<pepijndevos>
sigh... even at 1k sample depth glscopeclient is borderline unusable. Pretty much any action hangs the ui for a second because it needs to acquire the lock that is held for receiving the next waveform.
<pepijndevos>
Maybe there should be more fifos? Seems bad in general that the UI acquires locks depending on IO, that's basically doing IO in the UI thread.
<azonenberg>
pepijndevos: the io done in the UI thread is supposed to be almost instantaneous
<azonenberg>
but rigol is sloooow
<pepijndevos>
well yea, but if that IO needs to wait on some other slow IO...
<miek>
this sounds like the driver still isn't caching all the meta stuff
<azonenberg>
queueing write-only operations to yet another thread might make sense, for things like start/stop
<azonenberg>
but yeah that sounds like missing caching
<azonenberg>
the intent is, during normal operation, for the UI thread to never touch the socket
<azonenberg>
it should only lock the cache mutex, which is needed because STL data structures like std::map are not atomic
<azonenberg>
but the scope thread should never hold that mutex for more than a few microseconds
<monochroma>
probably need to isolate the UI completely from IO tasks, UI threads only run when background IO threads have completed and are 100% ready for the UI to render something
<azonenberg>
it should do all of the I/O and *then* grab the mutex, update the cache, and immediately free
<azonenberg>
monochroma: the issue is that when you click e.g. start capture
<azonenberg>
that's an asynchronous event that goes direct to the scope driver
<azonenberg>
But it *should* be almost instantaneous
<azonenberg>
incidentally this is why all of the scopes i'm building have separate command and data streams on different sockets
<azonenberg>
pepijndevos: do more poking around in gdb
<azonenberg>
Look at where the UI thread is blocking
<pepijndevos>
azonenberg, happy to gdb more, but pretty sure I'll just find the scope thread in recv and the UI thread in whatever button I pressed.
<azonenberg>
pepijndevos: those two look like the big offenders
<azonenberg>
they're called from the right click menu to set checkboxes in the menu
<azonenberg>
among other thigns
<azonenberg>
So if they block it bogs down everything
<azonenberg>
Anybody have time to write a fix? i'm in the middle of some other stuff but can review a PR if somebody submits one
m4ssi has quit [Remote host closed the connection]
<azonenberg>
miek: ?
<azonenberg>
at some point we should do a general code quality review of a lot of the drivers
<azonenberg>
the LeCroy one is heavily optimized as i use it on a daily basis, most of the others are clearly not up to that level of quality/maturity
<miek>
mmmaybe, i don't have the new project structure setup yet
<pepijndevos>
What would be interesting is do the get config thing and see where it hangs until I hover the play button
<azonenberg>
pepijndevos: yeah
<pepijndevos>
I also managed to hang it hard by fiddling with the frontpanel witht he scope running remotely, but that might be expected.
<azonenberg>
Not expected, but not surprising either :p
<azonenberg>
i had a few crashes in the lecroy driver from that. i think i've fixed those
<azonenberg>
but it's easy to have bugs when settings change under your feet
<pepijndevos>
If it doesn;t block the Ui thread and has timeouts it should be fine. But it seems neither is fine.
<azonenberg>
Yeah
<azonenberg>
I pushed very hard on not blocking the UI thread in the lecroy driver
<azonenberg>
but it's very easy to write code that blocks if you're not careful about it
<pepijndevos>
How do you avoid that when you need to do IO? I would imagine either your API is blocking, or you go with futures and event loops and stuff, so now is this different per driver?
<azonenberg>
reads of settings should always cache
<azonenberg>
writes might block if an acquisition is in progress but this should be quick
<azonenberg>
longer term we might want to think about queueing writes that don't need an acknowledgement, like changing vertical scale
<azonenberg>
as a general rule i didn't do a lot of testing with scopes that take more than a couple of ms to respond to a command
<azonenberg>
Here's a concept: transports will have two functions for sending commands
<azonenberg>
the first one is for sending a command that needs a response
<pepijndevos>
Right, so to work properly with scopes that take a second to acquire a waveform... ouch
<azonenberg>
actually no we dont even need that
<azonenberg>
Just have SendCommand() use a mutex sync to write to a fifo of outbound commands, remove synchronization code from drivers around SendCommand() calls that aren't reading replies
<azonenberg>
(since you don't care when that write executes in relation to other stuff)
<pepijndevos>
Can you just open two sockets to the same scope?
<azonenberg>
No
<azonenberg>
Most scopes only support one concurrent connection
<azonenberg>
otherwise this would be 100x easier
<azonenberg>
I have ideas, let me open a ticket and we'll work on it
<_whitenotifier-b>
[scopehal] azonenberg opened issue #179: Add write queueing to transports for better handling of high-latency instruments - https://git.io/JJcOu
<_whitenotifier-b>
[scopehal] azonenberg labeled issue #179: Add write queueing to transports for better handling of high-latency instruments - https://git.io/JJcOu
<azonenberg>
but hey, thanks for being a beta tester and finding an area we need to improve on :
<azonenberg>
:)
<pepijndevos>
Maybe there should be a single command fifo with a thread executing commands and use futures to notify the sender of the response. the UI is probably event based anyway
<azonenberg>
relatively few if any UI accesses need to round trip to the scope
<azonenberg>
almost all queries need an immediate response and it's OK if it's a bit stale
<azonenberg>
and most commands can be deferred until the current waveform is done with no problem
<azonenberg>
but shouldn't block the UI until that point
<azonenberg>
Write command queueing should solve this nicely
<pepijndevos>
right okay
<azonenberg>
It's just going to require a fair bit of refactoring across many drivers
<pepijndevos>
hrm...
<azonenberg>
Not something that will happen quickly. easily a day or two of work
<azonenberg>
But this is one of the reasons why i'm holding off on even considering a general release, publishing binaries, etc
<pepijndevos>
I have some holiday in september, who knows...
<azonenberg>
i know there are still major changes to be made and breaking refactoring to be done
<azonenberg>
I want to do it myself, this is a major architectural change that touches delicate parts of the whole project
<azonenberg>
i know that code better than anyone else right now
<azonenberg>
Just need a bit of time
<azonenberg>
Maybe i can work on it later this week
<azonenberg>
Longer term i'd like to refactor this into a more general attribute framework that doesnt need every driver to manage their own caching
<azonenberg>
there's a bunch of duplicated code i think can be cleaned up and unified
<Degi>
yess
<azonenberg>
but command queueing comes first and that will touch a lot of stuff too
<miek>
something else to think about longer term: how do we invalidate that cache? is there even a good way?
<Degi>
SetChannelAttenuation is a stub
<azonenberg>
miek: No, there's not
<azonenberg>
That's what the "refresh" button is for
<azonenberg>
time based will lead to random lag spikes, i tried that and it was really disconcerting
<azonenberg>
i update the cache with metadata from waveform headers every trigger if it's included in the wavedesc
<azonenberg>
the PROPER solution is for the scope (if it has front panel controls at all) to *push* notifications about updates to the client when you move a knob
<azonenberg>
basically all of the scpi apis for these scopes are not designed for performance
<azonenberg>
they're designed for 1970s era gpib scripting
<pepijndevos>
lemme guess, no scope ever has a push api?
<azonenberg>
nope
<azonenberg>
you know how i figure out if there's a new waveform to display?
<azonenberg>
polling a status register in a loop
<Degi>
Wait didnt I implmeent that
<azonenberg>
on every scope i've checked
<Degi>
Hey who deleted my never-ending switch case statements
<azonenberg>
Degi: was there some merge conflicting going on or something?
<azonenberg>
look at history
<azonenberg>
i know there were some fixes to the rigol driver recently and somebody might have screwed up a merge
<Degi>
Can I just copy paste it from the copy I have of it
<pepijndevos>
New proposal: find a buffer overflow and inject working code in the scope
<Degi>
lol
<azonenberg>
Degi: lol sure
<azonenberg>
pepijndevos: lol
<azonenberg>
there are actually some... siglent? scopes people have written open firmware for iirc
<azonenberg>
pepijndevos: and ziggggggy was working on a C# API for the lecroy scopes that would run on the scope and we could write our own, sane, remote control server in C#
<azonenberg>
bypassing the maui scpi stack entirely
<miek>
my master will be way out of date, the change was on the branch rigol_caching
<Degi>
Wait wtf did I git clone
<azonenberg>
status of that is apparently up in the air as he no longer works at lecroy, although apparently management planned to release it anyway
<Degi>
Ok whatever I cloned it again its there now
<Degi>
Truly some galaxy brain code //+ 0.1 in case atten is for example 0.049999 or so, to round it to 0.05 which turns to an int of 500
<miek>
oops, i accidentally bought another bench multimeter
<Degi>
Ahh I should get a proper multimeter sometime
<azonenberg>
miek: which one?
<miek>
agilent 34410a
<azonenberg>
ah 6 1/2 digit?
<azonenberg>
I'm pretty happy with my R&S HMC8012s, 5 3/4 digit, although i think you can probably get better bang for the buck by now
<miek>
yup. i'm most excited about an ethernet port on the back though!
<azonenberg>
the nice thing about them is they're smallish and fit next to my power supplies
<azonenberg>
scopehal supports DMMs
<azonenberg>
would love a driver
<azonenberg>
although there isnt yet much of a UI for them
<azonenberg>
but the APIs exist and i've scripted some experiments using it
<miek>
cool. yeah, i'll probably write something at some point
<azonenberg>
I also have the early beginnings of a power supply control utility
<miek>
i've got a keithley 6.5 digit meter already, but i've been too lazy to do anything with the rs232/gpib on it
<pepijndevos>
azonenberg, the efforts to unlock the rigol scopes seem to have resulted in ways to get ssh on the scope and such. Probably trivial to run a daemon on the scope, but an api, not so much probably...
<azonenberg>
The other thing that needs to happen is a function generator UI, and we need to figure out how to integrate it with glscopeclient
<Degi>
So how was that with mutex and so on
<azonenberg>
because most scopes don't support multiple concurrent connections
<azonenberg>
Which means you cant just have one socket for the func gen and one for the scope function
<azonenberg>
Degi: you should acquire m_mutex when accessing the socket
<azonenberg>
release it as soon as you're done doing network operations
<Degi>
and m_cacheMutex for cache?
<azonenberg>
acquire m_cacheMutex when reading or writing the cache
<Degi>
The lock_guard does that for me automatically and releases after the function ends?
<azonenberg>
Correct
<Degi>
So theres like lock_guard<recursive_mutex> lock2(m_mutex);
<azonenberg>
But you can create a new scope with curly braces to hold it for a shorter time
<pepijndevos>
azonenberg, maybe another reason to have a single socket thread with a queue and async api, so both applications can send it commands simultaneausly
<azonenberg>
pepijndevos: well it's more in terms of higher level structure
<azonenberg>
both "apps" would have to be in the same thread
<azonenberg>
the same application*
<Degi>
WE could have a server and then gui clients heh
<azonenberg>
But then how do we handle standalone function generators?
<Degi>
like cups
<azonenberg>
Degi: so i actually consiered that
<azonenberg>
some kind of multi client proxy
<azonenberg>
i honestly think that is the cleanest solution
<azonenberg>
just havent had time to define the interface for it
<Degi>
I guess VNC works too heh
<Degi>
Yeah, like minecraft...
<azonenberg>
anyway minimize time holding the cache mutex as this can block the UI thead, and especially avoid holding it when the user socket is held too
<Degi>
SO it writes to the cache later on, is it in this case right that its held for the whole call
<pepijndevos>
You could maybe have a veeery generic command multiplexer daemon that acts as a scope that can accept multiple commands, but serializes them to the scope over one socket
<azonenberg>
Degi: no, that's bad. because now while waiting for the reply from DISP? another thread can't call a get*() function
<azonenberg>
pepijndevos: yes the challenge is to handle multiple request/response pairs
<azonenberg>
and not confusing them
<azonenberg>
you'd have to have higher level scpi protocol awareness i think
<azonenberg>
when yousee a ? command hold until you get a reply
<azonenberg>
basically
<Degi>
How do I do it with for only some time? Replace lock_guard<recursive_mutex> lock2(m_mutex); with lock_guard<recursive_mutex> lock2(m_mutex) {stuff}?
<pepijndevos>
azonenberg, yea it'd have to be at the scpi level for sure
<azonenberg>
Degi: see LeCroyOscilloscope::GetChannelVoltageRange()
<azonenberg>
for a good example
<azonenberg>
i acquire the cache mutex, check if i have a hit
<azonenberg>
if miss, free it, acquire the main mutex, do the network stuff
<Degi>
ahh
<azonenberg>
Then acquire the cache mutex just long enough to update the cache
<azonenberg>
then free everything
<azonenberg>
Holding the cache mutex momentarily while the main mutex is locked is fine
<azonenberg>
what you want to avoid is blocking on attempting to get the main mutex while the cache mutex is already held
<Degi>
Thx
<azonenberg>
again this needs to become a more generic settings framework longer term
<pepijndevos>
This would also simplify the current command queue complications. the rigol driver could just have two sockets for commands and data, and the siggen could have a third. no queue neede in the ui
<azonenberg>
caching should be done in the base Oscilloscope class
<Degi>
So if I have cache mutex and do network req it hangs till thats done
<azonenberg>
Yeah. Avoid that
<azonenberg>
pepijndevos: well the thing is, this will also add latency and require you run a server on your local box
<azonenberg>
i want to avoid requiring such a proxy
<azonenberg>
It's something i would allow you to use if you want to use the func gen and scope features on a combined instrument
<azonenberg>
but not something that should be required to only use one chunk of an instrument
<azonenberg>
initial area for MAXWELL firmware so far
<azonenberg>
Brown = xilinx DDR controller (it's massive, lol)
<azonenberg>
pink = debug logic that wont be in the final design
<azonenberg>
blue = input serialization and buffering
<azonenberg>
green = ethernet and tcp/ip
<azonenberg>
yellow = triggering
<azonenberg>
no actual compression or saving of sample data is implemented yet
<azonenberg>
the dram controller just refreshes forever, and the network stack responds to pings, accepts TCP connections on 5025, and bridges that out to a UART going to the stm32
<azonenberg>
the trigger logic just outputs intermediate status to a debug IP
<azonenberg>
Overall i'm at 22% LUT, 8% FF, 15% BRAM, 85% IO right now
<azonenberg>
The IP stack is 1G only, no 10G/40G support yet
<azonenberg>
no 10 Gsps channel support yet
<azonenberg>
no trig in/out, no pps, lots of more niche features not implemented
<azonenberg>
Just getting started, basically trying to sanity check the pinout and clocking structure before finalizing th ePCB
<azonenberg>
so far it all looks solid
<azonenberg>
Except the trigger logic is failing timing slightly so i have to optimize it more
<Degi>
And gets the response of 1078 bytes length at 5.7837
<Degi>
Which is some way after 5.7025
<Degi>
;(
<Degi>
The connection is over a shared gigabit switch
<azonenberg>
Sounds like the firmware is stupidly slow then
<azonenberg>
and there's nothing we can really do but heavy caching to keep the UI responsive
<azonenberg>
and minimize time spent waiting on network traffic that isnt absolutely essential
<azonenberg>
Degi: you see why BLONDEL is doing all of the waveform processing in FPGA now? :p
<Degi>
I can make a damn UART go faster .-.
<noopwafel>
my code for the ds1054z grabs 250k points per request, I think you can do more also
<Degi>
Yes that decreases the speed by like 20% between 1k and 250 k
<Degi>
Like wtf is the scope doing
<noopwafel>
but I think it's clear that 'responsiveness' is not exactly on Rigol's priority list, see also: the scope UI
<Degi>
Hmh is it slow?
<Degi>
Lets just make our own scope but add FibreChannel ha
<noopwafel>
there's usually like 0.5s lag on the ds1054z ui :x try e.g. moving the vertical
<noopwafel>
anyway this is kind of off-topic :p but yeah the Rigol network interface is sadly frustratingly slow
<Degi>
On the MSO5 its solved by freezing the waveform and moving it graphically and when done sampling again
<miek>
yeah, the slow rigol UI was what pushed me towards getting an old agilent instead
<miek>
i had the fancier version of this scope in an old job, and i guess i was spoiled by it :p
<azonenberg>
miek: yeah even the lower end lecroy stuff (rebadged siglent) is a bit slow
<azonenberg>
like wavesurfer 3000 etc
<azonenberg>
i'm spoiled by working on scopes with five-digit price tags lol
<Degi>
Hmm I have a tektronix, sadly its broke
<noopwafel>
I mean there are plenty of second-hand scopes in the <$5k range which manage this fine
<noopwafel>
rigol situation is just a bit frustrating since they're everywhere+affordable
<azonenberg>
Yeah
<azonenberg>
And they're awful
<azonenberg>
that's what BLONDEL etc are meant to address, although not as cheaply as rigol
<Degi>
Wanna make PCIe scope when this thing works out for really cheap
<noopwafel>
yeah I mean, I love picoscope for <$1k
<Degi>
HMCAD1511, some cheaping out on analog components could give 4 ch for like 300 €
<miek>
i mean, there's some issues with them, but i think the 1054z is still a great deal for what it is
<Degi>
They all kinda waste on having a screen
<Degi>
If it is a PCIe card: No buttons, no screen, no PSU, no case (besides a metal bracket), no housing
<noopwafel>
azonenberg: do you have a BOM estimate for BLONDEL?
<noopwafel>
ah nm found it in logs, 1k-1.5k
<Degi>
Oh and you dont need RAM
<noopwafel>
(want)
<azonenberg>
noopwafel: Yeah it's next on the list after MAXWELL is done
<azonenberg>
I pushed it ahead because i had somebody who wanted to buy one right away, and because it wasn't blocking on corgi to finish the probe power work
<noopwafel>
makes sense
<azonenberg>
And my good scope is out for service right now which is a partial blocker to BLONDEL characterization
<noopwafel>
I do miss having knobs
<azonenberg>
sooooo
<noopwafel>
when vacation happens (aug?) I will build something with potentiometers in a laser-cut case and see how satisfying that is
<azonenberg>
that brings me to an idea i came up with
<Degi>
Some kinda USB device with knobs and E Ink displays as labels would be nice
<azonenberg>
i wanted to build something similar to what SEMs use for their controls
<Degi>
A lan cable?
<Degi>
:P
<azonenberg>
a usb attached gizmo with channel select buttons (maybe a touchscreen that shows actual channel names or something?) and encoders for selecting scale/offset
<azonenberg>
that you could connect to glscopeclient
<noopwafel>
ah yeah
<Degi>
Hm yes something that outputs keycode impulses or acts as a joystick
<Degi>
And not only glscopeclient would be nice... Like actually putting e ink there or so heh
<azonenberg>
Yeah. It's been on my mental roadmap for a while but i havent had time to think of actual requirements yet
<noopwafel>
yeah I was thinking potentiometers plus a Teensy or so, for the cheap-and-easy hack, but it'd be lovely to have something with display etc too
<Degi>
Why not rotary encoders
<Degi>
e ink cost monies .-.
<miek>
i wouldn't be surprised if the HID spec has an entry for this
<miek>
..right after the magic carpet controller
<noopwafel>
so I was originally looking for midi controllers
<noopwafel>
I figured there'd be something widely-available with a bunch of knobs that I could just buy
<noopwafel>
buttons with lights so you can do channel on/off, etc
<Degi>
Yes like 2 RGB LEDs on each button / slider / knob heh
<miek>
ooh, that's neat. i like the encoder & led combo. i tried to use something like that before but it had pots so syncronising with the PC was annoying
<Degi>
Rotary encoder with button would be nice
<Degi>
Or double rotary encoder like on some scopes. Sometimes you can pull them out too to activate a switch
<noopwafel>
seems like it's all controllable over midi, I'm tempted to just grab one