<azonenberg>
Initial testing: plugged it into a 1000base-X SFP+ to generate test waveforms. It works fine in manual mode working with the front panel
<azonenberg>
when i connect glscopeclient i get some errors, it seems like i'm mis-parsing the list of installed software options
Degi_ has joined #scopehal
Degi has quit [Ping timeout: 260 seconds]
Degi_ is now known as Degi
vitalmixofnutrie has quit [Ping timeout: 258 seconds]
vitalmixofnutrie has joined #scopehal
apo has quit [Ping timeout: 260 seconds]
apo has joined #scopehal
StM has quit [Ping timeout: 260 seconds]
StM has joined #scopehal
<azonenberg>
welp, mso6 is definitely a bit buggy still
<monochroma>
:o
<azonenberg>
(the driver i mean)
<azonenberg>
on top of the instability scope side in the scpi stack i already knew about
<azonenberg>
i'm chasing several bugs where it seems like the data it can return to specific commands is different depending on unknown instrument state
<azonenberg>
also, the list of installed software options... *stab stab*
<azonenberg>
LeCroy is sane and just returns a list of alphanumeric option code
<azonenberg>
Meanwhile, you know what the MSO6 returns"
<azonenberg>
they use commas, colons, and semicolons as delimiters in different parts of the string
<azonenberg>
And those are legal non-escape characters in other parts :p
<azonenberg>
So you can't just split the string at commas and parse each option
<azonenberg>
you have to do a full context-sensitive decode of the whole thing because whether a comma is freeform description text or an option delimiter depends on how many semicolons have come before or after
<azonenberg>
at a high level Tek's SCPI API seems far better *designed* than LeCroy's, but LeCroy's *implementation* is much more stable and orthogonal. For the most part
electronic_eel has quit [Ping timeout: 260 seconds]
electronic_eel_ has joined #scopehal
<_whitenotifier>
[scopehal] azonenberg pushed 1 commit to master [+0/-0/±2] https://git.io/JLsoy
<_whitenotifier>
[scopehal] azonenberg 294de11 - Various bug fixes to MSO6 driver
<azonenberg>
So far I'm getting a disappointing 5 WFM/s from the MSO64 over gig-e with 2 channels @ 500 kS memory depth
<azonenberg>
i suspect there's lots of room to optimize though
<azonenberg>
actually, less than that. 268 WFM / 1 min = 4.46 WFM/s
<azonenberg>
Looking at the profiling report i spent 55.5 sec of that minute waiting on the socket
<_whitenotifier>
[scopehal] azonenberg pushed 1 commit to master [+0/-0/±1] https://git.io/JLsM8
<_whitenotifier>
[scopehal] azonenberg c03f7c3 - Initial implementation of write-only queueing for TektronixOscilloscope. See #179.
<_whitenotifier>
[scopehal] azonenberg pushed 2 commits to master [+0/-0/±3] https://git.io/JLGHZ
<_whitenotifier>
[scopehal] azonenberg 6aa7957 - Sanity checks re external trigger
<_whitenotifier>
[scopehal] azonenberg cc69d1c - TektronixOscilloscope: reverted nonblocking conversion due to instability
<azonenberg>
Holy moley mso6 is SLOW
<azonenberg>
I don't know how much i can do on this
<azonenberg>
T+ 0: arm trigger
<azonenberg>
T+ 5 ms: query trigger state
<azonenberg>
T+ 7.5 ms: TCP ACK
<azonenberg>
T+ 11.7 ms: query status returns 'triggered'
<azonenberg>
Then there's a 40ms delay for some reason, before I send the ACK at 51 ms (investigating this)
<azonenberg>
T+ 51.9 ms: scope finally sends the newline after the trigger status
<azonenberg>
T+ 52 ms: I start sending query commands
<azonenberg>
52.1 ms: I send the WFMO? query
<azonenberg>
52.2: ACK
<azonenberg>
57.8: first tiny segment of the reply
<azonenberg>
99.8: I ACK
<azonenberg>
it basically looks like the TCP is running in lockstep mode with only a single packet in flight. The window size coming from the instrument is 227 which is microscopic
<azonenberg>
And for some reason my computer is taking a long time to send ACKs
<azonenberg>
Which means the scope blocks the next segment for tens of ms at a time waiting for me to ack
<azonenberg>
This feels like a bad interaction between a poorly designed embedded tcp stack and tcp delayed ack on the linux side
<azonenberg>
and confirmed. With TCP_QUICKACK itt's about 6 ms from WFMO? to reply
<azonenberg>
vs nearly 100
<azonenberg>
monochroma: ^ check out this derpiness
<azonenberg>
Tek's braindead TCP/IP stack sends each of the fields in the WFMO? as a separate TCP segment
<azonenberg>
and can only have one segment in flight at a time
<azonenberg>
And it won't send another segment until the last one is ACKed
futarisIRCcloud has joined #scopehal
<monochroma>
oh wow
<azonenberg>
and apparently the thing is running linux of some sort
<azonenberg>
i have no idea how they screwed up linux networking this badly
<monochroma>
yeah wow
<monochroma>
the scope market is more screwed up than i thought :P
<apo>
azonenberg │ Holy moley mso6 is SLOW <- My rigol can tx something on the order of 50 kB/s over any channel (ethernet/USB host/USB device)
<apo>
this is suboptimal when you have 140 MB of data
<azonenberg>
apo: your rigol isn't a scope with a five figure price tag
<azonenberg>
i expected lecroy level performance off this
<apo>
Nope, just four figures
<azonenberg>
this is like tek's flagship mainstream scope
<azonenberg>
how is it this bad
<monochroma>
azonenberg: i guess this across the board testing of the network interfaces of scopes really shows that no one is using them as network attached ADCs :P
<azonenberg>
i really lucked out buying lecroy
<azonenberg>
lol
<monochroma>
apparently, good to know :P
<azonenberg>
I'm glad i didnt buy a mso6 not knowing what i was getting into
<_whitenotifier>
[scopehal] azonenberg pushed 1 commit to master [+0/-0/±4] https://git.io/JLGx9
<_whitenotifier>
[scopehal] azonenberg 4d9cee5 - Updated xptools. SCPIScocketTransport disables delayed-ACK by default. TektronixOscilloscope now uses 8 bit format for MSO6 waveform download.
<monochroma>
makes it feel more like a very expensive CRT era DSO with those capabilities
<azonenberg>
Yes
<azonenberg>
if you showed me that network capture i'd say it feels like it's running vxworks on a 68k or something, with a 10baseT interface
<azonenberg>
Not linux on an i5 with gig-e
<azonenberg>
i literally don't know what tcp flags i would have to set in order to make linux screw up networking this catastrophically
<monochroma>
watch as you find a teardown and somehow they stuffed a coldfire in there to be the network interface :P
<azonenberg>
Window size 227 bytes
<azonenberg>
seriously, how/why
<monochroma>
i wonder if they did something weird like use raw sockets and ported some code over from an embedded platform
<azonenberg>
oh, ok so that window size was my misreading
<azonenberg>
i had a truncated capture and missed the window scaling during startup
<d1b2>
<daveshah> something like a handwritten TCP/IP stack in Tcl 😄
<azonenberg>
no wait
<azonenberg>
yeah, ok so the actual window is 29056, it's a scale of 128
<monochroma>
daveshah: LOL
<azonenberg>
But that doesnt change the fact that it's lockstepping and only having one tcp segment in flight
<azonenberg>
interestingly it seems to use several segments when downloading waveform data
<azonenberg>
so i really dont know what they did
<azonenberg>
Here i was thinking i was going to be spending the evening doing performance tuning to get it up to 50+ WFM/s
<azonenberg>
and i'd have it doing all kinds of cool stuff alongside the lecroy scopes
<azonenberg>
instead i'm fighting with the thing choking on a CURV? query, hanging and then finally leaving an error in the log saying the requested channel isn't enabled
<azonenberg>
when i know full well it is
<monochroma>
how were they supposed to know anyone was actually going to use the ethernet port? :P
<azonenberg>
lol
<azonenberg>
Really makes me want to see if it's less of a trainwreck over USB
<azonenberg>
let me find a cable...
<azonenberg>
hmm, i dont seem to have any usb3 cables handy
<azonenberg>
at least not with B ports on them
<azonenberg>
guess i'm using usb
<azonenberg>
2
<azonenberg>
So when connected via usbtmc i get "data depleted" errors which i need to investigate
<azonenberg>
but only when downloading waveforms, other functionality seeeeems to work
<monochroma>
maybe the system is in a bad state after the ethernet testing?
<azonenberg>
no i rebooted it
<azonenberg>
i already crashed the lxi service once
<azonenberg>
lots of manual tuning to get to this point, super unstable, the auto sync wizard doesnt work
<azonenberg>
and obviously some DCD because of the different risetimes of the frontends
<azonenberg>
but... it works!
futarisIRCcloud has quit [Quit: Connection closed for inactivity]
Degi has quit [Ping timeout: 240 seconds]
Degi has joined #scopehal
rue_mohr has joined #scopehal
<d1b2>
<j4cbo> who does make good scopes in the 4-digit price range, anyway? I’ve been pretty frustrated by a number of things on my R&S RTB2K and will probably want to replace it someday
<monochroma>
azonenberg: hah!
<d1b2>
<mubes> Hanging my nose over a tek mso44...interested to hear what your experience of the mso6 is like. There are likely to be two rtb2004s on the second hand market soon.
<florolf_>
azonenberg: i think i might be able to dump /proc/sys/net / look what process is actually handling this traffic later this week if you really want to know
<florolf_>
the userland is based on some ubuntu lts, fwiw
florolf_ is now known as florolf
<d1b2>
<atx> Hm, is it normal that glscopeclient with DS1054Z has like 20 seconds latency?
<d1b2>
<atx> The poor thing is also trying to sing via its beeper
EmilJ has joined #scopehal
<monochroma>
atx: that sounds correct, iirc the rigols take forever to send data out
<d1b2>
<atx> Interesting
<d1b2>
<atx> For what its worth, the Rigol UltraScope app seems to be much snappier
<d1b2>
<atx> Looks like it reads 2000 bytes of waveform data for each channel
<d1b2>
<atx> Hm, I don't quite get how it works
<d1b2>
<atx> It just seems to read waveform data from 1 to 1200
<d1b2>
<atx> But, there has to be some sort of resampling involved
<d1b2>
<atx> Or maybe it just sets the memory depth to the minimum
<d1b2>
<atx> Can't quite figure out where though
<d1b2>
<atx> Right, I get it now. The :WAV:MODE command has multiple parameters "NORMal" reads downsampled data from the screen, while "RAW" reads data from the internal memory buffer
<d1b2>
<atx> Okay, patching scopehal to use the normal mode makes it work at ~1.4 WFM/s
<d1b2>
<atx> Well, at least using the interface is much less iritating
<d1b2>
<TiltMeSenpai> I wonder if SCPI is translated into VX11 or the other way around internally
<d1b2>
<atx> It might be faster, but this is about the same refresh rate that the Rigol UltraScope thing gets
<d1b2>
<TiltMeSenpai> I recall vx11 essentially encapsulating SCPI commands, but with some sort of async notification functions that not everything uses
<d1b2>
<atx> Yes, but from what I saw the Rigol app seems to do everything sequentially
<d1b2>
<TiltMeSenpai> yeah, in that case, we probably wouldn't get a speedup by using vx11
<d1b2>
<atx> I wonder if there is a way of getting the extended resolution bits
<d1b2>
<atx> Because now I am getting 8 bits per sample, while running in high res mode
<d1b2>
<TiltMeSenpai> the ones that are filtered out in NORM mode?
<d1b2>
<atx> Which should add a bit or two
<d1b2>
<TiltMeSenpai> if NORM is filtering stuff out, is there a way to have glscopeclient use NORM for free running mode and RAW for single shot triggers?
<d1b2>
<atx> Oh actually there is a special mode for that even
<d1b2>
<atx> MAX
<d1b2>
<atx> Which acts like NORM in run mode and like RAW in stop mode
<d1b2>
<TiltMeSenpai> oh wait dope
<d1b2>
<TiltMeSenpai> this sounds awesome, thank you for figuring that out
<d1b2>
<atx> But, glscopeclient seems to be stopping the scope for every read at the moment
<d1b2>
<atx> Which might be because that is the only way to get the data out in RAW mode
<d1b2>
<atx> I wonder what happens if I just throw all that away and let the thing run
<d1b2>
<atx> Now it's pinned at (very stable) 1.5 WFM/s
<d1b2>
<atx> iftop says I am reading at 62kb/s from the scope
<d1b2>
<atx> Seems like it should be able to do much more
<d1b2>
<atx> I might be able to stare at the TCP trace and figure out which commands take time
<d1b2>
<atx> The :WAV:PRE command takes ~50ms, not great
<d1b2>
<atx> Let's see what happens if I just hard code the results
<d1b2>
<atx> Okay, that gets us to 7.4WFM/s, modest but nice
<d1b2>
<TiltMeSenpai> that seems plenty reasonable
<d1b2>
<TiltMeSenpai> still with 1 channel hardcoded?
<d1b2>
<atx> Yes
<d1b2>
<atx> So it would be 4 times less with all channels unfortunately
<d1b2>
<TiltMeSenpai> maybe we figure out how to sample only selected channels, move :WAV:PRE to somewhere where it's not getting pinged every read, and leave it with a note saying "don't touch the scope interface if you're using this patch set"?
<d1b2>
<TiltMeSenpai> I don't know if there's a way to tell if someone messed with the scope dials without using :WAV:PRE, so it would probably do weird things if someone messed with the interface between those values getting updated
<d1b2>
<atx> So, I enabled the "WORD" mode, which makes it send 2 bytes per sample instead of one The update rate is unchanged and now it is sending 120kB/s
<d1b2>
<atx> So the Ethernet interface is not getting overloaded at least
<d1b2>
<atx> Maybe there is some way to get asynchronous notifications using the VXI-11 thing?
<d1b2>
<atx> About scope state changes?
<d1b2>
<atx> One may also use :SYST:LOCK ON to lock the interface :)
<d1b2>
<TiltMeSenpai> all I know about VXI-11 is it's supposedly some RPC thing. I assume that means there should be a way for the scope to notify you of changes, but I haven't read enough of the Sun RPC spec that it's supposedly based on to figure out where that occurs
<d1b2>
<TiltMeSenpai> and then there's the matter of if Rigol bothered to implement those functions
<d1b2>
<atx> There is a python library
<d1b2>
<TiltMeSenpai> does glscopeclient have scope specific settings? we could add a "interface locked" setting that would cause data acquisition to use cached :WAV:PRE values and set :SYST:LOCK ON maybe
<d1b2>
<TiltMeSenpai> yeah the python library doesn't look like it implements everything VX11 is capable of, mostly encapsulated SCPI stuff. There's some async functionality but I haven't figured that out in the couple hours I poked at that library
<d1b2>
<atx> Yeah, it also seems to be dead
<d1b2>
<atx> I seem to remember some more extensive thing
<d1b2>
<atx> pyvisa maybe
<d1b2>
<atx> glscopeclient is asking about trigger status, which does not seem to take long
<d1b2>
<atx> And it can be used to figure out the if the preambule needs to be updated
<d1b2>
<atx> But, there is still the issue of getting the ranges
<d1b2>
<atx> So, running just :WAV:DATA in a python script gives ~125ms per invocation
<d1b2>
<atx> Which about corresponds to what the patched glscopeclient is getting
<d1b2>
<atx> Running two simultaneous clients doubles the read time, so it is shared between clients
<d1b2>
<TiltMeSenpai> maybe ~7WF channels/S is as fast as we can go :/
<d1b2>
<TiltMeSenpai> still pretty usable
<d1b2>
<atx> Indeed
m4ssi has joined #scopehal
<azonenberg>
TiltMeSenpai: as of now i cache as much as i can
<azonenberg>
and provide a toolbar "flush cache" button if you make changes on the scope UI
<azonenberg>
if the rigol driver does not cache things it could, that's a bug
<d1b2>
<atx> Right, I see the code
<d1b2>
<atx> This should be reasonably easy to implement then
<d1b2>
<atx> I might get around to writing a proper patch this weekend
<azonenberg>
Great
<azonenberg>
I'm still fighting stability and other issues on the Tek MSO6 driver right now
<azonenberg>
i have it up to almost 10 WFM/s on a single channel if you get really lucky
<azonenberg>
but it seems to be unreliable
<azonenberg>
the multi scope flow is also buggy, but it was bad with lecroy scopes too
m4ssi has quit [Remote host closed the connection]