azonenberg changed the topic of #scopehal to: libscopehal, libscopeprotocols, and glscopeclient development and testing | Online hackathon December 19th all day | | Logs:
<codysseus> Nice!
<azonenberg> Initial testing: plugged it into a 1000base-X SFP+ to generate test waveforms. It works fine in manual mode working with the front panel
<azonenberg> when i connect glscopeclient i get some errors, it seems like i'm mis-parsing the list of installed software options
Degi_ has joined #scopehal
Degi has quit [Ping timeout: 260 seconds]
Degi_ is now known as Degi
vitalmixofnutrie has quit [Ping timeout: 258 seconds]
vitalmixofnutrie has joined #scopehal
apo has quit [Ping timeout: 260 seconds]
apo has joined #scopehal
StM has quit [Ping timeout: 260 seconds]
StM has joined #scopehal
<azonenberg> welp, mso6 is definitely a bit buggy still
<monochroma> :o
<azonenberg> (the driver i mean)
<azonenberg> on top of the instability scope side in the scpi stack i already knew about
<azonenberg> i'm chasing several bugs where it seems like the data it can return to specific commands is different depending on unknown instrument state
<azonenberg> also, the list of installed software options... *stab stab*
<azonenberg> LeCroy is sane and just returns a list of alphanumeric option code
<azonenberg> Meanwhile, you know what the MSO6 returns"
<azonenberg> OptionCode : OptionType ; Description (may include commas ; LicenseType ,
<azonenberg> they use commas, colons, and semicolons as delimiters in different parts of the string
<azonenberg> And those are legal non-escape characters in other parts :p
<azonenberg> So you can't just split the string at commas and parse each option
<azonenberg> you have to do a full context-sensitive decode of the whole thing because whether a comma is freeform description text or an option delimiter depends on how many semicolons have come before or after
<azonenberg> at a high level Tek's SCPI API seems far better *designed* than LeCroy's, but LeCroy's *implementation* is much more stable and orthogonal. For the most part
electronic_eel has quit [Ping timeout: 260 seconds]
electronic_eel_ has joined #scopehal
<_whitenotifier> [scopehal] azonenberg pushed 1 commit to master [+0/-0/±2]
<_whitenotifier> [scopehal] azonenberg 294de11 - Various bug fixes to MSO6 driver
<azonenberg> So far I'm getting a disappointing 5 WFM/s from the MSO64 over gig-e with 2 channels @ 500 kS memory depth
<azonenberg> i suspect there's lots of room to optimize though
<azonenberg> actually, less than that. 268 WFM / 1 min = 4.46 WFM/s
<azonenberg> Looking at the profiling report i spent 55.5 sec of that minute waiting on the socket
<_whitenotifier> [scopehal] azonenberg pushed 1 commit to master [+0/-0/±1]
<_whitenotifier> [scopehal] azonenberg 9214fb6 - TektronixOscilloscope: optimized MSO6 waveform download
<azonenberg> 4.73 WFM/s, still awful but better i guess :p
vitalmixofnutrie has quit [Ping timeout: 268 seconds]
d1b2 has quit [Remote host closed the connection]
d1b2 has joined #scopehal
juli966 has joined #scopehal
<_whitenotifier> [scopehal-docs] azonenberg pushed 1 commit to master [+0/-0/±2]
<_whitenotifier> [scopehal-docs] azonenberg ab008c1 - Documented instability on Tek MSO6 with LAN transport
<_whitenotifier> [scopehal] azonenberg pushed 2 commits to master [+0/-0/±7]
<_whitenotifier> [scopehal] azonenberg 4f97508 - TektronixOscilloscope: AVX2 waveform conversion
<_whitenotifier> [scopehal] azonenberg c03f7c3 - Initial implementation of write-only queueing for TektronixOscilloscope. See #179.
<_whitenotifier> [scopehal] azonenberg pushed 2 commits to master [+0/-0/±3]
<_whitenotifier> [scopehal] azonenberg 6aa7957 - Sanity checks re external trigger
<_whitenotifier> [scopehal] azonenberg cc69d1c - TektronixOscilloscope: reverted nonblocking conversion due to instability
<azonenberg> Holy moley mso6 is SLOW
<azonenberg> I don't know how much i can do on this
<azonenberg> T+ 0: arm trigger
<azonenberg> T+ 5 ms: query trigger state
<azonenberg> T+ 7.5 ms: TCP ACK
<azonenberg> T+ 11.7 ms: query status returns 'triggered'
<azonenberg> Then there's a 40ms delay for some reason, before I send the ACK at 51 ms (investigating this)
<azonenberg> T+ 51.9 ms: scope finally sends the newline after the trigger status
<azonenberg> T+ 52 ms: I start sending query commands
<azonenberg> 52.1 ms: I send the WFMO? query
<azonenberg> 52.2: ACK
<azonenberg> 57.8: first tiny segment of the reply
<azonenberg> 99.8: I ACK
<azonenberg> it basically looks like the TCP is running in lockstep mode with only a single packet in flight. The window size coming from the instrument is 227 which is microscopic
<azonenberg> And for some reason my computer is taking a long time to send ACKs
<azonenberg> Which means the scope blocks the next segment for tens of ms at a time waiting for me to ack
<azonenberg> This feels like a bad interaction between a poorly designed embedded tcp stack and tcp delayed ack on the linux side
electronic_eel_ has quit [Quit: - Chat comfortably. Anywhere.]
electronic_eel has joined #scopehal
<azonenberg> and confirmed. With TCP_QUICKACK itt's about 6 ms from WFMO? to reply
<azonenberg> vs nearly 100
<azonenberg> monochroma: ^ check out this derpiness
<azonenberg> Tek's braindead TCP/IP stack sends each of the fields in the WFMO? as a separate TCP segment
<azonenberg> and can only have one segment in flight at a time
<azonenberg> And it won't send another segment until the last one is ACKed
futarisIRCcloud has joined #scopehal
<monochroma> oh wow
<azonenberg> and apparently the thing is running linux of some sort
<azonenberg> i have no idea how they screwed up linux networking this badly
<monochroma> yeah wow
<monochroma> the scope market is more screwed up than i thought :P
<apo> azonenberg │ Holy moley mso6 is SLOW <- My rigol can tx something on the order of 50 kB/s over any channel (ethernet/USB host/USB device)
<apo> this is suboptimal when you have 140 MB of data
<azonenberg> apo: your rigol isn't a scope with a five figure price tag
<azonenberg> i expected lecroy level performance off this
<apo> Nope, just four figures
<azonenberg> this is like tek's flagship mainstream scope
<azonenberg> how is it this bad
<monochroma> azonenberg: i guess this across the board testing of the network interfaces of scopes really shows that no one is using them as network attached ADCs :P
<azonenberg> i really lucked out buying lecroy
<azonenberg> lol
<monochroma> apparently, good to know :P
<azonenberg> I'm glad i didnt buy a mso6 not knowing what i was getting into
<_whitenotifier> [scopehal] azonenberg pushed 1 commit to master [+0/-0/±4]
<_whitenotifier> [scopehal] azonenberg 4d9cee5 - Updated xptools. SCPIScocketTransport disables delayed-ACK by default. TektronixOscilloscope now uses 8 bit format for MSO6 waveform download.
<monochroma> makes it feel more like a very expensive CRT era DSO with those capabilities
<azonenberg> Yes
<azonenberg> if you showed me that network capture i'd say it feels like it's running vxworks on a 68k or something, with a 10baseT interface
<azonenberg> Not linux on an i5 with gig-e
<azonenberg> i literally don't know what tcp flags i would have to set in order to make linux screw up networking this catastrophically
<monochroma> watch as you find a teardown and somehow they stuffed a coldfire in there to be the network interface :P
<azonenberg> Window size 227 bytes
<azonenberg> seriously, how/why
<monochroma> i wonder if they did something weird like use raw sockets and ported some code over from an embedded platform
<azonenberg> oh, ok so that window size was my misreading
<azonenberg> i had a truncated capture and missed the window scaling during startup
<d1b2> <daveshah> something like a handwritten TCP/IP stack in Tcl 😄
<azonenberg> no wait
<azonenberg> yeah, ok so the actual window is 29056, it's a scale of 128
<monochroma> daveshah: LOL
<azonenberg> But that doesnt change the fact that it's lockstepping and only having one tcp segment in flight
<azonenberg> interestingly it seems to use several segments when downloading waveform data
<azonenberg> so i really dont know what they did
<azonenberg> Here i was thinking i was going to be spending the evening doing performance tuning to get it up to 50+ WFM/s
<azonenberg> and i'd have it doing all kinds of cool stuff alongside the lecroy scopes
<azonenberg> instead i'm fighting with the thing choking on a CURV? query, hanging and then finally leaving an error in the log saying the requested channel isn't enabled
<azonenberg> when i know full well it is
<monochroma> how were they supposed to know anyone was actually going to use the ethernet port? :P
<azonenberg> lol
<azonenberg> Really makes me want to see if it's less of a trainwreck over USB
<azonenberg> let me find a cable...
<azonenberg> hmm, i dont seem to have any usb3 cables handy
<azonenberg> at least not with B ports on them
<azonenberg> guess i'm using usb
<azonenberg> 2
<azonenberg> So when connected via usbtmc i get "data depleted" errors which i need to investigate
<azonenberg> but only when downloading waveforms, other functionality seeeeems to work
<monochroma> maybe the system is in a bad state after the ethernet testing?
<azonenberg> no i rebooted it
<azonenberg> i already crashed the lxi service once
juli966 has quit [Quit: Nettalk6 -]
<monochroma> heh
juli966 has joined #scopehal
<azonenberg> lots of manual tuning to get to this point, super unstable, the auto sync wizard doesnt work
<azonenberg> and obviously some DCD because of the different risetimes of the frontends
<azonenberg> but... it works!
futarisIRCcloud has quit [Quit: Connection closed for inactivity]
Degi has quit [Ping timeout: 240 seconds]
Degi has joined #scopehal
rue_mohr has joined #scopehal
<d1b2> <j4cbo> who does make good scopes in the 4-digit price range, anyway? I’ve been pretty frustrated by a number of things on my R&S RTB2K and will probably want to replace it someday
<monochroma> azonenberg: hah!
<d1b2> <mubes> Hanging my nose over a tek mso44...interested to hear what your experience of the mso6 is like. There are likely to be two rtb2004s on the second hand market soon.
<florolf_> azonenberg: i think i might be able to dump /proc/sys/net / look what process is actually handling this traffic later this week if you really want to know
<florolf_> the userland is based on some ubuntu lts, fwiw
florolf_ is now known as florolf
<d1b2> <atx> Hm, is it normal that glscopeclient with DS1054Z has like 20 seconds latency?
<d1b2> <atx> The poor thing is also trying to sing via its beeper
EmilJ has joined #scopehal
<monochroma> atx: that sounds correct, iirc the rigols take forever to send data out
<d1b2> <atx> Interesting
<d1b2> <atx> For what its worth, the Rigol UltraScope app seems to be much snappier
<d1b2> <atx> Not sure how it does that
<monochroma> i specifically rememebr the ethernet interfaced stuff, not sure about USB
<d1b2> <atx> This is over ethernet
<d1b2> <atx> I would say latency is under a second
<miek> afaik it streams the display rather than the actual data
<d1b2> <atx> (the youtube video is over USB, but I am testing over Ethernet at the moment and it works the same way)
<d1b2> <atx> That seems like a possibility
<d1b2> <dale6998> @atx that avatar, yoooooo
<d1b2> <atx> But, honestly, the rendering in the application looks worse than what is displayed on the builtin screen
<d1b2> <dale6998> awesome
<d1b2> <atx> @dale6998 From Tao's Analysis. I think around a third of the propositions therein are left as exercises :P
<d1b2> <atx> Let me check the traffic with wireshark
<d1b2> <atx> I seem to remember reading the DS1054Z SCPI manual and I do not think there was any "screen streaming" option
<d1b2> <atx> Maybe its just undocumented
<monochroma> yeah, would be interesting if there was secretly a faster way to get waveforms out
<d1b2> <atx> Looks like its just using bog-standard VXI-11
<d1b2> <atx> Not really sure what is the difference between that and SCPI
<d1b2> <atx> Looks like it reads 2000 bytes of waveform data for each channel
<d1b2> <atx> Hm, I don't quite get how it works
<d1b2> <atx> It just seems to read waveform data from 1 to 1200
<d1b2> <atx> But, there has to be some sort of resampling involved
<d1b2> <atx> Or maybe it just sets the memory depth to the minimum
<d1b2> <atx> Can't quite figure out where though
<d1b2> <atx> Right, I get it now. The :WAV:MODE command has multiple parameters "NORMal" reads downsampled data from the screen, while "RAW" reads data from the internal memory buffer
<d1b2> <atx> Okay, patching scopehal to use the normal mode makes it work at ~1.4 WFM/s
<d1b2> <atx> It's somewhat low-res though...
futarisIRCcloud has joined #scopehal
<d1b2> <atx> Well, at least using the interface is much less iritating
<d1b2> <TiltMeSenpai> I wonder if SCPI is translated into VX11 or the other way around internally
<d1b2> <atx> It might be faster, but this is about the same refresh rate that the Rigol UltraScope thing gets
<d1b2> <TiltMeSenpai> I recall vx11 essentially encapsulating SCPI commands, but with some sort of async notification functions that not everything uses
<d1b2> <atx> Yes, but from what I saw the Rigol app seems to do everything sequentially
<d1b2> <TiltMeSenpai> yeah, in that case, we probably wouldn't get a speedup by using vx11
<d1b2> <atx> I wonder if there is a way of getting the extended resolution bits
<d1b2> <atx> Because now I am getting 8 bits per sample, while running in high res mode
<d1b2> <TiltMeSenpai> the ones that are filtered out in NORM mode?
<d1b2> <atx> Which should add a bit or two
<d1b2> <TiltMeSenpai> if NORM is filtering stuff out, is there a way to have glscopeclient use NORM for free running mode and RAW for single shot triggers?
<d1b2> <atx> Oh actually there is a special mode for that even
<d1b2> <atx> MAX
<d1b2> <atx> Which acts like NORM in run mode and like RAW in stop mode
<d1b2> <TiltMeSenpai> oh wait dope
<d1b2> <TiltMeSenpai> this sounds awesome, thank you for figuring that out
<d1b2> <atx> But, glscopeclient seems to be stopping the scope for every read at the moment
<d1b2> <atx> Which might be because that is the only way to get the data out in RAW mode
<d1b2> <atx> I wonder what happens if I just throw all that away and let the thing run
<d1b2> <atx> Now it's pinned at (very stable) 1.5 WFM/s
<d1b2> <atx> iftop says I am reading at 62kb/s from the scope
<d1b2> <atx> Seems like it should be able to do much more
<d1b2> <atx> This is also not ideal
<d1b2> <atx> Forcing it to read from just a single channel yields 6 WFM/s
<d1b2> <atx> I wonder if using multiple connections in parallel would make this faster
<d1b2> <atx> Seems unlikely, but I am still bothered by the fact that it can only do like 60kB/s over that 100Mb/s Ethernet port
<d1b2> <TiltMeSenpai> do we know where it's choking? if the scope side is struggling, the mso5k doesn't even have a multi-core processor, does it
<d1b2> <atx> No idea, I don't even know how the hardware is designed
<d1b2> <atx> iirc the more expensive Rigol scopes have Zynqs
<d1b2> <atx> Which should be completely capable of saturating a 100Mb/s port
<d1b2> <atx> But, I would guess the DS1054Z has some higher-end-ish micro
<d1b2> <atx> I might be able to stare at the TCP trace and figure out which commands take time
<d1b2> <atx> The :WAV:PRE command takes ~50ms, not great
<d1b2> <atx> Let's see what happens if I just hard code the results
<d1b2> <atx> Okay, that gets us to 7.4WFM/s, modest but nice
<d1b2> <TiltMeSenpai> that seems plenty reasonable
<d1b2> <TiltMeSenpai> still with 1 channel hardcoded?
<d1b2> <atx> Yes
<d1b2> <atx> So it would be 4 times less with all channels unfortunately
<d1b2> <TiltMeSenpai> maybe we figure out how to sample only selected channels, move :WAV:PRE to somewhere where it's not getting pinged every read, and leave it with a note saying "don't touch the scope interface if you're using this patch set"?
<d1b2> <TiltMeSenpai> I don't know if there's a way to tell if someone messed with the scope dials without using :WAV:PRE, so it would probably do weird things if someone messed with the interface between those values getting updated
<d1b2> <atx> So, I enabled the "WORD" mode, which makes it send 2 bytes per sample instead of one The update rate is unchanged and now it is sending 120kB/s
<d1b2> <atx> So the Ethernet interface is not getting overloaded at least
<d1b2> <atx> Maybe there is some way to get asynchronous notifications using the VXI-11 thing?
<d1b2> <atx> About scope state changes?
<d1b2> <atx> One may also use :SYST:LOCK ON to lock the interface :)
<d1b2> <TiltMeSenpai> all I know about VXI-11 is it's supposedly some RPC thing. I assume that means there should be a way for the scope to notify you of changes, but I haven't read enough of the Sun RPC spec that it's supposedly based on to figure out where that occurs
<d1b2> <TiltMeSenpai> and then there's the matter of if Rigol bothered to implement those functions
<d1b2> <atx> There is a python library
<d1b2> <TiltMeSenpai> does glscopeclient have scope specific settings? we could add a "interface locked" setting that would cause data acquisition to use cached :WAV:PRE values and set :SYST:LOCK ON maybe
<d1b2> <TiltMeSenpai> yeah the python library doesn't look like it implements everything VX11 is capable of, mostly encapsulated SCPI stuff. There's some async functionality but I haven't figured that out in the couple hours I poked at that library
<d1b2> <atx> Yeah, it also seems to be dead
<d1b2> <atx> I seem to remember some more extensive thing
<d1b2> <atx> pyvisa maybe
<d1b2> <atx> glscopeclient is asking about trigger status, which does not seem to take long
<d1b2> <atx> And it can be used to figure out the if the preambule needs to be updated
<d1b2> <atx> But, there is still the issue of getting the ranges
<d1b2> <atx> So, running just :WAV:DATA in a python script gives ~125ms per invocation
<d1b2> <atx> Which about corresponds to what the patched glscopeclient is getting
<d1b2> <atx> Running two simultaneous clients doubles the read time, so it is shared between clients
<d1b2> <TiltMeSenpai> maybe ~7WF channels/S is as fast as we can go :/
<d1b2> <TiltMeSenpai> still pretty usable
<d1b2> <atx> Indeed
m4ssi has joined #scopehal
<azonenberg> TiltMeSenpai: as of now i cache as much as i can
<azonenberg> and provide a toolbar "flush cache" button if you make changes on the scope UI
<azonenberg> if the rigol driver does not cache things it could, that's a bug
<d1b2> <atx> Right, I see the code
<d1b2> <atx> This should be reasonably easy to implement then
<d1b2> <atx> I might get around to writing a proper patch this weekend
<azonenberg> Great
<azonenberg> I'm still fighting stability and other issues on the Tek MSO6 driver right now
<azonenberg> i have it up to almost 10 WFM/s on a single channel if you get really lucky
<azonenberg> but it seems to be unreliable
<azonenberg> the multi scope flow is also buggy, but it was bad with lecroy scopes too
m4ssi has quit [Remote host closed the connection]