<whitequark>
i think the reason your code doesn't work that well is that you're doing a lot more than necessary
<d1b2>
<Attie> the plan with the checksum was to be more confident in what's going on
<whitequark>
don't add checksums (USB has them already), don't process sample buffers in Python, *especially* don't copy samples (line 227)
<d1b2>
<Attie> i currently have low confidence that data is put into the FIFO when it needs to be
<whitequark>
don't try to guess when f.write will write to disk
<whitequark>
for streaming samples to disk, the following should be sufficient:
<whitequark>
while True: f.write(await iface.read())
<whitequark>
now, it's true that you do need some framing, so you can't do *quite* that
<whitequark>
hm
<whitequark>
you have inputs that are not synchronized to the FPGA clock
<whitequark>
you need some FFSynchronizers to do that, or your gateware will misbehave
<whitequark>
that might be the cause of the issue you're trying to find with checksums
FFY00 has quit [Remote host closed the connection]
<whitequark>
to handle framing, I strongly suggest that you (a) rely on `iface.read()`'s internal buffering, and (b) avoid auto-flush
FFY00 has joined #glasgow
<d1b2>
<Attie> okay, i'll take a look at these things
<whitequark>
if you flush once you finish a frame, you will, in most circumstances, greatly reduce the amount of work that has to be done in Python
<d1b2>
<Attie> good to know - as in line 176?
<whitequark>
yes
<d1b2>
<Attie> i presume there'll be a flush signal i need to assert then
<whitequark>
yes
<whitequark>
in_fifo.flush
<d1b2>
<Attie> yup
<whitequark>
though in this case, it's not strictly necessary--if you write enough bytes to the buffer it'll flush all on its own
<d1b2>
<Attie> (i added auto_flush while working on the not-resetting-properly issue, thinking it was a startup flush / purge)
<whitequark>
True is the default
<whitequark>
you have to use False explicitly
<d1b2>
<Attie> ok
<d1b2>
<Attie> would you mind filling me in a bit on when w_rdy is set?
<d1b2>
<Attie> if I don't wait for it, then things go really bad, which i imagine is to be expected
<whitequark>
when the FIFO is not full
<whitequark>
and yeah, if you write to the FIFO when it's not ready, it'll just drop that byte on the floor
<d1b2>
<Attie> yeah
<whitequark>
on the other hand, if you wait instead of writing, you'll lose sync
<d1b2>
<Attie> yip
<d1b2>
<Attie> which i think is where lots of my issues are
<d1b2>
<Attie> i'll look into FFSynchronizer() and this more tomorrow
<d1b2>
<Attie> thanks very much for your input!
<whitequark>
currently the most reasonable way to handle this is to add another FIFO in front of the FIFO you get from the interface
<whitequark>
hm, wait, no
<d1b2>
<Attie> i was wondering about something like that
<d1b2>
<Attie> oh
<whitequark>
there are more ways than that
<whitequark>
hmm
<whitequark>
i would suggest adding a counter for the skipped samples that you have nowhere to write
<whitequark>
thus splitting your subtarget into two FSMs
<whitequark>
the first one either writes a sample or records the fact that it can't write a sample, the second one actually writes things
<d1b2>
<Attie> I was going to add a register to hold that info, but didn't yet
<whitequark>
so I don't recall how I2S works exactly
<d1b2>
<Attie> can you calrify "writes a sample" vs "actually writes things"?
<whitequark>
but I'm thinking something like the following might work...
<d1b2>
<Attie> i2s is fundamentally: clock, word clock, and data... each channel is framed by the word clock, to give you sync
<whitequark>
you encounter a new frame. while there is space in FIFO, you keep writing. once there is no more space in FIFO, you transition to a new, special, state where first you count how many samples are there in the frame that is in progress
<whitequark>
if stb_sample and !w_rdy then skipped++; if stb_sample and w_rdy then w_data=0; w_en=1; if !stb_sample and w_rdy then w_data=0; w_en=1; skipped--;
<whitequark>
basically you ensure that the frame you already started will be padded with zeroes
<whitequark>
since by now you have no way to escape from the frame, but you gotta finish it *somehow*
<d1b2>
<Attie> okay
<whitequark>
then you have a different issue, new frames starting when the FIFO is full
<whitequark>
i *think* the most reasonable way to solve that one is as follows
<d1b2>
<Attie> [side note: i just tried again with auto_flush=False, and got a perfect 5sec capture, and much more sensibly sized reads]
<d1b2>
<Attie> [...and zero blinks out of LED0]
<whitequark>
hm
<whitequark>
actually the second issue is pretty tricky
<whitequark>
honestly
<whitequark>
if with auto_flush=False it works fine in practice, you might get away with "on overflow the applet borks itself and stays borked until restart"
<whitequark>
this is what i did in a few other places
<d1b2>
<Attie> i think it might be the way for this
<whitequark>
handling overflow gracefully is really hard and the payoff can often be minimal
<d1b2>
<Attie> yeah
<d1b2>
<Attie> when flushing the fifo... what does that actually mean?
<whitequark>
since that code is not only complex but is bug-prone as well
<d1b2>
<Attie> it pushes data over USB?
<whitequark>
so, glasgow applets use a stream-based interface, like TCP
<whitequark>
but USB is packet-based, like UDP
<whitequark>
you have to packetize somehow
<d1b2>
<Attie> i was seeing incoming buffers of ~300 bytes, which feels small and poor wrt overhead
<whitequark>
i didn't want the packetization to be directly user-visible because that sucks for a variety of reasons
<whitequark>
for OUT packets i just use a FIFO
<d1b2>
<Attie> now i see stable ~19kiB buffers
<whitequark>
for IN packets, no dice; ideally you want sending full-sized packets, but you also sometimes want non-full-sized packets
<whitequark>
the way it works is:
<d1b2>
<Attie> *16KiB
<whitequark>
with auto_flush=True, the FX2 crossbar stuffs the FX2-side buffers from the applet-side buffers while the latter fill up. when the applet-side buffer becomes empty, the crossbar cuts a new packet.
<whitequark>
this works "the way people expect" in that it never results in in_fifo writes getting stuck forever in the FX2
<whitequark>
so it's good for novices who program their first applet. but performance sucks.
<whitequark>
with auto_flush=False, the FX2 crossbar always cuts a new packet when the FX2 512-byte buffer fills up
<whitequark>
and also when flush is 1
<whitequark>
(it also sends ZLPs as necessary)
<whitequark>
in your case, since you never explicitly flush, you just get a stream of full-sized packets
<whitequark>
then, the PC-side code has two more layers of buffering
<whitequark>
first, it aggregates the 512-byte packets it gets from the device into something more substantial, because if it doesn't, throughput and latency would suck
<whitequark>
that's the 16 KiB buffers you're seeing
<whitequark>
it can only aggregate until it receives a non-full-sized packet (basically until the applet flushes), so maybe you *shouldn't* explicitly assert .flush here
<whitequark>
the next layer of buffering is something similar to what you tried to do with your _read_fifo_in function
<d1b2>
<Attie> ok... so when the fifo is "being flushed", is there a hold put on it or similar? could this explain why i was dropping so many samples while transferring relatively tiny packets?
<whitequark>
well
<whitequark>
the answer to your question as asked is "no", but basically yes
<d1b2>
<Attie> [i just captured what "audibly sounded" like a perfect ~3.20 song]
<d1b2>
<Attie> heh - a good answer then 🙂
<whitequark>
and the answer is "yes" because, well, how do i put it
<whitequark>
because USB is kinda badly designed
<whitequark>
the HCD has to poll the device for each IN packet, and we use bulk transfers for reasons too complex to go into here
<whitequark>
the HCD will repeatedly poll the device, modern ones even several times per microframe, as long as the device is sending full length packets
<whitequark>
but when you don't send a full length packet, the HCD will not poll the device again in the same microframe, and in general it assumes, when scheduling, that you're done for a while
<whitequark>
when this happens, the FX2-side buffers, which are fairly small (1K or 2K, depending on the OS, with one applet), fill up
<d1b2>
<Attie> i see, yes that makes sense
<whitequark>
when that happens, the FPGA-side buffer fills up
<whitequark>
when that happens, you drop samples
<whitequark>
the FPGA-side buffer is just 512 bytes by default
<whitequark>
extending it is *usually* a sign something went wrong elsewhere
<whitequark>
but it's occasionally truly necessary
<sorear>
well you have 16KB total on the hx8k
<whitequark>
yes, which is why it's so small by default
<whitequark>
you might want to use those for other things. so one BRAM per EP it is
<d1b2>
<Attie> great, thanks very much for the support wq!
<whitequark>
np
<d1b2>
<Attie> i seem to now have what is a "reasonably" working applet... starting to remove some of the crud
<whitequark>
congrats
<d1b2>
<Attie> and i should look into FFSynchronizer()
FFY00 has quit [Read error: Connection reset by peer]
<d1b2>
<Attie> ... i should go, thanks again wq
<d1b2>
<Attie> if you wanted to take a look over it as it stands, then please feel free - i just cleaned and pushed, TODOs at the top
<whitequark>
tomorrow prob
<d1b2>
<Attie> np... bye
_whitelogger has joined #glasgow
<_whitenotifier-f>
[glasgow] brainstorm deleted a comment on issue #151: False positive results in selftest - https://git.io/JU5xM
electronic_eel has quit [Ping timeout: 265 seconds]
electronic_eel has joined #glasgow
PyroPeter_ has joined #glasgow
PyroPeter has quit [Ping timeout: 258 seconds]
PyroPeter_ is now known as PyroPeter
balrog has quit [Ping timeout: 260 seconds]
Stormwind_mobile has quit [Remote host closed the connection]
Stormwind_mobile has joined #glasgow
balrog has joined #glasgow
ma1 has quit [Quit: ma1]
_whitelogger has joined #glasgow
ma1 has joined #glasgow
jevinskie[m] has joined #glasgow
_whitelogger has joined #glasgow
<_whitenotifier-f>
[glasgow] russss commented on pull request #210: Move factory flashing instructions and add basic example to README - https://git.io/JUFaR
Stormwind_mobile has quit [Read error: Connection reset by peer]
Stormwind_mobile has joined #glasgow
samlittlewood has joined #glasgow
Stormwind_mobile has quit [Ping timeout: 265 seconds]