sakirious has quit [Quit: Ping timeout (120 seconds)]
sakirious has joined #nmigen
proteusguy has joined #nmigen
Bertl is now known as Bertl_oO
jeanthom has joined #nmigen
lkcl has quit [Ping timeout: 272 seconds]
lkcl has joined #nmigen
jeanthom has quit [Ping timeout: 265 seconds]
emeb_mac has joined #nmigen
bqq has joined #nmigen
bqq has left #nmigen ["Leaving"]
lkcl has quit [Ping timeout: 256 seconds]
lkcl has joined #nmigen
<lsneff>
whitequark: the vcd parsing crate I'm using is terribly, terribly slow, I guess I'll have to rewrite itt
<lsneff>
With some extremely buffering it's down to about a second for a 36 MB vcd, about a third is unnecessary allocations in the vcd crate :/
jeanthom has joined #nmigen
<d1b2>
<mardab> is there an ELI5 on why migen has split into omigen and nmigen?
jeanthom has quit [Ping timeout: 268 seconds]
<d1b2>
<TiltMeSenpai> omigen is migen, nmigen is new migen
<d1b2>
<mardab> I know, but why? Was there no agreement on future direction?
<tpw_rules>
so i'm reengaging one of my projects and building using the nmigen cli and yosys (both latest master) now says: Warning: Wire top.U$$1.U$$0.timer.do_decrement_SB_DFFSR_Q_D_SB_LUT4_O_I3 has an unprocessed 'init' attribute.
<tpw_rules>
(along with some other similar ones) is that bad?
jeanthom has joined #nmigen
jeanthom has quit [Ping timeout: 240 seconds]
chipmuenk has quit [Quit: chipmuenk]
lkcl has quit [Ping timeout: 240 seconds]
lkcl has joined #nmigen
<whitequark>
mardab: it was not technically feasible to implement the goals i had for nmigen within the existing migen codebase
<whitequark>
(especially while keeping complete compatibility for downstream software)
<whitequark>
tpw_rules: it is probably benign; there are some open yosys issues related to the clean pass which might have that effect
<d1b2>
<mardab> Thanks for clarification
<lkcl>
lsneff: that's probably because it's text-based format. a common mistake made by people who've grown up with json and xml. there's a binary format "fst" which gtkwave supports, and the package contains a stand-alone vcd2fst command
<lkcl>
it's by no means perfect: some information is lost (strings for example)
<JJJollyjim>
that's an attack on young people i haven't heard before lol
<whitequark>
yeah, a common mistake that is as old as unix
<whitequark>
get that elitism out of this channel, lkcl
<lkcl>
it's just a fact. i'm pretty much verbatim 22 years ago quoting Andrew Tridgell, the lead developer of samba.
<lkcl>
we had to get high performance code running on pentium I and II systems.
<lkcl>
XML had just come out, beginning to take over from ASN-I (because it was "easier")
<lkcl>
text-based protocols equals terrible performance. this is simply a fact.
<d1b2>
<uep> I suggest the world is not as simple as your preconceptions
<lsneff>
lkcl: I'm working on a tool to convert a vcd into something much more quickly grokable, for use in a vcd viewer.
<whitequark>
yeah, what uep said
<lkcl>
lsneff: ah nice. so, can i recommend looking up the fst format, which is in gtkwave and some other viewer tools?
<lsneff>
Json can be parsed at GBs per second, text based isn't necessarily slow
<lkcl>
i seem to recall somewhere it was the default format from some closed-source simulator tools
<lsneff>
Fst is basically impossible to implement without using the library from gtkwave
<whitequark>
yep, that is also my conclusion about fst
<lkcl>
lsneff: ahh
<whitequark>
i would very much look forward to a format better than fst in that aspect
<lsneff>
I will try to document
<lkcl>
two systems which i'm aware of which will give high performance. i'm assuming multi-gigabyte read/write data structures
<lkcl>
the first is datadraw. the second is LMDB.
<lsneff>
Right now, I'm working on the temporary streaming format used in the conversion from vcd to whatever
<lkcl>
datadraw is... well, you have to see the website on it :)
<whitequark>
waveform dumps are append-only; any write support is extra complexity we don't need
<lkcl>
ah then interestingly, LMDB will help there: it has a special append-only mode.
<lkcl>
that's assuming that a key-value store is useful
<JJJollyjim>
i don't think it is
<lsneff>
Not really
<lkcl>
the caveat is that it has to be "the largest key", which if it's time-based (sequential event numbering) you could get away with
<lkcl>
ah no, wait, lmdb database files aren't cross-machine portable, no sorry, forget that one
<lsneff>
I came up with a clever way of storing variable length bit vectors in rust
<lkcl>
oo that's interesting and unexpected - gnetman, which converts netlists between VHDL, Verilog and SPICE, uses datadraw
<lkcl>
half the memory and 7x faster than straight c data structures lol
nfbraun has joined #nmigen
<whitequark>
the requirements for storing netlists and for storing waveform dumps are extremely different
<whitequark>
from their front page, DataDraw seems reasonable for storing graphs (although it is very funny they compare themselves to STL, given that STL is widely known to be quite inefficient... maybe it wasn't at the time that was written)
<whitequark>
but it is not reasonable for an append-only format designed primarily for interchange. for example, I couldn't use it in CXXRTL (CXXRTL has no external dependencies) and it would be a major obstale for people using Rust
<whitequark>
while at the same time the append-only time-series nature greatly reduces the requirements for a serialization library
<lkcl>
yyeah that's tricky. LMDB (which is a B+ Tree with a special append mode) still seems to fit all the requirements, except the initialisation is done to the page-boundary of the machine (LMDB uses shmem copy-on-write semantics)
<lkcl>
so you copy a database from one machine to another, performance sucks because of different page sizes.
<lkcl>
i know some people who did an append-only key-value store... what the hell was it... zdb?
<lkcl>
stupidly quick because nothing was ever removed
<lkcl>
lsneff: if you think it would help i can look that up
<lkcl>
key question: is compatibility with a "pre-existing" file format sought here
<lkcl>
(vcd)
<lkcl>
or is the goal to make a new one
<lkcl>
or
<lkcl>
is the goal to have efficient *in-memory* representations of the data, once read?
<whitequark>
no, vcd is a terrible and very underspecified file format that no one should be keeping compatibility with
<lkcl>
:)
<lkcl>
that answers that one lol. (and my response is: whewwww :) )
<lkcl>
ok, so what are the access patterns? what are the primary keys, and are there any secondary keys?
<lkcl>
as in: what queries on the data are made? "give me events between time X and time Y"?
* lkcl
trying to think what else people would want to query.