00:03 <duck2> well, by 1) using pypy magic, 2) not using prjxray.db_cache, 3) not using threads in prjxray_create_edges.py and also rearranging some code in build_channels, i could bring the build into the vpr stage in a LXC container with 2gb RAM and 1gb swap. it takes roughly 40 mins or maybe a hour

00:10 <duck2> honestly i don't think i can get away with vpr with this much ram but let's see

01:15 <duck2> litghost: indeed, vtr uses pugixml which needs 2x the file size in memory to build a dom tree https://github.com/zeux/pugixml/issues/166

01:15 <tpb> Title: Incrementally reading very large files · Issue #166 · zeux/pugixml · GitHub (at github.com)

01:58 <duck2> in fact, rr_graph.xml would be a very neat json file but i guess that would be hacking too much

02:54 <litghost> duck2: I think a zero parse format is worth investing in the future (e.g. captain proto or flatbuffers). I have a VPR branch with flatbuffer support, but it turns out the flatbuffer support in python is actually really terrible :(

13:04 <duck2> what is the problem with python's flatbuffers support? i looked at the tutorial and it does not seem worse than, say, javascript

14:58 <litghost> duck2: It is implemented in python, using ctypes. It is very very slow. It actually turns it to be faster to write out the rr graph in XML, and have VPR load the XML and write it back to flatbuffers.

14:58 <litghost> duck2: It also doesn't support incremental writing, which may become important for larger graphs.

15:36 <duck2> i think converting to libxml2 SAX would be good in terms of memory usage and ease of impl, since the current code is suitable for sax: processes one element at a time

15:45 <duck2> litghost: it's very sad. maybe pypy can help with that though, it runs pure python code quite fast.

17:09 <litghost> Duck2: yes, sax parsing was what I was implying above