02:22 <mithro> syed.ahemd: At the moment is it pretty quiet with people on holidays, I expect that acomodi and litghost will be back in Jan and able to help then

02:24 <mithro> hansfbaier: You are correct that the 071-dump_all takes a pretty large amount of disk space

02:35 <hansfbaier> mithro: I now really understand the reason for the ticket: In order to build the part for the Nitefury, the fully bonded part had to be built first. It creates a TON of JSON files and hung my linux system because create_node_tree consumed all my 32GB of RAM.

02:36 <hansfbaier> mithro: I think if we could use sqlite instead of JSON that would save a HUGE amount of disk space and memory

02:36 <hansfbaier> mithro: because the data is inherently relational and thus a relational database would be a great fit

02:36 <hansfbaier> mithro: Also sqlite has simple python bindings

02:37 <hansfbaier> mithro: I will look into moving 071-dump_all to sqlite

02:39 <mithro> hansfbaier: Dunno how those work, but it might be a quick solution to put a compression filter on the json output?

02:39 <hansfbaier> mithro: it would be the proper way to deduplicate (= normalize) the data, because each item is stored exactly once

02:40 <hansfbaier> mithro: But in order to to make a database schema I have to understand the logical structure of the data

02:40 <mithro> hansfbaier: Probably - but sometimes starting with an easier solution gets you to the end goal faster :-)

02:41 <mithro> hansfbaier: https://pypi.org/project/compress-json/

02:41 <tpb> Title: compress-json · PyPI (at pypi.org)

02:42 <hansfbaier> mithro: The compression does not solve my problem: create_node_tree still would have to uncompress everything and then it would take even more memory

02:42 <hansfbaier> mithro: And I am quite reluctant to buy more memory because, now I have 32GB and I even rarely maxed out 16G

02:43 <mithro> hansfbaier: Spin up a cloud machine?

02:43 <mithro> hansfbaier: https://www.lucidchart.com/techblog/2019/12/06/json-compression-alternative-binary-formats-and-compression-methods/ seems like an interesting page

02:43 <tpb> Title: JSON Compression: Alternative Binary Formats and Compression Methods - Lucidchart (at www.lucidchart.com)

02:45 <mithro> I wonder if we should be using ujson in some places too - https://pypi.org/project/ujson/

02:45 <tpb> Title: ujson · PyPI (at pypi.org)

02:45 <hansfbaier> mithro: saving space is not the only advantage, you also can create indices for oft used tables, this will speed up the processing of the data by orders of magnitude

02:46 <mithro> hansfbaier: I'm sure that litghost might have some thoughts on that but is currently on holidays until Jan

02:47 <mithro> Anyway, I'm heading out

02:47 <hansfbaier> mithro: I will discuss it in the ticket. I would aim at a proof of concept limited to 071-dump_all

02:59 <hansfbaier> mithro: sqlite seems to handle large datasets quite well: https://stackoverflow.com/questions/1033309/sqlite-for-large-data-sets

02:59 <tpb> Title: SQLite for large data sets? - Stack Overflow (at stackoverflow.com)

04:02 <umarcor> mithro: what about using yaml and anchors?

14:42 <mithro> umarcor: I think I would take json+compression over yaml+anchors personally...