<woodruffw>
watzon: do you need bit or byte granularity?
<woodruffw>
one option to consider is kaitai, you can write your spec out in YAML and have it compiled to ruby: https://kaitai.io/ (i haven't used the ruby compiler before, but the python and C/C++ ones work well)
niv has quit [Client Quit]
niv has joined #ruby
niv has quit [Client Quit]
niv has joined #ruby
niv has quit [Client Quit]
<watzon>
Just byte granularity is good, but ideally I need something like Python's struct or Crystal's IO::ByteFormat. I might be able to get away with using `Array.pack` and `String.unpack`, but it's pretty limited.
niv has joined #ruby
niv has quit [Client Quit]
niv has joined #ruby
<watzon>
My problem with more declarative methods is that I'm working with 1200+ generated classes, and the declarative methods would require major changes to the codegen
niv has quit [Client Quit]
niv has joined #ruby
rzjian has joined #ruby
niv has quit [Client Quit]
niv has joined #ruby
niv has quit [Client Quit]
niv has joined #ruby
Sina has quit [Quit: Connection closed for inactivity]
<naftilos76>
Can you pls have a look and tell me if there is any way to improve it?
<naftilos76>
After using this method in a loop i get a max of 200K/sec but suddenly it drops down to a few per sec
<naftilos76>
off course the reason for this is because i am using 5000 randomly generated user ids and 50000 randomly generated topic ids
<naftilos76>
if i drop the value range of these two i get better performance
<naftilos76>
the was i am doing it is the least memory-hungry approach based on my understanding
<naftilos76>
if i just drop all data inside an array the performance just explodes
<naftilos76>
but it takes a lot of memory space
<naftilos76>
and searching/deleting gets slower
<naftilos76>
can anybody help?
iNs has quit [Quit: bubye]
iNs has joined #ruby
fercell has quit [Ping timeout: 260 seconds]
fercell has joined #ruby
x0n has quit [Ping timeout: 240 seconds]
x0n has joined #ruby
Swyper has joined #ruby
Swyper has quit [Ping timeout: 264 seconds]
x0n has quit [Ping timeout: 256 seconds]
<nakilon>
naftilos76 was it you who did the caching thing few weeks ago?
<naftilos76>
yes
<nakilon>
I said you'll face the perfomance drop on reaching 3GB of memory, what did you reach?
<naftilos76>
i cannot say i noticed such a thing
<nakilon>
what memory did your process reach?
<naftilos76>
i am running stats in order to find a good structure
<naftilos76>
i reached 50GB
<nakilon>
_OO
<naftilos76>
the problem is that i keep noticing sudden drop of performance
<nakilon>
would be 5GB if you used different language
<naftilos76>
that is records/sec
<naftilos76>
that is my last resort
<nakilon>
I told you you'd notice it, just weird how you got this far
<naftilos76>
probably will move to C++
jenrzzz has quit [Ping timeout: 240 seconds]
weaksauce has quit [Ping timeout: 240 seconds]
jenrzzz has joined #ruby
x0n has joined #ruby
fercell has quit [Ping timeout: 272 seconds]
fercell has joined #ruby
<naftilos76>
there is no cure i guess
split-brain has joined #ruby
coniptor has quit [Ping timeout: 240 seconds]
coniptor has joined #ruby
<nakilon>
maybe look at the existing in-mem databases for that
rzjian has quit [Quit: WeeChat 3.0]
<nakilon>
50GB Ruby process is just wild
jenrzzz has quit [Ping timeout: 240 seconds]
DaRock has quit [Ping timeout: 265 seconds]
moldorcoder7 has quit [Quit: %bye mirc%]
techsethi has quit [Ping timeout: 260 seconds]
moldorcoder7 has joined #ruby
jenrzzz has joined #ruby
techsethi has joined #ruby
jenrzzz has quit [Ping timeout: 272 seconds]
<ua_>
naftilos76, nice, was it 50gb in one process? i did some heavy processing in ruby but it was parallelized by splitit it among 48 cores
<naftilos76>
one single process
<naftilos76>
a few threads though
<naftilos76>
3-4 threads not sure
jenrzzz has joined #ruby
johnny526 has quit [Quit: Connection closed]
jenrzzz has quit [Ping timeout: 240 seconds]
jenrzzz has joined #ruby
Technodrome has joined #ruby
jenrzzz has quit [Ping timeout: 260 seconds]
jenrzzz has joined #ruby
jenrzzz has quit [Ping timeout: 246 seconds]
akem1 has quit [Quit: leaving]
akem has quit [Quit: leaving]
jenrzzz has joined #ruby
moldorcoder7 has quit [Ping timeout: 265 seconds]
jenrzzz has quit [Ping timeout: 256 seconds]
elphe has quit [Quit: leaving]
moldorcoder7 has joined #ruby
jenrzzz has joined #ruby
jenrzzz has quit [Ping timeout: 246 seconds]
yann-kaelig has joined #ruby
Swyper has joined #ruby
jenrzzz has joined #ruby
Swyper has quit [Ping timeout: 258 seconds]
jenrzzz has quit [Ping timeout: 260 seconds]
jenrzzz has joined #ruby
jenrzzz has quit [Ping timeout: 240 seconds]
gearnode has joined #ruby
m27frogy has joined #ruby
jenrzzz has joined #ruby
vondruch has quit [Ping timeout: 265 seconds]
postmodern has quit [Quit: Leaving]
jenrzzz has quit [Ping timeout: 246 seconds]
MuffinPimp has quit [Quit: Goodbye.]
moldorcoder7 has quit [Quit: %bye mirc%]
MuffinPimp has joined #ruby
moldorcoder7 has joined #ruby
jenrzzz has joined #ruby
greengriminal has joined #ruby
<linuus[m]>
Does anyone here have experience with RbNaCl? I have some tests that are failing randomly, with different errors. Sometimes it's that the `nonce` is 11 bytes, but 12 was expected. And sometimes it's `Ciphertext failed verification.`. Any ideas what might cause these things? Can it be some string encoding issues or something? It's really confusing that it's random...
<linuus[m]>
It fails when decrypting btw.
jenrzzz has quit [Ping timeout: 265 seconds]
jenrzzz has joined #ruby
<adam12>
linuus[m]: Is it repeatable with same test seed?
<linuus[m]>
Nope
<adam12>
linuus[m]: Do you test in parallel?
<linuus[m]>
No. Also, it fails randomly if I only run a single test so it's not a test order thing either.
<linuus[m]>
I got the nonce length error again now, the nonce is indeed 12 bytes when it's created (I printed it when encoding) but it says it's 11 when decoding.
<adam12>
linuus[m]: Ruby version? Which interpreter are you using? MRI?
<adam12>
(CRUby/YARV)
Technodrome has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
<linuus[m]>
MRI, yes
jenrzzz has quit [Ping timeout: 256 seconds]
<linuus[m]>
I'm using `RbNaCl::Random.random_bytes(box.nonce_bytes)` to generate the nonce btw. (Where `box` is the specific cipher class
<adam12>
linuus[m]: Can you make a small reproducible example?
dionysus69_ has quit [Quit: Konversation terminated!]
Mia has joined #ruby
jla has quit [Ping timeout: 265 seconds]
Axy has quit [Ping timeout: 272 seconds]
whysthatso has quit [Ping timeout: 256 seconds]
jenrzzz has quit [Ping timeout: 240 seconds]
jenrzzz has joined #ruby
jenrzzz has quit [Ping timeout: 240 seconds]
jenrzzz has joined #ruby
jenrzzz has quit [Ping timeout: 240 seconds]
jenrzzz has joined #ruby
merlin1991 has joined #ruby
Rudd0 has quit [Ping timeout: 246 seconds]
deviantfero has joined #ruby
jenrzzz has quit [Ping timeout: 272 seconds]
naftilos76 has quit [Ping timeout: 260 seconds]
jenrzzz has joined #ruby
akem has joined #ruby
dionysus69 has joined #ruby
dionysus69 has joined #ruby
Technodrome has joined #ruby
whysthatso has joined #ruby
m_antis has joined #ruby
jenrzzz has quit [Ping timeout: 246 seconds]
jenrzzz has joined #ruby
hsmath has joined #ruby
jenrzzz has quit [Ping timeout: 264 seconds]
nofxx_ has joined #ruby
nofxx has quit [Ping timeout: 256 seconds]
jenrzzz has joined #ruby
akem has quit [Ping timeout: 256 seconds]
akem has joined #ruby
jenrzzz has quit [Ping timeout: 256 seconds]
Cork has quit [Remote host closed the connection]
Cork has joined #ruby
jla has joined #ruby
jenrzzz has joined #ruby
<burgestrand>
linuus[m] maybe I'm late to the party, but pack/unpack for A is a bit iffy, since unpacking removes trailing nulls/spaces and in your case you don't really want that :D
<burgestrand>
it's not exactly clear why you're using pack on your random bytes, they're already packed in a way, they have no meaning
Technodrome has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
burgestrand has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
Rudd0 has joined #ruby
trautwein has quit [Ping timeout: 264 seconds]
teardown has quit [Ping timeout: 268 seconds]
akem_ has joined #ruby
orbyt_ has joined #ruby
ChmEarl has joined #ruby
DTZUZU_ has joined #ruby
DTZUZU has quit [Read error: Connection reset by peer]
DTZUZU__ has joined #ruby
trautwein has joined #ruby
shtirlic has quit [Remote host closed the connection]
DTZUZU_ has quit [Ping timeout: 256 seconds]
trautwein has quit [Ping timeout: 264 seconds]
fercell has quit [Ping timeout: 265 seconds]
fercell has joined #ruby
DTZUZU__ is now known as DTZUZU
trautwein has joined #ruby
jenrzzz has quit [Ping timeout: 272 seconds]
trautwein has quit [Ping timeout: 258 seconds]
jenrzzz has joined #ruby
m_antis has quit [Read error: Connection reset by peer]
orbyt_ has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
Alexandra_V has joined #ruby
Technodrome has joined #ruby
orbyt_ has joined #ruby
orbyt_ has quit [Client Quit]
fercell has quit [Ping timeout: 265 seconds]
fercell has joined #ruby
fercell has quit [Ping timeout: 264 seconds]
fercell has joined #ruby
akasha has joined #ruby
<akasha>
"x&y".gsub(/(&)/, '\1') returns 'x&y' but "x&y".gsub(/(&)/, '\\\1') does not return 'x\&y'. how do I get gsub to return the capture group contents preceded by a backslash (\)?
<akasha>
using some other method to replace characters with their escaped versions is fine too