<nakilon>
rubydoc.info is 503ing; are there alternatives with gem docs?
al2o3-cr has joined #ruby
hiroaki has joined #ruby
akem has quit [Quit: leaving]
akem has joined #ruby
jenrzzz has joined #ruby
jenrzzz has quit [Ping timeout: 240 seconds]
schaerli has quit [Ping timeout: 240 seconds]
schaerli has joined #ruby
<adam12>
nakilon: yard server --gemfile
dviola has left #ruby ["WeeChat 3.0"]
dviola has joined #ruby
venmx has quit [Ping timeout: 240 seconds]
iNs_ has joined #ruby
iNs has quit [Ping timeout: 268 seconds]
schaerli has quit [Ping timeout: 264 seconds]
venmx has joined #ruby
jenrzzz has joined #ruby
venmx has quit [Ping timeout: 256 seconds]
venmx has joined #ruby
jenrzzz has quit [Ping timeout: 256 seconds]
jenrzzz has joined #ruby
naftilos76 has joined #ruby
cuerbot has joined #ruby
<naftilos76>
Hi everyone, i am trying to build a caching system that supports super fast retrieval of records. I am using ruby 3.0.0. I have already created methods t add records, get records and delete records that are older than a date. The core of the system is based on a ruby hash. The records are mqtt messages. The idea is to have a caching system that has 2-3 days in RAM and if anyone wants records dated older that that then i will have to get those records
<naftilos76>
from the database.
<naftilos76>
Trying to use the database without the caching system ended up in the retrieval of a few records lasting >2-3 seconds after mysql went up to a few millions of rows.
elcuervo has quit [Ping timeout: 272 seconds]
<havenwood>
naftilos76: You might actually consider Rinda::TupleSpace for a talk like this.
<naftilos76>
i guess i am gonna have to put to the test to see if it performs like i hope it will
<havenwood>
naftilos76: For that ^ cache, I set it to have a reaper check every ten minutes for keys that expire in 24 hours.
dinfuehr_ has quit [Ping timeout: 256 seconds]
<naftilos76>
i do something like that in mine
<havenwood>
naftilos76: If it doesn't suit you, we can show how to make your own thread-based reaper around Hash or however else.
<havenwood>
But Rinda::TupleSpace already has that implemented, is thread safe, and other niceities.
<havenwood>
I'd say give it a look! :0
<havenwood>
:)*
dviola has quit [Ping timeout: 264 seconds]
<naftilos76>
i have already done it but the performance is problematic after a certain extend
dinfuehr has joined #ruby
GodFather_ has joined #ruby
GodFather has joined #ruby
dviola has joined #ruby
<naftilos76>
i want it t be able to store 3K messages/sec
<naftilos76>
it works well
<naftilos76>
but after it reaches 10Million
<naftilos76>
performance starts to drop
<naftilos76>
it works well though for 1000-2000 messages/sec
<havenwood>
naftilos76: You're using Hash as the underlying data structure? Which version of Ruby?
dviola has left #ruby [#ruby]
<naftilos76>
off course i had to implement mutexes and stuff in order to make thread safer but i am not entirelly sure that i have done everytihng perfect
<naftilos76>
3.0.0
dviola has joined #ruby
<naftilos76>
i have implemented the search method with the binary method which is supper fast
<havenwood>
naftilos76: Hem, I'd be very curious to see how TupleSpace stack up.
<havenwood>
Also curious what the bottleneck is for your code.
<naftilos76>
ok you got my attention - i will give it a try
<naftilos76>
well it is a lot of it
<naftilos76>
i have 4 separate processes
cd has joined #ruby
<naftilos76>
the first is just getting messages from the local mosquitto server
<naftilos76>
the second is get the messages from the first and bulk-saving to mysql
<naftilos76>
and sending to a Queue class-based third process
<nakilon>
17:29:47 <adam12> nakilon: yard server --gemfile
<naftilos76>
the last one is the cache implementation
jenrzzz has joined #ruby
<nakilon>
but I don't have yard, ruby or computer, I just wanna read about some method
<naftilos76>
all comms are implemented via sockets which is crazy fast much faster that i would require
<naftilos76>
so the last caching process
<naftilos76>
uses a hash like this: cache[unique_user_id][mqtt_topic][time_stamp_of_the_message_in_floating_point]
<naftilos76>
and every [user_uid][topic][timestamp] is an array
<naftilos76>
so i have multiple user_uids, multiple topic names and offcourse every topic has millions of timestamps with each timestamp being a array with message info
<naftilos76>
right now it has got 20Million records loaded
<naftilos76>
with the access time for say 10 records being well under 1ms
<naftilos76>
for 1000 records it would still be under 10ms so it is a nice solution BUT every now and then
<havenwood>
naftilos76: On the Hash end of things, #dig might help if you're not already using it.
<nakilon>
Ruby would be the last language I would use to "improve perfomance" of anything
<havenwood>
naftilos76: Folk do when the task is complex enough, since a more sophisticated solution can be much faster than quick primatives.
<havenwood>
naftilos76: Hence odd things like supercomputing with Ruby.
<havenwood>
oops, wrong person
<havenwood>
nakilon: ^
<naftilos76>
the access time climbs up to several seconds but fortuantelly that is like 0.001% of the cases
<nakilon>
it has the biggest memory consumption and your goal was to build a cache that is about optimizing memory
<havenwood>
na<tab> failed me, hah
<naftilos76>
i thought about implementing the last part process which is the caching system with C/C++ but i have already spent a few months on this!
<naftilos76>
is there anything standard happening if i use a hash with 20Milion records ? I mean is there a standard practise that says "do it in this way" ?
<naftilos76>
i know this is a very general question...
<nakilon>
from my observations I know that when you run out of RAM and/or ruby process reaches >3gb, it starts spending most of the time on garbage collection or something like that
<nakilon>
also in Ruby to implement some sort of caching system you will have to rely on its magic garbage collector, no real ability to optimize the memory consumption -- the best you can do is to recreate Ruby objects and hope it won't leak
Technodrome has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
<naftilos76>
nakilon: i was experimenting with GR the other day and understood that memory allocated does not get de-allocated if i explicitly do not use GR.start. Isn't that true?
<nakilon>
it might depend on classes you use and how you build scopes, and might change between major versions of Ruby
<naftilos76>
ok
Technodrome has joined #ruby
<naftilos76>
havenwood: dig looks nice but i what i am actually doing is retrieving records between dates that have 6 decimals after the seconds so using dig is no good for me.
<nakilon>
if I was facing my data structure becomes slow after reaching a certain size I would make it automatically drop the least used portions of data out of memory determining what to throw out using some of many existing caching algorithms
<naftilos76>
nakilon: i have already assigned that to a thread, that is, to delete records older than a specified timestamp - which is somewhere close to 2 days
<naftilos76>
however that means that 3K/sec becomes 3K*60*60*24*2=518.4Million records
<naftilos76>
which is a huge number
<naftilos76>
i guess i will have to comprimise for less
<nakilon>
did you plot the memory/time consumption? does it grow continuously? if it is getting only worse I would probably try a process recreation once in an hour for example -- that would deal with possible memory leak; dumping the current data and then loading it, maybe even doing it in background to when you kill the process another one is already ready
<nakilon>
to work
<naftilos76>
This is one line of the stats i generate regarding access time to the records: 10-100ms: 1.079% (30667), 100-1000ms: 0.121% (3444), over 1sec: 0.038% (1068)
<naftilos76>
so far ihas been working for 3.2 hrs and has consumed 1.5GBytes of RAM
<naftilos76>
you are right - i will add this to the stats
<nakilon>
you might compare the growth of RAM used in fact to how much it HAD to use by your estimation or walking through the data summing the .size and stuff
<nakilon>
if they don't align that would mean the memory leak
shokohsc has joined #ruby
<naftilos76>
i couls do that certainly. since all requests are sent to the caching system via sockets, could i build a mechanism that sets a time out period, rejects the connection and retries to get the data it requires?
<naftilos76>
that would definitely solve one of my problems
<naftilos76>
that would force that 0.038% >1sec to get eliminated
<naftilos76>
nakilon: i tried named pipes which worked really well and fast but what about multiples requests? Thats why i moved to sockets which is crazy fast since there is not PHY!
<naftilos76>
Sockets worked really well - i am very pleased with it and never got corrupted data
<nakilon>
naftilos76 can you provide some link to read about sockets?
<bougyman>
Problem was: given a string of operations in the set [<Integer>, '+', '-', 'POP', 'DUP'], perform operations on a stack, with <Integer> being a PUSH <Integer> to the stack. Return -1 on any error. integers on the stack must be in the range 0..(2**20 - 1)
<leah2>
i dont get the error checking for - but else it looks ok to me
<bougyman>
what do you mean?
<bougyman>
that's checking for underflow.
cow[moo] has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
<leah2>
didnt pop already do that?
<leah2>
it checks that - doesnt yield a negative result
<leah2>
ah ok
<bougyman>
right.
<leah2>
it's a value underflow, not a stack underflow :)
<bougyman>
Woops, you're right. The comment is wrong.
<bougyman>
same in plus, that's a value overflow, not a stack overflow.
<bougyman>
Oh well.
<bougyman>
Thanks, leah.
<leah2>
np
<leah2>
i had to do a test too recently, but just to review the application process :D
<bougyman>
At least I don't feel like it was time wasted, now. That was such a letdown.
<leah2>
it was a quite intricate activerecord query, i think we'll have to make it a bit easier
<bougyman>
I've done 4 of these this month, all the other 3 gave immediate results / feedback.
<leah2>
was there a test suite or something?
<bougyman>
Yeah, they show you 4 tests and hide 8, as per usual.
<leah2>
ok
<bougyman>
It's the results of the hiddens that they didn't show me. Like not even pass /fail.
<leah2>
ah meh
<bougyman>
My fave thus far was bridgewaters, it scored each set.
<bougyman>
and then came up with an overall score.
<bougyman>
I really can't believe i'm still in the running for that one. 3 people I know have applied and all have been weeded out thus far.
<bougyman>
One of them I'm sure is a better programmer than I.
<leah2>
i find it a bit silly to do these tests for people that have some kind of track record...
cow[moo] has joined #ruby
<bougyman>
The thing I loved most about bridgewater is that my tech panel after the test went through the thing line by line and we talked about the thought process of the solution iteratively.
<bougyman>
That's been my favorite interview of the month.
<bougyman>
Funny thing is, if I get that job, I'm not sure I'll take it.
<leah2>
heh
<bougyman>
It's more of the challenge of it.
<bougyman>
They're supposed to be one of the hardest places to break into.
<bougyman>
Their compensation package is simply ridiculous. So that's a plus. But that culture thing... dunno how or if I could get used to it.
<leah2>
and they do rubu?
<bougyman>
Well I'm in the SRE space, so ruby is everywhere.
<bougyman>
Puppet, Chef, etc.
<leah2>
ah, legacy deployment stacks :p
* leah2
ducks
<bougyman>
Puppet is Legacy?
<bougyman>
I can kind of see it with Chef, though they have a lot of cloud-native stuff too.
<leah2>
my impression was that most people moved to ansible
<leah2>
or full k8s
<leah2>
but i dont have a big sample
<bougyman>
Yeah, i'll be doing plenty of k8s there. The Go assessment was trivial shit, though.
<leah2>
:)
<bougyman>
Place I'm at now is all Helm. Helm 2, even. bleh.
<leah2>
ick
<bougyman>
Helm is the Chef of k8s.
<leah2>
luckily i had something to say in the last project and vetoed helm quickly
<bougyman>
We end up writing operators that basically play the role of kustomize
<leah2>
i'd consider helm3 now, maybe
<bougyman>
Helm3 has some fixes. I think the Helm3 + kustomize route handles almost all use-cases.
<bougyman>
But then helm purists get mad.
<bougyman>
because Tiller isn't playing the role of Big Brother
powerhouse has quit [Quit: Leaving]
<leah2>
depending on how many external packages you need, reusing upstream helm charts saves a lot of time
<leah2>
tiller, aka the backdoor :p
<bougyman>
Indeed.
<bougyman>
Yeah, once I got used to the pattern of wrapping upstream charts everything went smoothly. My first mew months of helm was forking upstream charts. That's a nightmare.
<bougyman>
*few
<apotheon>
new months
<apotheon>
mew months
<apotheon>
few months
<apotheon>
hmm
<apotheon>
Maybe I should start using "mew" instead of "new" in IRC, and see if people notice.
<bougyman>
I'm really getting sick of the k8s complexities. I live in the service mesh (istio) space all day every day, where each problem is a unique snowflake.
<leah2>
apotheon: meow
<leah2>
yeah we used istio too
<bougyman>
That's the biggest reason I think I'm gonna take a much lower paying job for a startup where I get to make all the technical decisions.
<bougyman>
Then I can finally live in a nomad world.
<bougyman>
I've pitched nomad to the last 3 joints and gotten no takers.
Nilium has quit [Quit: <kristin> Cower is like... I dunno. sometimes he seems senile and other times he seems like a middle schooler.]
Nilium has joined #ruby
<bougyman>
My mind groks HCL very well. Feels very akin to Prolog.
<apotheon>
Being similar to Prolog is probably not a great way to drive adoption, though it might be a great way to design something.
<bougyman>
It's the first mew language I've learned in a long time that I really enjoyed.
<bougyman>
If I can live in ruby, HCL (and sprinkingling of bash), I'd be in heaven.
<nakilon>
Breaking meows!
<apotheon>
nakilon: nice idea
<apotheon>
Array.mew
<bougyman>
Last few places I've been have been heavy on the python. Looking at python gives me headaches. Not metaphorically, actual pain.
<apotheon>
ouch
<bougyman>
Something about the assymetry, probably.
<nakilon>
Purray = Array
<apotheon>
I actually enjoyed writing Python at one job -- where the Python I wrote was almost entirely for Flask.
<bougyman>
I don't mind writing in it.
<apotheon>
I had to look at it, too.
<bougyman>
I keep the indentation to a minimum and write super short methods.
<apotheon>
(obviously)
<bougyman>
It's looking at other people's 16 levels of indent that hurts.
<apotheon>
Oh, yeah, that's awful.
<apotheon>
Flask is very Rubyish for Python, and encourages writing more Rubyish code.
<nakilon>
*Meowish
<apotheon>
I've seen far too many superfunctions in other people's Python code. It drives me a bit nuts.
<bougyman>
and then there's the list comprehension "geniuses" that pride themselves on their golfing.
<bougyman>
That's probably the worst.
<apotheon>
I once had to deal with a single Ruby method that was over a thousand lines of code, and I still preferred refactoring that over dealing with most Python code.
<nakilon>
alias blep sleep
<nakilon>
alias purrs puts
<nakilon>
this can be a gem
<apotheon>
I think I broke nakilon . . .
weaksauce has joined #ruby
alextee has quit [Ping timeout: 264 seconds]
orbyt_ has joined #ruby
Technodrome has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
cadeskywalker has quit [Quit: WeeChat 2.3]
cadeskywalker has joined #ruby
alextee has joined #ruby
ruurd has quit [Read error: Connection reset by peer]
ruurd has joined #ruby
Technodrome has joined #ruby
xf3rno has joined #ruby
sy has left #ruby [#ruby]
xf3rno has quit [Ping timeout: 264 seconds]
cnsvc has quit [Ping timeout: 268 seconds]
cnsvc has joined #ruby
jla has quit [Quit: Leaving.]
naftilos76 has quit [Ping timeout: 246 seconds]
MzrE has joined #ruby
apotheon has quit [Ping timeout: 240 seconds]
apotheon has joined #ruby
TCZ has joined #ruby
MzrE has quit [Quit: Leaving]
jla has joined #ruby
postmodern has joined #ruby
elcontrastador has joined #ruby
jla has quit [Ping timeout: 246 seconds]
Technodrome has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
cow[moo] has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
arahael2 is now known as arahael
GodFather has joined #ruby
GodFather_ has joined #ruby
dfucci has quit [Ping timeout: 240 seconds]
dfucci has joined #ruby
dfucci has quit [Ping timeout: 246 seconds]
TCZ has quit [Remote host closed the connection]
naftilos76 has joined #ruby
neshpion has quit [Quit: neshpion]
ruby[bot] has quit [Remote host closed the connection]
ruby[bot] has joined #ruby
dfucci has joined #ruby
dfucci has quit [Ping timeout: 264 seconds]
dfucci has joined #ruby
dfucci has quit [Ping timeout: 246 seconds]
naftilos76 has quit [Ping timeout: 272 seconds]
orbyt_ has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]