<warmwaffles>
So we decided to actually stick this data from google into an actual data storage engine. Ruby was just taking too long to do aggregate data
knowtheory has quit [Quit: Computer has gone to sleep]
knowtheory has joined #rom-rb
knowtheory has quit [Ping timeout: 240 seconds]
<dkubb>
warmwaffles: interesting. what datastore?
<warmwaffles>
good ole SQL, we need aggregate and data grouping
<warmwaffles>
plus the data in the table is throw away, so if we make changes, it wont be a big deal
<warmwaffles>
we are doing spikes right now
<warmwaffles>
Team like mutant though, we are definitely going to give it a go
<warmwaffles>
I had to talk the team out of doing a NoSQL solution because the data definitely had relational information
<warmwaffles>
Though I am extremely intrigued by all these little libraries
<warmwaffles>
Still using that Transform class you helped me with last night.
<dkubb>
yeah, that sounds like a good choice
<dkubb>
I personally wouldn't do a ton of transformations in ruby
<dkubb>
probably just enough to get it into a sane format and them dump it into a db
<dkubb>
it also depends on whether it's a one-off or if it has to be run continuously. if it's a one-off, then I would do whatever gets me the answer fastest, even if it takes longer to run
<dkubb>
for something continuous, which it sounds like this is, I would focus on getting the data into the database as quickly as possible. I'd use yajl-ruby to parse the json as a stream, and then for each "node" I would transform and write to the db
<dkubb>
minimizing buffering can make a big difference in this case
<warmwaffles>
yea
<warmwaffles>
dkubb: have you used sidekiq at all?
<dkubb>
warmwaffles: no I haven't yet
dkubb has quit [Quit: Leaving...]
postmodern has joined #rom-rb
snusnu has quit [Quit: Leaving.]
snusnu has joined #rom-rb
snusnu has quit [Client Quit]
mbj has joined #rom-rb
dkubb has joined #rom-rb
<mbj>
dkubb: hi
moonglum has joined #rom-rb
skade has joined #rom-rb
mbj has quit [Read error: Connection reset by peer]
mbj has joined #rom-rb
skade has quit [Quit: Computer has gone to sleep.]
skade has joined #rom-rb
skade has quit [Client Quit]
skade has joined #rom-rb
skade has quit [Client Quit]
skade has joined #rom-rb
postmodern has quit [Quit: Leaving]
zekefast has joined #rom-rb
moonglum has quit [Ping timeout: 272 seconds]
mbj has quit [Quit: Lost terminal]
snusnu1 has joined #rom-rb
zekefast has quit [Quit: Leaving.]
cored has joined #rom-rb
cored has joined #rom-rb
cored has quit [Changing host]
zekefast has joined #rom-rb
snusnu1 has quit [Quit: Leaving.]
breakingthings has joined #rom-rb
knowtheory has joined #rom-rb
moonglum has joined #rom-rb
gildo has joined #rom-rb
CraigBuchek has joined #rom-rb
skade has quit [Quit: Computer has gone to sleep.]
skade has joined #rom-rb
skade has quit [Client Quit]
snusnu has joined #rom-rb
skade has joined #rom-rb
skade has quit [Quit: Computer has gone to sleep.]
bf4 has joined #rom-rb
irclogger__ has joined #rom-rb
skade has joined #rom-rb
snusnu has quit [Quit: Leaving.]
kleech has joined #rom-rb
mkristian has joined #rom-rb
mkristian has quit [Read error: Connection reset by peer]
skade has quit [Quit: Computer has gone to sleep.]
snusnu has joined #rom-rb
mbj has joined #rom-rb
breakingthings has quit []
skade has joined #rom-rb
zekefast has quit [Quit: Leaving.]
skade has quit [Quit: Computer has gone to sleep.]