<mynameisdebian>
I have an array (a) with about 4.5 million members, each itself being an array containing a good amount of data. All of this was extracted from a CSV about 1GB in size. I have over 7GB free. When I run this code in IRB I get an error "failed to allocate memory": https://gist.github.com/AumCoin/4c2993fc548b8966503b96e74ac48013 . Anyone know why this can't allocate the memory? Is there a workaround?
<mynameisdebian>
Might be wrong about having 7GB free, but the machine is a headless Linux instance with nothing else running on it, and has 8GB RAM
Azure has quit [Read error: Connection reset by peer]
Azure has joined #ruby
witlesswonder__ has joined #ruby
AJA4351 has joined #ruby
witlesswonder_ has quit [Ping timeout: 260 seconds]
xarthna has quit [Ping timeout: 240 seconds]
AJA4350 has quit [Ping timeout: 276 seconds]
AJA4351 is now known as AJA4350
Intelo has quit [Ping timeout: 240 seconds]
hiroaki has quit [Ping timeout: 245 seconds]
fphilipe has joined #ruby
n88 has quit [Ping timeout: 240 seconds]
fphilipe has quit [Ping timeout: 250 seconds]
<Iambchop>
mynameisdebian: can you provide a small example input and output for your problem?
<leftylink>
if I may make a completely uneducated guess. I would assume that `a[1...a.length]` would create another array. maybe Ruby just didn't have enough memory to do that. I don't know offhand whether Ruby is able to optimise this to a copy-on-write (whether it even does such a thing)
<baweaver>
If you do a range subquery it totally does
<baweaver>
mynameisdebian: what's the input look like?
<leftylink>
I was unclear and now I want to know
<leftylink>
does "it totally does" mean "it totally does create a new array" or "it totally does do a copy-on-write"
<baweaver>
Also you left something off
<leftylink>
I guess "do" is not the right word
<leftylink>
"create a virtual array that has copy-on-write effect" I guess
<baweaver>
m[3]
<baweaver>
That means you iterated once already
<baweaver>
So you're leaving some parts of the code off that are consuming more memory
drincruz has joined #ruby
<baweaver>
The other thing is never read in the entire file like that for larger sizes
<mynameisdebian>
ok guys, my instance is frozen, rbeooting and I'll try to answer some of these questions
hiroaki has joined #ruby
poontangmessiah has quit [Remote host closed the connection]
poontangmessiah has joined #ruby
poontangmessiah has quit [Max SendQ exceeded]
<baweaver>
You probably want CSV.foreach
<mynameisdebian>
This is the whole code, except ignore the uncommented comment line in the middle: https://dpaste.de/CtbL
poontangmessiah has joined #ruby
<baweaver>
Yeah, you're performing a ton more allocations
poontangmessiah has quit [Max SendQ exceeded]
poontangmessiah has joined #ruby
poontangmessiah has quit [Remote host closed the connection]
ogres has quit [Quit: Connection closed for inactivity]
<baweaver>
because some people still have data in that format.
<mynameisdebian>
CSV because that's just how the data is structured, and I need to do some calculations that are either very hard to do directly on a DB, or that I do not know how to do on a DB
<SeepingN>
I have lots of data in csv. but when it's ...what was that, 11 million lines that I saw?
<mynameisdebian>
4.5 million
<SeepingN>
uh ya
<SeepingN>
awk maybe? lol definitely need soemthing that reads in chunks
<baweaver>
Yeah
<baweaver>
CSV.foreach
<Iambchop>
and you're trying to group the entries by the fourth column?
<baweaver>
al2o3-cr / havenwood: Did you know it did that?
<baweaver>
I sure didn't
<baweaver>
"If the n argument is given, maximum n elements are returned as an array. These n elements are sorted by the value from the given block, in descending order."
<NL3limin4t0r>
You can build the gem using `gem build gemspec_file.gemspec` then install with `gem install /path/to/gem_file.gem`.
fphilipe has joined #ruby
<NL3limin4t0r>
If you're using bundler you don't have to install the gem to test it out. You can use it with `bin/console` instead.
poontangmessiah has joined #ruby
Intelo has quit [Ping timeout: 265 seconds]
darkwingchuck has joined #ruby
Intelo has joined #ruby
cyclonis_ has joined #ruby
bradleyprice has joined #ruby
cyclonis has quit [Ping timeout: 240 seconds]
rafadc has joined #ruby
<Bish>
but im not making my own gem.. im trying to modify one
<Bish>
isn't it colliding in some sort?
<Bish>
NL3limin4t0r:
bradleyprice has quit [Ping timeout: 240 seconds]
<kp666[m]>
Bish: you could build from the modified code or you could give that as a source in your gemfile
<NL3limin4t0r>
Those manuals offer more info than only say how to initialize a new gem. They talk about folder structure, building and installing the gem localy, publishing the gem etc.
<NL3limin4t0r>
So they should also apply to a forked gem.
blackmesa has joined #ruby
queip has quit [Ping timeout: 250 seconds]
dionysus69 has quit [Remote host closed the connection]
dionysus69 has joined #ruby
queip has joined #ruby
DTZUZO has quit [Ping timeout: 265 seconds]
DTZUZO has joined #ruby
queip has quit [Ping timeout: 246 seconds]
queip has joined #ruby
DTZUZO has quit [Ping timeout: 240 seconds]
mossplix_ has quit [Remote host closed the connection]
mossplix has joined #ruby
cyclonis has joined #ruby
DTZUZO has joined #ruby
cyclonis_ has quit [Ping timeout: 245 seconds]
mossplix has quit [Ping timeout: 240 seconds]
lucasb has joined #ruby
mossplix has joined #ruby
DTZUZO has quit [Ping timeout: 268 seconds]
dionysus69 has quit [Remote host closed the connection]
dionysus69 has joined #ruby
DTZUZO has joined #ruby
DTZUZO has quit [Ping timeout: 240 seconds]
DTZUZO has joined #ruby
zodd has joined #ruby
darkwingchuck has quit [Ping timeout: 245 seconds]
jacksoow_ has joined #ruby
jacksoow has quit [Ping timeout: 240 seconds]
stryek has joined #ruby
DTZUZO has quit [Ping timeout: 265 seconds]
DTZUZO has joined #ruby
Obast has joined #ruby
DTZUZO has quit [Ping timeout: 268 seconds]
<Obast>
Hi. Ive created a hash like this: myhash=["Ekonomistyrning" => "Ekonomi", "Fastigheter" => "Fastighet"] When I try 'puts myhash["Fastigheter"]' I get 'TypeError: no implicit conversion of String into Integer'. I suppose I am missing some basic ruby?
no_gravity has joined #ruby
DTZUZO has joined #ruby
sandstrom has joined #ruby
jacksoow_ has quit [Ping timeout: 252 seconds]
n88 has joined #ruby
akem_ has joined #ruby
<NL3limin4t0r>
Obast: That's because you use `[...]` and not `{...}`.
thad_the_man_2 has quit [Remote host closed the connection]
blackmesa has quit [Ping timeout: 246 seconds]
<NL3limin4t0r>
You should either change the definition to use {}, or change the access to `myhash[0]["Fastigheter"]`
<no_gravity>
Hello! How would this line look like in Ruby? $names=['Sue','Joe','Min']; foreach($names as $name) echo "Hello $name!\n";
inkvar has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
<Obast>
A bit strange result. So when you put "hash code/data" inside array brackets ruby kind of creates a hash behind the scenes, and adds the result to the array?
<NL3limin4t0r>
Will for example have 3 parameters, 1, 2 and {option_a: true, option_b: false}
<zodd>
adam12, I fixed my DBI-ODBC issue. Turned out I forgot to install the dbi-odbc gem :P
akem_ has quit [Quit: Leaving]
mossplix has quit [Remote host closed the connection]
<zodd>
error handling could have been better
akemhp has joined #ruby
Intelo_ has joined #ruby
mossplix has joined #ruby
darkwingchuck has joined #ruby
<NL3limin4t0r>
Obast: However implicit hash convertion only kicks in for the last thing in the list. `[a: 1, 2]` would yield an exception and not `[{a: 1}, 2]`
<NL3limin4t0r>
&>> [a: 1, 2]
<rubydoc>
stderr: -e:4: syntax error, unexpected ']', expecting =>... check link for more (https://carc.in/#/r/7t4d)
DTZUZO has quit [Ping timeout: 276 seconds]
Intelo has quit [Ping timeout: 276 seconds]
<NL3limin4t0r>
no_gravity: If you strings don't contain any spaces you could also use the %w syntax
<no_gravity>
NL3limin4t0r: What is that |xyz| syntax doing?
darkwingchuck has quit [Ping timeout: 268 seconds]
poontangmessiah has quit [Remote host closed the connection]
<NL3limin4t0r>
no_gravity: The "each" method loops though each element in the list executing the { ... } block for each element. The element itself will be passed as parameter. With `|name|` you basically say accept one parameter and give it the name "name".
GodFather has joined #ruby
<NL3limin4t0r>
Basically the same as `as $name` in your code.
<NL3limin4t0r>
You can name it anything you want, as long as it is an valid variable name.
<ruby[bot]>
no_gravity: we in #ruby do not like pastebin.com, it loads slowly for most, has ads which are distracting and has terrible formatting. Please use https://gist.github.com
<no_gravity>
ruby[bot]: The /raw/ version has no adds, perfect formatting and loads blazingly fast.
DTZUZO has joined #ruby
mossplix has joined #ruby
Technodrome has joined #ruby
queip has quit [Ping timeout: 240 seconds]
brendan- has joined #ruby
SuspiciousMinds is now known as SzeregowyPosel
no_gravity has quit [Quit: leaving]
blackmesa has quit [Ping timeout: 245 seconds]
queip has joined #ruby
s2013 has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
darkwingchuck has joined #ruby
s2013 has joined #ruby
s2013 has quit [Client Quit]
blackmesa has joined #ruby
SzeregowyPosel has quit [Quit: Bye Bye]
<al2o3-cr>
names = {'Joe', 'Sue', 'Min'} for _, name in ipairs(names) do print(("Hello %s!"):format(name)) end
<al2o3-cr>
noodle: theres one for you in lua for your collection
<al2o3-cr>
oh, they've gone. sorry noodle
BTRE has quit [Remote host closed the connection]
bradleyprice has joined #ruby
blackmesa has quit [Ping timeout: 276 seconds]
<NL3limin4t0r>
al2o3-cr: yep, was about to add some aditional info then I saw I tagged the wrong nickname
BTRE has joined #ruby
<al2o3-cr>
NL3limin4t0r: yep, i don't know how i didn't see them leave i've got join/leaves on too.
<al2o3-cr>
*join/quit
darkwingchuck has quit [Ping timeout: 240 seconds]
<NL3limin4t0r>
al2o3-cr: I have smart_filter enabled so I'm not seeing all join/quit messages.
<NL3limin4t0r>
I find them to clutter the actual messages, since irc is pretty quite most days.
<al2o3-cr>
NL3limin4t0r: i should activate that at some point, it does get a bit distracting sometimes.
bradleyprice has quit [Remote host closed the connection]
<RougeR>
return super(params.empty? ? params : query) if Setting.log_storage_type&.casecmp?('POSTGRES')
<RougeR>
out of curiosity
<RougeR>
in your opinion is that okay code
<RougeR>
im totally anticipating my manager to shit on it
<RougeR>
hes from a .net background...struggle to get him to give in to ruby sometimes
akem__ has joined #ruby
<RougeR>
changed the setting to 'log_store' instead of 'log_storage_type' its still obviosuly but shorter
<phaul>
well.. it's all subjective. one thing I useon my personal stuff is reek.Now I have a policy that if reek and rubocop are both happy then it's good
<RougeR>
eyyyyyy
<RougeR>
(query || params)
<RougeR>
can use this
<RougeR>
as query is nil
<RougeR>
mmm interesting
<RougeR>
rubocop is not screaming
wildtrees has joined #ruby
<RougeR>
all its got is branch asignment too high
<RougeR>
but fuck that lol
<RougeR>
actually this looks nice
<RougeR>
return super(query || params) if Setting.log_store&.casecmp?('POSTGRES')
akemhp_ has quit [Ping timeout: 240 seconds]
<phaul>
it's more difficult to keep reek happy :) that actually sometimes requires redesign
<RougeR>
not used reek much may check it
<RougeR>
lol whast your max line char
<RougeR>
we have some monsters :l
<phaul>
but others don't care about reek.. as I said it's subjective
<RougeR>
i try keep under 120
sameerynho has joined #ruby
<RougeR>
i know rubocop is 80
<phaul>
home projects: 80. new job: 120
<RougeR>
cool cool
<RougeR>
interesting
<RougeR>
i duno what our logest is at work lol
<RougeR>
too long...
AJA4351 has joined #ruby
AJA4350 has quit [Ping timeout: 268 seconds]
AJA4351 is now known as AJA4350
istrasci has joined #ruby
<istrasci>
How do you override a method in a class from an included module?
<phaul>
there is prepend as well as include
<phaul>
but I'm not 100% I understand the question yet
<phaul>
but if you prepend then the prpepended module will win over the class
<jeremycw>
I think that if it doesn't act like where it shouldn't be called where. Call it query or something so there's no expectation.
<istrasci>
If I have `class A; include B; end`, and B#c exists, I can call `A.new.c`.
<jeremycw>
otherwise someone is going to write code that will only work on postgress and not on elasticsearch without even realizing.
wildtrees has quit [Quit: Leaving]
<istrasci>
I want to override the method `c` to be something like `def c; some_condition? ? st_else : call_B#c; end`.
<phaul>
istrasci: you just include and if you overide in the class you can use super to invoke the included modules method
<istrasci>
That's what I was thinking, but I thought `super` was just for pure inheritance.
<phaul>
include under the hood is pure inheritance
<istrasci>
phaul: Although it doesn't seem to be working for me. I'll play around with it some more.
<phaul>
&>> module B; def foo; 13; end; end; class A; include B; def foo; super * 2; end; end; A.new.foo
<phaul>
but I haven't realized how long single liner I wrote :D
<Net>
supsup: is this a filesystem?
<Net>
you want to see if the currentPage is a file under the link directory?
<supsup>
i want to know if the current 'doc''s children collection contain the currentPage compared against the collections link property
<supsup>
i'm drawing a navigation, have no problems making the child active, but i cannot make the parent active
<supsup>
guessing i need to make a helper function that basically passes in the currentPage and the current docs collection
<supsup>
which would trigger more recursion if more collections are in there.
<jhass>
might very well be a regression, I switched from a old thoughtbot/neat to CSS grid
<supsup>
and does not find a match at the current level
Nicmavr has quit [Read error: Connection reset by peer]
Nicmavr has joined #ruby
<phaul>
possible. I can't say for sure but I kind of remember scrollable div boxes in this case
`nutcr4ck3r` has joined #ruby
jacksoow has joined #ruby
mossplix has joined #ruby
<RougeR>
phaul,
<RougeR>
haha okay that was a lot of dicking about
<RougeR>
i just went and looked at the active record source code in the end and used their where method signature
Intelo has joined #ruby
orbyt_ has joined #ruby
nutcr4ck3r has quit [Ping timeout: 246 seconds]
cloaked1_ has joined #ruby
<Net>
supsup: did you figure it out?
<supsup>
working on it
<supsup>
prob will figure it out
<Net>
👍🏼
<Net>
I think I understand what you mean now if you need to talk about it
<Net>
> i'm drawing a navigation
<Net>
saying that helped clear it up
<supsup>
i don't normally use ruby, so a bit of a syntax curve
<Net>
I'm sure you're aware, you can just write a regular recursive function
cloaked1 has quit [Ping timeout: 268 seconds]
<RougeR>
what you trying to do>
mossplix has quit [Remote host closed the connection]
mossplix has joined #ruby
schne1der has quit [Ping timeout: 245 seconds]
mossplix has quit [Ping timeout: 240 seconds]
hiroaki_ has quit [Ping timeout: 246 seconds]
AJA4351 has joined #ruby
AJA4350 has quit [Ping timeout: 240 seconds]
AJA4351 is now known as AJA4350
SzeregowyPosel has joined #ruby
tdy has quit [Ping timeout: 240 seconds]
fphilipe has joined #ruby
<jhass>
phaul: should be better now
dbz__ has joined #ruby
<Frankenstein>
if I have an array `foo = [1,2,3,4]` is there any semantic difference (or potential pitfalls) between `bar *foo` and `bar(*foo)`?
darkwingchuck has quit [Ping timeout: 240 seconds]
<jhass>
not for ruby but it's one char short of bar * foo that is bar() * foo, so some people argue it's harder to read
dbz_ has quit [Ping timeout: 240 seconds]
<jhass>
whereas bar(* foo) still does what you want
<Frankenstein>
ah, ok
fphilipe has quit [Ping timeout: 250 seconds]
<jhass>
additionally bar*foo also doesn't do what you want ofc
<jhass>
or bar* foo
nutcr4ck3r has joined #ruby
darkwingchuck has joined #ruby
<Frankenstein>
this is starting to remind me of c
<jhass>
:D
<jhass>
except in C two of the three do the same thing and people can't agree on what's the right way to write it
`nutcr4ck3r` has quit [Ping timeout: 265 seconds]
<Frankenstein>
yeah, at least it means different things in ruby
ellcs has quit [Ping timeout: 264 seconds]
darkwingchuck has quit [Ping timeout: 240 seconds]
`nutcr4ck3r` has joined #ruby
nutcr4ck3r has quit [Ping timeout: 268 seconds]
Fernando-Basso has joined #ruby
darkwingchuck has joined #ruby
reber has joined #ruby
Axy has joined #ruby
Axy has quit [Changing host]
Axy has joined #ruby
Mia has quit [Ping timeout: 240 seconds]
cloaked1 has joined #ruby
cloaked1 has joined #ruby
cloaked1 has quit [Changing host]
`nutcr4ck3r` has quit [Ping timeout: 240 seconds]
cloaked1_ has quit [Ping timeout: 265 seconds]
drincruz has quit [Ping timeout: 268 seconds]
greengriminal has quit [Quit: This computer has gone to sleep]
reber has quit [Ping timeout: 240 seconds]
Technodrome has joined #ruby
darkwingchuck has quit [Ping timeout: 265 seconds]
bambanx_ has joined #ruby
bradleyprice has quit [Remote host closed the connection]
nutcr4ck3r has joined #ruby
bambanx has quit [Ping timeout: 265 seconds]
nutcr4ck3r has left #ruby [#ruby]
greengriminal has joined #ruby
ogres has joined #ruby
greengriminal has quit [Client Quit]
jeremycw has quit [Ping timeout: 276 seconds]
bambanx_ has quit [Quit: Leaving]
AJA4351 has joined #ruby
bambanx has joined #ruby
supsup has quit [Remote host closed the connection]
AJA4350 has quit [Ping timeout: 246 seconds]
AJA4351 is now known as AJA4350
crichoux has quit [Ping timeout: 245 seconds]
bradleyprice has joined #ruby
jhass has quit [Quit: Bye]
absolutejam has quit [Ping timeout: 240 seconds]
dbz__ has quit [Remote host closed the connection]
xGrind has joined #ruby
dbz_ has joined #ruby
gix has quit [Quit: Client exiting]
jhass has joined #ruby
sameerynho has quit [Ping timeout: 245 seconds]
dbz_ has quit [Ping timeout: 276 seconds]
darkwingchuck has joined #ruby
darkwingchuck has quit [Ping timeout: 240 seconds]
mynameisdebian has joined #ruby
<mynameisdebian>
baweaver, thanks for all your help yesterday
blackmesa1 has joined #ruby
thebananaking has joined #ruby
SzeregowyPosel has quit [Quit: Bye Bye]
<mynameisdebian>
I am working on an app for my work. My boss wants me to do various kinds of testing, some to catch when my code breaks down the line, and some to catch when the upstream DB I am reading from changes. I don't have any experience with testing in general. I have heard of unit testing and various Ruby testing suites, but I'm not 100% either of those relate to the matter at hand. Should I be using unit testing or a testing suite for this, and can anyone
<mynameisdebian>
recommend a guide somewhere that will bring me up to speed on what kinds of things should be tested and how?
bradleyprice has quit [Remote host closed the connection]
xarthna_ has quit [Ping timeout: 268 seconds]
tpanarch1st has quit [Ping timeout: 240 seconds]
dbz_ has joined #ruby
nowhere_man has quit [Ping timeout: 276 seconds]
wickedbloodfart has joined #ruby
cloaked1 has quit [Ping timeout: 240 seconds]
wickedbloodfart has quit [Client Quit]
eljimmy has quit [Quit: This computer has gone to sleep]
eljimmy has joined #ruby
eljimmy has quit [Client Quit]
eljimmy has joined #ruby
eljimmy has quit [Client Quit]
cyclonis_ has joined #ruby
eljimmy has joined #ruby
stryek has quit [Quit: Connection closed for inactivity]
eljimmy has quit [Client Quit]
eljimmy has joined #ruby
cyclonis has quit [Ping timeout: 268 seconds]
eljimmy has quit [Client Quit]
tpanarch1st has joined #ruby
eljimmy has joined #ruby
eljimmy has quit [Client Quit]
eljimmy has joined #ruby
eljimmy has quit [Client Quit]
eljimmy has joined #ruby
eljimmy has quit [Client Quit]
CableNinja_ has quit [Ping timeout: 240 seconds]
<Iambchop>
mynameisdebian: there is a "testing" section in the books link linked from the channel topic https://goo.gl/wpGhoQ
<mynameisdebian>
Iambchop, meant to thank you too
<mynameisdebian>
also thanks
Axy has quit [Read error: Connection reset by peer]
Mia has joined #ruby
Mia has joined #ruby
Mia has quit [Changing host]
frem has quit [Quit: Connection closed for inactivity]
xGrind has quit [Quit: Saindo]
fphilipe has joined #ruby
cyclonis_ has quit [Quit: -a- IRC for Android 2.1.55]
cyclonis has joined #ruby
ExoUNX has quit [Quit: later fam...]
AJA4350 has quit [Ping timeout: 240 seconds]
SzeregowyPosel has joined #ruby
fphilipe has quit [Ping timeout: 245 seconds]
Mia has quit [Read error: Connection reset by peer]
Mia has joined #ruby
Mia has quit [Changing host]
Mia has joined #ruby
Intelo has quit [Ping timeout: 240 seconds]
darkwingchuck has joined #ruby
Mia has quit [Read error: Connection reset by peer]
<Frankenstein>
is there a good light-weight migrations tool for a non-rails project?
Mia has joined #ruby
Mia has quit [Changing host]
Mia has joined #ruby
darkwingchuck has quit [Ping timeout: 240 seconds]