chendo_ has quit [Quit: Computer has gone to sleep.]
fbernier has joined #ruby-lang
fbernier has quit [Client Quit]
fbernier has joined #ruby-lang
JustinCampbell has quit [Remote host closed the connection]
JustinCampbell has joined #ruby-lang
JustinCampbell has quit [Remote host closed the connection]
kvirani has joined #ruby-lang
kvirani has quit [Remote host closed the connection]
savage- has quit [Ping timeout: 246 seconds]
chessguy has quit [Remote host closed the connection]
cndiv has joined #ruby-lang
chessguy has joined #ruby-lang
chendo_ has joined #ruby-lang
z3r00ld_ has joined #ruby-lang
Swimming_Bird has quit [Quit: Computer has gone to sleep.]
ltd has quit [Ping timeout: 264 seconds]
towski has quit [Remote host closed the connection]
<z3r00ld_>
hello, i am trying to use net-https, but getting this error: "/usr/lib/ruby/1.8/net/http.rb:567:in `initialize': can't convert OpenSSL::X509::Certificate into String (TypeError)"
<Mon_Ouie>
Or maybe just the contents of that file
chessguy has quit [Remote host closed the connection]
krohrbaugh has quit [Quit: Leaving.]
<z3r00ld_>
i did convert certs and keys to .pem format, do i need to strip the --BEGIN-- and --END-- lines ?, OpenSSL reads the certs and keys file ok though, its the net-http which has issue
jtoy has quit [Quit: jtoy]
chessguy has joined #ruby-lang
gsav has joined #ruby-lang
Mon_Ouie has quit [Ping timeout: 240 seconds]
gsav has quit [Read error: Connection reset by peer]
ltd has joined #ruby-lang
diegoviola has joined #ruby-lang
justinmcp has quit [Ping timeout: 240 seconds]
<z3r00ld_>
Mon_Ouie: directly passing ca file gives this error: "not enough data (OpenSSL::X509::CertificateError)"
chessguy has quit [Remote host closed the connection]
ltd has quit [Ping timeout: 246 seconds]
VGoff is now known as VGoff_afk
arooni-mobile has joined #ruby-lang
robbyoconnor has joined #ruby-lang
robbyoconnor has quit [Read error: Connection reset by peer]
m3nd3s has quit [Remote host closed the connection]
m3nd3s has joined #ruby-lang
m3nd3s has quit [Read error: Connection reset by peer]
m3nd3s_ has joined #ruby-lang
m3nd3s_ has quit [Remote host closed the connection]
kain_ has joined #ruby-lang
kain_ has quit [Client Quit]
kain_ has joined #ruby-lang
justinmcp has quit [Remote host closed the connection]
kain has quit [Ping timeout: 252 seconds]
justinmcp has joined #ruby-lang
justinmcp has quit [Remote host closed the connection]
justinmcp has joined #ruby-lang
justinmcp has quit [Remote host closed the connection]
postmodern has quit [Remote host closed the connection]
postmodern has joined #ruby-lang
jperry2_ has quit [Quit: jperry2_]
jaska has quit [Ping timeout: 240 seconds]
Asad_ has quit [Quit: Asad_]
havenn has joined #ruby-lang
gsav has quit [Read error: Connection reset by peer]
yxhvuud has quit [Ping timeout: 246 seconds]
itcharlie has left #ruby-lang [#ruby-lang]
havenn has quit [Remote host closed the connection]
sn0wb1rd has joined #ruby-lang
gsav has joined #ruby-lang
Asad_ has joined #ruby-lang
sjuxax has joined #ruby-lang
<sjuxax>
Can anyone help me understand what is happening with memory usage at http://dpaste.com/802836/? I have also tried removing elements from l with delete_if, but this is really slow and never drops memory either.
<zenspider>
sjuxax: what's the question?
savage- has joined #ruby-lang
<sjuxax>
zenspider: the question is: why isn't the memory associated with the list freed when the list is set to nil, and again why is it not freed when the GC is explicitly called?
<sjuxax>
The only data I have put into the program is taken away when I set it to nil, so GC should take away the 10GB allocation.
<sjuxax>
err, 1.6GB
<zenspider>
ruby's GC is a conservative GC. it doesn't guarantee that things will be collected at any time.
<sjuxax>
I am more concerned that in real life, a use case similar to this slows my code WAY down as it gets farther (using a hash as a temporary lookup table, making the hash zero out at the top of some loop; ultimately millions of rows go through this hash, but should never really have more than ~400k-~500k at a time). It causes both potential memory contention and makes the program really, really slow, presumably as the number of objects for GC to scan never
<zenspider>
conservative means that it takes a fast pass through memory instead of walking every object... if something looks like it points to a valid object, that object is kept. it makes for faster GC passes, but means that some stuff gets kept around
<sjuxax>
OK, so there is no way to invalidate whole chunks of data like that? Can I trigger "deep" introspection or something to make it really release everything I don't need anymore?
<sjuxax>
Is there a trick I can do that will make things definitely look like they don't point to anything else?
<sjuxax>
basically, I need to quickly get this resolved, keep memory usage and GC collection time under control
<bnagy>
sticking your memory hungry code in a separate method might help
<bnagy>
jruby has a better GC
savage- has quit [Remote host closed the connection]
Asad_ has quit [Quit: Asad_]
<zenspider>
for me, the RSS for that process tops around 644520 (I'm only doing 10m iterations on the loop). If I bother to re-iterate and replace each string with "" then I can get the first GC to drop to 426924
<zenspider>
bnagy: yes, but jruby uses about 2x the ram at the start
<bnagy>
sure, and the start is slower
<bnagy>
but for long running processes it has an advantage
<zenspider>
eh. that doesn't seem pertinent here. I assume this process is long standing or it wouldn't be an issue
<zenspider>
sjuxax: what are you actually trying to do?
<zenspider>
sjuxax: are you profiling against 1.8 or 1.9?
<zenspider>
I'm gonna guess 1.8
savage- has joined #ruby-lang
<zenspider>
so... 1.9.3's GC seems to be much better with this type of profile
<zenspider>
I'm guessing because they throw away the parse tree and do other stuff to optimize stings
<zenspider>
I can even get rid of my replace/clear trick and the memory goes down much more on each iteration
<zenspider>
BUT... it starts much higher too
<zenspider>
4.6m -> .4m in 1.8 w/ my tricks
arooni-mobile has quit [Ping timeout: 268 seconds]
<zenspider>
prolly has a lot of tweaks from nari's GC work in 1.9. I dunno, I haven't tracked that much
<zenspider>
I know they did a lot of stuff to cheat with smaller strings, so your code exampleis a little suspect. I multiplied the string by 12 to actually match the size
swarley_ has joined #ruby-lang
<sjuxax>
zenspider: As I said, I am trying to use a hash as a temporary lookup table. It get created and stores several items. I use these items to look up things. When I find something I want, I append a single UUID to a UUID list with uuid_str.dup. In total, there should be about 5000 UUIDs which is like 180 bytes. Then, after a while, I zero the hash back out to an empty hash: {}. Memory never seems to get released, and the program gets slower and slower as
<sjuxax>
I am using 1.9
alvaro_o has quit [Quit: Ex-Chat]
arooni-mobile has joined #ruby-lang
<sjuxax>
Yeah, the "369" was a typo, it was really a 36-character string (a UUID). but yes, using even bigger strings should demonstrate the same issue
<sjuxax>
The process is not long running, without this hash lookup thing the whole process finishes in 20 mins. It is parsing some huge text files that we receive from a third-party, and making INFILEs for insertion to our MySQL database.
cndiv has joined #ruby-lang
chendo_ has quit [Quit: Computer has gone to sleep.]
krohrbaugh has joined #ruby-lang
<sjuxax>
Could this be a bug or should it just be considered normal behavior?
<bnagy>
where are you getting the uuids from, out of interest?
snorkdude has joined #ruby-lang
snorkdude has quit [Remote host closed the connection]
sandbags_ has quit [Remote host closed the connection]
<sjuxax>
In real life, I am generating them with SecureRandom, which produces a string, so .dup should create a separate and unrelated string. In this sample which shows the same thing, I am inserting that literal sample string a lot of times.
<bnagy>
yeah I'm just wondering about the speed not the memory profile
cndiv has quit []
<sjuxax>
FYI python2 doesn't do this. Equivalent code there produces a much tighter list/array to start with for roughly equivalent number of items (33 million in python v. 27 million in Ruby; python using 242M of memory instead of 1.6G), and l = [] in Python instantly drops us back to 0.0% memory use (5K).
MrOnFireMr has quit [Ping timeout: 256 seconds]
gsav has quit [Ping timeout: 252 seconds]
chendo has quit [Ping timeout: 244 seconds]
sepp2k has joined #ruby-lang
chendo has joined #ruby-lang
chendo has quit [Changing host]
chendo has joined #ruby-lang
gsav has joined #ruby-lang
pr0ton has quit [Remote host closed the connection]
gsav has quit [Client Quit]
gsav has joined #ruby-lang
pr0ton has joined #ruby-lang
sn0wb1rd_ has joined #ruby-lang
sn0wb1rd has quit [Ping timeout: 255 seconds]
sn0wb1rd_ is now known as sn0wb1rd
Sambalero has quit [Remote host closed the connection]
jtoy has joined #ruby-lang
pr0ton has quit [Quit: pr0ton]
havenn has joined #ruby-lang
sn0wb1rd has quit [Ping timeout: 244 seconds]
sn0wb1rd has joined #ruby-lang
havenn has quit [Ping timeout: 244 seconds]
S1kx has quit [Ping timeout: 255 seconds]
burgestrand has joined #ruby-lang
znowi_ has quit [Ping timeout: 252 seconds]
ryanf has quit [Quit: leaving]
znowi has joined #ruby-lang
jtoy has quit [Quit: jtoy]
cndiv has joined #ruby-lang
deryl has quit [Read error: Connection reset by peer]
deryl has joined #ruby-lang
arooni-mobile has quit [Ping timeout: 252 seconds]
mantono has quit [Ping timeout: 244 seconds]
jtoy has joined #ruby-lang
mantono has joined #ruby-lang
gsav has quit [Read error: Connection reset by peer]
arooni-mobile has joined #ruby-lang
xyzodiac has quit [Quit: Computer has gone to sleep.]
mistym has quit [Remote host closed the connection]
burgestrand has quit [Quit: Leaving.]
Sambalero has joined #ruby-lang
justinmcp has joined #ruby-lang
steveklabnik has quit [Ping timeout: 268 seconds]
chessguy has quit [Remote host closed the connection]
robbyoconnor has joined #ruby-lang
robbyoconnor has quit [Client Quit]
steveklabnik has joined #ruby-lang
MrOnFireMr has joined #ruby-lang
d3vic3 has quit [Ping timeout: 248 seconds]
robbyoconnor has joined #ruby-lang
dhruvasagar has joined #ruby-lang
d3vic3 has joined #ruby-lang
robbyoconnor has quit [Read error: Connection reset by peer]
robbyoconnor has joined #ruby-lang
swarley_ has quit [Ping timeout: 246 seconds]
robbyoconnor has quit [Ping timeout: 240 seconds]
gmci has quit [Quit: Computer has gone to sleep.]
rippa has joined #ruby-lang
Sambalero has quit [Remote host closed the connection]
diegoviola has quit [Ping timeout: 268 seconds]
jaska has joined #ruby-lang
dfr|mac has joined #ruby-lang
arooni-mobile has quit [Ping timeout: 252 seconds]
WillMarshall has joined #ruby-lang
dhruvasagar has quit [Ping timeout: 246 seconds]
justinmcp has quit [Remote host closed the connection]
dhruvasagar has joined #ruby-lang
seoaqua has joined #ruby-lang
rue|w has joined #ruby-lang
stardiviner has joined #ruby-lang
MrOnFireMr has quit [Ping timeout: 252 seconds]
Mon_Ouie has joined #ruby-lang
rippa has quit [Ping timeout: 248 seconds]
|Vargas| has joined #ruby-lang
|Vargas| has quit [Changing host]
|Vargas| has joined #ruby-lang
z3r00ld has joined #ruby-lang
WillMarshall has quit [Quit: Computer has gone to sleep.]
postmodern has quit [Remote host closed the connection]
josh^ has joined #ruby-lang
z3r00ld has quit [Read error: Connection reset by peer]
jtoy has quit [Quit: jtoy]
z3r00ld has joined #ruby-lang
z3r00ld has quit [Read error: Connection reset by peer]
z3r00ld has joined #ruby-lang
WillMarshall has joined #ruby-lang
ryanf has joined #ruby-lang
sjuxax has quit [Read error: Connection reset by peer]
<njero>
If I change it to -provision it works... is the "-" required?
judofyr has quit [Remote host closed the connection]
judofyr has joined #ruby-lang
judofyr has quit [Read error: Connection reset by peer]
jxie has joined #ruby-lang
judofyr has joined #ruby-lang
elux has joined #ruby-lang
<whitequark>
njero: option parser is for options
<whitequark>
use ARGV and catch commands manually
<whitequark>
alternatively, take a look at gem trollop
sailias has joined #ruby-lang
<whitequark>
there's a nice example of what you're trying to do
<njero>
thanks whitequark, I almost used trollop googling around for how to solve this :)
<injekt>
njero: there's also slop
g0bl1n has quit [Quit: g0bl1n]
<deryl>
i used trollop for dtf. was fast and easy to set up, you can either use it as a gem or just download the single file from the homepage and put it in your lib/ dir then require it. was extremely easy in comparison to OptionParser to use. and i do mean easy
<injekt>
:/
<injekt>
you're not helping
mistym has joined #ruby-lang
<deryl>
i don't see how i'm not but ok
<injekt>
deryl: i was kidding.. i was trying to promote slop
<deryl>
ah :)
<mistym>
slop <3
<njero>
injekt: does slop handle commands... because trollop surely does :)
<injekt>
njero: yes, you wanted commands, I wouldn't have mentioned slop if it didn't support them :)
<mistym>
The opts.flag? style of option checking doesn't seem to get on well with --long-style options, unless there's something I was missing. (Of course I can always do opts['long-style'] but that's not nearly as pretty ;) )
Hakon has joined #ruby-lang
tooky has quit [Remote host closed the connection]
ruurd has quit [Quit: Leaving...]
<darix>
mistym: it didnt work in my testing either.
GarethAdams has joined #ruby-lang
<darix>
i think i even asked injekt about it back then ^^
<injekt>
mistym: yeah, obviously there's not much you can do if ruby doesn't support - in method names. You'd have to use send or present?('foo-bar')
<mistym>
injekt: Yeah, just seems unfortunate because - is really common in long-form option names.
<injekt>
if you have a better idea, i'm of course all ears :)
<injekt>
yeah
<injekt>
I think someone said about allowing foo_bar? to return true for a 'foo-bar'
<mistym>
Translate - to _? opts.long_form? matches --long-form
<injekt>
:P
<mistym>
Great minds think alike!
outoftime has joined #ruby-lang
<mistym>
Theoretically you get a minor option name collision. On the other hand, defining both --foo-bar and --foo_bar options and treating them separately is degenerate madness, so breaking that would be a feature anyway ;)
abletony84 has joined #ruby-lang
<injekt>
tbh I thought I merged something like that, but I guess not. I'm more than happy to, though
<abletony84>
In my home_helper.rb over at http://pastie.org/pastes/4753794 - how do I display my forums alphabetically? Now whenever a forum is visited, that forum appears at the bottom of the list.
<injekt>
yeah maybe that was my first thought
<injekt>
abletony84: forums.sort.map
<abletony84>
injekt: hey thank you so much man!! :-)
<injekt>
hey man you're totally welcome bro
<abletony84>
:)
dhruvasagar has quit [Ping timeout: 255 seconds]
rue|w has quit [Remote host closed the connection]
rue|w has joined #ruby-lang
dhruvasagar has joined #ruby-lang
chimkan has joined #ruby-lang
tooky has joined #ruby-lang
gsav has joined #ruby-lang
chendo has quit [Ping timeout: 252 seconds]
chendo has joined #ruby-lang
chendo has quit [Changing host]
chendo has joined #ruby-lang
gsav has quit [Client Quit]
gsav has joined #ruby-lang
gsav_ has joined #ruby-lang
gsav has quit [Client Quit]
wyhaines has joined #ruby-lang
<hagabaka>
slop should automagically implement the options
<roelof>
I get this error message : path_to_enlightenment.rb:12:in `require': ./about_symbols.rb:66: syntax error, unexpected '\n', expecting tCOLON2 or '[' or '.' (SyntaxError) from path_to_enlightenment.rb:12
xyzodiac has quit [Quit: Computer has gone to sleep.]
imajes has joined #ruby-lang
coryf has quit [Read error: Connection reset by peer]
rolfb has joined #ruby-lang
chickenflu has joined #ruby-lang
workmad3 has joined #ruby-lang
coryf has joined #ruby-lang
<roelof>
nobody ?
chickenflu has quit [Quit: leaving]
coryf_ has joined #ruby-lang
rippa has quit [Ping timeout: 244 seconds]
roelof has quit [Quit: Page closed]
cantonic has quit [Quit: cantonic]
cantonic has joined #ruby-lang
coryf has quit [Ping timeout: 268 seconds]
coryf_ has quit [Read error: Connection reset by peer]
coryf has joined #ruby-lang
swarley_ has joined #ruby-lang
elux has joined #ruby-lang
tonni has joined #ruby-lang
marc3000 has left #ruby-lang [#ruby-lang]
cantonic has quit [Quit: cantonic]
jarib has quit [Excess Flood]
jarib has joined #ruby-lang
derpops has quit [Ping timeout: 268 seconds]
nariyal has quit [Quit: Computer has gone to sleep.]
<jtoy>
any one have recommendations for a fast disk persistent key/value store
<rindolf>
jtoy: Google LevelDB is not too bad from my experience.
<swarley>
jtoy, i like SQLite3
<rindolf>
jtoy: but I have not used it extensively (and only used it from C code).
<swarley>
file based SQL
<jtoy>
rindolf: I was planning to use that, but its a library? I want to use it as a server that any language an access
<rindolf>
swarley: SQLite3 is an SQL database.
<swarley>
ah, sorry
<rindolf>
jtoy: it's a library, yes.
<swarley>
i misread
<rindolf>
jtoy: you can write a server for it.
<jtoy>
rindolf: I might try that
cored has quit [Ping timeout: 264 seconds]
z3r00ld has quit [Quit: Leaving.]
cored has joined #ruby-lang
z3r00ld has joined #ruby-lang
<rindolf>
jtoy: there are also Berkeley DB (which has a GPL-like licence, so may not be suitable for you), and Tokyo Cabinet (LGPL) and Kyoto Cabinet (GPLed).
z3r00ld has left #ruby-lang [#ruby-lang]
toretore has quit [Quit: Leaving]
z3r00ld has joined #ruby-lang
xyzodiac has joined #ruby-lang
<z3r00ld>
hello, I am trying to send a GET request to a https URL which requires certificate, key and ca certificate to authenticate and i am getting this error: "/usr/lib/ruby/1.8/net/http.rb:567:in `initialize': can't convert OpenSSL::X509::Certificate into String (TypeError)"
<z3r00ld>
OpenSSL reads the certificate properly, format is pem, but net-https gives error and unable to read the certificate
<z3r00ld>
swarley: it throws error on declaration of "session.ca_file = OpenSSL::X509::Certificate.new ca" and so for the "session.cert" and "session.key", net-https says unable to read cert/file, type error can't convert it into string
<kith>
thx guys
<rindolf>
(Note: it's a page out of a site I originated and maintain.).
<kith>
i'm just trying to find out how a vendor does a trick i wrote lotsa ruby code for :)
<kith>
out of curiosity
<kith>
so far, i'm horrified how they charge those amounts of money for that kinda solution... :D