<owen1>
so i don't feel confident that redis is a good solution. i was thinking of rabbitMQ but it's as easy to manage/configure as redis.
<owen1>
and most of the examples are java.
bzalasky has joined #ruby-lang
<bnagy>
owen1: I use beanstalkd as a queue
kain has joined #ruby-lang
<bnagy>
there are good gems for AMQP though, I use rabbitmq for 'general' messaging ( pub sub etc )
cordax has joined #ruby-lang
<bnagy>
bunny is nice and simple, and hot-bunnies is pretty slick if you're on jruby, but it's a super-thin wrapper
<bnagy>
I think sidekiq etc use redis under the hood, don't they?
<owen1>
bnagy: how many rabbitMQ servers do u use? i think i want 2. so if rabbit1 is down , rabbit2 takes over until rabbit1 recovers.
<bnagy>
I just use one, but I think rabbitmq clusters fine
<bnagy>
I mean it should, enough telcos use it
<owen1>
bnagy: how many consumers do u have and are they on a different hosts than the rabbit?
<bnagy>
some number of hundreds, and yes
<owen1>
wow
<bnagy>
but like I said, beanstalkd is my main work queue
<bnagy>
so large work items go onto that, rabbit is just smallish comms
cordax has quit [Quit: Computer has gone to sleep.]
<owen1>
are both publishers and consumers of rabbit are ruby? also, do the consumers send an ack back to the rabbit to tell it they're done? (so a job will not get lost)
cordax has joined #ruby-lang
<bnagy>
owen1: again, my 'jobs' are all on beanstalkd :)
<bnagy>
but amqp supports acks for messages, yeah
<owen1>
bnagy: u said u use rabbit for general messaging. can u give a concrete example please?
<bnagy>
I do a lot of pub / sub stuff. Heartbeats, work results, status messages etc
<bnagy>
but all 'noncritical'
<owen1>
bnagy: interesting. so u have multiple clients that send their status to rabbit, and a central dashboard that reads that?
<owen1>
also, why do u use both rabbit and beanstalkd? is there a feature that u like in beanstalkd or there is it just as a learning excercise?
<bnagy>
beanstalkd is very fast and very simple and works well with large items
<bnagy>
owen1: kind of, except no 'dashboard' but that would be trivial if I wrote any web code ever
<owen1>
do u guys suggest that i'll send failed work-items to a failed queue or to a seperate DB? another DB might be nice since i can query the failures easily and persist them seperately.
cordax has quit [Quit: Computer has gone to sleep.]
<bnagy>
it completely depends on your workflow
<owen1>
i need to show reports in a web interface based on failures - ie: message of type A, 1003 failures, yahoo API timeout
<owen1>
it seems like based on the quries i'll have to run, external db is better.
<bnagy>
imho just send a failure message to a queue or via however you do your comms and let a failure message consumer deal with it
<bnagy>
and if it makes sense for it to go into a DB have the failure consumer do that
robbyoconnor has joined #ruby-lang
<bnagy>
then you don't have to refactor tons of crap if you decide to change your approach
vlad_starkov has joined #ruby-lang
jg has joined #ruby-lang
glebm has quit [Ping timeout: 264 seconds]
<owen1>
bnagy: nice, so i'll have some logic, like - only if an item was retry 3 times it is deleted from the 'fail' queue.
havenwood has quit [Remote host closed the connection]
<r0bgleeson>
owen1: why not something like sidekiq? it sounds like you're rebuilding some of what it already does
havenwood has joined #ruby-lang
vlad_starkov has quit [Ping timeout: 258 seconds]
<owen1>
r0bgleeson: never heard about that
<owen1>
let me google
<owen1>
btw, my publisher is ruby and my consumers are node.js
<r0bgleeson>
you'd need to write a node.js client then
<r0bgleeson>
it uses redis as its message queue
<r0bgleeson>
you only really ever pass arguments through redis (like some sort of ID or whatever)
<owen1>
can it handle 300 messages/sec
<r0bgleeson>
i dont know
stardiviner has quit [Ping timeout: 248 seconds]
<r0bgleeson>
what is the content of your messages?
<owen1>
json, less than 1 k
<owen1>
10 keys with some strings
havenwood has quit [Ping timeout: 252 seconds]
<r0bgleeson>
so if you went with Sidekiq, you would create a publisher as a Sidekiq work that would publish the message(JSON), and the consumer(JS) client could read from the redis queue. anyway, dont consider this too much, i havent looked into if it would be good enough
vlad_sta_ has quit [Remote host closed the connection]
tylersmith has joined #ruby-lang
r0kis has joined #ruby-lang
r0kis has quit [Killed (idoru (Spam is off topic on freenode.))]
madb055 has quit [Remote host closed the connection]
madb055 has joined #ruby-lang
tylersmith has quit [Quit: tylersmith]
faen has joined #ruby-lang
maxmanders has quit [Quit: Computer has gone to sleep.]
mytrile has joined #ruby-lang
mytrile has quit [Read error: Connection reset by peer]
mytrile_ has joined #ruby-lang
banisterfiend has joined #ruby-lang
ryez has quit [Quit: Page closed]
jg has quit [Ping timeout: 255 seconds]
sepp2k has quit [Remote host closed the connection]
mrfelix has joined #ruby-lang
Gaelan has quit [Remote host closed the connection]
Gaelan has joined #ruby-lang
postmodern has quit [Quit: Leaving]
mrfelix has quit [Ping timeout: 256 seconds]
Gues_____ has joined #ruby-lang
tsion has quit [Quit: Leaving]
<mbj>
whitequark: '"foo"' is lexed as a single tSTRING token with value "foo", getting source maps for delimiters is difficult, what about lexing it with tSTRING_BEG, tSTRING_CONTENTS and tSTRING_END ?
<mbj>
whitequark: We already have a builder for that case
<mbj>
whitequark: We could generate source ranges via tok[0].chr, tok[-1].chr from the actions in parser. But IMHO it is more clean to just remove the single token tSTRING, okay we also have ?a style strings, but these could be handled via a tSTRING_BEG_Q and tSTRING_CONTENTS ?
<mbj>
whitequark: For now I'll not change the lexer and parser actions and handle this in the builder, but in the longerm I think the remove of the tSTRING token would remove some actions.
jg has joined #ruby-lang
madb055 has quit [Remote host closed the connection]
<andersbr>
I'm generating some ruby code, and I want to be able to write out a string to a file in such a way that, when the file is eval'ed or required, the string is recreated. It seems to be way more difficult than I'd expected.
carloslopes has quit [Remote host closed the connection]
dhruvasagar has joined #ruby-lang
dr_bob has quit [Read error: Connection reset by peer]
dr_bob has joined #ruby-lang
chessguy has quit [Ping timeout: 256 seconds]
Oligarchy has quit [Quit: Reconnecting]
Oligarchy has joined #ruby-lang
Oligarchy has quit [Client Quit]
mossplix has quit [Read error: Connection reset by peer]
<rue>
A quine?
<shachaf>
andersbr: It's a fun exercise!
<shachaf>
There's a trick to it which has many other applications.
<andersbr>
Does it involve eval?
<shachaf>
No, no eval.
mbj has joined #ruby-lang
<shachaf>
Well, you can do it without eval.
krohrbaugh has joined #ruby-lang
<dr_bob>
String#inspect should do the job.
<andersbr>
shachaf: Is that what you had in mind?
<manveru>
today matz has birthday :)
<shachaf>
There are lots of ways to make it easier.
maxmanders has joined #ruby-lang
Mon_Ouie has quit [Ping timeout: 255 seconds]
krohrbaugh has quit [Ping timeout: 252 seconds]
mossplix has joined #ruby-lang
wallerdev has quit [Quit: wallerdev]
<dr_bob>
shachaf, birthday? ;-)
<kalleth>
trying to do a gem install middleman but nothing happens... when I run gem install --debug middleman i get lots of OpenSSL::SSL::SSLError - read would block messages
<kalleth>
its a new environment so its possible i missed a package
Gaelan has quit [Remote host closed the connection]
Gaelan has joined #ruby-lang
<rue>
Ehmg, it’s dr_bob
security is now known as megha
ruby-lang355 has joined #ruby-lang
chessguy has joined #ruby-lang
ruby-lang355 has quit [Client Quit]
dr_bob has quit [Quit: Tune in next week when you'll hear Dr. Bob say...]
chessguy has quit [Ping timeout: 258 seconds]
chessguy has joined #ruby-lang
wallerdev has joined #ruby-lang
mbj_ has joined #ruby-lang
mbj has quit [Ping timeout: 252 seconds]
banisterfiend has joined #ruby-lang
<banisterfiend>
.
mossplix has quit [Read error: Connection reset by peer]
solars has quit [Ping timeout: 256 seconds]
roz has joined #ruby-lang
mbj_ is now known as mbj
andersbr has quit [Quit: andersbr]
mossplix has joined #ruby-lang
krohrbaugh has joined #ruby-lang
apeiros has joined #ruby-lang
lsumner has joined #ruby-lang
krohrbaugh has quit [Ping timeout: 252 seconds]
solars has joined #ruby-lang
carloslopes has joined #ruby-lang
countdigi has joined #ruby-lang
mistym has joined #ruby-lang
mistym has joined #ruby-lang
yankeefan04 has joined #ruby-lang
yankeefan04 has quit [Remote host closed the connection]
yankeefan04 has joined #ruby-lang
chessguy has quit [Remote host closed the connection]
chessguy has joined #ruby-lang
vlad_sta_ has quit [Remote host closed the connection]
dhruvasagar has quit [Ping timeout: 246 seconds]
spuk has quit [Changing host]
spuk has joined #ruby-lang
chessguy has quit [Ping timeout: 256 seconds]
lsumner has quit [Remote host closed the connection]
bzalasky has joined #ruby-lang
chessguy has joined #ruby-lang
roz has quit [Ping timeout: 258 seconds]
banisterfiend has quit [Remote host closed the connection]
lsumner has joined #ruby-lang
maxmanders has quit [Quit: Computer has gone to sleep.]
totallymike has joined #ruby-lang
sush24 has joined #ruby-lang
<whitequark>
mbj: no I don't think that is a good idea
<whitequark>
first, tSTRING is used for most simple strings. it is an important optimization, and in the future the lexer will take advantage of it more
<whitequark>
e.g. right now, it shines at everything non-text-literal--outperforming RP at least by 2x even while collecting the maps--and sucks hard at strings
<whitequark>
being slower by 2x-4x
<mbj>
whitequark: "foo" is lexed as a single tSTRING token, just like ?c on non 1.8, without a real sign of distinction between "" and ? strings it blows up the later stages.
<whitequark>
hrm
<whitequark>
I'm against adding much new tokens to the parse.y files. the deviations from mri should be as small as possible; that allows us to verify the behavior against mri easily
<mbj>
I understand that reasoning
<mbj>
The other option is to detect the "?" as leading character in the builder
lsumner has quit [Remote host closed the connection]
security has joined #ruby-lang
megha has quit [Ping timeout: 264 seconds]
<whitequark>
this is probably the right thing to do.
<whitequark>
the strings should be unambiguous. you can have either ", ', ?, or % as the leading char
<mbj>
and in case of ? you dont have a closing delimiter
<whitequark>
yep
<whitequark>
it's a bit of logic but not that much
<mbj>
yeah
<whitequark>
you can put it to a Source::Map descendant IMO, they will generally handle various source-related quirks in themselves
<mbj>
whitequark: Within Source::Map we do not have access to the ruby version we parse (currently)
mossplix has quit [Read error: Connection reset by peer]
<mbj>
So ?c in 1.8 vs non 1.8 would not belong there.
<mbj>
forget this
<mbj>
We do not see ?c as a string literal in 1.8 mode ;)
andersbr has joined #ruby-lang
<mbj>
s/literal/token/
<whitequark>
mbj: yeah, ?c is an (int) mapped to the "?c" token.
<whitequark>
there could be a method in the Source::Map::Integer to query this info.
gix has joined #ruby-lang
<mbj>
whitequark: Lets see where this ends. Once we have the literal source maps the process will speed up
<mbj>
whitequark: Still dont have much time, so I do not advertise anything ;)
redgetan has joined #ruby-lang
<whitequark>
ok
lsumner has joined #ruby-lang
lsegal has joined #ruby-lang
mwjcomputing has joined #ruby-lang
mytrile has joined #ruby-lang
maxmanders has joined #ruby-lang
bzalasky has quit [Remote host closed the connection]
krohrbaugh has joined #ruby-lang
lsumner has quit [Remote host closed the connection]
krohrbaugh has quit [Ping timeout: 245 seconds]
sepp2k has joined #ruby-lang
apeiros has quit [Remote host closed the connection]
canton7 has joined #ruby-lang
chessguy has quit [Remote host closed the connection]
lsumner has joined #ruby-lang
jg has quit [Ping timeout: 246 seconds]
mytrile has quit [Remote host closed the connection]
<judofyr>
whitequark: I have a slight problem now though: there's places where I want to trim newlines, but I want the AST to still contain the correct line-numbers
<whitequark>
if you'd write it in ragel that would be trivial :)
<judofyr>
well, I'm mostly wondering how the communication with the parser/AST would be
<whitequark>
ummm
<whitequark>
don't think about lexing/parsing in terms of "communication"
<whitequark>
they're not friends talking to each other, they're mortal enemies trying to achieve a armistice
<whitequark>
well I overdid that metaphor maybe
<judofyr>
:D
<whitequark>
but: lexer produces a token stream, parser consumes it
<whitequark>
the parser does not, ever, ask the lexer "what is the current lineno"
<judofyr>
whitequark: let me phrase it another way: the generated Ruby-code has to line up with the template source. how would you store that line information the AST?
<whitequark>
or otherwise you'll get the ruby_parser's "line numbers are slightly off, parsing LR sucks" (no, you do.)
<whitequark>
judofyr: :3
<judofyr>
I don't really care about "this is lineno x", I care about "I need to insert a newline in the generated Ruby here"
Banistergalaxy has quit [Ping timeout: 258 seconds]
<judofyr>
I've had explicit s(:newline)-nodes earlier
chessguy has quit [Remote host closed the connection]
<whitequark>
well
<whitequark>
look how parser does it :D
<whitequark>
add an attr_reader :line to your AST::Node subclass
mytrile has joined #ruby-lang
<whitequark>
add a field to all your tokens with the line number for token
<whitequark>
and yes, that means that you need to split the tokens on newlines and be able to concatenate them back in parser
apeiros has quit [Ping timeout: 256 seconds]
<whitequark>
that's trivial to do in the grammar, slightly less trivial to represent in the AST
<judofyr>
that would make the code generator more complex though
<whitequark>
yep
<judofyr>
as I would have to keep track of the previous and current Node, and insert \n as needed
<judofyr>
and sometime I would also want to combine and split nodes
<whitequark>
splitting is simple
<whitequark>
combining, not so
<whitequark>
easiest way is to do 1:n src:ruby correspondence
<judofyr>
e.g `Hello <%= world %>!` can be combined to s(:dynamic, 'Hello #{world}!')
<whitequark>
well. don't do that :)
<judofyr>
whitequark: it's a neat optimization
<whitequark>
it's not an abstract syntax tree anymore in this case, for example
<whitequark>
and it's not neat, it breaks things for you
* whitequark
needs to fix the last parser bug which prevents it from handling 1.8 & 1.9 stdlibs
<judofyr>
I agree that it's not really an AST anymore, but I don't think it's premature optimization
<whitequark>
judofyr: do you have benchmarks?
<whitequark>
if not, it is
<whitequark>
besides
<whitequark>
parsing in a templating engine is a one-time task
<whitequark>
if you are not writing php, that is
<whitequark>
so it doesn't even matter as long as it only spans seconds and not minutes
chessguy has joined #ruby-lang
<judofyr>
whitequark: oh, I mean this as an optimization step for improving the generated code. e.g. `_buf << "Hello #{world}!"` over `_buf << 'Hello '; _buf << world; _buf << '!'`
<whitequark>
ah...
<whitequark>
it's even simpler then, make this optimization in your backend / code generator
<whitequark>
eg translate every (dynamic) node to _buf << "single interpolated string"
<judofyr>
sure, but it's also neat to have the optimizer separate because you can have different backends
<judofyr>
e.g. buffering into an array and then joining vs buffering into a string
<whitequark>
AST is a really wrong and inconvenient place for doing any optimizations or transformations. I dunno why is this idea so persistent
<whitequark>
judofyr: no
<whitequark>
optimizer is intrinsically coupled to the backend
<whitequark>
besides, it doesn't make sense to have distinct backends for just ruby because it's just overengineering and/or masturbation
<whitequark>
pick the fastest one and go with it
maxmanders has joined #ruby-lang
<whitequark>
the problem domain is too simple to require this form of variability
<judofyr>
whitequark: having custom backends is useful if you e.g. want to support streaming. you could have a backend that just pushes to a buffer
<whitequark>
judofyr: class StreamBuffer; def <<(str); @socket.send str; end; end
krohrbaugh has quit [Quit: Leaving.]
<whitequark>
class ArrayBuffer; def <<(str); @array << str; end; def get; @array.join; end; end
<whitequark>
etc
brianpWins has joined #ruby-lang
<judofyr>
whitequark: I do think I actually agree with you :)
<judofyr>
whitequark: but still, when you're just going to produce Ruby in the end, an AST is a surprisingly nice format to work with
KM has joined #ruby-lang
pkrnj has joined #ruby-lang
<whitequark>
the first thing you understand when you begin with compilers... they're already complicated enough. you do not need MORE code there.
<judofyr>
what would be a "better" solution?
<whitequark>
*to work with
<whitequark>
hm
<whitequark>
it depends on the problem domain :)
KM is now known as Guest88304
<whitequark>
so far, I do not see what would a backend-independent optimization step constitute
<judofyr>
that I can agree
Gaelan has quit [Remote host closed the connection]
Gaelan has joined #ruby-lang
<judofyr>
but I still find s(:newline)-nodes and AST a surprisingly nice way to work with things
<whitequark>
well, it doesn't exactly work for you, does it? :)
<whitequark>
(if yes, why would you ask? :))
<judofyr>
it's been working fine enough when I've been writing custom lexer/parsers
<judofyr>
(hey, it works surprisingly fine for template languages)
<judofyr>
but when you have the whole "lexer produces tokens", "parser consumes them", it isn't as easy to just say `res << s(:newline)`
<judofyr>
I could incorporate it in the grammar, but it feels a bit dirty
nazty has quit [Read error: Connection reset by peer]
as has joined #ruby-lang
Gaelan_ has joined #ruby-lang
<yxhuvud>
whitequark: isn't the whole concept of transforming something to an AST more or less *by itself* a backend independent optimization step?
<whitequark>
yxhuvud: wat?
<yxhuvud>
(it is quite possible to parse without them, though quite horrible)
<whitequark>
optimization is something that makes your target program faster
<whitequark>
(or makes it consume less space, etc)
<judofyr>
yxhuvud: AST construction is not really about performance; it's about making it easier to work with
<yxhuvud>
judofyr: yes, which is also an optimization.
as has quit [Remote host closed the connection]
<whitequark>
yxhuvud: no, it isn't.
<judofyr>
yxhuvud: not the type of optimization we were talking about
Gaelan has quit [Ping timeout: 246 seconds]
as has joined #ruby-lang
<whitequark>
first, do not confuse optimizing compilers, which transform the programs being compiled, with optimization of the compiler itself
<whitequark>
second, conceptual simplification of something is not an optimization, by any means
nmeum has quit [Quit: WeeChat 0.4.1-dev]
nmeum has joined #ruby-lang
<judofyr>
whitequark: I guess if I'm going down the lexer/parser/AST road I might as well tag line numbers to nodes
<whitequark>
judofyr: well, yes? this is how it is generally done
rolfb has joined #ruby-lang
<judofyr>
whitequark: I've noticed
as has quit [Remote host closed the connection]
tommylommykins has joined #ruby-lang
<judofyr>
whitequark: you ever dabbled in HTML/JS? I'm actually quite proud of the number of ugly hacks that works great together in Mockle. e.g. look at the source code of http://stuff.judofyr.net/mockle-example/ and compare it to the DOM nodes in the Inspector
as has joined #ruby-lang
j3r1ch0 has quit [Ping timeout: 246 seconds]
mbj has quit [Ping timeout: 276 seconds]
<whitequark>
neat
<judofyr>
I'm using <noframes> :D
<whitequark>
also cool error reporting
<judofyr>
yeah, I'm trying to make it as designer-friendly as possible
<whitequark>
I wish our internal editor was as good >_<
<whitequark>
I have developed a templating language, you see
<whitequark>
it's like liquid but it was actually *designed* as opposed to banging regexps together
<whitequark>
also several hundred or thousand times or so faster
<judofyr>
oh, Liquid is so terrible
as has joined #ruby-lang
<judofyr>
(implementation-wise)
<judofyr>
I think I have 2-3 re-implementations
<whitequark>
yeah. my one is compiled to ruby and also supports precise error reporting
<whitequark>
just like parser :3
rippa has joined #ruby-lang
as has quit [Remote host closed the connection]
andersbr has quit [Quit: andersbr]
<whitequark>
both compile-time & run-time
<judofyr>
I think I called mine Solid :)
<judofyr>
Sat Feb 20 17:53:00 2010 +0100
meise has quit [Ping timeout: 245 seconds]
<whitequark>
but the guy who should've integrated all of this with an editor on our website (it's a cms for musicians, http://amplifr.com), well, didn't do that
<whitequark>
and now it totally sucks :/
meise has joined #ruby-lang
as has joined #ruby-lang
<whitequark>
there's also an internal reporting app for liquor errors these days. I think
sush24 has quit [Quit: This computer has gone to sleep]
<judofyr>
whitequark: mockle.rb will have crappy compile-time error reporting because you should just be able to open it in the browser and see it there
<judofyr>
so much nicer to show errors in a browser than in a terminal
<judofyr>
coloring, positioning, etc
sepp2k has quit [Quit: Leaving.]
krohrbaugh has joined #ruby-lang
as has quit [Remote host closed the connection]
<whitequark>
dunno
as has joined #ruby-lang
* whitequark
has all that in the terminal
<whitequark>
it's about ten loc
mossplix has joined #ruby-lang
tylersmith has joined #ruby-lang
apeiros has joined #ruby-lang
as has quit [Remote host closed the connection]
andersbr has joined #ruby-lang
as has joined #ruby-lang
aderyabin has quit [Ping timeout: 258 seconds]
tylersmith has quit [Client Quit]
<judofyr>
whitequark: as in, more user-friendly. you can show the whole source, jump to the line. do proper syntax highlighting. anyway.
<judofyr>
night!
judofyr has quit [Remote host closed the connection]
brianpWins has quit [Quit: brianpWins]
thone_ has joined #ruby-lang
yankeefa_ has quit [Remote host closed the connection]
brianpWins has joined #ruby-lang
thone has quit [Ping timeout: 255 seconds]
mbj has joined #ruby-lang
mytrile has quit [Remote host closed the connection]
anachronistic has joined #ruby-lang
sepp2k has joined #ruby-lang
chessguy has quit [Remote host closed the connection]