<SeanTAllen>
Shayanjm: I assume you ended up here because of my post last went up today. If yes, and you found it interesting, what other things about Wallaroo would you be interested in learning about (we are looking to narrow down/augment our blog post ideas)
dipin has quit [Quit: dipin]
dipin has joined #wallaroo
dipin has quit [Client Quit]
vaninwagen has joined #wallaroo
vaninwagen_ has joined #wallaroo
vaninwagen has quit [Ping timeout: 248 seconds]
bitcrusher has quit [Read error: Connection reset by peer]
dipin has joined #wallaroo
_whitelogger has joined #wallaroo
vaninwagen_ is now known as vaninwagen
dipin has quit [Quit: dipin]
dipin has joined #wallaroo
_whitelogger has joined #wallaroo
_whitelogger has quit [Ping timeout: 258 seconds]
_whitelogger_ has joined #wallaroo
<Shayanjm>
SeanTAllen: the latest blog post definitely put it front-of-mind, but I heard about you guys when you first released your demo
<Shayanjm>
Generally speaking, I'm very interested in the consistency guarantees provided by your system
<Shayanjm>
so would love any literature/deep dives on that sort of stuff
<SeanTAllen>
Shayanjm: we are still working on those. couple of open projects. will be writing up eventually.
<Shayanjm>
Awesome! Looking forward to it
<SeanTAllen>
things we need to do: we can't handle multiple failures at one time. that is slated for work starting in november.
<Shayanjm>
Right
<SeanTAllen>
and we dont support "exactly once" at source or sink yet
<SeanTAllen>
that is coming in november/december as we finish off kafka support
<SeanTAllen>
first we are getting our kafka client to handle things like leadership changes and what not.
<SeanTAllen>
i'm interested in what about consistency guarantees you are interested in knowing
<SeanTAllen>
is it more
<SeanTAllen>
high level guarantees or the "how we implemented that, what are the edge cases, catches etc?"
<Shayanjm>
definitely the latter. Generally curious about your mentality when building this. What tradeoffs were considered, what edge cases exist, and how many gotchas are there?
<Shayanjm>
re: exactly once, in my experience exactly-once delivery is "nice to have" - but at least once + local caching is usually good enough re: ordering (not strict by any means) and optimizes primarily for throughput
<Shayanjm>
though i'd imagine ordering guarantees may be important in your case
<SeanTAllen>
we talk about "exactly once message processing"
<SeanTAllen>
so we do at least once delivery w a deduplication strategy
<Shayanjm>
yeah that's a good distinction
<Shayanjm>
delivery vs. processing
<SeanTAllen>
we guarantee between any two points that dont involve a split that messages arrive in the same order they were received
<SeanTAllen>
the problem with exactly-once conversations that i often see is that no one defines their terms and end up arguing because they are talking about different things
<Shayanjm>
so that's super interesting
<Shayanjm>
how do you offer that guarantee? like what are the mechanisms that provide that?
<SeanTAllen>
that's probably most internet conversations
<SeanTAllen>
Pony guarantees message ordering between actors
<SeanTAllen>
so "two points" is actor to actor in our system
<SeanTAllen>
and right now we are relying on TCP to do the proper thing with network delivery
<SeanTAllen>
which in theory it does, and almost always in practice as well
<Shayanjm>
ah gotcha
<SeanTAllen>
the pony term for it is "causal messaging"
<Shayanjm>
very interesting. I guess I haven't actually poked at any of the code - are these "two actors" defined in the same software context? are they not just two replicas of the same application communicating over some RPC?
<Shayanjm>
i.e: Pony is aware that it is communicating with an actor, whether it's sending some payload or receiving?
<Shayanjm>
I guess I could probably trivially answer this question by looking at the code lol
<SeanTAllen>
Pony doesnt have distributed actors yet
<SeanTAllen>
so all actors are in the same process
<SeanTAllen>
a message send involves sticking something on the receiving actors queue
<SeanTAllen>
and actors have to read their queue and process it in order
<SeanTAllen>
that is where the causal messaging comes wrong
<SeanTAllen>
we added worker to worker communication over TCP to Wallaroo
<SeanTAllen>
and maintain the same ordering guarantees between the TCP end points
<SeanTAllen>
hope that makes sense
<SeanTAllen>
"two actors" was literally two pony actors
<SeanTAllen>
pony being an actor based language ala erlang
<Shayanjm>
gotcha, makes perfect sense
<Shayanjm>
all very cool stuff! You guys are based in NY, right?
<slfritchie>
Yes, though a few of us are elsewhere in the US, Canada, or Europe.