<SeanTAllen>
nemosupremo: can you share your python source? there's probably some error in the source we arent currently handling and printing out a nice message for. we had a couple of those in the past related to Python objects being different than what was expected. We probably missed a case where that was possible.
<SeanTAllen>
But yes nemosupremo that would be #960 that you are probably running into
<nemosupremo>
I get to '|v|v|v|Initializing Local Topology|v|v|v|' before it segfaults
<nemosupremo>
@SeanTAllen there are 2 imports you can comment out (from x import y) that you can remove
<nemosupremo>
ok, I exited the docker container, then started it again and it worked
<SeanTAllen>
that's really weird
<SeanTAllen>
yeah it works for me over here fine.
<SeanTAllen>
hmmm...
<SeanTAllen>
i wonder what could have happened
<SeanTAllen>
so you restarted the docker container and it worked... hmmm... ok i'm going to be thinking on that. and try to figure out something i can do to try and recreate.
<nemosupremo>
weird, ever since I restarted the docker container, I haven't been able to reproduce the issue
<SeanTAllen>
some days, computers are not my favorite things
<nemosupremo>
I must have messed up the environment somehow, sorry
<SeanTAllen>
i make no such assumption. but something happened. im bummed we cant reproduce.
<nemosupremo>
One thing I noticed is if I ctrl+c the machida application, and restart it, I get stuck on 'Need ClusterInitializer to inform that topology is ready'
<nemosupremo>
is that the wrong way to stop the application?
<SeanTAllen>
that should work, we found a bug in the linux signal handling that we setup so that it wasn't working correctly and doing a clean shutdown. that is fixed on master. will be fixed in the next release.
<SeanTAllen>
it works on OSX, not Linux at the moment.
<SeanTAllen>
there's a cluster shutdown tool you can use to shut it down cleanly or...
<SeanTAllen>
all the files it would remove on clean shutdown can be manually removed in /tmp
<nemosupremo>
If its broken on Linux, that might mean its broken inside docker?
<SeanTAllen>
ya
<SeanTAllen>
it would be an issue in docker
<SeanTAllen>
for my application called "a" for application module, this is the files it removes on shutdown...
<SeanTAllen>
Here's a question we've been discussing internally nemosupremo...
<SeanTAllen>
let's say you have a cluster of 3 wallaroo workers running and you ctrl-c one of them... should only that one shut down or should it do a clean shutdown of all 3? currently it will do all 3.
<nemosupremo>
It should only shut down that worker, but depending on how wallaroo is architectured, I don't know if thats the right answer. Another thing - I don't think I'm giving a good answer either. When starting out, one thing that wasn't immediately clear to me was "what is a wallaroo cluster". Right now, when I ctrl+c I can't tell if I'm shutting down a worker node, or if I'm shutting down a worker process? Is wallaroo multi tenant?
<nemosupremo>
Other "cluster services" have "masters" and "nodes" and its pretty clear what doing a ctrl+c, or a kill -9, or a terminate will do
<SeanTAllen>
each worker runs a single application, perhaps in conjunction with other workers, so it is not multi-tenant in the way most folks would think of it
<nemosupremo>
I may also be very confused, because wallaroo seems too easy to use
<SeanTAllen>
worker node/process would be the same thing in wallaroo i think. what is the difference in your mind between a worker node and a worker process?
<SeanTAllen>
the basic idea is there is no master, except when you have cluster join/leave events and then 1 node makes itself the "master" for work redistribution.
<SeanTAllen>
but the rest of the time, there's no master, just a series of workers that are working together in a cluster.
<nemosupremo>
a worker node = the actual process wallaroo is running. a worker process = a thread that is running my application.
<nemosupremo>
that may be confusing, because I was confused about how wallaroo is architected
<nemosupremo>
Sometimes I want to ctrl+c my process (I have a bug in my application), sometimes I want to ctrl+c my node (I don't need 3 servers)
<nemosupremo>
wallaroo seems very magical in that I can just spin up a process and it will join my cluster and start distributing work to it!
<SeanTAllen>
there's no distinction like that in wallaroo... there are workers
<SeanTAllen>
workers are unix processes
<SeanTAllen>
the work together
<SeanTAllen>
you can't shutdown a given thread within a worker
<SeanTAllen>
so when you say "worker process" i think of process as "unix process"
<nemosupremo>
Right, I think I understand now, I came in with a very Storm/Spark oriented model where you have your "storm nodes" and developers write sinks/spouts
<SeanTAllen>
for the "i dont need 3 servers", in wallaroo that would be, i have 3 workers, i only want two. there's a tool for that, it allows you to shrink the cluster size.
<SeanTAllen>
im quite familiar with storm, happy to try and talk in terms of its terminology if it helps
<nemosupremo>
Ok, now that I understand it a bit more I think ctrl+c should kill the entire thing
<nemosupremo>
in that case I would try to own the fact that the developer doesn't care about how many resources wallaroo is running
<nemosupremo>
and if it runs on 3 or 9 nodes is an operator detail
<nemosupremo>
(and he has tools to shrink/grow applications)
<SeanTAllen>
im going to disappear on you soon. wife is home and i will be finished cooking dinner soon.
<nemosupremo>
no problem
<SeanTAllen>
slfritchie: tends to be more of a night owl. he might be around.
<nemosupremo>
just want to say that seems very magical. I used storm in pre-1.0, and adding cluster capacity was almost always something you had to think about
<SeanTAllen>
a couple of us ran storm in production for several years and that experience has informed some of the Wallaroo experience
<SeanTAllen>
unfortunately, it came out a couple months before Twitter announced heron. That pretty much killed the book sales.
<SeanTAllen>
ok dinner time for me. ill check back in the morning to see if you have more questions. if you log off and log back on, check the irc logs for responses.
<SeanTAllen>
the Running Wallaroo section of the book as more information https://docs.wallaroolabs.com/book/running-wallaroo/running-wallaroo.html. Hopefully it can provide some answers. And fire away with questions you have. It helps us figure out what we need to improve in the documentation.
<nemosupremo>
With the Kafka Source - it doesn't look like there is any way to get the partition key of an incoming message
nemosupremo has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
<SeanTAllen>
That’s an interesting comment. What would you use the partition key for nemosupremo?
nemosupremo has joined #wallaroo
<nemosupremo>
@SeanTAllen 1.) In this specific application I'm working on, since it uses the RandomKafka Partitioner, it actually stores data in the key that is not stored in the message. Implemention detail. 2.) I might be mistaken on partitioning works in wallaroo, but to me it seems like in a lot of cases it would make sense to partition off of the kafka key (its something I've done often in home grown frameworks)
<SeanTAllen>
that's something we could possibly do, but, how to go about it could be a little tricky. the "simplest" thing would be for every computation etc to take a message and message-metadata object. which could be nothing.
<SeanTAllen>
or the kafka decoder could combine message and metadata together (but that involves a decent amount of copying and could be slow).
<SeanTAllen>
is there other metadata from kafka you can see using?
<SeanTAllen>
hmmm i suppose in a decoder that you could have a message that takes both payload and headers, no copying there but there's extra allocation for the wrapper object, still, that seems doable.
<SeanTAllen>
and given not every source might have metadata, that probably makes more sense
<SeanTAllen>
definitely something we will be discussing internally.
nemosupremo has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
nemosupremo has joined #wallaroo
nemosupremo has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
<nemosupremo>
more specifically, line 47 looks like a bug
<nemosupremo>
if self.letter != votes.letter, then the votes get mixed
<nemosupremo>
or, if wallaroo is supposed to ensure that never happens, line 47 looks like a foot gun
<aturley>
nemosupero the total votes object is guaranteed to always be on the same letter
<aturley>
when TotalVotes objects are created, though, there is no way to tell them which letter that will be.
<aturley>
ideally self.letter would only be set the first time update was called.
<aturley>
i seem to recall a few of us had a discussion around what to do in this example, and we decided that we wanted to avoid getting bogged down in the mechanism for only setting the letter on the first call to update(...)
<aturley>
^^ nemosupremo (sorry, i typed your username wrong up at the top)
<nemosupremo>
Can you use `to_state_partition` if you don't know how many partitions you will have beforehand?
<slfritchie>
Currently, Wallaroo's state partition scheme needs to know how many in advance. It's a restriction we wish to loosen. I'm not aware of when that might happen, though.
<nemosupremo>
@slfritchie how are people partitioning in the real world today?
<nemosupremo>
just select an arbitrary number and bucket into those?
<nisanharamati>
predefined keys/buckets
<nisanharamati>
So the current approach for unbounded/unknown keys it to treat it like database sharding
<nisanharamati>
where you partition on something known (like the first letter of the word in the word_count example)
<nisanharamati>
and then use an unbounded hashmap to store the data inside each shard
<SeanTAllen>
nemosupremo: we are planning on adding "dynamic partition keys" within the next few months. that would allow you to not know the number of partitions ahead of time and could then have a state object per word in an example like the word count one or say, a key per client in another system etc.