21:35 <nemosupremo> Does the giles receiver work in 0.4.0?

21:36 <nemosupremo> The encoder is being executed, but received.txt stays at 0 bytes

21:47 <nisanharamati> Yes, but it depends on the output data being formatted correctly

21:48 <nisanharamati> you can try replacing it with `nc` to see whether the output is (a) arriving and (b) formatted as expected

21:48 <nemosupremo> makes sense, I see the format in the docs

21:48 <slfritchie> Our release test process has moved away from using giles/receiver/receiver everywhere but AFAIK it works. If your encoder isn't spitting out 4 byte length + binary payload of the message, then giles receiver will interpret the first 4 bytes as a length, calculate a wrong number of bytes to read, and then wait a long time for more data.

21:48 <nemosupremo> I'm not formatting it at all

22:05 <nemosupremo> I'm trying to understand how data "flows" in wallaroo in order to optimize my database lookups. I'm using a kafka sink and this is what I see - 1000 (or a batch) of messages get "decoded", then my messages get "stuck" in a state computation step. What I would like to do as many db look ups as I can in parallel

22:06 <nemosupremo> nevermind

22:06 <nemosupremo> as I was explaining it to myself I figured it out

22:26 <nisanharamati> nice. What was it? (And consequently, is there anything we could improve in the docs or examples to help new users avoid that issue?)

22:48 <nemosupremo> @nisanharamati I think it only applies to me, but using a combination of partitioning with compute_multi I can get "batches" that I can work with at once

22:48 <nemosupremo> One thing I noticed though is that compute_multi doesn't drop null elements

22:49 <nemosupremo> (compute will drop a message if I return None)

22:49 <nisanharamati> So that `[1, 2, 3, None, 5, 6]` would be received as the sequence `[1, 2, 3, 5, 6]` at the next step, instead of the original?

22:50 <nemosupremo> As I understand it if I return `[1,2,3,None,5,6]` from compute_multi

22:51 <nemosupremo> my next compute steps should receive `1`, `2`, `3`,`5`,`6`

22:51 <nemosupremo> but I get `3`,`None`,`5`

22:53 <nisanharamati> I think that's a bug

22:53 <nisanharamati> I'll open an issue to look into it.

22:53 <nisanharamati> Thanks for pointing it out!

22:54 <nemosupremo> ok, to a bit clearer, this was the output of a computation_multi, that went into state_computation_multi, and in the partition function I was seeing the null

22:56 <nisanharamati> Thanks

22:56 <aturley> nemosupremo you *only* see `3`, `None`, `5` at the next computation?

22:57 <nemosupremo> what do you mean only?

22:57 <aturley> or are you saying that you were surprised to see the `None` along with the other values?

22:57 <aturley> do you see 1, 2, 3, None, 5, 6?

22:57 <aturley> or do you see 3, None, 5?

22:57 <nemosupremo> I was surprised to see the None

22:57 <nemosupremo> I was lazy in typing 1,2,...

22:57 <aturley> ok.

22:58 <aturley> that's expected behavior. perhaps the documentation is unclear in this case.

23:08 <aturley> let me rephrase that. the behavior that you've described is consistent with how i understood the code.

23:08 <aturley> that said, i can see how it could be unexpected.

23:13 <nemosupremo> aturley: https://docs.wallaroolabs.com/book/python/api.html#computation

23:13 <nemosupremo> "If the incoming data isn't an integer, we filter (drop) the message by returning None."

23:13 <nemosupremo> It's a bit surprising you can drop a message from a compute step by returning None

23:13 <nemosupremo> but that isn't true when using compute_multi

23:13 <nemosupremo> The developer could just filter out his own null's

23:14 <nemosupremo> so I don't think this is a stop the world issue

23:16 <aturley> the intention was that the user could return `None` to indicate that, for a given incoming message, there would be no outgoing messages.

23:23 <aturley> nemosupremo i think your line of reasoning makes sense. i'm not sure all users would expect that behavior, but i'm beginning to think that most users wouldn't be surprised if they found out it worked the way you expected.

23:24 <nemosupremo> Yep - in my case it was converting a compute step to a compute_multi step

23:24 <nemosupremo> any common possible reason, of the top of anyone's head I would see just "Killed" as the output after running machida?

23:26 <nemosupremo> is to_state_partition -> to_state_partition legal in wallaroo?

23:28 <nisanharamati> should be legal

23:29 <nisanharamati> scratch that, _is_ legal.

23:29 <nisanharamati> I think I've only seen `Killed` from machida if I had `exit(0)` in my python code

23:30 <nemosupremo> I removed the second to_state_partition, and now I get a segmentation fault after the first to_state_partition completes on one partition

23:34 <nisanharamati> Is the second partition using the same function or same set of keys? (e.g. same list object)

23:35 <nemosupremo> err

23:35 <nisanharamati> sounds like a different error than I'm imagining

23:35 <nemosupremo> by partition, I mean I see the compute function exit once and print out data

23:35 <nemosupremo> *execute once

23:35 <nemosupremo> and then seg faults after I return

23:36 <nisanharamati> so it's: `decode -> stateful_partition.partition(data) -> stateful_partition.compute(data) -> segfault` ?

23:36 <nemosupremo> yep

23:37 <nemosupremo> I've had a couple segfaults already but I could see where it was happening in "my" code, and it was usually just a python error or something that didn't get its stack traced

23:37 <nemosupremo> but these feels like the segfault is happening in the framework

23:38 <nemosupremo> my function is essentially `def join(d,s): s.lookup(d), return d`

23:38 <nemosupremo> I can print just before the return

23:40 <nemosupremo> if my stateful_partition.compute does nothing, I get a segfault, so the problem must be in the previous step somehow...

23:41 <nisanharamati> Could be either in the framework barfing over `d`, or in `s`s garbage collection, `__del__` method or `atexit` hook, if it gets collected

23:41 <nisanharamati> if you switched the partitioned step to non-partitioned, does it still segfault?

23:42 <nemosupremo> yes it does

23:42 <nemosupremo> Feels like I'm programming in C 😂

23:44 <nisanharamati> would you mind sharing a gist of your app and some sample input data, if it's not sensitive data?

23:44 <nisanharamati> I'm running up against the wall of my imagination of where it could be going wrong

23:45 <nemosupremo> 'm trying to figure out how to send it to you because the data comes from a live kafka source

23:46 <nisanharamati> the fields in a single decoded message would be fine. I can send it in from there

23:46 <nisanharamati> you could `print repr(bs)` in the decoder

23:47 <nemosupremo> I just printed out some msgs and I can't send you the message content, although I think the actual messages may be irrelevant

23:52 <nemosupremo> okay

23:54 <nemosupremo> https://gist.github.com/nemosupremo/8d7d7ca00615031802f11cae5322b470

23:54 <nemosupremo> This causes a segfault after about ~800 messages

23:54 <nemosupremo> message content does not matter

23:55 <nemosupremo> to_state_partition references a DeviceEventsWrapper, you can use.a Votes state object as I don't touch the state ither

23:59 <nemosupremo> FUCK

23:59 <nemosupremo> nvm