15:03 <cajually> is Alan Mosca here? or anyone else that participated in the implementation of the current decorator based python api

15:05 <SeanTAllen> Alan is no longer working on Wallaroo. Nisan (who isnt on IRC at the moment, but should be in an hour or so) and aturley are most likely to have answers to questions. Is there something in particular you were looking to know cajually ?

15:10 <cajually> Just some details in how the current way we are createing pickle:able decorated functions, it appears to work by accident in python2 based on the error messages I'm getting

15:11 <cajually> I think I've found a way forward, just checking if there is someone with a lot more pickle experience than myself as it is a quite awkward module to work with

15:11 <SeanTAllen> I think Nisan is the best person to talk to on that. I remember that was a particularly sticky area and if I remember correctly, it was Nisan who helped address. I'll drop him a note to swing by when he starts (he's on west coast of North America).

15:12 <cajually> But it raises a larger question, why do we rely on pickle, is it because of ease of serialization?

15:13 <SeanTAllen> We wanted to allow for a way for people to get started without having to worry about serialization. You can override it if you want, but its a good place to start.

15:14 <cajually> It has imho a larger than desired capability to serialize python objects, I'm not sure that allowing a creep between function and state is best

15:14 <cajually> but I've stepped between productivity and purity before..

15:15 <SeanTAllen> This example overriding the serialization from pickle to a binary form that performs better: https://github.com/WallarooLabs/wallaroo/blob/master/testing/performance/apps/python/market_spread/market_spread.py#L64

15:15 <SeanTAllen> "it" being Pickle cajually ?

15:16 <cajually> yes exactly

15:18 <SeanTAllen> We have considered having pickle as the default to be important to getting people using Wallaroo easily and then, issues around serializing differently can be raised. Serialization being a bit of an advanced concept that be a lot of cognitive overhead for folks starting out learning Wallaroo if we didn't provide a default.

15:18 <SeanTAllen> That has been our thinking.

15:22 <cajually> that is fair, sorry my hotel wifi is almost nonexistent

15:23 <cajually> I think that having a good interface is very important and I realize a middle way where strictly state would be stored is never transparent, pickling can be tho

15:26 <SeanTAllen> There are definitely tradeoffs. Serialization/deserialization is something we plan on revisting at some point, right now it isn't a top priority, but we have had a few discussions about things we could do in the future.

15:26 <nisanharamati> Morning all

15:26 <SeanTAllen> howdy nisanharamati

15:27 <nisanharamati> @cajually re pickling and new-style classes, there's an issue on how to support them, which is one possible way to go about it: https://github.com/WallarooLabs/wallaroo/issues/1938

15:27 <nisanharamati> Can also address lambdas at the same time (name them globally based on the hash of the `__code__` object or something like that)

15:28 <cajually> I'll be back in 15min~

15:50 <cajually> I mostly wanted to know if there was a strong opinion for pickle

15:51 <cajually> nisanharamati: that is very related to an issue I'm having

15:51 <nisanharamati> I'm glad it helps!

15:51 <cajually> the local functools.wrapped old style classes can't be pickled in python 3

15:52 <cajually> because the local classes can't be discovered and apparently this is explicitly not supported by pickle

15:54 <cajually> I've been trying to change this into a decorator class builder builder pattern but that has the issue that the `def`:ed object is not that same as the decorated and breaks, because I can't use wraps

15:55 <cajually> I'm sure there is some combination of these tools that will work and the best way to solve it is going to involve understanding pickle better

15:55 <cajually> eh only one "decorator class builder pattern"

15:56 <nisanharamati> For lambdas: https://github.com/WallarooLabs/wallaroo/issues/1921

15:57 <cajually> I can imagine that would be as impossible currently

15:57 <nisanharamati> yeah

15:58 <nisanharamati> but once you start doing name mangling an reassignment to the global scope like the POC in the code snippet in #1938, a lot of things become possible

15:59 <cajually> I think there is a path forward here for sure

15:59 <nisanharamati> e.g. in this instance, once you called the application builder with the lambda, the app builder identifies it as a lambda, recreates it as a regular function, and assigns it to the global scope, which then makes it pickleable

15:59 <cajually> yeah that i definitely doable

16:00 <cajually> we could grab the .name or .__name__, add something to make it unique and jam it in globals

16:01 <cajually> pickle has weird restrictions for being stdlib

16:01 <nisanharamati> it's also indeterminate in terms of how long pickling can take

16:02 <nisanharamati> which is not ideal for a low-latency needs

16:02 <cajually> anyway I like the idea of adding the __serialize__// __dese.. pseudo interface

16:02 <cajually> check if those attrs exists and use them then otherwise use pickle format 3 o

16:05 <nisanharamati> I've got a bit of time before my next meeting, would you like me to whip up a branch with those changes applied to `wallaroo.py`, @cajually ?

16:05 <nisanharamati> you can then rebase or cherry pick that into your work branch

16:06 <cajually> I'm not yet familiar with what the process for such a change should look like

16:07 <cajually> My branch so far is an exploration in what needs to be done for python2 and python3 support in machida, I think my machida supports python3 only now and the issues are entirely in pickling

16:08 <cajually> what would be the best way to for you to spend your time on the python stuff I can't say. Do we expect anyone to be angry if we break the current API for example?

16:08 <SeanTAllen> cajually: break in what sort of way? having python2 and python3 as different languages and APIs is something we expected to happen.

16:09 <cajually> but it would be nice if wallaroo.py did the same thing as before once you ported your compute functions

16:09 <nisanharamati> I think other than the pickling bit, our API is compatible with python 3

16:10 <cajually> I get no other unit test failures

16:10 <nisanharamati> Supporting new-style class pickling is beneficial for python2 as well

16:11 <cajually> I agree

16:11 <nisanharamati> it gives users the ability to base their state objects off of existing classes for example

16:11 <cajually> Which I feel is the main reason to suport pickling

16:11 <cajually> over say your state has to be a dict of primitive

16:12 <cajually> nisanharamati: have you looked into using update_wrapper directly?

16:13 <cajually> I was thinking that maybe we can avoid a closure by using a class also

16:14 <cajually> as the decorator, but I guess we will still have one

16:15 <cajually> But the best solution is probably the global solution we talked about earlier, it will definitely work

16:16 <nisanharamati> The code in #1938 is loosely based on update_wrapper

16:16 <cajually> Unless there is more magic in pickle that I don't know

16:16 <cajually> ah of course

16:16 <nisanharamati> update_wrapper lets you pickle

16:16 <nisanharamati> but to unpickle, you need it in the global scope or you can't find it

16:16 <cajually> it has to the it

16:17 <nisanharamati> Once it's in global scope, you can also make some optimizations like using a local lookup table for functions instead of unpickling them

16:18 <cajually> gah, my connection, I'm going to give up for tonight. I think that is solution is worth working on for sure

16:22 <cajually> If you don't manage to get your code up in abranch that's ok, I can probably get enough of the snippoets working in my branch to get pickling working and tests passing tomorrow. Just gotta find a coworking space first:)

16:25 <cajually> great talking tonight/mornig and see you later

23:48 <nisanharamati> @cajually the updated API in `wallaroo.py` is now in `master`. Try rebasing your work off of the latest `master` code and see if it works!

23:51 <nisanharamati> One thing to keep in mind: there's a lot of python2 glue code in the integration test harness. That glue does not need updating to Python3 as part of the project, as it's running machida, Go, and Pony based wallaroo application in subprocesses.

23:51 <nisanharamati> Specifically, anything in `testing/tools/integration/`, and `testing/correctness/tests/` can be left alone for now Python2.

23:51 <nisanharamati> *as python2

23:52 <nisanharamati> I'm out for the rest of the day, but I'll check back here tomorrow morning and see how things are going.