<UrsusArcanus>
Hello Pony fans. After flirting with Pony for a couple of years now I am finally committing myself to actually learning it.
<UrsusArcanus>
was curious if anyone is working with chacha20 + poly1305 or other crypto libs in Pony land
<UrsusArcanus>
or good ol' salsa20 for that matter
<UrsusArcanus>
over the years here I know that Sylvan's been working on distributed Pony intermittently. Does anyone know if reference capabilities are extended to the distributed API?
<UrsusArcanus>
I've been toying with a gRPC replacement that needs to work across hundreds (or potentially thousands) of nodes at varying latencies
<UrsusArcanus>
part of the concept is to have a meta-API that can delegate calls between servers and pass data around between nodes in tuples.
<UrsusArcanus>
refcaps leave my jaw quite literally on the floor in terms of what Pony is capable of on a single machine . I'm still wrapping my mind around it but it seems so much more intuitive than async/await style interfaces for managing concurrency.
<UrsusArcanus>
so I was wondering if the distributed project had intended to share some of that goodness between machines.
<UrsusArcanus>
@aturley - appreciated your FFI posts. still watching and re-watching to grok all of it, but having gone thru some small experiments in crystal and Nim here recently I like Pony's near seamless approach.
<SeanTAllen>
UrsusArcanus: im not aware of anyone working with those libraries. The user mailing list and the slack would be other good places to check.
<SeanTAllen>
Sylvan hasn't been working on distributed Pony. No work on that has been done in a while as far as I am aware. It's not one of the higher priorities for anyone using Pony that I'm aware of. If someone wanted to take that on, the core team would be happy to help. We certainly learned a lot about what would be needed from working on Wallaroo.
jemc has quit [Ping timeout: 250 seconds]
<SeanTAllen>
what did you mean by: "so I was wondering if the distributed project had intended to share some of that goodness between machines." I didn't really follow that.
<UrsusArcanus>
thx Sean. I meant in implementing refcaps in the distributed library
<UrsusArcanus>
btw - I'm the same crazy that was asking if anyone was using pony for blockchain stuff a while ago.
<UrsusArcanus>
Instead, I went to work for one of the largest btc companies on the planet. we lost ~ 1 billion USD. so... new project time! :D
<SeanTAllen>
what do you mean by refcaps in the distributed library?
<SeanTAllen>
still not getting it
<UrsusArcanus>
the ability to set reference capabilities on shared data as part of the interface. like - imagine being able to define box or tag on a protobuf msg
<SeanTAllen>
im still not getting it.
<SeanTAllen>
once its bytes on the wire and in another process, why does it matter?
<UrsusArcanus>
microservices mesh hell.
<UrsusArcanus>
the "serverless" trend is turning all of our code into container ran functions on a cloud machine
<SeanTAllen>
sorry, just not getting it
<SeanTAllen>
i guess i need a concrete use case to understand
<UrsusArcanus>
basically, I want to wire actors together between runtimes
<UrsusArcanus>
across a multi-hop network
<UrsusArcanus>
so the program doens't know if it has many cores or many physical machines
<UrsusArcanus>
the actors themselves are addressable (in a utopia like world)
<UrsusArcanus>
in a highly connected microservices environment, we might want to share a struct of data that's potentially being updated in somem other service
<SeanTAllen>
i followed up until then end
<SeanTAllen>
then you lost me
<SeanTAllen>
sounds like you want RDMA
<UrsusArcanus>
kiiiind of
<UrsusArcanus>
so, linda works by establishing a couple of simple primitives on how you can take or place tuples on a shared stack
<UrsusArcanus>
imagine a lot of pony clients working against a set of tuples in a similar way ... except being passed as messages between actors
<SeanTAllen>
again, sorry, just not getting it
<SeanTAllen>
i dont get what a "shared stack" is when its cross physical machine boundaries
<UrsusArcanus>
the machine boundary / network isn't important to the program. it just thinks it's messaging another actor, which looks like its local, but is on a different IP
<UrsusArcanus>
erlang does this by making everything immutable
<SeanTAllen>
sorry, i dont get it
<UrsusArcanus>
no worries :)
<SeanTAllen>
i dont see what that has to do with refcaps
<SeanTAllen>
you arent sharing memory in the erlang case you just mentioned
<SeanTAllen>
the data exists, somewhere, managed by an actor and you send messages to that actor
<SeanTAllen>
but early you mentioned "sharing" that data
<SeanTAllen>
which sounds like more than 1 copy to me
<SeanTAllen>
which is a different problem entirely
<SeanTAllen>
So, just not getting it
<SeanTAllen>
something i'm missing
<SeanTAllen>
I'd need a real concrete example
<UrsusArcanus>
yes, for me, too ;)
<UrsusArcanus>
if I understand correctly, the refcaps are a much better way of handling mutexes between multiple processes that want references (of some kind) to the same piece of data.
<UrsusArcanus>
so some can read, only 1 can write, or maybe nobody can write (val iirc)
<UrsusArcanus>
in a distributed system, each node has it's own local copy of the data. they indeed to not have a reference to it.
<UrsusArcanus>
*do not
<UrsusArcanus>
what i'm suggesting is that, what if we could *emulate* refcaps by tagging that data when it's passed to an API / function call on a remote machine
<UrsusArcanus>
a common pattern that pops up in microservices is that we have to relay API calls between machines
theodus has joined #ponylang
<UrsusArcanus>
a message arrives to some terminal node in an kind of graph of API calls, and get some processing done to it.
<UrsusArcanus>
then it performs a callback with the modified data
<UrsusArcanus>
not sure if that's concrete enough
<SeanTAllen>
nope
<UrsusArcanus>
:D
<SeanTAllen>
" the refcaps are a much better way of handling mutexes between multiple processes that want references (of some kind) to the same piece of data." <-- not sure what that means but it doesn't sound quite right to me
<SeanTAllen>
"mutexes between multiple processes" sounds incorrect to me
<UrsusArcanus>
well, one mutex ... multiple processes
<SeanTAllen>
again, sounds incorrect to me
<SeanTAllen>
refcaps are about safe usage of memory with a single process that features concurrency
<SeanTAllen>
so for example, you can't send mutable data between actors unless its isolated and you give up your alias to it.
<SeanTAllen>
effectively "give it away"
<UrsusArcanus>
right
<SeanTAllen>
there's nothing about multiple processes or mutexes
<UrsusArcanus>
behind the scenes, pony is managing the threading for us though, correct? so if I have 1000 actors spun up across say 10 types, i have no control over what process or what thread is accessing memory referenced by the actor
<SeanTAllen>
if you wanted to use shared memory between processes, there's nothing in Pony's refcaps that will stop you from f'ing that up. Refcaps are compile time and do not cross process boundaries. A pony program has no concept of refcaps at runtime.
<SeanTAllen>
are you using "process" in the erlang sense?
<UrsusArcanus>
no in the posix sense -
<SeanTAllen>
then no
<SeanTAllen>
"so if I have 1000 actors spun up across say 10 types, i have no control over what process or what thread is accessing memory referenced by the actor"
<SeanTAllen>
that isn't correct
<SeanTAllen>
pony is single process
<UrsusArcanus>
does pony call down to any fork() magic ? Or it's just rnning as single multi-threaded proc?
<UrsusArcanus>
ah - ok
<SeanTAllen>
there are Y actors and there are X number of scheduler threads which will work together to run those actors and process their work
<SeanTAllen>
all data is owned by a particular actor basically:
<UrsusArcanus>
my mind is still down in CSP land
<SeanTAllen>
can be shared if its immutable with other actors
<SeanTAllen>
can be "given" to another actor if its mutable and was declared `iso` so there is only a single alias to it and the originating actor gives up the alias to that memory
theodus has quit [Quit: Leaving]
<SeanTAllen>
however, for GC purposes, the actor that allocated memory always owns it and will eventually be the one to GC it.
<UrsusArcanus>
I have some experiments to do to understand this a bit better!
<UrsusArcanus>
if i understand right... multiple actors could have a box reference to data, but only a single actor may have an iso at any given time
<UrsusArcanus>
as box is read only
<UrsusArcanus>
i think where i'm a little confused is with rerf
<UrsusArcanus>
*ref
<UrsusArcanus>
"ref -(reference) alias is R/W, other aliases can be R/W"
<UrsusArcanus>
I need to rewatch the vids
<SeanTAllen>
no
<SeanTAllen>
multiple actors can have val reference to data
<SeanTAllen>
box is the union of (val, ref)
<SeanTAllen>
ref is plain old mutable
<UrsusArcanus>
ahhhhhhh
<SeanTAllen>
you can have as many as you want but its not shareable
<SeanTAllen>
you can never send a ref to another actor
<UrsusArcanus>
got it!
<UrsusArcanus>
"ref is plain old mutable" <--- the ah-ha moment
<UrsusArcanus>
thanks for chiming in during coffee time. I'll be back with more newbie questions I'm sure
<SeanTAllen>
sure thing.
Foaly has joined #ponylang
UrsusArcanus has quit [Ping timeout: 252 seconds]
Foaly has quit [Quit: Now 'mid shadows deep falls blessed sleep.]
endformationage has quit [Ping timeout: 272 seconds]
Foaly has joined #ponylang
ashirase has quit [Remote host closed the connection]
Foaly has quit [Quit: Now 'mid shadows deep falls blessed sleep.]