<codec_>
So far I have done all problems until day 11 but there is an issue I ran into from time to time is that my computer start swapping (and freezing) because of too much memory allocation
<codec_>
This is often because I do immutable string manipulation, so the obvious solution is to use mutable string to reuse memory
<codec_>
But I have wonder is there is any simpler way to collect garbage from time to time
mollymorphic has quit [Ping timeout: 250 seconds]
<codec_>
Last time someone gave me a hint on how to call the garbage collector -using @pony_triggergc[None](@pony_ctx[Pointer[None]]()) -
<codec_>
but it does not seems to collect any memory. Why is that ?
benq has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
benq has joined #ponylang
mollymorphic has joined #ponylang
<codec_>
my bad, I just saw that the call to the gc force the actor to ge gc-ed the next time it is scheduled
<codec_>
so it won't run the gc if the actor is running a loop
nisanharamati has joined #ponylang
codec_ has quit [Ping timeout: 260 seconds]
samuell has joined #ponylang
aturley has quit [Read error: Connection reset by peer]
user10032 has joined #ponylang
notsgnik has quit [Ping timeout: 260 seconds]
mollymorphic has quit [Ping timeout: 240 seconds]
mollymorphic has joined #ponylang
<slfritchie>
Correct. Also, in my limited experience, the runtime doesn't make a lot of effort to return memory to the OS. So looking at RSS or total process size won't tell you how effective GC was at reclaiming stale memory.
oraoro has joined #ponylang
benq has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
oraoro has quit [Ping timeout: 248 seconds]
mollymorphic has quit [Ping timeout: 248 seconds]
benq has joined #ponylang
mollymorphic has joined #ponylang
profetes__ has quit [Quit: Leaving]
endformationage has joined #ponylang
mollymorphic has quit [Ping timeout: 250 seconds]
olof has quit [Ping timeout: 268 seconds]
mollymorphic has joined #ponylang
benq has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
benq has joined #ponylang
_andre has quit [Quit: leaving]
trevorriles has joined #ponylang
mollymorphic has quit [Ping timeout: 268 seconds]
codec_ has joined #ponylang
<codec_>
actually I have rewritten my code so it runs on a actor calling itself to do the computation
<codec_>
this way the runtime can free (or at least reuse) the unsued memory
<codec_>
and it practice I see no noticeable memory allocation
<codec_>
I wish there were a compiler option so an actor can be preempted by the runtime and run a gc pass when it start to use too much memory
<codec_>
The good thing with the current state is that memory allocation that go out of proportion are easily noticable
<codec_>
The bad one is that it can make the system way slower than a "traditional GC" if the computer starts to swap on almost any instruction
<codec_>
Sean told me some time ago, that they are people working on a way to bring a more traditional "preemptive" threading model for some actor
mollymorphic has joined #ponylang
<codec_>
If someone has any bit of information about this, please let me know as I would be happy to know the current status and the direction it take
mollymorphic has quit [Ping timeout: 240 seconds]
trevorriles has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
mollymorphic has joined #ponylang
mollymorphic has quit [Ping timeout: 268 seconds]
stephen has joined #ponylang
stephen is now known as Guest42999
gokr has joined #ponylang
codec_ has quit [Quit: Page closed]
benq has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
benq has joined #ponylang
mollymorphic has joined #ponylang
mollymorphic has quit [Ping timeout: 248 seconds]
<SeanTAllen>
codec_ i'd like to point out a few things about the non pre-emptive scheduler:
<SeanTAllen>
1) yes you need to account for giving up slices in order to not use too much memory however, the simplicity of the scheme allows for...
Guest42999 has quit [Ping timeout: 260 seconds]
<SeanTAllen>
2) backpressure built into the run (that you can see me working on here https://github.com/ponylang/ponyc/pull/2264) that has little to no overhead on normal execution
<SeanTAllen>
the lack of backpressure is a means where with preemptive or cooperative scheduling, that you can have runaway memory growth.
<SeanTAllen>
building a backpressure system with little to no overhead is pretty much impossible to do at the application level
<SeanTAllen>
build a backpressure system like the one we will soon have in Pony would be very hard to do with little to no overhead if you had a pre-emptive scheduler.
codec_ has joined #ponylang
<SeanTAllen>
the "cost" here is that you need structure your code a little differently in order to play nice with the scheduler
<SeanTAllen>
however, the advantages that are about to flow from that are huge
<SeanTAllen>
I recognize that needing to think about your memory usage for and scheduling can be annoying however...
<SeanTAllen>
1) there's no direct connection between cooperative and preemptive and when gc happens.
<codec_>
thanks for the explanation, yet I am not familiar with the terminology. What is backpressure?
<SeanTAllen>
you can implement a preemptive scheduler and have it run for a given actor til after a behavior has executed.
<SeanTAllen>
so you are conflating some ideas some.
<SeanTAllen>
every technical decision generally has things it makes easier and things it makes more difficult.
<SeanTAllen>
you are very correct that some things are made more difficult by cooperative scheduling.
<SeanTAllen>
you have to think about giving up the scheduler periodically if you have a "long running" behavior
<SeanTAllen>
the fact that currently gc will not be attempted til after a behavior has finished executing also has a cost as you have noticed
<codec_>
Sure
<SeanTAllen>
the also come with advantages. the ease of being able to implement backpressure into the runtime is going to be a huge win.
<SeanTAllen>
and we can do it without having to examine queue sizes (which would be a massive performance hit)
<SeanTAllen>
so codec_: backpressure, lets talk about that for a moment
<SeanTAllen>
if a part of a system is overloaded, you have 2 options on how to address
<SeanTAllen>
you can shed load
<SeanTAllen>
or you can exert backpressure
<SeanTAllen>
backpressure is some means of having upstream producers slow down so they dont swamp downstream consumers
<SeanTAllen>
TCP features backpressure
<SeanTAllen>
eventually OS buffers fill up and the machine will stop accepting new data
<SeanTAllen>
which causes buffers on the sending machines to fill
<SeanTAllen>
until eventually system calls to send data start failing
<SeanTAllen>
that's backpressure
<codec_>
I understand, does it exist in other context that IO ?
<SeanTAllen>
in the not so distant future, the pony runtime will feature backpressure that can prevent "normal" sending patterns from overloading a given actor
<SeanTAllen>
yes, the choice of backpressure or load shedding or failure exists in any system where you have producers and consumers
<SeanTAllen>
imagine a pony program with 2 actors
<SeanTAllen>
if actor A sends more messages to actor B than it can process and it does this constantly, the program will run out of memory because the mailbox for actor B will grow and grow and grow
<SeanTAllen>
detecting and dealing with that can be incredibly expense performance wise
<SeanTAllen>
current testing on my backpressure work shows little to no performance impact, that's going to a huge win.
<SeanTAllen>
with a pre-emptive scheduler, that wouldn't be possible. at least not using the technique i'm using.
<SeanTAllen>
tradeoffs.
<SeanTAllen>
ill take having to rewrite some algos and having backpressure because high performance backpressure is a very difficult thing to achieve.
<codec_>
so basically, you are saying that you have found a way to slow down an actor sending too much messages so the other can catch up (ie process the message) without any performance penalties?
<SeanTAllen>
especially in a generalized, non specific fashion like how it is being added to the pony runtime
<SeanTAllen>
codec_: yes. without any performance penalties for an application that isn't in a "backpressure" situation
<codec_>
because if it is in a backpresseure situtation where and why does the performance occurs?
<codec_>
*performance drop
<SeanTAllen>
exterting backpressure will result in a change in performance. the goal was to have little to no performance impact when backpressure wasn't being exerted or when it was being exerted occassionally
<SeanTAllen>
it drops because backpressure means intentionally slowing down producers
<SeanTAllen>
which means lowering their performance so that the system can continue to function and remain in a state of equilibrium
<codec_>
yes, but isn't it possible to run more consumers and still maximizing CPU usage?
<SeanTAllen>
however if A send to B in a "maintainable"/"no backpressure needed" state, then, there shouldn't be a performance cost for that
<SeanTAllen>
how do you know you need more consumers codec_?
<codec_>
well let's say we some actors of type A that produce data and them to actors of Type B that consume them
<codec_>
if the A actors run too often they may overload the system
<SeanTAllen>
yes
<codec_>
so to decrease load we might need to run more often the actors of type B
<SeanTAllen>
any producer can in theory overload the system
<SeanTAllen>
well you may or may not be able to create more actors of type B
<SeanTAllen>
if actor B is a socket and you need ordering guarantees, you cant start more actors of type B
<codec_>
I don't mean more
<codec_>
but scheduling them with higher priorities
<SeanTAllen>
sure
<SeanTAllen>
thats one way to look at it
<SeanTAllen>
the question is...
<SeanTAllen>
how do you know that actor B is falling behind?
<SeanTAllen>
note, the simple way is to say "its queue is growing larger"
bimawa2 has quit [Ping timeout: 240 seconds]
<SeanTAllen>
however, observing a queue size is a very expensive operation and will have an impact on application performance
<codec_>
yes, I didn't think of that
<SeanTAllen>
so, your idea of scheduling with higher priorities is one way to describe what my backpressure work does
bimawa_ has quit [Ping timeout: 268 seconds]
<SeanTAllen>
what it really does though, is skips scheduler runs for actors that are causing "overloads"
<codec_>
sounds really cool
<SeanTAllen>
the really tricky part is how do you know an actor is overloaded
<SeanTAllen>
without observing its queue length
<SeanTAllen>
you need a cheap heuristic in order to be able to do that
<SeanTAllen>
the cooperative nature of the pony scheduler helps with that
<SeanTAllen>
each actor has a batch size, by default, its 100
<SeanTAllen>
any actor can run up to 100 behavior calls before it gives up the scheduler
<SeanTAllen>
if it hits 100, then the runtime stops executing that actor and schedules the next one
<SeanTAllen>
the heuristic im using is...
<SeanTAllen>
if an actor can keep up with the messages it is being sent then it should never hit the max of 100
<SeanTAllen>
i tried being more tricky but so far, that simple heuristic works
<SeanTAllen>
if we hit the max size where we give up the scheduler, we denote the actor as "overloaded"
<SeanTAllen>
sending to an overloading actor can get other actors to be unscheduled until the actor they sent to is no longer overloaded
<SeanTAllen>
there's a bit more to it, but that is the basics of it
<SeanTAllen>
i dont know how this could be done with a pre-emptive scheduler but its something that i'm thinking about because
<SeanTAllen>
being able to do preemptive scheduling AND have backpressure would be great
<codec_>
thanks really like to get a glimpse of the inner workings of pony
<SeanTAllen>
however, there's no obvious, cheap heuristic that i've been able to come up with for a preemptive scheduler
<SeanTAllen>
basically, tradeoffs.
<codec_>
yeah, personally for the kind of app I am writing right now I wouldn't mind giving away a bit of performance if it make writing code a bit easier
<SeanTAllen>
i write 3 different app level backpressure systems last year
<SeanTAllen>
non ended up working for all the traffic patterns i could throw at them
<SeanTAllen>
all of them had large performance impacts
<codec_>
I was just wondering if it could be possible that the actor can tell the pony runtime that it want a more traditional threading model
<SeanTAllen>
in some cases, depending on the traffic pattern, 2/3 of messages based would be for dealing with backpressure and not applicaiton work
<SeanTAllen>
i dont know what a "more traditional threading model" means
<codec_>
yeah that is a lot
<codec_>
well the pony runtime could allocate a native thread for this specific actor
<codec_>
so it won't starve the other
<codec_>
and potentially could be garbage collected from time to time
<SeanTAllen>
Sylvan has discussed that idea with me
<SeanTAllen>
I think its a bad idea but he says he is going to convince me at some point
benq has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
<SeanTAllen>
codec_: youve conflated when gc happens again with scheduling
<SeanTAllen>
the fact that gc is done after a behavior is running is an implementation detail
benq has joined #ponylang
<SeanTAllen>
that could be true whether its preemptive or cooperative multitasking
<codec_>
I am not sure to follow, you mean actors could be garbage collected anytime theorically ?
<SeanTAllen>
yes
<SeanTAllen>
its something Sylvan and I have discussed. How we can do that with minimal overhead.
<SeanTAllen>
We recognize the short comings of the existing implementation and are trying to find ways to address them while having a limited performance impact.
<SeanTAllen>
are you familiar with Erlang and the BEAM vm codec_?
<codec_>
so you mean the problems I have run into (system start swapping way too much) could get away in a future version of pony?
<codec_>
not at all
<SeanTAllen>
Erlang has a preemptively scheduled actor system in its vm
<SeanTAllen>
You can write non-erlang code to run within the VM
<SeanTAllen>
these are called NIFs
<SeanTAllen>
writing NIFs is a black art as slfritchie can attest
<SeanTAllen>
it's a back art in part because you need to be conscious when writing NIFs to play nice with the preemptive scheduler
<codec_>
NIFs is basically FFI, for calling native code?
<SeanTAllen>
Yes
<SeanTAllen>
Pony would end up with the same kind of problem with a preemptive scheduler
<SeanTAllen>
Unless we find something clever to do, it will make writing and use code via FFI much more difficult
<SeanTAllen>
tradeoffs.
<codec_>
I don't get it. Why the preemptive scheduler create problems?
<SeanTAllen>
my point is, it isnt as simple as "preemptive scheduling is better"
<SeanTAllen>
there are tradeoffs and you make some things easier and some things harder
<SeanTAllen>
the general pony philosophy is, "its ok to make easy things harder if you make really hard things easier"
<SeanTAllen>
so if/when we introduce a preemptive scheduler, it can't make hard things harder.
<SeanTAllen>
and it needs to make harder things easier.
<codec_>
I will read the article later, but the problem you are talking about are only about performance or they concern other domains (like thread safety and such) ?
<SeanTAllen>
making easy things easier isn't a win if ti makes hard things harder.
<SeanTAllen>
the problems that FFI can introduce are many
<SeanTAllen>
how to incorporate FFI code into a scheduler is made harder in general by preemptive scheduling
<SeanTAllen>
thread safety can definitely be made harder by FFI as you can be dropping down into a language like C that can stomp on memory, use globals and everything else you can do with C
<codec_>
sure it happened to me quite often, but isn't the same regardless of the scheduling?
<SeanTAllen>
yes
<codec_>
to come back to the root, the issues I ran into with the pony runtime were: 1- the memory wasn't gc collected before the system start to swap 2- I ran into some threading issues (it seems) when doing a SDL app
<codec_>
So it seems I could get 1 for free if the GC implementation is changed in Pony
<codec_>
am I correct?
<SeanTAllen>
thats not quite true
<SeanTAllen>
we could swap in any garbage collector
<SeanTAllen>
but changing that implementation doesnt matter
<SeanTAllen>
its when and how it gets triggered
nisanharamati has quit [Quit: Connection closed for inactivity]
<SeanTAllen>
we could insert a check to see if gc should be done after every single instruction is run
<SeanTAllen>
instead of after each behavior.
<SeanTAllen>
that could fix your problem now
<SeanTAllen>
it would also be a very expensive thing to do
<SeanTAllen>
thats an extreme
<codec_>
sure, so from what your said earlier I understood that there was a solution to do GC without any real performance drop
<SeanTAllen>
that said, some garbage collector implementations might be better than others for being able to trigger gc closer to when its needed with less performance overhead
<codec_>
while avoiding the situation where the system has eaten all of its memory
<SeanTAllen>
i didnt say that. i said Sylvan and I were discussing how that could be done.
<SeanTAllen>
you can do that now by restructuring your code. thats one solution.
<codec_>
and do you think there is a better alternative ?
<SeanTAllen>
we are looking at how it could be done without restructuring code.
<SeanTAllen>
i think that we will eventually come up with something where we think the tradeoffs are reasonable.
<SeanTAllen>
we are playing around with ideas.
<codec_>
I see
<codec_>
About 2, has anybody tried to do GUI app, or simple game in Pony?
<SeanTAllen>
yes
<codec_>
Did they were succesful ?
<codec_>
Because I ran into really weird issues that I cannot explain at all
<codec_>
I managed to completely freeze my computer when tring to do a hello world SDL app with Pony.
<codec_>
My best guess was that I did something wrong with the threads but I am not really sure about this
<SeanTAllen>
Yes, my understand is that some folks have been succesful with SDL
<SeanTAllen>
my understanding is that its kind of particular about threads
<codec_>
sorry?
<SeanTAllen>
there's something you need to do with threads for it to work
<SeanTAllen>
i dont remember the particulars
<SeanTAllen>
it has come up in IRC before
<SeanTAllen>
it would be awesome if someone wrote up a Pony pattern for how to do it
<codec_>
actually that is my secret goal
<SeanTAllen>
jemc when he is around might be able to direct you in the proper direction
<SeanTAllen>
or you can also try the mailing list
<SeanTAllen>
someone there might know
<SeanTAllen>
i dont remember much
<codec_>
I want to write some simple app with common libs to get people started easily