<DanC_>
hmm... primitive _init takes a parameter of type Env. so any module can get an Env just by declaring a primitive?
jemc has quit [Quit: WeeChat 1.4]
jemc has joined #ponylang
copy` has quit [Quit: Connection closed for inactivity]
trapped has joined #ponylang
bb010g has quit [Quit: Connection closed for inactivity]
jemc has quit [Ping timeout: 246 seconds]
Jbbouille has joined #ponylang
trapped has quit [Read error: Connection reset by peer]
Jbbouille has quit [Ping timeout: 250 seconds]
srenatus has joined #ponylang
<sylvanc>
ponysaurus: x.bitwidth()
<sylvanc>
having a separate call for a byte size vs a bit size seems unnecessary, although of course always open to be showing it's worth it
<sylvanc>
if you are doing a writebuffer, and you need .bitwidth(), it's possible you are doing it wrong :)
<sylvanc>
sounds like maybe you are accepting a union type to be written to the buffer?
<sylvanc>
if so, every caller will end up allocating memory
<sylvanc>
DanC: yeow, that's an oversight from long ago!
<sylvanc>
i think primitive _init should take no arguments
<sylvanc>
thanks for the heads up
aturley has joined #ponylang
aturley has quit [Ping timeout: 264 seconds]
trapped has joined #ponylang
BrotherLy has joined #ponylang
trapped has quit [Read error: Connection reset by peer]
trapped has joined #ponylang
BrotherLy_ has quit [Ping timeout: 244 seconds]
_andre has joined #ponylang
aturley has joined #ponylang
aturley has quit [Ping timeout: 248 seconds]
BrotherLy has quit [Quit: Leaving]
Praetonus has joined #ponylang
aturley has joined #ponylang
aturley has quit [Ping timeout: 268 seconds]
ponysaurus has quit [Ping timeout: 250 seconds]
aturley has joined #ponylang
jemc has joined #ponylang
mcguire has left #ponylang [#ponylang]
copy` has joined #ponylang
graaff has joined #ponylang
amclain has joined #ponylang
jemc has quit [Quit: WeeChat 1.4]
jemc has joined #ponylang
jemc has quit [Quit: WeeChat 1.4]
jemc has joined #ponylang
srenatus has quit [Quit: Connection closed for inactivity]
ponysaurus has joined #ponylang
<SeanTAllen>
@Praetonus i dont understand why a type for "collection of arrays" would be needed and how that is different than Seq[Array[A]] or why that is preferrable to Seq[Seq[A]], can you explain?
<ponysaurus>
sylvanc: Yes, I was accepting a union type. I am going to clean that up.
<ponysaurus>
@jemc, @praetonus .. : The following makes more sense, would like to hear your thoughts: Have a list of arrays as the writebuffer data structure. Keep allocating a new buffer, as we fill up (possibly exponentially increase the size?).
<ponysaurus>
somehow provide an implementation to extract a sendable packet from the HEAD of the buffer. Then manage the buffer and the arrays that are getting used up at the head?
<ponysaurus>
does this sound more reasonable?
<ponysaurus>
*keep allocating for a new array (NOT buffer)
<Praetonus>
SeanTAllen: The difference is that the collection I'm proposing provides a direct access to its elements. The collection of arrays is just a storage detail. In an array, all elements are stored contiguously. In a linked list, elements are separate. Here, the idea for the storage is to have nodes (like a linked list) of contiguous storage (like arrays). Some kind of hybrid structure
<Praetonus>
But the key really is the direct Seq access to elements. The collection could then be used directly in WriteBuffer and even be retreived directly by the user since it provides a safe Seq interface
<Praetonus>
The underlying arrays wouldn't be accessible directly
<ponysaurus>
I was just about to type the first sentence, yes!
<ponysaurus>
type your first sentence
<SeanTAllen>
"the underlying arrays wouldn't be accessible directly", that sounds like "access" means a copy. am i misinterpreting that?
<SeanTAllen>
i'm not sure i am going to understand without at least some pseudo code
<ponysaurus>
how would you provide safe and sendable access to the first N elements without a copy?
<ponysaurus>
this is the tricky part
<Praetonus>
By access, I mean there won't be a array(i: USize) returning the ith array
<Praetonus>
But there will be a apply(i: USize) returning the ith global element
<ponysaurus>
of course not, by access I don't mean that either. But how do we extract the first few elements without a copy?
<Praetonus>
e.g if there are two arrays of size 500 and you ask for the 750th element, you get the 250th of the second array
<jemc>
ponysaurus: to avoid copying, my inclination is that when I want to "take" some bytes from WriteBuffer object, it gives me back a ByteSeqIter rather than a ByteSeq
<jemc>
the ByteSeqIter can be used with `writev`
<jemc>
so if you have the N bytes you want to return segmented over multiple arrays internally, you return those multiple arrays to avoid copying as much as possible
<Praetonus>
I'd say the whole object is the ByteSeq. We just have to keep the global size to ensure all accesses to elements are in the structure bounds
<ponysaurus>
hahaha .. brings me back to my issue about ByteSeqIter ;)
<jemc>
see scatter/gather IO
<SeanTAllen>
what issue is that ponysaurus ?
<jemc>
Praetonus: TBH I'm not seeing how your proposed data structure is that helpful for either (Read)Buffer *or* WriteBuffer
<SeanTAllen>
in my experience praetonus, "just" tends to hide a lot of complexity
<jemc>
it sounds like your DS is just a different implementation of an Array or List - I don't see how it solves the problems we're talking about (or at least the ones *I'm* talking about)
<jemc>
not that it wouldn't be a useful data structure - I just see it as a separate conversation I think
<SeanTAllen>
i am very confused at this point. i think some code, pseudo or otherwise would help quite a bit
<ponysaurus>
SeanTAllen: #691
<jemc>
ponysaurus: yeah, that's something we need to fix - but I don't think it invalidates my point :)
<ponysaurus>
@jemc, I am going to try and code that up with ByteSeqIter. What's a best way to collaborate in cases like this? github gist?
<ponysaurus>
i am sure I will bother you here, with questions
<jemc>
gist is fine - or you can point to a different branch on your personal repo
<ponysaurus>
ok .. will do
<jemc>
but basically, I would picture something like this:
<ponysaurus>
go on ..
<ponysaurus>
:)
<Praetonus>
jemc: You're correct, ByteSeqIter could be used for this. However, getting a Seq from WriteBuffer could be useful, for example to allow some final modifications before sending
<Praetonus>
SeanTAllen: I'll make a basic implementation and put the link on Github, it will be better if we have actual code to discuss
<Praetonus>
SeanTAllen: On a completely unrelated subject, I can't find the invite on Github. Is it hidden in a special tab or something like that?
<SeanTAllen>
Praetonus: they send an emaik
<SeanTAllen>
Praetonus: they send an email
<jemc>
ponysaurus: internally, I have a List of ByteSeqs, some may be Array[U8] and some may be String: ['F', 'O', 'O'], "BAR", ['B', 'A', 'Z'] - I only add elements to the tail, and only remove elements from the front - in the public API, I can `append` an entire ByteSeq (Array[U8] or String) by pushing to the tail of the list - I can `push` a U8 to the list by either pushing it onto the last ByteSeq or creating
<jemc>
a new ByteSeq with one element (or varying the strategy based on the circumstances, whichever is deemed to be the most efficient algorithm)
<SeanTAllen>
it also says they can be accepted at "github.com/ponylang" Praetonus
<Praetonus>
SeanTAllen: Ah yes, here it is
<jemc>
ponysaurus: to remove from the front, I remove whole ByteSeqs at a time - if for some reason I need to limit the number of elements to some maximum N, I may need to in some cases split a ByteSeq on the boundary and copy its contents into two resulting ByteSeqs, one kept at the head and the other as the tail of the return ByteSeqIter
<jemc>
however most of the time I would think you wouldn't need to limit the number of elements, so you would just extract the entire contents as a ByteSeqIter and continue with an empty list
<jemc>
I think your use case of fixed size packets is the more specialized one, compared to the generic use case of arbitrary-sized packets
<jemc>
not all transports have framing, so the difference between two fixed size packets and a single arbitrary packet combining both of their contents is often a difference without a distinction
<ponysaurus>
This makes sense, however, i still don't like resizing the existing ByteSeq at the tail, every time I push a number to it.
<ponysaurus>
jemc
<ponysaurus>
jemc:
graaff has quit [Quit: Leaving]
<jemc>
there's room to tweak the optimal algorithm there - however it should be noted that Array has some semi-"smart" resize logic when it resizes, so that even if you declare size 1, it starts at 8 and increases by powers of 2
<ponysaurus>
jemc: And also, am I the only one who wishes for fixed sized packets? MTU in network transport is still a useful thing
<jemc>
ponysaurus: you're not the only one - I just think that this is not the only use case we need to support
<ponysaurus>
ok, thanks for the resize info
<ponysaurus>
yes, and I completely understand the don't care about packet size case. But, you should take care of people who care about packet sizes. Because, once you make that choice, you may lose them :(
<ponysaurus>
jemc
<jemc>
this is one reason why I'm always harping about third-party packages over stdlib ones - it's often the shortest path to usefulness if we each solve our own specific problems and share the solutions with others who want to use and contribute to them - rather than trying to make something specialized into something generic and waiting/working for community consensus on that generic solution, when everyone has
<ponysaurus>
@jemc: all in all, thanks a lot for your feedback
<jemc>
different needs and priorities
<jemc>
ponysaurus: yes, I'm not saying it can't support fixed size packets - just saying we probably want to support arbitrary packet sizes if it's going to be in stdlib
<ponysaurus>
I agree, this isn't a compiler issue. And can be made into a 3rd party package first. I will do that.
<jemc>
the problem right now with 3rd party packages is that we don't have a package manager yet :D
<jemc>
but yes, you can certainly move faster when you're not trying to get everyone in a diverse group of people to agree on something
<ponysaurus>
and then the stdlib can take it, if it makes sense. This looks like a much more logical route.
Matthias247 has joined #ponylang
<ponysaurus>
jemc: Just to summarize my thoughts: Imagine a WriteBuffer as a list of fixed size ByteSeqs. Maintain a HEAD and TAIL. Tail to where the user keeps writing. Head from where the user demands a ByteSeqIter of a certain length. Now I am not sure if pony has a way to provide this ByteSeqIter without copying overhead. But I believe this will provide what’s necessary for both parties who care about the packet size and to those who d
<ponysaurus>
I would like to have people's opinion on this. And this will definitely fit my use case, once issue 691 is fixed. I am going to focus on trying to get something like this implemented for myself.
<jemc>
ponysaurus: using a List[ByteSeq], if your packet size happens to fall on a boundary between ByteSeqs, it should be possible without copying to sever the link of the linked list at that boundary
<shepheb>
I made ponyc segfault. pulling and rebuilding, and I'll try with the latest.
<jemc>
if your packet size doesn't happen to fall on a boundary, I don't think there's any way to avoid copying - which is one big reason why one might not want to use an exact packet size
<jemc>
you could instead try for a packet size *near* some given size - rounding to the nearest Array boundary - if your concern is MTU
<jemc>
errr.. ByteSeq boundary
<shepheb>
no segfault on master. never mind then.
<ponysaurus>
@jemc : Ok, thanks!
prettyvanilla has quit [Quit: Konversation terminated!]
prettyvanilla has joined #ponylang
aturley_ has joined #ponylang
prettyvanilla has quit [Excess Flood]
prettyvanilla has joined #ponylang
aturley has quit [Ping timeout: 276 seconds]
<DanC>
re package manager... I wonder if it would help to use nix until you manage to do something better
<jemc>
DanC: I think the main point of contention in the package manager issue is not about the actual management of packages, but about defining how the user experience looks - where do you define dependencies and how does it integrate (or not integrate) with the compiler
<jemc>
there are some great ideas in that thread, some of them conflicting - I have some ideas about how to try to support a variety of use patterns via plugins to a generic `pony` binary, but haven't gotten around to fleshing these out yet
<jemc>
but I think the answer is yes - we should support things like nix and gx that already work well for managment of packages
<jemc>
but have a nice user experience and some flexibility to that experience so users can do the advanced things they need for their use case
<jemc>
in my own personal roadmap for pony work, I'm planning to try to take up this mantle and start some work on this as soon as the `process` package is done
<jemc>
ideally we can have a flexible-enough plugin system that everyone can be happy with how their dependencies are defined and fetched
<DanC>
"nice user experience" might be a bit generous for nix. it took me quite a while to learn, and I'm still only comfortable with a small part of it.
_andre has quit [Quit: leaving]
<ponysaurus>
jemc: A ByteSeq == (String | Array[U8] val). So appending to an existing ByteSeq is impossible since it's immutable if it were an Array[U8] val. Am I missing something?
<jemc>
DanC: I'm talking mainly about the tooling/integration with the Pony environment - but I don't know hardly anything about nix workings at the moment
<jemc>
which is why my first integration would probably use gx
<doublec>
Is the process package being developed in a github repo somewhere?
<jemc>
ponysaurus: a type always declares a "default" cap, but that doesn't stop you from using it with another cap
<jemc>
for example: `let x: ByteSeq iso = recover iso [as U8: 71, 72, 73] end`
<ponysaurus>
ahhhh .. ok! Didn't know that!!!
<jemc>
that said, for ease of implementation, you might want to keep everything but the final ByteSeq as a val, and use iso only for the final one
<jemc>
so once you start a new tail, `consume` the old one into a `val` in your list - should be no need to mutate it after that
<jemc>
that's up to you though - I just mention because often `val` is easier to handle than `iso`
<ponysaurus>
yes, this is exactly what I was hoping to do. I admit, code is a lot cleaner with `val` than `iso`
<jemc>
doublec: as far as I know, the code for the process package isn't yet public
trapped has quit [Read error: Connection reset by peer]
ponysaurus has quit [Ping timeout: 250 seconds]
Matthias247 has quit [Read error: Connection reset by peer]