<marler8997>
might not be useful though because any code that takes a string ending in newline would usually also have to handle it not ending ina newline
<marler8997>
std.testing.expectEqualSlices?
<marler8997>
I'll let you know if I find anything else about the build times
<marler8997>
the new API also allows the client to know how much is available without having to try every size. For example, if you ask for 100 bytes, but the allocator has to give you 4096 (like mmap), then there was no way for you to know that you actually had 4096 bytes available, you would just have to keep resizing until it moved memory
<marler8997>
tgshultz, the previous API didn't allow the client to say how much data should be copied, so if an allocator needed to move memory, it would always copy the entire buffer. The new API allows the client to decide when to move memory and how much to copy
<marler8997>
the only other thing I see that could explain it is that I had to disable the "recursive async function" test, but I would be surprised if that one test took 30 minutes
<marler8997>
yes seems suspicious to me as well...that seems like to much performance gain
<andrewrk>
right that's the main purpose of marler8997's changes
<marler8997>
the big change here is that arraylist now takes advantage of the full capacity of allocations, if we really got this much performance boost from just that it will be incredible
<marler8997>
sourcehut took 30 minutes, not sure how to see build times for other runs on sourcehut, anyone know?
<marler8997>
oh wow, and the dron CI only took 29 minutes, it usually take around 1h 15m !!!
<marler8997>
not sure if this is correct, but the Linux Azure build with the new allocator interface only took 1h 25m, it seems like before it was consistently taking over 1h 55m
<mq32>
marler8997: i thik it's just a quirk in the semantic analysis
<marler8997>
if we restricted "return" to only be a statement (not an expression) then we could make return return x a syntax error, but then you wouldn't be able to return in the middle of an expression
<marler8997>
yeah I get "unreachable code detected" with return return x
<marler8997>
I would expect the type of the expression "return x;" to be "noreturn" (ironically)...so I wouldn't expect return return x; to pass semantic analysis...is this the case?
<marler8997>
andrewrk, new allocator interface is done, PR still needs work but the interface should be done if you want to take a look
2020-06-24
<andrewrk>
marler8997, sounds good to me
<marler8997>
I think idea #1 works, if that sounds good to you I'll go with that
<marler8997>
I take that back, you have to lookup the capacity again
<marler8997>
then again, performaing an in-place resize that's already reserved should be a simple addition
<marler8997>
same issue we discussed when returning the full capacity in allocFn
<marler8997>
that does seem like a disadvantage
<marler8997>
ok thinking more...
<marler8997>
oh
<marler8997>
idea #1 doesn't have this problem
<marler8997>
because the code will be using the full capacity buffer, it would be easy to acciddently pass the full buffer back into another allocator function
<marler8997>
it would need to track the requested length, and the full capacity returned by allocFn
<marler8997>
so something like array_list
<marler8997>
I'm imagining code that does want to take advantage of the full buffer
<andrewrk>
marler8997, e.g. allocAllowExtra(Foo, 10) internally would call the resize function to track more Foos before returning
<andrewrk>
marler8997, not sure I follow- if you used Allocator.alloc API, it would ignore any extra capacity returned, so there would be no "full buffer". if you called the advanced function, and got a bigger buffer, that's what you'd be tracking, before you passed it to something
<marler8997>
I think I'm leaning towards #1...still thinking though
<marler8997>
programmers could easily pass in the full buffer instead, violating the requirement that all buffers track the requested size
<marler8997>
that function might resize it
<marler8997>
for idea #2, imagine you allocate a buffer and pass it to a function
<andrewrk>
marler8997 over here going on the hero's journey
<marler8997>
in the meantime I can work on adding the return sizes back into allocFn and resizeFn
<marler8997>
ok
<andrewrk>
marler8997, yeah that sounds good, and I have another idea to solve the other problems you ran into, just give me minute to formulate it
<marler8997>
so...I'll modify allocFn/resizeFn to return full capacity, sound good?
<marler8997>
maybe there's a way to modify the API to not need capacityFn, if not, then I suppose we can keep it
<marler8997>
I'll keep capacityFn in for the time being
<marler8997>
but yes, now that I see an example I think we should support returning the full capacity from allocFn
<marler8997>
We'd have to change resizeFn to return the size as well
<marler8997>
makes me want to go back to the original design without the capacityFn
<marler8997>
thinking...
<marler8997>
so that one has to perform some sort of lookup to get the full capacity?
<marler8997>
but haven't seen one yet
<marler8997>
I did have the same thought though, i was looking out for allocators that might take advantage of that
<marler8997>
well the work to get the capacity is just an align
<marler8997>
We dont' have an allocator that does that yet
<marler8997>
possibly
<andrewrk>
marler8997, just musing here, I wonder if it would be better perf to also return the capacity from the alloc function, since the data is probably available in the implementation. could potentially prevent doing the lookup based on slice twice, in the case that the client wants the capacity
<marler8997>
ok
<andrewrk>
marler8997, that makes sense to me
<marler8997>
I don't think adding that extra requirement will really be an issue
<marler8997>
sounds like we need to accept any size between the requested size and the full capacity?
<marler8997>
when allocating an array of items (larger than 1) and the full capacity isn't divisible by the element size
<marler8997>
ok found an issue with requiring that the buffer size be either the requested size or the full capacity
<marler8997>
my immediate answer would be that errors trigger control flow like "try" and "errdefer"
<marler8997>
it will
<marler8997>
will be a simple change
<marler8997>
yeah I can do that
<marler8997>
so the semantics you proposed yesterday
<marler8997>
yes, they can pass in a buffer with the size they requested, or the full capacity
<marler8997>
I removed the length from the return value of allocFn and resizeFn, the client already knows they size they requested, and the allocator already needs to support capacityFn, so it made sense to remove length from the other functions, it actually made alot of the code simpler as well
<marler8997>
I saved the branch that doesn't include the capacityFn function as well
<marler8997>
I started explaining the issues yesterday but you didn't respond
<marler8997>
andrewrk, I ran int a few places where code needed to know the full capacity of an allocation that didn't have access to the allocFn return value
<marler8997>
yeah that'll be good
<marler8997>
to prevent this from happening in the future, zig could host it's own msys packages
<marler8997>
andrewrk, I ran into a few issues with the latest proposed memory allocator interface. I was able to address them by adding an additional function to the interface to get the full capacity of an allocation.
<marler8997>
I think it's this, "dash" use to depend on "msys2-base", and "filesystem" provided "msys2-base". But the new version of "filesystem" does not provide "msys2-base" and the new version of "dash" does not depend on "msys2-base". But pacman tries to update "filesystem" before dash, so it fails because it breaks the "dash" dependency on "msys2-base"
<marler8997>
looks like that pipelines xml you mentioned in the issue has the exact same issue
<marler8997>
don't thank me yet, we'll see if I can get it to work first :)
<andrewrk>
marler8997, oh man thank you so much for doing that, I was putting that off today
<marler8997>
this issue also comes up during realloc, first we try to resize in place, but if that fails, it may return a size larger than what was passed in, so we have to restore the original size, but we don't know the original size
<marler8997>
failing_allcoator keeps track of the memory allocated and memory freed, and validates it has all been allocated and freed
<marler8997>
another place where this can matter, is when invalidating memory
<marler8997>
one way to solve this, would be to add an allocator function, getFullSize, instead of returnning the full size from allocFn
<marler8997>
for example, in failing_allocator.zig, when a buffer is freed, it doesn't know whether the buffer being passed in is the requested size of the full size, so it doesn't know how much memory is actually being freed
<marler8997>
I'm seeing an issue with the latest allocator proposal, one is that there are some places where a client needs to get the full size of an allocation when they possibly have a partial size
<shakesoda>
marler8997: i did
<marler8997>
shakesoda, you missed andrew's last twitch stream :)
<marler8997>
andrewrk, I don't consider it push back, I consider us all pushing forward together
<andrewrk>
marler8997, I know you just put a lot of work into this, I'm sorry to push back again. But I really think this is going to make zig's "you have to think about allocators now" approach more palatable for people in general
<marler8997>
andrewrk, I'll work on that proposal
<marler8997>
andrewrk, yes I think it's a good proposal
<marler8997>
pixelherodev, do you have another idea?
<marler8997>
so if they need to reslice it to a value in the middle, they need to call resize
<marler8997>
can they pass in a size in between the 2?
<marler8997>
that might work
<marler8997>
right
<marler8997>
there are some allocators that already track the exact size (like the C allocator)
<marler8997>
I decided to punt and say that either could track it, it's up to the caller
<marler8997>
the questio is, when you need a slice of an allocation, who tracks the size, the allocator, or the client?
<marler8997>
I think the current InexactAllocator accomplishes that goal
<marler8997>
that would work for something like mmap
<marler8997>
oh, requested or actual size
<marler8997>
so mmap wouldn't know there is a 3rd page to unmap
<marler8997>
but I thought the proposal was, the client can slice the allocation without telling the allocator
<marler8997>
is that right andrew?
<marler8997>
not that you sub-slice the buffer without calling resize?
<marler8997>
oh, you're saying that you call resize
<marler8997>
if you allocate 3 pages, and resize it to 2 pages, you leak the 3rd page
<marler8997>
andrewrk, actually no
<marler8997>
oh, allowing resize to accept a sub-slice does put the burden on the allocator, I misread what you were saying
<marler8997>
this doesn't increase the burden on allocators
<marler8997>
if you request 10 bytes, but you get 4096, you want to know you actually have 4096 bytes
<marler8997>
so that clients that want inexact allocations (access to the full allocation) can have it
<marler8997>
so then we don't have 2 interfaces
<marler8997>
to allow clients to reslice before freeing
<marler8997>
pixelherodev, that was the proposed interface
<marler8997>
if you allocate 3 pages with mmap, and resize it to only be 2 pages when you free, then that 3rd page will leak
<marler8997>
mmap kinda
<marler8997>
pixherodev, yes
<marler8997>
and alot of code requires that behavior
<marler8997>
some allocatos cannot resize things to be exactly a particular size
<marler8997>
but the problem is, that allocators cannot resize allocations to be smaller
<marler8997>
tgshultz, yes
<marler8997>
New ideas are very much appreciated, and I happy to discuss
<marler8997>
thanks, and I'm not claiming I know all the answers
<andrewrk>
marler8997, you've clearly put a lot of work into this, I just want to make it clear that work is appreciated, and your observations are valuable, having done that work
<marler8997>
but I suppose I will explain it again
<marler8997>
*chat
<marler8997>
I already explained this example in the cat
<marler8997>
I answered your question
<marler8997>
what do you mean so?
<marler8997>
You asked what inherintly inexact meant
<marler8997>
if you request 10 bytes, it will reserver 4096 (hence the name inexact)
<marler8997>
because it's minimum allocation size is 4K, so if you request anything less than 4K, it still needs to reserve/allocate 4K
<marler8997>
meaning it can only allocated certain sizes of allocations
<marler8997>
mmap is an example
<marler8997>
this is a disaster of an interface
<marler8997>
which means that buffers returned by std.mem.Allocator, would only be able to pass them back to some of the functions, but not others
<marler8997>
you can't mix the 2, because an inheritnly inexact allocator would need additional metadata to track the exact sizes of inexact allocations
<marler8997>
the problem came up when I provide functions to allocate "exactly" and functions to allocate inexactly
<marler8997>
I started with that
<marler8997>
> Clients should never use anything other than std.mem.Allocator
<marler8997>
> There shouldn't be multiple allocator interfaces...I'm not sure where this rule comes from or what it means exactly
<marler8997>
to make then exact
<marler8997>
A way to share code between inexact allocators
<marler8997>
You can think of InexactAllocator as an internal detail if you want
<marler8997>
If you don't care about inexact allocations, you can forget about it and just use ExactAllocator
<marler8997>
The InexactAllocator already supports that rule
<marler8997>
so the interface is less flexible, because it only allows the allocator to manage the exact size
<marler8997>
Either in the allocator (using InexactToExactAllocator) or just storing the exact size itself
<marler8997>
by supporting an InexactAllocator, the caller can decide where to store the exact size if it is required
<marler8997>
that puts the burden on the allocator to track the size if it requires it
<andrewrk>
marler8997, another idea: "The length of `buf` is a hint to the allocator, to help it find the allocated block more efficiently. The hint may be too small, but too large is illegal behavior. === new part ===> If a given allocation returns the exact size requested, the hint must be exactly correct."
<marler8997>
And code using allocators, could decalre which kind they need, ExactAllocator or InexactAllocator
<marler8997>
but the thing is, some allocator inherintly support exact allocations (the C heap allocator) and some do not. If you do not, you would implement the InexactAllocator interface, if you do, you would support the ExactAllocator interface.
<marler8997>
it's what arraylist would be using for example
<marler8997>
I think having an inexact allocator interface is pretty useful in general
<marler8997>
maybe
<marler8997>
at minimum, I would expect most code to use InexactAllocator, and an ExactAllocator would be an one-off thing someone could use
<marler8997>
one thought is we may be able to remove the ExactAllocator once clients' logic change to support inexact allocations
<marler8997>
this allocator stuff has been kicking my butt
<andrewrk>
marler8997, I see, thanks - let me think about that
<marler8997>
but the mmap allocator would not
<marler8997>
the c allocator would be fine with that
<marler8997>
and then only passed a 4K buffer when freeing it
<marler8997>
but the caller changes it to 4K
<marler8997>
say you allocate a 16K buffer from mmap
<marler8997>
for example
<marler8997>
yes, because allocators may need the exact size
<marler8997>
the interface allows for that
<marler8997>
yeah
<marler8997>
I have a few different branches with different attempts
<marler8997>
using 2 different types ended up being alot simpler, but I'm definityl open to other ideas
<marler8997>
I tried that
<marler8997>
we could solve that by prefixing all the functions with something like exact... and inexact...
<marler8997>
so if you allocate an inexact allocation, you can't pass it to exact allocation functions
<marler8997>
is that you can't mix allocations
<marler8997>
the touble I had with mixing the two
<marler8997>
so functions could indicate whether the allocator they use needs to be exact or not
<marler8997>
so this design stems from wanting the type system to be able to distinguish between exact and inexact allocators
<marler8997>
I started out that way
<marler8997>
it could
<marler8997>
and vice versa
<marler8997>
you can't take an exact allocation and pass it to an inexact allocation function
<marler8997>
so for page allocator, exact and inexact allocations cannot be mixed
<marler8997>
thinking...
<andrewrk>
marler8997, couldn't std.mem.Allocator have all the functions of both ExactAllocator and InexactAllocator? I'm confused why there is a need for separation
<marler8997>
alexnask, thanks
<marler8997>
anyone know where that proposal is for custom integer type ranges?
<marler8997>
I recognize the first one, I can't remember exactly where though
<marler8997>
which reference "See, it's a play on words, and that's where the humor comes in"?
<marler8997>
it gets so unfunny, the joke overflows...that's why Zig was made, to protect from that
<marler8997>
andrewrk, great
<andrewrk>
marler8997, I'm having a look now!
<marler8997>
they must have turned the switch in the last few minutes
<marler8997>
whoa, did github just get a new look?
<marler8997>
andrewrk, it took me a few attempts and some time to think, but I think I've come up with a way to start integrating the new allocator interface. I've updated my PR when you get a chance to take a look: https://github.com/ziglang/zig/pull/5064
<marler8997>
andrewrk, yes that's what I thought, my explanations on reddit may not have come accross that way
<andrewrk>
marler8997, btw `nosuspend` does not modify the behavior of suspend points - it's actually an *assertion* that there will be no suspension at runtime
<marler8997>
I created it to be very minimal, I think could be a good starting off point
<marler8997>
andrewrk, if you like I could work on getting an initial http client into the standard lib, based on my ziget library https://github.com/marler8997/ziget
<marler8997>
yeah feel free to use it however you like
<andrewrk>
marler8997, nice, that looks handy. looking at the build setup it makes me want to hurry up and get the package manager done
<marler8997>
I've ported my python zig update tool to zig
2020-06-13
<greenfork>
marler8997_: they are strings
<marler8997_>
greenfork: out of curiosity are the keys strings or some other type?
<marler8997_>
for now I'll just make it work without referencing the dependencie's build.zig file
<marler8997_>
that's alright, I think in the end, the zig build system would probably download the dependency into the zig-cache
<marler8997_>
but now I'd like to take the settings from it's build.zig file, so I don't have to duplicate them
<marler8997_>
In the past, I just add the dependencies "index file" (the main zig file) as a package
<marler8997_>
Maybe you have a better suggestion for how I can use a dependency from another repo?
<marler8997_>
build.zig is in another repo, which I'm assuming is outside my current repo, so it doesn't allow me to import because it's outside my package path
<marler8997_>
Hmmm, this may not be the right way to accomplish what I'm doing
<marler8997_>
ok I'll play with that
<marler8997_>
anyone familair with invoking build.zig files from other build.zig files, is that a thing?
<marler8997_>
but now when I'm using that library in another project, even though I'm setting the "ssl" package, the modules in my http client project cannot find the "ssl" package
<marler8997_>
In my http client project, I use package path to set the "ssl" package to either an openssl or nossl variation
<marler8997_>
I'm having an issue with package-path, not sure if it is a bug or not
2020-06-03
<marler8997>
found an ownership bug deep in rational.zig...what a headache
<marler8997>
ah new allocator found bug in std.fs.path.zig
<marler8997>
yeah that's what I ended up doing
<marler8997>
easy way to turn a comptime_int into a string at comptime?
2020-06-02
<marler8997>
ok
<marler8997>
I think it would be easy to make it a separate change, don't know for sure though
<marler8997>
so it sounds like a way to set memory to "undefined", which in debug mode should cause access to panic/assert in some way
<marler8997>
that would make sense
<marler8997>
it's being used on newly allocated memory
<marler8997>
and/or what it's doing
<marler8997>
anyone able to explain when @memset(..., undefined, ...) should be called?
<marler8997>
ok cool
<andrewrk>
marler8997, makes sense to me
<marler8997>
well this is just the allocator interface, most programs wouldn't call it directly
<pixelherodev>
marler8997: why's that? What if it fails?
<marler8997>
andrewrk, one modification, I think it would be better to have resizeFn just return usize instead of Error!usize
<marler8997>
something we could decide to add if needed
<marler8997>
yeah an optional reallocFn could solve that issue
<marler8997>
c's heap is a bit cripped because of it
<marler8997>
yeah
<marler8997>
c heap doesn't suppor tit
<marler8997>
fengb...nope
<marler8997>
andrewrk, that's one way to do it, that's fine with me
<andrewrk>
marler8997, I think it could be used heuristically. e.g. it would determine whether or not to call realloc. we'd still have to check realloc return value to see if it moved the ptr and in that case free it and return failure
<marler8997>
andrewrk, oh I've never heard of that, looks like that would alleviate not being able to call realloc
<andrewrk>
marler8997, hmmm. good point. maybe c_allocator could take advantage of malloc_usable_size ?
<marler8997>
however, this interface doesn't have a path to call C's realloc
<marler8997>
yeah that was a nice bonus
<andrewrk>
marler8997, I like that we won't be pointlessly passing alignment for reallocs. it's a nonsense use case to try to increase the alignment of something
<marler8997>
I'll get to work on it today
<marler8997>
it was something I already was thinking of
<marler8997>
yeah
<marler8997>
it would
<marler8997>
but will require some work to get that interface to work with current clients who expect the length to be exact
<marler8997>
this fits my 5151 proposal more closely as well
<andrewrk>
marler8997, yeah, and note that the higher level abstractions can still be built upon this proposed interface
<marler8997>
andrewrk, removing the requirement that lengths have to be exact will making the allocator simpler
<marler8997>
For them I wrote a tool that allows you to call managed code from a native process written in D. Was thinking of supporting Zig as well if the need ever comes up, but it might not.
<marler8997>
There's a few, a game developer, some investments companies and other tech/tooling companies
<marler8997>
With a company that uses D (not too many), but I'm moving away from the language as I see where the language is going
<marler8997>
Did it for about 6 months, was a good experience
<marler8997>
I'm stepping away from my second job to have more time to work on personal projects and Zig. If you have other directiosn you need help with let me know as well