<cshenton>
Are you guys open to contributions to std.meta? Or are things a bit too unstable at the moment?
cshenton has quit [Remote host closed the connection]
cshenton has joined #zig
Aransentin has quit [Remote host closed the connection]
adamkowalski has joined #zig
adamkowalski has quit [Ping timeout: 268 seconds]
cshenton has quit [Remote host closed the connection]
adamkowalski has joined #zig
<adamkowalski>
cshenton: awesome thanks!
<adamkowalski>
do you know how to handle the case where you get a slice instead? I noticed the type info says it's a pointer and the child was a void
cshenton has joined #zig
<daurnimator>
cshenton: we're happy for contributions anywhere :P
<cshenton>
Ah so that's where slices are. So the Pointer field of the TypeInfo enum has a field size that records the pointer type, so I can ad that case in.
<pixelherodev>
z
<pixelherodev>
Whoops - sorry about that!
<adamkowalski>
cshenton: yeah I think we went down the same path. I started out with something that looked like what you had since I took inspiration from how expectEqual was implemented, but I coudln't figure out how to match on slices
<daurnimator>
pixelherodev: z is for zig?
<cshenton>
Matching on slices isn't to bad, you just add .Pointer to the switch statement and check the contents in the block. Let me slap that together and if it looks good I'll PR it.
<adamkowalski>
Well before you do that, we should talk about eltype
<adamkowalski>
i'm a little concerned that for array the child is called type
<adamkowalski>
for pointer it's called child
<adamkowalski>
and now we are introducing eltype
<adamkowalski>
I think we should keep consistency if possible
<cshenton>
I think that's reasonable.
<cshenton>
eltype is just what julia uses in its stdlib, so I defaulted to it, consistency is more important
<adamkowalski>
ah you are a julia user too?
<adamkowalski>
awesome, I could use your help on a project i'm working on
<cshenton>
Also would you consider [*]T a valid container type? Or just arrays and slices?
<adamkowalski>
i'm building out a reverse mode automatic differntiation package
<adamkowalski>
have you used TensorFlow, PyTorch, or Flux
<cshenton>
Yup, used all of them
<adamkowalski>
Awesome, I'm currently implementing a library called compute graph, so it's similar to tensorflow v1 and it's a static graph approach
protty has quit [Remote host closed the connection]
<adamkowalski>
I figured that way we can get distribution and parallization by analyzing the graph
<cshenton>
Ah nice, you should also check out cpp-taskflow for inspiration.
<adamkowalski>
i'm still working out exactly how to make everything work, but can you take a look at this test case and see what you think of the API so far?
<cshenton>
The internals are much more readable than tensorflow
<adamkowalski>
I may have blatently stole the names from tensorflow, but hopefully it makes reading the code easier
<adamkowalski>
I'm currently working on expanding from just working on scalars to working on tensors, hence the question I had earlier haha
<cshenton>
This is a good start but I can think of ways of improving it. If each op returned both an op-id and a reference to the graph, you could chain operators:
<cshenton>
like graph.mul(a, b).add(c); or something
<cshenton>
Or you could have that classic split and offer both like `c = Constant(&graph, 2.0)` or `c = graph.add_constant(2.0)`
<cshenton>
A lot of this is a matter of taste though.
<adamkowalski>
the reason why I dind't want to make the methods be on the graph itself is because then there will be some number of "blessed functions"
<adamkowalski>
you can't add new methods of your own that would be callable like that
<cshenton>
Yep, that's a good point.
<adamkowalski>
so I figured the consistency would be nice
<adamkowalski>
unless we have ufcs
<cshenton>
I was under the impression that wasn't likely to happen.
<adamkowalski>
for constant I was going to use the ArrayInfo based on the var thats passed in and I will automatically construct the appropriate tensor
<cshenton>
Nice
<adamkowalski>
Since I don't have something like numpy or julias native array type, I also need to build a eager tensor which is the thing that will actually get passed into your operations
<adamkowalski>
I'm not sure if there is anything I can do about that virtual call I have for the operation while maintaining the ability for users of the library to add custom operaitons
<adamkowalski>
if you have any ideas i'm all ears
<cshenton>
So tensorflow handles user-defined c++ ops with some macro magic, and python ops (which are more subgraphs) with inheritance.
<adamkowalski>
But all the actual eager tensor ops will be done using static compile time polymorphism and only the graph needs to be runtime polymorphic
<adamkowalski>
at first I tried giving the graph itself an element type to get away from that
<adamkowalski>
but you need to be able to injest different kind of data
<adamkowalski>
so now i'm going to use a union(enum) over all the different primitive types that the cpu offers
<adamkowalski>
that way you can have strings and one hot encode them to turn them into vector space representations and use word embeddings
<adamkowalski>
but the thing I could really use help with is implementing the backprop
<cshenton>
Yeah managing the allocations for that will be a bit awkward.
<adamkowalski>
I was going to have a backward method on each operation which will be responsible for managing that
<adamkowalski>
and you can chain ops that have backward defiend
<adamkowalski>
the allocations are all in the arena that the graph uses
<adamkowalski>
then when you're done with the graph you deallocate the whole arena at once
<adamkowalski>
the session uses it's own arena allocator as well, so the actuall values needed to compute the node will be ephemeral
<cshenton>
Tensorflow's GPU backend will probably have something like that, I know tensorflow dominates your GPU memory, so they probably use an arena.
<adamkowalski>
the actual backprop computations are pretty tricky haha. I might study how ChainRules.jl works to find out the derivative of everything is
<cshenton>
Even if you're using a static graph, I think you'll make your life easy to use dual numbers.
<adamkowalski>
dual numbers are great for forward mode auto diff
<adamkowalski>
reverse mode auto diff scales better if the number of outputs is small but the number of inputs is large
<adamkowalski>
I think we should have both though
<adamkowalski>
what are your thoughts on graph mode anyway?
<cshenton>
So in terms of cutting edge stuff there's a push towards autodiff being a language level thing.
<cshenton>
i.e. autodiff happens on IR inside the compiler
<adamkowalski>
right like Zygote.jl
<cshenton>
pytorch are doing this with torchscript, and julia with zygote
<cshenton>
yup
<adamkowalski>
torchscript and tf.function both seem to be just jit compiles though
<adamkowalski>
it seems like the trend moved away from graphs, but now we're trying to add them back, just under the hood
<adamkowalski>
julia is a HUGE pain to deploy
<adamkowalski>
and deploying tensorflow/pytorch is all pretty much getting a trace of your program -> creating a graph -> exporting and reading in C++
<adamkowalski>
so I figured let's just use the graph in the first place haha
<cshenton>
That's not unreasonable, this split between what's convenient for research vs. deployment is pretty fundamental.
<adamkowalski>
right, unless we can get andrewrk to help us build something like this into the compiler itself
<adamkowalski>
but I think the explicit nature of this approach has advantages
<adamkowalski>
each time you create an operation it allocates, which may fail, so you need to "try" it
<adamkowalski>
and executing the graph allocates so same goes
<adamkowalski>
also since shapes are known at runtime (for now) they may mismatch, which may fila
<adamkowalski>
fail*
<cshenton>
So I've seen teams attempt to roll their own graph autodiff framework before.
<cshenton>
In java
<cshenton>
And they really struggled because they sort of went halfway on ergonomics and performance and got the worst of both worlds.
<adamkowalski>
I care a lot about reliability, and scalability. I think that matters much more then prototyping. Plus I think it's not that bad to write models even with this explicit API
<adamkowalski>
I just forces you to think about allocation, failure, and types
<cshenton>
For sure
<adamkowalski>
We have all these models currently written in Python and it's a huge pain to refactor
<cshenton>
So the place where current frameworks fail is definitely things like compilation / startup time.
<adamkowalski>
there is no compiler so when you add a parameter to a function or remove one, how do you find all the places in your code that it is used
<adamkowalski>
we just hope our unit test catches it
<cshenton>
The push towards dynamic graphs is really about improving developer iteration times.
<adamkowalski>
right, but I disaggree with the premise
<adamkowalski>
I think that then you end up with a bunch of code which maybe you got out the door slightly faster
<adamkowalski>
but just like dynamic typing vs static typing, the tracer baced approaches are harder to maintain in my opinion
<adamkowalski>
development slows down as you increase the number of lines of code, and intent is lost because you don't have the documentaiton that types / shapes can provide
<cshenton>
I agree, my point was that a graph computation framework in zig would have that as a key value-add.
<cshenton>
Good iteration times without things being hard to debug at scale.
<adamkowalski>
I hope so! hopefully as I work on this more and more you and others can keep reviewing the code and making sure you like the direction
<cshenton>
I'd be keen to keep an eye on it, I'm using zig for games engine stuff primarily, but I used to do a lot of probabilistic programming stuff so I'm quite familiar with these kinds of weird scientific computing use cases.
<adamkowalski>
probabilistic programming is definitily a huge use case. being able to backprop through a distribution is really important
<adamkowalski>
in tensorflow that was a huge pain, but you could solve it with the reparametization trick
<adamkowalski>
i don't think you can do that in Julia yet either
<cshenton>
Yeah tensorflow-probability fixed a lot of that.
<adamkowalski>
true, honestly I like a lot of things about tensorflow, I just don't like that it's in Python haha
<cshenton>
I personally think the pytorch c++ apis are really nice. They factored them out really well.
<adamkowalski>
I think we can do better though
<cshenton>
Well the difficulty with backprop through samplers / likelihoods is that writing them in autodiff friendly form is awkward.
<adamkowalski>
they use shared pointer everywhere
<cshenton>
oof
<adamkowalski>
it's reference semantics which doesn't make sense in C++ which is so value oriented
ur5us has joined #zig
<adamkowalski>
They claim it's because it helps the Python data scientists transition
<adamkowalski>
another use case we want to support is differential equations, just like Julia. It works so well with Flux
<adamkowalski>
but first thing is first. I just want a simple "dense/linear/affine" layer that I can backprop through and use that to solve MNIST
<adamkowalski>
then write a simple article about why Zig for machine learning, and get people excited and contributing
<adamkowalski>
then we can do convolutions, lstms, and attention mechanisms
<cshenton>
One thing to keep in mind is that the existing frameworks are the way they are not because the engineers were bad.
<cshenton>
But because the users want a particular interface.
<adamkowalski>
oh no I definitely agree with that, I think everyone working on those projects is doing an amazing job
<cshenton>
So I guess the question is, who are the users who want this kind of API for graph computation? It might be they're in a particular field, so focussing on features in that field would be the way to go.
<adamkowalski>
I think we have the benefit of hindsight and not needing to be backwards compatible though
<adamkowalski>
I think the users of this API want something that is meant for production rather then research
<adamkowalski>
however, I am hoping the API will be pleasant enough that you can still use it for research
<adamkowalski>
i'm not sure there must be this clear distinction, I guess you could just argue it's slightly less ergonomic
<adamkowalski>
but it will make sure that you handle all possible error conditions, and when you need to extend it, you don't need to drop into a lower level language
<adamkowalski>
finally you can deploy it without needing swift, javascript, java, etc as you can just use web assembly or if you target the C ABI you can have much greater reach
cshenton has quit [Remote host closed the connection]
adamkowalski has quit [Quit: Lost terminal]
dddddd has quit [Remote host closed the connection]
ur5us has quit [Ping timeout: 260 seconds]
dingenskirchen has quit [Ping timeout: 248 seconds]
return0e has quit [Read error: Connection reset by peer]
return0e has joined #zig
darithorn has joined #zig
cshenton has joined #zig
<cshenton>
adamkowalski: One thing you could do is to target your zig library as a backend for https://github.com/tensorflow/mlir or https://onnx.ai/ or even the protobuf serialisation of tensorflow's internal graph format.
<cshenton>
Because then you could have a user story like: transition your existing models using this path, and start writing new models with the front end inside the library itself.
<cshenton>
Either way, I think it's a compelling use case. Even a replacement for the tensorflow c++ api that isn't a nightmare to build would be a huge value add for a lot of use cases.
<leeward>
C is good for defeating its own purpose.
<mq32>
oh yeah
<leeward>
Though that's machine code, not assembler language.
wilsonk has quit [Ping timeout: 268 seconds]
<leeward>
/pedant
return0e_ has quit []
<fengb>
mq32: are you casting a string literal to a function pointer?
<mq32>
fengb: yeah, so? :D
<mq32>
that's called "inline machine code" :D
wilsonk has joined #zig
<Snektron>
That shouldnt work
<Snektron>
Unless you have some system without virtual memory that is
BitPuffin has quit [Quit: killed]
D3zmodos has quit [Quit: killed]
aperezdc has quit [Quit: killed]
Snektron has quit [Quit: killed]
fengb has quit [Quit: killed]
Demos[m] has quit [Quit: killed]
vegai has quit [Quit: killed]
AlexMax has quit [Quit: killed]
Demos[m] has joined #zig
return0e has quit [Read error: Connection reset by peer]
return0e has joined #zig
Snektron has joined #zig
<Snektron>
How long are static builds hosted on ziglang.org/builds?
<Snektron>
apart from major versions
BitPuffin has joined #zig
dtz has joined #zig
mattmurr has joined #zig
D3zmodos has joined #zig
AlexMax has joined #zig
fengb has joined #zig
vegai has joined #zig
mahmudov has quit [Ping timeout: 260 seconds]
hoppetosse has joined #zig
mahmudov has joined #zig
<leeward>
Snektron: Why wouldn't it work with virtual memory?
<hoppetosse>
would it be feasible to have the compiler output the actual active field on "access of inactive union field" safety check?
<hoppetosse>
*not compiler, stacktrace
darithorn has joined #zig
<fengb>
leeward: most OSs mark data separately from executable code so that’d most likely segfault when trying to execute data
<leeward>
fengb: Oh, so not virtual memory.
<leeward>
Yeah, I know about NX bits.
<leeward>
It's more of a pain, but NX can be cleared on .bss.
<leeward>
Or whatever page table entry holds .bss, anyway.
lygaret has joined #zig
adamkowalski has joined #zig
<leeward>
Whoo, LoggingAllocator really doesn't work, does it?
clktmr has left #zig [#zig]
leeward has quit [Remote host closed the connection]
leeward has joined #zig
<lygaret>
hey, if I have a slice of bytes, of a known length, is there a way to cast that to an array of the same length? I'm trying to use `std.mem.bytesAsValue(Header, &data[8..16])`, but I'm getting a compile error of: `error: expected *[N]u8 , passed [*]const u8`
<lygaret>
er: `expected *[N]u8 , passed *[]const u8` (prev was from trying `data[8..16].ptr`)
<lygaret>
it seems I'm missing something, but I'm having trouble figuring out why these wouldn't be the same type?