2021-02-27

<marler8997> there currently are no keywords
<ifreund> marler8997: was just looking at the readme of stitch again. I'm not a huge fan of the fact that variables are prefixed with the same symbol ($) as keywords when referencing them. IMO it would be nice to use a different symbol to distinguish them from keywords
<marler8997> compiling openssl statically does make the ziget binary go from about 800 KB to 9 MB :)
<marler8997> took a couple hours to get the build.zig file together to get just the right set of source files
<marler8997> g-w1, I got ziget to compile OpenSSL statically from source!
<marler8997> I must say, I like how clever our buildpkgs solution is, however, when I think about how long it takes to explain it to people and compare that with @tryImport I'm less enchanted
<marler8997> yeah I was half joking he would do that, but also half expecting him to :)

2021-02-26

<marler8997> actually, I could generate buildpkgs-PKGNAME.zig
<marler8997> I was thinking each package would get it's own
<marler8997> right, trying to figure out how to generate buildpkgs.zig
<marler8997> hmmm
<marler8997> oh...
<marler8997> got it
<marler8997> so, is there one Compilation object for the full package set?
<marler8997> and each package has it's own sub-package set
<marler8997> I'm going to have to go back to the @import("buildpkgs") instead of using std.builtin because we're only compiling one version of builtin.zig per compilation (not one per package)
<marler8997> yeah, still working on a bug though
<andrewrk> marler8997, that was fast
<marler8997> I've got multi-level build.zig dependencies worked out now, with a test in the PR!
<marler8997> there's a "run.zig" file which pretty much just takes a command to run, so build.zig builds run.zig as an exe and passes the command for it to run, then calls it
<marler8997> daurnimator, I think I found a way around your idea to add a whitelist for running external commands in build.zig that I'm not sure how you could plug
<marler8997> g-w1, I had an idea to put the packages in builtin.zig instead, I was able to leverage more existing code, I've updated the PR if you want to have a look
<marler8997> union(enum)
<marler8997> g-w1, look at this beauty...even tests that not using has with comptime will create a compile error: https://github.com/ziglang/zig/pull/8072/commits/aec0f135356de6878fd90d5466236c0a4c407e54
<marler8997> I should write a test that ensures this causes has without comptime to fail
<marler8997> actually that might make it always return true?
<marler8997> maybe I should be marking the call to isComptime as comptime
<marler8997> right
<marler8997> why would it have a compile error every time?
<marler8997> seems to work
<marler8997> there's the commit that now forces the user to specify "comptime" when the call has
<marler8997> g-w1, it's not a pretty solution, but at least now the function will always fail when comptime is not specified
<g-w1> marler8997: i think the inline thing with comptime behavior is implimented in stage2 (not 100% sure tho)
<g-w1> marler8997: exploit peer type resolution ftwpub fn isComptime() bool {var t: bool = true; const x = if (t) @as(u7, 0) else @as(u8, 0); return @TypeOf(x) == u7;}
<marler8997> here's the issue where it looks like the plan is to use inline to enable this: https://github.com/ziglang/zig/issues/425#issuecomment-754572088
<marler8997> ooo, how to do?
<marler8997> then their build will work in some cases, and fail in others, which is what I'm trying to avoid
<marler8997> because if the caller forgets to specify comptime
<marler8997> but that's the problem I'm trying to avoid
<marler8997> g-w1, yeah that works
<marler8997> g-w1, callconv(.Inline) not enough, maybe the comptime block could do it
<marler8997> I will try
<marler8997> oooo, would that work?
<marler8997> buildpkgs only exposes the list of packages given by the user
<marler8997> I know there was a proposal about this as well
<marler8997> like, if comptime_bool existed that would be one solution
<marler8997> does anyone have a clever idea that would force hasPkg to be done at comptime, so the user doesn't accidently miss it and then their build.zig file won't work with a particular configuration
<marler8997> and I can't add a tryImport wrapper to hasPkg because Zig is planning on removing the ability to use a comptime string to call @import
<marler8997> @tryImport didn't have this problem because it returned ?type so it HAD to be compiled at runtime
<marler8997> maybe someone has an idea here, with our new build "hasPkg" design, hasPkg returns a bool so it's not automatically computed at comptime
<marler8997> when's the llvm12 transition going to happen?
<marler8997> I'll check there
<marler8997> it's good to get these features tested in some real world scenarios
<marler8997> if we can get it to support them, I'll probably add support for it in some of my projects, including ziget/zigup
<marler8997> anyone on here familair with how zigmod works, I wanted to discuss what it would take to get it to support dynamic build dependencies with the newly proposed feature
<marler8997> that's a common solution
<marler8997> Zig could also support a response file if they get too long
<marler8997> *chance
<marler8997> much less of a change of it becoming an issue for build.zig's direct dependencies
<marler8997> daurnimator, could be a problem for specifying pkgs on the project itself
<daurnimator> marler8997: I don't think --pkg-begin would work......... command line arguments can only be so long
<marler8997> it's just a simple list of packages and an interface to query them
<marler8997> actually it can be implemented today
<marler8997> actually I think it could be done completely independently now that I think about it
<marler8997> part 2 is pretty small, just adding a needed_package field to Builder, and a bit of logic in build_runner to print them and error if they are missing
<marler8997> that's part 2
<marler8997> of course the user could provide it as well
<marler8997> the package manager would provide --pkg-being to build.zig
<marler8997> PR for dynamic build deps part 1 here: https://github.com/ziglang/zig/pull/8072

2021-02-25

<marler8997> daurnimator, maybe we could try out your idea by supporting a comptimeBuild entry point
<marler8997> here's the proposal of our design: https://github.com/ziglang/zig/issues/8070
<marler8997> noam what's ur github name?
<marler8997> oh right, I remember that one
<marler8997> ikskuh what are you working on that's going to break build.zig? (out of curiosity)
<marler8997> everyone else as well, mikdusan, g-w1, ifruend, noam
<marler8997> thanks for the help ikskuh
<marler8997> this stuff is so tricky for our humans brains to think about
<marler8997> should rename buildimports to buildpkgs, since this only works for packages
<marler8997> link in a bit
<marler8997> I'm putting together a gist
<marler8997> So we don't even need #2206 to implement this!
<marler8997> all we do is provide a function that takes a comptime string with the package name and it only returns true for the packages is has
<marler8997> we don't need to scan build.zig
<marler8997> oh wait a second
<marler8997> so it's just extra steps to do @tryImport
<marler8997> oh wait no, this mechanism still needs to print the needed packages back to the package manager
<marler8997> yeah that sounds easy
<marler8997> the "issue implemented"?
<marler8997> if andrewrk, approves, then I'll be happy to make a PR for this
<marler8997> yeah, I'm totally fine with this solution as well
<marler8997> although, I suppose my proposed solution isn't much better
<marler8997> we are making the build process quite a bit more compelx
<marler8997> because it would work
<marler8997> I find this solution acceptable
<marler8997> this is actually functionally equivalent to @tryImport, it's just implemented through a dynamic module rather than supported in the general case
<marler8997> then we take those imports and create a module with comptime values for each import that indicates whether or not it has been included
<marler8997> before compiling build.zig, we scan it for all the imports it makes (we can limit it to package imports as that's the only ones I have a use case for)
<marler8997> so let me make sure I have this right
<marler8997> you might be onto something here
<marler8997> loading...buffering...
<marler8997> thinking...
<marler8997> look at the example I sent
<marler8997> knowing all the @import tokens doesn't tell us which packages are required
<marler8997> it's running the logic inside build.zig that tells us which ones it needs based on the command-line options
<marler8997> the problem isn't finding all possible imports
<marler8997> how would that work?
<marler8997> just by the title I don't see how we can use logic inside Zig to enable/disable the @import
<marler8997> this doesn't seem possible, where's this proposal?
<marler8997> i.e. can -Dandroid enable/disable an import?
<marler8997> and can these *potential* imports use build.zig's runtime logic to enable/disable them?
<marler8997> what do you mean by scan?
<marler8997> was that something you think might help with this problem?
<marler8997> ikskuh I'm not familiar with that proposal
<ikskuh> marler8997: have you seen andrews proposal on using string literals instead of comptime strings for @import() ?
<marler8997> we spent a bunch of time on it yesterday and almost had a solution but it ultimately fell through and we still needed @tryImport
<marler8997> well if you get any ideas let me know and we can discuss
<ikskuh> marler8997: not sure yet
<marler8997> ikskuh I'm all ears if you have alternative solutions :)
<ikskuh> marler8997: doesn't look too bad, even though im still opposed. no technical critique :D
<marler8997> I made the "optional dependency" "opt-in"
<marler8997> ikskuh have a look at the @tryImport example I put together: https://gist.github.com/marler8997/db2a97faad5c4d58b68689c5efc95787
<marler8997> there can always be --ignore-whitelist
<daurnimator> marler8997: uh, no different to what build.zig looks like today, except that build() would be called as `comptime build()`
<marler8997> daurnimator you don't have an example of what such a build.zig would look like do you?
<daurnimator> marler8997: the std libraries build.zig would have the default whitelist; then a build.zig would specify which (if any) additional commands it needs
<marler8997> yeah that's what I meant
<daurnimator> marler8997: or what do you mean?
<daurnimator> marler8997: project's build.zig
<marler8997> is this whitelist in the standard lib, or in the project's build.zig file?
<marler8997> for your orc idea, would this whitelist be a list in the zig standard library?
<marler8997> I'm still unsure whether it would be too restrictive
<marler8997> It's an interesting idea. You're sort of forcing people to make their build logic more correct
<marler8997> so at no point can you pass a function to the Builder that gets executed at runtime, interesting
<marler8997> ah
<daurnimator> marler8997: correct.
<marler8997> oh wait a second, does this pure Dag have no ability to add runtime code to it?
<daurnimator> marler8997: so I haven't figured this out yet, but I imagine you can add to the white list, so e.g. `zig build --allow-build-subprocess orc=/path/to/orc`
<marler8997> we're at runtime now, so anything is free to use the network/full filesystem/whatever
<daurnimator> marler8997: you create the DAG, and each DAG step has an action in it, e.g. "run zig build-obj" or "run zig build-lib"
<marler8997> you have a custom build step to call the orc tool, how to you prevent it from calling anything else?
<daurnimator> marler8997: what do you mean?
<marler8997> how do you restrict runtime like that?
<marler8997> you can still run subprocesses during the actual build
<marler8997> but that doesn't require DAG modification right?
<marler8997> like?
<marler8997> which is good, except if someone has a valid use case that doesn't allow this, would have to think if there is one
<marler8997> it makes the DAG more pure
<marler8997> yeah I definitely think that's worth consideration
<daurnimator> marler8997: subprocesses should be from a predefined list (essentially only `zig build-*` commands)
<marler8997> can't run subprocesses?
<daurnimator> marler8997: 1. can't reach the network. 2. can't read anything outside of build/src dir (e.g. my /etc/passwd file)
<marler8997> so what can't we do then?
<marler8997> oh true
<daurnimator> marler8997: sure it can, `@embedFile` is a thing
<marler8997> so you don't want build.zig to be able to read/process non-zig files to generate it's DAG?
<marler8997> like touch the filesystem (aside from imports)
<marler8997> give me an example of what we can't do at comptime
<marler8997> daurnimator, you're probably more familair with comptime restrictions, what benefit would be get by limiting build.zig to comptime?
<marler8997> with my example, compiling and querying build.zig could all be done in the download setp
<marler8997> daurnimator, like I said you could be right but I'd have to think about it
<marler8997> it might be confusing because zig's build.zig doesn't actually just do the build step
<marler8997> daurnimator, you could be right, I'm not sure, but my example allows for that flow
<daurnimator> marler8997: IMO a build/package system needs the following 'phases': `download` -> `unpack` -> `build` -> `package`
<marler8997> normals don't seem to relate, maybe some fellow programmers feel the same way?
<marler8997> I have this theory that I have a horrible memory, and I constantly have to relearn/revaluate everything I know. It makes me very detail oriented and I get lost if I don't understand everything.
<marler8997> people have been bringing new and interesting ideas and we've really made some progress
<marler8997> everyone's input on this has actually been very helpful
<marler8997> I'm just laser focused at the moment
<marler8997> lol, actually I think my memory is pretty shit
<marler8997> you did the same thing to me when I was talking about comptime build options so now we're even
<marler8997> lol
<marler8997> "the runtime thing" is kinda the whole thing :)
<mikdusan> daurnimator: notwithstanding the runtime thing marler8997 just mentioned, none of what I said precludes user-interaction required if we setup package manager defaults that way
<marler8997> if (b.option(bool, "android")) { @import("android"); }
<marler8997> and how do we know which imports we need? We only know this at runtime
<marler8997> how do we hook into the compiler to report missing imports, and not also cause other compile errors from the resulting type not being valid?
<marler8997> poor mans IPC
<marler8997> oh right, well that's what the --get-package-needs is
<marler8997> IPCs?
<marler8997> so long as we agree this use case is valid, if we can solve this without @tryImport I'd love to hear ideas
<marler8997> ok, here's a realistic example, with a step-by-step of what's going on: https://gist.github.com/marler8997/db2a97faad5c4d58b68689c5efc95787
<marler8997> what kind of server/client, for what purpose?
<marler8997> mikdusan, yeah I hadn't gotten that far in the comptime build options yet, but I got there eventually :)
<mikdusan> marler8997: yeah I was staring at my gist thinking `if (enable_llvm)` is comptime, I don't see how to handle missing package: https://gist.github.com/mikdusan/a4617066c986d46bd415916f6d0b2263
<marler8997> but shoot, I don't think comptime build options solves this, that's freakin lame
<marler8997> try kinda means "error if this fails" so it doesn't really match
<marler8997> I was also considering optionalImport
<marler8997> that's fine with me
<marler8997> in build.zig if (b.option(bool, "android")) { if (@tryImport("androidbuild")) |ab| { ... } else { b.needPackage("android"); } }
<marler8997> with @tryImport, what would happen is package-manger would compile build.zig, then it would run it to get the package list it needs
<marler8997> yeah I don't think this works
<marler8997> but then the imports will still fail
<marler8997> this means that the package manager MUST evaluate build.zig at compile time
<marler8997> yeah definitely, I'm trying to work through a potential solution
<marler8997> wait no that doesn't work, because the @imports will fail
<marler8997> build --zig-internal-list-packages
<marler8997> if it interacts with the build exe function, I wonder if there's a way build_runner could output the packages it needs?
<marler8997> is it interacting with the resulting build exe, or is package-manager re-evaluating the comptime build function?
<marler8997> ok, so then how does the package manager use build.zig?
<marler8997> Does it work the same way? you still compile a build exe and run it the same way?
<marler8997> how does the zig build flow work if build.zig is invoked at comptime?
<marler8997> today the flow is, compile build.zig, then run the exe to get inputs and outputs and run various build actions
<marler8997> like, when is the compiler running, how does the flow work here?
<marler8997> I'm trying to picture how the package manager and "zig build" would be using build.zig at comptime
<marler8997> but how does evaluating it at comptime help the package manager know what packages it needs?
<daurnimator> marler8997: if build.zig can be evaluated at comptime, then there's no reason you couldn't call it from any comptime code/context.
<marler8997> I guess the point you're saying it, builder is the result of running build from build.zig
<marler8997> are you sure that's correct?
<marler8997> groking...
<marler8997> just for clarification, this would mean the packages that build.zig uses would be in packages.json, but we wouldn't necessarily download them unless we determine that build.zig needs them
<marler8997> how does that tell the package manager which package it needs?
<daurnimator> marler8997: my proposal to g-w1 the other week was that `pub fn build()` would be evaluated at comptime
<marler8997> with the current package manager design, this would require the package manager to have some way to interact with build.zig to know which packages it is importing for any given comptime-configuration set
<marler8997> but it does allow build.zig to change what it needs to import based on the build configuration
<daurnimator> marler8997: that's good IMO
<marler8997> this one doesn't give build.zig any opportunity to try to handle a missing package
<marler8997> yea definitely not equal
<mikdusan> marler8997: the solutions are thus not even remotely equal
<marler8997> with this solution, I'm not sure how a package would know which packages need to be retrieved
<marler8997> it's just a compile error
<mikdusan> marler8997: what happens when @import("opensslbuild") fails because not found?
<marler8997> I was just copying the Builder.option function
<daurnimator> marler8997: use the field name as the option name; and make the help come from a doc-comment?
<daurnimator> marler8997: why like that?
<marler8997> here's what comptime build options would look like in a build.zig: https://gist.github.com/marler8997/7a5e8c34078660d7762e68ffd5d8bb5e
<marler8997> yeah true, I don't know if you've considered full comptime build.zig
<marler8997> I know it will work so no proof-of-concept needed
<marler8997> sure thing, let me know if you want to discuss sometime as well, if you want to discuss soon I'll hold off on prototyping the comptime options idea and wait instead
<marler8997> yup
<marler8997> g-w1, instead of pre_build_running.zig, we probably want something like extract_comptime_options.zig
<marler8997> but allowing some options to be comptime is another solution, with no language changes required
<marler8997> (this is why @tryImport can get around this)
<marler8997> this will always fail because b.option is known at runtime, so the @import must always pass
<marler8997> whears today we have if (b.option(bool, "androidEnabled")) { const androidbuild = @import("androidbuild"); ... }
<marler8997> if (comptimeOptionAndroidEnabled) { const androidbuild = @import("androidbuild"); ... }
<marler8997> this solves the @tryImport problem because now we can use those comptime_options to know whether or not we need certain imports
<marler8997> the current idea is to allow build.zig to specify some options as comptime
<marler8997> andrewrk, ok thanks, knowing where you stand saves me alot of time and effort, even knowing you're not sure
<andrewrk> marler8997, yes my answer is, "I don't know, I need to think about it."
<marler8997> I'm not sure it's even a solution yet
<marler8997> and Zig's package manager probably won't have a way to hook into a dependency to run a pre-build step like this, so it actually probably isn't even a viable solution
<marler8997> every project that uses my project will also need to run this "pre-package-manager-build-tool"
<marler8997> that's always an option, but is ugly, and will make my project not work as a dependency for other projects
<marler8997> you can do this with a tool that runs before Zig's package/manager and "zig build"
<marler8997> this means we don't use b.comptime_option
<marler8997> it's a solution that works with the existing language, so that's definitely a plus
<marler8997> you don't need to rebuild it, even if the user provides different comptime options
<marler8997> you only need to recompile pre_build_runner.zig when build.zig changes
<marler8997> only 1 compilation ever
<marler8997> not 1 compliation every time
<marler8997> it's not just about speed, but storage as well
<marler8997> that can be cached
<marler8997> this is 1 extra compilation, that's it end of story
<marler8997> that's 1 compilation per argument permutation
<marler8997> nah, caching every arg is completely different
<marler8997> ok I think I have enough to create a PR with a proof-of-concept in Zig
<marler8997> but it should work
<marler8997> it adds this extra compilation before build.zig is compiled
<marler8997> it's convuluted and complicated
<marler8997> that's pretty clever if I do say so myself
<marler8997> it relys on Zig being lazy, and it looks for a special symbol like "comptime_options" inside build.zig
<marler8997> it imports build.zig but doesn't call "build"
<marler8997> pre_build_runner.zig
<marler8997> we could possibly do something clever
<marler8997> it needs to know what options are comptime before it compiles build.zig
<marler8997> but, it doesn't seem like something that should be exposed to the end-user
<marler8997> we could use a different prefix like "zig build -Efoo=bar"
<marler8997> zig build does not even know the options until after it executes the "build" function from build.zig at runtime
<marler8997> that' doesn't run until runtime
<marler8997> oh, builder
<marler8997> b's comptime_options? what's b?
<marler8997> because that's how it knows it needs to rebuild build.zig
<marler8997> zig build needs to know which options the user has given are comptime
<marler8997> but how to expose them on the command-line?
<marler8997> right
<marler8997> we already need to distinguish between comptime and runtime options, it would be crazy to make "zig build run ARGS" comptime
<marler8997> you're going to recompile build.zig N exact duplicates of build.zig
<marler8997> why should every project pay the cost of comptime args when only a few need it?
<marler8997> sure, but we're talking about a corner case
<marler8997> means you don't need to rebuild build.zig
<marler8997> how would zig build know if it was a comptime or runtime option?
<marler8997> right
<marler8997> comptime options would need a special syntax in "zig build"
<marler8997> why don't we let build.zig say what options are comptime?
<marler8997> ooo wait a second
<marler8997> oh right, yes
<marler8997> build.zig wouldn't actually import buildoptions, build_runner.zig would
<marler8997> we could make b.option return a comptime value, so we'd still be using the exact same API
<marler8997> g-w1 huh?
<marler8997> it sounds like your answer is, "I don't know I need to think about it" from your first impression, do you have an inkling about whether you would be more likely to accept @tryImport or comptime build options?
<marler8997> not only does not only apply to non-leaf nodes, it also only applies to nodes that have dynamic dependencies
<marler8997> there's no question that this feature is a corner case
<marler8997> be back
<marler8997> andrewrk, do you think making build options available at comptime for build.zig is likely to be accepted?
<marler8997> I'll be back in like 30
<marler8997> and the bigger question is, whether build.zig could be executed entirely at comptime
<marler8997> the question is whether this would be accepted