sb0 changed the topic of #m-labs to: ARTIQ, Migen, MiSoC, Mixxeo & other M-Labs projects :: fka #milkymist :: Logs http://irclog.whitequark.org/m-labs
<mithro>
sb0: You would need to do "MigenSet({x, y, ...})"
cr1901_modern has joined #m-labs
<sb0>
mithro, the order is lost as soon as you use {}
<sb0>
whitequark, it's in the commit message iirc
<mithro>
sb0: can't talk now, I'll explain further later this afternoon
<sb0>
also, if you have () already, making it ({}) just adds clutter
mindrunner has quit [Ping timeout: 250 seconds]
mumptai has quit [Ping timeout: 250 seconds]
siruf has quit [Ping timeout: 250 seconds]
mindrunner has joined #m-labs
siruf_ has joined #m-labs
siruf_ is now known as siruf
mumptai has joined #m-labs
mumptai has quit [Ping timeout: 240 seconds]
mumptai has joined #m-labs
<sb0>
you'd think "git mv" would handle submodules buried into the directory you are moving. no. it borks your repository and any git command results in crap like "fatal: Not a git repository: misoc/software/src/compiler-rt/../../.git/modules/software/compiler-rt"
<sb0>
mithro, things that depend on the order things are defined/included are unavoidable and fine
<sb0>
mithro, your patch should not attempt to "fix" that; instead, it should make the definitions deterministic
<mithro>
sb0: you talking about the latest patch I sent? If you ignore the last bullet, the first two things I listed are definite cases where the same definition order will produce different output.
<sb0>
in _build_xst_files you are sorting right before the output
<sb0>
I would sort where the non-determinism happens
<sb0>
sprinkling "sorted" all over just makes things messy and potentially slow...
<mithro>
sb0: yes - that is my longer goal, having those output files be sorted is nice though (and if I was creating them by hand I would make them sorted)
<sb0>
I wonder why python just doesn't make set and dict iteration deterministic
<mithro>
sb0: well - if you do a PYTHONHASHSEED=0 then it is in theory
<sb0>
it's fucking annoying, and python itself has the problem - their bytecode compiler produces different output for the same source all the time
<mithro>
sb0: I was pondering making the "make.py" check the environment for that and if it didn't exist "re-exec itself" if that value set.
<sb0>
bah. no. the solution is to pass deterministic iterators to whatever generates the output files.
<sb0>
create migen signals and csrs in deterministic order
<sb0>
sort the output of os.path.walk
<sb0>
the rule for migen is: it should produce deterministic output for the same signal creation order
<mithro>
sb0: yes - I would think migen producing deterministic output for the same signal creation order is the goal
<sb0>
sorted(crappy_python_structure, key=lambda x: x.duid) does it
<sb0>
of course, the user should also be careful e.g. when creating signals/logic based on the contents of a dict/set
<mithro>
"the user should also be careful e.g. when creating signals/logic based on the contents of a dict/set" - There are at least a few cases in migen itself which don't follow that rule. I'm pretty sure it is the cause of all the non-determinism in migen at the moment (not sure about misoc).
<mithro>
Anywhere migen does a "for x in set/dict:" is a problem if they do / cause any type of creation
<sb0>
well, then migen should be fixed
<mithro>
sb0: That is the plan!
<sb0>
whitequark, can we get rid of __cxxabi_config.h somehow?
<mithro>
sb0: I still think the latest patch I sent is useful - having the .prj and similar files be in sorted order for any definition order is still a good thing
<sb0>
not sure about that for the include path
<sb0>
what if the same include file is in two places?
<sb0>
also, it adds code.
<mithro>
sb0: The multiple files in the include path with the same name is definitely a bit hairy - its definitely a case which is going to cause subtle bugs (if the thing is expecting one of the files to be included and another is expect the other)
FabM has quit [Quit: ChatZilla 0.9.92 [Firefox 40.0.3/20150826023504]]
<_florent_>
mithro: I'm not sure we really need that and I agree with sb0: if we really need that, this should be done where the at non-determinism happens otherwise we will have to introduce that for all toolchains.
<_florent_>
I think you are messing with this because ISE generates random builds and sometimes takes 2 hours to build
<mithro>
_florent_ / sb0: lets see if I can fix the other non-determinism and then we can re-evaluate that
<_florent_>
In a such situation, ISE has clearly difficulties with the design and it's better to optimize the design or add some placement constraints to guide ISE than trying to get determinism everywhere because changing one line of code will of course introduce non-determinism and could reintroduce builds that takes 2 hours...
FabM has joined #m-labs
<_florent_>
So before touching that we should optimize the HDMI2USB design to give ISE some margin. I'll try to look at that.
<mithro>
_florent_: I'm interested in trying to make the HDMI2USB firmware reproducible in the long term
<_florent_>
ok, but do you still have the issue with build that take random time?
<mithro>
_florent_: yes, but I'm not too worried about that
<mithro>
_florent_: Can't really talk right now - but I would like to sync up later if you have time
<sb0>
mithro, how exactly is soc.get_constant() returning a non-deterministic iterable?
<sb0>
_florent_, check_csr_region() is incomplete.
<sb0>
ah, no, sorry, it's ok, since all regions have the same size.
<whitequark>
sb0: ok, to recap status of with parallel: the problem I've stumbled onto is that in the new compiler, `def x(...):` is the same as `x = def(...):`
<whitequark>
where def() is a hypothetical construct creating a multi-line anonymous function
<whitequark>
which is Python's semantics.
<whitequark>
however, an unfortunate consequence is that you can't inline anything statically, because it may be overridden later
<whitequark>
now you may suggest I do the same as in your old compiler and just make sure it isn't actually overridden, but the problem is that ties something that our compiler guarantees (interleaving) to an output of a transformation which does not always finish successfully and when not, is unable to coherently explain why that happened
<whitequark>
I tried a few 'elegant' way to integrate it in the language but none were a particular improvement
<sb0>
what do you mean by overridden?
<whitequark>
def x(): return 1
<whitequark>
def x(): return 2
<whitequark>
^ legal python
<sb0>
_florent_, if there is a function "register_sdram_phy" that takes the PHY as parameter, then bruteforcing common names (sdrphy, ddrphy) in make.py is twice a horrible and unacceptable hack
<sb0>
whitequark, we don't have to support this particular construct
<sb0>
throw an error on the re-definition?
<whitequark>
sb0: it's not about this particular construct
<sb0>
the problem is you cannot statically resolve function names in order to inline them, right?
<whitequark>
functions in general. some functions appear as locals, but when we do embedding, functions also appear as attributes
<sb0>
eg you can have some algorithm that assigns x to a function, and then later there is x() in a "with parallel"
<whitequark>
such as in with parallel: ttl0.pulse()
<whitequark>
what currently happens in the latter case is that self.ttl0.pulse is resolved at runtime; self, ttl0, etc have classes, and the usual method lookup chain works
<sb0>
what about inlining (and for now, supporting) only those functions that clearly are static?
<sb0>
runtime resolution also sounds slow
<whitequark>
I believe sure LLVM optimizes it out in trivial cases
<whitequark>
but the thing is that it's /possible/ to do ttl = self.ttl0 if x else self.ttl1; ttl.pulse()
<whitequark>
and the pulse() doesn't have to be the same, etc
[florian] has quit [Ping timeout: 250 seconds]
FabM has quit [Ping timeout: 250 seconds]
<whitequark>
as for "clearly are static", well, sure. with parallel:, by definition, can only work with functions that are clearly static
FabM has joined #m-labs
<whitequark>
the problem I have is a definition of "clearly static" that is equally suited for a human and a compiler to understand
<sb0>
not necessarily, if we have multiple cpus/threads then with parallel can support dynamic functions.
rjo has quit [Ping timeout: 250 seconds]
<sb0>
but I would not look into that, especially considering how late this new compiler is
<whitequark>
I wouldn't either, we have a lot more issues to solve before multicore
[florian] has joined #m-labs
[florian] has joined #m-labs
rjo has joined #m-labs
<sb0>
so. what is the problem with a simple static analyzer that resolves functions, which is invoked whenever a function call in a "with parallel" block?
<sb0>
*there is
<sb0>
it can immediately throw an error right now if the function cannot be statically resolved
<whitequark>
would you say that such a static analyzer only try to resolve names and chains of names?
<sb0>
yes
<whitequark>
i.e. no constant propagation, no DCE, just see if a name is defined once and only once and fail otherwise?
<whitequark>
yes, I can do that
<whitequark>
yeah, that is much less terrible, assuming you can fall back to hardware at some point in the future
<sb0>
multicore sounds not only possible but also potentially necessary for performance reasons
<whitequark>
when you do multicore, you immediately bump into having to define a memory model, which is why i'm saying there are more issues
<whitequark>
but yes
<sb0>
sure, and there are higher priority items right now...
<sb0>
maybe we could look into this fancy timecounter-based model that was published recently
<whitequark>
hm
<whitequark>
just how many cores do you want? dozens?
<sb0>
I don't know
<whitequark>
and what I'm talking about is not really hardware issues. the hardware is fairly well understood
<whitequark>
what I'm talking about is software, because Python does not have a notion of memory model nor any good tools to deal with concurrency
<whitequark>
what happens when you mutate the environment of the same closure in parallel?
<sb0>
what happens right now? it works because of the GIL?
<whitequark>
yes
<whitequark>
oh, actually it's far worse than that
<whitequark>
don't forget that our allocator is fundamentally thread-local
<whitequark>
and our memory safety fundamentally depends on invariants that are derived from everything happening on the same thread
<sb0>
yeah. anyway. that's for much later. right now there are other, often more down-to-earth problems, like UDP not working on windows
<whitequark>
ok
<cr1901_modern>
stupid q, but what does "interleaving" mean in the context of Artiq? (6:17:46)
<sb0>
whitequark, do you happen to know if there is a flag to set the ld include path?
<sb0>
ie an option that sets a directory that is searched to process "INCLUDE" directives in linker scripts
<whitequark>
I believe it searches the paths defined with -L
<sb0>
no, it doesn't
<whitequark>
hm
<whitequark>
INCLUDE filename
<whitequark>
Include the linker script filename at this point. The file will be searched for in the current directory, and in any directory specified with the -L option.
<GitHub113>
misoc/master 8e860e3 Sebastien Bourdeauducq: Merge branch 'master' of github.com:m-labs/misoc
<_florent_>
sb0> can't we just expect the user to pass an appropriate path in there? > from what I remember, I introduced that because toolchains requires paths with "/" (even on windows) when path provided by os.path.join on windows machines uses "\". But maybe that's not the right place to do that.
<sb0>
oh right, it does \ -> / on windows, not the other way around
<sb0>
well. if the toolchain cannot be fixed, there should be at least a comment about why that is needed
<sb0>
but I would complain to the toolchain vendor. it's *their* bug.
<_florent_>
I'll redo the test for each toolchain and eventually report bugs
<sb0>
also, that replacement workaround should only be executed when dealing with the buggy toolchain, not all of them
<_florent_>
ack
rohitksingh has quit [Ping timeout: 260 seconds]
rohitksingh has joined #m-labs
rohitksingh has quit [Client Quit]
<terpstra>
@whitequark rust is awesome
<terpstra>
when it gets a bit more mature, i totally want all our embedded coding done in rust.
<whitequark>
I used to be extremely positive about rust, but then I wrote a lot of rust code
antgreen has quit [Ping timeout: 240 seconds]
<sb0>
_florent_, do you really need the "with_identifier" and "with_timer" options in SoC?
<sb0>
those cores are very small
ylamarre has joined #m-labs
antgreen has joined #m-labs
antgreen has quit [Ping timeout: 265 seconds]
<cr1901_modern>
sb0: Those options actually do make a difference when building MiSoC for the smaller Spartan-3, where all saved space counts.
vn has joined #m-labs
vn has quit [Client Quit]
ylamarre has quit [Ping timeout: 240 seconds]
mumptai has quit [Quit: Verlassend]
ylamarre has joined #m-labs
ylamarre has quit [Read error: Connection reset by peer]