ChanServ changed the topic of #picolisp to: PicoLisp language | Channel Log: https://irclog.whitequark.org/picolisp/ | Check also http://www.picolisp.com for more information
orivej has quit [Ping timeout: 260 seconds]
aw- has joined #picolisp
<Regenaxer> Cool tankf33der
<Regenaxer> But I must first find the build-bug
<Regenaxer> If we fix other errors, the bug will not appear again and we don't know whether it is fixed or not (like last time)
<Regenaxer> Only in the current state we can reproduce it
<tankf33der> this one is suspecius
<tankf33der> also i could build ll file via gc+ run
<Regenaxer> I'm back at tracing in the build what exactly goes wrong
<Regenaxer> It is definitely not 'pack'
<Regenaxer> I was wrong yesterday
<Regenaxer> We had *exactly* that error a few months ago. Remember? We did gc+ tests, and found a bug in 'for' iirc
<Regenaxer> So we thought we fixed it
<Regenaxer> But 'for' was *not* the reason
<Regenaxer> Only the symptom disappeared
<Regenaxer> So if we now change anything, by fixing other bugs, we will not find this one again
<Regenaxer> It has absolute prioritY
<Regenaxer> I also cannot continue with src/db.l etc
<Regenaxer> I'm tracing back the faulty value
<tankf33der> i will bulld ll via gc+
<Regenaxer> it happens when 'lup' in src/subr.l is compiled
<Regenaxer> in 'if'
<Regenaxer> it is the expression (cons Z (lup N (cdr X))) )
<Regenaxer> compiles as %95 = call i64 @cons(i64 %82, i64 %94)
<Regenaxer> *this* result, "%95" is the one that is garbled
<Regenaxer> That's what I know so far
<Regenaxer> I suspect it is not a gc issue
_whitelogger has joined #picolisp
<tankf33der> in yesterday bug no crash but just wrong result in gc+
orivej has joined #picolisp
aw- has quit [Quit: Leaving.]
<tankf33der> bug without fixing it would help to understand what mechanism triggered the bug.
<Regenaxer> yes, but how to know then it is the build-bug?
<Regenaxer> If we don't find the reason *now*, it will disappear again
<Regenaxer> I'm sure it is the same bug as we saw a few months ago
<Regenaxer> then we fixed 'for', and it disappeared
<Regenaxer> But it was wrong
<tankf33der> maybe tree constructed the way in f4-48 and lup returned NIL now.
<Regenaxer> lup is not executed here, it is compiled
<Regenaxer> bbl
jibanes_ has joined #picolisp
jibanes has quit [Ping timeout: 240 seconds]
rob_w has joined #picolisp
orivej has quit [Ping timeout: 240 seconds]
mtsd has joined #picolisp
<Regenaxer> ret
<tankf33der> http://ix.io/2tXs
<tankf33der> this one also fails in gc+
<Regenaxer> ok
<tankf33der> i gonna find all of them.
<Regenaxer> Now I think the build problem is in 'let'
<Regenaxer> (if (atom X)
<Regenaxer> X
<Regenaxer> (let Z (save (lup N (car X)))
<Regenaxer> (cons Z (lup N (cdr X))) ) ) )
<Regenaxer> 'if' gets the broken value
<Regenaxer> but in the body of 'let' it is still there
<Regenaxer> the return of 'cons'
<Regenaxer> which is "%95"
<Regenaxer> So it must be 'let' (?)
<Regenaxer> llvm~let
<Regenaxer> not pico~let of course
<Regenaxer> Whas is called there is 'bind'!
<Regenaxer> 'bind' would be a good candidate, it is normally not used often
<Regenaxer> Inspecting 'bind'
<Regenaxer> Aahh!
<tankf33der> :)
<Regenaxer> It is not the 'bind' function! But it is the garbage collector itself
<Regenaxer> (let P (val $Bind) # Bind frames
<Regenaxer> (while P
<Regenaxer> (mark (val P)) # Mark saved value
<Regenaxer> (setq P (val 3 P)) ) )
<Regenaxer> When marking the bind frames, it marks only the values
<Regenaxer> but not the symbols!
<Regenaxer> Normally, these are internal symbols, so it does not matter
<Regenaxer> But in the pil21 compiler they are transient symbols like "%95"
<Regenaxer> This *must* be the reason
<tankf33der> T.
<Regenaxer> Can this also be the reason for some of your gc+ tests?
<Regenaxer> I think not
<Regenaxer> no transients bound (?)
<tankf33der> I found already 5 failed tests under gc+
<Regenaxer> good
<Regenaxer> I fix gc, then see
<tankf33der> sure.
<tankf33der> In general i can test all in 3 hours of manual work.
<Regenaxer> wow
<Regenaxer> grr
<Regenaxer> no
<tankf33der> to run all code base under gc+ would take days or weeks.
<Regenaxer> the wrong output is still there
<Regenaxer> strange
<Regenaxer> 87222,87223c87222
<Regenaxer> < ret i64 %102
<Regenaxer> ---
<Regenaxer> < %102 = phi i64 [%77, %$15], [%95, %$16] ; #
<Regenaxer> > ret i64 void
<Regenaxer> I can't believe
<Regenaxer> too bad
<Regenaxer> I must think again
<Regenaxer> Now another business
<Regenaxer> bbl
<Regenaxer> ret
<Regenaxer> ok, it must be 'bind', but the marking of the symbol in 'gc' is indeed only necessary for pathological cases
<Regenaxer> That's why I left it out
<Regenaxer> But for correctness, better mark here
<Regenaxer> pil64 and pil32 do them mark too
peterhil has quit [Read error: Connection reset by peer]
peterhil has joined #picolisp
<beneroth> Hi Regenaxer
<beneroth> Is (+Aux +Key) expected to work ?
<Regenaxer> hi beneroth
<Regenaxer> Have not tried. Should work, but does it make sense?
<beneroth> use case is a (+Aux +Key +Link) (date time) NIL (account))
<Regenaxer> hmm, no
<Regenaxer> With +Aux it is no longer a +Key
<beneroth> I'm only interested to have the last updated entry in the index
<beneroth> ok
<beneroth> so +Ref, or custom indexing prefix class which builts a 'lst key value
<Regenaxer> +Key has the direct value, and +Aux makes a list of it
<beneroth> list is ok
<beneroth> but just only one of it xD
<beneroth> hm
<Regenaxer> yes, but the key is broken
<beneroth> yeah I know
<Regenaxer> won'w find it any more
<beneroth> the unique check on the index is against the whole index entry, so key + value
<beneroth> right?
<Regenaxer> yes
<Regenaxer> What is the last updated?
<beneroth> last account activity
<beneroth> basically
<Regenaxer> Can't you keep it is a global place?
<Regenaxer> not in the index
<beneroth> I like to have it in the DB
<Regenaxer> yes, but not the index
<Regenaxer> anyway, sorry, I'm up to the neck in debugging bind
<beneroth> ok
<beneroth> no worries
<Regenaxer> can't concentrate now
<beneroth> okay sorry for the interuption
<beneroth> all fine
<beneroth> good hunt!
<beneroth> you will get the bugs, I'm sure of it
orivej has joined #picolisp
<Regenaxer> Hmm, but *now* I found it
<Regenaxer> build passe
<Regenaxer> s
<Regenaxer> It was 'prog1'!!
<Regenaxer> and prog2 the same
<Regenaxer> I did not save the value
Blue_flame has quit [Quit: killed]
<Regenaxer> I relied on it being stored in '@'
<Regenaxer> but the body of prog1 may set '@' to some other value, so the old value is no longer safe
<Regenaxer> This explains why it was seldom
<Regenaxer> only if '@' is overwritten by the body
aw- has joined #picolisp
<Regenaxer> Must check for other such cases too
<Regenaxer> tankf33der, may prog1/2 be involved in your cases?
<tankf33der> sure.
<Regenaxer> oh
<Regenaxer> Let me check for more cases, then I'll release
<tankf33der> ok
<Regenaxer> looks good
Blue_flame has joined #picolisp
<Regenaxer> released
<Regenaxer> puh
<Regenaxer> That was a tough one
<Regenaxer> And as I suspected, buggy since long times
<Regenaxer> If you have time, please run all tests again :)
<tankf33der> ok
<tankf33der> i will
<Regenaxer> Your last example, (diff (fish atom L1) (fish atom L2)), does now use prog1 though
<Regenaxer> and also not f4-48
<Regenaxer> 'sect' and 'diff'
<tankf33der> not in this cases, but inside pil21-tests
<Regenaxer> ok
peterhil has quit [Ping timeout: 256 seconds]
<tankf33der> tests passed, now gc+
<Regenaxer> ok
<tankf33der> http://ix.io/2tXX
<tankf33der> they both keep failing
<Regenaxer> fail only with gc+?
<tankf33der> yeap!
<Regenaxer> btw, have you updated your gc+ source? src/gc.l has one new line (not important though)
<Regenaxer> great!
<Regenaxer> So 'diff' and 'sect'
<Regenaxer> uniq is more involved
<Regenaxer> or it is 'fish'
<tankf33der> all updated. i use git. fetching every 15mins in working hours.
<beneroth> congratulations
<beneroth> Regenaxer, what the chance, when I mused about prog1 xD
<beneroth> zufall
<Regenaxer> yeah
<beneroth> but how could that corrupt stuff?
<beneroth> or you used prog1 wrong, as I did?
<Regenaxer> The value was not safe
<beneroth> safe means?
<Regenaxer> saved in a stack frame, so that gc marks it
<Regenaxer> (prog1 (list 1 2 3) (doSomething))
<Regenaxer> returns (1 2 3), right?
<Regenaxer> But if (doSomething) causes gc, the list must be safe
<Regenaxer> prog1 immediately stores the value in '@'
<Regenaxer> so it *is* safe
<Regenaxer> but if (prog1 (list 1 2 3) (and (foo) (doSomething)))
<Regenaxer> the 'and' writes the result of (foo) into '@'
<beneroth> aye
<Regenaxer> so the list (1 2 3) is no longer referred to
<beneroth> and gets gc'ed
<Regenaxer> only a variable in prog1
<Regenaxer> gc does not know this variable
<Regenaxer> so it frees the list
<beneroth> isn't that ok? it's not bound anywhere anymore, so for what is it still used?
<Regenaxer> pil32 and pil64 also stored in '@', but also saved the list in a stack structure
<Regenaxer> It is not ok, because prog1 returns that value
<Regenaxer> so the caller of prog1 reveives a bad pointer
<Regenaxer> pointer to gc'd memory, or a new cell etc
<beneroth> ah right
<beneroth> it is returned, but not bound to @ anymore
<beneroth> right
<Regenaxer> yes
<Regenaxer> This is always a tricky issue
<beneroth> hehe yeah
<Regenaxer> For performance, do not save more than necessary
<Regenaxer> but saving less *may* give a problem
<Regenaxer> Hard to detect
<beneroth> well many languages solved it by saving more than necessary and then throwing hardware at it :P
<Regenaxer> T
<beneroth> or to be exact, the defined "necessary" as "everything we might want to provide in a debug/reflection/throw event
<beneroth> so like.. whole stack, minimum xd
<Regenaxer> yes, but this is the same in pil
<Regenaxer> (trail) traverses these structures
<Regenaxer> It is a "spaghetti" stack
<beneroth> hm,..well T
<beneroth> yeah interlinked cells = random access memory = spaghetti, no?
<beneroth> naturally
<Regenaxer> Interlinked cells on the hardware stack
<Regenaxer> aka many stacks
<Regenaxer> but synchronous
<beneroth> yeah, I mean the stack is "random access"
<beneroth> which implies "fragmented"
<beneroth> anyway, good job :)
<Regenaxer> took too long ;)
<Regenaxer> stupid error
<beneroth> you're relativity for "too long" is rather tiny :) that is good, but don't feel too bad about it :)
<Regenaxer> ok :)
<Regenaxer> Anyway there are more bugs
<Regenaxer> in pil64 it felt easier
<Regenaxer> as the stack was under full control
<beneroth> yeah I wonder how this one could result in "uninitialized memory usage" error in gc+
<Regenaxer> in llvm it is better than in C though
<beneroth> if they're related
<beneroth> ok
<beneroth> how is it compared to the ARM pil64 implementation efforts?
<Regenaxer> the "uninitialized memory usage" is probably secondary
<beneroth> that one gave you a pretty hard time, and turned out to be excellent
<Regenaxer> reference of the reference of an invalid pointer
<beneroth> T, probably a corruption side effect
<beneroth> heisenbug typical
<Regenaxer> yes
<Regenaxer> Next one is 'fish' perhaps
<Regenaxer> something is wrong
<beneroth> I just hope the llvm will be low maintenance. I understand llvm might still be a moving target, no?
<Regenaxer> But I also want to finish src/db.l
<beneroth> oh fish
<beneroth> I never used fish
<beneroth> but i can see nice uses
<Regenaxer> I think the things we need in llvm are pretty stable
<beneroth> is fish old? I kinda missed it
<Regenaxer> yes, quite
<beneroth> might be handy for editing DSL AST's
<Regenaxer> (pp 'test)
<Regenaxer> uses 'fish' for example
<beneroth> yeah pats
<beneroth> nice
<beneroth> nested fill
<Regenaxer> lib/form.l and lib/dbgc.l toi
<beneroth> I think I might in many cases be more interested in the recursion mechanic of fish then its return value
<Regenaxer> yes, but that is quite normal recursion
<Regenaxer> 'fish' is just for efficiency
<Regenaxer> recursing on the Lisp level is more expensive
<beneroth> yeah so I figure
<beneroth> more bindings to maintain, more functionc alls
<Regenaxer> exactly
<beneroth> fish is also depth-first?
<Regenaxer> yes
<beneroth> so also quite some state/binding maintenance
<Regenaxer> first CDR, then CAR it seems
<beneroth> stack maintenance, basically, right?
<beneroth> oh
<beneroth> haha
<beneroth> so right before left depth-first, okay :)
<beneroth> so much to learn from random discussions :)
<Regenaxer> :)
<beneroth> people without IRC missing it
<beneroth> on the other hand, if it would be more crowded, we would have less such discussions
<Regenaxer> right
<Regenaxer> more trolls
<beneroth> oh yes
<beneroth> and some people appear to read the archive even when they're not active here
<Regenaxer> some, yes, but hard to read
<Regenaxer> the log page is very slow
<Regenaxer> and gobbled with join/quit garbage
<Regenaxer> For that reason I filter orivej here
<Regenaxer> He messes up the log, and does not react when addressing him
<Regenaxer> (hmm, in case he reacts *now*, I won't notice ;)
orivej has quit [Ping timeout: 265 seconds]
<Regenaxer> So 'fish' is all right?
<Regenaxer> What causes gc+ to fail?
<beneroth> Regenaxer, orivej just lost connection once more
<beneroth> wasting other peoples resources this way is unethical, I find.
<beneroth> but yeah
<Regenaxer> yeah, at least when he obviously never watches here
<beneroth> T
<Regenaxer> It is ok if technically no better connection is available
<Regenaxer> then use only if necessary
<beneroth> it wastes resources of the freenode project, mainly
<Regenaxer> T
<Regenaxer> a little only
<beneroth> also green IT etc., my main argument against wasteful software and hardware throwing
<beneroth> T
<Regenaxer> Oh no!
<Regenaxer> Got the error again
<Regenaxer> build error :(
<Regenaxer> Can't believe :(
<Regenaxer> Same procedure once more
<beneroth> I try to think up candidates for similar mistakes as prog1/prog2, but I fail to come up with any. I thought about (cond) and (case), but the @ usage is just normal condition handling, and the return value is anyway prg-dependent
<Regenaxer> Not easy
<beneroth> how did you figure out to look at prog1 ?
<Regenaxer> It depends on the situation, what is sure to be safe
<beneroth> T
<Regenaxer> I put 'msg' calls into src/lib/llvm.l
<beneroth> ok, good old brute force log debugging :)
<Regenaxer> Then redirected stderr to "a" for building with pil64
<Regenaxer> and to "b" for building with pil32
<Regenaxer> then diff
<beneroth> right. elegant.
<Regenaxer> only a single line different usually
<Regenaxer> hmm, exactly the same place
<Regenaxer> As if I did not fix 'prog1'
<Regenaxer> Fixed it wrongly?
<beneroth> or build did reuse something mistakenly?
<Regenaxer> yes, something stupid
<beneroth> afk
<Regenaxer> cu :)
<Regenaxer> uh, stupid indeed!
<Regenaxer> I had moved to another machine, and didn't make clean
<Regenaxer> Now it is good
<Regenaxer> puh!
<Regenaxer> Looks like I should take some vacation
rob_w has quit [Remote host closed the connection]
<aw-> hi Regenaxer beneroth
<aw-> Regenaxer: is it considered "bad practice" to litter the global namespace with global variables?
<aw-> hundreds, maybe thousands of new symbols
<Regenaxer> Hi aw-!
<Regenaxer> I personally do not worry if I have many globals
<Regenaxer> or many symbols in general
<Regenaxer> The reader slows down a little ;)
<aw-> hmmm
<Regenaxer> Ideally put them into some namespace, they they are out of the way
<aw-> how so?
<Regenaxer> Reading a symbol involves a tree lookup
patrixl has quit [Ping timeout: 246 seconds]
<Regenaxer> to see if it is already interned
<Regenaxer> and use it if so
<Regenaxer> The tree degenerates, however, if they are all created in alphabetical order
<Regenaxer> i.e. when 'read'ing them the first time
<Regenaxer> (or created with 'intern')
patrixl has joined #picolisp
<aw-> interesting..
<Regenaxer> thousands are ok
<Regenaxer> if on a normal pil with gui:
<Regenaxer> : (length (all))
<Regenaxer> -> 2762
<Regenaxer> no problem
<aw-> what about one million symbols?
<Regenaxer> also ok, if they are created in random order
<aw-> is the problem caused by the amount, or their size?
<Regenaxer> then the tree is 20 to 30 levels deep
<Regenaxer> mainly number of syms in the tree
<Regenaxer> if the names are longer than 7 ASCII chars, lookup takes longer due to longer name comparisons
<Regenaxer> (short nums vs. bignums)
<aw-> would it be more efficient to use one list instead, which contains millions of cons pairs?
<Regenaxer> would be a lot worse
<Regenaxer> needs a million comparisons instead of 30
<aw-> i did some tests, seems to use more much memory
<aw-> :\
<Regenaxer> the tree needs 1.5 cells on the average
<Regenaxer> for an optimally balanced tree
<Regenaxer> half the nodes have 2 cells
<Regenaxer> and half (the leaf nodes) have 1 cell
<Regenaxer> plus the symbols themselves of course
<Regenaxer> *what* uses much more memory?
<aw-> a list of million cons pairs
<Regenaxer> much more memory than what?
<aw-> much more than a million symbols
<aw-> i didnt test a million sorry
<aw-> i only tested with a few hundred
<Regenaxer> thats ok
<Regenaxer> a cons pair takes one cell
<Regenaxer> a list of them takes 2 cells then
<Regenaxer> one for the pair, and one more for the list entry
<Regenaxer> Try to draw a box diagram
<aw-> cells diagram?
<Regenaxer> yes
<Regenaxer> car and cdr boxes
<aw-> the value in the cdr, not a pointer
<Regenaxer> as in the ref
rob_w has joined #picolisp
<Regenaxer> it is always a pointer
<aw-> but in a list, the cdr is a pointer to the car of the cons cell
<Regenaxer> except if it is a short num
<aw-> oh i thought in cons, the cdr was the value itself
<Regenaxer> in a list, the cdr is a pointer to the car of the *next* cell
<beneroth> back
<aw-> right right
<aw-> i knew that
<Regenaxer> ok
<Regenaxer> both car and cdr have a value
<Regenaxer> which is a pointer or a short num
<Regenaxer> anyway
<Regenaxer> a list needs N cells
<beneroth> hi aw- o/
<aw-> yes
<Regenaxer> a tree 1.5 * N cells
<Regenaxer> (ideally)
<aw-> hmm.. but you said earlier the tree is more efficient
<Regenaxer> "tree" is 'idx' or namespace
<beneroth> aw-, it is more efficient to query
<Regenaxer> for searching
<aw-> ohhhh ok ok
<Regenaxer> If you really have many millions, a DB might be better
<Regenaxer> needs more space, but on disk ;)
<beneroth> because of RAM limits :)
<beneroth> if all fits in ram, then no db needed
<Regenaxer> in mem only what is accessed
<Regenaxer> yes
<Regenaxer> but if only a small part is needed at a moment ...
<beneroth> aw-, don't worry too much about too many symbols. main thing to worry about is if you can handle it as programmer, keeping the overview, avoiding name shadowing/clashing. with that namespaces can help.
<aw-> yes yes DB.. i'm working with only in-memory data now
<Regenaxer> if just one tree is involved, 'idx' with perhaps 'balance' or 'hash' is best
<beneroth> pil cells are basically two pointers.. if you think about it, a program with many millions of pointers in RAM is actually not so special, especially in other languages where many indirect pointers fly around
<beneroth> so I wouldn't worry much unless RAM capacity or mental/conceptual capacity is at limits
<Regenaxer> right
<Regenaxer> mental/conceptual capacity is the critical point
<beneroth> T
<Regenaxer> make life easy
<beneroth> easy to handle, not easy to achieve :)
<aw-> well you already made it clear with this "Regenaxer: Reading a symbol involves a tree lookup "
<aw-> that's the information i was looking for
<beneroth> reading: parsing of a file
<beneroth> or interning a symbol.. so also when reading from network
peterhil has joined #picolisp
<Regenaxer> In that regard internal symbols are the easiest way
<beneroth> during execution, it's just pointers, and sometimes comparison of symbol names
<Regenaxer> you just "have" it
<beneroth> T, it's easier than all possible alternatives :P
<Regenaxer> the reader takes care of creating and finding it
<Regenaxer> T
<aw-> ok perfect
<aw-> so my implementation is correct ;)
<aw-> maybe i'll look into namespaces once more
<Regenaxer> we hope so
<Regenaxer> (correct I mean ;)
<aw-> thank you
<Regenaxer> :)
<beneroth> my advice: use namespaces as minimally as meaningful. namespacing everything in pil is premature optimization which likely makes things worse (mostly the conceptual fragmentation)
<aw-> yes i have a very specific use-case for it, will do some tests
<Regenaxer> beneroth, exactly
<beneroth> better to think of them as more powerful (named) transient scopes than what is called namespaces in mainstream programming
<beneroth> the term namespace as used in pil is technically correct and accurate, but still quite different in a way the term is usually understood by programmers
<beneroth> the key point is that pil namespaces do namespace Symbols, not names
<Regenaxer> Good to know. As usuall, I have only faint ideas of what "normal" programmers think
<Regenaxer> T
<Regenaxer> names do not exist in pil
<Regenaxer> separate from symbols
<beneroth> I might be wrong, or worded it wrong, its complicated. But maybe "normal programmers think of namespacing as namespacing names (while in fact its often not just that, but has kinda that effect to the using programmer)
<beneroth> aye
<beneroth> names are just an attribute of symbols to refer to them in human text
<Regenaxer> right
<Regenaxer> only for I/O
<beneroth> aw-, so for making a huge tree of interned symbols, the picolisp namespacing might actually be very powerful to make lookups even more efficient, as namespacing is practically splitting the main symbol trees (it's 2 trees, technically) into multiple sub-trees
<beneroth> and it would allow multiple symbols having the same name (when they are in different namespaces)
<beneroth> and shorter names are better (less space, less time for name comparison)
<Regenaxer> or in no namespace at all
<beneroth> Regenaxer, you're killing (arg) in pil21, right?
<Regenaxer> yes
<beneroth> kk.. I must remind myself...
<Regenaxer> good
<beneroth> I have here a (cond ((... (next)) (.. (arg))) :)
<Regenaxer> I will rewrite all use cases also in pil64
<beneroth> saves some (let) :)
<Regenaxer> yeah
<Regenaxer> but (arg) is expensive
<beneroth> yeah and the let isn't gone, just invisible, and always payed even when not needed
<beneroth> it's ok
<Regenaxer> T
<Regenaxer> (arg) was not a good idea from the beginning
<beneroth> one pil21 becomes stables/finalized we need to create a wiki page or so to note all the conceptual differences and few breaking changes
<Regenaxer> right
<Regenaxer> atm there is a small file
<Regenaxer> I tried to note all cases
<beneroth> I plan to make a docu for ht library. I think I fully grokked it. but my schedule is all over the place. feel free to remind/nag me to do it :)
<Regenaxer> ok :)
<Regenaxer> We have still time
<Regenaxer> What I'm worried about most is testing all DB stuff
<Regenaxer> Not finished yet, and the existing parts are not even tried yet
<beneroth> I don't know when I can invest time in that, but I certainly will before I move my stuff
<Regenaxer> They are all needed to do a first test
<beneroth> yeah
<beneroth> but most of it is stuff covered in other tests
<beneroth> only a few but very critical pieces
<beneroth> as most is implemented in pil anyway :)
<Regenaxer> Perhaps make same tests on old and new, and do a bytewise compare of the DB and journal files
<beneroth> T
<beneroth> right plio format is not changing, not like between pil32 and pil64
<Regenaxer> yes
<beneroth> so DB migration should also not be needed :D
<Regenaxer> but for example the transaction logs exist only a short moment
<Regenaxer> only if pil crashes inside a transaction
<beneroth> for a start, running a replica from a pil64 DB under pil21 might be an idea (if you don't require the replica)
<beneroth> that should cover the lowest level of binary file interactions, no?
<Regenaxer> the "journal"
<Regenaxer> thats easier
<beneroth> T
<beneroth> I enable the crash journal always
<Regenaxer> On old pils I tested by exiting the program inside a transaction
<Regenaxer> in commit
<beneroth> can't rule out sudden power offs on most instances, even when they should be impossible
<Regenaxer> I use it always on PilBox
<beneroth> T
<Regenaxer> yeah
<Regenaxer> on a phone even more dangerous
<beneroth> or do like tankf33der testing stuff.. a program which is doing DB stuff all the time with multiple processes, then kill -9 processes randomly
<Regenaxer> also good
<beneroth> once setup this could also be run on phone
<beneroth> or we should just implement a jepsen test for pilDB :)
<Regenaxer> jepsen?
<beneroth> jepsen.io, testing framework (and a incredible guy behind it), mainly used to test distributed databases
<Regenaxer> good
<beneroth> but recently he did a test of postgresql and found some buggy special cases
<beneroth> (non-distributed)
<Regenaxer> Such a test would be useful indeed
<beneroth> tests stuff like transaction isolation, dirty writes, dirty reads (all the possible flavours), and for distributed databases network delays, interruptions, splits, etc.
<beneroth> on postgres it found some special transaction isolation violation http://jepsen.io/analyses/postgresql-12.3
<beneroth> problem is, all those standards and definitions are often not well-understood, too often even misunderstood by DBMS devs :)
<beneroth> pilDB when used with transaction (also for reading) is the perfect text book serializable transactions.. because it's factually serialized, no fancy optimizing and trickery going on, we just pay the potential performance cost for it (if that matters is highly use case dependent)
<Regenaxer> never measured, but relatively low overhead
<beneroth> T
<Regenaxer> the data is just written twice
<beneroth> just locking, so no parallelism
<Regenaxer> plus syncing
<Regenaxer> yes
<Regenaxer> Sync takes probably most time
<beneroth> guess so
<Regenaxer> it waits for the first write to be on disk
<beneroth> pilDB architecture is pretty simple, but that rules out many categories of possible bugs
<Regenaxer> yeah, hope so
<beneroth> and the cost for that doesn't matter except for very busy cases
<beneroth> I'm certain.
<Regenaxer> I never noticed a real delay
<Regenaxer> but it all depends of course
<beneroth> well if two concurrent sessions both make (dbSync) ...(commit)/(rollback), they block each other.
<Regenaxer> yep
<beneroth> other DBMS go huge complicated ways to optimize that situation
<beneroth> but if it actually matters is application specific
<Regenaxer> T
<beneroth> and I think we should have the tools to do partial locks, but the difficulty of course is to correctly manage them
<Regenaxer> Heavy committing are costly anyway
<Regenaxer> also for the IPC
<beneroth> T, but other DBMS have here the same problems
<beneroth> only so much you can optimize there away
<Regenaxer> Instead of partial locks, separate DBs are easier
<beneroth> most DBMS this days do only append data, no physical in place updates
<Regenaxer> I see
<beneroth> and then have some job periodically (or slowly concurrently) garbage collecting that mess again
<beneroth> so the cost is moved to the future + interest + additional space cost
<beneroth> WAL
<beneroth> the main benefit is not even the performance gain when writing stuff, but isolation
<beneroth> then you just read all data up to a certain journal entry and got a snapshoted state
<Regenaxer> makes sense but complicated
<beneroth> T
<beneroth> the such achieved snapshot isolation is better than pilDB reads without a transaction, as you might read stuff from multiple commits in pilDB
<beneroth> but in practice other DBMS don't do this by default, only when the programmer is explicitly asking for it the right way, by default they do some less careful trickery to gain performance and hope for the best
<Regenaxer> :)
<beneroth> and most DB users/programmers these days don't know about this, so the end result is anyway potentially risky :)
<beneroth> so without understanding the DB stuff a programmer is probably better of with pilDB than mainstream databases
<beneroth> I would suspect
<beneroth> except for one very specific category of read errors
<Regenaxer> if he follows some simple rules
<beneroth> and maybe performance problems, but I believe them when I see them :)
<beneroth> yeah well.. a programmer without insight/understanding will anyway not get so far with pil. not the target audience :)
<Regenaxer> T
<beneroth> what I said here only really applies to old-school SQL databases, the "nosql" "distributed" databases are mostly a big mess
<beneroth> all kind of pitfalls, but most don't care
<beneroth> a number of companies died because of it, but *shrug* :)
<Regenaxer> oh
<Regenaxer> ok :) back to work
<beneroth> T
<beneroth> :)
<Regenaxer> ;)
<beneroth> oh, Mozilla just fired their security team: https://twitter.com/MichalPurzynski/status/1293220570885062657
<beneroth> apparently?
* beneroth is going angry every week because firefox is leaking memory all over the place, suddenly crashes from out of memory, crashing from corruption via video content, and other unbearable stuff
<rob_w> Regenaxer: i moved to picoLisp-20.6 .. all looked fine but i now get the following
<rob_w> this /usr/bin/picolisp: symbol lookup error: /usr/lib/picolisp/lib/ext: undefined symbol: crypt
<rob_w> but compile works with -lcrypto
<Regenaxer> oha
<Regenaxer> in ext
<Regenaxer> and -lcrypt ?
<Regenaxer> src/Makefile seems to use -lcrypt on Linux
<rob_w> but how can the compile work
<rob_w> hmm ok i will repatch and test
<Regenaxer> hmm, the linker should complain?
<Regenaxer> You do not use pil32's Makefile?
<rob_w> yes but i have to patch it to make it work inside my toolchain , in that i changed to -lcrypto
<Regenaxer> ok, still strange
<rob_w> ok letme try again
<rob_w> i am really confused about lcrypt vs lcrypto
<tankf33der> one of them from openssl
<tankf33der> you have it, right?
<tankf33der> you have installed openssl-devel packages too, ok?
<rob_w> yes .. -lcrypt works ! wtf
<rob_w> at least i get further and get a different problem which is ok
<Regenaxer> :)
<rob_w> whis is only a wrong path on myside .. so that concludes my merge to 20.6 - from 16.6 ;)
<Regenaxer> big step
<rob_w> definitly , for me !
<Regenaxer> :)
<rob_w> if only i got rid of arm32 .. but its not the time yet ..
<Regenaxer> arm32 is not good?
<Regenaxer> ah, you mean arm64 would be better :)
<Regenaxer> true
<rob_w> yeah
mtsd has quit [Quit: Leaving]
<rob_w> https://pastebin.pl/view/d62a60b3 this is my current makefile patch if interested
<Regenaxer> wow
<rob_w> mostly simple stuff .. stripping is done in my toolchain , CC is reset to the cross compiler .. only thing i worry is -fPIC but all works for now
<Regenaxer> should be ok
<aw-> hmmm
<rob_w> ok .. i worried as you once mentiond you have some hardcode conventions about addresses and pointers ..
<Regenaxer> only the size
<aw-> rob_w: hi
<Regenaxer> 32 bits on pil32
<rob_w> hi aw-
<aw-> i'm also running arm64, didnt have to make changes
<aw-> perhaps you're missing libssl
<rob_w> i am compiling via yocto toolchain , so strip and CC is handled inside
<aw-> oh ok
<rob_w> so it just needs to be abit more abstract compared to a "native" compile
<aw-> what are you doing with yocto?
<aw-> are you using balena?
<Regenaxer> rob_w, would LLVM make sense on your system?
<rob_w> no idea
<Regenaxer> I'm in the hope that Makefile gets more portable then (?)
<rob_w> aw-: i am building images for our custom boards with it .. basicly a simplified poky image with our stuff added
<rob_w> Regenaxer: ive no idea about llvm and friends
<Regenaxer> pil21 is based on it
<Regenaxer> I got the impression that all world moves to llvm
<rob_w> well i am happy to get to your latest stuff , so maybe i will find some time to grab you and try to get it work if that is your latest/best
<rob_w> but for today iam logging off .. need some shadow and some iced drinks .. cya guys and thx alot again
<Regenaxer> Good idea ;)
<Regenaxer> su
<Regenaxer> cu
<Regenaxer> tankf33der, strange the we have "ASM = clang" in Makefile
rob_w has quit [Quit: Leaving]
<Regenaxer> Should be "CC = clang", right?
<Regenaxer> and ASM should be llvm-as
<Regenaxer> can later be opt -O3
<tankf33der> yea, i told you. in general not a problem
<Regenaxer> yes, but confusing :)
<Regenaxer> I change it
<tankf33der> next: -lffi you should take via pkg-conf too
<tankf33der> and for readline headers and -I opt too. cant be compiled on non linux. freebsd failed.
<tankf33der> all saved to issues, later.
<tankf33der> afk.
<Regenaxer> ok
<Regenaxer> Let me know
orivej has joined #picolisp
jibanes_ is now known as jibanes
xkapastel has joined #picolisp
tankf33der_ has joined #picolisp
orivej has quit [Ping timeout: 264 seconds]
tankf33der has quit [*.net *.split]
Blue_flame has quit [Write error: Connection reset by peer]
orivej has joined #picolisp
Blue_flame has joined #picolisp
tankf33der_ has quit []
tankf33der_ has joined #picolisp
tankf33der_ has quit [Client Quit]
tankf33der_ has joined #picolisp
tankf33der_ has left #picolisp [#picolisp]
tankf33der has joined #picolisp
<tankf33der> i've completed run 860 tasks on gc+.
<tankf33der> list of failed functions:
<tankf33der> sect, fish.
<tankf33der> 4clojure, rng, 5 crypto functions, two years from adventofcode - all passed and never run before.
<Regenaxer> So only sect and fish failed?
<tankf33der> yeap. they will wait for you.
<Regenaxer> oh ;)
<Regenaxer> I looked at 'fish', but it looks correct ;)
<Regenaxer> Can you reproduce an error with 'fish' alone?
<tankf33der> with several lines i've post.
<tankf33der> too much
<tankf33der> ?
<Regenaxer> http://ix.io/2tXX ?
<tankf33der> for example.
<tankf33der> also i have another variation.
<Regenaxer> ok
<Regenaxer> we will find
<Regenaxer> Perhaps I draw a diagram tomorrow
<Regenaxer> 'sect' is simpler than 'fish'
<tankf33der> btw, maybe start check diff.
<tankf33der> fish itself works.
<Regenaxer> 'diff' is similar to 'sect'
<tankf33der> diagram of what ?
<Regenaxer> stack structures
<Regenaxer> link frames
<tankf33der> ok
<tankf33der> (diff (range 1 3) (range 2 10 3))
<tankf33der> this is wrong. so diff is bad.
<Regenaxer> hehe, diff and sect are exactly the same, except for while <-> unless
<Regenaxer> no
<Regenaxer> when <-> unless
<Regenaxer> but error only in gc+ ?
<Regenaxer> or always ?
<tankf33der> only in gc+
<Regenaxer> ok, good
<Regenaxer> hmm, diff is very simple
<Regenaxer> data seem safi
<Regenaxer> safe
<Regenaxer> ah, no
<Regenaxer> 'Z' is not
<Regenaxer> yes, this is probably the problem
<Regenaxer> I will fix tomorrow
<Regenaxer> good find! :)
<Regenaxer> nuff for today
<Regenaxer> gnite! :)
xkapastel has quit [Quit: Connection closed for inactivity]
<beneroth> Mozilla is indeed doing suicide (my interpretation): https://blog.mozilla.org/blog/2020/08/11/changing-world-changing-mozilla/
<beneroth> security team firefd
<beneroth> new rendering engine servo (written in rust, one of the few serious render engine projects that is not decades old) is killed
<beneroth> MDN team (the documentation about web standards) is killed
<beneroth> I predict Mozilla going slowly but inevitably die
<beneroth> and for the shorttime future Google controls the web 100%
<beneroth> this is another major blow to a free and open web
mmmattyx has joined #picolisp