<Regenaxer>
Interlinked cells on the hardware stack
<Regenaxer>
aka many stacks
<Regenaxer>
but synchronous
<beneroth>
yeah, I mean the stack is "random access"
<beneroth>
which implies "fragmented"
<beneroth>
anyway, good job :)
<Regenaxer>
took too long ;)
<Regenaxer>
stupid error
<beneroth>
you're relativity for "too long" is rather tiny :) that is good, but don't feel too bad about it :)
<Regenaxer>
ok :)
<Regenaxer>
Anyway there are more bugs
<Regenaxer>
in pil64 it felt easier
<Regenaxer>
as the stack was under full control
<beneroth>
yeah I wonder how this one could result in "uninitialized memory usage" error in gc+
<Regenaxer>
in llvm it is better than in C though
<beneroth>
if they're related
<beneroth>
ok
<beneroth>
how is it compared to the ARM pil64 implementation efforts?
<Regenaxer>
the "uninitialized memory usage" is probably secondary
<beneroth>
that one gave you a pretty hard time, and turned out to be excellent
<Regenaxer>
reference of the reference of an invalid pointer
<beneroth>
T, probably a corruption side effect
<beneroth>
heisenbug typical
<Regenaxer>
yes
<Regenaxer>
Next one is 'fish' perhaps
<Regenaxer>
something is wrong
<beneroth>
I just hope the llvm will be low maintenance. I understand llvm might still be a moving target, no?
<Regenaxer>
But I also want to finish src/db.l
<beneroth>
oh fish
<beneroth>
I never used fish
<beneroth>
but i can see nice uses
<Regenaxer>
I think the things we need in llvm are pretty stable
<beneroth>
is fish old? I kinda missed it
<Regenaxer>
yes, quite
<beneroth>
might be handy for editing DSL AST's
<Regenaxer>
(pp 'test)
<Regenaxer>
uses 'fish' for example
<beneroth>
yeah pats
<beneroth>
nice
<beneroth>
nested fill
<Regenaxer>
lib/form.l and lib/dbgc.l toi
<beneroth>
I think I might in many cases be more interested in the recursion mechanic of fish then its return value
<Regenaxer>
yes, but that is quite normal recursion
<Regenaxer>
'fish' is just for efficiency
<Regenaxer>
recursing on the Lisp level is more expensive
<beneroth>
yeah so I figure
<beneroth>
more bindings to maintain, more functionc alls
<Regenaxer>
exactly
<beneroth>
fish is also depth-first?
<Regenaxer>
yes
<beneroth>
so also quite some state/binding maintenance
<Regenaxer>
first CDR, then CAR it seems
<beneroth>
stack maintenance, basically, right?
<beneroth>
oh
<beneroth>
haha
<beneroth>
so right before left depth-first, okay :)
<beneroth>
so much to learn from random discussions :)
<Regenaxer>
:)
<beneroth>
people without IRC missing it
<beneroth>
on the other hand, if it would be more crowded, we would have less such discussions
<Regenaxer>
right
<Regenaxer>
more trolls
<beneroth>
oh yes
<beneroth>
and some people appear to read the archive even when they're not active here
<Regenaxer>
some, yes, but hard to read
<Regenaxer>
the log page is very slow
<Regenaxer>
and gobbled with join/quit garbage
<Regenaxer>
For that reason I filter orivej here
<Regenaxer>
He messes up the log, and does not react when addressing him
<Regenaxer>
(hmm, in case he reacts *now*, I won't notice ;)
orivej has quit [Ping timeout: 265 seconds]
<Regenaxer>
So 'fish' is all right?
<Regenaxer>
What causes gc+ to fail?
<beneroth>
Regenaxer, orivej just lost connection once more
<beneroth>
wasting other peoples resources this way is unethical, I find.
<beneroth>
but yeah
<Regenaxer>
yeah, at least when he obviously never watches here
<beneroth>
T
<Regenaxer>
It is ok if technically no better connection is available
<Regenaxer>
then use only if necessary
<beneroth>
it wastes resources of the freenode project, mainly
<Regenaxer>
T
<Regenaxer>
a little only
<beneroth>
also green IT etc., my main argument against wasteful software and hardware throwing
<beneroth>
T
<Regenaxer>
Oh no!
<Regenaxer>
Got the error again
<Regenaxer>
build error :(
<Regenaxer>
Can't believe :(
<Regenaxer>
Same procedure once more
<beneroth>
I try to think up candidates for similar mistakes as prog1/prog2, but I fail to come up with any. I thought about (cond) and (case), but the @ usage is just normal condition handling, and the return value is anyway prg-dependent
<Regenaxer>
Not easy
<beneroth>
how did you figure out to look at prog1 ?
<Regenaxer>
It depends on the situation, what is sure to be safe
<beneroth>
T
<Regenaxer>
I put 'msg' calls into src/lib/llvm.l
<beneroth>
ok, good old brute force log debugging :)
<Regenaxer>
Then redirected stderr to "a" for building with pil64
<Regenaxer>
and to "b" for building with pil32
<Regenaxer>
then diff
<beneroth>
right. elegant.
<Regenaxer>
only a single line different usually
<Regenaxer>
hmm, exactly the same place
<Regenaxer>
As if I did not fix 'prog1'
<Regenaxer>
Fixed it wrongly?
<beneroth>
or build did reuse something mistakenly?
<Regenaxer>
yes, something stupid
<beneroth>
afk
<Regenaxer>
cu :)
<Regenaxer>
uh, stupid indeed!
<Regenaxer>
I had moved to another machine, and didn't make clean
<Regenaxer>
Now it is good
<Regenaxer>
puh!
<Regenaxer>
Looks like I should take some vacation
rob_w has quit [Remote host closed the connection]
<aw->
hi Regenaxer beneroth
<aw->
Regenaxer: is it considered "bad practice" to litter the global namespace with global variables?
<aw->
hundreds, maybe thousands of new symbols
<Regenaxer>
Hi aw-!
<Regenaxer>
I personally do not worry if I have many globals
<Regenaxer>
or many symbols in general
<Regenaxer>
The reader slows down a little ;)
<aw->
hmmm
<Regenaxer>
Ideally put them into some namespace, they they are out of the way
<aw->
how so?
<Regenaxer>
Reading a symbol involves a tree lookup
patrixl has quit [Ping timeout: 246 seconds]
<Regenaxer>
to see if it is already interned
<Regenaxer>
and use it if so
<Regenaxer>
The tree degenerates, however, if they are all created in alphabetical order
<Regenaxer>
i.e. when 'read'ing them the first time
<Regenaxer>
(or created with 'intern')
patrixl has joined #picolisp
<aw->
interesting..
<Regenaxer>
thousands are ok
<Regenaxer>
if on a normal pil with gui:
<Regenaxer>
: (length (all))
<Regenaxer>
-> 2762
<Regenaxer>
no problem
<aw->
what about one million symbols?
<Regenaxer>
also ok, if they are created in random order
<aw->
is the problem caused by the amount, or their size?
<Regenaxer>
then the tree is 20 to 30 levels deep
<Regenaxer>
mainly number of syms in the tree
<Regenaxer>
if the names are longer than 7 ASCII chars, lookup takes longer due to longer name comparisons
<Regenaxer>
(short nums vs. bignums)
<aw->
would it be more efficient to use one list instead, which contains millions of cons pairs?
<Regenaxer>
would be a lot worse
<Regenaxer>
needs a million comparisons instead of 30
<aw->
i did some tests, seems to use more much memory
<aw->
:\
<Regenaxer>
the tree needs 1.5 cells on the average
<Regenaxer>
for an optimally balanced tree
<Regenaxer>
half the nodes have 2 cells
<Regenaxer>
and half (the leaf nodes) have 1 cell
<Regenaxer>
plus the symbols themselves of course
<Regenaxer>
*what* uses much more memory?
<aw->
a list of million cons pairs
<Regenaxer>
much more memory than what?
<aw->
much more than a million symbols
<aw->
i didnt test a million sorry
<aw->
i only tested with a few hundred
<Regenaxer>
thats ok
<Regenaxer>
a cons pair takes one cell
<Regenaxer>
a list of them takes 2 cells then
<Regenaxer>
one for the pair, and one more for the list entry
<Regenaxer>
Try to draw a box diagram
<aw->
cells diagram?
<Regenaxer>
yes
<Regenaxer>
car and cdr boxes
<aw->
the value in the cdr, not a pointer
<Regenaxer>
as in the ref
rob_w has joined #picolisp
<Regenaxer>
it is always a pointer
<aw->
but in a list, the cdr is a pointer to the car of the cons cell
<Regenaxer>
except if it is a short num
<aw->
oh i thought in cons, the cdr was the value itself
<Regenaxer>
in a list, the cdr is a pointer to the car of the *next* cell
<beneroth>
back
<aw->
right right
<aw->
i knew that
<Regenaxer>
ok
<Regenaxer>
both car and cdr have a value
<Regenaxer>
which is a pointer or a short num
<Regenaxer>
anyway
<Regenaxer>
a list needs N cells
<beneroth>
hi aw- o/
<aw->
yes
<Regenaxer>
a tree 1.5 * N cells
<Regenaxer>
(ideally)
<aw->
hmm.. but you said earlier the tree is more efficient
<Regenaxer>
"tree" is 'idx' or namespace
<beneroth>
aw-, it is more efficient to query
<Regenaxer>
for searching
<aw->
ohhhh ok ok
<Regenaxer>
If you really have many millions, a DB might be better
<Regenaxer>
needs more space, but on disk ;)
<beneroth>
because of RAM limits :)
<beneroth>
if all fits in ram, then no db needed
<Regenaxer>
in mem only what is accessed
<Regenaxer>
yes
<Regenaxer>
but if only a small part is needed at a moment ...
<beneroth>
aw-, don't worry too much about too many symbols. main thing to worry about is if you can handle it as programmer, keeping the overview, avoiding name shadowing/clashing. with that namespaces can help.
<aw->
yes yes DB.. i'm working with only in-memory data now
<Regenaxer>
if just one tree is involved, 'idx' with perhaps 'balance' or 'hash' is best
<beneroth>
pil cells are basically two pointers.. if you think about it, a program with many millions of pointers in RAM is actually not so special, especially in other languages where many indirect pointers fly around
<beneroth>
so I wouldn't worry much unless RAM capacity or mental/conceptual capacity is at limits
<Regenaxer>
right
<Regenaxer>
mental/conceptual capacity is the critical point
<beneroth>
T
<Regenaxer>
make life easy
<beneroth>
easy to handle, not easy to achieve :)
<aw->
well you already made it clear with this "Regenaxer: Reading a symbol involves a tree lookup "
<aw->
that's the information i was looking for
<beneroth>
reading: parsing of a file
<beneroth>
or interning a symbol.. so also when reading from network
peterhil has joined #picolisp
<Regenaxer>
In that regard internal symbols are the easiest way
<beneroth>
during execution, it's just pointers, and sometimes comparison of symbol names
<Regenaxer>
you just "have" it
<beneroth>
T, it's easier than all possible alternatives :P
<Regenaxer>
the reader takes care of creating and finding it
<Regenaxer>
T
<aw->
ok perfect
<aw->
so my implementation is correct ;)
<aw->
maybe i'll look into namespaces once more
<Regenaxer>
we hope so
<Regenaxer>
(correct I mean ;)
<aw->
thank you
<Regenaxer>
:)
<beneroth>
my advice: use namespaces as minimally as meaningful. namespacing everything in pil is premature optimization which likely makes things worse (mostly the conceptual fragmentation)
<aw->
yes i have a very specific use-case for it, will do some tests
<Regenaxer>
beneroth, exactly
<beneroth>
better to think of them as more powerful (named) transient scopes than what is called namespaces in mainstream programming
<beneroth>
the term namespace as used in pil is technically correct and accurate, but still quite different in a way the term is usually understood by programmers
<beneroth>
the key point is that pil namespaces do namespace Symbols, not names
<Regenaxer>
Good to know. As usuall, I have only faint ideas of what "normal" programmers think
<Regenaxer>
T
<Regenaxer>
names do not exist in pil
<Regenaxer>
separate from symbols
<beneroth>
I might be wrong, or worded it wrong, its complicated. But maybe "normal programmers think of namespacing as namespacing names (while in fact its often not just that, but has kinda that effect to the using programmer)
<beneroth>
aye
<beneroth>
names are just an attribute of symbols to refer to them in human text
<Regenaxer>
right
<Regenaxer>
only for I/O
<beneroth>
aw-, so for making a huge tree of interned symbols, the picolisp namespacing might actually be very powerful to make lookups even more efficient, as namespacing is practically splitting the main symbol trees (it's 2 trees, technically) into multiple sub-trees
<beneroth>
and it would allow multiple symbols having the same name (when they are in different namespaces)
<beneroth>
and shorter names are better (less space, less time for name comparison)
<Regenaxer>
or in no namespace at all
<beneroth>
Regenaxer, you're killing (arg) in pil21, right?
<Regenaxer>
yes
<beneroth>
kk.. I must remind myself...
<Regenaxer>
good
<beneroth>
I have here a (cond ((... (next)) (.. (arg))) :)
<Regenaxer>
I will rewrite all use cases also in pil64
<beneroth>
saves some (let) :)
<Regenaxer>
yeah
<Regenaxer>
but (arg) is expensive
<beneroth>
yeah and the let isn't gone, just invisible, and always payed even when not needed
<beneroth>
it's ok
<Regenaxer>
T
<Regenaxer>
(arg) was not a good idea from the beginning
<beneroth>
one pil21 becomes stables/finalized we need to create a wiki page or so to note all the conceptual differences and few breaking changes
<Regenaxer>
right
<Regenaxer>
atm there is a small file
<Regenaxer>
I tried to note all cases
<beneroth>
I plan to make a docu for ht library. I think I fully grokked it. but my schedule is all over the place. feel free to remind/nag me to do it :)
<Regenaxer>
ok :)
<Regenaxer>
We have still time
<Regenaxer>
What I'm worried about most is testing all DB stuff
<Regenaxer>
Not finished yet, and the existing parts are not even tried yet
<beneroth>
I don't know when I can invest time in that, but I certainly will before I move my stuff
<Regenaxer>
They are all needed to do a first test
<beneroth>
yeah
<beneroth>
but most of it is stuff covered in other tests
<beneroth>
only a few but very critical pieces
<beneroth>
as most is implemented in pil anyway :)
<Regenaxer>
Perhaps make same tests on old and new, and do a bytewise compare of the DB and journal files
<beneroth>
T
<beneroth>
right plio format is not changing, not like between pil32 and pil64
<Regenaxer>
yes
<beneroth>
so DB migration should also not be needed :D
<Regenaxer>
but for example the transaction logs exist only a short moment
<Regenaxer>
only if pil crashes inside a transaction
<beneroth>
for a start, running a replica from a pil64 DB under pil21 might be an idea (if you don't require the replica)
<beneroth>
that should cover the lowest level of binary file interactions, no?
<Regenaxer>
the "journal"
<Regenaxer>
thats easier
<beneroth>
T
<beneroth>
I enable the crash journal always
<Regenaxer>
On old pils I tested by exiting the program inside a transaction
<Regenaxer>
in commit
<beneroth>
can't rule out sudden power offs on most instances, even when they should be impossible
<Regenaxer>
I use it always on PilBox
<beneroth>
T
<Regenaxer>
yeah
<Regenaxer>
on a phone even more dangerous
<beneroth>
or do like tankf33der testing stuff.. a program which is doing DB stuff all the time with multiple processes, then kill -9 processes randomly
<Regenaxer>
also good
<beneroth>
once setup this could also be run on phone
<beneroth>
or we should just implement a jepsen test for pilDB :)
<Regenaxer>
jepsen?
<beneroth>
jepsen.io, testing framework (and a incredible guy behind it), mainly used to test distributed databases
<Regenaxer>
good
<beneroth>
but recently he did a test of postgresql and found some buggy special cases
<beneroth>
(non-distributed)
<Regenaxer>
Such a test would be useful indeed
<beneroth>
tests stuff like transaction isolation, dirty writes, dirty reads (all the possible flavours), and for distributed databases network delays, interruptions, splits, etc.
<beneroth>
problem is, all those standards and definitions are often not well-understood, too often even misunderstood by DBMS devs :)
<beneroth>
pilDB when used with transaction (also for reading) is the perfect text book serializable transactions.. because it's factually serialized, no fancy optimizing and trickery going on, we just pay the potential performance cost for it (if that matters is highly use case dependent)
<Regenaxer>
never measured, but relatively low overhead
<beneroth>
T
<Regenaxer>
the data is just written twice
<beneroth>
just locking, so no parallelism
<Regenaxer>
plus syncing
<Regenaxer>
yes
<Regenaxer>
Sync takes probably most time
<beneroth>
guess so
<Regenaxer>
it waits for the first write to be on disk
<beneroth>
pilDB architecture is pretty simple, but that rules out many categories of possible bugs
<Regenaxer>
yeah, hope so
<beneroth>
and the cost for that doesn't matter except for very busy cases
<beneroth>
I'm certain.
<Regenaxer>
I never noticed a real delay
<Regenaxer>
but it all depends of course
<beneroth>
well if two concurrent sessions both make (dbSync) ...(commit)/(rollback), they block each other.
<Regenaxer>
yep
<beneroth>
other DBMS go huge complicated ways to optimize that situation
<beneroth>
but if it actually matters is application specific
<Regenaxer>
T
<beneroth>
and I think we should have the tools to do partial locks, but the difficulty of course is to correctly manage them
<Regenaxer>
Heavy committing are costly anyway
<Regenaxer>
also for the IPC
<beneroth>
T, but other DBMS have here the same problems
<beneroth>
only so much you can optimize there away
<Regenaxer>
Instead of partial locks, separate DBs are easier
<beneroth>
most DBMS this days do only append data, no physical in place updates
<Regenaxer>
I see
<beneroth>
and then have some job periodically (or slowly concurrently) garbage collecting that mess again
<beneroth>
so the cost is moved to the future + interest + additional space cost
<beneroth>
WAL
<beneroth>
the main benefit is not even the performance gain when writing stuff, but isolation
<beneroth>
then you just read all data up to a certain journal entry and got a snapshoted state
<Regenaxer>
makes sense but complicated
<beneroth>
T
<beneroth>
the such achieved snapshot isolation is better than pilDB reads without a transaction, as you might read stuff from multiple commits in pilDB
<beneroth>
but in practice other DBMS don't do this by default, only when the programmer is explicitly asking for it the right way, by default they do some less careful trickery to gain performance and hope for the best
<Regenaxer>
:)
<beneroth>
and most DB users/programmers these days don't know about this, so the end result is anyway potentially risky :)
<beneroth>
so without understanding the DB stuff a programmer is probably better of with pilDB than mainstream databases
<beneroth>
I would suspect
<beneroth>
except for one very specific category of read errors
<Regenaxer>
if he follows some simple rules
<beneroth>
and maybe performance problems, but I believe them when I see them :)
<beneroth>
yeah well.. a programmer without insight/understanding will anyway not get so far with pil. not the target audience :)
<Regenaxer>
T
<beneroth>
what I said here only really applies to old-school SQL databases, the "nosql" "distributed" databases are mostly a big mess
<beneroth>
all kind of pitfalls, but most don't care
<beneroth>
a number of companies died because of it, but *shrug* :)
* beneroth
is going angry every week because firefox is leaking memory all over the place, suddenly crashes from out of memory, crashing from corruption via video content, and other unbearable stuff
<rob_w>
Regenaxer: i moved to picoLisp-20.6 .. all looked fine but i now get the following
<rob_w>
this /usr/bin/picolisp: symbol lookup error: /usr/lib/picolisp/lib/ext: undefined symbol: crypt
<rob_w>
but compile works with -lcrypto
<Regenaxer>
oha
<Regenaxer>
in ext
<Regenaxer>
and -lcrypt ?
<Regenaxer>
src/Makefile seems to use -lcrypt on Linux
<rob_w>
but how can the compile work
<rob_w>
hmm ok i will repatch and test
<Regenaxer>
hmm, the linker should complain?
<Regenaxer>
You do not use pil32's Makefile?
<rob_w>
yes but i have to patch it to make it work inside my toolchain , in that i changed to -lcrypto
<Regenaxer>
ok, still strange
<rob_w>
ok letme try again
<rob_w>
i am really confused about lcrypt vs lcrypto
<tankf33der>
one of them from openssl
<tankf33der>
you have it, right?
<tankf33der>
you have installed openssl-devel packages too, ok?
<rob_w>
yes .. -lcrypt works ! wtf
<rob_w>
at least i get further and get a different problem which is ok
<Regenaxer>
:)
<rob_w>
whis is only a wrong path on myside .. so that concludes my merge to 20.6 - from 16.6 ;)
<Regenaxer>
big step
<rob_w>
definitly , for me !
<Regenaxer>
:)
<rob_w>
if only i got rid of arm32 .. but its not the time yet ..
<rob_w>
mostly simple stuff .. stripping is done in my toolchain , CC is reset to the cross compiler .. only thing i worry is -fPIC but all works for now
<Regenaxer>
should be ok
<aw->
hmmm
<rob_w>
ok .. i worried as you once mentiond you have some hardcode conventions about addresses and pointers ..
<Regenaxer>
only the size
<aw->
rob_w: hi
<Regenaxer>
32 bits on pil32
<rob_w>
hi aw-
<aw->
i'm also running arm64, didnt have to make changes
<aw->
perhaps you're missing libssl
<rob_w>
i am compiling via yocto toolchain , so strip and CC is handled inside
<aw->
oh ok
<rob_w>
so it just needs to be abit more abstract compared to a "native" compile
<aw->
what are you doing with yocto?
<aw->
are you using balena?
<Regenaxer>
rob_w, would LLVM make sense on your system?
<rob_w>
no idea
<Regenaxer>
I'm in the hope that Makefile gets more portable then (?)
<rob_w>
aw-: i am building images for our custom boards with it .. basicly a simplified poky image with our stuff added
<rob_w>
Regenaxer: ive no idea about llvm and friends
<Regenaxer>
pil21 is based on it
<Regenaxer>
I got the impression that all world moves to llvm
<rob_w>
well i am happy to get to your latest stuff , so maybe i will find some time to grab you and try to get it work if that is your latest/best
<rob_w>
but for today iam logging off .. need some shadow and some iced drinks .. cya guys and thx alot again
<Regenaxer>
Good idea ;)
<Regenaxer>
su
<Regenaxer>
cu
<Regenaxer>
tankf33der, strange the we have "ASM = clang" in Makefile
rob_w has quit [Quit: Leaving]
<Regenaxer>
Should be "CC = clang", right?
<Regenaxer>
and ASM should be llvm-as
<Regenaxer>
can later be opt -O3
<tankf33der>
yea, i told you. in general not a problem
<Regenaxer>
yes, but confusing :)
<Regenaxer>
I change it
<tankf33der>
next: -lffi you should take via pkg-conf too
<tankf33der>
and for readline headers and -I opt too. cant be compiled on non linux. freebsd failed.
<tankf33der>
all saved to issues, later.
<tankf33der>
afk.
<Regenaxer>
ok
<Regenaxer>
Let me know
orivej has joined #picolisp
jibanes_ is now known as jibanes
xkapastel has joined #picolisp
tankf33der_ has joined #picolisp
orivej has quit [Ping timeout: 264 seconds]
tankf33der has quit [*.net *.split]
Blue_flame has quit [Write error: Connection reset by peer]
orivej has joined #picolisp
Blue_flame has joined #picolisp
tankf33der_ has quit []
tankf33der_ has joined #picolisp
tankf33der_ has quit [Client Quit]
tankf33der_ has joined #picolisp
tankf33der_ has left #picolisp [#picolisp]
tankf33der has joined #picolisp
<tankf33der>
i've completed run 860 tasks on gc+.
<tankf33der>
list of failed functions:
<tankf33der>
sect, fish.
<tankf33der>
4clojure, rng, 5 crypto functions, two years from adventofcode - all passed and never run before.
<Regenaxer>
So only sect and fish failed?
<tankf33der>
yeap. they will wait for you.
<Regenaxer>
oh ;)
<Regenaxer>
I looked at 'fish', but it looks correct ;)
<Regenaxer>
Can you reproduce an error with 'fish' alone?