<DKordic>
I meant to say, there might be a pattern to the madness :) .
<beneroth>
madness is often interpreting patterns where there is none - this is a pattern :)
freemint has joined #picolisp
<beneroth>
Regenaxer, are you here?
<beneroth>
(lup) seems not to work when a transient is used to hold the idx tree. Is this indented?
<beneroth>
-> NIL
<beneroth>
: (idx 'A)
<beneroth>
: (idx 'A (1 . b) T)
<beneroth>
-> NIL
<beneroth>
: (idx 'A (2 . c) T)
<beneroth>
-> ((1 . b) (2 . c))
<beneroth>
: (lup A 2)
<beneroth>
-> (2 . c)
<beneroth>
: (idx '"A" (1 . b) T)
<beneroth>
-> NIL
<beneroth>
: (idx '"A" (2 . c) T)
<beneroth>
-> NIL
<beneroth>
: (idx '"A")
<beneroth>
-> ((1 . b) (2 . c))
<beneroth>
: (lup '"A" 2)
<beneroth>
-> "A"
<beneroth>
tested with both pil 19.7.6 and 20.3.25
<beneroth>
wanted to use an idx scoped to a file. will use flat assoc list as workaround.
<Regenaxer>
The reason is probably that the value is not NIL initially
<Regenaxer>
Try (off "A") first
<Regenaxer>
hmm, no
<Regenaxer>
"A" looks clean
<beneroth>
tried
<Regenaxer>
Any var should be OK
<Regenaxer>
also a cell
<beneroth>
so, a bug in (lup) ?
<Regenaxer>
so transient should not matter
<beneroth>
oh wait, lup doesn't take a 'var
<beneroth>
maybe that was the mistake
<Regenaxer>
ah, you quoted it
<Regenaxer>
(lup '"A" 2)
<Regenaxer>
'lup' takes the tree, not the var
<beneroth>
yep
<beneroth>
that as the mistake
<Regenaxer>
Good
<beneroth>
so BCK issue, not bug
<Regenaxer>
:)
<beneroth>
BCK = between chair and keyboard ;-)
<beneroth>
strange though that (lup) returns "A") then
<Regenaxer>
yeah
<beneroth>
: (lup 'F)
<beneroth>
-> F
<beneroth>
consistent though :)
<Regenaxer>
it returns atomic data as they are
<Regenaxer>
expecting NIL for an empty list
<Regenaxer>
This is often in pil
<Regenaxer>
it returns what it gets
<beneroth>
shit in shit out
<beneroth>
okay
<Regenaxer>
true
<beneroth>
it is alright, as long as there is no stack corruption
<Regenaxer>
yep
<beneroth>
but in my original case I didn't make the 'var mistake
<beneroth>
well not directly
<beneroth>
I did (and "A" (lup @ ...))
<beneroth>
but then (lup) always returns "A"
<Regenaxer>
because "A" is not NIL?
<beneroth>
I do (off "A") at the start of the file
<Regenaxer>
ie (val '"A") -> "A"
<Regenaxer>
ah, ok
<Regenaxer>
then (and "A" (lup @ should be fine
<beneroth>
I do
<beneroth>
(if (and "*MuIdx" (lup "*MuIdx" (; M uuid)))
<beneroth>
(cdr @)
<beneroth>
"*MuIdx" -- List expected
<beneroth>
!? (cdr @)
<beneroth>
? "*MuIdx"
<beneroth>
-> "*MuIdx"
<beneroth>
is the debug repl within the file transient scope?
<Regenaxer>
yes, as long as you don't hit Enter
<Regenaxer>
Enter leaves the repl (and it starts new)
<beneroth>
(off "*MuIdx") is the second statement in that file, after `(once T)
<Regenaxer>
ok
<beneroth>
aah
<beneroth>
I do a (load) afterwards
<beneroth>
so my NIL setting is lost
<Regenaxer>
yesss
<beneroth>
yep now it works!
<beneroth>
case closed
<Regenaxer>
Perfect :)
<beneroth>
thanks for the support :)
<Regenaxer>
Welcome :)
<freemint>
Regenaxer, if you are still around. Can you give a brief overview/pointer how PilDB and multiple processes accessing it works again?
<beneroth>
I can give you that, too
<beneroth>
the essential point is that: initial picolisp process opens the database using (pool), and then usually gets reduced to doing message handling and accepting new connections
<beneroth>
new connections are handled in forked child processes
<beneroth>
by default, picolisp established two pipes between each child and the parent (one for each sending direction)
<freemint>
ok, so if i want to access data in a child, what happens?
<beneroth>
(this is way the parent might run into the standard linux limit on the number of open files / file descriptors in use, as it opens two file descriptors per child - effectively you have a limit of ca. 512 concurrent children - this limit can be increased)
<beneroth>
child (as in standard linux forking) inherits the file descriptor(s) to the open database file(s)
<beneroth>
database functions like (db) and (collect) etc. load external symbols - everything in the database is external symbols
<beneroth>
when an external symbols is accessed in picolisp code, it is first attempted to read from memory, if not loaded into memory it will be loaded into memory automatically
<beneroth>
loaded from the database file8s), that is
<beneroth>
so you have a cache of external symbols (which can be records, but also b-tree index parts)
<beneroth>
per child process = database session
<beneroth>
you can start an transaction using (dbSync) - among other things (explained later) it sets a write-lock on the database
<beneroth>
parallel (sibling) processes can still read from the database files, but not write
<beneroth>
(commit 'upd) can be called to finish the transaction
<beneroth>
(commit) in itself = write changes on external symbols (in memory/cache) into database file(s)
<beneroth>
now when working with multiple processes (concurrent database sessions), this 'upd argument to (commit) is important!
<freemint>
How does Pil keep track what values are modified?
<beneroth>
it basically tells commit to call (apply 'upd Lst) in sibling processes, wherein Lst is the list of changed external symbols
<beneroth>
dirty bit on external symbol pointer
<beneroth>
see doc64/structures
<beneroth>
the sibling processes receive this 'upd message when the do (wait) - which is also called within (dbSync)
<beneroth>
'upd is per standard just an alias for 'wipe, basically it calls 'wipe on the external symbols received from the sibling process
<freemint>
where wipe purges local cache
<beneroth>
'wipe = delete from memory, so basically cache eviction
<beneroth>
exactly
<beneroth>
the messaging between the siblings is the same mechanism as in (tell)
<beneroth>
basically it is picolisp code sent to the parent process (who is/should be most of the time within (wait))
<beneroth>
who then just resends the message to all other picolisp children
<beneroth>
and they read the pipe when they are in (wait) - during (dbSync), but also during any I/O wait or so...
<beneroth>
through this design the I (Isolation) of ACID is achieved: each database session = another child = with its own memory/database cache
<beneroth>
questions?
<beneroth>
brb
<freemint>
In how far is sharing the same FD necessary. I might want to port the picolisp DB to another programming language, which doesn't work via fork but spawn "empty processes instead".
<beneroth>
not necessary
<beneroth>
but the communication between the siblings has to established
<freemint>
over a parent, for example
<beneroth>
aye, but ofc doesn't have to be via a parent :)
<freemint>
Do you see ways how PilDB could be speed up with shared memory and atomic ops between processes?
<beneroth>
for locking there are actually kinda 2 layers: 1) (lock) function within picolisp - that one is used within (dbSync), and is actually a cooperative lock = the other check if to obtain the lock
<beneroth>
the other locking is happening when (commit) is writing the actual database files..then they're locked in the filesystem (using fcntl)
<beneroth>
so a non-cooperative process (being a sibling or also completely disconnected) may read the database file at any time (unless it is (very short) not available because of fcntl)
<beneroth>
but shouldn't write it
<freemint>
Ok, how is lock done in picolisp without shared memory?
<beneroth>
the newer picolisp function (blk) allows reading from a foreign database file
<beneroth>
look into (lock)
<beneroth>
I actually don't know how that one works, if it is communicated via the tell-channels via the parent or via another way.. I guess it also uses the tell mechanism
<beneroth>
but I really don't know
<beneroth>
maybe check out the source of (lock)
<beneroth>
afk
<freemint>
thanks
<Regenaxer>
Thanks beneroth! Good explanations!
<Regenaxer>
Good night
alexshendi has quit [Quit: Yaaic - Yet another Android IRC client - http://www.yaaic.org]