rob_w has quit [Read error: Connection reset by peer]
orivej has quit [Ping timeout: 260 seconds]
andyjpb has quit [Ping timeout: 276 seconds]
karswell_ has quit [Read error: No route to host]
karswell_ has joined #picolisp
pierpa has quit [Quit: Page closed]
<yunfan>
which library do you use for you backend service?
rob_w_ has quit [Read error: Connection reset by peer]
<Regenaxer>
'$ pil +' has all you need
<Regenaxer>
plus @lib/too.l loaded only for maintenance, not production runtime
rob_w has joined #picolisp
<xificurC>
Regenaxer: Hi! Is it OK to post small findings like these here? I would fix typos myself but I don't think you made it world accessible, right
<xificurC>
psh has #!bin/picolisp as its interpreter, which means it expects me to be in a folder where picolisp can be relatively found at ./bin/picolisp
<Regenaxer>
There are two psh's I think (as there are also two pil's)
<xificurC>
eh?
<Regenaxer>
bin/psh uses the relative path
<xificurC>
I went through INSTALL and did (cd src64 && make) then (cd src && make tools gate)
<Regenaxer>
But /usr/lib/picolisp/bin/psh uses /usr/bin/picolisp
<Regenaxer>
yes, this is good
<xificurC>
after this I only have one psh
<xificurC>
two pils ./pil and ./bin/pil
<xificurC>
psh is at ./bin/psh
<Regenaxer>
ok, yes, the global one is in the distro
<Regenaxer>
I use it as $ bin/psh <port> or $ bin/psh <port> <sesId>
<xificurC>
what distro
<Regenaxer>
Debian, Ubuntu etc.
<xificurC>
ah ok
<xificurC>
why didn't it connect though -_-
<Regenaxer>
Same with 'pil'
<xificurC>
I have the project running with - pil @lib/http.l @lib/xhtml.l @lib/form.l --server 8080 project.l +
<Regenaxer>
yes, good
<xificurC>
if I run bin/psh 8080 it exits straight away. doc/app.html says I should get a prompt
<Regenaxer>
You either call pil with a relative path, or have it in /usr/bin/pil
<Regenaxer>
I always have all local
<Regenaxer>
./pil ...
<Regenaxer>
then bin/psh ...
<Regenaxer>
in fact I have ./bin in my PATH, so just $ psh works too
<xificurC>
ok, so, from the beginning :) I created a folder ~/tmp/picoapp/ and created project.l in it
<Regenaxer>
They are just scripts, check the pathes inside
<xificurC>
are you saying you always develop where your pil is installed?
<xificurC>
or that you copy the whole thing over to a new place and start developing with the whole thing?
<Regenaxer>
Yes, I do, but this is not necessary
<Regenaxer>
I have a local install
<Regenaxer>
and *all projects* relative to that
<Regenaxer>
./pil app/main.l ...
<Regenaxer>
./pil app2/main.l ...
<Regenaxer>
./pil app3/main.l ...
<Regenaxer>
But you may also install picolisp somewhere else
<Regenaxer>
then
<Regenaxer>
/path/to/pil app/main.l ...
<Regenaxer>
../path/to/pil app/main.l ...
<Regenaxer>
etc
<Regenaxer>
as you like
<Regenaxer>
Just look into the scripts, they are simple
<xificurC>
I added a (prinl "HI") and (prinl "BYE") into bin/psh, they do print
<Regenaxer>
So in your case, where did you install picolisp?
<xificurC>
so psh starts OK
<Regenaxer>
sure
<Regenaxer>
where did you install picolisp?
<xificurC>
I untared at ~/.local/lib/
<xificurC>
then did a bunch of sudo ln -s, as is in the manual
<Regenaxer>
ok, but not needed
<Regenaxer>
you can call it simply with relative path
<xificurC>
I understand, what I don't understand is why does psh exit?
<Regenaxer>
ok
<Regenaxer>
Did you supply a port?
<Regenaxer>
the one your app is listening at?
<xificurC>
yes, app is running as pil @lib/http.l @lib/xhtml.l @lib/form.l --server 8080 project.l +
<xificurC>
and I'm starting psh as bin/psh 8080
<Regenaxer>
ok
<Regenaxer>
I just tried with global: $ pil app/main.l -main -go -wait
<Regenaxer>
then: $ psh 8080
<Regenaxer>
Works
<Regenaxer>
(though I usually don't use the global one (from the Debian distro) for production
<xificurC>
I believe you :)
<Regenaxer>
:)
clacke[m] has quit [Ping timeout: 260 seconds]
<Regenaxer>
So if you installed in .local, and want to use it globally
<Regenaxer>
you need to put links in /usr/bin and /usr/lib as described
<xificurC>
I'll wipe the installation and try one more time
fwirt[m] has quit [Ping timeout: 260 seconds]
<xificurC>
locally
<Regenaxer>
ok
<Regenaxer>
You can also keep what you have, and extract another one somewhere else
<Regenaxer>
You can have as many different local installs as you like
<Regenaxer>
different versions for example, they dont interfer
<xificurC>
I get that, thanks :)
<xificurC>
I just like to keep things clean
<Regenaxer>
:)
<Regenaxer>
yeah
<xificurC>
ok, clean install works
<xificurC>
heisenbug
<xificurC>
can I pick your brain on something Regenaxer?
<Regenaxer>
yes :)
<xificurC>
I was reading about select on my way to work
<Regenaxer>
in the train I hope, not in the car ;)
clacke[m] has joined #picolisp
<xificurC>
in the traffic jam :)
<Regenaxer>
ah :)
<xificurC>
but I got called off to fix something quickly, be back soon (hopefully)
<Regenaxer>
good
fwirt[m] has joined #picolisp
<xificurC>
well I have to wait for a colleague to fix something first :)
<Regenaxer>
perfect ;)
<xificurC>
Regenaxer: ok, so you have a huge select there - (select (@@) ((nr +Ord @Nr) (dat +Ord @Dat) (nm +CuSu @Cus (cus +Ord)) (ort +CuSu @Ort (cus +Ord)) (nm +Item @Item (itm +Pos) ord) (nm +CuSu @Sup (sup +Item) (itm +Pos) ord) )
<xificurC>
let's say I understand the clauses
<Regenaxer>
ok
<xificurC>
my question is, do you *need* all clauses to find each order?
<xificurC>
doesn't the first (nr +Ord @Nr) already find all of them?
<Regenaxer>
They are combined via OR
<Regenaxer>
yes, would find, but there may be more criteria
<xificurC>
via OR? I read the generators stop when the first one finishes
<Regenaxer>
Only searching the number might result in a complete traversal of the DB
<Regenaxer>
yes
<Regenaxer>
if also eg part of the name is given, the results may be found faster
<Regenaxer>
all these indexes are searched in "parallel"
<Regenaxer>
not really parallel, but one after the other step by step
<Regenaxer>
Whichever index is more specific will terminate the whole search if exhausted
<Regenaxer>
So it is an AND, sorry
<Regenaxer>
all criteria must be fulfilled
<xificurC>
sorry I got called off again, reading
<Regenaxer>
no hurry
<xificurC>
OK, so you're giving more generator clauses basically to optimize the search/traversal
<Regenaxer>
yes
<Regenaxer>
The user enters all criteria he is interested in
<xificurC>
if you knew for sure 1 of them is the fastest every time you would only put that 1 down
<Regenaxer>
yes, but this depends on how precise the user is
<xificurC>
ok
<Regenaxer>
eg if he knows the exact number, the search terminates immediately
<xificurC>
ok, I get it now :) Now for the "little report" part
<xificurC>
first you have a typo, you talk about item #2 but say #3 once
<xificurC>
"namely all positions with item #3 OR from the year 2007"
<Regenaxer>
ok, where is that?
<Regenaxer>
select.html ?
<xificurC>
yes
<xificurC>
so I take it this example can be explained the same way - both generator clauses would traverse *all* +Pos's, you just supply both in order to optimize the query
<Regenaxer>
Well, they do never traverse *all*, but only those few that match the pattern
<xificurC>
actually the fact that the first exhausted generator terminates the traversal means by definition that you cannot supply a generator that would traverse only a subset of the objects you need to actually look at
<Regenaxer>
No, it is always just a subset
<xificurC>
because if that was the one to terminate you would be left with a partial match
<Regenaxer>
*all* would mean NIL as pattern, but these indexes are not even looked at
<Regenaxer>
Each criterium typically only searches a small part
<Regenaxer>
I change #3
mtsd has joined #picolisp
<xificurC>
yes but (nr +Item @Nr (itm +Pos)) and (dat +Ord @Year pos)) should generate a superset of the objects you need to look at
<xificurC>
if gen#1 generates (1 .. 10) and gen#2 generates (5 .. 10) and I need to look at (3 .. 6) I'm in trouble
<xificurC>
or is this an abstract example that doesn't really happen in the real world
<Regenaxer>
No, they are not all generated
<Regenaxer>
Whe example is fine, it is the search :)
<Regenaxer>
It does not generate all results
<xificurC>
I understand the generators are not exhausted
<xificurC>
this is not SQL :)
<Regenaxer>
yes
<xificurC>
but in my above example
<Regenaxer>
Most hits are immediately rejected
<Regenaxer>
the "sets" are never generated
<xificurC>
if gen#2 terminates the search while gen#1 only got to 4 noone has checked 5
<Regenaxer>
All objects which do not match *all* criteria are discarded
<Regenaxer>
The generators return one object after the other
<Regenaxer>
eg first #1
<Regenaxer>
as this does not match the date, the date index is tried
<Regenaxer>
now we get the right date, and if the number is OK, we have a hit
<Regenaxer>
In this way after a few iterations only the *shortest* range is searched
<Regenaxer>
It uses a LRU scheme
<xificurC>
I can see I'm not making myself clear, so let me think of an example that would get us to synchronize
<Regenaxer>
When one index returns a "good" object, it is continued
<Regenaxer>
ok
<Regenaxer>
The point is that the indexes are tried alternatingly
<Regenaxer>
so the one with the most *total* matches is continued
<Regenaxer>
If one index covers 90% of the DB, and one only 1% (more specific), then the latter one will be exhausted quickly
<Regenaxer>
So the 1% one either finds a match quickly, or terminates without *any* hit, because the results dont fit the 90% criterium
<xificurC>
all I was trying to wrap my head around is that you can't write a generator clause the is 1% specific when you need to look through 90%
<Regenaxer>
no
<xificurC>
because the generators are combined with an AND
<xificurC>
right?
<Regenaxer>
we don't look through the 90%
<Regenaxer>
lets say you have a million items
<Regenaxer>
number 1 ... 1000000
<Regenaxer>
and names "A" ... "Z"
<Regenaxer>
search is (7 - 9) and (D - Y)
<Regenaxer>
so it starts with eg D
<Regenaxer>
no success in the first hit, it has number 123456
<Regenaxer>
so 7 is tried
<Regenaxer>
it is probable that the name is (D - Y)
<Regenaxer>
so the numeric one is continued
<Regenaxer>
The three values (7 - 9) will exhaust soon, as *most* returned objects will be (D - Y)
<Regenaxer>
In the end the search will just look at objects (7 - 9)
<xificurC>
I understand that. Let me give this one more stab :)
anddam has quit [Quit: WeeChat 2.2-dev]
<Regenaxer>
ok
<xificurC>
let's say in the report example there's another class +VipOrd for VIP orders
<xificurC>
obviously this is a small subset of the orders
<Regenaxer>
I dont have the example in my mind though
<xificurC>
app/er.l
<xificurC>
there's +Ord(ers) which have +Pos(itions) that point to +Itm(s)
<Regenaxer>
right
<xificurC>
so there's a specialized order +VipOrd
<Regenaxer>
I just did not get the VIP
<Regenaxer>
ok, understand
<xificurC>
I'm making this up
<xificurC>
:)
<Regenaxer>
yeah
<xificurC>
so, I cannot add a generator clause (dat +VipOrd @Year pos) to the search at this point, right?
<xificurC>
because if that's 10 orders from a million I won't get a full report
<Regenaxer>
So it could be a (rel vip (+Ref +Bool)) right?
<Regenaxer>
then you search only these 10 objects
<xificurC>
yes, or a few more from the other generator clauses, right?
<Regenaxer>
yes
<Regenaxer>
You can also make a subclass +VipOrd
<Regenaxer>
it would have its own index
<Regenaxer>
with only 10 entries then
<xificurC>
that's what I meant, yes
<xificurC>
a subclass
<Regenaxer>
yes, perfect
<xificurC>
and that's what I meant when saying "all generator clauses have to be a superset of what you want to look at"
<xificurC>
if I want a report of *all* orders I cannot put a generator in that only looks at VIP orders
<Regenaxer>
I'm not sure if "superset" is the right term
<Regenaxer>
you can combine
<xificurC>
I'm sure it is not since it didn't make it clear for you what I'm talking about
<Regenaxer>
you search for +Ord in some clauses and for +VipOrd in others
<Regenaxer>
practically I would go simply for a 'vip' bool relation
<Regenaxer>
but a c
<Regenaxer>
lass is also fine
<xificurC>
yeah, would make more sense, I was just desperate for an example to demonstrate :)
<Regenaxer>
For example, I normally have such subclasses for invoice types
<Regenaxer>
Then search is faster if the type is known
<Regenaxer>
smaller index
<xificurC>
yeah
<Regenaxer>
I use +Ref2 to get a small and a big index
<Regenaxer>
Each number gets entered in 2 index trees
<xificurC>
reading the first clause and looking at the definitions in app/er.l I see I misunderstood something. (nr +Item @Nr (itm +Pos)) , this is saying to use an index on +Pos or +Item?
<Regenaxer>
it uses both
<Regenaxer>
first searches the number, then the position from the item
<xificurC>
but an +Item has no link to a +Pos, only the other way around
<Regenaxer>
yes, but a +Ref
<xificurC>
I'm trying to imagine this as a plain old loop over lisp data
<xificurC>
I need to loop through +Pos's and find their number through the itm reference
<xificurC>
I would read that generator as: walk through the itm index of Pos and bind @Nr to the item number by following the itm reference to the +Item object and taking the nr field
<xificurC>
does that sound correct?
<Regenaxer>
hmm, (nr +Item @Nr (itm +Pos)) means to search for the number first
<Regenaxer>
for each found item travers the positions where it is used
<Regenaxer>
possibly thousands of positions if the item was sold many times
<Regenaxer>
It is a multiplicator
<Regenaxer>
giving all orders where this item appears in
<xificurC>
ah ok, and since +Pos is indexed by nr too basically the lookup will be fast
<xificurC>
in +Item it is a traversal, in +Pos it is a lookup of the nr being traversed right now
<Regenaxer>
+Pos has no number index, but one from +Ord
<Regenaxer>
a +Joint to be exact
<xificurC>
(rel itm (+Ref +Link) NIL (+Item))
<Regenaxer>
yes, so it is two traversals
<xificurC>
so there's 2 indexes?
<Regenaxer>
For each item found by number, find all positions that refer to it
<Regenaxer>
Yes, (rel itm (+Ref +Link) is a tree too
<xificurC>
ok
<Regenaxer>
+Link from pos to item, and index back
<xificurC>
so if walking nr is O(m) and looking up nr in itm is O(log(n)) then the whole traversal is O(m*log(n))?
<Regenaxer>
walking nr is also O(log(n))
<xificurC>
how can you walk n items in log(n) time? :)
<Regenaxer>
it is a B-Tree
<Regenaxer>
both are B~Trees
<xificurC>
yes but I'm looking for the complexity of a full walk
<Regenaxer>
for all orders referring to that item?
<xificurC>
walking '(1 2 3 4 5) is O(n). walking an aray is O(n). walking a btree is O(n) too
<Regenaxer>
it is O(n) * O(n) then
<xificurC>
hm, but you said "For each item found by number, find all positions that refer to it"
<Regenaxer>
yes
<xificurC>
that sounds like (for X Items (lookup X Positions))
<xificurC>
for is O(n) but lookup is O(log(n))
<Regenaxer>
Yes. It is the product of the number of items by the number of positions
<xificurC>
it's like N times do a log(N) operation
<Regenaxer>
The searched number of tree nodes is log(N)
<Regenaxer>
But in both cases you get O(n) results on the level above
<Regenaxer>
the iteration of the pos tree for a given item is O(n)
<Regenaxer>
gives only the exact results
<Regenaxer>
Same as getting all items with a given number
<Regenaxer>
So both operations are O(n)
<xificurC>
so lookup of an +Ord by itm is O(n)?
<Regenaxer>
while O(log(n)) tree nodes are traversed
<Regenaxer>
yes
<xificurC>
why, isn't a lookup in a btree O(log(n))?
<Regenaxer>
from pos -> ord it is O(1)
<xificurC>
crap, lookup of +Pos by itm I meant
<Regenaxer>
yes, in the tree
<Regenaxer>
As I said: while O(log(n)) tree nodes are traversed
<Regenaxer>
no
<Regenaxer>
even less
<Regenaxer>
the hits are all in the same node
<Regenaxer>
so only O(n)
<Regenaxer>
no
<xificurC>
but the point of an index is to make the search faster. If it is O(n) it might have as well be unindexed
<Regenaxer>
near O(1) I mead
<Regenaxer>
yes :)
<Regenaxer>
near O(1) I mean
<xificurC>
I think balanced tree lookups are O(log(n))
<Regenaxer>
The access to the first item is log(n)
<Regenaxer>
then it is O(1)
<xificurC>
so overall O(log(n))
<xificurC>
big-O is hard :)
<Regenaxer>
I would say less
<xificurC>
well, you mean in this case there's a 1 to many relationship
<Regenaxer>
the first item is log(n), then the node is in memory and traversed linearly
<xificurC>
so you find the first in O(log(n)) and the rest are O(1)
<Regenaxer>
One number to one item, then many positions
<xificurC>
unfortunately in big-O I think that's still O(log(n)) :)
<xificurC>
yes
<Regenaxer>
yes, just physically it is less
<xificurC>
yes, I understand
<xificurC>
maybe it could be O(log(n)/k) where k is the average number of +Pos's for 1 item
<Regenaxer>
right
<Regenaxer>
average
<xificurC>
thanks, this was very helpful :)
<xificurC>
gotta get some work done now :(
<Regenaxer>
yes, me too :)
<xificurC>
thank you for your time and patience, hope our communication issues didn't get you mad at any point
<Regenaxer>
Not at all! :)
<Regenaxer>
I continued meanwhile with other stuff here too
<xificurC>
I didn't, so stop bragging :p
<Regenaxer>
:D
orivej has joined #picolisp
<yunfan>
Regenaxer: do we have any http app container solution? like python's wsgi container
<yunfan>
Regenaxer: i found picolisp's http.l got quite a high benchmarks when i use it in single process
<yunfan>
but when i dont saw any multi worker tools, so i could cheat in python using some multiworker mangers, like gunicorn
<Regenaxer>
Hi yunfan, not sure what you mean with "app container"
<yunfan>
Regenaxer: well do you know fastcgi or wsgi?
<Regenaxer>
nope
<xificurC>
yunfan: is your goal getting parallel (multicore) execution?
<yunfan>
xificurC: yes, many performance could gained by simply run many process of the same service
<Regenaxer>
I'm really bad when it comes about knowing other systems. Too lazy to look around, found it always easier going ahead and writing just what I needed ;)
<xificurC>
yunfan: the solution used here is different then. Picolisp achieves multicore by forking processes and communicating back to the parent process via an IPC mechanism if necessary
andyjpb has joined #picolisp
<Regenaxer>
yeah
<xificurC>
if you're building an HTTP server then each session gets a new (forked) process so you are already parallel
<yunfan>
xificurC: well almost every enterprises use nginx or haproxy in their network's edge for handle incoming request, and then dispatch the request to different logic server.
<Regenaxer>
This is done by httpGate in pil
<yunfan>
some just use reverse proxy, but since http 1.x is a little prolixity, so someone found some litely protocol like fastcgi and wsgi
<xificurC>
out of my scope at this point :) Not sure if you could use nginx instead of httpGate
<Regenaxer>
There are some people who did afaik
<Regenaxer>
But I don't know nginx
<xificurC>
these are different topics though. One is getting parallel (multicore) on 1 server. The other is load balancing across multiple servers
<yunfan>
i am quite sure i need to choose from nginx or haproxy, cause i am not work alone
<xificurC>
both can be achieved
<yunfan>
xificurC: yes, load balancing
<Regenaxer>
It should be possible if you let httpGate listen at another port and forward from another proxy
<xificurC>
yunfan: I guess you can do load balancing with nginx/haproxy and have multiple servers running the same picolisp application, each accessing one pilDB