ChanServ changed the topic of #picolisp to: PicoLisp language | Channel Log: | Check also for more information
xkapastel has quit [Quit: Connection closed for inactivity]
razzy has quit [Read error: Connection reset by peer]
orivej has joined #picolisp
<Regenaxer> tankf33der:
<Regenaxer> - pil32
<Regenaxer> 42.564 sec
<Regenaxer> 27.654 sec
<Regenaxer> 4.713 sec
<Regenaxer> - pil64
<Regenaxer> 11.462 sec
<Regenaxer> Always the second run is a lot faster
<Regenaxer> So this can be fixed with (gc <size>)
<Regenaxer> We expected it to be slower and bigger, as a lot of symbols are interned now
<Regenaxer> Should we make an option to *not* intern again? E.g. with an optional argument?
_whitelogger has joined #picolisp
<Regenaxer> Could be a 5th arg 'FI' (= flag intern)
<tankf33der> morning
<tankf33der> this is not second run
<tankf33der> there is two bench in code. check again
<Regenaxer> Ah_ I see
<Regenaxer> ok
<Regenaxer> Anyway, interning one million symbols takes space and time, which may be unnecessary in most use cases
<tankf33der> i see
rob_w has joined #picolisp
<Regenaxer> eg:
<Regenaxer> (if (and (> DX 26) (not FI))
<Regenaxer> (box)
<Regenaxer> (intern ...
<Regenaxer> ?
<Regenaxer> Or lets just keep it as it is?
<Regenaxer> I never had such big grids
<tankf33der> performance not the main problem
<tankf33der> task failed completly
<Regenaxer> yes, but why or where?
<tankf33der> i expect 400000, return 0
<tankf33der> not found yet
<tankf33der> looks like use and with doest not do the job anymore
<Regenaxer> strange
<Regenaxer> only because of interned symbols? Can't be, something else must be broken
<Regenaxer> What if the naming is wrong?
<Regenaxer> Then symbols might clash if they have the same name by accident
<Regenaxer> Ha! Looks like this is indeed the case!
<Regenaxer> This gives:
<Regenaxer> 2224
<Regenaxer> 1002223
<Regenaxer> Should be one more, right?
<Regenaxer> So one symbol clashed
<Regenaxer> I do a brute-force search for that symbol
<Regenaxer> Haha, I know!!!
<Regenaxer> It is not really a bug, but indeed a name clash
<Regenaxer> However not *inside* the grid, but with existing symbols
<tankf33der> ok
<tankf33der> sounds good.
<Regenaxer> To be exact, it is 'if2'
<tankf33der> :)
<Regenaxer> So the whole thing was perhaps not a geod idea
<tankf33der> also i had idea to capitalize grid's symbols
<Regenaxer> yeah, but we will never be sure
<Regenaxer> Even the old grid was not safe
<Regenaxer> Eg 'd', 'e', 'u'
<Regenaxer> Hmm, no
<Regenaxer> d1, e1 ...
<Regenaxer> Using a namespace would solve it
<Regenaxer> iirc I recommended that in the mailing list
<Regenaxer> on Jul 1st
<Regenaxer> Current version is unusable
<Regenaxer> Shall we revert and forget the matter?
<tankf33der> yea. capitalized symbols helped.
<Regenaxer> ok, but looks ugly e.g. in chess
<Regenaxer> and violates naming convention
<Regenaxer> a1 etc. are globals
<tankf33der> write to ML what happened and revert.
<Regenaxer> OK, yes
<tankf33der> afk.
<Regenaxer> Thanks a lot!
<Regenaxer> Done
beneroth has joined #picolisp
<beneroth> Hi all
<Regenaxer> Hi beneroth!
aw- has joined #picolisp
<beneroth> Regenaxer, I have a entity with 3 +Ref properties, no +Key property. I like to have a pilog query to get the combined objects from all 3 indexes. I figure select predicate is the right for this, using a combinated generic clause without any filtering. 1) select predicate ensures uniqueness of results, right? 2) how to do a combinated generic clause without a filter parameter?
<beneroth> e.g. (? (select (@@) ((name +Town)))) and (? (select (@@) ((zip +Town)))) combined into a single quer
<beneroth> query
<Regenaxer> Optimal would be +Aux +Ref
<Regenaxer> then only a single tree access
<aw-> Regenaxer beneroth hi!
<beneroth> hi aw- !
<Regenaxer> (if the combination is unique, else a 'collect')
<Regenaxer> hi aw-!
<aw-> brb
aw- has quit [Quit: Leaving.]
<beneroth> Regenaxer, I want pilog for doing (dump), and also only for that I need the combined output...
<beneroth> I see your point about +Aux.
<Regenaxer> ok, no +Aux then
<beneroth> T
<Regenaxer> Why do you want to avoid the filters?
<beneroth> I simply want all objects
<beneroth> for dump
<Regenaxer> Ah, not AND ?
<beneroth> T, OR :)
<Regenaxer> combine the indexes sequentially?
<Regenaxer> yes
<beneroth> T, and ensure unique output
<Regenaxer> thaws easy
<beneroth> from the select predicate documentation: ((tel +CuSu @Tel mob +CuSu @Tel))
<Regenaxer> (select (@@) ((ref1 +Cls ref2 +Cls ref3 +Cls)) ...
<beneroth> like this, but without @Tel filter
<Regenaxer> I see
<Regenaxer> if @Tel is empty it is ignored
<beneroth> aye, I tried that
<beneroth> (select (@@) ((ref1 +Cls ref2 +Cls ref3 +Cls) is not working for me
<Regenaxer> So 3 (db ...) in sequence?
<Regenaxer> select is overkill
<Regenaxer> yes, needs @Var :)
<beneroth> would work, but unique, and (dump) wants a single pilog query, no? :)
<Regenaxer> right
<Regenaxer> ok, then this will work
<Regenaxer> (select (@@) ((ref1 +Cls (NIL . T) ref2 +Cls ...
<Regenaxer> so the value is not empty and thus not skipped
<beneroth> already tried.
<beneroth> (select (@@) ((ref1 +Cls (NIL . T))) and (select (@@) ((ref2 +Cls (NIL . T))) works, but not combined
<Regenaxer> I mean combined
<Regenaxer> but better is to drop into Lisp
<Regenaxer> and do 'seq'
<Regenaxer> Then you get all objects regardless of index
<beneroth> I did: (? (select (@@) ((name +Town (NIL . T) zip +Town (NIL . T)))))
<Regenaxer> yes, thought so
<Regenaxer> or ... @Rng '(NIL . T) .. (select (@@) ((name +Town @Rng zip +Town @Rng ... ?
<beneroth> nope
<Regenaxer> hmm, should work ...
<beneroth> I might do something wrong, but I don't think so
<Regenaxer> (trace 'init)
<Regenaxer> (trace 'step)
<Regenaxer> I do this always to see what is searched
<beneroth> : (? @N '(NIL . T) (select (@@) ((name +Town @N zip +Town @N))))
<beneroth> init : (name . +Town) (NIL) (T . T)
<beneroth> -> NIL
<beneroth> init = ((((NIL) T . T) ((("Aarau" . {2440}) NIL) (("Aarau" . {5612}) NIL) (("Aarau" . {107762}) NIL) (("Adliswil" . {32755}) NIL) (("Affoltern" . {33736}) NIL) (("Allendorf (Eder)" . {136660}) NIL) (("Altstätt$
<Regenaxer> good, the tree is inited
<Regenaxer> and 'step'?
<beneroth> : (? @N '(NIL . T) (select (@@) ((name +Town @N zip +Town @N))))
<beneroth> init : (name . +Town) (NIL) (T . T)
<beneroth> init = ((((NIL) T . T) ((("Aarau" . {2440}) NIL) (("Aarau" . {5612}) NIL) (("Aarau" . {107762}) NIL) (("Adliswil" . {32755}) NIL) (("Affoltern" . {33736}) NIL) (("Allendorf (Eder)" . {136660}) NIL) (("Altstätten" . {10753}) NIL) (("Anwil" . {150531}) NIL) (("Arlington, VA 22201" . {74756}) NIL) (("Bad Homburg vor der Höhe" . {137312}) NIL)) ((("Bad Zurzach" . {135357}) {135360}) (("Basel" . {2265}) {130062}) (("Bern" . {130060}) {130061}
<beneroth> ) (("Bielefeld" . {41657}) {135645}) (("Brügg" . {10610}) {135644}) (("Cologny" . {6164}) {131250})) ((("Edinburgh EH3 8HE, Scotland" . {130207}) {135362}) (("Muralto" . {13314}) {225514}) (("Tann" . {14140}) {225513}))))
<beneroth> step : ((((NIL) T . T) ((("Aarau" . {2440}) NIL) (("Aarau" . {5612}) NIL) (("Aarau" . {107762}) NIL) (("Adliswil" . {32755}) NIL) (("Affoltern" . {33736}) NIL) (("Allendorf (Eder)" . {136660}) NIL) (("Altstätten" . {10753}) NIL) (("Anwil" . {150531}) NIL) (("Arlington, VA 22201" . {74756}) NIL) (("Bad Homburg vor der Höhe" . {137312}) NIL)) ((("Bad Zurzach" . {135357}) {135360}) (("Basel" . {2265}) {130062}) (("Bern" . {130060}) {130061}
<beneroth> ) (("Bielefeld" . {41657}) {135645}) (("Brügg" . {10610}) {135644}) (("Cologny" . {6164}) {131250})) ((("Edinburgh EH3 8HE, Scotland" . {130207}) {135362}) (("Muralto" . {13314}) {225514}) (("Tann" . {14140}) {225513})))) T
<beneroth> step = {2440}
<beneroth> step : ((((NIL) T . T) ((("Aarau" . {5612}) NIL) (("Aarau" . {107762}) NIL)
<beneroth> uh, sorry for flood
<Regenaxer> np
<Regenaxer> So it correctly returns objects
<Regenaxer> step = {2440}
<beneroth> T
<beneroth> why don't they get returned?
<Regenaxer> No further filter, right?
<beneroth> T
<Regenaxer> I tried here: (? (select (@@) ((nm +User (NIL . T)) (nam +User (NIL . T)))))
<Regenaxer> works
<beneroth> in SQL there is UNION to merge multiple result sets into a single one. I cannot see yet how to do this within pilog
<beneroth> uh OK
<Regenaxer> it is a merge
<beneroth> ah w
<beneroth> yes that works for me too
<beneroth> but that is an AND, not an OR, no?
<Regenaxer> hmm, yes, here too if I take 'em' instead of 'nm'
<beneroth> I like to get all objects which are at least in one of the 3 +Ref indexes
<beneroth> as I cannot guarantee that all objects are in all 3 indexes
<Regenaxer> But I used that in production
<Regenaxer> also with sparse values
<Regenaxer> hmm
<beneroth> aj
<beneroth> aaaah
<beneroth> right
<Regenaxer> ?
<beneroth> it is an AND, but every generation clause should be satisfied?
<beneroth> no
<beneroth> no
<beneroth> (? (select (@@) ((nm +User (NIL . T)) (nam +User (NIL . T)))))
<Regenaxer> I thought it is OR
<Regenaxer> no
<beneroth> excludes objects with have only nam or nm set
<beneroth> no?
<Regenaxer> this is parallel
<Regenaxer> stops when *one* index is exhausted
<Regenaxer> ((nm +User @S nam +User @S)) searche first 'nm', then 'nam'
<beneroth> aye, this I want, without @S :)
<Regenaxer> yep
<Regenaxer> I used that always with @S and filters
<beneroth> yeah
<beneroth> doesn't make sense otherwise, well, except for (dump) :)
<beneroth> we go for lunch now. I will think about the best way to solve this.
<beneroth> thank you, Regenaxer :)
<beneroth> bbl
<Regenaxer> I investigate here
<Regenaxer> should work without filter
<beneroth> thank you
<Regenaxer> My test was wrong
<Regenaxer> one paren too much
<Regenaxer> Now works
<Regenaxer> I looked up old sources ;)
<Regenaxer> Must be (select (@@) (((em +User @S nm +User @S))))
<Regenaxer> eg. I have:
<Regenaxer> ...
<Regenaxer> (@LK +Land @Lnd (lnd +Ort) (ort @Cls))
<Regenaxer> (cont +Ort @Cont (ort @Cls))
<Regenaxer> ((exp +Ort @Rel imp +Ort @Rel) (ort @Cls))
<Regenaxer> (bra @Cls @Bra)
<Regenaxer> ...
<Regenaxer> here it is the line ((exp +Ort @Rel imp +Ort @Rel) (ort @Cls))
<Regenaxer> So it needs one *more* paren! ;)
<Regenaxer> Envoy your lunch first!
aw- has joined #picolisp
aw- has quit [Client Quit]
aw- has joined #picolisp
<Regenaxer> aw- wb!
<aw-> hi, thanks
aw- has quit [Quit: Leaving.]
<beneroth> back
<beneroth> Regenaxer, you're still having a parameter in the generator clause
<beneroth> I see no difference with additional parens
<beneroth> Regenaxer, ugly solution for my dump: I could do 3 dumps: (dump (db name +Town)) (dump (db zip +Town)) (dump (db country +Town)) -> ugly, as multiple time the same (object ...) gets generated, but would work.
<beneroth> better: not using pilog, (let All NIL (iter (tree 'name '+Town) '((E) (push1 'All E))) (iter (tree 'zip '+Town) '((E) (push1 'All E))) ...)
<beneroth> (for E All (dumpExt E))
orivej has quit [Ping timeout: 268 seconds]
<Regenaxer> Sorry, was away
xkapastel has joined #picolisp
<Regenaxer> beneroth, no, it must work now
<Regenaxer> It needs an extra paren, because it groups several indexes
<beneroth> you mean (select (@@) (((em +User @S nm +User @S)))) with @S being?
<Regenaxer> The parameter is needed, but (NIL . T) is OK
<beneroth> T, works!
<Regenaxer> You could do (? @S '(NIL . T) .., but diect is better
* beneroth rereading select doc about that extra parent
<beneroth> Regenaxer, aye, that I did now
<Regenaxer> (select (@@) (((em +User (NIL . T) nm +User (NIL . T)))))
<beneroth> aye, works
<beneroth> better because it is one less indirection?
<Regenaxer> Now it is all clear to me again :)
<Regenaxer> yes, no binding
<Regenaxer> but should not be noticeable
<beneroth> T
<beneroth> grouping of multiple index requires again () around
<beneroth> naturally
<Regenaxer> yeah
<Regenaxer> and to separate it from further trails
<Regenaxer> ((exp +Ort @Rel imp +Ort @Rel) (ort @Cls))
<Regenaxer> as opposed to
<Regenaxer> (exp +Ort @Rel (ort @Cls))
<Regenaxer> for a single index
<beneroth> aye, thanks!
<beneroth> that was a a bit stupid from my side, should have read more properly
<beneroth> ah, btw. Regenaxer do you get my special characters? Über Hügel nächste fördern.
rob_w has quit [Quit: Leaving]
<Regenaxer> Ahh, cool! I see Umlauts now :)
<Regenaxer> New client?
<beneroth> yes. HexChat. successor to what I used before, but less bugs apparently :)
<beneroth> looks the same
<Regenaxer> Perfect
<beneroth> thanks for you time, once more. I believe I finally got to a level of (user) understanding of pilog I don't forgot 2 seconds after a successful query :)
<Regenaxer> :)
<beneroth> the syntax is a bit tricky, parens do matter (unlike as in lisp, where they seem natural to me, no effort), but the resulting database code is beautifully short and clear :)
<Regenaxer> As you see, I forget it too rather often
<beneroth> tricky stuff. well, it db querying is complex material :)
<Regenaxer> yeah
<beneroth> not the single units of complexity, but the sheer number and the many ways stuff interacts
<Regenaxer> ... and is connectet to each other
<beneroth> aye
<beneroth> pilog code is really beautiful once one get the query together :)
<beneroth> s/get/got
<Regenaxer> at least it is compact. Whether Pi(Pro)log is beautiful is also a matter of (intellectual) taste
<Regenaxer> I still have problems with it
<Regenaxer> get my brain around it
razzy has joined #picolisp
razzy has quit [Changing host]
razzy has joined #picolisp
orivej has joined #picolisp
<beneroth> Regenaxer, last time I wrote SQL I wrote over 10k LOC for a complex data sync logic, mostly boiler plate code because a proper (request) is pretty hard in SQL..
<Regenaxer> wow
<Regenaxer> Last time I wrote SQL was around 1996
<Regenaxer> 23 years ago :)
<beneroth> nice uptime :D
<Regenaxer> yeah
<beneroth> how did you manage to implement pilog, especially the pilDB parts of it? just learning and grokking it for the moment? or just following some textbook?
<beneroth> I'm not yet daring to wrap my head around this pilog backtracking tree (the side-effect of prove)
<Regenaxer> For Pilog I had a book, "Implementations of Prolog" or so
<Regenaxer> PilDB came from alone ;)
<beneroth> awesome
<beneroth> whats the upper number of objects your current pilDB projects contain/manage (a single instance, not a distributed setup overall) ?
<beneroth> battle tested load numbers, so to say
<Regenaxer> Not as big as I had for SmApper 10 years ago
<Regenaxer> a few million
<beneroth> a few million at the moment? and SmApper ?
<Regenaxer> But for testing I imported some OSM
<Regenaxer> SmApper was around 800 million iirc
<beneroth> right, I think I remember
<beneroth> ah nice
<Regenaxer> Let me check OSM
<beneroth> no need
<Regenaxer> ok, seems I erased it again ;)
<beneroth> no problem
<beneroth> I'm still in the kilo range. will move to 20k+ the next days. and might get a project going into 100k+ to million range into the pipeline.
<Regenaxer> great
<beneroth> so far performance was always better than with "comparable" state of the art microsoft stack
<beneroth> while my applications are not really well tuned yet.
<Regenaxer> ok
<beneroth> even including some blunders (e.g. packing strings instead of working in stream etc)
<Regenaxer> The biggest bottleneck is usually import
<Regenaxer> That's why I made 'create'
<beneroth> T
<beneroth> oh right, I should look at it!
<beneroth> ha
<Regenaxer> I should also look
<Regenaxer> forgot all
<beneroth> I haven't used it yet
<beneroth> but I'm about to parse a 20k+ line CSV, sounds like a job for using create
<Regenaxer> I think my biggest DB currently is
<Regenaxer> output of dbck
<beneroth> I see
<Regenaxer> So the first numbers should be added
<Regenaxer> 1537513 symbols
<beneroth> impressive
<Regenaxer> It is an ERP system
<Regenaxer> BTG should be also about that size
<beneroth> well the majority of symbols is probably indexes, not records, right?
<beneroth> hm
<Regenaxer> The big one in the beginning is just items
<Regenaxer> 923042
<beneroth> so majority is not index
<beneroth> I just checked 'create. good examples.
<Regenaxer> And (old) BTG is this:
<beneroth> will try out. my parsing is a bit more involved, as I cannot fully trust the input. I have to validate during import.
<beneroth> Regenaxer, many more files, including some weakly used ones :D
<beneroth> for the moment I go with a standard set of predefined files with different block sizes
<Regenaxer> So BTG is smaller: 785547
<Regenaxer> Yeah, I used too many files back then
<Regenaxer> 20 years ago
<Regenaxer> A bit paranoid
<Regenaxer> But hardware was slower then
<Regenaxer> For recent projects I always used 12 - 16 files only
<Regenaxer> also becaus I need to save file counts for the many DBs on mobiles
<beneroth> yeah I see
<beneroth> I use more then I need at the moment.
<beneroth> Regenaxer, there is yet no best way to use (create) with output from (dump) (object ...) right? maybe for the future a variant of (dump) writting plio would be an idea?
<Regenaxer> Yeah, such things would be nice and useful
<beneroth> we'll get there. no need to hurry :)
<Regenaxer> T
<Regenaxer> In osm/create.l I did in fact use plio
<Regenaxer> i.e. piped curl to a plio file
<beneroth> nice
<Regenaxer> input OSM is xml
<Regenaxer> So I avoided to create the huge XML file
<beneroth> I apparently have an old version of your OSM code laying around.. without create.l
<Regenaxer> Maybe I did not publish this
<Regenaxer> only the very first old osm
<Regenaxer> OSM is good to experiment with large data
<beneroth> T
<Regenaxer> 'create' should work fine with 'dump'ed data, as it generates its input with a 'prg'
<beneroth> ah right
<beneroth> I see. I will try it out :)
ubLIX has joined #picolisp
ubLIX has quit [Quit: ubLIX]
xkapastel has quit [Quit: Connection closed for inactivity]
xkapastel has joined #picolisp