ChanServ changed the topic of #picolisp to: PicoLisp language | Channel Log: https://irclog.whitequark.org/picolisp/ | Check also http://www.picolisp.com for more information
ubLIX has quit [Quit: ubLIX]
alexshendi has quit [Ping timeout: 250 seconds]
rick42 has joined #picolisp
jibanes has quit [Ping timeout: 246 seconds]
jibanes has joined #picolisp
orivej has quit [Ping timeout: 240 seconds]
_whitelogger has joined #picolisp
_whitelogger_ has joined #picolisp
_whitelogger has quit [Ping timeout: 250 seconds]
CSchmidt has joined #picolisp
CSchmidt has quit []
_whitelogger has joined #picolisp
<Regenaxer> razzy: Good plan!
<Regenaxer> But it does not make much sense to stick close to that book
<Regenaxer> PicoLisp is too different I think
<Regenaxer> Better represent the "world" in a DB
<razzy> i like that aproach in "book" more than picolisp ;]
<razzy> than picolisp lost some of lisp power to me
<tankf33der> lol
<Regenaxer> Hi tankf33der
<Regenaxer> razzy, which power is lost?
<tankf33der> morning all
<Regenaxer> I think the opposite is true
<razzy> i cannot prove it yet
<razzy> sorry
<Regenaxer> Do you even understand it?
<Regenaxer> what you are talking about?
<razzy> more than half of it :]
<Regenaxer> tankf33der: 'create' more than doubled its speed yesterday
<Regenaxer> Hundred million objects with 3 indexes import in 53 minutes now
<razzy> ` evaluate in runtime? or pre-runtime, macro-time?
<tankf33der> Regenaxer: good i found fastest laptop i can and defenetly must try reimplement ny taxi from scratch
<Regenaxer> Was 4 hours first, then 2, now less than one
<tankf33der> Regenaxer: sounds good
<Regenaxer> Why do you not try the version I sent?
<tankf33der> i save it and will use
<tankf33der> problem i dont remember details of import
<Regenaxer> I have not tested it though
<Regenaxer> the csv?
<tankf33der> yea
<tankf33der> i completely dont remember details of the task
<Regenaxer> Best to test with a small one first
<Regenaxer> you don't have the old CSVs any more?
<tankf33der> all i have this
<Regenaxer> I think thats what I used
<Regenaxer> dec2018.l is the current
<Regenaxer> So "yellow_tripdata_2017-01.csv" is needed ;)
<tankf33der> i can send you a link
<tankf33der> will you try then?
<Regenaxer> yes, good
<Regenaxer> But what is the problem? Just run it on the first few lines first
<Regenaxer> But I will try
<Regenaxer> To debug it
<Regenaxer> razzy: '`' is a *read* macro
<tankf33der> searching you a csv file
<tankf33der> several GB
<razzy> so it is evaluated in macro time, with variables at the time.
<razzy> yes?
<Regenaxer> tankf33der, for a first try the first 10 lines are enough
<Regenaxer> razzy: No
<tankf33der> tankf33der: i will try in working hours.
<Regenaxer> razzy, ok
<razzy> Regenaxer: ?
<Regenaxer> There is no "macro time", but it is the dynamic state at read time
<Regenaxer> tankf33der, just 'head 100 x.csv >y.csv'
<razzy> i think common lisp has macro time before compilation/interpretation
<Regenaxer> I don't want to download several GiB now ;)
<razzy> picolisp ` uses variables that are accesible in that place in program?
<Regenaxer> razzy, what does *that* mean?
<razzy> current place in program?
<Regenaxer> Pil *always* uses the current dynamic bindings
<Regenaxer> Try to understand that
<razzy> hmm,.. i will ask another way
<Regenaxer> Not "current place"
<Regenaxer> that's a static view
<Regenaxer> ie lexical binding
<Regenaxer> pil uses dynamic binding
<razzy> picolisp ` does not evaluate when loading code from file. yes?
<Regenaxer> of course it does
<Regenaxer> thats the main point
<Regenaxer> also, there is no difference
<razzy> sorry my bad
<Regenaxer> it is *always* loading from a file kind of
<Regenaxer> What excactly do you want to achieve?
<razzy> i want to learn if there is some limits to ` use
<Regenaxer> No limits
<razzy> how different variables behave in there
<Regenaxer> There are no "different" variables
<Regenaxer> all are dynamically bound
<Regenaxer> No "special" variables as in CL
<razzy> macro time in common lisp is evaluated before run of program
<Regenaxer> What excactly do you want to achieve?
<Regenaxer> What excactly do you want to achieve?
<Regenaxer> What excactly do you want to achieve?
<razzy> now i tried to explain why i asked
<razzy> i am good now propably
<Regenaxer> tankf33der, I download it. It is only 815M
<tankf33der> ok
<Regenaxer> takes only 2 min
<tankf33der> to import or to download?
<Regenaxer> download
<Regenaxer> Interrupt
_whitelogger has joined #picolisp
<Regenaxer> I want to sort the file on the first date
orivej has joined #picolisp
<tankf33der> in memory or in file?
<Regenaxer> Standard Unix tools
<Regenaxer> I prepare the stream a little to make the import faster
<Regenaxer> experimenting with 'sort', tough ;)
<Regenaxer> I cut off the first field also
<Regenaxer> let me try
<Regenaxer> Works
<Regenaxer> Import running
<Regenaxer> first pass done
<Regenaxer> finishes soon
<Regenaxer> Done
<Regenaxer> about 9 minutes
<Regenaxer> Nearly 10 million +Cab objects
<Regenaxer> Tested code: http://ix.io/1w6M
<tankf33der> i want try too
<Regenaxer> good
<Regenaxer> Should be no problem
<Regenaxer> First process (about 2/3 of the time) used about 500 MiB, then the other 4 processes about 360 to 450 MiB each. So I GiB might be a little small
<Regenaxer> Total time 8 min 12 sec here
<tankf33der> so when i will import 2017-2.csv it will work too
<Regenaxer> Probably
<tankf33der> and next monthes and years
<Regenaxer> I think one bug still
<Regenaxer> The first two lines are not considered in the sort
<Regenaxer> better cut them off first
<Regenaxer> I'll use tail -n +2
<Regenaxer> How about this: http://ix.io/1w6R ?
<tankf33der> i see
<Regenaxer> hmm, no, something went wrong ;)
<Regenaxer> needs +3
<Regenaxer> So final version is http://ix.io/1w6S
<Regenaxer> You just need to keep the file name variable
<Regenaxer> (in (list "sh" "-c" (pack "cut -c3- yellow_tripdata_2017-01.csv" " |tail -n +3 |sort"))
<Regenaxer> oops
<Regenaxer> (in (list "sh" "-c" (pack "cut -c3- " "yellow_tripdata_2017-01.csv" " |tail -n +3 |sort"))
<Regenaxer> You know what I mean
<Regenaxer> bbl
<tankf33der> also i need modify code to possible import other monthes
freemint has joined #picolisp
orivej has quit [Quit: No Ping reply in 180 seconds.]
orivej has joined #picolisp
orivej has quit [Read error: Connection timed out]
orivej has joined #picolisp
alexshendi has joined #picolisp
alexshendi has quit [Client Quit]
alexshendi has joined #picolisp
<Regenaxer> ret
<Regenaxer> They have another format?
orivej has quit [Quit: No Ping reply in 180 seconds.]
orivej has joined #picolisp
orivej has quit [Client Quit]
orivej has joined #picolisp
<tankf33der> no, the same
<Regenaxer> So no code modification needed?
<tankf33der> but inside db it will sorted, right ?
<tankf33der> no
<tankf33der> what if i import 2017-01, 2016-12, 2017-02
<Regenaxer> Not directly sorted, but inserted into b-trees
<tankf33der> it will ok inside db, right ?
<Regenaxer> (in (list "sh" "-c" (pack "cut -c3- " File " |tail -n +3 |sort"))
<Regenaxer> yes
<Regenaxer> should be ;)
<tankf33der> ok
<tankf33der> i import a month and try requests
<Regenaxer> Best if the whole is a funtion
<tankf33der> then year and all data
<Regenaxer> '((File) ...
<Regenaxer> yeah
<Regenaxer> It will get slower when it gets bigger
orivej has quit [Quit: No Ping reply in 180 seconds.]
orivej has joined #picolisp
_whitelogger has joined #picolisp
orivej has quit [Ping timeout: 245 seconds]
<tankf33der> start importing 2017-01
<Regenaxer> Great!
<Regenaxer> How much RAM?
<tankf33der> 8GB
<Regenaxer> Perfect
<Regenaxer> Same as here
<Regenaxer> It also helps if you do something like .pil/tmp -> /ssd/
<Regenaxer> (mounted SSD)
<Regenaxer> But only a few percent
<tankf33der> i will do from tmpfs
<Regenaxer> RAM disk?
<tankf33der> yea
<tankf33der> BTW, i have ssd too
<Regenaxer> Better use RAM for process usage
<tankf33der> ok
<Regenaxer> The temp files get very big
<Regenaxer> about the same size as the DB
orivej has joined #picolisp
<Regenaxer> It first creates lots of temp files in PLIO format
<Regenaxer> which are the imported in a sorted order
<Regenaxer> s/are the/are then/
<tankf33der> so
<tankf33der> import works
<tankf33der> (create) huge progress. :)
<Regenaxer> ok
<Regenaxer> How huge?
<tankf33der> import was a day :)
<tankf33der> now minutes.
<Regenaxer> Yeah
<Regenaxer> The first process will become small and fork 4 processes
<Regenaxer> 4 large ones
<tankf33der> seen
<tankf33der> i can sort and delete not required fields before import
<Regenaxer> yes, but I think this is not a bottleneck
<Regenaxer> traversed only once
<tankf33der> of course.
<tankf33der> so, will compare if csv file is the same in range years 2009-2018
<tankf33der> ===========
<tankf33der> push '*Bye '(call "rm" "-r" *Pool))
<tankf33der> (push '*Fork '(del '(call "rm" "-r" *Pool) '*Bye))
<tankf33der> i dont need this, right ?
orivej_ has joined #picolisp
<Regenaxer> yeah
orivej has quit [Ping timeout: 272 seconds]
<tankf33der> i want to import next month.
<Regenaxer> for testing only
<Regenaxer> Surely better not delete any more :)
freemint2 has joined #picolisp
freemint has quit [Ping timeout: 250 seconds]
<Regenaxer> bbl
Regenaxer has left #picolisp [#picolisp]
jibanes has quit [Ping timeout: 268 seconds]
jibanes has joined #picolisp
jibanes has quit [Ping timeout: 250 seconds]
jibanes has joined #picolisp
<razzy> big data are interesting :]
<tankf33der> razzy: bigdata is this
<tankf33der> this one new york taxi task is not big data, just training
<razzy> he he , do you have pornhub data?
Regenaxer has joined #picolisp
<razzy> Regenaxer: i groked fill, and i must say, that (fill) is more powerfull for heavy user. i was confused with @ magic
<Regenaxer> Great :)
<razzy> for ocasional user (me) is ussage without @ :]
<razzy> how do i get pointer to some cell? i think it is solved in simul.l but i am hoping for more easy way. cannot find function for that
<Regenaxer> There as a raw function 'adr', you can calculate the pointer from it
<razzy> hm
<Regenaxer> What do you want to use the pointer for?
<razzy> if i want to change value cell? i know it is not recommended
<razzy> value of cell
<Regenaxer> You can change it any time in Lisp
<Regenaxer> 'set' and 'con'
<razzy> in long cadddr expression?
<Regenaxer> yes, after obtaining the cell in some way
<razzy> (set '(caddar *var) 5) will work?
<Regenaxer> yes, but not as you expect
<Regenaxer> you replase the atom 'caddar'
<Regenaxer> replace
<razzy> how do i change value of (caddar *var). easily?
<Regenaxer> You really do not understand. Please think
<Regenaxer> You still don't seem to have an idea aoout even the most basic Lisp mechanisms
<Regenaxer> I have no time no
<Regenaxer> afp
<razzy> one answer would be do not build deep lists.
<razzy> i think reaching into long list and changing variable is lisp *core* mechanism :]
<razzy> now i am at (set (format (pack "$" `(adr (caaadr a)))) 5) but i cannot format "$num" it seems
mtsd has joined #picolisp
<razzy> maybe new version help with that?
<razzy> no newer version do not help :(
<Regenaxer> ret
<Regenaxer> razzy, try to *understand* first
<razzy> i think i do
<Regenaxer> You did not grok the concepts of PicoLisp at all
<Regenaxer> I give up trying to explain
<razzy> what is wrong with my aproach? should i pay you? i can a little :]
<razzy> if picolisp do not support deep lists. just say so
<Regenaxer> No, I'm not able to teach you
<Regenaxer> You are a hopeless case
<Regenaxer> Or are you a troll? Don't do such jokes
<Regenaxer> afp
<razzy> no troll :]
<razzy> just incosistent memories about lisp programming
<mtsd> razzy, sounds like a good opportunity for a fresh start then?
<razzy> picolisp seemed like fresh start :]
<razzy> little code so far :]
<mtsd> It has been for me anyway. It does take some time and effort to wrap your head around concepts unfamiliar to you, but it is well worth it
<mtsd> But, as with every new thing we learn, no one is able to do the work for you. It is like running, training for a marathon:
<mtsd> You have to do all the running, every single mile. No one else can do that instead of you. But you are also the one reaping all the benefits from doing so
<mtsd> I struggle with things from time to time too, but each time I overcome them and learn, I have become a little better than I was before.
<razzy> mtsd: i am overcoming something now
<razzy> i will report my strugle with code
<razzy> mtsd: my strugle is here https://ptpb.pw/nyMb the aproach is bad only by half
<razzy> mtsd: i want to change value of deep list. i think in other lisps it is possible
<razzy> also combination of (adr (caaadr a)) is often segfault :]. unless you have one layer of list beneth you :]. (adr (caaadr)) seem usefull to me
<razzy> Regenaxer: i really think my aproach is lispy. i am learning about picolisp
mtsd has quit [Quit: WeeChat 1.6]
<razzy> Regenaxer: you maybe hate this aproach. but i like it https://ptpb.pw/nyMb pls, tell me what is wrong with it.
<beneroth> razzy, there is no practical use to your approach.
orivej_ has quit [Ping timeout: 250 seconds]
cobax has quit [Ping timeout: 264 seconds]
orivej has joined #picolisp
<razzy> beneroth: if i spread data to several lists, i will have to track more lists and data ussage grows by 20%.
<razzy> or i learn picoDB, there could be answer :)
orivej has quit [Ping timeout: 250 seconds]
orivej has joined #picolisp
<razzy> if pilDB has some magical way to shorten data lists
<razzy> ah, nevermind. it works. i was one lvl of car too deep.and still having trouble accepting that if i use (set (caaar a) 6) , (caaar) return adress naturally to set. i hope it is picolisp shortcut. or there is somthing very bad with my understanding.