ChanServ changed the topic of #picolisp to: PicoLisp language | Channel Log: https://irclog.whitequark.org/picolisp/ | Check also http://www.picolisp.com for more information
_whitelogger has joined #picolisp
_whitelogger has joined #picolisp
aw- has quit [Quit: Leaving.]
_whitelogger has joined #picolisp
_whitelogger has joined #picolisp
_whitelogger has joined #picolisp
aw- has joined #picolisp
Blukunfando has quit [Ping timeout: 240 seconds]
dexen has quit [Ping timeout: 264 seconds]
<Regenaxer> aw-: How about storing the length values not in binary but as decimal ASCII?
<Regenaxer> I still think mixing binary data is not a good idea
<Regenaxer> Just think of operating on the files with standard tools, grepping or editing
<Regenaxer> All give trouble for mixed data
<aw-> hi Regenaxer
<aw-> the packet format is binary
<aw-> but some parts of the packet contains utf-8 strings
<Regenaxer> ok, just my opinion
<aw-> it's not my design or my decision
<Regenaxer> So better write all in C
<Regenaxer> As ht:Read
<aw-> why? so far it works perfectly
<Regenaxer> such things are not good to write in Lisp
<aw-> ...
<Regenaxer> I find it easier in C
<aw-> that's not something I like to hear
<Regenaxer> and surely more efficient
<Regenaxer> ok :)
<aw-> you're basically telling me i'm wasting my time with picolisp
<Regenaxer> Not in general, just for low-level stuff
<aw-> hmm...
<aw-> wouldn't we be better of fixing picolisp to work well with low-level stuff?
<Regenaxer> I also dont think I waste my time with Lisp
<Regenaxer> but for some things other langs are better
<Regenaxer> like Asm or C
<aw-> yes i agree
<aw-> i'm aware of the advantages of C
<Regenaxer> You *do* fix Lisp if you write a C function
<Regenaxer> just like the lib/ht stuff
<aw-> but parsing network packets seems like something which should work perfectly in PicoLisp without needing C or external libs
<Regenaxer> I feel it best to separate such things
<aw-> every other high level language supports that kind of work
<Regenaxer> Yes, as you say you can do it
<Regenaxer> and you even did
<aw-> indeed
<Regenaxer> I just feel it is not optimal
<Regenaxer> feels wrong
<Regenaxer> Like writing a device driver in COBOL ;)
<aw-> i guess it's similar to tankf33der's crypto implementations in picolisp? slow, not optimal
<Regenaxer> yeah, though pil has good bignum support, so it is close
<aw-> so you don't think we should write device drivers in PicoLisp either?
<Regenaxer> T
<aw-> :( disappointed
<aw-> what can I do? i dont want to write C
<Regenaxer> No, it is just *my* feeling
<Regenaxer> You need *some* native calls
<aw-> can i just write a C glue function that picolisp can talk to?
<aw-> is that your suggestion?
<Regenaxer> I said I would modify ht:Read
<Regenaxer> not taking a lenght arg, but reading it from the data
<aw-> oh
<aw-> i can't change the packet structure - not my design
<Regenaxer> Not the data
<Regenaxer> You need a behavior similar to ht:Read
<Regenaxer> the problem is the length
<Regenaxer> So modify ht:Read
<Regenaxer> ht:Read takes care of the utf8
<aw-> in src/ht.c ?
<aw-> or src64/ht.l?
<Regenaxer> yes, or pil21/src/ht.l
<Regenaxer> write a new lib for pil21
<Regenaxer> my:Read
<Regenaxer> or whatever name
<aw-> ok.. but i dont know that syntax for pil21 or even pilAsm
<aw-> it'll take me 1 year to finish that
<Regenaxer> ok, then a C lib callable with 'native'
<Regenaxer> How was the state yesterday?
<aw-> ?
<Regenaxer> Writing all with (native ... "read" works?
<Regenaxer> Perhaps this is the easiest then
<Regenaxer> just more syscall overhead
<aw-> with the buffer? yeah i gave up and just used the code you gave me a few weeks ago
<aw-> tweaked it a bit
<aw-> one sec i'll post my changes, it's a bit different
<Regenaxer> this can be improved
<Regenaxer> read a large buffer
<Regenaxer> 8 k
<Regenaxer> and parse with 'struct'
<Regenaxer> this is perhaps even easier than calling many read's
<aw-> hmmm
<aw-> yeah that might work, because i do have a maximum size for each string
<aw-> i know it should "never be more than N"
<aw-> so i can make an N buffer
<aw-> ok ok.. let me try your native/read/struct/buffer approach now
<Regenaxer> good
<aw-> wait
<Regenaxer> If you parse packets, it might make sense wo read the whole packet into a buffer and parse it with 'struct'
<aw-> no i cant, for security/DoS reasons
<aw-> i need to parse it in small increments and validate at each step
<Regenaxer> I see, makes sense
_whitelogger has joined #picolisp
<aw-> ok i got it, not sure if it's good
<aw-> one sec
<aw-> ok i'm definitely doing somethiung wrong
<aw-> i get back a list of bytes
<aw-> one sec
<aw-> Regenaxer: should i allocate the memory with alloc first? before passing the buffer to (native) ?
<aw-> or can I pass it inline like (native "@" "read" 'N Fd '(Buf ...)) ?
<aw-> ok i got it
<aw-> Regenaxer: it works, but it doesn't handle multi-byte UTF-8 characters
<aw-> i guess i shouldn't use (struct Mem 'S) ? or.. it works but i see extra garbage at the end of my string unless i add a null-byte
<Regenaxer> Sorry, I'm on the road atm
<Regenaxer> I thought (struct Mem 'S) is good
<aw-> (struct Mem 'S) expects my string to be null-terminated?
<Regenaxer> sure
<aw-> oh ok
<Regenaxer> : (setq P (%@ "malloc" 'P 99))
<Regenaxer> -> 519303016448
<Regenaxer> -> NIL
<Regenaxer> : (struct P NIL 65 66 0)
<Regenaxer> $: (struct P 'S)
<Regenaxer> -> "AB"
<aw-> oh ok
<aw-> ok ok i must have made a mistake earlier
<aw-> oops, Cnt is leaking in that function.. well anyways just to test
<aw-> hmmm.. this code doesn't work as expected
<aw-> i change the inputs and it appends garbage at the end of the string
<aw-> i think i need to add a null byte there
<Regenaxer> Still out in the rain. Can look only sporadically
<Regenaxer> I thought no separate read for Cnt
<Regenaxer> all in one buffer
<Regenaxer> get the count with (struct P '(B B))
<Regenaxer> then (struct (+ P 2) 'S)
<Regenaxer> afp
<aw-> oh interesting idea, ok i'll try that
<aw-> thanks
<aw-> yeah that doesn't work (pil64)
<aw-> so i'll just do 2 reads
<Regenaxer> It works in pil64
<Regenaxer> 'P' is not supported, so (setq P (native "@" "malloc" 'N 99))
<aw-> Regenaxer: in '(Cnt (1 . B) . 0) and '(Cnt (4 . I) . 0) what is the difference?
<aw-> why does '(Cnt (1 . I) . 0) not work?
<Regenaxer> Size 1?
<Regenaxer> btw. the ". 0" just wastes time
<Regenaxer> why fill the buffer with zero if you read it immediately?
dexen has joined #picolisp
orivej has joined #picolisp
<aw-> good question
<aw-> yes size 1 byte
<Regenaxer> But you specify 'I'
<Regenaxer> so the buffer needs 4 bytes
<aw-> ok that's what i thought
<aw-> but it doesn't work if I do '(Cnt (4 . I))
<aw-> '(Cnt (1 . B)) works fine
<aw-> and '(Cnt (4 . I) . 0) also works fine
<aw-> ok yeah i want an Int anyways
<aw-> good
<Regenaxer> The init makes no sense
<Regenaxer> Seems you don't read 4 bytes
<Regenaxer> what if you do '(Cnt (4 . I) . 127) ?
<Regenaxer> Again bad result?
<Regenaxer> if so, you don't read the right 4 bytes
<aw-> that works
<aw-> (native "@" "read" 'N Fd '(Cnt (4 . I) . 127) 1)
<Regenaxer> as I say!
<Regenaxer> makes sense
<aw-> you said makes no sense
<Regenaxer> What *I* said makes sense: "if so, you don't read the right 4 bytes" ;)
<Regenaxer> Here we see how bad an unnecessary inititialization is!
<Regenaxer> It covers the real error
<Regenaxer> have you tried '(Cnt (4 . I) . 127) ?
<Regenaxer> Ah, ok, yes
<Regenaxer> anyway, don't you see the error?
<Regenaxer> read ... 1)
<Regenaxer> you read just 1 byte
<aw-> yes im reading 1 byte
<aw-> from a 4 byte buffer
<aw-> hmmmm
<aw-> so an Int is 32-bits?
<Regenaxer> but then you cant read the strigg
<Regenaxer> string
<Regenaxer> I thought the length is 2 bytes
<Regenaxer> you must read the exact length
<aw-> sorry, i changed it since yesterday
<Regenaxer> otherwise you cannot read the data next time
<aw-> the length is 1 byte
<aw-> always
<aw-> i thought it was 1 or 2 but it's always 1
<Regenaxer> then '(Cnt (1 . B))
<Regenaxer> if you extract 4 bytes with 'I' but read only 1 byte you get garbage
<aw-> yes that's what i was seeing
<aw-> thats why i asked if an Int is 32-bits
<aw-> i just saw in (doc 'native)
<aw-> indeed it is
* aw- facepalms
<Regenaxer> yes, but your count is 1 byte
<aw-> right, my count is 1 byte, not 4 bytes, so (1 . B) is what i need
<Regenaxer> Yes
<aw-> ok sorry i have one last question for you about this
<Regenaxer> but if it is only one byte, you can save one read
<Regenaxer> buffer size 257 bytes
<aw-> hmmm
<Regenaxer> but only if there is just *1* read
<Regenaxer> if you want to continue with more data, you need 2 reads
<aw-> yes I do want to continue with more data afterwards
<Regenaxer> ok, then two syscalls are needed
<Regenaxer> first count, then the exact size
<aw-> ok no problem
<aw-> ok so my question is: since the string I'm reading is NOT null-terminated, do I still need a 256 byte buffer?
<Regenaxer> all right
<aw-> max string length will be 255
<Regenaxer> hmm, you must store the 0 yourself
<aw-> ok yes that's what I thought
<aw-> so how?
<Regenaxer> (byte (+ P length) 0)
<Regenaxer> (byte (+ P Length) 0)
<Regenaxer> and yes, one more byte allocated
<Regenaxer> -> 256
<aw-> ahhhhh (byte)
<aw-> i tried earlier with (byte) and it kept segfaulting
<Regenaxer> it is the fastest
<Regenaxer> 'struct' also works
<Regenaxer> but 'struct' has more overhead
<aw-> i think i was doing (byte (+ P 1)) or something
<Regenaxer> this reads a byte from memory, no storing
<aw-> does (byte) us strdup?
<Regenaxer> so the string parsing crashed
<Regenaxer> Why strdup?
<aw-> it doesn't copy the string to add the null byte?
<Regenaxer> strdup malloc()s a buffer, and copies all bytes up to NULL
<Regenaxer> byte gives a number 0 ... 255
<aw-> strndup()
<Regenaxer> Why would you do that?
<aw-> yeah.. im not really sure how (byte) works
<Regenaxer> no sense to *copy* the string
<Regenaxer> sigh
<Regenaxer> it reads or stores a byte
<Regenaxer> you want to store a NULL *byte*, no?
<aw-> yes
<Regenaxer> to have the string terminated
<Regenaxer> then use (struct P 'S)
<Regenaxer> to get utf8
<Regenaxer> no need to copy the string and later free() it
<aw-> i see
<aw-> ok so far so good
<aw-> i'm gonna go read a C book
<Regenaxer> Not needed I think. man pages are good
<Regenaxer> hmm, ok, perhaps you are right
<Regenaxer> memory handling of C
<Regenaxer> is not in man pages ;)
<aw-> haha
<Regenaxer> We still have a bug in pil21 family RPC
<Regenaxer> 'sync' does not work sometimes
<Regenaxer> hard to debug
<Regenaxer> :(
<aw-> is that used only for DB?
<Regenaxer> not only
<Regenaxer> I noticed when testing app/ though
<Regenaxer> but all (tell ..) or (boss ..) etc depend on it
<Regenaxer> in the app/ case it was also not DB
<Regenaxer> but syncing processes after locking an object in the GUI
<Regenaxer> took me since yesterday to find
<Regenaxer> almost a day
<Regenaxer> the sync call is in a finally clause
<Regenaxer> so a 'throw' fails when unwinding the stack
<Regenaxer> Really tough to trace down where exactly it failed
<Regenaxer> cause the throw never arrived
<Regenaxer> I thought catch/throw is the culprit
<aw-> ohch
<aw-> ouch*
<Regenaxer> :)
<aw-> i have some code doing (throw) as well but didn't notice any issues
<Regenaxer> only in pil21
<aw-> yes in pil21
<Regenaxer> and catch and throw seem fine
<aw-> none of my tests failed.. either the bug is really obscure, or my tests are incomplete ;)
<Regenaxer> but finally clauses are executed in throw
<Regenaxer> ok, good to know
<Regenaxer> It is in line 253 in lib/form.l
<Regenaxer> (in old pil; the file changed in pil21)
<Regenaxer> there is (sync) in a 'finally' clause
<Regenaxer> there it hangs
<Regenaxer> don't know the reason yet
<Regenaxer> basically the RPC works, I did some other tests
<Regenaxer> ok, must go. Will debug more later
<aw-> ok see you
<Regenaxer> cu :)
orivej has quit [Ping timeout: 256 seconds]
<Regenaxer> ret
<tankf33der> i will add tests for OOP this weekend.
<Regenaxer> Great!
<Regenaxer> I don't find it
<Regenaxer> the bug in family IPC
anddam has quit [*.net *.split]
anddam has joined #picolisp
orivej has joined #picolisp
orivej has quit [Ping timeout: 272 seconds]
<Regenaxer> Ha! Found it
orivej has joined #picolisp
<tankf33der> oop passed tests, added for future run.
<Regenaxer> Cool
<Regenaxer> misc/stress.l gives an error
<Regenaxer> but *not* with (gc 99)
<Regenaxer> No idea what might be the reason
<tankf33der> so, need gc+ run :)
<Regenaxer> could be indeed gc, or something still with IPC
<Regenaxer> yes, gc+ might give an idea :)
<Regenaxer> misc/stress.l does not much in heap, I rather suspect it is an sync/IPC issue
<tankf33der> misc/stress.l from picolisp dist ?
<Regenaxer> yes, exactly the same
<tankf33der> i will try asap
<Regenaxer> super
<Regenaxer> it is:
<Regenaxer> $ ./pil misc/stress.l -main -go -bye; rm db/test jnl db/test2
<Regenaxer> or, in my case, ../pil21/pil misc/stress.l ...
<tankf33der> $ ./pil21 misc/stress.l -main -go -bye; rm db/test jnl db/test2
<tankf33der> in my case.
<tankf33der> :)
<Regenaxer> yes, I know, you renamed pil
<Regenaxer> I keep the directories separate, too many differences
<tankf33der> stress.l gives error fast?
<Regenaxer> No, a little over 3 minutes
<tankf33der> ok
<Regenaxer> It runs through, but then the DB sanity checks fail
<Regenaxer> So it is probably not a "normal" gc problem
<Regenaxer> At the end something like
<Regenaxer> 1 2 (1168 . 1168)
<Regenaxer> {3} (973 . {640}) dat +A {1}
<Regenaxer> {553} -- Bad sequence
<Regenaxer> ?
<tankf33der> i see process have 130MB, so (gc 99) is not the case.
<Regenaxer> But with ... misc/stress.l -main -"gc 99" -go - ... it passes
<tankf33der> 1 2 (1175 . 1175)
<tankf33der> {5} -- Bad sequence
<tankf33der> ?
<tankf33der> {3} (1014 . {675}) dat +A {1}
<tankf33der> yes, crashed. ok
<Regenaxer> yeah
<Regenaxer> also -'gc 9' seems ok
<Regenaxer> Each child process inherits the heap size
<Regenaxer> hmm
<Regenaxer> now an error even with 99
<Regenaxer> db/test db/test2 differ: byte 7, line 1
<Regenaxer> So there is trouble ;)
<Regenaxer> It worked several times with 99
<tankf33der> {3} (999 . {673}) dat +A {1}
<tankf33der> db/test db/test2 differ: byte 7, line 1
<tankf33der> 1 2 (1168 . 1168)
<tankf33der> {3} (999 . {734}) key +A {1}
<tankf33der> with -"gc 256"
<Regenaxer> Same here
<Regenaxer> It is not a gc issue
<Regenaxer> another heisenbug
<Regenaxer> timings in process syncing
<tankf33der> running gc-plus
<Regenaxer> good
<tankf33der> $ pil21 db.l
<tankf33der> [db.l:14] !? (val *DB)
<tankf33der> Bad DB file
<tankf33der> ?
<tankf33der> under gc-plus
<Regenaxer> I see
<tankf33der> oop.l passed, btw.
<Regenaxer> great
<tankf33der> line 14 is ^^^
<Regenaxer> good, very basic tests
<Regenaxer> Without gc+ it passes, right?
<tankf33der> for a several weeks already
<tankf33der> and latest today tests too.
<Regenaxer> ok
<tankf33der> all basic tests passed too: flow, io, mapping, lists.
<tankf33der> all basic tests passed in gc+ too: flow, io, mapping, lists.
<Regenaxer> Why is line 14 = 22 ?
<Regenaxer> ah
<Regenaxer> no
<Regenaxer> this one (test *DB (val *DB) (lieu *DB)) ?
<tankf33der> aaaaaaa.
<tankf33der> stop
<tankf33der> wrong. forget it, checking again.
<Regenaxer> ok
<tankf33der> db.l passed.
<tankf33der> i have my own stress variation, doing.
<tankf33der> passed.
<tankf33der> now misc/stress.l
<Regenaxer> :)
<Regenaxer> The above db.l does not pass under gc+ ?
<tankf33der> tankf33der> db.l passed.
<Regenaxer> also with gc+ ?
<tankf33der> of course.
<tankf33der> all above under gc+
<Regenaxer> ok
<tankf33der> now misc/stress.l under gc+
<Regenaxer> app/ demo crashes in certain cases
<Regenaxer> I will debug tomorrow, cause I can reproduce
<Regenaxer> misc/stress.l under gc+ will not run so very much longer I think
<Regenaxer> cause the system calls and syncing are the sage
<Regenaxer> ... are the same
<tankf33der> maybe somehow make it smaller.
<tankf33der> required minimum
<tankf33der> only
<tankf33der> in will sleep in 20mins
<Regenaxer> Lets stop for today
<tankf33der> cu
<Regenaxer> Good night!