_whitelogger has joined #picolisp
_whitelogger has joined #picolisp
aw- has quit [Quit: Leaving.]
_whitelogger has joined #picolisp
_whitelogger has joined #picolisp
_whitelogger has joined #picolisp
aw- has joined #picolisp
Blukunfando has quit [Ping timeout: 240 seconds]
dexen has quit [Ping timeout: 264 seconds]
<
Regenaxer>
aw-: How about storing the length values not in binary but as decimal ASCII?
<
Regenaxer>
I still think mixing binary data is not a good idea
<
Regenaxer>
Just think of operating on the files with standard tools, grepping or editing
<
Regenaxer>
All give trouble for mixed data
<
aw->
the packet format is binary
<
aw->
but some parts of the packet contains utf-8 strings
<
Regenaxer>
ok, just my opinion
<
aw->
it's not my design or my decision
<
Regenaxer>
So better write all in C
<
Regenaxer>
As ht:Read
<
aw->
why? so far it works perfectly
<
Regenaxer>
such things are not good to write in Lisp
<
Regenaxer>
I find it easier in C
<
aw->
that's not something I like to hear
<
Regenaxer>
and surely more efficient
<
aw->
you're basically telling me i'm wasting my time with picolisp
<
Regenaxer>
Not in general, just for low-level stuff
<
aw->
wouldn't we be better of fixing picolisp to work well with low-level stuff?
<
Regenaxer>
I also dont think I waste my time with Lisp
<
Regenaxer>
but for some things other langs are better
<
Regenaxer>
like Asm or C
<
aw->
i'm aware of the advantages of C
<
Regenaxer>
You
*do* fix Lisp if you write a C function
<
Regenaxer>
just like the lib/ht stuff
<
aw->
but parsing network packets seems like something which should work perfectly in PicoLisp without needing C or external libs
<
Regenaxer>
I feel it best to separate such things
<
aw->
every other high level language supports that kind of work
<
Regenaxer>
Yes, as you say you can do it
<
Regenaxer>
and you even did
<
Regenaxer>
I just feel it is not optimal
<
Regenaxer>
feels wrong
<
Regenaxer>
Like writing a device driver in COBOL ;)
<
aw->
i guess it's similar to tankf33der's crypto implementations in picolisp? slow, not optimal
<
Regenaxer>
yeah, though pil has good bignum support, so it is close
<
aw->
so you don't think we should write device drivers in PicoLisp either?
<
aw->
:( disappointed
<
aw->
what can I do? i dont want to write C
<
Regenaxer>
No, it is just
*my* feeling
<
Regenaxer>
You need
*some* native calls
<
aw->
can i just write a C glue function that picolisp can talk to?
<
aw->
is that your suggestion?
<
Regenaxer>
I said I would modify ht:Read
<
Regenaxer>
not taking a lenght arg, but reading it from the data
<
aw->
i can't change the packet structure - not my design
<
Regenaxer>
Not the data
<
Regenaxer>
You need a behavior similar to ht:Read
<
Regenaxer>
the problem is the length
<
Regenaxer>
So modify ht:Read
<
Regenaxer>
ht:Read takes care of the utf8
<
aw->
in src/ht.c ?
<
aw->
or src64/ht.l?
<
Regenaxer>
yes, or pil21/src/ht.l
<
Regenaxer>
write a new lib for pil21
<
Regenaxer>
my:Read
<
Regenaxer>
or whatever name
<
aw->
ok.. but i dont know that syntax for pil21 or even pilAsm
<
aw->
it'll take me 1 year to finish that
<
Regenaxer>
ok, then a C lib callable with 'native'
<
Regenaxer>
How was the state yesterday?
<
Regenaxer>
Writing all with (native ... "read" works?
<
Regenaxer>
Perhaps this is the easiest then
<
Regenaxer>
just more syscall overhead
<
aw->
with the buffer? yeah i gave up and just used the code you gave me a few weeks ago
<
aw->
tweaked it a bit
<
aw->
one sec i'll post my changes, it's a bit different
<
Regenaxer>
this can be improved
<
Regenaxer>
read a large buffer
<
Regenaxer>
and parse with 'struct'
<
Regenaxer>
this is perhaps even easier than calling many read's
<
aw->
yeah that might work, because i do have a maximum size for each string
<
aw->
i know it should "never be more than N"
<
aw->
so i can make an N buffer
<
aw->
ok ok.. let me try your native/read/struct/buffer approach now
<
Regenaxer>
If you parse packets, it might make sense wo read the whole packet into a buffer and parse it with 'struct'
<
aw->
no i cant, for security/DoS reasons
<
aw->
i need to parse it in small increments and validate at each step
<
Regenaxer>
I see, makes sense
_whitelogger has joined #picolisp
<
aw->
ok i got it, not sure if it's good
<
aw->
ok i'm definitely doing somethiung wrong
<
aw->
i get back a list of bytes
<
aw->
Regenaxer: should i allocate the memory with alloc first? before passing the buffer to (native) ?
<
aw->
or can I pass it inline like (native "@" "read" 'N Fd '(Buf ...)) ?
<
aw->
Regenaxer: it works, but it doesn't handle multi-byte UTF-8 characters
<
aw->
i guess i shouldn't use (struct Mem 'S) ? or.. it works but i see extra garbage at the end of my string unless i add a null-byte
<
Regenaxer>
Sorry, I'm on the road atm
<
Regenaxer>
I thought (struct Mem 'S) is good
<
aw->
(struct Mem 'S) expects my string to be null-terminated?
<
Regenaxer>
: (setq P (%@ "malloc" 'P 99))
<
Regenaxer>
-> 519303016448
<
Regenaxer>
: (struct P NIL 65 66 0)
<
Regenaxer>
$: (struct P 'S)
<
Regenaxer>
-> "AB"
<
aw->
ok ok i must have made a mistake earlier
<
aw->
oops, Cnt is leaking in that function.. well anyways just to test
<
aw->
hmmm.. this code doesn't work as expected
<
aw->
i change the inputs and it appends garbage at the end of the string
<
aw->
i think i need to add a null byte there
<
Regenaxer>
Still out in the rain. Can look only sporadically
<
Regenaxer>
I thought no separate read for Cnt
<
Regenaxer>
all in one buffer
<
Regenaxer>
get the count with (struct P '(B B))
<
Regenaxer>
then (struct (+ P 2) 'S)
<
aw->
oh interesting idea, ok i'll try that
<
aw->
yeah that doesn't work (pil64)
<
aw->
so i'll just do 2 reads
<
Regenaxer>
It works in pil64
<
Regenaxer>
'P' is not supported, so (setq P (native "@" "malloc" 'N 99))
<
aw->
Regenaxer: in '(Cnt (1 . B) . 0) and '(Cnt (4 . I) . 0) what is the difference?
<
aw->
why does '(Cnt (1 . I) . 0) not work?
<
Regenaxer>
Size 1?
<
Regenaxer>
btw. the ". 0" just wastes time
<
Regenaxer>
why fill the buffer with zero if you read it immediately?
dexen has joined #picolisp
orivej has joined #picolisp
<
aw->
good question
<
aw->
yes size 1 byte
<
Regenaxer>
But you specify 'I'
<
Regenaxer>
so the buffer needs 4 bytes
<
aw->
ok that's what i thought
<
aw->
but it doesn't work if I do '(Cnt (4 . I))
<
aw->
'(Cnt (1 . B)) works fine
<
aw->
and '(Cnt (4 . I) . 0) also works fine
<
aw->
ok yeah i want an Int anyways
<
Regenaxer>
The init makes no sense
<
Regenaxer>
Seems you don't read 4 bytes
<
Regenaxer>
what if you do '(Cnt (4 . I) . 127) ?
<
Regenaxer>
Again bad result?
<
Regenaxer>
if so, you don't read the right 4 bytes
<
aw->
(native "@" "read" 'N Fd '(Cnt (4 . I) . 127) 1)
<
Regenaxer>
as I say!
<
Regenaxer>
makes sense
<
aw->
you said makes no sense
<
Regenaxer>
What
*I* said makes sense: "if so, you don't read the right 4 bytes" ;)
<
Regenaxer>
Here we see how bad an unnecessary inititialization is!
<
Regenaxer>
It covers the real error
<
Regenaxer>
have you tried '(Cnt (4 . I) . 127) ?
<
Regenaxer>
Ah, ok, yes
<
Regenaxer>
anyway, don't you see the error?
<
Regenaxer>
read ... 1)
<
Regenaxer>
you read just 1 byte
<
aw->
yes im reading 1 byte
<
aw->
from a 4 byte buffer
<
aw->
so an Int is 32-bits?
<
Regenaxer>
but then you cant read the strigg
<
Regenaxer>
I thought the length is 2 bytes
<
Regenaxer>
you must read the exact length
<
aw->
sorry, i changed it since yesterday
<
Regenaxer>
otherwise you cannot read the data next time
<
aw->
the length is 1 byte
<
aw->
i thought it was 1 or 2 but it's always 1
<
Regenaxer>
then '(Cnt (1 . B))
<
Regenaxer>
if you extract 4 bytes with 'I' but read only 1 byte you get garbage
<
aw->
yes that's what i was seeing
<
aw->
thats why i asked if an Int is 32-bits
<
aw->
i just saw in (doc 'native)
<
Regenaxer>
yes, but your count is 1 byte
<
aw->
right, my count is 1 byte, not 4 bytes, so (1 . B) is what i need
<
aw->
ok sorry i have one last question for you about this
<
Regenaxer>
but if it is only one byte, you can save one read
<
Regenaxer>
buffer size 257 bytes
<
Regenaxer>
but only if there is just
*1* read
<
Regenaxer>
if you want to continue with more data, you need 2 reads
<
aw->
yes I do want to continue with more data afterwards
<
Regenaxer>
ok, then two syscalls are needed
<
Regenaxer>
first count, then the exact size
<
aw->
ok no problem
<
aw->
ok so my question is: since the string I'm reading is NOT null-terminated, do I still need a 256 byte buffer?
<
Regenaxer>
all right
<
aw->
max string length will be 255
<
Regenaxer>
hmm, you must store the 0 yourself
<
aw->
ok yes that's what I thought
<
Regenaxer>
(byte (+ P length) 0)
<
Regenaxer>
(byte (+ P Length) 0)
<
Regenaxer>
and yes, one more byte allocated
<
aw->
ahhhhh (byte)
<
aw->
i tried earlier with (byte) and it kept segfaulting
<
Regenaxer>
it is the fastest
<
Regenaxer>
'struct' also works
<
Regenaxer>
but 'struct' has more overhead
<
aw->
i think i was doing (byte (+ P 1)) or something
<
Regenaxer>
this reads a byte from memory, no storing
<
aw->
does (byte) us strdup?
<
Regenaxer>
so the string parsing crashed
<
Regenaxer>
Why strdup?
<
aw->
it doesn't copy the string to add the null byte?
<
Regenaxer>
strdup malloc()s a buffer, and copies all bytes up to NULL
<
Regenaxer>
byte gives a number 0 ... 255
<
Regenaxer>
Why would you do that?
<
aw->
yeah.. im not really sure how (byte) works
<
Regenaxer>
no sense to
*copy* the string
<
Regenaxer>
it reads or stores a byte
<
Regenaxer>
you want to store a NULL *byte*, no?
<
Regenaxer>
to have the string terminated
<
Regenaxer>
then use (struct P 'S)
<
Regenaxer>
to get utf8
<
Regenaxer>
no need to copy the string and later free() it
<
aw->
ok so far so good
<
aw->
i'm gonna go read a C book
<
Regenaxer>
Not needed I think. man pages are good
<
Regenaxer>
hmm, ok, perhaps you are right
<
Regenaxer>
memory handling of C
<
Regenaxer>
is not in man pages ;)
<
Regenaxer>
We still have a bug in pil21 family RPC
<
Regenaxer>
'sync' does not work sometimes
<
Regenaxer>
hard to debug
<
aw->
is that used only for DB?
<
Regenaxer>
not only
<
Regenaxer>
I noticed when testing app/ though
<
Regenaxer>
but all (tell ..) or (boss ..) etc depend on it
<
Regenaxer>
in the app/ case it was also not DB
<
Regenaxer>
but syncing processes after locking an object in the GUI
<
Regenaxer>
took me since yesterday to find
<
Regenaxer>
almost a day
<
Regenaxer>
the sync call is in a finally clause
<
Regenaxer>
so a 'throw' fails when unwinding the stack
<
Regenaxer>
Really tough to trace down where exactly it failed
<
Regenaxer>
cause the throw never arrived
<
Regenaxer>
I thought catch/throw is the culprit
<
aw->
i have some code doing (throw) as well but didn't notice any issues
<
Regenaxer>
only in pil21
<
Regenaxer>
and catch and throw seem fine
<
aw->
none of my tests failed.. either the bug is really obscure, or my tests are incomplete ;)
<
Regenaxer>
but finally clauses are executed in throw
<
Regenaxer>
ok, good to know
<
Regenaxer>
It is in line 253 in lib/form.l
<
Regenaxer>
(in old pil; the file changed in pil21)
<
Regenaxer>
there is (sync) in a 'finally' clause
<
Regenaxer>
there it hangs
<
Regenaxer>
don't know the reason yet
<
Regenaxer>
basically the RPC works, I did some other tests
<
Regenaxer>
ok, must go. Will debug more later
orivej has quit [Ping timeout: 256 seconds]
<
tankf33der>
i will add tests for OOP this weekend.
<
Regenaxer>
I don't find it
<
Regenaxer>
the bug in family IPC
anddam has quit [*.net *.split]
anddam has joined #picolisp
orivej has joined #picolisp
orivej has quit [Ping timeout: 272 seconds]
<
Regenaxer>
Ha! Found it
orivej has joined #picolisp
<
tankf33der>
oop passed tests, added for future run.
<
Regenaxer>
misc/stress.l gives an error
<
Regenaxer>
but
*not* with (gc 99)
<
Regenaxer>
No idea what might be the reason
<
tankf33der>
so, need gc+ run :)
<
Regenaxer>
could be indeed gc, or something still with IPC
<
Regenaxer>
yes, gc+ might give an idea :)
<
Regenaxer>
misc/stress.l does not much in heap, I rather suspect it is an sync/IPC issue
<
tankf33der>
misc/stress.l from picolisp dist ?
<
Regenaxer>
yes, exactly the same
<
tankf33der>
i will try asap
<
Regenaxer>
$ ./pil misc/stress.l -main -go -bye; rm db/test jnl db/test2
<
Regenaxer>
or, in my case, ../pil21/pil misc/stress.l ...
<
tankf33der>
$ ./pil21 misc/stress.l -main -go -bye; rm db/test jnl db/test2
<
tankf33der>
in my case.
<
Regenaxer>
yes, I know, you renamed pil
<
Regenaxer>
I keep the directories separate, too many differences
<
tankf33der>
stress.l gives error fast?
<
Regenaxer>
No, a little over 3 minutes
<
Regenaxer>
It runs through, but then the DB sanity checks fail
<
Regenaxer>
So it is probably not a "normal" gc problem
<
Regenaxer>
At the end something like
<
Regenaxer>
1 2 (1168 . 1168)
<
Regenaxer>
{3} (973 . {640}) dat +A {1}
<
Regenaxer>
{553} -- Bad sequence
<
tankf33der>
i see process have 130MB, so (gc 99) is not the case.
<
Regenaxer>
But with ... misc/stress.l -main -"gc 99" -go - ... it passes
<
tankf33der>
1 2 (1175 . 1175)
<
tankf33der>
{5} -- Bad sequence
<
tankf33der>
{3} (1014 . {675}) dat +A {1}
<
tankf33der>
yes, crashed. ok
<
Regenaxer>
also -'gc 9' seems ok
<
Regenaxer>
Each child process inherits the heap size
<
Regenaxer>
now an error even with 99
<
Regenaxer>
db/test db/test2 differ: byte 7, line 1
<
Regenaxer>
So there is trouble ;)
<
Regenaxer>
It worked several times with 99
<
tankf33der>
{3} (999 . {673}) dat +A {1}
<
tankf33der>
db/test db/test2 differ: byte 7, line 1
<
tankf33der>
1 2 (1168 . 1168)
<
tankf33der>
{3} (999 . {734}) key +A {1}
<
tankf33der>
with -"gc 256"
<
Regenaxer>
Same here
<
Regenaxer>
It is not a gc issue
<
Regenaxer>
another heisenbug
<
Regenaxer>
timings in process syncing
<
tankf33der>
running gc-plus
<
tankf33der>
$ pil21 db.l
<
tankf33der>
[db.l:14] !? (val *DB)
<
tankf33der>
Bad DB file
<
tankf33der>
under gc-plus
<
tankf33der>
oop.l passed, btw.
<
tankf33der>
line 14 is ^^^
<
Regenaxer>
good, very basic tests
<
Regenaxer>
Without gc+ it passes, right?
<
tankf33der>
for a several weeks already
<
tankf33der>
and latest today tests too.
<
tankf33der>
all basic tests passed too: flow, io, mapping, lists.
<
tankf33der>
all basic tests passed in gc+ too: flow, io, mapping, lists.
<
Regenaxer>
Why is line 14 = 22 ?
<
Regenaxer>
this one (test *DB (val *DB) (lieu *DB)) ?
<
tankf33der>
aaaaaaa.
<
tankf33der>
wrong. forget it, checking again.
<
tankf33der>
db.l passed.
<
tankf33der>
i have my own stress variation, doing.
<
tankf33der>
passed.
<
tankf33der>
now misc/stress.l
<
Regenaxer>
The above db.l does not pass under gc+ ?
<
tankf33der>
tankf33der> db.l passed.
<
Regenaxer>
also with gc+ ?
<
tankf33der>
of course.
<
tankf33der>
all above under gc+
<
tankf33der>
now misc/stress.l under gc+
<
Regenaxer>
app/ demo crashes in certain cases
<
Regenaxer>
I will debug tomorrow, cause I can reproduce
<
Regenaxer>
misc/stress.l under gc+ will not run so very much longer I think
<
Regenaxer>
cause the system calls and syncing are the sage
<
Regenaxer>
... are the same
<
tankf33der>
maybe somehow make it smaller.
<
tankf33der>
required minimum
<
tankf33der>
in will sleep in 20mins
<
Regenaxer>
Lets stop for today
<
Regenaxer>
Good night!