<Regenaxer>
or implement the bit operations in Lisp
<beneroth>
and Bytes aren't really bytes. maybe name more accurately? CharBytes?
<aw->
yes sorry naming is bad
<beneroth>
pretty tricky stuff
<aw->
its just an example for testing
<beneroth>
yeah no worries
<Regenaxer>
well, a list of bytes
<beneroth>
naming is the most difficult task
<Regenaxer>
and off ry one ;)
<beneroth>
Regenaxer, it might be a list of multi-byte-values
<Regenaxer>
bd
<Regenaxer>
ah
<Regenaxer>
no
<Regenaxer>
I think single bytes, no?
<beneroth>
no? than I misunderstand
<Regenaxer>
from (rd 1)
<aw->
(rd 1) is one byte, but (char) is not a utf-8 char.. not necessarily 1 byte
<Regenaxer>
T
<aw->
err
<aw->
(char) is utf-8 char
<beneroth>
Regenaxer, the argument to 'make-bytes is Bytes, but the return value is a list of numbers, which is the argument for utf8-str
<aw->
1-4 bytes?
<aw->
beneroth: please dont focus on the name of variables and functions
<aw->
it's just an example
<Regenaxer>
hehe :)
<beneroth>
aw-, yeah, but Regenaxer just though all Bytes are the same :P
<aw->
i could call them Peanuts and it changes nothing
<beneroth>
kinda proving my point
<beneroth>
sure
<beneroth>
call one list Peanuts and the other Bytes
<Regenaxer>
Some bytes are more equal than others ;)
<beneroth>
exactly
<aw->
haha yeah, like 1 + 1 = 3
<beneroth>
more like (setq One 2)
<Regenaxer>
yep
<beneroth>
technically makes no difference, but it stinks :P
<beneroth>
(+ 1 One) -> 3
<aw->
beneroth: can you suggest a code optimization to my paste above?
<Regenaxer>
One nice exercise would be poking a symbol name with 'byte' and 'adr'
<Regenaxer>
Just that 'adr' changes with pil21 a little
<aw->
oh you mentioned that yesterday
<beneroth>
rename the argument of utf8-str, call it CharBytes or CharNums or so
<beneroth>
Regenaxer, yeah
<aw->
64-bit only though.. i need it to work on 32-bit
<beneroth>
aw-, because the Bytes in both functions is a complete different data structure
<Regenaxer>
on 32 bit it works too
<Regenaxer>
can have up to 4 bytes in the name cell
<Regenaxer>
so enough
<aw->
beneroth: .... that's not optimization, that's just wasting time now. The problem is NOT the variable names
<beneroth>
aw-, ok. I don't want to step on your toes.. just pointing out that Regenaxer misunderstood the example at first exactly because of the names.
<Regenaxer>
If you poke e.g. 3 bytes, the symbol contains the ready-made char
<aw->
Regenaxer: "(byte 'num ['cnt]) -> cnt
<aw->
(64-bit version only)"
<Regenaxer>
oh, sorry!
<Regenaxer>
I forgot
<aw->
hmm.. but maybe i can (seek) 3 bytes, and then (skip) 3 ?
<Regenaxer>
no, same problem
<Regenaxer>
seek is not supported in high-level read
<beneroth>
implement something in C and use native. or pilasm and make a ht-style library.
<beneroth>
for best optimization
<Regenaxer>
What if you use the code from pil21?
<aw->
:\
<Regenaxer>
It *is* Lisp already
<Regenaxer>
getChar
<beneroth>
Regenaxer, paste or file?
<aw->
(getChar)
<aw->
i had the file open all day today
<Regenaxer>
change it to parse a list of bytes
<Regenaxer>
ok
<aw->
yeah that's not a bad option too
<Regenaxer>
beneroth, which paste?
<beneroth>
nevermind
<Regenaxer>
the line ((== C (hex "FF")) (i32 TOP)) can be removed
<Regenaxer>
and (call $Get) replaced with (++ Lst) or so
<Regenaxer>
'shl' -> '>>' with negative arg
razzy has joined #picolisp
<Regenaxer>
Ha! *If* the max length is less than 128, you could use (char) for everything!!
<aw->
i looked at src64/io.l, getChar is much shorter in pil21
<Regenaxer>
is that the case?
<aw->
no
<Regenaxer>
ok
<Regenaxer>
yeah, asm is longer
<aw->
max length is 2^16-1
<Regenaxer>
but I think same algo
<Regenaxer>
ok
<aw->
(2^16)-1
<aw->
shl is shift left?
<Regenaxer>
yes
<Regenaxer>
on Lisp we have only '>>'
<aw->
so (>> (- B) .. ?
<Regenaxer>
it is always constant
<Regenaxer>
(>> -6 ..
<aw->
oh ok
<aw->
ohhh i see
<aw->
(shl B 6) is really (>> -6)
<aw->
ok i get it
<Regenaxer>
yes
<aw->
what is this: (shl (& C 7) 6) ?
<Regenaxer>
(>> -6 (& C 7))
<Regenaxer>
take the lowes 3 bits and shift
<aw->
ok great
<aw->
thanks
<Regenaxer>
That's indeed the easiest and cleanest
<Regenaxer>
and probably faster than a pipe
<beneroth>
certainly faster
<beneroth>
no buffers, no extra process
<beneroth>
nice solution
<beneroth>
Regenaxer, so pil21 picolisp-libraries could be written in pil-llvm (or how it is called?), which is more friendly than pilAsm, right?
<Regenaxer>
right
<Regenaxer>
shared libs
Blukunfando has quit [Read error: Connection reset by peer]
<aw->
yes i love that 'feature' of pil21
<aw->
Regenaxer: (in Fd (char)) <-- "the next character from the current input stream is returned as a single-character transient symbol"
<Regenaxer>
pilSrc is easier than pilAsm, but it is still necessary to think in structures of the underlying machinery (now LLVM)
<aw->
is there another way to get the 'next character from input stream as single-character transient?
<aw->
or is my only choice to use getChar from oi.l ?
<Regenaxer>
(peek)
<aw->
hmmm
<aw->
but it doesn't advance the read pointer
<Regenaxer>
yes
<Regenaxer>
(char) returns the next and advances
<Regenaxer>
What other choice would you desire?
<aw->
well, no i was just looking at other options outside of using (pipe) and then reading with (char)
<aw->
and outside of using a custom getChar
<aw->
i'm looking at totally different approach now
Blukunfando has joined #picolisp
<aw->
maybe i can loop and decrement a counter for each char i read
<aw->
until the counter = le0
<aw->
should error if the count is lower than zero
<aw->
that means it read more bytes than it should
<Regenaxer>
But the problem is that the binary counts are non-utf8
<Regenaxer>
intermixed
<Regenaxer>
so (char) fails
<Regenaxer>
gets out of sync
<aw->
hmm
<aw->
what if there's only one binary read, and the rest is all (char) ?
<Regenaxer>
mom, brb
<Regenaxer>
ret
<Regenaxer>
Why should that help? The counts in between are binary again as I understand it
<aw->
oh, I think i got it
<aw->
using loop
<aw->
wow this is so much simpler
<Regenaxer>
cool
<aw->
wait i'll send to pastebin
<Regenaxer>
ok
<aw->
ahhh no it doesn't work.. again likely because of (char)
<aw->
i tried to do (in Fd ..) twice
<aw->
Regenaxer: when you say (char) advances
<aw->
does that mean the next call to (char) contains the previous read?
<beneroth>
kinda
<aw->
that's what i'm seeing
<Regenaxer>
It returns the next prefetched char
<beneroth>
first (char) reads actually 2 characters: 1 (return value) and 1 (peek buffer)
<beneroth>
next (char) then returns the (peek) buffer and puts a new char into it
<aw->
because i do (rd 1) and i get the next byte, and if I do (char) then i get the byte that was BEFORE the (rd 1)
<beneroth>
(or hangs, if the input stream is a pipe with nothing more buffered)
<aw->
hmmm
<aw->
so the data is not _lost_
<beneroth>
T, it is in an additional buffer in between
<Regenaxer>
To be exact, the look-ahead reads only always a *byte*
<beneroth>
oh okay
<Regenaxer>
as this is enough for delimiters
<beneroth>
Regenaxer, what is (peek) doing then? reading until it has a full char in the look-ahead buffer?
<Regenaxer>
Yes
<beneroth>
I see
<beneroth>
I also see why :)
<Regenaxer>
When called, it looks at that byte
<Regenaxer>
and fetches more if necessary
<beneroth>
yeah, because that might be enough to abort/cancel here
orivej has joined #picolisp
<beneroth>
not prematurely reading more than asked
<Regenaxer>
yeah
<Regenaxer>
maybe (peek) is not completely correct then
<beneroth>
Regenaxer, btw. is my understanding correct that (eol) never gets reseted?
<Regenaxer>
No, it will reset
<Regenaxer>
It also looks at that byte
<beneroth>
ok thx
<Regenaxer>
The global $Chr variable
<Regenaxer>
Hmm, so (peek) may be buggy in fact
<beneroth>
how?
<beneroth>
you mean in general, or just in pil21 ?
<Regenaxer>
In general
<Regenaxer>
I have never tried
<aw->
hmmm
<Regenaxer>
but if the next char is more than one byte, (peek) will fetch it
<Regenaxer>
So the next (char) will return something different
<aw->
how do i get the value in the buffer?
<Regenaxer>
Which value?
<beneroth>
you mean: without reading/updating the buffer?
<beneroth>
the byte value?
<Regenaxer>
the global $Chr ?
<aw->
wait
<beneroth>
I guess it is not a byte, it is a picolisp number, right Regenaxer ?
<aw->
the value that was prefetched with (char)
<Regenaxer>
it is not just 'char'
<beneroth>
simple answer: (peek) but... (peek) might do more reading
<Regenaxer>
All reading always fetches the next byte
<beneroth>
it's all text-input functions
<beneroth>
or well, all input functions except (rd 'num) ? Regenaxer ?
<Regenaxer>
(peek) is used typically by parsers, and the delimiters are always single-byte, so I never thought that 'peek' might be wrong
<Regenaxer>
yes
<beneroth>
aw-, char/peek/line/skip/read ...
<Regenaxer>
binary read is separate
<beneroth>
but not plio read? or plio read, too?
<Regenaxer>
(echo) also maintains the look-ahead
<Regenaxer>
so you can mix (echo) with char, line, read etc.
<Regenaxer>
Yes, plio is separate
<beneroth>
makes sense
<aw->
hah
<aw->
ok
<aw->
i got it!
<beneroth>
though (echo) doesn't stumble over NULL bytes, while (till) always stops at a NULL byte (afaik)
<aw->
but
<aw->
this seems like a bug in PicoLisp
<beneroth>
aw-, I'm excited for your solution
<aw->
for this to work, is so strange
<Regenaxer>
I think none stop at a null byte
<Regenaxer>
only -1 (EOF)
<Regenaxer>
But *symbol* names with a null byte are truncated
<Regenaxer>
like in C
<beneroth>
not same thing, I know.
<aw->
ok let me cleanup this example and you guys can checkc
<aw->
Regenaxer: please have a look when you have time
<aw->
beneroth too ;)
<Regenaxer>
beneroth, right
<Regenaxer>
aw-, but isn't this what I proposed and you said it won't work?
<Regenaxer>
i.e. use (char) to get the count?
<Regenaxer>
Works only for less than 128
<beneroth>
aw-, (bytes) is not what you believe it is. (bytes) is PLIO, that is meant by "encoded binary format", it's not about usual binary bytes...
<aw->
ohhh
<beneroth>
and yeah, I also think your counters go of when 1 char <> 1 byte
<beneroth>
s/of/off
<aw->
but i have a Japanese character in there, it's 3 bytes
<beneroth>
for the test you read from a file, but in production it should read from a socket/pipe ?
<aw->
beneroth: yes
<Regenaxer>
Minor detail: (<= 0 N) ls (le0 N)
<beneroth>
aw-, I think you should retrieve the last character with (peek) instead of (char), because (char) will block if the input buffer is empty (because your file signals EOF, but the socket is probably not doing that)
<beneroth>
I ran into this problem some weeks ago
<Regenaxer>
In any case, I think reading mixed formats won't work
<Regenaxer>
Why don't you do the reading with the bitwise converter?
<aw->
yeah.. this approach doesn't work
<Regenaxer>
and read the list as before?
<aw->
getChar ?
<Regenaxer>
yes, a variant of that
razzy has left #picolisp [#picolisp]
<Regenaxer>
Even no list needed
<Regenaxer>
Just use (rd 1) in the conversion routine
<Regenaxer>
instead of (call $Get) in 'getChar'
<Regenaxer>
So you can base all your input on (rd1)
<Regenaxer>
(rd 1)
<Regenaxer>
The count is always 2 bytes?
<beneroth>
mixing text processing and byte processing in picolisp is not comfortable. Doing either thing is well, but mixing is hell :)
<Regenaxer>
you seem to read only one
<Regenaxer>
beneroth, well stated :)
<aw->
one or two.. depends
<aw->
2 bytes i think
<Regenaxer>
How do you know?
<Regenaxer>
If 2 bytes, you need to know the endianess
<aw->
big endian
<aw->
msb then lsb
<beneroth>
probably guaranteed to be big endian in the network protocoll
<beneroth>
T
<Regenaxer>
So (rd 2)
<Regenaxer>
I would do (rd 2) and then loop with (rd 1) building the UTF-8 chars with the standard algorithm
<aw->
ok one sec i'm trying now
<aw->
hmmm ok i think i have one small mistake
<aw->
wait
<aw->
i don't fully grok the getChar yet
<aw->
so i think i'm not doing it correctly
<aw->
Regenaxer: i need to (loop (getChar (rd 1)...)
<aw->
something like that, right?
<Regenaxer>
I would not pass an argument
<Regenaxer>
i.e. do the first (rd 1) in the function
<Regenaxer>
So it returns the next char, by reading 1 ... 4 bytes
<Regenaxer>
I would not even write a separate function
<Regenaxer>
but do the (rd 1) together with a decrement of the count directly
<aw->
ughhh
<aw->
one sec
<aw->
getChar only reads ONE char, right?
<Regenaxer>
Otherwise the function needs to return *two* values
<Regenaxer>
yes
<Regenaxer>
1 ... 4 bytes
<aw->
so how do I "do the first (rd 1) in getChar ?
<aw->
or.. why would i do it there?
<Regenaxer>
so it must return the char AND the size, so that the caller can decrement
<Regenaxer>
No getChar
<Regenaxer>
Just do it inline
<Regenaxer>
it is easier :)
<aw->
?
<aw->
(de getChar ()
<aw->
(let C (rd 1)
<aw->
??
<Regenaxer>
Otherwise you must keep the count in a global and derement it
<Regenaxer>
Yes,
<Regenaxer>
but what with the Count?
<aw->
what count?
<Regenaxer>
The count you talk about all the time
<aw->
well that's why i was doing it outside of getChar
<Regenaxer>
(rd 2)
<Regenaxer>
sigh
<Regenaxer>
yes
<Regenaxer>
but then you need to return the SIZE also from getchar, no?
<Regenaxer>
Or use a globel which is decremented
<Regenaxer>
You must decrement for each (rd 1) in the function
<Regenaxer>
So no function is needed
<Regenaxer>
it is only a handful of lines, right?
<aw->
soo, too much information too quickly,
<aw->
give me a few minutes
<aw->
i dont even understand what i'm looking at yet
<Regenaxer>
I replaced (ifn (...)) with (if (=0 (...))
<aw->
what is different?
<Regenaxer>
only that
<aw->
oh
<aw->
that works!
<Regenaxer>
Sorry
<aw->
no problem
<Regenaxer>
We must always keep in mind that pil21src looks like Lisp but it is not!
<Regenaxer>
Also new for me
<Regenaxer>
It is LLVM with s-expr syntax
<aw->
right...
<Regenaxer>
temptating to port as it is ;)
<aw->
indeed.. well (shl) kind of gives it away
<aw->
and also: (de i32 getChar ((i32 . C))
<Regenaxer>
right
<Regenaxer>
shl and shr shift 32 or 64 bits
<aw->
ok
<aw->
thanks for your help
<Regenaxer>
and even 128 bits to get overflow bits
<Regenaxer>
Welcome
<aw->
this is much more efficient than my first implementation
<aw->
with (pipe)
<Regenaxer>
have you measured?
<aw->
yes
<aw->
0.004 sec old approach
<aw->
0.000 sec new approach
<aw->
and this is for **very** small data, 10 bytes?
<aw->
haha
<Regenaxer>
wuff
<Regenaxer>
yeah, fork overhead
Blukunfando has quit [Ping timeout: 256 seconds]
<beneroth>
if you keep the pipe open, that overhead is only payed once ;-)
<beneroth>
nice discussion you had
<beneroth>
very interesting
<aw->
beneroth: true, but for a network app which has potentially many connections, it's bad
<beneroth>
T
<beneroth>
well, if you need one pipe per connection
<Regenaxer>
yeah, could keep a worker pipe in an Fd
<beneroth>
I've cases where I do that. But yeah aw-'s use case needs to send the input from the client connections through that, so one pipe would be a bottleneck
<beneroth>
I figure
<Regenaxer>
T
<Regenaxer>
And tedious to manage
aw- has quit [Quit: Leaving.]
aw- has joined #picolisp
mtsd has quit [Quit: Leaving]
<aw->
Regenaxer: i just noticed an error in your latest version, trying to debug now
<Regenaxer>
great
<Regenaxer>
in pil21?
<aw->
no no sorry
<aw->
the code you pasted earlier
<aw->
utf-8 ;)
<Regenaxer>
ah
<Regenaxer>
ok :)
<aw->
not a bug
<aw->
ok fixed
<aw->
your version was reading the entire stream
<aw->
because you had (while (rd 2) ...
<Regenaxer>
yeah
<aw->
i want it to read only up to the end of the utf-8 chars
<aw->
so i can call the same function multiple times
<Regenaxer>
good
<aw->
easy change
rob_w has quit [Quit: Leaving]
Blukunfando has joined #picolisp
mtsd has joined #picolisp
mtsd has quit [Quit: mtsd]
orivej has quit [Ping timeout: 240 seconds]
jibanes has quit [Ping timeout: 240 seconds]
jibanes has joined #picolisp
orivej has joined #picolisp
orivej_ has joined #picolisp
orivej has quit [Quit: No Ping reply in 180 seconds.]