ChanServ changed the topic of #picolisp to: PicoLisp language | Channel Log: https://irclog.whitequark.org/picolisp/ | Check also http://www.picolisp.com for more information
_whitelogger has joined #picolisp
orivej has quit [Ping timeout: 256 seconds]
_whitelogger has joined #picolisp
dexen has quit [Ping timeout: 246 seconds]
rob_w has joined #picolisp
orivej has joined #picolisp
dexen has joined #picolisp
<aw-> Regenaxer: here?
<Regenaxer> yup
<aw-> hi
<Regenaxer> hi :)
<aw-> in pil21, is there an easier way to write this (native) result specification: (list 'Attr (64 N . 4) . 0)
<Regenaxer> You could (64 N N N N) . 0
<Regenaxer> nit easier
<Regenaxer> well, it is constant
<Regenaxer> no 'list' needed
<Regenaxer> '(Attr (64 N . 4) . 0)
<Regenaxer> so less 'cons'ing at runtime
<aw-> oh good
<aw-> that works in pil64 too
<Regenaxer> yes
<aw-> ok this is good
<aw-> thanks
<Regenaxer> :)
<Regenaxer> Also, 'list' does not what you want
<Regenaxer> : (list 'Attr (64 N . 4) . 0)
<Regenaxer> -> (Attr (64 N . 4))
<Regenaxer> You'd need 'cons'
<Regenaxer> : (cons 'Attr (64 N . 4) 0)
<Regenaxer> -> (Attr (64 N . 4) . 0)
<aw-> oh wow you're right
<aw-> well i had already fixed it to '(Attr (64 N N N N) . 0)
<Regenaxer> ok :)
<aw-> is that wrong?
<Regenaxer> No, looks good
<aw-> ok thank you
rob_w has quit [Quit: Leaving]
patrixl has quit [Read error: Connection reset by peer]
patrixl has joined #picolisp
<aw-> Regenaxer: i think i asked you already, but if I do (in Fd (rd 4)) to read 4 bytes.. it reads as an Int.. is there a way to conver that to 4 separate bytes?
<Regenaxer> Instead of converting, better read the bytes directly
<Regenaxer> (in Fd (make (do 4 (rd 1))))
<Regenaxer> (in Fd (make (do 4 (link (rd 1)))))
<Regenaxer> Conversion would need & and >> operations
<aw-> right.. ok that's what i was doing (make (do ...)
<aw-> thanks
<Regenaxer> :)
orivej has quit [Ping timeout: 240 seconds]
<Regenaxer> Fixed a bug in ht:Fmt
<tankf33der> i wonder if i could implement this exponent reducing
<tankf33der> hm
<Regenaxer> yeah, symbolic algebra
<Regenaxer> Demo "app/" seems to work fine in pil21 now
<Regenaxer> oops, no
<Regenaxer> hangs in (sync)
<Regenaxer> Any new insights with the "uninitialized" write() buffer?
<Regenaxer> Or is it fake news?
<tankf33der> written to issue’s list for history.
<Regenaxer> Well, *is* it an issue?
<tankf33der> i could repeat it with readline calls too
<Regenaxer> What exactly goes wrong?
<tankf33der> you use raw memory and sanitizer generates false alarm
<Regenaxer> What is raw memory?
<tankf33der> raw memory is something you didnt filled with zeroes before using :)
<Regenaxer> Filling memory with zero is nonsense, no?
<Regenaxer> and the buffer we write is completely filled with data
<Regenaxer> There is something we don't understand about this test
<tankf33der> or you stepped out or collapsed with something
<tankf33der> we could fill and try again
<Regenaxer> No, I cannot fill all memory I use
<Regenaxer> heaps, and many buffer
<Regenaxer> that's nonsense
<Regenaxer> Isn't there some better explanation of the check tools?
<tankf33der> nope.
<Regenaxer> What is "address+undefined sanitizers"?
<Regenaxer> No description somewhere?
<tankf33der> will play with this ^^^
<Regenaxer> thanks!
<Regenaxer> I read the docs
<tankf33der> http://ix.io/2yGG
<tankf33der> see, now it tells where uninit memory was born
<tankf33der> ==========
<tankf33der> ...
<tankf33der> Uninitialized value was created by a heap allocation
<tankf33der> #0 0x43366f in realloc (/root/pil21/bin/picolisp+0x43366f)
<tankf33der> #1 0x4be075 in alloc (/root/pil21/bin/picolisp+0x4be075)
<tankf33der> ...
<tankf33der> ==========
<beneroth> question is, does the checker check for some "nice to have" standard or finding a real issue
<Regenaxer> right
<Regenaxer> alloc() if course does not init
<Regenaxer> it is the same place as yesterday
<Regenaxer> writing 159 bytes
<Regenaxer> all initialized!
<beneroth> use = reading of those memory at #0 0x4c017d in wrBytes (/root/pil21/bin/picolisp+0x4c017d)
<beneroth> ?
<Regenaxer> not sure
<beneroth> or it doesn't like the writing without having set it previously to 0 (which of course is irrelevant)
<Regenaxer> strace yesterday:
<Regenaxer> write(1, "93326215443944152681699238856266"..., 159) = 159
<Regenaxer> the written number is correct
<tankf33der> Regenaxer: alloc() is calloc(), right ?
<Regenaxer> It calls realloc() iirc
<tankf33der> because malloc() does not init memory
<tankf33der> calloc() does.
<tankf33der> calloc() does by zeroes.
<Regenaxer> I dont want to waste time initializing
<Regenaxer> not useful at all
<beneroth> which is good practice, but unnecessary performance cost if you can guarantee to never read from that memory without writting first
<Regenaxer> sure
<beneroth> well possible that the checker just checks for idealistic reasons...
<Regenaxer> else you get nonsense data
<Regenaxer> and if initialized to zero it still gives nonsense
<Regenaxer> I dont think so
<Regenaxer> No software does that
<beneroth> well reading from possible uninitialized or deleted memory is also a main vector to attack C programs (overwriting the value of function pointers)
<beneroth> initializing to zero can block such a security attack
<tankf33der> you could modify just for testing and then return back.
<beneroth> T
<Regenaxer> If a bad process can read the heap, it does not help
<Regenaxer> free()d mem is just normal heap
<beneroth> its a clang feature? so from the same guys who hate dynamic sized arrays just by principle?
<Regenaxer> Even that I wont believe ;)
<Regenaxer> No, it may well be I have indeed a fault
<Regenaxer> but which one?
<beneroth> you need to factor more cargo-culting into your model of how other devs think, Regenaxer ;-)
<beneroth> T
<beneroth> it might be a real issue
<Regenaxer> yes, quite probable
<Regenaxer> beneroth, btw, I introduced a new concept in pil21 namespaces
<Regenaxer> 2 days ago
<Regenaxer> I like it :)
<Regenaxer> It concerns the (private)s
<Regenaxer> (private) is also in pil21 recently, but I think it is wrong
<Regenaxer> or at least dangerous
<Regenaxer> pil21 now has a special namespace built-in, in addition to the internal, transient and external namespaces
<Regenaxer> this 'priv' namespace behaves in a special way
<Regenaxer> that's why I said "a new concept"
<Regenaxer> (despite I try to avoid too many concepts)
<beneroth> the amount of namespaces might become too big xD
<Regenaxer> Nono :)
<beneroth> I still use "normal" transient often, but I also burned myself with not thinking about a (load) in the middle ;)
<Regenaxer> yes
<Regenaxer> And transients look ugly when used as variables
<Regenaxer> I rewrote @lib/xhtml.l and @lib/form.l with (private)
<Regenaxer> in pil21 I mean
<Regenaxer> But transients continue to work as before
<Regenaxer> Still nice for a quick local symbol
<Regenaxer> But 'priv' in pil21 is always there
<Regenaxer> something between transients and a normal namespace
<Regenaxer> I'll explain later
<Regenaxer> When it is clear that it works out well
<Regenaxer> But so far it looks good
<aw-> Regenaxer: not long ago we discussed parsing a list like this: (2 97 98 5 99 100 26085) so it returns -> "ab" and "cd日"
<beneroth> Regenaxer, yeah I think I understand :)
<aw-> the problem was mixing rd and char
<aw-> so you suggested an alternate utf-8 parsing
<beneroth> hi aw- :)
<aw-> hi beneroth
<aw-> so i wonder if i can just add a null byte at the end of each string
<aw-> like this: (2 97 98 0 5 99 100 26085 0)
<aw-> or (3 97 98 0 6 99 100 26085 0)
<aw-> but in that case i guess i dont need the length byte
<aw-> ok never mind ;)
<aw-> actually i'm just curious
<Regenaxer> hmm, I don't remember exactly
<aw-> how does (char) behave when there's a null byte in the next character?
<Regenaxer> 'C' cannot be used?
<aw-> 'C' ? this is a pure picolisp thing, not native
<Regenaxer> I think (char) returns "^@'
<Regenaxer> I mean the 'C' spec
<Regenaxer> like 'B' and 'N' etc
<aw-> (in Fd (char))
<Regenaxer> Ah, I thought for 'native'
<Regenaxer> ok, it returns an anonymous symbol:
<Regenaxer> : (out "a" (wr 0)) (in "a" (char))
<Regenaxer> -> $344610442461
Regenaxer has left #picolisp [#picolisp]
Regenaxer has joined #picolisp
<aw-> hmmm
<Regenaxer> cause it is a symbol without name
<Regenaxer> Null-bytes cannot be represented in Pil symbol names
<Regenaxer> like in C
<Regenaxer> they are a terminator
<Regenaxer> So you are reading non-UTF chars?
<Regenaxer> (char) is not suitable then
<Regenaxer> it needs UTF-8
<aw-> no no they are UTF-8
<Regenaxer> ok
<Regenaxer> : (out "a" (wr 66 0))
<Regenaxer> : (in "a" (list (char) (char))
<Regenaxer> -> ("B" NIL)
<Regenaxer> So it returns NIL
<Regenaxer> The issue is problematic
<aw-> ahhh ok
<aw-> it returns NIL
<Regenaxer> yes, but I'm not sure if this is guaranteed
<Regenaxer> NULL bytes are kind of illegal
<Regenaxer> beyond the standard data
<Regenaxer> You also cannot well edit such files
<Regenaxer> Can't you take another delimiter?
<Regenaxer> Ideal is "\t"
<aw-> ideally i would have no delimiter
<Regenaxer> cause you have a count?
<aw-> yes, with a length byte at the beginning of the list
<Regenaxer> A delimiter is easer and more powerful
<Regenaxer> eg if the data are more than 256 bytes
<aw-> ok wait
<aw-> let me show you an example
<aw-> compare these two:
<aw-> (in (pipe (wr 4 97 230 151 165 4 100 230 151 165)) (make (rd 1) (link (char) (char)) (rd 1) (link (char) (char))))
<aw-> (in (pipe (wr 5 97 230 151 165 0 5 100 230 151 165 0)) (make (rd 1) (link (char) (char)) (rd 1) (link (char) (char))))
<aw-> i'm trying to understand why the second option works correctly, and the first doesn't
<Regenaxer> hmm, both seem wrong
<Regenaxer> you skip the count
<aw-> yes i know.. it's just an example
<Regenaxer> then read one or twe
<Regenaxer> What's the difference then?
<aw-> ("a" "日" "d" "日")
<aw-> that's the correct result
<aw-> but in the first example i get something different
<Regenaxer> Anyway, I told you that rd and high-level ffered read
<Regenaxer> cant ae mixed
<Regenaxer> unpredictaale
<aw-> ahh ok
<Regenaxer> oops
<aw-> so the second example, with the null bytes, is just coincidence?
<Regenaxer> "ffer" I typed "buffered?
<aw-> it works "by luck" ?
<Regenaxer> yes, depending on the prefetched buffer contents
<aw-> i see
<Regenaxer> I think we thought about read() with native
<Regenaxer> then 'struct' to read the string from the buffer
<aw-> i thought because (char) detects the null byte then it knows to stop that's the end
<Regenaxer> (char) does not specially look for null bytes
<Regenaxer> it simply does not expect them
<aw-> ah
<aw-> ok that makes sense
<Regenaxer> it checks for -1 (eof)
<Regenaxer> so if null arrives, it is mkChar() or so
<Regenaxer> which makes a transient symbol
<Regenaxer> then either NIL or anonymous
<Regenaxer> not sure atm
<Regenaxer> ah, I think we found you can use a function from 'ht'!
<Regenaxer> That was the last insight
<Regenaxer> (ht:Read) ?
<Regenaxer> it does what you need
<Regenaxer> read a given number of bytes
<Regenaxer> into a list of *char*s
<aw-> right
<Regenaxer> : (out "a" (wr 65 66 67 0))
<Regenaxer> -> 0
<Regenaxer> : (in "a" (ht:Read 2))
<Regenaxer> -> ("A" "B")
<Regenaxer> $: (in "a" (ht:Read 3))
<Regenaxer> -> ("A" "B" "C")
<aw-> ohhh right
<aw-> i do remember now
<Regenaxer> Should work with any UTF-8, Kanji, Smileys etc
<Regenaxer> yeah, I forgot too
<aw-> hmmm.. doesnt work
<aw-> one sec
<Regenaxer>
<Regenaxer> : (out "a" (prinl "ab☺cd"))
<Regenaxer> -> "ab☺cd"
<Regenaxer> $: (in "a" (ht:Read 7))
<Regenaxer> -> ("a" "b" "☺" "c" "d")
<aw-> it doesnt accept a variable?
<aw-> (ht:Read Var) ?
<Regenaxer> Why should it not?
<aw-> !? (ht:Read Length)
<aw-> ?
<aw-> NIL -- Small number expected
<Regenaxer> Cant be
<aw-> wait wait
<aw-> i know
<Regenaxer> it should evaluate the normal way
<aw-> sorry
<aw-> it was in a (make
<aw-> haha
<Regenaxer> Problem is the count
<Regenaxer> you can't use (rd 1)
<Regenaxer> (ht:Read (rd 1))
<Regenaxer> cause of the buffering problem
<Regenaxer> (ht:Read (format (char))) or so neither
<Regenaxer> the count byte is binary, right?
<aw-> i can do it early so it's fine
<aw-> yes it is
<aw-> i do (setq Length (in Fd (rd 1)))
<aw-> in a (loop
<Regenaxer> collect all lengths first?
<Regenaxer> The problem is that this format is not good for the Lisp reader
<Regenaxer> Better write a small tool in C
<aw-> why not?
<Regenaxer> to preprocess it
<Regenaxer> you cant mix (rd 1) and ht:Read
<aw-> i'm not
<Regenaxer> ht:Read reads the stream
<aw-> hmmmm
<Regenaxer> 'ht' does in fact someting like that
<Regenaxer> with "chunked" reading
<Regenaxer> always a count followed by data
<Regenaxer> A messy format
<Regenaxer> So if you had that format, you can use lib/ht.so right out of the box
<aw-> too complex
<Regenaxer> yeah, a mess
<Regenaxer> kiss :)
<aw-> ok so
<aw-> (ht:Read (rd 1)) wont work
<Regenaxer> I think so
<aw-> but (let L (rd 1) (ht:Read L)) is ok?
<aw-> i did tests and it works for me
<Regenaxer> Thats exactly the same!
<aw-> i know
<aw-> that's why i dont understand why you said it won't work
<Regenaxer> ht:Read uses (getChar)
<Regenaxer> it is not in sync with (rd)
<Regenaxer> it uses the global prefetch $Chr
<Regenaxer> '_rd' uses 'getBinary'
<Regenaxer> which has its own buffer
<Regenaxer> no char look-ahead needed
<Regenaxer> so faster
<aw-> hmmm
<Regenaxer> If you cannot change the format or write a filter util, you can modify ht:Read perhaps
<Regenaxer> or use native
<Regenaxer> (%@ "read" Fd Buf1 1)
<Regenaxer> (byte Buf1)
<Regenaxer> then use that byte as a count to
<Regenaxer> (%@ "read" Fd Buf2 Cnt)
<Regenaxer> then 'sruct' to get the string
<Regenaxer> you must read Cnt+1 to get also the nullbyte
<aw-> so make a buffer to store the byte, then extract with (struct) ?
<Regenaxer> (strucw Buf2 'S ... gets the string
<aw-> oh ok
<Regenaxer> yes, alloc or 'buf' if pil21
<Regenaxer> malloc I mean
<aw-> hmm
<aw-> im a bit confused
<Regenaxer> Only 1 'Buf' is needed
<Regenaxer> malloc 256
<aw-> instead of using (in Fd) .. i use (native "read" Fd) ???
<Regenaxer> yep
<aw-> i didnt know that was possible hahaha
<Regenaxer> :)
<aw-> why 256?
<Regenaxer> The string cannot be longer
<Regenaxer> Single-byte count
<Regenaxer> (%@ "read" Fd Buf 1) gets the count
<aw-> 256 bits
<aw-> not 256 bytes
<Regenaxer> (%@ "read" Fd Buf (byte Buf)) should do
<aw-> right
<Regenaxer> You have a *bit* count?
<aw-> no no sorry nevermind
<aw-> why you choose 256 bytes for maximum string length?
<Regenaxer> 255 plus null byte
<aw-> i already know the string length in advance
<Regenaxer> can't be more
<Regenaxer> No, you alloc the buffer *once*
<Regenaxer> then read in a loop
<Regenaxer> and even for a single read 256 is best
<Regenaxer> you 'read' two times
<Regenaxer> see above
<aw-> ok i understand, but i want to know what is '256' ?
<aw-> arbitrary number
<aw-> or 255 + null byte, why 255?
<Regenaxer> no!
<aw-> where is it defined?
<Regenaxer> the length can max be 255, right?
<Regenaxer> one byte!!
<aw-> well.. if I use one byte yes
<aw-> i might use 2 bytes for the length
<aw-> so 65536+nullbyte
<Regenaxer> so you *have* it under control?
<aw-> yes i do
<Regenaxer> Why such a stupid format?
<Regenaxer> I give up
<aw-> ahahaha
<aw-> :(
<aw-> well i didn't invent it
<aw-> it's based on TLV: Type-length-value
<Regenaxer> ok :)
<Regenaxer> Anyway, you got the idea
<Regenaxer> If two bytes count, you must be prepared to realloc() your buffer
<Regenaxer> or allocate 65536 in the beginning
<aw-> hmmm... what advantage does native provide?
<aw-> in this case
<Regenaxer> You know a better way?
<aw-> if I do (rd 1) or (rd 2) then i get the 1 or 2 bytes count
<aw-> so can't i use that value to perform malloc with the exact buffer size?
<Regenaxer> yes
<Regenaxer> but you can't (native "read" then
<Regenaxer> As I said, pil binary read also buffers
<Regenaxer> so read() from that Fd gives the *next* 8192 bytes or so
<aw-> ahhh i see
<aw-> ok that was the missing link
<aw-> well in that case, i'll force it to use only 1 byte count (255+nullbyte)
<aw-> but it seems wasteful of resources
<Regenaxer> you can use 2 bytes with native too
<Regenaxer> (native "@" "read" Fd Buf 2)
<Regenaxer> then (byte Buf) and (byte (inc Buf))
<Regenaxer> or better 'struct'
<aw-> haha yeah good one
<aw-> ok let me add this to my notes
<aw-> i'll test tomorrow
<Regenaxer> then aetter 4 bytes
<Regenaxer> (struct Buf 'I)
<Regenaxer> gives the number if 4 bytes
<Regenaxer> with (byte (inc Buf)) you must take care of endianess
<Regenaxer> or, you must do that anyway
<aw-> yes it's network data so no problem
<Regenaxer> if the two bytes in the file are big-endian, and your CPU is little etc
<Regenaxer> ok
<Regenaxer> then two bytes, shifted and or'ed
<aw-> last time you suggested I use the a variation of getChar()
<aw-> works without native
<Regenaxer> really?
<aw-> is the native approach faster than that?
<Regenaxer> Which getChar()?
<aw-> from io.l
<aw-> or somethign similar to that
<Regenaxer> ah, you modified it
<aw-> yes
<Regenaxer> but this is what ht:Read does
<Regenaxer> It still needs the count handling
<aw-> yes i modified it to count
<Regenaxer> and addreses only the assembly of utf8 chars
<aw-> or.. you modified it ;)
<aw-> yes exactly
<Regenaxer> with a count? hehe, I garbage-collected already ;)
<aw-> haha yeah i'm still on a 2-week old problem
<Regenaxer> So yes, doing lots of native read()s is slower
<aw-> hahahaa
<Regenaxer> system calls
<Regenaxer> :)
<aw-> ok
<aw-> well thanks for looking at this with me (again)
<Regenaxer> Then best is to modify Read
<aw-> sorry to distract from your high-speed coding sessions ;)
<Regenaxer> to take the count by itself from the stream
<Regenaxer> No problem!!
<Regenaxer> I'm idling here, Masako does a Sushi course
<Regenaxer> Will listen podcasts now
<aw-> great
<aw-> bedtime for me
<Regenaxer> yeah
<Regenaxer> sleep well!
<aw-> thanks, g'nite
<Regenaxer> :)
orivej has joined #picolisp
orivej has quit [Ping timeout: 272 seconds]