ChanServ changed the topic of #picolisp to: PicoLisp language | Channel Log: https://irclog.whitequark.org/picolisp/ | Picolisp latest found at http://www.software-lab.de/down.html | check also http://www.picolisp.com for more information
m_mans2 has quit [Ping timeout: 260 seconds]
m_mans has joined #picolisp
Regenaxer has quit [Ping timeout: 255 seconds]
linus`__ has joined #picolisp
linus` has quit [Read error: Connection reset by peer]
aw- has joined #picolisp
aw- has quit [Quit: Leaving.]
libertas has quit [Ping timeout: 248 seconds]
aw- has joined #picolisp
aw- has quit [Read error: Connection reset by peer]
aw- has joined #picolisp
libertas has joined #picolisp
orivej_ has quit [Ping timeout: 250 seconds]
Regenaxer has joined #picolisp
<aw-> hi all
<Regenaxer> Hi aw-!
aw- has left #picolisp [#picolisp]
aw- has joined #picolisp
<aw-> hi Regenaxer
<aw-> i have an issue with http.l
<aw-> i think i brought this up a while back but I don't remember how we resolved it
<Regenaxer> hmm, or not solved? ;)
<aw-> when I do a curl -X POST http://localhost -F file=@file.tgz
<aw-> it's a multipart/form-data request
<Regenaxer> ok
<aw-> it takes _forever_ to upload the file
<aw-> ~20 seconds to upload a 300Mb file
<aw-> on localhost
<Regenaxer> Is it emu?
<aw-> I narrowed it down to (_htMultipart)
<aw-> no
<aw-> this is 64-bit arm64
<Regenaxer> ok
<aw-> but same problem on 64-bit x86_64
<aw-> it seems the (out F (echo ... is really slow
<aw-> (unless (out F (echo (pack "^M^J" *MPartLim)))
<aw-> (call "rm" "-f" F) ) )
<aw-> i can see the file upload in ~/.pil/tmp/NNNNN/file.tgz
<aw-> it's always so slow to process
<Regenaxer> It is inside 'echo'?
<Regenaxer> (trace 'echo) ?
<Regenaxer> or (unless (out (tmp S) (bench (echo @))) ?
<aw-> i don't think so, no
<aw-> oh yes
<aw-> it's (echo) that's slow
<aw-> just did the bench, 19.9s
<Regenaxer> Strange, I never observed this
<Regenaxer> Is this also when called in the repl?
<aw-> no, i'm not running the server in a repl
<Regenaxer> (bench (in "file300MiB" (out "/dev/null" (echo))))
<Regenaxer> yes, but a test
<aw-> why /dev/null ?
<aw-> that's not what i'm testing
<Regenaxer> To see if 'echo' is the bottleneck
<aw-> 13.298 sec
<aw-> also much too long
<aw-> 300MiB file should take under 1sec
<Regenaxer> no idea
<aw-> even if the file is in a tmpfs (memory), it takes the same amount of time to run that test
<Regenaxer> hmm
<Regenaxer> I have never measured 'echo'
<Regenaxer> ok:
<Regenaxer> : (out "f" (do (* 30 1024 1024) (prinl "123456789")))
<Regenaxer> -> "123456789"
<Regenaxer> : (bench (in "f" (out "/dev/null" (echo))))
<Regenaxer> 1.604 sec
<Regenaxer> -> T
<Regenaxer> So 'echo' is not the bottleneck
<aw-> it's my computer?
<aw-> i just ran the same test
<aw-> (bench (in "f" (out "/dev/null" (echo))))
<aw-> -> T
<aw-> 19.362 sec
<Regenaxer> oh
<aw-> is there a setting i need to change?
<aw-> maybe ulimit or buffer setting ?
<aw-> ulimit -s ?
<Regenaxer> I did the above test on this server
<Regenaxer> on my Arm64 tablet it takes 5 sec
<aw-> interesting..
<aw-> is your tablet faster than my computer? haha
<Regenaxer> I think it is when writing to compact flash
<Regenaxer> CF write is slow
<aw-> nono
<aw-> i'm doing this all in memory
<aw-> in a tmpfs
<Regenaxer> Cause otherwise the tablet has roughly the same speed as this server
<aw-> can you show your ulimit settings?
<Regenaxer> core file size (blocks, -c) 0
<Regenaxer> data seg size (kbytes, -d) unlimited
<Regenaxer> scheduling priority (-e) 0
<Regenaxer> file size (blocks, -f) unlimited
<Regenaxer> pending signals (-i) 31044
<Regenaxer> max locked memory (kbytes, -l) 16384
<Regenaxer> max memory size (kbytes, -m) unlimited
<Regenaxer> open files (-n) 1024
<Regenaxer> pipe size (512 bytes, -p) 8
<Regenaxer> POSIX message queues (bytes, -q) 819200
<Regenaxer> real-time priority (-r) 0
<Regenaxer> stack size (kbytes, -s) 65536
<Regenaxer> cpu time (seconds, -t) unlimited
<Regenaxer> max user processes (-u) 31044
<Regenaxer> did this go through?
<aw-> yes
<aw-> thanks
<aw-> ok one sec i'm comparing
<aw-> yeah... no difference
<Regenaxer> I'm wrong. Writing to /dev/null does not go to CF anyway
<aw-> when I strace, i see it's reading 8192 bytes at a time
<aw-> or maybe Kbytes
<Regenaxer> no, bytes
<Regenaxer> it is BUFSIZ
<aw-> ?
<aw-> src64/sys/arm64.linux.defs.l:(equ BUFSIZ 8192) ?
<aw-> is there an alternative to using (echo) here ?
<aw-> maybe i can do (while (line) (prin) ?
<Regenaxer> What do you get for 'fibo'?
<Regenaxer> $ pil misc/fibo.l +
<Regenaxer> : (bench (fibo 36))
<Regenaxer> 1.137 sec
<Regenaxer> -> 14930352
<Regenaxer> : (bench (fibo 36))
<Regenaxer> 1.083 sec
<Regenaxer> -> 14930352
<Regenaxer> No, line/prin is not a good idea
<Regenaxer> Much more expensive, allocates lots of memory, and wont work on binary data
<aw-> (bench (fibo 36))
<aw-> -> 14930352
<aw-> 8.718 sec
<aw-> :O
<Regenaxer> on my tablet it is 2.2 sec
<aw-> crazy
<aw-> on my Mac with 32-bit pil it's 9sec
<Regenaxer> yeah, pil32 is slower as it has no short numbers
<aw-> 2sec on my colocation server
<Regenaxer> and generates garbage during arithmetics
<Regenaxer> ok
<aw-> 64-bit x86
<aw-> i have difficulty believing my arm64 CPU is _that_ slow
<Regenaxer> Well, 'echo' is not really optimized too
<Regenaxer> But usually I found it OK
<Regenaxer> What kind of machine is it? RasPI?
<aw-> rock64
<aw-> Quad-Core ARM Cortex A53, 4GB RAM
<Regenaxer> For embedded usage? Or a notebook or desktop?
<aw-> i'm using as desktop
<aw-> ok well, if (echo) performs well on x86-64 then that's fine.. that's my target anyways
<aw-> i only use the arm64 for dev
<Regenaxer> good thing!
<Regenaxer> Does your server have do get such 300 MiB files such often?
<aw-> yes
<aw-> sometimes up to 800Mb
<Regenaxer> I see
<Regenaxer> Using (call "dd" ...) instead of 'echo' might help, but then more needs to be changed
<Regenaxer> eg. how to detect errors
<Regenaxer> to call (call "rm" "-f" (tmp S))
<Regenaxer> The (unless ..) check is to make sure the file arrived completely
<Regenaxer> Also, I don't want to depend on an external program like 'dd' here
<Regenaxer> May not be available in Android apps for example
<Regenaxer> And normally the network is the bottleneck here
<aw-> right
<aw-> yeah like i said it's fine for now since my target is x86-64
<Regenaxer> In principle arm64 is same speed or faster than x84
<aw-> yeah.. dunno
<aw-> i'm onto something else now, this is not that big of a deal since it's fine on x86
<aw-> again in http.l
<aw-> i don't understand in (_htSet)
<aw-> (if (= " multipart/form-data; boundary" (lowc (till "=" T)))
<aw-> how does that work?
<aw-> there's no input
<aw-> doesn
<aw-> doesn't (till) require an input?
<Regenaxer> (till) is an input
<Regenaxer> returns the data till "=" as a symbol
<aw-> hmm
<aw-> yeah..
<aw-> i still don't understand how this works
<aw-> this http server is so complex
<aw-> ("content-type:"
<aw-> *MPartLim (append '(- -) (cdr (line)))
<aw-> (setq
<aw-> (if (= " multipart/form-data; boundary" (lowc (till "=" T)))
<aw-> *MPartEnd (append *MPartLim '(- -)) )
<aw-> (line) ) )
<Regenaxer> The boundary is terminated by a "="
<aw-> i don't understand this
<Regenaxer> I would recommend to single-step through the execution to see what it does
<aw-> it loops through every line in the request
<Regenaxer> yep
<Regenaxer> only the header
<aw-> the case statement checks each header
<Regenaxer> T
<aw-> (lowc (till " ^M^J" T))
<Regenaxer> "X-Pil:" is special, sent by httpGate
<aw-> actually
<aw-> can you explain this:
<aw-> (case (lowc (till " ^M^J" T))
<aw-> ("setq:" (host *Host (cdr (line))))
<aw-> oops
<aw-> (case (lowc (till " ^M^J" T))
<aw-> ("host:" (setq *Host (cdr (line))))
<aw-> i don't understand that part
<aw-> how can "host:" be separated from what comes after the : ?
<Regenaxer> The call (till " ^M^J" T) returns the first part of the line
<Regenaxer> normally until a space
<aw-> hmmm
<aw-> so then why do we have (cdr (line)) ?
<Regenaxer> The cdr is to throw away the space
<aw-> ohhhhhh i see
<aw-> wow that's really not obvious
<Regenaxer> We could call 'clip', but thats more expensive
<aw-> ok so going back to the ("content-type:"
<Regenaxer> So this assumes there is only a single space
<aw-> why is there a (line) at the end there?
<aw-> it's not used for anything
rob_w has joined #picolisp
<Regenaxer> To skip return and/or linefeed
<aw-> an empty content-type ?
<Regenaxer> all the clauses in this 'case' call (line) in some way
<Regenaxer> Whatever the content-type line contains, we must skip it to continue
<aw-> but it doesn't skip it
<aw-> it freezes
<aw-> if I make an request with an arbitrary content-type, ex: multipart/mixed, then the server just freezes
<Regenaxer> really? But other content types should appear often (?)
<Regenaxer> Hmm, perhaps (till "=" T) is not optimal
<Regenaxer> Should be (till "=^M^J" T) perhaps?
<aw-> no that's only for the multipart/form-data which has a boundary
<aw-> i'm trying to add a condition for supporting other content-types
<aw-> and it's incredibly painful
<Regenaxer> Still, (till "=" T) is perhaps incorrect?
<aw-> i think it's ok.. i don't think that's the issue
<Regenaxer> Where exactly does it freeze?
<Regenaxer> If the line does not contain a "=", it will continue to parse the input stream till an unpredictable location
<Regenaxer> I think I should change this
<aw-> ok wait
<aw-> i'll show you what i did
<aw-> ah nvrmind it doesn't work
<Regenaxer> I changed the two 'till' calls
<Regenaxer> I think it should be safe
<Regenaxer> or is there a situation where it might harm?
<aw-> what do you mean 'safe' ?
<Regenaxer> Nothing else breaks
<Regenaxer> I changed to (till "=^M^J" T)
dtornabene has joined #picolisp
<aw-> why?
<aw-> there's also the problem of sending data as content
<aw-> right now http.l only accepts key=value as data
<aw-> you can't send just a string as data.. that limits the types of requests it can handle
<Regenaxer> For example?
<Regenaxer> In pil applications all kinds of data are POSTed
<aw-> ex: curl -X POST -d 'a random string of data' http://localhost
<aw-> yes but those are POST'd via web form, right?
<aw-> those are all key=value
<aw-> but you should be able to just send a value
<Regenaxer> Why? What should happen with that value on the server?
<aw-> and then parse the data based on the content-type
<Regenaxer> I don't see any need for that
<aw-> ex, if someone specifies Content-Type: image/svg+xml ... my server should be able to parse the content directly
<aw-> yes there's lots of nede
<aw-> need*
<Regenaxer> Sorry telephone
<aw-> ok, i'll try to hack on this a bit more and figure out how to add my own content-types
<Regenaxer> Will take some hours here
<Nistur> mornin'
mtsd has joined #picolisp
<Regenaxer> Hi Nistur!
<Regenaxer> Hi mtsd!
<mtsd> Hi Regenaxer!
<mtsd> My picolisp t-shirt arrived yesterday :)
<mtsd> Very nice
<Nistur> mtsd: :)
<Nistur> I'm wearing mine today again :P
<mtsd> Good work, Nistur :)
<mtsd> It is going to be standard work uniform for me, I think
<aw-> picolisp t-shirt?
<Regenaxer> mtsd: Great! :)
<Nistur> 14 total tshirts sold so far :P
<mtsd> Uniforming the community ;)
orivej has joined #picolisp
<aw-> wow nice! what's the difference with Premium Tee?
<aw-> only price?
<tankf33der> does anyone want post to reddit.com/r/lisp Tshirt link ?
<aw-> why the hell does this website have 45 javascript assets?
<aw-> i guess most ecommerce websites are like this nowadays
<Nistur> aw-: yeah, all tshirt sites I tried had a million and one hooks into everything :( it sucked.
<aw-> good business, how many have you sold so far?
<aw-> oh 14...
<aw-> 16 now
<Nistur> I didn't order both myself, I just got a premium one which seems pretty nice
<Nistur> haha :)
<aw-> cool
<Nistur> teespring has unfortunately high shipping costs :( but the actual tshirt costs are considerably lower than other sites I tried (most were $20-25 per shirt)
<Nistur> I put the prices of the shirts pretty much at cost, so I'm not earning anything from this :P It was just because I wanted a picolisp tshirt, and it looked like other people might be interested in one too
<aw-> +1
mtsd has quit [Remote host closed the connection]
mtsd has joined #picolisp
alexshendi has joined #picolisp
mtsd has quit [Remote host closed the connection]
mtsd has joined #picolisp
mtsd has quit [Remote host closed the connection]
dtornabene has quit [Read error: Connection reset by peer]
dtornabene has joined #picolisp
dtornabene_ has joined #picolisp
dtornabene has quit [Ping timeout: 240 seconds]
m_mans has quit [Ping timeout: 240 seconds]
m_mans has joined #picolisp
<sriram_> Nistur> thats wonderful....I am looking into getting one too..many thanks for letting us know about this..
<sriram_> just wondering how the sizes are set up
<Nistur> sriram_: I am tempted to say something facetious, like "well, they start small, and get bigger until they're huge"... :P
<sriram_> :)
<Nistur> sriram_: I got an XL... it's probably one of the smaller XLs I've got, they usually go from 'baggy' to 'practically a dress' on me, which my wife always complains about :P But it's safer to buy a bit larger, because a L would have been tight and I don't like tight
<sriram_> i guess i will start ordering at the smallest size that will fit me and stop when it actually fits :)
<Nistur> anyway, I would say they probably run a little on the small side, but are relatively average... they're not noticably small
<sriram_> ah thats the info I am looking for...I dont like tight as well and was wondering
<sriram_> if XL would do
<sriram_> like you, I dont like it when a snug fit..so was debating L/XL/2XL
<aw-> i ordered two, because i hate having a "favourite t-shirt" that's hard to replace
<aw-> Nistur: thanks for setting it up
<aw-> unrelated, I finally switched my email client, from Thunderbird to Alpine (Pine from the good ol')
<aw-> days
<Nistur> aw-: no problem
* Nistur is using gnus pretty much entirely for his own emails these days
<aw-> never heard of it, emacs reader?
<Nistur> yup
aw- has quit [Quit: Leaving.]
dtornabene_ has quit [Quit: Leaving]
<tankf33der> nasty one again.
<Nistur> tankf33der: it's not showing up in r/lisp
<tankf33der> i can see it, try F5
<Nistur> I have it on 'new' obviously
<Nistur> and there's nothing newer than 15 hours ago
<tankf33der> front page
* Nistur blows raspberries
<Nistur> dunno
<Nistur> still not showing up for me
<Nistur> on two machines, let me check my phone
<Nistur> nope not there either
<Nistur> tankf33der: maybe it needs approval and is just showing up on yours because you're signed in?
* Nistur shrugs
<tankf33der> yea, dont show if loggedout
<Nistur> I am logged in on one box and my phone, and not logged in on another, so yeah, it must just be you that can see it :P
_whitelogger has joined #picolisp
rob_w has quit [Quit: Leaving]
grp has joined #picolisp
himmAllRight has joined #picolisp
alexshendi has quit [Ping timeout: 264 seconds]
_whitelogger has joined #picolisp
alexshendi has joined #picolisp
alexshendi has quit [Ping timeout: 255 seconds]
_whitelogger has joined #picolisp
reed_ has joined #picolisp
rick42 has joined #picolisp
alexshendi has joined #picolisp
grp has quit [Quit: box shutting down...]
reed_ has quit [Ping timeout: 260 seconds]