Blue_flame has quit [Write error: Connection reset by peer]
Blue_flame has joined #picolisp
orivej has quit [Ping timeout: 244 seconds]
orivej has joined #picolisp
orivej has quit [Ping timeout: 240 seconds]
orivej has joined #picolisp
orivej has quit [Ping timeout: 264 seconds]
orivej_ has joined #picolisp
orivej has joined #picolisp
orivej_ has quit [Ping timeout: 264 seconds]
alexshendi has joined #picolisp
orivej has quit [Ping timeout: 240 seconds]
orivej has joined #picolisp
orivej has quit [Ping timeout: 256 seconds]
rob_w has joined #picolisp
<Nistur>
mornin'
<Regenaxer>
Hi Nistur
<Nistur>
o7
<aw->
hi Nistur Regenaxer
<aw->
Regenaxer: i have a question regarding (out "file" (prinl "str"))
<aw->
how do I 'catch it in case there's a problem with "file" (ex: read-only)
<aw->
i tried (catch T (out ...)) but it doesn't seem to work
<Regenaxer>
Hi aw-
<Regenaxer>
To catch errors, you need to pass a list or strings to 'catch'
<Regenaxer>
s/or/of
<Regenaxer>
the strings are meant to be substring candidates of the error message
<aw->
oh ok.. like (catch '("Permission denied") ...) ?
<Regenaxer>
yeah
<aw->
there's no way to catch all?
<Regenaxer>
this is the normal error handler then
<Regenaxer>
you can catch *any* error with '(NIL)
<Regenaxer>
or '("")
<aw->
oh ok! that's what I wanted
<aw->
thanks
<Regenaxer>
:)
<aw->
oh one more Q
<aw->
(out "file1" (in "file2" (echo))) ... if file2 is 100MB, or 1GB, will it have a big performance impact?
<Regenaxer>
Not sure, as 'echo' does buffered transfers, iy is not optimal
<Regenaxer>
Perhaps (call "cp" ...) is faster
alexshendi has quit [Read error: Connection reset by peer]
<aw->
i see
<Regenaxer>
Perhaps you could do some benchmarks of typical cases. I never measured the performance
<aw->
yes i'm trying now with (bench)
<aw->
it's pretty slow
<Regenaxer>
I suspected so
<Regenaxer>
'echo' buffers, but operates character by character, to stay intermixable with other I/O functions (read, line, char, peek etc.)
<Regenaxer>
It is not really intended for bulk transfers
<aw->
i see, yes i want to avoid that for my use case
<Regenaxer>
Doesn't matter perhaps for TCP transfers
<aw->
i'm trying to copy contents from A -> B, while B is open for writes by another process
<aw->
so i can't use (call 'cp)
<Regenaxer>
Why not?
<Regenaxer>
The processes buffer independently anyway
<Regenaxer>
Perhaps you need a lock?
<Regenaxer>
'ctl' or 'acquire' / 'release'
<aw->
if the file is open and i delete it without closing the FD, will it continue receiving writes?
<Regenaxer>
yes
<Regenaxer>
It vanishes not before all links to it are gone
<Regenaxer>
hardlinks
<aw->
and what happens to that data once i close the FD?
<Regenaxer>
It is gone
<aw->
exactly
<Regenaxer>
unless some link exists
<aw->
that's what i'm trying to avoid
<aw->
hmmm right
<aw->
hardlinks.. maybe i can use that
<tankf33der>
aw-: try mmap and use c funtion via native to read-write in memory regions.
<Regenaxer>
The processes are in the same family?
<aw->
Regenaxer: yes they are
<Regenaxer>
So you could also 'tell'
<Regenaxer>
I do something like that in log files
<Regenaxer>
stdout and stderr are redirected
<aw->
can you show me the code?
<Regenaxer>
then at night a 'cat' is done, and (rewind)
<Regenaxer>
so the appending starts new
<aw->
yes I do (rewind) already, and also using (ctl) to lock it
<Regenaxer>
ok
<aw->
i tried (call 'cat ">" ...) doesnt work
<aw->
(call 'cat "file1" ">" "file2") ..
<Regenaxer>
No, ">" is a shell operator
<aw->
yes i know :(
<aw->
same as | doesn't work
<aw->
how do you use 'cat from picolisp?
<aw->
cat file1 -o file2; # would be nice ;)
<aw->
there's no -o
<Regenaxer>
cat is not useful
<Regenaxer>
moment
<Regenaxer>
I try to find an example
<aw->
(call 'script.sh) would work too, but that's ugly
<Regenaxer>
The cronjobs just do
<Regenaxer>
(when (info Log)
<Regenaxer>
(out (pack "bak/" Log)
<Regenaxer>
(in Log (echo)) )
<Regenaxer>
<Regenaxer>
then lib/clrlog.l is loaded by the parent
<Regenaxer>
it does only
<Regenaxer>
(out NIL (rewind))
<Regenaxer>
the parent is started with >log/xxx 2>&1
<aw->
same as me.. well i guess you never tried with 100MB log file ;)
<Regenaxer>
right, they are not such big :)
<Regenaxer>
so (call "cp" should be fine
<Regenaxer>
So 'cat' is not useful, it is just a cp to stdout
<Regenaxer>
Otherwise, call 'open' and then native "write" with a large buffer
<beneroth>
hi aw-, Regenaxer :)
<beneroth>
aw-, the important bit to understand about (catch) is that the two way it can be used are pretty independent of each other, and must not be mixed
<Regenaxer>
Hi beneroth
<Regenaxer>
beneroth, right, there is not a single 'catch' for both
<beneroth>
either catch errors by using a 'str as an argument, or list of 'str or list with empty string as catch-all'("")
<Regenaxer>
Would need (catch T (catch '("") ..
<beneroth>
aw-, or catch symbols thrown with (throw) - this is more like a a goto to handle exits out of nestings or such, not really meant for error management
<beneroth>
instead of wrapping things into (catch 'str|'(str ...) ...) you can also set the *Err variable
<beneroth>
depends on the situation what is more appropriate
<beneroth>
Regenaxer, the loss of the error cause 'any argument is still something I would like to have improved eventually :D
<Regenaxer>
yeah
<Regenaxer>
hmm, where was it lost? In the *Err handler?
<beneroth>
aw-, what I also sometimes do is (let ("quit" quit quit '((Msg Err) (quit (pack "error context prefix " Msg) Err))) ... it's kinda a "re-throw" of an (quit) with decorating the error message.. this way the strings in multiple quit calls can be shorter, and still be prefixed with a common identifier
<beneroth>
Regenaxer, (catch)
<beneroth>
Regenaxer, the message (first 'any argument to 'quit) gets returned, and is also available in *Msg
<Regenaxer>
yes, but how to pass it?
<beneroth>
but the second argument.. why not just also put it into a global?
<beneroth>
or make *Msg value a cons pair or list of those 2 values
<Regenaxer>
Not good perhaps, as this can be anything
<Regenaxer>
So never gc'd
<beneroth>
it can also be anything within a cons
<beneroth>
hmm
<Regenaxer>
same
<Regenaxer>
imagine it is a DB object
<beneroth>
I see. but why doesn't that argument apply to the 'str ending up in *Msg in normal usage?
<Regenaxer>
perhaps the root object
<beneroth>
ok
<Regenaxer>
A transient symbol does no harm
<beneroth>
good then ... make the return value of (catch) a cons pair... no binding if it doesn't get handled :)
<beneroth>
I see your point
<Regenaxer>
Catch has no return value in case of error
<beneroth>
good one to think of
<Regenaxer>
it never has its own
<Regenaxer>
it returns what was thrown
<beneroth>
Catch returns the catched message in case of error, else it returns the result of the 'prg
<beneroth>
or so I understood it.. wrong?
<beneroth>
hm
<Regenaxer>
No, it does not return
<beneroth>
ah I see
<Regenaxer>
the error handler unwinds the stack
<beneroth>
so I should just use the first argument of (quit) more wildly? :P
<beneroth>
hm
<beneroth>
no also not working
<beneroth>
ok I see
<Regenaxer>
If you call quit, it is under your control
<Regenaxer>
store it in a global
<beneroth>
what I want,... I should do explicitly at the point of the 'quit call, because it depends heavily on the situation
<beneroth>
yeah
<beneroth>
T
<Regenaxer>
makes sense
<beneroth>
I haven't though of unintended bindings lingering around because of that
<beneroth>
good point
<Regenaxer>
I'm always paranoid because of DB objects
<beneroth>
yeah I can fully understand.. could mess up quite a lot :)
<Regenaxer>
but possibly other things like a huge list
<Regenaxer>
yeah, and hard to detect
<beneroth>
btw. there is no way to determine if a symbol is interned at all (does exist) without creating it except for iterating over (all), right?
alexshendi has joined #picolisp
<Regenaxer>
not only (all), but possibly other namespaces too
<beneroth>
(namespaces don't matter for this question, I'm just argueing about the current environment)
<beneroth>
hehe
<Regenaxer>
ah, ok
<Regenaxer>
So, yes
<Regenaxer>
eg 'print' always checks that way
<Regenaxer>
also 'pr' (but not 'rd')
<beneroth>
kinda: does 'sym exist, if yes use it as a function, else do something else... I practically do (intern (pack "construct-symbol-somehow)) and do a 'fun? or 'isa on the result
<Regenaxer>
I think this is not a good idea
<beneroth>
hehe
<Regenaxer>
The sym might exist unexpectedly
<Regenaxer>
eg you typed it in the repl
<beneroth>
my use case: check if a specific class exists, if yes use that one for instancing, else use a more general one
<beneroth>
T
<beneroth>
possible but very unlikely (custom naming conventions in the way)
<Regenaxer>
for a class it should be easy
<Regenaxer>
ie if it is not empty (val or prop)
<beneroth>
yes, the point is, during the check I construct a interned symbol
<beneroth>
which then lingers around forever, right?
<Regenaxer>
right
<Regenaxer>
you can 'zap' it later
<beneroth>
there is no 'gc for interned names, right? not unless the whole interned tree "goes out of scope"... either by being in a namespace where the namespace symbol got its value changed, or by the process being terminated
<beneroth>
right?
<beneroth>
T
<Regenaxer>
yes, but 'zap' does not care about the whole interned tree
<beneroth>
so if my check fail, and no class definition exist, I should 'zap the symbol if I'm sure that it only exists as a side effect of my check and I'm sure that it cannot exist for some other reason, correct?
<Regenaxer>
yes, if it is empty it should be ok
<beneroth>
ok thanks :)
<Regenaxer>
Here a small namespace might be easier
<beneroth>
good point
<Regenaxer>
then you are sure
<beneroth>
I'm sure anyway because of a name prefix, but yeah
<Regenaxer>
and 'export' if all ok
<beneroth>
hm
<Regenaxer>
ok, I see
<beneroth>
nice idea
<beneroth>
I'm amazed that we actually understand each other over this few chat lines.. our discussions get more involved every time :)
<Regenaxer>
yeah, indeed :)
<Regenaxer>
'export' is not right perhaps
<Regenaxer>
just 'intern'
<Regenaxer>
So what 'import' does
<beneroth>
T
<Regenaxer>
interesting anyway :)
<aw->
hey beneroth
<aw->
thanks!
<aw->
although i must admit: (let ("quit" quit quit '((Msg Err) (quit (pack "error context prefix " Msg)
<aw->
it's pretty crazy
<aw->
not my kind of pil code ;)
<Regenaxer>
What if "normal" catch/throw is used for such self-generated errors?
<Regenaxer>
I think I never felt the need for catching 'quit's
<beneroth>
Regenaxer, I would say (throw) is for errors you can recover from, and (quit) for errors which really should never happen during regular operation
<Regenaxer>
yes, so 'quit' will not be caught
<beneroth>
though I put bad/malicious input (not complying with the rfc for example) in the second bucket
<Regenaxer>
T
<Regenaxer>
It ends up in *Err
<beneroth>
yeah just a catch all, to stop current operation and go back to normal main loop
<Regenaxer>
I see
<Regenaxer>
I think better (bye) then
<Regenaxer>
it may be in an inconsistent state
<beneroth>
well my use case is processing of a web request, so than I would like to deliver a proper HTTP 500 error if this is still possible (no response started sending yet)
<Regenaxer>
Typically a child process anyway
<Regenaxer>
ok, this can be done in *Err too
<beneroth>
T
<beneroth>
I only have catches in the few places where I expect such things to happen (invalid http stream). if a more severe error is issues (e.g. out of memory, files broken, dunno) then I find the server should crash violently
<Regenaxer>
yep
<beneroth>
better crash early and entirely than hidding severe faults or making them tolerable enough so the issue never gets fixed
<Regenaxer>
As I'm going more extreme with 'native' in pil21, it gets noisy to have lots of (native "@" "foo" ...) calls. What if we also had a short form?
<Regenaxer>
eg
<Regenaxer>
(%@ "foo" ...)
<Regenaxer>
(%@ "socket" 'I AF_INET6 Type 0)
<Regenaxer>
instead of (native "@" "socket" 'I AF_INET6 Type 0)
<rob_w>
cheers all , hope ya all doing very well !
<Regenaxer>
Cheers rob_w :)
<rob_w>
how is the weather over there ?
<Regenaxer>
oh, are you far away?
<rob_w>
no .. just chatting , i could actually see if there is a cloud over your place ;-))
<rob_w>
i am just returing from a 2weeks off , so its coding time again
<Regenaxer>
:)
<Regenaxer>
I see no cloud over your place, 3300 m to the east
<rob_w>
pure east ? thought be more south to you .. hang on
<Regenaxer>
yes, a little south too
<Regenaxer>
You are right
<Regenaxer>
much more south than east
<rob_w>
ok good
<Regenaxer>
OK, implemented (%@ "foo" 'I Arg)
<Regenaxer>
I think I like it
<Regenaxer>
What do other 'native' experts here say?
<Regenaxer>
I find the full native call too verbose
<aw->
Regenaxer: i like it
<Regenaxer>
Great!
<aw->
i would like native to be simpler though
<Regenaxer>
I released it
<aw->
it was quite painful to wrap my head around it the first time
<Regenaxer>
yeah
<Regenaxer>
I had to look up the refs too
<Regenaxer>
Now I'm more into it, as I ported it to pil21, extended it, and used it more extensively in the networking stuff
<aw->
the biggest pain point is specifying the "result specification", it's so difficult to get that right
<Regenaxer>
agreed
<Regenaxer>
depends though
<Regenaxer>
structs are the hard part
<Regenaxer>
I extended the result specs a little
<Regenaxer>
there is now also 'W' and 'P'
<aw->
yes structs
<Regenaxer>
'W' is signed 16 bits
<Regenaxer>
'P' is pointer (unsigned 64 bits)
<aw->
ohhh
<aw->
P might be useful
<Regenaxer>
yeah
<aw->
that means we dont need to figure out the struct anymore?
<Regenaxer>
more often used than 'N'
<Regenaxer>
The struct spec is the same
<Regenaxer>
in the result spec
<Regenaxer>
the 'struct' *function* is a little more useful
<aw->
sorry, i meant the pointer
<Regenaxer>
eg (struct P 'I)
<Regenaxer>
returns an int at that place
<aw->
hmmm
<Regenaxer>
you needed (car (struct P '(I))) before
<Regenaxer>
same for 'B', 'N' etc
<Regenaxer>
also 'W' and 'C'
<aw->
so C code like: int *fun(void); .. could we do (%@ "foo" 'P) ?
<Regenaxer>
though (struct P 'I) is the same as (byte P)
<Regenaxer>
yes
<Regenaxer>
this stayed the same
<aw->
ok
<Regenaxer>
except for %@
<aw->
awesome
<aw->
will (native) still exist? or do you plan to remove it?
<Regenaxer>
No, it is still needed for other libraries
<aw->
older code, ok
<Regenaxer>
it replaces only (native "@" ...
<Regenaxer>
not only older
<Regenaxer>
(native "libcrypt.so" ...
<Regenaxer>
but "@" is so often needed
<aw->
oh right sorry
<Regenaxer>
you can check lib/net.l in pil21
<Regenaxer>
not complete yet
<Regenaxer>
We (tankfeeder and I) also found a portable way for constants
<aw->
Regenaxer: unrelated: what can you think of which may cause (in Fd (when (rd) ...) to just freeze forever waiting for data even though "technically" it was sent
<Regenaxer>
Is it a socket?
<aw->
yes
<Regenaxer>
Usually best to wrap it into a 'task'
<Regenaxer>
Are you sure it was sent?
<tankf33der>
60 mins
<Regenaxer>
tankf33der: ?
<Regenaxer>
ah, pulls?
<tankf33der>
yea
<Regenaxer>
ok, enough
<aw->
yes sure
<aw->
oops
<aw->
it's in deadlock
<Regenaxer>
'ctl'?
<aw->
client and server are both reading on the TCP socket
<Regenaxer>
Probably it read *some* data but expects more
<aw->
hmmm
<aw->
if using (when (rd) ..) does it stop at newline?
orivej has joined #picolisp
<Regenaxer>
no, there are no newlines
<beneroth>
aw-, use (poll) to check if there is something to read
<Regenaxer>
it stops if the expr is not complete but the read buffer is empty
<beneroth>
(rd) is reading directly the binary stream
<Regenaxer>
yes
<Regenaxer>
but poll will not help if the expr is only partially read
<Regenaxer>
ie already started reading
<Regenaxer>
For pipes this does not happen if data is shorter than PIPE_BUF
<Regenaxer>
normally 4096 bytes
<Regenaxer>
but no guarantee for TCP
<beneroth>
while (char) and (peek) work together with a hidden buffer variable, which could end up in a situation where (peek) returns something (there is something in the buffer, 1 character (which might be bigger than 1 byte - UTF-8) but (char) hangs, because (char) does not only read one character but also shoveling another one into the 'peek-buffer - and if there is no other one, (char) hangs, even when the return value of (char) is already known/defined
<beneroth>
I just discovered recently :-)
<beneroth>
changed my one-byte-character reading into (char (rd 1)) and voilà :)
<Regenaxer>
ok
<Regenaxer>
good if you read only char and never (read)
<beneroth>
Regenaxer, yeah, so any TCP reading with a instabile/untrusted client should be wrapped into an (abort) timeout, to abort on a hanging TCP connection, yes?
<Regenaxer>
the one-char look ahead is needed for (read)
<beneroth>
I see
<beneroth>
so (read) is kinda based on (char) and (peek) - makes sense
<Regenaxer>
yes, abort may be needed
<Regenaxer>
right
<beneroth>
good
<beneroth>
thanks for confirmation
<Regenaxer>
always needs the next char to detect delimiters etc
<aw->
ok im using abort
<aw->
it fixes my problem but it seems like a hack
<beneroth>
yeah I read now practically bytewise, so I can validate the expected input on every byte and abort if the input misbehaves
<Regenaxer>
or use a separate process which may wait
<Regenaxer>
and then sends via pipe
<beneroth>
only possible issue than is either a bug in my validation logic or a too-long hanging TCP connection
<beneroth>
aw-, might be hack in your use case, but don't forget a connection might go stale anytime for complete legitimate reasons
<beneroth>
it should happen and usually doesn't, but it can
<beneroth>
network is inherently unreliable, so you might made to wait forever
<beneroth>
the measures you want to deploy against that depends of course of the size of the risk.. if you control both ends of a connection, than it's likely (you rule out malicious TCP behaviour from the other end)
<beneroth>
if the network connection is within a single datacenter, then it's probably more reliable than a connection through the internet
<Regenaxer>
A separate child could do forever (while (rd) (tell SomePid 'doSomething (lit @]
<beneroth>
T
<beneroth>
and if you accept more connection in the meantime, you can be DoS'ed (on purpose or accident) by running out of children/file descriptors
<Regenaxer>
right
<beneroth>
question is, if this thread/risk is in any way relevant
orivej has quit [Ping timeout: 264 seconds]
<beneroth>
how likely is it? how bad would the impact be? how long would it take you to detect it? how long would it take you to go back into a healthy state after detection?
orivej has joined #picolisp
<Regenaxer>
yeah
<beneroth>
in all consequence the answers to these questions depends on how the resulting system is used
<Regenaxer>
I thought you never can really protect from DOS attacks
<beneroth>
therefore this cannot be solved once for all in a library, it has always to be dealt with on application/whole system level somewhat.. maybe indirectly by having certain guarantees in a library, but I don't think there is one size fits all
<beneroth>
Regenaxer, you can.. it depends what the bottleneck is of the Denial of Service
<beneroth>
DoS is essentially running out of a resource
<beneroth>
some resource consumption can be monitored
<beneroth>
and guarded
<beneroth>
guarding costs resources (performance?), so too much guarding can have the same effect as an actual DOS
<Regenaxer>
yes, number of accepts
<beneroth>
T
<aw->
if i do (while (rd) .. ) how does it know when to stop reading data?
<Regenaxer>
but if a botnet attacks the infrastructure ...
<Regenaxer>
It never stops
<Regenaxer>
it read 1 expr at a time and sends it
<beneroth>
but you have to be cautious to not build a guard system which gets triggered by to simple symptoms.. then maybe those symptoms might be triggered by accident or maliciously, causing a guard reaction even when the resource is not scarce at all
<Regenaxer>
runs forever until the connection is closed
<Regenaxer>
(while (rd/ ...
<beneroth>
Regenaxer, the real big problem you cannot defend is a DDOS (likely from millions of IoT devices) which overload your ISP's internet connection
<Regenaxer>
T
<beneroth>
so your resources get never DOSes.. but the street is blocked to reach you
<beneroth>
on that level it's a numbers game.. who has more resources. and defense takes more resources as offense.
<beneroth>
but the probability for that is not so large... more likely a buggy client hanging your server somehow
<Regenaxer>
aw, could be modelled after app/main.l
<Regenaxer>
'go'
<beneroth>
Regenaxer, (while (poll Socket) (rd)) would not work?
<Regenaxer>
it might stop too early
<Regenaxer>
no data *yet*
<beneroth>
ah T
<Regenaxer>
(task (port @) (unless (fork) ...
<beneroth>
T
<Regenaxer>
then exit
<beneroth>
aw-, the proper way would be (task).. which is basically async I/O
<aw->
beneroth: ?
<beneroth>
well (task) has two modes: async I/O and/or timers. Correct, Regenaxer ?
<Regenaxer>
beneroth, but the task may block too
<aw->
i dont see how task changes anything
<Regenaxer>
yes
<Regenaxer>
yeah, best is a fork
<aw->
i'm doing now (abort 5 (in *Sock (when (rd) ...)))
<Regenaxer>
see 'go' in app/
<Regenaxer>
insted of (pr ... which answers to the client
<beneroth>
aw-, it removes the too early (nothing to read YET) issue.. as the task is only sleeping until something arrives, not actively polling.
<Regenaxer>
do a 'tell'
<aw->
but the (abort) kills the TCP connection and i lose the data that should have been sent
<Regenaxer>
this will never block in the other processes
<beneroth>
aw-, it doesn't remove the "the other never sends or stopped mid sentence" issue.. that one you cannot solve, only guard with timeout, so (abort)
orivej has quit [Ping timeout: 256 seconds]
orivej_ has joined #picolisp
<Regenaxer>
yes, an abort might be good in the forked child too
<aw->
beneroth: this is happening on localhost between 2 processes on the same machine
<beneroth>
aw-, so you have data you wanna send. and the other doesn't answer. and your only copy of the data is in the pipe buffer?
<Regenaxer>
the point is that it 'tell's either all or nothing
<aw->
also it's in a forked child
<Regenaxer>
if the packets are not too big
<aw->
and the packets are not big
<aw->
just a number between 1 and 100
<Regenaxer>
perfect
<beneroth>
what is the issue/symptoms?
<aw->
beneroth: issue is deadlock between client/server
<beneroth>
so most likely a reading error on one end
<aw->
both are blocked on (rd)
<beneroth>
why do both want to read?
<aw->
client sends to server, server responds to client
<aw->
so, both read and both send, just the order is reversed
<beneroth>
both should be (task), so they don't block, they only got waked up when there is some input for them to process
<aw->
like ping pong
<Regenaxer>
should not lock
<aw->
(task) is just an abstraction though, it doesnt solve anything
<Regenaxer>
(pr (rd))
<beneroth>
it does, it hands the issue to the kernel
<beneroth>
no?
<beneroth>
I might be wrong here
<Regenaxer>
The strange thing here is that *both* read
<beneroth>
problem is: your process needs to know when to read
<Regenaxer>
seems a logical error
<beneroth>
aye
<beneroth>
aw-, both ends of the connection may (at any time) initiate a exchange?
<beneroth>
that is way you have both listening?
<aw->
hmmm
<aw->
Regenaxer: might be a logical error
<beneroth>
otherwise only one should have the ability to initiate an exchange, so that one is not listening until it sent something
<aw->
client does (pr) and then (rd).. server does (rd) and then (pr)
<beneroth>
client - server or peer to peer?
<beneroth>
ok
<aw->
but for some reason, SOMETIMES, both are stuck on (rd) at the same time
<beneroth>
(flush) ?
<Regenaxer>
then a task will be needed
<aw->
this is the definition of deadlock
<beneroth>
T
<beneroth>
why is B not receiving anything when the A is finished with writting ?
<aw->
the client wouldn't go to (rd) if it didn't (pr), it's not possible
<beneroth>
because it is buffered and A-buffer waits for more?
<beneroth>
yeah, so the (pr) is not received, for some reason
<aw->
so for some reason, the data is not going through the socket
<Regenaxer>
yes, but without task it prints
<beneroth>
T
<Regenaxer>
I think you need a task on both sides
<beneroth>
Regenaxer, no idea what your context is
<Regenaxer>
both may init a query on the other side, no?
<Regenaxer>
at any moment?
<beneroth>
no
<beneroth>
client server
<Regenaxer>
ok
<beneroth>
client initiates
<aw->
Regenaxer can you confirm: on a TCP socket, if the server is not listening with (rd) and the client sends with (pr), the data will BUFFER?
<beneroth>
else you're right :)
<aw->
or does the data get lost?
<beneroth>
aw-, I suspect your (pr) is buffering on the sender site
<Regenaxer>
yes, buffers up to some amount
<beneroth>
T
<aw->
right.. so even without a listener, when the server does (rd) it technically _should_ see the data that was already sent
<Regenaxer>
So the server should do a task on the socket
<aw->
what would (task) solve?
<Regenaxer>
when something arrives, it wakes up
<beneroth>
aw-, you could switch from (rd) to (rd 1).. reading bytes instead of PLIO, then you know if nothing arrives (sender is buffer and hasn't actually sent anything), or (rd) blocks because the end of the exchange is not sent by the sender correctly, so it expects more data when there is none
<Regenaxer>
the server can do other things meanwhile
<Regenaxer>
a GUI
<beneroth>
task solves not knowing when you have to listen
<Regenaxer>
if there are no other duties, the server can stay in (rd) forever
<Regenaxer>
and prints if something arrives
<beneroth>
doesn't apply in aw-'s problem here as far as I understand
<aw->
i have a feeling this will not solve the problem
<aw->
it's just adding a layer of abstraction over something that's already broken
<Regenaxer>
yes, task is not needed
<Regenaxer>
if there is nothing else to di
<Regenaxer>
do
<Regenaxer>
thats seldom
<beneroth>
Regenaxer, (pr) is buffering, so (flush) or leaving (out ..) would be needed?
<Regenaxer>
usually you sync with other stuff
<Regenaxer>
commits from other siblings
<beneroth>
Regenaxer, you bring more problems into this.
<aw->
wait i have more info
<aw->
because i have logs, one sec
<Regenaxer>
yes, pr buffers
<beneroth>
Regenaxer, either aw- was wrong and the hanging is not happening on (rd), or it has nothing to do with your point
<beneroth>
Regenaxer, good
<Regenaxer>
yeah
<Regenaxer>
as both (rd)
<beneroth>
aw-, put a (flush) after your (pr) and try again :)
<beneroth>
if that doesn't solve it, change (for debugging purposes only, no useful data will come out of it) the (rd) to a (rd 1) call...reading 1 byte instead of a plio expression. test if (rd 1) is returning or blocking.
<beneroth>
to determine if it is a network/pipe problem or if it is a protocol problem
<Regenaxer>
flush is not needed if you do (out Sock (pr @))
<beneroth>
we don't know if he does that or not
<Regenaxer>
I think it is not a network issue
<beneroth>
but yes, leaving (out) automatically calls (flush), important to note here
<Regenaxer>
Why do *both* read?
<beneroth>
well his messages are very small, if its just small integers
<beneroth>
it might be (out Sock (pr) (in Sock (rd))) :P
<Regenaxer>
ok, if data are lost, both rd
<Regenaxer>
yes
<aw->
ok i just checked
orivej_ has quit [Ping timeout: 246 seconds]
<beneroth>
I'm sure the problem is solvable and just a little mistake, not a big weird magic issue
<aw->
the client (pr) was received and processed by the server, and then the client did (rd) and aborted after 5s because the server never sent back its (pr)..
<aw->
so it seems they are not both stuck in (rd)
<beneroth>
ok
<Regenaxer>
did you (out Sock ... in the server?
<Regenaxer>
after (in Sock (rd)) ?
<aw->
yes of course
<beneroth>
and when is the (out..) expression left? immediately after (pr) ?
<beneroth>
if not, then put a (flush) after (pr)
<Regenaxer>
~
<Regenaxer>
T
<beneroth>
s/left/exited
<aw->
ok wait i'll try
<Regenaxer>
In any case, such code works:
<Regenaxer>
(task (port @) # Set up query server in t
<Regenaxer>
(let? Sock (accept @)
<Regenaxer>
(unless (fork) # Child process
<Regenaxer>
(sync)
<Regenaxer>
(while (rd)
<Regenaxer>
(in Sock
<Regenaxer>
(tell)
<Regenaxer>
(out Sock
<Regenaxer>
(pr (eval @)) ) ) )
<Regenaxer>
(bye) )
<Regenaxer>
(close Sock) ) )
<Regenaxer>
(forked) )
<Regenaxer>
omit the task wrapper here
<Regenaxer>
if it is the only duty
<beneroth>
:)
<Regenaxer>
Point is (out Sock (pr ...
<Regenaxer>
for proper flushing as beneroth explained
<aw->
sorry, i fixed it, stupid error
<beneroth>
and that (out Sock) is exited
* aw-
facepalms
<beneroth>
aw-, it's always so, no worries
<Regenaxer>
oh :)
<Regenaxer>
yes
<beneroth>
happens to all of us
<beneroth>
Regenaxer, well in usual stacks it's quite more common that something is not working because of the stack
<aw->
on the server i was doing (in *Sock (while (rd) ...))
<beneroth>
?
<beneroth>
so (rd) blocks until something arrives.. and then?
* beneroth
cannot see the cause of the issue yet
<aw->
wait let me double-check
<aw->
sorry
<aw->
not fixed
<Regenaxer>
:(
<beneroth>
aw-, your log said the issue was with the sending/receiving of the server response
<beneroth>
so.. how does the (out *Socket) on the server look like?
<beneroth>
maybe you can compare the server response sending code (out *Socket ...) with the equivalent in the client, as the client sending works.
<Regenaxer>
beneroth, today I tried something I never did before:
<Regenaxer>
(+Swap +List +Ref +Link)
<Regenaxer>
I was not sur if it works
<Regenaxer>
sure
<beneroth>
and?
<Regenaxer>
But the list may be quite long
<beneroth>
I expect it to work
<Regenaxer>
*Seems* to work, yes
<Regenaxer>
No stress tests yet
<Regenaxer>
or in GUI etc.
<beneroth>
well the +Swap doesn't care about its content. and the +Ref +Link doesn't care about its symbol
<beneroth>
so should work :)
<beneroth>
but yeah, nice point
<Regenaxer>
:)
<beneroth>
I usually create a separate Entity for such situations
<Regenaxer>
'(+Swap/R +Chart)
<Regenaxer>
yes, but it makes search dialogs more involved
<beneroth>
maybe my relational db background.. why do you keep the List instead?
<beneroth>
ah okay
<Regenaxer>
The additional difficulty here is that the objects are in another DB
<Regenaxer>
in Pips
<beneroth>
I'm less sure about (+Swap +List +Joint) - what you think? :)
<beneroth>
haha
<beneroth>
I see
<Regenaxer>
yeah, I +Joint is even more involved
<beneroth>
the relation class prefix system is very nice
<Regenaxer>
yeah
<beneroth>
I just recently started to implement a prefix class on my own
<beneroth>
very simple, very powerful
<Regenaxer>
I cannot use +Joint here anyway, as the remote DB is not writable
<Regenaxer>
in general
<Regenaxer>
may be on another continent ;)
<Regenaxer>
and offline
<beneroth>
there is a bit of undeclared context one has to keep in mind in that stuff (maybe your Forth background?), but one I wrapped my head around it comes very natural
<Regenaxer>
yes, no compile time checks
<beneroth>
yeah
alexshendi has quit [Ping timeout: 272 seconds]
<beneroth>
well not needed, if you do it right
<beneroth>
and if you do wrong, than most likely it is wrong enough that you will notice that something is off :)
<Regenaxer>
yes, if you test it at least once ;)
<beneroth>
debugging might be a bit of a pain, but (edit) helps :)
<Regenaxer>
T
<beneroth>
what I've a bit more problem with, is the fact that the whole daemon objects are.. well one way
<beneroth>
things points to things, but you don't know what points to you
<beneroth>
which I need for my additional meta-schema-metadata on my addition db layer
<Regenaxer>
in some cases you do
<beneroth>
T
<Regenaxer>
'cls' attribute
<Regenaxer>
and in +Joint
<beneroth>
so for the moment, some of that stuff is both in DB and in daemon objects. as the daemon objects follow from the metadata stored in the db (or well the class definitions, I still generate the code not daemon objects directly)
<beneroth>
maybe I can merge this two representations of the schema more, later on
<beneroth>
but I want to keep full pilDB compatibility, it's just a too nice feature - and I have that so far.
<Regenaxer>
cool
<beneroth>
just the amount of classes is a bit bigger
<Regenaxer>
Should be no problem
<beneroth>
and I think I grokked ht library
<Regenaxer>
classes are very small
<beneroth>
T
<beneroth>
and RAM is cheap :)
<beneroth>
these days
<Regenaxer>
T
<Regenaxer>
relatively
<beneroth>
yeah.. as everything
<Regenaxer>
Some programs easily use it up
<beneroth>
well not many outside of simulations, machine learning, games
<beneroth>
well okay, you can bloat your hardware with running many virtual OS on them... docker :)
<Regenaxer>
Or just caching a big DB
<beneroth>
wasting of resources always gets as wasteful as possible
<beneroth>
T
<Regenaxer>
but caching is not wasting :)
<beneroth>
but in many cases DB only consists of RAM these days.. that was unthinkable a few years ago...
<beneroth>
T
<beneroth>
caching is not wasting
<beneroth>
having 5 copies when 4 would be sufficient is wasting :)
<Regenaxer>
yep
<beneroth>
but look at snap package format.. and Golang binaries.. all pack all libraries.. easier to deploy, no more unix spirit
<beneroth>
or well docker.. you don't update a library, you replace the whole OS image :)
<Regenaxer>
sadly, yes, no more unix spirit
<beneroth>
logic saved (having not to figure out the differences and dependencies) bought by space (and network traffic, probably)
<beneroth>
so I'm not even sure that this is illegitimate, if the logic to figure out the relationship of things really is legitimately so complex.
<beneroth>
though I suspect complexity never should reach those depths in the first place :)
<Regenaxer>
Especially as packe systems take care of dependencies rather well
<Regenaxer>
package
<beneroth>
well only insomuch the package maintainers correctly maintain the metadata
<beneroth>
I suspect the main driver for snap et al is that software developers are too lazy to learn to pack packages
<beneroth>
which has it's point, as the tooling is pretty bad for most package managers afaik
<Regenaxer>
true
<beneroth>
but the people who went into it don't care to make it more approachable, as they grokked it
<beneroth>
which is basically the lisp curse, in a way, I think
<beneroth>
the same sociological underlying mechanic, I mean
<Regenaxer>
probably
<Regenaxer>
as in Pil too
<beneroth>
80% solution works for me, so nobody ever creates a 100% solution
<Regenaxer>
hard to grok
<beneroth>
instead we have dozens of 80% solutions
<beneroth>
because it's easy to make a solution
<beneroth>
when its harder, there is more incentive to complete a 80% solution instead of creating a new one from stack
<beneroth>
Regenaxer, I'm currently have to code some C#. But I think in PicoLisp... I think (default Var ...) and write if (Var == null) Var = ...
<beneroth>
*g*
<beneroth>
PicoLisp is so much nicer
<Regenaxer>
hehe, I know such situations
<Regenaxer>
'listen' now in pil21
<Regenaxer>
4 lines vs. 32 lines in pil64 ;)
<Regenaxer>
no
<Regenaxer>
3 lines vs. 32 lines in pil64 ;)
<Regenaxer>
or 4 lines vs. 33 lines
<Regenaxer>
well, anyway ;)
<beneroth>
not bad
<Regenaxer>
In C in pil32 it was 15
<beneroth>
how many bytes in the compiled version?
<beneroth>
not more in pil21 than in pil64 ?
<Regenaxer>
it is no longer compiled
<Regenaxer>
networking is in Lisp now
<beneroth>
oh
<Regenaxer>
@lib/net.l instead of src64/net.l
<beneroth>
does this also means more fine-grade access to sockets without native ?
<beneroth>
I mean.. configuration of sockets, if that is the right word
<Regenaxer>
At the moment the same finegrade
<beneroth>
ok
<Regenaxer>
but can be more easily customized now
<beneroth>
so not very fine but good enough for usual use cases
<beneroth>
:)
<Regenaxer>
yeah
<beneroth>
and extremely simple
<Regenaxer>
yes
<Regenaxer>
and mostly native calls
<Regenaxer>
as before
<beneroth>
good
<beneroth>
:)
<tankf33der>
Regenaxer: what about connection functin ?
<Regenaxer>
Not yet
<tankf33der>
ok
<Regenaxer>
ATM building 'host'
<Regenaxer>
I tested list with a 'connect' from a pil64 process
<Regenaxer>
listen
<Regenaxer>
'host' is done
<aw->
Regenaxer: i have one request for pil21
<aw->
not sure if you can/want to do it
<Regenaxer>
ok?
<aw->
UNIX domain sockets
<Regenaxer>
hmm
<aw->
do you think you can implement that "easily" ?
<Regenaxer>
As I said above, networking can now be easily customized
<Regenaxer>
I don't know, never looked at domain sockets
<Regenaxer>
What is the advantage over named pipes?
<Regenaxer>
or TCP sockets?
<aw->
over named pipes, they work both ways (bi-directional)
<aw->
and over TCP sockets, they dont have all the TCP connection overhead
<Regenaxer>
ok, but two pipes are fine too
<Regenaxer>
ok, less overhead is good
<Regenaxer>
I don't know how much additional infrastructure is needed
<Regenaxer>
Are they normal file descriptors?
<Regenaxer>
ie (in .. (out .. work?
<aw->
depends
<aw->
you can make them as sockets
<aw->
so you work with it using similar to TCP
<aw->
(listen) and (send)
<aw->
or (receive) or whatever it's called
<Regenaxer>
I see
orivej has joined #picolisp
<aw->
i don't know if it's complex to add to pil21, but in C it's fairly easy to use
<Regenaxer>
Then it is the same
<Regenaxer>
Along the line of @lib/net.l
<Regenaxer>
You could create an additional lib
<aw->
right!
<aw->
oh that's in picolisp
<aw->
cool
<aw->
interesting..
<Regenaxer>
yeah
<Regenaxer>
easier to customize and extend
<Regenaxer>
or modify
<aw->
yes very
<aw->
wow this is awesome
<Regenaxer>
even at runtime as ever
<aw->
much simpler than pilAsm
<aw->
maybe I can attempt to implement it
<Regenaxer>
cool
<aw->
i didn't reach much about pil21
<aw->
does that code get compiled into the picolisp binary?
<Regenaxer>
No, it is normal interpreted code
<aw->
s/reach/read/
<Regenaxer>
mostly native calls anyway
<aw->
hmmm... so not slower?
<Regenaxer>
I don't think so
<Regenaxer>
The bottlenecks are in the system calls
<aw->
right
<aw->
ok thanks, i'll look more into it tomorrow
<Regenaxer>
No hurry, pil21 is still changing a lot
<aw->
do you have a roadmap document?
<aw->
or a todo list or something to show what is still missing
<Regenaxer>
No explicit document
<Regenaxer>
I even don't check, just continue what seems necessary ;)
<Regenaxer>
What is definitely missing after networking is DB
<Regenaxer>
@src/db.l
<Regenaxer>
But that is also already half-implemented
<Regenaxer>
I think tankf33der knows much better what is still missing
<aw->
ok
<aw->
oh i just saw (lines) is not supported?
<aw->
why not?
<aw->
i have some code using that
<Regenaxer>
Did you need it?
<Regenaxer>
Oh
<Regenaxer>
It can be easily implemented with 'till'
<Regenaxer>
(in File (let N 0 (while (from "\n") (inc 'N))))
<Regenaxer>
So not 'till'
<aw->
well it's not _very_ important.. i can do (call 'wc "-l" File)
<Regenaxer>
yeah
<Regenaxer>
It is an unnecessarily complex function for that simple task if in the base system
<aw->
ok no problem
<Regenaxer>
What is more a problem is the lack of (arg)
<Regenaxer>
(arg 'cnt) is there
<Regenaxer>
but not without arguments
<Regenaxer>
It can't be implemented in an efficient way
<aw->
maybe you'll figure it out eventually
<Regenaxer>
No, can't be implemented
<Regenaxer>
I don't have the stack structures for that in llvm
<Regenaxer>
Also, it is a stupid function
<Regenaxer>
ie. calling a function to get a value we already have
<Regenaxer>
So better keep the value fetched with (next) in a 'let'
<aw->
i see
<Regenaxer>
At the beginning I was also not happy about that, but now I find it is better
<Regenaxer>
Concerning performance, I think pil21 is generally about 30 to 50 percent slower
<Regenaxer>
I hope this can still be optimized
<Regenaxer>
using LLVM optimizers
<Regenaxer>
It is hard to beat pil64, due to the assembly language
<aw->
oh wow, that's a big difference
<aw->
but it's not complete yet, optimize later
<Regenaxer>
yeah
<Regenaxer>
these percentages are raw interpreter speed
<Regenaxer>
Normal programs will be a lot less
<Regenaxer>
as soon as I/O or system calls are involved
<Regenaxer>
tankf33der, 'connect' seems to work too now :)
<Regenaxer>
If I see it right, @lib/net.l should be finished
<tankf33der>
optimized llvm can beat asm.
<tankf33der>
or run on equal speed.
<Regenaxer>
haha, I don't think so
<Regenaxer>
never
<Regenaxer>
Too much overhead between functions
<Regenaxer>
We should make a contest
<Regenaxer>
Inviting LLVM gurus to compete
<tankf33der>
Regenaxer: already did it.
<tankf33der>
you just dont remember.
<Regenaxer>
you called 'opt'
<tankf33der>
yeap
<Regenaxer>
But it was not as fast as pil64
<tankf33der>
equal, +- several %.
<tankf33der>
Regenaxer: why net.l not in Makefile ?
<Regenaxer>
Did you try misc/bench.l ?
<Regenaxer>
net.l is nothing to make
<Regenaxer>
it is a normal lib
<tankf33der>
there is no misc/bench.l file
<tankf33der>
i can compile pil21 over opt and it will run any file in pil21-test with the same speed.