<m_mans>
sorry for interruption. I see strange bug (probably) with svg->pdf generation while starting application directly by httpGate (via 'names' file)
<Regenaxer>
no interruption at all! :)
<m_mans>
maybe problem is the environment for rsvg-convert
<Regenaxer>
might be. What error do you see?
<m_mans>
I see this in log: Invalid byte sequence in conversion input
<beneroth>
cess11, oh Intel added an off-switch for the management engine for NSA, so they can disable it on their devices. officially that switch doesn't exist.
miskatonic has joined #picolisp
miskatonic has quit [Quit: ERC Version 5.3 (IRC client for Emacs)]
miskatonic has joined #picolisp
miskatonic has quit [Remote host closed the connection]
<m_mans>
concerning rsvg-convert: it works in shell even without env-vars
<beneroth>
m_mans, might it run under a different user through httpGate then when called manually?
<m_mans>
it seems wrong guess about env
<beneroth>
different access rights etc
<m_mans>
no, user is the same: app
<m_mans>
another guess is tty
<beneroth>
T
<Regenaxer>
I think it does not need tty
<beneroth>
T
<Regenaxer>
I use it in several processes started by httpGate without tty
<m_mans>
I had similar issues in FreeBSD, everything worked in shell, but not with httpgate
<Regenaxer>
strange
<m_mans>
yeah
<beneroth>
very strange
<Regenaxer>
Did you find the reason in FreeBSD?
<m_mans>
I don't remember well
<m_mans>
seems not
<Regenaxer>
Hmm, what else is different httpGate <-> shell?
<m_mans>
there I had some issues when trying mount ISOs as folders
<m_mans>
I could strace it, right?
<Regenaxer>
tty, env
<Regenaxer>
yes, good idea
<m_mans>
let me try :)
<Regenaxer>
thx
<m_mans>
I can't give up in my case :)
<Regenaxer>
:)
<m_mans>
btw, can 'screen' be used to wrap httpGate?
<m_mans>
I mean in init.d script
<Regenaxer>
"wrap"?
<Regenaxer>
you mean not in background?
<m_mans>
I mean to start httpGate inside 'screen' session
<Regenaxer>
yes, I use tmux
<m_mans>
you don't use init.d script?
<Regenaxer>
httpGate does not care about the console I would say
<tankfeeder>
new quiz in 5min
<Regenaxer>
I put a single start script into /etc/rc.local
<Regenaxer>
that's simpler :)
<m_mans>
T, not httpGate itself, but some 'child' programs maybe
<m_mans>
hm, that's interesting. How do you start httpGate in rc.local?
<Regenaxer>
The start script starts httpGate, watchdog and inotifywait
<m_mans>
so, no tmux involved there?
<beneroth>
m_mans, today you can also use "@reboot <script>" in crontab instead
<Regenaxer>
right
<Regenaxer>
My current script is:
<Regenaxer>
echo "Set iptables ..."
<Regenaxer>
/home/app/7fach/setIptables
<Regenaxer>
<Regenaxer>
echo "Starting PicoLisp Application Server ..."
<m_mans>
line 499: [pid 9339] write(2, "Invalid byte sequence in convers"..., 42) = 42
<m_mans>
9339 is rsvg-convert
<beneroth>
Regenaxer is away
<beneroth>
I'm not sure about the setup. Is the input for rsvg-convert going through httpGate? how so?
<beneroth>
maybe you can redirect the input for rsvg-convert in a file
<beneroth>
for manual inspection/testing
<m_mans>
thanks beneroth, I see
<beneroth>
I'm still confused how httpGate should be able to have an effect
<beneroth>
I'm successfully generating PDF, but I don't use httpGate in that setup...
<m_mans>
it's not input who goes through httpGate, but httpGate is started by init system, so it's initially detached from tty, and all other pil-related processes are childs of httpGate
<beneroth>
yeah
<beneroth>
in my setup tty is also detached.
<m_mans>
if I start ~/picoLisp/pil ... - then everything is fine
<beneroth>
you sure httpGate uses that pil, and not another global installation?
<beneroth>
doesn't make sense, but is a likely source of problems sometimes
<m_mans>
good question
<beneroth>
I usually work with a global installation (as described in @INSTALL)
<beneroth>
but yeah, has the disadvantage that all apps on the same computer use the same picolisp version
<m_mans>
no, PicoLisp is the same
<m_mans>
local from /home/app/picoLisp
<m_mans>
btw, you said "my setup is also detached". What do you mean?
<beneroth>
ok. good. unlikely that this would have anything to do with this issue anway.
<Regenaxer>
So it does something with charsets at that moment
rgrau has joined #picolisp
alexshendi has joined #picolisp
aw- has quit [Ping timeout: 268 seconds]
aw- has joined #picolisp
aw- has quit [Client Quit]
alexshendi has quit [Ping timeout: 248 seconds]
<beneroth>
good find Regenaxer
<Regenaxer>
:)
orivej has quit [Remote host closed the connection]
orivej has joined #picolisp
alexshendi has joined #picolisp
rgrau has quit [Ping timeout: 250 seconds]
orivej has quit [Ping timeout: 248 seconds]
miskatonic has joined #picolisp
freemint_ has joined #picolisp
orivej has joined #picolisp
<freemint_>
beneroth, i might get an VM is 1 GB good enough or should i ask for more? They would ask me to write an index for mailman mailing lists.
orivej has quit [Ping timeout: 240 seconds]
orivej has joined #picolisp
<beneroth>
freemint_, is probably good enough. question is how much the OS uses of that 1 GB.
<beneroth>
I assume you want to do the indexing with pil DB
orivej has quit [Read error: Connection reset by peer]
<freemint_>
yes
orivej has joined #picolisp
<beneroth>
I would assume 1 GB to be enough.
<beneroth>
the thing with VMs / VPSes is that you can always increase RAM if you see that you need more :)
<freemint_>
i know
<beneroth>
let me check the RAM statistics of my pil app server (running multiple apps
<freemint_>
thanks
<beneroth>
too bad, I have no RAM statistics, only CPU, IO, traffic, etc
<cess11>
Wouldn't the amount of users matter?
<freemint_>
less than 100 current request a few 100MB of text
<beneroth>
good point cess11
<cess11>
Then I guess it would probably do.
<freemint_>
can you recommend a docker system?
<cess11>
Linux?
<freemint_>
ofc
<beneroth>
I would not use docker unless you really want to build a big sharding environment
<freemint_>
ok
<freemint_>
just want to ask about it
<beneroth>
or if you want a big test environment with various different machines.
<freemint_>
what does your RAM stat say?
<beneroth>
doesn't exist. anyway, I run multiple pil apps on it, all pretty db heavy. memory usage is seldom over 200 MB
<cess11>
Same in my experience.
<beneroth>
pil has pretty good RAM usage due the cell system
<cess11>
Don't think I've managed to surpass 1 GB without sort of forcing it.
<cess11>
Very efficient.
<freemint_>
ok
<beneroth>
well cell is a small block of memory. by using smaller blocks of memory, picoLisp wastes less memory. other language VMs/runtimes usually work with larger blocks of memory, or with memory-allocated-on-request... so more waste and more allocation happening
miskatonic has quit [Quit: ERC Version 5.3 (IRC client for Emacs)]
orivej has quit [Ping timeout: 240 seconds]
<beneroth>
actually this is also an advantage gained by not using arrays
<beneroth>
arrays require to be in single chunks of RAM... picolisp lists can be completely fragmented into whichever cells happened to be free during allocation of the list
<freemint_>
T
<freemint_>
on the other hand array based datastructures like tries can be more efficient.
<Regenaxer>
freemint_: PicoLisp as of today has a new byte-level peek/poke function
<Regenaxer>
I've only twittered it so far
<freemint_>
whoohooo
<Regenaxer>
It was necessary for PilMCU
<freemint_>
is it documented somewhere?
<Regenaxer>
to acces hardware ports
<Regenaxer>
yes, it is also in pil32
<beneroth>
cool
<Regenaxer>
(doc 'byte)
<Regenaxer>
In normal pil64 it is pretty useless I believe
<freemint_>
why?
<beneroth>
well you never know when you want to work with some serial ports or so :P
<freemint_>
it is not yet in the wiki
<Regenaxer>
Perhaps only low-level manipulations in structures for 'native'
<freemint_>
usefull
<Regenaxer>
beneroth, exactly
<Regenaxer>
UART
<Regenaxer>
freemint_, the wiki has no function ref
<beneroth>
Regenaxer, well freemint_ could allocate his array with (native "@" "malloc" ...) and do pointer-arithmetic on it with (byte), right?
<Regenaxer>
yep
<beneroth>
cool
<Regenaxer>
as in the example of the ref
<freemint_>
i am happy
<Regenaxer>
great :)
<beneroth>
oh overlooked it
<beneroth>
nice
<Regenaxer>
So we have double-benefit
<beneroth>
well there aren't many sane usecases, but some there are
<Regenaxer>
For Georg's PilMCU and freemint_ :)
<beneroth>
I could imagine working with an bit-table array retrieved from another system
<Regenaxer>
beneroth, right
<freemint_>
thanks for me being an usecase
<Regenaxer>
hehe
<freemint_>
or pictures
<Regenaxer>
For large-scale byte manipulations I would write a C function
<freemint_>
you could
<Regenaxer>
yeah, inline-C is easy in pil
<Regenaxer>
easier than using 'byte'
<Regenaxer>
Just if there is no C compiler, as in PilOS or PilMCU
<freemint_>
well that is a cool project
<freemint_>
C interpreter in LISP for backward compability
<freemint_>
*hides* not mine
<Regenaxer>
haha, bizarre idea
<Regenaxer>
ok :)
<cess11>
C is small.
<cess11>
So could probably be done pretty fast by someone who knows how.
<Regenaxer>
true
<freemint_>
Is there a (native "@" "malloc" 'N 8) equivaleng for PilOS or PilMCU
<Regenaxer>
nope
<Regenaxer>
The whole memory is occupied by heap and stack
<Regenaxer>
But system areas exist
<Regenaxer>
hardware ports, and perhaps reserved memory areas
<freemint_>
would it be possible to tell the garbage collector to ignore certain "pages" of cells?
<Regenaxer>
In PilOS, for example, the VGA frame buffer is memory at hex 000
<Regenaxer>
B00
<beneroth>
sorry, still have to say this. will not say anything further on that topic if you can't discuss it.
<beneroth>
Regenaxer, I'm not intending to destroy the good mood, but I still believe that (till) stopping at NULL-bytes should be documented somewhere. this has nothing to do with NULL bytes in symbol names. also, afaik NULL is a valid UTF-8 character, so it is not invalid within UTF-8 text files. (echo) does not stop at NULL, behaves according to documentation, and because it is meant for binary data I understand. I see no reason why picolisp and (till) should not
<beneroth>
be the right choice to handle a weird text file which happens to contains NULL bytes. (I fixed my issue by doing (eof)/(peek) after each (till), either we're EOF, or the (peek) returns one of the stop arguments we gave to (till), or it returns another character meaning it stopped because of a NULL-byte (which gets implicitly skipped, good so!).
<Regenaxer>
The gc ignores all outside the heap
<Regenaxer>
A "weird text file" will give lots of other problems
<beneroth>
I had no other problems than this. (well ok, multiple formats for dates used, but thats is easy to handle..)
<cess11>
Yeah. It can contaminate the VM and mess things up in ways that aren't immediately noticable.
<Regenaxer>
Input to pil must be clean. If it is not, it must be filtered externally (in a pipe or so)
<beneroth>
cess11, can it? how?
<freemint_>
could you set aside some "pages" in PilOS for 'malloc-ing
<beneroth>
Regenaxer, ok, so you say picolisp is not intended for doing such filtering.
<Regenaxer>
freemint_, that's what I meant with "reserved"
<Regenaxer>
like the text buffer
<beneroth>
freemint_, in your use case, if you run pilOS on special built hardware, you can easily tweak your pilOS imagine a bit to have additional reserved memory for your super-duper computing
<Regenaxer>
beneroth, you will open a pandora box
<cess11>
I don't know, I just know that I've put things in there that weren't healthy for those processes and later found they weren't reliable.
<Regenaxer>
it is better handled outside
<freemint_>
is the reserved space limited by another thing than availability of RAM
<cess11>
Sometimes web servers send weird encodings when they don
<Regenaxer>
it is easy to do (in '(iconv ...
<cess11>
't recognise the User-Agent.
<freemint_>
oh i see
<beneroth>
cess11, I still see no possibility that bad input can get the pil VM into a unintended state. you say this is possible and you saw it?
<Regenaxer>
I would say the same
<Regenaxer>
just the resulting data may be nonsense
<beneroth>
as me or as cess11
<Regenaxer>
as beneroth
<beneroth>
ok
<beneroth>
well you can make mistakes and have bad or even malicious data in your db if you just blindly eat the input, but that is another issue.
<Regenaxer>
In any case, as input is not executed
<beneroth>
exactly. dangerous stuff is functions like ($tim) which do interpretation of raw input.
<cess11>
I've seen it start showing new parentheses instead of chars I've just entered, making it nonsense.
<cess11>
It can mess up tmux too.
<Regenaxer>
right
<Regenaxer>
escape sequences
<beneroth>
well that is then a messed up TTY, not a messed up pil VM, no?
<Regenaxer>
T
<Regenaxer>
$ stty sane
<cess11>
Perhaps. I'll try if I manage to end up there again.
<beneroth>
Regenaxer, I don't see the problem/pandoras box in documenting that (till) stops at NULL bytes. I don't ask for handling NULL bytes within the (till) implementation. it is an edge case.
<Regenaxer>
Just the ref?
<beneroth>
cess11, I managed messed up TTYs especially when (raw) is in play somewhere.. but afaik this only affects TTY.
<Regenaxer>
yes, only visibility of console text
<cess11>
'(quit) also didn't work, neither '(bye), but didn't manage much else, so I assumed it stopped working somehow.
<Regenaxer>
I think you must type ^J instead of <return>
<Regenaxer>
if tty is in raw mode
<Regenaxer>
such things
<beneroth>
Regenaxer, well correctly the ref for (till) states that it reads till a character contained in any is found or eof. which is wrong in my view. unless any implicitly includes the NULL byte. the similar (by usage, not implementation) function (echo) correctly only stops at sym or eof.
<Regenaxer>
right, it assumes "clean" data stream input
<beneroth>
so yes, all I would do is mention in ref for (till) that it also stops at NULL byte.
<beneroth>
well, I found no source stating that NULL is not valid in a UTF-8 text file.
<Regenaxer>
the pandora box is that NULL byte must be mentioned *everywhere* then
<Regenaxer>
all input
<Regenaxer>
not specific to 'till'
<Regenaxer>
the behavior is not defined
<beneroth>
good argument. I'm not convinced, trying to think of what else it could affect
<freemint_>
If i would modify PilOS i would modify it so that the heap is shared between cell "pages" an raw pages ignored by GC
<Regenaxer>
the implementation is not prepared for NULLDE bytes
<freemint_>
what is the right word for "pages" of cells?
<Regenaxer>
beneroth, everywhere a symbol name is involved
<Regenaxer>
freemint_, there are no cell pages in PilOS
<Regenaxer>
PilOS grows the heap up, until it gets close to the stack
<beneroth>
which size are the pil64 cell pages?
<beneroth>
according to OS or something else?
<freemint_>
good question
<Regenaxer>
beneroth, *only* for symbol names
<Regenaxer>
and numbers perhaps, as they are first parsed as symbols
<Regenaxer>
echo does not care
<Regenaxer>
it operates on a byte stream
<Regenaxer>
But 'read' may read symbols
<beneroth>
Regenaxer, I don't care about NULL byte being not returned! My issue is that (till) stops at them.
<Regenaxer>
this is a random behavior
<beneroth>
it is not.
<beneroth>
or not as I saw it.
<Regenaxer>
I did not implement it "absichtlich"
<Regenaxer>
it is a consequence
<beneroth>
yes.
<Regenaxer>
cause NULL is a "non-char"
<beneroth>
it is a consequence of the underlying C function used. right?
<Regenaxer>
it is assumed NEVER to appear in the input stream!!
<Regenaxer>
that'a all
<Regenaxer>
if it appears, there is no guarantee :)
<beneroth>
then you have to differ between text input streams and others. as it is assumed and OK to appear when using an input stream with (echo).
<Regenaxer>
There is no C function
<beneroth>
T
<Regenaxer>
So the rule is: Don't send NULL
<Regenaxer>
same as: Don't pull the power chord!
<cess11>
Or search and remove when importing the data?
<beneroth>
consequence: you can't use picolisp to handle untrusted input (e.g. everything from internet)
<Regenaxer>
or: Don't smash your tablet on the wall!
<Regenaxer>
yep
<beneroth>
that is a pretty big limitation.
<Regenaxer>
NEVER trust input from outside!
<freemint_>
that is very serious: consequence: you can't use picolisp to handle untrusted input (e.g. everything from internet)
<Regenaxer>
no
<Regenaxer>
it is not
<Regenaxer>
this is for any system
<Regenaxer>
you cant trust input
<Regenaxer>
of course!!!!
<Regenaxer>
But pil is safe I believe
<Regenaxer>
as long as you don't 'eval'
<beneroth>
T. my point is that I should be able to handling of untrusted input on picolisp lisp level.
<beneroth>
exactly
<Regenaxer>
If you send non-UTF8 you have the same
<Regenaxer>
unpredictable data
<freemint_>
there is a difference "between do not trust input". "Picolisp can not handle untrusted input"
<beneroth>
but what you say is "never even dare to read from a input stream which might contain NULL" which, to be consequent & secure, means I must NOT and never do (in (connect "google.com" 80) ..)
<beneroth>
freemint_, T
<beneroth>
that is what I mean!
<Regenaxer>
What is the problem?
<Regenaxer>
Show me a scenario where we have trouble
<Regenaxer>
PicoLisp handles the input
<Regenaxer>
it does not crash
<Regenaxer>
it gets garbage data
<cess11>
Sure you can trust Google and connect to them. They are quite reliable.
<beneroth>
sure
<beneroth>
maybe
<beneroth>
if my DNS is not redirect.
<Regenaxer>
but the the input source may send garbage directly
<beneroth>
(in (connect "google.com" 80) (till)) -> oh there is a null byte in it.
<beneroth>
yes
<beneroth>
that is what I say. allow the input source to send garbage. give me the tools to handle it correctly on lisp level. we have this tools.
<Regenaxer>
So NULL is just one aspect of the pandora box
<beneroth>
what is another then?
<cess11>
It could probably be handled in the VM, but it is probably easier to clean it with some tool. w3m, hxnormalize or what might be appropriate.
Regenaxer has left #picolisp [#picolisp]
<beneroth>
it doesn't need to be handled in the VM I think.
Regenaxer has joined #picolisp
<Regenaxer>
hit ^C
<beneroth>
nothing to change in the VM.
<beneroth>
ok, good.
<beneroth>
I don't like when you are mad and leave.
<Regenaxer>
I wrote:
<freemint_>
sorry for offtop what is 'rc and .rc for?
<Regenaxer>
If pil would skip the NULL byte or handle it is some defined way, the outside source can get *exactly* the same result by sendig other bytes
<beneroth>
Regenaxer, don't change the null byte handling. leave it as it is.
<Regenaxer>
freemint_: resource
<beneroth>
or are you saying: the current handling is not stable/to be trusted? I think it is stable and trustable.
<freemint_>
is resource some standardized format?
<beneroth>
freemint_, I think it isn't.
<Regenaxer>
freemint_, you have foo.rc often in Unix
<freemint_>
mhh i feel like Regenaxer kinda felt attacked
<beneroth>
possible
<freemint_>
his reactions were very defensive. Even by my standards and questions
<beneroth>
all I wish for is: no undocumented behaviour.
<freemint_>
i understand that
<freemint_>
cany you built what you want with 'rd?
<beneroth>
I can built it with current (till), no problem.
<beneroth>
could be done with (rd), but would be very wasteful
<freemint_>
for some definition of very wastefull
<beneroth>
T
<beneroth>
checking every byte on lisp level is much slower then doing it directly on asm/VM level as (till) does
<freemint_>
maybe a factor of 6?
<freemint_>
the disk is the bottleneck anyhow
<beneroth>
I guess as he is a solo worker he is probably not used to be criticized beside the obvious critic on his choices which are a result from his principles and therefore easy to defend.
<freemint_>
it produces more heat for the same problem and takes resources away from other processes
<beneroth>
well I think I find out all the input functions which prematurely stop at NULL and propose to change their documentation. one last time.
<freemint_>
ok
<freemint_>
also check functions which work on transient symbols
<beneroth>
I don't think those are many. only those which read until something. so (line) is a candidate).
<beneroth>
no, symbol names must not contain NULL, this is already documented.
<freemint_>
Ok
<cess11>
It isn't sure it will stop when encountering it in some other context.
<freemint_>
can you explain transient symbols to me
<freemint_>
i never really got them
<cess11>
'chop, 'pack, 'split and probably some more are probably the best introduction to them.
<beneroth>
two things: 1) they behave like normal symbols but when their value is not set they evaluate to their name. 2) they form an implicit namespace, so two times "symbol" in one file is the same symbol, but "symbol" two times in two different files are two different symbols.
<beneroth>
that is the "transient" part
<freemint_>
what is a file. If i reopen a file does it count as a new file?
<beneroth>
see (===) function
<beneroth>
yes, per parsing I would assume.
<beneroth>
you can have multiple transient scopes within one file by using (===)
<freemint_>
i used them as strings in the past
<freemint_>
can you interpret things in a choose transient scope?
<freemint_>
*choosen
<beneroth>
that is the main use, but they are real symbols. so you can assign them a value or even set a property on them
<freemint_>
I know
<beneroth>
no, you can only act within the transient scope you happen to be in.
<freemint_>
ok. Do you know why they were created
<beneroth>
yes, to prevent symbol clashes. many library functions have all argument variables as transient symbols, therefore they will never get the value of a similar named symbol on a higher layer/scope
<beneroth>
its highly related to how the parsing/reader works.
<freemint_>
ah
<beneroth>
the reader "translates" the source code to cells and pointers.
<freemint_>
ok
<freemint_>
i found an use for them
<freemint_>
thanks for the info
<beneroth>
say your source is "(setq A 5)" -> during parsing (before execution!) this is translated to memory. so a a list is built of 4 elements: [cdr->next-cell|car->setq][cdr->next cell|car->A][cdr->NIL|car->5]
<beneroth>
the thing after the -> is a memory pointer
orivej has joined #picolisp
<freemint_>
where do you count 4 cells?
<freemint_>
i count 3
<beneroth>
right
<beneroth>
4 elements. stored in 3 cons pairs. the last element is the NIL.
<beneroth>
so I say the symbol A is stored at pointer 5555.
<beneroth>
NIL at pointer 000.
<beneroth>
5 is at some other pointer which points to a number (remember: the type (symbol, list or number) is coded into the pointers!)
<beneroth>
the firs time the reader finds A, it just takes an empty cell and says that there A is stored. the next time it finds A, it remembers which pointer it used for A the last time, and re-uses that pointer.
<freemint_>
i know you can stop
<beneroth>
well if the source code is not 'A but "A", in two different files or with a (===) in between, then the reader will not lookup the pointer of its previous "A" but store it in a new location