<yunfan>
then which app do you use to listen on 8080 on termux?
<Regenaxer>
Any app I'm developing (?)
<Regenaxer>
./pil someProject/main.l -main -go +
<yunfan>
you mean that http.l could serve static html page?
<Regenaxer>
sure
<Regenaxer>
any files
<yunfan>
without specific code?
<Regenaxer>
See line 205 and following in @lib/http.l
<yunfan>
i mean like i want to server uri for `localhost:8080/js/*.js` all the .js page could be mapping to a local directory
<yunfan>
ok
<Regenaxer>
It handles no wildcards
<yunfan>
tested, perfect
<Regenaxer>
:)
<xificurC>
Regenaxer: you often mention you're using picolisp in production. Care to give a ballpark figure of the amount of customers and their respective fields?
<xificurC>
I'm curious what industries do you work with and what is the volume :)
<Regenaxer>
uh, that's 35 years now ;)
<Regenaxer>
Almost all fields
<Regenaxer>
graphics, ERP, logistics, some embedded
<Regenaxer>
If I scan the records, about 40 customers over the years
<Regenaxer>
Currently I have projects open with 7 customers
<Regenaxer>
or 6 and a half, one is idle
<xificurC>
that's enough to get a feel, thanks :) Any tip on largest pilDB?
<Regenaxer>
And we have our own projects which we try to market, currently one
<Regenaxer>
The largest I had between 2005 and 2009, about 700 million objects (distributed er 60 DBs, still pil32 at that time)
<xificurC>
you really needed 60 dbs for that volume or was it a customer requirement
<Regenaxer>
Currently the biggest DB is an ERP customer with about one 700 thousand items
<Regenaxer>
It was highly parallel, distributed across many machines
<beneroth>
hi Regenaxer, xificurC, yunfan
<Regenaxer>
Hi beneroth
<xificurC>
hi beneroth, healthy?
<beneroth>
Regenaxer, that was the one optimised for input processing?
<beneroth>
nah, better but still not really well
<xificurC>
:(
<Regenaxer>
It used a C++ frontend to scan storage systems
<yunfan>
beneroth: hi
<Regenaxer>
At Infineon, BMW and some other clients
<Regenaxer>
Infineon was the main driving force
<Regenaxer>
But the system turned out rather useless and they abandoned it
<beneroth>
you should make a page with a list of clients, at least on 7fach or software-lab website.
<Regenaxer>
ie the investors turned off :)
<Regenaxer>
I think the clients don't like that, especially for such abandoned projects
<beneroth>
I mean you could use such a list as a marketing tool ;-)
<Regenaxer>
yeah
<beneroth>
yeah of course, successful projects with happy clients are better for reputation :D
<beneroth>
though I guess politics usually ruined it, not the tech...
<Regenaxer>
BTW, just at the moment I'm working on a system which will have up to 40 DBs (beneroth knows)
<yunfan>
Regenaxer: how about use some famous kv database as the storage backend?
<beneroth>
yunfan, why, Regenaxer knows pilDB better, and it works better :D
<Regenaxer>
I have no idea what the famous kv is ;)
<beneroth>
key-value
<Regenaxer>
ah
<beneroth>
good for simple primitive stuff.
<xificurC>
I have a small project here at work that I would like to try to port to picolisp if time permits. To try out the language with something "real". From a high level view I think I know where to start, but there's 1 I don't and can already ask :) The current implementation has a POST URL which is used for filling up the db
<beneroth>
but not good for a complex DB you want to query
<Regenaxer>
berkeley db?
<xificurC>
so another process has to do a curl -XPOST and send a JSON, which the backend processes by storing it in a database. What would be an equivalent in pil?
<beneroth>
I guess yunfan means younger stuff
<Regenaxer>
xificurC, I always used CVS to import
<yunfan>
beneroth: i just saw the talk between Regenaxer and xificurC and they mentioned millions of records over 60 DB
<beneroth>
yeah
<Regenaxer>
a CVS file could be posted to the db
<yunfan>
Regenaxer: yes berkeley db is good
<yunfan>
google's leveldb is also a choice
<xificurC>
CVS? Do you mean CSV?
<yunfan>
also there's wiredtiger and sophia
<Regenaxer>
sorry, yes
jibanes has quit [Ping timeout: 248 seconds]
<Regenaxer>
XML is also fine, I use it if absolutely necessary
<beneroth>
it has much overhead
<xificurC>
is there a csv parser?
<beneroth>
there are also JSON parsers
<Regenaxer>
There is a general csv parser, but I use TAB separated files
<Regenaxer>
no parser needed
<Regenaxer>
just (while (split (line) "\t") ..
<xificurC>
unless there's a tab in the data :)
<Regenaxer>
right
<beneroth>
xificurC, look at (from) and (till) :)
<Regenaxer>
"normal" data don't contain tabs, as the user cannot enter it :)
<beneroth>
(unless there is a NULL byte in the data :P sorry Regenaxer )
<xificurC>
text might
<Regenaxer>
a tab moves to the next input field in most GUIs
<Regenaxer>
yes, same with null bytes
<beneroth>
I had to parse a DB export with NULL bytes within text values. horrible!
<xificurC>
haha
<beneroth>
took me long to even detect the problem.
<Regenaxer>
yeah
<beneroth>
can't see them in the debug console, you know? :D
<Regenaxer>
A general
<Regenaxer>
CSV was discussed here too
<Regenaxer>
I posted one, and m_mans had one too
<Regenaxer>
or tankf33der?
<xificurC>
where is this general csv parser? I can't grep it in the sources, just one (de csv) in @lib/xhtml.l which probably isn't a parser
<beneroth>
xificurC, if you need to validate the input fully then creating a little FSM with (state) is handy
<Regenaxer>
xificurC, anyway, your POST would send some file, which is then imported
<xificurC>
as usual then :)
<Regenaxer>
T
<xificurC>
thanks
jibanes has joined #picolisp
aw- has joined #picolisp
<aw->
hi all
<Regenaxer>
Hi aw-
<aw->
Regenaxer: i saw your concurrent/production DB size, i didn't expect it to be that big! Those are useful numbers to post on the website or in the docs
<Regenaxer>
Perhaps, yes, but rather specific, and useless without full project detail
<Nistur>
mornin' all
<Regenaxer>
Hi Nistur
<Nistur>
how are things?
<Regenaxer>
good I hope :)
<Nistur>
:)
<Nistur>
I miss when computers just _worked_.
<Regenaxer>
So not good?
<Nistur>
for me? No.
andyjpb has joined #picolisp
<Nistur>
Not at all :P
<Nistur>
I'm currently wrestling with Windows not wanting to do anything I tell it (it's almost entirely unresponsive for some reason) and I discovered yesterday that almost all my colleagues don't respect me, or agree with me... nd I think they probably dislike me now. One of my main roles on this project, according to the CTO and my direct line manager right now is to bring the project up towards industry
<Nistur>
standards, and also help educate the other programmers... a lot of them have 'as much' experience as me... ie about 8 years... but they have been at that company, making small games, since uni... so they're stuck with some VERY bad habits, and I'm coming in and poo-pooing over their work. They're getting defensive and disagreeing with me...
mtsd has joined #picolisp
<Regenaxer>
Uh, sounds really bad!
<Nistur>
yes.
<Regenaxer>
What kind of bad habits for example?
<mtsd>
Good morning all
<Nistur>
one of the guys, I believe was a lecturer and he seems to do a lot of things 'by the book' favouring elegant, and 'correct' code over that which runs faster. I can give a concrete example of code in the codebase, but it requires more explanation, so I'll give a slightly more abstract example
<Regenaxer>
Hi mtsd!
<Nistur>
Regenaxer: how would you write a fibonacci generator? And then how would you write one if it was performance critical?
<Nistur>
o7 mtsd
<mtsd>
Hi Regenaxer, Hi Nistur
<mtsd>
Nistur, sorry to hear about the situation at work.
<Regenaxer>
Nistur, see misc/fibo.l - there are about 5 versions of fibonacci
<Nistur>
I expected that :P
<Nistur>
but the point is, that the normal 'correct' way is the recursive way, because it's short and elegant
<Regenaxer>
6 in fact
<Nistur>
but it's hellishly inefficient
<Regenaxer>
yes
<Nistur>
you can do optimisations on it, like caching the results of them
<Regenaxer>
Recursion wich memoization is the fastest
<Nistur>
but best way is probably the naive approach to just loop forwards
<Regenaxer>
cachedFibo
<Nistur>
yeah
<Nistur>
but that then leaves the cache hanging around, which in gamedev at least would be wasteful
<Regenaxer>
yes, takes space
<Nistur>
but the point I'm trying to make is, this guy wouldn't consider anything like that. He'd write the simple, elegant, recursive function
<Nistur>
and that is _right_
<Regenaxer>
But the cache could be a local variable
<Regenaxer>
I see
<Nistur>
I have found one recursive function which calls itself 1650 times.
<Nistur>
and it was called 4x in a frame
<Nistur>
true, it was at that point still TECHNICALLY loading
<Nistur>
so it's less problematic
<Regenaxer>
In many cases this might be OK. fibo is especially inefficient
<Nistur>
but this is a general purpose solution he provided for the junior programmer to make it more elegant
<Regenaxer>
In some cases you almost cannot avoid recursion, eg in trees
<Nistur>
(basically typechecked message passing in a hierarchical entity-component system)
<Nistur>
other bad practices, for gamedev at least, are that _everything_ is done with strings
<Nistur>
all IDs are strings
<Nistur>
everywhere
<Nistur>
and I have brought this up and got the rebuttal that strings are not slow, strings are not problems any more
<Nistur>
... but
<Nistur>
they are causing a lot of memory allocations
<Regenaxer>
yep
<Nistur>
and they are magnitudes slower than an int comparison, and if you use an enumeration or something, no more clear
<Nistur>
so it's stupid
<Nistur>
plus, we're using a lot of hard coded strings
<Nistur>
which is jjust NASTY
<mtsd>
"_everything_ is done with strings" Tcl heritage in there? ;)
<Nistur>
mtsd: this is C# with Unity3D
<Nistur>
so it's more copy/paste from SO heritage, than Tcl
<Nistur>
because it's "easy"
<mtsd>
Just joking, haha
<Nistur>
and now has become the "done thing"
<Regenaxer>
Is execution too slow?
<Nistur>
and then I wrote a serialiser in C# because I am making a save system... their current one is... shocking... I got attacked for writing this serialiser when I should have used one of the ones that the company already has... despite me having looked into them and none have what I needed... I am adding versioning to the fields so I can deserialise only the fields that were serialised in the first place,
<Nistur>
regardless of whether we add more fields into the code or not, and I'm adding CRC... and it's all in the serialiser, so the actual save code doesn't have to worry about it
<Nistur>
... I then looked at the existing system
<Nistur>
which uses "PlayerPrefs" (a wrapper around NSUserDefaults on iOS)... it is basically a string-indexed map of values
<Nistur>
it _actually_ uses something called "ObscuredPrefs" because PlayerPrefs is insecure
<Nistur>
which writes all values into a string and then 'encrypts' them
<Nistur>
and it 'encrypts' the keys
<Nistur>
... the 'encryption' is actually just a very VERY slow XOR loop
<Nistur>
and, the system follows particularly bad practice, and does two allocations more than necessary
<Nistur>
it's
<Nistur>
... ridiculous
<Nistur>
and I cannot believe anyone looked at this and thought it was worth using
<Nistur>
I mean XOR masking data in memory is one thing, and I've seen it done a lot
<Nistur>
but as 'encryption' (it's referred to in the documentation as "simple symmetric encryption") it's laughable
<Nistur>
and I'm worried by the fact that they are seriously advocating this over what I wrote. I mean, the serialisation is using C#'s reflections, so my code itself is pretty minimal (and I've written it about 10x now on various projects. I can pretty much write it blindfolded) and I'm using C#'s inbuilt cryptography suite for _actual_ encryption.
<Nistur>
... apologies for that rant :P
<Regenaxer>
np
<Regenaxer>
XOR encryption is fine, as long as you use a vernam string of bits
<Regenaxer>
the *only* proven unbreakable encryption ;)
<Nistur>
this, I believe, falls back to using "4441" to XOR it with
<Regenaxer>
hehe, not really infinite nor random :)
<Nistur>
during the debates I had with one of the other programmers about these things, I was told "that every single programmer around [him] disagrees with [me]"
<Nistur>
oh, another example I can give. I was asked, whether I would be saving deltas to 'cloud storage' of the save files because 'when the saves get big - and they will' I will be expected to
<Nistur>
I have never seen a save system in a game I've worked on, and most have been _MUCH_ more complex than this, which saves more than about 100kB... and deltas are complex to manage (relatively speaking)
<Nistur>
so I said that I couldn't imagine that happening, and I jokingly added 'unless you start adding inventory items as strings'
<Nistur>
the response: 'what else would you save them as? IDs?'
<Nistur>
so, that told me a lot. They started writing save systems that stored deltas, and made far more complex systems because of this... because they didn't consider writing smaller save files
mtsd has quit [Quit: Leaving]
mtsd has joined #picolisp
orivej has joined #picolisp
inara has quit [Quit: Leaving]
inara has joined #picolisp
styx has joined #picolisp
<beneroth>
omg Nistur
<beneroth>
I feel you
orivej has quit [Ping timeout: 268 seconds]
<Nistur>
I'd rather you didn't ;)
<beneroth>
haha
<beneroth>
my day gets worse and weirder
<beneroth>
still sick
<Regenaxer>
:(
<Nistur>
:(
<beneroth>
my current pil project client (a NGO) is collapsing right now, as it seems
<beneroth>
they got new people in, and apparently the new ones manage to piss of old the old leaders with the necessary knowledge etc.
<beneroth>
one after another
<Regenaxer>
ow
<beneroth>
no issue for me, I think. but sad.
mtsd has quit [Quit: Leaving]
jibanes has quit [Ping timeout: 256 seconds]
pierpal has quit [Quit: Poof]
pierpal has joined #picolisp
<xificurC>
are you guys doing pil as contractors? How do you get clients?
<Regenaxer>
This is the difficult thing indeed. Most of my customers don't care (or don't even know the difference), only 2 so far contacted me be because they explicitly wanted pil)
<Regenaxer>
I got all customers via mouth propaganda
<Regenaxer>
It never worked when I tried to acquire by myself
pierpal has quit [Quit: Poof]
pierpal has joined #picolisp
rob_w has quit [Quit: Leaving]
styx has quit [Quit: styx]
pierpal has quit [Quit: Poof]
pierpal has joined #picolisp
alexshendi has joined #picolisp
<xificurC>
typo man returns: Thus, it is impossible to type to more characters than allowed into the field. (app.html)
<Regenaxer>
Thanks! :)
<xificurC>
Regenaxer: there's a bit of a bootstrapping problem in your previous answer. If you never acquired by yourself how did you start?
<Regenaxer>
Friends of mine had a hardware company, and needed software support for a CAD system
<Regenaxer>
thus also mouth propaganda
<Regenaxer>
Also, at the same time I worked free-lance for the company I had worked for in Japan
<Regenaxer>
The typo: Best to replace "to" with "in"?
<Regenaxer>
or just empty?
<Regenaxer>
Probably best to just remove the "to"
<Regenaxer>
done
<xificurC>
yes, just remove
<xificurC>
and thanks for the answer
<Regenaxer>
:)
<xificurC>
any reason why you use such short names for relations? Why nm and not name, which is pretty short too?
<xificurC>
I understand on 64 there's a limit where a transient's name will fit in a single cell
<Regenaxer>
yes, in fact I use name too sometimes, "full" name
<Regenaxer>
yes, but this is no problem. The symbol 'nm' exists only once per namespace
<Regenaxer>
Where it *is* a very little bit critical is disk space
<Regenaxer>
one byte more increases the size of each object ;)
<Regenaxer>
One million objects is roughly one megabyte
<Regenaxer>
But this also does not really matter of course
<xificurC>
probably not gonna matter compared to what the value takes :)
<Regenaxer>
exactly
<xificurC>
did you ever make a comparison of DB size compared to more "traditional" DBs?
<Regenaxer>
I like short identifiers in geneaal, find them more readable
<Regenaxer>
No, such comparison is difficult as it depends on the model
<Regenaxer>
mostly the number of indexes, and how elaborate the indexes are
<xificurC>
I thought so, wanted to ask anyway :)
<Regenaxer>
The data themselves are easy to measure with the 'size' function
<Regenaxer>
ie the PLIO representation size
<xificurC>
yesterday I was running a sample from app.html and left the browser open with a session, so I was on a port different than 8080. This session expires after some time defined in pil. After that it was painful to navigate back to the home page, pressin Back 10 times left me nowhere. Is this ever an issue?
<Regenaxer>
Why navigate back?
<Regenaxer>
Just go to localhost:port again?
<Regenaxer>
btw, the session should not expire if JS is enabled
<Regenaxer>
the client pings every few minutes
<xificurC>
what does it mean to have JS enabled
<Regenaxer>
in the browser
<xificurC>
re navigate back - it's a typical way to use a browser, no?
<Regenaxer>
yes, but if the session expired all pages in the history are meaningless
<Regenaxer>
Within a session back works well
<xificurC>
exactly :)
<xificurC>
yes but what enables JS?
<Regenaxer>
So normally (with JS) the session terminates only if the browser is closed
<Regenaxer>
Normally JS is on by default (unless it is a text browser)
<Regenaxer>
disabled in the browser settings
<Regenaxer>
I normally disable JS
<Regenaxer>
and enable explicitly only for certain sites
<xificurC>
well, I had JS on and the session expired
<Regenaxer>
(including localhost ;)
<Regenaxer>
hmm, the window or tab was still open?
<xificurC>
nevermind, I'll ask again if it happens and I have the setup
<xificurC>
yes
<Regenaxer>
ok
<xificurC>
also, reloading the main page kept opening new sessions (ports)
<Regenaxer>
yes, this is correct
<Regenaxer>
eg localhost:8080
<Regenaxer>
The server assumes a new client connected
<Regenaxer>
This is nasty in the repl
<xificurC>
why do you need a new port for each session? And what is the solution when running a web site and only 80/443 can be guaranteed to work?
<Regenaxer>
as then two sessions are running in the same terminal
<Regenaxer>
The solution is httpGate
<Regenaxer>
I have normally only port 443 open in iptables
<Regenaxer>
and 22
<Regenaxer>
The port is needed so that each session sees its own transactions
<xificurC>
so sessions still open ports but httpGate takes care of handling the which-request-goes-to-which-session part
<Regenaxer>
yes, in combination with base href in each session
<Regenaxer>
it generates domain/123/url
<xificurC>
needed sounds like it's the only way to do it and every other language/framework *has to* do it the same way
<Regenaxer>
where 123 is the port
<Regenaxer>
nono, only in pil ;)
<Regenaxer>
I don't know at all how other frameworks handle sessions
<Regenaxer>
Never cared ;)
<Regenaxer>
with cookies perhaps
<beneroth>
re
<beneroth>
yeah, usually
<Regenaxer>
Hi beneroth
<Regenaxer>
how is health?
<beneroth>
feeling better with some calories intus :D
<Regenaxer>
perfect :)
<Regenaxer>
Lets start weekend :)
<beneroth>
yeah
<Regenaxer>
I still have trouble
<Regenaxer>
deadlocks
<beneroth>
on Android? or DB, the issue tankf33der brought up?
<Regenaxer>
Normally deadlocks never happen in pil db
<Regenaxer>
no
<Regenaxer>
it is the multi-DB setup for BTG
<Regenaxer>
accessing the files directly with 'blk'
<Regenaxer>
including locking
<Regenaxer>
I'm aware that if I access a remote DB in a transaction (dbSync) I may produce a deadlock under certain conditions
<beneroth>
sorry yeah I think you solved both I mentioned. My head is still totally fuzzy
<Regenaxer>
But here I don't have such a condition
<Regenaxer>
even not a transaction
<Regenaxer>
no worry :)
<Regenaxer>
Must be something simple, I just don't find it atm
<beneroth>
but the "concurrent" accessors get stuck?
<beneroth>
must be outside pil, in filesystem or kernel then, no?
<Regenaxer>
yes, one process seems to lock the DB even if it is not a transaction
<Regenaxer>
no, I dont think so. They wont lock user files
<Regenaxer>
I cant reproduce well
<Regenaxer>
saw it in strace when the thing hung
<Regenaxer>
waiting to get a read lock
<Regenaxer>
xificurC, concerning the port
<Regenaxer>
I think this is the cleanest solution
<Regenaxer>
each session has its own private connections
jibanes has joined #picolisp
<xificurC>
I'm sure you have your reasons. I'm usually asking find and understand those reasons :)
<xificurC>
to find*
<beneroth>
you do well, xificurC :)
<xificurC>
a cookie backed session is kept locally, so a user might be able to continue his session even after restarting the browser
<beneroth>
in Regenaxers webframework: session = (long-) running process (and its memory)
<xificurC>
I guess you don't have this option when keeping the session on the server
<beneroth>
cookie is just the identifier, as it is the port number here
<Regenaxer>
you can copy/paste the session even to another client
<beneroth>
or well, yes you can also store everything in the cookie but thats horrible.. does not work for larger data :)
<Regenaxer>
I sometimes move from w3m to chrome
<Regenaxer>
Nasty is that a cookie is global
<beneroth>
no and yes
<xificurC>
can a session theoretically be restored after e.g. a day of inactivity? The browser is closed so no keep-alive happening
<Regenaxer>
what to do when you want more sessions?
<beneroth>
security-wise its very bad
<beneroth>
opening another browser xD
<Regenaxer>
well, if you *know* the session id, why not?
<beneroth>
xificurC, when the server-process for the session is not there anymore, then the session is not there anymore
<Regenaxer>
However without browser the session terminates in 5 mins or so
<beneroth>
there is no session-object stored in the DB, or thelike
<xificurC>
yeah so you could keep the session around if you wish to