<xificurC>
Regenaxer: hi :) @app/er.l, how does one know what sizes to prefer to optimize for perfomance? How does one know how many separate files are needed?
<Regenaxer>
Hi xificurC!
<Regenaxer>
Not an easy question, and not very critical unless the DBs are very big
<Regenaxer>
I use a few rules of thumb
<Regenaxer>
I usually check during development how big typical objects of that class get
<Regenaxer>
'size'
<Regenaxer>
Then it is ok if most of them fit into a single block
<xificurC>
don't worry then, if I ever get to a point of a larger DB I will ask again :)
<Regenaxer>
For indexes I use 2 (256 bytes) for small indexes, else 4
<Regenaxer>
yes, good :)
<xificurC>
I read +Dep 3 times now and still have no clue what's it doing :D
<Regenaxer>
+Dep is concerned about the time when a value is cleared
<Regenaxer>
then dependent values are cleared too
<Regenaxer>
We have in +Pos (rel ord (+Dep +Joint)
<Regenaxer>
if the +Joint to the order is lost, i.e. the position is deleted
<Regenaxer>
then the link to the item should be deleted too, so that the pos may rest in peace (be garbage collected)
<Regenaxer>
A +Pos with no order is no usable
<xificurC>
so the (rel ord (+Dep +Joint) (itm) pos (+Ord)) says if the ord gets cleared the itm relation should get automatically cleared as well?
<Regenaxer>
yes
<Regenaxer>
the +Ref
<Regenaxer>
ie. the index tree entry
<xificurC>
ok
<Regenaxer>
if in this model the price had a +Ref too, it would be wise to give (itm pr) to +Dep
<xificurC>
if a relationship is +Fold(ed) how does it manifest in a pilog search? Does one need to use fold in the search?
<Regenaxer>
No, the pilog rules know about the types. But the filter clauses need the proper predicates
<Regenaxer>
*ID and *Tab are globals in @lib/xhtml.l and @lib/form.l
<xificurC>
loading them in a pil session doesn't make (doc '*ID) work :(
<Regenaxer>
So above: (+IdxFold +String) needs part in the filters
<Regenaxer>
*ID has no doc, it is just a global to communicate an object
<Regenaxer>
and 'allow'ed global ;)
<Regenaxer>
It has no predictable value outside a HTTP transaction
<xificurC>
and url> has doc?
<Regenaxer>
nope
<Regenaxer>
In @lib/db.l there are only stubs
<Regenaxer>
It is up to the application to define methods
<xificurC>
how are the results interpreted though. E.g. +Sal says (and (may Customer) (list "app/sal.l" '*ID This))
<Regenaxer>
the list, if non-NIL is applied to 'mkUrl' by 'url>
<Regenaxer>
It gives a properly formatted URL
<Regenaxer>
You can see it in the browser
<Regenaxer>
with '=' and '&' etc.
<Regenaxer>
An URL in pil can encode all atoms and 1-level lists
<Regenaxer>
eg +123 is a number
<Regenaxer>
and $abc is an internal symbol
<xificurC>
ok so a list is a redirect
<Regenaxer>
(1 abc {3}) is "+1_$abc_-3" iirc
<Regenaxer>
not redirect
<Regenaxer>
it can also be in links
<xificurC>
ah, a link to another URL, ok
<Regenaxer>
or POSTed by a button
<Regenaxer>
yes
<xificurC>
one more then I'll go reading more and maybe start coding :)
<Regenaxer>
ok :)
<xificurC>
the POSTing of data we talked about friday
<xificurC>
let's say I want localhost:8080/upload to accept a tab-delimited file
<xificurC>
then I should (de upload ???)
<xificurC>
how can I tell upload should expect a POST and how can I get my hands on the data being sent
<Regenaxer>
I don't think so
<Regenaxer>
you don't get a stream
<Regenaxer>
I would upload a file, and import that
<Regenaxer>
Not sure atm if there is a direct way
<Regenaxer>
eg wiki/upload.l
<Regenaxer>
I use similar things to import CSV this way
<xificurC>
where is wiki/upload.l?
<Regenaxer>
wiki.tgz
<Regenaxer>
software-lab.de
<Regenaxer>
but these are just files
<Regenaxer>
moment
<Regenaxer>
there is another example we discussed here recently
<xificurC>
looking at it from a high level I would like the process to send an e.g. curl -XPOST --data "a\tb\c" and the server to accept that and put it in the DB
<Regenaxer>
hmm, can't find
<Regenaxer>
But there is osm.tgz
<xificurC>
lost a t in the string, "a\tb\tc"
<Regenaxer>
importes OSM XML files
<Regenaxer>
check software-lab.de/import.l
<Regenaxer>
Imports various CSV formats and charsets
<Regenaxer>
Most text is German though
<Regenaxer>
Perhaps too involved for an example
<Regenaxer>
The OSM demo is also described in an article on picolisp.com
<xificurC>
well osm/import.l seems to show a form that accepts a file to import
<Regenaxer>
yes, also the other import.l
<Regenaxer>
Always the same mechanism, a button which accepts the file and hen imports it
<xificurC>
I don't mind building a simple form if it can be simply automated with 1 curl
<Regenaxer>
Not sure
<Regenaxer>
The '(+Drop +Button) triggers it
<xificurC>
in the end I want the upload to be the equivalent of curl -XPOST ...
<xificurC>
not something one does manually
<Regenaxer>
I never tried
<Regenaxer>
Always import on button press of the user
<Regenaxer>
Or have a cronjob which checks for files and imports them
<xificurC>
the application is for looking through deployments. So whenever someone fires a deployment in Jenkins the script behind it sends me over some information about the deployment (what project, sever, environment etc)
<Regenaxer>
In any case, out of the box the pil server acceps only multipart/form-data in POST
<Regenaxer>
a file attachment is stored in the process-local tmp/ dir
<Regenaxer>
Then the process may do with it whatever it likes
<Regenaxer>
store in a blob, import etc.
<Regenaxer>
So it should definitely work
<Regenaxer>
You need authentication etc. to avoid a security hole
<xificurC>
that should work I guess. curl -F
<beneroth>
xificurC, you can easy program such a backend.
<beneroth>
easy possible with pil.
<xificurC>
beneroth: hi, what do you mean
<beneroth>
the webpage taking a POST with a few arguments to store in DB.
<xificurC>
yeah, I just didn't imagine I need to build an actual page with a form to accept data. Not a big issue though I guess
<Regenaxer>
You need no page
<Regenaxer>
no button etc.
<Regenaxer>
a function or a x.l file are ok
<Regenaxer>
I forgot the detail, but you receive the file name in a variable
<xificurC>
well you said I need to send multipart/form-data. So if I was to send curl -F 'data=1\t2\t3' to localhost:8080/import
<Regenaxer>
and the file itself in (tmp ...)
<Regenaxer>
yes
<Regenaxer>
buw localhost:8080/import.l
<Regenaxer>
or localhost:8080/!import
<beneroth>
you would to have to do that regardless which technology you use. unless you have a database which accepts everything via POST, but then you probably would want to limit the possible DB actions :)
<Regenaxer>
! is for a Lisp function
<xificurC>
doesn't matter, let's say it is a function, so !import. Now what, how do I get to data=1\t2\t3?
<Regenaxer>
If you know the filename, when the function 'upload' finds the file in (tmp "file.name")
<Regenaxer>
otherwise you must also post the name
<xificurC>
you mean function import in this case
<Regenaxer>
as <upload> does in whe browser
<Regenaxer>
yes, 'import'
<beneroth>
Regenaxer, no file is uploaded. the upload is one POST argument with name "data" and value "1\t2\t3", so I understand the example here.
<Regenaxer>
But 'import' is already a function ;)
<Regenaxer>
namespace import
<beneroth>
so the question boils down to: how to access post arguments with form.l webframework? :)
<Regenaxer>
form.l is not needed
<Regenaxer>
only http.l
<beneroth>
T
<xificurC>
well if we are going through a form it would be simpler for now to send env=local;project=foo etc. But if that's not built into the framework I can generate a file
<Regenaxer>
see line 302
<Regenaxer>
there the file is posted to tmp
<Regenaxer>
multipart/form-data
<beneroth>
xificurC, go with POST arguments instead of a file. no need to create and upload a file.
andyjpb has joined #picolisp
<beneroth>
(a file would be no problem)
<Regenaxer>
if the file is already on the same machine, right
<xificurC>
yes but how do I get to the arguments? :)
<Regenaxer>
you post the path to the file in a variable
<xificurC>
I don't have a file, I will create one if necessary, otherwise I would just send the data as is
<Regenaxer>
yes, if same machine, only the path
<Regenaxer>
then it could read a pipe
<xificurC>
not the same machine, and there's no file unless required by pil. Then it would be a temporary file that would be deleted after the curl
<Regenaxer>
The point is you trigger the reading with a GET
<Regenaxer>
Yes, so I would not create a file
<Regenaxer>
open a socket and use rpc
<Regenaxer>
sending the data in a pipe
<beneroth>
its not on the same machine
<beneroth>
he just wants to send a few data fields via HTTP POST and store it in DB.
<beneroth>
no biggie.
<Regenaxer>
no POST
<Regenaxer>
useless
<Regenaxer>
GET
<beneroth>
T, GET is good enough.
<beneroth>
not nice though, using GET to send data, in the spirit of the http methods :P
<Regenaxer>
the sending process can send the data straight via the socket in the same transaction
<Regenaxer>
Normal rpc, I do that a lot
<Regenaxer>
copying whold DBs in case of PilBox apps
<Regenaxer>
beneroth, consider what we do with ssl -> replica
<Regenaxer>
it is the same
<xificurC>
I have no idea how to do that. I would need a sample curl command that does the send and a snippet on the server that reads it and stores it
<Regenaxer>
A GET, sending a stream of binary data
<Regenaxer>
I would use pil instead of curl
<Regenaxer>
or bin/ssl
<Regenaxer>
bin/ssl checks periodically if there is something to send
<Regenaxer>
and transmits it as a single transaction
<Regenaxer>
all or nothing
<xificurC>
can't put pil on the servers
<Regenaxer>
Then curl can surely do it
<Regenaxer>
I don't know curl well enough
<Regenaxer>
A typical receiver is:
<Regenaxer>
(de rpc ()
<Regenaxer>
(when (= *SSLKey (rd))
<Regenaxer>
(while (rd)
<Regenaxer>
(sync)
<Regenaxer>
(tell)
<Regenaxer>
(pr (eval @))
<Regenaxer>
(flush) ) ) )
<xificurC>
but you're saying you would send the data as if by (wr) and have (de import () ... (rd) ...)
<Regenaxer>
instead of (rd) you need binary reads
<beneroth>
I have the impression Regenaxer talks about a much more complex use case than xificurC :)
<Regenaxer>
no, I just don't find a simple example
<xificurC>
feel free to chime in beneroth :)
<beneroth>
haha
<Regenaxer>
if the file is ascii, it would be (while (split (line) "\r")
<Regenaxer>
then (put> ...
<beneroth>
I would recommend xificurC to write his client bash script as he sees fit, and then post the curl call here so we can help him writing the pil backend to receive the http request.
<xificurC>
it's not even a file yet, just a bunch of strings
<Regenaxer>
Yes
<xificurC>
1 request sends 1 object
<Regenaxer>
The problem is curl
<beneroth>
why?
<Regenaxer>
make a GET !rpc
<Regenaxer>
then send a stream
<xificurC>
the object would be (rel env) (rel prj) (rel server)
<Regenaxer>
xificurC, if they are short requests, then just GET with a few args
<xificurC>
and I would need something like curl -F 'env=1;prj=foo;server=12'
<Regenaxer>
I see
<Regenaxer>
yes, no file needed
<xificurC>
it might grow later but this should be adequate for now
<Regenaxer>
you could GET an URL like "!getObj?abc&dez"
<xificurC>
in other languages you usually get your hands on a parameter that holds the form's arguments in a map
<Regenaxer>
(de getObj (Str1 Str2) ...
<Regenaxer>
no parameter names needed
<Regenaxer>
(de getObj (Str Num) ...
<Regenaxer>
"!getObj?abc&+123"
<Regenaxer>
you can directly pass any Lisp atom or simple list
<Regenaxer>
So the whole project really becomes trivial ;)
<xificurC>
so you & them together and encode with the rules described for serializing data in URLs and the functions will get them as arguments
<Regenaxer>
(de getObj (Env Foo Server) ...
<xificurC>
function*
<Regenaxer>
"!getObj?+1&foo&+12"
<xificurC>
that sounds a bit simpler than binary reads :)
<Regenaxer>
(de getObj (Env Prj Server) ...
<Regenaxer>
yes, a lot
<Regenaxer>
if 'getObj' is allowed, it is called with 1 "foo" 12
<xificurC>
it's a bit tricky that it's a GET considering the modern conventions but I don't care too much :)
<Regenaxer>
There is not much difference between a GET and a POST, just
<Regenaxer>
where the args are passed
<xificurC>
here when @lib/user.l is run *Menu is globally defined?
<xificurC>
and can access it via *Menu
<Regenaxer>
yes, see the first line in lib/xhtml.l
<Regenaxer>
used in <menu>
<beneroth>
Regenaxer, strictly speaking GET should only be used for retrieving data, POST for creating or editing, and PUT for creating. strictly speaking, no technical effect.
<Regenaxer>
"should" is some official recommendation?
<beneroth>
so it is in the RFC for HTTP, if I don't misremember strongly
<xificurC>
so ?foo=+1&bar=+2 is equivalent to (setq foo 1) (setq bar 2) whereas ?+1&+2 would send it as arguments?
<Regenaxer>
But GET *always* also sends something :)
<Regenaxer>
xificurC, a little different
<Regenaxer>
?*Foo=+1&*Bar=+2 is like setq
<Regenaxer>
?foo=+1&bar=+2 puts whe values in the http property iirc
<Regenaxer>
yes, 'http'
<Regenaxer>
(put 'foo 'http 1)
<Regenaxer>
Normally I would use a global *Foo
<xificurC>
I see that in _htSet, ok
<Regenaxer>
(allow '*Foo)
<Regenaxer>
yes, _htSet
<xificurC>
what about ?+1&+2 ?
<Regenaxer>
then the values are passed as arguments
<Regenaxer>
both to a function or even a *.l file
<xificurC>
to a .l file it's in (args)?
<Regenaxer>
The file would do (next)
<Regenaxer>
yes
<xificurC>
ok. Not very complicated, just need to learn once :)
<Regenaxer>
yeah
<xificurC>
re security - I was thinking Friday on my way home
<xificurC>
if every session is identified by the port it opens for that session
<xificurC>
isn't it trivial to steal a user's session?
<Regenaxer>
It uses also a long session id
<Regenaxer>
you need to copy/paste it
<xificurC>
or snap it with a camera, ok :)
<Regenaxer>
yes
<xificurC>
still no cookies, all in the URL
<Regenaxer>
or look into the log file, it is recorded there too
<xificurC>
you're not a cookieman
<Regenaxer>
for debugging
<Regenaxer>
right, no *dependence* on cookies, but I use them sometimes
<Regenaxer>
A cookie could be used, but in the URL it is easy to have many sessions in the same browser
<xificurC>
if the GET for a database upload would really irk someone would a POST work just the same?
<Regenaxer>
yes, as I said it is only the place where the params are sent which differs
<Regenaxer>
I think the real difference is only that you use UOST if you have to pass a lot of data
<Regenaxer>
B
<Regenaxer>
Posting a form does *also* return something
<beneroth>
no, conceptionally a GET should never change anything on a server.
<Regenaxer>
A new html form normally
<Regenaxer>
Why not?
rob_w has quit [Remote host closed the connection]
<Regenaxer>
There is no reason
<beneroth>
its a convention.
<Regenaxer>
And you will almost always have side effects
<Regenaxer>
I don't mind
<beneroth>
so you can repeat a GET as often as you like, and also cache the result, because it is not changing something on the server.
<Regenaxer>
In today's servers it will always change something
<beneroth>
you don't have to mind. I just want to point out that this convention exists and that it can be found in the RFC about the HTTP standard, it's not an arbitrary convention only a few people follow.
<Regenaxer>
logs, statistics etc
<Regenaxer>
yeah
<Regenaxer>
POST is a bit more heavy
<Regenaxer>
I prefer GET if it suffices
<beneroth>
well technically POST is not heavier, necessarily :P
<Regenaxer>
in case of pil it is. GET needs no parameter names
<Regenaxer>
And GET even starts a session, a *big* side effect
<beneroth>
neither does POST. thats a requriement from your POST handling in your http.l
<Regenaxer>
true
<Regenaxer>
In pil POST does not work without var=val in the body
<Regenaxer>
only in the URL, like in GET
<beneroth>
your http.l and form.l are optimized for the default case of an ERP session. perfect for that :)
<Regenaxer>
Not just ERP, any kind of interaction
<Regenaxer>
GET gets the page, and POST does button presses
<Regenaxer>
that's all
<Regenaxer>
but all can have side effects
<Regenaxer>
So basically it is as you say
<Regenaxer>
To trigger any action in the server, the pil GUI needs a POST
<Regenaxer>
But as side effect you can do anything with the arguments
<beneroth>
my point is in differencing between the raw possibilities of picolisp and the context of a specific web framework (being the default http.l and form.l)
<beneroth>
you can easily implement some other behaviour with pil.
<Regenaxer>
What are the differences of raw possibilities? I don't know other systems
<Regenaxer>
yes
<beneroth>
and that you can do that is a point worth pointing out, because thats usually not easily done in other such systems.
<beneroth>
even when its obvious from.. well the nature of things ^^
<Regenaxer>
I was surprised to learn that PHP can't handle "val1&val2&val3"
<beneroth>
me too
<xificurC>
I love it how you don't have to care about what the rest of the world is doing in the past 10 years
<Regenaxer>
only var=val
<Regenaxer>
haha, me too :)
<xificurC>
maybe even more
<Regenaxer>
yes
<beneroth>
Regenaxer can be very happy about this :D
<xificurC>
you're like a caveman by now. The funny thing is if you look outside nothing's changed, just people changing their opinions
<Regenaxer>
:)
<xificurC>
wearing blue instead of purple
<beneroth>
but yeah, somehow he has difficulty to relate to people asking for help because he doesn't know (and hardly can imagine) the weird ways something is done "the common way"
<beneroth>
:P
<Regenaxer>
sadly, yes
<xificurC>
that's why you're here beneroth, no?
<Regenaxer>
But I learn it here
<xificurC>
:)
<beneroth>
ah, thats the price for a lot of missed headaches
<xificurC>
well, opinions and standards come and go
<beneroth>
T. Regenaxer only cares about real practical results, and in the end that's the only thing that makes a difference
<xificurC>
yes. I take it as a bonus, only shows the tools are durable
<beneroth>
T
<xificurC>
hype-resistant
<beneroth>
no-nonsense
<xificurC>
should be in the title
<xificurC>
slowly getting to the point where I should start writing code. The scary part!
<Regenaxer>
Most people are not even aware of unnecessary restrictions
<beneroth>
T
<Regenaxer>
yes, go ahead :)
<beneroth>
used to think inside the box, they can't see the box anymore :)
<Regenaxer>
T
<xificurC>
well, you like forth, there's no restrictions there :)
<xificurC>
did you follow what is Chuck Moore doign right now?
<Regenaxer>
not recently
<Regenaxer>
funny things probably
<beneroth>
Regenaxer, and you pointing out them that they live in a box naturally causes distress and push-back ;-) most people are not very good in questioning their own reality tunnels.
<beneroth>
Regenaxer, was there a decisive argument about the VPN/security non-issue, or did you "win" by pure stubborness? :P
<beneroth>
xificurC, most restrictions are not in the nature of the things.
<Regenaxer>
the latter :)
<Regenaxer>
We "agreed"
<beneroth>
okay :)
<Regenaxer>
The final argument was that Let's encrypt needs port 443
<beneroth>
wut? how came Let's Encrypt into this?
<Regenaxer>
And I invented a PUK which is needed for the first remote login
<beneroth>
ah okay
<Regenaxer>
I need certs to connect from outside
<Regenaxer>
avoid man in the middle
<beneroth>
yeah of course.
<beneroth>
ports not being bound to the protocol of the connection is also a fact most company IT people have a hard time to get :)
<Regenaxer>
The VPN would really be impractical, if using arbitrary mobile devices
<Regenaxer>
yes, and at the end of the day IT is glad if they don't have to do all those installations
<beneroth>
oh good xD
<Regenaxer>
So everybody is happy now
<beneroth>
sometimes such people like such needless work because then they don't have to do something they like even less
<Regenaxer>
T
<Regenaxer>
They just need to demonstrate their power, and that they are indisposable
<beneroth>
yeah. the silly thing that it's very stupid, arguably the most sane reaction to someone demonstrating "indispensableness" is to build in some security against it :P
<beneroth>
but well, it's even more stupid to believe employees to the disposable. if they are, they're not made good use of.
<Regenaxer>
T
andyjpb has quit [Ping timeout: 256 seconds]
<xificurC>
liking short identifiers you have one of the longest nicknames Regenaxer
<Regenaxer>
oh, true, "abu" was not available
<Regenaxer>
"Regenaxer" goes back 50 years, I was not programming yet
<Regenaxer>
I was inventing anagrams while being bored in the classroom