<cobax>
What does picolisp mean by "distributed" when it says "creating networks of distributed databases is built into the core as well"? In what ways can it help me simplify my scaling stack tech?
<aw->
cobax hi
<cobax>
aw-: o/
<aw->
the builin database in picolisp can be accessed from remote picolisp clients without any additional work
<aw->
it's like.. 1 line of code
<cobax>
so I can scale the database horizontally by adding more commodity servers with very minimal configuration? and it's only scaling the storage and not the processing?
<aw->
however using the picolisp builtin database is not as easy as just connecting to Redis.. assuming you don't already have PicoLisp code/knowledge
<aw->
yes exactly
<aw->
but it all depends how deep into picolisp you already are
<cobax>
Hmm that is interesting, I wonder, does the technique used to achieve that have a name? Reason I ask is, it seems Picolisp has a good grasp of simplicity, and I really wonder if you have figured a solution for horizontally scaling web apps, but it seems you figured half of that - scaling storage but not processing.
<aw->
not really a name?
<aw->
what are you looking for?
<aw->
it's just a linux process you access over TCP
<aw->
there's nothing magic about it
xkapastel has quit [Quit: Connection closed for inactivity]
<cobax>
If there is no name, I am looking for what is the technique that Picolisp figured out, that seemingly other databases haven't, since scaling every other database is not easy.
<aw->
wait
<aw->
i think you're confusing something as well
<cobax>
I probably am, I am terrible at devops.
<aw->
what does "distributed database" and "horizontal scaling" mean to you?
<aw->
a lot of people tend to misinterpret these things and then set their expectations too high
<cobax>
it means "oh my god, I got too many clients all of a sudden, let me spin up another linode machine, git clone my repo there, run it and add this one line and now my web app is twice as capable"
<aw->
no
<aw->
that is not at all what the picolisp database does
<cobax>
what is the name of what I said, do you know, then?
<aw->
yes it's called fairy tale
<cobax>
I thought that was what was meant by scaling
<aw->
for starters, there's a few things that you need to specify before you begin thinking about "scaling"
<aw->
1) the size of your database
<aw->
2) the number of queries or transactions per second
<aw->
3) your service usage patterns
<aw->
4) your budget
<aw->
there's probably more, but i'll explain what PicoLisp DB can do
<cobax>
Okay, please.
<aw->
ex: your website has 100 unique clients/hour and suddenly you're getting 1000 unique clients/hour, and each client is entirely isolated from the other, but you still sometimes want to gather data on ALL clients or whatever
<aw->
PicoLisp DB can solve that, by letting you run a query on server X, and it will query all the other DBs and fetch the data and then you can process it however you want
<aw->
but it doesn't scale _writes_
<aw->
you **must** make a write to the exact DB server that holds the client info you want to edit
<aw->
it's essentially like sharding
<aw->
so you'll have one database on each server you spin up, but their contents will be different
<aw->
of course you can transfer/sync the database contents to/from all servers.. normally people do that when they want to scale _reads_
<aw->
but you're on your own for that, PicoLisp doesn't have a builtin mechanism to transfer the entire DB from A->B and A->C.. but you can code that easily in a few lines of code, assuming the DB isn't too large
<aw->
not sure if that makes sense
<aw->
in any case, if you're looking to replace an existing "stack" with a picolisp DB, you're probably looking in the wrong place
<cobax>
I think I understand now and I appreciate the explanation. From my understanding those capabilities mirror those of MUMPS, as it scales reads horizontally too but with explicit machine indexing.
orivej has joined #picolisp
<aw->
right
<aw->
now if you want to have way less code to maintain, and much cleaner more efficient code (cmpared to popular languages) then picolisp and its DB are a good fit
<cobax>
I don't mind the explicit machine naming, I think explicit things are better in programming, and I think what picolisp db achieves is a feat in itself.
<aw->
what is MUMPS? i only know the disease
<cobax>
Yes I entirely appreciate the level of simplification that picolisp brings
<cobax>
MUMPS is a programming language from 1966 with an embedded database
<cobax>
it powers half the hospitals in the USA
<aw->
i see
<cobax>
connecting them all to any programmer from the system, so they can read freely from any hospital
<cobax>
and write too
<cobax>
so I was curious how high that stack had picolisp climbed
<aw->
yeah you can write to different databases from one location too, but you'll need to code the functionality yourself
<cobax>
right, I understand that, and I prefer having to implement those things myself
<cobax>
if only I could not just read and write to/from other machines but execute code in them... wouldn't that be scaling but with my having to still code something on top?
<aw->
for the record, i'm not a huge fan of PicoLisp DB because the syntax is strange and complex.. but that's my personal preference
<aw->
others use it quite a lot
<cobax>
Yes the syntax is complex but I usually write my own macros on top of crazy syntaxes
<cobax>
I don't suffer them
<aw->
yes you can transfer direct PicoLisp functions and code from one server to another
<aw->
quite easily
<cobax>
not transfer, run
<cobax>
can I?
<aw->
transfer and run
<aw->
yes
<cobax>
well, then if I can do both those things, how is that not scaling? I mean, sure, I have to code on top, but I want that.
<cobax>
So I can easily code a queue and workers that scale horizontally by just adding more linodes?
<aw->
like i said, many people have different definitions of scaling
<cobax>
and I can scale a web app?
<aw->
yes i've done that and even released code that works exactly that way
<aw->
if you want to code some stuff on your own, then you should get familiar with PLIO for sending arbitrary lisp expressions over TCP
<cobax>
I know nothing about PLIO, can you send me a link?
<aw->
PLIO uses (pr) and (rd)
<aw->
yeah just read the picolisp docs for 'pr and 'rd functions
<aw->
you can combine them with (in *Socket .. to read from a TCP socket and (out *Socket ... to write to it
<aw->
where *Socket is the TCP socket you create with (connect Host Port)
<cobax>
so you use those commands in your picolisp-supervisor?
<aw->
no i use them in the picolisp-kv client/server
<aw->
but the supervisor has some workers which use the client/server code to send/read data to/from remote hosts
<aw->
and that "data" can be code which gets run on the other end
<aw->
it can be anything, the joys of Lisp ;)
<cobax>
So in other words, let me see if I understand this; Picolisp allows me to code such that, with discipline and with some extra effort on my part, I can remain within Picolisp (not have to configure docker, kubernetes, other scaling software) and code a web app that can become more capable easily in both code execution and data storage by adding
<cobax>
servers?
<aw->
yes pretty much
<aw->
but your resource usage will be so much lower that you probably wont need to add servers AHHAHAHAA
<cobax>
That is amazing. And yes look up Snabb Switch, you're right about not needing to add servers, with Lua on bare metal they do 40 million requests per second
<cobax>
I think I only have one last question for now, and I really appreciate all your clarifications
<cobax>
"and that "data" can be code which gets run on the other end" -> can you directly call a function on another machine? If so, how?
<aw->
no prob
<cobax>
As in, could I start a clean-up worker on every machine, with some sort of crontab function that I coded on just one machine?
<aw->
yes you can, but the other end needs to be setup to receive code and execute it
<cobax>
ahhh...
<aw->
yes, you're essentially opening a backdoor to your server by doing that
<aw->
because it will run _anything_ you send it
<cobax>
okay, I get that. That is entirely reasonable. So I could set it up with my own password system or otherwise my own security system and that would be on me.
<aw->
yes exactly
<aw->
that's what i did in my picolisp-kv database
<aw->
there's a few layers of security to add on top of it
<aw->
picolisp only cares about the core essential functionality
<cobax>
That is amazing. Man, you have helped me a lot tonight. I will be looking over your code to learn from it.
<cobax>
Picolisp has made so many good decisions.
<aw->
yes it has
<cobax>
Only caring about the core essential functionality is what I want from a language.
<aw->
ok great to hear, there's lots of other great picolisp code out there, search around
<aw->
come back when you have more questions
<aw->
in a few hours Regenaxer will be here, he's the picolisp author
<aw->
he knows everything
<aw->
and answers everything
<cobax>
Dr. Burger, is that right?
<aw->
yes, great guy, very helpful and friendly
<cobax>
Yes he is very helpful indeed, I have chatted with him before.
<aw->
oh great
<aw->
btw all my picolisp code is MIT licensed, feel free to rip it out and use what you need
<cobax>
Thank you, I will come back when I have more questions. Thank you so much, once again. I am excited about learning Picolisp! You're like me, I also prefer MIT and to share as much as possible.
<aw->
or use it as inspiration, read the codes, combine how you want
<aw->
great!
<cobax>
You and I know the best ideas are never widely adopted so we always have the competitive edge anyways ;)
<cobax>
Thank you and enjoy the rest of your day!
<tankf33der>
morning all
<tankf33der>
bought and setup my first vps, from hetzner
mtsd has joined #picolisp
<Regenaxer>
Good morning all! Wow, long discussion :)
<Regenaxer>
Thanks aw-! You did a wonderful way of explaining!
<tankf33der>
May 6 05:19:38 delta mail.info smtpd[21393]: 44c1fb0ff74a53a6 smtp bad-input result="500 5.5.1 Invalid command: Pipelining not supported"