<beneroth>
maybe because he has a new one now for a few years? :)
<beneroth>
(but that is about AI & ML, not lisp)
<beneroth>
aw-, thx for the hint at NPM, I had a good laugh
<beneroth>
its unbelievable
<aw->
hey bene
aw- has quit [Quit: Leaving.]
shpx has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
shpx has joined #picolisp
shpx has quit [Client Quit]
orivej has joined #picolisp
orivej has quit [Quit: No Ping reply in 180 seconds.]
orivej has joined #picolisp
orivej has quit [Ping timeout: 268 seconds]
orivej has joined #picolisp
orivej has quit [Ping timeout: 240 seconds]
orivej has joined #picolisp
orivej has quit [Ping timeout: 256 seconds]
<beneroth>
Regenaxer, does rel (+List +Blob) work? (not tested yet.) looking at the implementation, I can't see how.
rob_w has quit [Quit: Leaving]
<beneroth>
didn't find anything about that topic in ML
<beneroth>
I guess its not possible - so one should do a (+List +Link) instead
<Regenaxer>
ret
<beneroth>
welcome back Regenaxer
<Regenaxer>
No, this won't work, because the blob is named like the relation
<beneroth>
aye
<Regenaxer>
(rel pdf (+Blob)
<Regenaxer>
gives <path>.pdf
<Regenaxer>
I make special +Doc objects
<beneroth>
so (rel pdfs (+List +Link) (+PDF)) (class +PDF) (rel file (+Blob)) is the right way?
<Regenaxer>
they have also other attrs
<Regenaxer>
yep
<beneroth>
alright, got it
<Regenaxer>
The wiki doest that too iirc
<beneroth>
afaik yes
<Regenaxer>
(class +Bin +Entity)
<Regenaxer>
(rel nm (+String)) # Name
<Regenaxer>
(rel mt (+String)) # Mime type
<Regenaxer>
(rel d (+Blob)) # Binary data
<Regenaxer>
(when (== X 'd)
<Regenaxer>
(dm upd> (X Old)
<Regenaxer>
(put> This 'mt (mimetype (blob This 'd))) ) )
<Regenaxer>
Such things
<Regenaxer>
(rel bin (+List +Link) (+Bin)) # Binaries
<beneroth>
if one doesn't expect malicious input, one could even deduplicate blob data this way by using a md5 hash of the blob file. re-using already stored blobs, when available
<Regenaxer>
yes
freemint__ has joined #picolisp
<freemint__>
Hi
<Regenaxer>
Hi freemint__
<beneroth>
hey freemint__
<freemint__>
Hi Regenaxer beneroth
<freemint__>
How did it go yesterday?
rob_w has joined #picolisp
<freemint__>
(Your presentation of the chat derived thing)
<Regenaxer>
Not presentation, just discussion of concepts
<Regenaxer>
Went good, I have the prototype running
<Regenaxer>
:)
<freemint__>
What did they say?
<freemint__>
(roughly)
<Regenaxer>
Good
<Regenaxer>
But I ended up not using SSE
<Regenaxer>
It turned out that a simple POST is enough
<Regenaxer>
There are no asynchronous messages popping up in the background
<freemint__>
Will it end up in picoLisp.tar.gz anyway?
<Regenaxer>
It is already
<Regenaxer>
I released immediately
<freemint__>
I will make a presentation about picolisp to FeM.
<Regenaxer>
Great, very nice!
<freemint__>
But i have no concept what i want to mention/do yet
<Regenaxer>
oh :)
<freemint__>
I would defenilty put the database to good use and do things they can not deny are usefull but only lisp can do
<freemint__>
(like having code in the database or something.
<Regenaxer>
Like equivalence code <-> data
<freemint__>
Yeah
<freemint__>
Does something come to mind ?
<freemint__>
If the end product is usefull all the better
<Regenaxer>
You could scan the rosettacode solutions to look for ideas perhaps
<freemint__>
nice tipp
<freemint__>
Is there a web spider or something like that in picolisp already?
<freemint__>
or an html parser?
<Regenaxer>
there is lib/scrape.l
<freemint__>
I was encouraged to write an search engine for their mailing list archive
<Regenaxer>
scraping the gui
<freemint__>
not the gui but arbitrary html
<Regenaxer>
there is a wiki article about scape too
<Regenaxer>
The gui *is* just html
<Regenaxer>
js is ignored here
<Regenaxer>
in rosetta are also several scraping tasks
<freemint__>
ok ... do you have other ideas which would highlight the power of picolisp better?
<Regenaxer>
hmm, hard for me because I'm too long into it
<freemint__>
Can you imagine an useful appliance for code in the database?
<Regenaxer>
I never had the need
<Regenaxer>
Source files are all right ;)
<Regenaxer>
But you can of course easily store expressions in relations
<Regenaxer>
(rel code (+Any))
freemint__ has quit [Ping timeout: 260 seconds]
freemint has joined #picolisp
<freemint>
when did you put pilog rules in the db and why?
<freemint>
and noone linked a library either ... i see
<freemint>
seems like Regenaxer has not needed it yet
<beneroth>
btw. I have a point for your database pitch
<freemint>
tell me
<beneroth>
the database is kinda graph.. but better than what is traditional understood under graph (afaik), as it also supports relation indexes similar to mainstream SQL databases
<beneroth>
graph databases can do JOINs much better than relational databases
<beneroth>
less costly for graph databases
<beneroth>
so graph database architecture is of benefit whenever your data has a graph structure (trees, nesting, etc)
<freemint>
i know that ... but i guess that a well tweaked sql will be picolisp still
<freemint>
so what is the selling point?
<beneroth>
what do you mean?
<beneroth>
I don't think you can achieve picolisp features on top of traditional sql... not without heavy cost on maintainability and probably on performance too
<beneroth>
application programming wise pil DB is OOP, no ORM framework required
<freemint>
ok i think you are right but that is hard to demo
<beneroth>
and pilog (aka having a rule engine to do DB queries) is a big feature, too
<beneroth>
T
<freemint>
I need something "fassbares"
<freemint>
"greifbares"
<beneroth>
well, what does your client/audience care about?
<beneroth>
that they can maintain it when you are gone (safety for them) ? then pil-based is probably not for them, doesn't matter what the technical qualities are
<freemint>
either something cool, something usefull,
<beneroth>
do they not so much care about that but about having a very flexible powerful (graph) DB ?
<freemint>
They would, but it does not make for a nice presentation either
<beneroth>
I would say flexibility (ability to re-arrange things after initial development, during maintaining phase) is the biggest feature of pil DB and picolisp apps in general
<beneroth>
this + quick development time (arguably requires some level of picolisp skill by the developer(s), though comparable to what is required from a dev to be productive in another tech)
<beneroth>
maybe don't try to sell them the full package at first
<beneroth>
propose them to make a prototype with picolisp, if you like
<freemint>
this are things that only lead to akward question "Why have not you implemented anything usefull yet"
<beneroth>
which things lead to this question?
<freemint>
i do not need to propose to make a prototype i want to to have prototype to show
<freemint>
to discuss how it went
<beneroth>
whats holding you back?
<freemint>
not having a good idea
<beneroth>
because you don't understand the task fully? or because you don't understand the possibilities?
<beneroth>
this is not a critic
<freemint>
(scraping the mail server and building an search index failed because no cookie handling and no https
<beneroth>
understanding the task is the key, the hard part
<freemint>
cookie handling in curl will only get more ugly
<freemint>
yeah
<beneroth>
why do you get the emails via web? IMAP?
<freemint>
mail archive
<freemint>
in Mailman
<beneroth>
public, but behind authentication? or internal?
<beneroth>
ah
<freemint>
internal but port 80 gives a redirect to https
<beneroth>
GNU mailman?
<freemint>
yeah
<beneroth>
well I guess there would be an easier & faster way to get the mails.
<freemint>
was setup many years ago and has no search function
<freemint>
which is?
<freemint>
parse the server format?
<beneroth>
can't you just talk the mailman server admin and getting a copy (don't has to be a full one to make a proof of concept even) on a usb stick or so?
<freemint>
i could
<beneroth>
it's probably just a text file format
<freemint>
mhh ok
<beneroth>
copying everything individually by HTTPS is a lot of network traffic which much overhead
<beneroth>
you can already work on other parts until you have the data, at least partially
<beneroth>
you can develop the entities & relations (what do you store, what do you index how...)
<freemint>
how can i build features which can be only implemtented in picolisp
<freemint>
What features would you want from a mail archive of a huge hacker space?
<yumaikas>
freemint: You don't. You build features that are *easiest* to build in picolisp
<beneroth>
everything which can be implemented in a turing machine can be implemented in every turing-complete programming language
<beneroth>
exactly, what yumaikas just said
<freemint>
beneroth, nitpcking
<beneroth>
important nit, I find
<yumaikas>
freemint: Also, if this is for a hackerspace, why does it matter which language you use?
<beneroth>
if you know another way to do this task easily, don't use picolisp
<freemint>
which i agree with
<freemint>
beneroth, I do not . i want to get a project outthere in picolisp
<beneroth>
unless you want, but then its not technical reasons :)
<beneroth>
ok
<freemint>
which things would you want from a mail archive?
<yumaikas>
Mail associated to users, for starters
<yumaikas>
Mail sent by date
<yumaikas>
A way to view an entire thread in one page
<freemint>
i thought about per user too
<beneroth>
so I guess you want to have a list of +Email addresses, +Aux date time of the emails (maybe even multiple datetimes, sending time and receive time etc), and of course a text index of the content
<beneroth>
yeah
<yumaikas>
freemint: But why do you need to sell picolisp as the language for the project?
<freemint>
yumaikas, i do not need to "sell picolisp"
<freemint>
if i want the project i can have it and do it how i want
<beneroth>
think about what you want to store and what not. think about what you want to query (what do you want to find out?) - this influences what you want to store and (more importantly) how you want to store it
<yumaikas>
Ah, ok
<beneroth>
so you are looking for reasons to convince yourself?
<beneroth>
:)
<freemint>
myself ... and making a battle plan
<beneroth>
I can understand you. was the same. then I just went ahead. turned out to be happy with this way
<beneroth>
well the battle plan will look roughly the same independent of tech stack choice.
<freemint>
What about email dedupplication if they were sent to multiple lists
<freemint>
a quoting from other email graph?
<beneroth>
I wouldn't deduplicate whole emails, but maybe chunks of content
<freemint>
beneroth, may the wording was bad
<beneroth>
no worries
<freemint>
something like "this email was also seen on this mailing list"
<beneroth>
well you chose the parts of the email which are the "same" on both lists when it is the same email (there are surely some data that is different, this one you want to leave out of the comparison), and you want to make a hash from this data
<beneroth>
this hash you put in an +Ref index (or whatever the equivalent is on the stack you use)
<beneroth>
then you can query for that hash to find all associated emails, which are associated with a mailing list
<freemint>
I am against hashing it is very error prone
<beneroth>
how so?
<beneroth>
hashing is like everything with computers: all about the input.
<freemint>
an email client changes the linefeed order -> other hash
<beneroth>
well, then you have to remove the linefeed order from the hash input
<beneroth>
same email must generate same hash
<beneroth>
so you have to look at test data and identify what constitutes "sameness"
<beneroth>
you want a function to produce the same input for the hash function from the same emails
<beneroth>
this hash input has already to be equal
<freemint>
I won't go with hashes
<beneroth>
you only use the hash so that you don't have to store so much in the db index
<freemint>
reason 1:
<freemint>
It is all text
<beneroth>
what is all text?
<yumaikas>
freemint: For this, I would use hashes
<yumaikas>
Keep the text around so you can resolve collisions
<freemint>
reason 2: there are obvious heuristics
<freemint>
reason 3: language is relativly low entropy
<beneroth>
you can represent a hash as text, but as everything in computers, you can also represent it as an integer number.
<freemint>
reason 4: the dataset fits in RAM
<yumaikas>
but use the hashes to keep from going O(N^2) in the deduping process
<beneroth>
make it without hash then first
<freemint>
yumaikas, I do think that will be a problem
<beneroth>
you can always add it later when you see its use
<freemint>
beneroth, yeah
<yumaikas>
Do or don't thing N^2 will be a problem?
<freemint>
yumaikas, No it will not because reasonable heuristcs cut it down so fast...
<yumaikas>
*think
<yumaikas>
ok
<freemint>
You can make a N day radius for de-dup and you got almost all duplications (for let's say N = 7)
<freemint>
Does anything speak against keeping a full text index?
<freemint>
Do you know in which lambdau full text indexes scale?
<beneroth>
just start coding. you are more likely to overthink stuff in advance then to not think enough in advance. the real problems you will find underway, and so the real solutions.
<freemint>
I want to get a better sample of the possiblity space before i start
<freemint>
(visual stuff) of what?
<beneroth>
try not to blow this too big at first, it will grow anyway. try to complete a minimal valuable product (mvp) first.
<freemint>
T
<beneroth>
e.g. relations betweens lists / email addresses
<beneroth>
or topics
<freemint>
here is a claim: to get a good seed (MVP) you need to have good idea where you want to go.
<freemint>
and i want to get your opinions since i value them
<freemint>
Let's say i end up workin on it till 2020.
<freemint>
What should you expect of an email archive written in 2020
<yumaikas>
freemint: What are the 3 biggest problems you have with the current email archive? Solve those first
<yumaikas>
Start by solving specific problems, and then go from there
<beneroth>
yep, start small. once you finished the fundament, you can easily grow on top of it
<freemint>
i want to ask you what problems you see
<beneroth>
this is opinion gained from experience
<freemint>
i am asking your opinion since you have more exp
<yumaikas>
freemint: I don't use this mailing archive, so I don't know
<freemint>
yumaikas, why don't you
<yumaikas>
freemint: First of all, I don't know which one you're trying to add stuff to
<beneroth>
Perfection is attained
<beneroth>
but when there is nothing left to take away.
<beneroth>
not when there is nothing left to add
<yumaikas>
Second of all, I generally use lobsters/reddit/IRC for loose discussion, not email
<freemint>
What makes them not a good source of information even though the most bright people in the world commented on mailinglits
<beneroth>
keep it simple.
<yumaikas>
freemint: Most mailing archives are very much shop talk
<freemint>
yumaikas, what does reddit offer what mailing lists can not
<beneroth>
T, though I like mailing lists.
<freemint>
Is the social difference enforced by technical difference
<beneroth>
don't value things against each other. take the best of everything.
<yumaikas>
A better UI, a different community?
<yumaikas>
Eh, sometimes comparison of contrast can be good to
<yumaikas>
*too
<yumaikas>
freemint: I recently wrote a wiki (in Erlang, not picolisp) at idea.junglecoder.com
<freemint>
Do you see technical differences between reddit and email which lead to a difference in user behavior?
<freemint>
Why are email archives not as fun to browse as reddit is?
<freemint>
(or hackernews)
<freemint>
(Do i ask stupid questions?)
<freemint>
Am i being ignored now
<yumaikas>
That was a project born for 2 goals: Get a list of ideas out of my head, and make a place where I can record stuff I find interesting
<yumaikas>
freemint: No, I'm just a bit busy elsewhere as well
<freemint>
cool
<freemint>
and ben.....?
<yumaikas>
freemint: But, it's been rather interesting as a wiki, and now that I've been using it for about 3-4 weeks, I've got a more ideas for what to do with it than time to implement them
<freemint>
yumaikas, That is my default state
<yumaikas>
(And the wiki has also given me a place to dump ideas into for later perusal)
<freemint>
are you interested in to accouting?
<yumaikas>
A little bit
<yumaikas>
The notion of double-entry bookkeeping seems interesting
<freemint>
If you are in it for curius you might want to look at throughput accounting
<yumaikas>
We'll seee
<yumaikas>
*see
orivej has joined #picolisp
<beneroth>
freemint, I was shortly away
<freemint>
i see
<freemint>
when you have some thoughts on my question tell me
<beneroth>
I will have a look at throughput accounting, thx for the tip
<beneroth>
I find good that you ask fundamental questions
<beneroth>
but I don't have much of an answer right now.. I'm not a user really of reddit. mailing lists I only read the picolisp one regularly, only sometimes others. HN I visit irregularly.
<beneroth>
I don't think the stuff is that much different, the difference lies in shape and presentation. which is a huge difference from a UX/psychological perspective. not so big from data storage perspective, I find.
<beneroth>
sorry, all in all I'm also rather busy right now.
<beneroth>
my biggest advice is: try to keep it simple. start small, only add whats required for your task. attempt to fully understand your task (what you want to achieve). limit yourself.
<beneroth>
the meaning is not to put restrictions on yourself to keep you from doing something big. but its very easy to dream something big up without never realising it. also makes one afraid of the task, if it becomes to big. better start small. its also easier to adapt and grow when doing only small things, so the act of developing/changing/growing is easier, so the thing is more likely to actually become big in reality.
<beneroth>
so is my experience and so is my current approach to software development. it's hard. I still need to become better at trying to do less before trying to do more.