rob_w changed the topic of #picolisp to: PicoLisp language | Channel Log: | Check also for more information || Attention due to latest spam floods this channel will only allow registered users to send messages - check
razzy has quit [Ping timeout: 256 seconds]
razzy has joined #picolisp
pierpa has quit [Quit: Page closed]
pierpal has joined #picolisp
arlen has joined #picolisp
arlen has quit [Remote host closed the connection]
orivej has quit [Ping timeout: 256 seconds]
_whitelogger has joined #picolisp
<Regenaxer> razzy: ulimit -s unlimited
<Regenaxer> But I'm not sure if your problem is really the stack
<Regenaxer> If ulimit is unlimited, the stack grows until it swaps
<Regenaxer> if not, you get either a segfault if it overflows
<Regenaxer> or an explicit error if you are in coroutines (I don't expect you do)
_whitelogger has joined #picolisp
pierpal has quit [Quit: Poof]
pierpal has joined #picolisp
pierpal has quit [Read error: Connection reset by peer]
<razzy> no error, 100percent processor for several hours, 20percent of memory, no swap. (i think normally, stack goes to swapping yes?)
<razzy> if necesssary
<Regenaxer> stack goes to swapping only if ulimit is unlimited
<Regenaxer> Why should it be the stack? Is the program very recursive?
<Regenaxer> I rather think you got into an infinite loop
<razzy> it depends on data
<razzy> is it possible to read chars that have influnce on program flow?
<Regenaxer> I don't think so
<Regenaxer> Did you properly debug? Should be easy
<Regenaxer> Learn 'trace' and 'debug'
<razzy> do not know how, yet
<razzy> if everything is evaluated, the data should too
<Regenaxer> I would not start to program in pil without. It is *the* advantage of having a REPL
<Regenaxer> Not everything is evaluated
<Regenaxer> So 'trace' your function and whatch what they do!
<razzy> aye
<Regenaxer> Concerning evluation, during reading only read macros are evaluated
<Regenaxer> ie 'read'
<Regenaxer> 'char' or 'line' etc do not evaluate anything
<razzy> that might be isue :]
<Regenaxer> ok, I see, then '`' or '~' trigger them
<Regenaxer> 'read' is used only for reading s-expressions
<Regenaxer> numbers, symbols or lists
<Regenaxer> raw data are normally parsed on a lower level
<razzy> is there stream library for picolisp?
<Regenaxer> For which purpose?
<razzy> manipulating stream of data?
<razzy> binary utf8?
<Regenaxer> Why you need a library for that?
<Regenaxer> i/o is always a stream in pil
<Regenaxer> 'rd' can read on the byte-level
<razzy> when i use read, it read whole file, yes?
<Regenaxer> What else should a lib do? In general, in pil you can often do directly where you need a lib in other langs
<Regenaxer> no
<razzy> Regenaxer: how do i pause stream from rd?
<Regenaxer> 'read' reads *one* s-expr
<Regenaxer> pause? By not reading ;)
<razzy> that easy?
<Regenaxer> Maybe I did not understand the question
<Regenaxer> (rd 1) reads only a single byte
<Regenaxer> (char) reads a character, 1 .. 3 bytes only
<razzy> utf8 has even 4 bytes char
<Regenaxer> yes, or 5
<Regenaxer> but pil limits to 16 bit unicode
<Regenaxer> historically
<razzy> dammed
<Regenaxer> you need such chars?
<razzy> maybe
<razzy> open question so far
<Regenaxer> I did it like Java back then
<Regenaxer> Could be extended, but is a lot of places to change
<razzy> dammed,... java is poisoned
<Regenaxer> indeed
<Regenaxer> It was the first lang I saw with UTF-8 support in the mid-90s
<razzy> will picolisp be erasing or changing core functions?
<razzy> it is small enough to change
<Regenaxer> Seldom
<razzy> small size will atract more people :]
<Regenaxer> right
<Regenaxer> But I see no reason atm to remove any core function
<razzy> maybe erase/change 1 core function per year? on christmass?
<Regenaxer> would not change the size much
<Regenaxer> haha, ok
<Regenaxer> Changes happen, possibly in a backwards compatible way
<Regenaxer> and bug fixes
<razzy> and choose the most problematic?
<Regenaxer> Which ones are problematic?
<Regenaxer> I would not remove any
<Regenaxer> I use them all somewhere
<razzy> char should read UTF8 char
<razzy> for first :]
<Regenaxer> It does
<razzy> not if it is 5bytes long
<Regenaxer> right
<Regenaxer> But I don't want chars above 16 bits
<razzy> it will mess whole encoding
<Regenaxer> would break other places
<Regenaxer> Must go, see you!
<Regenaxer> afp
<razzy> you could make it adaptible
<razzy> and backwards compliant
<razzy> Regenaxer: if your code is on several places, you did not written lisp enough code
erkin has joined #picolisp
Nazca has joined #picolisp
orivej has joined #picolisp
Nazca has quit [Remote host closed the connection]
pierpal has joined #picolisp
<Regenaxer> ret
<Regenaxer> No comment
anderson9 has joined #picolisp
anderson9 has quit [Remote host closed the connection]
michagogo12 has joined #picolisp
michagogo12 has quit [Remote host closed the connection]
pierpal has quit [Read error: Connection reset by peer]
pierpal has joined #picolisp
<razzy> sorry :]
orivej has quit [Ping timeout: 272 seconds]
orivej has joined #picolisp
orivej has quit [Ping timeout: 256 seconds]
orivej has joined #picolisp
freemint has joined #picolisp
<freemint> Hello
NyanCat27 has joined #picolisp
NyanCat27 has quit [Ping timeout: 244 seconds]
haza-w14 has joined #picolisp
haza-w14 has quit [Killed (Sigyn (Spam is off topic on freenode.))]
orivej has quit [Ping timeout: 256 seconds]
<freemint> Guten Abend
pierpal has quit [Quit: Poof]
pierpal has joined #picolisp
<razzy> hello :]
pmn has joined #picolisp
pmn has quit [K-Lined]
orivej has joined #picolisp
orivej has quit [Ping timeout: 248 seconds]
<freemint> hello razzy do you have any questions remaining?
<razzy> well yes :] freemint a lot of them
<freemint> feel free to ask them if you can write them down
<razzy> what is output of echo,.. is it stream?
<razzy> it does not behave as any stream i encounter yet
pierpal has quit [Ping timeout: 240 seconds]
<Regenaxer> I would not call it a stream
<Regenaxer> It simply echoes the input channel to the output channel
<Regenaxer> I would not call it a stream because I have no idea what a stream means
<razzy> is it a string?
karstensrage1 has joined #picolisp
karstensrage1 has quit [Remote host closed the connection]
alexshendi has left #picolisp [#picolisp]
<razzy> echo does interesting thing, it fowarding data before it stopped reading
<razzy> how do i dissassemble it?
<Regenaxer> : (vi 'echo)
<Regenaxer> It really does only echo
<Regenaxer> with some specs like start/end or stop pattern
<Regenaxer> No, not what I meant
<razzy> uhh, 4 days in, and still do not have proper compresser :]
<Regenaxer> Why not use an existing one?
<Regenaxer> I would not roll my own in Lisp
<razzy> but i am close i think :]
<razzy> Regenaxer: what would you roll in?
<Regenaxer> I would use 'pipe' to call some good one
<Regenaxer> bzip?
<Regenaxer> I have no preferrences
<razzy> i want custom one :]
<Regenaxer> ok
<Regenaxer> tankf33der knows better about such algos
<Regenaxer> But in an application I would surely just call something existent and proven
<tankf33der> T
<Regenaxer> (pipe (exec "bzip" ...
<Regenaxer> Hi tankf33der
<tankf33der> or compression library
<tankf33der> good evening
<Regenaxer> right
<Regenaxer> :)
<razzy> picolisp has compression library?
<Regenaxer> He meant C libs
<Regenaxer> no need in pil
<razzy> it is not hard to test compression algorithms :]
<Regenaxer> true, at least easier than encryption
<razzy> compression and encryption are very similiar data transformations
<razzy> what is hard is to test quality of encryption
<tankf33der> start point
<Regenaxer> the math
<Regenaxer> and to be sure not to have security holes
<Regenaxer> all no issue with compression
<razzy> tankf33der: well 10 years of standard human reading
<razzy> thx :]
<razzy> on second thought. not very well documented
<razzy> i know what i want to do, i just need to write it :]
<Regenaxer> perfect ;)
<tankf33der> razzy do you a pseudocode?
<tankf33der> do you have a pseudocode?
<tankf33der> razzy: looks like you fan of silicon valley tv series
<tankf33der> razzy: you know there is also compression algo in the script, right?
<razzy> i feel that there should by way less core functions in picolisp. there should be lik 50 of them and the rest in separate library :]
<razzy> it would ease me a start :]
<Regenaxer> Would be possible, but I see no advantage
<razzy> tankf33der: i do have pseudocode in my papers:]
<Regenaxer> ignore funs you don't neet atm
<Regenaxer> need
<razzy> Regenaxer: the non-essentials are making noise to cut through
<Regenaxer> But then the libs would too
<tankf33der> writing compression as first code on new language is extreme like base jumping
<Regenaxer> Everything in the base system is essential
<Regenaxer> And the base is easy to browse with (doc 'doc)
<razzy> tankf33der: i jumped long time ago :]
<tankf33der> razzy: ok then
<razzy> now it is just fallout
<razzy> tankf33der: problem is, i keep changing my pseudocode to fit what i can do in picolisp
<tankf33der> this is not a proble
<tankf33der> show pseudocode
<razzy> tankf33der: working on it
<tankf33der> ok
freemint has quit [Ping timeout: 240 seconds]
<razzy> tankf33der: well for start i picked up this. pay atention to first commented part.
freemint has joined #picolisp
<freemint> back
Lumpi7 has joined #picolisp
Lumpi7 has quit [Remote host closed the connection]
<freemint> i am working on my first database backed program. Just testing the water so far. It will run from console.
<freemint> Is there another way to make entities other than 'obj?
<Regenaxer> yes, 'new' mainly, and 'request'
<freemint> I am unsure what way to go here, i ask when i am done reading the relevant docs
<Regenaxer> 'obj' is only a convenience function for constant initialization
<Regenaxer> Internally it all boils down to 'new' anyway
<freemint> Should i modify the 'T method?
<Regenaxer> Normally not modify, but override
<Regenaxer> you mean of +Entity, right?
<freemint> let me specify
<freemint> I want to create an own user login system.
<freemint> Each user has name, hash and pepper
<razzy> tankf33der: better explained how to count strings in stream file
<freemint> the pepper is randomly generated each time a new password is set and the hash is set to password concatenated with pepper. The create user function takes name and password but stored, name, hash and pepper
<freemint> how would i go about that?
<razzy> tankf33der: basicly frequency analysis of text file
<Regenaxer> Why is the existing one not enough?
<Regenaxer> With "pepper" you mean "salt"?
pierpal has joined #picolisp
<freemint> The existing one is good, but i want my one as a little project to get going with database
<freemint> pepper is crypto term for per user salt
<Regenaxer> ok, good
<razzy> tankf33der: will you help me?
<Regenaxer> freemint, I see, never heard
<razzy> i like the terminus technicus
<freemint> the idea of per user salt (pepper) is to have the attacker not reuse the hash result between users
<tankf33der> razzy: i will help you only with picolisp
<tankf33der> gmt+2 working hours
<tankf33der> bed
<Regenaxer> tankf33der, so Riga is in the same zone as Berlin? I thought one more east
<razzy> tankf33der: do you understand second pseudocode? or should i work it differently?
<tankf33der> berlen -1 hour
<freemint> Regenaxer how would i create such user? If i were to do it with normal objects i would write an own T method.
<Regenaxer> You create a random pepper? Then T is fine
<Regenaxer> Look eg wiki/er.l
<Regenaxer> in +User T
<freemint> But database confuses me for 2 reasons. Whether i should use the constructor is unclear and how to fill the database is a little unclear too. new does not take attributes
<Regenaxer> (dm T @
<Regenaxer> (pass super 'ca (date) 'new T) )
<freemint> let me check
<freemint> what is the @ for in this example?
<Regenaxer> I do not call the T method a constructor, it is rather an initializer
<Regenaxer> @ is for more arguments
<Regenaxer> (new! '(+User) 'nm "Hugo" ...
<Regenaxer> Each T in subclasses usually adds more properties
<freemint> so this creates an Timestamp and all arguments are thrown away?
<Regenaxer> Why thrown away?
<Regenaxer> it *adds* a timestamp and a "new" flag
<freemint> oh i did not knew what pass does
<Regenaxer> yes, passing the args
<freemint> that makes sense
<Regenaxer> tankf33der, so you are GMT+3
<freemint> Will be going to FrosCon this year
<freemint> i might give a lightning talk about pil
<Regenaxer> cool!
<Regenaxer> In St.Augustin again?
<freemint> yes
<freemint> What happens if set an already existing value in a key?
<freemint> is checking for username uniqueness a think that should happen in the constructor ?
<Regenaxer> Then you have changed the key
<Regenaxer> I would not check it manually, but rely on +Key
<Regenaxer> mis> will check
<Regenaxer> automatically in the gui
<freemint> i go without gui for now
<Regenaxer> Then call mis>
<freemint> what does mis> do normally
<Regenaxer> It checks for any mismatch
<Regenaxer> eg data type
<freemint> on what would i call miss?
<Regenaxer> Before you set a value
<Regenaxer> I never called it myself iirc
<Regenaxer> only gui
<freemint> ok
<Regenaxer> But I think better use 'request' or 'obj'
<Regenaxer> you don't get double keys then
<Regenaxer> instead it uses the existing object
<freemint> how do they react to a double key?
<freemint> ahh
<Regenaxer> 'request' is typical for imports
<freemint> so i write the T method ignorant of this constraint?
<Regenaxer> yes
<freemint> what do you mean by imports?
<Regenaxer> Importing data, normally CSV
<Regenaxer> excel or other DBs
<freemint> ok, why wouldn't you use it for normal data creation?
<freemint> (like on a web server?)
<Regenaxer> If you *know* it is new you don't need 'request'
<Regenaxer> eg a "New" button
<Regenaxer> then normally an empy object is created
<Regenaxer> or with some defaults
<Regenaxer> Then when the user types a name 'mis>' complain
<Regenaxer> s
<freemint> ok
<freemint> my code looks like this
<Regenaxer> So in the gui objects are empty initially, but an import will hit existing ones
<freemint> (dm T (nm pw . @ ) (let R (rand) (pass super 'nm nm 'ppr R 'pw (hash (cons pw R)))))
<freemint> other than it being very hacky . Is that fine?
<Regenaxer> Better Nm and Pw
<Regenaxer> otherwise fine I think
<freemint> so i would create a new user with (new! '(+User) "Mark" "BattreyHorseStapple" )
<freemint> ?
<Regenaxer> right
<freemint> how can i check whether there already is a mark who is an user?
<Regenaxer> 'mis>', or just 'db' or 'fetch'
<Regenaxer> (when (db 'nm '+User "Mark") ...
<freemint> thanks
<Regenaxer> (when (fetch '(nm . +User) "Mark")
<Regenaxer> the latter for +Key
<freemint> these things are obvious once you did them a few times
<Regenaxer> 'db' works also for non-unique
<Regenaxer> indeed
<freemint> also you definitly want to use a if
<Regenaxer> The question is what to do if "Mark" exists
<Regenaxer> use the existing one?
<Regenaxer> then (request ... is better
<freemint> *confused*
<Regenaxer> (request '(+User) 'nm "Mark")
<freemint> that did not confuse me
<Regenaxer> Then you are always unique
<Regenaxer> The GUI complains if "Mark" exists, so without GUI you must be sure *beforehand* that the data are good
<Regenaxer> eg when importing from a CSV
<Regenaxer> you can't give an error then
<Regenaxer> So I always use 'request' in imports
<freemint> i will understand this another day
<Regenaxer> yes, no big thing
<Regenaxer> Just experiment
orivej has joined #picolisp
<Regenaxer> Getting sleepy
<Regenaxer> I'm off for today
<freemint> good night
<Regenaxer> Good night!
<Regenaxer> afp
<freemint> good night
<freemint> oh it works so wonderful
erkin has quit [Read error: Connection reset by peer]
Sveta17 has joined #picolisp
Sveta17 has quit [Remote host closed the connection]
<freemint> razzy do you have an idea how you want to compress?
<razzy> freemint: yes very much
<razzy> freemint: want to help me?
<freemint> can you explain your basic algorithm?
<razzy> something between hofman and lzw with some deep learning heuristic
pierpa has joined #picolisp
<razzy> but for start, i need stemlined frequency analysis
<razzy> streamlined
<freemint> how will you the deep learning?
<freemint> *will you do the
<razzy> deep learning does not have to be complicated
<freemint> then you are probably using the wrong word. Deep learning is always complicated
<razzy> symbols and variables in lists
<freemint> what do you mean by deep learning?
<freemint> what do you have in mind for that?
<razzy> structure several layers of variables that propagates error
<razzy> and learn
<freemint> how will you do the derivatives?
<razzy> there is standard neuron
<freemint> What activation function do you intend to use?
<razzy> not very important
<freemint> Is this function implemented in PicoLisp and if not can you implement it7
<razzy> if i understand picolisp well enough, yes i can
<freemint> How will you ensure that the do not need the neural network to decompress?
<razzy> that is the plan
<razzy> it is just heuristic to help find good compress
<freemint> what compression primitives do you have to work with?
<razzy> what do you mean compression primitives?
<freemint> the networks gives you an heuristics on what algorithm to choose when compressing?
<razzy> could be but not yet
<freemint> what is the network for then?
<razzy> in broad sense, network suggest which algoritms to use
<razzy> yes
<razzy> very broad sense
<freemint> based on what metrics does the network decide?
<razzy> freemint: will you help? or you just mining info
<freemint> does it see the complete stream and if so how does it memorize stuff?
<freemint> I am trying to help since you seem very confused about what you want to do
<razzy> i might seem that way :]
<freemint> (Nobody in there right mind would pick picolisp for this task i think) (no insult) also i am curious why you did choose picolisp for this
<razzy> i would say that no normal sane person would choose picolisp
<freemint> yeah
<freemint> so why did you choose it?
<razzy> little baggage, lispy enough.
<freemint> ok
<razzy> scheme was another choice
<razzy> or racket
<freemint> one problem i see is that file the file access functions in PicoLisp are not perfect for that. (on the the other hand scheme would be just as bad and picolisp easier to extend)
<freemint> If i had to choose i would go with clojure
<freemint> all the Java libraries for compression would be helpful
<razzy> freemint: picolisp has wonderfull file access imho
<freemint> it lacks jumping to a position in a file
<freemint> (i think)
<freemint> anyhow what questions do you have?
<razzy> there is peek, but i did not need that
<razzy> i do not know how it works
<razzy> clojure is fucking corporate animall
<freemint> I would never think in terms of tables in picolisp for counting things
<freemint> build an idx tree
<freemint> (doc 'idx)
<razzy> i want to check my code and rewrite it to be picolisp compliant
<freemint> tables in picolisp are lists and when you a 60000 picolisp will follow so many pointers to increment an number
<razzy> i writen table, meant lists structure
<freemint> yeah, go with a tree when you can.
<razzy> 120k cell lists seems ok
<freemint> lwzCompress takes an compression-table?
<razzy> could, depend on implementation
<razzy> imho
<freemint> doing one operation on this list of 120000 will take roughly 2ms on my machine
<freemint> anyway ... optimization comes later
<razzy> yop
<freemint> if lwzCompress Takes an compression-table. where will this table be generated?
<razzy> lzwCompress is bad name. compress generate Comp_table
<freemint> a comp-table and a compressed file?
<razzy> Comp-table is compression table with statistics and how to decompress
<razzy> out file suppose to be compressed
<razzy> i am not there yet
<freemint> all you still failing to fill your buffer?
<razzy> i think i know how to fill my buffer
<razzy> but to write it and test it takes time
<razzy> there are bugs
<freemint> (while 1000>(lenght Buffer)) this line is wrong on so many levels
<razzy> yes, it is my pseudocode
<freemint> (doc 'make) is your friend.
<freemint> What do you intend to do with the buffer?
<razzy> use it as in section 2.1 and 2.2
<freemint> do you want the buffer to always be 1000 elements long?
<razzy> yes i reconsidered 100 long
<razzy> but normal list is ok
<razzy> no advantage imho in making it special
<freemint> does the buffer contain the file or something else?
<razzy> bits of file
<razzy> readed over and over and crossover adressed
<freemint> ok. then i do not grasp what you try to do with the file...
<razzy> file is GBs long
<freemint> 2.0 is very vague
<freemint> i know
<razzy> it is explained in 2.1 and 2.2
<freemint> no "frequency of repeating characters" is badly worded
<razzy> for start i want to count frequency of repeating strings
<razzy> count number of repeating strings
<freemint> how do you know that a string is repeating?
<freemint> (without having the complete file in memory)
<freemint> where does a string start and end?
<razzy> freemint: 2.1 and 2.2
<razzy> it reads the file and make some guesses.
<freemint> so you look for exactly 100 byte long strings?
<freemint> and look how often they appear?
<razzy> up to 100 char long strings
<razzy> yes
<freemint> so if you read "abha" you try to insert "abha" "bha" "ha" and "a" ?
<razzy> i try to make table from all of it
<freemint> what is "all" ? your algorithm is very underspecified
<freemint> is my abha example right?
<freemint> (assuming abha is your buffer)
<razzy> key is, that when string is meeted again, it is solidified. when is not meeted again, it may be replaced with other one
<freemint> that is obvious
<freemint> but what (sub-) strings do you try to insert
<razzy> for start all of them
<razzy> later deep learning helps
<freemint> are you aware that if you process your file in chuncks you loose sub-strings?
<razzy> not if you do not pop strings that you meet for first time
<freemint> example "abhabhabhabha" processed in chunks of four is "abha" "bhaa" "haab" "aabh"
<freemint> in an optimal compression scheme we should say the we see abh 4 times
<razzy> will you write it for me if i write whole example?
<freemint> write what?
<razzy> picolisp
<freemint> well ... i do not want to steal the learning experience but if that helps you ... i would invest 1 hour trying to write atleast
<freemint> or did you take the algorithm from somewhere else ... maybe that resource describes him better
<freemint> ?
<freemint> then you would not have to write anything
<razzy> no, as far as i know, unique
<razzy> i am writing example
<freemint> honestly from what i know about compression your approach seems hopelessly naive. 1. Deep learning will not be able to help you since every file is different 2. you are using a fixed length for maximum symbol length 3. the number of substrings explodes with the length of the file 4. you attempt to fix that problem with a maximum symbol length which is choose arbitrarily without proper statistical or theoretical grounding.
<freemint> But the problem sounds fun, so i see what i can do
<freemint> razzy why is the example taking so long?
<razzy> i am tired and read to hofman
<freemint> do you mean Huffman-Encoding?
<freemint> "Huffman"
<razzy> yes, do not know how close i am to it
<freemint> well i can tell you
<razzy> you are right about exploding, some heuristic should help there
<freemint> there are no general purpose heuristics which make it shorter without make compression more bad in many cases
<freemint> what makes approach different from Huffman is that he has constant word length
<freemint> your extension to longer symbols brings new problems:
<freemint> your problem is that your substrings will overlapp
<razzy> yes
<freemint> "ablabablab" lab and ab or ablab or abl and ab
<freemint> will all be good encodings and there is no way how you can recovers that information from a frequency list
<razzy> solution is to use heuristic, "prefer 2 longest"
<freemint> first off heuristics are not solutions
<freemint> heuristics are approximations
<razzy> well yes
<razzy> i do not need perfect solution, just better than everyone else
<freemint> 2. any heuristic a neural network can come up with is likely bullshit
<freemint> 3. The level on which you can specify these heuristics is inappropriate for the task
<freemint> 4. you will not Winn the hutter price with that approach
<razzy> in random example yes,.. but human text or usefull data has rules to them. neural networks pick them up
<razzy> 4. maybe :D
<freemint> are you ducking kidding?
<razzy> ducking?
<freemint> fucking
<razzy> not really, hutter is good :]
<freemint> if i were to attempting to solve this prize i would build exploit structural information like XML tags
<freemint> sorry but you are totally going down the wrong path
<razzy> we will see
<razzy> or not
<razzy> :D
<freemint> sorry but if you are trying to claim an improvement on that prize, i think i can not help you
<razzy> free why?
<razzy> freemint: why?
<razzy> are you competing?