hitchhikingcoder has quit [Remote host closed the connection]
pjb has quit [Ping timeout: 252 seconds]
makomo has quit [Ping timeout: 252 seconds]
Kundry_Wag has joined #lisp
khisanth_ has quit [Ping timeout: 244 seconds]
Kundry_Wag has quit [Ping timeout: 260 seconds]
dented42 has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
frodef has quit [Ping timeout: 245 seconds]
khisanth_ has joined #lisp
djh has quit [*.net *.split]
_bryan has quit [*.net *.split]
payphone has quit [*.net *.split]
grobe0ba has quit [*.net *.split]
grobe0ba_ has joined #lisp
grobe0ba_ is now known as grobe0ba
Kundry_Wag has joined #lisp
Kundry_Wag has quit [Ping timeout: 252 seconds]
Jesin has quit [Quit: Leaving]
<no-defun-allowed>
i'm giving cl-vep a real test today
<no-defun-allowed>
it's going to green screen........my english language video
hitchhikingcoder has joined #lisp
<AeroNotix>
no-defun-allowed: what videos do you make?
Roy_Fokker has quit [Read error: Connection reset by peer]
<no-defun-allowed>
ones where i need half decent green screening
<no-defun-allowed>
we're doing a presentation on pidgins and creoles in eng lang so we have different places green screened
<no-defun-allowed>
also, pigeons
Kundry_Wag has joined #lisp
<AeroNotix>
no-defun-allowed: Where is this broadcast?
<AeroNotix>
school or youtube or something?
<no-defun-allowed>
nowhere, it's just for school
<AeroNotix>
ah ok
hitchhikingcoder has quit [Quit: hitchhikingcoder]
<AeroNotix>
I found an internal advertisement I was made to do ten years ago while I worked at HP on an old hard drive the other day. Other dimensions could feel the cringe.
<no-defun-allowed>
wow
Kundry_Wag has quit [Ping timeout: 245 seconds]
asarch has joined #lisp
milanj has quit [Quit: This computer has gone to sleep]
robotoad has quit [Quit: robotoad]
Oladon has quit [Quit: Leaving.]
robotoad has joined #lisp
Kundry_Wag has joined #lisp
dddddd has quit [Remote host closed the connection]
Kundry_Wag has quit [Ping timeout: 252 seconds]
robotoad has quit [Ping timeout: 272 seconds]
Oladon has joined #lisp
quazimodo has joined #lisp
Kundry_Wag has joined #lisp
mathrick has quit [Ping timeout: 240 seconds]
asarch has quit [Ping timeout: 260 seconds]
Kundry_Wag has quit [Ping timeout: 252 seconds]
fortitude_ has joined #lisp
<Pixel_Outlaw>
Is there a way to print a literal escape sequence? Thinking about toying with some ReGIS in xterm.
<Colleen>
Pixel_Outlaw: jackdaniel said at 2018.09.12 05:06:21: it seems that for some reason trivial-garbage's make-weak-hash-table doesn't remf this keyword argument before passing to make-hash-table in sbcl
<Colleen>
Pixel_Outlaw: jackdaniel said at 2018.09.12 05:06:53: what is weird, because I have newest trivial-garbage and all works just fine for me. do you have some hacked version in your local-projects?
robotoad has joined #lisp
mathrick has joined #lisp
nowhere_man has quit [Ping timeout: 240 seconds]
Lycurgus has joined #lisp
orivej has quit [Ping timeout: 260 seconds]
Kundry_Wag has joined #lisp
Kundry_Wag has quit [Ping timeout: 246 seconds]
prometheus_falli has quit [Ping timeout: 256 seconds]
housel has quit [Remote host closed the connection]
housel has joined #lisp
meepdeew has quit [Remote host closed the connection]
Pixel_Outlaw has quit [Quit: Leaving]
Kundry_Wag has joined #lisp
quazimodo has quit [Ping timeout: 260 seconds]
Kundry_Wag has quit [Ping timeout: 252 seconds]
Pixel_Outlaw has joined #lisp
Pixel_Outlaw has quit [Client Quit]
nonlinear has quit [Ping timeout: 244 seconds]
Kundry_Wag has joined #lisp
nonlinear has joined #lisp
Kundry_Wag has quit [Ping timeout: 272 seconds]
Bike has quit [Quit: Lost terminal]
SaganMan has quit [Read error: Connection reset by peer]
pierpal has quit [Quit: Poof]
pierpal has joined #lisp
quazimodo has joined #lisp
Kundry_Wag has joined #lisp
Kundry_Wag has quit [Ping timeout: 260 seconds]
pierpal has quit [Ping timeout: 260 seconds]
Fare has quit [Quit: Leaving]
quazimodo has quit [Ping timeout: 240 seconds]
Shinmera has quit [Remote host closed the connection]
Oladon has quit [Quit: Leaving.]
Kundry_Wag has joined #lisp
Kundry_Wag has quit [Ping timeout: 252 seconds]
<beach>
Good morning everyone!
Kundry_Wag has joined #lisp
<gendl>
Good morning beach. I keep forgetting, are you RS?
<gendl>
I'm just going to bed (told my wife I would be there 2 hours ago but got stuck finishing a few things on our new web-based object browser)
<beach>
gendl: I am, yes.
<beach>
"beach" is just a literal translation of my last name into English.
<gendl>
Oh right - I should have thought of that.
<gendl>
Playa
anewuser has quit [Quit: anewuser]
<beach>
How are you these days?
Kundry_Wag has quit [Ping timeout: 252 seconds]
<gendl>
can't rightly complain much, although I still do... been through some challenges lately,
<beach>
Oh!
nirved_ has joined #lisp
<gendl>
you know about one of them. That looks like (knock wood) coming to a conclusion here real soon.
<beach>
OK, good to hear.
<beach>
Something to do with the ALU, I am guessing?
<gendl>
not ALU - an ongoing legal issue - i've told you about it
<beach>
Oh, that one! Yes, I remember.
<gendl>
ALU has been in limbo, as far as I'm personally concerned, it's a bit far down on my list.
nirved has quit [Ping timeout: 240 seconds]
<beach>
I understand.
<gendl>
RS is technically still president and things are kind of in a holding pattern until he gets a new one
<gendl>
CLF is active, and we're overdue for launching that ASDF fundraider.
<gendl>
oops -- i mean raiser lol
<beach>
Heh!
<gendl>
yep, we're gonna raid funds out of ASDF har
<gendl>
we have $5000 in matching funds and they're finally in the proper bank account and Stripe is hooked up to it
<beach>
Sounds good.
<gendl>
... so i just have to get the website spun up and re-skinned (the one from Quicklisp from almost two years ago)
Kundry_Wag has joined #lisp
<gendl>
then start publicizing.
_bryan has joined #lisp
<beach>
Shinmera is good with websites.
<beach>
Perhaps he can help.
<gendl>
I'll pass it by him before launching.
<beach>
He did the new one for ELS.
<beach>
By the way, McCLIM development is very active these days, in case you are looking for a fundraising target in the future.
<no-defun-allowed>
morning beach
<gendl>
yes, fundraisers should really be going back-to-back
<gendl>
two years has been far too long of a pause between the QL one and now.
<beach>
If you say so.
<gendl>
well, I think if we can demonstrate some modicum of organization, there are deep-pocketed donors out there who can help us out
<beach>
That sounds very good.
<gendl>
i think it'll be great to have a balance between a good number of grass-roots donors of smaller amounts, and a few deep-pocketed well-wishers who can see that their larger donations are going to good use
<gendl>
that's my wishful thinking, but that is kind of what happened for the QL fundraiser.
<beach>
I totally agree.
Kundry_Wag has quit [Ping timeout: 260 seconds]
<beach>
I would definitely be in the first category.
<beach>
Seeing no-defun-allowed, I was reminded about the importance of coming to ELS so that one can recognize the people here.
<gendl>
I still have to make the XML file to index the ELS proceedings with ACM... I didn't see a hard deadline for that so I'm hoping they'll still accept.
makomo has joined #lisp
<gendl>
I did get the Report to them.
<beach>
I see.
<gendl>
... so yeah, I've got a bit of a backlog of volunteer work to make good on
<no-defun-allowed>
when's ELS?
<beach>
no-defun-allowed: I have told you this before, but I'll say it again. I think you should do what I recommended to froggey, namely put up a crowd funding site to pay your way to ELS 2019.
frgo has quit [Remote host closed the connection]
<beach>
no-defun-allowed: April 1 and 2 in Genoa.
<no-defun-allowed>
do you really think i should go?
<beach>
Absolutely!
frgo has joined #lisp
<no-defun-allowed>
i can't though, that's a school week
<beach>
:(
<gendl>
no-defun-allowed: where are you from.
Shinmera has joined #lisp
<no-defun-allowed>
smelbourne, australia
<no-defun-allowed>
the week after is school holidays
<no-defun-allowed>
another off-by-one error by the victorian government
<gendl>
Ah.
<beach>
I like Melbourne. The tram system is fantastic!
<gendl>
no-defun-allowed is an example of why we still need ILCs
<gendl>
in alternate locations than Europe every time.
<beach>
Sure, yes.
<no-defun-allowed>
yay i'm an example
<gendl>
yep you're a motivation.
<gendl>
to have another meeting somewhere on that side of the planet
<no-defun-allowed>
anyways i do need to do school work for a while, i've got to hack up a video for tomorrow
<no-defun-allowed>
which reminds me of two things for cl-vep:
<no-defun-allowed>
- fix the god damn slow ffmpeg interface
<gendl>
and i'm off to bed, it's 2am. I need to hang out in here more to stay motivated.
<no-defun-allowed>
- make the parallel-mapvideo buffer guess the frame count that can fit in memory instead of a fixed frame count
<no-defun-allowed>
night
<beach>
'night gendl.
lemonpepper24 has quit [Ping timeout: 245 seconds]
frgo has quit [Ping timeout: 246 seconds]
<Shinmera>
school week schmool week, go anyway.
<beach>
Heh!
<Shinmera>
There's been lectures every year I've gone
<no-defun-allowed>
Shinmera: i'd fail year 12 then
<Shinmera>
because of one week?
<no-defun-allowed>
probably
Inline has quit [Quit: Leaving]
<no-defun-allowed>
you're expected to have a medical certificate for days off, and anything below 90% attendance or some shit is failing too
<Shinmera>
talk to your department lead if you can get the week off to attend an important (!) conference that will further your studies (!)
<no-defun-allowed>
i guess that could happen
<Shinmera>
or the secretary of studies, whatever it is you have over there
<no-defun-allowed>
would that be at school or higher up?
<beach>
I totally agree with Shinmera.
<beach>
no-defun-allowed: The school should be enough.
<beach>
no-defun-allowed: Let me know if you need a letter.
<no-defun-allowed>
that'd probably be needed
<beach>
OK then, first check whether it is possible, hinting that you can get a letter from me.
shrdlu68 has joined #lisp
reverse_light has joined #lisp
<no-defun-allowed>
i don't know how to go about that though
nirved_ is now known as nirved
<beach>
Go to the head of the school. Say something like "I have become involved in the Common Lisp community, and I have received an invitation to come to ELS 2019... Would there be a possibility for me to get that week off from school?"
<no-defun-allowed>
beach: even if i did go i still think it'd go over my head
<beach>
no-defun-allowed: It is a common mistake to believe that the talks are the important part of a conference like this.
<no-defun-allowed>
okay then
<beach>
The important part is to chat with the participants between talk, at lunch, at diner, etc.
<beach>
dinner.
<no-defun-allowed>
okay then
<beach>
Plus, you would get to go to Genoa.
<no-defun-allowed>
is it going to cost anything if i manage to get a flight there? i've never been to europe or planned much myself
<beach>
Like I said, set up a crowdfunding site.
<no-defun-allowed>
okay
<beach>
You need to pay for the hotel or equivalent.
<beach>
And there is a small fee for attending ELS.
<no-defun-allowed>
fair enough
<beach>
Less than 100€ as I recall.
<no-defun-allowed>
that's not so bad
<beach>
Right. We intentionally keep it modest for students.
<beach>
Actually, it is quite modest for others as well.
<no-defun-allowed>
the school does a lot of extracurricular stuff so maybe they'll let me go for the week
<beach>
no-defun-allowed: Oh, and it's a joint event with the <programming> conference.
<beach>
no-defun-allowed: That could be fun for you as well.
<jackdaniel>
but wikipedia article is better and it mentions that
<jackdaniel>
"In 1995, Netscape Communications recruited Brendan Eich with the goal of embedding the Scheme programming language into its Netscape Navigator."
Kundry_Wag has quit [Ping timeout: 240 seconds]
scymtym has quit [Ping timeout: 244 seconds]
<aeth>
Mission failed.
<shrdlu68>
Mission perverted.
frgo has joined #lisp
<shrdlu68>
"But CL implementations! What big binaries you produce", said the Little Red Riding Hood.
stnutt has quit [Ping timeout: 252 seconds]
Kundry_Wag has joined #lisp
<no-defun-allowed>
"All the better to debug myself with."
<Jachy>
beach: was there much overlap in speakers between ELS and <Programming> the previous time they were co-located? I assume lots of attendees of ELS would go to <Programming> while they're there, did the reverse happen too?
housel has quit [Read error: Connection reset by peer]
<jackdaniel>
Jachy: unfortunately (imho) things were happening at the same time, so nobody would want to miss ELS to attend programming
<jackdaniel>
we were just one block of four or five in the pool
<jackdaniel>
but people had nice chat in the cafeteria during the lunch break
<Shinmera>
Programming attendance also had a much steeper price tag, if I remember correctly
stnutt has joined #lisp
<beach>
Jachy: I don't think there was a lot of overlap.
Kundry_Wag has quit [Ping timeout: 246 seconds]
<makomo>
morning!
<beach>
Hello makomo.
stnutt has quit [Ping timeout: 240 seconds]
<no-defun-allowed>
hey makomo
<makomo>
hello :-)
stnutt has joined #lisp
<no-defun-allowed>
is opticl usually fast?
Cymew has joined #lisp
<jackdaniel>
only on Fridays and Wednesdays
<jackdaniel>
Thursdays are 50-50, Sundays are a nightmare
<aeth>
should remove the cloud features then
<no-defun-allowed>
damn, i'll write my own pbm blitter then
<aeth>
no-defun-allowed: iirc it depends on the format and the direction because it's just a wrapper over other things.
<aeth>
so e.g. reading PNGs is fast
<no-defun-allowed>
writing pbm to ffmpeg is the bottleneck right now
<Shinmera>
if the png is compressed, which is almost always, it's slow :(
<jackdaniel>
his point is that mfiano wrote an optimized png loader (and opticl uses that)
orivej has joined #lisp
<Shinmera>
My point is that it's still unbearably slow for compressed pngs
<no-defun-allowed>
cool
<no-defun-allowed>
my problem is writing PPMs to ffmpeg though
<aeth>
There are probably fewer than 30 people who write fast CL.
<aeth>
So there's naturally a shortage in fast CL libraries for niche things.
<|3b|>
opticl has some options for fast, but like most pixel manipulation libs is easy to use slowly :)
<|3b|>
a large part of png reading time was zlib last i checked, so need to rewrite another lib to improve it much :(
<Shinmera>
yes
<|3b|>
(and probably rewrite pngload again too after that)
* |3b|
wants a version that can do streaming reads, so i can just start a bunch of threads and ignore some of the speed problems :p
<|3b|>
while limiting memory usage if i happen to want to load a bunch of 4kx4k images at once or something
<|3b|>
but for now i have other things to do, so png will have to wait
trittweiler has joined #lisp
<Shinmera>
While we're on image formats, cl-jpg can't even read all jpgs :/
<no-defun-allowed>
yeah ):
<jackdaniel>
lucky for us it is an opensource library which we can improve to read all of them! :)
<no-defun-allowed>
it feels slightly faster with a custom ppm writer but i'm Totally Not Biased
Kundry_Wag has joined #lisp
<no-defun-allowed>
okay but the writer doesn't work so
<no-defun-allowed>
guess i'm stuck on 2.3fps
<|3b|>
at least we tried to get that mostly right for pngload compared to the other libs :/
<|3b|>
how big are your frames?
shka_ has joined #lisp
<no-defun-allowed>
1080x720 right now
<no-defun-allowed>
(that's the recording resolution of a macbook air)
<shka_>
good morning
<|3b|>
8-bit rgb i assume?
<no-defun-allowed>
well there is a float->8 bit conversion step
<no-defun-allowed>
hmmm, what if that's the bottleneck?
<no-defun-allowed>
i swear if that's it
gpiero has joined #lisp
<|3b|>
might be, if you aren't declaring types properly
<no-defun-allowed>
yeah
cl-arthur has joined #lisp
Kundry_Wag has quit [Ping timeout: 252 seconds]
Kundry_Wag has joined #lisp
varjag has joined #lisp
cyraxjoe has quit [Ping timeout: 245 seconds]
* |3b|
can write random 1080x720 png files at ~6fps, ppm at 15 with opticl
<|3b|>
filling each pixel with (floor (* 255 (random 1.0))) for each file
<|3b|>
(random value for each component)
<no-defun-allowed>
hmm i see
scymtym has joined #lisp
<no-defun-allowed>
i've tried some type definitions and i've got sbcl's (speed 3) down to 3 complaints
<aeth>
asarch: If you're using SBCL for a lot of 2D array code, SBCL 1.4.10 and later has fixes for bounds checking on 2D arrays
<asarch>
Thank you!
<asarch>
:-)
<|3b|>
use :report :graph (or :type :graph if you call report directly instead of just with-profiling)
<no-defun-allowed>
ok
<no-defun-allowed>
good timing, i forgot to add :report and nothing happened
<|3b|>
if you end up using it much, probably also want to install and use the slime slime-sprof contrib, easier to navigate, but harder to paste for other people to look at :)
<no-defun-allowed>
i see
<no-defun-allowed>
it seems it doesn't take multiple threads well
Kundry_Wag has quit [Ping timeout: 240 seconds]
<|3b|>
(and you probably want other people to look at it when you start out, not obvious what the various internal things indicate about your code)
<no-defun-allowed>
no it just doesn't like me in general
schweers has joined #lisp
<|3b|>
might try :mode :time
pfdietz has quit [Ping timeout: 252 seconds]
<no-defun-allowed>
i'm fairly sure it's gonna be the image read/write though, i've tuned the effects very carefully
<no-defun-allowed>
it prints out function calls but not relevant ones
<|3b|>
looks like array stuff is in the mask-blend lambda
<no-defun-allowed>
right then
<|3b|>
in each block, the line with #s under 'total.' (and name shifted to the left a bit) is the function being measured in that block, above it shows what called it with how much to the left
<no-defun-allowed>
gotcha
<|3b|>
so in 3rd block the slow-...-ref is called by opticl (1%) and mask-blend (5%)
wigust has joined #lisp
<|3b|>
and in first block you can see same info, and that mask-blend is also making full calls to +,-,*
<no-defun-allowed>
i see
<no-defun-allowed>
you know what kdenlive does better than video editing?
<no-defun-allowed>
crashing.
<|3b|>
:)
<no-defun-allowed>
every time i run a parallel-mapvideo, it complains the worked on video is invalid and crashes
<no-defun-allowed>
also it seems i should have taken a background picture for every clip i masked
<no-defun-allowed>
eek, no type declarations
<shrdlu68>
dim: Getting "Failed to connect to pgsql...Can't resolve foreign symbol "SSL_load_error_strings""
arduo has joined #lisp
<no-defun-allowed>
also i have to do a beach and set the heap size relatively high to get the worker->writer cache working well enough
<no-defun-allowed>
43% realtime is pretty okay actually
* |3b|
always builds sbcl with heap a bit under ram size
<|3b|>
hmm, actually i guess i've been building with 16gb heap out of 32gb ram lately, i should raise that
<no-defun-allowed>
nice
<no-defun-allowed>
i've got mine at half my machine's ram, 4gb
<no-defun-allowed>
if it doesn't work at 4gb, something's really wrong
<no-defun-allowed>
hasn't the ram price inflation bothered you a bit?
<no-defun-allowed>
this single stick of 2400mhz 8gb ram was au$70
<no-defun-allowed>
my dad's 16gb ddr3 was $100 five years ago
<|3b|>
i tend to overspend on computer stuff anyway :/
<no-defun-allowed>
i see
<makomo>
no-defun-allowed: lul, that made me laugh
<no-defun-allowed>
what did i say? i said a lot
<makomo>
"to do a beach" :')
<no-defun-allowed>
:3
Kundry_Wag has joined #lisp
schweers has quit [Remote host closed the connection]
<shrdlu68>
dim: Runnning the dimitri/pgloader container fails with "The alien function "CRYPTO_num_locks" is undefined".
<no-defun-allowed>
i do need to work on the cache though, and make it use a "maximum pixel count" instead of "maximum frame count"
<makomo>
shrdlu68: i had that problem as well
<no-defun-allowed>
i also need to work on cl-decentralise but that's not doing any school projects for me
<makomo>
shrdlu68: i remember having to install an older version of SSL (1.1.0, as opposed to 1.1.1, i think)
<makomo>
and then there was the problem of cl-plus-ssl not having the correct name of the shared lib in its CFFI defs
Kundry_Wag has quit [Ping timeout: 260 seconds]
<shrdlu68>
makomo: ..this is in the docker image, shouldn't that bundle up everything it needs?
<minion>
Xof, memo from flip214: please send me the link with the full results (whole year) of your 'Using Lisp-based pseudocode to probe student understanding' Moodle experiments; ISTR that you posted that in June or so. I'd like to forward that. Thanks a lot!
marvin2 has quit [Ping timeout: 245 seconds]
dddddd has joined #lisp
milanj has joined #lisp
Kundry_Wag has joined #lisp
Kundry_Wag has quit [Ping timeout: 246 seconds]
mingus has joined #lisp
Kundry_Wag has joined #lisp
Kundry_Wag has quit [Ping timeout: 252 seconds]
jaziz has joined #lisp
jaziz has quit [Quit: Leaving]
esrse has quit [Ping timeout: 272 seconds]
sigjuice has quit [Ping timeout: 252 seconds]
sigjuice has joined #lisp
Kundry_Wag has joined #lisp
Kundry_Wag has quit [Ping timeout: 252 seconds]
Kaisyu has quit [Quit: Connection closed for inactivity]
arduo has quit [Read error: Connection reset by peer]
xificurC has joined #lisp
arduo has joined #lisp
Kundry_Wag has joined #lisp
<xificurC>
I have an unknown number of lines I'm reading. I want to work through the entries on-the-fly, not accumulating everything first, because the volume will not fit in RAM. So there's no way to tell the final amount. A line needs to be parsed (json) and processed. The processing is fast, the parsing is slow. I want to parallelize this so I picked up lp
<xificurC>
arallel. I cannot find a solution that would ensure the tasks pushing onto the queue will really all be processed. This is what I've come up with but it still doesn't ensure all messages get processed. http://sprunge.us/1dK2Yw
Kundry_Wag has quit [Ping timeout: 252 seconds]
Essadon has joined #lisp
* |3b|
would probably have the worker loop waiting on the queue, and exit loop when it gets an :exit message (or anything distinguishable from input)
<xificurC>
|3b|: I thought about that, but who should send that message? How can I make sure that message gets on the queue last?
<|3b|>
reader
<|3b|>
(loop for line = ... finally (send :exit))
<|3b|>
though i think i'm thinking of other API
<xificurC>
does lparallel.queue guarantee that (submit-task channel (lambda () (push-queue :first))) (push-queue :second) (eq :first (pop-queue)) (eq :second (pop-queue)) ?
<phoe>
yes, the queue is FIFO
<|3b|>
ok, with submit-task, i think reader needs to count tasks and loop on (Receive-result) until it gets that many
<xificurC>
are the pushes serialized?
<phoe>
if you have eight processors, then you'll need to (finally (loop repeat 8 do (push-queue :exit)))
<phoe>
so each processor consumes one :exit and finishes
<|3b|>
with submit-task you don't need a specific :exit task, since each line is a separate task, you just need to wait for all of them to finish
<phoe>
oh right
<phoe>
if you create a kernel, then it will take care of that
<phoe>
the kernel threads will just wait for new tasks without consuming any resources.
<|3b|>
in either case, if lines are independent and you want more than 1 parser task, you can just add a serial # and sort results at end or something
<xificurC>
how do I wait for all of them to finish?
<phoe>
each submitted task returns a result
<|3b|>
with submit-task, 1 receive-result per submit-task
<phoe>
so if you submit 9001 tasks, you (loop repeat 9001 do (receive-result))
<xificurC>
but I don't know how many of them I'll get
fluxit has quit [Quit: ...]
<|3b|>
count them as you read lines
<phoe>
in that case--- exactly
<|3b|>
then wait for results after end of file
<phoe>
count them as you read lines
<phoe>
and when you're done reading, (loop repeat count do (receive-result))
<xificurC>
but not each line is processed and some fail within the submit-task call
<phoe>
in that case program your code to handle the error somehow and return a result anyway.
<|3b|>
right, but you catch the error, so task returns something
<phoe>
and then when you count the results, you can catch the "error happened" results and filter them out.
<xificurC>
right
<|3b|>
might want an atomic incf there too
<xificurC>
but how to count?
<xificurC>
right, I tried incf but it doesn't work, as expected
<xificurC>
(let ((channel (make-channel)) (x 0)) (loop repeat 1000 do (submit-task channel (lambda () (incf x) :done))) (loop repeat 1000 do (receive-result channel)) x) --> anything between 960 an 1000
<|3b|>
return success/fail and count in the receive results and you don't have to use atomics
<|3b|>
right, INCF isn't generally atomic
<|3b|>
it has to read, add and write. another thread could read before the write, and then overwrite the first write
<|3b|>
most threaded lisps have an atomic incf that only works on specific types of data (since otherwise it would need locks)
<xificurC>
I thought I could make it a fixnum and get an assembler inc call :)
<|3b|>
(or you can use locks yourself, but that's more work and probably slower)
<|3b|>
you can, if you use the special call :)
<|3b|>
it doesn't know the inc won't overflow into a bignum for example
<|3b|>
('fixnum' may or may not be the right thing for the special call)
arduo has quit [Ping timeout: 252 seconds]
<|3b|>
and for that matter, asm inc might not be atomic without special modifiers
<xificurC>
yeah, so to sum up the only way to make sure is to count the lines that were processed and receive that many results. Counting needs to be done either atomically or with some locking mechanism. So I need to look at sbcl if it provides an atomic increment or go for a lock
<|3b|>
not 'only way', but seems like appropriate way with the submit-task API
<|3b|>
and counting can be done by the receive-result loop with no atomic or lock
<|3b|>
if multiple threads want to update a shared variable, you need atomics or locks
<xificurC>
well queues don't give another option either
<|3b|>
with queues you could implement something like submit-task yourself :)
<xificurC>
how can I count without atomics or locks?
<|3b|>
(which was the "send :exit message" idea)
justinmcp has quit [Quit: No Ping reply in 180 seconds.]
<|3b|>
count on a single thread
<xificurC>
I don't think I can. Let me check
<|3b|>
(the atomics or locks are hidden by the queue implementation in that case)
justinmcp has joined #lisp
<xificurC>
I'm reading lines from a stream. I can't wait for the stream to be read completely, so I have to fire up the receiver loop before there's anything to receive
<|3b|>
actually, reading closer, you were using both strategies... you submit a task to add to the queue
<_death>
you could also just split the file and process the parts, then merge results
<|3b|>
with submit-task you don't have a receiver loop
<xificurC>
_death: it's not a file, it's a stream, think stdin
<|3b|>
lparallel starts some worker threads, and they all wait for tasks to be submitted
<xificurC>
|3b|: I was using a queue inside because I need to start the receiver loop before reading all lines
<|3b|>
you don't need a receiver loop
<shka_>
ok, so if it was a file, i would recommend cl-ds line-by-line -> process -> in-parallel pipe
<xificurC>
still waiting for enlightenment, sorry
<|3b|>
(unless you want stats while reading or something, but even then just counting results is probably fast enough to do between lines)
<shka_>
since it is a stream, this won't work, but strategy remains the same
<|3b|>
ok, for specifics need to work out details
<|3b|>
first thing is do you need any incremental results or just at end of whole thing?
<shka_>
xificurC: you will need thread to read stream, lparallel:future that will do processing, and lparallel.queue with fixed size to get the result
<xificurC>
I can't have the 1 loop that is reading the stream both submit and receive because then I'm blocking and processing line by line again
Kundry_Wag has joined #lisp
<|3b|>
(like printing progress report, or writing output if parsed data won't fit in ram or whatever)
<|3b|>
you don't have to block, just check once in a while
<xificurC>
|3b|: at end
<shka_>
so basicly do the following -> on thread A: read line, create future, push it into fixed size queue
<|3b|>
ok, next is processing of lines independent?
<shka_>
on future, do processing, return result
<shka_>
on thread C, read queue, handle results
<shka_>
xificurC: makes sense?
<xificurC>
|3b|: no, each line is setf-ing something on slowlogs and groklogs
<|3b|>
(like line N can be parsed without knowing results of parsing line N - x)
<|3b|>
do they have to be in order?
<shka_>
basicly two independent loops, in two threads, processing on the lparallel worker, and threads are talking to each other with queue
<xificurC>
shka_: I'm ok with the processing part, not ok how to ensure all lines get processed
<|3b|>
and is setf-ing slow?
<|3b|>
and is the something large enough to worry about storing all of them?
<shka_>
xificurC: lparallel future can forward error
<xificurC>
|3b|: yes lines are independent, setf is fast
<xificurC>
phone
vlatkoB_ has joined #lisp
Kundry_Wag has quit [Ping timeout: 240 seconds]
<shka_>
or you can handle error in the worker, and return tuple of results, where first value is processing result, and the second is the processing status
Kundry_Wag has joined #lisp
<shka_>
don't have time right now to demonstrate how this works, but maybe in the evening if you want to
<|3b|>
looking at code some more, you are probably losing results due to lack of atomics/locks too
<shka_>
anyway, this approach maintains upper memory usage, does not require explicit locks on your side and usually works good enough
<shka_>
give it a try
<shka_>
it is usually also simple to understand
ebrasca has quit [Read error: Connection reset by peer]
<shka_>
i think that it is a good fit for your problem
vlatkoB has quit [Ping timeout: 245 seconds]
Kundry_Wag has quit [Ping timeout: 260 seconds]
<xificurC>
back
<xificurC>
|3b|: true, I'm probably losing data there too, didn't realize yet
vibs29 has quit [Ping timeout: 250 seconds]
vibs29 has joined #lisp
<xificurC>
shka_: I understand your solution but don't see how it solves my problem of ensuring all input gets processed. The 2 facts that it's coming from a stream and is of unknown length make the problem hard
<|3b|>
untested, and assuming add-*log doesn't actually do anything beyond PUSH, adjust as needed
fluxit has joined #lisp
<|3b|>
oops, typo, fixed
<|3b|>
and again :/
zfree has joined #lisp
<|3b|>
and rearranged a bit just for appearance
<|3b|>
actually that closure is probably bad (unportable at best)
<xificurC>
if I'm reading correctly this will read all lines and submit them as tasks first, eating up a lot of memory
<|3b|>
updated again, don't close over loop variables unless you checked spec to know it is safe (and know why it wouldn't be)
<|3b|>
right, it uses arbitrary memory if parsing is slower than reading
<_death>
so do it in batches
<|3b|>
you can add rate limiting if needed, though you need to merge the 2 loops in that case
<|3b|>
(or add some out-of-band communication between the tasks and read thread)
<|3b|>
but your original code did that too i think
nckx has quit [Ping timeout: 240 seconds]
orivej has quit [Ping timeout: 252 seconds]
<xificurC>
|3b|: I could move your receiving loop before the submitting one and receive with try-receive-result and a timeout perhaps?
<xificurC>
so the setfing would take place on the main thread, and when the try-receive-result succeeds it will increment/decrement some counter
<xificurC>
this way since processing is faster than parsing the memory would stay limited
<|3b|>
is the problem with memory the input to parse or output or both?
<xificurC>
the result is small, it's just a small report. What takes a lot of memory are the lines and the jsons parsed from them
<|3b|>
no, i mean result of parse-line (which is the slow part, right?)
<|3b|>
but sounds like answer is "both"
<shka_>
xificurC: well, EVERYTHING eventually lands in the worker and queue
<shka_>
that's how it is ensured
nckx has joined #lisp
<|3b|>
so in that case, need rate limiting, and we either need to merge the read and process loops or run them at same time
<xificurC>
|3b|: yes output of parse-line is slow and heavy
<xificurC>
shka_: what do you mean, what is the conclusion of your statement?
<|3b|>
so i assume your add-slowlog and add-groklog just extract some part of the parsed json to save?
<xificurC>
yes
<|3b|>
ok, need to think about what primitives are easily available (do you care about running on anything besides sbcl?) to decide which
<shka_>
xificurC: my conclusion is that every line of stream will be read, sent to process, and result of it will be available on the consuming end of the queue
<shka_>
given this framework, you can insert your logic to ensure error handling
<xificurC>
I'm not sure I made this clear but the main issue encountered was OOM. When the pipeline is processed as-we-go the GC can clean up some space
<shka_>
that's why queue has fixed size
<|3b|>
looks like bx-threads only gives us locks and conditions, so need to either find an atomic portability lib, use sbcl specific, or use locks for rate limiting
msb has quit [Ping timeout: 244 seconds]
<|3b|>
ah, fixed-size queue works too, good idea
<shka_>
lparallel.queue can do that
<xificurC>
why do I need rate limiting if the processing loop is mostly starved?
<|3b|>
right
<xificurC>
shka_: what is the behavior of the fixed size queue when it gets full and there's a push?
<|3b|>
reader loop will fill queue with entire input text if parsing is much slower than reading
<shka_>
xificurC: block
scymtym has quit [Ping timeout: 260 seconds]
<shka_>
and wait until there is space again
suskeyhose has joined #lisp
m00natic has joined #lisp
msb has joined #lisp
<|3b|>
so you need to both not read entire file into ram, and not store entire output in ram
<shka_>
anyway, this is common way to do this stuff
<|3b|>
for first we need to rate-limit submitting tasks, for latter we need to check results regularly
<xificurC>
great, that starts to make sense :)
<shka_>
xificurC: so as I said, try two thread, futures and queue
<shka_>
it should work just fine
<xificurC>
I think I understand enough to implement now
Kundry_Wag has joined #lisp
<xificurC>
at least I'll try
<shka_>
good
<xificurC>
thanks |3b| and shka_
<xificurC>
it only took me 1 hour here. 1 hungry hour. gotta eat something. Thank you very much for your help
Kundry_Wag has quit [Ping timeout: 260 seconds]
orivej has joined #lisp
trittweiler has joined #lisp
<|3b|>
hmm, is there any way to see if submit-task will block?
<|3b|>
if not, i think it needs separate threads to submit and receive, which means need to synchronize count between them :/
j`ey has quit [Remote host closed the connection]
<|3b|>
actually i guess if you know size, you can just stop submitting and wait for results
<shka_>
|3b|: that's why lparallel.queue solve the problem
<shka_>
as i said, one producer thread, one consumer thread, lparallel workers and queue between producer and consumer
<shka_>
and futures in the queue
<|3b|>
shka_: you mean running your own worker loops instead of the kernel api (submit-task+receive-result)?
<shka_>
i mean literally this: thread A reads stream, pushes lparallel:futures into the queue
<shka_>
lparallel do it's thing in the kernel
<shka_>
thread B pulls from the queue
<|3b|>
ah, right, futures handles the workers
<shka_>
voila, problem solved
* |3b|
needs to use lparallel more :/
<shka_>
heh
<shka_>
it is pretty great!
<|3b|>
yeah, though not quite what i wanted last time i did threaded stuff, so probably need to implement my own thing anyway :(
<shka_>
anyway, that's what i do in cl-ds.threads:in-parallel and it works just fine
<shka_>
well, you can try
<shka_>
but lparallel is the best i have seen so far
* |3b|
needs to run some things in specific threads, and make sure some tasks don't block
<shka_>
in all programing languages
<|3b|>
yeah, domain specific problems it doesn't know about :)
<shka_>
ok
<shka_>
well, this can be done with lparallel as well
<shka_>
but it does not support this style of design explicitly
<shka_>
basicly you would have to open channel and hijack worker
<|3b|>
yeah, might end up with (parts of) it in the background
<shka_>
which can be done but it kinda defeats the purpose :D
<|3b|>
right
Kaisyu has joined #lisp
<|3b|>
if i'm just using it to start threads that each run a single task indefinitely and i manually schedule small work between them, might as well start my own threads :)
<gendl>
good morning boys and girls
<shka_>
gendl: hi
longshi has joined #lisp
<phoe>
heyyy
suskeyhose has quit [Ping timeout: 260 seconds]
<shka_>
i have a path
<shka_>
pathname even
Kundry_Wag has joined #lisp
<shka_>
i want to create pathname based on it, such that result will contain n deepest elements of the original pathname
<shka_>
how do i do such thing?
<shka_>
for instance /usr/lib/foo/bar should become /foo/bar given n = 2
cl-arthur has quit [Ping timeout: 260 seconds]
<jackdaniel>
and it should be still absolute pathname?
<|3b|>
pathname object not a namestring (or a normal string containing a posix path or whatever)?
<jackdaniel>
my take would be: (make-pathname :default *pathname* :drectory (let ((d (pathname-directory *pathname*))) (list* (car d) (cddr d))))
<|3b|>
hmm, 'valid pathname directory' definition doesn't allow for keywords (aside from 'implementation defined' case)
<jackdaniel>
(not default meant to prevent losing file name)
<pfdietz>
You can create a new pathname defaulting to the old one, but with the :directory set to a different value. However, I'm not sure what "deepest n" would mean in the presence of :wild-inferiors.
<gendl>
right, be he wants the first parts (i.e. the deepest parts), not the last parts.
<gendl>
(I assume he. Maybe I should say he/she/it).
<gendl>
or he/she/they
<gendl>
or just they
Kundry_Wag has joined #lisp
<gendl>
anyway so much for my attempts at political correctness. Anything we can do to encourage more females in here is welcome. ELS needs more females on its review committees etc. Let's start a slogan "Lisp is for girls."
<gendl>
(I think Guido wore a t-shirt "Python is for Girls." We need to steal that.
Kundry_Wag has quit [Ping timeout: 252 seconds]
longshi has quit [Ping timeout: 246 seconds]
<_death>
just wait until someone proposes to rename :wild-inferiors
<gendl>
is freenode accessible in China? I’m guessing no.
pierpal has joined #lisp
nckx has quit [Ping timeout: 240 seconds]
<shka_>
oh my, pathnames are though
<shka_>
i think that function that does that should be in cl-fad
<shka_>
anyway, thanks guys
Bike has joined #lisp
nowhere_man has joined #lisp
shrdlu68 has quit [Ping timeout: 245 seconds]
nckx has joined #lisp
marvin2 has joined #lisp
Kundry_Wag has joined #lisp
razzy has joined #lisp
Kundry_Wag has quit [Ping timeout: 246 seconds]
<gendl>
shka_: also check uiop. One of uiop's stated goals is to obsolete little utility libraries like cl-fad.
<gendl>
shka_: where are you from, if I may ask?
suskeyhose has joined #lisp
flazh has quit [Ping timeout: 240 seconds]
jmercouris has quit [Remote host closed the connection]
<shka_>
gendl: poland
phoe has quit [Ping timeout: 252 seconds]
<Demosthenex>
so i'm trying to make my restful api downloader multithreaded with lparallel, i've got most of it ok except the rate limiter. each rest request return how many requests against the limit in the headers, and i was calculating a delay based on the returned value which is local for threads, but now i'm hitting the rate limit constantly.
orivej has quit [Ping timeout: 252 seconds]
<Demosthenex>
i'm kind of uncertain what to add to that that's threadsafe
<Demosthenex>
though maybe there isn't much point, given i can hit the rate limit without threads, i spend most of my time waiting.
<|3b|>
if just a shared number is enough (and you can find a portability lib or limit implementations), atomics, otherwise a mutex and shared variable
<|3b|>
yeah, if your rate limit is less than processing rate, threads just give you a bit higher burst rate
<|3b|>
(and a lot more complexity)
<Demosthenex>
yeah.
orivej has joined #lisp
kushal has quit [Remote host closed the connection]
kushal has joined #lisp
<Demosthenex>
given that i was calculating a sleep from the response header, i thought i was safe :P
<|3b|>
xificurC: well, it's linux implementations is 9 loc if you want to reimplement it :p
nbunjeva1 has joined #lisp
<xificurC>
|3b|: you mean inline it :)
<|3b|>
CL the language has no concept of CPU or memory to give you counts
<AeroNotix>
xificurC: sbcl doesn't provide anything for this itself any way
<dlowe>
just run /usr/bin/nproc :)
<AeroNotix>
xificurC: in the SBCL source they use the exact same syscall on Linux to achieve this but there are no public interfaces that expose this value directly.
<AeroNotix>
but it's an internal test module (for threading). So it's not available
<AeroNotix>
.
nbunjeva3 has quit [Ping timeout: 252 seconds]
igemnace has joined #lisp
<AeroNotix>
xificurC: if you're dead against using a library for this (don't know why you would be realistically, we have lots of RAM these days) then you can c+p the code which sbcl uses?
<AeroNotix>
but then you're not portable
<Xach>
/lastlog xach
<Xach>
whee
<xificurC>
AeroNotix: just feels a bit left-pad-ish, that's all
<AeroNotix>
xificurC: welcome to lisp dependencies
<AeroNotix>
xificurC: It's not a bad thing. If sbcl itself provided a function for this your interface to it would be exactly the same
<AeroNotix>
eh, whatever. I'm not dying on this hill
<xificurC>
there's no hill, don't worry
<AeroNotix>
xificurC: the main "issue" is that simply the standard didn't mandate a function for exposing the number of CPUs on the host OS. So it doesn't exist in implementations. That library provides that function.
Lycurgus has quit [Quit: Exeunt]
<AeroNotix>
I don't know what else that library should do aside from what it says on the tin :)
<AeroNotix>
Though I am somewhat surprised there isn't a little sb-* function for it
<AeroNotix>
Can someone run this for me: (progn (ql:quickload :stmx) (and (stmx:hw-transaction-supported?) (= stmx.asm:+transaction-started+ (stmx.asm:transaction-begin)))) ?
<AeroNotix>
uh, someone on sbcl <1.4.10 FYI, stmx is broken for 1.4.10 at the minute
flazh has joined #lisp
phoe has joined #lisp
dyelar has joined #lisp
<xificurC>
AeroNotix: doing, on 1.4.5. btw that progn doesn't work, the reader barks package does not exist
<xificurC>
AeroNotix: returned NIL
Kundry_Wag has joined #lisp
<xificurC>
hw-transaction-supported? returns NIL
<AeroNotix>
xificurC: derp, yeah I already had the package loaded in my repl. Inside the progn that won't work you're right
<AeroNotix>
xificurC: okay your processor doesn't support RTM any way.
astalla has quit [Ping timeout: 246 seconds]
<xificurC>
|3b| shka_, I managed to rewrite my code to indeed run in parallel, all cores are running on 100% after the changes. The only problem is that it's giving wrong results :) http://sprunge.us/lk0LeM
Kundry_Wag has quit [Ping timeout: 272 seconds]
<|3b|>
xificurC: does that give you any results at all?
orivej has joined #lisp
<shka_>
xificurC: not what i meant
<shka_>
it should deadlock imho
<|3b|>
no, looks like it should just not get any results
<shka_>
also (lparallel:future (lparallel.queue:push-queue (ignore-errors (parse-line line)) queue))
<shka_>
this is not what i meant
jkordani has joined #lisp
<shka_>
it should be (lparallel.queue:push-queue (lparallel:future ...
<shka_>
also, no source and sink threads
<|3b|>
it fills queue with futures, then remaining futures just sit there doing nothing, and nothing adds to queue
<Shinmera>
shka_: I don't think pathname-utils already has what you wanted, but it has very similar functions.
<|3b|>
actually i guess you get 100 results (and any more that happen to fit in before read finishes)
<xificurC>
I get many results, tens of thousands, but the numbers are always a bit different
<|3b|>
maybe not, back to "i don't use it enough to tell what it does with that odd code" :)
<shka_>
xificurC: you are doing it wrong, anyway
<xificurC>
shka_: I know, I wrote it that way first, but I can't do it that way :)
<|3b|>
yeah, it isn't waiting for stuff to finish processing
<xificurC>
I need the main thread to handle the add-* calls because they setf the variables
<xificurC>
but the queue is empty in the end, look at the assert
shifty has quit [Ping timeout: 246 seconds]
<shka_>
that does not sound right
<shka_>
first of
<shka_>
you can do popping stuff on your main thread
<shka_>
and it will work just fine
<shka_>
secondly, this should not work any faster then single thread
<shka_>
eh, just let me write it
<xificurC>
firstly, yes, but I can't run over the whole stream before starting processing.
<xificurC>
secondly, it does, it is around 4x faster, just not correct :)
* |3b|
tries finishing writing it again too :p
<xificurC>
just remember that add-slowlog and add-groklog modify slowlogs/groklogs by setfing on it
<|3b|>
on yeah, you still close over loop variables, don't do that :p
<xificurC>
so that part of the code must run on the main thread
<|3b|>
(especially with threaded stuff)
<xificurC>
|3b|: why
<|3b|>
because it might not be a new binding each iteration, so could get overwritten by next iteration before it is processed
dale_ has joined #lisp
dale_ is now known as dale
<xificurC>
what does one do then?
orivej has quit [Ping timeout: 244 seconds]
<|3b|>
(let ((line line)) (lambda () .... line ...))
<|3b|>
create a new binding every time yourself, so the closure captures that instead of the (possibly) reused loop variable
<djeis[m]>
You need to explicitly create a new local around the future.
<xificurC>
hey, (let ((line line)) ...) solved all the issues
<xificurC>
where can I read about this issue? I'd need a more detailed explanation because I didn't understand it from the short description you gave |3b|
<shka_>
xificurC: lparallel manual for instance
<|3b|>
do you know C?
<xificurC>
|3b|: I "know" C. I didn't write much of it but can read it to some extent. I know some assembly too if that helps
<|3b|>
imagine if your loop allocates a buffer once, then stores a pointer to that buffer each time it sends work to other thread, then next iteration overwrites the buffer, so all the threads are looking at same data
Kundry_Wag has joined #lisp
<xificurC>
shka_: you mean the part about "dynamic variables and worker context"? Is the loop variable dynamic?
grumble has quit [Remote host closed the connection]
<|3b|>
that's effect you get when loop only creates 1 binding and just assigns to it every iteration. closure saves that binding, so all threads are looking at same thing (though you are more likely to move it into a new binding within the thread than passing around pointers in c, so less likely to see partial updates)
<xificurC>
shka_: you don't force the futures anywhere, I guess you just forgot
<djeis[m]>
The loop variable is lexical, it just might be allocated once at the start of the loop and then shared by each future.
<shka_>
xificurC: yeah, i am at work, kinda in the rush
<|3b|>
other option (and only correct one in C) is to allocate a new buffer (and new pointer) every iteration
<xificurC>
ok, starting to understand
<|3b|>
if LOOP is implemented to create a new binding every iteration, then each closure would only see the binding active when the closure was created, giving same effect
<|3b|>
CL doesn't specify which way LOOP is implemented, so you have to manually create a new binding yourself to make sure
<|3b|>
(equivalent to manually making a copy of the pointer to pass to other threads in the single-buffer C example)
grumble has joined #lisp
<|3b|>
though in CL, creating a binding doesn't make a copy, so not exactly the same thing
<xificurC>
shka_: your implementation keeps order of lines. Forcing each future in this fashion might block the main thread for quite some time, no? Depending on how lparallel handles the threads there might be more stalls. Although the main thread doesn't do much work so there shouldn't be much time lost
<|3b|>
(if in CL you reused the same array every iteration, that would actually be closest to the single-buffer C example, and still wouldn't work even with new bindings for each closure, since each binding would still store same array object)
<|3b|>
xificurC: forcing the future will either parse it in that thread, or get results calculated by a worker thread
<|3b|>
so it will sometimes duplicate some of the work done by workers if it happens to catch up with them
<xificurC>
I think my solution can't deadlock, don't see how that would happen
<makomo>
jackdaniel: i guess one could infer it from "the expansion of a define-modify-macro is equivalent to the following: (...)" but i always end up wondering if they thought about all of the implications of the example
<makomo>
jackdaniel: right :-)
<jackdaniel>
which is not an example (so it is binding as part of the standard)
<xificurC>
|3b|: "A future is a promise which is fulfilled in parallel. When a future is created, a parallel task is made from the code passed."
<makomo>
jackdaniel: ah, true. ok
<jackdaniel>
examples per se are not part of the standard, but this is not an example but purt of the description
<jackdaniel>
part*
<xificurC>
based on that I would assume a future will be fulfilled by BODY, and forcing means blocking waiting for it. In this case I'm blocking waiting for it's result of pushing on the queue
<|3b|>
makomo: i don't see anything in the spec requiring it to name anything, function or otherwise :)
<xificurC>
s/it's/its/
<|3b|>
xificurC: yeah, sounds reasonable to me
<makomo>
|3b|: good point :D
<|3b|>
xificurC: ah, i think problem is if you fill queue before any work finishes?
<|3b|>
try adding a sleep before the parse-line call
<xificurC>
|3b|: problem? The code is running fine :)
<|3b|>
i mean try adding sleep to see the problem
<|3b|>
(if i'm correct that there is a problem)
<|3b|>
and maybe print a dot or something to see if it gets stuck, since large enough sleep might make it take too long overall if it doesn't
lumm has joined #lisp
<|3b|>
problem i see is that if queue fills up and you try to push onto it, you will block waiting for something to pop the queue, which happens on the blocked thread
<xificurC>
the push is in a future
scymtym has quit [Ping timeout: 260 seconds]
<xificurC>
not on the main thread
lumm has quit [Client Quit]
<|3b|>
ok, so you are back to lacking rate-limiting on input
<|3b|>
so instead of blocking in that case you load whole file into ram
xrash has joined #lisp
<|3b|>
main thread is creating futures as fast as it can, each storing a line of input
gector has quit [Read error: Connection reset by peer]
gector has joined #lisp
<xificurC>
yes, that would be the case if the popper was slower than the pusher
<xificurC>
but I checked and the memory usage is the same
<xificurC>
*same as the single threaded version
gravicappa has joined #lisp
orivej has joined #lisp
varjag has quit [Quit: ERC (IRC client for Emacs 24.5.1)]
<xificurC>
sleepy version works too. lots of dots :)
<|3b|>
same memory use?
<xificurC>
yes, +10MB
<shka_>
it is kinda complicated and ugly, though
SaganMan has joined #lisp
warweasle has joined #lisp
<xificurC>
indeed
meepdeew has joined #lisp
pierpal has quit [Ping timeout: 252 seconds]
scymtym has joined #lisp
Kaisyu has quit [Quit: Connection closed for inactivity]
SenasOzys has joined #lisp
frgo has quit [Remote host closed the connection]
Achylles has joined #lisp
gravicappa has quit [Ping timeout: 245 seconds]
orivej has quit [Ping timeout: 252 seconds]
Pixel_Outlaw has joined #lisp
Kundry_Wag has quit [Remote host closed the connection]
lnostdal has quit [Ping timeout: 272 seconds]
meepdeew has quit [Remote host closed the connection]
lnostdal has joined #lisp
SaganMan has quit [Quit: WeeChat 1.6]
SaganMan has joined #lisp
Kundry_Wag has joined #lisp
frodef has quit [Ping timeout: 260 seconds]
Kundry_Wag has quit [Remote host closed the connection]
lnostdal has quit [Remote host closed the connection]
lnostdal has joined #lisp
frgo has joined #lisp
orivej has joined #lisp
frgo has quit [Ping timeout: 240 seconds]
pierpal has joined #lisp
Inline has joined #lisp
Inline has quit [Read error: Connection reset by peer]
Inline has joined #lisp
gpiero has quit [Remote host closed the connection]
pierpal has quit [Read error: Connection reset by peer]
orivej has quit [Ping timeout: 260 seconds]
frgo has joined #lisp
mkolenda has quit [Remote host closed the connection]
mkolenda has joined #lisp
heisig has quit [Ping timeout: 260 seconds]
pjb has joined #lisp
wooden_ has joined #lisp
trittweiler has quit [Ping timeout: 252 seconds]
Lycurgus has joined #lisp
Spaceman77 has joined #lisp
nbunjeva1 has quit [Quit: leaving]
nsrahmad has joined #lisp
warweasle is now known as warweasle_afk
<Spaceman77>
Hello. i am wondering if lisps can be used in image processing
<Spaceman77>
I know it's a stupid question, of course it can
<Shinmera>
The answer is yes
<Spaceman77>
But where should i look if i want to analyze an image pixel-by-pixel and apply filters
<Spaceman77>
I intend to create a `simultaneous localization and mapping` system that uses only 1 or 2 cameras. I try to create 3D maps from these images and determine the position of the robot
<Spaceman77>
I've heard lisp is the language to do impossible things :)
frodef has joined #lisp
<Spaceman77>
I am still considering using C++ for it, but Lisp made me curios
shrdlu68 has joined #lisp
<dim>
you might like clasp, an experimental CL compiler that uses LLVM so that you can build apps in both C++ and CL
<dlowe>
Spaceman77: that does mean you don't have to get weird right away
igemnace has joined #lisp
<dlowe>
Spaceman77: many people just want to use the new toys immediately and get frustrated
<dlowe>
but you can ease into it
<Shinmera>
dim: anyway the problem is primarily that a lot of lisp libraries don't run on clasp yet
<dim>
Spaceman77: it got me about a week of everyday programming to get past the parens/syntax and the basics stuff, including the *whole* syntax and a first approach at the standard api
<Spaceman77>
Hm, what dialect? CLisp?
<Shinmera>
dim: due to bugs and lack of compat library support
<djeis[m]>
Spaceman77: You actually get used to the syntax fairly quickly, and after that it’s not that much weirder than any other language for the usual stuff. The difference is the rabbit hole can go much deeper :)
<shrdlu68>
Spaceman77: In what way do you find it alien?
nydel has quit [Ping timeout: 252 seconds]
<dlowe>
Spaceman77: feel free to join us in #clschool :)
<Spaceman77>
shrdlu68: Nothing and everything. I've never seen a single language being praised so much by some high profile people
<Spaceman77>
I've been meaning to read SICP for half a year now, but i keep putting it aside
<Spaceman77>
I just want to become a better programmer.
<shrdlu68>
Spaceman77: Ah, the "If you're so smart why are your poor" paradox for Lisp.
<djeis[m]>
Spaceman77: well, every language you learn will help with that, especially languages that work differently than you’re used to. Lisp is definitely a good choice for expanding your horizons tho.
<Spaceman77>
I like programming. I like solving problems and thinking about abstractions and whatnot until what i have is the most elegant and readable solution.
<djeis[m]>
Then at the least you’ll find lisp very interesting :)
<drmeister>
Hello
<drdo>
Spaceman77: Have you watched the SICP lectures?
<Spaceman77>
drdo: No
<drdo>
Definitely nice, even if you are an experienced programmer
<djeis[m]>
And they’re all up on YouTube.
<Spaceman77>
I want to learn lisp. I am studying and intend to work in robotics, and it seems like such a "doomed" field. Every problem is almost impossible to solve. There is no true AI, sensors lie etc. etc. etc.
<dlowe>
That's pretty much every software domain everywhere
<Spaceman77>
dlowe: i wouldn't say so
<Spaceman77>
blogs and e-commerce are trivial
<Spaceman77>
apps on your phone that gets you food or taxi exist
<dlowe>
And they're riddled with problems
<Spaceman77>
sure
<dlowe>
all of them
<Spaceman77>
but the populace still uses them and interacts with them
<Spaceman77>
wheres your robot?
<dlowe>
vacuuming my house, I'm guessing
<Shinmera>
got plenty of those in factories
<Spaceman77>
when will it make you breakfast? go to the mall for you?
<dim>
the more you dig into the details, the more it seems impossible that anything works, usually saying that a domain is more complex than another one only means that you've been digging more in that first domain ;-)
<dlowe>
Managing your expectations is part of life
<dlowe>
My compiler is not likely to make me breakfast either
<dlowe>
(other than providing an adequate heat source)
<Spaceman77>
I may sound frustrated/ that i am venting
nsrahmad has left #lisp ["Leaving"]
shrdlu68 has quit [Ping timeout: 245 seconds]
<Spaceman77>
I just want to tackle some impossible problems. Is Lisp an adequate language to tackle impossible problems?
<pjb>
Spaceman77: yes.
<pjb>
Spaceman77: AFAIK, it's the only one. (well, perhaps ometa too, but I'm cheating here).
<aeth>
Spaceman77: What you get in Lisp is being able to trivially do anything at compile time that you can do at run time.
<aeth>
I would say "Well, within reason." but you can even get input in the user at compile time, but then you basically turn compile time into run time.
<Spaceman77>
Oh, this is curious. I've heard of this wacky stuff you can do with Lisp. Is it true that Lisp can interpret itself and change itself?
<pjb>
(defparameter *version* #.(progn (format *query-io* "What version are we compiling? ") (finish-output *query-io*) (read-line *query-io*)))
<pjb>
Spaceman77: Not Common Lisp. Modifying Common Lisp is undefined. Nasal daemons.
<Spaceman77>
Scheme?
<pjb>
Nope. It's worse.
<dlowe>
Spaceman77: The idea behind lisp is that your code can execute inside the compiler.
<pjb>
Scheme doesn't even have proper symbols…
<Spaceman77>
weird
<dlowe>
Spaceman77: it's like having a code generator built into the system
<Spaceman77>
What are the implications of lisp executing in a compiler?
schweers has quit [Ping timeout: 250 seconds]
<Spaceman77>
What can i do with this?
<dlowe>
Spaceman77: Well, it means that we don't have to have another language when we want to write code that operates on code
warweasle_afk is now known as warweasle
<Spaceman77>
So Lisp can operate on its own source code?
<dim>
Spaceman77: e.g. in pgloader I benefit from that by parsing the command language that the users give me into a lisp program, that I compile at run-time and then execute
<dlowe>
Spaceman77: the source code is made of nested lists. The text files that are parsed into nested lists are a convenience for humans.
<|3b|>
Spaceman77: are you familiar with c++ template metaprogramming? if so, imagine being able to use a sane programming language instead of template hacks
<dim>
if you've been doing some C/C++ before, imagine that the preprocessor would be in C/C++ rather than this #define pseudo-language
<dlowe>
Spaceman77: therefore, manipulating the soruce code programmatically is manipulating nested lists
<pjb>
Spaceman77: Lisp is a language, it doesn't have any source.
<pjb>
Spaceman77: try to refine your concepts!
<pjb>
Spaceman77: read some.
<dlowe>
okay, I'll step away from the dogpile
<dim>
pjb: you're asking a lot to a newby discovering lisp
<Bike>
putting assembly into different languages is kind of a pain. you know how gcc does it, right? this weird shit with strings and dependencies? not great
dented42 has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
scymtym has joined #lisp
Roy_Fokker has joined #lisp
je4i has joined #lisp
gravicappa has quit [Ping timeout: 245 seconds]
phoe has quit [Ping timeout: 244 seconds]
Fare has quit [Ping timeout: 260 seconds]
<no-defun-allowed>
Is the ELS invite only?
Colleen has quit [Remote host closed the connection]
<frodef>
They allowed me in at one point, so I cannot imagine so.. :)
<Xach>
no-defun-allowed: no. you must register to attend but registration is open to anyone.
<no-defun-allowed>
Ok, thanks.
Bike has quit [Ping timeout: 256 seconds]
lemonpepper24 has joined #lisp
<gendl>
no-defun-allowed: The ECLM was (and likely will continue to be if it happens again) invite-only for speakers. But of course not for attendees.
<gendl>
ELS doesn't generally put out invitations for presenters, it accepts paper and demo submissions and selects enough to fill the program.
AroPar has joined #lisp
AroPar has quit [Max SendQ exceeded]
AroPar has joined #lisp
varjag has quit [Quit: ERC (IRC client for Emacs 26.1)]
AroPar has quit [Client Quit]
Essadon has quit [Quit: Qutting]
orivej has joined #lisp
nowhere_man has quit [Remote host closed the connection]
trn has joined #lisp
nowhere_man has joined #lisp
mrSpec has quit [Ping timeout: 276 seconds]
Lycurgus has quit [Quit: Exeunt]
dented42 has joined #lisp
Fare has joined #lisp
shifty has joined #lisp
<no-defun-allowed>
ok then
<no-defun-allowed>
good thing i don't have anything to present
<jasom>
okay, any volunteers to watch my kids in california while I fly to Europe?
dented42 has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
dented42 has joined #lisp
dented42 has quit [Client Quit]
<no-defun-allowed>
^^ but s/kids/cats s/california/melbourne
nowhere_man has quit [Remote host closed the connection]
PuercoPop has quit [Ping timeout: 240 seconds]
nowhere_man has joined #lisp
lnostdal has quit [Read error: Connection reset by peer]
astronavt has joined #lisp
mange has joined #lisp
Denommus has quit [Remote host closed the connection]
quazimodo has joined #lisp
anewuser has joined #lisp
<jasom>
no-defun-allowed: fly your cats to california and they can watch my kids! 8-)
<no-defun-allowed>
okay then
robotoad_ has quit [Quit: robotoad_]
dented42 has joined #lisp
Fare has quit [Ping timeout: 252 seconds]
dented42 has quit [Client Quit]
<oni-on-ion>
yes
meepdeew has joined #lisp
je4i has quit [Quit: ERC (IRC client for Emacs 25.3.1)]
meepdeew has quit [Ping timeout: 252 seconds]
ebrasca has joined #lisp
<adlai>
no-defun-allowed: however, i do believe they only allow declamations, macros, and long-form method combinations.
<no-defun-allowed>
okay then
<adlai>
however, you should have no trouble if you send a human representative for your defun.
<no-defun-allowed>
ok i hired the guy that sometimes steals my laptop to represent my function that adds two to a number
<Demosthenex>
i'm loading swank in a separate process from emacs on the command line, and connecting with slime to it. how can i tell swank not to make a repl on that terminal? it's conflicting with reading sb-sys:*stdin*
pjb has quit [Ping timeout: 240 seconds]
robotoad has joined #lisp
deba5e12 has quit [Read error: Connection reset by peer]