<sb0>
the general consensus seems to be that fds and other OS resources should be explicitly closed in python
sb0 has quit [Quit: Leaving]
<whitequark>
yes
<whitequark>
you shouldn't rely on GC, as no guarantees are provided
<whitequark>
and it's actually quite easy to exhaust limited OS resources without triggering a GC run
<whitequark>
especially on Windows, I think msvcrt has some ridiculously small amount of fds by default, like 64
<whitequark>
or maybe 256
<rjo>
none of this is a problem here.
key2 has joined #m-labs
<rjo>
then the general consensus would also be to not use a GC at all because its use means not freeing memory explicitly.
<rjo>
but please feel free to patch that in bit2bin
<whitequark>
(not use a GC) no. the problem is not "not freeing resources explicitly"
<whitequark>
the problem is that GC can and will respond to memory pressure by freeing stuff, but it has no way to respond to fd pressure
<rjo>
yes. there is also no guarantee that memory pressure can be resolved that way. its the same problem.
<whitequark>
absolutely not
<whitequark>
you are tying two different kinds of resources into one
mumptai has joined #m-labs
<whitequark>
I have no idea why are you even arguing about this, it's a well-known problem that is trivial to demonstrate (open /dev/null in a loop)
<whitequark>
you will never have a python process dying from OOM if you allocate and discard an empty class in a loop.
<rjo>
nope. look at this very case. the fd will be closed eventually. same as with memory and a gc. it will be freed eventually.
<rjo>
this is a script
<whitequark>
gc will respond to memory pressure by collection
<whitequark>
gc will not respond to fd pressure in any way
<rjo>
what's the point?
<whitequark>
the point of what?
<rjo>
of that distinction with respect to the case at hand.
<rjo>
there is no fd pressure. there is no memory pressure.
<whitequark>
I don't care what the case at hand is, just don't write such sloppy code
<whitequark>
especially when there is no complexity from doing it right
<rjo>
you should make an effort to figure out what the topic of a discussion is before joining a discussion
<whitequark>
no. I am aware what the case at hand is. I just don't care about it.
<rjo>
don't make sloppy comments.
<rjo>
why should anybody care about your comments if you don't care about their issue?
<whitequark>
*shrug* you are free to continue writing crappy code then.
<rjo>
isn't it common practice to free resources lazily in scripts?
<rjo>
if you feel so offended, go ahead and improve it.
<rjo>
don't whine about it
<whitequark>
oh, I'll improve it, but education is also important to prevent this from coming up in the future
<rjo>
don't make parrot statements like "not explicitly closing fds is bad" if they are irrelevant for the case at hand.
<rjo>
sure. i do know when to close files.
* mumptai
never saw a script actually freeing any resources, and a few actually crashing in corner cases because of that
<whitequark>
not explicitly closing fds is bad irrelevant of the case where it happens
<rjo>
but would you close stdin just because you don't need it and you want to free resources?
<whitequark>
today it's a script, tomorrow it's in a library
<larsc>
and in one week it's in the kernel!
<rjo>
if you would write every shell one-liner with the option of it being useable as a library function in mind, we would not be were we are.
<whitequark>
the stdin comparison is irrelevant. stdin is a global resource, and without any additional actions it already has global lifetime
<rjo>
so closing stdin does not free an fd?
<whitequark>
as for shell one-liners: yes. we would have lived in a much better world if the unix "worse is better" ideology was not applied
<whitequark>
I am not arguing that closing stdin does not free an fd
<whitequark>
I am rather saying that a function that allocates a file and a file object should ensure that their lifetimes are the same as a part of its contract
<whitequark>
... because not doing this leads to the problems I described above
<whitequark>
there's no function that allocates stdin, how is stdin relevant?
<whitequark>
a file descriptor and a file object*
<whitequark>
and this is not just about fds, this applies just as well to any sort of OS resources or even external library resources
<whitequark>
e.g. allocating *memory* via imagemagick that's invisible to the GC of your language is a common cause of OOMs when you're doing image processing
<rjo>
yeah. but this is artificial. if you know that your script exits soon, you can be epsilon-sloppy and lazily close the fd, either if gc hits it (by refcount or by time) or if the script exits.
<whitequark>
yes. which is why I didn't call your code "broken", just "sloppy". it works, and it is bad practice
<whitequark>
I suppose it wouldn't be a problem if Python actually guaranteed refcounting
<rjo>
sure. about the severity level of spelling errors.
* whitequark
says something about HTTP Referer:
<rjo>
(spelling errors that do not impact the content) that is.
<rjo>
does refcounting not work in this case?
<whitequark>
ok. fine. I concur that this isn't a problem in CPython, and anything we write is unlikely to end up anywhere except CPython, because those implementations barely work.
<rjo>
HTTP Referer is even more harmless: It eventually leads to the other side (the language) shifting.
<whitequark>
it's not harmless, I've seen bugs stemming from people trying to spell it correctly in projects I worked on
<rjo>
that sentence of yours describes a lot of the reasons why python is successful.
<whitequark>
which?
<whitequark>
lack of alternative implementations?
<rjo>
ack. but it might well be "eventually harmless" because the originally correct side concedes.
<rjo>
no the sentence ".. this isn't a problem ... barely work."
<whitequark>
I mean, yes, I've never heard of anyone actually using PyPy in production, and almost anyone with Jython
<rjo>
"it is easy to get somethig that barely works in python and the problems in doing so are not that dramatic"
<whitequark>
and that's a fairly substantial problem with Python if you ask me. a few years ago I wrote some graph-crunching in Ruby and then was able to scale it 3x by adding three more cores and running on JRuby
<whitequark>
fortunately irrelevant to ARTIQ
<whitequark>
a more relevant part is that Java has excellent tooling, such as for profiling CPU, time, etc. I could really use that for the compiler.
<rjo>
true. but IME in practice average-joe is completely swamped in problems of his own creation and 3x speed improvement is not what he is looking for nor what he should look for.
<whitequark>
CPU, memory, etc*
<whitequark>
jvisualvm is specifically what I'm thinking of, it was an immense help and nothing I've seen for bare Python comes close
<rjo>
... and worse: it seems that average-joe creates fewer (big) problems for himself in python than in other languages.
<rjo>
never heard of jvisualvm
<whitequark>
I'm completely unconvinced about Python helping to eliminate problems
<rjo>
i was bitterly complaining to a google guy in boulder that was in the java/android/dalvik team and i wanted to know why they went for java. and they said similar things (tooling, maturity, correctness...)
<whitequark>
every time I refactor the compiler, I spend more than half of the time fixing trivial bugs produced by inattentiveness that could have been caught by a basic (parametric) type system
<rjo>
not eliminating problems. correlating with a less likely creation of problems due to user misunderstanding/inexperience/stupidity.
<rjo>
yes. it generally does sound like "python was not designed to be compiled statically" (if that last part is what we are doing)
<whitequark>
that's not quite what I mean. what we are doing is something I generally would argue *against*
<whitequark>
i.e. using a type system for performance
<whitequark>
ARTIQ Python is a highly unusual case when that's actually justified
<whitequark>
the goal of a type system is first and foremost to ensure correctness, and dynlangs have many features, lack of type system a comparatively minor one, that make ensuring correctness very difficult
<whitequark>
such as encouraging aliasing and mutation, making immutable objects complicated to use, and specifically in Python the rampant use of exceptions
<rjo>
ack.
<whitequark>
the story of Python is rather similar to story of Go, as is its reception
<whitequark>
people *love* go.
<whitequark>
and python.
<whitequark>
in both cases you see that a major part of the language's success story is that it explicitly avoids making things artificially harder to appease a use case that most don't care about
<rjo>
yet many people -- while aware of these issues to varying degrees -- feel that they could never get that much shit done with any other language than python.
<whitequark>
and in both cases they have performed absolutely no steps towards making things, if you will, artificially easier
<whitequark>
a really indicative example in Go is the "res, err := foo(); if err != nil { ... }" pattern.
<whitequark>
it's present in every single API and while it's not artificially hard (like C's errno or out parameters would be), it is also not artificially easy (like pattern matching on a Result type, or even to some degree exceptions)
<rjo>
true.
<whitequark>
as for "they could never get that much shit done in any other language". well. i used to argue that no language will ever match the productivity of Ruby.
<rjo>
ruby would probably be the king if it had been present and well known a few years earlier.
cyrozap has quit [Ping timeout: 276 seconds]
<whitequark>
for the purposes of this discussion ruby and python are wholly interchangeable
<whitequark>
they have so little serious difference in their design that they're pretty much the same thing.
<rjo>
yes
key2 has quit [Ping timeout: 252 seconds]
<whitequark>
and the only thing I can say about such a statement about productivity is that it is surely understandable, especially if one's coming from C or C++ (I started with C++), but it's more a sign of relative personal immaturity
<whitequark>
than any particular massive benefit of the design
<rjo>
no. for python in science it is what your peers speak (mostly) and where the libaries are.
<rjo>
this is not a personal decision
<whitequark>
the key part is "in science". the use of programming in science that does not specifically concern itself with computing is a relatively narrow domain
<whitequark>
for example, you would not be productive writing a distributed, fault-tolerant system in Python, anywhere near as you would doing it in Erlang. but you also wouldn't be doing one in all likelihood
<rjo>
absolutely. but if somebody asks "which language?" the answer needs to be "where?" or "for what?"
<whitequark>
yes.
<whitequark>
but then we aren't really talking about the language.
<whitequark>
if all your peers speak are fortran 77, you are going to be writing fortran 77
cyrozap has joined #m-labs
<rjo>
and that's why i just never stumble over a problem for myself where i could use go or rust or haskell even though i would be totally happy to spend the 20% overhead in time for learning the language.
<rjo>
no. for f77 the "problems" with it were big enough to start a/many new langauges.
<cr1901_modern>
Like the fact it was meant for punched cards :)
<whitequark>
we still use some punchcard languages. G-code, for one. and that academic image storage format, I forgot how it was called
<whitequark>
actually, wasn't hdf such a language?
<cr1901_modern>
Fair point. My main problem with Fortran is there are sorely missing built-ins. There is no sort; you either do it yourself, or pay NAG for one that can deal with arbitrary dimensions.
<whitequark>
rjo: you could totally reap the benefits of Rust while writing CPython extensions
<cr1901_modern>
I don't want to spend my time coding e.g. a sort. And libraries for arbitrary dimension transformations are limited
<whitequark>
there's a no-boilerplate Ruby bridge, not sure if someone already wrote that in Python
<whitequark>
for*
<rjo>
whitequark: maybe for that. but for me that would compete with doing it cython.
<rjo>
and if the extension is numerical i would probably try julia.
<whitequark>
last time I checked julia barely worked and no one especially cared about that
<rjo>
i have no opinion of julia. the julia guys just constantly go to the conference i go to. that's why i feel urged to form an opinion by trying it.