tomchapi_ has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
Oliphaunte has quit [Read error: Connection reset by peer]
Oliphaunte has joined #crystal-lang
Oliphaunte has quit [Remote host closed the connection]
Oliphaunte has joined #crystal-lang
tomchapin has joined #crystal-lang
onionhammer has joined #crystal-lang
soveran has joined #crystal-lang
soveran has quit [Ping timeout: 276 seconds]
zodiak has joined #crystal-lang
zodiak_ has quit [Ping timeout: 246 seconds]
tomchapin has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
pawnbox has joined #crystal-lang
franzejr_ has quit [Remote host closed the connection]
pawnbox has quit [Ping timeout: 244 seconds]
Oliphaunte has quit [Remote host closed the connection]
Oliphaunte has joined #crystal-lang
Raimondii has joined #crystal-lang
Raimondi has quit [Ping timeout: 240 seconds]
Raimondii is now known as Raimondi
pawnbox has joined #crystal-lang
pawnbox has quit [Ping timeout: 240 seconds]
Oliphaunte has quit [Remote host closed the connection]
Oliphaunte has joined #crystal-lang
Oliphaunte has quit [Ping timeout: 240 seconds]
Oliphaunte has joined #crystal-lang
pawnbox has joined #crystal-lang
pawnbox has quit [Ping timeout: 240 seconds]
soveran has joined #crystal-lang
soveran has quit [Ping timeout: 276 seconds]
Oliphaunte has quit [Remote host closed the connection]
Oliphaunte has joined #crystal-lang
pawnbox has joined #crystal-lang
pawnbox has quit [Ping timeout: 260 seconds]
Oliphaunte has quit [Remote host closed the connection]
pawnbox has joined #crystal-lang
pawnbox has quit [Remote host closed the connection]
pawnbox has joined #crystal-lang
pawnbox has quit [Ping timeout: 244 seconds]
pawnbox has joined #crystal-lang
Oliphaunte has joined #crystal-lang
Oliphaunte has quit [Ping timeout: 276 seconds]
soveran has joined #crystal-lang
soveran has quit [Ping timeout: 244 seconds]
pawnbox has quit [Remote host closed the connection]
pawnbox has joined #crystal-lang
pawnbox has quit [Remote host closed the connection]
pawnbox has joined #crystal-lang
Oliphaunte has joined #crystal-lang
soveran has joined #crystal-lang
Oliphaunte has quit [Ping timeout: 250 seconds]
snsei has quit [Remote host closed the connection]
snsei has joined #crystal-lang
snsei has quit [Read error: Connection reset by peer]
snsei has joined #crystal-lang
snsei has quit [Remote host closed the connection]
snsei has joined #crystal-lang
qard has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
snsei has quit [Remote host closed the connection]
Oliphaunte has joined #crystal-lang
Oliphaunte has quit [Ping timeout: 260 seconds]
qard has joined #crystal-lang
jmoriau has joined #crystal-lang
Philpax has quit [Ping timeout: 244 seconds]
crack08 has joined #crystal-lang
bjz has joined #crystal-lang
bjz has quit [Ping timeout: 244 seconds]
snsei has joined #crystal-lang
Oliphaunte has joined #crystal-lang
gettalong has joined #crystal-lang
snsei has quit [Ping timeout: 244 seconds]
gettalong has quit [Read error: Connection reset by peer]
gettalong has joined #crystal-lang
Oliphaunte has quit [Ping timeout: 276 seconds]
<gettalong>
Hi everyone! I was looking at Crystal this weekend and wanted to see how it performs in comparison to Ruby.
<gettalong>
There are some benchmarks out there that have Crystal in the league of C. I also found a benchmark of the JSON library at https://github.com/kostya/benchmarks
<gettalong>
This looking good I tried implementing a PDF lexer in Crystal to test whether it performs that good for my use case.
<gettalong>
Using the Json lexer/parser as example I came up with a solution that compiled and I ran some tests.
<gettalong>
The problem is now: The compiled Crystal program is not that much faster than the Ruby solution, ~9s for Crystal and 12s for Ruby...
<BlaXpirit>
gettalong, run it with --release; change `class` to `struct`
<gettalong>
Since this is my first time writing Crystal it probably (or surely) looks awful but that is not my main concern which is performance. Any tips or hints on how this code could be made faster? Or are there any docs out there which deal with parsing ?
<gettalong>
Thanks BlaXpirit - I compiled with "crystal compile --release src/hexapdf.cr", so this is taken care of.
<BlaXpirit>
gettalong, consume_float seems dangerous, because integers can overflow. best to store it as string then convert probably
<BlaXpirit>
gettalong, have you tried changing the small classes to `struct`?
<gettalong>
I have changed the classes to structs, no improvement
<gettalong>
Also, the code in consume_float etc. is taken from the Json library of Crystal and adapted to use UInt8 instead of Char, so it should probably be alright (if I didn't introduce any stray bugs)
bjz has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
<BlaXpirit>
:(
<gettalong>
My Ruby version uses the StringScanner class for parsing a binary string. What I noticed is that if I don't use the StringScanner in Ruby but a similar approach as Crystal does with JSON (or what I did in the Crystal version of my code), it is twice as slow, probably because I need to loop so much in the Ruby code.
<gettalong>
However, I thought that this should be no problem in Crystal because it gets compiled, so everything should be roughly at the same level
<gettalong>
Thanks for you input, BlaXpirit! The code is in there but I removed the comment
<BlaXpirit>
oh, I'm sorry, it really is there
<gettalong>
Btw. just reading my 21MiB test file byte by byte is 14x faster in Crystal than in Ruby but that is something I expected
<gettalong>
I think I will dig some more into existing files like json.cr that do this kind of parsing and see if I can get my code faster
Philpax has joined #crystal-lang
Oliphaunte has joined #crystal-lang
Oliphaunte has quit [Ping timeout: 258 seconds]
<jhass>
parse_until_whitespace_or_delimiter looks a bit dangerous, it allocates a string with almost each call. Do we actually do something similar in the JSON lexer?
<jhass>
could use char's in LITERAL_STRING_ESCAPE_MAP
<jhass>
445 should be a .delete if anything
<jhass>
parse_name looks optimizable too
<gettalong>
parse_until_whitespace_or_delimiter needs to create a string because this string is then needed to identify the keyword (parse_keyword) or the PDF name (parse_name) - I don't know how this could be done without allocating the string
<gettalong>
jhass: line 445: like this: str = str.delete("\0", "\t", "\n", "\f", "\r", " ") ?
<jhass>
no just like in Ruby
<jhass>
.delete("\0\t\n\f\r ")
<jhass>
but the list of keywords is finite and known at least, no?
soveran has quit [Remote host closed the connection]
<gettalong>
jhass: thanks! Yes, the list of keywords is finite and known
<gettalong>
but would that help? We would still need to read each byte and generate a string accordingly
<jhass>
well for starters you could seed the string pool with them
<jhass>
then they'll be static from the data section and not permanent heap allocations
soveran has joined #crystal-lang
<jhass>
but in theory you could build up a big nested case/when that returns literals or symbols
<jhass>
it might be possible to generate such with a macro, I'm not sure
<gettalong>
jhass: thanks for your input! I will look into that. One question though: How would pre-seeding the string pool help with heap allocations? Wouldn't I still need to build the string beforehand with MemoryIO or String::Builder before calling string_pool.get?
<jhass>
yes that'll actually probably not make it faster just lower on memory
<jhass>
well and thus a bit less GC work, so there's that
trapped has joined #crystal-lang
<gettalong>
okay, good to know - thanks
pawnbox has quit [Remote host closed the connection]
pawnbox has joined #crystal-lang
pawnbox has quit [Remote host closed the connection]
gettalong has quit [Quit: gettalong]
Oliphaunte has joined #crystal-lang
bjz has quit [Ping timeout: 244 seconds]
Oliphaunte has quit [Ping timeout: 244 seconds]
bjz has joined #crystal-lang
Davy_CC has quit [Ping timeout: 264 seconds]
Davy_CC has joined #crystal-lang
pabs has quit [Ping timeout: 264 seconds]
pabs has joined #crystal-lang
ssvb has quit [Ping timeout: 244 seconds]
pawnbox has joined #crystal-lang
endou_ has quit [Ping timeout: 252 seconds]
endou has joined #crystal-lang
snsei has joined #crystal-lang
gettalong has joined #crystal-lang
snsei has quit [Ping timeout: 250 seconds]
franzejr_ has quit [Read error: Connection reset by peer]
Oliphaunte has joined #crystal-lang
Oliphaunte has quit [Remote host closed the connection]
Oliphaunte has joined #crystal-lang
Oliphaunte has quit [Remote host closed the connection]
Oliphaunte has joined #crystal-lang
Oliphaunte has quit [Remote host closed the connection]
snsei has joined #crystal-lang
Oliphaunte has joined #crystal-lang
Oliphaunte has quit [Ping timeout: 240 seconds]
snsei has quit [Remote host closed the connection]
gettalong has quit [Quit: gettalong]
ponga has joined #crystal-lang
<RX14->
I'll probably work on a multipart/form-data interface for the standard library based on the existing shard tonight.
<FromGitter>
<sdogruyol> @RX14 that would be awesome
RX14- is now known as rx14
<rx14>
Yeah, your shard doesn't have a high-level interface really
<FromGitter>
<sdogruyol> which one are you talking about?
gettalong has joined #crystal-lang
<rx14>
You made the multi part shared right?
<rx14>
Or is there two
<rx14>
Shard*
<FromGitter>
<sdogruyol> actually i failed at making one :P
<rx14>
Oh, it doesn't work?
<FromGitter>
<sdogruyol> it's not working
<FromGitter>
<sdogruyol> guess i'm not good at writing parsers :/
<rx14>
Ah well I'll sort it out somehow
<FromGitter>
<sdogruyol> rx14 that'd be really great
ponga has quit []
Oliphaunte has joined #crystal-lang
no0p has quit [Ping timeout: 240 seconds]
no0p has joined #crystal-lang
SolarSailor has joined #crystal-lang
<rkeene>
Any dependency manager that depends on Git is broken :-/
<jhass>
shards happily takes a path option
soveran has quit [Remote host closed the connection]
<dom96>
rkeene: why?
<rkeene>
dom96, People who build packages do so from hosts that have no network access (to prevent external influences from changing the package)
<dom96>
ahh, so you mean package managers that can *only* work with Git are broken.
pawnbox has quit [Read error: Connection reset by peer]
pawnbox has joined #crystal-lang
SolarSailor has quit [Quit: My MacBook Pro has gone to sleep. ZZZzzz…]
SolarSailor has joined #crystal-lang
SolarSailor has quit [Client Quit]
<rx14>
rkeene: surely if there is no network access then there is no way of downloading dependencies at all, do you manually place them? If you're doing that why use a package manager at all?
pawnbox has quit [Remote host closed the connection]
Raimondii has joined #crystal-lang
pawnbox has joined #crystal-lang
Raimondi has quit [Ping timeout: 240 seconds]
<rkeene>
rx14, Right, that's the point, the dependencies have to be downloaded, packaged, and installed separately. If you use Git to store them then it's much harder to get a serializable object which can be used to create a package and verified to be the same thing later on (like, 10 years into the future)
SolarSailor has joined #crystal-lang
gettalong has quit [Ping timeout: 264 seconds]
Philpax has quit [Ping timeout: 276 seconds]
<rx14>
How would you propose to do it?
pawnbox has quit [Ping timeout: 272 seconds]
SolarSailor has quit [Client Quit]
Raimondii is now known as Raimondi
pawnbox has joined #crystal-lang
<rx14>
Besides, you don't have to package all the dependencies in crystal, in fact there's not much point
<jhass>
Using git actually avoids having to reimplement data verification in a custom package format, one could even easily reuse its signature support
<jhass>
a shards package that fetches the repos for offline usage into a well defined site directory should be fairly straight forward to implement too
snsei has joined #crystal-lang
soveran has joined #crystal-lang
soveran has quit [Changing host]
soveran has joined #crystal-lang
<rkeene>
jhass, For the hundreds of other packages, it's simple -- verify the SHA256 of the serialized archive (tarball, zip, etc) that holds a particular release
<rx14>
Why is that harder than verifying the HEAD of a git repo?
<rkeene>
The HEAD of the git repo won't be the same now as in 10 years, when you may be recompiling the package
<jhass>
shards.lock contains the git sha or git tag in question
snsei has quit [Ping timeout: 276 seconds]
<rx14>
Yes but that's like complaining the sha256 of the latest yarn all release won't be the same
soveran has quit [Ping timeout: 244 seconds]
<rx14>
Of course it won't, because there's been a new release
<rkeene>
Right, but you don't have any network access when building the package
<jhass>
you don't need to. git is not SVN
<rkeene>
rx14, But we don't want to want to use the "latest" release, we want to use the same release every time, exactly the same.
<rx14>
Exactly
<jhass>
git checkout <sha>
<rx14>
So you check out the same sha
soveran has joined #crystal-lang
<rx14>
And yojre guaranteed to get the same sourcr
<rx14>
If you're worried about the commit disappearing then mirror the repo
<rkeene>
I do mirror all my sources, on multiple levels -- which is another thing that is harder with git, since it's not a serialized format...
<rkeene>
And it's more difficult to do a shallow checkout of Git repo that is not HEAD, it may be fixed now but it never worked for me the the past
<rkeene>
(and git-archive is completely broken -- it generates archives that have differing contents)
<rx14>
Git remotes are simple directories
ssvb has joined #crystal-lang
<rx14>
You can easily vendor in all the deps by including the libs dir in your tarball
<rkeene>
That just means that I'm doing the verification work manually, and most likely wrong then.
<rx14>
Huh?
<rx14>
You verify once, when you first build that package revision, then to make it reproducable just tar the whole dir
<rkeene>
Verifying that the upstream sources are unmolested from release-to-release
<rkeene>
Right, but my software will have multiple releases
<rkeene>
Over time
<rx14>
How is verifying they are unmolested harder with git?
<rkeene>
And the upstream should not change over that time, unless I verify that I'm getting it from the same entity
<rx14>
So you store the sha of the dependency's directory
<rkeene>
That's not harder with git, that's harder with a package manager that uses git -- since it may try to "help" by updating things, which is the wrong thing to do.
<rx14>
But it doesn't touch anything unless you run it
<rx14>
And shards doesn't do thag
<rkeene>
Yes, that would be the manual verification I was talking about
<rkeene>
But I WOULD run it to fetch additional dependencies as the software developed
<rx14>
Its a simple bash for loop
<rx14>
Yes but its likely all the dependencies updated anyway due to the nature of crystal's development
<rx14>
Early and stuff gets updated quick
<rx14>
Even when its not generating a sha of a directory is a one-ljner in bash
<rkeene>
And now, of course, I have all those changes in my version control system since they are part of my release
<rkeene>
I'd rather have external applications be separate packages and use the distribution package manager to assemble the parts... It's less of a problem with Crystal since it lacks "gems" (from Ruby, which are terrible) and Python's terribly terrible distutils -- but several applications built into a distribution may need to be tied to the same version of several packages to interoprate, if those dependencies are only managed upstream then you are managing only the l
crack08 has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
<jhass>
Crystal doesn't build shared objects
<jhass>
it's an all source compiler
<rkeene>
It doesn't matter for this, the shared code doesn't have to be in the form of a shared object
<jhass>
it does matter for compatibility considerations
<rkeene>
For example, if you have multiple applications talking to an SQLite3 DB, they need to be using compatible versions of SQLite3 (which is usually included inline and not linked as a shared object)
<rkeene>
Since newer versions of SQLite3 can create databases that older versions cannot read
<rx14>
but you can deal with that on a case-by-case basis, most crystal packages won't need to be version-locked like that
<rkeene>
The need for same version remains the same because there are external factors, you need to interoperate via external resources such as the filesystem
<rkeene>
But the need isn't on a per-package basis, it's on an integration basis -- the need doesn't exist in either package, just in the system that happens to incoroprate them both and expect them to operate on the same datafiles
<rx14>
but it does, because most crystal shards don't operate in that way
<crystal-gh>
[crystal] asterite closed pull request #2876: Fix the order of output in "no overload matches" error (master...no-overload-error) https://git.io/voaYG
<FromGitter>
<taylorfinnell> ahh i just am bad at searching watch and guard are good queries
<FromGitter>
<taylorfinnell> thanks guys!
soveran has quit [Ping timeout: 244 seconds]
<FromGitter>
<taylorfinnell> i now fear i just searched poorly, but are there any ui\graphics libs? not for games but more of for a small gui, more specifically an icon in the system tray
<FromGitter>
<taylorfinnell> that looks like it may be perfect
<FromGitter>
<taylorfinnell> the github says "(well, forget about Windows for now, this is Crystal)" is Windows cross compilation not really supported yet?
<BlaXpirit>
crystal can't run on windows, no
<FromGitter>
<taylorfinnell> gotcha, thanks again guys
<BlaXpirit>
sigh, this is such a minimal binding
<BlaXpirit>
@taylorfinnell i really doubt that this supports system tray. gtk may be good though