<FromGitter>
<rurounijones_gitlab> @straight-shoota I tried that and it didn't appear to work but I made a few more changes and my original stuff worked. I think I maybe forgot to compile (I also forgot to actually use puts when outputting a debug message) >.< - Thanks for the confirmation.
<repo>
has anyone here worked with kafka in crystal?
<repo>
i'm experiencing serious performance issues with packetzero/kafka.cr at least
<repo>
also bums me out, that i can't consume in bulk
olbat[m] has joined #crystal-lang
<FromGitter>
<confact> How do I close files in a each? I use File.read and also check if it is not a directory. I get error `Too many open files (Errno)`. ⏎ ⏎ The code is prorotype to try to make it work: ⏎ ⏎ ``` code paste, see link``` ... [https://gitter.im/crystal-lang/crystal?at=5e466c7f25f1d250fed6c06c]
beepdog has joined #crystal-lang
juanfra_ has joined #crystal-lang
erdnaxeli has joined #crystal-lang
<FromGitter>
<confact> Found the issue i think, was using open on Dir, which i thought was the same as read for file. but it wasn't. Using open with a block instead.
alexherbo2 has joined #crystal-lang
<repo>
am i correct to assume crystal-lang/crystal:alpine images don't support -Dpreview_mt yet?
<repo>
i get some linking errors to GC when trying to compile with -Dpreview_mt
<FromGitter>
<straight-shoota> It should work when you build bdwgc from master.
<FromGitter>
<straight-shoota> I didn't go that mile in the Docker image because preview_mt is not production ready anyway and the problem will solve itself as soon as a the APK package is updated to the next bdwgc release (which should probably be released soonish).
<raz>
ah, interesting little gotcha i bumped into today:
<raz>
foo = 0.to_u16 ; bar = 0.to_u8 ; foo += bar + 256
<raz>
this gives an OverflowError, as it's evaluated from right to left
<raz>
makes sense i guess (not a bug), but caught my by surprise :)
<repo>
one thing really bums me off: crystal shards that dont't define `requires` in the files they are needed in.
<repo>
this is mostly all shards
alexherbo2 has quit [Client Quit]
<repo>
it makes it near impossible to use just parts of a shard
<repo>
especially if it's something huge like lucky
<FromGitter>
<Blacksmoke16> doesnt really matter since unused stuff isnt included in the binary
<repo>
mmh
<repo>
i guess you're right about that
<repo>
another side effect though is using something like syntastic or neomake or other code checking tools in an editor like vim
<repo>
the single file you have opened won't compile
<repo>
because of missing requires
<repo>
so you can't check for compile time errors
<repo>
it's much cleaner imho with explicit requires anyway. it's immediately clear where you should look for constants used in the file you have open
<FromGitter>
<Blacksmoke16> ES6 import system would be 💯
<repo>
true
<repo>
and throw in pattern matching and destructuring while you're at it
<FromGitter>
<asterite> @rurounijones_gitlab `puts Time.measure do` means `puts(Time.measure) do`. That's essentially the difference between `do/end` and `{ }`: `do` binds to the left-most call, `{` binds to the right-most call
<repo>
asterite: do you have an opinion regarding explicit requires?
<repo>
Blacksmoke16:
<repo>
sorry
<repo>
Blacksmoke16: what would happen if a shard requires some lib (c-lib bindings) that adds a @[Link ..] annotation?
<repo>
if i never reference code, that uses this lib the c lib won't be linked against?
<FromGitter>
<Blacksmoke16> probably? im not super familiar with c binding stuff
<FromGitter>
<straight-shoota> repo, it's super easy to simply try: `lib LibDoesntExcist; end` compiles perfectly fine. You can add annotations as you like.
alexherbo2 has joined #crystal-lang
<repo>
straight-shoota: that wasn't my question
<repo>
it was related to the explicit requiring
<repo>
so if a shard of which i want to use a tiny portion has somewhere in its codebase a lib which has a @[Link] annotation, that @[Link] annotation will most likely make it to the final build command
<repo>
although
<repo>
that can't be true
<repo>
no wait. it can
<repo>
this is why we don't blindly require the whole stdlib. There are parts of it that depend on runtime libs like libyaml or libxml2 and so on which you wouldn't need if you don't use the code that needs them.
<repo>
i get this error when trying to use YAML::Serializable: undefined method 'new_from_yaml_node'
<FromGitter>
<Blacksmoke16> did you require yaml?
<FromGitter>
<Blacksmoke16> also what type are you trying to use it on
<repo>
yeah i required yaml
<repo>
hm on a model
<repo>
*module
<FromGitter>
<Blacksmoke16> makes sense
<repo>
i'll add an included hook
<FromGitter>
<christopherzimmerman> I've looked through a bunch of shards, but before I work on it, is there a crystal shard that does anything like https://github.com/llgcode/draw2d
sagax has quit [Ping timeout: 268 seconds]
<FromGitter>
<watzon> @christopherzimmerman not that I've seen, but it would probably be possible to build something like that on top of https://github.com/stumpycr/stumpy_png and the like
<FromGitter>
<watzon> On another note, it would be really nice if Crystal had a real `Any` type. Not `JSON::Any`, but a type that can really represent anything. We have pretty good pattern matching in Crystal with `case` statements, so it should be possible.
<FromGitter>
<watzon> Scala has something like that
<FromGitter>
<kinxer> @watzon How does that differ from Object?
<FromGitter>
<kinxer> @christopherzimmerman I've been thinking about working on something like that as well. I agree with @watzon that building on top of the stumpycr repos is the way to go.
<FromGitter>
<watzon> You can't use Object/Class/Int/Float and lots of other types as param types in a lot of cases
<FromGitter>
<watzon> It won't compile
<FromGitter>
<kinxer> Ah... I see. What's the use-case for Any?
<FromGitter>
<kinxer> That is, what motivates having an `Any` type as opposed to using `Object` and generics?
<FromGitter>
<kinxer> And improving where abstract classes can be used for typing.
<FromGitter>
<watzon> Object would be fine if it worked. My main issue right now is that sometimes you don't know what the type of an object is going to be at compile time, or sometimes it's a pain in the ass to write out all of the possible types. For instance, when you have a Hash that's assigned to a class variable. Currently, you have to know exactly the type of that Hash so that you can type it correctly ⏎ ⏎ ```v =
<FromGitter>
<watzon> So it would require the RFC for exhaustive cases to go through
<FromGitter>
<kinxer> Presumably it will be added at some point soon, though.
<FromGitter>
<watzon> But once that's available, an Any type should be possible
<FromGitter>
<watzon> At least as far as I can tell
alexherbo24 has joined #crystal-lang
<FromGitter>
<watzon> Crap, tried to edit the post and deleted it
<FromGitter>
<watzon> Fucking gitter
<FromGitter>
<kinxer> I'm worried about how you'd allocate memory with that. Would that have to be a compile time check for all of the types it *could* be, and then just make a union of that?
<FromGitter>
<watzon> Yeah I think with something like that it would end up adding a little more runtime overhead
alexherbo2 has quit [Ping timeout: 240 seconds]
alexherbo24 is now known as alexherbo2
<FromGitter>
<watzon> But it would be amazingly convenient. I've had to resort to using JSON::Any or writing my own Any type several times
<FromGitter>
<kinxer> I don't think runtime checking of `Any` will work without some compiler additions.
<FromGitter>
<kinxer> But I'm far from an expert on the language internals.
<FromGitter>
<watzon> As it is the whole thing would have to be a compiler addition. As it is now I don't think the compiler is equipped to handle something like this, but at the very least it should be possible.
<FromGitter>
<kinxer> Ah... Gotcha.
<FromGitter>
<kinxer> I personally would prefer to focus on allowing abstract supertypes for type restrictions, thus allowing the use of Object.
<FromGitter>
<kinxer> Probably more work for that, though.
<FromGitter>
<watzon> That would be nice. It would be nice to be able to use Object, Class, Int, Float, etc in unions
<FromGitter>
<watzon> Which you can't right now
<FromGitter>
<kinxer> Yeah. `Number` and its abstract subtypes would be very useful to use for type restrictions.
<FromGitter>
<watzon> Very much so. I've needed that a number of times, but I've had to instead specifically type out each one of the subtypes for a union
<FromGitter>
<straight-shoota> > *<repo>* this is why we don't blindly require the whole stdlib. There are parts of it that depend on runtime libs like libyaml or libxml2 and so on which you wouldn't need if you don't use the code that needs them. ⏎ ⏎ That's only an issue if the code is actually called. You can `require "yaml"` without having `libyaml` available as long as no libyaml function is ever in a reachable code path.
<FromGitter>
... That's only problematic when some code is executed upon requiring a file. Regex, or the bindings to libprcre, are an example for this: It initializes libpcre upon being required, so the library is linked even if no regular expression is ever used in the code.
<FromGitter>
<christopherzimmerman> @kinxer @watzon I will definitely use that for the png rendering, I like how go's is quite general. Unfortunately I would also have to implement pretty much all of Go's "image" package, might do that separately.
<FromGitter>
<watzon> Might not be the worst idea
<FromGitter>
<watzon> I was actually going to do so myself a few days ago
<FromGitter>
<watzon> I've been looking into nice go libraries that could/should be ported to Crystal
<FromGitter>
<christopherzimmerman> My entire motivation for the draw library is that I want a plotting lib in pure crystal, and Go has some great ones to reference, but they *all* require draw2d
<FromGitter>
<watzon> A good plotting library would go a long way
<FromGitter>
<watzon> I want something like jupyter notebook too
<FromGitter>
<christopherzimmerman> I was thinking of writing a kernel for it, there are a lot of pretty easy to follow examples
<FromGitter>
<christopherzimmerman> But compilation time would be an issue
<FromGitter>
<watzon> Yeah, but we're use to that with crystal play and eveything. It would just be a convenient format. Especially if you built in the ability to view the generated LLVM IR and assembly as well.
sagax has joined #crystal-lang
<FromGitter>
<christopherzimmerman> It would also bring over people from the scientific community. I would absolutely love more people helping with numerical libraries.
<FromGitter>
<watzon> For sure
<FromGitter>
<watzon> Anything we can do to get people away from Python
<FromGitter>
<christopherzimmerman> Not sure if we'll ever get the new programmers away from Python too easy to use, but I'm hoping the group that uses C/C++ for the computing and Python for the visualization can be won over.
<FromGitter>
<christopherzimmerman> I've been really trying to take advantage of Crystal stuff to improve over numpy, rather than just porting stuff over like I was doing before, so now:
<FromGitter>
<christopherzimmerman> With numpy, that creates 5 intermediate arrays, with crystal, just one
<FromGitter>
<watzon> Very nice!
<FromGitter>
<watzon> We need this shit in Crystal for sure
<FromGitter>
<watzon> How are things going on the ndarray front?
<FromGitter>
<christopherzimmerman> I have pretty much everything done I want to get done in an initial version, so broadcasting, memory views, all the slicing I need.
<FromGitter>
<christopherzimmerman> I want to abstract storage from just a pointer so I can add GPU storage, but it's in a real good place right now.
<FromGitter>
<watzon> That's awesome. That's another place crystal needs some help, GPU computation
<FromGitter>
<watzon> We need a cuda library
<FromGitter>
<christopherzimmerman> Yea, I started with OpenCL since I don't have a card compatible with Cuda :P
<FromGitter>
<watzon> Same, for now anyway
<FromGitter>
<christopherzimmerman> My issue with using OpenCL in crystal is that I couldn't figure out how to bind to this pattern: ```
<FromGitter>
<christopherzimmerman> `cl_platform_id` is an abstract type, and I don't know how to mimic that in bindings
<FromGitter>
<watzon> Hmmm yeah me neither, I've never dealt with that
<FromGitter>
<watzon> @asterite or @bew might know
<FromGitter>
<christopherzimmerman> In `v` for example, I just do `struct C.cl_platform_id` and it works, but crystal doesn't allow for empty structs
<FromGitter>
<watzon> Haha I didn't know you were using V
<FromGitter>
<christopherzimmerman> Once I figure that out, I have 99% of the code written to allow users to store ndarrays on gpu with no overhead costs.
<FromGitter>
<straight-shoota> You're talking about `encoding`, right? Because there's no mention of that in the generated XML.
<FromGitter>
<watzon> Maybe I'm thinking about the wrong property, but it's this `<?xml version=\"1.0\"?>`
<FromGitter>
<watzon> That's encoding isn't it?
<FromGitter>
<straight-shoota> that's `version`, not `encoding`
<FromGitter>
<watzon> Ahhhh duh
<FromGitter>
<straight-shoota> And AFAIK it's a mandatory attribute
<FromGitter>
<watzon> It's mandatory according to the spec, but I don't think it should be forced on us. There are plenty of cases where you may not want that.
<FromGitter>
<watzon> You can always just remove the first line
<FromGitter>
<watzon> But meh
<FromGitter>
<straight-shoota> Yeah, I guess there could be use cases for skipping the document setup. For example when only building a document fragment.
<FromGitter>
<watzon> Exactly, line in my case it's just basic HTML formatting that I'm doing. Taking some HTML entities and building out a formatted HTML string.
<FromGitter>
<straight-shoota> @j8r ⏎ ⏎ > I don't know, should `reuse_port` be true by default on a framework using `HTTP::Server`? ⏎ ⏎ I'd recommend against reuse_port as default because it can be confusing when the port is already bound by a different application (or maybe an old build you don't remember is still running). It's better to fail when the port is taken, especially during development. That's where users will
<FromGitter>
<j8r> @straight-shoota thanks!
sorcus has joined #crystal-lang
<sorcus>
It's possible to write array into file and then read it as `Array` without convertation to `JSON`, `YAML` and other?
<FromGitter>
<j8r> you have to define a format anyway
<FromGitter>
<watzon> May as well write it as JSON
<FromGitter>
<j8r> the simpler format depends on the content
<sorcus>
Thanks :-)
<FromGitter>
<j8r> if there are no newlines inside, a new line separated format is fine
<FromGitter>
<j8r> or CSV
<FromGitter>
<j8r> the advantage with JSON is the native support in Crystal
<sorcus>
j8r: Yeah, and this is faster than YAML :-D
<FromGitter>
<christopherzimmerman> They promote some fairly small projects, and I think with Crystal's rise, should have a good chance.
<FromGitter>
<watzon> That would be really cool
<FromGitter>
<watzon> Very good for Crystal
<raz>
it would be great to get rid of python in that space
<raz>
it still mystifies me how it ever got so popular. i guess the only reason is because the alternatves are even worse...
<FromGitter>
<Blacksmoke16> @bcardiff `TypeNode#name` example in the blog post got borked a bit
<FromGitter>
<Blacksmoke16> seems to be missing the macro code
<FromGitter>
<bcardiff> Grrr. thanks
<raz>
i'm slowly warming up with macros. they're the devil, but such a useful devil.
<FromGitter>
<bcardiff> @Blacksmoke16 it will be fixed in a little. thanks
<raz>
i kinda wonder if at some point in the future they could be expanded to become "real" crystal (i.e. kinda like JSP/ERB instead of the current meta-language)
early has quit [Quit: Leaving]
<FromGitter>
<Blacksmoke16> 👍
ur5us has joined #crystal-lang
<FromGitter>
<christopherzimmerman> Welp, updated to 0.33.0 and 95% of my tests broke haha.
<FromGitter>
<christopherzimmerman> I'm assuming this isn't intended:
<FromGitter>
<dscottboggs_gitlab> the problem comes with complex configurations that also need to be human-readable. I don't know much better than YAML unless the data-model just coincidentally happens to work well in INI/TOML or CSV
<raz>
j8r, oh, that first/second bit is actually a nice idea
<sorcus>
j8r: Looks good :-)
<raz>
i like that format. but i'll admit the current yaml hasn't given me major problems either.
<FromGitter>
<j8r> yes
<raz>
the only problem i have is that my vim blows the indentation when i copy/paste the dep from a readme. every. single. time.
<FromGitter>
<dscottboggs_gitlab> yeah, I like your sherd.ini better than shard.yml
<FromGitter>
<rurounijones_gitlab> Is it worth reporting somewhere when we get a "Invalid memory access (signal 11)" error?
<FromGitter>
<j8r> I was frustrated by shards dependency resolution, and slow downloads of dependencies
<raz>
j8r: me too. i even looked at the code with the intent to parallelize it.
<FromGitter>
<j8r> but I am not clever enough to deal with dependencies
<raz>
but then quickly realized it's way over my head
<raz>
ll
<raz>
same
<FromGitter>
<j8r> @dscottboggs_gitlab thanks!
<raz>
maybe someone could bug the author to add it. but imagine he's probably busy working on a space shuttle or something.
<FromGitter>
<j8r> now I m working on a web router, because a bit frustrated by Kemal :)
<raz>
j8r: looked at router.cr? it looks pretty good
<FromGitter>
<j8r> Yep already, but very basic
<raz>
true. but it does the proper radix tree thing, so performance is top notch
<FromGitter>
<j8r> I have made a custom tree instead of radix
<FromGitter>
<j8r> early benchmarks seems to be similar to Kemal
<raz>
i don't know what kemal uses
<FromGitter>
<j8r> this allows me to store handlers, and query parameteres inside the tree
<FromGitter>
<j8r> Nearly all Crystal routers use Radix
<raz>
yea i think the router would rarely be a bottleneck anyway
<raz>
outside of benchmarks
<FromGitter>
<straight-shoota> kemal uses radix
<FromGitter>
<j8r> I need to finish tests, the swagger support and I release it
<raz>
i recently built a simple router using File.glob
<FromGitter>
<dscottboggs_gitlab> yeah Crystal HTTP server shards are a dime a dozen TBH
<raz>
actually i think they all just use the stdlib HTTP::Server under the hood
<FromGitter>
<j8r> The issue is: how to store handlers inside the tree nodes?
<raz>
nothing wrong with that, but credit where its due :)
<FromGitter>
<j8r> the use case is to have a set of routes under authentication
<raz>
have the tree nodes be objects/structs?
<FromGitter>
<straight-shoota> the router shouldn't need to care about authentication
<FromGitter>
<dscottboggs_gitlab> > how to store handlers ⏎ ⏎ as a `Hash(Route, Proc(Context, Result))`?
<FromGitter>
<dscottboggs_gitlab> ^^ in this case optimization woud come from `Route#hash`
<FromGitter>
<j8r> the router is not aware of authentication, but aware of handlers and the action
<FromGitter>
<j8r> a given Node has action, queries, and its associate handlers
<FromGitter>
<dscottboggs_gitlab> Authentication is a middleware. A middleware is a `Proc(Context, Result)` which does some preprocessing on the request or result
<FromGitter>
<straight-shoota> what's the difference between action and handlers?
<FromGitter>
<j8r> in action I mean a route action
<FromGitter>
<j8r> handlers are executed before or after
<raz>
so handlers = per route middleware?
<FromGitter>
<dscottboggs_gitlab> I mean, that's how it is in express, kemal, flask or sanic, gorilla/mux, etc
<FromGitter>
<j8r> maybe it's a better name
<FromGitter>
<j8r> `HTTP::Handler` are compatible with mine handlers - but what's the diff with middlewares?
<raz>
i think middlewares execute for all requests, regardless of routinig
<raz>
(at least that's how it is in most frameworks)
<FromGitter>
<j8r> for now there is a Initializer, NotFound, PreAction, PostAction, and Finalizer
<FromGitter>
<dscottboggs_gitlab> either before or after the route-based handler, depending on the middleware configureation
<FromGitter>
<dscottboggs_gitlab> can one configure more than one Pre/PostAction?
<FromGitter>
<j8r> yes, a la `HTTP::Handler`
<FromGitter>
<dscottboggs_gitlab> gotta go for a bit ✌️
<FromGitter>
<j8r> there is `property next : PostAction(R)? = nil` for instance, with `R` the result type of the route action
<FromGitter>
<straight-shoota> Sounds pretty complicated. I don't like middleware except for very basic protocol stuff. Having middleware attached to individual routes doesn't make any sense in my book.
<FromGitter>
<j8r> authentication behind certain routes?
<FromGitter>
<j8r> and send logs to a specific location for certain routes
<FromGitter>
<j8r> or compression, CORS for others
<FromGitter>
<j8r> lots of use cases
<FromGitter>
<j8r> another one is to serve certain locations with a StaticFile handler
<FromGitter>
<j8r> raz: so a request logger can usually be considered a middleware, I guess?
<raz>
j8r: yep
<FromGitter>
<straight-shoota> These use cases IMO belong all in the controller, not the router
<FromGitter>
<straight-shoota> > *<raz>* t.ly/GZLvB ⏎ ⏎ Are you aware of https://github.com/schovi/baked_file_system which has been around for years? It's seems rucksack is more difficult to use. Are there any advantages?
<raz>
straight-shoota: yes. the advantage is that it doesn't load the files into RAM at startup
<raz>
well, it doesn't load them at all. ram is precious :)
<FromGitter>
<Blacksmoke16> Athena is where its at 😉
<FromGitter>
<straight-shoota> yeah, that might be preferable in some usecases for heavier assets
<FromGitter>
<straight-shoota> But it also means a running program will break when the executable is removed or updated
<FromGitter>
<Blacksmoke16> @j8r could also take an event based approach versus `HTTP::Handler` style
<FromGitter>
<Blacksmoke16> then your code just has to emit it and other code can do whatever they want with it
<FromGitter>
<Blacksmoke16> chose to act, ignore it etc
<FromGitter>
<j8r> I have seen your library, in fact my solution is a bit between the 2
<FromGitter>
<Blacksmoke16> is a good pattern imo, the handler stack is simple but not very flexible
<raz>
straight-shoota: yup that's a valid concern, should perhaps warn about that in the readme. my use-case is webapps that may want to bundle hundred of megs of assets but not waste the RAM on it (can be expensive when hosting in the cloud)
<FromGitter>
<j8r> @straight-shoota but where does you store the information?
<FromGitter>
<j8r> The router nodes only store, do not execute anything
<FromGitter>
<straight-shoota> @j8r what information?
<FromGitter>
<j8r> The route information: http query params, what hanlders should it executes before (like auth)
<FromGitter>
<j8r> And of course the route action proc
<FromGitter>
<straight-shoota> raz, I see it inserts a 16k separator to indicate the data section. That's probably already more than most simple web apps would have baked in :D
<raz>
straight-shoota: but it doesn't load that into RAM either :p
<FromGitter>
<straight-shoota> yeah
<raz>
my app here bundles desktop-app downloads for win and osx (electron). those weigh in at ~200M alone
<FromGitter>
<straight-shoota> @j8r Sry, not following. The router is supposed to match request paths to handlers/controllers. Information is passed to that.
<raz>
but yea, i agree, for most apps it won't be a concern either way. i just wanted a solution where i don't have to worry about asset size at all.
<FromGitter>
<j8r> At the end, it stores the information through the controller then
<FromGitter>
<straight-shoota> yeah
<FromGitter>
<j8r> Ok, thanks, interesting – I'm thinking about the advantages? (Except conceptualy)
<FromGitter>
<j8r> Ho I think I've already done it on the project
<FromGitter>
<j8r> There is a way to group routes actions together – which is the purpose of a controller, right?
<raz>
having a smarter router can be useful tho. see e.g. roda framework in ruby
<FromGitter>
<Blacksmoke16> right a controller should consist of common endpoints
<raz>
there is no law that frameworks have to adhere the traditional middleware/controller split (at least none that i'm aware of :D)
<FromGitter>
<Blacksmoke16> good practice tho
<raz>
depends on the design goals i'd say
<raz>
if you wanna build yet another sinatra/rails, then yea
<FromGitter>
<j8r> It is "good" to attach middlewares/handlers to controllers?
<FromGitter>
<Blacksmoke16> like for auth?
<FromGitter>
<j8r> Yep
<raz>
well, auth is one of the classic cases where the central middleware split breaks down. often you want to apply diifferent auth patterns to different route prefixes.
<FromGitter>
<j8r> Right
<raz>
so most frameworks have some more or less awkward way to specify that outside of the route definitions
<FromGitter>
<j8r> I don't know if it is a thing to have nested controllers