<jhass>
then all the String and Enumerable fun stuff
<holst>
oh, thats fantastic
<jhass>
depending on your portability requirements you also may want to scout through /proc for the info you're after, the data in there is usually easier to parse and it's faster because you just need to read a file rather than spawning a subprocess
<FromGitter>
<j8r> one could implement a new format, and transform any object to another one
<FromGitter>
<naqvis> agree
<FromGitter>
<christopherzimmerman> Yea, the closest I could get was NamedTuple mapping, and I wasn't really a fan of that to keep track of types. I guess checking once per op isn't too big a performance hit.
<FromGitter>
<christopherzimmerman> What is backing your Datacols right now? Just an Array, Pointer, Slice?
<FromGitter>
<j8r> I was going to ask a similar question, it is related to CSV/TSV I guess?
<FromGitter>
<naqvis> Array :P
<FromGitter>
<naqvis> One dimensional array of specific type
<FromGitter>
<naqvis> e.g `Int32Col` is storing its data inside `Array(Int32?)`
<FromGitter>
<j8r> that's `@values : Array`
<FromGitter>
<naqvis> yup
<FromGitter>
<christopherzimmerman> If you're interested, I would be interested in working on a PR to make the backend a bit more generic. I'm working on Apache Arrow, Vulkun / Cuda backing for my library right now, it would be cool to support Dataframe operations on a GPU.
<FromGitter>
<j8r> there is a bit of duplication
<FromGitter>
<naqvis> sure thing and more than welcome
<FromGitter>
<naqvis> true, but again as said above, I'm not a big fan of macros
<FromGitter>
<naqvis> and avoid them until there is no option left
* FromGitter
* Blacksmoke16 wants his custom macro methods
<FromGitter>
<naqvis> for sure, would appreciate and welcome PR
<FromGitter>
<christopherzimmerman> They can muddy up code if there are a ton of conditional paths, but for just literal repetition it makes things a lot more clear I think.
<FromGitter>
<naqvis> :P @Blacksmoke16 the Macro man
<FromGitter>
<naqvis> I didn't try to perform any benchmarking on stdlib collection interface and also didn't want to optimize things until there is a solid need for this
<FromGitter>
<naqvis> but i'm sure `num.cr` is definitely going to outperform
<FromGitter>
<j8r> macro man vs dup man... not sure what's better haha
<FromGitter>
<naqvis> on my macbook, whole test suite completes in 1.5s or so
<FromGitter>
<naqvis> so I think, it should be reasonable enough for smaller datasets of few MBs
HumanG33k has joined #crystal-lang
<FromGitter>
<naqvis> and the heaviest dataset i've tried is around 10M
<FromGitter>
<christopherzimmerman> For operations on a flat array, writing the operation on a standard library container will probably be faster. The power of a `Tensor` is that it supports data in n-dimensions, so there is a lot of overhead to iterate through an arbitrarily strided and shaped array.
<FromGitter>
<naqvis> agree
<FromGitter>
<christopherzimmerman> I would love to see an Apache arrow backend. It's really nice for tabular data. Plus it would allow for data IO from pretty much every other popular dataframe library.
<FromGitter>
<naqvis> yeah true and also looking forward to that
<FromGitter>
<naqvis> as that will open crystal door to huge ecosystems of all other languages
HumanG33k has quit [Remote host closed the connection]
HumanG33k has joined #crystal-lang
<FromGitter>
<wyhaines> So..... ⏎ ⏎ You have an object, a Hash or an Array or something else, that has a type like `Hash(String, Int32)` or `Array(UInt64)`, as an example. You want to programmatically access the contained types -- `String, Int32` and `UInt64` so that you can generate a union type from them. How?
<FromGitter>
<wyhaines> True. It's an effort to perform some duck typing magic. ⏎ ⏎ I have a data structure. I want to be able to `merge` anything that has an `#each` method. ⏎ ⏎ `Hash` does what @Blacksmoke16 showed, basically. But then you can only merge other hashes. You can't merge a hash with a hash-like something-else. ... [https://gitter.im/crystal-lang/crystal?at=5f9716be631a250ab293d30f]
<FromGitter>
<HertzDevil> how then do you tell that a `Proc` is neither `Array`-like nor `Hash`-like?
<FromGitter>
<HertzDevil> seems like it's easier to use `Enumerable` directly
<FromGitter>
<HertzDevil> every `Hash(K, V)` is also an `Enumerable({K, V})`
<FromGitter>
<Blacksmoke16> that would be the better option yea, overloads for non key/value types would prob work as well
<FromGitter>
<wyhaines> I can do it for a `merge!` where the incoming data simply has to survive an `.as(X)` type modification into the current structure's type restrictions.
<FromGitter>
<HertzDevil> (this approach then raises the question of whether `Array(Tuple(String, Int32))` should behave like `Hash(String, Int32)`)
<FromGitter>
<wyhaines> Hmmm. Is every array type an `Enumerable(X)`? *thinks*
<FromGitter>
<naqvis> yes it is
<FromGitter>
<HertzDevil> kinda sad crystal doesn't have a module for hash-like types
<FromGitter>
<wyhaines> @HertzDevil Yeah. I'm building a Splay Tree implementation that implements most of the Hash interface. In many cases the code to implement a method is the same as the Hash code.
<hightower2>
O hey @HertzDevil, awesome work on bindgen !
duane has quit [Ping timeout: 258 seconds]
duane has joined #crystal-lang
louis771 has quit [Ping timeout: 260 seconds]
<holst>
Is this Crystal-ideomatic or, is there a better way? spawn name: "webserver" do server.listen end
<holst>
hmm, you got me, maybe I did not even try that
<holst>
If that works it means that server.listen indeed uses Fibers I assume
<FromGitter>
<Blacksmoke16> it odes
<FromGitter>
<Blacksmoke16> does
<holst>
mm, indeed it works!
<FromGitter>
<Blacksmoke16> 👍
<holst>
I like the built in playground so much I would like all my production code to look like this. I attach to the runtime and I get to see a realtime log and metrics correlated with the code. Brilliant
<holst>
Too bad the workflow does not scale to other languages
<FromGitter>
<Blacksmoke16> isnt that what the log module is for? 😉
<FromGitter>
<wyhaines> The Enumerable(X) approach worked well. ⏎ ⏎ I can merge anything array-like or hash-like, now, which is *perfect*.