jhass changed the topic of #crystal-lang to: The Crystal programming language | http://crystal-lang.org | Crystal 0.28.0 | Fund Crystal's development: http://is.gd/X7PRtI | GH: https://github.com/crystal-lang/crystal | Docs: http://crystal-lang.org/docs/ | API: http://crystal-lang.org/api/ | Gitter: https://gitter.im/crystal-lang/crystal
<FromGitter> <nsuchy> 300
<FromGitter> <nsuchy> Not 30
<FromGitter> <nsuchy> It’s a data feed and the production feed is several GB
<FromGitter> <nsuchy> I want to optimize well as possible before I switch from the Node.js processor to the Crystal processor
<FromGitter> <nsuchy> It’s my understanding SQLite was also a bottleneck, not sure how faster the Crystal library is compared to Node.js
laaron has quit [Quit: ZNC 1.7.1 - https://znc.in]
laaron has joined #crystal-lang
<FromGitter> <Blacksmoke16> depending on what you're doing, prob a lot
<FromGitter> <nsuchy> a lot of inserts
<FromGitter> <nsuchy> like a lot
<FromGitter> <Blacksmoke16> could prob make the inserts in a single query
<FromGitter> <nsuchy> Parsing huge JSON feeds (several GB of data) and caching into SQLite for internal use
<FromGitter> <Blacksmoke16> gotcha
<FromGitter> <nsuchy> how do I build as release
<FromGitter> <nsuchy> typing --release gets an LLVM error
<FromGitter> <Blacksmoke16> `crystal build --release my_app.cr`
<FromGitter> <Blacksmoke16> oh?
<FromGitter> <nsuchy> using with run though
<FromGitter> <nsuchy> not build
<FromGitter> <Blacksmoke16> easiest to just use build
<FromGitter> <nsuchy> ```code paste, see link``` [https://gitter.im/crystal-lang/crystal?at=5ce9e4039404bf2aedad16b9]
<FromGitter> <nsuchy> using build
<FromGitter> <Blacksmoke16> how did you install crystal
<FromGitter> <nsuchy> asdf
<FromGitter> <nsuchy> it's a version manager tool
<FromGitter> <Blacksmoke16> yea something prob didnt get installed right
<FromGitter> <Blacksmoke16> :shrug:
<FromGitter> <Blacksmoke16> cant say i ever saw that error before
<FromGitter> <nsuchy> Nathaniels-MacBook-Pro% crystal --version ⏎ Crystal 0.28.0 (2019-04-17) ⏎ ⏎ LLVM: 3.9.1 ⏎ Default target: x86_64-apple-macosx [https://gitter.im/crystal-lang/crystal?at=5ce9e46f9404bf2aedad1984]
<FromGitter> <Blacksmoke16> maybe try with llvm 4.0.0
<FromGitter> <nsuchy> I don't have any way to change the version
<FromGitter> <Blacksmoke16> asdf is using an older version of llvm i guess
<FromGitter> <Blacksmoke16> vs brew?
<FromGitter> <Blacksmoke16> yea upgrading llvm seems to be the solution as far as i can tell
<FromGitter> <nsuchy> hmm okay
<FromGitter> <nsuchy> I'll uninstall the asdf version
<FromGitter> <nsuchy> I have little incentive to use older versions anyways
<FromGitter> <nsuchy> its not like ruby where certain packages require old ruby versions
<FromGitter> <nsuchy> yeah so homebrew has llvm 6
<FromGitter> <Blacksmoke16> 👍
<FromGitter> <nsuchy> weird though asdf is using such an old version
<FromGitter> <nsuchy> probably to support older crystal versions
<FromGitter> <Blacksmoke16> :shrug:
<FromGitter> <tenebrousedge> @Blacksmoke16 are you on OSX?
<FromGitter> <Blacksmoke16> my work laptop is yes
<FromGitter> <Blacksmoke16> but desktop at home is debian
<FromGitter> <nsuchy> ```code paste, see link``` [https://gitter.im/crystal-lang/crystal?at=5ce9e6c78f019114aeb88216]
<FromGitter> <nsuchy> Wow
<FromGitter> <Blacksmoke16> 👍
<FromGitter> <nsuchy> 6 seconds
<FromGitter> <Blacksmoke16> looks about right
<FromGitter> <nsuchy> wow
<FromGitter> <Blacksmoke16> remind me of that guy that moved his ruby code over and went from 9min 13 sec to 1min 54 sec
<FromGitter> <Blacksmoke16> then did release mode and went down to 18s
<FromGitter> <tenebrousedge> yeah that's kinda what I love about this language. I mean, Ruby is a dream, but Crystal is bottled lightning
<FromGitter> <nsuchy> yes like
<FromGitter> <nsuchy> now I just need to write the database insert logic
<FromGitter> <nsuchy> and I can have a full picture to benchmark
<FromGitter> <nsuchy> anything else I can do to speed up the parsing?
<FromGitter> <nsuchy> 6 seconds is great
<FromGitter> <nsuchy> would anything shave off time?
<FromGitter> <Blacksmoke16> reminds me a project i did, ruby to crystal
<FromGitter> <nsuchy> It uses JSON.mapping() and a lot of subclasses
<FromGitter> <Blacksmoke16> structs
<FromGitter> <Blacksmoke16> went from 22 sec to 20ms
<FromGitter> <Blacksmoke16> but im going to give most of that credit to less shitty code
<FromGitter> <Blacksmoke16> :s
<FromGitter> <nsuchy> Wait really?
<FromGitter> <nsuchy> I'm willing to listen
<FromGitter> <Blacksmoke16> of what? my project or structs?
<FromGitter> <nsuchy> Of structs
<FromGitter> <Blacksmoke16> yea just replace class with struct
<FromGitter> <Blacksmoke16> and try again
<FromGitter> <Blacksmoke16> imma take a guess and say 3 sec
<FromGitter> <Blacksmoke16> no i mean like
<FromGitter> <nsuchy> what do structs look like?
<FromGitter> <Blacksmoke16> `class MyClass`
<FromGitter> <Blacksmoke16> `struct MyClass`
<FromGitter> <Blacksmoke16> ^
<FromGitter> <nsuchy> is there any code changes needed
<FromGitter> <Blacksmoke16> no
<FromGitter> <nsuchy> other than change class to struct
<FromGitter> <Blacksmoke16> prob not if all you're doing is reading in data from josn
<FromGitter> <Blacksmoke16> imma drop down to 500ms guess
<FromGitter> <Blacksmoke16> should be a noticeable improvement
<FromGitter> <nsuchy> ```code paste, see link``` [https://gitter.im/crystal-lang/crystal?at=5ce9e97becdf942b4c4759af]
<FromGitter> <nsuchy> :(
<FromGitter> <nsuchy> The structs are inside a module if that makes a difference
<FromGitter> <Blacksmoke16> all the time must be in the parsing of the JSON itself
<FromGitter> <Blacksmoke16> it shouldnt
<FromGitter> <tenebrousedge> the JSON is being written to SQLite after this?
<FromGitter> <Blacksmoke16> i think he's just doing a from_json on the big file
<FromGitter> <nsuchy> not yet
<FromGitter> <nsuchy> Just using Struct name .from_json
<FromGitter> <Blacksmoke16> yea id imagine most of the time there is spent on parsing/reading json?
<FromGitter> <Blacksmoke16> vs newing up structs/classes
<FromGitter> <nsuchy> :|
laaron has quit [Remote host closed the connection]
<FromGitter> <tenebrousedge> but that could be parallelized probably, and if there's no intermediate transformation that needs to happen to the strings, you could maybe use some low-level construct instead of `String`
laaron has joined #crystal-lang
<FromGitter> <Blacksmoke16> Would fibers help read the file faster?
<FromGitter> <nsuchy> How would I go about that?
<FromGitter> <Blacksmoke16> IDK
<FromGitter> <nsuchy> For example, `@vectorString="CVSS:3.0/AV:N/AC:L/PR:L/UI:R/S:C/C:L/I:L/A:N",` the value needs to get copied into SQLite that's it
griddle has joined #crystal-lang
<FromGitter> <nsuchy> Big inserts would also be nice
<FromGitter> <nsuchy> If I could pop off say 20 items from an array and insert them into the correct table and then go to the next 20 items
<FromGitter> <Blacksmoke16> Probably pretty easy
<FromGitter> <tenebrousedge> parallelizing the read on the other hand would probably be more trouble than it's worth
<FromGitter> <nsuchy> @tenebrousedge the feeds I am parsing are in multitudes of GB
laaron has quit [Quit: ZNC 1.7.1 - https://znc.in]
<FromGitter> <nsuchy> I'm willing to try anything
<FromGitter> <tenebrousedge> I would write an iterator that would take n lines at a time, and then pass those to different fibers for processing/insertion
<FromGitter> <nsuchy> Can I do that with file.open?
<FromGitter> <nsuchy> For example `myobject = Entity.from_json(File.open("./src/example-json-files/2018.json"))`
<FromGitter> <Blacksmoke16> Try doing file.read instead
laaron has joined #crystal-lang
<FromGitter> <Blacksmoke16> Probably won't make much of a difference tho
<FromGitter> <nsuchy> Not true
<FromGitter> <nsuchy> huge difference
<FromGitter> <nsuchy> ```code paste, see link``` [https://gitter.im/crystal-lang/crystal?at=5ce9ed2a9404bf2aedad4661]
<FromGitter> <tenebrousedge> 👀
<FromGitter> <Blacksmoke16> Nice
<FromGitter> <nsuchy> So use File.read instead of File.open
<FromGitter> <Blacksmoke16> well depends on what you want
<FromGitter> <nsuchy> just need READ access
<FromGitter> <nsuchy> Reproducible ⏎ ⏎ ```code paste, see link``` [https://gitter.im/crystal-lang/crystal?at=5ce9ed8b879f4478c7e18ce5]
<FromGitter> <Blacksmoke16> yea file.read just opens it in read mode, copys the IO to a string and returns the string
<FromGitter> <nsuchy> I have to do this times multiple GB
<FromGitter> <nsuchy> willing to go a lot of effort to save on processing time
laaron has quit [Remote host closed the connection]
<FromGitter> <Blacksmoke16> hmm
<FromGitter> <Blacksmoke16> well at this point im out of ideas
laaron has joined #crystal-lang
<FromGitter> <nsuchy> `File.read()` is amazing in itsel
<FromGitter> <Blacksmoke16> what happens if you did like `File.read("path", encoding: "UTF-8")`
<FromGitter> <nsuchy> ```code paste, see link``` [https://gitter.im/crystal-lang/crystal?at=5ce9eecdb313d723142c20bb]
<FromGitter> <Blacksmoke16> i figured
<FromGitter> <nsuchy> yeah :\
<FromGitter> <Blacksmoke16> prob best you going to get less you have another idea
<FromGitter> <nsuchy> yeah
<FromGitter> <nsuchy> :|
laaron has quit [Remote host closed the connection]
laaron has joined #crystal-lang
<FromGitter> <Blacksmoke16> refactored how i handle env specific config
<FromGitter> <Blacksmoke16> ```code paste, see link``` [https://gitter.im/crystal-lang/crystal?at=5ce9f23fecdf942b4c478bc2]
<FromGitter> <Blacksmoke16> is all in one file now, can define custom settings for each env, including custom envs. is based off of env var
<FromGitter> <Blacksmoke16> also determines the default behavior for the logger
<FromGitter> <Blacksmoke16> STDOUT and file if on dev, none on testing, and file if its >= warning on prod
<FromGitter> <watzon> Ok this is extremely basic so I don't know why it doesn't work. Why is it that including a module does't add its methods as class methods, and is there a way to do so? https://carc.in/#/r/6zac
<FromGitter> <watzon> That's pretty cool @Blacksmoke16
<FromGitter> <Blacksmoke16> use `extend` instead of `include`
<FromGitter> <watzon> https://carc.in/#/r/6zag
<FromGitter> <watzon> Then it breaks the class variables
<FromGitter> <Blacksmoke16> huh
<FromGitter> <Blacksmoke16> dunno
<FromGitter> <Blacksmoke16> and thanks :)
<FromGitter> <Blacksmoke16> it's getting there
<FromGitter> <watzon> Sure is :)
<FromGitter> <watzon> I don't know if this is a bug or expected behavior. I'm going to assume the latter, but I find it very unintuitive coming from how Ruby does it.
<FromGitter> <watzon> The only thing that I've found that works is to do all instance variable declarations in a `macro included` and then include the module, but that ruins the whole purpose of the module since the instance vars get declared in the class that's doing the including instead of the module.
<FromGitter> <Blacksmoke16> dunno
<FromGitter> <Blacksmoke16> anyway im off to bed o/
<FromGitter> <watzon> Well good night!
laaron has quit [Remote host closed the connection]
laaron has joined #crystal-lang
laaron has quit [Quit: ZNC 1.7.1 - https://znc.in]
laaron has joined #crystal-lang
laaron has quit [Quit: ZNC 1.7.1 - https://znc.in]
laaron has joined #crystal-lang
laaron has quit [Quit: ZNC 1.7.1 - https://znc.in]
laaron has joined #crystal-lang
<FromGitter> <yorci> can anyone explain me why when pass variable with parantheses to macro variable which is passed, it yields passed variable name instead of variable content? ⏎ ⏎ https://carc.in/#/r/6zdj
<FromGitter> <yorci> or i must ask how can i invoke method which is passed as variable to macro ?
<FromGitter> <watzon> @yorci because the value of your variable is not known at compile time
<FromGitter> <watzon> macros are expanded at compile time
<FromGitter> <yorci> i see, i must pass the method name directly right?
<FromGitter> <tenebrousedge> @yorci https://carc.in/#/r/6ze3
<FromGitter> <watzon> Yes
<FromGitter> <watzon> Either that or I think you can use a constant
<FromGitter> <tenebrousedge> not like that. I tried it o_o
<FromGitter> <tenebrousedge> there is also `method_missing`
<FromGitter> <yorci> these are not fullfil my needs, i am trying to make something like event register and fire macro, registering is okay, its registering in my program files but event fire is problematic, i get data from api, its a hash variable which is contains method name and arguments, so i need to invoke method by that method name in hash, i mean i think that way is more flexible and modular.
<FromGitter> <yorci> but if icant do that dynamicly, i need to use a switch case i think
<FromGitter> <tenebrousedge> well, why not just use a hash where the values are `Proc`s?
<FromGitter> <yorci> what you mean
<FromGitter> <yorci> hash is actually a json object
<FromGitter> <yorci> {"command": "method1", "args": ["t1","t2","t3"]} ⏎ ⏎ such as
<FromGitter> <tenebrousedge> https://carc.in/#/r/6ze7
<FromGitter> <watzon> As @tenebrousedge said, use procs
<FromGitter> <yorci> hmm you mean nevermind about macros, just create a hash of procs and register method as proc, fire that method by proc.call, am i understand right?
<FromGitter> <tenebrousedge> I mean, if you have a good reason not to, it seems like an easy solution
<FromGitter> <tenebrousedge> you may want to define a `default_block` on your hash for any missing keys
<FromGitter> <yorci> it looks pretty mess but modular 😃 thank you, i think i will make this way.
<FromGitter> <watzon> You can use a method to register an event like this
<FromGitter> <watzon> Obviously you probably want to do a little bit more there, but that's the basic idea
<FromGitter> <yorci> yes actually i was tried procs first but procs needs to static argument size & type and return type, so i change it to define method and invoke it by macro. ⏎ ⏎ its hard to seperate logics compile time / run time for me as an long term interpreter developer 😃 , i've never write code flexible as much as i want 😃
<FromGitter> <watzon> Crystal needs more hash functions, ie native sha2/3
<FromGitter> <watzon> Or at the very least better documentation around them, because I can't find SHA-256 and SHA-512
laaron has quit [Quit: ZNC 1.7.1 - https://znc.in]
laaron has joined #crystal-lang
<FromGitter> <watzon> Figured it out. OpenSSL::Digest needs some docs
laaron has quit [Quit: ZNC 1.7.1 - https://znc.in]
laaron has joined #crystal-lang
<FromGitter> <r00ster91> I'm trying to run a Crystal executable on a Raspberry Pi. I tried to cross compile with the target `aarch64-linux-gnu` and with `aarch64-linux-musl` and then run the `cc` command on the Raspberry Pi but both .o files fail with ` file not recognized: File format not recognized`
ShalokShalom has joined #crystal-lang
ShalokShalom62 has joined #crystal-lang
ShalokShalom62 has quit [Remote host closed the connection]
<FromGitter> <r00ster91> oh the correct target seems to be `arm-unknown-linux-gnueabihf `
ashirase has quit [Ping timeout: 258 seconds]
ashirase has joined #crystal-lang
DTZUZU has quit [Ping timeout: 248 seconds]
laaron has quit [Quit: ZNC 1.7.1 - https://znc.in]
laaron has joined #crystal-lang
_whitelogger has joined #crystal-lang
_whitelogger has joined #crystal-lang
ShalokShalom has quit [Remote host closed the connection]
DTZUZU has joined #crystal-lang
<FromGitter> <alex-lairan> Hello ! ⏎ ⏎ I have a JSON like `"[{\"text\":\"Should I go ?\",\"responses\":[\"yes\",\"no\"]}]"` ⏎ ⏎ To read the JSON, I created a ... [https://gitter.im/crystal-lang/crystal?at=5cea9811879f4478c7e5a461]
<FromGitter> <Blacksmoke16> Array(QuestionLoader).from_json(params[:questions])`
<FromGitter> <Blacksmoke16> its an array of that type, not just a single obj of that type
<FromGitter> <alex-lairan> Yes, thanks ! ⏎ ⏎ I've forget the Array :D
<FromGitter> <Blacksmoke16> np
_whitelogger has joined #crystal-lang
<FromGitter> <Blacksmoke16> is there not a way to generate `<<: *development` using yaml nodes builder?
<FromGitter> <Blacksmoke16> seems to be missing
laaron has quit [Remote host closed the connection]
laaron has joined #crystal-lang
<FromGitter> <Blacksmoke16> `All "emitting" methods support specifying a "reference" object that will be associated to the emitted object, so that when that reference object is emitted again an anchor and an alias will be created. This generates both more compact documents and allows handling recursive data structures.`
<FromGitter> <Blacksmoke16> sounds like using reference does that
laaron has quit [Remote host closed the connection]
laaron has joined #crystal-lang
<FromGitter> <dscottboggs_gitlab> https://carc.in/#/r/6zg5 ⏎ ⏎ :D phew I'm really glad that works
<FromGitter> <tenebrousedge> why would it not?
<FromGitter> <dscottboggs_gitlab> > I've forget the Array :D ⏎ ⏎ I've definitely done that :D
<FromGitter> <dscottboggs_gitlab> > why would it not? ⏎ ⏎ not every language likes to do overloads, and since macros are more dynamic I wasn't sure if they would work right or just overwrite the previous one
<FromGitter> <tenebrousedge> ahhhhh
<FromGitter> <tenebrousedge> thanks for explaining
<FromGitter> <dscottboggs_gitlab> I could've gotten an error saying that `test "arg1"` was not a valid call because it was expecting `test` to take 2 args. that would've been real annoying :p
<FromGitter> <dscottboggs_gitlab> np :)
<FromGitter> <Blacksmoke16> ayy i figured it out
<FromGitter> <Blacksmoke16> ```code paste, see link``` [https://gitter.im/crystal-lang/crystal?at=5ceaa9a5b313d7231430d5cc]
<FromGitter> <Blacksmoke16> kinda annoying but ill take it