<FromGitter>
<tenebrousedge> the JSON is being written to SQLite after this?
<FromGitter>
<Blacksmoke16> i think he's just doing a from_json on the big file
<FromGitter>
<nsuchy> not yet
<FromGitter>
<nsuchy> Just using Struct name .from_json
<FromGitter>
<Blacksmoke16> yea id imagine most of the time there is spent on parsing/reading json?
<FromGitter>
<Blacksmoke16> vs newing up structs/classes
<FromGitter>
<nsuchy> :|
laaron has quit [Remote host closed the connection]
<FromGitter>
<tenebrousedge> but that could be parallelized probably, and if there's no intermediate transformation that needs to happen to the strings, you could maybe use some low-level construct instead of `String`
laaron has joined #crystal-lang
<FromGitter>
<Blacksmoke16> Would fibers help read the file faster?
<FromGitter>
<nsuchy> How would I go about that?
<FromGitter>
<Blacksmoke16> IDK
<FromGitter>
<nsuchy> For example, `@vectorString="CVSS:3.0/AV:N/AC:L/PR:L/UI:R/S:C/C:L/I:L/A:N",` the value needs to get copied into SQLite that's it
griddle has joined #crystal-lang
<FromGitter>
<nsuchy> Big inserts would also be nice
<FromGitter>
<nsuchy> If I could pop off say 20 items from an array and insert them into the correct table and then go to the next 20 items
<FromGitter>
<Blacksmoke16> Probably pretty easy
<FromGitter>
<tenebrousedge> parallelizing the read on the other hand would probably be more trouble than it's worth
<FromGitter>
<nsuchy> @tenebrousedge the feeds I am parsing are in multitudes of GB
<FromGitter>
<tenebrousedge> I would write an iterator that would take n lines at a time, and then pass those to different fibers for processing/insertion
<FromGitter>
<nsuchy> Can I do that with file.open?
<FromGitter>
<nsuchy> For example `myobject = Entity.from_json(File.open("./src/example-json-files/2018.json"))`
<FromGitter>
<Blacksmoke16> is all in one file now, can define custom settings for each env, including custom envs. is based off of env var
<FromGitter>
<Blacksmoke16> also determines the default behavior for the logger
<FromGitter>
<Blacksmoke16> STDOUT and file if on dev, none on testing, and file if its >= warning on prod
<FromGitter>
<watzon> Ok this is extremely basic so I don't know why it doesn't work. Why is it that including a module does't add its methods as class methods, and is there a way to do so? https://carc.in/#/r/6zac
<FromGitter>
<watzon> Then it breaks the class variables
<FromGitter>
<Blacksmoke16> huh
<FromGitter>
<Blacksmoke16> dunno
<FromGitter>
<Blacksmoke16> and thanks :)
<FromGitter>
<Blacksmoke16> it's getting there
<FromGitter>
<watzon> Sure is :)
<FromGitter>
<watzon> I don't know if this is a bug or expected behavior. I'm going to assume the latter, but I find it very unintuitive coming from how Ruby does it.
<FromGitter>
<watzon> The only thing that I've found that works is to do all instance variable declarations in a `macro included` and then include the module, but that ruins the whole purpose of the module since the instance vars get declared in the class that's doing the including instead of the module.
<FromGitter>
<Blacksmoke16> dunno
<FromGitter>
<Blacksmoke16> anyway im off to bed o/
<FromGitter>
<watzon> Well good night!
laaron has quit [Remote host closed the connection]
<FromGitter>
<yorci> can anyone explain me why when pass variable with parantheses to macro variable which is passed, it yields passed variable name instead of variable content? ⏎ ⏎ https://carc.in/#/r/6zdj
<FromGitter>
<yorci> or i must ask how can i invoke method which is passed as variable to macro ?
<FromGitter>
<watzon> @yorci because the value of your variable is not known at compile time
<FromGitter>
<watzon> macros are expanded at compile time
<FromGitter>
<yorci> i see, i must pass the method name directly right?
<FromGitter>
<watzon> Either that or I think you can use a constant
<FromGitter>
<tenebrousedge> not like that. I tried it o_o
<FromGitter>
<tenebrousedge> there is also `method_missing`
<FromGitter>
<yorci> these are not fullfil my needs, i am trying to make something like event register and fire macro, registering is okay, its registering in my program files but event fire is problematic, i get data from api, its a hash variable which is contains method name and arguments, so i need to invoke method by that method name in hash, i mean i think that way is more flexible and modular.
<FromGitter>
<yorci> but if icant do that dynamicly, i need to use a switch case i think
<FromGitter>
<tenebrousedge> well, why not just use a hash where the values are `Proc`s?
<FromGitter>
<yorci> what you mean
<FromGitter>
<yorci> hash is actually a json object
<FromGitter>
<yorci> {"command": "method1", "args": ["t1","t2","t3"]} ⏎ ⏎ such as
<FromGitter>
<watzon> As @tenebrousedge said, use procs
<FromGitter>
<yorci> hmm you mean nevermind about macros, just create a hash of procs and register method as proc, fire that method by proc.call, am i understand right?
<FromGitter>
<tenebrousedge> I mean, if you have a good reason not to, it seems like an easy solution
<FromGitter>
<tenebrousedge> you may want to define a `default_block` on your hash for any missing keys
<FromGitter>
<yorci> it looks pretty mess but modular 😃 thank you, i think i will make this way.
<FromGitter>
<watzon> You can use a method to register an event like this
<FromGitter>
<watzon> Obviously you probably want to do a little bit more there, but that's the basic idea
<FromGitter>
<yorci> yes actually i was tried procs first but procs needs to static argument size & type and return type, so i change it to define method and invoke it by macro. ⏎ ⏎ its hard to seperate logics compile time / run time for me as an long term interpreter developer 😃 , i've never write code flexible as much as i want 😃
<FromGitter>
<watzon> Crystal needs more hash functions, ie native sha2/3
<FromGitter>
<watzon> Or at the very least better documentation around them, because I can't find SHA-256 and SHA-512
<FromGitter>
<r00ster91> I'm trying to run a Crystal executable on a Raspberry Pi. I tried to cross compile with the target `aarch64-linux-gnu` and with `aarch64-linux-musl` and then run the `cc` command on the Raspberry Pi but both .o files fail with ` file not recognized: File format not recognized`
ShalokShalom has joined #crystal-lang
ShalokShalom62 has joined #crystal-lang
ShalokShalom62 has quit [Remote host closed the connection]
<FromGitter>
<r00ster91> oh the correct target seems to be `arm-unknown-linux-gnueabihf `
<FromGitter>
<Blacksmoke16> is there not a way to generate `<<: *development` using yaml nodes builder?
<FromGitter>
<Blacksmoke16> seems to be missing
laaron has quit [Remote host closed the connection]
laaron has joined #crystal-lang
<FromGitter>
<Blacksmoke16> `All "emitting" methods support specifying a "reference" object that will be associated to the emitted object, so that when that reference object is emitted again an anchor and an alias will be created. This generates both more compact documents and allows handling recursive data structures.`
<FromGitter>
<Blacksmoke16> sounds like using reference does that
laaron has quit [Remote host closed the connection]
laaron has joined #crystal-lang
<FromGitter>
<dscottboggs_gitlab> https://carc.in/#/r/6zg5 ⏎ ⏎ :D phew I'm really glad that works
<FromGitter>
<tenebrousedge> why would it not?
<FromGitter>
<dscottboggs_gitlab> > I've forget the Array :D ⏎ ⏎ I've definitely done that :D
<FromGitter>
<dscottboggs_gitlab> > why would it not? ⏎ ⏎ not every language likes to do overloads, and since macros are more dynamic I wasn't sure if they would work right or just overwrite the previous one
<FromGitter>
<tenebrousedge> ahhhhh
<FromGitter>
<tenebrousedge> thanks for explaining
<FromGitter>
<dscottboggs_gitlab> I could've gotten an error saying that `test "arg1"` was not a valid call because it was expecting `test` to take 2 args. that would've been real annoying :p
<FromGitter>
<Blacksmoke16> er i added the anchors to test/prod, but yea
<FromGitter>
<dscottboggs_gitlab> that's really interesting... I wish YAML wasn't so complicated to deal with :/
<FromGitter>
<j8r> yes, YAML can be confusing at times for humans, and is not simple to parse for machines
<FromGitter>
<Blacksmoke16> much cleaner than the JSON version of this would be tho
<FromGitter>
<Blacksmoke16> as i dont think you could extend other objects in json
<FromGitter>
<Blacksmoke16> would have have to duplicate the whole obj for each env, instead of just changing what you want
<FromGitter>
<dscottboggs_gitlab> or have that be a separate thing
<FromGitter>
<Blacksmoke16> like separate files?
<FromGitter>
<dscottboggs_gitlab> Like you could have a configurations section, then have a like `{"development": ["conf1", "conf2"] }`, then like `{"production": ["conf1", "conf3", "conf4"]}` which would each be merged together
<FromGitter>
<Blacksmoke16> true
<FromGitter>
<j8r> the logic has to be done ourselves
<FromGitter>
<Blacksmoke16> yea, vs that yaml code works with like from_yaml
laaron has quit [Remote host closed the connection]
<FromGitter>
<j8r> https://crystal-lang.org/reference/syntax_and_semantics/if_var.html#limitations ⏎ ⏎ > The above logic doesn’t work with instance variables, class variables and variables bound in a closure. The value of these kinds of variables could potentially be affected by another fiber after the condition was checked, rendering it nil.
<FromGitter>
<tenebrousedge> that the variable could be `nil` at runtime is not a bug, neither is aggressively checking for that. It's somewhat annoying, but it prevents problems
<FromGitter>
<nsuchy> interesting so because it's an instance of a class it can randomly become nil?
<FromGitter>
<j8r> if you want, yes
<jhass>
not randomly, but at any point given concurrent writes to it
<FromGitter>
<j8r> spawning fibers can change its value
<FromGitter>
<nsuchy> it's mapped to an object now
Vexatos has joined #crystal-lang
<jhass>
you don't need to make your mapping nilable if the fields are not optional to you. You'll get a runtime error at parse time then if the assumption doesn't hold
<FromGitter>
<nsuchy> Would this switch to "Not available" if nil is returned
<jhass>
sure
<FromGitter>
<nsuchy> I'm gonna pass data_type to SQLite
<FromGitter>
<nsuchy> so that would be readable enough for me
<FromGitter>
<Blacksmoke16> should the columns in the db be null if that property wasnt in the json? or is there some default value
<FromGitter>
<nsuchy> it serves as a helpful error message
<FromGitter>
<nsuchy> the other project devs don't like null data
<jhass>
take care you don't just invent your own null type here though
<FromGitter>
<Blacksmoke16> ^
<jhass>
that's easily worse then having nulls
<FromGitter>
<nsuchy> I'll discuss with them
<FromGitter>
<nsuchy> for now though
<FromGitter>
<nsuchy> default string provided by Crystal is necessary
<FromGitter>
<nsuchy> I can always improve the code later
<FromGitter>
<nsuchy> :D
<FromGitter>
<Blacksmoke16> if there are valid defaults for stuff you can define those in your mapping so if a property is missing it gets that value
<FromGitter>
<Blacksmoke16> also could be used for arrays, set it to empty array if not there etc
<FromGitter>
<Blacksmoke16> but i wouldnt just give defaults for sake of skipping nil checks
<FromGitter>
<nsuchy> it's more about, other developers, on 100s of projects written in different languages, not doing nil checks
<FromGitter>
<nsuchy> :P
<jhass>
basically if any code ever has to do something == "Not available" anywhere, you're doing it wrong
<FromGitter>
<Blacksmoke16> ^
<FromGitter>
<nsuchy> Even if my project was 100% nil-safe, every other project would have to to be okay with null values in SQLite
<FromGitter>
<Blacksmoke16> sounds like they got into a bad habit
<FromGitter>
<Blacksmoke16> null is a totally valid value
<FromGitter>
<nsuchy> yes
<FromGitter>
<nsuchy> I agree
<FromGitter>
<nsuchy> Although I'm one noob programmer who got lucky enough to find out about crystal
<FromGitter>
<nsuchy> Can't exactly rewrite the rest of the company's software even if I wanted to
<FromGitter>
<Blacksmoke16> fair enough
<FromGitter>
<tenebrousedge> a perfect world wouldn't need `NULL`, I think. But it's at least theoretically possible that this is not the best of all possible worlds
<FromGitter>
<Blacksmoke16> well it has its uses
<FromGitter>
<Blacksmoke16> like a record that hasnt been deleted yet with a deleted_at column
<jhass>
an alternative could be not inserting those rows and generate a report about not inserted items at the end of the report. If it's really just about noticing that
<FromGitter>
<nsuchy> @Blacksmoke16 Fun fact, most data is never deleted, it's retained for legal reasons and users are lied to about it being deleted xD
<FromGitter>
<Blacksmoke16> thats the point of the deleted_at column
<FromGitter>
<nsuchy> `is_deleted = true`
<FromGitter>
<nsuchy> ;)
<FromGitter>
<nsuchy> totally gone
<FromGitter>
<Blacksmoke16> lets you know when a row was deleted w/o actually deleting it
<FromGitter>
<Blacksmoke16> deleted_at would be better imo
<FromGitter>
<nsuchy> except when legal comes knocking
<jhass>
"for legal reasons" is funny in GDPR times :P
<FromGitter>
<Blacksmoke16> as you inherently get *when* it was deleted as well
<FromGitter>
<nsuchy> jhass: Very true, the bureaucracy demands fake deletes though for legal dept
<FromGitter>
<nsuchy> @Blacksmoke16 will pass that idea to the project team
<FromGitter>
<Blacksmoke16> 👍 same idea with created_at and updated_at
<FromGitter>
<j8r> Not only Facebook do that? what a surprise
<FromGitter>
<Blacksmoke16> soft deleting is also useful for reporting
<FromGitter>
<Blacksmoke16> as you're able to retain the relations between records
<FromGitter>
<tenebrousedge> theoretically the record with the `deleted_at DEFAULT NULL` column could be normalized to another table of deletions. Any null element could be an absent row in another table. But this is clearly only possible in theory
<FromGitter>
<Blacksmoke16> true
<FromGitter>
<nsuchy> okay so far so good
<FromGitter>
<nsuchy> SQLite doesn't seem to be hit too hard performance wise
<FromGitter>
<nsuchy> writing out those insert queries
<FromGitter>
<tenebrousedge> how long does it take?
<FromGitter>
<nsuchy> 500 keys took 3 seconds
<FromGitter>
<nsuchy> there's a lot of tables though
<FromGitter>
<nsuchy> I'm not using transactions yet either
<FromGitter>
<nsuchy> just a bunch of single queries
<FromGitter>
<nsuchy> @tenebrousedge So if I switch to a transaction the issue will go away?
<FromGitter>
<davidcolombogit> No "Actually, SQLite will easily do 50,000 or more INSERT statements per second on an average desktop computer. But it will only do a few dozen transactions per second."
<FromGitter>
<davidcolombogit> Transactions are the slow thing
<FromGitter>
<nsuchy> so we need to group each insert statement into one big transaction?
<FromGitter>
<tenebrousedge> one transaction would probably be faster
<FromGitter>
<nsuchy> I have `DB.open "sqlite3://./src/example-json-files/dbname.sqlt" do |db| ... end`
<FromGitter>
<nsuchy> what would I put inside to do a transaction with one query
<FromGitter>
<nsuchy> (in production there would be several queries)
<FromGitter>
<davidcolombogit> We would still need dozens of transactions to add more than a million rows xD
<FromGitter>
<nsuchy> @davidcolombogit you're making the asumption a transaction can only have one query
<FromGitter>
<nsuchy> in fact they can have several
<FromGitter>
<davidcolombogit> Yes
<FromGitter>
<davidcolombogit> But by default, each INSERT statement is its own transaction.
<FromGitter>
<nsuchy> So we group them into one transaction
<FromGitter>
<nsuchy> problem hopefully solved
<FromGitter>
<davidcolombogit> What do you think about that
<FromGitter>
<davidcolombogit> " ⏎ Another option is to run PRAGMA synchronous=OFF. This command will cause SQLite to not wait on data to reach the disk surface, which will make write operations appear to be much faster. But if you lose power in the middle of a transaction, your database file might go corrupt."
<FromGitter>
<nsuchy> `tx.commit` is at the bottom
<FromGitter>
<nsuchy> nothing gets written to the DB
<FromGitter>
<tenebrousedge> so, one question is why you're not storing the raw json in SQLite https://www.sqlite.org/draft/json1.html and why SQLite is being used instead of Mongo or Postgres, which have even better JSON support
<FromGitter>
<davidcolombogit> Well, MySQL should do the job also
<FromGitter>
<nsuchy> any reason why transactions are doing nothing
<FromGitter>
<Blacksmoke16> Did you commit the transaction
<FromGitter>
<nsuchy> yes
<FromGitter>
<nsuchy> tx.commit
<FromGitter>
<nsuchy> Queries as `tx.connection.query("INSERT INTO IMPACT (ID, BASEMETRICV3, BASEMETRICV2) VALUES (?, ?, ?)", [cve_item_id, "", ""])`
<FromGitter>
<Blacksmoke16> and they're handled in the same way?
<FromGitter>
<Blacksmoke16> because it would be easy to write some slow code, then refactor it in a better way to get a big improvement
<FromGitter>
<Blacksmoke16> im just a bit skeptical that the lang change alone resulted in *that* big of an improvement
<FromGitter>
<nsuchy> /shrug
<FromGitter>
<nsuchy> A rewrite into Crystal forced us to use better practices
<FromGitter>
<nsuchy> Crystal is at least partially responsible
<FromGitter>
<tenebrousedge> 👍
<FromGitter>
<jgaskins> That, I think, is one of the big wins with Crystal over other languages that goes unnoticed a lot.
<FromGitter>
<davidcolombogit> Definitely
<FromGitter>
<Blacksmoke16> i could go with that
<FromGitter>
<jgaskins> You can gain a fair bit of performance by porting code directly and then tune it to use things like stack-based objects, replace allocations of large strings with direct IO, etc, and gain even more performance.
<FromGitter>
<jgaskins> That last mile is unavailable in some languages.
<FromGitter>
<nsuchy> Also SQLite bulk transactions are awesome
<FromGitter>
<davidcolombogit> Signing off for today folks, have great night (or day depending on your country xD) ⏎ Thanks for the help so far!
<FromGitter>
<Blacksmoke16> o/
<FromGitter>
<jgaskins> 👋
<FromGitter>
<nsuchy> 😄😄😄
ashirase has quit [Ping timeout: 246 seconds]
DTZUZU has quit [Ping timeout: 258 seconds]
ashirase has joined #crystal-lang
<FromGitter>
<nsuchy> Crystal is amazing
<FromGitter>
<nsuchy> Can not believe we got a 14 hour script to do the same job in just a second
<FromGitter>
<Daniel-Worrall> Scripting is accessible but compiled is efficient
jokke2 is now known as jokke
jokke has quit [Quit: WeeChat 2.4]
jokke has joined #crystal-lang
<FromGitter>
<Blacksmoke16> i noticed that yaml anchors are built before the key
<FromGitter>
<Blacksmoke16> which kinda breaks my stuff :/