HumanG33k has quit [Remote host closed the connection]
HumanG33k has joined #crystal-lang
HumanG33k has quit [Remote host closed the connection]
HumanG33k has joined #crystal-lang
postmodern has joined #crystal-lang
oddp has quit [Ping timeout: 240 seconds]
<FromGitter>
<watzon> @mamantoha any idea what it would take to get MITM HTTPS proxy functionality working in your http_proxy project?
<FromGitter>
<Blacksmoke16> as a proxy server?
<FromGitter>
<watzon> Yeah. I've got it somewhat working, but for some reason when I log out the `context` for each request I only get `CONNECT` methods. Idk what's going on.
<FromGitter>
<Blacksmoke16> not easier to use something already existing? https://docs.traefik.io/ is pretty legit if thats what you're trying to do
<FromGitter>
<watzon> I'm trying to add an example to Marionette showing how to use a proxy server for capturing HARs
<FromGitter>
<watzon> Trying to keep things small and in Crystal ideally
<FromGitter>
<Blacksmoke16> ah, prob not that then
<FromGitter>
<plambert> `HTTP::Server::Context` only contains an `HTTP::Request` and an `HTTP::Response`.
<FromGitter>
<plambert> And `HTTP::Request` has `remote_address` but I see no way to tell what the local address was.
<FromGitter>
<plambert> Calling `getsockname` would work if I had a `Socket` to call it on... and obviously I could save the `Socket` objects returned by the `bind_tcp` and `bind_tls` calls. But that still doesn't tell me on which `Socket` a given request was received.
<FromGitter>
<plambert> Here's what I'm trying to do: write an `HTTP::Server` based web server that, if a request comes in on the HTTP port (80) always returns a 302 redirect to HTTPS on port 443.
<FromGitter>
<plambert> It looks right now like I will have to create two different `HTTP::Server` instances, one for each binding, and spawn one in a separate fiber?
<FromGitter>
<grkek> Why would you want to do that?
<FromGitter>
<grkek> Makes no sense just use NGINX to redirect non-https requests
sp3ncer has joined #crystal-lang
sp3ncer has left #crystal-lang [#crystal-lang]
<FromGitter>
<grkek> If youre still desperate just start two servers one with 80 port it will take http and just redirect to https on port 443
<FromGitter>
<plambert> Hmm.
<FromGitter>
<plambert> I'm not sure why you care why I want to do it.
<FromGitter>
<plambert> Of course I could use nginx.
<FromGitter>
<plambert> There's no reason for me to use Crystal at all... there are many other languages to choose from.
<FromGitter>
<plambert> Why does it make sense to write a web server in Crystal at all, when it does nothing that nginx or apache or 800 others don't do?
<FromGitter>
<plambert> Anyway...
<FromGitter>
<plambert> Yes, as I said, I'll have to create two different `HTTP::Server` instances and spawn them in separate fibers.
<FromGitter>
<plambert> I am not, however, desperate.
<FromGitter>
<plambert> Thank you for taking the time to respond. I do appreciate it.
<FromGitter>
<plambert> I'm sorry you think what I'm trying to do "makes no sense."
<FromGitter>
<plambert> To me, it makes no sense to run two different services to respond to requests on different ports.
<FromGitter>
<plambert> There's literally no reason to do that except "a feature isn't yet implemented in `HTTP::Server`"
<FromGitter>
<plambert> Which is not a complaint... it's obviously not important to many people.
<FromGitter>
<bararchy> @watzon nice! looks great :) btw, you can check our fork, some of it might not be relevant for you, but it has alot of battle-tested code and fixes, also MT safety and other improvments
oddp has joined #crystal-lang
jhass|off has joined #crystal-lang
jhass has quit [*.net *.split]
jhass|off is now known as jhass
alexherbo2 has joined #crystal-lang
<FromGitter>
<sam0x17> way to get name of current shards target from a macro? (guessing no)
<mps>
Blacksmoke16: however you like, my IRC trigger recognize both. :)
<mps>
Blacksmoke16: thanks for notice, will test later today
<FromGitter>
<sam0x17> @plambert assuming you are behind a load balancer (as is the case for most people running in some sort of cluster or instance group in [insert cloud vendor here], you can do something like what I do here in spider gazelle to redirect http to https behind a google load balancer: ⏎ ⏎ ``` code paste, see link``` ⏎ ⏎ Others are right to question you -- typically you do not run HTTPS directly for a web
<FromGitter>
... instance regardless of the language/environment, but instead defer this task to a load balancer. Even if you are just running a website off of a single VPS, normally you would do this part in apache or nginx. ... [https://gitter.im/crystal-lang/crystal?at=5ef4540a405be935cdbfc680]
<FromGitter>
<sam0x17> in that last case though, I am not sure what stuff you have to do to get an SSL cert to work -- have never done it that way in pure crystal
<FromGitter>
<sam0x17> but yeah TLDR: crystal doesn't give you protocol directly, but it's often available via a header (`X-Forwarded-Proto` in this case) if you are behind a load balancer or nginx or something
<sorcus>
`Unhandled exception: Hostname lookup for localhost failed: No address found (Socket::Addrinfo::Error)` - `crystal play` not working X-)
<sorcus>
`127.0.0.1 localhost` in `/etc/hosts` resolve my problem. :-)
<sorcus>
Hmm... Can i get `Bytes[48, 49, 49, 48]` from `Bytes[48, 49]` and `Bytes[49, 48]`?
<jhass>
mmh, what for?
<jhass>
if for writing somewhere, just write it twice?
<FromGitter>
<codenoid> hi folks, it's 25-06-2020 how you doing, hope everything getting better
<sorcus>
jhass: Mmm, nope. Not for writing.
<jhass>
sorcus: probably won't get prettier than expanded = Bytes.new(src.size * 2); expanded.copy_from(src); expanded[src.size, src.size].copy_from(src) (didn't test, but something along those lines)
<jhass>
well the other approach is writing it twice to an IO::Memory, get the slice from that then I guess
<jhass>
if it's really just 2 to 4, I would also consider Bytes[src[0], src[1], src[0], src[1]]
<sorcus>
jhass: Yeah, your example looks working :-)
<sorcus>
jhass: Thanks you :-)
<jhass>
yw
postmodern has quit [Quit: Leaving]
livcd has joined #crystal-lang
<FromGitter>
<bararchy> hi @codenoid long time
HumanG33k has quit [Ping timeout: 240 seconds]
<FromGitter>
<sam0x17> how to write an empty `Set(String)` literal? it lets me do `Set(String){""}` but not `Set(String){}`?
<FromGitter>
<sam0x17> (can't use `Set(String).new` because macro stuff)
<jhass>
I think .new is the only way, sorry
<jhass>
Set(String) {a, b} is just syntax sugar for tmp = Set(String).new; tmp << a; tmp << b; tmp; maybe your macro can skip the sugar?
HumanG33k has quit [Remote host closed the connection]
HumanG33k has joined #crystal-lang
<jhass>
looks terrible tbh :P
<FromGitter>
<evvo> :D
<FromGitter>
<Zenohate> Hello everyone here, I wanted to ask where can I find the reference for the memory layout of structs. The reference says the structs have a well defined layout but it is not documented directly within or linked from here
<FromGitter>
<bararchy> @Zenohate can I ask out of interest what do you need that for?
<FromGitter>
<Zenohate> I want to move a prototype software from C+
<FromGitter>
<Zenohate> 1) to crystal, and it relies on memory mapped data a lot
<FromGitter>
<Zenohate> to make it any more elegant in crystal than it is in C++, I would need to be able to reliably mapped structure to crystal ones
<FromGitter>
<bararchy> Got you
Flipez has quit [Ping timeout: 258 seconds]
ghanima has quit [Ping timeout: 260 seconds]
<FromGitter>
<Zenohate> Issue #2422 seems to imply that Crystal follows C structures layout, but as far as I know, there is no definitive layout for these, so does it means that in crystal we are left with no concept of alignment in memory and no way to reason about it? ⏎ ⏎ If that is the case, I would actually have to reimplement the structs i would be working with as pseudo properties that I would map by hand or with a macro
<FromGitter>
... inside of a StaticArray(UInt8) which would in theory perfectly work (albeit not being the top elegant solution but I had to do the same in C++ anyway)
<yxhuvud>
You could probably file an issue about it being less obvious what the guarantees are than the text imply. But what you could also is to use a struct defined in a lib namespace, then you would get the same layout as the layout the used c compiler would give on your local machine.
<FromGitter>
<Zenohate> The issue is that C compilers give no guarantees over structs layout, hence the fact that I had to hack it into my preleminary C++ prototype. ⏎ ⏎ I have been scouring the reference manual of LLVM which is also very silent on the topic, most likely because most of the time you want the layout to be platform dependent for optimization reasons
<FromGitter>
<Zenohate> Okay, so I guess that what is the most likely to give me any control would be using the `[@Packed]` attribute, and/or the byte array shenanigans trick, but I would be in favor of removing the part that says that the layout of a Struct is well defined in the documentation, replacing with something on the line of: ⏎ ⏎ "packed structs have a well defined layout, struct have a layout that is well defined
<FromGitter>
... within each platform, following the C layout for a struct on the same platform."
<FromGitter>
<asterite> @sam0x17 that's a bug, I think empty literals when the generic type is known should work well. The problem is that that should also work with aliases, so maybe we should move the error to the semantic phase.
alexherbo21 has joined #crystal-lang
alexherbo2 has quit [Ping timeout: 265 seconds]
alexherbo21 is now known as alexherbo2
<FromGitter>
<phykos> why if I pass to the compiler `--emit asm` it still generated an executable?
<FromGitter>
<Blacksmoke16> because its saying "also emit x"
<FromGitter>
<phykos> oh
<FromGitter>
<phykos> kk
<FromGitter>
<Blacksmoke16> so builds what you told it, but also creates a file with the asm
<FromGitter>
<phykos> `Comma separated list of types of output for the compiler to emit. You can use this to see the generated LLVM IR, LLVM bitcode, assembly, and object files.` ⏎ thats what man says
<jhass>
sorcus: While BitArray itself is a struct, it's underlying datastore is Pointer.malloc'ed, so a heap reference
<sorcus>
jhass: Mmm... So i can't create a copy of this, without affect to original instance?
<jhass>
mmh, it should implement `dup` properly but does not. So atm, short of monkey patching a dup in, yeah. PR to add dup is welcome I think :)
andremedeiros has joined #crystal-lang
<andremedeiros>
Hello everyone! I've been playing around with Crystal and would really love to contribute. What's the best way for a newcommer to grab an issue and start working on the compiler?
<jhass>
sorcus: mmh, well there's no Pointer#dup you could delegate to, so I'm afraid it's a little hardcer than that :)
<sorcus>
jhass: Oh. Ok. :-D
<andremedeiros>
sorcus: what are you trying to solve?
<jhass>
andremedeiros: Can't say there's a bad way! Try to find something that bugs your personally, ask here or on the forum for pointers about the specific thing. Before starting anything big or changing any semantic from what currently happens, make sure to open an issue to discuss your proposal :)
<sorcus>
andremedeiros: Some work with bits... :-D
<sorcus>
jhass: What should i have to do? Add a `dup` to `Pointer`? X-)
<jhass>
nah, that doesn't make too much sense :)
<jhass>
inside BitArray, you got a @pointer and a @size. You gotta allocate some new memory, copy the current data there and use that to construct a new BitArray
<andremedeiros>
without too much context, `dup` are specialized methods that fit the data structure you're trying to duplicate.
<andremedeiros>
copying memory back and forth won't do it for a lot of cases because structures might be referring to other bits of memory you need to duplicate too (think linked lists, hash values, etc)
<jhass>
in this case it's fine though :)
<andremedeiros>
ah yeah, it's just a pointer with a size it looks like!
<andremedeiros>
sorcus: mind if I take a stab at implementing a dup for you?
<sorcus>
andremedeiros: That's would be great i think. *HAPPY*
<sorcus>
jhass: Mmm, can you explain one more thing?
<jhass>
sorcus: structs are copied by value, you always get a new one! In the case of BitArray, the struct just wraps a pointer though. So if you copy the outer struct, the copy has the same inner pointer as the original
alexherbo2 has quit [Remote host closed the connection]
<sorcus>
jhass: Mmm, cool. And the `dup` method has an impact on performance or it's the same as copying by value?
<sorcus>
jhass: Sorry for this questions, i just try to understand more things... X-)
<jhass>
it's definitely slower, since it needs to obtain new memory from the heap, whereas (most) struct copying happens on the stack
<sorcus>
jhass: Hmmm... So malloc is a heap-only allocation method?
<jhass>
yes
<sorcus>
jhass: And it's not possible to use a stack for bits in BitArray?
<andremedeiros>
sorcus: i'm on it
<jhass>
It's possible but then it would need to become something similar to StaticArray, where the size of the array must be known at compile time
<FromGitter>
<phykos> Is there any multi threading library?
<andremedeiros>
jhass: what's the rule of thumb for using new vs initialize
<jhass>
I haven't really figured out one :D
<jhass>
I guess in this case it can make sense to make initialize just private def initialize(@pointer, @size) and have the new's handle everything else
<sorcus>
jhass: Hmmm... Very interesting. I learned a little bit more today. :-D
<jhass>
andremedeiros: but neither is really wrong, Crystal just synthesizes a new for every initialize after all
<sorcus>
phykos: Crystal's stdlib?
<jhass>
physkos: Compiling with -Dpreview_mt enables an experimental runtime that runs your fibers (spawn {}) on a thread pool
<FromGitter>
<samuell> Folks, I have been doing some very simple and naive performance comparison of different language implementations of a simple task: Read a file containing DNA letters line by line, and calculate the fraction of G and C:s, to the count of all G, C, A and Ts ("GC-content"): https://github.com/samuell/gccontent-benchmark ⏎ ⏎ I just added my naive Crystal implementation there, but as these are my very first
<FromGitter>
... lines of Crystal, there's probably a lot of room for optimizations.
<jhass>
mmh, the only thing I see is doing the line parsing yourself to avoid allocating the intermediate string for it
<jhass>
so a state variable "comment", which you flip on seeing '>' and then flip back on seeing '\n'
<jhass>
and then you can each_char on the file instead of each_line
<FromGitter>
<samuell> @jhass Good points
<jhass>
Char is just a thin wrapper around UInt32, I doubt you gain a lot going by bytes, but of course the above would work bytewise as well
<jhass>
well actually nvm, UTF-8 decoding is kinda expensive, so yeah, going by bytes might actually be faster
<FromGitter>
<Blacksmoke16> dont suppose doing `&+=` would help, assuming the context wouldn't allow for overflow?
<FromGitter>
<samuell> The fact is actually ASCII, so UTF-8 is not needed ... but didn't figure out how to change that
<jhass>
sure, if we can be sure of that or don't care about it
<andremedeiros>
jhass: turns out properties are protected so I can Pointer.copy_to directly hopefully
<FromGitter>
<samuell> I know the File.new() takes an encoding parameter, but never found out what the possible values are except UTF-8 ...
<jhass>
andremedeiros: sounds good
<FromGitter>
<Blacksmoke16> @samuell also fwiw you're not closing the file
<FromGitter>
<Blacksmoke16> i doubt it has any bearing on the benchmark, but might be a better habit to use `File.open` so it gets closed at the end
<FromGitter>
<samuell> @Blacksmoke16 Ah, thanks for the heads up
<FromGitter>
<Blacksmoke16> yea `File.open` closes file after yielding, otherwise just need to call `.close` on it
<FromGitter>
<asterite> @samuell is it necessary to read line by line? What happens if you read byte per byte? I guess newlines won't be taken into account for the count so it'll work the same, but probably faster (no strings allocated at all)
<FromGitter>
<asterite> Oh, I guess lines with `>` needs to be skipped...
<FromGitter>
<samuell> @asterite Reading byte-by-byte should be OK too, if we can just probably skip header lines (those starting with '>'). I have restrained from doing things like memory mapping files etc in the benchmark, as these problems often require parsing files of hundreds of gigabytes in size.
<FromGitter>
<samuell> Exactly
<FromGitter>
<asterite> I don't think there's any way to further optimize the code unless you do what jhass says, which is to have a byte buffer and use `io.read(...)` or `io.peek` and then manually parsing the lines. There's no way to reuse string memory in Crystal and the only performance problem I see in the code is that each line allocates a new string. But the code is still one of the fastest out there, so I think it's fine?
<FromGitter>
<samuell> I like this version in that it is pretty readable, and intelligible too.
<jhass>
I don't see a need for a byte buffer, just skip bytes after '>' until a '\n'
<FromGitter>
<samuell> @rbpynet Not sure if what you're after, but if you install the Crystal docs in https://devdocs.io/ , they will be available offline too.
<mps>
Blacksmoke16: np, thanks for upgrading and making it
<FromGitter>
<rbpynet> @Blacksmoke16 @samuell awesome thanks! I'm checking it now
<andremedeiros>
Does Crystal have a story of how to embed files at compile time, kind of like go generate?
<andremedeiros>
wow the api for this is so freaking nice
<andremedeiros>
coming from go, this is just so coool
svipal has joined #crystal-lang
<svipal>
hey
andremedeiros has quit [Remote host closed the connection]
andremedeiros has joined #crystal-lang
andremedeiros has quit [Quit: ZNC 1.8.1 - https://znc.in]
<FromGitter>
<wontruefree> I gave this talk a while ago about the case for crystal. It is a little rough but I think presents a good case for Crystal https://youtu.be/VyjKNy-uUmg
<FromGitter>
<wontruefree> check it out and let me know
<FromGitter>
<wontruefree> it has come a long way since the 2 years I originally gave it
<svipal>
I really really like crystal tbh
<sorcus>
:-)
andremedeiros has joined #crystal-lang
andremedeiros has quit [Quit: ZNC 1.8.1 - https://znc.in]
andremedeiros has joined #crystal-lang
<FromGitter>
<wontruefree> you are in the right place then
<FromGitter>
<Daniel-Worrall> Devdocs is on like 0.31 or 0.32. I have a pr for 0.34 open and I tried to update it to 0.35 but the manual checks failed so I have to look into it when I have time. Devdocs is good, but they're just not keeping it updated right now
<andremedeiros>
any reason why this couldn't be done dynamically?
<andremedeiros>
i realize I should put my code where my mouth is -- Daniel, mind if I push another PR that works off GH releases?
<FromGitter>
<j8r> huf, oprypin, you are another fan of pushd/popd
<FromGitter>
<j8r> I don't see the benefit here vs good old cd
<oprypin>
j8r, it's like `Dir.cd { }` u know
<FromGitter>
<Daniel-Worrall> Having it update like that is against the devdocs design philosophy I think. Also all I was doing was bumping the versions from what was already in place. You're welcome to do your own PR though. I want to see latest crystal on devdocs
<FromGitter>
<j8r> oprypin I know, that's why I stick with cd ;=
<FromGitter>
<j8r> ;)
<andremedeiros>
split
<andremedeiros>
^ disregard
HumanG33k has quit [Quit: Leaving]
HumanG33k has joined #crystal-lang
HumanG33k has quit [Remote host closed the connection]
HumanG33k has joined #crystal-lang
ep4sh has joined #crystal-lang
deavmi has quit [Quit: Eish! Load shedding.]
deavmi has joined #crystal-lang
<sorcus>
Good night :-)
<svipal>
night !
ep4sh has quit [Quit: WeeChat 1.9.1]
<FromGitter>
<j8r> What's the "danger" of using `Array(Pointer(SomeStruct))` vs `Array(SomeClass)`? How can I know if using a pointer is safe or not?
<oprypin>
j8r, i was gonna say "like obviously dangerous dont do it" but yeah i cant point out anything in particular immediately
<oprypin>
j8r, but how do you even create a pointer to a struct persistently?
<oprypin>
you just cant do `x = SomeStruct.new; a << pointerof(x)`, it'll be pointing to garbage
<FromGitter>
<j8r> Hum in fact, in my case it is more like `StaticArray(Pointer(Tile), SIZE).new { t = Tile.new; pointerof(t) }`
<FromGitter>
<j8r> may not change anything
<oprypin>
j8r, umm have u tried printing that array
<FromGitter>
<j8r> it prints pointers
<FromGitter>
<j8r> (of course)
<oprypin>
j8r, can u provide the output
<FromGitter>
<j8r> ho, seems to be all the same pointer
<oprypin>
oh what do u know
<oprypin>
j8r, `t = Tile.new` puts an item on the program's stack, then you take a pointer of it, and that's then disposed of
<oprypin>
next time the exact same memory location is reused because why not
<oprypin>
more than that, after you're done with all of this, that memory location will be used for yet another thing
<oprypin>
`a = StaticArray(Pointer(Tile), SIZE).new { t = Tile.new; pointerof(t) }; t2 = Tile.new; p a[0]` will likely be equal to `t2` now
<oprypin>
ok no that one is harder to reproduce
<FromGitter>
<j8r> I see, not good for my case
<oprypin>
j8r, it's not even that it's dangerous long-term, it's imediately just broken
<oprypin>
if you really have to use a pre-existing Tile struct, just create a class wrapper for it. `class TileStuff; @x : Tile; end`
<oprypin>
though i think Box(T) happens to be such a class too.... though this is super weird and i dont think anyone has done this https://carc.in/#/r/9bsb
DTZUZU has quit [Quit: WeeChat 2.8]
<FromGitter>
<j8r> I will have to find something else then...
<FromGitter>
<j8r> I have some literature about how games find nearest entities, the "broad phase"
<FromGitter>
<j8r> I'll start simple: group entities in chunk, then check this.
<oprypin>
BSP i guess
<FromGitter>
<j8r> yeah there are several different options with different kind of trees
<oprypin>
j8r, basically yea you can split the world into a grid of squares and you know that if the things are not in the same square they couldnt possibly be near
<oprypin>
binary space partitioning is basically... what if i then divided those squares... and then what if i divided even those squares
<oprypin>
but probably unnecessary
<FromGitter>
<j8r> I aim to have a huge map, that's why I used an Array of static arrays to have the maximum length
<oprypin>
j8r, wait, now that one i don't quite understand the purpose of
<FromGitter>
<j8r> Yes at first I was thinking to use Tiles for this squares
<FromGitter>
<j8r> of?
<oprypin>
why do you need static arrays
<FromGitter>
<j8r> because I cannot have an Array of 1 000 000 elements
<FromGitter>
<j8r> but I can have 10 000 of static arrays, which are sized like 1 000
<oprypin>
why cant you? O_o
DTZUZU has joined #crystal-lang
<oprypin>
and why do you need to
<FromGitter>
<j8r> not correct numbers, but you see
<oprypin>
and a shitton of unexpected memory region copies
<FromGitter>
<j8r> I see this coming
<oprypin>
anyway, when u get tired, change to array and see performance improev
<FromGitter>
<j8r> right, I will compare in real world
<FromGitter>
<j8r> Array is fine
<FromGitter>
<plambert> @sam0x17 Thank you for the well thought out reply. This server is for my personal website, running on a micro intel server in my garage. There's no separate load balancer, or anything like that.
csaba has quit [Ping timeout: 246 seconds]
<FromGitter>
<j8r> BTW I can use the "jhass tricks" with Hash too :D
<oprypin>
[00:15:59] <oprypin> j8r, oh it does? nice. yea mybe thats the way. and each of the chunks themselves might be one array (i assume that's what jhass suggested)
<FromGitter>
<j8r> I meant, one hash instead of 2
<FromGitter>
<j8r> (I was initially doing this :/)
<oprypin>
oof
<oprypin>
of course it'd be `Hash({Int32, Int32},`
mps has left #crystal-lang [#crystal-lang]
<FromGitter>
<j8r> ha good idea
<FromGitter>
<plambert> @sam0x17 I've got a little bit of experience with large scale websites, and luckily I'm not directly responsible for one now, though I guess it depends on a person's definition of "large scale." However, this is a toy server for in my garage, because I need named virtual hosts (or host-based routing, or whatever the kids are calling it these days) and listen sockets on multiple ports. Except of course "need"
<FromGitter>
... isn't the right word for a side project. ;)
<FromGitter>
<j8r> instead of each time converting an integer to X and Y coordinates
csaba has joined #crystal-lang
<FromGitter>
<plambert> @sam0x17 While I totally understand that a lot of emphasis is placed on the sort of use cases where Ruby on Rails is used, there's no reason not to expand the use cases when the opportunity presents itself. I have a patch for `HTTP::Request` that adds the local address and protocol info when it's available. Once I've tested it out a bit, I'll submit a PR. My code is awful, so I look forward to a lot of
<FromGitter>
... great feedback before it's even close to mergeable. ;)