<mbj>
dkubb: All of these depend on reproduction of the magic encoding header.
<dkubb>
ahh ok
<dkubb>
does parser not expose it?
<mbj>
Not explicitly.
<mbj>
Indirectly via the encoding of the string children of str nodes.
<dkubb>
I would expect parser to see it, and then change some state that affects the ast production while parsing the file
<mbj>
Exactly. I'd need to scan the AST on how str nodes are actually encoded.
<mbj>
But TBH, I'm probably gonna saw: Unparser only supports UTF-8 files.
<mbj>
s/saw/say/
<dkubb>
each ast node should almost have a slot saying what the encoding was when it was created
<dkubb>
I think that's fine
<mbj>
node = s(:str, "foo")
<mbj>
node.children.first.encoding would work
<mbj>
It reflects the encoding mode parser was in while parsing.
<dkubb>
right, but you don't necessarily always have a string node in the ast
<mbj>
Yeah.
<dkubb>
I guess the encoding header only affects string nodes though, or does it?
<mbj>
I'll probably also affect nodes that use str nodes internally.
<mbj>
regexp, xstr
<mbj>
and dsym
<mbj>
also I'd expect sym reflects the encoding.
<mbj>
But you can also have an .rb files without any of these.
<mbj>
BTW: 10e100000000 gets internally represented as s(:int, Infinity)
<mbj>
While preprocessing I catch these and replace it with the AST for Float::INFINITY
<dkubb>
I wonder if I should do that in sql
<dkubb>
*do that kind of thing
<dkubb>
when parsing literal numbers, I need to maintain the precision, so I'm probably going to encode :float nodes using BigDecimal
<dkubb>
at least internally in the parsed ast
<mbj>
I needed to do this to support normalize(parse(source)) == normalize(parse(generate(normalize(source))))
<mbj>
Because the nature how parser works is asymetric.
<dkubb>
BigDecimal is a bit annoying in how it evaluates 2E3, it'll expand it to 2000.0
<mbj>
Its probably a good choice NOT to map to the native ruby primitives.
<dkubb>
yeah?
<mbj>
As you did, mapping float to BigDecimal.
<mbj>
I agree here.
<dkubb>
I had another idea about making a number object that can losslessly represent the literal numbers
<mbj>
You dont need to do calulus. So this is also a nice idea.
<dkubb>
BigDecimal is probably good enough for :float. I think Integer can represent :integer though
<mbj>
*calculus
<mbj>
dkubb: I'm gonna add the rubyspec round trip as an integration spec to unparser. Than release and see how much unparser releated bug reports come in.
<mbj>
dkubb: We could add more code to the self test. But I think rubyspec should be enough.