<snusnu>
dkubb: are you thinking about api like attribute :person_id, :references => people.id ?
<snusnu>
dkubb: i'm not entirely sure what you're getting at .. do you want to omit declaring the FK attribute explicitly like we did in DM1 ?
<snusnu>
dkubb: cause imo that doesn't feel right at the schema level
<snusnu>
dkubb: i'd like to see all attribute definitions explicitly .. foreign keys could provide sugared api like the :references option above, but then they would have the limitation of not being able to assign a name to that FK
mbj has quit [Ping timeout: 245 seconds]
xargoon has quit [Read error: Operation timed out]
xargoon has joined #rom-rb
<snusnu>
dkubb: btw, offtopic, i was just thinking about implementing a Pointer class in ruby ;)
<snusnu>
dkubb: kinda generalized from your relvar implementation
<snusnu>
dkubb: the idea being that it's an instance that holds a pointer to an otherwise immutable object
_br_ has quit [Read error: Operation timed out]
_br_ has joined #rom-rb
bf4 has joined #rom-rb
bf4 has quit [Ping timeout: 252 seconds]
<postmodern>
snusnu, ping
<postmodern>
snusnu, i'd love to help but it seems things are still in flux
<postmodern>
snusnu, also i wish the specs would be normalized
<postmodern>
snusnu, im more used to 90% code coverage, so if I need to refactor some internal method, I can run the specs for it separately
<postmodern>
snusnu, but i think i will start rewriting code I developed for Ronin, against ROM libraries
<postmodern>
snusnu, like write a virtus-optparse
<snusnu>
postmodern: hey, yeah, re specs for internal methods, how can i say it, i'm not totally against having them, and i'm not totally for having them .. as long as they are of help, they're obviously useful, but there's a thin line between being useful and being a maintenance burden
<postmodern>
snusnu, if there's no specs for a method, you can guarantee i wont touch it
<snusnu>
sometimes i feel like we would benefit from some directory that contains those tests, and we commit to killing them regularly (with the ability to resurrect them)
<snusnu>
postmodern: i totally understand you btw, i fully realize how they are a crucial aid for someone stepping fresh into a codebase, trying to help out
<snusnu>
the thing is, we had/have projects where there are SO many specs, that it just isn't fun working on the project anymore
<snusnu>
because literally every change you make, results in shitloads of spec updates
jgaskins has quit [Quit: This computer has gone to sleep]
<postmodern>
snusnu, if you have too many specs, your methods are probably too complex
<postmodern>
snusnu, also i hate the unit vs. integration duplication of tests in DM1
bf4 has joined #rom-rb
<snusnu>
postmodern: fwiw, i agree that at some point, projects reach a state where that maintenance burden becomes slightly less, because the system gets more and more stable overall
<snusnu>
postmodern: however, during initial development, which often turns out to be a spike more than anything else, it surely hurts velocity
<postmodern>
snusnu, i don't test during code spikes
<snusnu>
exactly, neither do i (at least not more than i need to verify my assumptions)
<postmodern>
snusnu, i usually write tests after the code has settled down and works in my examples
<postmodern>
snusnu, then i merge the branch
<snusnu>
the thing is, a lot of rom code are spikes ;)
<postmodern>
snusnu, but that's besides the point, I really don't know where ROM is going or what it's going to end up looking like, so I can't really contribute beyond refactoring
<snusnu>
well, that's probably not entirely true either, but yeah, we oftentimes experiment with an approach, only to find out it doesn't really lead us anywhere
<snusnu>
postmodern: so, that'd be a more or less isolated task to work on ;)
<snusnu>
postmodern: axiom-schema is built on top of whitequark/ast, so processing should be fairly simple
<postmodern>
snusnu, i don't understand why you need an AST?
bf4 has quit [Ping timeout: 264 seconds]
<snusnu>
postmodern: because it's an easily processable IR that's fairly easy to construct from a DSL
<postmodern>
snusnu, this appears to be exactly the same as your internal object representation
<postmodern>
snusnu, why do you need IR?
<postmodern>
snusnu, im sorry but i do not understand the end goal?
<postmodern>
also your not really generated IR code, your just generating another AST
<postmodern>
which looks almost identical to the first AST
<snusnu>
postmodern: instead of defining ad hoc objects that encapsulate the declared schema definition and are used to build the "real" (immutable) axiom objects, we've decided that by using an AST, we don't have to write those intermediate objects ourselves, and once we have a standardized IR, other things (like migrations) can easily be built too
<postmodern>
snusnu, i still dont follow
<snusnu>
postmodern: btw the current axiom-schema contains no processing so far, i only started it a few days ago, and am currently finishing the building of the ast
<snusnu>
postmodern: hmm
<postmodern>
what is the goal here? like what is axiom-schema supposed to do?
<snusnu>
axiom schema is supposed to be the part that allows you to define all (base) relations your app will work with .. based on that info, mappers will be generated that work with those relations
<snusnu>
also based on that info, DDL can be generated to perform migrations
<snusnu>
without that code being mixed into the objects that perform the actual work
<snusnu>
we want the code that makes up rom during *runtime*, to not be intertwined with code that only exists to build up objects we need
<postmodern>
sounds like your creating something even more complex than DM1 :(
<snusnu>
since we focus on immutability as much as possible, we need a lot of object building .. we often "collect" data (in mutable objects) that is then used to construct the objects used at runtime .. those are immutable
<postmodern>
I wouldn't even bother generating migrations from the mappers
<snusnu>
?
<snusnu>
neither would i, they have nothing to do with each other ..
<snusnu>
it seems like i couldn't yet describe the plan properly
<postmodern>
then don't :)
<postmodern>
decouple the migrations from the mappers entirely
<snusnu>
they are entirely decoupled of course
<postmodern>
they are connected via the base-relation code
<snusnu>
how so?
<snusnu>
sorry, now i can't follow ;)
<postmodern>
ok this is why i can't contribute to ROM :/
<snusnu>
well, what can i say then …
<postmodern>
there's just too many layers for me to understand
<postmodern>
and it's not entirely clear how they communicate
<snusnu>
re your thoughts on rom being more complex than DM1 … there's 2 sides to that imo .. on the one hand, *of course* it is more complex, because it's a freakin' datamapper and no activerecord
<snusnu>
on the other hand, we try hard, to work with simple isolated pieces, that by themselves, are actually not that complex at all
<postmodern>
i've seen a dozen trivial datamapper-like libraries so far
<postmodern>
much less complex than ROM
<postmodern>
also it seems like there's an impudence mismatch with the migrations
tpitale has joined #rom-rb
<postmodern>
dkubb was talking about using veritas (now axiom) to express the migrations as relational algebra
<snusnu>
yeah, migrations, i should've been more precise, what i was referring to was more like auto_migrate! in datamapper, a way to generate the DDL that describes your db
<postmodern>
also i don't understand why you are generating a second AST from your base-relation DSL, which is basically an AST itself
<postmodern>
ooooh
<snusnu>
fwiw, i appreciate your scepticism wrt the AST, it's just that a few things become simpler if we go down that route, at least that#s what we believe .. so e.g. re the DSL objects essentially being an AST, of course that's true in a way, but by splitting the responsibilities into more smaller pieces, we actually can fight the complexity
<postmodern>
seems like you just need some example DDL output from a real database
<snusnu>
sorry, dunno if that made any sense ..
<postmodern>
then define classes to store that generalized data
<postmodern>
then dump out the same data from the mappers, and diff them
<postmodern>
snusnu, adding more layers doesn't necessarily simplify things either
<snusnu>
not adding them neither
<postmodern>
snusnu, duck typing actually might be a better fit here
<postmodern>
snusnu, you should add simple methods for walking your AST
<snusnu>
postmodern: sorry, somehow i'm missing the context for that tdiff thing, are you talking about "real" migrations now, and how they could be done, sry, i don't get it
<postmodern>
snusnu, talking about auto-migrations, or more specifically auto_upgrades!
<postmodern>
snusnu, the need to compare the actual DB schema against the desired schema
<postmodern>
snusnu, i haven't done much with real-migrations, i did write some tsort code for sorting explicit migrations by their dependencies
<snusnu>
yeah ok, well that surely looks like it could work, but tbh, i dunno the exact plans dkubb has wrt to migrations as RA ops
<snusnu>
we're probably not yet really near the point where we could start working on such a thing, or rather, there seems to be other areas that need to be done earlier
<postmodern>
snusnu, man you guys need a road map badly :)
<snusnu>
full ack lol
<postmodern>
what's the status for write support?
<postmodern>
can it do inserts yet?
<postmodern>
sans unit-of-work, that's a whole other barrel of monkeys
<snusnu>
the memory adapter fully supports it, no sql support so far tho :/
<postmodern>
what's the blocker?
<snusnu>
yeah, we've decided to leave out UoW for now, it's a nice to have, not a need to have
<snusnu>
well, i dunno tbh
<snusnu>
re the blocker
<snusnu>
it's just that this is typically no area of the code i'm much involved with
<snusnu>
that's stuff that's mostly on dkubb's plate, or on mbjs
<snusnu>
mbj iirc has done insert support for arango db adapter already
<postmodern>
oh cool!
<snusnu>
re sql inserts, i guess you should talk to dkubb about his plans for this
<snusnu>
i'd surely love to see it soon :p
<postmodern>
trying to find where arango's write support is now
<snusnu>
can't find it either, maybe my memory tricked me
<postmodern>
would love to know what everyone has been working on over the last couple months
<postmodern>
last i remember, axiom had full read support on postgres
<postmodern>
and could successfully join tuples from an in-memory DB
kenphused has joined #rom-rb
<postmodern>
snusnu, also RE using ASTs, I'm more fond of the Emitter pattern
<postmodern>
snusnu, where you have some of tree of objects and an Emitter which walks the objects, emitting syntax for them
<postmodern>
snusnu, that way you keep your emitting/syntax logic separate from the objects that store the data
adkron has joined #rom-rb
<snusnu>
since the first rom release, a few things have happened …
<snusnu>
axiom got support for #wrap and #group, which basically means we now support embedded values (#wrap) and embedded collections (#group)
<snusnu>
i've rewritten rom-mapper to work on top of mbj/ducktrap and adds support for #wrap and #group
<snusnu>
using ducktrap, rom-mapper code is now a lot simpler, as ducktrap handles the transformation in both directions
<snusnu>
since axiom relations can now have EVs and ECs, and the mapper on top of ducktrap can handle those, we can now map arbitrary objects with n EV and ECs
<snusnu>
mbj is working on the next version of ducktrap called morpher, which features general awesomeness but he's probably the best guy to describe that to you
bf4 has joined #rom-rb
<snusnu>
i've started work on axiom-schema to decouple schema representation from rom-relation because really, it should be usable with axiom standalone, no rom involved
<snusnu>
i will continue to work on axiom-schema until it's able to build proper axiom (base) relations
<snusnu>
then i will rip that part out of rom-relation and integrate that with axiom-schema instead
<snusnu>
another change to the current rom-mapper master is that the mapper will not be responsible for renaming attributes in the future
<snusnu>
this sort of mapping/renaming will be handled by axiom-schema, which sill simply generate an axiom relation with a rename operation applied
<snusnu>
from the new rom-mapper's pov, it only ever sees *one* tuple, and knows how to map that into an object, and back
<snusnu>
once all that's integrated, we will be able to define arbitrarily joined relations, with a few #wrap and #group operations
<snusnu>
those will yield tuples that have their respective EVs and ECs wrapped and grouped
<snusnu>
initially, you will have to declare the respective mapper that is able to map these tuples yourself, but eventually, we will be able to generate mappers for all the relations you register with axiom-schema
bf4 has quit [Ping timeout: 252 seconds]
lfox has joined #rom-rb
<snusnu>
and there's quite a few missing pieces too …
misfo has joined #rom-rb
misfo has left #rom-rb [#rom-rb]
<snusnu>
the axiom sql adapter(s) need to support #insert #update and #delete
<snusnu>
axiom also needs support for server generated values (like keys) .. currently, you can basically only use UUIDs or the like
<snusnu>
axiom furthermore needs support for foreign key constraints
lfox has quit [Client Quit]
<snusnu>
on the rom side, we're currently completely missing a unit of work, but as i said before, that will probably come last
lfox has joined #rom-rb
<snusnu>
you might have also noticed that we basically have no support for the concept of relationships as one's used to, but that's only half of the story
<snusnu>
we have indeed decided that relationship (dsl support) should come last, but it might as well turn out that we won't implement it at all, simply because it's not needed
<snusnu>
instead we're going to focus on establishing proper FK constraints with axiom-schema, and see how far we can go with realworld apps, that define the relations (aka sql views) needed to work with object structures typically obtained by walking a chain of relationships
<snusnu>
as an example: Axiom::Schema.build { relation(:people_with_tasks) { people.join(tasks.rename(id: :task_id, person_id: :id, name: :task_name)).group(:tasks, [:task_id, :task_name]) } }
<snusnu>
this will basically yield tuples like: { id: 1, name: 'John', tasks: [{id: 1, task_name: 'doit'}]} .. which the mapper can digest
<snusnu>
to be exact, the above tuple should've been: { id: 1, name: 'John', tasks: [{task_id: 1, task_name: 'doit'}]}
<snusnu>
well no, not really .. anyways ...
<snusnu>
that's roughly what we've been up to and are planning to do ...
<snusnu>
postmodern: ^^^^
<dkubb>
snusnu: I'm going to create a gist with a preliminary road map, and once we get some consensus I'll create an issue under the rom-rb/rom project with all the tasks
<postmodern>
snusnu, ah ha
<dkubb>
I mean we all know the roadmap, but it's not accessible to people not doing the day to day dev
<snusnu>
dkubb: that'd be very awesome
<postmodern>
snusnu, btw the DSL in axiom-schema needs work to make it easier to interact with
<dkubb>
also, I could say at a high level "snusnu is working on the mapper" that doesn't really have any direct relationship to what you're doing
<dkubb>
I mean, right now you have the mapper that needs a schema, so you're dropping down to the schema level to get things pushed down from ROM
<postmodern>
dkubb, also are you sure you all have the same roadmap ;)
<dkubb>
postmodern: well, that's the thing too :)
<dkubb>
postmodern: I'm waiting for someone to say "wait, I'm not doing that!"
<dkubb>
postmodern: I'd love to get more outside help, but it has been a bit discouraging because I've put a lot of effort into getting a few people up to speed, and on a specific path, then nothing has come of it.. so it's been a waste of time for the most part, aside from supporting snusnu, mbj and solnic's efforts
<snusnu>
dkubb: (and i'm not at all happy with the name Attribute(::Name)
<dkubb>
postmodern: so what I'll do is a roadmap for snusnu, mbj and myself.. and then if others start to poke around and contribute I'll expand it, but I'm probably going to do it alongside any contributions, rather than do a ton of up-front preparation for people. I just don't have time to set up a runway for people and have them not follow through
<postmodern>
snusnu, i still don't grok why you are emitting s-exps with the exact same data in them
<dkubb>
postmodern: this is kind of the same thing with DM1. I've spent way more time supporting people in their efforts to contribute than I've ever gotten back in terms of effort (aside from a few core people).. so I'm a bit discouraged atm in expanding it too much
<postmodern>
dkubb, that would help a lot, since i don't understand what the goals are for each layer
<dkubb>
postmodern: I think snusnu just wants a temporary, easy to manipulate object, in which to build things up so that he can "finalize" it into some usable layer. not saying we couldn't do the same thing with full fledged objects, I guess maybe he finds it easier to reason about until the "real" objects are fully fleshed out (?)
<postmodern>
dkubb, what's wrong with the existing object? the s-exp tree and the object tree are basically the same representation
<dkubb>
postmodern: I'm not sure
<dkubb>
postmodern: my assumption was that eventually we'd collapse it and remove the s-exp, but I haven't voiced that yet ;)
<snusnu>
postmodern: the thing is, there are no DSL objects that could act like an ast, the DSL objects don't store state at all, they only expose the methods needed to setup the schema
<snusnu>
postmodern: i don't want to overload objects with doing this and doing that, i want small, simple objects that have one and only one purpose
<snusnu>
i realize it *could* be done with "ad hoc" dsl objects as ast, but i have the feeling that it'd end up in more complicate, tighter coupled code
<postmodern>
snusnu, i'm a bit confused here? your DSL objects definitely store the schema information
<postmodern>
snusnu, your DSL is basically an AST
<snusnu>
the DSL objects expose methods like #database #attribute #key etc .. they do not store any of those tho, they only have a ref to an ast builder
<snusnu>
because they're simply needed to *build* the *runtime* objects
<postmodern>
this seems needlessly complex
<dkubb>
one other option is DSL specific objects that accept user input, but has-a axiom primitives underneath
<snusnu>
it depends on how you look at it
<snusnu>
i don't see much complexity in fewer, simpler objects
<postmodern>
why not ignore the DSL entirely, and just create nested objects with the necessary data and methods to get the data?
<snusnu>
because for example then you run into all kinds of circular refs, tsorting, finalization, checking if something's valid or not, etc
<snusnu>
i want to collect the data i need to build the runtime system, check that, then build the runtime system
<snusnu>
not do everything scattered around everywhere
<postmodern>
snusnu, resolve references in a separate step
<snusnu>
why add future like objects etc, objects that are only needed for constructing the system, if an ast can hold all that info, and allows for an easy, standardized, documentable layout to process the stuff into whatever you like?
<snusnu>
e.g. axiom will surely have a ForeignKey object at some point
<snusnu>
but i can already tell you that this *will not be* the foreign key object needed to *create* one
<snusnu>
so you end up with 2 foreign key objects
<snusnu>
why
<snusnu>
i don't see it
<postmodern>
snusnu, i think this is a case of aesthetics
<snusnu>
for sure!
<postmodern>
snusnu, you are viewing the use of s-exps as reducing complexity
<snusnu>
yeah
<postmodern>
snusnu, when in actually you are still creating another representation
<snusnu>
i don't get that tbh, i have no other representation, the ast is the representation
<snusnu>
i have no other objects that store the representation
<postmodern>
snusnu, perhaps you should focus on finding a better representation to work with, instead of transforming the initial one
<postmodern>
man all these layers and transformations is making me not want to use ROM
<snusnu>
what can i say ....
<postmodern>
it's difficult to even discuss one individual layer
<snusnu>
how exactly?
<postmodern>
i don't even know what you mean by DSL Builder?
<snusnu>
there is no DSL builder afair
<snusnu>
there's an AST builder
<postmodern>
i don't understand why this is even a DSL instead of PORO that represent the schema
<snusnu>
i guess that's just different approaches to designing the system
<dkubb>
postmodern: have you audited the code under https://github.com/rom-rb/ ? I had a couple months off rom stuff for $work, and when I came back it only took about an hour or so to catch up on the code solnic and snusnu had written because the layers are relatively simple to understand
<postmodern>
dkubb, with the amount of layers and communication between them, there is bound to be issues
<postmodern>
dkubb, and not to mention performance hit from creating all these objects and destroying them
<snusnu>
postmodern: when you go down the immutable road pretty aggressive, you need to construct systems differently .. mutability only happens at very few places, mostly at the boundary of the system, when you collect the data needed to construct the immutable runtime objects
<snusnu>
lemme put it this way, i don't care much about performance, i want it to be correct
<postmodern>
snusnu, i don't understand how immutability applies here? rom-schema is just supposed to represent the schema?
<snusnu>
it can always be optimized
<postmodern>
snusnu, create an optimizer that walks the schemas and creates a new one?
<snusnu>
immutability applies in that the complete rom stack is built with immutable objects as much as possible
<postmodern>
snusnu, using an AST here just seems like an impudence mismatch
<snusnu>
again, i don't get it, you want dedicated objects for building up runtime objects, while i reuse a general purpose data structure, what's the big deal?
<postmodern>
snusnu, also isn't that kind of dogmatic, to take immutability to that far of an extreme?
<snusnu>
except that i tend to think that the ast leads to less code
<postmodern>
snusnu, sort of like the argument about how much mutation coverage we should shot for
<snusnu>
i don't think it is, no, because it's not for the sake of immutability, it's because i like systems built that way more, they're easier to reason about
<postmodern>
snusnu, you would only have one set of objects and an optimizer
<dkubb>
we do have an optimizer that works on relations already btw
<postmodern>
i just don't understand why we need to use an AST
<postmodern>
since it contains the exact same data as your schema objects
<postmodern>
in the exact same structure
<snusnu>
dude, the schema objects don't store state, they just don't
<snusnu>
heck, there *are* no schema objects
<snusnu>
there are objects, that expose api, that is needed to construct a schema
<postmodern>
snusnu, so what kind of state changes occur?
<snusnu>
instead of storing the collected data in ivars in dedicated objects, i store them in slots in the ast
<snusnu>
really, the only difference i can see, is that you'd like to store that state used to build the schema in ivars in objects you'd write, while i store them in an ast that already has been written for me
<postmodern>
snusnu, what does that give you?
<snusnu>
i don't have to write those classes
<postmodern>
in this context define state?
<snusnu>
that will never be needed in the running system
<snusnu>
that are only ever used when constructing the runtime
<snusnu>
that have ambiguous names with the actual real objects, that represent the same content during *runtime*
<snusnu>
those will surely expose api relevant to their *operation*, not their *construction*
<postmodern>
snusnu, so why does the DSL exist at all?
<postmodern>
snusnu, why not express everything in the AST directly?
<snusnu>
it's simply a layer to instantiate the objects we need at runtime
<snusnu>
because that wouldn't look nice
<snusnu>
;)
<postmodern>
snusnu, i say just write those classes
<snusnu>
for whose benefit exactly?
<dkubb>
actually, I think I can explain this approach
<snusnu>
mine, no
<postmodern>
snusnu, mine, when I'm debugging ROM :)
<snusnu>
heh
<snusnu>
thought so ;)
<postmodern>
snusnu, there's only a couple of them anyways
<dkubb>
from working with snusnu, solnic, mbj and others I've noticed two things.. snusnu and solnic like to do things top down. they like to start with the end interface and work down, refining things as they work towards the center
<dkubb>
there's nothing wrong with that. I think I and mbj like to work bottom up
<dkubb>
I think snusnu is using the ast as a stand-in for a complete object model while he works out how the interface for defining schemas should work
<dkubb>
because he is most interested in making sure the interface works well and makes sense first
<dkubb>
how I assumed things would work is that he'd get it the way he likes it, with some knowledge about how it works and then would refactoring it (or I/mbj would) so that some of the intermediate objects are removed, if they can be
<postmodern>
i guess i'll just have to continue waiting for ROM to settle down
<snusnu>
in fact dkubb, what i'm concerned with, is the *construction* phase of the runtime objects we need .. i don't want to mix those runtime objects with api needed for their construction, this sort of code has nothing to do over there
<dkubb>
I've come to realize everyone approaches things differently and as long as there's forward progress *and* the interfaces are stable, that's the main thing for me.. I don't mind doing passes over someone else's code, nor do I mind others doing passes over my code to make different facets work nice
<dkubb>
snusnu: I think it may be possible to support construction and runtime objects without signficantly complicating things
<dkubb>
snusnu: but I'm more interested in you getting something solid in place from an interface perspective
<dkubb>
snusnu: my general approach to DSL stuff is that I have builder objects that provide the DSL methods, and they has-a the immutable objects underneath. as the DSL is evaluated, the builders would replace the objects "inline" using the standard transformations available in the immutable objects
<dkubb>
so for example, the attribute :name, String call
<dkubb>
it would do: @header = @header.extend([:name, String])
<dkubb>
I'm simplifying of course, but the builder object would have immutable objects representing the current state of the system as a snapshot
<dkubb>
postmodern: is this more along the lines you were thinking about?
<postmodern>
well it depends on the type of DSL
<dkubb>
the problem right now is that axiom doesn't have *all* of the domain objects to represent what the DSL currently describes. it will though
<dkubb>
I already have keys, and I think it would be relatively easy to add foreign key data
<postmodern>
but yeah, basically you would start with a Context class, with methods that initialized other objects and stored them
<postmodern>
for DSLs with potentially lots of data, I use Structs or just plain Hashes
bf4 has joined #rom-rb
<postmodern>
benefit to this approach is everything can be programmatically initialized via the Structs, allowing you to bypass the DSL methods entirely
<snusnu>
fwiw, i still believe that using an ast will make building up the actual runtime objects much easier, checking validity will be easier, no future objects will have to be involved, and it will leave us with a documented format on which to build on top of
<postmodern>
also if you want to get fancy, you can override the Struct's #initialize method
<snusnu>
to me, it's about different phases, 1) collect data 2) validate/verify collected data 3) build target objects
<snusnu>
why would i want to start building my runtime system, before i even had the chance for checking its validity?
<postmodern>
snusnu, i tend to put verification in the #initialize method so validation happens immediately
<snusnu>
i tend to send validated objects into #initialize
<snusnu>
and have an external object do the validation
<postmodern>
snusnu, what kinds of validations are we talking about here?
<snusnu>
any, basically
<snusnu>
certainly not active_model ones only
<snusnu>
:p
<postmodern>
snusnu, validates_presence_of ?
<postmodern>
snusnu, we are still talking about the schema right?
<snusnu>
nah, really, any kind of checks, if i initalize a runtime object, i want my #initialize to be dumb, assigning ivars, nothing mroe
<postmodern>
i am confused again, what do you mean by runtime object?
<snusnu>
sorry, i mean an immutable object that is necessary for operation
bf4 has quit [Ping timeout: 272 seconds]
<postmodern>
snusnu, still confused, what is the goal here?
<snusnu>
heh, i was mostly referring to your mention of verification in #initialize method so that validation happens immediately
<postmodern>
all i understand is that we have a DSL for describing the schema in a generic way
<postmodern>
and that generates a big schema object with base relations and attributes etc
<postmodern>
what validation/verification has to happen there?
<snusnu>
right, and it completely suffices for that object to respond to #[]
<snusnu>
what else would it need, methods for building it?
<postmodern>
maybe a custom #initialize method if you wanted to specify defaults
<postmodern>
snusnu, the example DSL in the specs makes me think we could actually use Virtus for this
<postmodern>
snusnu, since it already has an Attribute class, which can accept additional options
<snusnu>
why would we depend on virtus for this simple piece?
<postmodern>
snusnu, well you are declaring attributes with names, types and options, seems very virtus (pun)
<snusnu>
(btw, i'm honestly enjoying this discussion, i hope i don't give any other impression)
<snusnu>
yeah, that's indeed similar, but imo virtus has a few loc too many to even get a slight chance here ;)
<postmodern>
snusnu, ok so option two, roll your own Attribute with Struct
<snusnu>
fwiw, i started off with dedicated objects like you suggested, but i was using anima .. turned out it needed way more code
<snusnu>
i dont need an attribute
<snusnu>
axiom has one already
* postmodern
looks up what anima is
<snusnu>
i just need to collect the data to create an axiom attribute
<snusnu>
postmodern: mbj/anima
<postmodern>
snusnu, what was the missing code that you had to write?
<postmodern>
snusnu, seems like there'd be just as much code to write if you used s-exps
<snusnu>
i would have to write objects that encapsulate all the data i collect for constructing the axiom objects, i would have to have objects that'd store collections of those, all those objects would exist with the same name, but different api in other parts of axiom/rom code .. and i would only write them, so that i could instantiate them, and after building the runtime axiom objects, they'd be thrown away
<snusnu>
so i figured, hey, there's a general purpose data structure that allows me to collect "typed" data, that's easy to transform/validate/process later on, it's an ast
<snusnu>
so i went ahead and used it
<snusnu>
saved me ~5-7 classes
<postmodern>
so what are the axiom objects really?
<snusnu>
Axiom::{Header, Attribute, Relation, Relation::Base} plus the currently non existent ForeignKey
<postmodern>
couldn't we create those directly from the DSL?
<snusnu>
yes we could
<snusnu>
would it be easier, i dunno, i doubt it
<snusnu>
would i find it better? no, i guess not
<snusnu>
it's a matter of taste after all
<postmodern>
just wondering if there is an intermediate step that has to happen between the DSL and Axiom
<postmodern>
if there's no intermediate step, I would just have the DSL create the Axiom objects
<postmodern>
i mean, the classes already exist and I assume represent the data you need?
<postmodern>
s/represent/encapsulate
<snusnu>
there's one issue at the minimum, and that#s declaration of foreign keys, if their referenced base relation hasn't yet been creatd
<postmodern>
ah ha
<snusnu>
and then there's regular relations (non base ones) . .they could depend on other such relations
<snusnu>
so they'd need to be tsorted somehow
<postmodern>
well you could require everything be defined top-down?
<snusnu>
if we do not create the runtime axiom object while we read the dsl, we don't get into situations where we've built up the system, only to find out that the last statement breaks
<snusnu>
we can read/validate the collected data, and bail out, or proceed with transformation/processing knowing that all will be good
<snusnu>
this allows for more offensive coding
<postmodern>
or you could create a proxy object for foreign keys, which attempt to resolve the relationship on method calls
<snusnu>
right, and i wanted to avoid that
<postmodern>
so you wanted an explicit validation step?
<snusnu>
yeah, i wanted all steps to be as simple as possible
<postmodern>
i think i see now
<postmodern>
so i would store the attributes using Axiom objects
<postmodern>
the ForeignKeys require some tricky logic to resolve/validate their relationships
<snusnu>
i realize that there's always a tradeoff with introducing yet another layer, i fully agree with you that it's no silver bullet, but complexity exposes itself differently, depending on the eye of the beholder, while more small, fine grained pieces may startle newcomers, once you're more familiar, you might be able to appreciate the simplicity *inside* the simple pieces more
<postmodern>
i would shot for a compromise between the two
<snusnu>
that'd be the ideal, yes
<postmodern>
obviously you shouldn't need to re-implement an Attribute class
<snusnu>
exactly, and i could avoid it
<postmodern>
also, can we redesign the DSL itself?
<snusnu>
sure, i'm open to suggestions
<postmodern>
we could design the DSL to be more declarative
<postmodern>
for instance you'd define the foreign-keys after all the base_relations?
<snusnu>
i'd love to see examples of what you have in mind
<snusnu>
currently? no .. do you want that?
<postmodern>
well it would mean that when a foreign-key is defined, your base_relation is guaranteed to already be in some Hash table
<postmodern>
it's a cheat, but avoids resolving the relations
<snusnu>
with using the ast and no futures, we don't have to mind this .. we just collect the ast, and when processing it, first make all base relations, then go add the fks . .while their declaration, is local to the relation they belong to
<postmodern>
snusnu, why can't that be done in the DSL?
<snusnu>
it could probably, but then the DSL would have another job to do .. whereas the ast processing can be done in concrete, dedicated steps, with one goal
lfox has quit [Quit: ZZZzzz…]
<postmodern>
snusnu, why can't those concrete steps be done after the DSL block has been evaled?
<postmodern>
snusnu, like you could store the foreign-keys in a simple Hash of sorts, then create the actual Axiom base relations from the populated attributes/foreign-key info
<snusnu>
i guess it depends on where you draw the line between a good amount of responsibilities for a class, and too much … i know, there are always infinite ways to reach a goal, the ones we choose, we choose largely based on preference
<postmodern>
your DSL could have two basic methods
<postmodern>
#initialize and #finalize
<postmodern>
#finalize would convert the collected data into Axiom objects
<postmodern>
if it can't resolve a foreign-key it would raise some exception
<snusnu>
that's more or less what i do tho, but it seems like i design objects differently, maybe a bit stricter wrt separation of concerns
<postmodern>
@relations could be a Hash of Arrays of Axiom::Attributes
<snusnu>
it will eventually be one
<postmodern>
and @foreign_keys would just be a Hash :fkey => [:relation, :attribute]
<postmodern>
i could probably write up some pseudo code for this
<postmodern>
snusnu, btw can base_relations have multiple databases or just one?
<snusnu>
base relations are basically tables, they must be in one specific database
<snusnu>
regular relations / virtual relations / views .. whatever you call them, (we favor just "relations") can span multiple databases
<postmodern>
snusnu, was wondering why you declare a database, then specify it again within the base_relation block
<postmodern>
snusnu, seems like you'd just want database() to accept a block
<snusnu>
the database is for specifiying the adapters to use, the database inside a base relation, ties that base relation to one adapter
<postmodern>
snusnu, or use an options Hash: base_relations(:users, database: :memory)
<snusnu>
think of it this way, a schema spans multiple databases, you tell it the adapters it will need, then you associate specific base relations to a specific, registered adapter
<postmodern>
ah ha
<postmodern>
snusnu, one way to simplify the DSL is to move as much stuff into options Hashes
<postmodern>
snusnu, reducing the number of DSLy methods you need
<snusnu>
i completely disagree, option hashes imo never simplify things
<snusnu>
but that's a matter of taste again
<postmodern>
snusnu, well if your only setting an ivar
<postmodern>
@ivar = options[:foo]
<postmodern>
def foo(x); @ivar = x; end
<snusnu>
oh, sorry, i guess i got you wrong, we could/should *of course* allow to declare multiple databases with one call to #database, accepting a hash
<postmodern>
snusnu, i was thinking of putting that into base_relation()
<postmodern>
snusnu, so only the body contains attribute info
<postmodern>
snusnu, or even allow #database to accept a block
<snusnu>
hmm, the reasoning behind stating the databases explicitly for the schema, is so that the builder can know that you're referring to an unknown symbol, for which no connection uri is present
<snusnu>
the connection uris are needed for setting up the adapters, no need to state them multiple times .. the databases are given a logical name you can then refer to
<snusnu>
i expect rom will have no such thing as DataMapper.setup
<snusnu>
you'd define the schema, and then maybe a ROM::Environment that gets the schema injected (altho i'm not yet sure if we'll even need such a thing)
<postmodern>
snusnu, only catch would be that if you specified a fkey to another db, you'd need to specify [:db, :relation, :attribute]
<snusnu>
we could do that i guess, and inside the schema block, we'd also expose #relation, and that would be a virtual relation, spanning multiple databases, or modelling a view inside a single one
<postmodern>
alternatively, you could do: relation(:users, database: :foo) do ... end
<snusnu>
good point re cross repo fks
<postmodern>
the fact that the axiom-schema example has database declared twice annoyed me
<snusnu>
yeah, there's definitely room for improvement in the dsl
<snusnu>
it was the first thing i came up with
<postmodern>
snusnu, so expanding on the idea of nesting data in blocks, instead of resolving it afterwards
<postmodern>
snusnu, what if we nested the relations?!
<postmodern>
snusnu, like a user has many posts, so why not define the post relationship within the user relationship
<postmodern>
snusnu, this way in the DSL, the user relationship would always get created first
<postmodern>
snusnu, that way the post relationship could specify foreign-keys to user
<postmodern>
hmm that might not work for m..n relations
<postmodern>
but it could for 1..n
<postmodern>
er never mind
<snusnu>
tbh, i want to postpone any thoughts on relationships for now …
<snusnu>
off the top of my head, one counter argument against a lot of nesting in the dsl, is that it'd need more different evaluation contexts
<postmodern>
i forget how foreign-keys must be specified for Axiom::Header
<snusnu>
oh, that's not set in stone, in fact, not implemented at all
<postmodern>
snusnu, true, but i usually just override #initialize to accept a block
<postmodern>
snusnu, ah ha!
<snusnu>
in fact, i don't expect that FK's will be given to relation headers
<snusnu>
they're database wide constraints
<postmodern>
snusnu, so why are we resolving the foreign-keys?
<snusnu>
they span multiple relations (2 to be exact)
<snusnu>
we're resolving them, because we need to make sure that we don't try to send DDL to the db that it can't handle
<snusnu>
if an FK gets declared, we should know wether the referenced attribute actually exists
<snusnu>
and here you can get into situations where you define an FK to a table, that's not yet defined in the DSL at this point
<postmodern>
snusnu, right
<snusnu>
so we either need futures, or break up the steps
<postmodern>
snusnu, or just force the user to define the tables in order ;)
<postmodern>
snusnu, it's mean but effective
<snusnu>
hah right, i started out with that, and i really thought for a *while*, why cannot we just do it like that
<snusnu>
then i guess i loved the challenge, heh
<postmodern>
snusnu, could you think of a case where you'd want to define an fkey, but not explicitly define it's destination?
<postmodern>
snusnu, like maybe the table/column already exists, but you don't tell the schema about it?
<snusnu>
if it were only me, i guess i'd be totally fine with having to declare the fks after everything else
<snusnu>
hmm
<snusnu>
it could be argued tho, that we probably don't *have to* support that
<snusnu>
like, if you expect the schema to work, tell it everything it needs to know
<snusnu>
maybe there could be a somewhat not-so-strict mode too
<snusnu>
but you definitely raise a good point here
<postmodern>
snusnu, also couldn't the DB just execute all the table creation DDL statements
<postmodern>
snusnu, then add the foreign-keys afterwards
<snusnu>
that's what it will probably do anyway
<snusnu>
once we get around to adding auto_migrate! like functionality
<postmodern>
so we might be able to get away without resolving things
<snusnu>
easier to generate
<snusnu>
hmm not really tho i guess, we still want to catch cases where the schema contains a logical error, catching cases where an unknown attribute is referenced
<postmodern>
snusnu, is the user defining this schema?
<postmodern>
snusnu, or is ROM sending us this information?
<snusnu>
yes, but it should also be reflectable from a database
bf4 has joined #rom-rb
<snusnu>
so, both ways
<postmodern>
ok seems like we have two competing problems
<postmodern>
a user could technically give us a fkey that points to a pre-existing table
<postmodern>
but the database would give us the complete picture
<postmodern>
brb pizza place is about to close
<snusnu>
yeah, in the end, only the database would know if there's a table that's not "mapped" into the schema
snusnu1 has joined #rom-rb
bf4 has quit [Ping timeout: 272 seconds]
snusnu has quit [Ping timeout: 264 seconds]
<snusnu1>
heh, my notebook crashed again, stupid battery is very close to die :p
snusnu1 has quit [Client Quit]
snusnu has joined #rom-rb
<snusnu>
postmodern: re an FK that points to a base relation not defined in the schema, we could probably add a #foreign_key! method that'd skip checks
<dkubb>
snusnu: postmodern: nice conversation I missed
<snusnu>
re nesting the relation definitions inside #database blocks, i actually like that, but it kinda implies that relation names must be unique within databases, which actually isn't really true in our case, we need schema wide unique relation names
<dkubb>
one argument against forcing the user to specify relations in order is that some relationships are circular in nature
<snusnu>
yeah
<dkubb>
I'm not talking about the case where you have parent_id that reference id
adkron has quit [Ping timeout: 245 seconds]
<dkubb>
you could have 3 tables: A -> B -> C, and then have C link back to A. at some point a FK is nullable, but this does happen in some cases
<dkubb>
I kind of don't like the idea of forcing people to specify FKs as a second step either. I mean, we could in a v1, but I would probably want to resolve the dependency
<dkubb>
it's not like it's a super difficult problem if approached properly
<snusnu>
i agree
<snusnu>
we're all set for that resolving step anyway, i fully expect to implement it
<dkubb>
yeah
<dkubb>
the tsort module will also blow up when there are true circular dependencies with no way to "break the chain"
<snusnu>
dkubb: re the point of having schema wide unique relation names, could/should axiom support some renaming of relations?
<snusnu>
i expect databases in the wild to have equally named relations
<dkubb>
so that's kind of a nice side effect we'll get.. it will help identify impossible schemas early
<dkubb>
snusnu: are you thinking of something like: base_relation(:people, name: 'customers') ?
<snusnu>
dkubb: exactly
<dkubb>
I would actually recommend we do something similar for attribute too
<dkubb>
rather than an explicit rename keyword
<dkubb>
it overloads attribute, but it might be nicer working with a schema where the primary names for things are what we plan on using in higher levels
<dkubb>
everytime I refer to this file I'll see there's a "zip" attribute, I might miss the fact that it's being renamed to zipcode
<dkubb>
it depends on what pov you're coming at the schema from
<dkubb>
if you're doing DDL generation, then maybe it makes more sense.. but if you're working at the object layer you probably care more about zipcode
<snusnu>
dkubb: heh, i guess you read it the wrong way? the way i imagine it, is that :zip is the name in object land, and :zipcode is the backend name
<dkubb>
oh I see
<dkubb>
hmm
<dkubb>
I always thoght it was like "rename the zip attribute to zipcode"
<snusnu>
i agree that #rename might not be the best api, but i was initially careful to overload attribute options (mainly because i dunno exactly how the "type mapping" will eventually be implemented, along with constraints and such)
<dkubb>
I think the type mapping and just use Axiom::Types.infer
<snusnu>
yeah, that's what i thought, so i was careful to mix options not intended for Axiom::Type into the mix
<snusnu>
also, i couldn't think of an option name that would make what#s happening any clearer
<snusnu>
dkubb: fwiw, re type mapping, i had the thought of "resolving" the type differently this time, not via const lookup madness
<dkubb>
once we figure out the right option name, we could use the same thing in base_relation()
<snusnu>
dkubb: what about some method_missing that looks up the type somewhere specific?
<snusnu>
dkubb: so we'd have attribute :id, integer
<dkubb>
snusnu: you could have a const_missing hook that delegates to Axiom::Types.infer
<snusnu>
if that works for all cases, i'm fine with it
<dkubb>
hmm, I'd rather probably stick with something closer resembling other code in the stack
<snusnu>
i just vaguely remember issues back in DM1, nothing concrete anymore tho
<snusnu>
dkubb: am i understanding it right when i think that axiom would need to implement base relation renaming?
<snusnu>
i.e., there's currently no support for it, right?
<dkubb>
snusnu: yeah, there's no support for that. I'm not sure if we would need to or if we would just resolve the name in the schema
<snusnu>
good point
<dkubb>
snusnu: the base relation always tracks the true name inside, the name given to base_relation() in the schema is the name the "registry" will use to map to the base relation. the base relation name and the registry name don't necessarily need to match
<snusnu>
of course heh
<dkubb>
one is the physical name and one is the logical name
<snusnu>
right
<snusnu>
great, then we could change the dsl to what postmodern suggested, and only provide #database and #relation inside the schema block, and additionally, provide #relation inside the #database block
<dkubb>
I'd be alright with nesting
<dkubb>
I was going to spike something super quick here for your review
<snusnu>
a #relation block in schema toplevel would build a virtual relation, and inside a #database block, it'd build a base relation
<dkubb>
it's not immutable objects in the DSL layer, but it's easy to audit and we can make it thread-safe relatively easily since the mutation is relatively explicit
<dkubb>
I realize this code reads like "my first dsl", but this is what I banged out off the top of my head ;)
bf4 has joined #rom-rb
<postmodern>
dkubb, you could make those methods private to restrict outside tampering
<dkubb>
yeah
<postmodern>
dkubb, ah i forgot you extend an empty Axiom::Relation
<postmodern>
i see now
<postmodern>
what definition of immutable do you all bide by?
<dkubb>
yeah, so I basically use a "table dum" which is a dumb name, but it's common in relational model docs.. it basically means an empty relation with no tuples
<dkubb>
mutation has to happen at some level, I just try to minimize it and make it explicit when it does happen
bf4 has quit [Ping timeout: 245 seconds]
<dkubb>
I try to keep it out of the bulk of the system
<dkubb>
in many cases, if I can return a new immutable object that represents the new state after an operation, I sometimes will do it. it just depends on the case
<dkubb>
in both examples, mine and snusnu, mutation is still happening. in one case you're mutating an AST and in my case you're mutating either a registy of databases, a registry of relations, or a placeholder that represents the current state of the relation given the DSL operations evaluated at a specific pointin time
<dkubb>
I just wanted to provide mine as an example of another approach, one that uses the existing axiom objects as part of the process, rather than just the end product
<snusnu>
dkubb: yeah, that's basically the way i did it in my first spike too, what you don't get with that tho, is an IR that can then be used to be transformed into all sorts of things, say 1) the actual axiom objects, 2) a DDL version of it … you'd have to either add additional objects, or additional api to those objects
<snusnu>
also, the whole simplicity starts to break down more and more once you add the rest of the features, like a finalization step, eventually, the code grows, and the responsibilities (the way i "predict" it) will get more and more mixed
<snusnu>
i'm not saying it's not doable or not feasible, at all
<snusnu>
i'm just thinking that decoupling the IR from the actual code, will give us benefits later on, with more options of "isolated" processors
<snusnu>
but really, i should be sleeping for hours already, even for my rythm .. :p
<dkubb>
no worries
<snusnu>
one last thing, i generally like the idea of checking my input *before* i start trusting it .. that doesn't mean i need an AST datastructure, i could build one using objects too, but starting to build the target objects right when receiving the first sentence of description of the whole system, seems weird
<dkubb>
the axiom objects do all sorts of checking on their own
<dkubb>
to make sure they are internally valid. probably logic we wouldn't want to duplicate in schema
<snusnu>
yeah, but they're (at least currently) concerned with relations mostly, not with stuff spanning multiple relations
<dkubb>
in many cases, if we identify something in schema that needs to be validated I'll need to push it down into axiom itself anyway.. like the "no duplicate pk" constraint we identified today
<snusnu>
i wouldn't do any validation checks of "relation internal" stuff
<dkubb>
yeah, stuff spanning schemas won't be validated in axiom relations
<snusnu>
i'd leave that to axiom
<dkubb>
in many cases we'll identify these constraints at higher levels
<dkubb>
I'm find with adding the checks at the higher level as long as we push down as much as we can
<dkubb>
it's only natural that we'll discover more things at the front-facing parts of the code
<dkubb>
constraints should be pushed down to as low a layer as possible
<snusnu>
fully agreed
<snusnu>
and i guess i'm really out now ;)
<dkubb>
once stuff is pushed down, we can remove the higher level checks
<dkubb>
ok, ttyl!
<snusnu>
thx for a nice discussion postmodern and dkubb!
snusnu has quit [Quit: Leaving.]
<postmodern>
good night, hope it helped
<dkubb>
I'm wondering if these "wrapper" objects using axiom primitives for storing the raw data might be a decent approach
<dkubb>
obviously they're going to do other things, like having dependency resolution performed for FKs, but at the core they're just manipulating primitives that already have much of the behaviour
<postmodern>
yeah it's a DSL for building the internal representation from the external one
<dkubb>
I need to break out the DSL book and see if there are common names for this approach. it's been so long since I read it