<solnic>
mbj: dude, those mutations are killed with —rspec-unit
<solnic>
so, no, those are not uncovered ones, it’s just that with —rspec mutant’s doesn’t work the same as with —rspec-unit
<solnic>
mbj: I’d expect that I would get 100% with —rspec because I get 100% with —rspec-unit as simple as that
<solnic>
if I don’t, I consider this to be a bug in the new system
<mbj>
solnic: I found the problem :D
<mbj>
Old mutant master simply was blind at some spots.
<mbj>
Yeah those mutations had been killed from --rspec-unit but NOT from your spec.
<mbj>
You wrote a spec for Foo::Bar#baz
<mbj>
but this spec did not killed all mutations in Foo::Bar#baz
<mbj>
Some other spec did this for you
<mbj>
got my point? So instead of adding more complexity, just kill all mutations from narrow example groups
<mbj>
it is easy fo the ones I saw.
<mbj>
Or dont write narrow specs at all.
<solnic>
mbj: this misses the whole point
<solnic>
mbj: this is very confusing, you said it would expand
<solnic>
it does not do that
<solnic>
the spec which kills mutation in ROM::Relation.build is ROM::Mapping spec
<solnic>
so it seems like mutant is not expanding specs
<solnic>
you said it should expand from Foo::Bar#baz to Foo::Bar* to Foo* eventually
<solnic>
so, it doesn’t do that
<solnic>
I think it stops at Foo::Bar* and doesn’t run all specs for Foo*
<solnic>
I know this un-killed mutation is easy to kill but that’s not the point man :)
<mbj>
solnic: it only expands if you dont have a dedicated example group
<mbj>
it expands till it finds example groups that matches
<solnic>
WHY?
<solnic>
that’s so bad man
<solnic>
that’s not flexible at all, this still adds the constraint that you must kill all mutations via tests for a specific method
<solnic>
I *want* to have separate specs per method because it’s easier to work with small spec files
<solnic>
sooooo I’m very confused right now :)
<mbj>
solnic: if you write a spec for a specific method, that spec is reponsible for killing all mutations in that method, right?
<solnic>
no
<mbj>
why?
<solnic>
because no
<solnic>
;)
<mbj>
It does not make sense all other specs need to be executed allso till the mutation is covered.
<solnic>
because it’s a much bigger effort to kill all mutations through dedicated spec files
<mbj>
For A) speed, and B) spec is your most exact documenation if you miss a point here you are lost.
<solnic>
it results in much more specs than it’s actually needed
<mbj>
nope
<solnic>
yes
<solnic>
I just had to add one
<solnic>
dude
<solnic>
look at my commits
<mbj>
forgot to push?
<solnic>
mbj: "its(:mapper) { should be(mapper) }"
<solnic>
I had to add this
<solnic>
otherwise I wouldn’t have to
<solnic>
in this case it’s quite OK
<mbj>
because of implicit coverage
<mbj>
The problem, implicit coverage is weak
<mbj>
and slows mutant down, a lot.
<solnic>
because this is actually part of the public API
<mbj>
that is my point
<mbj>
if you specify parts of the public api, this specs must cover all mutation of that specific public api.
<solnic>
but I can see cases where it would force me to write more specs just to kill something
<solnic>
I don’t consider implicit coverage to be a bad thing
<mbj>
Wich ones??
<mbj>
If you dont write a spec for a public method, you get implicit coverage.
<mbj>
Lets say we define def foo? !!@foo; end
<mbj>
and you dont write a spec for this, this one will get implictly coverage.
<mbj>
So we have the best from both worlds, no need to add an extra spec for simple stuff
<solnic>
mbj: lemme see what other mutations are not killed
<solnic>
mbj: yeah so, good example - Relation#each
<solnic>
now I’m forced to write stupid detailed specs for each method separately
<solnic>
even though its mutations are killed by other tests
<solnic>
imho this kind of implicit coverage is totally fin
<solnic>
fine*
<mbj>
solnic: mom lemme run
<mbj>
solnic: I see your point, what about to expand if mutation is not killed, but report this fact?
<solnic>
mbj: what about expand and say nothing? :)
<mbj>
mutant is a tool to verify your specs "do the right ting"
<mbj>
s/ting/thing/
<mbj>
if you specified Relation#each BUT it does not kill all mutations, I'd like to know htis.
<solnic>
well, I don’t care, but if it is valuable to you then sure, why not have mutant telling you this
<solnic>
in general I don’t like when tools forcing opinions in such a strong way, so if mutant can be flexible then it should be, people will come to their own conclusions wrt mutation testing anyway
<mbj>
solnic: You ruin execution time this way.
<mbj>
And I'd like to know this.
<mbj>
So I'm okay to expand, but report.
<solnic>
doesn’t matter
<solnic>
yeah sure, you can report
<solnic>
but I think we should find other ways to improve perf
<solnic>
like parallel execution
<mbj>
yeah
<solnic>
tons of extra brittle tests is way worse than mutant running few minutes longer imho
<mbj>
But I see this also form a "specification" point of view. If you write a spec that does not cover the method and rely on another spec doing this for you, you miss a significant portion of explicitlness.
<solnic>
dude
<solnic>
unit testing is about testing objects in an isolation
<solnic>
not methods
<solnic>
testing methods in isolation is crazy
<solnic>
and yes I know you and dkubb think it’s great, I think it leads to test suites that are hard to maintain and slows down development
<solnic>
just look at the #each case in rom-relation
<solnic>
now mutant forces me to write specs that check if each works, well, of course it works because it is used by #to_a which is used in shit loads of other specs
<solnic>
if #each was broken all those specs would fail
<solnic>
if I were to kill those mutations in each_spec.rb then I’d write REDUNDANT tests
<solnic>
redundant tests are evil
<solnic>
that’s an extreme anti-pattern and I’m really surprised you and dkubb didn’t see this and don’t consider this to be a problem
<solnic>
lemme remind you I have spec-file-per-method because eventually I realized smaller files are easier to work with even if this means *more* files
<solnic>
I don’t have spec-file-per-method because I want to fully cover those methods via individual files because that’s crazy IMHO
<mbj>
solnic: I'm already changing it, no need to convince me now :D
<solnic>
I also agree there are cases where this could mean incidental, implicit coverage which is not good, like in Relation#mapper :)
<solnic>
so, after all, we are both right to some extent :D
<solnic>
mbj: ok :)
<mbj>
I accept we have different opinion here, as long as be both can use the tool I'm happy.
<mbj>
You have to accept reports, I have to accept reports :D
<mbj>
For you these reports say: I'd did my job
<solnic>
fine by me
<mbj>
For me these reports say: I have still to do my job.
<solnic>
well, I will try to explain this at eurucamp :)
<mbj>
BUT, mutant will exit 0 in case of implicit coverage.
<mbj>
solnic: I already understood this.
<solnic>
you still don’t agree :)
<mbj>
We both have a differend opinions about the granularity of "unit" :D
<solnic>
it boils down to this: I test my objects in isolation, you test individual *methods* in isolation
<solnic>
and I think the latter is *crazzzzyyyy*
<mbj>
I plan to write my own unit test framework
<solnic>
but that’s ok, your code your rules, if you are fine maintaining such enormous test suites that’s not really my problem ;)
<mbj>
That will target got be the "best mutation killer ever"
<solnic>
damn, I need to talk about this with dkubb one day
<mbj>
lets fly him to europe :D
<solnic>
mbj: actually, what about minitest?
<solnic>
I’ve header it’s very small and hackable
<solnic>
header? wtf auto-correct
<solnic>
heard*
<mbj>
solnic: Lets say it this way, I know heckle internals. And minitest is from the same facet of community.
<mbj>
And heckle internals suck hard.
<mbj>
This is no prove minitest sucks, but this reduced my willingness to look into it.
<mbj>
Also I did a spike integration once, and I was very unhappy with it.
<solnic>
oh ok
<solnic>
too bad
<solnic>
well, the world needs another test framework I suppose
<solnic>
I think we could build a really solid one given our…interest in testing techniques :)
<solnic>
ok I’m gonna shut up finally and let you do the work ;)
<mbj>
lulz
<mbj>
My test framework should allow to combine all those spec/unut/foo/{class_methods/,}*_spec.rb into a single file
<mbj>
while still being ideomatic and focusing in invariants.
<mbj>
Also it should reduce terminal duplication to a minimum.
<mbj>
Lots of rom-style "defaults". If you do "describe Foo::Bar, '#baz'" you get some defaults for free
<mbj>
For example that framework should setup object for you (with zero arguments to constructor)
<mbj>
or if you give object_arguments { [foo, bar] } with arguments, etc.
<mbj>
I had a hack for rspec once, where these defautls get inferred from the file name, remeber?
<mbj>
I plan this + even more consistency
<solnic>
dude count me in, I’d love to help
<solnic>
I mean rom is my priority right now but after sql adapter is done I want to do other stuff too
<mbj>
solnic: I see my work as strong support, just imagine how uneasy our live would have been with heckle :D
<mbj>
crashing all the time, inflexible like hell and incorrect :D
<mbj>
And forcing us into 1.8 :D
<mbj>
moving to office
<mbj>
back in some minutes
<mbj>
solnic: I'm at 3.8k opensource commits, maybe I crack 4k till eurucamp
<mbj>
But more important, I STILL DONT HAVE A TALK :D
<mbj>
busy weekend :(
<solnic>
mbj: oops, you gotta start working on it :)
<solnic>
…and we gotta start working on the workshop
<mbj>
hehe
<mbj>
solnic: So lets skip slides and go directly to the Q&A
<mbj>
:D
mbj has quit [Quit: leaving]
postmodern has quit [Quit: Leaving]
jfredett has joined #rom-rb
snusnu has joined #rom-rb
snusnu has quit [Quit: Leaving.]
cored has joined #rom-rb
_whitelogger has joined #rom-rb
jfredett has joined #rom-rb
knowtheory has joined #rom-rb
snusnu has joined #rom-rb
mbj has joined #rom-rb
<mbj>
hola
knowtheory has quit [Quit: Computer has gone to sleep]
<snusnu>
mbj: yo
<snusnu>
mbj: i have a mutant question
<snusnu>
mbj: well, actually, maybe not ..
<snusnu>
mbj: i was about to ask if it's necessary that mutant reads all examples prior to executing a very specific one
<snusnu>
mbj: it shows that the examples work fine, but i cannot mutate them, because mutant chokes about other failing specs
<snusnu>
mbj: note how the error mutant raises, is due to a spec *outside* of the scope that mutant is supposed to mutate
<snusnu>
mbj: this pretty much means that as long as any specs fail to load, i cannot mutate anything (even if the thing i want to mutate, loads and runs fine=
<snusnu>
cored: hey
dkubb has joined #rom-rb
<cored>
snusnu: what's up
<cored>
dkubb: hi
<dkubb>
cored: hello
<dkubb>
cored: how's it going?
<mbj>
snusnu: use the -r flag, I added it again
<mbj>
snusnu: Most likely I forgot to require spec_helper.rb
<mbj>
snusnu: mutant -r ./spec/spec_helper.rb
<cored>
dkubb: fine, you?
<snusnu>
mbj: be mutant --rspec -r ./spec/spec_helper.rb ::Substation::DSL::Registry#[]= gives me the same output
<snusnu>
dkubb: re your question about yardstick, i had an idea yesterday (not yardstick specific), lemme try to recall it
<mbj>
dkubb: +1 for squezing out backports
<dkubb>
snusnu: I'm going to try to get the config working better today
<cored>
I'm trying to get devtools working on a Rails project
<cored>
getting a TIMEOUT error from rubygems trying to install rake vesion 1.10.x
<snusnu>
so what if we add enabled: false flags to all devtools configs, and then have something inside devtools, that makes it so that when git branch is not master, certain configs are disabled by default .. (maybe based on another config setting) .. even if we don't switch behavior based on git branch, it seems like having the enabled: false flag in devtools config files, seems better than removing the configs altogether
<dkubb>
yeah, I think whatever we do we should have less-strict settings in feature branches, stricter settings in master, and the strictest settings when we do a gem release
<snusnu>
dkubb, mbj: i grepped my source, and i *never* do include Adamantium, i *always* do include Adamantium::Flat
<snusnu>
dkubb: yeah, full ack
<mbj>
snusnu: equalizer
<mbj>
snusnu: equalizer deep frezes the generated module
<mbj>
snusnu: this is a wild guess :D
<snusnu>
mbj: oh noes … now that you say it, i remember it does, but, why?
<snusnu>
mbj: oh ok, it deep freezes the generated module, so what?
<dkubb>
equalizer deep freezes its state and the state provided to it
<snusnu>
aha
<dkubb>
but it doesn't freeze whatever it's mixed into
<snusnu>
dkubb: why does it deep_freeze stuff?
<mbj>
snusnu: I said, only a wild guess
<snusnu>
dkubb: that's too invasive?
<mbj>
snusnu: just place a p caller here /Users/snusnu/.gem/ruby/1.9.3/gems/adamantium-0.0.11/lib/adamantium/class_methods.rb:17 and wait for the crash.
<mbj>
snusnu: nope, sorry a p self :D
<dkubb>
snusnu: deep freezing by default is a safe default. you shouldn't really be passing in state into another object if that state is mutable.. if it is, then you can modify internals of another object breaking encapsulation
<mbj>
snusnu: you aready staring a t a stack trace :D
<mbj>
dkubb: Let us exclude all kind_of?(::Class) from freezing.
<snusnu>
dkubb: ok, but that means that i can *never* safely equalize on anything outside of my control, is that desired?
<dkubb>
snusnu: what an initializer should do is make sure that it doesn't just use the provided object as-is
<dkubb>
mbj: I can't do that without breaking axiom-types
<dkubb>
well, not breaking it, but changing it to be mutable
<snusnu>
dkubb: in my particular case, i inject classes (i could as well inject instances, but that's not actually necessary, i only rely on #call)
<dkubb>
I suppose I could just freeze by hand
<snusnu>
dkubb: so duping the class won't help
<mbj>
dkubb: You remember the days when I said axiom-types should NOT be real instancs, not classes :D
<dkubb>
:P
<cored>
in fact, which is the fastest way to get devtools working on a Rails project?
<dkubb>
mbj: yeah, although the inheritance did result in some nice benefits. I see inheritance-like behaviour in some ROM stuff and I wonder if it could be done more elegantly leveraging ruby's built-in behaviour
<mbj>
dkubb: you can also do this via modules.
<mbj>
dkubb: With classes you get an "unusable" interface .new
<mbj>
dkubb: But dont let discuss this now :D
<dkubb>
cored: I don't recall doing anything special. I just added it to my Gemfile
<mbj>
dkubb: it is okay for me.
<mbj>
snusnu: Can I reproduce this crash easily?
<cored>
dkubb: trying it
<mbj>
I debugged deep freeze triggerer very often.
<dkubb>
mbj: assuming we turn off class freezing, a deep-freeze related bug is almost always a sign of a bug in the code
<mbj>
dkubb: not always
<dkubb>
mbj: I saw "almost always" :P
<dkubb>
*said
<dkubb>
mbj: if you pass in a mutable object into another object's constructor, you can manipulate the new object's internal state by modifying the object you passed in
<snusnu>
one side effect of Adamantium{::Flat} is that we cannot stub methods on infected objects anymore, now that might be a good thing in general, but it certainly affects the design of the system plus tests (i have to admit, i haven't yet encountered a case where the necessary refactoring made the code worse tho)
<dkubb>
yeah, that's definately an annoying thing
<mbj>
dkubb: hehe
<dkubb>
although I don't do that so often. I find myself not stubbing "real" objects very often
<snusnu>
dkubb: yeah, that's what i meant with refactorings never leading to worse code
<dkubb>
yeah
<snusnu>
dkubb: i found myself having to inject stuff into #initialize, so that i can inject a double
<dkubb>
I see some of our techniques as gentle nudges in nicer directions.. usually anyway. I will admit that once in a while it does lead to worse code
<dkubb>
but that's where we always have to question it and make sure we think about it and get some review when we're not sure
<cored>
weird
<cored>
I got an error before which is not here now, I just remove the entire gemset
<cored>
anyways, can I run devtools against just one file?
<snusnu>
mbj, dkubb: fwiw, here's the output i get when i add puts(self.inspect) as first line in Adamantium::ClassMethods#new : http://pastie.org/pastes/8208735/text
<mbj>
snusnu: OMG, I'm so stupid
<mbj>
snusnu: nope, I'm not.
<snusnu>
lol don't say that, it's wrong
<snusnu>
heh
<mbj>
snusnu: just undershugared
<snusnu>
the first statement i.e
<mbj>
lulz
<snusnu>
:p
<cored>
now the ci :-)
<solnic>
mbj: not without asking you a ton of questions probably ;P
<dkubb>
cored: you'll probably have to do it a few times and tweak the config files each time
<snusnu>
but yeah, the config object containing the observers gets equalized on those
<snusnu>
mbj: ^^
<cored>
dkubb: flog is disable for 1.9.3, why?
<solnic>
mbj: what’s “next wider selector”?
<snusnu>
mbj: the issue only came up, once i made all processors observable .. previously, as you know, there was Substation::Action which was the only observable thing .. now every processor is observable, hence the observers moved to Processor::Config, where they get equalized .. that must be the problem
<snusnu>
cored: it's score incompatible with 2.0.0
<dkubb>
cored: most likely when you see stuff disabled for one version and not another, its because they return incompatible scores with other versions
<snusnu>
dkubb: which btw makes me think, do we really want to follow 2.0 scores for now? 1.9.3 is our primary dev env?
<cored>
oh and now I notice something code climate reports stuffs on a controller but I can't see any offences now with devtools on that particular controler or on any
<dkubb>
snusnu: is it your primary dev env? I always dev in 2.0 now
<snusnu>
dkubb: well yeah, i develop on 1.9.3 currently
<dkubb>
snusnu: 2.0 is actually stable now, so I figured it was a better choice than 1.9.3
<solnic>
ugh me too
<dkubb>
hmm
<solnic>
also, mutant is faster on 1.9.3
<solnic>
like, significantly faster
<dkubb>
should we all downgrade to an older version of ruby, or should we upgrade to the latest stable?
<dkubb>
that's really weird
<dkubb>
I should test that
<cored>
I'm first planning to go with the code smells duplication and code complexity which was what code climate report, and then go with mutant
<snusnu>
tbh, i don't care much, but i can already see us developing against 2.0, using something not supported in 1.9.3 .. thus getting broken builds when we were sure that we'll be good after push
<snusnu>
and then it's backports all over again
<snusnu>
;)
<dkubb>
heh
<dkubb>
the differences between 2.0 and 1.9.3 aren't very wide
<dkubb>
which is good
<dkubb>
not like supporting 1.8.7 and 2.0
<snusnu>
yeah, i imagine so
<dkubb>
mbj: have you seen speed differences between 1.9.3 and 2.0?
<snusnu>
dkubb: my reasoning was more like, ok, realistically, 1.9.3 will be the most common ruby rom apps will be deployed to .. that's just a guess too, but given the slow adoption of 1.9, it seems like a reasonable guess
<dkubb>
mbj: in mutant I mean
<cored>
I think probably there are others problems like undefined method `source_line' for #<Parser::Source::Map:0x00000004fa2860>
<dkubb>
snusnu: yeah, that's a good argument. maybe we're better off targetting 1.9 and then it'll just work in 2.0
<cored>
and invalid option: --rpec-dm2
<solnic>
cored: change to —rspec
<dkubb>
cored: you can remove that from the config. I will do it in devtools now
<snusnu>
dkubb: imo we *should* fix build problems in 2.0 immediately, but it's ok to dev for 1.9 for the time being .. we certainly don't want to use ruby-2.0 only features
<solnic>
mbj: ping ;)
<cored>
dkubb: ok
<cored>
solnic: which particular config file?
<cored>
solnic: can't find that option
<solnic>
cored: config/mutant.yml just remove strategy since devtools defaults to —rspec
<cored>
solnic: ok
<dkubb>
snusnu: ok, good point
<cored>
also the YourLib part should be remove
<cored>
I'm guessing
<dkubb>
is everyone ok if I change flog to run under 1.9 only for now? it may break some scores
<dkubb>
cored: what version of ruby are you using?
<cored>
1.9.3
<dkubb>
cored: as a test you could try changing max_statements to 0. if this doesn't cause some kind of error then we can safely assume reek isn't running
<cored>
ok
<cored>
switched same thing
<cored>
so, not running
<dkubb>
ok, I've got one slightly invasive test
<dkubb>
do bundle show devtools to find where it is installed
<dkubb>
then look in tasks/metrics/reek.rake to find the --quiet switch in the reek config. remove it and then run the rake task again
<cored>
ok
<solnic>
geez this new strategy gives a significant speed boost
<dkubb>
cored: oh I think you may have found a bug in our reek task
<snusnu>
cored: i'm happy to describe the desired functionality again, just ping me once you wanna start working on it
<cored>
dkubb: same thing
<dkubb>
cored: it looks like by default it only scans the lib directory
<cored>
snusnu: sure, thanks
<cored>
dkubb: oh
<dkubb>
cored: lemme see about changing it to scan "lib" and "app"
<solnic>
at least in rom-relation
<solnic>
so, new strategy is basically 3 x faster for me :)
<snusnu>
yeah
<solnic>
sucks
<snusnu>
solnic: interesting, iirc mbj saw roughly the same speedup .. on my box, for substation, it "only" dropped from ~280 secs to ~240 secs
<solnic>
hmm really?
<solnic>
lemme push so you can test ok?
<solnic>
snusnu: I just tweaked it a bit
_whitelogger_ has joined #rom-rb
<snusnu>
solnic: i thought mutant was already working like you described in your commit msg
<solnic>
snusnu: no it didn't
<solnic>
that’s why I was so surprised and a little bit confused
<snusnu>
so, what's different now?
<solnic>
previously, before my commit, it would stop at the first exact match
<snusnu>
ah ok
<solnic>
so, if you have a dedicated spec group for a method
<solnic>
it would only run those examples
<solnic>
w/o expanding the scope at all
<snusnu>
i get it now
<solnic>
yeah
<snusnu>
i assume it stops tho, once it killed all mutations at any scope?
cored has quit [Ping timeout: 256 seconds]
<solnic>
snusnu: yes it starts with the exact match and go on expanding if the first one didn’t kill the mutation
<snusnu>
right
<snusnu>
soundsgood
cored has joined #rom-rb
cored has joined #rom-rb
<cored>
c
<cored>
found some smells but it's also throw an exception in the end
<snusnu>
solnic: fwiw, i find the first "widening" reasonable, going from Foo::Bar#baz to Foo::Bar .. what's with Foo tho?
<solnic>
snusnu: it runs all unit tests since it’s the root namespace
<solnic>
basically ending up with what —rspec-unit used to be
<snusnu>
solnic: ok, make the nesting 4 or 5 levels deep
<solnic>
snusnu: ah I see what you did there
<dkubb>
solnic: I asked mbj about widening it even further to use ''
<snusnu>
solnic: it seems to me that it could just as well go from Bar#baz to bar, and then do the equivalent to --rspec-unit
<dkubb>
solnic: which allows the edge cases where you use core_ext extension
<dkubb>
even though that's icky
<solnic>
snusnu: yeah I will work a bit more on this thing
<snusnu>
solnic: eventually tho, i'd like it to be more intelligent even .. say i have an (abstrac) base class, i want to be able to kill all mutations from the concrete subclasses i have, i don't want to write extra tests where i basically make subject { Class.new(SomeAbstractClass) }
<solnic>
actually…no, it doesn’t duplicate anything
<snusnu>
solnic: same goes for modules
<solnic>
since it matches example group descriptions
<cored>
why devtools stills have flay on it
<cored>
if reek reports duplication problems?
<solnic>
and then gets unique example groups only
<dkubb>
cored: none of the metrics have perfect overlap. and the duplication that reek reports about is only within one single method so it's not really comparable
jfredett has joined #rom-rb
<snusnu>
solnic: my issue is not so much with duplication, but with potentially running mutant on a "meaningless" namespace … by meaningless, i mean that there's probably no semantic relationship between Foo::Bar::Baz and Foo::bar
<cored>
dkubb: got it
<snusnu>
solnic: ah ok, so it doesn't do anything anyway, because it finds no examples for the "intermediate" steps?
<cored>
dkubb: btw, have same issue as reek with metrics:flay
<snusnu>
solnic: in any case, you could probably call my problem related to duplication ;) of running tests
<cored>
dkubb: probably same thing of not looking at app
<snusnu>
solnic: what i'd love to have is this: if it sees a class/module which has subclasses/hosts, it should try to kill all mutations via those, and if it can, shouldn't require to have any tests for the baseclass/module
<snusnu>
dkubb: that is inline with your thoughts on that topic too, right?
<snusnu>
if it finds out that it cannot kill mutations via subclasses/hosts, then it's ok for it to complain .. you should then either write new specs for the uncovered baseclass method, or ask yourself why you introduced the (obviously unused) method in the first place
<snusnu>
and i suspect there are valid usecases for such "currently unused" methods
<snusnu>
especially in a system that's designed to be extensible, say, via some sort of template method
<snusnu>
even then, most likely the code you have, should already exercise those tho
<solnic>
heh from 670 seconds down to 250 seconds on travis :)
<solnic>
dkubb, snusnu: ^^
<snusnu>
nice
theCrab has joined #rom-rb
<dkubb>
snusnu: I'm not sure. This may get kind of abstract, but I think it depends on how you view inheritance. are you inheriting the implementation or interface? if it's the former then you can assume both implementations are the same, and testing one tests the other. if it's the latter then you have to assume both classes are black boxes and test them as if the other doesn't exist; specs can be shared of course
<dkubb>
snusnu: in most things I fall on the side of thinking about the interfaces more than anything, but I haven't thought much about this specific case. I suspect I am leaning in the same direction though
postmodern has joined #rom-rb
<snusnu>
dkubb: i agree, in any case, if mutant *finds* subclasses/hosts, it can try to kill via those, if it finds dedicated specs for the baseclass/module, it should of course not complain either .. in any case, it's a decision the developer has to make
mbj has joined #rom-rb
<snusnu>
dkubb: but imo mutant shouldn't blindly force you to write potentially superfluous tests
<mbj>
hi
<mbj>
dkubb: you need support, regarding the "first match must kill" :D ?
<snusnu>
completely offtopic .. does anyone know the *reason* why ruby devs decided to *not* have Boolean but TrueClass and FalseClass instead?
<solnic>
snusnu: because
<dkubb>
I would bet it was due to performance considerations
<dkubb>
maybe they need to test on the type a lot or something
<dkubb>
people have tried to get ruby-core to add #to_bool type methods
<dkubb>
and the reason for not wanting it was the same, perf
<dkubb>
I would love to have a method that is called in boolean context
<solnic>
so we ended up writing a ton of libs that provide to_boolean :D
<dkubb>
it would make lots of things so nice.. like proxy objects
<solnic>
haha rbx finished mutant task in 23 minutes (vs > 50 minutes that was terminated by travis)
<solnic>
mbj: ^
<dkubb>
mbj: with your recent changes, does it make paralellization easier or harder or about the same?
<mbj>
dkubb: the same
<mbj>
solnic: nice!
<snusnu>
btw, how would the gist linked by mbj impact performance?
<mbj>
I think we can soon run mutant in self test mode, it was terminated for the same timeout before.
<mbj>
I heard extending or including modules in primitives creates an "extra" lookup table for regular dispatches.
<mbj>
So there is a penalty for all primitive operations on the same type.
<mbj>
BUT the community accepts active_support, so we should not even talk about this.
jfredett has quit [Quit: Leaving.]
<solnic>
mbj: I’m reeeeeealy surprised you prefer testing methods in isolation, that’s crazy, an object is not a collection of methods that work independently, it’s a whole thing and should be treated like that IMHO /cc dkubb
<mbj>
And if mri would add a few special cases to the C implementation of Object#kind_of, I thin the difference would only be measureable in microbenchmarks.
<snusnu>
solnic: the thing is the public api, if some method is dedicated public api, it should have a spec, so that with internal refactorings, the public api doesn't break
<snusnu>
mbj: or rather, your specs tell you it does
<mbj>
Its all about the interface, I like to make 100% clear what my interface does, via specs.
<solnic>
but dude, all tests for a given object make sure if it’s not broken
<snusnu>
ehm, solnic^^
<snusnu>
but if you have to change a spec for a public api method, you have a chance to examine it, and potentially bump version and rerelease
<dkubb>
I like testing my interfaces, not the implementation. I don't like relying on a side effect that method A uses method B to verify that I've tested method B's interface
<snusnu>
solnic: i can't exactly put my finger on it, but the stuff you're doing, is really close to incidental coverage .. saying that without judging in any way
<mbj>
solnic: We'll have LOTS of fun arguing about this with beer :D
<snusnu>
hah, +1
<solnic>
eventually I will be screaming and jumping in agony
<solnic>
but that’s ok
<snusnu>
and fun for us
<snusnu>
:p
<solnic>
well, I think it’s taking arguments with incidental coverage way too far
<solnic>
but that’s OK
<solnic>
maybe I’ll change my mind, maybe you will
<snusnu>
exactly
<mbj>
Its fun to have differend opinons. Talking with people that agree all the time is boring
<solnic>
I think it is a good idea to have mutant warn you about specs for other methods/objects that kill something
<solnic>
forgot to mention that
<mbj>
solnic: So we are 99.9% in sync
<solnic>
I do agree in most of the cases a dedicated spec should kill mutations
<mbj>
solnic: I'd nuke that warings, you not.
<snusnu>
solnic: i do agree that it all depends on what you view as the black box, with oop, it's definitely fair enough to view the object as the black box, but from a stable public api pov, it's just equally fair enough, to treat the supported messages as blackboxes as well
<solnic>
as I described in the commit msg
<solnic>
snusnu: see, I dunno, I’m not sure :)
<solnic>
need to digest that
<snusnu>
fair enough again
<solnic>
my point is basically that you should have only one test that check something
<solnic>
if you have two or more tests checking THE SAME THING
<solnic>
then imho you’re doing it wrong
<mbj>
solnic: Imagine a spec framework that lets you test THE SAME THING without duplication?
<solnic>
wat? :)
<solnic>
dooh I just realized I’m hungry, bbiab
<snusnu>
take a simple example tho, you decided to make a method public api, people rely on it, now you changed your code which changed your public api method, you adjust your specs, and since there's no *dedicated* spec for that one public api method, you might overlook that breaking change .. which is more unlikely imo, if there were a dedicated spec in the first place
<mbj>
solnic: you know my plans to develop a spec framework dedicated to kill mutations :D
<mbj>
snusnu: +1
<dkubb>
snusnu: +1
<mbj>
snusnu: A VERY good point!
<mbj>
snusnu: The one I was looking for so long!
<snusnu>
hah, well i'm glad then ;)
<dkubb>
just because one method uses another method doesn't mean it will always be the case. if the only way an interface is tested is via a side effect, then it's not really tested imho
<snusnu>
yeah, i agree
<solnic>
snusnu: no I won’t because mutant will tell me about that
<snusnu>
how?
<snusnu>
from mutant's pov, all mutations are killed
<snusnu>
because you already adjusted your specs
<dkubb>
mutant probably can't tell you that. if method A uses method B, and you change B's interface and update it's usage in A mutant won't know there'sa change
<solnic>
you don’t understand my point (because I probably don’t explain this correctly)
<dkubb>
client code using method B will break though
<solnic>
I have specs for all my public methods
<solnic>
but
<mbj>
solnic: The example group that does your incidential coverage just might be another after the refactoring
<solnic>
I do not have all mutations killed via dedicated spec-per-method approach
<solnic>
see my point?
<mbj>
solnic: this is only true for public interfaces
<snusnu>
solnic: i know you have them, and then all is good .. as long as you kill all mutations via dedicated specs for public api method, all is well
<solnic>
mostly because in some cases I’d have to write specs that are redundant
<mbj>
solnic: for private interfaces you are *free*
<snusnu>
yeah
<mbj>
solnic: I'm sorry to say this, but I'm thinking about to revert that change you did, or make it available via a configuration flag.
<dkubb>
I don't think anyone is saying private interfaces need to have direct tests.. as far as I'm concerned those are implementation details and I try not to test implementation details
<mbj>
snisnus argument is very good.
<mbj>
*snusnus
<snusnu>
lools weird
<snusnu>
:p
<snusnu>
solnic "doesn't get it" because he's already doing it
<snusnu>
;)
<snusnu>
if a project has a dedicated spec for every public method tagged @api public .. all is good
<dkubb>
I need to get on that mutant-yard plugin that can scope tests to just @api public methods
<snusnu>
yeah, that'd be cool
<dkubb>
I would also want to exclude methods and classes that have @private
<snusnu>
yeah, that'd be cool too
<snusnu>
mutant needs to know what humans consider public api, otherwise it has no chance
<solnic>
mbj: you’re joking right? with that reverting?
<snusnu>
since not every public method is public api, it needs hints, period
<mbj>
solnic: refactoring to a configuration option
<mbj>
solnic: think about snusnus argument
<mbj>
solnic: Mutant is a tool to verify specs describe the public interface very exactly
<mbj>
solnic: So if I allow incidential coverage I lower the maximum value of the tool
<solnic>
this is ridiculous but whatever
<mbj>
solnic: It is okay for not using the maximum power
<mbj>
solnic: But I dont want to lower the maxmimum power a tool has.
<mbj>
solnic: Just *thinking* load.
<mbj>
s/load/loud/
<dkubb>
mbj: I think I agree with you. metrics defaults should be strict with the ability to loosen them. if you go the other way, people are lazy and will never "raise the bar" on their own
<snusnu>
fwiw, i tend to agree with mbj, if you the human say: i want to test this public api method with this exact spec, mutant should not allow you to slip through and rely on incidental coverage
<snusnu>
you can easily not rely on the most exact matcher in this case
<snusnu>
solnic: don't get me wrong, but your approach relies on being smart all the time, we all consider ourselves smart, but we all know that we actually aren't really smart, all the time
<snusnu>
;)
<snusnu>
tools shine at proving that we were doing something "wrong" out of habit and/or laziness
<mbj>
solnic: What about an --rspec-all, with warnings
<mbj>
solnic: I'd use that for myself with libraries under development
<dkubb>
ok this time that failure wasn't my fault! :P
<mbj>
Before I push 1.0 I'd switch to --rspec and try to reduce spec duplication via shared specs.
<dkubb>
flog now runs under 1.9.3 only
<solnic>
dkubb: I already updated flog threshold but in a separate commit
<solnic>
it’s green
<solnic>
snusnu: yes my approach totally relies on being smart and knowing what you’re doing
<solnic>
not being smart is not very smart ;P
<snusnu>
heh
<snusnu>
but having something remind you if for some weird/crazy reason you're actually not being smart atm, is a good thing too, right?
<snusnu>
:p
<solnic>
mbj: I think —rspec should do what it does and warn you
<solnic>
snusnu: hey I said I want warnings right?
<snusnu>
solnic: warnings for what? that you changed the behavior of a public api method? how would mutant do that?
<mbj>
solnic: this is "soft"-ware, can be changed later, deal: I accept --rspec-all for now and we see how well it goes?
<mbj>
Or we can go --rspec with the expansion and --rspec-strict being less permissive.
<mbj>
but I wanna have a mode that reduces coverage
<snusnu>
solnic: if it's a warning that says: dude, you have specs that test public api, but not in a dedicated spec, than yes, it could do that (given some yard integration)
<snusnu>
solnic: otoh, i'm not entirely sure how it would do that, but i guess it could, given it tracks which specs kill what subjects
<mbj>
snusnu: I think he is talking about warnings when example scope is widened to kill a spec.
<snusnu>
mbj: i can already see me ignore those, not being smart while doing so
<snusnu>
:p
<snusnu>
then again, i hate warnings, most likely i would not ignore them after all
<dkubb>
warnings are interesting things
<dkubb>
if you aren't overwhelmed, and a new one pops up many people will attempt to fix them
<mbj>
I'm okay with two strategies
<mbj>
and see how they compeate
<dkubb>
but I think more people will ignore them to just work on a feature or are under pressure to ship
<snusnu>
yeah, it's becoming problematic once there are so many, that your brain just sees a wall of ugly text
<mbj>
We had three inflexible ones before
<dkubb>
and once they've gotten used to them they will ignore them
<snusnu>
yeah
<mbj>
BTW someone knews how warnings got activated in mutant?
<mbj>
stupid question maybe
<dkubb>
<-- this guy
<mbj>
dkubb: not who, HOW :D
<mbj>
dkubb: point me to the commit :D
<dkubb>
oh, hmm
<dkubb>
I added --warnings to a few of my gems' .rspec files
<dkubb>
and a few rom related gems too
<snusnu>
imo a future mutant should be totally strict, and require a dedicated method spec killing all mutations, for any @api public method it encounters .. it should *not* care about other public/protected/private methods
<dkubb>
it's possible rspec sets $VERBOSE based on that
<dkubb>
snusnu: I think mbj didn't want a hard dep on YARD
<snusnu>
dkubb: it should be realized via some sort of plugin
<dkubb>
snusnu: I would be ok with this as a mutant plugin, at least until we get all the kinks worked out
<dkubb>
yeah, if we end up using it 95% of the time, then we can talk about including it in mutant itself
<mbj>
dkubb: A hard dep on yard would be okay
<dkubb>
oh really?
<mbj>
dkubb: But I dont want to force the user to use YARD.
<mbj>
dep != use
<dkubb>
oh yeah
<snusnu>
yard is the defacto ruby doc anyway, every ruby lib using rdoc, is well, i'm not gonna say it ...
<mbj>
so I'm okay with developing this feature in tree and split later.
<dkubb>
I think in the absence of @api tags it would just do what it does now
<mbj>
I'd love to see yard using parser
<snusnu>
that said, i realize that there are *a lot* of gems that don't use yard but rdoc :p
<dkubb>
if rdoc was not bundled with ruby I don't think it would be used as much as yard
<snusnu>
bbiab
snusnu has quit [Quit: Leaving.]
<dkubb>
mbj: parser can parse comments now can't it? or at least track them
<mbj>
dkubb: It allows to get comments per associated node, yeah
<mbj>
so comments are NOT part of the ast
<dkubb>
yeah, I think I fixed some bugs in rubocop around that
<mbj>
but you can query an object per node to get the comments
<dkubb>
there's an associator object I think
<mbj>
Yeah, IMHO it was a good decision NOT to make the comments part of the ast
<mbj>
but I plan to reproduce comments in unparser
<mbj>
+ some other optimations to honor 80chars unparser could make a nice prettifier
<dkubb>
that would be sweet
<dkubb>
yeah, run unparsed code through rubocop for verification
<mbj>
heh, yes
<dkubb>
which we talked about before I think
<mbj>
I'd love rubocop coulr replace reek
<mbj>
reek is outdated and broken and inflexible
<dkubb>
rubocop has a good community and lots of momentum
<solnic>
+9001
<mbj>
dkubb: OT, do you deploy your jruby installations with oracle or openjdk?
<mbj>
s/dkubb: //
<dkubb>
mbj: I haven't deployed jruby in production recently
<mbj>
I had some crashes with both
<mbj>
My own installations still work sane
<dkubb>
mbj: I would probably just start with openjdk on principle ;)
<mbj>
heh
<dkubb>
but I'd still use oracle as a last resort
<mbj>
same here
<mbj>
but this time I'm bound to oracle (current client)
<mbj>
my own stuff happily deploys via openjdk
<solnic>
you’d be surprised by how many people actually dislike yard
snusnu has joined #rom-rb
<dkubb>
oh yeah, I'm sure there are. probably not more than the number that dislike rdoc though.. but that comes with the territory of being the default implementation
<dkubb>
based on what I've seen in the wild ruby developers dislike writing documentation in general :P
<mbj>
dkubb: heh
<mbj>
The only reason I wrote mutant and keep zombie broken: The last place where it is allowed to cowboy code :D
<mbj>
Nope tbh, its just lots of work doing unparser and mutant :D
<dkubb>
yeah, mutant represents a ton of work
<solnic>
mbj: are you adding second strategy or just adding warnings to —rspec?
<mbj>
solnic: still no consensus
<mbj>
dkubb: BTW you mentioned the fact I'm talking to the PIT maintaner
<mbj>
dkubb: It gave me some ideas already, especially how to measure kill times better
<solnic>
mbj: I think the less strategies the better because new users wouldn’t have to learn about the 2 strategies
<mbj>
solnic: TBH, I'm confused now
<mbj>
wont decide this today
<mbj>
BTW once tihs is decided, we can IMHO release 0.3.0
<mbj>
non --pre
<solnic>
mbj: I believe you can just add one line
<solnic>
which spits a warning when a mutation wasn’t killed after the first group run
<mbj>
I'd love to default to strict rules
<mbj>
with the ability to lower the strictness to personal preference
<mbj>
but tool default should be optimum
<mbj>
I like compilers that dont emit warnings
<mbj>
just accept or error
<mbj>
no grey areas
<solnic>
well so maybe a configuration flag
<solnic>
—strict
<solnic>
or something
<solnic>
—strict and —no-strict
<mbj>
with --strict as default
<mbj>
but I think two strategies would be easier to support
<mbj>
two subclasses of Rspec
<mbj>
*rspec killer
<solnic>
not sure, it maybe discourage people because nobody is writing specs like that
<solnic>
well except you and almost me
<solnic>
;P
<dkubb>
imho you should do something special to be less strict
<mbj>
nobody will write method specs
<solnic>
ok let’s start with strict
<mbj>
sure
<solnic>
and see what people will say
<mbj>
and this is the point
<mbj>
if people do describe Foo::Bar { my_specs_here }
<mbj>
they will be happy in strict mode
<solnic>
that’s what people do in 99% cases
<solnic>
heck
<solnic>
a lot of libs don’t even have unit tests
<mbj>
so they can use strict mode
<solnic>
mostly integration ones
<dkubb>
if you start off less strict it is really hard to raise it up later
jfredett has joined #rom-rb
<mbj>
see --rspec-dm2 envored Namespace#method
<mbj>
s/envored/enforced/
<mbj>
and --rspec does not
<solnic>
well, true otoh many people will try to use mutant with existing libs/projects
<mbj>
so they'll be happy ootb
<solnic>
so strict by default might scare them off ;)
<mbj>
solnic: nope
<solnic>
oh wait
<mbj>
only if they use method level granularity
<solnic>
rite :)
<mbj>
and most people will do describe Namespace { here the dragons }
<mbj>
so strict default is perfect, IMHO
<solnic>
snusnu: wdyt about 2 modes for mutant —strict and —no-strict?
<mbj>
--strict and --no-strict are just the same as --rspec-weak and --rspec
<mbj>
s/weak/something-more-positive/
<solnic>
yeah but later on you can expand it for other runners where it would mean the same thing
<solnic>
that’s why having 2 strategies wouldn’t be smart
<solnic>
so I vote for this extra conf option
<mbj>
good point
<dkubb>
internally it could still use two strategy classes
<mbj>
the question, is this premature optimization of the future, we dont know of other spec frameworks will/can work in strict/no-strict modes.
<mbj>
double dispatch
<mbj>
on two "signals"
<mbj>
dontlike, but hey we have consensus
<dkubb>
let's think about minitest
<mbj>
provide two strategies/flags whatever, use strict by default
<dkubb>
that's the most requested one atm
<dkubb>
I would expect the same kind of spec layout differences as with rspec
<mbj>
I think we should start just like with new axiom adapters
<mbj>
first read-all-tuples/run-all-specs
<mbj>
than feedback cycle it :D
<dkubb>
I'm fine with doing the simplest thing that could work, based on the current knowledge we possess. I don't have a super strong opinion on specific switches, beyond my preference that the default should be the strictest possible setting, you should have to do something extra to loosen constraints
<dkubb>
it goes without saying, but there shouldn't be any extra "stricter" mode
<mbj>
I do not plan to use minitest in one of my projects
<mbj>
So I'll not push this forward now
<mbj>
Apart from the fact I could use a stable minitest integration for my rails story :D
<dkubb>
no, I think it approaches it in a different way
<dkubb>
it's more about reporting the state
<mbj>
mutcov
<mbj>
dkubb: BTW, I talked to snusnu once about mutcov, I could host some central "report your metrics here" service
<mbj>
so we share metrics across developer machines
<mbj>
much better than "violatil" storage on developer devbox
<solnic>
yeah mutcov site would be awesome :)
<solnic>
would love to help building it if I could only find time
<mbj>
solnic: I'd start with mutcov.org
<mbj>
and if there are enough requests for private repos add a mutcov.com
<solnic>
dammit I clicked
<mbj>
but only lateeeeeeeeerrrr
<mbj>
solnic: this is snusnus job to click on all links :D
<solnic>
snusnu: ^^ go
* dkubb
is scared to click now
<solnic>
haha it’s not live that’s the problem :)
<dkubb>
I've been burned too many times by friends posting innocuous looking links in chat and then clicking on them
<dkubb>
you cannot unsee some things
jfredett has quit [Quit: Leaving.]
<mbj>
dkubb: curl, always browse with curl before :D
<dkubb>
heh
<mbj>
going to sleep, cu tomorrow
mbj has quit [Quit: leaving]
<dbussink>
dkubb: no set_trace_func or TracePoint atm btw on rbx
<dbussink>
we might add support for the latter
<dbussink>
but i'd rather develop better tools :)
<dkubb>
ahh I see, better libraries than those
<dkubb>
dbussink: mbj and I were discussing using those within mutant a bit on twitter today
<dbussink>
what would you use it for then?
<dkubb>
dbussink: what mutant would do is run the specs once and trace which lines are being executed, then relate that to each spec. from then on, whenever a line is mutated mutant would know precisely which specs to execute
<dkubb>
dbussink: right now we kind of don't really know the association, beyond using naming conventions, so we kind of have to do a scattershot approach in some cases
<dbussink>
i think that would be easier with more fine graining coverage support
<dbussink>
fine grained
<dbussink>
enable coverage, run spec, store coverage
<dbussink>
reset coverage and go to next spec
<dkubb>
dbussink: oh you're talking about using the Coverage lib?
<dbussink>
afaik even the coverage api can already do that
<dkubb>
that's better than the set_trace_func stuff
<dkubb>
I find that api confusing
<dbussink>
yeah
<dbussink>
anyway, nite for me
mbj has joined #rom-rb
<snusnu>
solnic, mbj, dkubb: i'm fine with —no-strict and —strict, the latter being the default
<mbj>
snusnu: nice
<snusnu>
but tbh, i don't see the value in —no-strict .. but i'm not opposed either
<mbj>
I'm okay with providing less strict modes for libs in development
<snusnu>
also, i do not care *that* much about how other people use rspec .. if they're inclined to do mutation testing, they might just be inclined to do it full on
<snusnu>
mbj: imo that doesn't help
<snusnu>
as dkubb said, starting out non strict, will make it much harder to go strict eventually
<mbj>
snusnu: I'll start strict
<mbj>
But only mention --non-strict for cases where developer chooses to loose some requirement
<mbj>
like we all adjust rubocop/reek/flay/flog
<snusnu>
i just don't cloud mutant's sharp focus, for usecases where people write shitty specs … do you really expect those to do mutation testing?
<snusnu>
just because mutant exists and is cool?
<snusnu>
i doubt it
<mbj>
we all need a --cowboy and a --james-cameron mode :D - dkub
<mbj>
*dkubb
<snusnu>
well, nobody forces us to run mutant on our code
<snusnu>
:p
<snusnu>
so much for cowboy code area
<mbj>
heh
<mbj>
finally hitting bed, cu
mbj has quit [Quit: leaving]
jfredett has joined #rom-rb
<solnic>
snusnu: what about methods like Foo.build? if I use this method in all of the tests for Foo then what’s the point in having a separate test for this method?
<solnic>
esp that this method…builds an object, so the only way to have full coverage of this method is to either inspect object’s *state* or…use it methods in tests to see if it behaves like expect which is…redundant since you already have tests for that
<snusnu>
solnic: i'm not that opposed against testing state, i usually have equalized objects, and i don't mind an it { should eql(expected) }
<solnic>
that’s a smell IMHO
<solnic>
unless the state is public
<snusnu>
solnic: so tell me exactly why, leaving no blurry stuff
<snusnu>
solnic: nope it's not imo
<solnic>
well, I test behaviour, not state
<snusnu>
for equality tests, protected visibility is enough, and that's good as it is
<snusnu>
well, why
<snusnu>
always
<snusnu>
as a hard rule
<snusnu>
i don't see the point
<solnic>
because that’s what counts
<snusnu>
you have tests for behaviro, why do it again
<solnic>
geez exactly
<snusnu>
just because state checking is bad, i don't buy it
<solnic>
it is terribly bad
<snusnu>
when used wrongly
<solnic>
an object can have some state that allows it to do its job and that state might change over the course of development even though the behaviour might not change
<snusnu>
i followed too many "don't do this it's generally bad" .. there are *always* exceptions to the rules, that's what makes them rules
<snusnu>
right, and you have tests for that
<solnic>
yes, the exception here is when the state is public
<solnic>
yes, so why the hell would I write a special test that check object state
<snusnu>
to make sure the interface didn't break
<solnic>
dude
<snusnu>
i.e., it is an additional hint
<solnic>
if the interface was broken
<snusnu>
yeah yeah, you'd change your other specs
<solnic>
most or even all of the tests would fail for that object
<snusnu>
;)
<snusnu>
know what? let's not reduce to a single "special" method, .build
<snusnu>
;)
<solnic>
well
<solnic>
I have 3 cases like that, #each and 2 builder methods
<solnic>
both are not fully mut-covered through their dedicated specs
<solnic>
and I think it’s totally fine
<solnic>
because if I wrote those specs I would introduce redundancy in my test suite
<solnic>
because I would have to check behaviour that’s tested in other places
<solnic>
like the fact that #each actually works
<solnic>
or that builder method create valid objects
<snusnu>
but you'd nail down public api behavior, so that once you change it, you don't need to rely on being smart, to find out
<solnic>
so you think growing your test suite with redundant specs is better?
<solnic>
not to mention that IF I broke anything either my tests or mutant would tell me about it?
<solnic>
I need to see some clear examples
<snusnu>
it's just a feeling, but i don't think that redundancy would be *that* much in a properly designed system, with mutant really only checking @api public methods
<solnic>
because I’m giving you examples and you’re giving me some nice sounding theories
<snusnu>
also, i don't mind a bit of redundancy in specs .. tbh, while writing it, yeah, i somehow hate duplicate setup .. but when browsing specs after some time, i hate shared contexts
<snusnu>
dude, i totally see how you can cover it all, with potentially doing no harm with a little bit of "incidental" coverage … i'm just not sure if for long lived projects with sharp, nailed down public api, the little bit of extra overhead really is the problem
<solnic>
snusnu: well, I dunno
<solnic>
how many projects have you written like that, maintained for some time, extended, refactored etc?
<snusnu>
time will tell probably, which somehow implies that we take both approaches to be able to compare them :p
<snusnu>
^^
<snusnu>
btw, i should add that i'm not going to curse you if you (we) decide that it's fine for special methods in rom, to be covered "incidentally"
<snusnu>
i'm only thinking hard about the issue :)
<snusnu>
if we find it to be a "rule" that methods needing that special treatment are all somewhat alike, then that's fine
<solnic>
well, I don’t think extreme approaches are good, and this feels extreme to me
<snusnu>
one could say that mutation testing in general is extreme
<snusnu>
but that doesn't add anything to the discussion ;)
<solnic>
no I think heckle was extreme
<solnic>
mutant is totally cool
<snusnu>
heh
<solnic>
I find it mostly very easy to kill mutations and I immediately see the positive impact it has on my code
<snusnu>
absolutely
<snusnu>
same here
<solnic>
I couldn’t say the same about heckle
<snusnu>
tbh, i haven't used it extensively enough to be able to compare it
<solnic>
the problem with heckle was that every single thing had to have specs
<solnic>
or at least that was our setup
<solnic>
bah
<solnic>
it’s really great with mutant now
<snusnu>
dkubb's heckle grew out of axiom, and axiom is somewhat special imo … the kind of api it exposes, isn't quite like other stuff
<solnic>
no it’s not
<solnic>
actually, once I stopped speccing every-single-class-in-the-lib it started feeling really good
<snusnu>
also, using the strictest form of an (otherwise comparably "weak") mutation tester seemed (and still seems) feasible .. axiom is hardcore foundational stuff
<solnic>
like what we discussed today isn’t a huge issue for me
<solnic>
rom-relation had 6 un-killed mutations
<solnic>
from over 8000
<solnic>
blah
<solnic>
800
<snusnu>
right, i was surprised for a moment ;)
<snusnu>
rom-relation is much smaller than substation, and substation has ~1400 mutations currently
<snusnu>
axiom somewhere near 5000 iirc
<solnic>
yeah rom-relation is tiny
<snusnu>
and that's awesome
<snusnu>
re not writing specs for every single class in the lib … i also can't wait for the @api public and @private integration with mutant
<snusnu>
that will have a huge impact on the amount of methods we'll have to spec
<snusnu>
again, depending on the project domain
<snusnu>
obviously
<solnic>
snusnu: hmm? but new —rspec already solves that
<solnic>
if there’s no spec file it’ll expand
<solnic>
so, no need to write tests for stuff that’s @private or @api private
<snusnu>
well, what can i say, i've mentioned quite a few times, that i want it to force me to write a spec for every @api public method … and yeah, @private support isn't all that useful, as those classes probably shouldn't contain @api public methods :p
<snusnu>
also, it'll be interesting to see what mbj comes up with, once he writes the rspec replacement he's talking about .. maybe that, plus something in mutant can work around the issue with the special methods you encounter
<snusnu>
because i *do see* the point you make, it *is* duplication .. i'm just not entirely sure if that duplication is harmful to *me*
<snusnu>
in any case, i know that mbj made mutant happy with symlinking specs in some cases … a new testing framework targeted for mutant might be able to bake this in without falling back to symlinks
<snusnu>
these are just wild thoughts tho
<solnic>
I have a bunch of internal private objects
<solnic>
those are a subject to change w/o changing the behaviour of my lib
<solnic>
the are used by other objects that actually expose public interfaces
<solnic>
those objects are the ones I’m covering with tests
<solnic>
so far so good
<snusnu>
ack
<solnic>
if I were to spec the internal ones I’d have to write tests for public ones and mock the internals as dependencies
<solnic>
which I do not want to do :P
<snusnu>
me neither
<solnic>
cool
<snusnu>
really, our *only* difference is in our thoughts on mutant forcing a dedicated spec for a specific public api method to kill all mutation
<snusnu>
apart from that, we're in complete sync i guess
<solnic>
yeah lemme put it this way
<snusnu>
what would be cool, is if mutant printed out superfluous specs … i have tons of them in substation, because i did —rspec-dm2 style … otoh, i don't need it *now* … because i haven't yet nailed down the public api :)
<snusnu>
with new mutant, my first step is to try and see what happens if i just nuke an offending spec, when refactoring :) only given it most likely is private api of course
<solnic>
I see lots of improvements in our dev style, it’s changing all the time and we’re clearly going in the right direction
<snusnu>
yeah, i see it the same way
<solnic>
once ROM is released and axiom is battle-tested in the wild we’ll see what worked well and what didn't
<snusnu>
yeah, everything else is speculation, but yeah, we can say one thing i guess .. most of the code should behave *as specified* .. which of course doesn't mean it doesn't contain bugs, might well be that we specified "wrong" behavior, but yeah, that's a different class of bug
<solnic>
as you pointed out - axiom is a very special library based on solid theoretical foundation, so I totally get the enormous effort to have it tested in 10000%
<solnic>
it’s different with ROM though, at least IMHO, I’d prefer to get the alpha out even if docs aren’t 1000% and there are 6 un-killed mutations via dedicated specs and get to know what people think about it
<snusnu>
the other tools we use, should help us with moving us in a direction, where fixing bugs becomes easier
<snusnu>
full ack
<solnic>
anyhow, it’s gonna happen in a little over a week yay :D
<snusnu>
even tho 6 unkilled mutations sounds arbitrary, i guess we'd just kill those prior to release :p
<snusnu>
yay!
<solnic>
well, those are killed in non-strict mode ;P
<snusnu>
lulz
<snusnu>
so, they are not actually unkilled
<snusnu>
see, i get you ;)
<snusnu>
and really, it's totally fine at this stage .. it's NOT like we nailed down the public api, 1.0style already
<snusnu>
but i probably wouldn't want to let that number grow and grow over time .. it'll be hard to switch to strict mode for a future James Cameron release
<snusnu>
but we can only speculate about what our toolchain will be able to do at this point in time ...
jfredett has quit [Quit: Leaving.]
<solnic>
I think I cover most of the stuff by default in strict-mode
<dkubb>
the "it" blocks are called examples in rspec
<solnic>
well those read nicely
<snusnu>
i'm almost certain that no one will ever complain about not being able to read rom spec output in nice english
<snusnu>
just a guess of course ;)
<solnic>
I guess you’re right
<dkubb>
I was more curious about what it was
<dkubb>
I mean, I do try to write descriptions that make sense
<dkubb>
but I don't measure it or audit it regularly
<snusnu>
it's different with end user projects maybe
<snusnu>
but with library code ...
<snusnu>
imo a lot of the info already is in the context description
<snusnu>
anyway
<snusnu>
i see, ah, that context, ok, green? cool
<dkubb>
I might just try it in a local branch over time
<dkubb>
I do try to write spec examples that make sense both in the source and in the specdoc output. if looking over a report once a month to see what the output looks like helps me become better at that I may just do it
<snusnu>
ok guys, i hear alarm bells ringing because we need consistency … can i keep on *testing*, or do i have to come up with boilerplate english text in the future?
<snusnu>
because i *will* write that text if it's necessary, but i don't want to write boilerplate crap
<snusnu>
;)
<dkubb>
snusnu: are you talking about the README template we discussed a bit last week?
<snusnu>
dkubb: heh no, still talking about: describe Relation, #delete, it 'deletes tuples from the relation'
<dkubb>
oh heh
<dkubb>
I dunno. what does the specdoc output show? I do sometimes use that as a guide
<snusnu>
no idea, and tbh, i don't care much ;)
<snusnu>
but that's my personal opinion of course
<dkubb>
I figured you cared because you asked about it :P
<snusnu>
solnic, dkubb: actually, my first thought when i saw solnic writing spec descriptions like that was, ok, i know, he's writing a lot of application specs and he's gotten used to it :p
<snusnu>
dkubb: hehe
<snusnu>
dkubb: i actually do not care about the spec output .. i want dots :)
<snusnu>
dkubb: that's why i strongly prefer it { should_not include(whatever) } over the long form
<dkubb>
snusnu: I do try to use the tools as they were designed. our usage of subject{} is a bit off what the rspec-core team wanted, but I think it's a reasonable change.. so when I'm not sure if something is correct I sometimes try to understand the original intention
<snusnu>
dkubb: yeah, and my feeling is, that the original intention is (rightfully) deducted from application specs
<dkubb>
yeah, most of the time I care about dots too
<snusnu>
dkubb: in which our style probably doesn't make that much immediate sense, but for our stuff, it's different imo
<dkubb>
I dunno, if I was concerned I would check the specdoc output and whichever made the most sense I would choose that as a convention
<snusnu>
well, then our style looses big time .. with no loss for ourselves (if i dare say so)
<snusnu>
the output doesn't read nice at all .. but hey, it's green or red
<snusnu>
:p
<snusnu>
only sometimes it's not (looking at jruby on travis ;)
<snusnu>
i just have this feeling, that our "audience" are developers, and i like to think that most of those, don't need to rely on specdoc output to get an idea what stuff is doing or not, they just look at the code/specs, because they have to do that anyway
<snusnu>
when developing an app for a client that might be different
<dkubb>
I always wonder about developers who use cucumber for libs
<snusnu>
i *absolutely* don't get it
<dkubb>
is it because it helps the way they think?
<dkubb>
or is it because they're just in a pattern, or want to use something cool?
<snusnu>
that may be, yeah, dunno
<snusnu>
the latter seems more likely to me
<dkubb>
I know sometimes I like to see things from a different pov
<dkubb>
for all I know cucumber does give you a different perspective on the code. ime it was such a pain
<dkubb>
I may try it again *if* someone I was working with had lots of experience with it and could guide me, but for the most part I just see it as alternate syntax for writing integration tests
<snusnu>
it might be nice as an acceptance test tool … stuff like FIT imo is totally justified .. but yeah, that's a different audience .. requirements engineering is done differently with apps and libs
<snusnu>
but i can't really comment, cause i haven't used both cucumber and FIT
<snusnu>
or FITness, or what it's called
<snusnu>
altho i suspect it's good stuff for domain modelling .. like, when you really need your customers to tell you about the domain, and hardcode facts about it in an executable spec
<snusnu>
re libs using cucumber, i cried a little bit, when i made specs for a small reek commit pass, and then saw a broken cucumber build because it was expecting a different line number to be printed out somewhere, because i added, well, locs
<snusnu>
:p
<solnic>
lolwat
<snusnu>
!
<snusnu>
the fact it was checking line numbers makes sense, it was testing its own output, i only never saw the point in using cucumber for that
<solnic>
snusnu: and why do you think I did this huh?
<snusnu>
i know why you did it, that's why i said it's fine with me
<snusnu>
slightly OT, will Mapper.build be public api?
<snusnu>
i guess so
<solnic>
snusnu: not sure, it is public now
<solnic>
we have a del that builds mappers
<solnic>
so I dunno
<solnic>
maybe if you want to inject a mapper with custom loader/dumper
<snusnu>
solnic: i guess it will at least be what dm1 had api protected, a distinction we don't make anymore
<solnic>
then having Mapper.build comes in handy
<snusnu>
yeah
<snusnu>
wait, it was called @api semipublic
<solnic>
yes
<solnic>
I think people might wanna use Mapper.build
<solnic>
we’ll see
<snusnu>
yeah
<snusnu>
solnic: btw, i'd love to have an ultrasimple substation/rom app ready for euruko, thinking about doing a lightning talk
<snusnu>
solnic: maybe an initial mutcov
<snusnu>
as i hate to build useless apps
<snusnu>
and of course i meant eurucamp
<solnic>
snusnu: gonna be hard w/o key generator
<solnic>
time for bed
<solnic>
good night
<snusnu>
solnic: ok good night
<dkubb>
just generate keys using uuids
<dkubb>
simple_uuid is nice and easy
<snusnu>
dkubb: as solnic mentioned it, wwyt about axiom going through some KeyFactory api whenever it sets keys?
<snusnu>
dkubb: it be cool to be able to inject whatever strategy one wants to use to generate keys
<snusnu>
dkubb: including server generated values
<dkubb>
snusnu: well, I'd probably want to get it all working in the context of an adapter first before introducing it in the main part of axiom
<dkubb>
snusnu: there's actually quite a few issues you have to consider with server generated keys, and I would rather deal with them locally in an adapter first
<dkubb>
snusnu: for example, say you're inserting into a join. you have to propagate the inserts in the right order, and you have to figure out some way for the id to be communicated down to the other branch of the insert
<dkubb>
and you have to handle it for more than just a single id.. you can't assume it's one serial id
<dkubb>
you could have a natural join with 4 keys, all of which are decided at insert time
<dkubb>
you also have to handle more than join, it needs to be handled across every operation that is writable
<snusnu>
i see, i was mostly thinking about some api we can use in rom for now (it can very well be implemented in rom) to do simple stuff that makes the memory adapter usable without laying the burden on the app, to pass in keys by itself … i'd love to get to a point soon, where i don't need to expect the app to change once persistent adapters are introduced
<snusnu>
maybe i want to much at this point?
<dkubb>
I think it's too much
<snusnu>
my idea was mainly just, ok, if there's nothing built in that generates keys, lemme inject something that does
<dkubb>
we have so little information about how it should work, except at the highest level, that any decision we make at this point will be the wrong api
<dkubb>
I would just say "use UUIDs" for now
<snusnu>
and generate them from "rom's client", the app?
<dkubb>
that's simple and easy, and you can generate them far before axiom sees them
<dkubb>
you could, or you could have an enumerable that does it
<snusnu>
yeah, tho i wanted to avoid that, but i can see how it might be the most feasible thing to do at this point
<snusnu>
having some Sequence class that gives me uuids will be good enough
<snusnu>
i thought we could maybe already get closer to the real thing, but yeah, i didn't think about the edge cases obviously
<snusnu>
dkubb: my thoughts, iirc, were something like: if axiom/rom sees a tuple with one or many undefined key attributes in it, send that tuple to some injected class by doing key_factory.call(tuple) and expect that to return a new "complete" tuple … i realize how that was just a quick thought tho
<snusnu>
dkubb: ok, it would probably also need the relation, or at least the header (but that's accessible in tuple, isn't it?)
<snusnu>
the header i mean
<snusnu>
anyway, if you say that won't work, i guess i'll just believe you, it's not like i spent much more than 5min on that topic
<dkubb>
yeah, I probably won't design the api until I actually start to implement it