From: ruby-core@... Date: 2017-11-03T06:31:25+00:00 Subject: [ruby-core:83655] [Ruby trunk Feature#13901] Add branch coverage Issue #13901 has been updated by marcandre (Marc-Andre Lafortune). mame (Yusuke Endoh) wrote: > What do you think about the format of `Coverage.result`? Having a character position is a big improvement on just a line #, that's for sure. It makes it possible to pinpoint exactly what we're talking about, and with the help of `parser` or similar, makes it possible to get all the rest of the information that might be wanted. I imagine you are planning on covering `||`, `&&`, `&.` and `? :`, right? > I'm also afraid about performance degradation by pure Ruby implementation. How long does it take to run the test suite of ActiveSupport with and without coverage measurement? I'm getting a 15% increase in time to run the tests for activesupport, for example (not counting instrumenting or analysis, just running `rake`). We have not looked much into this, but I'm not worried about this. It's not something that's needed quickly usually (since you want to run the full test suite), and a pure Ruby implementation should only scale linearly in the worst case. Our implementation uses only trackers of the form `$_global[1][2] += 1`, and often storage in a local variable (like `temp = foo.bar(1,2,3); $_global[1][2] +=2; temp`) or some condition like `... if temp` or ` && $_global_var[...] +=1`. This is all optimized by Ruby and doesn't require method dispatch. We also use the strict minimum number of trackers required (except for multiple assignments where we use one more tracker than theoretically possible). > It is already possible to identify the last position, but we need more work. This is just an implementation issue, but a NODE object has just one word for keeping the information because of the GC limitation. I'm trying to detach NODEs from GC management, so that we can freely change the data structure of NODEs. (This is my personal idea, but I want this work to lead to the embedded parser API like "perser" gem.) This sounds great. The precise beginning (column and line) is the only thing strictly necessary though; the rest could be obtained by parsing the code with `parser` and locating the node in question. In any case, tools may have to do that anyways for really good output, because beginning and end is not necessarily enough. For example `parser` has very detailed information about a node; it can tell you the beginning and end of the whole node's expression, but also of the keyword, for example, etc. Also it gives information about children nodes. We use this to output very precisely, for example the punctuation like ,(); are considered non executable, etc. For example, deep-cpver will color `0&.nonzero?&.foo(1, 2, 3).nil?` all in green, but "foo(1, 2, 3)" in red (well, the (), are gray, actually) A minor issue is that branch coverage will not solve cases of exceptions being raised. Check the last line (char coverage) of: $ deep-cover -e "puts(:a, notdefined, :c) rescue nil" The first argument (:a) is in green, the last one (:c) in red, because `notdefined` raised a NoMethodError. Is there a plan to implement such detailed coverage? ---------------------------------------- Feature #13901: Add branch coverage https://bugs.ruby-lang.org/issues/13901#change-67682 * Author: mame (Yusuke Endoh) * Status: Open * Priority: Normal * Assignee: * Target version: ---------------------------------------- I plan to add "branch coverage" (and "method coverage") as new target types of coverage.so, the coverage measurement library. I'd like to introduce this feature for Ruby 2.5.0. Let me to hear your opinions. ## Basic Usage of the Coverage API The sequence is the same as the current: (1) require "coverage.so", (2) start coverage measurement by `Coverage.start`, (3) load a program being measured (typically, a test runner program), and (4) get the result by `Coverage.result`. When you pass to `Coverage.start` with keyword argument "`branches: true`", branch coverage measurement is enabled. test.rb ~~~ require "coverage" Coverage.start(lines: true, branches: true) load "target.rb" p Coverage.result ~~~ target.rb ~~~ 1: if 1 == 0 2: p :match 3: else 4: p :not_match 5: end ~~~ By measuring coverage of target.rb, the result will be output (manually formatted): ~~~ $ ruby test.rb :not_match {".../target.rb" => { :lines => [1, 0, nil, 1, nil], :branches => { [:if, 0, 1] => { [:then, 1, 2] => 0, [:else, 2, 4] => 1 } } } ~~~ `[:if, 0, 1]` reads "if branch at Line 1", and `[:then, 1, 2]` reads "then clause at Line 2". So, `[:if,0,1] => { [:then,1,2]=>0, [:else,2,4]=>0 }` reads "the branch from Line 1 to Line 2 has never executed, and the branch from Line 1 to Line 4 has executed once." The second number (`0` of `[:if, 0, 1]`) is a unique ID to avoid conflict, just in case where multiple branches are written in one line. This format of a key is discussed in "Key format" section. ## Why needed Traditional coverage (line coverage) misses a branch in one line. Branch coverage is useful to find such untested code. See the following example. target.rb ~~~ p(:foo) unless 1 == 0 p(1 == 0 ? :foo : :bar) ~~~ The result is: ~~~ {".../target.rb" => { :lines => [1, 1], :branches => { [:unless, 0, 1] => { [:else, 1, 1] => 0, [:then, 2, 1] => 1 }, [:if, 3, 2] => { [:then, 4, 2] => 0, [:else, 5, 2] => 1 } } }} ~~~ Line coverage tells coverage 100%, but branch coverage shows that the `unless` statement of the first line has never taken true and that the ternary operator has never taken true. ## Current status I've already committed the feature in trunk as an experimental feature. To enable the feature, you need to set the environment variable `COVERAGE_EXPERIMENTAL_MODE` = `true`. I plan to activate this feature by default by Ruby 2.5 release, if there is no big problem. ## Key format The current implementation uses `[