From: mame@... Date: 2017-11-02T23:20:35+00:00 Subject: [ruby-core:83651] [Ruby trunk Feature#13901] Add branch coverage Issue #13901 has been updated by mame (Yusuke Endoh). Marc-Andre, thank you for letting me know! I didn't know it. DeepCover is so great! I'd like to cooperate with you! What do you think about the format of `Coverage.result`? If you have any request, let me know. I'd like to consider it. It would be difficult for the flexibility of pure Ruby, but we'd like to provide a feature for filtering and controlling measurement target. I'm also afraid about performance degradation by pure Ruby implementation. How long does it take to run the test suite of ActiveSupport with and without coverage measurement? Anyway, it would be a good diversity that there are multiple approaches. Thank you for your effort! Our goal is also the visualization like Istanbul, finer coverage measurement than a line. For the purpose, Yuichiro Kaneko and I are working on improvement of NODE representation so that each NODE object keeps track of not only lineno of the original code, but also more precise and useful position information: * The first lineno and first column number that the NODE begins * The last lineno and last column number that the NODE ends Yuichiro Kaneko has already implemented the column number of the beggining of the NODE in trunk. Here is the current format of `Coverage.result` for Example 1: ~~~~ {"/home/mame/work/ruby/target.rb"=> {:lines=>[1, 0, 1, 0, nil, 1, nil], :branches=> {[:if, 0, 3, 0]=>{[:then, 1, 4, 2]=>0, [:else, 2, 6, 2]=>1}, [:if, 3, 1, 0]=>{[:then, 4, 2, 2]=>0, [:else, 5, 3, 0]=>1}}, :methods=>{}}} ~~~~ `[:if, 0, 3, 0]` reads "the `if` statement that starts from line 3 and column 0", and `[:then, 1, 4, 2]` reads "the `if` statement that starts from line 4 and column 2". (Please ignore the second element. When the column number is not implemented yet, it was needed as a unique number to avoid a conflict.) It is already possible to identify the last position, but we need more work. This is just an implementation issue, but a NODE object has just one word for keeping the information because of the GC limitation. I'm trying to detach NODEs from GC management, so that we can freely change the data structure of NODEs. (This is my personal idea, but I want this work to lead to the embedded parser API like "perser" gem.) ---------------------------------------- Feature #13901: Add branch coverage https://bugs.ruby-lang.org/issues/13901#change-67677 * Author: mame (Yusuke Endoh) * Status: Open * Priority: Normal * Assignee: * Target version: ---------------------------------------- I plan to add "branch coverage" (and "method coverage") as new target types of coverage.so, the coverage measurement library. I'd like to introduce this feature for Ruby 2.5.0. Let me to hear your opinions. ## Basic Usage of the Coverage API The sequence is the same as the current: (1) require "coverage.so", (2) start coverage measurement by `Coverage.start`, (3) load a program being measured (typically, a test runner program), and (4) get the result by `Coverage.result`. When you pass to `Coverage.start` with keyword argument "`branches: true`", branch coverage measurement is enabled. test.rb ~~~ require "coverage" Coverage.start(lines: true, branches: true) load "target.rb" p Coverage.result ~~~ target.rb ~~~ 1: if 1 == 0 2: p :match 3: else 4: p :not_match 5: end ~~~ By measuring coverage of target.rb, the result will be output (manually formatted): ~~~ $ ruby test.rb :not_match {".../target.rb" => { :lines => [1, 0, nil, 1, nil], :branches => { [:if, 0, 1] => { [:then, 1, 2] => 0, [:else, 2, 4] => 1 } } } ~~~ `[:if, 0, 1]` reads "if branch at Line 1", and `[:then, 1, 2]` reads "then clause at Line 2". So, `[:if,0,1] => { [:then,1,2]=>0, [:else,2,4]=>0 }` reads "the branch from Line 1 to Line 2 has never executed, and the branch from Line 1 to Line 4 has executed once." The second number (`0` of `[:if, 0, 1]`) is a unique ID to avoid conflict, just in case where multiple branches are written in one line. This format of a key is discussed in "Key format" section. ## Why needed Traditional coverage (line coverage) misses a branch in one line. Branch coverage is useful to find such untested code. See the following example. target.rb ~~~ p(:foo) unless 1 == 0 p(1 == 0 ? :foo : :bar) ~~~ The result is: ~~~ {".../target.rb" => { :lines => [1, 1], :branches => { [:unless, 0, 1] => { [:else, 1, 1] => 0, [:then, 2, 1] => 1 }, [:if, 3, 2] => { [:then, 4, 2] => 0, [:else, 5, 2] => 1 } } }} ~~~ Line coverage tells coverage 100%, but branch coverage shows that the `unless` statement of the first line has never taken true and that the ternary operator has never taken true. ## Current status I've already committed the feature in trunk as an experimental feature. To enable the feature, you need to set the environment variable `COVERAGE_EXPERIMENTAL_MODE` = `true`. I plan to activate this feature by default by Ruby 2.5 release, if there is no big problem. ## Key format The current implementation uses `[