From: sam.rawlins@...
Date: 2014-03-03T22:28:22+00:00
Subject: [ruby-core:61271] [ruby-trunk - Feature #9508] Add method coverage and branch coverage metrics

Issue #9508 has been updated by Sam Rawlins.


Endoh-san and Eric, thanks both for considering these changes. I'm in favor of changing the format of `Coverage.result` because of the reasons I outline in [#14](#note-14), but I can give more evidence that it should be a very safe change:

After a code search on GitHub [1], and after looking at the Code Metrics section of The Ruby Toolbox [2], it is clear that the Ruby community in general (at least open source) rely almost 100% on the SimpleCov gem [3]. Many of the other metrics tools found in the Ruby Toolbox use SimpleCov (coveralls, flog, rails_best_practices, and more) and never call `Coverage.result` manually. I can find no other libraries currently maintained that execute "Coverage.result".

And I still just don't like the idea of _some_ Coverage results being duped/frozen/cleared, while others are not.

Another idea was suggested to me, where we introduce an optional parameter to `Coverage.start`, which defaults to `:lines`. Old libraries can continue to call `Coverage.start`, which will cause `Coverage.result` to return a Hash of Ruby files mapped to line counts. New libraries (or updated SimpleCov) can call `Coverage.start(:all)`, which will cause `Coverage.result` to return the new, more complicated format.

This solution, however, makes it _harder_ for SimpleCov to be compatible with old AND new format: the library must figure out whether Coverage.start takes an optional argument or not (or use something like RUBY_VERSION), track that, and parse `Coverage.result` differently based on the answer. I would again just prefer upgrading the format to this solution, allowing SimpleCov to just test `coverage.is_a? Array`.

[1] https://github.com/search?l=ruby&q=%22Coverage.start%22&ref=searchresults&type=Code
[2] https://www.ruby-toolbox.com/categories/code_metrics
[3] https://github.com/colszowka/simplecov

----------------------------------------
Feature #9508: Add method coverage and branch coverage metrics
https://bugs.ruby-lang.org/issues/9508#change-45607

* Author: Sam Rawlins
* Status: Open
* Priority: Normal
* Assignee: 
* Category: 
* Target version: 
----------------------------------------
Since the Coverage extension was introduced in Ruby 1.9, Ruby has had built-in line code coverage. Ruby should support more of the basic code coverage metrics [1]. I have a pull request on GitHub ( https://github.com/ruby/ruby/pull/511 ) to add Method Coverage (Function Coverage) and Branch Coverage. I'd love feedback to improve it.

Currently, with this feature, Coverage.result would look like:

    {"/Users/sam/code/ruby/cov_method.rb" => {
      lines: [1, 2, 2, 20, nil, nil, 2, 2, 2, nil, 0, nil, nil, nil, 1, 0, nil, nil, 1, 1, nil, nil, 1],
      methods: {1=>2, 15=>0, 19=>1},
      branches: {8=>2, 11=>0}
    }}

which includes
* the current Ruby line coverage report,
* as well as a method report (The method defined on line 1 was called 2 times; the method on line 15 was called 0 times; ...),
* and a branch report (the branch on line 8 was called 2 times; the branch on line 11 was called 0 times).

Branches
--------

Branches include the bodies of if, elsif, else, unless, and when statements, which are all tracked with this new feature. However, this feature is not aware of void bodies, for example:

    if foo
      :ok
    end

will report that only one branch exists in the file. It would be better to declare that there is a branch body on line 2, and a void branch body on line 3, or perhaps line 1. This would require the keys of the [:branch] Hash to be something other than line numbers. Perhaps label_no? Perhaps nd_type(node) paired with line or label_no?

More Coverage
-------------

I think that Statement Coverage, and Condition Coverage could be added to this feature, using the same techniques.

Caveats
-------

I was not very clear on the bit-arrays used in ruby.h, and just used values for the new macros that seemed to work.

Also, I would much rather use Ranges to identify a branch, so that a Coverage analyzer like SimpleCov won't need any kind of Ruby parser to identify and highlight a full chunk of code as a tested branch, or a not tested branch. I'm trying to find how that could be implemented...

[1] Wikipedia has good definitions: http://en.wikipedia.org/wiki/Code_coverage

---Files--------------------------------
pull-request-511.patch (26.7 KB)
pull-request-511.patch (38.5 KB)
pull-request-511.patch (57 KB)


-- 
http://bugs.ruby-lang.org/