From: "ioquatix (Samuel Williams) via ruby-core" <ruby-core@...>
Date: 2024-02-20T01:03:30+00:00
Subject: [ruby-core:116859] [Ruby master Feature#20282] Enhancing Ruby's Coverage with Per-Test Coverage Reports

Issue #20282 has been updated by ioquatix (Samuel Williams).


`Coverage.suspend` and `Coverage.resume` does not work for multi-thread test runner unfortunately, unless there is something I'm missing. In my integration with `vscode` and `sus`, this can be a problem, as the test host can run multiple tests at the same time in order to handle the requests from `vscode`.

>  Why don't you understand? Is this because of my bad English? Frankly, I am very tired of communicating with you.

I'm sorry to hear that. When we discussed it last time, you suggested introducing some kind of "coverage2" library:

> I don't have time right now, but I would like to rebuild the library as a pure external gem in future, i.e. coverage2 or something, with dynamically generated code, etc. in mind from the beginning.

This proposal does not change the existing interface, but introduce new interface which conceivably be in a gem. However, such a `coverage2` cannot exist without some support from Ruby, e.g. instrumentation of instruction sequence, and so on. That is outlined in the proposal, e.g. `Coverage.enable_coverage(iseq)` etc.

----------------------------------------
Feature #20282: Enhancing Ruby's Coverage with Per-Test Coverage Reports
https://bugs.ruby-lang.org/issues/20282#change-106893

* Author: ioquatix (Samuel Williams)
* Status: Open
* Priority: Normal
----------------------------------------
As Ruby applications grow in complexity, the need for more sophisticated testing and coverage analysis tools becomes paramount. Current coverage tools in Ruby offer a good starting point but fall short in delivering the granularity and flexibility required by modern development practices. Specifically, there is a significant gap in "per-test coverage" reporting, which limits developers' ability to pinpoint exactly which tests exercise which lines of code. This proposal seeks to initiate a discussion around improving Ruby's coverage module to address this gap.

## Objectives

The primary goal of this initiative is to introduce support for per-test coverage reports within Ruby, focusing on three key areas:

1. Scoped Coverage Data Capture: Implementing the capability to capture coverage data within user-defined scopes, such as global, thread, or fiber scopes. This would allow for more granular control over the coverage analysis process.

2. Efficient Data Capture Controls: Developing mechanisms to efficiently control the capture of coverage data. This includes the ability to exclude specific files, include/ignore/merge eval'd code, to ensure that the coverage data is both relevant and manageable.

3. Compatibility and Consistency: Ensuring that the coverage data is exposed in a manner that is consistent with existing coverage tools and standards. This compatibility is crucial for integrating with a wide array of tooling and for facilitating a seamless developer experience.

## Proposed Solutions

The heart of this proposal lies in the introduction of a new subclassable component within the Coverage module, tentatively named `Coverage::Capture`. This component would allow users to define custom coverage capture behaviors tailored to their specific needs. Below is a hypothetical interface for such a mechanism:

```ruby
class Coverage::Capture
  def self.start
    self.new.tap(&:start)
  end
  
  # Start receiving coverage callbacks.
  def start
  end

  # Stop receiving coverage callbacks.
  def stop
  end

  # User-overridable statement coverage callback.
  def statement(iseq, location)
    fetch(iseq)&.statement_coverage.increment(location)
  end

  # Additional methods for branch/declaration coverage would follow a similar pattern.
end

class MyCoverageCapture < Coverage::Capture
  # Provides efficient data capture controls - can return nil if skipping coverage for this iseq, or can store coverage data per-thread, per-fiber, etc.
  def fetch(iseq)
    @coverage[iseq] ||= Coverage.default_coverage(iseq)
  end  
end

# Usage example:
my_coverage_capture = MyCoverageCapture.start
# Execute test suite or specific tests
my_coverage_capture.stop
# Access detailed coverage data
puts my_coverage_capture.coverage.statement_coverage
```

In addition, we'd need a well defined interface for `Coverage.default_coverage`, which includes line, branch and declaration coverage statistics. I suggest we take inspiration from the proposed interface defined by the vscode text editor: https://github.com/microsoft/vscode/blob/b44593a612337289c079425a5b2cc7010216eef4/src/vscode-dts/vscode.proposed.testCoverage.d.ts - this interface was designed to be compatible with a wide range of coverage libraries, so represents the intersection of that functionality.

```ruby
# Hypothetical interface (mostly copied from vscode's proposed interface):
module Coverage
  # Contains coverage metadata for a file
  class Target
    attr_reader :instruction_sequence
    attr_accessor :statement_coverage, :branch_coverage, :declaration_coverage, :detailed_coverage

    # @param statement_coverage [Hash(Location, StatementCoverage)] A hash table of statement coverage instances keyed on location.
    # Similar structures for other coverage data.
    def initialize(instruction_sequence, statement_coverage, branch_coverage=nil, declaration_coverage=nil)
      @instruction_sequence = instruction_sequence
      @statement_coverage = statement_coverage
      @branch_coverage = branch_coverage
      @declaration_coverage = declaration_coverage
    end
  end

  # Coverage information for a single statement or line.
  class StatementCoverage
    # The number of times this statement was executed, or a boolean indicating
    # whether it was executed if the exact count is unknown. If zero or false,
    # the statement will be marked as un-covered.
    attr_accessor :executed
  
    # Statement location (line number? or range? or position? AST?)
    attr_accessor :location
  
    # Coverage from branches of this line or statement. If it's not a
    # conditional, this will be empty.
    attr_accessor :branches
  
    # Initializes a new instance of the StatementCoverage class.
    #
    # @parameter executed [Number, Boolean] The number of times this statement was executed, or a
    # boolean indicating whether it was executed if the exact count is unknown. If zero or false,
    # the statement will be marked as un-covered.
    #
    # @parameter location [Position, Range] The statement position.
    #
    # @parameter branches [Array(BranchCoverage)] Coverage from branches of this line.
    # If it's not a conditional, this should be omitted.
    def initialize(executed, location, branches=[])
      @executed = executed
      @location = location
      @branches = branches
    end
  end

  # Coverage information for a branch
  class BranchCoverage
    # The number of times this branch was executed, or a boolean indicating
    # whether it was executed if the exact count is unknown. If zero or false,
    # the branch will be marked as un-covered.
    attr_accessor :executed
  
    # Branch location.
    attr_accessor :location
  
    # Label for the branch, used in the context of "the ${label} branch was
    # not taken," for example.
    attr_accessor :label
  
    # Initializes a new instance of the BranchCoverage class.
    #
    # @param executed [Number, Boolean] The number of times this branch was executed, or a
    # boolean indicating whether it was executed if the exact count is unknown. If zero or false,
    # the branch will be marked as un-covered.
    #
    # @param location [Position, Range] (optional) The branch position.
    #
    # @param label [String] (optional) Label for the branch, used in the context of
    # "the ${label} branch was not taken," for example.
    def initialize(executed, location=nil, label=nil)
      @executed = executed
      @location = location
      @label = label
    end
  end

  # Coverage information for a declaration
  class DeclarationCoverage
    # Name of the declaration. Depending on the reporter and language, this
    # may be types such as functions, methods, or namespaces.
    attr_accessor :name
  
    # The number of times this declaration was executed, or a boolean
    # indicating whether it was executed if the exact count is unknown. If
    # zero or false, the declaration will be marked as un-covered.
    attr_accessor :executed
  
    # Declaration location.
    attr_accessor :location
  
    # Initializes a new instance of the DeclarationCoverage class.
    #
    # @param name [String] Name of the declaration.
    #
    # @param executed [Number, Boolean] The number of times this declaration was executed, or a
    # boolean indicating whether it was executed if the exact count is unknown. If zero or false,
    # the declaration will be marked as un-covered.
    #
    # @param location [Position, Range] The declaration position.
    def initialize(name, executed, location)
      @name = name
      @executed = executed
      @location = location
    end
  end
end
```

By following this format, we will be compatible with a wide range of external tools.



-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/postorius/lists/ruby-core.ml.ruby-lang.org/