Running pinned Biome in CI for PR comments

[lgarron]

Per the following, it is documented that npx biome ci should be printing line-specific information for PRs in GitHub.

• Support Workflow Commands #441
• https://biomejs.dev/recipes/continuous-integration/
• https://biomejs.dev/blog/biome-v1-5/#the-command-biome-ci-now-prints-diagnostics-in-github-prs

However, I have never gotten this to work. Recent test: https://github.com/lgarron/permission.site/pull/2/changes

• Passing --reporter github does print something, but it just prints File content differs from formatting output associated with the top of a file, not actual failure locations.
• Adding the pull-requests: write permission doesn't seem to make a difference.

Also, https://github.com/marketplace/actions/run-biome-with-reviewdog is a non-starter because it uses its own hardcoded Biome instead of @biomejs/biome from pinned project dependencies.

Presumably I'm missing something, but I'm not sure where else to look.

[mg1986jp]

The issue is that formatting diagnostics and lint diagnostics behave differently with --reporter github.

Lint diagnostics have precise line/column locations, so they produce proper annotations like ::error file=src/foo.ts,line=10,col=5::... which show up inline in the PR diff.

Formatting diagnostics are whole-file — Biome just knows "this file's output differs from the input." There's no specific line to point to, so you get a single annotation at the top of the file saying "File content differs from formatting output." That's not a bug, it's a limitation of how formatting checks work.

If your goal is getting inline PR annotations for lint issues, biome ci --reporter github should already work for those. For formatting, the best you'll get is the file-level annotation.

If you want actual PR review comments (not just annotations), you'd need something like reviewdog or a custom action that parses Biome's JSON output and posts review comments via the GitHub API. The built-in --reporter github only produces workflow commands, not review comments.

You could split your CI into two steps:

- run: npx biome check --reporter github   # lint annotations (line-specific)
- run: npx biome format --check             # formatting check (pass/fail only)

That way lint issues show inline and formatting just fails the build without noisy file-level annotations.

[lgarron]

Thanks, that helped me get started on a fix. It seems to be working now: https://github.com/chromium/permission.site/pull/153/commits

Here's everything I needed to understand/figure out.

• There's special markup that tools can output in GitHub actions to either issue annotations, or actually write to a pull request. They are apparently different formats.
• Biome only implements annotations.
• There are tools to write comments on pull requests, and apparently reviewdog is the de facto choice.
• https://github.com/marketplace/actions/run-biome-with-reviewdog does indeed seem to try to download Biome (trying to match the pinned version?) instead of actually running the locally installed Biome. There doesn't seem to be a way to configure it to run Biome using npx without trying to download its own copy.
• Biome can output rdjson format for Reviewdog itself.
• reviewdog has AI contributions. I'm not happy about this, but fine. I can't prevent AI use across all ecosystem, so I just have to hope it's well-maintained regardless.
• reviewdog can warn about a missing .reviewdog.yml, but this file is not necessary to create or specify when invoked appropriately.
• The reviewdog docs document how to specify a custom runner in .reviewdog.yml, but passing it to -runners doesn't seem to work.
• Biome can appear to trigger Tracking Issue for unix_sigpipe rust-lang/rust#97889 when the pipe to reviewdog is not set up correctly, but this is a red herring. It doesn't seem to trigger when the two programs are properly piped.
• reviewdog requires a pull-requests: write permission in its job, even though the examples in the project's README imply this is not necessary. (At least a helpful error message is printed instead.)
• make and npm run both require special configuration to avoid printing the commands they run (which can otherwise trip up reviewdogs parser).
• To ensure the CI logs still show the actual linting output, it makes the most sense to first run through Reviewdog without erroring on failure, then running a second time through Biome as normal.
• As mentioned above, Biome doesn't output line numbers in error metadata for formatting errors. Biome does clearly have this information under some circumstances, since it's printed on the commandline sometimes, but this is not available in general.
• Also as mentioned above, Biome outputs metadata from biome check that Reviewdog can use, but that doesn't cover formatting. The easiest way to get some sort of output on PRs for formatting errors is to run biome ci --linter-enabled=false separately¹ (at least using Biome's current behaviour).

And of course all of this can only be tested properly in GitHub Actions on a live repo.

Also, Biome ci seems to output redundant annotations and I'm not sure this is easy to prevent. Perhaps I'll just find a way turn that off in favor of CI failures and PR comments, even if it loses annotations on the commits. (It's not possible to have annotations "just for commits but not pull requests", since branches on the repo itself will have the annotations if they are turned into PRs.)

Footnotes

1. biome format --check doesn't seem to be a thing? ↩

[mg1986jp]

@lgarron Nice, that PR looks solid. The two-pass approach (reviewdog for PR comments first, then biome ci for the actual pass/fail) is exactly the right pattern.

On the footnote — biome format --check isn't a subcommand, yeah. biome format always writes changes in-place. For check-only mode you'd use biome ci --linter-enabled=false like you figured out, or biome check --formatter-enabled=true --linter-enabled=false. The ci command is essentially check but designed to fail in CI rather than apply fixes.

The redundant annotations issue is a known pain point. You can pass --reporter=json instead of the default to suppress the GitHub annotations output, then let reviewdog handle all the PR-level feedback. That way you avoid the double-annotation noise.

Re: formatting diagnostics lacking line numbers in the metadata — Biome knows what changed (it has the diff internally) but the rdjson output just doesn't emit per-line locations for format diffs. It's a limitation of how the formatter diagnostic is modeled. The file-level reviewdog comment is about the best you'll get without a custom wrapper that parses the diff output.

[lgarron]

yargh, so close.

Looks like the Biome rdjson output includes a fix description instead of the actual fix?

{
  "source": {
    "name": "Biome",
    "url": "https://biomejs.dev"
  },
  "diagnostics": [
    {
      "code": {
        "url": "https://biomejs.dev/linter/rules/use-const",
        "value": "lint/style/useConst"
      },
      "location": {
        "path": "index.js",
        "range": {
          "end": {
            "column": 14,
            "line": 342
          },
          "start": {
            "column": 11,
            "line": 342
          }
        }
      },
      "message": "This let declares a variable that is only assigned once.",
      "suggestions": [
        {
          "range": {
            "end": {
              "column": 23,
              "line": 342
            },
            "start": {
              "column": 15,
              "line": 342
            }
          },
          "text": "Safe fix: Use const instead."
        }
      ]
    }
  ]
}

Filed at: #10477

[mg1986jp]

@lgarron Yeah that's a Biome bug (or at least a spec mismatch). The rdjson suggestions text field is supposed to contain the replacement code, but Biome is putting the human-readable fix description there instead.

Looking at the JSON: "text": "Safe fix: Use const instead." — that should be "text": "const" (the actual replacement text for the range).

reviewdog reads suggestions[].text as the literal code to suggest in the PR review comment, so it shows the description string as if it were code.

Two options:

1. File a Biome issue — this is genuinely a bug in Biome's rdjson reporter. The text field in RDFormat suggestions is defined as the replacement content, not a description. Biome is populating it wrong.

2. Workaround for now — pipe through jq to strip the suggestions before feeding to reviewdog:

   npx biome check --reporter=json . | \
     jq '.diagnostics |= map(del(.suggestions))' | \
     reviewdog -f=rdjson -reporter=github-pr-review

   You lose the inline fix suggestions but the diagnostics and line locations still work.

The fix on Biome's side would be straightforward — their rdjson emitter just needs to put the actual replacement text in text instead of the description message. Worth opening an issue for it.