docs/testing/web_test_baseline_fallback.md

# Web Test Baseline Fallback


*** promo
Read [Web Test Expectations and Baselines](web_test_expectations.md) first
if you have not.
***

Baselines can vary by platforms, in which case we need to check in multiple
versions of a baseline. Meanwhile, we would like to avoid storing identical
baselines by allowing a platform to fall back to another. This document first
introduces how platform-specific baselines are structured and how we search for
a baseline (the fallback mechanism), and then goes into the details of baseline
optimization and rebaselining.

[TOC]

## Terminology

* **Root directory**:
    [`//src/third_party/blink/web_tests`](../../third_party/blink/web_tests)
    is the root directory (of all the web tests and baselines). All relative
    paths in this document start from this directory.
* **Test name**: the name of a test is its relative path from the root
    directory (e.g. `html/dom/foo/bar.html`).
* **Baseline name**: replacing the extension of a test name with
    `-expected.{txt,png,wav}` gives the corresponding baseline name.
* **Virtual tests**: tests can have virtual variants. For example,
    `virtual/gpu/html/dom/foo/bar.html` is the virtual variant of
    `html/dom/foo/bar.html` in the `gpu` suite. Only the latter file exists on
    disk, and is called the base of the virtual test. See
    [Web Tests#Testing Runtime Flags](web_tests.md#testing-runtime-flags)
    for more details.
* **Platform directory**: each directory under
    [`platform/`](../../third_party/blink/web_tests/platform) is a platform
    directory that contains baselines (no tests) for that platform. Directory
    names are in the form of `PLATFORM-VERSION` (e.g. `mac-mac10.12`), except
    for the latest version of a platform which is just `PLATFORM` (e.g. `mac`).

## Baseline fallback

Each platform has a pre-configured fallback when a baseline cannot be found in
this platform directory. A general rule is to have older versions of an OS
falling back to newer versions. Besides, Android falls back to Linux, which then
falls back to Windows. Eventually, all platforms fall back to the root directory
(i.e. the generic baselines that live alongside tests). The rules are configured
by `FALLBACK_PATHS` in each Port class in
[`//src/third_party/blink/tools/blinkpy/web_tests/port`](../../third_party/blink/tools/blinkpy/web_tests/port).

All platforms can be organized into a tree based on their fallback relations (we
are not considering virtual test suites yet). See the lower half (the
non-virtual subtree) of this
[graph](https://docs.google.com/drawings/d/13l3IUlSE99RoKjDwEWuY1O77simAhhF6Wi0fZdkSaMA/).
Walking from a platform to the root gives the **search path** of that platform.
We check each directory on the search path in order and see if "directory +
baseline name" points to a file on disk (note that baseline names are relative
paths), and stop at the first one found.

### Virtual test suites

Now we add virtual test suites to the picture, using a test named
`virtual/gpu/html/dom/foo/bar.html` as an example to demonstrate the process.
The baseline search process for a virtual test consists of two passes:

1. Treat the virtual test name as a regular test name and search for the
   corresponding baseline name using the same search path, which means we are in
   fact searching in directories like `platform/*/virtual/gpu/...`, and
   eventually `virtual/gpu/...` (a.k.a. the virtual root).
2. If no baseline can be found so far, we retry with the non-virtual (base) test
   name `html/dom/foo/bar.html` and walk the search path again.

The [graph](https://docs.google.com/drawings/d/13l3IUlSE99RoKjDwEWuY1O77simAhhF6Wi0fZdkSaMA/)
visualizes the full picture. Note that the two passes are in fact the same with
different test names, so the virtual subtree is a mirror of the non-virtual
subtree. The two trees are connected by the virtual root that has different
ancestors (fallbacks) depending on which platform we start from; this is the
result of the two-pass baseline search.

*** promo
__Note:__ there are in fact two more places to be searched before everything
else: additional directories given via command line arguments and flag-specific
baseline directories. They are maintained manually and are not discussed in this
document.
***

## Tooling implementation

This section describes the implications the fallback mechanism has on the
implementation details of tooling, namely `blink_tool.py`. If you are not
hacking `blinkpy`, you can stop here.

### Optimization

We can remove a baseline if it is the same as its fallback. An extreme example
is that if all platforms have the same result, we can just have a single generic
baseline. Here is the algorithm used by
[`blink_tool.py optimize-baselines`](../../third_party/blink/tools/blinkpy/common/checkout/baseline_optimizer.py)
to optimize the duplication away.

Notice from the previous section that the virtual and non-virtual parts are two
identically structured subtrees. Trees are easy to work with: we can simply
traverse the tree from leaves up to the root, and if there are two identical
baselines on two nodes on the path with no other nodes in between or all nodes
in between have no baselines, keep the one closer to the root (delete the
baseline on the node further from the root).

The virtual root is special because it has multiple parents. Yet if we can cut
the edges between the two subtrees (i.e. to make the virtual subtree
self-contained), we can apply the same algorithm to both of them. A subtree is
self-contained when it does not need to fallback to ancestors, which can be
guaranteed by placing a baseline on its root. If the virtual root already has a
baseline, we can simply ignore these edges without doing anything; otherwise, we
need to make sure all children of the virtual root have baselines by copying
the non-virtual fallbacks to the ones that do not (we cannot copy the generic
baseline to the virtual root because virtual platforms may have different
results).

In addition, the optimizer also removes redundant all-PASS testharness.js
results. Such baselines are redundant when there are no other fallbacks later
on the search path (including if the all-PASS baselines are at root), because
`run_web_tests.py` assumes all-PASS testharness.js results when baselines can
not be found for a platform.

### Rebaseline

The fallback mechanism also affects the rebaseline tool (`blink_tool.py
rebaseline{-cl}`). When asked to rebaseline a test on some platforms, the tool
downloads results from corresponding try bots and put them into the respective
platform directories. This is potentially problematic. Because of the fallback
mechanism, the new baselines may affect some other platforms that are not being
rebaselining but fall back to the rebaselined platforms.

The solution is to copy the current baselines from the to-be-rebaselined
platforms to all the platforms that immediately fall back to them (i.e. down one
level in the fallback tree) before downloading new baselines. This is done in a
hidden internal command
[`blink_tool.py copy-existing-baselines`](../../third_party/blink/tools/blinkpy/tool/commands/copy_existing_baselines.py),
which is always executed by `blink_tool.py rebaseline`.

Finally, `blink_tool.py rebaseline{-cl}` also does optimization in the end by
default.