docs/testing/web_tests_with_manual_fallback.md - chromium/src - Git at Google

 # Web Tests with Manual Fallback

 Some Blink features cannot be automatically tested using the Web Platform. Prime
 examples are the APIs that require
 [user activation](https://html.spec.whatwg.org/multipage/interaction.html#tracking-user-activation)
 (also known as _a user gesture_), such as [Fullscreen](https://developer.mozilla.org/en-US/docs/Web/API/Fullscreen_API).
 Automated tests for these Blink features must rely on special APIs, which are
 only exposed in testing environments, and are therefore not available in a
 normal browser session.

 A popular pattern used in these tests is to rely on the user to perform some
 manual steps in order to run the test case in a normal browser session. These
 tests are effectively
 [manual tests](https://web-platform-tests.org/writing-tests/manual.html), with
 additional JavaScript code that automatically performs the desired manual steps,
 when loaded in an environment that exposes the needed testing APIs.

 ## Motivation

 Web tests that degrade to manual tests in the absence of testing APIs have
 the following benefits.

 * The manual test component can be debugged in a normal browser session, using
   the rich [developer tools](https://developer.chrome.com/devtools). Tests
   without a manual fallback can only be debugged in the test runner.
 * The manual tests can run in other browsers, making it easy to check whether
   our behavior matches other browsers.
 * The web tests can form the basis for manual tests that are contributed to
   [web-platform-tests](./web_platform_tests.md).

 Therefore, the desirability of adding a manual fallback to a test heavily
 depends on whether the feature under test is a Web Platform feature or a
 Blink-only feature, and on the developer's working style. The benefits above
 should be weighed against the added design effort needed to build a manual test,
 and the size and complexity introduced by the manual fallback.

 ## Development Tips

 A natural workflow for writing a web test that gracefully degrades to a
 manual test is to first develop the manual test in a browser, and then add code
 that feature-checks for testing APIs, and uses them to automate the test's
 manual steps.

 Manual tests should minimize the chance of user error. This implies keeping the
 manual steps to a minimum, and having simple and clear instructions that
 describe all the configuration changes and user gestures that match the effect
 of the Blink-specific APIs used by the test.

 ## Example

 Below is an example of a fairly minimal test that uses a Blink-Specific API
 (`window.eventSender`), and gracefully degrades to a manual test.

 ```html
 <!doctype html>
 <meta charset="utf-8">
 <title>DOM: Event.isTrusted for UI events</title>
 <link rel="help" href="https://dom.spec.whatwg.org/#dom-event-istrusted">
 <link rel="help" href="https://dom.spec.whatwg.org/#constructing-events">
 <meta name="assert"
     content="Event.isTrusted is true for events generated by user interaction">
 <script src="../../resources/testharness.js"></script>
 <script src="../../resources/testharnessreport.js"></script>

 <p>Please click on the button below.</p>
 <button>Click Me!</button>

 <script>
 'use strict';

 setup({ explicit_timeout: true });

 promise_test(() => {
   const button = document.querySelector('button');
   return new Promise((resolve, reject) => {
     const button = document.querySelector('button');
     button.addEventListener('click', (event) => {
       resolve(event);
     });

     if (window.eventSender) {
       eventSender.mouseMoveTo(button.offsetLeft, button.offsetTop);
       eventSender.mouseDown();
       eventSender.mouseUp();
     }
   }).then((clickEvent) => {
     assert_true(clickEvent.isTrusted);
   });

 }, 'Click generated by user interaction');

 </script>
 ```

 The test exhibits the following desirable features:

 * It has a second specification URL (`<link rel="help">`), because the paragraph
   that documents the tested feature (referenced by the primary URL) is not very
   informative on its own.
 * It links to the
   [WHATWG Living Standard](https://wiki.whatwg.org/wiki/FAQ#What_does_.22Living_Standard.22_mean.3F),
   rather than to a frozen version of the specification.
 * It contains clear instructions for manually triggering the test conditions.
   The test starts with a paragraph (`<p>`) that tells the tester exactly what to
   do, and the `<button>` that needs to be clicked is clearly labeled.
 * It disables the timeout mechanism built into `testharness.js` by calling
   `setup({ explicit_timeout: true });`
 * It checks for the presence of the Blink-specific testing APIs
   (`window.eventSender`) before invoking them. The test does not automatically
   fail when the APIs are not present.
 * It uses [Promises](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Promise)
   to separate the test setup from the assertions. This is particularly helpful
   for manual tests that depend on a sequence of events to occur, as Promises
   offer a composable way to express waiting for asynchronous events that avoids
   [callback hell](http://stackabuse.com/avoiding-callback-hell-in-node-js/).

 Notice that the test is pretty heavy compared to a minimal JavaScript test that
 does not rely on testing APIs. Only use testing APIs when the desired testing
 conditions cannot be set up using Web Platform APIs.
	# Web Tests with Manual Fallback

	Some Blink features cannot be automatically tested using the Web Platform. Prime
	examples are the APIs that require
	[user activation](https://html.spec.whatwg.org/multipage/interaction.html#tracking-user-activation)
	(also known as _a user gesture_), such as [Fullscreen](https://developer.mozilla.org/en-US/docs/Web/API/Fullscreen_API).
	Automated tests for these Blink features must rely on special APIs, which are
	only exposed in testing environments, and are therefore not available in a
	normal browser session.

	A popular pattern used in these tests is to rely on the user to perform some
	manual steps in order to run the test case in a normal browser session. These
	tests are effectively
	[manual tests](https://web-platform-tests.org/writing-tests/manual.html), with
	additional JavaScript code that automatically performs the desired manual steps,
	when loaded in an environment that exposes the needed testing APIs.

	## Motivation

	Web tests that degrade to manual tests in the absence of testing APIs have
	the following benefits.

	* The manual test component can be debugged in a normal browser session, using
	the rich [developer tools](https://developer.chrome.com/devtools). Tests
	without a manual fallback can only be debugged in the test runner.
	* The manual tests can run in other browsers, making it easy to check whether
	our behavior matches other browsers.
	* The web tests can form the basis for manual tests that are contributed to
	[web-platform-tests](./web_platform_tests.md).

	Therefore, the desirability of adding a manual fallback to a test heavily
	depends on whether the feature under test is a Web Platform feature or a
	Blink-only feature, and on the developer's working style. The benefits above
	should be weighed against the added design effort needed to build a manual test,
	and the size and complexity introduced by the manual fallback.

	## Development Tips

	A natural workflow for writing a web test that gracefully degrades to a
	manual test is to first develop the manual test in a browser, and then add code
	that feature-checks for testing APIs, and uses them to automate the test's
	manual steps.

	Manual tests should minimize the chance of user error. This implies keeping the
	manual steps to a minimum, and having simple and clear instructions that
	describe all the configuration changes and user gestures that match the effect
	of the Blink-specific APIs used by the test.

	## Example

	Below is an example of a fairly minimal test that uses a Blink-Specific API
	(`window.eventSender`), and gracefully degrades to a manual test.

	```html
	<!doctype html>
	<meta charset="utf-8">
	<title>DOM: Event.isTrusted for UI events</title>
	<link rel="help" href="https://dom.spec.whatwg.org/#dom-event-istrusted">
	<link rel="help" href="https://dom.spec.whatwg.org/#constructing-events">
	<meta name="assert"
	content="Event.isTrusted is true for events generated by user interaction">
	<script src="../../resources/testharness.js"></script>
	<script src="../../resources/testharnessreport.js"></script>

	<p>Please click on the button below.</p>
	<button>Click Me!</button>

	<script>
	'use strict';

	setup({ explicit_timeout: true });

	promise_test(() => {
	const button = document.querySelector('button');
	return new Promise((resolve, reject) => {
	const button = document.querySelector('button');
	button.addEventListener('click', (event) => {
	resolve(event);
	});

	if (window.eventSender) {
	eventSender.mouseMoveTo(button.offsetLeft, button.offsetTop);
	eventSender.mouseDown();
	eventSender.mouseUp();
	}
	}).then((clickEvent) => {
	assert_true(clickEvent.isTrusted);
	});

	}, 'Click generated by user interaction');

	</script>
	```

	The test exhibits the following desirable features:

	* It has a second specification URL (`<link rel="help">`), because the paragraph
	that documents the tested feature (referenced by the primary URL) is not very
	informative on its own.
	* It links to the
	[WHATWG Living Standard](https://wiki.whatwg.org/wiki/FAQ#What_does_.22Living_Standard.22_mean.3F),
	rather than to a frozen version of the specification.
	* It contains clear instructions for manually triggering the test conditions.
	The test starts with a paragraph (`<p>`) that tells the tester exactly what to
	do, and the `<button>` that needs to be clicked is clearly labeled.
	* It disables the timeout mechanism built into `testharness.js` by calling
	`setup({ explicit_timeout: true });`
	* It checks for the presence of the Blink-specific testing APIs
	(`window.eventSender`) before invoking them. The test does not automatically
	fail when the APIs are not present.
	* It uses [Promises](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Promise)
	to separate the test setup from the assertions. This is particularly helpful
	for manual tests that depend on a sequence of events to occur, as Promises
	offer a composable way to express waiting for asynchronous events that avoids
	[callback hell](http://stackabuse.com/avoiding-callback-hell-in-node-js/).

	Notice that the test is pretty heavy compared to a minimal JavaScript test that
	does not rely on testing APIs. Only use testing APIs when the desired testing
	conditions cannot be set up using Web Platform APIs.