Blame - docs/testing/web_tests_with_manual_fallback.md - chromium/src.git

blob: 188efd648376503c2fd71468f2c07e9891370753 [file] [log] [blame] [view]

Kent Tamura	59ffb02	2018-11-27 05:30:56	[diff] [blame]	1	# Web Tests with Manual Fallback
pwnall	59aadcb	2017-01-26 23:27:21	[diff] [blame]	2
				3	Some Blink features cannot be automatically tested using the Web Platform. Prime
				4	examples are the APIs that require
Mustaq Ahmed	3a40f9f2	2022-10-11 19:16:17	[diff] [blame]	5	[user activation](https://html.spec.whatwg.org/multipage/interaction.html#tracking-user-activation)
				6	(also known as _a user gesture_), such as [Fullscreen](https://developer.mozilla.org/en-US/docs/Web/API/Fullscreen_API).
pwnall	59aadcb	2017-01-26 23:27:21	[diff] [blame]	7	Automated tests for these Blink features must rely on special APIs, which are
				8	only exposed in testing environments, and are therefore not available in a
				9	normal browser session.
				10
				11	A popular pattern used in these tests is to rely on the user to perform some
				12	manual steps in order to run the test case in a normal browser session. These
				13	tests are effectively
Philip Jägenstedt	3a3d5b8	2018-05-31 15:25:35	[diff] [blame]	14	[manual tests](https://web-platform-tests.org/writing-tests/manual.html), with
pwnall	59aadcb	2017-01-26 23:27:21	[diff] [blame]	15	additional JavaScript code that automatically performs the desired manual steps,
				16	when loaded in an environment that exposes the needed testing APIs.
				17
				18	## Motivation
				19
Kent Tamura	59ffb02	2018-11-27 05:30:56	[diff] [blame]	20	Web tests that degrade to manual tests in the absence of testing APIs have
pwnall	59aadcb	2017-01-26 23:27:21	[diff] [blame]	21	the following benefits.
				22
				23	* The manual test component can be debugged in a normal browser session, using
				24	the rich [developer tools](https://developer.chrome.com/devtools). Tests
				25	without a manual fallback can only be debugged in the test runner.
				26	* The manual tests can run in other browsers, making it easy to check whether
				27	our behavior matches other browsers.
Kent Tamura	59ffb02	2018-11-27 05:30:56	[diff] [blame]	28	* The web tests can form the basis for manual tests that are contributed to
foolip	eda32ab	2017-02-16 19:21:58	[diff] [blame]	29	[web-platform-tests](./web_platform_tests.md).
pwnall	59aadcb	2017-01-26 23:27:21	[diff] [blame]	30
				31	Therefore, the desirability of adding a manual fallback to a test heavily
				32	depends on whether the feature under test is a Web Platform feature or a
				33	Blink-only feature, and on the developer's working style. The benefits above
				34	should be weighed against the added design effort needed to build a manual test,
				35	and the size and complexity introduced by the manual fallback.
				36
				37	## Development Tips
				38
Kent Tamura	59ffb02	2018-11-27 05:30:56	[diff] [blame]	39	A natural workflow for writing a web test that gracefully degrades to a
pwnall	59aadcb	2017-01-26 23:27:21	[diff] [blame]	40	manual test is to first develop the manual test in a browser, and then add code
				41	that feature-checks for testing APIs, and uses them to automate the test's
				42	manual steps.
				43
				44	Manual tests should minimize the chance of user error. This implies keeping the
				45	manual steps to a minimum, and having simple and clear instructions that
				46	describe all the configuration changes and user gestures that match the effect
				47	of the Blink-specific APIs used by the test.
				48
				49	## Example
				50
				51	Below is an example of a fairly minimal test that uses a Blink-Specific API
				52	(`window.eventSender`), and gracefully degrades to a manual test.
				53
				54	```html
				55	<!doctype html>
				56	<meta charset="utf-8">
				57	<title>DOM: Event.isTrusted for UI events</title>
				58	<link rel="help" href="https://dom.spec.whatwg.org/#dom-event-istrusted">
				59	<link rel="help" href="https://dom.spec.whatwg.org/#constructing-events">
				60	<meta name="assert"
				61	content="Event.isTrusted is true for events generated by user interaction">
				62	<script src="../../resources/testharness.js"></script>
				63	<script src="../../resources/testharnessreport.js"></script>
				64
				65	<p>Please click on the button below.</p>
				66	<button>Click Me!</button>
				67
				68	<script>
				69	'use strict';
				70
				71	setup({ explicit_timeout: true });
				72
				73	promise_test(() => {
				74	const button = document.querySelector('button');
				75	return new Promise((resolve, reject) => {
				76	const button = document.querySelector('button');
				77	button.addEventListener('click', (event) => {
				78	resolve(event);
				79	});
				80
				81	if (window.eventSender) {
				82	eventSender.mouseMoveTo(button.offsetLeft, button.offsetTop);
				83	eventSender.mouseDown();
				84	eventSender.mouseUp();
				85	}
				86	}).then((clickEvent) => {
				87	assert_true(clickEvent.isTrusted);
				88	});
				89
				90	}, 'Click generated by user interaction');
				91
				92	</script>
				93	```
				94
				95	The test exhibits the following desirable features:
				96
				97	* It has a second specification URL (`<link rel="help">`), because the paragraph
				98	that documents the tested feature (referenced by the primary URL) is not very
				99	informative on its own.
				100	* It links to the
				101	[WHATWG Living Standard](https://wiki.whatwg.org/wiki/FAQ#What_does_.22Living_Standard.22_mean.3F),
				102	rather than to a frozen version of the specification.
				103	* It contains clear instructions for manually triggering the test conditions.
				104	The test starts with a paragraph (`<p>`) that tells the tester exactly what to
				105	do, and the `<button>` that needs to be clicked is clearly labeled.
				106	* It disables the timeout mechanism built into `testharness.js` by calling
				107	`setup({ explicit_timeout: true });`
				108	* It checks for the presence of the Blink-specific testing APIs
				109	(`window.eventSender`) before invoking them. The test does not automatically
				110	fail when the APIs are not present.
				111	* It uses [Promises](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Promise)
				112	to separate the test setup from the assertions. This is particularly helpful
				113	for manual tests that depend on a sequence of events to occur, as Promises
				114	offer a composable way to express waiting for asynchronous events that avoids
				115	[callback hell](http://stackabuse.com/avoiding-callback-hell-in-node-js/).
				116
				117	Notice that the test is pretty heavy compared to a minimal JavaScript test that
				118	does not rely on testing APIs. Only use testing APIs when the desired testing
				119	conditions cannot be set up using Web Platform APIs.