Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Define the scope of web-platform-tests #215

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Define the scope of web-platform-tests #215

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
60b04c5
Add a quick and dirty draft of what might be the de-facto scope of WPT
gsnedders Dec 6, 2024
4e73ec1
Update scope-of-wpt.md
gsnedders Dec 6, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 60 additions & 0 deletions rfcs/scope-of-wpt.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
# RFC 215: Defining the Scope of web-platform-tests

### Summary

Define the scope of web-platform-tests to specifications which might reasonably be called "web specifications"
and which are intended to be implemented by web browsers.

Additionally, define what "tentative" might reasonably be used for.

This is intended to codify existing practice,
rather than represent any change.


### Details

Historically, what's been in scope for web-platform-tests has been somewhat nebulous,
and largely based on a common understanding of participants,
especially those on the Core Team.
We should put this on surer footing,
and make it easier for others to understand what is and isn't reasonable to add.

The primary aim of web-platform-tests is to test
[web specifications](https://github.com/w3c/browser-specs/blob/11a71b738f5e41f9239fdcd2074153388c8c6b8b/README.md#spec-selection-criteria)
[intended to be implemented by browsers](https://github.com/w3c/browser-specs/blob/11a71b738f5e41f9239fdcd2074153388c8c6b8b/README.md#categories),
both RFC 2119 "must" and RFC 2119 "should"
implementation requirements.

A specification does not need to have cross-vendor support for its tests to be included in web-platform-tests.
(XXX: Should we set a bar of one-vendor support or are we okay with zero-vendor support?)

Additionally, other features can be added as
[tentative](https://web-platform-tests.org/writing-tests/file-names.html#:~:text=.tentative,-%3A%20)
tests:

* Web browser behavior currently being explored via an
[explainer](https://tag.w3.org/explainers/),
but without a specification yet written.
(XXX: require prototyping to have started?)

* Historic but unspecified features in web browsers,
especially where major browsers are interoperable,
where there exists a long-term intention to specify them.
(Note: this excludes features which have been deliberately removed from specifications.
These are explicitly out of scope.)
Comment on lines +31 to +44
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This arguably doesn't actually match up with what the documentation currently says:

Indicates that a test makes assertions not yet required by any specification, or in contradiction to some specification. This is useful when implementation experience is needed to inform the specification. It should be apparent in context why the test is tentative and what needs to be resolved to make it non-tentative.

The "in contradiction to some specification" is kinda troubling — this implies that if some spec change is made, with no expectation of implementations changing behaviour any time soon, you could change the existing tests to be tentative and add new non-tentative tests for the new behaviour. IMO, that should be against policy — anything that is tentative should be on a path to becoming non-tentative. We should probably be explicit in the scope of what can be tentative that everything which is tentative should be on a path to becoming non-tentative.

I don't think the two things listed here are a complete list of cases; we also have cases:

The other thing that stands out is:

% rg --glob '*tentative*' 'https?://((bugs\.webkit\.org|bugs\.chromium\.org|crbug\.com|issues\.chromium\.org)/|github\.com/.*/(issue|pull))' --files-without-match | wc -l
    2078

Which shows it's very often not obvious "why the test is tentative and what needs to be resolved to make it non-tentative", though that is really a separate issue. (And looking at a few random tests, the commit history often doesn't make it obvious either.)


Features which are optional
(in RFC 2119 terms:
where the conformance criteria is
"may" or "may not")
may be included as
[optional tests](https://web-platform-tests.org/writing-tests/file-names.html#:~:text=.optional,-%3A%20)
or
[optional subtests](https://web-platform-tests.org/writing-tests/testharness-api.html#optional-features).


### Risks

We over-constrain what is allowed in web-platform-tests,
potentially raising the bar to change what is allowed to "submit a new RFC",
leading browser vendors to reduce what they submit to the project.