Reduce reliance on JS in benchmark, focus more on HTML/CSS

[vprelovac]

Javascript is not the official language of the web documents, HTML is.

Currently the benchmark relies excessively on measuring JS Frameworks performance when the fact is for the most of the web, JS is not even necessary to function, let alone uses any of the JS frameworks used in the test.

Almost all browsers have the option to disable Javascript.

Wikipedia as well as the very site this is hosted on (Gthub) work without Javascript. Apple.com, Mozilla.org and Google.com all load and render without Javascript. Even when JS is used on a webpage, joint usage of all JS frameworks in the test currently, fade in comparison to the proportion vanilla JS is used.

This should be enough to argument that focusing on JavaScript frameworks (that come and go) is not the best way to be refelective of the real-world Web.

In fact the largest user of javascript on the web is ad-tech and tracking technology. Most of the web works fine without and JS enabled.

Even Web apps that use JS, are less than 1% of the web. There are mostly apps built by giant corporations, while the most of the web remain personal sites, blogs and static HTML. For this test to be the de-facto "browser" benchmark and achieve its goal of being " reflective of the real-world Web", it should measure hollistically across all things that browsers do. Spending time in JS heavy web apps certainly does not account for as much usage as the test in current form attributes (almost 100%!).

For this reason I would like to propose focusing on HTML/CSS performance more in the benchmark. If Javascript is to be measured, vanilla JS should be prioritized.

[bgrins]

There is generally interest in capturing more complex HTML/CSS workloads (including news and social feed type pages), but also methodology challenges to doing that within Speedometer. In particular we need to avoid network calls and atm we're only able to capture measurements synchronously (although the latter is being worked on).

I have some experiments with testing native HTML element interactions like toggling <details> (main...bgrins:Speedometer:mdn) and <dialog> elements which can cover some of these use cases neatly within the existing runner.

I've also played around with full and partial page rendering tests (https://bgrins.github.io/data-ui-tests/layout/) which also fit within the current runner although they don't perfectly represent the user experience with types of applications you are talking about since they don't include network & parsing within the measured time.

[vprelovac]

I am glad you are thinking more along these lines. Network calls can be avoided by caching content to be displayed or loading it from a static pre-included file.

There is so much more that browsers do then run heavy JS frameworks.

Here is just an example of types of benchmarks that can be included for the hollistic coverage of browsers performance.

HTML5 - For example https://testdrive-archive.azurewebsites.net/performance/chalkboard/
CSS3 https://testdrive-archive.azurewebsites.net/performance/mazesolver/default.html
WebAssembly https://pspdfkit.com/webassembly-benchmark/
WebGL https://luic.github.io/WebGL-Performance-Benchmark/

[camillobruni]

I like the CSS-heavy workload you suggested, and we're working towards adding more DOM and CSS heavy workloads for the current release.

Some comment regarding the other workloads:

• WebAssembly benchmarks are typically better suited for an upcoming JetStream release (they can be easily run standalone and typically don't stress DOM interactions / Web APIs too much)
• WebGL and CSS animations are more suited for MotionMark since you quickly get into framerate-based metrics. Speedometer is currently not meant to handle this

[vprelovac]

Thanks for acknowledgment. I listed those other options as a way to bring attention to the other parts of browser performance that can be measured vs strict focus on arbitrary JS frameworks that the test currently almost exclusively focuses on (which is in no shape or form reflective of the real-world Web unless we want to embrace a dystopian view of the "web" being whatever the current top 10 biggest websites are).

Including CSS heavy workload is certainly a great first step.

I also previously suggested this: #35

To me this is an interesting part of browser performance that is not measured anywhere (construction and traversing the DOM tree etc) where all curent browsers are miserably slow for some reason.

Thanks everyone for doing a great job with Speedometer 3!