Speedometer 2.0:
A Benchmark for Modern Web App Responsiveness

In 2014, the WebKit team at Apple released Speedometer 1.0, a benchmark for web app responsiveness. It simulates user interactions in web applications, using TodoMVC to orchestrate adding, completing, and removing todo items. Speedometer repeats these actions using DOM APIs that were extensively used in real-world applications. The performance of these kinds of operations depends on the speed of the JavaScript engine, DOM APIs, layout, CSS style resolution and other parts of the browser engine.

Browser engineers have been optimizing their engines using Speedometer as a proxy for real-world use of popular frameworks for a number of years. Originally, Speedometer included implementations of todo apps in six popular JavaScript frameworks and libraries in heavy use: Ember, Backbone, AngularJS, jQuery, Flight, and an early version of React. It also included vanilla JavaScript.

The web developer ecosystem has evolved significantly since Speedometer 1.0 was first released, as have the trends in what libraries, frameworks, and programming paradigms are used. Developers now commonly use transpilers, module bundlers, and recently-introduced frameworks when creating new sites. This is why, for the last year, engineers from WebKit and Chromium have been collaborating on a new version of Speedometer that better reflects the frameworks, tools, and patterns in wide use today.

Today, we are pleased to announce the Speedometer 2.0 benchmark. We hope this new version of Speedometer helps browser vendors optimize their browser engines for the modern Web.

Support for modern JavaScript frameworks and libraries

Over the last three years, a growing number of real-world sites have been written using React — a JavaScript library for authoring user interfaces. Derivatives such as Preact and Inferno have also gained popularity. Speedometer 2.0 includes web apps implemented using these libraries. It also includes an entry using React and Redux — a popular state management library.

Webpack and Rollup are popular JavaScript module bundlers frequently used with these libraries and Speedometer 2.0 includes output generated by these tools.

Ember.js, which featured in the original Speedometer, now has a dedicated tool to create new projects, and provides a more streamlined deployment process for authors. In addition, there were large changes to the core Ember framework over the years. To incorporate these changes in Ember.js itself and the way developers use Ember.js today, Speedometer 2.0 includes an implementation using the latest Ember, built using Ember CLI.

Another framework we observed gaining traction is Vue.js — a progressive solution aimed at being incrementally adoptable. Similar to Ember, Vue.js has prescriptive tooling for getting started and Speedometer 2.0 includes a Vue.js implementation built using the Vue CLI.

It’s of course true that not all real-world sites are being built using these solutions. Many are still deployed using libraries that were popular when Speedometer 1.0 was authored, which is one reason Speedometer 2.0 also includes updates to implementations written in AngularJS, Backbone.js, and Flight.

ES2015 JavaScript and Babel support

Speedometer 1.0 included a version of the todo app implemented with vanilla JavaScript — i.e. without using any libraries or frameworks. At the time, web developers primarily wrote their applications in the version of JavaScript known as ES5. Today, modern browsers have excellent support of ES2015 (also known as ES6), a more evolved version of JavaScript. Speedometer 2.0 now includes a todo app implemented using ES2015 features like classes, const, let, arrow functions, and template literals.

Although measuring vanilla JavaScript performance has high value, a growing number of developers today also use transpilers like Babel to transpile the latest versions of JavaScript code back to a version supporting all browsers they care about. To reflect this workflow, Speedometer 2.0 includes an ES2015 implementation that uses ES Modules and has ES5 output generated by Babel. In the future, as browsers gain full support for native ES Modules, we hope to evolve the benchmark to also track an implementation that isn’t bundled or translated.

TypeScript support

TypeScript is a typed superset of JavaScript that has been gaining traction in the web developer community. It offers types as a first-class syntax, generally fast compilation, and rich tooling for type-aware auto-completion and error highlighting during iteration.

Today, one of the largest users of TypeScript is Angular. To enable browsers to measure the kinds of output a TypeScript app might generate, Speedometer 2.0 includes an Angular implementation written in TypeScript, transpiled to ES5. We’re hopeful that browsers optimizing for this implementation will be able to offer the same wins as more frameworks introduce TypeScript support.

Future-facing: functional programming

The front-end developer community has been shifting in the direction of borrowing more patterns from functional programming. This has been demonstrated with the growth of interest in technologies like Elm and PureScript, both of which transpile down to JavaScript. To enable browsers to optimize for these patterns, Speedometer 2.0 includes implementations for both of these technologies.

Updates in score calculation

Speedometer 1.0 calculated a final score for Web App Responsiveness using the arithmetic mean of the run time needed to add, mark completed, and remove 100 todo items in each implementation in order to incentivize browser vendors to optimize the slowest framework or library. Unfortunately, this resulted in some implementation of a todo app getting 25× more weight compared to another as we added more libraries and frameworks to Speedometer 2.0. It became particularly problematic when we added back a debug build of Ember — it was more than 4× slower than the Ember production build. However, only a small fraction of websites deployed with Ember use debug builds.

In Speedometer 2.0, we’ve changed the score to be computed as the geometric mean against different implementations of the todo app. The final score is computed as the arithmetic mean of the geometric means computed for each iteration of the benchmark.

Future

Speedometer 2.0 has been an exciting collaboration between browser vendors. We would like to build on this collaboration in future iterations of the benchmark by working more closely with framework authors and the developer community to identify broadly-used patterns, frameworks, and tools for which browser engines could be optimized.