The web developer ecosystem has evolved significantly since Speedometer 1.0 was first released, as have the trends in what libraries, frameworks, and programming paradigms are used. Developers now commonly use transpilers, module bundlers, and recently-introduced frameworks when creating new sites. This is why, for the last year, engineers from WebKit and Chromium have been collaborating on a new version of Speedometer that better reflects the frameworks, tools, and patterns in wide use today.
Today, we are pleased to announce the Speedometer 2.0 benchmark. We hope this new version of Speedometer helps browser vendors optimize their browser engines for the modern Web.
Ember.js, which featured in the original Speedometer, now has a dedicated tool to create new projects, and provides a more streamlined deployment process for authors. In addition, there were large changes to the core Ember framework over the years. To incorporate these changes in Ember.js itself and the way developers use Ember.js today, Speedometer 2.0 includes an implementation using the latest Ember, built using Ember CLI.
Another framework we observed gaining traction is Vue.js — a progressive solution aimed at being incrementally adoptable. Similar to Ember, Vue.js has prescriptive tooling for getting started and Speedometer 2.0 includes a Vue.js implementation built using the Vue CLI.
It’s of course true that not all real-world sites are being built using these solutions. Many are still deployed using libraries that were popular when Speedometer 1.0 was authored, which is one reason Speedometer 2.0 also includes updates to implementations written in AngularJS, Backbone.js, and Flight.
let, arrow functions, and template literals.
Today, one of the largest users of TypeScript is Angular. To enable browsers to measure the kinds of output a TypeScript app might generate, Speedometer 2.0 includes an Angular implementation written in TypeScript, transpiled to ES5. We’re hopeful that browsers optimizing for this implementation will be able to offer the same wins as more frameworks introduce TypeScript support.
Future-facing: functional programming
Updates in score calculation
Speedometer 1.0 calculated a final score for Web App Responsiveness using the arithmetic mean of the run time needed to add, mark completed, and remove 100 todo items in each implementation in order to incentivize browser vendors to optimize the slowest framework or library. Unfortunately, this resulted in some implementation of a todo app getting 25× more weight compared to another as we added more libraries and frameworks to Speedometer 2.0. It became particularly problematic when we added back a debug build of Ember — it was more than 4× slower than the Ember production build. However, only a small fraction of websites deployed with Ember use debug builds.
In Speedometer 2.0, we’ve changed the score to be computed as the geometric mean against different implementations of the todo app. The final score is computed as the arithmetic mean of the geometric means computed for each iteration of the benchmark.
Speedometer 2.0 has been an exciting collaboration between browser vendors. We would like to build on this collaboration in future iterations of the benchmark by working more closely with framework authors and the developer community to identify broadly-used patterns, frameworks, and tools for which browser engines could be optimized.