Surfin' Safari

Web Inspector Console Improvements

Posted by Joseph Pecoraro on Wednesday, June 24th, 2015 at 2:00 pm

The console is an essential part of Web Inspector. Evaluating expressions in the quick console is one of the primary ways of interacting with the inspected page. Logs, errors, and warnings emitted from the page show up in the console and exploring or interacting with these objects is a given while debugging.

We recently improved both the Console and Object views in Web Inspector to make it more powerful and fun to use. Our main focus was getting quicker access to useful data and modernizing it to work better with the new changes in JavaScript.

Basics – Object Previews, Trees, and $n

Object previews allow you to see the first few properties without needing to expand them. You’ll notice that each evaluation provides you with a “$n” debugger variable to refer back to that object later. These special variables are known only to the tools, so you won’t be cluttering the page with temporary variables. $0 still exists and refers to the current selected node in the DOM Tree.

Object Preview

When expanded, the full object tree view cleanly separates properties and API. Again, we use object previews where possible to reveal more data at a glance. The icons for each property correspond to the type of the value for that property. For example, in the image below you’ll see properties with number values have a blue N icon, strings with a red S, functions a green F, etc. The icons give objects a visual pattern, and makes it easy to visually find a particular property or an unexpected change in the normal data an object holds.

Object Tree

Supporting New Types

Web Inspector has always had great support for inspecting certain built-in JavaScript types such as Arrays and DOM types like Nodes. Web Inspector has improved those views and now has comprehensive support for all of the built-in JavaScript types. This including the new ES6 types (Symbol, Set, Map, WeakSet, WeakMap, Promises, Classes, Iterators).

Array, Set, and Map object trees

WebKit’s tools are most useful when they show internal state of objects, known only to the engine, that is otherwise inaccessible. For example, showing the current status of Promises:

Promises

Or upcoming values of native Iterators:

Iterators

Other interesting cases are showing values in WeakSets and WeakMaps, or showing the original target function and bound arguments for bound functions.

API View

When expanding an object’s prototype you get a great API view showing what methods you can call on the object. The API view always provides parameter names for user functions and even provides curated versions for native functions. The API view makes it really convenient to lookup or discover the ways that you can interact with objects already available to you in the console.

Array API ViewLocal Storage Object Tree

As an added bonus, if you are working with ES6 Classes and log a class by its name or its constructor you immediately get the API view for that class.

Interactivity

Object trees are more interactive. Hover a property icon to see the property’s descriptor attributes. Hover the property name to see the exact path you can use to access the property. Getters can be invoked, and their results can be further explored.

Property Descriptor tooltipProperty Path tooltip

Context menus also provide more options. One of the most powerful features is that with any value in an Object tree you can use the context menu and select “Log Value” to re-log the value to the Console. This immediately creates a $n reference to the live object, letting you interact with it or easily reference it again later.

Console Messages

Console messages have also had a UI refresh, making logs, errors, warnings, and their location links stand out more:

Console Messages

Feedback

These enhancements are available to use in WebKit Nightly Builds. We would love to hear your feedback! You can send us quick feedback on Twitter (@JosephPecoraro, @xeenon), file a bug report, or even consider contributing your own enhancements!

Introduction to WebKit Content Blockers

Posted by Benjamin Poulain on Friday, June 12th, 2015 at 4:26 pm

Browser extensions have been a big part of modern browsers for a while now. With extensions, everyone can change their browser to their preferences.

Today, there are several models to extend browsers. Most extensions are written in JavaScript and loaded by the browser, following a model introduced by Mozilla over a decade ago. That model is also used with WebKit. It is the classical way to write extensions for OS X Safari, Web/Epiphany, and other browsers.

On OS X and iOS, there is also the concept of App Extensions with a different approach to security and performance. They are essentially little sandboxed applications that are launched on demand to extend some specific piece of functionality, known as an extension point.

The JavaScript extensions model has been great for many use cases, but there is one category of extensions where members of the WebKit project felt we should do better: the content blocking extensions. Such extensions are the most popular kind; they let users decide what should load and not load, who can track them, what should be visible on pages, etc.

The reason we are unhappy about the JavaScript-based content blocking extensions is they have significant performance drawbacks. The current model uses a lot of energy, reducing battery life, and increases page load time by adding latency for each resource. Certain kinds of extensions also reduce the runtime performance of webpages. Sometimes, they can allocate tremendous amounts of memory, which goes against our efforts to reduce WebKit’s memory footprint.

It is an area were we want to do better. We are working on new tools to enable content blocking at a fraction of the cost.

One new feature, we are developing allows describing content blocking rules in a structured format ahead-of-time, declaratively, rather than running extension-provided code at the moment a decision about blocking needs to be made. This model allows for WebKit to compile the ruleset into a format that’s very efficient to apply to loads and page content.

Content blockers in action

Before I dive into the details, let’s see what a declarative extension look like in the new format.

In essence, each content blocker extension is a list of rules that tells the engine how to act when loading a resource.

The rules are written in JSON format. For example, here is an extension with two rules:


    [
        {
            "trigger": {
                "url-filter": "evil-tracker.js"
            },
            "action": {
                "type": "block"
            }
        },
        {
            "trigger": {
                "url-filter": ".*",
                "resource-type": ["image", "style-sheet"]
                "unless-domain": ["reputable-content-server.com"]
            },
            "action": {
                "type": "block-cookies"
            }
        }
    ]

The first rule activates for any URL that contains the string “evil-tracker.js”. When the first rule is activated, the load is blocked.

The second rule activates for any resource loaded as an image or a style sheet on any domain except “reputable-content-server.com”. When the rule is activated, cookies are stripped from the request before sending it to the server.

The rules are passed to the engine by the browser. In iOS Safari, it is done through the native app extension mechanism. In OS X Safari, browser extensions can provide their rules through a new API. If you hack on WebKit, MiniBrowser also lets you load rule sets directly from the Debug menu.

Once the rules are passed to WebKit, they are compiled into an efficient bytecode format. The engine then executes this bytecode for each resource request, and uses the result to modify the request or inject CSS.

The bytecode is executed for each resource in the network subsystem. The goal is to reduce the latency between a request being created by the page and the request being actually dispatched over the network.

Content blocker format

The content blocker rules are passed in JSON format. The top level object is an array containing every rule that needs to be loaded.

Each rule of the content blocker is a dictionary with two parts: a trigger which activates the rule, and an action defining what to do when the rule is activated.

A typical rule set looks like this:


    [
        {
            "trigger": {
                …
            },
            "action": {
                …
            }
        },
        {
            "trigger": {
                …
            },
            "action": {
                …
            }
        }
    ]

The order of the rules is important. For every extension, the actions are applied in order. There is an action that skip all the rules that appears before the current one: “ignore-previous-rules”.

Let’s dive into the “trigger” and “action” objects.

Trigger definition

The “trigger” defines what properties activate a rule. When the rule is activated, its action is queued for execution. When all the triggers have been evaluated, the actions are applied in order.

Currently, the triggers are based on resource load information: the url and type of each resource, the domain of the document, and the relation of the resource to the document.

The valid fields in the trigger are:

  • “url-filter” (string, mandatory): matches the resource’s URL.
  • “url-filter-is-case-sensitive”: (boolean, optional): changes the “url-filter” case-sensitivity.
  • “resource-type”: (array of strings, optional): matches how the resource will be used.
  • “load-type”: (array of strings, optional): matches the relation to the main resource.
  • “if-domain”/”unless-domain” (array of strings, optional): matches the domain of the document.

The most important field, and the only mandatory one, is “url-filter”. In this field, you define a regular expression that will be evaluated against the URL of each resource. It is possible to match every URL by matching every character (e.g. “.*”) but in general it is better to be as precise as possible to avoid unforeseen side effects.

The syntax of the regular expression is a strict subset of JavaScript regular expressions. It is introduced later in this post.

It is possible to change the case-sensitivity of “url-filter” with the field “url-filter-is-case-sensitive”. By default, the matching is case-insensitive.

The optional field “resource-type” specifies the type of load to match. The content of this field is an array with all the types of load that can activate the trigger. The possible values are:

  • “document”
  • “image”
  • “style-sheet”
  • “script”
  • “font”
  • “raw” (any untyped load, like XMLHttpRequest)
  • “svg-document”
  • “media”
  • “popup”

Since the triggers are evaluated before the load starts, these types defines how the engine intends to use the resource, not necessarily the type of the resource itself (for example, <img src=”something.css”> is identified as an image.)

If “resource-type” is not specified, the default is to match all types of resources.

The field “load-type” defines the relation between the domain of the resource being loaded and the domain of the document. The two possible values are:

  • “first-party”
  • “third-party”

A “first-party” load is any load where the URL has the same security origin as the document. Every other case is “third-party”.

Finally, it is possible to make a trigger conditional on the URL of the main document. In this case, the rule only applies for a particular domain, or only outside of a particular domain.

The domain filters are “if-domain” and “unless-domain”, those fields are exclusive. In those fields, you can provide a domain filter for the main document.

It is important to be careful with the trigger to avoid rules being unexpectedly activated. Since it is impossible to test the entire web to validate a trigger, it is advised to be as specific as possible.

Action definition

The “action” part of the dictionary defines what the engine should do when a resource is matched by a trigger.

Currently, the action object has only 2 valid fields:

  • “type” (string, mandatory): defines what to do when the rule is activated.
  • “selector” (string, mandatory for the “css-display-none” type): defines a selector list to apply on the page.

There are 3 types of actions that limit resources: “block”, “block-cookies”, “css-display-none”. There is an additional type that does not have any impact on the resource but changes how the content extension behaves: “ignore-previous-rules”.

The action “block” is the most powerful one. It tells the engine to abort loading the resource. If the resource was cached, the cache is ignored and the load will still fail.

The action “block-cookies” changes the way the resource is requested over the network. Before sending the request to the server, all cookies are stripped from the header. Safari has its own privacy policy that applies on top of this rule. It is only possible to block cookies that would otherwise be accepted by the privacy policy, combining “block-cookies” and “ignore-previous-rules” still follows the browser’s privacy settings.

The action “css-display-none” acts on the CSS subsystem. It lets you hide elements of the page based on selectors. When this action is set, there should be a second entry named “selector” containing a selector list. Any element matching the selector list has its “display” property set to “none” which hides it.

Every selector supported by WebKit is supported for content extensions, including compound selectors and the new selectors from CSS Selectors Level 4. For example, the following action definition is valid:


    "action": {
        "type": "css-display-none",
        "selector": "#newsletter, :matches(.main-page, .article) .annoying-overlay"
    }

Finally, there is the action type “ignore-previous-rules”. All it does is ignore every rule before the current one if the trigger is activated. Note that it is not possible to ignore the rules of an other extension. Each extension is isolated from the others.

The Regular expression format

Triggers support filtering the URLs of each resource based on regular expression.

All strings in “url-filter” are interpreted as regular expressions. You have to be careful to escape regular expression control characters. Typically the dot appears in filters and needs to be escaped (for example, “energy-waster.com” should appear as “energy-waster\.com”.

The format is a strict subset of JavaScript regular expressions. Syntactically, everything supported by JavaScript is reserved but only a subset will be accepted by the parser. An unsupported expression results in a parse error.

The following features are supported:

  • Matching any character with “.”.
  • Matching ranges with the range syntax [a-b].
  • Quantifying expressions with “?”, “+” and “*”.
  • Groups with parenthesis.

It is possible to use the beginning of line (“^”) and end of line (“$”) marker but they are restricted to be the first and last character of the expression. For example, a pattern like “^bar$” is perfectly valid, while “(foo)?^bar$” causes a syntax error.

All URL matching is done against the canonical version of the URL. As such, you can expect the URL to be completely ASCII. The domain will already be punycode encoded. Both the scheme and domain are already lowercase. The resource part of the URL is already percent encoded.

Since the URL is known to be ASCII, the url-filter is also restricted to ASCII. Patterns with non-ASCII characters result in a parse error.

Privacy

We have been building these features with a focus on providing better control over privacy. We wanted to enable better privacy filters, and that is what has been driving the feature set that exists today.

There is a whole universe of features that can take advantage of the content blocker API, around privacy or better user experience. We would love to hear your feedback about what works well, what needs improvement, and what is missing.

A major benefit of the declarative content blocking extension model is that the extension does not see the URLs of pages and resources the user browsed to or had a page request. WebKit itself does not keep track of what rules have been executed on which URLs; we do not track you by design.

Everything has been developed in the open; everyone is welcome to audit and improve the code. The main part of content blockers lives in Source/WebCore/contentextensions.

Performance advice

A big focus of this feature is performance. We are trying to have good scalability with minimal performance impact.

If the rule compiler detects that a set of rules would negatively impact user experience, it refuses to load them and returns an error.

There are parameters in your extensions that do impact performance. In this section, I will give some general rules to get good performance. There are a few big themes to maximize performance:

  • Avoid quantifiers (“*”, “+”, “?”) in “url-filter” as much as possible.
  • CSS rules are best defined before any “ignore-previous-rules”
  • Make the trigger as specific as possible.
  • Group rules with similar actions.

Minimize quantifiers in regular expressions

Avoiding quantifiers helps us optimize the triggers in the backend. Quantifiers are useful but they increase the matching possibilities which tends to reduce performance. We have found that many existing privacy extensions sometimes use quantifiers excessively which tends to be costly.

One particularly bad case is quantifier appearing in the middle of a string. Cases like:


    foo.*bar

tends to be slower than “foo” and “bar” separately. Using too many of them can cause the rule set to be rejected.

One exception to that rule is common prefixes. For example, if you have many rules such as those:


    https?://user-tracker.com
    https?://we-follow-you.com
    https?://etc.com

The rules are grouped by the prefix “https?://, and it only counts as one rule with quantifiers.

Have CSS rules before “ignore-previous-rules”

When compiling the rules, we group the CSS rules whenever we determine they will be used together. For example, if a set of rules applies on every page (by using the filter “.*”), a special stylesheet is prepared for them to be ready to use them instantly when the page has loaded.

When a “ignore-previous-rules” appears, it forces the compiler to break the stylesheet since the rules appearing before an action “ignore-previous-rules” are all dismissed when the action is activated.

Use specific triggers

Good triggers try to exclude everything that could be activated by accident. Regular expressions should be as specific as possible to achieve your desired goal. Specifying the resource types if possible, and using domain filter if the rule should only apply on certain domains.

Having specific rules is important to avoid changing pages inadvertently, but it is also useful for performance. Having few actions to execute is a good idea.

Grouping rules with the same actions

Finally, grouping rules with similar actions can simplify the execution. Ideally, your extension will have all the rules blocking loads, followed by all the rules blocking cookies, etc.

Since rules are evaluated in order, it is useful to have them grouped together since matching a trigger means all the following rules of the same action can be skipped.

For example, if all the “block” rules are together, as soon as the first one is activated, all the following block rules are skipped since they have the same actions. The trigger evaluation continues on the first following rule with a different action.

We want your feedback

We have been developing these capabilities with the goal of achieving better privacy without incurring an unreasonable performance cost. We have intentionally limited ourselves to just a few features that could be improved.

If you start building content blocker extensions, it would help us if you send us your JSON files. Having many different use cases will help us optimize the code better.

For short questions, you can contact me on Twitter. For longer questions, you can email webkit-help or file bug report. If you’re interested in hacking on the code, feel free to ask me, Alex Christensen, Brady Eidson and Sam Weinig for help.

For questions about Safari’s adoption of these APIs, contact Brian Weinstein or Jon Davis.

Building WebKit for iOS Simulator

Posted by Daniel Bates on Tuesday, January 27th, 2015 at 9:01 am

I am proud to formally announce that you can now build and run top-of-tree WebKit for iOS in the iOS Simulator. We have updated the pages on webkit.org with details on building for iOS Simulator. For your convenience, I have summarized the steps to get you up and running below:

  1. Install Xcode 6.1.1.
  2. Get the Code.
  3. Enable Xcode to build command line tools by running sudo Tools/Scripts/configure-xcode-for-ios-development in the Terminal.
  4. Build WebKit for iOS Simulator by running Tools/Scripts/build-webkit --ios-simulator.
  5. Launch Safari in the iOS Simulator with the WebKit version you built by running Tools/Scripts/run-safari --ios-simulator.

Early Warning System (EWS) bots for iOS are running to help contributors catch build breakage before a patch is landed. The EWS bots build 32-bit iOS WebKit for ARMv7 hardware. We chose to build this configuration because it will most likely reveal build errors that differ from the configuration built by the existing Mac EWS bots.

We are working to bring up support for running layout tests, build and test build bots and additional iOS EWS configurations to help contributors notice build issues and regressions in WebKit for iOS.

We have always encouraged you to file all WebKit bugs that you find. Since upstreaming iOS WebKit to open source in early 2014, we have tracked iOS WebKit bugs in bugs.webkit.org. Now that you are able to build and run iOS WebKit yourself, we invite you to help fix them!

Using ARIA 1.0 and the WebKit Accessibility Node Inspector

Posted by James Craig on Tuesday, June 10th, 2014 at 10:40 am

On the heels of the 25th birthday of the Web, WAI-ARIA 1.0—the specification for Accessible Rich Internet Applications—is a W3C Recommendation, thanks in part to WebKit’s implementation. Most major web applications use ARIA and related techniques to improve accessibility and general usability.

Many web developers are familiar with the simple parts of ARIA, such as retrofitting roles in legacy or otherwise non-semantic uses of HTML like <div role="button" tabindex="0">, but ARIA has other equally important uses:

  • Improving languages like SVG where no accessibility semantics exist natively.
  • Augmenting technologies like EPUB that build on existing HTML semantics.
  • Allowing accessibility in native implementations, like the sub-DOM controls of <video> elements.
  • Supporting accessibility and full keyboard access when HTML is insufficient, such as with data grids, tree controls, or application dialogs.
  • Enabling accessible solutions where there is no equivalent semantic or functionality. For example, HTML has no concept similar to “live” regions.

More on these topics below, including how to diagnose and debug problems using new accessibility inspection features in the WebKit Web Inspector.

Example 1: ARIA in an SVG Map of Africa

The Scalable Vector Graphics (SVG) language does not include semantics to indicate what type of content is represented, such as a chart, illustration, or interactive application control. However, ARIA roles and attributes can be used in SVG today for both raster- and vector-based images, and the SVG Working Group recently adopted ARIA officially into SVG 2.

The following video shows VoiceOver’s touchscreen navigation of an accessible map. It uses a simple role="img" on each country path, and an aria-labelledby relationship to associate that country path with the text label. After watching the video, view the source of the test case SVG map of Africa to see how it works.

Closed captioned video showing VoiceOver on iOS with touch screen navigation of African map in SVG+ARIA.

Prior to WebKit’s implementation of ARIA in SVG, the best opportunity for a blind individual to experience spatial data like charts and maps was to print expensive tactile graphics on swell paper or with a modified braille embosser. Along with WebKit’s first native implementation of accessible MathML, accessible graphics represent new possibilities in the category of study collectively referred to as STEM: science, technology, engineering, and math.

Note: The test case SVG map of Africa is based on an original by Peter Collingridge, with accessibility added for the demo.

Introducing the Accessibility Node Inspector

Recent nightly builds of WebKit include a new accessibility section in the node properties of the Web Inspector. Select any DOM element to see relevant accessibility information, such as the computed role.

The properties and relationships displayed come from WebCore. Accessibility children and parent nodes cannot be detected through a JavaScript interface and are not a one-to-one mapping to the DOM, so these relationships have not previously been available to web developers. Many other accessibility properties are likewise not detectable through the rendered DOM or a JavaScript interface.

We’ll use the WebKit Accessibility Node Inspector to diagnose and inspect the examples below.

Complex ARIA Widget Examples

Many of the original features of ARIA (such as dialogs, landmarks, and menus) have been adopted into recent versions of HTML as elements. However, there are interaction patterns, common on the Web since the 1990s, that still have no native support or unreliable rendering support in HTML, such as date pickers, combo boxes, tab sets, data grids, tree controls, and more. Web developers must render all these controls themselves, using custom code or a JavaScript framework.

Making ARIA widgets can be challenging. The examples below illustrate some of the trickier bits of implementation. Debugging these controls is made easier by observing accessibility properties in the Web Inspector.

Example 2: Selectable List Box with Full Keyboard Support Using Native Focus Events

This demo was created in 2009 for Apple’s World Wide Developer Conference (WWDC) and uses the “roaming tabindex” technique on a custom list box.

Assistive technologies do not change the DOM, so there’s no hidden magic happening. JavaScript running in the page uses standard event handling and DOM interfaces like setAttribute() and focus(). View the source or step through in the WebKit debugger for a deeper understanding.

Closed captioned video showing an accessible “managed focus” list box demo

For a full explanation of the techniques and test case roaming tabindex demo used in the video, see WWDC 2009: Improving Accessibility in Web Applications.

Example 3: Combo Box with a “Status” Live Region

During the life cycle of a web application, there may be multiple points of user interest. In the list box example, the web application moves focus to the updated item, but moving focus is not always appropriate. In a combo box, keyboard focus should remain on the text field so the user can continue typing. The selected item in the related drop-down list is conveyed to the API when the selection changes, allowing a a screen reader to speak updates to both elements. Some combo boxes have an additional status display to indicate the total number of results. In this demo, we’ll use an ARIA “live region” for the status updates.

As with the previous example, there’s no hidden magic occurring in the DOM. JavaScript running in the page uses standard event handling and DOM interfaces like setAttribute(). View the source or step through in the WebKit debugger for a deeper understanding of the techniques.

Closed captioned video showing an accessible combo box demo with live region support

As this combo box demo shows, the ability for an assistive technology to simultaneously follow and report changes to multiple points of user interest was never possible in web applications prior to ARIA.

Major Contributors to WebKit’s ARIA Implementation

WebKit’s implementation of ARIA played a significant part in the ARIA 1.0 Recommendation milestone, and many individuals collaborated on the work.

The initial implementation was completed in 2008 by Alice Liu Barraclough and Beth Dakin. Much of the remaining ARIA implementation in WebCore, as well as the Mac and iOS platform mapping, was completed by Chris Fleizach. Sam White made major improvements to WebKit’s accessibility architecture. Jon Honeycutt, Brent Fulgham, Dominic Mazzoni, Mario Sánchez Prada, and others completed the platform mapping to the Windows and Linux accessibility APIs. Credit for the ARIA test harness and WebKit test results goes to Michael Cooper, Jon Gunderson, Mona Heath, Joseph Scheuhammer, Rich Schwerdtfeger, and others. The full list of working group acknowledgements is available in the ARIA spec.

The Web is a more enabling resource for everyone because of the efforts of these individuals. Thank you!

Future Direction of ARIA

ARIA 1.0 has much room for improvement, but it’s an incredibly important step toward ensuring equal access, opportunity, and usability of the web platform.

Future work on ARIA will cover additional semantics for HTML, SVG, and EPUB, and some of the work proposed includes non-declarative JavaScript accessibility API support for custom view technologies like WebGL and Canvas. There is also work to be done for rich text editing that is beyond the capability of contenteditable, such as the custom display and input-proxied views that are used on application suites like Google Docs and iWork for iCloud.

A Call to Action

Many of the widget libraries available in JavaScript frameworks do NOT include support for accessibility and keyboard behavior. If you are a front-end engineer, you have an opportunity to change this situation.

When you contribute to JavaScript UI libraries, include support for accessibility. Test your code for accessibility and keyboard behavior using focus() where appropriate. Detect and update your web app state based on user focus events. Don’t just style the CSS of controls to look focused; use real keyboard focus.

The amount of effort it takes to add and test for accessibility is well worth the fit-and-finish it will bring to your web app. You’ll improve the experience for all users.

Additional Video Resources

Each of these videos is about an hour in length. They cover ARIA and related techniques in detail.

Introducing the JetStream benchmark suite

Posted by Filip Pizlo on Monday, June 2nd, 2014 at 12:42 pm

Today we are introducing a new WebKit JavaScript benchmark test suite, JetStream. JetStream codifies what our de facto process has been — to combine latency and throughput benchmarks with roughly equal weighting, and capturing both metrics of traditional JavaScript programming styles as well as new JavaScript-based technologies that have captured our imaginations. Scores on JetStream are a good indicator of the performance users would see in advanced web applications like games.

Optimizing the performance of our JavaScript engine is a high priority for the WebKit project. Examples of some of the improvements we introduced in the last year include concurrent compilation, generational GC, and the FTL JIT. Engineering such improvements requires focus: we try to prioritize high-impact projects over building and maintaining complex optimizations that have smaller benefits. Thus, we motivate performance work with benchmarks that illustrate the kinds of workloads that WebKit users will likely encounter. This philosophy of benchmark-driven development has long been part of WebKit.

The previous state of JavaScript benchmarking

As we made enhancements to the WebKit JavaScript engine, we found that no single benchmark suite was entirely representative of the scenarios that we wanted to improve. We like that JSBench measures the performance of JavaScript code on popular websites, but WebKit already does very well on this benchmark. We like SunSpider for its coverage of commonly-used language constructs and for the fact that its running time is representative of the running time of code on the web, but it falls short for measuring peak throughput. We like Octane, but it skews too far in the other direction: it’s useful for determining our engine’s peak throughput but isn’t sensitive enough to the performance you’d be most likely to see on typical web workloads. It also downplays novel JavaScript technologies like asm.js; only one of Octane’s 15 benchmarks was an asm.js test, and this test ignores floating point performance.

Finding good asm.js benchmarks is difficult. Even though Emscripten is gaining mindshare, its tests are long-running and until recently, lacked a web harness. So we built our own asm.js benchmarks by using tests from the LLVM test suite. These C and C++ tests are used by LLVM developers to track performance improvements of the clang/LLVM compiler stack. Emscripten itself uses LLVM to generate JavaScript code. This makes the LLVM test suite particularly appropriate for testing how well a JavaScript engine handles native code. Another benefit of our new tests is that they are much quicker to run than the Emscripten test suite.

Having good JavaScript benchmarks allows us to confidently pursue ambitious improvements to WebKit. For example, SunSpider guided our concurrent compilation work, while the asm.js tests and Octane’s throughput tests motivated our work on the FTL JIT. But allowing our testing to be based on a hodgepodge of these different benchmark suites has become impractical. It’s difficult to tell contributors what they should be testing if there is no unified test suite that can tell them if their change had the desired effect on performance. We want one test suite that can report one score in the end, and we want this one score to be representative of WebKit’s future direction.

Designing the new JetStream benchmark suite

Different WebKit components require different approaches to measuring performance. For example, for DOM performance, we just introduced the Speedometer benchmark. In some cases, the obvious approach works pretty well: for example, many layout and rendering optimizations can be driven by measuring page load time on representative web pages. But measuring the performance of a programming language implementation requires more subtlety. We want to increase the benchmarks’ sensitivity to core engine improvements, but not so much so that we lose perspective on how those engine improvements play out in real web sites. We want to minimize the opportunities for system noise to throw off our measurements, but anytime a workload is inherently prone to noise, we want a benchmark to show this. We want our benchmarks to represent a high-fidelity approximation of the workloads that WebKit users are likely to care about.

JetStream combines a variety of JavaScript benchmarks, covering a variety of advanced workloads and programming techniques, and reports a single score that balances them using a geometric mean. Each test is run three times and scores are reported with 95% confidence intervals. Each benchmark measures a distinct workload, and no single optimization technique is sufficient to speed up all benchmarks. Some benchmarks demonstrate tradeoffs, and aggressive or specialized optimization for one benchmark might make another benchmark slower. Demonstrating trade-offs is crucial for our work. As discussed in my previous post about our new JIT compiler, WebKit tries to dynamically adapt to workload using different execution tiers. But this is never perfect. For example, while our new FTL JIT compiler gives us fantastic speed-ups on peak throughput tests, it does cause slight regressions in some ramp-up tests. New optimizations for advanced language runtimes often run into such trade-offs, and our goal with JetStream is to have a benchmark that informs us about the trade-offs that we are making.

JetStream includes benchmarks from the SunSpider 1.0.2 and Octane 2 JavaScript benchmark suites. It also includes benchmarks from the LLVM compiler open source project, compiled to JavaScript using Emscripten 1.13. It also includes a benchmark based on the Apache Harmony open source project’s HashMap, hand-translated to JavaScript. More information about the benchmarks included in JetStream is available on the JetStream In Depth page.

We’re excited to be introducing this new benchmark. To run it, simply visit browserbench.org/JetStream. You can file bugs against the benchmark using WebKit’s bug management system under the Tools/Tests component.