Surfin' Safari

The Acid 3 Test

Posted by Dave Hyatt on Thursday, March 6th, 2008 at 12:29 pm

The Acid 3 Test has been officially released. The test has been in development for some time, with much of that development happening in the open.

The Acid 3 test is far more complex than the Acid 2 test. It covers a wider range of standards and consists of many more individual tests. Browsers have to render a sequence of boxes that display dynamically in a stairstep pattern. For every cluster of tests passed successfully, the boxes will fill in with a color, which signifies that all of the tests covered by that block have passed.

If you run Acid 3 on the shipping versions of current browsers (Firefox 2, Safari 3, Opera 9, IE7), you’ll see that they all score quite low. For example Safari 3 scores a 39/100. This percentage score is a bit misleading however. The situation with all four browser engines really isn’t that bad.

You can think of the Acid 3 test as consisting of 100 individual test suites. In order for a browser engine to claim one of these precious 100 points, it has to pass a whole battery of tests around a specific standard. In other words it’s like the browser is being asked to take 100 separate exams and score an A+ on each test in order to get any credit at all.

The reality is that all of the browsers are doing much better than their scores would have you believe, since the engines are often passing a majority of the subtests and experiencing minor failures that cost them the point for that section.

Shipping Safari scores a 39/100 with some significant rendering errors. We’ve been working hard since the test surfaced and are pleased to report that we’ve entered the “A” range on the test with a score of 90/100.

So what did we fix to gain so many points?

Bug 17064 has all the details, but here are the highlights.

Support for CSS3 Selectors
We added support for all of the remaining CSS3 selectors. These include selectors like nth-child, nth-of-type, last-child, last-of-type, etc. These selectors were already implemented in KHTML, and the KHTML developers had even kindly provided patches for us in the relevant WebKit bugs. Therefore it was a simple matter of taking those patches, updating them to the WebKit codebase, and then merging them in. A big thanks to the KHTML developers for their hard work in this area.

Parsing Bugs
WebKit had a number of minor parsing bugs that Acid 3 targeted. The boxes did not render properly because of an obscure parsing bug that the test exploited (thanks, Hixie). In addition a number of other parsing bugs kept us from completely passing individual tests. We have updated our parser to be much closer to the HTML5-specified parsing rules.

WebKit has also never parsed DOCTYPEs before. I re-wrote WebKit’s DOCTYPE parsing to match the HTML5 specification, and so now if you put a DOCTYPE into your page it will be present in the DOM. In addition many bugs centered around proper mode resolution (quirks vs. strict) have now been fixed. You can document.write a DOCTYPE for example in a new document and have the correct mode be selected.

SVG
Acid3 has many SVG tests. We’ve been hard at work making these tests pass. In particular SVG font support and other aspects of the SVG DOM have been tested. Many of the remaining 10 points are SVG failures. We’ll be working on SVG animation in order to pass the last few SVG tests.

DOM
Acid3 tests a lot of DOM level 2 features, like traversal and ranges. It particularly focuses on the “liveness” of objects, e.g., making sure everything updates properly when you dynamically change a document by adding/removing nodes. Most of our failures in this area had to do with not behaving properly in the presence of these dynamic changes (even though we tended to pass the more static tests).

Now that we’re closing in on 100%, we’ll be blogging about each fix as it happens, so that you can follow our progress from the blog.

16 Responses to “The Acid 3 Test”

  1. Roberto Says:

    Yay

    can’t wait for A+ ;)

  2. squareman Says:

    Congrats. When can we see this WK rendering engine in an official release? It took so very long to see some very old fixed bugs for Safari 2 make it into Safari 3.

    Also how about some overview on where it *doesn’t* pass?

  3. xfinite Says:

    Webkit proves once again it = TEH AWESOME. You guys rock.
    Any indication you can give about how many of these wonderful fixes will make it into Safari 3.1?

    Thanks for making the web a better place. :)

  4. Mark Rowe Says:

    Apple does not comment about it’s plans for future releases. Releases will happen when they happen.

  5. atppp Says:

    Safari is the best! You guys rock!

  6. Tycho Says:

    I saw that this was the case last night. The best part about Acid 3 is NOT that WebKit trunk is beating Internet Explorer so badly. It’s that Internet Explorer 5.5 beats 6.0 and 7.0!

    http://www.anomalousanomaly.com/2008/03/06/acid-3/

    Hehe. Silly Microsoft.

    Can’t wait to see the current WebKit code make it into a Safari release. Until then I’mma use the nightly builds far more often.

  7. Sebastian Says:

    I just downloaded and installed the nightly build… When I run the test, I score only 86/100 + some design glitches. I even created a new user, so I would have default settings, but it’s still not working. These tests fail:

    Test 2: reached expectation 13 when expecting expectation 12
    Test 11: when inserting <a> into Document with another child: wrong exception raised; code = 1
    Test 13: collapsed is wrong after deletion
    Test 23: no exception for createElementNS(‘null’, ‘:div’)
    Test 25: failed to raise exception
    Test 69: Undefined value
    Test 70: UTF-8 encoded XML document with invalid character did not have a well-formedness error
    Test 73: expected: 10, got: 1 – click event handler called the wrong number of times
    Test 74: getSVGDocument failed for referencing an svg document.
    Test 75: Value undefined (result of expression anim.beginElement) is not object.
    Test 76: expected: 0, got: 100 – Incorrect animVal value after svg animation.
    Test 77: expected: 3, got: 0 – getNumberOfChars returned incorrect string length.
    Test 78: expected: 90, got: 0 – getRotationOfChar(0) failed.
    Test 79: expected: 34, got: 33 – SVGSVGTextElement.getNumberOfChars() incorrect

    Am I missing something?

    Thanks
    Sebastian

  8. dngnta Says:

    I also got only 87/100 on the most recent nightly builds on Windows. However, I had overlooked an update to Safari 3.0.4 beta in Apple Software Update – once I let that one install, I started getting 90/100 as expected.

    So if you’re not getting 90/100 on the latest nightlies, check to see if the underlying Safari installation can be updated.

  9. Sebastian Says:

    Hm… My Safari reports it is Version 3.0.4 (5523.15). Is that not the most current version? Software Update doesn’t offer me an update, where would I get it from?

  10. deirdre Says:

    Sebastian, running a nightly of WebKit with Safari won’t change the Safari version. The version you have is the most current release of Safari.

    If you want to try a WebKit nightly with Safari, the easy way is to download the nightly from http://nightly.webkit.org/

  11. Jeffrey Says:

    Running Safari 3.0.4 (523.15) with webkit build 30881 gives me 88/100 with an empty cache. After refreshing the page I get 90/100. Test 77 and 78 seem to be failing with an empty cache.

    Test 77 failed: expected ’4776′ but got ’5550′ – getComputedTextLength failed.
    Test 78 failed: expected ’90′ but got ’0′ – getRotationOfChar(0) failed.

  12. Sebastian Says:

    I don’t know why my last comment does not appear here, but anyways, I said that I’m using the webkit application, not Safari to run the test.

    I just used Safari to check its version as dngnta suggested.

    Does anybody have another idea?

    Thanks!

  13. lhawkins Says:

    Like Jeffrey wrote, I too get different results when I refresh the page. In fact I get 90/100 with the current nightly on first load, but if I refresh I get 88, then 89, then 90 most of the time, although it does seem somewhat related to network latency. If there is a visible delay in the animation I usually get the 88, very time I get a 90 it seems “smoother”.

  14. Sebastian Says:

    Deirdre just asked me to add to the information above,

    I’m on PPC (PowerBook G4), and no matter how often I refresh, I am always stuck at 86. This doesn’t depend on how many Applications I’m running at the time.

    Clearing the cache also doesn’t make any difference.

  15. alienvenom Says:

    With the newest version of Safari released today (v3.1) I am only able to get 75/100 on Windows. :(

  16. solipsism Says:

    That is waht you should be getting with Safari 3.1 and its accompanying WebKit engine. It bests everyother stable AND beta engines, except for WebKit, of course. That is damn good.