Updated on 23. June 2014
Last weekend I watched a video about Java8 and Nashorn. This is a much faster successor to Rhino by Oracle. They claimed to have passing all of more than 11000 tests of es5 spec, even though no browser vendor made compatibility to so scrupulous degree. The presentation was from late 2012, hence I decided to check, how things are going for now.
You can run them by yourself here, but I already did that for most popular browsers and even not popular ones. And moreover, I found that tests often fail not because of the feature in description is implemented incorrectly. Simpler, some tests are not independent. Extrapolating that to all tests would be an survival bias. Hence I'm not saying all tests are badly written. But if one wants to interpret the fact of test fail, reading description is not enough. On the bad note, though, a pair of tests are out of tune with specification. They expect some certain behavior, when it is clearly stated that it may be implementation-dependent.
Here's a hg repo viewer, where separate tests could be found.
TL;DR
If you don't want to go deep into details about different tests and my rants about their quality, here's a list what you'd better not to do from es5:
- test your code in different browsers, even if you're using bulletproof library like jQuery - these tests themselves couldn't be run in IE8 due to bug in UI
- in Firefox one cannot change numerical properties on
window
- in Firefox
[[Writable]], [[Configurable]] and [[Enumerable]] are incorrectly set to false when not provided in re-defining properties on window - just pass them for every defineProperty call.
- do not rely on features of strict mode. For example, sometimes primitives used as
this are still wrapped into objects - in special cases or in browsers (IE9) which, do not support strict mode at all. I personally faced a bug emerged from combination of jQuery.each on array of primitives and using strict equality for this.
Using strict mode as a helper for early problem detection is encouraged, though.
- do not rely exclusively on feature presence - some implementations are rather buggy - e.g.
defineProperty in Safari 5
- IE before 11th use slightly different representation for
Date.toString, hence do not rely on exact values.
- some old browsers still parse strings as octal numbers, starting from zero - use
parseInt(n, 10) or Number(n) if you're sure there're no trailing characters.
String.prototype.localeCompare returns incorrect results for non-canonical Unicode representations in many browsers- do no not rely on assumption that code will fail at some certain point with some certain error. This is mostly OK for tests, though.
- hardly one would use that in real code, but messing with numeric properties on
Array.prototype is not only difficult to understand, but is also vulnerable to subtle differences in implementations
Detailed analysis of failed tests
Listing full test names could be too "noisy" for visual perception, hence I compacted several names with syntax similar to imports in es6. E.g.: "1.1.2, 1.1.3" is written as "1.1.{2,3}".
First of all I need to mention, that six Date tests do not fail if and only if you live California or elsewhere near. Relax, it is neither discrimination, nor location tracking - just OS setting, timezone should be UTC-8.
15.5.4.9_CE
Tests that String.prototype.localeCompare returns 0 when comparing Strings that are considered canonically equivalent by the Unicode standard
All browsers except Chrome and Firefox 29 beta fails that test.
Interesting enough, strings from test case ("\u1111\u1171\u11B6" vs "\ud4db") are even displayed differently in FF29- (and strangely Chrome33-34): the first string looks like linear Hangul, whilst the second one is normal Jamo:
퓛 vs 퓛
OK, mainstream browsers go first in some arbitrary order.
Chrome 32+
11.2.3-3_3
Call arguments are not evaluated before the check is made to see if the object is actually callable (undefined member)
I.e. in expression ({}).b(f()); engine should call f() before .b is evaluated and detected as non-callable, whilst for ({}).a.b(f()); browser should not call f(). In the second case Chrome incorrectly calls f() as well.
S15.8.2.8_A6
Checking if Math.exp is approximately equals to its mathematical values on the set of 64 argument values; all the sample values is calculated with LibC
Chrome35 have similar bugs in 3 more tests.
The relative error is 2-50, while precision of IEEE754 doubles used in javascript, is 52 bits. So this error hardly matters, unless you're doing a REAL SCIENCE.
Moreover, specification [15.8.2] doesn't impose any certain requirements on computation accuracy for math functions:
The behaviour of the functions acos, asin, atan, atan2, cos, exp, log, pow, sin, sqrt, and tan is not precisely specified here except to require specific results for certain argument values that represent boundary cases of interest. For other argument values, these functions are intended to compute approximations to the results of familiar mathematical functions, but some latitude is allowed in the choice of approximation algorithms.
Test itself is rather arguable. Are those approximations enough or not? Even if this is a problem, it's not a big one.
Safari 7 (MacOS/iOS)
10.2.1.1.3-4-{16,18}-s
TypeError should be thrown when changing global.NaN or global.undefined in strict mode.
Safari 6 (iOS)
Surprisingly it doesn't fail the previous test.
S10.4.2_A1.{1,2}_T{1-11}
eval within global execution context - scopes are not chained properly
15.5.4.9_3
Tests that String.prototype.localeCompare treats a missing "that" argument, undefined, and "undefined" as equivalent.
str.localeCompare(), str.localeCompare(undefined) and str.localeCompare("undefined") should be the same for any possible value of str.
Firefox 27-33
10.4.3-1-{104,106}
Strict mode should not ToObject thisArg if not an object.
In es3 and non-strict es5, primitive passed as this, are wrapped into object (String, Number or Boolean). In strict mode primitives should not be unwrapped. This works in general wherever strict mode is enabled and supported.
In getter, defined on Object.prototype and used on primitive value (via boxing), this still should be a primitive. And this is where Firefox fails.
Actually, if you write cross-browser code, you shouldn't rely on aforementioned feature of strict mode.
15.11.4.4-8-2
Error.prototype.toString return empty string when 'name' is empty string and 'msg' is undefined
FF returns "Error" if name is empty. Not a serious bug, again.
Numerous bugs with Object.defineProperty.
Sometimes (6 from 15.2.3.6-4-{292-296}) checks interfere with implementation details of mapping between arguments object and actual named arguments like this:
(function (a) {
Object.defineProperty(arguments, '0', { value: 2 });
return a;
}(1));
BTW in strict mode all browsers demonstrate the same behavior as Firefox.
-354-4, -354-13, -360-3, -360-7, -531-4, -531-13, -538-3, -538-7 and 15.2.3.7-6-a-24
In these cases there's really an error. All 3 properties: writable, enumerable and configurable become false on re-definition if they're not provided. This is detected on window only and doesn't happen for normal objects. -531-17 and -538-7 fails even earlier due to inability to redefine numerical properties on window.
15.4.4.{14-22}
Problems with Array.prototype methods applied to global object.
The same reason as just mentioned - it fails because one cannot declare numerical keys on window - they continue to be undefined.
Actually, every method of Array.prototype has a NOTE like:
The reduce function is intentionally generic; it does not require that its this value be an Array object. Therefore it can be transferred to other kinds of objects for use as a method. Whether the reduce function can be applied successfully to a host object is implementation-dependent.
I.e. spec doesn't require ability to successfully apply those methods to host objects! So tests are just incorrect.
S15.4.4.4_A3_T1
[[Prototype]] of Array instance is Array.prototype, [[Prototype] of Array.prototype is Object.prototype
This test fails due to implementation details of concat (not own property remains not own in Firefox).
15.4.4.4-5-c-i-1
Array.prototype.concat will concat an Array when index property (read-only) exists in Array.prototype (Step 5.c.i)
It fails because Array.prototype.concat.call(1)[0] results in number primitive, and not Number object. The check of copy's properties existence actually passes.
IE11
The latest 11th version fails only one test about localeCompare
IE10
IE11 in IE10 emulation mode shows the same behavior.
It fails one more test, which is rather funny:
S15.5.4.7_A1_T11
Fails for eastern hemisphere (UTC+N timezones), cause IE does not display leading zeroes for dates: new Date(0).toString() → "Thu Jan 1 04:00:00 UTC+0400 1970"
Yeah, it sounds so arcane, but check in code is really like that:
(date.indexOf(date.getTimezoneOffset()>0 ? '31' : '01')) === 8
One way or another, it reveals a piece of non-standard behavior of IE.
IE9
IE9 fails significantly more tests additionally to aforementioned two.
And true IE9 shows results somewhat different from higher versions in corresponding emulation mode (surprise-surprise). The only good point about that is that both emulation modes (IE11@9 and IE10@9) works the same.
Of course, it also can't pass 500+ tests about strict mode. I didn't even check 'em all - just filtered out.
8.12.9-9-c-i_{1,2} (passes in emulation mode)
Redefine a configurable accessor property to be a data property on a non-extensible object
with preventExtensions.
10.6-13-a-2 (passes in emulation mode)
A direct call to arguments.callee.caller should work
(function (called) {
function test1(flag) { if (!flag) test2(); else called = true; }
function test2() { arguments.callee.caller(true); }
test1();
return called;
}(false));
Rather tricky and strange.
11.2.3-3_{1-2,5-8}
Call arguments are evaluated before the check is made to see if the object is actually callable.
It fails differently than in Chrome.
15.11.4.{2,3}-1
Error.prototype.{name,message} are not enumerable
15.2.4.4-{1,2}
Using Object.prototype.valueOf for non-object values
Expected result: primitive value should be wrapped into corresponding object.
Workaround: if you need to wrap primitive into object, Object(value) seems to be working properly everywhere.
S15.2.4.4_A{12-15}
Object.prototype.valueOf invoked with null/undefined should throw an error.
15.4.4.4-5-c-i-1
Array.prototype.concat will concat an Array when index property (read-only) exists in Array.prototype (Step 5.c.i)
The same as in Firefox.
15.5.4.11-1 (passes in emulation mode)
'this' object used by the replaceValue function of a String.prototype.replace invocation
S15.5.4.14_A1_T{6-9}, ...A2_T7
Split should not coerce its argument to string "undefined" if its argument is undefined.
15.9.5.40_1
Date.prototype.setFullYear - Date.prototype is itself an instance of Date
Test just calls Date.prototype.setFullYear(2012); and expects 2012 to be read back. Hardly you'll need that in practice, but this test fails only in IE9. All other higher and lower versions aren't affected.
Old non-IEs
They are not widespread at all for now, hence only short overviews for each. Nonetheless, number of such client will not drop abruptly in foresee future - because of vendor-lock or similar situation.
Safari 5 (Win)
- even though, it claims support of defining properties, it fails on hundreds of tests regarding them.
- and on dozens regarding strict mode
parseInt still accepts octal.JSON.parse('{"__proto__":[]}'); returns object with 2 __proto__ properties!
Opera 12
Surprisingly good, only 4 bugs.
parseInt still accepts octal.- Does not treat whitespace characters from
[\u2000-\u23ff] as those in regular expressions. (2 tests).
String.prototype.localeCompare does not treat missing 'that' argument as undefined.
Android 4.3 (Galaxy S3)
parseInt still accepts octal.String.prototype.localeCompare does not treat missing 'that' argument as undefined.- Failed to add a property to an object when the object's prototype has a property with the same name and [[Writable]] set to false. Non-writable property on a prototype written to.
- Call arguments are not evaluated before the check is made
Android 4.1 (ASUS fonepad)
JSON.parse('{"__proto__":[]}'); returns object with 2 __proto__ properties!String.prototype.localeCompare does not treat missing 'that' argument as undefined.- do not mess with
function.length for fn instances.
- numerous bugs with
Array.prototype methods applied to the global object (window)
IE8
After few hours of night hacking I made test suite running in IE8 - there're several things, which break in the GUI and test runner itself.
Of course, there're no es5 features, but they could be polyfilled to some extent. Nonetheless, some bugs due to subtle discrepancies in syntax parsing can arise.
Bugs are limitless, but I'll try to summarize them in concise way. Keep in mind, that most bugs from IE9 are also reproduced, but they're too minor comparing to the HELL happening below.
Incompatibilities described in terms how IE8 behaves. Correct behavior is either stated explicitly or considered obvious.
Scope
- indirect
eval call does not set scope to the global context
- variable name spoiling:
- function declared within
if block
- named function expressions spoil the outside scope.
- multiple names for function
- name used in
catch block
TypeError is used instead of ReferenceError for referencing variabledelete behave differently in unusual positions. Like delete foo();- global properties couldn't be removed with
delete operator, workaround
- it's only possible to define
length variable with var keyword in a global scope. Otherwise you'll get an error, trying to set it.
- it's possible to delete
prototype property.
Syntax
- trailing comma adds extra empty element to array and fail with SyntaxError in object literals
- most reserved words can't be used for dot-notation, in object literals and function names. More or less real use cases for them are:
jQuery("<div>", { class: "someClass" })jQuery(someSelector).css({ float: "left" })
The first one fails in IE, while the second one doesn't, because it isn't a reserved word in IE8. For the same reason IE8 allows other names such as let, interface and so on.
And both pieces of code fails in deprecated YUI compressor.
- the following constructions should generate syntax error:
if (...) { ... }; elsedo {}; while(..)try {...}; finally{...}
\\v VERTICAL TAB, U+2028 LINE SEPARATOR and U+2029 PARAGRAPH SEPARATOR can mess your code everywhere: in code itself, RegExp literals, comments.
Regex
- many whitespace characters are not considered as
\s. E.g. U+00A0 or separators from previous point.
- Symbol "İ"
U+0130 is considered as \w (ru-RU locale)
- backreference used before actual capturing-group incorrectly considered as charcode escape.
/\1(A)(B)\2/.exec("\1ABB"); vs /\1(A)(B)\2/.exec("ABB");
/[^]/ should match anything and /[]/ should match nothing, RegExpError is thrown insteadRegExpError is thrown instead of SyntaxError for wrong syntaxRegExpError is thrown instead of TypeError for wrong flags- flag duplications is allowed in both literals (
/./ii) and dynamic construction via RegExp
- non-matched capturing groups become
"" instead of undefined
- non-matched capturing groups inside repetition do not overwrite matched ones from previous matches
/(z)((a+)?(b+)?(c))*/.exec("zaacbbbcac")
(z)((aa)( ) (c))
(( ) (bbb)(c))
((a) ( ) (c))
^ ^ ^
a bbb c
Here "bbb" incorrectly "fall-through": match[4] should be undefined or at least ""
Numbers
parseInt still accepts octalNumber.prototype.toString(undefined) fails with errorNumber.prototype.toFixed(Infinity) should fail with TypeError(1e21).toFixed() should return 1e+21, not 1000000000000000000000
Arrays
- No
indexOf and lastIndexOf as well as other functional stuff.
Array.isArray is missingarray.slice(n, undefined) always returns an empty array[].sort(undefined) fails with errorArray.prototype.splice on objects doesn't remove values- setting non-integer (but numeric) length for array should fail
Strings
trim and lastIndexOf methods are missing"".match() returns null instead of """".search() returns null instead of 0- empty strings are not included if
separator argument is regex: "101101".split(/1/) === ["0", "0"]
- capturing groups are not included in the final results:
"010".split(/(1)/) === ["0", "0"]
split should not coerce to string undefined passed as separator argumentsplit should coerce limit argument with ToUint32- incorrect
length value for fromCharCode, replace, slice, search, indexOf and lastIndexOf methods
Misc
- ES5 methods of
Object are not supported. defineProperty and getOwnPropertyDescriptor are present. Alas, MSDN tells us that function
... supports DOM objects but not user-defined objects. The enumerable and configurable attributes can be specified, but they are not used
Date.prototype.toISOString is missingObject.prototype.toString incorrectly returns "[object Object]" when applied to window, DOM nodes, undefined or null.- JSON shouldn't allow characters
U+0010 through U+001F