Updated on 23. June 2014
Last weekend I watched a video about Java8 and Nashorn. This is a much faster successor to Rhino by Oracle. They claimed to have passing all of more than 11000 tests of es5 spec, even though no browser vendor made compatibility to so scrupulous degree. The presentation was from late 2012, hence I decided to check, how things are going for now.
You can run them by yourself here, but I already did that for most popular browsers and even not popular ones. And moreover, I found that tests often fail not because of the feature in description is implemented incorrectly. Simpler, some tests are not independent. Extrapolating that to all tests would be an survival bias. Hence I'm not saying all tests are badly written. But if one wants to interpret the fact of test fail, reading description is not enough. On the bad note, though, a pair of tests are out of tune with specification. They expect some certain behavior, when it is clearly stated that it may be implementation-dependent.
Here's a hg repo viewer, where separate tests could be found.
TL;DR
If you don't want to go deep into details about different tests and my rants about their quality, here's a list what you'd better not to do from es5:
- test your code in different browsers, even if you're using bulletproof library like jQuery - these tests themselves couldn't be run in IE8 due to bug in UI
- in Firefox one cannot change numerical properties on
window
- in Firefox
[[Writable]]
, [[Configurable]]
and [[Enumerable]]
are incorrectly set to false
when not provided in re-defining properties on window
- just pass them for every defineProperty
call.
- do not rely on features of strict mode. For example, sometimes primitives used as
this
are still wrapped into objects - in special cases or in browsers (IE9) which, do not support strict mode at all. I personally faced a bug emerged from combination of jQuery.each
on array of primitives and using strict equality for this
.
Using strict mode as a helper for early problem detection is encouraged, though.
- do not rely exclusively on feature presence - some implementations are rather buggy - e.g.
defineProperty
in Safari 5
- IE before 11th use slightly different representation for
Date.toString
, hence do not rely on exact values.
- some old browsers still parse strings as octal numbers, starting from zero - use
parseInt(n, 10)
or Number(n)
if you're sure there're no trailing characters.
String.prototype.localeCompare
returns incorrect results for non-canonical Unicode representations in many browsers
- do no not rely on assumption that code will fail at some certain point with some certain error. This is mostly OK for tests, though.
- hardly one would use that in real code, but messing with numeric properties on
Array.prototype
is not only difficult to understand, but is also vulnerable to subtle differences in implementations
Detailed analysis of failed tests
Listing full test names could be too "noisy" for visual perception, hence I compacted several names with syntax similar to imports in es6. E.g.: "1.1.2, 1.1.3" is written as "1.1.{2,3}".
First of all I need to mention, that six Date
tests do not fail if and only if you live California or elsewhere near. Relax, it is neither discrimination, nor location tracking - just OS setting, timezone should be UTC-8.
15.5.4.9_CE
Tests that String.prototype.localeCompare
returns 0 when comparing Strings that are considered canonically equivalent by the Unicode standard
All browsers except Chrome and Firefox 29 beta fails that test.
Interesting enough, strings from test case ("\u1111\u1171\u11B6" vs "\ud4db") are even displayed differently in FF29- (and strangely Chrome33-34): the first string looks like linear Hangul, whilst the second one is normal Jamo:
퓛 vs 퓛
OK, mainstream browsers go first in some arbitrary order.
Chrome 32+
11.2.3-3_3
Call arguments are not evaluated before the check is made to see if the object is actually callable (undefined member)
I.e. in expression ({}).b(f());
engine should call f()
before .b
is evaluated and detected as non-callable, whilst for ({}).a.b(f());
browser should not call f()
. In the second case Chrome incorrectly calls f()
as well.
S15.8.2.8_A6
Checking if Math.exp is approximately equals to its mathematical values on the set of 64 argument values; all the sample values is calculated with LibC
Chrome35 have similar bugs in 3 more tests.
The relative error is 2-50, while precision of IEEE754 doubles used in javascript, is 52 bits. So this error hardly matters, unless you're doing a REAL SCIENCE.
Moreover, specification [15.8.2] doesn't impose any certain requirements on computation accuracy for math functions:
The behaviour of the functions acos
, asin
, atan
, atan2
, cos
, exp
, log
, pow
, sin
, sqrt
, and tan
is not precisely specified here except to require specific results for certain argument values that represent boundary cases of interest. For other argument values, these functions are intended to compute approximations to the results of familiar mathematical functions, but some latitude is allowed in the choice of approximation algorithms.
Test itself is rather arguable. Are those approximations enough or not? Even if this is a problem, it's not a big one.
Safari 7 (MacOS/iOS)
10.2.1.1.3-4-{16,18}-s
TypeError
should be thrown when changing global.NaN
or global.undefined
in strict mode.
Safari 6 (iOS)
Surprisingly it doesn't fail the previous test.
S10.4.2_A1.{1,2}_T{1-11}
eval
within global execution context - scopes are not chained properly
15.5.4.9_3
Tests that String.prototype.localeCompare
treats a missing "that" argument, undefined, and "undefined" as equivalent.
str.localeCompare()
, str.localeCompare(undefined)
and str.localeCompare("undefined")
should be the same for any possible value of str
.
Firefox 27-33
10.4.3-1-{104,106}
Strict mode should not ToObject thisArg if not an object.
In es3 and non-strict es5, primitive passed as this
, are wrapped into object (String, Number or Boolean). In strict mode primitives should not be unwrapped. This works in general wherever strict mode is enabled and supported.
In getter, defined on Object.prototype
and used on primitive value (via boxing), this
still should be a primitive. And this is where Firefox fails.
Actually, if you write cross-browser code, you shouldn't rely on aforementioned feature of strict mode.
15.11.4.4-8-2
Error.prototype.toString
return empty string when 'name' is empty string and 'msg' is undefined
FF returns "Error" if name is empty. Not a serious bug, again.
Numerous bugs with Object.defineProperty
.
Sometimes (6 from 15.2.3.6-4-{292-296}) checks interfere with implementation details of mapping between arguments
object and actual named arguments like this:
(function (a) {
Object.defineProperty(arguments, '0', { value: 2 });
return a;
}(1));
BTW in strict mode all browsers demonstrate the same behavior as Firefox.
-354-4, -354-13, -360-3, -360-7, -531-4, -531-13, -538-3, -538-7 and 15.2.3.7-6-a-24
In these cases there's really an error. All 3 properties: writable
, enumerable
and configurable
become false
on re-definition if they're not provided. This is detected on window
only and doesn't happen for normal objects. -531-17 and -538-7 fails even earlier due to inability to redefine numerical properties on window.
15.4.4.{14-22}
Problems with Array.prototype
methods applied to global object.
The same reason as just mentioned - it fails because one cannot declare numerical keys on window
- they continue to be undefined
.
Actually, every method of Array.prototype
has a NOTE like:
The reduce
function is intentionally generic; it does not require that its this value be an Array object. Therefore it can be transferred to other kinds of objects for use as a method. Whether the reduce
function can be applied successfully to a host object is implementation-dependent.
I.e. spec doesn't require ability to successfully apply those methods to host objects! So tests are just incorrect.
S15.4.4.4_A3_T1
[[Prototype]] of Array instance is Array.prototype, [[Prototype] of Array.prototype is Object.prototype
This test fails due to implementation details of concat
(not own property remains not own in Firefox).
15.4.4.4-5-c-i-1
Array.prototype.concat will concat an Array when index property (read-only) exists in Array.prototype (Step 5.c.i)
It fails because Array.prototype.concat.call(1)[0]
results in number
primitive, and not Number
object. The check of copy's properties existence actually passes.
IE11
The latest 11th version fails only one test about localeCompare
IE10
IE11 in IE10 emulation mode shows the same behavior.
It fails one more test, which is rather funny:
S15.5.4.7_A1_T11
Fails for eastern hemisphere (UTC+N timezones), cause IE does not display leading zeroes for dates: new Date(0).toString()
→ "Thu Jan 1 04:00:00 UTC+0400 1970"
Yeah, it sounds so arcane, but check in code is really like that:
(date.indexOf(date.getTimezoneOffset()>0 ? '31' : '01')) === 8
One way or another, it reveals a piece of non-standard behavior of IE.
IE9
IE9 fails significantly more tests additionally to aforementioned two.
And true IE9 shows results somewhat different from higher versions in corresponding emulation mode (surprise-surprise). The only good point about that is that both emulation modes (IE11@9 and IE10@9) works the same.
Of course, it also can't pass 500+ tests about strict mode. I didn't even check 'em all - just filtered out.
8.12.9-9-c-i_{1,2} (passes in emulation mode)
Redefine a configurable accessor property to be a data property on a non-extensible object
with preventExtensions
.
10.6-13-a-2 (passes in emulation mode)
A direct call to arguments.callee.caller should work
(function (called) {
function test1(flag) { if (!flag) test2(); else called = true; }
function test2() { arguments.callee.caller(true); }
test1();
return called;
}(false));
Rather tricky and strange.
11.2.3-3_{1-2,5-8}
Call arguments are evaluated before the check is made to see if the object is actually callable.
It fails differently than in Chrome.
15.11.4.{2,3}-1
Error.prototype.{name,message} are not enumerable
15.2.4.4-{1,2}
Using Object.prototype.valueOf
for non-object values
Expected result: primitive value should be wrapped into corresponding object.
Workaround: if you need to wrap primitive into object, Object(value)
seems to be working properly everywhere.
S15.2.4.4_A{12-15}
Object.prototype.valueOf
invoked with null
/undefined
should throw an error.
15.4.4.4-5-c-i-1
Array.prototype.concat will concat an Array when index property (read-only) exists in Array.prototype (Step 5.c.i)
The same as in Firefox.
15.5.4.11-1 (passes in emulation mode)
'this' object used by the replaceValue
function of a String.prototype.replace
invocation
S15.5.4.14_A1_T{6-9}, ...A2_T7
Split should not coerce its argument to string "undefined" if its argument is undefined
.
15.9.5.40_1
Date.prototype.setFullYear - Date.prototype is itself an instance of Date
Test just calls Date.prototype.setFullYear(2012);
and expects 2012
to be read back. Hardly you'll need that in practice, but this test fails only in IE9. All other higher and lower versions aren't affected.
Old non-IEs
They are not widespread at all for now, hence only short overviews for each. Nonetheless, number of such client will not drop abruptly in foresee future - because of vendor-lock or similar situation.
Safari 5 (Win)
- even though, it claims support of defining properties, it fails on hundreds of tests regarding them.
- and on dozens regarding strict mode
parseInt
still accepts octal.
JSON.parse('{"__proto__":[]}');
returns object with 2 __proto__
properties!
Opera 12
Surprisingly good, only 4 bugs.
parseInt
still accepts octal.
- Does not treat whitespace characters from
[\u2000-\u23ff]
as those in regular expressions. (2 tests).
String.prototype.localeCompare
does not treat missing 'that' argument as undefined.
Android 4.3 (Galaxy S3)
parseInt
still accepts octal.
String.prototype.localeCompare
does not treat missing 'that' argument as undefined.
- Failed to add a property to an object when the object's prototype has a property with the same name and [[Writable]] set to false. Non-writable property on a prototype written to.
- Call arguments are not evaluated before the check is made
Android 4.1 (ASUS fonepad)
JSON.parse('{"__proto__":[]}');
returns object with 2 __proto__
properties!
String.prototype.localeCompare
does not treat missing 'that' argument as undefined.
- do not mess with
function.length
for fn instances.
- numerous bugs with
Array.prototype
methods applied to the global object (window
)
IE8
After few hours of night hacking I made test suite running in IE8 - there're several things, which break in the GUI and test runner itself.
Of course, there're no es5 features, but they could be polyfilled to some extent. Nonetheless, some bugs due to subtle discrepancies in syntax parsing can arise.
Bugs are limitless, but I'll try to summarize them in concise way. Keep in mind, that most bugs from IE9 are also reproduced, but they're too minor comparing to the HELL happening below.
Incompatibilities described in terms how IE8 behaves. Correct behavior is either stated explicitly or considered obvious.
Scope
- indirect
eval
call does not set scope to the global context
- variable name spoiling:
- function declared within
if
block
- named function expressions spoil the outside scope.
- multiple names for function
- name used in
catch
block
TypeError
is used instead of ReferenceError
for referencing variable
delete
behave differently in unusual positions. Like delete foo();
- global properties couldn't be removed with
delete
operator, workaround
- it's only possible to define
length
variable with var
keyword in a global scope. Otherwise you'll get an error, trying to set it.
- it's possible to delete
prototype
property.
Syntax
- trailing comma adds extra empty element to array and fail with SyntaxError in object literals
- most reserved words can't be used for dot-notation, in object literals and function names. More or less real use cases for them are:
jQuery("<div>", { class: "someClass" })
jQuery(someSelector).css({ float: "left" })
The first one fails in IE, while the second one doesn't, because it isn't a reserved word in IE8. For the same reason IE8 allows other names such as let
, interface
and so on.
And both pieces of code fails in deprecated YUI compressor.
- the following constructions should generate syntax error:
if (...) { ... }; else
do {}; while(..)
try {...}; finally{...}
\\v
VERTICAL TAB, U+2028
LINE SEPARATOR and U+2029
PARAGRAPH SEPARATOR can mess your code everywhere: in code itself, RegExp literals, comments.
Regex
- many whitespace characters are not considered as
\s
. E.g. U+00A0
or separators from previous point.
- Symbol "İ"
U+0130
is considered as \w
(ru-RU locale)
- backreference used before actual capturing-group incorrectly considered as charcode escape.
/\1(A)(B)\2/.exec("\1ABB");
vs /\1(A)(B)\2/.exec("ABB");
/[^]/
should match anything and /[]/
should match nothing, RegExpError
is thrown instead
RegExpError
is thrown instead of SyntaxError
for wrong syntax
RegExpError
is thrown instead of TypeError
for wrong flags
- flag duplications is allowed in both literals (
/./ii
) and dynamic construction via RegExp
- non-matched capturing groups become
""
instead of undefined
- non-matched capturing groups inside repetition do not overwrite matched ones from previous matches
/(z)((a+)?(b+)?(c))*/.exec("zaacbbbcac")
(z)((aa)( ) (c))
(( ) (bbb)(c))
((a) ( ) (c))
^ ^ ^
a bbb c
Here "bbb"
incorrectly "fall-through": match[4]
should be undefined
or at least ""
Numbers
parseInt
still accepts octal
Number.prototype.toString(undefined)
fails with error
Number.prototype.toFixed(Infinity)
should fail with TypeError
(1e21).toFixed()
should return 1e+21
, not 1000000000000000000000
Arrays
- No
indexOf
and lastIndexOf
as well as other functional stuff.
Array.isArray
is missing
array.slice(n, undefined)
always returns an empty array
[].sort(undefined)
fails with error
Array.prototype.splice
on objects doesn't remove values
- setting non-integer (but numeric) length for array should fail
Strings
trim
and lastIndexOf
methods are missing
"".match()
returns null
instead of ""
"".search()
returns null
instead of 0
- empty strings are not included if
separator
argument is regex: "101101".split(/1/) === ["0", "0"]
- capturing groups are not included in the final results:
"010".split(/(1)/) === ["0", "0"]
split
should not coerce to string undefined
passed as separator
argument
split
should coerce limit
argument with ToUint32
- incorrect
length
value for fromCharCode
, replace
, slice
, search
, indexOf
and lastIndexOf
methods
Misc
- ES5 methods of
Object
are not supported. defineProperty
and getOwnPropertyDescriptor
are present. Alas, MSDN tells us that function
... supports DOM objects but not user-defined objects. The enumerable and configurable attributes can be specified, but they are not used
Date.prototype.toISOString
is missing
Object.prototype.toString
incorrectly returns "[object Object]"
when applied to window
, DOM nodes, undefined
or null
.
- JSON shouldn't allow characters
U+0010
through U+001F