Today I visited jsPerf and now I am wondering…
Example: http://jsperf.com/concatenation-vs-join
Can anyone tell me?
Thanks in advance.
I wrote Benchmark.js, which jsPerf uses.
"ops/sec
" stands for operations per second.
That is how many times a test is projected to execute in a second.
A test is repeatedly executed until it reaches the minimum time needed to get a percentage uncertainty for the measurement of less than or equal to 1%
. The number of iterations will vary depending on the resolution of the environment’s timer and how many times a test can execute in the minimum run time. We collect completed test runs for 5
seconds (configurable), or at least 5
runs (also configurable), and then perform statistical analysis on the sample. So, a test may be repeated 100,000
times in 50 ms
(the minimum run time for most environments), and then repeated 100
times more (5
seconds). A larger sample size (in this example, 100
), leads to a smaller margin of error.
We base the decision of which test is faster on more than just ops/sec by also accounting for margin of error. For example, a test with a lower ops/sec but higher margin of error may be statistically indistinguishable from a test with higher ops/sec and lower margin of error.
We used a welch t-test, similar to what SunSpider uses, but switched to an unpaired 2-sample t-test for equal variance (the variance is extremely small) because the welch t-test had problems comparing lower ops/sec and higher ops/sec with small variances which caused the degrees of freedom to be computed as less than 1
. We also add a 5.5%
allowance on tests with similar ops/sec because real world testing showed that identical tests can swing ~5%
from test to re-test. T-tests are used to check that differences between tests are statistically significant.