pixelhandler.dev

How Much Faster is HTMLBars Than Handlebars?

Billy Heaton 12 min read
Ember.js Templates

When I shared the data with Kris Seldon, he asked whether I had normalized the data and calculated the geometric mean, since that would be a proper way to compare benchmarks. I found the document How not to lie with statistics, The correct way to summarize benchmark results by Philip J. Fleming and John J. Wallace. So, I wrote a query for my data set that would group the metrics data in an effort to normalize the measurements taken from actual users on various devices and group the measurements into the benchmarks that I could generate some reporting data from.
To itentify the benchmarks I could report on I wrote some filters for the various userAgent strings within my metrics data. I considered the combination of the operating system and the browser, e.g. Safari on Macintosh, or iPad combined with a type of metric, e.g. index_view. Then I compared the Ember versions.

With the normalized set of data, I created a query that would take the latest measurements, up to 100, from the normalized data set and calculated the geopmentric mean for the various benchmarks. I calculated the product of each duration in the normalize set; then raised that value by an exponent that is the reciprocal of the number of factors in that product. (Geometric Mean: multiply the numbers and then take the nth root of the product.) The calculation looked something like this Math.round(Math.pow(2 * 3 * 5 * 8 * 13 * 21, 1/6) * 1000) / 1000 (I rounded to the nearest thousandth). As for the normalization of the data, the metrics can be reported by the Ember Version, platform (Macintosh/Windows) and browser (Chrome, Safari, Firefox, iPad, iPhone, Android) for the specific metric type, e.g. index_view, post_view or the metric_table (for the long list of 1,000 items).

I'd like to share my measurements and calculations…

Performance Report

For the upgrade on my site to HTMLBars from Handlebars 1.3…

Across the board, for the benchmarks I am intersted in, it looks like HTMLBars brought a 10-22% gain in rendering speeds

Index View: Home Page
Platform Browser v1.8 Mean v1.10 Mean Difference Gain/Loss %
Macintosh Safari 162.343 86.653 75.69 47%
Macintosh Chrome 183.794 187.945 -4.151 -2%
Windows Chrome 136.289 211.616 -75.327 -55%
iOS iPhone 450.503 352.764 97.739 22%
Overall 206.89 186.73 20.16 10%



Post View: Article Pages
Platform Browser v1.8 Mean v1.10 Mean Difference Gain/Loss %
iOS iPad 801.595 375.945 425.65 53%
iOS iPhone 348.711 302.05 46.661 13%
Android Various 850.716 862.627 -11.911 -1%
Macintosh Safari 109.321 117.684 -8.363 -8%
Macintosh Chrome 145.244 163.481 -18.237 -13%
Windows Chrome 203.063 232.081 -29.018 -14%
Macintosh Firefox 147.432 140.634 6.798 5%
Windows Firefox 200.542 163.704 36.838 18%
Overall 262.68 237.34 25.34 10%



Metrics Table: Long List of 1,000 Records
Platform Browser v1.8 Mean v1.10 Mean Difference Gain/Loss %
iOS iPad 3323.655 2170.638 1153.017 35%
iOS iPhone 1580.537 1268.771 311.766 20%
Macintosh Safari 448.611 302.707 145.904 33%
Macintosh Chrome 903.33 646.06 257.27 28%
Windows Chrome 812.356 829.604 -17.248 -2%
Macintosh Firefox 865.249 743.429 121.82 14%
Overall 1069.48 832.20 237.27 22%


The Details

Below is a look at the calculations for each of the benchmarks that I included in the summary above.

The Index (Home) Page

Safari on Macintosh

HTMLbars shaved of 76 milliseconds, a 46% gain in rendering speed

{
  fast: 32.94,
  geometric_mean: 86.653,
  measurments: 100,
  name: "index_view",
  slow: 819,
  emberVersion: "1.10",
  platform: "Macintosh",
  browser: "Safari"
}

{
  fast: 105.18,
  geometric_mean: 162.343,
  measurments: 4,
  name: "index_view",
  slow: 244.669,
  emberVersion: "1.8",
  platform: "Macintosh",
  browser: "Safari"
}

Chrome on Macintosh

Only a 4 millisecond difference, a 2% decrease in rendering speed

{
  fast: 65.41,
  geometric_mean: 187.945,
  measurments: 100,
  name: "index_view",
  slow: 9547.233,
  emberVersion: "1.10",
  platform: "Macintosh",
  browser: "Chrome"
}

{
  fast: 90.804,
  geometric_mean: 183.794,
  measurments: 70,
  name: "index_view",
  slow: 550.173,
  emberVersion: "1.8",
  platform: "Macintosh",
  browser: "Chrome"
}

Chrome on Windows

Handlebars 1.3 was 74 milliseconds faster, a 35% gain in rendering speed

{
  fast: 81.023,
  geometric_mean: 211.616,
  measurments: 100,
  name: "index_view",
  slow: 1397,
  emberVersion: "1.10",
  platform: "Windows",
  browser: "Chrome"
}

{
  fast: 95.374,
  geometric_mean: 136.289,
  measurments: 13,
  name: "index_view",
  slow: 172.329,
  emberVersion: "1.8",
  platform: "Windows",
  browser: "Chrome"
}

Android

Only had measurements on Ember v1.10.0

{
  fast: 520.524,
  geometric_mean: 841.979,
  measurments: 18,
  name: "index_view",
  slow: 1382.816,
  emberVersion: "1.10",
  platform: null,
  browser: "Android"
}

iPad

Only had measurements on Ember v1.10.0

{
  fast: 155,
  geometric_mean: 484.614,
  measurments: 34,
  name: "index_view",
  slow: 1120,
  emberVersion: "1.10",
  platform: null,
  browser: "iPad"
}

iPhone

HTMLBars shaved off 98 milliseconds, a 22% gain in rendering speed

{
  fast: 143,
  geometric_mean: 352.764,
  measurments: 62,
  name: "index_view",
  slow: 1490,
  emberVersion: "1.10",
  platform: null,
  browser: "iPhone"
}

{
  fast: 330,
  geometric_mean: 450.503,
  measurments: 7,
  name: "index_view",
  slow: 588,
  emberVersion: "1.8",
  platform: null,
  browser: "iPhone"
}

So for the index page the gains from HTMLBars I noticed were on Safari on Macintosh and on the iPhone. However using Chrome on Windows Handlebars was faster.

I didn't have lots of data for the home page in Ember v1.8.1 but did get loads of traffic on some of my posts, let's see how the individual posts pages did.

Post (Article) Pages

iPad

HTMLBars shaved off about 426 milliseconds, a 53% gain in rendering speed

{
  fast: 116,
  geometric_mean: 375.945,
  measurments: 100,
  name: "post_view",
  slow: 2051,
  emberVersion: "1.10",
  platform: null,
  browser: "iPad"
}

{
  fast: 193,
  geometric_mean: 801.595,
  measurments: 18,
  name: "post_view",
  slow: 1449,
  emberVersion: "1.8",
  platform: null,
  browser: "iPad"
}

iPhone

HTMLBars shaved off about 47 milliseconds, an 11% gain in rendering speed

{
  fast: 8,
  geometric_mean: 302.05,
  measurments: 100,
  name: "post_view",
  slow: 2139,
  emberVersion: "1.10",
  platform: null,
  browser: "iPhone"
}

{
  fast: 174,
  geometric_mean: 348.711,
  measurments: 32,
  name: "post_view",
  slow: 985,
  emberVersion: "1.8",
  platform: null,
  browser: "iPhone"
}

Android

Handlebars 1.3 was 12 milliseconds faster, no rendering speed gains from upgrade to HTMLBars

{
  fast: 204.868,
  geometric_mean: 862.627,
  measurments: 100,
  name: "post_view",
  slow: 4610.199,
  emberVersion: "1.10",
  platform: null,
  browser: "Android"
}

{
  fast: 275.503,
  geometric_mean: 850.716,
  measurments: 74,
  name: "post_view",
  slow: 2747.738,
  emberVersion: "1.8",
  platform: null,
  browser: "Android"
}

Safari on Macintosh

Handlebars 1.3 was 8 milliseconds faster, no rendering speed gains from upgrade to HTMLBars

{
  fast: 20.164,
  geometric_mean: 117.684,
  measurments: 100,
  name: "post_view",
  slow: 5616.18,
  emberVersion: "1.10",
  platform: "Macintosh",
  browser: "Safari"
}

{
  fast: 50.377,
  geometric_mean: 109.321,
  measurments: 100,
  name: "post_view",
  slow: 2551.46,
  emberVersion: "1.8",
  platform: "Macintosh",
  browser: "Safari"
}

Chrome on Macintosh

Handlebars 1.3 was 18 milliseconds faster, no rendering speed gains from upgrade to HTMLBars

{
  fast: 19.82,
  geometric_mean: 163.481,
  measurments: 100,
  name: "post_view",
  slow: 4220.102,
  emberVersion: "1.10",
  platform: "Macintosh",
  browser: "Chrome"
}

{
  fast: 18.673,
  geometric_mean: 145.244,
  measurments: 100,
  name: "post_view",
  slow: 2884.554,
  emberVersion: "1.8",
  platform: "Macintosh",
  browser: "Chrome"
}

Chrome on Windows

Handlebars 1.3 was 29 milliseconds faster, no rendering speed gains from upgrade to HTMLBars

{
  fast: 31.697,
  geometric_mean: 232.081,
  measurments: 100,
  name: "post_view",
  slow: 13214,
  emberVersion: "1.10",
  platform: "Windows",
  browser: "Chrome"
}

{
  fast: 56.472,
  geometric_mean: 203.063,
  measurments: 100,
  name: "post_view",
  slow: 3696,
  emberVersion: "1.8",
  platform: "Windows",
  browser: "Chrome"
}

Firefox on Macintosh

HTMLBars was 7 milliseconds faster, only a minor gain in rendering speed

{
  fast: 45.745,
  geometric_mean: 140.634,
  measurments: 100,
  name: "post_view",
  slow: 1247.685,
  emberVersion: "1.10",
  platform: "Macintosh",
  browser: "Firefox"
}

{
  fast: 71.173,
  geometric_mean: 147.432,
  measurments: 62,
  name: "post_view",
  slow: 387.791,
  emberVersion: "1.8",
  platform: "Macintosh",
  browser: "Firefox"
}

Firefox on Windows

HTMLBars was 37 milliseconds faster, an 18% gain in rendering speed

{
  fast: 73.865,
  geometric_mean: 163.704,
  measurments: 100,
  name: "post_view",
  slow: 1860,
  emberVersion: "1.10",
  platform: "Windows",
  browser: "Firefox"
}

{
  fast: 69.896,
  geometric_mean: 200.542,
  measurments: 100,
  name: "post_view",
  slow: 1305.268,
  emberVersion: "1.8",
  platform: "Windows",
  browser: "Firefox"
}

Metrics Table (long list of 1,000 records)

iPad

I only had a couple measurments…

Looks like HTMLBars can be 1153 milliseconds faster, a 35% gain in rendering speed

{
  fast: 2062,
  geometric_mean: 2170.638,
  measurments: 2,
  name: "metrics_table",
  slow: 2285,
  emberVersion: "1.10",
  platform: null,
  browser: "iPad"
}

{
  fast: 3103,
  geometric_mean: 3323.655,
  measurments: 2,
  name: "metrics_table",
  slow: 3560,
  emberVersion: "1.8",
  platform: null,
  browser: "iPad"
}

iPhone

HTMLBars was 312 milliseconds faster, a 20% gain in rendering speed

{
  fast: 675,
  geometric_mean: 1268.771,
  measurments: 8,
  name: "metrics_table",
  slow: 1568,
  emberVersion: "1.10",
  platform: null,
  browser: "iPhone"
}

{
  fast: 871,
  geometric_mean: 1580.537,
  measurments: 8,
  name: "metrics_table",
  slow: 5380,
  emberVersion: "1.8",
  platform: null,
  browser: "iPhone"
}

Safari on Macintosh

HTMLBars was 146 milliseconds faster, a 48% gain in rendering speed

{
  fast: 239.835,
  geometric_mean: 302.707,
  measurments: 13,
  name: "metrics_table",
  slow: 416.321,
  emberVersion: "1.10",
  platform: "Macintosh",
  browser: "Safari"
}

{
  fast: 287,
  geometric_mean: 448.611,
  measurments: 6,
  name: "metrics_table",
  slow: 568.253,
  emberVersion: "1.8",
  platform: "Macintosh",
  browser: "Safari"
}

Chrome on Macintosh

HTMLBars was 257 milliseconds faster, a 28% gain in rendering speed

{
  fast: 414.107,
  geometric_mean: 646.06,
  measurments: 27,
  name: "metrics_table",
  slow: 1584.423,
  emberVersion: "1.10",
  platform: "Macintosh",
  browser: "Chrome"
}

{
  fast: 457.012,
  geometric_mean: 903.33,
  measurments: 27,
  name: "metrics_table",
  slow: 1564.946,
  emberVersion: "1.8",
  platform: "Macintosh",
  browser: "Chrome"
}

Chrome on Windows

Handlebars 1.3 was 17 milliseconds faster, a 2% decrease in rendering speed

{
  fast: 615.905,
  geometric_mean: 829.604,
  measurments: 3,
  name: "metrics_table",
  slow: 1142.15,
  emberVersion: "1.10",
  platform: "Windows",
  browser: "Chrome"
}

{
  fast: 765.64,
  geometric_mean: 812.356,
  measurments: 2,
  name: "metrics_table",
  slow: 861.923,
  emberVersion: "1.8",
  platform: "Windows",
  browser: "Chrome"
}

Firefox on Windows

I did not collect enough measurements to make a comparison

Firefox on Macintosh

The data is limited to 2 and 4 measurements, I've incluced the metrics, and unique userAgents to show the individual measuremnts within the normalized set, for example the Firefox versions include versions 33.0 and 37.0.

In this limited set, HTMLBars is 122 milliseconds faster, a 14% gain in rendering speed

{
  "durations": [
    696.713,
    903.421,
    660.65,
    734.586
  ],
  "fast": 660.65,
  "geometric_mean": 743.429,
  "measurments": 4,
  "metrics": [
    {
      "date": "2015-02-17T06:04:15.251Z",
      "duration": 696.713,
      "emberVersion": "1.10.0",
      "id": "8b6994ee-268b-43ce-8f26-96ddfcf466a7",
      "pathname": "/metrics"
    },
    {
      "date": "2015-02-17T06:04:06.243Z",
      "duration": 903.421,
      "emberVersion": "1.10.0",
      "id": "28bad254-c3b9-462e-9966-4bc2fcc8fc63",
      "pathname": "/metrics"
    },
    {
      "date": "2015-02-17T06:02:24.487Z",
      "duration": 660.65,
      "emberVersion": "1.10.0",
      "id": "11dfee70-d635-41fd-8e69-2df4f1c70dc1",
      "pathname": "/metrics"
    },
    {
      "date": "2015-02-17T06:02:24.021Z",
      "duration": 734.586,
      "emberVersion": "1.10.0",
      "id": "b58722d9-c9eb-4ab2-bac7-41ba187a3646",
      "pathname": "/metrics"
    }
  ],
  "name": "metrics_table",
  "slow": 903.421,
  "userAgents": [
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:33.0) Gecko/20100101 Firefox/33.0",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:37.0) Gecko/20100101 Firefox/37.0"
  ],
  "emberVersion": "1.10",
  "platform": "Macintosh",
  "browser": "Firefox"
}

{
  "metric": {
    "durations": [
      746.559,
      1002.809
    ],
    "fast": 746.559,
    "geometric_mean": 865.249,
    "measurments": 2,
    "metrics": [
      {
        "date": "2015-02-17T03:02:27.682Z",
        "duration": 746.559,
        "emberVersion": "1.8.1",
        "id": "d828e5b6-c5c2-4fee-8bf3-2d08d97ed36d",
        "pathname": "/metrics"
      },
      {
        "date": "2015-02-17T03:02:11.844Z",
        "duration": 1002.809,
        "emberVersion": "1.8.1",
        "id": "e65effb9-4756-4642-8397-1757c4f3be15",
        "pathname": "/metrics"
      }
    ],
    "name": "metrics_table",
    "slow": 1002.809,
    "userAgents": [
      "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:33.0) Gecko/20100101 Firefox/33.0",
      "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:37.0) Gecko/20100101 Firefox/37.0"
    ],
    "emberVersion": "1.8",
    "platform": "Macintosh",
    "browser": "Firefox"
  }
}

Perhaps I'll use the queries and API endpoints for my metrics collection and reporting to see how the next version of Ember.js does, or see how a feature change impacts performance.

About the Author

Billy Heaton
Billy Heaton

Software engineer with two decades of experience who favors Ruby and JavaScript to build web applications