Technology Evaluation - Internationalising and Localising UI strings

From wiki.gpii
Jump to: navigation, search


To support the work on the dev PMT, Steve Githens asked if we could add the ability to internationalise messages to gpii-handlebars. I first looked into the support within handlebars itself, they suggest making a helper that accepts some kind of message key.

Writing a helper is not difficult, but we need to look around at what might power this helper, i.e. what would actually be taking a locale, a message string, and variables, and turning that into internationalised text to be displayed.

The Requirements

We need an approach that:

  1. Lets us provide locale-specific wording in our user interfaces.
  2. Supports cleanly interpolating variables.
  3. Works in supported browsers (for client-side rendering).
  4. Works in supported Node versions (for server-side rendering).
  5. Is open source with a compatible license.
  6. Does not introduce an inordinate amount of complexity, for example, by:
    1. Requiring us to convert node libraries using browserify
    2. Requiring loading many and/or very large libraries.
    3. Forcing us to adopt large parts of another framework just to handle this specific problem.

The Candidates

First, we have the i18n support build into Infusion itself. We should consider this as one starting point, but also look at what else is out there. With that in mind, I reviewed every library mentioned in this excellent overview on StackOverflow (including a few mentioned within those projects).

Here is a summary table of all candidates:

Library Meets Requirements Node Support Browser Support Community Complexity Licence
browser-i18n Possibly (when paired with node-i18n or node-i18n2) no yes Long-lived project with very very infrequent updates (years between commits). Low (single library). MIT
counterpart No (requires browserify) yes only via browserify Modest but consistent activity over the years. Primarily a single maintainer. Low (single library). MIT
ECMAScript i18n API No (no support for strings) yes, in later versions. yes, in most modern browsers. It’s a standard, so the community is the ECMAScript community. Very low (built into the language)  ?
i10n.js No (no node support) no yes Seems like a more or less single maintainer, and the project seems dead at this point. Even their docs site is a 404. Low (single library). MIT
i18n-node-2 Possibly (when paired with browser-i18n) yes no Limited activity, year-long gap between recent commits. Low (single library). MIT
i18n-node Possibly (when paired with browser-i18n) yes no Consistent, but spiky commits. Low (single library). MIT
i18next Yes yes yes By far the healthiest-seeming community with huge momentum, especially recently. Good industry adoption, but limited core committers. Low (single library). MIT
Intl.js No (only supports dates and numbers) yes yes Seems to have less recent activity, but otherwise healthy. Low (single library). MIT
jQuery globalize No (no node support) no yes Massive and vibrant. Low (only requires jQuery). MIT
jQuery Localisation Plugin No (no node support) no yes Very low activity, and none since 2012. Seems dead. Low (only requires jQuery). GPL and MIT
jQuery.i18Now No (no node support) no yes No activity since late 2014. Low (only requires jQuery). MIT
js-lingui No (too complex) maybe yes New project, very active. Seems to be mainly a single maintainer. High (claims neutrality, but heavily skewed towards React) MIT
l10ns No (too complex and not a strong enough community) yes yes Seems like mainly a single maintainer, and not much recent activity. Moderate. They have their own templating and lookup languages. Apache 2.0
l20n.js No (no node support) no yes Seems healthy and fairly active. They look like they’re steadily gaining momentum. Low (single library). Apache 2.0
moment.js No (only dates) yes yes Steady flow of commits and releases, modest group of core committers. Low (single library). MIT
numbro.js No (only numbers) yes yes Slower, but steady flow of commits and releases. Low (single library). MIT
requirejs-i18n No (too complex) yes yes Massive effort, less activity recently. High (requires using a different module loader). MIT
YUI internationalisation No (no node support) no yes No activity since late 2014. (Yahoo stopped development in 2014.) Moderate to high, requires using YUI. BSD

After the initial review, the final candidates are:

  1. Infusion itself
  2. i18next

After discussions in the PCP meeting on Thursday, July 21, 2017, it was agreed that I would examine the top candidates above and write up a "Hello World" with each.

Detailed Review

In preparing this review, I created a test harness and two prototype implementations, one with Infusion, one with i18next. You can see this work here in a repository on GitHub.

For each of the candidates, I looked at:

  1. Message lookup (keys, message bundle format)
  2. Variable interpolation
  3. Speed


Within the infusion package itself, internationalisation is mainly implemented with regards to the preferences framework. There is a message loader component, which retrieves message bundles. There is also a message resolver that takes a message bundle, message key, and variable data and produces string output. The message resolver works in both node and the browser unaltered. The message loader is a client-side component, for the initial tests I bypassed that and only tested "preloaded" message bundles. In practice we would need something to handle message loading for both node and the client side (see below).

Message Lookup

A message bundle in Infusion looks something like the following:

  "static":       "I can eat glass, and it doesn't hurt me.",
  "static_sa":    "काचं शक्नोम्यत्तुम् । नोपहिनस्ति माम् ॥",
  "static_sa_IN": "काचं शक्नोम्यत्तुम् । नोपहिनस्ति माम् ॥"

The first example presents a message key when the locale and language are unknown. The second example presents a message key when only the language is known. The third example shows a message key that is suffixed with a full locale (i.e. language and country). To look up the translated text, we use code like:

demo.infusion.translate("static", {}, "sa_IN");
demo.infusion.translate("static", {}, "sa");
// both return "काचं शक्नोम्यत्तुम् । नोपहिनस्ति माम् ॥"

If we attempt translation with a locale for which there is no message, a default locale is used. The existing Infusion components did not provide a failover strategy that would work in both the browser and node. For my prototype, I double-registered messages as demonstrated above. For a real implementation, we would need to discuss how best to merge language files and failover sensibly from a locale that does not have a translation to the same language in another country.

If a message key is supplied that cannot be matched in any language, the key itself is used as the message template.

demo.infusion.translate("Not no body, not no how.", {}, "en");
// "Not no body, not no how."

As we will see in the examples in the next section, this behavior allows us to use "inline" templates.

Variable Interpolation

In Infusion, messages may include placeholders for variable content, which look something like:

demo.infusion.translate("hello, %planet", { planet: "world"}, "en");
// "hello, world"

The message resolver makes use of the string templating system, and the syntax for variable substitution is exactly the same. The longest matching top-level key after the percent sign is replaced with the variable content. If no matching key is found for text after a percent sign, the original text is left intact, as in:

demo.infusion.translate("%title", {} }, "en");
// "%title"

This lends itself well to replacing "layers" of variables in multiple steps, for example, replacing the port and hostname of a REST endpoint in one stage, and replacing URL parameters in a later stage. When rendering text to be displayed, however, it does present a risk that placeholders may be visible to end users.

The resolver can also work with array content, as shown in the following example:

demo.infusion.translate("%0", ["hello", "world"], "en")
// "hello"

However, the resolver does not currently support "deep" paths, as demonstrated in the following example:

demo.infusion.translate("%deep.path", { deep: { path: "yo" } }, "en");
// "[object Object].path"

This seems to be a limitation of the underlying fluid.stringTemplate method:

fluid.stringTemplate("%deep.path", { deep: { path: "yo" }});
// "[object Object].path"

Finally, the resolver does not provide a means of describing the "root" of an object, as is possible with model transformations or with Handlebars itself, where a single "dot" indicates the current "context" in its entirety.


The performance of the demo infusion component was perfectly acceptable:

Performance, Infusion
Environment Translations / s Variable Interpolations / s
Node.js 70,354 78,345
Firefox 53.0 91,949 103,435
PhantomJS 2.1 112,300 113,500
Opera 46.0 141,307 155,930
Safari 10.1 184,748 152,626
Chrome 59.0 161,055 173,589
IE 11.0 77,489 80,598

Infusion Summary

Infusion meets our needs, although we will need to write a bit more code to handle message loading and failover. Based on the types of client-side data I have worked with, it seems useful to add support for deep paths to the %variable notation used by fluid.stringTemplate.


Message Lookup

A message bundle in i18next looks something like the following (presented as a Javascript Object to allow for inline comments):

"sa-IN": { // language slash locale, note that the dashes are required
     translation: { // namespace
          "key": "काचं शक्नोम्यत्तुम् । नोपहिनस्ति माम् ॥" // message key and value
"sa": { // language slash locale, note that the dashes are required
     translation: { // namespace
          "key": "काचं शक्नोम्यत्तुम् । नोपहिनस्ति माम् ॥" // message key and value

In theory, this kind of double-registration is not necessary, as i18next does support smart loading, starting with locale, failing back to language, and then failing back to the overall default. However, for the prototype, I was initializing the language bundles from a transformed common format, and couldn't figure out how to handle that correctly. I ended up writing a crude failover strategy, as I did with Infusion.

As Infusion does, if we attempt to translate a message key that cannot be resolved, i18next renders the key as the message.

demo.infusion.translate("Not no body, not no how.", {}, "en");
// "Not no body, not no how."

As you will see in the examples, this is useful for demonstrating "inline" templates. In testing this feature, I noticed that characters like the colon have special meaning in i18next message keys. This is apparently configurable, but I wasn't able to completely disable this in my limited testing.

Variable Interpolation

The syntax i18next uses for variable interpolation should be familiar to anyone who has written a Handlebars template:

demo.i18next.translate("hello, {{planet}}", { planet: "world"}, "en")
// "hello, world"

Although i18next uses a handlebars like syntax for variables, notably it does not support referencing the root of the current context using a dot character. Like Handlebars, i18next does support "deep" references using "dot notation" comparable to what we use with fluid.get within Infusion.

demo.i18next.translate("{{deep.path}}", { deep: { path: "yo" } }, "en");
// "yo"

Unlike infusion, when a variable cannot be found, i18next renders an empty string.

demo.i18next.translate("title = {{title}}", {}, "en")
// "title = "

There are cases in which this kind of pattern might be useful, although in most cases I would assume we're checking for the existence of variables using Handlebars, which has much smarter support for conditional rendering of blocks of text.

Like Infusion, i18next supports working with array data as well:

demo.i18next.translate("{{0}}", ["hello"], "en")
// "hello"


The performance of i18next was acceptable (see below for speed comparison):

Performance, i18next
Environment Translations / s Variable Interpolations / s
Node.js 39,871 30,953
Firefox 53.0 34,861 31,632
PhantomJS 2.1 50,978 40,642
Opera 46.0 92,299 63,012
Safari 10.1 97,628 65,313
Chrome 59.0 98,561 67,634
IE 11.0 25,931 20,799

i18next Summary

In addition to making the core requirements well, i18next has an incredible range of features. Here are a few that caught my eye in a detailed review of their documentation:

  1. Message keys in i18next use suffixing to support highly complex plurals, including different plurals for a range of "contexts".
  2. Support for "nesting" of message keys using the $(childKey) syntax, meaning that you can compose longer messages out of smaller message parts.
  3. Support for custom formatting within a variable placeholder, so that you can control the format of an individual date or translation.

Of these, it seems most likely that we would need at least rudimentary support for plurals.

Detailed Review Conclusion

Both Infusion and i8next meet our core requirements equally well, and require building out the same missing infrastructure. In all environments, Infusion was noticeably faster than i18next (which makes sense, as it is a leaner approach):

How much faster is the Infusion approach than the i18next approach?
Environment Translations / s Variable Interpolations / s
Node.js 76% faster 153% faster
Firefox 53.0 164% faster 227% faster
PhantomJS 2.1 120% faster 179% faster
Opera 46.0 53% faster 147% faster
Safari 10.1 89% faster 134% faster
Chrome 59.0 63% faster 157% faster
IE 11.0 199% faster 288% faster

In general, the preference is to extend and improve our core libraries rather than introduce new ones. In this case, the question seems to be whether we need enough of the unique value i18next provides, or whether we should extend and improve the faster and more familiar message resolver bundled with Infusion.

Practical Questions Raised

In addition to the up-front requirements, the deep review and subsequent discussions highlighted a few more practical concerns for wider discussion.

Loading and Bundling Messages

The existing Infusion implementation only loads messages from within a client-side component. i18next provides its own loading strategy, but only within node. For the longer term, we need an approach that works for both the server and client.

Whichever we choose, I would propose that (as with JSON schemas and handlebars templates), we use a piece of middleware to assemble all available messages on the server side. This can make use of package-relative paths and other features available in node. We could then make a simpler client-side component that retrieves complete message bundles via a REST call.

Live Reloading Concerns

Neither of the evaluated packages natively helps with the key problem of "live reloading", i.e. delivering new message bundles when they are updated. On the server-side, we need something like the "watcher" developed for gpii-handlebars that will look for filesystem changes and reload the data as needed. Antranig and I discussed moving this to its own micromodule for reuse here.

On the client-side, we need a means of making a browser aware of updated message content. Assuming that we use the kind of middleware approach I outlined above, Antranig and I discussed three approaches to live reloading:

  1. Documenting the requirement for a full browser reload to pick up message changes, and living with that in the short term.
  2. Exposing all available message bundles via a REST interface, and using client-side polling to pick up server-side changes. Although this is only a short-to-medium term solution, with proper caching, the performance impact would be fairly low.
  3. Long term, we discussed using the Nexus to relay changes like these from the server to the client.

Where to Continue with This Work

Although parts of this work would take place in other packages, we should agree where it most makes sense for the general functionality to live. My suggestion would be either:

  1. In the gpii-handlebars package.
  2. Within its own micromodule.
  3. Within Infusion itself.


In the architecture meeting on August 2nd, we discussed this review and agreed that:

  1. We do not currently need the unique features of i18next, and will proceed with an Infusion-based approach.
  2. We agreed that I would create two tickets and submit two pull requests:
    1. A ticket/pull against Infusion, to move the messageResolver out of the renderer
    2. A ticket/pull against gpii-handlebars, to add the core of the new i18n approach there.