Elixir implementation of CLDR/ICU
Cldris an Elixir library for the Unicode Consortium's Common Locale Data Repository (CLDR). The intentions of CLDR, and this library, is to simplify the locale specific formatting of numbers, lists, currencies, calendars, units of measure and dates/times. As of November 2020 and
ex_cldrVersion 2.18.0,
Cldris based upon CLDR version 38.0.0.
The first step is to define a module that will host the desired
Cldrconfiguration and the functions that serve as the public API. This module is referred to in this documentation as a
backendmodule. For example:
@doc """ Define a backend module that will host our Cldr configuration and public API.Most function calls in Cldr will be calls to functions on this module. """ defmodule MyApp.Cldr do use Cldr, locales: ["en", "fr", "zh", "th"], default_locale: "en"
end
This strategy means that different configurations can be defined and it also means that one
Cldrimplementation won't interfer with implementations in other, potentially dependent, applications.
The functions you are mostly likely to use are:
MyApp.Cldr.default_locale/0
MyApp.Cldr.put_locale/1
MyApp.Cldr.get_locale/0
MyApp.Cldr.known_locale_names/0
MyApp.Cldr.Locale.new/1
To access the raw Cldr data for a locale the
Cldr.Configmodule is available. Note that the functions in
Cldr.Configare typically used by library authors. The most useful function is:
Cldr.Config.get_locale/2which returns a map of all the CLDR data known to
Cldr. Since this data is read from a file, parsed and then formatted it is a function that should be used with care due to the material performance implications.
Cldruses this function during compilation to build functions that return the relevant data with higher performance and these functions are to be preferred over the use of
Cldr.Config.get_locale/2.
Support multiple languages and locales in your application
Need to support formatting numbers, dates, times, date-times, units and lists in one language or many
Access the data maintained in the CLDR repository in a functional manner
Parse an Accept-Language http header or a language tag
It is highly likely that you will also want to install one or more of the dependent packages that provide localization and formatting for a particular data domain. See Additional Cldr Packages below.
Add
ex_cldrand the JSON library of your choice as a dependencies to your
mixproject:
defp deps do [ {:ex_cldr, "~> 2.0"}, # Posion or any other compatible json library # that implements `encode!/1` and `decode!/1` # :jason is recommended {:jason, "~> 1.0"} # {:poison, "~> 2.1 or ~> 3.0"} ] end
then retrieve
ex_cldrand the JSON library from hex:
elixir mix deps.get mix deps.compile
ex_cldrincludes only basic functions to maintain the CLDR data repository in an accessible manner and to manage locale definitions. Additional functionality is available by adding additional packages:
Each of these packages includes
ex_cldras a dependency so configuring any of these additional packages will automatically install
ex_cldr.
Cldrattempts to maximise runtime performance at the expense of additional compile time. Where possible
Cldrwill create functions to encapsulate data at compile time. To perform these optimizations for all 541 locales known to Cldr wouldn't be an effective use of your time or your computer's. Therefore
Cldrrequires that you configure the locales you want to use.
The preferred way to configure
Cldris to define the configuration in your backend module. This removes any dependency on your
mix.exsand therefore simplifies deployment as a release. However configuration can also be defined in other ways:
In
mix.exsa global configuration can be defined under the
:ex_cldrkey. Although any valid configuration keys can be used here, only the keys
:json_library,
:cacertfileand
default_localeare considered valid. Other configuration keys may be used to aid migration from
Cldrversion 1.x but a deprecation message will be printed during compilation. Here's an example of global configuration:
config :ex_cldr, default_locale: "en", default_backend: MyApp.Cldr, json_library: Jason, cacertfile: "path/to/cacertfile"
Note that the
:json_librarykey can only be defined at the global level since it is required during compilation before any backend module is compiled.
On most platforms other than Windows the
:cacertfilewill be automatically detected. Any configured
:cacertfilewill take precedence on all platforms.
If configuration beyond the keys
:default_locale,
:cacertfileor
:json_libraryare defined a deprecation warning is printed at compile time noting that configuration should be moved to a backend module.
The preferred configuration method is to define the configuration in the backend module. The configuration keys are the same so the preferred way to achieve the same configuration as defined in the global example is:
defmodule MyApp.Cldr do use Cldr, default_locale: "en", locales: ["fr", "en", "bs", "si", "ak", "th"], gettext: MyApp.Gettext, data_dir: "./priv/cldr", otp_app: :my_app, precompile_number_formats: ["¤¤#,##0.##"], precompile_transliterations: [{:latn, :arab}, {:thai, :latn}], providers: [Cldr.Number], generate_docs: true, force_locale_download: false end
In the backend configuration example above the
:otp_appkey has been defined. This means that configuration for
Cldrhas been defined in
mix.exsunder the key
:my_appwith the sub-key
MyApp.Cldr. For example:
defmodule MyApp.Cldr do use Cldr, otp_app: :my_app end
# In mix.exs config :my_app, MyApp.Cldr, default_locale: "en", locales: ["fr", "en", "bs", "si", "ak", "th"], gettext: MyApp.Gettext, data_dir: "./priv/cldr", precompile_number_formats: ["¤¤#,##0.##"], precompile_transliterations: [{:latn, :arab}, {:thai, :latn}]
Multiple backends can be configured under a single
:otp_appif required.
When building the consolidated configuration the following priority applies:
The configuration keys available for
Cldrare:
default_localespecifies the default locale to be used for this backend. The default locale in case no other locale has been set is
"en-001". The default locale calculated as follows:
:default_localekey, then this is the priority
:default_localekey, then a configured
Gettextdefault locale for this backend is chosen
:default_localekey is specified and no
Gettextmodule is configured, or is configured but has no default set, use
Cldr.default_locale/0which returns either the default locale configurated in
mix.exsunder the
ex_cldrkey or then the system default locale will is currently
en-001
locales: Defines what locales will be configured in
Cldr. Only these locales will be available and an exception
Cldr.UnknownLocaleErrorwill be raised if there is an attempt to use an unknown locale. This is the same behaviour as
Gettext. Locales are configured as a list of binaries (strings). For convenince it is possible to use wildcard matching of locales which is particulalry helpful when there are many regional variances of a single language locale. For example, there are over 100 regional variants of the "en" locale in CLDR. A wildcard locale is detected by the presence of
.,
[,
*and
+in the locale string. This locale is then matched using the pattern as a
regexto match against all available locales. The example below will configure all locales that start with
en-and the locale
fr.
use Cldr, default_locale: "en", locales: ["en-*", "fr"]
There is one additional setting which is
:allwhich will configure all 541 locales. This is highly discouraged since it will take many minutes to compile your project and will consume more memory than you really want. This setting is there to aid in running the test suite. Really, don't use this setting.
:gettext: configures
Cldrto use a
Gettextmodule as an additional source of locales you want to configure. Since
Gettextuses the Posix locale name format (locales with an '_' in them) and
Cldruses the Unicode format (a '-' as the subtag separator),
Cldrwill transliterate locale names from
Gettextinto the
Cldrcanonical form.
:data_dir: indicates where downloaded locale files will be stored. The default is
:code.priv_dir(otp_app)where
otp_appis the app defined under the
:otp_appconfiguration key. If that key is not specified then the
:ex_cldrapp is used. It is recommended that an
:otp_appkey is specified in your backend module configuration.
:precompile_number_formats: provides a means to have user-defined format strings precompiled at application compile time. This has a performance benefit since precompiled formats execute approximately twice as fast as formats that are not precompiled.
:precompile_transliterations: defines those transliterations between the digits of two different number systems that will be precompiled. The is a list of 2-tuples where each tuple is of the form
{from_number_system, to_number_system}where each number system is expressed as an atom. The available number systems is returned by
Cldr.Number.System.systems_with_digits/0. The default is the empty list
[].
:precompile_date_time_formats: provides a means to have user-defined date, time and date time format strings precompiled at application compile time. This has a performance benefit since precompiled formats execute approximately twice as fast as formats that are not precompiled. These formats are used by excldrdate_times.
:precompile_interval_formats: provides a means to have user-defined interval format strings precompiled at application compile time. This has a performance benefit since precompiled formats execute approximately twice as fast as formats that are not precompiled. These formats are used by excldrdate_times.
:providers: a list of modules that provide
Cldrfunctionality to be compiled into the backend module. See the providers section below.
:generate_docsdefines whether or not to generate documentation for the modules built as part of the backend. Since these modules represent the public API for
ex_cldr, the default is
true. Setting this key to
false(the atom
false, not a falsy value) which prevent the generation of docs for this backend.
:supress_warningsdefines whether warnings are logged when a provider module is configured but not available. It also controls whether warnings are logged when a number format is compiled at runtime. Its purpose is to help identify those formats which might best be added to the
:precompile_number_formatsconfiguration. The default is
false. Warning are not logged when set to
true.
:force_locale_downloaddetermines whether to always download locale files during compilation. Locale data is
ex_cldrversion dependent. When a new version of
ex_cldris installed, no locales are installed and therefore locales are downloaded at compilation time as required. This ensures that the right version of the locale data is always associated with the right version of
ex_cldr. However if locale data is being cached in CI/CD there is some possibility that there can be a version mismatch. Since reproducable builds are important, setting the
force_locale_download: truein a backend or in global configuration adds additional certainty. The default setting is
falsethereby retaining compatibility with existing behaviour. The configuration can also be made dependent on
mixenvironment as shown in this example:
defmodule MyApp.Cldr do use Cldr, locales: ["en", "fr"], default_locale: "en", force_locale_download: Mix.env() == :prod end
The data maintained by CLDR is quite large and not all capabilities are required by all applications. Hence
Cldrhas additional optional functionality that can be provided through additional
hexpackages. In order to support compile-time additions to a configured
backend, any package can define a provider that will be called at compile time.
The currently known providers and their
hexpackage names are:
| Hex Package | Provider Module | Comment | | :------------------- | :---------------- | :------------------------------------------ | | excldrnumbers | Cldr.Number | Formatting of numbers, currencies | | excldrlists | Cldr.List | Formatting of lists | | excldrunits | Cldr.Unit | Formatting of SI and Imperial units | | excldrterritories | Cldr.Territory | Formatting of territory (country) data | | excldrlanguages | Cldr.Language | Formatting of language information | | excldrdatestimes | Cldr.DateTime | Formatting of dates, times & datetimes | | exmoney | Money | Operations and formatting of a money type | | ex_messages | Cldr.Message | Formatting of ICU-formatted messages |
Any library author can create a provider module by exposing a function called
cldr_backend_provider/1that takes a
Cldr.Configstruct as a single parameter. The function should return an AST that is inserted into the
backendmodule being compiled.
Providers are configured on each backend module under the
:providerskey. It must be a list of provider modules. For example:
elixir defmodule MyApp.Cldr do use Cldr, locales: ["en", "zh"], default_locale: "en", providers: [Cldr.Number, Cldr.List] endIf :providers is
nil(the default),
Cldrwill attempt to configure all of the providers described above if they have been installed as
deps. If you don't wish to invoke any providers, use the empty list
[].
backendmodule by following the configuration instructions
config.exsfiles. Only the keys
:default_localeand
:json_libraryare supported in the global configuration
Cldr.some_functionto
MyApp.Cldr.some_function. Or better still, alias your backend module where required. ie.
alias MyApp.Cldr, as: Cldr
Cldrcan be installed from either github or from hex.
If installed from github then all 541 locales are installed when the repo is cloned into your application deps.
If installed from hex then only the locales "en", "en-001" and "root" are installed. When you configure additional locales these will be downloaded during application compilation.
The
Cldr.Numbermodule implemented in the excldrnumbers package provides number formatting. The public API for number formatting is
MyApp.Cldr.Number.to_string/2. Some examples: ```elixir iex> MyApp.Cldr.Number.to_string 12345 "12,345"
iex> MyApp.Cldr.Number.to_string 12345, locale: "fr" "12 345"
iex> MyApp.Cldr.Number.to_string 12345, locale: "fr", currency: "USD" "12 345,00 $US"
iex> MyApp.Cldr.Number.to_string 12345, format: "#E0" "1.2345E4"
iex(> MyApp.Cldr.Number.to_string 1234, format: :roman "MCCXXXIV"
iex> MyApp.Cldr.Number.to_string 1234, format: :ordinal "1,234th"
iex> MyApp.Cldr.Number.tostring 1234, format: :spellout "one thousand two hundred thirty-four" ``
Seeh MyApp.Cldr.Number
andh MyApp.Cldr.Number.tostring
iniex` for further information.
The
Cldr.Listmodule provides list formatting and is implemented in the excldrlists package. The public API for list formating is
Cldr.List.to_string/2. Some examples: ```elixir iex> MyApp.Cldr.List.to_string(["a", "b", "c"], locale: "en") "a, b, and c"
iex> MyApp.Cldr.List.tostring(["a", "b", "c"], locale: "en", format: :unitnarrow) "a b c"
iex> MyApp.Cldr.List.tostring(["a", "b", "c"], locale: "fr") "a, b et c" ``
Seeh MyApp.Cldr.List
andh MyApp.Cldr.List.tostring
iniex` for further information.
The
Cldr.Unitmodule provides unit localization and is implemented in the excldrunits package. The public API for unit localization is
Cldr.Unit.to_string/3. Some examples: ```elixir iex> MyApp.Cldr.Unit.to_string 123, :gallon "123 gallons"
iex> MyApp.Cldr.Unit.to_string 1234, :gallon, format: :long "1 thousand gallons"
iex> MyApp.Cldr.Unit.to_string 1234, :gallon, format: :short "1K gallons"
iex> MyApp.Cldr.Unit.to_string 1234, :megahertz "1,234 megahertz"
iex> MyApp.Cldr.Unit.availableunits [:acre, :acrefoot, :ampere, :arcminute, :arcsecond, :astronomicalunit, :bit, :bushel, :byte, :calorie, :carat, :celsius, :centiliter, :centimeter, :century, :cubiccentimeter, :cubicfoot, :cubicinch, :cubickilometer, :cubicmeter, :cubicmile, :cubicyard, :cup, :cupmetric, :day, :deciliter, :decimeter, :degree, :fahrenheit, :fathom, :fluidounce, :foodcalorie, :foot, :furlong, :gforce, :gallon, :gallonimperial, :generic, :gigabit, :gigabyte, :gigahertz, :gigawatt, :gram, :hectare, :hectoliter, :hectopascal, :hertz, :horsepower, :hour, :inch, ...] ``
Seeh MyApp.Cldr.Unit
andh MyApp.Cldr.Unit.to_string
iniex` for further information.
Formatting of relative dates and date times is supported in the
Cldr.DateTime.Relativemodule implemented in the excldrdates_times package. The public API is
MyApp.Cldr.DateTime.to_string/2and
MyApp.Cldr.DateTime.Relative.to_string/2. Some examples: ```elixir iex> MyApp.Cldr.Date.tostring Date.utctoday() {:ok, "Aug 18, 2017"}
iex> MyApp.Cldr.Time.tostring Time.utcnow {:ok, "11:38:55 AM"}
iex> MyApp.Cldr.DateTime.tostring DateTime.utcnow {:ok, "Aug 18, 2017, 11:39:08 AM"}
iex> MyApp.Cldr.DateTime.Relative.to_string 1, unit: :day, format: :narrow {:ok, "tomorrow"}
iex> MyApp.Cldr.DateTime.Relative.to_string(1, unit: :day, locale: "fr") "demain"
iex> MyApp.Cldr.DateTime.Relative.to_string(1, unit: :day, format: :narrow) "tomorrow"
iex> MyApp.Cldr.DateTime.Relative.to_string(1234, unit: :year) "in 1,234 years"
iex> MyApp.Cldr.DateTime.Relative.to_string(1234, unit: :year, locale: "fr") "dans 1 234 ans" ```
There is an experimental plurals module for Gettext called
Cldr.Gettext.Plural. It is configured in
Gettextby:
elixir defmodule MyApp.Gettext do use Gettext, plural_forms: Cldr.Gettext.Plural end
Cldr.Gettext.Pluralwill fall back to
Gettextpluralisation if the locale is not known to
Cldr. This module is only compiled if
Gettextis configured as a dependency in your project.
Note that
Cldr.Gettext.Pluraldoes not guarantee to return the same
plural indexas
Gettext's own pluralization engine which can introduce some compatibility issues if you plan to mix plural engines.
Cldrprovides two plugs to aid integration into an HTTP workflow. These two plugs are:
Cldr.Plug.AcceptLanguagewhich will parse an
accept-languageheader and resolve the best matched and configured
Cldrlocale. The result is stored in
conn.private[:cldr_locale]which is also returned by
Cldr.Plug.AcceptLanguage.get_cldr_locale/1.
Cldr.Plug.SetLocalewhich will look for a locale in the several places and then call
Cldr.put_locale/2and
Gettext.put_locale/2if configured so to do. Finally, The result is stored in
conn.private[:cldr_locale]which is then available through
Cldr.Plug.SetLocale.get_cldr_locale/1. The plug will look for a locale in the following locations depending on the plug configuration:
path_params
query_params
body_params
cookies
accept-languageheader
session
See
Cldr.Plug.SetLocalefor a description of how to configure the plug.
In addition, note that when migrating from
ex_cldr1.x versions, a backend needs to be configured for both plugs. In the simplest case an example would be: ```elixir plug Cldr.Plug.SetLocale, apps: [:cldr], cldr: MyApp.Cldr
plug Cldr.Plug.AcceptLanguage, cldr_backend: MyApp.Cldr ```
If you are using
Cldr.Plug.SetLocalewithout Phoenix and you plan to use
:path_paramto identify the locale of a request then
Cldr.Plug.SetLocalemust be configured after
plug :matchand before
plug :dispatch. For example: ```elixir defmodule MyRouter do use Plug.Router
plug :match
plug Cldr.Plug.SetLocale, apps: [:cldr, :gettext], from: [:path, :query], gettext: MyApp.Gettext, cldr: MyApp.Cldr
plug :dispatch
get "/hello/:locale" do send_resp(conn, 200, "world") end end ```
If you are using
Cldr.Plug.SetLocalewith Phoenix and you plan to use the
:path_paramto identify the locale of a request then
Cldr.Plug.SetLocalemust be configured in the router module, not in the endpoint module. This is because
conn.path_paramshas not yet been populated in the endpoint. For example: ```elixir defmodule MyAppWeb.Router do use MyAppWeb, :router
pipeline :browser do plug :accepts, ["html"] plug :fetchsession plug Cldr.Plug.SetLocale, apps: [:cldr, :gettext], from: [:path, :query], gettext: MyApp.Gettext, cldr: MyApp.Cldr plug :fetchflash plug :protectfromforgery plug :putsecurebrowser_headers end
scope "/:locale", HelloWeb do pipe_through :browser
get "/", PageController, :index
end
end ```
Note that
Cldrdefines locale strings according to the IETF standard as defined in RFC5646.
Cldralso implements the
uextension as defined in RFC6067 and the
textension defined in RFC6497. This is also the standard used by W3C.
The IETF standard is slightly different to the ISO/IEC 15897 standard used by Posix-based systems; primarily in that ISO 15897 uses a "_" separator whereas IETF and W3C use "-".
Locale string are case insensitive but there are common conventions:
Unicode defines the U extension which support defining the requested treatment of CLDR data formats. For example, a locale name can configure the requested:
For example, the following locale name will request the use of the timezone
Australia/Sydney, and request the use of
accountingformat when formatting currencies:
elixir iex> MyApp.Cldr.validate_locale "en-AU-u-tz-ausyd-cf-account" {:ok, %Cldr.LanguageTag{ canonical_locale_name: "en-Latn-AU", cldr_locale_name: "en-AU", extensions: %{}, gettext_locale_name: "en", language: "en", language_subtags: [], language_variant: nil, locale: %{currency_format: :accounting, timezone: "Australia/Sydney"}, private_use: [], rbnf_locale_name: "en", requested_locale_name: "en-AU", script: "Latn", territory: "AU", transform: %{} }}The implementation of these extensions is governed by each library in the
ex_cldrfamily. As of January 2020, excldrnumbers version 2.10 implements the following
Uextension keys:
cf(currency format)
cu(currency)
nu(number system)
Other libraries in the family will progressively implement other extension keys.
-(dash), not a
_. (underscore). If you configure a
Gettextmodule then
Cldrwill transliterate
Gettext's
_into
-for compatibility.
pt-PTis the locale referring to Portugese as used in Portugal.
Cldra locale name is always a
binaryand never an
atom. Internally a locale is parsed and stored as a
Cldr.LanguageTagstruct.
Cldrcan be retrieved by
Cldr.known_locale_names/1to get the locales known to this configuration of
Cldrand
Cldr.all_locale_names/0to get the locales available in the CLDR data repository.
Tests cover the full 566 locales defined in CLDR. Since
Cldrattempts to maximize the work done at compile time in order to minimize runtime execution, the compilation phase for tests is several minutes.
Tests are run on Elixir 1.6 and later.
Cldrwill not run on Elixir versions before 1.6.
See the file
DEVELOPMENT.md