A time and space-efficient implementation of Unicode text. Suitable for performance critical use, both in terms of large data quantities and high speed.

Note: Read below the synopsis for important notes on the use of this module.

This module is intended to be imported qualified, to avoid name clashes with Prelude functions, e.g.

import qualified Data.Text as T

To use an extended and very rich family of functions for working with Unicode text (including normalization, regular expressions, non-standard encodings, text breaking, and locales), see the text-icu package.

This package provides both strict and lazy Text types. The strict type is provided by the Data.Text module, while the lazy type is provided by the Data.Text.Lazy module. Internally, the lazy Text type consists of a list of strict chunks.

The strict Text type requires that an entire string fit into memory at once. The lazy Text type is capable of streaming strings that are larger than memory using a small memory footprint. In many cases, the overhead of chunked streaming makes the lazy Text type slower than its strict counterpart, but this is not always the case. Sometimes, the time complexity of a function in one module may be different from the other, due to their differing internal structures.

Each module provides an almost identical API, with the main difference being that the strict module uses Int values for lengths and counts, while the lazy module uses Int64 lengths.

A Text value is a sequence of Unicode scalar values, as defined in §3.9, definition D76 of the Unicode 5.2 standard. As such, a Text cannot contain values in the range U+D800 to U+DFFF inclusive. Haskell implementations admit all Unicode code points (§3.4, definition D10) as Char values, including code points from this invalid range. This means that there are some Char values (corresponding to Surrogate category) that are not valid Unicode scalar values, and the functions in this module must handle those cases.

Within this module, many functions construct a Text from one or more Char values. Those functions will substitute Char values that are not valid Unicode scalar values with the replacement character "�" (U+FFFD). Functions that perform this inspection and replacement are documented with the phrase "Performs replacement on invalid scalar values". The functions replace invalid scalar values, instead of dropping them, as a security measure. For details, see Unicode Technical Report 36, §3.5.)

This package uses the term character to denote Unicode code points.

Note that this is not the same thing as a grapheme (e.g. a composition of code points that form one visual symbol). For instance, consider the grapheme "ä". This symbol has two Unicode representations: a single code-point representation U+00E4 (the LATIN SMALL LETTER A WITH DIAERESIS code point), and a two code point representation U+0061 (the "A" code point) and U+0308 (the COMBINING DIAERESIS code point).

Starting from text-1.3 fusion is no longer implicit, and pipelines of transformations usually allocate intermediate Text values. Users, who observe significant changes to performances, are encouraged to use fusion framework explicitly, employing Data.Text.Internal.Fusion and Data.Text.Internal.Fusion.Common.