Node.js � Character Encodings

# Node.js � Character Encodings

Anna Henningsen Node.js CTC she/her

---

Why did this happen?

---

# So … what’s a character encoding?

Maps strings to some binary representation:

* A → `41`
* é → `c3 a9`
* あ → `e3 81 82`

This is _always_ necessary for storage/transfer of strings!

Most modern encodings are built upon/subset of *Unicode*:

! → 33 
" → 34 
\# → 35 
… 
🎉 → 127881

---

! → 33 
" → 34 
\# → 35 
… 
🎉 → 127881

Need a way to convert these *code points* into sequence of bytes:

UTF-8:

* 1–6 bytes/character
* Fully ASCII-compatible

UTF-16:

* 2–4 bytes/character
* JavaScript-y
* Big vs Little Endian

---

Non-Unicode character sets:

* ISO-8859-…
* ASCII
* more local charsets

ASCII < ISO-8859-1 < Unicode 
(character → code point mapping)

ASCII < UTF-8 
(string → bytes)

---

# Encodings are part of your APIs!

- Passing strings in buffers? You *need* to specify an encoding!  
- Best practice: Always use UTF-8 without BOM
- Other encodings are not necessarily easier/faster

UTF-8 nicenesses:
- Easy enough to detect
- No misinterpretations
- Efficient for most characters
- Keeps lexicograpical sorting

---

# UTF-8

<pre style="font-size:130%">
 0x00000000 - 0x0000007F:
 0xxxxxxx

0x00000080 - 0x000007FF:
     110xxxxx 10xxxxxx

0x00000800 - 0x0000FFFF:
     1110xxxx 10xxxxxx 10xxxxxx

0x00010000 - 0x001FFFFF:
 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
</pre>

---

# UTF-16

Characters >= U+10000 are turned into “surrogate pairs”: 
`'😊' === '\ud83d\ude0a'`

U+D800...U+DBFF for the first half 
U+DC00...U+DFFF for the second half

Actually encoding these as bytes leaves two possibilities:

- Ă → U+0102 → <code>01 02</code>
- Ă → U+0102 → <code>02 01</code>

For UTF-16, always prepend the BOM (U+FEFF) to distinguish!

---

# Pitfalls

- Using the wrong encoding
- Interpreting BOMs  
- Assuming Buffer and character boundaries align
- Splitting strings between UTF-16 surrogate code units
- Splitting inside combined characters, e.g. `á` vs `a´`
- Making assumptions about string display lengths

---

# Typical misinterpretations

> MÃ©xico

UTF-8 interpreted as ISO-8859-1

> M�xico

ISO-8859-1 interpreted as UTF-8

> 썍碩捩੯ / ䷃ꥸ楣漊

UTF-8 interpreted as UTF-16

---

# Handling all this in Node

Built-in encodings: `latin1`, `utf8`, `utf16le` and `ascii`

> Primary API: `Buffer.from('abc', 'utf8')`, `buffer.toString('utf8')`

**Don’t**:

```js
let data = '';
stream.on('data', chunk => { data += chunk });
```

This works 95 % of the time…

`chunk1 = <Buffer e2 88> ` ` chunk2 = <Buffer 9e>`

→ `data = '��'` 😞

---

# Better:

Use `.setEncoding()`:

```js
let data = '';
stream.setEncoding('utf8');
stream.on('data', chunk => { data += chunk });
```

**or**

```js
stream.pipe(bl((err, data) => { data.toString('utf8') … }));
```

Or, for `Writable` instances:
 `writable.setDefaultEncoding(enc)`

---

# Advanced builtins for character encoding handling

* Buffer → Buffer: `buffer.transcode(src, from, to)`
 * Uses ICU for transcoding  
* `string_decoder`: stream-like raw interface

```js
> const sd = new string_decoder.
      StringDecoder();
> sd.write(Buffer.from([0xE2,
                        0x82]))
''
> sd.end(Buffer.from([0xAC]))
'€'
```

---

# Web (& Node 8+) builtins

- `TextDecoder`, `TextEncoder` convert `Uint8Array` ↔ `string`

```js
const decoder = new TextDecoder('shift_jis');
let string = '';
let buffer;
while (buffer = getNextChunkSomehow()) {
  string += decoder.decode(buffer, { stream: true });
}
string += decoder.decode(); // end-of-stream
```

---

# Handling surrogates properly

* `'💩' === '\ud83d\udca9'`
* → `💩` has length 2, `'\ud83d'` doesn’t render on its own
* Don’t split in between

* Approach 1: Explicitly checking for surrogates
  * Fast, but not elegant
  * Test `str.codePointAt(i)` against range `\ud800 - \udfff`
* Approach 2: `[...str]`
  * Slow, but readable and modern

```js
[...'🅐🅑🅒'] ~ ['🅐', '🅑', '🅒']
```

---

# Surrogate handling II

* Avoid `str.charCodeAt(index)`
* `str.codePointAt()` is the cool way 😎

```js
'💩'.charCodeAt(0)  === 0xD83D
'💩'.charCodeAt(1)  === 0xDCA9
'💩'.codePointAt(0) === 0x1F4A9
```

* Likewise: `String.fromCharCode()` > `String.fromCodePoint()`

---

# Handling other encodings

* `iconv-lite`
* `iconv`

For browsers:

* `new TextEncoder('utf8').encode('abc') // → Uint8Array`
* `new TextDecoder('utf8').decode(bytes) // → String`

---

# ES6 Unicode regexes:

Always use `/.../u`

```js
> /"."/.test('"😊"')
false
> /"."/u.test('"😊"')
true
```

Also fixes:
- Quantifiers (e.g. in         `/😊*/u`)
- Unicode escapes (e.g.     `\u{1f389}`)
- Generally everything works for non-BMP characters

---

# One more Node.js pitfall

`crypto.createHash('sha256').update('über').digest('hex')`

Node v4: `e200d3dfd3c0de62c…`
Node v6: `b51c854170449c382…`

Why?

* Previously `hash('ç') === hash('ǧ')`
* Incompatible with impls defaulting to UTF-8
* Again: Always be explicit about encodings!
  * Specify the encoding, or pass in Buffers

---

# Much more weirdness:

- Right-to-Left/Left-to-Right indicators
- Zero-Width-Joiner sequences
- Variation selectors
- …

---

# Final notes

Legacy stuff:
- `latin1` is called `binary` in Node < 6 for … reasons
- `ucs2` is an alias for `utf16le` for … reasons

Node treats `base64` and `hex` like character encodings, but:
  - Base64 maps: Bytes → String
  - UTF-8 maps: String → Bytes

Things to do:
- Test with non-ASCII characters
- Test with emoji
- Depending on context, test with CJK characters

---

Questions?

Resources:

- http://www.unicode.org/faq/
- https://codepoints.net/ & `unicode(1)`
- manpages for Unicode, UTF-8
- Node.js Buffer docs