Workspace

Text is bytes: ASCII, UTF-8 and UTF-16

3 day streakD

Text is bytes: ASCII, UTF-8 and UTF-16

A string has at least four different lengths. Learn which one each limit means, and how a careless slice ships broken text to a model.

BasicsLesson 2 of 616 min

Unicode assigns every character a number called a code point, written like U+1F642. An encoding turns code points into bytes. The two you meet every day are UTF-8, used by files, HTTP and most APIs, and UTF-16, used inside JavaScript, Java and Windows strings.

Code point rangeUTF-8 bytesUTF-16 code unitsExample
U+0000 to U+007F11A, {, 7
U+0080 to U+07FF21é, ñ, ß
U+0800 to U+FFFF31日, €, ✓
U+10000 and above42 (surrogate pair)🙂, 𝄞

UTF-8 is backward-compatible with ASCII: every ASCII byte means the same thing. Bytes of a multi-byte character always start with 10, so a decoder can tell a continuation byte from the start of a character. That property makes UTF-8 self-synchronising, which matters when a stream is cut in the middle.

Figure 2.1

One string, four lengths

GraphemeHi␠🙂
Code pointsU+0048U+0069U+0020U+1F642
UTF-8 bytes486920F09F9982
UTF-16 units004800690020D83DDE42
.length
5
UTF-8 bytes
7
Code points
4
Graphemes
4
“Hi �”

Broken: ends with a lone surrogate 0xD83D. Encoded to UTF-8 it becomes the replacement character.

Each column is one user-perceived character, a grapheme. Below it are its code points, UTF-8 bytes and UTF-16 code units. Drag the slice to see where .slice(0, n) cuts a surrogate pair in half.

The four lengths

  • UTF-16 code units: what JavaScript's .length counts. "🙂".length is 2.
  • Bytes: what a network or storage limit counts. new TextEncoder().encode("🙂").length is 4.
  • Code points: what [...str].length counts. [..."🙂"].length is 1.
  • Graphemes: what a person counts as one character. A family emoji such as 👨‍👩‍👧 is 1 grapheme, 5 code points, 8 UTF-16 units and 18 UTF-8 bytes.

A fifth length matters in agent work: tokens. A tokenizer splits text into pieces from a learned vocabulary. Common English words are often one token, while rare words, code identifiers and emoji may take several. You cannot convert between bytes and tokens with a constant.

Worked example: truncating tool output

A tool returns 2 MB of logs and the harness keeps the first 4,000 characters. .slice(0, 4000) can end between the two halves of a surrogate pair. The result contains a lone surrogate, which is not valid Unicode. Encoding it to UTF-8 produces the replacement character �, and some JSON encoders reject it outright.

truncate.tsTypeScript
const encoder = new TextEncoder();
const decoder = new TextDecoder("utf-8", { fatal: false });

/** Truncate to at most maxBytes of UTF-8 without splitting a character. */
export function truncateUtf8(text: string, maxBytes: number) {
  const bytes = encoder.encode(text);
  if (bytes.length <= maxBytes) return { text, truncated: false };

  let end = maxBytes;
  // Step back past continuation bytes (10xxxxxx) to a character boundary.
  while (end > 0 && (bytes[end] & 0b1100_0000) === 0b1000_0000) end--;

  return {
    text: decoder.decode(bytes.subarray(0, end)),
    truncated: true,
    omittedBytes: bytes.length - end,
  };
}

// Graphemes: do not split 👨‍👩‍👧 or é written as e + U+0301
const segmenter = new Intl.Segmenter("en", { granularity: "grapheme" });
export const graphemeCount = (s: string) => [...segmenter.segment(s)].length;

Report what was cut. A note like “truncated 1.8 MB, full output saved to /tmp/run-42.log” lets the model ask for more.

Where you will meet it

Tool output limits

Production harnesses cap tool results by bytes or tokens, never by .length, and save the overflow to a file the model can read later.

Database columns

MySQL's legacy utf8 charset stores at most 3 bytes per character, so emoji fail to insert. utf8mb4 fixes it. Postgres limits are in bytes too.

Prompt budgets

Model context windows are measured in tokens. Use the provider's token-counting endpoint or tokenizer to budget, then keep a safety margin.

Check yourself

Answer out loud before you open each one.

What does "🙂".length return in JavaScript, and why?

2. JavaScript strings are UTF-16, and 🙂 (U+1F642) is above U+FFFF, so it needs a surrogate pair.

How many UTF-8 bytes does é take? And 日?

é (U+00E9) takes 2 bytes. 日 (U+65E5) takes 3.

Why is str.length wrong for a byte limit or a token budget?

It counts UTF-16 code units. Bytes depend on the encoding, and tokens depend on the tokenizer's vocabulary. All three diverge on non-ASCII text.

Try it

Take a 10 KB sample of real tool output with some emoji and accented names. Print its .length, UTF-8 byte length, code point count and grapheme count, then count its tokens with your model provider's tokenizer. Note the ratios.

A good place to start

Explore a pattern

A little guidance

Start with a pattern, read through a problem, and mark it solved when the idea clicks. Save anything you want to revisit.

New tutorials are added one at a time. Your bookmarks and solved checkmarks stay in this browser.

Explore the roadmap