Text is bytes: ASCII, UTF-8 and UTF-16
A string has at least four different lengths. Learn which one each limit means, and how a careless slice ships broken text to a model.
Unicode assigns every character a number called a code point, written like U+1F642. An encoding turns code points into bytes. The two you meet every day are UTF-8, used by files, HTTP and most APIs, and UTF-16, used inside JavaScript, Java and Windows strings.
| Code point range | UTF-8 bytes | UTF-16 code units | Example |
|---|---|---|---|
| U+0000 to U+007F | 1 | 1 | A, {, 7 |
| U+0080 to U+07FF | 2 | 1 | é, ñ, ß |
| U+0800 to U+FFFF | 3 | 1 | 日, €, ✓ |
| U+10000 and above | 4 | 2 (surrogate pair) | 🙂, 𝄞 |
UTF-8 is backward-compatible with ASCII: every ASCII byte means the same thing. Bytes of a multi-byte character always start with 10, so a decoder can tell a continuation byte from the start of a character. That property makes UTF-8 self-synchronising, which matters when a stream is cut in the middle.
One string, four lengths
| Grapheme | H | i | ␠ | 🙂 |
|---|---|---|---|---|
| Code points | U+0048 | U+0069 | U+0020 | U+1F642 |
| UTF-8 bytes | 48 | 69 | 20 | F09F9982 |
| UTF-16 units | 0048 | 0069 | 0020 | D83DDE42 |
.length- 5
- UTF-8 bytes
- 7
- Code points
- 4
- Graphemes
- 4
Broken: ends with a lone surrogate 0xD83D. Encoded to UTF-8 it becomes the replacement character.
.slice(0, n) cuts a surrogate pair in half.The four lengths
- UTF-16 code units: what JavaScript's
.lengthcounts."🙂".lengthis 2. - Bytes: what a network or storage limit counts.
new TextEncoder().encode("🙂").lengthis 4. - Code points: what
[...str].lengthcounts.[..."🙂"].lengthis 1. - Graphemes: what a person counts as one character. A family emoji such as 👨👩👧 is 1 grapheme, 5 code points, 8 UTF-16 units and 18 UTF-8 bytes.
A fifth length matters in agent work: tokens. A tokenizer splits text into pieces from a learned vocabulary. Common English words are often one token, while rare words, code identifiers and emoji may take several. You cannot convert between bytes and tokens with a constant.
Worked example: truncating tool output
A tool returns 2 MB of logs and the harness keeps the first 4,000 characters. .slice(0, 4000) can end between the two halves of a surrogate pair. The result contains a lone surrogate, which is not valid Unicode. Encoding it to UTF-8 produces the replacement character �, and some JSON encoders reject it outright.
const encoder = new TextEncoder();
const decoder = new TextDecoder("utf-8", { fatal: false });
/** Truncate to at most maxBytes of UTF-8 without splitting a character. */
export function truncateUtf8(text: string, maxBytes: number) {
const bytes = encoder.encode(text);
if (bytes.length <= maxBytes) return { text, truncated: false };
let end = maxBytes;
// Step back past continuation bytes (10xxxxxx) to a character boundary.
while (end > 0 && (bytes[end] & 0b1100_0000) === 0b1000_0000) end--;
return {
text: decoder.decode(bytes.subarray(0, end)),
truncated: true,
omittedBytes: bytes.length - end,
};
}
// Graphemes: do not split 👨👩👧 or é written as e + U+0301
const segmenter = new Intl.Segmenter("en", { granularity: "grapheme" });
export const graphemeCount = (s: string) => [...segmenter.segment(s)].length;Report what was cut. A note like “truncated 1.8 MB, full output saved to /tmp/run-42.log” lets the model ask for more.
Where you will meet it
Production harnesses cap tool results by bytes or tokens, never by .length, and save the overflow to a file the model can read later.
MySQL's legacy utf8 charset stores at most 3 bytes per character, so emoji fail to insert. utf8mb4 fixes it. Postgres limits are in bytes too.
Model context windows are measured in tokens. Use the provider's token-counting endpoint or tokenizer to budget, then keep a safety margin.
Check yourself
Answer out loud before you open each one.
What does "🙂".length return in JavaScript, and why?
2. JavaScript strings are UTF-16, and 🙂 (U+1F642) is above U+FFFF, so it needs a surrogate pair.
How many UTF-8 bytes does é take? And 日?
é (U+00E9) takes 2 bytes. 日 (U+65E5) takes 3.
Why is str.length wrong for a byte limit or a token budget?
It counts UTF-16 code units. Bytes depend on the encoding, and tokens depend on the tokenizer's vocabulary. All three diverge on non-ASCII text.
Try it
Take a 10 KB sample of real tool output with some emoji and accented names. Print its .length, UTF-8 byte length, code point count and grapheme count, then count its tokens with your model provider's tokenizer. Note the ratios.