Unicode and UTF-8 for Software Engineers: What Actually Breaks
Text handling bugs show up in production as garbled names, truncated emoji and failed logins. A short mental model prevents most of them, and it is a frequent follow-up in coding interviews ("reverse this string", "count characters").
Three layers
- Code points: Unicode assigns every character a number, written like U+00E9 (é) or U+1F600 (an emoji). There are over a million possible code points.
- Encodings: UTF-8 stores each code point as 1 to 4 bytes and is backward compatible with ASCII. UTF-16 uses 2 or 4 bytes (surrogate pairs above U+FFFF) and is what JavaScript strings and Java char use internally.
- Grapheme clusters: what a user sees as one character may be several code points: a letter plus a combining accent, a flag (two regional indicators), or a family emoji joined with zero-width joiners.
Where code breaks
- Length: "length" may mean bytes, UTF-16 units, code points or graphemes. A database column limit in bytes can cut a character in half. Validate user-facing limits in graphemes and storage limits in bytes.
- Reversing or slicing: reversing by UTF-16 units splits surrogate pairs; reversing by code points detaches accents. Use a grapheme-aware library.
- Comparison: é can be one code point or e plus a combining mark. Normalize (NFC is common) before comparing or using text as a key, for example usernames.
- Case: case mapping is locale dependent (the Turkish dotted and dotless i) and can change length (German ß uppercases to SS). Use case folding for case-insensitive matching.
- Mojibake: decoding UTF-8 bytes as Latin-1 turns é into é. Declare encodings explicitly at every boundary: HTTP headers, files, database connections.
- MySQL's legacy utf8 charset holds only 3 bytes per character, so emoji fail; use utf8mb4.
- Security: visually identical characters from different scripts (homoglyphs) enable spoofed usernames and domains; restrict scripts or apply confusable checks.
Rules of thumb
Use UTF-8 everywhere at the edges, decode bytes to text as early as possible and encode as late as possible, normalize identifiers, and never assume one character equals one byte.
Interview tip
If asked to reverse or truncate a string, ask whether the input may contain non-ASCII text. Mentioning code points versus graphemes is a strong signal of real-world experience.
Disclosure: I help build Linkjob AI, an AI copilot for job interviews - https://www.linkjob.ai/