String Encoding & Unicode
Transcript
Made automatically from the video's sound, so it may contain mistakes. Click a line to play from there.
Right, let's get straight into it. If you caught our last explainer on floating point maths, you'll remember how 64-bit binary floats hide silent, nasty rounding traps right beneath our everyday numbers. Well, strings do exactly the same thing.
They hide distinct encoding layers beneath what looks like a single, simple character on your screen. Just think about these JavaScript outputs for a second. You ask for the length of the face palm emoji, and it returns seven.
The rocket ship, two. I mean, why on earth is one emoji seven times longer than another? And honestly, it gets even weirder when you try some naive string slicing. Say you take a family emoji and try to slice off just the first character.
You literally amputate the family. You're left with just a single father. Or worse, one of those dreaded replacement diamonds. It completely shatters the illusion that strings are just simple arrays of characters, doesn't it?
But the sheer madness of text encoding really peaks when we try to check for equality. Imagine you have two visually identical pixel for pixel cafe strings. They look exactly the same.
Yet a strict JavaScript equality check says false. How can cafe not equal cafe? Well, it all comes down to the bytes hiding underneath. Welcome to the explainer!
Today we're tackling one of the genuinely hard problems in computing string encoding and Unicode. Our mission? To dive right beneath the surface of the screen, decode the multi layered architecture of strings, and figure out why our basic assumptions about text are just so often wrong.
Here's a quick look at our roadmap. We'll start with the illusion of text, dive into the four layer string stack, unpack normalization collisions, expose some terrifying invisible security attacks, and finally wrap up with a solid architecture blueprint.
The illusion of text, or why strings are not just character arrays. We really need to transition away from our foundational, everyday assumptions about how computers store human language, because frankly, they are fundamentally flawed. Using code points for iterating over a string is wrong.
They are not the basic unit of writing. One grapheme could consist of multiple code points. Basically, treating modern Unicode code points like legacy ASCII characters is a massive trap.
It's a trap developers fall into every single day. The four-layer string stack. We're gonna look at the actual structural architecture of how a string is built, working step by step from the bare metal all the way up to the screen.
Think of this like putting text under a microscope. Down at the bottom layer, we have our raw encoded bytes. Up one level, we get to code units. Above that, we hit code points, specifically the 21-bit Unicode scalar code points.
And finally, at the very top, we reach extended grapheme clusters. This top layer is what we humans actually perceive as a single character. Now, the contrast between encodings at those base layers is pretty stark. On one side you've got Rob Pike and Ken Thompson's brilliant UTF-8 standard from RFC 3629, variable 1 to 4 bytes, perfectly self-synchronizing.
On the other side, the legacy 16-bit UTF-16 surrogate pairs, which JavaScript, Java, and C sharp still heavily rely on. And you guessed it, those surrogate pairs are exactly what's causing those bizarre length bugs we saw with the emojis earlier.
So, up at that top layer, we have the UAX hashtag 29 extended grapheme cluster. It is essentially a horizontally segmentable unit that combines base letters, diacritics, UTS hashtag 51 Fitzpatrick skin tones, presentation selectors, and zero-width joiners.
Because a single character you see on screen can be a massive mashup of all these distinct parts. Safe modern languages like Rust explicitly ban integer indexing on strings. You just can't grab index zero safely anymore.
It's a thing of the past. Let's hit section three. Normalization and cross-system collisions. This is where we see what happens when identical text fails equality, and it's pure chaos.
Different operating systems and databases will literally store the exact same text using entirely different byte combinations. Think about how UAX number 15 normalization handles this. We have canonical NFC, which uses composed characters, so an E is stored as a single combined point.
But then you have NFD, which decomposes it into an E plus a combining accent mark. And then, oh boy, we have the destructive NFKC and NFKD compatibility forms. These will actually strip formatting away entirely and irreversibly turn something like a superscript 2 into a plain old 2.
And these normalization layers connect directly to some real historic disasters. We're talking MySQL's infamous 3-byte UTF-8 or UTF-8 MB3 truncation bugs. We're talking the 2018 iOS Telegu Glyphshaper crash. And then there's Mac OS APFS forcing NFD, which causes raw byte key collisions against Linux and AWS S3 environments that heavily prefer NFC.
These aren't just theoretical issues. They break raw byte lookups, they ruin password hashes, and they literally crash operating systems. Right, section 4.
Invisible Unicode Security Attacks. Hacking what you can't see. We urgently need to pivot from accidental system bugs to how malicious actors intentionally weaponize this invisible complexity against developers like us.
Seriously, think about this for a second. How do you hack a system using malicious code that you literally cannot see during a pull request review? It's a terrifying reality.
There are exploits that look perfectly normal, completely benign to a human reviewer, but they secretly instruct the machine to do something completely different. Take the IDN homograph attack, for example. You could have an ASCII Apple.com right next to a Cyrillic confusable Apple.com.
They are pixel identical. The human eye sees the exact same word, but because of those UTS39 confusibles, the computer reads a completely different set of bytes underneath. It is literally the perfect phishing domain.
Beyond phishing, we have things like Trojan Source, CVE-2021-42574, which uses bidirectional overrides to physically flip the rendering of text, hiding logic backdoors directly in the code itself. And then you've got plane 14 tags, which let attackers smuggle entirely invisible prompt injections into large language models without triggering any basic filters.
You literally cannot see the attack happening. All of this leads us straight into what we call the vibe coder trap. Relying on default AI code generation often spits out naive string slicing, like taking a substring up to 140 characters, or using basic ASCII regexes for global names, or storing unnormalized Argonne 2 ID password hashes.
When you rely on these naive defaults or confuse UI character counts with UTF-8 bytecaps, you are opening the door wide for these invisible Unicode attacks. Finally, section 5, the architecture blueprint, our modern string handling survival guide.
Let's strip away the panic and lay out a pragmatic, authoritative blueprint for handling strings safely in modern systems. Here are the concrete rules you need to follow. First, enforce UTF 8 everywhere, specifically UTF-8 MB4 in your databases.
Second, use NFC canonicalization on ingress, especially for passwords following OWASP guidance. Though, quick side note, avoid blind NFKC on pros. Third, rely exclusively on intl.segmenter for grapheme truncation in your UI.
And finally, always use buffer.byte length for checking your storage limits. Oh, and absolutely remember to strip by die controls and plane 14 tags on any untrusted input. However, please apply the you aren't gonna need it architecture principle here.
Do not overengineer. You absolutely should keep fast order one ASCII byte indexing for internal identifiers. Things like UUIDs, hex hashes, and ISO timestamps.
Simple internal IDs just do not need complex graphene cluster logic. Keep it fast and simple. Well, we have covered a massive amount of ground today on string architecture. But I want to leave you with a final provocative thought.
Let's say your strings are fully secured, properly encoded, and perfectly normalized. What happens when your architecture serves that perfectly encoded string from a stale cache?
Have a think about that, because it sets us up perfectly for our next explainer on cache invalidation. See you then.