Learn
What Is Unicode?
One number for every character, in every script.
Short answer
Unicode gives every character in every writing system one unambiguous number, called a code point. Before it, the same byte meant different letters in different systems, so text sent between them arrived scrambled.
The problem it solved
Before Unicode, each language region used its own byte encoding. The same byte meant different letters in different systems, so text sent between them arrived scrambled. Unicode assigns every character a single unambiguous number, called a code point, so text means the same thing everywhere.
Code points and encodings
A code point is written like U+0041 for A. How that number becomes bytes is a separate choice: UTF-8 uses one to four bytes per character and is now standard on the web, largely because plain ASCII stays a single byte.
What mojibake actually is
When text arrives as a run of nonsense characters, what has happened is that bytes encoded in one scheme were decoded using another. The bytes are intact; only the interpretation is wrong. Before Unicode this was routine in any correspondence crossing a language boundary, and the reason it is now rare is that almost everything agreed on one encoding.
Why UTF-8 won
Its decisive property is that plain ASCII remains a single byte per character and is byte-identical to what came before. That meant existing English-language systems kept working unchanged while gaining the ability to represent every other script, so adoption cost almost nothing — a rare case where the technically sound option was also the easy one.
Read next
Sources
- Unicode Consortium — Code Charts
The official code charts for every block referenced here.
- RFC 3629 — UTF-8, a transformation format of ISO 10646
The specification defining UTF-8's one-to-four byte encoding.