UTF-8000: Unlimited UTF-8

To test the reference implementation, you can run: pipx install UTF-8000

[!IMPORTANT] Disclaimer: UTF-8000 is an independent project and is ~~not~~ endorsed by or affiliated with the Unicode Consortium.

TL;DR & Examples

The following table illustrates the bit-pattern evolution from standard ASCII and UTF-8 into the expanded UTF-8000 specification.

Format	Bytes	Bit Pattern (Simplified)
ASCII	1	`0 xxxxxxx`
UTF-8	2	`110 xxxx` $\rightarrow$ `10 xxxxxx`
	3	`1110 xxxx` $\rightarrow$ `10 xxxxxx` $\rightarrow$ `10 xxxxxx`
	4	`11110 xxx` $\rightarrow$ `10 xxxxxx` $\rightarrow$ `10 xxxxxx` $\rightarrow$ `10 xxxxxx`
UTF-8000	5	`111110 xx` $\rightarrow$ `10 xxxxxx` $\times 4$
	6	`1111110 x` $\rightarrow$ `10 xxxxxx` $\times 5$
	7	`11111110` $\rightarrow$ `10 xxxxx` $\rightarrow$ `10 xxxxxx` $\times 5$
	8	`11111111` $\rightarrow$ `10 0 xxxxx` $\rightarrow$ `10 xxxxxx` $\times 6$
	9	`11111111` $\rightarrow$ `10 10 xxxx` $\rightarrow$ `10 x xxxxx` $\rightarrow$ `10 xxxxxx` $\times 6$
	10	`11111111` $\rightarrow$ `10 110 xxx` $\rightarrow$ `10 xx xxxx` $\rightarrow$ `10 xxxxxx` $\times 7$
...	...	...
	22	`11 111111` $\rightarrow$ `10 111111` $\rightarrow$ `10 111111` $\rightarrow$ `10 110 xxx` $\rightarrow$ `10 xx xxxx` $\rightarrow$ `10 xxxxxx` $\times 16$

The 22-byte example provided above isn't a "special case"; rather, it serves as a prototypical demonstration of how UTF-8000 utilizes multiple start bytes to handle massive codepoints.

The Only Special Cases

UTF-8000 maintains almost total consistency, inheriting only two specific exceptions from UTF-8:

ASCII: Handled exactly as is.
2-byte UTF-8: Requires $\text{4}$ mandatory content bits to prevent overlong encoding (whereas all longer units require $\text{5}$ ).

Anatomy of a Code Unit

Below is a conceptual breakdown of the 22-byte example mentioned in the TL;DR.

Key components identified in the anatomy:

Start Bytes: The initial sequence of bytes.
Final Start Byte: The last byte in the start sequence.
Continuation Bytes: All bytes following the start sequence.
Content Bits: The actual data bits of the codepoint.
Mandatory Content Bits: Specific bits used to validate encoding length; these may be split across the final start byte and the first non-start byte.

The core innovation of UTF-8000 is the clear methodology for dividing the first byte's highest bits into self-synchronization bits and start bits, and the subsequent "striping" of these start bits across continuation bytes to allow for infinite expansion.

Glossary

Terms are listed in order of conceptual introduction.

Term	Definition
Codepoint	An unsigned, non-negative integer.
Code Unit	The full sequence of bytes used to represent one codepoint.
First Byte	The unique byte that initiates a code unit. Its prefix is either `0` (for ASCII) or `11` (for multi-byte). Also called the first start byte or non-continuation byte.
Continuation Byte	Any byte following the first byte in a multi-byte sequence. It always begins with the prefix `10`.
Self-Synchronization Prefix	The leading bits of any byte that identify it as either a first byte or a continuation byte.

Deep Dive: Self-Synchronization

The prefixes are designed as a prefix-free tree:

0 $\rightarrow$ First byte (ASCII)
1
- 0 $\rightarrow$ Continuation byte
- 1 $\rightarrow$ First byte (Multi-byte)

This architecture allows for byte-level self-synchronization. If you jump to a random position in a file, you can immediately determine your context:

If the byte starts with 0 or 11, you are at a first byte and can start decoding.
If it starts with 10, you are in the middle of a unit and must seek forward to the next first byte.

This is critical for:

Random Access: No need to read the file from the beginning to determine state.
Error Recovery: If a corrupt byte (e.g., 0xC0) is found, the decoder can insert a replacement character $\text{U+FFFD}$ ( $\text{�}$ ) and skip ahead to the next valid first byte.

Understanding Start Bytes

Start bytes appear contiguously at the beginning of a code unit. By allowing multiple start bytes, UTF-8000 can encode codepoints of any size.

Note: In casual conversation, ASCII bytes are sometimes called start bytes, though they technically lack "start bits."
Structure: A code unit begins with the first byte, followed by zero or more continuation bytes that also function as start bytes.
UTF-8 vs UTF-8000: In standard UTF-8, the "start byte" and "first byte" are the same thing because UTF-8 only ever requires one start byte, whether usin...