Text Encoding

• Encoding defines how bytes in a file map to characters
• A text file is just bytes — without knowing the encoding, you can’t correctly interpret the content
• IANA maintains character sets as well as encodings
  • https://www.iana.org/assignments/character-sets

Concepts

• Code Point
  • Logical ID given to single character
  • A
    • U+0041 (Unicode Code point)
    • 65 (ASCII Code point)
• Character Set
  • Set of Characters/Code Points
  • Examples
    • Unicode (every imaginable character on earth)
    • ASCII (128 characters)
• Encoding
  • Algorithm defining how code points are physically stored as binary bytes on disk
  • Traditionally character sets and encodings were fused together but in case of Unicode, it is different
  • If you know encoding you can determine the character set automatically
  • Examples
    • UTF-8 (1-4 bytes per character)
    • ANSI_X3.4-1968 — ASCII (1 byte per character)
    • ISO-8859-1 (1 byte per character)

ASCII

• 1 byte per character, 128 characters (0–127)
• Only covers English letters, digits, and basic symbols
• Any byte above 127 is undefined in ASCII
• 0-31: control characters (non-printable)
• 32-126: printable characters
• Tricks
  • 'A' to 'a' difference is 32 (0x20) - toggle case by flipping bit 5
  • Digit char to number: '5' - '0' = 53 - 48 = 5

Dec	Hex	Char	Notes
0	00	\0	null terminator (end of C strings)
9	09	\t	horizontal tab
10	0A	\n	line feed; newline on Unix/Linux
13	0D	\r	carriage return; part of Windows \r\n
27	1B	ESC	escape
32	20		space
48	30	0	first digit
57	39	9	last digit
65	41	A	first uppercase
90	5A	Z	last uppercase
97	61	a	first lowercase
122	7A	z	last lowercase
127	7F	DEL	delete

Unicode & UTF-8

• Unicode: a universal standard assigning a unique code point to every character in every language on earth
  • It is Character Set
  • U+0041 = A
  • U+1F600 = 😀
• Unicode Encodings
  • UTF-8: the most common encoding of Unicode on the web and Linux
    • Variable length: 1-4 bytes per character
    • Backwards compatible with ASCII: the first 128 code points are identical
    • hello is still 5 bytes in UTF-8, same as ASCII
  • UTF-16: 2 or 4 bytes per character; used internally by Windows and Java
  • UTF-32: fixed 4 bytes per character; simple but wasteful
• References
  • Joel Spolsky: https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/
  • Niki: https://tonsky.me/blog/unicode/

BOM (Byte Order Mark)

• Optional magic bytes at the very start of a text file declaring its encoding
• Similar in structure to magic numbers (fixed bytes at offset 0), but identifies encoding rather than file format
• UTF-8 BOM is optional and almost never used on Linux/macOS/Modern Windows
• Rarely used in practice today — UTF-8’s distinctive byte patterns are reliable enough for heuristic detection
• Can cause subtle bugs e.g. a script with a UTF-8 BOM will fail because #!/bin/bash is no longer at byte 0, or a CSV’s first column header has invisible \xEF\xBB\xBF prepended

Encoding	BOM bytes
UTF-8	EF BB BF
UTF-16 LE	FF FE
UTF-16 BE	FE FF

How Editors Detect Encoding

1. BOM — if present, definitive; editor reads first bytes and matches known signatures
2. Explicit declaration in file
   • XML <?xml version="1.0" encoding="ISO-8859-1"?>
   • HTML <meta charset="UTF-8">
3. Heuristics — if neither above exists:
   • All bytes 0–127 —> safe to assume ASCII/UTF-8
   • Valid UTF-8 multi-byte patterns (10xxxxxx continuation bytes) —> likely UTF-8
   • Many null bytes —> likely UTF-16
4. Configured default — editor’s fallback setting (VS Code: files.encoding)

• Without a BOM or declaration there is no guaranteed detection — wrong guess causes mojibake (garbled text)
  • e.g. 0xE9 is é in Latin-1 but invalid alone in UTF-8
• Platform defaults
  • Linux and macOS assume UTF-8; modern Windows (10 1903+) also defaults to UTF-8
  • Older Windows defaulted to system locale encodings (e.g. Windows-1252 for Western Europe) hence Notepad used to add BOM for UTF-8 files

iconv

• Converts a file from one encoding to another

iconv -f <from-encoding> -t <to-encoding> input.txt -o output.txt
 
# Convert Windows-1252 to UTF-8
iconv -f WINDOWS-1252 -t UTF-8 input.txt -o output.txt
 
# List all supported encodings
iconv -l

locale / LANG

• Linux uses the LANG environment variable to determine the default encoding

echo $LANG
# en_US.UTF-8
 
locale        # show all locale settings
locale -a     # list all available locales

• Format: language_TERRITORY.encoding e.g. en_US.UTF-8, de_DE.UTF-8
• This is what tools like file, iconv, and editors fall back to when no encoding is specified
• See Locale

Line Endings

• Different OSes use different byte sequences to mark the end of a line
  • Unix/Linux/macOS: LF
    • \n (0A)
  • Windows: CRLF
    • \r\n (0D 0A)
  • Old macOS (pre OS X)
    • CR - \r (0D)
• file detects line endings

file notes.txt
# notes.txt: ASCII text, with CRLF line terminators   -- Windows file
# notes.txt: ASCII text                               -- Unix file

• Converting between line endings

dos2unix file.txt      # CRLF --> LF
unix2dos file.txt      # LF --> CRLF

• Common issue: Windows-edited files checked into git on Linux showing ^M characters or diff noise
  • Fix in git: git config --global core.autocrlf input (macOS/Linux)