Experiments

4 min read

File Formats

Identifying File Formats

  • The files are generally identified by:
    • Filename Extension
    • File Header
    • Magic Number
  • file command can be used to detect the file type
    • it does not rely on extension
    • it checks file header and magic bytes
    • it uses uses libmagic library to detect files
      • libmagic loads /usr/share/misc/magic.mgc database containing both file header and magic bytes
      • then uses pattern matching to identify file type

Filename Extension

  • Used by Windows and MacOS
  • traditionally 3 letter since older FAT system had this limitation
  • but now can be any number of letters

File Header

  • Binary files have a binary header at the start containing metadata: format version, size, resolution, color space, etc.
  • Structured text formats have textual headers e.g. <?xml version="1.0"?>, <!DOCTYPE html>
  • Plain .txt files have no header - the file starts directly with the content bytes
  • file command can identify using file header
> file index.html
index.html: HTML document, ASCII text

Magic Number

  • Plain .txt files have no magic number — writing hello produces exactly 5 bytes (68 65 6C 6C 6F), nothing before it
echo -n "hello" > test.txt
 
xxd test.txt
# 00000000: 6865 6c6c 6f   hello
 
wc -c test.txt
# 5 test.txt
  • Shebang (#!/bin/bash) is an example of a textual magic number — must be at byte 0 for the kernel to recognize it
    • XML/HTML are not reliable magic numbers since comments or whitespace can appear before the identifier
  • Binary formats can also have ASCII-readable signatures e.g. PDF starts with %PDF — but the rest of the file is binary
  • Many designers deliberately chose readable ASCII bytes so the format is self-describing in a hex dump
FormatHexASCIIMnemonic
GIF47 49 46 38 37 61GIF87aformat name + version
PDF25 50 44 46%PDFliteral %PDF
ZIP/JAR50 4B 03 04PK..Phil Katz — creator of PKZIP
Windows EXE4D 5AMZMark Zbikowski — MS-DOS developer
BMP42 4DBMBitMap
ELF (Linux bin)7F 45 4C 46.ELF”ELF” with leading non-ASCII byte
MP3 (ID3)49 44 33ID3literal ID3
JPEGFF D8 FFÿØÿnon-ASCII, no mnemonic
PNG89 50 4E 47 0D 0A 1A 0A.PNG\r\n.\nengineered to detect corruption
  • PNG magic bytes are the most deliberate design:
    • 89 — non-ASCII, prevents being misread as a text file
    • PNG — human-readable identification
    • 0D 0A — Windows CR+LF: if a transfer converts line endings this changes, failing the magic check
    • 1A — DOS Ctrl-Z (EOF marker): detects DOS text-mode corruption
    • 0A — Unix LF: detects the reverse conversion
  • Inspecting magic numbers
    • hexdump or xxd or od (octal dump)
      • View raw bytes
    • strings <file>
      • Extracts printable ASCII strings from a binary
      • Useful for quick inspection without a hex editor
# type of file
> file picture.jpg
picture.jpg: JPEG image data, JFIF standard 1.01
 
# Hex representation output: [Offset]: [16 bytes] [ASCII]
# 2-letter hex = 1 byte
# Next offset after first line with 16 bytes = 17 (decimal) = 10 (hex)
# [FF D8 FF]: JPEG magic bytes
> xxd picture.jpeg | head -2 # first two lines of output
00000000: ffd8 ffe0 0010 4a46 4946 0001 0100 0001  ......JFIF......
00000010: 0001 0000 ffdb 0043 0006 0405 0605 0406  .......C........
 
# Extract readable text from binary
> strings picture.jpeg | head -1
JFIF

MIME Type

file --mime-type picture.jpg
# picture.jpg: image/jpeg
  • MIME Sniffing: when Content-Type is missing or wrong, browsers inspect the first bytes of the response to guess the type
    • Security risk: a server serving a malicious script as text/plain may still be executed if the browser sniffs it as text/javascript
    • Mitigated by the X-Content-Type-Options: nosniff response header