Experiments

9 min read

Linux Filesystem

Filesystem
├── Reserved / Metadata Region
│   ├── Boot block (optional / reserved)
│   ├── Superblock
│   ├── Block bitmap
│   ├── Inode bitmap
│   └── Inode table
│
└── Allocatable Region
    └── Data blocks
  • inode calculation example??

Block

  • aka logical blocks
  • small fixed-size unit of storage used by the filesystem
  • often 4KB
  • Filesystem divides disk space into blocks
  • They can store
    • Superblock data
    • Inode tables
    • Bitmaps
    • Data blocks

Superblock

  • metadata about entire filesystem as a whole such as size
    • Total blocks
    • Free blocks
    • Block size
    • Total Inodes
    • Free Inodes
    • Filesystem type
    • Filesystem state
# for ext4 formatted filesystems
dumpe2fs /dev/sda3
tune2fs -l /dev/sda3

Inode

  • Index node
  • Stores metadata about a file, except its filename
    • owner
    • permissions
    • timestamps
    • file type
    • link count
    • pointers to data blocks
  • See File metadata
# inode metadata
stat <file/directory>
 
# inode number of file
ls -i <file>
# inode number of directory
ls -id <directory>

Directory

  • A directory is a special type of file used to organize files and folders
  • It contains multiple directory entries
  • A directory inode points to the data blocks containing its directory entries
  • Directory Entry
    • maps a filename to an inode number
notes.txt  --> inode 123
image.png  --> inode 124
report.pdf --> inode 125
  • Check directory contents along with inodes
ls -il <directory>

Inode table

  • A filesystem structure that stores all inodes
  • Each inode occupies an entry in the inode table
  • inode table is stored in the metadata region of filesystem along with inode bitmap
  • Directory entries use inode numbers to locate the corresponding inode

Data block

  • A block used specifically for storing actual file contents
  • Inodes point to several data blocks which are used to store the data in the file
  • inode —> block1, block2, block3
  • Indirect block: Inode has limited space for block pointers
    • If a file needs more data blocks than can be directly referenced by the inode, the inode can point to another block called an indirect block
    • inode —> indirect block —> block101, block102, block103
    • Indirect blocks store pointers/addresses, not actual file contents
    • These extra pointer blocks are allocated dynamically when more block references are needed
  • Data blocks can contain:
    • File contents
    • Directory contents
    • Indirect block information (block addresses)

Bitmap

  • A compact data structure used by the filesystem to track allocation status of resources
  • Each bit represents whether a resource is free or in use
  • Types:
    • Block bitmap
      • Tracks free/used data blocks
    • Inode bitmap
      • Tracks free/used inodes

Journaling

  • A filesystem mechanism that records intended filesystem changes in a journal before applying them to the actual filesystem
  • Helps maintain consistency and recover from crashes or power failures
  • Flow
    • Filesystem operation
    • Write change to journal
    • Apply filesystem changes
    • Mark journal entry complete

File descriptors

  • A file descriptor is an integer used to reference an opened file or I/O resource
  • It can represent files, sockets, pipes, terminals, and other I/O resources
  • Each process gets a File Descriptor Table (per-process, but lives in kernel memory)
    • Programs run in user mode and never directly access the FDT
    • The kernel maintains a separate FDT for each process inside its task_struct (as files_struct)
    • When a syscall like open() or read() is made, the CPU switches to kernel mode, the kernel looks up the FDT, and returns to user mode with just the result
    • Maps fd integer (index) —> pointer to open file description (struct file)
    • Userspace only ever sees the fd integer; all actual pointers stay inside the kernel
  • Standard File descriptor
    • 0 —> stdin
    • 1 —> stdout
    • 2 —> stderr
# List down file descriptors of a process
ls -l /proc/{pid}/fd
 
# See info of fd from open file description
cat /proc/{pid}/fdinfo/{fd}
  • There is a limit to number of file descriptors that can be opened by a process
# provides control over resources available to the shell and any child processes it creates
# check limit of open file descriptors
ulimit -n # Ubuntu
# 1048576
 
# see all limits
ulimit -a

System-wide Open File Table

  • A kernel-managed global table containing open file descriptions
  • Shared across the entire operating system
  • File descriptors from different processes may reference the same open file description
    • fork() — child inherits parent’s fds, both point to the same open file description (shared offset)
    • So, the parent and child share the exact same cursor/offset. If the child reads 10 bytes, the parent’s offset moves forward too.
  • Two independent processes calling open() on the same file get separate open file descriptions, each with their own offset
    • Each process maintains its own reading/writing cursor position, preventing them from accidentally overwriting each other’s progress.
    • Even though they have separate entries in this table, both entries point to the exact same single entry in the Inode Table.
  • Each open file description contains:
    • File Offset: The exact byte location where the process is currently reading or writing
    • Status Flags: File access mode and behavior
      • O_RDONLY, O_WRONLY, O_APPEND, O_NONBLOCK
    • Reference Count: Tracks how many different processes are using this specific file description
    • Inode Reference: Points to the corresponding inode
# list open files
lsof -p <pid>
 
# list open files based on internet address
lsof -i[TCP|UDP][@address][:port]

File Descriptor Layers

  • These tables are all in-memory structures managed by the kernel
  • File Descriptor Table (per-process)
    • maps: fd integer --> open file description (Open File Table)
  • Open File Table (system-wide)
    • maps: open file descriptions --> inode (Inode Table)
  • Inode Table (system-wide/VFS layer)
    • not to be confused with Disk Inode Table
    • contains metadata and points to data blocks

Open File Flow

  • open("/path/to/notes.txt") system call
  • Filesystem searches directory
  • Directory entry found
    • notes.txt --> inode 123
  • Read inode 123 and get the Metadata
  • Kernel creates an open file description
    • Initializes file offset
    • Stores status flags (read/write/append, etc.)
    • References the inode
  • Kernel creates an entry for fd in the process file descriptor table
  • Return fd to application

Read File Flow

  • read(fd, buffer, size) system call
  • Kernel locates inode
  • Follow direct/indirect block pointers as needed
  • Read data blocks
  • Copy content to buffer
  • Return bytes read

Delete File Flow

  • unlink("notes.txt") system call
  • Filesystem locates the directory entry
  • Directory entry is removed
    • notes.txt --> inode 123 mapping is deleted
  • Inode (hard) link count is decreased
  • Check whether link count becomes 0
    • If No:
      • Other hard links still exist
      • File remains alive
    • If Yes
      • Check whether the file is still open by a process
      • If Yes:
        • Keep inode and data blocks
        • File remains alive until all open references are closed
      • If No:
        • Data blocks become free
        • Inode becomes free
        • Block/inode Bitmaps are updated

Close File Flow

  • close(fd) system call
  • Kernel removes the file descriptor
  • Open file references are decreased
  • If hard link count = 0 and open file references = 0
    • Data blocks become free
    • Inode becomes free
    • Block/inode Bitmaps are updated

VFS (Virtual File System)

  • aka Virtual Filesystem switch
  • A kernel abstraction layer that provides a uniform interface between user space and different filesystem implementations
  • User space programs call the same system calls (open, read, write) regardless of the underlying filesystem
  • VFS translates these calls into filesystem-specific operations (ext4, xfs, tmpfs etc.)
  • Enables mounting different filesystem types at different mount points simultaneously
flowchart TD
    A["User Space Applications (nano, ls, cp)"]
    B["VFS - Virtual File System"]
    C[ext4]
    D[exfat]
    E[tmpfs]
    F["Main Disk (HDD / SSD)"]
    G["USB Drive"]
    H["System RAM"]

    A -->|"System Calls <br> (open, read, write)"| B
    B --> C & D & E
    C --> F
    D --> G
    E --> H

VFS Core Objects

  • VFS defines four core in-memory objects - each mirrors a concept already in the filesystem
  • The dentry cache (dcache) is why repeated ls or open calls on the same path are fast
    • filename —> inode mapping is already in memory
  • When a filesystem is mounted, it registers its own implementations of these objects with VFS
VFS ObjectFilesystem conceptDescription
superblockSuperblockRepresents a mounted filesystem instance
Holds metadata like block size, inode count
inodeInodeIn-memory representation of a file/directory
Populated from disk inode on first access
dentryDirectory EntryMaps a filename to an inode
Cached in the dentry cache to avoid repeated disk lookups
fileOpen File DescriptionRepresents an open file for a process
Holds offset, flags, pointer to dentry

Filesystem and Partitions

  • A filesystem is created inside a partition, not on the raw disk
    • /dev/sda = whole disk (contains partition table: MBR or GPT)
    • /dev/sda1, /dev/sda2 = partitions — each gets its own filesystem
  • One partition = one filesystem (standard)
  • All filesystem structures (superblock, inode table, bitmaps, data blocks) are scoped within the partition
# create ext4 filesystem on a partition
mkfs.ext4 /dev/sda1

Common Linux Filesystems

  • List of filesystems currently mounted: /proc/mounts
    • mount is legacy tool to view the same data
    • findmnt gives a human-friendly tree view

Disk Filesystems

  • ext4 (Fourth Extended Filesystem)
    • Default on Ubuntu, Debian, and most general-purpose distros
    • Supports journaling, extents, large files
    • Max file size: 16TB; max volume: 1EB
  • xfs
    • Default on RHEL, CentOS, Fedora
    • High performance for large files and parallel I/O
    • Supports journaling; cannot shrink a volume
  • btrfs (B-tree Filesystem)
    • Modern filesystem with advanced features
    • Built-in snapshots, subvolumes, checksums, transparent compression
    • Default on openSUSE; used by Fedora
  • vfat / exfat
    • Cross-platform (Windows/macOS/Linux)
    • vfat (FAT32)
      • max file size 4GB
      • used for EFI boot partition (/boot/efi)
    • exfat
      • removes the 4GB limit
      • used on large USB drives and SD cards

Virtual / Pseudo Filesystems

  • Exist entirely in RAM, No disk backing
  • Contents are lost on reboot
  • Exposed as a filesystem so userspace can use standard file tools (cat, ls, echo) to interact with the kernel
  • Most files show size 0 in ls -l
    • content is generated on-demand at read() time, so no size is known upfront
    • Exception: tmpfs files have real sizes since tmpfs actually stores data in RAM
FilesystemMount pointPurpose
tmpfs/tmp, /run, /dev/shmGeneral temp storage in RAM
devtmpfs/devDevice nodes, populated by kernel at boot
devpts/dev/ptsPseudo-terminal slave devices
procfs/procProcess info and kernel parameters
sysfs/sysKernel device/driver object tree
securityfs/sys/kernel/securityLinux Security Module (LSM) interfaces
tracefs/sys/kernel/tracingKernel tracing (ftrace)
debugfs/sys/kernel/debugKernel debugging interfaces
configfs/sys/kernel/configUserspace-driven kernel object configuration
cgroup2/sys/fs/cgroupControl groups v2 (resource limiting)
pstore/sys/fs/pstorePersistent storage for crash/panic logs
bpf/sys/fs/bpfPinned BPF maps and programs
efivars/sys/firmware/efi/efivarsRead/write UEFI firmware variables

Swap

  • Not a filesystem - no directory entries, inodes, or file operations
  • Raw memory extension: kernel pages out cold memory to swap space, reclaims RAM
  • Can be a dedicated partition (/dev/sdaX) or a swap file