Handling Storage Space
"No space left on device" is one of the most common alerts you'll receive as an SRE — and it can crash production applications.
Checking Disk Usage
df — Disk Filesystem
# Show disk space on all mounted filesystems (human-readable)
$ df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 792M 1.2M 791M 1% /run
/dev/sda1 98G 45G 48G 49% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/sdb1 200G 89G 111G 45% /data
/dev/sdc1 50G 12G 38G 24% /backupColumns:
| Column | Meaning |
|---|---|
Size | Total size of filesystem |
Used | Space currently used |
Avail | Space available |
Use% | Percentage used |
Mounted on | Where the filesystem attaches |
# Show inodes (file count limit, not disk space)
$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 6553600 234567 6319033 4% /
# Show filesystem type
$ df -Th
Filesystem Type Size Used Avail Use% Mounted on
/dev/sda1 ext4 98G 45G 48G 49% /
tmpfs tmpfs 792M 1.2M 791M 1% /rundu — Directory Usage
# Show size of a directory (human-readable)
$ du -sh /var/log
1.2G /var/log
# Show all directories, sorted by size
$ du -h --max-depth=1 /var | sort -h
4.0K /var/local
8.0K /var/opt
64K /var/cache
1.2G /var/log
48G /var/lib
# Show top 20 largest items
$ du -h /var 2>/dev/null | sort -rh | head -20Finding Large Files
# Find files larger than 100MB
$ find / -type f -size +100M 2>/dev/null
/var/lib/docker/overlay2/abc123.../data.img
/var/log/archive/old-logs.tar.gz
# Find files larger than 1GB in /home
$ find /home -type f -size +1G 2>/dev/null
# Find the 10 largest files in a directory
$ find /var/log -type f -exec ls -lh {} \; 2>/dev/null | sort -k5 -rh | head -10
# Find largest files by extension
$ find / -type f -name "*.log" -size +100M 2>/dev/nullMounting Devices
When you attach a new disk (like an AWS EBS volume), it appears as a block device but doesn't automatically appear as a usable filesystem. You must mount it.
lsblk — List Block Devices
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 100G 0 disk
├─sda1 8:1 0 98G 0 part /
├─sda14 8:14 0 1M 0 part
└─sda15 8:15 0 106M 0 part /boot/efi
sdb 8:16 0 500G 0 disk ← New unformatted disk
sdc 8:32 0 1TB 0 disk ← Another new diskMounting a Filesystem
# 1. Create a filesystem (format the partition)
$ sudo mkfs.ext4 /dev/sdb
mke2fs 1.46.5 (30-Dec-2021)
Creating filesystem with 131072000 4k blocks and 32768000 inodes
Filesystem UUID: 3e4a5b6c-7d8e-9f0a-1b2c-3d4e5f6a7b8c
Allocating group tables: done
Writing superblocks and filesystem accounting: done
# 2. Create a mount point directory
$ sudo mkdir -p /mnt/data
# 3. Mount the filesystem
$ sudo mount /dev/sdb /mnt/data
# 4. Verify
$ df -h /mnt/data
Filesystem Size Used Avail Use% Mounted on
/dev/sdb 493G 73M 467G 1% /mnt/datamkfs.ext4 destroys all data on the partition. Only run this on NEW disks that have no data. Never format a disk with existing data.Unmounting
# Unmount when done
$ sudo umount /mnt/data
# Force unmount (if busy)
$ sudo umount -f /mnt/data
# Lazy unmount (unmount when not busy)
$ sudo umount -l /mnt/data/etc/fstab — Persistent Mounts
To mount a filesystem automatically at boot, add an entry to /etc/fstab:
# Get the UUID of the filesystem
$ sudo blkid /dev/sdb
/dev/sdb: UUID="3e4a5b6c-7d8e-9f0a-1b2c-3d4e5f6a7b8c" TYPE="ext4"
# Add to /etc/fstab
$ sudo nano /etc/fstab
# Add this line:
UUID=3e4a5b6c-7d8e-9f0a-1b2c-3d4e5f6a7b8c /mnt/data ext4 defaults 0 2/etc/fstab columns:
<device> <mount point> <type> <options> <dump> <pass>
UUID=... /mnt/data ext4 defaults 0 2
| Column | Meaning |
|---|---|
| 1 | Device (UUID is preferred over /dev/sdb) |
| 2 | Mount point |
| 3 | Filesystem type |
| 4 | Mount options |
| 5 | Dump (backup, 0=don't) |
| 6 | fsck pass (0=no check, 1=root, 2=others) |
# Test fstab entry without rebooting
$ sudo mount -a
# If no errors, the entry is valid
# Verify it mounted
$ df -h /mnt/dataDisk Partitions
fdisk — Partition Table
# View partition table
$ sudo fdisk -l /dev/sda
Disk /dev/sda: 100 GiB, 107374182400 bytes, 209715200 sectors
Disk model: Virtual Disk
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Device Start End Sectors Size Type
/dev/sda1 2048 20482047 20480000 9.8G Linux filesystem
/dev/sda2 20482048 40962047 20480000 9.8G Linux filesystempartprobe — Notify Kernel of Changes
# After partitioning, inform the kernel
$ sudo partprobe /dev/sdbLVM (Logical Volume Manager)
LVM provides flexible storage management — resize volumes online, add disks without downtime.
# Show LVM status
$ sudo lvdisplay
# Show physical volumes
$ sudo pvdisplay
# Show volume groups
$ sudo vgdisplay
# Extend a logical volume (while mounted)
$ sudo lvextend -L +50G /dev/vg00/lv_data
$ sudo resize2fs /dev/vg00/lv_dataQuick Reference
| Task | Command |
|---|---|
| Show disk space | df -h |
| Show directory sizes | du -sh /path |
| Find large files | find / -size +100M |
| List block devices | lsblk |
| Format partition | sudo mkfs.ext4 /dev/sdb |
| Mount | sudo mount /dev/sdb /mnt/data |
| Unmount | sudo umount /mnt/data |
| Add persistent mount | Edit /etc/fstab |
| Test mounts | sudo mount -a |
Practice Challenge
- Run
df -h— what filesystems are mounted and how full are they? - Run
du -sh /var/log— how much space do logs use? - Run
lsblk— what block devices are attached? - Find files larger than 10MB in your home directory:
find ~ -type f -size +10M - Check the UUID of a partition:
sudo blkid /dev/sda1