How to Fix “No Space Left on Device” Errors

Getting a “no space left on device” error can be frustrating, especially when you still have data you need to save. This error occurs when the storage partition you are trying to write to is completely full. There are several potential causes and solutions for this error on Linux systems. As a system administrator with over 10 years of experience, I have helped troubleshoot and resolve this issue many times.

What Causes This Error

There are a few common culprits for the “no space left on device” error:

  • Running processes still holding deleted files open – Even if a file is deleted, a process may still have an open handle to that file, reserving the space until the process is restarted.
  • Too many small files filling up inodes – Inodes store metadata and have a separate limit from overall disk space. Running out of inodes can happen before running out of space.
  • Log files, databases, or other writable files growing unchecked – Any files that steadily grow can eventually fill up a disk. Logs and database files are common culprits.
  • Failing disk with bad sectors – Over time, disk errors can reduce available good sectors until none are left for writes.

Check Available Disk Space

First, check the disk usage with the df command:

df -h

If any partition shows 100% full, that is your culprit.

Next, check inode usage with df -i:

df -i

If inodes are 100% used even with free space left, you have an inode shortage.

Fix 1 – Restart Processes Holding Open Deleted Files

Find processes still holding deleted files:

lsof | grep deleted

Then restart those processes. This will free up reserved space.

Fix 2 – Delete Unneeded Files

If there are no open deleted files, delete unneeded files and directories to free up space.

Some helpful commands:

  • du -sh * – Check folder sizes
  • find /var -size +1G – Find large files over 1GB
  • ncdu – Interactive disk usage browser

Fix 3 – Expand File System

If the partition is on a logical volume, you can expand the volume group and logical volume:

  1. Extend the volume group with free space
  2. Extend the logical volume
  3. Expand the filesystem

This will add free space without deleting data.

Fix 4 – Add Disks or Expand Filesystem

For physical partitions, add more disks and expand the filesystem or switch to a larger disk. This also expands space without deleting data.

Fix 5 – Disable Features Generating Large Files

For databases, logs, and other data sources, tune configuration to stop runaway disk usage. This includes log rotation, deletion policies, and growth limits.

Conclusion

Getting “no space left on device” errors requires both troubleshooting the root cause and fixing the immediate space shortage. Following this process will help resolve these errors:

  1. Check disk usage with df and df -i
  2. Identify problem processes holding deleted files open
  3. Delete unneeded files
  4. Expand logical volumes or add new disks
  5. Tune configurations generating large files

With filesystems expanding into the terabytes, running out of space is less common today. But when it happens, these steps will get your system back up and running. Let me know if you have any other questions!