Postgresql | No space left on device

Question

I am getting space issue while running a batch process on PostgreSQL database.

However, df -h command shows that machine has enough space

below is the exact error

org.springframework.dao.DataAccessResourceFailureException: PreparedStatementCallback; SQL [INSERT into BATCH_JOB_INSTANCE(JOB_INSTANCE_ID, JOB_NAME, JOB_KEY, VERSION) values (?, ?, ?, ?)]; ERROR: could not extend file "base/16388/16452": No space left on device
  Hint: Check free disk space.

What is causing this issue?

EDIT

postgres data directory is /var/opt/rh/rh-postgresql96/lib/pgsql/data

df -h /var/opt/rh/rh-postgresql96/lib/pgsql/data
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda2      100G   63G   38G  63% /

Answer 1

Most likely there are some queries that create large temporary files which fill up your hard disk temporarily. These files will be deleted as soon as the query is done (or has failed), so the file system has enough free space when you look.

Set log_temp_files = 10240 in postgresql.conf (and reload) to log all temporary files exceeding 10 MB, then you can check the log file to see if this is indeed the reason.

Try to identify the bad queries and fix them.

If temporary files are not the problem, maybe temporary tables are. They are dropped automatically when the database session ends. Does your application use temporary tables?

Another possibility might be files created by something else than the database.