IOError: No space left on device - which device?

Question

I'm uploading a small file (8.5 Mb) to a flask test server.

When the file finishes uploading, the server reports:

    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/wtforms/form.py", 
        line 212, in __call__
    return type.__call__(cls, *args, **kwargs)
    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/flask_wtf/form.py", line 49, in __init__
        formdata = request.form
    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/local.py", 
        line 338, in __getattr__
    return getattr(self._get_current_object(), name)
    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/utils.py", 
         line 71, in __get__
    value = self.func(obj)
    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/wrappers.py", 
         line 484, in form
    self._load_form_data()
    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/flask/wrappers.py", 
        line 165, in _load_form_data
    RequestBase._load_form_data(self)
    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/wrappers.py", 
        line 356, in _load_form_data
    mimetype, content_length, options)
    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/formparser.py", 
        line 193, in parse
    content_length, options)
    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/formparser.py",
        line 99, in wrapper
    return f(self, stream, *args, **kwargs)
    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/formparser.py",
        line 210, in _parse_multipart
    form, files = parser.parse(stream, boundary, content_length)
    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/formparser.py", 
        line 520, in parse
    return self.cls(form), self.cls(files)
    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/datastructures.py", 
        line 373, in __init__
    for key, value in mapping or ():
    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/formparser.py", 
        line 518, in <genexpr>
    form = (p[1] for p in formstream if p[0] == 'form')
    File "/home/ubuntu/.virtualenvs/eco_app/lib/python2.7/site-packages/werkzeug/formparser.py", 
        line 494, in parse_parts
        _write(ell)
IOError: [Errno 28] No space left on device

Now, the server has plenty of free space - over 3Gb.

I've looked at the Werkzeug github repo to try to find the location that Werkzeug is trying to write to, but can't track it down.

I've also checked tempfile.gettempdir() which gives /var/tmp as the temporary file directory, but this folder is practically empty, so I don't think that's the location that's creating the issue.

How do I find the device that has no space?

Answer 1

@Tom Hunt's comment was on the right track.

This unix SE answer explains what happened.

As a protection against low disc space, some daemons automatically "shadows" the current /tmp/ dir with a ram disc if the the root partition runs out of disc space. Sadly there's no automatic reversion of that process once enough disc space is free again.

I unmounted /tmp directory and followed Nitesh's suggestion:

sudo umount /tmp
sudo echo 'MINTMPKB=0' > sudo /etc/default/mountoverflowtmp

and now uploads are working properly.