Handling large file uploads with Flask
What would be the best way to handle very large file uploads (1 GB +) with Flask?
My application essentially takes multiple files assigns them one unique file number and then saves it on the server depending on where the user selected.
How can we run file uploads as a background task so the user does not have the browser spin for 1hour and can instead proceed to the next page right away?
- Flask development server is able to take massive files (50gb took 1.5 hours, upload was quick but writing the file into a blank file was painfully slow)
- If I wrap the app with Twisted, the app crashes on large files
- I've tried using Celery with Redis but this doesn't seem to be an option with posted uploads
- I'm on Windows and have fewer options for webservers
Answer
I think the super simple way to get around that simply sends the file in lots of small parts/chunks. So there are going to be two parts to making this work, the front-end (website) and backend (server).
For the front-end part, you can use something like Dropzone.js which has no additional dependencies and decent CSS included. All you have to do is add the class dropzone to a form and it automatically turns it into one of their special drag and drop fields (you can also click and select).
However, by default, dropzone does not chunk files. Luckily, it is really easy to enable.
Here's a sample file upload form with DropzoneJS and chunking enabled:
<html lang="en">
<head>
<meta charset="UTF-8">
<link rel="stylesheet"
href="https://cdnjs.cloudflare.com/ajax/libs/dropzone/5.4.0/min/dropzone.min.css"/>
<link rel="stylesheet"
href="https://cdnjs.cloudflare.com/ajax/libs/dropzone/5.4.0/min/basic.min.css"/>
<script type="application/javascript"
src="https://cdnjs.cloudflare.com/ajax/libs/dropzone/5.4.0/min/dropzone.min.js">
</script>
<title>File Dropper</title>
</head>
<body>
<form method="POST" action='/upload' class="dropzone dz-clickable"
id="dropper" enctype="multipart/form-data">
</form>
<script type="application/javascript">
Dropzone.options.dropper = {
paramName: 'file',
chunking: true,
forceChunking: true,
url: '/upload',
maxFilesize: 1025, // megabytes
chunkSize: 1000000 // bytes
}
</script>
</body>
</html>
And Here's the Back-end part using flask:
import logging
import os
from flask import render_template, Blueprint, request, make_response
from werkzeug.utils import secure_filename
from pydrop.config import config
blueprint = Blueprint('templated', __name__, template_folder='templates')
log = logging.getLogger('pydrop')
@blueprint.route('/')
@blueprint.route('/index')
def index():
# Route to serve the upload form
return render_template('index.html',
page_name='Main',
project_name="pydrop")
@blueprint.route('/upload', methods=['POST'])
def upload():
file = request.files['file']
save_path = os.path.join(config.data_dir, secure_filename(file.filename))
current_chunk = int(request.form['dzchunkindex'])
# If the file already exists it's ok if we are appending to it,
# but not if it's new file that would overwrite the existing one
if os.path.exists(save_path) and current_chunk == 0:
# 400 and 500s will tell dropzone that an error occurred and show an error
return make_response(('File already exists', 400))
try:
with open(save_path, 'ab') as f:
f.seek(int(request.form['dzchunkbyteoffset']))
f.write(file.stream.read())
except OSError:
# log.exception will include the traceback so we can see what's wrong
log.exception('Could not write to file')
return make_response(("Not sure why,"
" but we couldn't write the file to disk", 500))
total_chunks = int(request.form['dztotalchunkcount'])
if current_chunk + 1 == total_chunks:
# This was the last chunk, the file should be complete and the size we expect
if os.path.getsize(save_path) != int(request.form['dztotalfilesize']):
log.error(f"File {file.filename} was completed, "
f"but has a size mismatch."
f"Was {os.path.getsize(save_path)} but we"
f" expected {request.form['dztotalfilesize']} ")
return make_response(('Size mismatch', 500))
else:
log.info(f'File {file.filename} has been uploaded successfully')
else:
log.debug(f'Chunk {current_chunk + 1} of {total_chunks} '
f'for file {file.filename} complete')
return make_response(("Chunk upload successful", 200))