File Upload¶

Uploading an image is the same as uploading any file, so we'll cover that. Here are the general principles:

You need a fancier kind of FORM element, specifically a different ENCTYPE that handles the binary data of a file by specially encoding it.
You need a new kind of INPUT element, which gives you a widget for specifying the file.
Some different processing in the script for getting the data from the file(s) and putting the data either into the filesystem or into a BLOB.

HTML's File Upload INPUT¶

Use the POST method for your FORM. Recall that one limitation of the GET method is that URLs usually have some bound on length, and we don't want to have that with a file.
ENCTYPE="multipart/form-data" must be an attribute of the FORM.
INPUT TYPE=FILE NAME=name-of-file-input

Here's an example:

{% extends "base.html" %}
{% block main %}
<p>Form to upload a file:
<form method="post" action="" enctype="multipart/form-data">
    <p><label>NM: <input type="text" name="nm"></label></p>
    <p><label>Pic: <input type="file" name="pic"></label></p>
    <p><input type="submit" value="upload"></p>
</form>

{% if nm != '' and src != '' %}
<p>Last upload: {{nm}}</p>
<p> <img src="{{src}}"></p>
{% endif %}
{% endblock %}

File Upload Security Concerns¶

File upload has a number of unique security concerns. We'll discuss:

filename naming: Don't allow a file to be named something that could be executed via the web, such as name.php or name.cgi. (Did you notice that the linked reading mentioned XSS attacks?)
file contents: be sure it really is an image (or whatever), rather than some unexpected content.
file size: a malicious user could try to overwhelm your server or use up all the disk space.
access to uploaded content: if you can, don't put the uploaded file in a location accessible from the web. This helps you avoid people using your system as, for example, a porn server. Flask allows us to implement a route that serves images, so we'll know what images we are serving.
access to the upload script: if you can, require people to be authenticated before they upload anything. Then, hopefully, you can force them to be responsible for their actions. It's not a bad idea to keep track of who uploads each file.

MIME Types¶

When the back-end delivers content to the browser, it is often labeled with the type of data. For example, an image might have a "image/jpeg" header. That's its MIME_type.

MIME is for Multimedia Internet Mail Extensions. It was orginally for emailing attachments. The attachment was labeled as to its type, so that helper applications could be invoked appropriately, without necessarily depending on a filename extension.
The HTTP headers from the server must always announce the MIME type of the data being returned to the client. When the data comes from a file, it figures it out in a variety of ways, such as the file extension and the "file" command.

Working Flask Example¶

Copy upload to your account:

cd ~/cs304/
source venv/bin/activate
cp -r ~cs304/pub/downloads/upload upload
cd upload

I'll demo and explain the following:

create-table-picfile.sql to create the table
insert-picfiles.py to bulk-insert the existing pics
show-picfiles.sql to list the people with pictures
app.py the example app that does file upload to files
delete-image.sh to delete an image

Running It¶

create the table using create-table-picfile.sql
bulk-insert the files using insert-picfiles.py
list them, just to see using picfiles.sql
run app-upload-files.py
display one picture; notice the URL
display all pictures; notice the URL
upload a picture
(optional) delete the picture using delete-image.sh
repeat

I recorded a video demonstration of file upload in videos

Flask Code¶

In a moment, we'll look at the app.py code. It's lengthy, but comprehensible.

Here's a road map:

There's some configuration (upload directory name, etc) at lines 18-19
There's a route @app.route('/pic/<nm>') (line 25) which just sends a single picture to the browser. It does that by using the flask send_from_directory function, getting the filename out of the database.
There's a route @app.route('/pics/') (line 39) that sends a page with all the images on it. Each image is actually shown by the first route, but this shows how we can combined database queries with images.
There's a route @app.route('/upload/', methods=["GET", "POST"]) (line 48) that does the upload. The meat of it is only a dozen lines of code or so.

Here's the complete app:

'''This example implements file upload to the filesystem'''

from flask import (Flask, render_template, make_response, request, redirect,
                   url_for, session, flash, send_from_directory, Response)
from werkzeug.utils import secure_filename
app = Flask(__name__)

import sys, os
import cs304dbi as dbi

import secrets
app.secret_key = secrets.token_hex()

# This gets us better error messages for certain common request errors
app.config['TRAP_BAD_REQUEST_ERRORS'] = True

# new for file upload
app.config['UPLOADS'] = 'uploads'
app.config['MAX_CONTENT_LENGTH'] = 1*1024*1024 # 1 MB

@app.route('/')
def index():
    return render_template('base.html')

@app.route('/pic/<nm>')
def pic(nm):
    conn = dbi.connect()
    curs = dbi.dict_cursor(conn)
    numrows = curs.execute(
        '''select filename from picfile where nm = %s''',
        [nm])
    if numrows == 0:
        flash('No picture for {}'.format(nm))
        return redirect(url_for('index'))
    row = curs.fetchone()
    return send_from_directory(app.config['UPLOADS'],row['filename'])

    
@app.route('/pics/')
def pics():
    conn = dbi.connect()
    curs = dbi.dict_cursor(conn)
    curs.execute('''select nm,name,filename
                    from picfile inner join person using (nm)''')
    pics = curs.fetchall()
    return render_template('all_pics.html',n=len(pics),pics=pics)

@app.route('/upload/', methods=["GET", "POST"])
def file_upload():
    if request.method == 'GET':
        return render_template('form.html',src='',nm='')
    else:
        try:
            nm = int(request.form['nm']) # may throw error
            f = request.files['pic']
            user_filename = f.filename
            ext = user_filename.split('.')[-1]
            filename = secure_filename('{}.{}'.format(nm,ext))
            pathname = os.path.join(app.config['UPLOADS'],filename)
            f.save(pathname)
            conn = dbi.connect()
            curs = dbi.dict_cursor(conn)
            curs.execute(
                '''insert into picfile(nm,filename) values (%s,%s)
                   on duplicate key update filename = %s''',
                [nm, filename, filename])
            conn.commit()
            flash('Upload successful')
            # decided to just re-render the form (rather than use
            # post-redirect-get) so that we can display the uploaded
            # picture.
            return render_template('form.html',
                                   src=url_for('pic',nm=nm),
                                   nm=nm)
        except Exception as err:
            flash('Upload failed {why}'.format(why=err))
            return render_template('form.html',src='',nm='')
            

if __name__ == '__main__':
    if len(sys.argv) > 1:
        # arg, if any, is the desired port number
        port = int(sys.argv[1])
        assert(port>1024)
    else:
        port = os.getuid()
    dbi.conf()                  # connect to personal database
    app.debug = True
    app.run('0.0.0.0',port)

The HTML Code¶

There are two parts to the HTML code:

the browser requesting a particular picture from the server so that it can be displayed (this would be a GET request) with the server sending the picture data to the browser, and
the browser sending a picture to the server in order for it to be uploaded.

Let's look at them in order. The browser can request a picture in a number of ways but the most common is as the SRC attribute of an <IMG> tag. In our example code, the template file looks like:

<p><img src="{{url_for('pic',nm=pic['nm'])}}" alt="{{pic['name']}}"></p>

In the code above, the pic variable is a dictionary read from the database. It has two keys, the nm and the name of the person. We need the nm for the URL, and therefore for the url_for function, which refers to our pic(nm) handler function.

Now let's look at uploading a picture. That requires a special form:

<form method="post" action="{{url_for('file_upload')}}" enctype="multipart/form-data">
    <p><label>NM: <input type="text" name="nm"></label></p>
    <p><label>Pic: <input type="file" name="pic"></label></p>
    <p><input type="submit" value="upload"></p>
</form>

Notice that the data (the nm, which is text, and the pic, which is a file) are sent to our file_upload function, which we saw above.

Caching¶

As we know, browsers cache files, including CSS, JS and images. That can cause trouble when we upload a new image to replace an existing one. I've solved this and I have a decent writeup, but haven't yet had time to write for the web. Let me know if you run into this in your project.

Security Concerns¶

Very high. Bad stuff can happen when you allow untrustworthy stuff onto your server.

Control the access: don't allow anonymous people to upload stuff for all to see. Require logins, sessions, etc.
Control the filename. Don't let people name uploads because they might name it as a command or something executable via the web.
Control the size; don't let people overwhelm your server with huge files.
Control the location. You don't want people being able to use your server as an image server, serving porn or something.
Control the content. In the shell, you can check using the Linux file command to check what kind of data this is. In Python, you have to use an extra module, such as imghdr or filemagic

File Upload with Project Teams¶

The basics of file upload stay the same when you are using it in a team project, but there are some practical considerations that are useful to know. Let's talk about those now.

First, what's the problem we are trying to solve? Suppose Alice and Betty are working together. Alice implements file upload. She tests it, uploading files to a uploads folder in her personal account. (For concreteness, suppose that's /students/aa123/cs304/project/upload.) The code also puts some entities in the database, saying that entity "Edith" has a profile picture, "ee999.jpg" in the uploads folder. So far, so good, and it works great for Alice. When Betty runs the app, she displays Edith's page and gets a 404 for Edith's profile picture, because "ee999.jpg" isn't in her (Betty's) uploads folder. (It's in Alice's uploads folder.)

That's the problem. It's not a big problem, but it's annoying. So, how do we solve it? Conceptually, we have to do the following:

have a single shared location to put the uploaded files. We'll use the team account
make that shared location writeable by all team members
make sure all copies of the app upload to that shared folder
make the uploaded files readable by all team members

To do the first, we need to create a folder in the team account. Suppose we decide to call it uploads. We just need to (1) login to the team account and (2) create the folder:

mkdir ~/uploads

The second step is to make the folder writeable by anyone. (By default, folders are only writable by their owner, for obvious security reasons. Here, we need to change that default.) The chmod command does that:

chmod a+wrx ~/uploads

For the third step, you'll want to configure your app.py to upload to the uploads folder in the team account. Suppose that the team account for Alice and Betty is team_ab with home directory /students/team_ab/. So the code in app.py might be:

app.config['UPLOADS'] = '/students/team_ab/uploads/'

Finally, when any team member uploads a file using the app, that file gets created with default permissions that typically do not allow others to read the file. (Again, this is for obvious privacy reasons.) So, in our python code, after we save the file, we need to change its permissions with the chmod system call:

import os    # at the top of the file

@app.route('/upload/', methods=["GET", "POST"])
def file_upload():
            ...
            pathname = os.path.join(app.config['UPLOADS'],filename)
            f.save(pathname)
            os.chmod(pathname, 0o444)    # readable by owner, group and others
            ...

The os.chmod function can be used in other ways as well, but that octal value 0x444 (which starts with the digit zero and the lowercase letter "o") will do the trick.

Summary¶

File upload is an important feature of many web databases, not just FaceBook and Yelp. You should consider its usefulness in your project.
File upload can be a security risk, so you should be careful.

Specific technology pieces:

form enctype: <FORM ENCTYPE="MULTIPART/FORM-DATA" ...>
form inputs: <INPUT TYPE=FILE NAME=name-of-file-input>
folders: you'll probably create a folder for uploaded files, say, uploads, next to templates and static. You could, of course, have different folders for different kinds of files, etc.
files: you'll need a naming scheme for your uploaded files.
- You could use a counter: fileNNN.jpg
- You could use an ID: fileNM.jpg
- You could use a timestamp: file-2022-04-01-23-01-12.jpg (your phone's camera does that)
You'll probably store the filename in an appropriate database table, unless the naming scheme allows the filename to be inferred from other data.
Flask's request object has a files dictionary, containing all the uploaded files, as Python file objects.
file objects have a .save() method, allowing you to save the file to your desired pathname
You'll construct your pathname from the uploads folder and the chosen filename using os.path.join(folder, file) from the Python os module.
Avoid user data in the filename/pathname, though that is possible.
Use secure_filename if there's any user data in the filename
from werkzeug.utils import secure_filename
To deliver files to the browser, use send_from_directory(uploaddir, filename)
The browser requests files (e.g. images) in a separate request, so you have to have a route/endpoing to respond to those requests.

Appendix¶

The following is optional

The Python `imghdr` Package¶

Being able to identify the type of a file is important. We can't trust the way that the file is named; we need to look at the actual data. One way is the Python imghdr package.

You can try it by running python and using the following code:

import imghdr
imghdr.what('uploads/123.jpg')

File Upload¶

File Upload with Flask¶

Issues¶

Images in Files¶

File Upload¶

HTML's File Upload INPUT¶

File Upload Security Concerns¶

MIME Types¶

Working Flask Example¶

Running It¶

Flask Code¶

The HTML Code¶

Caching¶

Security Concerns¶

File Upload with Project Teams¶

Summary¶

Appendix¶

The Python imghdr Package¶

The Python `imghdr` Package¶