Cookies

The topic is cookies, but not the type you eat (though we may have some). Cookies are a technique for maintaining sessions, which we'll talk more about next time. (There are other techniques, such as hidden inputs and url-rewriting, but we'll focus on cookies.)

Sessions are important because most interesting web applications involve a series of interactions. People visit Facebook and they stay a while. People go to Amazon or eBay and they shop for a while. They build up state such as history or a shopping cart, with or without login.

Sessions are a challenge because the underlying HTTP protocol is stateless. Each http request and response is treated as if it's the only one, ever. Even if a user has made 1000 prior requests to a particular application, when that application starts running, it has no prior knowledge of that user. The application has to

• Figure out who this user is, and
• remember what that user was doing.

Cookies are the primary way of remembering something across transactions, including who someone is. They can also be used for many other things, both more and less interesting:

• A movie showtimes application I used to use had a form to specify your zip code and it would show you movie listings for your area. It would remember the zip code in a cookie, so that you didn't have to supply it again when you went to the webpage again. Weather websites could do the same (and it's annoying when they don't).
• A web application could store preference settings, such as myYahoo or iGoogle.
• A web application like Gmail can store identity or authentication information in a cookie, so that you only have to login once per session, instead of with each transaction. (What a pain that would be!)
• To track unique visitors (for bragging or charging advertisers)
• What examples can you think of?

What are Cookies?¶

Cookies are small bits of text saved with your browser (so Chrome, Firefox and Safari will each have its own set of cookies). You can think of them as key/value pairs. For example "zip=02481" for the movie showtimes application.

Cookies are set in the headers of your HTTP response. Therefore, you have to decide on setting the cookie and set it before doing anything else. (Though some web technologies, such as the default settings of PHP, will buffer your response (hold the whole thing in memory), just in case you decide to add a cookie after printing some or most of your response. In general, though, you should not count on this.)

Cookies are the primary means for maintaining state (remembering stuff) from one request to the next.

Along with a reply, server sends some additional information in the header. Here's what the response to an HTTP request might look like:

 
HTTP/1.0 200 OK 
Content-Length: 141 
Content-Type: text/html 
Set-Cookie: zip=02491; 
    domain=cs.wellesley.edu; 
    expires=Mon, 23-Apr-2019

... content follows 

Let's take a look at that cookie:

• The first name-value pair determines the name of the cookie and a special value, often used to identify the user. Here the name is zip and the value is 02481.
• The domain value determines what sites can access this cookie (usually only the site that issued it). Here that's cs.wellesley.edu. That means that the browser will only send this cookie back to cs.wellesley.edu, not to other servers.
• Other values determine when the cookie expires, which URLs require this cookies, etc.

Clients have the option of refusing cookies (meaning that they do not store them or do not submit them with requests), though in practice the web relies so much on cookies that it would be difficult to do anything but casual browsing without accepting cookies.

Most browsers will refuse a cookie if it directs them to submit information to a third party (not the site that issued the cookie). You can usually configure this. Hunt around in your browser and you'll find it.

Cookies live on the client¶

For all page requests (in other words, every time you visit a web page), your browser (the client) compares the web page's URL to the cookies in the cookie jar and sends all the unexpired cookies that match the domain and path. Here's what a request might look like:

 
GET /index.html HTTP/1.0 
Accept: text/html, image/gif, image/jpeg 
Accept-Language: en 
Cookie: zip=02481 

• The total size of a cookie must be less than 4KB. In practice, it's usually only a few dozen bytes.
• The client may hold up to 20 cookies for a given domain, and up to 300 cookies total.
• A request might carry more than one cookie.
• Your browser can also add cookies to the cookie jar; it doesn't have to come from the server. Your web page can do the same, using JavaScript.

Google Analytics Cookies¶

If you look in your cookie jar, you will almost certainly see these cookies: _utma, _utmc, _utmz. These are from Google Analytics, a widely used service to track who uses a website. To learn more, try Google Analytics Cookies Ultimate Guide. They are set (in my browser), for domain .wellesley.edu, so of course your page matches.

Security and Privacy¶

Since cookies live on the client, what does that mean in terms of privacy and security? Here are a few common scenarios:

• The browser is running on your own computer (laptop, desktop, smart phone, whatever). In that case, the cookies are stored on that computer and are therefore as safe/unsafe as any other file on your computer.
• The browser is running on a shared computer, but one with your own unique login, such as the ones here at Wellesley. In this case, the cookies are stored in the same folder as your preferences, browser history, cache, and other personal stuff. On the Macs in L180 and E101, the general Firefox code, libraries, plug-ins and the like are in one common folder (/Applications/Firefox), but my personal stuff and your personal stuff are in folders that belong just to us (/Users/sanderso/ versus /Users/ehildret/ versus /Users/youracct/). So, this is more vulnerable than a machine you have physical control of, but it is still relatively secure. Someone would have to hack into your account or into that computer to view your cookies.
• The browser is running on a shared computer, and there is no login process. For example, the Natick public library has a number of computers scattered about for searching the catalog, which is done via a web browser. If I go to a website that sets a cookie, it'll be set in that browser, and if you use the machine after me, you'd be able to view the cookie (unless I or the website clears the cookies when I'm done). In this case, there's barely a fig leaf protecting your privacy.

In general, you should note that cookies are not secure. They have the following vulnerabilities:

• The user can see the values you have set. If you set a cookie saying deadbeat-customer=true, they will see that.
• The user can modify the values you have set. They can do that via a browser, a JavaScript, and custom apps. They can change the cookie to deadbeat-customer=false. We'll see one way to do that later in this reading. We'll also see this in class or in a demo.
• The cookie value is sent along with the HTTP request and response, which are not encrypted, so the cookie value is easily seen by anyone with a network packet sniffer. (Later in the course, we'll see how to use HTTPS so that both directions are encrypted.)

When we get to Flask's sessions, we will see how Flask introduces some safeguards to protect against some of these insecurities.

A Schematic of the Request-Reply Model, with Cookies¶

It can be helpful to have a picture of what's happening. Here's a picture of basic scenario involving two request/response pairs and the cookie that the server sets in the first response and receives with the second request.

1. Browser sends a request to the server. Since the user has never visited that site before, there are no cookies, so none are sent.
2. The server notices that there are no cookies, so it assumes that this is a new visitor, and it generates a new ID for the user and includes that ID as a cookie in the reply. The reply might even be tailored for new visitors.
3. The browser later (minutes or months later, but before the cookie expires) makes another request at that site and sends back the cookie it got in step 2.
4. The server gets the request with the cookie, realizes this is a returning visitor, and can generate a customized reply.

Cookie Attributes¶

Web browsers and web servers that support cookies have to handle the following attributes:

• name: anything you want for your web app, within limits
• value: the value your web app wants to get back
• domain: the server to supply them to, typically your own
• expires: the time that the cookie is invalid. If not specified, it expires when the browser exits.
• Max-Age: the number of seconds until the cookie expires; an alternative to the "Expires" attribute.
• path: the browser only returns the cookie for URLs below this
• secure: the browser will only return this cookie using HTTPS
• HttpOnly: the browser won't let JavaScript access this cookie via document.cookie.

In most cases, cookies are not deleted. It's easier to set the value to an empty value (such as zero or the empty string) and test for that. You can also set its expiration time to the past, which will have the effect of removing the cookie.

Session Diagram¶

The following is a diagram of what a somewhat typical session might look like. (Click on the diagram to see a bigger version.) There are four situations that the back end app (on the right) might have to deal with:

1. (blue) Very first visit. You can tell because there's no cookie and there's no form being sub mitted; it's just a simple GET request. Respond with a page that includes a login form.
2. (green) Login. You can tell because the user is submitting a login form. Respond with a normal page, including a logout button. The response headers will set the cookies for the first time.
3. (yellow) normal, logged-in usage. You can tell because the request includes a cookie. There might be lots of different use-cases, differentiated in lots of ways. (E.g. searching for data, submitting a comment, rating a product, adding a tag, uploading a picture, ...). The response will depend on the use-case, but will include headers that set the cookies. Cookies are pretty much always set in a response. They're usually set to the same value, but with a new expiration value. Thus, if the user continues to interact with the system, the cookies never expire. (That is, even if the cookie is set to expire in 5 minutes, the user can stay logged in for hours, as long as they do something every five minutes.)
4. (red) logout. You can tell because the request is of a logout form. (You can think of this as being just another use-case of the web app.)

Cookie Values¶

What do we set the cookie to? Cookies are limited to 4K, so we have two options:

• Store everything in the cookie, but be conscious of the space limit
• Store a unique identifier in the cookie, and use that identifier as a key into a table kept in the database.

The latter is the most common approach, as you probably saw when you looked in your browser's cookie jar. (What does your gmail cookie look like?). How do we make a unique identifier? Here are some options:

• use the computer's time (say, milliseconds since the epoch), possibly with the process id of our current web app.
• use the key (auto_increment) generated by some table
• use a server-supplied session identifier (punt to someone else)

We won't pursue this further, since Flask does not create a unique identifier. We'll look at Flask's approach for sessions next time.

Cookies from Code¶

Whatever way you use cookies, you have to remember:

• They go out with the header of the response, so typically very early. Flask has a special mechanism for adding them to the response, after rendering but before sending it to the browser.
• You typically read cookies early in the process as well.

In the next sections, we'll look at how to manage cookies using Flask. There will be a web page that allows the user to "log in" (identify themself) and thereafter it counts how many visits to the page there are. It also supplies a "logout" button when the person is logged in.

Just for additional detail, the web app will print out the cookie information it got from the browser, and the response will have a button that will run some JavaScript to show you the cookies as the browse sees them.

When you read the code, look for the four use cases:

1. No cookie, no form data, so first visit. Give them an empty form. This corresponds to the Welcome (blue) phase in the cookie session figure.
2. No cookie, form data is present, so record the person's name. This corresponds to the login (green) phase in the cookie session figure.
3. Cookie, but nothing else, so an ordinary visit. This corresponds to the normal (yellow) phase inthe cookie session figure.
4. Delete cookie. This corresponds to the logout (red) phase in the cookie session figure.

Note that cookies are available to JavaScript running in the browser (unless the HTTPOnly attribute is set to true, which I have not done), so all the examples will let you see JavaScript's view of the cookies.

Cookies in Flask¶

Please read this very short tutorial on Flask - Cookies

Here are the basic facts from that tutorial:

• the request object has a cookies property which is a dictionary of key-value pairs: request.cookies.get('zip')
• there is a set_cookie method on the response object to set a key-value pair: resp.set_cookie('zip','02481')
• to get a response object, use the make_response function wrapped around a call to render_template or redirect: make_response(render_template('movie-listing.html'...))

Those ingredients might be used like this, where we read a cookie value (we assume one is set), look up listings for that zip code, and render a result, setting the cookie again.

zip_code = request.cookies.get('zip')
listings = movies_for(zip_code)
resp = make_response(render_template('movie-listing.html',
                                     location=zip_code,
                                     movies=listings))
resp.set_cookie('zip',zip_code)
return resp

Cookies Demo¶

Here's a demo of the same ideas as the cookie session figure above. It opens in another tab/window.

CS 304 cookie demo

Remember, the four use cases are:

1. A first visit; no cookie is set
2. A login: send a name to the server
3. A normal visit; the cookie sends the name to the server
4. A logout; the server will log the person out and delete (expire) the cookie

Note that these are not real logins with usernames and passwords and proper security. We'll learn that a bit later. But I'm using a name and a number of visits as the values of the cookies. In fact, if you have two browsers installed on your laptop, try visiting the CS 304 cookie demo in each browser. You can use different names and different numbers of visits, which shows that the cookie values are specific to your browser.

Below is the code that implements the CS 304 cookie demo. It has these parts:

• a base template that shows the cookie values that the app got
• a splash page that is shown when no one is "logged" in
• a cookie page that greets the user by name and shows their visit count, and allows them to re-visit and to delete the cookies.
• the python file that implements all the behavior.

Base Template

Note that both pages will report all cookies that the Flask app receives. These are sent in the allCookies dictionary when we render the template. The base template also provides a way to display a message from Python. This demo does not use flash() because flash relies on sessions which involves a special kind of cookie, and I wanted to focus on just the two cookies this demo is about.

<!doctype html>
<html> 
    <head>
        <title>Flask Cookies Example</title>
    </head>

    <body>

{% if msg %}
<p>Message: {{msg}}</p>
{% endif %}

        <p>From base: These are all the cookies the application can see:</p>
        <ul>
            {% for key in allCookies %}
            <li>{{key}} : {{allCookies[key]}}</li>
            {% endfor %}
        </ul>

{% block page_content %}
<p>This block is overridden in the inheriting templates. </p>
{% endblock %}

    </body>
</html>

Splash Page

Here is the splash page with "login" form:

{% extends "base.html" %}
{% block page_content %}

<h1>Splash Page</h1>

<p>Welcome to our app!</p>

<form method="POST" action=""> 
    <label for="login_name"> your name: </label>
    <input type="text" name="login_name">
    <input type="submit">
</form>

{% endblock %}

Cookies page

Here is the page for a normal visit. It provides a number of ways to visit the page again (thereby incrementing the visit count), including both GET and POST. You can also click the "reload" button in your browser.

{% extends "base.html" %}
{% block page_content %}

        <h1>Hello, {{ name }}!</h1>

        {{ visitMsg }}

        <a href="">visit with link</a>
        <form method="GET" action="">
            <input type="submit" name="submit-btn" value="Visit again using GET">
        </form>
        <form method="POST" action="">
            <input type="submit" name="submit-btn" value="Visit again using POST">
        </form>
        <form method="POST" action=""> 
            <input type="submit" name="submit-btn" value="Delete cookie">
        </form>
{% endblock %}

cookies.py

Finally, here is the Python/Flask code that implements the four use cases. In this code, I made several debatable design decisions.

• I combined all four cases into a single route, so that all the logic is in one place, but in other uses of cookies, these could all be different routes:
  • one for initial greeting,
  • one for login,
  • many for normal usage, and
  • one for logging out.
• As mentioned above, I used a msg instead of flash() so as to avoid the session cookie
• Because I didn't use flash(), I decided not to use the POST-REDIRECT-GET pattern when handling POST requests.

The code starts out with an IF statement distinguishing whether there is a cookie or not. If there's no cookie, it's either the first visit (so send them a splash page with the login form) or they are POSTing the login form (so log them in). If there is a cookie, we then distinguish GET versus POST. Note that "routine" visits could be done either with GET or POST, so case 3 appears on both sides of the IF. They can only logout by POSTing the delete form.

from flask import (Flask, render_template, make_response,
                   request, redirect, url_for, flash)

app = Flask(__name__)

@app.route('/', methods=["GET", "POST"])
def index():
    flaskname = request.cookies.get('flaskname')
    if not flaskname:
        print('no cookie set')
        if request.method == "GET":
            print('case 1: first visit, just render splash page')
            return render_template('splash.html',
                                   msg='Welcome!',
                                   allCookies=request.cookies)
        else:
            print('case 2: user submitted a form with their name')
            resp = make_response(
                render_template('cookies.html',
                                msg = 'logged in; this is your first visit',
                                name=request.form['login_name'],
                                allCookies=request.cookies))
            resp.set_cookie('flaskvisits', str(1))
            resp.set_cookie('flaskname',
                            request.form.get('login_name'))
            return resp
    else:
        print("cookie is set")
        print("so either they are continuing or logging out")
        visits = request.cookies.get('flaskvisits')
        newVisitCount = int(visits)+1
        visitMsg = ("You have visited {n} times before this."
                    .format(n=str(newVisitCount)))
        if request.method == "GET":
            print('case 3: just a regular visit')
            resp = make_response(
                render_template('cookies.html',
                                msg=visitMsg,
                                name=flaskname,
                                allCookies=request.cookies))
            resp.set_cookie('flaskvisits', str(newVisitCount))
            return resp
        else:
            print('a form is posted')
            # I'm not doing the post-redirect-get pattern
            if request.form['submit-btn'] == "Visit again using POST": 
                print('case 3 redux: user posts the Visit again form')
                resp = make_response(
                    render_template('cookies.html',
                                    msg=visitMsg,
                                    name=flaskname,
                                    allCookies=request.cookies))
                resp.set_cookie('flaskvisits', str(newVisitCount))
                return resp
            elif request.form['submit-btn'] == "Delete cookie": 
                print('case 4: user wants to delete the cookies')
                goodbyeMsg = ("Goodbye {name}, you visited {n} times."
                              .format(name=flaskname, n=visits))
                resp = make_response(
                    render_template('splash.html',
                                    msg=goodbyeMsg,
                                    name=flaskname,
                                    allCookies=request.cookies))
                resp.set_cookie('flaskname', '', expires=0)
                resp.set_cookie('flaskvisits', '', expires=0)
                return resp

if __name__ == '__main__':
    import sys,os
    if len(sys.argv) > 1:
        # arg, if any, is the desired port number. Use 5000 when developing locally
        port = int(sys.argv[1])
        assert(port>1024)
    else:
        port = os.getuid()
    app.debug = True
    app.run('0.0.0.0',port)

Even though this is pretty complicated, it's less than 100 lines of code, so take a few minutes to read it and recognize the different aspects of getting and setting cookie values.

Cookie Demo Video¶

Here is a video of the Cookie Demo. It's less than 10 minutes.

I apologize that it starts with a black screen, but there is a voiceover before I share my screen, so I can't clip the black part out.

Developing with Cookies¶

One of the difficult things about using cookies is that when your re-start your app, it might not restart from "scratch". That is, your prior cookies may still be in the cookie jar. We'll see that in the demo. I will also demonstrate how to delete the old cookies so that you can start over from scratch.

Another option is to run the application in an "incognito mode" browser window or "private window" and close the window when you are done. Those modes don't save cookies.

Using the Chrome Developer Tools¶

Because cookies are a "hidden" part of the HTTP requests and responses, observing them and debugging them can be harder. Fortunately, the Chrome Developer tools make this easy. (Other developer tools can do this as well, but I'll only teach you how it's done in Chrome). There will be a video demonstration of these tools, but I'll do a brief textual description now.

See Chrome DevTools

In both descriptions below, you enter the Chrome Developer tools in the usual way. I usually use command-option-j on my Mac.

Network Tab

First is the network tab. You'll find that at the top of the devtools window, and if you click on it, it will show you the browsers's requests and the server's response. It will only show you new ones though, so usually you switch to that tab and then issue a request (say by clicking a link or submitting a form). In the screenshot below, I have highlighted (with magenta ovals) the following noteable areas:

• The chosen network tab (note the blue underline)
• The request itself (localhost in this case)
• The set-cookie header in the response. Here, the flaskvisits cookie is set to 18.

Application Tab

To discover what cookies are set for different applications, you can use the DevTools applications tab. In the screenshot below, I chose that tab, then toggled the triangle next to cookies and chose the http://localhost:8080/ tab. Both are highlighted with magenta ovals. The devtools then shows the two cookie values.

If you double-click on any of the cookie values, you can modify them. If you right-click on the value, you'll get a menu of options, including "delete", so you can delete the value when you are developing and want to start over. Alternatively, if a line is highlighted, pressing the delete key will delete it.

Video of Dev Tools¶

Here is a Video of the Cookies Demo with the Dev Tools

Like the earlier one, this starts with a black screen, but there is a voiceover before I share my screen, so I can't clip the black part out. Sorry.