Express: Ports, Routes, Handlers, Endpoints and Parameterized Endpoints

In this reading, we'll get started learning about Express, a web application framework for Node.js.

As we discussed at the beginning of the semester, web applications can be viewed as a three tier architecture:

• the front-end or web browser
• the back-end of database tier
• the middle tier, which runs on the server, receiving requests from the front end and responding to them, possibly connecting to the database to serve those requests

Here's a picture we discussed:

So far in this course, we've been focussing on the front-end (the browser). Then we switched to the back-end, writing some Node.js scripts that can talk to a MongoDB database. Now we're going to start work with the middle tier, stitching those pieces together.

In this reading, we'll get started learning about Express, a web application framework for Node.js.

Express is a module that I've installed in our omnibus node_modules folder, which you have symlinks to, so you already have access to it. Express will allow us to create a web server.

What does that mean? A web server is software that

• runs continuously,
• listening to the network
• on a particular port
• for HTTP requests from browsers, and then
• it parses the request, representing the information in it as an object,
• routes the request to the appropriate code to generate a response, and,
• sends the response back to the browser

Apache is the industrial-strength web server that we run on Tempest for serving web pages (like this one). There are other web servers, meaning software packages that provide web services like Apache and Node.js+Express.

Web Application Frameworks¶

Express is a web application framework, meaning that it helps to build web applications (by "web application", I mean a web server dedicated to a particular application, such as FreeCycle).

A web application framework systematizes some important but routine aspects of web applications. Most modern web applications share the following features:

• They use some kind of templating system for creating the responses. This allows separating the computed, varying parts of the response from the static HTML and other front-end stuff. (These are sometimes called the views; see below.)
• They handle a number of different use cases (ways that the app is used) in a single program.
• Often, these different use cases are accessed via different URLs. For example, the IMDB has one URL for each person and one for each movie, but of course these are generated by software when requested. In general a web application comprises a set of URLs. The web framework has just one main back-end script (plus, of course, supporting modules and files).
• The different use cases are distinguished by some initial logic that analyzes the incoming request and decides how to handle it. A web framework centralizes that logic. This logic is also called routing. (It is also sometimes called the controller; see below.)

Frameworks usually involve a long-term process (in the sense of an operating system process). That process lasts across many requests, so templates don't have to be re-read from disk each time, and other persistent information can be retained in memory.

Model-View-Controller¶

Web frameworks are often described as fitting into the Model-View-Controller (MVC) paradigm. In that analysis:

• The database is the model.
• The logic to determine the use case and how to handle it is the controller.
• The routes and templates that determine what the user sees are called the views.

This terminology is helpful if you already understand MVC, but if you don't, it's useless. Make of it what you will.

Background Concepts¶

There are a few background concepts that we'll run into when using Express, so I'd like to describe those first.

Process¶

A process, in computer science, is a program that is currently executing. Programs that are executing have memory allocated to them that they can store useful stuff in. We all know that if our laptop runs out of battery power or crashes, certain things don't get saved, such as the changes we made to a MS Word document or to a file being edited by Emacs. That's because the changes were in memory but not (yet) saved to disk. (Some applications save to crash recovery files and other kinds of safety nets; don't let those features confuse the issue.)

Some processes run quickly and exit right away. For example, an ls command runs for less than a second. Other processes stick around for long periods. (I might have Chrome, Firefox, and Emacs all running for weeks on my computer.)

Express is more like the latter: a long-running process. Indeed, it might run for weeks. Essentially, Express starts an infinite loop in our Node.js program/process, where the loop listens for web requests and handles them. When we develop and debug our apps, we'll probably run Node/Express for minutes or hours, killing and restarting it as needed to load new code and such.

Ports¶

A server computer like Tempest has a number of processes listening to the network. Each process listens on a different port. A port is just a 16-bit number from 1 to 65536. You can think of a port as a numbered door (the French word for door is porte) to the machine. Incoming network packets specify which port they should go to, and then they're handled by the process listening on that port.

Ports numbered less than 1024 are reserved for the operating system. Here are some standard services and the ports they listen on:

• 22: ssh (this is how you ssh to a machine)
• 25: sendmail
• 80: http (Apache)
• 443: https (Apache using secure connections)

There are many others; it's not important to remember any of these numbers. The point is that different services listen on different ports. Think of them as different offices in town hall: you go to different offices to pay your taxes, schedule a building inspection, complain about trash collection, etc.

If Apache is running on Tempest listening on port 80, and you run your Express application on Tempest, what port is your application running on? It can't listen on port 80 without interfering with the normal web services that Tempest provides. (You wouldn't be able to open port 80 anyhow, since it's a special port number.) It can't listen on any other ports that are in use, either.

Furthermore, only one process can listen to any port, so if each student in CS 304 is running an Express process, we all have to use different ports.

Consequently, we all need a way to open a port on Tempest that is unique to us. Fortunately, each user on a Unix system has a user id (UID) as well as a username. The UID is a unique number that will be perfect to keep us all separate. (If you're curious, you can find out your UID by logging in and using the id command.) We'll write our Express programs to open up a port number corresponding to our UID. So if your UID is 6543, you'll open port 6543.

Note that if you are working on your own laptop, there is less contention for ports, so you could use other choices.

When using a web browser, if you want to send your web requests to a port other than the default HTTP port (80), simply add a colon and the port number to the URL. For example, when you're developing with Express on your own machine, you might use a URL like this:

http://localhost:3000/hello

The Express documentation will have examples like that a lot.

Tempest URLs¶

When we develop our Express apps, instead of localhost and port 3000, we'll each use URLs like this, where the 6543 is our UID and hence our port:

http://cs.wellesley.edu:6543/hello

The CS 304node course account has UID 9160, so when I login to the CS 304 course account and run my Express app, it opens port 9160 and I access it at:

http://cs.wellesley.edu:9160/hello

This will be tricky for the first day or two, and then it'll become completely routine.

Express¶

Since Express is a common, well-used web framework, good documentation already exists for it. You're welcome to consult outside tutorials, such as expressjs.com.

Please read the following pages from that site:

• Express Starter
• Express Routing
• Express Static Files

Express Routing¶

One of the crucial features that Express provides is routing. That means that Express listens on the network for web requests, and when one comes in, it does the following:

• parses the information in the request and creates an object to represent that information.
• it creates a second object that will represent the response (initially empty)
• it looks at the endpoint of the URL to determine what function will handle this request.
• it invokes the handler function with these two objects: the request and the response.
• the handler function reads information from the request object to understand the details of the request and uses methods on response object to build the response. The response is probably a web page of some sort.
• The handler function returns the response to Express, which formats it and sends it to the browser.

Our work is almost entirely in writing handler functions for different kinds of requests.

Here's a visualization:

Hello World Example¶

Here's code from the Express Starter page, above:

const express = require('express')
const app = express()
const port = 3000

app.get('/', (req, res) => {
  res.send('Hello World!')
})

app.listen(port, () => {
  console.log(`Example app listening on port ${port}`)
})

1. The first step just loads the Express module (from node_modules) into our Node.js process.
2. The second step creates an object that will be our web application, storing it in a global variable (actually, a constant, since we won't ever re-assign its value). Conventionally, this is called app. Almost everything will hang on app.
3. This step defines what port to open. We'll do something a little more complicated so that we can open a port depending on our UID.

The next step, app.get is complicated. The get method of app takes two arguments, an endpoint and a function. The "endpoint" is the end of a URL, after the https://cs.wellesley.edu:9160/ part. If I have a website with several URLs:

• https://cs.wellesley.edu:9160/ the main page
• https://cs.wellesley.edu:9160/about the "about us" page
• https://cs.wellesley.edu:9160/contact page of contact information
• https://cs.wellesley.edu:9160/order a form for ordering merch

The endpoints are the /, /about, /contact and /order.

In the Hello World example, the endpoint is the shortest possible endpoint, namely /.

The second argument to get is a function, which is almost always an arrow function with two arguments, usually called req (short for request) and res (short for response). This function is a callback function. When a request comes in that matches the endpoint, Express invokes the callback function, providing two objects:

• the first has information about the incoming request, and
• the second is an object that allows the callback to generate a response to the request.

In other words, information coming in from the browser and information going out to the browser.¹

We can also refer to the callback function as a handler, since it handles the request. In general, Express maintains a set of mappings between endpoints and handlers, and routes incoming requests to the appropriate handler(s).

In this particular handler, the code is very simple:

app.get('/', (req, res) => {
  res.send('Hello World!')
})

The handler function just uses the send method of the response object to send a string back to the browser.

Most of our work in building our web applications will be in writing handler functions.

The last step is more straightforward:

app.listen(port, () => {
  console.log(`Example app listening on port ${port}`)
})

This code just opens the desired network port and starts an infinite loop, listening for requests on that port.

Hello World, Again¶

There's a version of the Hello World example that we can run on our server. The code is a lot longer, due to some extra packages, explained below. We'll run this example in class.

const path = require('path');
require("dotenv").config({ path: path.join(process.env.HOME, '.cs304env')});
const express = require('express');
const morgan = require('morgan');
const serveStatic = require('serve-static');

// our modules loaded from cwd

const { Connection } = require('./connection');
const cs304 = require('./cs304');

// Create and configure the app

const app = express();

// Morgan reports the final status code of a request's response
app.use(morgan('tiny'));

app.use(cs304.logStartRequest);

app.use(serveStatic('public'));

// ================================================================
// custom routes here

// This is our simple Hello World example
app.get('/', (req, res) => {
    return res.send('Hello World!');
});

// ================================================================
// postlude

const serverPort = cs304.getPort(8080);

// this is last, because it never returns
app.listen(serverPort, function() {
    console.log(`listening on ${serverPort}`);
    console.log(`visit http://cs.wellesley.edu:${serverPort}/`);
    console.log('^C to exit');
});

The code breaks down into three phases, separated by lines of equals signs.²

• the prelude, which loads a bunch of Node modules and configures the app
• the main part, which has all the routes and handlers
• the postlude, which starts the listener loop on the desired port

As the semester progresses, we'll add a bit more to the prelude. For now, I wanted to keep it relatively lean, even though it's a lot more code than the very concise "Hello World" above. Your work will mostly be in the main section.

Let's look at some of that extra code in the prelude:

require("dotenv").config({ path: path.join(process.env.HOME, '.cs304env')});

This line is from our MongoDB queries boilerplate. It reads the .cs304env file in our home directory so that we can connect to the MongoDB database. This process doesn't connect to the database, but since you've seen this line of code before, I've left it in.

const morgan = require('morgan');

It's nice to get a console.log message saying when a response has been generated. Morgan is a module that does that, so we load it.

const serveStatic = require('serve-static');

Very soon, we'll want to send web pages, pictures, and other static files to the browser, instead of just "Hello World!", and this module helps with that.

// our modules loaded from cwd

const { Connection } = require('./connection');
const cs304 = require('./cs304');

More lines of code that you've seen before, to help with connecting to the MongoDB server.

// Morgan reports the final status code of a request's response
app.use(morgan('tiny'));

This line configures Morgan. Morgan can be terse or verbose and can be configured in lots of ways; this configures a pretty terse output.

app.use(cs304.logStartRequest);

Morgan is useful, but it comes at the end of the response, after any error messages. That can be a little confusing. I like seeing a console.log message at the start of the request, so I created some middleware to do that. (More about middleware in a moment.)

app.use(serveStatic('public'));

This finishes the configuration of the module that serves static files.

Running Hello World¶

We run our Express apps the same way we run other Node programs: we cd to the folder with the .js file in it, and run it using node. Like this:

cd ~cs304node/apps/express1/
node hello-world.js

We will usually name our files server.js, but in this case, I wanted to put several examples in one folder, so I had to give them more specific names.

We then can go to a web browser to http://cs.wellesley.edu:9160 (substituting your port number for the 9160) and see the results.

The node program will print console.log output to the terminal window. It might look like this:

[cs304node@tempest express1] node hello-world.js
using UID as port 9160
listening on 9160
GET / at 11:59:12 AM
GET / 200 12 - 22.639 ms
GET / at 11:59:15 AM
GET / 200 12 - 0.889 ms

The first two lines are from the "postlude", where our app is starting up.

The second pair of lines is from a single request from our browser:

GET / at 11:59:12 AM
GET / 200 12 - 22.639 ms

The first line is when the request starts, and it is printed by the cs304.logStartRequest middleware. It just reports the method of request (GET), the endpoint (/) and the time of day (12:59:12 AM). The time of day isn't that useful, but it help distinguish one request from another. More on methods later.

The second line is when the response is sent, and it is printed by the Morgan middleware. It reports the method and endpoint, and then the response code (200 is success), the number of bytes sent ("Hello World!" is 12 bytes long), and the amount of time the request took.

If your handler does any console.log statements, they would be printed between these two statements.

MiddleWare¶

Express is written in a modular way, with extra modules (such as Morgan) implemented as middleware. In the context of Express, what this means is that when a request comes in, it goes through a chain or pipeline of functions. Each function takes three arguments:

• a request object (req)
• a response object (res)
• a next function, which represents the next function in the chain. Typically, this argument is called next but that's just convention.

The function can do whatever it wants, but often it works by modifying either the request or response objects.

If the function invokes the next function, the chain continues. If it doesn't, the chain ends.

To add a function to the chain by the app.use method. Here's an example from the middleware page:

const express = require('express')
const app = express()

app.use((req, res, next) => {
  console.log('Time:', Date.now())
  next()
})

That middleware function just prints the time and continues the chain. The cs304.logStartRequest is a variation on that function. Like this:

/* This function provides something similar to morgan('tiny') but at
 * the beginning of the request rather than at the end. You can add it
 * to the middleware like this:

app.use(cs304.logStartRequest);
*/

function logStartRequest(req, res, next) {
    let now = new Date();
    let now_time = now.toLocaleTimeString();
    console.log(`${req.method} ${req.url} at ${now_time}`);
    next();
}

You can see how it accesses the method and url (the endpoint) from the request object.

Request Methods¶

The vast majority of web requests are trying to get an HTML page from the server, or, more generally getting a file, whether an HTML file, CSS file, JS file, image file, or something like that.

Clicking on a hyperlink generates a GET request, as we saw above with the hello-world.js app.

Some web requests are sending information to the server. When you post a comment or a picture to social media, you are uploading information to a server somewhere. That kind of web request is a POST request, or we say that the request is using the POST method.

Because this is already a long reading with lots of new concepts, I will defer a discussion of GET versus POST, though feel free to read that page if you have time.

Parameterized Routes (Endpoints)¶

The mapping from endpoints to handlers that Express maintains doesn't have to just be fixed strings as endpoints. The endpoint can also be a pattern that then matches against a variety of actual URLs. Therefore, all the different URLs matching some pattern can be handled by one handler function.

As an example, suppose we have a in-memory dictionary of world capitals. This is essentially a tiny database. (We could put all 190+ countries in there, but I'm just going to put a few in there.) We can then have routes (endpoints) to show the complete dictionary, namely /all and also a parameterized endpoint to look up the capital of a specified country, namely /capital/:country, which is then a pattern that matches any URL that ends with /capital/ and some additional text. Let's see the code:

app.get('/', (req, res) => {
    return res.send('App to Look up World Capitals');
});

// Here's a tiny database of countries and their capitals

capitals = {Canada: 'Ottowa',
            China: 'Beijing',
            France: 'Paris',
            India: 'New Delhi',
            Nigeria: 'Abuja',
            US: 'Washington, D. C.'};

// route to return all the known capitals

app.get('/all', (req, res) => {
    let page = '<h1>All Capitals</h1>';
    Object.keys(capitals).forEach( country => {
        let city = capitals[country];
        page += `<p>${city} is the capital of ${country}</p>`;
    });
    res.send(page);
});

// route to look up the capital of a particular country, supplied in the URL

app.get('/capital/:country', (req, res) => {
    let country = req.params.country;
    let city = capitals[country];
    let page = '<h1>Capital</h1>';
    page += `<p>${city} is the capital of ${country}</p>`;
    res.send(page);
});

We can make requests like:

/
/all
/capital/Canada
/capital/China
/capital/France

We'll see this in action in class.

Express Summary¶

• Express is a web application framework.
• Express listens on a port: just a number to distinguish one listener from another
• Express starts up an infinite loop, listening on the port
• Express handles routing for us, allowing us to set up handlers for different endpoints.
• We define handlers by using app.get(endpoint, handler)
• The handler is a function, typically an arrow function with two arguments: req and res
• Other features are defined by middleware, which are functions that run in a chain before the final handler.
• Endpoints can be parameterized, allowing one handler to handle all the endpoints matching some pattern
• Parameters in an endpoint are done with a colon: /fixed/:param

There is a bit more after this summary, so don't stop reading yet.

VPN and SSH Tunnel¶

The section on Tempest URLs works great from on-campus, but fails if we are off-campus (which is true for me when I work from home, and for many of you when you are off-campus for cross-registered classes, visiting home, etc.). That's because the campus firewall only allows a handful of ports to be open "through" the firewall (ports 22, 80, and 443 for example, but not 9160).

There are two choices we have:

• Use the VPN. That changes your laptop to be inside the campus firewall, and you can access your app on the port with your UID, as described above. This is relatively easy and works. It can be a little slow (traffic goes back and forth to campus extra times because of the VPN), but I've never really noticed much lag. On the other hand, I live 3 miles from campus, not on the other side of the world.
• Use an SSH tunnel. The SSH program can set up a "tunnel" through the firewall, relaying traffic to port XYZ on Tempest to port ABC on your laptop.

The SSH incantation to set up a relay from 9160 on tempest to 8080 on my laptop is the following command, which I run in a terminal on my laptop, not on tempest:

ssh -L 8080:localhost:9160 anderson@cs.wellesley.edu

That logs me into my personal account (which I then ignore) and also sets up the SSH tunnel. I then go to my laptop's browser and access this URL:

localhost:8080

(8080, like 5000 and 8000, is commonly used for this purpose, but the number doesn't matter. But there's also no reason to get creative.)

This is described in our FAQ as well: Express from off-campus

There's also a video on our videos page.

I use SSH tunneling most of the time. I've even written a shell alias so that I don't have to remember the incantation. Ask me how if you're curious.