Reading on Ajax and JSON

revised for Fall 2020

This reading covers the concepts behind Ajax and JSON, and why they fit together, and then explores a particular example, with code. We won't cover all aspects of Ajax and JSON, but we'll see more later in the course.

Ajax¶

What is Ajax? Why should we care?

It can be successfully argued that no single technology shift has transformed the landscape of the web more than Ajax. The ability to make asynchronous requests back to the server without the need to reload entire pages has enabled a whole new set of user-interaction paradigms and made DOM-scripted applications possible.

Bear Bibeault & Yehuda Katz
jQuery in Action, 2nd edition

Ajax is a technique where a user's interaction with the page causes some JavaScript to run, which connects (asynchronously) to a server to retrieve some information, and then the page is updated with the results. It's the essential functionality behind such many nice web applications, such as:

• Facebook menus (e.g. specifying the state and place where you went to high school). Posting to someone's timeline; liking a post, commenting on a post.
• Gmail. You can read and delete an email without reloading the whole Gmail window. You can compose and send an email without reloading the whole Gmail window.
• Google Docs. You can edit a document and the browser will automatically save to the cloud without you having to reload the whole document.
• Google maps, etc.

Ajax Concepts¶

Ajax can be used in two main modes, which also correspond to the methods of an HTML form.

• The web browser can GET some data from the back end. This data can be combined with the existing data on the page. You see this in Facebook, when it retrieves posts, comments, and so forth without you having to reload the page.
• The web browser can POST some data to the back-end. The back-end can do whatever it needs to with it. Gmail might POST a message you've sent, sending it to a Gmail server that will then forward it to the recipients. Similarly, when you click a like button on Facebook, that information is sent to the back end, updating counters on Facebook's servers.

Of course, these ideas can be combined, so when you like a Facebook post, it not only increments the counters in the back end, but returns the new value (which might include increments from other people), and your browser can update the number of likes that you see.

The difference between a normal form submission to the back-end and a submission via Ajax is that the response from a normal interaction has to be the entire web page, while with an Ajax interaction, the response might be only the relevant information, such as "your message was sent" or "the number of likes is now 25." That smaller payload is quicker to load over a network, and (usually) less of the page needs to be re-renderered.

Slideshow on Ajax Interaction¶

Consider the following slideshow on Ajax interaction. Here, the HTML part of the page is in blue and the JavaScript is in green. Click on the links below to go to that slide. The context of this slideshow is that the back-end is computing some values for us, based on values from the browser. In this example, we are sending the number 100 and getting the number 50 back. (The example uses PHP in the back-end, but it could just as easily be Flask.)

The harry.php script does the computation. The JS function in the browser that sends the data (the number 100) is named fred, and the JS function that receives the response (the number 50) is named george. The george function inserts the number 50 into a list on the page.

1. User types a value into the form
2. User clicks the submit button
3. Fred sends form data to Harry
4. Harry returns result to George
5. George inserts result into page

An alternative explanation that you're welcome to consult is this gentle introduction to Ajax.

Callbacks¶

There's an important concept depicted in that interaction, namely the notion of a callback. When the user clicks the button, the function fred runs immediately, sending the data to the server. However, the response from the server is not immediate. It could take tens of milliseconds or longer for the response to arrive.

We don't want fred to wait around for the response; that would lock up the browser, allowing nothing to proceed until the response arrives. If we did this, which is possible, the interaction is called synchronous. Normally, however, the processing is asynchronous. What that means is that fred tells the browser:

When the response arrives, invoke the function george with the data

The fred function can then finish, returning to its caller, and the browser proceeds.

Once the response arrives, whenever that is, george runs and does whatever we want done with the response. In this example, we wanted it added to a list of results on the page.

Ajax Response Types¶

The response from the back-end can come in many forms. In our example above, it was just a number. In general, it could be snippets of text or HTML to insert onto the page. It could be XML to be processed (this is the X in the acronym AJAX). Nowadays, it's most likely to be JSON. Let's take a detour to talk about this kind of data.

JSON¶

JSON, the JavaScript Object Notation, is a new standard for information interchange between programs, competing in the same market niche as XML. Ajax will allow us to get data from the back end, typically formatted as JSON. So, let's see how JSON works.

JSON is a subset of the JavaScript language, omitting things like function definitions. It's intended to comprise hierarchical data, like arrays and objects, and non-compositional data like numbers, strings, and booleans. Here's an example that we've seen before:

{"title": "Dr. Zhivago", 
 "tt": 59113, 
 "year": 1965, 
 "stars": ["Omar Sharif", "Julie Cristie", "Geraldine Chaplin"], 
 "isGreat": true, 
 "director": {"name": "David Lean", 
              "nm": 180, 
              "birth": "1908-03-25"}, 
 "IMDB rating": 8.0} 

Note that values can be scalars or compound; here we have an array literal of strings for the stars and an object for the director.

See JSON.org for more. That site also has a ton of info on packages and modules to process JSON notation in a variety of languages.

JSON notation comprises the following:

• null
• strings
• numbers
• booleans
• arrays (denoted with square brackets around any number of JSON values)
• objects (denoted with curly braces around a set of key/value pairs, where the keys are strings and the value can be any JSON data, the pairs are separated with commas, and a colon appears after a key)

Modern JavaScript implementations come with two functions that are useful for working with JSON:

• The JSON.stringify function turns a data structure into a string.
• The JSON.parse function turns a string back into a data structure.

Thus, if the back-end wanted to send us some complex data, instead of a simple number, sending it as JSON makes it easy to work with.

Extended Example: Collatz¶

For the rest of this reading, we'll look at an example using Flask and JavaScript/jQuery. In class, we'll download and run this example and I'll answer your questions, but it's best to be prepared. This reading will help you prepare.

Here it is in action: cs304collatz

In the example, the back-end will compute the Collatz function:

def collatz(num): 
    if num %2 == 0: 
        return num / 2 
    else: 
        return (num * 3) + 1 

This is, of course, a trivial function that we don't need a server-based solution for, but imagine that the number it is returning is, say, the number of likes of a FaceBook post.

High-Level View¶

Let's first get a conceptual overview of the workings of the Ajax behavior. First, a general picture:

Now, some concrete steps:

1. The user can submit a number to the back end by filling out and submitting a form via Ajax, rather than via the normal form-submission procedures.
2. The back end will compute the result, and instead of rendering a complete page, will just send back the essential information, something like one of these JSON objects:

    
   {error: false, in: 50, out: 25} 
   {error: false, in: 25, out: 76} 
   {error: true, err: "Could not convert 'five' to integer"} 
   

3. The front end (the browser) will receive that response and invoke a callback function (that we write) that will either report the error or add the in/out pair to a list in the browser.
4. We'll also add a feature where we can click on any of the list of in/out pairs in the browser, and the output value will be submitted and, the response will be added to the list.

We won't necessarily talk about these steps in this order. In fact, we won't start with any of them, we'll start with progressive enhancement.

Progressive Enhancement¶

Ajax is very slick, but you have to have JavaScript in your browser to use it. Most people do, of course, but there are special screen-readers for the blind and other special web browsers for people who need assistive technology, and not all of those unusual web browsers have JavaScript. There may also be people who have chosen to turn JavaScript off for some reason.

You could decide that you just don't care about such people, but you don't have to. The idea of progressive enhancement is to start with a functional but old-fashioned, 1990s-style web application, with no JavaScript, and then use the power of JavaScript to hide the old and reveal the new (or convert the old into the new). Then, people without JavaScript just see the old version, and people with JavaScript see the new version.

So, we're first going to look at the code that handles the non-Ajax usage. First, here's the relevant part of the HTML page:

<form id="number-submit-form" 
       method='GET' 
       action="{{url_for('getCollatzNormal')}}"> 
    <p><label for="form-num">Form Number</label>: 
        <input type="text" name="num" id="form-num"></p> 
    <p><input type="submit" id="btn-submit" value="submit"></p> 
</form> 

{% if x %} 
<p id="result">The next value after  
<span class="x">{{x}}</span>  
is  
<span class="y">{{y}}</span></p> 
{% endif %} 

In this case, I decided to use the GET method because it makes it easier to see the data (it's in the URL).

And here is the back-end code that makes the non-Ajax form work:

@app.route('/') 
def index(): 
    return render_template('collatz.html',title='Collatz') 

@app.route('/getCollatzNormal/', methods=['GET','POST']) 
def getCollatzNormal(): 
    if request.method == 'POST': 
        num = request.form.get('num') 
    else: 
        num = request.args.get('num') 
    try: 
        x = int(num) 
    except: 
        flash('Could not convert {num} to int'.format(num)) 
        return render_template('collatz.html',title='Collatz') 
    y = collatz(x) 
    return render_template('collatz.html', 
                           title='Collatz', x=x, y=y) 

This is just two simple routes. I've omitted the definition of collatz, since we saw it earlier. The route handles both GET and POST, though our form uses GET.

Handling an Ajax Response¶

Let's start on the Ajax behavior, but we'll start at the end of the Ajax behavior, where we add the results to the list on the page. (Recall our high-level view of the Ajax behavior.) Why at the end? Because the beginning is going to mention the addToList function we are about to define, and it's useful to know what that is referring to.

Let's assume that the back end will send us a response that looks like:

{error: false, in: 50, out: 25} 
{error: false, in: 25, out: 76} 
{error: true, err: "Could not convert 'five' to integer"} 

What do we want to do with it? We want to add it to the end of the list. We'll write a JS/JQ function that does that. But first, let's add some stuff to our page to put error messages and results:

<div id="errors"></div> 

<ul id="collatzNumbers"></ul> 

Now, with our HTML expanded a little, we can write a function to update those elements:

function addToList(obj) { 
    if(obj.error) { 
        $("#errors").empty().html('Error: '+obj.err); 
    } else { 
        $("<li>") 
            .attr("data-num",obj.out) 
            .text("the number "+obj.in+" produces "+obj.out) 
            .appendTo("#collatzNumbers"); 
    } 
} 

addToList({error: false, in: 50, out: 25});         
addToList({error: false, in: 27, out: 82});         

This code implements the function and shows two examples of it. Those are added to the list when the page loads. You'll see that in action in class.

The function adds a data-num attribute to the LI element, with the value being the output of the pair. We'll use this to add our behavior where clicking on a list item submits the output of that pair so that we can see what's next in the sequence.

Note how in our JQ code, sometimes we select the destination (#errors) and then operate on it, and sometimes we create some structure and then add it to some element (#collatzNumbers). jQuery accommodates many styles.

Jsonify the Back End¶

How does the back end create and send such a response? The back-end processing is very similar to the normal way; the only difference is the response object. Flask provides a function called jsonify that will turn a Python data structure into a JSON string suitable for sending back to the browser. Here's the route:

@app.route('/getCollatzAjax/', methods=['GET','POST']) 
def getCollatzAjax(): 
    if request.method == 'POST': 
        num = request.form.get('num') 
    else: 
        num = request.args.get('num') 
    try: 
        x = int(num) 
        y = collatz(x) 
        print(x,y) 
        return jsonify( {'error': False, 'in': x, 'out': y} ) 
    except Exception as err: 
        return jsonify( {'error': True, 'err': str(err) } ) 

Contrast that with our original, non-Ajax version; they are very similar! In particular, they process the request data in the same way. In fact, the back-end can't even tell whether the request was made using Ajax or the normal method. This route is called *Ajax because of the response, not the request.

Note that we have to handle errors in a different way. Flashing works because we render a template, but the Ajax response isn't rendering a template so flashing is useless. Here, we put a flag in the response saying whether there was an error and what it was. The front end is responsible for showing it somewhere (or ignoring it). We saw that handling in the addToList() function.

Making an Ajax Request¶

Now the centerpiece of the Ajax technology: making a request.

The jQuery library makes this much easier. The essential step is using one of the following methods:

• jQuery.get
• jQuery.post

Notice that there's no selector here; the functions are used like this:

$.get(url,data,success,datatype); 
$.post(url,data,success,datatype);

All arguments except the first are optional. The second argument allows you to send arbitrary form data, encoded in several different ways. Today, we'll use a JSON literal.

function sendNum(num) { 
    console.log("Sending "+num+" to the back end"); 
    $.get(URL,{'num': num},addToList); 
}

Note that the third argument is the addToList function from the last section. This function sends the data to the back-end, but it does not wait for the response. It hands the addToList function to the browser and says,

when the response arrives, invoke this function with the response as an argument

Event Handlers¶

Now, we need to set up an event handler to get the data from the form and send it via Ajax. What we'll do is attach an event handler function to the submit button of the form. Note that clicking on that button would normally submit the form in the old-fashioned way, but we don't want that default behavior. Therefore, we'll use a method of the event object that the browser will invoke our event handler with. Here's the code:

$("#btn-submit").on("click", function (event) { 
    if(progressive_enhancement_on) { 
        // don't do the normal form submission: 
        event.preventDefault();   
        var num = $("#form-num").val(); 
        sendNum(num); 
    } 
}); 

Did you see the use of the progressive_enhancement_on global? That's how we'll be able to tell if we are in "normal" mode or progressive-enhanced "ajax" mode. If we are in normal mode, this event handler does nothing. If we are in "ajax" mode, the event handler turns off normal form submission (by using event.preventDefault() and instead makes an Ajax request. It pulls the input value out of the form to send to the back end.

Clicking on List Items¶

Looking back at our high-level view, you remember that we also implemented a behavior where the user could click on items in the list. Let's implement that now.

It turns out that there's a phenomenon called event bubbling where a click on, say, one of our LI elements also counts as a click on the UL that is its parent, the ancestor of that UL, and so forth all the way up to the top of the document tree. We say that the event bubbles up and every element on that ancestry path can choose to handle the event or not.

Therefore, instead of adding an event handler to every LI in the UL#collatzNumbers, we'll add one event handler to the static ancestor and handle everything there.

But how does the single event handler know which of its many descendants was actually clicked? jQuery has a nice feature of delegated handlers, where you can specify a selector for the descendants to be matched, and the magic JS keyword called this is bound to the particular descendant that was clicked.

Here it is in action:

$("#collatzNumbers").on( 
        "click",       // event to handle 
        "[data-num]",  // which descendants are handled 
        function(event) { 
            var num = $(this).attr("data-num"); 
            sendNum(num); 
        }); 

The selector is the second argument to the .on method, and the handler function moves to third place. Here, we used [data-num] as the selector, meaning every descendant that has an attribute named data-num. The keyword this is bound to the particular element that was clicked, and we can then pull the attribute value off the element and send it to the back end using our Ajax function.

Adding Progressive Enhancement¶

Now, to add the progressive enhancement, we can see that our JS/JQ code just sets a global variable and changes the text of the button, so that we know that it is on or not:

var progressive_enhancement_on = false; 
$("#btn-progressive").on('click', function () { 
    progressive_enhancement_on = !progressive_enhancement_on; 
    if(progressive_enhancement_on) { 
        $("#btn-progressive").text('Progressive Enhancement ON'); 
    } else { 
        $("#btn-progressive").text('Progressive Enhancement OFF'); 
    }         
}); 

In real life, we would just turn progressive enhancement on, under the assumption that anyone who has JavaScript will want the better interface, but for the purposes of this class, it's useful to be able to turn it on/off so that we can see both behaviors. As you read the JS/JQ code below, keep in mind the global JS variable progressive_enhancement_on.

The whole thing¶

Here's the whole of the front-end. In real life, we would put the JS/JQ code in a separate file, but here I wanted to make it easy to look back and forth between the JS/JQ and HTML, since the first is operating on the second and there are many connections between them.

Several things to note:

• A button that we will use to turn progressive enhancement on/off
• An empty element where we can put error messages. The CSS makes that a red, bold font.
• a form to submit a number. In this case, I decided to use the GET method because it makes it easier to see the data (it's in the URL).
• Some conditional code to report a pair of numbers
• An (empty) list of past results.
• Some JS files that will be loaded after loading jQuery
• A JS function to handle clicking on the progressive enhancement toggle.

Summary¶

We've come a long way! We learned

• how to submit data using forms and GET/POST
• how to do the same thing from JS, even using progressive enhancement
• how to process a JSON response from the back-end
• how to update the DOM
• how to add an event handler and use bubbling to improve our code