Shaders¶

Throughout the semester, we've talked about rendering and discussed the mathematics for various color computations, but we've hand-waved about how the shader actually computes the color of each pixel. In this reading, we'll start to get concrete about that.

Outside Readings¶

I've found the following articles helpful, but they are optional. I hope that my description will be sufficient. Focus on the concepts, rather than the code

• WebGL Fundamentals
• WebGL How it Works
• WebGL Shaders
• MDN WebGL Tutorial
• TutorialsPoint Graphics Pipeline

Rendering Pipeline¶

We can think of the rendering process as a pipeline where we put vertices and other information in at one end and pixels come out the other end. Historically (more than 20 years ago), none of that pipeline was programmable. Programmers could set various parameters and such, but these were only variations on fixed code that was hard-wired into the graphics card. But then an innovation came along that allowed code to be sent to the graphics card for execution. The result has been a revolution, both in computer graphics and the use of graphics cards for other high-performance computing, such as machine learning.

The rendering pipeline has several stages and those stages can vary across different implementations, but here's a pipeline described by TutorialsPoint (also see links above)

• vertex shader: maps vertices to clip space.
• primitive assembly (creates triangles)
• rasterization (creates pixels/fragments)
• fragment shader
• fragment operations

In WebGL, two of those stages are programmable, namely the vertex shader and the fragment shader. We will look at those now.

Vertex Shader¶

The vertex shader transforms vertex coordinates into other vertex coordinates. This gets them ready for the creation of triangles and for later computations by the fragment shader. (It doesn't do any "shading", but these programmable pipeline stages are all called "shaders".)

The output of the vertex shader is done by assigning a value (a four-place vector in homogeneous coordinates) to a variable called gl_Position. The coordinate system is called "clip space".

We've talked about lots of coordinate systems. Here's a brief reminder of the series of transformations:

Model Space → World Space → View Space → Clip Space → NDC → Screen Space

• Model space is the local coordinate system of an object, say a box, sphere, or torus.
• World space is how the vertices of the object land in the entire scene.
• View space is the coordinates from the point of view of the camera and after projection (but not perspective division)

Clip space is a homogeneous coordinate system where:

• x, y, and z values typically range from -w to w (before perspective division). In this case, you can think of w as half the width of the box-like view volume.
• Clip space is after all transformations: model → world → view → projection.
• The vertex shader does all of this, taking the vertices from the model space to clip space.
• The GPU performs a perspective divide after the vertex shader, turning gl_Position from (x, y, z, w) into normalized device coordinates (NDC):

$$ \left( \frac{x}{w}, \frac{y}{w}, \frac{z}{w} \right) $$

We are familiar with NDC. Clip space is similar, but not in the range [-1,+1]. You can think of the coordinates as being in the range [-w,+w], because if they are outside that range, they will be clipped off.

In some cases, clipping will result in different triangles. Consider a triangle one of whose vertices is outside the box. The point of the triangle is chopped off and the new triangles are created that lie entirely within the view volume, like so:

In summary:

• The vertex shader transforms object coordinates to clip space coordinates
• The output variable, gl_Position, is in clip space.
• The GPU automatically does a perspective divide to move from clip space to NDC.
• After NDC, coordinates are mapped to the screen (viewport).

Fragment Shader¶

The fragment shader, on the other hand, actually does compute colors. Its output is a four-place vector: RGBA, with all values between 0 and 1. The output variable is called gl_FragColor.

It's called a fragment shader, not a pixel shader. These are almost the same thing, but not quite. Let's discuss what happens to fragments.

Fragments versus Pixels¶

A fragment has been called a "candidate pixel":

• A fragment is a data structure generated during rasterization, representing a potential contribution to a pixel on the screen.
• A pixel is the actual final colored dot you see on the screen after all processing.

A fragment is like a "pixel-in-waiting" that still needs to pass some tests. Here’s why not all fragments become pixels:

1. Depth Test Fails
   • If a fragment is behind something else (based on the depth buffer, AKA the z-buffer), it's thrown out.
   • Example: Drawing a tree behind a wall — you don't want the tree's fragments overwriting the wall’s.
2. Backface Culling
   • Fragments from triangles facing away can be ignored before they even produce fragments, depending on render settings.
3. Stencil Test Fails
   • We haven't discussed stencils, but they are used for more advanced masking effects (e.g., only draw inside a portal window).
4. Alpha Test or Blending Rules
   • If your shader writes a transparent color and blending is off or misconfigured, it might get discarded.
   • We saw the alpha test in the context of transparent texels in textures
5. Manually Discarded in Shader
   • You can use the discard keyword in a fragment shader if you have some reason to discard a fragment:
   • if (gl_FragCoord.x < 100.0) discard;

But, when we are writing a fragment shader, we can just think about the color we want the pixel to be, even if it ends up getting discarded later in the pipeline.

So, we can think of the pipeline like this:

1. Vertex Shader → Calculates position (gl_Position)
2. Clipping and conversion to NDC
3. Rasterization → Turns triangles into fragments
4. Fragment Shader → Runs per fragment, outputs color (gl_FragColor)
5. Depth Test (Z-test) → Compares fragment depth to what's already in the depth buffer
6. Other Tests & Blending → Stencil test, alpha blending, etc.
7. Write to Framebuffer → If all tests pass

We've talked a bit about the output of the two shaders, so let's talk about their inputs.

Buffers, Uniforms, Varying¶

One of the most important things that the shaders need is information about the vertices. These are conveyed in objects called buffers. We saw buffers back when we talked about custom geometry where we learned that a BufferGeometry object (which all of our objects are based on) has attributes that are typed arrays. Some of the attributes are:

• position spatial coordinates for the vertex
• normal coordinates of the normal vector for vertex
• uv texture coordinates for the vertex

These values are available to the shaders. Buffers are the primary way that the shaders access the values from our program. For efficiency, the data is copied to GPU memory.

Let's think back to our custom geometry, where we created ordinary JS arrays of coordinates, then we copied them to typed arrays and made those arrays the attributes of the BufferGeometry object. Let's walk through the steps of how that vertex data gets to the GPU.

1. We define geometry in JavaScript:

const positions = new Float32Array([
  0, 0, 0,
  1, 0, 0,
  0, 1, 0
]);
geometry.setAttribute('position', new THREE.BufferAttribute(positions, 3));

2. Three.js wraps that into WebGL buffers. That is, under the hood, Three.js calls WebGL’s gl.bufferData(...), which copies the typed array to GPU memory. In the GPU memory, these are called Vertex Buffer Objects or VBO.

3. This copying happens only once, unless you modify the buffer. That saves a ton of data transmission, which particularly important if we are in a render loop.

If the data is static, it’s uploaded once and reused for every frame. If you modify it (e.g., for animation), Three.js needs to know, so you must flag it:

geometry.attributes.position.needsUpdate = true;

That tells Three.js: “Hey, re-upload this buffer to the GPU.”

Uniforms¶

Another important input to the shaders are called uniforms, because they are read-only and don't change during the rendering process. Uniforms are global, read-only variables that shaders can access — they’re constant during the draw call, meaning every vertex or fragment gets the same value. Uniforms are used to pass "global" state into the shaders.

What Kind of Data Do Uniforms Hold?¶

Some data that might end up in a uniform might be the modelView matrix. We'll see an example of that in a moment.

How do we specify these?¶

In Threejs, uniforms are part of the material:

const material = new THREE.ShaderMaterial({
  uniforms: {
    uTime: { value: 0.0 },
    uColor: { value: new THREE.Color(0xff0000) }
  },
  vertexShader,
  fragmentShader
});

Given a material like the above, the shader code can reference "global" variables like uTime and uColor

Automatic Uniforms¶

Three.js automatically injects some uniforms into shader materials, including:

• modelMatrix — transforms object space → world space
• viewMatrix — transforms world space → camera (view) space
• modelViewMatrix — combo: object → camera space
• projectionMatrix — camera projection (perspective or orthographic)
• normalMatrix — for transforming normals correctly

The WebGL Shader Language¶

Before we look at our first examples of shaders, a few notes about the GLSL (WebGL) language:

• The syntax looks a lot like C, Java and JavaScript, both of the latter being in the C family of languages.
• The language is strongly typed, in the sense that every variable has to have a declared type.
• The language is special-purpose for CG, so in addition to simple types like float it has types like
  • vec3 for a three-place vector of floats
  • mat4 for a 4x4 matrix of floats
  • sampler2D for a two-dimensional texture
  • we saw some of these in the table about uniforms, above
  • See WebGL - Shaders for more

Enough preliminaries. Let's see some examples of shaders in Threejs.

Demo: Phong By Hand¶

The following demo has a cube with green Phong material, but we've done the vertex shader and the fragment shader "by hand" instead of using the built-in Threejs materials.

• Phong By Hand

The following sections will discuss the two shaders.

Vertex Shader Example¶

Recall that the vertex shader needs to convert the coordinates of the box (which are just in the usual local coordinates, written into a buffer) to clip space coordinates. Here's the vertex shader in its entirety:

    varying vec3 vNormal;
    varying vec3 vPosition;
    void main() {
      vNormal = normalize(normalMatrix * normal);
      vPosition = (modelViewMatrix * vec4(position, 1.0)).xyz;
      gl_Position = projectionMatrix * modelViewMatrix * vec4(position, 1.0);
    }

The first two lines declare two variables of type varying. These variables are created in the pipeline to pass information to later stages. In particular, the vertex shader can pass information to the fragment shader. Look back at the webgl graphics pipeline. Here, the fragment shader, which will be computing the Phong model, needs to know the normal vector for this vertex and also its position, so those will be computed and put in the varying variables.

Then, we begin a function named main. The first function in any C program is called main, so that name is used here.

The first line of main computes vNormal. It takes the normalMatrix and multiplies it by the normal for this vertex (that variable comes from the buffer, which originated in our geometry object) to transform the normal vector based on any modelview transformations that have been done to this box. (For example, the box might have been translated, rotated or scaled, all of which would affect the normal vectors. There are lots of rotations in the demo above.) The normalMatrix is similar in spirit to the modelView matrix in that it tranforms things based on how they are placed in the scene, but differs in that it works correctly for normal vectors as opposed to positions.

The second line computes vPosition. It starts with the position variable, which comes from the buffer attribute of that name, appends a 1.0 on the end to create a vec4 in homogeneous coordinates. Then it multiplies that vec4 with the modelView matrix (which is a mat4) to transform it to world coordinates. Finally, the code copies just the xyz components (dropping the w component) into the vPosition variable. The vPosition variable will be used by the fragment shader as well.

Finally, the last line does a similar calculation, but also multiplies by the projectionMatrix which converts from world space to clip space. Clip space is from the point of view of the camera. The final value is kept in homogeneous coordinates because the next stage of the pipeline will do perspective division and so it needs that w value.

I think it's cool that all that stuff we discussed at the beginning of the course with matrix multiplications of positions in homogenous coordinates are being done in real code that we can see!

Phong Fragment Shader¶

Now, let's look at the fragment shader. That code is a bit longer, but don't be intimidated.

    uniform vec3 uLightPosition;
    uniform vec3 uLightColor;
    uniform vec3 uAmbientColor;
    uniform vec3 uSpecColor;
    uniform float uShininess;
    uniform vec3 uBaseColor;
    varying vec3 vNormal;
    varying vec3 vPosition;

    void main() {
      vec3 normal = normalize(vNormal);
      vec3 lightDir = normalize(uLightPosition - vPosition);
      vec3 viewDir = normalize(-vPosition);
      vec3 reflectDir = reflect(-lightDir, normal);

      float diff = max(dot(normal, lightDir), 0.0);
      float spec = 0.0;
      if (diff > 0.0) {
        spec = pow(max(dot(viewDir, reflectDir), 0.0), uShininess);
      }

      vec3 ambient = uAmbientColor * uBaseColor;
      vec3 diffuse = diff * uLightColor * uBaseColor;
      vec3 specular = spec * uLightColor * uSpecColor;

      vec3 color = ambient + diffuse + specular;
      gl_FragColor = vec4(color, 1.0);
    }

In the Phong model, we know that we need to know the position of the light, its color, the color of the ambient light and so forth. That accounts for all the uniform declarations at the top of the code. Those are all, in a sense, global variables usable by this function. We also see declarations of the varying values coming from the vertex shader, which are additional global variables we can use.

When main begins, we first normalize the surface normal, meaning to make it unit length. The normalize() function is built-in to GSLS, the shader language.

We compute the direction to the light from this vertex as lightDir just by subtracting the location of this vertex (in world coordinates) from the light's position (which is also in world coordinates). And, of course, we normalize the resulting vector.

Next, we compute the direction to the eye or viewpoint (viewDir) as the negative of the vertex position (vPosition). That makes sense because the camera (the eye) is at the origin.

The GLSL language also has a built-in function to compute the vector that is the reflection of a given vector. Very cool.

Next, we take the dot product of the surface normal and the light direction to compute the magnitude of the diffuse component of the light. If the dot product is negative, the surface is facing away from the light, so we use zero instead. If the surface is facing the light (diff > 0.0), we also compute the magnitude of the specular component.

Finally, we take the base color of the object, multiply it by these various magnitudes, add them up and that's the color of this fragment!

Setting the Uniforms¶

We saw that the fragment shader makes liberal use of these uniforms. How does that information get specified? It's specified in the shader material, which Threejs will then put in the uniforms. Here's the code that sets up the box:

  const uniforms = {
    uLightPosition: { value: new THREE.Vector3(5, 5, 5) },
    uLightColor: { value: new THREE.Color(1, 1, 1) },
    uAmbientColor: { value: new THREE.Color(0.1, 0.1, 0.1) },
    uSpecColor: { value: new THREE.Color(1, 1, 1) },
    uShininess: { value: 30.0 },
    uBaseColor: { value: new THREE.Color(0x00ff80) }
  };

  const material = new THREE.ShaderMaterial({
    vertexShader,
    fragmentShader,
    uniforms
  });

  const geometry = new THREE.BoxGeometry();
  const cube = new THREE.Mesh(geometry, material);
  scene.add(cube);

As you can see, all the work goes into the material; the geometry is the same stuff we've always used.

Oh, and we do need to move the cube around (which will change the modelView matrix):

  function animate() {
    requestAnimationFrame(animate);
    cube.rotation.x += 0.01;
    cube.rotation.y += 0.01;
    renderer.render(scene, camera);
  }
  animate();

Summary¶

• GSLS is a whole new language for programming the two shaders
• The vertex shader computes the gl_Position of the vertex in Clip Space coordinates (similar to NDC but not normalized to [-1,+1]
• The fragment shader computes the gl_FragColor of the fragment in RGBA
• The shaders get the vertex information from vertex buffer objects
• The shaders get other "global" information from uniforms
• The vertex shader can compute additional values and store them in varying variables

Demos¶

• Phong By Hand