Validation and Verification

Arguably, Validation and Verification (V & V) are the most important aspects of simulation. Many books start with these concepts. What are they?

• Verification: Did we build the system right?

  In other words, does the simulation model work the way we intended it to work? In programming, this is called debugging.

• Validation: Did we build the right system?

  A simulation model is often supposed to match some real-world system. Does ours? You can have a perfectly debugged M/M/1 queuing simulation, but if the real-world system is an M/M/2 queuing system, the results aren't going to be useful.

These two steps not only sound similar, but in practice they can be quite similar. For example, one way to verify a system is to run it on some test data and see if it gets the correct result. If that test data and correct result are collected from the real-world system, this step is also validation.

However verification can also be looking at internal consistency and general magnitude of numbers.

Examples

Here are some examples we can discuss.

Ideal Gas Simulations

• Verification: The animation helps a lot here. At first, I just had a single turtle, and when I noticed that it was always on the lower right side, I looked into the random positioning code and found a bug. I also tried forcing the heading to be zero (due north) and tested to see that it just bounced up and down between the top and bottom. A few other consistency checks as well.

  In the Extend model, I checked that the attributes were properly being set (which took a while!) and that the number of items going through the queue were correct (10 items times 10 rounds should be 100 items), and other kinds of internal consistency. I also turned on the animation, so that I could be sure that the items were flowing around in a circle, as desired. However, the animation doesn't help ensure that the calculations of position are correct; that also had to be done. Comparing to the StarLogo model helped.

• Validation: Almost nothing was done here. I did make the mean speed be 300, because I remembered that the mean speed of air molecules at standard temperature is 300 meters per second. Because of the way the calculations work, the position of the walls and the distances between them to are real distances in meters. The turtle/item should go 300 meters total distance in 1 second.

  However, there are certain approximations I can't make, such as the number of turtles versus the number of gas molecules. This is one of the necessary approximations of the simulation.

Grocery Store

• Verification: We checked (or should have) that as the distribution of "number of items" moved towards higher values, the number that went through the express lanes decreased. For the multiple regular lanes, we checked that the number that went through each lane was roughly similar (none were zero, for example). That shows that our system is functioning roughly correctly.
• Validation: We basically made up an arrival rate. To do this for real, we'd want to keep track of arrivals (to the register) in an actual store. For this, we might post people in the store, watching the lines, for several hours or days, to collect data. Then we have to match that data (using histograms and summary statistics) to find a good fit. There is statistical software that will do that fit for us, including some in Extend.

  We also made up a distribution of number of items. Here, we can probably go over register tapes or other store records to find appropriate data for getting the distribution right.

  Service times. Fortunately, we may be able to get that data from register tapes and such.

  Finally, we have to validate the model's behavior. The person who observes arrival rates, should also record queue lengths and departure rates, so that we can compare those with how our model behaves. If they match reasonably well (perhaps some kind of statistical test), we can have good confidence in the model.

Once we have good confidence in the model, we can then try adding the express lanes in the model and see if the performance is better. If it is, we can add the express lanes in the real store, confident that it will have something like the desired effect. For best results, we should confirm that it does.

Note: there is a major movement in the business world to do just this kind of thing. There are buzzwords like TQM (Total Quality Management) and 6sigma that are based on this idea of measuring and evaluating things (although not always via simulation).

Lake Pollution

The issues here are very similar to the ones above.

• Verification: Without the random addition of additional pollution, we should get a couple of simple exponential decay curves. You may be able to think of other consistency checks.
• Validation: Consider a STELLA model that was presented this year at Ruhlman by an Environmental Studies course modeling (I think) phosphate levels in Lake Waban. The model was certainly configured with measured values, such as the rates of inflow and outflow, the current phosphate levels and so forth. They then would also have to check to see if predicted levels the next month or so were approximately correct. Ideally, they would check several times over a longish period.

1. This work is licensed under a Creative Commons License
2. 
3. 
4. 
5.