Children seem to need to have physical objects to pretend with at first; only later can they pretend without any placeholder object. But what's so hard about learning to think about imaginary objects? Andreya, Allison, and Alex decided to try to train Hidden Markov Models to recognize different pretending actions, to find out what was computationally hard about this problem.

The system was trained to recognize actions with physical objects (lower left box), and then was tested on motions using imaginary objects. The system also recalled a binary image of the imagined object, and attempted to place it in the correct place in the image.

Determining rotation from hand orientation and grasp turns out to be rather difficult (top), though it's easier when the rotation is treated as a function of the motion (bottom). This suggests that children may use placeholder objects in pretending because the function for deciding object location from hand pose is difficult to learn. Understanding this learning better could lead to new methodologies in training gesture-based controllers, such as those slated for next-generation game consoles.