# The Fractal Quality of Decisions in Games

In setting out to map decisions, one of the first challenges we
encountered was to define what counts as a "decision." Different
theoretical approaches to games have different perspectives on this. For
example, most game theory modelling of decisions, and some
player-modelling approaches (see e.g., [@holmgard2014generative]),
side-step the issue by focusing on (possibly simultaneous) turn-based
decision-making, where one turn is one decision (see also
[@mawhorter2018choice] for work in games studies that takes this
approach). Similarly, reinforcement learning often models each frame of a
game as a decision, although this approach means each "decision" is a
tiny part of a larger-scale behavior (see [@ladosz2022exploration] for an
overview of specifically "deep" reinforcement learning focused on the
kinds of exploration we're interested in here). To address this issue,
hierarchical decision models have been used in both reinforcement
learning [@kulkarni2016hierarchical] and other game AI contexts such as
in planning agents for real-time strategy games ([@robertson2014review]
provides a review). In fact, the real-time strategy game literature makes
an explicit distinction between "strategic" and "tactical" decision
contexts which hints at the fact that multiple layers of decision are
relevant to gameplay, and hierarchical task network planning explicitly
uses a hierarchy of decisions [Ibid.].

But it is not only real-time strategy games where this is true: in the
exploration games we're focused on, decision happen at several levels:

1. At the level of frame-to-frame button presses, both decisions of what
   to press when, and executions of carrying that out control the
   player's avatar's basic course of action in the game.
2. Above this level, there are moment-to-moment decisions about actions
   to take, like "attack that enemy" or "get up to that platform." The
   outcomes of these decisions form micro-level plans which the first
   layer of decisions is used to carry out.
3. Above that level, are meso-level decisions about where to go or what
   to do, like "kill the enemies in this room" or "go north to the next
   room.
4. But of course, those decisions are in service of carrying out even
   higher-level decisions, like "go to the fountain room" or "farm this
   area for more money."

...we can continue to identify higher decision levels, like "find the red
key" or "level up to level 50," and above that, whole-game-level goals,
like "beat the final boss" or "find the true ending." One could even
continue with goals outside of the game context, like "play some Hollow
Knight" or "show off to my new friend."

These decisions each break down into actions which themselves involve
further decisions about the details, which break down again, and so on.
Rather than talking about "the N levels of decisions in games" we instead
see things as a continuous scale of fractal decision-making. At the
bottom, there is pure execution and reflex, where the carrying out of a
decision no longer involves further decisions. But the decisions above
that do not necessarily resolve themselves into distinct layers, and
instead many decisions may cross between multiple "layers," so it seems
simplest to just think of the entire structure as "fractal:" decisions
lead to plans of action, which in turn are carried out by making
lower-level decisions, down to some limit of simple actions like "press
the 'A' button."

An important note: not all of these plans of action are really the result
of a conscious decision, but instead at multiple levels of
decision-making there may be subconscious processes which result in
action plans that structure lower-level decisions. In addition, some
actions taken (at multiple levels) are mistakes, rather than the result
of correctly following higher-level plans. We do not have the space here
to delve into these aspect in detail, and for now treat consciously- and
subconsciously-decided action plans as equivalent, and treat mistakes as
decision outcomes (but will note them when they happen).

When we talk about "decision mapping" in games, we do not intend to map
all decisions at every level. Instead, we try to focus on a meso-level of
decision, around the granularity of "go to the next room to the right."
This focus allows our maps to capture the key exploration moves that the
player makes without getting too bogged down in the details of movement.
Some of our journals will also attempt to capture certain higher-level
decisions also related to exploration, like "find a way to get to the
elevator." We also try to focus on only decisions relevant to exploring
the virtual space of the game, as opposed to those that consider powering
up or defeating enemies, although these concerns are often intertwined.


# References:


% Typical paper on modeling player decisions
@article{holmgard2014generative,
  title={Generative agents for player decision modeling in games},
  author={Holmgard, Christoffer and Liapis, Antonios and Togelius, Julian and Yannakakis, Georgios N},
  year={2014},
  publisher={Foundations of Digital Games}
}

% My own work here on choice poetics: formal analysis of individual
% decisions...
@inproceedings{mawhorter2018choice,
  title={Choice poetics by example},
  author={Mawhorter, Peter and Zegura, Carmen and Gray, Alex and Jhala, Arnav and Mateas, Michael and Wardrip-Fruin, Noah},
  booktitle={Arts},
  volume={7},
  number={3},
  pages={47},
  year={2018},
  organization={MDPI}
}

% A recent Deep Reinforcement Learning review centered on sparse-reward games
% and exploration strategies, with plenty of game mentions even though it's not
% exclusively game-focused.
@article{ladosz2022exploration,
author = {Ladosz, Pawel and Weng, Lilian and Kim, Minwoo and Oh, Hyondong},
title = {Exploration in Deep Reinforcement Learning: A Survey},
year = {2022},
issue_date = {Sep 2022},
publisher = {Elsevier Science Publishers B. V.},
address = {NLD},
volume = {85},
number = {C},
issn = {1566-2535},
url = {https://doi.org/10.1016/j.inffus.2022.03.003},
doi = {10.1016/j.inffus.2022.03.003},
journal = {Inf. Fusion},
month = {sep},
pages = {122},
numpages = {22},
keywords = {Intrinsic motivation, Exploration, Sparse reward problems, Deep reinforcement learning}
}

% Hierarchical deep reinforcement learning
@article{kulkarni2016hierarchical,
  title={Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation},
  author={Kulkarni, Tejas D and Narasimhan, Karthik and Saeedi, Ardavan and Tenenbaum, Josh},
  journal={Advances in neural information processing systems},
  volume={29},
  year={2016}
}

% Review of RTS AI with sections on HTN planning and strategic/tactical
% decision sections.
@article{robertson2014review,
  title={A review of real-time strategy game AI},
  author={Robertson, Glen and Watson, Ian},
  journal={Ai Magazine},
  volume={35},
  number={4},
  pages={75--104},
  year={2014}
}