An Investigation of Decision Noise and Horizon-Adaptive Exploration in the Explore-Exploit Dilemma

Abstract

Human and animals constantly face the tradeoff between exploring something new and exploiting what one has learned to be good. In this dissertation, I studied the properties of the heuristics that humans use to make the explore-exploit decisions. In the first study, I examined the nature of randomness in human behavior that is adaptive in exploration. Human decision making is inherently variable. While this variability is often seen as a sign of sub-optimality in human behavior, recent work suggests that randomness can actually be adaptive. A little randomness in explore-exploit decisions is remarkably effective as it encourages us to explore options we might otherwise ignore. From a modeling perspective, behavioral variability is essentially the variance that can not be explained by a model and is modeled as the level of decision noise. However, what we have called ”decision noise” in previous researches could actually just be missing deterministic components from the model, it is difficult to tell whether decision noise truly arises from a stochastic process. Here we show that, while both random and deterministic noise drive variability in behavior, the noise driving random exploration is predominantly random. In the second study, we further asked where the randomness in behavior comes from. In particular, we examined one candidate theory known as deep exploration that decisions are made through mental simulation in which behavioral variability can potentially come from the stochastic sampling process during such mental simulation. In the context of a stopping problem, we showed that deep exploration successfully accounts for the simultaneous strategic adaptation of stopping threshold and the adaptation of the level of behavioral variability in the task, suggesting a potential mechanism for how adaptive behavioral variability in human behavior is achieved. In the third study, we examined factors that modulate the behavioral adaptation of strategy and behavioral variability to the horizon context in explore-exploit decisions. One key factor in explore-exploit decisions is the planning horizon, i.e. how far ahead one plans when making the decision. Previous work has shown that humans can adapt the level of exploration to the horizon context, specifically, people are more biased towards less-known option (known as directed exploration) and behave more randomly (known as random exploration) in longer horizon context. However, Sadeghiyeh et al. (2018) showed that this horizon adaptive exploration critically depends on how the value information of the options are obtained by the participants, and participants only show horizon adaptive exploration when the value information is gained by action triggered responses (Active version), and don’t show horizon adaption if the information is presented without actions to retrieve them (Passive version). In the Passive version, participants showed no horizon adaptive directed or random exploration. This is true even if the same participant has played the Active version first. I conducted a series of experiments to further investigate what behavioral factors kill the horizon adaptive exploration in the passive condition. This work reveals a more complicated nature of explore-exploit decisions and suggests the influence of action on how subjective utility is computed in the brain.