Week 9 - Westbrook & Braver, 2016 PDF

Preview

Summary

Weekly readings for PSYC20006 Biological Psychology...

Description

Neuron

Review Dopamine Does Double Duty in Motivating Cognitive Effort Andrew Westbrook 1,* and Todd S. Braver 1,2 1Department

of Psychological & Brain Sciences, Washington University in St. Louis, St. Louis, MO 63130, USA of Radiology, Washington University in St. Louis, St. Louis, MO 63110, USA *Correspondence: [email protected] http://dx.doi.org/10.1016/j.neuron.2015.12.029 2Department

Cognitive control is subjectively costly, suggesting that engagement is modulated in relationship to incentive state. Dopamine appears to play key roles. In particular, dopamine may mediate cognitive effort by two broad classes of functions: (1) modulating the functional parameters of working memory circuits subserving effortful cognition, and (2) mediating value-learning and decision-making about effortful cognitive action. Here, we tie together these two lines of research, proposing how dopamine serves ‘‘double duty’’, translating incentive information into cognitive motivation. Why is thinking effortful? Unlike physical exertion, there is no readily apparent metabolic cost (relative to ‘‘rest’’, which is already metabolically expensive) (Raichle and Mintun, 2006). And yet, we avoid engaging in demanding activities even when doing so might further valuable goals. This appears particularly true when goal pursuit requires extended allocation of working memory for cognitive control. One hypothesis is that cognitive effort avoidance is intended to minimize opportunity costs incurred by the allocation of working memory (Kurzban et al., 2013). If this is true, it suggests not only that working memory is allocated opportunistically, but also that allocation policies entail sophisticated cost-benefit decision-making that is sensitive to as yet unknown cost and incentive functions. In any case, the phenomenon raises a number of questions: How do brains track effort costs? What information is being tracked? How can incentives overcome such costs? What mechanisms mediate adaptive working memory allocation? Working memory capacity is sharply limited, especially in the domain of cognitive control, involving abstract, flexible, hierarchical rules for behavior selection. Optimizing working memory allocation is thus critical for optimizing behavior. Prevalent computational frameworks have proposed reward- or expectancy-maximization algorithms for working memory allocation (Botvinick et al., 2001; Donoso et al., 2014; O’Reilly and Frank, 2006). Yet, these frameworks largely neglect that working memory allocation itself carries affective valence. High subjective costs drive disengagement, whereas sufficient incentive drives engagement. That is, allocation of working memory is a motivated process. In this review, we argue that modulatory functions of the midbrain dopamine (DA) system translate cost-benefit information into adaptive working memory allocation. DA has been implicated in numerous processes including, but not limited to, motivation, learning, working memory, and decision-making. There are two largely independent literatures that ascribe disparate functional roles to DA with relevance to motivated cognition. First, DA influences the allocation of working memory directly by modulating the functional parameters of working memory circuits. For example, DA tone in the prefrontal cortex (PFC) influences the stability of working memory repre-

sentations, with higher extrasynaptic tone promoting greater stability, to a limit (Seamans and Yang, 2004). Phasic DA efflux may also push beyond the limit and toggle the PFC into a labile state such that working memory representations can be flexibly updated (Braver et al., 1999). Additionally, DA may support the learning of more sophisticated (and hierarchical) allocation policies via synaptic depression and potentiation in corticostriatal loops (Frank et al., 2001; O’Reilly and Frank, 2006). Second, DA is critical for action selection. Specifically, DA trains value functions for action selection via phasic reward prediction error dynamics potentiating behaviors that maximize reward with respect to effort in a given context (see Niv, 2009 for a review). DA tone in the striatum and the medial PFC also promotes preparatory and instrumental behaviors in response to conditioned stimuli and particularly effortful behavior (Kurniawan et al., 2011; Salamone and Correa, 2012). Here, we tie together these largely independent lines of research by proposing how the very same functional properties of DA encoding incentive information translate incentives into cognitive motivation by regulating working memory. Specifically, we propose that DA dynamics encoding incentive state promote subjectively costly working memory operations experienced as conscious, phenomenal effort. As we detail below, our proposal makes use of the concept of a ‘‘control episode’’ during goal pursuit (cf. ‘‘attentional episodes’’, see Duncan, 2013), involving stable maintenance of the goal state at higher-levels of the control hierarchy, along with selective updating of lower level rules for guiding behavior during completion of subgoals, as progress is made toward the ultimate goal state. We review the ways in which DA dynamics encoding a net cost-benefit of goal engagement and persistence result in adaptive working memory allocation. As such, DA translates incentive motivation into cognitive effort. Motivated Cognition Why Cognitive Effort Matters Cognitive effort is an everyday experience. The subjective costliness of cognitive effort is consequential, sometimes driving disengagement from otherwise highly valuable goals.

Neuron 89, February 17, 2016 ª2016 Elsevier Inc. 695

Neuron

Review Yet, surprisingly little is known about this phenomenon. It is neither clear what makes tasks effortful, nor why task engagement is apparently aversive in the first place (Inzlicht et al., 2014; Kurzban et al., 2013). Beyond a quizzical influence over goal-directed behavior, there are numerous reasons to care about cognitive effort. First, expenditure is critical for career and educational success, economic decision-making, and attitude formation (Cacioppo et al., 1996; von Stumm et al., 2011). Second, deficient effort may be a significant component of neuropsychiatric disorders for which avolition, anhedonia, and inattention feature prominently, such as attention deficit hyperactivity disorder (ADHD) (Volkow et al., 2011), depression (Hammar et al., 2011), and schizophrenia (Strauss et al., 2015). Effort avoidance may also contribute to declining cognitive performance in healthy aging (Hess and Ennis, 2012; Westbrook et al., 2013). Engagement with certain kinds of cognitive tasks appears negatively valenced, indicating a subjective cost. Subjectively inflated effort costs might undermine cognitive engagement and thereby performance. Control-Demanding Tasks Are Valenced Not all tasks are effortful. Tasks requiring allocation of working memory for cognitive control, however, appear to be (Botvinick et al., 2009; Dixon and Christoff, 2012; Dreisbach and Fischer, 2012; Kool et al., 2010; Massar et al., 2015; McGuire and Botvinick, 2010; Schouppe et al., 2014; Westbrook et al., 2013). Individuals allowed to select freely between tasks differing only in the frequency with which working memory must be reallocated for cognitive control express a progressive preference for the option with lower reallocation demands (Kool et al., 2010; McGuire and Botvinick, 2010). Critically even when offered larger reward, decision-makers discount reward as a function of effort costs, thus selecting smaller reward with lower demands over larger reward with higher demands (Massar et al., 2015; Westbrook et al., 2013). Under what conditions might cognitively demanding tasks acquire affective valence? By one account, tasks demanding cognitive control involve response conflict (Botvinick et al., 2001) or frequent errors (Brown and Braver, 2005; Holroyd and Coles, 2002) and as such are less likely to be successful, thus engendering avoidance learning to bias behavior toward tasks with higher chances of success (Botvinick, 2007). Multiple lines of evidence suggest that conflict is aversive. First, conflict in the context of a Stroop task predicts overt avoidance (Schouppe et al., 2012). Also, trial-wise variation in subjective frustration with a stop-signal task predicts BOLD signal in the anterior cingulate cortex (ACC), otherwise implicated in conflict detection (Spunt et al., 2012). In another study (McGuire and Botvinick, 2010), participant ratings of their desire to avoid a conflictinducing task correlated positively with individual differences in recruitment of ACC and also dorsolateral PFC, putatively involved in working memory maintenance of task sets. Moreover, the dorsolateral PFC correlation remained after controlling for performance differences (reaction time, RTs, and error rates), indicating that the desire to avoid the task did not simply reflect perceived failure. Finally, interesting interactions between affect and cognitive control also support the notion that conflict is aversive (Dreisbach and Goschke, 2004; Saunders and Inzlicht,

696 Neuron 89, February 17, 2016 ª2016 Elsevier Inc.

2015; Shackman et al., 2011). For example, individuals respond faster to affectively negative, and slower to affectively positive stimuli, following priming by conflicting versus non-conflicting Stroop trials (Dreisbach and Fischer, 2012). Avoidance learning to minimize loss may partly explain aversion to working memory allocation for cognitive control. Yet, it cannot be the full story. On the one hand, individuals avoid cognitive demand, even controlling for reward likelihood (Kool et al., 2010; McGuire and Botvinick, 2010; Westbrook et al., 2013). On the other, opportunity costs may reflect more than just the likelihood of failure during the current control episode; namely, they may reflect the value of missed opportunities (Kurzban et al., 2013). Finally, an adaptive system must also be judicious, and avoidance of all goals requiring cognitive control is clearly maladaptive. Decision-making must consider both costs and benefits. Indeed, there is growing evidence that the ACC is as important for biasing engagement with effortful, controldemanding tasks as it is for biasing avoidance (Shenhav et al., 2013). Incentives Motivate Cognitive Control If control is avoided because of subjective costs, increased incentives could offset costs, promoting control. Indeed, incentives yield control-mediated performance enhancements (see Botvinick and Braver, 2015; Pessoa and Engelmann, 2010 for review). Incentives enhance performance in control-demanding tasks encompassing visuospatial attention (Krebs et al., 2012; Small et al., 2005), task-switching (Aarts et al., 2010), working memory (Jimura et al., 2010), and context maintenance (Chiew and Braver, 2014; Locke and Braver, 2008), among others. Furthermore, incentives predict greater activity in controlrelated regions, including medial and lateral PFC. For example, incentives yield increased BOLD signal in the ACC, propagating to dorsolateral PFC, corresponding well with the canonical model by which the ACC monitors for control demands and recruits lateral PFC to implement control (Kouneiher et al., 2009). This particular study showed that incentives yielded an additive increase in BOLD signal, on top of demand-driven control signals. However, more recent work has shown that incentive information is not merely additive, but interactive: with increasing incentive-related activity under high task-demand conditions, thus more directly implicating incentives in the enhancement of cognitive control (Bahlmann et al., 2015), cf. Krebs et al. (2012). Beyond mean activity, incentives also enhance the fidelity of working memory representations. Task set representations are more distinctive, as revealed by multivariate pattern analysis of BOLD data, during incentivized working memory trials (Etzel et al., 2015). Interestingly, increased distinctiveness predicts individual differences in incentive-driven behavioral enhancement. Incentives not only drive more control-related activity, or higher fidelity task set representations, but they also affect the selection of more costly control strategies. For example, cognitive control may be recruited proactively, in advance of imperative events, or reactively, concurrent with event onset (Braver, 2012). Proactive control has behavioral advantages, but also incurs opportunity costs that bias reliance on reactive control. Incentives appear to offset costs, increasing proactive relative to reactive control, as reflected in sustained increases

Neuron

Review Figure 1. Incentive State Dynamics during a Control Episode State dynamics as exemplified by succession through mental multiplication task operations. In the image, points incentivize initial engagement. The costs (red line) mount with time-on-task and increasing maintenance and updating demands. The actors persist while the net incentive value of engagement (black line) remains positive, which occurs when costs are offset by incremental progress (e.g., at subgoal completion) and other incentives (green line). If the net incentive value goes negative, the actors are prone to disengagement.

in BOLD signal prior to imperative events, and attenuated phasic responses at event onsets, and this shift to proactive control predicts performance enhancements (see Jimura et al., 2010). Moreover, incentive-driven shifts to proactive control are larger among highly reward-sensitive individuals (Jimura et al., 2010). In sum, working memory operations are treated as subjectively costly. Whether apparent costliness reflects avoidance learning of behaviors with low likelihood of success, or opportunity costs, incentives can counterbalance costs, promoting working memory operations. Cost-benefit decision-making thus underlies working memory allocation for cognitive control. We propose that during goal pursuit, individuals engage in costly control episodes, remaining engaged to the extent that benefits outweigh costs. Moreover, we propose that DA solves a core computational problem of control episodes: namely, valuebased management of working memory for cognitive control that reflects not only prior reward learning, but also instantaneous effects of current incentive state. To illustrate, we consider an example control episode involving the demanding task of finding the product of two two-digit numbers, incentivized by points on an examination (without calculators; Figure 1). Control episodes may be initiated by incentive-driven (point-value cued) allocation of working memory to represent the goal state (finding the product). Throughout an episode, the actor must maintain high-level goal information (e.g., the original numbers), resisting interference from distractors, while flexibly updating targeted, lowerlevel representations of subgoals in a hierarchical fashion. Subgoals in our example include: (1) multiplying the ones column digits; (2) carrying the tens-digit value of that product; (3) adding that value to the product of the tens-digits, etc. Maintaining each subgoal is subjectively costly and thus the stability of goal representations should reflect the value of those goals. Similarly, updating operations, as required when subgoals are completed, are also subjectively costly. As each stage has its own costs,

and costs may accumulate in excess of perceived benefits, any stage may result in disengagement. We consider the mental multiplication example for illustrative purposes only; the general notion of a control episode should apply broadly to any hierarchically structured, temporally extended sequence of goal-directed behaviors that require working memory allocation (e.g., planning, problem-solving, and reasoning). In the sections that follow, we describe how DA mediates value-based working memory management during control episodes. Figure 2 provides an overview of critical functions that will be reviewed. Tonic DA, for example, influences the stability of working memory contents by direct action in PFC (Figure 2B), while phasic DA efflux in the striatum trains policies for value-based updating of working memory contents that reflect both the reward value of the goals to which they correspond and effort (updating and maintenance) costs (Figure 2C). While cached value-functions reflect past experience, their implementation is subject to instantaneous modulation by incentive state. Accordingly, we describe how DA and its projection targets encode net incentive state, dynamically accounting for goal state revaluation and generalized motivation. Such information is used to bias policies for working memory allocation actions (Figure 2D). Hence, DA does double duty in translating incentive information into cognitive effort both by functional modulation of working memory circuits (Figures 2B and 2C) and by influencing value-learning and decision-making about effortful action (Figures 2C and 2D). We take up each of these key duties in turn. DA and Working Memory Management Successful control episodes demand stable maintenance and also targeted, flexible updating of working memory, with DA appearing to play an important role in both processes. In the PFC, DA influences the stability of recurrent networks (Brunel and Wang, 2001; Seamans and Yang, 2004) and, thereby, the stability of short-term configurations that constitute control-related working memory representations (Cools and D’Esposito, 2011; Robbins and Arnsten, 2009). In the striatum, DA trains gating policies that come to determine the kinds of information that become represented in the PFC and the stimulus signals that drive updating of specific PFC subregions (Frank et al., 2001;

Neuron 89, February 17, 2016 ª2016 Elsevier Inc. 697

Neuron

Review Figure 2. Dopamine Does Double Duty during Control Episodes Double duty for DA in cognitive effort includes: (1) modulating the functionality of WM circuits including maintenance stability and specific flexibility for updating WM contents (yellow), and (2) developing and biasing value-based policies for WM allocation (blue). The images clockwise from upper left: (A) key anatomical loci of DA circuitry regulating control episodes; (B) tonic DA promotes stable and robust WM maintenance via PFC modulation; (C) phasic DA release encoding effortdiscounted reward trains allocation policies in striatum and ACC; and (D) phasic DA release and ramping tone in the striatum bias action selection toward costly WM updating in the lateral PFC, by potentiating updating generally, and updating in accordance with PFC-based action policy signals, in particular. The top-down policy signals reflect hierarchically higher-level goals and thus favor gating of contextually appropriate subgoals into WM. The insets are described in subsequent figures.

O’Reilly and Frank, 2006). Thereby, DA plays key roles in initiating and sustaining control episodes by functionally promoting both working memory stability and targeted flexibility. Promoting Stability of Higher-Order Goal Representations Working memory representations in the PFC (Miller and Cohen, 2001) (though see Riggall and Postle, 2012) are instantiated as temporarily stable, recurrent cortical pyramidal networks (Brunel and Wang, 2001). Extracellular DA promotes recurrent dynamics by increasing excitatory N-methyl-D-aspartate (NMDA) drive and also pruning firing external to such networks by exciting inhibitory gamma-Aminobutyric acid (GABA) interneurons (Berridge and Arnsten, 2013; Cools and D’Esposito, 2011; Seamans and Yang, 2004). The net effect of increasing DA (to a point) is to increase network-specific recurrent firing rates (Figure 3) and thus signal-to-noise ratio of working memory representations (Brunel and Wang, 2001). Evidence includes that, DA D1 receptor agonism sharpens spatial tuning in task-relevant PFC neurons in monkeys performing a spatial working memory task (Vijayraghavan et al., 2007).

698 Neuron 89, February 17, 2016 ª2016 Elsevier Inc.

Importantly, PFC DA changes dynamically, precisely when needed, to promote working memory maintenance. Salient, cognitive task-relevant events have been shown to drive mesocort...