Risk aversion

[This post is a stub that will be expanded as time goes on and I learn more, or figure out how to present the question better.]

Humans, and many animals, tend to like predictability. When things get crazy, chaotic, unpredictable – we tend to avoid those things. This is called risk aversion: preferring safe, predictable outcomes to unpredictables ones.

Take the choice between a guaranteed $1,000,000 or a 10% chance of $10,000,000 with a 90% chance of nothing at all. How many people would choose the riskier option? Very few, it turns out. We aren’t always risk-averse. When animals search for food, they tend to prefer safer areas to riskier ones until they start getting exceptionally peckish. Once starving, animals are often risk-seeking, and are willing to go to great lengths for the chance to get food.

Why are we risk-averse? There are a few reasons. First off, unpredictability means that the information we have about our environment is not as useful, and possibly downright wrong. On the other hand, it may just come from experience. Imagine that you are given the choice between two boxes, each of which will give a reward when opened, and rewards are reset when closed. One of these boxes will give you lots of rewards sometimes, and no rewards the rest of the time, while the other box will always give you a little reward. Over the long run the two boxes will give you the same amount of reward but when you start opening them up? You are likely to have a dry run from the risky box. Whenever you get no reward from a box, you feel more inclined to open the safer box. This gives you a nice little reward! So now you like this box a little better. Maybe you think it’s a good idea to peak in the risky box now? Ah, foiled again, that box sucks, better stick with the safe box that you know.

This is the basic logic behind the Reinforcement Learning model of risk-aversion as characterized by Yael Niv in 2002 (does anyone know an older reference?).

See also: Ellsberg Paradox, Prospect Theory

Deciding about deciding

In the field of decision-making, a typical laboratory experiment goes something like this: give a subject an option between two choices, let them decide, force them to do it again.  Put a novel variation on the way the decision is made and BAM, you’ve got yourself a little paper!  Mostly the decisions are something akin to choosing between a picture of a cake and a picture of a moldy cheese.  But a more realistic decision process might involve choosing whether the cake and moldy cheese are the best one can get; maybe you should look for something better!  One (might) call this a foraging decision, something that has been studied extensively in other contexts.  Let’s look at how the brain represents this decision.

The setup of the first experiment is a bit tricky.  Subjects were shown a choice of two rewards that they could choose between, or a set of other rewards that could be selected from randomly.  In the initial ‘foraging’ round, they got to decide whether to keep the two rewards, or get two new (random) ones for a small price.  This was repeated until they were satisfied with the two options, at which point they moved to a ‘decision’ round where they chose between the two rewards.  It is a bit unsurprising that subjects required a higher expected value from the ‘foraging’ option in order to choose it.  The authors call this their ‘foraging readiness’ but it would be more accurate to call it their level of risk-aversion.  It has been known for a long time that people prefer more sure options than more risky options, even if the economically rational man would have no preference.  I guess that’s a less sexy phrase, though.

The authors zeroed in on the anterior cingulate cortex (ACC).  Like pretty much everything that comes out of fMRI and cognitive studies, there is a lot of controversy about what exactly the ACC is doing (this isn’t a ding on fMRI or cognitive studies, it’s just really hard).  Here, researchers find that activity in ACC was positively correlated with the expected value of the foraging and negatively correlated with the expected value of the binary decision.  The BOLD signal in ACC was able to predict the number of times a subject would repeatedly search, as well as how the subject weighted the expected value of the foraging option.  And that last point is important!  Even though the researchers knew that the two new options would be chosen with equal probability, the subjects did not know that.  Or, they at least did not know that they could trust that information from the researchers who are notoriously unreliable in what they tell their subjects.   So the signal probably represents some measure of what their posterior probability distribution was, as well as how much they valued risky gains and losses, all convolved with the expected reward of each option.

Another recent paper looked at a visual task in monkeys and skipped the whole fMRI step, just putting electrodes directly into the dorsal region of ACC (dACC).  Monkeys were allowed to saccade between patches that would give a continual reward that decreased with time.  They then faced a real foraging decision: when do you leave a depleted patch to find a new source of reward?  Neurons in dACC seemed to increase their firing rate when the monkeys were making this decision.  The speed with which the firing rate increased was related to the travel time to a new patch (the cost of going to that patch of reward).  This increase continued until it reached a threshold related to the relative value of leaving the patch.

The authors are clear that the dACC signal itself is not sufficient for a leaving-decision; an observer would have to get information from other regions to determine what the threshold for leaving is.  But the data strongly suggests that dACC is coding the value of relative value of leaving a patch.

So what do the two studies together tell us about how ACC helps us make a decision?  The first paper tells us that ACC is representing the predicted cost of finding new options.  This calculation probably includes the predicted probability distribution of all available options, and will also include how many times (how long) someone is willing to go searching for a better option.  The second paper is in broad agreement, and claims that dACC represents the relative expected value but is an insufficient signal to tell the brain when to make that decision; it just encodes that signal.  It does however represent the maximum cost the brain is currently willing to bear to find a new option, just like the fMRI study shows.

These two papers are great together as they really show how (1) fMRI can be useful and (2) the differences in how the same question is framed in different subfields of neuroscience.

References

Neural mechanisms of foraging.  Kolling, Behrens, Mars, Rushworth.  Science (2012).  DOI: 10.1126/science.1216930

Neuronal basis of sequential foraging decisions in a patchy environment.  Hayden, Pearson, Platt.  Nature Neuroscience (2011).  DOI: 10.1038/nn.2856

Picture from