On Quantity of Information

“On Quantity of Information”

by Walter Pitts

Random remarks are traced by little boys
In wet cement; synapses in the brain
Die off; renewing uplift glyphs mountain
And valley in peneplane; the mouth rounds noise
To consonants in truisms: Thus expands law
Cankering the anoetic anonymous.
“If any love magic, he is most impious:
Him I cut off, who turn his world to straw,
Making him know Me.” So speaks the nomothete
Concealed in crystals, contracting myosin,
Imprisoning man by close-packing in his own kind.
We, therefore, exalt entropy and heat,
Fist-fight for room, trade place, momentum, spin,
Successful enough if life is undesigned.

From Nautilus

The man who asked the simplest question

“Claude Shannon answered a question that no one else was even asking.”

This is a nice little video essay on Claude Shannon; even as someone bathed in information theory day in, day out, I found it interesting. Sadly, it ends with a standard #einsteincomplex.

If you haven’t read it yet, James Gleick’s The Information is well worth reading… or at least the first three quarters is. As important as Shannon was, it’s worth remembering what Hamming had to say about him:

When you are famous it is hard to work on small problems. This is what did Shannon in. After information theory, what do you do for an encore? The great scientists often make this error. They fail to continue to plant the little acorns from which the mighty oak trees grow. They try to get the big thing right off. And that isn’t the way things go…

When you go to a new field, you have to start over as a baby. You are no longer the big mukity muk and you can start back there and you can start planting those acorns which will become the giant oaks. Shannon, I believe, ruined himself. In fact when he left Bell Labs, I said, “That’s the end of Shannon’s scientific career.” I received a lot of flak from my friends who said that Shannon was just as smart as ever. I said, “Yes, he’ll be just as smart, but that’s the end of his scientific career,” and I truly believe it was.

via kottke

Communication among animals (aka, I wasn’t droppin’ no eaves sir, honest.)

cocktail party

I have terrible hearing. I’m not hearing-impaired in any actual way, but whenever there is a lot of background noise – terrible music at a bar, the burbling of friends at a big party – I just cannot understand what people are saying even when they’re right nearby. I honestly spend most of time responding to what I guess they’re talking about. But this ability to separate what a friend is signaling from the background noise is not just a problem most of us are able to solve at “cocktail parties” but is also something that ubiquitous technology like cell phones have been developed to cope with.

A less understood problem is not how to detect and understand these signals, but how to convey them. Should you speak really loudly? Have a particularly distinctive voice? This is something that animals in the wild have to deal with all the time. Among the cacophony that is multiple species trying to chatter at each other, they have to decide how to send messages to each other that are both detectable and understandable from the background noise.

The traditional view has been that animals will act like to channels: partition the space so that they don’t interfere with each other too much. This bird over here will squawk loudly, this dove will coo softly, and so on. That would be the most informative way if each species were acting on their own. But of course there are other things to consider. Two birds may occupy the same ecological niche, worried about the same predators and needing to warn off other animals that are battling for the same food. If that was the driving evolutionary pressure, signals might end up more clustered than you’d otherwise expect.

In fact, the latter possibility is exactly what happens. Tobias et al. visited the Amazon and recorded the dense vocalizations of more than 300 animals throughout the day. Taking the principal components, they found that the three most relevant ways to describe the data are in pitch, duration, and pace of the signal. In fact, there is much more clustering than you’d expect from animals partitioning their signal. Although they are not able to test it directly, this suggests that there could be a lot of communication between different species. This interspecies communication shouldn’t be too shocking: we all understand a growl when we hear it, right?

Informationally-optimal filters for natural sounds (left) and experimentally measured cochlear filters (right)

Informationally-optimal filters for natural sounds (left) and experimentally measured cochlear filters (right)

One of the fundamental questions in neuroscience is how our sensory neurons are able to represent the world. An extremely fruitful line of research has been to study how neurons respond to natural stimuli. It makes sense, then, that sensory neurons have evolved to represent as much information as possible about the natural world – after all, why would you throw away information right away? An influential paper by Michael Lewicki proposed an answer for audition by finding the independent components of natural sounds. But no one has thought about this in an ecological context! Natural sounds have to compete – or cooperate – with vocalizations from other animals. Hopefully we will see evidence of that in the future.

References

Tobias JA, Planqué R, Cram DL, & Seddon N (2014). Species interactions and the structure of complex communication networks. Proceedings of the National Academy of Sciences of the United States of America, 111 (3), 1020-5 PMID: 24395769

M Lewicki (2002). Efficient coding of natural sounds Nature Neuroscience DOI: 10.1038/nn831

Photo from

 

Culture and human evolution

Edge has an excellent interview with Joseph Henrich on cultural and biological evolution.  He argues that the distinction between the two is fuzzy; he says they are inseparable but I think what he really means is that we don’t know how to separate them yet.  Although they are distinct concepts, they have feedback on each other which makes the separability difficult-to-impossible (though does not mean they are not distinct!).  To get an example of what he’s saying here:

Another example here is fire and cooking. Richard Wrangham, for example, has argued that fire and cooking have been important selection pressures, but what often gets overlooked in understanding fire and cooking is that they’re culturally transmitted—we’re terrible at making fires actually. We have no innate fire-making ability. But once you got this idea for cooking and making fires to be culturally transmitted, then it created a whole new selection pressure that made our stomachs smaller, our teeth smaller, our gapes or holdings of our mouth smaller, it altered the length of our intestines. It had a whole bunch of downstream effects.

We did not evolve the ability to make fire.  But once we were able to make fire, biological evolution took hold.  Cultural evolution drove biological evolution.  An important point that he makes is that culture and technology can only reach a certain level of richness in any given population level.  More complex societies require larger – or more connected – populations:

I began this investigation by looking at a case study in Tasmania. Tasmania’s an island off the coast of Southern Victoria in Australia and the archeological record is really interesting in Tasmania. Up until about 10,000 years ago, 12,000 years ago, the archeology of Tasmania looks the same as Australia. It seems to be moving along together. It’s getting a bit more complex over time, and then suddenly after 10,000 years ago, it takes a downturn. It becomes less complex.

The ability to make fire is probably lost. Bone tools are lost. Fishing is lost. Boats are probably lost. Meanwhile, things move along just fine back on the continent, so there’s this kind of divergence, and one thing nice about this experiment is that there’s good reason to believe that peoples were genetically the same.

You start out with two genetically well-intermixed peoples. Tasmania’s actually connected to mainland Australia so it’s just a peninsula. Then about 10,000 years ago, the environment changes, it gets warmer and the Bass Strait floods, so this cuts off Tasmania from the rest of Australia, and it’s at that point that they begin to have this technological downturn. You can show that this is the kind of thing you’d expect if societies are like brains in the sense that they store information as a group and that when someone learns, they’re learning from the most successful member, and that information is being passed from different communities, and the larger the population, the more different minds you have working on the problem.

If your number of minds working on the problem gets small enough, you can actually begin to lose information. There’s a steady state level of information that depends on the size of your population and the interconnectedness. It also depends on the innovativeness of your individuals, but that has a relatively small effect compared to the effect of being well interconnected and having a large population.

The analogy between brains and population level is a good one: in the brain, it is not the individual neurons that give rise to complex behavior, but the interactions between them.  The number of neurons determines the complexity of patterns that can be extracted from the environment.  A simple example in computer science is the perceptron; if you have one neuron, you can make a linear decision between two choices.  As you connect more and more neurons, you’re able to increase the complexity of the decision by adding another linear filter; eventually you can be arbitrarily complex, but at low numbers of neurons you’re going to be really limited in the number of patterns that you can decode.

But the level of complexity also has an impact on how we interact with each other:

In the Ultimatum Game, two players are allotted a sum of money, say $100, and the first player can offer a portion of this $100 to the second player who can either accept or reject. If the second player accepts, they get the amount of the money, and the first player gets the remainder. If they reject, both players get zero. Just to give you an example, suppose the money is $100, and the first player offers $10 out of the $100 to the second player. If the second player accepts, he gets the $10 and the first player gets $90. If he rejects, both players go home with zero. If you place yourself in the shoes of the second player, then you should be inclined to accept any amount of money if you just care about making money.

Now, if he offers you zero, you have the choice between zero and zero, so it’s ambiguous what you should do. But assuming it’s a positive amount, so $10, you should accept the $10, go home with $10 and let the other guy go home with $90. But in experiments with undergraduates, Western undergraduates, going back to 1982, behavioral economists find that students give about half, sometimes a little bit less than half, and people are inclined to reject offers below about 30 percent.

…I was thinking that the Machiguenga would be a good test of this, because if they also showed this willingness to reject and to make equal offers, it would really demonstrate the innateness of this finding, because they don’t have any higher level institutions, and it would be hard to make a kind of cultural argument that they were bringing something into the experiment that was causing this behavior.  I went and I did it in 1995 and 1996 there, and what I found amongst the Machiguenga was that they were completely unwilling to reject, and they thought it was silly. Why would anyone ever reject? They would almost explain the subgame perfect equilibrium, the solution that the economists use, back to me by saying, “Well, why would anybody ever reject? You lose money then.” And they made low offers, the modal offer was 15 percent instead of 50, and the mean comes out to be about 25 percent.

We found we were able to explain a lot of the variation in these offers with two variables. One was the degree of market integration. More market-integrated societies offered more, and less market integrated societies offered less. But also, there seemed to be other institutions, institutions of cooperative hunting seemed to influence offers. Societies with more cooperative institutions offered more, and these were independent effects.

This creates a puzzle because typically people think of small-scale kinds of societies, where you study hunter-gatherers and horticultural scattered across the globe (ranging from New Guinea to Siberia to Africa) as being very pro social and cooperative. This is true, but the thing is those are based on local norms for cooperation with kin and local interactions in certain kinds of circumstances. Hunter-gatherers are famous for being great at food sharing, but these norms don’t extend beyond food sharing. They certainly don’t extend to ephemeral or strangers, and to make a large-scale society run you have to shift from investing in your local kin groups and your enduring relationships to being willing to pay to be fair to a stranger.

This is something that is subtle, and what people have trouble grasping is that if you’re going to be fair to a stranger, then you’re taking money away from your family. In the case of these dictator games, in order to give 50 percent to this other unknown person, it meant you were going home with less money, and that meant your family was going to have less money, and your kids would have less money. To observe modern institutions, to not hire your brother-in-law when you get a fancy job or you get elected to an office is to hurt your family. Your brother-in-law doesn’t have a job now. He has to have whatever other job he has, a less good job.