National Geographic

The Wisdom of (Little) Crowds

In 1785, a French mathematician named Marie Jean Antoine Nicolas de Caritat (known as Marquis de Condorcet) used statistics to champion democracy.

Democracies are based on the collective decisions of large groups of people. But citizens aren’t experts on every topic, and so they may be prone to errors in the choices they make. And yet, Condorcet argued, it’s possible for a group of error-prone decision-makers to be surprisingly good at picking the best choice.

Condorcet’s logic was simple. Assume you have a group of people each independently making a choice about a question. Assume that they have a chance of making the wrong choice–but that their choices are better than random. If the decision they’re trying to make is either thumbs up or thumbs down, for example, their chance of picking the right answer only needs to be greater than 50 percent. The odds that a majority of them will pick the right answer is greater than the odds that any one of them will pick it on their own. What’s more, Condorcet argued that the group’s performance gets even better as its size goes up.

Condorcet’s argument is the foundation of what’s now commonly called the “wisdom of crowds.” Individuals who have imperfect understanding of a situation can band together to become good at collective decision-making.

There are some famous stories that illustrate the wisdom of crowds. Just over a century ago, Sir Francis Galton asked 787 people to guess the weight of an ox. None of them got the right answer, but, pooled together, their collective guess was almost perfect. In his book, The Wisdom of the Crowds, James Surowiecki writes about the game show “Who Wants to Be a Millionaire?” Contestants could get help answering questions either from an individual friend whom they considered an expert, or from a poll of the audience. The majority of the audience picked the right answer 91 percent of the time, while individual friends only did so 65 percent of the time.

Many scientists have used Condorcet’s idea (known as the jury theorem) as a launching pad for exploring collective decision-making. They’ve expanded the basic theory to include more features of crowds–such as the way information can move through them. They’ve tested out versions of the jury theorem on real groups of humans and animals. And their research has shown that crowds really can be wise. People can indeed make better decisions in groups than on their own. And while animals may not be able to pick presidents, they can also make good decisions in groups. It may be hard for an individual fish to recognize a predator in a murky ocean and escape in time. But a school of fish can pool its uncertain information to avoid enemies.

In fact, animals can make some remarkably sound choices among remarkably complicated options. In this feature I wrote for Smithsonian, I described the decisions that honeybee swarms make. Thomas Seeley, a Cornell biologist, has shown that a swarm of honeybees can choose among several possible locations to build a new hive. And they’re able to choose the best spot in terms of size, temperature, and other factors.

But now a leading expert on crowd decisions is starting to question some of the basic rules of the wisdom of crowds. In a new study, Iain Couzin of Princeton and his graduate student Albert Kao argue that, in most case, small groups are wiser than big ones.

This result came as a surprise to Couzin. For over a decade he’s been studying collective decision-making, combining mathematical models with sophisticated experiments on fish, insects, and other animals. (For more on Couzin’s work, check out fellow Phenom Ed Yong’s 2013 Wired feature and my 2007 profile in the New York Times.)

To develop their models for how animal swarms make decisions, Couzin and his colleagues have made certain assumptions. That’s how science always works–rather than try to replicate every facet of reality, you assume that some of them are irrelevant to the phenomenon you want to understand. But in recent years, Couzin and Kao started to question two of the most basic assumptions about collective decision-making.

The first assumption goes all the way back to Condorcet. It’s the idea that all the votes cast by a group are truly independent of one another. Each voter, in other words, makes a decision based on his or her own imperfect information about the subject. To do so, each voter has to gather information on a question on his or her own. In these conditions, the casting of each vote is like rolling a separate set of dice.

That might well be true in some cases, but Couzin and Kao could imagine many cases where it wouldn’t be. If people all gather information about a presidential candidate from different news sources, their votes will be based on independent sources of information. But if they all get their information only by watching the same show on MSNBC, their information isn’t independent. Instead, it’s what scientists describe as correlated.

A similar situation holds true for animals. If two fish are swimming on opposite sides of a school, they may have two entirely different fields of view of their surroundings. The information that one fish gets on one side is uncorrelated with the information that the other fish gets. And that means that the decisions they make based on cues in their environment are also uncorrelated. Their uncorrelated information gives them a collective wisdom that a single fish, with its limited amount of information, can’t gather.

By contrast, two fish swimming side by side see almost entirely the same scene. Their information is correlated. And that means their decisions are correlated, too. If one fish gets misled by a mirage, the other one is likely to be misled as well.

Couzin also became concerned by how wisdom-of-the-crowd experiments are set up. Scientists typically present a group of animals (or people) with a single cue they can use to make a decision. They might offer a school of fish a visual cue and see if they decide it’s a predator they have to escape.

But in the natural world, animals are swamped with information from lots of sources. Individuals on the lookout for predators may use not only their eyes, but also their ears and their noses.

This feature of real decision-making may have huge effects on how crowds perform. It’s not simply that each fish is keeping track of more than one cue. It’s also the fact that some cues may be very reliable and others may be unreliable. Some cues may be correlated, and others uncorrelated. And animals may learn to pay more attention to some cues and disregard others.

Couzin and Kao wondered how these factors could affect the wisdom of crowds. They put together a series of mathematical models that included correlation and several cues. In one model, for example, a group of animals had to choose between two options–think of two places to find food. But the cues for each choice were not equally reliable, nor were they equally correlated.

The scientists found that in these models, a group was more likely to choose the superior option than an individual. Even in these more realistic conditions, the wisdom of crowds survives.

Couzin and Kao expected that the bigger the group got, the wiser it would become. But they were surprised to find something very different. Small groups did better than individuals. But bigger groups did not do better than small groups. In fact, they did worse.  A group of 5 to 20 individuals made better decisions than an infinitely large crowd.

The problem with big groups is this: a faction of the group will follow correlated cues–in other words, the cues that look the same to many individuals. If a correlated cue is misleading, it may cause the whole faction to cast the wrong vote. Couzin and Kao found that this faction can drown out the diversity of information coming from the uncorrelated cue. And this problem only gets worse as the group gets bigger.

Small groups, Kao and Couzin found, can escape this trap. That’s because probability works differently in small groups as opposed to large ones. It’s not unheard of, for example, to roll the same number a few times in a row. But it’s really weird to do so a thousand times in a row. Likewise, in a small decision-making group, a lot of individuals may end up using uncorrelated cues–the ones that give wisdom to crowds.

Couzin and Kao’s analysis, which has just been published in the Proceedings of the Royal Society, doesn’t prove that the wisdom of big crowds is a fatally flawed idea. But it does serve as a warning that even simple factors can have a big impact on how groups make decisions. And it may help to explain how real animals form groups.

When scientists first came to appreciate how groups can make decisions, a question naturally arose: why don’t all animals live in gigantic groups? Some researchers argued that big groups had drawbacks that balanced the advantage they offered in making good decisions.

But Couzin and Kao wonder if such drawbacks don’t, in fact, exist. Perhaps animals live in smaller groups because smaller groups are better at making decisions.

Even the animals that do live in big groups may not actually be solving problems en masse. Only a small fraction of the group may actually be casting votes, while the rest follow their lead.

Couzin and Kao’s work also raises some questions about how we humans make decisions. If people are basing their decisions on the same information, they may be more prone to bad decisions in big groups. All things being equal, smaller groups might do better. And big groups might improve their choices if people avoided relying on the same sources of information.

In a sense, Couzin and Kao’s new study is an idea whose time has come. Only now is it becoming possible to study the perceptions and decisions of hundreds of animals as they respond to several cues at once. In years to come, Couzin, Kao, and their colleagues may be able to experiment on this model. And, in the process, they may finally be able to put Condorcet’s elegant idea to a natural test.

There are 15 Comments. Add Yours.

  1. Mike Lewinski
    April 22, 2014

    There’s a powerful argument for trial by jury in wisdom of the crowds. I argued this recently on a news story, where the cynic said that guilty people always choose juries because they know they can fool one. But innocent people should always choose a jury too, because the risk of false conviction in a trial by judge

  2. David Blincoe
    April 23, 2014

    Thank you for this work on correlation. I think the accuracy of the very large group breaks down at the place where w=dt fails from the original threat. Value comes from the larger metric and comes down.

  3. Gustavo Stolovitzky
    April 23, 2014

    The inference of gene-gene interactions in the cellular milieu, a problem known as Reverse Engineering (or inference) of gene regulatory networks, is of fundamental importance in systems biology. Indeed, the network of interactions between genes is a first step towards a deeper mechanistic understanding of the cascade of information within cells. In a recent work (http://www.nature.com/nmeth/journal/v9/n8/full/nmeth.2016.html), we reported that the wisdom of the (expert) crowds works well when we aggregated the predictions of gene regulatory networks resulting from dozens of teams that participated in a crowdsourced challenge (known as the DREAM challenges) to determine gene regulatory networks from gene expression data. The reason why the reconstruction of the gene-gene networks worked in aggregate, we showed, had much to thank to the different methodologies used to attempt to solve the problem. In the statistical jargon, these methodologies included regression, information theory, Bayesian networks, to name a few. In the absence of first principles, each of these methodologies brought something new to the pot: they extract different kinds of information from the data. Interestingly, when we aggregated predictions from the same statistical approach, the results were less impressive than when we aggregated predictions from across different approaches.

    In the absence of first principles to solve a research question, diversity of approaches is a necessity if we want to tap on the wisdom of the crowds for scientific predictions. Fashion in scientific approaches, which occurs in scientific research as in all human endeavors, create correlated predictions which can dampen the wisdom in the crowds.

  4. Benjamin Good
    April 24, 2014

    Fascinating, thank you. This seems like a beautiful illustration of the danger of the echo chamber effect on the internet. It also makes a lot of sense if you think about decision making from an evolutionary perspective. Diversity and periodic isolation of both idea generators and adapting organisms helps to keep the group from getting stuck in local optima.

  5. John Mashey
    April 25, 2014

    In many circumstances, small groups certainly work better than individuals, if there is interaction, not just voting.
    1) Consider a Trivia Night at a bar at a ski resort, with teams ranging from 2 to about 8. In a team of 8, 7 people might guess (wrong), but one person *knows* the answer and (usually) convinces the others.

    2) I’ve run program committees for conferences, using a “12 Angry Men” scheme. All the PC members would rate abstracts from 1 (definitely reject) to 5 (definitely accept). All those would be stuck in a big spreadsheet, sorted by average score, with each committee member’s votes shown.
    Just taking the top N results was seriously suboptimal. Almost always, there would be cases where all but one person would gives a 3 o4 4, and one a 1, so I’d ask: why? It would turn out the abstract sounded OK, most members weren’t that familiar with the work, but the one who was had convincing reasons to reject it.
    The same happened the other way, where one person knew the work well, but the abstract was not very well-written.

    3) In many years of managing engineering teams, going to board meetings, etc, it was quite clear that teams big enough for a variety of expertise, and small enough to communicate, performed well, most often because some person knew enough to avoid some really bad decisions.

    4) That seems a big different than estimation problems, where random differences average out. In these cases, the crucial factor is that someone in the group had strong information, and was allowed to present it, the group could see that.

  6. Kathy Burtscher
    April 26, 2014

    Thank you for this, it’s very thought provoking from my perspective, teaching Grade 1, and observing my students in patterns of choice making in work and play.

    In another direction, I wonder about the relationship between a virtual crowd and Cyberbullying (not a grade 1 problem, thankfully). If it is not community based, based on personal contact, then it is based on the information posted about a person, possibly from a single perspective.

  7. kevin
    April 27, 2014

    You should read great delusions and the madness of crowds that looks at how crowds can make horrible decisions that lead to bubbles and crashes

  8. monty mcintyre
    April 27, 2014

    One must be careful though. large and small groups can be prejudiced and have tunnel vision, viewing scenarios from different points of view. Go to any group meeting such as church,union function,Boy Scouts,etc. most of those attending are going to have the same thoughts and ultimate desires. Trying to change the bias is very,if not impossible,difficult. Oftentimes, emotion will trump practically and truth at the detriment of ones pocketbook

  9. Roderick T. Long
    April 28, 2014

    Aristotle said it long before Condorcet.

  10. Robin McClellan
    April 28, 2014

    While crowds can have some great outcomes, the CIA is finding that there are individuals who have great success in prediction, too. NPR’s story, “So You Think You’re Smarter Than A CIA Agent” (http://www.npr.org/blogs/parallels/2014/04/02/297839429/-so-you-think-youre-smarter-than-a-cia-agent) profiles the “The Good Judgment Project” (http://www.goodjudgmentproject.com/) that the CIA uses (along with their own intelligence!) to figure out what might happen next in places like North Korea.

  11. Hannah Sanderson
    April 30, 2014

    It seems as if it’s a matter of communication. Communication within a smaller group is generally more reliable than that of a large group where information can become muddled. What can also be considered is that there may be several, if not too many, who want to the “leader” in large groups, therefore confusing the important issues, whereas smaller groups can stay focused on the problem(s) at hand. In either case, it is easier for a smaller group to communicate within itself and learn which of them has the best information (this can fluctuate) in different situations..

  12. Lemper
    May 19, 2014

    “Condorcet’s logic was simple. Assume you have a group of people each independently making a choice about a question. Assume that they have a chance of making the wrong choice–but that their choices are better than random.”

    Hmm… this is a rather strong assumption, isn’t it?
    How is it explained that their choices can assumed to be better than random?

    • Monty Mcntyre
      May 20, 2014

      I am of the opinion that most people,both educated and not, make choices based upon some past experience. Most realize that if your head remains underwater,unassisted, you drown and this is not a random thought. Consequently,sane individuals keep their nose above water. I’m not sure many folks randomly make decisions. Seems as though this can lead to many problems. To randomly make decisions is an earmark for irresponsibility and mediocrity.
      Todays media is a prime example of how correlated information can lead many people in one direction regardless of how accurate it might be. Fox News stands as the sole conveyer of information that might make one an independent thinker. In my travels I listen to news extensively on XM radio and have taken notice that CNN conveys a lot of trivia to the listeners while FOX is actually delivering the current events. I have started referring to CNN as the trivia station and I am neither Democrat or Republican. Large groups of people listen to the liberal media and take its word as gospel completely blacklisting a notable channel as undeserving,old fashioned and out of tune to the times.Correlation is bad,independence is good. One needs only to study the French and Russian revolutions to understand why.
      Crowds, often times, cannot be trusted for tunnel vision sets in and mob mentality can take over!

      • Robin
        May 21, 2014

        The comments about Fox news really drive home that we tend to listen to news that we have a sympathetic listening for. I find NPR news “unbiased” but it probably means that it’s biases are mine. On the other hand, I find Fox news impossible to listen to. Questions that Fox news might ask someone often seem leading, inappropriate or irrelevant while the ones NPR reporters ask do not. The Fox news listener would find the opposite true.

        Much of this also relates to how ideology skews our view of things. When we are making decisions about whether to brake our car or not, there is no ideological component. When we make decisions on who should get government money, ideology is usually the main consideration (unless it’s us is getting the money!). That’s why we can use crowd wisdom to determine the weight of an ox but probably don’t want to use it to make decisions on whether or not to go to war or how to pursue a war that we are in.

        All too often I find polls that ask people whether they would do or support something that most people don’t have the facts or the background to answer. “Do you support longer prison sentences for drug offenders?” for example. Crowd wisdom may, through the electoral process, lead to longer sentences for drug offenders, but I don’t know that it leads to reduced drug use.

        • M McIntyre
          May 23, 2014

          I agree that most of us listen with view points skewered by our own bias. Enough people with the same bias and prejudices can lead large groups to do both meaningful and catastrophic events. After all, if one person can convince another that his viewpoints are valid that gives legitimacy to the first persons idea. There is power in numbers and this translates into more power exerted, but that does not mean the original idea is valid. Many have been been led astray ( Nazism, Communism, KKK,etc.) and I fear more are going to go down that same dreary path and stare into the abyss, possibly, because of bias. What is sad, there are generally small groups who know the truth but are drowned out by the many. “Never underestimate the power of stupid people in large groups”

Add Your Comments

All fields required.

Related Posts