Overselling the Science of Social Tipping Points
What was published was great. What was publicized was wrong.
I was browsing Reddit the other day and came across a rather scintillating claim in r/science: “A new study finds that when 25 percent of people in a group adopt a new social norm, it creates a tipping point where the entire group follows suit.”
This is just the sort of thing that’s right up my alley, so I decided to take a read. The link led me to a press release, so I was immediately a smidge cautious. To be clear, press releases of major academic works aren’t uncommon nor necessarily bad things. I think it’s important that research be broadcast and publicized by the institutions that house the researchers; they often have the bigger megaphone. It’s just that press-releases are notorious for dramatizing and over-extending the results of the study in order to make them seem more sexy or profound then they actually are.
And, unfortunately, this study proved to be no exception.
One of the first things you see on the page is a link to a YouTube video titled “How Many People Does it Take to Start a Revolution” with the 25% figure from the title emphasized smack-dab in the middle.
When you read the press release, the first three paragraphs are making some very large statements.
The release links the findings to “large-scale social change” like “gay marriage, gun laws, or race and gender equality.” You know, the few kinds of attitudes that people are more likely to have long-standing and stable opinions on because they tend to be well practiced and intersect with deep-rooted moral and psychological proclivities.
If this is true, then it would only take a relatively small number of people to cause a total shift in what the public believes. If true, this could be huge in terms of both science and society. Especially given the reports of sophisticated online operations spearheaded by the Russian government in elections around the globe — including the United States.
If true.
But it’s not.
“The press release then?…It, in short, was wrong.”
Well, let me walk that back a bit: It’s true that the Russian government is in the business of meddling in foreign elections. It’s true that there was a paper — a good paper, no less — published in one of the premier scientific journals in the world on the topic of contagion effects in expressed beliefs. It’s true that this paper found that having only 25% of network members avidly asserting a position was enough to cascade into the whole network asserting said position.
So if all that is true, where’s the falsehood. It’s a subtle but important one: It’s false that the study said anything about the kinds of attitudes that are highlighted in the press release. The study didn’t talk about gay marriage. It didn’t talk about racial or gender equality. It didn’t talk about gun laws.
None of it.
“None of them are structured in ways similar to how the experiment was set up. And that is apparent when you dig a bit deeper into these interactions.”
Let’s step back and look at what the study actually reported, then. As a graduate student at a top research institution, I’m fortunate in that I have institutional access to a cornucopia of academic journals, Science being one of them. So I’m in the privileged position that I can actually read what is being done and how well they cohere with what was reported.
The thrust of the paper comes in two parts: An experiment and a set of computational simulations. The experiment created 10 separate online communities of no fewer than 20 players. The players were given a picture of a face and presented with a name. Trials were ran until it was confirmed that all players associated that face and the name assigned. They were then randomly assigned into groups of two and told that they would receive 50 cents if they submitted the same name for the face — the same name that they had just learned. If one of them wrote a different name, both players received nothing. After they submitted the name, they could see what their partner wrote but not what the other 18 players had. This repeated over a series of rounds as participants tried to rack up as much cash as possible.
Here’s the catch: All of the communities had a number of confederates who would start writing a different name. In half of the communities, there were at least five of them constituting no less than 25% of the group. In the other half, there were fewer than five. All of these confederates would write the same name and would do so consistently. In the conditions with five or more players, given enough interactions, the group eventually converged on the answer offered by the confederates.
The second feature, the computer simulations, demonstrated that this effect was incredibly robust when varying a variety of different pertinent features about the experiment: The number of players, the “memory” length of each player, the number of rounds, etc. These models too converged on the 25% figure.
The paper is really good. I just want to emphasize that. I think the experimental set-up is interesting. I think that the broader topic that they’re addressing is really worth looking at from both an academic and social standpoint. And I love the usage of multiple modalities to hone in on the same question (e.g., using the computational simulations and experiments while still coming at the same results). I think their finding that threshold effects can emerge with such a small minority consistently pushing something in certain circumstances is a important contribution to our knowledge. I am not doubting the veracity of what they found.
But that’s the thing: In certain circumstances. The findings can only be extended to instances that have similar structural features. And all of the domains that are included in the press release are joined by the fact that none of them are structured in ways similar to how the experiment was set up. And that is apparent when you dig a bit deeper into these interactions.
First off, this was a series of two-player “mini-games” — population convergence wasn’t measured by a vote from the whole crowd but by the proportion submitting name “X” by round “Y.” Indeed, people probably interacted with the majority of others in the network, but they never did more than one person at a time. Facebook groups, Twitter conversations, Reddit threads — they all are characterized by having several people engaged in a conversation in concert. Sometimes it’s a disharmonious, cacophonous concert — but it’s a concert nonetheless. If 25% of your Facebook feed started insisting that Cousin Alice was actually named Jane, the other 75% of your friends would probably start insisting that they get their heads checked.
Second, it’s really really important that we consider the incentives offered to the participants. Remember there were only two options: Either you concurred with your partner and you scored cash or you didn’t and, well, didn’t. You only had two options here — and the money only came with conformity not correctness. We often have more than just two options presented to us in a conversation about our beliefs. We can be persuaded, hold our ground, sincerely moderate our position, or feign doing so to be polite — and that’s just a few options; we’re already at more than two. Further, there isn’t necessarily a reward for coming to a consensus on difficult topics. Rewards can come from reaffirming one’s own beliefs or from signalling your group-identity by virtue of you espousing its favorite positions. Heck, if you agree with the wrong kind of person your liable to face issues from your in-group (which, due to the anonymity of the subjects, was a wrinkle that wasn’t even included in the experimental design).
Finally, and perhaps most obviously, respondents weren’t talking about issues like gay marriage, gun control, or social equality. They were talking about an arbitrarily assigned name to a face they had never before seen. The amount that they cared of whether they got it right or wrong is much, much lower than the amount that they would care if they were talking about the kinds of topics that they are far more invested in. Further, the zealousness of the confederates could have just as well convinced them that they had initially misremembered the name. Especially if you consider how many dyads (groups of two people) would include a confederate if they were 25% of the population. Assuming that confederates were not assigned to each other (something that is neither confirmed nor denied in the research or its supplementary materials — this is admittedly an assumption of mine), then a full 50% of groups would have a confederate.
If you were told that you needed to agree with strangers for money, you didn’t know much nor were too deeply invested in what you were agreeing with, that you only got money for agreement and not converging on the “correct” answer, and half of the time you were paired with something they were consistently giving you a different answer, wouldn’t you start singing a different tune? According to the experiment and accompanying computer models, yes — if given enough time.
That’s important! That’s a helluva finding. There are lots of times where people are talking about things that they don’t otherwise care about and breaking into groups of two to discuss it. I could totally see this happening in a corporate brainstorming session if people were allowed to coordinate beforehand, as an example. But I can’t see this changing the nation’s minds about salient social issues. Not just because previous research has pointed to the importance of cohort replacement (e.g, people with old beliefs dying out to be replaced by those with new beliefs) but because these discussions often happen in groups with far foggier incentives than mere agreement.
The press release then? The thing that I saw on Reddit? The thing that tens of thousands of people saw and potentially incorporated into their schema of how social dynamics work? It was overselling the work far beyond what the data could assert.
It, in short, was wrong.
I get it. I do. Shoot, considering that I’m in a far more precarious professional position as a graduate student I really get it. There is a huge temptation to make your results look as good as they possibly can. Besides the prestige, the increased attention can lead to increased citations which are often critical for tenure and promotions. In the case of someone like me, it can very well make or break whether or not you’ll get one of the few, vanishing, and intensely coveted tenure track jobs in the academy. But when we gussy-up our findings beyond what the data can actually honestly assert, we’re doing a disservice to society.
Those inducted into the academic way of things can pierce through the jargon, dry language, and complicated mathematics and better understand the validity and veracity of the claims. Everyone else (read: roughly 80–90% of the country) have to take us and our summaries at face value.
I don’t know how much say the authors had over the content of the press release. It’s not too uncommon for those things to be entirely out of their hands. But it’s also not uncommon for researchers to get more say. I’m inclined to give these authors the benefit of the doubt. However, I am far less content to give a pass on the situation itself. We need to be better at sharing our research. It’s important that we make people care, absolutely, but it’s also imperative that we do so while being honest about what those findings entail. Otherwise, we can hardly stand to claim that what we are pursuing is the “Truth.”
Peter R. Licari is a PhD student in Political Science at the University of Florida specializing in American Politics, Political Behavior, and Political Methodology. The opinions expressed are his own. He can also be found on YouTube and on Twitter(@prlitics13). What little spare time remains is dedicated to long-distance running, video games with his ever-patient fiancee, and to oddly productive one-sided conversations with his cat, Asia.
