I have written about The Wisdom of Crowds before many times (see here, and here, and here…). In short, given a problem with a fully objective, quantifiable answer, taking the average of many, diverse people’s estimates will give a more accurate answer than the estimate of an expert.
Now there’s a new study that shows that you can provide multiple estimates yourself, by putting yourself in a different frame of mind — then average them. And that average is likely to be more accurate than either of your two guesses, though not as accurate as involving another person. Neat mind hack!
…participants were given detailed directions for making their follow-up guess: “First, assume that your first estimate is off the mark. Second, think about a few reasons why that could be. Which assumptions and considerations could have been wrong? Third, what do these new considerations imply?… Fourth, based on this new perspective, make a second, alternative estimate.” When the participants used the more involved method, the average was significantly more accurate than the first estimate. The “crowd within” achieved about half the accuracy gains that would have been achieved by averaging with a second person.
28 Responses to “One is a Wise Crowd”
Sorry, the comment form is closed at this time.
So the very old saying “Look at it from different angles” comes around again. I know this is about quantitative things, but I always end up thinking about Trammel whenever you talk about group think.
Soo… welcome to my mind?
You might find the latest NPR Radio Lab episode Into the Brain of a Liar interesting.
One thing mentioned is how people can delude themselves – lie to themselves essentially – by having more than one story or idea in their head at once. Reality versus the lie for example.
The flip side though is that having more than one story in your head, especially contradictory ones, is not a bad way to “crowd” source. Of course when you let one dominate, you may find yourself in denial.
Be all the people you can be.
Works for me.
Borg!
a “study” that says “thinking” is better than “not thinking” to get to a result…
amazing.
the wisdom of the crowd gets you as many nazi members as american idol winners….
you decide if were talking about wisdom or answers….
not the same thing.
in a few decades folks i hope will look back at this stuff and either laugh or cry.
I think you may have misunderstood the research, c3.
Let’s say you were asked for the date when say, the book WISDOM OF CROWDS came out. Unless you happen to actually know, you will come out with an estimate. Let’s say 1993.
Then you stop and think about it, and realize wait, why would I have blogged so many times about this, and maybe it has to be more current given all the mentions, etc. So you revise your estimate to 2008, last year.
BOTH estimates are wrong. But (1993+2008)/2 will be closer to the correct answer of 2004 than either estimate, most of the time.
If you added the guesses of more people with varying backgrounds and expertise, by the way, the answer gets more accurate, and it has nothing to do with “thinking” per se…
no i didnt misunderstand anything 😉
on “who wants to be a millionaire” everyday some sucker goes home with 1k because he didnt learn to think for themself and assumed the “audience” had a wisdom. They found out the audience members were just guessing, just because some gamer put a buzzer button in their hands and it was “expected” that they would push the button:)
I mock “WISDOM” as having any correlation to the “NUMBERS”.
I mock the masses answers as VALID when your goal is the objective answer… they can only provide a Subjective outcome. Whos agenda is served by the edit becomes the key issue of these “memes”
its a good thing to empower the mass to fuel the subjective nature of a human society. But when it is used to set objective answers in a culture, that culture becomes doomed.
why is the intelligent act of an individual “a hack”?
why is it that the human minds abilities are now the mimic to the machines?
pulling the plug on Kelly and Shirky would be wise.;)
@cube3: well the article starts of saying that if you average the estimates of a crowd of experts you are more likely to get a good answer… simple statistics, gaussian bell-curve… Then it goes on and says that if you average an underestimate with an overestimate, then you get a better result… But then, this is exactly how people who are good at working with uncertainty do estimates… So, yeah, I am with you, the “wisdom” is a misnomer. It would be better to call it “the statistics of expert-crowds”.
If crowds possessed wisdom then they would be able to create great art. They aren’t. Only small groups are capable of that. Crowds are good at providing approximate survey-information (wikipedia, slashdot, etc), not insightful information.
Agreed. Does that make us a crowd?
The wisdom of crowds is a marketing wonk’s dream of nirvana: the effortless sell. This is why they push Twitter so hard. When Ashton Kucher is a ‘broadcast network’ in his own right; something ain’t holy although I’d only worry if his followers become an entity in their own right.
The studio audience of “Who Wants to be a Millionaire” is correct 90%-95% of the time, according to various sources. And why not? They’re a self-selected pool of individuals with an interest in quiz shows.
I find “one is a wise crowd” an interesting concept, as it encourages looking at a problem from multiple viewpoints — a valuable skill in most any field of endeavor that doesn’t involve shooting people.
But I think the more important takeaway is that no matter how adept you are at considering multiple viewpoints in your own head, accuracy is improved by asking other people. At best you can incorporate their feedback to make your answer better. At worst it just confirms your initial assessment that you’re right and everybody else is an idiot.
I think several of you need to go read the book or at least the earlier links, because you have it wrong. It is NOT crowds of experts. It is crowds of diverse groups of people. It also has nothing to do with art, Twitter or Wikipedia (as I wrote on one of the earlier posts on the topic, claiming Wikipedia as an example of the wisdom of crowds is incorrect). It has little to do with Kelly or Shirky, either, really… don’t mix up the politics of social media with the specific topic. Wisdom of crowds is a documented phenomenon of rather limited application, and the canonical example is guessing how many jelly beans are in a jar.
No diversity of opinion. I think that makes you an echochamber. 😛
I mean, yes. People use the “Wisdom of the Crowds” data wrong. Raph wrote about this time ago. (I have this idealistic friend who keeps using the term whenever she talks about doing something with multiple people and I keep facepalming.) But just because people put a different spin on it for the public doesn’t invalidate the original conclusions.
It goes like this: The Science News Cycle. Except when a book gets published, it’s a “positive” effect.
the “wisdom of the crowds” is a documented phenonmenon.. yes. its limited application is usally totalitarism or game show mechanisms when people, not jellybeans are concerned.
The current meme to see all “existance” and especially “human” interactions as a model of how the digital machine works – is the echochamber thats needs to have its walls cracked.
i dont here many questions from these type of folk, only answers. thats the act of technologists, not scientists.
It is sampling and I understand that. It has become lore and that is how a fairly mundane idea gets spun up into a full blown marketing meme designed to justify inanity. Twitter is a good example of that.
Kelly and Shirky are the current day examples of the late-bound prophet, or blogging whichever way the wind blows just before the weather report. That is why we react that way. It became all too apparent a long time ago just how easily the crowd could be manipulated using the appointed heros. That is why I say the web is a pop phenomenon. Even a hulu hoop has serious applications but that makes it nonetheless, faddish.
Most crowds are echo chambers. The fear of being left out, of being other, of being last is so pervasive in this culture that echo chambers are easy to apply. See Chartres and for the same reason. A diverse crowd gathered together to celebrate a single saint isn’t a way to decide if today is a good day to drink wine.
Pop is a harsh business.
Ok, let’s do this.
1. Diversity of opinion: is this the initial state or superposition? Can it be shown that the opinions are diverse or is polysemy playing a role for example? Do members actually care or are they for example arguing infinite regress because the argument of note is not the test at hand (the tactic of exhaustion used when the outcome of the debate is not important but who wins is). Are their opinions significant (I have tons and tons of UFO reports but I’ve never seen one)?
2. Independence of members: some what like a jury pool, all you have is self-attestation and that is easy to game, in fact, all marketing relies on gaming this. They call that ‘relevance’ these days.
3. Decentralization: Again, hard to prove and easy to game.
4. A good method for aggregating opinions: Audit deep and often.
It isn’t that sampling won’t produce valid results. It produces results that are easy to game and when the outcome has a very high value to some party, you expect the mammals to game it. The Obama election (Axelrod’s Bot) is a perfect example of how to do this then claim wisdom (the People Have Spoken). So was Bush’s and in fact most elections back to at least John Kennedy.
In limited circumstances for limited decisions, it may be true but asking any one to trust the crowd to do brain surgery, great art, engineer or otherwise perform a task that DOES rely on deep well-practiced knowledge is foolish. Pick Pepsi vs Coke: ok.
The one good test of crowdsourcing is to see how long the crowd’s opinions remain stable over time. So far, Microsoft is still the world’s best software. Don’t like it? Well, you’re in the minority.
And so it goes.
yeah,
knowing facts. and earning wisdom using facts (or not using them) are different things.
whats going on is that we have technologists being banked upon to convince the mass that they “offer” the closest “product” that can simulate the human mind and the easiest way to do that is to systematicaly promote the lowering of the human minds abilities.
This is what advertising and propagada does. it promotes un earned knowledge to replace fear. and fools you into believing it’s wisdom.
Cube3, you’re either misrepresenting or misunderstanding the method here. Your talk of Nazi membership relates to problems of conformity, which comes from decisions and discussion within the group prior to making the decision, and the perceived cost of deviating from the group norm. (eg. the Asch studies.) That is quite different from ‘the Wisdom of Crowds’ as stated in this context, which is essentially talking about many people acting completely independently and their opinions then averaged. There is no pressure to conform and so people enter their best guesses, which typically contain some information of relevance about the variable being sampled.
It’s not exactly about replacing objective answers with subjective ones, but about using subjective methods to approximate objective answers that might be hard to determine through other means.
I didnt name this meme the “The WISDOM of the Crowds”
so im not the one misrepresenting or misunderstanding anything.
Im not part of the group blogging and conferenecing away in the tech press and now “civic” press about its need to be applied.
“There is no pressure to conform ”
prove that. on that i call BS.
the method above states clearly its own limitations, it instructs humans as one would a mechanical machine. The studies assumptions were already clear.
hammers seeing nails. those with agenda using studies to prove beliefs… seen it all before.
Ill take my brain chip in designer black, please.
Well, Raph, I disagree. Everybody with basic knowledge of math is an expert of jellybean estimates. Add some idiots claiming that there is a gazillion jellybeans in the jar and your average will be blown out of of proportions… Garbage in == garbage out, even for “wise” crowds…
What I dislike is the tabloid meme of “wisdom”, because it isn’t. There is nothing new or spectacular about this. The basic principles at work has been used in engineering and scientific methodologies for decades (or centuries). And it IS related to the basic nature of how Wikipedia and Slashdot improves quality…
I don’t know. An expert on jelly beans in a jar would know how many jelly beans are in a square inch. At least, certainly, those real little yummy ones with the steroid flavor packed into them. Because we’re talking about an expert here. So this expert would have a pretty good idea, once he sizes up the size of the jar, of almost exactly how many jelly beans there are in that jar. Unless, of course, this expert is extremely near sighted and wears glasses so think you could get a panoramic view of the Jolly Green Giants arse, in which case he may guess a bit high.
Actually, it turns out not, because those jelly beans jars frequently have a balloon inflated in the middle. Seriously. 🙂 So the expert has a sort of “box” in the way they approach the problem that more diverse viewpoints and backgrounds helps compensate for.
Then that’s just cheating. Actually, I was going to bring up the fact that most people guess quite low, so even with diversity the group result should be low. So this ballon sort of would make sense out of that.
But why wouldn’t an expert know that? Or is it a case of not knowing how big the balloon is?
In days of ol’, when the Andy of Mayberry’s stalked the world, before people realized that other people would actually go out of their way to reap the rewards of Jelly Bean Counting, there were people who were true experts. And they’d win more often than not. That must be the reason for the balloons in more contemporary times, eh?
(p.s. those of you who have never experienced the joys of a variety store soda fountain may be geek grandeur, but you don’t know what you’ve missed.)
It comes down to the practical worthlessness of the idea. The core “honest” requirement is independence. The individual contributors have to be black boxes and anything that opens that box to inspection violates the independence. What Larry is describing it the Google Ploy: claim to have the best means to aggregate, then feedback the results into the system to reinforce the samples thus violating both independence and decentralization. It’s a classic Schrodinger cat.
My guess here is what Larry objects to and I echo is the social twittering of a very mundane math proccess as a means to hijack public acceptance of what would otherwise be suspect conclusions. IOW, it elevates the Simon Cowell’s of the world to power and wealth while turning the talent into cattle in the iron barred walk. You can’t cheat an honest man but you can corrupt him and the easiest way to do that is to have the crowd applaud for him on the way up and boo him on the way down. In business, decline to work with the desperate even if they are thee.
len, I completely agree that the term (which is catchy but dubious) and the actual process get co-opted constantly for cases where they do not apply and used as justification. In fact, I am pretty sure I wrote a lengthy post about exactly that…
It helps to understand the interlocking four parts to the idea and when offered it as a justification, to question each one. That may be what a Wise One does best.
Otherwise the booking agent gets you to play a gig in a room where you simply don’t belong. 😉
so weve now justified “cheating” and “guessing” to poor Opie. Aunt B would be so upset.
good to see weve learned our lessons.:)
“when the president does it , its not illegal!” ;)–see what happens when Opie grows up to become Frank Capra…lol
thanks Len- thats pretty much it.
I still think it’s entertaining to watch len and cube tag team like sports commentators. Do you guys do this anywhere else?
Actually, I wonder if you could determine exactly how important independence is to the effect. If you could decrease independence by bits and see what happens to the quality of the results over a large set of samples, what does the curve look like? Does it plummet to nothing? Does it dip and come back up? Is it a gentle drop off? Does it go down at all?
Granted, there’s the extra variable of the nature of the interdependence, but eh. It’d be fun from a game theory point of view.