AskSpace Frappr!

Where are we? Add your pin to our AskSpace map.

The Wisdom of Crowds

Wikipedia

Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations, first published in 2004, is a book written by James Surowiecki about the aggregation of information in groups, resulting in decisions that, he argues, are often better than could have been made by any single member of the group. The book presents numerous case studies and anecdotes to illustrate its argument, and touches on several fields, primarily economics and psychology.

The opening anecdote relates Francis Galton's surprise that the crowd at a county fair accurately guessed the weight of an ox when their individual guesses were averaged (the average was closer to the ox's true weight than the estimates of most crowd members, and also closer than any of the separate estimates made by cattle experts).

The book relates to diverse collections of independently-deciding individuals, rather than crowd psychology as traditionally understood. There are parallels with statistical sampling theory—a diverse collection of independently-deciding individuals is likely to be more representative of the universe of possible outcomes, thereby producing a better prediction.

Its title is an allusion to Charles Mackay's Extraordinary Popular Delusions and the Madness of Crowds, published in 1843.

Types of Crowd Wisdom

He breaks down the advantages he sees in disorganized decisions into three main types, which he classifies as:

Cognition: Market judgement, which he argues can be much faster, more reliable, and less subject to political forces than the deliberations of experts, or expert committees

Coordination of behavior, such as optimizing the utilization of a popular restaurant, or not colliding in moving traffic flows. The book is replete with examples from experimental economics, but this section relies more on naturally occurring experiments such as pedestrians optimizing the pavement flow, or the extent of crowding in popular restaurants. He examines how common understanding within a culture allows remarkably accurate judgments about specific reactions of other members of the culture.

Cooperation: How groups of people can form networks of trust without a central system controlling their behavior or directly enforcing their compliance. This section is especially pro-free market.

Four Elements Required to Form a Wise Crowd

Not all crowds (groups) are wise. Consider, for example, mobs or crazed investors in a stock market bubble. Refer to Failures of crowd intelligence for more examples of unwise crowds. What key criteria separate wise crowds from irrational ones?

Diversity of opinion: Each person should have private information even if it's just an eccentric interpretation of the known facts.

Independence: People's opinions aren't determined by the opinions of those around them.

Decentralization: People are able to specialize and draw on local knowledge.

Aggregation: Some mechanism exists for turning private judgments into a collective decision.

Failures of crowd intelligence

Surowiecki studies situations (such as rational bubbles) in which the crowd produces very bad judgment, and argues that in these types of situations their cognition or cooperation failed because (in one way or another) the members of the crowd were too conscious of the opinions of others and began to emulate each other and conform rather than think differently. Although he gives experimental details of crowds collectively swayed by a persuasive speaker, he says that the main reason that groups of people intellectually conform is that the system for making decisions has a systematic flaw.

Surowiecki asserts that what happens when the decision making environment is not set up to accept the crowd, is that the benefits of individual judgments and private information are lost, and that the crowd can only do as well as its smartest member, rather than perform better (as he shows is otherwise possible). Detailed case histories of such failures include:

Too centralized: The Columbia shuttle disaster, which he blames on a hierarchical NASA management bureaucracy that was totally closed to the wisdom of low-level engineers.

Too divided: The U.S. Intelligence community failed to prevent the September 11, 2001 attacks partly because information held by one subdivision was not accessible by another. Surowiecki's argument is that crowds (of intelligence analysts in this case) work best when they choose for themselves what to work on and what information they need. (He cites the SARS-virus isolation as an example in which the free flow of data enabled laboratories around the world to coordinate research without a central point of control.)

Too imitative: Where choices are visible and made in sequence, an "information cascade" can form in which only the first few decision makers gain anything by contemplating the choices available: once this has happened it is more efficient for everyone else to simply copy those around them.

Is It Possible to Be Too Connected?

Surowiecki spoke on Independent Individuals and Wise Crowds, or Is It Possible to Be Too Connected?.

The question for all of us is, how can you have interaction without information cascades, without losing the independence that's such a key factor in group intelligence?

He recommends:

Keep your ties loose

Keep yourself exposed to as much information as possible

Tim O'Reilly[1] and others discuss the success of Google, wikis, blogging and Web 2.0 in the context of the wisdom of crowds.

Applications

Surowiecki is a very strong advocate of the benefits of decision markets, and regrets the failure of DARPA's controversial Policy Analysis Market (potential intellectual descendant of RAND's Delphi method and author John Brunner's Delphi pool) to get off the ground. He points to the success of public and internal corporate markets as evidence that a collection of individuals with varying points of view but the same motivation (to make a good guess) can produce an accurate aggregate prediction. According to Surowiecki the aggregate predictions have been shown to be more reliable than the output of any think tank. He advocates extensions of the existing futures markets even into areas such as terrorist activity, and prediction markets within companies.

To illustrate its thesis, he says that his publisher is able to publish a more compelling output by relying on individual authors under one-off contracts bringing book ideas to them. In this way they are able to tap into the wisdom of a much larger crowd than would be possible with an in-house writing team.

Will Hutton has argued that Surowiecki's analysis applies to value judgements as well as factual issues, with crowd decisions that "emerge of our own aggregated free will [being] astonishingly... decent". He concludes that "There's no better case for pluralism, diversity and democracy, along with a genuinely independent press." [2].

Quite a few other experimental sites have been constructed to explore the potential of building collective wisdom. One site, Opinion Republic (http://www.opinionrepublic.com) is an experiment to capture public opinion and then converge on the most broadly accepted opinions.

Prediction markets

Wikipedia

Also known as information markets, decision markets, idea futures, and virtual markets, prediction markets are speculative (i.e., betting) markets created for the purpose of making predictions. Assets are created whose final cash value is tied to a particular event (e.g., will the next US president be a Republican) or parameter (e.g., total sales next quarter). The current market prices can then be interpreted as predictions of the probability of the event or the expected value of the parameter.

... [CLB: Lots of information on, examples of, and links to prediction markets.]

ON AMAZON'S MECHANICAL TURK: DEFINING WORK 2.0

*michael parekh on IT

WORK 2.0: MICROCHUNKING WORK 1.0

Who'd have thunk? That potentially the next best use of the Internet since Google (via Bill Gross's GoTo.com/Overture efforts) could come from, not a bright-eyed, bushy-tailed, 'web two'ed' start-up at 285 Hamilton Ave. in Palo Alto (via Om Malik), but from a scrappy web services team at 'Web 1.0' Amazon.com?

At the admittedly very real risk of getting ahead of myself, this week's announcement of Amazon's Mechanical Turk (called AMT for the remainder of this post) may mark the beginning of a whole new Internet gold rush, not seen since the Search 2.0 gold rush since Google's IPO. AMT could be the shot that inspires the launch of a whole new stream of start-ups the world over, especially since this is about re-configuring outsourcing as we know it.

What is AMT? As the announcement dryly and geekily describes it,

Amazon Mechanical Turk provides a web services API for computers to integrate Artificial Artificial Intelligence directly into their processing by making requests of humans.

Developers use the Amazon Mechanical Turk web services API to submit tasks to the Amazon Mechanical Turk web site, approve completed tasks, and incorporate the answers into their software applications.

To the application, the transaction looks very much like any remote procedure call - the application sends the request, and the service returns the results. In reality, a network of humans fuels this Artificial Artificial Intelligence by coming to the web site, searching for and completing tasks, and receiving payment for their work.

Here's what a typical work request looks like, for Amazon's A9 search engine. TechCrunch explains AMT in simpler terms:

The “machine” is a web service that Amazon is calling “artificial artificial intelligence.” If you need a process completed that only humans can do given current technology (judgment calls, text drafting or editing, etc.), you can simply make a request to the service to complete the process. The machine will then complete the task with volunteers, and return the results to your software.

Volunteers are paid different amounts for each task, and money earned is deposited into their Amazon accounts. Amazon keeps a 10% margin on what the requester pays.

The Future of Media

A VC, by fred wilson

"I have seen the future of media and it looks like this:

Mashed Up Blog Posts at tech.memeorandum

Mashed Up Funny Videos at delicious

Mashed Up Playlists at webjay Here is the future of media:

1. Microchunk it - Reduce the content to its simplest form. Thanks Umair.
2. Free it - Put it out there without walls around it or strings on it. Thanks Stewart.
3. Syndicate it - Let anyone take it and run with it. Thanks Dave.
4. Monetize it - Put the monetization and tracking systems into the microchunk. Thanks Feedburner.

Leaving aside the rights issues, which I know are large, if I were a television executive right now, I'd take my content, microchunk it, put a couple calls to a video ad server in the middle of it, and let it go whereever it wants to go, safe in the knowledge that whenever the show is viewed, I'll get to run a couple 15 second spots in the middle of it (which I could change whenever I wanted to and which I could measure).

This is where media is going and its not going to be stopped.

[...]

What Microchunking Doesn't Mean

A VC, by fred wilson

"Microchunking means taking an object and reducing it to its smallest usable part.

It doesn't mean reducing it to something smaller than is usable.

One of my biggest pet peeves is the magazine industry's habit of taking an online story and breaking it into multiple pages. It forces you to click at the bottom of the page to get to see the next page, thereby generating more pageviews.

Whenever I link to a magazine story, I click on the 'print format' link to get a full page of the story and link to that.

That's as microchunk as I want it.

The Crowd Knows Best

CBS News | January 12, 2006

Who knows best: The bookie or a random bunch of bettors? Plugged-in Wall Street hotshots or people like you and I?

The answer, CBS News correspondent Serena Altschul says, may surprise you.

'Groups of people can be remarkably smart. And the really strange thing is they can actually be smarter than the smartest person within them,' believes author and New Yorker business columnist James Surowiecki.

Surowiecki, is not only a crowd pleaser with a best-selling book, he's a crowd champion. All that elbowing and jostling, he says, actually is a good thing.

"If you figure out a way to tap into the knowledge that say, a large group of have; you could really, dramatically improve like, the decisions you make, predictions you make. You can actually, in some ways, even forecast the future," he says.

Surowiecki admits that this logic diminishes the role of experts and authoritative figures, adding that, "it also goes against everything we've been taught about what groups are like."

And Surowiecki is not the first person to recognize this counterintuitive phenomenon. At the turn of last century, a British scientist named Francis Galton came across a crowd of fairgoers guessing the weight of oxen. He collected the guesses and averaged them out. The result, Surowiecki says was that the group guessed the ox weighed 1,197 pounds, one pound short of the actual weight.

This precision, Surowiecki says, is no fluke, rather, it is a trait groups exercise again and again.

On the game show "Who Wants To Be Millionaire," it happens almost every day. Contestants can choose advice from a lifeline expert or poll the audience

"The experts do pretty well -- they get the answer right about two-thirds of the time," Surowiecki says, "but the audience gets the answer right 91 percent of the time."

And there's nary a furlong between horse gamblers and stock traders.

What you're essentially doing when you are buying a stock, whether you know it or not, is offering up your judgment as to how much that stock is really worth. It is very hard for even the smartest money managers to do better than the stock market as a whole.

Which is why index funds, which hold a broad range of stocks with few changes, consistently beat managed funds.

John Bogle agrees and he should know. He's the legendary founder of The Vanguard Group, the second-biggest mutual fund in the world.

"In a given year, the index fund will beat 60 percent of managed funds and over decade, 89 percent," Bogle says.

"In the security markets we have no Lake Woebegone where all kids are above average. We are average in this business of professional investing and that is why index funds win because they give you market return at zero versus managerial guys who charge 20-25 percent," Bogle says.

What about when the crowd gets it completely wrong? For instance, the bursting of the prosperous stock market bubble in the late 1990s.

"Crowds go wrong when diversity really breaks down," Surowiecki says.

Lack of diversity, Surowiecki explains occurs when independent thought within the crowd ceases. He adds that, "crowds also break down when people start paying too much attention to what those around them are doing."

Surowiecki uses Amazon.com as an example. "Most of us weren't really saying to ourselves, 'What is Amazon.com really worth as a company? How much money is it gonna make over the next 20 years?'

"But what we were really saying to ourselves was, 'What does everybody else think Amazon.com is worth?' and, 'What is everyone else gonna be willing to pay for, for this stock if I buy it now? So, when I try to sell this a month from now.' And so, what happens is you get just wrapped in this loop," Surowiecki says.

"It's just this kind of vicious cycle, where everyone is really just looking at what everybody else is doing," he adds.

Surowiecki offers a second example, this time from the animal kingdom.

"Most of the time, says Surowiecki, it works for certain kinds of ants to follows the ants in front. That is, until one ant gets lost. Then it's curtains for everyone.

"And what happens is they eventually end up in this giant circle. And what happens is they will literally walk around and around, for hours, or maybe even days, until eventually, they get tired, and just die," he says.

Pressed to describe the ideal people to lead a crowd, Surowiecki says, "If we're trying to make a good decision, or predict the future, the knowledge we need is buried in the heads of people who you would never think to ask."

NewsFutures is an Internet prediction market where serious bettors try to predict the future. Using play money (because in the United States it is still illegal to bet real money on the Internet) bettors can gamble on just about anything.

"This is more than just sport and games. There's real information to be gathered from these markets," says Emile Servan-Schreiber, NewsFutures' chief executive officer. He adds of his site, "It taps directly into the collective intelligence of the audience."

Servan-Schreiber believes that "The one thing that is key is that you can trust people. And the Internet has showed that every time you put a community together through the Internet, it generates great, new possibilities for all of the rest of us."

Prediction markets are becoming so significant that Harvard Business School Professor Anita Elberse not only teaches the subject, she has made it her business.

Elberse looks to a Web site called the Hollywood Stock Exchange, which allows users to buy and sell motion pictures as if they were publicly traded stocks. The prices fluctuate depending on the success of the films at the box office.

In turn, Elberse says movie executives pay attention to the activity on the virtual stock exchange as a guide to how they should spend their studio's money.

The deference toward the gut feelings and opinions of the masses leads Elberse to conclude, "One expert will never know as much as a group of people."

So there you have it: crowd wisdom. That experts often know less than they think. Amateurs, together, know more than any one person knows, and that people can be trusted to find the truth.

It's an old lesson that Surowiecki is sharing anew.

"If you go back to Machiavelli and "The Prince," what does he say? He says, 'Do not surround yourself with yes men,' and that this is what the Prince has to watch out for.

"But," Surowiecki adds, "apparently what it comes down to, it's just very hard to follow that good advice."

Link: The Wisdom of Crowds

Wikipedia and the Wisdom of the Crowds

Knowledge@Wharton Can Wikipedia Survive Its Own Success?

It's not easy being Wikipedia, a free web encyclopedia created and edited by anonymous contributors. Just ask founder Jimmy Wales, who has seen his creation come under fire in just a few short months as the site fends off vandalism and charges of inaccurate entries. 'Wikipedia has always been in a state of change,' says Wales, in defense of his product.

That's putting it mildly"

On November 29, journalist John Seigenthaler, Sr., once a member of Robert Kennedy's staff, penned an op-ed piece in USA Today noting that an article on Wikipedia had incorrectly linked him to the assassination of Robert Kennedy and John F. Kennedy. The article, which stayed on the site for four months, stated that "John Seigenthaler, Sr. was the assistant to Attorney General Robert Kennedy in the early 1960s. For a brief time, he was thought to have been directly involved in the Kennedy assassinations of both John and his brother Bobby. Nothing was ever proven." Wikipedia eventually deleted the inaccurate information, and now even contains an entry entitled, "John Seigenthaler, Sr. Wikipedia biography controversy" explaining the history of its own error.

And last month, former MTV VJ and podcasting pioneer Adam Curry was accused of editing out Wikipedia's references to Kevin Marks, another early podcasting luminary. In a December 2 blog entry, Curry owned up to the switch and apologized.

Today, popular or potentially controversial Wikipedia entries, such as one on George W. Bush, greet readers with the following: "In response to recent vandalism, editing of this page by new or anonymous users has been temporarily disabled. Please discuss possible changes or request unprotection."

Wikipedia, founded in 2001 as a non-profit organization and supported mainly by donations, allows anyone with Internet access to edit its articles. The premise is that collective knowledge, which some call "open source" content, is every bit as valuable as professionally edited content. As a result, Wikipedia has become a hybrid encyclopedia/current events site that raises a number of interesting questions, including: Just how accurate is free content, given recent events at Wikipedia? Does the aggregate 'wisdom of the crowd' trump the expertise of knowledgeable individuals? Does Wikipedia's policing mechanism work? Does the controversy over Wikipedia merely reflect further tension between old and new media?

The answer to all those questions, according to experts at Wharton, boils down to two words: It depends. According to Eric Clemons, professor of operations and information management, it is foolish to take Wikipedia's definitions as gospel, but the site is still worth reading from time to time. Marketing professor Peter Fader notes that Wikipedia shows there is wisdom in crowds, but a better user rating system would filter out those who post bogus information. Joel Waldfogel, professor of business and public policy, agrees that much of the concern about Wikipedia is just a new spin on whether old media (printed encyclopedias, in this case) stand a chance against the new breed of instantly updatable online media.

For Kendall Whitehouse, senior director of information technology at Wharton, the most interesting question is "whether the wisdom of the crowd is ultimately a better approach compared to scholarly review and edited content." As Whitehouse observes, "Wikipedia's strength is that it has thousands of eyes looking at it. The hope is that errors will be quickly caught and corrected." But Whitehouse also points out that as the scale of Wikipedia's content expands even a thousand eyes can miss certain details -- which is apparently what happened in the case of the Seigenthaler material.

Wales doesn't dismiss concerns about Wikipedia and notes that he has implemented changes to prevent abuse, such as only allowing registered users to make certain types of changes. He adds, however, that Wikipedia has been slammed for mistakes simply because it's relatively new, and he compares the attention swirling around the site to stories that focused on quirky items for sale on eBay in 1999 and 2000. "The media loves a story," he says.

Growing Pains

Just the fact that Wikipedia has attracted so much attention indicates the clout it has achieved since being created almost five years ago. Indeed, a December 14 study of 42 entries by Nature magazine puts Wikipedia almost on a par with Britannica in terms of accurate science coverage. Nature found that the average science entry in Wikipedia had four errors while Britannica had three. The biggest complaint against Wikipedia was that the entries were confusing and poorly structured. Overall, concluded the Nature study, problems such as the Seigenthaler incident were the exception, not the rule.

Wikipedia has also grown at a rapid clip. There are 13,000 active contributors working on 1.8 million articles in more than 100 languages. On the English-language sites, 800 to 1,000 editors edit most of the entries, says Wales. According to Alexa, a web traffic tracking site, Wikipedia is the 37th most highly visited site on the Internet.

But with its clout comes additional scrutiny, says Fader, who views the most recent flaps about Seigenthaler, Curry and suggested policy changes as "tempests in a teapot. Wikipedia is as credible as anything else I find on the web. It comes up a lot ... but it doesn't mean that I stop my research there."

Indeed, the key may be putting Wikipedia into a broader context. It can be a research tool, but it's far from definitive. Wikipedia, itself, acknowledges this fact, noting on its site that "it differs from a paper-based reference source in some important ways. In particular, mature articles tend to be more comprehensive and balanced, while other (often fledgling) articles may still contain significant misinformation, unencyclopaedic content or vandalism. Users need to be aware of this in order to obtain valid information and avoid misinformation which has been recently added and not yet removed."

Despite that disclaimer, observers say Wikipedia is too often viewed as authoritative. David Winer, a software pioneer who contributed to several common standards used on the Internet and published one of the first weblogs, said in a June blog entry that the biggest issue with Wikipedia is that it is widely viewed as authoritative but "can easily be manipulated by people with an axe to grind."

Ari Friedman, a senior at Wharton who founded an online journal called Copyright that plans to adopt some of Wikipedia's techniques to vet copyright research and case studies, says it would be foolish to dismiss Wikipedia's open-content approach. But, he adds, there are trade-offs with disseminating information via the crowd approach. "The primary trade-off is not really 'speed for accuracy' as some might suggest, but rather 'speed for guarantee,'" says Friedman. "If the information absolutely must not be wrong for each and every person who views it, then collaborative content is not going to work very well."

Clemons notes that Wikipedia should never be the sole source of information. "I only use Wikipedia for things I know cold, looking for a terse description to share with my daughter doing homework so I don't have to write it myself. But if I don't know the subject, I would never consider Wikipedia as a source of information," he says, adding that the biggest issue is that Wikipedia provides no information on who posted the story, the writer's credentials or the motives behind changes.

Waldfogel agrees. "With these sites it is buyer beware. Anyone who is interested in the topic has to do more research. You can't think that these things are absolutely true."

At issue is whether collaborative content, such as Wikipedia, is up to standards of professionally-edited content. Wales acknowledges that some of the attention given to Wikipedia of late could be attributed to the ongoing debate over old media versus new media, but adds that the online encyclopedia functions more like a traditional editor driven outlet in which writers post articles and editors check them. "The way Wikipedia operates is far more traditional than people realize," says Wales. "We have a core group of editors who do the work."

Policing Wikipedia

What succeeded in getting Wikipedia to this juncture -- an open format where anyone can post articles -- may not work as the online encyclopedia continues to grow, says Fader. The issue: There is no rating system to lend credibility to people who create and edit articles. For instance, a system that keeps editors of articles anonymous, yet rates them much like eBay rates merchants, could enhance Wikipedia's credibility. A contributor known for accuracy or subject expertise would be given, say, five stars. A vandal or someone shamelessly plugging a product would get one star.

In response to recent events, Wales has made a few changes to better police the site. In addition to implementing a "semi-protection" policy where pages facing vandalism problems can only be edited by registered users with accounts older than four days, site administrators will add a delay to high profile pages, such as the President Bush page that is currently closed. The delay, similar to a "dump" button in radio that gives producers a few seconds to prevent something like crude language from hitting the airwaves, will mean that online vandals cannot quickly amend a page on a breaking news article, such as those posted when the Pope died. Wales notes that the length of the delay will be decided by the community on Wikipedia. "We have hundreds of people monitoring the site all the time. They will figure out the community norms."

Other changes are in development, but Wales didn't disclose anything specific, except to say that there are no plans to institute an eBay-like rating system such as the one Fader suggests. That type of system couldn't work and would "be destructive of the community," Wales suggests, because Wikipedia has 800 to 1,000 active editors. A rating system would be degrading as well as unrefined. For instance, a biologist may be an expert on biology topics, but turn into a ranting madman when it comes to the Israeli-Palestinian conflict. "In that example, the person would probably be rated a three-star overall," says Wales. "But the rating would lose the human judgment element. In our system, we learn over time that you need to keep an eye on that editor" when he or she is working on topics related to the Middle East.

Whitehouse acknowledges that some Wikipedia content may be inaccurate, but overall the policing system works. "Look at what Wikipedia has done in a short period of time. By and large, self policing has worked well."

Emerging Models

The challenge for Wikipedia will be to juggle the monitoring of highly visible pages and the low-traffic areas that harbor misinformation. "The process works when there is a back and forth that allows discussion and dialog," says Whitehouse. "The problem is policing the obscure articles no one is looking at."

It's unclear how the Wikipedia model will evolve, but some potential rivals have their own ideas. A site called Digital Universe recently launched offering "stewards" to review content and providing two tiers of articles -- those verified by experts and those not. The company bills itself as the "PBS of the Web" and a university of human thought and experience.

Friedman's Copyright, which will kick off in mid-February, is an open access, peer-reviewed journal on all aspects of copyright in the Internet age. Authors will submit detailed research, case studies and opinion pieces vetted by rapid peer review. Copyright will seek a balance between offering access to all users, but making sure information is accurate. According to Friedman, there is still a big role for collaborative content, but the filtering of vast amounts of information will become increasingly critical.

While Wikipedia is effective, it "allows anyone to post [information] and just prays that the experts will wind up posting more than the malicious or incompetent," says Friedman. Copyright's approach will be open to all, but submissions will be vetted by, among others, Fader, Lawrence Lessig, a law professor at Stanford Law School, and William M. Landes, a law professor at the University of Chicago, before being published.

Experts at Wharton agree that Wikipedia will weather its most recent flap, but its model may have to be tweaked. Whitehouse suggests that citizen-authored content is a valuable contribution to knowledge, but there is still a place for more edited information. Wikipedia's challenge: Find the middle ground.

Links

• Wikipedia's Co-founder Eyes a Digital Universe: CNET news.com
• Wikis, Weblogs and RSS: What Does the New Internet Mean for Business? Knowledge@Wharton
• Internet Encyclopedias Go Head to Head: Nature Magazine
• Copyrightjournal.org