Dion Hinchcliffe's Web 2.0 Blog
Web 2.0 University

Blog Feed

Subscribe By E-Mail

Enter your email address:

Delivered by FeedBurner



Web 2.0 Expo San Francisco 2008 Speaker

Enterprise 2.0 Conference 2008 Speaker

Web 2.0 Strategies 2008 Speaker

    follow me

    Dion's Facebook Status

    Recent Readers

    Web 2.0 Ajax SOA Power Panel

    Web 2.0, Ajax and SOA Power Panel with Dion Hinchcliffe and Jeremy Geelan
    Click above to watch a SYS-CON Power Panel discussion on Web 2.0, Ajax, and SOA with Dion Hinchcliffe, Jeremy Geelan, and other industry notables including SOA Web Services Journal Editor-in-Chief, Sean Rhody. Taped on Dec 7th, 2005 from the Reuter's TV studio in Times Square.

     

    Public Calendar

    Five Great Ways to Harness Collective Intelligence

    « H E email
    posted Tuesday, 17 January 2006
    Over the recent weekend I read Ellyssa Kroski's superbly researched and written new article, The Hype and Hullabaloo of Web 2.0.  It's a must-read piece whether you're a die-hard aficionado or a battle-hardened detractor.  The article is essentially an in-depth snapshot of the current state of affairs of our favorite topic. And it's particularly well-cited with supporting material that also makes excellent reading.  I definitely encourage you to give it a close read.

    One of the things Ellyssa discusses is some practical examples of harnessing collective intelligence, one of the linchpin techniques of successful Web 2.0 software.  The article describes this as when the "critical mass of participation is reached within a site or system, allowing the participants to act as a filter for what is valuable."  I single this definition out as an understatement.  While the article gives some excellent examples of collective intelligence and I'm not certainly faulting Ellyssa in the slightest, I'm just saying that observing that harnessing collective intelligence allows Web systems to be "better" is like saying Moore's Law states how computers get faster over time.  It wasn't for nothing that Einstein said that compound interest was the most powerful force in the universe.  And harnessing collective intelligence is about those very same exponential effects.

    But speaking in generalities invariably ends up derailing all too many discussions of Web 2.0 ideas.  So along this line, I thought I'd dive in and provide a list of concrete ways that Web software can leverage collective intelligence to particular effect.  If you're not sure about this, don't forget, the entire product that highly successful companies like eBay have to offer is the aggregation of their user contributions.  So too is the blogosphere.  The concepts here are potent and immensely powerful.  Thus, mastery of architectures of participation to create real value will be essential to success in the Web of the future.





    Five Great Ways to Harness Collective Intelligence


    1) Be The Hub of A Hard To Recreate Data Source - This is a classic Web 2.0 concept and success here often devolves to being the first entry with an above average implementation.  Examples include Wikipedia, eBay, and others which are almost entirely the sum of the content their users contribute.  And far from being a market short on remaining space, it's lack of imagination that's often the limiting factor for new players.  There is so much more terrific software like digg and del.icio.us waiting to be created.  So don't wait until it's perfect, get your collective intelligence technique out there that creates a user base virtually on its own from the innate usefulness of its data.  Just be careful and avoid crowded niches, like peer production news.

    2) Seek Collective Intelligence Out - This is the Google approach.  There is an endless supply of existing information waiting out there on the Web to be analyzed, derived, and leveraged.  In other words, you can be smart and use what already exists instead of waiting for it to be contributed.  For example, Google uses hyperlink analysis to determine the relevance of a given page and builds its own database of content which it then shares through its search engine.  Not only does this approach completely avoid a dependency on the ongoing kindness of strangers it also lets you build a very big content base from the outset.  This ultimately has interesting intellectual property implications, as I've discussed before.

    3) Trigger Large-Scale Network Effects - This is what Katrinalist and CivicSpace did and many others have done.  This is arguably harder to do than either of the methods above but it can be great in the right circumstance.  With one billion connected users on the Web, the potential network effects are theoretically almost limitless.  Smaller examples can be found in things like the Million Dollar Pixel Page.  That's not to say that network effects don't cut both ways and are probably not very repeatable, but when they happen, they can happen big.

    4) Provide A Folksonomy - Self-organization by your users can be a potent force to allow the content on your site or social software to be used in a way that more befits your community.  It's the law of unintended uses again, something Web 2.0 design patterns strongly encourage.  Allow users to tag the data they contribute or find and then make those tags available to others so they can discover and access things in dynamically evolving categorization schemes.  Use real-time feedback to display tag clouds of the most popular tags and data; you'll be amazed at how much better your software works. It worked for Flickr and del.icio.us and it'll probably work for you too.

    5) Create a Reverse Intelligence Filter - Like Ellyssa points out, the blogosphere is the greatest example of this and sites like Memeorandum have been using this to great effect.  The idea is that hyperlinks, trackbacks, and other information references can be counted and used as a reference to determine what it's important.  Combined with temporal filters and other techniques and you can create situation awareness engines easily.  It sounds similar to #2 but it's different in that it can be used with or without external data sources and is aimed not at finding but at eliding the irrelevant altogether as an active filter.

    Of course, there are other important ways that collective intelligence can be exploited but these are some of the main ones.  I'll continue to talk about this in future posts since effectively harnessing collective intelligence is one of the first order ideas in the Web 2.0 practice set.

    What other techniques for leveraging collective intelligence are there?

    links: del.icio.us    


    Web 2.0 is about participation! Please leave comments. Use wiki markup for links/images. (9)

    AddThis Social Bookmark Button

    1. laurence timms left...
    Tuesday, 17 January 2006 4:42 am :: http://chuquet.com

    I've been developing http://chuquet.com for over a year now, and I never knew that it was a Reverse Intelligence Filter. Wow.

    Chuquet.com has a reasonbly complex way of filtering 'interesting' stuff from rss/atom feeds based on the number of unique feed references to a link plus how recent the link was.

    This means that stuff that's getting a lot of interest over a short period of time will come up higher than items that everyone is linking to. You'll see a new item with four or five references hit the top while an older link with forty or more references appears lower down.

    Chuquet.com is still being enhanced, which is why I haven't done a lot of shouting about it. In the next couple of weeks it will be offering tag and opml-driven searches, plus RSS and OPML search result output.

    I'm still debating with myself as to whether I should allow users to 'rate' items a la reddit/digg or simply rely on the reference/time filter.


    2. Patrick left...
    Tuesday, 17 January 2006 11:48 am

    Thanks for the pointer to Ellyssa's article... that's the best overview I've read. I noticed that your examples of collective intelligence harnessing each involve a very public implementation. I wonder how the rules change for smaller communities, e.g. a private wiki in a 10 person company. Arguably the reverse filter is not as critical, since the scale is smaller, and *everything* is relevant by definition. (That said, if you participate in a bunch of private spaces, a summary view becomes more important).

    I think the folksonomy arguably becomes even more powerful in smaller, contained groups. Tagging applications here can possibly go beyond indexing/categorization/topic discovery and enable light process management, e.g. "in-process", "bob:review", "steve:favorite".


    3. Pete Cashmore left...
    Tuesday, 17 January 2006 3:43 pm :: http://www.mashable.com

    Excellent post - I enjoyed it.


    4. Dion Hinchcliffe left...
    Tuesday, 17 January 2006 7:57 pm :: http://web2.wsj2.com

    I agree Patrick and folksonomy is underrepresented in Web 2.0 software still and when done, is often done poorly. I look forward to lots of improvement and more widepread use.

    Additional information can be found here in an earlier article of mine if you're interested.

    Best,

    Dion


    5. mukara left...
    Tuesday, 4 July 2006 10:01 pm

    curious


    6. Andrew left...
    Sunday, 3 December 2006 9:28 am

    This was very helpful. We've created Public Insight Journalism at American Public Media (americanpublicmedia.org/publicinsight). We've built a network of 23,000 and regularly target questions to them, the answers to which public insight analysts (our version of reverse intel filters) sort, distill, and pass on to journalists to inform our news covreage.

    I'd say we do 1 and 2 great, but we could stand to figure out 3, 4 and 5.


    7. Bill Kuhns left...
    Tuesday, 23 January 2007 4:06 pm

    Could I suggest a sixth model for "harnessing collective intelligence"? These would be utterances combined from many sources that nonetheless achieve a high degree of coherence and result in what could be called a "self assembling message." Reputations and rumors are like this: in both cases, a lot of variations and conflicting ideas become ironed out by the paraadoxical network process of achieving fusion through diffusion. There's a terrific study of both the process and the effect of network utterances: Tamotau Shibutani's Improvised News. There are plenty of examples on the Web, from open source to collaboratoriums of all sorts. We need a name for these collective utterances. My candidate would be Noospeak -- indeed, could anything be further from Orwell's Newspeak than web-enabled Noospeak?

    (I'm writing a posting called "From Newspeak to Noospeak:)


    8. jean-luc left...
    Monday, 26 February 2007 10:14 am

    Folksonomy is at its beginning, so it's partial and incomplete. I believe that web 2.0 is mainly trying to organize the web : searching keyword with google is becoming less and less attractive because of the amount of data available. Web 2.0 companies provide a way to add metadata to the web content to be able to quickly and efficiently retrieve data. Harnessing Collective intelligence is either : - being able to brings a lot of people to do the job for you (1-Be The Hub of A Hard To Recreate Data Source or 4-Provide A Folksonomy), - finding a way to reorganize existing data to bring more efficient search capability (2-Seek Collective Intelligence Out or 5-Create a Reverse Intelligence Filter)

    I'm not sure about 3) Trigger Large-Scale Network Effects : This is more for me the use of a new technology to do what some people have done for years. When a union organizes a strike this is what I call Large-scale network effects.


    9. TJ left...
    Friday, 11 May 2007 6:27 pm :: http://blog.tektag.com

    This is a great article. Tim O'Reilly often talks about the participatory aspects of Web 2.0, but this is a great breakdown of the areas and ways it can applied. Something I think businesses need to heed. In our case at TekTag.com, we're just starting the journey toward providing the community a place for tech support using bookmarking and folksonomies. Feedback welcome.

    TJ