There is no product category that appears to be as locked up as Search. Google’s market share is a massive 72%, with the spoils going to Bing, Yahoo, Ask.com and a host of others. Yet there is a flurry of innovation going on in search and it is one of the very few segments of the technology market that is attracting abundant venture capital investments.
The innovations are driven by the emergence of the real-time web. It used to be that search engines crawled sites daily to update the indexes, but now the aim is to capture information, index it and present search results in real-time. Google’s innovation was page-rank to determine what information was the most relevant, which worked fantastically for webpages. That model needs an update for the content of the likes of Twitter and Facebook, where the relevance is determined by the number of friends of the poster, the number of times a message is repeated, the time at which it was posted and at what location.
And beyond that trend there is still the hope that one day the search engine will not respond to the character strings that I type, but to the question I have in my head, with full understanding of my interests and context. Also in that field, semantic search, progress is being made. In fact, for real-time search it is critical that these efforts succeed because 140-character messages give too little context to enable meaningful search results with traditional approaches.
I asked a number of the incumbent and new search engines for information about Al Gore, which I thought would be a topic with history, but also sure to be a lively topic in the real-time web (Update Nov 22: I wrote this on Saturday afternoon, Al Gore appeared on Saturday Night Live, quite a coincidence. I reshot the screen shots for this story on Sunday):
As a baseline, let’s ask Google:
Unless you have lived under a rock in the past decade, this mix of relevant web pages, advertisements and news about the topic will look familiar to you. But even in this well-known page there is a lot of innovation going on, click for instance on the “Wonder Wheel” in the left sidebar (click “show options” if you do not see the sidebar), it will show you topics that Google thinks are related to the topic “Al Gore”. Notice also that one can check for information on a topic that has become available within the last hour – real-time anyone?
Take a look at the search results presented by Bing:
The Bing results closely match the Google results, but then Bing adds search results in a range of additional categories like “El Gore Facts”, “El Gore Quotes”, “El Gore Biography”, “El Gore Accomplishments” and “El Gore Book”. This moves the Bing results in the direction of semantic engines like Hakia (discussed below).
Here are the results from Yahoo:
Yahoo offers about the same search results as do Google and Bing, but then it adds links to news sources like ABC and Huffington that provide remarkably good content. Yahoo also offers useful related searches.
So now take a look at some of the Real-Time search engines, starting with TweetMeme:
Tweetmeme counts the number of times tweets link to a URL and lists the URLs with the highest number of tweets that refer to them. This metric establishes relevance to the links, it makes it easier to make sense of what is discussed on Twitter. But that strength is also its weakness, I cannot learn any more than what gets tweeted, and from my experiments so far that does not teach me more about my topics than the gossip that surrounds them.
Tweetmeme shines in its own categorization. I follow its Technology category and often find useful information there that I have not found before.
Another way to find information that is currently discussed on the real-time web is CrowdEye:
CrowdEye has an algorithm that assigns a number between 1 and 100 to give relevance to the poster of the tweet about your topic. Like Tweetmeme, CrowdEye is a great site to browse and discover topics that are Hot in the Twittersphere, but it does provide less context than Google, Bing and Yahoo.
Another search engine for Twitter that I like is TWazzup:
TWazzup has many slick features, such as the ability to Retweet straight from the listed tweets, saved searches (very convenient for complex boolean searches), list of influential twitterers on the topic of your search, and more. The homepage of TWazzup is very sparse, I use this bookmark to populate it with the major news of the day.
There are a number of additional search engines which track what is being discussed on Twitter, check these links to see which one you like best: DailyRT, which prioritizes topics by their number of retweets, Topsy, another search engine that weighs the influence of the posters, and Mozzler, Scoopler and OneRiot , which take information from Twitter, Digg, Delicious and more. And Bing has a Twitter search page with the same functionality:
Semantic Search adds capability to search engines to parse the meaning of the information it indexes. Hakia is an impressive example of this:
It provides information about Al Gore in categories such as News, Biography, Awards, Speaches, Quotes, Blogs, Interviews, Controversies, Contact Data and more, much more. So far, the information I have found on Hakia has always been genuinely useful. Other sites that offer semantic search capabilities are Yebol and Powerset, absolutely the best search for Wikipedia.
And last but certainly not least is the Ellerdale project. The website is still very much a beta, but the potential of its technology platform is easily recognized:
Ellerdale parses the internet to discover “topics”. It manages to recognize that Schwarzenegger, Governator, Arnold and California Governor are the same thing. Even more exciting is that it can apply that semantic understanding to index the real-time web. It can see that a Twitter entry about “arnold” is really about Schwarzenegger and show it in the search results for the California Governor.
