The CLEVER project was a research project in Web search led by Jon Kleinberg at IBM's Almaden Research Center. Techniques developed in CLEVER included various forms of link analysis, including the HITS algorithm.
The CLEVER search engine incorporates several algorithms that make use of the Web's hyperlink structure for discovering high-quality information. It can be exceedingly difficult to locate resources on the World Wide Web that are both high-quality and relevant to a user's informational needs. Traditional automated search methods for locating information on the Web are easily overwhelmed by low-quality and unrelated content. Second generation search engines have to have effective methods for focusing on the most authoritative documents. The rich structure implicit in hyperlinks among Web documents offers a simple, and effective, means to deal with many of these problems.
Members of the Clever project have come up with a mathematical algorithm that views the Net as simply web pages pointing at each other. It also takes into account the notion of hubs, which point to quality content and link information together, and the idea of authority pages, which are often written by specialists in certain fields.
Bill Cody, senior manager of exploratory data management research at IBM's Almaden Research Center, said: "Web searches provide a lot of information, some good, some bad. But the people providing good hubs usually point to authority pages and authority pages generally know of good hubs. The algorithm enables us to find them and so provide users with quality information rather than the regular list of irrelevant web pages." He added that the algorithm had also been used to find Internet based communities using the same principle of finding links between like and like. It was used to discover 300,000 communities worldwide, only four per cent of which turned out to be spurious. About two thirds of these still existed, he claimed, with about half now appearing on Yahoo as mature communities.
Cody said that the tool could be used for targeted advertising purposes or for enabling users to find out more information about incipient communities, but declined to say whether IBM had plans to turn Clever into a commercial product or not.