Jinfo: Search Trails: Back to the Future

Search Trails: Back to the Future
Jinfo Blog

31st July 2007

Abstract

Humans are great at discovering new things, but not always so good at remembering how we found them. Since the first caveman discovered fire, the knowledge has been passed down through the ages. In a way we've all been touched by that first flame and the idea of it is still burning.

Item

Fire was a fantastic discovery, but human language is our best discovery yet as a way of infecting others with new ideas. With language we can pass more than DNA on to the next generation: we can pass on discoveries too.

But spoken language is ephemeral ('Sorry, how do I make it spark again? I can't hear properly; The Neanderthals are shouting.') So humankind came up with ways of fossilizing our ideas into written symbols on various media: cave walls, stone tablets, papyrus scrolls, books and more recently 1GB memory sticks.

Bringing order to chaos

The Internet is a massive archaeological pile of fossilized ideas. But armed with an average query of 2.5 keywords and Google, we can still dive into the pile and come out relatively unscathed. But for how long? The pile of information is growing and growing. According to the World Wide Web Consortium (W3C), there are 1,173 million Internet users as of June 2007, compared with 938 million two years ago.

Currently Google only crawls part of the pile. Vast tracts of valuable information reside in topic-specific search engines out of the Google crawler's reach.

The Google PageRank algorithm <http://www.google.com/technology/> tries to bring order to some of the pile by ranking Web pages: 'Google provides its views on pages' relative importance,' according to the Google website. But the polarity of PageRank is still skewed in favour of the publishers. PageRank is conferred by Web publishers to other Web publishers in the form of referral links - not by users.

Consequently Web publishers, ably assisted by an army of search engine optimisers (SEOs), are playing Google's PageRank for profit. The raison d'etre of SEOs is to create optimally relevant pages tuned to the Google algorithm so their clients' pages appear high in Google's results. Although these pages appear relevant they lack a crucial ingredient: authority.

It was 'authority' conferred by referral links that propelled Google's PageRank to be the premier retrieval algorithm for the Web, and it is authority that is ebbing away as the SEOs and Web publishers take control of their position in Google's results by manipulating PageRank.

Power to the people

We need a system of Darwinian information selection where the users, not publishers, decide the best answer for a given query. Shouldn't users be given more authority in deciding what is, or isn't relevant? Social search provides this by determining the relevance of search results in accordance with actual usage.

Every day millions of people search the Web and apply human intellect to making search discoveries, yet this effort is mostly wasted. The pile grows and grows but remains untouched by the humans sifting through it. The promise of social search is to harness this communal effort for the good of all. But how can we bring human order to the pile?

I've been wrestling with this question for the past six years and I believe the answer lies in the prescient vision of Dr Vannevar Bush in his 1945 seminal paper "As We May Think" <http://www.theatlantic.com/doc/194507/bush>. Bush describes a machine called a Memex that augments your memory and searching powers by helping you to create and share 'trails of association' between things in the 'common record' (analogous to the Internet).

Back in 1945 this concept must have seemed wild but Bush successfully predicted the future by directly influencing it. Bush inspired Ted Nelson (hypertext), Sir Tim Berners-Lee (WWW) and many others to realise his vision of an 'interlinked common record'. However, there are still major parts of his functional specification missing. Trailblazing, for example, is a crucial, yet largely overlooked part of Bush's invention:

'There is a new profession of trail blazers, those who find delight in the task of establishing useful trails through the enormous mass of the common record. The inheritance from the master becomes, not only his additions to the world's record, but for his disciples the entire scaffolding by which they are erected.'

Trailblazing

Every day millions of people blaze search trails into the pile but most of that search effort just goes up in smoke.

How can all this search effort be harnessed simply and unobtrusively? Fortunately something as simple as saving and displaying a search trail can capture the association between human search desire and its satisfaction.

Let me give you a personal example. I spent my first five years at the University of Queensland in Australia trudging from the car park (known then as the Dust Bowl) up a winding path to lectures. The winding path, colloquially called the Goat Trail, was etched into the grass thanks to the collective unconscious of all the students rushing to lectures - a planner could not have designed a more optimal route.

Around November each year the Goat Trail would inexplicably change route - it suddenly diverted around a large pair of jacaranda trees. The jacaranda trees at the university flower beautifully and smell even better, but local legend has it that they are deadly to students - especially around exam time. The legend warns that if a jacaranda flower lands on your head you're certain to fail your exams. New students who had never heard of the brain-busting jacaranda flower were spared failure thanks to following the Goat Trail. I remember marvelling at the high technology of a simple path on the ground. It was, in essence, a meme-filtering machine.

The Goat Trail actively encoded the association between search desire and the destination (lecture hall) and was malleable enough to move with the times (watch out for those flowers). A similar system is required for Web search. Who needs all that fancy AJAX-ian widgetry on the Web? Search engines just need to provide a simple system of way- finding - we can trust humans to do the rest.

A trail is a simple method of showing the way. On the Web a search trail begins with a search engine, typically 2.5 keywords, a sequence of clicks and ends hopefully, but not always, at a relevant result.

Over time the association between a search keyword (desire) and its most traversed destination becomes stronger. This is analogous to human memory. Each time a memory trail is traversed the synaptic gaps fire between neurons and the engram, or unconscious memory, is reinforced. The more times this trail of neurons fires the stronger the memory becomes. If it fires less frequently, the memory fades.

Saving memory lane

The real achievement of human memory is not what we remember but what we forget. Everyday life is full of forgettable factoids and our memory does a great job of filtering them out. Shouldn't search engines do the same?

Despite what the publishers say a lot of the pile is worth forgetting, and we need a system that behaves just like human memory on a communal scale.

I believe large-scale trailblazing can act like a global search memory: reinforcing but also fading the associations between search desires (keywords) and destinations (URLs).

Vannevar Bush lamented the 'artificiality of indexing' and hoped that 'selection by association, rather than indexing, may yet be mechanised'. I hope so too. The time has come for a pure interaction layer that sits above the Web and acts as an associative lens that can burn trails through the pile.

It's time to let the users leave their mark on the pile - not just publishers. There is a simple democratic truth to a physical trail in the grass - it's transparent, open and honest. Millions of micro- discoveries are made every day and we have the means to pass them on to each other - search trails offer a simple way for us to do it.

Related FreePint links:

"Life of the Party: Social Web Browsers" By Stephanie Taylor <http://www.freepint.com/issues/150207.htm#feature>
"An Introduction to Search Engines and Web Navigation" Written by Mark Levene Reviewed by David Stuart <http://www.freepint.com/bookshelf/introsearch.htm>
"Searching for success: an update on search engine developments" By Duncan Parry <http://www.freepint.com/issues/150404.htm#tips>
"Extreme Searcher's Guide to Web Search Engines" Written by Randolph Hock Reviewed by Nick Lloyd <http://www.freepint.com/bookshelf/extreme.htm>
"What's the Big Deal With Social Search?" By Chris Sherman, The ClickZ Network <http://searchenginewatch.com/showPage.html?page=3623153>