Search Engines in 1998, Before Google Takes the Spotlight
Google makes the transition from Stanford project to company over 1998, but it is portals like Yahoo! and portal-wannabes like AltaVista that feature in Danny Sullivan's Search Engine Watch this year.
1998 was the last year of the web before Google began setting the agenda in search. During that year, there were hints that Google was emerging — even before it officially incorporated in September, it was gaining credence among industry insiders as a search technology to watch. But for all of 1998, the leading names in search were either portals (Yahoo, Netscape, Excite) or search engines trying to turn themselves into portals (AltaVista, Infoseek, Lycos).
With the portals, often the search function was either partly or fully outsourced. So when assessing how popular a particular search engine was, you first needed to work out how it was integrated into the leading portals or directories of the day.
Take AltaVista, for example. Although it did have a destination site — at the time located at altavista.digital.com — it also syndicated its search technology to partners like Yahoo and Netscape. So while many users visited AltaVista directly, a large proportion of search query volume came from its presence on these popular portals.
Search Alliances
At the beginning of 1998, Danny Sullivan’s Search Engine Watch — one of the first search industry blogs — had a feature called the "Search Engine Alliances Chart." For Yahoo, Sullivan wrote that if the main directory didn't "turn up any useful links," then Yahoo made it easy for users to try a search engine:
"With a click, the query originally sent to the Yahoo catalog is "piped" or forwarded to any of the major search engines. Because so many people use Yahoo, search engines listed first on Yahoo pages have a strategic advantage over others. AltaVista is the preferred search engine. It is listed prominently, and the results returned from a piped AltaVista query appear as part of the Yahoo web site."
In its own help pages, Yahoo explained that "Alta Vista is what's called a search engine, and it specializes in indexing every single web page it can find." That wording is a reminder that web search was still a relatively foreign concept to web users in 1998.
In hindsight, we can see that 1998 was an inflexion point between the directories era of the web (led by Yahoo) and the emerging search era.
The dominant browser company, Netscape, also had a list of preferred search engines for its homepage cum portal (Netcenter). In early 1998 it had a fairly large list to choose from, including Yahoo:
By mid-1998, Netscape had a visually attractive search form that highlighted its own search, plus five other services (Yahoo was not among them). As explained by Search Engine Watch in a June 1, 1998 update of its alliances page:
"By default, the Net Search page displays a different form for one of six search services automatically. These services are called "premier" providers. They have paid Netscape the most money to appear on the page, so they get the most prominent treatment. The rotation isn't quite random. Netscape gets 25 percent of the traffic, as does Excite. None of the others receives more than a 15 percent share."
It wasn't just the portals forming alliances — the search engines also occasionally joined forces. In the second edition of the book The AltaVista Search Revolution, published in April 1998, it was explained that the subject search in AltaVista was provided by LookSmart (which at the time was primarily a directory, like Yahoo):
"So, where did Browse by Subject come from? It's from the LookSmart directory, actually. LookSmart is a guide to the Web, with a specially selected list of Web sites. Editors hand-select sites for inclusion in the directory, categorize them appropriately, and provide mini-reviews. AltaVista licensed the technology and use of the directory to help provide AltaVista users with a comprehensive searching solution."
Google Arises
While the portals and wannabe-portals were making partnerships and tweaking their directories, a revolution in search was quietly underway at Stanford University, by PhD students Larry Page and Sergey Brin. The public coming out of their project, Google, was in April 1998 at the 7th international World Wide Web conference in Brisbane. There Page and Brin presented a paper entitled The Anatomy of a Large-Scale Hypertextual Web Search Engine. The introduction suggested that it was already better than existing search engines:
"In this paper, we present Google, a prototype of a large-scale search engine which makes heavy use of the structure present in hypertext. Google is designed to crawl and index the Web efficiently and produce much more satisfying search results than existing systems. The prototype with a full text and hyperlink database of at least 24 million pages is available at http://google.stanford.edu/."
By July 1998, Larry Page had even coined the term "googling." In a Google announcements list on eGroups, Page outlined the latest updates — including "some enhancements to our search code that produce more matches for many queries, improving performance significantly" — and then ended the message with a cheery, "have fun and keep googling!"
The first mention of Google in Danny Sullivan's Search Engine Watch (SEW) was an August 4, 1998 article entitled Counting Clicks and Looking at Links:
"The last time some students at Stanford University got involved with categorizing the web, it turned into a little site you may have heard of called Yahoo. That alone makes me surprised someone hasn't yet swooped in to carry off Google developers Larry Page, Sergey Brin and Craig Silverstein into portal heaven. Even more so is the fact that the engine they've put together is really good and even has a catchy name."
Despite the potential of Google, which Sullivan clearly picks up on, it is "portal heaven" news that occupies SEW's attention for the rest of 1998. On November 4, it reported on the latest AltaVista features:
"AltaVista introduced a number of significant new search features in October, including the display of relevant results from the Ask Jeeves answer service, a new filtering option and a photo search service."
So, more partnerships (AskJeeves) and more vertical search options (photo search). This was portal-thinking at its height from AltaVista. It wasn't about focusing on one single thing, as Google at this point was doing. AltaVista wanted to compete on features — the more the better — with established portals like Yahoo, Netscape and Excite.
But as the web grew, so did the need for effective search. One of the reasons Google began to gain traction the following year was its singular focus on producing good search results, which it presented as simply as possible as a list of plain blue links.
First though, Google had to become a company and move off Stanford's web servers. On September 4, 1998, Page and Brin filed for incorporation. By the end of the year, the "beta" website was up and running on google.com.
Things would never be the same again in search — or on the web, for that matter. AltaVista (now owned by Compaq) didn't yet know it, but the age of the portals was quickly coming to an end.
Buy the Book
My Web 2.0 memoir, Bubble Blog: From Outsider to Insider in Silicon Valley's Web 2.0 Revolution, is now available to purchase:
- Paperback, US$19.99: Amazon; Bookshop.org
- eBook, US$9.99: Amazon Kindle Store; Apple Books; Google Play
Or search for "Bubble Blog MacManus" on your local online bookstore.