Is bigger better, when it comes to the size of a search engine's index? Not necessarily. However, a large index can help those who seek unusual or hard-to-find information. Consequently, when you seek the obscure, consider using a search engine with a large index. However, for general searches or for when looking for information about popular topics, a large index does not necessarily equal better results.
Current Size Comparison
Search Engine Sizes Over Time
Search Engine Size War I
SW-II
SW-III
SW-IV
The size figures below are unaudited and self-reported by each search engine (for audited figures, see the Search Engine Showdown web site listed below, which makes the best attempt at this). Figures show how many textual documents have been indexed, which includes HTML files, text documents, PDF files, Microsoft Office documents and other similar files. Image and multimedia files are not included. Nor are Google Groups discussion posts.
Billions Of Textual Documents Indexed
As of Sept 2, 2003
KEY: GG=Google, ATW=AllTheWeb, INK=Inktomi, TMA=Teoma, AV=AltaVista. See the Major Search Engines page for links to these services.
Search Engine Sizes Over Time
The chart below shows how self-reported search engine sizes have changed over the years. Only search engine still crawling the web are shown on the chart. Thus, players such as Northern Light, Excite, Infoseek and others that no longer crawl for their results are not displayed.
Billions Of Textual Documents Indexed
December 1995-September 2003
KEY: See above.
Search Engine Size War I: Sept 1997-June 1999
When AltaVista appeared in December 1995, it used an index much larger than any of the other search engines at that time. Thus, competition forced others to increase their sizes in early 1996. But after these moves, sizes stayed about the same through September 1997. It was then that AltaVista and Inktomi really began the first of the Search Engine Size Wars, competing to claim the bragging rights of being biggest. Inktomi gave up after a couple of months, but Northern Light jumped in to compete with AltaVista to hit the 150 million page mark.
MILLIONS Of Textual Documents Indexed
December 1995-June 1999
KEY: See above plus NL=Northern Light, EX=Excite, LY=Lycos, GO=GO/Infoseek
Search Engine Size War I:
December 1997-June 1999
When AltaVista appeared in December 1995, it used an index much larger than any of the other search engines at that time. Thus, competition forced others to increase their sizes in early 1996. After these initial moves, sizes stayed about the same through September 1997. But by the end of that year, AltaVista and Inktomi began the first of the serious Search Engine Size Wars, competing to claim the bragging rights of being biggest. Inktomi failed to keep up, but Northern Light jumped in to compete with AltaVista in pursuit of the 150 million page mark.
MILLIONS Of Textual Documents Indexed
December 1995-June 1999
KEY: See above plus NL=Northern Light, EX=Excite, LY=Lycos, GO=GO/Infoseek
Search Engine Size War II:
September 1999-June 2000
Just when AltaVista and Northern Light were celebrating hitting the 150 million document mark, newcomer AllTheWeb appeared with a record-setting index size of 200 million documents. Suddenly, a new round of size escalation began. The title of biggest flip-flopped between AllTheWeb and AltaVista at first. However, Google put a decisive stop to the war in June 2000, when it set a new benchmark of 500 million pages indexed and started growing well past its challengers.
MILLIONS Of Textual Documents Indexed
September 1999-March 2002
KEY: See above plus NL=Northern Light
Search Engine Size War III:
June 2002-December 2002
After a long period of being the size king, AllTheWeb grabbed the title back from Google by declaring it had broken the 2 billion document mark. Soon after, Google grew the number of documents it reported indexing up to the 3 billion page mark. Inktomi also released a new version of its search engine that claimed this index level.
Billions Of Textual Documents Indexed
June 2002-September 2003
KEY: See aboveSearch Engine Size War IV:
August 2003-???
In August 2003, AllTheWeb claimed an index of 3.3 billion documents, putting it just past Google's self-reported figure. Google responded within days by increasing its self-reported figure to 3.2 billion. It's likely that over the coming months, new competition to hit the 4 billion document mark may happen.
