Surveying the Search Engines

Google & SEO | September 10th 2009


After diving straight into the deep end and discussing redirects, search engine crawlers and site optimization I think it’s time for a little background. It’s important to know what options we have out there and the differences between the search engines may affect the way we work with them. We mention Google 100 times a day in different contexts, but what is Google? What other options does it give me? And who are its competitors?

Google– was established in September 1998 by Larry Page & Sergey Brin as a Stanford University project. Today it is the number one search engine with well over 60% of worldwide searches. Google offers a wide range of features including the ‘Cache’ option that allows you to easily see how the crawler perceives your site or older versions of web pages. It also provides spell checking, access to dictionary definitions, maps, telephone numbers and more. Google has its own free toolbar that has gained a lot of popularity and operates the Knol site which is an information community where many articles are published by Google users.

To get listed in Google you can submit your site directly or create links to it from other ranked websites.

In addition, Google also operates advertisement programs through AdWords, which is a Pay Per Click (PPC) program where webmasters can create short ads and choose the keywords.  Payment is made depending on the demand for that keyword and according to the number of times the ad was clicked.

Yahoo – this is the oldest directory on the web, launched in 1994. Originally this was a place where human editors arranged websites into categories. Later, in 2002 Yahoo shifted towards crawler-based listings that came from Google until February 2004. Today, Yahoo uses its own technology but it maintains a ‘human’ directory under the Yahoo Directory Home Page.

The search engine includes easy to use tabs above the search box, making it possible to search for images, Yellow Page listings or shopping results. Inclusion in the commercial listings includes a fee, although non-commercial content is accepted free of charge. The search engine also launched a content acquisition program in 2004, where sites can pay to be included in the crawler results, although Yahoo assures that this does not guarantee ranking. This program also brings in content from non-profit organizations for free.

Just like Google, Yahoo also sells paid placement ads that appear on its site and are distributed to other sites. The current Yahoo crawler technology is a combination of AltaVista and AllTheWeb technology together with Inktomi (a crawler based search engine company developed from UC Berkley), purchased in March 2003.

Submitting to Yahoo is possible through their site, and you can choose a category for your Flash website.

Ask – this search engine rose to prominence in 1998 & 1999. The concept was the use of ‘natural’ language. This means that you can ask a question in a ‘natural’ format and find search results, thanks to unique technology. In reality however, at the starting point at least, there were 100 human editors monitoring the search log – making sure the search engine’s performance was up to par, even when the technology was not. Later the technology was improved as Ask acquired Direct Hit in 1999 and then, in 2001, Teoma’s unique index and search relevancy technology. Today, Ask also uses crawler based technology to provide results for its users. The algorithm which Teoma uses is known as ExpertRank.

You cannot list your free website with Ask.com for free. You can either wait until its crawler finds you – again, through incoming links from other, ranked sites, or you can add a sponsored link for PPC.

Other Search Engines to Consider

AllTheWeb.com – powered by Yahoo, AllTheWeb is lighter and more customizable. It focuses on web searches but news, videos, pictures, MP3 and FTP searches are also provided.

AllTheWeb provides an easy and free submission process.

AOL.com – this search engine has internal and external search results. This depends on whether you are using AOL as a service provider or not. The ‘external’ version’s listings come directly from Google’s crawler based index, meaning that the same search in both search engines will come up with almost identical results. Why use AOL search? Because you are an AOL user. Even so, the ‘internal’ version provides links to content that is only available within the AOL online service. Certain things that are available with Google are not available with an AOL search, such as the cached option.

To get listed in AOL you need to be listed with Google, as AOL virtually duplicates Google’s editorial and ad listings.

HotBot – provides access to the three major crawler based search engines on the web, Yahoo, Google and Teoma. Its ‘Choose a Search Engine’ interface was launched in December 2002, but HotBot had a long history prior to that as a search brand.

HotBot debuted in May 1996 and initially had a strong following amongst hardcore searchers looking for comprehensive and quality results. At the time, search results were provided by Inktomi. HotBot’s popularity began to drop when it switched over to using Direct Hit’s search results for its main listings in 1999. This is about the time Google entered the scene, and Direct Hit’s results could not compete, causing both search engines to drop.

To get listed in HotBot you need to be listed by the three major crawlers it uses to day.

More and More Choices

AltaVista was launched in 1995 and served as the Google of the day for a while. Unfortunately when AltaVista turned into a portal in 1998 it lost track of the importance of the search and overtime, the relevancy of the results dropped as did the freshness of the listings and web coverage.

Today AltaVista is focused on searches again and results come from Yahoo. Certain tabs above the search box allow you to find images, MP3/audio files, video files and human category listings.

AltaVista is good for those Yahoo users who prefer a lighter interface.

Live Search is Microsoft’s web search engine. It is designed to compete with market leaders such as Google and Yahoo. It offers a few innovative features, such as the option of viewing more results on the same page (as opposed to clicking through subsequent pages) and the option of customizing the amount of information displayed for each search query. For example, you can choose to view only the title, or a short summary, or a long summary… depending on your needs. You can also save searches that will continue to be automatically updated.

Lycos was launched in 1994 and is one of the oldest search engines around. In April 1999 it stopped crawling the web for listings on its own and instead began to provide human powered results for popular  queries and crawler based results through Yahoo for other search terms.

Its ‘Fast Forward’ option allows you to see results on one side of your screen while the actual page are listed on the other. Relevant human-compiled informational categories appear at the bottom of the results page.

To get listed on Lycos you need to be listed on AllTheWeb.com.

The Open Directory uses volunteer editors who catalog the web. It was launched in June 1998 under the name NewHoo and was purchased by the AOL Time Warner owned Netscape in November 1998.

You can view searches directly from the Open Directory but their interface is not recommended. The site has no backup results in case there is no match in the human-compiled database. Also, the site rankings by keywords are poor, and an alphabetical order is used when you browse the categories.

A better option is to use the version offered by Google, called the Google Directory. Google uses refined algorithms to analyze links and push the better pages from the human database to the top. Also, when viewing the sites by category, they will be listed in order of their page rank. This means the most popular sites will be on top based on link analysis.


Tagged with: .

5 comments

Leave a Reply

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>