On search engines and how they work

Most people are familiar with search engines and how to use them. However if you are going to build a website and get traffic to it you need to understand how search engines determine which results to present. This is fairly straightforward in principle but the practical details are extremely complex.

The first thing that the search engine has to do is to go out and find all of the content on the web. The do this by sending out what are called robots to crawl the web. These robots will visit millions of sites each day in order to determine what they are about and what search terms they should be ranked for. The robots will constantly go back to the pages that they have already visited and indexed in order to see if they have been updated. They will also follow the links on these pages which is how they find new sites. If you have created a new website the best way to get it indexed by the search engines is to get a link from a site that is already indexed and which is updated frequently.

In order to determine what a site is about and which search terms a site should be ranked for the search engines have created an algorithm. The algorithm is constantly being changed in order to improve the results and to put a stop to people who are trying to game the system. In the past the algorithm used to be based almost exclusively on how often keywords appeared in the text. Over the years search engines have improved dramatically and can now tell what a site is about based on the text.

Once the search engine has found a site and determined what it is about the next thing that it has to do is to determine the order in which all of the sites that are relevant to a search term should appear in the results. The goal is to have the most relevant and best quality sites appear at the top of the results. In practice this does not happen in a lot of cases because the robots are not able to judge quality. The way that quality is determined is largely by the number of links that a site has pointing at it and where they come from. This does leave room to game the system but it is a lot harder than it used to be.

The final step is to present the results when somebody searches for a term. When you search for something the search engine will look through its index for pages that match that term and then present them in the order that they have been ranked by the algorithm. Given the billions of pages that currently exist this is quite a feat.