What is a search engine, and how does it work?

One of the questions we’re often asked by our clients is how search engines decide to rank your website. A good place to start is probably at the very beginning:


What is a search engine?

The great thing about the internet is that there are literally millions of pages of information on any given topic you could hope to learn about. The bad thing about the internet is that there are literally millions of pages of…

See where we’re going with that?

Search engines are sites on the internet which were designed to help people find information stored on other sites. The term ‘search engine’ is often used generically to describe both spider/crawler-based search engines and human-powered directories. These two types of search engines gather their listings in very different ways. For the purpose of this blog post, we’ll focus on crawler-based search engines, such as the almighty Google.

The ways various search engines work are different, but they all have three things in common:
1. They search the Internet based on words that are defined as important
2. They keep an index of those words, and where they were found
3. They allow users to look for words or combinations of words found in that index.

During the crawling process, search engines use small programs called crawlers or spiders that are continuously browsing the web. Spiders are directed by a program that provides the URLs for the spider to scan.

During the indexing process, a copy of the web pages that have been found and scanned by the spiders is stored. From this copy, the website’s indexing program records the words from each page along with the URLs where this word has been found, in a large table.

When the search engine returns the results to search queries, it is able to present the most relevant results to that particular query. Results appear according to ranking criteria, which is determined by formulas or algorithms used to quantify the relevance of the content against the words that the user is searching for.

Although the algorithm formulas are a secret, the numbers of times that a particular keyword and its position in the web copy are the main factors that determine the relevance of the page to that particular keyword. The most relevant results are listed first in the search engine result pages. Other things that influence search engine rankings are use of bold text, internal links to other pages, page titles, meta tags and incoming links from other respected websites.

Paid Results vs Organic Results

Most search engines deliver search results in two ways – ‘paid’ results and ‘organic’ results. Organic search results occur as a result of the process outlined above.

Paid inclusion is a search engine marketing model in which web site owners pay a search engine company to guarantee their sites will show up in search results. In addition to guaranteeing that site will be indexed, paid inclusion may ensure that the search engine’s spider visits the site more frequently, and may also give the site’s owners the option to submit information about their pages more often. Different search engines treat paid inclusion results differently. Some indicate the paid inclusion results as advertisements while others display them as results alongside non-paid search results.

What is search engine marketing (SEM)?

SEM refers to the process of building or making changes to a web site in order to increase its ranking in the results pages of search engines and directories. Mostly, the term SEM refers to the paid methods on improving a site’s ranking, including pay per inclusion, pay per click, banner ads or any other marketing programs offered by search engines.

What is search engine optimisation (SEO)?

SEO refers to the process of building or altering a web site’s structure and content so that it ranks higher in the organic, crawler-based listings of search engines. These changes may involve:
• rewriting body copy
• altering title or meta tags
• removal of frames, flash content
• incorporating internal links, incoming links, and exchanging links with other related and credible web sites
• optimising URLs
• optimising the directory structure
• optimising any content that’s incorporated using PDF files
• incorporating a site map, and a user-friendly 404 page

If you want some help on either the SEO or SEM front, give us a shout. We’ll take a look at your site, and let you know what we can do to help.