Spiders And Search Engines

Written by Gregory Rouse Monday, 25 April 2011 03:45

When a potential customer searches for products and services they use a search engine to do it. So I think we know why it's important to know how search engines work.

When a potential customer searches for products and services they use a search engine to do it. So I think we know why it's important to know how search engines work.

Spiders are not only the creepy crawlers that you squish with your shoe, but also the name for the software robots that search engines use to index websites.

A 'spider' is an automated software program that is run by the search engine system. It's a text-based reader that doesn't see pictures, colors or video, it only reads text.

The spider visits a website, reads the content on the actual site, the site's Meta tags and also follows the links that the site connects with. The spider then returns all that information back to a central depository, where the data is indexed.

Not only will the spider index your website, but it will index any website you link to. Now some search engines only index a certain number of pages on each site, so no need to create millions of pages if you don't want to.

Search engines do return to check on changes to the website, so new content is always good. How often they do this depends on the engine.

When you ask a search engine to locate information, it is actually searching through the index which it has created and not actually searching the Web. Different search engines produce different rankings because not every search engine uses the same algorithm to search through its indexes.

The algorithm is a mathematical formula that gives points based on certain criteria or key factors that it deems important. Algorithms are unique to each search engine, they're the "secret sauce" and each search engine uses its own recipe. The goal is to provide the best search results for each user and so search engines are always changing and updating their recipes.

One of the things that we know a search engine algorithm scans for is the frequency and location of keywords on a web page, but it can also detect artificial keyword stuffing or keyword spamming.

Another key factor we know the algorithms analyze are the way that pages link to other pages on the Web. By checking how pages link to each other, an engine can both determine what a page is about and if the keywords of the linked pages are similar to the keywords on the original page.

If you plan on optimizing a website it's important to know how search engines work.