SEO - The Other Side

Nothing is more telling about the 'behind the scenes' operation of search engines than developing one's own search engine. In developing a search engine, many factors must be considered including speed, accuracy of the search engine results pages (SERP's), and the exclusion of spam sites or low-quality content. Many talk about SEO as a form of trial and error... "I did this and I ranked at this number for this keyword in this search engine". But these trial and error methods can only scratch the surface of the 'behind the scenes' operation of the search engine and everyone's goal, to rank higher. Some webmasters have got it down and can rank low quality websites very well on the major search engines. Maybe they've found the loop holes with trial and error that exist with present search technology. When these loop holes are shut, those guys that ranked well now rank poorly.

The SERP Quality Standard
From the point of view of the search engine developer, the higher quality results that can be returned for all searches, the more useful the search engine. There are two basic ways to look at this problem and many more complex ways that can be used in place of or in conjuction with the basic ways. The first is to eliminate low quality content by identifying it and EXCLUDING it. (Here and in the following, exclude means actual exclusion of a page/site or giving a page/site a lower ranking because it is identified by search engine filters as being of low quality, spam, or belonging to a low quality site). The search engine developer may perform this by scanning the search engine for techniques that currently cause low quality sites to rank higher than they should and reducing the rank of sites that match any patterns that are identified as being indicative of a low quality site. The second method is to identify high quality content and INCLUDE it, where include means to rank it higher. This is essentially the principle behind Google's Page Rank system where high quality sites have been identified by the number of incoming links and the quality of the sites/pages that the incoming links are on.

Attacks on Both Methods of Search Engine Development
The first method of exclusion of low quality results is both easy for the search engine developer to emplace and easy for the spammer to get around. But really, it's nothing more than the spammer's attempt to stay one step ahead of the search engine developer. So long as the spammer can continue to find new loop holes in search engine development as the old ones are closed, they can always rank well in the SERP's.

The second method of inclusion of quality sites can be looked at in at least two unique ways. One follows the principles of Google's Page Rank system and similar systems where the internet as a whole determines the value of a page. This worked great for Google in it's early days when the internet was free from links intended to boost page rank, but this too has slowly become a method that can no longer be sustained on today's internet to provide quality search engine results. The other way to include quality sites looks at the page itself and how it has features that make it a quality page for it's intended purpose or audience, not directly considering any possible exclusions to the page. This has yet to be acomplished by any search engine that I know of because the technology necessary to perform this has not yet been formalized.

Data from internet toolbars
It seems these days that many major search engines (Google, Yahoo, Alexa to name a few) are offering toolbars to internet surfers as a form of convenient searching; many even offer additional features like pop-up blocking or page rank display. Why are the search engines so kind to make this offer? First it will mean that a toolbar user will be more likely to perform a search at the site of the toolbar... this increases traffic and revenues for the search engine. But most importantly, it provides the technology for the development of future search engines. Toolbars can track the paths that users take on the internet, the sites they visit, the sites they revisit, the sites they never visit again. The toolbars allow the search engines of the future to gain intelligence. This knowledge puts the structure of the internet-user interface in the hands of the search engine developer. This technology is one step above anything available today because it is based not on the structure of the entire internet, but on the browsing patterns of an intelligent internet user in conjunction with internet structure.

Search Engine Patterning
Patterns are useful for identifying and organizing information. The internet as a whole is one giant pattern with many subpatterns and slight discontinuities. The power of a pattern-based search engine would be quite impressive, much more powerful than anything we know today. Spam sites would sink like a lead ship without affecting smaller quality sites. The technology is not so simple as what we have today, but entirely feasible.

	Site Search