How to Optimize a Database-Driven Website for the Search Engines
Problems and solutions in optimizing a database-driven website for the search engines
There are problems optimizing a website for the search engines, where the pages of that site are created from pieces pulled from a database. If you use cgi, Cold Fusion, Microsoft ASP, or other various proprietary shopping carts like xCart -- these make the kind of web pages where the URL (as you see it in your browser's address bar) contains question marks ("?") or other symbols. The links within this kind of website don't go to existing html pages. The links are set up so that when you click on them, the pages are created instantly for you from information and HTML code stored in a database on the server. It is all put together for you instantly "on the fly" when you click on the link.
Basically, search engine spiders aren't smart enough to figure out how to interact with a database to create those pages, so sometimes they never make it past the first page of the site. While indexing your site and trying to follow the links from your main page, if a search engine spider finds a question mark in the URL that you are linking to, the spider will disregard that link and move on. If your whole site is built this way, the busy spider leaves your site without being able to index anything.
None of the search engines can interface directly to your database and read what is in it -- they just aren't set up to do that.
If you have a database-driven site then special actions need to be taken handle the behavior of the search engine spiders. We do not cover them in detail on this page, this is just an outline of what you need to do:
The main (home) page of the site should link to several static pages (which will need to be created if they don't already exist) which then interact with the database. It helps to have about a dozen of these static (non-changing) pages: these are .html pages which are NOT created by the database "on the fly". Typically they would describe the main categories of what you are selling. The links FROM these "category" pages can create other pages based on information pulled from the database, but these static category pages themselves should not be created newly every time someone clicks a link from the main page of the site.
Each of these static category HTML pages should then be optimized using the appropriate key word phrases for that category. These static HTML pages will then be picked up and indexed by the search engines after going through the submission process. You stand a much better chance of getting good search engine rankings for your website by creating and optimizing static pages like this.
There are other more technical ways to handle it, some of which involve changes to the configuration of the server, and there are various software solutions which will make your site look more friendly to a search engine -- but this method of making some static "category" pages is usually the simplest to implement and meets with the least resistance from your webmaster. The other methods one can use are described in this article at the Web Developer's Journal and depend upon whether one is using ColdFusion, CGI scripts, ASP, or some other method of displaying the information from your database on your web pages.
For another tutorial (with solutions for your specific problem) on this subject, go here: Search Engine Ethics.
For another view on this, see Jill Whalen's article about optimizing dynamic sites.
If you have a database-driven site, contact us for an inspection and suggestions about what else can and should be done. We've seen some beautifully designed websites--very pretty, interactive and doing a good job of selling products--which were invisible to the search engines. The search engine spiders are simply not designed to crawl through the site and index pages created from a database in response to clicks from visitors.
The workaround outlined above using category pages work well to get your site listed and doing better in the search engines. Unfortunately, it involves some re-design of the way the site works. Your webmaster may have some resistance to this because it means a lot of work -- but it's the only way we know of to get good placement in the search engines for a website built around a database of products and information about them.
Another article by Dirk Brockhausen telling how to use the mod_rewrite module on an Apache server to get rid of the question marks in a URL that are bothersome to the search engines.
For even more information about this whole subject, see this WebMonkey article by Todd Elliott from 1997 -- which is still mostly true even though it is five years old now. (This is not a new problem!)
Return to Top of Page.
