Make Your Dynamic Web site
Search Engine Friendly
by Brad S. Konia
Do you have a dynamic
Web site or are thinking of building one? If so, failing to make your
dynamic site search engine friendly can dramatically reduce your visibility
in the search engines. Even though some engines spider dynamic URLs,
others still do not. Those that do spider them may still place artificial
limits on how deep they will travel within such links.
A dynamic site
is one in which the pages are generated on the fly, usually from a database.
You can often recognize one by seeing symbols such as a question mark,
ampersand or other special symbols in the URL.
Let's say that
you're a web designer and a client asks you to build a Web site for
him to sell his products online. If the client has one or two products,
then a static HTML site would be all that's needed. But what if the
client has a database containing hundreds or thousands of products?
To build a Web site like that using static HTML pages would involve
creating a separate page for each product, meaning that you might have
to create thousands of pages.
A dynamic Web site
can solve this problem. Unfortunately, many Web sites make use of this
technique without fully realizing the dangers relating to search engine
visibility.
There are programs
available that will automatically generate static template-based HTML
pages from a database, but for a variety of reasons, this is usually
not the best approach. Most designers prefer to work with active server
technologies such as PHP or ASP to create truly dynamic Web sites. This
saves time and simplifies maintenance.
The only problem
with dynamic Web sites is that they're not nearly as search engine friendly
as static sites. Some engines will not index them or will index only
a limited number of pages. If you've already spent a lot of money to
have a dynamic site designed and built, you're not going to want to
scrap all that work and start over with a static HTML site.
So how can you
retain the functionality of your dynamic Web site, but make it search
engine friendly? First, let me give you an example of how a typical
dynamic Web site functions.
Suppose that you
had a real estate Web site in which customers could view all of your
available properties online. Instead of creating a separate HTML page
for each property, you could put all the information for your properties
into a database. The database might contain fields like:
-
Property ID
-
Property Name
-
Address
-
City
-
Asking Price
-
Owner
-
Pictures,etc...
Your Web site would
most likely include some type of search form so that your users could
search for a list of properties that fulfill various criteria. For instance,
someone might search for a list of properties in Miami, Florida with
asking prices below $300,000.
After conducting
this search, the results page would contain a list of properties that
meet the criteria and perhaps a thumbnail picture of each property.
Then the user could click on the property name or picture to view more
detailed information.
On a typical dynamic
Web site, the hyperlinks to the property detail pages would contain
URLs similar to the following:
http://www.xyzWeb
site.com/property-detail.php?id=57
The property-detail.php
page is a dynamic page that the server builds on the fly. The information
after the question mark in the URL is passed to the server so it knows
which property to display on the detail page. In this case, it will
display detail information for property #57. Therefore, this single
page can actually display detailed information for an unlimited number
of properties. This is an extremely efficient way to manage your Web
site, but unfortunately, some search engines will not index dynamic
pages.
There are two basic
problems in indexing a dynamic Web site:
-
Search engines
will not utilize your search box.
If the only
way to access your dynamic content is by first conducting a query
on your site, a search engine will miss this content entirely! You
must provide access to all your Web pages by doing nothing more
than navigating links. This means that you'll need to have one or
more pages organized by product or category that eventually drills
down to every page in your database that you may want indexed by
a search engine. Without this, an engine will never "see" the vast
content that your site has to offer.
Offering both
methods of locating information on your site will also help improve
your site's navigation. Notice how Yahoo.com gives visitors an option
to conduct a site search or to browse its catalog by traveling a
series of links.
-
Some engines
won't index pages containing a question mark or other special symbols.
Or, they may limit how deep they will spider such sites.
How can we
get around this problem? The solution differs depending on what
server software your Web site runs. Ask your system administrator
or hosting service if you are unsure whether your site runs on Apache,
Microsoft IIS, or another server.
For Apache-based
Web sites...
Most Unix
and Linux sites run a server application called Apache. For these
people, the solution to the dynamic site dilemma is in the Apache
module called "mod-rewrite." Mod-rewrite is a powerful scripting
program that will translate URLs based on the patterns that you
define. In layman's terms, this will allow you to feed the search
engines URLs that appear to be static, but are actually dynamic.
Most Unix
and Linux sites run a server application called Apache. For these
people, the solution to the dynamic site dilemma is in the Apache
module called "mod-rewrite." Mod-rewrite is a powerful scripting
program that will translate URLs based on the patterns that you
define. In layman's terms, this will allow you to feed the search
engines URLs that appear to be static, but are actually dynamic.
As an example,
consider the dynamic URL: http://www.yourWeb site.com/yourscript.php?id=123.
The above
URL passes the variable called "id" to a script called yourscript.php.
This script builds a dynamic page based on the product ID. How can
we make this more search engine friendly?
With mod-rewrite,
you can get the same result using the following URL:
http://www.yourWeb
site.com/productid123.htm
Notice how
the "offending" question mark symbol has been removed from the URL.
This second URL is much more search engine friendly. In addition,
by using the mod-rewrite technique, it will function exactly the
same as the first URL.
When mod-rewrite
sees "productid123.htm", it knows to translate that into "yourscript.php?id=123."
This translation takes place behind the scenes, so the URL in the
browser's address bar will continue to display "productid123.htm"
while your database program sees the URL it is expecting to see.
It is important to understand that there is no re-direction taking
place. It's simply a URL translation.
Rather than
getting into a detailed technical description of how this all works,
I'll give you step-by-step instructions on how to configure this
on your Apache-based server. The main requirement is that your site
be hosted on a server running the Apache web server software and
that the mod-rewrite module is installed (it usually is).
Apache is
by far the most popular web server for Linux, so if your site is
hosted on a Linux server, chances are it's running Apache. Apache
is also available for Windows, but most Windows servers use Microsoft's
IIS Web Server. If your site is hosted on a Microsoft IIS server,
then see the section of this article addressing Microsoft servers.
The instructions
below assume that you're using PHP as your scripting language, but
this could easily be adapted to any scripting language.
Below are the
instructions to implement mod-rewrite. If you are not experienced
in Web site development and scripting, I recommend you forward this
article to someone with expertise in this area. For an experienced
Web developer or programmer, these changes should not take long to
implement. In many cases, your hosting service may be able to make
the changes for you although they may charge a fee.
Instructions:
-
Open Notepad
on your Windows machine (or a comparable text editor if you're using
a Macintosh). Avoid using MS Word or any type of word processor
because these programs add extra formatting characters by default
that will cause problems.
-
Copy and
paste the following text into Notepad:
RewriteEngine
on
RewriteBase /basedir
RewriteRule ^productid([^.]+).*$ yourscript.php?id=$1 [T=application/x-httpd-php]
-
Change /basedir
to the name of the directory containing your dynamic pages. This
will normally be just a "/" unless the pages are in a subdirectory,
in which case it would be a "/" followed by the name of the subdirectory.
-
In the "RewriteRule"
line, after the question mark, change "id" to whichever variable
you're using to pass your product id in order to display the product
detail page. Navigate to the link on your site that displays a product
detail page from your database. Study the URL in your browser. Normally
you'll only have a question mark followed by the product ID. However,
if other parameters exist, you'll need to rewrite those too.
-
In the "RewriteRule"
line, change yourscript.php to the name of your dynamic product
detail page. This will be the script name seen in the URL often
ending in ASP, PHP, or various other extensions. The script name
will normally precede the question mark.
-
If your site
does not use PHP, change "T=application/x-httpd-php" to the MIME
type for the language that you use.
If you have
questions, the technical documentation for the mod-rewrite functions
can be found at: http://httpd.apache.org/docs/mod/mod_rewrite.html
-
Save the
file to your local computer. Name it htaccess.txt.
-
Upload the
file to your Web site in ASCII mode.
-
Configure
your FTP program to display hidden files. Hidden files are files
that start with a dot on UNIX or Linux based operating systems.
Most FTP programs have an option to display hidden files either
on the preference screen or in the settings for the individual FTP
site. Some FTP programs will allow you to add parameters to the
list command that will display hidden files. In that case, the parameter
that you would need to add to the ls command is "-al" (without the
quotes).
-
Test your
FTP program to make sure you can view hidden files. Try renaming
a non-essential file on your Web site to a filename that starts
with a dot. Make sure that it shows up in the directory listing.
Then make sure you can rename it back to its original file name.
-
Rename htaccess.txt
to .htaccess (starts with a dot, and no file extension). This will
activate your changes.
CAUTION:
If you have done something wrong in the htaccess file, your site may
stop working after you rename the file. In that case, rename .htaccess
back to htaccess.txt and the site should start working again. That's
why it's important that your FTP program is configured to see hidden
files.
If you set things
properly, at this point your web server will automatically rewrite
URLs of the format: http://www.yourWeb site.com/yourscript.php?id=123
to: http://www.yourWeb site.com/productid123.htm
The rewriting
takes place behind the scenes, so the URL in the address bar will
always display in the new format, without the question mark.
For Microsoft
IIS Web sites...
If your Web
site runs on the Microsoft IIS server, you can obtain the same basic
functionality I described for Apache Web sites using a program called
ISAP rewrite. In the interest of brevity, I'll simply point you to
the Web site offering more
information on this program.
Read this article
regarding Apache servers and then visit the above link. In this context,
you should understand how you could apply ISAP rewrite to accomplish
the same goal as Apache's mod-rewrite.
Whether you're
running a Microsoft or Apache server, test your changes by browsing
and searching your site to see if it worked. Instead of having URLs
containing question marks followed by the product ID, the product ID
should become part of the page name. Ampersands and commas should also
be eliminated whenever possible. Hyphens, underscores, and periods should
be fine.
Once you have
verified that the change worked as expected, you must go into your site
and change the URLs that point to each of your detail/product pages
so that they conform to the new format. This is critical since these
are the links that the search engines will follow to find your product
pages. If you leave them using the old syntax containing the troublesome
symbols, then the search engines may still avoid spidering the links.
If the bulk of
your dynamically generated pages are not being indexed today, then don't
short-change yourself any longer. Implementing this one change on your
Web server can improve your site's potential visibility by a hundred-fold
or more! The next step is to make sure those detail pages are properly
optimized so that they'll rank well. WebPosition Gold's Page Critic
is the perfect match for this task.
Brad Konia is
the CEO of Market My Site, a leading search engine optimization and
online marketing firm. Market My Site offers a suite of full-service
and self-service solutions designed to increase traffic to your Web
site and convert that traffic into paying customers. Mr. Konia is also
the author of "Search Engine Optimization with WebPosition Gold 2" and
the developer of the official WebPosition Gold 2 Video Tutorial. Visit
Market My Site on the web.
This article is copyrighted
and has been reprinted with permission from FirstPlace Software, the
makers of WebPosition
Gold. FirstPlace Software helped define the SEO industry with the
introduction of the first product to track your rankings on the major
search engines and to help you improve those rankings. A free
trial of WebPosition Gold is available from their Web site.