INDEX TIME: First, make sure you've allowed enough
time to become indexed. The amount of time to allow is
sometimes listed on the search engine's submission page.
Unfortunately, the engine's own advertised times are often
inaccurate or out of date.
WebPosition's Submitter report and WebPosition's URL
Verification report will both tell you how much time you
should allow for each engine before being concerned about
not being indexed. Average index times often range from one
to eight weeks depending on the engine. Some engines like
AltaVista and Inktomi offer paid options if you wish to be
indexed more quickly.
-
ALREADY INDEXED: Be sure you're not already
indexed but just don't know it. Unfortunately, none of the
major engines are kind enough to e-mail or notify you as
to if and when you've been indexed.
In addition, you cannot simply do a search on a keyword
that applies to your Web site and expect it to pop up at the
top. In fact, you must take pro-active steps to optimize
your pages for each search engine. If you don't, it's very
unlikely you'll find your Web site except on the most
specific of searches.
The method to determine if a page or domain has been
indexed varies from one engine to another, and in many
cases, it's difficult to tell for sure whether your pages
are in fact in there. Never assume that you're not indexed
just because you searched for a bunch of keywords and you
did not come up in the first few pages of results. You could
be there (i.e., indexed) but be buried near the bottom.
In addition, it's not very practical to check the status
of a number of pages on each major engine each week.
Fortunately, WebPosition Gold has a URL verification feature
in the Reporter that makes this process much easier. Each
time you run a mission, it will report which URLs exist and
do not exist in each engine. If you're using WebPosition and
are not finding your URLs after submitting, be sure to see
this page for common pitfalls to watch out for.
-
MISSING PAGE: Make sure you have uploaded the
pages to your site before submitting them. This one will
seem obvious to many people, but submitting a page that
does not exist or submitting with a subtle typo in the URL
is a goof we might all make at one time or another. If
you're using WebPosition's Submitter, there's a checkbox
on tab 2 labeled "Verify that each page exists on Web site
before submitting." This option defaults so that
WebPosition will verify that all your URLs are valid and
actually exist before submitting them. This is important
since not all search engines will notify you if the URL
does not exist when you submit.
-
ROADMAP FROM HOME PAGE: Some engines have been
known to drop pages that cannot be traveled to from the
home page. HotBot has been rumored to do this. You may
want to consider submitting your home page that links
either directly or indirectly to your doorway pages. Think
of your Web site as a series of roads (i.e., links) from
one page to another. If there's no road from your home
page to the page you want indexed, a search engine may
decide the page is unimportant or of low-quality. You
could submit the page directly, but the engine may reject
it or may drop it at a later date when it finds no "road."
-
EXTERNAL LINKS: Some search engines such as Google
and HotBot have been known to refuse to index Web sites
that do not have any other Web sites linking to them. Or,
they may index your home page but refuse to index any
other pages until you achieve at least one or more links
from another domain. Or, they may index you for a while
but then "prune" their database later of all Web sites
that did not achieve any external links within a certain
period of time. However, do not worry! You simply need to
establish some links and when that's done, resubmit both
your pages and the pages that link to you. Once you have
links to your Web site, it becomes much easier to get
indexed, stay indexed, and to achieve top rankings.
To check your link popularity and for tips on how to
increase the links to your Web site see our
free service.
-
FRAMES: If you have content inside HTML frames,
this can cause problems with submissions. For example, the
search engine may index the main content of the page, but
not the surrounding menu frame. Visitors to your site will
then find some information but may not see the associated
menu! It's generally best if you can create non-framed
versions of your pages. You should then submit the
non-frames versions of your pages, which can of course
link to your framed Web site. Alternatively, you can enter
your relevant text within the NOFRAMES area of a framed
page that most search engine spiders will read. However,
don't expect to achieve high rankings while optimizing the
NOFRAMES area. Optimizing a NON-framed page will often
achieve better results.
-
SPIDER BLOCKS: Search engine spiders cannot index
sites that require any kind of registration or password. A
spider cannot fill out a form of any kind. The same rule
applies regarding indexing of content from a searchable
database. That's because the spider cannot fill out a form
to query that database. The solution is to create static
pages that the engines will be able to find and index
without performing a special action on your site.
Depending on the database system you have, there are
utility programs out there that help you do this, as well
as companies that can assist you.
-
FREE SITES: Many engines no longer index pages from
free web sites or they limit the number of pages they will
index from these hosts. Sometimes they will get too many
"junk" submissions from free web site domains such as
Geocities or others. Therefore, some engines choose not to
index anyone for some of these domains. Or, more commonly,
they limit the number of pages they will accept.
It's always best to buy your own domain name (very
important) and place it on a respected, paid service to
avoid being discriminated against. The free traffic you can
generate from the search engines is just too valuable to be
sacrificed for the small savings a free hosting service
provides. In addition, free hosts are often unreliable or
force you to display banners that send valuable visitors
away from your Web site soon after arriving. That can cost
you sales.
-
GUILT THROUGH ASSOCIATION: If your Web site shares
the same IP address as many other Web sites on your host's
Web server, then you may find your IP quietly banned from
something another Web site on the same server did! It's
always best to ask your hosting service if your domain
name has its own unique IP assigned to it. If not, ask
them to move it to its own IP to avoid the potential of
having your submissions ignored because of something that
a site sharing your IP did. We've heard from many people
who tried everything to be indexed only to find it was a
snap once they changed hosting services.
-
SUBMISSION LIMITS: Make sure you're submitting
within the recommended limits. Some engines do not like
more than a certain number of submissions per day for the
same domain. If you exceed the limit, you may find that
all your submissions for that day are ignored.
Fortunately, WebPosition's submitter will warn you
regarding current limits and help keep you within them.
Some submission consultants feel it is dangerous to submit
more than ONE page a day to an engine for a given Web
site. For those who wish to be ultra-conservative in their
approach, the WebPosition Submitter includes a checkbox to
limit submissions to one URL per day per engine.
-
DYNAMIC PAGES: Dynamic pages are often ignored by
the search engine spiders. In fact, any URL containing
special symbols like a question mark (?) or an ampersand
(&) will be ignored by many engines. Pages generated on
the fly from a database often contain these symbols. In
this situation, it's important to generate "static"
versions of each page you wish to be indexed. In regard to
the search engines, the simpler the page is, the better.
Does this mean, for example, having a JavaScript to count
visits to the page will prevent you from being indexed, or
lower your rankings? No. It simply means that the search
engine will most likely ignore the JavaScript and index
the remaining areas of the page. There is evidence that
going too far with fancy scripts and code on a page can
hurt your rankings if the bulk of your page consists of
java or VB scripts.
-
NON-INDEXABLE CONTENT: It's important to know the
types of content that the average search engine cannot
index. Most engines cannot index text that is embedded in
images. Text that appears in multimedia files (audio and
video) will not be indexed. Most engines cannot index
information that is generated by Java applets or in XML
coding.
-
LARGE PAGES: If your site has a slow connection or
the pages are very complex and take a long time to load,
it might time out before the spider can index all the
text. For the benefit of your visitors and the search
engines, limit your page size to 50K or less. In fact,
most Webmasters recommend that your page size PLUS the
size of all your images on the page should not exceed
50K-70K total. If it does, many people on dial up
connections will leave before the page fully loads.
-
DEEP LINKS: If you submit just your home page, don't
expect a search engine to travel more than one or two
links away from the home page or from the page that you
submitted. Over time they may venture deeper into your
site, but don't count on it happening quickly. You'll
often need to submit pages individually that appear
further down into your site or create more direct links
from the home page. This way the search engines can find
them. Visible links are preferable. If its not practical
to provide a short link path from the home page to all
your other Web pages, there are methods to create links
that won't be seen by the average visitor. However, you
should be
aware of the dangers before going that route.
The technique of submitting one page that then links to
multiple other pages you want found is called creating a
"hallway page" or a site map. In many cases, this will not
only get you indexed in cases where they are ignoring you,
but it will often improve your rankings. That's because
many engines assign "bonus points" to pages they find on
their own versus pages that were specifically submitted to
them.
-
UNRELIABLE HOSTS: If your Web site fails to respond
when the search engine spider pays a visit, you will not
be indexed. Even worse, if you are indexed and they pay a
visit when your site is down, you could be removed from
their database! Consequently, it pays to have a reliable
hosting service that is up 99.5% of the time. However, at
some point a spider is going to hit that other 0.5% and
end up yanking your pages by mistake. Therefore, it pays
to keep a close eye on your listings and resubmit when
needed.
-
SPAM: If you have ever used any questionable
techniques that might be considered an overt attempt at
spamming (i.e., excessive repetition of keywords, same
color text as background, or other things that the
WebPosition Page Critic warns you about), an engine may
ignore or reject your submissions. If you're having
trouble getting indexed in the expected amount of time,
make sure your site is spam-free.
-
REDIRECTS: If your site contains redirects or
meta refresh tags, these things can sometimes cause the
engines to have trouble indexing your site. Generally they
will index the page that it is redirecting TO, but if it
thinks you are trying to "trick" the engine by using
"cloaking" or IP redirection technology that it can
detect, there is a chance that it may not index the site
at all.
-
PROPER DIRECTORY SUBMISSIONS: If you're
submitting to a directory site like Yahoo, Open Directory,
Looksmart, or others, then a human being will review your
site. They must decide if the site is of sufficient
"quality" before they will list it. I recommend you read
the submission guide on the Guided Submission tab of the
WebPosition Submitter. It contains tips on how to improve
your chances of obtaining a good listing in these
directories. Getting listed in major directories first can
help you get listed elsewhere.
-
INDEX TIMES CAN FLUCTUATE: WebPosition will tell
you the average index time of each search engine. However,
this is only an average. Sometimes engines will index
sites every 30 days fairly consistently and then suddenly
stop indexing most sites for several months. This can be
frustrating, but it does happen. Generally a major engine
will not go more than three to four months without
refreshing its index.
-
PAGE LIMITS: If you have many pages indexed but
are having a hard time getting new ones recognized, be
aware that there are limits. Each search engine will only
spider so many pages of your Web site. This may range from
a few dozen or three or four hundred depending on the
engine. Some people have even been successful in getting
far more pages indexed depending on the engine. Google is
one engine that tends to crawl deeper into your site.
However, how deep they go may depend on factors like your
link popularity. Sites with higher link popularity are
deemed "worthier" of more thorough indexing.
-
RANDOM ERRORS: Sometimes the engines just lose
submissions at random through technical errors and bugs.
After all, they are managing a database of hundreds of
millions of pages. Therefore, some people like to resubmit
once or twice a month for good merit in case they do lose
a submission. Certainly if you've followed all the "rules"
and are still not listed, by all means, re-submit!
Sometimes a little persistence is all that's needed.
-
LINK FARMS: If you've ever struggled to improve
your link popularity, you may have been tempted by services
offering instant fixes. Many such services are based on a
network of sites that promise to link to you in exchange
for you linking to them. While there's nothing wrong with
exchanging links with other sites, you must be wary of such
services that are considered to be spam by the major
engines. These services are often termed "link farms."
Google in particular has been known to drop Web sites
that participate in certain link exchange networks. If the
links are being exchanged between unrelated sites for the
sole purpose of increasing one's link popularity, then it's
a good bet that Google and potentially other engines may
choose to penalize all the sites participating in the
network.
Therefore, avoid any kind of reciprocal link service
where all members are required to link to a common Web site
or script. If some of the text on the link page and/or the
page name is the same among most or all linking parties,
then watch out. It can become trivial for Google or another
engine to identify all the sites in the linking service and
to drop them from the index, or reduce their ranking. When
reviewing any kind of link building service, be careful to
note whether there's any way an engine could identify the
members of the service.
If you've already joined one of these services and
discovered your site dropped from the index, quickly remove
any "tainted" link pages and then re-submit. If your page
does not appear within the engines normal indexing time,
then you may need to e-mail the engine and ask them to
re-list you.