Thanks for asking.
Yes, there's a way to exclude certain content (customer's pages) from
your Google website listings, and I'll explain the process. However,
in fairness to your current search results listing and site rank, I
will also explain why this might not be desirable in terms of your
site's current, overall Google ranking.
As our standard disclaimer, please note that as independent
contractors, Google Answers Researchers do not speak officially for
Google itself. We can, however, offer insight into webmaster use of
the search engine and PageRank system, based upon our own experiences,
observation and study.
Keeping the Googlebot Away
----------------------------------------------------------------------
The Googlebot is a well behaved search engine spider. It follows the
instructions and restrictions listed in the robots.txt file.
Robots.txt is a special file of instructions that is placed in the
root directory, i.e. http://www.doitgulfcoast.com/robots.txt. The
robots.txt file tells search engine spiders where they can "legally"
look for files (pages) to download and index.
The robots.txt file consists of records. Each record has two fields: a
User-agent line, and one or more Disallow lines. The robots.txt file
should be created in a simple text editor (like Notepad) in Unix line
ender mode. Do not attempt to create a robots.txt file in an HTML
editor.
Google's User-agent is the Googlebot. To make a Google specific
record, your first record line would be:
User-agent: googlebot
The second part of the record consists of Disallow: directives. You
may specify directories, or files. You may use * as a wildcard.
Disallow: /customers/
or
Disallow: /customers/xyz/index.html
or
Disallow: /customers/b*
The following tutorials describe the robots.txt file creation process
in detail, with numerous examples, and also explain how avoid common
mistakes.
Robots.txt Tutorials
--------------------
SearchTools - Search Indexing Robots and Robots.txt
http://www.searchtools.com/robots/robots-txt.html
SearchEngineWorld - Robots.txt File Tutorial
http://www.searchengineworld.com/robots/robots_tutorial.htm
Once you've created your robots.txt file, you may use this online
Robots.txt Validator tool to check your work and verify that all your
file syntax is correct.
SearchEngineWorld - Robots.txt Validator
http://www.searchengineworld.com/cgi-bin/robotcheck.cgi
META Tag Exclusions
-------------------
The Googlebot also obeys robot exclusion instructions coded into the
META tags in the <HEAD> section of individual HTML pages.
Example:
<HTML>
<HEAD>
<META NAME="ROBOTS" CONTENT="NOINDEX,NOFOLLOW">
<META NAME="DESCRIPTION" CONTENT="This page ....">
<TITLE>...</TITLE>
</HEAD>
<BODY>
Using NAME="ROBOTS" will cause the instructions to be applied to all
robots that recognize and use these META tag directives. If you wish
the instructions to be Google-specific, use NAME="GOOGLEBOT" instead.
There are four directives that may be used in a robots META tag.
The CONTENT field of the meta tag may contain:
index,noindex,follow,nofollow
The directives are separated by commas. Contradictory directives
should be avoided (i.e. index and noindex in the same CONTENT field.)
-- INDEX tells the robot it is ok to index the page.
-- NOINDEX tells the robot it is NOT ok to index the page.
-- FOLLOW tells the robot it is ok to follow links on the page.
-- NOFOLLOW tells the robot it is NOT ok to follow links on the page.
Typical Examples:
<META NAME="ROBOTS" CONTENT="INDEX,FOLLOW">
<META NAME="ROBOTS" CONTENT="NOINDEX,NOFOLLOW">
You'll find further resource information and examples at:
SearchEngineWorld - Using a Robots Meta Tag
http://searchengineworld.com/metatag/robots.htm
HTML Author's Guide to the Robots META tag
http://www.robotstxt.org/wc/meta-user.html
Remember that these type of changes to not take effect immediately,
even if you submit the pages for a new crawl. The changes are made
when Google republishes the index, usually about once a month. Google
offers direct information and links about Googlebot behavior at:
Google Information for Webmasters
://www.google.com/webmasters/3.html#B3
Pros and Cons
----------------------------------------------------------------------
By excluding customer pages, you'll neaten up your listings in the
search results. Only your homepage and directory pages will be
displayed.
But... that might not help your overall ranking. When the pages are
excluded from the Google index, any content on those pages is also
"subtracted" from your site's current relevancy for hundreds of
keywords. Site relevancy is also, in part, dependent upon the size of
the entire site. Subtracting 150 of your site's pages may cause well
cause a drop in the rankings of the remaining pages, unless their
relevancy is sheltered by inbound linking from other sites and pages
with high PageRank of their own. Your directory pages do not currently
have that type of linkage, and their keyword density is all over the
map.
If you do exclude customer pages, you might wish to add add
descriptive content to the individual category listings, or break up
the category pages into individual categories, rather than listing
several categories on the same page. A keyword density analyzer will
help you focus your text on desirable phrases.
SearchEngineWorld - Online Tools - Keyword Density Analyzer
http://www.searchengineworld.com/cgi-bin/kwda.cgi
One more caution. Websites are often penalized for using optimization
techniques that Google frowns upon. Keyword stuffing uses repetition
of keywords and phrases in an attempt to artificially raise keyword
density, and thereby improve a page's ranking. Keyword stuffing, both
by its position on a page, and lack of the grammatical elements of
normal speech/text is quite easily detected by search engines. Such
pages are penalized by Google, and severe abuse can result in being
banned from the Google index.
Answer Strategy
----------------------------------------------------------------------
My answer was compiled from personal bookmarks and knowledge of search
engine optimization, earned as webmaster and web developer, working in
tandem with SEO specialists on website development.
If anything I've said is unclear or if you (egads!) discover a broken
link, please, let me know. I'll be happy to make it right.
---larre |
