I take care of a popular website, Chabad.org. It is a content-rich
website with thousands of articles. It is the "headquarters" website
for Chabad Worldwide, a non-profit organization.

We also have 150 affiliate sites. The affiliate sites are websites for
the local chapters of the organization.

All content (articles) is handled centraly in a Content Management
System (CMS).

My problem is as follows: Since these affiliate sites were launched 8
months ago, google has only indexed the home page of these affiliate
websites and no other pages, the most importatn one being the
articlecco.asp file. Now, the important point here is that local
articles are accessed by a file called articlecco.asp. A careful
analysis of our web logs revealed that Articlecco.asp is not even
touched by googlebot. This is not good and my question is why not and
what can we do to make it available.

I know that you will find sections in the the affiliate sites that
have the NOINDEX meta tag in them. That is ok. Those sections are NOT
calling articlecco.asp.

Here are some affiliate sites you can look at:
www.chabad.ca
www.chabadtexas.org
www.chabadofcommack.com
www.chabadwestside.org

So you should know: all links in the site are to an
"article.asp?(number)", and that file redirects to the proper
template, either default.asp or articlecco.asp etc.

I was thinking that it could be session thing, because our site works
with session variables. But I was informed that if the client cannot
set the session the server will. Also, I tried a text based browser,
lynx, disabled session, and I was able to browse the site properly.

Also: All articles get indexed by google in the headquarters site,
Chabad.org. Articlecco.asp only applies to the local sites and is not
used at all in chabad.org.

We do not have any robot.txt file.

Looking forward to your response.

Shalom Moishier -

Please understand that Google Answers Researchers are not employees of
Google, and we are not privy the the workings of the Google search
engine rankings. Google closely guards its search algorithms, so it
would be hard to give you aspecific reason for your problems with your
affiliate sites. I think you have really answered your own questions
yourself, though, and what I offer are educated guesses and I will
make some suggestions.

I would direct your attention to Google's Design and Content
Guidelines, where specific recommendations are made to enhance your
listing(s) in Google.

1. You mention the use of dynamic pages, and Google specifically
adfdresses the use of dynamic pages, saying, "If you decide to use
dynamic pages (i.e., the URL contains a '?' character), be aware that
not every search engine spider crawls dynamic pages as well as static
pages. It helps to keep the parameters short and the number of them
small."
[ - ://www.google.com/webmasters/guidelines.html ]

2. In addition, there are recommendations to "Make sure that your
TITLE and ALT tags are descriptive and accurate." I notice that you
don't use ALT tags at all.
[ - ://www.google.com/webmasters/guidelines.html ]

3. You also state, "our site works with session variables..." Google
addresses this issue, too, "Allow search bots to crawl your sites
without session ID's or arguments that track their path through the
site. These techniques are useful for tracking individual user
behavior, but the access pattern of bots is entirely different. Using
these techniques may result in incomplete indexing of your site, as
bots may not be able to eliminate URLs that look different but
actually point to the same page."

============

Recommendatons:

Recommendations:

1. Make sure your page meets as many of Google's guidelines as
possible. Google's guidelines are at:
  - ://www.google.com/webmasters/guidelines.html

2. There is an excellent information page on search engines and search
engine rankings by one of the Internet's foremost Search Engine
experts, Danny Sullivan. This is Search Engine Watch,
  - http://www.searchenginewatch.com/

which has information, news, and other helpful hints to help you
understand listing and rankings in the various search engines. Whether
or not you subscribe, using the site for informational purposes can
and will help you achieve your goals.

================
Search terms:

 - Google listings
 - Google webmaster information


I hope the above addresses your concerns and that you'll soon be on a
more comfortable footing!

Warm regards,
Serenata

---

Request for Answer Clarification by moishier-ga on 20 Mar 2003 18:33 PST

Hi Serenata,

I thank you for your response, but unfortunately it is inadequate.
There seems to be something fundametaly wrong. Adding ALT tags will
not help me. The proof to that is that virtually the whole Chabad.org
site is indexedincluding all the types of templates, yet the
articlecco.asp template was not indexed at all, not even once!! There
is something missing, and something important! I am not sure what it
is and that is why I asked.

I am not happy with your answer.

---

Clarification of Answer by serenata-ga on 20 Mar 2003 19:45 PST

Oops!
I posted my clarification in 'Comments' below.
Serenata

---

Clarification of Answer by serenata-ga on 20 Mar 2003 20:06 PST

One additional bit I found for  you on Search Engine Watch's site:

Here is some further authority about your dynamically generated pages:

"Generating pages via CGI or database-delivery? Expect that some of
the search engines won't be able to index them. Consider creating
static pages whenever possible, perhaps using the database to update
the pages, not to generate them on the fly. Also, avoid symbols in
your URLs, especially the ? symbol. Search engines tend to choke on
it."
"Search Engine Placement Tips," Danny Sullivan, Updated October 14,
2002.
[ - http://www.searchenginewatch.com/webmasters/tips.html ]

If you want to get these pages indexed, there are some discussions of
how to generate dynamic text on a static page, which may help you get
them indexed.

I hope this helps,
Serenata

Did you see the reference to No. 1???

The dynamic pages, ""If you decide to use dynamic pages (i.e., the URL
contains a '?' character), be aware that not every search engine
spider crawls dynamic pages as well as static pages."

and the reference to the session variables? "Allow search bots to
crawl your sites without session ID's or arguments that track their
path through the site. These techniques are useful for tracking
individual user behavior, but the access pattern of bots is entirely
different. Using   these techniques may result in incomplete indexing
of your site, as bots may not be able to eliminate URLs that look
different but actually point to the same page."

I was certainly NOT suggesting the addition of alt tags is the sole
answer ... but in this case it would also enhance what you are trying
to say, especially on those affiliate pages using dynamic site feeds
(the ? references in the URL) and the sessions variables will not help
getting the bot to follow those links.

I also recommended looking at Danny Sullivan's pages, because his
immense wealth of knowledge can most assuredly help you with your
site's design and with your Google (and other search engines as well)
listing.

As I also stated, I am not privy to the algorithms, but I doubt Google
would mention the dynamic links and the sessions variables with
instructions on how to get listed and what to avoid if there wasn't a
problem with it.

Consider the alt tags leaving out some other knowledge the bots are
looking for.

Once again, that link for the Guidelines and the technical
explanations that Google has for Webmasters is at:

- ://www.google.com/webmasters/guidelines.html 

Under the header "Design and Content Guidelines"

Warm regards,
Serenata