Google Answers Logo
View Question
 
Q: Listing of Dynamically Generated Web Pages on Google ( Answered,   2 Comments )
Question  
Subject: Listing of Dynamically Generated Web Pages on Google
Category: Computers > Internet
Asked by: abe-ga
List Price: $35.00
Posted: 24 Jun 2003 13:58 PDT
Expires: 24 Jul 2003 13:58 PDT
Question ID: 221283
Our online store Beadstore.com is having trouble getting our items
listed on Google.  All of our plain html pages return search results
just fine -- however, our dynamically generated item pages do not seem
to register with Google.

For instance: a search for the statement "Millefiore & Fancy Venetian
Trade Beads (Under $150)" returns this page:
http://www.beadstore.com/african.html.  However a search for this
statement: "Long strand of uniform small trade bead strands" does not
return item # trade 87 on this page:
http://www.beadstore.com/dwos-bin/catalog?nccust=beadstore&page=BEADEDART&Category=Mille&limit=10&sortby=Name

Google states in its policies that it does index dynamically created
pages.  My ISP has assured me that they allow searching by bots. 
Answers suggesting packages guaranteed to improve listings or
promising submission to 10,000 search engines will be rejected.  So
will answers telling me to use Google's web page  submission page
(that's been tried and I will not spam them).  Many of these pages
have been up for 1+ years and still have not been indexed.  Again,
please note that I am not looking for help in improving my listing
position or anything else – only that my item pages get listed by
Google.

At the point I just do not understand what the problem is – I'm
assuming there is a technical fix, but credit will also be provided
for clearly and unambiguously diagnosing the problem.  Please assume
some minimal computer competence, but very little knowledge of
servers, etc.

Thanks for your help,
Abe
Answer  
Subject: Re: Listing of Dynamically Generated Web Pages on Google
Answered By: robertskelton-ga on 24 Jun 2003 16:53 PDT
 
Hi there,

As independent contractors, none of us at Google Answers are privy to
the inner workings of the Google search engine. Unless a direct answer
can be had from a Google search engineer, the best help anyone can
give you is an educated guess.

Having said that, this is from Google's guidelines:

Reasons your site may not be included
-------------------------------------
Your pages are dynamically generated. We are able to index dynamically
generated pages. However, because our web crawler can easily overwhelm
and crash sites serving dynamic content, we limit the amount of
dynamic pages we index.
://www.google.com/webmasters/2.html

Obviously they have made a choice not to index your dynamic pages.
Factors they possibly use to determine this are:

- PageRank of the home page 
- Number of parameters in the URL
- Session IDs in the URL


Your PageRank is 4. When showing results for backward links to a web
page, pages with a PR of less than 4 are ignored. One would expect
that the minimum PR required to have your dynamic pages indexed would
be at least 4, and quite possibly higher. A site like yours, with a PR
of 5, has had dynamic pages indexed by Google:
www.shipwreck-beads.com

None of the other sites in the same category as you have had their
dynamic pages indexed:
http://directory.google.com/Top/Shopping/Jewelry/Supplies/Beads/S/?il=1

The above site also has less parameters in the URL. According to
GoogleGuy (a Google employee and the closest you will get to an
official Google answer):

"We're getting better on dynamic pages every month thanks to better
analysis. I think we crawl dynamic pages better than any general
search engine at this point.."

"In general, it's still a good idea to keep the number of parameters
short. But we are getting better over time"
http://www.webmasterworld.com/forum3/12370.htm

Doing a search for sites that include the parameter "nccust"...

://www.google.com/search?as_qdr=all&q=allinurl%3A+nccust%3D

...I find the only dynamic pages indexed are from netstores.com - so
you are far from alone. It is possible that Google has decided
"nccust=beadstore" is an end-user customer ID, and has tried to visit
pages after removing this parameter - which doesn't work.

So what is the answer to your problem? Another discussion at
WebMasterWorld discusses the possibilities, but there does not appear
to be a guaranteed solution:

Tips for Helping Google Index Dynamic Pages
http://www.webmasterworld.com/forum3/6784.htm

This search brings up many more discussions on the topic:

Google Search: site:webmasterworld.com google dynamic mod rewrite
://www.google.com/search?q=site%3Awebmasterworld.com+google+dynamic+mod+rewrite


Best wishes,
robertskelton-ga
Comments  
Subject: Re: Listing of Dynamically Generated Web Pages on Google
From: waynebe-ga on 24 Jun 2003 14:09 PDT
 
You need to remove the ? from your page.  This ? indicates that it is
a dynamic page.  However, this requires a coding change for your web
site, which can only be done by someone technical.  Please see:

http://asp101.com/articles/wayne/extendingnames/default.asp

However, the code here will not work with your page, since it is most
likely Apache, Perl or CGI.

-WAyne
Subject: Re: Listing of Dynamically Generated Web Pages on Google
From: hammer-ga on 24 Jun 2003 15:25 PDT
 
Actually, since you include the link with all parameters statically on
your page, Google should try to follow it. I believe that the problem
is that your link has too many parameters. I think the maximum that
Google will index for a dynamic link is two parameters.

- Hammer

Important Disclaimer: Answers and comments provided on Google Answers are general information, and are not intended to substitute for informed professional medical, psychiatric, psychological, tax, legal, investment, accounting, or other professional advice. Google does not endorse, and expressly disclaims liability for any product, manufacturer, distributor, service or service provider mentioned or any opinion expressed in answers or comments. Please read carefully the Google Answers Terms of Service.

If you feel that you have found inappropriate content, please let us know by emailing us at answers-support@google.com with the question ID listed above. Thank you.
Search Google Answers for
Google Answers  


Google Home - Answers FAQ - Terms of Service - Privacy Policy