I'd like to find a way to have Google spider my website:
http://www.sisterstreet.com .. I have a lot of people linking to the
site, however, since the entire site is dynamically generated, spiders
don't like it. Is there a way to either:

a) Dump to static pages for google spider to index while allowing
humans to see the dynamic content? In that case, how would I configure
Apache to do that.

b) Tell the spider that "it's REALLY ok" to index the dynamic content.
Promise!

Thanks!

---

Clarification of Question by sisterstreet-ga on 04 Dec 2002 10:31 PST

I don't want to reprogram the site to better use mod_perl (as I've
seen Scoop sites do to get around spidering issues). I do want to not
effect the end-users.. My guess is some sort of mod_rewrite commands
that would direct the googlebot to a static page.. or, a mod_rewrite
command that will change the standard

http://www.sisterstreet.com/cgi-bin/boards/board_show.pl?bid=16

to something like

http://www.sisterstreet.com/cgi-bin/boards/board_show.pl/bid=16

without hurting anything else.

Thanks

What I would say is don't just have googles spider find your site,
advertize on googles site as a "AdWord"
https://adwords.google.com/select/?hl=en

This would be the option I would pick I think, it would make it so
that you would have even better advertizing then just in with everyone
else.


Hope this helps :)

Dear sisterstreet-ga,

There's two things to keep in mind when trying to get spiders to index
your dynamically-generated sites.

The first thing is that spiders follow /links/. Regardless of whether
the links lead to static pages, or to dynamically-generated ones, the
spiders and bots request the pages behind those links and index them.
So it's not the dynamic nature of your site that's an obstacle. It's a
question of site organization. What does this mean? Simple -- provide
an 'Archives'-type link somewhere on the front page. Make sure all the
posts, or dynamic pages, on your site are accessible through the
archives (a common way to archive is by year/month, or by topic). When
a spider hits your web site, it will follow the Archive link, and
crawl and index all of that information.

The other thing to keep in mind is the frequency of 'spidering'. How
often will the Googlebot and its ilk re-index your site, since it's
dynamically changing? This depends on each individual search engine's
policy, and there's not much you can do to change this or speed this
up, as far as I know. Just be aware that new content will take several
days or weeks to make it into the indices of the search engines.

Hope this helps!