I was wondering if anyone had any idea of how often Google..
a) Spiders new sites (in terms of which have been newly submitted to
Google)
b) Spiders existing sites (in terms of re-spidering already indexed
pages and pages which have NOT previosuly been indexed)
c) Refreshes its cache of sites

If you need any clarification, please let me know.

Hi there,

There are actually two different spider processes at Google, the
regular GoogleBot (aka DeepBot) and the FreshBot. The purpose of the
FreshBot is to re-index every day or two sites which have constantly
changing content. It sometimes picks up on new sites which have not
yet made it to the main index.

Here is a FAQ page about the GoogleBot at WebMasterWorld:
http://www.webmasterworld.com/forum3/9213.htm

a) Any new sites get spidered sometime within the month after they are
submitted, which may or may not be before the next major index update.
There appears to be no priority given to new sites.

Sometimes a new site gets noticed by FreshBot and appears in the index
really quickly, often with a good PageRank, because the PageRank has
been estimated, not calculated. The new site can disappear from search
results just as quickly.

b) Existing sites can get spidered at any time within the month
following a major Google update. Basically they are re-spidering every
site they know of, and it takes them most of a month to do so. Some
webmasters report being spidered at roughly the same time each month.

c) The cached copy is a replica of what the GoogleBot saw when it last
visited. There is a way of finding out when that was. Search for it
using Comet Web Search, then click on "archived copy" to see Google's
cache of the page, and a note stating the date of the cache.

Comet Web Search
http://search.cometsystems.com/

Here is a history of when Google has updated its main index:
http://www.webmasterworld.com/forum3/2657.htm

As you can see, it is roughly once a month, and despite much
speculation, there is no way to work out when it will happen next, not
even by following the phases of the moon.


I hope this helps. Information of this type is only an amalgamation of
opinions. Google does not make much information available to the
public, and that includes Google Answers researchers - we are very
seperate from Google the search engine.

Best wishes,
robertskelton-ga