Google Answers Logo
View Question
 
Q: Search engine indexing question ( Answered 5 out of 5 stars,   0 Comments )
Question  
Subject: Search engine indexing question
Category: Computers > Internet
Asked by: mvuijlst-ga
List Price: $25.00
Posted: 12 Jun 2003 06:36 PDT
Expires: 12 Jul 2003 06:36 PDT
Question ID: 216423
Why am I not listed in Google? (NO!! It's not what you think! It's not
one of *those* questions, this is an honest-to-goodness bona fide
technical non-commercial really serious question :)

Clarification of Question by mvuijlst-ga on 12 Jun 2003 06:43 PDT
The situation is the following: I've a personal website at
http://www.waterwijk.be. This website got indexed incredibly fast
(within a day or so of being put on-line), probably because I plugged
it on my weblog at http://blogs.salon.com/0001236.

However: a day or two later, it had completely vanished from Google.
And I am at a loss as to why it's gone.

The content is non-commercial and relevant to the topic, links on the
site don't rely on javascript, there's no weird or exotic headers,
there's no spoofing or cloaking going on, the content is changed daily
(by hand), the URLs are clean, ...

Under "normal" circumstances a search for "waterwijk", and certainly
"waterwijk gent" (the last bit being the name of the city, not an
abbreviation of gentleman :) should bring up www.waterwijk.be. And it
did, for a day or two. It still does at msn.com (#3 at
http://search.msn.com/results.asp?q=waterwijk today). But not at
Google.

My question is: is there something technically wrong with the site
that would make Google (and presumably other search engines too)
remove it from their indexes?
Answer  
Subject: Re: Search engine indexing question
Answered By: robertskelton-ga on 12 Jun 2003 13:25 PDT
Rated:5 out of 5 stars
 
Hi there,

I have visited your site (well done, it looks great!) and I can find
no reason, technical or otherwise, for it not to be indexed. The good
news is that you have done nothing wrong, and your site will most
likely appear permanently when Google has its next major index update.

The cause of being indexed for just two days is Google's FreshBot.

There was a time when you just submitted your website's URL, waited a
month or two, it appeared and that was that. These days the process
can be more complicated, especially when it comes to Google.
   
Freshbot   
--------   
   
When you submit your URL to Google, some time in the next month or so
the Googlebot will pay your site a visit and index most or all of it.
When the entire main index is refreshed (used to be the first week of
each month, but it is going through an unpredictable phase at present)
your site should appear and stay there. Sometimes you have to wait
until the following main index refresh for it to appear.
   
Google also has Freshbot - it looks for websites which typically have
constantly changing content (like weblogs), and re-indexes them every
day or two. If you ever see any Google search results with a date
beside them, that is when Freshbot last visited them. Sometimes
Freshbot will unintentionally find other sites by following a link...
   
It appears that www.waterwijk.be was found by the FreshBot, and is now
going through an unstable period which often occurs to new sites found
by FreshBot. One of the most common queries I receive here (and
elsewhere) is why does my site appear, then disappear? The following
comes from GoogleGuy, a Google search engineer who often answers
queries posted at WebMasterWorld:

   
GoogleGuy: Everflux and Fresh Crawls   
------------------------------------   
   
If your site is new, or hasn't shown up in Google for long, it may be
because our "fresh crawl" (which runs each day) was finding your site
instead of our main crawl (which runs about once a month). Our "fresh
crawl" is a newer feature, and we're still experimenting with which
pages to crawl, how deeply to crawl, etc. We even reserve the right to
(gasp!) not do a fresh crawl on some days because we're doing tests or
reviewing new code. Someone wrote in recently and said "my site got in
Google three weeks ago, and you've dropped me four times!"  Nope, it's
just that we don't always crawl the same pages in our fresh crawl, and
we don't always crawl to the same depth. As we do a full crawl of the
web, we find most of the sites from our fresh crawl and put them in
our regular index. My advice on our fresh crawl is to view it as a
nice "bonus" on top of Google's deep index. Users can always search
our full index, but sometimes we can serve up even fresher pages as an
extra nicety.
   
The rest of the discussion is at:   
http://www.webmasterworld.com/forum3/6339.htm   
   
Basically, if your site is picked up by FreshBot one day, but not the
next, the site's listing reverts to its normal listing. If your site
didn't previously have a regular listing, it disappears. Here are some
more opinions at WebMasterWorld:
   
Google also has what we call the fresh-bot, also nicknamed "minty". It
crawls much more frequently, but not as deeply. It is used to update
pages in the index which change often. After a freshbot crawl you site
may appear for a few days with an updated listing, called a "fresh
listing". However, if the freshbot does not return within a few more
days, your "fresh listing" will disappear, and your listing will
revert to whatever it was during the previous deep-crawl, meaning the
one at least one month ago. If you have never had a "previous" deep
crawl, then your site may disappear. If the fresh bot returns to your
site, your fresh listing may re-appear. Therefore, during the first 30
to 59 days of a site's life, it may drop in and out of Google's
results page.
http://www.webmasterworld.com/forum3/6743.htm   
   
When we do a new fresh crawl, it currently replaces the last fresh
crawl. And again, we do experiments from time to time, so different
crawls may visit different pages.(GoogleGuy)
http://www.webmasterworld.com/forum3/6339.htm   
   
My site is new and isn't in the Main Index yet, but was in for a day,
due to fresh bot crawl 3 days ago. freshbot came back this morning
(whew), grabbed some pages, but I'm nowhere to be found... The point
is that pages found by freshbot have an unsure life in Google's index.
http://www.webmasterworld.com/forum3/5196-3-15.htm   
 
 
   
Search Strategy: searching WebMasterWorld for "freshbot"   
   
   
This is a huge subject, with thousands of people guessing at what is
going on, and rare happy instances of someone like GoogleGuy giving
semi-official answers.
   
 
Best wishes,   
robertskelton-ga
mvuijlst-ga rated this answer:5 out of 5 stars
Thanks for this!

Comments  
There are no comments at this time.

Important Disclaimer: Answers and comments provided on Google Answers are general information, and are not intended to substitute for informed professional medical, psychiatric, psychological, tax, legal, investment, accounting, or other professional advice. Google does not endorse, and expressly disclaims liability for any product, manufacturer, distributor, service or service provider mentioned or any opinion expressed in answers or comments. Please read carefully the Google Answers Terms of Service.

If you feel that you have found inappropriate content, please let us know by emailing us at answers-support@google.com with the question ID listed above. Thank you.
Search Google Answers for
Google Answers  


Google Home - Answers FAQ - Terms of Service - Privacy Policy