Google Answers Logo
View Question
Q: Google Spider Crawls ( Answered 5 out of 5 stars,   1 Comment )
Subject: Google Spider Crawls
Category: Miscellaneous
Asked by: hellbent-ga
List Price: $2.00
Posted: 04 Nov 2003 20:54 PST
Expires: 04 Dec 2003 20:54 PST
Question ID: 272740
I have removed my robots.txt file from my webroot, now Google can
crawl my site. Will Google crawl my site (
again now the robot file is gone?
Subject: Re: Google Spider Crawls
Answered By: serenata-ga on 05 Nov 2003 11:24 PST
Rated:5 out of 5 stars
Hi Hellbent ~

There are two ways to get a site listed on Google:

1. Submit your site via the add URL page,
://, although there is no guarantee they
will either crawl or add your site -

     "We add and update new sites to our index each time we
      crawl the web, and we invite you to submit your URL here.
      We do not add all submitted URLs to our index, and we
      cannot make any predictions or guarantees about when or
      if they will appear. [From Google's "Submit Your Site"]
   - ://

2. or Google's "best way":

     "The best way to ensure Google finds your site is for your
      page to be linked from lots of pages on other sites.
      Google's robots jump from page to page on the Web via
      hyperlinks, so the more sites that link to you, the more
      likely it is that we'll find you quickly."
   - ://

Even if Google does crawl your site, there is no guarantee it will be
listed in Google's index. Google states, "We DO NOT add all submitted
URLs to our index, and cannot predict when or if they will appear."
   - ://


Removing obstacles which may have been included in your robot.txt file
allows Google to crawl your site unobstructed, but that isn't all you
need to do. There are things you, as a website owner or webmaster
should do which will increase your chances of being crawled - and more
importantly, LISTED, in Google's search results.

Links and PageRank

As mentioned above, links to your site are important and help build
PageRank, which is explained in Google's "Our Search: Google

     "The heart of our software is PageRank?, a system for
      ranking web pages developed by our founders Larry Page
      and Sergey Brin at Stanford University. And while we
      have dozens of engineers working to improve every aspect
      of Google on a daily basis, PageRank continues to provide
      the basis for all of our web search tools."

and its PageRank in "PageRank Explained"

     "PageRank relies on the uniquely democratic nature of the
      web by using its vast link structure as an indicator of
      an individual page's value. In essence, Google interprets
      a link from page A to page B as a vote, by page A, for
      page B. But, Google looks at more than the sheer volume
      of votes, or links a page receives; it also analyzes the
      page that casts the vote. Votes cast by pages that are
      themselves "important" weigh more heavily and help to make
      other pages "important."

      Important, high-quality sites receive a higher PageRank,
      which Google remembers each time it conducts a search. Of
      course, important pages mean nothing to you if they don't
      match your query. So, Google combines PageRank with
      sophisticated text-matching techniques to find pages that
      are both important and relevant to your search. Google
      goes far beyond the number of times a term appears on a
      page and examines all aspects of the page's content (and
      the content of the pages linking to it) to determine if
      it's a good match for your query."
   - ://

PageRank is only one factor in getting your site crawled and indexed.
Google offers additional recommendations in its Guidelines which can
be found here:
   - ://

I have also included some other information to help get your site
listed below. If you follow the guidelines and suggestions contained
in these pages, your chances are greatly increased of getting listed.

Important Google Links For Your Reference

Here are some important links from Google's Webmaster Information and
Webmaster Guidelines. Webmasters who follow the guidelines and avoid
Google's "Thou shalt nots" usually have no problem getting listed and
showing up under the search terms they desire.

   * How Do I Get My Site Listed on Google?
     - ://

   * My Web Pages Are Not Currently Listed (a good 'primer'
     on how and why Google works so well)
     - ://

   * PageRank Information (covers both Google's PageRank and
     - ://

   * Webmaster Guidelines (contains both the dos and don'ts)
     - ://

   * Google Facts & Fiction (can  you buy your way to a
     high ranking in Google?)
     - ://

   * Search Engine Optimizers (some good advice on what to look
     for if you're going to hire a Search Engine Optimizer)
     - ://

   * Frequently Asked Questions (pretty much what it says, but
     definitely worth wading through)
     - ://

Other Sources of Information

There is also good information from many of the top search engine
optimization experts, such as

   * Detlev Johnson, Search Engine Guide

   * Danny Sullivan, Search Engine Watch

   * Jill Whalen, High Rankings

   * Shari Thurow, Web Pro News (and quoted all over the Internet)

who all have columns or newsletters to which you can subscribe and
keep abreast of the best way to use good content for better
positioning in search engine results.

In addition, Webmaster World - - has
discussion boards on most of the search engines. While some of the
discussions are anecdotal and/or questions for information, there is
usually enough discussion to keep abreast of what seems to be

There is an entire section devoted to Google at:

It never hurts to keep track among these discussions, but remember,
trying to optimize for search engines only is like trying to hit a
moving target. You'll notice among the more experienced contributors
to the discussions - plus the SEO experts listed above - that there
really is no substitute for content, relevant links and good HTML.


To answer your question, removing the robots.txt and submitting your
URL may get your site crawled by Google; but even if the site is
crawled, there is no guarantee it will be listed under the terms you
want in Google's index.

Following Google's recommendations in its Webmaster Information and
Guidelines, and getting relevant links TO your site to help build your
PageRank can help your chances of getting your site listed.

It is Google's intention to deliver relevant content in answer to a
search query, and it is your responsibility to make sure your site has
content which can both be indexed and be included under the search
terms you desire.

Search strategies -

I relied on information available on Google's site about its search
technology, Information for Webmasters and Guidelines plus other
bookmarked information used in the regular course of business.

Good luck with your site submission,

hellbent-ga rated this answer:5 out of 5 stars and gave an additional tip of: $5.00
Very Complete Answer. Thank you very much.

Subject: Re: Google Spider Crawls
From: serenata-ga on 07 Nov 2003 19:40 PST
Hi again Hellbent ~

Thank you very much for the rating and the tip. That was a very
generous thing for you to do.

Thanks again,

Important Disclaimer: Answers and comments provided on Google Answers are general information, and are not intended to substitute for informed professional medical, psychiatric, psychological, tax, legal, investment, accounting, or other professional advice. Google does not endorse, and expressly disclaims liability for any product, manufacturer, distributor, service or service provider mentioned or any opinion expressed in answers or comments. Please read carefully the Google Answers Terms of Service.

If you feel that you have found inappropriate content, please let us know by emailing us at with the question ID listed above. Thank you.
Search Google Answers for
Google Answers  

Google Home - Answers FAQ - Terms of Service - Privacy Policy