Question for ACERESEARCHER: Ace, your last answers helped out a ton.
Only, now I no longer have my Google S.E. listing for my photography
portfolio. Kinda of a trade off (www.uscuties.com is now listed, but
my www.onemodelplace.com/photographer_list.cfm/P_ID=4735 is not). I
would like to have both listed, as they somewhat go hand in hand.
My question, however, is, how do I keep the uscuties.com listing
updated? It seems the info crawler(?) has passed me by, as it still
shows a very old page. I made changes to accommodate a higher
standing, but without the periodic Google updating, it hasn't done any
good yet! Do I need to periodically re-submit the site to receive
periodic updating?

'Uncle Sammy'
U.S. Cuties & American Beauties
www.USCUTIES.com

---

Request for Question Clarification by aceresearcher-ga on 05 Mar 2003 01:56 PST

Welcome back, unclesammy, and thanks for requesting me again.

I apologize for all the downtime since you have posted this Question;
hopefully the Answers site difficulties have now been resolved.

Something you should know in case you decide to post more Questions is
that the computerized "administrator" for this site is programmed to
automatically lock any Questions with the magic "G" word in them, for
first review by the Editors (which is what happened to your Question).
It may be anywhere from a couple of hours to more than a day before
the lock gets released and the Question can be Answered by a
Researcher. So, for future reference, if you need to use that word, I
recommend using the spelling G00gle (zeroes instead of o's).

It looks like your site placement in Google's search results has
indeed improved; it comes up 1st for "US cuties" and 3rd for "cuties
beauties", and it's got a Page Rank of 3; that's great. We'll see if
we can improve upon that.

Google usually runs its massive crawl of the web (known as "The
Dance") every 4 to 6 weeks; but it will index sites more often if it
decides that they are worth doing so. Convincing the Googlebot that
your site is one of those worth indexing more often is another thing.

However, there are indeed a couple of things you can do to "invite"
robots to crawl your site.

Let me gather some information for you; then I will come back and post
it as an Answer. I just wanted to make sure you knew that I had
noticed your Question and am working on it.

Thanks,

aceresearcher

Hello again, unclesammy!


One of the things you can do to encourage robots to visit your site is
to upload a robots.txt file to your server at
http://www.uscuties.com/robots.txt

The contents should simply be:

User-agent: *
Disallow: 

Polite Search Engines such as Google, when accessing a site, will
first look for a robots.txt file to see if you have left any
instructions for them on how you would like them to access your site.
The robots.txt file shown above simply says, "Come on in! I don't have
any pages that I want you to NOT index, so take a look at everything!"

If you did have a directory that you didn't want robots to index, your
robots.txt file should look something like:

User-agent: *
Disallow: /logs
Disallow: /misc

where http://www.uscuties.com/logs might be the directory where you
keep your server logs, misc might be a directory of miscellaneous
stuff that you aren't interested in getting listed in search engine
results, and you don't want either of these spidered by robots.

***HOWEVER*** 
Not all Search Engines are as polite as Google, and they may index the
files and directories that you disallow in your robots.txt file
anyway. In fact, some hackers and other nefarious characters actually
troll the web looking for directories disallowed in /robots.txt files,
specifically looking in them for confidential information. /robots.txt
files are just NOT a good way to secure confidential data.


You've done a good job with your <meta> tags for keywords and
description. Now you will want to add a "Welcome" <meta> tag
invitation to robots to spider your site. At the top of your main
page, where you have all your <meta> tags, add the following:
<meta name="Robots" content="index,follow"> 
<meta name="revisit-after" content="10 Days">

When robots access your page, this will tell them that you welcome
them and that you would like them to follow your links and spider your
site. Some robots will also index the "revisit" information and use
that to decide when to look at your site again. There's no guarantee
that the Googlebot will obey your wishes, but you've done everything
you can to encourage it to do so.
(NOTE: 10 days is probably a reasonable revisit time to specify. DON'T
ask robots to visit your site every day. This could cause a huge drain
on your server bandwidth, and it is not likely that the Googlebot
would honor a request to be spidered that frequently anyway.)


Finally, one of the biggest factors Google and other search engines
look at in determining how often to visit a site is how often the site
is updated. Every time you upload a new version of your front page to
the server, the date/time stamp on it is changed, and robots look at
this date/time stamp to decide if the content has changed. For this
reason, even if you have no changes to make to your website, I
recommend that you upload a fresh copy of your front page to your
server every 1-2 weeks, so that this date/time stamp will be updated,
and robots will know that someone is actively maintaining your site.


For information on robots.txt files, I encourage you to check out the
following resources:

The Web Robots Pages
http://www.robotstxt.org/wc/robots.html

Microsoft Knowledge Base Article - 217103  
"How to Write a Robots.txt File" 
http://support.microsoft.com/default.aspx?scid=KB;en-us;q217103


For more information on developing a Google-friendly website, I
recommend that you study the information in Google's Help Department:
://www.google.com/webmasters 
Guidelines 
://www.google.com/webmasters/guidelines.html 
Facts & Fiction (myths dispelled) 
://www.google.com/webmasters/facts.html 
Search Engine Optimization (SEO) 
://www.google.com/webmasters/seo.html 
Frequently Asked Questions (FAQ) 
://www.google.com/webmasters/faq.html 
User Support Discussion Forum 
http://groups.google.com/groups?q=google.public.support.general 
 

Another fabulous resource is the forum at WebmasterWorld.com: 
http://www.webmasterworld.com 
 
and at Search Engine World: 
http://www.searchengineworld.com 
 
I encourage you to visit these sites and learn more about making your
site attractive and friendly to Search Engines.


I hope that this Answer provides exactly the information that you
needed!

Best wishes,

aceresearcher

---

Clarification of Answer by aceresearcher-ga on 08 Mar 2003 02:25 PST

By the way, unclesammy,

Google still does have your One Model Place page in its index (you can
enter the URL in the Google Search box to find out whether this is the
case), it just doesn't have a page rank, so it's probably not going to
come up too high on search engine results.
http://www.onemodelplace.com/photographer_list.cfm?P_ID=4735

It looks like the robots.txt file for One Model Place has not been
properly set up. If you plug "http://www.onemodelplace.com/robots.txt"
into the robots.txt file validator at:
http://www.searchengineworld.com/cgi-bin/robotcheck.cgi
you'll see a list of syntax error messages.

You may want to consider e-mailing the webmaster of One Model Place
and (in a very friendly, polite way) giving them the information that
I have given you about robots <meta> tags, robots.txt files, and the
link for the above validator.

Regards,

aceresearcher

---

Request for Answer Clarification by unclesammy-ga on 13 Mar 2003 16:25 PST

Hey, Ace, thanks for the reply. I guess there is only so much I can
expect, in terms of ranking. Much as I would like a closer spot to the
first page, when 'cuties' is searched for, until a base of members are
linked, there's not much else I can do.

I've done the meta insert, as suggested, so we'll see if that helps
being visited more often.

Wish I had the time and ability to make better use of the other links
you included. I'll keep 'em and when time is allowing....

Til the next time, though, thanks, and I wish you the best.

S-

---

Clarification of Answer by aceresearcher-ga on 13 Mar 2003 16:52 PST

unclesammy,

Thanks again for requesting me!

I DO encourage you to send the information on <robots> meta tags, 
robots.txt files, and the robots.txt validator page, on to the
proprietor of One Model Place. I think those might help his site's
rankings, and by osmosis, yours.

Regards and Best Wishes!

Very thorough!

This might or might not be something you know about, but there's tags
that you insert, either manually or with software, in front of
webpages. They're called meta tags. They tell the search engine
spiders or crawlers what the website contains in a nutshell and some
other kinds of info like how often to check back and etc. You can read
all about this on this website:
http://searchenginewatch.com/webmasters/index.html

(I'd recommend you read several of the articles on that site to find
out more about how search engines work and how web crawlers index
sites and whatnot. It's interesting stuff)

Kris,

Thank you for the 5-star rating and the generous tip. It is my
pleasure to be able to assist you.

Regards,

Ace