Google Answers Logo
View Question
 
Q: Robots and Spider ( Answered 5 out of 5 stars,   2 Comments )
Question  
Subject: Robots and Spider
Category: Computers > Internet
Asked by: respree-ga
List Price: $5.00
Posted: 04 Nov 2002 15:00 PST
Expires: 04 Dec 2002 15:00 PST
Question ID: 98716
In general, what is the general rule of thumb or SEO consensus of how
many characters a spider will read while indexing a page.  Does it
read the entire page or stop after it hits a certain amount of
characters.

I have put my keywords in an H1 tag and used CCS so that I don't have
this big ugly font.  For now, it is at the very top of my page, which
I don't like, but I wanted it to be the FIRST thing the spider sees. 
However, it makes my page layout look undesirable (and amateurish). 
Thoughts?  Suggestions? Comments?

Example:
http://www.respree.com/product/pages/sku/LADLEA-SA015
Answer  
Subject: Re: Robots and Spider
Answered By: theta-ga on 05 Nov 2002 09:48 PST
Rated:5 out of 5 stars
 
You seem to have paid attention to the various search engine
optimization tips.
Your meta tags conform to the acceptable standards(no keyword
spamming,reasonably short page description) and your title contains
the relevant page description info.


To answer the first part of your question, the amount of text indexed
by a search engine varies a lot from search engine to search engine,
and also from time to time, as the search algorithms evolve.
For example :
Lycos indexes the title, headings, subheadings and the hyperlinks to
other sites, along with the first 20 lines of text, while AltaVista
claims to index all words, even the articles, "a," "an," and "the."
Some search engines also generate a text excerpt of your page,
containing elements that they believe represent your site content.
So, you should ensure that the lines of text at the top of your page
contain as many relevant keywords as possible.You should also make
sure that any paragraph beginings contain some keywords or phrases
referring to the site content.You can see the article links presented
at the end of this answer for tips on your page structure.

To answer the second part of your question:
What you want to do to optimize your web page, is ensure that enough
relevant words appear near the beginning of the page.
- If you want to stick with your current idea, here is a suggestion.
You can replace the non clickable text in the first line with the 'You
are here:" link in the menu.So, instead of the line '~ Prints: New -
Starry Night 1889 ~ Vincent Van Gogh next item->' , you can show
something like this
 '> Prints > By Genre > Post Impressionism > Vincent Van Gogh > Starry
Night (1889)'
  with all the words except the painting name as hyperlinks.This puts
the relevant words at the top of your page and at the same time
provide users with a useful navigational aid.

- Another problem that your page suffers from is the fact that you are
using tables to display your menu sidebar, and this sidebar appears
before the main body of text. As a result, words like 'home contact us
search shopping cart' etc. appear before the main subject of your
page, which is not what you want.
To see what you can do to remedy this situation, check out the
following page :
      Search Engine Hints: Tables
      ( http://www.delorie.com/web/ses-hint.html )
If you want to see what the text on your page will look like to a
search engine, head over to :
      Delorie.com : Search Engine Simulator
      (http://www.delorie.com/web/ses.cgi )
You can use this site to help ensure that the most important keywords
always appear in the text excerpt generated by the spider.

Relevant Links :
 - Free Search Engine Secrets
   ( http://www.kaleidoscope-dts.com/secrets.html )
   The page contains links to sites that will help you analyze and
optimize your webpage for optimal placement. Definitely worth a read.

 - Search Engine Placement Tips By Danny Sullivan
   ( http://www.searchenginewatch.com/webmasters/tips.html )

 - All About Title Tags
  (http://www.highrankings.com/allabouttitles.htm )

 - Death Of A Meta Tag By Danny Sullivan
   ( http://www.searchenginewatch.com/sereport/02/10-meta.html )
   Meta tags are supported by by only one major crawler-based search
engine - Inktomi

 - WebDevelopers Journal : Web Site Promotion Articles
( www.webdevelopersjournal.com/articles/site_promotion/web_site_promotion_guide.html
)

 - WebMaster T's : SEO Tips
  ( http://www.tsworldofdesign.com/search_engine/ )
 
==============
Hope this helped.
If you need any clarifications, just ask!
:)

Request for Answer Clarification by respree-ga on 05 Nov 2002 14:34 PST
Thank you for your response and for the additional SEO tips and links.

Could you provide me with a 2-3 links on articles discussing (even in
general terms) how many characters the spiders will read.  Based on
your answer ("varies a lot from SE to SE), I'm afraid I wasn't able to
come to a reasonable conclusion as to whether to keep my 'keyword
phrases' at the top of the page, or if I would be hurting myself to
move them to the place I would rather have them.

I realize each engine is different (maybe even drastically), but I was
wondering if there was some kind of 'general concensus' with SEO
experts as to my question about bots will stop reading.

Many thanks again.

Clarification of Answer by theta-ga on 06 Nov 2002 12:21 PST
If you have read the comment provided by concentration-ga and read the
article links give, you will note that most of the modern engines in
use today (AstalaVista,Google,FAST) index the whole page and analyse
all its content to determine keywords which best represent the
content.
These keywords are affected by the following :
 - Word placement : words which appear repeatedly at the top of the
page are considered to be more relevant than words which make their
appearence at the bottom of the page.
 - Page Title - Paragraph/Section Titles : Words which appear as part
of titles are considered more likely to represent the actual page
content.So the page title is important.The engines determine section
titles by the HTML FONT tags. More important titles are usually in
larger fonts while less relevant text is usually in smaller font.

Also, Lycos is the only search engine for which I have found
information limiting the number of lines of text scanned(20 lines).
My recommendation to you would be to make sure that the text appears
2-3 times within the first 150 words on your page.This should be
enough to appease Lycos.
For the other search engine, which are much more popular than Lycos
and are more likely to affect the traffic on your page, you should
concentrate on structuring your document to emphasize keywords with
the proper headings, paragraph beginnings etc.Please see the links in
my orig answer for this.
The major fact which affects your ranking in many engines is the
number of links to your website, and you should concentrate on
cultivating these.
Make sure you submit your site to the DMOZ web directory (
www.dmoz.org ).
This affects your listing in many engines.

In short, I do not think that you should spend more time fretting
about the placement of one line of text on your page.The search
engines of today are too clever to depend on just this one fact to
optimize your page ranking.

As for links : I was able to find no relevant articles discussing the
search engine text index limit.The Lycos/Altavista example was from an
article on SearchEngineWatch( www.searchenginewatch.com ) . See the
links in the orig. answer for more details.
I highly recommend you to read the articles that concertation-ga
pointed out in his comment.Here are the links again :
Anatomy of a search engine -
FAST : ( www.searchenginewatch.com/searchday/02/sd1031-in-fast.html )
GOOGLE : ( www.searchenginewatch.com/searchday/02/sd1030-in-google.html)

I hope the above will help you make a decision.
respree-ga rated this answer:5 out of 5 stars
Wonderful.  Many thanks for your research.

Comments  
Subject: Re: Robots and Spider
From: funkywizard-ga on 04 Nov 2002 15:12 PST
 
i did something similar on my website www.akddr.com. The first
paragraph of text i would prefer not to have where it is, but it is
the highest place in my html that i could put the info, and i wanted
it to be spidered well, so there it has stayed. I look forward to a
definitive answer as much as you do!
Subject: Re: Robots and Spider
From: concertation-ga on 05 Nov 2002 07:12 PST
 
Your title is certainly the most important indexing criteria for
search engines and should be placed as close as possible to the <HEAD>
tag of your page even before your meta tags. Their are a lot of even
free tools on the Web which offer detailed analysis of your page
before submitting it to search engines (addURL). Example for your URL
: Title (78 characters) Meta Description (71 characters) Meta Keywords
(163 characters) URL (53 characters) Texte visible (221 words). Your
title should be informative and represent the contents of your page
for instance "New" is an empty word; Respree is redundant with your
domain name, etc. -  you have to choose your title carefully and not
exceed the 100 characters for 7 - 10 words.  Although each search
engine is specific, modern third generation SEs read and index the
entire page (see: Anatomy of a search engine at
http://searchenginewatch.com/searchday/02/sd1031-in-fast.html and
http://searchenginewatch.com/searchday/02/sd1030-in-google.html).
Moreover search engines base now their rankings amongst other on your
page popularity i.e. link richness between your site and other sites
pointing to your page, the number of times your page is retrieved...

Important Disclaimer: Answers and comments provided on Google Answers are general information, and are not intended to substitute for informed professional medical, psychiatric, psychological, tax, legal, investment, accounting, or other professional advice. Google does not endorse, and expressly disclaims liability for any product, manufacturer, distributor, service or service provider mentioned or any opinion expressed in answers or comments. Please read carefully the Google Answers Terms of Service.

If you feel that you have found inappropriate content, please let us know by emailing us at answers-support@google.com with the question ID listed above. Thank you.
Search Google Answers for
Google Answers  


Google Home - Answers FAQ - Terms of Service - Privacy Policy