my website http://www.sohodecor.com  was up and running since Oct
2003, during the Chrismas season, I have change the entire product
line to be Chrismas ornament products. Google at that time was able to
index my entire site about 400+ pages. I can use the product name that
shows on title to find the specific page on Google. after Chrismas, I
spend like 1.5 months to reconstruct entire site to be spring/summer
product line. However, since the update, Google only index my top page
http://store.sohodecor.com ( it really should be
http://www.sohodecor.com  although those two look the same but it is
reading different from different site, I make it the same), and won't
spider further to index my entire site about 728 pages. now I become
helpless that all of those hard word of creating description for each
product become useless because google won't go there...  those pages
that are still on the Google index are old pages that didn't exist
anymore and were marked Supplemental Result .

Few days ago I try to add my site to some garden related directory,
and add their links to my infomation page at
http://store.sohodecor.com/info.html , originally this page was on the
index along with store.sohodecor.com , now it disapear. don't know if
Google think I was doing link spam ? or I linked a bad site that
decrease my ranking ?  I know only keep those link that has high
Pagerank on info page.  but still, Google won't index it anymore.

I have been trying to add my site to DMOZ, but still in waiting list.
other paid inclusion site I used like inktomi are expensive. Does
Yahoo directory worth to try? will it make help Google to index my
entire site again and delete the old pages?

Now Google seems visit my site every 2 days( from the date info in
result page), should I manuly submit all the sub pages ?

Thank you for the help it's greatly apperciated. 

Phil

Hi Sohodecor ~

I can see that you're having problems with your listings on Google,
and I know how frustrating that can be.

Let's start with the double listings:

There are two different listings, apparently with the same content.
One is the yahoo store, http://store.sohodecor.com
and the other is your domain, http://www.sohodecor.com.


Both can be found listed in Google's index by searching for the URL:

1. Search for: store.sohodecor.com -
------------------------------------- 

   "Showing web page information for store.sohodecor.com

    SohoDecor.com -- Ornaments for your Home and Garden --
    Ornament | Garden Decor | Ceramic Ball Ornament | Holiday
    Bouncy Ornament | Bella Ornament Light Catcher Ornament |
    Rustic Ornament | Ornament Wind Chime ...

    Google can show you the following information for this URL:

    * Show Google's cache of store.sohodecor.com
    * Find web pages that are similar to store.sohodecor.com
    * Find web pages that link to store.sohodecor.com
    * Find web pages that contain the term "store.sohodecor.com"
  - ://www.google.com/search?hl=en&ie=UTF-8&oe=UTF-8&q=store.sohodecor.com&btnG=Google+Search


Clicking on "Find web pages that link to store.sohodecor.com" did not
match any documents.
  - ://www.google.com/search?hl=en&lr=&ie=UTF-8&oe=UTF-8&q=link:DGaYb35pinsJ:store.sohodecor.com/

Clicking on "Find web pages that contain the term
"store.sohodecor.com" returns 257 pages, most of which are Winter
and/or seasonal ornaments, and those pages all return to your first
page.


2. Search for: www.sohodecor.com -
-----------------------------------

    "Showing web page information for www.sohodecor.com
     SohoDecor.com -- Ornaments for your Home and Garden --
     Ornament | Garden Decor | Ceramic Ball Ornament |
     Holiday Bouncy Ornament | Bella Ornament Light Catcher
     Ornament | Rustic Ornament | Ornament Wind Chime ...

     Google can show you the following information for this URL:

    * Show Google's cache of www.sohodecor.com
    * Find web pages that are similar to www.sohodecor.com
    * Find web pages that link to www.sohodecor.com
    * Find web pages that contain the term "www.sohodecor.com"
  - ://www.google.com/search?hl=en&lr=&ie=UTF-8&oe=UTF-8&q=www.sohodecor.com&btnG=Google+Search


Clicking on "Find web pages that link to www.sohodecor.com shows no returns.
  - ://www.google.com/search?hl=en&lr=&ie=UTF-8&oe=UTF-8&q=link:GNGWS8ek52kJ:www.sohodecor.com/


Clicking on "Find web pages that contain the term "www.sohodecor.com"
returns 9 pages.
_________________________________


===================
Yahoo Search
===================

Searching Yahoo's search results only shows one listing - which is for
a PDF file which is no longer there:

     "http://www.sohodecor.com/PO_Form1.pdf
      www.sohodecor.com/PO_Form1.pdf - More pages from this site

and clicking on "More pages from this site" does bring up your website.
  - http://search.yahoo.com/search?p=www.sohodecor.com&&fr=fp-pull-web-t&vst=0&vs=www.sohodecor.com


Searching Yahoo's directory produced the following results:

    "We didn't find any Directory pages containing www.sohodecor.com"


Your store URL, store.sohodecor.com, did produce results on Yahoo's
search engine, although it is not listed in Yahoo's directory, either.



======================
All The Web Search
======================

Searching for your domain, www.sohodecor.com, produces no results:

    "No entries for www.sohodecor.com in our index."
  - http://alltheweb.com/search?avkw=fogg&cat=web&cs=utf-8&q=www.sohodecor.com&_sb_lang=pref


Nor does your Yahoo store URL produce any results:
    "No entries for store.sohodecor.com in our index."



================
MSN Search
================

Your Yahoo store does snow up in MSN's Search results, or at least two
pages, the Wind Chimes and Bird Feeder pages.

  - http://search.msn.com/pass/results.asp?RS=CHECKED&FORM=MSNH&v=1&q=&q=store.sohodecor.com&cp=1252


Your domain name, www.sohodecor.com, likewise shows up,
  - http://search.msn.com/pass/results.aspx?q=www.sohodecor.com&FORM=SMCRT

____________________________


So what does this mean to you?

If this were simply a Google anomaly, the advice might be to wait;
however, when the other major search engines are experiencing problems
as well, then a closer look is in order.


====================================
Duplicate sites/Duplicate Content
====================================

This last Publishers Conference in Orlando said basically the same
thing. For good placement in free search engine results, the best
advice is still to follow Google's Guidelines. Important Google links
will be included at the bottom of this answer.


=====================
Duplicate Content
=====================

You mentioned - and are right to be concerned about - duplicate
content on the two URLs.

A check with Whois show your domain is also hosted by Yahoo. I am a
Web Designer/Developer and many of my clients use Yahoo Stores. I know
Yahoo has the ability for its merchants to either permanently redirect
their domain to their Yahoo Store, or to redirect the Yahoo store
results so that it always looks like their domain, and the Yahoo store
URLs do not show.

You need to make a choice one way or the other and get Yahoo to fix it
for you so that only one URL is showing.

Most search engines will penalize you for duplicate content. Some
either show only one domain, will 'penalize' you in lower placement,
or may drop both listings altogether.

The fact that you have made some design errors which literally hobble
search engine crawlers doesn't help, either. You now have those same
errors on two different websites.


============================
HTML - Back to the Basics
============================

When I first looked at your site, I thought it looked "odd". I have
taken the liberty of doing a "quick and dirty" screen shot of what I
see when I look at your page. By "quick and dirty", I mean that I did
not optimize this graphic for anything other than to show you what I
am seeing in several browsers.

Take a look at http://209.151.82.182/soho/sohodecor.jpg ... I suspect
that is NOT what you want visitors to see.

This is what the page looks like, however, in Mozilla 1.6, Firebird,
Opera, and IE 5.5. The site does look as you intended in IE 6.x, but
more savvy users are starting to make choices other than IE as their
browsers of choice.

This led to taking a close look at the coding, and there were some
important design elements missing.

Shari Thurow, one of the leading authorities in web design and search
engine optimization (and author of the book "Search Engine
Visibility"), recently made the following observation with regard to
HTML:

     "Clean HTML is absolutely imperative for search engine
      indexing. Browsers are extremely forgiving when it comes
      to displaying pages with "unclean" HTML (unclosed tags,
      no quotation marks, etc.).  Search engine spiders are
      not so forgiving.  Even something as simple as a missing
      quotation mark on the <.a href="page.html"> can cause a
      spider to not index text or a link." (See: Link Exchange
      Digest, July 3, 2003, "Clean HTML")
 - http://list.audettemedia.com/SCRIPTS/WA.EXE?A2=ind0307&L=led&D=1&T=0&H=1&O=D&F=&S=&P=282

She explained how errors in HTML can affect your ability to be
indexed, and ultimately, ranked.

Among those elements missing are:

1. DOCTYPE Declaration
----------------------

DOCTYPES are essential to the proper rendering and functioning of web
documents in compliant browsers. It is also essential for the search
engines to understand and follow the coding contained on your pages.

DOCTYPE is explained and discussed further in "A List Apart",
  - http://www.alistapart.com/stories/doctype/

and in Web Design Group's article, "Choosing a DOCtype",
  - http://www.htmlhelp.com/tools/validator/doctype.html


You do not have a DOCTYPE Declaration and you should add one to every
page on your site.


2. No close Head </head> tag
----------------------------

There is no close of your head tag. Some search engines, in this case,
IE., assumes it's there when it reaches the <body> tag; however, not
all browsers make such assumptions, which is why there's that rather
alarming blank place.

In its Technical Guidelines, Google suggests downloading and using the
Lynx browser,

     "Use a text browser such as Lynx to examine your site,
      because most search engine spiders see your site much
      as Lynx would. If fancy features such as Javascript,
      cookies, session ID's, frames, DHTML, or Flash keep you
      from seeing all of your site in a text browser, then
      search engine spiders may have trouble crawling your site."
   - ://www.google.com/webmasters/guidelines.html


Lynx is only getting to the first mapped image with links. 

At the present time, this is basically what search engines are seeing"

   "[yhst-9901986642544_1779_573.jpg]
    [yhst-9901986642544_1779_39386.jpg]
    [yhst-9901986642544_1779_138486.gif] 
    [USEMAP:yhst-9901986642544_1780_117595266.gif] pad
    [EMBED]
    SohoDecor Bird Feeder SohoDecor Bird House SohoDecor
    Charisma Rooters SohoDecor Welcome Sign SohoDecor
    Hummingbird Feeders SohoDecor Spring Garden Decor
    Collections
    ____________________________________________

    Butterfly Feeders | Floating Candle | Door Decor |
    Wall Decor | Lantern | Garden Accessory Wind Chimes |
    Welcome Signs | Bird Feeders | Bird Houses | Votive
    Candle Holder Planter & Stake | Rooters | Shelf Sitters
    | Stepping Stones | Sun Catchers | Wine Bottle Charms
    SohoDecor.com - Your Source of Christmas Ornaments and
    Garden Decorations Copyright 2004 LivingSoho, Inc. &
    SohoDecor.com - All rights reserved.

    [botomicon.jpg]"


You can download the Lynx browser here:
    - http://lynx.browser.org/


You will definitely want to correct the design deficiencies in order
to make it more user friendly and search engine friendly.

Remember, too, that the contents of your Flash cannot be indexed,
which means that basically you're not giving much to the search
engines to find and crawl.


==========================================
Important Google Links For Your Reference
==========================================

I am including important links from Google's site. This information
will help you understand Google's goals and responsibility to the web
searcher (and not the webmaster or site owner).

Webmasters who follow the guidelines and avoid Google's "Thou shalt
nots" usually have no problem getting listed and showing up under the
search terms they desire.

   * Google Today (absolutely the best information you
     can read about the "how and why" of Google's results
     - ://www.google.com/corporate/today.html

   * How Do I Get My Site Listed on Google?
     - ://www.google.com/webmasters/1.html

   * My Web Pages Are Not Currently Listed (a good 'primer'
     on how and why Google works so well)
     - ://www.google.com/webmasters/2.html

   * PageRank Information (covers both Google's PageRank and
     SERPS)
     - ://www.google.com/webmasters/4.html

   * Webmaster Guidelines (contains both the dos and don'ts)
     - ://www.google.com/webmasters/guidelines.html

   * Google Facts & Fiction (can  you buy your way to a
     high ranking in Google?)
     - ://www.google.com/webmasters/facts.html

   * Search Engine Optimizers (some good advice on what to look
     for if you're going to hire a Search Engine Optimizer)
     - ://www.google.com/webmasters/seo.html

   * Frequently Asked Questions (pretty much what it says, but
     definitely worth wading through)
     - ://www.google.com/webmasters/faq.html

   * Remove Content from Google's Index (just in case you feel
     a burning need to start all over again)
     - ://www.google.com/remove.html



================================
Other Sources of Information
================================

There is also good information from many of the top search engine
optimization experts, such as

   * Detlev Johnson, Search Engine Guide
     - http://www.searchengineguide.com/detlev/

   * Danny Sullivan, Search Engine Watch
     - http://www.searchenginewatch.com/

   * Jill Whalen, High Rankings
     - http://www.highrankings.com/

   * Shari Thurow, Web Pro News (and quoted all over the Internet)
     - www.webpronews.com/

who all have columns or newsletters to which you can subscribe and
keep abreast of the best way to use good content for better
positioning in search engine results.

In addition, Webmaster World - http://www.webmasterworld.com/ - has
discussion boards on most of the search engines. While some of the
discussions are anecdotal and/or questions for information, there is
usually enough discussion to keep abreast of what seems to be
happening.

There is an entire section devoted to Google at:
   - http://www.webmasterworld.com/forum3/

It never hurts to keep track of the discussions; but remember, trying
to optimize for search engines only is like trying to hit a moving
target. You'll notice among the more experienced contributors to the
discussions - plus the SEO experts listed above - that there really is
no substitute for content, relevant links and good HTML.


===========
Summary
===========

Your question was directed specifically at getting your pages indexed.
In order to do so, you need to clean up the design.

There really is no reason to use an image map for your left hand
margins, when text on the blue would be easier for search engines to
find and index.

Besides the design problems, there are problems with the fact that you
have duplicate content on two sites, even though they may be
interlinked.

Duplicate content is specifically seen as a method of spamming, and to
keep from being penalized, it would be best to ask your Host to
permanently redirect one of your sites to the other and then only try
to list the actual site on the Web.

You did not ask about ultimate search results placement for the search
terms you want. Although it is beyond the scope of your question, it
would be wise to remember that high placement requires a combination
of links TO your site, relevant content on your page and good HTML to
make your site both user friendly and search engine friendly.

If the other search engines are still admitting that designing for
Google's standards will ensure placement in their own search engines,
it would behoove you to make sure your site meets the guidelines set
out in the Google links pages I mentioned above.


Search strategies
===================

Other than the specific searches conducted and mentioned above, I
relied on the material I use in the daily course of my own business as
a Web designer/developer and search engine optimist.


Making the changes to your site with Google's Guidelines and good HTML
practices will greatly enhance the chances the search engine crawlers
will find and index your site.

Best wishes,

Serenata
Google Answers Researcher

---

Request for Answer Clarification by sohodecor-ga on 01 Mar 2004 12:52 PST

Dear Serenata,

Thank you very much for such a professional answer. 

After reading your reply, I realize that there are many design errors
I have made and need to be addressed in order for search engine like
Google to reindex me. However, about the duplicate site, I have
contacted Yahoo's merchant tech support and webhosting tech support,
their answers are making me very frustrating.
1.They tell me the way I have done is perfectly ok, because from
www.sohodecor.com all the link is directly store.SOHODECOR.COM which
is still under the same Domain name, just different server. They deny
that it will be consider the spam, and tell me most of their Merchant
Solution customer all have same configuration. The old version of
Yahoo Store may be change able but not Merchant Solution.(starting
like half years ago) Example he used is if www.ibm.com link to a
different server like store.ibm.com, is it a spam? Having a different
server doing different job is pretty common nowadays.

2.They spend like 20 minutes to talk to higher level and search the
history, couldn?t find any related back issues.  (call twice for
different person , same result)

Would you kindly be able to provide more answer on defining how a
search engine thinks a site is doing the spamming?  What are the key
points?
Currently www.sohodecor.com/index.html and
store.sohodecor.com/index.htm look the same, if I make them look
differently, will it still be consider the duplicated content?

I also have ?text link? on bottom of every page, even domain?s index
page is missing many critical html element, other page in store all
have text link, why Google still not able to index down further.  I am
thinking it?s too far away from top? ?

I am doing those Html changes now, but you still be able to see the
unchanged content on the web.  Please kindly answer me what else I can
do without Yahoo?s help.

Wish you have a wonderful day. 

Phil

---

Clarification of Answer by serenata-ga on 02 Mar 2004 14:56 PST

Hi again, Phil ~

Only Google knows the specifics of its algorithm, and they're not
going to share them. They do, however, offer guidelines with
recommendations and suggestions.

Their Quality Guidelines - Specific recommendations specifically state:

     "Don't create multiple pages, subdomains, or domains
      with substantially duplicate content."
   - ://www.google.com/webmasters/guidelines.html


And as I showed you, Google has specifically identified two different
URLS / domains with substantially the same content, one your own
domain and the other your Yahoo store.

You can get your host to direct your domain to your store and don't
try to list your domain, just list your store. As an alternative, you
can incorporate your store into your domain site and not really point
to the store.

Whether or not this affects you now, has affected you in the past, or
will change in the future is only known by Google.

There is some discussion about Google and the Yahoo stores on
Webmaster World's site,

   - http://www.webmasterworld.com/forum3/

but this is more anecdotal than 'official', unless the post is by
GoogleGuy (who is a Google engineer).

As for whether or not your main page (index.html) is listed or some
internal page is listed in search results, remember that search
engines return PAGES, not sites, so it would not be unreasonable for
your index.html page to rank lower than a specific page with relevant
content. If a search engine is working right, it actually should
return the internal page with more relevant information, and not the
more generic index.html page.

What works for those who don't move much during any search engine
update or change in algorithms is still the basics: relevant content,
links TO the site, and good design structure to make it user and
search engine friendly.

Hope this helps,

Serenata

Very satisfy... always recommanded.

Question : Are others like Yahoo etc. indexing your site ? or is it
just Google not indexing your site ?

Thank you for asking, 
Few month ago while Google and Yahoo are still together, whatever I
found in Google, I found the same result(even ranking) on Yahoo. After
that day Yahoo don't use Google's result, I am complete out from
Yahoo. In the beginning of this month(Feb), I sign up Inktomi for my
top page and other 3 Url. Now Yahoo is using those Url that I "buy"
from Inktomi.
Basicly after sepration of Google and Yahoo, no search engine index my
page without I puchase the lising service.
Feel free to ask me if anything unclear.

Good day.

Phil

Hello, 
Question : is http://www.msn.com or http://www.alltheweb.com indexing
your site ?  Perhaps you should look into the google guidelines and
check to see if everything is in order. GO TO
://www.google.com/webmasters/guidelines.html
If so, give it some time.

Very well explained and easy to understand. Thank you. 
Nando.