I need information about how the google robot works in relation to the
"noindex, no follow" or "noindex, follow" meta tag between 2 websites
:
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
I have 2 websites (Let's call them X.COM and Y.COM).
X.COM is already indexed by Google and I would like to keep it that way.
Y.COM is NOT indexed by Google. And I would also like to keep it that way as well.
To understand the question, you will need to know the basic structure
of site X.COM :
I will use 1000 pages as an example of the # of pages this site has.
Both X.COM and Y.COM contain 1000 indexable pages each (X.COM,
X.COM/PAGEa.html, X.COM/PAGEb.html, X.COM/PAGEc.html, and so on and
the same for Y.COM).
X.COM contains 10 links that point to its own domain second level
pages, such as http://X.COM/PAGEk.html, http://X.COM/PAGEl.html,
X.COM/PAGEm.html and so on).
All of X.COM's secondary and third level 1000 pages are linked
somewhere throughout the X.COM domain, such as through category pages,
site maps, etc.
Each page on X.COM also contains 5 outbound links pointing to site
Y.COM (Y.COM/PAGEa.html, Y.COM/PAGEb.html, Y.COM/PAGEc.html, and so
on.)
All 1000 links of Y.COM are linked to from X.COM throughout the various pages.
Question :
If I use the "noindex, nofollow" meta tag on all the pages of Y.COM,
how will the google robot function in relation to the rest of the
pages of X.COM when indexing X.COM?
When the Google robots go to X.COM for an index, will the Google robot
stop in its tracks when it encounters the "noindex" meta tag at Y.COM
and cease to index the rest of the 1000 pages of X.COM?
Since these 2 sites are intertwined via links, I want to make sure
that virtually all the 1000 pages of X.COM continue to get indexed,
even if I add the "no index, no follow" meta tag to all the pages at
Y.COM.
To clarify, here is the structure of X.COM :
X.COM has these links in it, in this order, from left to right and top
to bottom on the page, in the body.
X.COM/PAGEa.html
X.COM/PAGEb.html
Y.COM/PAGEa.html (no index meta tag)
X.COM/PAGEc.html
X.COM/PAGEd.html
Y.COM/PAGEb.html (no index meta tag)
X.COM/PAGEe.html
X.COM/PAGEf.html
When the google robot goes to X.COM it indexes X.COM/PAGEa.html and
X.COM/PAGEb.html and proceeds to Y.COM/PAGEa.html where it encounters
the "no index meta tag". Will it then bounce back out to the previous
page it was on at X.COM and continue to index X.COM/PAGEc.html and
X.COM/PAGEd.html and continue from there to Y.COM/PAGEb.html (where it
will not index this page due to the no index metatag) and then bounce
back out again to X.COM and continue again to X.COM/PAGEe and
X.COM/PAGEf.html, and so on all the way through the rest of the pages
of X.COM?
Or will the "no index meta tag" stop the google robot in its tracks
once it encounters the pages on Y.COM, so that, in essence, the rest
of the pages in the sequence on X.COM will no longer be indexed by
Google, as before?
Will this be true of all secondary and third pages of X.COM?
In other words, to summarize, will adding the "no index metag" to my
pages at Y.COM disrupt the standard indexing at X.COM since these 2
sites are linked together on every page?
In order to index all the linked pages within X.COM but not any of the
pages at Y.COM, what html meta tag should I use for this purpose? "no
index, no follow" or "no index, follow"? etc. or is there a better way
to do this? |
Clarification of Question by
applesofgold-ga
on
02 Mar 2004 11:07 PST
Hi, basically I wanted to be sure and I wasn't sure how the google bot
behaves. It could really effect our website listings if portions of
the site were not indexed or ranked due to my misuse of the metatag.
So to clarify, you're saying that as long as I use "no index, follow"
on all the pages of website Y.com - that all of the pages in X.com
will still get indexed? There won't be any interruption of indexing
all of X.COM's sequence of pages?
What should I put on "X.com" to make sure all the pages get indexed?
"index, follow"?
And I know I'm being redundant here - but my main concern was that if
I use the noindex meta tag on y.com, that x.com's pages would get
disrupted and not index correctly because x.com and y.com are so
intertwined through links. But you're saying this wouldn't be the
case?
If you can confirm your response with as much details as possible, I
would appreciate it - just, if anything, to confirm this information.
I want to be sure of what I am doing when it comes to this.
Thank you!
|
Request for Question Clarification by
robertskelton-ga
on
02 Mar 2004 11:20 PST
If you want Google to index a site completely, there is no need for
any robots tags.
Google state that they conform to standard robots.txt and robots meta
tags. However, it is impossible to guarantee that just because the
Google can index a site, or should index a site, that it will.
Sometimes they choose to not index an entire site, for reasons they do
not disclose.
There may be something within the Google algorithm that says if sites
are too intertwined, don't index them because they could be spam.
Probably not, but there is no way of being certain.
The real solution is to only have one website, or if you need two,
keep them separate.
|
Clarification of Question by
applesofgold-ga
on
02 Mar 2004 11:59 PST
Hi, exactly. That's kind of my point and why I am seeking this answer
- It is necessary for my purposes to have 2 different sites that are
intertwined. However, I don't want google to think I am spamming
them, so I would like to index only 1 (which is already indexed by
google), but not index the other (which is not yet indexed). So I am
looking for solutions to separate these sites as far as google is
concerned, so that google does not recognize one site, but it
continues to recognize the other.
That is why, I'm trying to confirm or determine if using the "no
index, follow" on the second site would be the right solution. I'm
only hesitant to do it in the case that using this meta tag would
interfere with google's current index of my original site (x.com).
It would not be a possible solution for me at this time to have only 1
site. The 2 sites, by necessity, have to be intertwined due to the
software I'm using and the way things are set up.
So the conclude, assuming that my sites have to be intertwined, would
using the appropriate "no index, follow" meta tag be a feasible
solution to my issue? It is important that the current set up at x.com
does not get effected by meta tags at y.com since they are
intertwined.
Anything definate you can give me based on this 1 particular aspect of things?
Am I on the right track? Barring google's undisclosed reasons for not
indexing sites? Just as far as this one specific area is concerned,
I'm assuming that the meta tag won't effect things negatively on the
site I want indexed, but I needed confirmation, since I'm not an
expert on this - I just know some basics of indexing.
|
