My url was removed by Google's robot more than 7 weeks ago due to my
new web host's inability to recognize or deal with the "&" sign. Up
until that time, I had #1 ranking under just about any phrase with
"r&b" leading the title. Then, Inktomi removed me from their listing.
My web tech constructed some custom error pages, as hits to my site
went to 404 Error pages. Then Google Team saw there were disallow
issues and made me aware of them. My tech fixed those problems in my
html. She changed that page's url suffix from "r&b/" to
"varietydance". I then had her change it back to "r&b/". I immediately
resubitted to Google at the Google Resubmit page. It has now been over
6 weeks, and the Google robot still does not recognize my "r&b motown
bands" or "r&b dance bands".
How do I get the Google robot to accept that page's url with the "&"
in it? |
Request for Question Clarification by
webadept-ga
on
09 Sep 2003 15:44 PDT
Hi,
I've read your question and few times now and I'm still scratching my
head. Can you post a link to your site so I can see what it is you are
trying to do? It sounds like you want pages called
http://www.MySite.com/r&b_dance_bands.html which is not going to work
very well. But I would like you to clarify that and let us see your
real site pages.
You also might want to take a look at this page.
http://answers.google.com/answers/pricing.html
thanks,
webadept-ga
|
Clarification of Question by
levotb-ga
on
09 Sep 2003 17:10 PDT
Hi, Webadept-ga!
My highly ranked website (web-wide) has a page whose url is NOT being
recognized at all by Google robot/spiders. The site is
http://www.davidlevinent.com and the inner page is
http://www.davidlevinent.com/r&b/ . Until I changed we hosts
officially in late April and up until my previous web host went out of
business in June, I was ranked #1 at Google and Google-related engines
(Yahoo, AOL, etc.) under "r&b motown bands", "r&b dance bands", "r&b
rock bands", etc. Because my new host, Infinology.com, had a problem
with "&" signs, it suddenly dropped my site from its directory. It was
on Thursday evening. I spoke with their tech, and by the following
Monday, it was determined that Infinology had what he called "a url
scan issue". He then performed the scan to the url that had been
removed, and he said everything should be fine. He asked to speak with
my tech, Jill in Florida, and they straightened things out. HOWEVER,
because of the passage of three of four days, the Google robot saw
nothing but "404 Error links" to my site and to that page, and removed
me completely from it's listings--except with regards to my other
rankings under Classical, Ethnic Bands, etc. or anything BUT the
Variety Dance Bands url.
To make my Index page (which was visible but with "xed out' button
links) and all "r&b"-suffix pages work (a previous name for that page
was "r&b/R.htm"), Jill created FOUR custom error pages so that all my
pages/urls on the site were linked to non-404 Error pages.
Then, Google Team sent me the following response to my email re:
removal of that url:
"Thank you for your note.
Upon reviewing your robots.txt file located at,
http://www.davidlevinent.com/robots.txt, it disallows our robots from
crawling the r&b directory.
User-agent: *
Disallow: /cgi-bin/
Disallow: /logs/
Disallow: /Images/
Disallow: /_notes/
Disallow: /images/
Disallow: /oldfiles/
Disallow: /Scanned Photos/
Disallow: /test/
Disallow:/r&b/
Please remove the disallow line "Disallow:/r&b/" and the next time our
robots recrawl your site, the r&b directory should get crawled.
Regards,
The Google Team"
This was done immediately by Jill, my tech.
Then, my tech mistakenly changed the directory name for the url from
http://www.davidlevinent.com/r&b to
http://www.davidlevinent.com/varietydance/.
I found out about this and a few days later, asked her to make it
"r&b/". Six weeks later, I am STILL not visible ANYwhere under "r&b
motown bands" or "r&b dance bands"! I have properly resubmitted that
url ("r&b/") numerous times, each time successfully.
Now, finally, everything seems to be fine...except the url is STILL
not being recognized.
The Google Team tells me "Be patient. Our spider will recognize the
new url." I'm afraid I do not share their assurance. They cannot/will
not provide a satisfactory answer as to "why" this is happening!
Precious time is passing. They don't say if there is a problem with
the "&" sign or not. My tech fixed the "disallow" issues to their
satisfaction.
As the current url for that important page on my site has the "&" in
it, I need to know asap if THAT (the "&") is "the problem at Google".
If it is, my tech can change once again the directory name for that
url back to "varietydance/".
The reason I haven't changed the name AGAIN is because each time I do
so, costs me 5-8 weeks of time for recognition. That is insane and I
have already paid dearly in both time and exposure waiting to be
recognized.
Other sites list "R&B" and are highly ranked at Google, so why the
problem with MY url? It is a properlty constructed/functioning url.
Thanks and I look forward to hearing from you. DL
|
Clarification of Question by
levotb-ga
on
09 Sep 2003 17:19 PDT
Hi webadept-ga!
My site is www.davidlevinent.com. The Variety Dance Bands
page--http://www.davidlevinent.com/r&b/ is the url not being
recognized. No, I am NOT describing "
http://www.MySite.com/r&b_dance_bands.html ". I am describing:
http://www.davidlevinent.com/r&b/
I hope this clarifies the matter. DL
|
Request for Question Clarification by
webadept-ga
on
09 Sep 2003 18:47 PDT
Hi again,
I'm not going to take this on as a "real" question, because I doubt
I'm going to convince you. :-)
The & symbol in a URL sting is reserved, meaning that is normally used
to produce a "GET" string from a form to a CGI or PHP or some other
page that can use the variables that follow. It is also used for
dynamic pages, so that a page can render itself according to some
variable.
If you go to Google and use the search, and then look up at the URL
that is produced, you will see this in action.
://www.google.com is the main page.. you put in a search for
"Dance Lessons" and we get :
://www.google.com/search?hl=en&ie=UTF-8&oe=UTF-8&q=Dance+Lessons
The ? symbols starts the string
hl=en
&ie=UTF-8
&oe=UTF-8
&q=Dance+Lessons
What is a major concern for search engine bot creators is their bot
getting trapped inside a website. This has two unwanted effects when
this happens. The first is that the bot is stuck and wastes several
hours sitting on your site, and not getting anywhere else and the
second is that the bot slows your site down because it is repetitively
looping through. Both situations are extremely bad for both parties.
What is probably happening here is that the Google bot is seeing your
page URL's as poorly formed dynamic pages and therefore, not going
into them. I seriously doubt it has anything to do with your ISP,
after all I can see the pages you posted, so they are being severed
correctly.
I have to agree with our comment below, no one visiting your page
cares what the URL is, really. Nor do the Search Engines. It use to be
that we posted url's with keywords in them to boost our levels on
searches, but that stopped working back in 2000, so really I would
change the name of the pages to r_and_b or rb or something of that
nature. Using a reserved symbol is simply poor practice to begin with.
Now, you say you moved the page seven weeks ago and lost the Google
ranking. This is common, in fact your loss of listing on Google could
simply be because you moved the page, and could have nothing at all to
do with the & symbol. Seeing it in a page name would send red flags up
for anyone like myself who works on the web as a programmer. So if one
of the Google folks told you that's probably the problem, I'm not
surprised. In fact had you called me I would have told you that was
the problem too, completely overlooking the fact that you have moved
your page. It is such a glaring error in page naming that it really
warps the mind.
I could be up to 3 months before Google gets you back on the engines.
That's what they say in the FAQ's anyway, though I've never heard of
anyone having to wait that long. I would resubmit the main page, take
out those &'s from the page names and wait for this months Google
Dance (indexing routine) to be over, about the first of October.
Hope all this works out for you.
webadept-ga
|
Request for Question Clarification by
webadept-ga
on
09 Sep 2003 18:55 PDT
Hi again,
I just saw that robots.txt thing. Wow.. <smile> don't feel alone, I've
had that happen to myself back in the day as well.
Believe the Googles. The Google Dance, (what we call the indexing
period that Google does, because during it all kinds of things can
happen to page ranks and listings) happens only once a month. Google
bots will hit your pages several times a week, but the indexing
happens once a month. I don't know how long ago you fixed this
robots.txt error, but it won't be fixed in the Google engine until the
dance is over. This one or the next. But it will be fixed.
My first comments still hold though. I would take that & out of there,
it is not "well formed" in any fasion. Google and other engines may
still index the pages, but at this point, why gamble?
webadept-ga
|
