Often when you see a CGI script linked on a website it will have an
URL like this:

http://sitename.com/forum.cgi?post=1234

Search engines typically ignore URLs like this because they can get
caught in infinite loops with dynamic programs.  It's possible,
however, to represent that information in a way that makes it look
more like a static html page instead of a dynamic one.  Like this:

http://sitename.com/forum.cgi/post/1234/

The idea is that this makes it look less like a dynamic page and so
search engines will index it.  I want to make sure search engines
don't discriminate the page URL for just because it has .cgi in the
path though.

The question... I want to find me examples of one of more pages with
/.cgi/ in their path that come up in search engines.  So provide me
with search terms for each of the major engines that will generate at
least one result with /.cgi/ in the path.  If you can find a page in
most of the other major engines but not in one, I'd like to know that
as well.

Major search engines: google, lycos, altavista, excite, hotbot, and infoseek.

Also, for bonus points, find indexed URLs with any of the following
characters in them: "?", "&", "=", or "+".

Hi, 

Cool question.. After searching the web as much has I have been the
last few years I read you question and thought "That's Changed now"..
and was sure I could find many indexed pages in the Google and
AlltheWeb databases.. and I was right.

I'm going to post the queries I used, but really, I could have found
as many with just about any query I think. The engines are much better
at handling cgi and asp pages these days then they are with frames,
believe it our not. The thing to watch out for is to make sure those
cgi pages don't trap the spiders. Otherwise those may get
"blacklisted" off the robots list of places to go. There are plenty of
indexed pages out there.

Queries... On Google

/*.cgi?   "cgi?"

://www.google.com/search?q=+%22cgi%3F%22&num=100&hl=en&lr=&ie=UTF-8&oe=UTF-8&as_qdr=all&start=100&sa=N

is the url for the search I used. 

[FreeCode] - Browse project tree
www.freecode.com/browse/219/?topic=219


www.nsf.gov/cgi-bin/getpub?gpg

NASA Technical Report Server (NTRS), v 2.0
www.netskills.ac.uk/TonicNG/cgi/sesame?tng 

Wireless Communities
... WWW]http://www.wifi.ee/?p=us&lang=eng; Finland, Helsinki
[WWW]http://www.mediapoli.com/wireless/index.html;
Finland, Helsinki [WWW]http://www.wlanvoorumi.com/cgi ... 
www.personaltelco.net/index.cgi/WirelessCommunities - 53k - 18 Aug
2002 - Cached - Similar pages


The David and Lucile Packard Foundation: Fellowships for Science ... 
www.packard.org/index.cgi?page=scifellow - 8k - 18 Aug 2002 - Cached -
Similar pages

license-discuss@opensource.org: Threads for August 2002
[<<-]Messages by date] [eSubscribe] [eFAQ] Threads for
August 2002. August 2002. [2] Re: question about the term "source ...
www.crynwr.com/cgi-bin/ezmlm-cgi?3 - 5k - 18 Aug 2002 - Cached -
Similar pages

Join or leave the EDUPAGE list
Join or leave the EDUPAGE list. This screen allows you to join or
leave
the EDUPAGE list. To confirm your identity and prevent third ... 
listserv.educause.edu/cgi-bin/ wa.exe?SUBED1=edupage&A=1 - 7k - Cached
- Similar pages

lots ... lots .. more 

Yahoo.. is the same as Google so we won't bother posting the same ones
here

All the web -- Used the advanced search area and asked for cgi in the
URL and ? in the URL

This is the link for the actual search

http://www.alltheweb.com/search?advanced=1&cat=web&type=phrase&q=.cgi&jsact=&l=any&ics=utf-8&cs=utf-8&wf%5Bn%5D=3&wf%5B0%5D%5Br%5D=&wf%5B0%5D%5Bq%5D=cgi%3F&wf%5B0%5D%5Bw%5D=url.all%3A&wf%5B1%5D%5Br%5D=%2B&wf%5B1%5D%5Bq%5D=%26&wf%5B1%5D%5Bw%5D=url.all%3A&wf%5B2%5D%5Br%5D=-&wf%5B2%5D%5Bq%5D=&wf%5B2%5D%5Bw%5D=&dincl=&dexcl=&limip=&age=&size%5Bp%5D=%3D&size%5Bv%5D=&size%5Bx%5D=0&hits=100&nooc=off

We got several examples of what you are talking about. 

CGI Pull Down Menu Redirector
Have you ever wanted a pull-down menu that redirects users to a
specified webpage based upon the menu option they selected? If so,
then this CGI script is perfect for you. This script will work with
all major browsers and is very easy to install
http://webdevelopment.developersnetwork.com/Articles.asp?... (26.9 kB)

HotFreeSite: Add a New Listing
free hot stuff including free samples, fonts, games, graphics,
Webmaster freebies, email service, Web hosting, chat programs,
screensavers, contests, java, javascript and MP3s.
http://www.hotfreesite.com/cgi-bin/add.cgi (32.5 kB)


106.  CGI Pull Down Menu Redirector | Web Host Search, Web Hosting
Companies
Have you ever wanted a pull-down menu that redirects users to a
specified webpage based upon the menu option they selected? If so,
then this CGI script is perfect for you. This script will work with
all major browsers and is very easy to install
http://www.webhostsearch.info/Articles/Articles.asp?... (32.2 kB)


 111.  An interactive introduction to HTML and CGI scripts on the WWW
(courseware)
It is a hypertext link to a web page containing examples of the HTML
features under discussion. The reader can if they wish modify the text
to try out their own ideas of style and formatting
http://www.is-edu.hcmuns.edu.vn/WebLib/Books/.../HTML&... (15.4 kB)


134.  No Title
http://new.search.com/search?channel=1&q=Spill: A Fictional Pilgrimage
from Valencia to Santiago in the Fifteenth Century&tag=ex.pt.savvy..se
http://magnus.bleken.no/seelog.cgi?logfile=/index_main.html (976.4 kB)



258.  not on the second copy that # appears as an in some modes. # #
zoom_in.url # Like...
etc. If you don't want to mess with HTML # (one of the main benefits
of this CGI is that you can just upload your # images and not have to
mess with a bunch of HTML authoring), the file can # contain just
plain text, but note that if you have multiple...
http://harkless.org/dan/software/image_album?2002-03-30 (83.1 kB)


308.  ICQ Inc. - ICQ Search - Category "GNATS"
ICQ Search results for GNATS
http://www.icqit.com/cat.adp?id=173312 (14.2 kB)


313.  Davezilla | One blog to rule them all
A thorn in the side of the A-List
http://www.davezilla.com/index.php?p=395&c=1 (13.0 kB)



Another search of the AlltheWeb Site, this time using a query that
said
\.cig/ must be in the URl came up with these, which are more to your
point. The slash before the . dot is to escape it, so its really used
in the query.

Here's the link to the query itself
http://www.alltheweb.com/search?t=all&q=.cgi&c=web&o=100&h=100&l=any&av=1&wf%5Bn%5D=3&wf%5B0%5D%5Bq%5D=%5C.cgi%2F&wf%5B0%5D%5Bw%5D=url.all%3A&wf%5B1%5D%5Br%5D=%2B&wf%5B2%5D%5Br%5D=-&no=on&qtf=n&size%5Bp%5D=%3D&size%5Bx%5D=0&ics=utf-8&cs=utf-8


 147.  Dr Clue's HTML Guide [] CGI Programming and C++ Classes for the
Common Gateway...
Information covering how CGI works and the different methods of
interaction. Includes a free C++ class that compiles on both Unix and
Windows , along with several pieces of code that exploit it.
more hits from: http://www.drclue.net/F1.cgi/HTML/CGI/CGI.html (9.2
kB)


278.  MSp ČR - Zadávání podmínek pro výběr subjektu
Odštěpný závod nebo jiná organizační složka podniku zapi
http://www.justice.cz/cgi-bin/sqw1250.cgi/or/l_hled.sqw (4.5 kB)


Using AltaVista the query best suited was  url:*.cgi/*
http://www.altavista.com/sites/search/web?q=url%3A*cgi%2F*&pg=q&avkw=tgz&kl=XX&stq=20


Wells Fargo - 2nd mortgage company   
Click here for Wells Fargo 2nd mortgage company!
www.wellsfargo.com/?tag.cgi=938668148671...391145549207448 • Translate


428.  Explore the States
Click on the name of the state you want to explore. Choose a State:
Alabama Alaska Arizona Arkansas California Colorado Connecticut
Delaware District of Columbia Florida Georgia Hawaii Idaho Illinois
Indiana Iowa Kansas Kentucky Louisiana Maine Maryland...
http://www.americaslibrary.gov/cgi-bin/page.cgi/es (13.6 kB)

Lycos is unfamilar to me, because I never get good results there I
don't keep up on how to search it. But, I searched for one of the
links I found on AlltheWeb and found it right off.

I didn't cheat by using a URL search.. My query was this :
Click on the name of the state you want to explore

And it came up with this one 4 down. 

4.  Explore the States  Fast Forward » - Click on the name of the
state you want to explore. Choose a State: Alabama Alaska Arizona
Arkansas California Colorado Connecticut Delaware District of Columbia
Florida Georgia Hawaii Idaho Illinois...
http://www.americaslibrary.gov/cgi-bin/page.cgi/es 



Excite is a Meta Search engine so it has no advance features in it, it
simple queries the real engines

HotBot is Lycos with a different front end.

Infoseek is Overture with a different front end. Overture is more or
less a pay or play listing area, and doesn't have very good advanced
search features.

So I used our : "Click on the name of the state you want to explore"
query

and got this page
http://www.americaslibrary.gov/cgi-bin/page.cgi/es


Hope that was as fun for you as it was for me. 

Thanks, 

webadept-ga

just what I needed thanks.

This is an interesting question and a common Internet marketing
problem. How much does databased pages and dynamic generated pages
cause search engines not to list pages or rank them highly? There has
been much change in this area just as webadept stated.

And with all due respect to your question, I might propose that you
are asking for an answer to justify a choice in web design. Maybe you
already have a site that includes scripted pages and are wondering
about what problems the site has in rankings? Or are you considering a
web proposal that includes scripted pages on a web site and you are
hoping that this wont mean you are dooming your site not to have high
rankings for those scripted pages?

Just because a researcher can determinedly find scripted pages in
search engines means that scripted pages are not bad for rankings or
for some other reason. Webadept found quite a few but usually had to
add "cgi" to the search term somehow to find those pages. And often
using an advanced search function, which is more exact but not what
the majority of searchers use.

An interesting question I thought of would be: Do scripted pages show
in the search results in the top 30 for the keywords your site will
use?

There are a myriad of other questions: If a page is scripted what
would its ranking be and if it was the same basic content but static
would the ranking be higher?

Some sites show often in search results with scripted pages but often
are aided by high link popularity like Ebay or amazon.

http://search.msn.com/results.asp?q=ionic+breeze+quadra&spoff=on&origq=6+week+body+makeover&RS=CHECKED&FORM=SMCRT&v=1&cfg=SMCINITIAL&nosp=0&thr=

#6 is Ebay. I have seen them on lots of scripted rankings.

I have seen some high rankings for scripted pages but usually as I
said either on Ebay or Amazon pages. For some fairly competitive terms
I have seen scripted pages often on Inktomi powered search engines
such as MSN. Here is one where a scripted page gets a number one
ranking:

http://search.msn.com/results.asp?q=6+week+body+makeover&spoff=on&origq=&RS=CHECKED&FORM=SMCRT&v=1&cfg=SMCINITIAL&nosp=0&thr=

Or the same as the first example : one where the scripted pages are
#6, 7, and 8. It is below:

http://search.msn.com/results.asp?q=ionic+breeze+quadra&spoff=on&origq=6+week+body+makeover&RS=CHECKED&FORM=SMCRT&v=1&cfg=SMCINITIAL&nosp=0&thr=

That is one of the terms I know of where a large amount of the terms
are based on scripted pages. And if you didn't know the top directory
terms are based on Looksmart PPC advertising so those don't really
count. Many of the real calculated rank terms on those pages are
scripted.

But in reality I think you should determine if what I put forth in the
first paragraph is true.

Should your question be would my site get equal rankings and number of
pages listed with scripted pages? Will scripted pages cause me further
development time or advertising costs than a static site? On the other
hand what are the drawbacks of static pages to organizing, updating,
and uploading pages to the site?

This is a tough question and might be multi staged to get the best
answer and might take more input as to what your site and industry
are.

It might take some serious effort from web design team, marketing
team, and those with the financial interests.

Good Luck,

insideinfo-ga