Google Answers Logo
View Question
 
Q: Spiders, Bots, Web Trawlers, Knowledge Management, Competitive Intelligence ( Answered,   1 Comment )
Question  
Subject: Spiders, Bots, Web Trawlers, Knowledge Management, Competitive Intelligence
Category: Computers
Asked by: wag265-ga
List Price: $100.00
Posted: 21 Jun 2002 14:50 PDT
Expires: 28 Jun 2002 14:50 PDT
Question ID: 31350
Spiders, Bots, Web Trawlers, Knowledge Management, Competitive
Intelligence, Business Intelligence - all have been somewhat
dissapointing in their evolution as that evolution relates to
efficiently acquiring, converting and presenting information (from the
massive world wide web) in a cost-beneficial, user-friendly format.

Relevare is a UK based company of 18 supposed technical gurus, and
they seem to have (and promote themselves as having) syncretized and
synthesized the heretofore fragmented, multi-faceted, contibuting
elements necessary for a true "soup to nuts", comprehensive,
one-stop-shop solution.

See www.relevare.com

Relevare is the "best" company/offering I have found for the turnkey
solution I need for searching (and constantly re-searching) the web,
filtering down to relevant data, filtering out irrelevant data,
organizing that data and delivering that data to my business and my
non-technical employees in a quickly accessible (Icons), well
taxonomically organized, meaningful, non-laborious, usable format.

Relevare's website promotional blurbs seem to answer most, if not all
of, the software and multi-element integration deficiencies that my
research indicated were still prevalent in this arena until mid-2001.

I know and understand everything contained in Relevare's website
(partnership with Autonomy, various product and service offerings,
etc.) and recently had a three hour conference call with their sales
and technical team.

Please research Relevare's site in full detail to get up to speed with
my current level of understanding. Then, to sufficiently answer my
"Question", I need the "Answer" to provide the following:

A. Delivery to me of the full scope (pros and cons) of all available
published commentary, criticism, research, articles, product testing
and documentation of any kind regarding Relevare's (and their
partner's applicable) products, services, track record and
performance.

B. Delivery to me of a list of Relevare's best current competitors and
as much distinguishing information and documentation (same as Part A.
above) on those competitors as possible.

C. Delivery to me of your, the researcher's personal opinion, of
Relevare (informed by either your first hand knowledge of or your
exposure to Relevare or solely by your analysis of the research -
please indicate to what degree for both) and your ranking of the top
five to ten competitors (including Relevare) in this specific sector.

Thanks,

wag265

Request for Question Clarification by davidsar-ga on 21 Jun 2002 15:26 PDT
I had a lock on this one for a while, but I’m going to return it to
the answer pool with a (hopefully) constructive suggestion or two:

-To properly answer this is probably a full day or two day research
task – not something that’s conducive to Google Answers and its system
of locking a question an hour at a time.

-I could answer any ONE of your questions A, B or C but not all three
at one sitting.  Please consider breaking them up and posting them as
individual questions.

-If your goal is one of “efficiently acquiring, converting and
presenting information (from the
massive world wide web) in a cost-beneficial, user-friendly format”, I
think you already have two of the best tools ever invented at your
disposal – Google and Google Answers.  Careful searches on Google will
return highly relevant results.  But for those not inclined to learn
advanced searching techniques, the Google Answer team can do it for
you – and what could be more user-friendly than a query system that
interacts with the questioner and asks for clarification whenever it’s
needed.

-From the brief look I had, Relevare is a very sophisticated system,
but one that makes use of a much smaller overall “piece” of the web
(about 250 million pages) than Google’s 2 billion and counting pages. 
Are you sure this is the way you want to go?

-Anyhow...good luck in quest.  

Dave
Answer  
Subject: Re: Spiders, Bots, Web Trawlers, Knowledge Management, Competitive Intelligence
Answered By: webadept-ga on 21 Jun 2002 18:25 PDT
 
Hi, and WOW, this is a very bizarre quest you have me on here, and I
have to say the answers are stranger than fiction.

First off, I didn't find a single write up on the company from an
outside source since, March 29, 2001. At that time OpinionWire in the
UK wrote up a small article and merely states that the company exists
and that they are up against IBM and Compuware with their product.
That's it... nothing more. Very odd for a Search Engine, especially a
search engine, because these companies get written up and judged by
just about everyone.

But it doesn't stop there. One of the things I read on the website was
that their special crawler called "bumblebee" is out there 24/7
looking at websites. I'm sure you read that too. I found several logs
which listed the bumblebee as having been there and I've posted below
some links to those logs so you can see what they look like. My real
reason for doing that was to show you a log, because I got curious.
See, I run several websites, and I have never heard of this robot. A
robot has a name and they identify themselves in the logs, and being a
webmaster, I check those logs to see who has been around.

On my personal server, for the last 12 months, not one entry shows
this bumblebee robot. Now, my websites (there are five on this server)
aren't especially important or even interesting to anyone not in the
"internet" business, so I looked at 14 of my customer's logs, checking
them for the last 3 months. Not a single hit. Not one. Some of those
websites are "relevant" sites, so perhaps this robot is only cruising
through the UK and Europe. That would not equal a "World of
Information".

The write-ups I did find on the website itself were all on the Flash
Design, nothing about the service. It's a good design, takes a bit to
get use to, and its hard to read on a larger screen, but that's the
only reason I've found anyone has paid the slightest attention to this
company since last year. Even these write-ups were small and hardly
worth mentioning.

Quoted Definition from Webopedia.com

Web Portal 
 
A Web site or service that offers a broad array of resources and
services, such as e-mail, forums, search engines, and on-line shopping
malls. The first Web portals were online services, such as AOL, that
provided access to the Web, but by now most of the traditional search
engines have transformed themselves into Web portals to attract and
keep a larger audience.

A Web portal is commonly referred to as simply a portal. 

http://www.pcwebopaedia.com/TERM/W/Web_portal.html

That's a good definition of what Internet types think of when using
the term Portal. Relevare is apparently going one step past this and
creating a web portal that is specifically for your business or needs
created from information that is on the Internet. An "AOL for You"
kind of thing. Sounds like a good idea and there is nothing out there,
right now, that is in the same box. IBM's Portal deals with your
company's documentation and databases. Excel files, Word Documents,
etc., all get logged into this software Portal which then tries to
make some sense of it all for you, and allows you to find information
quickly when your company documents start looking as wide in breath as
the Internet does. This is completely different. Compuware is the same
thing.

On the other side of the spectrum you have web portal websites.
BlackWebPortal, and SemanticWeb.org are good examples of these.
Portals like these are websites which focus on a particular interest
and hunt down on the information they can find for that subject.
Sometimes they miss depending on their methods, but most of the time
they hit and do a good job. MSN.com and AOL.com, and even Yahoo.com
can be put into this category as well.

Another rather odd feature or lack of feature actually, is that there
is no demo for the Relevare system. I can't find one. Normally a web
service like this would have a demo, since they are obviously running
the bumblebee somewhere (not the US or Canada, but somewhere). So they
have some type of criteria set for the robot and it's presumably
bringing back "relevant" information for that criteria. So, why no
demo? This could explain the lack of write-ups, since no one really
has anything to judge their accuracy or relevancy by.

If they had a demo I could then search the internet using the same
criteria and check to see how "relevant" they are, since I do this
kind of work all the time, a spot check of a few "search strings"
could see what they are getting and what they are leaving behind.
Then, I could write something up and give you more information. But,
no demo.


Personal Thoughts. 

davidsar-ga had a good point with using Google-Answers. I'll put that
here and leave it alone. He's written out a good description of how
you could use us and we would love to have you.

"Soup-To-Nuts" type solutions rarely are. There were several hundred
of these types of "Complete Solutions to Information on the Web"
companies started up in the late 1990's and none of them survived.
That doesn't mean that this one wouldn't. One of my favorite sayings
is "Everyday someone is doing something that someone else said was
impossible" but the resources involved with keeping a service like
this running are huge. Not to mention your company's needs changing
periodically. What happens then? More over, what happens if they can't
change with you?

Most robots hit my servers about once a month, sometimes a little
sooner, most are once a month, so if you change criteria with one of
these, its not going to show up until next month. This type of service
would have to be out there everyday. Everyday, to get you "new"
websites that are meeting your criteria. Santa Claus does it in a
night, but I don't know many others that can.

As a webmaster, if I was getting scanned by the same robot everyday, I
would block that robot from my site. I know several other webmasters
who would do the same thing and most of the big news sites would as
well (my opinion). That type of action is considered "rude" by
internet folks. So if you are getting your information from a
particular website and they block this ill-behaved robot, where does
that leave you? They could of course change the name of their robot,
which is done by the more "rude" people that own them, but eventually
they are not going to be welcome, and poof, they're gone.

Advice : 

I know you didn't ask for it, but I'm going to put it in here anyway.
I would, if I was going to pay for something like this, want to see it
work for a month. I'm not sure what they are charging for the service,
that's not on the website either, but I would ask to see it in action,
and perhaps let one of us go through it as well. If not a month, at
least some type of demo? And set it up for someone, if not a Google
Researcher, someone who knows how to search the web.

A real let down is to find out that the pages you are being served are
not in fact pages that match a criteria, but pages that have paid to
match your criteria. So that you are being fed information that is not
really "relevant" but profitable. For them I mean.

Spiders.com -- No outside write-ups. 

Bots.com -- Didn't find a write-up here either from an outside source.

Web Trawlers -- Here's something to try, do a search on Google using
this term. +"Web Trawlers" and you'll get a listing of hits that back
up my "Opinion" earlier. Most webmasters don't like these things, we
like them if they are well behaved, but the ones that aren't ... well,
they are a fly in the ointment. I couldn't even find these guys on the
web as a service at webtrawlers.com, so I'm not sure why you mentioned
them.

The rest appear to be obscure as well. 


IBM's Portal
http://www-3.ibm.com/pvc/products/portal_server/index.shtml


Compuware
http://www.compuware.com/


BlackWebPortal.com
http://www.blackwebportal.com/

SemanticWeb.org
http://www.semanticweb.org/

relevare
http://www.relevare.com/site/


Links of Interest

OpinionWire: Iconic Vortals
http://www.serverworldmagazine.com/opinionw/2001/03/29_vortals.shtml

List of Access Logs With Bumblebee Entries. 
chobhamvillage.org.uk
http://www.chobhamvillage.org.uk/logs/cho_hosting_access_log

etonvillage.org.uk
http://www.etonvillage.org.uk/logs/eto_hosting_access_log

key-property.co.uk
http://www.etonvillage.org.uk/logs/eto_hosting_access_log


Query Schema

+relevare
+relevare +review
+relevare +opinion
+bumblebee
+bumblebee +review
+relevare +portal
+IBM +portal
+Compuware +portal

much much more... 

Thanks for the question and good luck,

webadept-ga
Comments  
Subject: Re: Spiders, Bots, Web Trawlers, Knowledge Management, Competitive Intelligence
From: davidsar-ga on 21 Jun 2002 19:19 PDT
 
I've got to add my thumb's down to webadept's...I wasn't all that
impressed with what I saw either.

Important Disclaimer: Answers and comments provided on Google Answers are general information, and are not intended to substitute for informed professional medical, psychiatric, psychological, tax, legal, investment, accounting, or other professional advice. Google does not endorse, and expressly disclaims liability for any product, manufacturer, distributor, service or service provider mentioned or any opinion expressed in answers or comments. Please read carefully the Google Answers Terms of Service.

If you feel that you have found inappropriate content, please let us know by emailing us at answers-support@google.com with the question ID listed above. Thank you.
Search Google Answers for
Google Answers  


Google Home - Answers FAQ - Terms of Service - Privacy Policy