I searched on ‘apteryx’ the other day to see if anyone’s come by to
help me out with my question about software for pricing inventory
(#182608) and saw that no one has (why not? I wonder--I thought that
would be an easy one, straight facts)--and that is when I saw that you
had answered one of Granny’s marvelous survey questions by listing me
among your favorite customers.  Wowee!  What a thrill!  I am still
grinning.  To be on anybody’s list would be a real honor and a
tremendous compliment.  To be on yours, dear Pink, with your
superlative standards of research and delivery, your remarkable
presence, and your polished-to-gleaming style, is like getting the
prettiest valentine in the class--and a dozen roses besides.

To say thank you, I’d like to offer this question to you.

Please be patient with me, now.  This is going to take a minute.

When I first got the hang of web-surfing, just a little over two years
ago, and tried out various search engines for the first time, I was
awed, as youngsters can never be who take it for granted, at the
depth, breadth, and sheer mass of information and opinion and creative
work that can be found on the Internet.  I still have a deep love of
books and libraries and traditional scholarship (and wooden
card-catalog drawers you can browse in), but the magical allure of the
web is almost irresistible.  I spent one whole Sunday afternoon just
reading the things that came up when I typed “BP” into Google and
clicked “search.”

Hard on the heels of that initial astonished excitement, however, came
a disturbing question:  how does one evaluate the quality of
information found by this means?

To be sure, one might (and should) ask the same question of what one
finds in print.  Plenty of worthless junk gets into books, magazines,
and newspapers and always has.  Nonetheless, there is a process that
acts as a filter of a sort, and there are some barriers to wholesale
delivery of written material to the public that require effort to
overcome, such as the mechanics of production, the hurdles of
distribution, and the challenge of getting a prospective audience to
pay for your product.  In the year or more that it takes a scholarly
book to go from an author’s hands through the acquisitions process and
the steps of developmental editing, author’s revision, manuscript
editing, reviewing, copyediting, production, proofreading, and
indexing, the content is subject to many rigorous tests.  The
financial investment that the process requires and the dependence of
its success on acceptance by a community of prospective critics say
something about the level of confidence one might place in its merit.

By contrast, anyone can put anything on the Internet, at very little
expense and with no significant resources required beyond a personal
commitment of time and energy.  There is no vetting of any sort, no
necessity of rigor or discipline to the content, no measure of quality
applied.  What it takes to put content onto the Internet is simply
nothing like what it takes to bring a book or magazine to publication.

And the quality of much of what we can find on the Internet is
conspicuously wanting.  It may not be too difficult to pick out the
defrauders, deceivers, and dissemblers who write pseudo-facts in
service of some ideology or cause; some quality of monomania, some
edge of hysteria or shrillness may give them away.  But even if we put
these aside, along with the clever hoaxes that have no true harmful
intent but still may grossly mislead, we see the work of the unwitting
who perpetrate error out of sheer arrogance:  the transcribers of Poe,
for instance, who render the line “as of someone gently rapping” as
“as if . . . ,” perhaps thinking to correct Poe’s impeccable grammar
with something that sounds more customary to their own ignorant ear,
or quoters of the allegedly Hippocratic “primum non nocere” who insert
a comma into the expression as if they were punctuating an English
sentence.  We find the admiring students’ flawed renditions of a
teacher’s lessons imperfectly understood, or the postings of the
careless or foolish who simply cannot differentiate their own
perceptions from anything objectively verifiable and write reckless
nonsense masquerading as reporting.  Until our educators have put
aside self-esteem as the paramount value in education and committed
themselves to rescuing discrimination from the bad rap it got in the
sixties, when it became equated with blind, unreasoning prejudice
although it actually means the opposite--a necessity not only for clear
rational thought but indeed for any sentient creature’s survival—-who
will show us how to tell good information from bad, to differentiate
between fact and opinion, to recognize propaganda, to question the
authenticity of data, and to reject specious logic?  How can our young
people, in particular, mine the Internet’s amazing treasury of
knowledge and pick out the gold from all the dross?  These are
questions that trouble me.

And so my question to you, PinkFreud, is this:  in your online
research, what discriminators do you use to recognize sound
information, to test what is offered as fact, to weed out fringe
sources and dubious data and special-interest interpretations, to
validate the quality of the references you quote and cite?  What
questions do you ask, or what cues do you look for, or what signs of
legitimacy, or what reliable authorities?  How, in sum, do you judge
what is good information?

---

Request for Question Clarification by pinkfreud-ga on 17 Apr 2003 11:26 PDT

Apteryx,

I'm tickled "pink" to be asked for my view on an interesting subject.
I am going to take my time preparing an answer for you. Gathering my
thoughts sometimes is a process somewhat like herding cats, so I'll be
working on this for a few more hours. Just wanted you to know that I'm
not ignoring your question. ;-)

~Pink

---

Clarification of Question by apteryx-ga on 17 Apr 2003 13:40 PDT

Take your time, Pink.  I'm interested in your thinking more than I am
in your speed.  The question will be right here waiting.

Herding cats, of course, is easier if you know what they like to eat. 
The right treats are like a beacon to them.

Hello, apteryx! Thanks for flagging this question with my username. 
It's nice to be noticed. :-)

In approaching the matter of the reliability of Internet materials,
I'll give a few examples of online information that I found less than
credible, and the reasons behind my judgments of these sites.

Sometimes information seems suspect because the authority of the
originating source is doubtful, in the context of the Web site where
the information is found. For instance, I would be a bit leery of
quoting (as a sole authority) any site that uses a plenitude of
animated graphics as decoration. Likewise, there's nothing quite like
a blast of uninvited midi music to turn me off and make me want to
dump a site from consideration. Pop-up advertisements are another
warning sign that this is not the sort of site I want to inflict upon
my Google Answers customers. While I have occasionally used source
material from pages which committed the abovementioned abominations, I
generally steer clear of sites that are filled with hyperkinetic
dancing smiley-faces, midi songs that would wake the dead (or at least
make them roll in their graves, if they were musicians,) and ads that
tell me I have won something, or should punch the monkey, or have an
unprecedented opportunity to spy on everybody in the universe with a
handy-dandy webcam.

Here's a fascinating question that I tried to tackle last year; I
ended up commenting without answering:

http://answers.google.com/answers/main?cmd=threadview&id=77755

I would have loved to have answered the question by citing an
authoritative source that said the stringy things on bananas are
called "phloem bundles." In point of fact, I think this is a plausible
answer. However, I could not find this exact phrase elsewhere, and I
hesitated to cite as an authority a person who identifies herself only
as "armadillogirl" on her Web site.

Of course, much the same argument could be made against taking the
word of someone who calls herself "pinkfreud." ;-)

Earlier today, I tackled a question regarding the reasons for my
having chosen Leonardo da Vinci as the world's greatest genius ever:

http://answers.google.com/answers/main?cmd=threadview&id=191712

Since I didn't want to pull the entire answer out of my posterior, I
looked for some interesting links with which to dress up my verbiage.
Here's one site that I rejected:

Mind Publications
http://www.mindpub.com/art204.htm

I knew I was in trouble here when I came to this sentence: "He was a
world class anatomist, acrhitect, artist, botanist, city planner,
costume and stage designer, cher, humorist, emgomeer, hunoirst,
engineer, equestrian, philosopher, and phuysicist." Sheesh. Anyone who
doesn't even take the time to run a spell-check on his prose isn't a
very impressive authority on "geniouses" (his spelling.) And what does
Cher have to do with it? I bet she can't spell either.

In addition to decorative trappings and poorly-constructed content,
another warning sign that a source of information may be less than
reliable is that the site has an identifiable agenda, or takes a
well-defined stance of some kind on a controversial issue, WHETHER OR
NOT THE ISSUE IS RELATED TO THE INFORMATION I WANT TO QUOTE. (Please
excuse my shouting in the last sentence, but it's difficult to place
emphasis within the narrow constraints of GA's answer box, which does
not permit HTML conveniences such as italics, bolding, or
underlining.) If I perceive a Web site as being an issue-driven
mouthpiece for propaganda, I am going to be skeptical of accepting the
accuracy of *any* material I find on such a site, even if the material
is not part of the hot-button issue that the site is promoting (or
discouraging.) There have been several instances in the past when I
was seeking information which led me to Web sites related to religious
matters. Much to my alarm, I found that it is not uncommon for both
pro-religious and anti-religious sites to misquote scripture from the
world's great holy books. In addition, I've found numerous re-worded
poems and much misattributed prose on "agenda" sites. Even if I happen
to share the belief system of the site in question, my BS-detecting
radar goes into high gear when I visit such a site; I would hesitate
to use this kind of site as a sole authority on anything.

Humans have a tendency to accept information that is found in more
than one source. Ever since I was a rather cynical child who learned
that advertising was full of weasel-words, I've been cautious about
assuming that seeing something in print a number of times means it is
likely to be true. The Internet contains many statements of factual
error, some of which are reduplicated an astounding number of times.
One of my favorites: "A duck's quack does not echo."

Google Web Search: "a duck's quack does not echo"
://www.google.com/search?hl=en&ie=UTF-8&oe=UTF-8&q=%22a+duck%27s+quack+does+not+echo

Although you'll get 153 hits on this phrase, all the repetition proves
is that quackery of another kind echoes quite a bit. Anyone who has
ever lived near a pond where ducks tend to flock can tell you that the
echoes darn near drive you crazy. I don't know who started this
trivio-erratum, but it's pure malarkey, and you can take that to
AFLAC.

My colleage larre-ga, one of Google Answers' very finest Researchers,
took on the quack-echo question once, and I suspect that the
customer's poor rating of the answer had to do with the infernal
frequency of the misinformation that infests the Internet:

http://answers.google.com/answers/main?cmd=threadview&id=55003

Off the top of my head (which is where I keep such things,) here's a
list of factors that can influence my judgment regarding the
trustworthiness of Internet info:

1.  Overall appearance of the site
2.  Credentials (if any) of the site's author or Webmaster
3.  Types of advertising that appear on the site (pop-ups are the 
    spawn of Satan)
4.  Hosting of the site (generally, AOL = bad, .edu = good)
5.  Proper use of English (with leniency for a site run by a 
    non-native user of English)
6.  Apparent objectivity and lack of axe-grinding or propagandizing
7.  Adequate citation of sources (citing only the author's name 
    won't cut it)
8.  Links to other sites that are known to be reliable 
9.  There is no factor #9
10. Brownie points for good humor and wit (genuinely funny people are,
    in my experience, more careful with facts and tend to be more 
    trustworthy than are humorless wretches.)
 
I feel rather uneasy about preparing an answer that is mostly just me
jabbering, and doesn't have any non-Google links in it, so here are a
few good pages on the subject of evaluating the reliability of
information:

Washington State University: Savvy tips for evaluating websites
http://wsutoday.wsu.edu/completestory.asp?StoryID=527

Forbes: For 53% Reliable Information, Click Here 
http://www.forbes.com/technology/ebusiness/2003/01/31/cx_da_0131topnews.html

Johns Hopkins University: Evaluating Information Found on the Internet
http://www.library.jhu.edu/elp/useit/evaluate/

Wellness Junction: Tips for Finding Reliable Information on the
Internet
http://www.wellnessjunction.com/athome/selfcare/trust.htm

Humboldt State University Library: Using Internet Sources
http://library.humboldt.edu/infoservices/literacy/module5/mod5inet.htm

This is the seach string that brought me the best results:

Google Web Search: "reliable information" + "internet"
://www.google.com/search?hl=en&ie=UTF-8&oe=UTF-8&q=%22reliable+information%22+internet

Thank you so much for asking my opinion on this most interesting
subject. It's always a pleasure to be able to offer assistance to a
customer who has a track record of being gracious, intelligent, and
insightful (and whose name makes a super-terrific "Scrabble" word.)

* * * * * * * 

My best to you,
Pink

---

Clarification of Answer by pinkfreud-ga on 17 Apr 2003 14:51 PDT

Addendum:

Right after I finished posting my answer, I went to the forum where
Researchers relax and share useful stuff with our Google Answers
Researcher compatriots (we call ourselves GARs, which sounds kinda
pirate-y. I must post my GAR pirate poem for you someday.)

When I arrived at the forum, two private messages awaited me. Both
offered useful info related to your question.

My colleague luciaphile has shared with me this collection of links:

http://www2.vuw.ac.nz/staff/alastair_smith/evaln/evaln.htm

And my colleague tehuti offers this:

http://sosig.ac.uk/desire/internet-detective.html

Thanx, and a tip of the pink hat, to luciaphile and tehuti. One of the
things that makes Google Answers really special is the team spirit
among Researchers. Yay team! :-D

~Pink

Beautifully done, as always, Pink.  Your own opinions by themselves
were quite enough and all I asked for.  I particularly liked your list
of ten criteria and was charmed by #9, which I take to mean that
there's a factor of sheer intuition or gut instinct at work (as I
would expect of someone with your exquisite sensitivity to the
language).  The additional resources are excellent, too, and help me
think more about what I *didn't* ask you, which is how ordinary folks
with far less discernment than you can learn to evaluate the quality
of online resources.

I'd tip you with Au if I could, but you'll have to settle for Ag. 
Find me in MN and I'll buy you lunch.

Thank you very much for the five stars, the tip, and the wit. There
are times when this job is an outright pleasure. This question
provided me with one of those times.

~Pink

Oops. I have just noticed that, after having chided someone for not
using a spell-checker, I proceeded to misspell the word "colleague" in
my answer.

Gee, is my face pink. ;-)

What a wonderful answer! I agree wholeheartedly with PinkFreud:

"...there's nothing quite like a blast of uninvited midi music to turn
me off and make me want to dump a site from consideration. Pop-up
advertisements are another warning sign that this is not the sort of
site I want to inflict upon
my Google Answers customers. While I have occasionally used source
material from pages which committed the abovementioned abominations, I
generally steer clear of sites that are filled with hyperkinetic
dancing smiley-faces, midi songs that would wake the dead (or at least
make them roll in their graves, if they were musicians,) and ads that
tell me I have won something, or should punch the monkey, or have an
unprecedented opportunity to spy on everybody in the universe with a
handy-dandy webcam."

I think the midi files annoy me more than the pop-ups.  The few times
I've cited one of them as a source, I've warned the customer about the
music.  I am at a loss to understand why the designers of those sites
fail to include a "stop" button for the music - they are sooooo simple
to add.

Hello Apteryx,

Completely agreed with Pink (Factor #9 included).
I'd like to add that certain content is almost always questionable,
even if otherwise I'd trust the source. Such as:

1. Quotes and their origin.
2. Something that sounds really unbelievable.
3. A "fact" that could hurt a company's public image, or could hurt an
individual.
4. Someone wants to sell something.
5. A person or organization needs help of some sort.
6. A warning is issued, something that would somehow affect my health
or PC.
7. Something is true but cannot be explained by science.
8. Conspiracies preventing this "fact" from being published.
9. A "fact" that makes you feel really good, or really bad.
10. It's a statistic.


1. Quotes and their origin.

There are so many misquotes running around online, you really have to
validate the source, and find out to who many other people a
buzz-word, phrase or quip has been attributed to. Think of the
infamous "I created the Information Highway" (paraphrased) which is
supposedly coined by Al Gore, and makes the 'Net-savvy crowd snicker
-- however, he never coined it, nor did he _say_ he coined it. I won't
even try to find any information related to it because, yes, it
_might_ just be true he said it, or he was just quoted out of context,
who knows! But if someone asked me for the fact only, when it comes to
quotes, I'd make sure I find reliable sources.


2. Something that sounds really unbelievable.

If something is really, highly shocking, funny, but makes sense in a
strange sense of "I knew it!" or "I was always suspecting that!", then
probably it's a lie. The more a "fact" awes you the more likely you
will tell it on, and when you spread a lie it becomes a fact to many
people pretty soon, simply because they hear it over and over. Urban
legends are part of this. If your gut-reaction is "I don't believe
it", well, then don't! Better to try to track down the reality behind
the myth.


3. A "fact" that could hurt a company's public image.

For instance at this moment there's an image making its circles
through the email world. It's a sexist depiction of a couple wearing
Puma shoes, to spare you the details (and the link to the image, as
it's sort of adult-content). As soon as I got the email -- in the
style of "Check out this new Puma advertisement!" -- I was not taking
for granted that indeed the originator of this campaign is Puma (and
indeed, they now officially distance themselves from the pictures).
There's a certain mentality of paranoid, Orwellian "them-against-us"
in popular culture, which helps spread bad news about the big ones in
business.

Another variant of this is negative information about individuals.
Someone might be interested to simply ruin this person's credibility
in an act of revenge (especially after being "let off" from a job).
And bad-mouthing is so much more effective online, since it's easy to
publish information anonymously.


4. Someone wants to sell something.

Well, no need to elaborate on this one really. If someone has a strong
interest that you believe her, there might be more behind her "facts"
than meet the eye. I guess everybody knows that ads lie. Matter of
fact we take it for granted, right? So if someone tells you his new
soft drink will make you run faster than before, we simply reject this
"fact".


5. A person or organization needs help of some sort.

These hoaxes are often send out via email by the innocent, but
dangerously naive, people. The same crowd that double-clicks on links
in their browser. Now when they find out that little Billy located
somewhere in a romantic French village needs money (or loads of
post-cards) to cure his sickness, of course they want to help! They do
so by sending this message out to hundreds of others, and there's a
good chance some of them (you guessed it, the link-double-clickers)
will pass it on. Especially out of the instinct of saving money; "Well
I can't help directly but I do my share by asking others to do so".


6. A warning is issued, something that would somehow affect my health
or PC.

There's some funny emails that tell you to delete this or that file
from your computer hard-disk immediately, or else bad things could
happen. People who are naturally suspecting a virus everywhere will be
so relieved they have been informed in time, they rush to open their
Windows Explorer, and delete the file in question. Which might either
be completely harmless, or of crucial importance to the Operating
System -- and _not_ a virus at all.

So, if someone would ask me to research wether this or that file is
harmful or not, I would check only with authoritative sources, and
double-check their information.


7. Something is true but cannot be explained by science.

Really, I appreciate people who are into new age, healing crystals,
and all that. They probably live a life much happier than the cynical
ones. But if you don't even attempt to prove it, or only with methods
that in themselves are not proven, it's plain dubious to me. That
doesn't mean I cannot appreciate what I cannot understand or reason
about (like art), it just means I don't count those things as
belonging to the world of hard facts.


8. Conspiracies preventing this "fact" from being published.

Many hype their content by boasting about how this information has
been locked up so many years by the government. In general, this would
ring my alarm bell. OK, it might just be true, and I would certainly
look into it if I have the time, but mostly conspiracy theories are
just lies (entertaining ones, nevertheless).


9. A "fact" that makes you feel really good, or really bad.

People tend to believe what they _want_ to believe. Things that make
them feel good. So much spam starts with "You've won the lottery".

And then we have those people who believe everything you tell them if
it's bad news. Because they're born pessimistics, and always curious
as to why they have been spared from bad fate so far. They will almost
sigh with relief when they find out that sickness finally hit them ().

So, if a doctor tells me I'm dying tomorrow, _or_ that I don't have to
worry at all, I'm highly suspicious. (And I never win in lottery,
either.)


10. It's a statistic.

I don't really need to explain this one. There's a saying that
statistics are like bikinis. ("What they reveal is suggestive, but
what they conceal is vital.") I don't say statistics are always
questionable in content, but yes, that one _has_ to question them to
use them in research. Here especially public opinion votes, or product
comparison tests. It's always important who sponsored the poll, who's
publishing the comparison, and what other statistics are available on
the subject. Some people would believe just about anything as long as
it comes with some nice graphs and numbers.

And, since I already got carried away, I would like to quote from this
Monty Python sketch (which also includes, yes, a statistic):

"Presenter: Good evening. Tonight 'Spectrum' looks at one of the major
problems in the world today - that old vexed question of what is going
on. Is there still time to confront it, let alone solve it, or is it
too late? What are the figures, what are the facts, what do people
mean when they talk about things? Alexander Hardacre of the Economic
Affairs Bureau.

(Cut to equally intense pundit in front of a graph with three
different coloured columns with percentages at the top. He talks with
great authority)

Hardacre: In this graph, this column represents 23% of the population.
This column represents 28% of the population, and this column
represents 43% of the population.

(Cut back to presenter.) 

Presenter: Telling figures indeed, but what do they mean to you, what
do they mean to me, what do they mean to the average man in the
street? With me now is Professor Tiddles of Leeds University...

(Pull out to reveal bearded professor sitting next to presenter.) 

Presenter: ... Professor, you've spent many years researching into
things, what do you think?

Professor: I think it's too early to tell."


I hope this helps.

<< 1. Quotes and their origin. 
 
There are so many misquotes running around online, you really have to
validate the source, and find out to who many other people a
buzz-word, phrase or quip has been attributed to. Think of the
infamous "I created the Information Highway" (paraphrased) which is
supposedly coined by Al Gore, and makes the 'Net-savvy crowd snicker
-- however, he never coined it, nor did he _say_ he coined it. I won't
even try to find any information related to it because, yes, it
_might_ just be true he said it, or he was just quoted out of context,
who knows! But if someone asked me for the fact only, when it comes to
quotes, I'd make sure I find reliable sources. >>

This "quote" is indeed an urban myth; Phil Agre debunks it here:
http://commons.somewhere.com/rre/2000/RRE.Al.Gore.and.the.Inte1.html

Great answer, Pink!

This is an interesting topic to me. Determining the quality of
information is one of those things that I tend to do very quickly and
almost without thinking about it.

Most libraries have prepared worksheets for their patrons on some
basic things to look for in an internet based source:

1. Authority:
Who's presenting this information? What are their credentials? Do they
have a bias? Do they seem reliable?

2. Affiliation:
Look at the domain name (.edu; .com; .gov, etc.). If it comes from an
university, is it still a personal site? Are they selling something?
Is there information about the group or person that created the page?
Does the group or person have a bias?

3. Accuracy:
How are they backing up their information? If it's scholarly, is there
a bibliography? Spelling and correct grammar are things I like to see,
but there are instances (topics not well represented on the WWW) where
I'll take what I can get--as long as I can supplement the information
I find with something more substantive.

4. Currency:
How old is this information? Is there a date saying the site's been
updated? Are there a lot of dead links? When was the page created?

5. Purpose:
What was the aim in creating the web page? Did that effect the
content? I'm all for the objective and the unbiased, but there are
times when motivations will be significant (e.g. an organization of
sufferers of a particular illness or disease have a very personal
motive driving them to get more information than say, some student
who's put up a casual web page on the same topic for a class project).

6. Audience:
Who is this directed to? Sometimes it's useful to purposely look at
sites designed for children--when I was learning HTML, for example, I
found that those pages tended to help me a lot more than the ones
meant for adult.

7. Why:
Something I notice a lot is that people often try and approach each
and every search in the same fashion. That can be just as much a
hindrance as a help. Each search you do (and this goes for print
material as well) is different. It's good to think about why you're
looking for information in the first place, because your criteria for
inclusion will vary greatly depending upon your motivations.

Regards,
luciaphile-ga

http://www.llrx.com/features/verifying.htm
http://www.philosophy.unimelb.edu.au/reason/critical