Over the last year I've noticed that we often get invalid refers in
our web stats.  They're usually from Internet Explorer users,
typically only show 1 link from a particular URL, and often are from
sites I know are not linking to us (www.cnn.com).


i have this theory that IE is sending a referer of the current page
when the user is clicking on a bookmark or some other non-direct link
in the browser but haven't been able to back this up.   I think this
partially because I've seen some of the sites I know I've visited
refered to in the stats, as if maybe I clicked back to my site and IE
send the referer from the site i was on.


I'd like to find out more information about this problem, other people
having it, discussions about it, and ideally a MS knowledgebase
article regarding it.  I've done a lot of searching myself, but
there's millions of pages that mention "referer" and it's very hard
(for me) to find references to it.

---

Clarification of Question by davious-ga on 19 Jun 2002 09:20 PDT

I wanted to clarify.  The URLs aren't usually just a host name,
usually it's a complete URL to a specific page.  In January, we had a
hit from:

http://hoohoo.ncsa.uiuc.edu/docs/tutorials/includes.html
The user-agent was "Mozilla/4.0 (compatible; MSIE 5.01; Windows 98)"

We also had a hit from:
http://www.microsoft.com/WINDOWS2000/techinfo/reskit/en-us/iisbook/c02_tools_and_utilities_in_use.asp
User Agent "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)"

These are both sites that I can see our customers using, but they
certainly don't link to us and I think it's unlikely that they are
hardcoded as spoofed referers (although that was a good call, otwo and
arlenegreen).

It may be an issue with that specific version of IE.  I'll see if I
can find any other examples with different user agents.

Hi,

Can you check the version of IE in the logs? Is it different? Or do
you get the messages from only one version of IE?

In many bannerblocking programs you can set the referaddress yourself.
If you see many hits from cnn.com, it is probably because it has a
well known and easy to remember sitename.

One example of such a bannerblocking program is privoxy (
www.privoxy.org ). It has the option:
+hide-referer
which has the following alternatives:
* Remove completely (breaks images on some free web hosts).
* Fake as the root directory of the site (fools checks for in-site
links.)
* Fake as this web address ( default is www.google.com )

Relatively common problem...because referer spoofing is relatively
easy to accomplish. Also you can download and install such things as
A4proxy (a local proxy server) that will allow the surfer to do things
like hide their IP and change HTTP variables.

The reason people choose to do this is mostly internet
security/privacy related. Some people don't want the information about
where they have been to be listed anywhere and people who are
concerned about such things will often say that the referer.log
information is a great gaping security hole. For instance, I set up a
proxy server for our work LAN because my boss, after watching some
internet privacy guru on TV, decided that he didn't want anyone to
know what we had been up to. My business really doesn't need that kind
of bordering on the paranoid security but many do.

Another reason you might be seeing this is due to dynamic pages. 

Hopefully this will help you to analyze your logs a bit more in depth.
My best advice to you would be to search under internet security and
privacy referer logs and see what comes up. The people most likely to
be discussing the topics you are talking about would be within that
group. Perhaps some nice researcher will come along and point you in
the right direction. If no one does I'll check back later and try to
find you people to hook up with but I don't have the time right now.

Hmmmm...now that is whole 'nother kettle of fish.

Now I'm curious. I'll dig around some more tonight and see if I find
any known bugs that cause this. Passing strange, this.

It is my understanding in working with weblogs, that simply typing in
a URL (or accessing a bookmark), will report the last paged accessed
as the referer (without a direct link).  A good weblog analyzer (like
AWStats) should distinguish between these accesses (though I'm not
sure how it works).

I believe that is an issue with IE 5.01, unless you find a referer
that shows a different version.

I was seeing this phenomenon as early as 1996 or 1997 - if it's a bug,
it's been around for a very long time.