What is the secret behind Amazon's recommendation system?

Hello Jbigart,

The secret behind Amazon.com's recommendation system is a large data
pool on user shopping behavior, combined with relatively low-level
intelligence on the algorithmic side.

Steven Johnson, founder of Feed Magazine and author of Interface
Culture, tells the following in an interview with Amazon.com:
http://www.amazon.de/exec/obidos/tg/feature/-/217324/ref%3Ded%5Fcp%5Fle%5F1%5F4/302-1180530-0945615
"If you've built up a long purchase history with Amazon, you'll tend
to get pretty sophisticated recommendations. (...) The software
doesn't know what it's like to read a book, or what you feel like when
you read a particular book. All it knows is that people who bought
this book also bought these other ones; or that people who rated these
books highly also rated these books highly, etc. Out of that elemental
data something more nuanced can emerge--if you set up the system
correctly, and give it enough data."

Let me give a simplified example. When you log in to Amazon.com, and
only then, you will get your personal recommendations. Let's assume
you're very much into books about technology history. Promptly, you're
presented with new recommendations in just this area.

How do they know you're interested in history of technology?
- They keep all your past buys stored in their database.

How do they find out which books might interest you now?
- They compare the shopping history data of other users and find out
which patterns resemble yours.

To continue with the example, let's say you've bought "History of the
Internet". Many other people who bought "History of the Internet" also
bought "Evolution of Computing". Amazon.com's server-side software
will check if you got that one, and if not, serve it to you as
personal recommendation.

Added to that, you can also rate Amazon's recommendations. But this is
just another way of lessening the importance given to items you
shopped for when computing new recommendations, based on the same
approach.

And the more you will buy, the more refined their recommendations will
be.
Imagine you bought a cooking book by Bill Gates. You don't especially
care about new recipes, but out of your interest in computing, you
wanted to know more about what kind of food a successful computer
business man prepares for himself. If this is the only book you ever
bought, suddenly you'll be receiving off-target recommendations for
many more cooking books.
This is a far-stretched example, but it emphasizes Amazon.com doesn't
really have a clue what you might want now and in the future; all they
can do is make "educated" guesses based on topic groups and what you
wanted in the past.

I hope this explained it!

j_phillipp-
Thank you - great stuff!  The problem that I have with collaborative
filtering is that there are no random connections (or variation).  I
am still waiting for the random Amazon recommendation :) - but I guess
that kind of defeats the purpose!

Good answer but here are a few further comments:

1.  You can improve your recommendations if you let them know
what else you own, rating what you have, and telling them which 
of there recommendations are bad for you (so you can rid yourself 
of cook book recs :-)
2.  Items in your shopping cart and on your wish list effect
recommendations, too.
3.  If you click on the "Why was I recommended this" link, you
can find out exactly what items caused this recommendation.  Some
of these connections are a bit strange.

I find that because I've bought so many gifts for my whole family,
I get recommended just about everything--what can I say, I have a
pre-teen, a teenager, and an elderly MIL so apparently I am simultenously
dealing with puberty and menopause.

Puffin!  How could you *not* want cookbook recommendations?  Shocking!

missy-ga <--madly grinning cookbook collector