I need advice on what web server hardware, software and settings to
use in two scenarios:

Scenario A: There are 20 files on the server, on average 30kb in size.
The web server recieves ten million requests per day. Each static file
is thus requested on average 500 000 times per day.

Scenario B: There are 20 million files on the server, on average 60kb
in size. The web server recieves one million requests per day. Each
day there are about 100 000 unique files recieving most of the
requests, but all of the files are access stochastically.

Give recommendations for optimaln hardware configuration in the two
scenarios. What is the best storage option in each case, to put the
files in a database or on disk? What operating system, web server
software and settings should be used in each of the two scenarios? If
a clustered architecture is needed, describe it.

---

Request for Question Clarification by keystroke-ga on 23 Aug 2006 06:12 PDT

I do not believe there is any such software that can push data to the
clients FASTER than internet connection allows.

Comuguy is looking along the same lines as myself. I have been
thinking about this question since the day you asked it. I wrote a
long clarification about disk drive configurations and mirroring etc.

Would you like me to answer this question in the clarification for you
to look over it and say "yes this is the type of answer I was looking
for" or would you prefer comuguy to answer this for you?

--Keystroke-ga

Hello there mooksha-ga.

I will add this comment with no suggestion to Operating Systems so
that you will have an Architectural answer instead of a 'Do it with
this OS, or do it with that OS.'

First off, when talking web servers you really do not want to do any
type of clustering on your front-end (Web Servers).  Clustering may be
a smart requirement on the backend depending on the amount of data you
are serving and the budget you are working with.

Let's look at the first scenario:
...20 files @ 30k each, 10,000,000 requests per day...
So we are working with almost 6MB of data being served 10M times a
day.  That is approximately 42,000 requests in an hour or
approximately 7,000/min or 116/sec.  In this scenario you should not
need any type of clustering.  You would need a good script in place to
make sure all you data (6MB) is replicated from a central location
(File Server) and kept up to date on your web server(s).  If you are
indeed getting 10M hits a day you probably want to have at least 2 web
servers, although I would suggest 3 if the data is that important that
10M people are accessing it in a day, in a 'Load Balanced'
configuration.  The two servers actually look as 1 to the outside
world and requests are bounced (Load Balanced) between each...

We can get VERY detailed in the design here so what I can do is offer
up the information above as a taste and if you would like more
information and further the discussion if you reply back.  I can
create some diagrams of each scenario as well as some detailed
information.  I work in the IT field and Internet Technology is my
main area of expertise.

Hi comuguy-ga,

thanks for the comment. I think I can work out the details of
load-balancing and so on for myself. But what I don't know is which
web server is preferrable when serving static files in scenarios like
the two mentioned. If there's any obscure software that can push
static files faster than Apache or similar.

What do you think?