Hello,

I'm looking to build a free web mail solution for the public with some
added value services for a fee.  The service will be for my own
domains and an unbranded version to allow other webmasters to use it
for theirs.  It should also be suportive of multilingual interfaces.

I was thinking of starting out from scratch with the free standard
Linux resources which are bundled with the OS and those available for
free on the net and build on that.

I need advice as to what to focus on in terms of weaknesses in the
above.  Do I think about the resource management application which
would do the load balancing between servers and bandwidth optimization
where some of the very active users get moved to idle servers with
plenty of bandwidth and CPU resources available?  Where do I look for
proper spam protection where I give users and webmaster the ability to
contribute and alter methods of screening?  Should I invest in adding
some IQ and manageability to a few ProcMail based recipes out there? 
Where do I go for affordable or free virus protection for those mail
boxes and SMTP outbound traffic as I’d like to provide web and
SMTP/POP3/IMAP access if possible?

I have access to plenty of development resources, at a cost of course
which I would like to manage well, though start out with the right
initial building blocks to optimize my results.

Having low level access to my infrastructure is important to allow me
to fine-tune it and provide unique services to be able to compete. 
So, this makes is preferable not to get branded and closed products
unless they can’t be beat and don’t really need improving.

A response to this question is not only technical, but it is business
oriented as well.  I can use some good common sense as I’m out of that
rare common commodity these days so comments would be appreciated. :-)

Thank you.

/Lizardnation

Lizardnation-ga,

Thanks for contacting Google Answers!

First and foremost, building a web-based mail solution is not the
daunting enterprise it once used to be.  There has been a significant
amount of development, even very recently, in the e-mail sector
[particularly on Linux].   I'm not sure what level of knowledge you're
coming from, so i'll start from the beginning.

Your choice of MTA is critical here.  You want something that will be
able to function well in a high-performance environment, but you also
need something that is capable of acting well in a distributed
fashion.  The most common MTAs are Sendmail, qmail, Exim, and Postfix.
 Exim is commonly looked to at many high-profile webmail
establishments, due to a good mix of features, functionality, and
speed.

Resource management becomes a big issue when you are under the load of
several hundred users simultaneously.  Open-source products have been
developed on the software side for load balancing lately, but by far
the most common implementation seen is that of a load-balancing switch
capable of being intelligent about IMAP, SMTP, and POP3 traffic within
an internetwork and handling these connections statefully.  
Load-balancing switch manufacturers include F5 and Foundry Networks. 
The need to handle stateful connections is probably your primary
requirement at least for IMAP and POP3, simple TCP load
balancing/request stuff will probably not work in this regard.  One
thing that might be interesting to do more research on is the resource
needs of different daemons, but this doesn't speak to your original
question.  None of these items mentioned will probably be able to help
you with techniques such as "weighted" LB where you gauge the activity
of user /x/ and forward him to a server where his resources are better
managed.

As far as spam detection, Spam Assassin (http://spamassassin.org/) is
becoming a very popular system for the detection of spam even in
large, distributed systems.  It is somewhat unsuited for
high-performance networks given its dependence on Perl and inability
(at last check) to pipeline messages or thread them.  It can only
handle one message at a time per process, and this may be unsuitable
for larger networks.  It does allow customizable scoring and
independent configuration.

Virus protection - http://www.amavis.org/ is a good potential piece of
architecture to build on.  Check out the talk one of the contributors
gave at Rubi Con, http://www.amavis.org/rubi/mgp00001.html.  It is
becoming used at many sites and probably will suit your needs nicely.

As far as a core system or architecture for building your system on,
http://www.squirrelmail.org/ is used by many web hosting providers
(Dreamhost for sure and I believe several others).  It is somewhat
customizable and fairly open, so it may be a good piece for you to
build on.  Other examples are IMP (http://www.horde.org/imp/), MailMan
(http://www.endymion.com/products/mailman/), and
http://www.courier-mta.org/.

Hey, good luck with this stuff.  It looks like a very interesting
project to work on.  Lots of folks have tried to tackle this problem
using integrated and diverse approaches to the same problems.  The
largest one that tends to be mentioned in the hosting community is
performance and stateful management of connections.  It used to be the
architecture of these sorts of systems, but as tools have become more
available - and I mean REALLY recently, like the past few months -
those concerns have died down to be replaced with a strong development
effort towards customizability.

Also, another thing to consider is an LDAP-based infrastructure, check
out http://www.openldap.org/.

Let me know if I can be of further assistance.

Thanks,
arimathea-ga

Research methodology:

- Google searches on: webmail squirrelmail "mail infrastructure" MTA
- Freshmeat (http://www.freshmeat.net) searches
- Personal knowledge

MTAs
 Exim - http://www.exim.org
 Sendmail - http://www.sendmail.org
 qmail - http://www.qmail.org
 Postfix - http://www.postfix.org

Load Balancing
 Foundry Networks - http://www.foundrynet.com
 F5 - http://www.f5.com

LB software
 Eddie - http://eddie.sourceforge.net
 LVS - http://www.linuxvirtualserver.org
 balance - http://balance.sourceforge.net
 mod_backhand - http://www.backhand.org/
 Freshmeat Search - http://freshmeat.net/search/?q=load+balancing&section=projects

Webmail systems:
 Squirrelmail - http://www.squirrelmail.org/  
 IMP - http://www.horde.org/IMP/
 MailMan - http://www.endymion.com/products/mailman/

Thanks Arimathea,

Covered all points rather well.  I specially appreciated the
background on other providers what they've used.  If you have more of
that laying around, post it please. O:-)

Thank you.

/Lizardnation