I have created a mailing list(newsletter) program in php, and the
subscriber list has the potential of getting quite large. I wanted
some pointers as to the best way to keep the stress down on the server
while using the mail() in a loop.

Thanks,

Aaron;

Hi, 

I've written one of these that I'm working on right now, actually
re-writing in another language (Perl) to get it working again. The
list has reached over 500 and most of the last subscribers are not
getting their emails. Mine does quite a bit of processing before
sending the list, subscribers can choose to get the message daily,
weekly or pick a set of days in the week to get the messages.

The reason I'm pulling it out of PHP and processing in Perl is because
of the server settings for PHP. In the php.ini file on the server,
which tells the server how to act with PHP scripts, there is a setting
called max_execution_time. On default this is set to 30, as in 30
seconds. Most ISP's don't change this. If you are having a similar
problem, this is the cause. The program is running, and timing out.

One thing you can try is asking the ISP to reset this, but don't be
surprised if they say no. Its a safeguard for them against poorly
written scripts and one they don't want to give up. If this is your
server, then perhaps you can change it yourself. You'll need to
restart the apache server or whatever to re-set the php.ini. I don't
recommend this if anyone but you is running php scripts on the same
server.

Using envelopes is another good idea. I can point you to some good
sites on what these are. Basically its sending several messages in the
same "package" thus keeping your processing down. To do this you need
to sort your list by domain name. The links I can find for this will
be in Perl, but will describe the method well enough that if you are
reasonably proficient in PHP it shouldn't be a problem getting the
same effect.

Processing the list on a single send is pretty straight forward, as
you know. I don't know if you are grabbing this from a database or a
file, so if you want help in those areas, I need to know that. If its
a database, make sure that the email field is indexed. That will speed
up your query and keep the database from working too hard at the same
time.

Some guys drop the list to a file, and then send from the file, which
only causes another disk read, so don't do that. I'm not really sure
why they do that, and I've never gotten a straight answer that sounded
logical to me. Also, on the send loop, if you don't need anything
except the email address, the don't pull anything else. Pull as little
as possible before the loop starts, that way the query runs faster. If
you don't need it, don't pull it.

Keep as much of the "sending code" in the same function as you can.
Each call to an outside function slows you down. The trade off here is
that your code is less readable and harder to maintain. Separate
functions are easy to read and troubleshoot. At the very least, keep
the calls to includes out completely. Make one program, no includes,
to send the mail. Process everything with a different program.

Keeping the code fast is good as far as it goes, but your are still
sending basically one message at a time. So your program will pause on
each communication with the email server. There is really nothing you
can do about this, unless you own the server. Then there are methods
of talking directly to Sendmail, and I've got a link below that will
explain some of that. If this is an ISP then there's nothing you are
going to do about that. This is where you are going to slow down and
with PHP and the php.ini, time out.

If you need clarification on this, please don't hesitate to ask. Your
question is a bit vague as to what the problem might be and where you
are looking to "speed things up". Also if you need clarification
please let me know what server OS you are working on, what mail
program you are working on and whether this is your server or an
ISP's. Also, is this a database or a text file holding the email
addresses.


Links of Interest :

Email Header Info
http://www.angelfire.com/pop/cram/spam/SMTP.htm

PHP Manual / Mail / mail
http://www.zend.com/manual/function.mail.php


Perl Advisor: A MIME is a Terrible thing to Waste
http://www.samag.com/documents/s=1152/sam0104f/0104f.htm

Mail::BulkMail | Mojo Mail | An Easy And Powerful Mailing List Manager
http://mojo.skazat.com/support/documentation/Mail_Bulkmail.pm.html



Thanks for the question and good luck 

webadept-ga

---

Request for Answer Clarification by acorn71-ga on 22 Jun 2002 18:15 PDT

Hello,

Thanks for your feedback. The server is hosted by a web host and I do
not have full access, only SSH. I know it is running on Linux but I do
not know the version. I also know it is a CGI version of PHP. I am
grabbing the emails from a database. My main concern is that the
server does not time out during a mailing, thus stopping execution. I
wanted to get some advice as to the best ways to avoid this. I saw
there was a comment to my question that suggested sending the mails
using Bcc in my script, do you also reccommend this? I like your idea
of using "packages" containing the same domain name. I saw another
comment that pointed out the use of set_time_limit() and wanted to
know if you thought that might work for me in this case.

---

Clarification of Answer by webadept-ga on 22 Jun 2002 19:10 PDT

Hi again, 

The problem with set_time_limit is that if the server is running in
safe_mode, it has no effect at all. Most ISP's have their server set
in safe mode. Also, I'm not sure of this, but I believe that the
php.ini on the server that sets max_execution_time over-rides the
value set by set_time_limit if it is greater than the
max_execution_time. In other words if max_execution_time is set as 30,
and you set set_time_limit to 60, the script will be killed by the
server at 30. themastermind1-ga appears to think differently, but
since your ISP is probably running in safe_mode it won't matter
anyway.

BCC is a method of doing an easy envelope type of deal, and it may
work for you. It doesn't work for mine, and you need to set a
"receiver" in the TO field so that field isn't blank. You see mail
like this periodically where the To: says something like
list@newsgroup.com, and the personal address doesn't exist any more.
The addresses in the BCC field are stripped by the receiving server,
and the field is blank with it reaches the mail client. Doing this
will certainly help, and the commenter below ( mohsen-ga ) has a good
code example of how to do this. His comment on sending one at a time
is accurate as well, as I explained before, in the answer, that is the
main limiting factor in mail lists. The more you talk to the mail
server (in this case probably Sendmail) the more you slow your script
down and take up memory.

The only trouble with BCC is the Sendmail server you are using still
has to break up the message and send them out one at a time to the
proper domains, so your ISP is going to see the hits on their server,
and may send you a nasty note if it becomes to big of a hit, so
packaging these out in sets of ten or so is a good idea, but it will
still be noticeable, since you are rapid firing 100 sets of 10 at the
mail server.

I knew you were going to need clarification, it was my first thought
when I decided to answer the question, because every setup is
different. PHP is a great language, but its not really designed to
take on this type of task. Not large lists. I didn't find this out
until I started using it, and found its limitations simply too great
in this area. speedywise-ga points this out in his comment as well.

If your ISP has Perl available I would look into running that, but
there are limitations with ISP's there as well, and each is different.
I have the apache server set not to let CGI's run too long myself, and
I believe this is common practice. Shell scripts are good if you have
access to that on the server, again like speedywise-ga suggests. Can
be difficult to access databases with this, and difficult to write at
times. It sounds like speedywise-ga has his own server, and not an
ISP. Or perhaps speedywise-ga just has more than normal access to the
server. The ISP's I deal with normally don't allow access to shell
scripts.

I was talking with a friend about this problem and working with ISP's
and he suggested putting a script call inside one of the front pages
of the website. This call would grab the next 20 people on the list
and send them the message, updating those 20 as being sent. This of
course suggests that you get enough hits during the day and that you
aren't sending the newsletter anymore than once a week. As I saw this
suggestion, I thought it was a little much, I don't like elaborate
schemes in my code setups. I prefer easier and more straight forward
methods. Thought I would mention it though.

I would like to thank the commenters on this question, and invite
others to put in their suggestions. Really what this boils down to is
you experimenting and seeing what works for your setup.

One thing you need to do is find out your server's setup, and see what
you have to work with.

Make a single file, something called php_check.php and put this code
inside.

// Show all information, defaults to INFO_ALL
phpinfo();

// Show just the module information.
// phpinfo(8) yields identical results.
phpinfo(INFO_MODULES);

This will give you a page that lists all the settings on the server.
Call it as a URL. It will tell you max_execution_time, the safe_mode
setting, everything. ISP's don't mind this normally because it keeps
their phone from ringing all the time :-)

Hope this helps. 

webadept-ga

Hi,

As I understand from your question, you call mail() for each entry
in your address list. That would not be a good way, because every
call to mail() function, forks a sendmail process which causes
poor performance and resource utilization. A better approach is
to call mail for every chunk of say 10 entries. Then the forked sendmail
will deal with sending your message to each of 10 entries. Assuming your
list is REALY long, it would be worth too to sleep awhile when
you call mail() for say 15 times. This will let sendmail to send
the emails that you have passed to it thus more resource will be freed
(when an email is passed to sendmail, it wouldn't deliver it instantly.
becuase sendmail has to connect to each users's MTA and 
trnasfer the content. these all require time).

Putting it all together, here is what you need to implement. You can
consider it a PSEUDO code because I have not tested it:

#################################################
$to = $your_own_email_or_some_dummy_one;
foreach $entry in $mylist
{
   $numEntries++;
   if ($numEntries == 1)
      $achunk = $entry;  
   else
      $achunk .= ", $entry";
 
   if ($numEntries == 10) // let's send email to this chunk
   { 
     $bccTo = "BCC: " . $achunk;  // BCC header prevents others from 
                                  // seeing your list entries  
     mail($to, $subject, $message, $bccTo);
     $numEtnries = 0;
     $numChunks++;
    }
 
    if (numChunks == 15) // if sent 15 chunks
    {
        sleep(300);   //sleep 5 minutes
        $numChunks = 0;
    }
 }

  $bccTo = "BCC: " . $achunk;  //last chunk
  mail($to, $subject, $message, $bccTo);

################################################

You may adjust 10 and 15 to get the best result.

Recommended link: http://www.php.net/manual/en/function.mail.php

Hope it was helpful,
mohsen,

A comment on what the the answerer said. You can infact override the
max_execution_time in the php.ini file using the set_time_limit()
function in php. Giving that function a value of 0 for example, makes
the execution time unlimited. Also, it is possible to change the
values in the ini file for the local environment using the ini_set()
function.

I also use PHP on my website, and I think it's a great tool - the best
programming language for the web. However, I don't recommend using PHP
(or any other web script) for mailing lists. The reason is that web
servers (like apache) have a time limit for every web script (like CGI
or PHP) which runs as a result of an HTTP request, and these time
limits are usually quite short. When this time limit is reached, the
process is killed by the web server.

I have used another method for sending mail to a list of users, and I
have used it to send mail up to 50,000 users successfully. What I used
is a shell script, which runs in the background on a list of users,
and sends mail to each of them separately. I do recommend sending mail
to each one separately, and putting them as the recipient of the
letter (To: header). After each mail, the script "sleeps" for 1
second, in order to avoid causing too much load on the server, and let
other processes work. This script can be written in any language, as
long as it runs as a separate process and not from the web. On my
server it took about 22 hours to send this letter to 50,000 users -
about 1.5 seconds per user in average (including "sleep" for 1
second).

Good luck!
Speedywise.