I would like a small program for testing the maximum possible threads
generated through a JVM, sun 1.4.2-4 when running on a NPTL kernel
(RHEL3).
basically to create an increasing number of threads and report and 
error on failure.

I do not have a detailed knowledge of Java so a step by step guide of
how to compile and run any solution would be appreciated.

1) There are several books on this topic. Take a look at "Taming Java
Threads" and the other books listed under the "Customers who bought
this book also bought" heading of this page,

http://www.amazon.com/exec/obidos/tg/detail/-/1893115100/ref=pd_sim_books_2/002-5971162-5908802?v=glance&s=books

2) Take a look at the Volanomark benchmark which tests network
throughput with high thread counts,

http://www.volano.com/benchmarks.html

Running the scalability tests on your platform will provide an
estimate. Specifically, look at Tables 3 and 4 in the latest test
results,

http://www.volano.com/report/index.html

Table 3 will show you what options you may need to use to run a very
large number of threads with your JVM. Table 4 gives you an idea of
what other systems  are capable of.

3) I've written the Java test program below. Copy-n-paste this program
into an editor and save it as MaxThreads.java. To run this, compile
with the command

     javac MaxThreads.java

and run with

     java MaxThreads

The results will vary from system to system. I believe you'll find
that the amount of RAM in the system is the limiting factor. Depending
upon your application, you may exhaust another resource first. On my
system (RedHat 9, 512 MB) I can create around 3600 threads before
running out of memory,

main invoked
run invoked: 100
run invoked: 200
...
run invoked: 3400
run invoked: 3500
Exception: java.lang.OutOfMemoryError: unable to create new native thread

Each thread that is created has stack space associated with it. By
decreasing the amount of stac space reserved for each thread I can get
~4000 threads by running with,

     java -Xss100k MaxThreads

Again, check Table 3 of the Volanomark page for more tuning parameters
for your system. Run "java -X" for a list of parameters.

Note that this is a micro-benchmark. A real application will likely be
able to create fewer threads because it will be using memory for other
things too (with less memory available for the stack space required
per thread).

Let me know what you find on your system. 

// MaxThreads.java
import java.lang.*;
import java.util.*;

public class MaxThreads
{
    public static class Data {
        private int count = 0;

        public synchronized int increment() {
            this.count++;
            return count;
        }
        public synchronized void noop() {
            int i = this.count;
            return;
        }
    }

    public static class Worker extends Thread
    {
        private Data data;

        public Worker (Data d) {
            data = d;
        }

        public void run()
        {
            int myNum = this.data.increment();
            if ( myNum % 100 == 0) {
                System.out.println("run invoked: " + myNum);
            }

            while (true) {
                // sleep up to one second
                Random r = new Random();
                int ms = r.nextInt(1000);
                try {
                    this.sleep(ms);
                }
                catch (InterruptedException e) {
                }

                // do something
                this.data.noop();
            }
        }
    }

    public static void main(String[] args)
    {
        System.out.println("main invoked");
        Data data = new Data();

        while (true) {
            try {
                Worker w = new Worker(data);
                w.start();
            }
            catch (Throwable t) {
                System.out.println("Exception: " + t);
                System.exit(1);
            }
        }
    }
 }

Thanks for your comment, the included program worked well. For interested on a
RHEL3 machine 2GB ram, SUN 1.4.02-04 JVM default setting gave 5600 max, with
-Xss100k gave 14200 (limited by the default thread setting of the
machine) on increasing this in /proc/sys/kernel/threads-max, maximum
was 16300 with the error being cannot allocate alternate signal stack.
I had looked at the volano test previously but don't have 2 servers in
order to run the netclient and netserver option, the loop test gave
results of 12k+.
References look useful as well. Thanks for your comment, it answers my
question well. Beyond the original question my request stemmed from
the fact that the following program reported a limit of 306 threads
which seemed strange. Program for interest:-

#include <stdio.h>
#include <unistd.h>
#include <pthread.h>
#define MAX_THREADS 10000
int i;

void run(void) {
char c;
if (i < 10)
printf("Address of c = %u KB\n", (unsigned int) &c / 1024);
sleep(60 * 60);
}
int main(int argc, char *argv[]) {
int rc = 0;
  pthread_t thread[MAX_THREADS];
 printf("Creating threads ...\n");
  for (i = 0; i < MAX_THREADS && rc == 0; i++) {
   rc = pthread_create(&(thread[i]), NULL, (void *) &run, NULL);
 if (rc == 0) {
  pthread_detach(thread[i]);
 if ((i + 1) % 100 == 0)
printf("%i threads so far ...\n", i + 1);
 }
 else
printf("Failed with return code %i creating thread %i.\n",
rc, i + 1);
}
  exit(0);
}