This is a bit open ended, but I am looking for a compendium of
articles/forum threads etc that deal with the following set of issues:

What recommended optimizations should be made on Windows registeries
as well as Java invocation (memory/stack) to support a high
performance network i/o environment. We seem to be running out of
threads without hitting the peak cpu/memory on our server. Its the
number of connections that are important, not the throughput of
traffic on the connections.


Sun Java 1.4.2/Java 5 running on Windows 2003 servers
A tcp socket oriented client server environment that needs to
establish > 2500 simultaneous long lived TCP connections
Windows Hardware has sufficient memory ( > 2GB and 2 CPUs) but is
running out of  threads

---

Clarification of Question by push-ga on 22 Jan 2006 12:21 PST

An update - mostly for completion and for information to others who
may be running into this problem

We were running this program as a Windows Service. Changing it to
running it as a standalone program seems to have removed the thread
exhaustion problem. We are easily exceeding 7000 sessions on a Windows
2003 server.

We are still working on discovering why or how the windows service
manager may be handling resource allocations differently.

2500 simultaneous TCP connections each with its own thread?  Perhaps
reduce the number of thread & assign blocks of connections to a thread
pool.  They all can't process at the same time anyway & the scheduler
will be eating up a bunch of CPU it probably doesn't need to.

If these are all threads then you may be running out of stack space in
your process.  This blog
(http://blogs.msdn.com/oldnewthing/archive/2005/07/29/444912.aspx)
shows this happens around 2000 thread.

What sort of application requires 2500 simultaneous connections?  This
info would help quite a bit with suggestions.

Hi

Thanks

Yes- A non blocking i/o architecture is a possibility and under
development but we want to exhaust the limits of the one thread per
connection model before a deviation of that magnitude.

The server is a high throughput low latency messaging server. Each
client must be always connected to the server and send and recieve
messages.
Changing the -Xss flag to Java does create a variation in number of
threads supported before we get the "run out of native threads" error.

A point of interest is that running the java program as a stand alone
program versus running it as a windows service seems to better
performance ( > 2700 connections) on the former. In other words,
resource exhaustion happens faster when running as a Windows Service
in case that provides a clue.

I also understand that Windows 2000 had the per process thread limit
but 2003 should not have it down at that level.

You might want to look at JRockit...

http://www.bea.com/framework.jsp?CNT=index.htm&FP=/content/products/jrockit/

It is a JVM specfically written to address your application. It uses
an interesting many-threads to many-sockets model to get higher
scalability and performance using the old locking IO Java model. And
it is free.

You should also check out the Volano reprot at...

http://www.volano.com/report/

It tests the maximum number of connections/throughput different JVMs
and configurations can handle before breaking. It has not been updated
in a while, but it is still interesting to look at.

-josh