I run a popular photography web site, photoSIG (www.photosig.com). 
photoSIG a Java web application running under Sun JDK 1.4.2_08 on Red Hat
Linux 9.  The servlet container is Tomcat 5.5.9.  I am having a
serious problem with memory leaks.  Specifically, after serving requests
for 12-24 hours, the application begins throwing OutOfMemoryErrors.  I
recently ran the system under Java 1.5.0_04 so I could use the jmap
utility to figure out what was using up all of the memory.  The worst
offenders were:

Size    Count   Class description
-------------------------------------------------------
704300504       445840  char[]
43027816        143114  byte[]
10528920        438705  java.lang.String
8939384 61498   * ConstMethodKlass
8192424 341351  java.util.HashMap$Entry
4433096 61498   * MethodKlass
4346784 64548   java.lang.Object[]
4107904 52500   int[]
3945912 10303   java.util.HashMap$Entry[]
3718704 232419  java.lang.Integer
3593168 90131   * SymbolKlass
3430624 5341    * ConstantPoolKlass
2458816 76838   java.sql.Timestamp
2276016 4682    * ConstantPoolCacheKlass
2147272 5341    * InstanceKlassKlass
1338520 3631    * MethodDataKlass

In other words, 670M+ of the 768M heap (Tomcat runs with -Xmx768M) was
being used up by char[] objects.  These were most likely the char[]
objects that are in the String objects, since the object counts are
close.  But this is as far as I've gotten with diagnosing the problem.
 I can't figure out where all those char[] objects are coming from or
why they aren't being reclaimed by the garbage collector.  I've not
been able to use a profiler to figure out the source of the problem. 
I can run the profiler in development, but I've not been able to
reproduce the problem in that environment.  The char[] objects
accumulate over 12 or more hours of heavy usage (4-5 requests per
second), and I don't even know if it's just a steady accumulation or
if it's one specific event that generates lots of them all at once; 
if it's one specific event, then I may not even encounter that event
in development.  And if I run the profiler in production, then the
system runs so slowly that it's unusable.

Please note that I'm not looking for guesses or general suggestions.
I'm looking for a specific explanation for why I've so many 
uncollected char[] objects or a course of action that I can take
that will lead me to such an explanation.  I can run tests on the
system and provide you with additional information if necessary.  If
you want to diagnose the problem on a production system then I can
create an account for you, but I'll only do that if you have a
diagnostic strategy that sounds good;  I won't provide accounts to
people who just want to poke around.

You wrote: "I don't even know if it's just a steady accumulation or
if it's one specific event that generates lots of them all at once; 
if it's one specific event, then I may not even encounter that event
in development."

It's significantly more likely, based on how the way your application
is used over time, that memory leaks slowly over time with each and
every request.  To prove this to yourself, I suggest monitoring your
memory usage over a period and tabulating time vs memory usage.  Plot
this against your expected memory requests in that time frame and I'd
suspect you'd fine a correlation between the two - most likely you'll
see your memory usage increase linearly with # of requests.

Unlike C/C++, your Java garbage collector should clean up dynamically
allocated memory.  However, if there is some data object which
maintains state on your server, to which memory allocated is with each
and every requests, you may find a problem.  I'm not exactly sure how
and/or why this would be done...

I'd suggest thinking in terms of what objects may be persistant --
independent and always existing regardless of whether there is a
request or not.  Is memory allocated to any of these objects when a
request does come in?  This may be some sort of data structure to
which "nodes" are added upon an incoming request.  Perhaps some "node"
needs to be removed upon satisfaction of that request.

Regards,
oilime

Thanks for your comment.

I've been monitoring the memory utilization for months.  I have about
84,000 samples at one-minute intervals for each of my four servers. 
You can download the data for one server ("andrew") here:

http://www.willisboyce.com/private/memorymonitor-andrew.zip

It's a zipped Excel spreadsheet, one row per minute.  The three
columns are free, total, and used.  When you open it, you'll see a
graph of one memory failure.  This one took about 21 hours to play
out.  The reductions in total memory occur when I restart Tomcat.

I know that something is holding memory and isn't letting it go, and I
know that the leaked memory consists of char[] objects (see above). 
What I can't
figure out is how to determine what's doing the leaking.

Here's an idea.  What if the garbage collection can't reclaim strings
that are created by Web requests that never "run to completion".  On
one hand I would test this by creating a well-behaved test load in
which every single Web request is allowed to complete.  Monitoring
memory use over a four hour period should show whether the memory
leaks a lot, a little, or not at all under this type of well-behaved
load.

If that idea turns out to have merit, the solutions may be of two kinds:

1) Exert intellectual effort to rewrite the code in such a way that
time-outs and similar events are handled with better clean-up logic.

2) Shutdown and restart the application periodically (which I'm sure
you've already instituted).

regards, mathtalk-ga

You are very close. The char[] objects do come from
String/StringBuffer creation and the JVM is refusing to free them
because something else is still referencing them. There is a tool
called hat from java.net that you can use with a profile dump (hprof)
and this will tell what other object still refers to them.

However there are going to be 3 places to look. 1) the jdbc driver
(the sql timestamp is the clue) 2) String manipulation you are already
doing (the infamous one in 1.4 was the subString and related calls,
they create a new string object for you but will not free the original
string until all subString objects are de-referenced 3) Using another
library like DOM that keeps a tree of your char references in memory
or those hashtables in the map listing.

Thanks for the suggestions.

Unfortunately I can't use HPROF because it slows down the application
too much.  By "too much" mean that the performance of the server
degrades to such an extent that it cannot serve pages;  the database
connection pool gets exhausted and nearly all requests time out. 
Nevertheless I did attempt to run the system for about an hour under
HPROF.  After about an hour, I attempted to dump the heap, and nothing
happened.  The same configuration dumped the heap in development, so I
know that I was doing it right.  The server was simply too overloaded
to do anything.

I suspect the JDBC driver also due to the large number of byte[]
objects in the heap.  These seem to be created by the MySQL JDBC
driver, which presumably uses a binary protocol to talk to the MySQL
server.  The version of the driver that I'm using, 3.0.11, is a little
out of date, but I've not found anything in the release notes for the
more recent drivers suggesting that any memory leaks have been
patched.  One thing that I did notice recently is that the application
is a little sloppy about closing PreparedStatement objects.  I cleaned
that up in development but haven't tested the change in production
yet.

Take a look at this... http://www.yourkit.com/eap-download/index.jsp
Can't hurt :-)

Thanks to those who responded.  It appears that closing the
PreparedStatement objects resolved the problem.

For more information:

http://www.photosig.com/go/forums/read?id=225164