I have a java applet that needs to upload files from a client machine
to a web server using a servlet. the problem i am having is that in
the current scheme, files larger than 17-20MB throw an out of memory
error. is there any way we can get around this problem? i will post
the client and server side code for reference.

Client Side Code:


import java.io.*;
import java.net.*;

// this class is a client that enables transfer of files from client
// to server. This client connects to a servlet running on the server
// and transmits the file.
public class fileTransferClient
{

   private static final String FILENAME_HEADER = "fileName";
   private static final String FILELASTMOD_HEADER = "fileLastMod";


   // this method transfers the prescribed file to the server.
   // if the destination directory is "", it transfers the file to
"d:\\".
   //11-21-02 Changes : This method now has a new parameter that
references the item
   //that is being transferred in the import list.
   public static String transferFile(String srcFileName, String
destFileName,
                                     String destDir, int itemID)
   {
      if (destDir.equals(""))
      {
         destDir = "E:\\FTP\\incoming\\";
      }

      // get the fully qualified filename and the mere filename.
      String fqfn = srcFileName;
      String fname =
fqfn.substring(fqfn.lastIndexOf(File.separator)+1);

      try
      {
         //importTable importer = jbInit.getImportTable();
         // create the file to be uploaded and a connection to
servlet.
         File fileToUpload = new File(fqfn);
         long fileSize = fileToUpload.length();

         // get last mod of this file.
         // The last mod is sent to the servlet as a header.
         long lastMod = fileToUpload.lastModified();
         String strLastMod = String.valueOf(lastMod);

         URL serverURL = new URL(webadminApplet.strServletURL);
         URLConnection serverCon = serverURL.openConnection();

         // a bunch of connection setup related things.
         serverCon.setDoInput(true);
         serverCon.setDoOutput(true);

         // Don't use a cached version of URL connection.
         serverCon.setUseCaches (false);
         serverCon.setDefaultUseCaches (false);

         // set headers and their values.
         serverCon.setRequestProperty("Content-Type",
                                      "application/octet-stream");
         serverCon.setRequestProperty("Content-Length",
                                     
Long.toString(fileToUpload.length()));
         serverCon.setRequestProperty(FILENAME_HEADER, destDir +
destFileName);
         serverCon.setRequestProperty(FILELASTMOD_HEADER, strLastMod);

         if (webadminApplet.DEBUG) System.out.println("Connection with
FTP server established");

         // create file stream and write stream to write file data.
         FileInputStream fis = new FileInputStream(fileToUpload);
         OutputStream os = serverCon.getOutputStream();

         try
         {
            // transfer the file in 4K chunks.
            byte[] buffer = new byte[4096];
            long byteCnt = 0;
            //long percent = 0;
            int newPercent = 0;
            int oldPercent = 0;
            while (true)
            {
               int bytes = fis.read(buffer);
               byteCnt += bytes;
               //11-21-02 :
               //If itemID is greater than -1 this is an import file
transfer
               //otherwise this is a header graphic file transfer.
               if (itemID > -1)
               {
                  newPercent = (int) ((double) byteCnt/ (double)
fileSize * 100.0);
                  int diff = newPercent - oldPercent;
                  if (newPercent == 0 || diff >= 20)
                  {
                    oldPercent = newPercent;
                    jbInit.getImportTable().displayFileTransferStatus
                                                  (itemID,
newPercent);
                  }
               }
               if (bytes < 0) break;
               os.write(buffer, 0, bytes);
            }
            os.flush();
            if (webadminApplet.DEBUG) System.out.println("No of bytes
sent: " + byteCnt);
         }
         finally
         {
            // close related streams.
            os.close();
            fis.close();
         }

         if (webadminApplet.DEBUG) System.out.println("File
Transmission complete");

         // find out what the servlet has got to say in response.
         BufferedReader reader = new BufferedReader(
                        new
InputStreamReader(serverCon.getInputStream()));
         try
         {
            String line;
            while ((line = reader.readLine()) != null)
            {
               if (webadminApplet.DEBUG) System.out.println(line);
            }
         }
         finally
         {
            // close the reader stream from servlet.
            reader.close();
         }

      } // end of the big try block.

      catch (Exception e)
      {
         System.out.println("Exception during file transfer:\n" + e);
         e.printStackTrace();
         return("FTP failed. See Java Console for Errors.");
      }  // end of catch block.

      return("File: " + fname + " successfully transferred.");

   }  // end of method transferFile().


} // end of class fileTransferClient

Server side code:

import java.io.*;
import javax.servlet.*;
import javax.servlet.http.*;
import java.util.*;
import java.net.*;

// This servlet class acts as an FTP server to enable transfer of
files
// from client side.
public class FtpServerServlet extends HttpServlet
{

   String ftpDir = "D:\\pub\\FTP\\";
   private static final String FILENAME_HEADER = "fileName";
   private static final String FILELASTMOD_HEADER = "fileLastMod";

   public void doGet(HttpServletRequest req, HttpServletResponse resp)
                                     throws ServletException,
IOException
   {
      doPost(req, resp);
   }


   public void doPost(HttpServletRequest req, HttpServletResponse
resp)
                                     throws ServletException,
IOException
   {
      // ### for now enable overwrite by default.
      boolean overwrite = true;

      // get the fileName for this transmission.
      String fileName = req.getHeader(FILENAME_HEADER);

      // also get the last mod of this file.
      String strLastMod = req.getHeader(FILELASTMOD_HEADER);

      String message = "Filename: " + fileName + " saved
successfully.";
      int status = HttpServletResponse.SC_OK;

      System.out.println("fileName from client: " + fileName);

      // if filename is not specified, complain.
      if (fileName == null)
      {
         message = "Filename not specified";
         status = HttpServletResponse.SC_INTERNAL_SERVER_ERROR;
      }

      else
      {
         // open the file stream for the file about to be transferred.
         File uploadedFile = new File(fileName);

         // check if file already exists - and overwrite if necessary.
         if (uploadedFile.exists())
         {
            if (overwrite)
            {
               // delete the file.
               uploadedFile.delete();
            }
         }

         // ensure the directory is writable - and a new file may be
created.
         if (!uploadedFile.createNewFile()) 
         {
            message = "Unable to create file on server. FTP failed.";
            status = HttpServletResponse.SC_INTERNAL_SERVER_ERROR;
         }

         else
         {
            // get the necessary streams for file creation.
            FileOutputStream fos = new FileOutputStream(uploadedFile);
            InputStream is = req.getInputStream();

            try
            {
               // create a buffer. 4K!
               byte[] buffer = new byte[4096];

               // read from input stream and write to file stream.
               int byteCnt = 0;
               while (true)
               {
                  int bytes = is.read(buffer);
                  if (bytes < 0) break;
                  byteCnt += bytes;

                  // System.out.println(buffer);
                  fos.write(buffer, 0, bytes);
               }

               // flush the stream.
               fos.flush();
            } // end of try block.

            finally
            {
               is.close();
               fos.close();

               // set last mod date for this file.
               uploadedFile.setLastModified((new
Long(strLastMod)).longValue());
            } // end of finally block.

         }  // end - the new file may be created on server.

      }  // end - we have a valid filename.

      // set response headers.
      resp.setContentType("text/plain");
      resp.setStatus(status);
      if (status != HttpServletResponse.SC_OK)
      {
         getServletContext().log("ERROR: " + message);
      }

      // get output stream.
      PrintWriter out = resp.getWriter();
      out.println(message);

   }  // end of doPost().


} // end of class FtpServerServlet

Hello,

Unfortunately you're running up against a "bug" in Sun's
implementation of the URLConnection class. Basically, the problem is
that HTTP requires the content length to be set as a header. The
implementation of the URLConnection takes the view that it can't know
the content length until you've finished writing to the outputstream,
so it uses an in-memory buffer to store your data until you've
finished. This is why you are running into out of memory issues.
Actually, if you took a look at your server log you'd find that your
servlet never actually receives any data, since it's being buffered by
the client outputstream and this runs out of memory before it is able
to post the data.

The annoying thing is that since you're uploading a file you do know
the  content length, so URLConnection could write the data directly.
Unfortunately there's no way to set the content length on the
URLConnection class. (When you set it via the line below you're just
setting a header value - the URLConnection doesn't *know* that it's
the content length).

serverCon.setRequestProperty("Content-Length", 
                             Long.toString(fileToUpload.length())); 

Here's a discussion of the bug at the Sun Java developer site
(unfortunately I believe you have to register as a developer in order
to view the bug database - at least it's free).

http://developer.java.sun.com/developer/bugParade/bugs/4212479.html

Sun's position is that this isn't a bug and is fixed for HTTP 1.1, but
as you'll see from the comments, many people find that this isn't so
and want the "bug" reopened.

Also, take a look at the first 3 results from the the following google
groups search - it turns up some interesting discussions of this
problem
on the comp.lang.java.programmer newsgroup.

http://groups.google.com/groups?q=urlconnection+large+file

So, that's the bad news. If you were trying to do a post to a generic
server that you didn't control then you would have look at using
sockets - but this has problems if your applet is behind a firewall.

However, the good news is that since you're posting to your own server
you can modify your servlet code to accept the file in smaller chunks
and to stitch them back together. Let me know if you want some help on
this.

Cheers,

Eadfrith

Yes I am posting to my own server. Can you give me some ideas on what to do??

I have a couple of questions:

1. What environment are you running your applet in? Does it need to
run behind a firewall? If not then one simple solution to your problem
would be to replace the Sun URLConnection class with a free,
open-source replacement called HTTPClient:

http://www.innovation.ch/java/HTTPClient/

This class solves the problem with the buffered outputstream. Here's a
table comparing the two classes:

http://www.innovation.ch/java/HTTPClient/urlcon_vs_httpclient.html

The problem with HTTPClient is that since it's built on top of a
generic socket it cannot be used in an applet running behind a
firewall, due to the sandbox security restrictions enforced by the
browser.

2. What files are you uploading? How well do they compress, and is
their content consistent? If so then you could wrap the (buffered)
output stream in a GZIPOutputStream which would compress the file on
the fly. If files compressed down to <17Mb in all cases then this
would get over the buffer size problem. I hesitate to suggest this
since it isn't a very robust solution. However, if the nature of the
files is sufficiently constrained then it would be a simple answer.

If neither 1 or 2 apply then we need to break up the file on the
client side and send it chunks. The servlet then needs to identify the
chunks and stitch them back together. The problems would be in
maintaining state information on the server (using the session object)
and the cleaning up in cases where a client aborted the transfer
before all chunks were sent. This may be more than a $10 solution,
especially given that I won't get the $10 anyway! If nobody else steps
in and I have some time later on today I'll have a stab.

Cheers,

Eadfrith

yes the applet does run behind a firewall. i'm uploading files of
different formats (powerpoints, pdfs, jpegs, mpegs, txt, docs, xls
etc..) essentially all these files are compressable .. how does
GZIPOutputStream work.. do you know what level of compression I can
achieve using it? If you find the solution before I get an answer
here, I'll cancel this question on the site and send you a check
instead!!! haha. thanks for your help - it is much appreciated!!

another quick question - what do you feel about sending a multi part
HTTP request from the applet to the servlet - do you think that might
be a possible solution? Would you happen to have any ideas on this??
Thanks again!

Here are a couple of classes that do file transfer in chunks. I'm
sorry but I didn't have time to understand your code so I just wrote
new code - hopefully you can adapt this to suit your needs. Also, it's
had very limited testing, so beware!

If you compile and install the Server class under your servlet engine
you can test the code using the command:

> java upload.Client bigFile.pdf
http://localhost/transfer/UploadServlet

where the application context is transfer and the Server servlet has
been mapped to /UploadServlet

The code is very rudimentary. Given time it could be adapted into a
useful package. Let me know if you have any questions on the code.

No need to pay :-)

Cheers,

Eadfrith


////////////////// Code //////////////////////

package upload;

import java.io.*;
import java.net.*;
import javax.servlet.http.HttpServletResponse;

public class Client
{
  public static final int BUFFER_SIZE = 4096;

  public static final int MAX_CHUNK_SIZE = 1000 * BUFFER_SIZE; //
~4.1MB

  public static final String FILE_NAME_HEADER = "Transfer-File-Name";

  public static final String CLIENT_ID_HEADER = "Transfer-Client-ID";

  public static final String FILE_CHUNK_HEADER =
"Transfer-File-Chunk";

  public static final String FILE_CHUNK_COUNT_HEADER =
"Transfer-File-Chunk-Count";

  public static void main(String[] args)
  {
    transferFile(args[0], args[1]);
  }

  public static void transferFile(String fileName, String urlString)
  {
    try
    {
      URL url = new URL(urlString);

      File file = new File(fileName);
      long fileSize = file.length();

      FileInputStream in = new FileInputStream(file);

      int nChunks = (int)(fileSize/MAX_CHUNK_SIZE);
      if(fileSize % MAX_CHUNK_SIZE > 0)
      {
        nChunks++;
      }

      byte[] buf = new byte[BUFFER_SIZE];
      long bytesRemaining = fileSize;

      String clientID = String.valueOf((long)(Long.MIN_VALUE *
Math.random()));

      for (int i=0; i<nChunks; i++)
      {
        HttpURLConnection conn =
(HttpURLConnection)url.openConnection();
        conn.setRequestMethod("PUT");
        conn.setDoOutput(true);
        conn.setDoInput(true);
        conn.setUseCaches (false);

        int chunkSize = (int)((bytesRemaining > MAX_CHUNK_SIZE) ?
                              MAX_CHUNK_SIZE : bytesRemaining);
        bytesRemaining -= chunkSize;

        conn.setRequestProperty("Content-Type",
"application/octet-stream");
        conn.setRequestProperty("Content-Length",
String.valueOf(chunkSize));

        conn.setRequestProperty(CLIENT_ID_HEADER, clientID);
        conn.setRequestProperty(FILE_NAME_HEADER, fileName);
        conn.setRequestProperty(FILE_CHUNK_COUNT_HEADER,
String.valueOf(nChunks));
        conn.setRequestProperty(FILE_CHUNK_HEADER, String.valueOf(i));

        OutputStream out = conn.getOutputStream();
        int bytesRead = 0;
        while(bytesRead < chunkSize)
        {
          int read = in.read(buf);
          if(read == -1)
          {
            break;
          }
          else if(read > 0)
          {
            bytesRead += read;
            out.write(buf, 0, read);
          }
        }
        out.close();

        if(conn.getResponseCode() != HttpServletResponse.SC_OK)
        {
          System.err.println(conn.getResponseMessage());
        }

      }
    }
    catch (IOException e)
    {
      e.printStackTrace();
    }
  }
}


package upload;

import java.io.*;
import javax.servlet.*;
import javax.servlet.http.*;
import java.util.*;
import java.net.*;

public class Server extends HttpServlet
{
  public static final int BUFFER_SIZE = 4096;

  public void doGet(HttpServletRequest req, HttpServletResponse resp)
                                     throws ServletException,
IOException
   {
      doPut(req, resp);
   }


   public void doPost(HttpServletRequest req, HttpServletResponse
resp)
                                     throws ServletException,
IOException
   {
     doPut(req, resp);
   }

   public void doPut(HttpServletRequest req, HttpServletResponse resp)
                                     throws ServletException,
IOException
   {
     String fileName = req.getHeader(Client.FILE_NAME_HEADER);
     if(fileName == null)
     {
       resp.sendError(HttpServletResponse.SC_INTERNAL_SERVER_ERROR,
                      "Filename not specified");
     }

     String clientID = req.getHeader(Client.CLIENT_ID_HEADER);
     if(null == clientID)
     {
       resp.sendError(HttpServletResponse.SC_INTERNAL_SERVER_ERROR,
                      "Missing Client ID");
     }

     int nChunks = req.getIntHeader(Client.FILE_CHUNK_COUNT_HEADER);
     int chunk = req.getIntHeader(Client.FILE_CHUNK_HEADER);

     if(nChunks == -1 || chunk == -1)
     {
       resp.sendError(HttpServletResponse.SC_INTERNAL_SERVER_ERROR,
                      "Missing chunk information");
     }

     if(chunk == 0)
     {
       // check permission to create file here
     }

     OutputStream out = null;
     if(nChunks == 1)
     {
       out = new FileOutputStream(fileName);
     }
     else
     {
       out = new FileOutputStream(getTempFile(clientID), (chunk > 0));
     }

     InputStream in = req.getInputStream();
     byte[] buf = new byte[BUFFER_SIZE];
     int bytesRead = 0;
     while(true)
     {
       int read = in.read(buf);
       if(read == -1)
       {
         break;
       }
       else if(read > 0)
       {
         bytesRead += read;
         out.write(buf, 0, read);
       }
     }
     in.close();
     out.close();

     if(nChunks > 1 && chunk == nChunks-1)
     {
       File tmpFile = new File(getTempFile(clientID));
       File destFile = new File(fileName);
       if(destFile.exists())
       {
         destFile.delete();
       }
       if(!tmpFile.renameTo(destFile))
       {
         resp.sendError(HttpServletResponse.SC_INTERNAL_SERVER_ERROR,
                        "Unable to create file");
       }
       else
       {
         resp.setStatus(HttpServletResponse.SC_OK);
       }
     }
     else
     {
       resp.setStatus(HttpServletResponse.SC_OK);
     }
   }

   static String getTempFile(String clientID)
   {
     return "c://temp/" + clientID + ".tmp";
   }
}

That was an excellent solution. Thank you very much for your help. I
really appreciate it. I noticed that when i "netstat"ed from my
machine, a bunch of sockets had been created... do you think this
might cause some data corruption due to missing packets/chunks? Since
a lot of our files are not too big, chances of that happening are
statistically minimal but I wanted to get an idea on how connection
dependent this solution would be. But thank you .. this has definitely
given me the right idea on what to do.. was also wondering abt how to
use a multi part Http request.. do you think that would buy me
anything over the solution I already have.

Glad the code was useful. The problem with the sockets may be helped
by inserting a call to disconnet the HttpURLConnection once a chunk of
the file has been sent. From the javadoc for HttpURLConnection:

"Calling the disconnect() method may close the underlying socket if a
persistent connection is otherwise idle at that time."

The reason I didn't put the call in initially was that the javadoc for
the disconnect method states:

"Indicates that other requests to the server are unlikely in the near
future. Calling disconnect() should not imply that this
HttpURLConnection instance can be reused for other requests."

In our case we do know that we're going to be making other requests -
so I thought it would be more efficient not to call disconnect in the
hope that it would reuse the same socket. However, it seems like it's
maybe creating a socket per HttpURLConnection, which isn't then being
closed promptly. As with much of Sun's documentation, many of the
important details are missing!

As for your question about multipart requests, I don't believe it
would help. I think they're mainly intended for situations where a
user, via a standard browser, wants to upload many files as part of a
single form post. The multipart request protocol (see documentation
link below) indicates how the files should be packed into the post
data so that they can be unpacked again on the server. Unfortunately I
don't believe that it would help with our problem, which is the
buffering of output by the HttpURLConnection class. Reasonable
question though.

http://www.ietf.org/rfc/rfc1867.txt

If you do want to look further into multipart requests you may find
this link useful too:

http://www.jguru.com/faq/view.jsp?EID=160

Cheers,

Eadfrith



Multipart requests.