I have implemented a multithreaded overlapped named pipe server on
Windows 2003 server, which opens up 26 instances of a named pipe. I
have 26 clients connecting to it, and the clients are permanently
connected. The clients are running Windows 2000 Pro SP4. The network
protocol is TCP/IP.

If I disconnect a client (for example by rebooting it or physically
disconnecting from the network), the server thread disconnects the
named pipe on the appropriate instance, then waits for a new
connection. When the client is restarted or reconnected The client can
reconnect to the pipe. This invariably works.

The server is issuing watchdog messages every 5 seconds to all connected clients.

However, unfortunately part of the network is experiencing problems.
This is being looked into. The clients on this section of the network
occasionally fail to receive the watchdog messages. I believe this is
due to the network problems, certainly the problem is not seen on
other clients.

After 5 minutes of no messages, the client closes the handle on its
end of the named pipe, then re-opens it after about 10 seconds, to try
to reconnect.

The server sees the client close the handle, so it calls
DisconnectNamedPipe, then ConnectNamedPipe. From logs, I can see that
the server does this without error and is waiting for a new
connection.

Usually this is successful. However, occasionally (say 1% of the time)
the client cannot reconnect, and GetLastError() returns 231, or
ERROR_PIPE_BUSY. The server is then sat there waiting for a client to
connect, but the client cannot connect because it says all pipe
instances are busy. The client keeps retrying, but fails.

The client can be pinged at this point, and browsed over the network,
but the named pipe seems to be permanently broken somehow.

At this point the only solution is to restart the server application -
restarting the client computer has no effect. Presumably I could also
fix it by disconnecting all pipe instances at the server, closing the
handle and recreating the named pipe, but as far as the server is
aware, there is no problem (yes, there is an unused connection but it
may be deliberate - the client may be offline for maintenance), so I
cannot detect this situation.

Brief server pseudo-code:

CreateNamedPipe()
start 26 threads, each thread:
loop:
   ConnectNamedPipe()
   Wait for (connect) event
   WriteFile/ReadFile until disconnection
   CancelIO
   DisconnectNamedPipe()
   goto loop

All return values are checked for errors.

My question is: why is the client reporting ERROR_PIPE_BUSY when the
server has called a ConnectNamedPipe()?

Are You sure that the ConnectNamedPipe is being called after the
DisconnectNamedPipe when it detects the client is closed?

Create some logging to ensure that when you see these 1% errors that
it did infact close the old pipe and create a new one.

The best I can for now..

Yes, as I show in the pseudo-code loop the DisconnectNamedPipe must
occur before the ConnectNamedPipe, and logging bears this out. The
reason it breaks out of the WriteFile/ReadFile section is usually a
"broken pipe" error when you pull the network cable or "pipe ended" if
you shut down the client (or errors to that effect, anyway), and there
is no apparent difference in the error when this problem occurs.
Unfortunately it's impossible to force this error, it happens only on
site, and it's one of those situations where you don't find out for
several hours.
Note that I don't "close" the pipe and create a new one, I simply
disconnect and reconnect an instance.
Also wouldn't ConnectNamedPipe fail if I hadn't called
DisconnectNamedPipe first? When I create the named pipe originally, I
tell it 26 instances, then immediately start threads which
ConnectNamedPipe on all of them. If one thread didn't do a
DisconnectNamedPipe, presumably the subsequent ConnectNamedPipe would
fail.

Dave

P.S. Incidentally this is written in C using MSVC 6 and the latest
service packs & SDK (I think March 2003).

You should call WaitNamedPipe() if your client's CreateFile() returns
ERROR_PIPE_BUSY.

Excerpt from MSDN:

// Try to open a named pipe; wait for it, if necessary.
while (1)
{ 
	hPipe = CreateFile(
		lpszPipename,   // pipe name
		GENERIC_READ |  // read and write access
		GENERIC_WRITE,
		0,              // no sharing
		NULL,           // default security attributes
		OPEN_EXISTING,  // opens existing pipe
		0,              // default attributes
		NULL);          // no template file

	// Break if the pipe handle is valid.
	if (hPipe != INVALID_HANDLE_VALUE)
		break;

	// Exit if an error other than ERROR_PIPE_BUSY occurs.
	if (GetLastError() != ERROR_PIPE_BUSY)
	{
		printf("Could not open pipe");
		return 0;
	}

	// All pipe instances are busy, so wait for 20 seconds.
	if (!WaitNamedPipe(lpszPipename, 20000))
	{
		printf("Could not open pipe");
		return 0;
	}
}

Thanks for your comment. 

I don't call WaitNamedPipe, because I can't have the client end just
sitting there for 20 seconds, as it has other things to do. Yes, I
realise I could do it in a thread but I didn't see the point.
Instead, if the client fails to CreateFile (for any reason), it simply
tries again every 5 seconds, basically on a WM_TIMER message.

Are you saying that WaitNamedPipe is mandatory if CreateFile fails
with ERROR_PIPE_BUSY? If so, then you may have found the problem.

Yes. I think WaitNamedPipe() is a mandatory procedure to open server
named pipe. In fact, I always call WaitNamedPipe() (and wait forever
:) before CreateFile() on client side. This is the same way 'Win32
Systems Programming' 2nd page 309-310 says.

Interesting. I did a quick Google using the terms WaitNamedPipe
mandatory and found this:
http://www.developer2.com/features/stories/33490.html.
Although the terms do not appear together, the example client code
here also calls WaitNamedPipe before CreateFile.
I don't have 'Win32 Systems Programming', I mainly use MSDN or the SDK docs. 
Perhaps WaitNamedPipe does do something "magic" that CreateFile on its
own doesn't do. If this is the case, then it looks like (yet another
!) case of Microsofts documentation being inadequate. AFAIK nowhere
does it say that WaitNamedPipe is necessary, it's there in the
samples, yes, but then so is printf and that's not going to do much ;)
WaitNamedPipe just looks like a high-efficiency way of waiting for an
instance to become free.
Really all I can do is mod the code to put in a WaitNamedPipe before
the CreateFile on the client side. I'm assuming I don't have to use 20
seconds, I can use, say, 500 milliseconds, although now I'm beginning
to wonder if 20 seconds is also mandatory....
As I mentioned before unfortunately this problem is random. Sometimes
we have weeks between occurrences. What I will do is update the code
on, say, half the clients and then see what happens. What I do know is
that there are certain clients that exhibit the problem more than
others.
So thanks, I don't yet know if this counts as THE answer, so I don't
know what happens now - it's my first Google question!
Dave

Having reflected on the above for all of 10 minutes :), I have my
doubts that this is the answer.
When the condition occurs, rebooting the clients has no effect. The
only way of clearing it is to close down the pipe at the server and
restart it. So unless calling WaitNamedPipe at a client somehow
affects the server, this cannot resolve the problem. :(

could reproduce and solve this problem. outline:

fOk = WaitNamedPipe(sz, NMPWAIT_USE_DEFAULT_WAIT); //or try other wait values!

if (fOk)  {
   hpipe = CreateFile(sz, GENERIC_READ, 0, NULL, OPEN_EXISTING, 0, NULL);

// for some reasons pipe MIGHT BE NOT ready yet (do not trust fOK) 
   while (hpipe == INVALID_HANDLE_VALUE)
      hpipe = CreateFile(sz, GENERIC_READ, 0, NULL, OPEN_EXISTING, 0, NULL);
...

It seems that this question has timed out, nevertheless i am courious
what the reason for this behaviour could be. a problem with completion
ports ? or or simply a mistake we made ? some smart people out there ?

additional info
client: xpprof, sp2, msvc6, mfc; server w2k, sp4, msvc6, plain c (ntservice)