I am a co-admin on a webserver running redhat linux 7. A few month ago
I believe someone broke in and hijacked it. Since then the server is
running very slow. Some of the strange symtoms are:

1. certan commands in /bin/ and apache/bin always have a modification
date as of the present minute. When I try to replace a buggy ps with a
fresh copy it gets replaced with the buggy one right away.

2. when I use certain commands (such as ls, ps, rm) a quarter of the
time they go out of control and start eating up all the system memory.
For example, if an ls command goes wierd it can eat up 33% of the
memory.

I would like to know the best way to fix this problem. I'd prefer not
to reinstall redhat linux because the box is running 150+ websites
(most of them small traffic).
www.rainforestweb.org is one site we host, I provide this so you can
see just how slow it really is.

---

Request for Question Clarification by maniac-ga on 04 Nov 2003 17:29 PST

Hello Dwenaus,

Hmm. You say that you would prefer to not reinstall Red Hat, but that
is the most straight forward solution. I can guide you through a
method where you can save the contents of your hosted web sites and
replace the operating system with a more secure version. I recommend
this method since it will be far more certain to work for you.

If you readlly don't want to reinstall Red Hat, let me suggest a few
references to provide information on diagnosis and recover and then
based on what you see - we can walk through an approach to clean your
machine.

The Honeynet Project
  http://www.honeynet.org/
A group that sets up systems to capture information about exploits. In
particular, I suggest a review of a few of the "Scan of the Month"
challenges. The last one
  http://www.honeynet.org/scans/scan29/index.html
is of a Red Hat 7.2 system that was penetrated through the web server
(apache) and may be of particular interest (though the symptoms are
different). The top rated evaluations all describe ways to analyze the
system "safely" to determine the compromised parts of the system.

For other Red Hat examples, see
  http://www.honeynet.org/scans/scan18/
A Red Hat 6.2 system compromised within 8 hours of installation.
  http://www.honeynet.org/scans/scan22/
A Red Hat system compromised through the FTP server.
  http://www.honeynet.org/scans/scan25/
Another Red Hat system compromised through the web server.

Depending on how you want to proceed, please advise.
  --Maniac