I know a little bit about networking, but I have a situation on my
home network that puzzles me, and which presents a small but
unacceptable security risk.  This will be a rather long question
(because I want to provide complete information to best assist a
researcher).  It's possible (and likely) that I'm raising some points
that are totally irrelevant to my two goals (stated at the bottom of
this question).  If you know what's wrong with my setup and I raise
irrelevant points, you need not bother to explain why those points are
irrelevant.  Just ignore them.

I have a somewhat sophisticated home network, with seven client
machines (Windoze NT and XP, and one kick-butt AIX client for
variety), two Linux servers, and two Linksys broadband routers.  My
question will refer to my network diagram which you can find at
http://www.users.qwest.net/~davidfilmer/home_network.jpg (this is a
high-resolution image; it may not display well in a browser, but ought
to be nice and clear if downloaded and viewed in an image viewer at a
decent size). The diagram is accurate except that my upstream router
is now a BEFSX41 instead of the BEFSR41 as shown.  There is also a
typo in the note adjacent to the BEFSR11; the IP address for port
forwarding should read "172.22.1.100" (not 172.16.1.100).  (The
qwest.net server hosting the network diagram, btw, has nothing to do
with my home network or my question.)

The webserver (the machine at lower left corner of the diagram) is a
hardened Mandrake 9.1 box running Apache 1.3x and is on its own
private network (172.22.1.xxx), which is being NAT'ted through a
Linksys BEFSR11 broadband router (this router is there only to do
hardware NAT; it does not actually connect to the broadband connection
- the upstream Linksys BEFSX41 does that).  Both the WAN and LAN
addresses of this router are hard-coded.  This machine is a "real"
webserver hosting a (very small, low traffic) public website for a
charitable orginization.  I use port forwarding (http/https on 80/443,
plus SSH on 22) through the two Linksys devices to make this possible.
 This aspect is functioning - people can see my webserver (but I can't
see it myself - more later)...

The other server (near center page) is a stock RedHat 8.0 server for
my internal network. This server can see the internet, but the
internet cannot see it (that's the plan, anyway).  And I firewall it
(ipfilter), and it does masquerading and proxy serving and the whole
thing.  And it works great.

The purpose of my network design is to isolate the webserver from the
rest of my home network.  If somebody cracks the webserver I don't
want them getting anywhere else.

The problem is that it's not working that way.  From my webserver
console, I can ping 10.11.1.1 (the upstream BEFSX41 broadband router)
and 10.11.1.99 (my internal Redhat 8.0 Linux server).  I can even SSH
to the RedHat server.  I can (from a browser) connect to and
administer the BEFSX41 (the upstream Linksys device).

I don't think that I should have any access to the 10.x.x.x network
from that Mandrake webserver.  That's the idea, anyway.

From my RedHat internal server, I cannot ping 10.11.1.2 (the WAN side
of the BEFSR11), and I cannot ping or SSH to the webserver (which is
running sshd).  So I can ping/ssh from the webserver to the RedHat
server, but not from the RedHat server to the webserver (so the
BEFSR11 seems to be properly blocking improper inbound traffic).

OK that's not good.  If somebody roots the webserver, they have
visibility to my RedHat server, which could (potentially) give access
to my entire network.  I've got hyper-secret stuff on there (well, not
really, but I don't want my stuff getting trashed or my machines being
zombie'fied or anything).

But it gets stranger (and much more annoying):  My webserver is
operating on the public IP address 12.231.186.??? which is forwarded
through the two Linksys boxes (I'm masking the last octet of my public
IP address in this posting - I'll provide it to a Google researcher
via e-mail if requested).  *YOU* could connect to that IP address just
fine.  However, *I* cannot connect to it from any machine on my home
network (it times out).

On my webserver, I can connect to http://localhost and it works fine. 
If I connect to http://172.22.1.1 then I see the admin panel for the
BEFSR11 (as expected).  HOWEVER, if I connect to http://10.11.1.2 (the
WAN side of the BEFSR11, which I shouldn't even be able to see from
the webserver) then I see... um, my webserver's own index.html (same
as locally browsing to localhost).  That's strange - it's like it's
looping somehow.  BTW, from the webserver, I am also unable to connect
to the public IP address (http://12.231.186.???).

On the BEFSR11 and BEFSX41, I have done *NOTHING NOTHING NOTHING* to
the default configuration except to specify the LAN/WAN IP addresses
and set port forwarding as specified on the network diagram (see typo
note in second paragraph).  I have no DMZ host (gads!), no static
routes, no dynamic routing, nor ANY OTHER THING that's not default,
except IP addresses and Port Forwarding.  I've even done hardware
resets to be SURE that I didn't do something strange.  Both Linksys
devices have up-to-date firmware ('1.44.2, Dec 13 2002' for the
BEFSR11 and '41 - v1.44, Nov 22 2002' for the BEFSX41)

The routing tables for the Linksys devices look like this:

Routing table for befsr11:
   Dest      Subnet         Default    Hop    Interface
  LAN IP      Mask           Gateway   Count
----------  -------------   --------- -----   ---------
   0.0.0.0  0.0.0.0         10.11.1.1   1       WAN
 10.11.1.0  255.255.255.0   0.0.0.0     1       WAN
172.22.1.0  255.255.255.0   0.0.0.0     1       LAN

Routing Table for BEFSX41:
   Dest          Subnet         Default     Hop    Interface
  LAN IP          Mask          Gateway    Count
----------   --------------  ------------  -----   ---------
0.0.0.0             0.0.0.0  12.231.186.1     1       WAN
10.11.1.0     255.255.255.0       0.0.0.0     1       LAN
12.231.186.0  255.255.255.0       0.0.0.0     1       WAN

Here's `netstat -rn` on my Mandrake webserver (this ought not matter
in any event, but here it is anyway):
Destination     Gateway         Genmask         Flags   MSS Window 
irtt Iface
172.22.1.0      0.0.0.0         255.255.255.0   U         0 0         
0 eth0
127.0.0.0       0.0.0.0         255.0.0.0       U         0 0         
0 lo
0.0.0.0         172.22.1.1      0.0.0.0         UG        0 0         
0 eth0

I'm running a Shorewall firewall on the Mandrake webserver.  However,
it makes absolutely no difference if I disable the firewall completely
(with `shorewall clear`).

Something is bad-wrong with my setup.  I have two objectives:
1) completely isolate the webserver from the rest of my network, such
that even if a cracker rooted my webserver s/he could go no farther
(no visibility to the private 10.x.x.x network).
2) enable any machine on my internal network to connect to the public
IP address of the webserver (12.231.186.???).  I want to go "out to
the internet" and back in to accomplish this (ie, I don't want to set
static routes across my Linksys devices).

Can anyone assist me please in accomplishing my objectives???  Please
feel free to post followup inquiries if I have not provided some
detail(s).  And don't worry about making me feel like an idiot by
pointing out something very obvious and stupid that I've totally
missed.

Thanks!

---

Clarification of Question by davidfilmer-ga on 10 Apr 2003 04:12 PDT

OOOOOPPPPPPSSSSS - the network diagram link should have read
http://www.users.qwest.net/~davidfilmer/images/network.jpg

Well, I got my head around it pretty quickly.  You've got some
fundamental problems with your network topology combined with the
idiosynchrocies of the Linksys routers.  So let's dive into the
problems...

First of all, and this is an aside, which doesn't answer your
question, but it my personal opinion.  I  believe your solution is too
paranoid.  Your first line of defense is the initial router.  If you
only forward port 80 and 22 to the webserver, and you are running
Apache and OpenSSH and keep up with the latest patches, the chances of
that box being rooted is negligible.  Especially since you can keep an
eye on it.  That being said, let's move on to how to solve the
problem.

A box can see everything in every subnet upstream from it.  (Where
upstream is towards the internet).  While you can set it up to blind
it, you have to do that explicitly, and it's just a better assumption
that it can see everything, so let's look at your webserver.  When you
go one level upstream from it, you have two boxes in that subnet, the
BEFSR11 and the Redhat box.  In order to isolate the webserver, you
want it to be UPSTREAM from the rest of your private network.  That
seems backwards, you would think to isolate it you want it as deep as
possible, but that's not the case.  Let's look at your problems and
what is causing them.

1)  Not being able to connect to the webserver from your private
network.  That threw me off guard for awhile, but then it occurred to
me.  The two levels of NAT are the problem.  The request bubbles up to
the WAN side of the BEFSX41 which sees a request coming from either
172.22.X.X or 192.168.X.X and either way, just views it as a request
from an unroutable address, so it drops it, which is why it is timing
out, rather than giving you a useful error.  Disabling the "Block WAN
Requests" feature on the Advanced page of the BEFSX41 might solve the
problem, but I'm not sure (I have it disabled on mine).  The best
solution, in my opinion, would be to set up your redhat box as a DNS
server, and then use split DNS, so when a request is made for the
webserevr name from your private network, it sends it to 10.11.1.2 and
it'll get forwarded to the webserver (with the given topology, which
I'll talk about later).

2)  Webserver weirdness.  When you connect to http://localhost, that
works since localhost turns into 127.0.0.1 and it never goes over the
wire.  http://172.22.1.1 connects to the admin page since the BEFSR11
is receiving the request from the _LAN_ side of the BEFSR11.  When you
connect to http://10.11.1.2, the request bubbles up to the _WAN_ side
of your BEFSR11, which says, "Oh, I need to forward WAN side requests
on port 80 to 172.22.1.100" which is why you see your website.

But really, the topology needs to change.  Looking at the diagram, the
first thing I see that needs to be done is to move the BEFSR11 between
the RedHat box and the BEFSX41.  The webserver will have a 10.11.X.X
IP, and the WAN address of the Redhat box will be the 172.22.1.100. 
The reasoning for this is that the BEFSR11 will be the firewall for
your private network, and you don't care what happens on the other
side of it.  once you do that, you don't need to run NAT on the redhat
box anymore, just it be another box on the same subnet as the rest of
your machines.  This will greatly simplify the network, and probably
solve all of your problems.

Let me know if that needs to be clarified more.  I wish I could just
draw a diagram in here.  I'll start at the Internet, and go backwards

Internet->Cable Modem->BEFSX41

Now, the BEFSX41 will have the webserver and the BEFSR11 hanging off
of it.

Your BEFSR11 will have the uplink port of the switch plugged into it's
one port, and then plug all of the other machines, redhat box
included, into the switch.  The private network is completely isolated
from the webserver (they have the BEFSR11 between them).  This is the
standard method for isolating webservers in networks.

Hope this helps, lemme know if you need any clarification.

You need logs, logs, and more logs.  Analyze with a sniffer, on each
side of each device.  Look
for overzealous NAT rules, NAT rules that hit in the wrong order, or
translations that never happen.
Make sure the traffic passing through each device is really passing on
the rules you're expecting.
Think about how each device will route the packets and the responses
to the packets.

  Personally, I would have made the Linux box a plain firewall with
three interfaces, one for the
web server, and one for your 192.168.1 network.  The Samba server
should be on a machine on
the 192.168.1 network.  Just keep the linux firewall stable and with a
very small and
understandable (verifiable) rule base.