I currently have two non bonded T1's coming into a cisco 3640. I have
done multiple tests and it is apparent that multiple customers coming
from "at least different ip addresses" seem to load balance fairly
well. However, I have a single large customer which drives a large
amount of traffic. These are all HTTP requests which are done in an
"open/close" per connection -- somewhere around 40 per second.
Unfotunately, the load balancing seems to not properly balance. That
is, it prefers to use, say, 10% of one of the T1s and 90% of the
other. Then at some point in time which seems somewhat random, It
decides to switch -- use 90% of the first line and 10% of the other.
And, again much of this traffic is originated from the same ip
address, although they are distinct connections (it is not one big
keep-alive or something). I have placed the MRTG graphs of the two
individual lines and the "merged" ethernet out (ie whats on my local
network vs. the wan) at
http://www.rumordude.com/t1s.html
From these images, it should be fairly obvious that there is
individual line inversion going on here -- that is, it fills up one of
the pipes and then decides to go to the other, somewhat sporadically -
even if the one pipe is not full.
The router config is as follows:
-----------------------------------------------------
Using 1444 out of 129016 bytes
!
version 12.2
service timestamps debug uptime
service timestamps log uptime
no service password-encryption
!
hostname "Cisco3640"
!
enable secret 5 xxxx
enable password xxxx
!
ip subnet-zero
!
!
!
ip audit notify log
ip audit po max-events 100
!
call rsvp-sync
!
!
!
!
!
!
!
interface Ethernet0/0
bandwidth 10000
ip address 206.40.146.33 255.255.255.224
no ip mroute-cache
full-duplex
no cdp enable
no mop enabled
!
interface Serial0/0
ip address 204.194.10.250 255.255.255.252
no ip mroute-cache
!
interface Serial0/1
ip address 206.129.33.62 255.255.255.252
no ip mroute-cache
no cdp enable
!
interface Ethernet1/0
no ip address
no ip mroute-cache
shutdown
half-duplex
no cdp enable
!
interface Serial1/0
no ip address
no ip mroute-cache
shutdown
no cdp enable
!
interface Serial1/1
no ip address
no ip mroute-cache
shutdown
no cdp enable
!
router bgp 65387
bgp log-neighbor-changes
network 206.40.146.32 mask 255.255.255.224
neighbor 204.194.10.249 remote-as 1
neighbor 206.129.33.61 remote-as 2
!
ip classless
ip route 0.0.0.0 0.0.0.0 204.194.10.249
ip route 0.0.0.0 0.0.0.0 206.129.33.61
no ip http server
!
no cdp run
!
snmp-server group xxxx v3 auth read notify
snmp-server community xxxx RO
snmp-server community xxxx RW
no snmp-server enable traps tty
!
dial-peer cor custom
!
!
!
!
!
line con 0
exec-timeout 0 0
line aux 0
line vty 0 4
password xxxx
login
!
end
Cisco3640#show interface
Ethernet0/0 is up, line protocol is up
Hardware is AmdP2, address is 0030.945c.e701 (bia 0030.945c.e701)
Internet address is 206.40.146.33/27
MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:18, output 00:00:07, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops:
0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
271 packets input, 30209 bytes, 0 no buffer
Received 189 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected
334 packets output, 20915 bytes, 0 underruns
80 output errors, 0 collisions, 1 interface resets
0 babbles, 0 late collision, 0 deferred
80 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
Serial0/0 is up, line protocol is down
Hardware is QUICC with integrated T1 CSU/DSU
Internet address is 204.194.10.250/30
MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation HDLC, loopback not set
Keepalive set (10 sec)
Last input never, output 00:00:02, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops:
0
Queueing strategy: weighted fair
Output queue: 0/1000/64/0 (size/max total/threshold/drops)
Conversations 0/1/256 (active/max active/max total)
Reserved Conversations 0/0 (allocated/max allocated)
Available Bandwidth 1158 kilobits/sec
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
29 input errors, 0 CRC, 29 frame, 0 overrun, 0 ignored, 0 abort
258 packets output, 9544 bytes, 0 underruns
0 output errors, 0 collisions, 47 interface resets
0 output buffer failures, 0 output buffers swapped out
1 carrier transitions
DCD=up DSR=up DTR=up RTS=up CTS=up
Serial0/1 is up, line protocol is down
Hardware is QUICC with integrated T1 CSU/DSU
Internet address is 206.129.33.62/30
MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation HDLC, loopback not set
Keepalive set (10 sec)
Last input never, output 00:00:04, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops:
0
Queueing strategy: weighted fair
Output queue: 0/1000/64/0 (size/max total/threshold/drops)
Conversations 0/1/256 (active/max active/max total)
Reserved Conversations 0/0 (allocated/max allocated)
Available Bandwidth 1158 kilobits/sec
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
252 packets output, 9000 bytes, 0 underruns
0 output errors, 0 collisions, 46 interface resets
0 output buffer failures, 0 output buffers swapped out
1 carrier transitions
DCD=up DSR=up DTR=up RTS=up CTS=up
Ethernet1/0 is administratively down, line protocol is down
Hardware is AmdP2, address is 0030.945c.e711 (bia 0030.945c.e711)
MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops:
0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected
0 packets output, 0 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
Serial1/0 is administratively down, line protocol is down
Hardware is QUICC with integrated T1 CSU/DSU
MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation HDLC, loopback not set
Keepalive set (10 sec)
Last input never, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops:
0
Queueing strategy: weighted fair
Output queue: 0/1000/64/0 (size/max total/threshold/drops)
Conversations 0/0/256 (active/max active/max total)
Reserved Conversations 0/0 (allocated/max allocated)
Available Bandwidth 1158 kilobits/sec
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 packets output, 0 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
DCD=down DSR=up DTR=down RTS=down CTS=down
Serial1/1 is administratively down, line protocol is down
Hardware is QUICC with integrated T1 CSU/DSU
MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation HDLC, loopback not set
Keepalive set (10 sec)
Last input never, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops:
0
Queueing strategy: weighted fair
Output queue: 0/1000/64/0 (size/max total/threshold/drops)
Conversations 0/0/256 (active/max active/max total)
Reserved Conversations 0/0 (allocated/max allocated)
Available Bandwidth 1158 kilobits/sec
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 packets output, 0 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
DCD=down DSR=up DTR=down RTS=up CTS=down
Cisco3640#
Cisco3640#
Cisco3640#show ip route
Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B -
BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS
inter area
* - candidate default, U - per-user static route, o - ODR
P - periodic downloaded static route
Gateway of last resort is not set
206.40.146.0/27 is subnetted, 1 subnets
C 206.40.146.32 is directly connected, Ethernet0/0
Cisco3640#
-----------------------------------------------------
My theory is that the router decides that one ofthe particular lines
has the best route to the "large customer" which drives the
significant traffic and so it prefers to use that line until some
performance metric it hit where it decides it better start looking at
another.
I have examined just using static routing with two defautl gateways
and dumping BGP all together, but of course this runs the risk of a
single line going down (but these go to the same ISP anyway so the
failover situation is a bit of a far fetched scenario anyway). |
