Technical Alerts

TCP port exhaustion can starve all new TCP connection attempts.

Technical Alerts ID:    TFA42
Version:    4.0
Status:    Published
Published date:    09/21/2010
Updated:    02/16/2011
 

Affected products and versions

The use of a restricted TCP port range and port randomization is available on the ProxySG from SGOS v5.4.

Problem description

Symptomatically, the machine appears to stop forwarding traffic. Yet, existing established connections work. Access to the Management Console and CLI via SSH work as well. However, new TCP connections are not possible and users will be unable to obtain web pages, map CIFs shares (if CIFs is intercepted), or really access any TCP application that requires a new connection setup (e.g. 3-way handshake).

If you experience these symptoms, then the issue could be TCP port exhaustion.  TCP port exhaustion occurs when the 4-tuple that defines a TCP connection (source IP (src_ip), source port (src_port), destination IP (dst_ip), destination port (dst_port)) uses every source port available and the src_ip, dst_ip and dst_port remain constant.

It is most common to see this issue when a common server (dst_ip) and a common destination application such as HTTP (dst_port=80) or CIFS (dst_port=445) are accessed from a proxy's singular outgoing interface (src_ip). If  reflect client IP is not being used or an explicit connection is used from the client to the proxy , then the proxy will use it's own IP address as the source IP to communicate with a server/application (OCS). This sets up the limitation of 16K source ports to establish a unique 4-tuple, because the source IP, destination IP and destination port will be constants and the only variable is therefore the source port.

On the Proxy SG, the default  range of available source ports is 16K, staring at 49152 and ending at 65535. Under normal conditions, it is unlikely that this range will be exhausted since there are limitations on how many incoming client connections can be accepted.

However, under certain abnormal conditions, where TCP connections are not terminated as expected (4-way handshake), then it is possible that 4-tuples will be held up in a timed state before they are released for re-use. For example, If only one side of the TCP connection has closed , then the  TCP state will be FIN_WAIT_2. Thus, if the Proxy SG sends a FIN and receives an ACK to that FIN but never receives a FIN  from the other side of the connection, then it will hold the 4-tuple in the FIN_WAIT_2 state. There is no specification for how long a connection should be kept in FIN_WAIT_2. Thus, various products will have different timers. On the Proxy SG, the timer is set at 10 minutes by default, before the connections are released back to the pool. At this time, this is not configurable on the Proxy SG.

The other common wait state for connections in TCP is TIME_WAIT. This is the final stage of the TCP state machine and is defined to take 2 times the maximum segment lifetime. It is meant to insure that no packets remain on the segment from the prior connection before releasing the connection for re-use. This is configurable on the Proxy SG.

If many, many connections are held up in either one of these wait states then the possibility of port exhaustion increases greatly because the system must continue to use new ports in the defined range.

Additionally, the Proxy SG uses a randomization algorithm for selection of the next available source port for a connection. This is done as an additional security measure, primarily to defeat session hijacking. Blue Coat Support has determined that in cases where port exhaustion occurs then this algorithm may engender some undesirable side-effects, preventing other applications  from establishing connections besides the application affected by the original port exhaustion. This is due to the selection algorithm itself.

On account of this last issue, you may find that many or all of your applications are not able to create new connections.

In order to determine if you are faced with this issue, pull the TCP connection table from the advanced URL https://<proxy_ip:>8082/TCP/Connections. This could take a few minutes to complete as a listing. Filter the output for just the source ip, destination ip and destination port you suspect and then count the connections. If it approaches 16K, you know you have a port exhaustion issue.


 

Status

There is a new randomization algorithm being incorporated into SGOS which will avoid the global side effect that prevents other applications from creating TCP connections. The original application that experienced the port exhaustion issue must be mitigated (see below workaround steps)

Additionally, the 10 minute FIN_WAIT_2 timer will be adjustable to a lower value (one minute) which will allow the reuse of these timed out connections and free up ports.

Workaround

Here are some recommended mitigation steps you may follow to alleviate and potentially avoid this condition at the SG. In the case of misbehaving servers or network elements  that are interfering with the proper use of the TCP/IP protocol, it is best to determine the source of the issue and correct it first and foremost. Additionally, at the Proxy SG there is a knob and a switch that can be used to help mitigate the issue in the short term.

The knob increases the available port range from the 16K default to a larger number. This is done using the following hidden CLI command at the SG:

tcp-ip inet-lowport <value between 16,384 and 49,152>

This will increase the range of available source ports to the difference between 65,535 (max ports available) and this lowport specification. So, if you use 16,384 as the lowport value, then there are 48K ports in the new range.

Secondly, Blue Coat recommends that you turn off the randomization algorithm when faced with this condition. This is done with the following hidden CLI command:

tcp-ip tcp-randomize-port disable

Hopefully, this will be sufficient to mitigate the issue in the short term.

As stated, though, in the case of misbehaving servers or network devices that are causing the TCP connection table to fill up with timed entries (e.g. FIN_WAIT_2, TIME_WAIT), it is best to correct the external problem for a complete solution.

Resolution

The problem is resolved in the following versions of SGOS:

SGOS 5.5 code branch:  Fixed in SGOS 5.5.4.1.  Search for "tcp-ip tcp-fast-finwait2-recycle" in the release notes (should be the top of page 21 in the release notes PDF).  SGOS 5.5.4.1 and accompanying release notes can be downloaded at https://bto.bluecoat.com/download/product/41 .

SGOS 5.4 code branch:  Fixed in SGOS 5.4.5.1.  Search for bug 145266 in the SGOS 5.4.5.1 release notes. SGOS 5.4.5.1 and accompanying release notes can be downloaded at https://bto.bluecoat.com/download/product/17 .

For information on how to upgrade SGOS on your ProxySG, please see KB3608 titled "How do I upgrade SGOS on my ProxySG?

 


Rate this Page

Please take a moment to complete this form to help us better serve you.

Did this document help answer your question?
 
 
If you are finished providing feedback, please click the RATE CONTENT button. Otherwise, please add more detail in the following text box and then click RATE CONTENT.
 
 

Your response will be used to improve our document content.

Ask a Question