Monday, December 07, 2009

Windows, TcpTimedWaitDelay/MaxUserPort

在服务器上装的操作系统为windowXP professional SP3,在上面跑了很多东西,包括oracle 10g,tomcat,ftp server,还有一个java程序,最近经常碰到一个问题,就是操作系统会跑出异常信息:
An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full(windows socket error:由于系统缓冲区空间不足或者队列已满,不能执行套接字上的操作.(10055),on API 'connect').
于是使用"netstat -an"来查看,觉得系统当前占用的端口也不多,参考下面:
Proto Local Address Foreign Address State
TCP 192.168.2.9:1047 192.168.2.9:1521 ESTABLISHED
TCP 192.168.2.9:1072 192.168.2.9:1521 ESTABLISHED
TCP 192.168.2.9:1073 192.168.2.9:1521 ESTABLISHED
TCP 192.168.2.9:1079 192.168.2.9:1521 ESTABLISHED
TCP 192.168.2.9:1190 192.168.2.9:1521 ESTABLISHED
TCP 192.168.2.9:1335 192.168.2.9:1521 ESTABLISHED
TCP 192.168.2.9:1340 192.168.2.9:1521 ESTABLISHED
TCP 192.168.2.9:1387 192.168.2.9:1521 ESTABLISHED
TCP 192.168.2.9:1447 192.168.2.9:8443 CLOSE_WAIT
TCP 192.168.2.9:1494 192.168.2.9:1521 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.9:1047 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.9:1072 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.9:1073 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.9:1079 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.9:1190 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.9:1335 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.9:1340 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.9:1387 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.9:1494 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.9:1592 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.9:1727 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.9:2020 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.9:3152 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.9:3153 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.9:3154 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.9:4133 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.9:4435 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.9:4457 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.9:4471 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.9:4498 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.9:4521 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.9:4537 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.9:4570 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.9:4580 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.9:4766 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.9:4774 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.9:4837 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.9:4880 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.9:4982 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.110:3103 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.110:3104 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.110:3105 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.111:1922 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.113:5047 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.116:3713 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.116:3714 TIME_WAIT
TCP 192.168.2.9:1521 192.168.2.116:3734 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.117:1324 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.117:1325 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.117:1326 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.117:2897 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.117:2898 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.117:2899 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.118:2101 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.118:2105 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.118:2171 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.118:2172 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.118:2490 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.118:2491 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.118:2714 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.121:47877 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.121:47878 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.121:47879 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.121:47880 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.121:47881 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.121:47882 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.121:47883 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.121:47884 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.121:47885 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.121:47886 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.121:47887 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.121:47888 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.121:47889 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.121:47890 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.121:47891 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.121:47892 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.121:47893 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.121:47894 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.121:47895 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.121:47896 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.121:47897 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.121:47898 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.121:47899 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.121:47900 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.121:47901 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.121:47902 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.121:47903 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.121:47904 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.121:47905 ESTABLISHED
TCP 192.168.2.9:1521 192.168.2.121:47906 ESTABLISHED
TCP 192.168.2.9:1592 192.168.2.9:1521 ESTABLISHED
TCP 192.168.2.9:1657 192.168.2.9:8443 CLOSE_WAIT
TCP 192.168.2.9:1727 192.168.2.9:1521 ESTABLISHED
TCP 192.168.2.9:2020 192.168.2.9:1521 ESTABLISHED
TCP 192.168.2.9:2156 192.168.2.9:8443 CLOSE_WAIT
TCP 192.168.2.9:3152 192.168.2.9:1521 ESTABLISHED
TCP 192.168.2.9:3153 192.168.2.9:1521 ESTABLISHED
TCP 192.168.2.9:3154 192.168.2.9:1521 ESTABLISHED
TCP 192.168.2.9:3380 192.168.2.9:8443 CLOSE_WAIT
TCP 192.168.2.9:3724 192.168.2.9:8080 CLOSE_WAIT
TCP 192.168.2.9:3740 192.168.2.9:8080 CLOSE_WAIT
TCP 192.168.2.9:3764 192.168.2.9:8080 CLOSE_WAIT
TCP 192.168.2.9:3938 192.168.2.9:4476 TIME_WAIT
TCP 192.168.2.9:3938 192.168.2.9:4982 TIME_WAIT
TCP 192.168.2.9:4132 192.168.2.9:8080 CLOSE_WAIT
TCP 192.168.2.9:4133 192.168.2.9:1521 ESTABLISHED
TCP 192.168.2.9:4159 192.168.2.9:8080 CLOSE_WAIT
TCP 192.168.2.9:4385 192.168.2.9:8080 CLOSE_WAIT
TCP 192.168.2.9:4435 192.168.2.9:1521 ESTABLISHED
TCP 192.168.2.9:4457 192.168.2.9:1521 ESTABLISHED
TCP 192.168.2.9:4471 192.168.2.9:1521 ESTABLISHED
TCP 192.168.2.9:4476 192.168.2.9:1158 TIME_WAIT
TCP 192.168.2.9:4476 222.138.226.80:2001 SYN_SENT
TCP 192.168.2.9:4498 192.168.2.9:1521 ESTABLISHED
TCP 192.168.2.9:4521 192.168.2.9:1521 ESTABLISHED
TCP 192.168.2.9:4537 192.168.2.9:1521 ESTABLISHED
TCP 192.168.2.9:4570 192.168.2.9:1521 ESTABLISHED
TCP 192.168.2.9:4580 192.168.2.9:1521 ESTABLISHED
TCP 192.168.2.9:4766 192.168.2.9:1521 ESTABLISHED
TCP 192.168.2.9:4774 192.168.2.9:1521 ESTABLISHED
TCP 192.168.2.9:4837 192.168.2.9:1521 ESTABLISHED
TCP 192.168.2.9:4880 192.168.2.9:1521 ESTABLISHED
TCP 192.168.2.9:4982 192.168.2.9:1521 ESTABLISHED
TCP 192.168.2.9:5900 192.168.2.110:3304 ESTABLISHED
TCP 192.168.2.9:5900 192.168.2.113:4931 ESTABLISHED
从上面看来,顶多也就使用了125个user port(指建立socket连接时,操作系统给客户端自动分配的端口), 然后查看系统资源占用情况,总体来看也没有出现资源不足的情况.
CPU使用:4%
PF Usage: the amount of paging files being used by the system. If your computor is running near the maximum,
you can increase the page file size.

# Totals, total for numbers of handles, threads and processed running on the computor.
[总数]
# handles, a value used to uniquely identify a resource, such as a file or registry key, so that a program can
# access it.
句柄数:55234
# threads, a object within process that run program instructions.
线程数:788
# processed, a executable program.
进程:47

# Physical Memory
# The total physical memory, also called RAM, installed on your computor.
[物理内存(K)]
总数:3134356
# Available, represents the amount of free memory that is available for use.
可用数:1445808
# System Cache, shows the current physical memory used to map pages on open files .
系统缓存:624104

# Commit Charge
# Memory allocated to programs and the operating system, because of the memory copied to the paging files,
# called virtual memory, the value listed under PEAK may exceed the maximam physical memory. The value
# for Total is the same as that dipicted in the Paging File Usage History graph.
[认可用量(K)]
总数:1971300
限制:7670780
峰值:1989332

# Kernel Memory, memory used by system kernel and device drivers.
[核心内存(K)]
总数:63468
# Paged, is the memory can be copied to paging files, thereby free the physical memory. This physical memory
# can then be used by the operating system.
分页数:46444
# Nonpaged, is the memory that remains resident in the physical memory, and will not be copied out to the
# paging files.
未分页:16996

What is paging file?
A hidden file on the hard disk that Windows uses to hold parts of programs and data files that do not fit in memory. The paging file and physical memory, or RAM, comprise virtual memory.
Windows moves data from the paging file to memory as needed and moved data from memory to the paging file to make room for new data. Paging file is also called a swap file.

What is nonpaged pool?
Operating system memory that is never paged to disk. Paging is the moving of infrequently used parts of a program's working memory from RAM to another storage medium, usually the hard dist. In Task Manager, the amount of memory used by a process, in kilobytes.

What is paged pool?
The system-allocated virtual memory that has been charged to a process and that can be paged. Paging is the moving of infrequently-used parts of the program's working memory from RAM to another storage medium, usually the hard disk.

What is virtual memory?
Temporary storage used by a computer to run programs that need more memory that it has. For example, programs could have access to 4 gigabytes of virtual memory on a computer's hard drive, even if the computer has only 32 megabytes of RAM,. The program data that does not currently fit in the computer's memory is saved into paging files.

上面解释了以下任务管理器中的性能监控的一些因素.
最后关于socket错误的原因从微软的官网找到了(http://support.microsoft.com/kb/196271).
If you try to set up TCP connections from ports that are greater than 5000, the local computer responds with the following WSAENOBUFS (10055) error message:
An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full.
Important This section, method, or task contains steps that tell you how to modify the registry. However, serious problems might occur if you modify the registry incorrectly. Therefore, make sure that you follow these steps carefully. For added protection, back up the registry before you modify it. Then, you can restore the registry if a problem occurs. For more information about how to back up and restore the registry, click the following article number to view the article in the Microsoft Knowledge Base:
322756 (http://support.microsoft.com/kb/322756/ ) How to back up and restore the registry in Windows


The default maximum number of ephemeral TCP ports is 5000 in the products that are included in the "Applies to" section. A new parameter has been added in these products. To increase the maximum number of ephemeral ports, follow these steps:

1. Start Registry Editor.
2. Locate the following subkey in the registry, and then click Parameters:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
3. On the Edit menu, click New, and then add the following registry entry:
Value Name: MaxUserPort
Value Type: DWORD
Value data: 65534
Valid Range: 5000-65534 (decimal)
Default: 0x1388 (5000 decimal)
Description: This parameter controls the maximum port number that is used when a program requests any available user port from the system. Typically, ephemeral (short-lived) ports are allocated between the values of 1024 and 5000 inclusive. After the release of security bulletin MS08-037, the behavior of Windows Server 2003 was changed to more closely match that of Windows Server 2008 and Windows Vista. For more information about Microsoft security bulletin MS08-037, click the following article numbers to view the articles in the Microsoft Knowledge Base:
951746 (http://support.microsoft.com/kb/951746/ ) MS08-037: Description of the security update for DNS in Windows Server 2008, in Windows Server 2003, and in Windows 2000 Server (DNS server-side): July 8, 2008
951748 (http://support.microsoft.com/kb/951748/ ) MS08-037: Description of the security update for DNS in Windows Server 2003, in Windows XP, and in Windows 2000 Server (client side): July 8, 2008
953230 (http://support.microsoft.com/kb/953230/ ) MS08-037: Vulnerabilities in DNS could allow spoofing
4. Exit Registry Editor, and then restart the computer.

Note An additional TCPTimedWaitDelay registry parameter determines how long a closed port waits until the closed port can be reused.

从上面可以看到,window操作系统还有一个TCPTimeWaitDelay的参数, 这个参数表示:在确定 TCP/IP 可释放已关闭连接并重用其资源前,必须经过的时间。关闭和释放之间的此时间间隔通称 TIME_WAIT 状态或两倍最大段生命周期(2MSL)状态。此时间期间,重新打开到客户机和服务器的连接的成本少于建立新连接。减少此条目的值允许 TCP/IP 更快地释放已关闭的连接,为新连接提供更多资源。如果运行的应用程序需要快速释放和创建新连接,而且由于 TIME_WAIT 中存在很多连接,导致低吞吐量,则调整此参数。
默认的MaxUserPort时1024-5000,TCPTimeWaitDelay是4分钟,所以如果在4分钟内发起了大约4000个连接,这时就会发生异常(10055).