java springboot 大量close_wait连接 timeout waiting for connection from pool

系统在运行了6-7天后，突然出现timeout waiting for connection from pool，上次出现此问题因为时间紧迫，重启后恢复正常；但此次重启5分钟后，又出现此问题，这个问题不能再拖下去了，异常栈如下

org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
	at com.my.package.HttpClientUtil.postWithJson(HttpClientUtil.java:292)

查看关键代码发现，方法返回了CloseableHttpResponse 未进行关闭。

import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
public static CloseableHttpResponse postWithJson(String url, String jsonParam)  {
        CloseableHttpClient httpClient = HttpClients.createDefault();
        HttpPost httpPost = new HttpPost(url);
        httpPost.setEntity(new StringEntity(jsonParam, "UTF-8"));
        httpPost.setHeader("Content-Type", "application/json;charset=UTF-8");
        return httpClient.execute(httpPost);
}

查看端口情况，netstat -anp|grep port|grep -i close_wait ，发现大量close_wait。

close_wait

复习下tcp的4次挥手，客户端发起fin，则服务端进入close_wait，而笔者的情况服务端就是java，因为资源未释放，所以一直未发送FIN，也就未进入Last-Ack。注意，客户端和服务端是可以互换的，谁先发起fin，谁就是客户端。

结合代码推断，客户端发起fin，但java使用http连接后未释放，导致未发送fin，所以未进入last-ack，最终导致http 连接池满了，后续请求因为没有可用连接，在等待30秒后，爆出异常。
修改后代码

public static String postWithJson(String url, String jsonParam) {
        CloseableHttpClient httpClient = HttpClients.createDefault();
        HttpPost httpPost = new HttpPost(url);
        httpPost.setEntity(new StringEntity(jsonParam, "UTF-8"));
        httpPost.setHeader("Content-Type", "application/json;charset=UTF-8");
        //使用try---with resource语法，确保关闭response，从而发送fin，从close_wait进入last-ack
        try (CloseableHttpResponse response = httpClient.execute(httpPost)) {
            HttpEntity entity = response.getEntity();
            return EntityUtils.toString(entity);
        } catch (IOException e) {
            //log
        }
}

还有一个问题，距离上次启动已经好几天了，为什么这么长时间没有报错，而这次重启5分钟后就会报错？
这是因为调用端在200的时候，读取流，而读取流后会关闭流。

	CloseableHttpResponse response = HttpClientUtil.postWithJson();
     int statusCode = response.getStatusLine().getStatusCode();
     if (statusCode == 200) {
         HttpEntity entity = response.getEntity();
         //读取流
         String result = EntityUtils.toString(entity);
         if (result.contains("200")) {
         }
      //其它

org.apache.http.util.EntityUtils
private static String toString(
            final HttpEntity entity,
            final ContentType contentType) throws IOException {
        final InputStream inStream = entity.getContent();
        if (inStream == null) {
            return null;
        }
        try {
            
        } finally {
        //关闭了流
            inStream.close();
        }
    }

后续

在解决完后，看了网上其它案例线上大量CLOSE_WAIT的原因深入分析 ，与其中提到的第3种情形是一样的，只不过它的是开启事务后，未释放sql连接导致。

3、代码问题，MySQL连接无法释放

目前看起来应该是代码质量问题，加之本次数据有异常，触发到了以前某个没有测试到的点，目前看起来很有可能是这个原因