记一次服务无法正常登陆的问题分析

于 2020-11-27 14:03:20 发布
阅读量2.2k 收藏
点赞数
CC 4.0 BY-SA版权
分类专栏： arthas jstack top
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/blog_zxb/article/details/110229805
本文记录了一次服务无法正常登陆的问题分析过程，通过查看日志、监控线程、分析jstack、研究源码和使用arthas在线诊断，最终发现是由于nginx启动用户skg-web缺少文件夹权限导致，解决方案是将文件夹权限改为属主skg-web。

摘要生成于 C知道 ，由 DeepSeek-R1 满血版支持， 前往体验 >

服务无法正常登陆的问题分析

看日志

20-11-24 00:33:23.906 ERROR [-,57ef3ff8026ee399,57ef3ff8026ee399,false] [XNIO-2 task-7][ImageUtils] download.
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
        at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51)
        at sun.nio.ch.IOUtil.write(IOUtil.java:148)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:504)
        at org.xnio.nio.NioSocketConduit.write(NioSocketConduit.java:184)
        at io.undertow.server.protocol.http.HttpResponseConduit.write(HttpResponseConduit.java:647)
        at org.xnio.conduits.AbstractStreamSinkConduit.write(AbstractStreamSinkConduit.java:55)
        at org.xnio.conduits.ConduitStreamSinkChannel.write(ConduitStreamSinkChannel.java:158)
        at io.undertow.channels.DetachableStreamSinkChannel.write(DetachableStreamSinkChannel.java:179)
        at io.undertow.server.HttpServerExchange$WriteDispatchChannel.write(HttpServerExchange.java:2059)
        at org.xnio.channels.Channels.writeBlocking(Channels.java:152)
        at io.undertow.servlet.spec.ServletOutputStreamImpl.writeTooLargeForBuffer(ServletOutputStreamImpl.java:218)
        at io.undertow.servlet.spec.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:145)
        at io.undertow.servlet.spec.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:128)
        at org.springframework.session.web.http.OnCommittedResponseWrapper$SaveContextServletOutputStream.write(OnCommittedResponseWrapper.java:614)
        at com.jw.service.admin.util.ImageUtils.download(ImageUtils.java:86)
        at com.jw.service.admin.web.NoAuthController.appDownload(NoAuthController.java:207)

20-11-24 20:25:55.637 ERROR [-,,,] [XNIO-2 Accept][server] Exception accepting request, closing server channel TCP server (NIO) <6bed2093>
java.io.IOException: Too many open files
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
        at org.xnio.nio.QueuedNioTcpServer.handleReady(QueuedNioTcpServer.java:467)
        at org.xnio.nio.QueuedNioTcpServerHandle.handleReady(QueuedNioTcpServerHandle.java:38)
        at org.xnio.nio.WorkerThread.run(WorkerThread.java:561)

看top线程数

top -Hp 9526
[skg-admin@ecs-skgapp-0005 ~]$ top -Hp 9526
top - 11:35:45 up 142 days, 19:07,  1 user,  load average: 0.06, 0.05, 0.05
Threads: 120 total,   0 running, 120 sleeping,   0 stopped,   0 zombie
%Cpu(s):  2.3 us,  2.2 sy,  0.0 ni, 95.5 id,  0.0 wa,  0.0 hi,  0.1 si,  0.0 st
KiB Mem : 16265864 total,   932272 free,  8214468 used,  7119124 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  7638548 avail Mem

看jstack栈

jstack 9526 > jstack.tmp
"XNIO-2 task-4" #82 prio=5 os_prio=0 tid=0x00007fe83407f000 nid=0x2596 runnable [0x00007fe80702e000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
        at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:593)
        at sun.security.ssl.InputRecord.read(InputRecord.java:532)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
        - locked <0x0000000099b5c940> (a java.lang.Object)
        at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
        at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
        - locked <0x0000000099b5ca00> (a sun.security.ssl.AppInputStream)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
        - locked <0x0000000099b5ce68> (a java.io.BufferedInputStream)
        at sun.net.www.MeteredStream.read(MeteredStream.java:134)
        - locked <0x0000000099b5ca18> (a sun.net.www.http.KeepAliveStream)
        at java.io.FilterInputStream.read(FilterInputStream.java:133)
        at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3393)
        at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3386)
        at com.jw.service.admin.util.ImageUtils.download(ImageUtils.java:81)
        at com.jw.service.admin.web.NoAuthController.appDownload(NoAuthController.java:207)

看源码

public static void download(HttpServletResponse response, String fileUrl, String fileName) {
        InputStream cin = null;
        ByteArrayOutputStream outStream = null;
        response.setContentType("application/octet-stream");// 指明response的返回对象是文件流
        response.setHeader("Content-Disposition", "attachment;filename=" + fileName);// 设置在下载框默认显示的文件名
        OutputStream os = null;
        try {
             URL url = new URL(fileUrl); 
            HttpURLConnection httpConn = (HttpURLConnection) url.openConnection(); 
            httpConn.connect(); 
            cin = httpConn.getInputStream(); 
            outStream = new ByteArrayOutputStream(); 
            byte[] buffer = new byte[1024]; 
            int len = 0; 
            while ((len = cin.read(buffer)) != -1) { 
                outStream.write(buffer, 0, len); 
            } 
            byte[] fileData = outStream.toByteArray(); 
            os = response.getOutputStream();
            os.write(fileData);
            os.flush();
        } catch (IOException e) {
            logger.error("download. ", e);
        } finally {
            try {
                if (os != null) {
                    os.close();
                }
            } catch (IOException e) {
                logger.error("",e);
            }
        }
    }

使用arthas在线诊断

[arthas@21740]$ watch com.jw.service.admin.util.ImageUtils download '{params,returnObj,throwExp}' -v -n 5 -x 3 '1==1'
Press Q or Ctrl+C to abort.
Affect(class count: 1 , method count: 1) cost in 218 ms, listenerId: 1
Condition express: 1==1 , result: true
ts=2020-11-27 11:39:01; [cost=650.804856ms] result=@ArrayList[
    @Object[][
        @ShiroHttpServletResponse[
            DEFAULT_SESSION_ID_PARAMETER_NAME=@String[JSESSIONID],
            context=@ServletContextImpl[io.undertow.servlet.spec.ServletContextImpl@106e2767],
            request=@ShiroHttpServletRequest[org.apache.shiro.web.servlet.ShiroHttpServletRequest@346accac],
            response=@StatHttpServletResponseWrapper[com.alibaba.druid.support.http.WebStatFilter$StatHttpServletResponseWrapper@2c315a73],
        ],
        @String[https://xxx.obs.cn-south-1.myhuaweicloud.com:443/app/20201030/other/332a19b5-f84c-40a2-b435-687ac8f818b9.1.12_2020-10-30_1112_1_%E9%BB%98%E8%AE%A4_sign.apk?AccessKeyId=K2GBDYPOVLJ4FP81RSXP&Expires=1606455540&Signature=f9knunzNfwt3ggyKXBzxd9X9uKc%3D],
        @String[app/20201030/other/332a19b5-f84c-40a2-b435-687ac8f818b9.1.12_2020-10-30_1112_1_默认_sign.apk],
    ],
    null,
    null,
]
Condition express: 1==1 , result: true

最后查看nginx日志发现，使用skg-web用户启动的nginx没有创建文件夹的权限，发现该文件夹的所属用户是root，将文件夹所属修改为skg-web解决该问题：chown skg-web:skg-web -R *

2020/11/27 13:35:14 [crit] 3556#0: *82719380 mkdir() "/usr/local/nginx/proxy_temp/3/85" failed (13: Permission denied) while reading upstream, client: 100.125.146.168, server: app.skg.com, request: "GET /admin/noAuth/appDownload?fileKey=app/20201030/other/332a19b5-f84c-40a2-b435-687ac8f818b9.1.12_2020-10-30_1112_1_%E9%BB%98%E8%AE%A4_sign.apk HTTP/1.1", upstream: "http://127.0.0.1:8185/admin/noAuth/appDownload?fileKey=app/20201030/other/332a19b5-f84c-40a2-b435-687ac8f818b9.1.12_2020-10-30_1112_1_%E9%BB%98%E8%AE%A4_sign.apk", host: "app.skg.com:8186", referrer: "https://app.skg.com/qrCodeDownload"

[root@ecs-skgapp-0001 nginx]# ll
total 40
drwxr-xr-x  2 skg-web root 4096 Nov 13 17:03 client_body_temp
drwxr-xr-x  2 root    root 4096 Nov 25 11:30 conf
drwxr-xr-x  2 skg-web root 4096 Jul  8 10:09 fastcgi_temp
drwxr-xr-x  2 root    root 4096 Jul  8 10:09 html
drwxr-xr-x  2 root    root 4096 Nov 27 13:36 logs
drwxr-xr-x 12 skg-web root 4096 Jul  8 10:09 proxy_temp
drwxr-xr-x  2 root    root 4096 Jul  8 10:09 sbin
drwxr-xr-x  2 skg-web root 4096 Jul  8 10:09 scgi_temp
drwxr-xr-x  2 root    root 4096 Jul  8 10:09 ssl
drwxr-xr-x  2 skg-web root 4096 Jul  8 10:09 uwsgi_temp
[root@ecs-skgapp-0001 nginx]# cd proxy_temp/
[root@ecs-skgapp-0001 proxy_temp]# ls
0  1  2  3  4  5  6  7  8  9
[root@ecs-skgapp-0001 proxy_temp]# ll
total 40
drwxr-xr-x  8 root root 4096 Jul  8 10:09 0
drwxr-xr-x 10 root root 4096 Jul  8 10:09 1
drwxr-xr-x  9 root root 4096 Jul  8 10:09 2
drwxr-xr-x 10 root root 4096 Jul  8 10:09 3
drwxr-xr-x 10 root root 4096 Jul  8 10:09 4
drwxr-xr-x 10 root root 4096 Jul  8 10:09 5
drwxr-xr-x 10 root root 4096 Jul  8 10:09 6
drwxr-xr-x 10 root root 4096 Jul  8 10:09 7
drwxr-xr-x  9 root root 4096 Jul  8 10:09 8
drwxr-xr-x  7 root root 4096 Jul  8 10:09 9
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值