webMagic-0.7.3出现javax.net.ssl.SSLException: Received fatal alert: protocol_version错误

最新推荐文章于 2025-06-11 19:44:53 发布

想养一只雪狐

最新推荐文章于 2025-06-11 19:44:53 发布

阅读量4.9k

点赞数 3

CC 4.0 BY-SA版权

分类专栏：其他文章标签： webMagic

本文链接：https://blog.csdn.net/kswkly/article/details/84184620

其他专栏收录该内容

16 篇文章

订阅专栏

针对WebMagic-0.7.3版本仅支持TLSv1的问题，本文介绍了一种解决方案，通过修改源码中的HttpClientGenerator与HttpClientDownloader类，使爬虫能够支持TLSv1.2，从而访问更多站点。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

WebMagic-0.7.3版本默认的HttpClient只会用TLSv1去请求，对于某些只支持TLS1.2的站点（例如 https://juejin.im/），就会报错：

javax.net.ssl.SSLException: Received fatal alert: protocol_version
	at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
	at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
	at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2023)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1125)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at us.codecraft.webmagic.downloader.HttpClientDownloader.download(HttpClientDownloader.java:85)

在未来的新版本中将会解决这个问题，现在作者的解决方案如下：
1、首先复制源码中的 HttpClientGenerator 与 HttpClientDownloader 到自己的项目中。
在这里插入图片描述
2、修改 HttpClientGenerator 类代码中的buildSSLConnectionSocketFactory 方法，该为如下即可。

private SSLConnectionSocketFactory buildSSLConnectionSocketFactory() {
	try {
		return new SSLConnectionSocketFactory(createIgnoreVerifySSL(), new String[]{"SSLv3", "TLSv1", "TLSv1.1", "TLSv1.2"},
                null,
                new DefaultHostnameVerifier()); // 优先绕过安全证书
	} catch (KeyManagementException e) {
        logger.error("ssl connection fail", e);
    } catch (NoSuchAlgorithmException e) {
        logger.error("ssl connection fail", e);
    }
	return SSLConnectionSocketFactory.getSocketFactory();
}

3、修改 HttpClientDownloader 中引用的 HttpClientGenerator 为你修改后的类。
4、设置爬虫 Spider 的 Downloader 为你修改的 HttpClientDownloader如下：。

Spider.create(new GithubRepoPageProcessor()).setDownloader(new HttpClientDownloader()).addUrl("https://github.com/code4craft").thread(5).run();