fails to work with tor/socks proxy

[izidan]

Would you please provide sample on how it should work using tor/socks proxy.

tried both of the following settings using .defaults() and still no luck

{ proxy: "socks://127.0.0.1:9050" }

{ agentClass: SocksProxyAgent, agentOptions: { protocol: "socks:", host: "localhost", port: 9050 } }

Hi @izidan,

If it works with request or request-promise then it'll work with Cloudscraper with very few exceptions.
This isn't one of those exceptions, those options will be passed to the underlying library as you'd expect.

Replace your calls to cloudscraper with request-promise to make sure your code works as intended with request first. There is documentation: https://github.com/request/request#requestoptions-callback

const cloudscraper = require('request-promise') // <-- Does it work in this case?

If the fault lies with request, the issue should be opened there.

[izidan]

it throws the following error when trying the following default settings even though it request module works by itself when calling directly using the same settings

{ agentClass: require('socks-proxy-agent'), agentOptions: { protocol: 'socks:', host: '127.0.0.1', port: '9050' }}

CaptchaError: captcha
at validate (.\node_modules\cloudscraper\index.js:236:11)
at onCloudflareResponse (.\node_modules\cloudscraper\index.js:197:5)
at onRequestResponse (.\node_modules\cloudscraper\index.js:176:5)
at Request. (.\node_modules\cloudscraper\index.js:137:7)
at Object.onceWrapper (events.js:286:20)
at Request.emit (events.js:198:13)
at Request. (.\node_modules\request\request.js:1161:10)
at Request.emit (events.js:198:13)
at IncomingMessage. (.\node_modules\request\request.js:1083:12)
at Object.onceWrapper (events.js:286:20)
at IncomingMessage.emit (events.js:203:15)
at endReadableNT (_stream_readable.js:1129:12)
at process._tickCallback (internal/process/next_tick.js:63:19)

[izidan]

same error as above when setup defaults with the following settings

{ agent: (new require('socks-proxy-agent'))('socks://127.0.0.1:9050') }

[izidan]

and when called with default settings being { proxy: 'socks://127.0.0.1:9050' } it throws the following error

RequestError: Error: tunneling socket could not be established, statusCode=501
at onRequestResponse (.\node_modules\cloudscraper\index.js:153:21)
at Request. (.\node_modules\cloudscraper\index.js:132:7)
at Object.onceWrapper (events.js:286:20)
at Request.emit (events.js:198:13)
at Request.onRequestError (.\node_modules\request\request.js:881:8)
at ClientRequest.emit (events.js:198:13)
at ClientRequest.onConnect (.\node_modules\tunnel-agent\index.js:168:23)
at Object.onceWrapper (events.js:286:20)
at ClientRequest.emit (events.js:198:13)
at Socket.socketOnData (_http_client.js:475:11)
at Socket.emit (events.js:198:13)
at addChunk (_stream_readable.js:288:12)
at readableAddChunk (_stream_readable.js:269:11)
at Socket.Readable.push (_stream_readable.js:224:10)
at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:94:17) +0ms

request doesn't throw exceptions when you get a CAPTCHA, Cloudscraper does
It's working with both libraries, or you wouldn't get the CAPTCHA error.

You may log the response that you're getting:

const cloudscraper = require('cloudscraper')
cloudscraper.get(url).then(console.log, error => console.error(error.response.body.toString('utf-8')))

and when called with default settings being { proxy: 'socks://127.0.0.1:9050' } it throws the following error

I'm guessing that those settings aren't working with request either.
I'm going to close this issue now since it is working when it works with the request library.

[izidan]

using const cloudscraper = require('request-promise') with both of the agent or agentClass + agentOptions options works fine as the html is returned with the captcha check, although it fails to go past the security page

Yeh, unfortunately Cloudscraper can't solve those for you but we do have some convenience methods to help solve them using a third party service. It's mentioned in the README and there are examples.

I hope that helps

[izidan]

I am really confused with your reply above, Cloudscraper already works to resolve the captcha and pass using normal http proxy yet it fails with socks proxy, and I believe this has nothing to do with request or request-promise

simply this works

node -e "require('cloudscraper').defaults({ proxy: 'http://54.37.21.29:3128' }).get('https://www.nicholashumphreys.com').then(console.log, console.error)"

whilst the same page over socks proxy fails

node -e "require('cloudscraper').defaults({ agentClass: require('socks-proxy-agent'), agentOptions: { protocol: 'socks:', host: '127.0.0.1', port: '9050' } }).get('https://www.nicholashumphreys.com').then(console.log, console.error)"

I shall leave it to you to figure out why and please reopen the issue as it is a bug in the captcha handler itself.

I am really confused with your reply above, Cloudscraper already works to resolve the captcha and pass using normal http proxy yet it fails with socks proxy, and I believe this has nothing to do with request or request-promise

I apologize if my response wasn't clear but Cloudscraper has never handled CAPTCHA, has never solved CAPTCHA ever. If you have a CAPTCHA handler, please share the code so I'll know what we're dealing with here.

whilst the same page over socks proxy fails

You're connecting to the target server using two different IP's. It's most probable that the target has blocked Tor users. I don't find it odd in the least that the web proxy gets through without a CAPTCHA but Tor gets flagged.

I shall leave it to you to figure out why and please reopen the issue as it is a bug in the captcha handler itself.

Please identify a problem with the library as I have not. I'll reopen this for a while.

[izidan]

@pro-src you are right, it seems to be an issue with cloudflare blocking the tor proxy itself even though it works via the tor browser itself, I shall close the issue and do further investigation. Thanks for your time.

If it works in the Tor Browser then I'd definitely like to have it working in Cloudscraper. The only Tor related conversation besides this issue is #202.

I'd be glad to know what your investigation yields.