爬虫——post请求，代理，cookie_爬虫 post代理-CSDN博客

本文链接：https://blog.csdn.net/Light__1024/article/details/88742234

本文介绍了Python爬虫中如何发起POST请求，处理Cookie以及利用代理进行网络请求。在POST请求中，重点讲解了改写参数、封装字典格式和回调函数的使用。对于Cookie，强调了其在post请求后的自动处理和callback的重要性。同时，文章提到了在爬虫项目中设置代理的方法，包括在中间件自定义类以及在settings中配置代理，确保每次请求都能通过代理进行。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

发起post请求

改写def start_requests(self):
封装参数字典格式
回调函数yield scrapy.FormRequest(url=url, formdata=data, callback=self.parse)




    class PostdemoSpider(scrapy.Spider):
        name = 'postDemo'
        # allowed_domains = ['www.baidu.com']
        start_urls = ['https://fanyi.baidu.com/sug']

        '''
        最开始是有start_request函数的，默认是get请求
        def start_requests(self):
            for url in self.start_urls:
                yield scrapy.Request(url=url,callback=self.parse)

        发起post请求常见方式：
            一定要对start_requests方法进行重写。
            Request（）方法中给method属性赋值成post
            FormRequest（）进行post请求的发送

        '''

        def start_requests(self):
            print('start_request')
            data = {'kw': 'dog'}

            for url in self.start_urls:
                yield scrapy.FormRequest(url=url, formdata=data, callback=self.parse)

        def parse(self, response):
            # print(response.text)
            pass

cookie

post请求之后自动处理cookie，直接处理相关页面即可，注意callback
一定要注意新建工程的设置


import scrapy


class DoubanSpider(scrapy.Spider):
    name = 'douban'
    # allowed_domains = ['www.douban.com']
    start_urls = ['https://accounts.douban.com/j/mobile/login/basic']

    def start_requests(self):
        print('start')
        for url in self.start_urls:
            data={
                'ck':'',
                'name':'',
                'password':'',
                'remember':'false',
                'ticket':''
            }

            yield scrapy.FormRequest(url=url,formdata=data,callback=self.parse)

    def parse(self, response):
        print('登陆')
        url='https://www.douban.com/people/193627830/'
        yield scrapy.Request(url=url, callback=self.parseBySecond)

    def parseBySecond(self,response):
        print('写入')
        with open ('./test.html','w',encoding='utf-8') as f:
            f.write(response.text)

代理

就是在中间件里自定义一个类，然后写个process_request
request.meta[‘proxy’] =‘https://代理ip'
settings里设置，每次请求都会被更改

class DoubanSpider(scrapy.Spider):
    name = 'douban'
    # allowed_domains = ['www.douban.com']
    start_urls = ['https://www.baidu.com/s?wd=ip']

    def parse(self, response):
        with open('baidu.html','w',encoding='utf-8') as f:
            f.write(response.text)

DOWNLOADER_MIDDLEWARES = {
   'three.middlewares.MyProxies': 543,
}


class MyProxies(object):
    def process_request(self, request, spider):
        request.meta['proxy'] = 'https://157.230.150.101:8080'