asyncio aiohttp请求解决API接口耗时问题

HADLM

已于 2022-08-17 10:10:41 修改

阅读量927

点赞数

CC 4.0 BY-SA版权

分类专栏： python 文章标签： asyncio

于 2021-05-28 09:05:53 首次发布

本文链接：https://blog.csdn.net/qq_42631707/article/details/117351228

python 专栏收录该内容

32 篇文章

订阅专栏

本文讲述了如何通过Python的asyncio库和aiohttp实现协程并发请求，从而显著减少大屏API接口的总耗时。原始接口耗时6-7秒，通过并发处理，将耗时降低到约3秒。主要方法是将第二个接口的多次请求转换为并发IO操作，实现了接口调用效率的大幅提升。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

现象：需要实现一个大屏的API，需要展示50-100份数据列表，单数这些数据的组成部分，由2个http接口数据才能取到，第一个接口：取的是这50-100份数据，另一个接口：根据这50-100份数据中的一部分数据去获取相关的其它数据，所以这是一个两个接口串行，并将第二个接口串行循环完成后的过程。

问题：第一个接口的耗时大概是2-3S，第二个接口的耗时大概是0.3S，这样实现后，实现的这个大屏的API接口，最低耗时也要达到5-6s，经常耗时7-8s，耗时太长，不可接受。

解决方案：第一个接口的耗时没办法再进行缩减，只能从第二接口下手，如果能把第二个接口的时间缩减到一秒内，这样是能够接受的。

采用：python asyncio请求有关的依赖aiohttp。

通过协程的概念，完成IO请求的并发。

@action(methods=['get'], detail=False)
    def list_alarm_limit30(self, request):
        num_objs = KeyValueConfig.objects.filter(key='alarm_number')
        if not num_objs:
            return Response({
                "result": False,
                "code": "NO",
                "message": "请到后台配置告警数量key:alarm_number",
                "data": []
            })
        num_obj = num_objs[0].value
        try:
            num_obj = int(num_obj)
        except Exception as e:
            num_obj = 100
            logger.error('配置告警数量的值错误：{0}'.format(str(e)))
        biz_data = search_business(params={"bk_username": admin_role if admin_role else "admin"})
        biz_dict = {i['bk_biz_id']: i['bk_biz_name'] for i in biz_data}
        alarm_time = time.time()
        data = list_alarm_instance(bk_biz_ids=[i['bk_biz_id'] for i in biz_data], limit=num_obj)['data']
        print('告警列表获取耗时：{0}'.format(time.time() - alarm_time))
        data = sorted(data, key=lambda keys: keys['id'], reverse=True)
        data = data[:num_obj]
        alarm_data = []
        tasks_list = []
        # 创建一个asyncio事件任务循环（协程）
        loop = asyncio.new_event_loop()
        asyncio.set_event_loop(loop)
        for i in data:
            source_start_time = datetime.datetime.strptime(i['begin_time'][:19], '%Y-%m-%d %H:%M:%S')
            source_end_time = source_start_time + datetime.timedelta(minutes=2)
            objs = VoiceMsg.objects.filter(call_time__range=(source_start_time, source_end_time))
            # -1 代表未配置
            # 0  通知失败
            # 1  通知成功
            voice_content = -1
            phone = None
            if objs:
                if objs.filter(status=True):
                    voice_content = 1
                    phone = objs.filter(status=True)[0].phone
                else:
                    voice_content = 0
                    phone = objs.filter(status=False)[0].phone
            end_time = datetime.datetime.strptime(i['end_time'][:19], '%Y-%m-%d %H:%M:%S')
            end_time = i['end_time'][:19] if end_time > source_start_time else '--'

            dimension_dict = i['origin_alarm']['dimension_translation']
            if 'bk_topo_node' in dimension_dict:
                dimension_dict.pop('bk_topo_node')
            if 'bk_target_cloud_id' in dimension_dict:
                dimension_dict.pop('bk_target_cloud_id')
            if 'bk_cloud_id' in dimension_dict:
                dimension_dict.pop('bk_cloud_id')
            dimension_list = ['{0}({1})'.format(i['display_name'], i['display_value'])
                              for i in dimension_dict.values()]
            dimension = ' - '.join(dimension_list)
            content = i['origin_alarm']['anomaly']['{0}'.format(i['level'])]['anomaly_message']
            data_dict = {
                "id": i['id'],
                "bk_biz_name": biz_dict.get(i['bk_biz_id']),
                "source_time": i['begin_time'][:19],
                "end_time": end_time,
                "source_name": i['origin_config']['strategy_name'],
                "alarm_type": LEVEL_TYPE.get(str(i['level'])),
                "alarm_content": {
                    "dimension": "维度信息：{0}".format(dimension),
                    "content": "告警内容：{0}".format(content),
                },

                "source_url": "{0}/o/bk_monitorv3/?bizId={1}#/event-center/detail/{2}".format(BK_PAAS_HOST,
                                                                                              i['bk_biz_id'],
                                                                                              i['id']),
                "voice_status": voice_content,
                'phone': phone
            }
			#  添加IO事件任务到循环列表
            tasks_list.append(asyncio.ensure_future(get_status_log(i['id'])))
            alarm_data.append(data_dict)
        start_time = time.time()
        # 事件任务执行 返回所有结果
        result = asyncio.get_event_loop().run_until_complete(asyncio.gather(*tasks_list))
        api_ids_list = []
        for data_obj in alarm_data:
            for re_obj in result:
                if data_obj['id'] == re_obj['event_id']:
                    data_obj['comment'] = re_obj['comment']
                    api_ids_list.append(re_obj['event_id'])
                    break
        print("日志消耗时间：{0}".format(time.time() - start_time))
        queryset = KeyValueConfig.objects.filter(key='event_ids')
        is_voice = False
        if queryset:
            database_ids_list = [int(i) for i in queryset[0].value.split(',')]
            api_alone_ids = [id_obj for id_obj in api_ids_list if id_obj not in database_ids_list]
            print(api_alone_ids)
            is_voice = True if api_alone_ids else False
        KeyValueConfig.objects.update_or_create(
            defaults={
                "key": "event_ids",
                "value": ','.join([str(i) for i in api_ids_list]),
                "mark": "当前大屏告警的ids"
            },
            key="event_ids"
        )
        flush_objs = KeyValueConfig.objects.filter(key='flush_time')
        flush_time = int(flush_objs[0].value) if flush_objs else '180000'
        return Response({"data": alarm_data, "count": len(alarm_data), "is_voice": is_voice, 'flush_time': flush_time})

具体第二个接口请求封装成IO请求

async def get_status_log(event_id):
    params = {
        'bk_app_code': settings.APP_CODE,
        'bk_app_secret': settings.SECRET_KEY,
        "bk_username": admin_role if admin_role else "admin",
        "id": event_id,
    }
    session = aiohttp.ClientSession()
    session.headers.update({'X-DATA-REQUEST-ID': get_request_id()})
    session.headers.update({'Content-Type': 'application/json; chartset=utf-8'})
    async with session.request(
            method="POST",
            url=BK_PAAS_HOST + '/api/c/compapi/v2/monitor_v3/get_event_log/',
            data=json.dumps(params),
            verify_ssl=False
    ) as response:
        content = json.loads(await response.content.read())
        # 关闭IO请求
        await session.close()
        comment = "--"
        for status_obj in content.get('data', []):
            if status_obj['operate'] == 'ANOMALY_NOTICE':
                comment = "已通知" if status_obj['status'] == 'SUCCESS' else '通知失败'
                break
        return {
            "event_id": event_id,
            "comment": comment
        }

效果：原本大屏API要展示100条数据，第二个接口耗时3-4S，API总耗时6-7S，现在通过协程实现，第二个接口耗时不到一秒，现在大屏API总耗时三秒左右。
100次接口的耗时和一次接口的耗时差不多。