Llama中文大模型-模型量化

最新推荐文章于 2025-08-04 16:57:17 发布

蓝鲸123

最新推荐文章于 2025-08-04 16:57:17 发布

阅读量996

点赞数 9

CC 4.0 BY-SA版权

文章标签： llama

本文链接：https://blog.csdn.net/TH_NUM/article/details/136274435

文章讲解了如何利用4bit压缩的FlagAlpha/Llama2-Chinese-13b-Chat模型进行文本生成，包括环境设置和代码示例。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

对中文微调的模型参数进行了量化，方便以更少的计算资源运行。目前已经在Hugging Face上传了13B中文微调模型FlagAlpha/Llama2-Chinese-13b-Chat的4bit压缩版本FlagAlpha/Llama2-Chinese-13b-Chat-4bit，具体调用方式如下：

环境准备：

pip install git+https://github.com/PanQiWei/AutoGPTQ.git

from transformers import AutoTokenizer
from auto_gptq import AutoGPTQForCausalLM
model = AutoGPTQForCausalLM.from_quantized('FlagAlpha/Llama2-Chinese-13b-Chat-4bit', device="cuda:0")
tokenizer = AutoTokenizer.from_pretrained('FlagAlpha/Llama2-Chinese-13b-Chat-4bit',use_fast=False)
input_ids = tokenizer(['<s>Human: 怎么登上火星\n</s><s>Assistant: '], return_tensors="pt",add_special_tokens=False).input_ids.to('cuda')        
generate_input = {
    "input_ids":input_ids,
    "max_new_tokens":512,
    "do_sample":True,
    "top_k":50,
    "top_p":0.95,
    "temperature":0.3,
    "repetition_penalty":1.3,
    "eos_token_id":tokenizer.eos_token_id,
    "bos_token_id":tokenizer.bos_token_id,
    "pad_token_id":tokenizer.pad_token_id
}
generate_ids  = model.generate(**generate_input)
text = tokenizer.decode(generate_ids[0])
print(text)

关注博主即可阅读全文