Jetson AGX Orin 配置LLM Deepseek-R1 7B模型

SZ-Jun

已于 2025-03-30 11:08:40 修改

阅读量2k

点赞数 31

CC 4.0 BY-SA版权

文章标签： ubuntu 大模型人工智能语言模型 Deepseek

于 2025-03-29 21:23:41 首次发布

本文链接：https://blog.csdn.net/zengwj2014/article/details/146703815

测试环境：

硬件： AGX ORIN 32G 套件

软件： Jetpack6.2 ，R36.4.3

按照官方测试数据，AGX ORIN 64套件可以跑70B 模型，为什么测试7B ，测试机器是AGX ORIN 32G 内存套件，图方便直接使用内部64G EMMC 存储，不想麻烦去挂载SSD盘，存储空间大小限制，64G EMMC存储空间刚好只能满足7B模型运行大小, 同样方法在ORIN NANO/NX 套件也是可以运行,内存不够时候需要释放相关的内存，如关闭系统桌面

sudo init 3     # stop the desktop

采用方式是英伟达官方提供适配测试demo方式，参考链接如下，

https://www.jetson-ai-lab.com/models.html

上次听Jeff 老师课程之后，尝试按照NVIDIA 方式在AGX orin 平台适配和运行Deepseek，折腾几天，发现很多时候总是链接报错，猜测原因是NVIDIA 方式Docker 采用NIM 微服务调用方式，对接open-ai和huggingface 平台，这两个鬼佬的平台对国内访问不友好，最后折腾出来能跑起来模型，对遇到问题总结下，

国内用户不能科学上网方式，建议用ollma 方式适配deepseek和其他大模型简单快捷，在国内别折腾了

1. Docker 安装

参考链接

🔖 SSD + Docker - NVIDIA Jetson AI Lab

sudo apt update
sudo apt install -y nvidia-container curl
curl https://get.docker.com | sh && sudo systemctl --now enable docker
sudo nvidia-ctk runtime configure --runtime=docker

sudo systemctl restart docker
sudo usermod -aG docker $USER
newgrp docker

sudo apt install -y jq
sudo jq '. + {"default-runtime": "nvidia"}' /etc/docker/daemon.json | \
  sudo tee /etc/docker/daemon.json.tmp && \
  sudo mv /etc/docker/daemon.json.tmp /etc/docker/daemon.json

sudo systemctl daemon-reload && sudo systemctl restart docker

2. 参考官方docker 运行脚本，了解模型需要资源

3. 提前从huggingface 找到和下载模型到本地（git clone ），后续不需要在docker 里面再访问huggingface.co下载，huggingface 资源访问需要科学上网才能获取

git clone https://huggingface.co/dusty-nv/DeepSeek-R1-Distill-Qwen-7B-q4f16_ft-MLC

3. 机器安装openai, 需要科学上网

pip install openai

4.拷贝模型文件到指定目录下在面，这个目录对应后面 docker -v 路径参数

cecport@ubuntu:~$ ls -l /home/cecport/cache/mlc_llm/dusty-nv/DeepSeek-R1-Distill-Qwen-7B-q4f16_ft-MLC/
total 3805012
-rwxrwxrwx 1 cecport cecport   9855032 3月 25 00:26 aarch64-cu126-sm87.so
-rw-r--r-- 1 root    root         1977 3月 27 21:44 mlc-chat-config.json
-rw-r--r-- 1 root    root       134600 3月 27 21:44 ndarray-cache.json
-rwxrwxrwx 1 cecport cecport 272498688 3月 25 00:27 params_shard_0.bin
-rwxrwxrwx 1 cecport cecport 33947648 3月 24 23:57 params_shard_10.bin
-rwxrwxrwx 1 cecport cecport 29605888 3月 25 00:26 params_shard_11.bin
-rwxrwxrwx 1 cecport cecport 67895296 3月 25 00:16 params_shard_12.bin
-rwxrwxrwx 1 cecport cecport 33947648 3月 24 23:57 params_shard_13.bin
-rwxrwxrwx 1 cecport cecport 67895296 3月 25 00:16 params_shard_14.bin

5.运行docker 模型相关的服务，需要下载 llm server 镜像和启动相关容器

由于国内网络访问Huggingface受限，我们需要通过镜像网站： https://hf-mirror.com 下载模型和相关的资源，增加参数

-e HF_ENDPOINT=https://hf-mirror.com \

docker run -it --rm \
--name llm_server \
--gpus all \
-p 9000:9000 \
-e DOCKER_PULL=always --pull always \
-e HF_ENDPOINT=https://hf-mirror.com \
-e HF_HUB_CACHE=/root/.cache/huggingface \
-v /home/cecport/cache:/root/.cache \
dustynv/mlc:r36.4.0 \
    sudonim serve \
      --model dusty-nv/DeepSeek-R1-Distill-Qwen-7B-q4f16_ft-MLC \
      --quantization q4f16_ft \
      --max-batch-size 1 \
      --chat-template deepseek_r1_qwen \
      --host 0.0.0.0 \
      --port 9000

启动正常结果，其实这个也是报错，访问hugging face 433端口失败，国内网络原因，折腾很多方式会有失败信息，之前模型下载本地关联起来，最后模型还是能挂载起来，

写一个 open-ai.py 测试脚本

# use 'pip install openai' before running this
from openai import OpenAI

client = OpenAI(
base_url = 'http://0.0.0.0:9000/v1',
api_key = 'foo' # not enforced
)

chat = [{
'role': 'user',
'content': 'ni hao'
}]

completion = client.chat.completions.create(
model='default', # not enforced
messages=chat,
temperature=0.2,
top_p=0.7,
stream=True
)

for chunk in completion:
if chunk.choices[0].delta.content is not None:
print(chunk.choices[0].delta.content, end='')

执行之后得到deepseek 回复，说明搭建模型环正常运行，

cecport@ubuntu:~/deepseek$ python open-ai.py
<think>

</think>

你好！很高兴能为你提供帮助。请告诉我你需要什么类型的帮助，我会尽力为你提供详细的解答。

cecport@ubuntu:~/deepseek$

6. 安装OPEN-WEBUI

需要网页浏览窗口，方便与deepseek 模型对话操作

cecport@ubuntu:~$ docker run -it --rm \
--name open-webui \
--network=host \
-e PORT=8080 \
-e ENABLE_OPENAI_API=True \
-e ENABLE_OLLAMA_API=False \
-e OPENAI_API_BASE_URL=0.0.0.0:9000/v1 \
-e OPENAI_API_KEY=foo \
-e AUDIO_STT_ENGINE=openai \
-e AUDIO_TTS_ENGINE=openai \
-e HF_ENDPOINT=https://hf-mirror.com \
-e AUDIO_STT_OPENAI_API_BASE_URL=0.0.0.0:8990/v1 \
-e AUDIO_TTS_OPENAI_API_BASE_URL=0.0.0.0:8995/v1 \
-v /home/cecport/cache/open-webui:/app/backend/data \
-e DOCKER_PULL=always --pull always \
-e HF_HUB_CACHE=/root/.cache/huggingface \
-v /home/cecport/cache:/root/.cache \
ghcr.io/open-webui/open-webui:mai

open webui 运行成功之后显示图片

7. 通过PC端网页访问登陆open webui IP地址：端口，输入账号，邮箱，密码（务必记住，后续登陆需正确信息才能进入）。

最后发现页面里面没有发现大模型，这个问题很奇怪，到腾研究下也找不到原因，通过open webui错误信息判断是open webui 访问失败，明明通过网页可以访问http://0.0.0.0:9000/v1/models，猜测是open webui 本身程序或者软件版本兼任有问题

错误地方日志：

|INFO | open_webui.routers.openai:get_all_models:379 - get_all_models() - {}
| ERROR | open_webui.routers.openai:send_get_request:78 - Connection error: 0.0.0.0:9000/v1/models - {}

8. 参考网上高人方法，在open -webui 设置选项里面 --外部链接，把模型相关的链接信息添加进去，发现可以把模型加载近来，

9. 先问一个问题测试下deepseek，感觉回答还算可以

如果想更完美回复，可以上传一些本地知识库文件上去，回答会更完美些

对话模式下，CPU ，GPU和内存运行状态

10.

遗留问题，benchmark 跑分总是有错误，找不到原因，可能还是网络原因导致有些文件下载失败，缺失导致，

docker run -it --rm \
--name llm_server \
--network=host \
--runtime=nvidia \
--gpus all \
-p 9000:9000 \
-e DOCKER_PULL=always --pull always \
-e HF_ENDPOINT=https://hf-mirror.com \
-e HF_HUB_CACHE=/root/.cache/huggingface \
-v /home/cecport/cache:/root/.cache \
dustynv/mlc:r36.4.0 \
sudonim bench stop \
--model dusty-nv/DeepSeek-R1-Distill-Qwen-7B-q4f16_ft-MLC \
--quantization q4f16_ft \
--max-batch-size 1 \
--chat-template deepseek_r1_llama \
--host 0.0.0.0 \
--port 9000

错误信息如下

/usr/local/lib/python3.10/dist-packages/transformers/utils/hub.py:128: FutureWarning: Using `TRANSFORMERS_CACHE` is deprecated and will be removed in v5 of Transformers. Use `HF_HOME` instead.
warnings.warn(
[2025-03-29 11:55:31] INFO request_processor.py:47: Fixing number of concurrent requests: 2
[2025-03-29 11:55:38] INFO request_processor.py:279: Warmup with 3 request(s)...
0%|                                                                                                                                                                                                              | 0/3 [00:00<?, ?it/s][2025-03-29 11:55:38] INFO api_endpoint.py:279: API endpoint errored when sending request: Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/aiohttp/connector.py", line 1115, in _wrap_create_connection
    sock = await aiohappyeyeballs.start_connection(
File "/usr/local/lib/python3.10/dist-packages/aiohappyeyeballs/impl.py", line 104, in start_connection
    raise first_exception
File "/usr/local/lib/python3.10/dist-packages/aiohappyeyeballs/impl.py", line 82, in start_connection
    sock = await _connect_sock(
File "/usr/local/lib/python3.10/dist-packages/aiohappyeyeballs/impl.py", line 174, in _connect_sock
    await loop.sock_connect(sock, address)
File "/usr/lib/python3.10/asyncio/selector_events.py", line 501, in sock_connect
    return await fut
File "/usr/lib/python3.10/asyncio/selector_events.py", line 541, in _sock_connect_cb
    raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 111] Connect call failed ('0.0.0.0', 9000)

The above exception was the direct cause of the following exception:

下载过程访问错误解决

（1） docker: Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

（2）iptables failed: iptables --wait -t raw -A PREROUTING -p tcp
-d 172.19.0.2 --dport 9000 ! -i br-de7b322f65f3 -j DROP: iptables v1.8.7 (legacy): can't initialize
iptables table `raw': Table does not exist (do you need to insmod?)

修改 /etc/docker/daemon.json

cecport@ubuntu:~$ cat /etc/docker/daemon.json
{
"runtimes": {
    "nvidia": {
      "args": [],
      "path": "nvidia-container-runtime"
    }
},
"default-runtime": "nvidia",
"iptables": false,
"registry-mirrors": ["https://docker.registry.cyou",
"https://docker-cf.registry.cyou",
"https://dockercf.jsdelivr.fyi",
"https://docker.jsdelivr.fyi",
"https://dockertest.jsdelivr.fyi",
"https://mirror.aliyuncs.com",
"https://dockerproxy.com",
"https://mirror.baidubce.com",
"https://docker.m.daocloud.io",
"https://docker.nju.edu.cn",
"https://docker.mirrors.sjtug.sjtu.edu.cn",
"https://docker.mirrors.ustc.edu.cn",
"https://mirror.iscas.ac.cn",
"https://docker.rainbond.cc"]
}