sglang 部署Qwen2VL7B，大模型部署，速度测试，深度学习
sglang
项目github仓库：
https://github.com/sgl-project/sglang
项目说明书：
https://sgl-project.github.io/start/install.html
资讯：
https://github.com/sgl-project/sgl-learning-materials?tab=readme-ov-file#the-first-sglang-online-meetup
快得离谱：
[外链图片转存中…(img-E3n1Ivz9-1731913508383)]
图来源：https://lmsys.org/blog/2024-09-04-sglang-v0-3/
Docker使用：
docker run --gpus device=0 \--shm-size 32g \-p 30000:30000 \-v /root/xiedong/Qwen2-VL-7B-Instruct:/Qwen2-VL \--env "HF_TOKEN=abc-1234" \--ipc=host \-v /root/xiedong/Qwen2-VL-72B-Instruct-GPTQ-Int4:/root/xiedong/Qwen2-VL-72B-Instruct-GPTQ-Int4 \lmsysorg/sglang:latest \python3 -m sglang.launch_server --model-path /Qwen2-VL --host 0.0.0.0 --port 30000 --chat-template qwen2-vl --context-length 8192 --log-level-http warning启动成功：
接口文档：
http://101.136.22.140:30000/docs
速度测试代码
import time
from openai import OpenAI# 初始化OpenAI客户端
client = OpenAI(api_key='YOUR_API_KEY', base_url='http://0.0.0.0:30000/v1')# 定义图像路径
image_paths = ["/root/xiedong/Qwen2-VL-72B-Instruct-GPTQ-Int4/demo256.jpeg","/root/xiedong/Qwen2-VL-72B-Instruct-GPTQ-Int4/demo512.jpeg","/root/xiedong/Qwen2-VL-72B-Instruct-GPTQ-Int4/demo768.jpeg","/root/xiedong/Qwen2-VL-72B-Instruct-GPTQ-Int4/demo1024.jpeg","/root/xiedong/Qwen2-VL-72B-Instruct-GPTQ-Int4/demo1280.jpeg","/root/xiedong/Qwen2-VL-72B-Instruct-GPTQ-Int4/demo2560.jpeg"
]# 设置请求次数
num_requests = 10# 存储每个图像的平均响应时间
average_speeds = {}# 遍历每张图片
for image_path in image_paths:total_time = 0# 对每张图片执行 num_requests 次请求for _ in range(num_requests):start_time = time.time()# 发送请求并获取响应response = client.chat.completions.create(model="/Qwen2-VL",messages=[{'role': 'user','content': [{'type': 'text','text': 'Describe the image please',}, {'type': 'image_url','image_url': {'url': image_path,},}],}],temperature=0.8,top_p=0.8)# 记录响应时间elapsed_time = time.time() - start_timetotal_time += elapsed_time# 打印当前请求的响应内容（可选）print(f"Response for {image_path}: {response.choices[0].message.content}")# 计算并记录该图像的平均响应时间average_speed = total_time / num_requestsaverage_speeds[image_path] = average_speedprint(f"Average speed for {image_path}: {average_speed} seconds")# 输出所有图像的平均响应时间
for image_path, avg_speed in average_speeds.items():print(f"{image_path}: {avg_speed:.2f} seconds")
速度测试结果
sglang 测试结果：
| Model | 显存占用 (MiB) | 分辨率 | 处理时间 (秒) |
|---|---|---|---|
| Qwen2-VL-7B-Instruct | 70G | 256 x 256 | 1.71 |
| 512 x 512 | 1.52 | ||
| 768 x 768 | 1.85 | ||
| 1024 x 1024 | 2.05 | ||
| 1280 x 1280 | 1.88 | ||
| 2560 x 2560 | 3.26 |
纯transformer，不用加速框架，我之前的测了一张图的速度是：5.22 seconds，很慢。
附录-vllm速度测试
启动：
docker run --gpus device=0 \-v /root/xiedong/Qwen2-VL-7B-Instruct:/Qwen2-VL \-v /root/xiedong/Qwen2-VL-72B-Instruct-GPTQ-Int4:/root/xiedong/Qwen2-VL-72B-Instruct-GPTQ-Int4 \-p 30000:8000 \--ipc=host \vllm/vllm-openai:latest \--model /Qwen2-VL --gpu_memory_utilization=0.9
代码：
import time
import base64
from openai import OpenAI# 初始化OpenAI客户端
client = OpenAI(api_key='YOUR_API_KEY', base_url='http://0.0.0.0:30000/v1')# 定义图像路径
image_paths = ["/root/xiedong/Qwen2-VL-72B-Instruct-GPTQ-Int4/demo256.jpeg","/root/xiedong/Qwen2-VL-72B-Instruct-GPTQ-Int4/demo512.jpeg","/root/xiedong/Qwen2-VL-72B-Instruct-GPTQ-Int4/demo768.jpeg","/root/xiedong/Qwen2-VL-72B-Instruct-GPTQ-Int4/demo1024.jpeg","/root/xiedong/Qwen2-VL-72B-Instruct-GPTQ-Int4/demo1280.jpeg","/root/xiedong/Qwen2-VL-72B-Instruct-GPTQ-Int4/demo2560.jpeg"
]# 设置请求次数
num_requests = 10# 存储每个图像的平均响应时间
average_speeds = {}# 将图片转换为 Base64 编码的函数
def image_to_base64(image_path):with open(image_path, "rb") as img_file:return base64.b64encode(img_file.read()).decode('utf-8')# 遍历每张图片
for image_path in image_paths:total_time = 0# 将图片转换为 Base64 编码image_base64 = image_to_base64(image_path)# 对每张图片执行 num_requests 次请求for _ in range(num_requests):start_time = time.time()# 发送请求并获取响应response = client.chat.completions.create(model="/Qwen2-VL",messages=[{'role': 'user','content': [{'type': 'text','text': 'Describe the image please',}, {'type': 'image_url','image_url': {'url': f"data:image/jpeg;base64,{image_base64}", # 使用Base64编码的图片},}],}],temperature=0.8,top_p=0.8)# 记录响应时间elapsed_time = time.time() - start_timetotal_time += elapsed_time# 打印当前请求的响应内容（可选）print(f"Response for {image_path}: {response.choices[0].message.content}")# 计算并记录该图像的平均响应时间average_speed = total_time / num_requestsaverage_speeds[image_path] = average_speedprint(f"Average speed for {image_path}: {average_speed} seconds")# 输出所有图像的平均响应时间
for image_path, avg_speed in average_speeds.items():print(f"{image_path}: {avg_speed:.2f} seconds")
速度：
| Model | 显存占用 (MiB) | 分辨率 | 处理时间 (秒) |
|---|---|---|---|
| Qwen2-VL-72B-Instruct-GPTQ-Int4 | 70G | 256 x 256 | 1.50 |
| 512 x 512 | 1.59 | ||
| 768 x 768 | 1.61 | ||
| 1024 x 1024 | 1.67 | ||
| 1280 x 1280 | 1.81 | ||
| 2560 x 2560 | 1.97 |
https://www.dong-blog.fun/post/1856