OpenCompass傻瓜式入门教程

1 我也许不是傻瓜，却只想做个傻瓜

OpenCompass在github中说祝贺 OpenCompass 作为大模型标准测试工具被Meta AI官方推荐，实力不俗。但官方教程真是写得那个随心所欲，我只想无脑照着教程，快速跑起来，再慢慢深入了解。但折腾半天，不是这里错就是那里有问题，根本跑不起来，差点就劝退了。于是有了这篇入门教程，力争无脑操作，躺平做个傻瓜！

2 环境要求

1. 最好是Linux或WSL，Windows可能会有不期而遇的错误
2. 必须是python 3.10，我一上来就用了python 3.12，装了半天也装不上
3. 直接源码安装！千万别pip install opencompass，然后照着官方教程，直接劝退

3 安装

3.1 下载源码

git clone https://github.com/open-compass/opencompass opencompass

3.2 创建虚拟环境

conda create -n open-compass python=3.10 -y
conda activate open-compass
python --version

3.3 安装

cd opencompass
pip install -e . -i https://mirrors.aliyun.com/pypi/simple/
pip list | grep opencompass

opencompass 0.4.2 /home/ubuntu/ws/opencompass

4 下载数据

# 先进入到opencompass目录下
# 注意，opencompass下还有一个opencompass文件夹，别进错了  
# cd opencompasswget https://github.com/open-compass/opencompass/releases/download/0.2.2.rc1/OpenCompassData-core-20240207.zip
# 会自动解压到data文件夹中
unzip OpenCompassData-core-20240207.zip

5 查看支持的模型和数据集

OpenCompass采用基于配置文件与命名约定的运行机制。运行时指定的模型和数据集参数并非随意传递，而是通过预定义的名称与配置文件映射关系来定位。用户只需指定预定义的名称，系统就能自动关联到对应的配置文件。要查看所有可用的配置映射关系，可以运行命令python tools/list_configs.py。

# cd opencompass
# conda activate open-compass
python tools/list_configs.py

+-----------------------------------------------+-------------------------------------------------------------------------------------+
| Model                                         | Config Path                                                                         |
|-----------------------------------------------+-------------------------------------------------------------------------------------|
| README                                        | opencompass/configs/models/qwen/README.md                                           |
| README                                        | opencompass/configs/models/hf_internlm/README.md                                    |
| accessory_llama2_7b                           | opencompass/configs/models/accessory/accessory_llama2_7b.py                         |+-----------------------------------------------+-------------------------------------------------------------------------------------+
| Dataset                                       | Config Path                                                                         |
|-----------------------------------------------+-------------------------------------------------------------------------------------|
| ARC_c_clean_ppl                               | opencompass/configs/datasets/ARC_c/ARC_c_clean_ppl.py                               |
| ARC_c_cot_gen_926652                          | opencompass/configs/datasets/ARC_c/ARC_c_cot_gen_926652.py                          |
| ARC_c_few_shot_gen_e9b043                     | opencompass/configs/datasets/ARC_c/ARC_c_few_shot_gen_e9b043.py                     |

6 评测

opencompass主要有两种运行方式，一种指定模型路径，opencompass直接加载运行模型进行评测；一种是指定配置文件，opencompass加载解析配置文件，加载本地模型或调用模型的服务化API进行评测。

6.1 指定模型路径

# cd opencompass
# conda activate open-compass# 必须是在opencompass目录下，才有run.py这个脚本 
# --hf-type 指定chat或base模型类型
# --hf-path 指定本地模型的路径，open-compass会以huggingface的方式加载模型
# --datasets 指定数据集名称，必须是在tools/list_configs.py中python run.py \--hf-type chat \--hf-path /mnt/d/models/Qwen/Qwen2.5-1.5B-Instruct \--datasets demo_gsm8k_chat_gen \--work-dir eval_results/hf/Qwen2.5-1.5B-Instruct \--debug

6.2 指定配置文件

6.2.1 评测本地qwen2.5模型

6.2.1.1 查看opencompass支持的qwen2.5模型

# cd opencompass
# conda activate open-compasspython tools/list_configs.py | grep hf_qwen2_5

| hf_qwen2_57b_a14b                             | opencompass/configs/models/qwen/hf_qwen2_57b_a14b.py                                |
| hf_qwen2_5_0_5b_instruct                      | opencompass/configs/models/qwen2_5/hf_qwen2_5_0_5b_instruct.py                      |
| hf_qwen2_5_14b_instruct                       | opencompass/configs/models/qwen2_5/hf_qwen2_5_14b_instruct.py                       |
| hf_qwen2_5_1_5b_instruct                      | opencompass/configs/models/qwen2_5/hf_qwen2_5_1_5b_instruct.py                      |
| hf_qwen2_5_32b_instruct                       | opencompass/configs/models/qwen2_5/hf_qwen2_5_32b_instruct.py                       |
| hf_qwen2_5_3b_instruct                        | opencompass/configs/models/qwen2_5/hf_qwen2_5_3b_instruct.py                        |
| hf_qwen2_5_72b_instruct                       | opencompass/configs/models/qwen2_5/hf_qwen2_5_72b_instruct.py                       |
| hf_qwen2_5_7b_instruct                        | opencompass/configs/models/qwen2_5/hf_qwen2_5_7b_instruct.py                        |

6.2.1.2 创建配置文件

以Qwen2.5-1.5B-Instruct为例，即hf_qwen2_5_1_5b_instruct，可看到它对应的配置文件为opencompass/configs/models/qwen2_5/hf_qwen2_5_1_5b_instruct.py ，以它为模板，拷贝后修改如下，并命名为 local_hf_qwen2_5_1_5b_instruct.py：

from opencompass.models import HuggingFacewithChatTemplatemodels = [dict(type=HuggingFacewithChatTemplate,abbr='local_qwen2.5-1.5b-instruct-hf',path='/mnt/d/models/Qwen/Qwen2.5-1.5B-Instruct',  # 指定本地路径max_out_len=4096,batch_size=8,run_cfg=dict(num_gpus=1),)
]

6.2.1.3 再次使用python tools/list_configs.py | grep hf_qwen2_5进行查看

| hf_qwen2_57b_a14b                             | opencompass/configs/models/qwen/hf_qwen2_57b_a14b.py                                |
| hf_qwen2_5_0_5b_instruct                      | opencompass/configs/models/qwen2_5/hf_qwen2_5_0_5b_instruct.py                      |
| hf_qwen2_5_14b_instruct                       | opencompass/configs/models/qwen2_5/hf_qwen2_5_14b_instruct.py                       |
| hf_qwen2_5_1_5b_instruct                      | opencompass/configs/models/qwen2_5/hf_qwen2_5_1_5b_instruct.py                      |
| hf_qwen2_5_32b_instruct                       | opencompass/configs/models/qwen2_5/hf_qwen2_5_32b_instruct.py                       |
| hf_qwen2_5_3b_instruct                        | opencompass/configs/models/qwen2_5/hf_qwen2_5_3b_instruct.py                        |
| hf_qwen2_5_72b_instruct                       | opencompass/configs/models/qwen2_5/hf_qwen2_5_72b_instruct.py                       |
| hf_qwen2_5_7b_instruct                        | opencompass/configs/models/qwen2_5/hf_qwen2_5_7b_instruct.py                        |
| local_hf_qwen2_5_1_5b_instruct                | opencompass/configs/models/qwen2_5/local_hf_qwen2_5_1_5b_instruct.py                |

6.2.1.4 运行

# cd opencompass
# conda activate open-compass# --models 要指定tools/list_configs.py中定义的模型配置文件
python run.py \--models local_hf_qwen2_5_1_5b_instruct \--datasets demo_gsm8k_chat_gen \--work-dir eval_results/hf/Qwen2.5-1.5B-Instruct \--debug

6.2.2 评测ollama模型

6.2.2.1 创建配置文件eval_ollama.py

from mmengine.config import read_base  from opencompass.models import OpenAI  
from opencompass.partitioners import NaivePartitioner  
from opencompass.runners import LocalRunner  
from opencompass.tasks import OpenICLInferTask  with read_base():  from opencompass.configs.datasets.cmmlu.cmmlu_gen import cmmlu_datasets  api_meta_template = dict(round=[  dict(role='HUMAN', api_role='HUMAN'),  dict(role='BOT', api_role='BOT', generate=True),  
], )  models = [  dict(  abbr='ollama',  type=OpenAI,  path='qwen3:0.6b',  # 指定ollama中的模型，可通过ollama list进行查看，如果不存在，则必须先下载   # 若不在这里指定，则需要在shell中使用 export OPENAI_BASE_URL=http://192.168.56.1:11434/v1# 不能 import os，否则会报错，所以也不能使用os.environ['OPENAI_BASE_URL'] = 'http://192.168.56.1:11434/v1' # 这里要指定windows的IP，使用localhost访问不到 openai_api_base='http://192.168.56.1:11434/v1/chat/completions',  # 若不在这里指定，则需要在shell中使用 export OPENAI_API_KEY=Nonekey='None',  meta_template=api_meta_template,  query_per_second=1,  max_out_len=2048,  max_seq_len=2048,   batch_size=2  ),  ]  infer = dict(  partitioner=dict(type=NaivePartitioner),  runner=dict(type=LocalRunner,  max_num_workers=2,  task=dict(type=OpenICLInferTask)),  
)  datasets = cmmlu_datasetsif __name__ == '__main__':  from opencompass.cli.main import main  import sys  from pathlib import Path  sys.argv.append(str(Path(__file__)))  sys.argv.extend(['--work-dir', 'eval_results/ollama/qwen3_0_6b'])  sys.argv.append('--debug')  main()

注意：

1. 代码中指定的path='qwen3:0.6b', 是指定ollama中的qwen3:0.6b模型，可通过ollama list进行查看，如果不存在，则必须先下载
2. 脚本中指定的openai_api_base='http://192.168.56.1:11434/v1/chat/completions'，其中的192.168.56.1是windows的IP，若直接使用localhost，无法从WSL访问到ollama。

6.2.2.2 启动ollama服务

在windows中打开powershell

# 配置ollama环境变量
$env:OLLAMA_HOST="0.0.0.0:11434"
$env:OLLAMA_MODELS="D:\models\ollama"
$env:OLLAMA_DEBUG="2"# 启动ollama服务
ollama serve

6.2.2.3 运行方式1

# cd opencompass
# conda activate open-compass# 不加 --debug，可能会报错
# 08/14 23:36:45 - OpenCompass - ERROR - /home/ubuntu/ws/opencompass/opencompass/runners/local.py - _launch - 241 - task OpenICLInfer[ollama/cmmlu-chinese_civil_service_exam] fail, see outputs/default/20250814_233629/logs/infer/ollama/cmmlu-chinese_civil_service_exam.out
python run.py eval_ollama.py --debug

6.2.2.4 运行方式2

这种方式可以pycharm中使用debug方式运行，方便debug

# cd opencompass
# conda activate open-compasspython eval_ollama.py

6.2.3 使用lmdeploy加速评测

默认transformers加载模型，推理比较慢。使用lmdeploy加载模型，加快推理速度。

6.2.3.1 安装lmdeploy

conda activate open-compass
pip install lmdeploy

如果CUDA不是12+，请参考官方教程进行安装：使用 LMDeploy 加速评测 — OpenCompass 0.4.2 documentation

6.2.3.2 查看支持lmdeploy加速的qwen2.5模型

# cd opencompass
# conda activate open-compasspython tools/list_configs.py lmdeploy_qwen2_5

+--------------------------------+----------------------------------------------------------------------+
| Model                          | Config Path                                                          |
|--------------------------------+----------------------------------------------------------------------|
| lmdeploy_qwen2_5_0_5b_instruct | opencompass/configs/models/qwen2_5/lmdeploy_qwen2_5_0_5b_instruct.py |
| lmdeploy_qwen2_5_14b           | opencompass/configs/models/qwen2_5/lmdeploy_qwen2_5_14b.py           |
| lmdeploy_qwen2_5_14b_instruct  | opencompass/configs/models/qwen2_5/lmdeploy_qwen2_5_14b_instruct.py  |
| lmdeploy_qwen2_5_1_5b          | opencompass/configs/models/qwen2_5/lmdeploy_qwen2_5_1_5b.py          |
| lmdeploy_qwen2_5_1_5b_instruct | opencompass/configs/models/qwen2_5/lmdeploy_qwen2_5_1_5b_instruct.py |
| lmdeploy_qwen2_5_32b           | opencompass/configs/models/qwen2_5/lmdeploy_qwen2_5_32b.py           |
| lmdeploy_qwen2_5_32b_instruct  | opencompass/configs/models/qwen2_5/lmdeploy_qwen2_5_32b_instruct.py  |
| lmdeploy_qwen2_5_3b_instruct   | opencompass/configs/models/qwen2_5/lmdeploy_qwen2_5_3b_instruct.py   |
| lmdeploy_qwen2_5_72b           | opencompass/configs/models/qwen2_5/lmdeploy_qwen2_5_72b.py           |
| lmdeploy_qwen2_5_72b_instruct  | opencompass/configs/models/qwen2_5/lmdeploy_qwen2_5_72b_instruct.py  |
| lmdeploy_qwen2_5_7b            | opencompass/configs/models/qwen2_5/lmdeploy_qwen2_5_7b.py            |
| lmdeploy_qwen2_5_7b_instruct   | opencompass/configs/models/qwen2_5/lmdeploy_qwen2_5_7b_instruct.py   |
+--------------------------------+----------------------------------------------------------------------+

6.2.3.3 创建配置文件

以lmdeploy_qwen2_5_1_5b_instruct为模板，拷贝创建local_lmdeploy_qwen2_5_1_5b_instruct.py,修改如下：

from opencompass.models import TurboMindModelwithChatTemplatemodels = [dict(type=TurboMindModelwithChatTemplate,abbr='local-qwen2.5-1.5b-instruct-turbomind',path='/mnt/d/models/Qwen/Qwen2.5-1.5B-Instruct',  # 指定模型路径engine_config=dict(session_len=16384, max_batch_size=16, tp=1),gen_config=dict(top_k=1, temperature=1e-6, top_p=0.9, max_new_tokens=4096),max_seq_len=16384,max_out_len=4096,batch_size=16,run_cfg=dict(num_gpus=1),)
]

6.2.3.4 运行

# cd opencompass
# conda activate open-compass# --models 要指定tools/list_configs.py中定义的模型配置文件
python run.py \--models local_lmdeploy_qwen2_5_1_5b_instruct \--datasets demo_gsm8k_chat_gen \--work-dir eval_results/lmdeploy/Qwen2.5-1.5B-Instruct \--debug