想用一下 MiniCPM-o-2_6-int4
import torch
from modelscope import AutoModel, AutoTokenizer
# 加载模型和分词器
model = AutoModel.from_pretrained(
'openbmb/MiniCPM-o-2_6-int4',
torch_dtype=torch.bfloat16,
device_map="cuda:0",
trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(
'openbmb/MiniCPM-o-2_6-int4',
trust_remote_code=True
)
# 如果你需要进行量化操作,可以使用其他量化工具
# 例如,使用 torch.quantization 进行量化
# 这里只是一个示例,具体的量化操作需要根据你的需求进行调整
model.eval()
model.qconfig = torch.quantization.default_qconfig
torch.quantization.prepare(model, inplace=True)
# 进行校准等操作
torch.quantization.convert(model, inplace=True)
# 初始化 TTS(假设模型支持 TTS 功能)
# 这里假设模型有一个 init_tts 方法
if hasattr(model, 'init_tts'):
model.init_tts()
但是发现跑不起来了
会报错
Downloading Model from https://www.modelscope.cn to directory: /home/pon/.cache/modelscope/hub/models/OpenBMB/MiniCPM-V-2_6-int4
2025-05-13 05:47:18,136 - modelscope - WARNING - Using branch: master as version is unstable, use with caution
Traceback (most recent call last):
File "/home/pon/code/me/modelscope_example/MiniCPM-V_2-6_int4-video.py", line 8, in <module>
model = AutoModel.from_pretrained('OpenBMB/MiniCPM-V-2_6-int4', trust_remote_code=True) # sdpa or flash_attention_2, no eager
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/pon/.local/share/virtualenvs/modelscope_example-DACykz4b/lib/python3.11/site-packages/modelscope/utils/hf_util/patcher.py", line 247, in from_pretrained
module_obj = module_class.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/pon/.local/share/virtualenvs/modelscope_example-DACykz4b/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 558, in from_pretrained
model_class = get_class_from_dynamic_module(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/pon/.local/share/virtualenvs/modelscope_example-DACykz4b/lib/python3.11/site-packages/transformers/dynamic_module_utils.py", line 570, in get_class_from_dynamic_module
return get_class_in_module(class_name, final_module, force_reload=force_download)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/pon/.local/share/virtualenvs/modelscope_example-DACykz4b/lib/python3.11/site-packages/transformers/dynamic_module_utils.py", line 267, in get_class_in_module
module_spec.loader.exec_module(module)
File "<frozen importlib._bootstrap_external>", line 940, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/home/pon/.cache/huggingface/modules/transformers_modules/MiniCPM-V-2_6-int4/modeling_minicpmv.py", line 14, in <module>
from .resampler import Resampler
File "/home/pon/.cache/huggingface/modules/transformers_modules/MiniCPM-V-2_6-int4/resampler.py", line 682, in <module>
) -> List[Tensor]:
^^^^
NameError: name 'List' is not defined. Did you mean: 'pdist'?
看到是因为 huggingface/modules/transformers_modules/MiniCPM-V-2_6-int4/resampler.py
里面用了 List
但是没有 from typing import List
看了一下在 modelscope 和 huggingface 都是错的