openLLM

GitHub地址

安装

# or pip3 install openllm
pip install openllm
openllm hello

• 聊天界面

OpenLLM provides a chat UI at the /chat endpoint for the launched LLM server at http://localhost:3000/chat.

• 使用

openllm serve llama3.2:1b

openllm run llama3:8b
openllm model list
openllm repo update
openllm model get llama3.2:1b

• 添加模型

GitHub地址

openllm repo add <repo-name> <repo-url>
openllm repo add nightly https://github.com/bentoml/openllm-models@nightly

• test

export VLLM_DEVICE=cpu
openllm serve gemma2:2b