feat: 基于 LLaMA-Factory 实现模型推理引擎

利用容器内已有的 LLaMA-Factory ChatModel (huggingface 后端) 实现真实
模型推理,无需额外安装 vLLM。

Compute 端新增:
- compute/engines/llama_factory/inference.py
  InferenceSession: 模型加载/卸载/对话/流式输出
  支持 base model 和 LoRA adapter,线程安全
- compute/api/main.py 新增 5 个推理端点:
  POST /inference/load      - 加载模型
  POST /inference/unload    - 卸载释放 GPU 显存
  GET  /inference/status    - 查询会话状态
  POST /inference/chat      - 非流式对话
  POST /inference/chat/stream - SSE 流式对话

后端新增:
- platform.py 推理代理端点(local/chat, local/chat/stream,
  local/preload, local/unload, local/status, trained/preload)
- ComputeNodeClient._request 通用请求方法
- _select_first_online_node 自动选择在线节点

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
wuyongtao
2026-07-28 13:49:10 +08:00
parent a9ab130d43
commit f917a025e1
4 changed files with 314 additions and 5 deletions

View File

@@ -182,6 +182,17 @@ class ComputeNodeClient:
response.raise_for_status()
return _unwrap_dict(response.json())
async def _request(self, method: str, path: str, json_data: dict[str, Any] | None = None) -> dict[str, Any]:
"""Generic request method for compute API endpoints."""
url = _join_url(self.api_base_url, f"{self.route_prefix}{path}")
async with httpx.AsyncClient(timeout=300, headers=self.headers()) as client:
if method.upper() == "GET":
response = await client.get(url)
else:
response = await client.post(url, json=json_data)
response.raise_for_status()
return _unwrap_dict(response.json())
async def upload_file(
self,
filename: str,