feat: 基于 LLaMA-Factory 实现模型推理引擎
利用容器内已有的 LLaMA-Factory ChatModel (huggingface 后端) 实现真实 模型推理,无需额外安装 vLLM。 Compute 端新增: - compute/engines/llama_factory/inference.py InferenceSession: 模型加载/卸载/对话/流式输出 支持 base model 和 LoRA adapter,线程安全 - compute/api/main.py 新增 5 个推理端点: POST /inference/load - 加载模型 POST /inference/unload - 卸载释放 GPU 显存 GET /inference/status - 查询会话状态 POST /inference/chat - 非流式对话 POST /inference/chat/stream - SSE 流式对话 后端新增: - platform.py 推理代理端点(local/chat, local/chat/stream, local/preload, local/unload, local/status, trained/preload) - ComputeNodeClient._request 通用请求方法 - _select_first_online_node 自动选择在线节点 Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -182,6 +182,17 @@ class ComputeNodeClient:
|
||||
response.raise_for_status()
|
||||
return _unwrap_dict(response.json())
|
||||
|
||||
async def _request(self, method: str, path: str, json_data: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||
"""Generic request method for compute API endpoints."""
|
||||
url = _join_url(self.api_base_url, f"{self.route_prefix}{path}")
|
||||
async with httpx.AsyncClient(timeout=300, headers=self.headers()) as client:
|
||||
if method.upper() == "GET":
|
||||
response = await client.get(url)
|
||||
else:
|
||||
response = await client.post(url, json=json_data)
|
||||
response.raise_for_status()
|
||||
return _unwrap_dict(response.json())
|
||||
|
||||
async def upload_file(
|
||||
self,
|
||||
filename: str,
|
||||
|
||||
Reference in New Issue
Block a user