feat: 模型推理异步加载与对话链路修复,同步基线

模型推理全异步化改造:
- 计算节点 InferenceSession 改为后台线程异步加载模型,load 立即返回,
  加载期间事件循环保持响应(/inference/status 与 /health 不阻塞)
- 后端模型加载改为异步派发 + 轮询对账器(reconcile_inference_loads),
  任务状态由 starting 自动推进到 ready/error,解决多节点启动超时
  (timeout of 120000ms exceeded)
- 推理删除/卸载改为任务感知 + 短超时,删除先删记录再 best-effort 卸载,
  不再被不可达节点阻塞;同节点新模型替换旧任务标记失效
- 流式对话透传 task_id/node_id 路由到真正加载模型的算力节点,
  useStreamChat 解析 SSE 错误帧以干净文案展示
- 对话历史按任务 id 本地持久化,退出重进可恢复;移除页脚提示文本
- 新增后端推理异步加载与计算节点异步状态机单元测试

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
wuyongtao
2026-08-04 16:59:34 +08:00
parent 250e060271
commit 0271942ba5
21 changed files with 1272 additions and 245 deletions

View File

@@ -1,5 +1,6 @@
from __future__ import annotations
import asyncio
import json
import os
import math
@@ -717,7 +718,9 @@ def create_app() -> FastAPI:
@app.post(f"{route_prefix}/inference/unload")
async def inference_unload() -> dict[str, Any]:
"""Unload the currently loaded model and free GPU memory."""
return get_inference_session().unload()
# Teardown (gc.collect + cuda.empty_cache) can take a while; run it off
# the event loop so /health and /inference/status stay responsive.
return await asyncio.to_thread(get_inference_session().unload)
@app.get(f"{route_prefix}/inference/status")
async def inference_status() -> dict[str, Any]:
@@ -737,7 +740,10 @@ def create_app() -> FastAPI:
messages = payload.get("messages") or []
if not messages:
raise HTTPException(status_code=400, detail="messages is required")
result = get_inference_session().chat(
# Generation is long-running; run it in a thread so the event loop keeps
# serving /inference/status and /health during inference.
result = await asyncio.to_thread(
get_inference_session().chat,
messages=messages,
temperature=float(payload.get("temperature", 0.95)),
top_p=float(payload.get("top_p", 0.7)),