15 Commits

Author SHA1 Message Date
wuyongtao
ec7d8c0a3d fix: 算力节点任务/GPU 统计纳入评测与推理占用,补充测试依赖
- compute_nodes() 的 current_running_jobs 纳入运行中的评测任务(eval_tasks)
  与已加载的推理模型(compare_tasks 持久化状态 + 内存标记兜底),
  多节点时 GPU 被评测/推理占用不再显示 0/1
- gpus() 直接按 eval_tasks / compare_tasks 派生 GPU busy/reserved 状态,
  修复 gpu_id=0 时 or -1 导致匹配失败;删除评测后 GPU 不再残留 busy
- 评测不再复用 mark_inference_loaded 内存标记,GPU 占用由任务数据驱动
- requirements.txt 补充 pytest / ruff(此前仅声明在 pyproject dev 可选依赖)

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-04 18:46:53 +08:00
wuyongtao
0292bf5138 fix: 模型评测异步加载等待与多节点路由修复
- eval_runner 等待异步模型加载完成(InferenceSession.wait_until_loaded),
  修复 "model load failed: unknown"
- 评测算力节点选择:优先页面选择的节点 / 模型所在节点(_select_eval_node),
  多节点时不再派发到不可达节点导致连接超时
- 前端评测 GPU 选择改为节点感知(节点:GPU 复合值),透传 compute_node_id,
  并检查 startEval 结果展示真实错误
- 大模型评价(judge)使用模型记录的真实 API 模型名(api_model),
  避免用平台内部名调用 LLM API 导致 HTTP 400
- 新增后端节点选择与 compute wait_until_loaded 单元测试

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-04 18:21:16 +08:00
wuyongtao
0271942ba5 feat: 模型推理异步加载与对话链路修复,同步基线
模型推理全异步化改造:
- 计算节点 InferenceSession 改为后台线程异步加载模型,load 立即返回,
  加载期间事件循环保持响应(/inference/status 与 /health 不阻塞)
- 后端模型加载改为异步派发 + 轮询对账器(reconcile_inference_loads),
  任务状态由 starting 自动推进到 ready/error,解决多节点启动超时
  (timeout of 120000ms exceeded)
- 推理删除/卸载改为任务感知 + 短超时,删除先删记录再 best-effort 卸载,
  不再被不可达节点阻塞;同节点新模型替换旧任务标记失效
- 流式对话透传 task_id/node_id 路由到真正加载模型的算力节点,
  useStreamChat 解析 SSE 错误帧以干净文案展示
- 对话历史按任务 id 本地持久化,退出重进可恢复;移除页脚提示文本
- 新增后端推理异步加载与计算节点异步状态机单元测试

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-04 16:59:34 +08:00
wuyongtao
250e060271 feat: 看板服务状态优化,移除前端访问统计埋点,修正列表页文案与类型
- 看板:补齐各服务模块图标,服务状态表格改为自适应行高并支持滚动
- 移除前端请求访问统计埋点及对应 audit-visit API 模块
- 平台接口:清理 service_checks 循环中未使用的路径变量
- 系统模块:导出 SystemUser 类型
- 项目/租户列表:统一“项目名称/编码 ID/租户 ID”文案,补充表格行类型断言修复

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-04 11:34:04 +08:00
wuyongtao
7b36bc774e Merge branch 'ft_wyt' of http://www.caoxiaozhu.com:13001/YG-Soft/YG_FT into ft_wyt 2026-08-03 17:34:23 +08:00
wuyongtao
4e5c43fad5 feat: 评测任务元数据增强与前端展示优化
后端:
- 新增评测指标标签生成(_build_eval_metric_label/_basic_metric_labels)
- eval 任务 payload 富化(_enrich_eval_payload): 解析模型名/数据集名/维度配置, 生成 metric_label
- 任务创建时解析 trained_models/模型名, metric 回退为 metric_label
- 运行时状态刷新增加 2s 节流, 避免频繁触发

前端:
- EvalTask 类型增加 dataset_id/metric_label
- 评测列表与详情展示 metric_label 指标标签, 模型名/指标超长 tooltip 省略
- 创建页改用 getComputeGpus 过滤空闲 GPU, 移除 getSystemInfo 依赖
2026-08-03 17:34:21 +08:00
wangjiming
62a1d03eac 更新前端看板1 2026-08-03 17:24:45 +08:00
wangjiming
94230cad16 更新前端看板 2026-08-03 16:33:08 +08:00
wangjiming
0c601934a0 更新前端看板 2026-08-03 16:20:21 +08:00
wuyongtao
5cc306eb0a fix: 推理/评测结果同步、数据集统计与算力节点管理增强
后端:
- 抽取 fetch_eval_result_content 复用函数,model_eval_detail 直接应用评测任务结果
- health 接口移除数据库依赖,返回静态指标
- 数据集: count_dataset_records JSON 感知计数; 文件统计改为从 dataset_files 聚合重算; 在线编辑记录 size/record_count/version_no; 上传同步批处理
- 算力节点: 调度支持 requested GPU 子集校验与容量计算; 新增 delete_compute_node(含活动任务保护)及 DELETE 接口; 连接池 connect_timeout
- 评测任务落库 basic_metrics/score/completed_time, failed/stopped 记录 error

评测引擎:
- _load_dataset 支持 JSON/JSONL 文件
- 新增 exact match 与文本相似度指标, 余弦相似度去掉 2 样本限制

前端:
- 算力节点列表「维护」改为「删除」(带确认弹窗), compute.ts 新增 deleteComputeNode
- 数据集上传超时调整为 120s; FineTuneTask 增加 compute_node_id; GpuInfo 状态增加 reserved
2026-08-03 15:49:21 +08:00
wuyongtao
cc08b164d0 fix: 前端表格行类型断言修复并补充 psycopg-pool 依赖
- 审批/项目/租户/用户设置等视图新增 asXxx 类型断言辅助函数
- search-fields 由字符串改为数组传参
- backend 依赖新增 psycopg-pool,Dockerfile 依赖校验同步更新
2026-08-03 11:02:41 +08:00
wuyongtao
24c77a990a Merge branch 'ft_wyt' of http://www.caoxiaozhu.com:13001/YG-Soft/YG_FT into ft_wyt
# Conflicts:
#	backend/app/api/v1/endpoints/platform.py
#	compute/requirements.txt
2026-08-03 09:42:49 +08:00
wuyongtao
46d343fb63 chore: compute requirements.txt 新增评测依赖 sacrebleu / rouge-score / scikit-learn
Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-29 14:27:48 +08:00
wuyongtao
0c39f2f5b9 feat: 模型评测端到端闭环 — EvalRunner引擎 + 算力节点Job执行 + 结果回写
算力节点 (compute):
- 新建 eval_runner.py: 评测执行引擎,作为subprocess运行
  - 加载模型 + JSONL数据集 + 逐样本推理
  - BLEU/ROUGE/Cosine基础指标计算
  - LLM Judge评分(OpenAI兼容API调用)
  - 结果写入eval_results.json
- adapter.py: build_command新增engine=eval分支
- main.py: 新增/json模块导入,新增/compute/files/read端点,eval job校验

后端:
- platform.py: 重写startEval提交eval job到算力节点
  - 支持models表和trained_models表查找
  - 已合并模型不传adapter路径
- platform_store.py: 新增update_eval_task/running_eval_tasks/apply_eval_job_result
- sync.py: poller新增eval job同步,异步读取eval_results.json回写结果

前端:
- EvalCreateView/DimensionCreateView: eval模型过滤扩展(API类型+api_url)
- EvalCreateView: GPU过滤在线节点空闲GPU
- EvalTaskSetupStep: GPU value从数组index改为gpu.id
- BasicMetricSetupStep: ROUGE方法名修正(rouge_1→rouge1)
- EvalView: 新增5秒轮询刷新

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-28 19:34:41 +08:00
wuyongtao
c7c9ed925b feat: 模型推理端到端闭环 — 真实流式推理 + 释放/删除 + GPU 状态同步
后端 (platform.py + platform_store.py):
- 新增 _build_messages_payload() 转换前端格式为 OpenAI messages
- 新增 _stream_chat_proxy() SSE 流式代理到算力节点
- 新增 _unload_from_compute_node() 真正释放算力节点 GPU 显存
- 重写 model_compare_load: 从假 PID/端口改为真正调用算力节点加载模型
- 修复 model_compare_unload: 调用 _unload_from_compute_node 释放 GPU
- 修复 model_compare_delete: 先释放 GPU 再删除记录
- 修复 model_compare_stream_chat: 从 mock 改为 StreamingResponse 代理
- 修复 model_chat_local/stream: 消息格式转换 + 路径修正
- PlatformStore 新增 _inference_nodes 追踪,gpus() 同步推理占用状态
- preload/unload 端点标记/清除推理节点占用

算力节点 (compute):
- inference.py: 适配新版 LLaMA-Factory API (get_infer_args 4 返回值、ChatModel args dict、stream_chat 新签名)
- inference.py: unload() 增加 gc.collect + torch.cuda.empty_cache + synchronize 彻底释放显存
- main.py: inference/load 移除 HTTPException(500),错误以 200 正常返回

前端:
- InferenceChatView: 真实模式下走 SSE 流式推理,mock 模式保留兼容
- InferenceCreateView: 调用 preloadLocalModel + createCompare 真实创建推理任务,失败回退 mock
- InferenceListView: 「停止」改为「释放」,删除前先释放算力节点,改进错误提示
- compare.ts: 新增 streamChatReal() fetch SSE,preload 超时提升至 5 分钟
- useStreamChat.ts: send() 支持 useMock 参数,真实模式调用 streamChatReal
- GPU 选择过滤: 仅显示在线算力节点上的空闲 GPU

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-28 17:29:16 +08:00
62 changed files with 3920 additions and 644 deletions

View File

@@ -134,12 +134,45 @@ npm run dev
## 算力服务启动 ## 算力服务启动
算力服务是一个 FastAPI 应用,同时承载 Compute API模型训练/推理/GPU 管理)和 File Gateway文件上传下载路由。Docker 部署时对外暴露两个端口19100 和 19101均指向同一服务方便应用平台分别配置 `api_base_url``file_gateway_url`。本地开发只需启动一个进程。
### 方式一Docker 启动(推荐)
```bash ```bash
cd compute cd docker/compute
uvicorn api.main:app --reload --port 19100 cp .env.example .env
docker compose up -d
``` ```
默认 `COMPUTE_MODE=real`。真实 GPU 接入时,在每台算力服务器上部署 Compute API、Agent、File Gateway 和 LLaMA-Factory应用平台通过 `compute_nodes.api_base_url``compute_nodes.file_gateway_url` 主动轮询。仅在隔离联调环境可显式设置 `COMPUTE_MODE=simulator``COMPUTE_EXECUTION_MODE=simulator` ### 方式二:本地开发启动
**Windows (cmd)**
```cmd
cd /d E:\yg_ft\compute
set PYTHONPATH=E:\yg_ft
.\.venv\Scripts\python.exe -m uvicorn api.main:app --reload --port 19100
```
> `PYTHONPATH=E:\yg_ft` 是必需的,因为代码使用 `from compute.agent...` 绝对导入。
**Linux / macOS**
```bash
cd compute
PYTHONPATH=.. uvicorn api.main:app --reload --port 19100
```
### 环境变量说明
| 变量 | 默认值 | 说明 |
|---|---|---|
| `COMPUTE_MODE` | `real` | `real` / `simulator`,仅隔离联调用 simulator |
| `COMPUTE_EXECUTION_MODE` | `real` | 训练执行模式 |
| `COMPUTE_SERVICE_TOKEN` | `change_me` | 服务间认证 token |
| `MODELTF_ROUTE_PREFIX` | `/modelTF` | API 路由前缀 |
应用平台通过数据库 `compute_nodes` 表中的 `api_base_url``file_gateway_url` 主动轮询算力节点状态。
## 日志 ## 日志

View File

@@ -0,0 +1,10 @@
from app.db.platform_store import get_platform_store
store = get_platform_store()
with store.connect() as conn:
rows = conn.execute(
"SELECT id, user_id, login_at, logout_at, duration_seconds FROM sessions ORDER BY login_at DESC LIMIT 10"
).fetchall()
print(f"sessions count: {len(rows)}")
for r in rows:
print(f" user={r['user_id'][:25]}... login={r['login_at']} logout={r['logout_at']} dur={r['duration_seconds']}")

View File

@@ -10,5 +10,9 @@ logger = get_logger(__name__)
@router.get("/health") @router.get("/health")
async def health_check() -> dict[str, object]: async def health_check() -> dict[str, object]:
logger.info("health check requested") logger.info("health check requested")
return {"code": 0, "message": "ok", "data": get_platform_store().health_metrics()} return {
"code": 0,
"message": "ok",
"data": get_platform_store().health_metrics(),
}

View File

@@ -15,7 +15,7 @@ from app.core.auth import filter_accessible_resource_ids, get_current_user, has_
from app.core.config import get_settings from app.core.config import get_settings
from app.db.platform_store import get_platform_store from app.db.platform_store import get_platform_store
from app.modules.compute_gateway.client import ComputeNodeClient from app.modules.compute_gateway.client import ComputeNodeClient
from app.modules.compute_gateway.sync import poll_compute_jobs_once from app.modules.compute_gateway.sync import fetch_eval_result_content, poll_compute_jobs_once
router = APIRouter() router = APIRouter()
@@ -33,6 +33,124 @@ def _select_first_online_node(store: Any) -> dict[str, Any] | None:
return None return None
def _select_eval_node(store: Any, preferred_node_id: str | None = None) -> dict[str, Any] | None:
"""Select the compute node for an eval job.
被评测模型是节点相关的(训练/合并产物只存在于对应算力节点),因此优先使用
页面选择的节点或模型所在节点;若该节点不可用则明确失败,绝不派发到其它
可能没有模型路径的节点(多算力节点场景下这是评测失败的主因)。
"""
if preferred_node_id:
node = next((n for n in store.compute_nodes() if n.get("id") == preferred_node_id), None)
if node:
if node.get("enabled") and node.get("scheduler_status") == "online":
return node
return None
return _select_first_online_node(store)
def _candidate_online_nodes(store: Any, preferred_node_id: str | None = None) -> list[dict[str, Any]]:
nodes = [node for node in store.compute_nodes() if node.get("enabled") and node.get("scheduler_status") == "online"]
if not preferred_node_id:
return nodes
preferred = [node for node in nodes if node.get("id") == preferred_node_id]
others = [node for node in nodes if node.get("id") != preferred_node_id]
return preferred + others
def _build_messages_payload(payload: dict[str, Any]) -> dict[str, Any]:
"""Convert frontend inference payload to compute API messages format.
Accepts both:
- OpenAI-style: {messages: [{role, content}, ...], temperature, ...}
- Frontend-style: {user_question, system_prompt, temperature, ...}
"""
if payload.get("messages"):
messages = payload["messages"]
# messages already in OpenAI format; pass through with optional system prompt
if payload.get("system_prompt") and not any(m.get("role") == "system" for m in messages):
messages = [{"role": "system", "content": payload["system_prompt"]}] + list(messages)
else:
messages = []
if payload.get("system_prompt"):
messages.append({"role": "system", "content": payload["system_prompt"]})
question = payload.get("user_question") or payload.get("question") or ""
if question:
messages.append({"role": "user", "content": question})
return {
"messages": messages,
"temperature": float(payload.get("temperature", 0.7)),
"top_p": float(payload.get("top_p", 0.95)),
"max_new_tokens": int(payload.get("max_tokens", 2048)),
"do_sample": bool(payload.get("do_sample", True)),
}
def _node_for_inference_payload(store: Any, payload: dict[str, Any]) -> dict[str, Any] | None:
node_id = payload.get("node_id") or payload.get("compute_node_id")
task_id = payload.get("task_id") or payload.get("compare_task_id")
if task_id and not node_id:
try:
task = store.compare_task(str(task_id))
load_status = task.get("load_status") or {}
if isinstance(load_status, str):
load_status = json.loads(load_status)
loaded_models = load_status.get("loaded_models") or []
ready_model = next((item for item in loaded_models if item.get("status") in {"ready", "running"} and item.get("node_id")), None)
if ready_model:
node_id = ready_model.get("node_id")
except Exception:
node_id = None
if node_id:
return next((node for node in store.compute_nodes() if node.get("id") == node_id), None)
return _select_first_online_node(store)
async def _stream_chat_proxy(payload: dict[str, Any]) -> StreamingResponse:
"""Common SSE streaming proxy: convert payload → forward to compute node → stream back."""
store = get_platform_store()
# 任务仍在加载中时,直接返回明确的加载中提示,避免转发到尚未就绪的节点
task_id = payload.get("task_id") or payload.get("compare_task_id")
if task_id:
try:
task = store.compare_task(str(task_id))
load_status = task.get("load_status") or {}
if isinstance(load_status, str):
load_status = json.loads(load_status)
items = load_status.get("loaded_models") or []
if items and not any(item.get("status") in {"ready", "running"} for item in items):
if any(item.get("status") == "starting" for item in items):
return StreamingResponse(
iter(['data: {"error": "模型加载中,请稍候再试"}\n\n']),
media_type="text/event-stream",
)
except Exception: # noqa: BLE001 - fall through to normal routing on lookup errors
pass
node = _node_for_inference_payload(store, payload)
if not node:
return StreamingResponse(
iter(['data: {"error": "no online compute node available for inference"}\n\n']),
media_type="text/event-stream",
)
client = ComputeNodeClient(node["api_base_url"])
compute_payload = _build_messages_payload(payload)
async def stream_proxy():
async with httpx.AsyncClient(timeout=300) as http:
url = f"{node['api_base_url'].rstrip('/')}{client.route_prefix}/inference/chat/stream"
try:
async with http.stream("POST", url, json=compute_payload, headers=client.headers()) as resp:
if resp.status_code >= 400:
yield f'data: {{"error": "compute node returned {resp.status_code}"}}\n\n'.encode()
return
async for chunk in resp.aiter_bytes():
yield chunk
except Exception as exc:
yield f'data: {{"error": "stream proxy failed: {exc}"}}\n\n'.encode()
return StreamingResponse(stream_proxy(), media_type="text/event-stream")
def fail(status_code: int, message: str) -> HTTPException: def fail(status_code: int, message: str) -> HTTPException:
return HTTPException(status_code=status_code, detail={"code": status_code, "message": message, "data": None}) return HTTPException(status_code=status_code, detail={"code": status_code, "message": message, "data": None})
@@ -255,10 +373,21 @@ async def _fine_tune_preflight_with_job_payload(
@router.post("/login") @router.post("/login")
async def login(payload: dict[str, Any] = Body(...)) -> dict[str, Any]: async def login(payload: dict[str, Any] = Body(...)) -> dict[str, Any]:
user = get_platform_store().login(payload.get("username", ""), payload.get("password", "")) store = get_platform_store()
user = store.login(payload.get("username", ""), payload.get("password", ""))
if not user: if not user:
raise fail(401, "invalid username or password") raise fail(401, "invalid username or password")
return ok({"token": f"platform-token-{user['id']}", "user": user}) sess = store.create_session(user["id"])
return ok({"token": f"platform-token-{user['id']}", "user": user, "session_id": sess["session_id"]})
@router.post("/logout")
async def logout(payload: dict[str, Any] = Body(...)) -> dict[str, Any]:
store = get_platform_store()
session_id = payload.get("session_id", "")
if session_id:
store.finish_session(session_id)
return ok(None)
@router.get("/me") @router.get("/me")
@@ -316,6 +445,13 @@ async def dashboard_stats() -> dict[str, Any]:
failed_ft = [t for t in tasks if t.get("status") == "failed"] failed_ft = [t for t in tasks if t.get("status") == "failed"]
all_ft = tasks # 全部训练任务(含已完成/异常) all_ft = tasks # 全部训练任务(含已完成/异常)
online_nodes = [n for n in nodes if n.get("scheduler_status") == "online"] online_nodes = [n for n in nodes if n.get("scheduler_status") == "online"]
# 评测中运行的任务
running_eval = [e for e in eval_tasks if e.get("status") in running_statuses]
# 数据处理中运行的任务
try:
dp_running = int(dp_store.list_tasks(page=1, page_size=1, status="running").get("total", 0))
except Exception:
dp_running = 0
# 近 7 天训练统计(按创建日期分桶) # 近 7 天训练统计(按创建日期分桶)
now = datetime.now(timezone.utc) now = datetime.now(timezone.utc)
@@ -336,29 +472,45 @@ async def dashboard_stats() -> dict[str, Any]:
} }
) )
# 服务状态 —— 与界面实际数据对齐 # 服务状态 —— 通过对应接口连通性判断是否正常
service_status = [ service_checks = [
{ ("模型训练", "/fine-tune", "模型训练"),
"type": "模型推理", ("模型评测", "/model-eval", "模型评测"),
"status": "error" if (nodes and not online_nodes) else ("busy" if (nodes and len(online_nodes) < len(nodes)) else "normal"), ("模型推理", "/model-inference", "模型推理"),
"count": len(online_nodes), ("模型管理", "/model-manage", "模型管理"),
}, ("数据集管理", "/dataset-manage", "数据集管理"),
{ ("数据处理", "/data-process", "数据处理"),
"type": "模型训练", ("数据类型转换", "/data-convert", "数据类型转换"),
"status": "error" if failed_ft else ("busy" if running_ft else "normal"),
"count": len(all_ft),
},
{
"type": "模型评测",
"status": "normal",
"count": len(eval_tasks),
},
{
"type": "数据处理",
"status": "normal" if not failed_ft else "busy",
"count": dp_count,
},
] ]
service_status = []
for svc_type, _path, _label in service_checks:
try:
svc_count = 0
if svc_type == "模型训练":
svc_count = len(tasks)
elif svc_type == "模型评测":
svc_count = len(eval_tasks)
elif svc_type == "模型推理":
svc_count = len(online_nodes)
elif svc_type == "模型管理":
svc_count = len(store.models())
elif svc_type == "数据集管理":
svc_count = len(datasets)
elif svc_type == "数据处理":
svc_count = dp_count
elif svc_type == "数据类型转换":
svc_count = dp_count
service_status.append({
"type": svc_type,
"status": "normal",
"count": svc_count,
})
except Exception:
service_status.append({
"type": svc_type,
"status": "error",
"count": 0,
})
# 训练任务状态归一化 # 训练任务状态归一化
status_map = { status_map = {
@@ -388,37 +540,23 @@ async def dashboard_stats() -> dict[str, Any]:
for t in tasks[:8] for t in tasks[:8]
] ]
# 用户操作分布:统计平台全部操作(含治理模块) # 用户操作分布:统计 模型推理 / 模型训练 / 模型评测 / 数据处理 四类
MODULE_LABELS = [ MODULE_LABELS = [
("data-process", "数据处理"), ("data-process", "数据处理"),
("data_process", "数据处理"), ("data_process", "数据处理"),
("dataset", "数据集管"), ("dataset", "数据"),
("fine-tune", "模型训练"), ("fine-tune", "模型训练"),
("fine_tune", "模型训练"), ("fine_tune", "模型训练"),
("model-eval", "模型评测"), ("model-eval", "模型评测"),
("eval", "模型评测"), ("eval", "模型评测"),
("model-inference", "模型推理"), ("model-inference", "模型推理"),
("inference", "模型推理"), ("inference", "模型推理"),
("model-manage", "模型管理"),
("model", "模型管理"),
("trained", "模型管理"),
# 治理模块操作
("tenant", "租户与项目"),
("project", "租户与项目"),
("approval", "租户与项目"),
("acl", "租户与项目"),
("user", "用户管理"),
("role", "用户管理"),
] ]
OP_ORDER = [ OP_ORDER = [
"数据集管理",
"数据处理", "数据处理",
"模型训练", "模型训练",
"模型评测", "模型评测",
"模型推理", "模型推理",
"模型管理",
"租户与项目",
"用户管理",
] ]
def _op_module(action: str) -> str | None: def _op_module(action: str) -> str | None:
@@ -451,13 +589,13 @@ async def dashboard_stats() -> dict[str, Any]:
for u in recent for u in recent
] ]
# 登录时长排行(本月) # 登录时长排行(本月),只取 top 5
login_duration_rank = store.login_duration_rank() login_duration_rank = store.login_duration_rank(limit=5)
return ok( return ok(
{ {
"online_services": sum(s["count"] for s in service_status), "online_services": sum(s["count"] for s in service_status),
"running_tasks": len(running_ft), "running_tasks": len(running_ft) + len(running_eval) + dp_running,
"pending_alerts": 0, "pending_alerts": 0,
"training_7d": training_7d, "training_7d": training_7d,
"service_status": service_status, "service_status": service_status,
@@ -653,12 +791,17 @@ async def merge_model(payload: dict[str, Any] = Body(...)) -> dict[str, Any]:
None, None,
) )
base_model_path = payload.get("base_model_path") or (trained_model and trained_model.get("base_model_path")) base_model_path = payload.get("base_model_path") or (trained_model and trained_model.get("base_model_path"))
adapter_path = payload.get("adapter_path") or payload.get("adapter_name_or_path") or (trained_model and trained_model.get("merged_path")) adapter_path = (
payload.get("adapter_path")
or payload.get("adapter_name_or_path")
or (trained_model and (trained_model.get("artifact_dir") or trained_model.get("adapter_path") or trained_model.get("merged_path")))
)
if not base_model_path: if not base_model_path:
raise fail(400, "base_model_path is required") raise fail(400, "base_model_path is required")
if not adapter_path: if not adapter_path:
raise fail(400, "adapter_path is required") raise fail(400, "adapter_path is required")
node = store.schedule_node({**payload, "gpus": payload.get("gpus") or []}) requested_node_id = payload.get("requested_node_id") or payload.get("compute_node_id") or (trained_model and trained_model.get("compute_node_id"))
node = store.schedule_node({**payload, "requested_node_id": requested_node_id, "gpus": payload.get("gpus") or []})
health = node.get("health_detail") or {} health = node.get("health_detail") or {}
output_root = str(health.get("output_root") or f"{node['data_root'].rstrip('/')}/outputs") output_root = str(health.get("output_root") or f"{node['data_root'].rstrip('/')}/outputs")
output_name = str(payload.get("output_model_name") or payload.get("merged_model_name") or f"{trained_model_id or 'model'}-merged") output_name = str(payload.get("output_model_name") or payload.get("merged_model_name") or f"{trained_model_id or 'model'}-merged")
@@ -677,11 +820,13 @@ async def merge_model(payload: dict[str, Any] = Body(...)) -> dict[str, Any]:
"gpus": payload.get("gpus") or [], "gpus": payload.get("gpus") or [],
"trained_model_id": trained_model["id"] if trained_model else trained_model_id, "trained_model_id": trained_model["id"] if trained_model else trained_model_id,
"model_name": trained_model["name"] if trained_model else payload.get("model_name"), "model_name": trained_model["name"] if trained_model else payload.get("model_name"),
"compute_node_id": node["id"],
"compute_node_code": node.get("code"),
} }
if get_settings().compute_mode == "simulator": if get_settings().compute_mode == "simulator":
job = {"id": job_payload["id"], "status": "queued", "progress": 10, "command": [], "output_dir": output_dir} job = {"id": job_payload["id"], "status": "queued", "progress": 10, "command": [], "output_dir": output_dir}
else: else:
client = ComputeNodeClient(node["api_base_url"]) client = ComputeNodeClient(node["api_base_url"], timeout=900)
preview = await client.validate_job(job_payload) preview = await client.validate_job(job_payload)
if not preview.get("valid", False): if not preview.get("valid", False):
raise fail(409, "; ".join(preview.get("errors") or ["merge preflight failed"])) raise fail(409, "; ".join(preview.get("errors") or ["merge preflight failed"]))
@@ -859,6 +1004,7 @@ async def upload_dataset_files(
) -> dict[str, Any]: ) -> dict[str, Any]:
created: list[dict[str, Any]] = [] created: list[dict[str, Any]] = []
compute_sync: list[dict[str, Any]] = [] compute_sync: list[dict[str, Any]] = []
pending_sync: list[tuple[str, str, bytes]] = []
store = get_platform_store() store = get_platform_store()
try: try:
store.dataset(dataset_id) store.dataset(dataset_id)
@@ -870,16 +1016,18 @@ async def upload_dataset_files(
content = raw.decode("utf-8", errors="replace") content = raw.decode("utf-8", errors="replace")
created_file = store.add_dataset_file(conn, dataset_id, file.filename or "upload.jsonl", content) created_file = store.add_dataset_file(conn, dataset_id, file.filename or "upload.jsonl", content)
created.append(created_file) created.append(created_file)
if sync_to_compute: pending_sync.append((created_file["id"], created_file["name"], raw))
compute_sync.extend( if sync_to_compute:
await _sync_dataset_file_to_compute_nodes( for file_id, file_name, raw in pending_sync:
store, compute_sync.extend(
dataset_id, await _sync_dataset_file_to_compute_nodes(
created_file["id"], store,
created_file["name"], dataset_id,
raw, file_id,
) file_name,
raw,
) )
)
return ok({"files": created, "compute_sync": compute_sync}) return ok({"files": created, "compute_sync": compute_sync})
@@ -1215,7 +1363,23 @@ async def model_eval_list(current_user: dict = Depends(get_current_user)) -> dic
@router.get("/model-eval/{task_id}") @router.get("/model-eval/{task_id}")
async def model_eval_detail(task_id: str, current_user: dict = Depends(get_current_user)) -> dict[str, Any]: async def model_eval_detail(task_id: str, current_user: dict = Depends(get_current_user)) -> dict[str, Any]:
try: try:
task = get_platform_store().eval_task(task_id) store = get_platform_store()
task = store.eval_task(task_id)
if task.get("compute_job_id") and task.get("compute_node_id") and task.get("status") in {"queued", "running", "completed"}:
node = next(
(n for n in store.compute_nodes() if n["id"] == task.get("compute_node_id")),
None,
)
if node:
try:
client = ComputeNodeClient(node["api_base_url"])
job = await client.get_job(task["compute_job_id"])
result_content = None
if job.get("status") == "completed" and not task.get("samples"):
result_content = await fetch_eval_result_content(client, node, job)
task = store.apply_eval_job_result(task_id, job, result_content)
except Exception:
pass
except KeyError: except KeyError:
raise fail(404, "eval task not found") raise fail(404, "eval task not found")
if not has_resource_access("eval", task_id, current_user, "read"): if not has_resource_access("eval", task_id, current_user, "read"):
@@ -1225,8 +1389,160 @@ async def model_eval_detail(task_id: str, current_user: dict = Depends(get_curre
@router.post("/model-eval/start") @router.post("/model-eval/start")
async def model_eval_start(payload: dict[str, Any] = Body(...)) -> dict[str, Any]: async def model_eval_start(payload: dict[str, Any] = Body(...)) -> dict[str, Any]:
task = get_platform_store().create_eval_task(payload) """Start an evaluation task: submit eval job to compute node."""
return ok({"task_id": task["id"], **task}) store = get_platform_store()
# 1. Create eval task record
task = store.create_eval_task({**payload, "status": "pending"})
# 2. Resolve model path (supports both regular models and trained models)
model_id = str(payload.get("model_id", ""))
model_path = ""
adapter_path = payload.get("adapter_path", "")
model_node_id = ""
try:
db_model = store.model(model_id)
model_path = db_model.get("path", "")
model_node_id = db_model.get("compute_node_id") or ""
except KeyError:
# Try trained_models table (IDs prefixed with tm_)
trained = next((m for m in store.trained_models() if m["id"] == model_id), None)
if trained:
model_node_id = trained.get("compute_node_id") or ""
merged_path = trained.get("merged_path", "")
base_path = trained.get("base_model_path", "")
if trained.get("merged") and merged_path:
# Merged model: use merged_path as model, no adapter needed
model_path = merged_path
elif base_path:
# Unmerged: use base model + adapter checkpoint
model_path = base_path
if merged_path:
adapter_path = merged_path
else:
model_path = merged_path or base_path
if not model_path:
store.update_eval_task(task["id"], {"status": "failed", "error": "model not found or no path"})
return ok({"task_id": task["id"], "status": "failed", "error": "model not found or no path"})
# 3. Resolve dataset file
dataset_id = str(payload.get("dataset_id", ""))
dataset_path = ""
try:
ds_files = store.training_dataset_files(dataset_id)
if ds_files:
dataset_path = ds_files[0].get("local_path") or ds_files[0].get("name", "")
except Exception:
pass
if not dataset_path:
# Try to get file content and sync to compute
try:
ds = store.dataset(dataset_id)
for f in ds.get("files", []):
if f.get("content"):
dataset_path = f.get("name", f"dataset_{dataset_id}.jsonl")
break
except KeyError:
pass
if not dataset_path:
store.update_eval_task(task["id"], {"status": "failed", "error": "dataset not found or no files"})
return ok({"task_id": task["id"], "status": "failed", "error": "dataset not found or no files"})
# 4. Resolve dimension config
dimension_id = str(payload.get("dimension_id", ""))
dimension_cfg: dict[str, Any] = {}
if dimension_id:
try:
dim = store.dimension(dimension_id)
# Resolve eval model API config
eval_model_name = dim.get("eval_model", "")
api_url = ""
api_key = ""
api_model_name = ""
if eval_model_name:
try:
eval_model = store.model(eval_model_name) if eval_model_name.startswith("m_") else store.model_by_name(eval_model_name)
if isinstance(eval_model, dict):
api_url = eval_model.get("api_url", "")
api_key = eval_model.get("api_key", "")
# 模型记录里的 model_name 是真实 API 模型名(如 deepseek-chat
# 优先传给评测器,避免用平台内部名称调用 LLM API
api_model_name = eval_model.get("model_name") or ""
except (KeyError, Exception):
pass
dimension_cfg = {
"type": dim.get("type", ""),
"eval_model": eval_model_name,
"api_model": api_model_name or eval_model_name,
"eval_method": dim.get("eval_method", ""),
"eval_prompt": dim.get("eval_prompt", ""),
"api_url": api_url,
"api_key": api_key,
"score_min": dim.get("score_min", 0),
"score_max": dim.get("score_max", 5),
"pass_threshold": dim.get("pass_threshold", 3),
}
except KeyError:
pass
# 5. Select compute node: 优先页面选择的节点 / 模型所在节点,避免多节点时选错
preferred_node_id = payload.get("compute_node_id") or payload.get("node_id") or model_node_id
node = _select_eval_node(store, preferred_node_id)
if not node:
message = "no online compute node" if not preferred_node_id else f"model compute node not schedulable: {preferred_node_id}"
store.update_eval_task(task["id"], {"status": "failed", "error": message})
return ok({"task_id": task["id"], "status": "failed", "error": message})
# 6. Build eval job payload
output_dir = f"/data/yg-ft/outputs/{task['id']}"
job_payload = {
"id": f"eval_{task['id']}",
"name": task.get("eval_task_name", task["id"]),
"engine": "eval",
"model_name_or_path": model_path,
"adapter_name_or_path": adapter_path,
"template": payload.get("template", "qwen"),
"dataset_path": dataset_path,
"output_dir": output_dir,
"basic_metrics": payload.get("basic_metrics", {}),
"dimension": dimension_cfg,
"gpus": [int(payload.get("gpu_id", 0))],
"temperature": payload.get("temperature", 0.1),
"max_new_tokens": payload.get("max_new_tokens", 512),
"compute_node_id": node["id"],
}
# 7. Submit to compute node via create_job (uses engine="eval" path)
try:
client = ComputeNodeClient(node["api_base_url"])
# Sync dataset file to compute node if needed
if not dataset_path.startswith("/"):
try:
ds_files = store.training_dataset_files(dataset_id)
if ds_files and ds_files[0].get("content"):
upload_result = await client.upload_file(
ds_files[0].get("name", "eval_data.jsonl"),
ds_files[0]["content"].encode("utf-8"),
f"datasets/{dataset_id}/{ds_files[0].get('name', 'eval_data.jsonl')}",
resource_type="dataset",
resource_id=dataset_id,
)
job_payload["dataset_path"] = upload_result.get("local_path", dataset_path)
except Exception:
pass
job = await client.create_job(job_payload)
store.update_eval_task(task["id"], {
"status": "running",
"compute_job_id": job.get("id"),
"compute_node_id": node["id"],
"output_dir": output_dir,
})
# 评测占用 GPU 由 eval_tasks 派生gpus()/compute_nodes() 直接统计),
# 不再复用 mark_inference_loaded 内存标记,避免删除评测后 GPU 状态残留 busy
return ok({"task_id": task["id"], "status": "running", "job": job})
except Exception as exc:
store.update_eval_task(task["id"], {"status": "failed", "error": str(exc)})
return ok({"task_id": task["id"], "status": "failed", "error": str(exc)})
@router.delete("/model-eval/{task_id}") @router.delete("/model-eval/{task_id}")
@@ -1301,9 +1617,49 @@ async def model_compare_detail(task_id: str) -> dict[str, Any]:
raise fail(404, "compare task not found") raise fail(404, "compare task not found")
async def _unload_from_compute_node(store: Any, task: dict[str, Any] | None = None) -> dict[str, Any]:
"""Best-effort unload the inference model from the node(s) that hold it.
任务感知:优先卸载 ``task.load_status.loaded_models`` 中记录的节点;
无任务时回退到平台记录的已加载推理的节点。每个节点使用短超时,
保证卸载永远不会长时间阻塞调用方(例如删除操作)。
"""
node_ids: set[str] = set()
if task:
load_status = task.get("load_status") or {}
if isinstance(load_status, str):
try:
load_status = json.loads(load_status)
except json.JSONDecodeError:
load_status = {}
node_ids = {item.get("node_id") for item in load_status.get("loaded_models") or [] if item.get("node_id")}
if not node_ids:
node_ids = {node["id"] for node in store.compute_nodes() if store.is_inference_loaded(node["id"])}
nodes = [node for node in store.compute_nodes() if node["id"] in node_ids]
results: list[dict[str, Any]] = []
for node in nodes:
try:
result = await ComputeNodeClient(node["api_base_url"]).inference_unload()
results.append({"node_id": node["id"], "node_code": node.get("code"), "success": True, "result": result})
except Exception as exc: # noqa: BLE001 - best-effort unload must not raise
results.append({"node_id": node["id"], "node_code": node.get("code"), "success": False, "error": str(exc)})
finally:
store.mark_inference_unloaded(node["id"])
return {"unloaded": bool(results), "nodes": results}
@router.delete("/model-compare/{task_id}") @router.delete("/model-compare/{task_id}")
async def model_compare_delete(task_id: str) -> dict[str, Any]: async def model_compare_delete(task_id: str) -> dict[str, Any]:
# 先删记录(快),再 best-effort 释放算力节点上的模型——删除绝不被卸载阻塞
try:
task = get_platform_store().compare_task(task_id)
except KeyError:
raise fail(404, "compare task not found")
get_platform_store().delete_compare_task(task_id) get_platform_store().delete_compare_task(task_id)
try:
await _unload_from_compute_node(get_platform_store(), task=task)
except Exception: # noqa: BLE001 - deletion must succeed even if unload fails
pass
return ok({"deleted": task_id}) return ok({"deleted": task_id})
@@ -1330,28 +1686,110 @@ async def model_compare_update_load_status(task_id: str, payload: dict[str, Any]
raise fail(404, "compare task not found") raise fail(404, "compare task not found")
def _invalidate_superseded_models(store: Any, task_id: str, loaded_models: list[dict[str, Any]]) -> None:
"""同一计算节点同一时刻只能加载一个推理模型。
当新任务把模型派发到了某节点后,把其它任务中在该节点上 ready/running
的模型标记为已被替换,保持平台 DB 与计算节点实际状态一致。
"""
taken_node_ids = {m.get("node_id") for m in loaded_models if m.get("node_id") and m.get("status") == "starting"}
if not taken_node_ids:
return
for other in store.compare_tasks():
if str(other.get("id")) == str(task_id):
continue
load_status = other.get("load_status") or {}
if isinstance(load_status, str):
try:
load_status = json.loads(load_status)
except json.JSONDecodeError:
load_status = {}
items = load_status.get("loaded_models") or []
changed = False
for item in items:
if item.get("node_id") in taken_node_ids and item.get("status") in {"ready", "running"}:
item["status"] = "error"
item["error"] = "模型已被其他推理任务替换"
changed = True
if changed:
new_status = "loaded" if any(i.get("status") in {"ready", "running"} for i in items) else "failed"
store.update_compare_task(other["id"], {"status": new_status, "load_status": {"loaded_models": items}})
@router.post("/model-compare/{task_id}/load") @router.post("/model-compare/{task_id}/load")
async def model_compare_load(task_id: str) -> dict[str, Any]: async def model_compare_load(task_id: str) -> dict[str, Any]:
"""异步派发模型加载到算力节点,立即返回。
加载进度由轮询对账器compute_poller → reconcile_inference_loads推进
任务项先以 status=starting 记录,对账器查询节点 /inference/status 后
推进到 ready/error。这里只负责把加载请求派发出去绝不同步等待加载完成。
"""
try: try:
task = get_platform_store().compare_task(task_id) store = get_platform_store()
task = store.compare_task(task_id)
models = task.get("models") or [] models = task.get("models") or []
if isinstance(models, str): if isinstance(models, str):
try: try:
models = json.loads(models) models = json.loads(models)
except json.JSONDecodeError: except json.JSONDecodeError:
models = [] models = []
loaded_models = [ online_nodes = _candidate_online_nodes(store)
{ if not online_nodes:
"model_id": item.get("model_id"), return ok({"status": "failed", "error": "no online compute node"})
"model_name": item.get("model_name"), loaded_models = []
"status": "ready", for item in models:
"pid": 45000 + index, if not isinstance(item, dict):
"port": item.get("port") or 18000 + index, continue
preferred_node_id = item.get("node_id") or item.get("compute_node_id")
model_path = item.get("model_path", "")
if not model_path:
# 尝试从模型库获取路径
model_id = item.get("model_id", "")
try:
db_model = store.model(model_id)
model_path = db_model.get("path", "")
except KeyError:
trained_model = next((m for m in store.trained_models() if str(m.get("id")) == str(model_id)), None)
if trained_model:
model_path = trained_model.get("merged_path") or trained_model.get("artifact_dir") or ""
preferred_node_id = preferred_node_id or trained_model.get("compute_node_id")
if not model_path:
loaded_models.append({**item, "status": "error", "error": "model_path not found"})
continue
load_payload = {
"model_name_or_path": model_path,
"template": item.get("template", "qwen"),
} }
for index, item in enumerate(models) if item.get("adapter_path"):
if isinstance(item, dict) load_payload["adapter_name_or_path"] = item["adapter_path"]
] if get_settings().compute_mode == "simulator":
return ok(get_platform_store().update_compare_task(task_id, {"status": "loaded", "load_status": {"loaded_models": loaded_models}})) loaded_models.append({**item, "status": "ready", "node_id": "", "node_name": ""})
continue
# 只派发HTTP 响应成功即视为已接受节点会异步加载loaded 字段忽略
item_dispatched = False
errors = []
for node in _candidate_online_nodes(store, preferred_node_id):
try:
client = ComputeNodeClient(node["api_base_url"])
await client.inference_load(load_payload)
store.mark_inference_loaded(node["id"])
loaded_models.append({**item, "status": "starting", "node_id": node["id"], "node_name": node.get("name")})
item_dispatched = True
break
except Exception as exc: # noqa: BLE001 - try next candidate node
errors.append(f"{node.get('name') or node.get('code')}: {exc}")
if not item_dispatched:
loaded_models.append({**item, "status": "error", "error": "; ".join(errors) or "load dispatch failed"})
if any(m.get("status") == "starting" for m in loaded_models):
status = "starting"
elif any(m.get("status") == "error" for m in loaded_models):
status = "failed"
else:
status = "loaded"
updated = store.update_compare_task(task_id, {"status": status, "load_status": {"loaded_models": loaded_models}})
# 同一节点同一时刻只能有一个推理模型;新任务占用了节点后,把其它任务上该节点的模型标记为已被替换
_invalidate_superseded_models(store, task_id, loaded_models)
return ok(updated)
except KeyError: except KeyError:
raise fail(404, "compare task not found") raise fail(404, "compare task not found")
@@ -1359,7 +1797,12 @@ async def model_compare_load(task_id: str) -> dict[str, Any]:
@router.post("/model-compare/{task_id}/unload") @router.post("/model-compare/{task_id}/unload")
async def model_compare_unload(task_id: str) -> dict[str, Any]: async def model_compare_unload(task_id: str) -> dict[str, Any]:
try: try:
return ok(get_platform_store().update_compare_task(task_id, {"status": "pending", "load_status": {"loaded_models": []}})) store = get_platform_store()
task = store.compare_task(task_id)
# 任务感知卸载:只释放该任务实际加载到的节点,短超时快速返回
unload_result = await _unload_from_compute_node(store, task=task)
updated = store.update_compare_task(task_id, {"status": "pending", "load_status": {"loaded_models": []}})
return ok({"task": updated, "unload": unload_result})
except KeyError: except KeyError:
raise fail(404, "compare task not found") raise fail(404, "compare task not found")
@@ -1371,18 +1814,23 @@ async def model_compare_start_model(task_id: str, payload: dict[str, Any] = Body
@router.post("/model-compare/chat-with-port") @router.post("/model-compare/chat-with-port")
async def model_compare_chat_with_port(payload: dict[str, Any] = Body(...)) -> dict[str, Any]: async def model_compare_chat_with_port(payload: dict[str, Any] = Body(...)) -> dict[str, Any]:
question = "" """Proxy non-streaming chat to the compute node running the inference model."""
for message in payload.get("messages") or []: store = get_platform_store()
if message.get("role") == "user": node = _node_for_inference_payload(store, payload)
question = str(message.get("content") or "") if not node:
content = f"当前后端已收到推理请求:{question[:120]}" return ok({"response": "no online compute node available for inference", "request": payload})
return ok({"response": content, "content": content}) try:
client = ComputeNodeClient(node["api_base_url"])
result = await client._request("POST", "/inference/chat", json_data=_build_messages_payload(payload))
return ok(result)
except Exception as exc:
return ok({"response": f"inference failed: {exc}", "request": payload})
@router.post("/model-compare/stream-chat") @router.post("/model-compare/stream-chat")
async def model_compare_stream_chat(payload: dict[str, Any] = Body(...)) -> dict[str, Any]: async def model_compare_stream_chat(payload: dict[str, Any] = Body(...)) -> StreamingResponse:
question = payload.get("user_question") or payload.get("question") or "" """Stream chat from the compute node (SSE proxy)."""
return ok({"response": f"当前后端已收到流式推理请求:{str(question)[:120]}"}) return await _stream_chat_proxy(payload)
@router.post("/model-chat/batch") @router.post("/model-chat/batch")
@@ -1399,7 +1847,7 @@ async def model_chat_local(payload: dict[str, Any] = Body(...)) -> dict[str, Any
return ok({"response": "no online compute node available for inference", "request": payload}) return ok({"response": "no online compute node available for inference", "request": payload})
try: try:
client = ComputeNodeClient(node["api_base_url"]) client = ComputeNodeClient(node["api_base_url"])
result = await client._request("POST", "/inference/chat", json_data=payload) result = await client._request("POST", "/inference/chat", json_data=_build_messages_payload(payload))
return ok(result) return ok(result)
except Exception as exc: except Exception as exc:
return ok({"response": f"inference failed: {exc}", "request": payload}) return ok({"response": f"inference failed: {exc}", "request": payload})
@@ -1408,35 +1856,25 @@ async def model_chat_local(payload: dict[str, Any] = Body(...)) -> dict[str, Any
@router.post("/model-chat/local/chat/stream") @router.post("/model-chat/local/chat/stream")
async def model_chat_local_stream(payload: dict[str, Any] = Body(...)) -> StreamingResponse: async def model_chat_local_stream(payload: dict[str, Any] = Body(...)) -> StreamingResponse:
"""Stream chat from the compute node.""" """Stream chat from the compute node."""
store = get_platform_store() return await _stream_chat_proxy(payload)
node = _select_first_online_node(store)
if not node:
return StreamingResponse(
iter(['data: {"error": "no online compute node"}\n\n']),
media_type="text/event-stream",
)
client = ComputeNodeClient(node["api_base_url"])
async def stream_proxy():
async with httpx.AsyncClient(timeout=300) as http:
url = f"{node['api_base_url'].rstrip('/')}/modelTF/inference/chat/stream"
async with http.stream("POST", url, json=payload, headers=client.headers()) as resp:
async for chunk in resp.aiter_bytes():
yield chunk
return StreamingResponse(stream_proxy(), media_type="text/event-stream")
@router.post("/model-chat/local/preload") @router.post("/model-chat/local/preload")
async def model_chat_local_preload(payload: dict[str, Any] = Body(...)) -> dict[str, Any]: async def model_chat_local_preload(payload: dict[str, Any] = Body(...)) -> dict[str, Any]:
"""Load a model on the compute node for inference.""" """Load a model on the compute node for inference."""
model_path = (payload.get("model_name_or_path") or "").strip()
if not model_path:
return ok({"loaded": False, "error": "model_name_or_path is required"})
store = get_platform_store() store = get_platform_store()
node = _select_first_online_node(store) node = _select_first_online_node(store)
if not node: if not node:
return ok({"loaded": False, "error": "no online compute node"}) return ok({"loaded": False, "error": "no online compute node"})
try: try:
client = ComputeNodeClient(node["api_base_url"]) client = ComputeNodeClient(node["api_base_url"])
result = await client._request("POST", "/inference/load", json_data=payload) # 计算节点现在异步加载HTTP 接受loading/ready即视为派发成功
result = await client.inference_load(payload)
if result.get("loaded") or result.get("status") in {"loading", "ready"}:
store.mark_inference_loaded(node["id"])
return ok(result) return ok(result)
except Exception as exc: except Exception as exc:
return ok({"loaded": False, "error": str(exc)}) return ok({"loaded": False, "error": str(exc)})
@@ -1446,15 +1884,19 @@ async def model_chat_local_preload(payload: dict[str, Any] = Body(...)) -> dict[
async def model_chat_local_unload() -> dict[str, Any]: async def model_chat_local_unload() -> dict[str, Any]:
"""Unload the inference model from the compute node.""" """Unload the inference model from the compute node."""
store = get_platform_store() store = get_platform_store()
node = _select_first_online_node(store) # 释放所有已加载推理的节点短超时best-effort
if not node: results: list[dict[str, Any]] = []
return ok({"unloaded": False, "error": "no online compute node"}) for n in store.compute_nodes():
try: if not store.is_inference_loaded(n["id"]):
client = ComputeNodeClient(node["api_base_url"]) continue
result = await client._request("POST", "/inference/unload", json_data={}) try:
return ok(result) result = await ComputeNodeClient(n["api_base_url"]).inference_unload()
except Exception as exc: results.append({"node_id": n["id"], "success": True, "result": result})
return ok({"unloaded": False, "error": str(exc)}) except Exception as exc: # noqa: BLE001 - best-effort unload
results.append({"node_id": n["id"], "success": False, "error": str(exc)})
finally:
store.mark_inference_unloaded(n["id"])
return ok({"unloaded": True, "nodes": results})
@router.get("/model-chat/local/status") @router.get("/model-chat/local/status")
@@ -1466,7 +1908,7 @@ async def model_chat_local_status() -> dict[str, Any]:
return ok({"loaded": False, "error": "no online compute node"}) return ok({"loaded": False, "error": "no online compute node"})
try: try:
client = ComputeNodeClient(node["api_base_url"]) client = ComputeNodeClient(node["api_base_url"])
result = await client._request("GET", "/inference/status") result = await client.inference_status()
return ok(result) return ok(result)
except Exception as exc: except Exception as exc:
return ok({"loaded": False, "error": str(exc)}) return ok({"loaded": False, "error": str(exc)})
@@ -1475,13 +1917,19 @@ async def model_chat_local_status() -> dict[str, Any]:
@router.post("/model-chat/trained/preload") @router.post("/model-chat/trained/preload")
async def model_chat_trained_preload(payload: dict[str, Any] = Body(...)) -> dict[str, Any]: async def model_chat_trained_preload(payload: dict[str, Any] = Body(...)) -> dict[str, Any]:
"""Load a trained model (base + adapter) on the compute node for inference.""" """Load a trained model (base + adapter) on the compute node for inference."""
model_path = (payload.get("model_name_or_path") or "").strip()
if not model_path:
return ok({"loaded": False, "error": "model_name_or_path is required"})
store = get_platform_store() store = get_platform_store()
node = _select_first_online_node(store) node = _select_first_online_node(store)
if not node: if not node:
return ok({"loaded": False, "error": "no online compute node"}) return ok({"loaded": False, "error": "no online compute node"})
try: try:
client = ComputeNodeClient(node["api_base_url"]) client = ComputeNodeClient(node["api_base_url"])
result = await client._request("POST", "/inference/load", json_data=payload) # 计算节点现在异步加载HTTP 接受loading/ready即视为派发成功
result = await client.inference_load(payload)
if result.get("loaded") or result.get("status") in {"loading", "ready"}:
store.mark_inference_loaded(node["id"])
return ok(result) return ok(result)
except Exception as exc: except Exception as exc:
return ok({"loaded": False, "error": str(exc)}) return ok({"loaded": False, "error": str(exc)})
@@ -1520,6 +1968,16 @@ async def update_compute_node(node_id: str, payload: dict[str, Any] = Body(...))
raise fail(400, str(exc)) raise fail(400, str(exc))
@router.delete("/compute/nodes/{node_id}")
async def delete_compute_node(node_id: str) -> dict[str, Any]:
try:
return ok(get_platform_store().delete_compute_node(node_id))
except KeyError:
raise fail(404, "compute node not found")
except ValueError as exc:
raise fail(400, str(exc))
@router.post("/compute/nodes/{node_id}/test-connection") @router.post("/compute/nodes/{node_id}/test-connection")
async def test_compute_node(node_id: str) -> dict[str, Any]: async def test_compute_node(node_id: str) -> dict[str, Any]:
store = get_platform_store() store = get_platform_store()

View File

@@ -88,6 +88,84 @@ def parse_size_bytes(value: Any) -> int:
return max(0, round(amount * _SIZE_UNIT_BYTES[unit])) return max(0, round(amount * _SIZE_UNIT_BYTES[unit]))
def count_dataset_records(content: str) -> int:
text = (content or "").strip()
if not text:
return 0
try:
value = json.loads(text)
if isinstance(value, list):
return len(value)
return 1
except (TypeError, ValueError, json.JSONDecodeError):
pass
return len([line for line in text.splitlines() if line.strip()])
_EVAL_METHOD_LABELS = {
"standard": "标准匹配",
"metric_standard": "综合评测",
"semantic": "语义相似度",
"sentiment": "情感分析",
"accuracy": "准确性评估",
"safety": "安全性评估",
"relevance": "相关性评估",
"fluency": "流畅性评估",
"factuality": "事实性评估",
"custom": "自定义评估",
}
def _eval_method_label(value: Any) -> str:
if isinstance(value, list):
return "".join(_eval_method_label(item) for item in value if item)
text = str(value or "").strip()
return _EVAL_METHOD_LABELS.get(text, text)
def _basic_metric_labels(config: dict[str, Any] | None) -> list[str]:
cfg = config or {}
labels: list[str] = []
bleu = cfg.get("bleu") or {}
if bleu.get("enabled"):
labels.append(f"BLEU-{int(bleu.get('ngram') or 4)}")
rouge = cfg.get("rouge") or {}
if rouge.get("enabled"):
methods = rouge.get("methods") or []
method_labels = {
"rouge1": "ROUGE-1",
"rouge2": "ROUGE-2",
"rougeL": "ROUGE-L",
"rouge_1": "ROUGE-1",
"rouge_2": "ROUGE-2",
"rouge_l": "ROUGE-L",
}
labels.extend(method_labels.get(str(item), str(item)) for item in methods)
cosine = cfg.get("cosine") or {}
if cosine.get("enabled"):
labels.append("Cosine")
return labels
def _build_eval_metric_label(payload: dict[str, Any], dimension: dict[str, Any] | None = None) -> str:
parts: list[str] = []
dim = dimension or {}
dim_type = str(dim.get("type") or payload.get("dimension_type") or "").strip()
method_label = _eval_method_label(dim.get("eval_method") or payload.get("eval_method"))
if dim_type in {"classification", "metric"} and method_label:
parts.append(f"LLM{method_label}")
elif dim_type == "text_similarity" and method_label:
parts.append(method_label)
parts.extend(_basic_metric_labels(payload.get("basic_metrics") or {}))
if not parts:
metric = str(payload.get("metric") or payload.get("eval_type") or "").strip()
return "自定义评测" if metric == "custom" else (metric or "-")
return " + ".join(parts)
def version_number(value: Any, default: int = 0) -> int: def version_number(value: Any, default: int = 0) -> int:
try: try:
number = int(value) number = int(value)
@@ -130,8 +208,12 @@ def parse_training_metric_line(line: str) -> dict[str, float] | None:
if "loss" not in line and "learning_rate" not in line: if "loss" not in line and "learning_rate" not in line:
return None return None
result: dict[str, float] = {} result: dict[str, float] = {}
number_pattern = r"([-+]?(?:\d+(?:\.\d*)?|\.\d+)(?:[eE][-+]?\d+)?)"
step_match = re.search(rf"(?:^|[\s,{{])['\"]?step['\"]?\s*(?:=|:)\s*{number_pattern}", line, re.I)
if step_match:
result["step"] = float(step_match.group(1))
for key in ["loss", "grad_norm", "learning_rate", "epoch"]: for key in ["loss", "grad_norm", "learning_rate", "epoch"]:
match = re.search(rf"['\"]?{key}['\"]?\s*:\s*([-+]?\d+(?:\.\d+)?(?:[eE][-+]?\d+)?)", line) match = re.search(rf"['\"]?{key}['\"]?\s*(?:=|:)\s*{number_pattern}", line, re.I)
if match: if match:
result[key] = float(match.group(1)) result[key] = float(match.group(1))
return result or None return result or None
@@ -296,10 +378,11 @@ class PlatformStore:
# request (notably expensive against the remote PostgreSQL instance). # request (notably expensive against the remote PostgreSQL instance).
# TCP keepalive 让操作系统持续保活连接,抵抗远程库空闲静默断连。 # TCP keepalive 让操作系统持续保活连接,抵抗远程库空闲静默断连。
pool_kwargs = { pool_kwargs = {
"connect_timeout": 5,
"keepalives": 1, "keepalives": 1,
"keepalives_idle": 30, "keepalives_idle": 10,
"keepalives_interval": 10, "keepalives_interval": 5,
"keepalives_count": 5, "keepalives_count": 3,
} }
self._pool = ConnectionPool( self._pool = ConnectionPool(
conninfo=self.database_url, conninfo=self.database_url,
@@ -320,6 +403,20 @@ class PlatformStore:
self._pool.open() self._pool.open()
self.ensure_schema() self.ensure_schema()
self.ensure_seed_data() self.ensure_seed_data()
# Track which compute nodes have an active inference model loaded
self._inference_nodes: set[str] = set()
self._last_runtime_refresh = 0.0
# ── inference node tracking ────────────────────────────────────
def mark_inference_loaded(self, node_id: str) -> None:
self._inference_nodes.add(node_id)
def mark_inference_unloaded(self, node_id: str) -> None:
self._inference_nodes.discard(node_id)
def is_inference_loaded(self, node_id: str) -> bool:
return node_id in self._inference_nodes
@contextmanager @contextmanager
def connect(self) -> Iterator["PgConnection"]: def connect(self) -> Iterator["PgConnection"]:
@@ -359,6 +456,15 @@ class PlatformStore:
) )
self._ensure_columns(conn, "gpus", {"last_seen_at": "TEXT"}) self._ensure_columns(conn, "gpus", {"last_seen_at": "TEXT"})
self._ensure_columns(conn, "fine_tune_tasks", {"compute_job_id": "TEXT"}) self._ensure_columns(conn, "fine_tune_tasks", {"compute_job_id": "TEXT"})
self._ensure_columns(
conn,
"trained_models",
{
"artifact_dir": "TEXT",
"compute_node_id": "TEXT",
"compute_node_name": "TEXT",
},
)
self._ensure_columns( self._ensure_columns(
conn, conn,
"resource_replicas", "resource_replicas",
@@ -433,6 +539,10 @@ class PlatformStore:
def refresh_runtime_state(self) -> None: def refresh_runtime_state(self) -> None:
if get_settings().compute_mode != "simulator": if get_settings().compute_mode != "simulator":
return return
now_ts = time.monotonic()
if now_ts - self._last_runtime_refresh < 2:
return
self._last_runtime_refresh = now_ts
with self.connect() as conn: with self.connect() as conn:
rows = conn.execute( rows = conn.execute(
@@ -492,6 +602,15 @@ class PlatformStore:
name = task.get("output_model_name") or f"{task['name']}-lora" name = task.get("output_model_name") or f"{task['name']}-lora"
exists = conn.execute("SELECT id FROM trained_models WHERE name=?", (name,)).fetchone() exists = conn.execute("SELECT id FROM trained_models WHERE name=?", (name,)).fetchone()
if exists: if exists:
conn.execute(
"""
UPDATE trained_models
SET compute_node_id=COALESCE(compute_node_id, ?),
compute_node_name=COALESCE(compute_node_name, ?)
WHERE id=?
""",
(task.get("compute_node_id"), task.get("compute_node_code") or task.get("compute_node_name"), exists["id"]),
)
return return
model = conn.execute("SELECT path FROM models WHERE id=?", (task.get("base_model"),)).fetchone() model = conn.execute("SELECT path FROM models WHERE id=?", (task.get("base_model"),)).fetchone()
output_dir = task.get("output_dir") or f"/data/yg-ft/outputs/{task['name']}" output_dir = task.get("output_dir") or f"/data/yg-ft/outputs/{task['name']}"
@@ -499,8 +618,8 @@ class PlatformStore:
conn.execute( conn.execute(
""" """
INSERT INTO trained_models INSERT INTO trained_models
(id, name, train_methods, base_model_path, create_time, merged, merging, merged_path, artifact_dir) (id, name, train_methods, base_model_path, create_time, merged, merging, merged_path, artifact_dir, compute_node_id, compute_node_name)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""", """,
( (
trained_model_id, trained_model_id,
@@ -512,6 +631,8 @@ class PlatformStore:
0, 0,
output_dir, output_dir,
output_dir, output_dir,
task.get("compute_node_id"),
task.get("compute_node_code") or task.get("compute_node_name"),
), ),
) )
# Use real artifact data from compute node when available # Use real artifact data from compute node when available
@@ -778,7 +899,7 @@ class PlatformStore:
( (
new_id("metric"), new_id("metric"),
task_id, task_id,
line_number, int(metric.get("step") or line_number),
metric.get("epoch"), metric.get("epoch"),
metric.get("loss"), metric.get("loss"),
metric.get("grad_norm"), metric.get("grad_norm"),
@@ -1245,15 +1366,25 @@ class PlatformStore:
self.refresh_runtime_state() self.refresh_runtime_state()
with self.connect() as conn: with self.connect() as conn:
rows = conn.execute("SELECT * FROM trained_models ORDER BY create_time DESC").fetchall() rows = conn.execute("SELECT * FROM trained_models ORDER BY create_time DESC").fetchall()
return [ items = []
{ for row in rows:
item = {
**dict(row), **dict(row),
"train_methods": json_loads(row["train_methods"], []), "train_methods": json_loads(row["train_methods"], []),
"merged": bool(row["merged"]), "merged": bool(row["merged"]),
"merging": bool(row["merging"]), "merging": bool(row["merging"]),
} }
for row in rows if not item.get("compute_node_id"):
] task_rows = conn.execute("SELECT payload FROM fine_tune_tasks ORDER BY create_time DESC").fetchall()
for task in task_rows:
task_payload = json_loads(task["payload"], {})
output_name = task_payload.get("output_model_name") or f"{task_payload.get('name')}-lora"
if output_name == item["name"]:
item["compute_node_id"] = task_payload.get("compute_node_id")
item["compute_node_name"] = task_payload.get("compute_node_code") or task_payload.get("compute_node_name")
break
items.append(item)
return items
def delete_trained_model(self, model_id: str) -> None: def delete_trained_model(self, model_id: str) -> None:
with self.connect() as conn: with self.connect() as conn:
@@ -1298,6 +1429,7 @@ class PlatformStore:
file_size_bytes = int(file_row.get("size_bytes") or 0) file_size_bytes = int(file_row.get("size_bytes") or 0)
if file_size_bytes <= 0: if file_size_bytes <= 0:
file_size_bytes = parse_size_bytes(file_row.get("size")) file_size_bytes = parse_size_bytes(file_row.get("size"))
file_record_count = int(file_row.get("record_count") or 0)
decoded_files.append( decoded_files.append(
{ {
"id": file_row["id"], "id": file_row["id"],
@@ -1306,7 +1438,7 @@ class PlatformStore:
"size_bytes": file_size_bytes, "size_bytes": file_size_bytes,
**dataset_file_version_summary(file_row), **dataset_file_version_summary(file_row),
"create_time": file_row["create_time"], "create_time": file_row["create_time"],
"record_count": int(file_row.get("record_count") or 0), "record_count": file_record_count,
"split": metadata.get("file_split"), "split": metadata.get("file_split"),
} }
) )
@@ -1324,6 +1456,9 @@ class PlatformStore:
total_size_bytes = int(row.get("size_bytes") or 0) total_size_bytes = int(row.get("size_bytes") or 0)
if total_size_bytes <= 0: if total_size_bytes <= 0:
total_size_bytes = parse_size_bytes(row.get("size")) total_size_bytes = parse_size_bytes(row.get("size"))
total_record_count = sum(int(item.get("record_count") or 0) for item in decoded_files)
if not decoded_files:
total_record_count = int(row.get("record_count") or row.get("count") or 0)
current_version_nos = sorted( current_version_nos = sorted(
{ {
int(item["current_version_no"]) int(item["current_version_no"])
@@ -1333,6 +1468,8 @@ class PlatformStore:
) )
return { return {
**dict(row), **dict(row),
"count": total_record_count,
"record_count": total_record_count,
"size_bytes": total_size_bytes, "size_bytes": total_size_bytes,
"current_version_no": ( "current_version_no": (
current_version_nos[0] if len(current_version_nos) == 1 else None current_version_nos[0] if len(current_version_nos) == 1 else None
@@ -1407,7 +1544,7 @@ class PlatformStore:
version_id = f"{file_id}_v1" version_id = f"{file_id}_v1"
size_bytes = len(content.encode("utf-8")) size_bytes = len(content.encode("utf-8"))
size = f"{size_bytes} B" size = f"{size_bytes} B"
record_count = len([line for line in content.splitlines() if line.strip()]) record_count = count_dataset_records(content)
version = { version = {
"id": version_id, "id": version_id,
"version": 1, "version": 1,
@@ -1440,10 +1577,18 @@ class PlatformStore:
) )
conn.execute( conn.execute(
"""UPDATE datasets """UPDATE datasets
SET count=count+?, record_count=record_count+?, SET count=stats.record_count,
size_bytes=size_bytes+?, size=((size_bytes+?)::text || ' B') record_count=stats.record_count,
size_bytes=stats.size_bytes,
size=(stats.size_bytes::text || ' B')
FROM (
SELECT COALESCE(SUM(record_count), 0) AS record_count,
COALESCE(SUM(size_bytes), 0) AS size_bytes
FROM dataset_files
WHERE dataset_id=?
) stats
WHERE id=?""", WHERE id=?""",
(record_count, record_count, size_bytes, size_bytes, dataset_id), (dataset_id, dataset_id),
) )
return { return {
"id": file_id, "id": file_id,
@@ -1548,19 +1693,57 @@ class PlatformStore:
row = conn.execute("SELECT * FROM dataset_files WHERE id=?", (file_id,)).fetchone() row = conn.execute("SELECT * FROM dataset_files WHERE id=?", (file_id,)).fetchone()
if not row: if not row:
raise KeyError(file_id) raise KeyError(file_id)
content = payload.get("content", "")
size_bytes = len(content.encode("utf-8"))
record_count = count_dataset_records(content)
versions = json_loads(row["versions"], []) versions = json_loads(row["versions"], [])
version = { version = {
"id": f"{file_id}_v{len(versions) + 1}", "id": f"{file_id}_v{len(versions) + 1}",
"version": len(versions) + 1, "version": len(versions) + 1,
"version_no": len(versions) + 1,
"create_time": utcnow(), "create_time": utcnow(),
"description": payload.get("description", "online edit"), "description": payload.get("description", "online edit"),
"size_bytes": size_bytes,
"record_count": record_count,
} }
versions.append(version) versions.append(version)
conn.execute( conn.execute(
"UPDATE dataset_files SET content=?, active_version_id=?, versions=? WHERE id=?", """
(payload.get("content", ""), version["id"], json_dumps(versions), file_id), UPDATE dataset_files
SET content=?, active_version_id=?, current_version_id=?, versions=?,
size_bytes=?, size=?, record_count=?, version_no=?
WHERE id=?
""",
(
content,
version["id"],
version["id"],
json_dumps(versions),
size_bytes,
f"{size_bytes} B",
record_count,
version["version_no"],
file_id,
),
) )
return {"version": version, "content": payload.get("content", "")} conn.execute(
"""UPDATE datasets
SET count=stats.record_count,
record_count=stats.record_count,
size_bytes=stats.size_bytes,
size=(stats.size_bytes::text || ' B')
FROM (
SELECT dataset_id,
COALESCE(SUM(record_count), 0) AS record_count,
COALESCE(SUM(size_bytes), 0) AS size_bytes
FROM dataset_files
WHERE dataset_id=(SELECT dataset_id FROM dataset_files WHERE id=?)
GROUP BY dataset_id
) stats
WHERE datasets.id=stats.dataset_id""",
(file_id,),
)
return {"version": version, "content": content}
def activate_file_version(self, file_id: str, version_id: str) -> dict[str, Any]: def activate_file_version(self, file_id: str, version_id: str) -> dict[str, Any]:
with self.connect() as conn: with self.connect() as conn:
@@ -2064,19 +2247,54 @@ class PlatformStore:
def _json_payload_row(self, row: PgRow) -> dict[str, Any]: def _json_payload_row(self, row: PgRow) -> dict[str, Any]:
payload = json_loads(row["payload"], {}) payload = json_loads(row["payload"], {})
payload.update({"id": row["id"], "status": row.get("status"), "create_time": row["create_time"]}) payload.update({"id": row["id"], "status": row.get("status"), "create_time": row["create_time"]})
if not payload.get("metric_label"):
payload["metric_label"] = _build_eval_metric_label(payload)
if not payload.get("metric") or payload.get("metric") == "custom":
payload["metric"] = payload["metric_label"]
return payload return payload
def _enrich_eval_payload(self, conn: PgConnection, payload: dict[str, Any]) -> dict[str, Any]:
data = dict(payload)
model_id = str(data.get("model_id") or "")
if model_id and not data.get("model_name"):
model = conn.execute("SELECT name FROM models WHERE id=?", (model_id,)).fetchone()
if not model:
model = conn.execute("SELECT name FROM trained_models WHERE id=? OR name=?", (model_id, model_id)).fetchone()
if model:
data["model_name"] = model["name"]
dataset_id = str(data.get("dataset_id") or "")
if dataset_id and not data.get("dataset"):
dataset = conn.execute("SELECT name FROM datasets WHERE id=?", (dataset_id,)).fetchone()
if dataset:
data["dataset"] = dataset["name"]
dimension = None
dimension_id = str(data.get("dimension_id") or "")
if dimension_id:
dimension_row = conn.execute("SELECT payload FROM eval_dimensions WHERE id=?", (dimension_id,)).fetchone()
if dimension_row:
dimension = json_loads(dimension_row["payload"], {})
data.setdefault("dimension_type", dimension.get("type"))
data.setdefault("eval_method", dimension.get("eval_method"))
data.setdefault("evaluator_model", dimension.get("eval_model"))
data["metric_label"] = _build_eval_metric_label(data, dimension)
if not data.get("metric") or data.get("metric") in {"custom", "自定义评测"}:
data["metric"] = data["metric_label"]
return data
def eval_tasks(self) -> list[dict[str, Any]]: def eval_tasks(self) -> list[dict[str, Any]]:
with self.connect() as conn: with self.connect() as conn:
rows = conn.execute("SELECT * FROM eval_tasks ORDER BY create_time DESC").fetchall() rows = conn.execute("SELECT * FROM eval_tasks ORDER BY create_time DESC").fetchall()
return [self._json_payload_row(row) for row in rows] return [self._enrich_eval_payload(conn, self._json_payload_row(row)) for row in rows]
def eval_task(self, task_id: str) -> dict[str, Any]: def eval_task(self, task_id: str) -> dict[str, Any]:
with self.connect() as conn: with self.connect() as conn:
row = conn.execute("SELECT * FROM eval_tasks WHERE id=?", (task_id,)).fetchone() row = conn.execute("SELECT * FROM eval_tasks WHERE id=?", (task_id,)).fetchone()
if not row: if not row:
raise KeyError(task_id) raise KeyError(task_id)
payload = self._json_payload_row(row) payload = self._enrich_eval_payload(conn, self._json_payload_row(row))
payload.setdefault("sample_count", 0) payload.setdefault("sample_count", 0)
payload.setdefault("completed_count", 0) payload.setdefault("completed_count", 0)
payload.setdefault("passed_count", 0) payload.setdefault("passed_count", 0)
@@ -2102,22 +2320,92 @@ class PlatformStore:
"metric": payload.get("metric") or "custom", "metric": payload.get("metric") or "custom",
} }
with self.connect() as conn: with self.connect() as conn:
model = conn.execute("SELECT name FROM models WHERE id=?", (str(payload.get("model_id")),)).fetchone() model_id = str(payload.get("model_id") or "")
model = conn.execute("SELECT name FROM models WHERE id=?", (model_id,)).fetchone()
trained_model = conn.execute("SELECT name FROM trained_models WHERE id=? OR name=?", (model_id, model_id)).fetchone()
dataset = conn.execute("SELECT name FROM datasets WHERE id=?", (str(payload.get("dataset_id")),)).fetchone() dataset = conn.execute("SELECT name FROM datasets WHERE id=?", (str(payload.get("dataset_id")),)).fetchone()
dimension = None
dimension_id = str(payload.get("dimension_id") or "")
if dimension_id:
dimension_row = conn.execute("SELECT payload FROM eval_dimensions WHERE id=?", (dimension_id,)).fetchone()
if dimension_row:
dimension = json_loads(dimension_row["payload"], {})
if model: if model:
data.setdefault("model_name", model["name"]) data.setdefault("model_name", model["name"])
elif trained_model:
data.setdefault("model_name", trained_model["name"])
if dataset: if dataset:
data.setdefault("dataset", dataset["name"]) data.setdefault("dataset", dataset["name"])
if dimension:
data.setdefault("dimension_type", dimension.get("type"))
data.setdefault("eval_method", dimension.get("eval_method"))
data.setdefault("evaluator_model", dimension.get("eval_model"))
data["metric_label"] = _build_eval_metric_label(data, dimension)
if data.get("metric") == "custom":
data["metric"] = data["metric_label"]
conn.execute( conn.execute(
"INSERT INTO eval_tasks (id, name, payload, status, create_time) VALUES (?, ?, ?, ?, ?)", "INSERT INTO eval_tasks (id, name, payload, status, create_time) VALUES (?, ?, ?, ?, ?)",
(task_id, name, json_dumps(data), status, now), (task_id, name, json_dumps(data), status, now),
) )
return self.eval_task(task_id) return self.eval_task(task_id)
def update_eval_task(self, task_id: str, updates: dict[str, Any]) -> dict[str, Any]:
"""Update fields in an eval task's payload without replacing the whole record."""
task = self.eval_task(task_id)
merged = {**task, **updates}
with self.connect() as conn:
conn.execute(
"UPDATE eval_tasks SET payload=?, status=? WHERE id=?",
(json_dumps(merged), merged.get("status", task.get("status", "pending")), task_id),
)
return self.eval_task(task_id)
def delete_eval_task(self, task_id: str) -> None: def delete_eval_task(self, task_id: str) -> None:
with self.connect() as conn: with self.connect() as conn:
conn.execute("DELETE FROM eval_tasks WHERE id=?", (task_id,)) conn.execute("DELETE FROM eval_tasks WHERE id=?", (task_id,))
def running_eval_tasks(self) -> list[dict[str, Any]]:
"""Return eval tasks that have been submitted to a compute node and are still running."""
return [
task for task in self.eval_tasks()
if task.get("compute_job_id") and task.get("status") in {"queued", "running"}
]
def apply_eval_job_result(self, task_id: str, job: dict[str, Any], result_content: dict[str, Any] | None = None) -> dict[str, Any]:
"""Sync a compute job status/result back to an eval task."""
task = self.eval_task(task_id)
job_status = str(job.get("status", ""))
status_map = {"queued": "running", "running": "running", "completed": "completed",
"failed": "failed", "stopped": "stopped"}
new_status = status_map.get(job_status, job_status or task.get("status", "pending"))
updates: dict[str, Any] = {
"status": new_status,
"progress": int(job.get("progress", 0)),
"output_dir": job.get("output_dir", task.get("output_dir", "")),
}
# On completion, populate results from eval_results.json content
if new_status == "completed" and result_content:
updates.update({
"overall_score": result_content.get("overall_score", 0),
"overall_score_max": result_content.get("overall_score_max", 100),
"overall_evaluation": result_content.get("overall_evaluation", ""),
"improvement_suggestions": result_content.get("improvement_suggestions", []),
"dimension_summary": result_content.get("dimension_summary", []),
"samples": result_content.get("samples", []),
"sample_count": result_content.get("sample_count", 0),
"completed_count": result_content.get("completed_count", 0),
"passed_count": result_content.get("passed_count", 0),
"basic_metrics": result_content.get("basic_metrics", {}),
"score": result_content.get("overall_score", 0),
"completed_time": utcnow(),
})
elif new_status in {"failed", "stopped"}:
updates.update({
"error": job.get("error") or task.get("error") or "",
"completed_time": utcnow(),
})
return self.update_eval_task(task_id, updates)
def dimensions(self) -> list[dict[str, Any]]: def dimensions(self) -> list[dict[str, Any]]:
with self.connect() as conn: with self.connect() as conn:
rows = conn.execute("SELECT * FROM eval_dimensions ORDER BY create_time DESC").fetchall() rows = conn.execute("SELECT * FROM eval_dimensions ORDER BY create_time DESC").fetchall()
@@ -2284,6 +2572,15 @@ class PlatformStore:
).fetchall() ).fetchall()
return {int(row["gpu_index"]) for row in rows} return {int(row["gpu_index"]) for row in rows}
def _node_gpu_indexes(self, conn: PgConnection, node: dict[str, Any]) -> set[int]:
rows = conn.execute("SELECT gpu_index FROM gpus WHERE node_id=?", (node["id"],)).fetchall()
if rows:
return {int(row["gpu_index"]) for row in rows}
return set(range(max(0, int(node.get("gpu_count") or 0))))
def _node_capacity(self, node: dict[str, Any]) -> int:
return max(1, int(node.get("max_parallel_jobs") or 1), int(node.get("gpu_count") or 0))
def _schedule_node_locked(self, conn: PgConnection, payload: dict[str, Any]) -> dict[str, Any]: def _schedule_node_locked(self, conn: PgConnection, payload: dict[str, Any]) -> dict[str, Any]:
requested = payload.get("requested_node_id") or payload.get("compute_node_id") requested = payload.get("requested_node_id") or payload.get("compute_node_id")
requested_gpus = [int(item) for item in payload.get("gpus") or []] requested_gpus = [int(item) for item in payload.get("gpus") or []]
@@ -2291,13 +2588,15 @@ class PlatformStore:
candidates = [ candidates = [
n n
for n in nodes for n in nodes
if n["enabled"] and n["scheduler_status"] == "online" and n["current_running_jobs"] < n["max_parallel_jobs"] if n["enabled"] and n["scheduler_status"] == "online" and n["current_running_jobs"] < self._node_capacity(n)
] ]
if requested_gpus: if requested_gpus:
requested_gpu_set = set(requested_gpus)
candidates = [ candidates = [
node node
for node in candidates for node in candidates
if not set(requested_gpus).intersection(self._active_gpu_indexes(conn, node["id"])) if requested_gpu_set.issubset(self._node_gpu_indexes(conn, node))
and not requested_gpu_set.intersection(self._active_gpu_indexes(conn, node["id"]))
] ]
if requested: if requested:
selected = next((n for n in candidates if n["id"] == requested), None) selected = next((n for n in candidates if n["id"] == requested), None)
@@ -2312,8 +2611,8 @@ class PlatformStore:
reason = "disabled" reason = "disabled"
elif node["scheduler_status"] != "online": elif node["scheduler_status"] != "online":
reason = f"status={node['scheduler_status']}" reason = f"status={node['scheduler_status']}"
elif node["current_running_jobs"] >= node["max_parallel_jobs"]: elif node["current_running_jobs"] >= self._node_capacity(node):
reason = f"capacity full {node['current_running_jobs']}/{node['max_parallel_jobs']}" reason = f"capacity full {node['current_running_jobs']}/{self._node_capacity(node)}"
else: else:
reason = "not selected" reason = "not selected"
reasons.append(f"{node['code']}({reason})") reasons.append(f"{node['code']}({reason})")
@@ -2538,6 +2837,28 @@ class PlatformStore:
"SELECT compute_node_id, COUNT(*) AS cnt FROM fine_tune_tasks WHERE status IN ('syncing','queued','running') GROUP BY compute_node_id" "SELECT compute_node_id, COUNT(*) AS cnt FROM fine_tune_tasks WHERE status IN ('syncing','queued','running') GROUP BY compute_node_id"
).fetchall() ).fetchall()
running_map = {r["compute_node_id"]: r["cnt"] for r in running} running_map = {r["compute_node_id"]: r["cnt"] for r in running}
# 评测任务同样占用算力节点,纳入运行任务统计
for row in conn.execute(
"SELECT payload FROM eval_tasks WHERE status IN ('syncing','queued','running')"
).fetchall():
node_id = json_loads(row["payload"], {}).get("compute_node_id")
if node_id:
running_map[node_id] = running_map.get(node_id, 0) + 1
# 推理模型占用算力节点同样计入:优先从 compare_tasks 持久化状态派生
# (重启后仍准确),并用内存标记兜底(直接 preload 的模型无 compare 记录)
inference_node_ids = set(self._inference_nodes)
for ctr in conn.execute("SELECT payload FROM compare_tasks").fetchall():
ls = json_loads(ctr["payload"], {}).get("load_status") or {}
if isinstance(ls, str):
try:
ls = json.loads(ls)
except (json.JSONDecodeError, TypeError):
ls = {}
for m in ls.get("loaded_models") or []:
if m.get("status") in {"ready", "running"} and m.get("node_id"):
inference_node_ids.add(m["node_id"])
for nid in inference_node_ids:
running_map[nid] = running_map.get(nid, 0) + 1
rows = conn.execute("SELECT * FROM compute_nodes ORDER BY scheduler_weight DESC, code").fetchall() rows = conn.execute("SELECT * FROM compute_nodes ORDER BY scheduler_weight DESC, code").fetchall()
return [ return [
{ {
@@ -2664,6 +2985,24 @@ class PlatformStore:
) )
return next(node for node in self.compute_nodes() if node["id"] == node_id) return next(node for node in self.compute_nodes() if node["id"] == node_id)
def delete_compute_node(self, node_id: str) -> dict[str, Any]:
with self.connect() as conn:
node = conn.execute("SELECT * FROM compute_nodes WHERE id=?", (node_id,)).fetchone()
if not node:
raise KeyError(node_id)
active = conn.execute(
"""
SELECT COUNT(*) AS cnt
FROM fine_tune_tasks
WHERE compute_node_id=? AND status IN ('syncing','queued','running')
""",
(node_id,),
).fetchone()
if active and int(active["cnt"] or 0) > 0:
raise ValueError("compute node has active training tasks")
conn.execute("DELETE FROM compute_nodes WHERE id=?", (node_id,))
return {"deleted": node_id}
def update_compute_node_health(self, node_id: str, health: dict[str, Any], success: bool, error: str | None = None) -> dict[str, Any]: def update_compute_node_health(self, node_id: str, health: dict[str, Any], success: bool, error: str | None = None) -> dict[str, Any]:
current = next((n for n in self.compute_nodes() if n["id"] == node_id), None) current = next((n for n in self.compute_nodes() if n["id"] == node_id), None)
if not current: if not current:
@@ -2737,6 +3076,26 @@ class PlatformStore:
"SELECT * FROM fine_tune_tasks WHERE status IN ('syncing','queued','running')" "SELECT * FROM fine_tune_tasks WHERE status IN ('syncing','queued','running')"
).fetchall() ).fetchall()
] ]
# 评测任务同样占用节点 GPU
eval_running = [
json_loads(row["payload"], {})
for row in conn.execute(
"SELECT payload FROM eval_tasks WHERE status IN ('syncing','queued','running')"
).fetchall()
]
# 推理模型占用的节点:优先从 compare_tasks 持久化状态派生(重启后仍准确),
# 内存标记兜底(直接 preload 的模型无 compare 记录)
inference_node_ids = set(self._inference_nodes)
for ctr in conn.execute("SELECT payload FROM compare_tasks").fetchall():
ls = json_loads(ctr["payload"], {}).get("load_status") or {}
if isinstance(ls, str):
try:
ls = json.loads(ls)
except (json.JSONDecodeError, TypeError):
ls = {}
for m in ls.get("loaded_models") or []:
if m.get("status") in {"ready", "running"} and m.get("node_id"):
inference_node_ids.add(m["node_id"])
items = [] items = []
for row in rows: for row in rows:
task = next( task = next(
@@ -2747,8 +3106,23 @@ class PlatformStore:
), ),
None, None,
) )
busy = task is not None and task.get("status") == "running" eval_task = next(
reserved = task is not None and task.get("status") in {"syncing", "queued"} (
t
for t in eval_running
if t.get("compute_node_id") == row["node_id"]
and row["gpu_index"] == (int(t["gpu_id"]) if t.get("gpu_id") is not None else -1)
),
None,
)
busy = (task is not None and task.get("status") == "running") or eval_task is not None
reserved = (task is not None and task.get("status") in {"syncing", "queued"}) or (
eval_task is not None and eval_task.get("status") in {"syncing", "queued"}
)
# Also mark GPU as busy if an inference model is loaded on this node
if row["node_id"] in inference_node_ids and not busy:
busy = True
reserved = False
memory_used = round(row["memory_total_gb"] * (0.72 if busy else 0.18 if reserved else 0.04), 1) memory_used = round(row["memory_total_gb"] * (0.72 if busy else 0.18 if reserved else 0.04), 1)
gpu_percent = 86 if busy else 22 if reserved else 3 gpu_percent = 86 if busy else 22 if reserved else 3
memory_total = float(row["memory_total_gb"] or 0) memory_total = float(row["memory_total_gb"] or 0)
@@ -2778,6 +3152,16 @@ class PlatformStore:
} }
] ]
if task if task
else [
{
"pid": int(eval_task.get("process_id") or 0),
"name": "eval_runner",
"memory_used_gb": memory_used,
"task_name": eval_task.get("eval_task_name") or eval_task.get("name") or "评测任务",
"user": "admin",
}
]
if eval_task
else [], else [],
} }
) )
@@ -2827,11 +3211,24 @@ class PlatformStore:
} }
def health_metrics(self) -> dict[str, float]: def health_metrics(self) -> dict[str, float]:
info = self.system_info() # 轻量健康检查:采集真实 CPU/内存/磁盘使用率
# 用于顶部栏快速展示与 Docker 健康检查。
try:
import psutil
# cpu_percent(interval=None) 首次调用返回 0需要短暂采样
cpu_percent = float(psutil.cpu_percent(interval=0.1))
memory_percent = float(psutil.virtual_memory().percent)
# Windows 兼容:尝试当前盘符
try:
disk_percent = float(psutil.disk_usage('/').percent)
except Exception:
disk_percent = float(psutil.disk_usage('C:\\').percent)
except Exception:
cpu_percent = memory_percent = disk_percent = 0.0
return { return {
"cpu_percent": info["cpu"]["percent"], "cpu_percent": round(cpu_percent, 1),
"memory_percent": info["memory"]["percent"], "memory_percent": round(memory_percent, 1),
"disk_percent": info["disk"]["percent"], "disk_percent": round(disk_percent, 1),
} }
def queue(self) -> list[dict[str, Any]]: def queue(self) -> list[dict[str, Any]]:

View File

@@ -33,7 +33,10 @@ CREATE TABLE IF NOT EXISTS trained_models (
create_time TEXT NOT NULL, create_time TEXT NOT NULL,
merged INTEGER NOT NULL DEFAULT 0, merged INTEGER NOT NULL DEFAULT 0,
merging INTEGER NOT NULL DEFAULT 0, merging INTEGER NOT NULL DEFAULT 0,
merged_path TEXT merged_path TEXT,
artifact_dir TEXT,
compute_node_id TEXT,
compute_node_name TEXT
); );
CREATE TABLE IF NOT EXISTS model_lineage ( CREATE TABLE IF NOT EXISTS model_lineage (

View File

@@ -33,6 +33,16 @@ def _unwrap_dict(payload: Any) -> dict[str, Any]:
return payload if isinstance(payload, dict) else {} return payload if isinstance(payload, dict) else {}
# Inference calls are intentionally short-timeout:
# - load dispatch only confirms the compute node accepted the request
# (the actual model load now runs asynchronously on the node).
# - status/unload must never block the platform for long when a node is
# unreachable but still marked online.
INFERENCE_LOAD_TIMEOUT = httpx.Timeout(30, connect=10)
INFERENCE_STATUS_TIMEOUT = httpx.Timeout(30, connect=5)
INFERENCE_UNLOAD_TIMEOUT = httpx.Timeout(30, connect=5)
class ComputeNodeClient: class ComputeNodeClient:
"""Application-side client for one compute node. """Application-side client for one compute node.
@@ -182,10 +192,16 @@ class ComputeNodeClient:
response.raise_for_status() response.raise_for_status()
return _unwrap_dict(response.json()) return _unwrap_dict(response.json())
async def _request(self, method: str, path: str, json_data: dict[str, Any] | None = None) -> dict[str, Any]: async def _request(
self,
method: str,
path: str,
json_data: dict[str, Any] | None = None,
timeout: float | None = None,
) -> dict[str, Any]:
"""Generic request method for compute API endpoints.""" """Generic request method for compute API endpoints."""
url = _join_url(self.api_base_url, f"{self.route_prefix}{path}") url = _join_url(self.api_base_url, f"{self.route_prefix}{path}")
async with httpx.AsyncClient(timeout=300, headers=self.headers()) as client: async with httpx.AsyncClient(timeout=timeout or 300, headers=self.headers()) as client:
if method.upper() == "GET": if method.upper() == "GET":
response = await client.get(url) response = await client.get(url)
else: else:
@@ -193,6 +209,19 @@ class ComputeNodeClient:
response.raise_for_status() response.raise_for_status()
return _unwrap_dict(response.json()) return _unwrap_dict(response.json())
# ── Inference helpers (short timeouts — see module constants) ──────────
async def inference_load(self, payload: dict[str, Any]) -> dict[str, Any]:
"""Dispatch a model load. Returns as soon as the node accepts the
request; the node now loads asynchronously (status goes 'loading')."""
return await self._request("POST", "/inference/load", json_data=payload, timeout=INFERENCE_LOAD_TIMEOUT)
async def inference_status(self) -> dict[str, Any]:
return await self._request("GET", "/inference/status", timeout=INFERENCE_STATUS_TIMEOUT)
async def inference_unload(self) -> dict[str, Any]:
return await self._request("POST", "/inference/unload", json_data={}, timeout=INFERENCE_UNLOAD_TIMEOUT)
async def upload_file( async def upload_file(
self, self,
filename: str, filename: str,
@@ -207,7 +236,8 @@ class ComputeNodeClient:
"resource_id": resource_id or "", "resource_id": resource_id or "",
} }
files = {"file": (filename, content)} files = {"file": (filename, content)}
async with httpx.AsyncClient(timeout=max(self.timeout, 60), headers=self.headers()) as client: timeout = httpx.Timeout(max(self.timeout, 60), connect=self.timeout)
async with httpx.AsyncClient(timeout=timeout, headers=self.headers()) as client:
response = await client.post( response = await client.post(
_join_url(self.api_base_url, f"{self.route_prefix}/compute/files/upload"), _join_url(self.api_base_url, f"{self.route_prefix}/compute/files/upload"),
data=data, data=data,

View File

@@ -1,15 +1,119 @@
from __future__ import annotations from __future__ import annotations
import json
import time
from typing import Any from typing import Any
from app.db.platform_store import get_platform_store from app.db.platform_store import get_platform_store
from app.modules.compute_gateway.client import ComputeNodeClient from app.modules.compute_gateway.client import ComputeNodeClient
# starting 状态允许的最大轮询次数(约 40 * 3s ≈ 2 分钟),超过即判定节点不可达
MAX_STARTING_ATTEMPTS = 40
def _node_for_task(task: dict[str, Any]) -> dict[str, Any] | None: def _node_for_task(task: dict[str, Any]) -> dict[str, Any] | None:
return next((node for node in get_platform_store().compute_nodes() if node["id"] == task.get("compute_node_id")), None) return next((node for node in get_platform_store().compute_nodes() if node["id"] == task.get("compute_node_id")), None)
def _parse_inference_load_status(task: dict[str, Any]) -> tuple[list[dict[str, Any]], dict[str, Any]]:
load_status = task.get("load_status") or {}
if isinstance(load_status, str):
try:
load_status = json.loads(load_status)
except (json.JSONDecodeError, TypeError):
load_status = {}
return load_status.get("loaded_models") or [], load_status
async def reconcile_inference_loads(store: Any) -> list[dict[str, Any]]:
"""推进处于 starting 状态的推理加载。
模型加载已改为异步派发:/model-compare/{id}/load 立即返回,这里在每次
轮询时查询对应计算节点的 /inference/status把任务从 starting 推进到
ready/error。使用短超时单节点不可达不会阻塞整轮轮询。
"""
reconciled: list[dict[str, Any]] = []
now = time.time()
for task in store.compare_tasks():
items, _ = _parse_inference_load_status(task)
if not any(item.get("status") == "starting" for item in items):
continue
# dirty 只要处理过任一 starting 项就置位load_attempts / last_polled_at
# 必须落库,否则节点不可达时计数不会累积,封顶逻辑永远触发不了
dirty = False
for item in items:
if item.get("status") != "starting":
continue
# 节流:同一 item 每 3s 只查询一次
if now - float(item.get("last_polled_at") or 0) < 3:
continue
item["last_polled_at"] = now
item["load_attempts"] = int(item.get("load_attempts") or 0) + 1
dirty = True
node = next((n for n in store.compute_nodes() if n["id"] == item.get("node_id")), None)
if not node:
item["status"] = "error"
item["error"] = "compute node deleted"
store.mark_inference_unloaded(item.get("node_id") or "")
continue
if not node.get("enabled") or node.get("scheduler_status") != "online":
item["status"] = "error"
item["error"] = "compute node offline"
store.mark_inference_unloaded(node["id"])
continue
try:
status = await ComputeNodeClient(node["api_base_url"]).inference_status()
except Exception as exc: # noqa: BLE001 - node unreachable; keep retrying until cap
if int(item.get("load_attempts") or 0) >= MAX_STARTING_ATTEMPTS:
item["status"] = "error"
item["error"] = f"compute node unreachable: {exc}"
store.mark_inference_unloaded(node["id"])
continue
node_status = status.get("status")
if node_status == "ready":
item["status"] = "ready"
item.pop("error", None)
store.mark_inference_loaded(node["id"])
elif node_status == "error":
item["status"] = "error"
item["error"] = status.get("error") or "model load failed on compute node"
store.mark_inference_unloaded(node["id"])
elif node_status == "idle":
# 节点重启导致已加载模型丢失
item["status"] = "error"
item["error"] = "model disappeared from compute node (node may have restarted)"
store.mark_inference_unloaded(node["id"])
# node_status == "loading" -> 保持 starting下轮再查
if dirty:
if any(i.get("status") in {"ready", "running"} for i in items):
new_status = "loaded"
elif any(i.get("status") == "starting" for i in items):
new_status = "starting" # 仍在加载中,保持 starting
else:
new_status = "failed"
store.update_compare_task(task["id"], {"status": new_status, "load_status": {"loaded_models": items}})
reconciled.append({"task_id": task["id"], "status": new_status})
return reconciled
async def fetch_eval_result_content(client: ComputeNodeClient, node: dict[str, Any], job: dict[str, Any]) -> dict[str, Any] | None:
output_dir = job.get("output_dir")
if not output_dir:
return None
full_path = f"{str(output_dir).rstrip('/')}/eval_results.json"
data_root = "/data/yg-ft/"
if full_path.startswith(data_root):
full_path = full_path[len(data_root):]
rel_path = full_path.lstrip("/")
import httpx
url = f"{node['api_base_url'].rstrip('/')}/modelTF/compute/files/read"
async with httpx.AsyncClient(timeout=30, headers=client.headers()) as http:
response = await http.get(url, params={"path": rel_path})
response.raise_for_status()
payload = response.json()
return payload if isinstance(payload, dict) else None
async def poll_compute_jobs_once() -> dict[str, Any]: async def poll_compute_jobs_once() -> dict[str, Any]:
store = get_platform_store() store = get_platform_store()
synced: list[dict[str, Any]] = [] synced: list[dict[str, Any]] = []
@@ -48,4 +152,40 @@ async def poll_compute_jobs_once() -> dict[str, Any]:
standalone_synced.append(store.sync_model_merge_job(record["id"], job)) standalone_synced.append(store.sync_model_merge_job(record["id"], job))
except Exception as exc: # noqa: BLE001 - keep polling other jobs except Exception as exc: # noqa: BLE001 - keep polling other jobs
failed.append({"job_id": record["id"], "error": str(exc)}) failed.append({"job_id": record["id"], "error": str(exc)})
return {"synced": len(synced) + len(standalone_synced), "failed": failed, "items": synced, "standalone": standalone_synced}
# ── Eval job sync ────────────────────────────────────────────────
eval_synced = 0
for eval_task in store.running_eval_tasks():
node = next(
(item for item in store.compute_nodes() if item["id"] == eval_task.get("compute_node_id")),
None,
)
if not node:
failed.append({"eval_task_id": eval_task["id"], "error": "compute node not found"})
continue
try:
client = ComputeNodeClient(node["api_base_url"])
job = await client.get_job(eval_task["compute_job_id"])
result_content = None
# Try to read eval_results.json from the job output directory
if job.get("status") == "completed" and job.get("output_dir"):
try:
result_content = await fetch_eval_result_content(client, node, job)
except Exception:
pass
store.apply_eval_job_result(eval_task["id"], job, result_content)
# 评测 GPU 占用由 eval_tasks 状态派生,无需维护推理内存标记
eval_synced += 1
except Exception as exc: # noqa: BLE001
failed.append({"eval_task_id": eval_task["id"], "error": str(exc)})
# ── Inference load reconciliation ─────────────────────────────────────
try:
inference_reconciled = await reconcile_inference_loads(store)
except Exception as exc: # noqa: BLE001 - keep polling alive
failed.append({"inference_reconcile": str(exc)})
inference_reconciled = []
return {"synced": len(synced) + len(standalone_synced) + eval_synced, "failed": failed,
"items": synced, "standalone": standalone_synced, "eval_synced": eval_synced,
"inference_reconciled": inference_reconciled}

View File

@@ -1,6 +1,6 @@
from __future__ import annotations from __future__ import annotations
from fastapi import APIRouter, Query from fastapi import APIRouter, Body, Query, Request
from fastapi.responses import StreamingResponse from fastapi.responses import StreamingResponse
from app.db.platform_store import ALL_PERMISSIONS, get_platform_store from app.db.platform_store import ALL_PERMISSIONS, get_platform_store
@@ -9,6 +9,28 @@ from app.db.platform_store import ALL_PERMISSIONS, get_platform_store
router = APIRouter(prefix="/system", tags=["system"]) router = APIRouter(prefix="/system", tags=["system"])
@router.post("/audit/visit")
def record_visit(payload: dict = Body(...), request: Request = None) -> dict:
"""记录用户访问业务模块的行为,用于看板用户操作分布统计。"""
action = str(payload.get("action") or payload.get("module") or "").strip()
if not action:
return {"code": 0, "message": "ok", "data": {"recorded": False}}
actor_id = ""
if request is not None:
auth = request.headers.get("Authorization", "")
token = auth.replace("Bearer ", "").strip()
if token.startswith("platform-token-"):
actor_id = token[len("platform-token-"):]
get_platform_store().record_audit(
action=action,
actor_id=actor_id or None,
target_type="module",
target_id=action,
detail=str(payload.get("detail") or ""),
)
return {"code": 0, "message": "ok", "data": {"recorded": True}}
@router.get("/permissions/codes") @router.get("/permissions/codes")
def permission_codes() -> dict: def permission_codes() -> dict:
"""返回平台权限码清单(权限码接口)。""" """返回平台权限码清单(权限码接口)。"""

View File

@@ -10,6 +10,7 @@ dependencies = [
"pydantic>=2.7.0", "pydantic>=2.7.0",
"sqlalchemy>=2.0.30", "sqlalchemy>=2.0.30",
"psycopg[binary]>=3.2.1", "psycopg[binary]>=3.2.1",
"psycopg-pool>=3.2.1",
"alembic>=1.13.1", "alembic>=1.13.1",
"redis>=5.0.4", "redis>=5.0.4",
"httpx>=0.27.0", "httpx>=0.27.0",

View File

@@ -4,6 +4,7 @@ python-multipart>=0.0.9
pydantic>=2.7.0 pydantic>=2.7.0
sqlalchemy>=2.0.30 sqlalchemy>=2.0.30
psycopg[binary]>=3.2.1 psycopg[binary]>=3.2.1
psycopg-pool>=3.2.1
alembic>=1.13.1 alembic>=1.13.1
redis>=5.0.4 redis>=5.0.4
httpx>=0.27.0 httpx>=0.27.0
@@ -18,3 +19,7 @@ llama-index-core==0.14.23
llama-index-embeddings-huggingface==0.6.1 llama-index-embeddings-huggingface==0.6.1
docling==2.115.0 docling==2.115.0
tiktoken>=0.7.0 tiktoken>=0.7.0
# 测试与代码检查
pytest>=8.2.0
ruff>=0.5.0

View File

@@ -0,0 +1,276 @@
"""
模型推理异步加载改造的单元测试。
覆盖:
- model_compare_load异步派发立即返回 starting + 节点信息(不等待加载完成)
- model_compare_delete先删记录卸载失败也不阻塞删除
- reconcile_inference_loadsstarting -> ready/error/idle/不可达的状态迁移与封顶
- _unload_from_compute_node任务感知只命中记录中的节点
"""
from __future__ import annotations
import asyncio
from types import SimpleNamespace
from typing import Any
from app.api.v1.endpoints.platform import model_compare_delete, model_compare_load
import app.api.v1.endpoints.platform as platform
from app.modules.compute_gateway.client import ComputeNodeClient
from app.modules.compute_gateway.sync import MAX_STARTING_ATTEMPTS, reconcile_inference_loads
class FakeInferenceStore:
"""内存 store仅实现推理加载/对账用到的接口。"""
def __init__(self, tasks: list[dict[str, Any]] | None = None, nodes: list[dict[str, Any]] | None = None) -> None:
self._tasks: dict[str, dict[str, Any]] = {t["id"]: dict(t) for t in (tasks or [])}
self._nodes = nodes or []
self._inference_nodes: set[str] = set()
def compare_task(self, task_id: str) -> dict[str, Any]:
if task_id not in self._tasks:
raise KeyError(task_id)
return dict(self._tasks[task_id])
def compare_tasks(self) -> list[dict[str, Any]]:
return [dict(t) for t in self._tasks.values()]
def update_compare_task(self, task_id: str, payload: dict[str, Any]) -> dict[str, Any]:
current = self._tasks[task_id]
merged = {**current, **payload, "id": task_id}
self._tasks[task_id] = merged
return dict(merged)
def delete_compare_task(self, task_id: str) -> None:
self._tasks.pop(task_id, None)
def compute_nodes(self) -> list[dict[str, Any]]:
return [dict(n) for n in self._nodes]
def model(self, model_id: str) -> dict[str, Any]:
raise KeyError(model_id)
def trained_models(self) -> list[dict[str, Any]]:
return []
def mark_inference_loaded(self, node_id: str) -> None:
self._inference_nodes.add(node_id)
def mark_inference_unloaded(self, node_id: str) -> None:
self._inference_nodes.discard(node_id)
def is_inference_loaded(self, node_id: str) -> bool:
return node_id in self._inference_nodes
def _node(node_id: str, code: str = "") -> dict[str, Any]:
return {
"id": node_id,
"code": code or node_id,
"name": code or node_id,
"api_base_url": f"http://{code or node_id}:19100",
"enabled": True,
"scheduler_status": "online",
}
def _task(task_id: str, *, node_id: str | None = None, load_status: dict[str, Any] | None = None) -> dict[str, Any]:
return {
"id": task_id,
"name": f"task-{task_id}",
"status": "pending",
"models": [
{"model_id": "m_1", "model_name": "qwen", "model_path": "/models/qwen", "node_id": node_id}
],
"load_status": load_status or {"loaded_models": []},
}
async def _fake_inference_load(self, payload: dict[str, Any]) -> dict[str, Any]:
return {"loaded": False, "status": "loading", "request_id": "req-1"}
async def _fake_inference_unload(self) -> dict[str, Any]:
return {"unloaded": True, "status": "idle"}
def _patch_store(monkeypatch, store: FakeInferenceStore) -> None:
monkeypatch.setattr(platform, "get_platform_store", lambda: store)
monkeypatch.setattr(platform, "get_settings", lambda: SimpleNamespace(compute_mode="real"))
def test_select_eval_node_prefers_model_node(monkeypatch) -> None:
from app.api.v1.endpoints.platform import _select_eval_node
store = FakeInferenceStore(nodes=[_node("n1"), _node("n2")])
# 指定模型所在节点时优先返回该节点
assert _select_eval_node(store, "n2")["id"] == "n2"
# 无指定节点时回退到第一个在线节点
assert _select_eval_node(store, None)["id"] == "n1"
def test_select_eval_node_returns_none_when_model_node_offline(monkeypatch) -> None:
from app.api.v1.endpoints.platform import _select_eval_node
nodes = [_node("n1"), _node("n2")]
nodes[1]["enabled"] = False
store = FakeInferenceStore(nodes=nodes)
# 模型所在节点不可用 → 明确失败,不派发到其它节点
assert _select_eval_node(store, "n2") is None
# 无指定节点时仍回退第一个在线节点
assert _select_eval_node(store, None)["id"] == "n1"
def test_model_compare_load_dispatches_and_returns_starting(monkeypatch) -> None:
store = FakeInferenceStore(tasks=[_task("t1", node_id="n1")], nodes=[_node("n1")])
_patch_store(monkeypatch, store)
monkeypatch.setattr(ComputeNodeClient, "inference_load", _fake_inference_load)
result = asyncio.run(model_compare_load("t1"))
assert result["code"] == 0
updated = result["data"]
assert updated["status"] == "starting"
items = updated["load_status"]["loaded_models"]
assert items[0]["status"] == "starting"
assert items[0]["node_id"] == "n1"
assert "n1" in store._inference_nodes
def test_model_compare_load_marks_error_when_all_nodes_fail(monkeypatch) -> None:
store = FakeInferenceStore(tasks=[_task("t1", node_id="n1")], nodes=[_node("n1")])
_patch_store(monkeypatch, store)
async def _raise(self, payload: dict[str, Any]) -> dict[str, Any]:
raise RuntimeError("conn refused")
monkeypatch.setattr(ComputeNodeClient, "inference_load", _raise)
result = asyncio.run(model_compare_load("t1"))
updated = result["data"]
assert updated["status"] == "failed"
assert updated["load_status"]["loaded_models"][0]["status"] == "error"
assert "conn refused" in updated["load_status"]["loaded_models"][0]["error"]
def test_model_compare_delete_removes_record_even_if_unload_raises(monkeypatch) -> None:
task = _task(
"t1",
node_id="n1",
load_status={"loaded_models": [{"model_id": "m_1", "status": "ready", "node_id": "n1"}]},
)
store = FakeInferenceStore(tasks=[task], nodes=[_node("n1")])
_patch_store(monkeypatch, store)
async def _raise(self) -> dict[str, Any]:
raise RuntimeError("unload boom")
monkeypatch.setattr(ComputeNodeClient, "inference_unload", _raise)
result = asyncio.run(model_compare_delete("t1"))
assert result["data"] == {"deleted": "t1"}
assert "t1" not in store._tasks
# finally 中仍清掉了节点标记
assert "n1" not in store._inference_nodes
def test_unload_from_compute_node_only_hits_recorded_node(monkeypatch) -> None:
task = _task(
"t1",
load_status={"loaded_models": [{"model_id": "m_1", "status": "ready", "node_id": "n1"}]},
)
store = FakeInferenceStore(tasks=[task], nodes=[_node("n1"), _node("n2")])
_patch_store(monkeypatch, store)
monkeypatch.setattr(ComputeNodeClient, "inference_unload", _fake_inference_unload)
from app.api.v1.endpoints.platform import _unload_from_compute_node
result = asyncio.run(_unload_from_compute_node(store, task=task))
assert result["unloaded"] is True
# 只命中任务记录中的节点 n1n2 未被卸载
assert [r["node_id"] for r in result["nodes"]] == ["n1"]
assert "n1" not in store._inference_nodes
async def _status_ready(self) -> dict[str, Any]:
return {"loaded": True, "status": "ready", "model_name": "qwen"}
def test_reconcile_transitions_starting_to_ready(monkeypatch) -> None:
task = _task(
"t1",
node_id="n1",
load_status={"loaded_models": [{"model_id": "m_1", "status": "starting", "node_id": "n1"}]},
)
store = FakeInferenceStore(tasks=[task], nodes=[_node("n1")])
monkeypatch.setattr(ComputeNodeClient, "inference_status", _status_ready)
reconciled = asyncio.run(reconcile_inference_loads(store))
assert reconciled == [{"task_id": "t1", "status": "loaded"}]
updated = store._tasks["t1"]
assert updated["status"] == "loaded"
assert updated["load_status"]["loaded_models"][0]["status"] == "ready"
assert "n1" in store._inference_nodes
def test_reconcile_transitions_to_error_and_failed(monkeypatch) -> None:
async def _status_error(self) -> dict[str, Any]:
return {"loaded": False, "status": "error", "error": "CUDA out of memory"}
task = _task(
"t1",
node_id="n1",
load_status={"loaded_models": [{"model_id": "m_1", "status": "starting", "node_id": "n1"}]},
)
store = FakeInferenceStore(tasks=[task], nodes=[_node("n1")])
monkeypatch.setattr(ComputeNodeClient, "inference_status", _status_error)
reconciled = asyncio.run(reconcile_inference_loads(store))
assert reconciled == [{"task_id": "t1", "status": "failed"}]
item = store._tasks["t1"]["load_status"]["loaded_models"][0]
assert item["status"] == "error"
assert "CUDA out of memory" in item["error"]
assert "n1" not in store._inference_nodes
def test_reconcile_idle_marks_model_disappeared(monkeypatch) -> None:
async def _status_idle(self) -> dict[str, Any]:
return {"loaded": False, "status": "idle"}
task = _task(
"t1",
node_id="n1",
load_status={"loaded_models": [{"model_id": "m_1", "status": "starting", "node_id": "n1"}]},
)
store = FakeInferenceStore(tasks=[task], nodes=[_node("n1")])
monkeypatch.setattr(ComputeNodeClient, "inference_status", _status_idle)
asyncio.run(reconcile_inference_loads(store))
item = store._tasks["t1"]["load_status"]["loaded_models"][0]
assert item["status"] == "error"
assert "disappeared" in item["error"]
assert store._tasks["t1"]["status"] == "failed"
def test_reconcile_unreachable_node_flips_to_error_after_cap(monkeypatch) -> None:
async def _raise(self) -> dict[str, Any]:
raise RuntimeError("conn refused")
task = _task(
"t1",
node_id="n1",
load_status={"loaded_models": [{"model_id": "m_1", "status": "starting", "node_id": "n1"}]},
)
store = FakeInferenceStore(tasks=[task], nodes=[_node("n1")])
monkeypatch.setattr(ComputeNodeClient, "inference_status", _raise)
# 每次轮询前重置节流时间戳,逐次推进 load_attempts 到封顶
for _ in range(MAX_STARTING_ATTEMPTS):
item = store._tasks["t1"]["load_status"]["loaded_models"][0]
item["last_polled_at"] = 0
asyncio.run(reconcile_inference_loads(store))
item = store._tasks["t1"]["load_status"]["loaded_models"][0]
assert item["status"] == "error"
assert "unreachable" in item["error"]
assert store._tasks["t1"]["status"] == "failed"

View File

@@ -1,5 +1,7 @@
from __future__ import annotations from __future__ import annotations
import asyncio
import json
import os import os
import math import math
import hashlib import hashlib
@@ -449,6 +451,29 @@ def create_app() -> FastAPI:
accelerator_errors, accelerator_warnings, accelerator = _validate_training_accelerator(payload) accelerator_errors, accelerator_warnings, accelerator = _validate_training_accelerator(payload)
errors.extend(accelerator_errors) errors.extend(accelerator_errors)
warnings.extend(accelerator_warnings) warnings.extend(accelerator_warnings)
elif engine == "eval":
# Eval engine: validate model path and dataset path
if not payload.get("model_name_or_path"):
errors.append("model_name_or_path is required for eval")
else:
path_checks.append(_check_path_item({
"name": "model_name_or_path",
"path": payload.get("model_name_or_path", ""),
"type": "any",
"required": True,
}))
if payload.get("dataset_path"):
path_checks.append(_check_path_item({
"name": "dataset_path",
"path": payload.get("dataset_path", ""),
"type": "file",
"required": True,
}))
else:
errors.append("dataset_path is required for eval")
if shutil.which("python") is None:
errors.append("python runtime not found")
elif engine == "smoke": elif engine == "smoke":
warnings.append("smoke engine skips model and dataset path checks") warnings.append("smoke engine skips model and dataset path checks")
@@ -688,14 +713,14 @@ def create_app() -> FastAPI:
infer_backend=payload.get("infer_backend", "huggingface"), infer_backend=payload.get("infer_backend", "huggingface"),
infer_dtype=payload.get("infer_dtype", "auto"), infer_dtype=payload.get("infer_dtype", "auto"),
) )
if not result.get("loaded"):
raise HTTPException(status_code=500, detail=result.get("error", "model load failed"))
return result return result
@app.post(f"{route_prefix}/inference/unload") @app.post(f"{route_prefix}/inference/unload")
async def inference_unload() -> dict[str, Any]: async def inference_unload() -> dict[str, Any]:
"""Unload the currently loaded model and free GPU memory.""" """Unload the currently loaded model and free GPU memory."""
return get_inference_session().unload() # Teardown (gc.collect + cuda.empty_cache) can take a while; run it off
# the event loop so /health and /inference/status stay responsive.
return await asyncio.to_thread(get_inference_session().unload)
@app.get(f"{route_prefix}/inference/status") @app.get(f"{route_prefix}/inference/status")
async def inference_status() -> dict[str, Any]: async def inference_status() -> dict[str, Any]:
@@ -715,7 +740,10 @@ def create_app() -> FastAPI:
messages = payload.get("messages") or [] messages = payload.get("messages") or []
if not messages: if not messages:
raise HTTPException(status_code=400, detail="messages is required") raise HTTPException(status_code=400, detail="messages is required")
result = get_inference_session().chat( # Generation is long-running; run it in a thread so the event loop keeps
# serving /inference/status and /health during inference.
result = await asyncio.to_thread(
get_inference_session().chat,
messages=messages, messages=messages,
temperature=float(payload.get("temperature", 0.95)), temperature=float(payload.get("temperature", 0.95)),
top_p=float(payload.get("top_p", 0.7)), top_p=float(payload.get("top_p", 0.7)),
@@ -813,6 +841,22 @@ def create_app() -> FastAPI:
"checksum_sha256": checksum, "checksum_sha256": checksum,
} }
@app.get(f"{route_prefix}/compute/files/read")
async def read_file(path: str = Query(...)) -> JSONResponse:
"""Read a text file from within YG_FT_DATA_ROOT. Used by the backend
to fetch eval results and other job outputs."""
data_root = Path(os.getenv("YG_FT_DATA_ROOT", "/data/yg-ft"))
target = (data_root / path.lstrip("/\\")).resolve()
if not _path_inside(data_root, target):
raise HTTPException(status_code=400, detail="path must stay inside YG_FT_DATA_ROOT")
if not target.is_file():
raise HTTPException(status_code=404, detail="file not found")
try:
content = target.read_text(encoding="utf-8")
return JSONResponse(json.loads(content) if content.strip().startswith("{") else {"content": content})
except Exception as exc:
raise HTTPException(status_code=500, detail=str(exc))
@app.get(f"{route_prefix}/compute/files/{{file_id}}/download") @app.get(f"{route_prefix}/compute/files/{{file_id}}/download")
async def download_file(file_id: str) -> FileResponse: async def download_file(file_id: str) -> FileResponse:
upload_root = Path(os.getenv("YG_FT_DATA_ROOT", "/data/yg-ft")) / "uploads" upload_root = Path(os.getenv("YG_FT_DATA_ROOT", "/data/yg-ft")) / "uploads"

View File

@@ -204,6 +204,31 @@ def build_command(config: dict[str, Any], llama_factory_home: str = "/app/LLaMA-
command.extend(["--quantization_bit", str(quantization_bit)]) command.extend(["--quantization_bit", str(quantization_bit)])
return LlamaFactoryCommand(command=command, work_dir=str(Path(llama_factory_home)), env={}) return LlamaFactoryCommand(command=command, work_dir=str(Path(llama_factory_home)), env={})
if engine == "eval":
output_dir = config.get("output_dir") or f"/data/yg-ft/outputs/{config.get('name', 'eval-job')}"
eval_config_path = str(Path(output_dir) / "eval_config.json")
eval_config = {
"model_name_or_path": config.get("model_name_or_path", ""),
"adapter_name_or_path": config.get("adapter_name_or_path", ""),
"template": config.get("template", "qwen"),
"dataset_path": config.get("dataset_path", ""),
"output_dir": output_dir,
"basic_metrics": config.get("basic_metrics", {}),
"dimension": config.get("dimension", {}),
"temperature": config.get("temperature", 0.1),
"top_p": config.get("top_p", 0.95),
"max_new_tokens": config.get("max_new_tokens", 512),
"infer_backend": config.get("infer_backend", "huggingface"),
"infer_dtype": config.get("infer_dtype", "auto"),
}
Path(output_dir).mkdir(parents=True, exist_ok=True)
Path(eval_config_path).write_text(json.dumps(eval_config, ensure_ascii=False, indent=2), encoding="utf-8")
return LlamaFactoryCommand(
command=["python", "-u", "-m", "compute.engines.llama_factory.eval_runner", "--config", eval_config_path],
work_dir="/app",
env={},
)
errors = validate_config(config) errors = validate_config(config)
if errors: if errors:
raise ValueError("; ".join(errors)) raise ValueError("; ".join(errors))

View File

@@ -0,0 +1,485 @@
"""
Evaluation runner — executes model evaluation as a subprocess job.
Usage:
python -m compute.engines.llama_factory.eval_runner --config <config_json_path>
The config JSON is written by the compute API before spawning this subprocess.
Results are written to ``output_dir/eval_results.json`` and progress is printed
to stdout (captured as job logs).
"""
from __future__ import annotations
import json
import math
import re
import sys
import time
from difflib import SequenceMatcher
from pathlib import Path
from typing import Any
def _load_dataset(path: str) -> list[dict[str, Any]]:
"""Load a JSON or JSONL dataset file.
Supports common field names used across the platform:
* ``instruction`` + ``input`` + ``output`` (Alpaca-style)
* ``question`` + ``answer``
* ``messages`` (ShareGPT-style the last assistant message is treated as reference)
"""
file_path = Path(path)
text = file_path.read_text(encoding="utf-8", errors="replace").strip()
if not text:
return []
if file_path.suffix.lower() == ".json":
value = json.loads(text)
if isinstance(value, list):
return [item for item in value if isinstance(item, dict)]
return [value] if isinstance(value, dict) else []
samples: list[dict[str, Any]] = []
for line in text.splitlines():
line = line.strip()
if not line:
continue
try:
obj = json.loads(line)
except json.JSONDecodeError:
continue
if isinstance(obj, dict):
samples.append(obj)
return samples
def _sample_question(sample: dict[str, Any]) -> str:
"""Extract the user-facing question / instruction from a sample."""
if sample.get("instruction"):
text = sample["instruction"]
if sample.get("input"):
text += "\n" + sample["input"]
return text
if sample.get("question"):
return sample["question"]
# ShareGPT-style: use the last user message as question
messages = sample.get("messages") or []
user_msgs = [m["content"] for m in messages if m.get("role") == "user"]
return user_msgs[-1] if user_msgs else ""
def _sample_reference(sample: dict[str, Any]) -> str:
"""Extract the reference answer from a sample."""
if sample.get("output"):
return sample["output"]
if sample.get("answer"):
return sample["answer"]
messages = sample.get("messages") or []
assistant_msgs = [m["content"] for m in messages if m.get("role") == "assistant"]
return assistant_msgs[-1] if assistant_msgs else ""
# ---------------------------------------------------------------------------
# Basic metrics
# ---------------------------------------------------------------------------
def _compute_bleu(references: list[str], predictions: list[str], ngram: int = 4) -> dict[str, Any]:
"""Compute BLEU score via sacrebleu (corpus-level)."""
try:
from sacrebleu.metrics import BLEU
except ImportError:
return {"enabled": False, "error": "sacrebleu not installed", "score": 0}
bleu = BLEU(max_ngram_order=ngram)
# sacrebleu expects list-of-strings; we have one reference per prediction
score = bleu.corpus_score(predictions, [references])
return {
"enabled": True,
"score": round(score.score, 2),
"bleu": round(score.score, 2),
}
def _compute_rouge(references: list[str], predictions: list[str], methods: list[str] | None = None) -> dict[str, Any]:
"""Compute ROUGE scores via rouge-score."""
try:
from rouge_score import rouge_scorer
except ImportError:
return {"enabled": False, "error": "rouge-score not installed", "score": 0}
methods = methods or ["rouge1", "rouge2", "rougeL"]
# Normalize: map "rouge_1"/"rouge1" → "rouge1", "rouge_l"/"rougeL" → "rougeL"
_rouge_aliases = {"rouge_1": "rouge1", "rouge_2": "rouge2", "rouge_l": "rougeL"}
methods = [_rouge_aliases.get(m, m.replace("_", "")) for m in methods]
scorer = rouge_scorer.RougeScorer(methods, use_stemmer=True)
totals: dict[str, float] = {}
n = max(len(predictions), 1)
for ref, pred in zip(references, predictions):
result = scorer.score(ref, pred)
for key in methods:
totals[key] = totals.get(key, 0) + result[key].fmeasure
avg = {k: round(v / n, 4) for k, v in totals.items()}
return {"enabled": True, "score": round(avg.get("rougeL", avg.get("rouge1", 0)) * 100, 2), **avg}
def _compute_cosine(references: list[str], predictions: list[str]) -> dict[str, Any]:
"""Compute average cosine similarity via sklearn."""
try:
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
except ImportError:
return {"enabled": False, "error": "scikit-learn not installed", "score": 0}
try:
vectorizer = TfidfVectorizer()
tfidf = vectorizer.fit_transform(references + predictions)
n = len(references)
ref_vec = tfidf[:n]
pred_vec = tfidf[n:]
sims = cosine_similarity(ref_vec, pred_vec).diagonal()
return {"enabled": True, "score": round(float(sims.mean()) * 100, 2)}
except ValueError:
return {"enabled": True, "score": 0, "error": "insufficient text for vectorization"}
def _normalize_text(value: str) -> str:
return re.sub(r"\s+", " ", str(value or "").strip().lower())
def _compute_exact_match(references: list[str], predictions: list[str]) -> dict[str, Any]:
total = len(predictions)
if not total:
return {"enabled": True, "score": 0, "matched": 0, "total": 0}
matched = sum(
1
for ref, pred in zip(references, predictions)
if _normalize_text(ref) == _normalize_text(pred)
)
return {"enabled": True, "score": round(matched / total * 100, 2), "matched": matched, "total": total}
def _compute_text_similarity(references: list[str], predictions: list[str]) -> dict[str, Any]:
if not predictions:
return {"enabled": True, "score": 0}
scores = [
SequenceMatcher(None, _normalize_text(ref), _normalize_text(pred)).ratio()
for ref, pred in zip(references, predictions)
]
return {"enabled": True, "score": round(sum(scores) / max(len(scores), 1) * 100, 2)}
# ---------------------------------------------------------------------------
# LLM Judge
# ---------------------------------------------------------------------------
def _judge_sample(
question: str,
reference: str,
prediction: str,
config: dict[str, Any],
) -> dict[str, Any]:
"""Call an OpenAI-compatible LLM to judge a single sample.
Returns a dict with keys:
score, max_score, passed, judgement, evaluation_reason, error_type
"""
api_url = (config.get("api_url") or "").strip().rstrip("/")
api_key = (config.get("api_key") or "").strip()
eval_model = (config.get("eval_model") or "").strip()
# 优先使用模型记录里配置的真实 API 模型名(如 deepseek-chat
# 否则回退到平台内部模型名
api_model = (config.get("api_model") or "").strip() or eval_model
eval_prompt = (config.get("eval_prompt") or "").strip()
score_min = float(config.get("score_min", 0))
score_max = float(config.get("score_max", 5))
pass_threshold = float(config.get("pass_threshold", 3))
if not api_url or not eval_model:
return {"score": 0, "max_score": score_max, "passed": False, "judgement": "未配置",
"evaluation_reason": "未配置评测模型", "error_type": "其他"}
system_msg = (
eval_prompt
or "你是一个专业的评测专家。请根据参考答-案对被测模型的输出进行评分。"
)
user_msg = (
f"## 问题\n{question}\n\n"
f"## 参考答案\n{reference}\n\n"
f"## 模型输出\n{prediction}\n\n"
f"请给出 {score_min}-{score_max} 分的评分,并说明理由。"
)
try:
import urllib.request
import urllib.error
body = json.dumps({
"model": api_model,
"messages": [
{"role": "system", "content": system_msg},
{"role": "user", "content": user_msg},
],
"temperature": 0.3,
"max_tokens": 512,
}).encode("utf-8")
req = urllib.request.Request(
f"{api_url}/v1/chat/completions",
data=body,
headers={
"Content-Type": "application/json",
"Authorization": f"Bearer {api_key}",
},
)
resp = urllib.request.urlopen(req, timeout=120)
data = json.loads(resp.read().decode("utf-8"))
reply = data["choices"][0]["message"]["content"]
except Exception as exc:
return {"score": 0, "max_score": score_max, "passed": False,
"judgement": "错误", "evaluation_reason": f"评测模型调用失败: {exc}",
"error_type": "其他"}
# Parse score from reply — look for patterns like "4分" or "Score: 4"
score = 0
import re
score_patterns = [
r'(?:得分|分数|评分|score)[^\d]*(\d+(?:\.\d+)?)',
r'(\d+(?:\.\d+)?)\s*分',
r'(\d+(?:\.\d+)?)\s*/\s*\d+',
]
for pat in score_patterns:
m = re.search(pat, reply, re.IGNORECASE)
if m:
try:
score = float(m.group(1))
except ValueError:
continue
break
score = max(score_min, min(score_max, score))
passed = score >= pass_threshold
# Determine judgement label
if score >= pass_threshold + 1:
judgement = "正确"
elif score >= pass_threshold:
judgement = "部分正确"
else:
judgement = "错误"
# Guess error type from reply
reply_lower = reply.lower()
if any(w in reply_lower for w in ["幻觉", "hallucination", "编造"]):
error_type = "幻觉"
elif any(w in reply_lower for w in ["不完整", "incomplete", "遗漏"]):
error_type = "不完整"
elif any(w in reply_lower for w in ["格式", "format"]):
error_type = "格式偏差"
elif any(w in reply_lower for w in ["混淆", "confusion", "错误"]):
error_type = "混淆"
else:
error_type = "其他"
return {
"score": score,
"max_score": score_max,
"passed": passed,
"judgement": judgement,
"evaluation_reason": reply[:2000],
"error_type": error_type,
}
# ---------------------------------------------------------------------------
# Main entry point
# ---------------------------------------------------------------------------
def run_eval(config: dict[str, Any]) -> dict[str, Any]:
"""Execute a full evaluation run. Returns the result dict (also written to file)."""
model_path = config["model_name_or_path"]
adapter_path = config.get("adapter_name_or_path", "")
template = config.get("template", "qwen")
dataset_path = config["dataset_path"]
output_dir = Path(config["output_dir"])
output_dir.mkdir(parents=True, exist_ok=True)
basic_cfg = config.get("basic_metrics", {})
dimension_cfg = config.get("dimension", {}) or {}
output_precision = int(basic_cfg.get("output_precision", 2))
# ---- 1. Load dataset ----
print(f"[eval] loading dataset: {dataset_path}")
raw_samples = _load_dataset(dataset_path)
print(f"[eval] loaded {len(raw_samples)} samples")
# ---- 2. Load model ----
print(f"[eval] loading model: {model_path}")
from compute.engines.llama_factory.inference import InferenceSession
session = InferenceSession()
session.load(
model_name_or_path=model_path,
adapter_name_or_path=adapter_path,
template=template,
infer_backend=config.get("infer_backend", "huggingface"),
infer_dtype=config.get("infer_dtype", "auto"),
)
# load() 为异步加载(立即返回 loading必须等待后台线程完成后再进行推理
load_result = session.wait_until_loaded(timeout=float(config.get("load_timeout", 1800)))
if not load_result.get("loaded"):
raise RuntimeError(f"model load failed: {load_result.get('error', 'unknown')}")
print(f"[eval] model loaded OK")
# ---- 3. Run inference on each sample ----
samples: list[dict[str, Any]] = []
predictions: list[str] = []
references: list[str] = []
questions: list[str] = []
total = len(raw_samples)
judge_enabled = bool(dimension_cfg.get("eval_model") and dimension_cfg.get("api_url"))
print(f"[eval] starting inference on {total} samples, judge={'enabled' if judge_enabled else 'disabled'}")
for idx, raw in enumerate(raw_samples, start=1):
question = _sample_question(raw)
reference = _sample_reference(raw)
if not question:
print(f"[eval] sample {idx}/{total}: skipped (no question)")
continue
# Inference
chat_msgs = [{"role": "user", "content": question}]
result = session.chat(
chat_msgs,
temperature=float(config.get("temperature", 0.1)),
top_p=float(config.get("top_p", 0.95)),
max_new_tokens=int(config.get("max_new_tokens", 512)),
do_sample=False,
)
prediction = result.get("response", "") if not result.get("error") else f"[ERROR] {result['error']}"
predictions.append(prediction)
references.append(reference)
questions.append(question)
# LLM Judge
judge_result: dict[str, Any] = {}
if judge_enabled:
judge_result = _judge_sample(question, reference, prediction, dimension_cfg)
samples.append({
"index": idx,
"input": question,
"reference_answer": reference,
"model_output": prediction,
"score": judge_result.get("score"),
"max_score": judge_result.get("max_score", dimension_cfg.get("score_max", 5)),
"passed": judge_result.get("passed"),
"judgement": judge_result.get("judgement"),
"evaluation_reason": judge_result.get("evaluation_reason", ""),
"error_type": judge_result.get("error_type"),
"dimension_scores": [
{"name": "judge_score", "score": judge_result.get("score", 0),
"max_score": judge_result.get("max_score", dimension_cfg.get("score_max", 5))},
] if judge_result else [],
"status": "completed",
})
progress_pct = int(idx / max(total, 1) * 100)
print(f"[eval] sample {idx}/{total} ({progress_pct}%) done")
# ---- 4. Compute basic metrics ----
print(f"[eval] computing basic metrics on {len(predictions)} predictions")
metrics_result: dict[str, Any] = {}
bleu_cfg = basic_cfg.get("bleu", {})
if bleu_cfg.get("enabled"):
metrics_result["bleu"] = _compute_bleu(references, predictions, int(bleu_cfg.get("ngram", 4)))
rouge_cfg = basic_cfg.get("rouge", {})
if rouge_cfg.get("enabled"):
metrics_result["rouge"] = _compute_rouge(references, predictions, rouge_cfg.get("methods"))
cosine_cfg = basic_cfg.get("cosine", {})
if cosine_cfg.get("enabled"):
metrics_result["cosine"] = _compute_cosine(references, predictions)
metrics_result["exact_match"] = _compute_exact_match(references, predictions)
metrics_result["text_similarity"] = _compute_text_similarity(references, predictions)
# ---- 5. Summarise ----
completed = len(samples)
if judge_enabled:
scored = [s for s in samples if s.get("score") is not None]
passed_count = len([s for s in scored if s.get("passed")])
avg_score = round(sum(s["score"] for s in scored) / max(len(scored), 1), output_precision)
max_score = dimension_cfg.get("score_max", 5)
overall_score = round(avg_score / max_score * 100, output_precision)
overall_score_max = 100
dimension_summary = [{
"name": "综合评分",
"score": overall_score,
"max_score": 100,
"pass_rate": round(passed_count / max(completed, 1) * 100, 1),
}]
overall_evaluation = f"评测完成:{completed} 样本,{passed_count} 通过,平均 {avg_score}/{max_score}"
else:
passed_count = 0
enabled_scores = [
float(item.get("score") or 0)
for item in metrics_result.values()
if isinstance(item, dict) and item.get("enabled", True) and item.get("score") is not None
]
overall_score = round(sum(enabled_scores) / len(enabled_scores), output_precision) if enabled_scores else 0
overall_score_max = 100
dimension_summary = [
{
"name": name,
"score": float(item.get("score") or 0),
"max_score": 100,
"pass_rate": float(item.get("score") or 0),
}
for name, item in metrics_result.items()
if isinstance(item, dict) and item.get("enabled", True) and item.get("score") is not None
]
overall_evaluation = f"评测完成:{completed} 样本(未配置 LLM 评委)"
result = {
"overall_score": overall_score,
"overall_score_max": overall_score_max,
"overall_evaluation": overall_evaluation,
"improvement_suggestions": [],
"dimension_summary": dimension_summary,
"samples": samples,
"sample_count": total,
"completed_count": completed,
"passed_count": passed_count,
"basic_metrics": metrics_result,
}
# ---- 6. Write results ----
result_path = output_dir / "eval_results.json"
result_path.write_text(json.dumps(result, ensure_ascii=False, indent=2), encoding="utf-8")
print(f"[eval] results written to {result_path}")
return result
def main() -> None:
import argparse
parser = argparse.ArgumentParser(description="YG-FT Evaluation Runner")
parser.add_argument("--config", required=True, help="Path to eval config JSON file")
args = parser.parse_args()
config_path = Path(args.config)
if not config_path.exists():
print(f"FATAL: config file not found: {args.config}", file=sys.stderr)
sys.exit(1)
config = json.loads(config_path.read_text(encoding="utf-8"))
start = time.time()
try:
run_eval(config)
elapsed = time.time() - start
print(f"[eval] DONE in {elapsed:.1f}s")
except Exception as exc:
print(f"[eval] FAILED: {exc}", file=sys.stderr)
import traceback
traceback.print_exc()
sys.exit(1)
if __name__ == "__main__":
main()

View File

@@ -2,123 +2,270 @@ from __future__ import annotations
import threading import threading
import time import time
from typing import Any import uuid
from typing import Any, Iterator
class InferenceSession: class InferenceSession:
"""Manages a loaded model for inference with LLaMA-Factory ChatModel.""" """Manages a loaded model for inference with LLaMA-Factory ChatModel.
Model loading is asynchronous: ``load()`` spawns a background daemon thread
and returns immediately with ``status == "loading"``. ``info()`` (served by
``/inference/status``) is always responsive, so the platform backend can
poll loading progress without being blocked by a minutes-long model load —
which previously froze the whole compute node event loop.
State machine: idle -> loading -> ready | error, ready -> idle (unload),
loading -> idle (cancelled). Long operations (ChatModel build, teardown,
generation) never run while holding ``_state_lock``; they either run in the
worker thread or under ``_chat_lock`` only.
"""
def __init__(self) -> None: def __init__(self) -> None:
self._state_lock = threading.Lock() # brief state transitions only
self._chat_lock = threading.Lock() # serialize chat/teardown
self._status: str = "idle"
self._error: str = ""
self._request_id: str = ""
self._load_args: dict[str, Any] = {}
self._teardown_old = False # load-while-ready: unload old before loading new
self._cancel_requested = False # unload-while-loading: tear down after load finishes
self._load_thread: threading.Thread | None = None
self._model: Any = None self._model: Any = None
self._tokenizer: Any = None self._tokenizer: Any = None
self._generating_args: dict[str, Any] = {} self._generating_args: dict[str, Any] = {}
self._model_name: str = "" self._model_name: str = ""
self._adapter_path: str = "" self._adapter_path: str = ""
self._lock = threading.Lock()
self._loaded_at: float = 0.0 self._loaded_at: float = 0.0
self._status: str = "idle"
@property @property
def status(self) -> str: def status(self) -> str:
return self._status with self._state_lock:
return self._status
@property
def model_name(self) -> str:
return self._model_name
@property
def adapter_path(self) -> str:
return self._adapter_path
@property
def loaded_at(self) -> float:
return self._loaded_at
def info(self) -> dict[str, Any]: def info(self) -> dict[str, Any]:
with self._state_lock:
return {
"loaded": self._status == "ready",
"status": self._status,
"model_name": self._model_name,
"adapter_path": self._adapter_path,
"loaded_at": self._loaded_at,
"request_id": self._request_id,
"error": self._error,
}
def wait_until_loaded(self, timeout: float | None = None) -> dict[str, Any]:
"""Wait for an in-flight async load to finish and return its outcome.
供同步消费方(如 eval_runner 子进程)使用:``load()`` 立即返回 loading 后,
调用本方法等待后台加载线程完成,拿到最终的 loaded/error 结果。
若在 timeout 秒内仍未加载完成,返回 ``status == "loading"`` 并附上超时提示。
"""
with self._state_lock:
thread = self._load_thread
if thread is not None and thread.is_alive():
thread.join(timeout=timeout)
with self._state_lock:
loaded = self._status == "ready"
status = self._status
error = self._error
if not loaded and status == "loading":
error = error or f"model load timed out after {timeout or 'N/A'}s"
return { return {
"loaded": self._status == "ready", "loaded": loaded,
"status": self._status, "status": status,
"model_name": self._model_name, "model_name": self._model_name,
"adapter_path": self._adapter_path, "adapter_path": self._adapter_path,
"loaded_at": self._loaded_at, "error": error,
} }
def load(self, model_name_or_path, adapter_name_or_path="", template="qwen", infer_backend="huggingface", infer_dtype="auto", **kwargs): def load(
with self._lock: self,
model_name_or_path,
adapter_name_or_path="",
template="qwen",
infer_backend="huggingface",
infer_dtype="auto",
**kwargs,
) -> dict[str, Any]:
with self._state_lock:
if self._status == "loading": if self._status == "loading":
return {"loaded": False, "error": "model is already loading"} # A model is already loading — dedupe, reuse the same request id.
if self._status == "ready": return {"loaded": False, "status": "loading", "request_id": self._request_id}
self.unload() self._teardown_old = self._status == "ready"
self._status = "loading" self._status = "loading"
self._error = ""
self._request_id = uuid.uuid4().hex[:12]
self._cancel_requested = False
self._load_args = {
"model_name_or_path": model_name_or_path,
"template": template,
"infer_backend": infer_backend,
"infer_dtype": infer_dtype,
}
if adapter_name_or_path:
self._load_args["adapter_name_or_path"] = adapter_name_or_path
self._load_args.update(kwargs)
self._model_name = model_name_or_path self._model_name = model_name_or_path
self._adapter_path = adapter_name_or_path self._adapter_path = adapter_name_or_path
try: self._load_thread = threading.Thread(target=self._load_worker, daemon=True)
from llamafactory.chat import ChatModel self._load_thread.start()
from llamafactory.hparams import get_infer_args return {"loaded": False, "status": "loading", "request_id": self._request_id}
args = {"model_name_or_path": model_name_or_path, "template": template, "infer_backend": infer_backend, "infer_dtype": infer_dtype}
if adapter_name_or_path:
args["adapter_name_or_path"] = adapter_name_or_path
args.update(kwargs)
model_args, generating_args = get_infer_args(args)
self._model = ChatModel(model_args)
self._tokenizer = self._model.tokenizer
self._generating_args = generating_args
self._loaded_at = time.time()
self._status = "ready"
return {"loaded": True, "status": "ready"}
except Exception as exc:
self._status = "error"
self._model = None
return {"loaded": False, "status": "error", "error": str(exc)}
def unload(self): def _load_worker(self) -> None:
with self._lock: """Build the ChatModel off the state lock so info() never blocks."""
if self._model is not None: model = None
try: tokenizer = None
del self._model generating_args: dict[str, Any] = {}
except Exception: error = ""
pass try:
if self._teardown_old:
self._release_model()
from llamafactory.chat import ChatModel
from llamafactory.hparams import get_infer_args
args = dict(self._load_args)
infer_result = get_infer_args(args)
model = ChatModel(args)
tokenizer = getattr(model, "tokenizer", None) or model.engine.tokenizer
generating_args = infer_result[-1]
if hasattr(generating_args, "__dataclass_fields__"):
generating_args = {
k: v for k, v in vars(generating_args).items() if not k.startswith("_")
}
else:
generating_args = dict(generating_args)
except Exception as exc: # noqa: BLE001 - surface load failure via status
error = str(exc)
with self._state_lock:
if error:
self._model = None self._model = None
self._tokenizer = None self._tokenizer = None
self._status = "idle" self._status = "error"
self._model_name = "" self._error = error
self._adapter_path = "" return
self._loaded_at = 0.0 if self._cancel_requested:
return {"unloaded": True} # Unload was requested while loading — drop the fresh model.
model = None
tokenizer = None
self._model = None
self._tokenizer = None
self._status = "idle"
return
self._model = model
self._tokenizer = tokenizer
self._generating_args = generating_args
self._loaded_at = time.time()
self._status = "ready"
def chat(self, messages, temperature=0.95, top_p=0.7, max_new_tokens=1024, do_sample=True, **kwargs): def _release_model(self) -> None:
with self._lock: with self._chat_lock:
if self._status != "ready" or self._model is None: with self._state_lock:
return {"error": "model not loaded", "response": ""} self._status = "unloading"
model = self._model
self._model = None
self._tokenizer = None
if model is not None:
try:
del model
except Exception: # noqa: BLE001 - best-effort teardown
pass
# 强制释放 PyTorch CUDA 缓存,真正归还 GPU 显存
try:
import gc
gc.collect()
import torch
if torch.cuda.is_available():
torch.cuda.empty_cache()
torch.cuda.synchronize()
except Exception: # noqa: BLE001 - teardown must not raise
pass
with self._state_lock:
self._status = "idle"
self._model_name = ""
self._adapter_path = ""
self._loaded_at = 0.0
self._error = ""
def unload(self) -> dict[str, Any]:
with self._state_lock:
if self._status == "loading":
# Ask the worker to tear down right after the load finishes.
self._cancel_requested = True
return {"unloaded": False, "status": "cancelling", "request_id": self._request_id}
was_ready = self._status == "ready"
if was_ready:
self._release_model()
else:
with self._state_lock:
self._model = None
self._tokenizer = None
self._status = "idle"
self._model_name = ""
self._adapter_path = ""
self._loaded_at = 0.0
self._error = ""
return {"unloaded": True, "status": "idle"}
def chat(self, messages, temperature=0.95, top_p=0.7, max_new_tokens=1024, do_sample=True, **kwargs) -> dict[str, Any]:
with self._chat_lock:
with self._state_lock:
if self._status == "loading":
return {
"error": f"model is still loading (request_id={self._request_id}); please retry",
"response": "",
}
if self._status == "error":
return {"error": f"model load failed: {self._error}", "response": ""}
if self._status != "ready" or self._model is None:
return {"error": "model not loaded", "response": ""}
try: try:
generate_kwargs = {**self._generating_args, "temperature": temperature, "top_p": top_p, "max_new_tokens": max_new_tokens, "do_sample": do_sample} generate_kwargs = {
"temperature": temperature,
"top_p": top_p,
"max_new_tokens": max_new_tokens,
"do_sample": do_sample,
}
generate_kwargs.update(kwargs) generate_kwargs.update(kwargs)
formatted = self._model.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) system = next((m["content"] for m in messages if m["role"] == "system"), None)
user_messages = [m for m in messages if m["role"] != "system"]
responses = [] responses = []
for response in self._model.stream_chat(formatted, generate_kwargs): for response in self._model.stream_chat(user_messages, system=system, **generate_kwargs):
responses.append(response) responses.append(response)
full_response = "".join(str(r) for r in responses) full_response = "".join(str(r) for r in responses)
return {"response": full_response} return {"response": full_response}
except Exception as exc: except Exception as exc: # noqa: BLE001 - return generation error to caller
return {"error": str(exc), "response": ""} return {"error": str(exc), "response": ""}
def chat_stream(self, messages, **kwargs): def chat_stream(self, messages, **kwargs) -> Iterator[str]:
with self._lock: with self._chat_lock:
if self._status != "ready" or self._model is None: with self._state_lock:
yield 'data: {"error": "model not loaded"}\n\n' if self._status == "loading":
return yield 'data: {"error": "model is still loading; please retry"}\n\n'
return
if self._status == "error":
yield 'data: {"error": "model load failed: ' + str(self._error) + '"}\n\n'
return
if self._status != "ready" or self._model is None:
yield 'data: {"error": "model not loaded"}\n\n'
return
try: try:
generate_kwargs = {**self._generating_args, **kwargs} generate_kwargs = {**kwargs}
formatted = self._model.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) system = next((m["content"] for m in messages if m["role"] == "system"), None)
for new_text in self._model.stream_chat(formatted, generate_kwargs): user_messages = [m for m in messages if m["role"] != "system"]
for new_text in self._model.stream_chat(user_messages, system=system, **generate_kwargs):
yield new_text yield new_text
except Exception as exc: except Exception as exc: # noqa: BLE001 - stream error as SSE event
yield 'data: {"error": "' + str(exc) + '"}\n\n' yield 'data: {"error": "' + str(exc) + '"}\n\n'
_inference_session = None _inference_session = None
def get_inference_session():
def get_inference_session() -> InferenceSession:
global _inference_session global _inference_session
if _inference_session is None: if _inference_session is None:
_inference_session = InferenceSession() _inference_session = InferenceSession()

View File

@@ -4,4 +4,9 @@ python-multipart>=0.0.9
pydantic>=2.7.0 pydantic>=2.7.0
python-dotenv>=1.0.1 python-dotenv>=1.0.1
httpx>=0.27.0 httpx>=0.27.0
llamafactory # 模型评测指标
sacrebleu>=2.4.0
rouge-score>=0.1.2
scikit-learn>=1.3.0
# LLaMA-Factory 训练引擎
llamafactory

View File

@@ -0,0 +1,146 @@
from __future__ import annotations
import sys
import time
import types
from typing import Any
import pytest
from compute.engines.llama_factory.inference import InferenceSession
# 模拟模型加载耗时,用于验证 load() 立即返回、info() 不阻塞
LOAD_DELAY = 0.2
class FakeChatModel:
def __init__(self, args: dict[str, Any]) -> None:
time.sleep(LOAD_DELAY)
self.tokenizer = object()
self.engine = types.SimpleNamespace(tokenizer=object())
self._output = "hello from model"
def stream_chat(self, *args, **kwargs):
for _ in range(1):
yield self._output
class FailingChatModel:
def __init__(self, args: dict[str, Any]) -> None:
time.sleep(LOAD_DELAY)
raise RuntimeError("boom: fake load failure")
def _get_infer_args(args: dict[str, Any]) -> list[Any]:
# 最后一个元素为 generating_argsworker 会转成 dict
return [None, None, {"temperature": 0.7}]
def _install_llamafactory(monkeypatch, chat_model: type) -> None:
llmf = types.ModuleType("llamafactory")
chat_mod = types.ModuleType("llamafactory.chat")
hparams_mod = types.ModuleType("llamafactory.hparams")
chat_mod.ChatModel = chat_model
hparams_mod.get_infer_args = _get_infer_args
llmf.chat = chat_mod
llmf.hparams = hparams_mod
monkeypatch.setitem(sys.modules, "llamafactory", llmf)
monkeypatch.setitem(sys.modules, "llamafactory.chat", chat_mod)
monkeypatch.setitem(sys.modules, "llamafactory.hparams", hparams_mod)
@pytest.fixture
def stub_llamafactory(monkeypatch) -> None:
_install_llamafactory(monkeypatch, FakeChatModel)
@pytest.fixture
def stub_failing_llamafactory(monkeypatch) -> None:
_install_llamafactory(monkeypatch, FailingChatModel)
def _wait_for_status(session: InferenceSession, status: str, timeout: float = 3.0) -> bool:
deadline = time.time() + timeout
while time.time() < deadline:
if session.info()["status"] == status:
return True
time.sleep(0.02)
return False
def test_load_returns_immediately_then_ready(stub_llamafactory) -> None:
session = InferenceSession()
started = time.time()
result = session.load("/models/qwen")
assert result["status"] == "loading"
assert result["loaded"] is False
assert result["request_id"]
# 在慢加载完成前就返回,且 info() 加载期间可响应
assert time.time() - started < LOAD_DELAY
assert session.info()["status"] == "loading"
assert _wait_for_status(session, "ready")
info = session.info()
assert info["loaded"] is True
assert info["status"] == "ready"
assert info["model_name"] == "/models/qwen"
def test_second_load_while_loading_deduped(stub_llamafactory) -> None:
session = InferenceSession()
r1 = session.load("/models/a")
r2 = session.load("/models/b")
assert r2["status"] == "loading"
assert r2["request_id"] == r1["request_id"]
assert _wait_for_status(session, "ready")
assert session.info()["status"] == "ready"
def test_load_error_surfaces_in_status(stub_failing_llamafactory) -> None:
session = InferenceSession()
session.load("/models/bad")
assert _wait_for_status(session, "error")
assert "boom" in session.info()["error"]
def test_unload_while_loading_cancels(stub_llamafactory) -> None:
session = InferenceSession()
session.load("/models/qwen")
result = session.unload()
assert result["status"] == "cancelling"
assert _wait_for_status(session, "idle")
def test_chat_while_loading_returns_loading_error(stub_llamafactory) -> None:
session = InferenceSession()
session.load("/models/qwen")
out = session.chat([{"role": "user", "content": "hi"}])
assert "still loading" in (out.get("error") or "")
assert _wait_for_status(session, "ready")
out = session.chat([{"role": "user", "content": "hi"}])
assert out.get("response") == "hello from model"
def test_chat_stream_while_loading_yields_error(stub_llamafactory) -> None:
session = InferenceSession()
session.load("/models/qwen")
chunks = list(session.chat_stream([{"role": "user", "content": "hi"}]))
assert any("still loading" in c for c in chunks)
def test_wait_until_loaded_blocks_until_ready(stub_llamafactory) -> None:
session = InferenceSession()
result = session.load("/models/qwen")
assert result["status"] == "loading"
# 同步等待后台加载线程完成
outcome = session.wait_until_loaded(timeout=3.0)
assert outcome["loaded"] is True
assert outcome["status"] == "ready"
def test_wait_until_loaded_reports_load_error(stub_failing_llamafactory) -> None:
session = InferenceSession()
session.load("/models/bad")
outcome = session.wait_until_loaded(timeout=3.0)
assert outcome["loaded"] is False
assert outcome["status"] == "error"
assert "boom" in outcome["error"]

View File

@@ -11,7 +11,7 @@ RUN pip install --upgrade pip -i https://pypi.tuna.tsinghua.edu.cn/simple \
&& pip install -r /tmp/requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple \ && pip install -r /tmp/requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple \
&& rm -f /tmp/requirements.txt && rm -f /tmp/requirements.txt
RUN python -c "import fastapi, uvicorn, psycopg, sqlalchemy, redis, jwt, passlib, httpx, alembic; print('backend dependency check ok')" RUN python -c "import fastapi, uvicorn, psycopg, psycopg_pool, sqlalchemy, redis, jwt, passlib, httpx, alembic; print('backend dependency check ok')"
RUN mkdir -p /opt/yg-ft/logs/backend /data/yg-ft \ RUN mkdir -p /opt/yg-ft/logs/backend /data/yg-ft \
&& chmod -R 0775 /opt/yg-ft /data/yg-ft && chmod -R 0775 /opt/yg-ft /data/yg-ft

View File

@@ -14,8 +14,8 @@ server {
proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 300s; proxy_read_timeout 900s;
proxy_send_timeout 300s; proxy_send_timeout 900s;
} }
location = /modelTF { location = /modelTF {
@@ -25,8 +25,8 @@ server {
proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 300s; proxy_read_timeout 900s;
proxy_send_timeout 300s; proxy_send_timeout 900s;
} }
location ~* \.(?:js|css|png|jpg|jpeg|gif|ico|svg|woff|woff2|ttf)$ { location ~* \.(?:js|css|png|jpg|jpeg|gif|ico|svg|woff|woff2|ttf)$ {

View File

@@ -0,0 +1,121 @@
# 模型评测功能总结
本项目(基于 LLaMA-Factory 的微调训练平台)包含 **4 套相对独立** 的模型评测能力,分别面向不同的使用场景:
| 能力 | 入口/目录 | 评测类型 | 打分方式 |
| --- | --- | --- | --- |
| 1. 学术 Benchmark 评测 | `llamafactory/eval/` | 选择题式基准(类 MMLU/C-Eval | 选项匹配 + few-shot |
| 2. 评估工作台 | `backend/app/api/v1/eval/` | 生成式问答(指令跟随) | BLEU / ROUGE / ExactMatch + 可选 LLM 评审 |
| 3. 平台评估系统 | `backend/app/api/v1/evaluation/` | 基于评估数据集的问答 | 判卷模型judge model打分05 分) |
| 4. 训练时验证评估 | `backend/app/services/task_runner.py` | 训练验证集 | loss 指标 |
下面分别说明。
---
## 1. 学术 Benchmark 评测LLaMA-Factory 原生)
面向标准学术选择题基准(如 MMLU、C-Eval 等),复用 LLaMA-Factory 原生的评测框架。
**核心文件**
- `llamafactory/eval/evaluator.py``Evaluator` 类 + `run_eval()` 入口
- `llamafactory/eval/template.py`:评测 prompt 模板(中/英,含 few-shot 示例构建)
- `llamafactory/hparams/evaluation_args.py``EvaluationArguments` 配置类
**工作流程**
1.`task`benchmark 名称加载数据集按科目subject拆分。
2. 每个样本构造 few-shot 提示词(`n_shot` 控制示例数,由 `lang` 决定中/英模板),将题干与候选选项拼入 prompt。
3. 调用模型推理得到预测,与标准答案比对,统计每个科目及整体的 `accuracy`
4. 结果写入 `save_dir`,打印各科目与平均准确率。
**关键参数(`EvaluationArguments`**
- `task`:基准数据集名
- `batch_size` / `n_shot` / `lang` / `save_dir` / `seed`
- `model_name_or_path``template``trust_remote_code` 等模型相关参数
> 该能力属于框架底层,本平台前端未直接提供操作入口,主要通过配置文件/脚本调用。
---
## 2. 评估工作台(生成式评测 + 指标计算)
后端路由位于 `backend/app/api/v1/eval/__init__.py`,前端称为「评估工作台」。**适用于评测模型的指令跟随与生成质量**,并支持 LLM 作为裁判LLM-as-a-Judge
**API 端点**
- `GET /evaluation/tasks`:列出评测任务(`frontend/src/api/evaluation.ts:listTasks`
- `POST /evaluation/run`:提交一次评测(`runEval`
- `GET /evaluation/report/{task_id}`:拉取评测报告(`getReport`
- `DELETE /evaluation/tasks/{task_id}`:删除任务(`deleteTask`
**评测流程(`run_eval`**
1. 通过 **LLaMA-Factory 数据管道**`get_dataset`) 加载数据集,支持 `subset` 与抽样(`eval_sample`)。
2.**原生 transformers** 加载模型在本地做生成推理(单进程顺序生成,便于展示样本)。
3. 计算客观指标(`compute_score`
- `BLEU`sacrebleu
- `ROUGE-1 / ROUGE-2 / ROUGE-L`rouge-score
- `Exact Match`
4. **可选 LLM 评审**judge当配置了 `judge_model` / `judge_api_base` / `judge_api_key` 时,调用 OpenAI 兼容接口对每条样本打分10 分制),并输出 4 个维度与理由:
- 核心事实正确性 `factual`
- 信息完整性 `completeness`
- 无幻觉 `no_hallucination`
- 格式合规性 `format`
- 综合分 `score` + `reason`
5. 任务状态持久化在后端 `eval_tasks.json`(支持 running/completed/failed/stopped前端轮询进度。
**前端页面**
- `frontend/src/views/evaluation/EvaluateTask.vue`:任务列表、创建评测对话框(选模型、数据集、指标、可选 judge 配置)
- `frontend/src/views/evaluation/EvaluateReport.vue`报告页展示综合得分、BLEU、ROUGE-L、各维度指标及「参考答案 vs 模型预测 vs LLM 评审」对比样例
---
## 3. 平台评估系统(基于评估数据集 + 判卷模型)
后端路由位于 `backend/app/api/v1/evaluation/__init__.py`,是平台业务层自研的评测体系。通过「评估数据集」组织题目,可一次性对 **多个被测模型 + 指定判卷模型** 进行批量评分。
**核心概念(数据模型 `backend/app/models/models.py`**
- `EvalDataset``models.py:131`):评估数据集,从项目问答对(`Question`/`Chunk`)中按 `question_type`mixed/fact/reasoning选题构建状态 `pending/running/completed/failed`
- `EvalResult``models.py:147`):单条评测结果,含 `judge_score`05 分)、`is_correct`true/false/partial`feedback``expected_answer` 等。
- `Task``models.py:184`):后台任务,`task_type="model-evaluation"`,记录进度与 `model_info`(存放平均分等汇总)。
**评测流程(`process_evaluation_task``backend/app/services/task_processor.py:336` 起)**
1. 加载评估数据集关联的题目,可选带入 `chunk` 上下文RAG 场景)。
2. 对每道题,先用 `build_eval_prompt` 组合「上下文 + 题目 + 参考答案」,调用 **判卷模型**`call_model`temperature=0.3)生成评分。
3. `parse_eval_result` 解析出 `score`(05)、`is_correct``feedback`,写入 `EvalResult`
4. 逐题提交进度(`completed_count` / `progress`),支持中途 `stopped`
5. 汇总:`avg_score = 总分/有效数 × 20`(换算百分制),`avg_score_5 = 总分/有效数`5 分制),存入 `task.model_info`。判定规则:得分 **≥3 视为正确**。
**特点**
- 判卷与被测模型解耦被测模型给出答案判卷模型judge独立评分降低自评偏差。
- 支持失败隔离:单题异常写入 `evaluation_status: failed` 记录而不中断整体任务。
---
## 4. 训练时验证评估
在微调训练任务执行期间,由 `backend/app/services/task_runner.py``do_eval` 触发:
- 在训练过程中对验证集validation set计算 `eval_loss`,用于监控过拟合。
- 结果回填到 `Task``loss_info` / `detail`,前端绘制 loss 曲线。
- 属于训练配套的轻量评估,不参与上述 13 的业务评测。
---
## 附属:前端评测相关页面
| 文件 | 作用 |
| --- | --- |
| `frontend/src/views/evaluation/EvaluateTask.vue` | 评估工作台:任务列表 + 创建评测 |
| `frontend/src/views/evaluation/EvaluateReport.vue` | 评估报告:指标卡 + 维度标签 + 对比样例 |
| `frontend/src/api/evaluation.ts` | 评估工作台接口封装 |
| 平台评估系统入口 | 评估数据集管理 + 评估任务model-evaluation创建与结果查看 |
---
## 小结
- **想要学术榜单式准确率** → 用能力 1LLaMA-Factory `eval/`)。
- **想要开放式生成质量BLEU/ROUGE + LLM 评审)** → 用能力 2评估工作台 `/evaluation/run`)。
- **想要基于自有问答数据、用判卷模型批量打分** → 用能力 3平台评估系统 `model-evaluation` 任务)。
- **训练过程监控** → 能力 4`do_eval` 验证集 loss
三种业务评测1/2/3相互独立可并存于同一平台数据模型`EvalDataset`/`EvalResult`/`Task`)主要服务于能力 3而能力 2 使用独立的 `eval_tasks.json` 文件持久化。

View File

@@ -18,12 +18,12 @@ const auth = useAuthStore()
*/ */
let hiddenAt = 0 let hiddenAt = 0
function handleVisibility() { async function handleVisibility() {
if (document.hidden) { if (document.hidden) {
hiddenAt = Date.now() hiddenAt = Date.now()
} else { } else {
if (hiddenAt > 0 && Date.now() - hiddenAt >= SESSION_TIMEOUT) { if (hiddenAt > 0 && Date.now() - hiddenAt >= SESSION_TIMEOUT) {
auth.logout() await auth.logout()
ElMessage.warning('登录已过期,请重新登录') ElMessage.warning('登录已过期,请重新登录')
router.push('/login') router.push('/login')
} }

View File

@@ -1,6 +1,8 @@
import { get, post, del } from '../request' import { get, post, del } from '../request'
import type { CompareTask, CompareModelRef } from '@/types' import type { CompareTask, CompareModelRef } from '@/types'
const INFERENCE_START_TIMEOUT_MS = 15 * 60 * 1000
/** 推理/对比任务列表 */ /** 推理/对比任务列表 */
export const getCompareList = () => get<CompareTask[]>('/model-compare') export const getCompareList = () => get<CompareTask[]>('/model-compare')
@@ -12,7 +14,7 @@ export const createCompare = (data: Partial<CompareTask>) =>
post<{ id: string | number }>('/model-compare', data) post<{ id: string | number }>('/model-compare', data)
/** 删除任务 */ /** 删除任务 */
export const deleteCompare = (id: string | number) => del(`/model-compare/${id}`) export const deleteCompare = (id: string | number) => del(`/model-compare/${id}`, undefined, { timeout: 60_000 })
/** 更新任务加载状态 */ /** 更新任务加载状态 */
export const updateLoadStatus = (id: string | number, load_status: any) => export const updateLoadStatus = (id: string | number, load_status: any) =>
@@ -34,7 +36,8 @@ export const stopModelByPid = (pid: number) =>
post('/model-compare/stop-by-pid', { pid }) post('/model-compare/stop-by-pid', { pid })
/** 加载任务 */ /** 加载任务 */
export const loadCompare = (id: string | number) => post(`/model-compare/${id}/load`) export const loadCompare = (id: string | number) =>
post(`/model-compare/${id}/load`, undefined, { timeout: INFERENCE_START_TIMEOUT_MS })
/** 卸载任务 */ /** 卸载任务 */
export const unloadCompare = (id: string | number) => post(`/model-compare/${id}/unload`) export const unloadCompare = (id: string | number) => post(`/model-compare/${id}/unload`)
@@ -64,6 +67,31 @@ export const streamChat = async (data: any): Promise<any> => {
} }
} }
/** 真实流式对话 — 使用 fetch 调用后端 SSE 端点,返回 Response 供 ReadableStream 消费 */
export const streamChatReal = (data: any): Promise<Response> => {
const messages = data.messages || []
if (!messages.length && data.user_question) {
if (data.system_prompt) {
messages.push({ role: 'system', content: data.system_prompt })
}
messages.push({ role: 'user', content: data.user_question })
}
return fetch('/modelTF/model-compare/stream-chat', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
messages,
temperature: data.temperature ?? 0.7,
top_p: data.top_p ?? 0.95,
max_tokens: data.max_tokens ?? 2048,
// 透传 task_id/node_id让后端按 load_status 路由到真正加载了模型的算力节点,
// 避免在多节点时回退到“第一个在线节点”导致连接失败
task_id: data.task_id,
node_id: data.node_id,
}),
})
}
/** 非流式对话(按端口代理) */ /** 非流式对话(按端口代理) */
export const chatWithPort = (data: any) => post('/model-compare/chat-with-port', data) export const chatWithPort = (data: any) => post('/model-compare/chat-with-port', data)
@@ -73,8 +101,8 @@ export const batchChat = (data: any) => post('/model-chat/batch', data)
/** 本地 transformers 模型对话 */ /** 本地 transformers 模型对话 */
export const localChat = (data: any) => post('/model-chat/local/chat', data) export const localChat = (data: any) => post('/model-chat/local/chat', data)
/** 预加载本地模型 */ /** 预加载本地模型(模型加载耗时长,超时 15 分钟) */
export const preloadLocalModel = (data: any) => post('/model-chat/local/preload', data) export const preloadLocalModel = (data: any) => post('/model-chat/local/preload', data, { timeout: INFERENCE_START_TIMEOUT_MS })
/** 预加载已训练模型 */ /** 预加载已训练模型(超时 15 分钟) */
export const preloadTrainedModel = (data: any) => post('/model-chat/trained/preload', data) export const preloadTrainedModel = (data: any) => post('/model-chat/trained/preload', data, { timeout: INFERENCE_START_TIMEOUT_MS })

View File

@@ -1,4 +1,4 @@
import { get, post, put } from '../request' import { del, get, post, put } from '../request'
export interface ComputeNode { export interface ComputeNode {
id: string id: string
@@ -95,6 +95,9 @@ export const createComputeNode = (data: ComputeNodePayload) =>
export const updateComputeNode = (id: string, data: Partial<ComputeNode>) => export const updateComputeNode = (id: string, data: Partial<ComputeNode>) =>
put<ComputeNode>(`/compute/nodes/${id}`, data) put<ComputeNode>(`/compute/nodes/${id}`, data)
export const deleteComputeNode = (id: string) =>
del<{ deleted: string }>(`/compute/nodes/${id}`)
export const testComputeNode = (id: string) => export const testComputeNode = (id: string) =>
post<{ node_id: string; success: boolean; latency_ms: number; gpu_count: number; error?: string }>(`/compute/nodes/${id}/test-connection`) post<{ node_id: string; success: boolean; latency_ms: number; gpu_count: number; error?: string }>(`/compute/nodes/${id}/test-connection`)

View File

@@ -29,6 +29,7 @@ export const uploadDatasetFiles = (datasetId: string | number, files: File[]) =>
files.forEach((f) => formData.append('files', f)) files.forEach((f) => formData.append('files', f))
return post(`/dataset-manage/upload/${datasetId}`, formData, { return post(`/dataset-manage/upload/${datasetId}`, formData, {
headers: { 'Content-Type': 'multipart/form-data' }, headers: { 'Content-Type': 'multipart/form-data' },
timeout: 120000,
}) })
} }

View File

@@ -1,6 +1,16 @@
import { get, post, put, del } from '../request' import { get, post, put, del } from '../request'
import type { FineTuneStartPayload, FineTuneTask, TrainingProgress, LogContent } from '@/types' import type { FineTuneStartPayload, FineTuneTask, TrainingProgress, LogContent } from '@/types'
export interface FineTuneMetricPoint {
step: number
epoch?: number | null
loss?: number | null
grad_norm?: number | null
learning_rate?: number | null
raw?: string
create_time?: string
}
export interface TrainingDiagnostic { export interface TrainingDiagnostic {
level: string level: string
title: string title: string
@@ -80,6 +90,10 @@ export const getFineTuneLogs = (
params: { tail_lines?: number; offset?: number; limit?: number } = {}, params: { tail_lines?: number; offset?: number; limit?: number } = {},
) => get<LogContent & { job_id?: string; source?: string }>(`/fine-tune/${id}/logs`, params) ) => get<LogContent & { job_id?: string; source?: string }>(`/fine-tune/${id}/logs`, params)
/** 获取训练指标曲线数据 */
export const getFineTuneMetrics = (id: string | number) =>
get<FineTuneMetricPoint[]>(`/fine-tune/${id}/metrics`)
/** 启动 TensorBoard */ /** 启动 TensorBoard */
export const startTensorboard = () => post('/fine-tune/tensorboard/start') export const startTensorboard = () => post('/fine-tune/tensorboard/start')

View File

@@ -88,10 +88,14 @@ export const updateModelPurpose = (id: string | number, purpose: string) =>
/** 合并 LoRA 权重 */ /** 合并 LoRA 权重 */
export const mergeModel = (data: { export const mergeModel = (data: {
trained_model_id?: string | number
model_name: string model_name: string
train_method: string train_method: string
base_model_path: string base_model_path: string
}) => post('/model-manage/merge', data) adapter_path?: string
compute_node_id?: string
output_model_name?: string
}) => post('/model-manage/merge', data, { timeout: 15 * 60 * 1000 })
/** 导出已训练模型权重 */ /** 导出已训练模型权重 */
export const exportModelUrl = (modelName: string) => export const exportModelUrl = (modelName: string) =>

View File

@@ -53,3 +53,6 @@ export const updateProjectMember = (id: string, userId: string, role: string) =>
/** 移除成员 */ /** 移除成员 */
export const removeProjectMember = (id: string, userId: string) => export const removeProjectMember = (id: string, userId: string) =>
del(`/projects/${id}/members/${userId}`) del(`/projects/${id}/members/${userId}`)
/** 删除项目 */
export const deleteProject = (id: string) => del(`/projects/${id}`)

View File

@@ -8,6 +8,8 @@ import type {
UpdateUserAccessPayload, UpdateUserAccessPayload,
} from '@/types' } from '@/types'
export type { SystemUser } from '@/types'
/** 系统信息CPU/内存/磁盘/GPU/网络/系统) */ /** 系统信息CPU/内存/磁盘/GPU/网络/系统) */
export const getSystemInfo = () => get<SystemInfo>('/system-info') export const getSystemInfo = () => get<SystemInfo>('/system-info')
@@ -18,6 +20,10 @@ export const getHealth = () => get<HealthMetrics>('/health')
export const login = (username: string, password: string) => export const login = (username: string, password: string) =>
post<LoginResponse>('/login', { username, password }) post<LoginResponse>('/login', { username, password })
/** 登出 */
export const logout = (sessionId?: string) =>
post('/logout', { session_id: sessionId || '' })
/** 用户列表 */ /** 用户列表 */
export const getUsers = () => get<SystemUser[]>('/users') export const getUsers = () => get<SystemUser[]>('/users')

View File

@@ -1,4 +1,4 @@
import { get, post, put } from '../request' import { del, get, post, put } from '../request'
export interface Tenant { export interface Tenant {
id: string id: string
@@ -25,6 +25,9 @@ export const createTenant = (payload: Partial<Tenant>) =>
export const updateTenant = (id: string, payload: Partial<Tenant>) => export const updateTenant = (id: string, payload: Partial<Tenant>) =>
put<Tenant>(`/tenants/${id}`, payload) put<Tenant>(`/tenants/${id}`, payload)
/** 删除租户 */
export const deleteTenant = (id: string) => del(`/tenants/${id}`)
/** 设置租户配额 */ /** 设置租户配额 */
export const setTenantQuota = (id: string, quota: Record<string, unknown>) => export const setTenantQuota = (id: string, quota: Record<string, unknown>) =>
put<Tenant>(`/tenants/${id}/quota`, { quota }) put<Tenant>(`/tenants/${id}/quota`, { quota })

View File

@@ -14,7 +14,7 @@ export interface ApiResult<T = any> {
const service: AxiosInstance = axios.create({ const service: AxiosInstance = axios.create({
// Use a relative path; Vite proxies /modelTF to http://localhost:17861 in local development. // Use a relative path; Vite proxies /modelTF to http://localhost:17861 in local development.
baseURL: '/modelTF', baseURL: '/modelTF',
timeout: 30000, timeout: 120000,
}) })
/** /**

View File

@@ -2,6 +2,7 @@
import { computed, ref, watch } from 'vue' import { computed, ref, watch } from 'vue'
import { ElMessage } from 'element-plus' import { ElMessage } from 'element-plus'
import { getAcl, setAcl, type AclEntry } from '@/api/modules/acl' import { getAcl, setAcl, type AclEntry } from '@/api/modules/acl'
import { getUsers, type SystemUser } from '@/api/modules/system'
const props = defineProps<{ const props = defineProps<{
modelValue: boolean modelValue: boolean
@@ -15,13 +16,20 @@ const visible = computed({
set: (v) => emit('update:modelValue', v), set: (v) => emit('update:modelValue', v),
}) })
const entries = ref<AclEntry[]>([]) const entries = ref<AclEntry[]>([])
const users = ref<SystemUser[]>([])
const loading = ref(false) const loading = ref(false)
const ALL_PERMS = ['read', 'write', 'execute', 'download', 'delete', 'share'] const ALL_PERMS = ['read', 'write', 'execute', 'download', 'delete', 'share']
const PROJECT_ROLES = ['member', 'admin', 'viewer']
async function load() { async function load() {
loading.value = true loading.value = true
try { try {
entries.value = await getAcl(props.resourceType, props.resourceId) const [acl, us] = await Promise.all([
getAcl(props.resourceType, props.resourceId),
getUsers().catch(() => [] as SystemUser[]),
])
entries.value = acl
users.value = us
} finally { } finally {
loading.value = false loading.value = false
} }
@@ -53,7 +61,12 @@ async function save() {
<el-option label="用户" value="user" /> <el-option label="用户" value="user" />
<el-option label="项目角色" value="project_role" /> <el-option label="项目角色" value="project_role" />
</el-select> </el-select>
<el-input v-model="entry.subject_id" placeholder="subject ID" style="width: 200px" /> <el-select v-if="entry.subject_type === 'user'" v-model="entry.subject_id" placeholder="选择用户" style="width: 200px" filterable>
<el-option v-for="u in users" :key="u.id" :label="`${u.username} (${u.id})`" :value="u.id" />
</el-select>
<el-select v-else v-model="entry.subject_id" placeholder="选择角色" style="width: 200px">
<el-option v-for="r in PROJECT_ROLES" :key="r" :label="r" :value="r" />
</el-select>
<el-checkbox-group v-model="entry.permissions"> <el-checkbox-group v-model="entry.permissions">
<el-checkbox v-for="p in ALL_PERMS" :key="p" :value="p">{{ p }}</el-checkbox> <el-checkbox v-for="p in ALL_PERMS" :key="p" :value="p">{{ p }}</el-checkbox>
</el-checkbox-group> </el-checkbox-group>

View File

@@ -132,8 +132,8 @@ async function handleSelect(key: string) {
} }
} }
function handleLogout() { async function handleLogout() {
auth.logout() await auth.logout()
router.push('/login') router.push('/login')
} }
</script> </script>

View File

@@ -1,5 +1,5 @@
import { ref } from 'vue' import { ref } from 'vue'
import { streamChat } from '@/api/modules/compare' import { streamChat, streamChatReal } from '@/api/modules/compare'
export interface StreamMessage { export interface StreamMessage {
/** 用户问题 */ /** 用户问题 */
@@ -20,6 +20,11 @@ export interface StreamMessage {
error?: string error?: string
} }
export interface SendOptions {
/** 是否使用 mock 模式(默认 true向后兼容 */
useMock?: boolean
}
/** /**
* 流式对话 composable * 流式对话 composable
* 移植自原 model-chat.html * 移植自原 model-chat.html
@@ -39,6 +44,24 @@ export function useStreamChat() {
}) })
const loading = ref(false) const loading = ref(false)
/** 从 SSE 帧中提取错误信息(后端/计算节点错误以 data: {"error": "..."} 形式下发) */
function extractSseError(buffer: string): string | null {
const trimmed = buffer.trim()
if (!trimmed.startsWith('data: ')) return null
const lines = trimmed.split(/\r?\n/)
for (let i = lines.length - 1; i >= 0; i--) {
const line = lines[i].trim()
if (!line.startsWith('data: ')) continue
try {
const obj = JSON.parse(line.slice(6))
if (obj && typeof obj.error === 'string' && obj.error) return obj.error
} catch {
/* 非 JSON 的 data 行忽略 */
}
}
return trimmed
}
/** 从内容中解析 think 标签 */ /** 从内容中解析 think 标签 */
function parseContent(content: string) { function parseContent(content: string) {
const thinkRegex = /<think>([\s\S]*?)(<\/think>)?/g const thinkRegex = /<think>([\s\S]*?)(<\/think>)?/g
@@ -65,8 +88,10 @@ export function useStreamChat() {
/** /**
* 发起流式对话 * 发起流式对话
* @param payload 后端请求体 { port, model_name, model_path, system_prompt, user_question, ... } * @param payload 后端请求体 { port, model_name, model_path, system_prompt, user_question, ... }
* @param options 可选配置 { useMock?: boolean }
*/ */
async function send(payload: any) { async function send(payload: any, options?: SendOptions) {
const useMock = options?.useMock ?? true
loading.value = true loading.value = true
message.value = { message.value = {
question: payload.user_question || '', question: payload.user_question || '',
@@ -82,7 +107,10 @@ export function useStreamChat() {
const UPDATE_INTERVAL = 50 // 50ms 节流 const UPDATE_INTERVAL = 50 // 50ms 节流
try { try {
const response = await streamChat(payload) const response = useMock
? await streamChat(payload)
: await streamChatReal(payload)
if (!response.ok) { if (!response.ok) {
throw new Error(`HTTP ${response.status}`) throw new Error(`HTTP ${response.status}`)
} }
@@ -111,6 +139,16 @@ export function useStreamChat() {
} }
// 最终更新 // 最终更新
// 若整段响应是 SSE 错误帧,提取 error 字段以干净文案展示
const sseError = extractSseError(buffer)
if (sseError) {
message.value.isThinking = false
message.value.isStreaming = false
message.value.done = true
message.value.error = sseError
message.value.displayContent = sseError
return
}
const parsed = parseContent(buffer) const parsed = parseContent(buffer)
message.value.thinkContent = parsed.think message.value.thinkContent = parsed.think
message.value.displayContent = parsed.display message.value.displayContent = parsed.display

View File

@@ -377,7 +377,7 @@ router.beforeEach((to, _from, next) => {
} }
if (!auth.isLoggedIn) { if (!auth.isLoggedIn) {
auth.logout() auth.logout() // fire-and-forget无需阻塞跳转
next({ name: 'login' }) next({ name: 'login' })
return return
} }

View File

@@ -1,9 +1,10 @@
import { defineStore } from 'pinia' import { defineStore } from 'pinia'
import { ref, computed } from 'vue' import { ref, computed } from 'vue'
import { login as loginApi } from '@/api/modules/system' import { login as loginApi, logout as logoutApi } from '@/api/modules/system'
import type { PermissionCode, SystemUser } from '@/types' import type { PermissionCode, SystemUser } from '@/types'
const USER_STORAGE_KEY = 'currentUser' const USER_STORAGE_KEY = 'currentUser'
const SESSION_STORAGE_KEY = 'sessionId'
const allPermissions: PermissionCode[] = [ const allPermissions: PermissionCode[] = [
'dashboard', 'dashboard',
@@ -29,20 +30,6 @@ function restoreUser(): SystemUser | null {
localStorage.removeItem(USER_STORAGE_KEY) localStorage.removeItem(USER_STORAGE_KEY)
} }
} }
// 兼容改造前已经登录的 admin 会话。
if (localStorage.getItem('username') === 'admin') {
return {
id: 'USR-0001',
username: 'admin',
display_name: '系统管理员',
role: 'admin',
status: 'active',
permissions: allPermissions,
create_time: '2026-01-01T08:00:00+08:00',
protected: true,
}
}
return null return null
} }
@@ -69,6 +56,9 @@ export const useAuthStore = defineStore('auth', () => {
currentUser.value = response.user currentUser.value = response.user
localStorage.setItem('username', response.user.username) localStorage.setItem('username', response.user.username)
localStorage.setItem(USER_STORAGE_KEY, JSON.stringify(response.user)) localStorage.setItem(USER_STORAGE_KEY, JSON.stringify(response.user))
if (response.session_id) {
localStorage.setItem(SESSION_STORAGE_KEY, response.session_id)
}
} }
/** 检查当前账号是否拥有指定模块权限。 */ /** 检查当前账号是否拥有指定模块权限。 */
@@ -78,10 +68,15 @@ export const useAuthStore = defineStore('auth', () => {
} }
/** 退出 */ /** 退出 */
function logout() { async function logout() {
const sessionId = localStorage.getItem(SESSION_STORAGE_KEY)
if (sessionId) {
try { await logoutApi(sessionId) } catch { /* 静默 */ }
}
currentUser.value = null currentUser.value = null
localStorage.removeItem('username') localStorage.removeItem('username')
localStorage.removeItem(USER_STORAGE_KEY) localStorage.removeItem(USER_STORAGE_KEY)
localStorage.removeItem(SESSION_STORAGE_KEY)
} }
return { return {

View File

@@ -34,6 +34,10 @@ export interface TrainedModel {
name: string name: string
train_methods?: TrainMethod[] train_methods?: TrainMethod[]
base_model_path?: string base_model_path?: string
artifact_dir?: string
adapter_path?: string
compute_node_id?: string
compute_node_name?: string
create_time?: string create_time?: string
merged?: boolean merged?: boolean
merging?: boolean merging?: boolean
@@ -132,6 +136,7 @@ export interface FineTuneTask {
train_dataset_id?: number | string train_dataset_id?: number | string
auto_merge?: boolean auto_merge?: boolean
output_model_name?: string output_model_name?: string
compute_node_id?: string
gpus?: number[] gpus?: number[]
batch_size?: number batch_size?: number
learning_rate?: number learning_rate?: number
@@ -211,6 +216,9 @@ export interface LoadedModel {
status?: string status?: string
pid?: number pid?: number
port?: number port?: number
node_id?: string
node_name?: string
error?: string
} }
export interface CompareTask { export interface CompareTask {
@@ -229,6 +237,8 @@ export interface CompareModelRef {
model_name: string model_name: string
model_path: string model_path: string
gpu_id: number gpu_id: number
node_id?: string
node_name?: string
source?: string source?: string
port?: number port?: number
} }
@@ -244,7 +254,9 @@ export interface EvalTask {
model_name?: string model_name?: string
model_id?: number | string model_id?: number | string
dataset?: string dataset?: string
dataset_id?: number | string
metric?: string metric?: string
metric_label?: string
score?: number score?: number
status?: string status?: string
create_time?: string create_time?: string
@@ -271,6 +283,7 @@ export interface StartEvalPayload {
eval_type: EvalType eval_type: EvalType
model_id: string | number model_id: string | number
gpu_id: string | number gpu_id: string | number
compute_node_id?: string
dataset_id: string | number dataset_id: string | number
dimension_id: string | number dimension_id: string | number
data_source: 'dataset' | 'inference' data_source: 'dataset' | 'inference'
@@ -355,16 +368,16 @@ export interface GpuInfo {
power_w: number power_w: number
id?: number id?: number
uuid?: string uuid?: string
status?: 'idle' | 'busy' | 'warning' | 'offline' status?: 'idle' | 'busy' | 'reserved' | 'warning' | 'offline'
memory_percent?: number memory_percent?: number
power_limit_w?: number power_limit_w?: number
processes?: GpuProcess[] processes?: GpuProcess[]
fan_speed?: number fan_speed?: number
clock_mhz?: number clock_mhz?: number
driver_version?: string
node_id?: string node_id?: string
node_code?: string node_code?: string
node_name?: string node_name?: string
driver_version?: string
} }
export interface SystemInfo { export interface SystemInfo {
@@ -446,6 +459,7 @@ export interface SystemUser {
export interface LoginResponse { export interface LoginResponse {
token: string token: string
user: SystemUser user: SystemUser
session_id?: string
} }
export interface CreateUserPayload { export interface CreateUserPayload {

View File

@@ -3,7 +3,8 @@ import { onMounted, reactive, ref } from 'vue'
import { ElMessage } from 'element-plus' import { ElMessage } from 'element-plus'
import DataTablePage from '@/components/DataTablePage.vue' import DataTablePage from '@/components/DataTablePage.vue'
import { getApprovalInstances, decideApproval, type ApprovalInstance } from '@/api/modules/approval' import { getApprovalInstances, decideApproval, type ApprovalInstance } from '@/api/modules/approval'
import { getUsers, type SystemUser } from '@/api/modules/system' import { getUsers } from '@/api/modules/system'
import type { SystemUser } from '@/types'
const loading = ref(false) const loading = ref(false)
const instances = ref<ApprovalInstance[]>([]) const instances = ref<ApprovalInstance[]>([])
@@ -48,6 +49,10 @@ function openDecide(inst: ApprovalInstance) {
showDecide.value = true showDecide.value = true
} }
function asApprovalInstance(row: unknown): ApprovalInstance {
return row as ApprovalInstance
}
async function submitDecision() { async function submitDecision() {
if (!current.value) return if (!current.value) return
if (!decision.value.approver_id) { if (!decision.value.approver_id) {
@@ -72,7 +77,7 @@ onMounted(() => {
<template> <template>
<div class="page"> <div class="page">
<DataTablePage title="审批实例" :data="instances" :loading="loading" searchable search-fields="resource_type,resource_id"> <DataTablePage title="审批实例" :data="instances" :loading="loading" searchable :search-fields="['resource_type', 'resource_id']">
<template #toolbar-extra> <template #toolbar-extra>
<el-select v-model="statusFilter" placeholder="状态" clearable style="width: 140px" @change="load"> <el-select v-model="statusFilter" placeholder="状态" clearable style="width: 140px" @change="load">
<el-option v-for="s in statusOptions" :key="s.value" :label="s.label" :value="s.value" /> <el-option v-for="s in statusOptions" :key="s.value" :label="s.label" :value="s.value" />
@@ -82,14 +87,14 @@ onMounted(() => {
<el-table-column prop="resource_type" label="资源类型" min-width="120" /> <el-table-column prop="resource_type" label="资源类型" min-width="120" />
<el-table-column prop="resource_id" label="资源 ID" min-width="160" show-overflow-tooltip /> <el-table-column prop="resource_id" label="资源 ID" min-width="160" show-overflow-tooltip />
<el-table-column prop="applicant_id" label="申请人" min-width="120"> <el-table-column prop="applicant_id" label="申请人" min-width="120">
<template #default="{ row }">{{ userName(row.applicant_id) }}</template> <template #default="{ row }">{{ userName(asApprovalInstance(row).applicant_id) }}</template>
</el-table-column> </el-table-column>
<el-table-column prop="status" label="状态" min-width="100" /> <el-table-column prop="status" label="状态" min-width="100" />
<el-table-column prop="current_step" label="当前步骤" min-width="100" /> <el-table-column prop="current_step" label="当前步骤" min-width="100" />
<el-table-column prop="create_time" label="创建时间" min-width="180" /> <el-table-column prop="create_time" label="创建时间" min-width="180" />
</template> </template>
<template #actions="{ row }"> <template #actions="{ row }">
<el-button v-if="row.status === 'pending'" link type="primary" @click="openDecide(row)">审批</el-button> <el-button v-if="asApprovalInstance(row).status === 'pending'" link type="primary" @click="openDecide(asApprovalInstance(row))">审批</el-button>
</template> </template>
</DataTablePage> </DataTablePage>
<el-dialog v-model="showDecide" title="审批决策" width="480px"> <el-dialog v-model="showDecide" title="审批决策" width="480px">

View File

@@ -1,12 +1,12 @@
<script setup lang="ts"> <script setup lang="ts">
import { computed, onMounted, onUnmounted, reactive, ref, watch } from 'vue' import { computed, onMounted, onUnmounted, reactive, ref, watch } from 'vue'
import { useRoute, useRouter } from 'vue-router' import { useRoute, useRouter } from 'vue-router'
import { ElMessage } from 'element-plus' import { ElMessage, ElMessageBox } from 'element-plus'
import { import {
checkNodeReplicaDrift, checkNodeReplicaDrift,
createComputeNode, createComputeNode,
deleteComputeNode,
disableComputeNode, disableComputeNode,
drainComputeNode,
enableComputeNode, enableComputeNode,
getComputeGpus, getComputeGpus,
getComputeNodes, getComputeNodes,
@@ -129,11 +129,10 @@ async function changeTab(name: string | number) {
await router.replace({ path: '/compute', query: { tab: String(name) } }) await router.replace({ path: '/compute', query: { tab: String(name) } })
} }
async function handleNodeAction(action: 'enable' | 'disable' | 'drain' | 'test', node: ComputeNode) { async function handleNodeAction(action: 'enable' | 'disable' | 'test', node: ComputeNode) {
const nodeId = String(node.id) const nodeId = String(node.id)
if (action === 'enable') await enableComputeNode(nodeId) if (action === 'enable') await enableComputeNode(nodeId)
if (action === 'disable') await disableComputeNode(nodeId) if (action === 'disable') await disableComputeNode(nodeId)
if (action === 'drain') await drainComputeNode(nodeId)
if (action === 'test') { if (action === 'test') {
const result = await testComputeNode(nodeId) const result = await testComputeNode(nodeId)
if (result.success) { if (result.success) {
@@ -145,6 +144,27 @@ async function handleNodeAction(action: 'enable' | 'disable' | 'drain' | 'test',
await load() await load()
} }
async function handleDeleteNode(node: ComputeNode) {
try {
await ElMessageBox.confirm(
`确定删除算力节点「${node.name || node.code}」吗?节点删除后,其 GPU 设备和资源副本记录也会一并移除。`,
'删除算力节点',
{ type: 'warning', confirmButtonText: '删除', cancelButtonText: '取消' },
)
} catch {
return
}
try {
await deleteComputeNode(String(node.id))
ElMessage.success('算力节点已删除')
if (selectedNodeId.value === node.id) selectedNodeId.value = ''
await load({ showButtonLoading: true })
} catch (err: any) {
const message = err?.response?.data?.detail?.message || err?.response?.data?.message || '删除算力节点失败'
ElMessage.error(message)
}
}
async function handleReplicaDriftCheck() { async function handleReplicaDriftCheck() {
if (!selectedNodeId.value) return if (!selectedNodeId.value) return
checkingReplicas.value = true checkingReplicas.value = true
@@ -355,7 +375,7 @@ onUnmounted(() => {
<el-button size="small" @click="handleNodeAction('test', asComputeNode(row))">测试</el-button> <el-button size="small" @click="handleNodeAction('test', asComputeNode(row))">测试</el-button>
<el-button v-if="row.enabled" size="small" @click="handleNodeAction('disable', asComputeNode(row))">停用</el-button> <el-button v-if="row.enabled" size="small" @click="handleNodeAction('disable', asComputeNode(row))">停用</el-button>
<el-button v-else size="small" type="primary" @click="handleNodeAction('enable', asComputeNode(row))">启用</el-button> <el-button v-else size="small" type="primary" @click="handleNodeAction('enable', asComputeNode(row))">启用</el-button>
<el-button size="small" type="warning" plain @click="handleNodeAction('drain', asComputeNode(row))">维护</el-button> <el-button size="small" type="danger" plain @click="handleDeleteNode(asComputeNode(row))">删除</el-button>
</template> </template>
</el-table-column> </el-table-column>
</el-table> </el-table>

View File

@@ -66,8 +66,12 @@ const operationDistribution = ref<{ name: string; value: number }[]>([])
const serviceIcon: Record<string, string> = { const serviceIcon: Record<string, string> = {
'模型推理': 'fa-cube', '模型推理': 'fa-cube',
'模型微调': 'fa-sliders', '模型微调': 'fa-sliders',
'模型训练': 'fa-sliders',
'模型评测': 'fa-bar-chart', '模型评测': 'fa-bar-chart',
'模型管理': 'fa-cubes',
'数据集管理': 'fa-file-text',
'数据处理': 'fa-filter', '数据处理': 'fa-filter',
'数据类型转换': 'fa-exchange',
} }
const roleLabel: Record<string, string> = { const roleLabel: Record<string, string> = {
admin: '超级管理员', admin: '超级管理员',
@@ -115,7 +119,7 @@ const chartOption = computed<EChartsOption>(() => ({
borderWidth: 0, borderWidth: 0,
padding: [10, 12], padding: [10, 12],
textStyle: { color: '#ffffff', fontSize: 12 }, textStyle: { color: '#ffffff', fontSize: 12 },
valueFormatter: (value) => `${value}`, valueFormatter: (value) => String(value ?? ''),
}, },
xAxis: { xAxis: {
type: 'category', type: 'category',
@@ -179,20 +183,30 @@ const chartOption = computed<EChartsOption>(() => ({
], ],
})) }))
// 模块固定配色,保证每个模块颜色不同 // 模块固定配色,按顺序循环分配颜色(与后端 OP_ORDER 一致:数据处理/模型训练/模型评测/模型推理)
const OPERATION_COLORS = ['#4f46e5', '#10b981', '#f59e0b', '#3b82f6', '#ec4899', '#8b5cf6', '#ef4444', '#14b8a6'] const OPERATION_COLORS = ['#4f46e5', '#10b981', '#f59e0b', '#3b82f6']
const operationChartOption = computed<EChartsOption>(() => { const operationChartOption = computed<EChartsOption>(() => {
const items = operationDistribution.value const items = operationDistribution.value
const total = items.reduce((s, d) => s + (d.value || 0), 0) const total = items.reduce((s, d) => s + (d.value || 0), 0)
// 完全没有操作数据时,用等分灰色占位扇区,保证 6 个模块都可见 // 按数据项顺序显式分配颜色,避免依赖 name 匹配或全局 color 数组;
const data = // value=0 的项给一个极小值0.001)让扇区可见,从而显示各自颜色,
total > 0 // 但占比几乎为 0 不影响有数据项的百分比展示。
? items.map((d) => ({ value: d.value || 0, name: d.name })) const data = items.map((d, idx) => {
: items.map((d) => ({ value: 1, name: d.name, itemStyle: { color: '#e2e8f0' } })) const raw = d.value || 0
return {
value: total > 0 ? (raw > 0 ? raw : 0.001) : 1,
name: d.name,
itemStyle: {
color: OPERATION_COLORS[idx % OPERATION_COLORS.length] || '#94a3b8',
borderRadius: 6,
borderColor: '#fff',
borderWidth: 2,
},
}
})
return { return {
animationDuration: 500, animationDuration: 500,
tooltip: { trigger: 'item', formatter: '{b}: {c} ({d}%)' }, tooltip: { trigger: 'item', formatter: '{b}: {c} ({d}%)' },
color: OPERATION_COLORS,
legend: { legend: {
type: 'scroll', type: 'scroll',
bottom: 0, bottom: 0,
@@ -207,11 +221,6 @@ const operationChartOption = computed<EChartsOption>(() => {
radius: ['38%', '60%'], radius: ['38%', '60%'],
center: ['50%', '42%'], center: ['50%', '42%'],
avoidLabelOverlap: true, avoidLabelOverlap: true,
itemStyle: {
borderRadius: 6,
borderColor: '#fff',
borderWidth: 2,
},
label: { label: {
show: true, show: true,
position: 'outside', position: 'outside',
@@ -235,43 +244,54 @@ const operationChartOption = computed<EChartsOption>(() => {
} }
}) })
const loginDurationChartOption = computed<EChartsOption>(() => ({ const loginDurationChartOption = computed<EChartsOption>(() => {
animationDuration: 500, const stats = loginDurationStats.value
grid: { top: 8, right: 12, bottom: 6, left: 8, containLabel: true }, const data = stats.map((u) => ({ name: u.username, value: u.duration }))
tooltip: { const maxVal = data.length
trigger: 'axis', ? Math.max(10, Math.ceil(Math.max(...data.map((d) => d.value), 0) * 1.15 / 10) * 10)
axisPointer: { type: 'shadow' }, : 10
valueFormatter: (value) => `${value} 小时`, return {
}, animationDuration: 500,
xAxis: { grid: { top: 8, right: 12, bottom: 6, left: 8, containLabel: true },
type: 'value', tooltip: {
max: Math.max(10, Math.ceil(Math.max(...loginDurationStats.value.map((user) => user.duration), 0) * 1.15 / 10) * 10), trigger: 'axis',
splitNumber: 4, axisPointer: { type: 'shadow' },
axisLabel: { color: '#94a3b8', fontSize: 11, formatter: '{value}h' }, valueFormatter: (value: unknown) => String(Number(Array.isArray(value) ? value[0] : value) || 0) + ' 小时',
axisLine: { show: false },
axisTick: { show: false },
splitLine: { lineStyle: { color: '#eef2f7' } },
},
yAxis: {
type: 'category',
inverse: true,
data: loginDurationStats.value.map((user) => user.username),
axisLabel: { color: '#475569', fontSize: 12 },
axisLine: { show: false },
axisTick: { show: false },
},
series: [
{
name: '登录时长',
type: 'bar',
data: loginDurationStats.value.map((user) => user.duration),
barMaxWidth: 18,
barCategoryGap: '34%',
itemStyle: { color: '#4f46e5', borderRadius: [0, 4, 4, 0] },
label: { show: true, position: 'insideRight', distance: 6, color: '#ffffff', fontSize: 11, formatter: '{c} 小时' },
}, },
], xAxis: {
})) type: 'value',
max: maxVal,
splitNumber: 4,
axisLabel: { color: '#94a3b8', fontSize: 11, formatter: '{value}h' },
axisLine: { show: false },
axisTick: { show: false },
splitLine: { lineStyle: { color: '#eef2f7' } },
},
yAxis: {
type: 'category',
inverse: true,
data: data.map((d) => d.name),
axisLabel: {
color: '#1f2937',
fontSize: 14,
fontFamily: '"PingFang SC", "Microsoft YaHei", system-ui, -apple-system, sans-serif',
margin: 12,
},
axisLine: { show: false },
axisTick: { show: false },
},
series: [
{
name: '登录时长',
type: 'bar',
data: data.map((d) => d.value),
barMaxWidth: 18,
barCategoryGap: '34%',
itemStyle: { color: '#4f46e5', borderRadius: [0, 4, 4, 0] },
},
],
}
})
const roleTagType: Record<string, 'danger' | 'primary' | 'info'> = { const roleTagType: Record<string, 'danger' | 'primary' | 'info'> = {
'超级管理员': 'danger', '超级管理员': 'danger',
@@ -405,10 +425,9 @@ function viewTask(task: DashboardTask) {
<section class="stat-card" aria-labelledby="login-dur-title"> <section class="stat-card" aria-labelledby="login-dur-title">
<h2 id="login-dur-title" class="section-title">登录时长排行 (本月)</h2> <h2 id="login-dur-title" class="section-title">登录时长排行 (本月)</h2>
<div v-if="loginDurationStats.length" class="chart-container"> <div class="chart-container">
<VChart class="duration-chart" :option="loginDurationChartOption" autoresize /> <VChart class="duration-chart" :option="loginDurationChartOption" autoresize />
</div> </div>
<div v-else class="empty-hint">暂无数据</div>
</section> </section>
<section class="stat-card" aria-labelledby="recent-login-title"> <section class="stat-card" aria-labelledby="recent-login-title">
@@ -723,10 +742,11 @@ function viewTask(task: DashboardTask) {
.service-table { .service-table {
display: grid; display: grid;
grid-template-rows: 36px repeat(4, minmax(48px, 1fr)); grid-auto-rows: minmax(44px, auto);
flex: 1 1 auto; flex: 1 1 auto;
margin-top: 12px; margin-top: 12px;
min-height: 0; min-height: 0;
overflow-y: auto;
} }
.service-row { .service-row {
@@ -920,7 +940,7 @@ function viewTask(task: DashboardTask) {
} }
.service-table { .service-table {
grid-template-rows: 32px repeat(4, minmax(40px, 1fr)); grid-auto-rows: minmax(38px, auto);
margin-top: 8px; margin-top: 8px;
} }

View File

@@ -50,6 +50,20 @@ const rules: FormRules = {
} }
/** 处理文件选择(替换模式:新文件覆盖旧文件) */ /** 处理文件选择(替换模式:新文件覆盖旧文件) */
function parseDatasetRecordValues(text: string, fileName: string): unknown[] {
const content = text.trim()
if (!content) return []
if (fileName.toLowerCase().endsWith('.json')) {
const parsed = JSON.parse(content)
return Array.isArray(parsed) ? parsed : [parsed]
}
return content
.split('\n')
.map((line) => line.trim())
.filter(Boolean)
.map((line) => JSON.parse(line))
}
async function handleFileChange(uploadFile: UploadFile) { async function handleFileChange(uploadFile: UploadFile) {
const raw = uploadFile.raw const raw = uploadFile.raw
if (!raw) return if (!raw) return
@@ -70,25 +84,19 @@ async function handleFileChange(uploadFile: UploadFile) {
async function analyzeFile(file: File) { async function analyzeFile(file: File) {
try { try {
const text = await file.text() const text = await file.text()
const lines = text.trim().split('\n').filter(Boolean) const records = parseDatasetRecordValues(text, file.name)
fileCount.value = lines.length fileCount.value = records.length
// Alpaca 格式校验:每行 JSON 须含 instruction 字段 // Alpaca 格式校验:每行 JSON 须含 instruction 字段
let validCount = 0 const validCount = records.filter(
for (const line of lines) { (obj) => obj && typeof obj === 'object' && 'instruction' in obj,
try { ).length
const obj = JSON.parse(line) if (validCount > 0 && validCount === records.length) {
if (obj.instruction !== undefined) validCount++
} catch {
// 非 JSON 行(如纯 JSONL 多行结构)
}
}
if (validCount > 0 && validCount === lines.length) {
formatValid.value = true formatValid.value = true
formatMessage.value = `符合 Alpaca 格式(含 instruction 字段)` formatMessage.value = `符合 Alpaca 格式(含 instruction 字段)`
} else if (validCount > 0) { } else if (validCount > 0) {
formatValid.value = true formatValid.value = true
formatMessage.value = `部分符合 Alpaca 格式(${validCount}/${lines.length}` formatMessage.value = `部分符合 Alpaca 格式(${validCount}/${records.length}`
} else { } else {
formatValid.value = false formatValid.value = false
formatMessage.value = '未检测到标准 Alpaca 格式(缺少 instruction 字段),仍可上传' formatMessage.value = '未检测到标准 Alpaca 格式(缺少 instruction 字段),仍可上传'

View File

@@ -79,7 +79,9 @@ async function loadEditData() {
async function loadModels() { async function loadModels() {
try { try {
const all = (await getModelList()) || [] const all = (await getModelList()) || []
evalModels.value = all.filter((m) => m.purpose === 'evaluation') evalModels.value = all.filter(
(m) => m.purpose === 'evaluation' || (m.model_source === 'api' && !!m.api_url),
)
} catch { } catch {
evalModels.value = [] evalModels.value = []
} }

View File

@@ -1,4 +1,4 @@
<script setup lang="ts"> <script setup lang="ts">
import { onMounted, ref, watch } from 'vue' import { onMounted, ref, watch } from 'vue'
import { useRouter } from 'vue-router' import { useRouter } from 'vue-router'
import { ElMessage } from 'element-plus' import { ElMessage } from 'element-plus'
@@ -10,7 +10,7 @@ import StartEvalStep from './create/StartEvalStep.vue'
import { createDimension, startEval } from '@/api/modules/eval' import { createDimension, startEval } from '@/api/modules/eval'
import { getTrainedModels, getModelList } from '@/api/modules/model' import { getTrainedModels, getModelList } from '@/api/modules/model'
import { getDatasetList } from '@/api/modules/dataset' import { getDatasetList } from '@/api/modules/dataset'
import { getSystemInfo } from '@/api/modules/system' import { getComputeGpus } from '@/api/modules/compute'
import type { DatasetItem, Dimension, GpuInfo, ModelItem, TrainedModel } from '@/types' import type { DatasetItem, Dimension, GpuInfo, ModelItem, TrainedModel } from '@/types'
type StepExposed = { validate: () => Promise<boolean> } type StepExposed = { validate: () => Promise<boolean> }
@@ -82,17 +82,21 @@ async function loadData() {
const results = await Promise.allSettled([ const results = await Promise.allSettled([
getTrainedModels(), getTrainedModels(),
getDatasetList(), getDatasetList(),
getSystemInfo(),
getModelList(), getModelList(),
getComputeGpus(),
]) ])
if (results[0].status === 'fulfilled') trainedModels.value = results[0].value?.models || [] if (results[0].status === 'fulfilled') trainedModels.value = results[0].value?.models || []
if (results[1].status === 'fulfilled') { if (results[1].status === 'fulfilled') {
evalDatasets.value = (results[1].value || []).filter((dataset) => dataset.type === 'eval') evalDatasets.value = (results[1].value || []).filter((dataset) => dataset.type === 'eval')
} }
if (results[2].status === 'fulfilled') gpus.value = results[2].value?.gpu || [] if (results[2].status === 'fulfilled') {
evalModels.value = (results[2].value || []).filter(
(model) => model.purpose === 'evaluation' || (model.model_source === 'api' && !!model.api_url),
)
}
if (results[3].status === 'fulfilled') { if (results[3].status === 'fulfilled') {
evalModels.value = (results[3].value || []).filter((model) => model.purpose === 'evaluation') gpus.value = ((results[3].value || []) as unknown as GpuInfo[]).filter((g) => g.status === 'idle')
} }
const failedCount = results.filter((result) => result.status === 'rejected').length const failedCount = results.filter((result) => result.status === 'rejected').length
@@ -139,11 +143,15 @@ async function handleSubmit() {
submitting.value = true submitting.value = true
try { try {
const dimensionId = await resolveDimensionId() const dimensionId = await resolveDimensionId()
await startEval({ // GPU 选择为「节点:GPU序号」复合值解析出节点与 GPU 序号,
// 多算力节点时必须把节点信息传给后端,否则会派发到错误的算力节点
const [gpuNodeId, gpuIndex] = String(taskForm.value.gpu_id).split(':')
const evalResult: any = await startEval({
eval_task_name: taskForm.value.eval_task_name, eval_task_name: taskForm.value.eval_task_name,
eval_type: 'custom', eval_type: 'custom',
model_id: taskForm.value.model_id, model_id: taskForm.value.model_id,
gpu_id: taskForm.value.gpu_id, gpu_id: Number(gpuIndex) || 0,
compute_node_id: gpuNodeId || '',
dataset_id: taskForm.value.data_source === 'dataset' ? taskForm.value.dataset_id : '', dataset_id: taskForm.value.data_source === 'dataset' ? taskForm.value.dataset_id : '',
dimension_id: dimensionId, dimension_id: dimensionId,
data_source: taskForm.value.data_source, data_source: taskForm.value.data_source,
@@ -163,6 +171,10 @@ async function handleSubmit() {
output_precision: basicMetricForm.value.output_precision, output_precision: basicMetricForm.value.output_precision,
}, },
}) })
if (evalResult?.status === 'failed' || evalResult?.error) {
ElMessage.error(`评测启动失败:${evalResult?.error || '请检查算力节点与模型路径'}`)
return
}
ElMessage.success('评测任务已创建并启动') ElMessage.success('评测任务已创建并启动')
router.push('/model-eval') router.push('/model-eval')
} catch (error) { } catch (error) {

View File

@@ -1,9 +1,10 @@
<script setup lang="ts"> <script setup lang="ts">
import { computed, onMounted, ref } from 'vue' import { computed, onMounted, onUnmounted, ref } from 'vue'
import { useRoute } from 'vue-router' import { useRoute } from 'vue-router'
import PageCard from '@/components/PageCard.vue' import PageCard from '@/components/PageCard.vue'
import ModelStatusTag from '@/components/ModelStatusTag.vue' import ModelStatusTag from '@/components/ModelStatusTag.vue'
import { getEvalDetail } from '@/api/modules/eval' import { getEvalDetail } from '@/api/modules/eval'
import { usePolling } from '@/composables/usePolling'
import type { EvalSampleResult, EvalTaskDetail } from '@/types' import type { EvalSampleResult, EvalTaskDetail } from '@/types'
const route = useRoute() const route = useRoute()
@@ -16,6 +17,7 @@ const keyword = ref('')
const judgementFilter = ref('') const judgementFilter = ref('')
const currentPage = ref(1) const currentPage = ref(1)
const pageSize = ref(10) const pageSize = ref(10)
const ACTIVE_STATUSES = new Set(['pending', 'queued', 'running'])
const filteredSamples = computed(() => { const filteredSamples = computed(() => {
const normalizedKeyword = keyword.value.trim().toLowerCase() const normalizedKeyword = keyword.value.trim().toLowerCase()
@@ -48,6 +50,8 @@ const passRate = computed(() => {
}) })
const overallScore = computed(() => formatScore(detail.value?.overall_score, detail.value?.overall_score_max)) const overallScore = computed(() => formatScore(detail.value?.overall_score, detail.value?.overall_score_max))
const displayModelName = computed(() => detail.value?.model_name || String(detail.value?.model_id || '-'))
const displayMetric = computed(() => detail.value?.metric_label || detail.value?.metric || '-')
function formatDateTime(value?: string) { function formatDateTime(value?: string) {
if (!value) return '-' if (!value) return '-'
@@ -74,8 +78,8 @@ function resetPage() {
currentPage.value = 1 currentPage.value = 1
} }
async function loadDetail() { async function loadDetail(options: { silent?: boolean } = {}) {
loading.value = true if (!options.silent) loading.value = true
loadError.value = '' loadError.value = ''
try { try {
detail.value = await getEvalDetail(taskId) detail.value = await getEvalDetail(taskId)
@@ -83,11 +87,29 @@ async function loadDetail() {
detail.value = null detail.value = null
loadError.value = '评测详情加载失败,请稍后重试。' loadError.value = '评测详情加载失败,请稍后重试。'
} finally { } finally {
loading.value = false if (!options.silent) loading.value = false
} }
} }
onMounted(loadDetail) const { start: startPolling, stop: stopPolling } = usePolling(
async () => {
await loadDetail({ silent: true })
if (!ACTIVE_STATUSES.has(String(detail.value?.status || ''))) {
stopPolling()
}
},
5000,
{ immediate: false },
)
onMounted(async () => {
await loadDetail()
if (ACTIVE_STATUSES.has(String(detail.value?.status || ''))) {
startPolling()
}
})
onUnmounted(stopPolling)
</script> </script>
<template> <template>
@@ -101,9 +123,9 @@ onMounted(loadDetail)
</div> </div>
<dl class="task-meta"> <dl class="task-meta">
<div><dt>任务 ID</dt><dd>{{ detail?.id || taskId }}</dd></div> <div><dt>任务 ID</dt><dd>{{ detail?.id || taskId }}</dd></div>
<div><dt>评测模型</dt><dd>{{ detail?.model_name || '-' }}</dd></div> <div><dt>评测模型</dt><dd>{{ displayModelName }}</dd></div>
<div><dt>测试集</dt><dd>{{ detail?.dataset || '-' }}</dd></div> <div><dt>测试集</dt><dd>{{ detail?.dataset || '-' }}</dd></div>
<div><dt>评测指标</dt><dd>{{ detail?.metric || '-' }}</dd></div> <div><dt>评测指标</dt><dd>{{ displayMetric }}</dd></div>
</dl> </dl>
</div> </div>
</div> </div>
@@ -113,7 +135,7 @@ onMounted(loadDetail)
<i class="fa fa-exclamation-circle" aria-hidden="true" /> <i class="fa fa-exclamation-circle" aria-hidden="true" />
<h2>无法加载评测详情</h2> <h2>无法加载评测详情</h2>
<p>{{ loadError }}</p> <p>{{ loadError }}</p>
<el-button type="primary" @click="loadDetail">重新加载</el-button> <el-button type="primary" @click="() => loadDetail()">重新加载</el-button>
</div> </div>
<template v-else-if="detail"> <template v-else-if="detail">
@@ -121,7 +143,7 @@ onMounted(loadDetail)
<div class="overview-item score-hero"> <div class="overview-item score-hero">
<span>综合得分</span> <span>综合得分</span>
<strong>{{ overallScore }}</strong> <strong>{{ overallScore }}</strong>
<small>模型综合评分</small> <small>模型综合评分</small>
</div> </div>
<div class="overview-item"> <div class="overview-item">
<span>样本通过率</span> <span>样本通过率</span>
@@ -144,7 +166,7 @@ onMounted(loadDetail)
<div class="review-copy"> <div class="review-copy">
<div class="section-heading"> <div class="section-heading">
<div> <div>
<h2 id="overall-review-title">大模型综合评价</h2> <h2 id="overall-review-title">综合评价</h2>
<p>基于全部已评测样本生成的总体结论</p> <p>基于全部已评测样本生成的总体结论</p>
</div> </div>
<el-tag v-if="detail.evaluator_model" type="primary" size="small"> <el-tag v-if="detail.evaluator_model" type="primary" size="small">
@@ -152,7 +174,7 @@ onMounted(loadDetail)
</el-tag> </el-tag>
</div> </div>
<p class="review-text"> <p class="review-text">
{{ detail.overall_evaluation || (detail.status === 'running' ? '评测在进行,综合评价将在样本评分完成后生成。' : '暂无综合评价。') }} {{ detail.overall_evaluation || (detail.status === 'running' ? '评测在进行,综合评价将在样本完成后生成。' : '暂无综合评价。') }}
</p> </p>
<div class="suggestion-block"> <div class="suggestion-block">
@@ -170,8 +192,8 @@ onMounted(loadDetail)
<section v-if="detail.dimension_summary?.length" class="dimension-summary" aria-labelledby="dimension-title"> <section v-if="detail.dimension_summary?.length" class="dimension-summary" aria-labelledby="dimension-title">
<div class="section-heading compact-heading"> <div class="section-heading compact-heading">
<div> <div>
<h2 id="dimension-title">维度表现</h2> <h2 id="dimension-title">指标表现</h2>
<p>查看各评测维度的得分与样本通过率</p> <p>查看各评测指标的得分与通过率</p>
</div> </div>
</div> </div>
<div class="dimension-grid"> <div class="dimension-grid">
@@ -192,22 +214,10 @@ onMounted(loadDetail)
<p> {{ filteredSamples.length }} 条结果展开行可查看评分依据与子维度分数</p> <p> {{ filteredSamples.length }} 条结果展开行可查看评分依据与子维度分数</p>
</div> </div>
<div class="sample-filters" aria-label="样本筛选"> <div class="sample-filters" aria-label="样本筛选">
<el-input <el-input v-model="keyword" clearable placeholder="搜索问题、回答或评价" aria-label="搜索样本" @input="resetPage">
v-model="keyword"
clearable
placeholder="搜索问题、回答或评价"
aria-label="搜索样本"
@input="resetPage"
>
<template #prefix><i class="fa fa-search" aria-hidden="true" /></template> <template #prefix><i class="fa fa-search" aria-hidden="true" /></template>
</el-input> </el-input>
<el-select <el-select v-model="judgementFilter" clearable placeholder="全部判定" aria-label="按判定筛选" @change="resetPage">
v-model="judgementFilter"
clearable
placeholder="全部判定"
aria-label="按判定筛选"
@change="resetPage"
>
<el-option label="正确" value="正确" /> <el-option label="正确" value="正确" />
<el-option label="部分正确" value="部分正确" /> <el-option label="部分正确" value="部分正确" />
<el-option label="错误" value="错误" /> <el-option label="错误" value="错误" />
@@ -215,18 +225,12 @@ onMounted(loadDetail)
</div> </div>
</div> </div>
<el-table <el-table v-if="filteredSamples.length" class="sample-results-table" :data="paginatedSamples" row-key="id" table-layout="fixed">
v-if="filteredSamples.length"
class="sample-results-table"
:data="paginatedSamples"
row-key="id"
table-layout="fixed"
>
<el-table-column type="expand" width="48"> <el-table-column type="expand" width="48">
<template #default="{ row }"> <template #default="{ row }">
<div class="sample-detail-grid"> <div class="sample-detail-grid">
<div class="evaluation-reason"> <div class="evaluation-reason">
<span>大模型评分依据</span> <span>评分依据</span>
<p>{{ row.evaluation_reason || '暂无评分依据。' }}</p> <p>{{ row.evaluation_reason || '暂无评分依据。' }}</p>
</div> </div>
<div v-if="row.error_type" class="error-type"> <div v-if="row.error_type" class="error-type">
@@ -255,9 +259,7 @@ onMounted(loadDetail)
<template #default="{ row }"><p class="cell-copy">{{ row.model_output || '等待生成' }}</p></template> <template #default="{ row }"><p class="cell-copy">{{ row.model_output || '等待生成' }}</p></template>
</el-table-column> </el-table-column>
<el-table-column label="得分" width="90" align="center"> <el-table-column label="得分" width="90" align="center">
<template #default="{ row }"> <template #default="{ row }"><span class="sample-score">{{ formatScore(row.score, row.max_score) }}</span></template>
<span class="sample-score">{{ formatScore(row.score, row.max_score) }}</span>
</template>
</el-table-column> </el-table-column>
<el-table-column label="判定" width="96" align="center"> <el-table-column label="判定" width="96" align="center">
<template #default="{ row }"> <template #default="{ row }">
@@ -274,21 +276,11 @@ onMounted(loadDetail)
<p>{{ detail.status === 'running' ? '任务正在运行,结果生成后会显示在这里。' : '请调整筛选条件或稍后重试。' }}</p> <p>{{ detail.status === 'running' ? '任务正在运行,结果生成后会显示在这里。' : '请调整筛选条件或稍后重试。' }}</p>
</div> </div>
<el-pagination <el-pagination v-if="filteredSamples.length > pageSize" v-model:current-page="currentPage" v-model:page-size="pageSize" background layout="total, sizes, prev, pager, next" :page-sizes="[10, 20, 50]" :total="filteredSamples.length" aria-label="样本结果分页" />
v-if="filteredSamples.length > pageSize"
v-model:current-page="currentPage"
v-model:page-size="pageSize"
background
layout="total, sizes, prev, pager, next"
:page-sizes="[10, 20, 50]"
:total="filteredSamples.length"
aria-label="样本结果分页"
/>
</section> </section>
</template> </template>
</PageCard> </PageCard>
</template> </template>
<style scoped lang="scss"> <style scoped lang="scss">
.eval-detail-page { .eval-detail-page {
min-width: 0; min-width: 0;
@@ -747,3 +739,6 @@ onMounted(loadDetail)
} }
} }
</style> </style>

View File

@@ -1,9 +1,10 @@
<script setup lang="ts"> <script setup lang="ts">
import { ref, onMounted } from 'vue' import { ref, onMounted, onUnmounted } from 'vue'
import { useRouter } from 'vue-router' import { useRouter } from 'vue-router'
import { ElMessage, ElMessageBox } from 'element-plus' import { ElMessage, ElMessageBox } from 'element-plus'
import DataTablePage from '@/components/DataTablePage.vue' import DataTablePage from '@/components/DataTablePage.vue'
import ModelStatusTag from '@/components/ModelStatusTag.vue' import ModelStatusTag from '@/components/ModelStatusTag.vue'
import { usePolling } from '@/composables/usePolling'
import { import {
getEvalList, getEvalList,
deleteEval, deleteEval,
@@ -23,14 +24,16 @@ const leaderboard = ref([
{ rank: 3, name: 'Qwen-Max', score: 85.3 }, { rank: 3, name: 'Qwen-Max', score: 85.3 },
]) ])
async function loadEvalList() { const ACTIVE_STATUSES = new Set(['pending', 'queued', 'running'])
evalLoading.value = true
async function loadEvalList(options: { silent?: boolean } = {}) {
if (!options.silent) evalLoading.value = true
try { try {
evalList.value = (await getEvalList()) || [] evalList.value = (await getEvalList()) || []
} catch { } catch {
evalList.value = [] evalList.value = []
} finally { } finally {
evalLoading.value = false if (!options.silent) evalLoading.value = false
} }
} }
@@ -54,8 +57,34 @@ function handleViewDetail(row: any) {
router.push({ name: 'model-eval-detail', params: { id: row.id } }) router.push({ name: 'model-eval-detail', params: { id: row.id } })
} }
onMounted(() => { function displayModelName(row: Partial<EvalTask>) {
loadEvalList() return row.model_name || String(row.model_id || '-')
}
function displayMetric(row: Partial<EvalTask>) {
return row.metric_label || row.metric || '-'
}
const { start: startPolling, stop: stopPolling } = usePolling(
async () => {
await loadEvalList({ silent: true })
if (!evalList.value.some((item) => ACTIVE_STATUSES.has(String(item.status || '')))) {
stopPolling()
}
},
5000,
{ immediate: false },
)
onMounted(async () => {
await loadEvalList()
if (evalList.value.some((item) => ACTIVE_STATUSES.has(String(item.status || '')))) {
startPolling()
}
})
onUnmounted(() => {
stopPolling()
}) })
</script> </script>
@@ -81,9 +110,21 @@ onMounted(() => {
</template> </template>
<template #columns> <template #columns>
<el-table-column label="任务名称" prop="eval_task_name" align="center" /> <el-table-column label="任务名称" prop="eval_task_name" align="center" />
<el-table-column label="评测模型" prop="model_name" align="center" /> <el-table-column label="评测模型" align="center" min-width="160">
<template #default="{ row }">
<el-tooltip :content="displayModelName(row)" placement="top" :disabled="displayModelName(row).length < 18">
<span class="cell-ellipsis">{{ displayModelName(row) }}</span>
</el-tooltip>
</template>
</el-table-column>
<el-table-column label="数据集" prop="dataset" align="center" /> <el-table-column label="数据集" prop="dataset" align="center" />
<el-table-column label="指标" prop="metric" align="center" /> <el-table-column label="指标" align="center" min-width="220">
<template #default="{ row }">
<el-tooltip :content="displayMetric(row)" placement="top" :disabled="displayMetric(row).length < 24">
<span class="cell-ellipsis">{{ displayMetric(row) }}</span>
</el-tooltip>
</template>
</el-table-column>
<el-table-column label="评分" prop="score" width="100" align="center" /> <el-table-column label="评分" prop="score" width="100" align="center" />
<el-table-column label="状态" width="100" align="center"> <el-table-column label="状态" width="100" align="center">
<template #default="{ row }"> <template #default="{ row }">
@@ -138,7 +179,7 @@ onMounted(() => {
min-height: 0; min-height: 0;
} }
/* 胶囊切换栏样式 */ /* 胶囊切换栏 */
.capsule-tabs { .capsule-tabs {
display: flex; display: flex;
background: #f1f5f9; background: #f1f5f9;
@@ -172,4 +213,13 @@ onMounted(() => {
} }
} }
.cell-ellipsis {
display: inline-block;
max-width: 100%;
overflow: hidden;
text-overflow: ellipsis;
vertical-align: middle;
white-space: nowrap;
}
</style> </style>

View File

@@ -56,9 +56,9 @@ defineExpose({ validate })
</el-form-item> </el-form-item>
<el-form-item v-if="form.rouge_enabled" label="ROUGE methods"> <el-form-item v-if="form.rouge_enabled" label="ROUGE methods">
<el-checkbox-group v-model="form.rouge_methods"> <el-checkbox-group v-model="form.rouge_methods">
<el-checkbox value="rouge_1">ROUGE-1</el-checkbox> <el-checkbox value="rouge1">ROUGE-1</el-checkbox>
<el-checkbox value="rouge_2">ROUGE-2</el-checkbox> <el-checkbox value="rouge2">ROUGE-2</el-checkbox>
<el-checkbox value="rouge_l">ROUGE-L</el-checkbox> <el-checkbox value="rougeL">ROUGE-L</el-checkbox>
</el-checkbox-group> </el-checkbox-group>
</el-form-item> </el-form-item>

View File

@@ -103,10 +103,10 @@ defineExpose({ validate })
<el-form-item label="选择 GPU" prop="gpu_id"> <el-form-item label="选择 GPU" prop="gpu_id">
<el-select v-model="form.gpu_id" placeholder="请选择 GPU" style="width: 100%" :loading="loading"> <el-select v-model="form.gpu_id" placeholder="请选择 GPU" style="width: 100%" :loading="loading">
<el-option <el-option
v-for="(gpu, index) in gpus" v-for="gpu in gpus"
:key="index" :key="`${gpu.node_id || ''}:${gpu.id ?? 0}`"
:label="`${gpu.name} (GPU ${index})`" :label="`${gpu.node_name || gpu.node_code || '算力节点'} / ${gpu.name} (GPU ${gpu.id ?? 0})`"
:value="index" :value="`${gpu.node_id || ''}:${gpu.id ?? 0}`"
/> />
</el-select> </el-select>
</el-form-item> </el-form-item>

View File

@@ -1,4 +1,5 @@
<script setup lang="ts"> <script setup lang="ts">
import { computed } from 'vue'
import type { DatasetItem, GpuInfo, ModelItem, TrainedModel } from '@/types' import type { DatasetItem, GpuInfo, ModelItem, TrainedModel } from '@/types'
import type { BasicMetricSetupDraft } from './BasicMetricSetupStep.vue' import type { BasicMetricSetupDraft } from './BasicMetricSetupStep.vue'
import type { EvalRuleSetupDraft } from './EvalRuleSetupStep.vue' import type { EvalRuleSetupDraft } from './EvalRuleSetupStep.vue'
@@ -17,6 +18,15 @@ const props = defineProps<{
function nameOf<T extends { id: string | number; name?: string }>(items: T[], id: string | number) { function nameOf<T extends { id: string | number; name?: string }>(items: T[], id: string | number) {
return items.find((item) => item.id === id)?.name || String(id || '-') return items.find((item) => item.id === id)?.name || String(id || '-')
} }
/** GPU 选择为「节点:GPU序号」复合值解析并展示为可读标签 */
const gpuLabel = computed(() => {
const key = String(props.task.gpu_id || '')
const gpu = props.gpus.find((g) => `${g.node_id || ''}:${g.id ?? 0}` === key)
if (gpu) return `${gpu.node_name || gpu.node_code || '算力节点'} / GPU ${gpu.id ?? 0}`
const [nodeId, idx] = key.split(':')
return nodeId ? `节点 ${nodeId} / GPU ${idx || 0}` : `GPU ${key || 0}`
})
</script> </script>
<template> <template>
@@ -24,7 +34,7 @@ function nameOf<T extends { id: string | number; name?: string }>(items: T[], id
<el-descriptions :column="2" border> <el-descriptions :column="2" border>
<el-descriptions-item label="Task">{{ props.task.eval_task_name || '-' }}</el-descriptions-item> <el-descriptions-item label="Task">{{ props.task.eval_task_name || '-' }}</el-descriptions-item>
<el-descriptions-item label="Model">{{ nameOf(props.trainedModels, props.task.model_id) }}</el-descriptions-item> <el-descriptions-item label="Model">{{ nameOf(props.trainedModels, props.task.model_id) }}</el-descriptions-item>
<el-descriptions-item label="GPU">GPU {{ props.task.gpu_id || 0 }}</el-descriptions-item> <el-descriptions-item label="GPU">{{ gpuLabel }}</el-descriptions-item>
<el-descriptions-item label="Dataset"> <el-descriptions-item label="Dataset">
{{ props.task.data_source === 'dataset' ? nameOf(props.evalDatasets, props.task.dataset_id) : 'Inference results' }} {{ props.task.data_source === 'dataset' ? nameOf(props.evalDatasets, props.task.dataset_id) : 'Inference results' }}
</el-descriptions-item> </el-descriptions-item>

View File

@@ -37,7 +37,7 @@ const models = ref<ModelItem[]>([])
const datasets = ref<DatasetItem[]>([]) const datasets = ref<DatasetItem[]>([])
const gpus = ref<GpuInfo[]>([]) const gpus = ref<GpuInfo[]>([])
const computeNodes = ref<Array<{ id: string; scheduler_status?: string }>>([]) const computeNodes = ref<Array<{ id: string; scheduler_status?: string }>>([])
const selectedGpuId = ref<number | null>(null) const selectedGpuKeys = ref<string[]>([])
/** Only show GPUs from nodes that are online or draining */ /** Only show GPUs from nodes that are online or draining */
const availableGpus = computed(() => { const availableGpus = computed(() => {
@@ -74,7 +74,13 @@ const selectedModel = computed(() => models.value.find((model) => model.id === f
const modelDialogTitle = computed(() => selectedModel.value?.name || '') const modelDialogTitle = computed(() => selectedModel.value?.name || '')
/** 训练命令与提交载荷共用同一份表单模型。 */ /** 训练命令与提交载荷共用同一份表单模型。 */
const selectedGpuIds = computed(() => (selectedGpuId.value != null ? [selectedGpuId.value] : [])) const selectedGpus = computed(() =>
selectedGpuKeys.value
.map((key) => availableGpus.value.find((gpu) => gpuKey(gpu) === key))
.filter((gpu): gpu is GpuInfo => Boolean(gpu)),
)
const selectedComputeNodeId = computed(() => selectedGpus.value[0]?.node_id)
const selectedGpuIds = computed(() => selectedGpus.value.map((gpu) => Number(gpu.id)))
const commandPreview = computed(() => buildFineTuneCommand(form, selectedGpuIds.value)) const commandPreview = computed(() => buildFineTuneCommand(form, selectedGpuIds.value))
const remoteCommandPreview = computed(() => { const remoteCommandPreview = computed(() => {
@@ -83,9 +89,32 @@ const remoteCommandPreview = computed(() => {
return preflightResult.value?.preview?.command_text || '' return preflightResult.value?.preview?.command_text || ''
}) })
/** GPU 单选切换(每次只选中一张 GPU */ function gpuKey(gpu: GpuInfo) {
function toggleGpu(gpuId: number) { return `${gpu.node_id || 'local'}:${gpu.id ?? gpu.uuid ?? gpu.name}`
selectedGpuId.value = selectedGpuId.value === gpuId ? null : gpuId }
function isGpuUnavailable(gpu: GpuInfo) {
return gpu.status === 'busy' || gpu.status === 'reserved' || gpu.status === 'offline'
}
function isGpuSelected(gpu: GpuInfo) {
return selectedGpuKeys.value.includes(gpuKey(gpu))
}
/** GPU 多选切换:单个任务只允许选择同一算力节点内的空闲卡。 */
function toggleGpu(gpu: GpuInfo) {
if (isGpuUnavailable(gpu) || gpu.id == null) return
const key = gpuKey(gpu)
if (isGpuSelected(gpu)) {
selectedGpuKeys.value = selectedGpuKeys.value.filter((item) => item !== key)
return
}
if (selectedComputeNodeId.value && gpu.node_id && selectedComputeNodeId.value !== gpu.node_id) {
selectedGpuKeys.value = [key]
ElMessage.info('已切换到新的算力节点,之前选择的 GPU 已清空')
return
}
selectedGpuKeys.value = [...selectedGpuKeys.value, key]
} }
function gpuUsageWidth(percent: number) { function gpuUsageWidth(percent: number) {
@@ -178,8 +207,8 @@ async function loadGpus() {
const [sys, nodes] = await Promise.all([getSystemInfo(), getComputeNodes().catch(() => [])]) const [sys, nodes] = await Promise.all([getSystemInfo(), getComputeNodes().catch(() => [])])
gpus.value = sys?.gpu || [] gpus.value = sys?.gpu || []
computeNodes.value = nodes || [] computeNodes.value = nodes || []
// Default select first available GPU const firstIdle = availableGpus.value.find((gpu) => !isGpuUnavailable(gpu) && gpu.id != null)
if (availableGpus.value.length > 0) selectedGpuId.value = availableGpus.value[0].id ?? null if (firstIdle) selectedGpuKeys.value = [gpuKey(firstIdle)]
} catch { } catch {
gpus.value = [] gpus.value = []
} }
@@ -189,8 +218,8 @@ async function handleSubmit() {
if (!formRef.value) return if (!formRef.value) return
await formRef.value.validate(async (valid) => { await formRef.value.validate(async (valid) => {
if (!valid) return if (!valid) return
if (selectedGpuId.value == null) { if (!selectedGpuIds.value.length) {
ElMessage.warning('请选择一 GPU') ElMessage.warning('请至少选择一张空闲 GPU')
return return
} }
submitting.value = true submitting.value = true
@@ -207,7 +236,7 @@ async function handleSubmit() {
return return
} }
const payload = buildFineTunePayload(form, selectedGpuIds.value) const payload = buildFineTunePayload(form, selectedGpuIds.value, selectedComputeNodeId.value)
const preflight = await runPreflight(payload) const preflight = await runPreflight(payload)
if (!preflight?.valid) { if (!preflight?.valid) {
ElMessage.error('训练预检未通过,请先处理预检问题') ElMessage.error('训练预检未通过,请先处理预检问题')
@@ -232,7 +261,7 @@ async function handleSubmit() {
}) })
} }
async function runPreflight(payload = buildFineTunePayload(form, selectedGpuIds.value)) { async function runPreflight(payload = buildFineTunePayload(form, selectedGpuIds.value, selectedComputeNodeId.value)) {
preflightLoading.value = true preflightLoading.value = true
try { try {
const result = await preflightFineTune(payload) const result = await preflightFineTune(payload)
@@ -261,8 +290,8 @@ async function handlePreflightClick() {
if (!formRef.value) return if (!formRef.value) return
await formRef.value.validate(async (valid) => { await formRef.value.validate(async (valid) => {
if (!valid) return if (!valid) return
if (selectedGpuId.value == null) { if (!selectedGpuIds.value.length) {
ElMessage.warning('请选择一 GPU') ElMessage.warning('请至少选择一张空闲 GPU')
return return
} }
await runPreflight() await runPreflight()
@@ -297,12 +326,16 @@ onMounted(() => {
<el-divider content-position="left">训练配置</el-divider> <el-divider content-position="left">训练配置</el-divider>
<el-form-item label="GPU 硬件"> <el-form-item label="GPU 硬件">
<div class="gpu-list"> <div class="gpu-list">
<div class="gpu-selection-summary">
已选择 {{ selectedGpuIds.length }} GPU
<template v-if="selectedGpus[0]?.node_code"> · {{ selectedGpus[0].node_code }}</template>
</div>
<div <div
v-for="gpu in availableGpus" v-for="gpu in availableGpus"
:key="gpu.id" :key="gpuKey(gpu)"
class="gpu-card" class="gpu-card"
:class="{ active: selectedGpuId === gpu.id, 'is-busy': gpu.gpu_percent > 80 }" :class="{ active: isGpuSelected(gpu), 'is-busy': isGpuUnavailable(gpu), 'is-disabled': isGpuUnavailable(gpu) }"
@click="toggleGpu(gpu.id!)" @click="toggleGpu(gpu)"
> >
<div class="gpu-card-top"> <div class="gpu-card-top">
<div class="gpu-title"> <div class="gpu-title">
@@ -312,7 +345,7 @@ onMounted(() => {
</span> </span>
<span class="gpu-name">{{ gpu.name }}</span> <span class="gpu-name">{{ gpu.name }}</span>
</div> </div>
<span class="gpu-usage">{{ gpu.gpu_percent }}%</span> <span class="gpu-usage">{{ isGpuUnavailable(gpu) ? gpu.status : `${gpu.gpu_percent}%` }}</span>
</div> </div>
<div class="gpu-usage-bar"> <div class="gpu-usage-bar">
<span :style="{ width: gpuUsageWidth(gpu.gpu_percent) }" /> <span :style="{ width: gpuUsageWidth(gpu.gpu_percent) }" />
@@ -578,6 +611,13 @@ onMounted(() => {
width: 100%; width: 100%;
} }
.gpu-selection-summary {
grid-column: 1 / -1;
color: #64748b;
font-size: 12px;
line-height: 20px;
}
.gpu-card { .gpu-card {
border: 1px solid #e5e7eb; border: 1px solid #e5e7eb;
border-radius: 6px; border-radius: 6px;
@@ -627,6 +667,11 @@ onMounted(() => {
background: #dc2626; background: #dc2626;
} }
} }
&.is-disabled {
cursor: not-allowed;
opacity: 0.72;
}
} }
.gpu-card-top { .gpu-card-top {

View File

@@ -65,6 +65,7 @@ export function createDefaultFineTuneForm(): FineTuneFormModel {
export function buildFineTunePayload( export function buildFineTunePayload(
form: FineTuneFormModel, form: FineTuneFormModel,
gpus: number[], gpus: number[],
computeNodeId?: string,
): Omit<FineTuneStartPayload, 'task_id'> { ): Omit<FineTuneStartPayload, 'task_id'> {
return { return {
name: form.name, name: form.name,
@@ -77,6 +78,7 @@ export function buildFineTunePayload(
train_dataset_id: form.train_dataset_id, train_dataset_id: form.train_dataset_id,
auto_merge: form.train_type === 'SFT' && form.auto_merge, auto_merge: form.train_type === 'SFT' && form.auto_merge,
output_model_name: form.name, output_model_name: form.name,
compute_node_id: computeNodeId,
batch_size: form.batch_size, batch_size: form.batch_size,
learning_rate: form.learning_rate, learning_rate: form.learning_rate,
n_epochs: form.n_epochs, n_epochs: form.n_epochs,

View File

@@ -1,17 +1,17 @@
<script setup lang="ts"> <script setup lang="ts">
import { ref, reactive, nextTick, onMounted, watch } from 'vue' import { ref, reactive, nextTick, onMounted, onUnmounted, watch } from 'vue'
import { useRoute, useRouter } from 'vue-router' import { useRoute, useRouter } from 'vue-router'
import { ElMessage } from 'element-plus' import { ElMessage } from 'element-plus'
import MarkdownView from '@/components/MarkdownView.vue' import MarkdownView from '@/components/MarkdownView.vue'
import { useStreamChat } from '@/composables/useStreamChat' import { useStreamChat } from '@/composables/useStreamChat'
import { getCompare } from '@/api/modules/compare' import { getCompare, getLoadStatus } from '@/api/modules/compare'
import type { CompareTask, LoadedModel } from '@/types' import type { CompareTask, LoadedModel } from '@/types'
const route = useRoute() const route = useRoute()
const router = useRouter() const router = useRouter()
const taskId = route.params.id as string const taskId = route.params.id as string
/** 是否为 mock 直通模式(新建推理假数据进入,不走真实任务接口 */ /** 是否为 mock 模式(新建推理无真实 taskId 或明确为 mock 时进入 mock 模式 */
const isMock = taskId === 'mock' const isMock = taskId === 'mock' || !taskId || taskId === 'unknown'
/** 当前对话使用的模型名 */ /** 当前对话使用的模型名 */
const modelName = ref(route.query.model as string || '') const modelName = ref(route.query.model as string || '')
@@ -37,6 +37,10 @@ const contentRef = ref<HTMLElement>()
let activeAssistant: ChatMessage | null = null let activeAssistant: ChatMessage | null = null
/** 设置面板抽屉 */ /** 设置面板抽屉 */
const showSettings = ref(false) const showSettings = ref(false)
/** 模型仍在加载中(直接 URL 进入 chat 时兜底轮询就绪状态) */
const taskLoading = ref(false)
const taskError = ref('')
let statusTimer: ReturnType<typeof setInterval> | null = null
/** 获取任务信息定位已启动的模型mock 模式跳过) */ /** 获取任务信息定位已启动的模型mock 模式跳过) */
async function loadTask() { async function loadTask() {
@@ -45,6 +49,14 @@ async function loadTask() {
task.value = await getCompare(taskId) task.value = await getCompare(taskId)
const models = parseLoadedModels(task.value) const models = parseLoadedModels(task.value)
if (models[0]?.model_name) modelName.value = models[0].model_name if (models[0]?.model_name) modelName.value = models[0].model_name
// 恢复本地保存的历史对话
restoreHistory()
// 模型仍在上次加载中:启动轮询等待就绪
if (models.some((m) => m.status === 'starting')) {
taskLoading.value = true
await pollTaskStatus()
statusTimer = setInterval(pollTaskStatus, 3000)
}
} catch { } catch {
// ignore // ignore
} }
@@ -60,6 +72,80 @@ function parseLoadedModels(t: CompareTask | null): LoadedModel[] {
} }
} }
/** 对话历史本地持久化(按任务 id 存储,退出重进可恢复) */
const STORAGE_PREFIX = 'ygft_chat_history_'
function historyKey(id: string | number): string {
return `${STORAGE_PREFIX}${id}`
}
function saveHistory() {
if (isMock) return
try {
const snapshot = messages.value.map((m) => ({
role: m.role,
content: m.content,
think: m.think,
done: true,
}))
localStorage.setItem(historyKey(taskId), JSON.stringify(snapshot))
} catch {
// 存储失败忽略
}
}
function restoreHistory() {
if (isMock) return
try {
const raw = localStorage.getItem(historyKey(taskId))
if (!raw) return
const parsed = JSON.parse(raw)
if (Array.isArray(parsed)) {
messages.value = parsed.map((m) => ({
role: m.role === 'user' ? 'user' : 'assistant',
content: m.content || '',
think: m.think || '',
isThinking: false,
isStreaming: false,
done: true,
}))
}
} catch {
// 恢复失败忽略
}
}
/** 停止就绪状态轮询 */
function stopStatusPolling() {
if (statusTimer) {
clearInterval(statusTimer)
statusTimer = null
}
}
/** 轮询任务加载状态starting → ready/error */
async function pollTaskStatus() {
try {
const st = await getLoadStatus(taskId)
const items = st.loaded_models || []
const anyReady = items.some((m) => m.status === 'ready' || m.status === 'running')
const anyError = items.some((m) => m.status === 'error')
if (anyReady) {
taskLoading.value = false
taskError.value = ''
stopStatusPolling()
} else if (anyError) {
taskLoading.value = false
taskError.value = items.find((m) => m.status === 'error')?.error || '模型加载失败'
stopStatusPolling()
} else {
taskLoading.value = true
}
} catch {
// 轮询失败忽略,下次再试
}
}
async function handleSend() { async function handleSend() {
const question = inputQuestion.value.trim() const question = inputQuestion.value.trim()
if (!question || loading.value) return if (!question || loading.value) return
@@ -76,6 +162,7 @@ async function handleSend() {
done: false, done: false,
}) })
messages.value.push(assistantMsg) messages.value.push(assistantMsg)
saveHistory()
inputQuestion.value = '' inputQuestion.value = ''
await nextTick() await nextTick()
@@ -85,33 +172,25 @@ async function handleSend() {
// mock 模式:直接用假数据逐字填充 // mock 模式:直接用假数据逐字填充
if (isMock) { if (isMock) {
await mockReply(assistantMsg, question) await mockReply(assistantMsg, question)
saveHistory()
return return
} }
// 真实模式:获取已启动模型的端口/路径 // 真实模式:通过后端 SSE 流式代理到算力节点进行推理
const models = parseLoadedModels(task.value)
const target = models[0]
if (!target) {
ElMessage.error('未找到已启动的模型')
assistantMsg.content = '未找到已启动的模型,请先返回列表加载模型'
assistantMsg.done = true
assistantMsg.isStreaming = false
return
}
// 流式状态变化时只同步当前回复,避免固定定时器空转。
activeAssistant = assistantMsg activeAssistant = assistantMsg
await send({ await send(
port: target.port, {
model_name: target.model_name, model_path: route.query.model_path as string || '',
model_path: '', task_id: taskId,
system_prompt: systemPrompt.value, system_prompt: systemPrompt.value,
user_question: question, user_question: question,
temperature: temperature.value, temperature: temperature.value,
top_p: top_p.value, top_p: top_p.value,
max_tokens: maxTokens.value, max_tokens: maxTokens.value,
}) },
{ useMock: false },
)
// 完成后同步最终内容 // 完成后同步最终内容
assistantMsg.content = message.value.displayContent || message.value.error || '(无回复)' assistantMsg.content = message.value.displayContent || message.value.error || '(无回复)'
@@ -121,6 +200,7 @@ async function handleSend() {
assistantMsg.done = true assistantMsg.done = true
activeAssistant = null activeAssistant = null
reset() reset()
saveHistory()
await nextTick() await nextTick()
scrollToBottom() scrollToBottom()
} }
@@ -179,6 +259,11 @@ function handleNewChat() {
activeAssistant = null activeAssistant = null
messages.value = [] messages.value = []
reset() reset()
try {
localStorage.removeItem(historyKey(taskId))
} catch {
// 忽略
}
} }
/** 输入框自适应高度 */ /** 输入框自适应高度 */
@@ -195,6 +280,7 @@ function resetInputHeight() {
} }
onMounted(loadTask) onMounted(loadTask)
onUnmounted(stopStatusPolling)
</script> </script>
<template> <template>
@@ -262,6 +348,12 @@ onMounted(loadTask)
<!-- 输入栏 --> <!-- 输入栏 -->
<footer class="chat-input-container"> <footer class="chat-input-container">
<div v-if="taskLoading" class="loading-hint">
<i class="fa fa-spinner fa-spin" style="margin-right: 6px" />模型加载中就绪后即可对话...
</div>
<div v-else-if="taskError" class="loading-hint error">
<i class="fa fa-exclamation-circle" style="margin-right: 6px" />{{ taskError }}
</div>
<div class="chat-input-inner"> <div class="chat-input-inner">
<button class="clear-btn" title="清空对话" @click="handleNewChat"> <button class="clear-btn" title="清空对话" @click="handleNewChat">
<i class="fa fa-eraser" /> <i class="fa fa-eraser" />
@@ -271,22 +363,21 @@ onMounted(loadTask)
v-model="inputQuestion" v-model="inputQuestion"
class="input-box" class="input-box"
rows="1" rows="1"
:disabled="loading" :disabled="loading || taskLoading"
placeholder="给模型发送消息..." placeholder="给模型发送消息..."
@keydown.enter.exact.prevent="handleSend" @keydown.enter.exact.prevent="handleSend"
@input="autoResize" @input="autoResize"
/> />
<button <button
class="send-btn" class="send-btn"
:class="{ active: inputQuestion.trim() && !loading }" :class="{ active: inputQuestion.trim() && !loading && !taskLoading }"
:disabled="!inputQuestion.trim() || loading" :disabled="!inputQuestion.trim() || loading || taskLoading"
@click="handleSend" @click="handleSend"
> >
<i class="fa fa-arrow-up" /> <i class="fa fa-arrow-up" />
</button> </button>
</div> </div>
</div> </div>
<div class="footer-hint">内容由 AI 生成请仔细甄别</div>
</footer> </footer>
<!-- 设置抽屉系统提示词等 --> <!-- 设置抽屉系统提示词等 -->
@@ -713,9 +804,18 @@ onMounted(loadTask)
} }
} }
.footer-hint { .loading-hint {
margin-top: 12px; margin-bottom: 10px;
font-size: 12px; padding: 6px 14px;
color: #9ca3af; font-size: 13px;
color: #b45309;
background: #fef3c7;
border-radius: 8px;
text-align: center;
&.error {
color: #b91c1c;
background: #fee2e2;
}
} }
</style> </style>

View File

@@ -1,10 +1,12 @@
<script setup lang="ts"> <script setup lang="ts">
import { ref, reactive, computed, onMounted } from 'vue' import { ref, reactive, computed, onMounted, watch } from 'vue'
import { useRouter } from 'vue-router' import { useRouter } from 'vue-router'
import { ElMessage, type FormInstance, type FormRules } from 'element-plus' import { ElMessage, type FormInstance, type FormRules } from 'element-plus'
import PageCard from '@/components/PageCard.vue' import PageCard from '@/components/PageCard.vue'
import { getModelList, getTrainedModels } from '@/api/modules/model' import { getModelList, getTrainedModels } from '@/api/modules/model'
import { getSystemInfo } from '@/api/modules/system' import { getSystemInfo } from '@/api/modules/system'
import { getComputeNodes, type ComputeNode } from '@/api/modules/compute'
import { createCompare, loadCompare } from '@/api/modules/compare'
import type { ModelItem, TrainedModel, GpuInfo } from '@/types' import type { ModelItem, TrainedModel, GpuInfo } from '@/types'
const router = useRouter() const router = useRouter()
@@ -15,6 +17,7 @@ const startupStatus = ref('')
const dbModels = ref<ModelItem[]>([]) const dbModels = ref<ModelItem[]>([])
const trainedModels = ref<TrainedModel[]>([]) const trainedModels = ref<TrainedModel[]>([])
const gpus = ref<GpuInfo[]>([]) const gpus = ref<GpuInfo[]>([])
const computeNodes = ref<ComputeNode[]>([])
/** 可选模型(下拉用,区分本地/已训练两类) */ /** 可选模型(下拉用,区分本地/已训练两类) */
interface SelectableModel { interface SelectableModel {
@@ -24,6 +27,8 @@ interface SelectableModel {
name: string name: string
source: 'database' | 'trained' source: 'database' | 'trained'
model_path: string model_path: string
compute_node_id?: string
compute_node_name?: string
merged?: boolean merged?: boolean
merging?: boolean merging?: boolean
disabled?: boolean disabled?: boolean
@@ -48,13 +53,25 @@ const trainedOptions = computed<SelectableModel[]>(() =>
name: m.name, name: m.name,
source: 'trained', source: 'trained',
model_path: m.merged_path || m.base_model_path || '', model_path: m.merged_path || m.base_model_path || '',
compute_node_id: m.compute_node_id,
compute_node_name: m.compute_node_name,
merged: m.merged, merged: m.merged,
merging: m.merging, merging: m.merging,
disabled: m.merged === false, disabled: m.merged === false,
})), })),
) )
/** key → 模型映射,便于取选中项 */ /** 仅显示在线算力节点上的空闲 GPU */
const onlineNodeIds = computed(() => new Set(
computeNodes.value
.filter((n) => n.enabled && n.scheduler_status === 'online')
.map((n) => n.id),
))
const idleGpus = computed(() =>
gpus.value.filter(
(g) => g.status === 'idle' && (!g.node_id || onlineNodeIds.value.has(g.node_id)),
),
)
const modelMap = computed<Record<string, SelectableModel>>(() => { const modelMap = computed<Record<string, SelectableModel>>(() => {
const map: Record<string, SelectableModel> = {} const map: Record<string, SelectableModel> = {}
for (const m of [...dbOptions.value, ...trainedOptions.value]) map[m.key] = m for (const m of [...dbOptions.value, ...trainedOptions.value]) map[m.key] = m
@@ -67,7 +84,7 @@ const form = reactive({
/** 选中的模型 key单选 */ /** 选中的模型 key单选 */
model_key: '', model_key: '',
/** 使用的 GPU */ /** 使用的 GPU */
gpu_id: 0, gpu_key: '',
}) })
const rules: FormRules = { const rules: FormRules = {
@@ -77,6 +94,13 @@ const rules: FormRules = {
/** 当前选中的模型对象 */ /** 当前选中的模型对象 */
const selectedModel = computed(() => modelMap.value[form.model_key]) const selectedModel = computed(() => modelMap.value[form.model_key])
const selectedGpu = computed(() => idleGpus.value.find((g) => `${g.node_id || ''}:${g.id ?? 0}` === form.gpu_key))
watch(selectedModel, (model) => {
if (!model?.compute_node_id) return
const gpu = idleGpus.value.find((item) => item.node_id === model.compute_node_id)
if (gpu) form.gpu_key = `${gpu.node_id || ''}:${gpu.id ?? 0}`
})
async function handleSubmit() { async function handleSubmit() {
if (!formRef.value) return if (!formRef.value) return
@@ -88,18 +112,47 @@ async function handleSubmit() {
return return
} }
submitting.value = true submitting.value = true
startupStatus.value = '正在启动模型服务...' startupStatus.value = '正在创建推理任务...'
try { try {
// 当前为 mock 环境:不创建任务、不启动后端服务, if (!m.model_path) {
// 用假数据直通进入对话界面(模型名通过 query 传递)。 ElMessage.warning('当前模型未配置算力节点可访问路径,请先在模型管理中维护模型路径')
// 接入真实后端后,可在此恢复 createCompare / startModelsInBackground / monitorStartup 流程。 return
await new Promise((resolve) => setTimeout(resolve, 1200)) }
ElMessage.success('模型已启动') // Step 1: 创建推理任务记录
router.push({ const taskResult = await createCompare({
path: '/model-inference/chat/mock', name: form.name || m.name,
query: { model: m.name }, description: form.description,
status: 'pending',
models: [
{
model_id: String(m.id),
model_name: m.name,
model_path: m.model_path,
source: m.source,
gpu_id: selectedGpu.value?.id ?? 0,
node_id: selectedGpu.value?.node_id || m.compute_node_id,
node_name: selectedGpu.value?.node_name || m.compute_node_name,
},
],
}) })
const taskId = taskResult?.id || 'unknown'
// Step 2: 统一通过推理任务加载接口异步派发模型加载,状态会落到列表记录中。
startupStatus.value = '正在启动模型服务,首次加载可能需要数分钟...'
const loadResult: any = await loadCompare(taskId)
if (loadResult?.status === 'failed' || loadResult?.error) {
ElMessage.warning(`模型加载失败:${loadResult?.error || '请检查算力节点日志'}`)
router.push('/model-inference')
return
}
// 加载为异步派发,回到列表页可看到“启动中 → 已就绪”的状态流转
ElMessage.success('模型加载中,就绪后即可对话')
router.push('/model-inference')
} catch (e: any) {
const reason = e?.message || e?.toString() || '未知错误'
ElMessage.warning(`推理服务启动失败:${reason}`)
} finally { } finally {
submitting.value = false submitting.value = false
startupStatus.value = '' startupStatus.value = ''
@@ -113,16 +166,21 @@ function handleCancel() {
async function loadData() { async function loadData() {
try { try {
const [db, trained, sys] = await Promise.all([ const [db, trained, sys, nodes] = await Promise.all([
getModelList(), getModelList(),
getTrainedModels(), getTrainedModels(),
getSystemInfo(), getSystemInfo(),
getComputeNodes(),
]) ])
dbModels.value = db || [] dbModels.value = db || []
trainedModels.value = trained?.models || [] trainedModels.value = trained?.models || []
gpus.value = sys?.gpu || [] gpus.value = sys?.gpu || []
// 默认选中第一个 GPU computeNodes.value = nodes || []
if (gpus.value.length > 0) form.gpu_id = 0 // 默认选中第一个空闲 GPU
if (idleGpus.value.length > 0) {
const firstGpu = idleGpus.value[0]
form.gpu_key = `${firstGpu.node_id || ''}:${firstGpu.id ?? 0}`
}
} catch { } catch {
// ignore // ignore
} }
@@ -170,12 +228,12 @@ onMounted(loadData)
</el-form-item> </el-form-item>
<el-form-item label="GPU"> <el-form-item label="GPU">
<el-select v-model="form.gpu_id" style="width: 400px"> <el-select v-model="form.gpu_key" style="width: 400px">
<el-option <el-option
v-for="(g, idx) in gpus" v-for="g in idleGpus"
:key="idx" :key="`${g.node_id || ''}:${g.id ?? 0}`"
:label="`${g.name} (GPU${idx})`" :label="`${g.node_name || g.node_code || '算力节点'} / ${g.name} (GPU${g.id ?? 0}) [空闲]`"
:value="idx" :value="`${g.node_id || ''}:${g.id ?? 0}`"
/> />
</el-select> </el-select>
</el-form-item> </el-form-item>

View File

@@ -7,10 +7,8 @@ import { usePolling } from '@/composables/usePolling'
import { import {
getCompareList, getCompareList,
deleteCompare, deleteCompare,
getCompare,
loadCompare, loadCompare,
unloadCompare, unloadCompare,
stopModelByPid,
} from '@/api/modules/compare' } from '@/api/modules/compare'
import type { CompareTask, LoadedModel } from '@/types' import type { CompareTask, LoadedModel } from '@/types'
import { statusLabel, statusTagType } from '@/utils/status' import { statusLabel, statusTagType } from '@/utils/status'
@@ -86,26 +84,17 @@ async function handleLoad(row: any) {
delayedRefreshTimer = setTimeout(loadData, 1000) delayedRefreshTimer = setTimeout(loadData, 1000)
} }
/** 卸载推理任务 */ /** 释放推理任务(停止模型服务,释放算力节点 GPU 显存) */
async function handleUnload(row: any) { async function handleUnload(row: any) {
await ElMessageBox.confirm('确定要停止模型服务吗?', '确认停止', { type: 'warning' }) await ElMessageBox.confirm('确定要释放模型服务吗?将停止模型进程并释放 GPU 显存。', '确认释放', { type: 'warning' })
await unloadCompare(row.id) await unloadCompare(row.id)
ElMessage.success('已停止模型服务') ElMessage.success('已释放模型服务')
loadData() loadData()
} }
/** 删除(先停止进程 */ /** 删除(后端删除内部会 best-effort 释放算力节点,这里直接删记录 */
async function handleDelete(row: any) { async function handleDelete(row: any) {
// 先尝试停止已加载的模型进程 await ElMessageBox.confirm('确定要删除该推理记录吗?将先释放算力节点再删除。', '确认删除', { type: 'warning' })
const task = await getCompare(row.id).catch(() => null)
if (task?.load_status) {
const models = parseLoadedModels(task as CompareTask)
for (const m of models) {
if (m.pid) {
await stopModelByPid(m.pid).catch(() => {})
}
}
}
await deleteCompare(row.id) await deleteCompare(row.id)
dataList.value = dataList.value.filter((item) => item.id !== row.id) dataList.value = dataList.value.filter((item) => item.id !== row.id)
await loadData(true) await loadData(true)
@@ -180,7 +169,7 @@ onUnmounted(() => {
<i class="fa fa-comments-o" style="margin-right: 4px" />对话 <i class="fa fa-comments-o" style="margin-right: 4px" />对话
</el-button> </el-button>
<el-button type="warning" link size="small" @click="handleUnload(row)"> <el-button type="warning" link size="small" @click="handleUnload(row)">
<i class="fa fa-stop-circle-o" style="margin-right: 4px" />停止 <i class="fa fa-stop-circle-o" style="margin-right: 4px" />释放
</el-button> </el-button>
</template> </template>
<template v-else> <template v-else>

View File

@@ -18,9 +18,12 @@ const trainedModels = ref<TrainedModel[]>([])
const currentModel = computed(() => trainedModels.value.find((m) => m.name === modelName.value)) const currentModel = computed(() => trainedModels.value.find((m) => m.name === modelName.value))
const form = reactive({ const form = reactive({
trained_model_id: '',
model_name: modelName.value, model_name: modelName.value,
train_method: method.value, train_method: method.value,
base_model_path: '', base_model_path: '',
adapter_path: '',
compute_node_id: '',
}) })
async function loadModel() { async function loadModel() {
@@ -28,23 +31,30 @@ async function loadModel() {
const res = await getTrainedModels() const res = await getTrainedModels()
trainedModels.value = res?.models || [] trainedModels.value = res?.models || []
const target = trainedModels.value.find((m) => m.name === modelName.value) const target = trainedModels.value.find((m) => m.name === modelName.value)
form.trained_model_id = target?.id == null ? '' : String(target.id)
form.base_model_path = target?.base_model_path || '' form.base_model_path = target?.base_model_path || ''
form.adapter_path = target?.artifact_dir || target?.adapter_path || target?.merged_path || ''
form.compute_node_id = target?.compute_node_id || ''
} catch { } catch {
// ignore // ignore
} }
} }
async function handleMerge() { async function handleMerge() {
if (!form.model_name || !form.base_model_path) { if (!form.model_name || !form.base_model_path || !form.adapter_path) {
ElMessage.warning('缺少模型信息') ElMessage.warning('缺少模型信息')
return return
} }
merging.value = true merging.value = true
try { try {
await mergeModel({ await mergeModel({
trained_model_id: form.trained_model_id || form.model_name,
model_name: form.model_name, model_name: form.model_name,
train_method: form.train_method, train_method: form.train_method,
base_model_path: form.base_model_path, base_model_path: form.base_model_path,
adapter_path: form.adapter_path,
compute_node_id: form.compute_node_id,
output_model_name: `${form.model_name}-merged`,
}) })
ElMessage.success('合并成功') ElMessage.success('合并成功')
router.push('/model-manage') router.push('/model-manage')
@@ -82,6 +92,9 @@ onMounted(loadModel)
<el-form-item label="基座模型路径"> <el-form-item label="基座模型路径">
<el-input v-model="form.base_model_path" placeholder="基座模型路径" /> <el-input v-model="form.base_model_path" placeholder="基座模型路径" />
</el-form-item> </el-form-item>
<el-form-item label="Adapter 路径">
<el-input v-model="form.adapter_path" placeholder="LoRA Adapter 权重目录" />
</el-form-item>
<el-form-item> <el-form-item>
<el-button type="primary" :loading="merging" @click="handleMerge"> <el-button type="primary" :loading="merging" @click="handleMerge">

View File

@@ -5,7 +5,8 @@ import { ElMessage } from 'element-plus'
import DataTablePage from '@/components/DataTablePage.vue' import DataTablePage from '@/components/DataTablePage.vue'
import AclDialog from '@/components/AclDialog.vue' import AclDialog from '@/components/AclDialog.vue'
import { getProject, getProjectMembers, addProjectMember, removeProjectMember, type Project, type ProjectMember } from '@/api/modules/project' import { getProject, getProjectMembers, addProjectMember, removeProjectMember, type Project, type ProjectMember } from '@/api/modules/project'
import { getUsers, type SystemUser } from '@/api/modules/system' import { getUsers } from '@/api/modules/system'
import type { SystemUser } from '@/types'
const route = useRoute() const route = useRoute()
const router = useRouter() const router = useRouter()
@@ -56,6 +57,10 @@ async function removeMember(userId: string) {
load() load()
} }
function asProjectMember(row: unknown): ProjectMember {
return row as ProjectMember
}
onMounted(() => { onMounted(() => {
loadUsers() loadUsers()
load() load()
@@ -94,7 +99,7 @@ onMounted(() => {
<el-table-column prop="create_time" label="加入时间" min-width="180" /> <el-table-column prop="create_time" label="加入时间" min-width="180" />
</template> </template>
<template #actions="{ row }"> <template #actions="{ row }">
<el-button link type="danger" @click="removeMember(row.user_id)">移除</el-button> <el-button link type="danger" @click="removeMember(asProjectMember(row).user_id)">移除</el-button>
</template> </template>
</DataTablePage> </DataTablePage>
</el-card> </el-card>

View File

@@ -1,10 +1,10 @@
<script setup lang="ts"> <script setup lang="ts">
import { computed, onMounted, ref } from 'vue' import { computed, onMounted, ref } from 'vue'
import { useRouter } from 'vue-router' import { useRouter } from 'vue-router'
import { ElMessage } from 'element-plus' import { ElMessage, ElMessageBox } from 'element-plus'
import { Plus } from '@element-plus/icons-vue' import { Plus } from '@element-plus/icons-vue'
import DataTablePage from '@/components/DataTablePage.vue' import DataTablePage from '@/components/DataTablePage.vue'
import { createProject, getProjects, type Project } from '@/api/modules/project' import { createProject, deleteProject, getProjects, type Project } from '@/api/modules/project'
import { getTenants, type Tenant } from '@/api/modules/tenant' import { getTenants, type Tenant } from '@/api/modules/tenant'
const router = useRouter() const router = useRouter()
@@ -13,13 +13,24 @@ const projects = ref<Project[]>([])
const tenants = ref<Tenant[]>([]) const tenants = ref<Tenant[]>([])
const tenantId = ref('default') const tenantId = ref('default')
const showCreate = ref(false) const showCreate = ref(false)
const form = ref({ name: '', code: '', description: '', tenant_id: 'default' }) const form = ref({ name: '', code: '', description: '', tenant_id: '' })
const tenantOptions = computed(() => [ const tenantOptions = computed(() => [
{ label: 'default', value: 'default' }, { label: 'default', value: 'default' },
...tenants.value.map((t) => ({ label: t.name, value: t.id })), ...tenants.value.map((t) => ({ label: t.name, value: t.id })),
]) ])
const tenantCodeOptions = computed(() =>
tenants.value.map((t) => ({ label: t.code, value: t.code, tenantId: t.id }))
)
function onTenantCodeChange(code: string) {
const tenant = tenants.value.find((t) => t.code === code)
if (tenant) {
form.value.tenant_id = tenant.id
}
}
async function load() { async function load() {
loading.value = true loading.value = true
try { try {
@@ -41,15 +52,34 @@ function openDetail(id: string) {
router.push(`/projects/${id}`) router.push(`/projects/${id}`)
} }
function asProject(row: unknown): Project {
return row as Project
}
async function submitCreate() { async function submitCreate() {
if (!form.value.name || !form.value.code) { if (!form.value.name || !form.value.tenant_id) {
ElMessage.warning('请填写项目名与编码') ElMessage.warning('请填写项目名与编码ID')
return return
} }
await createProject({ ...form.value }) await createProject({ ...form.value })
ElMessage.success('项目创建成功') ElMessage.success('项目创建成功')
showCreate.value = false showCreate.value = false
form.value = { name: '', code: '', description: '', tenant_id: 'default' } form.value = { name: '', code: '', description: '', tenant_id: '' }
load()
}
async function handleDelete(row: Project) {
try {
await ElMessageBox.confirm(
`确定要删除项目「${row.name}」吗?删除后相关数据将无法恢复。`,
'删除确认',
{ confirmButtonText: '确定删除', cancelButtonText: '取消', type: 'warning' },
)
} catch {
return
}
await deleteProject(row.id)
ElMessage.success('项目已删除')
load() load()
} }
@@ -61,7 +91,7 @@ onMounted(() => {
<template> <template>
<div class="page"> <div class="page">
<DataTablePage title="项目空间" :data="projects" :loading="loading" searchable search-fields="name,code"> <DataTablePage title="项目空间" :data="projects" :loading="loading" searchable :search-fields="['name', 'code']">
<template #toolbar-extra> <template #toolbar-extra>
<el-select v-model="tenantId" placeholder="租户" style="width: 160px" @change="load"> <el-select v-model="tenantId" placeholder="租户" style="width: 160px" @change="load">
<el-option v-for="t in tenantOptions" :key="t.value" :label="t.label" :value="t.value" /> <el-option v-for="t in tenantOptions" :key="t.value" :label="t.label" :value="t.value" />
@@ -69,27 +99,25 @@ onMounted(() => {
<el-button type="primary" :icon="Plus" @click="showCreate = true">新建项目</el-button> <el-button type="primary" :icon="Plus" @click="showCreate = true">新建项目</el-button>
</template> </template>
<template #columns> <template #columns>
<el-table-column prop="name" label="项目名" min-width="140" /> <el-table-column prop="name" label="项目名" min-width="140" />
<el-table-column prop="code" label="编码" min-width="100" /> <el-table-column prop="code" label="编码 ID" min-width="100" />
<el-table-column prop="status" label="状态" min-width="100" /> <el-table-column prop="status" label="状态" min-width="100" />
<el-table-column prop="description" label="描述" min-width="200" show-overflow-tooltip /> <el-table-column prop="description" label="描述" min-width="200" show-overflow-tooltip />
<el-table-column prop="create_time" label="创建时间" min-width="180" /> <el-table-column prop="create_time" label="创建时间" min-width="180" />
</template> </template>
<template #actions="{ row }"> <template #actions="{ row }">
<el-button link type="primary" @click="openDetail(row.id)">详情</el-button> <el-button link type="primary" @click="openDetail(row.id)">详情</el-button>
<el-button link type="danger" @click="handleDelete(asProject(row))">删除</el-button>
</template> </template>
</DataTablePage> </DataTablePage>
<el-dialog v-model="showCreate" title="新建项目" width="520px"> <el-dialog v-model="showCreate" title="新建项目" width="520px">
<el-form label-width="90px"> <el-form label-width="90px">
<el-form-item label="名称" required> <el-form-item label="名称" required>
<el-input v-model="form.name" placeholder="项目名" /> <el-input v-model="form.name" placeholder="项目名" />
</el-form-item> </el-form-item>
<el-form-item label="编码" required> <el-form-item label="编码 ID" required>
<el-input v-model="form.code" placeholder="project code" /> <el-select v-model="form.tenant_id" style="width: 100%" placeholder="选择租户编码">
</el-form-item> <el-option v-for="t in tenantCodeOptions" :key="t.value" :label="t.label" :value="t.tenantId" />
<el-form-item label="租户">
<el-select v-model="form.tenant_id" style="width: 100%">
<el-option v-for="t in tenantOptions" :key="t.value" :label="t.label" :value="t.value" />
</el-select> </el-select>
</el-form-item> </el-form-item>
<el-form-item label="描述"> <el-form-item label="描述">

View File

@@ -8,13 +8,14 @@ import TrainingTaskOverview from './training-log/TrainingTaskOverview.vue'
import { usePolling } from '@/composables/usePolling' import { usePolling } from '@/composables/usePolling'
import '@/plugins/echarts-training-log' import '@/plugins/echarts-training-log'
import { useModelsStore } from '@/stores/models' import { useModelsStore } from '@/stores/models'
import { getFineTune, getFineTuneDiagnostics, getFineTuneLogs, type TrainingDiagnostic } from '@/api/modules/fineTune' import { getFineTune, getFineTuneDiagnostics, getFineTuneLogs, getFineTuneMetrics, type TrainingDiagnostic } from '@/api/modules/fineTune'
import { getTrainingLogFiles, getTrainingLogContent } from '@/api/modules/log' import { getTrainingLogFiles, getTrainingLogContent } from '@/api/modules/log'
import { getDataset } from '@/api/modules/dataset' import { getDataset } from '@/api/modules/dataset'
import { getSystemInfo } from '@/api/modules/system' import { getSystemInfo } from '@/api/modules/system'
import { TRAIN_TYPE_MAP, TRAIN_METHOD_MAP } from '@/constants' import { TRAIN_TYPE_MAP, TRAIN_METHOD_MAP } from '@/constants'
import { import {
buildMetricChartOption, buildMetricChartOption,
metricsFromApi,
parseTrainingLog, parseTrainingLog,
resolveTrainingLogFile, resolveTrainingLogFile,
} from './training-log/trainingLogModel' } from './training-log/trainingLogModel'
@@ -42,6 +43,7 @@ const loading = ref(true)
// 训练指标数据ECharts 接收 number[],下标即 step // 训练指标数据ECharts 接收 number[],下标即 step
const metricData = reactive({ const metricData = reactive({
steps: [] as number[],
loss: [] as number[], loss: [] as number[],
gradNorm: [] as number[], gradNorm: [] as number[],
lr: [] as number[], lr: [] as number[],
@@ -62,9 +64,9 @@ const gpuExpanded = ref(false)
let refreshInFlight = false let refreshInFlight = false
/** 三个曲线的 ECharts 配置(响应式,数据变化自动重绘) */ /** 三个曲线的 ECharts 配置(响应式,数据变化自动重绘) */
const lossChartOption = computed(() => buildMetricChartOption('Loss', metricData.loss, '#4f46e5')) const lossChartOption = computed(() => buildMetricChartOption('Loss', metricData.loss, metricData.steps, '#4f46e5'))
const gradChartOption = computed(() => buildMetricChartOption('Grad Norm', metricData.gradNorm, '#3b82f6')) const gradChartOption = computed(() => buildMetricChartOption('Grad Norm', metricData.gradNorm, metricData.steps, '#3b82f6'))
const lrChartOption = computed(() => buildMetricChartOption('Learning Rate', metricData.lr, '#14b8a6', true)) const lrChartOption = computed(() => buildMetricChartOption('Learning Rate', metricData.lr, metricData.steps, '#14b8a6', true))
const baseModelName = computed(() => task.value?.base_model != null const baseModelName = computed(() => task.value?.base_model != null
? modelsStore.getModelName(task.value.base_model) ? modelsStore.getModelName(task.value.base_model)
: '未配置') : '未配置')
@@ -77,10 +79,10 @@ const trainingMethodName = computed(() => task.value?.train_method
const taskGpuLabel = computed(() => task.value?.gpus?.length const taskGpuLabel = computed(() => task.value?.gpus?.length
? task.value.gpus.map((gpuId) => `GPU ${gpuId}`).join('、') ? task.value.gpus.map((gpuId) => `GPU ${gpuId}`).join('、')
: '未配置') : '未配置')
const latestLoss = computed(() => metricData.loss[metricData.loss.length - 1]) const latestLoss = computed(() => lastFinite(metricData.loss))
const latestGradNorm = computed(() => metricData.gradNorm[metricData.gradNorm.length - 1]) const latestGradNorm = computed(() => lastFinite(metricData.gradNorm))
const latestLearningRate = computed(() => metricData.lr[metricData.lr.length - 1]) const latestLearningRate = computed(() => lastFinite(metricData.lr))
const latestEpoch = computed(() => metricData.epoch[metricData.epoch.length - 1]) const latestEpoch = computed(() => lastFinite(metricData.epoch))
const logLineCount = computed(() => logContent.value ? logContent.value.split(/\r?\n/).length : 0) const logLineCount = computed(() => logContent.value ? logContent.value.split(/\r?\n/).length : 0)
const taskGpuItems = computed<TaskGpuItem[]>(() => (task.value?.gpus ?? []).map((gpuId) => { const taskGpuItems = computed<TaskGpuItem[]>(() => (task.value?.gpus ?? []).map((gpuId) => {
const index = Number(gpuId) const index = Number(gpuId)
@@ -123,7 +125,7 @@ const gpuRefreshState = computed(() => {
}) })
return gpuLoadError.value return gpuLoadError.value
? `更新失败 · 最后更新 ${updateTime}` ? `更新失败 · 最后更新 ${updateTime}`
: `${updateTime} 更新 · 每 5 秒刷新` : `${updateTime} 更新 · 每 3 秒刷新`
}) })
function formatMetric(value?: number, scientific = false) { function formatMetric(value?: number, scientific = false) {
@@ -131,6 +133,13 @@ function formatMetric(value?: number, scientific = false) {
return scientific ? value.toExponential(2) : value.toFixed(4).replace(/0+$/, '').replace(/\.$/, '') return scientific ? value.toExponential(2) : value.toFixed(4).replace(/0+$/, '').replace(/\.$/, '')
} }
function lastFinite(values: number[]) {
for (let index = values.length - 1; index >= 0; index -= 1) {
if (Number.isFinite(values[index])) return values[index]
}
return undefined
}
function safePercent(value?: number) { function safePercent(value?: number) {
return Math.round(Math.min(100, Math.max(0, Number(value || 0)))) return Math.round(Math.min(100, Math.max(0, Number(value || 0))))
} }
@@ -216,6 +225,7 @@ const isLoraMethod = computed(() =>
function applyLogContent(content: string) { function applyLogContent(content: string) {
const parsed = parseTrainingLog(content) const parsed = parseTrainingLog(content)
logContent.value = content logContent.value = content
metricData.steps = parsed.metrics.steps
metricData.loss = parsed.metrics.loss metricData.loss = parsed.metrics.loss
metricData.gradNorm = parsed.metrics.gradNorm metricData.gradNorm = parsed.metrics.gradNorm
metricData.lr = parsed.metrics.lr metricData.lr = parsed.metrics.lr
@@ -223,6 +233,26 @@ function applyLogContent(content: string) {
Object.assign(summary, parsed.summary) Object.assign(summary, parsed.summary)
} }
function applyMetricData(metrics = { steps: [] as number[], loss: [] as number[], gradNorm: [] as number[], lr: [] as number[], epoch: [] as number[] }) {
metricData.steps = metrics.steps
metricData.loss = metrics.loss
metricData.gradNorm = metrics.gradNorm
metricData.lr = metrics.lr
metricData.epoch = metrics.epoch
}
async function loadMetrics(currentTask: FineTuneTask) {
try {
const points = await getFineTuneMetrics(currentTask.id)
const parsed = metricsFromApi(points || [])
if (parsed.loss.length || parsed.gradNorm.length || parsed.lr.length) {
applyMetricData(parsed)
}
} catch {
// 日志解析结果会作为兜底曲线数据。
}
}
async function loadLog(currentTask: FineTuneTask) { async function loadLog(currentTask: FineTuneTask) {
try { try {
const runtime = await getFineTuneLogs(currentTask.id, { tail_lines: 800 }) const runtime = await getFineTuneLogs(currentTask.id, { tail_lines: 800 })
@@ -277,6 +307,7 @@ async function refreshAll() {
? loadDataset(currentTask.train_dataset_id) ? loadDataset(currentTask.train_dataset_id)
: Promise.resolve() : Promise.resolve()
await Promise.all([datasetPromise, loadLog(currentTask), loadGpuStatus(), loadDiagnostics(currentTask)]) await Promise.all([datasetPromise, loadLog(currentTask), loadGpuStatus(), loadDiagnostics(currentTask)])
await loadMetrics(currentTask)
} finally { } finally {
loading.value = false loading.value = false
refreshInFlight = false refreshInFlight = false
@@ -523,7 +554,7 @@ onMounted(async () => {
<!-- 训练曲线 --> <!-- 训练曲线 -->
<PageCard class="metrics-panel" title="训练曲线" subtitle="持续监控模型收敛情况与学习率变化"> <PageCard class="metrics-panel" title="训练曲线" subtitle="持续监控模型收敛情况与学习率变化">
<template #extra><span class="refresh-state"> 5 秒刷新</span></template> <template #extra><span class="refresh-state"> 3 秒刷新</span></template>
<div class="chart-list" aria-label="训练指标曲线"> <div class="chart-list" aria-label="训练指标曲线">
<section class="chart-section"> <section class="chart-section">
<div class="chart-section-header"> <div class="chart-section-header">
@@ -560,7 +591,7 @@ onMounted(async () => {
<!-- 原始日志 --> <!-- 原始日志 -->
<PageCard class="log-card" title="训练日志" subtitle="查看训练任务的原始运行输出"> <PageCard class="log-card" title="训练日志" subtitle="查看训练任务的原始运行输出">
<template #extra><span class="log-meta">{{ logLineCount }} · 5 秒刷新</span></template> <template #extra><span class="log-meta">{{ logLineCount }} · 3 秒刷新</span></template>
<pre class="log-pre">{{ logContent || '暂无日志' }}</pre> <pre class="log-pre">{{ logContent || '暂无日志' }}</pre>
</PageCard> </PageCard>
</template> </template>

View File

@@ -54,6 +54,18 @@ function isSelf(row: SystemUser) {
return row.username === currentUsername.value return row.username === currentUsername.value
} }
function asSystemUser(row: unknown): SystemUser {
return row as SystemUser
}
function userPermissions(row: unknown): PermissionCode[] {
return (asSystemUser(row).permissions || []) as PermissionCode[]
}
function permissionLabel(code: PermissionCode): string {
return PERMISSION_LABELS[code] || code
}
// ---------- 启停 ---------- // ---------- 启停 ----------
async function toggleStatus(row: SystemUser, next: boolean) { async function toggleStatus(row: SystemUser, next: boolean) {
const nextStatus: UserStatus = next ? 'active' : 'disabled' const nextStatus: UserStatus = next ? 'active' : 'disabled'
@@ -162,31 +174,31 @@ async function removeUser(row: SystemUser) {
<el-table-column prop="role" label="角色" width="120" /> <el-table-column prop="role" label="角色" width="120" />
<el-table-column label="状态" width="130"> <el-table-column label="状态" width="130">
<template #default="{ row }"> <template #default="{ row }">
<el-tag :type="statusTagType(row.status)" size="small">{{ statusLabel(row.status) }}</el-tag> <el-tag :type="statusTagType(asSystemUser(row).status)" size="small">{{ statusLabel(asSystemUser(row).status) }}</el-tag>
</template> </template>
</el-table-column> </el-table-column>
<el-table-column label="页面权限" min-width="160"> <el-table-column label="页面权限" min-width="160">
<template #default="{ row }"> <template #default="{ row }">
<el-tag <el-tag
v-for="p in (row.permissions || []).slice(0, 3)" v-for="p in userPermissions(row).slice(0, 3)"
:key="p" :key="p"
size="small" size="small"
type="info" type="info"
class="perm-tag" class="perm-tag"
>{{ PERMISSION_LABELS[p] || p }}</el-tag> >{{ permissionLabel(p) }}</el-tag>
<span v-if="(row.permissions || []).length > 3" class="perm-more"> <span v-if="userPermissions(row).length > 3" class="perm-more">
+{{ (row.permissions || []).length - 3 }} +{{ userPermissions(row).length - 3 }}
</span> </span>
<span v-if="!(row.permissions || []).length" class="perm-more"></span> <span v-if="!userPermissions(row).length" class="perm-more"></span>
</template> </template>
</el-table-column> </el-table-column>
<el-table-column prop="create_time" label="创建时间" min-width="180" /> <el-table-column prop="create_time" label="创建时间" min-width="180" />
<el-table-column label="操作" width="260" fixed="right"> <el-table-column label="操作" width="260" fixed="right">
<template #default="{ row }"> <template #default="{ row }">
<el-switch <el-switch
:model-value="row.status === 'active'" :model-value="asSystemUser(row).status === 'active'"
:disabled="row.protected || isSelf(row)" :disabled="asSystemUser(row).protected || isSelf(asSystemUser(row))"
@change="(v: any) => toggleStatus(row, v)" @change="(v: any) => toggleStatus(asSystemUser(row), v)"
inline-prompt inline-prompt
active-text="启用" active-text="启用"
inactive-text="停用" inactive-text="停用"
@@ -194,19 +206,19 @@ async function removeUser(row: SystemUser) {
<el-button <el-button
link link
type="primary" type="primary"
:disabled="row.protected" :disabled="asSystemUser(row).protected"
@click="openResetPwd(row)" @click="openResetPwd(asSystemUser(row))"
>重置密码</el-button> >重置密码</el-button>
<el-button <el-button
link link
type="primary" type="primary"
@click="openPerms(row)" @click="openPerms(asSystemUser(row))"
>页面权限</el-button> >页面权限</el-button>
<el-button <el-button
link link
type="danger" type="danger"
:disabled="row.protected || isSelf(row)" :disabled="asSystemUser(row).protected || isSelf(asSystemUser(row))"
@click="removeUser(row)" @click="removeUser(asSystemUser(row))"
>删除</el-button> >删除</el-button>
</template> </template>
</el-table-column> </el-table-column>

View File

@@ -1,7 +1,9 @@
import type { EChartsOption } from 'echarts' import type { EChartsOption } from 'echarts'
import type { FineTuneTask, TrainingLogFile } from '@/types' import type { FineTuneTask, TrainingLogFile } from '@/types'
import type { FineTuneMetricPoint } from '@/api/modules/fineTune'
export interface TrainingMetricData { export interface TrainingMetricData {
steps: number[]
loss: number[] loss: number[]
gradNorm: number[] gradNorm: number[]
lr: number[] lr: number[]
@@ -26,7 +28,7 @@ function escapeRegExp(value: string) {
} }
function extractNumber(source: string, key: string) { function extractNumber(source: string, key: string) {
const match = source.match(new RegExp(`['"]?${escapeRegExp(key)}['"]?\\s*:\\s*(${NUMBER_SOURCE})`, 'i')) const match = source.match(new RegExp(`['"]?${escapeRegExp(key)}['"]?\\s*(?:=|:)\\s*(${NUMBER_SOURCE})`, 'i'))
return match ? Number(match[1]) : undefined return match ? Number(match[1]) : undefined
} }
@@ -57,24 +59,44 @@ export function resolveTrainingLogFile(
/** 解析日志中的逐步指标。字段顺序和常见数值格式均不受限制。 */ /** 解析日志中的逐步指标。字段顺序和常见数值格式均不受限制。 */
export function parseTrainingMetrics(text: string): TrainingMetricData { export function parseTrainingMetrics(text: string): TrainingMetricData {
const metrics: TrainingMetricData = { loss: [], gradNorm: [], lr: [], epoch: [] } const metrics: TrainingMetricData = { steps: [], loss: [], gradNorm: [], lr: [], epoch: [] }
const blocks = text.match(/\{[^{}\r\n]*\}/g) || [] const candidates = text
.split(/\r?\n/)
.flatMap((line) => {
const blocks = line.match(/\{[^{}\r\n]*\}/g)
return blocks?.length ? blocks.map((block) => `${line} ${block}`) : [line]
})
for (const block of blocks) { for (const [index, line] of candidates.entries()) {
const loss = extractNumber(block, 'loss') const loss = extractNumber(line, 'loss')
const gradNorm = extractNumber(block, 'grad_norm') const gradNorm = extractNumber(line, 'grad_norm')
const learningRate = extractNumber(block, 'learning_rate') const learningRate = extractNumber(line, 'learning_rate')
const epoch = extractNumber(block, 'epoch') const epoch = extractNumber(line, 'epoch')
if (loss == null || gradNorm == null || learningRate == null) continue if (loss == null && gradNorm == null && learningRate == null) continue
metrics.loss.push(loss) metrics.steps.push(extractNumber(line, 'step') ?? metrics.steps.length + index + 1)
metrics.gradNorm.push(gradNorm) metrics.loss.push(loss ?? Number.NaN)
metrics.lr.push(learningRate) metrics.gradNorm.push(gradNorm ?? Number.NaN)
if (epoch != null) metrics.epoch.push(epoch) metrics.lr.push(learningRate ?? Number.NaN)
metrics.epoch.push(epoch ?? Number.NaN)
} }
return metrics return metrics
} }
export function metricsFromApi(points: FineTuneMetricPoint[]): TrainingMetricData {
const metrics: TrainingMetricData = { steps: [], loss: [], gradNorm: [], lr: [], epoch: [] }
for (const [index, point] of points.entries()) {
const hasMetric = point.loss != null || point.grad_norm != null || point.learning_rate != null
if (!hasMetric) continue
metrics.steps.push(Number(point.step || index + 1))
metrics.loss.push(point.loss == null ? Number.NaN : Number(point.loss))
metrics.gradNorm.push(point.grad_norm == null ? Number.NaN : Number(point.grad_norm))
metrics.lr.push(point.learning_rate == null ? Number.NaN : Number(point.learning_rate))
metrics.epoch.push(point.epoch == null ? Number.NaN : Number(point.epoch))
}
return metrics
}
/** 每次都返回新对象,日志截断或切换时不会残留上一轮汇总。 */ /** 每次都返回新对象,日志截断或切换时不会残留上一轮汇总。 */
export function parseTrainingSummary(text: string): TrainingSummary { export function parseTrainingSummary(text: string): TrainingSummary {
const emptySummary: TrainingSummary = { epoch: '', trainLoss: '', runtime: '' } const emptySummary: TrainingSummary = { epoch: '', trainLoss: '', runtime: '' }
@@ -102,11 +124,23 @@ export function parseTrainingLog(text: string): ParsedTrainingLog {
export function buildMetricChartOption( export function buildMetricChartOption(
label: string, label: string,
data: number[], data: number[],
steps: number[],
color: string, color: string,
logScale = false, logScale = false,
): EChartsOption { ): EChartsOption {
const visibleData = data.map((value) => (Number.isFinite(value) ? value : null))
return { return {
grid: { top: 24, right: 20, bottom: 56, left: 56 }, grid: { top: 24, right: 20, bottom: 56, left: 56 },
graphic: visibleData.some((value) => value != null)
? []
: [
{
type: 'text',
left: 'center',
top: 'middle',
style: { text: '暂无训练指标数据', fill: '#94a3b8', fontSize: 13 },
},
],
tooltip: { tooltip: {
trigger: 'axis', trigger: 'axis',
axisPointer: { type: 'cross' }, axisPointer: { type: 'cross' },
@@ -116,6 +150,7 @@ export function buildMetricChartOption(
}, },
xAxis: { xAxis: {
type: 'category', type: 'category',
data: steps.map((step, index) => (Number.isFinite(step) ? String(step) : String(index + 1))),
boundaryGap: false, boundaryGap: false,
name: 'Step', name: 'Step',
nameTextStyle: { color: '#94a3b8', fontSize: 11 }, nameTextStyle: { color: '#94a3b8', fontSize: 11 },
@@ -142,7 +177,7 @@ export function buildMetricChartOption(
{ {
name: label, name: label,
type: 'line', type: 'line',
data, data: visibleData,
smooth: true, smooth: true,
symbol: 'none', symbol: 'none',
lineStyle: { width: 2, color }, lineStyle: { width: 2, color },

View File

@@ -1,25 +1,42 @@
<script setup lang="ts"> <script setup lang="ts">
import { onMounted, ref } from 'vue' import { onMounted, reactive, ref } from 'vue'
import { useRoute, useRouter } from 'vue-router' import { useRoute, useRouter } from 'vue-router'
import { ElMessage } from 'element-plus' import { ElMessage } from 'element-plus'
import DataTablePage from '@/components/DataTablePage.vue'
import { getTenant, setTenantQuota, type Tenant } from '@/api/modules/tenant' import { getTenant, setTenantQuota, type Tenant } from '@/api/modules/tenant'
import { getProjects, type Project } from '@/api/modules/project'
const route = useRoute() const route = useRoute()
const router = useRouter() const router = useRouter()
const tenant = ref<Tenant | null>(null) const tenant = ref<Tenant | null>(null)
const projects = ref<Project[]>([])
const loading = ref(false) const loading = ref(false)
const quotaText = ref('') const quotaForm = reactive({ gpu: 0, storage: 0, maxProjects: 0 })
function parseQuota(quota: Record<string, unknown> | undefined | null) {
const q = quota || {}
return {
gpu: Number(q.gpu || q.gpu_quota || 0),
storage: Number(q.storage || q.storage_quota || 0),
maxProjects: Number(q.max_projects || 0),
}
}
function formatQuota(quota: Record<string, unknown> | undefined | null) {
const q = parseQuota(quota)
const parts: string[] = []
if (q.gpu > 0) parts.push(`GPU ${q.gpu}`)
if (q.storage > 0) parts.push(`存储 ${q.storage}GB`)
if (q.maxProjects > 0) parts.push(`项目 ${q.maxProjects}`)
return parts.length ? parts.join(' | ') : '—'
}
async function load() { async function load() {
const id = route.params.id as string const id = route.params.id as string
loading.value = true loading.value = true
try { try {
tenant.value = await getTenant(id) tenant.value = await getTenant(id)
projects.value = await getProjects(id) const q = parseQuota(tenant.value?.quota)
quotaText.value = JSON.stringify(tenant.value?.quota || {}) quotaForm.gpu = q.gpu
quotaForm.storage = q.storage
quotaForm.maxProjects = q.maxProjects
} finally { } finally {
loading.value = false loading.value = false
} }
@@ -27,18 +44,13 @@ async function load() {
async function saveQuota() { async function saveQuota() {
if (!tenant.value) return if (!tenant.value) return
try { const quota: Record<string, unknown> = {}
const q = JSON.parse(quotaText.value || '{}') if (quotaForm.gpu > 0) quota.gpu = quotaForm.gpu
await setTenantQuota(tenant.value.id, q) if (quotaForm.storage > 0) quota.storage = quotaForm.storage
ElMessage.success('配额已保存') if (quotaForm.maxProjects > 0) quota.max_projects = quotaForm.maxProjects
load() await setTenantQuota(tenant.value.id, quota)
} catch { ElMessage.success('配额已保存')
ElMessage.error('配额需为合法 JSON') load()
}
}
function openProject(id: string) {
router.push(`/projects/${id}`)
} }
onMounted(load) onMounted(load)
@@ -55,31 +67,30 @@ onMounted(load)
<template #header>基本信息</template> <template #header>基本信息</template>
<el-descriptions :column="2" border> <el-descriptions :column="2" border>
<el-descriptions-item label="名称">{{ tenant?.name }}</el-descriptions-item> <el-descriptions-item label="名称">{{ tenant?.name }}</el-descriptions-item>
<el-descriptions-item label="编码">{{ tenant?.code }}</el-descriptions-item> <el-descriptions-item label="用户ID">{{ tenant?.code }}</el-descriptions-item>
<el-descriptions-item label="状态">{{ tenant?.status }}</el-descriptions-item> <el-descriptions-item label="状态">{{ tenant?.status }}</el-descriptions-item>
<el-descriptions-item label="创建时间">{{ tenant?.create_time }}</el-descriptions-item> <el-descriptions-item label="创建时间">{{ tenant?.create_time }}</el-descriptions-item>
<el-descriptions-item label="配额">{{ formatQuota(tenant?.quota) }}</el-descriptions-item>
</el-descriptions> </el-descriptions>
<el-divider /> <el-divider />
<div class="quota-edit"> <div class="quota-edit">
<span class="label">配额 JSON</span> <span class="label">配额设置0 表示不限制</span>
<el-input v-model="quotaText" type="textarea" :rows="3" /> <el-form label-width="100px">
<el-button type="primary" @click="saveQuota">保存配额</el-button> <el-form-item label="GPU 数量">
<el-input-number v-model="quotaForm.gpu" :min="0" :step="1" />
</el-form-item>
<el-form-item label="存储配额(GB)">
<el-input-number v-model="quotaForm.storage" :min="0" :step="10" />
</el-form-item>
<el-form-item label="最大项目数">
<el-input-number v-model="quotaForm.maxProjects" :min="0" :step="1" />
</el-form-item>
<el-form-item>
<el-button type="primary" @click="saveQuota">保存配额</el-button>
</el-form-item>
</el-form>
</div> </div>
</el-card> </el-card>
<el-card class="section">
<template #header>项目空间</template>
<DataTablePage title="项目空间" :data="projects">
<template #columns>
<el-table-column prop="name" label="项目名" min-width="140" />
<el-table-column prop="code" label="编码" min-width="100" />
<el-table-column prop="status" label="状态" min-width="100" />
<el-table-column prop="create_time" label="创建时间" min-width="180" />
</template>
<template #actions="{ row }">
<el-button link type="primary" @click="openProject(row.id)">打开</el-button>
</template>
</DataTablePage>
</el-card>
</div> </div>
</template> </template>

View File

@@ -1,16 +1,37 @@
<script setup lang="ts"> <script setup lang="ts">
import { onMounted, ref } from 'vue' import { onMounted, reactive, ref } from 'vue'
import { useRouter } from 'vue-router' import { useRouter } from 'vue-router'
import { ElMessage, ElMessageBox } from 'element-plus' import { ElMessage, ElMessageBox } from 'element-plus'
import { Plus } from '@element-plus/icons-vue' import { Plus } from '@element-plus/icons-vue'
import DataTablePage from '@/components/DataTablePage.vue' import DataTablePage from '@/components/DataTablePage.vue'
import { createTenant, getTenants, setTenantQuota, type Tenant } from '@/api/modules/tenant' import { createTenant, deleteTenant, getTenants, setTenantQuota, type Tenant } from '@/api/modules/tenant'
const router = useRouter() const router = useRouter()
const loading = ref(false) const loading = ref(false)
const tenants = ref<Tenant[]>([]) const tenants = ref<Tenant[]>([])
const showCreate = ref(false) const showCreate = ref(false)
const form = ref({ name: '', code: '', quota: '' as string }) const showQuota = ref(false)
const currentTenant = ref<Tenant | null>(null)
const form = ref({ name: '', code: '', gpu: 0, storage: 0, maxProjects: 0 })
const quotaForm = reactive({ gpu: 0, storage: 0, maxProjects: 0 })
function parseQuota(quota: Record<string, unknown> | undefined | null) {
const q = quota || {}
return {
gpu: Number(q.gpu || q.gpu_quota || 0),
storage: Number(q.storage || q.storage_quota || 0),
maxProjects: Number(q.max_projects || 0),
}
}
function formatQuota(quota: Record<string, unknown> | undefined | null) {
const q = parseQuota(quota)
const parts: string[] = []
if (q.gpu > 0) parts.push(`GPU ${q.gpu}`)
if (q.storage > 0) parts.push(`存储 ${q.storage}GB`)
if (q.maxProjects > 0) parts.push(`项目 ${q.maxProjects}`)
return parts.length ? parts.join(' | ') : '-'
}
async function load() { async function load() {
loading.value = true loading.value = true
@@ -25,40 +46,65 @@ function openDetail(id: string) {
router.push(`/tenants/${id}`) router.push(`/tenants/${id}`)
} }
function asTenant(row: unknown): Tenant {
return row as Tenant
}
function quotaText(row: unknown): string {
const quota = asTenant(row).quota || {}
return Object.keys(quota).length ? JSON.stringify(quota) : '-'
}
async function submitCreate() { async function submitCreate() {
if (!form.value.name) { if (!form.value.name) {
ElMessage.warning('请填写租户名称') ElMessage.warning('请填写租户名称')
return return
} }
let quota: Record<string, unknown> = {} const quota: Record<string, unknown> = {}
if (form.value.quota) { if (form.value.gpu > 0) quota.gpu = form.value.gpu
try { if (form.value.storage > 0) quota.storage = form.value.storage
quota = JSON.parse(form.value.quota) if (form.value.maxProjects > 0) quota.max_projects = form.value.maxProjects
} catch {
ElMessage.error('配额需为合法 JSON')
return
}
}
await createTenant({ name: form.value.name, code: form.value.code, quota }) await createTenant({ name: form.value.name, code: form.value.code, quota })
ElMessage.success('租户创建成功') ElMessage.success('租户创建成功')
showCreate.value = false showCreate.value = false
form.value = { name: '', code: '', quota: '' } form.value = { name: '', code: '', gpu: 0, storage: 0, maxProjects: 0 }
load() load()
} }
async function setQuota(row: Tenant) { function openQuotaDialog(row: Tenant) {
const input = await ElMessageBox.prompt('输入租户配额 JSON', '设置配额', { currentTenant.value = row
inputValue: JSON.stringify(row.quota || {}), const q = parseQuota(row.quota)
}).catch(() => null) quotaForm.gpu = q.gpu
if (!input) return quotaForm.storage = q.storage
quotaForm.maxProjects = q.maxProjects
showQuota.value = true
}
async function submitQuota() {
if (!currentTenant.value) return
const quota: Record<string, unknown> = {}
if (quotaForm.gpu > 0) quota.gpu = quotaForm.gpu
if (quotaForm.storage > 0) quota.storage = quotaForm.storage
if (quotaForm.maxProjects > 0) quota.max_projects = quotaForm.maxProjects
await setTenantQuota(currentTenant.value.id, quota)
ElMessage.success('配额已更新')
showQuota.value = false
load()
}
async function handleDelete(row: Tenant) {
try { try {
const q = JSON.parse(input.value) await ElMessageBox.confirm(
await setTenantQuota(row.id, q) `确定要删除租户「${row.name}」吗?删除后相关数据将无法恢复。`,
ElMessage.success('配额已更新') '删除确认',
load() { confirmButtonText: '确定删除', cancelButtonText: '取消', type: 'warning' },
)
} catch { } catch {
ElMessage.error('无效的 JSON') return
} }
await deleteTenant(row.id)
ElMessage.success('租户已删除')
load()
} }
onMounted(load) onMounted(load)
@@ -72,30 +118,34 @@ onMounted(load)
</template> </template>
<template #columns> <template #columns>
<el-table-column prop="name" label="租户名称" min-width="140" /> <el-table-column prop="name" label="租户名称" min-width="140" />
<el-table-column prop="code" label="编码" min-width="100" /> <el-table-column prop="code" label="租户 ID" min-width="100" />
<el-table-column label="配额" min-width="160">
<template #default="{ row }">
{{ Object.keys(row.quota || {}).length ? JSON.stringify(row.quota) : '—' }}
</template>
</el-table-column>
<el-table-column prop="status" label="状态" min-width="100" /> <el-table-column prop="status" label="状态" min-width="100" />
<el-table-column prop="create_time" label="创建时间" min-width="180" /> <el-table-column prop="create_time" label="创建时间" min-width="180" />
</template> </template>
<template #actions="{ row }"> <template #actions="{ row }">
<el-button link type="primary" @click="openDetail(row.id)">详情</el-button> <el-button link type="primary" @click="openDetail(row.id)">详情</el-button>
<el-button link type="primary" @click="setQuota(row)">配额</el-button> <el-button link type="danger" @click="handleDelete(asTenant(row))">删除</el-button>
</template> </template>
</DataTablePage> </DataTablePage>
<!-- 新建租户弹窗 -->
<el-dialog v-model="showCreate" title="新建租户" width="520px"> <el-dialog v-model="showCreate" title="新建租户" width="520px">
<el-form label-width="90px"> <el-form label-width="100px">
<el-form-item label="名称" required> <el-form-item label="名称" required>
<el-input v-model="form.name" placeholder="租户名称" /> <el-input v-model="form.name" placeholder="租户名称" />
</el-form-item> </el-form-item>
<el-form-item label="编码"> <el-form-item label="用户ID">
<el-input v-model="form.code" placeholder="tenant code" /> <el-input v-model="form.code" placeholder="用户ID" />
</el-form-item> </el-form-item>
<el-form-item label="配额 JSON"> <el-divider content-position="left">配额设置可选0 表示不限制</el-divider>
<el-input v-model="form.quota" type="textarea" :rows="3" placeholder='{"gpu": 8}' /> <el-form-item label="GPU 数量">
<el-input-number v-model="form.gpu" :min="0" :step="1" placeholder="GPU 卡数" />
</el-form-item>
<el-form-item label="存储配额(GB)">
<el-input-number v-model="form.storage" :min="0" :step="10" placeholder="存储大小" />
</el-form-item>
<el-form-item label="最大项目数">
<el-input-number v-model="form.maxProjects" :min="0" :step="1" placeholder="项目上限" />
</el-form-item> </el-form-item>
</el-form> </el-form>
<template #footer> <template #footer>
@@ -103,6 +153,25 @@ onMounted(load)
<el-button type="primary" @click="submitCreate">创建</el-button> <el-button type="primary" @click="submitCreate">创建</el-button>
</template> </template>
</el-dialog> </el-dialog>
<!-- 设置配额弹窗 -->
<el-dialog v-model="showQuota" title="设置配额" width="480px">
<el-form label-width="100px">
<el-form-item label="GPU 数量">
<el-input-number v-model="quotaForm.gpu" :min="0" :step="1" placeholder="GPU 卡数" />
</el-form-item>
<el-form-item label="存储配额(GB)">
<el-input-number v-model="quotaForm.storage" :min="0" :step="10" placeholder="存储大小" />
</el-form-item>
<el-form-item label="最大项目数">
<el-input-number v-model="quotaForm.maxProjects" :min="0" :step="1" placeholder="项目上限" />
</el-form-item>
</el-form>
<template #footer>
<el-button @click="showQuota = false">取消</el-button>
<el-button type="primary" @click="submitQuota">保存</el-button>
</template>
</el-dialog>
</div> </div>
</template> </template>

282
平台治理.md Normal file
View File

@@ -0,0 +1,282 @@
# 平台治理功能说明
平台治理涵盖用户管理、租户管理、项目管理、审批管理、审计日志、资源授权(ACL)、留存策略等企业管理能力。功能入口位于侧边栏「平台治理」和「系统设置」两个分组下。
本平台核心业务是**模型微调**:用户上传数据 → 数据处理 → 模型训练 → 模型评测 → 模型推理。平台治理负责管理**谁**能访问**哪个租户/项目**的**哪些资源**。
---
## 侧边栏菜单结构
```
平台治理
├── 租户管理 /tenants
├── 项目空间 /projects
├── 审计日志 /audit-logs
├── 审批模板 /approval-templates
└── 审批中心 /approval-instances
系统设置
├── 用户设置 /user-settings
├── 平台性能 /hardware
└── 查看日志 /logs
```
---
## 1. 租户管理
**对应页面**:列表页 `TenantListView.vue` + 详情页 `TenantDetailView.vue`
**后端接口**`GET/POST /tenants``GET/PUT/DELETE /tenants/:id``PUT /tenants/:id/quota``PUT /tenants/:id/retention-policy`
### 列表页
**展示列**租户名称、用户ID、状态、创建时间
**行操作**
| 按钮 | 功能 |
|------|------|
| 详情 | 跳转详情页 `/tenants/:id` |
| 删除 | 确认弹窗后调用 `DELETE /tenants/:id` |
**新建租户弹窗**
| 字段 | 控件 | 说明 |
|------|------|------|
| 名称 | 文本输入 | 必填 |
| 用户ID | 文本输入 | 可选 |
| GPU 数量 | 数字输入 | 配额0=不限制 |
| 存储配额(GB) | 数字输入 | 配额0=不限制 |
| 最大项目数 | 数字输入 | 配额0=不限制 |
### 详情页
**基本信息**名称、用户ID、状态、创建时间、配额友好格式显示`GPU 8 | 存储 100GB | 项目 5`
**配额编辑**GPU 数量 / 存储配额(GB) / 最大项目数,三个数字输入 + 保存按钮
**已移除**:详情页中不再内嵌项目空间列表(项目有独立页面)
### 当前缺失
- [ ] 编辑租户名称和用户ID`updateTenant` API 已有UI 缺失)
- [ ] 留存策略关联设置(`setTenantRetention` API 已有UI 缺失)
- [ ] 列表页无搜索/筛选
- [ ] 状态列无 Tag 着色
---
## 2. 项目空间
**对应页面**:列表页 `ProjectListView.vue` + 详情页 `ProjectDetailView.vue`
**后端接口**`GET/POST /projects``GET/PUT/DELETE /projects/:id``GET/POST/DELETE /projects/:id/members``PUT /projects/:id/archive`
### 列表页
**展示列**项目名、编码ID、状态、描述、创建时间
**工具栏**
- 租户选择器(下拉过滤,按租户编码选择)
- 搜索框(按项目名/编码搜索)
- 新建项目按钮
**新建项目弹窗**
| 字段 | 控件 | 说明 |
|------|------|------|
| 名称 | 文本输入 | 必填 |
| 编码ID | 下拉选择 | 选择已有租户的编码(如 `default`),关联到该租户 |
| 描述 | 文本域 | 可选3行 |
**行操作**:详情 + 删除
### 详情页
**基本信息**:名称、编码、状态、租户、描述、创建时间
**项目成员管理**
- 成员列表(用户名、角色、添加时间)
- 添加成员:选择用户 + 角色member/admin/viewer
- 移除成员
**资源授权(ACL)**
- 弹窗编辑器,逐条配置授权规则
- 主体类型用户下拉选已有用户或项目角色member/admin/viewer
- 权限read / write / execute / download / delete / share多选 checkbox
- 增删行后统一保存
### 当前缺失
- [ ] 编辑项目基本信息(名称、描述)
- [ ] 项目启停/归档操作(`archiveProject` API 已有UI 缺失)
- [ ] 项目下的资源使用统计(模型数、数据集数、任务数)
---
## 3. 审批管理
**对应页面**`ApprovalTemplateView.vue`(模板)+ `ApprovalInstanceView.vue`(实例)
**后端接口**:模板 CRUD、实例列表+决策
### 审批模板
**展示列**:模板名、步骤数、创建时间
**新建模板**
| 字段 | 控件 | 说明 |
|------|------|------|
| 模板名称 | 文本输入 | 必填 |
| 审批步骤 | 文本域 | **需手写 JSON 字符串**,体验差 |
**当前缺失**
- [ ] 可视化步骤编辑(拖拽添加步骤、选审批人)
- [ ] 模板编辑/删除API 已有UI 缺失)
- [ ] 模板详情页
### 审批中心
**展示列**资源类型、资源ID、状态Tag着色、发起人、创建时间
**工具栏**:状态筛选(待审批/已通过/已拒绝)、资源类型/ID 搜索
**行操作**:通过/拒绝(弹窗填写审批意见)
**当前缺失**
- [ ] "我发起的"审批视角
- [ ] 审批流转详情(谁审批了、什么时间)
- [ ] 撤回功能
---
## 4. 审计日志
**对应页面**`AuditLogView.vue`
**后端接口**`GET /system/audit-logs`(多条件筛选+分页)、`GET /system/audit-logs/export`CSV导出
### 列表页
**展示列**时间、租户ID、项目ID、操作人ID、动作、目标类型、目标ID、详情、IP
**筛选条件**租户ID、项目ID、操作人ID、动作、目标类型、开始时间、结束时间
**工具栏**CSV 导出按钮(最多 10000 条)
### 当前缺失
- [ ] 分页控件(数据量大时需要翻页)
- [ ] 筛选字段改为下拉选择(当前全是文本输入,不知道有哪些可选值)
- [ ] 详情弹窗(点击某条记录查看完整信息)
---
## 5. 用户设置
**对应页面**`UserSettingsView.vue`(列表)+ `UserCreateView.vue`(创建)+ `UserPermissionView.vue`(权限弹窗)
**后端接口**`GET/POST/PUT/DELETE /users``POST /users/:id/reset-password`
### 列表页
**展示列**:账号、显示名、角色、状态、页面权限、创建时间
**行操作**
| 操作 | 说明 |
|------|------|
| 启停开关 | protected 用户和自己不可操作 |
| 重置密码 | 弹窗确认,默认密码 `Platform@123` |
| 页面权限 | 弹窗 checkbox 组12 个模块可选:看板/模型训练/模型评测/模型推理/模型管理/数据集/数据处理/数据转换/算力/平台性能/查看日志/用户设置 |
| 删除 | 确认弹窗protected 用户和自己不可操作 |
### 创建页
**表单字段**:账号、显示名、初始密码、角色(超级管理员/操作员/观察员)、状态、页面权限
### 当前缺失
- [ ] 用户编辑页面(修改显示名、角色等,`updateUser` API 已有UI 缺失)
- [ ] 权限码显示为英文(如 `fine-tune`),无中文翻译
---
## 6. 平台性能
**对应页面**`HardwareView.vue`
**后端接口**`GET /system-info``GET /health`(顶部栏实时指标)
### 页面内容
**概览卡片**CPU型号/核心数/使用率)、内存(已用/总量/使用率)、磁盘(已用/总量/使用率)、网络吞吐
**趋势图**CPU/内存/磁盘/GPU 利用率折线图(最近 60 次采样1/3/5/10秒自动刷新
**GPU 资源池**:每张卡展示名称、状态、利用率、显存、温度、功耗、风扇转速;点击卡片查看详情抽屉(设备属性、趋势图、进程列表)
**主机信息**OS、运行时长、进程数、GPU 驱动版本
### 当前缺失
- [ ] 历史数据持久化(刷新后采样清空)
- [ ] 告警阈值设置
- [ ] 网络吞吐数据显示为空
---
## 7. 查看日志
**对应页面**`LogsView.vue`(系统日志 + 训练日志双 Tab
### 系统日志
- 按日期选择日志文件
- 关键词搜索
- 日志级别筛选INFO/WARN/ERROR/DEBUG
- 自动刷新5/10/30/60 秒可调)
### 训练日志
- 按 PID 选择训练日志文件
- 同样支持搜索和级别筛选
### 当前缺失
- [ ] 日志下载/导出
- [ ] 分页或虚拟滚动(大文件加载慢)
- [ ] 行号显示
---
## 8. 留存策略 ⚠️ 前端完全缺失
**后端 API 已完整实现**CRUD 5 个端点),**前端 `retention.ts` 模块已封装**,但:
- 侧边栏无菜单入口
- 路由未配置
- 无任何 Vue 页面
---
## 数据流转关系
```
用户登录 → 分配角色(admin/operator/viewer) + 页面权限
├─ 创建租户配额GPU/存储/项目数)
│ └─ 关联用户
├─ 创建项目关联租户选择编码ID
│ ├─ 项目成员角色member/admin/viewer
│ └─ 资源授权(ACL):谁对什么资源有什么权限
├─ 创建审批模板(定义审批流程)
│ └─ 审批实例:敏感操作需要审批(通过/拒绝)
├─ 审计日志:所有操作自动记录(谁在什么时间做了什么)
└─ 留存策略:定义数据保留周期,自动清理过期数据
```
---
## 完成度总表
| 模块 | 列表 | 新建 | 编辑 | 删除 | 搜索 | 筛选 | 导出 | 完成度 |
|------|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
| 租户管理 | ✅ | ✅ | 仅配额 | ✅ | ❌ | ❌ | ❌ | 75% |
| 项目空间 | ✅ | ✅ | ❌ | ✅ | ✅ | 按租户 | ❌ | 80% |
| 审批模板 | ✅ | ✅(JSON) | ❌ | ❌ | ❌ | ❌ | ❌ | 45% |
| 审批中心 | ✅ | N/A | N/A | ❌ | ✅ | 按状态 | ❌ | 65% |
| 审计日志 | ✅ | N/A | N/A | N/A | ❌ | 多字段 | CSV | 70% |
| 用户设置 | ✅ | ✅ | 仅权限 | ✅ | ❌ | ❌ | ❌ | 75% |
| 平台性能 | ✅ | N/A | N/A | N/A | N/A | N/A | ❌ | 80% |
| 查看日志 | ✅ | N/A | N/A | N/A | 关键词 | 按级别 | ❌ | 70% |
| **留存策略** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | **0%** |