feat: 模型评测端到端闭环 — EvalRunner引擎 + 算力节点Job执行 + 结果回写

算力节点 (compute):
- 新建 eval_runner.py: 评测执行引擎,作为subprocess运行
  - 加载模型 + JSONL数据集 + 逐样本推理
  - BLEU/ROUGE/Cosine基础指标计算
  - LLM Judge评分(OpenAI兼容API调用)
  - 结果写入eval_results.json
- adapter.py: build_command新增engine=eval分支
- main.py: 新增/json模块导入,新增/compute/files/read端点,eval job校验

后端:
- platform.py: 重写startEval提交eval job到算力节点
  - 支持models表和trained_models表查找
  - 已合并模型不传adapter路径
- platform_store.py: 新增update_eval_task/running_eval_tasks/apply_eval_job_result
- sync.py: poller新增eval job同步,异步读取eval_results.json回写结果

前端:
- EvalCreateView/DimensionCreateView: eval模型过滤扩展(API类型+api_url)
- EvalCreateView: GPU过滤在线节点空闲GPU
- EvalTaskSetupStep: GPU value从数组index改为gpu.id
- BasicMetricSetupStep: ROUGE方法名修正(rouge_1→rouge1)
- EvalView: 新增5秒轮询刷新

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
wuyongtao
2026-07-28 19:34:41 +08:00
parent c7c9ed925b
commit 0c39f2f5b9
12 changed files with 903 additions and 17 deletions

View File

@@ -48,4 +48,45 @@ async def poll_compute_jobs_once() -> dict[str, Any]:
standalone_synced.append(store.sync_model_merge_job(record["id"], job))
except Exception as exc: # noqa: BLE001 - keep polling other jobs
failed.append({"job_id": record["id"], "error": str(exc)})
return {"synced": len(synced) + len(standalone_synced), "failed": failed, "items": synced, "standalone": standalone_synced}
# ── Eval job sync ────────────────────────────────────────────────
eval_synced = 0
for eval_task in store.running_eval_tasks():
node = next(
(item for item in store.compute_nodes() if item["id"] == eval_task.get("compute_node_id")),
None,
)
if not node:
failed.append({"eval_task_id": eval_task["id"], "error": "compute node not found"})
continue
try:
client = ComputeNodeClient(node["api_base_url"])
job = await client.get_job(eval_task["compute_job_id"])
result_content = None
# Try to read eval_results.json from the job output directory
if job.get("status") == "completed" and job.get("output_dir"):
try:
full_path = f"{job['output_dir'].rstrip('/')}/eval_results.json"
# Convert absolute path to relative (strip YG_FT_DATA_ROOT prefix)
data_root = "/data/yg-ft/"
if full_path.startswith(data_root):
full_path = full_path[len(data_root):]
rel_path = full_path.lstrip("/")
import httpx
settings_path = f"{node['api_base_url'].rstrip('/')}/modelTF/compute/files/read"
async with httpx.AsyncClient(timeout=30, headers=client.headers()) as http:
read_resp = await http.get(settings_path, params={"path": rel_path})
if read_resp.status_code == 200:
result_content = read_resp.json()
except Exception:
pass
store.apply_eval_job_result(eval_task["id"], job, result_content)
# If job completed, un-mark inference loaded
if job.get("status") in {"completed", "failed", "stopped"}:
store.mark_inference_unloaded(node["id"])
eval_synced += 1
except Exception as exc: # noqa: BLE001
failed.append({"eval_task_id": eval_task["id"], "error": str(exc)})
return {"synced": len(synced) + len(standalone_synced) + eval_synced, "failed": failed,
"items": synced, "standalone": standalone_synced, "eval_synced": eval_synced}