feat: 模型评测端到端闭环 — EvalRunner引擎 + 算力节点Job执行 + 结果回写

算力节点 (compute):
- 新建 eval_runner.py: 评测执行引擎,作为subprocess运行
  - 加载模型 + JSONL数据集 + 逐样本推理
  - BLEU/ROUGE/Cosine基础指标计算
  - LLM Judge评分(OpenAI兼容API调用)
  - 结果写入eval_results.json
- adapter.py: build_command新增engine=eval分支
- main.py: 新增/json模块导入,新增/compute/files/read端点,eval job校验

后端:
- platform.py: 重写startEval提交eval job到算力节点
  - 支持models表和trained_models表查找
  - 已合并模型不传adapter路径
- platform_store.py: 新增update_eval_task/running_eval_tasks/apply_eval_job_result
- sync.py: poller新增eval job同步,异步读取eval_results.json回写结果

前端:
- EvalCreateView/DimensionCreateView: eval模型过滤扩展(API类型+api_url)
- EvalCreateView: GPU过滤在线节点空闲GPU
- EvalTaskSetupStep: GPU value从数组index改为gpu.id
- BasicMetricSetupStep: ROUGE方法名修正(rouge_1→rouge1)
- EvalView: 新增5秒轮询刷新

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
wuyongtao
2026-07-28 19:34:41 +08:00
parent c7c9ed925b
commit 0c39f2f5b9
12 changed files with 903 additions and 17 deletions

View File

@@ -204,6 +204,31 @@ def build_command(config: dict[str, Any], llama_factory_home: str = "/app/LLaMA-
command.extend(["--quantization_bit", str(quantization_bit)])
return LlamaFactoryCommand(command=command, work_dir=str(Path(llama_factory_home)), env={})
if engine == "eval":
output_dir = config.get("output_dir") or f"/data/yg-ft/outputs/{config.get('name', 'eval-job')}"
eval_config_path = str(Path(output_dir) / "eval_config.json")
eval_config = {
"model_name_or_path": config.get("model_name_or_path", ""),
"adapter_name_or_path": config.get("adapter_name_or_path", ""),
"template": config.get("template", "qwen"),
"dataset_path": config.get("dataset_path", ""),
"output_dir": output_dir,
"basic_metrics": config.get("basic_metrics", {}),
"dimension": config.get("dimension", {}),
"temperature": config.get("temperature", 0.1),
"top_p": config.get("top_p", 0.95),
"max_new_tokens": config.get("max_new_tokens", 512),
"infer_backend": config.get("infer_backend", "huggingface"),
"infer_dtype": config.get("infer_dtype", "auto"),
}
Path(output_dir).mkdir(parents=True, exist_ok=True)
Path(eval_config_path).write_text(json.dumps(eval_config, ensure_ascii=False, indent=2), encoding="utf-8")
return LlamaFactoryCommand(
command=["python", "-u", "-m", "compute.engines.llama_factory.eval_runner", "--config", eval_config_path],
work_dir="/app",
env={},
)
errors = validate_config(config)
if errors:
raise ValueError("; ".join(errors))