feat: 推理与评测支持多卡 GPU 选择,训练任务实时 GPU 监控
- 新增 GPU 选择归一化 helper,统一前端各形态的选择(gpu_indices/gpus/gpu_id)
- 评测与推理支持同一节点内多卡选择,校验所选 GPU 空闲后再派发
- 新增 /fine-tune/{id}/gpu-status 接口,训练日志页展示实时 GPU 指标
- 算力节点推理加载支持 CUDA_VISIBLE_DEVICES 多卡可见,nvidia-smi 进程级监控
- GPU 占用跟踪细化为按卡记录,覆盖评测任务、推理模型与对比任务
Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -73,7 +73,8 @@ async def reconcile_inference_loads(store: Any) -> list[dict[str, Any]]:
|
||||
if node_status == "ready":
|
||||
item["status"] = "ready"
|
||||
item.pop("error", None)
|
||||
store.mark_inference_loaded(node["id"])
|
||||
selected_gpus = item.get("gpu_indices") or item.get("gpus")
|
||||
store.mark_inference_loaded(node["id"], selected_gpus)
|
||||
elif node_status == "error":
|
||||
item["status"] = "error"
|
||||
item["error"] = status.get("error") or "model load failed on compute node"
|
||||
|
||||
Reference in New Issue
Block a user