feat: 更新后端平台模块、Compute引擎、前端组件及构建产物

- 更新 backend 平台 API、platform_store、compute_gateway sync
- 更新 compute agent/engine/adapter 及 API
- 更新 Docker 部署配置(app/compute)
- 新增 frontend/src/utils/ 工具模块
- 新增 scripts/ops_diagnostics.py 运维诊断脚本
- 新增 docs/2026-07-23-development-summary.md 开发总结
- 重构 frontend/dist 构建产物(新 hash)
- 更新前端多个视图组件及 API 模块

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
wuyongtao
2026-07-23 19:32:42 +08:00
parent f04dc479bb
commit b28cfbc6fa
193 changed files with 2647 additions and 424 deletions

View File

@@ -1095,8 +1095,21 @@ GPU 响应字段:
"engine": "llama_factory",
"base_model": "/data/yg-ft/models/Qwen2.5-7B",
"model_name_or_path": "/data/yg-ft/models/Qwen2.5-7B",
"dataset": "finance_train",
"dataset_dir": "/data/yg-ft/datasets",
"train_dataset_id": "ds_finance_train",
"dataset": "ygft_ds_finance_train",
"dataset_key": "ygft_ds_finance_train",
"dataset_dir": "/data/yg-ft/datasets/ds_finance_train",
"dataset_info": {
"ygft_ds_finance_train": {
"file_name": "train.jsonl",
"formatting": "alpaca",
"columns": {
"prompt": "instruction",
"query": "input",
"response": "output"
}
}
},
"output_dir": "/data/yg-ft/outputs/finance-sft-001",
"template": "qwen",
"train_method": "lora",
@@ -1108,6 +1121,16 @@ GPU 响应字段:
}
```
数据集启动规则:
- 页面选择的是平台数据集 ID后端提交 Compute Job 时会将其转换为 LLaMA-Factory 数据集 key。
- 单文件数据集使用 `--dataset ygft_{dataset_id}`;多文件数据集使用 `--dataset ygft_{dataset_id}_1,ygft_{dataset_id}_2`
- `dataset_dir` 指向目标算力节点上的独立数据集目录 `/data/yg-ft/datasets/{dataset_id}`
- Compute API 在 preflight 和启动训练前根据 `dataset_info` 生成 `{dataset_dir}/dataset_info.json`,避免 LLaMA-Factory 读取全局 `/data/yg-ft/datasets/dataset_info.json` 失败。
- `columns` 只声明训练文件实际存在的字段;`system``history` 等可选字段不能默认写入,否则样本缺少字段时 LLaMA-Factory 会在格式转换阶段报 `KeyError`
- 正式启动前,应用侧会把当前数据集文件内容同步到被调度的算力节点,确保在线编辑/版本切换后的训练文件被使用。
- Preflight 会校验 `dataset_info.columns` 对应字段是否能在样本文件中找到,并校验 PyTorch CUDA 可用性、所选 GPU 是否存在、显存是否满足 `MIN_TRAINING_GPU_MEMORY_GB`
- Compute 健康检查返回 `torch_cuda`,用于区分 `nvidia-smi` 可见但 PyTorch CUDA 初始化失败的环境问题。
应用侧轮询同步响应:
```json
@@ -1330,3 +1353,38 @@ LLaMA-Factory 引擎声明:
| 存储管理 | `/storage` | `GET /modelTF/quotas/usage``GET /modelTF/files/{id}/download-url``GET /modelTF/retention-policies``PUT /modelTF/retention-policies/{id}` | 磁盘占用、下载、留存 |
| 审计中心 | `/audit-logs``/login-logs``/download-logs` | `GET /modelTF/audit-logs``GET /modelTF/login-logs``GET /modelTF/download-logs` | 操作、登录、下载审计 |
| 训练引擎管理 | `/training-engines` | `GET /modelTF/training-engines``GET /modelTF/training-engines/{id}``GET /modelTF/training-engines/{id}/schema``POST /modelTF/training-engines/{id}/health-check` | 引擎能力和健康 |
## P1/P2/P3 runtime implementation note
The current backend/compute implementation has connected the B+D training runtime features below:
| Page module | API | Runtime behavior |
| --- | --- | --- |
| Training detail / logs `/training-log/:id` | `GET /modelTF/fine-tune/{id}/overview` | Returns task progress, parsed training metrics and real checkpoint records. |
| Training detail / loss chart `/training-log/:id` | `GET /modelTF/fine-tune/{id}/metrics` | Reads `fine_tune_metrics`, populated from Compute log polling and log proxy access. |
| Training detail / checkpoint list `/training-log/:id` | `GET /modelTF/fine-tune/{id}/checkpoints` | Reads `fine_tune_checkpoints`, populated from Compute scanning `output_dir/checkpoint-*`. |
| Merge weights `/model-manage/merge` | `POST /modelTF/model-manage/merge` | Submits a real Compute job using `llamafactory-cli export`; records the job in `compute_jobs`; updates `trained_models.merging/merged/merged_path` when queried after completion. |
| Compute ops / job detail | `GET /modelTF/compute/jobs/{job_id}` | Supports both fine-tune jobs and model merge/export jobs recorded in `compute_jobs`. |
| Compute ops / job logs | `GET /modelTF/compute/jobs/{job_id}/logs` | Proxies logs from the assigned Compute node for training and merge/export jobs. |
Operational diagnostic script:
```bash
APP_BASE_URL=http://localhost:17861 \
COMPUTE_BASE_URL=http://localhost:19100 \
COMPUTE_SERVICE_TOKEN=change_me \
DATABASE_URL=postgresql+psycopg://user:password@host:5432/yg_ft \
python scripts/ops_diagnostics.py
```
## P2/P3 runtime extension note
This iteration extends the B+D runtime implementation with production-facing model asset governance and compute operations:
| Page module | API | Description |
| --- | --- | --- |
| Trained model detail / artifacts | `GET /modelTF/model-manage/trained-models/{id}/artifacts` | Returns registered adapter, merged model and quantized/export artifacts from `model_artifacts`. |
| Trained model detail / lineage | `GET /modelTF/model-manage/trained-models/{id}/lineage` | Returns upstream/downstream relations from `model_lineage`, including base model to fine-tuned model and merge/export relations. |
| Merge/export task list | `GET /modelTF/model-manage/export-jobs?trained_model_id=xxx` | Returns model export and merge jobs from `model_export_jobs`. |
| Compute node replicas | `GET /modelTF/compute/nodes/{id}/replicas/drift` | Checks whether model/dataset/output replicas still exist on the compute node local disk and updates replica status. |
| Compute node replicas | `POST /modelTF/compute/nodes/{id}/replicas/repair` | Marks drifted replicas as `repair_pending` and creates a resource sync job for the operator/scheduler to process. |
The scheduler now uses the `scheduler_locks` table while starting training tasks. Node selection, task state update, resource sync job creation and GPU pre-allocation are written in one database transaction to reduce multi-worker GPU contention.