feat: 重构 Docker 配置结构,添加 compute 模块及新增文档
- 将 Dockerfile 和 docker-compose.yml 迁移至 docker/ 目录下统一管理 - 新增 compute 计算模块(API 入口、依赖配置) - 新增 docker/app 和 docker/compute 部署配置 - 新增 demo-development-plan.md 演示开发计划文档 - 更新后端 API 设计、部署计划、架构需求等文档 - 更新 postgres 数据库 schema Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -687,7 +687,7 @@ page=1&page_size=20&keyword=xxx&sort=-created_at
|
||||
|
||||
## 12. 仍需确认的问题
|
||||
|
||||
1. 部署形态:单机多 GPU、K8s、多训练节点,还是只在一台服务器上调度?
|
||||
1. 部署形态已确认:多算力节点仍按“单机多 GPU 节点”管理,不引入 K8s;每台 GPU 服务器独立部署 Compute API/Agent/File Gateway/LLaMA-Factory。
|
||||
2. 文件存储:使用本地磁盘、NAS、MinIO,还是对象存储?是否需要断点续传?
|
||||
3. 训练框架:是否固定使用 LLaMA-Factory?是否还要支持 Transformers 原生、DeepSpeed、Accelerate?
|
||||
4. 权限粒度:页面级权限是否足够,还是需要到数据集/模型/任务的所有者与项目空间级权限?
|
||||
@@ -836,12 +836,58 @@ page=1&page_size=20&keyword=xxx&sort=-created_at
|
||||
| 方法 | 路径 | 说明 |
|
||||
| --- | --- | --- |
|
||||
| GET | `/api/compute/nodes` | 算力节点列表 |
|
||||
| POST | `/api/compute/nodes` | 新增算力节点 |
|
||||
| GET | `/api/compute/nodes/{id}` | 算力节点详情 |
|
||||
| PUT | `/api/compute/nodes/{id}` | 编辑节点地址、权重、标签、路径和启用状态 |
|
||||
| POST | `/api/compute/nodes/{id}/test-connection` | 测试 Compute API/File Gateway 连通性 |
|
||||
| POST | `/api/compute/nodes/{id}/enable` | 启用节点 |
|
||||
| POST | `/api/compute/nodes/{id}/disable` | 禁用节点,不接收新任务 |
|
||||
| POST | `/api/compute/nodes/{id}/drain` | 进入维护模式,已有任务跑完后下线 |
|
||||
| POST | `/api/compute/nodes/{id}/health-check` | 主动触发节点健康检查 |
|
||||
| GET | `/api/compute/nodes/{id}/engines` | 节点训练引擎和版本 |
|
||||
| GET | `/api/compute/nodes/{id}/replicas` | 节点本地资源副本 |
|
||||
| GET | `/api/compute/gpus` | GPU 状态 |
|
||||
| GET | `/api/compute/queue` | 任务队列 |
|
||||
| GET | `/api/compute/jobs/{id}` | 算力任务详情 |
|
||||
| POST | `/api/compute/jobs/{id}/retry` | 重试任务 |
|
||||
| POST | `/api/compute/jobs/{id}/priority` | 调整优先级 |
|
||||
| POST | `/api/internal/compute-callbacks/jobs` | 算力平台任务回调 |
|
||||
| POST | `/api/internal/compute-sync/jobs/poll` | 应用平台主动轮询并同步算力任务状态 |
|
||||
| POST | `/api/internal/compute-sync/resources` | 调度前同步数据集/模型到目标节点 |
|
||||
|
||||
算力节点设计说明:
|
||||
|
||||
- 多算力节点仍按“单机多 GPU 节点”管理,每台 GPU 服务器是一条 `compute_nodes` 记录。
|
||||
- 每个可执行训练的节点都需要部署 `Compute API`、`Compute Agent`、`File Gateway` 和宿主机挂载的 LLaMA-Factory。
|
||||
- 节点之间默认不互相访问,应用平台主动访问所有节点的 Compute API/File Gateway。
|
||||
- 调度支持 `auto` 和 `manual`:普通用户默认自动调度,管理员或高级用户可手动指定节点。
|
||||
|
||||
算力节点响应字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "uuid",
|
||||
"code": "gpu-node-01",
|
||||
"name": "A800 Node 01",
|
||||
"api_base_url": "http://10.10.20.31:9100",
|
||||
"file_gateway_url": "http://10.10.20.31:9101",
|
||||
"enabled": true,
|
||||
"scheduler_status": "online",
|
||||
"scheduler_weight": 100,
|
||||
"tags": ["A800", "80GB", "llama_factory"],
|
||||
"gpu_count": 8,
|
||||
"current_running_jobs": 2,
|
||||
"max_parallel_jobs": 8,
|
||||
"data_root": "/data/yg-ft",
|
||||
"model_root": "/data/yg-ft/models",
|
||||
"log_root": "/opt/yg-ft/logs/compute",
|
||||
"last_health_check_at": "2026-07-20T12:00:00+08:00",
|
||||
"health_detail": {
|
||||
"compute_api": "ok",
|
||||
"file_gateway": "ok",
|
||||
"llama_factory": "ok"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
GPU 响应字段:
|
||||
|
||||
@@ -884,6 +930,13 @@ GPU 响应字段:
|
||||
"job_type": "fine_tune",
|
||||
"engine": "llama_factory",
|
||||
"priority": "normal",
|
||||
"scheduler": {
|
||||
"mode": "auto",
|
||||
"requested_node_id": null,
|
||||
"required_tags": ["A800"],
|
||||
"preferred_tags": ["llama_factory"],
|
||||
"min_gpu_memory_mb": 40960
|
||||
},
|
||||
"resource_request": {
|
||||
"gpu_count": 1,
|
||||
"gpu_ids": [0],
|
||||
@@ -897,10 +950,46 @@ GPU 响应字段:
|
||||
"dataset_path": "/data/ft-platform/.../datasets/train.jsonl",
|
||||
"training_args": {}
|
||||
},
|
||||
"callback_url": "http://app/api/internal/compute-callbacks/jobs"
|
||||
"status_sync_mode": "polling",
|
||||
"poll_interval_seconds": 10
|
||||
}
|
||||
```
|
||||
|
||||
手动指定节点时:
|
||||
|
||||
```json
|
||||
{
|
||||
"scheduler": {
|
||||
"mode": "manual",
|
||||
"requested_node_id": "uuid",
|
||||
"gpu_ids": [0, 1]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
调度前资源副本检查:
|
||||
|
||||
```json
|
||||
{
|
||||
"target_compute_node_id": "uuid",
|
||||
"resources": [
|
||||
{
|
||||
"resource_type": "model",
|
||||
"resource_id": "uuid",
|
||||
"required": true
|
||||
},
|
||||
{
|
||||
"resource_type": "dataset",
|
||||
"resource_id": "uuid",
|
||||
"required": true
|
||||
}
|
||||
],
|
||||
"sync_if_missing": true
|
||||
}
|
||||
```
|
||||
|
||||
如果目标节点缺少数据集或模型副本,应用平台通过 File Gateway 创建 `resource_sync_jobs`,同步完成后再提交训练任务。
|
||||
|
||||
### 13.7 文件网关与离线导入
|
||||
|
||||
| 方法 | 路径 | 说明 |
|
||||
@@ -1064,7 +1153,7 @@ LLaMA-Factory 引擎声明:
|
||||
| --- | --- | --- | --- |
|
||||
| 审批中心 | `/approvals`、`/approvals/pending`、`/approvals/mine`、`/approvals/:id` | `GET /api/approvals`、`POST /api/approvals`、`GET /api/approvals/{id}`、`POST /api/approvals/{id}/approve`、`POST /api/approvals/{id}/reject`、`POST /api/approvals/{id}/cancel` | 审批列表和审批动作 |
|
||||
| 审批设置 | `/approval-settings` | `GET /api/approval-templates`、`PUT /api/approval-templates/{id}` | 审批模板 |
|
||||
| 算力资源 | `/compute`、`/compute/gpus`、`/compute/queue`、`/compute/nodes` | `GET /api/compute/nodes`、`GET /api/compute/gpus`、`GET /api/compute/queue`、`POST /api/compute/jobs/{id}/retry`、`POST /api/compute/jobs/{id}/priority` | GPU、节点、队列 |
|
||||
| 算力资源 | `/compute`、`/compute/gpus`、`/compute/queue`、`/compute/nodes` | `GET/POST/PUT /api/compute/nodes`、`POST /api/compute/nodes/{id}/test-connection`、`POST /api/compute/nodes/{id}/enable`、`POST /api/compute/nodes/{id}/disable`、`POST /api/compute/nodes/{id}/drain`、`GET /api/compute/gpus`、`GET /api/compute/queue`、`POST /api/compute/jobs/{id}/retry`、`POST /api/compute/jobs/{id}/priority` | GPU、节点、队列、节点权重、标签、维护状态、资源副本 |
|
||||
| 存储管理 | `/storage` | `GET /api/quotas/usage`、`GET /api/files/{id}/download-url`、`GET /api/retention-policies`、`PUT /api/retention-policies/{id}` | 磁盘占用、下载、留存 |
|
||||
| 审计中心 | `/audit-logs`、`/login-logs`、`/download-logs` | `GET /api/audit-logs`、`GET /api/login-logs`、`GET /api/download-logs` | 操作、登录、下载审计 |
|
||||
| 训练引擎管理 | `/training-engines` | `GET /api/training-engines`、`GET /api/training-engines/{id}`、`GET /api/training-engines/{id}/schema`、`POST /api/training-engines/{id}/health-check` | 引擎能力和健康 |
|
||||
|
||||
Reference in New Issue
Block a user