feat: 新增 compute_gateway、compute_poller、agent 模块，重构前端 dist

- 新增 backend/app/modules/compute_gateway（client/sync）计算网关模块 - 新增 backend/app/workers/compute_poller 计算轮询 worker - 新增 compute/agent/process_manager 进程管理器 - 新增 scripts/ 脚本目录 - 更新 Docker 部署配置（app/compute/nginx） - 更新后端平台 API、数据库 SQL、core 配置 - 更新前端多个视图组件及 API 模块 - 重构 frontend/dist 构建产物（新 hash） - 更新多项文档 Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-22 17:32:59 +08:00
parent 1e438164c1
commit 836343b29e
180 changed files with 3352 additions and 263 deletions
--- a/docker/README.md
+++ b/docker/README.md
@@ -56,7 +56,7 @@ $images | ForEach-Object { docker pull $_ }
 | PostgreSQL | `15432` | `5432` | 开发阶段内置数据库 |
 | Redis | `16379` | `6379` | 开发阶段内置缓存 |
 | Compute API | `19100` | `9100` | 算力服务器 API |
-| File Gateway | `19101` | 后续服务端口 | 当前预留，后续拆出文件网关服务时使用 |
+| File Gateway | `19101` | `9100` | 当前由 Compute API 暴露文件网关契约，后续可拆为独立服务 |

 注意：`8000` 是后端容器内部端口，不作为宿主机对外访问端口。宿主机或浏览器应访问 `http://<app-server-ip>:17861/modelTF/health`；前端 Nginx 容器在 Docker 网络内部访问 `http://backend-api:8000/modelTF/...`。

@@ -142,6 +142,16 @@ docker compose logs --tail=80 frontend

 如果使用企业统一 PostgreSQL/Redis，修改 `docker/app/.env`：

+如果前端 Nginx 日志出现 `open() "/usr/share/nginx/html/modelTF/login" failed` 或 `open() "/usr/share/nginx/html/login" failed`，说明当前容器没有加载项目的 Nginx 代理配置，`/modelTF/*` 被当成静态文件查找。处理方式：
+
+```bash
+cd <repo-root>/docker/app
+docker compose up -d --force-recreate frontend
+docker compose exec frontend nginx -T | grep -n "location.*modelTF" -A12
+```
+
+正常配置中应存在 `location ^~ /modelTF/`，并代理到 `BACKEND_PROXY_PASS`，默认是 `http://backend-api:8000`。
+
 ```env
 DATABASE_URL=postgresql+psycopg://<user>:<password>@<postgres-host>:15432/<db>
 REDIS_URL=redis://<redis-host>:16379/0
@@ -192,10 +202,28 @@ GET http://<compute-server-ip>:19100/modelTF/v1/compute/health
 ```text
 ../../compute -> /app/compute
 ${YG_FT_DATA_ROOT_HOST} -> /data/yg-ft
-../../runtime/compute/logs -> /opt/yg-ft/logs/compute
-../../runtime/compute/training-logs -> /opt/yg-ft/logs/training
+${YG_FT_MODEL_ROOT_HOST} -> /data/yg-ft/models
+${YG_FT_DATASET_ROOT_HOST} -> /data/yg-ft/datasets
+${YG_FT_OUTPUT_ROOT_HOST} -> /data/yg-ft/outputs
+${COMPUTE_LOG_ROOT_HOST} -> /opt/yg-ft/logs/compute
+${TRAINING_LOG_ROOT_HOST} -> /opt/yg-ft/logs/training
 ```

+算力服务器启动前必须先在宿主机创建持久化目录，基座模型、训练数据、训练产物和训练日志都应落在宿主机磁盘上，不能只写入容器层。推荐默认目录：
+
+```bash
+cd <repo-root>/docker/compute
+mkdir -p data/yg-ft/models \
+  data/yg-ft/datasets \
+  data/yg-ft/outputs \
+  data/yg-ft/logs/compute \
+  data/yg-ft/logs/training
+```
+
+默认 `docker/compute/.env.example` 使用 `./data/yg-ft`，该相对路径以 `docker/compute/docker-compose.yml` 所在目录为基准，因此实际宿主机目录是 `<repo-root>/docker/compute/data/yg-ft`。如企业环境模型盘、数据盘、产物盘分盘挂载，可在 `docker/compute/.env` 中分别调整 `YG_FT_MODEL_ROOT_HOST`、`YG_FT_DATASET_ROOT_HOST`、`YG_FT_OUTPUT_ROOT_HOST`、`COMPUTE_LOG_ROOT_HOST`、`TRAINING_LOG_ROOT_HOST`，容器内路径建议保持 `/data/yg-ft/models`、`/data/yg-ft/datasets`、`/data/yg-ft/outputs`，避免训练参数和节点配置复杂化。
+
+页面上传数据集时，文件先进入 Backend API，再由 Backend API 调用目标算力节点的 `POST /modelTF/compute/files/upload`，写入容器内 `/data/yg-ft/datasets/{dataset_id}/`。在默认开发配置下，宿主机可在 `<repo-root>/docker/compute/data/yg-ft/datasets/{dataset_id}/` 看到对应文件。仅创建 bind mount 不会自动让应用侧上传文件出现在算力目录，必须通过这条 File Gateway 链路同步。
+
 ## 应用与算力分离部署

 应用服务器只需要主动访问算力服务器，不要求算力服务器回调应用服务器。
@@ -207,7 +235,7 @@ COMPUTE_API_BASE_URL=http://<compute-server-ip>:19100
 FILE_GATEWAY_BASE_URL=http://<compute-server-ip>:19101
 COMPUTE_SERVICE_TOKEN=change_me
 COMPUTE_STATUS_SYNC_MODE=polling
-COMPUTE_POLL_INTERVAL_SECONDS=10
+COMPUTE_POLL_INTERVAL_SECONDS=3
 COMPUTE_POLL_BATCH_SIZE=100
 ```

@@ -222,6 +250,8 @@ Frontend
    <- Backend Worker 定时轮询 Compute API
 ```

+算力服务默认开启服务间鉴权。`docker/compute/.env` 中保持 `COMPUTE_AUTH_ENABLED=true`，并确保 `COMPUTE_SERVICE_TOKEN` 与 `docker/app/.env` 一致；健康检查路径仍可用于容器探活。
+
 ## 多算力节点部署

 多算力节点仍按“单机多 GPU 节点”部署。每台 GPU 服务器都独立部署一套 `docker/compute`：
@@ -234,6 +264,8 @@ gpu-node-03: docker/compute + /data/yg-ft + 19100/19101

 节点之间默认不互访。应用平台主动访问每个节点的 Compute API/File Gateway，并通过 `compute_nodes`、`resource_replicas`、`resource_sync_jobs` 统一调度和同步。

+节点地址、权重、标签、启用状态和本地路径在前端“算力节点”页面动态维护。新增或编辑节点后，点击“测试”会由 Backend API 主动访问该节点的 `GET /modelTF/v1/compute/health` 和 `GET /modelTF/compute/resources/gpus`，并把健康信息与 GPU 清单同步到 PostgreSQL。
+
 ## 常用命令

 重新构建应用镜像：
--- a/docker/app/.env.example
+++ b/docker/app/.env.example
@@ -41,5 +41,6 @@ FILE_GATEWAY_BASE_URL=http://compute-api:9101
 # The application side polls Compute API for job state to avoid opening reverse network access.
 COMPUTE_MODE=real
 COMPUTE_STATUS_SYNC_MODE=polling
-COMPUTE_POLL_INTERVAL_SECONDS=10
+COMPUTE_POLL_INTERVAL_SECONDS=3
 COMPUTE_POLL_BATCH_SIZE=100
+COMPUTE_REQUEST_TIMEOUT_SECONDS=5
--- a/docker/app/docker-compose.yml
+++ b/docker/app/docker-compose.yml
@@ -21,6 +21,8 @@ services:
          ls -la /usr/share/nginx/html;
          exit 1;
        fi;
+        envsubst '$$BACKEND_PROXY_PASS' < /etc/nginx/templates/default.conf.template > /etc/nginx/conf.d/default.conf;
+        nginx -t;
        nginx -g 'daemon off;'
    networks:
      - yg-ft-app
@@ -58,8 +60,9 @@ services:
      FILE_GATEWAY_BASE_URL: ${FILE_GATEWAY_BASE_URL:-http://compute-api:9101}
      COMPUTE_MODE: ${COMPUTE_MODE:-real}
      COMPUTE_STATUS_SYNC_MODE: ${COMPUTE_STATUS_SYNC_MODE:-polling}
-      COMPUTE_POLL_INTERVAL_SECONDS: ${COMPUTE_POLL_INTERVAL_SECONDS:-10}
+      COMPUTE_POLL_INTERVAL_SECONDS: ${COMPUTE_POLL_INTERVAL_SECONDS:-3}
      COMPUTE_POLL_BATCH_SIZE: ${COMPUTE_POLL_BATCH_SIZE:-100}
+      COMPUTE_REQUEST_TIMEOUT_SECONDS: ${COMPUTE_REQUEST_TIMEOUT_SECONDS:-5}
      PYTHONPATH: /app
    volumes:
      - ../../backend:/app:ro
--- a/docker/compute/.env.example
+++ b/docker/compute/.env.example
@@ -8,14 +8,35 @@ FILE_GATEWAY_PORT=19101
 COMPUTE_API_IMAGE=yg-ft-compute-api:latest

 # The application server actively polls Compute API; compute server does not need reverse access.
+COMPUTE_AUTH_ENABLED=true
 COMPUTE_SERVICE_TOKEN=change_me
 ENABLE_APP_CALLBACK=false

 # LLaMA-Factory is provided by the official hiyouga/llamafactory base image.
 LLAMA_FACTORY_HOME=/app/LLaMA-Factory

+# Persistent host directories on the compute server.
+# Create these directories before starting docker compose. They are mounted into
+# the container so base models, datasets, training outputs and logs survive
+# container recreation or image upgrades.
 YG_FT_DATA_ROOT=/data/yg-ft
-YG_FT_DATA_ROOT_HOST=/data/yg-ft
+YG_FT_DATA_ROOT_HOST=./data/yg-ft
+YG_FT_MODEL_ROOT=/data/yg-ft/models
+YG_FT_MODEL_ROOT_HOST=./data/yg-ft/models
+YG_FT_DATASET_ROOT=/data/yg-ft/datasets
+YG_FT_DATASET_ROOT_HOST=./data/yg-ft/datasets
+YG_FT_OUTPUT_ROOT=/data/yg-ft/outputs
+YG_FT_OUTPUT_ROOT_HOST=./data/yg-ft/outputs
+TRAINING_LOG_ROOT=/opt/yg-ft/logs/training
+TRAINING_LOG_ROOT_HOST=./data/yg-ft/logs/training
+COMPUTE_LOG_ROOT_HOST=./data/yg-ft/logs/compute
+
+# Optional fallback used when nvidia-smi is unavailable.
+# Leave COMPUTE_GPU_COUNT=0 on real GPU servers with working NVIDIA runtime.
+COMPUTE_GPU_COUNT=0
+COMPUTE_GPU_NAME=NVIDIA A800-SXM4-80GB
+COMPUTE_GPU_MEMORY_GB=80
+COMPUTE_GPU_POWER_LIMIT_W=300

 LOG_DIR=/opt/yg-ft/logs/compute
 CUDA_VISIBLE_DEVICES=all
--- a/docker/compute/docker-compose.yml
+++ b/docker/compute/docker-compose.yml
@@ -5,15 +5,25 @@ services:
    gpus: all
    ports:
      - "${COMPUTE_API_PORT:-19100}:9100"
+      - "${FILE_GATEWAY_PORT:-19101}:9100"
    environment:
      COMPUTE_ENV: ${COMPUTE_ENV:-prod}
      COMPUTE_HOST_ID: ${COMPUTE_HOST_ID:-gpu-node-01}
      COMPUTE_EXECUTION_MODE: ${COMPUTE_EXECUTION_MODE:-real}
      MODELTF_ROUTE_PREFIX: ${MODELTF_ROUTE_PREFIX:-/modelTF}
+      COMPUTE_AUTH_ENABLED: ${COMPUTE_AUTH_ENABLED:-true}
      COMPUTE_SERVICE_TOKEN: ${COMPUTE_SERVICE_TOKEN:-change_me}
      ENABLE_APP_CALLBACK: ${ENABLE_APP_CALLBACK:-false}
      LLAMA_FACTORY_HOME: ${LLAMA_FACTORY_HOME:-/app/LLaMA-Factory}
      YG_FT_DATA_ROOT: ${YG_FT_DATA_ROOT:-/data/yg-ft}
+      YG_FT_MODEL_ROOT: ${YG_FT_MODEL_ROOT:-/data/yg-ft/models}
+      YG_FT_DATASET_ROOT: ${YG_FT_DATASET_ROOT:-/data/yg-ft/datasets}
+      YG_FT_OUTPUT_ROOT: ${YG_FT_OUTPUT_ROOT:-/data/yg-ft/outputs}
+      TRAINING_LOG_ROOT: ${TRAINING_LOG_ROOT:-/opt/yg-ft/logs/training}
+      COMPUTE_GPU_COUNT: ${COMPUTE_GPU_COUNT:-0}
+      COMPUTE_GPU_NAME: ${COMPUTE_GPU_NAME:-NVIDIA A800-SXM4-80GB}
+      COMPUTE_GPU_MEMORY_GB: ${COMPUTE_GPU_MEMORY_GB:-80}
+      COMPUTE_GPU_POWER_LIMIT_W: ${COMPUTE_GPU_POWER_LIMIT_W:-300}
      LOG_DIR: ${LOG_DIR:-/opt/yg-ft/logs/compute}
      CUDA_VISIBLE_DEVICES: ${CUDA_VISIBLE_DEVICES:-all}
      NVIDIA_VISIBLE_DEVICES: ${NVIDIA_VISIBLE_DEVICES:-all}
@@ -21,9 +31,12 @@ services:
      PYTHONPATH: /app
    volumes:
      - ../../compute:/app/compute:ro
-      - ${YG_FT_DATA_ROOT_HOST:-/data/yg-ft}:${YG_FT_DATA_ROOT:-/data/yg-ft}
-      - ../../runtime/compute/logs:/opt/yg-ft/logs/compute
-      - ../../runtime/compute/training-logs:/opt/yg-ft/logs/training
+      - ${YG_FT_DATA_ROOT_HOST:-./data/yg-ft}:${YG_FT_DATA_ROOT:-/data/yg-ft}
+      - ${YG_FT_MODEL_ROOT_HOST:-./data/yg-ft/models}:${YG_FT_MODEL_ROOT:-/data/yg-ft/models}
+      - ${YG_FT_DATASET_ROOT_HOST:-./data/yg-ft/datasets}:${YG_FT_DATASET_ROOT:-/data/yg-ft/datasets}
+      - ${YG_FT_OUTPUT_ROOT_HOST:-./data/yg-ft/outputs}:${YG_FT_OUTPUT_ROOT:-/data/yg-ft/outputs}
+      - ${COMPUTE_LOG_ROOT_HOST:-./data/yg-ft/logs/compute}:${LOG_DIR:-/opt/yg-ft/logs/compute}
+      - ${TRAINING_LOG_ROOT_HOST:-./data/yg-ft/logs/training}:${TRAINING_LOG_ROOT:-/opt/yg-ft/logs/training}
    networks:
      - yg-ft-compute
    healthcheck:
--- a/docker/nginx.conf.template
+++ b/docker/nginx.conf.template
@@ -7,11 +7,18 @@ server {

  client_max_body_size 200m;

-  location / {
-    try_files $uri $uri/ /index.html;
+  location ^~ /modelTF/ {
+    proxy_pass ${BACKEND_PROXY_PASS};
+    proxy_http_version 1.1;
+    proxy_set_header Host $host;
+    proxy_set_header X-Real-IP $remote_addr;
+    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+    proxy_set_header X-Forwarded-Proto $scheme;
+    proxy_read_timeout 300s;
+    proxy_send_timeout 300s;
  }

-  location /modelTF {
+  location = /modelTF {
    proxy_pass ${BACKEND_PROXY_PASS};
    proxy_http_version 1.1;
    proxy_set_header Host $host;
@@ -27,4 +34,8 @@ server {
    expires 30d;
    add_header Cache-Control "public, immutable";
  }
+
+  location / {
+    try_files $uri $uri/ /index.html;
+  }
 }