Merge branch 'ft_wyt' of http://www.caoxiaozhu.com:13001/YG-Soft/YG_FT into ft_wyt

- 支持从 PostgreSQL 数据库拉取结构化数据作为训练来源
- 新增 DPO (Direct Preference Optimization) 输出类型
- 支持 chosen/rejected 字段的编辑、校验和发布
- 完善数据预处理切分逻辑和元数据管理
- 移除 OCR 扫描 PDF 功能，保持基础文本解析能力

Co-Authored-By: Claude <noreply@anthropic.com>

.gitignore

@@ -12,8 +12,6 @@ __pycache__/
 build/
 develop-eggs/
 dist/
-!frontend/dist/
-!frontend/dist/**
 node_modules/
 *.tsbuildinfo
 downloads/
@@ -44,6 +42,7 @@ pip-delete-this-directory.txt
 # Runtime data and logs
 runtime/
 backend/runtime/
+backend/storage/
 logs/
 backend/logs/
 *.db
@@ -143,6 +142,7 @@ celerybeat.pid
 
 # Environments
 .env
+!.env.example
 .venv
 env/
 venv/
@@ -150,6 +150,16 @@ ENV/
 env.bak/
 venv.bak/
 
+# Local backend config (含数据库账号密码等敏感信息，勿提交)
+backend/config.yaml
+
+# Agent / IDE 工具产物，不应进版本库
+.codex-backups/
+.pnpm-store/
+.zcode/
+.claude/
+CLAUDE.md
+
 # Spyder project settings
 .spyderproject
 .spyproject
@@ -187,3 +197,17 @@ cython_debug/
 # PyPI configuration file
 .pypirc
 
+docker/llamafactory-latest.tar.gz
+# Compute data - 保留目录结构和 README，忽略子目录内容（日志、模型、数据集等）
+!docker/compute/data/yg-ft/logs/
+docker/compute/data/yg-ft/datasets/*
+docker/compute/data/yg-ft/models/*
+docker/compute/data/yg-ft/outputs/*
+docker/compute/data/yg-ft/logs/**
+!docker/compute/data/yg-ft/logs/compute/
+!docker/compute/data/yg-ft/logs/training/
+!docker/compute/data/yg-ft/**/.gitkeep
+!docker/compute/data/yg-ft/**/README.md
+
+# Offline deployment bundle - 离线部署包（镜像、运行时等大文件，不提交）
+docker/offline/

README.md

@@ -48,6 +48,41 @@ YG_FT/
 - 前端新增 `/compute` 算力节点页面，展示节点地址、权重、标签、启用状态、GPU、队列和资源副本。
 - `compute/engines/llama_factory/adapter.py` 提供 LLaMA-Factory 参数校验、命令生成和日志解析基础能力。
 
+## 前后端一键启动
+
+首次使用前请确保前端依赖已安装、PostgreSQL 已可用。之后在项目根目录执行：
+
+```bash
+bash ./start.sh
+```
+
+脚本会自动补装后端 `requirements.txt`，然后同时启动前端 `http://localhost:16801` 和后端
+`http://127.0.0.1:17861`，按 `Ctrl+C` 会同时停止两个服务。脚本不会自动安装
+前端依赖，也不会启动 PostgreSQL、Redis 或算力服务。
+
+仅检查依赖和端口而不启动服务：
+
+```bash
+bash ./start.sh --check
+```
+
+本地启动推荐只配置数据库主机。脚本会复用 `docker/app/.env` 中已有的
+`POSTGRES_USER`、`POSTGRES_PASSWORD` 和 `POSTGRES_DB`，端口默认使用
+PostgreSQL 标准端口 `5432`：
+
+```bash
+DATABASE_HOST='www.caoxiaozhu.com' bash ./start.sh
+```
+
+也可以在 `docker/app/.env` 中增加：
+
+```env
+DATABASE_HOST=www.caoxiaozhu.com
+```
+
+需要使用非标准端口时再设置 `DATABASE_PORT`。`DATABASE_URL` 仍可作为完整连接串
+高级覆盖项；终端环境变量优先级最高。脚本不会输出数据库密码。
+
 ## 后端启动
 
 ```bash
@@ -67,15 +102,37 @@ GET /modelTF/model-manage
 GET /modelTF/dataset-manage
 GET /modelTF/fine-tune
 GET /modelTF/compute/nodes
+GET /modelTF/data-convert
 ```
 
-本地运行时默认 PostgreSQL 连接：
+### 数据库配置
 
-```text
+后端通过 `backend/.env` 文件配置数据库连接（自动加载，`override=True`）：
-DATABASE_URL=postgresql+psycopg://yg_ft:change_me@localhost:15432/yg_ft
+
+```env
+DATABASE_URL=postgresql+psycopg://用户:密码@数据库地址:端口/库名
+COMPUTE_SERVICE_TOKEN=change_me
 ```
 
-本地启动前需要确保 PostgreSQL 已监听 `localhost:15432`，并已创建 `yg_ft` 数据库和 `yg_ft` 用户。后端启动后会自动创建当前运行表并写入内置管理员账号，运行数据统一写入 PostgreSQL。
+支持远程数据库。连接池参数已针对远程库优化（`connect_timeout=30`、`max_size=20`、`max_waiting=50`）。
+
+### 环境变量
+
+| 变量 | 默认值 | 说明 |
+|---|---|---|
+| `DATABASE_URL` | `postgresql+psycopg://yg_ft:change_me@localhost:15432/yg_ft` | 数据库连接串 |
+| `COMPUTE_SERVICE_TOKEN` | `""` | 算力服务认证 token，需与 compute 一致 |
+| `COMPUTE_STATUS_SYNC_MODE` | `polling` | `off` 禁用轮询（远程库慢时推荐） |
+| `COMPUTE_POLL_INTERVAL_SECONDS` | `3` | 轮询间隔秒数 |
+| `COMPUTE_REQUEST_TIMEOUT_SECONDS` | `5` | 调 compute 的超时秒数 |
+
+### 推荐启动命令（远程数据库）
+
+```cmd
+cd /d E:\yg_ft\backend
+set COMPUTE_STATUS_SYNC_MODE=off
+.\.venv\Scripts\python.exe -m uvicorn app.main:app --reload --port 17861
+```
 
 开发阶段内置登录账号：
 
@@ -98,12 +155,72 @@ npm run dev
 
 ## 算力服务启动
 
-```bash
+算力服务是一个 FastAPI 应用，同时承载 Compute API（模型训练/推理/GPU 管理）和 File Gateway（文件上传下载）路由。Docker 部署时对外暴露两个端口（19100 和 19101）均指向同一服务，方便应用平台分别配置 `api_base_url` 和 `file_gateway_url`。本地开发只需启动一个进程。
-cd compute
+
-uvicorn api.main:app --reload --port 19100
+### 方式一：Docker 启动（推荐）
+
+**1. 构建镜像**（首次或依赖变更后）：
+
+```cmd
+cd /d E:\yg_ft
+docker build -f docker/compute/Dockerfile.compute -t yg-ft-compute-api:latest .
 ```
 
-默认 `COMPUTE_MODE=real`。真实 GPU 接入时，在每台算力服务器上部署 Compute API、Agent、File Gateway 和 LLaMA-Factory，应用平台通过 `compute_nodes.api_base_url` 和 `compute_nodes.file_gateway_url` 主动轮询。仅在隔离联调环境可显式设置 `COMPUTE_MODE=simulator` 或 `COMPUTE_EXECUTION_MODE=simulator`。
+**2. 启动容器**：
+
+```cmd
+cd docker/compute
+cp .env.example .env
+docker compose up -d
+```
+
+**3. 验证**：
+
+```cmd
+curl http://localhost:19100/health
+```
+
+> 注意：构建上下文必须是项目根目录 `E:\yg_ft`（`docker build` 最后的 `.`），因为 Dockerfile 需要 `COPY compute/requirements.txt`。
+
+### 方式二：本地开发启动
+
+**Windows (cmd)：**
+
+```cmd
+cd /d E:\yg_ft\compute
+set PYTHONPATH=E:\yg_ft
+.\.venv\Scripts\python.exe -m uvicorn api.main:app --reload --host 0.0.0.0 --port 19100
+```
+
+> `PYTHONPATH=E:\yg_ft` 是必需的，因为代码使用 `from compute.agent...` 绝对导入。
+> `--host 0.0.0.0` 让其他机器可以通过 IP 访问（算力节点测试需要）。
+
+**Linux / macOS：**
+
+```bash
+cd compute
+PYTHONPATH=.. uvicorn api.main:app --reload --host 0.0.0.0 --port 19100
+```
+
+### 算力节点配置
+
+在平台的「算力节点」页面新增节点，填入：
+- **Compute API**：`http://你的IP:19100`
+- **File Gateway**：`http://你的IP:19101`
+
+本机测试用 `http://localhost:19100`。
+
+### 环境变量说明
+
+| 变量 | 默认值 | 说明 |
+|---|---|---|
+| `COMPUTE_MODE` | `real` | `real` / `simulator`，仅隔离联调用 simulator |
+| `COMPUTE_EXECUTION_MODE` | `real` | 训练执行模式 |
+| `COMPUTE_SERVICE_TOKEN` | `change_me` | 服务间认证 token，需与 backend 一致 |
+| `COMPUTE_AUTH_ENABLED` | `true` | 是否开启 token 认证 |
+| `MODELTF_ROUTE_PREFIX` | `/modelTF` | API 路由前缀 |
+
+应用平台通过数据库 `compute_nodes` 表中的 `api_base_url` 和 `file_gateway_url` 主动轮询算力节点状态。
 
 ## 日志
 

backend/.env.example

@@ -1,9 +0,0 @@
-APP_NAME=YG Fine-Tune Platform API
-APP_ENV=local
-API_PREFIX=/api
-LOG_LEVEL=INFO
-LOG_DIR=./logs
-LOG_FILE_PREFIX=backend
-LOG_ERROR_FILE_PREFIX=error
-LOG_MAX_BYTES=20971520
-LOG_RETENTION_DAYS=10

backend/_check_sessions.py

@@ -0,0 +1,10 @@
+from app.db.platform_store import get_platform_store
+
+store = get_platform_store()
+with store.connect() as conn:
+    rows = conn.execute(
+        "SELECT id, user_id, login_at, logout_at, duration_seconds FROM sessions ORDER BY login_at DESC LIMIT 10"
+    ).fetchall()
+    print(f"sessions count: {len(rows)}")
+    for r in rows:
+        print(f"  user={r['user_id'][:25]}... login={r['login_at']} logout={r['logout_at']} dur={r['duration_seconds']}")

backend/app/api/v1/endpoints/health.py

@@ -10,5 +10,9 @@ logger = get_logger(__name__)
 @router.get("/health")
 async def health_check() -> dict[str, object]:
     logger.info("health check requested")
-    return {"code": 0, "message": "ok", "data": get_platform_store().health_metrics()}
+    return {
+        "code": 0,
+        "message": "ok",
+        "data": get_platform_store().health_metrics(),
+    }
 

backend/app/api/v1/router.py

@@ -1,9 +1,26 @@
-from fastapi import APIRouter
+from fastapi import APIRouter
 
+from app.api.v1.endpoints.data_process import router as data_process_router
 from app.api.v1.endpoints.platform import router as platform_router
 from app.api.v1.endpoints.health import router as health_router
+from app.modules.tenant.router import router as tenant_router
+from app.modules.project.router import router as project_router
+from app.modules.approval.router import router as approval_router
+from app.modules.system.router import router as system_router
+from app.modules.retention.router import router as retention_router
+from app.modules.resource.router import router as resource_router
+from app.modules.gpu.router import router as gpu_router
+from app.modules.data_convert.router import router as data_convert_router
 
 api_router = APIRouter()
 api_router.include_router(health_router, tags=["health"])
+api_router.include_router(data_process_router, tags=["data-process"])
 api_router.include_router(platform_router, tags=["platform"])
-
+api_router.include_router(system_router, tags=["system"])
+api_router.include_router(tenant_router, tags=["tenant"])
+api_router.include_router(project_router, tags=["project"])
+api_router.include_router(approval_router, tags=["approval"])
+api_router.include_router(retention_router, tags=["retention"])
+api_router.include_router(resource_router, tags=["resource"])
+api_router.include_router(gpu_router, tags=["gpu-assignment"])
+api_router.include_router(data_convert_router, tags=["data-convert"])

backend/app/core/auth.py

@@ -0,0 +1,138 @@
+"""鉴权依赖：从 Authorization header 解析当前用户，提供权限校验。"""
+from __future__ import annotations
+
+from typing import Any
+
+from fastapi import Depends, HTTPException, Query, Request, status
+
+from app.db.platform_store import get_platform_store
+
+# 无需鉴权的路径前缀（健康检查、登录等）
+PUBLIC_PATHS = ("/health", "/login", "/system-info")
+
+
+def _extract_token(request: Request) -> str | None:
+    """从 Authorization header 提取 token（格式: Bearer platform-token-{user_id}）。"""
+    auth = request.headers.get("Authorization", "")
+    token = auth.replace("Bearer ", "").strip()
+    if token.startswith("platform-token-"):
+        return token[len("platform-token-"):]
+    return None
+
+
+def get_current_user(request: Request) -> dict[str, Any]:
+    """
+    FastAPI 依赖：解析当前登录用户。
+    - 公开路径（/health, /login 等）直接放行，返回匿名用户。
+    - 无 token 或 token 无效时抛 401。
+    - admin 用户标记为超级管理员，拥有全部权限。
+    """
+    path = request.url.path
+    # 去掉路由前缀后判断
+    for prefix in PUBLIC_PATHS:
+        if path.endswith(prefix):
+            return {"id": None, "username": "anonymous", "role": "viewer", "permissions": [], "protected": False}
+
+    user_id = _extract_token(request)
+    if not user_id:
+        raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="missing or invalid token")
+
+    store = get_platform_store()
+    for u in store.users():
+        if u.get("id") == user_id:
+            return u
+    raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="user not found")
+
+
+def require_admin(current_user: dict[str, Any] = Depends(get_current_user)) -> dict[str, Any]:
+    """FastAPI 依赖：要求当前用户是管理员（role=admin 或 protected）。"""
+    if current_user.get("role") == "admin" or current_user.get("protected"):
+        return current_user
+    raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="admin permission required")
+
+
+def is_admin(user: dict[str, Any]) -> bool:
+    """判断用户是否为管理员（admin 角色或 protected 标记）。"""
+    return user.get("role") == "admin" or user.get("protected", False)
+
+
+def has_resource_access(
+    resource_type: str,
+    resource_id: str,
+    user: dict[str, Any],
+    permission: str = "read",
+) -> bool:
+    """
+    检查用户对某资源是否有指定权限。
+    - admin/protected 用户直接放行（旁路）。
+    - 其他用户检查 acls 表中是否有对应授权。
+    """
+    if user.get("role") == "admin" or user.get("protected"):
+        return True
+
+    store = get_platform_store()
+    acls = store.get_acl(resource_type, resource_id)
+    user_id = user.get("id")
+    user_role = user.get("role")
+
+    for entry in acls:
+        # 按 user 授权
+        if entry.get("principal_type") == "user" and entry.get("principal_id") == user_id:
+            if _permission_covers(entry.get("permission"), permission):
+                return True
+        # 按 role 授权
+        if entry.get("principal_type") == "role" and entry.get("principal_id") == user_role:
+            if _permission_covers(entry.get("permission"), permission):
+                return True
+    return False
+
+
+def _permission_covers(granted: str | None, required: str) -> bool:
+    """权限覆盖判断：write/execute 覆盖 read；admin 覆盖一切。"""
+    if not granted:
+        return False
+    if granted == "admin":
+        return True
+    if granted == required:
+        return True
+    # write 覆盖 read
+    if required == "read" and granted in ("write", "execute"):
+        return True
+    return False
+
+
+def filter_accessible_resource_ids(
+    resource_type: str,
+    all_ids: list[str],
+    user: dict[str, Any],
+) -> list[str]:
+    """
+    从全部资源 ID 中过滤出当前用户可访问的 ID 列表。
+    - admin 直接返回全部。
+    - 普通用户查 acls 表取交集。
+    """
+    if user.get("role") == "admin" or user.get("protected"):
+        return all_ids
+
+    if not all_ids:
+        return []
+
+    store = get_platform_store()
+    user_id = user.get("id")
+    user_role = user.get("role")
+
+    # 查询该用户在该资源类型下有 read 权限的所有 resource_id
+    with store.connect() as conn:
+        rows = conn.execute(
+            """
+            SELECT DISTINCT resource_id FROM acls
+            WHERE resource_type=? AND (
+                (principal_type='user' AND principal_id=?)
+                OR (principal_type='role' AND principal_id=?)
+            )
+            """,
+            (resource_type, user_id, user_role),
+        ).fetchall()
+
+    accessible = {r["resource_id"] for r in rows}
+    return [rid for rid in all_ids if rid in accessible]

backend/app/core/config.py

@@ -1,6 +1,19 @@
 from dataclasses import dataclass
 from functools import lru_cache
 import os
+from typing import Any
+
+try:
+    from pathlib import Path as _Path
+
+    from dotenv import load_dotenv
+
+    # 显式指定 backend 目录下的 .env，并强制覆盖已有环境变量，
+    # 确保远程数据库配置生效，不被本地默认值或残留环境变量影响。
+    _env_path = _Path(__file__).resolve().parent.parent.parent / ".env"
+    load_dotenv(dotenv_path=_env_path, override=True)
+except ImportError:
+    pass
 
 
 def _int_env(name: str, default: int) -> int:
@@ -17,6 +30,24 @@ def _list_env(name: str, default: list[str]) -> list[str]:
     return [item.strip() for item in raw.split(",") if item.strip()]
 
 
+def _bool_env(name: str, default: bool) -> bool:
+    raw = os.getenv(name)
+    if raw is None or raw.strip() == "":
+        return default
+    return raw.strip().lower() in {"1", "true", "yes", "on"}
+
+
+def docs_kwargs(enabled: bool) -> dict[str, Any]:
+    """Swagger UI / ReDoc / OpenAPI schema 路由开关。
+
+    关闭时 FastAPI 不注册 /docs、/redoc、/openapi.json，访问一律返回 404，
+    避免未授权访问泄露 API 结构。
+    """
+    if enabled:
+        return {}
+    return {"docs_url": None, "redoc_url": None, "openapi_url": None}
+
+
 @dataclass(frozen=True)
 class Settings:
     app_name: str = os.getenv("APP_NAME", "YG Fine-Tune Platform API")
@@ -28,12 +59,15 @@ class Settings:
     compute_mode: str = os.getenv("COMPUTE_MODE", "real")
     compute_status_sync_mode: str = os.getenv("COMPUTE_STATUS_SYNC_MODE", "polling")
     compute_poll_interval_seconds: int = _int_env("COMPUTE_POLL_INTERVAL_SECONDS", 3)
+    compute_request_timeout_seconds: int = _int_env("COMPUTE_REQUEST_TIMEOUT_SECONDS", 5)
+    compute_service_token: str = os.getenv("COMPUTE_SERVICE_TOKEN", "")
     log_level: str = os.getenv("LOG_LEVEL", "INFO")
     log_dir: str = os.getenv("LOG_DIR", "./logs")
     log_file_prefix: str = os.getenv("LOG_FILE_PREFIX", "backend")
     log_error_file_prefix: str = os.getenv("LOG_ERROR_FILE_PREFIX", "error")
     log_max_bytes: int = _int_env("LOG_MAX_BYTES", 20 * 1024 * 1024)
     log_retention_days: int = _int_env("LOG_RETENTION_DAYS", 10)
+    enable_docs: bool = None  # type: ignore[assignment]
 
     def __post_init__(self) -> None:
         object.__setattr__(
@@ -49,6 +83,15 @@ class Settings:
                 ],
             ),
         )
+        # Swagger UI / ReDoc / OpenAPI 文档路由开关：
+        # 未显式配置 ENABLE_DOCS 时，仅本地/开发环境开放，生产环境默认关闭，
+        # 避免未授权访问泄露 API 结构。从运行时环境读取 APP_ENV，而非类定义时
+        # 缓存的默认值，保证生产默认关闭始终生效且便于测试。
+        object.__setattr__(
+            self,
+            "enable_docs",
+            _bool_env("ENABLE_DOCS", os.getenv("APP_ENV", "local") != "prod"),
+        )
 
 
 @lru_cache

backend/app/db/sql/000_full_init.sql

@@ -0,0 +1,755 @@
+-- ============================================================================
+-- YG Fine-Tune Platform — PostgreSQL 完整初始化脚本（一键建库建表）
+-- ============================================================================
+-- 用途：切换到新的 PG 数据集时，一次性创建平台运行所需的全部数据库对象与
+--       基础种子数据（幂等，可重复执行）。
+--
+-- 覆盖范围（与运行时代码实际使用的表一致）：
+--   001_platform_runtime.sql       平台核心表
+--   002_governance.sql             治理表（租户 / 审批 / 审计 / 留存）
+--   003_tenant_quota.sql           租户配额列
+--   003_model_path_governance.sql  模型可训练标识列
+--   002_data_process.sql           数据处理表 + 数据集扩展列
+--   本文件补充：data_convert_tasks（数据转换任务，运行时代码引用但原脚本缺失）
+--   种子数据：admin / operator 两个初始用户
+--
+-- 说明：
+--   * 本脚本通过 psql 执行，包含 DO $$ ... $$ 块与事务，不能用应用的
+--     executescript()（按分号切分）执行。
+--   * 应用启动时 PlatformStore.ensure_schema() 只会自动执行
+--     001 / 002_governance / 003_tenant_quota；数据处理表需另跑
+--     002_data_process.sql（本脚本已包含）。应用首次启动还会自动补充
+--     admin/operator 种子用户（本脚本已包含，二选一即可）。
+--   * 脚本内所有 DDL 均使用 IF NOT EXISTS / ADD COLUMN IF NOT EXISTS，
+--     可在已初始化的库上安全重复执行。
+--
+-- 执行步骤（详见 docs/database-config.md）：
+--   1. 以超级用户创建角色与数据库（必须单独执行，不能放进事务）：
+--        CREATE ROLE  yg_ft  LOGIN PASSWORD '请改为强密码';
+--        CREATE DATABASE yg_ft OWNER yg_ft;
+--   2. 连接目标库执行本脚本：
+--        psql "postgresql://yg_ft:密码@<host>:5432/yg_ft" -f backend/app/db/sql/000_full_init.sql
+--   3. 可选：为 superuser 授权
+--        ALTER ROLE yg_ft SUPERUSER;   -- 仅当需要执行 CREATE EXTENSION 等
+-- ============================================================================
+
+BEGIN;
+
+-- ============================================================================
+-- 一、平台核心表（来源：001_platform_runtime.sql）
+-- ============================================================================
+
+CREATE TABLE IF NOT EXISTS users (
+    id TEXT PRIMARY KEY,
+    username TEXT NOT NULL UNIQUE,
+    password_hash TEXT NOT NULL,
+    display_name TEXT NOT NULL,
+    role TEXT NOT NULL,
+    status TEXT NOT NULL,
+    permissions TEXT NOT NULL,
+    create_time TEXT NOT NULL,
+    last_login TEXT,
+    protected INTEGER NOT NULL DEFAULT 0
+);
+
+CREATE TABLE IF NOT EXISTS models (
+    id TEXT PRIMARY KEY,
+    name TEXT NOT NULL UNIQUE,
+    type TEXT NOT NULL,
+    purpose TEXT NOT NULL,
+    model_source TEXT NOT NULL,
+    description TEXT,
+    path TEXT,
+    api_url TEXT,
+    api_key TEXT,
+    online_model_name TEXT,
+    can_train INTEGER NOT NULL DEFAULT 0,
+    create_time TEXT NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS trained_models (
+    id TEXT PRIMARY KEY,
+    name TEXT NOT NULL UNIQUE,
+    train_methods TEXT NOT NULL,
+    base_model_path TEXT,
+    create_time TEXT NOT NULL,
+    merged INTEGER NOT NULL DEFAULT 0,
+    merging INTEGER NOT NULL DEFAULT 0,
+    merged_path TEXT,
+    artifact_dir TEXT,
+    compute_node_id TEXT,
+    compute_node_name TEXT
+);
+
+CREATE TABLE IF NOT EXISTS model_lineage (
+    id TEXT PRIMARY KEY,
+    child_resource_type TEXT NOT NULL,
+    child_resource_id TEXT NOT NULL,
+    parent_resource_type TEXT NOT NULL,
+    parent_resource_id TEXT NOT NULL,
+    relation_type TEXT NOT NULL,
+    compute_job_id TEXT,
+    payload TEXT NOT NULL,
+    create_time TEXT NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS model_artifacts (
+    id TEXT PRIMARY KEY,
+    model_id TEXT NOT NULL,
+    model_kind TEXT NOT NULL,
+    artifact_type TEXT NOT NULL,
+    path TEXT NOT NULL,
+    size_bytes BIGINT NOT NULL DEFAULT 0,
+    checksum_sha256 TEXT,
+    metadata TEXT NOT NULL,
+    compute_job_id TEXT,
+    create_time TEXT NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS model_export_jobs (
+    id TEXT PRIMARY KEY,
+    trained_model_id TEXT,
+    compute_job_id TEXT NOT NULL,
+    node_id TEXT,
+    export_type TEXT NOT NULL,
+    quantization_bit INTEGER NOT NULL DEFAULT 0,
+    status TEXT NOT NULL,
+    output_dir TEXT,
+    payload TEXT NOT NULL,
+    create_time TEXT NOT NULL,
+    completed_at TEXT
+);
+
+CREATE TABLE IF NOT EXISTS datasets (
+    id TEXT PRIMARY KEY,
+    name TEXT NOT NULL UNIQUE,
+    type TEXT NOT NULL,
+    storage_type TEXT NOT NULL,
+    source TEXT NOT NULL,
+    task_id TEXT,
+    size TEXT,
+    count INTEGER NOT NULL DEFAULT 0,
+    description TEXT,
+    create_time TEXT NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS dataset_files (
+    id TEXT PRIMARY KEY,
+    dataset_id TEXT NOT NULL REFERENCES datasets(id) ON DELETE CASCADE,
+    name TEXT NOT NULL,
+    size TEXT,
+    content TEXT NOT NULL,
+    active_version_id TEXT NOT NULL,
+    versions TEXT NOT NULL,
+    create_time TEXT NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS compute_nodes (
+    id TEXT PRIMARY KEY,
+    code TEXT NOT NULL UNIQUE,
+    name TEXT NOT NULL,
+    api_base_url TEXT NOT NULL,
+    file_gateway_url TEXT NOT NULL,
+    enabled INTEGER NOT NULL DEFAULT 1,
+    scheduler_status TEXT NOT NULL,
+    scheduler_weight INTEGER NOT NULL DEFAULT 100,
+    tags TEXT NOT NULL,
+    gpu_count INTEGER NOT NULL DEFAULT 0,
+    current_running_jobs INTEGER NOT NULL DEFAULT 0,
+    max_parallel_jobs INTEGER NOT NULL DEFAULT 2,
+    data_root TEXT NOT NULL,
+    model_root TEXT NOT NULL,
+    log_root TEXT NOT NULL,
+    api_version TEXT NOT NULL DEFAULT 'v1',
+    capabilities TEXT NOT NULL DEFAULT '[]',
+    description TEXT,
+    last_health_check_at TEXT,
+    health_detail TEXT NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS gpus (
+    id TEXT PRIMARY KEY,
+    node_id TEXT NOT NULL REFERENCES compute_nodes(id) ON DELETE CASCADE,
+    gpu_index INTEGER NOT NULL,
+    uuid TEXT NOT NULL,
+    name TEXT NOT NULL,
+    memory_total_gb DOUBLE PRECISION NOT NULL,
+    power_limit_w DOUBLE PRECISION NOT NULL,
+    base_temperature INTEGER NOT NULL,
+    last_seen_at TEXT
+);
+
+CREATE TABLE IF NOT EXISTS fine_tune_tasks (
+    id TEXT PRIMARY KEY,
+    name TEXT NOT NULL UNIQUE,
+    payload TEXT NOT NULL,
+    status TEXT NOT NULL,
+    progress INTEGER NOT NULL DEFAULT 0,
+    process_id INTEGER,
+    create_time TEXT NOT NULL,
+    start_time TEXT,
+    completed_at TEXT,
+    compute_node_id TEXT REFERENCES compute_nodes(id) ON DELETE SET NULL,
+    gpus TEXT NOT NULL,
+    sync_job_id TEXT,
+    compute_job_id TEXT
+);
+
+CREATE TABLE IF NOT EXISTS fine_tune_metrics (
+    id TEXT PRIMARY KEY,
+    task_id TEXT NOT NULL REFERENCES fine_tune_tasks(id) ON DELETE CASCADE,
+    step INTEGER NOT NULL,
+    epoch DOUBLE PRECISION,
+    loss DOUBLE PRECISION,
+    grad_norm DOUBLE PRECISION,
+    learning_rate DOUBLE PRECISION,
+    raw TEXT NOT NULL,
+    create_time TEXT NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS fine_tune_checkpoints (
+    id TEXT PRIMARY KEY,
+    task_id TEXT NOT NULL REFERENCES fine_tune_tasks(id) ON DELETE CASCADE,
+    step INTEGER NOT NULL,
+    name TEXT NOT NULL,
+    path TEXT NOT NULL,
+    size_bytes BIGINT NOT NULL DEFAULT 0,
+    create_time TEXT NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS compute_jobs (
+    id TEXT PRIMARY KEY,
+    task_id TEXT REFERENCES fine_tune_tasks(id) ON DELETE SET NULL,
+    node_id TEXT REFERENCES compute_nodes(id) ON DELETE SET NULL,
+    engine TEXT NOT NULL,
+    status TEXT NOT NULL,
+    command TEXT NOT NULL,
+    output_dir TEXT,
+    log_file TEXT,
+    payload TEXT NOT NULL,
+    create_time TEXT NOT NULL,
+    update_time TEXT NOT NULL,
+    completed_at TEXT
+);
+
+CREATE TABLE IF NOT EXISTS gpu_allocations (
+    id TEXT PRIMARY KEY,
+    task_id TEXT REFERENCES fine_tune_tasks(id) ON DELETE CASCADE,
+    compute_job_id TEXT,
+    node_id TEXT REFERENCES compute_nodes(id) ON DELETE CASCADE,
+    gpu_index INTEGER NOT NULL,
+    status TEXT NOT NULL,
+    create_time TEXT NOT NULL,
+    released_at TEXT
+);
+
+CREATE TABLE IF NOT EXISTS scheduler_locks (
+    lock_key TEXT PRIMARY KEY,
+    owner TEXT NOT NULL,
+    expires_at TEXT NOT NULL,
+    create_time TEXT NOT NULL,
+    update_time TEXT NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS resource_replicas (
+    id TEXT PRIMARY KEY,
+    node_id TEXT NOT NULL REFERENCES compute_nodes(id) ON DELETE CASCADE,
+    resource_type TEXT NOT NULL,
+    resource_id TEXT NOT NULL,
+    local_path TEXT NOT NULL,
+    status TEXT NOT NULL,
+    sync_status TEXT NOT NULL,
+    checksum_sha256 TEXT,
+    byte_size BIGINT NOT NULL DEFAULT 0,
+    last_checked_at TEXT,
+    last_error TEXT,
+    create_time TEXT NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS resource_sync_jobs (
+    id TEXT PRIMARY KEY,
+    target_node_id TEXT NOT NULL,
+    resources TEXT NOT NULL,
+    status TEXT NOT NULL,
+    progress INTEGER NOT NULL DEFAULT 0,
+    create_time TEXT NOT NULL,
+    completed_at TEXT
+);
+
+CREATE TABLE IF NOT EXISTS eval_tasks (
+    id TEXT PRIMARY KEY,
+    name TEXT NOT NULL,
+    payload TEXT NOT NULL,
+    status TEXT NOT NULL,
+    create_time TEXT NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS eval_dimensions (
+    id TEXT PRIMARY KEY,
+    name TEXT NOT NULL,
+    payload TEXT NOT NULL,
+    is_active INTEGER NOT NULL DEFAULT 1,
+    is_default INTEGER NOT NULL DEFAULT 0,
+    create_time TEXT NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS compare_tasks (
+    id TEXT PRIMARY KEY,
+    name TEXT NOT NULL,
+    payload TEXT NOT NULL,
+    status TEXT NOT NULL,
+    create_time TEXT NOT NULL
+);
+
+CREATE INDEX IF NOT EXISTS idx_fine_tune_status ON fine_tune_tasks(status);
+CREATE INDEX IF NOT EXISTS idx_fine_tune_compute_job ON fine_tune_tasks(compute_job_id);
+CREATE INDEX IF NOT EXISTS idx_fine_tune_compute_node_status ON fine_tune_tasks(compute_node_id, status);
+CREATE INDEX IF NOT EXISTS idx_model_lineage_child ON model_lineage(child_resource_type, child_resource_id);
+CREATE INDEX IF NOT EXISTS idx_model_lineage_parent ON model_lineage(parent_resource_type, parent_resource_id);
+CREATE INDEX IF NOT EXISTS idx_model_artifacts_model ON model_artifacts(model_kind, model_id, artifact_type);
+CREATE INDEX IF NOT EXISTS idx_model_export_jobs_model ON model_export_jobs(trained_model_id, create_time DESC);
+CREATE INDEX IF NOT EXISTS idx_model_export_jobs_compute ON model_export_jobs(compute_job_id);
+CREATE INDEX IF NOT EXISTS idx_fine_tune_metrics_task_step ON fine_tune_metrics(task_id, step);
+CREATE UNIQUE INDEX IF NOT EXISTS uq_fine_tune_metrics_task_step_epoch ON fine_tune_metrics(task_id, step, epoch);
+CREATE INDEX IF NOT EXISTS idx_fine_tune_checkpoints_task_step ON fine_tune_checkpoints(task_id, step);
+CREATE UNIQUE INDEX IF NOT EXISTS uq_fine_tune_checkpoints_task_path ON fine_tune_checkpoints(task_id, path);
+CREATE INDEX IF NOT EXISTS idx_compute_jobs_task ON compute_jobs(task_id);
+CREATE INDEX IF NOT EXISTS idx_compute_jobs_node_status ON compute_jobs(node_id, status);
+CREATE INDEX IF NOT EXISTS idx_gpu_allocations_node_status ON gpu_allocations(node_id, status);
+CREATE UNIQUE INDEX IF NOT EXISTS uq_gpu_allocations_active ON gpu_allocations(node_id, gpu_index) WHERE status IN ('allocated','running');
+CREATE INDEX IF NOT EXISTS idx_scheduler_locks_expires ON scheduler_locks(expires_at);
+CREATE INDEX IF NOT EXISTS idx_dataset_files_dataset ON dataset_files(dataset_id);
+CREATE INDEX IF NOT EXISTS idx_gpus_node ON gpus(node_id);
+CREATE UNIQUE INDEX IF NOT EXISTS uq_gpus_node_index ON gpus(node_id, gpu_index);
+CREATE INDEX IF NOT EXISTS idx_replicas_resource ON resource_replicas(resource_type, resource_id);
+CREATE UNIQUE INDEX IF NOT EXISTS uq_replicas_node_resource ON resource_replicas(node_id, resource_type, resource_id);
+CREATE INDEX IF NOT EXISTS idx_sync_jobs_node_status ON resource_sync_jobs(target_node_id, status);
+CREATE INDEX IF NOT EXISTS idx_eval_tasks_status ON eval_tasks(status);
+CREATE INDEX IF NOT EXISTS idx_eval_dimensions_active ON eval_dimensions(is_active);
+CREATE INDEX IF NOT EXISTS idx_compare_tasks_status ON compare_tasks(status);
+
+-- ---- 项目 / 租户 ----
+
+CREATE TABLE IF NOT EXISTS projects (
+    id TEXT PRIMARY KEY,
+    tenant_id TEXT NOT NULL DEFAULT 'default',
+    name TEXT NOT NULL,
+    code TEXT NOT NULL,
+    description TEXT,
+    quota TEXT,
+    status TEXT NOT NULL DEFAULT 'active',
+    create_time TEXT NOT NULL,
+    create_by TEXT,
+    updated_at TEXT
+);
+
+CREATE TABLE IF NOT EXISTS project_members (
+    project_id TEXT NOT NULL REFERENCES projects(id) ON DELETE CASCADE,
+    user_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
+    role TEXT NOT NULL DEFAULT 'member',
+    create_time TEXT NOT NULL,
+    PRIMARY KEY (project_id, user_id)
+);
+
+CREATE TABLE IF NOT EXISTS roles (
+    id TEXT PRIMARY KEY,
+    name TEXT NOT NULL,
+    permissions TEXT NOT NULL DEFAULT '[]',
+    create_time TEXT
+);
+
+CREATE TABLE IF NOT EXISTS sessions (
+    id TEXT PRIMARY KEY,
+    user_id TEXT NOT NULL,
+    issued_at TEXT NOT NULL,
+    expires_at TEXT NOT NULL,
+    ip TEXT
+);
+
+CREATE TABLE IF NOT EXISTS acls (
+    id TEXT PRIMARY KEY,
+    resource_type TEXT NOT NULL,
+    resource_id TEXT NOT NULL,
+    principal_type TEXT NOT NULL,
+    principal_id TEXT NOT NULL,
+    permission TEXT NOT NULL,
+    create_time TEXT
+);
+
+-- ============================================================================
+-- 二、治理表（来源：002_governance.sql）
+-- ============================================================================
+
+CREATE TABLE IF NOT EXISTS tenants (
+    id TEXT PRIMARY KEY,
+    name TEXT NOT NULL,
+    code TEXT,
+    status TEXT DEFAULT 'active',
+    owner_user_id TEXT,
+    quota TEXT,
+    retention_policy_id TEXT,
+    create_time TEXT
+);
+
+CREATE TABLE IF NOT EXISTS approval_templates (
+    id TEXT PRIMARY KEY,
+    name TEXT NOT NULL,
+    steps TEXT,
+    create_time TEXT
+);
+
+CREATE TABLE IF NOT EXISTS approval_instances (
+    id TEXT PRIMARY KEY,
+    template_id TEXT,
+    resource_type TEXT,
+    resource_id TEXT,
+    applicant_id TEXT,
+    status TEXT DEFAULT 'pending',
+    current_step INTEGER DEFAULT 0,
+    create_time TEXT
+);
+
+CREATE TABLE IF NOT EXISTS approval_steps (
+    id TEXT PRIMARY KEY,
+    instance_id TEXT,
+    step_index INTEGER,
+    approver_id TEXT,
+    status TEXT DEFAULT 'pending',
+    comment TEXT,
+    time TEXT
+);
+
+CREATE TABLE IF NOT EXISTS audit_logs (
+    id TEXT PRIMARY KEY,
+    tenant_id TEXT,
+    project_id TEXT,
+    actor_id TEXT,
+    action TEXT,
+    target_type TEXT,
+    target_id TEXT,
+    detail TEXT,
+    client_ip TEXT,
+    time TEXT
+);
+
+CREATE INDEX IF NOT EXISTS idx_audit_tenant ON audit_logs(tenant_id);
+CREATE INDEX IF NOT EXISTS idx_audit_project ON audit_logs(project_id);
+CREATE INDEX IF NOT EXISTS idx_audit_action ON audit_logs(action);
+CREATE INDEX IF NOT EXISTS idx_audit_time ON audit_logs(time);
+
+CREATE TABLE IF NOT EXISTS retention_policies (
+    id TEXT PRIMARY KEY,
+    name TEXT NOT NULL,
+    scope TEXT,
+    rule TEXT,
+    status TEXT DEFAULT 'active',
+    create_time TEXT,
+    create_by TEXT,
+    updated_at TEXT
+);
+
+-- ============================================================================
+-- 三、租户配额扩展（来源：003_tenant_quota.sql）
+-- ============================================================================
+
+ALTER TABLE tenants ADD COLUMN IF NOT EXISTS gpu_quota TEXT;
+ALTER TABLE tenants ADD COLUMN IF NOT EXISTS storage_quota TEXT;
+
+-- ============================================================================
+-- 四、模型路径治理（来源：003_model_path_governance.sql）
+--   models.can_train 已在建表语句中声明；以下为兼容旧库的幂等语句。
+-- ============================================================================
+
+ALTER TABLE models ADD COLUMN IF NOT EXISTS can_train INTEGER NOT NULL DEFAULT 0;
+ALTER TABLE trained_models ADD COLUMN IF NOT EXISTS artifact_dir TEXT;
+
+-- 按规则推定已有模型的 can_train（新库为空表，此语句为 no-op）
+UPDATE models
+SET can_train = CASE
+    WHEN path IS NOT NULL AND path != '' AND model_source IS NOT NULL AND model_source != 'api' THEN 1
+    ELSE 0
+END;
+
+-- ============================================================================
+-- 五、数据处理（来源：002_data_process.sql，去掉其外层 BEGIN/COMMIT）
+--   数据集扩展列
+-- ============================================================================
+
+ALTER TABLE datasets ADD COLUMN IF NOT EXISTS source_task_id TEXT;
+ALTER TABLE datasets ADD COLUMN IF NOT EXISTS size_bytes BIGINT NOT NULL DEFAULT 0;
+ALTER TABLE datasets ADD COLUMN IF NOT EXISTS record_count BIGINT NOT NULL DEFAULT 0;
+ALTER TABLE datasets ADD COLUMN IF NOT EXISTS metadata TEXT NOT NULL DEFAULT '{}';
+ALTER TABLE datasets ADD COLUMN IF NOT EXISTS tenant_id TEXT;
+ALTER TABLE datasets ADD COLUMN IF NOT EXISTS project_id TEXT;
+ALTER TABLE datasets ADD COLUMN IF NOT EXISTS owner_id TEXT;
+ALTER TABLE datasets ADD COLUMN IF NOT EXISTS created_by TEXT;
+ALTER TABLE datasets ADD COLUMN IF NOT EXISTS created_at TIMESTAMPTZ NOT NULL DEFAULT now();
+ALTER TABLE datasets ADD COLUMN IF NOT EXISTS updated_at TIMESTAMPTZ NOT NULL DEFAULT now();
+ALTER TABLE datasets ADD COLUMN IF NOT EXISTS deleted_at TIMESTAMPTZ;
+
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS storage_object_id TEXT;
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS current_version_id TEXT;
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS size_bytes BIGINT NOT NULL DEFAULT 0;
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS record_count BIGINT NOT NULL DEFAULT 0;
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS file_format VARCHAR(40);
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS checksum_sha256 CHAR(64);
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS version_no INTEGER NOT NULL DEFAULT 1;
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS source_task_id TEXT;
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS tenant_id TEXT;
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS project_id TEXT;
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS created_by TEXT;
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS metadata TEXT NOT NULL DEFAULT '{}';
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS created_at TIMESTAMPTZ NOT NULL DEFAULT now();
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS updated_at TIMESTAMPTZ NOT NULL DEFAULT now();
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS deleted_at TIMESTAMPTZ;
+
+-- ---- 数据处理任务 / 源文件 / 预览 / 结果 ----
+
+CREATE TABLE IF NOT EXISTS data_process_tasks (
+    id TEXT PRIMARY KEY,
+    name VARCHAR(150) NOT NULL,
+    description TEXT,
+    status VARCHAR(20) NOT NULL DEFAULT 'pending'
+        CHECK (status IN ('pending', 'running', 'completed', 'failed', 'stopped')),
+    process_type VARCHAR(20) NOT NULL
+        CHECK (process_type IN ('structured', 'unstructured', 'external')),
+    source_dataset_id TEXT REFERENCES datasets(id) ON DELETE SET NULL,
+    output_dataset_id TEXT REFERENCES datasets(id) ON DELETE SET NULL,
+    config TEXT NOT NULL DEFAULT '{}',
+    progress NUMERIC(5,2) NOT NULL DEFAULT 0 CHECK (progress >= 0 AND progress <= 100),
+    input_count BIGINT NOT NULL DEFAULT 0 CHECK (input_count >= 0),
+    output_count BIGINT NOT NULL DEFAULT 0 CHECK (output_count >= 0),
+    filtered_count BIGINT NOT NULL DEFAULT 0 CHECK (filtered_count >= 0),
+    duplicate_count BIGINT NOT NULL DEFAULT 0 CHECK (duplicate_count >= 0),
+    error_count BIGINT NOT NULL DEFAULT 0 CHECK (error_count >= 0),
+    failure_reason TEXT,
+    generation_run_id TEXT,
+    results_confirmed BOOLEAN NOT NULL DEFAULT TRUE,
+    workflow_step VARCHAR(20) NOT NULL DEFAULT 'create'
+        CHECK (workflow_step IN ('create', 'model', 'upload', 'preview', 'generate', 'results')),
+    preview_status VARCHAR(20) NOT NULL DEFAULT 'idle'
+        CHECK (preview_status IN ('idle', 'queued', 'running', 'completed', 'failed', 'cancelled')),
+    preview_progress NUMERIC(5,2) NOT NULL DEFAULT 0
+        CHECK (preview_progress >= 0 AND preview_progress <= 100),
+    preview_run_id TEXT,
+    preview_failure_reason TEXT,
+    preview_total_files INTEGER NOT NULL DEFAULT 0 CHECK (preview_total_files >= 0),
+    preview_completed_files INTEGER NOT NULL DEFAULT 0 CHECK (preview_completed_files >= 0),
+    tenant_id TEXT,
+    project_id TEXT,
+    owner_id TEXT,
+    approval_status VARCHAR(30) NOT NULL DEFAULT 'not_required',
+    created_by TEXT,
+    updated_by TEXT,
+    deleted_by TEXT,
+    started_at TIMESTAMPTZ,
+    completed_at TIMESTAMPTZ,
+    created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+    updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+    deleted_at TIMESTAMPTZ
+);
+
+ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS generation_run_id TEXT;
+ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS results_confirmed BOOLEAN NOT NULL DEFAULT TRUE;
+ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS workflow_step VARCHAR(20);
+ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_status VARCHAR(20);
+ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_progress NUMERIC(5,2);
+ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_run_id TEXT;
+ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_failure_reason TEXT;
+ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_total_files INTEGER;
+ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_completed_files INTEGER;
+
+ALTER TABLE data_process_tasks ALTER COLUMN workflow_step SET DEFAULT 'create';
+ALTER TABLE data_process_tasks ALTER COLUMN workflow_step SET NOT NULL;
+ALTER TABLE data_process_tasks ALTER COLUMN preview_status SET DEFAULT 'idle';
+ALTER TABLE data_process_tasks ALTER COLUMN preview_status SET NOT NULL;
+ALTER TABLE data_process_tasks ALTER COLUMN preview_progress SET DEFAULT 0;
+ALTER TABLE data_process_tasks ALTER COLUMN preview_progress SET NOT NULL;
+ALTER TABLE data_process_tasks ALTER COLUMN preview_total_files SET DEFAULT 0;
+ALTER TABLE data_process_tasks ALTER COLUMN preview_total_files SET NOT NULL;
+ALTER TABLE data_process_tasks ALTER COLUMN preview_completed_files SET DEFAULT 0;
+ALTER TABLE data_process_tasks ALTER COLUMN preview_completed_files SET NOT NULL;
+
+CREATE UNIQUE INDEX IF NOT EXISTS uq_data_process_tasks_name_alive
+    ON data_process_tasks(name) WHERE deleted_at IS NULL;
+CREATE INDEX IF NOT EXISTS idx_data_process_tasks_scope_status
+    ON data_process_tasks(tenant_id, project_id, status, created_at DESC)
+    WHERE deleted_at IS NULL;
+CREATE INDEX IF NOT EXISTS idx_data_process_tasks_creator_created
+    ON data_process_tasks(created_by, created_at DESC) WHERE deleted_at IS NULL;
+
+CREATE TABLE IF NOT EXISTS data_process_source_files (
+    id TEXT PRIMARY KEY,
+    task_id TEXT NOT NULL REFERENCES data_process_tasks(id) ON DELETE CASCADE,
+    storage_object_id TEXT,
+    name TEXT NOT NULL,
+    size_bytes BIGINT NOT NULL DEFAULT 0 CHECK (size_bytes >= 0),
+    record_count BIGINT NOT NULL DEFAULT 0 CHECK (record_count >= 0),
+    file_format VARCHAR(40),
+    checksum_sha256 CHAR(64) NOT NULL,
+    version_no INTEGER NOT NULL DEFAULT 1 CHECK (version_no > 0),
+    content TEXT NOT NULL,
+    content_preview TEXT,
+    metadata TEXT NOT NULL DEFAULT '{}',
+    tenant_id TEXT,
+    project_id TEXT,
+    created_by TEXT,
+    created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+    updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+    deleted_at TIMESTAMPTZ
+);
+
+CREATE INDEX IF NOT EXISTS idx_data_process_source_files_task
+    ON data_process_source_files(task_id, created_at) WHERE deleted_at IS NULL;
+CREATE UNIQUE INDEX IF NOT EXISTS uq_data_process_source_checksum_alive
+    ON data_process_source_files(task_id, checksum_sha256) WHERE deleted_at IS NULL;
+
+CREATE TABLE IF NOT EXISTS data_process_preview_items (
+    id TEXT PRIMARY KEY,
+    task_id TEXT NOT NULL REFERENCES data_process_tasks(id) ON DELETE CASCADE,
+    source_file_id TEXT REFERENCES data_process_source_files(id) ON DELETE CASCADE,
+    original_content TEXT NOT NULL DEFAULT '',
+    edited_content TEXT NOT NULL DEFAULT '',
+    source_start INTEGER CHECK (source_start IS NULL OR source_start >= 0),
+    source_end INTEGER CHECK (source_end IS NULL OR source_end >= 0),
+    source_start_line INTEGER CHECK (source_start_line IS NULL OR source_start_line > 0),
+    source_end_line INTEGER CHECK (source_end_line IS NULL OR source_end_line > 0),
+    token_count INTEGER NOT NULL DEFAULT 0 CHECK (token_count >= 0),
+    status VARCHAR(20) NOT NULL DEFAULT 'original'
+        CHECK (status IN ('original', 'modified', 'manual', 'invalid')),
+    quality_score TEXT NOT NULL DEFAULT '{}',
+    created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+    updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+    CHECK (source_start IS NULL OR source_end IS NULL OR source_end >= source_start),
+    CHECK (source_start_line IS NULL OR source_end_line IS NULL OR source_end_line >= source_start_line)
+);
+
+CREATE INDEX IF NOT EXISTS idx_data_process_preview_task_file
+    ON data_process_preview_items(task_id, source_file_id, created_at);
+
+CREATE TABLE IF NOT EXISTS data_process_results (
+    id TEXT PRIMARY KEY,
+    task_id TEXT NOT NULL REFERENCES data_process_tasks(id) ON DELETE CASCADE,
+    preview_item_id TEXT REFERENCES data_process_preview_items(id) ON DELETE SET NULL,
+    instruction TEXT NOT NULL,
+    input TEXT NOT NULL DEFAULT '',
+    output TEXT NOT NULL,
+    original_instruction TEXT,
+    original_input TEXT,
+    original_output TEXT,
+    status VARCHAR(20) NOT NULL DEFAULT 'valid'
+        CHECK (status IN ('valid', 'modified', 'invalid')),
+    error TEXT,
+    split VARCHAR(20) CHECK (split IS NULL OR split IN ('train', 'validation', 'test')),
+    quality_score TEXT NOT NULL DEFAULT '{}',
+    created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+    updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+
+CREATE INDEX IF NOT EXISTS idx_data_process_results_task_status
+    ON data_process_results(task_id, status, id);
+CREATE INDEX IF NOT EXISTS idx_data_process_results_task_split
+    ON data_process_results(task_id, split);
+
+-- ---- 数据集版本 / 记录 ----
+
+CREATE TABLE IF NOT EXISTS dataset_file_versions (
+    id TEXT PRIMARY KEY,
+    dataset_file_id TEXT NOT NULL REFERENCES dataset_files(id) ON DELETE CASCADE,
+    version_no INTEGER NOT NULL CHECK (version_no > 0),
+    storage_object_id TEXT NOT NULL,
+    content_preview TEXT,
+    description TEXT,
+    base_version_id TEXT REFERENCES dataset_file_versions(id) ON DELETE SET NULL,
+    size_bytes BIGINT NOT NULL DEFAULT 0 CHECK (size_bytes >= 0),
+    record_count BIGINT NOT NULL DEFAULT 0 CHECK (record_count >= 0),
+    checksum_sha256 CHAR(64) NOT NULL,
+    source_task_id TEXT REFERENCES data_process_tasks(id) ON DELETE SET NULL,
+    metadata TEXT NOT NULL DEFAULT '{}',
+    created_by TEXT,
+    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+
+ALTER TABLE dataset_file_versions ADD COLUMN IF NOT EXISTS source_task_id TEXT;
+ALTER TABLE dataset_file_versions ADD COLUMN IF NOT EXISTS metadata TEXT NOT NULL DEFAULT '{}';
+CREATE UNIQUE INDEX IF NOT EXISTS uq_dataset_file_versions_no_002
+    ON dataset_file_versions(dataset_file_id, version_no);
+CREATE INDEX IF NOT EXISTS idx_dataset_file_versions_source_task_002
+    ON dataset_file_versions(source_task_id) WHERE source_task_id IS NOT NULL;
+
+CREATE TABLE IF NOT EXISTS dataset_records (
+    id TEXT PRIMARY KEY,
+    dataset_id TEXT NOT NULL REFERENCES datasets(id) ON DELETE CASCADE,
+    dataset_file_id TEXT REFERENCES dataset_files(id) ON DELETE CASCADE,
+    version_id TEXT REFERENCES dataset_file_versions(id) ON DELETE CASCADE,
+    line_no INTEGER,
+    split VARCHAR(20) CHECK (split IS NULL OR split IN ('train', 'validation', 'test')),
+    instruction TEXT,
+    input TEXT,
+    output TEXT,
+    raw TEXT NOT NULL DEFAULT '{}',
+    status VARCHAR(20) NOT NULL DEFAULT 'valid'
+        CHECK (status IN ('valid', 'modified', 'invalid')),
+    source_task_id TEXT REFERENCES data_process_tasks(id) ON DELETE SET NULL,
+    source_result_id TEXT REFERENCES data_process_results(id) ON DELETE SET NULL,
+    preview_item_id TEXT REFERENCES data_process_preview_items(id) ON DELETE SET NULL,
+    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+
+ALTER TABLE dataset_records ADD COLUMN IF NOT EXISTS source_task_id TEXT;
+ALTER TABLE dataset_records ADD COLUMN IF NOT EXISTS source_result_id TEXT;
+ALTER TABLE dataset_records ADD COLUMN IF NOT EXISTS preview_item_id TEXT;
+CREATE INDEX IF NOT EXISTS idx_dataset_records_dataset_002
+    ON dataset_records(dataset_id, id);
+CREATE INDEX IF NOT EXISTS idx_dataset_records_source_task_002
+    ON dataset_records(source_task_id, source_result_id);
+CREATE INDEX IF NOT EXISTS idx_datasets_source_task_002
+    ON datasets(source_task_id) WHERE source_task_id IS NOT NULL;
+CREATE INDEX IF NOT EXISTS idx_dataset_files_source_task_002
+    ON dataset_files(source_task_id) WHERE source_task_id IS NOT NULL;
+
+-- ============================================================================
+-- 六、数据转换任务（data_convert_tasks）
+--   运行时 router（app/modules/data_convert/router.py）引用但原脚本缺失，本文件补齐。
+-- ============================================================================
+
+CREATE TABLE IF NOT EXISTS data_convert_tasks (
+    id TEXT PRIMARY KEY,
+    name TEXT NOT NULL,
+    description TEXT,
+    output_filename TEXT DEFAULT 'converted-data.jsonl',
+    status TEXT NOT NULL DEFAULT 'pending',
+    input_count INTEGER NOT NULL DEFAULT 0,
+    output_count INTEGER NOT NULL DEFAULT 0,
+    error_message TEXT,
+    create_time TEXT NOT NULL DEFAULT (to_char(now(), 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"')),
+    update_time TEXT NOT NULL DEFAULT (to_char(now(), 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"')),
+    deleted_at TIMESTAMPTZ
+);
+CREATE INDEX IF NOT EXISTS idx_data_convert_tasks_status ON data_convert_tasks(status);
+CREATE INDEX IF NOT EXISTS idx_data_convert_tasks_create_time ON data_convert_tasks(create_time DESC);
+
+-- ============================================================================
+-- 七、种子数据：初始管理员 / 操作员
+--   应用首次启动（ensure_seed_data）也会自动创建；此处提供以便脱离应用直接初始化。
+--   密码：admin / admin123，operator / operator123（上线前请改密）。
+-- ============================================================================
+
+INSERT INTO users
+    (id, username, password_hash, display_name, role, status, permissions, create_time, protected)
+VALUES
+    (
+        'u_admin', 'admin', 'pbkdf2_sha256$390000$ygft_init_salt_admin$2b6f31f22968c4f5a30bcf0acf066b7a0f58d4773d15c5ab898ba715ea87b5bd',
+        'Platform Admin', 'admin', 'active',
+        '["dashboard","fine-tune","model-eval","model-inference","model-manage","dataset","data-process","data-convert","compute","hardware","logs","user-settings"]',
+        to_char(now(), 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"'), 1
+    ),
+    (
+        'u_operator', 'operator', 'pbkdf2_sha256$390000$ygft_init_salt_op$525bf35d02ed26f37952cbd6862b0ae358b9d1a7fa0cbbf0217aa2b5dd544125',
+        'Platform Operator', 'operator', 'active',
+        '["dashboard","fine-tune","model-eval","model-inference","model-manage","dataset","data-process","data-convert","compute","hardware","logs"]',
+        to_char(now(), 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"'), 0
+    )
+ON CONFLICT (username) DO NOTHING;
+
+COMMIT;

backend/app/db/sql/001_platform_runtime.sql

@@ -33,7 +33,49 @@ CREATE TABLE IF NOT EXISTS trained_models (
     create_time TEXT NOT NULL,
     merged INTEGER NOT NULL DEFAULT 0,
     merging INTEGER NOT NULL DEFAULT 0,
-    merged_path TEXT
+    merged_path TEXT,
+    artifact_dir TEXT,
+    compute_node_id TEXT,
+    compute_node_name TEXT
+);
+
+CREATE TABLE IF NOT EXISTS model_lineage (
+    id TEXT PRIMARY KEY,
+    child_resource_type TEXT NOT NULL,
+    child_resource_id TEXT NOT NULL,
+    parent_resource_type TEXT NOT NULL,
+    parent_resource_id TEXT NOT NULL,
+    relation_type TEXT NOT NULL,
+    compute_job_id TEXT,
+    payload TEXT NOT NULL,
+    create_time TEXT NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS model_artifacts (
+    id TEXT PRIMARY KEY,
+    model_id TEXT NOT NULL,
+    model_kind TEXT NOT NULL,
+    artifact_type TEXT NOT NULL,
+    path TEXT NOT NULL,
+    size_bytes BIGINT NOT NULL DEFAULT 0,
+    checksum_sha256 TEXT,
+    metadata TEXT NOT NULL,
+    compute_job_id TEXT,
+    create_time TEXT NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS model_export_jobs (
+    id TEXT PRIMARY KEY,
+    trained_model_id TEXT,
+    compute_job_id TEXT NOT NULL,
+    node_id TEXT,
+    export_type TEXT NOT NULL,
+    quantization_bit INTEGER NOT NULL DEFAULT 0,
+    status TEXT NOT NULL,
+    output_dir TEXT,
+    payload TEXT NOT NULL,
+    create_time TEXT NOT NULL,
+    completed_at TEXT
 );
 
 CREATE TABLE IF NOT EXISTS datasets (
@@ -76,6 +118,9 @@ CREATE TABLE IF NOT EXISTS compute_nodes (
     data_root TEXT NOT NULL,
     model_root TEXT NOT NULL,
     log_root TEXT NOT NULL,
+    api_version TEXT NOT NULL DEFAULT 'v1',
+    capabilities TEXT NOT NULL DEFAULT '[]',
+    description TEXT,
     last_health_check_at TEXT,
     health_detail TEXT NOT NULL
 );
@@ -88,7 +133,8 @@ CREATE TABLE IF NOT EXISTS gpus (
     name TEXT NOT NULL,
     memory_total_gb DOUBLE PRECISION NOT NULL,
     power_limit_w DOUBLE PRECISION NOT NULL,
-    base_temperature INTEGER NOT NULL
+    base_temperature INTEGER NOT NULL,
+    last_seen_at TEXT
 );
 
 CREATE TABLE IF NOT EXISTS fine_tune_tasks (
@@ -103,7 +149,64 @@ CREATE TABLE IF NOT EXISTS fine_tune_tasks (
     completed_at TEXT,
     compute_node_id TEXT REFERENCES compute_nodes(id) ON DELETE SET NULL,
     gpus TEXT NOT NULL,
-    sync_job_id TEXT
+    sync_job_id TEXT,
+    compute_job_id TEXT
+);
+
+CREATE TABLE IF NOT EXISTS fine_tune_metrics (
+    id TEXT PRIMARY KEY,
+    task_id TEXT NOT NULL REFERENCES fine_tune_tasks(id) ON DELETE CASCADE,
+    step INTEGER NOT NULL,
+    epoch DOUBLE PRECISION,
+    loss DOUBLE PRECISION,
+    grad_norm DOUBLE PRECISION,
+    learning_rate DOUBLE PRECISION,
+    raw TEXT NOT NULL,
+    create_time TEXT NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS fine_tune_checkpoints (
+    id TEXT PRIMARY KEY,
+    task_id TEXT NOT NULL REFERENCES fine_tune_tasks(id) ON DELETE CASCADE,
+    step INTEGER NOT NULL,
+    name TEXT NOT NULL,
+    path TEXT NOT NULL,
+    size_bytes BIGINT NOT NULL DEFAULT 0,
+    create_time TEXT NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS compute_jobs (
+    id TEXT PRIMARY KEY,
+    task_id TEXT REFERENCES fine_tune_tasks(id) ON DELETE SET NULL,
+    node_id TEXT REFERENCES compute_nodes(id) ON DELETE SET NULL,
+    engine TEXT NOT NULL,
+    status TEXT NOT NULL,
+    command TEXT NOT NULL,
+    output_dir TEXT,
+    log_file TEXT,
+    payload TEXT NOT NULL,
+    create_time TEXT NOT NULL,
+    update_time TEXT NOT NULL,
+    completed_at TEXT
+);
+
+CREATE TABLE IF NOT EXISTS gpu_allocations (
+    id TEXT PRIMARY KEY,
+    task_id TEXT REFERENCES fine_tune_tasks(id) ON DELETE CASCADE,
+    compute_job_id TEXT,
+    node_id TEXT REFERENCES compute_nodes(id) ON DELETE CASCADE,
+    gpu_index INTEGER NOT NULL,
+    status TEXT NOT NULL,
+    create_time TEXT NOT NULL,
+    released_at TEXT
+);
+
+CREATE TABLE IF NOT EXISTS scheduler_locks (
+    lock_key TEXT PRIMARY KEY,
+    owner TEXT NOT NULL,
+    expires_at TEXT NOT NULL,
+    create_time TEXT NOT NULL,
+    update_time TEXT NOT NULL
 );
 
 CREATE TABLE IF NOT EXISTS resource_replicas (
@@ -114,6 +217,10 @@ CREATE TABLE IF NOT EXISTS resource_replicas (
     local_path TEXT NOT NULL,
     status TEXT NOT NULL,
     sync_status TEXT NOT NULL,
+    checksum_sha256 TEXT,
+    byte_size BIGINT NOT NULL DEFAULT 0,
+    last_checked_at TEXT,
+    last_error TEXT,
     create_time TEXT NOT NULL
 );
 
@@ -127,7 +234,102 @@ CREATE TABLE IF NOT EXISTS resource_sync_jobs (
     completed_at TEXT
 );
 
+CREATE TABLE IF NOT EXISTS eval_tasks (
+    id TEXT PRIMARY KEY,
+    name TEXT NOT NULL,
+    payload TEXT NOT NULL,
+    status TEXT NOT NULL,
+    create_time TEXT NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS eval_dimensions (
+    id TEXT PRIMARY KEY,
+    name TEXT NOT NULL,
+    payload TEXT NOT NULL,
+    is_active INTEGER NOT NULL DEFAULT 1,
+    is_default INTEGER NOT NULL DEFAULT 0,
+    create_time TEXT NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS compare_tasks (
+    id TEXT PRIMARY KEY,
+    name TEXT NOT NULL,
+    payload TEXT NOT NULL,
+    status TEXT NOT NULL,
+    create_time TEXT NOT NULL
+);
+
 CREATE INDEX IF NOT EXISTS idx_fine_tune_status ON fine_tune_tasks(status);
+CREATE INDEX IF NOT EXISTS idx_fine_tune_compute_job ON fine_tune_tasks(compute_job_id);
+CREATE INDEX IF NOT EXISTS idx_fine_tune_compute_node_status ON fine_tune_tasks(compute_node_id, status);
+CREATE INDEX IF NOT EXISTS idx_model_lineage_child ON model_lineage(child_resource_type, child_resource_id);
+CREATE INDEX IF NOT EXISTS idx_model_lineage_parent ON model_lineage(parent_resource_type, parent_resource_id);
+CREATE INDEX IF NOT EXISTS idx_model_artifacts_model ON model_artifacts(model_kind, model_id, artifact_type);
+CREATE INDEX IF NOT EXISTS idx_model_export_jobs_model ON model_export_jobs(trained_model_id, create_time DESC);
+CREATE INDEX IF NOT EXISTS idx_model_export_jobs_compute ON model_export_jobs(compute_job_id);
+CREATE INDEX IF NOT EXISTS idx_fine_tune_metrics_task_step ON fine_tune_metrics(task_id, step);
+CREATE UNIQUE INDEX IF NOT EXISTS uq_fine_tune_metrics_task_step_epoch ON fine_tune_metrics(task_id, step, epoch);
+CREATE INDEX IF NOT EXISTS idx_fine_tune_checkpoints_task_step ON fine_tune_checkpoints(task_id, step);
+CREATE UNIQUE INDEX IF NOT EXISTS uq_fine_tune_checkpoints_task_path ON fine_tune_checkpoints(task_id, path);
+CREATE INDEX IF NOT EXISTS idx_compute_jobs_task ON compute_jobs(task_id);
+CREATE INDEX IF NOT EXISTS idx_compute_jobs_node_status ON compute_jobs(node_id, status);
+CREATE INDEX IF NOT EXISTS idx_gpu_allocations_node_status ON gpu_allocations(node_id, status);
+CREATE UNIQUE INDEX IF NOT EXISTS uq_gpu_allocations_active ON gpu_allocations(node_id, gpu_index) WHERE status IN ('allocated','running');
+CREATE INDEX IF NOT EXISTS idx_scheduler_locks_expires ON scheduler_locks(expires_at);
 CREATE INDEX IF NOT EXISTS idx_dataset_files_dataset ON dataset_files(dataset_id);
 CREATE INDEX IF NOT EXISTS idx_gpus_node ON gpus(node_id);
+CREATE UNIQUE INDEX IF NOT EXISTS uq_gpus_node_index ON gpus(node_id, gpu_index);
 CREATE INDEX IF NOT EXISTS idx_replicas_resource ON resource_replicas(resource_type, resource_id);
+CREATE UNIQUE INDEX IF NOT EXISTS uq_replicas_node_resource ON resource_replicas(node_id, resource_type, resource_id);
+CREATE INDEX IF NOT EXISTS idx_sync_jobs_node_status ON resource_sync_jobs(target_node_id, status);
+CREATE INDEX IF NOT EXISTS idx_eval_tasks_status ON eval_tasks(status);
+CREATE INDEX IF NOT EXISTS idx_eval_dimensions_active ON eval_dimensions(is_active);
+CREATE INDEX IF NOT EXISTS idx_compare_tasks_status ON compare_tasks(status);
+
+-- ===================== Project / Tenant =====================
+
+CREATE TABLE IF NOT EXISTS projects (
+    id TEXT PRIMARY KEY,
+    tenant_id TEXT NOT NULL DEFAULT 'default',
+    name TEXT NOT NULL,
+    code TEXT NOT NULL,
+    description TEXT,
+    quota TEXT,
+    status TEXT NOT NULL DEFAULT 'active',
+    create_time TEXT NOT NULL,
+    create_by TEXT,
+    updated_at TEXT
+);
+
+CREATE TABLE IF NOT EXISTS project_members (
+    project_id TEXT NOT NULL REFERENCES projects(id) ON DELETE CASCADE,
+    user_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
+    role TEXT NOT NULL DEFAULT 'member',
+    create_time TEXT NOT NULL,
+    PRIMARY KEY (project_id, user_id)
+);
+
+CREATE TABLE IF NOT EXISTS roles (
+    id TEXT PRIMARY KEY,
+    name TEXT NOT NULL,
+    permissions TEXT NOT NULL DEFAULT '[]',
+    create_time TEXT
+);
+
+CREATE TABLE IF NOT EXISTS sessions (
+    id TEXT PRIMARY KEY,
+    user_id TEXT NOT NULL,
+    issued_at TEXT NOT NULL,
+    expires_at TEXT NOT NULL,
+    ip TEXT
+);
+
+CREATE TABLE IF NOT EXISTS acls (
+    id TEXT PRIMARY KEY,
+    resource_type TEXT NOT NULL,
+    resource_id TEXT NOT NULL,
+    principal_type TEXT NOT NULL,
+    principal_id TEXT NOT NULL,
+    permission TEXT NOT NULL,
+    create_time TEXT
+);

backend/app/db/sql/002_data_process.sql

@@ -0,0 +1,352 @@
+-- Data processing migration.
+--
+-- IMPORTANT: This file is intentionally NOT wired into application startup.
+-- Apply it explicitly in a controlled deployment, or call
+-- DataProcessStore.ensure_schema() from an administrative command.
+
+BEGIN;
+
+-- This migration targets the current runtime schema created by
+-- 001_platform_runtime.sql. Refuse the UUID/JSONB target-design schema instead
+-- of partially altering it with incompatible TEXT foreign keys.
+DO $$
+DECLARE
+    datasets_id_type TEXT;
+BEGIN
+    SELECT format_type(a.atttypid, a.atttypmod)
+      INTO datasets_id_type
+      FROM pg_attribute a
+      JOIN pg_class c ON c.oid = a.attrelid
+      JOIN pg_namespace n ON n.oid = c.relnamespace
+     WHERE n.nspname = current_schema()
+       AND c.relname = 'datasets'
+       AND a.attname = 'id'
+       AND a.attnum > 0
+       AND NOT a.attisdropped;
+    IF datasets_id_type IS NULL THEN
+        RAISE EXCEPTION '002_data_process.sql requires 001_platform_runtime.sql first';
+    END IF;
+    IF datasets_id_type <> 'text' THEN
+        RAISE EXCEPTION
+          '002_data_process.sql supports only the current TEXT runtime schema; found datasets.id type %',
+          datasets_id_type;
+    END IF;
+END $$;
+
+ALTER TABLE datasets ADD COLUMN IF NOT EXISTS source_task_id TEXT;
+ALTER TABLE datasets ADD COLUMN IF NOT EXISTS size_bytes BIGINT NOT NULL DEFAULT 0;
+ALTER TABLE datasets ADD COLUMN IF NOT EXISTS record_count BIGINT NOT NULL DEFAULT 0;
+ALTER TABLE datasets ADD COLUMN IF NOT EXISTS metadata TEXT NOT NULL DEFAULT '{}';
+ALTER TABLE datasets ADD COLUMN IF NOT EXISTS tenant_id TEXT;
+ALTER TABLE datasets ADD COLUMN IF NOT EXISTS project_id TEXT;
+ALTER TABLE datasets ADD COLUMN IF NOT EXISTS owner_id TEXT;
+ALTER TABLE datasets ADD COLUMN IF NOT EXISTS created_by TEXT;
+ALTER TABLE datasets ADD COLUMN IF NOT EXISTS created_at TIMESTAMPTZ NOT NULL DEFAULT now();
+ALTER TABLE datasets ADD COLUMN IF NOT EXISTS updated_at TIMESTAMPTZ NOT NULL DEFAULT now();
+ALTER TABLE datasets ADD COLUMN IF NOT EXISTS deleted_at TIMESTAMPTZ;
+
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS storage_object_id TEXT;
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS current_version_id TEXT;
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS size_bytes BIGINT NOT NULL DEFAULT 0;
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS record_count BIGINT NOT NULL DEFAULT 0;
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS file_format VARCHAR(40);
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS checksum_sha256 CHAR(64);
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS version_no INTEGER NOT NULL DEFAULT 1;
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS source_task_id TEXT;
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS tenant_id TEXT;
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS project_id TEXT;
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS created_by TEXT;
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS metadata TEXT NOT NULL DEFAULT '{}';
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS created_at TIMESTAMPTZ NOT NULL DEFAULT now();
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS updated_at TIMESTAMPTZ NOT NULL DEFAULT now();
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS deleted_at TIMESTAMPTZ;
+
+CREATE TABLE IF NOT EXISTS data_process_tasks (
+    id TEXT PRIMARY KEY,
+    name VARCHAR(150) NOT NULL,
+    description TEXT,
+    status VARCHAR(20) NOT NULL DEFAULT 'pending'
+        CHECK (status IN ('pending', 'running', 'completed', 'failed', 'stopped')),
+    process_type VARCHAR(20) NOT NULL
+        CHECK (process_type IN ('structured', 'unstructured', 'external')),
+    source_dataset_id TEXT REFERENCES datasets(id) ON DELETE SET NULL,
+    output_dataset_id TEXT REFERENCES datasets(id) ON DELETE SET NULL,
+    config TEXT NOT NULL DEFAULT '{}',
+    progress NUMERIC(5,2) NOT NULL DEFAULT 0 CHECK (progress >= 0 AND progress <= 100),
+    input_count BIGINT NOT NULL DEFAULT 0 CHECK (input_count >= 0),
+    output_count BIGINT NOT NULL DEFAULT 0 CHECK (output_count >= 0),
+    filtered_count BIGINT NOT NULL DEFAULT 0 CHECK (filtered_count >= 0),
+    duplicate_count BIGINT NOT NULL DEFAULT 0 CHECK (duplicate_count >= 0),
+    error_count BIGINT NOT NULL DEFAULT 0 CHECK (error_count >= 0),
+    failure_reason TEXT,
+    generation_run_id TEXT,
+    results_confirmed BOOLEAN NOT NULL DEFAULT TRUE,
+    workflow_step VARCHAR(20) NOT NULL DEFAULT 'create'
+        CHECK (workflow_step IN ('create', 'model', 'upload', 'preview', 'generate', 'results')),
+    preview_status VARCHAR(20) NOT NULL DEFAULT 'idle'
+        CHECK (preview_status IN ('idle', 'queued', 'running', 'completed', 'failed', 'cancelled')),
+    preview_progress NUMERIC(5,2) NOT NULL DEFAULT 0
+        CHECK (preview_progress >= 0 AND preview_progress <= 100),
+    preview_run_id TEXT,
+    preview_failure_reason TEXT,
+    preview_total_files INTEGER NOT NULL DEFAULT 0 CHECK (preview_total_files >= 0),
+    preview_completed_files INTEGER NOT NULL DEFAULT 0 CHECK (preview_completed_files >= 0),
+    tenant_id TEXT,
+    project_id TEXT,
+    owner_id TEXT,
+    approval_status VARCHAR(30) NOT NULL DEFAULT 'not_required',
+    created_by TEXT,
+    updated_by TEXT,
+    deleted_by TEXT,
+    started_at TIMESTAMPTZ,
+    completed_at TIMESTAMPTZ,
+    created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+    updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+    deleted_at TIMESTAMPTZ
+);
+
+ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS generation_run_id TEXT;
+-- 历史任务在引入六步确认流程前已经完成审核，默认保留为已确认；
+-- 新任务由创建接口显式写入 FALSE，并在第六步确认后转为 TRUE。
+ALTER TABLE data_process_tasks
+    ADD COLUMN IF NOT EXISTS results_confirmed BOOLEAN NOT NULL DEFAULT TRUE;
+UPDATE data_process_tasks
+SET results_confirmed=FALSE
+WHERE status <> 'completed' AND results_confirmed=TRUE;
+
+-- 先以可空列接入旧库，才能只回填历史行；随后再收紧默认值与约束。
+ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS workflow_step VARCHAR(20);
+ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_status VARCHAR(20);
+ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_progress NUMERIC(5,2);
+ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_run_id TEXT;
+ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_failure_reason TEXT;
+ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_total_files INTEGER;
+ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_completed_files INTEGER;
+
+CREATE TEMP TABLE data_process_workflow_backfill_ids ON COMMIT DROP AS
+SELECT id FROM data_process_tasks WHERE workflow_step IS NULL;
+
+UPDATE data_process_tasks task
+SET workflow_step = CASE
+    WHEN task.status IN ('running', 'failed', 'stopped') THEN 'generate'
+    WHEN task.status = 'completed' AND task.results_confirmed=FALSE THEN 'generate'
+    WHEN task.status = 'completed' THEN 'results'
+    ELSE 'create'
+END
+WHERE task.workflow_step IS NULL;
+UPDATE data_process_tasks
+SET preview_status='idle', preview_progress=0,
+    preview_total_files=0, preview_completed_files=0
+WHERE preview_status IS NULL OR preview_progress IS NULL
+   OR preview_total_files IS NULL OR preview_completed_files IS NULL;
+
+ALTER TABLE data_process_tasks ALTER COLUMN workflow_step SET DEFAULT 'create';
+ALTER TABLE data_process_tasks ALTER COLUMN workflow_step SET NOT NULL;
+ALTER TABLE data_process_tasks ALTER COLUMN preview_status SET DEFAULT 'idle';
+ALTER TABLE data_process_tasks ALTER COLUMN preview_status SET NOT NULL;
+ALTER TABLE data_process_tasks ALTER COLUMN preview_progress SET DEFAULT 0;
+ALTER TABLE data_process_tasks ALTER COLUMN preview_progress SET NOT NULL;
+ALTER TABLE data_process_tasks ALTER COLUMN preview_total_files SET DEFAULT 0;
+ALTER TABLE data_process_tasks ALTER COLUMN preview_total_files SET NOT NULL;
+ALTER TABLE data_process_tasks ALTER COLUMN preview_completed_files SET DEFAULT 0;
+ALTER TABLE data_process_tasks ALTER COLUMN preview_completed_files SET NOT NULL;
+
+DO $$
+BEGIN
+    IF NOT EXISTS (
+        SELECT 1 FROM pg_constraint
+        WHERE conrelid='data_process_tasks'::regclass
+          AND conname='ck_data_process_tasks_workflow_step'
+    ) THEN
+        ALTER TABLE data_process_tasks ADD CONSTRAINT ck_data_process_tasks_workflow_step
+            CHECK (workflow_step IN ('create', 'model', 'upload', 'preview', 'generate', 'results'));
+    END IF;
+    IF NOT EXISTS (
+        SELECT 1 FROM pg_constraint
+        WHERE conrelid='data_process_tasks'::regclass
+          AND conname='ck_data_process_tasks_preview_status'
+    ) THEN
+        ALTER TABLE data_process_tasks ADD CONSTRAINT ck_data_process_tasks_preview_status
+            CHECK (preview_status IN ('idle', 'queued', 'running', 'completed', 'failed', 'cancelled'));
+    END IF;
+    IF NOT EXISTS (
+        SELECT 1 FROM pg_constraint
+        WHERE conrelid='data_process_tasks'::regclass
+          AND conname='ck_data_process_tasks_preview_progress'
+    ) THEN
+        ALTER TABLE data_process_tasks ADD CONSTRAINT ck_data_process_tasks_preview_progress
+            CHECK (preview_progress >= 0 AND preview_progress <= 100);
+    END IF;
+    IF NOT EXISTS (
+        SELECT 1 FROM pg_constraint
+        WHERE conrelid='data_process_tasks'::regclass
+          AND conname='ck_data_process_tasks_preview_file_counts'
+    ) THEN
+        ALTER TABLE data_process_tasks ADD CONSTRAINT ck_data_process_tasks_preview_file_counts
+            CHECK (preview_total_files >= 0 AND preview_completed_files >= 0
+                   AND preview_completed_files <= preview_total_files);
+    END IF;
+END $$;
+
+CREATE UNIQUE INDEX IF NOT EXISTS uq_data_process_tasks_name_alive
+    ON data_process_tasks(name) WHERE deleted_at IS NULL;
+CREATE INDEX IF NOT EXISTS idx_data_process_tasks_scope_status
+    ON data_process_tasks(tenant_id, project_id, status, created_at DESC)
+    WHERE deleted_at IS NULL;
+CREATE INDEX IF NOT EXISTS idx_data_process_tasks_creator_created
+    ON data_process_tasks(created_by, created_at DESC) WHERE deleted_at IS NULL;
+
+CREATE TABLE IF NOT EXISTS data_process_source_files (
+    id TEXT PRIMARY KEY,
+    task_id TEXT NOT NULL REFERENCES data_process_tasks(id) ON DELETE CASCADE,
+    storage_object_id TEXT,
+    name TEXT NOT NULL,
+    size_bytes BIGINT NOT NULL DEFAULT 0 CHECK (size_bytes >= 0),
+    record_count BIGINT NOT NULL DEFAULT 0 CHECK (record_count >= 0),
+    file_format VARCHAR(40),
+    checksum_sha256 CHAR(64) NOT NULL,
+    version_no INTEGER NOT NULL DEFAULT 1 CHECK (version_no > 0),
+    content TEXT NOT NULL,
+    content_preview TEXT,
+    metadata TEXT NOT NULL DEFAULT '{}',
+    tenant_id TEXT,
+    project_id TEXT,
+    created_by TEXT,
+    created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+    updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+    deleted_at TIMESTAMPTZ
+);
+
+CREATE INDEX IF NOT EXISTS idx_data_process_source_files_task
+    ON data_process_source_files(task_id, created_at) WHERE deleted_at IS NULL;
+CREATE UNIQUE INDEX IF NOT EXISTS uq_data_process_source_checksum_alive
+    ON data_process_source_files(task_id, checksum_sha256) WHERE deleted_at IS NULL;
+
+CREATE TABLE IF NOT EXISTS data_process_preview_items (
+    id TEXT PRIMARY KEY,
+    task_id TEXT NOT NULL REFERENCES data_process_tasks(id) ON DELETE CASCADE,
+    source_file_id TEXT REFERENCES data_process_source_files(id) ON DELETE CASCADE,
+    original_content TEXT NOT NULL DEFAULT '',
+    edited_content TEXT NOT NULL DEFAULT '',
+    source_start INTEGER CHECK (source_start IS NULL OR source_start >= 0),
+    source_end INTEGER CHECK (source_end IS NULL OR source_end >= 0),
+    source_start_line INTEGER CHECK (source_start_line IS NULL OR source_start_line > 0),
+    source_end_line INTEGER CHECK (source_end_line IS NULL OR source_end_line > 0),
+    token_count INTEGER NOT NULL DEFAULT 0 CHECK (token_count >= 0),
+    status VARCHAR(20) NOT NULL DEFAULT 'original'
+        CHECK (status IN ('original', 'modified', 'manual', 'invalid')),
+    quality_score TEXT NOT NULL DEFAULT '{}',
+    created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+    updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+    CHECK (source_start IS NULL OR source_end IS NULL OR source_end >= source_start),
+    CHECK (source_start_line IS NULL OR source_end_line IS NULL OR source_end_line >= source_start_line)
+);
+
+CREATE INDEX IF NOT EXISTS idx_data_process_preview_task_file
+    ON data_process_preview_items(task_id, source_file_id, created_at);
+
+-- 子表在新库中到这里才存在；只修复本次新增 workflow_step 前的历史任务。
+UPDATE data_process_tasks task
+SET workflow_step = CASE
+    WHEN EXISTS (
+        SELECT 1 FROM data_process_preview_items preview
+        WHERE preview.task_id=task.id
+    ) THEN 'preview'
+    WHEN EXISTS (
+        SELECT 1 FROM data_process_source_files source_file
+        WHERE source_file.task_id=task.id AND source_file.deleted_at IS NULL
+    ) THEN 'upload'
+    ELSE task.workflow_step
+END
+WHERE task.id IN (SELECT id FROM data_process_workflow_backfill_ids)
+  AND task.status='pending';
+
+CREATE TABLE IF NOT EXISTS data_process_results (
+    id TEXT PRIMARY KEY,
+    task_id TEXT NOT NULL REFERENCES data_process_tasks(id) ON DELETE CASCADE,
+    preview_item_id TEXT REFERENCES data_process_preview_items(id) ON DELETE SET NULL,
+    instruction TEXT NOT NULL,
+    input TEXT NOT NULL DEFAULT '',
+    output TEXT NOT NULL,
+    chosen TEXT NOT NULL DEFAULT '',
+    rejected TEXT NOT NULL DEFAULT '',
+    original_instruction TEXT,
+    original_input TEXT,
+    original_output TEXT,
+    original_chosen TEXT,
+    original_rejected TEXT,
+    status VARCHAR(20) NOT NULL DEFAULT 'valid'
+        CHECK (status IN ('valid', 'modified', 'invalid')),
+    error TEXT,
+    split VARCHAR(20) CHECK (split IS NULL OR split IN ('train', 'validation', 'test')),
+    quality_score TEXT NOT NULL DEFAULT '{}',
+    created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+    updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+
+ALTER TABLE data_process_results ADD COLUMN IF NOT EXISTS chosen TEXT NOT NULL DEFAULT '';
+ALTER TABLE data_process_results ADD COLUMN IF NOT EXISTS rejected TEXT NOT NULL DEFAULT '';
+ALTER TABLE data_process_results ADD COLUMN IF NOT EXISTS original_chosen TEXT;
+ALTER TABLE data_process_results ADD COLUMN IF NOT EXISTS original_rejected TEXT;
+
+CREATE INDEX IF NOT EXISTS idx_data_process_results_task_status
+    ON data_process_results(task_id, status, id);
+CREATE INDEX IF NOT EXISTS idx_data_process_results_task_split
+    ON data_process_results(task_id, split);
+
+CREATE TABLE IF NOT EXISTS dataset_file_versions (
+    id TEXT PRIMARY KEY,
+    dataset_file_id TEXT NOT NULL REFERENCES dataset_files(id) ON DELETE CASCADE,
+    version_no INTEGER NOT NULL CHECK (version_no > 0),
+    storage_object_id TEXT NOT NULL,
+    content_preview TEXT,
+    description TEXT,
+    base_version_id TEXT REFERENCES dataset_file_versions(id) ON DELETE SET NULL,
+    size_bytes BIGINT NOT NULL DEFAULT 0 CHECK (size_bytes >= 0),
+    record_count BIGINT NOT NULL DEFAULT 0 CHECK (record_count >= 0),
+    checksum_sha256 CHAR(64) NOT NULL,
+    source_task_id TEXT REFERENCES data_process_tasks(id) ON DELETE SET NULL,
+    metadata TEXT NOT NULL DEFAULT '{}',
+    created_by TEXT,
+    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+
+ALTER TABLE dataset_file_versions ADD COLUMN IF NOT EXISTS source_task_id TEXT;
+ALTER TABLE dataset_file_versions ADD COLUMN IF NOT EXISTS metadata TEXT NOT NULL DEFAULT '{}';
+CREATE UNIQUE INDEX IF NOT EXISTS uq_dataset_file_versions_no_002
+    ON dataset_file_versions(dataset_file_id, version_no);
+CREATE INDEX IF NOT EXISTS idx_dataset_file_versions_source_task_002
+    ON dataset_file_versions(source_task_id) WHERE source_task_id IS NOT NULL;
+
+CREATE TABLE IF NOT EXISTS dataset_records (
+    id TEXT PRIMARY KEY,
+    dataset_id TEXT NOT NULL REFERENCES datasets(id) ON DELETE CASCADE,
+    dataset_file_id TEXT REFERENCES dataset_files(id) ON DELETE CASCADE,
+    version_id TEXT REFERENCES dataset_file_versions(id) ON DELETE CASCADE,
+    line_no INTEGER,
+    split VARCHAR(20) CHECK (split IS NULL OR split IN ('train', 'validation', 'test')),
+    instruction TEXT,
+    input TEXT,
+    output TEXT,
+    raw TEXT NOT NULL DEFAULT '{}',
+    status VARCHAR(20) NOT NULL DEFAULT 'valid'
+        CHECK (status IN ('valid', 'modified', 'invalid')),
+    source_task_id TEXT REFERENCES data_process_tasks(id) ON DELETE SET NULL,
+    source_result_id TEXT REFERENCES data_process_results(id) ON DELETE SET NULL,
+    preview_item_id TEXT REFERENCES data_process_preview_items(id) ON DELETE SET NULL,
+    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+
+ALTER TABLE dataset_records ADD COLUMN IF NOT EXISTS source_task_id TEXT;
+ALTER TABLE dataset_records ADD COLUMN IF NOT EXISTS source_result_id TEXT;
+ALTER TABLE dataset_records ADD COLUMN IF NOT EXISTS preview_item_id TEXT;
+CREATE INDEX IF NOT EXISTS idx_dataset_records_dataset_002
+    ON dataset_records(dataset_id, id);
+CREATE INDEX IF NOT EXISTS idx_dataset_records_source_task_002
+    ON dataset_records(source_task_id, source_result_id);
+CREATE INDEX IF NOT EXISTS idx_datasets_source_task_002
+    ON datasets(source_task_id) WHERE source_task_id IS NOT NULL;
+CREATE INDEX IF NOT EXISTS idx_dataset_files_source_task_002
+    ON dataset_files(source_task_id) WHERE source_task_id IS NOT NULL;
+
+COMMIT;

backend/app/db/sql/002_governance.sql

@@ -0,0 +1,68 @@
+-- 平台治理：租户 / 审批 / 审计（字段以 platform_store 实际写入为准）
+CREATE TABLE IF NOT EXISTS tenants (
+    id TEXT PRIMARY KEY,
+    name TEXT NOT NULL,
+    code TEXT,
+    status TEXT DEFAULT 'active',
+    owner_user_id TEXT,
+    quota TEXT,
+    retention_policy_id TEXT,
+    create_time TEXT
+);
+
+CREATE TABLE IF NOT EXISTS approval_templates (
+    id TEXT PRIMARY KEY,
+    name TEXT NOT NULL,
+    steps TEXT,
+    create_time TEXT
+);
+
+CREATE TABLE IF NOT EXISTS approval_instances (
+    id TEXT PRIMARY KEY,
+    template_id TEXT,
+    resource_type TEXT,
+    resource_id TEXT,
+    applicant_id TEXT,
+    status TEXT DEFAULT 'pending',
+    current_step INTEGER DEFAULT 0,
+    create_time TEXT
+);
+
+CREATE TABLE IF NOT EXISTS approval_steps (
+    id TEXT PRIMARY KEY,
+    instance_id TEXT,
+    step_index INTEGER,
+    approver_id TEXT,
+    status TEXT DEFAULT 'pending',
+    comment TEXT,
+    time TEXT
+);
+
+CREATE TABLE IF NOT EXISTS audit_logs (
+    id TEXT PRIMARY KEY,
+    tenant_id TEXT,
+    project_id TEXT,
+    actor_id TEXT,
+    action TEXT,
+    target_type TEXT,
+    target_id TEXT,
+    detail TEXT,
+    client_ip TEXT,
+    time TEXT
+);
+
+CREATE INDEX IF NOT EXISTS idx_audit_tenant ON audit_logs(tenant_id);
+CREATE INDEX IF NOT EXISTS idx_audit_project ON audit_logs(project_id);
+CREATE INDEX IF NOT EXISTS idx_audit_action ON audit_logs(action);
+CREATE INDEX IF NOT EXISTS idx_audit_time ON audit_logs(time);
+
+CREATE TABLE IF NOT EXISTS retention_policies (
+    id TEXT PRIMARY KEY,
+    name TEXT NOT NULL,
+    scope TEXT,
+    rule TEXT,
+    status TEXT DEFAULT 'active',
+    create_time TEXT,
+    create_by TEXT,
+    updated_at TEXT
+);

backend/app/db/sql/003_model_path_governance.sql

@@ -0,0 +1,18 @@
+-- 003_model_path_governance
+-- 模型路径治理：增加 can_train 标识，区分本地可训练模型与 API / 远程模型。
+-- 训练预检阶段依赖该字段拦截不适合 LLaMA-Factory 本地训练的基座模型。
+
+-- 1. models 表增加 can_train（默认 0，后设搬迁为 1 的规则如下）
+ALTER TABLE models ADD COLUMN IF NOT EXISTS can_train INTEGER NOT NULL DEFAULT 0;
+
+-- 2. 将已有模型按规则推定 can_train：
+--    - path 非空 且 model_source != 'api' → 可训练 (1)
+--    - 其余 → 不可训练 (0)
+UPDATE models
+SET can_train = CASE
+    WHEN path IS NOT NULL AND path != '' AND model_source IS NOT NULL AND model_source != 'api' THEN 1
+    ELSE 0
+END;
+
+-- 3. 给 trained_models 增加 artifact_dir（训练产物目录扫描结果目录）
+ALTER TABLE trained_models ADD COLUMN IF NOT EXISTS artifact_dir TEXT;

backend/app/db/sql/003_tenant_quota.sql

@@ -0,0 +1,3 @@
+-- 租户配额与保留策略扩展（如后续治理表需补列，可在此追加）
+ALTER TABLE tenants ADD COLUMN IF NOT EXISTS gpu_quota TEXT;
+ALTER TABLE tenants ADD COLUMN IF NOT EXISTS storage_quota TEXT;

backend/app/db/sql/004_permissions.sql

@@ -0,0 +1,22 @@
+-- ============================================================
+-- 权限体系扩展：GPU 分配表 + 资源所有权字段
+-- ============================================================
+
+-- GPU 分配表：管理员指定哪些用户可以使用哪些 GPU 卡
+CREATE TABLE IF NOT EXISTS gpu_assignments (
+    id          TEXT PRIMARY KEY,
+    node_id     TEXT NOT NULL REFERENCES compute_nodes(id) ON DELETE CASCADE,
+    gpu_index   INTEGER NOT NULL,
+    user_id     TEXT NOT NULL,
+    assigned_by TEXT,
+    assigned_at TEXT NOT NULL,
+    UNIQUE (node_id, gpu_index, user_id)
+);
+CREATE INDEX IF NOT EXISTS idx_gpu_assignments_user ON gpu_assignments(user_id);
+CREATE INDEX IF NOT EXISTS idx_gpu_assignments_gpu ON gpu_assignments(node_id, gpu_index);
+
+-- 资源所有权字段：用户创建的数据集/模型/训练产物/评测任务
+ALTER TABLE datasets ADD COLUMN IF NOT EXISTS created_by TEXT;
+ALTER TABLE models ADD COLUMN IF NOT EXISTS created_by TEXT;
+ALTER TABLE trained_models ADD COLUMN IF NOT EXISTS created_by TEXT;
+ALTER TABLE eval_tasks ADD COLUMN IF NOT EXISTS created_by TEXT;

backend/app/main.py

@@ -1,16 +1,20 @@
+import asyncio
+from contextlib import suppress
+
 from fastapi import FastAPI
 from fastapi.middleware.cors import CORSMiddleware
 
 from app.api.v1.router import api_router
-from app.core.config import get_settings
+from app.core.config import docs_kwargs, get_settings
 from app.core.logging import configure_logging, setup_request_logging
+from app.workers.compute_poller import run_compute_poller
 
 
 def create_app() -> FastAPI:
     settings = get_settings()
     configure_logging(settings)
 
-    app = FastAPI(title=settings.app_name)
+    app = FastAPI(title=settings.app_name, **docs_kwargs(settings.enable_docs))
     app.add_middleware(
         CORSMiddleware,
         allow_origins=settings.cors_allow_origins,
@@ -20,6 +24,19 @@ def create_app() -> FastAPI:
     )
     setup_request_logging(app)
     app.include_router(api_router, prefix=settings.route_prefix)
+
+    @app.on_event("startup")
+    async def start_workers() -> None:
+        app.state.compute_poller_task = asyncio.create_task(run_compute_poller())
+
+    @app.on_event("shutdown")
+    async def stop_workers() -> None:
+        task = getattr(app.state, "compute_poller_task", None)
+        if task:
+            task.cancel()
+            with suppress(asyncio.CancelledError):
+                await task
+
     return app
 
 

backend/app/modules/approval/router.py

@@ -0,0 +1,91 @@
+from __future__ import annotations
+
+from fastapi import APIRouter, Body
+from typing import Any
+
+from app.api.v1.endpoints.platform import ok, fail
+from app.db.platform_store import get_platform_store
+
+router = APIRouter(prefix="/approvals", tags=["approval"])
+
+
+@router.get("/templates")
+def list_templates() -> dict[str, Any]:
+    return ok(get_platform_store().approval_templates())
+
+
+@router.post("/templates")
+def create_template(payload: dict[str, Any] = Body(...)) -> dict[str, Any]:
+    if not payload.get("name"):
+        raise fail(400, "name 必填")
+    return ok(get_platform_store().create_approval_template(payload))
+
+
+@router.get("/templates/{template_id}")
+def get_template(template_id: str) -> dict[str, Any]:
+    try:
+        return ok(get_platform_store().approval_template(template_id))
+    except KeyError:
+        raise fail(404, "template not found")
+
+
+@router.put("/templates/{template_id}")
+def update_template(template_id: str, payload: dict[str, Any] = Body(...)) -> dict[str, Any]:
+    try:
+        return ok(get_platform_store().update_approval_template(template_id, payload))
+    except KeyError:
+        raise fail(404, "template not found")
+
+
+@router.delete("/templates/{template_id}")
+def delete_template(template_id: str) -> dict[str, Any]:
+    try:
+        return ok(get_platform_store().delete_approval_template(template_id))
+    except KeyError:
+        raise fail(404, "template not found")
+
+
+@router.get("")
+def list_instances(status: str | None = None) -> dict[str, Any]:
+    return ok(get_platform_store().approval_instances(status=status))
+
+
+@router.post("")
+def create_instance(payload: dict[str, Any] = Body(...)) -> dict[str, Any]:
+    for field in ("resource_type", "resource_id", "applicant_id"):
+        if not payload.get(field):
+            raise fail(400, f"{field} 必填")
+    try:
+        return ok(get_platform_store().create_approval_instance(payload))
+    except KeyError:
+        raise fail(404, "template not found")
+
+
+@router.get("/{instance_id}")
+def get_instance(instance_id: str) -> dict[str, Any]:
+    try:
+        return ok(get_platform_store().approval_instance(instance_id))
+    except KeyError:
+        raise fail(404, "instance not found")
+
+
+@router.post("/{instance_id}/steps/{step_index}/decision")
+def decide(
+    instance_id: str,
+    step_index: int,
+    payload: dict[str, Any] = Body(...),
+) -> dict[str, Any]:
+    if not payload.get("approver_id"):
+        raise fail(400, "approver_id 必填")
+    try:
+        return ok(
+            get_platform_store().decide_approval_step(
+                instance_id,
+                step_index,
+                approver_id=payload["approver_id"],
+                approved=bool(payload.get("approved", False)),
+                comment=payload.get("comment"),
+            )
+        )
+    except (KeyError, ValueError) as e:
+        raise fail(400, str(e))

backend/app/modules/compute_gateway/client.py

@@ -0,0 +1,247 @@
+from __future__ import annotations
+
+import time
+from typing import Any
+from urllib.parse import urljoin
+
+import httpx
+
+from app.core.config import get_settings
+
+
+def _join_url(base_url: str, path: str) -> str:
+    return urljoin(base_url.rstrip("/") + "/", path.lstrip("/"))
+
+
+def _unwrap_items(payload: Any) -> list[dict[str, Any]]:
+    if isinstance(payload, list):
+        return [item for item in payload if isinstance(item, dict)]
+    if isinstance(payload, dict):
+        data = payload.get("data")
+        if isinstance(data, dict) and isinstance(data.get("items"), list):
+            return [item for item in data["items"] if isinstance(item, dict)]
+        if isinstance(payload.get("items"), list):
+            return [item for item in payload["items"] if isinstance(item, dict)]
+        if isinstance(data, list):
+            return [item for item in data if isinstance(item, dict)]
+    return []
+
+
+def _unwrap_dict(payload: Any) -> dict[str, Any]:
+    if isinstance(payload, dict) and isinstance(payload.get("data"), dict):
+        return payload["data"]
+    return payload if isinstance(payload, dict) else {}
+
+
+# Inference calls are intentionally short-timeout:
+#   - load dispatch only confirms the compute node accepted the request
+#     (the actual model load now runs asynchronously on the node).
+#   - status/unload must never block the platform for long when a node is
+#     unreachable but still marked online.
+INFERENCE_LOAD_TIMEOUT = httpx.Timeout(30, connect=10)
+INFERENCE_STATUS_TIMEOUT = httpx.Timeout(30, connect=5)
+INFERENCE_UNLOAD_TIMEOUT = httpx.Timeout(30, connect=5)
+
+
+class ComputeNodeClient:
+    """Application-side client for one compute node.
+
+    The client accepts both current YG Compute API responses and common
+    wrapper shapes such as `{code,message,data}` to make future engine/node
+    adapters less brittle.
+    """
+
+    def __init__(self, api_base_url: str, token: str | None = None, timeout: float | None = None) -> None:
+        settings = get_settings()
+        self.api_base_url = api_base_url.rstrip("/")
+        self.token = token or settings.compute_service_token
+        self.timeout = timeout or settings.compute_request_timeout_seconds
+        self.route_prefix = settings.route_prefix.rstrip("/") or "/modelTF"
+
+    def headers(self) -> dict[str, str]:
+        if not self.token:
+            return {}
+        return {"X-Compute-Token": self.token}
+
+    async def test_connection(self) -> dict[str, Any]:
+        started = time.perf_counter()
+        health = await self.health()
+        gpus = await self.gpus()
+        return {
+            "success": True,
+            "latency_ms": int((time.perf_counter() - started) * 1000),
+            "health": health,
+            "gpus": gpus,
+        }
+
+    async def health(self) -> dict[str, Any]:
+        paths = [f"{self.route_prefix}/v1/compute/health", f"{self.route_prefix}/health", "/health"]
+        last_error = ""
+        async with httpx.AsyncClient(timeout=self.timeout, headers=self.headers()) as client:
+            for path in paths:
+                try:
+                    response = await client.get(_join_url(self.api_base_url, path))
+                    response.raise_for_status()
+                    return _unwrap_dict(response.json())
+                except Exception as exc:  # noqa: BLE001 - keep endpoint compatibility fallback broad
+                    last_error = str(exc)
+        raise RuntimeError(last_error or "compute health check failed")
+
+    async def gpus(self) -> list[dict[str, Any]]:
+        paths = [
+            f"{self.route_prefix}/compute/resources/gpus",
+            f"{self.route_prefix}/v1/compute/resources/gpus",
+            "/compute/resources/gpus",
+        ]
+        last_error = ""
+        async with httpx.AsyncClient(timeout=self.timeout, headers=self.headers()) as client:
+            for path in paths:
+                try:
+                    response = await client.get(_join_url(self.api_base_url, path))
+                    response.raise_for_status()
+                    return _unwrap_items(response.json())
+                except Exception as exc:  # noqa: BLE001
+                    last_error = str(exc)
+        raise RuntimeError(last_error or "compute gpu discovery failed")
+
+    async def create_job(self, payload: dict[str, Any]) -> dict[str, Any]:
+        async with httpx.AsyncClient(timeout=self.timeout, headers=self.headers()) as client:
+            response = await client.post(_join_url(self.api_base_url, f"{self.route_prefix}/compute/jobs"), json=payload)
+            response.raise_for_status()
+            return _unwrap_dict(response.json())
+
+    async def preview_job(self, payload: dict[str, Any]) -> dict[str, Any]:
+        async with httpx.AsyncClient(timeout=self.timeout, headers=self.headers()) as client:
+            response = await client.post(
+                _join_url(self.api_base_url, f"{self.route_prefix}/compute/jobs/preview"),
+                json=payload,
+            )
+            response.raise_for_status()
+            return _unwrap_dict(response.json())
+
+    async def validate_job(self, payload: dict[str, Any]) -> dict[str, Any]:
+        async with httpx.AsyncClient(timeout=self.timeout, headers=self.headers()) as client:
+            response = await client.post(
+                _join_url(self.api_base_url, f"{self.route_prefix}/compute/jobs/validate"),
+                json=payload,
+            )
+            response.raise_for_status()
+            return _unwrap_dict(response.json())
+
+    async def check_paths(self, paths: list[dict[str, Any]]) -> dict[str, Any]:
+        async with httpx.AsyncClient(timeout=self.timeout, headers=self.headers()) as client:
+            response = await client.post(
+                _join_url(self.api_base_url, f"{self.route_prefix}/compute/files/check-paths"),
+                json={"paths": paths},
+            )
+            response.raise_for_status()
+            return _unwrap_dict(response.json())
+
+    async def list_files(
+        self,
+        root: str = "data",
+        relative_path: str = "",
+        directories_only: bool = False,
+    ) -> dict[str, Any]:
+        async with httpx.AsyncClient(timeout=self.timeout, headers=self.headers()) as client:
+            response = await client.get(
+                _join_url(self.api_base_url, f"{self.route_prefix}/compute/files/list"),
+                params={"root": root, "relative_path": relative_path, "directories_only": directories_only},
+            )
+            response.raise_for_status()
+            return _unwrap_dict(response.json())
+
+    async def get_job(self, job_id: str) -> dict[str, Any]:
+        async with httpx.AsyncClient(timeout=self.timeout, headers=self.headers()) as client:
+            response = await client.get(_join_url(self.api_base_url, f"{self.route_prefix}/compute/jobs/{job_id}"))
+            response.raise_for_status()
+            return _unwrap_dict(response.json())
+
+    async def stop_job(self, job_id: str) -> dict[str, Any]:
+        async with httpx.AsyncClient(timeout=self.timeout, headers=self.headers()) as client:
+            response = await client.post(_join_url(self.api_base_url, f"{self.route_prefix}/compute/jobs/{job_id}/stop"))
+            response.raise_for_status()
+            return _unwrap_dict(response.json())
+
+    async def job_logs(
+        self,
+        job_id: str,
+        tail_lines: int | None = None,
+        offset: int | None = None,
+        limit: int | None = None,
+    ) -> dict[str, Any]:
+        params = {
+            key: value
+            for key, value in {"tail_lines": tail_lines, "offset": offset, "limit": limit}.items()
+            if value is not None
+        }
+        async with httpx.AsyncClient(timeout=self.timeout, headers=self.headers()) as client:
+            response = await client.get(
+                _join_url(self.api_base_url, f"{self.route_prefix}/compute/jobs/{job_id}/logs"),
+                params=params,
+            )
+            response.raise_for_status()
+            return _unwrap_dict(response.json())
+
+    async def import_local_file(self, payload: dict[str, Any]) -> dict[str, Any]:
+        async with httpx.AsyncClient(timeout=self.timeout, headers=self.headers()) as client:
+            response = await client.post(
+                _join_url(self.api_base_url, f"{self.route_prefix}/compute/files/import-local"),
+                json=payload,
+            )
+            response.raise_for_status()
+            return _unwrap_dict(response.json())
+
+    async def _request(
+        self,
+        method: str,
+        path: str,
+        json_data: dict[str, Any] | None = None,
+        timeout: float | None = None,
+    ) -> dict[str, Any]:
+        """Generic request method for compute API endpoints."""
+        url = _join_url(self.api_base_url, f"{self.route_prefix}{path}")
+        async with httpx.AsyncClient(timeout=timeout or 300, headers=self.headers()) as client:
+            if method.upper() == "GET":
+                response = await client.get(url)
+            else:
+                response = await client.post(url, json=json_data)
+            response.raise_for_status()
+            return _unwrap_dict(response.json())
+
+    # ── Inference helpers (short timeouts — see module constants) ──────────
+
+    async def inference_load(self, payload: dict[str, Any]) -> dict[str, Any]:
+        """Dispatch a model load. Returns as soon as the node accepts the
+        request; the node now loads asynchronously (status goes 'loading')."""
+        return await self._request("POST", "/inference/load", json_data=payload, timeout=INFERENCE_LOAD_TIMEOUT)
+
+    async def inference_status(self) -> dict[str, Any]:
+        return await self._request("GET", "/inference/status", timeout=INFERENCE_STATUS_TIMEOUT)
+
+    async def inference_unload(self) -> dict[str, Any]:
+        return await self._request("POST", "/inference/unload", json_data={}, timeout=INFERENCE_UNLOAD_TIMEOUT)
+
+    async def upload_file(
+        self,
+        filename: str,
+        content: bytes,
+        target_relative_path: str,
+        resource_type: str | None = None,
+        resource_id: str | None = None,
+    ) -> dict[str, Any]:
+        data = {
+            "target_relative_path": target_relative_path,
+            "resource_type": resource_type or "",
+            "resource_id": resource_id or "",
+        }
+        files = {"file": (filename, content)}
+        timeout = httpx.Timeout(max(self.timeout, 60), connect=self.timeout)
+        async with httpx.AsyncClient(timeout=timeout, headers=self.headers()) as client:
+            response = await client.post(
+                _join_url(self.api_base_url, f"{self.route_prefix}/compute/files/upload"),
+                data=data,
+                files=files,
+            )
+            response.raise_for_status()
+            return _unwrap_dict(response.json())

backend/app/modules/compute_gateway/sync.py

@@ -0,0 +1,191 @@
+from __future__ import annotations
+
+import json
+import time
+from typing import Any
+
+from app.db.platform_store import get_platform_store
+from app.modules.compute_gateway.client import ComputeNodeClient
+
+# starting 状态允许的最大轮询次数（约 40 * 3s ≈ 2 分钟），超过即判定节点不可达
+MAX_STARTING_ATTEMPTS = 40
+
+
+def _node_for_task(task: dict[str, Any]) -> dict[str, Any] | None:
+    return next((node for node in get_platform_store().compute_nodes() if node["id"] == task.get("compute_node_id")), None)
+
+
+def _parse_inference_load_status(task: dict[str, Any]) -> tuple[list[dict[str, Any]], dict[str, Any]]:
+    load_status = task.get("load_status") or {}
+    if isinstance(load_status, str):
+        try:
+            load_status = json.loads(load_status)
+        except (json.JSONDecodeError, TypeError):
+            load_status = {}
+    return load_status.get("loaded_models") or [], load_status
+
+
+async def reconcile_inference_loads(store: Any) -> list[dict[str, Any]]:
+    """推进处于 starting 状态的推理加载。
+
+    模型加载已改为异步派发：/model-compare/{id}/load 立即返回，这里在每次
+    轮询时查询对应计算节点的 /inference/status，把任务从 starting 推进到
+    ready/error。使用短超时，单节点不可达不会阻塞整轮轮询。
+    """
+    reconciled: list[dict[str, Any]] = []
+    now = time.time()
+    for task in store.compare_tasks():
+        items, _ = _parse_inference_load_status(task)
+        if not any(item.get("status") == "starting" for item in items):
+            continue
+        # dirty 只要处理过任一 starting 项就置位：load_attempts / last_polled_at
+        # 必须落库，否则节点不可达时计数不会累积，封顶逻辑永远触发不了
+        dirty = False
+        for item in items:
+            if item.get("status") != "starting":
+                continue
+            # 节流：同一 item 每 3s 只查询一次
+            if now - float(item.get("last_polled_at") or 0) < 3:
+                continue
+            item["last_polled_at"] = now
+            item["load_attempts"] = int(item.get("load_attempts") or 0) + 1
+            dirty = True
+            node = next((n for n in store.compute_nodes() if n["id"] == item.get("node_id")), None)
+            if not node:
+                item["status"] = "error"
+                item["error"] = "compute node deleted"
+                store.mark_inference_unloaded(item.get("node_id") or "")
+                continue
+            if not node.get("enabled") or node.get("scheduler_status") != "online":
+                item["status"] = "error"
+                item["error"] = "compute node offline"
+                store.mark_inference_unloaded(node["id"])
+                continue
+            try:
+                status = await ComputeNodeClient(node["api_base_url"]).inference_status()
+            except Exception as exc:  # noqa: BLE001 - node unreachable; keep retrying until cap
+                if int(item.get("load_attempts") or 0) >= MAX_STARTING_ATTEMPTS:
+                    item["status"] = "error"
+                    item["error"] = f"compute node unreachable: {exc}"
+                    store.mark_inference_unloaded(node["id"])
+                continue
+            node_status = status.get("status")
+            if node_status == "ready":
+                item["status"] = "ready"
+                item.pop("error", None)
+                store.mark_inference_loaded(node["id"])
+            elif node_status == "error":
+                item["status"] = "error"
+                item["error"] = status.get("error") or "model load failed on compute node"
+                store.mark_inference_unloaded(node["id"])
+            elif node_status == "idle":
+                # 节点重启导致已加载模型丢失
+                item["status"] = "error"
+                item["error"] = "model disappeared from compute node (node may have restarted)"
+                store.mark_inference_unloaded(node["id"])
+            # node_status == "loading" -> 保持 starting，下轮再查
+        if dirty:
+            if any(i.get("status") in {"ready", "running"} for i in items):
+                new_status = "loaded"
+            elif any(i.get("status") == "starting" for i in items):
+                new_status = "starting"  # 仍在加载中，保持 starting
+            else:
+                new_status = "failed"
+            store.update_compare_task(task["id"], {"status": new_status, "load_status": {"loaded_models": items}})
+            reconciled.append({"task_id": task["id"], "status": new_status})
+    return reconciled
+
+
+async def fetch_eval_result_content(client: ComputeNodeClient, node: dict[str, Any], job: dict[str, Any]) -> dict[str, Any] | None:
+    output_dir = job.get("output_dir")
+    if not output_dir:
+        return None
+    full_path = f"{str(output_dir).rstrip('/')}/eval_results.json"
+    data_root = "/data/yg-ft/"
+    if full_path.startswith(data_root):
+        full_path = full_path[len(data_root):]
+    rel_path = full_path.lstrip("/")
+    import httpx
+    url = f"{node['api_base_url'].rstrip('/')}/modelTF/compute/files/read"
+    async with httpx.AsyncClient(timeout=30, headers=client.headers()) as http:
+        response = await http.get(url, params={"path": rel_path})
+        response.raise_for_status()
+        payload = response.json()
+        return payload if isinstance(payload, dict) else None
+
+
+async def poll_compute_jobs_once() -> dict[str, Any]:
+    store = get_platform_store()
+    synced: list[dict[str, Any]] = []
+    failed: list[dict[str, str]] = []
+    for task in store.running_compute_tasks():
+        node = _node_for_task(task)
+        if not node:
+            failed.append({"task_id": task["id"], "error": "compute node not found"})
+            continue
+        try:
+            client = ComputeNodeClient(node["api_base_url"])
+            job = await client.get_job(task["compute_job_id"])
+            try:
+                logs = await client.job_logs(task["compute_job_id"], tail_lines=5000)
+                store.record_training_log_metrics(task["id"], str(logs.get("content") or ""))
+            except Exception:
+                pass
+            # P0-4: Force-fetch last log snippet when job reaches terminal state
+            if job.get("status") in {"failed", "stopped"}:
+                try:
+                    last_logs = await client.job_logs(task["compute_job_id"], tail_lines=200)
+                    job["log_snippet"] = str(last_logs.get("content") or "")[:8192]
+                except Exception:
+                    pass
+            synced.append(store.apply_compute_job(task["id"], job))
+        except Exception as exc:  # noqa: BLE001 - keep polling other jobs
+            failed.append({"task_id": task["id"], "error": str(exc)})
+    standalone_synced: list[dict[str, Any]] = []
+    for record in store.active_standalone_compute_jobs():
+        node = next((item for item in store.compute_nodes() if item["id"] == record.get("node_id")), None)
+        if not node:
+            failed.append({"job_id": record["id"], "error": "compute node not found"})
+            continue
+        try:
+            job = await ComputeNodeClient(node["api_base_url"]).get_job(record["id"])
+            standalone_synced.append(store.sync_model_merge_job(record["id"], job))
+        except Exception as exc:  # noqa: BLE001 - keep polling other jobs
+            failed.append({"job_id": record["id"], "error": str(exc)})
+
+    # ── Eval job sync ────────────────────────────────────────────────
+    eval_synced = 0
+    for eval_task in store.running_eval_tasks():
+        node = next(
+            (item for item in store.compute_nodes() if item["id"] == eval_task.get("compute_node_id")),
+            None,
+        )
+        if not node:
+            failed.append({"eval_task_id": eval_task["id"], "error": "compute node not found"})
+            continue
+        try:
+            client = ComputeNodeClient(node["api_base_url"])
+            job = await client.get_job(eval_task["compute_job_id"])
+            result_content = None
+            # Try to read eval_results.json from the job output directory
+            if job.get("status") == "completed" and job.get("output_dir"):
+                try:
+                    result_content = await fetch_eval_result_content(client, node, job)
+                except Exception:
+                    pass
+            store.apply_eval_job_result(eval_task["id"], job, result_content)
+            # 评测 GPU 占用由 eval_tasks 状态派生，无需维护推理内存标记
+            eval_synced += 1
+        except Exception as exc:  # noqa: BLE001
+            failed.append({"eval_task_id": eval_task["id"], "error": str(exc)})
+
+    # ── Inference load reconciliation ─────────────────────────────────────
+    try:
+        inference_reconciled = await reconcile_inference_loads(store)
+    except Exception as exc:  # noqa: BLE001 - keep polling alive
+        failed.append({"inference_reconcile": str(exc)})
+        inference_reconciled = []
+
+    return {"synced": len(synced) + len(standalone_synced) + eval_synced, "failed": failed,
+            "items": synced, "standalone": standalone_synced, "eval_synced": eval_synced,
+            "inference_reconciled": inference_reconciled}

backend/app/modules/data_convert/__init__.py

@@ -0,0 +1,3 @@
+from .router import router
+
+__all__ = ["router"]

backend/app/modules/data_convert/router.py

@@ -0,0 +1,356 @@
+from __future__ import annotations
+
+import json
+import os
+from pathlib import Path
+from typing import Any
+
+from fastapi import APIRouter, Body, Depends, File, UploadFile
+from fastapi.responses import FileResponse
+
+from app.api.v1.endpoints.platform import ok, fail
+from app.core.auth import get_current_user
+from app.db.platform_store import get_platform_store, new_id
+
+
+router = APIRouter(prefix="/data-convert", tags=["data-convert"])
+
+# 存储根目录
+STORAGE_ROOT = Path(__file__).resolve().parents[3] / "storage" / "data-convert"
+
+
+def _safe_output_filename(value: Any) -> str:
+    """输出文件名白名单校验：仅允许普通文件名，阻断 ``../``、``/``、``\\`` 等路径穿越。
+
+    转换结果始终写入 ``STORAGE_ROOT/<task_id>/output/<output_filename>``，
+    若文件名可被注入路径分隔符，将导致任意文件读写/删除。
+    """
+    name = str(value or "converted-data.jsonl").strip()
+    if (
+        not name
+        or name in {".", ".."}
+        or name != Path(name).name
+        or "/" in name
+        or "\\" in name
+        or any(ord(character) < 32 or ord(character) == 127 for character in name)
+    ):
+        raise fail(400, "output filename must be a plain file name")
+    return name
+
+
+def _task_output_path(task: dict[str, Any]) -> Path:
+    """返回经过白名单校验的转换输出文件路径（始终位于任务 output 目录内）。"""
+    return _output_dir(task["id"]) / _safe_output_filename(task.get("output_filename"))
+
+
+def _task_dir(task_id: str) -> Path:
+    return STORAGE_ROOT / task_id
+
+
+def _input_dir(task_id: str) -> Path:
+    return _task_dir(task_id) / "input"
+
+
+def _output_dir(task_id: str) -> Path:
+    return _task_dir(task_id) / "output"
+
+
+@router.get("")
+def list_tasks(
+    page: int = 1,
+    page_size: int = 20,
+    current_user: dict = Depends(get_current_user),
+) -> dict[str, Any]:
+    store = get_platform_store()
+    with store.connect() as conn:
+        rows = conn.execute(
+            "SELECT * FROM data_convert_tasks WHERE deleted_at IS NULL "
+            "ORDER BY create_time DESC LIMIT %s OFFSET %s",
+            (page_size, (page - 1) * page_size),
+        ).fetchall()
+        total = conn.execute(
+            "SELECT COUNT(*) FROM data_convert_tasks WHERE deleted_at IS NULL"
+        ).fetchone()[0]
+    return ok({"items": [dict(r) for r in rows], "total": total})
+
+
+@router.post("")
+def create_task(
+    payload: dict[str, Any] = Body(...),
+    current_user: dict = Depends(get_current_user),
+) -> dict[str, Any]:
+    name = str(payload.get("name") or "").strip()
+    if not name:
+        raise fail(400, "name is required")
+    task_id = new_id("dct")
+    output_filename = _safe_output_filename(payload.get("output_filename"))
+    description = str(payload.get("description") or "").strip()
+    store = get_platform_store()
+    with store.connect() as conn:
+        conn.execute(
+            "INSERT INTO data_convert_tasks (id, name, description, output_filename) "
+            "VALUES (%s, %s, %s, %s)",
+            (task_id, name, description, output_filename),
+        )
+    # 创建目录
+    _input_dir(task_id).mkdir(parents=True, exist_ok=True)
+    _output_dir(task_id).mkdir(parents=True, exist_ok=True)
+    return ok(_get_task(task_id))
+
+
+@router.get("/{task_id}")
+def get_task(
+    task_id: str,
+    current_user: dict = Depends(get_current_user),
+) -> dict[str, Any]:
+    task = _get_task(task_id)
+    if not task:
+        raise fail(404, "task not found")
+    # 附加输入文件列表
+    input_dir = _input_dir(task_id)
+    files = []
+    if input_dir.exists():
+        for f in sorted(input_dir.iterdir()):
+            if f.is_file():
+                files.append({"name": f.name, "size": f.stat().st_size})
+    task["input_files"] = files
+    return ok(task)
+
+
+@router.post("/{task_id}/source-files")
+async def upload_source_files(
+    task_id: str,
+    files: list[UploadFile] = File(...),
+    current_user: dict = Depends(get_current_user),
+) -> dict[str, Any]:
+    task = _get_task(task_id)
+    if not task:
+        raise fail(404, "task not found")
+    if task["status"] not in ("pending", "uploaded"):
+        raise fail(400, "task is not editable")
+    input_dir = _input_dir(task_id)
+    input_dir.mkdir(parents=True, exist_ok=True)
+    staged = []
+    for upload in files:
+        name = Path(upload.filename or "input.json").name
+        if not name.lower().endswith(".json"):
+            raise fail(415, f"only JSON files are supported: {name}")
+        target = input_dir / name
+        content = await upload.read()
+        target.write_bytes(content)
+        staged.append({"name": name, "size": len(content)})
+    store = get_platform_store()
+    # 标记上传完成
+    with store.connect() as conn:
+        conn.execute(
+            "UPDATE data_convert_tasks SET status='uploaded', update_time=NOW() WHERE id=%s",
+            (task_id,),
+        )
+    # 自动转换并导入数据集
+    try:
+        output_dir = _output_dir(task_id)
+        output_dir.mkdir(parents=True, exist_ok=True)
+        output_path = _task_output_path(task)
+        # 清空旧输出（如果重新上传）
+        if output_path.exists():
+            output_path.unlink()
+        input_count = 0
+        output_count = 0
+        for json_file in sorted(input_dir.iterdir()):
+            if not json_file.is_file() or not json_file.name.lower().endswith(".json"):
+                continue
+            input_count += 1
+            with open(json_file, "r", encoding="utf-8") as f:
+                data = json.load(f)
+            if isinstance(data, list):
+                records = data
+            elif isinstance(data, dict):
+                records = [data]
+            else:
+                raise ValueError(f"JSON must be object or array: {json_file.name}")
+            with open(output_path, "a", encoding="utf-8") as f:
+                for record in records:
+                    f.write(json.dumps(record, ensure_ascii=False) + "\n")
+                    output_count += 1
+        with store.connect() as conn:
+            conn.execute(
+                "UPDATE data_convert_tasks SET status='completed', "
+                "input_count=%s, output_count=%s, update_time=NOW() WHERE id=%s",
+                (input_count, output_count, task_id),
+            )
+        # 自动导入数据集
+        content = output_path.read_text(encoding="utf-8")
+        size_bytes = len(content.encode("utf-8"))
+        dataset = store.create_dataset({
+            "name": task["name"],
+            "type": "train",
+            "storage_type": "local",
+            "source": "upload",
+            "task_id": task_id,
+            "size": f"{size_bytes} B",
+            "count": output_count,
+            "description": f"由数据类型转换任务 {task_id} 自动导入",
+        })
+        dataset_id = dataset["id"]
+        with store.connect() as conn:
+            store.add_dataset_file(conn, dataset_id, _safe_output_filename(task.get("output_filename")), content)
+        return ok({
+            "staged_files": staged,
+            "auto_converted": True,
+            "dataset_id": dataset_id,
+            "input_count": input_count,
+            "output_count": output_count,
+        })
+    except Exception as exc:
+        with store.connect() as conn:
+            conn.execute(
+                "UPDATE data_convert_tasks SET status='failed', error_message=%s, update_time=NOW() WHERE id=%s",
+                (str(exc)[:500], task_id),
+            )
+        return ok({"staged_files": staged, "auto_converted": False, "error": str(exc)[:500]})
+
+
+@router.post("/{task_id}/run")
+def run_convert(
+    task_id: str,
+    current_user: dict = Depends(get_current_user),
+) -> dict[str, Any]:
+    task = _get_task(task_id)
+    if not task:
+        raise fail(404, "task not found")
+    if task["status"] not in ("uploaded", "completed", "failed"):
+        raise fail(400, "please upload source files first")
+    # 标记运行中
+    store = get_platform_store()
+    with store.connect() as conn:
+        conn.execute(
+            "UPDATE data_convert_tasks SET status='running', error_message='', update_time=NOW() WHERE id=%s",
+            (task_id,),
+        )
+    try:
+        input_dir = _input_dir(task_id)
+        output_dir = _output_dir(task_id)
+        output_dir.mkdir(parents=True, exist_ok=True)
+        output_path = _task_output_path(task)
+        input_count = 0
+        output_count = 0
+        for json_file in sorted(input_dir.iterdir()):
+            if not json_file.is_file() or not json_file.name.lower().endswith(".json"):
+                continue
+            input_count += 1
+            with open(json_file, "r", encoding="utf-8") as f:
+                data = json.load(f)
+            if isinstance(data, list):
+                records = data
+            elif isinstance(data, dict):
+                records = [data]
+            else:
+                raise ValueError(f"JSON must be object or array: {json_file.name}")
+            with open(output_path, "a", encoding="utf-8") as f:
+                for record in records:
+                    f.write(json.dumps(record, ensure_ascii=False) + "\n")
+                    output_count += 1
+        # 更新任务状态
+        with store.connect() as conn:
+            conn.execute(
+                "UPDATE data_convert_tasks SET status='completed', "
+                "input_count=%s, output_count=%s, update_time=NOW() WHERE id=%s",
+                (input_count, output_count, task_id),
+            )
+    except Exception as exc:
+        with store.connect() as conn:
+            conn.execute(
+                "UPDATE data_convert_tasks SET status='failed', error_message=%s, update_time=NOW() WHERE id=%s",
+                (str(exc)[:500], task_id),
+            )
+        raise fail(500, f"convert failed: {exc}")
+    return ok(_get_task(task_id))
+
+
+@router.get("/{task_id}/download")
+def download_result(
+    task_id: str,
+    current_user: dict = Depends(get_current_user),
+):
+    task = _get_task(task_id)
+    if not task:
+        raise fail(404, "task not found")
+    if task["status"] != "completed":
+        raise fail(400, "task is not completed")
+    output_path = _task_output_path(task)
+    if not output_path.exists():
+        raise fail(404, "output file not found")
+    return FileResponse(
+        str(output_path),
+        media_type="application/octet-stream",
+        filename=_safe_output_filename(task.get("output_filename")),
+    )
+
+
+@router.post("/{task_id}/import-as-dataset")
+def import_as_dataset(
+    task_id: str,
+    payload: dict[str, Any] = Body(default={}),
+    current_user: dict = Depends(get_current_user),
+) -> dict[str, Any]:
+    """把已转换的 JSONL 文件导入为数据集管理中的上传任务记录（source='task'）。"""
+    task = _get_task(task_id)
+    if not task:
+        raise fail(404, "task not found")
+    if task["status"] != "completed":
+        raise fail(400, "task is not completed")
+    output_path = _task_output_path(task)
+    if not output_path.exists():
+        raise fail(404, "output file not found")
+    content = output_path.read_text(encoding="utf-8")
+    dataset_name = str(payload.get("name") or task["name"]).strip()
+    description = str(payload.get("description") or f"由数据类型转换任务 {task_id} 导入").strip()
+    size_bytes = len(content.encode("utf-8"))
+    store = get_platform_store()
+    # 用 store 提供的接口创建数据集与文件
+    dataset = store.create_dataset({
+        "name": dataset_name,
+        "type": "train",
+        "storage_type": "local",
+        "source": "upload",
+        "task_id": task_id,
+        "size": f"{size_bytes} B",
+        "count": task["output_count"],
+        "description": description,
+    })
+    dataset_id = dataset["id"]
+    with store.connect() as conn:
+        store.add_dataset_file(conn, dataset_id, _safe_output_filename(task.get("output_filename")), content)
+    return ok({"dataset_id": dataset_id, "name": dataset_name})
+
+
+@router.delete("/{task_id}")
+def delete_task(
+    task_id: str,
+    current_user: dict = Depends(get_current_user),
+) -> dict[str, Any]:
+    task = _get_task(task_id)
+    if not task:
+        raise fail(404, "task not found")
+    store = get_platform_store()
+    with store.connect() as conn:
+        conn.execute(
+            "UPDATE data_convert_tasks SET deleted_at=NOW() WHERE id=%s",
+            (task_id,),
+        )
+    # 清理文件
+    import shutil
+    task_dir = _task_dir(task_id)
+    if task_dir.exists():
+        shutil.rmtree(task_dir, ignore_errors=True)
+    return ok({"deleted": task_id})
+
+
+def _get_task(task_id: str) -> dict[str, Any] | None:
+    store = get_platform_store()
+    with store.connect() as conn:
+        row = conn.execute(
+            "SELECT * FROM data_convert_tasks WHERE id=%s AND deleted_at IS NULL",
+            (task_id,),
+        ).fetchone()
+    return dict(row) if row else None

backend/app/modules/data_process/constants.py

@@ -0,0 +1,7 @@
+"""数据处理模块的共享限制。"""
+
+MAX_QA_PAIRS_PER_ITEM = 50
+MODEL_GENERATION_BATCH_SIZE = 10
+
+
+__all__ = ["MAX_QA_PAIRS_PER_ITEM", "MODEL_GENERATION_BATCH_SIZE"]

backend/app/modules/data_process/dataset_format.py

@@ -0,0 +1,158 @@
+"""Dataset format validation for Alpaca, ShareGPT, DPO, CPT formats.
+
+Used by the training preflight flow to validate that uploaded dataset files
+conform to the declared format before submitting to the compute node.
+"""
+
+from __future__ import annotations
+
+import json
+from typing import Any
+
+
+def _load_sample(path: str | None, content: str | None = None, max_samples: int = 20) -> list[dict[str, Any]]:
+    """Load up to max_samples records from JSONL file path or raw content string."""
+    try:
+        if content is not None:
+            text = content.strip()
+        elif path:
+            with open(path, "r", encoding="utf-8") as fh:
+                text = fh.read().strip()
+        else:
+            return []
+    except Exception:
+        return []
+
+    if not text:
+        return []
+
+    # 先按整文件 JSON（数组/单对象）解析，兼容 .json；失败再按 jsonl 逐行解析
+    try:
+        value = json.loads(text)
+    except json.JSONDecodeError:
+        value = None
+    if isinstance(value, list):
+        return [item for item in value[:max_samples] if isinstance(item, dict)]
+    if isinstance(value, dict):
+        return [value]
+
+    lines = text.splitlines()[:max_samples]
+    records: list[dict[str, Any]] = []
+    for line in lines:
+        line = line.strip()
+        if not line:
+            continue
+        try:
+            record = json.loads(line)
+        except json.JSONDecodeError:
+            continue
+        if isinstance(record, dict):
+            records.append(record)
+    return records
+
+
+def _check_alpaca(records: list[dict[str, Any]]) -> list[str]:
+    """Validate Alpaca format: requires 'instruction' field."""
+    errors: list[str] = []
+    if not records:
+        errors.append("Alpaca 格式数据集无有效记录")
+        return errors
+    missing_instruction = sum(1 for r in records if not r.get("instruction"))
+    if missing_instruction:
+        errors.append(
+            f"Alpaca 格式要求每条记录包含 instruction 字段，"
+            f"前{len(records)}条中有{missing_instruction}条缺失"
+        )
+    return errors
+
+
+def _check_sharegpt(records: list[dict[str, Any]]) -> list[str]:
+    """Validate ShareGPT format: requires 'messages' (list of dicts with role/content)."""
+    errors: list[str] = []
+    if not records:
+        errors.append("ShareGPT 格式数据集无有效记录")
+        return errors
+    bad = 0
+    for r in records:
+        messages = r.get("messages")
+        if not isinstance(messages, list) or not messages:
+            bad += 1
+            continue
+        for msg in messages:
+            if not isinstance(msg, dict) or "role" not in msg or "content" not in msg:
+                bad += 1
+                break
+    if bad:
+        errors.append(
+            f"ShareGPT 格式要求每条记录包含 messages 列表，"
+            f"每条消息需有 role 和 content 字段，前{len(records)}条中有{bad}条不符合"
+        )
+    return errors
+
+
+def _check_dpo(records: list[dict[str, Any]]) -> list[str]:
+    """Validate DPO format: requires 'chosen' and 'rejected' fields."""
+    errors: list[str] = []
+    if not records:
+        errors.append("DPO 格式数据集无有效记录")
+        return errors
+    missing_chosen = sum(1 for r in records if not r.get("chosen"))
+    missing_rejected = sum(1 for r in records if not r.get("rejected"))
+    if missing_chosen:
+        errors.append(f"DPO 格式要求 chosen 字段，前{len(records)}条中有{missing_chosen}条缺失")
+    if missing_rejected:
+        errors.append(f"DPO 格式要求 rejected 字段，前{len(records)}条中有{missing_rejected}条缺失")
+    return errors
+
+
+def _check_cpt(records: list[dict[str, Any]]) -> list[str]:
+    """Validate CPT format: requires 'text' field, should NOT have instruction/output."""
+    errors: list[str] = []
+    if not records:
+        errors.append("CPT 格式数据集无有效记录")
+        return errors
+    missing_text = sum(1 for r in records if not r.get("text"))
+    has_instruction = sum(1 for r in records if r.get("instruction") or r.get("output"))
+    if missing_text:
+        errors.append(f"CPT 格式要求 text 字段，前{len(records)}条中有{missing_text}条缺失")
+    if has_instruction:
+        errors.append(
+            f"CPT 格式不应包含 instruction/output 字段（疑似 Alpaca 格式），"
+            f"前{len(records)}条中有{has_instruction}条包含此类字段"
+        )
+    return errors
+
+
+FORMAT_VALIDATORS = {
+    "alpaca": _check_alpaca,
+    "alpaca_jsonl": _check_alpaca,
+    "sharegpt": _check_sharegpt,
+    "dpo": _check_dpo,
+    "cpt": _check_cpt,
+    "pt": _check_cpt,
+}
+
+
+def validate_dataset_format(
+    dataset_format: str,
+    content: str | None = None,
+    path: str | None = None,
+    max_samples: int = 20,
+) -> list[str]:
+    """Validate dataset content against expected format.
+
+    Args:
+        dataset_format: One of 'alpaca', 'sharegpt', 'dpo', 'cpt'.
+        content: Raw file content (JSONL text). Mutually exclusive with path.
+        path: File path to read content from.
+        max_samples: Maximum records to sample for validation.
+
+    Returns:
+        List of error messages (empty if valid).
+    """
+    fmt = str(dataset_format).lower().strip()
+    validator = FORMAT_VALIDATORS.get(fmt)
+    if not validator:
+        return [f"不支持的数据集格式: {dataset_format}，支持的格式: {', '.join(sorted(FORMAT_VALIDATORS))}"]
+    records = _load_sample(path=path, content=content, max_samples=max_samples)
+    return validator(records)

backend/app/modules/data_process/document_chunking.py

@@ -0,0 +1,499 @@
+"""基于 Docling 与 LlamaIndex 的文档切分实现。"""
+
+from __future__ import annotations
+
+import os
+import re
+import threading
+import unicodedata
+from dataclasses import dataclass
+from functools import lru_cache
+from io import BytesIO
+from typing import Any, Literal
+
+import tiktoken
+from docling_core.transforms.chunker.hierarchical_chunker import ChunkingSerializerProvider
+from llama_index.core import Document
+from llama_index.core.base.embeddings.base import BaseEmbedding
+from llama_index.core.node_parser import SemanticSplitterNodeParser, SentenceSplitter
+
+from app.modules.data_process.algorithms import normalize_text
+
+ChunkMethod = Literal["layout_hybrid", "semantic", "fixed"]
+
+_PAGE_FURNITURE = re.compile(
+    r"(?m)^\s*(?:第\s*\d+\s*页\s*共\s*\d+\s*页|[-—–]?\s*\d+\s*[/／]\s*\d+\s*[-—–]?)\s*$"
+)
+_COMPACT_CHARACTER = re.compile(r"[\w\u3400-\u4dbf\u4e00-\u9fff]", re.UNICODE)
+_CONVERTER_LOCK = threading.Lock()
+
+
+@dataclass(frozen=True, slots=True)
+class DocumentChunk:
+    """切片正文及其在原文件中的可追溯信息。"""
+
+    original_content: str
+    contextualized_content: str
+    source_start: int | None
+    source_end: int | None
+    source_start_line: int | None
+    source_end_line: int | None
+    token_count: int
+    heading_path: tuple[str, ...] = ()
+    source_pages: tuple[int, ...] = ()
+    doc_item_refs: tuple[str, ...] = ()
+    source_bboxes: tuple[dict[str, Any], ...] = ()
+
+
+def _sentence_chunks(text: str) -> list[str]:
+    """提供稳定的中英文句界，避免 LlamaIndex 默认分词器下载额外资源。"""
+
+    boundary = re.compile(
+        r".*?(?:\n\s*\n|[。！？!?；;](?:[\"'”’）】》]*)|\.(?:\s+|$)|$)",
+        re.DOTALL,
+    )
+    return [part for part in boundary.findall(text) if part]
+
+
+@lru_cache(maxsize=1)
+def _tokenizer() -> tiktoken.Encoding:
+    """加载 cl100k_base 编码器，优先在线下载，失败时使用本地缓存以支持离线环境。"""
+    import os
+    import base64
+
+    # 先设置缓存目录环境变量
+    offline_cache = os.path.expanduser("~/.cache/tiktoken")
+    os.environ.setdefault("TIKTOKEN_CACHE_DIR", offline_cache)
+
+    try:
+        # 尝试标准方式加载
+        return tiktoken.get_encoding("cl100k_base")
+    except Exception:
+        # 如果失败，尝试手动从本地文件构造
+        try:
+            from pathlib import Path
+
+            local_file = Path(offline_cache) / "9b5ad71b2ce5302211f9c61530b329a4922fc6a4"
+            if not local_file.exists():
+                # 尝试另一个可能的文件名
+                local_file = Path(offline_cache) / "cl100k_base.tiktoken"
+
+            if local_file.exists():
+                # 读取 BPE 文件内容
+                with open(local_file, "rb") as f:
+                    contents = f.read()
+
+                # 解析 BPE 文件
+                mergeable_ranks = {}
+                for line in contents.splitlines():
+                    if line:
+                        token, rank = line.split()
+                        mergeable_ranks[base64.b64decode(token)] = int(rank)
+
+                # 构造 Encoding 对象
+                import tiktoken.core
+                return tiktoken.core.Encoding(
+                    name="cl100k_base",
+                    pat_str=r"""'(?i:[sdmt]|ll|ve|re)|[^\r\n\p{L}\p{N}]?+\p{L}+|\p{N}{1,3}| ?[^\s\p{L}\p{N}]++[\r\n]*|\s*[\r\n]|\s+(?!\S)|\s+""",
+                    mergeable_ranks=mergeable_ranks,
+                    special_tokens={
+                        "<|endoftext|>": 100257,
+                        "<|fim_prefix|>": 100258,
+                        "<|fim_middle|>": 100259,
+                        "<|fim_suffix|>": 100260,
+                        "<|endofprompt|>": 100276,
+                    },
+                )
+        except Exception:
+            pass
+
+        raise RuntimeError(
+            f"无法加载 cl100k_base 编码器\n"
+            f"请确保以下任一条件满足:\n"
+            f"1. 服务器可以访问网络\n"
+            f"2. 本地存在缓存文件: {offline_cache}/9b5ad71b2ce5302211f9c61530b329a4922fc6a4"
+        )
+
+
+def _text_chunks(
+    text: str,
+    *,
+    chunk_size: int,
+    chunk_overlap: int,
+) -> list[DocumentChunk]:
+    normalized = normalize_text(text)
+    if not normalized:
+        return []
+    splitter = SentenceSplitter(
+        chunk_size=chunk_size,
+        chunk_overlap=chunk_overlap,
+        tokenizer=_tokenizer().encode,
+        chunking_tokenizer_fn=_sentence_chunks,
+        include_metadata=False,
+        include_prev_next_rel=False,
+    )
+    nodes = splitter.get_nodes_from_documents([Document(text=normalized)])
+    return _nodes_to_chunks(nodes, normalized)
+
+
+def chunk_fixed_text(
+    text: str,
+    *,
+    chunk_size: int,
+    chunk_overlap: int,
+) -> list[DocumentChunk]:
+    """使用 LlamaIndex SentenceSplitter 按句界控制固定 Token 长度。"""
+
+    return _text_chunks(text, chunk_size=chunk_size, chunk_overlap=chunk_overlap)
+
+
+@lru_cache(maxsize=1)
+def _semantic_embedding_model() -> BaseEmbedding:
+    # 模型可在部署环境覆盖；默认模型体积较小且适合中英文语义边界判断。
+    from llama_index.embeddings.huggingface import HuggingFaceEmbedding
+
+    return HuggingFaceEmbedding(
+        model_name=os.getenv("DATA_PROCESS_EMBEDDING_MODEL", "BAAI/bge-small-zh-v1.5"),
+        device=os.getenv("DATA_PROCESS_EMBEDDING_DEVICE", "cpu"),
+        trust_remote_code=False,
+    )
+
+
+def chunk_semantic_text(
+    text: str,
+    *,
+    chunk_size: int,
+    chunk_overlap: int,
+    breakpoint_percentile_threshold: int,
+    embed_model: BaseEmbedding | None = None,
+) -> list[DocumentChunk]:
+    """使用 LlamaIndex SemanticSplitter 识别主题跳变，再限制最大长度。"""
+
+    normalized = normalize_text(text)
+    if not normalized:
+        return []
+    splitter = SemanticSplitterNodeParser.from_defaults(
+        embed_model=embed_model or _semantic_embedding_model(),
+        breakpoint_percentile_threshold=breakpoint_percentile_threshold,
+        buffer_size=1,
+        sentence_splitter=_sentence_chunks,
+        include_metadata=False,
+        include_prev_next_rel=False,
+    )
+    semantic_nodes = splitter.get_nodes_from_documents([Document(text=normalized)])
+    result: list[DocumentChunk] = []
+    search_from = 0
+    for node in semantic_nodes:
+        content = node.get_content().strip()
+        if not content:
+            continue
+        start = _locate_text(normalized, content, search_from)
+        if start is None:
+            start = _locate_text(normalized, content, 0)
+        if start is None:
+            continue
+        if len(_tokenizer().encode(content)) <= chunk_size:
+            result.append(_make_text_chunk(normalized, start, start + len(content)))
+        else:
+            for child in _text_chunks(
+                content,
+                chunk_size=chunk_size,
+                chunk_overlap=chunk_overlap,
+            ):
+                if child.source_start is None or child.source_end is None:
+                    continue
+                result.append(
+                    _make_text_chunk(
+                        normalized,
+                        start + child.source_start,
+                        start + child.source_end,
+                    )
+                )
+        search_from = start + len(content)
+    return result
+
+
+def _nodes_to_chunks(nodes: list[Any], source_text: str) -> list[DocumentChunk]:
+    chunks: list[DocumentChunk] = []
+    search_from = 0
+    for node in nodes:
+        content = node.get_content().strip()
+        if not content:
+            continue
+        raw_start = getattr(node, "start_char_idx", None)
+        raw_end = getattr(node, "end_char_idx", None)
+        if (
+            isinstance(raw_start, int)
+            and isinstance(raw_end, int)
+            and source_text[raw_start:raw_end].strip() == content
+        ):
+            start = raw_start + len(source_text[raw_start:raw_end]) - len(source_text[raw_start:raw_end].lstrip())
+        else:
+            start = _locate_text(source_text, content, search_from)
+            if start is None:
+                start = _locate_text(source_text, content, 0)
+        if start is None:
+            continue
+        end = start + len(content)
+        chunks.append(_make_text_chunk(source_text, start, end))
+        search_from = max(search_from, end)
+    return chunks
+
+
+def _locate_text(source: str, content: str, start: int) -> int | None:
+    position = source.find(content, start)
+    return position if position >= 0 else None
+
+
+def _make_text_chunk(source: str, start: int, end: int) -> DocumentChunk:
+    content = source[start:end]
+    return DocumentChunk(
+        original_content=content,
+        contextualized_content=content,
+        source_start=start,
+        source_end=end,
+        source_start_line=source.count("\n", 0, start) + 1,
+        source_end_line=source.count("\n", 0, max(start, end - 1)) + 1,
+        token_count=len(_tokenizer().encode(content)),
+    )
+
+
+@lru_cache(maxsize=1)
+def _document_converter():
+    from docling.document_converter import DocumentConverter
+
+    return DocumentConverter()
+
+
+class _MarkdownSerializerProvider(ChunkingSerializerProvider):
+    def get_serializer(self, doc: Any):
+        from docling_core.transforms.chunker.hierarchical_chunker import ChunkingDocSerializer
+        from docling_core.transforms.serializer.markdown import (
+            MarkdownParams,
+            MarkdownTableSerializer,
+        )
+        from docling_core.types.doc import DocItemLabel
+
+        excluded = {
+            DocItemLabel.DOCUMENT_INDEX,
+            DocItemLabel.PAGE_HEADER,
+            DocItemLabel.PAGE_FOOTER,
+        }
+        return ChunkingDocSerializer(
+            doc=doc,
+            table_serializer=MarkdownTableSerializer(),
+            params=MarkdownParams(
+                labels=set(DocItemLabel) - excluded,
+                compact_tables=True,
+                image_placeholder="",
+                escape_html=False,
+                escape_underscores=False,
+            ),
+        )
+
+
+def _clean_layout_text(value: str) -> str:
+    return normalize_text(_PAGE_FURNITURE.sub("", value)).strip()
+
+
+def _compact_with_offsets(value: str) -> tuple[str, list[int]]:
+    compact: list[str] = []
+    offsets: list[int] = []
+    for index, character in enumerate(unicodedata.normalize("NFKC", value)):
+        if _COMPACT_CHARACTER.fullmatch(character):
+            compact.append(character.casefold())
+            offsets.append(index)
+    return "".join(compact), offsets
+
+
+def _project_layout_span(
+    source_text: str,
+    content: str,
+    *,
+    compact_source: str,
+    source_offsets: list[int],
+    compact_start: int,
+) -> tuple[int | None, int | None, int]:
+    compact_content, _ = _compact_with_offsets(content)
+    if len(compact_content) < 4:
+        return None, None, compact_start
+    position = compact_source.find(compact_content, compact_start)
+    if position < 0:
+        position = compact_source.find(compact_content)
+    if position < 0:
+        return None, None, compact_start
+    start = source_offsets[position]
+    end = source_offsets[position + len(compact_content) - 1] + 1
+    while start > 0 and source_text[start - 1] not in "\r\n":
+        start -= 1
+    while end < len(source_text) and source_text[end] not in "\r\n":
+        end += 1
+    return start, end, position + len(compact_content)
+
+
+def chunk_layout_document(
+    raw: bytes,
+    *,
+    filename: str,
+    source_text: str,
+    chunk_size: int,
+) -> list[DocumentChunk]:
+    """使用 Docling HybridChunker 按版面层级、列表与表格边界切分。"""
+
+    from docling.chunking import HybridChunker
+    from docling.datamodel.base_models import DocumentStream
+    from docling.exceptions import BaseError as DoclingError
+    from docling_core.transforms.chunker.tokenizer.openai import OpenAITokenizer
+    from docling_core.types.doc import DocItemLabel
+
+    try:
+        with _CONVERTER_LOCK:
+            conversion = _document_converter().convert(
+                DocumentStream(name=filename, stream=BytesIO(raw))
+            )
+    except DoclingError as exc:
+        raise ValueError(f"文档版面解析失败: {exc}") from exc
+    chunker = HybridChunker(
+        tokenizer=OpenAITokenizer(tokenizer=_tokenizer(), max_tokens=chunk_size),
+        serializer_provider=_MarkdownSerializerProvider(),
+        merge_peers=True,
+        repeat_table_header=True,
+    )
+    compact_source, source_offsets = _compact_with_offsets(source_text)
+    compact_start = 0
+    result: list[DocumentChunk] = []
+    excluded = {
+        DocItemLabel.DOCUMENT_INDEX,
+        DocItemLabel.PAGE_HEADER,
+        DocItemLabel.PAGE_FOOTER,
+    }
+    for raw_chunk in chunker.chunk(conversion.document):
+        doc_items = tuple(raw_chunk.meta.doc_items or ())
+        if doc_items and all(item.label in excluded for item in doc_items):
+            continue
+        content = _clean_layout_text(raw_chunk.text)
+        if not content:
+            continue
+        contextualized = _clean_layout_text(chunker.contextualize(raw_chunk)) or content
+        start, end, compact_start = _project_layout_span(
+            source_text,
+            content,
+            compact_source=compact_source,
+            source_offsets=source_offsets,
+            compact_start=compact_start,
+        )
+        original = source_text[start:end] if start is not None and end is not None else content
+        pages: set[int] = set()
+        refs: list[str] = []
+        bboxes: list[dict[str, Any]] = []
+        for item in doc_items:
+            refs.append(str(item.self_ref))
+            for provenance in item.prov or ():
+                pages.add(int(provenance.page_no))
+                bbox = provenance.bbox
+                bboxes.append(
+                    {
+                        "page": int(provenance.page_no),
+                        "left": float(bbox.l),
+                        "top": float(bbox.t),
+                        "right": float(bbox.r),
+                        "bottom": float(bbox.b),
+                        "origin": str(bbox.coord_origin.value),
+                    }
+                )
+        result.append(
+            DocumentChunk(
+                original_content=original,
+                contextualized_content=contextualized,
+                source_start=start,
+                source_end=end,
+                source_start_line=(source_text.count("\n", 0, start) + 1 if start is not None else None),
+                source_end_line=(
+                    source_text.count("\n", 0, max(start or 0, (end or 1) - 1)) + 1
+                    if end is not None
+                    else None
+                ),
+                token_count=len(_tokenizer().encode(contextualized)),
+                heading_path=tuple(str(item) for item in (raw_chunk.meta.headings or ())),
+                source_pages=tuple(sorted(pages)),
+                doc_item_refs=tuple(refs),
+                source_bboxes=tuple(bboxes),
+            )
+        )
+    return result
+
+
+def merge_short_chunks(
+    chunks: list[DocumentChunk],
+    *,
+    source_text: str,
+    min_token_count: int,
+    max_token_count: int,
+) -> list[DocumentChunk]:
+    """在不突破长度上限的前提下，把过短块并入相邻内容。"""
+
+    result: list[DocumentChunk] = []
+    index = 0
+    while index < len(chunks):
+        current = chunks[index]
+        if current.token_count >= min_token_count:
+            result.append(current)
+            index += 1
+            continue
+        if index + 1 < len(chunks):
+            combined = _combine_chunks(current, chunks[index + 1], source_text)
+            if combined.token_count <= max_token_count:
+                result.append(combined)
+                index += 2
+                continue
+        if result:
+            combined = _combine_chunks(result[-1], current, source_text)
+            if combined.token_count <= max_token_count:
+                result[-1] = combined
+                index += 1
+                continue
+        result.append(current)
+        index += 1
+    return result
+
+
+def _combine_chunks(
+    left: DocumentChunk,
+    right: DocumentChunk,
+    source_text: str,
+) -> DocumentChunk:
+    contextualized = "\n\n".join(
+        part for part in (left.contextualized_content, right.contextualized_content) if part
+    )
+    start = left.source_start
+    end = right.source_end
+    has_contiguous_source = (
+        start is not None
+        and left.source_end is not None
+        and right.source_start is not None
+        and end is not None
+        and left.source_end <= right.source_start
+    )
+    original = (
+        source_text[start:end]
+        if has_contiguous_source and start is not None and end is not None
+        else "\n\n".join(
+            part for part in (left.original_content, right.original_content) if part
+        )
+    )
+    if not has_contiguous_source:
+        start = None
+        end = None
+    return DocumentChunk(
+        original_content=original,
+        contextualized_content=contextualized,
+        source_start=start,
+        source_end=end,
+        source_start_line=left.source_start_line if start is not None else None,
+        source_end_line=right.source_end_line if end is not None else None,
+        token_count=len(_tokenizer().encode(contextualized)),
+        heading_path=left.heading_path or right.heading_path,
+        source_pages=tuple(sorted(set(left.source_pages) | set(right.source_pages))),
+        doc_item_refs=left.doc_item_refs + right.doc_item_refs,
+        source_bboxes=left.source_bboxes + right.source_bboxes,
+    )

backend/app/modules/data_process/generation.py

@@ -0,0 +1,622 @@
+"""数据处理任务的大模型生成适配器。"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import logging
+import re
+from collections.abc import Callable, Iterable, Mapping
+from typing import Any
+from urllib.parse import urlsplit, urlunsplit
+
+import httpx
+
+from app.modules.data_process.algorithms import normalize_text, stable_split_assignments
+from app.modules.data_process.constants import (
+    MAX_QA_PAIRS_PER_ITEM,
+    MODEL_GENERATION_BATCH_SIZE,
+)
+
+
+class ModelGenerationError(ValueError):
+    """模型配置、响应或调用失败。"""
+
+
+class _TerminalModelGenerationError(ModelGenerationError):
+    """使用相同参数重试也无法恢复的模型响应错误。"""
+
+
+OUTPUT_TYPE_STANDARD = "standard"
+OUTPUT_TYPE_REASONING = "reasoning"
+OUTPUT_TYPE_DPO = "dpo"
+SUPPORTED_OUTPUT_TYPES = {
+    OUTPUT_TYPE_STANDARD,
+    OUTPUT_TYPE_REASONING,
+    OUTPUT_TYPE_DPO,
+}
+REASONING_DETAIL_NORMAL = "normal"
+REASONING_DETAIL_DETAILED = "detailed"
+SUPPORTED_REASONING_DETAILS = {
+    REASONING_DETAIL_NORMAL,
+    REASONING_DETAIL_DETAILED,
+}
+MINIMAX_M3_API_HOSTS = {"api.minimax.io", "api.minimaxi.com"}
+MINIMAX_M3_MIN_COMPLETION_TOKENS = 4096
+logger = logging.getLogger(__name__)
+
+
+def _is_retryable_generation_error(exc: Exception) -> bool:
+    if isinstance(exc, _TerminalModelGenerationError):
+        return False
+    if isinstance(exc, httpx.HTTPStatusError):
+        status_code = exc.response.status_code
+        return status_code in {408, 425, 429} or status_code >= 500
+    if isinstance(exc, httpx.RequestError):
+        return True
+    return isinstance(exc, (json.JSONDecodeError, ModelGenerationError))
+
+
+def _is_official_minimax_m3(endpoint: str, model_name: str) -> bool:
+    host = (urlsplit(endpoint).hostname or "").casefold()
+    return host in MINIMAX_M3_API_HOSTS and model_name.casefold() == "minimax-m3"
+
+
+def chat_completions_url(value: str) -> str:
+    """把域名、基础 URL 或完整地址统一为 chat completions 地址。"""
+
+    raw = (value or "").strip()
+    if not raw:
+        raise ModelGenerationError("generation model api_url is required")
+    if "://" not in raw:
+        raw = f"https://{raw}"
+    parsed = urlsplit(raw)
+    if parsed.scheme not in {"http", "https"} or not parsed.hostname:
+        raise ModelGenerationError("generation model api_url must be an HTTP(S) host or URL")
+    if parsed.username or parsed.password:
+        raise ModelGenerationError("generation model api_url must not contain credentials")
+
+    path = parsed.path.rstrip("/")
+    if path.endswith("/chat/completions"):
+        target_path = path
+    elif path.endswith("/v1"):
+        target_path = f"{path}/chat/completions"
+    elif not path:
+        target_path = "/v1/chat/completions"
+    else:
+        target_path = f"{path}/v1/chat/completions"
+    return urlunsplit((parsed.scheme, parsed.netloc, target_path, "", ""))
+
+
+def _response_choice(payload: Mapping[str, Any]) -> Mapping[str, Any]:
+    try:
+        choice = payload["choices"][0]
+    except (KeyError, IndexError, TypeError) as exc:
+        raise ModelGenerationError("模型响应缺少 choices[0]") from exc
+    if not isinstance(choice, Mapping):
+        raise ModelGenerationError("模型响应 choices[0] 不是对象")
+    return choice
+
+
+def _response_finish_reason(payload: Mapping[str, Any]) -> str:
+    try:
+        return str(_response_choice(payload).get("finish_reason") or "").strip().lower()
+    except ModelGenerationError:
+        return ""
+
+
+def _response_content_length(payload: Mapping[str, Any]) -> int:
+    try:
+        message = _response_choice(payload).get("message")
+        if not isinstance(message, Mapping):
+            return 0
+        content = message.get("content")
+        if isinstance(content, str):
+            return len(content)
+        if isinstance(content, list):
+            return sum(
+                len(str(item.get("text") or ""))
+                for item in content
+                if isinstance(item, Mapping)
+            )
+    except ModelGenerationError:
+        pass
+    return 0
+
+
+def _raise_for_terminal_response(payload: Mapping[str, Any]) -> Mapping[str, Any]:
+    choice = _response_choice(payload)
+    base_response = payload.get("base_resp")
+    status_code: Any = None
+    status_message = ""
+    if isinstance(base_response, Mapping):
+        status_code = base_response.get("status_code")
+        status_message = re.sub(
+            r"\s+", " ", str(base_response.get("status_msg") or "")
+        ).strip()[:200]
+
+    if bool(payload.get("input_sensitive")) or status_code in {1026, "1026"}:
+        raise _TerminalModelGenerationError(
+            f"模型输入触发内容安全拦截（code={status_code or 1026}）"
+        )
+    if bool(payload.get("output_sensitive")) or status_code in {1027, "1027"}:
+        raise _TerminalModelGenerationError(
+            f"模型输出触发内容安全拦截（code={status_code or 1027}）"
+        )
+
+    finish_reason = str(choice.get("finish_reason") or "").strip().lower()
+    if finish_reason == "length":
+        raise _TerminalModelGenerationError(
+            "模型输出因达到 Token 上限被截断（finish_reason=length），"
+            "请提高最大输出长度后重试"
+        )
+    if finish_reason == "content_filter":
+        raise _TerminalModelGenerationError(
+            "模型输出被内容安全策略拦截（finish_reason=content_filter）"
+        )
+    if finish_reason in {"tool_calls", "function_call"}:
+        raise _TerminalModelGenerationError(
+            f"模型返回了当前生成任务不支持的工具调用（finish_reason={finish_reason}）"
+        )
+    if status_code not in {None, "", 0, "0"}:
+        detail = f"：{status_message}" if status_message else ""
+        raise _TerminalModelGenerationError(
+            f"模型服务返回业务错误（code={status_code}）{detail}"
+        )
+    return choice
+
+
+def _message_content(payload: Mapping[str, Any]) -> str:
+    choice = _raise_for_terminal_response(payload)
+    message = choice.get("message")
+    if not isinstance(message, Mapping):
+        raise ModelGenerationError("模型响应缺少 choices[0].message")
+    content = message.get("content")
+    if isinstance(content, str):
+        result = content
+    elif isinstance(content, list):
+        parts = [
+            str(item.get("text") or "")
+            for item in content
+            if isinstance(item, Mapping) and item.get("type") in {None, "text", "output_text"}
+        ]
+        result = "".join(parts)
+    elif content is None:
+        result = ""
+    else:
+        raise ModelGenerationError("模型响应 content 必须是文本")
+    if not result.strip():
+        raise ModelGenerationError("模型返回的最终内容为空，未生成可解析的 JSON")
+    return result
+
+
+def _json_documents(content: str) -> list[Any]:
+    decoder = json.JSONDecoder()
+    documents: list[Any] = []
+    cursor = 0
+    while cursor < len(content):
+        match = re.search(r"[\[{]", content[cursor:])
+        if not match:
+            break
+        start = cursor + match.start()
+        try:
+            value, end = decoder.raw_decode(content[start:])
+        except json.JSONDecodeError:
+            cursor = start + 1
+            continue
+        if isinstance(value, (Mapping, list)):
+            documents.append(value)
+        cursor = start + max(end, 1)
+    return documents
+
+
+def _json_payload(content: str) -> Any:
+    # 只移除模型在 JSON 之前自行输出的思考过程，不能破坏 JSON 字段中的训练内容。
+    cleaned = content.strip()
+    if re.match(r"^\s*<think>", cleaned, flags=re.IGNORECASE) and not re.match(
+        r"^\s*<think>[\s\S]*?</think>", cleaned, flags=re.IGNORECASE
+    ):
+        raise ModelGenerationError("模型思考内容未闭合，响应可能已被截断")
+    cleaned = re.sub(
+        r"^\s*(?:<think>[\s\S]*?</think>\s*)+",
+        "",
+        cleaned,
+        count=1,
+        flags=re.IGNORECASE,
+    ).strip()
+    fenced = re.fullmatch(r"```(?:json)?\s*([\s\S]*?)\s*```", cleaned, flags=re.IGNORECASE)
+    if fenced:
+        cleaned = fenced.group(1).strip()
+    try:
+        return json.loads(cleaned)
+    except json.JSONDecodeError as direct_error:
+        documents = _json_documents(cleaned)
+        if len(documents) == 1:
+            return documents[0]
+        if len(documents) > 1:
+            raise ModelGenerationError("模型响应包含多个 JSON 对象，无法确定应使用哪一个")
+        raise ModelGenerationError(
+            "模型响应中没有找到唯一且完整的 JSON 对象"
+            f"（第 {direct_error.lineno} 行，第 {direct_error.colno} 列）"
+        ) from direct_error
+
+
+def _result_items(payload: Any) -> list[Mapping[str, Any]]:
+    if isinstance(payload, list):
+        values = payload
+    elif isinstance(payload, Mapping):
+        nested = next(
+            (
+                payload[key]
+                for key in ("items", "results", "data", "records")
+                if isinstance(payload.get(key), list)
+            ),
+            None,
+        )
+        values = nested if isinstance(nested, list) else [payload]
+    else:
+        raise ModelGenerationError("model JSON must be an object or array")
+    items = [item for item in values if isinstance(item, Mapping)]
+    if not items:
+        raise ModelGenerationError("model JSON does not contain result objects")
+    return items
+
+
+def _prompt_messages(
+    prompt: str,
+    content: str,
+    count: int,
+    *,
+    start_index: int,
+    total_count: int,
+    output_type: str,
+    reasoning_detail: str,
+) -> list[dict[str, str]]:
+    end_index = start_index + count - 1
+    if output_type == OUTPUT_TYPE_REASONING:
+        schema = '{"items":[{"instruction":"...","input":"...","reasoning":"...","answer":"..."}]}'
+        detail_rule = (
+            "推理详细程度为“详细”：完整展开问题条件、来源依据、中间计算或推导，"
+            "并在得出答案前核对结论；每一步都必须能从来源内容中验证。"
+            if reasoning_detail == REASONING_DETAIL_DETAILED
+            else
+            "推理详细程度为“普通”：只保留得出答案所需的关键依据和必要步骤，"
+            "避免冗长复述、套话和无依据扩展。"
+        )
+        output_rule = (
+            "你正在生成用于训练推理模型的思维链数据，而不是普通问答数据。"
+            "instruction、reasoning 和 answer 均不得为空；reasoning 必须是基于来源内容、"
+            f"可核对的推理过程，answer 只写最终答案。{detail_rule}"
+            "这是思维链输出模式，即使其他提示语要求省略分析，也不得省略 reasoning。"
+            "不要自行添加 <think> 标签，系统会在保存时统一组装。"
+        )
+    elif output_type == OUTPUT_TYPE_DPO:
+        schema = (
+            '{"items":[{"instruction":"...","input":"...",'
+            '"chosen":"...","rejected":"..."}]}'
+        )
+        output_rule = (
+            "你正在生成用于直接偏好优化（DPO）的成对偏好数据。"
+            "instruction、chosen 和 rejected 均不得为空；chosen 必须是忠于来源、"
+            "准确完整的优选回答，rejected 必须是表面合理但存在明确质量缺陷的拒选回答。"
+            "两者不得相同；rejected 不得包含违法危险内容，也不得用空白、乱码或无关文本凑数。"
+            "不要输出分析过程或 <think> 标签。"
+        )
+    else:
+        schema = '{"items":[{"instruction":"...","input":"...","output":"..."}]}'
+        output_rule = (
+            "你正在生成标准监督微调问答数据。instruction 和 output 不得为空；"
+            "output 只写最终答案，禁止输出分析、推理过程或 <think> 标签。"
+        )
+    schema_instruction = (
+        f"必须只返回 JSON 对象，格式为 {schema}；items 必须包含 {count} 条。"
+        f"这是总计 {total_count} 条中的第 {start_index}-{end_index} 条，"
+        "各条必须使用不同的提问角度和表述，避免重复。"
+        f"{output_rule}不要输出 Markdown 代码围栏或 JSON 之外的说明。"
+    )
+    base_prompt = normalize_text(prompt) or "请根据来源内容生成可用于监督微调的问答数据。"
+    if "{{ content }}" in base_prompt:
+        user_prompt = base_prompt.replace("{{ content }}", content)
+        return [
+            {"role": "system", "content": schema_instruction},
+            {"role": "user", "content": user_prompt},
+        ]
+    return [
+        {"role": "system", "content": f"{base_prompt}\n{schema_instruction}"},
+        {"role": "user", "content": f"来源内容：\n{content}"},
+    ]
+
+
+def generate_model_records(
+    preview_items: Iterable[Mapping[str, Any]],
+    *,
+    model: Mapping[str, Any],
+    config: Mapping[str, Any],
+    task_id: str,
+    split: Mapping[str, int],
+    qa_pairs_per_item: int,
+    client: httpx.Client | None = None,
+    on_progress: Callable[[int, int], None] | None = None,
+) -> list[dict[str, Any]]:
+    """调用 OpenAI 兼容接口，将预览切片生成标准训练记录。
+
+    每个切片按安全批次调用模型；失败批次会产生一条可人工修复的
+    invalid 结果，已经成功的批次不会丢失。
+    """
+
+    if not 1 <= qa_pairs_per_item <= MAX_QA_PAIRS_PER_ITEM:
+        raise ModelGenerationError(f"qa_pairs_per_item must be in [1, {MAX_QA_PAIRS_PER_ITEM}]")
+    output_type = str(config.get("output_type") or OUTPUT_TYPE_STANDARD).strip().lower()
+    if output_type not in SUPPORTED_OUTPUT_TYPES:
+        raise ModelGenerationError(f"output_type must be one of {sorted(SUPPORTED_OUTPUT_TYPES)}")
+    reasoning_detail = str(
+        config.get("reasoning_detail") or REASONING_DETAIL_NORMAL
+    ).strip().lower()
+    if reasoning_detail not in SUPPORTED_REASONING_DETAILS:
+        raise ModelGenerationError(
+            f"reasoning_detail must be one of {sorted(SUPPORTED_REASONING_DETAILS)}"
+        )
+    endpoint = chat_completions_url(str(model.get("api_url") or ""))
+    model_name = str(model.get("online_model_name") or model.get("name") or "").strip()
+    if not model_name:
+        raise ModelGenerationError("generation model name is required")
+    is_minimax_m3 = _is_official_minimax_m3(endpoint, model_name)
+
+    temperature = float(config.get("temperature", 0.7))
+    max_tokens = int(config.get("max_tokens", 1024))
+    timeout = max(1.0, min(120.0, float(config.get("request_timeout_seconds", 60))))
+    retries = max(0, min(5, int(config.get("generation_retries", 2))))
+    headers = {"Content-Type": "application/json"}
+    api_key = str(model.get("api_key") or "").strip()
+    if api_key:
+        headers["Authorization"] = f"Bearer {api_key}"
+
+    owns_client = client is None
+    http_client = client or httpx.Client(timeout=timeout)
+    results: list[dict[str, Any]] = []
+    try:
+        preview_list = list(preview_items)
+        total_items = len(preview_list)
+        for item_index, item in enumerate(preview_list):
+            preview_id = str(item.get("id") or f"preview-{item_index + 1}")
+            content = normalize_text(
+                str(item.get("edited_content") or item.get("original_content") or "")
+            )
+            for batch_offset in range(0, qa_pairs_per_item, MODEL_GENERATION_BATCH_SIZE):
+                batch_count = min(
+                    MODEL_GENERATION_BATCH_SIZE,
+                    qa_pairs_per_item - batch_offset,
+                )
+                batch_start = batch_offset + 1
+                batch_end = batch_offset + batch_count
+                request_payload: dict[str, Any] = {
+                    "model": model_name,
+                    "messages": _prompt_messages(
+                        str(config.get("generation_prompt") or ""),
+                        content,
+                        batch_count,
+                        start_index=batch_start,
+                        total_count=qa_pairs_per_item,
+                        output_type=output_type,
+                        reasoning_detail=reasoning_detail,
+                    ),
+                    "temperature": temperature,
+                }
+                if is_minimax_m3:
+                    request_payload.update(
+                        reasoning_split=True,
+                        max_completion_tokens=max(
+                            max_tokens,
+                            MINIMAX_M3_MIN_COMPLETION_TOKENS,
+                        ),
+                    )
+                else:
+                    request_payload["max_tokens"] = max_tokens
+                if bool(config.get("json_mode", False)) and not is_minimax_m3:
+                    request_payload["response_format"] = {"type": "json_object"}
+
+                last_error: Exception | None = None
+                generated_items: list[Mapping[str, Any]] | None = None
+                for _ in range(retries + 1):
+                    try:
+                        response = http_client.post(
+                            endpoint,
+                            headers=headers,
+                            json=request_payload,
+                        )
+                        response.raise_for_status()
+                        body = response.json()
+                        if not isinstance(body, Mapping):
+                            raise ModelGenerationError("model response body must be a JSON object")
+                        try:
+                            candidate_items = _result_items(
+                                _json_payload(_message_content(body))
+                            )
+                        except ModelGenerationError as exc:
+                            logger.warning(
+                                "data process model response rejected task_id=%s model=%s "
+                                "finish_reason=%s response_chars=%s input_sensitive=%s "
+                                "output_sensitive=%s reason=%s",
+                                task_id,
+                                model_name,
+                                _response_finish_reason(body) or "missing",
+                                _response_content_length(body),
+                                bool(body.get("input_sensitive")),
+                                bool(body.get("output_sensitive")),
+                                str(exc),
+                            )
+                            raise
+                        if len(candidate_items) < batch_count:
+                            raise ModelGenerationError(
+                                "model response contains fewer result objects than requested: "
+                                f"expected {batch_count}, got {len(candidate_items)}"
+                            )
+                        generated_items = candidate_items
+                        break
+                    except (
+                        httpx.HTTPError,
+                        json.JSONDecodeError,
+                        ModelGenerationError,
+                    ) as exc:
+                        last_error = exc
+                        if not _is_retryable_generation_error(exc):
+                            break
+
+                if generated_items is None:
+                    error_message = str(last_error or "model generation failed")[:2000]
+                    failure_instruction = (
+                        f"模型生成失败，请人工补充（第 {batch_start}-{batch_end} 条）"
+                    )
+                    result_id = (
+                        "result_"
+                        f"{hashlib.sha256(f'{preview_id}:error:{batch_start}'.encode()).hexdigest()[:16]}"
+                    )
+                    results.append(
+                        {
+                            "id": result_id,
+                            "preview_item_id": preview_id,
+                            "instruction": failure_instruction,
+                            "input": content,
+                            "output": "",
+                            "chosen": "",
+                            "rejected": "",
+                            "original_instruction": failure_instruction,
+                            "original_input": content,
+                            "original_output": "",
+                            "original_chosen": "",
+                            "original_rejected": "",
+                            "status": "invalid",
+                            "error": error_message,
+                            "split": "train",
+                        }
+                    )
+                    continue
+
+                for batch_index, value in enumerate(generated_items[:batch_count]):
+                    variant_index = batch_offset + batch_index
+                    instruction = normalize_text(
+                        str(value.get("instruction") or value.get("question") or "")
+                    )
+                    input_text = normalize_text(
+                        str(value.get("input") or value.get("context") or "")
+                    )
+                    chosen = ""
+                    rejected = ""
+                    if output_type == OUTPUT_TYPE_REASONING:
+                        reasoning = normalize_text(
+                            re.sub(
+                                r"</?think>",
+                                "",
+                                str(value.get("reasoning") or value.get("analysis") or ""),
+                                flags=re.IGNORECASE,
+                            )
+                        )
+                        answer = normalize_text(
+                            re.sub(
+                                r"</?think>",
+                                "",
+                                str(
+                                    value.get("answer")
+                                    or value.get("final_answer")
+                                    or value.get("output")
+                                    or ""
+                                ),
+                                flags=re.IGNORECASE,
+                            )
+                        )
+                        output = (
+                            f"<think>\n{reasoning}\n</think>\n{answer}"
+                            if reasoning and answer
+                            else answer or (f"<think>\n{reasoning}\n</think>" if reasoning else "")
+                        )
+                        valid = bool(instruction and reasoning and answer)
+                        missing_error = "model result is missing instruction, reasoning or answer"
+                    elif output_type == OUTPUT_TYPE_DPO:
+                        chosen = normalize_text(str(value.get("chosen") or ""))
+                        rejected = normalize_text(str(value.get("rejected") or ""))
+                        chosen = normalize_text(
+                            re.sub(
+                                r"<think>[\s\S]*?(?:</think>|$)",
+                                "",
+                                chosen,
+                                flags=re.IGNORECASE,
+                            )
+                        )
+                        rejected = normalize_text(
+                            re.sub(
+                                r"<think>[\s\S]*?(?:</think>|$)",
+                                "",
+                                rejected,
+                                flags=re.IGNORECASE,
+                            )
+                        )
+                        output = chosen
+                        valid = bool(
+                            instruction
+                            and chosen
+                            and rejected
+                            and chosen.strip() != rejected.strip()
+                        )
+                        missing_error = (
+                            "model result is missing instruction, chosen or rejected, "
+                            "or chosen equals rejected"
+                        )
+                    else:
+                        output = normalize_text(
+                            str(
+                                value.get("output")
+                                or value.get("answer")
+                                or value.get("response")
+                                or ""
+                            )
+                        )
+                        output = normalize_text(
+                            re.sub(
+                                r"<think>[\s\S]*?(?:</think>|$)",
+                                "",
+                                output,
+                                flags=re.IGNORECASE,
+                            )
+                        )
+                        valid = bool(instruction and output)
+                        missing_error = "model result is missing instruction or output"
+                    raw_id = (
+                        f"{preview_id}:{variant_index + 1}:{instruction}:"
+                        f"{output}:{rejected}"
+                    )
+                    result_id = f"result_{hashlib.sha256(raw_id.encode()).hexdigest()[:16]}"
+                    results.append(
+                        {
+                            "id": result_id,
+                            "preview_item_id": preview_id,
+                            "instruction": instruction,
+                            "input": input_text,
+                            "output": output,
+                            "chosen": chosen,
+                            "rejected": rejected,
+                            "original_instruction": instruction,
+                            "original_input": input_text,
+                            "original_output": output,
+                            "original_chosen": chosen,
+                            "original_rejected": rejected,
+                            "status": "valid" if valid else "invalid",
+                            "error": (None if valid else missing_error),
+                            "split": "train",
+                        }
+                    )
+            if on_progress:
+                on_progress(item_index + 1, total_items)
+    finally:
+        if owns_client:
+            http_client.close()
+    assignments = stable_split_assignments(
+        [str(result["id"]) for result in results],
+        split,
+        seed=task_id,
+    )
+    for result, assignment in zip(results, assignments, strict=True):
+        result["split"] = assignment
+    return results
+
+
+__all__ = ["ModelGenerationError", "chat_completions_url", "generate_model_records"]

backend/app/modules/data_process/office_preview.py

@@ -0,0 +1,308 @@
+"""Word 与 Excel 原文件的安全、受限预览模型。
+
+预览只返回浏览器绘制所需的结构化数据，不返回或执行 Office 包中的活动内容。
+DOCX 的字符偏移与上传时的正文抽取规则保持一致，供前端定位当前切片。
+"""
+
+from __future__ import annotations
+
+import io
+import re
+from typing import Any
+
+from docx import Document
+from docx.oxml.table import CT_Tbl
+from docx.oxml.text.paragraph import CT_P
+from docx.table import Table
+from docx.text.paragraph import Paragraph
+from openpyxl import load_workbook
+
+from app.modules.data_process.algorithms import (
+    _MAX_WORKBOOK_COLUMNS,
+    _MAX_WORKBOOK_HEADER_SCAN_ROWS,
+    _infer_xlsx_header_region,
+    _normalize_spreadsheet_value,
+    _rewrite_xlsx_workbook_relationships,
+    _validate_office_archive,
+    _xlsx_sheet_merge_ranges,
+    normalize_text,
+)
+
+MAX_DOCX_PREVIEW_BLOCKS = 2_000
+MAX_XLSX_PREVIEW_ROWS = 200
+
+
+def _docx_alignment(paragraph: Paragraph) -> str:
+    value = paragraph.alignment
+    return {
+        0: "left",
+        1: "center",
+        2: "right",
+        3: "justify",
+        4: "distribute",
+        5: "justify",
+        7: "justify",
+        8: "distribute",
+        9: "distribute",
+    }.get(int(value) if value is not None else -1, "left")
+
+
+def _docx_heading_level(paragraph: Paragraph) -> int | None:
+    style = paragraph.style
+    if style is None:
+        return None
+    style_name = str(style.name or "")
+    style_id = str(style.style_id or "")
+    match = re.search(r"(?:heading|标题)\s*([1-6])", f"{style_name} {style_id}", re.IGNORECASE)
+    return int(match.group(1)) if match else None
+
+
+def build_docx_preview(raw: bytes) -> dict[str, Any]:
+    """把 DOCX 转为保留标题、段落和表格顺序的浏览器预览模型。"""
+
+    _validate_office_archive(raw, "docx")
+    try:
+        document = Document(io.BytesIO(raw))
+    except Exception as exc:
+        raise ValueError(f"invalid DOCX file: {exc}") from exc
+
+    blocks: list[dict[str, Any]] = []
+    source_cursor = 0
+    has_source_content = False
+    rendered_blocks = 0
+    truncated = False
+
+    def source_range(value: str) -> tuple[str, int, int] | None:
+        nonlocal source_cursor, has_source_content
+        text = normalize_text(value)
+        if not text:
+            return None
+        if has_source_content:
+            source_cursor += 2
+        start = source_cursor
+        source_cursor += len(text)
+        has_source_content = True
+        return text, start, source_cursor
+
+    for child in document.element.body.iterchildren():
+        if rendered_blocks >= MAX_DOCX_PREVIEW_BLOCKS:
+            truncated = True
+            break
+
+        if isinstance(child, CT_P):
+            paragraph = Paragraph(child, document)
+            located = source_range(paragraph.text)
+            if located is None:
+                continue
+            text, start, end = located
+            style_name = str(paragraph.style.name or "") if paragraph.style else ""
+            blocks.append(
+                {
+                    "type": "paragraph",
+                    "text": text,
+                    "style": style_name,
+                    "heading_level": _docx_heading_level(paragraph),
+                    "alignment": _docx_alignment(paragraph),
+                    "is_list": "list" in style_name.casefold() or "列表" in style_name,
+                    "source_start": start,
+                    "source_end": end,
+                }
+            )
+            rendered_blocks += 1
+            continue
+
+        if not isinstance(child, CT_Tbl):
+            continue
+        table = Table(child, document)
+        preview_rows: list[dict[str, Any]] = []
+        for row in table.rows:
+            if rendered_blocks >= MAX_DOCX_PREVIEW_BLOCKS:
+                truncated = True
+                break
+            cell_values = [normalize_text(cell.text) for cell in row.cells]
+            located = source_range("\t".join(cell_values))
+            if located is None:
+                continue
+            _, start, end = located
+            preview_rows.append(
+                {
+                    "cells": cell_values,
+                    "source_start": start,
+                    "source_end": end,
+                }
+            )
+            rendered_blocks += 1
+        if preview_rows:
+            blocks.append({"type": "table", "rows": preview_rows})
+        if truncated:
+            break
+
+    return {
+        "format": "docx",
+        "blocks": blocks,
+        "truncated": truncated,
+    }
+
+
+def build_xlsx_preview(
+    raw: bytes,
+    *,
+    sheet_index: int = 0,
+    offset: int = 0,
+    limit: int = 100,
+) -> dict[str, Any]:
+    """按工作表分页返回 XLSX 的表头和记录网格。"""
+
+    if sheet_index < 0 or offset < 0:
+        raise ValueError("sheet_index and offset must be non-negative")
+    if limit < 1 or limit > MAX_XLSX_PREVIEW_ROWS:
+        raise ValueError(
+            f"XLSX preview limit must be in [1, {MAX_XLSX_PREVIEW_ROWS}]"
+        )
+
+    _validate_office_archive(raw, "xlsx")
+    merged_by_sheet, normalized_targets = _xlsx_sheet_merge_ranges(raw)
+    workbook_raw = (
+        _rewrite_xlsx_workbook_relationships(raw, normalized_targets)
+        if normalized_targets
+        else raw
+    )
+    try:
+        workbook = load_workbook(
+            io.BytesIO(workbook_raw),
+            read_only=True,
+            data_only=True,
+            keep_links=False,
+        )
+    except Exception as exc:
+        raise ValueError(f"invalid XLSX file: {exc}") from exc
+
+    try:
+        sheets = [
+            {
+                "index": index,
+                "name": worksheet.title,
+                "state": worksheet.sheet_state,
+            }
+            for index, worksheet in enumerate(workbook.worksheets)
+        ]
+        if not sheets:
+            raise ValueError("XLSX workbook contains no worksheets")
+        if sheet_index >= len(sheets):
+            raise ValueError("XLSX worksheet index is out of range")
+
+        worksheet = workbook.worksheets[sheet_index]
+        reset_dimensions = getattr(worksheet, "reset_dimensions", None)
+        if callable(reset_dimensions):
+            reset_dimensions()
+        row_iterator = enumerate(worksheet.iter_rows(values_only=True), start=1)
+        buffered_rows: dict[int, tuple[Any, ...]] = {}
+
+        def normalized_values(row: tuple[Any, ...]) -> list[Any]:
+            values = list(row)
+            while values and values[-1] in {None, ""}:
+                values.pop()
+            if len(values) > _MAX_WORKBOOK_COLUMNS:
+                raise ValueError(
+                    f"XLSX worksheet {worksheet.title!r} exceeds "
+                    f"{_MAX_WORKBOOK_COLUMNS} columns"
+                )
+            return values
+
+        for row_number, row in row_iterator:
+            values = normalized_values(row)
+            if not values or all(value in {None, ""} for value in values):
+                continue
+            buffered_rows[row_number] = tuple(values)
+            if len(buffered_rows) >= _MAX_WORKBOOK_HEADER_SCAN_ROWS:
+                break
+
+        if not buffered_rows:
+            return {
+                "format": "xlsx",
+                "sheets": sheets,
+                "active_sheet": {
+                    "index": sheet_index,
+                    "name": worksheet.title,
+                    "columns": [],
+                    "rows": [],
+                    "offset": offset,
+                    "limit": limit,
+                    "has_more": False,
+                },
+            }
+
+        _, header_end_row, headers = _infer_xlsx_header_region(
+            worksheet.title,
+            buffered_rows,
+            merged_by_sheet.get(worksheet.title, ()),
+        )
+        preview_rows: list[dict[str, Any]] = []
+        record_index = 0
+        has_more = False
+
+        def append_row(row_number: int, values: tuple[Any, ...] | list[Any]) -> bool:
+            nonlocal record_index, has_more
+            row_values = list(values)
+            if len(row_values) > len(headers):
+                raise ValueError(
+                    f"XLSX worksheet {worksheet.title!r} has a row wider than its header"
+                )
+            row_values.extend([None] * (len(headers) - len(row_values)))
+            record = {
+                header: _normalize_spreadsheet_value(value)
+                for header, value in zip(headers, row_values, strict=True)
+            }
+            if not any(value not in {"", None} for value in record.values()):
+                return False
+            current_index = record_index
+            record_index += 1
+            if current_index < offset:
+                return False
+            if len(preview_rows) >= limit:
+                has_more = True
+                return True
+            preview_rows.append(
+                {
+                    "row_number": row_number,
+                    "record_index": current_index,
+                    "values": [record[header] for header in headers],
+                    "record": record,
+                }
+            )
+            return False
+
+        for row_number, values in buffered_rows.items():
+            if row_number > header_end_row and append_row(row_number, values):
+                break
+        else:
+            for row_number, row in row_iterator:
+                values = normalized_values(row)
+                if not values or all(value in {None, ""} for value in values):
+                    continue
+                if append_row(row_number, values):
+                    break
+
+        return {
+            "format": "xlsx",
+            "sheets": sheets,
+            "active_sheet": {
+                "index": sheet_index,
+                "name": worksheet.title,
+                "columns": headers,
+                "rows": preview_rows,
+                "offset": offset,
+                "limit": limit,
+                "has_more": has_more,
+            },
+        }
+    finally:
+        workbook.close()
+
+
+__all__ = [
+    "MAX_DOCX_PREVIEW_BLOCKS",
+    "MAX_XLSX_PREVIEW_ROWS",
+    "build_docx_preview",
+    "build_xlsx_preview",
+]

backend/app/modules/data_process/schema_cli.py

@@ -0,0 +1,76 @@
+"""数据处理运行表的显式检查与安装命令。"""
+
+from __future__ import annotations
+
+import argparse
+from urllib.parse import urlsplit
+
+from app.modules.data_process.store import DataProcessStore
+
+REQUIRED_TASK_COLUMNS = (
+    "generation_run_id",
+    "results_confirmed",
+    "workflow_step",
+    "preview_status",
+    "preview_progress",
+    "preview_run_id",
+    "preview_failure_reason",
+    "preview_total_files",
+    "preview_completed_files",
+)
+
+
+def _target_label(database_url: str) -> str:
+    parsed = urlsplit(database_url)
+    database = parsed.path.strip("/") or "(unknown)"
+    return f"{parsed.hostname or '(unknown)'}:{parsed.port or 5432}/{database}"
+
+
+def _schema_ready(store: DataProcessStore) -> bool:
+    with store.connect() as conn:
+        row = conn.execute(
+            """
+            SELECT COUNT(*) = %s AS ready
+            FROM information_schema.columns
+            WHERE table_schema=current_schema()
+              AND table_name='data_process_tasks'
+              AND column_name = ANY(%s)
+            """,
+            (len(REQUIRED_TASK_COLUMNS), list(REQUIRED_TASK_COLUMNS)),
+        ).fetchone()
+    return bool(row and row["ready"])
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(
+        description="检查或显式安装数据处理运行表（不会由应用启动自动执行）"
+    )
+    action = parser.add_mutually_exclusive_group(required=True)
+    action.add_argument("--check", action="store_true", help="只读检查迁移是否已安装")
+    action.add_argument("--apply", action="store_true", help="执行 002 数据处理迁移")
+    parser.add_argument(
+        "--yes",
+        action="store_true",
+        help="确认允许修改 DATABASE_URL 指向的数据库；与 --apply 同时使用",
+    )
+    args = parser.parse_args()
+
+    store = DataProcessStore()
+    target = _target_label(store.database_url)
+    if args.check:
+        ready = _schema_ready(store)
+        print(f"数据处理 schema：{'已安装' if ready else '未安装'}；目标：{target}")
+        return 0 if ready else 1
+    if not args.yes:
+        parser.error("--apply 必须同时提供 --yes，确认修改目标数据库")
+
+    print(f"正在安装数据处理 schema；目标：{target}")
+    store.ensure_schema()
+    if not _schema_ready(store):
+        raise RuntimeError("迁移执行后仍未检测到 generation_run_id")
+    print("数据处理 schema 安装完成")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

backend/app/modules/data_process/storage.py

@@ -0,0 +1,564 @@
+"""数据处理原始源文件的受控本地对象存储。"""
+
+from __future__ import annotations
+
+import os
+import re
+import stat
+import unicodedata
+import uuid
+from dataclasses import dataclass
+from functools import lru_cache
+from pathlib import Path, PurePosixPath
+from typing import Iterable, Iterator
+from urllib.parse import quote, unquote, urlsplit
+
+
+class DataProcessStorageError(ValueError):
+    """本地对象引用或文件系统状态不安全。"""
+
+
+@dataclass(frozen=True, slots=True)
+class StagedSourceObject:
+    """尚未发布的原始文件；绝对路径仅在存储模块内部流转。"""
+
+    reference: str
+    _temporary_path: Path
+    _relative_path: PurePosixPath
+
+
+def _default_storage_root() -> Path:
+    return Path(__file__).resolve().parents[3] / "storage" / "data-process"
+
+
+def _configured_storage_root() -> Path:
+    configured = os.getenv("DATA_PROCESS_STORAGE_DIR", "").strip()
+    if not configured:
+        return _default_storage_root()
+    path = Path(configured).expanduser()
+    # 相对配置固定以 backend 目录为基准，
+    # 避免从不同 cwd 启动时写入不同位置。
+    return path if path.is_absolute() else Path(__file__).resolve().parents[3] / path
+
+
+def _safe_component(value: str, label: str) -> str:
+    if not value or value in {".", ".."} or len(value) > 128:
+        raise DataProcessStorageError(f"invalid {label}")
+    if not value[0].isalnum() or any(
+        not (character.isalnum() or character in {"-", "_", "."})
+        for character in value
+    ):
+        raise DataProcessStorageError(f"invalid {label}")
+    return value
+
+
+def _safe_basename(value: str) -> str:
+    if not value or len(value.encode("utf-8")) > 255:
+        raise DataProcessStorageError("invalid source file name")
+    if value != Path(value).name or "/" in value or "\\" in value or "\x00" in value:
+        raise DataProcessStorageError("invalid source file name")
+    if value in {".", ".."} or any(
+        unicodedata.category(character).startswith("C") for character in value
+    ):
+        raise DataProcessStorageError("invalid source file name")
+    return value
+
+
+class LocalDataProcessStorage:
+    """只允许访问配置根目录下的版本化原始文件。"""
+
+    def __init__(self, root: str | os.PathLike[str] | Path | None = None) -> None:
+        configured = Path(root) if root is not None else _configured_storage_root()
+        configured = configured.expanduser()
+        if configured.exists() and configured.is_symlink():
+            raise DataProcessStorageError("data process storage root must not be a symlink")
+        configured.mkdir(parents=True, exist_ok=True, mode=0o700)
+        self._root = configured.resolve(strict=True)
+        # StagedSourceObject 本身是普通 dataclass，不能只依赖其中的路径字段判断
+        # 来源；只接受由当前存储实例实际签发的对象，
+        # 避免调用方伪造暂存路径。
+        self._issued_staged_objects: dict[Path, StagedSourceObject] = {}
+        self._ensure_directory(self._root / ".staging")
+
+    @property
+    def root(self) -> Path:
+        """仅供运维和测试检查；API 响应不得序列化该属性。"""
+
+        return self._root
+
+    def new_batch_id(self) -> str:
+        return f"batch-{uuid.uuid4().hex}"
+
+    def stage_bytes(
+        self,
+        *,
+        batch_id: str,
+        task_id: str,
+        source_file_id: str,
+        version: int,
+        name: str,
+        content: bytes,
+    ) -> StagedSourceObject:
+        batch_id = _safe_component(batch_id, "batch id")
+        task_id = _safe_component(task_id, "task id")
+        source_file_id = _safe_component(source_file_id, "source file id")
+        if isinstance(version, bool) or not isinstance(version, int) or version < 1:
+            raise DataProcessStorageError("invalid source file version")
+        basename = _safe_basename(name)
+        if not isinstance(content, bytes):
+            raise TypeError("content must be bytes")
+
+        batch_directory = self._ensure_directory(self._root / ".staging" / batch_id)
+        temporary_path = batch_directory / f"{source_file_id}-{uuid.uuid4().hex}.tmp"
+        flags = os.O_CREAT | os.O_EXCL | os.O_WRONLY
+        if hasattr(os, "O_NOFOLLOW"):
+            flags |= os.O_NOFOLLOW
+        descriptor = os.open(temporary_path, flags, 0o600)
+        try:
+            with os.fdopen(descriptor, "wb", closefd=True) as stream:
+                stream.write(content)
+                stream.flush()
+                os.fsync(stream.fileno())
+        except Exception:
+            temporary_path.unlink(missing_ok=True)
+            raise
+
+        relative_path = PurePosixPath(
+            task_id,
+            source_file_id,
+            f"v{version}",
+            basename,
+        )
+        reference = (
+            "local://data-process/"
+            f"{task_id}/{source_file_id}/v{version}/{quote(basename, safe='')}"
+        )
+        staged = StagedSourceObject(reference, temporary_path, relative_path)
+        self._issued_staged_objects[temporary_path] = staged
+        return staged
+
+    def stage_copy(
+        self,
+        *,
+        batch_id: str,
+        source_reference: str,
+        expected_source_task_id: str,
+        expected_source_file_id: str,
+        task_id: str,
+        source_file_id: str,
+        version: int,
+        name: str,
+    ) -> StagedSourceObject:
+        """为不可变源对象创建独立目录项，不把大文件重新读入内存。"""
+
+        batch_id = _safe_component(batch_id, "batch id")
+        task_id = _safe_component(task_id, "task id")
+        source_file_id = _safe_component(source_file_id, "source file id")
+        if isinstance(version, bool) or not isinstance(version, int) or version < 1:
+            raise DataProcessStorageError("invalid source file version")
+        basename = _safe_basename(name)
+        source_relative = self._relative_from_reference(source_reference)
+        if source_relative is None:
+            raise DataProcessStorageError("original source object is not available")
+        self._assert_expected_owner(
+            source_relative,
+            expected_task_id=expected_source_task_id,
+            expected_source_file_id=expected_source_file_id,
+        )
+        descriptor, source_info = self._open_read_descriptor(source_relative)
+        os.close(descriptor)
+
+        batch_directory = self._ensure_directory(self._root / ".staging" / batch_id)
+        temporary_path = batch_directory / f"{source_file_id}-{uuid.uuid4().hex}.tmp"
+        source_path = self._path_for_relative(source_relative)
+        try:
+            os.link(source_path, temporary_path, follow_symlinks=False)
+            copy_info = temporary_path.lstat()
+            if (
+                not stat.S_ISREG(copy_info.st_mode)
+                or source_info.st_dev != copy_info.st_dev
+                or source_info.st_ino != copy_info.st_ino
+            ):
+                raise DataProcessStorageError("source storage object changed while copying")
+        except Exception:
+            temporary_path.unlink(missing_ok=True)
+            raise
+
+        relative_path = PurePosixPath(
+            task_id,
+            source_file_id,
+            f"v{version}",
+            basename,
+        )
+        reference = (
+            "local://data-process/"
+            f"{task_id}/{source_file_id}/v{version}/{quote(basename, safe='')}"
+        )
+        staged = StagedSourceObject(reference, temporary_path, relative_path)
+        self._issued_staged_objects[temporary_path] = staged
+        return staged
+
+    def publish(self, objects: Iterable[StagedSourceObject]) -> None:
+        staged = list(objects)
+        published: list[StagedSourceObject] = []
+        try:
+            seen_temporary_paths: set[Path] = set()
+            for item in staged:
+                self._validate_staged_object(item, require_file=True)
+                if item._temporary_path in seen_temporary_paths:
+                    raise DataProcessStorageError("duplicate staged source object")
+                seen_temporary_paths.add(item._temporary_path)
+            for item in staged:
+                final_path = self._path_for_relative(item._relative_path)
+                self._ensure_directory(final_path.parent)
+                if final_path.exists() or final_path.is_symlink():
+                    raise DataProcessStorageError("source storage object already exists")
+                os.link(item._temporary_path, final_path, follow_symlinks=False)
+                published.append(item)
+                item._temporary_path.unlink()
+                self._fsync_directory(final_path.parent)
+        except Exception:
+            for item in reversed(published):
+                try:
+                    self.delete(item.reference)
+                except Exception:
+                    # 回滚必须尽量处理其余对象，并保留真正的发布异常。
+                    pass
+            for item in staged:
+                try:
+                    self.discard([item])
+                except Exception:
+                    pass
+            raise
+        self.discard(staged)
+
+    def discard(self, objects: Iterable[StagedSourceObject]) -> None:
+        staged = list(objects)
+        for item in staged:
+            self._validate_staged_object(item, require_file=False)
+
+        batch_directories: set[Path] = set()
+        first_error: Exception | None = None
+        for item in staged:
+            temporary_path = item._temporary_path
+            try:
+                temporary_path.unlink(missing_ok=True)
+            except Exception as exc:
+                if first_error is None:
+                    first_error = exc
+            else:
+                self._issued_staged_objects.pop(temporary_path, None)
+            batch_directories.add(temporary_path.parent)
+        for directory in batch_directories:
+            self._remove_empty_directory(directory)
+        if first_error is not None:
+            raise first_error
+
+    def read(self, reference: str) -> bytes | None:
+        """读取 local 引用；旧 ``db://`` 对象返回 ``None`` 由数据库正文兜底。"""
+
+        relative_path = self._relative_from_reference(reference)
+        if relative_path is None:
+            return None
+        descriptor, _ = self._open_read_descriptor(relative_path)
+        with os.fdopen(descriptor, "rb", closefd=True) as stream:
+            return stream.read()
+
+    def file_size(
+        self,
+        reference: str,
+        *,
+        expected_task_id: str,
+        expected_source_file_id: str,
+    ) -> int | None:
+        """返回受控 local 对象大小；旧 ``db://`` 对象没有原始文件。"""
+
+        relative_path = self._relative_from_reference(reference)
+        if relative_path is None:
+            return None
+        self._assert_expected_owner(
+            relative_path,
+            expected_task_id=expected_task_id,
+            expected_source_file_id=expected_source_file_id,
+        )
+        descriptor, info = self._open_read_descriptor(relative_path)
+        os.close(descriptor)
+        return info.st_size
+
+    def iter_bytes(
+        self,
+        reference: str,
+        *,
+        expected_task_id: str,
+        expected_source_file_id: str,
+        expected_size: int,
+        start: int = 0,
+        length: int | None = None,
+        chunk_size: int = 256 * 1024,
+    ) -> Iterator[bytes]:
+        """按范围流式读取原始文件，避免 PDF 预览把大文件整体载入内存。"""
+
+        relative_path = self._relative_from_reference(reference)
+        if relative_path is None:
+            raise DataProcessStorageError("original source object is not available")
+        self._assert_expected_owner(
+            relative_path,
+            expected_task_id=expected_task_id,
+            expected_source_file_id=expected_source_file_id,
+        )
+        if start < 0 or expected_size < 0 or chunk_size < 1:
+            raise DataProcessStorageError("invalid source byte range")
+        descriptor, info = self._open_read_descriptor(relative_path)
+        if info.st_size != expected_size:
+            os.close(descriptor)
+            raise DataProcessStorageError("source object size does not match metadata")
+        remaining = expected_size - start if length is None else length
+        if remaining < 0 or start + remaining > expected_size:
+            os.close(descriptor)
+            raise DataProcessStorageError("invalid source byte range")
+        with os.fdopen(descriptor, "rb", closefd=True) as stream:
+            stream.seek(start)
+            while remaining:
+                chunk = stream.read(min(chunk_size, remaining))
+                if not chunk:
+                    raise DataProcessStorageError("source object ended unexpectedly")
+                remaining -= len(chunk)
+                yield chunk
+
+    def validate_owner(
+        self,
+        reference: str,
+        *,
+        expected_task_id: str,
+        expected_source_file_id: str,
+    ) -> bool:
+        """校验 local 引用归属；旧 ``db://`` 引用无需文件系统处理。"""
+
+        relative_path = self._relative_from_reference(reference)
+        if relative_path is None:
+            return False
+        self._assert_expected_owner(
+            relative_path,
+            expected_task_id=expected_task_id,
+            expected_source_file_id=expected_source_file_id,
+        )
+        return True
+
+    def _open_read_descriptor(
+        self,
+        relative_path: PurePosixPath,
+    ) -> tuple[int, os.stat_result]:
+        path = self._path_for_relative(relative_path)
+        self._assert_controlled_parent(path)
+        try:
+            before_open = path.lstat()
+        except FileNotFoundError as exc:
+            raise DataProcessStorageError("source storage object does not exist") from exc
+        if stat.S_ISLNK(before_open.st_mode) or not stat.S_ISREG(before_open.st_mode):
+            raise DataProcessStorageError("source storage object is not a regular file")
+        flags = os.O_RDONLY
+        if hasattr(os, "O_NOFOLLOW"):
+            flags |= os.O_NOFOLLOW
+        descriptor = os.open(path, flags)
+        after_open = os.fstat(descriptor)
+        if (
+            not stat.S_ISREG(after_open.st_mode)
+            or before_open.st_dev != after_open.st_dev
+            or before_open.st_ino != after_open.st_ino
+        ):
+            os.close(descriptor)
+            raise DataProcessStorageError("source storage object changed while opening")
+        return descriptor, after_open
+
+    def delete(
+        self,
+        reference: str,
+        *,
+        expected_task_id: str | None = None,
+        expected_source_file_id: str | None = None,
+    ) -> bool:
+        """删除受控 local 对象；旧 ``db://`` 引用保持不变。"""
+
+        relative_path = self._relative_from_reference(reference)
+        if relative_path is None:
+            return False
+        if (expected_task_id is None) != (expected_source_file_id is None):
+            raise DataProcessStorageError("both expected storage owner fields are required")
+        if expected_task_id is not None and expected_source_file_id is not None:
+            self._assert_expected_owner(
+                relative_path,
+                expected_task_id=expected_task_id,
+                expected_source_file_id=expected_source_file_id,
+            )
+        path = self._path_for_relative(relative_path)
+        self._assert_controlled_parent(path)
+        try:
+            info = path.lstat()
+        except FileNotFoundError:
+            return False
+        if stat.S_ISLNK(info.st_mode) or not stat.S_ISREG(info.st_mode):
+            raise DataProcessStorageError("refusing to delete a non-regular storage object")
+        path.unlink()
+        self._fsync_directory(path.parent)
+        for directory in (path.parent, path.parent.parent, path.parent.parent.parent):
+            self._remove_empty_directory(directory)
+        return True
+
+    @staticmethod
+    def _assert_expected_owner(
+        relative_path: PurePosixPath,
+        *,
+        expected_task_id: str,
+        expected_source_file_id: str,
+    ) -> None:
+        task_id = _safe_component(expected_task_id, "expected task id")
+        source_file_id = _safe_component(
+            expected_source_file_id,
+            "expected source file id",
+        )
+        if relative_path.parts[:2] != (task_id, source_file_id):
+            raise DataProcessStorageError("source storage object owner mismatch")
+
+    def _relative_from_reference(self, reference: str) -> PurePosixPath | None:
+        if reference.startswith("db://"):
+            return None
+        parsed = urlsplit(reference)
+        if parsed.scheme != "local" or parsed.netloc != "data-process":
+            raise DataProcessStorageError("unsupported source storage reference")
+        if parsed.query or parsed.fragment or "\\" in parsed.path:
+            raise DataProcessStorageError("unsafe source storage reference")
+        raw_parts = parsed.path.lstrip("/").split("/")
+        if len(raw_parts) != 4:
+            raise DataProcessStorageError("unsafe source storage reference")
+        if any(re.search(r"%(?![0-9A-Fa-f]{2})", part) for part in raw_parts):
+            raise DataProcessStorageError("unsafe source storage reference")
+        try:
+            decoded = [unquote(part, encoding="utf-8", errors="strict") for part in raw_parts]
+        except UnicodeDecodeError as exc:
+            raise DataProcessStorageError("unsafe source storage reference") from exc
+        if any("/" in part or "\\" in part for part in decoded):
+            raise DataProcessStorageError("unsafe source storage reference")
+        canonical_parts = [
+            quote(decoded[0], safe="-_."),
+            quote(decoded[1], safe="-_."),
+            quote(decoded[2], safe="-_."),
+            quote(decoded[3], safe=""),
+        ]
+        if canonical_parts != raw_parts:
+            raise DataProcessStorageError("source storage reference is not canonical")
+        task_id = _safe_component(decoded[0], "task id")
+        source_file_id = _safe_component(decoded[1], "source file id")
+        version_component = decoded[2]
+        if not version_component.startswith("v") or not version_component[1:].isdigit():
+            raise DataProcessStorageError("invalid source file version")
+        version = int(version_component[1:])
+        if version < 1:
+            raise DataProcessStorageError("invalid source file version")
+        basename = _safe_basename(decoded[3])
+        return PurePosixPath(task_id, source_file_id, f"v{version}", basename)
+
+    def _path_for_relative(self, relative_path: PurePosixPath) -> Path:
+        if relative_path.is_absolute() or any(
+            part in {"", ".", ".."} for part in relative_path.parts
+        ):
+            raise DataProcessStorageError("storage path escapes the configured root")
+        path = self._root.joinpath(*relative_path.parts)
+        self._assert_controlled_parent(path)
+        return path
+
+    def _validate_staged_object(
+        self,
+        item: StagedSourceObject,
+        *,
+        require_file: bool,
+    ) -> None:
+        if not isinstance(item, StagedSourceObject):
+            raise DataProcessStorageError("invalid staged source object")
+        if self._issued_staged_objects.get(item._temporary_path) is not item:
+            raise DataProcessStorageError("staged source object was not issued by this storage")
+        expected_relative = self._relative_from_reference(item.reference)
+        if expected_relative is None or expected_relative != item._relative_path:
+            raise DataProcessStorageError("staged source object reference mismatch")
+        staging_root = self._root / ".staging"
+        try:
+            relative_temporary = item._temporary_path.relative_to(staging_root)
+        except ValueError as exc:
+            raise DataProcessStorageError("staged source object escapes staging") from exc
+        if len(relative_temporary.parts) != 2:
+            raise DataProcessStorageError("invalid staged source object path")
+        _safe_component(relative_temporary.parts[0], "batch id")
+        _safe_basename(relative_temporary.parts[1])
+        self._assert_controlled_parent(item._temporary_path)
+        try:
+            info = item._temporary_path.lstat()
+        except FileNotFoundError:
+            if require_file:
+                raise DataProcessStorageError("staged source object does not exist") from None
+            return
+        if stat.S_ISLNK(info.st_mode) or not stat.S_ISREG(info.st_mode):
+            raise DataProcessStorageError("staged source object is not a regular file")
+
+    def _ensure_directory(self, directory: Path) -> Path:
+        try:
+            relative = directory.relative_to(self._root)
+        except ValueError as exc:
+            raise DataProcessStorageError("storage path escapes the configured root") from exc
+        current = self._root
+        for component in relative.parts:
+            current = current / component
+            try:
+                current.mkdir(mode=0o700)
+            except FileExistsError:
+                pass
+            info = current.lstat()
+            if stat.S_ISLNK(info.st_mode) or not stat.S_ISDIR(info.st_mode):
+                raise DataProcessStorageError("storage path contains a symlink or non-directory")
+        return directory
+
+    def _assert_controlled_parent(self, path: Path) -> None:
+        try:
+            relative_parent = path.parent.relative_to(self._root)
+        except ValueError as exc:
+            raise DataProcessStorageError("storage path escapes the configured root") from exc
+        current = self._root
+        for component in relative_parent.parts:
+            current = current / component
+            if not current.exists():
+                continue
+            info = current.lstat()
+            if stat.S_ISLNK(info.st_mode) or not stat.S_ISDIR(info.st_mode):
+                raise DataProcessStorageError("storage path contains a symlink or non-directory")
+
+    @staticmethod
+    def _fsync_directory(directory: Path) -> None:
+        # Windows 不支持以 O_RDONLY 打开目录做 fsync，跳过即可。
+        # 数据完整性在 Linux 生产环境保障，Windows 开发环境忽略。
+        if os.name == "nt":
+            return
+        descriptor = os.open(directory, os.O_RDONLY)
+        try:
+            os.fsync(descriptor)
+        finally:
+            os.close(descriptor)
+
+    def _remove_empty_directory(self, directory: Path) -> None:
+        if directory in {self._root, self._root / ".staging"}:
+            return
+        self._assert_controlled_parent(directory / "placeholder")
+        try:
+            directory.rmdir()
+        except (FileNotFoundError, OSError):
+            return
+
+
+@lru_cache
+def get_data_process_storage() -> LocalDataProcessStorage:
+    return LocalDataProcessStorage()
+
+
+__all__ = [
+    "DataProcessStorageError",
+    "LocalDataProcessStorage",
+    "StagedSourceObject",
+    "get_data_process_storage",
+]

backend/app/modules/gpu/__init__.py

@@ -0,0 +1 @@
+"""GPU assignment management module."""

backend/app/modules/gpu/router.py

@@ -0,0 +1,77 @@
+"""GPU 算力分配管理路由。"""
+from __future__ import annotations
+
+from typing import Any
+
+from fastapi import APIRouter, Body, Depends, Request
+
+from app.api.v1.endpoints.platform import ok, fail
+from app.core.auth import get_current_user, is_admin
+from app.db.platform_store import get_platform_store
+
+router = APIRouter(prefix="/compute", tags=["gpu-assignment"])
+
+
+def _actor_id(request: Request) -> str | None:
+    auth = request.headers.get("Authorization", "")
+    token = auth.replace("Bearer ", "").strip()
+    if token.startswith("platform-token-"):
+        return token[len("platform-token-"):]
+    return None
+
+
+@router.get("/gpu-assignments")
+def list_assignments(current_user: dict = Depends(get_current_user)) -> dict[str, Any]:
+    """查看全部分配关系（仅 admin）。"""
+    if not is_admin(current_user):
+        raise fail(403, "admin permission required")
+    return ok(get_platform_store().gpu_assignments())
+
+
+@router.post("/gpu-assignments")
+def assign_gpus(
+    payload: dict[str, Any] = Body(...),
+    request: Request = None,
+    current_user: dict = Depends(get_current_user),
+) -> dict[str, Any]:
+    """批量分配 GPU（仅 admin）。body: { assignments: [{ node_id, gpu_index, user_id }] }"""
+    if not is_admin(current_user):
+        raise fail(403, "admin permission required")
+    assignments = payload.get("assignments") or []
+    if not assignments:
+        raise fail(400, "assignments 不能为空")
+    actor = _actor_id(request) if request else None
+    result = get_platform_store().assign_gpus(assignments, assigned_by=actor)
+    get_platform_store().record_audit(
+        action="gpu.assign",
+        actor_id=actor,
+        target_type="gpu",
+        detail=f"count={len(assignments)}",
+    )
+    return ok(result)
+
+
+@router.delete("/gpu-assignments/{assignment_id}")
+def unassign_gpu(
+    assignment_id: str,
+    request: Request = None,
+    current_user: dict = Depends(get_current_user),
+) -> dict[str, Any]:
+    """撤销 GPU 分配（仅 admin）。"""
+    if not is_admin(current_user):
+        raise fail(403, "admin permission required")
+    get_platform_store().unassign_gpu(assignment_id)
+    actor = _actor_id(request) if request else None
+    get_platform_store().record_audit(
+        action="gpu.unassign",
+        actor_id=actor,
+        target_type="gpu",
+        target_id=assignment_id,
+    )
+    return ok({"deleted": assignment_id})
+
+
+@router.get("/my-gpus")
+def my_gpus(current_user: dict = Depends(get_current_user)) -> dict[str, Any]:
+    """查看当前用户可用的 GPU 列表。"""
+    return ok(get_platform_store().gpu_assignments_for_user(current_user["id"]))

backend/app/modules/project/router.py

@@ -0,0 +1,244 @@
+from __future__ import annotations
+
+from fastapi import APIRouter, Body, Depends, Request
+from typing import Any
+
+from app.api.v1.endpoints.platform import ok, fail
+from app.core.auth import filter_accessible_resource_ids, get_current_user, has_resource_access, is_admin
+from app.db.platform_store import get_platform_store
+
+router = APIRouter(prefix="/projects", tags=["project"])
+
+
+def _actor(request: Request) -> str | None:
+    auth = request.headers.get("Authorization", "")
+    token = auth.replace("Bearer ", "").strip()
+    return token or None
+
+
+def _require_no_pending_approval(resource_type: str, resource_id: str) -> None:
+    """第 4 周：写操作审批拦截——存在待审批实例时拒绝执行。"""
+    store = get_platform_store()
+    pending = [
+        i for i in store.approval_instances(status="pending")
+        if i["resource_type"] == resource_type and i["resource_id"] == resource_id
+    ]
+    if pending:
+        raise fail(409, "存在待审批的变更，请先完成审批")
+
+
+def _require_approval_or_admin(
+    resource_type: str,
+    resource_id: str,
+    current_user: dict[str, Any],
+    action_desc: str = "",
+) -> dict[str, Any] | None:
+    """高风险操作审批旁路：admin 直接放行，普通用户创建审批实例（code=202）。"""
+    if is_admin(current_user):
+        return None
+    store = get_platform_store()
+    instance = store.create_approval_instance({
+        "resource_type": resource_type,
+        "resource_id": resource_id,
+        "applicant_id": current_user.get("id"),
+        "template_id": None,
+    })
+    return {
+        "code": 202,
+        "message": f"操作已提交审批，等待管理员批准：{action_desc}",
+        "data": {"approval_required": True, "approval_id": instance["id"]},
+    }
+
+
+@router.get("")
+def list_projects(
+    tenant_id: str = "default",
+    status: str | None = None,
+    keyword: str | None = None,
+    current_user: dict = Depends(get_current_user),
+) -> dict[str, Any]:
+    store = get_platform_store()
+    projects = store.projects(tenant_id=tenant_id, status=status, keyword=keyword)
+    # #1 ACL 过滤：admin 直接放行，普通用户只能看到自己被授权的项目
+    accessible_ids = set(
+        filter_accessible_resource_ids("project", [p["id"] for p in projects], current_user)
+    )
+    filtered = [p for p in projects if p["id"] in accessible_ids]
+    return ok(filtered)
+
+
+@router.post("")
+def create_project(payload: dict[str, Any] = Body(...), request: Request = None) -> dict[str, Any]:
+    store = get_platform_store()
+    proj = store.create_project(payload)
+    store.record_audit(
+        action="project.create",
+        actor_id=_actor(request) if request else None,
+        target_type="project",
+        target_id=proj["id"],
+        tenant_id=proj.get("tenant_id"),
+        detail=f"name={proj.get('name')}",
+    )
+    return ok(proj)
+
+
+@router.get("/{project_id}")
+def get_project(project_id: str, current_user: dict = Depends(get_current_user)) -> dict[str, Any]:
+    # #2 访问控制：普通用户无 read 权限则拒绝
+    if not has_resource_access("project", project_id, current_user, "read"):
+        raise fail(403, "no permission to access this project")
+    try:
+        return ok(get_platform_store().project(project_id))
+    except KeyError:
+        raise fail(404, "project not found")
+
+
+@router.put("/{project_id}")
+def update_project(
+    project_id: str,
+    payload: dict[str, Any] = Body(...),
+    request: Request = None,
+    current_user: dict = Depends(get_current_user),
+) -> dict[str, Any]:
+    if not has_resource_access("project", project_id, current_user, "write"):
+        raise fail(403, "no permission to update this project")
+    store = get_platform_store()
+    try:
+        proj = store.update_project(project_id, payload)
+    except KeyError:
+        raise fail(404, "project not found")
+    store.record_audit(
+        action="project.update",
+        actor_id=_actor(request) if request else None,
+        target_type="project",
+        target_id=project_id,
+        tenant_id=proj.get("tenant_id"),
+        detail=f"fields={','.join(payload.keys())}",
+    )
+    return ok(proj)
+
+
+@router.post("/{project_id}/archive")
+def archive_project(
+    project_id: str,
+    request: Request = None,
+    current_user: dict = Depends(get_current_user),
+) -> dict[str, Any]:
+    _require_no_pending_approval("project", project_id)
+    pending = _require_approval_or_admin("project", project_id, current_user, f"归档项目 {project_id}")
+    if pending:
+        return pending
+    store = get_platform_store()
+    try:
+        proj = store.archive_project(project_id)
+    except KeyError:
+        raise fail(404, "project not found")
+    store.record_audit(
+        action="project.archive",
+        actor_id=_actor(request) if request else None,
+        target_type="project",
+        target_id=project_id,
+        tenant_id=proj.get("tenant_id"),
+    )
+    return ok(proj)
+
+
+@router.delete("/{project_id}")
+def delete_project(
+    project_id: str,
+    request: Request = None,
+    current_user: dict = Depends(get_current_user),
+) -> dict[str, Any]:
+    _require_no_pending_approval("project", project_id)
+    pending = _require_approval_or_admin("project", project_id, current_user, f"删除项目 {project_id}")
+    if pending:
+        return pending
+    store = get_platform_store()
+    store.delete_project(project_id)
+    store.record_audit(
+        action="project.delete",
+        actor_id=_actor(request) if request else None,
+        target_type="project",
+        target_id=project_id,
+    )
+    return ok(None)
+
+
+@router.get("/{project_id}/members")
+def list_members(project_id: str, current_user: dict = Depends(get_current_user)) -> dict[str, Any]:
+    if not has_resource_access("project", project_id, current_user, "read"):
+        raise fail(403, "no permission to access this project")
+    try:
+        return ok(get_platform_store().project_members(project_id))
+    except KeyError:
+        raise fail(404, "project not found")
+
+
+@router.post("/{project_id}/members")
+def add_member(
+    project_id: str,
+    payload: dict[str, Any] = Body(...),
+    request: Request = None,
+    current_user: dict = Depends(get_current_user),
+) -> dict[str, Any]:
+    if not has_resource_access("project", project_id, current_user, "write"):
+        raise fail(403, "no permission to manage members of this project")
+    store = get_platform_store()
+    try:
+        member = store.add_project_member(project_id, payload)
+    except KeyError:
+        raise fail(404, "project not found")
+    store.record_audit(
+        action="project.member.add",
+        actor_id=_actor(request) if request else None,
+        target_type="project.member",
+        target_id=project_id,
+        detail=f"user_id={payload.get('user_id')},role={payload.get('role')}",
+    )
+    return ok(member)
+
+
+@router.put("/{project_id}/members/{user_id}")
+def update_member(
+    project_id: str,
+    user_id: str,
+    payload: dict[str, Any] = Body(...),
+    request: Request = None,
+    current_user: dict = Depends(get_current_user),
+) -> dict[str, Any]:
+    if not has_resource_access("project", project_id, current_user, "write"):
+        raise fail(403, "no permission to manage members of this project")
+    store = get_platform_store()
+    try:
+        member = store.update_project_member_role(project_id, user_id, payload)
+    except KeyError:
+        raise fail(404, "project or member not found")
+    store.record_audit(
+        action="project.member.update",
+        actor_id=_actor(request) if request else None,
+        target_type="project.member",
+        target_id=project_id,
+        detail=f"user_id={user_id},role={payload.get('role')}",
+    )
+    return ok(member)
+
+
+@router.delete("/{project_id}/members/{user_id}")
+def remove_member(
+    project_id: str,
+    user_id: str,
+    request: Request = None,
+    current_user: dict = Depends(get_current_user),
+) -> dict[str, Any]:
+    if not has_resource_access("project", project_id, current_user, "write"):
+        raise fail(403, "no permission to manage members of this project")
+    store = get_platform_store()
+    store.remove_project_member(project_id, user_id)
+    store.record_audit(
+        action="project.member.remove",
+        actor_id=_actor(request) if request else None,
+        target_type="project.member",
+        target_id=project_id,
+        detail=f"user_id={user_id}",
+    )
+    return ok(None)

backend/app/modules/resource/__init__.py

@@ -0,0 +1 @@
+"""Resource access control list (ACL) module."""

backend/app/modules/resource/router.py

@@ -0,0 +1,41 @@
+from __future__ import annotations
+
+from fastapi import APIRouter, Body, Request
+from typing import Any
+
+from app.api.v1.endpoints.platform import ok, fail
+from app.db.platform_store import get_platform_store
+
+router = APIRouter(prefix="/resources", tags=["resource"])
+
+
+def _actor(request: Request) -> str | None:
+    auth = request.headers.get("Authorization", "")
+    token = auth.replace("Bearer ", "").strip()
+    return token or None
+
+
+@router.get("/{resource_type}/{resource_id}/acl")
+def get_acl(resource_type: str, resource_id: str) -> dict[str, Any]:
+    """查询资源 ACL，返回按主体分组的权限列表。"""
+    return ok(get_platform_store().resource_acl(resource_type, resource_id))
+
+
+@router.put("/{resource_type}/{resource_id}/acl")
+def set_acl(
+    resource_type: str,
+    resource_id: str,
+    payload: dict[str, Any] = Body(...),
+    request: Request = None,
+) -> dict[str, Any]:
+    """设置资源 ACL，body: { entries: [{ subject_type, subject_id, permissions: [] }] }"""
+    entries = payload.get("entries") or []
+    result = get_platform_store().set_resource_acl(resource_type, resource_id, entries)
+    get_platform_store().record_audit(
+        action="resource.acl.set",
+        actor_id=_actor(request) if request else None,
+        target_type=resource_type,
+        target_id=resource_id,
+        detail=f"entries={len(entries)}",
+    )
+    return ok(result)

backend/app/modules/retention/router.py

@@ -0,0 +1,75 @@
+from __future__ import annotations
+
+from fastapi import APIRouter, Body, Request
+from typing import Any
+
+from app.api.v1.endpoints.platform import ok, fail
+from app.db.platform_store import get_platform_store
+
+router = APIRouter(prefix="/retention-policies", tags=["retention"])
+
+
+def _actor(request: Request) -> str | None:
+    auth = request.headers.get("Authorization", "")
+    token = auth.replace("Bearer ", "").strip()
+    return token or None
+
+
+@router.get("")
+def list_policies() -> dict[str, Any]:
+    return ok(get_platform_store().retention_policies())
+
+
+@router.post("")
+def create_policy(payload: dict[str, Any] = Body(...), request: Request = None) -> dict[str, Any]:
+    if not payload.get("name"):
+        raise fail(400, "name 必填")
+    policy = get_platform_store().create_retention_policy(payload)
+    get_platform_store().record_audit(
+        action="retention.create",
+        actor_id=_actor(request) if request else None,
+        target_type="retention_policy",
+        target_id=policy["id"],
+        detail=f"name={policy.get('name')}",
+    )
+    return ok(policy)
+
+
+@router.get("/{policy_id}")
+def get_policy(policy_id: str) -> dict[str, Any]:
+    try:
+        return ok(get_platform_store().retention_policy(policy_id))
+    except KeyError:
+        raise fail(404, "retention policy not found")
+
+
+@router.put("/{policy_id}")
+def update_policy(
+    policy_id: str, payload: dict[str, Any] = Body(...), request: Request = None
+) -> dict[str, Any]:
+    store = get_platform_store()
+    try:
+        policy = store.update_retention_policy(policy_id, payload)
+    except KeyError:
+        raise fail(404, "retention policy not found")
+    store.record_audit(
+        action="retention.update",
+        actor_id=_actor(request) if request else None,
+        target_type="retention_policy",
+        target_id=policy_id,
+        detail=f"fields={','.join(payload.keys())}",
+    )
+    return ok(policy)
+
+
+@router.delete("/{policy_id}")
+def delete_policy(policy_id: str, request: Request = None) -> dict[str, Any]:
+    store = get_platform_store()
+    store.delete_retention_policy(policy_id)
+    store.record_audit(
+        action="retention.delete",
+        actor_id=_actor(request) if request else None,
+        target_type="retention_policy",
+        target_id=policy_id,
+    )
+    return ok({"deleted": policy_id})

backend/app/modules/system/router.py

@@ -0,0 +1,115 @@
+from __future__ import annotations
+
+from fastapi import APIRouter, Body, Query, Request
+from fastapi.responses import StreamingResponse
+
+from app.db.platform_store import ALL_PERMISSIONS, get_platform_store
+
+
+router = APIRouter(prefix="/system", tags=["system"])
+
+
+@router.post("/audit/visit")
+def record_visit(payload: dict = Body(...), request: Request = None) -> dict:
+    """记录用户访问业务模块的行为，用于看板用户操作分布统计。"""
+    action = str(payload.get("action") or payload.get("module") or "").strip()
+    if not action:
+        return {"code": 0, "message": "ok", "data": {"recorded": False}}
+    actor_id = ""
+    if request is not None:
+        auth = request.headers.get("Authorization", "")
+        token = auth.replace("Bearer ", "").strip()
+        if token.startswith("platform-token-"):
+            actor_id = token[len("platform-token-"):]
+    get_platform_store().record_audit(
+        action=action,
+        actor_id=actor_id or None,
+        target_type="module",
+        target_id=action,
+        detail=str(payload.get("detail") or ""),
+    )
+    return {"code": 0, "message": "ok", "data": {"recorded": True}}
+
+
+@router.get("/permissions/codes")
+def permission_codes() -> dict:
+    """返回平台权限码清单（权限码接口）。"""
+    return {"code": 0, "message": "ok", "data": {"codes": ALL_PERMISSIONS}}
+
+
+@router.get("/permissions")
+def permissions_overview() -> dict:
+    """返回权限码清单与角色定义。"""
+    store = get_platform_store()
+    return {
+        "code": 0,
+        "message": "ok",
+        "data": {"codes": ALL_PERMISSIONS, "roles": store.roles()},
+    }
+
+
+@router.get("/audit-logs")
+def audit_logs(
+    tenant_id: str | None = Query(default=None, description="租户 ID"),
+    project_id: str | None = Query(default=None, description="项目 ID"),
+    actor_id: str | None = Query(default=None, description="操作人 ID"),
+    action: str | None = Query(default=None, description="动作类型"),
+    target_type: str | None = Query(default=None, description="目标类型"),
+    start_time: str | None = Query(default=None, description="ISO8601 起始时间"),
+    end_time: str | None = Query(default=None, description="ISO8601 结束时间"),
+    limit: int = Query(default=50, ge=1, le=200),
+    offset: int = Query(default=0, ge=0),
+) -> dict:
+    """审计日志查询：按租户/项目/操作人/动作/目标类型/时间范围分页过滤。"""
+    store = get_platform_store()
+    result = store.audit_logs(
+        tenant_id=tenant_id,
+        project_id=project_id,
+        actor_id=actor_id,
+        action=action,
+        target_type=target_type,
+        start_time=start_time,
+        end_time=end_time,
+        limit=limit,
+        offset=offset,
+    )
+    return {"code": 0, "message": "ok", "data": result}
+
+
+@router.get("/audit-logs/export")
+def audit_logs_export(
+    tenant_id: str | None = Query(default=None, description="租户 ID"),
+    project_id: str | None = Query(default=None, description="项目 ID"),
+    actor_id: str | None = Query(default=None, description="操作人 ID"),
+    action: str | None = Query(default=None, description="动作类型"),
+    target_type: str | None = Query(default=None, description="目标类型"),
+    start_time: str | None = Query(default=None, description="ISO8601 起始时间"),
+    end_time: str | None = Query(default=None, description="ISO8601 结束时间"),
+) -> StreamingResponse:
+    """审计日志导出：返回 CSV 流，与应用查询相同的过滤条件。"""
+    store = get_platform_store()
+    result = store.audit_logs(
+        tenant_id=tenant_id,
+        project_id=project_id,
+        actor_id=actor_id,
+        action=action,
+        target_type=target_type,
+        start_time=start_time,
+        end_time=end_time,
+        limit=10000,
+        offset=0,
+    )
+    items = result["items"]
+    columns = ["time", "tenant_id", "project_id", "actor_id", "action", "target_type", "target_id", "detail", "client_ip"]
+    header = ",".join(columns) + "\n"
+
+    def iter_rows():
+        yield header
+        for row in items:
+            yield ",".join(f'"{str(row.get(c, "") or "")}"' for c in columns) + "\n"
+
+    return StreamingResponse(
+        iter_rows(),
+        media_type="text/csv",
+        headers={"Content-Disposition": "attachment; filename=audit_logs.csv"},
+    )

backend/app/modules/tenant/router.py

@@ -0,0 +1,116 @@
+from __future__ import annotations
+
+from fastapi import APIRouter, Body, Request
+from typing import Any
+
+from app.api.v1.endpoints.platform import ok, fail
+from app.db.platform_store import get_platform_store
+
+router = APIRouter(prefix="/tenants", tags=["tenant"])
+
+
+def _actor(request: Request) -> str | None:
+    auth = request.headers.get("Authorization", "")
+    token = auth.replace("Bearer ", "").strip()
+    return token or None
+
+
+@router.get("")
+def list_tenants() -> dict[str, Any]:
+    return ok(get_platform_store().tenants())
+
+
+@router.post("")
+def create_tenant(payload: dict[str, Any] = Body(...), request: Request = None) -> dict[str, Any]:
+    store = get_platform_store()
+    try:
+        tenant = store.create_tenant(payload)
+    except KeyError as e:
+        raise fail(400, f"missing field: {e}")
+    store.record_audit(
+        action="tenant.create",
+        actor_id=_actor(request) if request else None,
+        target_type="tenant",
+        target_id=tenant["id"],
+        tenant_id=tenant["id"],
+        detail=f"name={tenant.get('name')}",
+    )
+    return ok(tenant)
+
+
+@router.get("/{tenant_id}")
+def get_tenant(tenant_id: str) -> dict[str, Any]:
+    try:
+        return ok(get_platform_store().tenant(tenant_id))
+    except KeyError:
+        raise fail(404, "tenant not found")
+
+
+@router.put("/{tenant_id}")
+def update_tenant(tenant_id: str, payload: dict[str, Any] = Body(...), request: Request = None) -> dict[str, Any]:
+    store = get_platform_store()
+    try:
+        tenant = store.update_tenant(tenant_id, payload)
+    except KeyError:
+        raise fail(404, "tenant not found")
+    store.record_audit(
+        action="tenant.update",
+        actor_id=_actor(request) if request else None,
+        target_type="tenant",
+        target_id=tenant_id,
+        tenant_id=tenant_id,
+        detail=f"fields={','.join(payload.keys())}",
+    )
+    return ok(tenant)
+
+
+@router.put("/{tenant_id}/quota")
+def set_quota(tenant_id: str, payload: dict[str, Any] = Body(...), request: Request = None) -> dict[str, Any]:
+    store = get_platform_store()
+    try:
+        tenant = store.set_tenant_quota(tenant_id, payload.get("quota", {}))
+    except KeyError:
+        raise fail(404, "tenant not found")
+    store.record_audit(
+        action="tenant.quota.set",
+        actor_id=_actor(request) if request else None,
+        target_type="tenant",
+        target_id=tenant_id,
+        tenant_id=tenant_id,
+    )
+    return ok(tenant)
+
+
+@router.put("/{tenant_id}/retention-policy")
+def set_retention(tenant_id: str, payload: dict[str, Any] = Body(...), request: Request = None) -> dict[str, Any]:
+    store = get_platform_store()
+    try:
+        tenant = store.set_tenant_retention(tenant_id, payload.get("retention_policy_id"))
+    except KeyError:
+        raise fail(404, "tenant not found")
+    store.record_audit(
+        action="tenant.retention.set",
+        actor_id=_actor(request) if request else None,
+        target_type="tenant",
+        target_id=tenant_id,
+        tenant_id=tenant_id,
+    )
+    return ok(tenant)
+
+
+@router.delete("/{tenant_id}")
+def delete_tenant(tenant_id: str, request: Request = None) -> dict[str, Any]:
+    store = get_platform_store()
+    try:
+        tenant = store.delete_tenant(tenant_id)
+    except KeyError:
+        raise fail(404, "tenant not found")
+    store.record_audit(
+        action="tenant.delete",
+        actor_id=_actor(request) if request else None,
+        target_type="tenant",
+        target_id=tenant_id,
+        tenant_id=tenant_id,
+        detail=f"name={tenant.get('name')}",
+    )
+    return ok(tenant)

backend/app/schemas/data_process.py

@@ -0,0 +1,417 @@
+from __future__ import annotations
+
+from enum import StrEnum
+from typing import Any, Literal
+from urllib.parse import parse_qs, urlsplit
+
+from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
+
+from app.modules.data_process.constants import MAX_QA_PAIRS_PER_ITEM
+
+
+def _config_value(config: dict[str, Any], snake_name: str, camel_name: str, default: Any) -> Any:
+    if snake_name in config:
+        return config[snake_name]
+    return config.get(camel_name, default)
+
+
+def _validate_process_config(config: dict[str, Any]) -> None:
+    output_type = _config_value(config, "output_type", "outputType", "standard")
+    if output_type not in {"standard", "reasoning", "dpo"}:
+        raise ValueError("output_type must be one of: standard, reasoning, dpo")
+
+    source_mode = _config_value(config, "source_mode", "sourceMode", "local")
+    if source_mode not in {"local", "external"}:
+        raise ValueError("source_mode must be one of: local, external")
+    external_source = _config_value(config, "external_source", "externalSource", None)
+    if external_source is not None:
+        if not isinstance(external_source, dict):
+            raise ValueError("external_source must be an object")
+        if any(
+            key.lower() in {"password", "secret", "token", "api_key"}
+            for key in external_source
+        ):
+            raise ValueError("external_source must not persist credentials")
+        external_url = str(external_source.get("url") or "").strip()
+        if external_url:
+            parsed_external_url = urlsplit(external_url)
+            sensitive_query_keys = {"password", "secret", "token", "api_key", "user", "username"}
+            if parsed_external_url.username or parsed_external_url.password or (
+                set(parse_qs(parsed_external_url.query)) & sensitive_query_keys
+            ):
+                raise ValueError("external_source URL must not contain credentials")
+
+    chunk_method = _config_value(config, "chunk_method", "chunkMethod", "layout_hybrid")
+    if not isinstance(chunk_method, str) or chunk_method not in {
+        "layout_hybrid",
+        "semantic",
+        "fixed",
+    }:
+        raise ValueError("chunk_method must be one of: layout_hybrid, semantic, fixed")
+
+    semantic_percentile = _config_value(
+        config,
+        "semantic_breakpoint_percentile",
+        "semanticBreakpointPercentile",
+        95,
+    )
+    if (
+        isinstance(semantic_percentile, bool)
+        or not isinstance(semantic_percentile, int)
+        or not 1 <= semantic_percentile <= 99
+    ):
+        raise ValueError("semantic_breakpoint_percentile must be an integer in [1, 99]")
+
+    split = _config_value(config, "dataset_split", "datasetSplit", None)
+    if split is not None:
+        if not isinstance(split, dict) or set(split) != {"train", "validation", "test"}:
+            raise ValueError("dataset_split must contain train, validation and test")
+        values = list(split.values())
+        if any(isinstance(value, bool) or not isinstance(value, int) for value in values):
+            raise ValueError("dataset_split values must be integers")
+        if any(value < 0 or value > 100 for value in values) or sum(values) != 100:
+            raise ValueError("dataset_split values must be in [0, 100] and total 100")
+
+    chunk_fields = {
+        "chunk_size",
+        "chunkSize",
+        "chunk_overlap",
+        "chunkOverlap",
+        "min_chunk_size",
+        "minChunkSize",
+    }
+    if chunk_fields.intersection(config):
+        chunk_size = _config_value(config, "chunk_size", "chunkSize", 800)
+        overlap = _config_value(config, "chunk_overlap", "chunkOverlap", 100)
+        minimum = _config_value(config, "min_chunk_size", "minChunkSize", 100)
+        if any(
+            isinstance(value, bool) or not isinstance(value, int)
+            for value in (chunk_size, overlap, minimum)
+        ):
+            raise ValueError("chunk_size, chunk_overlap and min_chunk_size must be integers")
+        if not 16 <= chunk_size <= 32_768:
+            raise ValueError("chunk_size must be in [16, 32768]")
+        if overlap < 0 or overlap >= chunk_size:
+            raise ValueError("chunk_overlap must be in [0, chunk_size)")
+        if minimum <= 0 or minimum > chunk_size or overlap + minimum > chunk_size:
+            raise ValueError("min_chunk_size and chunk_overlap exceed chunk_size")
+
+    temperature = _config_value(config, "temperature", "temperature", None)
+    if temperature is not None:
+        if isinstance(temperature, bool) or not isinstance(temperature, (int, float)):
+            raise ValueError("temperature must be a number")
+        if not 0 <= float(temperature) <= 2:
+            raise ValueError("temperature must be in [0, 2]")
+
+    max_tokens = _config_value(config, "max_tokens", "maxTokens", None)
+    if max_tokens is not None:
+        if isinstance(max_tokens, bool) or not isinstance(max_tokens, int):
+            raise ValueError("max_tokens must be an integer")
+        if not 1 <= max_tokens <= 32_768:
+            raise ValueError("max_tokens must be in [1, 32768]")
+
+    for snake_name, camel_name in (
+        ("qa_pairs_per_row", "qaPairsPerRow"),
+        ("qa_pairs_per_chunk", "qaPairsPerChunk"),
+    ):
+        pairs = _config_value(config, snake_name, camel_name, None)
+        if pairs is None:
+            continue
+        if (
+            isinstance(pairs, bool)
+            or not isinstance(pairs, int)
+            or not 1 <= pairs <= MAX_QA_PAIRS_PER_ITEM
+        ):
+            raise ValueError(
+                f"{snake_name} must be an integer in [1, {MAX_QA_PAIRS_PER_ITEM}]"
+            )
+
+
+class DataProcessStatus(StrEnum):
+    pending = "pending"
+    running = "running"
+    completed = "completed"
+    failed = "failed"
+    stopped = "stopped"
+
+
+class DataProcessWorkflowStep(StrEnum):
+    create = "create"
+    model = "model"
+    upload = "upload"
+    preview = "preview"
+    generate = "generate"
+    results = "results"
+
+
+class DataProcessPreviewStatus(StrEnum):
+    idle = "idle"
+    queued = "queued"
+    running = "running"
+    completed = "completed"
+    failed = "failed"
+    cancelled = "cancelled"
+
+
+class ProcessType(StrEnum):
+    structured = "structured"
+    unstructured = "unstructured"
+    external = "external"
+
+
+class DataProcessTaskCreate(BaseModel):
+    model_config = ConfigDict(extra="forbid")
+
+    name: str = Field(min_length=1, max_length=150)
+    description: str = ""
+    process_type: ProcessType
+    source_dataset_id: str | None = None
+    config: dict[str, Any] = Field(default_factory=dict)
+
+    @field_validator("name")
+    @classmethod
+    def normalize_name(cls, value: str) -> str:
+        value = value.strip()
+        if not value:
+            raise ValueError("task name cannot be empty")
+        return value
+
+    @model_validator(mode="after")
+    def validate_config(self) -> "DataProcessTaskCreate":
+        _validate_process_config(self.config)
+        return self
+
+
+class DataProcessTaskUpdate(BaseModel):
+    model_config = ConfigDict(extra="forbid")
+
+    name: str | None = Field(default=None, min_length=1, max_length=150)
+    description: str | None = None
+    process_type: ProcessType | None = None
+    source_dataset_id: str | None = None
+    config: dict[str, Any] | None = None
+
+    @field_validator("name")
+    @classmethod
+    def normalize_name(cls, value: str | None) -> str | None:
+        if value is None:
+            return None
+        value = value.strip()
+        if not value:
+            raise ValueError("task name cannot be empty")
+        return value
+
+    @model_validator(mode="after")
+    def validate_config(self) -> "DataProcessTaskUpdate":
+        if self.config is not None:
+            _validate_process_config(self.config)
+        return self
+
+
+class DataProcessWorkflowStepUpdate(BaseModel):
+    """仅保存创建向导位置，不修改配置或使下游产物失效。"""
+
+    model_config = ConfigDict(extra="forbid")
+
+    workflow_step: DataProcessWorkflowStep
+
+
+class DataProcessRegenerateRequest(BaseModel):
+    """以一份完整配置准备任务重新生成。
+
+    ``expected_updated_at`` 用于防止详情页的旧快照覆盖其他人刚刚
+    保存的配置。重新生成不允许改变处理类型，避免旧源文件在新解析
+    规则下被静默误用。
+    """
+
+    model_config = ConfigDict(extra="forbid")
+
+    name: str = Field(min_length=1, max_length=150)
+    description: str
+    process_type: ProcessType
+    config: dict[str, Any]
+    expected_updated_at: str = Field(min_length=1)
+
+    @field_validator("name")
+    @classmethod
+    def normalize_name(cls, value: str) -> str:
+        value = value.strip()
+        if not value:
+            raise ValueError("task name cannot be empty")
+        return value
+
+    @model_validator(mode="after")
+    def validate_config(self) -> "DataProcessRegenerateRequest":
+        _validate_process_config(self.config)
+        return self
+
+
+class DataProcessRepeatRequest(BaseModel):
+    """按已确认任务的完整快照创建一批独立的新生成结果。"""
+
+    model_config = ConfigDict(extra="forbid")
+
+    expected_updated_at: str = Field(min_length=1)
+    request_id: str = Field(
+        min_length=8,
+        max_length=80,
+        pattern=r"^[A-Za-z0-9_-]+$",
+    )
+
+
+class PreviewBuildRequest(BaseModel):
+    model_config = ConfigDict(extra="forbid")
+
+    replace_existing: Literal[True] = True
+    source_file_ids: list[str] | None = None
+    source_file_id: str | None = None
+
+    @model_validator(mode="after")
+    def validate_source_file_selection(self) -> "PreviewBuildRequest":
+        if self.source_file_ids is not None and self.source_file_id is not None:
+            raise ValueError("source_file_id and source_file_ids cannot be used together")
+        values = self.source_file_ids
+        if values is None and self.source_file_id is not None:
+            values = [self.source_file_id]
+        if values is None:
+            return self
+        normalized = list(dict.fromkeys(str(value).strip() for value in values))
+        if not normalized or any(not value for value in normalized):
+            raise ValueError("at least one non-empty source file id is required")
+        self.source_file_ids = normalized
+        self.source_file_id = None
+        return self
+
+
+class PreviewItemCreate(BaseModel):
+    model_config = ConfigDict(extra="forbid")
+
+    source_file_id: str | None = None
+    original_content: str = ""
+    edited_content: str = ""
+    source_start: int | None = Field(default=None, ge=0)
+    source_end: int | None = Field(default=None, ge=0)
+    source_start_line: int | None = Field(default=None, ge=1)
+    source_end_line: int | None = Field(default=None, ge=1)
+
+    @model_validator(mode="after")
+    def validate_ranges(self) -> "PreviewItemCreate":
+        if self.source_start is not None and self.source_end is not None:
+            if self.source_end < self.source_start:
+                raise ValueError("source_end must be greater than or equal to source_start")
+        if self.source_start_line is not None and self.source_end_line is not None:
+            if self.source_end_line < self.source_start_line:
+                raise ValueError(
+                    "source_end_line must be greater than or equal to source_start_line"
+                )
+        return self
+
+
+class PreviewItemUpdate(BaseModel):
+    model_config = ConfigDict(extra="forbid")
+
+    edited_content: str
+    expected_updated_at: str | None = None
+
+
+class GenerateRequest(BaseModel):
+    model_config = ConfigDict(extra="forbid")
+
+    replace_existing: Literal[True] = True
+
+
+class ExternalSourceRequest(BaseModel):
+    model_config = ConfigDict(extra="forbid")
+
+    type: str = Field(min_length=1, max_length=30)
+    url: str = Field(min_length=1, max_length=2048)
+    auth_mode: Literal["none", "basic"] = "none"
+    username: str | None = Field(default=None, max_length=150)
+    password: str | None = Field(default=None, max_length=500)
+    limit: int = Field(default=1000, ge=1, le=100_000)
+    connect_timeout_seconds: int = Field(default=5, ge=1, le=30)
+    statement_timeout_seconds: int = Field(default=30, ge=1, le=300)
+    ssl_mode: Literal["disable", "prefer", "require", "verify-ca", "verify-full"] = "prefer"
+
+
+class ExternalPullRequest(ExternalSourceRequest):
+    query: str | None = Field(default=None, max_length=20_000)
+    file_name: str = Field(default="external-data.jsonl", min_length=1, max_length=255)
+
+    @field_validator("file_name")
+    @classmethod
+    def validate_file_name(cls, value: str) -> str:
+        name = value.strip()
+        if not name.lower().endswith((".jsonl", ".ndjson")):
+            raise ValueError("external pull file_name must end with .jsonl or .ndjson")
+        return name
+
+
+class ResultUpdate(BaseModel):
+    model_config = ConfigDict(extra="forbid")
+
+    instruction: str | None = None
+    input: str | None = None
+    output: str | None = None
+    chosen: str | None = None
+    rejected: str | None = None
+    expected_updated_at: str | None = None
+
+
+class ResultRegenerateRequest(BaseModel):
+    model_config = ConfigDict(extra="forbid")
+
+    expected_updated_at: str = Field(min_length=1, max_length=100)
+
+
+class ResultBatchRegenerateItem(BaseModel):
+    model_config = ConfigDict(extra="forbid")
+
+    result_id: str = Field(min_length=1, max_length=100)
+    expected_updated_at: str = Field(min_length=1, max_length=100)
+
+
+class ResultBatchRegenerateRequest(BaseModel):
+    model_config = ConfigDict(extra="forbid")
+
+    items: list[ResultBatchRegenerateItem] = Field(min_length=1, max_length=100)
+
+    @model_validator(mode="after")
+    def validate_unique_results(self) -> "ResultBatchRegenerateRequest":
+        result_ids = [item.result_id for item in self.items]
+        if len(result_ids) != len(set(result_ids)):
+            raise ValueError("result_id values must be unique")
+        return self
+
+
+class DatasetSplit(BaseModel):
+    model_config = ConfigDict(extra="forbid")
+
+    train: int = Field(default=80, ge=0, le=100)
+    validation: int = Field(default=10, ge=0, le=100)
+    test: int = Field(default=10, ge=0, le=100)
+
+    @model_validator(mode="after")
+    def validate_total(self) -> "DatasetSplit":
+        if self.train + self.validation + self.test != 100:
+            raise ValueError("dataset split must total 100")
+        return self
+
+
+class PublishRequest(BaseModel):
+    model_config = ConfigDict(extra="forbid")
+
+    dataset_name: str = Field(min_length=1, max_length=150)
+    dataset_type: Literal["train", "test", "eval", "val", "other"] = "train"
+    storage_type: Literal["local"] = "local"
+    split: DatasetSplit = Field(default_factory=DatasetSplit)
+    format: Literal["alpaca_jsonl", "jsonl", "dpo"] = "alpaca_jsonl"
+    description: str = ""
+
+    @field_validator("dataset_name")
+    @classmethod
+    def normalize_dataset_name(cls, value: str) -> str:
+        value = value.strip()
+        if not value:
+            raise ValueError("dataset name cannot be empty")
+        return value

backend/app/workers/compute_poller.py

@@ -0,0 +1,31 @@
+from __future__ import annotations
+
+import asyncio
+
+from app.core.config import get_settings
+from app.core.logging import get_logger
+from app.modules.compute_gateway.sync import poll_compute_jobs_once
+
+
+logger = get_logger(__name__)
+
+
+async def run_compute_poller() -> None:
+    settings = get_settings()
+    if settings.compute_mode == "simulator" or settings.compute_status_sync_mode != "polling":
+        logger.info("compute poller disabled", extra={"compute_mode": settings.compute_mode})
+        return
+
+    interval = max(3, settings.compute_poll_interval_seconds)
+    logger.info("compute poller started", extra={"interval_seconds": interval})
+    while True:
+        try:
+            result = await poll_compute_jobs_once()
+            if result["synced"] or result["failed"]:
+                logger.info("compute jobs polled", extra={"result": result})
+        except asyncio.CancelledError:
+            logger.info("compute poller stopped")
+            raise
+        except Exception as exc:  # noqa: BLE001 - keep background polling alive
+            logger.exception("compute poller failed", extra={"error": str(exc)})
+        await asyncio.sleep(interval)

backend/pyproject.toml

@@ -10,12 +10,21 @@ dependencies = [
   "pydantic>=2.7.0",
   "sqlalchemy>=2.0.30",
   "psycopg[binary]>=3.2.1",
+  "psycopg-pool>=3.2.1",
   "alembic>=1.13.1",
   "redis>=5.0.4",
   "httpx>=0.27.0",
   "PyJWT>=2.8.0",
   "passlib[bcrypt]>=1.7.4",
   "python-dotenv>=1.0.1",
+  "pypdf[crypto]>=5.0.0",
+  "python-docx>=1.1.2",
+  "openpyxl>=3.1.5",
+  "python-pptx>=1.0.2",
+  "llama-index-core==0.14.23",
+  "llama-index-embeddings-huggingface==0.6.1",
+  "docling==2.115.0",
+  "tiktoken>=0.7.0",
 ]
 
 [project.optional-dependencies]

backend/requirements.txt

@@ -4,9 +4,22 @@ python-multipart>=0.0.9
 pydantic>=2.7.0
 sqlalchemy>=2.0.30
 psycopg[binary]>=3.2.1
+psycopg-pool>=3.2.1
 alembic>=1.13.1
 redis>=5.0.4
 httpx>=0.27.0
 PyJWT>=2.8.0
 passlib[bcrypt]>=1.7.4
 python-dotenv>=1.0.1
+pypdf[crypto]>=5.0.0
+python-docx>=1.1.2
+openpyxl>=3.1.5
+python-pptx>=1.0.2
+llama-index-core==0.14.23
+llama-index-embeddings-huggingface==0.6.1
+docling==2.115.0
+tiktoken>=0.7.0
+
+# 测试与代码检查
+pytest>=8.2.0
+ruff>=0.5.0

backend/tests/test_compare_inference_async.py

@@ -0,0 +1,276 @@
+"""
+模型推理异步加载改造的单元测试。
+
+覆盖：
+- model_compare_load：异步派发，立即返回 starting + 节点信息（不等待加载完成）
+- model_compare_delete：先删记录，卸载失败也不阻塞删除
+- reconcile_inference_loads：starting -> ready/error/idle/不可达的状态迁移与封顶
+- _unload_from_compute_node：任务感知，只命中记录中的节点
+"""
+
+from __future__ import annotations
+
+import asyncio
+from types import SimpleNamespace
+from typing import Any
+
+from app.api.v1.endpoints.platform import model_compare_delete, model_compare_load
+import app.api.v1.endpoints.platform as platform
+from app.modules.compute_gateway.client import ComputeNodeClient
+from app.modules.compute_gateway.sync import MAX_STARTING_ATTEMPTS, reconcile_inference_loads
+
+
+class FakeInferenceStore:
+    """内存 store，仅实现推理加载/对账用到的接口。"""
+
+    def __init__(self, tasks: list[dict[str, Any]] | None = None, nodes: list[dict[str, Any]] | None = None) -> None:
+        self._tasks: dict[str, dict[str, Any]] = {t["id"]: dict(t) for t in (tasks or [])}
+        self._nodes = nodes or []
+        self._inference_nodes: set[str] = set()
+
+    def compare_task(self, task_id: str) -> dict[str, Any]:
+        if task_id not in self._tasks:
+            raise KeyError(task_id)
+        return dict(self._tasks[task_id])
+
+    def compare_tasks(self) -> list[dict[str, Any]]:
+        return [dict(t) for t in self._tasks.values()]
+
+    def update_compare_task(self, task_id: str, payload: dict[str, Any]) -> dict[str, Any]:
+        current = self._tasks[task_id]
+        merged = {**current, **payload, "id": task_id}
+        self._tasks[task_id] = merged
+        return dict(merged)
+
+    def delete_compare_task(self, task_id: str) -> None:
+        self._tasks.pop(task_id, None)
+
+    def compute_nodes(self) -> list[dict[str, Any]]:
+        return [dict(n) for n in self._nodes]
+
+    def model(self, model_id: str) -> dict[str, Any]:
+        raise KeyError(model_id)
+
+    def trained_models(self) -> list[dict[str, Any]]:
+        return []
+
+    def mark_inference_loaded(self, node_id: str) -> None:
+        self._inference_nodes.add(node_id)
+
+    def mark_inference_unloaded(self, node_id: str) -> None:
+        self._inference_nodes.discard(node_id)
+
+    def is_inference_loaded(self, node_id: str) -> bool:
+        return node_id in self._inference_nodes
+
+
+def _node(node_id: str, code: str = "") -> dict[str, Any]:
+    return {
+        "id": node_id,
+        "code": code or node_id,
+        "name": code or node_id,
+        "api_base_url": f"http://{code or node_id}:19100",
+        "enabled": True,
+        "scheduler_status": "online",
+    }
+
+
+def _task(task_id: str, *, node_id: str | None = None, load_status: dict[str, Any] | None = None) -> dict[str, Any]:
+    return {
+        "id": task_id,
+        "name": f"task-{task_id}",
+        "status": "pending",
+        "models": [
+            {"model_id": "m_1", "model_name": "qwen", "model_path": "/models/qwen", "node_id": node_id}
+        ],
+        "load_status": load_status or {"loaded_models": []},
+    }
+
+
+async def _fake_inference_load(self, payload: dict[str, Any]) -> dict[str, Any]:
+    return {"loaded": False, "status": "loading", "request_id": "req-1"}
+
+
+async def _fake_inference_unload(self) -> dict[str, Any]:
+    return {"unloaded": True, "status": "idle"}
+
+
+def _patch_store(monkeypatch, store: FakeInferenceStore) -> None:
+    monkeypatch.setattr(platform, "get_platform_store", lambda: store)
+    monkeypatch.setattr(platform, "get_settings", lambda: SimpleNamespace(compute_mode="real"))
+
+
+def test_select_eval_node_prefers_model_node(monkeypatch) -> None:
+    from app.api.v1.endpoints.platform import _select_eval_node
+
+    store = FakeInferenceStore(nodes=[_node("n1"), _node("n2")])
+    # 指定模型所在节点时优先返回该节点
+    assert _select_eval_node(store, "n2")["id"] == "n2"
+    # 无指定节点时回退到第一个在线节点
+    assert _select_eval_node(store, None)["id"] == "n1"
+
+
+def test_select_eval_node_returns_none_when_model_node_offline(monkeypatch) -> None:
+    from app.api.v1.endpoints.platform import _select_eval_node
+
+    nodes = [_node("n1"), _node("n2")]
+    nodes[1]["enabled"] = False
+    store = FakeInferenceStore(nodes=nodes)
+    # 模型所在节点不可用 → 明确失败，不派发到其它节点
+    assert _select_eval_node(store, "n2") is None
+    # 无指定节点时仍回退第一个在线节点
+    assert _select_eval_node(store, None)["id"] == "n1"
+
+
+def test_model_compare_load_dispatches_and_returns_starting(monkeypatch) -> None:
+    store = FakeInferenceStore(tasks=[_task("t1", node_id="n1")], nodes=[_node("n1")])
+    _patch_store(monkeypatch, store)
+    monkeypatch.setattr(ComputeNodeClient, "inference_load", _fake_inference_load)
+
+    result = asyncio.run(model_compare_load("t1"))
+    assert result["code"] == 0
+    updated = result["data"]
+    assert updated["status"] == "starting"
+    items = updated["load_status"]["loaded_models"]
+    assert items[0]["status"] == "starting"
+    assert items[0]["node_id"] == "n1"
+    assert "n1" in store._inference_nodes
+
+
+def test_model_compare_load_marks_error_when_all_nodes_fail(monkeypatch) -> None:
+    store = FakeInferenceStore(tasks=[_task("t1", node_id="n1")], nodes=[_node("n1")])
+    _patch_store(monkeypatch, store)
+
+    async def _raise(self, payload: dict[str, Any]) -> dict[str, Any]:
+        raise RuntimeError("conn refused")
+
+    monkeypatch.setattr(ComputeNodeClient, "inference_load", _raise)
+
+    result = asyncio.run(model_compare_load("t1"))
+    updated = result["data"]
+    assert updated["status"] == "failed"
+    assert updated["load_status"]["loaded_models"][0]["status"] == "error"
+    assert "conn refused" in updated["load_status"]["loaded_models"][0]["error"]
+
+
+def test_model_compare_delete_removes_record_even_if_unload_raises(monkeypatch) -> None:
+    task = _task(
+        "t1",
+        node_id="n1",
+        load_status={"loaded_models": [{"model_id": "m_1", "status": "ready", "node_id": "n1"}]},
+    )
+    store = FakeInferenceStore(tasks=[task], nodes=[_node("n1")])
+    _patch_store(monkeypatch, store)
+
+    async def _raise(self) -> dict[str, Any]:
+        raise RuntimeError("unload boom")
+
+    monkeypatch.setattr(ComputeNodeClient, "inference_unload", _raise)
+
+    result = asyncio.run(model_compare_delete("t1"))
+    assert result["data"] == {"deleted": "t1"}
+    assert "t1" not in store._tasks
+    # finally 中仍清掉了节点标记
+    assert "n1" not in store._inference_nodes
+
+
+def test_unload_from_compute_node_only_hits_recorded_node(monkeypatch) -> None:
+    task = _task(
+        "t1",
+        load_status={"loaded_models": [{"model_id": "m_1", "status": "ready", "node_id": "n1"}]},
+    )
+    store = FakeInferenceStore(tasks=[task], nodes=[_node("n1"), _node("n2")])
+    _patch_store(monkeypatch, store)
+    monkeypatch.setattr(ComputeNodeClient, "inference_unload", _fake_inference_unload)
+
+    from app.api.v1.endpoints.platform import _unload_from_compute_node
+
+    result = asyncio.run(_unload_from_compute_node(store, task=task))
+    assert result["unloaded"] is True
+    # 只命中任务记录中的节点 n1，n2 未被卸载
+    assert [r["node_id"] for r in result["nodes"]] == ["n1"]
+    assert "n1" not in store._inference_nodes
+
+
+async def _status_ready(self) -> dict[str, Any]:
+    return {"loaded": True, "status": "ready", "model_name": "qwen"}
+
+
+def test_reconcile_transitions_starting_to_ready(monkeypatch) -> None:
+    task = _task(
+        "t1",
+        node_id="n1",
+        load_status={"loaded_models": [{"model_id": "m_1", "status": "starting", "node_id": "n1"}]},
+    )
+    store = FakeInferenceStore(tasks=[task], nodes=[_node("n1")])
+    monkeypatch.setattr(ComputeNodeClient, "inference_status", _status_ready)
+
+    reconciled = asyncio.run(reconcile_inference_loads(store))
+    assert reconciled == [{"task_id": "t1", "status": "loaded"}]
+    updated = store._tasks["t1"]
+    assert updated["status"] == "loaded"
+    assert updated["load_status"]["loaded_models"][0]["status"] == "ready"
+    assert "n1" in store._inference_nodes
+
+
+def test_reconcile_transitions_to_error_and_failed(monkeypatch) -> None:
+    async def _status_error(self) -> dict[str, Any]:
+        return {"loaded": False, "status": "error", "error": "CUDA out of memory"}
+
+    task = _task(
+        "t1",
+        node_id="n1",
+        load_status={"loaded_models": [{"model_id": "m_1", "status": "starting", "node_id": "n1"}]},
+    )
+    store = FakeInferenceStore(tasks=[task], nodes=[_node("n1")])
+    monkeypatch.setattr(ComputeNodeClient, "inference_status", _status_error)
+
+    reconciled = asyncio.run(reconcile_inference_loads(store))
+    assert reconciled == [{"task_id": "t1", "status": "failed"}]
+    item = store._tasks["t1"]["load_status"]["loaded_models"][0]
+    assert item["status"] == "error"
+    assert "CUDA out of memory" in item["error"]
+    assert "n1" not in store._inference_nodes
+
+
+def test_reconcile_idle_marks_model_disappeared(monkeypatch) -> None:
+    async def _status_idle(self) -> dict[str, Any]:
+        return {"loaded": False, "status": "idle"}
+
+    task = _task(
+        "t1",
+        node_id="n1",
+        load_status={"loaded_models": [{"model_id": "m_1", "status": "starting", "node_id": "n1"}]},
+    )
+    store = FakeInferenceStore(tasks=[task], nodes=[_node("n1")])
+    monkeypatch.setattr(ComputeNodeClient, "inference_status", _status_idle)
+
+    asyncio.run(reconcile_inference_loads(store))
+    item = store._tasks["t1"]["load_status"]["loaded_models"][0]
+    assert item["status"] == "error"
+    assert "disappeared" in item["error"]
+    assert store._tasks["t1"]["status"] == "failed"
+
+
+def test_reconcile_unreachable_node_flips_to_error_after_cap(monkeypatch) -> None:
+    async def _raise(self) -> dict[str, Any]:
+        raise RuntimeError("conn refused")
+
+    task = _task(
+        "t1",
+        node_id="n1",
+        load_status={"loaded_models": [{"model_id": "m_1", "status": "starting", "node_id": "n1"}]},
+    )
+    store = FakeInferenceStore(tasks=[task], nodes=[_node("n1")])
+    monkeypatch.setattr(ComputeNodeClient, "inference_status", _raise)
+
+    # 每次轮询前重置节流时间戳，逐次推进 load_attempts 到封顶
+    for _ in range(MAX_STARTING_ATTEMPTS):
+        item = store._tasks["t1"]["load_status"]["loaded_models"][0]
+        item["last_polled_at"] = 0
+        asyncio.run(reconcile_inference_loads(store))
+
+    item = store._tasks["t1"]["load_status"]["loaded_models"][0]
+    assert item["status"] == "error"
+    assert "unreachable" in item["error"]
+    assert store._tasks["t1"]["status"] == "failed"

backend/tests/test_data_convert_security.py

@@ -0,0 +1,57 @@
+"""data_convert 模块安全回归测试：输出文件名路径穿越与鉴权。
+
+- ``output_filename`` 必须通过白名单校验，阻断 ``../``、``/``、``\\`` 及控制字符，
+  否则转换结果可被写出到存储根目录之外（任意文件读写/删除）。
+- 所有 data_convert 路由必须挂载 ``get_current_user`` 鉴权依赖。
+"""
+from __future__ import annotations
+
+import pytest
+from fastapi import HTTPException
+
+from app.core.auth import get_current_user
+from app.modules.data_convert.router import _safe_output_filename, router
+
+
+def test_safe_output_filename_defaults() -> None:
+    assert _safe_output_filename(None) == "converted-data.jsonl"
+    assert _safe_output_filename("") == "converted-data.jsonl"
+
+
+def test_safe_output_filename_valid() -> None:
+    assert _safe_output_filename("converted-data.jsonl") == "converted-data.jsonl"
+    assert _safe_output_filename("my-data.v1.jsonl") == "my-data.v1.jsonl"
+    assert _safe_output_filename("  报告.jsonl  ") == "报告.jsonl"
+
+
+@pytest.mark.parametrize(
+    "bad",
+    [
+        "../../etc/passwd",
+        "../x.jsonl",
+        "a/b.jsonl",
+        r"a\b.jsonl",
+        "a\\b.jsonl",
+        "..",
+        ".",
+        "x\x00.jsonl",
+        "x\n.jsonl",
+        "x\t.jsonl",
+    ],
+)
+def test_safe_output_filename_rejects_traversal(bad: str) -> None:
+    with pytest.raises(HTTPException):
+        _safe_output_filename(bad)
+
+
+def test_all_data_convert_routes_require_auth() -> None:
+    for route in router.routes:
+        node = getattr(route, "dependant", None)
+        assert node is not None, f"route {route.path} has no dependency graph"
+        stack = list(node.dependencies)
+        calls: list = []
+        while stack:
+            dep = stack.pop()
+            stack.extend(getattr(dep, "dependencies", []))
+            calls.append(getattr(dep, "call", None))
+        assert get_current_user in calls, f"route {route.path} is missing get_current_user auth"

backend/tests/test_data_process_generation.py

@@ -0,0 +1,965 @@
+from __future__ import annotations
+
+import json
+
+import httpx
+import pytest
+
+from app.modules.data_process.generation import (
+    ModelGenerationError,
+    chat_completions_url,
+    generate_model_records,
+)
+
+
+def test_chat_completions_url_accepts_host_base_and_complete_url() -> None:
+    assert chat_completions_url("www.caoxiaozhu.com") == (
+        "https://www.caoxiaozhu.com/v1/chat/completions"
+    )
+    assert chat_completions_url("https://model.example/v1") == (
+        "https://model.example/v1/chat/completions"
+    )
+    complete = "https://model.example/openai/v1/chat/completions"
+    assert chat_completions_url(complete) == complete
+
+
+def test_generate_model_records_uses_prompt_auth_and_stable_split() -> None:
+    requests: list[httpx.Request] = []
+    progress_updates: list[tuple[int, int]] = []
+
+    def handler(request: httpx.Request) -> httpx.Response:
+        requests.append(request)
+        payload = json.loads(request.content)
+        assert payload["model"] == "qwen-plus"
+        assert payload["response_format"] == {"type": "json_object"}
+        assert "客户反馈页面加载慢" in payload["messages"][1]["content"]
+        assert "你正在生成标准监督微调问答数据" in payload["messages"][0]["content"]
+        assert "禁止输出分析、推理过程" in payload["messages"][0]["content"]
+        return httpx.Response(
+            200,
+            json={
+                "choices": [
+                    {
+                        "message": {
+                            "content": json.dumps(
+                                {
+                                    "items": [
+                                        {
+                                            "instruction": "请生成简洁客服回复",
+                                            "input": "客户反馈页面加载慢",
+                                            "output": "已收到反馈，我们正在排查。",
+                                        }
+                                    ]
+                                },
+                                ensure_ascii=False,
+                            )
+                        }
+                    }
+                ]
+            },
+        )
+
+    client = httpx.Client(transport=httpx.MockTransport(handler))
+    records = generate_model_records(
+        [{"id": "preview-1", "edited_content": "客户反馈页面加载慢"}],
+        model={
+            "name": "Qwen",
+            "online_model_name": "qwen-plus",
+            "api_url": "model.example",
+            "api_key": "test-secret",
+        },
+        config={
+            "generation_prompt": "请处理：{{ content }}",
+            "json_mode": True,
+            "temperature": 0.2,
+            "max_tokens": 512,
+        },
+        task_id="task-1",
+        split={"train": 100, "validation": 0, "test": 0},
+        qa_pairs_per_item=1,
+        client=client,
+        on_progress=lambda processed, total: progress_updates.append((processed, total)),
+    )
+
+    assert len(records) == 1
+    assert records[0]["status"] == "valid"
+    assert records[0]["split"] == "train"
+    assert requests[0].headers["Authorization"] == "Bearer test-secret"
+    assert progress_updates == [(1, 1)]
+
+
+def test_generate_model_records_builds_native_dpo_pair() -> None:
+    def handler(request: httpx.Request) -> httpx.Response:
+        payload = json.loads(request.content)
+        system_prompt = payload["messages"][0]["content"]
+        assert '"chosen"' in system_prompt
+        assert '"rejected"' in system_prompt
+        assert "直接偏好优化" in system_prompt
+        return httpx.Response(
+            200,
+            json={
+                "choices": [{
+                    "message": {
+                        "content": json.dumps({
+                            "items": [{
+                                "instruction": "系统如何处理扫描 PDF？",
+                                "input": "",
+                                "chosen": "仅在没有文本层时调用 OCR，并保留页码。",
+                                "rejected": "所有 PDF 都重复执行 OCR。",
+                            }],
+                        }, ensure_ascii=False),
+                    },
+                }],
+            },
+        )
+
+    records = generate_model_records(
+        [{"id": "preview-dpo", "edited_content": "扫描 PDF 缺少文本层时执行 OCR。"}],
+        model={"name": "model", "api_url": "https://model.example/v1"},
+        config={"output_type": "dpo", "generation_retries": 0},
+        task_id="task-dpo",
+        split={"train": 100, "validation": 0, "test": 0},
+        qa_pairs_per_item=1,
+        client=httpx.Client(transport=httpx.MockTransport(handler)),
+    )
+
+    assert records[0]["status"] == "valid"
+    assert records[0]["chosen"] == "仅在没有文本层时调用 OCR,并保留页码。"
+    assert records[0]["rejected"] == "所有 PDF 都重复执行 OCR。"
+    assert records[0]["output"] == records[0]["chosen"]
+
+
+def test_generate_model_records_rejects_equal_dpo_pair() -> None:
+    def handler(_request: httpx.Request) -> httpx.Response:
+        return httpx.Response(
+            200,
+            json={"choices": [{"message": {"content": json.dumps({
+                "items": [{
+                    "instruction": "问题",
+                    "chosen": "相同回答",
+                    "rejected": "相同回答",
+                }],
+            }, ensure_ascii=False)}}]},
+        )
+
+    records = generate_model_records(
+        [{"id": "preview-dpo-invalid", "edited_content": "来源"}],
+        model={"name": "model", "api_url": "https://model.example/v1"},
+        config={"output_type": "dpo", "generation_retries": 0},
+        task_id="task-dpo-invalid",
+        split={"train": 100, "validation": 0, "test": 0},
+        qa_pairs_per_item=1,
+        client=httpx.Client(transport=httpx.MockTransport(handler)),
+    )
+
+    assert records[0]["status"] == "invalid"
+    assert "chosen equals rejected" in records[0]["error"]
+
+
+def test_minimax_m3_uses_split_reasoning_and_completion_token_budget() -> None:
+    requests: list[dict[str, object]] = []
+
+    def handler(request: httpx.Request) -> httpx.Response:
+        payload = json.loads(request.content)
+        requests.append(payload)
+        return httpx.Response(
+            200,
+            json={
+                "choices": [{
+                    "finish_reason": "stop",
+                    "message": {
+                        "reasoning_content": "模型内部思考不应混入业务 JSON",
+                        "content": json.dumps({
+                            "items": [{
+                                "instruction": "申请编号有什么作用？",
+                                "reasoning": "来源说明它用于标识报销申请。",
+                                "answer": "它用于唯一标识一笔报销申请。",
+                            }],
+                        }, ensure_ascii=False),
+                    },
+                }],
+                "output_sensitive": False,
+                "base_resp": {"status_code": 0, "status_msg": ""},
+            },
+        )
+
+    records = generate_model_records(
+        [{"id": "preview-minimax", "edited_content": "申请编号用于标识报销申请。"}],
+        model={
+            "name": "MiniMax",
+            "online_model_name": "MiniMax-M3",
+            "api_url": "https://api.minimaxi.com/v1",
+        },
+        config={
+            "output_type": "reasoning",
+            "json_mode": True,
+            "max_tokens": 1024,
+            "generation_retries": 0,
+        },
+        task_id="task-minimax",
+        split={"train": 100, "validation": 0, "test": 0},
+        qa_pairs_per_item=1,
+        client=httpx.Client(transport=httpx.MockTransport(handler)),
+    )
+
+    assert records[0]["status"] == "valid"
+    assert len(requests) == 1
+    assert requests[0]["reasoning_split"] is True
+    assert requests[0]["max_completion_tokens"] >= 4096
+    assert "max_tokens" not in requests[0]
+    assert "response_format" not in requests[0]
+
+
+def test_minimax_m3_keeps_larger_configured_completion_budget() -> None:
+    requests: list[dict[str, object]] = []
+
+    def handler(request: httpx.Request) -> httpx.Response:
+        requests.append(json.loads(request.content))
+        return httpx.Response(
+            200,
+            json={
+                "choices": [{
+                    "message": {
+                        "content": json.dumps({
+                            "items": [{
+                                "instruction": "问题",
+                                "output": "这是满足测试要求的完整答案。",
+                            }],
+                        }, ensure_ascii=False),
+                    },
+                }],
+            },
+        )
+
+    generate_model_records(
+        [{"id": "preview-minimax-budget", "edited_content": "来源正文"}],
+        model={
+            "online_model_name": "MiniMax-M3",
+            "api_url": "https://api.minimax.io/v1",
+        },
+        config={"max_tokens": 8192, "generation_retries": 0},
+        task_id="task-minimax-budget",
+        split={"train": 100, "validation": 0, "test": 0},
+        qa_pairs_per_item=1,
+        client=httpx.Client(transport=httpx.MockTransport(handler)),
+    )
+
+    assert requests[0]["max_completion_tokens"] == 8192
+
+
+def test_minimax_m3_name_on_custom_proxy_keeps_generic_openai_parameters() -> None:
+    requests: list[dict[str, object]] = []
+
+    def handler(request: httpx.Request) -> httpx.Response:
+        requests.append(json.loads(request.content))
+        return httpx.Response(
+            200,
+            json={
+                "choices": [{
+                    "message": {
+                        "content": json.dumps({
+                            "items": [{
+                                "instruction": "问题",
+                                "output": "这是代理服务返回的完整答案。",
+                            }],
+                        }, ensure_ascii=False),
+                    },
+                }],
+            },
+        )
+
+    generate_model_records(
+        [{"id": "preview-minimax-proxy", "edited_content": "来源正文"}],
+        model={
+            "online_model_name": "MiniMax-M3",
+            "api_url": "https://model-proxy.example/v1",
+        },
+        config={"max_tokens": 1024, "json_mode": True, "generation_retries": 0},
+        task_id="task-minimax-proxy",
+        split={"train": 100, "validation": 0, "test": 0},
+        qa_pairs_per_item=1,
+        client=httpx.Client(transport=httpx.MockTransport(handler)),
+    )
+
+    assert requests[0]["max_tokens"] == 1024
+    assert requests[0]["response_format"] == {"type": "json_object"}
+    assert "reasoning_split" not in requests[0]
+    assert "max_completion_tokens" not in requests[0]
+
+
+def test_generate_model_records_extracts_json_surrounded_by_model_explanation() -> None:
+    content = "模型结果如下：\n```json\n" + json.dumps(
+        {
+            "items": [{
+                "instruction": "字段有什么作用？",
+                "output": "该字段用于唯一标识记录。",
+            }],
+        },
+        ensure_ascii=False,
+    ) + "\n```\n生成完毕。"
+    client = httpx.Client(
+        transport=httpx.MockTransport(
+            lambda _: httpx.Response(
+                200,
+                json={"choices": [{"message": {"content": content}}]},
+            )
+        )
+    )
+
+    records = generate_model_records(
+        [{"id": "preview-explanation", "edited_content": "字段用于唯一标识记录。"}],
+        model={"name": "model", "api_url": "https://model.example/v1"},
+        config={"generation_retries": 0},
+        task_id="task-explanation",
+        split={"train": 100, "validation": 0, "test": 0},
+        qa_pairs_per_item=1,
+        client=client,
+    )
+
+    assert records[0]["status"] == "valid"
+    assert records[0]["output"] == "该字段用于唯一标识记录。"
+
+
+def test_generate_model_records_reports_token_truncation_instead_of_json_error() -> None:
+    client = httpx.Client(
+        transport=httpx.MockTransport(
+            lambda _: httpx.Response(
+                200,
+                json={
+                    "choices": [{
+                        "finish_reason": "length",
+                        "message": {"content": ""},
+                    }],
+                    "output_sensitive": False,
+                },
+            )
+        )
+    )
+
+    records = generate_model_records(
+        [{"id": "preview-truncated", "edited_content": "来源正文"}],
+        model={"name": "model", "api_url": "https://model.example/v1"},
+        config={"generation_retries": 0},
+        task_id="task-truncated",
+        split={"train": 100, "validation": 0, "test": 0},
+        qa_pairs_per_item=1,
+        client=client,
+    )
+
+    assert records[0]["status"] == "invalid"
+    assert "Token" in records[0]["error"]
+    assert "截断" in records[0]["error"]
+
+
+def test_token_truncation_is_not_retried_even_when_json_looks_complete() -> None:
+    request_count = 0
+    content = json.dumps({
+        "items": [{
+            "instruction": "问题",
+            "output": "表面完整但服务端已声明截断。",
+        }],
+    }, ensure_ascii=False)
+
+    def handler(_: httpx.Request) -> httpx.Response:
+        nonlocal request_count
+        request_count += 1
+        return httpx.Response(
+            200,
+            json={
+                "choices": [{
+                    "finish_reason": "length",
+                    "message": {"content": content},
+                }],
+            },
+        )
+
+    records = generate_model_records(
+        [{"id": "preview-length", "edited_content": "来源正文"}],
+        model={"name": "model", "api_url": "https://model.example/v1"},
+        config={"generation_retries": 5},
+        task_id="task-length",
+        split={"train": 100, "validation": 0, "test": 0},
+        qa_pairs_per_item=1,
+        client=httpx.Client(transport=httpx.MockTransport(handler)),
+    )
+
+    assert request_count == 1
+    assert records[0]["status"] == "invalid"
+    assert "finish_reason=length" in records[0]["error"]
+
+
+def test_sensitive_model_response_is_not_retried_or_saved() -> None:
+    request_count = 0
+
+    def handler(_: httpx.Request) -> httpx.Response:
+        nonlocal request_count
+        request_count += 1
+        return httpx.Response(
+            200,
+            json={
+                "choices": [{
+                    "finish_reason": "stop",
+                    "message": {"content": "{}"},
+                }],
+                "output_sensitive": True,
+                "base_resp": {"status_code": 1027, "status_msg": "output sensitive"},
+            },
+        )
+
+    records = generate_model_records(
+        [{"id": "preview-sensitive", "edited_content": "来源正文"}],
+        model={"name": "model", "api_url": "https://model.example/v1"},
+        config={"generation_retries": 5},
+        task_id="task-sensitive",
+        split={"train": 100, "validation": 0, "test": 0},
+        qa_pairs_per_item=1,
+        client=httpx.Client(transport=httpx.MockTransport(handler)),
+    )
+
+    assert request_count == 1
+    assert records[0]["status"] == "invalid"
+    assert "安全拦截" in records[0]["error"]
+    assert "1027" in records[0]["error"]
+
+
+def test_empty_model_content_can_retry_then_succeed() -> None:
+    request_count = 0
+
+    def handler(_: httpx.Request) -> httpx.Response:
+        nonlocal request_count
+        request_count += 1
+        if request_count == 1:
+            return httpx.Response(
+                200,
+                json={"choices": [{"finish_reason": "stop", "message": {"content": ""}}]},
+            )
+        return httpx.Response(
+            200,
+            json={
+                "choices": [{
+                    "finish_reason": "stop",
+                    "message": {
+                        "content": json.dumps({
+                            "items": [{
+                                "instruction": "问题",
+                                "output": "第二次请求返回了完整答案。",
+                            }],
+                        }, ensure_ascii=False),
+                    },
+                }],
+            },
+        )
+
+    records = generate_model_records(
+        [{"id": "preview-empty-retry", "edited_content": "来源正文"}],
+        model={"name": "model", "api_url": "https://model.example/v1"},
+        config={"generation_retries": 1},
+        task_id="task-empty-retry",
+        split={"train": 100, "validation": 0, "test": 0},
+        qa_pairs_per_item=1,
+        client=httpx.Client(transport=httpx.MockTransport(handler)),
+    )
+
+    assert request_count == 2
+    assert records[0]["status"] == "valid"
+
+
+def test_multiple_top_level_json_documents_are_rejected_as_ambiguous() -> None:
+    first = json.dumps({
+        "items": [{"instruction": "问题一", "output": "答案一"}],
+    }, ensure_ascii=False)
+    second = json.dumps({
+        "items": [{"instruction": "问题二", "output": "答案二"}],
+    }, ensure_ascii=False)
+    client = httpx.Client(
+        transport=httpx.MockTransport(
+            lambda _: httpx.Response(
+                200,
+                json={"choices": [{"message": {"content": f"{first}\n{second}"}}]},
+            )
+        )
+    )
+
+    records = generate_model_records(
+        [{"id": "preview-ambiguous", "edited_content": "来源正文"}],
+        model={"name": "model", "api_url": "https://model.example/v1"},
+        config={"generation_retries": 0},
+        task_id="task-ambiguous",
+        split={"train": 100, "validation": 0, "test": 0},
+        qa_pairs_per_item=1,
+        client=client,
+    )
+
+    assert records[0]["status"] == "invalid"
+    assert "多个 JSON" in records[0]["error"]
+
+
+def test_generate_model_records_builds_reasoning_output_with_think_tags() -> None:
+    def handler(request: httpx.Request) -> httpx.Response:
+        payload = json.loads(request.content)
+        system_prompt = payload["messages"][0]["content"]
+        assert '"reasoning":"...","answer":"..."' in system_prompt
+        assert "你正在生成用于训练推理模型的思维链数据" in system_prompt
+        assert "推理详细程度为“普通”" in system_prompt
+        assert "系统会在保存时统一组装" in system_prompt
+        content = "<think>模型接口自己的分析</think>" + json.dumps(
+            {
+                "items": [
+                    {
+                        "instruction": "计算两项费用合计",
+                        "input": "交通费 30 元，餐费 20 元",
+                        "reasoning": "先识别两项费用，再计算 30 + 20。",
+                        "answer": "合计 50 元。",
+                    }
+                ]
+            },
+            ensure_ascii=False,
+        )
+        return httpx.Response(
+            200,
+            json={"choices": [{"message": {"content": content}}]},
+        )
+
+    records = generate_model_records(
+        [{"id": "preview-reasoning", "edited_content": "交通费 30 元，餐费 20 元"}],
+        model={"name": "model", "api_url": "https://model.example/v1"},
+        config={"output_type": "reasoning"},
+        task_id="task-reasoning",
+        split={"train": 100, "validation": 0, "test": 0},
+        qa_pairs_per_item=1,
+        client=httpx.Client(transport=httpx.MockTransport(handler)),
+    )
+
+    assert records[0]["status"] == "valid"
+    assert records[0]["output"] == (
+        "<think>\n先识别两项费用,再计算 30 + 20。\n</think>\n合计 50 元。"
+    )
+
+
+def test_generate_model_records_uses_detailed_reasoning_instruction() -> None:
+    def handler(request: httpx.Request) -> httpx.Response:
+        system_prompt = json.loads(request.content)["messages"][0]["content"]
+        assert "推理详细程度为“详细”" in system_prompt
+        assert "完整展开问题条件、来源依据、中间计算或推导" in system_prompt
+        return httpx.Response(
+            200,
+            json={
+                "choices": [
+                    {
+                        "message": {
+                            "content": json.dumps(
+                                {
+                                    "items": [
+                                        {
+                                            "instruction": "计算报销总额",
+                                            "reasoning": "条件为交通费 30 元和餐费 20 元。分别核对后相加，30 + 20 = 50。",
+                                            "answer": "报销总额为 50 元。",
+                                        }
+                                    ]
+                                },
+                                ensure_ascii=False,
+                            )
+                        }
+                    }
+                ]
+            },
+        )
+
+    records = generate_model_records(
+        [{"id": "preview-detailed", "edited_content": "交通费 30 元，餐费 20 元"}],
+        model={"name": "model", "api_url": "https://model.example/v1"},
+        config={"output_type": "reasoning", "reasoning_detail": "detailed"},
+        task_id="task-detailed",
+        split={"train": 100, "validation": 0, "test": 0},
+        qa_pairs_per_item=1,
+        client=httpx.Client(transport=httpx.MockTransport(handler)),
+    )
+
+    assert records[0]["status"] == "valid"
+    assert "分别核对后相加" in records[0]["output"]
+
+
+def test_generate_model_records_marks_reasoning_without_reasoning_field_invalid() -> None:
+    response = {
+        "choices": [
+            {
+                "message": {
+                    "content": json.dumps(
+                        {
+                            "items": [
+                                {
+                                    "instruction": "问题",
+                                    "answer": "只有最终答案",
+                                }
+                            ]
+                        },
+                        ensure_ascii=False,
+                    )
+                }
+            }
+        ]
+    }
+    client = httpx.Client(
+        transport=httpx.MockTransport(lambda _: httpx.Response(200, json=response))
+    )
+
+    records = generate_model_records(
+        [{"id": "preview-missing-reasoning", "edited_content": "来源正文"}],
+        model={"name": "model", "api_url": "https://model.example/v1"},
+        config={"output_type": "reasoning"},
+        task_id="task-missing-reasoning",
+        split={"train": 100, "validation": 0, "test": 0},
+        qa_pairs_per_item=1,
+        client=client,
+    )
+
+    assert records[0]["status"] == "invalid"
+    assert records[0]["output"] == "只有最终答案"
+    assert "reasoning" in records[0]["error"]
+
+
+def test_standard_output_removes_model_think_block() -> None:
+    content = json.dumps(
+        {
+            "items": [
+                {
+                    "instruction": "问题",
+                    "output": "<think>不应保存的分析</think>最终答案",
+                }
+            ]
+        },
+        ensure_ascii=False,
+    )
+    client = httpx.Client(
+        transport=httpx.MockTransport(
+            lambda _: httpx.Response(
+                200,
+                json={"choices": [{"message": {"content": content}}]},
+            )
+        )
+    )
+
+    records = generate_model_records(
+        [{"id": "preview-standard", "edited_content": "来源正文"}],
+        model={"name": "model", "api_url": "https://model.example/v1"},
+        config={"output_type": "standard"},
+        task_id="task-standard",
+        split={"train": 100, "validation": 0, "test": 0},
+        qa_pairs_per_item=1,
+        client=client,
+    )
+
+    assert records[0]["output"] == "最终答案"
+
+
+def test_generate_model_records_keeps_partial_failure_for_manual_repair() -> None:
+    client = httpx.Client(
+        transport=httpx.MockTransport(
+            lambda _: httpx.Response(200, json={"choices": [{"message": {"content": "not-json"}}]})
+        )
+    )
+    records = generate_model_records(
+        [{"id": "preview-1", "edited_content": "来源正文"}],
+        model={"name": "model", "api_url": "https://model.example/v1"},
+        config={"generation_retries": 1},
+        task_id="task-1",
+        split={"train": 80, "validation": 10, "test": 10},
+        qa_pairs_per_item=1,
+        client=client,
+    )
+
+    assert len(records) == 1
+    assert records[0]["status"] == "invalid"
+    assert records[0]["error"]
+
+
+def test_generate_model_records_batches_fifty_results_with_unique_ids() -> None:
+    requests: list[httpx.Request] = []
+
+    def handler(request: httpx.Request) -> httpx.Response:
+        requests.append(request)
+        batch_start = (len(requests) - 1) * 10 + 1
+        batch_end = batch_start + 9
+        payload = json.loads(request.content)
+        system_prompt = payload["messages"][0]["content"]
+        assert "items 必须包含 10 条" in system_prompt
+        assert f"第 {batch_start}-{batch_end} 条" in system_prompt
+        return httpx.Response(
+            200,
+            json={
+                "choices": [
+                    {
+                        "message": {
+                            "content": json.dumps(
+                                {
+                                    "items": [
+                                        {
+                                            "instruction": "同一问题",
+                                            "input": "来源正文",
+                                            "output": "同一答案",
+                                        }
+                                        for _ in range(batch_start, batch_end + 1)
+                                    ]
+                                },
+                                ensure_ascii=False,
+                            )
+                        }
+                    }
+                ]
+            },
+        )
+
+    records = generate_model_records(
+        [{"id": "preview-50", "edited_content": "来源正文"}],
+        model={"name": "model", "api_url": "https://model.example/v1"},
+        config={},
+        task_id="task-50",
+        split={"train": 100, "validation": 0, "test": 0},
+        qa_pairs_per_item=50,
+        client=httpx.Client(transport=httpx.MockTransport(handler)),
+    )
+
+    assert len(requests) == 5
+    assert len(records) == 50
+    assert len({record["id"] for record in records}) == 50
+    assert {record["instruction"] for record in records} == {"同一问题"}
+    assert all(record["status"] == "valid" for record in records)
+
+
+def test_generate_model_records_preserves_successful_batches_when_one_fails() -> None:
+    request_count = 0
+
+    def handler(_: httpx.Request) -> httpx.Response:
+        nonlocal request_count
+        request_count += 1
+        if request_count == 2:
+            return httpx.Response(500)
+        return httpx.Response(
+            200,
+            json={
+                "choices": [
+                    {
+                        "message": {
+                            "content": json.dumps(
+                                {
+                                    "items": [
+                                        {
+                                            "instruction": f"问题 {index}",
+                                            "output": f"答案 {index}",
+                                        }
+                                        for index in range(1, 11)
+                                    ]
+                                },
+                                ensure_ascii=False,
+                            )
+                        }
+                    }
+                ]
+            },
+        )
+
+    records = generate_model_records(
+        [{"id": "preview-partial", "edited_content": "来源正文"}],
+        model={"name": "model", "api_url": "https://model.example/v1"},
+        config={"generation_retries": 0},
+        task_id="task-partial",
+        split={"train": 100, "validation": 0, "test": 0},
+        qa_pairs_per_item=20,
+        client=httpx.Client(transport=httpx.MockTransport(handler)),
+    )
+
+    assert len(records) == 11
+    assert sum(record["status"] == "valid" for record in records) == 10
+    failed = next(record for record in records if record["status"] == "invalid")
+    assert "第 11-20 条" in failed["instruction"]
+    assert len({record["id"] for record in records}) == len(records)
+
+
+def test_generate_model_records_retries_short_batch_then_marks_it_invalid() -> None:
+    request_count = 0
+
+    def handler(_: httpx.Request) -> httpx.Response:
+        nonlocal request_count
+        request_count += 1
+        return httpx.Response(
+            200,
+            json={
+                "choices": [
+                    {
+                        "message": {
+                            "content": json.dumps(
+                                {
+                                    "items": [
+                                        {
+                                            "instruction": "只有一条",
+                                            "output": "不足本批要求数量",
+                                        }
+                                    ]
+                                },
+                                ensure_ascii=False,
+                            )
+                        }
+                    }
+                ]
+            },
+        )
+
+    records = generate_model_records(
+        [{"id": "preview-short", "edited_content": "来源正文"}],
+        model={"name": "model", "api_url": "https://model.example/v1"},
+        config={"generation_retries": 1},
+        task_id="task-short",
+        split={"train": 100, "validation": 0, "test": 0},
+        qa_pairs_per_item=10,
+        client=httpx.Client(transport=httpx.MockTransport(handler)),
+    )
+
+    assert request_count == 2
+    assert len(records) == 1
+    assert records[0]["status"] == "invalid"
+    assert "expected 10, got 1" in records[0]["error"]
+
+
+def test_generate_model_records_does_not_retry_non_retryable_http_errors() -> None:
+    request_count = 0
+
+    def handler(_: httpx.Request) -> httpx.Response:
+        nonlocal request_count
+        request_count += 1
+        return httpx.Response(401, json={"error": {"message": "unauthorized"}})
+
+    records = generate_model_records(
+        [{"id": "preview-auth", "edited_content": "来源内容"}],
+        model={
+            "api_url": "https://model.example/v1",
+            "online_model_name": "test-model",
+            "api_key": "invalid",
+        },
+        config={"generation_retries": 5},
+        task_id="task-auth",
+        split={"train": 100, "validation": 0, "test": 0},
+        qa_pairs_per_item=1,
+        client=httpx.Client(transport=httpx.MockTransport(handler)),
+    )
+
+    assert request_count == 1
+    assert records[0]["status"] == "invalid"
+    assert "401" in records[0]["error"]
+
+
+@pytest.mark.parametrize("status_code", [408, 425, 429, 500])
+def test_generate_model_records_retries_retryable_http_statuses(
+    status_code: int,
+) -> None:
+    request_count = 0
+
+    def handler(_: httpx.Request) -> httpx.Response:
+        nonlocal request_count
+        request_count += 1
+        if request_count == 1:
+            return httpx.Response(status_code)
+        return httpx.Response(
+            200,
+            json={
+                "choices": [{
+                    "message": {
+                        "content": json.dumps({
+                            "items": [{
+                                "instruction": "来源内容是什么？",
+                                "output": "这是用于验证可重试错误的来源内容。",
+                            }],
+                        }, ensure_ascii=False),
+                    },
+                }],
+            },
+        )
+
+    records = generate_model_records(
+        [{"id": "preview-retryable", "edited_content": "来源内容"}],
+        model={"name": "model", "api_url": "https://model.example/v1"},
+        config={"generation_retries": 1},
+        task_id="task-retryable",
+        split={"train": 100, "validation": 0, "test": 0},
+        qa_pairs_per_item=1,
+        client=httpx.Client(transport=httpx.MockTransport(handler)),
+    )
+
+    assert request_count == 2
+    assert records[0]["status"] == "valid"
+
+
+def test_generate_model_records_retries_transient_network_errors() -> None:
+    request_count = 0
+
+    def handler(request: httpx.Request) -> httpx.Response:
+        nonlocal request_count
+        request_count += 1
+        if request_count == 1:
+            raise httpx.ConnectError("temporary connection failure", request=request)
+        return httpx.Response(
+            200,
+            json={
+                "choices": [{
+                    "message": {
+                        "content": json.dumps({
+                            "items": [{
+                                "instruction": "网络恢复了吗？",
+                                "output": "临时连接错误后，第二次模型请求已经成功。",
+                            }],
+                        }, ensure_ascii=False),
+                    },
+                }],
+            },
+        )
+
+    records = generate_model_records(
+        [{"id": "preview-network", "edited_content": "网络重试来源"}],
+        model={"name": "model", "api_url": "https://model.example/v1"},
+        config={"generation_retries": 1},
+        task_id="task-network",
+        split={"train": 100, "validation": 0, "test": 0},
+        qa_pairs_per_item=1,
+        client=httpx.Client(transport=httpx.MockTransport(handler)),
+    )
+
+    assert request_count == 2
+    assert records[0]["status"] == "valid"
+
+
+@pytest.mark.parametrize("qa_pairs_per_item", [0, 51])
+def test_generate_model_records_rejects_out_of_range_count(
+    qa_pairs_per_item: int,
+) -> None:
+    with pytest.raises(ModelGenerationError, match=r"\[1, 50\]"):
+        generate_model_records(
+            [],
+            model={"name": "model", "api_url": "https://model.example/v1"},
+            config={},
+            task_id="task-invalid",
+            split={"train": 100, "validation": 0, "test": 0},
+            qa_pairs_per_item=qa_pairs_per_item,
+        )
+
+
+def test_generate_model_records_rejects_unknown_output_type() -> None:
+    with pytest.raises(ModelGenerationError, match="output_type"):
+        generate_model_records(
+            [],
+            model={"name": "model", "api_url": "https://model.example/v1"},
+            config={"output_type": "unknown"},
+            task_id="task-invalid-output",
+            split={"train": 100, "validation": 0, "test": 0},
+            qa_pairs_per_item=1,
+        )
+
+
+def test_generate_model_records_rejects_unknown_reasoning_detail() -> None:
+    with pytest.raises(ModelGenerationError, match="reasoning_detail"):
+        generate_model_records(
+            [],
+            model={"name": "model", "api_url": "https://model.example/v1"},
+            config={"output_type": "reasoning", "reasoning_detail": "verbose"},
+            task_id="task-invalid-reasoning-detail",
+            split={"train": 100, "validation": 0, "test": 0},
+            qa_pairs_per_item=1,
+        )

backend/tests/test_data_process_migration.py

@@ -0,0 +1,65 @@
+from __future__ import annotations
+
+from pathlib import Path
+
+from app.modules.data_process.schema_cli import REQUIRED_TASK_COLUMNS, _target_label
+
+
+def test_runtime_migration_fails_fast_on_incompatible_schema() -> None:
+    sql_path = (
+        Path(__file__).resolve().parents[1]
+        / "app"
+        / "db"
+        / "sql"
+        / "002_data_process.sql"
+    )
+    sql = sql_path.read_text(encoding="utf-8")
+
+    assert "requires 001_platform_runtime.sql first" in sql
+    assert "supports only the current TEXT runtime schema" in sql
+    assert "generation_run_id" in sql
+    assert "results_confirmed BOOLEAN NOT NULL DEFAULT TRUE" in sql
+    assert "WHERE status <> 'completed' AND results_confirmed=TRUE" in sql
+    assert "ADD COLUMN IF NOT EXISTS workflow_step VARCHAR(20)" in sql
+    assert "ADD COLUMN IF NOT EXISTS preview_status VARCHAR(20)" in sql
+    assert "ADD COLUMN IF NOT EXISTS preview_progress NUMERIC(5,2)" in sql
+    assert "ADD COLUMN IF NOT EXISTS preview_run_id TEXT" in sql
+    assert "ADD COLUMN IF NOT EXISTS preview_failure_reason TEXT" in sql
+    assert "ADD COLUMN IF NOT EXISTS preview_total_files INTEGER" in sql
+    assert "ADD COLUMN IF NOT EXISTS preview_completed_files INTEGER" in sql
+    assert "data_process_workflow_backfill_ids" in sql
+    assert "ck_data_process_tasks_workflow_step" in sql
+    assert "ck_data_process_tasks_preview_status" in sql
+    assert "ck_data_process_tasks_preview_progress" in sql
+    assert "ck_data_process_tasks_preview_file_counts" in sql
+    for value in ("create", "model", "upload", "preview", "generate", "results"):
+        assert f"'{value}'" in sql
+    for value in ("idle", "queued", "running", "completed", "failed", "cancelled"):
+        assert f"'{value}'" in sql
+    assert "CREATE TABLE IF NOT EXISTS data_process_results" in sql
+    assert "ADD COLUMN IF NOT EXISTS chosen TEXT NOT NULL DEFAULT ''" in sql
+    assert "ADD COLUMN IF NOT EXISTS rejected TEXT NOT NULL DEFAULT ''" in sql
+    assert "ADD COLUMN IF NOT EXISTS original_chosen TEXT" in sql
+    assert "ADD COLUMN IF NOT EXISTS original_rejected TEXT" in sql
+    assert sql.count("BEGIN;") == 1
+    assert sql.rstrip().endswith("COMMIT;")
+
+
+def test_schema_cli_target_label_never_contains_credentials() -> None:
+    label = _target_label("postgresql://secret-user:secret-password@db.example:5433/yg_ft")
+    assert label == "db.example:5433/yg_ft"
+    assert "secret" not in label
+
+
+def test_schema_check_requires_current_runtime_columns() -> None:
+    assert REQUIRED_TASK_COLUMNS == (
+        "generation_run_id",
+        "results_confirmed",
+        "workflow_step",
+        "preview_status",
+        "preview_progress",
+        "preview_run_id",
+        "preview_failure_reason",
+        "preview_total_files",
+        "preview_completed_files",
+    )

backend/tests/test_data_process_storage.py

@@ -0,0 +1,276 @@
+from __future__ import annotations
+
+from pathlib import Path, PurePosixPath
+
+import pytest
+
+from app.modules.data_process import storage as storage_module
+from app.modules.data_process.storage import (
+    DataProcessStorageError,
+    LocalDataProcessStorage,
+    StagedSourceObject,
+)
+
+
+def _stage(
+    storage: LocalDataProcessStorage,
+    *,
+    batch_id: str = "batch-main",
+    task_id: str = "task-1",
+    source_file_id: str = "source-1",
+    version: int = 1,
+    name: str = "source.txt",
+    content: bytes = b"payload",
+) -> StagedSourceObject:
+    return storage.stage_bytes(
+        batch_id=batch_id,
+        task_id=task_id,
+        source_file_id=source_file_id,
+        version=version,
+        name=name,
+        content=content,
+    )
+
+
+def _create_symlink(link: Path, target: Path, *, target_is_directory: bool = False) -> None:
+    try:
+        link.symlink_to(target, target_is_directory=target_is_directory)
+    except (NotImplementedError, OSError) as exc:
+        pytest.skip(f"当前平台不支持创建测试所需的符号链接: {exc}")
+
+
+def _assert_staging_empty(storage: LocalDataProcessStorage) -> None:
+    assert list((storage.root / ".staging").iterdir()) == []
+
+
+def test_stage_publish_read_delete_roundtrip_with_unicode_filename(tmp_path: Path) -> None:
+    storage = LocalDataProcessStorage(tmp_path / "storage")
+    content = "第一行\n第二行，100% 完成".encode()
+
+    staged = _stage(
+        storage,
+        name="中文 数据 100%.csv",
+        content=content,
+    )
+
+    assert "%20" in staged.reference
+    assert "%25" in staged.reference
+    storage.publish([staged])
+
+    assert storage.read(staged.reference) == content
+    assert storage.delete(staged.reference) is True
+    assert storage.delete(staged.reference) is False
+    _assert_staging_empty(storage)
+
+
+def test_stage_copy_creates_an_independently_deletable_source_object(
+    tmp_path: Path,
+) -> None:
+    storage = LocalDataProcessStorage(tmp_path / "storage")
+    original = _stage(storage, content=b"immutable source")
+    storage.publish([original])
+
+    copied = storage.stage_copy(
+        batch_id="batch-copy",
+        source_reference=original.reference,
+        expected_source_task_id="task-1",
+        expected_source_file_id="source-1",
+        task_id="task-2",
+        source_file_id="source-2",
+        version=1,
+        name="source.txt",
+    )
+    storage.publish([copied])
+
+    assert storage.read(copied.reference) == b"immutable source"
+    assert storage.delete(
+        original.reference,
+        expected_task_id="task-1",
+        expected_source_file_id="source-1",
+    ) is True
+    assert storage.read(copied.reference) == b"immutable source"
+    assert storage.delete(
+        copied.reference,
+        expected_task_id="task-2",
+        expected_source_file_id="source-2",
+    ) is True
+    _assert_staging_empty(storage)
+
+
+def test_db_reference_is_left_to_database_storage(tmp_path: Path) -> None:
+    storage = LocalDataProcessStorage(tmp_path / "storage")
+
+    assert storage.read("db://source-files/source-1") is None
+    assert storage.delete("db://source-files/source-1") is False
+
+
+def test_owned_source_can_be_streamed_by_byte_range(tmp_path: Path) -> None:
+    storage = LocalDataProcessStorage(tmp_path / "storage")
+    content = b"0123456789abcdef"
+    staged = _stage(storage, content=content)
+    storage.publish([staged])
+
+    assert storage.file_size(
+        staged.reference,
+        expected_task_id="task-1",
+        expected_source_file_id="source-1",
+    ) == len(content)
+    assert b"".join(storage.iter_bytes(
+        staged.reference,
+        expected_task_id="task-1",
+        expected_source_file_id="source-1",
+        expected_size=len(content),
+        start=4,
+        length=6,
+        chunk_size=2,
+    )) == b"456789"
+
+    with pytest.raises(DataProcessStorageError, match="owner mismatch"):
+        storage.file_size(
+            staged.reference,
+            expected_task_id="another-task",
+            expected_source_file_id="source-1",
+        )
+    with pytest.raises(DataProcessStorageError, match="does not match metadata"):
+        b"".join(storage.iter_bytes(
+            staged.reference,
+            expected_task_id="task-1",
+            expected_source_file_id="source-1",
+            expected_size=len(content) + 1,
+        ))
+
+
+@pytest.mark.parametrize(
+    "reference",
+    [
+        "local://data-process/../source-1/v1/file.txt",
+        "local://data-process/task-1/source-1/v1/file%2Fname.txt",
+        "local://data-process/task-1/source-1/v1/file.txt?download=1",
+        "local://data-process/task-1/source-1/v1/file.txt#fragment",
+        "https://data-process/task-1/source-1/v1/file.txt",
+    ],
+    ids=[
+        "parent-traversal",
+        "percent-encoded-slash",
+        "query",
+        "fragment",
+        "wrong-scheme",
+    ],
+)
+def test_unsafe_references_are_rejected(tmp_path: Path, reference: str) -> None:
+    storage = LocalDataProcessStorage(tmp_path / "storage")
+
+    with pytest.raises(DataProcessStorageError):
+        storage.read(reference)
+    with pytest.raises(DataProcessStorageError):
+        storage.delete(reference)
+
+
+def test_publish_rejects_intermediate_directory_symlink(tmp_path: Path) -> None:
+    storage = LocalDataProcessStorage(tmp_path / "storage")
+    outside = tmp_path / "outside"
+    outside.mkdir()
+    staged = _stage(storage, task_id="linked-task")
+    _create_symlink(
+        storage.root / "linked-task",
+        outside,
+        target_is_directory=True,
+    )
+
+    with pytest.raises(DataProcessStorageError, match="symlink|non-directory"):
+        storage.publish([staged])
+
+    assert list(outside.iterdir()) == []
+    _assert_staging_empty(storage)
+
+
+def test_target_symlink_is_never_followed_or_deleted(tmp_path: Path) -> None:
+    storage = LocalDataProcessStorage(tmp_path / "storage")
+    staged = _stage(storage, task_id="task-link", source_file_id="source-link")
+    outside_file = tmp_path / "outside.txt"
+    outside_file.write_bytes(b"outside sentinel")
+    final_path = storage.root.joinpath(*staged._relative_path.parts)
+    final_path.parent.mkdir(parents=True)
+    _create_symlink(final_path, outside_file)
+
+    with pytest.raises(DataProcessStorageError, match="already exists"):
+        storage.publish([staged])
+    with pytest.raises(DataProcessStorageError, match="regular file"):
+        storage.read(staged.reference)
+    with pytest.raises(DataProcessStorageError, match="non-regular"):
+        storage.delete(staged.reference)
+
+    assert final_path.is_symlink()
+    assert outside_file.read_bytes() == b"outside sentinel"
+    _assert_staging_empty(storage)
+
+
+def test_publish_rolls_back_first_object_when_second_target_collides(tmp_path: Path) -> None:
+    storage = LocalDataProcessStorage(tmp_path / "storage")
+    existing = _stage(
+        storage,
+        batch_id="batch-existing",
+        source_file_id="source-existing",
+        content=b"existing content",
+    )
+    storage.publish([existing])
+
+    first = _stage(
+        storage,
+        batch_id="batch-new",
+        source_file_id="source-new",
+        content=b"must be rolled back",
+    )
+    colliding_second = _stage(
+        storage,
+        batch_id="batch-new",
+        source_file_id="source-existing",
+        content=b"must not replace existing content",
+    )
+
+    with pytest.raises(DataProcessStorageError, match="already exists"):
+        storage.publish([first, colliding_second])
+
+    with pytest.raises(DataProcessStorageError, match="does not exist"):
+        storage.read(first.reference)
+    assert storage.read(existing.reference) == b"existing content"
+    _assert_staging_empty(storage)
+
+
+def test_publish_rejects_manually_forged_staged_object(tmp_path: Path) -> None:
+    storage = LocalDataProcessStorage(tmp_path / "storage")
+    temporary_path = storage.root / ".staging" / "batch-forged" / "forged.tmp"
+    temporary_path.parent.mkdir()
+    temporary_path.write_bytes(b"forged content")
+    relative_path = PurePosixPath("task-forged", "source-forged", "v1", "forged.txt")
+    forged = StagedSourceObject(
+        reference="local://data-process/task-forged/source-forged/v1/forged.txt",
+        _temporary_path=temporary_path,
+        _relative_path=relative_path,
+    )
+
+    with pytest.raises(DataProcessStorageError, match="was not issued"):
+        storage.publish([forged])
+    with pytest.raises(DataProcessStorageError, match="was not issued"):
+        storage.discard([forged])
+
+    assert temporary_path.read_bytes() == b"forged content"
+    assert not storage.root.joinpath(*relative_path.parts).exists()
+
+
+def test_relative_storage_configuration_is_anchored_to_backend_root(
+    tmp_path: Path,
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    relative_configuration = Path("relative-storage") / tmp_path.name
+    backend_root = Path(storage_module.__file__).resolve().parents[3]
+    monkeypatch.chdir(tmp_path)
+    monkeypatch.setenv("DATA_PROCESS_STORAGE_DIR", str(relative_configuration))
+    storage_module.get_data_process_storage.cache_clear()
+
+    try:
+        configured_root = storage_module._configured_storage_root()
+        assert configured_root == backend_root / relative_configuration
+        assert not configured_root.exists()
+    finally:
+        storage_module.get_data_process_storage.cache_clear()

backend/tests/test_docs_security.py

@@ -0,0 +1,76 @@
+"""Swagger / ReDoc / OpenAPI 文档路由安全开关测试。
+
+生产环境（APP_ENV=prod）默认关闭 /docs、/redoc、/openapi.json，
+避免未授权访问泄露 API 结构；本地开发环境默认开放，可用 ENABLE_DOCS 覆盖。
+"""
+from __future__ import annotations
+
+import pytest
+
+from app.core.config import docs_kwargs, get_settings
+
+
+@pytest.fixture(autouse=True)
+def _reset_settings_cache():
+    """每次测试前后清空 get_settings 的 lru_cache，避免环境变量互相污染。"""
+    get_settings.cache_clear()
+    yield
+    get_settings.cache_clear()
+
+
+def test_docs_kwargs_enabled() -> None:
+    assert docs_kwargs(True) == {}
+
+
+def test_docs_kwargs_disabled() -> None:
+    assert docs_kwargs(False) == {"docs_url": None, "redoc_url": None, "openapi_url": None}
+
+
+def test_docs_disabled_by_default_in_prod(monkeypatch) -> None:
+    monkeypatch.delenv("ENABLE_DOCS", raising=False)
+    monkeypatch.setenv("APP_ENV", "prod")
+    assert get_settings().enable_docs is False
+
+
+def test_docs_enabled_by_default_outside_prod(monkeypatch) -> None:
+    monkeypatch.delenv("ENABLE_DOCS", raising=False)
+    monkeypatch.setenv("APP_ENV", "local")
+    assert get_settings().enable_docs is True
+
+
+def test_docs_env_override_enables_in_prod(monkeypatch) -> None:
+    monkeypatch.setenv("ENABLE_DOCS", "true")
+    monkeypatch.setenv("APP_ENV", "prod")
+    assert get_settings().enable_docs is True
+
+
+def test_docs_env_override_disables_outside_prod(monkeypatch) -> None:
+    monkeypatch.setenv("ENABLE_DOCS", "false")
+    monkeypatch.setenv("APP_ENV", "local")
+    assert get_settings().enable_docs is False
+
+
+def test_create_app_disables_docs_in_prod(monkeypatch, tmp_path) -> None:
+    pytest.importorskip("fastapi")
+    from app.main import create_app
+
+    monkeypatch.delenv("ENABLE_DOCS", raising=False)
+    monkeypatch.setenv("APP_ENV", "prod")
+    monkeypatch.setenv("LOG_DIR", str(tmp_path))
+    app = create_app()
+    assert app.docs_url is None
+    assert app.redoc_url is None
+    assert app.openapi_url is None
+
+
+def test_create_app_enables_docs_outside_prod(monkeypatch, tmp_path) -> None:
+    pytest.importorskip("fastapi")
+    from app.main import create_app
+
+    monkeypatch.delenv("ENABLE_DOCS", raising=False)
+    monkeypatch.setenv("APP_ENV", "local")
+    monkeypatch.setenv("LOG_DIR", str(tmp_path))
+    app = create_app()
+    assert app.docs_url == "/docs"
+    assert app.redoc_url == "/redoc"
+    assert app.openapi_url == "/openapi.json"

backend/tests/test_document_chunking.py

@@ -0,0 +1,85 @@
+from __future__ import annotations
+
+from llama_index.core.embeddings import MockEmbedding
+
+from app.modules.data_process.document_chunking import (
+    DocumentChunk,
+    _compact_with_offsets,
+    _project_layout_span,
+    chunk_fixed_text,
+    chunk_semantic_text,
+    merge_short_chunks,
+)
+
+
+def test_fixed_splitter_preserves_offsets_and_token_limit() -> None:
+    text = "第一段说明苹果。第二段说明香蕉。\n第三段说明数据库。第四段说明索引。"
+    chunks = chunk_fixed_text(text, chunk_size=20, chunk_overlap=0)
+
+    assert len(chunks) > 1
+    assert all(chunk.source_start is not None for chunk in chunks)
+    assert all(chunk.source_end is not None for chunk in chunks)
+    assert all(
+        chunk.original_content == text[chunk.source_start : chunk.source_end]
+        for chunk in chunks
+        if chunk.source_start is not None and chunk.source_end is not None
+    )
+    assert all(chunk.token_count <= 20 for chunk in chunks)
+
+
+def test_semantic_splitter_uses_llamaindex_and_reapplies_maximum_size() -> None:
+    text = "第一段讨论水果。第二段继续讨论香蕉。第三段讨论数据库。第四段讨论索引。"
+    chunks = chunk_semantic_text(
+        text,
+        chunk_size=30,
+        chunk_overlap=0,
+        breakpoint_percentile_threshold=95,
+        embed_model=MockEmbedding(embed_dim=8),
+    )
+
+    assert len(chunks) >= 2
+    assert all(chunk.token_count <= 30 for chunk in chunks)
+    assert "".join(chunk.original_content for chunk in chunks) == text
+
+
+def test_layout_projection_ignores_layout_whitespace_but_keeps_source_lines() -> None:
+    source = "标题\n第一条 这是正文。\n第二条 后续正文。"
+    compact_source, offsets = _compact_with_offsets(source)
+    start, end, cursor = _project_layout_span(
+        source,
+        "第一条\n这是正文。",
+        compact_source=compact_source,
+        source_offsets=offsets,
+        compact_start=0,
+    )
+
+    assert source[start:end] == "第一条 这是正文。"
+    assert cursor > 0
+
+
+def test_short_layout_chunk_merges_with_neighbor_and_keeps_page_provenance() -> None:
+    source = "短标题\n这是一段足够长的正文内容，用于测试相邻切片合并。"
+    chunks = [
+        DocumentChunk("短标题", "短标题", 0, 3, 1, 1, 2, source_pages=(1,)),
+        DocumentChunk(
+            "这是一段足够长的正文内容，用于测试相邻切片合并。",
+            "这是一段足够长的正文内容，用于测试相邻切片合并。",
+            4,
+            len(source),
+            2,
+            2,
+            20,
+            source_pages=(1, 2),
+        ),
+    ]
+
+    merged = merge_short_chunks(
+        chunks,
+        source_text=source,
+        min_token_count=10,
+        max_token_count=100,
+    )
+
+    assert len(merged) == 1
+    assert merged[0].original_content == source
+    assert merged[0].source_pages == (1, 2)

backend/tests/test_governance.py

@@ -0,0 +1,791 @@
+"""
+平台治理功能集成测试 —— 覆盖第 1-4 周交付内容。
+
+测试策略：
+- 在导入 app 模块前 mock psycopg / psycopg_pool，避免依赖真实数据库驱动
+- 使用 FastAPI TestClient 对真实路由栈发起请求
+- 通过 mock.get_platform_store 替换为内存 FakeStore
+- 每周交付内容对应一组 test class，方便分阶段验收
+
+覆盖范围：
+  第 1 周 — 登录、当前用户、用户列表、权限码、日志查询
+  第 2 周 — 租户、项目、项目成员、资源 ACL
+  第 3 周 — 审批实例、审批模板、审计日志查询和导出
+  第 4 周 — 写操作审计、审批拦截、权限校验
+"""
+
+from __future__ import annotations
+
+import json
+import sys
+import types
+from contextlib import contextmanager
+from typing import Any, Iterator
+from unittest.mock import MagicMock, patch
+
+import pytest
+from fastapi import FastAPI
+from fastapi.testclient import TestClient
+
+# ============================================================
+# 在导入 app 之前 mock psycopg / psycopg_pool
+# ============================================================
+
+_psycopg_mock = types.ModuleType("psycopg")
+_psycopg_mock.PgConn = type("PgConn", (), {})
+_psycopg_mock.PostgresConnectionPool = MagicMock()
+_psycopg_mock.connection = MagicMock()
+sys.modules.setdefault("psycopg", _psycopg_mock)
+
+_psycopg_pool_mock = types.ModuleType("psycopg_pool")
+_psycopg_pool_mock.ConnectionPool = MagicMock()
+sys.modules.setdefault("psycopg_pool", _psycopg_pool_mock)
+
+# 现在安全导入 app 模块
+from app.api.v1.endpoints.platform import ok, fail  # noqa: E402
+from app.modules.tenant.router import router as tenant_router  # noqa: E402
+from app.modules.project.router import router as project_router  # noqa: E402
+from app.modules.approval.router import router as approval_router  # noqa: E402
+from app.modules.system.router import router as system_router  # noqa: E402
+from app.modules.retention.router import router as retention_router  # noqa: E402
+from app.modules.resource.router import router as resource_router  # noqa: E402
+from app.api.v1.endpoints.platform import router as platform_router  # noqa: E402
+
+PREFIX = "/modelTF"
+ADMIN_TOKEN = "platform-token-u_admin"
+OP_TOKEN = "platform-token-u_op"
+
+
+# ============================================================
+# FakePlatformStore —— 内存实现，模拟 PlatformStore 全部治理接口
+# ============================================================
+
+class FakePlatformStore:
+    """平台治理测试专用内存 store，确保测试不连接真实数据库。"""
+
+    def __init__(self) -> None:
+        self._users: list[dict[str, Any]] = [
+            {
+                "id": "u_admin",
+                "username": "admin",
+                "display_name": "Admin",
+                "role": "admin",
+                "status": "active",
+                "permissions": [
+                    "dashboard", "fine-tune", "model-eval", "model-inference",
+                    "model-manage", "dataset", "data-process", "data-convert",
+                    "compute", "hardware", "logs", "user-settings",
+                ],
+                "last_login": "2026-08-01T10:00:00Z",
+                "protected": True,
+            },
+            {
+                "id": "u_op",
+                "username": "operator",
+                "display_name": "Operator",
+                "role": "operator",
+                "status": "active",
+                "permissions": ["dashboard", "fine-tune"],
+                "last_login": "2026-08-01T11:00:00Z",
+                "protected": False,
+            },
+        ]
+        self._tenants: dict[str, dict[str, Any]] = {}
+        self._projects: dict[str, dict[str, Any]] = {}
+        self._members: dict[str, list[dict[str, Any]]] = {}
+        self._acl: dict[str, list[dict[str, Any]]] = {}
+        self._audit_logs: list[dict[str, Any]] = []
+        self._approval_templates: dict[str, dict[str, Any]] = {}
+        self._approval_instances: dict[str, dict[str, Any]] = {}
+        self._retention_policies: dict[str, dict[str, Any]] = {}
+        self._models: list[dict[str, Any]] = []
+        self._datasets: list[dict[str, Any]] = []
+        self._tasks: list[dict[str, Any]] = []
+        self._compute_nodes: list[dict[str, Any]] = []
+        self._gpus: list[dict[str, Any]] = []
+        self._sessions: list[dict[str, Any]] = []
+        self._seq = 0
+
+    @contextmanager
+    def connect(self) -> Iterator[Any]:
+        class FakeConn:
+            def execute(self, *a, **kw):
+                return []
+
+            def commit(self):
+                pass
+
+            def rollback(self):
+                pass
+
+            def close(self):
+                pass
+
+        yield FakeConn()
+
+    # ---- helpers ----
+
+    def _next_id(self, prefix: str) -> str:
+        self._seq += 1
+        return f"{prefix}_{self._seq}"
+
+    # ==================== 第1周：登录 / 用户 / 权限码 / 日志 ====================
+
+    def login(self, username: str, password: str) -> dict[str, Any] | None:
+        for u in self._users:
+            if u["username"] == username and u["status"] == "active":
+                if password in ("admin123", "operator123", "test123"):
+                    return dict(u)
+        return None
+
+    def create_session(self, user_id: str) -> dict[str, Any]:
+        import secrets
+        sid = secrets.token_hex(16)
+        return {"session_id": sid, "user_id": user_id}
+
+    def finish_session(self, session_id: str) -> None:
+        pass
+
+    def users(self) -> list[dict[str, Any]]:
+        return [dict(u) for u in self._users]
+
+    def create_user(self, payload: dict[str, Any]) -> dict[str, Any]:
+        u = {"id": self._next_id("u"), "protected": False, **payload}
+        self._users.append(u)
+        return u
+
+    def update_user(self, user_id: str, payload: dict[str, Any]) -> dict[str, Any]:
+        for u in self._users:
+            if u["id"] == user_id:
+                u.update(payload)
+                return u
+        raise KeyError(user_id)
+
+    def delete_user(self, user_id: str) -> None:
+        self._users = [u for u in self._users if u["id"] != user_id]
+
+    def roles(self) -> list[dict[str, Any]]:
+        return [
+            {"name": "admin", "display_name": "管理员"},
+            {"name": "operator", "display_name": "操作员"},
+            {"name": "viewer", "display_name": "访客"},
+        ]
+
+    def log_files(self, date: str | None = None) -> list[dict[str, Any]]:
+        return [{"name": "backend-2026-08-01.log", "size": "1 KB", "date": "2026-08-01"}]
+
+    def log_content(self, file: str) -> dict[str, Any]:
+        return {"file": file, "content": "[INFO] test line", "size": "1 KB"}
+
+    def training_log_files(self) -> list[dict[str, Any]]:
+        return [{"task_id": "ft_001", "name": "ft_001.log", "size": "2 KB"}]
+
+    def training_log_content(self, file: str) -> dict[str, Any]:
+        return {"file": file, "content": "epoch 0 loss 1.0", "size": "2 KB"}
+
+    # ==================== 第2周：租户 / 项目 / 成员 / ACL ====================
+
+    def tenants(self) -> list[dict[str, Any]]:
+        return list(self._tenants.values())
+
+    def tenant(self, tenant_id: str) -> dict[str, Any]:
+        if tenant_id not in self._tenants:
+            raise KeyError(tenant_id)
+        return dict(self._tenants[tenant_id])
+
+    def create_tenant(self, payload: dict[str, Any]) -> dict[str, Any]:
+        tid = self._next_id("tnt")
+        t = {"id": tid, "status": "active", "quota": "{}", "retention_policy_id": None,
+             "create_time": "2026-08-01T00:00:00Z", **payload}
+        self._tenants[tid] = t
+        return dict(t)
+
+    def update_tenant(self, tenant_id: str, payload: dict[str, Any]) -> dict[str, Any]:
+        self._tenants[tenant_id].update(payload)
+        return dict(self._tenants[tenant_id])
+
+    def set_tenant_quota(self, tenant_id: str, quota: dict[str, Any]) -> dict[str, Any]:
+        self._tenants[tenant_id]["quota"] = json.dumps(quota)
+        return dict(self._tenants[tenant_id])
+
+    def set_tenant_retention(self, tenant_id: str, retention_policy_id: str | None) -> dict[str, Any]:
+        self._tenants[tenant_id]["retention_policy_id"] = retention_policy_id
+        return dict(self._tenants[tenant_id])
+
+    def projects(self, *, tenant_id: str = "default", status: str | None = None, keyword: str | None = None) -> list[dict[str, Any]]:
+        result = []
+        for p in self._projects.values():
+            if p.get("tenant_id") != tenant_id:
+                continue
+            if status and p.get("status") != status:
+                continue
+            if keyword and keyword.lower() not in p.get("name", "").lower():
+                continue
+            result.append(dict(p))
+        return result
+
+    def project(self, project_id: str) -> dict[str, Any]:
+        if project_id not in self._projects:
+            raise KeyError(project_id)
+        return dict(self._projects[project_id])
+
+    def create_project(self, payload: dict[str, Any]) -> dict[str, Any]:
+        pid = self._next_id("prj")
+        p = {"id": pid, "status": "active", "quota": "{}", "create_time": "2026-08-01T00:00:00Z", **payload}
+        self._projects[pid] = p
+        self._members[pid] = []
+        return dict(p)
+
+    def update_project(self, project_id: str, payload: dict[str, Any]) -> dict[str, Any]:
+        self._projects[project_id].update(payload)
+        return dict(self._projects[project_id])
+
+    def archive_project(self, project_id: str) -> dict[str, Any]:
+        self._projects[project_id]["status"] = "archived"
+        return dict(self._projects[project_id])
+
+    def delete_project(self, project_id: str) -> None:
+        self._projects.pop(project_id, None)
+        self._members.pop(project_id, None)
+
+    def project_members(self, project_id: str) -> list[dict[str, Any]]:
+        return [dict(m) for m in self._members.get(project_id, [])]
+
+    def add_project_member(self, project_id: str, payload: dict[str, Any]) -> dict[str, Any]:
+        m = {"joined_at": "2026-08-01T00:00:00Z", **payload}
+        self._members.setdefault(project_id, []).append(m)
+        return m
+
+    def update_project_member_role(self, project_id: str, user_id: str, role: str) -> dict[str, Any]:
+        for m in self._members.get(project_id, []):
+            if m["user_id"] == user_id:
+                m["role"] = role
+                return m
+        raise KeyError(user_id)
+
+    def remove_project_member(self, project_id: str, user_id: str) -> None:
+        self._members[project_id] = [m for m in self._members.get(project_id, []) if m["user_id"] != user_id]
+
+    # ---- ACL ----
+
+    def get_acl(self, resource_type: str, resource_id: str) -> list[dict[str, Any]]:
+        key = f"{resource_type}:{resource_id}"
+        return [dict(a) for a in self._acl.get(key, [])]
+
+    def set_acl(self, resource_type: str, resource_id: str, entries: list[dict[str, Any]]) -> list[dict[str, Any]]:
+        key = f"{resource_type}:{resource_id}"
+        self._acl[key] = [dict(e) for e in entries]
+        return self.get_acl(resource_type, resource_id)
+
+    def resource_acl(self, resource_type: str, resource_id: str) -> list[dict[str, Any]]:
+        rows = self.get_acl(resource_type, resource_id)
+        grouped: dict[str, dict[str, Any]] = {}
+        for r in rows:
+            k = f"{r.get('principal_type')}:{r.get('principal_id')}"
+            bucket = grouped.setdefault(k, {
+                "subject_type": r.get("principal_type"),
+                "subject_id": r.get("principal_id"),
+                "permissions": [],
+            })
+            perm = r.get("permission")
+            if perm and perm not in bucket["permissions"]:
+                bucket["permissions"].append(perm)
+        return list(grouped.values())
+
+    def set_resource_acl(self, resource_type: str, resource_id: str, entries: list[dict[str, Any]]) -> list[dict[str, Any]]:
+        flat: list[dict[str, Any]] = []
+        for e in entries:
+            for perm in e.get("permissions") or []:
+                flat.append({
+                    "principal_type": e.get("subject_type"),
+                    "principal_id": e.get("subject_id"),
+                    "permission": perm,
+                })
+        self.set_acl(resource_type, resource_id, flat)
+        return self.resource_acl(resource_type, resource_id)
+
+    # ==================== 第3周：审批 / 审计 / 留存 ====================
+
+    def approval_templates(self) -> list[dict[str, Any]]:
+        return list(self._approval_templates.values())
+
+    def create_approval_template(self, payload: dict[str, Any]) -> dict[str, Any]:
+        tid = payload.get("id") or self._next_id("tpl")
+        t = {"id": tid, "steps": [], "create_time": "2026-08-01T00:00:00Z", **payload}
+        self._approval_templates[tid] = t
+        return dict(t)
+
+    def approval_instances(self, *, status: str | None = None) -> list[dict[str, Any]]:
+        result = []
+        for i in self._approval_instances.values():
+            if status and i.get("status") != status:
+                continue
+            result.append(dict(i))
+        return result
+
+    def approval_instance(self, instance_id: str) -> dict[str, Any]:
+        if instance_id not in self._approval_instances:
+            raise KeyError(instance_id)
+        return dict(self._approval_instances[instance_id])
+
+    def create_approval_instance(self, payload: dict[str, Any]) -> dict[str, Any]:
+        iid = self._next_id("appr")
+        inst = {
+            "id": iid,
+            "status": "pending",
+            "current_step": 0,
+            "steps": [],
+            "create_time": "2026-08-01T00:00:00Z",
+            **payload,
+        }
+        self._approval_instances[iid] = inst
+        return dict(inst)
+
+    def decide_approval_step(self, instance_id: str, step_index: int, *, approver_id: str, approved: bool, comment: str | None = None) -> dict[str, Any]:
+        inst = self._approval_instances[instance_id]
+        inst["status"] = "approved" if approved else "rejected"
+        inst["current_step"] = step_index + 1
+        return dict(inst)
+
+    def audit_logs(self, **kw) -> dict[str, Any]:
+        items = [dict(l) for l in self._audit_logs]
+        for filter_key in ("tenant_id", "project_id", "actor_id", "action", "target_type"):
+            val = kw.get(filter_key)
+            if val:
+                items = [l for l in items if l.get(filter_key) == val]
+        limit = kw.get("limit", 50)
+        offset = kw.get("offset", 0)
+        total = len(items)
+        items = items[offset:offset + limit]
+        return {"items": items, "total": total}
+
+    def record_audit(self, **kw) -> None:
+        log = {"id": self._next_id("log"), "time": "2026-08-01T12:00:00Z", **kw}
+        self._audit_logs.append(log)
+
+    # ---- 留存策略 ----
+
+    def retention_policies(self) -> list[dict[str, Any]]:
+        return list(self._retention_policies.values())
+
+    def retention_policy(self, policy_id: str) -> dict[str, Any]:
+        if policy_id not in self._retention_policies:
+            raise KeyError(policy_id)
+        return dict(self._retention_policies[policy_id])
+
+    def create_retention_policy(self, payload: dict[str, Any]) -> dict[str, Any]:
+        pid = payload.get("id") or self._next_id("rpol")
+        p = {"id": pid, "status": "active", "create_time": "2026-08-01T00:00:00Z", **payload}
+        self._retention_policies[pid] = p
+        return dict(p)
+
+    def update_retention_policy(self, policy_id: str, payload: dict[str, Any]) -> dict[str, Any]:
+        self._retention_policies[policy_id].update(payload)
+        return dict(self._retention_policies[policy_id])
+
+    def delete_retention_policy(self, policy_id: str) -> None:
+        self._retention_policies.pop(policy_id, None)
+
+    # ---- dashboard & other stubs ----
+
+    def login_duration_rank(self, limit: int = 8, days: int = 30) -> list[dict[str, Any]]:
+        return [{"user": "admin", "role": "admin", "duration": 10.0}]
+
+    def models(self) -> list[dict[str, Any]]:
+        return self._models
+
+    def datasets(self) -> list[dict[str, Any]]:
+        return self._datasets
+
+    def tasks(self) -> list[dict[str, Any]]:
+        return self._tasks
+
+    def eval_tasks(self) -> list[dict[str, Any]]:
+        return []
+
+    def compute_nodes(self) -> list[dict[str, Any]]:
+        return self._compute_nodes
+
+    def gpus(self) -> list[dict[str, Any]]:
+        return self._gpus
+
+    def compare_tasks(self) -> list[dict[str, Any]]:
+        return []
+
+    def trained_models(self) -> list[dict[str, Any]]:
+        return []
+
+    def system_info(self) -> dict[str, Any]:
+        return {"cpu": {}, "memory": {}}
+
+
+# ============================================================
+# 测试 fixtures
+# ============================================================
+
+@pytest.fixture(scope="module")
+def fake_store() -> FakePlatformStore:
+    return FakePlatformStore()
+
+
+def _build_client(store: FakePlatformStore) -> TestClient:
+    """构建 TestClient，patch 所有治理模块的 get_platform_store。"""
+    app = FastAPI()
+    app.include_router(platform_router, prefix=PREFIX)
+    app.include_router(system_router, prefix=PREFIX)
+    app.include_router(tenant_router, prefix=PREFIX)
+    app.include_router(project_router, prefix=PREFIX)
+    app.include_router(approval_router, prefix=PREFIX)
+    app.include_router(retention_router, prefix=PREFIX)
+    app.include_router(resource_router, prefix=PREFIX)
+
+    patches = [
+        patch("app.db.platform_store.get_platform_store", return_value=store),
+        patch("app.core.auth.get_platform_store", return_value=store),
+        patch("app.api.v1.endpoints.platform.get_platform_store", return_value=store),
+        patch("app.modules.system.router.get_platform_store", return_value=store),
+        patch("app.modules.tenant.router.get_platform_store", return_value=store),
+        patch("app.modules.project.router.get_platform_store", return_value=store),
+        patch("app.modules.approval.router.get_platform_store", return_value=store),
+        patch("app.modules.retention.router.get_platform_store", return_value=store),
+        patch("app.modules.resource.router.get_platform_store", return_value=store),
+    ]
+    for p in patches:
+        p.start()
+    client = TestClient(app, raise_server_exceptions=False)
+    client._fake_store = store  # type: ignore[attr-defined]
+    return client
+
+
+@pytest.fixture(scope="module")
+def client(fake_store: FakePlatformStore) -> TestClient:
+    c = _build_client(fake_store)
+    yield c
+
+
+def _admin_headers() -> dict[str, str]:
+    return {"Authorization": f"Bearer {ADMIN_TOKEN}"}
+
+
+def _op_headers() -> dict[str, str]:
+    return {"Authorization": f"Bearer {OP_TOKEN}"}
+
+
+# ============================================================
+# 第 1 周测试：登录、当前用户、用户列表、权限码、日志查询
+# ============================================================
+
+class TestWeek1AuthUserPermissionsLogs:
+    """第 1 周：登录、当前用户、用户列表、权限码、日志查询接口。"""
+
+    def test_login_success(self, client: TestClient):
+        resp = client.post(f"{PREFIX}/login", json={"username": "admin", "password": "admin123"})
+        assert resp.status_code == 200
+        data = resp.json()["data"]
+        assert data["token"] == ADMIN_TOKEN
+        assert data["user"]["username"] == "admin"
+
+    def test_login_invalid(self, client: TestClient):
+        resp = client.post(f"{PREFIX}/login", json={"username": "admin", "password": "wrong"})
+        assert resp.status_code == 401
+
+    def test_me_with_valid_token(self, client: TestClient):
+        resp = client.get(f"{PREFIX}/me", headers=_admin_headers())
+        assert resp.status_code == 200
+        assert resp.json()["data"]["username"] == "admin"
+
+    def test_me_without_token(self, client: TestClient):
+        resp = client.get(f"{PREFIX}/me")
+        assert resp.status_code == 401
+
+    def test_users_list(self, client: TestClient):
+        resp = client.get(f"{PREFIX}/users", headers=_admin_headers())
+        assert resp.status_code == 200
+        users = resp.json()["data"]
+        assert len(users) >= 2
+        assert any(u["username"] == "admin" for u in users)
+
+    def test_create_user(self, client: TestClient):
+        resp = client.post(
+            f"{PREFIX}/users",
+            json={"username": "tester", "display_name": "Tester", "role": "viewer", "password": "test123"},
+            headers=_admin_headers(),
+        )
+        assert resp.status_code == 200
+        assert resp.json()["data"]["username"] == "tester"
+
+    def test_permission_codes(self, client: TestClient):
+        resp = client.get(f"{PREFIX}/system/permissions/codes")
+        assert resp.status_code == 200
+        codes = resp.json()["data"]["codes"]
+        assert "dashboard" in codes
+        assert "user-settings" in codes
+
+    def test_permissions_overview(self, client: TestClient):
+        resp = client.get(f"{PREFIX}/system/permissions")
+        assert resp.status_code == 200
+        data = resp.json()["data"]
+        assert "codes" in data
+        assert "roles" in data
+
+    def test_log_files(self, client: TestClient):
+        resp = client.get(f"{PREFIX}/log-files", headers=_admin_headers())
+        assert resp.status_code == 200
+        files = resp.json()["data"]
+        assert len(files) >= 1
+
+    def test_log_content(self, client: TestClient):
+        resp = client.get(f"{PREFIX}/log-content", params={"file": "backend.log"}, headers=_admin_headers())
+        assert resp.status_code == 200
+        assert "content" in resp.json()["data"]
+
+    def test_training_log_files(self, client: TestClient):
+        resp = client.get(f"{PREFIX}/training-log-files", headers=_admin_headers())
+        assert resp.status_code == 200
+        assert len(resp.json()["data"]) >= 1
+
+    def test_training_log_content(self, client: TestClient):
+        resp = client.get(f"{PREFIX}/training-log-content", params={"file": "ft_001.log"}, headers=_admin_headers())
+        assert resp.status_code == 200
+        assert "content" in resp.json()["data"]
+
+
+# ============================================================
+# 第 2 周测试：租户、项目、项目成员、资源 ACL
+# ============================================================
+
+class TestWeek2TenantProjectACL:
+    """第 2 周：租户、项目、项目成员、资源 ACL。"""
+
+    def test_tenant_crud(self, client: TestClient):
+        # 创建
+        resp = client.post(f"{PREFIX}/tenants", json={"name": "Tenant-A", "code": "ta"}, headers=_admin_headers())
+        assert resp.status_code == 200
+        tid = resp.json()["data"]["id"]
+        # 查列表
+        resp = client.get(f"{PREFIX}/tenants", headers=_admin_headers())
+        assert resp.status_code == 200
+        assert any(t["id"] == tid for t in resp.json()["data"])
+        # 查详情
+        resp = client.get(f"{PREFIX}/tenants/{tid}", headers=_admin_headers())
+        assert resp.status_code == 200
+        assert resp.json()["data"]["name"] == "Tenant-A"
+        # 更新
+        resp = client.put(f"{PREFIX}/tenants/{tid}", json={"name": "Tenant-A2"}, headers=_admin_headers())
+        assert resp.status_code == 200
+        assert resp.json()["data"]["name"] == "Tenant-A2"
+
+    def test_tenant_quota(self, client: TestClient):
+        resp = client.post(f"{PREFIX}/tenants", json={"name": "Q-Tenant", "code": "qt"}, headers=_admin_headers())
+        tid = resp.json()["data"]["id"]
+        resp = client.put(f"{PREFIX}/tenants/{tid}/quota", json={"quota": {"gpu": 4}}, headers=_admin_headers())
+        assert resp.status_code == 200
+
+    def test_tenant_retention(self, client: TestClient):
+        resp = client.post(f"{PREFIX}/tenants", json={"name": "R-Tenant", "code": "rt"}, headers=_admin_headers())
+        tid = resp.json()["data"]["id"]
+        resp = client.put(f"{PREFIX}/tenants/{tid}/retention-policy", json={"retention_policy_id": "rpol_1"}, headers=_admin_headers())
+        assert resp.status_code == 200
+
+    def test_project_crud(self, client: TestClient):
+        # 创建项目
+        resp = client.post(f"{PREFIX}/projects", json={"name": "Proj-1", "code": "p1", "tenant_id": "default"}, headers=_admin_headers())
+        assert resp.status_code == 200
+        pid = resp.json()["data"]["id"]
+        # 查列表
+        resp = client.get(f"{PREFIX}/projects", params={"tenant_id": "default"}, headers=_admin_headers())
+        assert resp.status_code == 200
+        assert any(p["id"] == pid for p in resp.json()["data"])
+        # 查详情
+        resp = client.get(f"{PREFIX}/projects/{pid}", headers=_admin_headers())
+        assert resp.status_code == 200
+        assert resp.json()["data"]["name"] == "Proj-1"
+        # 更新
+        resp = client.put(f"{PREFIX}/projects/{pid}", json={"description": "updated"}, headers=_admin_headers())
+        assert resp.status_code == 200
+        # 归档
+        resp = client.post(f"{PREFIX}/projects/{pid}/archive", headers=_admin_headers())
+        assert resp.status_code == 200
+        assert resp.json()["data"]["status"] == "archived"
+
+    def test_project_members(self, client: TestClient):
+        resp = client.post(f"{PREFIX}/projects", json={"name": "Proj-M", "code": "pm", "tenant_id": "default"}, headers=_admin_headers())
+        pid = resp.json()["data"]["id"]
+        # 加成员
+        resp = client.post(f"{PREFIX}/projects/{pid}/members", json={"user_id": "u_op", "role": "developer"}, headers=_admin_headers())
+        assert resp.status_code == 200
+        # 列成员
+        resp = client.get(f"{PREFIX}/projects/{pid}/members", headers=_admin_headers())
+        assert resp.status_code == 200
+        assert len(resp.json()["data"]) >= 1
+        # 改角色
+        resp = client.put(f"{PREFIX}/projects/{pid}/members/u_op", json={"role": "maintainer"}, headers=_admin_headers())
+        assert resp.status_code == 200
+        # 删成员
+        resp = client.delete(f"{PREFIX}/projects/{pid}/members/u_op", headers=_admin_headers())
+        assert resp.status_code == 200
+
+    def test_resource_acl(self, client: TestClient):
+        # 设置 ACL
+        resp = client.put(
+            f"{PREFIX}/resources/model/m001/acl",
+            json={"entries": [{"subject_type": "user", "subject_id": "u_op", "permissions": ["read", "write"]}]},
+            headers=_admin_headers(),
+        )
+        assert resp.status_code == 200
+        result = resp.json()["data"]
+        assert len(result) == 1
+        assert set(result[0]["permissions"]) == {"read", "write"}
+        # 查询 ACL
+        resp = client.get(f"{PREFIX}/resources/model/m001/acl", headers=_admin_headers())
+        assert resp.status_code == 200
+        assert len(resp.json()["data"]) == 1
+
+
+# ============================================================
+# 第 3 周测试：审批实例、审批模板、审计日志查询和导出
+# ============================================================
+
+class TestWeek3ApprovalAudit:
+    """第 3 周：审批实例、审批模板、审计日志查询和导出。"""
+
+    def test_approval_template_crud(self, client: TestClient):
+        # 创建模板
+        resp = client.post(f"{PREFIX}/approvals/templates", json={"name": "delete-approval", "steps": [{"approver_id": "u_admin", "status": "pending"}]}, headers=_admin_headers())
+        assert resp.status_code == 200
+        tpl_id = resp.json()["data"]["id"]
+        # 查列表
+        resp = client.get(f"{PREFIX}/approvals/templates", headers=_admin_headers())
+        assert resp.status_code == 200
+        assert any(t["id"] == tpl_id for t in resp.json()["data"])
+
+    def test_approval_instance_flow(self, client: TestClient):
+        # 创建审批实例
+        resp = client.post(f"{PREFIX}/approvals", json={
+            "resource_type": "dataset", "resource_id": "ds_001",
+            "applicant_id": "u_op",
+        }, headers=_admin_headers())
+        assert resp.status_code == 200
+        iid = resp.json()["data"]["id"]
+        # 查详情
+        resp = client.get(f"{PREFIX}/approvals/{iid}", headers=_admin_headers())
+        assert resp.status_code == 200
+        assert resp.json()["data"]["status"] == "pending"
+        # 审批决策
+        resp = client.post(f"{PREFIX}/approvals/{iid}/steps/0/decision", json={
+            "approver_id": "u_admin", "approved": True, "comment": "ok",
+        }, headers=_admin_headers())
+        assert resp.status_code == 200
+        assert resp.json()["data"]["status"] == "approved"
+
+    def test_approval_instance_reject(self, client: TestClient):
+        resp = client.post(f"{PREFIX}/approvals", json={
+            "resource_type": "model", "resource_id": "m_002",
+            "applicant_id": "u_op",
+        }, headers=_admin_headers())
+        iid = resp.json()["data"]["id"]
+        resp = client.post(f"{PREFIX}/approvals/{iid}/steps/0/decision", json={
+            "approver_id": "u_admin", "approved": False, "comment": "no",
+        }, headers=_admin_headers())
+        assert resp.status_code == 200
+        assert resp.json()["data"]["status"] == "rejected"
+
+    def test_approval_missing_field(self, client: TestClient):
+        resp = client.post(f"{PREFIX}/approvals", json={"resource_type": "dataset"}, headers=_admin_headers())
+        assert resp.status_code == 400
+
+    def test_audit_logs_query(self, client: TestClient):
+        # 通过 API 写操作触发审计
+        client.post(f"{PREFIX}/tenants", json={"name": "Audit-Tenant", "code": "at"}, headers=_admin_headers())
+        # 查询
+        resp = client.get(f"{PREFIX}/system/audit-logs", params={"limit": 50}, headers=_admin_headers())
+        assert resp.status_code == 200
+        data = resp.json()["data"]
+        assert "items" in data
+        assert "total" in data
+        assert data["total"] >= 1
+
+    def test_audit_logs_filter_by_action(self, client: TestClient):
+        resp = client.get(f"{PREFIX}/system/audit-logs", params={"action": "tenant.create"}, headers=_admin_headers())
+        assert resp.status_code == 200
+        items = resp.json()["data"]["items"]
+        assert all(i.get("action") == "tenant.create" for i in items)
+
+    def test_audit_logs_export_csv(self, client: TestClient):
+        resp = client.get(f"{PREFIX}/system/audit-logs/export", headers=_admin_headers())
+        assert resp.status_code == 200
+        assert "text/csv" in resp.headers.get("content-type", "")
+        # CSV 首行是表头
+        lines = resp.text.strip().split("\n")
+        assert "time" in lines[0]
+
+
+# ============================================================
+# 第 4 周测试：写操作审计、审批拦截、权限校验
+# ============================================================
+
+class TestWeek4AuditInterceptPermission:
+    """第 4 周：写操作审计、审批拦截、权限校验。"""
+
+    def test_write_operation_produces_audit(self, client: TestClient, fake_store: FakePlatformStore):
+        # 清空审计日志便于断言
+        fake_store._audit_logs.clear()
+        # 创建租户 → 应产生 tenant.create 审计
+        client.post(f"{PREFIX}/tenants", json={"name": "W-Tenant", "code": "wt"}, headers=_admin_headers())
+        assert any(l["action"] == "tenant.create" for l in fake_store._audit_logs)
+        # 创建项目 → 应产生 project.create 审计
+        client.post(f"{PREFIX}/projects", json={"name": "W-Proj", "code": "wp", "tenant_id": "default"}, headers=_admin_headers())
+        assert any(l["action"] == "project.create" for l in fake_store._audit_logs)
+        # 设置 ACL → 应产生 resource.acl.set 审计
+        client.put(f"{PREFIX}/resources/model/w001/acl", json={"entries": []}, headers=_admin_headers())
+        assert any(l["action"] == "resource.acl.set" for l in fake_store._audit_logs)
+
+    def test_approval_intercept_on_project_archive(self, client: TestClient, fake_store: FakePlatformStore):
+        # 创建项目
+        resp = client.post(f"{PREFIX}/projects", json={"name": "I-Proj", "code": "ip", "tenant_id": "default"}, headers=_admin_headers())
+        pid = resp.json()["data"]["id"]
+        # 无待审批 → 可归档
+        resp = client.post(f"{PREFIX}/projects/{pid}/archive", headers=_admin_headers())
+        assert resp.status_code == 200
+
+    def test_approval_intercept_blocks_when_pending(self, client: TestClient, fake_store: FakePlatformStore):
+        # 创建项目
+        resp = client.post(f"{PREFIX}/projects", json={"name": "B-Proj", "code": "bp", "tenant_id": "default"}, headers=_admin_headers())
+        pid = resp.json()["data"]["id"]
+        # 注入一条待审批实例
+        fake_store.create_approval_instance({
+            "resource_type": "project",
+            "resource_id": pid,
+            "applicant_id": "u_op",
+        })
+        # 有待审批 → 归档应被拒绝
+        resp = client.post(f"{PREFIX}/projects/{pid}/archive", headers=_admin_headers())
+        assert resp.status_code == 409
+
+    def test_retention_policy_crud_with_audit(self, client: TestClient, fake_store: FakePlatformStore):
+        fake_store._audit_logs.clear()
+        # 创建
+        resp = client.post(f"{PREFIX}/retention-policies", json={"name": "30d-keep", "scope": "tenant"}, headers=_admin_headers())
+        assert resp.status_code == 200
+        rpid = resp.json()["data"]["id"]
+        assert any(l["action"] == "retention.create" for l in fake_store._audit_logs)
+        # 查列表
+        resp = client.get(f"{PREFIX}/retention-policies", headers=_admin_headers())
+        assert resp.status_code == 200
+        assert any(p["id"] == rpid for p in resp.json()["data"])
+        # 更新
+        resp = client.put(f"{PREFIX}/retention-policies/{rpid}", json={"status": "inactive"}, headers=_admin_headers())
+        assert resp.status_code == 200
+        assert resp.json()["data"]["status"] == "inactive"
+        # 删除
+        resp = client.delete(f"{PREFIX}/retention-policies/{rpid}", headers=_admin_headers())
+        assert resp.status_code == 200
+
+    def test_login_duration_rank_in_dashboard(self, client: TestClient):
+        resp = client.get(f"{PREFIX}/dashboard/stats", headers=_admin_headers())
+        assert resp.status_code == 200
+        data = resp.json()["data"]
+        assert "login_duration_rank" in data
+        assert "recent_login_users" in data
+        assert "service_status" in data
+        assert "training_7d" in data

backend/tests/test_platform_dataset_metadata.py

@@ -0,0 +1,103 @@
+from __future__ import annotations
+
+from contextlib import contextmanager
+from typing import Any, Iterator
+
+from app.db.platform_store import PlatformStore, dataset_file_version_summary, parse_size_bytes
+
+
+class _DatasetCursor:
+    def __init__(self, rows: list[dict[str, Any]]) -> None:
+        self.rows = rows
+
+    def fetchall(self) -> list[dict[str, Any]]:
+        return self.rows
+
+
+class _DatasetConnection:
+    def __init__(self) -> None:
+        self.queries: list[str] = []
+
+    def execute(self, sql: str, params: tuple[Any, ...] | None = None) -> _DatasetCursor:
+        self.queries.append(sql)
+        if "FROM datasets dataset" in sql:
+            return _DatasetCursor(
+                [
+                    {
+                        "id": "dataset-train",
+                        "name": "cash-数据集-训练集",
+                        "type": "train",
+                        "storage_type": "local",
+                        "source": "task",
+                        "task_id": "task-cash",
+                        "source_task_id": "task-cash",
+                        "task_name": "cash",
+                        "size": "0 B",
+                        "size_bytes": 0,
+                        "metadata": "{}",
+                    }
+                ]
+            )
+        if "FROM dataset_files" in sql or "FROM dataset_records" in sql:
+            return _DatasetCursor([])
+        raise AssertionError(f"unexpected query: {sql}")
+
+
+def test_parse_size_bytes_supports_legacy_units() -> None:
+    assert parse_size_bytes("21563 B") == 21563
+    assert parse_size_bytes("1.5 KB") == 1536
+    assert parse_size_bytes("2 MB") == 2 * 1024**2
+    assert parse_size_bytes(4096) == 4096
+    assert parse_size_bytes("unknown") == 0
+
+
+def test_dataset_file_version_summary_uses_active_version_metadata() -> None:
+    summary = dataset_file_version_summary(
+        {
+            "active_version_id": "file-1-v3",
+            "current_version_id": "file-1-v1",
+            "version_no": 1,
+            "versions": (
+                '[{"id":"file-1-v1","version":1},'
+                '{"id":"file-1-v3","version_no":3}]'
+            ),
+        }
+    )
+
+    assert summary == {
+        "active_version_id": "file-1-v3",
+        "current_version_id": "file-1-v3",
+        "current_version_no": 3,
+        "version_count": 2,
+    }
+
+
+def test_dataset_file_version_summary_uses_normalized_version_number_as_fallback() -> None:
+    summary = dataset_file_version_summary(
+        {
+            "active_version_id": "",
+            "current_version_id": None,
+            "version_no": 1,
+            "versions": "[]",
+        }
+    )
+
+    assert summary["current_version_no"] == 1
+    assert summary["version_count"] == 0
+
+
+def test_dataset_list_exposes_source_task_name() -> None:
+    store = PlatformStore.__new__(PlatformStore)
+    conn = _DatasetConnection()
+
+    @contextmanager
+    def connect() -> Iterator[_DatasetConnection]:
+        yield conn
+
+    store.connect = connect  # type: ignore[method-assign]
+
+    [dataset] = store.datasets()
+
+    assert dataset["task_name"] == "cash"
+    assert dataset["name"] == "cash-数据集-训练集"
+    assert any("task.name AS task_name" in query for query in conn.queries)

compute/README.md

@@ -25,5 +25,57 @@ compute/
 
 ## 运行模式
 
-- 默认 `COMPUTE_EXECUTION_MODE=real`，Compute API 只暴露健康检查和接口契约；真实训练执行器完成前，创建作业会返回未实现错误。
+- 默认 `COMPUTE_EXECUTION_MODE=real`，Compute API 会通过 `compute.agent.process_manager.ProcessManager` 启动真实 `llamafactory-cli train` 子进程，并将日志写入 `TRAINING_LOG_ROOT`。
+- 真实模式下 GPU 发现优先使用宿主机 `nvidia-smi`。如果部署环境暂时无法调用 `nvidia-smi`，可通过 `COMPUTE_GPU_COUNT`、`COMPUTE_GPU_NAME`、`COMPUTE_GPU_MEMORY_GB`、`COMPUTE_GPU_POWER_LIMIT_W` 声明兼容 GPU 清单，便于应用侧先完成节点登记和联调。
 - 仅隔离联调时可设置 `COMPUTE_EXECUTION_MODE=simulator`，启用内存状态机和合成 GPU/日志数据。该模式不得作为生产运行路径。
+- 服务间鉴权默认开启：设置 `COMPUTE_AUTH_ENABLED=true` 和一致的 `COMPUTE_SERVICE_TOKEN`，应用侧会通过 `X-Compute-Token` 调用 Compute API。
+- 真实训练作业会登记到 `TRAINING_LOG_ROOT/compute-jobs.json`。Compute API 重启后会恢复作业索引，继续提供状态、停止和日志查询。
+- 同一算力节点内按 GPU ID 做轻量锁定；已有运行中作业占用的 GPU 不允许再次提交，避免同机多 GPU 场景下误复用。
+
+真实执行前提：
+
+- 镜像或宿主机环境中 `llamafactory-cli` 可执行。
+- `LLAMA_FACTORY_HOME` 指向 LLaMA-Factory 工作目录。
+- 基座模型路径和数据集名称/目录已经在算力服务器本地可访问。
+- 应用侧训练任务中的 GPU、模型、数据集配置能映射到当前节点本地路径。
+
+## 应用侧接入
+
+应用平台通过“算力节点”页面维护每台 GPU 服务器的 `Compute API` 和 `File Gateway` 地址。点击连接测试时，Backend API 会主动调用：
+
+```text
+GET /modelTF/v1/compute/health
+GET /modelTF/compute/resources/gpus
+```
+
+连接成功后，应用侧会同步节点健康信息、能力标签和 GPU 清单到 PostgreSQL。多节点阶段仍按“每台算力服务器 = 单机多 GPU 节点”管理，每台服务器都部署 Compute API、Agent、File Gateway 契约和 LLaMA-Factory。
+
+训练闭环：
+
+```text
+Frontend 创建/启动训练
+  -> Backend API 选择 compute_nodes 节点
+  -> Backend API POST /modelTF/compute/jobs 到目标 Compute API
+  -> Compute API 启动 llamafactory-cli 子进程
+  -> Backend Worker 定时 GET /modelTF/compute/jobs/{id}
+  -> Backend API 同步 fine_tune_tasks 状态、进度、PID、日志路径和产物索引
+```
+
+## 当前接口能力
+
+日志接口：
+
+```text
+GET /modelTF/compute/jobs/{job_id}/logs?tail_lines=200
+GET /modelTF/compute/jobs/{job_id}/logs?offset=0&limit=500
+```
+
+返回 `content`、`metrics`、`total_lines`、`offset`、`limit`、`has_more`、`next_offset`，用于前端增量刷新和日志平台采集。
+
+文件导入：
+
+```text
+POST /modelTF/compute/files/import-local
+```
+
+该接口用于应用侧调度前把算力服务器本地可访问的模型/数据集路径导入到 `YG_FT_DATA_ROOT` 内部。目标路径会校验不能逃逸出 `YG_FT_DATA_ROOT`，源路径必须已存在于算力服务器本地或挂载目录。

compute/agent/process_manager.py

@@ -0,0 +1,281 @@
+from __future__ import annotations
+
+import os
+import json
+import contextlib
+import hashlib
+import signal
+import subprocess
+import time
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Any
+
+
+TERMINAL_STATUSES = {"completed", "failed", "stopped"}
+
+
+@dataclass
+class ManagedProcess:
+    id: str
+    name: str
+    command: list[str]
+    work_dir: str
+    log_path: Path
+    output_dir: str
+    gpus: list[int]
+    process: subprocess.Popen[Any] | None
+    created_at: float
+    pid: int | None = None
+    status: str = "running"
+    progress: int = 5
+    artifacts: list[dict[str, Any]] = field(default_factory=list)
+
+
+class ProcessManager:
+    def __init__(self, log_root: str) -> None:
+        self.log_root = Path(log_root)
+        self.log_root.mkdir(parents=True, exist_ok=True)
+        self.registry_path = self.log_root / "compute-jobs.json"
+        self.jobs: dict[str, ManagedProcess] = {}
+        self._load_registry()
+
+    def create_job(self, payload: dict[str, Any], command: list[str], work_dir: str) -> dict[str, Any]:
+        job_id = str(payload.get("id") or f"job_{int(time.time() * 1000)}")
+        if job_id in self.jobs and self.jobs[job_id].status not in TERMINAL_STATUSES:
+            raise ValueError(f"job {job_id} is already running")
+
+        output_dir = str(payload.get("output_dir") or f"/data/yg-ft/outputs/{payload.get('name', job_id)}")
+        Path(output_dir).mkdir(parents=True, exist_ok=True)
+        log_path = self.log_root / f"{job_id}.log"
+        env = os.environ.copy()
+        gpus = [int(item) for item in payload.get("gpus") or []]
+        locked = self.locked_gpus()
+        conflict = sorted(set(gpus).intersection(locked))
+        if conflict:
+            raise ValueError(f"gpu already locked: {conflict}")
+        if gpus:
+            env["CUDA_VISIBLE_DEVICES"] = ",".join(str(item) for item in gpus)
+        env.update({str(k): str(v) for k, v in payload.get("env", {}).items()})
+
+        cwd = work_dir if Path(work_dir).exists() else None
+        with log_path.open("ab") as log_file:
+            log_file.write(f"[INFO] starting job_id={job_id} command={' '.join(command)}\n".encode("utf-8"))
+            process = subprocess.Popen(
+                command,
+                cwd=cwd,
+                env=env,
+                stdout=log_file,
+                stderr=subprocess.STDOUT,
+            )
+
+        managed = ManagedProcess(
+            id=job_id,
+            name=str(payload.get("name") or job_id),
+            command=command,
+            work_dir=work_dir,
+            log_path=log_path,
+            output_dir=output_dir,
+            gpus=gpus,
+            process=process,
+            created_at=time.time(),
+            pid=process.pid,
+            progress=10,
+        )
+        self.jobs[job_id] = managed
+        data = self.serialize(managed)
+        self._save_registry()
+        return data
+
+    def get_job(self, job_id: str) -> dict[str, Any] | None:
+        job = self.jobs.get(job_id)
+        if not job:
+            return None
+        return self.serialize(job)
+
+    def list_jobs(self) -> list[dict[str, Any]]:
+        return [self.serialize(job) for job in self.jobs.values()]
+
+    def stop_job(self, job_id: str) -> dict[str, Any] | None:
+        job = self.jobs.get(job_id)
+        if not job:
+            return None
+        if job.status not in TERMINAL_STATUSES:
+            try:
+                if job.process is not None and os.name == "nt":
+                    job.process.terminate()
+                elif job.pid is not None:
+                    os.kill(job.pid, signal.SIGTERM)
+                if job.process is not None:
+                    job.process.wait(timeout=10)
+            except Exception:
+                if job.process is not None:
+                    job.process.kill()
+                elif job.pid is not None:
+                    with contextlib.suppress(Exception):
+                        os.kill(job.pid, signal.SIGKILL)
+            job.status = "stopped"
+            job.progress = min(job.progress, 99)
+        data = self.serialize(job)
+        self._save_registry()
+        return data
+
+    def logs(self, job_id: str) -> str:
+        job = self.jobs.get(job_id)
+        if not job or not job.log_path.exists():
+            return ""
+        return job.log_path.read_text(encoding="utf-8", errors="replace")
+
+    def serialize(self, job: ManagedProcess) -> dict[str, Any]:
+        code = job.process.poll() if job.process is not None else None
+        checkpoints = self._collect_checkpoints(job.output_dir)
+        if job.status not in TERMINAL_STATUSES:
+            if job.process is None and job.pid is not None and not self._pid_alive(job.pid):
+                job.status = "failed"
+                job.progress = min(job.progress, 99)
+                code = -1
+            elif code is None:
+                job.status = "running"
+                elapsed = max(0, int(time.time() - job.created_at))
+                job.progress = min(95, max(job.progress, 10 + elapsed // 6))
+            elif code == 0:
+                job.status = "completed"
+                job.progress = 100
+                job.artifacts = self._collect_artifacts(job.output_dir)
+            else:
+                job.status = "failed"
+                job.progress = min(job.progress, 99)
+        self._save_registry()
+        return {
+            "id": job.id,
+            "name": job.name,
+            "status": job.status,
+            "progress": job.progress,
+            "pid": job.pid,
+            "gpus": job.gpus,
+            "created_at": job.created_at,
+            "command": job.command,
+            "work_dir": job.work_dir,
+            "output_dir": job.output_dir,
+            "log_file": str(job.log_path),
+            "artifacts": job.artifacts,
+            "checkpoints": checkpoints,
+            "return_code": code,
+        }
+
+    def locked_gpus(self) -> set[int]:
+        locked: set[int] = set()
+        for job in self.jobs.values():
+            status = self.serialize(job)["status"]
+            if status in {"queued", "running"}:
+                locked.update(job.gpus)
+        return locked
+
+    def _collect_artifacts(self, output_dir: str) -> list[dict[str, Any]]:
+        root = Path(output_dir)
+        if not root.exists():
+            return []
+        artifacts: list[dict[str, Any]] = []
+        for path in root.rglob("*"):
+            if path.is_file():
+                digest = hashlib.sha256()
+                with path.open("rb") as handle:
+                    for chunk in iter(lambda: handle.read(1024 * 1024), b""):
+                        digest.update(chunk)
+                size = path.stat().st_size
+                artifacts.append(
+                    {
+                        "path": str(path),
+                        "name": path.name,
+                        "size": size,
+                        "size_bytes": size,
+                        "checksum_sha256": digest.hexdigest(),
+                    }
+                )
+        return artifacts[:200]
+
+    def _collect_checkpoints(self, output_dir: str) -> list[dict[str, Any]]:
+        root = Path(output_dir)
+        if not root.exists():
+            return []
+        checkpoints: list[dict[str, Any]] = []
+        for path in root.glob("checkpoint-*"):
+            if not path.is_dir():
+                continue
+            step = 0
+            try:
+                step = int(path.name.rsplit("-", 1)[-1])
+            except ValueError:
+                step = 0
+            size_bytes = sum(item.stat().st_size for item in path.rglob("*") if item.is_file())
+            checkpoints.append(
+                {
+                    "step": step,
+                    "name": path.name,
+                    "path": str(path),
+                    "size_bytes": size_bytes,
+                    "create_time": path.stat().st_mtime,
+                }
+            )
+        return sorted(checkpoints, key=lambda item: (int(item.get("step") or 0), str(item.get("name") or "")))
+
+    def _save_registry(self) -> None:
+        items = []
+        for job in self.jobs.values():
+            items.append(
+                {
+                    "id": job.id,
+                    "name": job.name,
+                    "command": job.command,
+                    "work_dir": job.work_dir,
+                    "log_path": str(job.log_path),
+                    "output_dir": job.output_dir,
+                    "gpus": job.gpus,
+                    "pid": job.pid,
+                    "created_at": job.created_at,
+                    "status": job.status,
+                    "progress": job.progress,
+                    "artifacts": job.artifacts,
+                }
+            )
+        self.registry_path.write_text(json.dumps(items, ensure_ascii=False, indent=2), encoding="utf-8")
+
+    def _load_registry(self) -> None:
+        if not self.registry_path.exists():
+            return
+        try:
+            items = json.loads(self.registry_path.read_text(encoding="utf-8"))
+        except json.JSONDecodeError:
+            return
+        for item in items if isinstance(items, list) else []:
+            if not isinstance(item, dict):
+                continue
+            pid = item.get("pid")
+            status = item.get("status", "failed")
+            if status not in TERMINAL_STATUSES and pid and not self._pid_alive(int(pid)):
+                status = "failed"
+            job = ManagedProcess(
+                id=str(item["id"]),
+                name=str(item.get("name") or item["id"]),
+                command=[str(part) for part in item.get("command") or []],
+                work_dir=str(item.get("work_dir") or ""),
+                log_path=Path(item.get("log_path") or self.log_root / f"{item['id']}.log"),
+                output_dir=str(item.get("output_dir") or ""),
+                gpus=[int(gpu) for gpu in item.get("gpus") or []],
+                process=None,
+                pid=int(pid) if pid else None,
+                created_at=float(item.get("created_at") or time.time()),
+                status=status,
+                progress=int(item.get("progress") or 0),
+                artifacts=item.get("artifacts") or [],
+            )
+            self.jobs[job.id] = job
+
+    def _pid_alive(self, pid: int) -> bool:
+        if pid <= 0:
+            return False
+        try:
+            os.kill(pid, 0)
+            return True
+        except OSError:
+            return False

compute/api/main.py

@@ -1,20 +1,43 @@
 from __future__ import annotations
 
+import asyncio
+import json
 import os
 import math
+import hashlib
+import shutil
+import subprocess
 import time
 from pathlib import Path
 from typing import Any
 
-from fastapi import FastAPI, HTTPException
+from fastapi import FastAPI, File, Form, HTTPException, Query, Request, UploadFile
+from fastapi.responses import FileResponse, JSONResponse, StreamingResponse
 
-from compute.engines.llama_factory.adapter import build_command, parse_log_line
+from compute.agent.process_manager import ProcessManager
+from compute.api.security import docs_kwargs
+from compute.engines.llama_factory.adapter import build_command, parse_log_line, prepare_runtime_files
+from compute.engines.llama_factory.inference import get_inference_session
 
 
 def create_app() -> FastAPI:
-    app = FastAPI(title="YG Fine-Tune Compute API")
+    app = FastAPI(title="YG Fine-Tune Compute API", **docs_kwargs())
     jobs: dict[str, dict[str, Any]] = {}
     route_prefix = os.getenv("MODELTF_ROUTE_PREFIX", "/modelTF").rstrip("/") or "/modelTF"
+    process_manager = ProcessManager(os.getenv("TRAINING_LOG_ROOT", "/opt/yg-ft/logs/training"))
+
+    @app.middleware("http")
+    async def compute_token_auth(request: Request, call_next):
+        token = os.getenv("COMPUTE_SERVICE_TOKEN", "")
+        auth_enabled = os.getenv("COMPUTE_AUTH_ENABLED", "true").lower() == "true"
+        public_paths = {f"{route_prefix}/health", "/health"}
+        if auth_enabled and token and request.url.path not in public_paths:
+            header_token = request.headers.get("x-compute-token", "")
+            auth_header = request.headers.get("authorization", "")
+            bearer_token = auth_header.removeprefix("Bearer ").strip() if auth_header.startswith("Bearer ") else ""
+            if header_token != token and bearer_token != token:
+                return JSONResponse({"detail": "invalid compute service token"}, status_code=401)
+        return await call_next(request)
 
     def now() -> float:
         return time.time()
@@ -25,6 +48,110 @@ def create_app() -> FastAPI:
     def execution_mode() -> str:
         return os.getenv("COMPUTE_EXECUTION_MODE", os.getenv("COMPUTE_MODE", "real")).lower()
 
+    def _int_env(name: str, default: int) -> int:
+        raw = os.getenv(name)
+        if raw is None or raw == "":
+            return default
+        return int(raw)
+
+    def _float_env(name: str, default: float) -> float:
+        raw = os.getenv(name)
+        if raw is None or raw == "":
+            return default
+        return float(raw)
+
+    def _path_inside(root: Path, candidate: Path) -> bool:
+        try:
+            candidate.resolve().relative_to(root.resolve())
+            return True
+        except ValueError:
+            return False
+
+    def _llama_factory_version() -> str:
+        for command in (["llamafactory-cli", "version"], ["llamafactory-cli", "--version"]):
+            try:
+                result = subprocess.run(command, capture_output=True, text=True, timeout=5)
+            except Exception:
+                continue
+            output = (result.stdout or result.stderr).strip()
+            if result.returncode == 0 and output:
+                return output.splitlines()[0][:120]
+        return ""
+
+    def torch_cuda_status() -> dict[str, Any]:
+        try:
+            import torch  # type: ignore[import-not-found]
+        except Exception as exc:  # noqa: BLE001 - keep health endpoint resilient
+            return {
+                "available": False,
+                "device_count": 0,
+                "torch_version": "",
+                "torch_cuda_version": "",
+                "error": f"torch import failed: {exc}",
+            }
+        try:
+            available = bool(torch.cuda.is_available())
+            device_count = int(torch.cuda.device_count())
+            devices = []
+            for index in range(device_count):
+                props = torch.cuda.get_device_properties(index)
+                devices.append(
+                    {
+                        "index": index,
+                        "name": props.name,
+                        "memory_total_gb": round(props.total_memory / 1024 / 1024 / 1024, 2),
+                    }
+                )
+            return {
+                "available": available,
+                "device_count": device_count,
+                "torch_version": str(torch.__version__),
+                "torch_cuda_version": str(torch.version.cuda or ""),
+                "devices": devices,
+                "error": "" if available else "torch cuda is not available",
+            }
+        except Exception as exc:  # noqa: BLE001 - expose CUDA initialization failures
+            return {
+                "available": False,
+                "device_count": 0,
+                "torch_version": str(getattr(torch, "__version__", "")),
+                "torch_cuda_version": str(getattr(torch.version, "cuda", "") or ""),
+                "devices": [],
+                "error": str(exc),
+            }
+
+    def _slice_log_content(
+        content: str,
+        tail_lines: int | None = None,
+        offset: int | None = None,
+        limit: int | None = None,
+    ) -> dict[str, Any]:
+        lines = content.splitlines()
+        total = len(lines)
+        if offset is not None or limit is not None:
+            start = max(0, offset or 0)
+            end = start + limit if limit else total
+            selected = lines[start:end]
+        else:
+            tail = tail_lines or 200
+            start = max(0, total - tail)
+            selected = lines[start:]
+        next_offset = start + len(selected)
+        return {
+            "content": "\n".join(selected),
+            "total_lines": total,
+            "offset": start,
+            "limit": len(selected),
+            "has_more": next_offset < total,
+            "next_offset": next_offset if next_offset < total else None,
+        }
+
+    def _safe_float(value: Any, default: float = 0) -> float:
+        try:
+            return float(str(value).replace("[N/A]", "").strip() or default)
+        except (TypeError, ValueError):
+            return default
+
     def job_status(job: dict[str, Any]) -> dict[str, Any]:
         if execution_mode() != "simulator":
             return job
@@ -74,9 +201,80 @@ def create_app() -> FastAPI:
             )
         return "\n".join(lines)
 
+    def real_gpu_resources() -> list[dict[str, Any]]:
+        query = (
+            "index,uuid,name,memory.total,memory.used,utilization.gpu,"
+            "temperature.gpu,power.draw,power.limit"
+        )
+        try:
+            result = subprocess.run(
+                ["nvidia-smi", f"--query-gpu={query}", "--format=csv,noheader,nounits"],
+                check=True,
+                capture_output=True,
+                text=True,
+                timeout=5,
+            )
+        except Exception:
+            return fallback_gpu_resources()
+
+        items: list[dict[str, Any]] = []
+        for line in result.stdout.splitlines():
+            parts = [part.strip() for part in line.split(",")]
+            if len(parts) < 9:
+                continue
+            idx, uuid, name, mem_total, mem_used, util, temp, power, power_limit = parts[:9]
+            total_gb = round(_safe_float(mem_total) / 1024, 2)
+            used_gb = round(_safe_float(mem_used) / 1024, 2)
+            memory_percent = round(used_gb / total_gb * 100, 1) if total_gb else 0
+            gpu_percent = int(_safe_float(util))
+            items.append(
+                {
+                    "id": int(idx),
+                    "gpu_index": int(idx),
+                    "uuid": uuid,
+                    "name": name,
+                    "status": "busy" if gpu_percent >= 5 or used_gb > 1 else "idle",
+                    "gpu_percent": gpu_percent,
+                    "memory_used_gb": used_gb,
+                    "memory_total_gb": total_gb,
+                    "memory_percent": memory_percent,
+                    "temperature": int(_safe_float(temp)),
+                    "power_w": round(_safe_float(power), 1),
+                    "power_limit_w": round(_safe_float(power_limit), 1),
+                    "processes": [],
+                }
+            )
+        return items
+
+    def fallback_gpu_resources() -> list[dict[str, Any]]:
+        count = _int_env("COMPUTE_GPU_COUNT", 0)
+        if count <= 0:
+            return []
+        name = os.getenv("COMPUTE_GPU_NAME", "Configured GPU")
+        memory_total = _float_env("COMPUTE_GPU_MEMORY_GB", 80.0)
+        power_limit = _float_env("COMPUTE_GPU_POWER_LIMIT_W", 300.0)
+        return [
+            {
+                "id": idx,
+                "gpu_index": idx,
+                "uuid": f"GPU-{host_id().upper()}-{idx}",
+                "name": name,
+                "status": "idle",
+                "gpu_percent": 0,
+                "memory_used_gb": 0,
+                "memory_total_gb": memory_total,
+                "memory_percent": 0,
+                "temperature": _int_env("COMPUTE_GPU_BASE_TEMPERATURE", 35),
+                "power_w": 0,
+                "power_limit_w": power_limit,
+                "processes": [],
+            }
+            for idx in range(count)
+        ]
+
     def gpu_resources() -> list[dict[str, Any]]:
         if execution_mode() != "simulator":
-            return []
+            return real_gpu_resources()
         active_jobs = [job_status(job) for job in jobs.values() if job["status"] in {"queued", "running"}]
         gpus: list[dict[str, Any]] = []
         for idx in range(4):
@@ -109,6 +307,191 @@ def create_app() -> FastAPI:
             )
         return gpus
 
+    def _validate_training_accelerator(payload: dict[str, Any]) -> tuple[list[str], list[str], dict[str, Any]]:
+        errors: list[str] = []
+        warnings: list[str] = []
+        if str(payload.get("engine") or payload.get("training_engine") or "llama_factory") == "smoke":
+            return errors, warnings, {}
+        requested_gpus = [int(item) for item in payload.get("gpus") or []]
+        if not requested_gpus:
+            warnings.append("no gpu selected; training will run on CPU")
+            return errors, warnings, {}
+        cuda = torch_cuda_status()
+        if not cuda.get("available"):
+            errors.append(f"torch cuda unavailable on compute node: {cuda.get('error') or 'unknown error'}")
+        device_count = int(cuda.get("device_count") or 0)
+        if device_count and max(requested_gpus) >= device_count:
+            errors.append(f"requested gpu index out of torch device range: requested={requested_gpus}, device_count={device_count}")
+        min_memory_gb = _float_env("MIN_TRAINING_GPU_MEMORY_GB", 4.0)
+        gpus = {int(item["gpu_index"]): item for item in gpu_resources() if "gpu_index" in item}
+        for gpu_index in requested_gpus:
+            gpu = gpus.get(gpu_index)
+            if not gpu:
+                errors.append(f"requested gpu not found by nvidia-smi: {gpu_index}")
+                continue
+            memory_total = float(gpu.get("memory_total_gb") or 0)
+            if memory_total and memory_total < min_memory_gb:
+                errors.append(
+                    f"gpu {gpu_index} memory too small: {memory_total}GB < required {min_memory_gb}GB"
+                )
+        return errors, warnings, cuda
+
+    def _check_path_item(item: dict[str, Any]) -> dict[str, Any]:
+        path = Path(str(item.get("path") or ""))
+        exists = path.exists()
+        expected_type = str(item.get("type") or "any")
+        ok = exists
+        if exists and expected_type == "dir":
+            ok = path.is_dir()
+        if exists and expected_type == "file":
+            ok = path.is_file()
+        return {
+            "name": item.get("name") or "",
+            "path": str(path),
+            "type": expected_type,
+            "required": bool(item.get("required", True)),
+            "exists": exists,
+            "is_dir": path.is_dir() if exists else False,
+            "is_file": path.is_file() if exists else False,
+            "byte_size": sum(child.stat().st_size for child in path.rglob("*") if child.is_file()) if exists and path.is_dir() else path.stat().st_size if exists and path.is_file() else 0,
+            "ok": ok or not item.get("required", True),
+        }
+
+    def _job_preview(payload: dict[str, Any], check_paths: bool) -> dict[str, Any]:
+        warnings: list[str] = []
+        runtime_files: list[dict[str, str]] = []
+        command_payload = {**payload, "require_dataset_files": check_paths}
+        if check_paths:
+            try:
+                runtime_files = prepare_runtime_files(command_payload)
+            except OSError as exc:
+                return {
+                    "valid": False,
+                    "errors": [f"prepare runtime files failed: {exc}"],
+                    "warnings": warnings,
+                    "engine": str(payload.get("engine") or payload.get("training_engine") or "llama_factory"),
+                    "command": [],
+                    "command_text": "",
+                    "work_dir": os.getenv("LLAMA_FACTORY_HOME", "/app/LLaMA-Factory"),
+                    "env": {},
+                    "runtime_files": [],
+                    "path_checks": [],
+                }
+        try:
+            command = build_command(command_payload, os.getenv("LLAMA_FACTORY_HOME", "/app/LLaMA-Factory"))
+        except ValueError as exc:
+            return {
+                "valid": False,
+                "errors": [part.strip() for part in str(exc).split(";") if part.strip()],
+                "warnings": warnings,
+                "engine": str(payload.get("engine") or payload.get("training_engine") or "llama_factory"),
+                "command": [],
+                "command_text": "",
+                "work_dir": os.getenv("LLAMA_FACTORY_HOME", "/app/LLaMA-Factory"),
+                "env": {},
+                "runtime_files": runtime_files,
+                "path_checks": [],
+            }
+
+        errors: list[str] = []
+        engine = str(payload.get("engine") or payload.get("training_engine") or "llama_factory")
+        path_checks: list[dict[str, Any]] = []
+        accelerator: dict[str, Any] = {}
+        if check_paths and engine != "smoke":
+            path_checks = [
+                _check_path_item(
+                    {
+                        "name": "model_name_or_path",
+                        "path": payload.get("model_name_or_path") or payload.get("base_model") or payload.get("base_model_path") or "",
+                        "type": "any",
+                        "required": True,
+                    }
+                )
+            ]
+            if engine in {"merge", "export", "llama_factory_export"} and payload.get("adapter_name_or_path"):
+                path_checks.append(
+                    _check_path_item(
+                        {
+                            "name": "adapter_name_or_path",
+                            "path": payload.get("adapter_name_or_path"),
+                            "type": "any",
+                            "required": True,
+                        }
+                    )
+                )
+            if payload.get("dataset_dir"):
+                path_checks.append(
+                    _check_path_item(
+                        {
+                            "name": "dataset_dir",
+                            "path": payload.get("dataset_dir"),
+                            "type": "dir",
+                            "required": True,
+                        }
+                    )
+                )
+            output_dir = Path(str(payload.get("output_dir") or "/data/yg-ft/outputs/training-job"))
+            path_checks.append(
+                _check_path_item(
+                    {
+                        "name": "output_parent",
+                        "path": str(output_dir.parent),
+                        "type": "dir",
+                        "required": False,
+                    }
+                )
+            )
+            errors.extend(
+                [f"{item['name']} path not available: {item['path']}" for item in path_checks if not item["ok"] and item["required"]]
+            )
+            if shutil.which(command.command[0]) is None:
+                errors.append(f"training command not found: {command.command[0]}")
+            if not Path(command.work_dir).exists():
+                errors.append(f"llama_factory_home not found: {command.work_dir}")
+            if engine not in {"merge", "export", "llama_factory_export"}:
+                accelerator_errors, accelerator_warnings, accelerator = _validate_training_accelerator(payload)
+                errors.extend(accelerator_errors)
+                warnings.extend(accelerator_warnings)
+        elif engine == "eval":
+            # Eval engine: validate model path and dataset path
+            if not payload.get("model_name_or_path"):
+                errors.append("model_name_or_path is required for eval")
+            else:
+                path_checks.append(_check_path_item({
+                    "name": "model_name_or_path",
+                    "path": payload.get("model_name_or_path", ""),
+                    "type": "any",
+                    "required": True,
+                }))
+            if payload.get("dataset_path"):
+                path_checks.append(_check_path_item({
+                    "name": "dataset_path",
+                    "path": payload.get("dataset_path", ""),
+                    "type": "file",
+                    "required": True,
+                }))
+            else:
+                errors.append("dataset_path is required for eval")
+            if shutil.which("python") is None:
+                errors.append("python runtime not found")
+
+        elif engine == "smoke":
+            warnings.append("smoke engine skips model and dataset path checks")
+
+        return {
+            "valid": not errors,
+            "errors": errors,
+            "warnings": warnings,
+            "engine": engine,
+            "command": command.command,
+            "command_text": " ".join(command.command),
+            "work_dir": command.work_dir,
+            "env": command.env,
+            "runtime_files": runtime_files,
+            "accelerator": accelerator,
+            "path_checks": path_checks,
+        }
+
     @app.get(f"{route_prefix}/health")
     async def health_check() -> dict[str, str]:
         return {
@@ -116,41 +499,130 @@ def create_app() -> FastAPI:
             "compute_host_id": os.getenv("COMPUTE_HOST_ID", "unknown"),
         }
 
+    @app.get("/health")
+    async def health_check_root() -> dict[str, str]:
+        return await health_check()
+
     @app.get(f"{route_prefix}/v1/compute/health")
-    async def compute_health_check() -> dict[str, str | bool]:
+    async def compute_health_check() -> dict[str, Any]:
         data_root = Path(os.getenv("YG_FT_DATA_ROOT", "/data/yg-ft"))
+        dataset_root = Path(os.getenv("YG_FT_DATASET_ROOT", str(data_root / "datasets")))
+        output_root = Path(os.getenv("YG_FT_OUTPUT_ROOT", str(data_root / "outputs")))
         llama_factory_home = Path(os.getenv("LLAMA_FACTORY_HOME", "/app/LLaMA-Factory"))
+        gpu_items = gpu_resources()
+        torch_cuda = torch_cuda_status()
         return {
             "status": "ok",
+            "api_version": "v1",
             "compute_host_id": os.getenv("COMPUTE_HOST_ID", "unknown"),
             "app_callback_enabled": os.getenv("ENABLE_APP_CALLBACK", "false").lower() == "true",
             "data_root": str(data_root),
             "data_root_exists": data_root.exists(),
+            "model_root": os.getenv("YG_FT_MODEL_ROOT", str(data_root / "models")),
+            "dataset_root": str(dataset_root),
+            "dataset_root_exists": dataset_root.exists(),
+            "output_root": str(output_root),
+            "output_root_exists": output_root.exists(),
+            "log_root": os.getenv("TRAINING_LOG_ROOT", "/opt/yg-ft/logs/training"),
             "llama_factory_home": str(llama_factory_home),
             "llama_factory_home_exists": llama_factory_home.exists(),
+            "llama_factory_version": os.getenv("LLAMA_FACTORY_VERSION", ""),
             "execution_mode": execution_mode(),
+            "gpu_count": _int_env("COMPUTE_GPU_COUNT", 0),
+            "nvidia_gpu_count": len(gpu_items),
+            "torch_cuda": torch_cuda,
+            "gpu_discovery_endpoint": f"{route_prefix}/compute/resources/gpus",
+            "capabilities": ["gpu_discovery", "torch_cuda_diagnostics", "llama_factory", "file_gateway", "job_polling"],
         }
 
     @app.get(f"{route_prefix}/v1/compute/jobs")
     async def list_jobs_alias() -> dict[str, list[dict[str, Any]]]:
-        return {"items": [job_status(job) for job in jobs.values()]}
+        items = process_manager.list_jobs() if execution_mode() != "simulator" else [job_status(job) for job in jobs.values()]
+        return {"items": items}
 
     @app.get(f"{route_prefix}/compute/resources/gpus")
     async def list_gpus() -> dict[str, Any]:
         return {"items": gpu_resources(), "compute_host_id": host_id()}
 
+    @app.get(f"{route_prefix}/v1/compute/resources/gpus")
+    async def list_gpus_v1() -> dict[str, Any]:
+        return {"items": gpu_resources(), "compute_host_id": host_id()}
+
+    @app.post(f"{route_prefix}/compute/jobs/preview")
+    async def preview_job(payload: dict[str, Any]) -> dict[str, Any]:
+        return _job_preview(payload, check_paths=False)
+
+    @app.post(f"{route_prefix}/compute/jobs/validate")
+    async def validate_job(payload: dict[str, Any]) -> dict[str, Any]:
+        return _job_preview(payload, check_paths=True)
+
+    @app.post(f"{route_prefix}/v1/compute/jobs/preview")
+    async def preview_job_v1(payload: dict[str, Any]) -> dict[str, Any]:
+        return await preview_job(payload)
+
+    @app.post(f"{route_prefix}/v1/compute/jobs/validate")
+    async def validate_job_v1(payload: dict[str, Any]) -> dict[str, Any]:
+        return await validate_job(payload)
+
+    @app.post(f"{route_prefix}/compute/files/check-paths")
+    async def check_paths(payload: dict[str, Any]) -> dict[str, Any]:
+        items = [_check_path_item(item) for item in payload.get("paths", []) if isinstance(item, dict)]
+        return {"valid": all(item["ok"] for item in items), "items": items}
+
+    @app.get(f"{route_prefix}/compute/files/list")
+    async def list_files(
+        root: str = Query(default="data"),
+        relative_path: str = Query(default=""),
+        directories_only: bool = Query(default=False),
+    ) -> dict[str, Any]:
+        roots = {
+            "data": Path(os.getenv("YG_FT_DATA_ROOT", "/data/yg-ft")),
+            "models": Path(os.getenv("YG_FT_MODEL_ROOT", os.getenv("YG_FT_DATA_ROOT", "/data/yg-ft") + "/models")),
+            "datasets": Path(os.getenv("YG_FT_DATASET_ROOT", os.getenv("YG_FT_DATA_ROOT", "/data/yg-ft") + "/datasets")),
+            "outputs": Path(os.getenv("YG_FT_OUTPUT_ROOT", os.getenv("YG_FT_DATA_ROOT", "/data/yg-ft") + "/outputs")),
+        }
+        base = roots.get(root)
+        if base is None:
+            raise HTTPException(status_code=400, detail="invalid root")
+        target = (base / relative_path.lstrip("/\\")).resolve()
+        if not _path_inside(base, target):
+            raise HTTPException(status_code=400, detail="path must stay inside selected root")
+        if not target.exists():
+            return {"root": root, "base_path": str(base), "relative_path": relative_path, "items": []}
+        items = []
+        for child in sorted(target.iterdir(), key=lambda path: (not path.is_dir(), path.name.lower())):
+            if directories_only and not child.is_dir():
+                continue
+            items.append(
+                {
+                    "name": child.name,
+                    "path": str(child),
+                    "relative_path": str(child.relative_to(base)).replace("\\", "/"),
+                    "type": "directory" if child.is_dir() else "file",
+                    "byte_size": child.stat().st_size if child.is_file() else 0,
+                }
+            )
+        return {"root": root, "base_path": str(base), "relative_path": relative_path, "items": items}
+
     @app.post(f"{route_prefix}/compute/jobs")
     async def create_job(payload: dict[str, Any]) -> dict[str, Any]:
+        payload = {**payload, "require_dataset_files": True}
+        try:
+            prepare_runtime_files(payload)
+        except OSError as exc:
+            raise HTTPException(status_code=400, detail=f"prepare runtime files failed: {exc}")
         try:
             command = build_command(payload, os.getenv("LLAMA_FACTORY_HOME", "/app/LLaMA-Factory"))
         except ValueError as exc:
             raise HTTPException(status_code=400, detail=str(exc))
-        if execution_mode() != "simulator":
-            raise HTTPException(
-                status_code=501,
-                detail="real compute executor is not implemented yet; set COMPUTE_EXECUTION_MODE=simulator only for isolated development",
-            )
         job_id = str(payload.get("id") or f"job_{int(now() * 1000)}")
+        if execution_mode() != "simulator":
+            try:
+                return process_manager.create_job({**payload, "id": job_id}, command.command, command.work_dir)
+            except FileNotFoundError as exc:
+                raise HTTPException(status_code=500, detail=f"training command not found: {exc.filename}")
+            except ValueError as exc:
+                raise HTTPException(status_code=409, detail=str(exc))
         job = {
             "id": job_id,
             "name": payload.get("name", job_id),
@@ -169,17 +641,29 @@ def create_app() -> FastAPI:
 
     @app.get(f"{route_prefix}/compute/jobs")
     async def list_jobs() -> dict[str, Any]:
-        return {"items": [job_status(job) for job in jobs.values()]}
+        items = process_manager.list_jobs() if execution_mode() != "simulator" else [job_status(job) for job in jobs.values()]
+        return {"items": items}
 
     @app.get(f"{route_prefix}/compute/jobs/{{job_id}}")
     async def get_job(job_id: str) -> dict[str, Any]:
         job = jobs.get(job_id)
+        if execution_mode() != "simulator":
+            job = process_manager.get_job(job_id)
+            if not job:
+                raise HTTPException(status_code=404, detail="job not found")
+            return job
+        job = jobs.get(job_id)
         if not job:
             raise HTTPException(status_code=404, detail="job not found")
         return job_status(job)
 
     @app.post(f"{route_prefix}/compute/jobs/{{job_id}}/stop")
     async def stop_job(job_id: str) -> dict[str, Any]:
+        if execution_mode() != "simulator":
+            job = process_manager.stop_job(job_id)
+            if not job:
+                raise HTTPException(status_code=404, detail="job not found")
+            return job
         job = jobs.get(job_id)
         if not job:
             raise HTTPException(status_code=404, detail="job not found")
@@ -188,22 +672,206 @@ def create_app() -> FastAPI:
         return job
 
     @app.get(f"{route_prefix}/compute/jobs/{{job_id}}/logs")
-    async def job_logs(job_id: str) -> dict[str, Any]:
+    async def job_logs(
-        job = jobs.get(job_id)
+        job_id: str,
-        if not job:
+        tail_lines: int | None = Query(default=200, ge=1, le=5000),
-            raise HTTPException(status_code=404, detail="job not found")
+        offset: int | None = Query(default=None, ge=0),
-        job = job_status(job)
+        limit: int | None = Query(default=None, ge=1, le=5000),
-        metrics = [parse_log_line(line) for line in job["logs"].splitlines()]
+    ) -> dict[str, Any]:
-        return {"job_id": job_id, "content": job["logs"], "metrics": [m for m in metrics if m]}
+        if execution_mode() != "simulator":
+            job = process_manager.get_job(job_id)
+            if not job:
+                raise HTTPException(status_code=404, detail="job not found")
+            content = process_manager.logs(job_id)
+        else:
+            job = jobs.get(job_id)
+            if not job:
+                raise HTTPException(status_code=404, detail="job not found")
+            job = job_status(job)
+            content = job["logs"]
+        window = _slice_log_content(content, tail_lines, offset, limit)
+        metrics = [parse_log_line(line) for line in window["content"].splitlines()]
+        return {"job_id": job_id, **window, "metrics": [m for m in metrics if m]}
+
+    # ── Inference Endpoints ───────────────────────────────────────────
+
+    @app.post(f"{route_prefix}/inference/load")
+    async def inference_load(payload: dict[str, Any]) -> dict[str, Any]:
+        """Load a model for inference using LLaMA-Factory ChatModel.
+
+        Expected payload:
+            model_name_or_path: str (required)
+            adapter_name_or_path: str (optional, for LoRA adapters)
+            template: str (default: "qwen")
+            infer_backend: str (default: "huggingface")
+            infer_dtype: str (default: "auto")
+        """
+        session = get_inference_session()
+        result = session.load(
+            model_name_or_path=payload.get("model_name_or_path", ""),
+            adapter_name_or_path=payload.get("adapter_name_or_path", ""),
+            template=payload.get("template", "qwen"),
+            infer_backend=payload.get("infer_backend", "huggingface"),
+            infer_dtype=payload.get("infer_dtype", "auto"),
+        )
+        return result
+
+    @app.post(f"{route_prefix}/inference/unload")
+    async def inference_unload() -> dict[str, Any]:
+        """Unload the currently loaded model and free GPU memory."""
+        # Teardown (gc.collect + cuda.empty_cache) can take a while; run it off
+        # the event loop so /health and /inference/status stay responsive.
+        return await asyncio.to_thread(get_inference_session().unload)
+
+    @app.get(f"{route_prefix}/inference/status")
+    async def inference_status() -> dict[str, Any]:
+        """Get the current inference session status."""
+        return get_inference_session().info()
+
+    @app.post(f"{route_prefix}/inference/chat")
+    async def inference_chat(payload: dict[str, Any]) -> dict[str, Any]:
+        """Chat with the loaded model (non-streaming).
+
+        Expected payload:
+            messages: list[dict] (OpenAI format)
+            temperature: float (default 0.95)
+            top_p: float (default 0.7)
+            max_new_tokens: int (default 1024)
+        """
+        messages = payload.get("messages") or []
+        if not messages:
+            raise HTTPException(status_code=400, detail="messages is required")
+        # Generation is long-running; run it in a thread so the event loop keeps
+        # serving /inference/status and /health during inference.
+        result = await asyncio.to_thread(
+            get_inference_session().chat,
+            messages=messages,
+            temperature=float(payload.get("temperature", 0.95)),
+            top_p=float(payload.get("top_p", 0.7)),
+            max_new_tokens=int(payload.get("max_new_tokens", 1024)),
+            do_sample=bool(payload.get("do_sample", True)),
+        )
+        if result.get("error"):
+            raise HTTPException(status_code=500, detail=result["error"])
+        return {"response": result["response"]}
+
+    @app.post(f"{route_prefix}/inference/chat/stream")
+    async def inference_chat_stream(payload: dict[str, Any]) -> StreamingResponse:
+        """Chat with streaming response (Server-Sent Events)."""
+        messages = payload.get("messages") or []
+        if not messages:
+            raise HTTPException(status_code=400, detail="messages is required")
+
+        def generate():
+            session = get_inference_session()
+            for chunk in session.chat_stream(
+                messages=messages,
+                temperature=float(payload.get("temperature", 0.95)),
+                top_p=float(payload.get("top_p", 0.7)),
+                max_new_tokens=int(payload.get("max_new_tokens", 1024)),
+                do_sample=bool(payload.get("do_sample", True)),
+            ):
+                yield chunk
+
+        return StreamingResponse(generate(), media_type="text/event-stream")
 
     @app.post(f"{route_prefix}/compute/files/upload")
-    async def upload_file(payload: dict[str, Any]) -> dict[str, Any]:
+    async def upload_file(
-        file_id = str(payload.get("id") or f"file_{int(now() * 1000)}")
+        file: UploadFile | None = File(default=None),
-        return {"id": file_id, "status": "available", "local_path": f"/data/yg-ft/uploads/{file_id}"}
+        target_relative_path: str | None = Form(default=None),
+        resource_type: str | None = Form(default=None),
+        resource_id: str | None = Form(default=None),
+    ) -> dict[str, Any]:
+        file_id = f"file_{int(now() * 1000)}"
+        data_root = Path(os.getenv("YG_FT_DATA_ROOT", "/data/yg-ft"))
+        data_root.mkdir(parents=True, exist_ok=True)
+        filename = Path(file.filename if file else file_id).name
+        if target_relative_path:
+            target = (data_root / target_relative_path.lstrip("/\\")).resolve()
+            if not _path_inside(data_root, target):
+                raise HTTPException(status_code=400, detail="target path must stay inside YG_FT_DATA_ROOT")
+        else:
+            target = data_root / "uploads" / f"{file_id}_{filename}"
+        if file:
+            target.parent.mkdir(parents=True, exist_ok=True)
+            with target.open("wb") as output:
+                while chunk := await file.read(1024 * 1024):
+                    output.write(chunk)
+        else:
+            target.parent.mkdir(parents=True, exist_ok=True)
+            target.write_text("", encoding="utf-8")
+        return {
+            "id": file_id,
+            "resource_type": resource_type,
+            "resource_id": resource_id,
+            "status": "available",
+            "local_path": str(target),
+            "byte_size": target.stat().st_size,
+            "checksum_sha256": hashlib.sha256(target.read_bytes()).hexdigest() if target.is_file() else "",
+        }
+
+    @app.post(f"{route_prefix}/compute/files/import-local")
+    async def import_local_file(payload: dict[str, Any]) -> dict[str, Any]:
+        source = Path(str(payload.get("source_path") or ""))
+        if not source.exists():
+            raise HTTPException(status_code=404, detail="source path not found")
+        data_root = Path(os.getenv("YG_FT_DATA_ROOT", "/data/yg-ft"))
+        data_root.mkdir(parents=True, exist_ok=True)
+        relative = str(payload.get("target_relative_path") or f"imports/{source.name}").lstrip("/\\")
+        target = (data_root / relative).resolve()
+        if not _path_inside(data_root, target):
+            raise HTTPException(status_code=400, detail="target path must stay inside YG_FT_DATA_ROOT")
+        target.parent.mkdir(parents=True, exist_ok=True)
+        if source.is_dir():
+            if target.exists():
+                shutil.rmtree(target)
+            shutil.copytree(source, target)
+            byte_size = sum(path.stat().st_size for path in target.rglob("*") if path.is_file())
+            checksum = ""
+        else:
+            shutil.copy2(source, target)
+            byte_size = target.stat().st_size
+            checksum = hashlib.sha256(target.read_bytes()).hexdigest()
+        return {
+            "id": str(payload.get("id") or f"file_{int(now() * 1000)}"),
+            "resource_type": payload.get("resource_type"),
+            "resource_id": payload.get("resource_id"),
+            "status": "available",
+            "local_path": str(target),
+            "byte_size": byte_size,
+            "checksum_sha256": checksum,
+        }
+
+    @app.get(f"{route_prefix}/compute/files/read")
+    async def read_file(path: str = Query(...)) -> JSONResponse:
+        """Read a text file from within YG_FT_DATA_ROOT.  Used by the backend
+        to fetch eval results and other job outputs."""
+        data_root = Path(os.getenv("YG_FT_DATA_ROOT", "/data/yg-ft"))
+        target = (data_root / path.lstrip("/\\")).resolve()
+        if not _path_inside(data_root, target):
+            raise HTTPException(status_code=400, detail="path must stay inside YG_FT_DATA_ROOT")
+        if not target.is_file():
+            raise HTTPException(status_code=404, detail="file not found")
+        try:
+            content = target.read_text(encoding="utf-8")
+            return JSONResponse(json.loads(content) if content.strip().startswith("{") else {"content": content})
+        except Exception as exc:
+            raise HTTPException(status_code=500, detail=str(exc))
 
     @app.get(f"{route_prefix}/compute/files/{{file_id}}/download")
-    async def download_file(file_id: str) -> dict[str, Any]:
+    async def download_file(file_id: str) -> FileResponse:
-        return {"id": file_id, "status": "ready", "download_url": f"{route_prefix}/compute/files/{file_id}/download"}
+        upload_root = Path(os.getenv("YG_FT_DATA_ROOT", "/data/yg-ft")) / "uploads"
+        # file_id 仅允许普通标识符，拒绝 ../、/、\ 等路径穿越字符。
+        if not file_id or not all(character.isalnum() or character in {"_", "-"} for character in file_id):
+            raise HTTPException(status_code=400, detail="invalid file id")
+        matches = list(upload_root.glob(f"{file_id}_*"))
+        if not matches:
+            raise HTTPException(status_code=404, detail="file not found")
+        # 解析符号链接后仍必须位于 upload 根目录内，防止符号链接指向目录外文件。
+        resolved = matches[0].resolve()
+        if not _path_inside(upload_root, resolved):
+            raise HTTPException(status_code=404, detail="file not found")
+        return FileResponse(resolved)
 
     return app
 

compute/api/security.py

@@ -0,0 +1,29 @@
+"""计算节点 API 安全配置：Swagger / ReDoc / OpenAPI 文档路由开关。"""
+from __future__ import annotations
+
+import os
+from typing import Any
+
+
+def docs_enabled() -> bool:
+    """判断 FastAPI 文档路由（/docs、/redoc、/openapi.json）是否开放。
+
+    显式配置 ENABLE_DOCS 时以之为准；否则仅在关闭 token 鉴权
+    （COMPUTE_AUTH_ENABLED=false，本地开发）时开放，生产环境默认关闭，
+    避免未授权访问泄露 API 结构。
+    """
+    raw = os.getenv("ENABLE_DOCS", "").strip().lower()
+    if raw in {"true", "false"}:
+        return raw == "true"
+    auth_enabled = os.getenv("COMPUTE_AUTH_ENABLED", "true").lower() == "true"
+    return not auth_enabled
+
+
+def docs_kwargs() -> dict[str, Any]:
+    """返回传入 FastAPI 的文档路由参数。
+
+    关闭时 FastAPI 不注册 /docs、/redoc、/openapi.json，访问一律返回 404。
+    """
+    if docs_enabled():
+        return {}
+    return {"docs_url": None, "redoc_url": None, "openapi_url": None}

compute/engines/llama_factory/adapter.py

@@ -1,5 +1,6 @@
 from __future__ import annotations
 
+import json
 import re
 from dataclasses import dataclass
 from pathlib import Path
@@ -13,40 +14,290 @@ class LlamaFactoryCommand:
     env: dict[str, str]
 
 
+def _load_dataset_preview(path: Path) -> list[dict[str, Any]]:
+    """Load a preview of JSON/JSONL records from a dataset file.
+
+    Content-sniffs instead of trusting the extension so that BOM-prefixed files,
+    JSONL files containing a single JSON array, and mislabeled extensions all work.
+    """
+    if not path.exists():
+        return []
+    text = path.read_text(encoding="utf-8-sig", errors="replace").strip()
+    if not text:
+        return []
+    try:
+        value = json.loads(text)
+    except json.JSONDecodeError:
+        value = None
+    if isinstance(value, list):
+        return [item for item in value[:20] if isinstance(item, dict)]
+    if isinstance(value, dict):
+        return [value]
+    items: list[dict[str, Any]] = []
+    for line in text.splitlines()[:20]:
+        line = line.strip()
+        if not line:
+            continue
+        try:
+            parsed = json.loads(line)
+        except json.JSONDecodeError:
+            continue
+        if isinstance(parsed, list):
+            items.extend(item for item in parsed[:20] if isinstance(item, dict))
+        elif isinstance(parsed, dict):
+            items.append(parsed)
+        if len(items) >= 20:
+            break
+    return items[:20]
+
+
+def _required_columns_for(formatting: str, columns: dict[str, Any]) -> list[str]:
+    """Required data columns per dataset format.
+
+    Mirrors LLaMA-Factory's leniency: optional columns (e.g. ``input`` / ``query``
+    in Alpaca) are never required, only fields the format structurally needs.
+    """
+    fmt = str(formatting or "").lower()
+    if fmt == "sharegpt":
+        return [str(columns.get("messages") or "messages")]
+    if fmt in {"dpo", "rm", "kto", "ppo"}:
+        return [str(columns[key]) for key in ("chosen", "rejected") if columns.get(key)]
+    if fmt in {"cpt", "pt", "pretrain"}:
+        return [str(columns.get("prompt") or columns.get("text") or "text")]
+    # alpaca family: prompt (instruction) + response (output) required,
+    # query (input) / history are optional and common to omit in jsonl datasets.
+    return [str(columns[key]) for key in ("prompt", "response") if columns.get(key)]
+
+
+def _validate_dataset_columns(config: dict[str, Any]) -> list[str]:
+    dataset_dir = config.get("dataset_dir")
+    dataset_info = config.get("dataset_info")
+    if not dataset_dir or not isinstance(dataset_info, dict):
+        return []
+    root = Path(str(dataset_dir))
+    errors: list[str] = []
+    for dataset_key, item in dataset_info.items():
+        if not isinstance(item, dict):
+            continue
+        file_name = item.get("file_name")
+        file_names = file_name if isinstance(file_name, list) else [file_name]
+        columns = item.get("columns") if isinstance(item.get("columns"), dict) else {}
+        required_columns = _required_columns_for(str(item.get("formatting") or ""), columns)
+        for name in file_names:
+            if not name:
+                continue
+            path = root / str(name).lstrip("/\\")
+            if not path.exists():
+                continue
+            try:
+                preview_rows = _load_dataset_preview(path)
+            except Exception as exc:  # noqa: BLE001 - expose malformed data as validation error
+                errors.append(f"dataset file parse failed: {path}: {exc}")
+                continue
+            if not preview_rows:
+                errors.append(f"dataset file has no valid object records: {path}")
+                continue
+            available = set().union(*(row.keys() for row in preview_rows))
+            missing = [column for column in required_columns if column not in available]
+            if missing:
+                errors.append(
+                    f"dataset columns missing in {path.name} for {dataset_key}: {', '.join(sorted(set(missing)))}"
+                )
+    return errors
+
+
 def validate_config(config: dict[str, Any]) -> list[str]:
     errors: list[str] = []
     if not config.get("base_model") and not config.get("model_name_or_path"):
         errors.append("base_model or model_name_or_path is required")
     if not config.get("dataset") and not config.get("dataset_dir"):
         errors.append("dataset or dataset_dir is required")
-    learning_rate = float(config.get("learning_rate", 0.0002))
+    try:
+        learning_rate = float(config.get("learning_rate", 0.0002))
+    except (TypeError, ValueError):
+        learning_rate = 0
     if learning_rate <= 0:
         errors.append("learning_rate must be greater than zero")
-    epochs = int(config.get("n_epochs", config.get("num_train_epochs", 1)))
+    try:
+        epochs = int(config.get("n_epochs", config.get("num_train_epochs", 1)))
+    except (TypeError, ValueError):
+        epochs = 0
     if epochs <= 0:
         errors.append("n_epochs must be greater than zero")
+    dataset_dir = config.get("dataset_dir")
+    dataset_info = config.get("dataset_info")
+    if config.get("require_dataset_files") and dataset_dir and isinstance(dataset_info, dict):
+        root = Path(str(dataset_dir))
+        for dataset_key, item in dataset_info.items():
+            if not isinstance(item, dict):
+                errors.append(f"dataset_info entry must be object: {dataset_key}")
+                continue
+            file_name = item.get("file_name")
+            file_names = file_name if isinstance(file_name, list) else [file_name]
+            for name in file_names:
+                if not name:
+                    errors.append(f"dataset_info file_name is required: {dataset_key}")
+                    continue
+                path = root / str(name).lstrip("/\\")
+                if not path.exists():
+                    errors.append(f"dataset file not found: {path}")
+        errors.extend(_validate_dataset_columns(config))
     return errors
 
 
+def _optional_arg(config: dict[str, Any], command: list[str], option: str, *keys: str) -> None:
+    for key in keys:
+        value = config.get(key)
+        if value is not None and value != "":
+            command.extend([option, str(value)])
+            return
+
+
+def _optional_bool_arg(config: dict[str, Any], command: list[str], option: str, *keys: str) -> None:
+    for key in keys:
+        value = config.get(key)
+        if value is True or str(value).lower() == "true":
+            command.extend([option, "true"])
+            return
+
+
+def _normalize_stage(config: dict[str, Any]) -> str:
+    raw = str(config.get("stage") or config.get("train_type") or "sft").strip().lower()
+    return {
+        "sft": "sft",
+        "dpo": "dpo",
+        "cpt": "pt",
+        "pt": "pt",
+        "pretrain": "pt",
+        "rm": "rm",
+        "ppo": "ppo",
+        "kto": "kto",
+    }.get(raw, raw or "sft")
+
+
+def prepare_runtime_files(config: dict[str, Any]) -> list[dict[str, str]]:
+    dataset_dir = config.get("dataset_dir")
+    dataset_info = config.get("dataset_info")
+    if not dataset_dir or not isinstance(dataset_info, dict):
+        return []
+    root = Path(str(dataset_dir))
+    root.mkdir(parents=True, exist_ok=True)
+    path = root / "dataset_info.json"
+    existing: dict[str, Any] = {}
+    if path.exists():
+        try:
+            loaded = json.loads(path.read_text(encoding="utf-8"))
+            existing = loaded if isinstance(loaded, dict) else {}
+        except json.JSONDecodeError:
+            existing = {}
+    existing.update(dataset_info)
+    path.write_text(json.dumps(existing, ensure_ascii=False, indent=2), encoding="utf-8")
+    return [{"name": "dataset_info", "path": str(path)}]
+
+
 def build_command(config: dict[str, Any], llama_factory_home: str = "/app/LLaMA-Factory") -> LlamaFactoryCommand:
+    engine = str(config.get("engine") or config.get("training_engine") or "llama_factory")
+    if engine in {"merge", "export", "llama_factory_export"}:
+        model_path = config.get("base_model") or config.get("model_name_or_path") or config.get("base_model_path")
+        adapter_path = config.get("adapter_name_or_path") or config.get("adapter_path") or config.get("lora_path")
+        output_dir = config.get("output_dir") or config.get("export_dir")
+        errors: list[str] = []
+        if not model_path:
+            errors.append("base_model or model_name_or_path is required")
+        if not adapter_path and engine == "merge":
+            errors.append("adapter_name_or_path or adapter_path is required")
+        if not output_dir:
+            errors.append("output_dir or export_dir is required")
+        if errors:
+            raise ValueError("; ".join(errors))
+        command = [
+            "llamafactory-cli",
+            "export",
+            "--model_name_or_path",
+            str(model_path),
+            "--template",
+            str(config.get("template", "qwen")),
+            "--finetuning_type",
+            str(config.get("train_method", config.get("finetuning_type", "lora"))),
+            "--export_dir",
+            str(output_dir),
+            "--export_size",
+            str(config.get("export_size", 2)),
+            "--export_device",
+            str(config.get("export_device", "cpu")),
+            "--export_legacy_format",
+            str(config.get("export_legacy_format", False)).lower(),
+        ]
+        if adapter_path:
+            command.extend(["--adapter_name_or_path", str(adapter_path)])
+        quantization_bit = int(config.get("export_quantization_bit", config.get("quantization_bit", 0)) or 0)
+        if quantization_bit in {4, 8}:
+            command.extend(["--quantization_bit", str(quantization_bit)])
+        return LlamaFactoryCommand(command=command, work_dir=str(Path(llama_factory_home)), env={})
+
+    if engine == "eval":
+        output_dir = config.get("output_dir") or f"/data/yg-ft/outputs/{config.get('name', 'eval-job')}"
+        eval_config_path = str(Path(output_dir) / "eval_config.json")
+        eval_config = {
+            "model_name_or_path": config.get("model_name_or_path", ""),
+            "adapter_name_or_path": config.get("adapter_name_or_path", ""),
+            "template": config.get("template", "qwen"),
+            "dataset_path": config.get("dataset_path", ""),
+            "output_dir": output_dir,
+            "basic_metrics": config.get("basic_metrics", {}),
+            "dimension": config.get("dimension", {}),
+            "temperature": config.get("temperature", 0.1),
+            "top_p": config.get("top_p", 0.95),
+            "max_new_tokens": config.get("max_new_tokens", 512),
+            "infer_backend": config.get("infer_backend", "huggingface"),
+            "infer_dtype": config.get("infer_dtype", "auto"),
+        }
+        Path(output_dir).mkdir(parents=True, exist_ok=True)
+        Path(eval_config_path).write_text(json.dumps(eval_config, ensure_ascii=False, indent=2), encoding="utf-8")
+        return LlamaFactoryCommand(
+            command=["python", "-u", "-m", "compute.engines.llama_factory.eval_runner", "--config", eval_config_path],
+            work_dir="/app",
+            env={},
+        )
+
     errors = validate_config(config)
     if errors:
         raise ValueError("; ".join(errors))
 
+    if engine == "smoke":
+        output_dir = config.get("output_dir") or f"/data/yg-ft/outputs/{config.get('name', 'training-smoke')}"
+        script = (
+            "import json, os, time; "
+            f"out={str(output_dir)!r}; "
+            "os.makedirs(out, exist_ok=True); "
+            "print('[INFO] smoke training started', flush=True); "
+            "\nfor step in range(1, 7):\n"
+            "    loss=round(1.8/(step+1), 4)\n"
+            "    lr=round(0.0002*(1-step/10), 8)\n"
+            "    print({'loss': loss, 'grad_norm': round(0.4 + step*0.03, 4), 'learning_rate': lr, 'epoch': round(step/6, 4)}, flush=True)\n"
+            "    time.sleep(0.4)\n"
+            "\nopen(os.path.join(out, 'adapter_config.json'), 'w', encoding='utf-8').write(json.dumps({'engine':'smoke','status':'completed'})); "
+            "print('***** train metrics *****', flush=True); "
+            "print('train_loss = 0.12', flush=True); "
+            "print('***** train metrics end *****', flush=True)"
+        )
+        return LlamaFactoryCommand(command=["python", "-u", "-c", script], work_dir="/app", env={})
+
     model_path = config.get("base_model") or config.get("model_name_or_path")
-    dataset = config.get("dataset") or config.get("dataset_dir")
+    dataset = config.get("dataset") or config.get("dataset_name")
+    dataset_dir = config.get("dataset_dir")
     output_dir = config.get("output_dir") or f"/data/yg-ft/outputs/{config.get('name', 'training-job')}"
     command = [
         "llamafactory-cli",
         "train",
         "--stage",
-        str(config.get("stage", "sft")).lower(),
+        _normalize_stage(config),
         "--do_train",
         "true",
         "--model_name_or_path",
         str(model_path),
         "--dataset",
-        str(dataset),
+        str(dataset or "default"),
         "--template",
         str(config.get("template", "qwen")),
         "--finetuning_type",
@@ -61,7 +312,32 @@ def build_command(config: dict[str, Any], llama_factory_home: str = "/app/LLaMA-
         str(config.get("n_epochs", 3)),
         "--save_steps",
         str(config.get("save_steps", 50)),
+        "--logging_steps",
+        str(config.get("logging_steps", 10)),
+        "--overwrite_output_dir",
+        "true",
+        "--plot_loss",
+        "true",
     ]
+    if dataset_dir:
+        command.extend(["--dataset_dir", str(dataset_dir)])
+    eval_dataset = config.get("eval_dataset")
+    if eval_dataset:
+        command.extend(["--eval_dataset", str(eval_dataset), "--do_eval", "true"])
+    _optional_arg(config, command, "--cutoff_len", "max_length", "cutoff_len")
+    _optional_arg(config, command, "--lr_scheduler_type", "lr_scheduler_type")
+    _optional_arg(config, command, "--warmup_ratio", "warmup_ratio")
+    _optional_arg(config, command, "--weight_decay", "weight_decay")
+    _optional_arg(config, command, "--lora_rank", "lora_rank", "rank")
+    _optional_arg(config, command, "--lora_alpha", "lora_alpha")
+    _optional_arg(config, command, "--lora_dropout", "lora_dropout")
+    _optional_arg(config, command, "--gradient_accumulation_steps", "gradient_accumulation_steps")
+    if not eval_dataset:
+        _optional_arg(config, command, "--val_size", "val_size")
+    _optional_arg(config, command, "--max_samples", "max_samples")
+    _optional_arg(config, command, "--preprocessing_num_workers", "preprocessing_num_workers")
+    _optional_bool_arg(config, command, "--fp16", "fp16")
+    _optional_bool_arg(config, command, "--bf16", "bf16")
     quantization_bit = int(config.get("quantization_bit", 0) or 0)
     if quantization_bit in {4, 8}:
         command.extend(["--quantization_bit", str(quantization_bit)])
@@ -77,4 +353,3 @@ def parse_log_line(line: str) -> dict[str, float] | None:
         if match:
             result[key] = float(match.group(1))
     return result or None
-

compute/engines/llama_factory/eval_runner.py

@@ -0,0 +1,491 @@
+"""
+Evaluation runner — executes model evaluation as a subprocess job.
+
+Usage:
+    python -m compute.engines.llama_factory.eval_runner --config <config_json_path>
+
+The config JSON is written by the compute API before spawning this subprocess.
+Results are written to ``output_dir/eval_results.json`` and progress is printed
+to stdout (captured as job logs).
+"""
+
+from __future__ import annotations
+
+import json
+import math
+import re
+import sys
+import time
+from difflib import SequenceMatcher
+from pathlib import Path
+from typing import Any
+
+
+def _load_dataset(path: str) -> list[dict[str, Any]]:
+    """Load a JSON or JSONL dataset file (jsonl-compatible).
+
+    Content-sniffs instead of trusting the extension so jsonl files with a BOM,
+    a single JSON array on one line, or mislabeled extensions all load correctly.
+
+    Supports common field names used across the platform:
+    * ``instruction`` + ``input`` + ``output``  (Alpaca-style)
+    * ``question`` + ``answer``
+    * ``messages``  (ShareGPT-style – the last assistant message is treated as reference)
+    """
+    file_path = Path(path)
+    text = file_path.read_text(encoding="utf-8-sig", errors="replace").strip()
+    if not text:
+        return []
+    try:
+        value = json.loads(text)
+    except json.JSONDecodeError:
+        value = None
+    if isinstance(value, list):
+        return [item for item in value if isinstance(item, dict)]
+    if isinstance(value, dict):
+        return [value]
+
+    samples: list[dict[str, Any]] = []
+    for line in text.splitlines():
+        line = line.strip()
+        if not line:
+            continue
+        try:
+            obj = json.loads(line)
+        except json.JSONDecodeError:
+            continue
+        if isinstance(obj, dict):
+            samples.append(obj)
+    return samples
+
+
+def _sample_question(sample: dict[str, Any]) -> str:
+    """Extract the user-facing question / instruction from a sample."""
+    if sample.get("instruction"):
+        text = sample["instruction"]
+        if sample.get("input"):
+            text += "\n" + sample["input"]
+        return text
+    if sample.get("question"):
+        return sample["question"]
+    # ShareGPT-style: use the last user message as question
+    messages = sample.get("messages") or []
+    user_msgs = [m["content"] for m in messages if m.get("role") == "user"]
+    return user_msgs[-1] if user_msgs else ""
+
+
+def _sample_reference(sample: dict[str, Any]) -> str:
+    """Extract the reference answer from a sample."""
+    if sample.get("output"):
+        return sample["output"]
+    if sample.get("answer"):
+        return sample["answer"]
+    messages = sample.get("messages") or []
+    assistant_msgs = [m["content"] for m in messages if m.get("role") == "assistant"]
+    return assistant_msgs[-1] if assistant_msgs else ""
+
+
+# ---------------------------------------------------------------------------
+# Basic metrics
+# ---------------------------------------------------------------------------
+
+def _compute_bleu(references: list[str], predictions: list[str], ngram: int = 4) -> dict[str, Any]:
+    """Compute BLEU score via sacrebleu (corpus-level)."""
+    try:
+        from sacrebleu.metrics import BLEU
+    except ImportError:
+        return {"enabled": False, "error": "sacrebleu not installed", "score": 0}
+    bleu = BLEU(max_ngram_order=ngram)
+    # sacrebleu expects list-of-strings; we have one reference per prediction
+    score = bleu.corpus_score(predictions, [references])
+    return {
+        "enabled": True,
+        "score": round(score.score, 2),
+        "bleu": round(score.score, 2),
+    }
+
+
+def _compute_rouge(references: list[str], predictions: list[str], methods: list[str] | None = None) -> dict[str, Any]:
+    """Compute ROUGE scores via rouge-score."""
+    try:
+        from rouge_score import rouge_scorer
+    except ImportError:
+        return {"enabled": False, "error": "rouge-score not installed", "score": 0}
+    methods = methods or ["rouge1", "rouge2", "rougeL"]
+    # Normalize: map "rouge_1"/"rouge1" → "rouge1", "rouge_l"/"rougeL" → "rougeL"
+    _rouge_aliases = {"rouge_1": "rouge1", "rouge_2": "rouge2", "rouge_l": "rougeL"}
+    methods = [_rouge_aliases.get(m, m.replace("_", "")) for m in methods]
+    scorer = rouge_scorer.RougeScorer(methods, use_stemmer=True)
+    totals: dict[str, float] = {}
+    n = max(len(predictions), 1)
+    for ref, pred in zip(references, predictions):
+        result = scorer.score(ref, pred)
+        for key in methods:
+            totals[key] = totals.get(key, 0) + result[key].fmeasure
+    avg = {k: round(v / n, 4) for k, v in totals.items()}
+    return {"enabled": True, "score": round(avg.get("rougeL", avg.get("rouge1", 0)) * 100, 2), **avg}
+
+
+def _compute_cosine(references: list[str], predictions: list[str]) -> dict[str, Any]:
+    """Compute average cosine similarity via sklearn."""
+    try:
+        from sklearn.feature_extraction.text import TfidfVectorizer
+        from sklearn.metrics.pairwise import cosine_similarity
+    except ImportError:
+        return {"enabled": False, "error": "scikit-learn not installed", "score": 0}
+    try:
+        vectorizer = TfidfVectorizer()
+        tfidf = vectorizer.fit_transform(references + predictions)
+        n = len(references)
+        ref_vec = tfidf[:n]
+        pred_vec = tfidf[n:]
+        sims = cosine_similarity(ref_vec, pred_vec).diagonal()
+        return {"enabled": True, "score": round(float(sims.mean()) * 100, 2)}
+    except ValueError:
+        return {"enabled": True, "score": 0, "error": "insufficient text for vectorization"}
+
+
+def _normalize_text(value: str) -> str:
+    return re.sub(r"\s+", " ", str(value or "").strip().lower())
+
+
+def _compute_exact_match(references: list[str], predictions: list[str]) -> dict[str, Any]:
+    total = len(predictions)
+    if not total:
+        return {"enabled": True, "score": 0, "matched": 0, "total": 0}
+    matched = sum(
+        1
+        for ref, pred in zip(references, predictions)
+        if _normalize_text(ref) == _normalize_text(pred)
+    )
+    return {"enabled": True, "score": round(matched / total * 100, 2), "matched": matched, "total": total}
+
+
+def _compute_text_similarity(references: list[str], predictions: list[str]) -> dict[str, Any]:
+    if not predictions:
+        return {"enabled": True, "score": 0}
+    scores = [
+        SequenceMatcher(None, _normalize_text(ref), _normalize_text(pred)).ratio()
+        for ref, pred in zip(references, predictions)
+    ]
+    return {"enabled": True, "score": round(sum(scores) / max(len(scores), 1) * 100, 2)}
+
+
+# ---------------------------------------------------------------------------
+# LLM Judge
+# ---------------------------------------------------------------------------
+
+def _judge_sample(
+    question: str,
+    reference: str,
+    prediction: str,
+    config: dict[str, Any],
+) -> dict[str, Any]:
+    """Call an OpenAI-compatible LLM to judge a single sample.
+
+    Returns a dict with keys:
+        score, max_score, passed, judgement, evaluation_reason, error_type
+    """
+    api_url = (config.get("api_url") or "").strip().rstrip("/")
+    api_key = (config.get("api_key") or "").strip()
+    eval_model = (config.get("eval_model") or "").strip()
+    # 优先使用模型记录里配置的真实 API 模型名（如 deepseek-chat），
+    # 否则回退到平台内部模型名
+    api_model = (config.get("api_model") or "").strip() or eval_model
+    eval_prompt = (config.get("eval_prompt") or "").strip()
+    score_min = float(config.get("score_min", 0))
+    score_max = float(config.get("score_max", 5))
+    pass_threshold = float(config.get("pass_threshold", 3))
+
+    if not api_url or not eval_model:
+        return {"score": 0, "max_score": score_max, "passed": False, "judgement": "未配置",
+                "evaluation_reason": "未配置评测模型", "error_type": "其他"}
+
+    system_msg = (
+        eval_prompt
+        or "你是一个专业的评测专家。请根据参考答-案对被测模型的输出进行评分。"
+    )
+    user_msg = (
+        f"## 问题\n{question}\n\n"
+        f"## 参考答案\n{reference}\n\n"
+        f"## 模型输出\n{prediction}\n\n"
+        f"请给出 {score_min}-{score_max} 分的评分，并说明理由。"
+    )
+
+    try:
+        import urllib.request
+        import urllib.error
+
+        body = json.dumps({
+            "model": api_model,
+            "messages": [
+                {"role": "system", "content": system_msg},
+                {"role": "user", "content": user_msg},
+            ],
+            "temperature": 0.3,
+            "max_tokens": 512,
+        }).encode("utf-8")
+
+        req = urllib.request.Request(
+            f"{api_url}/v1/chat/completions",
+            data=body,
+            headers={
+                "Content-Type": "application/json",
+                "Authorization": f"Bearer {api_key}",
+            },
+        )
+        resp = urllib.request.urlopen(req, timeout=120)
+        data = json.loads(resp.read().decode("utf-8"))
+        reply = data["choices"][0]["message"]["content"]
+    except Exception as exc:
+        return {"score": 0, "max_score": score_max, "passed": False,
+                "judgement": "错误", "evaluation_reason": f"评测模型调用失败: {exc}",
+                "error_type": "其他"}
+
+    # Parse score from reply — look for patterns like "4分" or "Score: 4"
+    score = 0
+    import re
+    score_patterns = [
+        r'(?:得分|分数|评分|score)[^\d]*(\d+(?:\.\d+)?)',
+        r'(\d+(?:\.\d+)?)\s*分',
+        r'(\d+(?:\.\d+)?)\s*/\s*\d+',
+    ]
+    for pat in score_patterns:
+        m = re.search(pat, reply, re.IGNORECASE)
+        if m:
+            try:
+                score = float(m.group(1))
+            except ValueError:
+                continue
+            break
+    score = max(score_min, min(score_max, score))
+    passed = score >= pass_threshold
+
+    # Determine judgement label
+    if score >= pass_threshold + 1:
+        judgement = "正确"
+    elif score >= pass_threshold:
+        judgement = "部分正确"
+    else:
+        judgement = "错误"
+
+    # Guess error type from reply
+    reply_lower = reply.lower()
+    if any(w in reply_lower for w in ["幻觉", "hallucination", "编造"]):
+        error_type = "幻觉"
+    elif any(w in reply_lower for w in ["不完整", "incomplete", "遗漏"]):
+        error_type = "不完整"
+    elif any(w in reply_lower for w in ["格式", "format"]):
+        error_type = "格式偏差"
+    elif any(w in reply_lower for w in ["混淆", "confusion", "错误"]):
+        error_type = "混淆"
+    else:
+        error_type = "其他"
+
+    return {
+        "score": score,
+        "max_score": score_max,
+        "passed": passed,
+        "judgement": judgement,
+        "evaluation_reason": reply[:2000],
+        "error_type": error_type,
+    }
+
+
+# ---------------------------------------------------------------------------
+# Main entry point
+# ---------------------------------------------------------------------------
+
+def run_eval(config: dict[str, Any]) -> dict[str, Any]:
+    """Execute a full evaluation run.  Returns the result dict (also written to file)."""
+    model_path = config["model_name_or_path"]
+    adapter_path = config.get("adapter_name_or_path", "")
+    template = config.get("template", "qwen")
+    dataset_path = config["dataset_path"]
+    output_dir = Path(config["output_dir"])
+    output_dir.mkdir(parents=True, exist_ok=True)
+    basic_cfg = config.get("basic_metrics", {})
+    dimension_cfg = config.get("dimension", {}) or {}
+    output_precision = int(basic_cfg.get("output_precision", 2))
+
+    # ---- 1. Load dataset ----
+    print(f"[eval] loading dataset: {dataset_path}")
+    raw_samples = _load_dataset(dataset_path)
+    print(f"[eval] loaded {len(raw_samples)} samples")
+
+    # ---- 2. Load model ----
+    print(f"[eval] loading model: {model_path}")
+    from compute.engines.llama_factory.inference import InferenceSession
+    session = InferenceSession()
+    session.load(
+        model_name_or_path=model_path,
+        adapter_name_or_path=adapter_path,
+        template=template,
+        infer_backend=config.get("infer_backend", "huggingface"),
+        infer_dtype=config.get("infer_dtype", "auto"),
+    )
+    # load() 为异步加载（立即返回 loading），必须等待后台线程完成后再进行推理
+    load_result = session.wait_until_loaded(timeout=float(config.get("load_timeout", 1800)))
+    if not load_result.get("loaded"):
+        raise RuntimeError(f"model load failed: {load_result.get('error', 'unknown')}")
+    print(f"[eval] model loaded OK")
+
+    # ---- 3. Run inference on each sample ----
+    samples: list[dict[str, Any]] = []
+    predictions: list[str] = []
+    references: list[str] = []
+    questions: list[str] = []
+
+    total = len(raw_samples)
+    judge_enabled = bool(dimension_cfg.get("eval_model") and dimension_cfg.get("api_url"))
+    print(f"[eval] starting inference on {total} samples, judge={'enabled' if judge_enabled else 'disabled'}")
+
+    for idx, raw in enumerate(raw_samples, start=1):
+        question = _sample_question(raw)
+        reference = _sample_reference(raw)
+        if not question:
+            print(f"[eval] sample {idx}/{total}: skipped (no question)")
+            continue
+
+        # Inference
+        chat_msgs = [{"role": "user", "content": question}]
+        result = session.chat(
+            chat_msgs,
+            temperature=float(config.get("temperature", 0.1)),
+            top_p=float(config.get("top_p", 0.95)),
+            max_new_tokens=int(config.get("max_new_tokens", 512)),
+            do_sample=False,
+        )
+        prediction = result.get("response", "") if not result.get("error") else f"[ERROR] {result['error']}"
+
+        predictions.append(prediction)
+        references.append(reference)
+        questions.append(question)
+
+        # LLM Judge
+        judge_result: dict[str, Any] = {}
+        if judge_enabled:
+            judge_result = _judge_sample(question, reference, prediction, dimension_cfg)
+
+        samples.append({
+            "index": idx,
+            "input": question,
+            "reference_answer": reference,
+            "model_output": prediction,
+            "score": judge_result.get("score"),
+            "max_score": judge_result.get("max_score", dimension_cfg.get("score_max", 5)),
+            "passed": judge_result.get("passed"),
+            "judgement": judge_result.get("judgement"),
+            "evaluation_reason": judge_result.get("evaluation_reason", ""),
+            "error_type": judge_result.get("error_type"),
+            "dimension_scores": [
+                {"name": "judge_score", "score": judge_result.get("score", 0),
+                 "max_score": judge_result.get("max_score", dimension_cfg.get("score_max", 5))},
+            ] if judge_result else [],
+            "status": "completed",
+        })
+
+        progress_pct = int(idx / max(total, 1) * 100)
+        print(f"[eval] sample {idx}/{total} ({progress_pct}%) done")
+
+    # ---- 4. Compute basic metrics ----
+    print(f"[eval] computing basic metrics on {len(predictions)} predictions")
+    metrics_result: dict[str, Any] = {}
+
+    bleu_cfg = basic_cfg.get("bleu", {})
+    if bleu_cfg.get("enabled"):
+        metrics_result["bleu"] = _compute_bleu(references, predictions, int(bleu_cfg.get("ngram", 4)))
+
+    rouge_cfg = basic_cfg.get("rouge", {})
+    if rouge_cfg.get("enabled"):
+        metrics_result["rouge"] = _compute_rouge(references, predictions, rouge_cfg.get("methods"))
+
+    cosine_cfg = basic_cfg.get("cosine", {})
+    if cosine_cfg.get("enabled"):
+        metrics_result["cosine"] = _compute_cosine(references, predictions)
+    metrics_result["exact_match"] = _compute_exact_match(references, predictions)
+    metrics_result["text_similarity"] = _compute_text_similarity(references, predictions)
+
+    # ---- 5. Summarise ----
+    completed = len(samples)
+    if judge_enabled:
+        scored = [s for s in samples if s.get("score") is not None]
+        passed_count = len([s for s in scored if s.get("passed")])
+        avg_score = round(sum(s["score"] for s in scored) / max(len(scored), 1), output_precision)
+        max_score = dimension_cfg.get("score_max", 5)
+        overall_score = round(avg_score / max_score * 100, output_precision)
+        overall_score_max = 100
+        dimension_summary = [{
+            "name": "综合评分",
+            "score": overall_score,
+            "max_score": 100,
+            "pass_rate": round(passed_count / max(completed, 1) * 100, 1),
+        }]
+        overall_evaluation = f"评测完成：{completed} 样本，{passed_count} 通过，平均 {avg_score}/{max_score} 分"
+    else:
+        passed_count = 0
+        enabled_scores = [
+            float(item.get("score") or 0)
+            for item in metrics_result.values()
+            if isinstance(item, dict) and item.get("enabled", True) and item.get("score") is not None
+        ]
+        overall_score = round(sum(enabled_scores) / len(enabled_scores), output_precision) if enabled_scores else 0
+        overall_score_max = 100
+        dimension_summary = [
+            {
+                "name": name,
+                "score": float(item.get("score") or 0),
+                "max_score": 100,
+                "pass_rate": float(item.get("score") or 0),
+            }
+            for name, item in metrics_result.items()
+            if isinstance(item, dict) and item.get("enabled", True) and item.get("score") is not None
+        ]
+        overall_evaluation = f"评测完成：{completed} 样本（未配置 LLM 评委）"
+
+    result = {
+        "overall_score": overall_score,
+        "overall_score_max": overall_score_max,
+        "overall_evaluation": overall_evaluation,
+        "improvement_suggestions": [],
+        "dimension_summary": dimension_summary,
+        "samples": samples,
+        "sample_count": total,
+        "completed_count": completed,
+        "passed_count": passed_count,
+        "basic_metrics": metrics_result,
+    }
+
+    # ---- 6. Write results ----
+    result_path = output_dir / "eval_results.json"
+    result_path.write_text(json.dumps(result, ensure_ascii=False, indent=2), encoding="utf-8")
+    print(f"[eval] results written to {result_path}")
+    return result
+
+
+def main() -> None:
+    import argparse
+    parser = argparse.ArgumentParser(description="YG-FT Evaluation Runner")
+    parser.add_argument("--config", required=True, help="Path to eval config JSON file")
+    args = parser.parse_args()
+
+    config_path = Path(args.config)
+    if not config_path.exists():
+        print(f"FATAL: config file not found: {args.config}", file=sys.stderr)
+        sys.exit(1)
+
+    config = json.loads(config_path.read_text(encoding="utf-8"))
+    start = time.time()
+    try:
+        run_eval(config)
+        elapsed = time.time() - start
+        print(f"[eval] DONE in {elapsed:.1f}s")
+    except Exception as exc:
+        print(f"[eval] FAILED: {exc}", file=sys.stderr)
+        import traceback
+        traceback.print_exc()
+        sys.exit(1)
+
+
+if __name__ == "__main__":
+    main()

compute/engines/llama_factory/inference.py

@@ -0,0 +1,272 @@
+from __future__ import annotations
+
+import threading
+import time
+import uuid
+from typing import Any, Iterator
+
+
+class InferenceSession:
+    """Manages a loaded model for inference with LLaMA-Factory ChatModel.
+
+    Model loading is asynchronous: ``load()`` spawns a background daemon thread
+    and returns immediately with ``status == "loading"``. ``info()`` (served by
+    ``/inference/status``) is always responsive, so the platform backend can
+    poll loading progress without being blocked by a minutes-long model load —
+    which previously froze the whole compute node event loop.
+
+    State machine: idle -> loading -> ready | error, ready -> idle (unload),
+    loading -> idle (cancelled). Long operations (ChatModel build, teardown,
+    generation) never run while holding ``_state_lock``; they either run in the
+    worker thread or under ``_chat_lock`` only.
+    """
+
+    def __init__(self) -> None:
+        self._state_lock = threading.Lock()  # brief state transitions only
+        self._chat_lock = threading.Lock()  # serialize chat/teardown
+        self._status: str = "idle"
+        self._error: str = ""
+        self._request_id: str = ""
+        self._load_args: dict[str, Any] = {}
+        self._teardown_old = False  # load-while-ready: unload old before loading new
+        self._cancel_requested = False  # unload-while-loading: tear down after load finishes
+        self._load_thread: threading.Thread | None = None
+        self._model: Any = None
+        self._tokenizer: Any = None
+        self._generating_args: dict[str, Any] = {}
+        self._model_name: str = ""
+        self._adapter_path: str = ""
+        self._loaded_at: float = 0.0
+
+    @property
+    def status(self) -> str:
+        with self._state_lock:
+            return self._status
+
+    def info(self) -> dict[str, Any]:
+        with self._state_lock:
+            return {
+                "loaded": self._status == "ready",
+                "status": self._status,
+                "model_name": self._model_name,
+                "adapter_path": self._adapter_path,
+                "loaded_at": self._loaded_at,
+                "request_id": self._request_id,
+                "error": self._error,
+            }
+
+    def wait_until_loaded(self, timeout: float | None = None) -> dict[str, Any]:
+        """Wait for an in-flight async load to finish and return its outcome.
+
+        供同步消费方（如 eval_runner 子进程）使用：``load()`` 立即返回 loading 后，
+        调用本方法等待后台加载线程完成，拿到最终的 loaded/error 结果。
+        若在 timeout 秒内仍未加载完成，返回 ``status == "loading"`` 并附上超时提示。
+        """
+        with self._state_lock:
+            thread = self._load_thread
+        if thread is not None and thread.is_alive():
+            thread.join(timeout=timeout)
+        with self._state_lock:
+            loaded = self._status == "ready"
+            status = self._status
+            error = self._error
+        if not loaded and status == "loading":
+            error = error or f"model load timed out after {timeout or 'N/A'}s"
+        return {
+            "loaded": loaded,
+            "status": status,
+            "model_name": self._model_name,
+            "adapter_path": self._adapter_path,
+            "error": error,
+        }
+
+    def load(
+        self,
+        model_name_or_path,
+        adapter_name_or_path="",
+        template="qwen",
+        infer_backend="huggingface",
+        infer_dtype="auto",
+        **kwargs,
+    ) -> dict[str, Any]:
+        with self._state_lock:
+            if self._status == "loading":
+                # A model is already loading — dedupe, reuse the same request id.
+                return {"loaded": False, "status": "loading", "request_id": self._request_id}
+            self._teardown_old = self._status == "ready"
+            self._status = "loading"
+            self._error = ""
+            self._request_id = uuid.uuid4().hex[:12]
+            self._cancel_requested = False
+            self._load_args = {
+                "model_name_or_path": model_name_or_path,
+                "template": template,
+                "infer_backend": infer_backend,
+                "infer_dtype": infer_dtype,
+            }
+            if adapter_name_or_path:
+                self._load_args["adapter_name_or_path"] = adapter_name_or_path
+            self._load_args.update(kwargs)
+            self._model_name = model_name_or_path
+            self._adapter_path = adapter_name_or_path
+            self._load_thread = threading.Thread(target=self._load_worker, daemon=True)
+            self._load_thread.start()
+            return {"loaded": False, "status": "loading", "request_id": self._request_id}
+
+    def _load_worker(self) -> None:
+        """Build the ChatModel off the state lock so info() never blocks."""
+        model = None
+        tokenizer = None
+        generating_args: dict[str, Any] = {}
+        error = ""
+        try:
+            if self._teardown_old:
+                self._release_model()
+            from llamafactory.chat import ChatModel
+            from llamafactory.hparams import get_infer_args
+
+            args = dict(self._load_args)
+            infer_result = get_infer_args(args)
+            model = ChatModel(args)
+            tokenizer = getattr(model, "tokenizer", None) or model.engine.tokenizer
+            generating_args = infer_result[-1]
+            if hasattr(generating_args, "__dataclass_fields__"):
+                generating_args = {
+                    k: v for k, v in vars(generating_args).items() if not k.startswith("_")
+                }
+            else:
+                generating_args = dict(generating_args)
+        except Exception as exc:  # noqa: BLE001 - surface load failure via status
+            error = str(exc)
+        with self._state_lock:
+            if error:
+                self._model = None
+                self._tokenizer = None
+                self._status = "error"
+                self._error = error
+                return
+            if self._cancel_requested:
+                # Unload was requested while loading — drop the fresh model.
+                model = None
+                tokenizer = None
+                self._model = None
+                self._tokenizer = None
+                self._status = "idle"
+                return
+            self._model = model
+            self._tokenizer = tokenizer
+            self._generating_args = generating_args
+            self._loaded_at = time.time()
+            self._status = "ready"
+
+    def _release_model(self) -> None:
+        with self._chat_lock:
+            with self._state_lock:
+                self._status = "unloading"
+                model = self._model
+                self._model = None
+                self._tokenizer = None
+            if model is not None:
+                try:
+                    del model
+                except Exception:  # noqa: BLE001 - best-effort teardown
+                    pass
+                # 强制释放 PyTorch CUDA 缓存，真正归还 GPU 显存
+                try:
+                    import gc
+
+                    gc.collect()
+                    import torch
+
+                    if torch.cuda.is_available():
+                        torch.cuda.empty_cache()
+                        torch.cuda.synchronize()
+                except Exception:  # noqa: BLE001 - teardown must not raise
+                    pass
+            with self._state_lock:
+                self._status = "idle"
+                self._model_name = ""
+                self._adapter_path = ""
+                self._loaded_at = 0.0
+                self._error = ""
+
+    def unload(self) -> dict[str, Any]:
+        with self._state_lock:
+            if self._status == "loading":
+                # Ask the worker to tear down right after the load finishes.
+                self._cancel_requested = True
+                return {"unloaded": False, "status": "cancelling", "request_id": self._request_id}
+            was_ready = self._status == "ready"
+        if was_ready:
+            self._release_model()
+        else:
+            with self._state_lock:
+                self._model = None
+                self._tokenizer = None
+                self._status = "idle"
+                self._model_name = ""
+                self._adapter_path = ""
+                self._loaded_at = 0.0
+                self._error = ""
+        return {"unloaded": True, "status": "idle"}
+
+    def chat(self, messages, temperature=0.95, top_p=0.7, max_new_tokens=1024, do_sample=True, **kwargs) -> dict[str, Any]:
+        with self._chat_lock:
+            with self._state_lock:
+                if self._status == "loading":
+                    return {
+                        "error": f"model is still loading (request_id={self._request_id}); please retry",
+                        "response": "",
+                    }
+                if self._status == "error":
+                    return {"error": f"model load failed: {self._error}", "response": ""}
+                if self._status != "ready" or self._model is None:
+                    return {"error": "model not loaded", "response": ""}
+            try:
+                generate_kwargs = {
+                    "temperature": temperature,
+                    "top_p": top_p,
+                    "max_new_tokens": max_new_tokens,
+                    "do_sample": do_sample,
+                }
+                generate_kwargs.update(kwargs)
+                system = next((m["content"] for m in messages if m["role"] == "system"), None)
+                user_messages = [m for m in messages if m["role"] != "system"]
+                responses = []
+                for response in self._model.stream_chat(user_messages, system=system, **generate_kwargs):
+                    responses.append(response)
+                full_response = "".join(str(r) for r in responses)
+                return {"response": full_response}
+            except Exception as exc:  # noqa: BLE001 - return generation error to caller
+                return {"error": str(exc), "response": ""}
+
+    def chat_stream(self, messages, **kwargs) -> Iterator[str]:
+        with self._chat_lock:
+            with self._state_lock:
+                if self._status == "loading":
+                    yield 'data: {"error": "model is still loading; please retry"}\n\n'
+                    return
+                if self._status == "error":
+                    yield 'data: {"error": "model load failed: ' + str(self._error) + '"}\n\n'
+                    return
+                if self._status != "ready" or self._model is None:
+                    yield 'data: {"error": "model not loaded"}\n\n'
+                    return
+            try:
+                generate_kwargs = {**kwargs}
+                system = next((m["content"] for m in messages if m["role"] == "system"), None)
+                user_messages = [m for m in messages if m["role"] != "system"]
+                for new_text in self._model.stream_chat(user_messages, system=system, **generate_kwargs):
+                    yield new_text
+            except Exception as exc:  # noqa: BLE001 - stream error as SSE event
+                yield 'data: {"error": "' + str(exc) + '"}\n\n'
+
+
+_inference_session = None
+
+
+def get_inference_session() -> InferenceSession:
+    global _inference_session
+    if _inference_session is None:
+        _inference_session = InferenceSession()
+    return _inference_session

compute/requirements.txt

@@ -1,5 +1,12 @@
 fastapi>=0.111.0
 uvicorn[standard]>=0.30.0
+python-multipart>=0.0.9
 pydantic>=2.7.0
 python-dotenv>=1.0.1
 httpx>=0.27.0
+# 模型评测指标
+sacrebleu>=2.4.0
+rouge-score>=0.1.2
+scikit-learn>=1.3.0
+# LLaMA-Factory 训练引擎
+llamafactory

compute/tests/test_eval_runner.py

@@ -0,0 +1,42 @@
+from __future__ import annotations
+
+import json
+
+from compute.engines.llama_factory.eval_runner import _load_dataset
+
+
+def _write(tmp_path, name: str, text: str) -> str:
+    path = tmp_path / name
+    path.write_text(text, encoding="utf-8")
+    return str(path)
+
+
+def test_load_jsonl_multiline(tmp_path) -> None:
+    path = _write(
+        tmp_path,
+        "eval.jsonl",
+        '{"question": "q1", "answer": "a1"}\n{"question": "q2", "answer": "a2"}\n',
+    )
+    assert _load_dataset(path) == [
+        {"question": "q1", "answer": "a1"},
+        {"question": "q2", "answer": "a2"},
+    ]
+
+
+def test_load_json_array(tmp_path) -> None:
+    path = _write(
+        tmp_path,
+        "eval.json",
+        json.dumps([{"question": "x", "answer": "y"}]),
+    )
+    assert _load_dataset(path) == [{"question": "x", "answer": "y"}]
+
+
+def test_load_jsonl_with_bom_and_embedded_array(tmp_path) -> None:
+    """jsonl 带 BOM 且单行内嵌 JSON 数组，都应正常加载。"""
+    path = _write(
+        tmp_path,
+        "eval.jsonl",
+        "" + json.dumps([{"question": "a", "answer": "b"}, {"question": "c", "answer": "d"}]),
+    )
+    assert len(_load_dataset(path)) == 2

compute/tests/test_file_download_security.py

@@ -0,0 +1,63 @@
+"""compute ``download_file`` 端点安全回归测试。
+
+修复前 ``file_id`` 直接拼进 glob 模式且不校验路径包含关系，可通过 ``../``
+穿越出 upload 目录，并在 Linux 上跟随符号链接读取任意文件。
+修复后：file_id 仅允许字母/数字/下划线/连字符，返回前对解析后的路径
+做 upload 根目录包含性校验。
+"""
+from __future__ import annotations
+
+from pathlib import Path
+
+import pytest
+from fastapi.testclient import TestClient
+
+
+def _make_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> TestClient:
+    monkeypatch.setenv("TRAINING_LOG_ROOT", str(tmp_path / "logs"))
+    monkeypatch.setenv("YG_FT_DATA_ROOT", str(tmp_path / "data"))
+    monkeypatch.setenv("COMPUTE_EXECUTION_MODE", "simulator")
+    monkeypatch.setenv("COMPUTE_AUTH_ENABLED", "false")
+    monkeypatch.delenv("ENABLE_DOCS", raising=False)
+    from compute.api.main import create_app
+
+    return TestClient(create_app())
+
+
+def _upload_root(tmp_path: Path) -> Path:
+    root = tmp_path / "data" / "uploads"
+    root.mkdir(parents=True, exist_ok=True)
+    return root
+
+
+def test_download_legit_file(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+    client = _make_client(tmp_path, monkeypatch)
+    (_upload_root(tmp_path) / "file_123456_hello.txt").write_text("HELLO-DOWNLOAD", encoding="utf-8")
+    response = client.get("/modelTF/compute/files/file_123456/download")
+    assert response.status_code == 200
+    assert response.content == b"HELLO-DOWNLOAD"
+
+
+def test_download_rejects_traversal_file_id(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+    client = _make_client(tmp_path, monkeypatch)
+    outside = tmp_path / "secret" / "passwd_1.txt"
+    outside.parent.mkdir(parents=True, exist_ok=True)
+    outside.write_text("TOP-SECRET", encoding="utf-8")
+    for file_id in ["..", "file.123", "..%2F..%2Fsecret%2Fpasswd", "file%20name"]:
+        response = client.get(f"/modelTF/compute/files/{file_id}/download")
+        assert response.status_code in (400, 404), f"file_id={file_id!r} -> {response.status_code}"
+        assert b"TOP-SECRET" not in response.content
+
+
+def test_download_blocks_symlink_escape(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+    client = _make_client(tmp_path, monkeypatch)
+    upload_root = _upload_root(tmp_path)
+    outside = tmp_path / "secret.txt"
+    outside.write_text("TOP-SECRET", encoding="utf-8")
+    try:
+        (upload_root / "file_999999_link.txt").symlink_to(outside)
+    except OSError:
+        pytest.skip("symlink creation not permitted on this platform")
+    response = client.get("/modelTF/compute/files/file_999999/download")
+    assert response.status_code == 404
+    assert b"TOP-SECRET" not in response.content

compute/tests/test_inference_session.py

@@ -0,0 +1,146 @@
+from __future__ import annotations
+
+import sys
+import time
+import types
+from typing import Any
+
+import pytest
+
+from compute.engines.llama_factory.inference import InferenceSession
+
+# 模拟模型加载耗时，用于验证 load() 立即返回、info() 不阻塞
+LOAD_DELAY = 0.2
+
+
+class FakeChatModel:
+    def __init__(self, args: dict[str, Any]) -> None:
+        time.sleep(LOAD_DELAY)
+        self.tokenizer = object()
+        self.engine = types.SimpleNamespace(tokenizer=object())
+        self._output = "hello from model"
+
+    def stream_chat(self, *args, **kwargs):
+        for _ in range(1):
+            yield self._output
+
+
+class FailingChatModel:
+    def __init__(self, args: dict[str, Any]) -> None:
+        time.sleep(LOAD_DELAY)
+        raise RuntimeError("boom: fake load failure")
+
+
+def _get_infer_args(args: dict[str, Any]) -> list[Any]:
+    # 最后一个元素为 generating_args，worker 会转成 dict
+    return [None, None, {"temperature": 0.7}]
+
+
+def _install_llamafactory(monkeypatch, chat_model: type) -> None:
+    llmf = types.ModuleType("llamafactory")
+    chat_mod = types.ModuleType("llamafactory.chat")
+    hparams_mod = types.ModuleType("llamafactory.hparams")
+    chat_mod.ChatModel = chat_model
+    hparams_mod.get_infer_args = _get_infer_args
+    llmf.chat = chat_mod
+    llmf.hparams = hparams_mod
+    monkeypatch.setitem(sys.modules, "llamafactory", llmf)
+    monkeypatch.setitem(sys.modules, "llamafactory.chat", chat_mod)
+    monkeypatch.setitem(sys.modules, "llamafactory.hparams", hparams_mod)
+
+
+@pytest.fixture
+def stub_llamafactory(monkeypatch) -> None:
+    _install_llamafactory(monkeypatch, FakeChatModel)
+
+
+@pytest.fixture
+def stub_failing_llamafactory(monkeypatch) -> None:
+    _install_llamafactory(monkeypatch, FailingChatModel)
+
+
+def _wait_for_status(session: InferenceSession, status: str, timeout: float = 3.0) -> bool:
+    deadline = time.time() + timeout
+    while time.time() < deadline:
+        if session.info()["status"] == status:
+            return True
+        time.sleep(0.02)
+    return False
+
+
+def test_load_returns_immediately_then_ready(stub_llamafactory) -> None:
+    session = InferenceSession()
+    started = time.time()
+    result = session.load("/models/qwen")
+    assert result["status"] == "loading"
+    assert result["loaded"] is False
+    assert result["request_id"]
+    # 在慢加载完成前就返回，且 info() 加载期间可响应
+    assert time.time() - started < LOAD_DELAY
+    assert session.info()["status"] == "loading"
+    assert _wait_for_status(session, "ready")
+    info = session.info()
+    assert info["loaded"] is True
+    assert info["status"] == "ready"
+    assert info["model_name"] == "/models/qwen"
+
+
+def test_second_load_while_loading_deduped(stub_llamafactory) -> None:
+    session = InferenceSession()
+    r1 = session.load("/models/a")
+    r2 = session.load("/models/b")
+    assert r2["status"] == "loading"
+    assert r2["request_id"] == r1["request_id"]
+    assert _wait_for_status(session, "ready")
+    assert session.info()["status"] == "ready"
+
+
+def test_load_error_surfaces_in_status(stub_failing_llamafactory) -> None:
+    session = InferenceSession()
+    session.load("/models/bad")
+    assert _wait_for_status(session, "error")
+    assert "boom" in session.info()["error"]
+
+
+def test_unload_while_loading_cancels(stub_llamafactory) -> None:
+    session = InferenceSession()
+    session.load("/models/qwen")
+    result = session.unload()
+    assert result["status"] == "cancelling"
+    assert _wait_for_status(session, "idle")
+
+
+def test_chat_while_loading_returns_loading_error(stub_llamafactory) -> None:
+    session = InferenceSession()
+    session.load("/models/qwen")
+    out = session.chat([{"role": "user", "content": "hi"}])
+    assert "still loading" in (out.get("error") or "")
+    assert _wait_for_status(session, "ready")
+    out = session.chat([{"role": "user", "content": "hi"}])
+    assert out.get("response") == "hello from model"
+
+
+def test_chat_stream_while_loading_yields_error(stub_llamafactory) -> None:
+    session = InferenceSession()
+    session.load("/models/qwen")
+    chunks = list(session.chat_stream([{"role": "user", "content": "hi"}]))
+    assert any("still loading" in c for c in chunks)
+
+
+def test_wait_until_loaded_blocks_until_ready(stub_llamafactory) -> None:
+    session = InferenceSession()
+    result = session.load("/models/qwen")
+    assert result["status"] == "loading"
+    # 同步等待后台加载线程完成
+    outcome = session.wait_until_loaded(timeout=3.0)
+    assert outcome["loaded"] is True
+    assert outcome["status"] == "ready"
+
+
+def test_wait_until_loaded_reports_load_error(stub_failing_llamafactory) -> None:
+    session = InferenceSession()
+    session.load("/models/bad")
+    outcome = session.wait_until_loaded(timeout=3.0)
+    assert outcome["loaded"] is False
+    assert outcome["status"] == "error"
+    assert "boom" in outcome["error"]

compute/tests/test_llama_factory_adapter.py

@@ -0,0 +1,92 @@
+from __future__ import annotations
+
+import json
+
+from compute.engines.llama_factory.adapter import _validate_dataset_columns, build_command
+
+
+def test_build_command_uses_explicit_validation_dataset_without_resplitting() -> None:
+    result = build_command(
+        {
+            "base_model": "/models/qwen",
+            "dataset": "ygft_dataset_train",
+            "eval_dataset": "ygft_dataset_validation",
+            "dataset_dir": "/datasets/example",
+            "output_dir": "/outputs/example",
+            "val_size": 0.1,
+        }
+    )
+
+    assert result.command[result.command.index("--dataset") + 1] == "ygft_dataset_train"
+    assert result.command[result.command.index("--eval_dataset") + 1] == (
+        "ygft_dataset_validation"
+    )
+    assert "--do_eval" in result.command
+    assert "--val_size" not in result.command
+
+
+def _write(tmp_path, name: str, lines: list[dict]) -> object:
+    path = tmp_path / name
+    path.write_text(
+        "".join(json.dumps(line, ensure_ascii=False) + "\n" for line in lines),
+        encoding="utf-8",
+    )
+    return path
+
+
+def test_jsonl_alpaca_without_input_column_passes_validation(tmp_path) -> None:
+    """纯 jsonl Alpaca 数据缺省 input 字段（常见），不应被校验拦截。"""
+    _write(tmp_path, "train.jsonl", [{"instruction": "hi", "output": "hello"}])
+    errors = _validate_dataset_columns(
+        {
+            "dataset_dir": str(tmp_path),
+            "dataset_info": {
+                "ygft_a": {
+                    "file_name": "train.jsonl",
+                    "formatting": "alpaca",
+                    "columns": {"prompt": "instruction", "query": "input", "response": "output"},
+                }
+            },
+        }
+    )
+    assert errors == []
+
+
+def test_jsonl_sharegpt_passes_validation(tmp_path) -> None:
+    """ShareGPT 格式 jsonl（messages）应通过校验。"""
+    _write(
+        tmp_path,
+        "msg.jsonl",
+        [{"messages": [{"role": "user", "content": "hi"}, {"role": "assistant", "content": "hello"}]}],
+    )
+    errors = _validate_dataset_columns(
+        {
+            "dataset_dir": str(tmp_path),
+            "dataset_info": {
+                "ygft_m": {
+                    "file_name": "msg.jsonl",
+                    "formatting": "sharegpt",
+                    "columns": {"messages": "messages"},
+                }
+            },
+        }
+    )
+    assert errors == []
+
+
+def test_jsonl_missing_response_still_rejected(tmp_path) -> None:
+    """缺 output（response）仍应报错——没有答案无法做有监督微调。"""
+    _write(tmp_path, "train.jsonl", [{"instruction": "hi"}])
+    errors = _validate_dataset_columns(
+        {
+            "dataset_dir": str(tmp_path),
+            "dataset_info": {
+                "ygft_a": {
+                    "file_name": "train.jsonl",
+                    "formatting": "alpaca",
+                    "columns": {"prompt": "instruction", "query": "input", "response": "output"},
+                }
+            },
+        }
+    )
+    assert errors and "output" in errors[0]

compute/tests/test_security.py

@@ -0,0 +1,40 @@
+"""计算节点文档路由（/docs、/redoc、/openapi.json）安全开关测试。
+
+生产默认（COMPUTE_AUTH_ENABLED=true）关闭文档路由，避免未授权泄露 API 结构；
+显式配置 ENABLE_DOCS 可覆盖默认行为。
+"""
+from __future__ import annotations
+
+from compute.api.security import docs_enabled, docs_kwargs
+
+
+def test_docs_disabled_when_auth_enabled(monkeypatch) -> None:
+    monkeypatch.delenv("ENABLE_DOCS", raising=False)
+    monkeypatch.setenv("COMPUTE_AUTH_ENABLED", "true")
+    assert docs_enabled() is False
+    assert docs_kwargs() == {"docs_url": None, "redoc_url": None, "openapi_url": None}
+
+
+def test_docs_enabled_when_auth_disabled(monkeypatch) -> None:
+    monkeypatch.delenv("ENABLE_DOCS", raising=False)
+    monkeypatch.setenv("COMPUTE_AUTH_ENABLED", "false")
+    assert docs_enabled() is True
+    assert docs_kwargs() == {}
+
+
+def test_docs_env_override_enables_with_auth(monkeypatch) -> None:
+    monkeypatch.setenv("ENABLE_DOCS", "true")
+    monkeypatch.setenv("COMPUTE_AUTH_ENABLED", "true")
+    assert docs_enabled() is True
+
+
+def test_docs_env_override_disables_without_auth(monkeypatch) -> None:
+    monkeypatch.setenv("ENABLE_DOCS", "false")
+    monkeypatch.setenv("COMPUTE_AUTH_ENABLED", "false")
+    assert docs_enabled() is False
+
+
+def test_docs_default_when_auth_env_missing(monkeypatch) -> None:
+    monkeypatch.delenv("ENABLE_DOCS", raising=False)
+    monkeypatch.delenv("COMPUTE_AUTH_ENABLED", raising=False)
+    assert docs_enabled() is False

design-qa.md

@@ -328,6 +328,57 @@ final result: blocked
 
 ---
 
+# Failed Result Regeneration Design QA
+
+## Evidence
+
+- Source visual truth: `/var/folders/nk/yks07zp14wb4rv3jqq0pt_4h0000gn/T/codex-clipboard-0c2e6f90-aaac-4817-9084-bc31496aed0a.png`.
+- Implementation route: `http://localhost:16801/data-process/:id/workflow`, step 6.
+- Browser evidence: `/Users/caoxiaozhu/.codex/visualizations/2026/07/27/019fa277-626b-76a1-af02-48948d66c7ec/data-process-empty-list.png`.
+- Viewport: 1280 × 720.
+- Target state: a completed task on step 6 with a selected invalid result.
+- Available browser state: authenticated data-process list with zero tasks; no representative invalid result could be opened without creating or mutating local business data.
+
+## Static and automated evidence
+
+- Invalid results render a primary plain `重新生成` button in the right side of the result header.
+- Valid results keep the existing `恢复生成结果` action.
+- While regeneration is running, the selected row shows a spinner, the button shows loading, editors are disabled, and the footer confirmation action is disabled.
+- A successful response replaces only the selected row in place; a failed second generation leaves the original invalid row untouched.
+- `regression-data-process-wizard.mjs`: passed.
+- `regression-data-process-detail.mjs`: passed.
+- `regression-data-process-list.mjs`: passed.
+- `vue-tsc -b --noEmit`: passed.
+- Relevant backend tests: 97 passed, with one third-party deprecation warning.
+
+## Required fidelity surfaces
+
+- Typography and hierarchy: the action uses the existing Element Plus small-button hierarchy and remains secondary to result content.
+- Spacing and layout rhythm: the action is placed in the existing flex header's right action slot, matching the source location.
+- Colors and tokens: the action uses the existing product primary color and existing Font Awesome refresh icon.
+- Copy and content: the visible label is exactly `重新生成`; only server-saved invalid results expose it.
+- Interaction: optimistic concurrency protects against overwriting a newer edit, and persistence happens only after the replacement passes generation and quality validation.
+
+## Findings
+
+- [P2] Representative rendered comparison unavailable
+  Location: step 6 result header with an invalid result selected.
+  Evidence: the authenticated local database currently contains zero data-processing tasks, so the target state cannot be reached without creating test business data or invoking the configured model.
+  Impact: automated and static checks prove the contract and placement, but cannot prove pixel-level spacing against the supplied screenshot.
+  Fix: open any existing failed result after one is available, capture the same 1280 × 720 state, and compare it side by side with the source screenshot.
+
+## Comparison history
+
+### Iteration 1 — blocked
+
+- Source screenshot was inspected and the implementation was aligned to its existing right-header action slot.
+- Local login and list navigation succeeded.
+- The target result state was unavailable because the local task list was empty.
+
+final result: blocked
+
+---
+
 # Dataset Version Actions Design QA
 
 ## Evidence

docker/README.md

@@ -56,7 +56,7 @@ $images | ForEach-Object { docker pull $_ }
 | PostgreSQL | `15432` | `5432` | 开发阶段内置数据库 |
 | Redis | `16379` | `6379` | 开发阶段内置缓存 |
 | Compute API | `19100` | `9100` | 算力服务器 API |
-| File Gateway | `19101` | 后续服务端口 | 当前预留，后续拆出文件网关服务时使用 |
+| File Gateway | `19101` | `9100` | 当前由 Compute API 暴露文件网关契约，后续可拆为独立服务 |
 
 注意：`8000` 是后端容器内部端口，不作为宿主机对外访问端口。宿主机或浏览器应访问 `http://<app-server-ip>:17861/modelTF/health`；前端 Nginx 容器在 Docker 网络内部访问 `http://backend-api:8000/modelTF/...`。
 
@@ -142,6 +142,16 @@ docker compose logs --tail=80 frontend
 
 如果使用企业统一 PostgreSQL/Redis，修改 `docker/app/.env`：
 
+如果前端 Nginx 日志出现 `open() "/usr/share/nginx/html/modelTF/login" failed` 或 `open() "/usr/share/nginx/html/login" failed`，说明当前容器没有加载项目的 Nginx 代理配置，`/modelTF/*` 被当成静态文件查找。处理方式：
+
+```bash
+cd <repo-root>/docker/app
+docker compose up -d --force-recreate frontend
+docker compose exec frontend nginx -T | grep -n "location.*modelTF" -A12
+```
+
+正常配置中应存在 `location ^~ /modelTF/`，并代理到 `BACKEND_PROXY_PASS`，默认是 `http://backend-api:8000`。
+
 ```env
 DATABASE_URL=postgresql+psycopg://<user>:<password>@<postgres-host>:15432/<db>
 REDIS_URL=redis://<redis-host>:16379/0
@@ -192,10 +202,28 @@ GET http://<compute-server-ip>:19100/modelTF/v1/compute/health
 ```text
 ../../compute -> /app/compute
 ${YG_FT_DATA_ROOT_HOST} -> /data/yg-ft
-../../runtime/compute/logs -> /opt/yg-ft/logs/compute
+${YG_FT_MODEL_ROOT_HOST} -> /data/yg-ft/models
-../../runtime/compute/training-logs -> /opt/yg-ft/logs/training
+${YG_FT_DATASET_ROOT_HOST} -> /data/yg-ft/datasets
+${YG_FT_OUTPUT_ROOT_HOST} -> /data/yg-ft/outputs
+${COMPUTE_LOG_ROOT_HOST} -> /opt/yg-ft/logs/compute
+${TRAINING_LOG_ROOT_HOST} -> /opt/yg-ft/logs/training
 ```
 
+算力服务器启动前必须先在宿主机创建持久化目录，基座模型、训练数据、训练产物和训练日志都应落在宿主机磁盘上，不能只写入容器层。推荐默认目录：
+
+```bash
+cd <repo-root>/docker/compute
+mkdir -p data/yg-ft/models \
+  data/yg-ft/datasets \
+  data/yg-ft/outputs \
+  data/yg-ft/logs/compute \
+  data/yg-ft/logs/training
+```
+
+默认 `docker/compute/.env.example` 使用 `./data/yg-ft`，该相对路径以 `docker/compute/docker-compose.yml` 所在目录为基准，因此实际宿主机目录是 `<repo-root>/docker/compute/data/yg-ft`。如企业环境模型盘、数据盘、产物盘分盘挂载，可在 `docker/compute/.env` 中分别调整 `YG_FT_MODEL_ROOT_HOST`、`YG_FT_DATASET_ROOT_HOST`、`YG_FT_OUTPUT_ROOT_HOST`、`COMPUTE_LOG_ROOT_HOST`、`TRAINING_LOG_ROOT_HOST`，容器内路径建议保持 `/data/yg-ft/models`、`/data/yg-ft/datasets`、`/data/yg-ft/outputs`，避免训练参数和节点配置复杂化。
+
+页面上传数据集时，文件先进入 Backend API，再由 Backend API 调用目标算力节点的 `POST /modelTF/compute/files/upload`，写入容器内 `/data/yg-ft/datasets/{dataset_id}/`。在默认开发配置下，宿主机可在 `<repo-root>/docker/compute/data/yg-ft/datasets/{dataset_id}/` 看到对应文件。仅创建 bind mount 不会自动让应用侧上传文件出现在算力目录，必须通过这条 File Gateway 链路同步。
+
 ## 应用与算力分离部署
 
 应用服务器只需要主动访问算力服务器，不要求算力服务器回调应用服务器。
@@ -207,7 +235,7 @@ COMPUTE_API_BASE_URL=http://<compute-server-ip>:19100
 FILE_GATEWAY_BASE_URL=http://<compute-server-ip>:19101
 COMPUTE_SERVICE_TOKEN=change_me
 COMPUTE_STATUS_SYNC_MODE=polling
-COMPUTE_POLL_INTERVAL_SECONDS=10
+COMPUTE_POLL_INTERVAL_SECONDS=3
 COMPUTE_POLL_BATCH_SIZE=100
 ```
 
@@ -222,6 +250,8 @@ Frontend
     <- Backend Worker 定时轮询 Compute API
 ```
 
+算力服务默认开启服务间鉴权。`docker/compute/.env` 中保持 `COMPUTE_AUTH_ENABLED=true`，并确保 `COMPUTE_SERVICE_TOKEN` 与 `docker/app/.env` 一致；健康检查路径仍可用于容器探活。
+
 ## 多算力节点部署
 
 多算力节点仍按“单机多 GPU 节点”部署。每台 GPU 服务器都独立部署一套 `docker/compute`：
@@ -234,6 +264,8 @@ gpu-node-03: docker/compute + /data/yg-ft + 19100/19101
 
 节点之间默认不互访。应用平台主动访问每个节点的 Compute API/File Gateway，并通过 `compute_nodes`、`resource_replicas`、`resource_sync_jobs` 统一调度和同步。
 
+节点地址、权重、标签、启用状态和本地路径在前端“算力节点”页面动态维护。新增或编辑节点后，点击“测试”会由 Backend API 主动访问该节点的 `GET /modelTF/v1/compute/health` 和 `GET /modelTF/compute/resources/gpus`，并把健康信息与 GPU 清单同步到 PostgreSQL。
+
 ## 常用命令
 
 重新构建应用镜像：

docker/app/.env

@@ -1,6 +1,8 @@
 APP_ENV=prod
 APP_NAME=YG Fine-Tune Platform API
 MODELTF_ROUTE_PREFIX=/modelTF
+# 禁止未授权访问 Swagger/ReDoc/OpenAPI 文档；调试时可临时改为 true
+ENABLE_DOCS=false
 CORS_ALLOW_ORIGINS=http://localhost:16801,http://127.0.0.1:16801
 
 FRONTEND_IMAGE=yg-ft-frontend-runtime:latest
@@ -13,15 +15,17 @@ POSTGRES_PORT=15432
 REDIS_PORT=16379
 
 POSTGRES_DB=yg_ft
-POSTGRES_USER=yg_ft
+POSTGRES_USER=root
-POSTGRES_PASSWORD=change_me
+POSTGRES_PASSWORD=8811614287327Leo
-DATABASE_URL=postgresql+psycopg://yg_ft:change_me@postgres:5432/yg_ft
+DATABASE_URL=postgresql+psycopg://root:8811614287327Leo@www.caoxiaozhu.com:5432/yg_ft
 
-REDIS_URL=redis://redis:6379/0
+# Redis 访问鉴权：requirepass 密码；REDIS_URL 已内嵌密码（redis://:<密码>@redis:6379/0）
+REDIS_PASSWORD=Tvhrf659WaX-S1B8FG6c2kSZK07XTv82
+REDIS_URL=redis://:Tvhrf659WaX-S1B8FG6c2kSZK07XTv82@redis:6379/0
 
-# Development uses the built-in PostgreSQL/Redis services in docker-compose.yml.
+# PostgreSQL uses the shared external database. The local postgres service is disabled in docker-compose.yml.
-# For enterprise infrastructure, replace DATABASE_URL/REDIS_URL and remove or disable those services.
+# Redis still uses the built-in service during current development.
-USE_BUILTIN_POSTGRES=true
+USE_BUILTIN_POSTGRES=false
 USE_BUILTIN_REDIS=true
 
 LOG_LEVEL=INFO
@@ -43,3 +47,4 @@ COMPUTE_MODE=real
 COMPUTE_STATUS_SYNC_MODE=polling
 COMPUTE_POLL_INTERVAL_SECONDS=10
 COMPUTE_POLL_BATCH_SIZE=100
+COMPUTE_REQUEST_TIMEOUT_SECONDS=5

docker/app/Dockerfile.backend

@@ -2,7 +2,8 @@ FROM python:3.12-slim
 
 ENV PYTHONDONTWRITEBYTECODE=1 \
     PYTHONUNBUFFERED=1 \
-    PIP_NO_CACHE_DIR=1
+    PIP_NO_CACHE_DIR=1 \
+    TIKTOKEN_CACHE_DIR=/opt/tiktoken_cache
 
 WORKDIR /app
 
@@ -11,11 +12,14 @@ RUN pip install --upgrade pip -i https://pypi.tuna.tsinghua.edu.cn/simple \
     && pip install -r /tmp/requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple \
     && rm -f /tmp/requirements.txt
 
-RUN python -c "import fastapi, uvicorn, psycopg, sqlalchemy, redis, jwt, passlib, httpx, alembic; print('backend dependency check ok')"
+RUN python -c "import fastapi, uvicorn, psycopg, psycopg_pool, sqlalchemy, redis, jwt, passlib, httpx, alembic; print('backend dependency check ok')"
 
 RUN mkdir -p /opt/yg-ft/logs/backend /data/yg-ft \
     && chmod -R 0775 /opt/yg-ft /data/yg-ft
 
+# 离线打包 tiktoken cl100k_base 词表，避免无网环境下运行时联网下载
+COPY docker/app/tiktoken /opt/tiktoken_cache
+
 EXPOSE 8000
 
 CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

docker/app/docker-compose.yml

@@ -21,6 +21,8 @@ services:
           ls -la /usr/share/nginx/html;
           exit 1;
         fi;
+        envsubst '$$BACKEND_PROXY_PASS' < /etc/nginx/templates/default.conf.template > /etc/nginx/conf.d/default.conf;
+        nginx -t;
         nginx -g 'daemon off;'
     networks:
       - yg-ft-app
@@ -30,8 +32,6 @@ services:
     image: ${BACKEND_API_IMAGE:-yg-ft-backend-api:latest}
     container_name: yg-ft-backend-api
     depends_on:
-      postgres:
-        condition: service_healthy
       redis:
         condition: service_healthy
     expose:
@@ -42,10 +42,11 @@ services:
       APP_ENV: ${APP_ENV:-prod}
       APP_NAME: ${APP_NAME:-YG Fine-Tune Platform API}
       MODELTF_ROUTE_PREFIX: ${MODELTF_ROUTE_PREFIX:-/modelTF}
+      ENABLE_DOCS: ${ENABLE_DOCS:-false}
       CORS_ALLOW_ORIGINS: ${CORS_ALLOW_ORIGINS:-http://localhost:16801,http://127.0.0.1:16801}
-      DATABASE_URL: ${DATABASE_URL:-postgresql+psycopg://yg_ft:change_me@postgres:5432/yg_ft}
+      DATABASE_URL: ${DATABASE_URL:-postgresql+psycopg://root:8811614287327Leo@www.caoxiaozhu.com:5432/yg_ft}
-      REDIS_URL: ${REDIS_URL:-redis://redis:6379/0}
+      REDIS_URL: ${REDIS_URL:-redis://:${REDIS_PASSWORD:-change_me}@redis:6379/0}
-      USE_BUILTIN_POSTGRES: ${USE_BUILTIN_POSTGRES:-true}
+      USE_BUILTIN_POSTGRES: ${USE_BUILTIN_POSTGRES:-false}
       USE_BUILTIN_REDIS: ${USE_BUILTIN_REDIS:-true}
       LOG_LEVEL: ${LOG_LEVEL:-INFO}
       LOG_DIR: ${LOG_DIR:-/opt/yg-ft/logs/backend}
@@ -58,8 +59,9 @@ services:
       FILE_GATEWAY_BASE_URL: ${FILE_GATEWAY_BASE_URL:-http://compute-api:9101}
       COMPUTE_MODE: ${COMPUTE_MODE:-real}
       COMPUTE_STATUS_SYNC_MODE: ${COMPUTE_STATUS_SYNC_MODE:-polling}
-      COMPUTE_POLL_INTERVAL_SECONDS: ${COMPUTE_POLL_INTERVAL_SECONDS:-10}
+      COMPUTE_POLL_INTERVAL_SECONDS: ${COMPUTE_POLL_INTERVAL_SECONDS:-3}
       COMPUTE_POLL_BATCH_SIZE: ${COMPUTE_POLL_BATCH_SIZE:-100}
+      COMPUTE_REQUEST_TIMEOUT_SECONDS: ${COMPUTE_REQUEST_TIMEOUT_SECONDS:-5}
       PYTHONPATH: /app
     volumes:
       - ../../backend:/app:ro
@@ -75,32 +77,37 @@ services:
       start_period: 20s
     restart: unless-stopped
 
-  postgres:
+  # PostgreSQL uses the shared external database configured by DATABASE_URL in docker/app/.env.
-    image: postgres:16-alpine
+  # Keep this local service commented out unless development needs an isolated database again.
-    container_name: yg-ft-postgres
+  # postgres:
-    environment:
+  #   image: postgres:16-alpine
-      POSTGRES_DB: ${POSTGRES_DB:-yg_ft}
+  #   container_name: yg-ft-postgres
-      POSTGRES_USER: ${POSTGRES_USER:-yg_ft}
+  #   environment:
-      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-change_me}
+  #     POSTGRES_DB: ${POSTGRES_DB:-yg_ft}
-      PGDATA: /var/lib/postgresql/data/pgdata
+  #     POSTGRES_USER: ${POSTGRES_USER:-yg_ft}
-    volumes:
+  #     POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-change_me}
-      - postgres_data:/var/lib/postgresql/data
+  #     PGDATA: /var/lib/postgresql/data/pgdata
-      - ../../backend/app/db/sql/001_platform_runtime.sql:/docker-entrypoint-initdb.d/001-platform-runtime.sql:ro
+  #   volumes:
-    ports:
+  #     - postgres_data:/var/lib/postgresql/data
-      - "${POSTGRES_PORT:-15432}:5432"
+  #     - ../../backend/app/db/sql/001_platform_runtime.sql:/docker-entrypoint-initdb.d/001-platform-runtime.sql:ro
-    networks:
+  #   ports:
-      - yg-ft-app
+  #     - "${POSTGRES_PORT:-15432}:5432"
-    healthcheck:
+  #   networks:
-      test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
+  #     - yg-ft-app
-      interval: 10s
+  #   healthcheck:
-      timeout: 5s
+  #     test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
-      retries: 5
+  #     interval: 10s
-    restart: unless-stopped
+  #     timeout: 5s
+  #     retries: 5
+  #   restart: unless-stopped
 
   redis:
     image: redis:7-alpine
     container_name: yg-ft-redis
-    command: ["redis-server", "--appendonly", "yes"]
+    command: ["redis-server", "--appendonly", "yes", "--requirepass", "${REDIS_PASSWORD:-change_me}"]
+    environment:
+      # redis-cli 健康检查免命令行传密码（避免 -a 泄露进程参数）
+      REDISCLI_AUTH: ${REDIS_PASSWORD:-change_me}
     volumes:
       - redis_data:/data
     ports:
@@ -119,5 +126,5 @@ networks:
     name: yg-ft-app
 
 volumes:
-  postgres_data:
+  # postgres_data:
   redis_data:

docker/compute/.env

@@ -0,0 +1,42 @@
+COMPUTE_ENV=prod
+COMPUTE_HOST_ID=gpu-node-01
+COMPUTE_EXECUTION_MODE=real
+MODELTF_ROUTE_PREFIX=/modelTF
+# 禁止未授权访问 Swagger/ReDoc/OpenAPI 文档；调试时可临时改为 true
+ENABLE_DOCS=false
+# Five-digit host ports exposed outside the compute server.
+COMPUTE_API_PORT=19100
+FILE_GATEWAY_PORT=19101
+COMPUTE_API_IMAGE=yg-ft-compute-api:latest
+
+# The application server actively polls Compute API; compute server does not need reverse access.
+COMPUTE_AUTH_ENABLED=true
+COMPUTE_SERVICE_TOKEN=change_me
+ENABLE_APP_CALLBACK=false
+
+# LLaMA-Factory is provided by the official hiyouga/llamafactory base image.
+LLAMA_FACTORY_HOME=/app/LLaMA-Factory
+
+YG_FT_DATA_ROOT=/data/yg-ft
+YG_FT_DATA_ROOT_HOST=./data/yg-ft
+YG_FT_MODEL_ROOT=/data/yg-ft/models
+YG_FT_MODEL_ROOT_HOST=./data/yg-ft/models
+YG_FT_DATASET_ROOT=/data/yg-ft/datasets
+YG_FT_DATASET_ROOT_HOST=./data/yg-ft/datasets
+YG_FT_OUTPUT_ROOT=/data/yg-ft/outputs
+YG_FT_OUTPUT_ROOT_HOST=./data/yg-ft/outputs
+TRAINING_LOG_ROOT=/opt/yg-ft/logs/training
+TRAINING_LOG_ROOT_HOST=./data/yg-ft/logs/training
+COMPUTE_LOG_ROOT_HOST=./data/yg-ft/logs/compute
+
+# Optional fallback used when nvidia-smi is unavailable.
+# Leave COMPUTE_GPU_COUNT=0 on real GPU servers with working NVIDIA runtime.
+COMPUTE_GPU_COUNT=0
+COMPUTE_GPU_NAME=NVIDIA A800-SXM4-80GB
+COMPUTE_GPU_MEMORY_GB=80
+COMPUTE_GPU_POWER_LIMIT_W=300
+
+LOG_DIR=/opt/yg-ft/logs/compute
+CUDA_VISIBLE_DEVICES=0
+NVIDIA_VISIBLE_DEVICES=all
+NVIDIA_DRIVER_CAPABILITIES=compute,utility

docker/compute/.env.example

@@ -2,20 +2,43 @@ COMPUTE_ENV=prod
 COMPUTE_HOST_ID=gpu-node-01
 COMPUTE_EXECUTION_MODE=real
 MODELTF_ROUTE_PREFIX=/modelTF
+# 禁止未授权访问 Swagger/ReDoc/OpenAPI 文档；调试时可临时改为 true
+ENABLE_DOCS=false
 # Five-digit host ports exposed outside the compute server.
 COMPUTE_API_PORT=19100
 FILE_GATEWAY_PORT=19101
 COMPUTE_API_IMAGE=yg-ft-compute-api:latest
 
 # The application server actively polls Compute API; compute server does not need reverse access.
+COMPUTE_AUTH_ENABLED=true
 COMPUTE_SERVICE_TOKEN=change_me
 ENABLE_APP_CALLBACK=false
 
 # LLaMA-Factory is provided by the official hiyouga/llamafactory base image.
 LLAMA_FACTORY_HOME=/app/LLaMA-Factory
 
+# Persistent host directories on the compute server.
+# Create these directories before starting docker compose. They are mounted into
+# the container so base models, datasets, training outputs and logs survive
+# container recreation or image upgrades.
 YG_FT_DATA_ROOT=/data/yg-ft
-YG_FT_DATA_ROOT_HOST=/data/yg-ft
+YG_FT_DATA_ROOT_HOST=./data/yg-ft
+YG_FT_MODEL_ROOT=/data/yg-ft/models
+YG_FT_MODEL_ROOT_HOST=./data/yg-ft/models
+YG_FT_DATASET_ROOT=/data/yg-ft/datasets
+YG_FT_DATASET_ROOT_HOST=./data/yg-ft/datasets
+YG_FT_OUTPUT_ROOT=/data/yg-ft/outputs
+YG_FT_OUTPUT_ROOT_HOST=./data/yg-ft/outputs
+TRAINING_LOG_ROOT=/opt/yg-ft/logs/training
+TRAINING_LOG_ROOT_HOST=./data/yg-ft/logs/training
+COMPUTE_LOG_ROOT_HOST=./data/yg-ft/logs/compute
+
+# Optional fallback used when nvidia-smi is unavailable.
+# Leave COMPUTE_GPU_COUNT=0 on real GPU servers with working NVIDIA runtime.
+COMPUTE_GPU_COUNT=0
+COMPUTE_GPU_NAME=NVIDIA A800-SXM4-80GB
+COMPUTE_GPU_MEMORY_GB=80
+COMPUTE_GPU_POWER_LIMIT_W=300
 
 LOG_DIR=/opt/yg-ft/logs/compute
 CUDA_VISIBLE_DEVICES=all

docker/compute/data/yg-ft/README.md

@@ -0,0 +1,38 @@
+# YG-FT Compute 数据目录说明
+
+本目录挂载到 `yg-ft-compute-api` 容器的 `/data/yg-ft`，用于持久化存储训练相关的数据。
+
+## 目录结构
+
+```
+data/yg-ft/
+├── datasets/     # 数据集存储目录
+├── models/       # 模型文件存储目录
+├── outputs/      # 训练/推理输出结果目录
+└── logs/
+    ├── compute/  # 计算服务运行日志
+    └── training/ # 训练任务执行日志
+```
+
+## 各目录说明
+
+### datasets/
+训练和评估所使用的数据集文件，包括 JSON、JSONL、CSV 等格式。数据集由用户上传或通过平台创建，供 LLaMA-Factory 等训练引擎读取。
+
+### models/
+存放模型文件，包括：
+- 预训练基座模型（如 LLaMA、Qwen 等）
+- 微调后的自定义模型权重
+- 合并后的部署模型
+
+### outputs/
+训练任务和推理任务的输出结果，包括：
+- 训练过程中的 checkpoint 文件
+- 评估结果和指标报告
+- 推理生成的结果文本
+
+### logs/compute/
+计算服务（compute-api）的运行时日志，用于排查服务启动、GPU 调度、健康检查等问题。
+
+### logs/training/
+各训练任务的执行日志，记录训练过程状态、报错信息等，便于追踪单个任务的运行情况。

docker/compute/data/yg-ft/datasets/.gitignore

@@ -0,0 +1,2 @@
+*
+!.gitignore

docker/compute/data/yg-ft/logs/compute/.gitignore

@@ -0,0 +1,2 @@
+*
+!.gitignore

docker/compute/data/yg-ft/logs/training/.gitignore

@@ -0,0 +1,2 @@
+*
+!.gitignore

docker/compute/data/yg-ft/models/.gitignore

@@ -0,0 +1,2 @@
+*
+!.gitignore

docker/compute/data/yg-ft/outputs/.gitignore

@@ -0,0 +1,2 @@
+*
+!.gitignore

docker/compute/docker-compose.yml

@@ -5,15 +5,27 @@ services:
     gpus: all
     ports:
       - "${COMPUTE_API_PORT:-19100}:9100"
+      - "${FILE_GATEWAY_PORT:-19101}:9100"
     environment:
       COMPUTE_ENV: ${COMPUTE_ENV:-prod}
       COMPUTE_HOST_ID: ${COMPUTE_HOST_ID:-gpu-node-01}
       COMPUTE_EXECUTION_MODE: ${COMPUTE_EXECUTION_MODE:-real}
       MODELTF_ROUTE_PREFIX: ${MODELTF_ROUTE_PREFIX:-/modelTF}
+      ENABLE_DOCS: ${ENABLE_DOCS:-false}
+      COMPUTE_AUTH_ENABLED: ${COMPUTE_AUTH_ENABLED:-true}
       COMPUTE_SERVICE_TOKEN: ${COMPUTE_SERVICE_TOKEN:-change_me}
       ENABLE_APP_CALLBACK: ${ENABLE_APP_CALLBACK:-false}
       LLAMA_FACTORY_HOME: ${LLAMA_FACTORY_HOME:-/app/LLaMA-Factory}
       YG_FT_DATA_ROOT: ${YG_FT_DATA_ROOT:-/data/yg-ft}
+      YG_FT_MODEL_ROOT: ${YG_FT_MODEL_ROOT:-/data/yg-ft/models}
+      YG_FT_DATASET_ROOT: ${YG_FT_DATASET_ROOT:-/data/yg-ft/datasets}
+      YG_FT_OUTPUT_ROOT: ${YG_FT_OUTPUT_ROOT:-/data/yg-ft/outputs}
+      TRAINING_LOG_ROOT: ${TRAINING_LOG_ROOT:-/opt/yg-ft/logs/training}
+      COMPUTE_GPU_COUNT: ${COMPUTE_GPU_COUNT:-0}
+      COMPUTE_GPU_NAME: ${COMPUTE_GPU_NAME:-NVIDIA A800-SXM4-80GB}
+      COMPUTE_GPU_MEMORY_GB: ${COMPUTE_GPU_MEMORY_GB:-80}
+      COMPUTE_GPU_POWER_LIMIT_W: ${COMPUTE_GPU_POWER_LIMIT_W:-300}
+      MIN_TRAINING_GPU_MEMORY_GB: ${MIN_TRAINING_GPU_MEMORY_GB:-4}
       LOG_DIR: ${LOG_DIR:-/opt/yg-ft/logs/compute}
       CUDA_VISIBLE_DEVICES: ${CUDA_VISIBLE_DEVICES:-all}
       NVIDIA_VISIBLE_DEVICES: ${NVIDIA_VISIBLE_DEVICES:-all}
@@ -21,9 +33,12 @@ services:
       PYTHONPATH: /app
     volumes:
       - ../../compute:/app/compute:ro
-      - ${YG_FT_DATA_ROOT_HOST:-/data/yg-ft}:${YG_FT_DATA_ROOT:-/data/yg-ft}
+      - ${YG_FT_DATA_ROOT_HOST:-./data/yg-ft}:${YG_FT_DATA_ROOT:-/data/yg-ft}
-      - ../../runtime/compute/logs:/opt/yg-ft/logs/compute
+      - ${YG_FT_MODEL_ROOT_HOST:-./data/yg-ft/models}:${YG_FT_MODEL_ROOT:-/data/yg-ft/models}
-      - ../../runtime/compute/training-logs:/opt/yg-ft/logs/training
+      - ${YG_FT_DATASET_ROOT_HOST:-./data/yg-ft/datasets}:${YG_FT_DATASET_ROOT:-/data/yg-ft/datasets}
+      - ${YG_FT_OUTPUT_ROOT_HOST:-./data/yg-ft/outputs}:${YG_FT_OUTPUT_ROOT:-/data/yg-ft/outputs}
+      - ${COMPUTE_LOG_ROOT_HOST:-./data/yg-ft/logs/compute}:${LOG_DIR:-/opt/yg-ft/logs/compute}
+      - ${TRAINING_LOG_ROOT_HOST:-./data/yg-ft/logs/training}:${TRAINING_LOG_ROOT:-/opt/yg-ft/logs/training}
     networks:
       - yg-ft-compute
     healthcheck:

docker/nginx.conf.template

@@ -7,19 +7,26 @@ server {
 
   client_max_body_size 200m;
 
-  location / {
+  location ^~ /modelTF/ {
-    try_files $uri $uri/ /index.html;
-  }
-
-  location /modelTF {
     proxy_pass ${BACKEND_PROXY_PASS};
     proxy_http_version 1.1;
     proxy_set_header Host $host;
     proxy_set_header X-Real-IP $remote_addr;
     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
     proxy_set_header X-Forwarded-Proto $scheme;
-    proxy_read_timeout 300s;
+    proxy_read_timeout 900s;
-    proxy_send_timeout 300s;
+    proxy_send_timeout 900s;
+  }
+
+  location = /modelTF {
+    proxy_pass ${BACKEND_PROXY_PASS};
+    proxy_http_version 1.1;
+    proxy_set_header Host $host;
+    proxy_set_header X-Real-IP $remote_addr;
+    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+    proxy_set_header X-Forwarded-Proto $scheme;
+    proxy_read_timeout 900s;
+    proxy_send_timeout 900s;
   }
 
   location ~* \.(?:js|css|png|jpg|jpeg|gif|ico|svg|woff|woff2|ttf)$ {
@@ -27,4 +34,8 @@ server {
     expires 30d;
     add_header Cache-Control "public, immutable";
   }
+
+  location / {
+    try_files $uri $uri/ /index.html;
+  }
 }

docs/2026-07-23-development-summary.md

@@ -0,0 +1,153 @@
+# 2026-07-23 开发总结
+
+本文档汇总 2026-07-23 当天围绕模型微调平台完成的系统开发内容。当前开发不再按临时 Demo 口径处理，以下能力均按后续可继续演进到生产系统的代码和数据结构推进。
+
+## 1. 今日完成内容
+
+### 1.1 模型训练最小闭环增强
+
+| 功能点 | 作用 | 对应页面/模块 | 用户如何操作 |
+| --- | --- | --- | --- |
+| 训练任务创建前预检 | 在任务进入运行前校验模型、数据集、算力节点、路径和 LLaMA-Factory 参数，减少任务启动后才失败的情况 | 模型微调列表、创建模型微调、训练日志 | 在“模型微调”页面创建任务并提交，系统自动执行预检；失败时页面提示具体原因 |
+| 训练数据同步到算力节点 | 将应用侧选择的数据集文件同步到目标 Compute 节点本地磁盘，保证 LLaMA-Factory 能读取到真实训练文件 | 创建模型微调、训练日志、算力节点/资源副本 | 用户选择数据集后提交训练任务，系统自动同步数据集到算力节点，无需手工进入容器复制 |
+| 动态生成训练参数 | 根据用户在页面选择的训练数据、基座模型、训练方法、输出目录等动态生成 LLaMA-Factory 训练参数 | 创建模型微调、训练日志 | 用户在创建任务表单中选择模型和数据集，启动训练后可在训练日志中查看实际执行命令 |
+| 训练日志和状态轮询 | 应用侧主动轮询 Compute API 获取训练状态、进度、日志、指标和 checkpoint | 训练日志详情页 | 用户进入训练日志页，可查看任务状态、日志输出、指标和 checkpoint 信息 |
+| 训练失败信息回传 | 训练命令失败后将失败状态和日志内容回传应用侧，便于定位数据格式、模型路径或参数问题 | 模型微调列表、训练日志详情页 | 任务失败后，用户进入训练日志页查看失败日志和错误堆栈 |
+
+### 1.2 B. 模型资产、训练与 LLaMA-Factory 任务能力
+
+| 功能点 | 作用 | 对应页面/模块 | 用户如何操作 |
+| --- | --- | --- | --- |
+| 训练模型产物 artifact 展示 | 展示训练/合并产生的 Adapter、合并模型、量化模型等产物路径、大小、checksum | 模型管理 -> 训练模型列表 | 用户进入“模型管理”，切换到“训练模型”，展开某个模型行查看“模型产物” |
+| 模型血缘 lineage 展示 | 记录和展示基座模型、训练模型、合并产物之间的来源关系 | 模型管理 -> 训练模型列表 | 用户展开训练模型行，在“模型血缘”区域查看父资源、子资源和对应 Compute Job |
+| 模型导出任务 export job 状态展示 | 展示模型合并/导出的任务状态、输出目录和创建时间 | 模型管理 -> 训练模型列表 | 用户展开训练模型行，在“导出任务”区域查看导出任务状态 |
+| export 完成后回填 checksum | Compute export 完成后扫描产物文件，回填大小和 `checksum_sha256`，为后续一致性校验和制品治理做准备 | 后端模型资产模块、模型管理页面 | 用户无需手工操作，任务完成后页面自动展示产物大小和 checksum |
+| 合并模型任务接入 Compute | 模型合并不再只是生成模拟任务，而是通过 Compute 节点调用模型合并任务并记录导出任务 | 模型管理 -> 合并权重 | 用户在训练模型列表点击“合并权重”，提交后系统创建 Compute 合并任务 |
+
+### 1.3 D. 算力平台、部署与运维能力
+
+| 功能点 | 作用 | 对应页面/模块 | 用户如何操作 |
+| --- | --- | --- | --- |
+| 算力节点动态配置 | 支持维护 Compute API、File Gateway、权重、标签、启用状态、调度状态等信息 | 算力节点 -> 节点 | 用户进入“算力节点”，点击“新增节点”或“编辑”，填写节点地址和调度参数 |
+| 节点连通性测试 | 验证应用服务器是否能访问 Compute API，并返回 GPU 发现结果和延迟 | 算力节点 -> 节点 | 用户在节点列表点击“测试”，查看连接成功或失败提示 |
+| GPU 状态展示 | 展示每个算力节点的 GPU 利用率、显存、温度、功耗、进程信息 | 算力节点 -> GPU | 用户进入“算力节点”，切换到“GPU”页签查看 |
+| 训练队列展示 | 展示 Compute 侧任务队列、状态、进度、节点和 GPU 分配 | 算力节点 -> 队列 | 用户切换到“队列”页签查看当前运行或等待任务 |
+| 资源副本列表 | 展示数据集、模型、训练产物在算力节点本地磁盘上的副本路径和同步状态 | 算力节点 -> 资源副本 | 用户切换到“资源副本”页签，选择目标节点查看本地副本 |
+| 资源副本漂移检测 | 检查副本路径是否仍存在、大小是否可读取，并将异常副本标记为漂移 | 算力节点 -> 资源副本 | 用户点击“漂移检测”，系统调用 Compute API 检查节点本地路径 |
+| 资源副本 repair 重同步 | 对漂移、失败、待修复副本执行真实重同步，而不只是标记状态 | 算力节点 -> 资源副本 | 用户点击“修复副本”，系统从权威源重新上传或导入到目标算力节点 |
+| 应用侧轮询模式 | 应用服务主动轮询 Compute API，避免需要 Compute 侧反向访问应用侧 | 后端 Compute Gateway、部署架构 | 用户无感知，部署时只需保证应用侧能访问各 Compute 节点 |
+| 本地磁盘挂载约定 | 基座模型、训练数据、训练产物、训练日志均通过宿主机目录挂载到 Compute 容器 | docker/compute 部署 | 运维人员在启动 Compute 服务前准备并挂载 `docker/compute/data/yg-ft/*` 目录 |
+
+### 1.4 前端状态中文化
+
+| 功能点 | 作用 | 对应页面/模块 | 用户如何操作 |
+| --- | --- | --- | --- |
+| 统一状态字典 | 将 `pending`、`running`、`completed`、`failed`、`online`、`synced`、`drifted` 等状态统一展示为中文 | 通用前端组件 `ModelStatusTag`、状态工具 `status.ts` | 用户在各列表页看到中文状态，不再直接看到英文状态值 |
+| 模型管理状态中文化 | 合并状态、导出任务状态、产物状态使用中文展示 | 模型管理 | 用户查看训练模型列表和展开详情时看到“未合并/合并中/已合并/已完成/失败”等中文 |
+| 算力节点状态中文化 | 节点状态、GPU 状态、队列状态、副本同步状态使用中文展示 | 算力节点 | 用户查看“节点/GPU/队列/资源副本”时看到中文状态标签 |
+| 推理/对比/评测/用户状态中文化 | 将推理任务、对比模型加载、评测任务、用户状态统一为中文标签 | 模型推理、模型对比、模型评测、用户设置 | 用户查看相关页面时看到“启动中/已就绪/已完成/启用”等中文状态 |
+
+### 1.5 前端构建产物更新
+
+| 功能点 | 作用 | 对应页面/模块 | 用户如何操作 |
+| --- | --- | --- | --- |
+| 刷新 `frontend/dist` | 保证 Docker 前端 nginx 容器可以直接加载最新页面代码 | 前端部署 | 用户重新访问前端服务即可看到最新页面 |
+| 重启前后端服务验证 | 验证源码挂载和 dist 挂载方式下服务可正常加载最新代码 | docker/app、docker/compute | 开发人员重启 `backend-api`、`frontend`、`compute-api` 后验证页面和接口 |
+
+## 2. 今日涉及的主要代码位置
+
+| 文件 | 说明 |
+| --- | --- |
+| `backend/app/api/v1/endpoints/platform.py` | 新增/完善训练预检、资源同步、模型 artifact/lineage/export job 接口、资源副本漂移检测和 repair |
+| `backend/app/db/platform_store.py` | 新增模型产物、血缘、导出任务、副本同步结果、artifact 查询等数据访问能力 |
+| `compute/agent/process_manager.py` | export/artifact 扫描时补充文件大小和 checksum |
+| `frontend/src/api/modules/model.ts` | 增加训练模型 artifact、lineage、export job API |
+| `frontend/src/api/modules/compute.ts` | 增加副本漂移检测、repair API 和副本字段 |
+| `frontend/src/views/model/ModelManageView.vue` | 增加训练模型展开详情：产物、血缘、导出任务 |
+| `frontend/src/views/compute/ComputeNodesView.vue` | 增加资源副本页签、漂移检测、repair 操作和状态中文化 |
+| `frontend/src/utils/status.ts` | 新增统一状态中文映射和标签类型映射 |
+| `frontend/src/components/ModelStatusTag.vue` | 改为复用统一状态字典 |
+| `frontend/src/components/DataTablePage.vue` | 增加表格展开事件支持 |
+| `frontend/dist/*` | 前端生产构建产物已更新 |
+
+## 3. 今日验证结果
+
+| 验证项 | 结果 |
+| --- | --- |
+| Python 编译检查 | 通过：`python -m compileall backend compute scripts` |
+| 前端生产构建 | 通过：`npm run build` |
+| 前端入口 | 通过：`http://127.0.0.1:16801/` 返回 200 |
+| 后端健康检查 | 通过：`/modelTF/health` 返回正常 |
+| Compute 健康检查 | 通过：`/modelTF/health` 返回正常 |
+| 模型导出任务接口 | 通过：`/modelTF/model-manage/export-jobs` 返回正常 |
+| 算力节点列表接口 | 通过：`/modelTF/compute/nodes` 返回正常 |
+| 资源副本漂移检测 | 通过：当前节点副本检测返回 `drifted: 0` |
+
+## 4. 当前仍需注意的问题
+
+| 问题 | 影响 | 建议 |
+| --- | --- | --- |
+| 训练数据格式仍依赖 LLaMA-Factory 约定 | 如果用户上传的数据字段不符合模板要求，训练仍会失败 | 下一步增加数据集格式校验和模板转换预检 |
+| 单机 MX350 显存较小 | 真实训练大模型时容易因显存不足失败 | 当前环境用于链路验证；真实训练应使用高显存 GPU 节点 |
+| resource replica repair 对大型模型仍是同步调用 | 大模型重同步可能耗时较长 | 下一步将 repair 完整异步化，并展示 sync job 进度 |
+| artifact checksum 目前在 Compute 扫描阶段计算 | 对超大目录递归扫描可能较慢 | 下一步支持分文件 checksum、manifest 文件和后台扫描 |
+| 模型评测、模型推理仍未形成完整生产闭环 | 目前页面已有基础能力，但后端表结构、任务运行、日志、治理能力还需补齐 | 下一步将评测和推理纳入正式开发计划 |
+
+## 5. 下一步开发计划
+
+### 5.1 B. 模型资产、训练与 LLaMA-Factory 任务
+
+| 优先级 | 开发任务 | 目标页面/模块 | 交付结果 |
+| --- | --- | --- | --- |
+| P0 | 数据集格式预检 | 创建模型微调、数据集管理 | 提交训练前校验 Alpaca/ShareGPT/OpenAI Messages 等格式，提示缺失字段 |
+| P0 | 训练参数可视化确认 | 创建模型微调 | 提交前展示最终 LLaMA-Factory 参数预览，减少参数不一致问题 |
+| P0 | 训练任务失败诊断 | 训练日志详情 | 识别常见错误：数据字段缺失、模型路径不存在、显存不足、依赖缺失，并生成中文诊断 |
+| P1 | checkpoint 管理 | 训练日志详情、模型管理 | 展示 checkpoint 列表、大小、路径、保留策略，支持标记最佳 checkpoint |
+| P1 | 模型产物 manifest | 模型管理 | 为每个训练/导出产物生成 manifest，记录文件清单、大小、checksum、来源任务 |
+| P1 | 合并/导出任务详情页 | 模型管理、训练日志 | 展示合并任务日志、状态、产物、失败原因 |
+| P2 | 模型版本治理 | 模型管理 | 支持版本号、标签、发布状态、归档状态、审批状态 |
+| P2 | 训练模板管理 | 系统设置或训练配置 | 将 LLaMA-Factory 模板、数据格式、默认超参做成可维护配置 |
+
+### 5.2 D. 算力平台、部署与运维
+
+| 优先级 | 开发任务 | 目标页面/模块 | 交付结果 |
+| --- | --- | --- | --- |
+| P0 | repair 异步化 | 算力节点 -> 资源副本 | repair 创建 sync job 后后台执行，页面展示进度和失败原因 |
+| P0 | 多算力节点调度策略 | 算力节点、创建模型微调 | 支持按标签、权重、空闲 GPU、显存要求选择节点 |
+| P1 | 节点资源水位告警 | 算力节点、硬件监控 | 展示磁盘、GPU、显存、训练日志目录水位和告警状态 |
+| P1 | 节点维护窗口 | 算力节点 | 支持维护中节点不再调度新任务，已有任务可继续或迁移 |
+| P1 | 文件副本治理 | 算力节点 -> 资源副本 | 支持副本清理、重建、过期策略和跨节点一致性检查 |
+| P2 | 部署健康巡检脚本 | 部署运维 | 一键检查 app、backend、redis、pg、compute、GPU、挂载目录、端口连通 |
+| P2 | Compute Agent 插件标准 | 算力平台 | 抽象 LLaMA-Factory 接入规范，预留其他训练框架 |
+
+### 5.3 模型评测页面开发计划
+
+| 优先级 | 开发任务 | 目标页面/模块 | 交付结果 |
+| --- | --- | --- | --- |
+| P0 | 评测任务表结构和接口补齐 | 模型评测列表、创建评测、评测详情 | 建立 `eval_tasks`、`eval_dimensions`、`eval_sample_results` 等运行表和接口 |
+| P0 | 创建评测任务真实提交 | 创建评测 | 支持选择模型、数据集、评测维度、GPU/节点，提交后生成评测任务 |
+| P0 | 评测任务运行闭环 | 评测详情 | 支持状态、进度、日志、样本级结果回传 |
+| P1 | 评测维度管理 | 评测维度创建/编辑 | 支持规则、Prompt、评分器、权重、适用数据集配置 |
+| P1 | 样本级评分展示 | 评测详情 | 展示每条样本的输入、模型输出、评分、原因、人工复核状态 |
+| P1 | 综合报告生成 | 评测详情、排行榜 | 生成维度汇总、综合分、问题样本、改进建议 |
+| P2 | 评测审批和审计 | 审批流、审计日志 | 评测任务创建、发布报告、删除报告纳入治理 |
+
+### 5.4 模型推理页面开发计划
+
+| 优先级 | 开发任务 | 目标页面/模块 | 交付结果 |
+| --- | --- | --- | --- |
+| P0 | 推理任务表结构和接口补齐 | 模型推理列表、新建推理 | 建立 `inference_tasks`、`inference_task_models`、`chat_sessions`、`chat_messages` |
+| P0 | 模型加载/卸载真实闭环 | 模型推理列表 | 支持选择训练产物加载推理服务，展示加载状态、端口、进程和错误 |
+| P0 | 单模型对话持久化 | 模型对话 | 保存会话、消息、参数、响应耗时、token 统计 |
+| P1 | 多模型对比任务 | 模型对比 | 支持多模型同时加载、同一问题并发请求、结果对比展示 |
+| P1 | 推理资源管控 | 模型推理、算力节点 | 支持 GPU 选择、并发限制、空闲自动卸载、异常进程清理 |
+| P1 | 推理日志和调用审计 | 日志、审计中心 | 记录加载、卸载、对话请求、失败原因、用户和租户信息 |
+| P2 | 推理服务发布 | 模型管理、模型推理 | 支持将某个训练模型发布为内部推理服务，并配置访问权限 |
+
+## 6. 建议的下一阶段顺序
+
+1. 先完成 B+D 的训练稳定性增强：数据格式预检、训练参数预览、失败诊断、repair 异步化。
+2. 再补齐模型评测的真实任务闭环：任务表、创建任务、运行状态、样本结果。
+3. 然后补齐模型推理闭环：加载/卸载、对话持久化、多模型对比。
+4. 最后统一治理能力：审批、审计、租户隔离、资源配额、保留策略和运维巡检。
+

docs/2026-07-24-work-plan.md

@@ -0,0 +1,281 @@
+# 2026-07-24 工作计划
+
+本文基于 `docs/2026-07-23-development-summary.md`、当前 B+D 开发进度，以及 2026-07-24 已完成的训练预检、失败诊断、资源修复异步化能力整理。后续开发仍按正式系统演进推进，不以临时演示能力作为交付标准。
+
+## 1. 当前完成基线
+
+### 1.1 训练创建与预检
+
+- 对应页面：`模型微调 / 创建训练任务`
+- 已完成能力：
+  - 创建训练前调用训练预检接口。
+  - 展示真实 LLaMA-Factory 命令预览。
+  - 展示预检节点、错误、警告和中文诊断建议。
+  - 后端预检不创建任务、不落库、不占用调度锁。
+  - 后端兼容 `base_model/train_dataset_id` 和 `model_id/dataset_id` 两套字段。
+- 对应接口：
+  - `POST /modelTF/fine-tune/preflight`
+  - `POST /modelTF/fine-tune/command-preview`
+
+### 1.2 训练日志与失败诊断
+
+- 对应页面：`系统日志 / 训练日志`
+- 已完成能力：
+  - 训练失败或停止后可查询诊断建议。
+  - 根据训练日志和失败原因识别模型路径、数据集字段、CUDA/GPU、LLaMA-Factory 命令等常见问题。
+  - 页面以中文展示失败诊断。
+- 对应接口：
+  - `GET /modelTF/fine-tune/{task_id}/diagnostics`
+
+### 1.3 算力节点与资源修复
+
+- 对应页面：`算力平台 / 算力节点`
+- 已完成能力：
+  - 算力节点可配置、可测试连接、可查看 GPU 与健康状态。
+  - 资源副本 repair 改为异步提交。
+  - 页面展示资源同步任务进度，避免长请求阻塞页面。
+- 对应接口：
+  - `POST /modelTF/compute/nodes/{node_id}/replicas/repair`
+  - `GET /modelTF/compute/sync-jobs/{sync_id}`
+
+## 2. 未完成任务清单
+
+### P0：真实训练成功闭环
+
+- 对应页面：
+  - `模型管理 / 新增模型`
+  - `数据集管理 / 上传数据集`
+  - `模型微调 / 创建训练任务`
+  - `系统日志 / 训练日志`
+- 未完成内容：
+  - 基座模型必须支持算力服务器本地路径校验，避免选择 API 模型或应用侧路径后进入训练。
+  - 数据集必须支持上传后格式校验，提前发现 Alpaca、ShareGPT、DPO、CPT 字段不匹配问题。
+  - 训练成功后需要完成模型产物扫描、产物入库、训练任务状态回填。
+  - 训练失败时需要强制拉取最后日志片段，保证失败原因可见。
+- 后端开发：
+  - 增强模型路径校验，明确区分 `本地训练模型`、`API 模型`、`已训练模型`。
+  - 增加数据集格式校验服务，支持字段级错误返回。
+  - 完善训练任务完成后的 artifact 回填逻辑。
+- Compute 开发：
+  - 训练结束后扫描输出目录。
+  - 返回产物列表、文件大小、目录结构、训练日志路径。
+- 验收标准：
+  - 使用算力节点可访问的本地模型路径和合法数据集，可以完成一次真实 LLaMA-Factory 训练。
+  - 训练成功后页面能看到完成状态、输出目录、模型产物。
+  - 训练失败时页面能看到中文诊断和最后错误日志。
+
+### P0：LLaMA-Factory 参数映射完善
+
+- 对应页面：`模型微调 / 创建训练任务`
+- 未完成内容：
+  - SFT、DPO、CPT 参数映射仍需细化。
+  - LoRA、Full、QLoRA、量化导出参数需要按训练方式校验。
+  - 不同模板与数据集格式之间的兼容关系需要预检。
+- 后端开发：
+  - 建立训练参数标准化层。
+  - 建立训练方式到 LLaMA-Factory 参数的映射表。
+  - 对无效组合返回中文错误，例如 DPO 缺少 rejected 字段、CPT 不应使用 instruction/output 格式等。
+- 验收标准：
+  - 页面选择不同训练方式时，预检能返回准确命令。
+  - 无效参数组合不能启动训练。
+
+### P1：资源副本 repair 自动重同步
+
+- 对应页面：`算力平台 / 算力节点`
+- 未完成内容：
+  - repair 当前已异步化，但还需要基于权威源路径自动重同步。
+  - 修复完成后需要重新校验副本状态。
+- 后端开发：
+  - 为模型、数据集、训练产物定义权威源路径。
+  - repair job 根据权威源自动发起重传或重新扫描。
+  - 修复完成后更新 replica 状态、checksum、错误原因。
+- Compute 开发：
+  - 支持接收重同步请求。
+  - 支持按资源类型写入目标路径并返回校验信息。
+- 验收标准：
+  - 将副本标记为异常后，点击修复可自动完成重同步并恢复为正常。
+
+### P1：artifact checksum 与 manifest
+
+- 对应页面：
+  - `模型管理 / 已训练模型`
+  - `模型管理 / 模型详情`
+  - `算力平台 / 资源副本`
+- 未完成内容：
+  - artifact checksum 目前仍是预留字段。
+  - 大目录需要 manifest 文件，避免每次递归扫描成本过高。
+- 后端开发：
+  - 增加 artifact checksum 回填逻辑。
+  - 增加 manifest 解析和存储字段。
+- Compute 开发：
+  - export 或训练完成后扫描文件并生成 checksum。
+  - 对大模型目录生成 manifest。
+- 验收标准：
+  - 模型产物列表能展示 checksum、大小、文件数、生成时间。
+  - 副本校验可以基于 checksum 判断一致性。
+
+### P1：模型导出闭环
+
+- 对应页面：
+  - `模型管理 / 已训练模型`
+  - `模型管理 / 导出任务`
+  - `模型管理 / 合并权重`
+- 未完成内容：
+  - 导出任务、量化导出、导出日志、失败重试、产物下载仍需完善。
+- 后端开发：
+  - 完善 export job 创建、查询、取消、重试接口。
+  - 导出完成后登记 artifact 和 lineage。
+- Compute 开发：
+  - 支持 LoRA 合并、GGUF/量化导出、导出日志回传。
+- 验收标准：
+  - 已训练模型可发起导出。
+  - 导出状态、日志、产物可在页面查看。
+
+### P1：多算力节点调度增强
+
+- 对应页面：`算力平台 / 算力节点`
+- 未完成内容：
+  - 当前已有节点配置和基础调度，但生产级调度策略仍需增强。
+  - 需要支持节点标签、权重、启用状态、容量、手动指定节点。
+- 后端开发：
+  - 增强调度策略：标签匹配、权重、当前任务数、GPU 占用、显存约束。
+  - 增加任务排队和等待原因。
+  - 增加 GPU 分配释放的异常恢复。
+- 前端开发：
+  - 创建训练时支持可选手动指定节点。
+  - 算力节点页展示容量、排队数、当前任务。
+- 验收标准：
+  - 多节点时可以自动选择合适节点。
+  - 节点不可用时页面能明确展示不可调度原因。
+
+### P2：训练日志实时性优化
+
+- 对应页面：`系统日志 / 训练日志`
+- 未完成内容：
+  - 当前依赖应用侧轮询，日志实时性和失败最后日志仍需增强。
+- 后端开发：
+  - 支持日志 offset/tail 增量读取。
+  - 任务失败时强制同步最后日志片段。
+  - 日志接口返回来源、偏移量、是否截断。
+- 前端开发：
+  - 日志页按 offset 增量刷新。
+  - 失败时自动跳到底部并展示最后错误。
+- 验收标准：
+  - 训练过程中日志持续刷新。
+  - 失败后无需手动刷新即可看到最后错误。
+
+### P2：权限、审计和治理落点补齐
+
+- 对应页面：
+  - `用户中心`
+  - `项目管理`
+  - `模型管理`
+  - `数据集管理`
+  - `模型微调`
+  - `算力平台`
+- 未完成内容：
+  - 训练链路中的租户、项目、用户权限校验还需要细粒度补齐。
+  - 审计事件需要覆盖训练创建、启动、停止、删除、导出、资源修复。
+- 后端开发：
+  - 接口增加项目/租户上下文校验。
+  - 增加审计事件写入。
+  - 删除和高风险操作进入审批流。
+- 验收标准：
+  - 用户只能访问授权项目内的模型、数据集和训练任务。
+  - 关键操作可以在审计日志中查询。
+
+### P2：模型评测页面真实闭环
+
+- 对应页面：`模型评测`
+- 未完成内容：
+  - 评测任务创建、运行、日志、指标、结果对比仍需接入真实后端。
+- 后端开发：
+  - 评测任务表、评测指标表、评测日志接口。
+  - 支持指定模型、数据集、评测模板和指标。
+- Compute 开发：
+  - 支持评测任务执行器。
+  - 返回指标结果和日志。
+- 验收标准：
+  - 可创建评测任务并看到运行状态、指标结果和失败原因。
+
+### P2：模型推理页面真实闭环
+
+- 对应页面：
+  - `模型推理 / 推理服务`
+  - `模型推理 / 对话测试`
+  - `模型对比`
+- 未完成内容：
+  - 推理服务启动、停止、健康检查、会话请求、资源释放仍需完善。
+- 后端开发：
+  - 推理服务实例管理接口。
+  - 对话请求代理接口。
+  - 推理日志和资源占用查询。
+- Compute 开发：
+  - 支持启动本地模型推理服务。
+  - 支持停止服务和释放 GPU。
+- 验收标准：
+  - 可从页面启动一个已训练模型的推理服务。
+  - 可进行对话测试并查看服务状态。
+
+## 3. 推荐开发顺序
+
+1. 完成真实训练成功闭环。
+2. 完成数据集格式校验和 LLaMA-Factory 参数映射。
+3. 完成 artifact、checksum、manifest 和模型导出闭环。
+4. 完成资源副本 repair 自动重同步。
+5. 完成多算力节点调度增强。
+6. 完成训练日志实时性优化。
+7. 补齐权限、审批、审计治理落点。
+8. 启动模型评测真实闭环开发。
+9. 启动模型推理真实闭环开发。
+
+## 4. 下一轮优先执行任务
+
+### 任务 1：训练模型路径治理
+
+- 页面：`模型管理 / 新增模型`、`模型微调 / 创建训练任务`
+- 内容：
+  - 新增模型时区分是否可用于训练。
+  - API 模型不能作为 LLaMA-Factory 本地训练基座。
+  - 本地模型路径必须是算力节点可访问路径。
+- 验收：
+  - 选择不可训练模型时，训练创建页预检直接给出中文错误。
+
+### 任务 2：数据集格式校验
+
+- 页面：`数据集管理 / 上传数据集`、`模型微调 / 创建训练任务`
+- 内容：
+  - 上传后扫描样本字段。
+  - 支持 Alpaca、ShareGPT、DPO、CPT 校验。
+  - 返回字段缺失、类型错误、空样本等问题。
+- 验收：
+  - `111.json` 这类数据可以明确判断是否满足当前训练模板。
+
+### 任务 3：训练完成产物入库
+
+- 页面：`模型微调 / 任务列表`、`模型管理 / 已训练模型`
+- 内容：
+  - Compute 训练成功后返回输出目录。
+  - 应用侧轮询后创建 trained model 记录。
+  - 写入 artifact、lineage、export job 初始状态。
+- 验收：
+  - 训练完成后无需手动登记，模型管理中自动出现新模型。
+
+### 任务 4：失败日志最后片段拉取
+
+- 页面：`系统日志 / 训练日志`
+- 内容：
+  - 任务失败时立即拉取最后 N 行日志。
+  - 页面展示最后错误、诊断建议和原始日志。
+- 验收：
+  - 训练失败后页面不再只看到“失败”，可以直接看到失败原因。
+
+## 5. 当前测试注意事项
+
+- 当前环境中已有算力节点可访问，但 GPU 为 2GB 显存，预检会提示显存不足，这是符合预期的生产校验结果。
+- 若要验证真实训练成功，需要提前准备：
+  - 算力节点可访问的本地基座模型目录。
+  - 合法训练数据集文件。
+  - 足够显存的 GPU。
+  - Compute 容器内可用的 LLaMA-Factory 和 `llamafactory-cli`。
+- 当前前端 `dist` 已按要求参与构建更新，后续修改前端页面后需要重新执行 `npm run build`。

docs/backend-api-design.md

@@ -271,7 +271,7 @@ page=1&page_size=20&keyword=xxx&sort=-created_at
 | POST | `/modelTF/dataset-manage` | 创建数据集 |
 | PUT | `/modelTF/dataset-manage/{id}` | 更新数据集 |
 | DELETE | `/modelTF/dataset-manage/{id}` | 删除数据集 |
-| POST | `/modelTF/dataset-manage/upload/{dataset_id}` | 上传文件，字段名 `files` |
+| POST | `/modelTF/dataset-manage/upload/{dataset_id}` | 上传文件，字段名 `files`；默认同步到启用的算力节点 `/data/yg-ft/datasets/{dataset_id}/` |
 | GET | `/modelTF/dataset-manage/download/{dataset_id}` | 打包下载数据集 |
 | GET | `/modelTF/dataset-manage/download/{dataset_id}/{file_id}` | 下载单文件 |
 
@@ -451,7 +451,9 @@ page=1&page_size=20&keyword=xxx&sort=-created_at
 | GET | `/modelTF/fine-tune/{id}` | 训练任务详情 |
 | GET | `/modelTF/fine-tune/check-name?name=xxx` | 任务名查重 |
 | POST | `/modelTF/fine-tune` | 创建训练任务记录 |
-| POST | `/modelTF/fine-tune/start` | 启动训练 |
+| POST | `/modelTF/fine-tune/{id}/command-preview` | 训练创建页/详情页命令预览，返回目标节点、Compute Job payload 和 LLaMA-Factory 命令 |
+| POST | `/modelTF/fine-tune/{id}/preflight` | 训练创建页启动前预检，校验节点、模型路径、数据集路径、引擎命令和训练参数 |
+| POST | `/modelTF/fine-tune/start` | 启动训练，应用侧选择算力节点并提交 Compute Job |
 | PUT | `/modelTF/fine-tune/{id}` | 更新任务 |
 | POST | `/modelTF/fine-tune/stop/{id}` | 停止任务 |
 | DELETE | `/modelTF/fine-tune/{id}` | 删除任务 |
@@ -494,6 +496,49 @@ page=1&page_size=20&keyword=xxx&sort=-created_at
 }
 ```
 
+训练启动前检查和命令预览：
+
+- 页面模块：`/fine-tune/create` 创建训练任务的“参数确认/启动训练”区域；`/training-log/:id` 训练详情页的“任务配置/命令查看”区域。
+- `POST /modelTF/fine-tune/{id}/command-preview`：不做远端路径强校验，只返回应用侧调度出的算力节点、标准 Compute Job payload、训练引擎命令和工作目录，供前端展示最终 LLaMA-Factory 启动命令。
+- `POST /modelTF/fine-tune/{id}/preflight`：启动前强校验，真实 `llama_factory` 会检查目标节点连通性、模型路径、数据集目录、LLaMA-Factory HOME、训练命令是否可用；`smoke` 引擎用于自动化闭环验收，会跳过模型/数据集路径检查。
+- `POST /modelTF/fine-tune/start`：内部先执行 preflight，预检失败返回 `409` 且任务保持 `pending`，预检通过后再写入 `syncing/queued/running` 运行态并提交 Compute Job。
+
+请求体可传启动覆盖参数：
+
+```json
+{
+  "requested_node_id": "node_xxx",
+  "gpus": [0],
+  "batch_size": 1,
+  "learning_rate": 0.0002,
+  "n_epochs": 1
+}
+```
+
+响应结构：
+
+```json
+{
+  "valid": true,
+  "errors": [],
+  "warnings": [],
+  "node": {
+    "id": "node_xxx",
+    "code": "gpu-node-01",
+    "scheduler_status": "online",
+    "gpu_count": 1
+  },
+  "job_payload": {},
+  "preview": {
+    "engine": "llama_factory",
+    "command": ["llamafactory-cli", "train", "..."],
+    "command_text": "llamafactory-cli train ...",
+    "work_dir": "/app/LLaMA-Factory",
+    "path_checks": []
+  }
+}
+```
+
 ### 7.2 训练日志详情页
 
 训练日志页还会联合调用：
@@ -841,7 +886,7 @@ page=1&page_size=20&keyword=xxx&sort=-created_at
 | POST | `/modelTF/compute/nodes` | 新增算力节点 |
 | GET | `/modelTF/compute/nodes/{id}` | 算力节点详情 |
 | PUT | `/modelTF/compute/nodes/{id}` | 编辑节点地址、权重、标签、路径和启用状态 |
-| POST | `/modelTF/compute/nodes/{id}/test-connection` | 测试 Compute API/File Gateway 连通性 |
+| POST | `/modelTF/compute/nodes/{id}/test-connection` | 测试 Compute API/File Gateway 连通性，并同步节点健康信息和 GPU 清单 |
 | POST | `/modelTF/compute/nodes/{id}/enable` | 启用节点 |
 | POST | `/modelTF/compute/nodes/{id}/disable` | 禁用节点，不接收新任务 |
 | POST | `/modelTF/compute/nodes/{id}/drain` | 进入维护模式，已有任务跑完后下线 |
@@ -853,6 +898,7 @@ page=1&page_size=20&keyword=xxx&sort=-created_at
 | GET | `/modelTF/compute/jobs/{id}` | 算力任务详情 |
 | POST | `/modelTF/compute/jobs/{id}/retry` | 重试任务 |
 | POST | `/modelTF/compute/jobs/{id}/priority` | 调整优先级 |
+| GET | `/modelTF/compute/jobs/{id}/logs` | 拉取算力任务训练日志，支持 tail/分页 |
 | POST | `/modelTF/internal/compute-sync/jobs/poll` | 应用平台主动轮询并同步算力任务状态 |
 | POST | `/modelTF/internal/compute-sync/resources` | 调度前同步数据集/模型到目标节点 |
 
@@ -862,6 +908,82 @@ page=1&page_size=20&keyword=xxx&sort=-created_at
 - 每个可执行训练的节点都需要部署 `Compute API`、`Compute Agent`、`File Gateway` 和宿主机挂载的 LLaMA-Factory。
 - 节点之间默认不互相访问，应用平台主动访问所有节点的 Compute API/File Gateway。
 - 调度支持 `auto` 和 `manual`：普通用户默认自动调度，管理员或高级用户可手动指定节点。
+- 节点地址、权重、标签、启用状态、最大并发和本地路径都由 `/compute` 算力节点页面维护。
+- 连接测试由应用后端发起，依次探测算力侧 `GET /modelTF/v1/compute/health` 和 `GET /modelTF/compute/resources/gpus`；返回包可为裸 JSON，也可为 `{code,message,data}` 包装结构。
+
+新增/编辑节点请求：
+
+```json
+{
+  "code": "gpu-node-01",
+  "name": "A800 Node 01",
+  "api_base_url": "http://10.10.20.31:19100",
+  "file_gateway_url": "http://10.10.20.31:19101",
+  "enabled": true,
+  "scheduler_status": "offline",
+  "scheduler_weight": 100,
+  "tags": ["A800", "80GB", "llama_factory"],
+  "max_parallel_jobs": 4,
+  "data_root": "/data/yg-ft",
+  "model_root": "/data/yg-ft/models",
+  "log_root": "/opt/yg-ft/logs/training",
+  "description": "北京机房训练节点"
+}
+```
+
+启动成功后，响应中的训练任务会包含 `compute_node_id`、`compute_job_id`、`process_id`、`status`、`progress`、`output_dir`、`log_file` 等字段。应用侧后台 worker 会按 `COMPUTE_POLL_INTERVAL_SECONDS` 定时调用目标算力节点查询 Compute Job，并回写训练任务状态。
+
+算力任务日志查询参数：
+
+| 参数 | 类型 | 必填 | 说明 |
+| --- | --- | --- | --- |
+| `tail_lines` | int | 否 | 默认 `200`，返回最后 N 行，范围 `1-5000` |
+| `offset` | int | 否 | 从第 N 行开始读取；当传入 `offset` 或 `limit` 时分页优先，忽略默认 tail 行数 |
+| `limit` | int | 否 | 分页读取行数，范围 `1-5000` |
+
+响应字段包括 `content`、`metrics`、`total_lines`、`offset`、`limit`、`has_more`、`next_offset`。前端训练详情页、训练日志页和算力队列页可以用该接口增量读取日志，避免一次性拉取大文件。
+
+任务维度实时日志接口：`GET /modelTF/fine-tune/{task_id}/logs?tail_lines=500`。该接口由应用后端按任务绑定的 `compute_node_id` 和 `compute_job_id` 转发到目标算力节点日志接口；如果训练尚未创建 Compute Job 或远端日志暂时不可达，则返回任务 `failure_reason`，用于页面展示启动失败、预检失败和远端训练失败原因。
+
+算力任务重试：
+
+```json
+{
+  "force": false,
+  "priority": "high",
+  "requested_node_id": "node_xxx",
+  "gpus": [0]
+}
+```
+
+默认只允许 `failed`、`stopped` 任务重试；如确需重新执行已完成任务，需要显式传 `force=true`。重试会清空旧的运行时字段，重新调度节点并创建新的 Compute Job。
+
+算力任务优先级：
+
+```json
+{
+  "priority": "low|normal|high|urgent"
+}
+```
+
+第一版优先级写入任务 payload，并影响 `/modelTF/compute/queue` 的展示排序；后续如接入独立队列调度器，可保持接口不变，将该字段映射到调度器优先级。
+
+连接测试响应：
+
+```json
+{
+  "node_id": "node_xxx",
+  "success": true,
+  "latency_ms": 35,
+  "gpu_count": 8,
+  "health": {
+    "status": "ok",
+    "api_version": "v1",
+    "execution_mode": "real",
+    "capabilities": ["gpu_discovery", "llama_factory", "file_gateway", "job_polling"]
+  }
+}
+```
 
 算力节点响应字段：
 
@@ -882,6 +1004,9 @@ page=1&page_size=20&keyword=xxx&sort=-created_at
   "data_root": "/data/yg-ft",
   "model_root": "/data/yg-ft/models",
   "log_root": "/opt/yg-ft/logs/compute",
+  "api_version": "v1",
+  "capabilities": ["gpu_discovery", "llama_factory"],
+  "description": "北京机房训练节点",
   "last_health_check_at": "2026-07-20T12:00:00+08:00",
   "health_detail": {
     "compute_api": "ok",
@@ -916,11 +1041,15 @@ GPU 响应字段：
 | 方法 | 路径 | 说明 |
 | --- | --- | --- |
 | POST | `/modelTF/compute/jobs` | 创建训练/评测/数据处理/推理任务 |
+| POST | `/modelTF/compute/jobs/preview` | 算力节点训练命令预览，不启动进程 |
+| POST | `/modelTF/compute/jobs/validate` | 算力节点训练启动前预检，校验参数、路径和引擎命令 |
 | GET | `/modelTF/compute/jobs/{id}` | 查询任务 |
 | POST | `/modelTF/compute/jobs/{id}/stop` | 停止任务 |
 | GET | `/modelTF/compute/jobs/{id}/logs` | 拉取日志 |
+| POST | `/modelTF/compute/files/check-paths` | 算力节点本地路径可用性检查 |
 | GET | `/modelTF/compute/resources/gpus` | 查询 GPU |
 | POST | `/modelTF/compute/files/upload` | 上传到算力本地磁盘 |
+| POST | `/modelTF/compute/files/import-local` | 从算力服务器本地路径导入到 `YG_FT_DATA_ROOT` |
 | GET | `/modelTF/compute/files/{id}/download` | 下载文件 |
 
 创建算力任务：
@@ -957,6 +1086,71 @@ GPU 响应字段：
 }
 ```
 
+当前 LLaMA-Factory 训练作业最小 payload：
+
+```json
+{
+  "id": "ft_xxx",
+  "name": "finance-sft-001",
+  "engine": "llama_factory",
+  "base_model": "/data/yg-ft/models/Qwen2.5-7B",
+  "model_name_or_path": "/data/yg-ft/models/Qwen2.5-7B",
+  "train_dataset_id": "ds_finance_train",
+  "dataset": "ygft_ds_finance_train",
+  "dataset_key": "ygft_ds_finance_train",
+  "dataset_dir": "/data/yg-ft/datasets/ds_finance_train",
+  "dataset_info": {
+    "ygft_ds_finance_train": {
+      "file_name": "train.jsonl",
+      "formatting": "alpaca",
+      "columns": {
+        "prompt": "instruction",
+        "query": "input",
+        "response": "output"
+      }
+    }
+  },
+  "output_dir": "/data/yg-ft/outputs/finance-sft-001",
+  "template": "qwen",
+  "train_method": "lora",
+  "gpus": [0],
+  "batch_size": 2,
+  "learning_rate": 0.0002,
+  "n_epochs": 3,
+  "save_steps": 50
+}
+```
+
+数据集启动规则：
+- 页面选择的是平台数据集 ID，后端提交 Compute Job 时会将其转换为 LLaMA-Factory 数据集 key。
+- 单文件数据集使用 `--dataset ygft_{dataset_id}`；多文件数据集使用 `--dataset ygft_{dataset_id}_1,ygft_{dataset_id}_2`。
+- `dataset_dir` 指向目标算力节点上的独立数据集目录 `/data/yg-ft/datasets/{dataset_id}`。
+- Compute API 在 preflight 和启动训练前根据 `dataset_info` 生成 `{dataset_dir}/dataset_info.json`，避免 LLaMA-Factory 读取全局 `/data/yg-ft/datasets/dataset_info.json` 失败。
+- `columns` 只声明训练文件实际存在的字段；`system`、`history` 等可选字段不能默认写入，否则样本缺少字段时 LLaMA-Factory 会在格式转换阶段报 `KeyError`。
+- 正式启动前，应用侧会把当前数据集文件内容同步到被调度的算力节点，确保在线编辑/版本切换后的训练文件被使用。
+- Preflight 会校验 `dataset_info.columns` 对应字段是否能在样本文件中找到，并校验 PyTorch CUDA 可用性、所选 GPU 是否存在、显存是否满足 `MIN_TRAINING_GPU_MEMORY_GB`。
+- Compute 健康检查返回 `torch_cuda`，用于区分 `nvidia-smi` 可见但 PyTorch CUDA 初始化失败的环境问题。
+
+应用侧轮询同步响应：
+
+```json
+{
+  "synced": 1,
+  "failed": [],
+  "items": [
+    {
+      "id": "ft_xxx",
+      "status": "running",
+      "progress": 35,
+      "compute_job_id": "ft_xxx",
+      "process_id": 52341,
+      "output_dir": "/data/yg-ft/outputs/finance-sft-001",
+      "log_file": "/opt/yg-ft/logs/training/ft_xxx.log"
+    }
+  ]
+}
+```
+
 手动指定节点时：
 
 ```json
@@ -1159,3 +1353,38 @@ LLaMA-Factory 引擎声明：
 | 存储管理 | `/storage` | `GET /modelTF/quotas/usage`、`GET /modelTF/files/{id}/download-url`、`GET /modelTF/retention-policies`、`PUT /modelTF/retention-policies/{id}` | 磁盘占用、下载、留存 |
 | 审计中心 | `/audit-logs`、`/login-logs`、`/download-logs` | `GET /modelTF/audit-logs`、`GET /modelTF/login-logs`、`GET /modelTF/download-logs` | 操作、登录、下载审计 |
 | 训练引擎管理 | `/training-engines` | `GET /modelTF/training-engines`、`GET /modelTF/training-engines/{id}`、`GET /modelTF/training-engines/{id}/schema`、`POST /modelTF/training-engines/{id}/health-check` | 引擎能力和健康 |
+## P1/P2/P3 runtime implementation note
+
+The current backend/compute implementation has connected the B+D training runtime features below:
+
+| Page module | API | Runtime behavior |
+| --- | --- | --- |
+| Training detail / logs `/training-log/:id` | `GET /modelTF/fine-tune/{id}/overview` | Returns task progress, parsed training metrics and real checkpoint records. |
+| Training detail / loss chart `/training-log/:id` | `GET /modelTF/fine-tune/{id}/metrics` | Reads `fine_tune_metrics`, populated from Compute log polling and log proxy access. |
+| Training detail / checkpoint list `/training-log/:id` | `GET /modelTF/fine-tune/{id}/checkpoints` | Reads `fine_tune_checkpoints`, populated from Compute scanning `output_dir/checkpoint-*`. |
+| Merge weights `/model-manage/merge` | `POST /modelTF/model-manage/merge` | Submits a real Compute job using `llamafactory-cli export`; records the job in `compute_jobs`; updates `trained_models.merging/merged/merged_path` when queried after completion. |
+| Compute ops / job detail | `GET /modelTF/compute/jobs/{job_id}` | Supports both fine-tune jobs and model merge/export jobs recorded in `compute_jobs`. |
+| Compute ops / job logs | `GET /modelTF/compute/jobs/{job_id}/logs` | Proxies logs from the assigned Compute node for training and merge/export jobs. |
+
+Operational diagnostic script:
+
+```bash
+APP_BASE_URL=http://localhost:17861 \
+COMPUTE_BASE_URL=http://localhost:19100 \
+COMPUTE_SERVICE_TOKEN=change_me \
+DATABASE_URL=postgresql+psycopg://user:password@host:5432/yg_ft \
+python scripts/ops_diagnostics.py
+```
+## P2/P3 runtime extension note
+
+This iteration extends the B+D runtime implementation with production-facing model asset governance and compute operations:
+
+| Page module | API | Description |
+| --- | --- | --- |
+| Trained model detail / artifacts | `GET /modelTF/model-manage/trained-models/{id}/artifacts` | Returns registered adapter, merged model and quantized/export artifacts from `model_artifacts`. |
+| Trained model detail / lineage | `GET /modelTF/model-manage/trained-models/{id}/lineage` | Returns upstream/downstream relations from `model_lineage`, including base model to fine-tuned model and merge/export relations. |
+| Merge/export task list | `GET /modelTF/model-manage/export-jobs?trained_model_id=xxx` | Returns model export and merge jobs from `model_export_jobs`. |
+| Compute node replicas | `GET /modelTF/compute/nodes/{id}/replicas/drift` | Checks whether model/dataset/output replicas still exist on the compute node local disk and updates replica status. |
+| Compute node replicas | `POST /modelTF/compute/nodes/{id}/replicas/repair` | Marks drifted replicas as `repair_pending` and creates a resource sync job for the operator/scheduler to process. |
+
+The scheduler now uses the `scheduler_locks` table while starting training tasks. Node selection, task state update, resource sync job creation and GPU pre-allocation are written in one database transaction to reduce multi-worker GPU contention.

docs/data-process-design.md

@@ -0,0 +1,279 @@
+# 数据处理接口与算法设计
+
+本文是 `team-development-plan.md` 板块 C 的落地契约，约束
+`/modelTF/data-process/*`、前端数据处理向导以及 PostgreSQL 数据模型。
+
+## 1. 处理闭环
+
+```text
+创建草稿任务
+  → 上传并登记源文件（格式、SHA-256、版本）
+  → 预处理（标准化、无效过滤、去重、可选脱敏）
+  → 构建可编辑预览（来源偏移与行号）
+  → 生成标准训练记录
+  → 质量评分与稳定数据集划分
+  → 人工编辑/恢复
+  → 幂等发布为数据集（保留完整来源链路）
+```
+
+任务只使用以下五种状态：
+
+```text
+pending ──start/generate──> running ──success──> completed
+   ▲                         │  ├──error───────> failed
+   │                         │  └──stop────────> stopped
+   └────────retry────────────┴────────retry─────┘
+```
+
+- `pending` 允许修改配置、增删源文件和重建预览。
+- `running` 拒绝重复启动、修改配置和删除任务。
+- `failed`、`stopped` 可重试；重试前清理上一次未完成结果。
+- `completed` 可编辑结果和发布；重复发布返回同一个数据集。
+- 非法状态转换返回 HTTP 409。
+- 每次生成分配独立 `generation_run_id`；停止或重试会使旧代次立即失效，
+  旧后台任务不能覆盖新代次的结果或状态。
+
+## 2. 接口契约
+
+所有路径由请求层统一添加 `/modelTF`，响应统一为
+`{ "code": 0, "message": "ok", "data": ... }`。
+
+### 任务与进度
+
+| 方法 | 路径 | 说明 |
+| --- | --- | --- |
+| GET | `/data-process` | 分页查询任务，支持 keyword/status/process_type |
+| POST | `/data-process` | 创建 `pending` 草稿 |
+| GET | `/data-process/{id}` | 查询任务详情，不内嵌全部结果 |
+| PUT | `/data-process/{id}` | 更新草稿配置 |
+| DELETE | `/data-process/{id}` | 软删除非运行任务 |
+| POST | `/data-process/{id}/start` | 重建预览并生成的一键编排入口 |
+| POST | `/data-process/{id}/generate` | 使用已确认预览生成结果 |
+| POST | `/data-process/{id}/stop` | 请求停止运行任务 |
+| GET | `/data-process/{id}/progress` | 查询阶段、进度与计数 |
+
+### 源文件与预览
+
+| 方法 | 路径 | 说明 |
+| --- | --- | --- |
+| POST | `/data-process/{id}/source-files` | multipart 上传，字段名 `files` |
+| DELETE | `/data-process/{id}/source-files/{file_id}` | 删除源文件及其预览 |
+| GET | `/data-process/{id}/source-files/{file_id}/content` | 按行窗口读取源文 |
+| POST | `/data-process/{id}/preview/build` | 后端预处理并重建预览 |
+| GET | `/data-process/{id}/preview` | 分页查询预览 |
+| POST | `/data-process/{id}/preview` | 手工增加预览条目 |
+| PUT | `/data-process/{id}/preview/{preview_id}` | 保存人工编辑 |
+| DELETE | `/data-process/{id}/preview/{preview_id}` | 删除预览条目 |
+
+上传批次先全部完成有界读取和解析，再在单个事务中登记；任一文件为空、超限、
+重复或格式非法时整批不落库，暂存原件也会一并清理。响应不回传整个文件，只返回
+逻辑对象引用、文件 ID、格式、原始字节数、记录数和原始 SHA-256。二进制文档必须
+由对应解析器显式处理；不支持的格式返回 415，绝不能静默替换成示例正文。
+
+原始上传字节与解析正文采用双层存储：原件默认保存在
+`backend/storage/data-process/<task_id>/<file_id>/v<version>/<安全文件名>`，数据库的
+`storage_object_id` 只保存 `local://data-process/...` 逻辑引用，不保存或返回宿主机
+绝对路径；完整解析正文继续保存在 `data_process_source_files.content`，列表摘要使用
+`content_preview`，因此 PDF、Office 等文件的预览无需反复解析原始二进制。可通过
+`DATA_PROCESS_STORAGE_DIR` 指定其他本地根目录；从 `start.sh` 启动时，该变量应在
+当前终端导出。历史 `db://data-process/...` 记录继续从数据库正文预览。
+
+单独删除源文件时先提交数据库软删除，再立即删除受控目录中的原件；若物理删除
+失败，接口仍按数据库结果返回成功并标记 `storage_cleanup_pending=true`，软删除记录
+中的逻辑引用可供运维补偿清理。任务软删除以及修改 `process_type` 导致的源文件
+软删除按留存数据处理，当前版本不自动物理清除。
+
+### 结果与发布
+
+| 方法 | 路径 | 说明 |
+| --- | --- | --- |
+| GET | `/data-process/{id}/results` | 分页查询，支持 keyword/status/split |
+| PUT | `/data-process/{id}/results/{result_id}` | 保存人工编辑并重评分 |
+| POST | `/data-process/{id}/results/{result_id}/restore` | 恢复生成时的原值 |
+| POST | `/data-process/{id}/publish` | 幂等发布为数据集 |
+
+## 3. 配置校验
+
+- `process_type`：`structured | unstructured | external`。
+- 数据集划分的 `train + validation + test` 必须等于 100，各项为 0～100。
+- `chunk_size` 为 16～32768 token；`chunk_overlap` 必须小于
+  `chunk_size`；`min_chunk_size` 不得大于 `chunk_size`。
+- `temperature` 为 0～2，`max_tokens` 为 1～32768。
+- 任务名称在未删除任务中唯一。
+- 选择 `generation_model_id` 后，启动生成时校验模型是否存在，并保存不含密钥的
+  模型版本快照。
+- 当前运行库沿用平台现有的单租户模式，不接受客户端提交 tenant/owner/operator
+  字段，避免伪造隔离上下文；接入平台可信认证上下文后再启用数据库中预留的
+  tenant/project 字段。
+
+## 4. 格式解析与标准化
+
+上传格式按处理类型约束：
+
+- 结构化数据支持 JSON、JSONL/NDJSON、CSV/TSV 和 XLSX。XLSX 能识别纵向、
+  横向合并单元格组成的多级表头，并稳定展平为 `销售.Q1` 一类字段；公式只读取
+  文件中已缓存的计算结果，不在服务端执行。
+- 非结构化数据支持 UTF-8/UTF-8 BOM 的 TXT、Markdown、JSON/JSONL，以及
+  文本型 PDF、DOCX 和 PPTX。PDF 按页抽取文本，DOCX 抽取段落与表格，PPTX
+  抽取幻灯片文本与表格，随后统一进入切片算法。
+- 旧版二进制 DOC、XLS、PPT 不直接解析，返回 415 并提示分别转换为
+  DOCX、XLSX、PPTX。
+- 扫描 PDF 没有文本层时明确提示需要 OCR；当前流程不执行 OCR。加密、损坏或
+  超出页数/工作表/行列/解压规模限制的文件整批拒绝。
+
+现代 Office 文件在交给解析库前检查 ZIP 成员路径、重复成员、加密标记、活动
+XML、单成员大小、总解压大小和压缩比，避免路径穿越、实体扩展与 ZIP bomb。
+
+结构化选项按固定顺序执行，关闭某项时不会隐式执行对应业务变换：
+
+1. `detect_structure`：展平嵌套对象；XLSX 上传解析阶段识别合并单元格和多级表头。
+2. `normalize_format`：字段名转 snake_case，执行 Unicode NFKC、换行和容器值规范化，
+   输出键顺序稳定的 canonical JSON；账号、邮编等字符串不会转成数值。
+3. `clean_invalid`：删除全空列和全空记录；存在 `id/uuid/key/code/*_id` 身份字段时，
+   删除身份字段残缺的行，但不会因备注等可选字段为空误删有效记录。
+4. `filter_anomaly`：仅对不少于 8 个样本的非身份数值字段使用 Tukey IQR 过滤离群行，
+   同时过滤明确乱码、不可打印或极端超长文本；小样本和 ID 字段不参与统计过滤。
+5. `deduplicate`：先按整行 canonical JSON 精确去重，再按非空
+   `id/uuid/key/code/*_id` 字段稳定保留首条；空关键值互不视为重复。
+6. `desensitize`：对结构化姓名字段和正文中的高置信上下文姓名、邮箱、手机号、
+   身份证号进行不可逆掩码，并分别记录命中数。
+
+非结构化“智能预处理”由六个可独立执行的底层选项组成：
+
+- `clean_invalid_content` 删除确定为空、不可读或纯重复符号的无效块。
+- `detect_document_structure` 识别 Markdown、中文章节和数字标题，切片不跨章节，
+  并在预览质量详情中保存 `heading_path`。
+- `merge_short_content` 在同一章节中合并短块，合并后不突破 `chunk_size`。
+- `filter_low_quality` 在生成前过滤乱码、不可打印、重复或极端超长内容。
+- `deduplicate_content` 先精确去重，再对足够长的内容进行保守近重复判断；数字或
+  否定含义变化时始终保留。
+- `preserve_context` 才启用相邻切片 overlap；关闭时切片不共享正文上下文，且上下文
+  永不跨文件或章节。
+
+表格、围栏代码块和连续列表保护是三个独立参数。启用时切点避开相应 Markdown
+块，关闭时允许按正常长度切分。
+
+脱敏是不可逆掩码：
+
+- 邮箱：`[EMAIL]`
+- 中国大陆手机号：`[PHONE]`
+- 18 位身份证号：`[ID_CARD]`
+- 高置信姓名：`[NAME]`
+
+源文件原文与脱敏后的预览分开保存，结果不得反向覆盖源文件。
+
+## 5. 切片算法
+
+首阶段只提供三种切片策略：
+
+- `structure` 先识别 Markdown、中文章节及编号标题，再由 LlamaIndex
+  `SentenceSplitter` 在章节内按段落和中英文句界限长；章节之间不共享 overlap。
+- `fixed` 使用 LlamaIndex `TokenTextSplitter` 按目标 token 窗口切分。
+- `custom` 使用用户给定分隔符，在找不到合适分隔点时回退到固定窗口。
+
+不提供 `semantic` 和旧 `heading` 配置；创建或更新任务时传入这些值会直接拒绝。
+LlamaIndex 只负责通用切分，原文 offset、行号、标题路径和 Markdown 保护块仍由
+项目适配层统一维护。
+
+首版使用可替换的确定性 token 估算器，中文字符、标点和英文词分别计数；
+所有偏移以 Python/JavaScript 都能稳定表达的 Unicode 文本偏移为准。
+
+算法必须满足：
+
+- 每轮游标严格前进，异常分隔符不能产生死循环。
+- overlap 是最大重叠量，尾部过短切片合并到上一片。
+- 代码块、Markdown 表格和连续列表在启用保护时不从中间切开。
+- 每个预览条目记录 `source_file_id`、字符偏移、起止行、token 数和算法版本。
+
+## 6. 生成与质量评分
+
+结构化记录优先识别以下字段：
+
+1. `instruction/input/output`
+2. `question/context/answer`
+3. `prompt/input/response`
+
+已有标准字段时只做标准化；需要语义生成时调用所选模型的 OpenAI 兼容接口，
+并固化模型 ID、模型版本、prompt、temperature、max_tokens 和 JSON mode 快照。
+模型地址可输入域名、`/v1` 基础地址或完整地址：例如输入
+`www.caoxiaozhu.com` 会规范为
+`https://www.caoxiaozhu.com/v1/chat/completions`，无需用户手工拼接路径。
+单条失败记录为 `invalid`，有限重试耗尽后继续处理下一条，避免整批丢失。
+
+每条结果总分为 0～100：
+
+```text
+总分 = 完整性 35% + 长度合理性 20% + 可读性 20%
+     + 来源相关性 15% + 非重复性 10%
+```
+
+- instruction 或 output 为空时格式硬失败并标记 `invalid`。
+- 开启短文本过滤且 output 低于 `min_output_length` 时标记过滤原因。
+- 评分详情、命中规则与过滤原因必须落库并返回前端，不只返回一个总分。
+
+## 7. 稳定划分
+
+划分不能依赖结果插入顺序。对每条记录计算：
+
+```text
+bucket = SHA256(task_id + ":" + result_id) mod 10000
+```
+
+按万分位阈值映射为 `train/validation/test`。同一任务重试、分页或进程重启后，
+同一结果仍落入相同 split。
+
+## 8. 发布与来源链路
+
+发布在一个数据库事务中完成：
+
+```text
+source_file
+  → data_process_task
+  → data_process_result
+  → dataset
+  → dataset_file + dataset_file_version
+  → dataset_record
+```
+
+只发布 `valid/modified` 且满足质量门槛的结果。输出 JSONL 先计算 checksum，
+再登记文件版本和记录。发布请求中的 split 会重新进行稳定划分。任务的
+`output_dataset_id` 是幂等键；重复调用返回已有数据集，目标数据集若已被外部
+删除则解除断链并重新发布。当前运行库只开放 `local` 存储类型，正文保存在
+当前平台的 `dataset_files.content`，不虚假宣称已上传 MinIO 或云存储。
+
+## 9. 安全边界
+
+- 文件名只保留 basename，响应不返回宿主机绝对路径。
+- 上传限制单文件、批次文件数与批次总大小，解析采用有界读取。
+- 外部数据源凭据不写日志、不进入 localStorage、不在详情接口回显。
+- 外部 PostgreSQL 只允许单条 `SELECT/WITH`、只读事务、5 秒连接超时、
+  30 秒语句超时和 50 MiB 响应上限；默认阻止回环、链路本地及私网地址。
+  可信内网部署必须显式设置 `DATA_PROCESS_ALLOW_PRIVATE_EXTERNAL_DB=true`。
+- SQL 迁移独立存放，应用启动不会隐式修改当前远程数据库。
+
+## 10. 迁移边界
+
+`backend/app/db/sql/002_data_process.sql` 只面向当前运行脚本
+`001_platform_runtime.sql` 的 TEXT/最小表模型。它会在执行前检查
+`datasets.id` 类型；若检测到 `docs/postgres-schema.sql` 的 UUID/JSONB 目标模型，
+会直接失败而不是进行一半成功、一半失败的危险迁移。目标模型后续应由独立
+Alembic 迁移和对应存储实现承接。
+
+`DataProcessStore.ensure_schema()` 仅供受控管理命令显式调用，API 路由和应用启动
+均不会自动执行该迁移。本次开发和测试没有修改任何远程数据库。
+
+在已加载 `DATABASE_URL` 的终端中可先只读检查：
+
+```bash
+cd backend
+.venv/bin/python -m app.modules.data_process.schema_cli --check
+```
+
+确认目标主机和数据库名称无误后，才显式执行：
+
+```bash
+cd backend
+.venv/bin/python -m app.modules.data_process.schema_cli --apply --yes
+```
+
+命令输出只显示主机、端口和数据库名，不显示用户名或密码。

docs/database-config.md

@@ -0,0 +1,164 @@
+# 数据库配置与初始化说明（PostgreSQL / Redis）
+
+> 记录平台的 **PostgreSQL 账号密码**、**Redis 账号密码**、**数据库地址在代码中的配置位置**，
+> 以及**切换 PG 数据集时如何执行完整初始化 SQL**。
+
+---
+
+## 1. 账号密码速查表
+
+### 1.1 PostgreSQL
+
+| 环境 | 地址 | 用户 | 密码 | 数据库 | 来源 |
+|------|------|------|------|--------|------|
+| 代码默认值 | `localhost:15432` | `yg_ft` | `change_me` | `yg_ft` | `config.py` / `session.py` 的 `DATABASE_URL` 兜底 |
+| Docker 部署 | `www.caoxiaozhu.com:5432` | `root` | `8811614287327Leo` | `yg_ft` | `docker/app/.env` 的 `DATABASE_URL` |
+| Docker 内置 Postgres（已注释） | `localhost:15432` | `root` | `8811614287327Leo` | `yg_ft` | `docker/app/docker-compose.yml` 注释掉的 postgres 服务 |
+
+> ⚠️ `change_me` 与 `8811614287327Leo` 均为默认/示例凭据，生产环境务必更换。
+
+### 1.2 Redis
+
+| 项 | 值 | 说明 |
+|----|-----|------|
+| 连接串 | `redis://:<REDIS_PASSWORD>@redis:6379/0` | 已内嵌密码；docker 网络内服务名 `redis`，端口 6379，db 0 |
+| 对外端口 | `16379`（`REDIS_PORT`） | 宿主机映射 |
+| 密码 | `docker/app/.env` 的 `REDIS_PASSWORD` | 已启用 `requirepass` 鉴权 |
+| 镜像 | `redis:7-alpine` | 已开启 AOF（`--appendonly yes`）+ `requirepass` |
+
+> **当前后端代码未使用 Redis**：`redis` 包已列入 `requirements.txt`，`REDIS_URL` 通过
+> docker-compose 注入容器，但全仓库 `backend/`、`compute/` 没有任何 `import redis` /
+> `Redis(...)` 连接代码。Redis 为后续功能预留；**即便如此仍已配置鉴权**，
+> 避免无密码实例对外暴露（纵深防御）。未来启用时按 `REDIS_URL` 连接即可。
+>
+> 健康检查通过容器环境变量 `REDISCLI_AUTH` 认证，不在进程参数中泄露密码。
+
+---
+
+## 2. 数据库地址在代码中的配置位置
+
+### 2.1 后端（backend）
+
+| 文件 | 作用 | 取值 |
+|------|------|------|
+| `backend/app/core/config.py` | **唯一权威配置**，`Settings.database_url` | `os.getenv("DATABASE_URL", "postgresql+psycopg://yg_ft:change_me@localhost:15432/yg_ft")` |
+| `backend/app/db/session.py` | SQLAlchemy 引擎（`get_db` / `session_scope`） | `os.getenv("DATABASE_URL", ...)` 同样兜底 |
+| `backend/app/db/platform_store.py` | **平台主存储**，直接用 psycopg 连接池 | 取 `settings.database_url`，`_psycopg_url()` 把 `postgresql+psycopg://` 转成 `postgresql://` |
+| `backend/app/modules/data_process/store.py` | 数据处理存储 | `get_settings().database_url` |
+
+> **环境变量加载顺序**：`config.py` 导入时会 `load_dotenv(backend/.env, override=True)`，
+> 即 **`backend/.env` 会覆盖系统环境变量**；docker 部署则直接由 compose 注入 `DATABASE_URL`。
+> 最终优先级：`backend/.env` / compose 注入的环境变量 > 代码内默认值。
+
+### 2.2 部署配置（docker）
+
+| 文件 | 关键项 |
+|------|--------|
+| `docker/app/.env` | `DATABASE_URL`、`POSTGRES_USER`、`POSTGRES_PASSWORD`、`REDIS_URL`、`REDIS_PORT` |
+| `docker/app/docker-compose.yml` | `backend-api` 环境透传上述变量；`redis` 服务定义 |
+
+```ini
+# docker/app/.env（节选）
+DATABASE_URL=postgresql+psycopg://root:8811614287327Leo@www.caoxiaozhu.com:5432/yg_ft
+POSTGRES_USER=root
+POSTGRES_PASSWORD=8811614287327Leo
+REDIS_PASSWORD=<强密码>                    # Redis requirepass（新增鉴权）
+REDIS_URL=redis://:<REDIS_PASSWORD>@redis:6379/0   # 连接串内嵌密码
+REDIS_PORT=16379
+USE_BUILTIN_POSTGRES=false   # 当前用共享外部库，内置 postgres 服务被注释
+USE_BUILTIN_REDIS=true       # Redis 用内置服务
+```
+
+---
+
+## 3. 完整初始化 SQL
+
+### 3.1 脚本位置
+
+| 脚本 | 用途 |
+|------|------|
+| **`backend/app/db/sql/000_full_init.sql`** | **一键初始化脚本（新增）**：建库表 + 索引 + 种子数据，幂等，覆盖全部 39 张运行表 |
+| `backend/app/db/sql/001_platform_runtime.sql` | 平台核心表（应用启动自动执行） |
+| `backend/app/db/sql/002_governance.sql` | 治理表（应用启动自动执行） |
+| `backend/app/db/sql/003_tenant_quota.sql` | 租户配额列（应用启动自动执行） |
+| `backend/app/db/sql/003_model_path_governance.sql` | 模型可训练列（**应用不自动执行**，已并入完整脚本） |
+| `backend/app/db/sql/002_data_process.sql` | 数据处理表（**应用不自动执行**，已并入完整脚本） |
+| `docs/postgres-schema.sql` | ⚠️ **目标设计稿**（UUID/JSONB），与运行时代码不兼容，**不要用于初始化** |
+
+> **重要**：`docs/postgres-schema.sql` 是规划中的“目标 schema”（UUID 主键、`ft_platform` schema 等），
+> 运行时代码明确拒绝该结构（`002_data_process.sql` 检测到 `datasets.id` 非 TEXT 会直接报错）。
+> 初始化请使用 **`000_full_init.sql`**。
+
+### 3.2 执行步骤（全新 PG 环境）
+
+**第 1 步：创建角色与数据库**（必须单独执行，不能放进事务）
+
+```sql
+-- 以超级用户（如 postgres）连接：
+CREATE ROLE  yg_ft LOGIN PASSWORD '请改为强密码';
+CREATE DATABASE yg_ft OWNER yg_ft;
+-- 如需应用执行 CREATE EXTENSION 等，可再授予超级用户（按需）：
+-- ALTER ROLE yg_ft SUPERUSER;
+```
+
+**第 2 步：执行完整初始化脚本**
+
+```bash
+psql "postgresql://yg_ft:密码@<host>:5432/yg_ft" \
+     -f backend/app/db/sql/000_full_init.sql
+```
+
+脚本特点：
+- 全程一个事务（`BEGIN; ... COMMIT;`），失败自动回滚
+- 所有 DDL 使用 `IF NOT EXISTS` / `ADD COLUMN IF NOT EXISTS`，**可重复执行**
+- 含 `DO $$...$$` 语句块，必须用 `psql` 执行（应用内部的按分号切分 `executescript()` 不适用）
+- 自动写入种子用户：`admin / admin123`、`operator / operator123`（登录后请改密）
+
+**第 3 步：校验**
+
+```sql
+SELECT count(*) FROM pg_tables WHERE schemaname = 'public';          -- 应 ≥ 39
+SELECT username, role, status FROM users;                            -- 应有 admin / operator
+```
+
+### 3.3 执行方式对比（三种途径）
+
+| 方式 | 覆盖范围 | 命令 |
+|------|----------|------|
+| **A. 完整脚本（推荐，切换新库）** | 全部 39 表 + 索引 + 种子 | `psql ... -f 000_full_init.sql` |
+| B. 应用自动初始化 | 001 + 002_governance + 003_tenant_quota + 种子用户；**不含**数据处理表、`models.can_train`、`data_convert_tasks` | 应用首次调用 `get_platform_store()` 时 `ensure_schema()` 自动执行 |
+| C. 数据处理表单独安装 | `002_data_process.sql` 全部内容 | 在 `backend/` 目录下：`python -m app.modules.data_process.schema_cli --apply --yes`（或 `--check` 只读检查） |
+
+> **缺口说明**：
+> - `models.can_train`（训练预检用）只在 `003_model_path_governance.sql` 中创建，应用启动**不会**自动执行；
+> - `data_convert_tasks`（数据转换任务表）运行时代码引用但**原 SQL 脚本缺失**；
+>   已统一并入 `000_full_init.sql` 补齐。若现有库缺这两项，执行一次完整脚本即可幂等补上。
+
+### 3.4 完整脚本包含的表（39 张）
+
+**核心**：users、models、trained_models、model_lineage、model_artifacts、model_export_jobs、
+datasets、dataset_files、compute_nodes、gpus、fine_tune_tasks、fine_tune_metrics、
+fine_tune_checkpoints、compute_jobs、gpu_allocations、scheduler_locks、resource_replicas、
+resource_sync_jobs、eval_tasks、eval_dimensions、compare_tasks、projects、project_members、
+roles、sessions、acls
+
+**治理**：tenants、approval_templates、approval_instances、approval_steps、audit_logs、retention_policies
+
+**数据处理**：data_process_tasks、data_process_source_files、data_process_preview_items、
+data_process_results、dataset_file_versions、dataset_records
+
+**数据转换**：data_convert_tasks（新增补齐）
+
+---
+
+## 4. 安全注意事项
+
+1. **更换默认密码**：`change_me`（代码兜底）、`8811614287327Leo`（部署）、`admin123`/`operator123`（种子用户）、`REDIS_PASSWORD` 上线前必须更换。
+2. **Redis 已加鉴权**：已配置 `requirepass` + `REDISCLI_AUTH` 健康检查；`REDIS_URL` 内嵌密码。若端口需暴露公网，仍建议用防火墙/安全组限制来源。
+3. **`docker/*/.env` 已入库，含明文凭据**：
+   - `backend/.env` 已被 `.gitignore` 排除；
+   - 但 `docker/app/.env`、`docker/compute/.env` 目前被 git 跟踪（`git ls-files` 可见），
+     其中的 `DATABASE_URL`、`POSTGRES_PASSWORD`、`COMPUTE_SERVICE_TOKEN` 等均为明文。
+   - **建议**：轮换这些凭据，将 `docker/*/.env` 移出版本库（`git rm --cached`）并改用
+     部署侧机密注入（如 docker secrets / CI 变量 / 环境变量模板），保留 `.env.example` 作为模板。
+4. **最小权限**：应用角色只需对业务库的 DML/DDL 权限，尽量避免 SUPERUSER。

docs/deployment-plan.md

@@ -228,6 +228,20 @@ GPU 算力服务器部署：
 
 多节点任务调度由应用平台统一完成。应用平台从 `compute_nodes` 读取节点地址、权重、标签、启用状态、维护状态和健康检查结果；从 `resource_replicas` 判断目标节点是否已有所需数据集/模型副本；缺失时创建 `resource_sync_jobs`，通过目标节点 File Gateway 同步资源。
 
+当前实现已支持在 `/compute` 算力节点页面新增和编辑节点。运维人员维护 `Compute API` 地址、`File Gateway` 地址、权重、标签、启用状态、最大并发和本地路径后，点击连接测试会由应用后端主动访问目标节点健康检查和 GPU 清单接口，并将 `health_detail`、`gpu_count`、`gpu_devices/gpus` 同步到 PostgreSQL。真实 GPU 服务器优先通过 `nvidia-smi` 发现 GPU；特殊环境可用 `COMPUTE_GPU_COUNT` 等环境变量声明兼容清单。
+
+训练运行闭环：
+
+- 前端启动训练后，Backend API 按 `compute_nodes` 的启用状态、调度状态、权重和并行任务数选择节点。
+- Backend API 向目标节点 `POST /modelTF/compute/jobs` 提交 LLaMA-Factory 训练作业，并在 `fine_tune_tasks.compute_job_id` 记录算力任务 ID。
+- Compute API 在真实模式下启动 `llamafactory-cli train` 子进程，训练日志写入 `TRAINING_LOG_ROOT/{job_id}.log`。
+- Backend API 启动后会运行应用侧轮询 worker，按 `COMPUTE_POLL_INTERVAL_SECONDS` 主动查询目标节点 `GET /modelTF/compute/jobs/{id}`，同步任务状态、进度、PID、输出目录、日志路径和产物索引。
+- 停止训练时，Backend API 优先调用目标节点 `POST /modelTF/compute/jobs/{id}/stop`，再回写应用任务状态。
+- 失败或停止任务可以通过 `POST /modelTF/compute/jobs/{id}/retry` 重试；重试会清空旧运行态，重新调度节点并创建 Compute Job。
+- 训练日志通过 `GET /modelTF/compute/jobs/{id}/logs` 读取，支持 `tail_lines`、`offset`、`limit`，用于训练详情页、训练日志页和日志平台采集。
+- Compute API 使用 `COMPUTE_SERVICE_TOKEN` 做服务间鉴权，应用侧请求携带 `X-Compute-Token`；健康检查接口保持可公开探活。
+- Compute API 会把本机训练作业登记到 `TRAINING_LOG_ROOT/compute-jobs.json`，服务重启后可恢复任务索引并继续暴露状态和日志。
+
 调度策略：
 
 - 默认自动调度，按节点健康、标签、GPU 空闲、队列长度、节点权重和资源副本命中率排序。
@@ -279,7 +293,7 @@ COMPUTE_API_BASE_URL=https://compute.internal:19100
 COMPUTE_SERVICE_TOKEN=***
 FILE_GATEWAY_BASE_URL=https://compute.internal:19101
 COMPUTE_STATUS_SYNC_MODE=polling
-COMPUTE_POLL_INTERVAL_SECONDS=10
+COMPUTE_POLL_INTERVAL_SECONDS=3
 COMPUTE_POLL_BATCH_SIZE=100
 ```
 
@@ -290,10 +304,21 @@ COMPUTE_ENV=prod
 COMPUTE_HOST_ID=gpu-node-01
 COMPUTE_API_PORT=19100
 FILE_GATEWAY_PORT=19101
+COMPUTE_AUTH_ENABLED=true
 COMPUTE_SERVICE_TOKEN=***
 ENABLE_APP_CALLBACK=false
 LLAMA_FACTORY_HOME=/app/LLaMA-Factory
 YG_FT_DATA_ROOT=/data/yg-ft
+YG_FT_DATA_ROOT_HOST=./data/yg-ft
+YG_FT_MODEL_ROOT=/data/yg-ft/models
+YG_FT_MODEL_ROOT_HOST=./data/yg-ft/models
+YG_FT_DATASET_ROOT=/data/yg-ft/datasets
+YG_FT_DATASET_ROOT_HOST=./data/yg-ft/datasets
+YG_FT_OUTPUT_ROOT=/data/yg-ft/outputs
+YG_FT_OUTPUT_ROOT_HOST=./data/yg-ft/outputs
+TRAINING_LOG_ROOT=/opt/yg-ft/logs/training
+TRAINING_LOG_ROOT_HOST=./data/yg-ft/logs/training
+COMPUTE_LOG_ROOT_HOST=./data/yg-ft/logs/compute
 LOG_DIR=/opt/yg-ft/logs/compute
 CUDA_VISIBLE_DEVICES=0,1,2,3
 ```
@@ -393,7 +418,7 @@ gpu-node-03 -> http://10.10.20.33:19100 / http://10.10.20.33:19101
 ```env
 ENABLE_APP_CALLBACK=false
 COMPUTE_SERVICE_TOKEN=change_me
-YG_FT_DATA_ROOT_HOST=/data/yg-ft
+YG_FT_DATA_ROOT_HOST=./data/yg-ft
 ```
 
 ## 12. 仍需确认的问题

docs/governance-user-guide.md

@@ -0,0 +1,411 @@
+# 平台治理功能使用指南
+
+> 版本：v1.0
+> 日期：2026-08-10
+> 适用版本：YG Fine-Tune Platform v1.0+
+
+---
+
+## 目录
+
+1. [快速入门](#1-快速入门)
+2. [用户与权限管理](#2-用户与权限管理)
+3. [GPU 算力分配](#3-gpu-算力分配)
+4. [资源可见性与隔离](#4-资源可见性与隔离)
+5. [审批流程管理](#5-审批流程管理)
+6. [审计日志查询](#6-审计日志查询)
+7. [常见问题与排查](#7-常见问题与排查)
+
+---
+
+## 1. 快速入门
+
+### 1.1 平台治理是什么？
+
+平台治理是一套**多租户、多角色、细粒度权限控制**体系，用于在多人协作使用 AI 微调平台时，确保：
+
+- 每个用户只能看到和操作自己有权限的资源
+- GPU 算力按需分配，避免资源争抢
+- 高风险操作（删除、停止任务）有审批记录可追溯
+- 所有操作都有审计日志
+
+### 1.2 三种内置角色
+
+| 角色 | 能做什么 | 不能做什么 |
+|---|---|---|
+| **超级管理员 (admin)** | 全部操作；管理用户、分配 GPU、审批、查看全部资源 | — |
+| **操作员 (operator)** | 创建数据集/模型、训练/评测/推理任务 | 管理用户、分配 GPU、修改他人权限 |
+| **观察员 (viewer)** | 查看被授权的资源 | 创建或修改任何资源 |
+
+### 1.3 入口在哪里？
+
+所有治理功能集中在左侧导航栏的 **「系统设置」** 分组下：
+
+```
+系统设置
+├── 用户设置        ← 用户 CRUD + 角色权限 + 密码管理
+├── 租户管理        ← 组织/团队（可选）
+├── 项目空间        ← 项目级资源隔离（可选）
+├── 审批模板        ← 定义哪些操作需要审批
+├── 审批中心        ← 处理待审批请求
+└── 审计日志        ← 查看所有操作记录
+```
+
+> ⚠️ 以上菜单**只有 admin 用户能看到**。普通用户登录后不会出现这些入口。
+
+---
+
+## 2. 用户与权限管理
+
+### 2.1 创建用户
+
+**路径**：`用户设置` → `创建用户`
+
+1. 以 admin 身份登录平台
+2. 进入「用户设置」页面
+3. 点击右上角「创建用户」按钮
+4. 填写信息：
+   - **账号**：登录用户名（如 `zhangsan`）
+   - **显示名称**：如 `张三`
+   - **密码**：初始密码（默认 `Platform@123`）
+   - **角色**：选择 `admin` / `operator` / `viewer`
+5. 点击保存
+
+创建后用户可以立即用该账号登录。
+
+### 2.2 管理用户权限
+
+**路径**：`用户设置` → 用户列表 → 操作列「页面权限」
+
+#### 给普通用户分配业务模块权限
+
+点击某用户的「页面权限」按钮，弹出对话框：
+
+```
+为 张三 分配可访问的页面模块：
+
+☑ 服务看板      ☑ 模型训练      ☑ 模型评测
+☑ 模型推理      ☑ 模型管理      ☑ 数据集管理
+☐ 数据处理      ☐ 数据类型转换  ☐ 算力节点    ← 不勾选则不可见
+☑ 平台性能      ☑ 查看日志
+
+[取消]  [保存]
+```
+
+勾选需要的模块，点「保存」即可。
+
+> **注意**：
+> - 「用户与权限」这个选项**只有 admin 能看到**，其他用户即使被赋权也不会显示
+> - admin 用户的权限**不可更改**，始终是全选状态且只读
+
+#### 限制说明
+
+| 权限码 | 说明 | 谁能拥有 |
+|---|---|---|
+| `user-settings` | 用户设置、租户管理、项目空间、审批、审计日志 | **仅 admin** |
+| `compute` | 算力节点、GPU 分配 | **仅 admin** |
+| 其他业务权限 | 训练、评测、推理、模型、数据集等 | admin 可分配给任何人 |
+
+### 2.3 重置用户密码
+
+**两种方式**：
+
+**方式一：管理员重置**
+1. 在用户列表中找到目标用户
+2. 点击「重置密码」
+3. 输入新密码，确认
+
+**方式二：用户自行修改**
+1. 用户登录后在「用户设置」页面点击「修改密码」按钮
+2. 输入旧密码 + 新密码（至少 6 位）
+3. 确认修改
+
+### 2.4 删除用户
+
+**路径**：`用户设置` → 用户列表 → 操作列「删除」
+
+> ⚠️ 删除用户时会**级联清理**其所有关联数据：
+> - 该用户创建的数据集、基座模型、微调产物、评测任务
+> - 该用户的 ACL 授权记录、GPU 分配记录
+> - 该用户的审批实例、审计日志、项目成员关系
+> - **训练任务保留不删**（避免算力节点上的物理任务数据不一致）
+
+---
+
+## 3. GPU 算力分配
+
+### 3.1 为什么需要 GPU 分配？
+
+当服务器有多张 GPU 卡（如 8×A800）时，需要指定**哪个用户能用哪张卡**：
+
+- 避免两个人同时选同一张卡导致训练冲突
+- 按团队/项目隔离算力资源
+- 控制每个用户的 GPU 配额
+
+### 3.2 分配 GPU（仅 admin）
+
+**路径**：`算力节点` → `GPU 分配` 标签页
+
+1. 以 admin 登录，进入「算力节点」页面
+2. 点击顶部的 **「GPU 分配」** 标签（只有 admin 可见）
+3. 点击 **「分配 GPU」** 按钮
+4. 填写：
+   - **算力节点**：选择节点（如 `gpu-node-01`）
+   - **GPU 序号**：卡号（0, 1, 2, ... 7）
+   - **用户**：选择要分配给谁
+5. 点「确认分配」
+
+示例：把节点 `gpu-node-01` 的第 0、1 号卡分配给用户 `zhangsan`：
+
+```
+┌─────────────────────────────────────┐
+│  分配 GPU                    [×]  │
+├─────────────────────────────────────┤
+│  算力节点:  [gpu-node-01 ▼]         │
+│  GPU 序号:  [0          ▲]         │
+│  用户:      [zhangsan    ▼]         │
+│                                     │
+│       [取消]     [确认分配]         │
+└─────────────────────────────────────┘
+```
+
+分配后的效果：
+
+| 用户 | 可用 GPU |
+|---|---|
+| admin | 全部 GPU（不需要显式分配） |
+| zhangsan | gpu-node-01 的 0、1 号卡 |
+| lisi | （未分配，不可用） |
+
+### 3.3 撤销分配
+
+在 GPU 分配列表中，每条记录右侧有「撤销」按钮，点击后确认即可移除该分配。
+
+### 3.4 用户视角：创建训练任务时的 GPU 选择
+
+- **admin**：下拉列表显示全部可用 GPU
+- **被分配了 GPU 的用户**：只显示被分配给自己的卡
+- **未分配任何 GPU 的用户**：显示提示「未分配 GPU，请联系管理员」，无法提交训练任务
+
+---
+
+## 4. 资源可见性与隔离
+
+### 4.1 自动生效的隔离规则
+
+无需手动配置，以下规则自动生效：
+
+| 资源类型 | admin 看到 | 普通用户看到 |
+|---|---|---|
+| **基座模型**（容器内注册的本地模型） | 全部 | **全部**（共享资源，有 model-manage 权限即可见） |
+| **数据集** | 全部 | **自己创建的** + 被 ACL 授权的 |
+| **微调产物**（训练输出的模型） | 全部 | **自己训练的** + 被 ACL 授权的 |
+| **评测任务** | 全部 | **自己创建的** + 被 ACL 授权的 |
+| **推理/对比任务** | 全部 | **自己创建的** + 被 ACL 授权的 |
+
+### 4.2 实际场景示例
+
+假设有三个用户：**admin**、**zhangsan**（算法工程师）、**lisi**（标注员）
+
+```
+zhangsan 上传了数据集 ds_alpaca、ds_sharegpt
+zhangsan 训练出了模型 ft_qwen_001
+lisi 上传了数据集 ds_label
+admin 注册了基座模型 Qwen3-1.7B
+```
+
+各用户看到的资源：
+
+| 用户 | 数据集 | 基座模型 | 微调产物 |
+|---|---|---|---|
+| **admin** | ds_alpaca, ds_sharegpt, ds_label (3个) | Qwen3-1.7B | ft_qwen_001 |
+| **zhangsan** | ds_alpaca, ds_sharegpt (2个) | Qwen3-1.7B | ft_qwen_001 |
+| **lisi** | ds_label (1个) | Qwen3-1.7B | (无) |
+
+### 4.3 ACL 资源授权（高级用法）
+
+如果 zhangsan 想让 lisi 也能看到自己的数据集 `ds_alpaca`：
+
+> 此功能需要在资源详情页提供「资源授权」按钮（前端已预留接口），当前可通过 API 直接操作：
+
+```bash
+# 授予 lisi 对 ds_alpaca 的读权限
+curl -X PUT /modelTF/resources/dataset/ds_alpaca_id/acl \
+  -H "Authorization: Bearer platform-token-admin" \
+  -d '{
+    "acls": [
+      {"principal_type": "user", "principal_id": "lisi_id", "permission": "read"}
+    ]
+  }'
+```
+
+---
+
+## 5. 审批流程管理
+
+### 5.1 哪些操作会触发审批？
+
+| 操作 | 触发条件 | 处理方式 |
+|---|---|---|
+| 删除他人的数据集 | 非 admin 删除别人创建的数据集 | 创建审批实例 或 admin 直接执行 |
+| 删除他人的模型 | 非 admin 删除别人创建的模型 | 同上 |
+| 停止他人的训练任务 | 非 admin 停止别人发起的任务 | 同上 |
+| 归档/删除项目空间 | 存在待审批变更时 | 拒绝执行 |
+
+**核心规则**：admin 做任何操作都直接执行（旁路）；普通用户操作他人资源时进入审批流程。
+
+### 5.2 审批流程示意
+
+```
+普通用户 lisi 尝试删除 zhangsan 的数据集
+                │
+                ▼
+    ┌───────────────────────┐
+    │ 后端检查：是 admin 吗？ │
+    └──────┬────────────────┘
+           │ 否
+           ▼
+    ┌───────────────────────┐
+    │ 创建审批实例            │
+    │ status = pending       │
+    │ 返回 202（待审批）     │
+    └──────────┬────────────┘
+               │
+               ▼
+    ┌───────────────────────┐
+    │ admin 在「审批中心」看到 │
+    │ 这条待审批请求           │
+    │ 点击「通过」或「拒绝」   │
+    └──────────┬────────────┘
+           │
+     ┌─────┴─────┐
+     │           │
+  通过        拒绝
+     │           │
+     ▼           ▼
+ 执行删除    不执行
++审计日志   +审计日志
+```
+
+### 5.3 管理审批
+
+**路径**：`系统设置` → `审批中心`
+
+1. 查看待审批列表（status=pending）
+2. 点击某条记录查看详情
+3. 决策：「通过」或「拒绝」
+4. 决策结果自动执行对应操作并记录审计日志
+
+**审批模板**（`系统设置` → `审批模板`）：定义每种操作需要几步审批、每步谁来审。默认模板都是单步（admin 审批即可）。
+
+---
+
+## 6. 审计日志查询
+
+### 6.1 什么是审计日志？
+
+平台上所有**写操作**和**敏感操作**都会自动记录审计日志，包括：
+
+- 用户创建/删除/修改
+- 权限变更
+- GPU 分配/撤销
+- 资源上传/删除
+- 训练任务启动/停止
+- 审批决策
+
+### 6.2 查询审计日志
+
+**路径**：`系统设置` → `审计日志`
+
+支持筛选条件：
+
+| 筛选项 | 说明 |
+|---|---|
+| 操作人 | 按用户 ID 过滤 |
+| 动作类型 | 如 `user.create`, `dataset.delete`, `gpu.assign` 等 |
+| 目标资源类型 | dataset / model / fine_tune_task 等 |
+| 时间范围 | 开始时间 ~ 结束时间 |
+
+### 6.3 导出审计日志
+
+审计日志页面底部有「导出 CSV」按钮，导出的文件包含当前筛选条件下的全部记录，可用于合规审计或问题追溯。
+
+### 6.4 日志保留策略
+
+审计日志受**留存策略**控制（`系统设置` → 租户管理 → 绑定留存策略）。默认保留 30 天，超期自动清理。
+
+---
+
+## 7. 常见问题与排查
+
+### Q1: 普通用户看不到某个菜单？
+
+检查两件事：
+1. 该用户是否有对应的**权限码**（admin 在「用户设置」→「页面权限」中分配）
+2. 该菜单是否属于 **admin 专属**（如「用户设置」「算力节点」——这些对非 admin 永远不可见）
+
+### Q2: 用户创建训练任务时报错"无权使用所选 GPU"
+
+说明该用户没有被分配所选择的 GPU 卡。解决方法：
+1. admin 进入「算力节点」→「GPU 分配」标签页
+2. 为该用户分配对应的 GPU
+3. 用户刷新页面重新选择 GPU
+
+### Q3: 删除用户后看板还显示残留数据？
+
+正常情况下 `delete_user` 会级联清理关联数据。如果仍有残留：
+- **训练任务**：设计上保留不删（避免算力节点物理数据不一致），这是预期行为
+- **登录时长排行**：可能来自旧的 session 记录，不影响功能，新登录后会更新
+
+### Q4: 审批实例一直 pending 没人处理？
+
+审批实例需要 admin 在「审批中心」手动处理。如果长时间无人处理：
+- 可以在数据库中直接将 `approval_instances.status` 改为 `rejected`
+- 或者由 admin 直接以自身身份执行该操作（admin 有旁路权限）
+
+### Q5: 如何查看当前所有 GPU 分配情况？
+
+```bash
+# admin 调用接口
+curl -H "Authorization: Bearer platform-token-admin" \
+  /modelTF/compute/gpu-assignments
+```
+
+返回格式：
+```json
+{
+  "data": [
+    {
+      "id": "ga_xxx",
+      "node_name": "A800 训练节点",
+      "gpu_index": 0,
+      "display_name": "张三",
+      "assigned_at": "2026-08-10T10:00:00"
+    }
+  ]
+}
+```
+
+### Q6: 用户忘记密码怎么办？
+
+两种方案：
+1. **admin 重置**：在「用户设置」→ 用户列表 →「重置密码」
+2. **用户自助修改**：用户登录后点击「修改密码」（需知道旧密码）
+
+如果是完全忘记且不是 admin，只能由 admin 重置。
+
+---
+
+## 附录：API 快速参考
+
+| 功能 | 方法 | 路径 | 鉴权 |
+|---|---|---|---|
+| 查看我的 GPU | GET | `/compute/my-gpus` | 登录用户 |
+| 查看 GPU 分配 | GET | `/compute/gpu-assignments` | admin |
+| 分配 GPU | POST | `/compute/gpu-assignments` | admin |
+| 撤销 GPU 分配 | DELETE | `/compute/gpu-assignments/{id}` | admin |
+| 修改自己的密码 | POST | `/users/me/password` | 登录用户 |
+| 查看审计日志 | GET | `/system/audit-logs` | admin |
+| 导出审计日志 | GET | `/system/audit-logs/export` | admin |
+| 查看审批列表 | GET | `/approvals` | 登录用户 |
+| 审批决策 | POST | `/approvals/:id/steps/:idx/decision` | 审批人 |

docs/permissions-design.md

@@ -0,0 +1,805 @@
+# 平台权限设计文档
+
+> 版本：v1.0
+> 日期：2026-08-02
+> 状态：设计基线，供后端实现和前端联调参照
+
+---
+
+## 目录
+
+1. [设计目标](#1-设计目标)
+2. [整体架构](#2-整体架构)
+3. [角色体系](#3-角色体系)
+4. [页面权限码](#4-页面权限码)
+5. [资源所有权与可见性](#5-资源所有权与可见性)
+6. [资源级 ACL（访问控制列表）](#6-资源级-acl访问控制列表)
+7. [GPU 算力分配与隔离](#7-gpu-算力分配与隔离)
+8. [审批拦截机制](#8-审批拦截机制)
+9. [审计日志](#9-审计日志)
+10. [接口鉴权流程](#10-接口鉴权流程)
+11. [数据库表结构](#11-数据库表结构)
+12. [API 接口清单](#12-api-接口清单)
+13. [前端权限控制](#13-前端权限控制)
+14. [安全设计补充](#14-安全设计补充)
+15. [实施计划](#15-实施计划)
+
+---
+
+## 1. 设计目标
+
+| 目标 | 说明 |
+|---|---|
+| **数据隔离** | 用户自己创建的数据集、模型、训练任务默认只有自己可见可操作；管理员可见全部 |
+| **权限分层** | 页面级（菜单/路由可见性）+ 资源级（单条数据的读/写/删）两层控制 |
+| **GPU 管控** | 多卡服务器上，管理员可指定哪些用户能使用哪些 GPU 卡 |
+| **审批拦截** | 删除他人资源、停止他人任务、发布模型等高风险操作需审批或管理员旁路 |
+| **审计可追溯** | 所有写操作和敏感操作产生审计日志，可按用户、动作、资源、时间筛选 |
+| **权限最小变更** | 只有管理员可修改用户角色和权限码；普通用户无法提权 |
+
+---
+
+## 2. 整体架构
+
+```
+┌──────────────────────────────────────────────────────┐
+│                     前端（Vue3）                       │
+│  ┌────────────┐  ┌────────────┐  ┌───────────────┐ │
+│  │ 路由守卫    │  │ 侧边栏过滤  │  │ 页面内按钮控制  │ │
+│  │ (permission)│  │ (permission)│  │ (ACL/owner)   │ │
+│  └──────┬─────┘  └──────┬─────┘  └───────┬───────┘ │
+│         └───────────────┴─────────────────┘          │
+│                         │ HTTP (Bearer token)        │
+└─────────────────────────┼────────────────────────────┘
+                          │
+┌─────────────────────────┼────────────────────────────┐
+│                    后端（FastAPI）                     │
+│  ┌──────────────┐  ┌────┴───────┐  ┌──────────────┐  │
+│  │ get_current  │  │ 资源可见性  │  │ GPU 分配校验  │  │
+│  │ _user (鉴权) │  │ 过滤器      │  │              │  │
+│  └──────┬───────┘  └────┬───────┘  └──────┬───────┘  │
+│         │               │                  │          │
+│  ┌──────┴───────────────┴──────────────────┘          │
+│  │              PlatformStore                        │  │
+│  │  users | acls | roles | gpu_assignments |         │  │
+│  │  datasets | models | fine_tune_tasks | ...        │  │
+│  └───────────────────────────────────────────────────│  │
+│  │              audit_logs (审计日志)                │  │
+│  └───────────────────────────────────────────────────│  │
+└─────────────────────────────────────────────────────────┘
+```
+
+**鉴权链路**：
+1. 请求到达 → `get_current_user` 从 `Authorization: Bearer platform-token-{user_id}` 解析当前用户
+2. 页面级权限 → 检查 `user.permissions` 是否包含路由对应的权限码
+3. 资源级权限 → 检查资源的 `created_by` 字段（所有权）或 `acls` 表（ACL 授权）
+4. GPU 权限 → 检查 `gpu_assignments` 表确认用户是否被分配了请求的 GPU
+
+---
+
+## 3. 角色体系
+
+### 3.1 内置角色
+
+| 角色 code | 中文名 | 说明 |
+|---|---|---|
+| `admin` | 超级管理员 | 拥有全部权限码；可见全部资源；可管理用户和 GPU 分配 |
+| `operator` | 操作员 | 可创建/操作自己的数据集、模型、训练任务；不可管理用户 |
+| `viewer` | 观察员 | 只读权限；可查看被授权的资源；不可创建或修改 |
+| `guest` | 访客 | 仅登录和看板；无业务操作权限（扩展预留） |
+
+### 3.2 角色与权限码映射
+
+| 权限码 | admin | operator | viewer |
+|---|---|---|---|
+| `dashboard` | ✅ | ✅ | ✅ |
+| `fine-tune` | ✅ | ✅ | — |
+| `model-eval` | ✅ | ✅ | — |
+| `model-inference` | ✅ | ✅ | — |
+| `model-manage` | ✅ | ✅ | — |
+| `dataset` | ✅ | ✅ | — |
+| `data-process` | ✅ | ✅ | — |
+| `data-convert` | ✅ | ✅ | — |
+| `compute` | ✅ | ✅ | — |
+| `hardware` | ✅ | ✅ | ✅ |
+| `logs` | ✅ | ✅ | ✅ |
+| `user-settings` | ✅ | — | — |
+
+### 3.3 权限修改规则
+
+- **只有 admin 角色的用户**可以修改其他用户的角色和权限码
+- admin 用户的 `protected=True` 标记，防止被删除或降级
+- 权限修改操作产生审计日志：`action=user.permission.update`
+- 用户可以查看自己的权限，不能修改自己的权限
+
+---
+
+## 4. 页面权限码
+
+| 权限码 | 对应路由 | 功能 |
+|---|---|---|
+| `dashboard` | `/dashboard` | 服务看板 |
+| `fine-tune` | `/fine-tune`, `/fine-tune/create`, `/training-log/:id` | 模型训练 |
+| `model-eval` | `/model-eval`, `/model-eval/create`, `/model-eval/:id` | 模型评测 |
+| `model-inference` | `/model-inference`, `/model-inference/create`, `/model-inference/chat/:id` | 模型推理 |
+| `model-manage` | `/model-manage`, `/model-manage/create`, `/model-manage/:id/edit`, `/model-manage/merge` | 模型管理 |
+| `dataset` | `/dataset`, `/dataset/create`, `/dataset/:id/preview` | 数据集管理 |
+| `data-process` | `/data-process`, `/data-process/create`, `/data-process/:id` | 数据处理 |
+| `data-convert` | `/data-convert`, `/tools` | 数据转换与工具 |
+| `compute` | `/compute` | 算力节点 |
+| `hardware` | `/hardware` | 平台性能 |
+| `logs` | `/logs`, `/training-log/:id` | 查看日志 |
+| `user-settings` | `/user-settings`, `/tenants`, `/projects`, `/approvals`, `/audit-logs` | 系统设置与平台治理 |
+
+---
+
+## 5. 资源所有权与可见性
+
+### 5.1 所有权模型
+
+每个用户可创建的资源都携带 `created_by`（或 `owner_id`）字段，标识资源所有者。
+
+| 资源类型 | 表 | 所有者字段 | 说明 |
+|---|---|---|---|
+| 数据集 | `datasets` | `created_by` | 用户上传/创建的数据集 |
+| 基座模型 | `models` | `created_by` | 登记的本地/API 模型 |
+| 训练产物 | `trained_models` | `created_by` | 微调产出的模型 |
+| 训练任务 | `fine_tune_tasks` | `payload.created_by` | 微调任务 |
+| 评测任务 | `eval_tasks` | `created_by` | 评测任务 |
+| 推理任务 | `compare_tasks` (payload) | `created_by` | 推理/对比任务 |
+| 数据处理任务 | `data_process_tasks` | `created_by` | 数据处理任务 |
+| 数据转换任务 | `data_convert_jobs` | `created_by` | 数据转换任务 |
+
+### 5.2 可见性规则
+
+```
+资源列表查询过滤逻辑：
+
+if user.role == "admin":
+    返回全部资源
+elif resource.created_by == user.id:
+    返回（资源所有者可见自己的资源）
+elif acl 中存在 (principal_type="user", principal_id=user.id, permission 包含 "read"):
+    返回（被 ACL 显式授权的资源）
+elif acl 中存在 (principal_type="role", principal_id=user.role, permission 包含 "read"):
+    返回（被角色级 ACL 授权的资源）
+else:
+    不可见
+```
+
+### 5.3 所有权操作矩阵
+
+| 操作 | admin | 资源所有者 | 其他被授权用户 | 其他用户 |
+|---|---|---|---|---|
+| 查看资源 | ✅ 全部 | ✅ 自己的 | ✅ ACL 授权范围内 | ❌ |
+| 编辑资源 | ✅ | ✅ 自己的 | ✅ ACL 含 write 时 | ❌ |
+| 删除资源 | ✅ | ✅ 自己的（需审批） | ❌ | ❌ |
+| 分享/授权 | ✅ | ✅ 自己的 | ❌ | ❌ |
+| 使用资源（训练/推理/评测） | ✅ | ✅ 自己的 | ✅ ACL 含 execute 时 | ❌ |
+
+### 5.4 数据集可见性示例
+
+```
+用户 A 创建了数据集 ds_A1 → 只有 A 和 admin 可见
+用户 A 通过 ACL 把 ds_A1 的 read 权限授给用户 B → B 也可见
+用户 A 通过 ACL 把 ds_A1 的 write 权限授给 operator 角色 → 所有 operator 可编辑
+管理员可在任何数据集上设置 ACL
+```
+
+---
+
+## 6. 资源级 ACL（访问控制列表）
+
+### 6.1 ACL 表结构
+
+```sql
+CREATE TABLE IF NOT EXISTS acls (
+    id              TEXT PRIMARY KEY,
+    resource_type   TEXT NOT NULL,      -- 资源类型: dataset / model / trained_model / fine_tune_task / ...
+    resource_id     TEXT NOT NULL,      -- 资源 ID
+    principal_type  TEXT NOT NULL,      -- 授权主体类型: user / role
+    principal_id    TEXT NOT NULL,      -- 授权主体 ID: user_id 或 role name
+    permission      TEXT NOT NULL,      -- 权限: read / write / execute / download / delete / admin
+    create_time     TEXT
+);
+```
+
+### 6.2 权限粒度
+
+| 权限值 | 含义 | 覆盖关系 |
+|---|---|---|
+| `read` | 查看资源详情、列表 | — |
+| `write` | 编辑资源内容/元数据 | 覆盖 `read` |
+| `execute` | 使用资源（如用数据集训练、用模型推理） | 覆盖 `read` |
+| `download` | 下载资源文件 | 独立权限 |
+| `delete` | 删除资源 | 独立权限（通常需审批） |
+| `admin` | 完全控制（含 ACL 管理） | 覆盖以上全部 |
+
+### 6.3 ACL 管理接口
+
+| 接口 | 方法 | 权限要求 | 说明 |
+|---|---|---|---|
+| `/resources/{type}/{id}/acl` | GET | admin 或资源所有者 | 查询资源 ACL |
+| `/resources/{type}/{id}/acl` | PUT | admin 或资源所有者 | 设置资源 ACL（全量替换） |
+
+### 6.4 ACL 管理规则
+
+- **admin** 可以在任何资源上设置 ACL
+- **资源所有者** 可以在自己的资源上设置 ACL
+- **被授权用户** 不能转授自己获得的权限
+- ACL 变更产生审计日志：`action=resource.acl.set`
+- 设置 ACL 时全量替换该资源的所有 ACL 条目
+
+### 6.5 前端 ACL 管理入口
+
+在数据集详情、模型详情、训练任务详情页面提供「资源授权」按钮，弹出 ACL 管理对话框：
+- 显示当前 ACL 列表（主体类型 + 主体名称 + 权限勾选）
+- 支持按用户或按角色添加授权
+- 权限以多选框形式展示（read / write / execute / download / delete）
+
+---
+
+## 7. GPU 算力分配与隔离
+
+### 7.1 设计背景
+
+服务器可能安装多张 GPU 卡（如 8×A100），需要精细化管控：
+- 管理员指定哪些用户可以使用哪些 GPU 卡
+- 未被分配的 GPU 卡对用户不可见或不可选
+- admin 可以使用全部 GPU
+
+### 7.2 GPU 分配表
+
+```sql
+CREATE TABLE IF NOT EXISTS gpu_assignments (
+    id          TEXT PRIMARY KEY,
+    gpu_id      TEXT NOT NULL,          -- gpus 表的外键（node_id + gpu_index 组合）
+    node_id     TEXT NOT NULL,          -- 算力节点 ID
+    gpu_index   INTEGER NOT NULL,       -- GPU 卡序号
+    user_id     TEXT NOT NULL,          -- 被分配的用户 ID
+    assigned_by TEXT,                  -- 分配操作人 ID（admin）
+    assigned_at TEXT NOT NULL,          -- 分配时间
+    UNIQUE (node_id, gpu_index, user_id)  -- 一张卡可分配给多个用户，但每对 (卡, 用户) 唯一
+);
+```
+
+### 7.3 分配规则
+
+| 规则 | 说明 |
+|---|---|
+| 谁可分配 | 只有 `admin` 角色可以分配 GPU |
+| admin 使用 | admin 可使用全部 GPU，不需要显式分配 |
+| 普通用户 | 只能使用 `gpu_assignments` 中分配给自己的 GPU |
+| 共享分配 | 一张 GPU 可分配给多个用户（非独占），但同时只能被一个任务占用 |
+| 默认策略 | 新用户默认不分配任何 GPU，由管理员显式分配 |
+
+### 7.4 GPU 分配接口
+
+| 接口 | 方法 | 权限 | 说明 |
+|---|---|---|---|
+| `/compute/gpu-assignments` | GET | admin | 查看全部分配关系 |
+| `/compute/gpu-assignments` | POST | admin | 批量分配（body: `{ assignments: [{ node_id, gpu_index, user_id }] }`） |
+| `/compute/gpu-assignments/{id}` | DELETE | admin | 撤销某条分配 |
+| `/compute/my-gpus` | GET | 登录用户 | 查看自己可用的 GPU 列表 |
+
+### 7.5 训练/评测/推理 GPU 选择校验
+
+当普通用户创建训练任务、评测任务、推理任务并选择 GPU 时：
+1. 后端检查 `gpu_assignments` 表，确认用户被分配了所选 GPU
+2. 未被分配的 GPU → 返回 403 `"无权使用 GPU {node}:{index}"`
+3. admin 用户跳过此校验
+
+### 7.6 前端 GPU 选择交互
+
+- 普通用户在创建任务选择 GPU 时，下拉列表只显示自己被分配的 GPU
+- admin 用户在下拉列表中可看到全部 GPU
+- 未分配任何 GPU 的用户，GPU 选择区域显示提示："未分配 GPU，请联系管理员"
+
+---
+
+## 8. 审批拦截机制
+
+### 8.1 需要审批的操作
+
+| 操作 | 触发条件 | 审批动作 code |
+|---|---|---|
+| 删除他人数据集 | 非 admin 删除 `created_by != user.id` 的数据集 | `dataset.delete` |
+| 删除他人模型 | 非 admin 删除 `created_by != user.id` 的模型 | `model.delete` |
+| 停止他人训练任务 | 非 admin 停止 `created_by != user.id` 的任务 | `fine_tune.stop` |
+| 发布模型到推理服务 | 任何用户（含 admin）发布到生产环境 | `model_service.publish` |
+| 删除项目空间 | 存在待审批变更时拒绝 | `project.delete` |
+| 归档项目空间 | 存在待审批变更时拒绝 | `project.archive` |
+| 导出训练产物 | 非 admin 导出他人训练的模型 | `trained_model.export` |
+
+### 8.2 审批流程
+
+```
+普通用户发起高风险操作
+         │
+         ▼
+  ┌──────────────┐        ┌──────────────────────┐
+  │ admin 旁路？ │───是──▶│ 直接执行 + 审计日志   │
+  └──────┬───────┘        └──────────────────────┘
+         │ 否
+         ▼
+  ┌──────────────────────┐
+  │ 创建审批实例          │
+  │ status=pending       │
+  │ 返回 202（待审批）    │
+  └──────────┬───────────┘
+             │
+             ▼
+  ┌──────────────────────┐
+  │ 管理员审批            │
+  │ POST /approvals/:id  │
+  │ /steps/:idx/decision │
+  └──────────┬───────────┘
+             │
+      ┌──────┴──────┐
+      │             │
+   approved      rejected
+      │             │
+      ▼             ▼
+  执行操作      不执行
+  +审计日志    +审计日志
+```
+
+### 8.3 审批模板
+
+审批模板定义了特定操作需要几步审批、每步的审批人是谁：
+
+```json
+{
+  "id": "tpl_001",
+  "name": "删除数据集审批",
+  "action": "dataset.delete",
+  "steps": [
+    { "approver_id": "u_admin", "step_name": "管理员审核" }
+  ]
+}
+```
+
+### 8.4 审批拦截点
+
+在项目模块的 `_require_no_pending_approval` 函数中，当存在待审批实例时拒绝执行新操作。其他模块通过 `_require_approval_or_admin` 函数实现 admin 旁路或创建审批实例。
+
+---
+
+## 9. 审计日志
+
+### 9.1 审计范围
+
+所有写操作和敏感操作必须产生审计日志：
+
+| 动作分类 | action 示例 |
+|---|---|
+| 用户管理 | `user.create`, `user.update`, `user.delete`, `user.permission.update` |
+| 租户管理 | `tenant.create`, `tenant.update`, `tenant.quota.set`, `tenant.retention.set` |
+| 项目管理 | `project.create`, `project.update`, `project.archive`, `project.delete`, `project.member.add`, `project.member.update`, `project.member.remove` |
+| 资源 ACL | `resource.acl.set` |
+| 模型管理 | `model.create`, `model.update`, `model.delete`, `model.merge` |
+| 数据集 | `dataset.create`, `dataset.update`, `dataset.delete`, `dataset.upload` |
+| 训练任务 | `fine_tune.create`, `fine_tune.start`, `fine_tune.stop`, `fine_tune.delete` |
+| 评测任务 | `eval.create`, `eval.start`, `eval.stop` |
+| 推理任务 | `inference.create`, `inference.start`, `inference.stop` |
+| 审批 | `approval.create`, `approval.decide` |
+| 留存策略 | `retention.create`, `retention.update`, `retention.delete` |
+| GPU 分配 | `gpu.assign`, `gpu.unassign` |
+
+### 9.2 审计日志字段
+
+```sql
+CREATE TABLE IF NOT EXISTS audit_logs (
+    id          TEXT PRIMARY KEY,
+    time        TEXT NOT NULL,         -- ISO8601 时间戳
+    tenant_id   TEXT,                 -- 租户 ID（可选）
+    project_id  TEXT,                 -- 项目 ID（可选）
+    actor_id    TEXT,                 -- 操作人 ID
+    action      TEXT NOT NULL,         -- 动作类型
+    target_type TEXT NOT NULL,         -- 目标资源类型
+    target_id   TEXT,                 -- 目标资源 ID
+    detail      TEXT,                 -- 详情摘要
+    client_ip   TEXT                  -- 客户端 IP
+);
+```
+
+### 9.3 查询与导出
+
+| 接口 | 方法 | 说明 |
+|---|---|---|
+| `/system/audit-logs` | GET | 分页查询，支持按 tenant_id / project_id / actor_id / action / target_type / start_time / end_time 筛选 |
+| `/system/audit-logs/export` | GET | CSV 导出，与应用查询相同的过滤条件 |
+
+---
+
+## 10. 接口鉴权流程
+
+### 10.1 Token 格式
+
+```
+Authorization: Bearer platform-token-{user_id}
+```
+
+登录成功后返回 `token` 和 `user` 信息。Token 中编码了 `user_id`，后端通过 `get_current_user` 解析。
+
+### 10.2 鉴权层级
+
+```
+请求到达
+  │
+  ├─ 1. 公开路径检查（/health, /login, /system-info）→ 直接放行
+  │
+  ├─ 2. Token 解析 → get_current_user
+  │     ├─ 无 token / token 无效 → 401
+  │     └─ 用户不存在 / 状态 disabled → 401
+  │
+  ├─ 3. 页面级权限码检查（路由守卫 / Depends）
+  │     └─ user.permissions 不含所需权限码 → 403
+  │
+  ├─ 4. 资源级权限检查（路由函数内）
+  │     ├─ admin → 全部放行
+  │     ├─ resource.created_by == user.id → 放行
+  │     ├─ ACL 检查 has_resource_access() → 有授权则放行
+  │     └─ 否则 → 403
+  │
+  ├─ 5. GPU 权限检查（训练/评测/推理创建时）
+  │     ├─ admin → 全部放行
+  │     ├─ gpu_assignments 检查 → 有分配则放行
+  │     └─ 否则 → 403
+  │
+  └─ 6. 审批拦截检查（高风险操作）
+        ├─ admin → 旁路，直接执行
+        ├─ 无待审批实例 → 可执行
+        ├─ 有待审批实例 → 409 "存在待审批的变更"
+        └─ 需要审批 → 202 "已创建审批实例"
+```
+
+### 10.3 FastAPI 依赖注入
+
+```python
+# 任何需要登录的接口
+@router.get("/datasets")
+async def list_datasets(user: dict = Depends(get_current_user)):
+    ...
+
+# 需要管理员权限的接口
+@router.post("/users")
+async def create_user(user: dict = Depends(require_admin)):
+    ...
+
+# 需要资源级权限检查的接口
+@router.delete("/datasets/{dataset_id}")
+async def delete_dataset(
+    dataset_id: str,
+    user: dict = Depends(get_current_user),
+):
+    if not has_resource_access("dataset", dataset_id, user, "delete"):
+        raise HTTPException(403, "forbidden")
+    ...
+```
+
+---
+
+## 11. 数据库表结构
+
+### 11.1 现有表（已实现）
+
+| 表名 | 用途 |
+|---|---|
+| `users` | 用户表（id, username, password_hash, role, status, permissions, protected） |
+| `roles` | 角色定义（name, permissions） |
+| `sessions` | 登录会话（user_id, issued_at, expires_at, ip） |
+| `acls` | 资源访问控制列表（resource_type, resource_id, principal_type, principal_id, permission） |
+| `audit_logs` | 审计日志（actor_id, action, target_type, target_id, time） |
+| `datasets` | 数据集（需补充 `created_by` 字段） |
+| `models` | 基座模型（需补充 `created_by` 字段） |
+| `trained_models` | 训练产物（需补充 `created_by` 字段） |
+| `fine_tune_tasks` | 训练任务（payload 中存储 `created_by`） |
+| `gpus` | GPU 设备（node_id, gpu_index, uuid, name, memory） |
+| `compute_nodes` | 算力节点 |
+| `tenants` | 租户 |
+| `projects` | 项目空间 |
+| `project_members` | 项目成员 |
+| `approval_templates` | 审批模板 |
+| `approval_instances` | 审批实例 |
+| `retention_policies` | 留存策略 |
+
+### 11.2 需新增/补充的表和字段
+
+#### 新增 `gpu_assignments` 表
+
+```sql
+CREATE TABLE IF NOT EXISTS gpu_assignments (
+    id           TEXT PRIMARY KEY,
+    node_id      TEXT NOT NULL REFERENCES compute_nodes(id) ON DELETE CASCADE,
+    gpu_index    INTEGER NOT NULL,
+    user_id      TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
+    assigned_by  TEXT,
+    assigned_at  TEXT NOT NULL,
+    UNIQUE (node_id, gpu_index, user_id)
+);
+CREATE INDEX IF NOT EXISTS idx_gpu_assignments_user ON gpu_assignments(user_id);
+CREATE INDEX IF NOT EXISTS idx_gpu_assignments_gpu ON gpu_assignments(node_id, gpu_index);
+```
+
+#### 补充 `created_by` 字段
+
+```sql
+-- 数据集表补充所有者
+ALTER TABLE datasets ADD COLUMN IF NOT EXISTS created_by TEXT;
+
+-- 基座模型表补充所有者
+ALTER TABLE models ADD COLUMN IF NOT EXISTS created_by TEXT;
+
+-- 训练产物表补充所有者
+ALTER TABLE trained_models ADD COLUMN IF NOT EXISTS created_by TEXT;
+
+-- 评测任务表补充所有者
+ALTER TABLE eval_tasks ADD COLUMN IF NOT EXISTS created_by TEXT;
+
+-- 推理任务表补充所有者
+-- 注意：inference_tasks 表尚未创建，后续建表时直接包含 created_by 字段
+```
+
+---
+
+## 12. API 接口清单
+
+### 12.1 鉴权接口
+
+| 接口 | 方法 | 鉴权 | 说明 |
+|---|---|---|---|
+| `/modelTF/login` | POST | 公开 | 登录，返回 token + user |
+| `/modelTF/me` | GET | Bearer token | 获取当前用户信息 |
+| `/modelTF/users` | GET | admin | 用户列表 |
+| `/modelTF/users` | POST | admin | 创建用户 |
+| `/modelTF/users/:id` | PUT | admin | 更新用户（角色/状态/权限） |
+| `/modelTF/users/:id` | DELETE | admin | 删除用户（protected 用户不可删） |
+| `/modelTF/users/:id/reset-password` | POST | admin | 重置密码 |
+| `/modelTF/system/permissions/codes` | GET | 登录 | 权限码清单 |
+| `/modelTF/system/permissions` | GET | 登录 | 权限码 + 角色定义 |
+
+### 12.2 资源 ACL 接口
+
+| 接口 | 方法 | 鉴权 | 说明 |
+|---|---|---|---|
+| `/modelTF/resources/:type/:id/acl` | GET | admin 或所有者 | 查询资源 ACL |
+| `/modelTF/resources/:type/:id/acl` | PUT | admin 或所有者 | 设置资源 ACL |
+
+### 12.3 GPU 分配接口
+
+| 接口 | 方法 | 鉴权 | 说明 |
+|---|---|---|---|
+| `/modelTF/compute/gpu-assignments` | GET | admin | 查看全部分配 |
+| `/modelTF/compute/gpu-assignments` | POST | admin | 批量分配 |
+| `/modelTF/compute/gpu-assignments/:id` | DELETE | admin | 撤销分配 |
+| `/modelTF/compute/my-gpus` | GET | 登录 | 查看自己可用 GPU |
+
+### 12.4 审批接口
+
+| 接口 | 方法 | 鉴权 | 说明 |
+|---|---|---|---|
+| `/modelTF/approvals/templates` | GET/POST | admin | 审批模板列表/创建 |
+| `/modelTF/approvals` | GET/POST | 登录 | 审批实例列表/创建 |
+| `/modelTF/approvals/:id` | GET | 登录 | 审批实例详情 |
+| `/modelTF/approvals/:id/steps/:idx/decision` | POST | 审批人 | 审批决策 |
+
+### 12.5 审计接口
+
+| 接口 | 方法 | 鉴权 | 说明 |
+|---|---|---|---|
+| `/modelTF/system/audit-logs` | GET | admin | 审计日志分页查询 |
+| `/modelTF/system/audit-logs/export` | GET | admin | CSV 导出 |
+
+### 12.6 租户/项目接口
+
+| 接口 | 方法 | 鉴权 | 说明 |
+|---|---|---|---|
+| `/modelTF/tenants` | GET/POST | admin | 租户列表/创建 |
+| `/modelTF/tenants/:id` | GET/PUT | admin | 租户详情/更新 |
+| `/modelTF/tenants/:id/quota` | PUT | admin | 设置配额 |
+| `/modelTF/tenants/:id/retention-policy` | PUT | admin | 绑定留存策略 |
+| `/modelTF/projects` | GET/POST | 登录 | 项目列表/创建 |
+| `/modelTF/projects/:id` | GET/PUT | 登录 | 项目详情/更新 |
+| `/modelTF/projects/:id/archive` | POST | admin 或所有者 | 归档（审批拦截） |
+| `/modelTF/projects/:id/members` | GET/POST | 登录 | 成员列表/添加 |
+| `/modelTF/projects/:id/members/:uid` | PUT/DELETE | admin 或所有者 | 改角色/移除 |
+
+### 12.7 留存策略接口
+
+| 接口 | 方法 | 鉴权 | 说明 |
+|---|---|---|---|
+| `/modelTF/retention-policies` | GET/POST | admin | 策略列表/创建 |
+| `/modelTF/retention-policies/:id` | GET/PUT/DELETE | admin | 策略详情/更新/删除 |
+
+---
+
+## 13. 前端权限控制
+
+### 13.1 路由守卫
+
+```typescript
+// router/index.ts
+router.beforeEach((to, _from, next) => {
+  const auth = useAuthStore()
+  auth.syncSession()
+
+  if (to.meta.public) {
+    next()
+    return
+  }
+
+  if (!auth.isLoggedIn) {
+    next({ name: 'login' })
+    return
+  }
+
+  if (!to.meta.skipPermission) {
+    const permission = requiredPermission(to.path, to.meta.permission)
+    if (permission && !auth.hasPermission(permission)) {
+      next({ name: 'permission-denied', replace: true })
+      return
+    }
+  }
+
+  next()
+})
+```
+
+### 13.2 侧边栏过滤
+
+```typescript
+// layouts/MainLayout.vue
+const visibleMenus = computed(() =>
+  allMenus.filter(menu => {
+    if (!menu.permission) return true
+    return auth.hasPermission(menu.permission)
+  })
+)
+```
+
+### 13.3 资源级按钮控制
+
+```vue
+<!-- 数据集详情页 -->
+<template>
+  <el-button v-if="canEdit" @click="handleEdit">编辑</el-button>
+  <el-button v-if="canDelete" @click="handleDelete">删除</el-button>
+  <el-button v-if="canManageAcl" @click="showAclDialog = true">资源授权</el-button>
+</template>
+
+<script setup>
+const canEdit = computed(() =>
+  isAdmin.value || resource.value.created_by === userId.value
+)
+const canDelete = computed(() =>
+  isAdmin.value || resource.value.created_by === userId.value
+)
+const canManageAcl = computed(() =>
+  isAdmin.value || resource.value.created_by === userId.value
+)
+</script>
+```
+
+### 13.4 GPU 选择过滤
+
+```vue
+<!-- 创建训练任务页 -->
+<template>
+  <el-select v-model="selectedGpus" multiple>
+    <el-option
+      v-for="gpu in availableGpus"
+      :key="gpu.id"
+      :label="`${gpu.node_name} GPU ${gpu.gpu_index}`"
+      :value="gpu.id"
+    />
+  </el-select>
+  <el-alert v-if="availableGpus.length === 0 && !isAdmin" type="warning">
+    未分配 GPU，请联系管理员
+  </el-alert>
+</template>
+
+<script setup>
+// 普通用户只看到 my-gpus 返回的列表
+// admin 看到全部 GPU
+const availableGpus = ref([])
+async function loadGpus() {
+  if (isAdmin.value) {
+    availableGpus.value = await getAllGpus()
+  } else {
+    availableGpus.value = await getMyGpus()
+  }
+}
+</script>
+```
+
+---
+
+## 14. 安全设计补充
+
+### 14.1 密码安全
+
+- 密码使用 PBKDF2-SHA256 存储（salt + 390000 次迭代）
+- 旧系统明文密码在首次登录时自动升级为哈希
+- 管理员可重置用户密码，用户不可自行修改密码（本期设计）
+- 默认密码：`platform123`（创建用户时由管理员设定）
+
+### 14.2 会话安全
+
+| 规则 | 说明 |
+|---|---|
+| Token 格式 | `platform-token-{user_id}` |
+| 会话超时 | 默认 30 分钟无操作自动过期 |
+| 并发会话 | 同一用户可有多会话，各自独立计时 |
+| 会话续期 | 前端定时调用 `auth.refresh()` 续期 |
+| 强制下线 | admin 可通过修改用户 status=disabled 使其 token 失效 |
+
+### 14.3 操作限流
+
+| 接口 | 限制 |
+|---|---|
+| `/login` | 同一 IP 5 次/分钟，失败后 30 秒冷却 |
+| 文件上传 | 单文件最大由配置控制，默认 2GB |
+| 训练任务创建 | 同一用户并发运行任务数受 GPU 分配限制 |
+
+### 14.4 数据安全
+
+| 规则 | 说明 |
+|---|---|
+| 软删除 | 数据集、模型、任务使用 `deleted_at` 标记，保留审计可追溯 |
+| 敏感字段 | API 密钥（`api_key`）在列表接口不返回明文 |
+| 下载审计 | 数据集下载产生审计日志，记录下载人和时间 |
+| 导出审计 | 训练产物导出产生审计日志 |
+
+### 14.5 多租户隔离
+
+| 规则 | 说明 |
+|---|---|
+| 租户隔离 | 同一租户内的资源相互可见；跨租户默认不可见 |
+| 项目隔离 | 项目内资源受项目 ACL 控制；项目间默认不可见 |
+| admin 旁路 | admin 可跨租户/项目访问全部资源 |
+| 配额管控 | 租户级配额限制 GPU 并发数、存储容量、最大项目数 |
+
+---
+
+## 15. 实施计划
+
+### 15.1 已实现
+
+| 功能 | 状态 |
+|---|---|
+| 登录/会话/Token | ✅ 已实现 |
+| 用户 CRUD + 权限码 | ✅ 已实现 |
+| 角色定义 | ✅ 已实现 |
+| 资源 ACL（acls 表 + 接口） | ✅ 已实现 |
+| 审计日志（查询 + 导出） | ✅ 已实现 |
+| 审批模板/实例 | ✅ 已实现 |
+| 项目空间 + 成员 | ✅ 已实现 |
+| 租户 + 配额 + 留存 | ✅ 已实现 |
+| 审批拦截（项目归档/删除） | ✅ 已实现 |
+| 资源所有权 ACL 字段适配（subject_type/permissions[]） | ✅ 已实现 |
+
+### 15.2 待实现
+
+| 功能 | 优先级 | 涉及表/接口 |
+|---|---|---|
+| GPU 分配表 + 接口 | P0 | `gpu_assignments` 表 + `/compute/gpu-assignments` + `/compute/my-gpus` |
+| 资源 `created_by` 字段补充 | P0 | `datasets` / `models` / `trained_models` / `eval_tasks` 表 ALTER |
+| 资源列表按 `created_by` + ACL 过滤 | P0 | `platform_store.py` 中 datasets/models/tasks 列表方法 |
+| GPU 选择校验（训练/评测/推理创建时） | P0 | `platform.py` 中 create_task/eval/inference |
+| 前端 GPU 下拉过滤 | P1 | 前端创建任务页面 |
+| 前端资源授权按钮 | P1 | 前端数据集/模型/任务详情页 |
+| 前端权限管理页面优化 | P1 | 前端用户设置页面 |
+| 审批拦截扩展（删除数据集/模型/停止任务） | P1 | `platform.py` 中 delete/stop 接口 |
+| 密码安全策略（用户自行修改） | P2 | 新增 `/users/me/password` 接口 |
+| 操作限流（login 限流） | P2 | 中间件或 SlowAPI |
+| 多租户隔离（按 tenant_id 过滤） | P2 | 各列表接口增加 tenant_id 过滤 |
+
+### 15.3 实施步骤
+
+1. **数据库迁移**：创建 `gpu_assignments` 表，为资源表补充 `created_by` 字段
+2. **后端接口**：实现 GPU 分配 CRUD + `my-gpus` + 创建任务时的 GPU 权限校验
+3. **资源过滤**：在 `datasets()` / `models()` / `tasks()` 等列表方法中按 `created_by` + ACL 过滤
+4. **前端适配**：GPU 下拉过滤、资源授权按钮、权限管理页面优化
+5. **审批扩展**：在删除/停止接口中接入 `_require_approval_or_admin`
+6. **测试补充**：扩展 `test_governance.py` 覆盖 GPU 分配、资源过滤、审批扩展场景

docs/postgres-schema.sql

@@ -1061,8 +1061,20 @@ CREATE TABLE IF NOT EXISTS compute_nodes (
   name varchar(150) NOT NULL,
   host varchar(200) NOT NULL,
   api_base_url text NOT NULL,
+  file_gateway_url text NOT NULL DEFAULT '',
   storage_node_id uuid REFERENCES storage_nodes(id) ON DELETE SET NULL,
   status varchar(40) NOT NULL DEFAULT 'online',
+  scheduler_status varchar(40) NOT NULL DEFAULT 'online',
+  scheduler_weight integer NOT NULL DEFAULT 100,
+  enabled boolean NOT NULL DEFAULT true,
+  max_parallel_jobs integer NOT NULL DEFAULT 1,
+  data_root text NOT NULL DEFAULT '/data/yg-ft',
+  model_root text NOT NULL DEFAULT '/data/yg-ft/models',
+  log_root text NOT NULL DEFAULT '/opt/yg-ft/logs/training',
+  api_version varchar(40) NOT NULL DEFAULT 'v1',
+  capabilities jsonb NOT NULL DEFAULT '[]'::jsonb,
+  description text,
+  health_detail jsonb NOT NULL DEFAULT '{}'::jsonb,
   agent_version varchar(80),
   gpu_count integer NOT NULL DEFAULT 0,
   last_heartbeat_at timestamptz,
@@ -1434,10 +1446,15 @@ ALTER TABLE fine_tune_tasks ADD COLUMN IF NOT EXISTS tenant_id uuid REFERENCES t
 ALTER TABLE fine_tune_tasks ADD COLUMN IF NOT EXISTS project_id uuid REFERENCES projects(id) ON DELETE SET NULL;
 ALTER TABLE fine_tune_tasks ADD COLUMN IF NOT EXISTS owner_id uuid REFERENCES users(id) ON DELETE SET NULL;
 ALTER TABLE fine_tune_tasks ADD COLUMN IF NOT EXISTS approval_status approval_status NOT NULL DEFAULT 'not_required';
+ALTER TABLE fine_tune_tasks ADD COLUMN IF NOT EXISTS compute_node_id uuid REFERENCES compute_nodes(id) ON DELETE SET NULL;
 ALTER TABLE fine_tune_tasks ADD COLUMN IF NOT EXISTS compute_job_id uuid REFERENCES compute_jobs(id) ON DELETE SET NULL;
 ALTER TABLE fine_tune_tasks ADD COLUMN IF NOT EXISTS resume_checkpoint_id uuid REFERENCES fine_tune_checkpoints(id) ON DELETE SET NULL;
 CREATE INDEX IF NOT EXISTS idx_fine_tune_tasks_scope_status
   ON fine_tune_tasks(tenant_id, project_id, status, created_at DESC) WHERE deleted_at IS NULL;
+CREATE INDEX IF NOT EXISTS idx_fine_tune_tasks_compute_job
+  ON fine_tune_tasks(compute_job_id) WHERE deleted_at IS NULL;
+CREATE INDEX IF NOT EXISTS idx_fine_tune_tasks_node_status
+  ON fine_tune_tasks(compute_node_id, status, created_at DESC) WHERE deleted_at IS NULL;
 
 ALTER TABLE inference_tasks ADD COLUMN IF NOT EXISTS tenant_id uuid REFERENCES tenants(id) ON DELETE SET NULL;
 ALTER TABLE inference_tasks ADD COLUMN IF NOT EXISTS project_id uuid REFERENCES projects(id) ON DELETE SET NULL;