Merge branch 'ft_wyt' of http://www.caoxiaozhu.com:13001/YG-Soft/YG_FT into ft_wyt

内容自然高度超出视口时，画布 flex 不收缩导致分页条被裁掉且无滚动
条可及；为 .data-table-page 的画布补充 flex: 1 1 auto 与 min-height: 0。

.gitignore

@@ -208,3 +208,10 @@ docker/compute/data/yg-ft/logs/**
 !docker/compute/data/yg-ft/logs/training/
 !docker/compute/data/yg-ft/**/.gitkeep
 !docker/compute/data/yg-ft/**/README.md
+
+# Offline deployment bundle - 离线部署包（镜像、运行时等大文件，不提交）
+docker/offline/
+
+# MinIO object storage data - 对象存储运行时数据，勿提交，保留目录结构
+docker/minio/data/*
+!docker/minio/data/.gitkeep

README.md

@@ -1,231 +1,59 @@
 # YG_FT 模型微调平台
 
-YG_FT 是一个面向企业治理场景的模型微调平台，覆盖用户中心、多租户、项目隔离、数据集管理、模型管理、训练任务、评测、推理、审批流、审计留存、算力调度和训练引擎适配。
+YG_FT 是面向多用户、多租户和多算力节点的模型训练与推理平台，提供数据集、模型、训练、权重合并、推理、评测、算力节点、项目隔离、权限和审计能力。
 
-当前前端已有基础页面，后端与算力平台已按多人协作开发方式建立工程骨架，并开始实现正式系统主链路能力。当前代码和 SQL 均作为后续生产演进基线维护，不再以一次性演示或静态 Mock 为开发准则。
+## 架构
-
-## 总体架构
 
 ```text
-YG_FT/
+浏览器 -> Frontend Nginx:16801 -> Backend API:17861
-  frontend/                 # 前端控制台
+                                      |-> PostgreSQL（元数据、权限、审计、任务状态）
-  backend/                  # FastAPI 应用平台后端
+                                      |-> Redis（缓存及任务辅助状态）
-    app/
+                                      |-> MinIO:19000（模型和数据唯一对象源）
-      api/v1/               # 对前端暴露的 REST API
+                                      |-> Compute API:19100
-      core/                 # 配置、日志、中间件、权限等基础能力
+                                                |-> Compute Agent/GPU/LLaMA-Factory
-      db/                   # 数据库连接、迁移、事务工具
+                                      `-> File Gateway:19101
-      modules/              # 业务模块目录
-      schemas/              # Pydantic 入参/出参模型
-      services/             # 跨模块应用服务
-      workers/              # 后台任务入口
-    requirements.txt        # 后端 Python 第三方依赖
-  compute/                  # 算力平台与训练框架适配层
-    api/                    # 内部 Compute API
-    agent/                  # 单机多 GPU 调度与进程管理
-    engines/llama_factory/  # LLaMA-Factory 适配器
-    file_gateway/           # 本地文件上传、下载、导入、产物管理
-  docs/                     # 需求、接口、数据库、开发计划和部署文档
-  docker/                   # 容器化配置
 ```
 
-## 平台分层
+Backend、MinIO 和 Compute 节点可以部署在不同服务器，不依赖跨服务器 Docker 网络，通过 IP、DNS 或负载均衡地址通信。Compute 节点只保存按需准备的本地缓存，MinIO 是模型、数据集、权重、评测报告和训练产物的唯一数据源。
 
-| 层级 | 职责 | 主要目录 |
+## 目录
-| --- | --- | --- |
-| 前端控制台 | 用户操作入口、任务看板、项目/模型/数据集/训练/审批/审计页面 | `frontend/` |
-| 应用平台后端 | 用户中心、多租户、RBAC/ABAC、项目隔离、元数据、审批流、审计、API 编排 | `backend/` |
-| 算力平台 | GPU 发现、资源锁定、训练进程管理、日志采集、产物归档、任务状态同步 | `compute/` |
-| 训练引擎 | 当前固定接入 LLaMA-Factory，预留其他训练平台适配标准 | `compute/engines/` |
-| 数据层 | PostgreSQL、Redis、本地文件存储、日志归档 | `docs/postgres-schema.sql` |
 
-## 当前开发基线
+| 目录 | 作用 |
+| --- | --- |
+| `frontend/` | Vue 3、TypeScript、Element Plus 控制台 |
+| `backend/app/api/v1/` | 平台 REST API |
+| `backend/app/core/` | 配置、认证、权限和日志 |
+| `backend/app/db/` | PostgreSQL 访问和初始化 SQL |
+| `backend/app/modules/` | 系统、资源、审批、数据处理和存储模块 |
+| `backend/app/workers/` | 节点轮询、任务对账和资源同步 |
+| `compute/` | Compute API、Agent、GPU 和训练引擎 |
+| `docker/` | 应用、MinIO、算力服务部署文件 |
+| `docs/` | 架构、权限、部署和测试文档 |
 
-- 使用 FastAPI 提供统一 API 响应结构 `{ code, message, data }`。
+## 端口
-- 本地运行阶段统一使用 PostgreSQL，后端启动时会在 PG 中初始化当前运行表和系统内置账号；模型、数据集、算力节点、GPU、微调任务等业务数据必须通过页面、接口或正式导入流程产生。
-- 支持登录、模型管理、数据集管理、微调任务创建/启动/停止/进度轮询。
-- 支持训练日志、loss 指标、checkpoint 和训练产物接口；真实训练执行器接入前，联调状态机必须通过显式环境变量开启。
-- 支持多算力节点、GPU、任务队列、资源副本和资源同步状态接口。
-- 前端新增 `/compute` 算力节点页面，展示节点地址、权重、标签、启用状态、GPU、队列和资源副本。
-- `compute/engines/llama_factory/adapter.py` 提供 LLaMA-Factory 参数校验、命令生成和日志解析基础能力。
 
-## 前后端一键启动
+| 服务 | 主机端口 | 容器端口 |
+| --- | ---: | ---: |
+| Frontend | 16801 | 80 |
+| Backend API | 17861 | 8000 |
+| Redis | 16379 | 6379 |
+| MinIO API/Console | 19000/19001 | 9000/9001 |
+| Compute API/File Gateway | 19100/19101 | 9100 |
 
-首次使用前，请先按下方“后端启动”和“前端启动”说明安装依赖，并确保
+## 启动
-PostgreSQL 已可用。之后在项目根目录执行：
 
 ```bash
-bash ./start.sh
+cd frontend && npm ci && npm run build && cd ..
+docker build -f docker/app/Dockerfile.backend -t yg-ft-backend-api:latest .
+docker build -f docker/app/Dockerfile.frontend -t yg-ft-frontend-runtime:latest .
+docker build -f docker/compute/Dockerfile.compute -t yg-ft-compute-api:latest .
+cd docker/minio && docker compose up -d
+cd ../app && docker compose up -d
+cd ../compute && docker compose up -d
 ```
 
-脚本会同时启动前端 `http://localhost:16801` 和后端
+## 权限与性能
-`http://127.0.0.1:17861`，按 `Ctrl+C` 会同时停止两个服务。脚本只负责
-启动前后端，不会自动安装依赖，也不会启动 PostgreSQL、Redis 或算力服务。
 
-仅检查依赖和端口而不启动服务：
+系统使用角色权限、资源 ACL、用户/项目/租户归属联合校验；删除为软删除，关键操作写入审计。模型合并前准备 base model 和 adapter，结果归档 MinIO；推理前按选择节点准备缓存。远程 PostgreSQL 延迟会影响全量列表和看板，页面慢时应检查浏览器 Network、Nginx、Backend 日志、连接池和节点可达性。
 
-```bash
+详细部署见 `docker/README.md`，测试见 `测试用例.md`，本次快照见 `docs/20260812/`。
-bash ./start.sh --check
-```
-
-本地启动推荐只配置数据库主机。脚本会复用 `docker/app/.env` 中已有的
-`POSTGRES_USER`、`POSTGRES_PASSWORD` 和 `POSTGRES_DB`，端口默认使用
-PostgreSQL 标准端口 `5432`：
-
-```bash
-DATABASE_HOST='www.caoxiaozhu.com' bash ./start.sh
-```
-
-也可以在 `docker/app/.env` 中增加：
-
-```env
-DATABASE_HOST=www.caoxiaozhu.com
-```
-
-需要使用非标准端口时再设置 `DATABASE_PORT`。`DATABASE_URL` 仍可作为完整连接串
-高级覆盖项；终端环境变量优先级最高。脚本不会输出数据库密码。
-
-## 后端启动
-
-```bash
-cd backend
-python -m venv .venv
-.venv\Scripts\activate
-pip install -r requirements.txt
-uvicorn app.main:app --reload --port 17861
-```
-
-默认接口前缀为 `/modelTF`，例如：
-
-```text
-GET /modelTF/health
-POST /modelTF/login
-GET /modelTF/model-manage
-GET /modelTF/dataset-manage
-GET /modelTF/fine-tune
-GET /modelTF/compute/nodes
-```
-
-本地运行时默认 PostgreSQL 连接：
-
-```text
-DATABASE_URL=postgresql+psycopg://yg_ft:change_me@localhost:15432/yg_ft
-```
-
-本地启动前需要确保 PostgreSQL 已监听 `localhost:15432`，并已创建 `yg_ft` 数据库和 `yg_ft` 用户。后端启动后会自动创建当前运行表并写入内置管理员账号，运行数据统一写入 PostgreSQL。
-
-开发阶段内置登录账号：
-
-| 角色 | 账号 | 密码 | 说明 |
-| --- | --- | --- | --- |
-| 超级管理员 | `admin` | `admin123` | 拥有当前全部页面权限 |
-| 操作员 | `operator` | `operator123` | 拥有业务操作相关页面权限 |
-
-以上账号仅用于本地开发和联调。生产环境初始化后应立即修改密码，或改为企业统一身份认证/管理员初始化流程。
-
-## 前端启动
-
-```bash
-cd frontend
-npm install
-npm run dev
-```
-
-前端开发服务默认运行在 `http://localhost:16801`，并通过 Vite proxy 将 `/modelTF` 转发到 `http://localhost:17861`。
-
-## 算力服务启动
-
-算力服务是一个 FastAPI 应用，同时承载 Compute API（模型训练/推理/GPU 管理）和 File Gateway（文件上传下载）路由。Docker 部署时对外暴露两个端口（19100 和 19101）均指向同一服务，方便应用平台分别配置 `api_base_url` 和 `file_gateway_url`。本地开发只需启动一个进程。
-
-### 方式一：Docker 启动（推荐）
-
-```bash
-cd docker/compute
-cp .env.example .env
-docker compose up -d
-```
-
-### 方式二：本地开发启动
-
-**Windows (cmd)：**
-
-```cmd
-cd /d E:\yg_ft\compute
-set PYTHONPATH=E:\yg_ft
-.\.venv\Scripts\python.exe -m uvicorn api.main:app --reload --port 19100
-```
-
-> `PYTHONPATH=E:\yg_ft` 是必需的，因为代码使用 `from compute.agent...` 绝对导入。
-
-**Linux / macOS：**
-
-```bash
-cd compute
-PYTHONPATH=.. uvicorn api.main:app --reload --port 19100
-```
-
-### 环境变量说明
-
-| 变量 | 默认值 | 说明 |
-|---|---|---|
-| `COMPUTE_MODE` | `real` | `real` / `simulator`，仅隔离联调用 simulator |
-| `COMPUTE_EXECUTION_MODE` | `real` | 训练执行模式 |
-| `COMPUTE_SERVICE_TOKEN` | `change_me` | 服务间认证 token |
-| `MODELTF_ROUTE_PREFIX` | `/modelTF` | API 路由前缀 |
-
-应用平台通过数据库 `compute_nodes` 表中的 `api_base_url` 和 `file_gateway_url` 主动轮询算力节点状态。
-
-## 日志
-
-后端日志模块位于 `backend/app/core/logging.py`，说明文档见：
-
-- `docs/backend-logging.md`
-
-默认输出：
-
-```text
-logs/backend-YYYY-MM-DD.log
-logs/error-YYYY-MM-DD.log
-```
-
-日志格式为 JSON Lines，单个文件不超过 20MB，只保留最近 10 天。
-
-## 主要文档
-
-- `docs/platform-architecture-requirements.md`：平台需求、功能模块、页面补全建议。
-- `docs/menu-functional-requirements.md`：当前菜单、二级路由、规划菜单、功能需求、接口和数据库映射。
-- `docs/backend-api-design.md`：FastAPI 接口分组、参数定义、权限说明。
-- `docs/postgres-schema.sql`：PostgreSQL 数据库脚本，包含权限、用户中心、多租户、审批、审计等模型。
-- `docs/system-development-plan.md`：多人协作开发计划，按前端、后端、DB、部署拆分。
-- `docs/team-development-plan.md`：3-4 人并行开发分工计划，按人员边界标注页面、接口、数据库和交付节奏。
-- `docs/first-version-development-plan.md`：当前系统主链路开发计划，覆盖前端、后端、DB、Compute API、GPU 和 LLaMA-Factory 适配。
-- `docs/backend-logging.md`：后端日志模块使用说明。
-- `docs/deployment-plan.md`：后期部署方案，覆盖单机算力服务器部署与应用/算力分离部署。
-- `docker/README.md`：Docker 部署入口，包含应用服务器和算力服务器两套 Compose 使用方式。
-
-## Docker 部署入口
-
-应用服务器：
-
-```bash
-cd docker/app
-cp .env.example .env
-docker compose up -d
-```
-
-算力服务器：
-
-```bash
-cd docker/compute
-cp .env.example .env
-docker compose up -d
-```
-
-两套 Compose 均采用代码外挂方式运行，镜像只包含运行时环境和第三方依赖。项目根目录不再保留 `Dockerfile` 和 `docker-compose.yml`，部署时统一进入 `docker/app` 或 `docker/compute` 目录执行。
-
-## 后续开发原则
-
-- 接口实现优先遵循 `docs/backend-api-design.md`。
-- 数据库实现优先遵循 `docs/postgres-schema.sql`，后续通过 Alembic 迁移管理变更。
-- 前端页面与后端接口、数据库表之间的映射以文档中的“对应页面/功能模块”为准。
-- 训练引擎适配必须通过 `compute/engines/` 下的标准接口，不在应用平台后端直接拼接训练命令。
-- 敏感信息不得写入日志，生产环境密钥通过环境变量或密钥管理系统注入。

backend/app/api/v1/endpoints/data_process.py

@@ -16,7 +16,7 @@ from dataclasses import asdict
 from pathlib import Path
 from threading import BoundedSemaphore, Lock
 from typing import Any, Literal
-from urllib.parse import quote, urlsplit
+from urllib.parse import parse_qs, quote, urlsplit
 
 import httpx
 import psycopg
@@ -676,8 +676,10 @@ def _run_generation(
                 qa_pairs_per_item=int(pairs or 1),
                 on_progress=report_progress,
             )
-        elif output_type == "reasoning":
+        elif output_type in {"reasoning", "dpo"}:
-            raise InvalidStateError("思维链输出必须配置可用的数据生成模型")
+            if output_type == "reasoning":
+                raise InvalidStateError("思维链输出必须配置可用的数据生成模型")
+            raise InvalidStateError("DPO 输出必须配置可用的数据生成模型")
         else:
             generated = generate_standard_records(
                 preview_items,
@@ -1077,7 +1079,10 @@ async def upload_source_files(
         with api_errors():
             task = store.get_task(task_id)
             process_type = str(task["process_type"])
-            if process_type == "external":
+            source_mode = str(
+                _value(task.get("config") or {}, "source_mode", "sourceMode", "local")
+            )
+            if process_type == "external" or source_mode == "external":
                 raise InvalidStateError(
                     "external tasks must import data through the external source endpoint"
                 )
@@ -1375,6 +1380,10 @@ def _external_postgres_connection(payload: ExternalSourceRequest) -> psycopg.Con
         raise fail(501, f"external data source type is not supported: {payload.type}")
     if parsed_url.username or parsed_url.password:
         raise fail(400, "database credentials must use the account and password fields")
+    if set(parse_qs(parsed_url.query)) & {
+        "password", "secret", "token", "api_key", "user", "username"
+    }:
+        raise fail(400, "database URL query must not contain credentials")
     if payload.auth_mode not in {"none", "basic"}:
         raise fail(400, "PostgreSQL supports only none or basic authentication")
     if payload.auth_mode == "basic" and not payload.username:
@@ -1413,10 +1422,14 @@ def _external_postgres_connection(payload: ExternalSourceRequest) -> psycopg.Con
                 "set DATA_PROCESS_ALLOW_PRIVATE_EXTERNAL_DB=true only in a trusted deployment",
             )
     kwargs: dict[str, Any] = {
-        "connect_timeout": 5,
+        "connect_timeout": payload.connect_timeout_seconds,
         "row_factory": dict_row,
         "application_name": "yg-ft-data-process-readonly",
-        "options": "-c default_transaction_read_only=on -c statement_timeout=30000",
+        "options": (
+            "-c default_transaction_read_only=on "
+            f"-c statement_timeout={payload.statement_timeout_seconds * 1000}"
+        ),
+        "sslmode": payload.ssl_mode,
     }
     if payload.auth_mode == "basic" and payload.username:
         kwargs["user"] = payload.username
@@ -1425,6 +1438,17 @@ def _external_postgres_connection(payload: ExternalSourceRequest) -> psycopg.Con
     return psycopg.connect(payload.url, **kwargs)
 
 
+def _assert_external_source_task(task: dict[str, Any]) -> None:
+    if str(task.get("process_type")) == "external":
+        return
+    config = task.get("config") or {}
+    source_mode = str(_value(config, "source_mode", "sourceMode", "local"))
+    if str(task.get("process_type")) != "structured" or source_mode != "external":
+        raise InvalidStateError(
+            "external source access requires a structured task with source_mode=external"
+        )
+
+
 @router.post("/{task_id}/external/test")
 def test_external_source(
     task_id: str,
@@ -1433,10 +1457,7 @@ def test_external_source(
 ) -> dict[str, Any]:
     with api_errors():
         task = store.get_task(task_id)
-        if str(task.get("process_type")) != "external":
+        _assert_external_source_task(task)
-            raise InvalidStateError(
-                "external source access requires an external data processing task"
-            )
         try:
             with _external_postgres_connection(payload) as conn:
                 conn.execute("SELECT 1 AS ok").fetchone()
@@ -1462,12 +1483,14 @@ def pull_external_source(
         raise fail(400, "a read-only SELECT or WITH query is required for external pull")
     with api_errors():
         task = store.get_task(task_id)
-        if str(task.get("process_type")) != "external":
+        _assert_external_source_task(task)
-            raise InvalidStateError("external pull requires an external data processing task")
         try:
             with _external_postgres_connection(payload) as conn:
                 conn.execute("SET TRANSACTION READ ONLY")
-                conn.execute("SET LOCAL statement_timeout = '30s'")
+                conn.execute(
+                    "SELECT set_config('statement_timeout', %s, true)",
+                    (f"{payload.statement_timeout_seconds}s",),
+                )
                 cursor = conn.execute(query)
                 rows: list[dict[str, Any]] = []
                 content_parts: list[str] = []
@@ -1518,6 +1541,9 @@ def pull_external_source(
                         "external_type": payload.type,
                         "external_host": urlsplit(payload.url).hostname,
                         "external_limit": payload.limit,
+                        "external_ssl_mode": payload.ssl_mode,
+                        "external_connect_timeout_seconds": payload.connect_timeout_seconds,
+                        "external_statement_timeout_seconds": payload.statement_timeout_seconds,
                     },
                 }
             ],
@@ -1566,7 +1592,10 @@ def _prepare_preview_items(
         for index, source in enumerate(sources):
             source_format = str(source.get("file_format") or "").lower()
             needs_structured_xlsx = not is_unstructured and source_format == "xlsx"
-            needs_layout_raw = is_unstructured and chunk_method == "layout_hybrid"
+            needs_layout_raw = (
+                is_unstructured
+                and chunk_method == "layout_hybrid"
+            )
             needs_pdf_noise = (
                 is_unstructured
                 and not needs_layout_raw
@@ -2038,6 +2067,8 @@ def restore_result(
             "instruction": current.get("original_instruction") or current.get("instruction") or "",
             "input": current.get("original_input") or current.get("input") or "",
             "output": current.get("original_output") or current.get("output") or "",
+            "chosen": current.get("original_chosen") or current.get("chosen") or "",
+            "rejected": current.get("original_rejected") or current.get("rejected") or "",
         }
         preview_id = current.get("preview_item_id")
         source_content = ""
@@ -2070,6 +2101,8 @@ def restore_result(
                 "instruction": restored["instruction"],
                 "input": restored["input"],
                 "output": restored["output"],
+                "chosen": restored["chosen"],
+                "rejected": restored["rejected"],
                 "quality_score": asdict(quality),
                 "expected_updated_at": current.get("updated_at"),
             },
@@ -2141,13 +2174,15 @@ def _generate_result_replacement(
     ).strip().lower()
     previous_instruction = str(current.get("instruction") or "")[:1000]
     previous_output = str(current.get("output") or "")[:1000]
+    previous_rejected = str(current.get("rejected") or "")[:1000]
     base_prompt = str(
         _value(config, "generation_prompt", "generationPrompt", "") or ""
     )
     regeneration_instruction = (
         "这是一次失败结果的重新生成。请使用新的提问角度和表达，"
         "不要复述旧结果。旧问题："
-        f"{previous_instruction or '无'}；旧答案：{previous_output or '无'}。"
+        f"{previous_instruction or '无'}；旧优选答案：{previous_output or '无'}；"
+        f"旧拒选答案：{previous_rejected or '无'}。"
     )
     runtime_config = {
         **config,

backend/app/api/v1/endpoints/health.py

@@ -2,6 +2,8 @@
 
 from app.core.logging import get_logger
 from app.db.platform_store import get_platform_store
+from app.core.config import get_settings
+from app.modules.storage.minio_store import ObjectStorageError, get_object_storage
 
 router = APIRouter()
 logger = get_logger(__name__)
@@ -9,10 +11,30 @@ logger = get_logger(__name__)
 
 @router.get("/health")
 async def health_check() -> dict[str, object]:
-    logger.info("health check requested")
+    # Health endpoints are called frequently by Docker and the frontend.
+    # Keep failures visible without emitting one INFO line per probe.
+    logger.debug("health check requested")
+    storage_status: dict[str, object] = {"enabled": get_settings().minio_enabled, "status": "disabled"}
+    if get_settings().minio_enabled:
+        try:
+            get_object_storage().ensure_bucket()
+            storage_status = {
+                "enabled": True,
+                "status": "ready",
+                "endpoint": get_settings().minio_endpoint,
+                "bucket": get_settings().minio_bucket,
+            }
+        except (ObjectStorageError, OSError) as exc:
+            logger.warning("MinIO health check failed: %s", exc)
+            storage_status = {
+                "enabled": True,
+                "status": "unavailable",
+                "endpoint": get_settings().minio_endpoint,
+                "error": str(exc),
+            }
     return {
         "code": 0,
         "message": "ok",
-        "data": get_platform_store().health_metrics(),
+        "data": {**get_platform_store().health_metrics(), "storage": storage_status},
     }
 

backend/app/api/v1/router.py

@@ -1,4 +1,4 @@
-from fastapi import APIRouter
+from fastapi import APIRouter
 
 from app.api.v1.endpoints.data_process import router as data_process_router
 from app.api.v1.endpoints.platform import router as platform_router
@@ -9,6 +9,8 @@ from app.modules.approval.router import router as approval_router
 from app.modules.system.router import router as system_router
 from app.modules.retention.router import router as retention_router
 from app.modules.resource.router import router as resource_router
+from app.modules.gpu.router import router as gpu_router
+from app.modules.data_convert.router import router as data_convert_router
 
 api_router = APIRouter()
 api_router.include_router(health_router, tags=["health"])
@@ -20,3 +22,5 @@ api_router.include_router(project_router, tags=["project"])
 api_router.include_router(approval_router, tags=["approval"])
 api_router.include_router(retention_router, tags=["retention"])
 api_router.include_router(resource_router, tags=["resource"])
+api_router.include_router(gpu_router, tags=["gpu-assignment"])
+api_router.include_router(data_convert_router, tags=["data-convert"])

backend/app/core/audit.py

@@ -0,0 +1,199 @@
+"""
+审计日志装饰器模块
+
+提供 @audit_log 装饰器，用于自动记录关键业务操作的审计日志。
+
+使用示例：
+    from app.core.audit import audit_log
+
+    @audit_log(action="create_dataset", target_type="dataset")
+    async def create_dataset(request: Request, ...):
+        ...
+"""
+
+from __future__ import annotations
+
+import asyncio
+import functools
+import time
+from datetime import datetime, timezone
+from typing import Any, Callable, Optional, TypeVar
+
+from fastapi import Request
+
+from app.core.logging import get_logger, request_id_var
+
+logger = get_logger("app.audit")
+
+F = TypeVar("F", bound=Callable[..., Any])
+
+
+def audit_log(
+    action: str,
+    target_type: str = "",
+    *,
+    detail_template: str = "",
+    extract_target_id: Optional[Callable[[Any], str]] = None,
+) -> Callable[[F], F]:
+    """
+    审计日志装饰器
+
+    Args:
+        action: 操作类型，如 create_dataset、update_model 等
+        target_type: 目标资源类型，如 dataset、model 等
+        detail_template: 日志详情模板（支持 format 参数）
+        extract_target_id: 从返回值中提取目标 ID 的函数
+
+    Returns:
+        装饰后的函数
+    """
+
+    def decorator(func: F) -> F:
+        if asyncio.iscoroutinefunction(func):
+            @functools.wraps(func)
+            async def async_wrapper(*args, **kwargs):
+                started_at = time.perf_counter()
+                trace_id = request_id_var.get("-")
+                try:
+                    result = await func(*args, **kwargs)
+                    elapsed_ms = (time.perf_counter() - started_at) * 1000
+                    target_id = _extract_target_id(result, kwargs, extract_target_id)
+                    detail = _build_detail(detail_template, kwargs)
+                    _record_audit(
+                        action=action,
+                        target_type=target_type,
+                        target_id=target_id,
+                        detail=detail,
+                        trace_id=trace_id,
+                        duration_ms=elapsed_ms,
+                    )
+                    return result
+                except Exception:
+                    logger.error(
+                        "审计日志记录失败 action=%s", action, exc_info=True
+                    )
+                    raise
+
+            return async_wrapper  # type: ignore
+        else:
+            @functools.wraps(func)
+            def sync_wrapper(*args, **kwargs):
+                started_at = time.perf_counter()
+                trace_id = request_id_var.get("-")
+                try:
+                    result = func(*args, **kwargs)
+                    elapsed_ms = (time.perf_counter() - started_at) * 1000
+                    target_id = _extract_target_id(result, kwargs, extract_target_id)
+                    detail = _build_detail(detail_template, kwargs)
+                    _record_audit(
+                        action=action,
+                        target_type=target_type,
+                        target_id=target_id,
+                        detail=detail,
+                        trace_id=trace_id,
+                        duration_ms=elapsed_ms,
+                    )
+                    return result
+                except Exception:
+                    logger.error(
+                        "审计日志记录失败 action=%s", action, exc_info=True
+                    )
+                    raise
+
+            return sync_wrapper  # type: ignore
+
+    return decorator
+
+
+def _extract_target_id(
+    result: Any, kwargs: dict, extractor: Optional[Callable[[Any], str]]
+) -> Optional[str]:
+    """从返回值或 kwargs 中提取目标 ID"""
+    if extractor:
+        try:
+            return extractor(result)
+        except Exception:
+            pass
+    if isinstance(result, dict):
+        return result.get("id")
+    # 尝试从路径参数中提取
+    for key in ("dataset_id", "model_id", "task_id", "resource_id"):
+        val = kwargs.get(key)
+        if val:
+            return str(val)
+    return None
+
+
+def _build_detail(template: str, kwargs: dict) -> str:
+    """构建审计详情"""
+    if not template:
+        return ""
+    try:
+        return template.format(**kwargs)
+    except (KeyError, IndexError):
+        return template
+
+
+def _record_audit(
+    action: str,
+    target_type: str,
+    target_id: Optional[str],
+    detail: str,
+    trace_id: str,
+    duration_ms: float,
+) -> None:
+    """通过已有的 record_audit 方法写入审计日志"""
+    try:
+        from app.db.platform_store import get_platform_store
+
+        store = get_platform_store()
+        store.record_audit(
+            action=action,
+            target_type=target_type or None,
+            target_id=target_id,
+            detail=f"{detail} trace_id={trace_id} duration_ms={duration_ms:.1f}" if detail else f"trace_id={trace_id} duration_ms={duration_ms:.1f}",
+        )
+    except Exception:
+        logger.error("写入审计日志失败 action=%s", action, exc_info=True)
+
+
+# ==================== 预定义的审计操作常量 ====================
+
+class AuditActions:
+    """预定义的审计操作类型"""
+    # 数据集操作
+    CREATE_DATASET = "create_dataset"
+    UPDATE_DATASET = "update_dataset"
+    DELETE_DATASET = "delete_dataset"
+
+    # 模型操作
+    CREATE_MODEL = "create_model"
+    UPDATE_MODEL = "update_model"
+    DELETE_MODEL = "delete_model"
+
+    # 微调任务
+    CREATE_FINE_TUNE = "create_fine_tune"
+    UPDATE_FINE_TUNE = "update_fine_tune"
+    DELETE_FINE_TUNE = "delete_fine_tune"
+
+    # 推理任务
+    CREATE_INFERENCE = "create_inference"
+    UPDATE_INFERENCE = "update_inference"
+    DELETE_INFERENCE = "delete_inference"
+
+    # 用户管理
+    CREATE_USER = "create_user"
+    UPDATE_USER = "update_user"
+    DELETE_USER = "delete_user"
+
+    # 租户管理
+    CREATE_TENANT = "create_tenant"
+    UPDATE_TENANT = "update_tenant"
+    DELETE_TENANT = "delete_tenant"
+
+    # 权限授权
+    GRANT_ACL = "grant_acl"
+    REVOKE_ACL = "revoke_acl"
+
+    # 系统配置
+    UPDATE_CONFIG = "update_config"

backend/app/core/auth.py

@@ -9,6 +9,13 @@ from app.db.platform_store import get_platform_store
 
 # 无需鉴权的路径前缀（健康检查、登录等）
 PUBLIC_PATHS = ("/health", "/login", "/system-info")
+OWNER_TABLES = {
+    "dataset": ("datasets", "created_by"), "model": ("models", "created_by"),
+    "trained_model": ("trained_models", "created_by"), "eval": ("eval_tasks", "created_by"),
+    "fine-tune": ("fine_tune_tasks", "payload"), "fine_tune_task": ("fine_tune_tasks", "payload"),
+    "compare": ("compare_tasks", "payload"), "inference": ("compare_tasks", "payload"),
+    "project": ("projects", "created_by"), "data_process": ("data_process_tasks", "created_by"),
+}
 
 
 def _extract_token(request: Request) -> str | None:
@@ -19,6 +26,9 @@ def _extract_token(request: Request) -> str | None:
         return token[len("platform-token-"):]
     return None
 
+def _session_token(user_id: str, session_id: str) -> str:
+    return f"platform-token-{user_id}.{session_id}"
+
 
 def get_current_user(request: Request) -> dict[str, Any]:
     """
@@ -33,14 +43,30 @@ def get_current_user(request: Request) -> dict[str, Any]:
         if path.endswith(prefix):
             return {"id": None, "username": "anonymous", "role": "viewer", "permissions": [], "protected": False}
 
-    user_id = _extract_token(request)
+    token_value = _extract_token(request)
-    if not user_id:
+    if not token_value:
         raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="missing or invalid token")
 
     store = get_platform_store()
-    for u in store.users():
+    user_id, _, session_id = token_value.partition(".")
-        if u.get("id") == user_id:
+    if session_id:
-            return u
+        with store.connect() as conn:
+            session = conn.execute(
+                "SELECT user_id, logout_at, expires_at FROM sessions WHERE id=?", (session_id,)
+            ).fetchone()
+        if not session or session["user_id"] != user_id or session["logout_at"]:
+            raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="session expired")
+        if session["expires_at"]:
+            from datetime import datetime, timezone
+            try:
+                if datetime.fromisoformat(str(session["expires_at"]).replace("Z", "+00:00")) <= datetime.now(timezone.utc):
+                    raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="session expired")
+            except ValueError:
+                pass
+    with store.connect() as conn:
+        user_row = conn.execute("SELECT * FROM users WHERE id=?", (user_id,)).fetchone()
+    if user_row:
+        return store._user(user_row)
     raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="user not found")
 
 
@@ -75,6 +101,23 @@ def has_resource_access(
     user_id = user.get("id")
     user_role = user.get("role")
 
+    owner_tables = OWNER_TABLES
+    table_info = owner_tables.get(resource_type)
+    if table_info and user_id:
+        table, column = table_info
+        with store.connect() as conn:
+            row = conn.execute(f"SELECT {column} FROM {table} WHERE id=?", (resource_id,)).fetchone()
+        if row:
+            owner = row[column]
+            if column == "payload":
+                try:
+                    import json
+                    owner = json.loads(owner or "{}").get("created_by")
+                except (TypeError, ValueError):
+                    owner = None
+            if owner == user_id:
+                return True
+
     for entry in acls:
         # 按 user 授权
         if entry.get("principal_type") == "user" and entry.get("principal_id") == user_id:
@@ -135,4 +178,63 @@ def filter_accessible_resource_ids(
         ).fetchall()
 
     accessible = {r["resource_id"] for r in rows}
+    if resource_type in OWNER_TABLES:
+        table, column = OWNER_TABLES[resource_type]
+        if column == "payload":
+            # payload 是 JSON 字符串，需要查出后解析 created_by
+            with store.connect() as conn:
+                owned = conn.execute(f"SELECT id, {column} FROM {table}").fetchall()
+            for row in owned:
+                try:
+                    import json
+                    payload = json.loads(row[column] or "{}")
+                    if payload.get("created_by") == user_id:
+                        accessible.add(row["id"])
+                except (TypeError, ValueError):
+                    pass
+        else:
+            with store.connect() as conn:
+                owned = conn.execute(f"SELECT id FROM {table} WHERE {column}=?", (user_id,)).fetchall()
+            accessible.update(row["id"] for row in owned)
     return [rid for rid in all_ids if rid in accessible]
+
+
+def filter_accessible_resource_ids_batch(
+    resource_type: str,
+    resource_ids: list[str],
+    user: dict[str, Any],
+) -> set[str]:
+    """Filter a list endpoint with one ACL query instead of one query per row."""
+    if is_admin(user):
+        return set(resource_ids)
+    if not resource_ids:
+        return set()
+    store = get_platform_store()
+    placeholders = ",".join("?" for _ in resource_ids)
+    with store.connect() as conn:
+        rows = conn.execute(
+            f"SELECT DISTINCT resource_id FROM acls WHERE resource_type=? AND resource_id IN ({placeholders}) "
+            "AND ((principal_type='user' AND principal_id=?) OR (principal_type='role' AND principal_id=?))",
+            (resource_type, *resource_ids, user.get("id"), user.get("role")),
+        ).fetchall()
+    accessible = {row["resource_id"] for row in rows}
+    table_info = OWNER_TABLES.get(resource_type)
+    if table_info and user.get("id"):
+        table, column = table_info
+        with store.connect() as conn:
+            owned = conn.execute(
+                f"SELECT id, {column} FROM {table} WHERE id IN ({placeholders})",
+                (*resource_ids,),
+            ).fetchall()
+        for row in owned:
+            owner = row[column]
+            # 如果列是 payload（JSON），需要解析后提取 created_by
+            if column == "payload":
+                try:
+                    import json
+                    owner = json.loads(owner or "{}").get("created_by")
+                except (TypeError, ValueError):
+                    owner = None
+            if owner == user["id"]:
+                accessible.add(row["id"])
+    return accessible

backend/app/core/config.py

@@ -1,6 +1,7 @@
 from dataclasses import dataclass
 from functools import lru_cache
 import os
+from typing import Any
 
 try:
     from pathlib import Path as _Path
@@ -29,6 +30,24 @@ def _list_env(name: str, default: list[str]) -> list[str]:
     return [item.strip() for item in raw.split(",") if item.strip()]
 
 
+def _bool_env(name: str, default: bool) -> bool:
+    raw = os.getenv(name)
+    if raw is None or raw.strip() == "":
+        return default
+    return raw.strip().lower() in {"1", "true", "yes", "on"}
+
+
+def docs_kwargs(enabled: bool) -> dict[str, Any]:
+    """Swagger UI / ReDoc / OpenAPI schema 路由开关。
+
+    关闭时 FastAPI 不注册 /docs、/redoc、/openapi.json，访问一律返回 404，
+    避免未授权访问泄露 API 结构。
+    """
+    if enabled:
+        return {}
+    return {"docs_url": None, "redoc_url": None, "openapi_url": None}
+
+
 @dataclass(frozen=True)
 class Settings:
     app_name: str = os.getenv("APP_NAME", "YG Fine-Tune Platform API")
@@ -41,6 +60,16 @@ class Settings:
     compute_status_sync_mode: str = os.getenv("COMPUTE_STATUS_SYNC_MODE", "polling")
     compute_poll_interval_seconds: int = _int_env("COMPUTE_POLL_INTERVAL_SECONDS", 3)
     compute_request_timeout_seconds: int = _int_env("COMPUTE_REQUEST_TIMEOUT_SECONDS", 5)
+    minio_enabled: bool = _bool_env("MINIO_ENABLED", False)
+    # MinIO is an independent service and may run on another host. The
+    # endpoint must therefore be reachable from the Backend container.
+    minio_endpoint: str = os.getenv("MINIO_ENDPOINT", "http://host.docker.internal:19000")
+    minio_access_key: str = os.getenv("MINIO_ACCESS_KEY", "minioadmin")
+    minio_secret_key: str = os.getenv("MINIO_SECRET_KEY", "minioadmin")
+    minio_bucket: str = os.getenv("MINIO_BUCKET", "yg-ft-resources")
+    minio_secure: bool = _bool_env("MINIO_SECURE", False)
+    storage_wait_seconds: int = _int_env("STORAGE_WAIT_SECONDS", 300)
+    storage_check_interval_seconds: int = _int_env("STORAGE_CHECK_INTERVAL_SECONDS", 10)
     compute_service_token: str = os.getenv("COMPUTE_SERVICE_TOKEN", "")
     log_level: str = os.getenv("LOG_LEVEL", "INFO")
     log_dir: str = os.getenv("LOG_DIR", "./logs")
@@ -48,6 +77,7 @@ class Settings:
     log_error_file_prefix: str = os.getenv("LOG_ERROR_FILE_PREFIX", "error")
     log_max_bytes: int = _int_env("LOG_MAX_BYTES", 20 * 1024 * 1024)
     log_retention_days: int = _int_env("LOG_RETENTION_DAYS", 10)
+    enable_docs: bool = None  # type: ignore[assignment]
 
     def __post_init__(self) -> None:
         object.__setattr__(
@@ -63,6 +93,15 @@ class Settings:
                 ],
             ),
         )
+        # Swagger UI / ReDoc / OpenAPI 文档路由开关：
+        # 未显式配置 ENABLE_DOCS 时，仅本地/开发环境开放，生产环境默认关闭，
+        # 避免未授权访问泄露 API 结构。从运行时环境读取 APP_ENV，而非类定义时
+        # 缓存的默认值，保证生产默认关闭始终生效且便于测试。
+        object.__setattr__(
+            self,
+            "enable_docs",
+            _bool_env("ENABLE_DOCS", os.getenv("APP_ENV", "local") != "prod"),
+        )
 
 
 @lru_cache

backend/app/core/logging.py

@@ -4,11 +4,12 @@ from contextvars import ContextVar
 from datetime import date, datetime, timedelta
 import json
 import logging
+import sys
 from logging import Handler, LogRecord
 from pathlib import Path
 import re
 import time
-from typing import Any
+from typing import Any, Callable, Optional
 from uuid import uuid4
 
 from fastapi import FastAPI, Request
@@ -17,6 +18,74 @@ from app.core.config import Settings, get_settings
 
 request_id_var: ContextVar[str] = ContextVar("request_id", default="-")
 
+# ==================== 敏感数据脱敏规则 ====================
+
+SENSITIVE_PATTERNS: dict[str, Callable | str] = {
+    "token": "***",
+    "password": "***",
+    "access_token": "***",
+    "refresh_token": "***",
+    "secret_key": "***",
+    "authorization": "***",
+    "bearer": "***",
+    "api_key": "***",
+    "private_key": "***",
+}
+
+def mask_value(key: str, value: Any) -> str:
+    """对单个值进行脱敏处理"""
+    if value is None:
+        return ""
+    str_val = str(value)
+    
+    handler = SENSITIVE_PATTERNS.get(key)
+    if callable(handler):
+        return handler(str_val)
+    elif isinstance(handler, str):
+        # 支持正则替换模式，如 r"1\d{3}\d{4}"
+        try:
+            return re.sub(handler, "***", str_val)
+        except re.error:
+            return "***"
+    return handler
+
+
+def mask_sensitive_dict(data: dict) -> dict:
+    """递归脱敏字典中的敏感字段"""
+    if not data or not isinstance(data, dict):
+        return data
+    
+    result = {}
+    for key, value in data.items():
+        result[key] = mask_value(key, value)
+    return result
+
+
+def mask_sensitive_string(text: str) -> str:
+    """从文本中脱敏常见敏感信息"""
+    if not text:
+        return text
+    
+    patterns = [
+        (r'Bearer\s+[A-Za-z0-9\-._]+', '***'),
+        (r'token\s*[:=]\s*', '***'),
+        (r'password\s*[:=]\s*', '***'),
+        (r'secret[_-]?key\s*[:=]', '***'),
+        (r'api[-_]?key\s*[:=]', '***'),
+        (r'private[_-]?key\s*[:=]', '***'),
+        (r'\d{11}', r'\d{3}\*\d{4}'),  # 手机号/身份证
+        (r'1[3-9]\d{9}', r'1\*{3}\*{4}'),  # 手机号
+    ]
+    
+    for pattern, replacement in patterns:
+        try:
+            text = re.sub(pattern, replacement, text, flags=re.IGNORECASE)
+        except re.error:
+            pass
+    return text
+
+
+# ==================== RequestId Filter ====================
 
 class RequestIdFilter(logging.Filter):
     def filter(self, record: LogRecord) -> bool:
@@ -24,8 +93,30 @@ class RequestIdFilter(logging.Filter):
         return True
 
 
+# ==================== Enhanced JSON Formatter ====================
+
 class JsonLogFormatter(logging.Formatter):
-    """Format one JSON object per line for ELK/Filebeat collection."""
+    """
+    增强的 JSON 日志格式化器，支持结构化字段输出。
+    
+    输出示例：
+    {
+      "@timestamp": "2026-08-17T18:30:00.123Z",
+      "level": "INFO",
+      "logger": "dataset.router",
+      "message": "数据集创建成功",
+      "module": "dataset.router",
+      "function": "create_dataset",
+      "file": "dataset/router.py",
+      "line": 45,
+      "process": 12345,
+      "thread": "MainThread",
+      "request_id": "req-abc123",
+      "user_id": "u_admin",
+      "client_ip": "192.168.1.100",
+      "extra": {...}
+    }
+    """
 
     def format(self, record: LogRecord) -> str:
         payload: dict[str, Any] = {
@@ -44,13 +135,26 @@ class JsonLogFormatter(logging.Formatter):
             "thread_name": record.threadName,
             "request_id": getattr(record, "request_id", "-"),
         }
+        
+        # 从 record 中提取额外字段（通过 extra 参数传入）
+        for attr in ("user_id", "client_ip", "target_type", "target_id", 
+                    "duration_ms", "status_code", "error"):
+            val = getattr(record, attr, None)
+            if val is not None:
+                payload[attr] = val
+        
+        # 处理异常信息
         if record.exc_info:
             payload["exception"] = self.formatException(record.exc_info)
         if record.stack_info:
             payload["stack"] = self.formatStack(record.stack_info)
+            
         return json.dumps(payload, ensure_ascii=False, separators=(",", ":"))
 
 
+# ==================== DateSizeRotatingFileHandler ====================
+# （保持不变，已有实现）
+
 class DateSizeRotatingFileHandler(Handler):
     """Rotate log files by date and size while keeping date in every file name."""
 
@@ -160,6 +264,157 @@ class DateSizeRotatingFileHandler(Handler):
                 path.unlink(missing_ok=True)
 
 
+# ==================== Structured Logger 封装 ====================
+
+class StructuredLogger:
+    """
+    结构化日志记录器，提供统一的日志接口。
+    
+    使用方式：
+        logger = get_structured_logger('dataset.router')
+        logger.info('创建数据集', dataset_id='ds_123')
+    """
+    
+    def __init__(self, name: str, module: str = ""):
+        self.logger = logging.getLogger(name)
+        self.name = name
+        self.module = module
+        
+    @property
+    def trace_id(self) -> str:
+        return request_id_var.get("-")
+    
+    def info(self, message: str, **extra: Any) -> None:
+        self._log("INFO", message, **extra)
+
+    def warning(self, message: str, **extra: Any) -> None:
+        self._log("WARNING", message, **extra)
+
+    def error(self, message: str, **extra: Any) -> None:
+        self._log("ERROR", message, **extra)
+
+    def debug(self, message: str, **extra: Any) -> None:
+        self._log("DEBUG", message, **extra)
+
+    def _log(self, level: str, message: str, **extra: Any) -> None:
+        """统一日志记录方法"""
+        log_entry: dict[str, Any] = {
+            "timestamp": datetime.utcnow().isoformat(),
+            "level": level,
+            "logger": self.name,
+            "module": self.module,
+            "message": message,
+            "trace_id": self.trace_id,
+            "extra": extra,
+        }
+        self.logger.log(getattr(logging, level, logging.INFO), json.dumps(log_entry, ensure_ascii=False, default=str))
+
+
+def get_structured_logger(name: str, module: str = "") -> StructuredLogger:
+    """获取结构化日志记录器"""
+    return StructuredLogger(name, module)
+
+
+# ==================== 快捷函数 ====================
+
+def get_logger(name: str) -> logging.Logger:
+    """获取标准 Python logger"""
+    return logging.getLogger(name)
+
+
+def set_request_id(request_id: str) -> None:
+    """设置当前请求的追踪 ID"""
+    request_id_var.set(request_id)
+
+
+def setup_request_logging(app: FastAPI) -> None:
+    """配置 FastAPI 请求日志中间件"""
+    logger = get_logger("app.access")
+
+    @app.middleware("http")
+    async def request_logging_middleware(request: Request, call_next):  # type: ignore[no-untyped-def]
+        request_id = request.headers.get("X-Request-ID") or str(uuid4())
+        token = request_id_var.set(request_id)
+        started_at = time.perf_counter()
+        try:
+            response = await call_next(request)
+            elapsed_ms = (time.perf_counter() - started_at) * 1000
+
+            noisy_paths = ("/health", "/system-info", "/compute/jobs/", "/model-eval/", "/model-compare/")
+            log_method = logger.debug if request.method == "GET" and response.status_code < 400 else logger.info
+            if any(request.url.path.endswith(path) or path in request.url.path for path in noisy_paths) and response.status_code < 400:
+                log_method = logger.debug
+            if response.status_code >= 400:
+                log_method = logger.warning
+            log_method(
+                "request completed method=%s path=%s status_code=%s duration_ms=%.2f client=%s",
+                request.method,
+                request.url.path,
+                response.status_code,
+                elapsed_ms,
+                request.client.host if request.client else "-",
+            )
+
+            # 5xx 系统错误自动写入操作日志（未被 @op_log 覆盖的系统级异常）
+            if response.status_code >= 500:
+                try:
+                    from app.core.op_log import log_operation, OpModule, OpStatus
+                    log_operation(
+                        module=OpModule.SYSTEM,
+                        action="request",
+                        target_type="api",
+                        target_name=request.url.path,
+                        status=OpStatus.FAILURE,
+                        error_message=f"HTTP {response.status_code} - 系统内部错误",
+                        error_type="HTTPError",
+                        detail=f'{{"method":"{request.method}","path":"{request.url.path}","status":{response.status_code}}}',
+                        func_name="request_logging_middleware",
+                        request=request,
+                        duration_ms=elapsed_ms,
+                    )
+                except Exception:
+                    pass  # 日志写入失败不影响主流程
+
+            response.headers["X-Request-ID"] = request_id
+            return response
+        except Exception:
+            elapsed_ms = (time.perf_counter() - started_at) * 1000
+            logger.exception(
+                "request failed method=%s path=%s duration_ms=%.2f client=%s",
+                request.method,
+                request.url.path,
+                elapsed_ms,
+                request.client.host if request.client else "-",
+            )
+
+            # 未被捕获的异常，写入操作日志
+            try:
+                import traceback as _tb
+                from app.core.op_log import log_operation, OpModule, OpStatus
+                log_operation(
+                    module=OpModule.SYSTEM,
+                    action="request",
+                    target_type="api",
+                    target_name=request.url.path,
+                    status=OpStatus.FAILURE,
+                    error_message=str(sys.exc_info()[1])[:1000] if sys.exc_info()[1] else "未知异常",
+                    error_type=type(sys.exc_info()[1]).__name__ if sys.exc_info()[1] else "UnknownError",
+                    error_traceback="".join(_tb.format_exception(*sys.exc_info()))[:5000],
+                    func_name="request_logging_middleware",
+                    detail=f'{{"method":"{request.method}","path":"{request.url.path}"}}',
+                    request=request,
+                    duration_ms=elapsed_ms,
+                )
+            except Exception:
+                pass  # 日志写入失败不影响主流程
+
+            raise
+        finally:
+            request_id_var.reset(token)
+
+
+# ==================== 配置函数 ====================
+
 def configure_logging(settings: Settings | None = None) -> None:
     settings = settings or get_settings()
 
@@ -209,45 +464,5 @@ def configure_logging(settings: Settings | None = None) -> None:
         logger.handlers.clear()
         logger.propagate = True
 
-
+    logging.getLogger("uvicorn.access").setLevel(logging.WARNING)
-def get_logger(name: str) -> logging.Logger:
+    logging.getLogger("psycopg.pool").setLevel(logging.ERROR)
-    return logging.getLogger(name)
-
-
-def set_request_id(request_id: str) -> None:
-    request_id_var.set(request_id)
-
-
-def setup_request_logging(app: FastAPI) -> None:
-    logger = get_logger("app.access")
-
-    @app.middleware("http")
-    async def request_logging_middleware(request: Request, call_next):  # type: ignore[no-untyped-def]
-        request_id = request.headers.get("X-Request-ID") or str(uuid4())
-        token = request_id_var.set(request_id)
-        started_at = time.perf_counter()
-        try:
-            response = await call_next(request)
-            elapsed_ms = (time.perf_counter() - started_at) * 1000
-            logger.info(
-                "request completed method=%s path=%s status_code=%s duration_ms=%.2f client=%s",
-                request.method,
-                request.url.path,
-                response.status_code,
-                elapsed_ms,
-                request.client.host if request.client else "-",
-            )
-            response.headers["X-Request-ID"] = request_id
-            return response
-        except Exception:
-            elapsed_ms = (time.perf_counter() - started_at) * 1000
-            logger.exception(
-                "request failed method=%s path=%s duration_ms=%.2f client=%s",
-                request.method,
-                request.url.path,
-                elapsed_ms,
-                request.client.host if request.client else "-",
-            )
-            raise
-        finally:
-            request_id_var.reset(token)

backend/app/core/op_log.py

@@ -0,0 +1,382 @@
+"""
+操作日志工具模块
+
+提供 @op_log 装饰器和 log_operation 函数，用于记录用户在各业务模块的详细操作。
+自动捕获成功/失败状态、完整报错堆栈、操作耗时等。
+
+核心设计：
+  - 失败操作必须清晰记录完整异常堆栈（traceback）
+  - 记录异常类型（如 RuntimeError / ValueError / ConnectionError）
+  - 记录具体出错的函数名和文件位置，方便定位 bug
+  - 记录 HTTP 状态码，方便区分用户错误(4xx)和系统错误(5xx)
+
+使用示例：
+    from app.core.op_log import op_log, OpModule, OpAction
+
+    @router.post("/inference/start")
+    @op_log(module=OpModule.INFERENCE, action=OpAction.START, target_type="inference")
+    async def start_inference(...):
+        ...
+"""
+
+from __future__ import annotations
+
+import asyncio
+import functools
+import json
+import time
+import traceback
+from datetime import datetime, timezone
+from typing import Any, Callable, Optional, TypeVar
+
+from fastapi import Request
+
+from app.core.logging import get_logger, request_id_var
+from app.db.platform_store import get_platform_store, new_id, utcnow
+
+logger = get_logger("app.op_log")
+
+F = TypeVar("F", bound=Callable[..., Any])
+
+
+class OpModule:
+    """业务模块常量"""
+    FINE_TUNE = "fine-tune"           # 模型训练
+    MODEL_EVAL = "model-eval"         # 模型评测
+    INFERENCE = "model-inference"     # 模型推理
+    MODEL_MANAGE = "model-manage"     # 模型管理
+    DATASET = "dataset"               # 数据集
+    DATA_PROCESS = "data-process"    # 数据处理
+    DATA_CONVERT = "data-convert"     # 数据类型转换
+    COMPUTE = "compute"              # 算力节点
+    SYSTEM = "system"                # 系统
+
+
+class OpAction:
+    """操作动作常量"""
+    CREATE = "create"
+    UPDATE = "update"
+    DELETE = "delete"
+    START = "start"
+    STOP = "stop"
+    UPLOAD = "upload"
+    DOWNLOAD = "download"
+    CONVERT = "convert"
+    MERGE = "merge"
+    IMPORT = "import"
+    LOGIN = "login"
+    LOGOUT = "logout"
+    PUBLISH = "publish"
+    RETRY = "retry"
+
+
+class OpStatus:
+    """操作状态常量"""
+    SUCCESS = "success"
+    FAILURE = "failure"
+
+
+def op_log(
+    module: str,
+    action: str,
+    target_type: str = "",
+    *,
+    target_name_param: str = "name",
+    detail_params: Optional[list[str]] = None,
+) -> Callable[[F], F]:
+    """
+    操作日志装饰器
+
+    自动记录：
+    - 谁在什么时间操作了什么
+    - 成功还是失败
+    - 失败时记录完整异常堆栈(traceback)、异常类型、异常消息
+    - 出错的函数名和文件位置，方便定位 bug
+    - 操作耗时(ms)
+    - 客户端 IP、请求路径
+
+    Args:
+        module: 业务模块（OpModule 常量）
+        action: 操作动作（OpAction 常量）
+        target_type: 资源类型
+        target_name_param: 从 kwargs 中提取目标名称的参数名
+        detail_params: 需要记录到 detail 的参数名列表
+    """
+    def decorator(func: F) -> F:
+        func_name = f"{func.__module__}.{func.__qualname__}"
+
+        if asyncio.iscoroutinefunction(func):
+            @functools.wraps(func)
+            async def async_wrapper(*args, **kwargs):
+                started_at = time.perf_counter()
+                trace_id = request_id_var.get("-")
+
+                user = _extract_user(args, kwargs)
+                request = _extract_request(args)
+
+                target_name = _get_param(kwargs, target_name_param, "")
+                target_id = _get_param(kwargs, "task_id", "") or _get_param(kwargs, "dataset_id", "") or _get_param(kwargs, "model_id", "")
+
+                detail_dict = _build_detail(detail_params, kwargs)
+                detail_str = json.dumps(detail_dict, ensure_ascii=False) if detail_dict else ""
+
+                try:
+                    result = await func(*args, **kwargs)
+                    elapsed_ms = (time.perf_counter() - started_at) * 1000
+
+                    if not target_id and isinstance(result, dict):
+                        target_id = str(result.get("id", ""))
+
+                    _write_log(
+                        module=module,
+                        action=action,
+                        target_type=target_type,
+                        target_id=str(target_id) if target_id else None,
+                        target_name=str(target_name) if target_name else None,
+                        status=OpStatus.SUCCESS,
+                        error_message="",
+                        error_type="",
+                        error_traceback="",
+                        func_name=func_name,
+                        detail=detail_str,
+                        user=user,
+                        request=request,
+                        trace_id=trace_id,
+                        duration_ms=elapsed_ms,
+                    )
+                    return result
+
+                except Exception as exc:
+                    elapsed_ms = (time.perf_counter() - started_at) * 1000
+
+                    # 捕获完整异常堆栈
+                    tb_lines = traceback.format_exception(type(exc), exc, exc.__traceback__)
+                    full_traceback = "".join(tb_lines)
+                    error_msg = str(exc)[:1000]
+                    error_type = type(exc).__name__
+
+                    _write_log(
+                        module=module,
+                        action=action,
+                        target_type=target_type,
+                        target_id=str(target_id) if target_id else None,
+                        target_name=str(target_name) if target_name else None,
+                        status=OpStatus.FAILURE,
+                        error_message=error_msg,
+                        error_type=error_type,
+                        error_traceback=full_traceback,
+                        func_name=func_name,
+                        detail=detail_str,
+                        user=user,
+                        request=request,
+                        trace_id=trace_id,
+                        duration_ms=elapsed_ms,
+                    )
+                    raise
+
+            return async_wrapper  # type: ignore
+        else:
+            @functools.wraps(func)
+            def sync_wrapper(*args, **kwargs):
+                started_at = time.perf_counter()
+                trace_id = request_id_var.get("-")
+
+                user = _extract_user(args, kwargs)
+                request = _extract_request(args)
+
+                target_name = _get_param(kwargs, target_name_param, "")
+                target_id = _get_param(kwargs, "task_id", "") or _get_param(kwargs, "dataset_id", "") or _get_param(kwargs, "model_id", "")
+
+                detail_dict = _build_detail(detail_params, kwargs)
+                detail_str = json.dumps(detail_dict, ensure_ascii=False) if detail_dict else ""
+
+                try:
+                    result = func(*args, **kwargs)
+                    elapsed_ms = (time.perf_counter() - started_at) * 1000
+
+                    if not target_id and isinstance(result, dict):
+                        target_id = str(result.get("id", ""))
+
+                    _write_log(
+                        module=module,
+                        action=action,
+                        target_type=target_type,
+                        target_id=str(target_id) if target_id else None,
+                        target_name=str(target_name) if target_name else None,
+                        status=OpStatus.SUCCESS,
+                        error_message="",
+                        error_type="",
+                        error_traceback="",
+                        func_name=func_name,
+                        detail=detail_str,
+                        user=user,
+                        request=request,
+                        trace_id=trace_id,
+                        duration_ms=elapsed_ms,
+                    )
+                    return result
+
+                except Exception as exc:
+                    elapsed_ms = (time.perf_counter() - started_at) * 1000
+
+                    tb_lines = traceback.format_exception(type(exc), exc, exc.__traceback__)
+                    full_traceback = "".join(tb_lines)
+                    error_msg = str(exc)[:1000]
+                    error_type = type(exc).__name__
+
+                    _write_log(
+                        module=module,
+                        action=action,
+                        target_type=target_type,
+                        target_id=str(target_id) if target_id else None,
+                        target_name=str(target_name) if target_name else None,
+                        status=OpStatus.FAILURE,
+                        error_message=error_msg,
+                        error_type=error_type,
+                        error_traceback=full_traceback,
+                        func_name=func_name,
+                        detail=detail_str,
+                        user=user,
+                        request=request,
+                        trace_id=trace_id,
+                        duration_ms=elapsed_ms,
+                    )
+                    raise
+
+            return sync_wrapper  # type: ignore
+
+    return decorator
+
+
+def log_operation(
+    *,
+    module: str,
+    action: str,
+    target_type: str = "",
+    target_id: str = "",
+    target_name: str = "",
+    status: str = OpStatus.SUCCESS,
+    error_message: str = "",
+    error_type: str = "",
+    error_traceback: str = "",
+    detail: str = "",
+    func_name: str = "",
+    user: Optional[dict] = None,
+    request: Optional[Request] = None,
+    duration_ms: float = 0,
+) -> None:
+    """手动记录操作日志（不方便用装饰器时使用）"""
+    trace_id = request_id_var.get("-")
+    _write_log(
+        module=module,
+        action=action,
+        target_type=target_type,
+        target_id=target_id or None,
+        target_name=target_name or None,
+        status=status,
+        error_message=error_message,
+        error_type=error_type,
+        error_traceback=error_traceback,
+        func_name=func_name,
+        detail=detail,
+        user=user,
+        request=request,
+        trace_id=trace_id,
+        duration_ms=duration_ms,
+    )
+
+
+def _build_detail(detail_params: Optional[list[str]], kwargs: dict) -> dict:
+    """从 kwargs 中提取需要记录的参数"""
+    detail_dict = {}
+    if detail_params:
+        for p in detail_params:
+            val = kwargs.get(p)
+            if val is not None:
+                detail_dict[p] = str(val)[:200]
+    return detail_dict
+
+
+def _extract_user(args: tuple, kwargs: dict) -> Optional[dict]:
+    """从函数参数中提取 current_user dict"""
+    for arg in args:
+        if isinstance(arg, dict) and "id" in arg and "username" in arg:
+            return arg
+    for v in kwargs.values():
+        if isinstance(v, dict) and "id" in v and "username" in v:
+            return v
+    return None
+
+
+def _extract_request(args: tuple) -> Optional[Request]:
+    """从函数参数中提取 Request 对象"""
+    for arg in args:
+        if isinstance(arg, Request):
+            return arg
+    return None
+
+
+def _get_param(kwargs: dict, key: str, default: str = "") -> str:
+    """安全获取参数值"""
+    val = kwargs.get(key, default)
+    if val is None:
+        return default
+    return str(val)
+
+
+def _write_log(
+    module: str,
+    action: str,
+    target_type: str,
+    target_id: Optional[str],
+    target_name: Optional[str],
+    status: str,
+    error_message: str,
+    error_type: str,
+    error_traceback: str,
+    func_name: str,
+    detail: str,
+    user: Optional[dict],
+    request: Optional[Request],
+    trace_id: str,
+    duration_ms: float,
+) -> None:
+    """写入操作日志到数据库"""
+    try:
+        store = get_platform_store()
+        log_id = new_id("op")
+        user_id = user.get("id") if user else None
+        username = user.get("username") if user else None
+        client_ip = None
+        req_method = None
+        req_path = None
+        if request:
+            client_ip = request.client.host if request.client else None
+            req_method = request.method
+            req_path = request.url.path
+
+        with store.connect() as conn:
+            conn.execute(
+                """
+                INSERT INTO operation_logs
+                (id, user_id, username, module, action, target_type, target_id,
+                 target_name, status, error_message, error_type, error_traceback,
+                 func_name, detail, client_ip,
+                 request_method, request_path, trace_id, duration_ms, create_time)
+                VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+                """,
+                (
+                    log_id, user_id, username, module, action,
+                    target_type or None, target_id, target_name,
+                    status,
+                    error_message[:1000] if error_message else None,
+                    error_type or None,
+                    error_traceback[:5000] if error_traceback else None,
+                    func_name or None,
+                    detail[:2000] if detail else None,
+                    client_ip, req_method, req_path, trace_id,
+                    round(duration_ms, 2), utcnow(),
+                ),
+            )
+    except Exception:
+        logger.error("写入操作日志失败 module=%s action=%s", module, action, exc_info=True)

backend/app/db/platform_store.py

@@ -239,10 +239,18 @@ def llama_dataset_info(dataset_key: str, file_names: list[str], formatting: str
     fmt = str(formatting).lower()
     for key, file_name in zip(llama_dataset_keys(dataset_key, file_names), file_names):
         if fmt == "sharegpt":
+            # 平台校验按 OpenAI 风格消息（role/content），故 tags 用 role/content
+            # 与 LLaMA-Factory 默认的 from/value 不同，需显式声明避免解析失败。
             result[key] = {
                 "file_name": file_name,
                 "formatting": "sharegpt",
                 "columns": {"messages": "messages"},
+                "tags": {
+                    "role_tag": "role",
+                    "content_tag": "content",
+                    "user_tag": "user",
+                    "assistant_tag": "assistant",
+                },
             }
         elif fmt == "dpo":
             result[key] = {
@@ -273,6 +281,47 @@ def llama_dataset_info(dataset_key: str, file_names: list[str], formatting: str
     return result
 
 
+def _sniff_dataset_format(sample_text: str, max_samples: int = 20) -> str:
+    """嗅探数据集内容格式（兼容 jsonl），返回 sharegpt / dpo / cpt / alpaca。
+
+    按内容而非文件名判断，纯 jsonl 数据集（如 ShareGPT messages、缺省 input 的
+    Alpaca）都能被正确识别，避免训练任务误按 alpaca 解析而失败。
+    """
+    text = (sample_text or "").strip()
+    if not text:
+        return ""
+    records: list[dict[str, Any]] = []
+    try:
+        value = json.loads(text)
+    except (TypeError, ValueError, json.JSONDecodeError):
+        value = None
+    if isinstance(value, list):
+        records = [item for item in value[:max_samples] if isinstance(item, dict)]
+    elif isinstance(value, dict):
+        records = [value]
+    else:
+        for line in text.splitlines()[:max_samples]:
+            line = line.strip()
+            if not line:
+                continue
+            try:
+                obj = json.loads(line)
+            except json.JSONDecodeError:
+                continue
+            if isinstance(obj, dict):
+                records.append(obj)
+    records = records[:max_samples]
+    if not records:
+        return ""
+    if all("messages" in record for record in records):
+        return "sharegpt"
+    if all(record.get("chosen") and record.get("rejected") for record in records):
+        return "dpo"
+    if all(record.get("text") and not (record.get("instruction") or record.get("output")) for record in records):
+        return "cpt"
+    return "alpaca"
+
+
 PASSWORD_HASH_ITERATIONS = 390_000
 
 
@@ -378,7 +427,7 @@ class PlatformStore:
         # request (notably expensive against the remote PostgreSQL instance).
         # TCP keepalive 让操作系统持续保活连接，抵抗远程库空闲静默断连。
         pool_kwargs = {
-            "connect_timeout": 5,
+            "connect_timeout": 30,
             "keepalives": 1,
             "keepalives_idle": 10,
             "keepalives_interval": 5,
@@ -388,14 +437,14 @@ class PlatformStore:
             conninfo=self.database_url,
             kwargs=pool_kwargs,
             min_size=2,
-            max_size=10,
+            max_size=20,
             # 借出前校验连接可用性，避免执行 SQL 时才发现 [BAD] 再重建。
             check=ConnectionPool.check_connection,
             # 不主动回收空闲连接（远程库约 10s 断，由 keepalive 维持），
             # 减少无谓的重建握手。
             max_idle=0,
-            # 请求最多排队等待 5s，避免雪崩时无限堆积。
+            # 请求最多排队等待，调大以适应远程库慢查询。
-            max_waiting=16,
+            max_waiting=50,
             open=False,
         )
         # 注意：不要在此调用 pool.wait()，它会阻塞等待 min_size 个连接就绪，
@@ -456,6 +505,20 @@ class PlatformStore:
             )
             self._ensure_columns(conn, "gpus", {"last_seen_at": "TEXT"})
             self._ensure_columns(conn, "fine_tune_tasks", {"compute_job_id": "TEXT"})
+            self._ensure_columns(
+                conn,
+                "sessions",
+                {
+                    "username": "TEXT",
+                    "login_at": "TEXT",
+                    "logout_at": "TEXT",
+                    "duration_seconds": "INTEGER",
+                    "issued_at": "TEXT",
+                    "expires_at": "TEXT",
+                    "ip": "TEXT",
+                    "create_time": "TEXT",
+                },
+            )
             self._ensure_columns(
                 conn,
                 "trained_models",
@@ -463,8 +526,15 @@ class PlatformStore:
                     "artifact_dir": "TEXT",
                     "compute_node_id": "TEXT",
                     "compute_node_name": "TEXT",
+                    "created_by": "TEXT",
+                    "tenant_id": "TEXT",
+                    "project_id": "TEXT",
+                    "deleted_at": "TEXT",
+                    "deleted_by": "TEXT",
                 },
             )
+            for table in ("models", "datasets", "eval_tasks"):
+                self._ensure_columns(conn, table, {"deleted_at": "TEXT", "deleted_by": "TEXT", "tenant_id": "TEXT", "project_id": "TEXT", "created_by": "TEXT"})
             self._ensure_columns(
                 conn,
                 "resource_replicas",
@@ -476,10 +546,46 @@ class PlatformStore:
                 },
             )
             schema_dir = Path(__file__).with_name("sql")
-            for extra in ("002_governance.sql", "003_tenant_quota.sql"):
+            for extra in (
+                "002_governance.sql",
+                "003_model_path_governance.sql",
+                "003_tenant_quota.sql",
+                "004_permissions.sql",
+            ):
                 extra_path = schema_dir / extra
                 if extra_path.exists():
                     conn.executescript(extra_path.read_text(encoding="utf-8"))
+            # data_convert_tasks 表补充 created_by 字段（用于数据隔离）
+            self._ensure_columns(conn, "data_convert_tasks", {"created_by": "TEXT"})
+            # 修复历史数据：将 data_convert_tasks.created_by 回填到关联的 datasets 记录
+            try:
+                conn.execute("""
+                    UPDATE datasets SET created_by = dct.created_by
+                    FROM data_convert_tasks dct
+                    WHERE datasets.task_id = dct.id
+                      AND datasets.source = 'upload'
+                      AND (datasets.created_by IS NULL OR datasets.created_by = '')
+                      AND dct.created_by IS NOT NULL
+                      AND dct.deleted_at IS NULL
+                """)
+            except Exception:
+                pass  # 列不存在时忽略
+            # 修复历史数据：为 eval_tasks 表回填 created_by（从 payload JSON 中提取）
+            try:
+                conn.execute("""
+                    UPDATE eval_tasks SET created_by = payload::json->>'created_by'
+                    WHERE (created_by IS NULL OR created_by = '')
+                      AND payload IS NOT NULL
+                      AND payload::json->>'created_by' IS NOT NULL
+                """)
+            except Exception:
+                pass  # 列或语法不支持时忽略
+            # 清理超过 7 天的操作日志
+            try:
+                cutoff = (datetime.now(timezone.utc) - timedelta(days=7)).isoformat()
+                conn.execute("DELETE FROM operation_logs WHERE create_time < %s", (cutoff,))
+            except Exception:
+                pass  # 表不存在时忽略，下次启动会建表
 
     def _column_names(self, conn: PgConnection, table_name: str) -> set[str]:
         columns = conn.execute(
@@ -618,8 +724,8 @@ class PlatformStore:
         conn.execute(
             """
             INSERT INTO trained_models
-            (id, name, train_methods, base_model_path, create_time, merged, merging, merged_path, artifact_dir, compute_node_id, compute_node_name)
+            (id, name, train_methods, base_model_path, create_time, merged, merging, merged_path, artifact_dir, compute_node_id, compute_node_name, created_by)
-            VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+            VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
             """,
             (
                 trained_model_id,
@@ -633,6 +739,7 @@ class PlatformStore:
                 output_dir,
                 task.get("compute_node_id"),
                 task.get("compute_node_code") or task.get("compute_node_name"),
+                task.get("created_by"),
             ),
         )
         # Use real artifact data from compute node when available
@@ -1209,7 +1316,13 @@ class PlatformStore:
 
     def create_user(self, payload: dict[str, Any]) -> dict[str, Any]:
         user_id = new_id("u")
-        permissions = payload.get("permissions") or (ALL_PERMISSIONS if payload.get("role") == "admin" else ["dashboard"])
+        role = payload.get("role", "user")
+        if role == "admin":
+            permissions = ALL_PERMISSIONS
+        else:
+            # 普通用户：默认拥有所有业务权限，仅排除 user-settings 和 compute
+            role = "user"
+            permissions = [p for p in ALL_PERMISSIONS if p not in ("user-settings", "compute")]
         with self.connect() as conn:
             conn.execute(
                 """
@@ -1222,7 +1335,7 @@ class PlatformStore:
                     payload["username"],
                     hash_password(payload.get("password", "platform123")),
                     payload.get("display_name") or payload["username"],
-                    payload.get("role", "viewer"),
+                    role,
                     payload.get("status", "active"),
                     json_dumps(permissions),
                     utcnow(),
@@ -1235,10 +1348,22 @@ class PlatformStore:
             row = conn.execute("SELECT * FROM users WHERE id=?", (user_id,)).fetchone()
             if not row:
                 raise KeyError(user_id)
+            is_admin = row["role"] == "admin" or bool(row["protected"])
+            if "permissions" in payload:
+                perms = payload["permissions"]
+                if is_admin:
+                    # 管理员权限不可更改，必须是全部
+                    perms = ALL_PERMISSIONS
+                else:
+                    # 非 admin 用户不能拥有 user-settings 和 compute 权限
+                    perms = [p for p in (perms or []) if p not in ("user-settings", "compute")]
+                payload = {**payload, "permissions": perms}
             values = {
                 "role": payload.get("role", row["role"]),
                 "status": payload.get("status", row["status"]),
-                "permissions": json_dumps(payload.get("permissions", json_loads(row["permissions"], []))),
+                "permissions": json_dumps(
+                    payload.get("permissions", json_loads(row["permissions"], []))
+                ),
             }
             conn.execute(
                 "UPDATE users SET role=?, status=?, permissions=? WHERE id=?",
@@ -1253,6 +1378,36 @@ class PlatformStore:
                 raise KeyError(user_id)
             if row["protected"]:
                 raise ValueError("protected user cannot be deleted")
+            # 级联删除该用户关联的数据
+            tables_to_clean = [
+                # 登录会话（避免残留 session 导致统计显示 user_id）
+                ("sessions", "user_id=?", [user_id]),
+                # ACL 授权
+                ("acls", "principal_type='user' AND principal_id=?", [user_id]),
+                # 审批实例（申请人）
+                ("approval_instances", "applicant_id=?", [user_id]),
+                # 审计日志
+                ("audit_logs", "actor_id=?", [user_id]),
+                # 项目成员
+                ("project_members", "user_id=?", [user_id]),
+                # GPU 分配
+                ("gpu_assignments", "user_id=?", [user_id]),
+                # 数据集
+                ("datasets", "created_by=?", [user_id]),
+                # 基座模型
+                ("models", "created_by=?", [user_id]),
+                # 微调产物
+                ("trained_models", "created_by=?", [user_id]),
+                # 评测任务
+                ("eval_tasks", "created_by=?", [user_id]),
+                # 对比/推理任务（payload 中 creator）
+                # 训练任务：仅标记为已删除或保留（有 compute_job_id 关联），不清物理数据
+            ]
+            for table_name, where_clause, params in tables_to_clean:
+                try:
+                    conn.execute(f"DELETE FROM {table_name} WHERE {where_clause}", params)
+                except Exception:
+                    pass  # 表可能不存在或字段不存在，跳过
             conn.execute("DELETE FROM users WHERE id=?", (user_id,))
 
     def reset_password(self, user_id: str, new_password: str) -> None:
@@ -1308,8 +1463,8 @@ class PlatformStore:
             conn.execute(
                 """
                 INSERT INTO models
-                (id, name, type, purpose, model_source, description, path, api_url, api_key, online_model_name, can_train, create_time)
+                (id, name, type, purpose, model_source, description, path, api_url, api_key, online_model_name, can_train, create_time, created_by)
-                VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+                VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                 """,
                 (
                     model_id,
@@ -1324,6 +1479,7 @@ class PlatformStore:
                     payload.get("online_model_name"),
                     can_train,
                     utcnow(),
+                    payload.get("created_by"),
                 ),
             )
             return dict(conn.execute("SELECT * FROM models WHERE id=?", (model_id,)).fetchone())
@@ -1360,12 +1516,12 @@ class PlatformStore:
 
     def delete_model(self, model_id: str) -> None:
         with self.connect() as conn:
-            conn.execute("DELETE FROM models WHERE id=?", (model_id,))
+            conn.execute("UPDATE models SET deleted_at=?, deleted_by=? WHERE id=?", (utcnow(), "system", model_id))
 
     def trained_models(self) -> list[dict[str, Any]]:
         self.refresh_runtime_state()
         with self.connect() as conn:
-            rows = conn.execute("SELECT * FROM trained_models ORDER BY create_time DESC").fetchall()
+            rows = conn.execute("SELECT * FROM trained_models WHERE deleted_at IS NULL ORDER BY create_time DESC").fetchall()
             items = []
             for row in rows:
                 item = {
@@ -1388,7 +1544,7 @@ class PlatformStore:
 
     def delete_trained_model(self, model_id: str) -> None:
         with self.connect() as conn:
-            conn.execute("DELETE FROM trained_models WHERE id=? OR name=?", (model_id, model_id))
+            conn.execute("UPDATE trained_models SET deleted_at=?, deleted_by=? WHERE id=? OR name=?", (utcnow(), "system", model_id, model_id))
 
     def datasets(self) -> list[dict[str, Any]]:
         with self.connect() as conn:
@@ -1397,6 +1553,7 @@ class PlatformStore:
                    FROM datasets dataset
                    LEFT JOIN data_process_tasks task
                      ON task.id=COALESCE(dataset.source_task_id, dataset.task_id)
+                   WHERE dataset.deleted_at IS NULL
                    ORDER BY dataset.create_time DESC"""
             ).fetchall()
             return [self._dataset(conn, row) for row in rows]
@@ -1487,16 +1644,33 @@ class PlatformStore:
 
     def create_dataset(self, payload: dict[str, Any]) -> dict[str, Any]:
         dataset_id = payload.get("id") or new_id("ds")
+        name = str(payload.get("name") or "").strip()
+        if not name:
+            raise ValueError("dataset name is required")
         with self.connect() as conn:
+            existing = conn.execute(
+                "SELECT id, deleted_at FROM datasets WHERE name=?",
+                (name,),
+            ).fetchone()
+            if existing and not existing.get("deleted_at"):
+                raise ValueError(f"dataset name already exists: {name}")
+            # Soft-deleted records remain in the database for audit/history and
+            # still participate in the legacy unique constraint. Free the name
+            # while retaining a traceable tombstone before creating the new row.
+            if existing:
+                conn.execute(
+                    "UPDATE datasets SET name=? WHERE id=?",
+                    (f"{name}__deleted__{existing['id']}", existing["id"]),
+                )
             conn.execute(
                 """
                 INSERT INTO datasets
-                (id, name, type, storage_type, source, task_id, size, count, description, create_time)
+                (id, name, type, storage_type, source, task_id, size, count, description, create_time, created_by)
-                VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+                VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                 """,
                 (
                     dataset_id,
-                    payload["name"],
+                    name,
                     payload.get("type", "train"),
                     payload.get("storage_type", "local"),
                     payload.get("source", "upload"),
@@ -1505,6 +1679,7 @@ class PlatformStore:
                     payload.get("count", 0),
                     payload.get("description"),
                     utcnow(),
+                    payload.get("created_by"),
                 ),
             )
             return self._dataset(conn, conn.execute("SELECT * FROM datasets WHERE id=?", (dataset_id,)).fetchone())
@@ -1535,8 +1710,7 @@ class PlatformStore:
 
     def delete_dataset(self, dataset_id: str) -> None:
         with self.connect() as conn:
-            conn.execute("DELETE FROM dataset_files WHERE dataset_id=?", (dataset_id,))
+            conn.execute("UPDATE datasets SET deleted_at=?, deleted_by=? WHERE id=?", (utcnow(), "system", dataset_id))
-            conn.execute("DELETE FROM datasets WHERE id=?", (dataset_id,))
 
     def add_dataset_file(self, conn: PgConnection, dataset_id: str, name: str, content: str) -> dict[str, Any]:
         now = utcnow()
@@ -1867,6 +2041,7 @@ class PlatformStore:
             "process_id": None,
             "train_duration": "",
             "create_time": now,
+            "created_by": payload.get("created_by"),
         }
         with self.connect() as conn:
             conn.execute(
@@ -1893,13 +2068,14 @@ class PlatformStore:
         task_id = str(payload.get("task_id") or payload.get("id"))
         current = self.task(task_id)
         merged = {**current, **payload, "id": task_id, "status": "syncing", "progress": 8}
-        selected_gpus = payload.get("gpus") or merged.get("gpus") or [0]
+        selected_gpus = payload.get("gpus") or merged.get("gpus") or []
         process_id = int(43000 + (time.time() % 10000))
         with self.connect() as conn:
             owner = f"start:{task_id}:{uuid.uuid4().hex[:8]}"
             if not self._acquire_scheduler_lock(conn, "compute-scheduler", owner):
                 raise RuntimeError("compute scheduler is busy, please retry")
             node = self._schedule_node_locked(conn, payload)
+            selected_gpus = list(node.get("selected_gpus") or selected_gpus)
             sync_job_id = new_id("sync")
             conn.execute(
                 """
@@ -1983,7 +2159,7 @@ class PlatformStore:
         task = self.task(task_id)
         merged = {**task, **(payload or {}), "id": task_id}
         node = self.select_compute_node(merged)
-        selected_gpus = merged.get("gpus") or [0]
+        selected_gpus = merged.get("gpus") or []
         return node, self._compute_job_payload_from_task_node(merged, node, selected_gpus)
 
     def prepare_compute_job_payload_from_payload(self, payload: dict[str, Any]) -> tuple[dict[str, Any], dict[str, Any]]:
@@ -1997,7 +2173,7 @@ class PlatformStore:
             "progress": int(payload.get("progress") or 0),
         }
         node = self.select_compute_node(transient_task)
-        selected_gpus = transient_task.get("gpus") or [0]
+        selected_gpus = transient_task.get("gpus") or []
         return node, self._compute_job_payload_from_task_node(transient_task, node, selected_gpus)
 
     def _compute_job_payload_from_task_node(
@@ -2076,18 +2252,38 @@ class PlatformStore:
         runtime_keys = llama_dataset_keys(dataset_key, runtime_file_names)
         training_keys = runtime_keys[: len(training_files)]
         validation_keys = runtime_keys[len(training_files) :]
-        dataset_format = str(task.get("dataset_format") or (dataset and dataset.get("formatting")) or "alpaca").lower()
+        # P0-2: 推导数据集格式并校验内容（兼容 jsonl：按内容嗅探 ShareGPT/DPO/CPT/Alpaca）
-        # P0-2: Validate dataset content against declared format
         train_type = str(task.get("train_type", task.get("train_method", ""))).upper()
-        expected_format = {
+        expected_format = {"DPO": "dpo", "CPT": "cpt"}.get(train_type)
-            "DPO": "dpo",
+        raw_format = str(
-            "CPT": "cpt",
+            task.get("dataset_format")
-        }.get(train_type)
+            or dataset_metadata.get("format")
+            or (dataset and dataset.get("formatting"))
+            or "alpaca"
+        ).lower()
+        sniffed_format = ""
+        content_samples: dict[str, str] = {}
+        if training_files:
+            with self.connect() as conn:
+                for file_entry in training_files:
+                    sample_row = conn.execute(
+                        "SELECT substr(content, 1, 400000) AS sample FROM dataset_files WHERE id=?",
+                        (str(file_entry["id"]),),
+                    ).fetchone()
+                    sample = (sample_row or {}).get("sample") or ""
+                    content_samples[str(file_entry["id"])] = sample
+                    if not sniffed_format:
+                        sniffed_format = _sniff_dataset_format(sample)
+        known_formats = {"sharegpt", "dpo", "cpt", "pt", "pretrain"}
         if expected_format:
             dataset_format = expected_format
+        elif raw_format in known_formats:
+            dataset_format = raw_format
+        else:
+            dataset_format = sniffed_format or raw_format or "alpaca"
         format_errors: list[str] = []
         for file_entry in training_files:
-            content = file_entry.get("content") or ""
+            content = content_samples.get(str(file_entry["id"])) or ""
             if content:
                 from app.modules.data_process.dataset_format import validate_dataset_format
                 file_errors = validate_dataset_format(dataset_format, content=content)
@@ -2129,7 +2325,7 @@ class PlatformStore:
                 for item in runtime_files
             ],
             "output_dir": output_dir,
-            "gpus": selected_gpus or task.get("gpus") or [0],
+            "gpus": selected_gpus if selected_gpus is not None else (task.get("gpus") or []),
             "compute_node_id": node["id"],
             "compute_node_code": node["code"],
         }
@@ -2139,7 +2335,7 @@ class PlatformStore:
         node = next((item for item in self.compute_nodes() if item["id"] == task.get("compute_node_id")), None)
         if not node:
             raise RuntimeError("compute node not found")
-        return node, self._compute_job_payload_from_task_node(task, node, task.get("gpus") or [0])
+        return node, self._compute_job_payload_from_task_node(task, node, task.get("gpus") or [])
 
     def apply_compute_job(self, task_id: str, job: dict[str, Any]) -> dict[str, Any]:
         status_map = {
@@ -2344,8 +2540,8 @@ class PlatformStore:
             if data.get("metric") == "custom":
                 data["metric"] = data["metric_label"]
             conn.execute(
-                "INSERT INTO eval_tasks (id, name, payload, status, create_time) VALUES (?, ?, ?, ?, ?)",
+                "INSERT INTO eval_tasks (id, name, payload, status, create_time, created_by) VALUES (?, ?, ?, ?, ?, ?)",
-                (task_id, name, json_dumps(data), status, now),
+                (task_id, name, json_dumps(data), status, now, data.get("created_by")),
             )
         return self.eval_task(task_id)
 
@@ -2362,7 +2558,7 @@ class PlatformStore:
 
     def delete_eval_task(self, task_id: str) -> None:
         with self.connect() as conn:
-            conn.execute("DELETE FROM eval_tasks WHERE id=?", (task_id,))
+            conn.execute("UPDATE eval_tasks SET deleted_at=?, deleted_by=? WHERE id=?", (utcnow(), "system", task_id))
 
     def running_eval_tasks(self) -> list[dict[str, Any]]:
         """Return eval tasks that have been submitted to a compute node and are still running."""
@@ -2581,27 +2777,71 @@ class PlatformStore:
     def _node_capacity(self, node: dict[str, Any]) -> int:
         return max(1, int(node.get("max_parallel_jobs") or 1), int(node.get("gpu_count") or 0))
 
+    @staticmethod
+    def _payload_gpu_indexes(payload: dict[str, Any]) -> list[int]:
+        raw = payload.get("gpu_indices")
+        if raw is None:
+            raw = payload.get("gpus")
+        if raw is None:
+            return []
+        try:
+            values = [int(item) for item in raw]
+        except (TypeError, ValueError) as exc:
+            raise RuntimeError("invalid GPU index") from exc
+        if any(item < 0 for item in values):
+            raise RuntimeError("GPU index must be non-negative")
+        return sorted(set(values))
+
+    def _select_node_gpus(
+        self,
+        conn: PgConnection,
+        node: dict[str, Any],
+        requested: list[int],
+        payload: dict[str, Any],
+    ) -> list[int]:
+        available = self._node_gpu_indexes(conn, node)
+        active = self._active_gpu_indexes(conn, node["id"])
+        allowed = payload.get("allowed_gpu_indices")
+        if allowed is not None:
+            available &= {int(item) for item in allowed}
+        if requested:
+            selected = set(requested)
+            if not selected.issubset(available):
+                raise RuntimeError(f"requested GPU is not available on compute node {node['code']}")
+            if selected.intersection(active):
+                raise RuntimeError(f"requested GPU is busy on compute node {node['code']}")
+            return sorted(selected)
+        if payload.get("allow_cpu") or payload.get("device") == "cpu":
+            return []
+        count = max(1, int(payload.get("gpu_count") or 1))
+        free = sorted(available - active)
+        if len(free) < count:
+            raise RuntimeError(f"compute node {node['code']} has only {len(free)} available GPU(s)")
+        return free[:count]
+
     def _schedule_node_locked(self, conn: PgConnection, payload: dict[str, Any]) -> dict[str, Any]:
         requested = payload.get("requested_node_id") or payload.get("compute_node_id")
-        requested_gpus = [int(item) for item in payload.get("gpus") or []]
+        requested_gpus = self._payload_gpu_indexes(payload)
         nodes = self._compute_nodes_locked(conn)
         candidates = [
             n
             for n in nodes
             if n["enabled"] and n["scheduler_status"] == "online" and n["current_running_jobs"] < self._node_capacity(n)
         ]
-        if requested_gpus:
-            requested_gpu_set = set(requested_gpus)
-            candidates = [
-                node
-                for node in candidates
-                if requested_gpu_set.issubset(self._node_gpu_indexes(conn, node))
-                and not requested_gpu_set.intersection(self._active_gpu_indexes(conn, node["id"]))
-            ]
         if requested:
             selected = next((n for n in candidates if n["id"] == requested), None)
-            if selected:
+            if not selected:
-                return selected
+                raise RuntimeError("selected compute node is unavailable")
+            selected["selected_gpus"] = self._select_node_gpus(conn, selected, requested_gpus, payload)
+            return selected
+        filtered = []
+        for node in candidates:
+            try:
+                node["selected_gpus"] = self._select_node_gpus(conn, node, requested_gpus, payload)
+                filtered.append(node)
+            except RuntimeError:
+                continue
+        candidates = filtered
         if not candidates:
             if not nodes:
                 raise RuntimeError("no available compute node: no compute node configured")
@@ -2613,6 +2853,8 @@ class PlatformStore:
                     reason = f"status={node['scheduler_status']}"
                 elif node["current_running_jobs"] >= self._node_capacity(node):
                     reason = f"capacity full {node['current_running_jobs']}/{self._node_capacity(node)}"
+                elif requested:
+                    reason = "selected node unavailable"
                 else:
                     reason = "not selected"
                 reasons.append(f"{node['code']}({reason})")
@@ -2751,6 +2993,112 @@ class PlatformStore:
                     result.append(dict(row))
             return result
 
+    def create_storage_object(self, payload: dict[str, Any]) -> dict[str, Any]:
+        object_id = str(payload.get("id") or new_id("object"))
+        with self.connect() as conn:
+            conn.execute(
+                """
+                INSERT INTO storage_objects
+                (id, resource_type, resource_id, version_id, bucket, object_key, file_name,
+                 content_type, checksum_sha256, byte_size, status, created_by, create_time)
+                VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+                ON CONFLICT (resource_type, resource_id, version_id, object_key)
+                DO UPDATE SET file_name=EXCLUDED.file_name, content_type=EXCLUDED.content_type,
+                    checksum_sha256=EXCLUDED.checksum_sha256, byte_size=EXCLUDED.byte_size,
+                    status=EXCLUDED.status, created_by=EXCLUDED.created_by
+                """,
+                (
+                    object_id, payload["resource_type"], payload["resource_id"], payload["version_id"],
+                    payload["bucket"], payload["object_key"], payload.get("file_name"), payload.get("content_type"),
+                    payload.get("checksum_sha256"), int(payload.get("byte_size") or 0), payload.get("status", "pending"),
+                    payload.get("created_by"), payload.get("create_time") or utcnow(),
+                ),
+            )
+            row = conn.execute(
+                "SELECT * FROM storage_objects WHERE resource_type=? AND resource_id=? AND version_id=? AND object_key=?",
+                (payload["resource_type"], payload["resource_id"], payload["version_id"], payload["object_key"]),
+            ).fetchone()
+            return dict(row)
+
+    def link_dataset_file_storage_object(self, file_id: str, storage_object_id: str) -> None:
+        """Link an uploaded dataset file to its canonical MinIO object."""
+        with self.connect() as conn:
+            conn.execute(
+                """
+                UPDATE dataset_files
+                SET storage_object_id=?
+                WHERE id=?
+                """,
+                (storage_object_id, file_id),
+            )
+            row = conn.execute(
+                "SELECT dataset_id FROM dataset_files WHERE id=?",
+                (file_id,),
+            ).fetchone()
+            if row:
+                conn.execute(
+                    "UPDATE datasets SET storage_type='minio' WHERE id=?",
+                    (row["dataset_id"],),
+                )
+
+    def storage_objects_for_resource(self, resource_type: str, resource_id: str) -> list[dict[str, Any]]:
+        with self.connect() as conn:
+            rows = conn.execute(
+                "SELECT * FROM storage_objects WHERE resource_type=? AND resource_id=? AND status='available' ORDER BY version_id, object_key",
+                (resource_type, resource_id),
+            ).fetchall()
+            return [dict(row) for row in rows]
+
+    def update_storage_object(self, object_id: str, payload: dict[str, Any]) -> dict[str, Any]:
+        allowed = {"status", "checksum_sha256", "byte_size", "content_type"}
+        fields = {key: value for key, value in payload.items() if key in allowed}
+        if fields:
+            assignments = ", ".join(f"{key}=?" for key in fields)
+            with self.connect() as conn:
+                conn.execute(f"UPDATE storage_objects SET {assignments} WHERE id=?", (*fields.values(), object_id))
+        with self.connect() as conn:
+            row = conn.execute("SELECT * FROM storage_objects WHERE id=?", (object_id,)).fetchone()
+            if not row:
+                raise KeyError(object_id)
+            return dict(row)
+
+    def create_storage_cache_job(self, payload: dict[str, Any]) -> dict[str, Any]:
+        job_id = str(payload.get("id") or new_id("cache"))
+        with self.connect() as conn:
+            conn.execute(
+                """
+                INSERT INTO storage_cache_jobs
+                (id, storage_object_id, node_id, direction, status, progress, local_path, error, create_time, completed_at)
+                VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+                """,
+                (job_id, payload["storage_object_id"], payload["node_id"], payload.get("direction", "download"),
+                 payload.get("status", "running"), int(payload.get("progress", 0)), payload.get("local_path"),
+                 payload.get("error"), payload.get("create_time") or utcnow(), payload.get("completed_at")),
+            )
+            row = conn.execute("SELECT * FROM storage_cache_jobs WHERE id=?", (job_id,)).fetchone()
+            return dict(row)
+
+    def update_storage_cache_job(self, job_id: str, payload: dict[str, Any]) -> dict[str, Any]:
+        allowed = {"status", "progress", "local_path", "error", "completed_at"}
+        fields = {key: value for key, value in payload.items() if key in allowed}
+        if fields:
+            assignments = ", ".join(f"{key}=?" for key in fields)
+            with self.connect() as conn:
+                conn.execute(f"UPDATE storage_cache_jobs SET {assignments} WHERE id=?", (*fields.values(), job_id))
+        with self.connect() as conn:
+            row = conn.execute("SELECT * FROM storage_cache_jobs WHERE id=?", (job_id,)).fetchone()
+            if not row:
+                raise KeyError(job_id)
+            return dict(row)
+
+    def storage_cache_jobs_for_node(self, node_id: str, limit: int = 100) -> list[dict[str, Any]]:
+        with self.connect() as conn:
+            rows = conn.execute(
+                "SELECT * FROM storage_cache_jobs WHERE node_id=? ORDER BY create_time DESC LIMIT ?",
+                (node_id, max(1, min(limit, 500))),
+            ).fetchall()
+            return [dict(row) for row in rows]
+
     def update_resource_replica_sync_result(
         self,
         replica_id: str,
@@ -3434,9 +3782,9 @@ class PlatformStore:
         login_at = utcnow()
         with self.connect() as conn:
             conn.execute(
-                "INSERT INTO sessions (id, user_id, username, login_at, create_time) "
+                "INSERT INTO sessions (id, user_id, username, login_at, issued_at, expires_at, create_time) "
-                "VALUES (%s, %s, (SELECT username FROM users WHERE id=%s), %s, %s)",
+                "VALUES (%s, %s, (SELECT username FROM users WHERE id=%s), %s, %s, %s, %s)",
-                (sid, user_id, user_id, login_at, login_at),
+                (sid, user_id, user_id, login_at, login_at, datetime.fromtimestamp(time.time() + 1800, timezone.utc).isoformat(), login_at),
             )
         return {"session_id": sid, "user_id": user_id, "login_at": login_at}
 
@@ -3485,12 +3833,16 @@ class PlatformStore:
 
         sessions 表列：login_at(TEXT), logout_at(TEXT), duration_seconds(INT)。
         优先用 duration_seconds；为空时回退计算 now-login_at（未登出）或 logout_at-login_at。
+
+        注意：使用 INNER JOIN 只统计仍存在于 users 表中的用户，
+        避免已删除用户的残留 session 记录导致显示 user_id（如 u_xxxx）。
         """
         with self.connect() as conn:
+            # 使用 INNER JOIN 而非 LEFT JOIN，确保只统计仍然存在的用户
             rows = conn.execute(
                 "SELECT s.user_id, s.login_at, s.logout_at, s.duration_seconds, "
                 "u.username, u.display_name, u.role "
-                "FROM sessions s LEFT JOIN users u ON s.user_id = u.id "
+                "FROM sessions s INNER JOIN users u ON s.user_id = u.id "
                 "WHERE s.login_at::timestamptz >= NOW() - make_interval(days => %s)",
                 (days,),
             ).fetchall()
@@ -3498,10 +3850,15 @@ class PlatformStore:
         agg: dict[str, dict[str, Any]] = {}
         for r in rows:
             uid = r["user_id"] or ""
+            # 优先使用 display_name，其次 username，最后才回退到 user_id
+            display = r["display_name"] or r["username"] or uid
+            # 如果回退到了 user_id 格式（说明用户信息不完整），标记为"未知用户"
+            if display == uid and display.startswith("u_") and len(display) > 10:
+                display = "（已删除用户）"
             bucket = agg.setdefault(
                 uid,
                 {
-                    "user": r["display_name"] or r["username"] or uid,
+                    "user": display,
                     "role": r["role"] or "",
                     "total": 0.0,
                 },
@@ -3912,8 +4269,8 @@ class PlatformStore:
             bucket = grouped.setdefault(
                 key,
                 {
-                    "subject_type": r.get("principal_type"),
+                    "principal_type": r.get("principal_type"),
-                    "subject_id": r.get("principal_id"),
+                    "principal_id": r.get("principal_id"),
                     "permissions": [],
                 },
             )
@@ -3931,8 +4288,8 @@ class PlatformStore:
             for perm in e.get("permissions") or []:
                 flat.append(
                     {
-                        "principal_type": e.get("subject_type"),
+                        "principal_type": e.get("principal_type") or e.get("subject_type"),
-                        "principal_id": e.get("subject_id"),
+                        "principal_id": e.get("principal_id") or e.get("subject_id"),
                         "permission": perm,
                     }
                 )
@@ -4005,6 +4362,138 @@ class PlatformStore:
                 "DELETE FROM retention_policies WHERE id=?", (policy_id,)
             )
 
+    # ===================== 平台治理：GPU 算力分配 =====================
+
+    def gpu_assignments(self) -> list[dict[str, Any]]:
+        """查询全部分配关系。"""
+        with self.connect() as conn:
+            rows = conn.execute(
+                """
+                SELECT ga.*, u.username, u.display_name,
+                       n.code AS node_code, n.name AS node_name, g.name AS gpu_name
+                FROM gpu_assignments ga
+                LEFT JOIN users u ON u.id = ga.user_id
+                LEFT JOIN compute_nodes n ON n.id = ga.node_id
+                LEFT JOIN gpus g ON g.node_id = ga.node_id AND g.gpu_index = ga.gpu_index
+                ORDER BY ga.assigned_at DESC
+                """
+            ).fetchall()
+        return [dict(r) for r in rows]
+
+    def gpu_assignments_for_user(self, user_id: str) -> list[dict[str, Any]]:
+        """查询某用户被分配的 GPU 列表。"""
+        with self.connect() as conn:
+            rows = conn.execute(
+                """
+                SELECT ga.node_id, ga.gpu_index,
+                       n.code AS node_code, n.name AS node_name,
+                       g.name AS gpu_name, g.uuid, g.memory_total_gb
+                FROM gpu_assignments ga
+                JOIN compute_nodes n ON n.id = ga.node_id
+                LEFT JOIN gpus g ON g.node_id = ga.node_id AND g.gpu_index = ga.gpu_index
+                WHERE ga.user_id = ?
+                ORDER BY n.code, ga.gpu_index
+                """,
+                (user_id,),
+            ).fetchall()
+        return [dict(r) for r in rows]
+
+    def assign_gpus(self, assignments: list[dict[str, Any]], assigned_by: str | None = None) -> list[dict[str, Any]]:
+        """批量分配 GPU（幂等：已存在的分配跳过）。"""
+        now = utcnow()
+        with self.connect() as conn:
+            for a in assignments:
+                node_id = a["node_id"]
+                gpu_index = a["gpu_index"]
+                user_id = a["user_id"]
+                existing = conn.execute(
+                    "SELECT id FROM gpu_assignments WHERE node_id=? AND gpu_index=? AND user_id=?",
+                    (node_id, gpu_index, user_id),
+                ).fetchone()
+                if existing:
+                    continue
+                aid = new_id("ga")
+                conn.execute(
+                    """
+                    INSERT INTO gpu_assignments (id, node_id, gpu_index, user_id, assigned_by, assigned_at)
+                    VALUES (?, ?, ?, ?, ?, ?)
+                    """,
+                    (aid, node_id, gpu_index, user_id, assigned_by, now),
+                )
+        return self.gpu_assignments()
+
+    def unassign_gpu(self, assignment_id: str) -> None:
+        with self.connect() as conn:
+            conn.execute("DELETE FROM gpu_assignments WHERE id=?", (assignment_id,))
+
+    def check_gpu_access(self, user_id: str, node_id: str, gpu_indices: list[int]) -> bool:
+        """检查用户是否被分配了指定节点的指定 GPU 卡。"""
+        if not gpu_indices:
+            return True
+        with self.connect() as conn:
+            rows = conn.execute(
+                """
+                SELECT gpu_index FROM gpu_assignments
+                WHERE user_id=? AND node_id=?
+                """,
+                (user_id, node_id),
+            ).fetchall()
+        assigned = {r["gpu_index"] for r in rows}
+        return all(idx in assigned for idx in gpu_indices)
+
+    def assigned_gpu_indexes(self, user_id: str, node_id: str) -> list[int]:
+        with self.connect() as conn:
+            rows = conn.execute(
+                "SELECT gpu_index FROM gpu_assignments WHERE user_id=? AND node_id=? ORDER BY gpu_index",
+                (user_id, node_id),
+            ).fetchall()
+        return [int(row["gpu_index"]) for row in rows]
+
+    # ===================== 平台治理：资源可见性过滤 =====================
+
+    def _filter_accessible_ids(
+        self, resource_type: str, all_ids: list[str], user: dict[str, Any]
+    ) -> list[str]:
+        """从全部资源 ID 中过滤出当前用户可访问的 ID 列表。
+        - admin 直接返回全部。
+        - 资源所有者可见（需调用方在 all_ids 中提供 owned ids）。
+        - ACL 授权的用户/角色可见。
+        """
+        if user.get("role") == "admin" or user.get("protected"):
+            return all_ids
+        if not all_ids:
+            return []
+        user_id = user.get("id")
+        user_role = user.get("role")
+        with self.connect() as conn:
+            rows = conn.execute(
+                """
+                SELECT DISTINCT resource_id FROM acls
+                WHERE resource_type=? AND (
+                    (principal_type='user' AND principal_id=?)
+                    OR (principal_type='role' AND principal_id=?)
+                )
+                """,
+                (resource_type, user_id, user_role),
+            ).fetchall()
+        accessible = {r["resource_id"] for r in rows}
+        return [rid for rid in all_ids if rid in accessible]
+
+    def change_password(self, user_id: str, old_password: str, new_password: str) -> bool:
+        """用户自行修改密码：验证旧密码后设置新密码。"""
+        with self.connect() as conn:
+            row = conn.execute("SELECT * FROM users WHERE id=?", (user_id,)).fetchone()
+            if not row:
+                raise KeyError(user_id)
+            matched, _ = verify_password(old_password, row["password_hash"])
+            if not matched:
+                return False
+            conn.execute(
+                "UPDATE users SET password_hash=? WHERE id=?",
+                (hash_password(new_password), user_id),
+            )
+        return True
+
 
 _store: PlatformStore | None = None
 

backend/app/db/sql/000_full_init.sql

@@ -0,0 +1,893 @@
+-- ============================================================================
+-- YG Fine-Tune Platform — PostgreSQL 完整初始化脚本（一键建库建表）
+-- ============================================================================
+-- 用途：切换到新的 PG 数据集时，一次性创建平台运行所需的全部数据库对象与
+--       基础种子数据（幂等，可重复执行）。
+--
+-- 覆盖范围（与运行时代码实际使用的表一致）：
+--   001_platform_runtime.sql       平台核心表
+--   002_governance.sql             治理表（租户 / 审批 / 审计 / 留存）
+--   003_tenant_quota.sql           租户配额列
+--   003_model_path_governance.sql  模型可训练标识列
+--   002_data_process.sql           数据处理表 + 数据集扩展列
+--   本文件补充：data_convert_tasks（数据转换任务，运行时代码引用但原脚本缺失）
+--   种子数据：admin / operator 两个初始用户
+--
+-- 说明：
+--   * 本脚本通过 psql 执行，包含 DO $$ ... $$ 块与事务，不能用应用的
+--     executescript()（按分号切分）执行。
+--   * 应用启动时 PlatformStore.ensure_schema() 只会自动执行
+--     001 / 002_governance / 003_tenant_quota；数据处理表需另跑
+--     002_data_process.sql（本脚本已包含）。应用首次启动还会自动补充
+--     admin/operator 种子用户（本脚本已包含，二选一即可）。
+--   * 脚本内所有 DDL 均使用 IF NOT EXISTS / ADD COLUMN IF NOT EXISTS，
+--     可在已初始化的库上安全重复执行。
+--
+-- 执行步骤（详见 docs/database-config.md）：
+--   1. 以超级用户创建角色与数据库（必须单独执行，不能放进事务）：
+--        CREATE ROLE  yg_ft  LOGIN PASSWORD '请改为强密码';
+--        CREATE DATABASE yg_ft OWNER yg_ft;
+--   2. 连接目标库执行本脚本：
+--        psql "postgresql://yg_ft:密码@<host>:5432/yg_ft" -f backend/app/db/sql/000_full_init.sql
+--   3. 可选：为 superuser 授权
+--        ALTER ROLE yg_ft SUPERUSER;   -- 仅当需要执行 CREATE EXTENSION 等
+-- ============================================================================
+
+BEGIN;
+
+-- ============================================================================
+-- 一、平台核心表（来源：001_platform_runtime.sql）
+-- ============================================================================
+
+CREATE TABLE IF NOT EXISTS users (
+    id TEXT PRIMARY KEY,
+    username TEXT NOT NULL UNIQUE,
+    password_hash TEXT NOT NULL,
+    display_name TEXT NOT NULL,
+    role TEXT NOT NULL,
+    status TEXT NOT NULL,
+    permissions TEXT NOT NULL,
+    create_time TEXT NOT NULL,
+    last_login TEXT,
+    protected INTEGER NOT NULL DEFAULT 0
+);
+
+CREATE TABLE IF NOT EXISTS models (
+    id TEXT PRIMARY KEY,
+    name TEXT NOT NULL UNIQUE,
+    type TEXT NOT NULL,
+    purpose TEXT NOT NULL,
+    model_source TEXT NOT NULL,
+    description TEXT,
+    path TEXT,
+    api_url TEXT,
+    api_key TEXT,
+    online_model_name TEXT,
+    can_train INTEGER NOT NULL DEFAULT 0,
+    create_time TEXT NOT NULL,
+    created_by TEXT,
+    tenant_id TEXT,
+    project_id TEXT,
+    deleted_at TEXT,
+    deleted_by TEXT
+);
+
+CREATE TABLE IF NOT EXISTS trained_models (
+    id TEXT PRIMARY KEY,
+    name TEXT NOT NULL UNIQUE,
+    train_methods TEXT NOT NULL,
+    base_model_path TEXT,
+    create_time TEXT NOT NULL,
+    merged INTEGER NOT NULL DEFAULT 0,
+    merging INTEGER NOT NULL DEFAULT 0,
+    merged_path TEXT,
+    artifact_dir TEXT,
+    compute_node_id TEXT,
+    compute_node_name TEXT,
+    created_by TEXT,
+    tenant_id TEXT,
+    project_id TEXT,
+    deleted_at TEXT,
+    deleted_by TEXT
+);
+
+CREATE TABLE IF NOT EXISTS model_lineage (
+    id TEXT PRIMARY KEY,
+    child_resource_type TEXT NOT NULL,
+    child_resource_id TEXT NOT NULL,
+    parent_resource_type TEXT NOT NULL,
+    parent_resource_id TEXT NOT NULL,
+    relation_type TEXT NOT NULL,
+    compute_job_id TEXT,
+    payload TEXT NOT NULL,
+    create_time TEXT NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS model_artifacts (
+    id TEXT PRIMARY KEY,
+    model_id TEXT NOT NULL,
+    model_kind TEXT NOT NULL,
+    artifact_type TEXT NOT NULL,
+    path TEXT NOT NULL,
+    size_bytes BIGINT NOT NULL DEFAULT 0,
+    checksum_sha256 TEXT,
+    metadata TEXT NOT NULL,
+    compute_job_id TEXT,
+    create_time TEXT NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS model_export_jobs (
+    id TEXT PRIMARY KEY,
+    trained_model_id TEXT,
+    compute_job_id TEXT NOT NULL,
+    node_id TEXT,
+    export_type TEXT NOT NULL,
+    quantization_bit INTEGER NOT NULL DEFAULT 0,
+    status TEXT NOT NULL,
+    output_dir TEXT,
+    payload TEXT NOT NULL,
+    create_time TEXT NOT NULL,
+    completed_at TEXT
+);
+
+CREATE TABLE IF NOT EXISTS datasets (
+    id TEXT PRIMARY KEY,
+    name TEXT NOT NULL UNIQUE,
+    type TEXT NOT NULL,
+    storage_type TEXT NOT NULL,
+    source TEXT NOT NULL,
+    task_id TEXT,
+    size TEXT,
+    count INTEGER NOT NULL DEFAULT 0,
+    description TEXT,
+    create_time TEXT NOT NULL,
+    created_by TEXT,
+    tenant_id TEXT,
+    project_id TEXT,
+    deleted_at TEXT,
+    deleted_by TEXT
+);
+
+CREATE TABLE IF NOT EXISTS dataset_files (
+    id TEXT PRIMARY KEY,
+    dataset_id TEXT NOT NULL REFERENCES datasets(id) ON DELETE CASCADE,
+    name TEXT NOT NULL,
+    size TEXT,
+    content TEXT NOT NULL,
+    active_version_id TEXT NOT NULL,
+    versions TEXT NOT NULL,
+    create_time TEXT NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS compute_nodes (
+    id TEXT PRIMARY KEY,
+    code TEXT NOT NULL UNIQUE,
+    name TEXT NOT NULL,
+    api_base_url TEXT NOT NULL,
+    file_gateway_url TEXT NOT NULL,
+    enabled INTEGER NOT NULL DEFAULT 1,
+    scheduler_status TEXT NOT NULL,
+    scheduler_weight INTEGER NOT NULL DEFAULT 100,
+    tags TEXT NOT NULL,
+    gpu_count INTEGER NOT NULL DEFAULT 0,
+    current_running_jobs INTEGER NOT NULL DEFAULT 0,
+    max_parallel_jobs INTEGER NOT NULL DEFAULT 2,
+    data_root TEXT NOT NULL,
+    model_root TEXT NOT NULL,
+    log_root TEXT NOT NULL,
+    api_version TEXT NOT NULL DEFAULT 'v1',
+    capabilities TEXT NOT NULL DEFAULT '[]',
+    description TEXT,
+    last_health_check_at TEXT,
+    health_detail TEXT NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS gpus (
+    id TEXT PRIMARY KEY,
+    node_id TEXT NOT NULL REFERENCES compute_nodes(id) ON DELETE CASCADE,
+    gpu_index INTEGER NOT NULL,
+    uuid TEXT NOT NULL,
+    name TEXT NOT NULL,
+    memory_total_gb DOUBLE PRECISION NOT NULL,
+    power_limit_w DOUBLE PRECISION NOT NULL,
+    base_temperature INTEGER NOT NULL,
+    last_seen_at TEXT
+);
+
+CREATE TABLE IF NOT EXISTS fine_tune_tasks (
+    id TEXT PRIMARY KEY,
+    name TEXT NOT NULL UNIQUE,
+    payload TEXT NOT NULL,
+    status TEXT NOT NULL,
+    progress INTEGER NOT NULL DEFAULT 0,
+    process_id INTEGER,
+    create_time TEXT NOT NULL,
+    start_time TEXT,
+    completed_at TEXT,
+    compute_node_id TEXT REFERENCES compute_nodes(id) ON DELETE SET NULL,
+    gpus TEXT NOT NULL,
+    sync_job_id TEXT,
+    compute_job_id TEXT
+);
+
+CREATE TABLE IF NOT EXISTS fine_tune_metrics (
+    id TEXT PRIMARY KEY,
+    task_id TEXT NOT NULL REFERENCES fine_tune_tasks(id) ON DELETE CASCADE,
+    step INTEGER NOT NULL,
+    epoch DOUBLE PRECISION,
+    loss DOUBLE PRECISION,
+    grad_norm DOUBLE PRECISION,
+    learning_rate DOUBLE PRECISION,
+    raw TEXT NOT NULL,
+    create_time TEXT NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS fine_tune_checkpoints (
+    id TEXT PRIMARY KEY,
+    task_id TEXT NOT NULL REFERENCES fine_tune_tasks(id) ON DELETE CASCADE,
+    step INTEGER NOT NULL,
+    name TEXT NOT NULL,
+    path TEXT NOT NULL,
+    size_bytes BIGINT NOT NULL DEFAULT 0,
+    create_time TEXT NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS compute_jobs (
+    id TEXT PRIMARY KEY,
+    task_id TEXT REFERENCES fine_tune_tasks(id) ON DELETE SET NULL,
+    node_id TEXT REFERENCES compute_nodes(id) ON DELETE SET NULL,
+    engine TEXT NOT NULL,
+    status TEXT NOT NULL,
+    command TEXT NOT NULL,
+    output_dir TEXT,
+    log_file TEXT,
+    payload TEXT NOT NULL,
+    create_time TEXT NOT NULL,
+    update_time TEXT NOT NULL,
+    completed_at TEXT
+);
+
+CREATE TABLE IF NOT EXISTS gpu_allocations (
+    id TEXT PRIMARY KEY,
+    task_id TEXT REFERENCES fine_tune_tasks(id) ON DELETE CASCADE,
+    compute_job_id TEXT,
+    node_id TEXT REFERENCES compute_nodes(id) ON DELETE CASCADE,
+    gpu_index INTEGER NOT NULL,
+    status TEXT NOT NULL,
+    create_time TEXT NOT NULL,
+    released_at TEXT
+);
+
+CREATE TABLE IF NOT EXISTS scheduler_locks (
+    lock_key TEXT PRIMARY KEY,
+    owner TEXT NOT NULL,
+    expires_at TEXT NOT NULL,
+    create_time TEXT NOT NULL,
+    update_time TEXT NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS resource_replicas (
+    id TEXT PRIMARY KEY,
+    node_id TEXT NOT NULL REFERENCES compute_nodes(id) ON DELETE CASCADE,
+    resource_type TEXT NOT NULL,
+    resource_id TEXT NOT NULL,
+    local_path TEXT NOT NULL,
+    status TEXT NOT NULL,
+    sync_status TEXT NOT NULL,
+    checksum_sha256 TEXT,
+    byte_size BIGINT NOT NULL DEFAULT 0,
+    last_checked_at TEXT,
+    last_error TEXT,
+    create_time TEXT NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS resource_sync_jobs (
+    id TEXT PRIMARY KEY,
+    target_node_id TEXT NOT NULL,
+    resources TEXT NOT NULL,
+    status TEXT NOT NULL,
+    progress INTEGER NOT NULL DEFAULT 0,
+    create_time TEXT NOT NULL,
+    completed_at TEXT
+);
+
+CREATE TABLE IF NOT EXISTS storage_objects (
+    id TEXT PRIMARY KEY,
+    resource_type TEXT NOT NULL,
+    resource_id TEXT NOT NULL,
+    version_id TEXT NOT NULL,
+    bucket TEXT NOT NULL,
+    object_key TEXT NOT NULL,
+    file_name TEXT,
+    content_type TEXT,
+    checksum_sha256 TEXT,
+    byte_size BIGINT NOT NULL DEFAULT 0,
+    status TEXT NOT NULL DEFAULT 'pending',
+    created_by TEXT,
+    create_time TEXT NOT NULL,
+    UNIQUE (resource_type, resource_id, version_id, object_key)
+);
+
+CREATE TABLE IF NOT EXISTS storage_cache_jobs (
+    id TEXT PRIMARY KEY,
+    storage_object_id TEXT NOT NULL REFERENCES storage_objects(id) ON DELETE CASCADE,
+    node_id TEXT NOT NULL REFERENCES compute_nodes(id) ON DELETE CASCADE,
+    direction TEXT NOT NULL,
+    status TEXT NOT NULL,
+    progress INTEGER NOT NULL DEFAULT 0,
+    local_path TEXT,
+    error TEXT,
+    create_time TEXT NOT NULL,
+    completed_at TEXT
+);
+
+CREATE INDEX IF NOT EXISTS idx_storage_objects_resource ON storage_objects(resource_type, resource_id, version_id);
+CREATE INDEX IF NOT EXISTS idx_storage_cache_jobs_node_status ON storage_cache_jobs(node_id, status);
+
+CREATE TABLE IF NOT EXISTS eval_tasks (
+    id TEXT PRIMARY KEY,
+    name TEXT NOT NULL,
+    payload TEXT NOT NULL,
+    status TEXT NOT NULL,
+    create_time TEXT NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS eval_dimensions (
+    id TEXT PRIMARY KEY,
+    name TEXT NOT NULL,
+    payload TEXT NOT NULL,
+    is_active INTEGER NOT NULL DEFAULT 1,
+    is_default INTEGER NOT NULL DEFAULT 0,
+    create_time TEXT NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS compare_tasks (
+    id TEXT PRIMARY KEY,
+    name TEXT NOT NULL,
+    payload TEXT NOT NULL,
+    status TEXT NOT NULL,
+    create_time TEXT NOT NULL
+);
+
+CREATE INDEX IF NOT EXISTS idx_fine_tune_status ON fine_tune_tasks(status);
+CREATE INDEX IF NOT EXISTS idx_fine_tune_compute_job ON fine_tune_tasks(compute_job_id);
+CREATE INDEX IF NOT EXISTS idx_fine_tune_compute_node_status ON fine_tune_tasks(compute_node_id, status);
+CREATE INDEX IF NOT EXISTS idx_model_lineage_child ON model_lineage(child_resource_type, child_resource_id);
+CREATE INDEX IF NOT EXISTS idx_model_lineage_parent ON model_lineage(parent_resource_type, parent_resource_id);
+CREATE INDEX IF NOT EXISTS idx_model_artifacts_model ON model_artifacts(model_kind, model_id, artifact_type);
+CREATE INDEX IF NOT EXISTS idx_model_export_jobs_model ON model_export_jobs(trained_model_id, create_time DESC);
+CREATE INDEX IF NOT EXISTS idx_model_export_jobs_compute ON model_export_jobs(compute_job_id);
+CREATE INDEX IF NOT EXISTS idx_fine_tune_metrics_task_step ON fine_tune_metrics(task_id, step);
+CREATE UNIQUE INDEX IF NOT EXISTS uq_fine_tune_metrics_task_step_epoch ON fine_tune_metrics(task_id, step, epoch);
+CREATE INDEX IF NOT EXISTS idx_fine_tune_checkpoints_task_step ON fine_tune_checkpoints(task_id, step);
+CREATE UNIQUE INDEX IF NOT EXISTS uq_fine_tune_checkpoints_task_path ON fine_tune_checkpoints(task_id, path);
+CREATE INDEX IF NOT EXISTS idx_compute_jobs_task ON compute_jobs(task_id);
+CREATE INDEX IF NOT EXISTS idx_compute_jobs_node_status ON compute_jobs(node_id, status);
+CREATE INDEX IF NOT EXISTS idx_gpu_allocations_node_status ON gpu_allocations(node_id, status);
+CREATE UNIQUE INDEX IF NOT EXISTS uq_gpu_allocations_active ON gpu_allocations(node_id, gpu_index) WHERE status IN ('allocated','running');
+CREATE INDEX IF NOT EXISTS idx_scheduler_locks_expires ON scheduler_locks(expires_at);
+CREATE INDEX IF NOT EXISTS idx_dataset_files_dataset ON dataset_files(dataset_id);
+CREATE INDEX IF NOT EXISTS idx_gpus_node ON gpus(node_id);
+CREATE UNIQUE INDEX IF NOT EXISTS uq_gpus_node_index ON gpus(node_id, gpu_index);
+CREATE INDEX IF NOT EXISTS idx_replicas_resource ON resource_replicas(resource_type, resource_id);
+CREATE UNIQUE INDEX IF NOT EXISTS uq_replicas_node_resource ON resource_replicas(node_id, resource_type, resource_id);
+CREATE INDEX IF NOT EXISTS idx_sync_jobs_node_status ON resource_sync_jobs(target_node_id, status);
+CREATE INDEX IF NOT EXISTS idx_eval_tasks_status ON eval_tasks(status);
+CREATE INDEX IF NOT EXISTS idx_eval_dimensions_active ON eval_dimensions(is_active);
+CREATE INDEX IF NOT EXISTS idx_compare_tasks_status ON compare_tasks(status);
+
+-- ---- 项目 / 租户 ----
+
+CREATE TABLE IF NOT EXISTS projects (
+    id TEXT PRIMARY KEY,
+    tenant_id TEXT NOT NULL DEFAULT 'default',
+    name TEXT NOT NULL,
+    code TEXT NOT NULL,
+    description TEXT,
+    quota TEXT,
+    status TEXT NOT NULL DEFAULT 'active',
+    create_time TEXT NOT NULL,
+    create_by TEXT,
+    updated_at TEXT
+);
+
+CREATE TABLE IF NOT EXISTS project_members (
+    project_id TEXT NOT NULL REFERENCES projects(id) ON DELETE CASCADE,
+    user_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
+    role TEXT NOT NULL DEFAULT 'member',
+    create_time TEXT NOT NULL,
+    PRIMARY KEY (project_id, user_id)
+);
+
+CREATE TABLE IF NOT EXISTS roles (
+    id TEXT PRIMARY KEY,
+    name TEXT NOT NULL,
+    permissions TEXT NOT NULL DEFAULT '[]',
+    create_time TEXT
+);
+
+CREATE TABLE IF NOT EXISTS sessions (
+    id TEXT PRIMARY KEY,
+    user_id TEXT NOT NULL,
+    username TEXT,
+    login_at TEXT,
+    logout_at TEXT,
+    duration_seconds INTEGER,
+    issued_at TEXT,
+    expires_at TEXT,
+    ip TEXT,
+    create_time TEXT
+);
+
+CREATE TABLE IF NOT EXISTS acls (
+    id TEXT PRIMARY KEY,
+    resource_type TEXT NOT NULL,
+    resource_id TEXT NOT NULL,
+    principal_type TEXT NOT NULL,
+    principal_id TEXT NOT NULL,
+    permission TEXT NOT NULL,
+    create_time TEXT
+);
+ALTER TABLE models ADD COLUMN IF NOT EXISTS deleted_at TEXT;
+ALTER TABLE models ADD COLUMN IF NOT EXISTS deleted_by TEXT;
+ALTER TABLE trained_models ADD COLUMN IF NOT EXISTS deleted_at TEXT;
+ALTER TABLE trained_models ADD COLUMN IF NOT EXISTS deleted_by TEXT;
+ALTER TABLE datasets ADD COLUMN IF NOT EXISTS deleted_at TEXT;
+ALTER TABLE datasets ADD COLUMN IF NOT EXISTS deleted_by TEXT;
+ALTER TABLE eval_tasks ADD COLUMN IF NOT EXISTS deleted_at TEXT;
+ALTER TABLE eval_tasks ADD COLUMN IF NOT EXISTS deleted_by TEXT;
+CREATE UNIQUE INDEX IF NOT EXISTS uq_acls_subject_permission
+    ON acls(resource_type, resource_id, principal_type, principal_id, permission);
+CREATE INDEX IF NOT EXISTS idx_acls_resource ON acls(resource_type, resource_id);
+CREATE INDEX IF NOT EXISTS idx_acls_principal ON acls(principal_type, principal_id);
+
+-- ---- 权限扩展（来源：004_permissions.sql） ----
+
+CREATE TABLE IF NOT EXISTS gpu_assignments (
+    id TEXT PRIMARY KEY,
+    node_id TEXT NOT NULL REFERENCES compute_nodes(id) ON DELETE CASCADE,
+    gpu_index INTEGER NOT NULL,
+    user_id TEXT NOT NULL,
+    assigned_by TEXT,
+    assigned_at TEXT NOT NULL,
+    UNIQUE (node_id, gpu_index, user_id)
+);
+CREATE INDEX IF NOT EXISTS idx_gpu_assignments_user ON gpu_assignments(user_id);
+CREATE INDEX IF NOT EXISTS idx_gpu_assignments_gpu ON gpu_assignments(node_id, gpu_index);
+
+ALTER TABLE datasets ADD COLUMN IF NOT EXISTS created_by TEXT;
+ALTER TABLE models ADD COLUMN IF NOT EXISTS created_by TEXT;
+ALTER TABLE trained_models ADD COLUMN IF NOT EXISTS created_by TEXT;
+ALTER TABLE eval_tasks ADD COLUMN IF NOT EXISTS created_by TEXT;
+
+-- ============================================================================
+-- 二、治理表（来源：002_governance.sql）
+-- ============================================================================
+
+CREATE TABLE IF NOT EXISTS tenants (
+    id TEXT PRIMARY KEY,
+    name TEXT NOT NULL,
+    code TEXT,
+    status TEXT DEFAULT 'active',
+    owner_user_id TEXT,
+    quota TEXT,
+    retention_policy_id TEXT,
+    create_time TEXT
+);
+
+CREATE TABLE IF NOT EXISTS approval_templates (
+    id TEXT PRIMARY KEY,
+    name TEXT NOT NULL,
+    steps TEXT,
+    create_time TEXT
+);
+
+CREATE TABLE IF NOT EXISTS approval_instances (
+    id TEXT PRIMARY KEY,
+    template_id TEXT,
+    resource_type TEXT,
+    resource_id TEXT,
+    applicant_id TEXT,
+    status TEXT DEFAULT 'pending',
+    current_step INTEGER DEFAULT 0,
+    create_time TEXT
+);
+
+CREATE TABLE IF NOT EXISTS approval_steps (
+    id TEXT PRIMARY KEY,
+    instance_id TEXT,
+    step_index INTEGER,
+    approver_id TEXT,
+    status TEXT DEFAULT 'pending',
+    comment TEXT,
+    time TEXT
+);
+CREATE INDEX IF NOT EXISTS idx_approval_instances_applicant ON approval_instances(applicant_id, status);
+CREATE INDEX IF NOT EXISTS idx_approval_instances_resource ON approval_instances(resource_type, resource_id, status);
+CREATE INDEX IF NOT EXISTS idx_approval_steps_approver ON approval_steps(approver_id, status);
+
+-- Compatibility for databases created from an older approval_steps definition.
+ALTER TABLE approval_steps ADD COLUMN IF NOT EXISTS id TEXT;
+UPDATE approval_steps
+SET id = 'astep_' || md5(concat_ws(':', instance_id, step_index, coalesce(approver_id, ''), coalesce(time, '')))
+WHERE id IS NULL;
+CREATE UNIQUE INDEX IF NOT EXISTS idx_approval_steps_id ON approval_steps(id);
+
+CREATE TABLE IF NOT EXISTS audit_logs (
+    id TEXT PRIMARY KEY,
+    tenant_id TEXT,
+    project_id TEXT,
+    actor_id TEXT,
+    action TEXT,
+    target_type TEXT,
+    target_id TEXT,
+    detail TEXT,
+    client_ip TEXT,
+    time TEXT
+);
+
+CREATE INDEX IF NOT EXISTS idx_audit_tenant ON audit_logs(tenant_id);
+CREATE INDEX IF NOT EXISTS idx_audit_project ON audit_logs(project_id);
+CREATE INDEX IF NOT EXISTS idx_audit_action ON audit_logs(action);
+CREATE INDEX IF NOT EXISTS idx_audit_time ON audit_logs(time);
+
+CREATE TABLE IF NOT EXISTS retention_policies (
+    id TEXT PRIMARY KEY,
+    name TEXT NOT NULL,
+    scope TEXT,
+    rule TEXT,
+    status TEXT DEFAULT 'active',
+    create_time TEXT,
+    create_by TEXT,
+    updated_at TEXT
+);
+
+-- Compatibility for the earlier retention policy schema
+-- (resource_type/retention_days). Keep legacy columns if they exist.
+ALTER TABLE retention_policies ADD COLUMN IF NOT EXISTS scope TEXT;
+ALTER TABLE retention_policies ADD COLUMN IF NOT EXISTS rule TEXT;
+ALTER TABLE retention_policies ADD COLUMN IF NOT EXISTS status TEXT DEFAULT 'active';
+ALTER TABLE retention_policies ADD COLUMN IF NOT EXISTS create_by TEXT;
+ALTER TABLE retention_policies ADD COLUMN IF NOT EXISTS updated_at TEXT;
+DO $$
+BEGIN
+    IF EXISTS (
+        SELECT 1 FROM information_schema.columns
+        WHERE table_schema='public' AND table_name='retention_policies' AND column_name='resource_type'
+    ) AND EXISTS (
+        SELECT 1 FROM information_schema.columns
+        WHERE table_schema='public' AND table_name='retention_policies' AND column_name='retention_days'
+    ) THEN
+        EXECUTE $migration$
+            UPDATE retention_policies
+            SET scope = COALESCE(scope, resource_type),
+                rule = COALESCE(rule, retention_days::text),
+                status = COALESCE(status, 'active'),
+                updated_at = COALESCE(updated_at, create_time)
+            WHERE scope IS NULL OR rule IS NULL OR status IS NULL OR updated_at IS NULL
+        $migration$;
+    ELSE
+        UPDATE retention_policies
+        SET status = COALESCE(status, 'active'),
+            updated_at = COALESCE(updated_at, create_time)
+        WHERE status IS NULL OR updated_at IS NULL;
+    END IF;
+END $$;
+
+-- ============================================================================
+-- 三、租户配额扩展（来源：003_tenant_quota.sql）
+-- ============================================================================
+
+ALTER TABLE tenants ADD COLUMN IF NOT EXISTS gpu_quota TEXT;
+ALTER TABLE tenants ADD COLUMN IF NOT EXISTS storage_quota TEXT;
+
+-- ============================================================================
+-- 四、模型路径治理（来源：003_model_path_governance.sql）
+--   models.can_train 已在建表语句中声明；以下为兼容旧库的幂等语句。
+-- ============================================================================
+
+ALTER TABLE models ADD COLUMN IF NOT EXISTS can_train INTEGER NOT NULL DEFAULT 0;
+ALTER TABLE trained_models ADD COLUMN IF NOT EXISTS artifact_dir TEXT;
+ALTER TABLE trained_models ADD COLUMN IF NOT EXISTS compute_node_id TEXT;
+ALTER TABLE trained_models ADD COLUMN IF NOT EXISTS compute_node_name TEXT;
+
+-- 按规则推定已有模型的 can_train（新库为空表，此语句为 no-op）
+UPDATE models
+SET can_train = CASE
+    WHEN path IS NOT NULL AND path != '' AND model_source IS NOT NULL AND model_source != 'api' THEN 1
+    ELSE 0
+END;
+
+-- ============================================================================
+-- 五、数据处理（来源：002_data_process.sql，去掉其外层 BEGIN/COMMIT）
+--   数据集扩展列
+-- ============================================================================
+
+ALTER TABLE datasets ADD COLUMN IF NOT EXISTS source_task_id TEXT;
+ALTER TABLE datasets ADD COLUMN IF NOT EXISTS size_bytes BIGINT NOT NULL DEFAULT 0;
+ALTER TABLE datasets ADD COLUMN IF NOT EXISTS record_count BIGINT NOT NULL DEFAULT 0;
+ALTER TABLE datasets ADD COLUMN IF NOT EXISTS metadata TEXT NOT NULL DEFAULT '{}';
+ALTER TABLE datasets ADD COLUMN IF NOT EXISTS tenant_id TEXT;
+ALTER TABLE datasets ADD COLUMN IF NOT EXISTS project_id TEXT;
+ALTER TABLE datasets ADD COLUMN IF NOT EXISTS owner_id TEXT;
+ALTER TABLE datasets ADD COLUMN IF NOT EXISTS created_by TEXT;
+ALTER TABLE datasets ADD COLUMN IF NOT EXISTS created_at TIMESTAMPTZ NOT NULL DEFAULT now();
+ALTER TABLE datasets ADD COLUMN IF NOT EXISTS updated_at TIMESTAMPTZ NOT NULL DEFAULT now();
+ALTER TABLE datasets ADD COLUMN IF NOT EXISTS deleted_at TIMESTAMPTZ;
+
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS storage_object_id TEXT;
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS current_version_id TEXT;
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS size_bytes BIGINT NOT NULL DEFAULT 0;
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS record_count BIGINT NOT NULL DEFAULT 0;
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS file_format VARCHAR(40);
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS checksum_sha256 CHAR(64);
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS version_no INTEGER NOT NULL DEFAULT 1;
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS source_task_id TEXT;
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS tenant_id TEXT;
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS project_id TEXT;
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS created_by TEXT;
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS metadata TEXT NOT NULL DEFAULT '{}';
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS created_at TIMESTAMPTZ NOT NULL DEFAULT now();
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS updated_at TIMESTAMPTZ NOT NULL DEFAULT now();
+ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS deleted_at TIMESTAMPTZ;
+
+-- ---- 数据处理任务 / 源文件 / 预览 / 结果 ----
+
+CREATE TABLE IF NOT EXISTS data_process_tasks (
+    id TEXT PRIMARY KEY,
+    name VARCHAR(150) NOT NULL,
+    description TEXT,
+    status VARCHAR(20) NOT NULL DEFAULT 'pending'
+        CHECK (status IN ('pending', 'running', 'completed', 'failed', 'stopped')),
+    process_type VARCHAR(20) NOT NULL
+        CHECK (process_type IN ('structured', 'unstructured', 'external')),
+    source_dataset_id TEXT REFERENCES datasets(id) ON DELETE SET NULL,
+    output_dataset_id TEXT REFERENCES datasets(id) ON DELETE SET NULL,
+    config TEXT NOT NULL DEFAULT '{}',
+    progress NUMERIC(5,2) NOT NULL DEFAULT 0 CHECK (progress >= 0 AND progress <= 100),
+    input_count BIGINT NOT NULL DEFAULT 0 CHECK (input_count >= 0),
+    output_count BIGINT NOT NULL DEFAULT 0 CHECK (output_count >= 0),
+    filtered_count BIGINT NOT NULL DEFAULT 0 CHECK (filtered_count >= 0),
+    duplicate_count BIGINT NOT NULL DEFAULT 0 CHECK (duplicate_count >= 0),
+    error_count BIGINT NOT NULL DEFAULT 0 CHECK (error_count >= 0),
+    failure_reason TEXT,
+    generation_run_id TEXT,
+    results_confirmed BOOLEAN NOT NULL DEFAULT TRUE,
+    workflow_step VARCHAR(20) NOT NULL DEFAULT 'create'
+        CHECK (workflow_step IN ('create', 'model', 'upload', 'preview', 'generate', 'results')),
+    preview_status VARCHAR(20) NOT NULL DEFAULT 'idle'
+        CHECK (preview_status IN ('idle', 'queued', 'running', 'completed', 'failed', 'cancelled')),
+    preview_progress NUMERIC(5,2) NOT NULL DEFAULT 0
+        CHECK (preview_progress >= 0 AND preview_progress <= 100),
+    preview_run_id TEXT,
+    preview_failure_reason TEXT,
+    preview_total_files INTEGER NOT NULL DEFAULT 0 CHECK (preview_total_files >= 0),
+    preview_completed_files INTEGER NOT NULL DEFAULT 0 CHECK (preview_completed_files >= 0),
+    tenant_id TEXT,
+    project_id TEXT,
+    owner_id TEXT,
+    approval_status VARCHAR(30) NOT NULL DEFAULT 'not_required',
+    created_by TEXT,
+    updated_by TEXT,
+    deleted_by TEXT,
+    started_at TIMESTAMPTZ,
+    completed_at TIMESTAMPTZ,
+    created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+    updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+    deleted_at TIMESTAMPTZ
+);
+
+ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS generation_run_id TEXT;
+ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS results_confirmed BOOLEAN NOT NULL DEFAULT TRUE;
+ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS workflow_step VARCHAR(20);
+ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_status VARCHAR(20);
+ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_progress NUMERIC(5,2);
+ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_run_id TEXT;
+ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_failure_reason TEXT;
+ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_total_files INTEGER;
+ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_completed_files INTEGER;
+
+ALTER TABLE data_process_tasks ALTER COLUMN workflow_step SET DEFAULT 'create';
+ALTER TABLE data_process_tasks ALTER COLUMN workflow_step SET NOT NULL;
+ALTER TABLE data_process_tasks ALTER COLUMN preview_status SET DEFAULT 'idle';
+ALTER TABLE data_process_tasks ALTER COLUMN preview_status SET NOT NULL;
+ALTER TABLE data_process_tasks ALTER COLUMN preview_progress SET DEFAULT 0;
+ALTER TABLE data_process_tasks ALTER COLUMN preview_progress SET NOT NULL;
+ALTER TABLE data_process_tasks ALTER COLUMN preview_total_files SET DEFAULT 0;
+ALTER TABLE data_process_tasks ALTER COLUMN preview_total_files SET NOT NULL;
+ALTER TABLE data_process_tasks ALTER COLUMN preview_completed_files SET DEFAULT 0;
+ALTER TABLE data_process_tasks ALTER COLUMN preview_completed_files SET NOT NULL;
+
+CREATE UNIQUE INDEX IF NOT EXISTS uq_data_process_tasks_name_alive
+    ON data_process_tasks(name) WHERE deleted_at IS NULL;
+CREATE INDEX IF NOT EXISTS idx_data_process_tasks_scope_status
+    ON data_process_tasks(tenant_id, project_id, status, created_at DESC)
+    WHERE deleted_at IS NULL;
+CREATE INDEX IF NOT EXISTS idx_data_process_tasks_creator_created
+    ON data_process_tasks(created_by, created_at DESC) WHERE deleted_at IS NULL;
+
+CREATE TABLE IF NOT EXISTS data_process_source_files (
+    id TEXT PRIMARY KEY,
+    task_id TEXT NOT NULL REFERENCES data_process_tasks(id) ON DELETE CASCADE,
+    storage_object_id TEXT,
+    name TEXT NOT NULL,
+    size_bytes BIGINT NOT NULL DEFAULT 0 CHECK (size_bytes >= 0),
+    record_count BIGINT NOT NULL DEFAULT 0 CHECK (record_count >= 0),
+    file_format VARCHAR(40),
+    checksum_sha256 CHAR(64) NOT NULL,
+    version_no INTEGER NOT NULL DEFAULT 1 CHECK (version_no > 0),
+    content TEXT NOT NULL,
+    content_preview TEXT,
+    metadata TEXT NOT NULL DEFAULT '{}',
+    tenant_id TEXT,
+    project_id TEXT,
+    created_by TEXT,
+    created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+    updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+    deleted_at TIMESTAMPTZ
+);
+
+CREATE INDEX IF NOT EXISTS idx_data_process_source_files_task
+    ON data_process_source_files(task_id, created_at) WHERE deleted_at IS NULL;
+CREATE UNIQUE INDEX IF NOT EXISTS uq_data_process_source_checksum_alive
+    ON data_process_source_files(task_id, checksum_sha256) WHERE deleted_at IS NULL;
+
+CREATE TABLE IF NOT EXISTS data_process_preview_items (
+    id TEXT PRIMARY KEY,
+    task_id TEXT NOT NULL REFERENCES data_process_tasks(id) ON DELETE CASCADE,
+    source_file_id TEXT REFERENCES data_process_source_files(id) ON DELETE CASCADE,
+    original_content TEXT NOT NULL DEFAULT '',
+    edited_content TEXT NOT NULL DEFAULT '',
+    source_start INTEGER CHECK (source_start IS NULL OR source_start >= 0),
+    source_end INTEGER CHECK (source_end IS NULL OR source_end >= 0),
+    source_start_line INTEGER CHECK (source_start_line IS NULL OR source_start_line > 0),
+    source_end_line INTEGER CHECK (source_end_line IS NULL OR source_end_line > 0),
+    token_count INTEGER NOT NULL DEFAULT 0 CHECK (token_count >= 0),
+    status VARCHAR(20) NOT NULL DEFAULT 'original'
+        CHECK (status IN ('original', 'modified', 'manual', 'invalid')),
+    quality_score TEXT NOT NULL DEFAULT '{}',
+    created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+    updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+    CHECK (source_start IS NULL OR source_end IS NULL OR source_end >= source_start),
+    CHECK (source_start_line IS NULL OR source_end_line IS NULL OR source_end_line >= source_start_line)
+);
+
+CREATE INDEX IF NOT EXISTS idx_data_process_preview_task_file
+    ON data_process_preview_items(task_id, source_file_id, created_at);
+
+CREATE TABLE IF NOT EXISTS data_process_results (
+    id TEXT PRIMARY KEY,
+    task_id TEXT NOT NULL REFERENCES data_process_tasks(id) ON DELETE CASCADE,
+    preview_item_id TEXT REFERENCES data_process_preview_items(id) ON DELETE SET NULL,
+    instruction TEXT NOT NULL,
+    input TEXT NOT NULL DEFAULT '',
+    output TEXT NOT NULL,
+    chosen TEXT NOT NULL DEFAULT '',
+    rejected TEXT NOT NULL DEFAULT '',
+    original_instruction TEXT,
+    original_input TEXT,
+    original_output TEXT,
+    original_chosen TEXT,
+    original_rejected TEXT,
+    status VARCHAR(20) NOT NULL DEFAULT 'valid'
+        CHECK (status IN ('valid', 'modified', 'invalid')),
+    error TEXT,
+    split VARCHAR(20) CHECK (split IS NULL OR split IN ('train', 'validation', 'test')),
+    quality_score TEXT NOT NULL DEFAULT '{}',
+    created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+    updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+
+-- Keep existing databases compatible with the current generation result model.
+ALTER TABLE data_process_results ADD COLUMN IF NOT EXISTS chosen TEXT NOT NULL DEFAULT '';
+ALTER TABLE data_process_results ADD COLUMN IF NOT EXISTS rejected TEXT NOT NULL DEFAULT '';
+ALTER TABLE data_process_results ADD COLUMN IF NOT EXISTS original_chosen TEXT;
+ALTER TABLE data_process_results ADD COLUMN IF NOT EXISTS original_rejected TEXT;
+
+CREATE INDEX IF NOT EXISTS idx_data_process_results_task_status
+    ON data_process_results(task_id, status, id);
+CREATE INDEX IF NOT EXISTS idx_data_process_results_task_split
+    ON data_process_results(task_id, split);
+
+-- ---- 数据集版本 / 记录 ----
+
+CREATE TABLE IF NOT EXISTS dataset_file_versions (
+    id TEXT PRIMARY KEY,
+    dataset_file_id TEXT NOT NULL REFERENCES dataset_files(id) ON DELETE CASCADE,
+    version_no INTEGER NOT NULL CHECK (version_no > 0),
+    storage_object_id TEXT NOT NULL,
+    content_preview TEXT,
+    description TEXT,
+    base_version_id TEXT REFERENCES dataset_file_versions(id) ON DELETE SET NULL,
+    size_bytes BIGINT NOT NULL DEFAULT 0 CHECK (size_bytes >= 0),
+    record_count BIGINT NOT NULL DEFAULT 0 CHECK (record_count >= 0),
+    checksum_sha256 CHAR(64) NOT NULL,
+    source_task_id TEXT REFERENCES data_process_tasks(id) ON DELETE SET NULL,
+    metadata TEXT NOT NULL DEFAULT '{}',
+    created_by TEXT,
+    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+
+ALTER TABLE dataset_file_versions ADD COLUMN IF NOT EXISTS source_task_id TEXT;
+ALTER TABLE dataset_file_versions ADD COLUMN IF NOT EXISTS metadata TEXT NOT NULL DEFAULT '{}';
+CREATE UNIQUE INDEX IF NOT EXISTS uq_dataset_file_versions_no_002
+    ON dataset_file_versions(dataset_file_id, version_no);
+CREATE INDEX IF NOT EXISTS idx_dataset_file_versions_source_task_002
+    ON dataset_file_versions(source_task_id) WHERE source_task_id IS NOT NULL;
+
+CREATE TABLE IF NOT EXISTS dataset_records (
+    id TEXT PRIMARY KEY,
+    dataset_id TEXT NOT NULL REFERENCES datasets(id) ON DELETE CASCADE,
+    dataset_file_id TEXT REFERENCES dataset_files(id) ON DELETE CASCADE,
+    version_id TEXT REFERENCES dataset_file_versions(id) ON DELETE CASCADE,
+    line_no INTEGER,
+    split VARCHAR(20) CHECK (split IS NULL OR split IN ('train', 'validation', 'test')),
+    instruction TEXT,
+    input TEXT,
+    output TEXT,
+    raw TEXT NOT NULL DEFAULT '{}',
+    status VARCHAR(20) NOT NULL DEFAULT 'valid'
+        CHECK (status IN ('valid', 'modified', 'invalid')),
+    source_task_id TEXT REFERENCES data_process_tasks(id) ON DELETE SET NULL,
+    source_result_id TEXT REFERENCES data_process_results(id) ON DELETE SET NULL,
+    preview_item_id TEXT REFERENCES data_process_preview_items(id) ON DELETE SET NULL,
+    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+
+ALTER TABLE dataset_records ADD COLUMN IF NOT EXISTS source_task_id TEXT;
+ALTER TABLE dataset_records ADD COLUMN IF NOT EXISTS source_result_id TEXT;
+ALTER TABLE dataset_records ADD COLUMN IF NOT EXISTS preview_item_id TEXT;
+CREATE INDEX IF NOT EXISTS idx_dataset_records_dataset_002
+    ON dataset_records(dataset_id, id);
+CREATE INDEX IF NOT EXISTS idx_dataset_records_source_task_002
+    ON dataset_records(source_task_id, source_result_id);
+CREATE INDEX IF NOT EXISTS idx_datasets_source_task_002
+    ON datasets(source_task_id) WHERE source_task_id IS NOT NULL;
+CREATE INDEX IF NOT EXISTS idx_dataset_files_source_task_002
+    ON dataset_files(source_task_id) WHERE source_task_id IS NOT NULL;
+
+-- ============================================================================
+-- 六、数据转换任务（data_convert_tasks）
+--   运行时 router（app/modules/data_convert/router.py）引用但原脚本缺失，本文件补齐。
+-- ============================================================================
+
+CREATE TABLE IF NOT EXISTS data_convert_tasks (
+    id TEXT PRIMARY KEY,
+    name TEXT NOT NULL,
+    description TEXT,
+    output_filename TEXT DEFAULT 'converted-data.jsonl',
+    status TEXT NOT NULL DEFAULT 'pending',
+    input_count INTEGER NOT NULL DEFAULT 0,
+    output_count INTEGER NOT NULL DEFAULT 0,
+    error_message TEXT,
+    create_time TEXT NOT NULL DEFAULT (to_char(now(), 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"')),
+    update_time TEXT NOT NULL DEFAULT (to_char(now(), 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"')),
+    deleted_at TIMESTAMPTZ
+);
+CREATE INDEX IF NOT EXISTS idx_data_convert_tasks_status ON data_convert_tasks(status);
+CREATE INDEX IF NOT EXISTS idx_data_convert_tasks_create_time ON data_convert_tasks(create_time DESC);
+
+-- ============================================================================
+-- 七、种子数据：初始管理员 / 操作员
+--   应用首次启动（ensure_seed_data）也会自动创建；此处提供以便脱离应用直接初始化。
+--   密码：admin / admin123，operator / operator123（上线前请改密）。
+-- ============================================================================
+
+INSERT INTO users
+    (id, username, password_hash, display_name, role, status, permissions, create_time, protected)
+VALUES
+    (
+        'u_admin', 'admin', 'pbkdf2_sha256$390000$ygft_init_salt_admin$2b6f31f22968c4f5a30bcf0acf066b7a0f58d4773d15c5ab898ba715ea87b5bd',
+        'Platform Admin', 'admin', 'active',
+        '["dashboard","fine-tune","model-eval","model-inference","model-manage","dataset","data-process","data-convert","compute","hardware","logs","user-settings"]',
+        to_char(now(), 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"'), 1
+    ),
+    (
+        'u_operator', 'operator', 'pbkdf2_sha256$390000$ygft_init_salt_op$525bf35d02ed26f37952cbd6862b0ae358b9d1a7fa0cbbf0217aa2b5dd544125',
+        'Platform Operator', 'operator', 'active',
+        '["dashboard","fine-tune","model-eval","model-inference","model-manage","dataset","data-process","data-convert","compute","hardware","logs"]',
+        to_char(now(), 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"'), 0
+    )
+ON CONFLICT (username) DO NOTHING;
+
+COMMIT;

backend/app/db/sql/001_platform_runtime.sql

@@ -234,6 +234,39 @@ CREATE TABLE IF NOT EXISTS resource_sync_jobs (
     completed_at TEXT
 );
 
+CREATE TABLE IF NOT EXISTS storage_objects (
+    id TEXT PRIMARY KEY,
+    resource_type TEXT NOT NULL,
+    resource_id TEXT NOT NULL,
+    version_id TEXT NOT NULL,
+    bucket TEXT NOT NULL,
+    object_key TEXT NOT NULL,
+    file_name TEXT,
+    content_type TEXT,
+    checksum_sha256 TEXT,
+    byte_size BIGINT NOT NULL DEFAULT 0,
+    status TEXT NOT NULL DEFAULT 'pending',
+    created_by TEXT,
+    create_time TEXT NOT NULL,
+    UNIQUE (resource_type, resource_id, version_id, object_key)
+);
+
+CREATE TABLE IF NOT EXISTS storage_cache_jobs (
+    id TEXT PRIMARY KEY,
+    storage_object_id TEXT NOT NULL REFERENCES storage_objects(id) ON DELETE CASCADE,
+    node_id TEXT NOT NULL REFERENCES compute_nodes(id) ON DELETE CASCADE,
+    direction TEXT NOT NULL,
+    status TEXT NOT NULL,
+    progress INTEGER NOT NULL DEFAULT 0,
+    local_path TEXT,
+    error TEXT,
+    create_time TEXT NOT NULL,
+    completed_at TEXT
+);
+
+CREATE INDEX IF NOT EXISTS idx_storage_objects_resource ON storage_objects(resource_type, resource_id, version_id);
+CREATE INDEX IF NOT EXISTS idx_storage_cache_jobs_node_status ON storage_cache_jobs(node_id, status);
+
 CREATE TABLE IF NOT EXISTS eval_tasks (
     id TEXT PRIMARY KEY,
     name TEXT NOT NULL,
@@ -319,9 +352,14 @@ CREATE TABLE IF NOT EXISTS roles (
 CREATE TABLE IF NOT EXISTS sessions (
     id TEXT PRIMARY KEY,
     user_id TEXT NOT NULL,
-    issued_at TEXT NOT NULL,
+    username TEXT,
-    expires_at TEXT NOT NULL,
+    login_at TEXT,
-    ip TEXT
+    logout_at TEXT,
+    duration_seconds INTEGER,
+    issued_at TEXT,
+    expires_at TEXT,
+    ip TEXT,
+    create_time TEXT
 );
 
 CREATE TABLE IF NOT EXISTS acls (

backend/app/db/sql/002_data_process.sql

@@ -268,9 +268,13 @@ CREATE TABLE IF NOT EXISTS data_process_results (
     instruction TEXT NOT NULL,
     input TEXT NOT NULL DEFAULT '',
     output TEXT NOT NULL,
+    chosen TEXT NOT NULL DEFAULT '',
+    rejected TEXT NOT NULL DEFAULT '',
     original_instruction TEXT,
     original_input TEXT,
     original_output TEXT,
+    original_chosen TEXT,
+    original_rejected TEXT,
     status VARCHAR(20) NOT NULL DEFAULT 'valid'
         CHECK (status IN ('valid', 'modified', 'invalid')),
     error TEXT,
@@ -280,6 +284,11 @@ CREATE TABLE IF NOT EXISTS data_process_results (
     updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
 );
 
+ALTER TABLE data_process_results ADD COLUMN IF NOT EXISTS chosen TEXT NOT NULL DEFAULT '';
+ALTER TABLE data_process_results ADD COLUMN IF NOT EXISTS rejected TEXT NOT NULL DEFAULT '';
+ALTER TABLE data_process_results ADD COLUMN IF NOT EXISTS original_chosen TEXT;
+ALTER TABLE data_process_results ADD COLUMN IF NOT EXISTS original_rejected TEXT;
+
 CREATE INDEX IF NOT EXISTS idx_data_process_results_task_status
     ON data_process_results(task_id, status, id);
 CREATE INDEX IF NOT EXISTS idx_data_process_results_task_split

backend/app/db/sql/002_governance.sql

@@ -51,10 +51,18 @@ CREATE TABLE IF NOT EXISTS audit_logs (
     time TEXT
 );
 
-CREATE INDEX IF NOT EXISTS idx_audit_tenant ON audit_logs(tenant_id);
+-- 幂等升级 audit_logs 表：新增字段（已存在则跳过）
-CREATE INDEX IF NOT EXISTS idx_audit_project ON audit_logs(project_id);
+ALTER TABLE audit_logs ADD COLUMN IF NOT EXISTS trace_id TEXT;
-CREATE INDEX IF NOT EXISTS idx_audit_action ON audit_logs(action);
+ALTER TABLE audit_logs ADD COLUMN IF NOT EXISTS request_method TEXT;
-CREATE INDEX IF NOT EXISTS idx_audit_time ON audit_logs(time);
+ALTER TABLE audit_logs ADD COLUMN IF NOT EXISTS request_path TEXT;
+ALTER TABLE audit_logs ADD COLUMN IF NOT EXISTS status_code INTEGER;
+ALTER TABLE audit_logs ADD COLUMN IF NOT EXISTS duration_ms REAL;
+ALTER TABLE audit_logs ADD COLUMN IF NOT EXISTS extra JSONB;
+
+-- 索引（已存在则跳过）
+CREATE INDEX IF NOT EXISTS idx_audit_trace ON audit_logs(trace_id);
+CREATE INDEX IF NOT EXISTS idx_audit_actor_time ON audit_logs(actor_id, time);
+CREATE INDEX IF NOT EXISTS idx_audit_target ON audit_logs(target_type, target_id);
 
 CREATE TABLE IF NOT EXISTS retention_policies (
     id TEXT PRIMARY KEY,
@@ -66,3 +74,38 @@ CREATE TABLE IF NOT EXISTS retention_policies (
     create_by TEXT,
     updated_at TEXT
 );
+
+-- ===================== 操作日志表 =====================
+-- 记录用户在每个业务模块的详细操作（成功/失败、报错信息等）
+CREATE TABLE IF NOT EXISTS operation_logs (
+    id TEXT PRIMARY KEY,
+    user_id TEXT,               -- 操作用户 ID
+    username TEXT,              -- 操作用户名（冗余，便于查询）
+    module TEXT,                -- 业务模块：fine-tune/model-eval/model-inference/dataset/data-process/data-convert/model-manage
+    action TEXT,                -- 具体动作：create/start/stop/delete/upload/convert 等
+    target_type TEXT,           -- 资源类型
+    target_id TEXT,             -- 资源 ID
+    target_name TEXT,           -- 资源名称（便于阅读）
+    status TEXT NOT NULL,       -- success / failure
+    error_message TEXT,         -- 失败时的报错信息
+    detail TEXT,                -- 操作详情 JSON（参数摘要）
+    client_ip TEXT,             -- 客户端 IP
+    request_method TEXT,        -- HTTP 方法
+    request_path TEXT,          -- 请求路径
+    trace_id TEXT,              -- 链路追踪 ID
+    duration_ms REAL,           -- 耗时(ms)
+    create_time TEXT            -- 操作时间
+);
+
+-- 幂等升级 operation_logs 表：新增字段（已存在则跳过）
+ALTER TABLE operation_logs ADD COLUMN IF NOT EXISTS error_type TEXT;       -- 异常类型：RuntimeError / ValueError / ConnectionError
+ALTER TABLE operation_logs ADD COLUMN IF NOT EXISTS error_traceback TEXT; -- 完整异常堆栈
+ALTER TABLE operation_logs ADD COLUMN IF NOT EXISTS func_name TEXT;       -- 出错的函数名
+
+-- 索引
+CREATE INDEX IF NOT EXISTS idx_op_user_time ON operation_logs(user_id, create_time DESC);
+CREATE INDEX IF NOT EXISTS idx_op_module_time ON operation_logs(module, create_time DESC);
+CREATE INDEX IF NOT EXISTS idx_op_status ON operation_logs(status, create_time DESC);
+CREATE INDEX IF NOT EXISTS idx_op_action ON operation_logs(action);
+CREATE INDEX IF NOT EXISTS idx_op_create_time ON operation_logs(create_time DESC);
+CREATE INDEX IF NOT EXISTS idx_op_error_type ON operation_logs(error_type);

backend/app/db/sql/003_model_path_governance.sql

@@ -16,3 +16,5 @@ END;
 
 -- 3. 给 trained_models 增加 artifact_dir（训练产物目录扫描结果目录）
 ALTER TABLE trained_models ADD COLUMN IF NOT EXISTS artifact_dir TEXT;
+ALTER TABLE trained_models ADD COLUMN IF NOT EXISTS compute_node_id TEXT;
+ALTER TABLE trained_models ADD COLUMN IF NOT EXISTS compute_node_name TEXT;

backend/app/db/sql/004_permissions.sql

@@ -0,0 +1,22 @@
+-- ============================================================
+-- 权限体系扩展：GPU 分配表 + 资源所有权字段
+-- ============================================================
+
+-- GPU 分配表：管理员指定哪些用户可以使用哪些 GPU 卡
+CREATE TABLE IF NOT EXISTS gpu_assignments (
+    id          TEXT PRIMARY KEY,
+    node_id     TEXT NOT NULL REFERENCES compute_nodes(id) ON DELETE CASCADE,
+    gpu_index   INTEGER NOT NULL,
+    user_id     TEXT NOT NULL,
+    assigned_by TEXT,
+    assigned_at TEXT NOT NULL,
+    UNIQUE (node_id, gpu_index, user_id)
+);
+CREATE INDEX IF NOT EXISTS idx_gpu_assignments_user ON gpu_assignments(user_id);
+CREATE INDEX IF NOT EXISTS idx_gpu_assignments_gpu ON gpu_assignments(node_id, gpu_index);
+
+-- 资源所有权字段：用户创建的数据集/模型/训练产物/评测任务
+ALTER TABLE datasets ADD COLUMN IF NOT EXISTS created_by TEXT;
+ALTER TABLE models ADD COLUMN IF NOT EXISTS created_by TEXT;
+ALTER TABLE trained_models ADD COLUMN IF NOT EXISTS created_by TEXT;
+ALTER TABLE eval_tasks ADD COLUMN IF NOT EXISTS created_by TEXT;

backend/app/main.py

@@ -5,7 +5,7 @@ from fastapi import FastAPI
 from fastapi.middleware.cors import CORSMiddleware
 
 from app.api.v1.router import api_router
-from app.core.config import get_settings
+from app.core.config import docs_kwargs, get_settings
 from app.core.logging import configure_logging, setup_request_logging
 from app.workers.compute_poller import run_compute_poller
 
@@ -14,7 +14,7 @@ def create_app() -> FastAPI:
     settings = get_settings()
     configure_logging(settings)
 
-    app = FastAPI(title=settings.app_name)
+    app = FastAPI(title=settings.app_name, **docs_kwargs(settings.enable_docs))
     app.add_middleware(
         CORSMiddleware,
         allow_origins=settings.cors_allow_origins,

backend/app/modules/approval/router.py

@@ -1,21 +1,23 @@
 from __future__ import annotations
 
-from fastapi import APIRouter, Body
+from fastapi import APIRouter, Body, Depends
 from typing import Any
 
 from app.api.v1.endpoints.platform import ok, fail
 from app.db.platform_store import get_platform_store
+from app.core.auth import get_current_user, is_admin
 
 router = APIRouter(prefix="/approvals", tags=["approval"])
 
 
 @router.get("/templates")
-def list_templates() -> dict[str, Any]:
+def list_templates(current_user: dict = Depends(get_current_user)) -> dict[str, Any]:
     return ok(get_platform_store().approval_templates())
 
 
 @router.post("/templates")
-def create_template(payload: dict[str, Any] = Body(...)) -> dict[str, Any]:
+def create_template(payload: dict[str, Any] = Body(...), current_user: dict = Depends(get_current_user)) -> dict[str, Any]:
+    if not is_admin(current_user): raise fail(403, "admin permission required")
     if not payload.get("name"):
         raise fail(400, "name 必填")
     return ok(get_platform_store().create_approval_template(payload))
@@ -30,7 +32,8 @@ def get_template(template_id: str) -> dict[str, Any]:
 
 
 @router.put("/templates/{template_id}")
-def update_template(template_id: str, payload: dict[str, Any] = Body(...)) -> dict[str, Any]:
+def update_template(template_id: str, payload: dict[str, Any] = Body(...), current_user: dict = Depends(get_current_user)) -> dict[str, Any]:
+    if not is_admin(current_user): raise fail(403, "admin permission required")
     try:
         return ok(get_platform_store().update_approval_template(template_id, payload))
     except KeyError:
@@ -38,7 +41,8 @@ def update_template(template_id: str, payload: dict[str, Any] = Body(...)) -> di
 
 
 @router.delete("/templates/{template_id}")
-def delete_template(template_id: str) -> dict[str, Any]:
+def delete_template(template_id: str, current_user: dict = Depends(get_current_user)) -> dict[str, Any]:
+    if not is_admin(current_user): raise fail(403, "admin permission required")
     try:
         return ok(get_platform_store().delete_approval_template(template_id))
     except KeyError:
@@ -46,13 +50,15 @@ def delete_template(template_id: str) -> dict[str, Any]:
 
 
 @router.get("")
-def list_instances(status: str | None = None) -> dict[str, Any]:
+def list_instances(status: str | None = None, current_user: dict = Depends(get_current_user)) -> dict[str, Any]:
-    return ok(get_platform_store().approval_instances(status=status))
+    items = get_platform_store().approval_instances(status=status)
+    return ok(items if is_admin(current_user) else [item for item in items if item.get("applicant_id") == current_user.get("id")])
 
 
 @router.post("")
-def create_instance(payload: dict[str, Any] = Body(...)) -> dict[str, Any]:
+def create_instance(payload: dict[str, Any] = Body(...), current_user: dict = Depends(get_current_user)) -> dict[str, Any]:
-    for field in ("resource_type", "resource_id", "applicant_id"):
+    payload["applicant_id"] = current_user.get("id")
+    for field in ("resource_type", "resource_id"):
         if not payload.get(field):
             raise fail(400, f"{field} 必填")
     try:
@@ -62,9 +68,12 @@ def create_instance(payload: dict[str, Any] = Body(...)) -> dict[str, Any]:
 
 
 @router.get("/{instance_id}")
-def get_instance(instance_id: str) -> dict[str, Any]:
+def get_instance(instance_id: str, current_user: dict = Depends(get_current_user)) -> dict[str, Any]:
     try:
-        return ok(get_platform_store().approval_instance(instance_id))
+        item = get_platform_store().approval_instance(instance_id)
+        if not is_admin(current_user) and item.get("applicant_id") != current_user.get("id"):
+            raise fail(403, "no permission to access approval")
+        return ok(item)
     except KeyError:
         raise fail(404, "instance not found")
 

backend/app/modules/compute_gateway/client.py

@@ -192,6 +192,27 @@ class ComputeNodeClient:
             response.raise_for_status()
             return _unwrap_dict(response.json())
 
+    async def prepare_cache(self, payload: dict[str, Any]) -> dict[str, Any]:
+        async with httpx.AsyncClient(timeout=httpx.Timeout(900, connect=30), headers=self.headers()) as client:
+            response = await client.post(
+                _join_url(self.api_base_url, f"{self.route_prefix}/compute/cache/prepare"),
+                json=payload,
+            )
+            response.raise_for_status()
+            return _unwrap_dict(response.json())
+
+    async def cache_status(self, resource_id: str, version_id: str | None = None) -> dict[str, Any]:
+        params = {"resource_id": resource_id}
+        if version_id:
+            params["version_id"] = version_id
+        async with httpx.AsyncClient(timeout=self.timeout, headers=self.headers()) as client:
+            response = await client.get(
+                _join_url(self.api_base_url, f"{self.route_prefix}/compute/cache/status"),
+                params=params,
+            )
+            response.raise_for_status()
+            return _unwrap_dict(response.json())
+
     async def _request(
         self,
         method: str,
@@ -245,3 +266,8 @@ class ComputeNodeClient:
             )
             response.raise_for_status()
             return _unwrap_dict(response.json())
+
+    async def upload_file_to_url(self, path: str, upload_url: str, object_key: str = "", content_type: str = "application/octet-stream") -> dict[str, Any]:
+        return await self._request("POST", "/compute/files/upload-to-url", json_data={
+            "path": path, "upload_url": upload_url, "object_key": object_key, "content_type": content_type,
+        }, timeout=900)

backend/app/modules/data_convert/__init__.py

@@ -0,0 +1,3 @@
+from .router import router
+
+__all__ = ["router"]

backend/app/modules/data_convert/router.py

@@ -0,0 +1,378 @@
+from __future__ import annotations
+
+import json
+import os
+from pathlib import Path
+from typing import Any
+
+from fastapi import APIRouter, Body, Depends, File, UploadFile
+from fastapi.responses import FileResponse
+
+from app.api.v1.endpoints.platform import ok, fail
+from app.core.auth import get_current_user, is_admin
+from app.core.op_log import op_log, OpModule, OpAction
+from app.db.platform_store import get_platform_store, new_id
+
+
+router = APIRouter(prefix="/data-convert", tags=["data-convert"])
+
+# 存储根目录
+STORAGE_ROOT = Path(__file__).resolve().parents[3] / "storage" / "data-convert"
+
+
+def _safe_output_filename(value: Any) -> str:
+    """输出文件名白名单校验：仅允许普通文件名，阻断 ``../``、``/``、``\\`` 等路径穿越。
+
+    转换结果始终写入 ``STORAGE_ROOT/<task_id>/output/<output_filename>``，
+    若文件名可被注入路径分隔符，将导致任意文件读写/删除。
+    """
+    name = str(value or "converted-data.jsonl").strip()
+    if (
+        not name
+        or name in {".", ".."}
+        or name != Path(name).name
+        or "/" in name
+        or "\\" in name
+        or any(ord(character) < 32 or ord(character) == 127 for character in name)
+    ):
+        raise fail(400, "output filename must be a plain file name")
+    return name
+
+
+def _task_output_path(task: dict[str, Any]) -> Path:
+    """返回经过白名单校验的转换输出文件路径（始终位于任务 output 目录内）。"""
+    return _output_dir(task["id"]) / _safe_output_filename(task.get("output_filename"))
+
+
+def _task_dir(task_id: str) -> Path:
+    return STORAGE_ROOT / task_id
+
+
+def _input_dir(task_id: str) -> Path:
+    return _task_dir(task_id) / "input"
+
+
+def _output_dir(task_id: str) -> Path:
+    return _task_dir(task_id) / "output"
+
+
+@router.get("")
+def list_tasks(
+    page: int = 1,
+    page_size: int = 20,
+    current_user: dict = Depends(get_current_user),
+) -> dict[str, Any]:
+    store = get_platform_store()
+    with store.connect() as conn:
+        if is_admin(current_user):
+            # 管理员可见全部
+            rows = conn.execute(
+                "SELECT * FROM data_convert_tasks WHERE deleted_at IS NULL "
+                "ORDER BY create_time DESC LIMIT %s OFFSET %s",
+                (page_size, (page - 1) * page_size),
+            ).fetchall()
+            total = conn.execute(
+                "SELECT COUNT(*) FROM data_convert_tasks WHERE deleted_at IS NULL"
+            ).fetchone()[0]
+        else:
+            # 普通用户只能看到自己创建的
+            user_id = current_user.get("id")
+            rows = conn.execute(
+                "SELECT * FROM data_convert_tasks WHERE deleted_at IS NULL AND created_by=%s "
+                "ORDER BY create_time DESC LIMIT %s OFFSET %s",
+                (user_id, page_size, (page - 1) * page_size),
+            ).fetchall()
+            total = conn.execute(
+                "SELECT COUNT(*) FROM data_convert_tasks WHERE deleted_at IS NULL AND created_by=%s",
+                (user_id,)
+            ).fetchone()[0]
+    return ok({"items": [dict(r) for r in rows], "total": total})
+
+
+@router.post("")
+@op_log(module=OpModule.DATA_CONVERT, action=OpAction.CREATE, target_type="convert_task", target_name_param="name")
+def create_task(
+    payload: dict[str, Any] = Body(...),
+    current_user: dict = Depends(get_current_user),
+) -> dict[str, Any]:
+    name = str(payload.get("name") or "").strip()
+    if not name:
+        raise fail(400, "name is required")
+    task_id = new_id("dct")
+    output_filename = _safe_output_filename(payload.get("output_filename"))
+    description = str(payload.get("description") or "").strip()
+    user_id = current_user.get("id")
+    store = get_platform_store()
+    with store.connect() as conn:
+        conn.execute(
+            "INSERT INTO data_convert_tasks (id, name, description, output_filename, created_by) "
+            "VALUES (%s, %s, %s, %s, %s)",
+            (task_id, name, description, output_filename, user_id),
+        )
+    # 创建目录
+    _input_dir(task_id).mkdir(parents=True, exist_ok=True)
+    _output_dir(task_id).mkdir(parents=True, exist_ok=True)
+    return ok(_get_task(task_id))
+
+
+@router.get("/{task_id}")
+def get_task(
+    task_id: str,
+    current_user: dict = Depends(get_current_user),
+) -> dict[str, Any]:
+    task = _get_task(task_id)
+    if not task:
+        raise fail(404, "task not found")
+    # 附加输入文件列表
+    input_dir = _input_dir(task_id)
+    files = []
+    if input_dir.exists():
+        for f in sorted(input_dir.iterdir()):
+            if f.is_file():
+                files.append({"name": f.name, "size": f.stat().st_size})
+    task["input_files"] = files
+    return ok(task)
+
+
+@router.post("/{task_id}/source-files")
+@op_log(module=OpModule.DATA_CONVERT, action=OpAction.UPLOAD, target_type="convert_task", target_name_param="task_id")
+async def upload_source_files(
+    task_id: str,
+    files: list[UploadFile] = File(...),
+    current_user: dict = Depends(get_current_user),
+) -> dict[str, Any]:
+    task = _get_task(task_id)
+    if not task:
+        raise fail(404, "task not found")
+    if task["status"] not in ("pending", "uploaded"):
+        raise fail(400, "task is not editable")
+    input_dir = _input_dir(task_id)
+    input_dir.mkdir(parents=True, exist_ok=True)
+    staged = []
+    for upload in files:
+        name = Path(upload.filename or "input.json").name
+        if not name.lower().endswith(".json"):
+            raise fail(415, f"only JSON files are supported: {name}")
+        target = input_dir / name
+        content = await upload.read()
+        target.write_bytes(content)
+        staged.append({"name": name, "size": len(content)})
+    store = get_platform_store()
+    # 标记上传完成
+    with store.connect() as conn:
+        conn.execute(
+            "UPDATE data_convert_tasks SET status='uploaded', update_time=NOW() WHERE id=%s",
+            (task_id,),
+        )
+    # 自动转换并导入数据集
+    try:
+        output_dir = _output_dir(task_id)
+        output_dir.mkdir(parents=True, exist_ok=True)
+        output_path = _task_output_path(task)
+        # 清空旧输出（如果重新上传）
+        if output_path.exists():
+            output_path.unlink()
+        input_count = 0
+        output_count = 0
+        for json_file in sorted(input_dir.iterdir()):
+            if not json_file.is_file() or not json_file.name.lower().endswith(".json"):
+                continue
+            input_count += 1
+            with open(json_file, "r", encoding="utf-8") as f:
+                data = json.load(f)
+            if isinstance(data, list):
+                records = data
+            elif isinstance(data, dict):
+                records = [data]
+            else:
+                raise ValueError(f"JSON must be object or array: {json_file.name}")
+            with open(output_path, "a", encoding="utf-8") as f:
+                for record in records:
+                    f.write(json.dumps(record, ensure_ascii=False) + "\n")
+                    output_count += 1
+        with store.connect() as conn:
+            conn.execute(
+                "UPDATE data_convert_tasks SET status='completed', "
+                "input_count=%s, output_count=%s, update_time=NOW() WHERE id=%s",
+                (input_count, output_count, task_id),
+            )
+        # 自动导入数据集
+        content = output_path.read_text(encoding="utf-8")
+        size_bytes = len(content.encode("utf-8"))
+        dataset = store.create_dataset({
+            "name": task["name"],
+            "type": "train",
+            "storage_type": "local",
+            "source": "upload",
+            "task_id": task_id,
+            "size": f"{size_bytes} B",
+            "count": output_count,
+            "description": f"由数据类型转换任务 {task_id} 自动导入",
+            "created_by": task.get("created_by") or current_user.get("id"),
+        })
+        dataset_id = dataset["id"]
+        with store.connect() as conn:
+            store.add_dataset_file(conn, dataset_id, _safe_output_filename(task.get("output_filename")), content)
+        return ok({
+            "staged_files": staged,
+            "auto_converted": True,
+            "dataset_id": dataset_id,
+            "input_count": input_count,
+            "output_count": output_count,
+        })
+    except Exception as exc:
+        with store.connect() as conn:
+            conn.execute(
+                "UPDATE data_convert_tasks SET status='failed', error_message=%s, update_time=NOW() WHERE id=%s",
+                (str(exc)[:500], task_id),
+            )
+        return ok({"staged_files": staged, "auto_converted": False, "error": str(exc)[:500]})
+
+
+@router.post("/{task_id}/run")
+@op_log(module=OpModule.DATA_CONVERT, action=OpAction.CONVERT, target_type="convert_task", target_name_param="task_id")
+def run_convert(
+    task_id: str,
+    current_user: dict = Depends(get_current_user),
+) -> dict[str, Any]:
+    task = _get_task(task_id)
+    if not task:
+        raise fail(404, "task not found")
+    if task["status"] not in ("uploaded", "completed", "failed"):
+        raise fail(400, "please upload source files first")
+    # 标记运行中
+    store = get_platform_store()
+    with store.connect() as conn:
+        conn.execute(
+            "UPDATE data_convert_tasks SET status='running', error_message='', update_time=NOW() WHERE id=%s",
+            (task_id,),
+        )
+    try:
+        input_dir = _input_dir(task_id)
+        output_dir = _output_dir(task_id)
+        output_dir.mkdir(parents=True, exist_ok=True)
+        output_path = _task_output_path(task)
+        input_count = 0
+        output_count = 0
+        for json_file in sorted(input_dir.iterdir()):
+            if not json_file.is_file() or not json_file.name.lower().endswith(".json"):
+                continue
+            input_count += 1
+            with open(json_file, "r", encoding="utf-8") as f:
+                data = json.load(f)
+            if isinstance(data, list):
+                records = data
+            elif isinstance(data, dict):
+                records = [data]
+            else:
+                raise ValueError(f"JSON must be object or array: {json_file.name}")
+            with open(output_path, "a", encoding="utf-8") as f:
+                for record in records:
+                    f.write(json.dumps(record, ensure_ascii=False) + "\n")
+                    output_count += 1
+        # 更新任务状态
+        with store.connect() as conn:
+            conn.execute(
+                "UPDATE data_convert_tasks SET status='completed', "
+                "input_count=%s, output_count=%s, update_time=NOW() WHERE id=%s",
+                (input_count, output_count, task_id),
+            )
+    except Exception as exc:
+        with store.connect() as conn:
+            conn.execute(
+                "UPDATE data_convert_tasks SET status='failed', error_message=%s, update_time=NOW() WHERE id=%s",
+                (str(exc)[:500], task_id),
+            )
+        raise fail(500, f"convert failed: {exc}")
+    return ok(_get_task(task_id))
+
+
+@router.get("/{task_id}/download")
+def download_result(
+    task_id: str,
+    current_user: dict = Depends(get_current_user),
+):
+    task = _get_task(task_id)
+    if not task:
+        raise fail(404, "task not found")
+    if task["status"] != "completed":
+        raise fail(400, "task is not completed")
+    output_path = _task_output_path(task)
+    if not output_path.exists():
+        raise fail(404, "output file not found")
+    return FileResponse(
+        str(output_path),
+        media_type="application/octet-stream",
+        filename=_safe_output_filename(task.get("output_filename")),
+    )
+
+
+@router.post("/{task_id}/import-as-dataset")
+def import_as_dataset(
+    task_id: str,
+    payload: dict[str, Any] = Body(default={}),
+    current_user: dict = Depends(get_current_user),
+) -> dict[str, Any]:
+    """把已转换的 JSONL 文件导入为数据集管理中的上传任务记录（source='task'）。"""
+    task = _get_task(task_id)
+    if not task:
+        raise fail(404, "task not found")
+    if task["status"] != "completed":
+        raise fail(400, "task is not completed")
+    output_path = _task_output_path(task)
+    if not output_path.exists():
+        raise fail(404, "output file not found")
+    content = output_path.read_text(encoding="utf-8")
+    dataset_name = str(payload.get("name") or task["name"]).strip()
+    description = str(payload.get("description") or f"由数据类型转换任务 {task_id} 导入").strip()
+    size_bytes = len(content.encode("utf-8"))
+    store = get_platform_store()
+    # 用 store 提供的接口创建数据集与文件
+    dataset = store.create_dataset({
+        "name": dataset_name,
+        "type": "train",
+        "storage_type": "local",
+        "source": "upload",
+        "task_id": task_id,
+        "size": f"{size_bytes} B",
+        "count": task["output_count"],
+        "description": description,
+        "created_by": task.get("created_by") or (current_user.get("id") if current_user else None),
+    })
+    dataset_id = dataset["id"]
+    with store.connect() as conn:
+        store.add_dataset_file(conn, dataset_id, _safe_output_filename(task.get("output_filename")), content)
+    return ok({"dataset_id": dataset_id, "name": dataset_name})
+
+
+@router.delete("/{task_id}")
+@op_log(module=OpModule.DATA_CONVERT, action=OpAction.DELETE, target_type="convert_task", target_name_param="task_id")
+def delete_task(
+    task_id: str,
+    current_user: dict = Depends(get_current_user),
+) -> dict[str, Any]:
+    task = _get_task(task_id)
+    if not task:
+        raise fail(404, "task not found")
+    store = get_platform_store()
+    with store.connect() as conn:
+        conn.execute(
+            "UPDATE data_convert_tasks SET deleted_at=NOW() WHERE id=%s",
+            (task_id,),
+        )
+    # 清理文件
+    import shutil
+    task_dir = _task_dir(task_id)
+    if task_dir.exists():
+        shutil.rmtree(task_dir, ignore_errors=True)
+    return ok({"deleted": task_id})
+
+
+def _get_task(task_id: str) -> dict[str, Any] | None:
+    store = get_platform_store()
+    with store.connect() as conn:
+        row = conn.execute(
+            "SELECT * FROM data_convert_tasks WHERE id=%s AND deleted_at IS NULL",
+            (task_id,),
+        ).fetchone()
+    return dict(row) if row else None

backend/app/modules/data_process/__init__.py

@@ -1 +1,46 @@
-"""Data processing module."""
+"""数据处理模块：从原始文件接入到生成标准训练数据的全流程。
+
+整体分层
+--------
+- ``algorithms/``  纯算法层（无副作用：不访问 DB / 文件系统 / 网络）。
+                  解析、格式检测、质量评分、去重、数据集切分、结构化预处理。
+                  API、后台任务与测试共同复用。
+- ``store/``       持久层（PostgreSQL）。按 Mixin 拆分：
+                  tasks / source_files / preview / generation / results / datasets。
+                  构造时不连库、不迁移；部署方须显式执行 002 迁移（见 schema_cli）。
+- 其余顶层文件    围绕上述两层的“服务 / 适配器”模块，由 endpoints 编排。
+
+顶层模块速查
+------------
+constants.py          共享常量（MAX_QA_PAIRS_PER_ITEM、MODEL_GENERATION_BATCH_SIZE）。
+storage.py            原始源文件的本地对象存储（受控暂存于 storage/data-process）。
+office_preview.py     Word/Excel 原文件的安全受限预览（仅返回绘制所需的结构化数据）。
+document_chunking.py  基于 Docling / LlamaIndex 的文档切分。
+dataset_format.py     Alpaca/ShareGPT/DPO/CPT 数据集格式校验（训练提交前预检）。
+generation.py         大模型生成适配器，把预览内容转为标准 instruction/output 记录。
+schema_cli.py         data_process 运行表的显式检查 / 安装命令（运维工具，应用启动不自动调用）。
+
+调用关系
+--------
+``app/api/v1/endpoints/data_process.py`` 是编排入口，组合调用以上模块与两个子包；
+``dataset_format.py`` 另被 ``db/platform_store.py`` 用于训练预检。
+
+典型链路
+--------
+上传源文件 → storage 暂存 → algorithms.parse_text_content 解析
+        → office_preview / document_chunking 处理 → store 落库预览
+        → generation 调模型生成 → store 写 results → dataset_format 校验后发布。
+
+导入约定
+--------
+为避免循环导入，本 ``__init__`` 不统一再导出；请按需从子模块直接导入：
+
+    from app.modules.data_process.store import DataProcessStore
+    from app.modules.data_process.algorithms import estimate_token_count
+    from app.modules.data_process.generation import generate_model_records
+"""
+
+# 注意：为了避免循环导入，不在此处导入所有内容
+# 请直接从子模块导入所需功能
+
+__all__ = ["store", "algorithms"]

backend/app/modules/data_process/algorithms/__init__.py

@@ -0,0 +1,145 @@
+"""数据处理算法模块。
+
+重构为多个按职责拆分的子模块：
+
+- types.py: 类型定义、常量与 dataclass
+- text_utils.py: 文本解码、归一化与格式检测
+- format_detection.py: 文档结构检测
+- parsers/: PDF / Office / JSON / CSV 解析器
+- quality.py: 质量评分与去重
+- transforms.py: 数据集分割
+- structured_processing.py: 结构化数据预处理
+
+使用方式：
+    from app.modules.data_process.algorithms import estimate_token_count
+
+子模块之间存在导入分层，顶层按依赖顺序导入以避免循环导入。
+"""
+
+from __future__ import annotations
+
+# Layer 0: 类型与常量（叶子，不依赖内部模块）
+from .types import (
+    DatasetSplit,
+    DocumentHeading,
+    DocumentNoiseSpan,
+    DocumentStructure,
+    MAX_QA_PAIRS_PER_ITEM,
+    ParsedText,
+    PdfPageText,
+    ProcessedStructuredRecord,
+    QualityScore,
+    SUPPORTED_TEXT_FORMATS,
+    StructuredPreprocessOption,
+    TextFormat,
+    _MAX_WORKBOOK_COLUMNS,
+    _MAX_WORKBOOK_HEADER_SCAN_ROWS,
+)
+
+# Layer 2: 文本工具（仅依赖 types；对 parsers 的依赖在函数体内延迟导入）
+from .text_utils import (
+    _normalize_spreadsheet_value,
+    decode_utf8,
+    detect_text_format,
+    normalize_text,
+    parse_text_content,
+    parse_utf8_text,
+    structured_json_dumps,
+)
+
+# Layer 1: 解析器（依赖 types 与 text_utils）
+from .parsers import (
+    _infer_xlsx_header_region,
+    _rewrite_xlsx_workbook_relationships,
+    _validate_office_archive,
+    _xlsx_sheet_merge_ranges,
+    detect_pdf_document_noise,
+    extract_pdf_page_texts,
+    remove_document_noise,
+)
+
+# Layer 3: 数据转换与质量评分
+from .transforms import stable_split, stable_split_assignments
+from .quality import (
+    content_quality_flags,
+    deduplicate_structured_records,
+    estimate_token_count,
+    fingerprints_are_near_duplicate,
+    is_low_quality_content,
+    is_near_duplicate,
+    near_duplicate_fingerprint,
+    record_fingerprint,
+    score_quality,
+)
+
+# Layer 4: 结构化数据处理（依赖 text_utils / quality / transforms / parsers）
+from .structured_processing import (
+    canonical_record_json,
+    desensitize_pii,
+    desensitize_structured_record,
+    expand_to_context_boundaries,
+    extract_structured_records,
+    filter_anomalous_structured_records,
+    flatten_structured_record,
+    generate_standard_records,
+    merge_short_blocks,
+    normalize_structured_record,
+    preprocess_structured_records,
+    preprocess_structured_records_with_lineage,
+    protected_context_ranges,
+)
+
+# Layer 5: 文档结构检测（依赖 structured_processing）
+from .format_detection import detect_document_structure
+
+__all__ = [
+    "DatasetSplit",
+    "DocumentHeading",
+    "DocumentNoiseSpan",
+    "DocumentStructure",
+    "MAX_QA_PAIRS_PER_ITEM",
+    "ParsedText",
+    "PdfPageText",
+    "ProcessedStructuredRecord",
+    "QualityScore",
+    "SUPPORTED_TEXT_FORMATS",
+    "StructuredPreprocessOption",
+    "TextFormat",
+    "_MAX_WORKBOOK_COLUMNS",
+    "_MAX_WORKBOOK_HEADER_SCAN_ROWS",
+    "_normalize_spreadsheet_value",
+    "canonical_record_json",
+    "content_quality_flags",
+    "decode_utf8",
+    "deduplicate_structured_records",
+    "desensitize_pii",
+    "desensitize_structured_record",
+    "detect_document_structure",
+    "detect_pdf_document_noise",
+    "detect_text_format",
+    "estimate_token_count",
+    "expand_to_context_boundaries",
+    "extract_pdf_page_texts",
+    "extract_structured_records",
+    "filter_anomalous_structured_records",
+    "fingerprints_are_near_duplicate",
+    "flatten_structured_record",
+    "generate_standard_records",
+    "is_low_quality_content",
+    "is_near_duplicate",
+    "merge_short_blocks",
+    "near_duplicate_fingerprint",
+    "normalize_structured_record",
+    "normalize_text",
+    "parse_text_content",
+    "parse_utf8_text",
+    "preprocess_structured_records",
+    "preprocess_structured_records_with_lineage",
+    "protected_context_ranges",
+    "record_fingerprint",
+    "remove_document_noise",
+    "score_quality",
+    "stable_split",
+    "stable_split_assignments",
+    "structured_json_dumps",
+]

backend/app/modules/data_process/algorithms/format_detection.py

@@ -0,0 +1,119 @@
+"""数据处理算法 - 格式检测。"""
+
+from __future__ import annotations
+
+import re
+
+from .structured_processing import protected_context_ranges
+from .text_utils import normalize_text
+from .types import DocumentHeading, DocumentStructure
+
+
+def detect_document_structure(text: str) -> DocumentStructure:
+    """识别 Markdown、中文章节和数字编号标题及不可拆分块。"""
+
+    normalized = normalize_text(text)
+    if not normalized:
+        return DocumentStructure(
+            line_count=0,
+            paragraph_count=0,
+            headings=(),
+            code_block_count=0,
+            table_block_count=0,
+            list_block_count=0,
+        )
+
+    code_ranges = protected_context_ranges(
+        normalized,
+        preserve_code_blocks=True,
+        preserve_tables=False,
+        preserve_lists=False,
+    )
+    table_candidates = protected_context_ranges(
+        normalized,
+        preserve_code_blocks=False,
+        preserve_tables=True,
+        preserve_lists=False,
+    )
+    list_candidates = protected_context_ranges(
+        normalized,
+        preserve_code_blocks=False,
+        preserve_tables=False,
+        preserve_lists=True,
+    )
+    table_ranges = tuple(
+        item
+        for item in table_candidates
+        if not any(
+            item[0] < code_end and item[1] > code_start
+            for code_start, code_end in code_ranges
+        )
+    )
+    list_ranges = tuple(
+        item
+        for item in list_candidates
+        if not any(
+            item[0] < code_end and item[1] > code_start
+            for code_start, code_end in code_ranges
+        )
+    )
+    markdown_heading = re.compile(r"^\s*(?P<marks>#{1,6})\s+(?P<title>.+?)\s*#*\s*$")
+    chinese_heading = re.compile(
+        r"^\s*(?P<title>第[一二三四五六七八九十百千万0-9]+[章节篇部分].*)$"
+    )
+    numbered_heading = re.compile(
+        r"^\s*(?P<number>\d+(?:\.\d+)*)(?:[、.]|\s+)\s*(?P<title>\S.*)$"
+    )
+
+    headings: list[DocumentHeading] = []
+    cursor = 0
+    for line_number, raw_line in enumerate(normalized.splitlines(keepends=True), start=1):
+        line = raw_line.rstrip("\n")
+        line_end = cursor + len(line)
+        if not any(range_start <= cursor < range_end for range_start, range_end in code_ranges):
+            match = markdown_heading.match(line)
+            if match:
+                headings.append(
+                    DocumentHeading(
+                        level=len(match.group("marks")),
+                        title=normalize_text(match.group("title")),
+                        line_number=line_number,
+                        start=cursor,
+                        end=line_end,
+                    )
+                )
+            else:
+                match = chinese_heading.match(line)
+                if match:
+                    headings.append(
+                        DocumentHeading(
+                            level=1,
+                            title=normalize_text(match.group("title")),
+                            line_number=line_number,
+                            start=cursor,
+                            end=line_end,
+                        )
+                    )
+                else:
+                    match = numbered_heading.match(line)
+                    if match:
+                        headings.append(
+                            DocumentHeading(
+                                level=min(6, match.group("number").count(".") + 1),
+                                title=normalize_text(match.group("title")),
+                                line_number=line_number,
+                                start=cursor,
+                                end=line_end,
+                            )
+                        )
+        cursor += len(raw_line)
+
+    paragraphs = [part for part in re.split(r"\n\s*\n", normalized) if part.strip()]
+    return DocumentStructure(
+        line_count=len(normalized.splitlines()),
+        paragraph_count=len(paragraphs),
+        headings=tuple(headings),
+        code_block_count=len(code_ranges),
+        table_block_count=len(table_ranges),
+        list_block_count=len(list_ranges),
+    )

backend/app/modules/data_process/algorithms/parsers/__init__.py

@@ -0,0 +1,19 @@
+"""文档解析器模块。"""
+
+from .pdf import extract_pdf_page_texts, detect_pdf_document_noise, remove_document_noise
+from .office import (
+    _validate_office_archive,
+    _rewrite_xlsx_workbook_relationships,
+    _xlsx_sheet_merge_ranges,
+    _infer_xlsx_header_region,
+)
+
+__all__ = [
+    'extract_pdf_page_texts',
+    'detect_pdf_document_noise',
+    'remove_document_noise',
+    '_validate_office_archive',
+    '_rewrite_xlsx_workbook_relationships',
+    '_xlsx_sheet_merge_ranges',
+    '_infer_xlsx_header_region',
+]

backend/app/modules/data_process/algorithms/parsers/csv_parser.py

@@ -0,0 +1,89 @@
+"""数据处理算法 - CSV 解析。"""
+
+from __future__ import annotations
+
+import csv
+import io
+import re
+from typing import Any
+
+from ..text_utils import normalize_text
+
+
+def _source_line_offsets(
+    text: str,
+    start_line: int,
+    end_line: int,
+) -> tuple[int, int]:
+    """把 1-based 物理行范围转换为左闭右开的字符范围。"""
+
+    line_starts = [0]
+    line_starts.extend(match.end() for match in re.finditer("\n", text))
+    if start_line < 1 or end_line < start_line or end_line > len(line_starts):
+        raise ValueError("source line range is outside normalized text")
+    source_start = line_starts[start_line - 1]
+    source_end = (
+        line_starts[end_line] - 1
+        if end_line < len(line_starts)
+        else len(text)
+    )
+    return source_start, source_end
+
+
+def _extract_csv_records_with_locators(
+    text: str,
+) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
+    """从 CSV 文本中提取记录及其行级定位信息。"""
+
+    normalized_text = normalize_text(text)
+    if not normalized_text:
+        return [], []
+
+    try:
+        dialect = csv.Sniffer().sniff(normalized_text[:8192], delimiters=",\t;")
+    except csv.Error:
+        dialect = csv.excel
+    reader = csv.DictReader(io.StringIO(normalized_text), dialect=dialect)
+    if not reader.fieldnames:
+        raise ValueError("CSV header is required")
+    headers = [normalize_text(header or "") for header in reader.fieldnames]
+    if any(not header for header in headers):
+        raise ValueError("CSV header cannot be empty")
+    if len(set(headers)) != len(headers):
+        raise ValueError("CSV headers must be unique")
+    reader.fieldnames = headers
+
+    records: list[dict[str, Any]] = []
+    locators: list[dict[str, Any]] = []
+    source_lines = normalized_text.splitlines()
+    previous_end_line = reader.line_num
+    for row in reader:
+        end_line = reader.line_num
+        start_line = previous_end_line + 1
+        previous_end_line = end_line
+        while start_line < end_line and not source_lines[start_line - 1].strip():
+            start_line += 1
+        if None in row:
+            raise ValueError("CSV row has more fields than the header")
+        normalized_row = {
+            key: normalize_text(value or "")
+            for key, value in row.items()
+        }
+        if any(value for value in normalized_row.values()):
+            records.append(normalized_row)
+            source_start, source_end = _source_line_offsets(
+                normalized_text,
+                start_line,
+                end_line,
+            )
+            locators.append(
+                {
+                    "kind": "csv",
+                    "record_index": len(records),
+                    "start_line": start_line,
+                    "end_line": end_line,
+                    "source_start": source_start,
+                    "source_end": source_end,
+                }
+            )
+    return records, locators

backend/app/modules/data_process/algorithms/parsers/json_parser.py

@@ -0,0 +1,395 @@
+"""数据处理算法 - JSON/JSONL 解析。"""
+
+from __future__ import annotations
+
+import json
+import re
+from collections.abc import Mapping, Sequence
+from decimal import Decimal
+from typing import Any
+
+from ..text_utils import _normalize_format, _normalize_value, normalize_text
+from ..types import (
+    _DuplicateJsonKeyError,
+    _JSON_ENVELOPE_KEYS,
+    _JSON_RECORD_ARRAY_KEYS,
+    _JSON_RESPONSE_METADATA_KEYS,
+    _JSON_WRAPPER_METADATA_KEYS,
+    _MAX_JSON_DEPTH,
+)
+
+
+def _record_from_value(value: Any, *, normalize: bool = True) -> dict[str, Any]:
+    if isinstance(value, Mapping):
+        return dict(_normalize_value(value)) if normalize else dict(value)
+    return {"value": _normalize_value(value) if normalize else value}
+
+def _json_pointer_segment(value: Any) -> str:
+    return str(value).replace("~", "~0").replace("/", "~1")
+
+def _reject_duplicate_json_keys(pairs: list[tuple[str, Any]]) -> dict[str, Any]:
+    result: dict[str, Any] = {}
+    for key, value in pairs:
+        if key in result:
+            raise _DuplicateJsonKeyError(f"duplicate JSON object key: {key!r}")
+        result[key] = value
+    return result
+
+def _reject_json_constant(value: str) -> Any:
+    raise ValueError(f"non-finite JSON number is not allowed: {value}")
+
+def _skip_json_whitespace(text: str, offset: int) -> int:
+    while offset < len(text) and text[offset] in " \t\r\n":
+        offset += 1
+    return offset
+
+def _validate_json_nesting(text: str) -> None:
+    """在构造 Python 对象前限制容器深度，避免依赖解释器递归阈值。"""
+
+    depth = 0
+    in_string = False
+    escaped = False
+    for char in text:
+        if in_string:
+            if escaped:
+                escaped = False
+            elif char == "\\":
+                escaped = True
+            elif char == '"':
+                in_string = False
+            continue
+        if char == '"':
+            in_string = True
+        elif char in "[{":
+            depth += 1
+            if depth > _MAX_JSON_DEPTH:
+                raise ValueError(
+                    f"JSON nesting exceeds the supported depth of {_MAX_JSON_DEPTH}"
+                )
+        elif char in "]}":
+            depth = max(0, depth - 1)
+
+def _strict_json_loads(text: str) -> tuple[Any, int, int]:
+    """严格解析单个 JSON 值并返回其左闭右开源码区间。"""
+
+    start = _skip_json_whitespace(text, 0)
+    if start >= len(text):
+        raise ValueError("JSON content is empty")
+    _validate_json_nesting(text)
+    decoder = json.JSONDecoder(
+        object_pairs_hook=_reject_duplicate_json_keys,
+        parse_float=Decimal,
+        parse_int=int,
+        parse_constant=_reject_json_constant,
+        strict=True,
+    )
+    try:
+        payload, end = decoder.raw_decode(text, start)
+    except json.JSONDecodeError as exc:
+        raise ValueError(
+            f"invalid JSON at line {exc.lineno}, column {exc.colno}: {exc.msg}"
+        ) from exc
+    except RecursionError as exc:
+        raise ValueError("JSON nesting exceeds the supported depth") from exc
+    except _DuplicateJsonKeyError as exc:
+        raise ValueError(str(exc)) from exc
+    except ValueError as exc:
+        # parse_int/parse_float/parse_constant 的异常也必须稳定映射为客户端错误。
+        raise ValueError(f"invalid JSON number: {exc}") from exc
+    trailing = _skip_json_whitespace(text, end)
+    if trailing != len(text):
+        line = text.count("\n", 0, trailing) + 1
+        line_start = text.rfind("\n", 0, trailing) + 1
+        column = trailing - line_start + 1
+        raise ValueError(
+            f"invalid JSON at line {line}, column {column}: extra data"
+        )
+    return payload, start, end
+
+def _json_value_end(text: str, start: int) -> int:
+    """在已验证 JSON 中定位一个值的结束偏移，不对数值做二次解析。"""
+
+    if start >= len(text):
+        raise ValueError("invalid JSON source span")
+    first = text[start]
+    if first == '"':
+        escaped = False
+        for offset in range(start + 1, len(text)):
+            char = text[offset]
+            if escaped:
+                escaped = False
+            elif char == "\\":
+                escaped = True
+            elif char == '"':
+                return offset + 1
+        raise ValueError("invalid JSON source span")
+    if first in "[{":
+        stack = [first]
+        in_string = False
+        escaped = False
+        for offset in range(start + 1, len(text)):
+            char = text[offset]
+            if in_string:
+                if escaped:
+                    escaped = False
+                elif char == "\\":
+                    escaped = True
+                elif char == '"':
+                    in_string = False
+                continue
+            if char == '"':
+                in_string = True
+            elif char in "[{":
+                stack.append(char)
+            elif char in "]}":
+                expected = "[" if char == "]" else "{"
+                if not stack or stack[-1] != expected:
+                    raise ValueError("invalid JSON source span")
+                stack.pop()
+                if not stack:
+                    return offset + 1
+        raise ValueError("invalid JSON source span")
+    end = start
+    while end < len(text) and text[end] not in " \t\r\n,]}":
+        end += 1
+    if end == start:
+        raise ValueError("invalid JSON source span")
+    return end
+
+def _json_object_value_spans(
+    text: str,
+    start: int,
+    end: int,
+) -> dict[str, tuple[int, int]]:
+    """返回已验证 JSON 对象直接子字段的值区间。"""
+
+    if text[start] != "{" or text[end - 1] != "}":
+        raise ValueError("JSON source value is not an object")
+    result: dict[str, tuple[int, int]] = {}
+    offset = _skip_json_whitespace(text, start + 1)
+    key_decoder = json.JSONDecoder()
+    while offset < end - 1:
+        key, key_end = key_decoder.raw_decode(text, offset)
+        if not isinstance(key, str):
+            raise ValueError("invalid JSON object key")
+        offset = _skip_json_whitespace(text, key_end)
+        if offset >= end or text[offset] != ":":
+            raise ValueError("invalid JSON object member")
+        value_start = _skip_json_whitespace(text, offset + 1)
+        value_end = _json_value_end(text, value_start)
+        result[key] = (value_start, value_end)
+        offset = _skip_json_whitespace(text, value_end)
+        if offset >= end - 1:
+            break
+        if text[offset] != ",":
+            raise ValueError("invalid JSON object member")
+        offset = _skip_json_whitespace(text, offset + 1)
+    return result
+
+def _json_array_item_spans(
+    text: str,
+    start: int,
+    end: int,
+) -> list[tuple[int, int]]:
+    """返回已验证 JSON 数组中每个直接元素的源码区间。"""
+
+    if text[start] != "[" or text[end - 1] != "]":
+        raise ValueError("JSON source value is not an array")
+    result: list[tuple[int, int]] = []
+    offset = _skip_json_whitespace(text, start + 1)
+    while offset < end - 1:
+        item_end = _json_value_end(text, offset)
+        result.append((offset, item_end))
+        offset = _skip_json_whitespace(text, item_end)
+        if offset >= end - 1:
+            break
+        if text[offset] != ",":
+            raise ValueError("invalid JSON array item")
+        offset = _skip_json_whitespace(text, offset + 1)
+    return result
+
+def _json_span_at_path(
+    text: str,
+    root_span: tuple[int, int],
+    path: Sequence[str],
+) -> tuple[int, int]:
+    span = root_span
+    for key in path:
+        try:
+            span = _json_object_value_spans(text, *span)[key]
+        except KeyError as exc:
+            raise ValueError(f"JSON source path cannot be located: {key}") from exc
+    return span
+
+
+def _pure_json_record_wrapper(
+    payload: Any,
+) -> tuple[list[Any], tuple[str, ...]] | None:
+    """识别不会与业务字段冲突的纯记录包装对象。"""
+
+    if not isinstance(payload, Mapping):
+        return None
+
+    def direct_wrapper(
+        value: Mapping[str, Any],
+        metadata_keys: frozenset[str] | set[str] = _JSON_WRAPPER_METADATA_KEYS,
+    ) -> tuple[list[Any], tuple[str, ...]] | None:
+        candidates = [
+            key
+            for key in _JSON_RECORD_ARRAY_KEYS
+            if isinstance(value.get(key), list)
+        ]
+        if len(candidates) != 1:
+            return None
+        record_key = candidates[0]
+        records = value[record_key]
+        if any(not isinstance(record, Mapping) for record in records):
+            return None
+        if any(
+            key != record_key and key not in metadata_keys
+            for key in value
+        ):
+            return None
+        return records, (record_key,)
+
+    direct = direct_wrapper(payload)
+    if direct is not None:
+        return direct
+
+    envelope_keys = [
+        key
+        for key in _JSON_ENVELOPE_KEYS
+        if isinstance(payload.get(key), Mapping)
+    ]
+    if len(envelope_keys) != 1:
+        return None
+    envelope_key = envelope_keys[0]
+    if any(
+        key != envelope_key and key not in _JSON_RESPONSE_METADATA_KEYS
+        for key in payload
+    ):
+        return None
+    nested = direct_wrapper(
+        payload[envelope_key],
+        _JSON_RESPONSE_METADATA_KEYS,
+    )
+    if nested is None:
+        return None
+    records, nested_path = nested
+    return records, (envelope_key, *nested_path)
+
+
+def _json_record_locator(
+    text: str,
+    *,
+    record_index: int,
+    json_pointer: str,
+    span: tuple[int, int],
+) -> dict[str, Any]:
+    source_start, source_end = span
+    start_line = text.count("\n", 0, source_start) + 1
+    last_character = max(source_start, source_end - 1)
+    end_line = text.count("\n", 0, last_character) + 1
+    return {
+        "kind": "json",
+        "record_index": record_index,
+        "json_pointer": json_pointer,
+        "source_start": source_start,
+        "source_end": source_end,
+        "start_line": start_line,
+        "end_line": end_line,
+    }
+
+
+def _extract_structured_records_with_locators(
+    text: str,
+    file_format: str,
+) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
+    """从 JSON、JSONL 或 CSV 中提取记录。
+
+    JSON 根数组始终表示多条记录；对象仅在满足纯包装契约时展开，其他
+    对象均视为一条业务记录。JSON 字段和值在解析阶段保持原样，只有用户
+    明确选择 ``normalize_format`` 后才会规范化。
+    """
+
+    normalized_format = _normalize_format(file_format)
+    if normalized_format not in {"json", "jsonl", "csv"}:
+        raise ValueError("structured record extraction only supports JSON, JSONL and CSV")
+
+    if normalized_format == "json":
+        if _skip_json_whitespace(text, 0) == len(text):
+            return [], []
+        payload, root_start, root_end = _strict_json_loads(text)
+        values: Sequence[Any]
+        pointer_path: tuple[str, ...] = ()
+        record_spans: list[tuple[int, int]]
+        if isinstance(payload, list):
+            values = payload
+            record_spans = _json_array_item_spans(text, root_start, root_end)
+        else:
+            wrapper = _pure_json_record_wrapper(payload)
+            if wrapper is None:
+                values = [payload]
+                record_spans = [(root_start, root_end)]
+            else:
+                values, pointer_path = wrapper
+                array_span = _json_span_at_path(
+                    text,
+                    (root_start, root_end),
+                    pointer_path,
+                )
+                record_spans = _json_array_item_spans(text, *array_span)
+        if len(record_spans) != len(values):
+            raise ValueError("JSON record source spans do not match parsed records")
+        records = [_record_from_value(value, normalize=False) for value in values]
+        pointer_prefix = "".join(
+            f"/{_json_pointer_segment(segment)}" for segment in pointer_path
+        )
+        locators = [
+            _json_record_locator(
+                text,
+                record_index=index + 1,
+                json_pointer=(
+                    f"{pointer_prefix}/{index}"
+                    if pointer_path or isinstance(payload, list)
+                    else ""
+                ),
+                span=record_spans[index],
+            )
+            for index in range(len(records))
+        ]
+        return records, locators
+
+    if normalized_format == "jsonl":
+        records: list[dict[str, Any]] = []
+        locators: list[dict[str, Any]] = []
+        source_offset = 0
+        for line_number, line in enumerate(text.split("\n"), start=1):
+            line_content_end = len(line)
+            if _skip_json_whitespace(line, 0) == line_content_end:
+                source_offset += len(line) + 1
+                continue
+            try:
+                value, value_start, value_end = _strict_json_loads(line)
+            except ValueError as exc:
+                raise ValueError(
+                    f"invalid JSONL at line {line_number}: {exc}"
+                ) from exc
+            records.append(_record_from_value(value, normalize=False))
+            locators.append(
+                {
+                    "kind": "jsonl",
+                    "record_index": len(records),
+                    "start_line": line_number,
+                    "end_line": line_number,
+                    "source_start": source_offset + value_start,
+                    "source_end": source_offset + value_end,
+                }
+            )
+            source_offset += len(line) + 1
+        return records, locators
+
+    # CSV 分支独立放在 csv_parser 中，避免与 JSON 机制耦合。
+    from .csv_parser import _extract_csv_records_with_locators
+
+    return _extract_csv_records_with_locators(text)

backend/app/modules/data_process/algorithms/parsers/office.py

@@ -0,0 +1,700 @@
+"""数据处理算法 - Office 文档解析。"""
+
+from __future__ import annotations
+
+import io
+import re
+import unicodedata
+import zipfile
+import xml.etree.ElementTree as ET
+from collections.abc import Mapping, Sequence
+from pathlib import PurePosixPath
+from typing import Any
+from urllib.parse import unquote, urlsplit
+
+from docx import Document
+from docx.oxml.table import CT_Tbl
+from docx.oxml.text.paragraph import CT_P
+from docx.table import Table
+from docx.text.paragraph import Paragraph
+from openpyxl import load_workbook
+from openpyxl.utils.cell import range_boundaries
+from pptx import Presentation
+
+from ..text_utils import _append_bounded_text, _normalize_spreadsheet_value, normalize_text
+from ..types import (
+    _MAX_ARCHIVE_COMPRESSION_RATIO,
+    _MAX_ARCHIVE_ENTRIES,
+    _MAX_ARCHIVE_ENTRY_BYTES,
+    _MAX_ARCHIVE_UNCOMPRESSED_BYTES,
+    _MAX_PRESENTATION_SLIDES,
+    _MAX_WORKBOOK_CELLS,
+    _MAX_WORKBOOK_COLUMNS,
+    _MAX_WORKBOOK_HEADER_ROWS,
+    _MAX_WORKBOOK_HEADER_SCAN_ROWS,
+    _MAX_WORKBOOK_MERGED_RANGES,
+    _MAX_WORKBOOK_ROWS,
+    _MAX_WORKBOOK_SCANNED_ROWS,
+    _MAX_WORKBOOK_SHEETS,
+    TextFormat,
+)
+
+_XLSX_REPORT_METADATA_PATTERN = re.compile(
+    r"^(?:报表|报告|标题|说明|备注|制表|统计|日期|时间|期间|"
+    r"report|title|note|remark|date|time|period)\b",
+    flags=re.IGNORECASE,
+)
+
+
+def _validate_office_archive(raw: bytes, file_format: TextFormat) -> None:
+    """在交给 Office 解析库前限制 ZIP 包规模并拒绝活动 XML。"""
+
+    required_members = {
+        "docx": {"[Content_Types].xml", "word/document.xml"},
+        "xlsx": {"[Content_Types].xml", "xl/workbook.xml"},
+        "pptx": {"[Content_Types].xml", "ppt/presentation.xml"},
+    }
+    try:
+        with zipfile.ZipFile(io.BytesIO(raw)) as archive:
+            members = archive.infolist()
+            if len(members) > _MAX_ARCHIVE_ENTRIES:
+                raise ValueError(
+                    f"{file_format.upper()} archive contains too many entries "
+                    f"(limit {_MAX_ARCHIVE_ENTRIES})"
+                )
+
+            names: set[str] = set()
+            total_size = 0
+            for member in members:
+                path = PurePosixPath(member.filename.replace("\\", "/"))
+                if path.is_absolute() or ".." in path.parts:
+                    raise ValueError(
+                        f"{file_format.upper()} archive contains an unsafe member path"
+                    )
+                if member.filename in names:
+                    raise ValueError(
+                        f"{file_format.upper()} archive contains duplicate member names"
+                    )
+                names.add(member.filename)
+                if member.flag_bits & 0x1:
+                    raise ValueError(f"encrypted {file_format.upper()} files are not supported")
+                if member.is_dir():
+                    continue
+                if member.file_size > _MAX_ARCHIVE_ENTRY_BYTES:
+                    raise ValueError(
+                        f"{file_format.upper()} archive entry exceeds "
+                        f"{_MAX_ARCHIVE_ENTRY_BYTES} bytes"
+                    )
+                total_size += member.file_size
+                if total_size > _MAX_ARCHIVE_UNCOMPRESSED_BYTES:
+                    raise ValueError(
+                        f"{file_format.upper()} archive expands beyond "
+                        f"{_MAX_ARCHIVE_UNCOMPRESSED_BYTES} bytes"
+                    )
+                if member.file_size >= 1024 * 1024:
+                    if member.compress_size <= 0:
+                        raise ValueError(
+                            f"{file_format.upper()} archive has an unsafe compression ratio"
+                        )
+                    ratio = member.file_size / member.compress_size
+                    if ratio > _MAX_ARCHIVE_COMPRESSION_RATIO:
+                        raise ValueError(
+                            f"{file_format.upper()} archive has an unsafe compression ratio"
+                        )
+
+            missing = required_members[file_format] - names
+            if missing:
+                raise ValueError(
+                    f"invalid {file_format.upper()} package: missing "
+                    f"{', '.join(sorted(missing))}"
+                )
+
+            for member in members:
+                if member.is_dir() or not member.filename.lower().endswith((".xml", ".rels")):
+                    continue
+                with archive.open(member) as stream:
+                    prefix = stream.read(min(member.file_size, 1024 * 1024)).upper()
+                if b"<!DOCTYPE" in prefix or b"<!ENTITY" in prefix:
+                    raise ValueError(
+                        f"{file_format.upper()} archive contains unsupported active XML"
+                    )
+    except zipfile.BadZipFile as exc:
+        raise ValueError(f"invalid {file_format.upper()} file: not an Office ZIP package") from exc
+
+def _extract_docx_text(raw: bytes) -> str:
+    _validate_office_archive(raw, "docx")
+    try:
+        document = Document(io.BytesIO(raw))
+    except Exception as exc:
+        raise ValueError(f"invalid DOCX file: {exc}") from exc
+
+    parts: list[str] = []
+    total = 0
+    for child in document.element.body.iterchildren():
+        if isinstance(child, CT_P):
+            total = _append_bounded_text(parts, Paragraph(child, document).text, total)
+            continue
+        if isinstance(child, CT_Tbl):
+            table = Table(child, document)
+            for row in table.rows:
+                cells = [normalize_text(cell.text) for cell in row.cells]
+                total = _append_bounded_text(parts, "\t".join(cells), total)
+    return normalize_text("\n\n".join(parts))
+
+def _presentation_shape_text(shape: Any) -> list[str]:
+    if getattr(shape, "has_table", False):
+        return [
+            "\t".join(normalize_text(cell.text) for cell in row.cells)
+            for row in shape.table.rows
+        ]
+    if getattr(shape, "has_text_frame", False):
+        return [shape.text]
+    child_shapes = getattr(shape, "shapes", None)
+    if child_shapes is not None:
+        values: list[str] = []
+        for child in child_shapes:
+            values.extend(_presentation_shape_text(child))
+        return values
+    return []
+
+def _extract_pptx_text(raw: bytes) -> str:
+    _validate_office_archive(raw, "pptx")
+    try:
+        presentation = Presentation(io.BytesIO(raw))
+    except Exception as exc:
+        raise ValueError(f"invalid PPTX file: {exc}") from exc
+    if len(presentation.slides) > _MAX_PRESENTATION_SLIDES:
+        raise ValueError(
+            f"PPTX contains too many slides (limit {_MAX_PRESENTATION_SLIDES})"
+        )
+
+    parts: list[str] = []
+    total = 0
+    for slide in presentation.slides:
+        slide_parts: list[str] = []
+        for shape in slide.shapes:
+            slide_parts.extend(_presentation_shape_text(shape))
+        total = _append_bounded_text(parts, "\n".join(slide_parts), total)
+    return normalize_text("\n\n".join(parts))
+
+def _xml_local_name(tag: str) -> str:
+    return tag.rsplit("}", 1)[-1]
+
+def _resolve_xlsx_relationship_target(
+    archive: zipfile.ZipFile,
+    target: str,
+    *,
+    source_part: str = "xl/workbook.xml",
+) -> str:
+    """按 OPC URI 规则解析内部关系目标，并保证结果仍位于 ZIP 根内。"""
+
+    raw_target = target.strip()
+    if (
+        raw_target != target
+        or not raw_target
+        or "\\" in raw_target
+        or any(unicodedata.category(char).startswith("C") for char in raw_target)
+        or re.search(r"%(?![0-9A-Fa-f]{2})", raw_target)
+    ):
+        raise ValueError("XLSX workbook contains an unsafe worksheet path")
+    try:
+        parsed = urlsplit(raw_target)
+    except ValueError as exc:
+        raise ValueError("XLSX workbook contains an unsafe worksheet path") from exc
+    if parsed.scheme or parsed.netloc or parsed.query or parsed.fragment:
+        raise ValueError("XLSX workbook contains an unsafe worksheet path")
+    if re.search(r"%(?:2[fF]|5[cC]|0{2})", parsed.path):
+        raise ValueError("XLSX workbook contains an unsafe worksheet path")
+    try:
+        decoded_path = unquote(parsed.path, encoding="utf-8", errors="strict")
+    except UnicodeDecodeError as exc:
+        raise ValueError("XLSX workbook contains an unsafe worksheet path") from exc
+    if not decoded_path or "\\" in decoded_path or "\x00" in decoded_path or "%" in decoded_path:
+        raise ValueError("XLSX workbook contains an unsafe worksheet path")
+
+    parts = [] if decoded_path.startswith("/") else list(PurePosixPath(source_part).parent.parts)
+    for part in decoded_path.lstrip("/").split("/"):
+        if part in {"", "."}:
+            continue
+        if part == "..":
+            if not parts:
+                raise ValueError("XLSX workbook contains an unsafe worksheet path")
+            parts.pop()
+            continue
+        if unicodedata.category(part[0]).startswith("C") or any(
+            unicodedata.category(char).startswith("C") for char in part
+        ):
+            raise ValueError("XLSX workbook contains an unsafe worksheet path")
+        parts.append(part)
+    if not parts:
+        raise ValueError("XLSX workbook contains an unsafe worksheet path")
+
+    member_name = "/".join(parts)
+    try:
+        member = archive.getinfo(member_name)
+    except KeyError as exc:
+        raise ValueError(
+            f"XLSX worksheet relationship target does not exist: {member_name}"
+        ) from exc
+    if member.is_dir():
+        raise ValueError("XLSX worksheet relationship target must be a file")
+    return member_name
+
+def _rewrite_xlsx_workbook_relationships(
+    raw: bytes,
+    replacements: Mapping[str, str],
+) -> bytes:
+    """把已验证的 worksheet Target 改为解析库稳定支持的包内绝对路径。"""
+
+    relationships_member = "xl/_rels/workbook.xml.rels"
+    output = io.BytesIO()
+    with zipfile.ZipFile(io.BytesIO(raw)) as source, zipfile.ZipFile(output, "w") as target:
+        target.comment = source.comment
+        for member in source.infolist():
+            if member.filename == relationships_member:
+                root = ET.fromstring(source.read(member))
+                pending = dict(replacements)
+                for element in root:
+                    if _xml_local_name(element.tag) != "Relationship":
+                        continue
+                    relationship_id = element.attrib.get("Id")
+                    if relationship_id in pending:
+                        element.set("Target", pending.pop(relationship_id))
+                if pending:
+                    raise ValueError(
+                        "XLSX workbook relationship changed during normalization"
+                    )
+                content = ET.tostring(root, encoding="utf-8", xml_declaration=True)
+                target.writestr(member, content)
+                continue
+            if member.is_dir():
+                target.writestr(member, b"")
+                continue
+            with source.open(member) as source_stream, target.open(
+                member,
+                "w",
+                force_zip64=True,
+            ) as target_stream:
+                while chunk := source_stream.read(1024 * 1024):
+                    target_stream.write(chunk)
+    return output.getvalue()
+
+def _xlsx_sheet_merge_ranges(
+    raw: bytes,
+) -> tuple[
+    dict[str, tuple[tuple[int, int, int, int], ...]],
+    dict[str, str],
+]:
+    """流式读取 XLSX 合并单元格，不把工作表 XML 整体载入内存。"""
+
+    relationship_namespace = (
+        "http://schemas.openxmlformats.org/officeDocument/2006/relationships"
+    )
+    try:
+        with zipfile.ZipFile(io.BytesIO(raw)) as archive:
+            relationships: dict[str, tuple[str, str, str]] = {}
+            with archive.open("xl/_rels/workbook.xml.rels") as stream:
+                for _, element in ET.iterparse(stream, events=("end",)):
+                    if _xml_local_name(element.tag) != "Relationship":
+                        element.clear()
+                        continue
+                    relationship_id = element.attrib.get("Id")
+                    target = element.attrib.get("Target")
+                    target_mode = element.attrib.get("TargetMode", "Internal")
+                    relationship_type = element.attrib.get("Type", "")
+                    if relationship_id:
+                        if relationship_id in relationships:
+                            raise ValueError(
+                                "XLSX workbook contains duplicate relationship identifiers"
+                            )
+                        relationships[relationship_id] = (
+                            target or "",
+                            target_mode,
+                            relationship_type,
+                        )
+                    element.clear()
+
+            sheet_paths: dict[str, str] = {}
+            normalized_targets: dict[str, str] = {}
+            with archive.open("xl/workbook.xml") as stream:
+                for _, element in ET.iterparse(stream, events=("end",)):
+                    if _xml_local_name(element.tag) != "sheet":
+                        element.clear()
+                        continue
+                    title = element.attrib.get("name")
+                    relationship_id = element.attrib.get(
+                        f"{{{relationship_namespace}}}id"
+                    )
+                    if title and relationship_id:
+                        relationship = relationships.get(relationship_id)
+                        if relationship is None:
+                            raise ValueError(
+                                f"XLSX worksheet {title!r} references a missing relationship"
+                            )
+                        target, target_mode, relationship_type = relationship
+                        if target_mode.strip().lower() != "internal":
+                            raise ValueError(
+                                f"XLSX worksheet {title!r} uses an external relationship"
+                            )
+                        if not relationship_type.endswith("/worksheet"):
+                            raise ValueError(
+                                f"XLSX worksheet {title!r} has an invalid relationship type"
+                            )
+                        member_name = _resolve_xlsx_relationship_target(
+                            archive,
+                            target,
+                        )
+                        sheet_paths[title] = member_name
+                        canonical_target = f"/{member_name}"
+                        if target != canonical_target:
+                            normalized_targets[relationship_id] = canonical_target
+                    element.clear()
+
+            result: dict[str, tuple[tuple[int, int, int, int], ...]] = {}
+            total_ranges = 0
+            for title, member_name in sheet_paths.items():
+                ranges: list[tuple[int, int, int, int]] = []
+                with archive.open(member_name) as stream:
+                    for _, element in ET.iterparse(stream, events=("end",)):
+                        if _xml_local_name(element.tag) != "mergeCell":
+                            element.clear()
+                            continue
+                        reference = element.attrib.get("ref")
+                        if reference:
+                            try:
+                                boundaries = range_boundaries(reference)
+                            except ValueError as exc:
+                                raise ValueError(
+                                    f"XLSX worksheet {title!r} has an invalid merged range"
+                                ) from exc
+                            ranges.append(boundaries)
+                            total_ranges += 1
+                            if total_ranges > _MAX_WORKBOOK_MERGED_RANGES:
+                                raise ValueError(
+                                    "XLSX workbook contains too many merged ranges "
+                                    f"(limit {_MAX_WORKBOOK_MERGED_RANGES})"
+                                )
+                        element.clear()
+                result[title] = tuple(ranges)
+            return result, normalized_targets
+    except (KeyError, ET.ParseError, zipfile.BadZipFile) as exc:
+        raise ValueError(f"invalid XLSX workbook structure: {exc}") from exc
+
+def _xlsx_header_end_row(
+    first_row: int,
+    rows: Mapping[int, Sequence[Any]],
+    merged_ranges: Sequence[tuple[int, int, int, int]],
+) -> int:
+    """在固定深度内闭包表头合并关系，忽略越界或跨空行的可疑级联。"""
+
+    header_end = first_row
+    maximum_end = first_row + _MAX_WORKBOOK_HEADER_ROWS - 1
+    has_header_hierarchy = any(
+        max_column > min_column and min_row == first_row
+        for min_column, min_row, max_column, _ in merged_ranges
+    )
+    changed = True
+    while changed:
+        changed = False
+        for min_column, min_row, max_column, max_row in merged_ranges:
+            if min_row < first_row or min_row > header_end or max_row < first_row:
+                continue
+            if max_column == min_column and not has_header_hierarchy:
+                continue
+            candidate = max_row + 1 if max_column > min_column else max_row
+            if candidate <= header_end or candidate > maximum_end:
+                continue
+            if (
+                max_column > min_column
+                and len(_xlsx_nonempty_values(rows.get(min_row, ()))) < 2
+                and len(_xlsx_nonempty_values(rows.get(candidate, ()))) < 2
+            ):
+                continue
+            if any(
+                not rows.get(row_number)
+                for row_number in range(header_end + 1, candidate + 1)
+            ):
+                continue
+            header_end = candidate
+            changed = True
+    return header_end
+
+def _xlsx_headers(
+    title: str,
+    rows: Mapping[int, Sequence[Any]],
+    first_row: int,
+    header_end: int,
+    merged_ranges: Sequence[tuple[int, int, int, int]],
+) -> list[str]:
+    width = max((len(row) for row in rows.values()), default=0)
+    for min_column, min_row, max_column, max_row in merged_ranges:
+        if min_row <= header_end and max_row >= first_row:
+            width = max(width, max_column)
+    if width > _MAX_WORKBOOK_COLUMNS:
+        raise ValueError(
+            f"XLSX worksheet {title!r} exceeds {_MAX_WORKBOOK_COLUMNS} columns"
+        )
+
+    matrix = [
+        list(rows.get(row_number, ())) + [None] * (width - len(rows.get(row_number, ())))
+        for row_number in range(first_row, header_end + 1)
+    ]
+    for min_column, min_row, max_column, max_row in merged_ranges:
+        if min_row > header_end or max_row < first_row:
+            continue
+        source_row = max(first_row, min_row) - first_row
+        source_column = min_column - 1
+        source = matrix[source_row][source_column]
+        for row_number in range(max(first_row, min_row), min(header_end, max_row) + 1):
+            for column_number in range(min_column, max_column + 1):
+                matrix[row_number - first_row][column_number - 1] = source
+
+    headers: list[str] = []
+    for column in range(width):
+        components: list[str] = []
+        for row in matrix:
+            component = normalize_text(str(row[column] or ""))
+            if component and (not components or component != components[-1]):
+                components.append(component)
+        header = ".".join(components)
+        if not header:
+            raise ValueError(f"XLSX worksheet {title!r} contains an empty header")
+        headers.append(header)
+    if len(set(headers)) != len(headers):
+        raise ValueError(f"XLSX worksheet {title!r} contains duplicate headers")
+    return headers
+
+def _xlsx_nonempty_values(row: Sequence[Any]) -> list[Any]:
+    return [value for value in row if value not in {None, ""}]
+
+def _infer_xlsx_header_region(
+    title: str,
+    rows: Mapping[int, Sequence[Any]],
+    merged_ranges: Sequence[tuple[int, int, int, int]],
+) -> tuple[int, int, list[str]]:
+    """从有界前缀中推断表头，区分报表说明、多级表头和正文合并。"""
+
+    ordered_rows = sorted(rows)
+    if not ordered_rows:
+        return 0, 0, []
+    first_nonempty_row = ordered_rows[0]
+    horizontal_merge_rows = {
+        min_row
+        for min_column, min_row, max_column, _ in merged_ranges
+        if max_column > min_column
+    }
+    candidates: list[tuple[float, int, int, list[str]]] = []
+    for first_row in ordered_rows:
+        raw_values = _xlsx_nonempty_values(rows[first_row])
+        if not raw_values:
+            continue
+        if len(raw_values) < 2 and first_row in horizontal_merge_rows:
+            continue
+
+        header_end = _xlsx_header_end_row(first_row, rows, merged_ranges)
+        header_rows = {
+            row_number: rows[row_number]
+            for row_number in range(first_row, header_end + 1)
+            if row_number in rows
+        }
+        try:
+            headers = _xlsx_headers(
+                title,
+                header_rows,
+                first_row,
+                header_end,
+                merged_ranges,
+            )
+        except ValueError:
+            if header_end == first_row:
+                continue
+            header_end = first_row
+            try:
+                headers = _xlsx_headers(
+                    title,
+                    {first_row: rows[first_row]},
+                    first_row,
+                    first_row,
+                    merged_ranges,
+                )
+            except ValueError:
+                continue
+
+        data_rows = [rows[row_number] for row_number in ordered_rows if row_number > header_end]
+        if any(len(row) > len(headers) for row in data_rows):
+            continue
+
+        text_ratio = sum(isinstance(value, str) for value in raw_values) / len(raw_values)
+        score = text_ratio * 6 + min(len(raw_values), 4)
+        if data_rows:
+            first_data_values = _xlsx_nonempty_values(data_rows[0])
+            score += 4 * min(len(first_data_values), len(headers)) / len(headers)
+            if first_data_values:
+                score += (
+                    2
+                    * sum(
+                        not isinstance(value, str)
+                        for value in first_data_values
+                    )
+                    / len(first_data_values)
+                )
+        first_label = normalize_text(str(raw_values[0]))
+        if len(raw_values) <= 2 and _XLSX_REPORT_METADATA_PATTERN.match(first_label):
+            score -= 8
+        candidates.append((score, first_row, header_end, headers))
+
+    if not candidates:
+        first_row = first_nonempty_row
+        headers = _xlsx_headers(
+            title,
+            {first_row: rows[first_row]},
+            first_row,
+            first_row,
+            (),
+        )
+        return first_row, first_row, headers
+    _, first_row, header_end, headers = max(
+        candidates,
+        key=lambda candidate: (candidate[0], -candidate[1]),
+    )
+    return first_row, header_end, headers
+
+def _extract_xlsx_records(
+    raw: bytes,
+) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
+    _validate_office_archive(raw, "xlsx")
+    merged_by_sheet, normalized_targets = _xlsx_sheet_merge_ranges(raw)
+    workbook_raw = (
+        _rewrite_xlsx_workbook_relationships(raw, normalized_targets)
+        if normalized_targets
+        else raw
+    )
+    try:
+        workbook = load_workbook(
+            io.BytesIO(workbook_raw),
+            read_only=True,
+            data_only=True,
+            keep_links=False,
+        )
+    except Exception as exc:
+        raise ValueError(f"invalid XLSX file: {exc}") from exc
+
+    records: list[dict[str, Any]] = []
+    locators: list[dict[str, Any]] = []
+    total_cells = 0
+    try:
+        if len(workbook.worksheets) > _MAX_WORKBOOK_SHEETS:
+            raise ValueError(
+                f"XLSX contains too many worksheets (limit {_MAX_WORKBOOK_SHEETS})"
+            )
+        for sheet_index, worksheet in enumerate(workbook.worksheets):
+            reset_dimensions = getattr(worksheet, "reset_dimensions", None)
+            if callable(reset_dimensions):
+                reset_dimensions()
+            merged_ranges = merged_by_sheet.get(worksheet.title, ())
+            sheet_rows = 0
+            scanned_rows = 0
+            row_iterator = enumerate(
+                worksheet.iter_rows(values_only=True),
+                start=1,
+            )
+            buffered_rows: dict[int, Sequence[Any]] = {}
+
+            def normalized_row_values(
+                row: Sequence[Any],
+                sheet_title: str = worksheet.title,
+            ) -> list[Any]:
+                values = list(row)
+                while values and values[-1] in {None, ""}:
+                    values.pop()
+                if len(values) > _MAX_WORKBOOK_COLUMNS:
+                    raise ValueError(
+                        f"XLSX worksheet {sheet_title!r} exceeds "
+                        f"{_MAX_WORKBOOK_COLUMNS} columns"
+                    )
+                return values
+
+            for row_number, row in row_iterator:
+                scanned_rows += 1
+                if scanned_rows > _MAX_WORKBOOK_SCANNED_ROWS:
+                    raise ValueError(
+                        f"XLSX worksheet {worksheet.title!r} exceeds "
+                        f"{_MAX_WORKBOOK_SCANNED_ROWS} scanned rows"
+                    )
+                values = normalized_row_values(row)
+                if not values or all(value in {None, ""} for value in values):
+                    continue
+                buffered_rows[row_number] = tuple(values)
+                if len(buffered_rows) >= _MAX_WORKBOOK_HEADER_SCAN_ROWS:
+                    break
+
+            if not buffered_rows:
+                continue
+            _, header_end_row, headers = _infer_xlsx_header_region(
+                worksheet.title,
+                buffered_rows,
+                merged_ranges,
+            )
+
+            def append_record(
+                row_number: int,
+                values: Sequence[Any],
+                record_headers: Sequence[str] = tuple(headers),
+                locator_sheet_index: int = sheet_index,
+                sheet_title: str = worksheet.title,
+            ) -> None:
+                nonlocal total_cells, sheet_rows
+                row_values = list(values)
+                if len(row_values) > len(record_headers):
+                    raise ValueError(
+                        f"XLSX worksheet {sheet_title!r} has a row wider than its header"
+                    )
+                row_values.extend([None] * (len(record_headers) - len(row_values)))
+                record = {
+                    header: _normalize_spreadsheet_value(value)
+                    for header, value in zip(record_headers, row_values, strict=True)
+                }
+                if not any(value not in {"", None} for value in record.values()):
+                    return
+                sheet_record_index = sheet_rows
+                sheet_rows += 1
+                total_cells += len(record_headers)
+                if sheet_rows > _MAX_WORKBOOK_ROWS:
+                    raise ValueError(
+                        f"XLSX worksheet {sheet_title!r} exceeds "
+                        f"{_MAX_WORKBOOK_ROWS} data rows"
+                    )
+                if total_cells > _MAX_WORKBOOK_CELLS:
+                    raise ValueError(
+                        f"XLSX workbook exceeds {_MAX_WORKBOOK_CELLS} populated cells"
+                    )
+                records.append(record)
+                locators.append(
+                    {
+                        "kind": "xlsx",
+                        "record_index": len(records),
+                        "sheet_index": locator_sheet_index,
+                        "sheet_name": sheet_title,
+                        "row_number": row_number,
+                        "sheet_record_index": sheet_record_index,
+                    }
+                )
+
+            for row_number, values in buffered_rows.items():
+                if row_number > header_end_row:
+                    append_record(row_number, values)
+
+            for row_number, row in row_iterator:
+                scanned_rows += 1
+                if scanned_rows > _MAX_WORKBOOK_SCANNED_ROWS:
+                    raise ValueError(
+                        f"XLSX worksheet {worksheet.title!r} exceeds "
+                        f"{_MAX_WORKBOOK_SCANNED_ROWS} scanned rows"
+                    )
+                values = normalized_row_values(row)
+                if not values or all(value in {None, ""} for value in values):
+                    continue
+                append_record(row_number, values)
+    finally:
+        workbook.close()
+    return records, locators

backend/app/modules/data_process/algorithms/parsers/pdf.py

@@ -0,0 +1,299 @@
+"""数据处理算法 - PDF 文档解析。"""
+
+from __future__ import annotations
+
+import io
+import math
+import re
+import unicodedata
+from collections.abc import Sequence
+from typing import Literal
+
+from pypdf import PdfReader
+
+from ..text_utils import normalize_text
+from ..types import (
+    DocumentNoiseSpan,
+    PdfPageText,
+    _MAX_EXTRACTED_TEXT_CHARS,
+    _MAX_PDF_PAGES,
+    _PdfLine,
+)
+
+_PDF_PAGE_NUMBER_LINE_PATTERN = re.compile(
+    r"^(?:页次\s*)?(?:第\s*)?(?P<page>\d+)\s*页\s*"
+    r"(?:(?:[/／]\s*)?共\s*(?P<total>\d+)\s*页)?$"
+)
+_PDF_FRACTION_PAGE_LINE_PATTERN = re.compile(
+    r"^[—–-]?\s*(?P<page>\d+)\s*[/／]\s*(?P<total>\d+)\s*[—–-]?$"
+)
+_PDF_CLASSIFICATION_LABEL_PATTERN = re.compile(
+    r"^(?:(?:秘密等级|密级)\s*)?(?:商密|秘密|机密|绝密)"
+    r"\s*(?:[【\[(（][^】\])）]{1,8}[】\])）])?$"
+)
+_TOC_TITLE_PATTERN = re.compile(r"^(?:目\s*录|contents)$", re.IGNORECASE)
+_TOC_LEADER_ENTRY_PATTERN = re.compile(
+    r"(?:[.．…·•]\s*){3,}\s*\d{1,4}\s*$"
+)
+_TOC_NUMBERED_ENTRY_PATTERN = re.compile(
+    r"^(?:第[\u3400-\u4dbf\u4e00-\u9fff]{1,12}章|附表\s*\d+|\d+(?:\.\d+)+)"
+    r"\s+.+\s+\d{1,4}\s*$",
+    re.IGNORECASE,
+)
+_MARGIN_TEMPLATE_KEYWORDS = (
+    "页",
+    "页次",
+    "版本",
+    "文件编码",
+    "秘密等级",
+    "密级",
+    "商密",
+    "confidential",
+)
+
+
+def extract_pdf_page_texts(raw: bytes) -> tuple[PdfPageText, ...]:
+    """提取 PDF 各页文本，并保留与切片字符偏移一致的页范围。"""
+
+    if b"%PDF-" not in raw[:1024]:
+        raise ValueError("invalid PDF file: missing PDF header")
+    try:
+        reader = PdfReader(io.BytesIO(raw), strict=True)
+    except Exception as exc:
+        raise ValueError(f"invalid PDF file: {exc}") from exc
+    if reader.is_encrypted and not reader.decrypt(""):
+        raise ValueError("password-protected PDF files are not supported")
+    if len(reader.pages) > _MAX_PDF_PAGES:
+        raise ValueError(f"PDF contains too many pages (limit {_MAX_PDF_PAGES})")
+
+    pages: list[PdfPageText] = []
+    total = 0
+    has_text = False
+    for page_number, page in enumerate(reader.pages, start=1):
+        try:
+            text = normalize_text(page.extract_text() or "")
+        except Exception as exc:
+            raise ValueError(
+                f"failed to extract text from PDF page {page_number}: {exc}"
+            ) from exc
+        if not text:
+            pages.append(
+                PdfPageText(
+                    page_number=page_number,
+                    text="",
+                    source_start=total,
+                    source_end=total,
+                )
+            )
+            continue
+        if has_text:
+            total += 2
+        start = total
+        total += len(text)
+        if total > _MAX_EXTRACTED_TEXT_CHARS:
+            raise ValueError(
+                f"extracted document text exceeds {_MAX_EXTRACTED_TEXT_CHARS} characters"
+            )
+        pages.append(
+            PdfPageText(
+                page_number=page_number,
+                text=text,
+                source_start=start,
+                source_end=total,
+            )
+        )
+        has_text = True
+    if not has_text:
+        raise ValueError(
+            "PDF contains no extractable text; scanned or image-only PDF files are not supported"
+        )
+    return tuple(pages)
+
+def _pdf_page_lines(page: PdfPageText) -> tuple[_PdfLine, ...]:
+    lines: list[_PdfLine] = []
+    local_offset = 0
+    for raw_line in page.text.splitlines(keepends=True):
+        content = raw_line.rstrip("\r\n")
+        leading = len(content) - len(content.lstrip())
+        trailing = len(content.rstrip())
+        text = content.strip()
+        if text:
+            lines.append(
+                _PdfLine(
+                    text=text,
+                    start=page.source_start + local_offset + leading,
+                    end=page.source_start + local_offset + trailing,
+                )
+            )
+        local_offset += len(raw_line)
+    return tuple(lines)
+
+def _is_standalone_page_number(
+    text: str,
+    *,
+    physical_page: int,
+    page_count: int,
+) -> bool:
+    normalized = unicodedata.normalize("NFKC", text).strip()
+    match = _PDF_PAGE_NUMBER_LINE_PATTERN.fullmatch(normalized)
+    if match is None:
+        match = _PDF_FRACTION_PAGE_LINE_PATTERN.fullmatch(normalized)
+    if match is None or int(match.group("page")) != physical_page:
+        return False
+    total = match.groupdict().get("total")
+    return total is None or int(total) == page_count
+
+def _margin_signature(text: str) -> str:
+    normalized = unicodedata.normalize("NFKC", text).casefold()
+    normalized = re.sub(r"\s+", " ", normalized).strip()
+    if any(keyword in normalized for keyword in _MARGIN_TEMPLATE_KEYWORDS):
+        normalized = re.sub(r"\d+", "#", normalized)
+    return normalized
+
+def _has_margin_metadata_keyword(text: str) -> bool:
+    normalized = unicodedata.normalize("NFKC", text).casefold()
+    return any(keyword in normalized for keyword in _MARGIN_TEMPLATE_KEYWORDS)
+
+def _has_meaningful_margin_signature(signature: str) -> bool:
+    return len(re.sub(r"[#\W_]+", "", signature, flags=re.UNICODE)) >= 2
+
+def _is_toc_leader_entry(text: str) -> bool:
+    return bool(_TOC_LEADER_ENTRY_PATTERN.search(text))
+
+def _is_toc_numbered_entry(text: str) -> bool:
+    return bool(_TOC_NUMBERED_ENTRY_PATTERN.fullmatch(text))
+
+def detect_pdf_document_noise(
+    pages: Sequence[PdfPageText],
+) -> tuple[DocumentNoiseSpan, ...]:
+    """识别 PDF 中的独立页码、重复页边内容和高置信目录。
+
+    规则只查看每页顶部 5 行和底部 3 行来推断页眉页脚；目录必须有
+    明显的点引导线密度，避免仅因正文中出现“目录”或章节标题而误删。
+    """
+
+    page_lines = tuple(_pdf_page_lines(page) for page in pages)
+    detected: dict[tuple[int, int], DocumentNoiseSpan] = {}
+
+    def mark(
+        line: _PdfLine,
+        kind: Literal["page_number", "repeated_margin", "table_of_contents"],
+    ) -> None:
+        detected.setdefault(
+            (line.start, line.end),
+            DocumentNoiseSpan(
+                start=line.start,
+                end=line.end,
+                kind=kind,
+            ),
+        )
+
+    for page, lines in zip(pages, page_lines, strict=True):
+        for line in lines:
+            if _is_standalone_page_number(
+                line.text,
+                physical_page=page.page_number,
+                page_count=len(pages),
+            ):
+                mark(line, "page_number")
+        outer_margin_lines = (*lines[:2], *lines[-2:])
+        for line in outer_margin_lines:
+            if _PDF_CLASSIFICATION_LABEL_PATTERN.fullmatch(line.text):
+                mark(line, "repeated_margin")
+
+    # 只在三页及以上文档中推断通用页眉页脚，避免短文档误删。
+    if len(pages) >= 3:
+        signature_pages: dict[str, set[int]] = {}
+        candidate_lines: list[tuple[int, _PdfLine, str]] = []
+        for page_index, lines in enumerate(page_lines):
+            boundary_lines = (
+                *((line, index < 2) for index, line in enumerate(lines[:5])),
+                *((line, index < 2) for index, line in enumerate(reversed(lines[-3:]))),
+            )
+            seen_ranges: set[tuple[int, int]] = set()
+            for line, is_outer_margin in boundary_lines:
+                line_range = (line.start, line.end)
+                if (
+                    line_range in seen_ranges
+                    or line_range in detected
+                    or len(line.text) > 160
+                ):
+                    continue
+                seen_ranges.add(line_range)
+                if not is_outer_margin and not _has_margin_metadata_keyword(line.text):
+                    continue
+                signature = _margin_signature(line.text)
+                if not _has_meaningful_margin_signature(signature):
+                    continue
+                signature_pages.setdefault(signature, set()).add(page_index)
+                candidate_lines.append((page_index, line, signature))
+        minimum_pages = max(3, math.ceil(len(pages) * 0.3))
+        repeated_signatures = {
+            signature
+            for signature, matching_pages in signature_pages.items()
+            if len(matching_pages) >= minimum_pages
+        }
+        for _, line, signature in candidate_lines:
+            if signature in repeated_signatures:
+                mark(line, "repeated_margin")
+
+    # 先依据强证据判定目录页，再补充删除少量不带点引导线的编号目录项。
+    toc_active = False
+    for lines in page_lines:
+        content_lines = [
+            line for line in lines if (line.start, line.end) not in detected
+        ]
+        leader_entries = [line for line in content_lines if _is_toc_leader_entry(line.text)]
+        titles = [line for line in content_lines if _TOC_TITLE_PATTERN.fullmatch(line.text)]
+        starts_toc = bool(titles and len(leader_entries) >= 2)
+        is_toc_dense = bool(
+            len(leader_entries) >= 3
+            and len(leader_entries) / max(1, len(content_lines)) >= 0.5
+        )
+        if not (starts_toc or (toc_active and is_toc_dense)):
+            toc_active = False
+            continue
+        toc_active = True
+        for line in content_lines:
+            if (
+                line in titles
+                or _is_toc_leader_entry(line.text)
+                or _is_toc_numbered_entry(line.text)
+            ):
+                mark(line, "table_of_contents")
+
+    return tuple(sorted(detected.values(), key=lambda span: (span.start, span.end)))
+
+def remove_document_noise(
+    text: str,
+    spans: Sequence[DocumentNoiseSpan],
+    *,
+    source_offset: int = 0,
+) -> str:
+    """按原文绝对偏移移除噪声，不改动调用方保留的原文及偏移。"""
+
+    text_end = source_offset + len(text)
+    intersections = sorted(
+        (
+            max(0, span.start - source_offset),
+            min(len(text), span.end - source_offset),
+        )
+        for span in spans
+        if span.start < text_end and span.end > source_offset
+    )
+    if not intersections:
+        return text
+    parts: list[str] = []
+    cursor = 0
+    for start, end in intersections:
+        if end <= cursor:
+            continue
+        if start > cursor:
+            parts.append(text[cursor:start])
+        cursor = end
+    parts.append(text[cursor:])
+    cleaned = normalize_text("".join(parts))
+    return re.sub(r"\n{3,}", "\n\n", cleaned)
+
+def _extract_pdf_text(raw: bytes) -> str:
+    return "\n\n".join(page.text for page in extract_pdf_page_texts(raw) if page.text)

backend/app/modules/data_process/algorithms/quality.py

@@ -0,0 +1,343 @@
+"""数据处理算法 - 质量评分和去重。"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import re
+import unicodedata
+from collections import Counter
+from collections.abc import Iterable, Mapping, Sequence
+from copy import deepcopy
+from typing import Any
+
+from .text_utils import normalize_text
+from .types import (
+    _MAX_ANOMALY_TEXT_CHARS,
+    _MOJIBAKE_MARKERS,
+    _TOKEN_PATTERN,
+    ProcessedStructuredRecord,
+    QualityScore,
+)
+
+
+def estimate_token_count(text: str) -> int:
+    """粗略估计文本的 token 数量。"""
+    return len(_TOKEN_PATTERN.findall(text))
+
+
+def content_quality_flags(
+    text: str,
+    *,
+    min_chars: int = 20,
+    min_tokens: int = 5,
+    max_chars: int = _MAX_ANOMALY_TEXT_CHARS,
+) -> tuple[str, ...]:
+    """返回非结构化内容的确定性低质量原因。"""
+
+    if min_chars < 0 or min_tokens < 0 or max_chars <= 0:
+        raise ValueError("content quality limits must be non-negative")
+    normalized = normalize_text(text)
+    if not normalized:
+        return ("empty_content",)
+    flags: list[str] = []
+    if len(normalized) < min_chars or estimate_token_count(normalized) < min_tokens:
+        flags.append("content_too_short")
+    if len(normalized) > max_chars:
+        flags.append("content_too_long")
+    if any(marker in normalized for marker in _MOJIBAKE_MARKERS):
+        flags.append("mojibake")
+    nonspace = [char for char in normalized if not char.isspace()]
+    if nonspace:
+        readable_ratio = sum(
+            char.isprintable()
+            and unicodedata.category(char) not in {"Co", "Cs", "Cn"}
+            for char in nonspace
+        ) / len(nonspace)
+        if readable_ratio < 0.85:
+            flags.append("low_printable_ratio")
+        if len(nonspace) >= 100:
+            most_common = Counter(nonspace).most_common(1)[0][1]
+            if most_common / len(nonspace) > 0.9:
+                flags.append("repetitive_content")
+    return tuple(dict.fromkeys(flags))
+
+def is_low_quality_content(
+    text: str,
+    *,
+    min_chars: int = 20,
+    min_tokens: int = 5,
+    max_chars: int = _MAX_ANOMALY_TEXT_CHARS,
+) -> bool:
+    """判断内容是否命中任一低质量规则。"""
+
+    return bool(
+        content_quality_flags(
+            text,
+            min_chars=min_chars,
+            min_tokens=min_tokens,
+            max_chars=max_chars,
+        )
+    )
+
+def _deduplicate_structured_entries(
+    entries: Sequence[ProcessedStructuredRecord],
+) -> list[ProcessedStructuredRecord]:
+    """仅按整条 canonical JSON 稳定去重，避免误删同 ID 的更新记录。"""
+
+    # canonical_record_json 位于 structured_processing，延迟导入以断开循环依赖。
+    from .structured_processing import canonical_record_json
+
+    exact_seen: set[str] = set()
+    unique: list[ProcessedStructuredRecord] = []
+    for entry in entries:
+        record = entry.record
+        fingerprint = hashlib.sha256(canonical_record_json(record).encode("utf-8")).hexdigest()
+        if fingerprint in exact_seen:
+            continue
+        exact_seen.add(fingerprint)
+        unique.append(
+            ProcessedStructuredRecord(
+                entry.source_index,
+                deepcopy(dict(record)),
+            )
+        )
+    return unique
+
+def deduplicate_structured_records(
+    records: Sequence[Mapping[str, Any]],
+) -> list[dict[str, Any]]:
+    """仅按整条 canonical JSON 稳定去重。"""
+
+    entries = [
+        ProcessedStructuredRecord(index, deepcopy(dict(record)))
+        for index, record in enumerate(records)
+    ]
+    return [entry.record for entry in _deduplicate_structured_entries(entries)]
+
+def _near_duplicate_features(text: str, shingle_size: int) -> tuple[str, ...]:
+    if isinstance(shingle_size, bool) or not isinstance(shingle_size, int):
+        raise TypeError("shingle_size must be an integer")
+    if shingle_size <= 0:
+        raise ValueError("shingle_size must be greater than 0")
+    tokens = re.findall(
+        r"[\u3400-\u4dbf\u4e00-\u9fff]|[A-Za-z0-9_]+",
+        normalize_text(text).casefold(),
+    )
+    if not tokens:
+        return ()
+    if len(tokens) < shingle_size:
+        return ("\x1f".join(tokens),)
+    return tuple(
+        "\x1f".join(tokens[index : index + shingle_size])
+        for index in range(len(tokens) - shingle_size + 1)
+    )
+
+def near_duplicate_fingerprint(text: str, *, shingle_size: int = 3) -> str:
+    """生成 64 位 SimHash 指纹，用于低成本近重复候选筛选。"""
+
+    if isinstance(shingle_size, bool) or not isinstance(shingle_size, int):
+        raise TypeError("shingle_size must be an integer")
+    if shingle_size <= 0:
+        raise ValueError("shingle_size must be greater than 0")
+    features = Counter(_near_duplicate_features(text, shingle_size))
+    if not features:
+        return "0" * 16
+    vector = [0] * 64
+    for feature, weight in features.items():
+        digest = int.from_bytes(hashlib.sha256(feature.encode("utf-8")).digest()[:8], "big")
+        for bit in range(64):
+            vector[bit] += weight if digest & (1 << bit) else -weight
+    fingerprint = sum(1 << bit for bit, value in enumerate(vector) if value >= 0)
+    return f"{fingerprint:016x}"
+
+def fingerprints_are_near_duplicate(
+    left: str,
+    right: str,
+    *,
+    max_hamming_distance: int = 3,
+) -> bool:
+    """比较两个 64 位十六进制 SimHash 指纹。"""
+
+    if isinstance(max_hamming_distance, bool) or not isinstance(max_hamming_distance, int):
+        raise TypeError("max_hamming_distance must be an integer")
+    if not 0 <= max_hamming_distance <= 64:
+        raise ValueError("max_hamming_distance must be in [0, 64]")
+    if not re.fullmatch(r"[0-9a-fA-F]{16}", left) or not re.fullmatch(
+        r"[0-9a-fA-F]{16}", right
+    ):
+        raise ValueError("fingerprints must be 16-character hexadecimal strings")
+    distance = (int(left, 16) ^ int(right, 16)).bit_count()
+    return distance <= max_hamming_distance
+
+def is_near_duplicate(
+    left: str,
+    right: str,
+    *,
+    shingle_size: int = 3,
+    similarity_threshold: float = 0.9,
+    max_hamming_distance: int = 3,
+) -> bool:
+    """结合词片 Jaccard 和 SimHash 判断两段内容是否近重复。"""
+
+    if isinstance(similarity_threshold, bool) or not isinstance(
+        similarity_threshold, (int, float)
+    ):
+        raise TypeError("similarity_threshold must be a number")
+    if not 0 <= similarity_threshold <= 1:
+        raise ValueError("similarity_threshold must be in [0, 1]")
+    if isinstance(max_hamming_distance, bool) or not isinstance(max_hamming_distance, int):
+        raise TypeError("max_hamming_distance must be an integer")
+    if not 0 <= max_hamming_distance <= 64:
+        raise ValueError("max_hamming_distance must be in [0, 64]")
+    left_normalized = normalize_text(left)
+    right_normalized = normalize_text(right)
+    if not left_normalized or not right_normalized:
+        return left_normalized == right_normalized
+    if left_normalized.casefold() == right_normalized.casefold():
+        return True
+    left_features = set(_near_duplicate_features(left_normalized, shingle_size))
+    right_features = set(_near_duplicate_features(right_normalized, shingle_size))
+    union = left_features | right_features
+    similarity = len(left_features & right_features) / len(union) if union else 1.0
+    if similarity >= similarity_threshold:
+        return True
+    return fingerprints_are_near_duplicate(
+        near_duplicate_fingerprint(left_normalized, shingle_size=shingle_size),
+        near_duplicate_fingerprint(right_normalized, shingle_size=shingle_size),
+        max_hamming_distance=max_hamming_distance,
+    )
+
+def record_fingerprint(record: Mapping[str, Any]) -> str:
+    """计算与字典键顺序无关的稳定记录指纹。"""
+
+    canonical = {
+        "instruction": normalize_text(str(record.get("instruction") or "")),
+        "input": normalize_text(str(record.get("input") or "")),
+        "output": normalize_text(str(record.get("output") or "")),
+    }
+    raw = json.dumps(canonical, ensure_ascii=False, sort_keys=True, separators=(",", ":"))
+    return hashlib.sha256(raw.encode("utf-8")).hexdigest()
+
+def _readability_score(text: str) -> float:
+    if not text:
+        return 0.0
+    nonspace = [char for char in text if not char.isspace()]
+    if not nonspace:
+        return 0.0
+    printable_ratio = sum(char.isprintable() for char in nonspace) / len(nonspace)
+    useful_ratio = sum(
+        char.isalnum() or "\u3400" <= char <= "\u9fff" or unicodedata.category(char).startswith("P")
+        for char in nonspace
+    ) / len(nonspace)
+    return round(100 * (0.65 * printable_ratio + 0.35 * useful_ratio), 2)
+
+
+def _internal_duplicate_score(text: str) -> float:
+    units = [unit.strip().lower() for unit in re.split(r"[\n。！？!?；;]+", text) if unit.strip()]
+    if len(units) <= 1:
+        return 100.0
+    return round(100 * len(set(units)) / len(units), 2)
+
+
+def _source_relevance_score(record: Mapping[str, Any], source_content: str) -> float:
+    """估算结果与来源文本的词元覆盖率。
+
+    这是无外部模型依赖、可重复的首版评分。没有来源文本（例如人工新增结果）
+    时不扣分；存在来源时，以结果中的有效词元被来源覆盖的比例计分。
+    """
+
+    source = normalize_text(source_content)
+    if not source:
+        return 100.0
+    candidate = normalize_text(
+        "\n".join(
+            str(record.get(field) or "") for field in ("instruction", "input", "output")
+        )
+    )
+
+    def semantic_tokens(text: str) -> set[str]:
+        return {
+            token.lower()
+            for token in _TOKEN_PATTERN.findall(text)
+            if token.isalnum() or "\u3400" <= token <= "\u9fff"
+        }
+
+    source_tokens = semantic_tokens(source)
+    candidate_tokens = semantic_tokens(candidate)
+    if not candidate_tokens:
+        return 0.0
+    if not source_tokens:
+        return 0.0
+    return round(100 * len(candidate_tokens & source_tokens) / len(candidate_tokens), 2)
+
+
+def score_quality(
+    record: Mapping[str, Any],
+    *,
+    min_output_length: int = 20,
+    source_content: str = "",
+    known_fingerprints: Iterable[str] = (),
+    threshold: float = 60.0,
+) -> QualityScore:
+    """按完整性、长度、可读性、来源相关性和重复度计算质量分。"""
+
+    if min_output_length <= 0:
+        raise ValueError("min_output_length must be greater than 0")
+    if not 0 <= threshold <= 100:
+        raise ValueError("threshold must be in [0, 100]")
+
+    instruction = normalize_text(str(record.get("instruction") or ""))
+    input_text = normalize_text(str(record.get("input") or ""))
+    output = normalize_text(str(record.get("output") or ""))
+    flags: list[str] = []
+
+    completeness = 100.0
+    if not instruction:
+        completeness -= 50
+        flags.append("missing_instruction")
+    if not output:
+        completeness -= 50
+        flags.append("missing_output")
+
+    output_length = len(output)
+    length_score = round(min(100.0, output_length / min_output_length * 100), 2)
+    if output_length < min_output_length:
+        flags.append("output_too_short")
+
+    readability = _readability_score("\n".join((instruction, input_text, output)))
+    if readability < 70:
+        flags.append("low_readability")
+
+    relevance = _source_relevance_score(record, source_content)
+    if source_content and relevance < 30:
+        flags.append("low_source_relevance")
+
+    fingerprint = record_fingerprint(record)
+    known = set(known_fingerprints)
+    duplicate = 0.0 if fingerprint in known else _internal_duplicate_score(output)
+    if duplicate == 0:
+        flags.append("duplicate_record")
+    elif duplicate < 70:
+        flags.append("repetitive_output")
+
+    overall = round(
+        completeness * 0.35
+        + length_score * 0.20
+        + readability * 0.20
+        + relevance * 0.15
+        + duplicate * 0.10,
+        2,
+    )
+    hard_valid = bool(instruction and output)
+    return QualityScore(
+        overall=overall,
+        completeness=completeness,
+        length=length_score,
+        readability=readability,
+        relevance=relevance,
+        duplicate=duplicate,
+        is_valid=hard_valid and overall >= threshold,
+        flags=tuple(flags),
+        fingerprint=fingerprint,
+    )

backend/app/modules/data_process/algorithms/structured_processing.py

@@ -0,0 +1,809 @@
+"""数据处理算法 - 结构化数据处理。"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import math
+import re
+from collections.abc import Iterable, Mapping, Sequence
+from copy import deepcopy
+from datetime import date, datetime, time
+from decimal import Decimal
+from typing import Any
+
+from .parsers.json_parser import _extract_structured_records_with_locators
+from .quality import (
+    _deduplicate_structured_entries,
+    content_quality_flags,
+    estimate_token_count,
+)
+from .text_utils import _normalize_field_name, normalize_text, structured_json_dumps
+from .transforms import stable_split_assignments
+from .types import (
+    _CHINESE_NAME_CONTEXT_PATTERN,
+    _EMAIL_PATTERN,
+    _ENGLISH_NAME_CONTEXT_PATTERN,
+    _ID_CARD_PATTERN,
+    _IDENTITY_FIELD_PATTERN,
+    _MAX_STRUCTURED_DEPTH,
+    _MAX_STRUCTURED_FIELDS,
+    _NAME_FIELD_NAMES,
+    _PHONE_PATTERN,
+    _STRUCTURED_OPTIONS,
+    MAX_QA_PAIRS_PER_ITEM,
+    ProcessedStructuredRecord,
+    StructuredPreprocessOption,
+)
+
+
+def _canonical_value(value: Any) -> Any:
+    if value is None or isinstance(value, (bool, int)):
+        return value
+    if isinstance(value, Decimal):
+        if not value.is_finite():
+            raise ValueError("non-finite JSON number is not allowed")
+        return value
+    if isinstance(value, float):
+        if not math.isfinite(value):
+            raise ValueError("non-finite JSON number is not allowed")
+        return value
+    if isinstance(value, str):
+        return normalize_text(value)
+    if isinstance(value, (datetime, date, time)):
+        return value.isoformat()
+    if isinstance(value, Mapping):
+        normalized: dict[str, Any] = {}
+        for key, item in sorted(value.items(), key=lambda pair: str(pair[0])):
+            normalized_key = normalize_text(str(key))
+            if not normalized_key:
+                raise ValueError("structured record contains an empty field name")
+            if normalized_key in normalized:
+                raise ValueError(
+                    f"structured record fields collide after normalization: {normalized_key}"
+                )
+            normalized[normalized_key] = _canonical_value(item)
+        return normalized
+    if isinstance(value, (list, tuple)):
+        return [_canonical_value(item) for item in value]
+    if isinstance(value, (set, frozenset)):
+        items = [_canonical_value(item) for item in value]
+        return sorted(
+            items,
+            key=lambda item: structured_json_dumps(item, sort_keys=True),
+        )
+    if isinstance(value, (bytes, bytearray, memoryview)):
+        return bytes(value).hex()
+    return normalize_text(str(value))
+
+
+def _is_empty_value(value: Any) -> bool:
+    if value is None:
+        return True
+    if isinstance(value, str):
+        return not normalize_text(value)
+    if isinstance(value, Mapping):
+        return not value or all(_is_empty_value(item) for item in value.values())
+    if isinstance(value, (list, tuple, set, frozenset)):
+        return not value or all(_is_empty_value(item) for item in value)
+    return False
+
+
+def _is_name_field(field: Any) -> bool:
+    raw_field = normalize_text(str(field))
+    if not raw_field:
+        return False
+
+    # 只匹配明确表示自然人姓名的字段，避免将 table_name、product_name、
+    # chinese_name 等业务名称或元数据字段误判为个人敏感信息。
+    if _normalize_field_name(raw_field, "snake_case") in _NAME_FIELD_NAMES:
+        return True
+
+    # detect_structure 会使用点号生成扁平化路径（例如 profile.name）；此时仅
+    # 判断最后一个路径段，不能退回到宽泛的 ``*_name`` 后缀匹配。
+    if "." not in raw_field:
+        return False
+    leaf_field = raw_field.rsplit(".", 1)[-1]
+    return _normalize_field_name(leaf_field, "snake_case") in _NAME_FIELD_NAMES
+
+
+def _embedded_structure(value: Any) -> Any:
+    if not isinstance(value, str):
+        return value
+    candidate = value.strip()
+    if not candidate or candidate[0] not in "[{":
+        return value
+    try:
+        parsed = json.loads(candidate)
+    except json.JSONDecodeError:
+        return value
+    return parsed if isinstance(parsed, (Mapping, list)) else value
+
+
+def _structured_options(options: Iterable[str] | Mapping[str, Any]) -> set[str]:
+    if isinstance(options, str):
+        raise TypeError("options must be an iterable or mapping of option names")
+    if isinstance(options, Mapping):
+        enabled = {str(key) for key, value in options.items() if bool(value)}
+    else:
+        enabled = {str(option) for option in options}
+    unknown = enabled - _STRUCTURED_OPTIONS
+    if unknown:
+        raise ValueError(f"unsupported structured preprocess options: {', '.join(sorted(unknown))}")
+    return enabled
+
+
+def _clean_invalid_structured_entries(
+    entries: Sequence[ProcessedStructuredRecord],
+) -> list[ProcessedStructuredRecord]:
+    if not entries:
+        return []
+    fields: list[str] = []
+    for entry in entries:
+        record = entry.record
+        for field in record:
+            if field not in fields:
+                fields.append(field)
+    active_fields = [
+        field
+        for field in fields
+        if any(not _is_empty_value(entry.record.get(field)) for entry in entries)
+    ]
+    if not active_fields:
+        return []
+    cleaned: list[ProcessedStructuredRecord] = []
+    for entry in entries:
+        record = entry.record
+        values = {field: deepcopy(record.get(field)) for field in active_fields}
+        # 清洗只依据整行是否为空。外键、父级 ID 等字段天然允许为空，不能
+        # 因为字段名以 *_id 结尾就把它们全部提升为联合必填项。
+        if all(_is_empty_value(value) for value in values.values()):
+            continue
+        cleaned.append(ProcessedStructuredRecord(entry.source_index, values))
+    return cleaned
+
+
+def _percentile(values: Sequence[float], fraction: float) -> float:
+    if not values:
+        raise ValueError("cannot calculate a percentile of an empty sequence")
+    ordered = sorted(values)
+    position = (len(ordered) - 1) * fraction
+    lower = math.floor(position)
+    upper = math.ceil(position)
+    if lower == upper:
+        return ordered[lower]
+    weight = position - lower
+    return ordered[lower] * (1 - weight) + ordered[upper] * weight
+
+
+def _filter_anomalous_structured_entries(
+    entries: Sequence[ProcessedStructuredRecord],
+    *,
+    iqr_multiplier: float = 1.5,
+) -> list[ProcessedStructuredRecord]:
+    """按字段级数值 IQR、乱码和极端文本长度过滤异常记录。"""
+
+    if iqr_multiplier <= 0:
+        raise ValueError("iqr_multiplier must be greater than 0")
+    numeric_values: dict[str, list[float]] = {}
+    text_lengths: dict[str, list[float]] = {}
+    for entry in entries:
+        record = entry.record
+        for field, value in record.items():
+            if (
+                isinstance(value, (int, float))
+                and not isinstance(value, bool)
+                and not _IDENTITY_FIELD_PATTERN.search(
+                    _normalize_field_name(field, "snake_case")
+                )
+            ):
+                number = float(value)
+                if math.isfinite(number):
+                    numeric_values.setdefault(field, []).append(number)
+            elif isinstance(value, str) and value:
+                text_lengths.setdefault(field, []).append(float(len(value)))
+
+    numeric_bounds: dict[str, tuple[float, float]] = {}
+    for field, values in numeric_values.items():
+        # 小样本不做统计异常判断，避免把合法长尾值误删。
+        if len(values) < 8:
+            continue
+        first_quartile = _percentile(values, 0.25)
+        third_quartile = _percentile(values, 0.75)
+        spread = third_quartile - first_quartile
+        numeric_bounds[field] = (
+            first_quartile - iqr_multiplier * spread,
+            third_quartile + iqr_multiplier * spread,
+        )
+
+    text_upper_bounds: dict[str, float] = {}
+    for field, lengths in text_lengths.items():
+        if len(lengths) < 8:
+            continue
+        first_quartile = _percentile(lengths, 0.25)
+        third_quartile = _percentile(lengths, 0.75)
+        spread = third_quartile - first_quartile
+        text_upper_bounds[field] = max(512.0, third_quartile + 3 * spread)
+
+    accepted: list[ProcessedStructuredRecord] = []
+    for entry in entries:
+        record = entry.record
+        anomalous = False
+        for field, value in record.items():
+            if (
+                isinstance(value, (int, float))
+                and not isinstance(value, bool)
+                and not _IDENTITY_FIELD_PATTERN.search(
+                    _normalize_field_name(field, "snake_case")
+                )
+            ):
+                number = float(value)
+                if not math.isfinite(number):
+                    anomalous = True
+                    break
+                bounds = numeric_bounds.get(field)
+                if bounds and not bounds[0] <= number <= bounds[1]:
+                    anomalous = True
+                    break
+            if isinstance(value, str):
+                flags = content_quality_flags(value, min_chars=0, min_tokens=0)
+                if {"content_too_long", "mojibake", "low_printable_ratio"} & set(flags):
+                    anomalous = True
+                    break
+                upper_bound = text_upper_bounds.get(field)
+                if upper_bound is not None and len(value) > upper_bound:
+                    anomalous = True
+                    break
+        if not anomalous:
+            accepted.append(
+                ProcessedStructuredRecord(
+                    entry.source_index,
+                    deepcopy(dict(record)),
+                )
+            )
+    return accepted
+
+
+def _preview_content(item: Mapping[str, Any]) -> str:
+    for field in ("edited_content", "editedContent", "original_content", "originalContent", "content"):
+        value = item.get(field)
+        if value is not None:
+            return normalize_text(str(value))
+    return ""
+
+
+def _standard_fields(content: str) -> tuple[str, str, str]:
+    if not content:
+        return "", "", ""
+
+    try:
+        payload = json.loads(content)
+    except json.JSONDecodeError:
+        payload = None
+    if isinstance(payload, Mapping):
+        instruction = next(
+            (
+                str(payload[key])
+                for key in ("instruction", "question", "prompt")
+                if payload.get(key) is not None
+            ),
+            "",
+        )
+        input_text = next(
+            (str(payload[key]) for key in ("input", "context") if payload.get(key) is not None),
+            "",
+        )
+        output = next(
+            (str(payload[key]) for key in ("output", "answer", "response") if payload.get(key) is not None),
+            "",
+        )
+        if instruction or output:
+            return normalize_text(instruction), normalize_text(input_text), normalize_text(output)
+
+    question_answer = re.match(
+        r"^\s*(?:问|question)\s*[:：]\s*(.+?)(?:\n|\r\n?)\s*(?:答|answer)\s*[:：]\s*(.+)\s*$",
+        content,
+        flags=re.IGNORECASE | re.DOTALL,
+    )
+    if question_answer:
+        return normalize_text(question_answer.group(1)), "", normalize_text(question_answer.group(2))
+
+    lines = [line.strip() for line in content.splitlines() if line.strip()]
+    first_line = re.sub(r"^(?:问|question)\s*[:：]\s*", "", lines[0], flags=re.IGNORECASE)
+    output = normalize_text("\n".join(lines[1:])) if len(lines) > 1 else normalize_text(content)
+    return normalize_text(first_line), "", output
+
+
+def _protected_markdown_ranges(
+    text: str,
+    *,
+    preserve_code_blocks: bool,
+    preserve_tables: bool,
+    preserve_lists: bool,
+) -> list[tuple[int, int]]:
+    """找出不应从中间切开的 Markdown 代码块、表格和连续列表。"""
+
+    lines: list[tuple[int, int, str]] = []
+    cursor = 0
+    for raw_line in text.splitlines(keepends=True):
+        end = cursor + len(raw_line)
+        lines.append((cursor, end, raw_line.rstrip("\r\n")))
+        cursor = end
+    if cursor < len(text) or not lines:
+        lines.append((cursor, len(text), text[cursor:]))
+
+    ranges: list[tuple[int, int]] = []
+    code_line_indexes: set[int] = set()
+    if preserve_code_blocks:
+        open_block: tuple[int, str, int] | None = None
+        for index, (start, end, content) in enumerate(lines):
+            fence = re.match(r"^\s*(`{3,}|~{3,})", content)
+            if not fence:
+                continue
+            marker = fence.group(1)[0]
+            length = len(fence.group(1))
+            if open_block is None:
+                open_block = (index, marker, length)
+                continue
+            first_index, open_marker, open_length = open_block
+            if marker == open_marker and length >= open_length:
+                ranges.append((lines[first_index][0], end))
+                code_line_indexes.update(range(first_index, index + 1))
+                open_block = None
+        if open_block is not None:
+            first_index = open_block[0]
+            ranges.append((lines[first_index][0], len(text)))
+            code_line_indexes.update(range(first_index, len(lines)))
+
+    if preserve_tables:
+        index = 0
+        while index + 1 < len(lines):
+            if index in code_line_indexes:
+                index += 1
+                continue
+            header = lines[index][2].strip()
+            separator = lines[index + 1][2].strip().strip("|")
+            cells = [cell.strip() for cell in separator.split("|")]
+            if (
+                "|" not in header
+                or len(cells) < 2
+                or not all(re.fullmatch(r":?-{3,}:?", cell) for cell in cells)
+            ):
+                index += 1
+                continue
+            end_index = index + 1
+            while (
+                end_index + 1 < len(lines)
+                and end_index + 1 not in code_line_indexes
+                and lines[end_index + 1][2].strip()
+                and "|" in lines[end_index + 1][2]
+            ):
+                end_index += 1
+            ranges.append((lines[index][0], lines[end_index][1]))
+            index = end_index + 1
+
+    if preserve_lists:
+        list_pattern = re.compile(r"^\s*(?:[-+*]|\d+[.)])\s+\S")
+        continuation_pattern = re.compile(r"^\s{2,}\S")
+        index = 0
+        while index < len(lines):
+            if index in code_line_indexes or not list_pattern.match(lines[index][2]):
+                index += 1
+                continue
+            end_index = index
+            item_count = 1
+            while end_index + 1 < len(lines) and end_index + 1 not in code_line_indexes:
+                next_line = lines[end_index + 1][2]
+                if list_pattern.match(next_line):
+                    item_count += 1
+                    end_index += 1
+                elif continuation_pattern.match(next_line):
+                    end_index += 1
+                else:
+                    break
+            if item_count >= 2:
+                ranges.append((lines[index][0], lines[end_index][1]))
+            index = end_index + 1
+
+    merged: list[tuple[int, int]] = []
+    for start, end in sorted(ranges):
+        if merged and start < merged[-1][1]:
+            merged[-1] = (merged[-1][0], max(merged[-1][1], end))
+        else:
+            merged.append((start, end))
+    return merged
+
+
+def extract_structured_records(text: str, file_format: str) -> list[dict[str, Any]]:
+    """从 JSON、JSONL 或 CSV 中提取规范化记录。"""
+
+    records, _ = _extract_structured_records_with_locators(text, file_format)
+    return records
+
+def canonical_record_json(record: Mapping[str, Any]) -> str:
+    """生成与字段顺序无关、可用于比较和落库的 canonical JSON。"""
+
+    if not isinstance(record, Mapping):
+        raise TypeError("record must be a mapping")
+    return structured_json_dumps(_canonical_value(record), sort_keys=True)
+
+def normalize_structured_record(
+    record: Mapping[str, Any],
+    *,
+    field_name_style: str = "snake_case",
+) -> dict[str, Any]:
+    """规范字段名、Unicode/空白、容器类型和不可 JSON 化的标量。"""
+
+    if not isinstance(record, Mapping):
+        raise TypeError("record must be a mapping")
+    normalized: dict[str, Any] = {}
+    for key, value in record.items():
+        normalized_key = _normalize_field_name(key, field_name_style)
+        if not normalized_key:
+            raise ValueError("structured record contains an empty field name")
+        if normalized_key in normalized:
+            raise ValueError(
+                f"structured record fields collide after normalization: {normalized_key}"
+            )
+        normalized[normalized_key] = _canonical_value(value)
+    return dict(sorted(normalized.items()))
+
+def flatten_structured_record(
+    record: Mapping[str, Any],
+    *,
+    separator: str = ".",
+) -> dict[str, Any]:
+    """把嵌套对象展平；数组保留为 canonical JSON 兼容值。"""
+
+    if not isinstance(record, Mapping):
+        raise TypeError("record must be a mapping")
+    if not separator:
+        raise ValueError("separator cannot be empty")
+    flattened: dict[str, Any] = {}
+
+    def visit(value: Any, path: tuple[str, ...], depth: int) -> None:
+        if depth > _MAX_STRUCTURED_DEPTH:
+            raise ValueError(
+                f"structured record nesting exceeds {_MAX_STRUCTURED_DEPTH} levels"
+            )
+        value = _embedded_structure(value)
+        if isinstance(value, Mapping):
+            if not value and path:
+                key = separator.join(path)
+                flattened[key] = {}
+                return
+            for child_key, child_value in value.items():
+                normalized_key = normalize_text(str(child_key))
+                if not normalized_key:
+                    raise ValueError("structured record contains an empty field name")
+                visit(child_value, (*path, normalized_key), depth + 1)
+            return
+        key = separator.join(path)
+        if key in flattened:
+            raise ValueError(f"structured record fields collide while flattening: {key}")
+        flattened[key] = _canonical_value(value)
+        if len(flattened) > _MAX_STRUCTURED_FIELDS:
+            raise ValueError(
+                f"structured record exceeds {_MAX_STRUCTURED_FIELDS} flattened fields"
+            )
+
+    for field, value in record.items():
+        field_name = normalize_text(str(field))
+        if not field_name:
+            raise ValueError("structured record contains an empty field name")
+        visit(value, (field_name,), 1)
+    return flattened
+
+def filter_anomalous_structured_records(
+    records: Sequence[Mapping[str, Any]],
+    *,
+    iqr_multiplier: float = 1.5,
+) -> list[dict[str, Any]]:
+    """按字段级数值 IQR、乱码和极端文本长度过滤异常记录。"""
+
+    entries = [
+        ProcessedStructuredRecord(index, deepcopy(dict(record)))
+        for index, record in enumerate(records)
+    ]
+    return [
+        entry.record
+        for entry in _filter_anomalous_structured_entries(
+            entries,
+            iqr_multiplier=iqr_multiplier,
+        )
+    ]
+
+def desensitize_pii(text: str) -> tuple[str, dict[str, int]]:
+    """掩码邮箱、手机号、身份证号及有明确上下文的姓名。"""
+
+    if not isinstance(text, str):
+        raise TypeError("text must be str")
+    counts: dict[str, int] = {"email": 0, "phone": 0, "id_card": 0}
+
+    def replace(pattern: re.Pattern[str], replacement: str, kind: str, value: str) -> str:
+        def replacer(_: re.Match[str]) -> str:
+            counts[kind] += 1
+            return replacement
+
+        return pattern.sub(replacer, value)
+
+    masked = replace(_EMAIL_PATTERN, "[EMAIL]", "email", text)
+    masked = replace(_ID_CARD_PATTERN, "[ID_CARD]", "id_card", masked)
+    masked = replace(_PHONE_PATTERN, "[PHONE]", "phone", masked)
+
+    def replace_context_name(match: re.Match[str]) -> str:
+        counts["name"] = counts.get("name", 0) + 1
+        return f"{match.group('label')}{match.group('separator')}[NAME]"
+
+    masked = _CHINESE_NAME_CONTEXT_PATTERN.sub(replace_context_name, masked)
+    masked = _ENGLISH_NAME_CONTEXT_PATTERN.sub(replace_context_name, masked)
+    counts["total"] = sum(counts.values())
+    return masked, counts
+
+def desensitize_structured_record(
+    record: Mapping[str, Any],
+) -> tuple[dict[str, Any], dict[str, int]]:
+    """递归脱敏结构化姓名字段及任意文本中的手机号、邮箱、身份证号。"""
+
+    if not isinstance(record, Mapping):
+        raise TypeError("record must be a mapping")
+    counts: dict[str, int] = {"email": 0, "phone": 0, "id_card": 0}
+
+    def add_counts(values: Mapping[str, int]) -> None:
+        for kind, count in values.items():
+            if kind != "total" and count:
+                counts[kind] = counts.get(kind, 0) + count
+
+    def visit(value: Any, field: Any = "") -> Any:
+        if isinstance(value, Mapping):
+            return {key: visit(item, key) for key, item in value.items()}
+        if isinstance(value, list):
+            return [visit(item, field) for item in value]
+        if isinstance(value, tuple):
+            return [visit(item, field) for item in value]
+        if _is_name_field(field) and not _is_empty_value(value):
+            counts["name"] = counts.get("name", 0) + 1
+            return "[NAME]"
+        if isinstance(value, str):
+            masked, found = desensitize_pii(value)
+            add_counts(found)
+            return masked
+        return deepcopy(value)
+
+    masked = {key: visit(value, key) for key, value in record.items()}
+    counts["total"] = sum(counts.values())
+    return masked, counts
+
+def preprocess_structured_records_with_lineage(
+    records: Iterable[Mapping[str, Any]],
+    options: Iterable[str] | Mapping[str, Any],
+) -> list[ProcessedStructuredRecord]:
+    """执行结构化预处理，并保留每条结果在原始输入中的稳定索引。"""
+
+    enabled = _structured_options(options)
+    current: list[ProcessedStructuredRecord] = []
+    for source_index, record in enumerate(records):
+        if not isinstance(record, Mapping):
+            if "clean_invalid" in enabled:
+                continue
+            raise TypeError("structured records must contain mappings")
+        value = deepcopy(dict(record))
+        if "detect_structure" in enabled:
+            value = flatten_structured_record(value)
+        if "normalize_format" in enabled:
+            value = normalize_structured_record(value)
+        current.append(ProcessedStructuredRecord(source_index, value))
+
+    if "clean_invalid" in enabled:
+        current = _clean_invalid_structured_entries(current)
+    if "filter_anomaly" in enabled:
+        current = _filter_anomalous_structured_entries(current)
+    if "deduplicate" in enabled:
+        current = _deduplicate_structured_entries(current)
+    if "desensitize" in enabled:
+        current = [
+            ProcessedStructuredRecord(
+                entry.source_index,
+                desensitize_structured_record(entry.record)[0],
+            )
+            for entry in current
+        ]
+    return current
+
+def preprocess_structured_records(
+    records: Iterable[Mapping[str, Any]],
+    options: Iterable[str] | Mapping[str, Any],
+) -> list[dict[str, Any]]:
+    """按界面选项执行确定性、无副作用的结构化数据预处理。"""
+
+    return [
+        entry.record
+        for entry in preprocess_structured_records_with_lineage(records, options)
+    ]
+
+def protected_context_ranges(
+    text: str,
+    *,
+    preserve_code_blocks: bool = True,
+    preserve_tables: bool = True,
+    preserve_lists: bool = True,
+) -> tuple[tuple[int, int], ...]:
+    """返回代码块、表格和列表的不可拆分区间。
+
+    返回的偏移量基于规范化后的文本；调用方应在同一份 ``normalize_text``
+    结果上使用这些区间。
+    """
+
+    normalized = normalize_text(text)
+    return tuple(
+        _protected_markdown_ranges(
+            normalized,
+            preserve_code_blocks=preserve_code_blocks,
+            preserve_tables=preserve_tables,
+            preserve_lists=preserve_lists,
+        )
+    )
+
+def expand_to_context_boundaries(
+    text: str,
+    start: int,
+    end: int,
+    *,
+    preserve_paragraph: bool = True,
+    preserve_code_blocks: bool = True,
+    preserve_tables: bool = True,
+    preserve_lists: bool = True,
+) -> tuple[int, int]:
+    """将一个文本区间扩展到段落及受保护 Markdown 结构边界。"""
+
+    normalized = normalize_text(text)
+    if isinstance(start, bool) or isinstance(end, bool):
+        raise TypeError("start and end must be integers")
+    if not isinstance(start, int) or not isinstance(end, int):
+        raise TypeError("start and end must be integers")
+    if not 0 <= start <= end <= len(normalized):
+        raise ValueError("start and end must define a valid normalized text range")
+
+    expanded_start = start
+    expanded_end = end
+    if preserve_paragraph and normalized:
+        paragraph_start = normalized.rfind("\n\n", 0, start)
+        expanded_start = 0 if paragraph_start < 0 else paragraph_start + 2
+        paragraph_end = normalized.find("\n\n", end)
+        expanded_end = len(normalized) if paragraph_end < 0 else paragraph_end
+
+    ranges = protected_context_ranges(
+        normalized,
+        preserve_code_blocks=preserve_code_blocks,
+        preserve_tables=preserve_tables,
+        preserve_lists=preserve_lists,
+    )
+    changed = True
+    while changed:
+        changed = False
+        for range_start, range_end in ranges:
+            overlaps = range_start < expanded_end and range_end > expanded_start
+            contains_boundary = (
+                range_start <= expanded_start < range_end
+                or range_start < expanded_end <= range_end
+            )
+            if not overlaps and not contains_boundary:
+                continue
+            next_start = min(expanded_start, range_start)
+            next_end = max(expanded_end, range_end)
+            if (next_start, next_end) != (expanded_start, expanded_end):
+                expanded_start, expanded_end = next_start, next_end
+                changed = True
+    return expanded_start, expanded_end
+
+def merge_short_blocks(
+    blocks: Iterable[str],
+    *,
+    min_token_count: int = 100,
+    separator: str = "\n\n",
+) -> list[str]:
+    """按原顺序合并短内容块，并把末尾残块归入前一块。"""
+
+    if isinstance(min_token_count, bool) or not isinstance(min_token_count, int):
+        raise TypeError("min_token_count must be an integer")
+    if min_token_count <= 0:
+        raise ValueError("min_token_count must be greater than 0")
+    if not isinstance(separator, str):
+        raise TypeError("separator must be str")
+
+    merged: list[str] = []
+    pending: list[str] = []
+    pending_tokens = 0
+    for block in blocks:
+        if not isinstance(block, str):
+            raise TypeError("blocks must contain strings")
+        normalized = normalize_text(block)
+        if not normalized:
+            continue
+        token_count = estimate_token_count(normalized)
+        if not pending and token_count >= min_token_count:
+            merged.append(normalized)
+            continue
+        pending.append(normalized)
+        pending_tokens += token_count
+        if pending_tokens >= min_token_count:
+            merged.append(separator.join(pending))
+            pending = []
+            pending_tokens = 0
+
+    if pending:
+        tail = separator.join(pending)
+        if merged:
+            merged[-1] = separator.join((merged[-1], tail))
+        else:
+            merged.append(tail)
+    return merged
+
+def generate_standard_records(
+    preview_items: Iterable[Mapping[str, Any]],
+    *,
+    qa_pairs_per_item: int = 1,
+    semantic_enrichment: bool = False,
+    split: Mapping[str, int] | None = None,
+    split_seed: str = "",
+) -> list[dict[str, Any]]:
+    """把预览内容确定性转换为标准 instruction/input/output 记录。
+
+    该函数只负责本地标准化，不冒充 LLM；服务层可将其作为无模型模式或
+    LLM 响应解析后的统一落库步骤。
+    """
+
+    if not 1 <= qa_pairs_per_item <= MAX_QA_PAIRS_PER_ITEM:
+        raise ValueError(
+            f"qa_pairs_per_item must be in [1, {MAX_QA_PAIRS_PER_ITEM}]"
+        )
+    prefixes = (
+        "请结合实际情况说明：",
+        "请用通俗易懂的方式说明：",
+        "请从实际应用角度说明：",
+        "请简洁自然地说明：",
+        "请详细解答：",
+    )
+    results: list[dict[str, Any]] = []
+    for item_index, item in enumerate(preview_items):
+        content = _preview_content(item)
+        instruction, input_text, output = _standard_fields(content)
+        preview_id = str(item.get("id") or f"preview-{item_index + 1}")
+        for variant_index in range(qa_pairs_per_item):
+            variant_instruction = instruction
+            if variant_index:
+                if semantic_enrichment:
+                    prefix = prefixes[variant_index % len(prefixes)]
+                    if variant_index >= len(prefixes):
+                        prefix = (
+                            f"{prefix.removesuffix('：')}"
+                            f"（问法 {variant_index + 1}）："
+                        )
+                    variant_instruction = f"{prefix}{instruction}"
+                else:
+                    variant_instruction = f"{instruction}(问法 {variant_index + 1})"
+            raw_id = f"{preview_id}:{variant_index + 1}"
+            result_id = f"result_{hashlib.sha256(raw_id.encode('utf-8')).hexdigest()[:16]}"
+            status = "valid" if variant_instruction and output else "invalid"
+            results.append(
+                {
+                    "id": result_id,
+                    "preview_item_id": preview_id,
+                    "instruction": variant_instruction,
+                    "input": input_text,
+                    "output": output,
+                    "original_instruction": variant_instruction,
+                    "original_input": input_text,
+                    "original_output": output,
+                    "status": status,
+                    "split": "train",
+                }
+            )
+    assignments = stable_split_assignments(
+        [str(result["id"]) for result in results],
+        split,
+        seed=split_seed,
+    )
+    for result, assignment in zip(results, assignments, strict=True):
+        result["split"] = assignment
+    return results

backend/app/modules/data_process/algorithms/text_utils.py

@@ -0,0 +1,328 @@
+"""数据处理算法 - 文本处理工具。"""
+
+from __future__ import annotations
+
+import csv
+import io
+import json
+import math
+import re
+import unicodedata
+from datetime import date, datetime, time
+from decimal import Decimal
+from pathlib import Path
+from typing import Any, Literal
+from collections.abc import Mapping
+
+from .types import ParsedText, TextFormat
+
+# 格式别名映射
+_FORMAT_ALIASES: dict[str, TextFormat] = {
+    "json": "json",
+    "jsonl": "jsonl",
+    "ndjson": "jsonl",
+    "csv": "csv",
+    "tsv": "csv",
+    "md": "markdown",
+    "markdown": "markdown",
+    "txt": "txt",
+    "text": "txt",
+    "pdf": "pdf",
+    "docx": "docx",
+    "xlsx": "xlsx",
+    "pptx": "pptx",
+}
+
+# 旧版 Office 格式映射
+_LEGACY_OFFICE_FORMATS: dict[str, str] = {
+    "doc": "docx",
+    "xls": "xlsx",
+    "ppt": "pptx",
+}
+
+# Office Open XML 格式集合
+_OFFICE_OPEN_XML_FORMATS = {"docx", "xlsx", "pptx"}
+
+# 文本提取限制
+_MAX_EXTRACTED_TEXT_CHARS = 20_000_000
+
+def decode_utf8(raw: bytes | bytearray | memoryview | str) -> str:
+    """严格解码 UTF-8 文本，并移除可选 BOM。
+
+    不使用 ``errors='replace'``，避免上传内容损坏后仍被静默接收。
+    """
+
+    if isinstance(raw, str):
+        return raw.removeprefix("\ufeff")
+    if not isinstance(raw, (bytes, bytearray, memoryview)):
+        raise TypeError("raw must be bytes-like or str")
+    try:
+        return bytes(raw).decode("utf-8-sig")
+    except UnicodeDecodeError as exc:
+        raise ValueError(f"content is not valid UTF-8 at byte {exc.start}") from exc
+
+def parse_utf8_text(raw: bytes | bytearray | memoryview | str) -> str:
+    """``decode_utf8`` 的语义化别名，供上传服务直接调用。"""
+
+    return decode_utf8(raw)
+
+def normalize_text(text: str) -> str:
+    """规范 Unicode、换行和行尾空白，同时保留段落结构。"""
+
+    if not isinstance(text, str):
+        raise TypeError("text must be str")
+    normalized = unicodedata.normalize("NFKC", text.removeprefix("\ufeff"))
+    normalized = normalized.replace("\r\n", "\n").replace("\r", "\n")
+    normalized = "".join(
+        char
+        for char in normalized
+        if char in {"\n", "\t"} or not unicodedata.category(char).startswith("C")
+    )
+    lines = [re.sub(r"[\t \f\v]+$", "", line) for line in normalized.split("\n")]
+    return "\n".join(lines).strip()
+
+def _normalize_format(value: str | None) -> TextFormat | None:
+    if value is None:
+        return None
+    normalized = value.strip().lower().removeprefix(".")
+    if normalized in _LEGACY_OFFICE_FORMATS:
+        replacement = _LEGACY_OFFICE_FORMATS[normalized]
+        raise ValueError(
+            f"legacy .{normalized} format is not supported; "
+            f"convert the file to .{replacement} and upload again"
+        )
+    try:
+        return _FORMAT_ALIASES[normalized]
+    except KeyError as exc:
+        raise ValueError(f"unsupported text format: {value}") from exc
+
+def detect_text_format(
+    *,
+    filename: str | None = None,
+    text: str = "",
+    file_format: str | None = None,
+) -> TextFormat:
+    """按显式格式、扩展名和内容特征依次识别文本格式。"""
+
+    explicit = _normalize_format(file_format)
+    if explicit:
+        return explicit
+
+    if filename:
+        suffix = Path(filename).suffix.lower().removeprefix(".")
+        if suffix in _LEGACY_OFFICE_FORMATS:
+            replacement = _LEGACY_OFFICE_FORMATS[suffix]
+            raise ValueError(
+                f"legacy .{suffix} format is not supported; "
+                f"convert the file to .{replacement} and upload again"
+            )
+        detected = _FORMAT_ALIASES.get(suffix)
+        if detected:
+            return detected
+
+    stripped = text.strip()
+    if stripped:
+        if stripped[0] in "[{":
+            try:
+                json.loads(stripped)
+            except json.JSONDecodeError:
+                pass
+            else:
+                return "json"
+
+        nonempty_lines = [line for line in stripped.splitlines() if line.strip()]
+        if len(nonempty_lines) > 1:
+            try:
+                for line in nonempty_lines:
+                    json.loads(line)
+            except json.JSONDecodeError:
+                pass
+            else:
+                return "jsonl"
+
+        if re.search(r"(?m)^(?:#{1,6}\s+|```|~~~)", stripped) or re.search(
+            r"(?m)^\s*\|.+\|\s*$", stripped
+        ):
+            return "markdown"
+
+        sample = stripped[:8192]
+        try:
+            dialect = csv.Sniffer().sniff(sample, delimiters=",\t;")
+            rows = list(csv.reader(io.StringIO(sample), dialect))
+            if len(rows) >= 2 and len(rows[0]) >= 2:
+                return "csv"
+        except csv.Error:
+            pass
+
+    return "txt"
+
+def _append_bounded_text(parts: list[str], value: Any, total: int) -> int:
+    text = normalize_text(str(value or ""))
+    if not text:
+        return total
+    total += len(text)
+    if total > _MAX_EXTRACTED_TEXT_CHARS:
+        raise ValueError(
+            f"extracted document text exceeds {_MAX_EXTRACTED_TEXT_CHARS} characters"
+        )
+    parts.append(text)
+    return total
+
+def _normalize_spreadsheet_value(value: Any) -> Any:
+    if value is None:
+        return ""
+    if isinstance(value, str):
+        return normalize_text(value)
+    if isinstance(value, (datetime, date, time)):
+        return value.isoformat()
+    if isinstance(value, (bool, int, float)):
+        return value
+    return normalize_text(str(value))
+
+def _normalize_value(value: Any) -> Any:
+    if isinstance(value, str):
+        return normalize_text(value)
+    if isinstance(value, Mapping):
+        return {normalize_text(str(key)): _normalize_value(item) for key, item in value.items()}
+    if isinstance(value, list):
+        return [_normalize_value(item) for item in value]
+    return value
+
+def parse_text_content(
+    raw: bytes | bytearray | memoryview | str,
+    *,
+    filename: str | None = None,
+    file_format: str | None = None,
+) -> ParsedText:
+    """安全解析 UTF-8 文本、文本型 PDF 和现代 Office 文件。"""
+
+    # 解析器依赖 text_utils（normalize_text 等），这里延迟导入以断开循环依赖。
+    from .parsers.json_parser import _extract_structured_records_with_locators
+    from .parsers.office import _extract_docx_text, _extract_pptx_text, _extract_xlsx_records
+    from .parsers.pdf import _extract_pdf_text
+
+    detected_format = detect_text_format(
+        filename=filename,
+        text="",
+        file_format=file_format,
+    )
+    if detected_format in _OFFICE_OPEN_XML_FORMATS or detected_format == "pdf":
+        binary = _binary_bytes(raw, detected_format)
+        if detected_format == "pdf":
+            text = _extract_pdf_text(binary)
+            return ParsedText(format=detected_format, text=text, records=())
+        if detected_format == "docx":
+            text = _extract_docx_text(binary)
+            return ParsedText(format=detected_format, text=text, records=())
+        if detected_format == "pptx":
+            text = _extract_pptx_text(binary)
+            return ParsedText(format=detected_format, text=text, records=())
+
+        records, record_locators = _extract_xlsx_records(binary)
+        text = "\n".join(
+            json.dumps(record, ensure_ascii=False, separators=(",", ":"))
+            for record in records
+        )
+        return ParsedText(
+            format=detected_format,
+            text=normalize_text(text),
+            records=tuple(records),
+            record_locators=tuple(record_locators),
+        )
+
+    decoded_text = decode_utf8(raw)
+    detected_format = detect_text_format(
+        filename=filename,
+        text=decoded_text,
+        file_format=file_format,
+    )
+    # JSON/JSONL 是有损规范化的禁区：NFKC、控制字符删除或 trim 都可能改变字段值、
+    # 掩盖非法输入，甚至把原本合法的字符串变成语法错误。其他格式保持历史行为。
+    text = (
+        decoded_text
+        if detected_format in {"json", "jsonl"}
+        else normalize_text(decoded_text)
+    )
+    records: list[dict[str, Any]] = []
+    record_locators: list[dict[str, Any]] = []
+    if detected_format in {"json", "jsonl", "csv"}:
+        records, record_locators = _extract_structured_records_with_locators(
+            text,
+            detected_format,
+        )
+    return ParsedText(
+        format=detected_format,
+        text=text,
+        records=tuple(records),
+        record_locators=tuple(record_locators),
+    )
+
+def _normalize_field_name(value: Any, style: str) -> str:
+    name = normalize_text(str(value))
+    if style == "preserve":
+        return name
+    if style == "lower":
+        return name.lower()
+    if style != "snake_case":
+        raise ValueError("field_name_style must be snake_case, lower or preserve")
+    name = re.sub(r"(?<=[a-z0-9])(?=[A-Z])", "_", name)
+    name = re.sub(r"[^\w\u3400-\u4dbf\u4e00-\u9fff]+", "_", name, flags=re.UNICODE)
+    return re.sub(r"_+", "_", name).strip("_").lower()
+
+def structured_json_dumps(value: Any, *, sort_keys: bool = False) -> str:
+    """序列化紧凑 JSON，并把 ``Decimal`` 保持为原值对应的 JSON 数字。
+
+    标准库会要求先把 ``Decimal`` 转成 float 或字符串；前者可能静默舍入，
+    后者会改变 JSON 类型。这里直接输出有限 Decimal 的十进制表示。
+    """
+
+    def serialize(item: Any) -> str:
+        if item is None:
+            return "null"
+        if item is True:
+            return "true"
+        if item is False:
+            return "false"
+        if isinstance(item, int):
+            return str(item)
+        if isinstance(item, Decimal):
+            if not item.is_finite():
+                raise ValueError("non-finite JSON number is not allowed")
+            return str(item)
+        if isinstance(item, float):
+            if not math.isfinite(item):
+                raise ValueError("non-finite JSON number is not allowed")
+            return json.dumps(item, allow_nan=False)
+        if isinstance(item, str):
+            return json.dumps(item, ensure_ascii=False)
+        if isinstance(item, Mapping):
+            pairs: list[tuple[str, Any]] = []
+            seen_keys: set[str] = set()
+            for key, child in item.items():
+                if not isinstance(key, str):
+                    raise TypeError("JSON object keys must be strings")
+                if key in seen_keys:
+                    raise ValueError(f"duplicate JSON object key: {key!r}")
+                seen_keys.add(key)
+                pairs.append((key, child))
+            if sort_keys:
+                pairs.sort(key=lambda pair: pair[0])
+            return "{" + ",".join(
+                f"{json.dumps(key, ensure_ascii=False)}:{serialize(child)}"
+                for key, child in pairs
+            ) + "}"
+        if isinstance(item, (list, tuple)):
+            return "[" + ",".join(serialize(child) for child in item) + "]"
+        raise TypeError(f"value of type {type(item).__name__} is not JSON serializable")
+
+    return serialize(value)
+
+def _binary_bytes(
+    raw: bytes | bytearray | memoryview | str,
+    file_format: TextFormat,
+) -> bytes:
+    if isinstance(raw, str):
+        raise ValueError(f"{file_format.upper()} content must be uploaded as binary data")
+    if not isinstance(raw, (bytes, bytearray, memoryview)):
+        raise TypeError("raw must be bytes-like or str")
+    return bytes(raw)

backend/app/modules/data_process/algorithms/transforms.py

@@ -0,0 +1,81 @@
+"""数据处理算法 - 数据集转换和分割。"""
+
+from __future__ import annotations
+
+import hashlib
+import math
+from collections.abc import Mapping, Sequence
+
+from .types import DatasetSplit
+
+
+def stable_split(
+    value: str | int,
+    split: Mapping[str, int] | None = None,
+    *,
+    seed: str = "",
+) -> DatasetSplit:
+    """按稳定哈希将记录划分到 train/validation/test。"""
+
+    ratios = dict(split or {"train": 80, "validation": 10, "test": 10})
+    required = {"train", "validation", "test"}
+    if set(ratios) != required:
+        raise ValueError("split must contain exactly train, validation and test")
+    if any(isinstance(value, bool) or not isinstance(value, int) or value < 0 for value in ratios.values()):
+        raise ValueError("split ratios must be non-negative integers")
+    if sum(ratios.values()) != 100:
+        raise ValueError("split ratios must sum to 100")
+
+    digest = hashlib.sha256(f"{seed}:{value}".encode("utf-8")).digest()
+    bucket = int.from_bytes(digest[:8], "big") % 10_000
+    train_boundary = ratios["train"] * 100
+    validation_boundary = train_boundary + ratios["validation"] * 100
+    if bucket < train_boundary:
+        return "train"
+    if bucket < validation_boundary:
+        return "validation"
+    return "test"
+
+def stable_split_assignments(
+    values: Sequence[str | int],
+    split: Mapping[str, int] | None = None,
+    *,
+    seed: str = "",
+) -> list[DatasetSplit]:
+    """按稳定顺序和精确配额批量划分数据集。
+
+    单条哈希分桶只能在大样本下近似比例。这里先按哈希稳定排序，再用
+    最大余数法计算各切分配额，确保小数据集也严格遵循配置比例。
+    """
+
+    ratios = dict(split or {"train": 80, "validation": 10, "test": 10})
+    # 复用单条划分的参数校验，避免两套规则逐渐漂移。
+    stable_split("validation", ratios, seed=seed)
+    if not values:
+        return []
+
+    split_order: tuple[DatasetSplit, ...] = ("train", "validation", "test")
+    exact = {name: len(values) * ratios[name] / 100 for name in split_order}
+    quotas = {name: math.floor(exact[name]) for name in split_order}
+    remaining = len(values) - sum(quotas.values())
+    remainder_order = sorted(
+        split_order,
+        key=lambda name: (-(exact[name] - quotas[name]), split_order.index(name)),
+    )
+    for name in remainder_order[:remaining]:
+        quotas[name] += 1
+
+    ranked_indices = sorted(
+        range(len(values)),
+        key=lambda index: (
+            hashlib.sha256(f"{seed}:{values[index]}".encode("utf-8")).digest(),
+            index,
+        ),
+    )
+    assignments: list[DatasetSplit] = ["train"] * len(values)
+    cursor = 0
+    for name in split_order:
+        for index in ranked_indices[cursor : cursor + quotas[name]]:
+            assignments[index] = name
+        cursor += quotas[name]
+    return assignments

backend/app/modules/data_process/algorithms/types.py

@@ -0,0 +1,266 @@
+"""数据处理模块使用的无副作用算法。
+
+本模块不访问数据库、文件系统或网络，便于 API、后台任务和测试共同复用。
+所有偏移量均为 Python 字符串偏移量。
+"""
+
+from __future__ import annotations
+
+import csv
+import hashlib
+import io
+import json
+import math
+import re
+import unicodedata
+import xml.etree.ElementTree as ET
+import zipfile
+from collections import Counter
+from collections.abc import Iterable, Mapping, Sequence
+from copy import deepcopy
+from dataclasses import dataclass
+from datetime import date, datetime, time
+from decimal import Decimal
+from pathlib import Path, PurePosixPath
+from typing import Any, Literal
+from urllib.parse import unquote, urlsplit
+
+from docx import Document
+from docx.oxml.table import CT_Tbl
+from docx.oxml.text.paragraph import CT_P
+from docx.table import Table
+from docx.text.paragraph import Paragraph
+from openpyxl import load_workbook
+from openpyxl.utils.cell import range_boundaries
+from pptx import Presentation
+from pypdf import PdfReader
+
+# 避免循环导入：直接定义常量而不是从 constants 导入
+MAX_QA_PAIRS_PER_ITEM = 50
+
+TextFormat = Literal[
+    "json",
+    "jsonl",
+    "csv",
+    "markdown",
+    "txt",
+    "pdf",
+    "docx",
+    "xlsx",
+    "pptx",
+]
+DatasetSplit = Literal["train", "validation", "test"]
+StructuredPreprocessOption = Literal[
+    "clean_invalid",
+    "detect_structure",
+    "deduplicate",
+    "normalize_format",
+    "filter_anomaly",
+    "desensitize",
+]
+
+SUPPORTED_TEXT_FORMATS: tuple[TextFormat, ...] = (
+    "json",
+    "jsonl",
+    "csv",
+    "markdown",
+    "txt",
+    "pdf",
+    "docx",
+    "xlsx",
+    "pptx",
+)
+
+_FORMAT_ALIASES: dict[str, TextFormat] = {
+    "json": "json",
+    "jsonl": "jsonl",
+    "ndjson": "jsonl",
+    "csv": "csv",
+    "tsv": "csv",
+    "md": "markdown",
+    "markdown": "markdown",
+    "txt": "txt",
+    "text": "txt",
+    "pdf": "pdf",
+    "docx": "docx",
+    "xlsx": "xlsx",
+    "pptx": "pptx",
+}
+_LEGACY_OFFICE_FORMATS: dict[str, str] = {
+    "doc": "docx",
+    "xls": "xlsx",
+    "ppt": "pptx",
+}
+_OFFICE_OPEN_XML_FORMATS = {"docx", "xlsx", "pptx"}
+_MAX_ARCHIVE_ENTRIES = 10_000
+_MAX_ARCHIVE_UNCOMPRESSED_BYTES = 512 * 1024 * 1024
+_MAX_ARCHIVE_ENTRY_BYTES = 128 * 1024 * 1024
+_MAX_ARCHIVE_COMPRESSION_RATIO = 200
+_MAX_EXTRACTED_TEXT_CHARS = 20_000_000
+_MAX_PDF_PAGES = 2_000
+_MAX_PRESENTATION_SLIDES = 2_000
+_MAX_WORKBOOK_SHEETS = 100
+_MAX_WORKBOOK_ROWS = 100_000
+_MAX_WORKBOOK_SCANNED_ROWS = 200_000
+_MAX_WORKBOOK_COLUMNS = 256
+_MAX_WORKBOOK_CELLS = 2_000_000
+_MAX_WORKBOOK_HEADER_ROWS = 8
+_MAX_WORKBOOK_HEADER_SCAN_ROWS = 64
+_MAX_WORKBOOK_MERGED_RANGES = 100_000
+_MAX_STRUCTURED_FIELDS = 1_024
+_MAX_STRUCTURED_DEPTH = 16
+_MAX_JSON_DEPTH = 64
+_MAX_ANOMALY_TEXT_CHARS = 1_000_000
+_STRUCTURED_OPTIONS = {
+    "clean_invalid",
+    "detect_structure",
+    "deduplicate",
+    "normalize_format",
+    "filter_anomaly",
+    "desensitize",
+}
+_IDENTITY_FIELD_PATTERN = re.compile(r"(?:^|[._])(?:id|uuid|key|code)$|(?:^|[._]).+_id$")
+_MOJIBAKE_MARKERS = ("\ufffd", "锟斤拷", "烫烫烫", "屯屯屯", "Ã", "Â", "â€")
+_JSON_RECORD_ARRAY_KEYS = ("records", "data", "items", "rows")
+_JSON_ENVELOPE_KEYS = ("response", "payload")
+_JSON_WRAPPER_METADATA_KEYS = frozenset(
+    {
+        "page",
+        "page_size",
+        "pageSize",
+        "per_page",
+        "perPage",
+        "total",
+        "total_count",
+        "totalCount",
+        "count",
+        "offset",
+        "limit",
+        "cursor",
+        "next_cursor",
+        "nextCursor",
+        "has_more",
+        "hasMore",
+    }
+)
+_JSON_RESPONSE_METADATA_KEYS = _JSON_WRAPPER_METADATA_KEYS | {
+    "success",
+    "status",
+    "code",
+    "message",
+    "error",
+}
+_NAME_FIELD_NAMES = {
+    "name",
+    "full_name",
+    "fullname",
+    "real_name",
+    "contact_name",
+    "customer_name",
+    "recipient_name",
+    "姓名",
+    "中文姓名",
+    "真实姓名",
+    "联系人",
+    "联系人姓名",
+    "客户姓名",
+    "收件人",
+    "收件人姓名",
+}
+
+_EMAIL_PATTERN = re.compile(
+    r"(?<![\w.+-])[A-Za-z0-9.!#$%&'*+/=?^_`{|}~-]+"
+    r"@[A-Za-z0-9](?:[A-Za-z0-9-]{0,61}[A-Za-z0-9])?"
+    r"(?:\.[A-Za-z0-9](?:[A-Za-z0-9-]{0,61}[A-Za-z0-9])?)+(?![\w.-])"
+)
+_PHONE_PATTERN = re.compile(r"(?<!\d)(?:(?:\+|00)?86[-\s]?)?1[3-9]\d{9}(?!\d)")
+_ID_CARD_PATTERN = re.compile(r"(?<!\d)(?:\d{17}[\dXx]|\d{15})(?!\d)")
+_CHINESE_NAME_CONTEXT_PATTERN = re.compile(
+    r"(?P<label>姓名|真实姓名|联系人(?:姓名)?|收件人)"
+    r"(?P<separator>\s*(?:[:：=]|为)\s*|\s+)"
+    r"(?P<name>[\u3400-\u4dbf\u4e00-\u9fff·]{2,8})"
+)
+_ENGLISH_NAME_CONTEXT_PATTERN = re.compile(
+    r"(?im)(?P<label>full\s+name|contact\s+name|name)"
+    r"(?P<separator>\s*[:=]\s*)"
+    r"(?P<name>[A-Za-z][A-Za-z'’-]*(?:[ \t]+[A-Za-z][A-Za-z'’-]*){0,3})"
+)
+_TOKEN_PATTERN = re.compile(r"[\u3400-\u4dbf\u4e00-\u9fff]|[A-Za-z0-9_]+|[^\s]")
+
+
+@dataclass(frozen=True, slots=True)
+class ParsedText:
+    """文本、文档或工作簿的统一解析结果。"""
+
+    format: TextFormat
+    text: str
+    records: tuple[dict[str, Any], ...]
+    record_locators: tuple[dict[str, Any], ...] = ()
+
+@dataclass(frozen=True, slots=True)
+class ProcessedStructuredRecord:
+    """保留原始记录索引的结构化预处理结果。"""
+
+    source_index: int
+    record: dict[str, Any]
+
+@dataclass(frozen=True, slots=True)
+class PdfPageText:
+    """PDF 物理页在统一提取文本中的字符范围。"""
+
+    page_number: int
+    text: str
+    source_start: int
+    source_end: int
+
+@dataclass(frozen=True, slots=True)
+class DocumentNoiseSpan:
+    """PDF 中可安全从展示内容移除的文本范围。"""
+
+    start: int
+    end: int
+    kind: Literal["page_number", "repeated_margin", "table_of_contents"]
+
+@dataclass(frozen=True, slots=True)
+class QualityScore:
+    """标准 instruction/input/output 记录的可解释质量分。"""
+
+    overall: float
+    completeness: float
+    length: float
+    readability: float
+    relevance: float
+    duplicate: float
+    is_valid: bool
+    flags: tuple[str, ...]
+    fingerprint: str
+
+@dataclass(frozen=True, slots=True)
+class DocumentHeading:
+    """文档标题的位置和层级。"""
+
+    level: int
+    title: str
+    line_number: int
+    start: int
+    end: int
+
+@dataclass(frozen=True, slots=True)
+class DocumentStructure:
+    """无需外部模型即可复现的文档结构摘要。"""
+
+    line_count: int
+    paragraph_count: int
+    headings: tuple[DocumentHeading, ...]
+    code_block_count: int
+    table_block_count: int
+    list_block_count: int
+
+@dataclass(frozen=True, slots=True)
+class _PdfLine:
+    text: str
+    start: int
+    end: int
+
+class _DuplicateJsonKeyError(ValueError):
+    """严格 JSON 解析时发现同一对象内的重复键。"""

backend/app/modules/data_process/dataset_format.py

@@ -26,6 +26,16 @@ def _load_sample(path: str | None, content: str | None = None, max_samples: int
     if not text:
         return []
 
+    # 先按整文件 JSON（数组/单对象）解析，兼容 .json；失败再按 jsonl 逐行解析
+    try:
+        value = json.loads(text)
+    except json.JSONDecodeError:
+        value = None
+    if isinstance(value, list):
+        return [item for item in value[:max_samples] if isinstance(item, dict)]
+    if isinstance(value, dict):
+        return [value]
+
     lines = text.splitlines()[:max_samples]
     records: list[dict[str, Any]] = []
     for line in lines:

backend/app/modules/data_process/document_chunking.py

@@ -57,7 +57,64 @@ def _sentence_chunks(text: str) -> list[str]:
 
 @lru_cache(maxsize=1)
 def _tokenizer() -> tiktoken.Encoding:
-    return tiktoken.get_encoding("cl100k_base")
+    """加载 cl100k_base 编码器，优先在线下载，失败时使用本地缓存以支持离线环境。"""
+    import os
+    import base64
+
+    # 先设置缓存目录环境变量
+    offline_cache = os.path.expanduser("~/.cache/tiktoken")
+    os.environ.setdefault("TIKTOKEN_CACHE_DIR", offline_cache)
+
+    try:
+        # 尝试标准方式加载
+        return tiktoken.get_encoding("cl100k_base")
+    except Exception:
+        # 如果失败，尝试手动从本地文件构造
+        try:
+            from pathlib import Path
+
+            local_file = Path(offline_cache) / "9b5ad71b2ce5302211f9c61530b329a4922fc6a4"
+            if not local_file.exists():
+                # 尝试另一个可能的文件名
+                local_file = Path(offline_cache) / "cl100k_base.tiktoken"
+
+            if local_file.exists():
+                # 读取 BPE 文件内容
+                with open(local_file, "rb") as f:
+                    contents = f.read()
+
+                # 解析 BPE 文件
+                mergeable_ranks = {}
+                for line in contents.splitlines():
+                    if line:
+                        token, rank = line.split()
+                        mergeable_ranks[base64.b64decode(token)] = int(rank)
+
+                # 构造 Encoding 对象（模块顶部已 import tiktoken，
+                # 此处不能再 import tiktoken.core，否则会把 tiktoken
+                # 变成局部变量，使函数开头的 tiktoken.get_encoding 抛
+                # UnboundLocalError）
+                return tiktoken.core.Encoding(
+                    name="cl100k_base",
+                    pat_str=r"""'(?i:[sdmt]|ll|ve|re)|[^\r\n\p{L}\p{N}]?+\p{L}+|\p{N}{1,3}| ?[^\s\p{L}\p{N}]++[\r\n]*|\s*[\r\n]|\s+(?!\S)|\s+""",
+                    mergeable_ranks=mergeable_ranks,
+                    special_tokens={
+                        "<|endoftext|>": 100257,
+                        "<|fim_prefix|>": 100258,
+                        "<|fim_middle|>": 100259,
+                        "<|fim_suffix|>": 100260,
+                        "<|endofprompt|>": 100276,
+                    },
+                )
+        except Exception:
+            pass
+
+        raise RuntimeError(
+            f"无法加载 cl100k_base 编码器\n"
+            f"请确保以下任一条件满足:\n"
+            f"1. 服务器可以访问网络\n"
+            f"2. 本地存在缓存文件: {offline_cache}/9b5ad71b2ce5302211f9c61530b329a4922fc6a4"
+        )
 
 
 def _text_chunks(
@@ -205,9 +262,17 @@ def _make_text_chunk(source: str, start: int, end: int) -> DocumentChunk:
 
 @lru_cache(maxsize=1)
 def _document_converter():
-    from docling.document_converter import DocumentConverter
+    from docling.datamodel.base_models import InputFormat
+    from docling.datamodel.pipeline_options import PdfPipelineOptions
+    from docling.document_converter import DocumentConverter, PdfFormatOption
 
-    return DocumentConverter()
+    pipeline_options = PdfPipelineOptions()
+    pipeline_options.do_ocr = False
+    return DocumentConverter(
+        format_options={
+            InputFormat.PDF: PdfFormatOption(pipeline_options=pipeline_options),
+        }
+    )
 
 
 class _MarkdownSerializerProvider(ChunkingSerializerProvider):

backend/app/modules/data_process/generation.py

@@ -29,7 +29,12 @@ class _TerminalModelGenerationError(ModelGenerationError):
 
 OUTPUT_TYPE_STANDARD = "standard"
 OUTPUT_TYPE_REASONING = "reasoning"
-SUPPORTED_OUTPUT_TYPES = {OUTPUT_TYPE_STANDARD, OUTPUT_TYPE_REASONING}
+OUTPUT_TYPE_DPO = "dpo"
+SUPPORTED_OUTPUT_TYPES = {
+    OUTPUT_TYPE_STANDARD,
+    OUTPUT_TYPE_REASONING,
+    OUTPUT_TYPE_DPO,
+}
 REASONING_DETAIL_NORMAL = "normal"
 REASONING_DETAIL_DETAILED = "detailed"
 SUPPORTED_REASONING_DETAILS = {
@@ -40,6 +45,25 @@ MINIMAX_M3_API_HOSTS = {"api.minimax.io", "api.minimaxi.com"}
 MINIMAX_M3_MIN_COMPLETION_TOKENS = 4096
 logger = logging.getLogger(__name__)
 
+# 问题表述风格规则：防止模型产出“请描述/请说明”式模板化问句。
+_QUESTION_STYLE_RULE = (
+    "各条问题必须覆盖不同的信息点并使用不同的句式，只替换关键词套用同一句式视为重复。"
+    "问题表述要像真实用户自然提出的问题：具体、口语化、直奔信息点，"
+    "避免“请描述”“请说明”“根据文档”等模板化开头，"
+    "也不要把原文句子直接改成问句；多条问题时交替使用直接疑问、场景式提问、追问式等句式。"
+    "表述示例（仅示意风格，不要照搬内容）："
+    "避免——“请描述系统的权限控制机制”；"
+    "推荐——“不同角色能看到的菜单不一样，平台是怎么控制的？”"
+)
+
+# 任务配置未提供提示语时的兜底，与前端内置默认提示语保持同等信息量。
+_DEFAULT_GENERATION_PROMPT = (
+    "你是一名专业的数据生成专家。请基于来源内容生成高质量、"
+    "可直接用于监督微调的问答数据：问题聚焦核心信息点、"
+    "表述像真实用户自然提出的问题，具体、口语化，多条问题使用不同句式；"
+    "答案严格依据来源内容，准确、完整、语言自然，不引入来源之外的信息。"
+)
+
 
 def _is_retryable_generation_error(exc: Exception) -> bool:
     if isinstance(exc, _TerminalModelGenerationError):
@@ -285,6 +309,18 @@ def _prompt_messages(
             "这是思维链输出模式，即使其他提示语要求省略分析，也不得省略 reasoning。"
             "不要自行添加 <think> 标签，系统会在保存时统一组装。"
         )
+    elif output_type == OUTPUT_TYPE_DPO:
+        schema = (
+            '{"items":[{"instruction":"...","input":"...",'
+            '"chosen":"...","rejected":"..."}]}'
+        )
+        output_rule = (
+            "你正在生成用于直接偏好优化（DPO）的成对偏好数据。"
+            "instruction、chosen 和 rejected 均不得为空；chosen 必须是忠于来源、"
+            "准确完整的优选回答，rejected 必须是表面合理但存在明确质量缺陷的拒选回答。"
+            "两者不得相同；rejected 不得包含违法危险内容，也不得用空白、乱码或无关文本凑数。"
+            "不要输出分析过程或 <think> 标签。"
+        )
     else:
         schema = '{"items":[{"instruction":"...","input":"...","output":"..."}]}'
         output_rule = (
@@ -293,11 +329,11 @@ def _prompt_messages(
         )
     schema_instruction = (
         f"必须只返回 JSON 对象，格式为 {schema}；items 必须包含 {count} 条。"
-        f"这是总计 {total_count} 条中的第 {start_index}-{end_index} 条，"
+        f"这是总计 {total_count} 条中的第 {start_index}-{end_index} 条。"
-        "各条必须使用不同的提问角度和表述，避免重复。"
+        f"{_QUESTION_STYLE_RULE}{output_rule}"
-        f"{output_rule}不要输出 Markdown 代码围栏或 JSON 之外的说明。"
+        "不要输出 Markdown 代码围栏或 JSON 之外的说明。"
     )
-    base_prompt = normalize_text(prompt) or "请根据来源内容生成可用于监督微调的问答数据。"
+    base_prompt = normalize_text(prompt) or _DEFAULT_GENERATION_PROMPT
     if "{{ content }}" in base_prompt:
         user_prompt = base_prompt.replace("{{ content }}", content)
         return [
@@ -461,9 +497,13 @@ def generate_model_records(
                             "instruction": failure_instruction,
                             "input": content,
                             "output": "",
+                            "chosen": "",
+                            "rejected": "",
                             "original_instruction": failure_instruction,
                             "original_input": content,
                             "original_output": "",
+                            "original_chosen": "",
+                            "original_rejected": "",
                             "status": "invalid",
                             "error": error_message,
                             "split": "train",
@@ -479,6 +519,8 @@ def generate_model_records(
                     input_text = normalize_text(
                         str(value.get("input") or value.get("context") or "")
                     )
+                    chosen = ""
+                    rejected = ""
                     if output_type == OUTPUT_TYPE_REASONING:
                         reasoning = normalize_text(
                             re.sub(
@@ -508,6 +550,36 @@ def generate_model_records(
                         )
                         valid = bool(instruction and reasoning and answer)
                         missing_error = "model result is missing instruction, reasoning or answer"
+                    elif output_type == OUTPUT_TYPE_DPO:
+                        chosen = normalize_text(str(value.get("chosen") or ""))
+                        rejected = normalize_text(str(value.get("rejected") or ""))
+                        chosen = normalize_text(
+                            re.sub(
+                                r"<think>[\s\S]*?(?:</think>|$)",
+                                "",
+                                chosen,
+                                flags=re.IGNORECASE,
+                            )
+                        )
+                        rejected = normalize_text(
+                            re.sub(
+                                r"<think>[\s\S]*?(?:</think>|$)",
+                                "",
+                                rejected,
+                                flags=re.IGNORECASE,
+                            )
+                        )
+                        output = chosen
+                        valid = bool(
+                            instruction
+                            and chosen
+                            and rejected
+                            and chosen.strip() != rejected.strip()
+                        )
+                        missing_error = (
+                            "model result is missing instruction, chosen or rejected, "
+                            "or chosen equals rejected"
+                        )
                     else:
                         output = normalize_text(
                             str(
@@ -527,7 +599,10 @@ def generate_model_records(
                         )
                         valid = bool(instruction and output)
                         missing_error = "model result is missing instruction or output"
-                    raw_id = f"{preview_id}:{variant_index + 1}:{instruction}:{output}"
+                    raw_id = (
+                        f"{preview_id}:{variant_index + 1}:{instruction}:"
+                        f"{output}:{rejected}"
+                    )
                     result_id = f"result_{hashlib.sha256(raw_id.encode()).hexdigest()[:16]}"
                     results.append(
                         {
@@ -536,9 +611,13 @@ def generate_model_records(
                             "instruction": instruction,
                             "input": input_text,
                             "output": output,
+                            "chosen": chosen,
+                            "rejected": rejected,
                             "original_instruction": instruction,
                             "original_input": input_text,
                             "original_output": output,
+                            "original_chosen": chosen,
+                            "original_rejected": rejected,
                             "status": "valid" if valid else "invalid",
                             "error": (None if valid else missing_error),
                             "split": "train",

backend/app/modules/data_process/storage.py

@@ -28,6 +28,9 @@ class StagedSourceObject:
 
 
 def _default_storage_root() -> Path:
+    data_root = os.getenv("YG_FT_DATA_ROOT", "").strip()
+    if data_root:
+        return Path(data_root).expanduser() / "data-process"
     return Path(__file__).resolve().parents[3] / "storage" / "data-process"
 
 
@@ -531,6 +534,10 @@ class LocalDataProcessStorage:
 
     @staticmethod
     def _fsync_directory(directory: Path) -> None:
+        # Windows 不支持以 O_RDONLY 打开目录做 fsync，跳过即可。
+        # 数据完整性在 Linux 生产环境保障，Windows 开发环境忽略。
+        if os.name == "nt":
+            return
         descriptor = os.open(directory, os.O_RDONLY)
         try:
             os.fsync(descriptor)

backend/app/modules/data_process/store/__init__.py

@@ -0,0 +1,69 @@
+"""数据处理存储层。"""
+
+from .base import (
+    StoreBase,
+    DataProcessStoreError,
+    NotFoundError,
+    ConflictError,
+    InvalidStateError,
+    utcnow,
+    new_id,
+    repeat_task_id,
+    json_dumps,
+    TASK_STATUSES,
+    EDITABLE_STATUSES,
+    ACTIVE_PREVIEW_STATUSES,
+    WORKFLOW_STEPS,
+    _decode_row,
+    _preview_config_changed,
+    _reasoning_output_is_valid,
+    _source_storage_descriptor,
+)
+from .tasks import TasksMixin
+from .source_files import SourceFilesMixin
+from .preview import PreviewMixin
+from .generation import GenerationMixin
+from .results import ResultsMixin
+from .datasets import DatasetsMixin
+
+
+class DataProcessStore(
+    StoreBase,
+    TasksMixin,
+    SourceFilesMixin,
+    PreviewMixin,
+    GenerationMixin,
+    ResultsMixin,
+    DatasetsMixin,
+):
+    """数据处理持久层。
+
+    构造函数不会连接数据库或执行迁移。部署方必须显式执行 002 SQL，
+    或在受控的管理命令中调用 :meth:`ensure_schema`，避免应用启动时
+    修改远程数据库。
+    """
+    pass
+
+
+def get_data_process_store() -> DataProcessStore:
+    """获取数据处理存储实例。"""
+    return DataProcessStore()
+
+
+__all__ = [
+    "DataProcessStore",
+    "DataProcessStoreError",
+    "NotFoundError",
+    "ConflictError",
+    "InvalidStateError",
+    "get_data_process_store",
+    "utcnow",
+    "new_id",
+    "repeat_task_id",
+    "TASK_STATUSES",
+    "EDITABLE_STATUSES",
+    "_decode_row",
+    "_preview_config_changed",
+    "_reasoning_output_is_valid",
+    "_source_storage_descriptor",
+]

backend/app/modules/data_process/store/base.py

@@ -0,0 +1,305 @@
+"""数据处理存储层 - 基础设施。
+
+包含：异常类、工具函数、常量定义、基类。
+"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import re
+import uuid
+from collections.abc import Iterator
+from contextlib import contextmanager
+from datetime import UTC, date, datetime
+from decimal import Decimal
+from pathlib import Path
+from typing import Any
+
+import psycopg
+from psycopg.rows import dict_row
+
+from app.core.config import get_settings
+
+# 常量定义
+TASK_STATUSES = {"pending", "running", "completed", "failed", "stopped"}
+EDITABLE_STATUSES = {"pending", "failed", "stopped", "completed"}
+ACTIVE_PREVIEW_STATUSES = {"queued", "running"}
+WORKFLOW_STEPS = {"create", "model", "upload", "preview", "generate", "results"}
+
+_PREVIEW_CONFIG_ALIASES = {
+    "preprocess_options": "preprocessOptions",
+    "chunk_method": "chunkMethod",
+    "chunk_size": "chunkSize",
+    "chunk_overlap": "chunkOverlap",
+    "min_chunk_size": "minChunkSize",
+    "semantic_breakpoint_percentile": "semanticBreakpointPercentile",
+    "preserve_tables": "preserveTables",
+    "preserve_code_blocks": "preserveCodeBlocks",
+    "preserve_lists": "preserveLists",
+}
+
+_UNSTRUCTURED_PREVIEW_DEFAULTS: dict[str, Any] = {
+    "chunk_method": "layout_hybrid",
+    "chunk_size": 800,
+    "chunk_overlap": 100,
+    "min_chunk_size": 100,
+    "semantic_breakpoint_percentile": 95,
+    "preserve_tables": True,
+    "preserve_code_blocks": True,
+    "preserve_lists": True,
+}
+
+_REGENERATION_MARKER_KEY = "_regeneration_prepared"
+_REPEAT_SOURCE_TASK_KEY = "_repeat_source_task_id"
+_REPEAT_REQUEST_KEY = "_repeat_request_id"
+_INTERNAL_CONFIG_KEYS = {
+    _REGENERATION_MARKER_KEY,
+    _REPEAT_SOURCE_TASK_KEY,
+    _REPEAT_REQUEST_KEY,
+}
+
+
+# 异常类
+class DataProcessStoreError(RuntimeError):
+    pass
+
+
+class NotFoundError(DataProcessStoreError):
+    pass
+
+
+class ConflictError(DataProcessStoreError):
+    pass
+
+
+class InvalidStateError(DataProcessStoreError):
+    pass
+
+
+# 工具函数
+def utcnow() -> str:
+    return datetime.now(UTC).replace(microsecond=0).isoformat().replace("+00:00", "Z")
+
+
+def new_id(prefix: str) -> str:
+    return f"{prefix}_{uuid.uuid4().hex[:20]}"
+
+
+def repeat_task_id(source_task_id: str, request_id: str) -> str:
+    """按源任务和请求幂等键生成稳定的新任务 ID。"""
+    digest = hashlib.sha256(f"{source_task_id}:{request_id}".encode()).hexdigest()
+    return f"dpt_{digest[:20]}"
+
+
+def json_dumps(value: Any) -> str:
+    return json.dumps(value, ensure_ascii=False, separators=(",", ":"))
+
+
+def _database_url(value: str) -> str:
+    return value.replace("postgresql+psycopg://", "postgresql://")
+
+
+def _json_value(value: Any, default: Any) -> Any:
+    if value is None or value == "":
+        return default
+    if isinstance(value, (dict, list)):
+        return value
+    try:
+        return json.loads(value)
+    except (TypeError, json.JSONDecodeError):
+        return default
+
+
+def _task_output_type(task: dict[str, Any]) -> str:
+    config = _json_value(task.get("config"), {})
+    if not isinstance(config, dict):
+        return "standard"
+    return str(config.get("output_type") or config.get("outputType") or "standard")
+
+
+def _task_reasoning_detail(task: dict[str, Any]) -> str:
+    config = _json_value(task.get("config"), {})
+    if not isinstance(config, dict):
+        return "normal"
+    return str(
+        config.get("reasoning_detail")
+        or config.get("reasoningDetail")
+        or "normal"
+    )
+
+
+def _reasoning_output_is_valid(value: Any) -> bool:
+    match = re.fullmatch(
+        r"\s*<think>\s*(?P<reasoning>[\s\S]*?)\s*</think>\s*(?P<answer>[\s\S]+?)\s*",
+        str(value or ""),
+        flags=re.IGNORECASE,
+    )
+    return bool(
+        match
+        and match.group("reasoning").strip()
+        and match.group("answer").strip()
+        and all(
+            tag not in part.lower()
+            for tag in ("<think", "</think")
+            for part in (match.group("reasoning"), match.group("answer"))
+        )
+    )
+
+
+def _dpo_fields_are_valid(row: dict[str, Any]) -> bool:
+    chosen = str(row.get("chosen") or "").strip()
+    rejected = str(row.get("rejected") or "").strip()
+    return bool(chosen and rejected and chosen != rejected)
+
+
+def _preview_config_value(config: dict[str, Any], key: str, default: Any) -> Any:
+    if key in config:
+        return config[key]
+    return config.get(_PREVIEW_CONFIG_ALIASES[key], default)
+
+
+def _normalized_preprocess_options(config: dict[str, Any]) -> Any:
+    value = _preview_config_value(config, "preprocess_options", [])
+    if isinstance(value, (list, tuple, set)):
+        return tuple(sorted({str(item) for item in value}))
+    return value
+
+
+def _preview_config_projection(
+    process_type: str,
+    config: dict[str, Any],
+) -> dict[str, Any]:
+    """只投影会改变预览切片的配置。
+
+    生成模型、提示词、温度等参数不影响源文切片，因此不应该
+    破坏用户已经校对过的预览内容。
+    """
+    projection: dict[str, Any] = {
+        "preprocess_options": _normalized_preprocess_options(config),
+    }
+    if process_type != "unstructured":
+        return projection
+    for key, default in _UNSTRUCTURED_PREVIEW_DEFAULTS.items():
+        projection[key] = _preview_config_value(config, key, default)
+    return projection
+
+
+def _preview_config_changed(
+    process_type: str,
+    current_config: dict[str, Any],
+    next_config: dict[str, Any],
+) -> bool:
+    return _preview_config_projection(process_type, current_config) != _preview_config_projection(
+        process_type, next_config
+    )
+
+
+def _regeneration_marker(task: dict[str, Any]) -> dict[str, Any] | None:
+    config = task.get("config")
+    if not isinstance(config, dict):
+        return None
+    marker = config.get(_REGENERATION_MARKER_KEY)
+    if not isinstance(marker, dict) or marker.get("prepared") is not True:
+        return None
+    return marker
+
+
+def _is_regeneration_prepared(task: dict[str, Any]) -> bool:
+    return _regeneration_marker(task) is not None
+
+
+def _business_config(config: dict[str, Any] | None) -> dict[str, Any]:
+    """过滤只供服务端维护的工作流标记。"""
+    return {
+        key: value
+        for key, value in (config or {}).items()
+        if key not in _INTERNAL_CONFIG_KEYS
+    }
+
+
+def _public_task(item: dict[str, Any] | None) -> dict[str, Any] | None:
+    """从 API 任务快照中移除服务端内部工作流标记。"""
+    if item is None:
+        return None
+    public = dict(item)
+    config = public.get("config")
+    if isinstance(config, dict):
+        public["config"] = _business_config(config)
+    return public
+
+
+def _serialize_value(value: Any) -> Any:
+    if isinstance(value, (datetime, date)):
+        return value.isoformat().replace("+00:00", "Z")
+    if isinstance(value, Decimal):
+        return float(value)
+    return value
+
+
+def _source_storage_descriptor(
+    payload: dict[str, Any],
+    task_id: str,
+    file_id: str,
+) -> tuple[str, dict[str, Any]]:
+    storage_object_id = str(
+        payload.get("storage_object_id")
+        or f"db://data-process/{task_id}/{file_id}/v1"
+    )
+    expected_local_prefix = f"local://data-process/{task_id}/{file_id}/v1/"
+    expected_database_reference = f"db://data-process/{task_id}/{file_id}/v1"
+    if storage_object_id.startswith(expected_local_prefix) and len(storage_object_id) > len(
+        expected_local_prefix
+    ):
+        storage_backend = "local"
+    elif storage_object_id == expected_database_reference:
+        storage_backend = "database"
+    elif storage_object_id.startswith(("local://data-process/", "db://data-process/")):
+        raise DataProcessStoreError("source storage object owner mismatch")
+    else:
+        raise DataProcessStoreError("unsupported source storage object reference")
+    metadata = {
+        **(payload.get("metadata") or {}),
+        "storage_backend": storage_backend,
+    }
+    return storage_object_id, metadata
+
+
+def _decode_row(row: dict[str, Any] | None) -> dict[str, Any] | None:
+    if row is None:
+        return None
+    item = {key: _serialize_value(value) for key, value in row.items()}
+    for key, default in {
+        "config": {},
+        "metadata": {},
+        "quality_score": {},
+        "versions": [],
+        "output_datasets": [],
+    }.items():
+        if key in item:
+            item[key] = _json_value(item[key], default)
+    return item
+
+
+class StoreBase:
+    """数据处理存储基类。"""
+
+    def __init__(self, database_url: str | None = None) -> None:
+        self.database_url = _database_url(database_url or get_settings().database_url)
+
+    @contextmanager
+    def connect(self) -> Iterator[psycopg.Connection[dict[str, Any]]]:
+        with psycopg.connect(self.database_url, row_factory=dict_row) as conn:
+            try:
+                yield conn
+                conn.commit()
+            except Exception:
+                conn.rollback()
+                raise
+
+    def ensure_schema(self) -> None:
+        """显式安装数据处理表；API 路由和应用启动流程不会调用此方法。"""
+        schema_path = Path(__file__).resolve().parents[3] / "db" / "sql" / "002_data_process.sql"
+        sql = schema_path.read_text(encoding="utf-8")
+        with self.connect() as conn, conn.cursor() as cursor:
+            cursor.execute(sql)

backend/app/modules/data_process/store/datasets.py

@@ -0,0 +1,555 @@
+"""数据处理存储层 - 数据集发布。"""
+
+from __future__ import annotations
+
+from typing import Any
+from collections.abc import Sequence
+
+import hashlib
+
+import psycopg
+
+from app.core.config import get_settings
+from app.modules.storage.minio_store import get_object_storage
+
+from .base import (
+    StoreBase,
+    utcnow,
+    new_id,
+    repeat_task_id,
+    json_dumps,
+    _json_value,
+    _decode_row,
+    _public_task,
+    _business_config,
+    _preview_config_value,
+    _preview_config_changed,
+    _preview_config_projection,
+    _normalized_preprocess_options,
+    _regeneration_marker,
+    _is_regeneration_prepared,
+    _task_output_type,
+    _task_reasoning_detail,
+    _reasoning_output_is_valid,
+    _dpo_fields_are_valid,
+    _source_storage_descriptor,
+    NotFoundError,
+    ConflictError,
+    InvalidStateError,
+    EDITABLE_STATUSES,
+    ACTIVE_PREVIEW_STATUSES,
+    WORKFLOW_STEPS,
+    _REGENERATION_MARKER_KEY,
+    _REPEAT_SOURCE_TASK_KEY,
+    _REPEAT_REQUEST_KEY,
+)
+
+from ..algorithms import stable_split_assignments
+
+class DatasetsMixin:
+    """数据集发布 Mixin。"""
+
+    def get_generation_model(self, model_id: str) -> dict[str, Any]:
+        with self.connect() as conn:
+            row = conn.execute(
+                """
+                SELECT id, name, type, purpose, model_source, description, path,
+                       api_url, api_key, online_model_name, create_time
+                FROM models WHERE id=%s
+                """,
+                (model_id,),
+            ).fetchone()
+        if not row:
+            raise NotFoundError("generation model not found")
+        return _decode_row(row) or {}
+
+    def save_generation_model_snapshot(
+        self,
+        task_id: str,
+        model_snapshot: dict[str, Any],
+        *,
+        generation_run_id: str,
+    ) -> dict[str, Any]:
+        # API 密钥仅用于本次调用，绝不能进入任务配置、详情响应或审计快照。
+        safe_snapshot = {
+            key: value for key, value in model_snapshot.items() if key != "api_key"
+        }
+        with self.connect() as conn:
+            task = self._task_in_connection(conn, task_id, for_update=True)
+            if (
+                task["status"] != "running"
+                or task.get("generation_run_id") != generation_run_id
+            ):
+                raise InvalidStateError("generation run is no longer active")
+            config = dict(task.get("config") or {})
+            config["generation_model_snapshot"] = safe_snapshot
+            row = conn.execute(
+                """
+                UPDATE data_process_tasks SET config=%s, updated_at=%s
+                WHERE id=%s AND generation_run_id=%s RETURNING *
+                """,
+                (json_dumps(config), utcnow(), task_id, generation_run_id),
+            ).fetchone()
+        return _decode_row(row) or {}
+
+    def publish(self, task_id: str, payload: dict[str, Any]) -> dict[str, Any]:
+        """按精确配额发布训练、验证、测试三个独立数据集。"""
+        with self.connect() as conn:
+            task = self._task_in_connection(conn, task_id, for_update=True)
+            if _is_regeneration_prepared(task):
+                raise InvalidStateError(
+                    "regeneration must start and complete before publishing"
+                )
+            if task["status"] != "completed":
+                raise InvalidStateError("only a completed task can be published")
+            if not task.get("results_confirmed"):
+                raise InvalidStateError("results must be confirmed before publishing")
+            rows = conn.execute(
+                """
+                SELECT * FROM data_process_results
+                WHERE task_id=%s ORDER BY created_at, id
+                """,
+                (task_id,),
+            ).fetchall()
+            if not rows:
+                raise InvalidStateError("task has no results to publish")
+            invalid_count = sum(
+                1
+                for row in rows
+                if row["status"] == "invalid"
+                or not str(row.get("instruction") or "").strip()
+                or not str(row.get("output") or "").strip()
+                or (
+                    _task_output_type(task) == "reasoning"
+                    and not _reasoning_output_is_valid(row.get("output"))
+                )
+                or (
+                    _task_output_type(task) == "dpo"
+                    and not _dpo_fields_are_valid(row)
+                )
+            )
+            if invalid_count:
+                raise InvalidStateError(f"task contains {invalid_count} invalid results")
+
+            now = utcnow()
+            requested_split = payload.get("split") or {
+                "train": 80,
+                "validation": 10,
+                "test": 10,
+            }
+            assignments = stable_split_assignments(
+                [str(row["id"]) for row in rows],
+                requested_split,
+                seed=task_id,
+            )
+            if _task_output_type(task) == "dpo":
+                records = [
+                    {
+                        "instruction": row["instruction"],
+                        "input": row["input"],
+                        "chosen": row["chosen"],
+                        "rejected": row["rejected"],
+                        "split": assignment,
+                    }
+                    for row, assignment in zip(rows, assignments, strict=True)
+                ]
+            else:
+                records = [
+                    {
+                        "instruction": row["instruction"],
+                        "input": row["input"],
+                        "output": row["output"],
+                        "split": assignment,
+                    }
+                    for row, assignment in zip(rows, assignments, strict=True)
+                ]
+            split_order = ("train", "validation", "test")
+            split_counts = {
+                split_name: assignments.count(split_name) for split_name in split_order
+            }
+            split_specs: list[dict[str, Any]] = []
+            use_minio = bool(get_settings().minio_enabled)
+            for split_name in split_order:
+                split_records = [
+                    (source_row, record)
+                    for source_row, record in zip(rows, records, strict=True)
+                    if record["split"] == split_name
+                ]
+                file_id = new_id("dfile")
+                version_id = new_id("dfv")
+                content = "".join(
+                    json_dumps(record) + "\n" for _, record in split_records
+                )
+                raw = content.encode("utf-8")
+                split_specs.append(
+                    {
+                        "split": split_name,
+                        "records": split_records,
+                        "file_id": file_id,
+                        "version_id": version_id,
+                        "content": content,
+                        "raw": raw,
+                        "checksum": hashlib.sha256(raw).hexdigest(),
+                        "storage_object_id": (
+                            f"db://data-process/{task_id}/{file_id}/v1"
+                        ),
+                    }
+                )
+            source_result_ids = [row["id"] for row in rows]
+            common_metadata = {
+                "source": "data_process",
+                "storage_backend": "minio" if use_minio else "database",
+                "source_task_id": task_id,
+                "output_type": _task_output_type(task),
+                "reasoning_detail": _task_reasoning_detail(task),
+                "source_file_ids": [item["id"] for item in self._source_ids(conn, task_id)],
+                "source_result_ids": source_result_ids,
+                "format": (
+                    "dpo"
+                    if _task_output_type(task) == "dpo"
+                    else payload.get("format") or "alpaca_jsonl"
+                ),
+                "split": requested_split,
+            }
+
+            existing_datasets = conn.execute(
+                """
+                SELECT * FROM datasets
+                WHERE source='task' AND deleted_at IS NULL
+                  AND (
+                      source_task_id=%s
+                      OR (source_task_id IS NULL AND task_id=%s)
+                  )
+                ORDER BY created_at, id
+                """,
+                (task_id, task_id),
+            ).fetchall()
+            existing_by_split: dict[str, dict[str, Any]] = {}
+            primary_existing = None
+            for existing in existing_datasets:
+                existing_metadata = _json_value(existing.get("metadata"), {})
+                existing_split = str(existing_metadata.get("dataset_split") or "")
+                if existing_split in split_order:
+                    existing_by_split[existing_split] = existing
+                if str(existing["id"]) == str(task.get("output_dataset_id") or ""):
+                    primary_existing = existing
+            if primary_existing and "train" not in existing_by_split:
+                # 兼容旧版“一个数据集包含三个文件”的发布物，原数据集复用为训练集。
+                existing_by_split["train"] = primary_existing
+
+            existing_group_metadata = _json_value(
+                (primary_existing or {}).get("metadata"), {}
+            )
+            base_dataset_name = str(
+                existing_group_metadata.get("base_dataset_name")
+                or payload["dataset_name"]
+            ).strip()
+            for suffix in ("-训练集", "-验证集", "-测试集"):
+                if base_dataset_name.endswith(suffix):
+                    base_dataset_name = base_dataset_name[: -len(suffix)].rstrip()
+                    break
+            split_group_id = str(
+                existing_group_metadata.get("split_group_id")
+                or f"dsg_{hashlib.sha256(task_id.encode()).hexdigest()[:20]}"
+            )
+            dataset_ids = {
+                spec["split"]: str(existing_by_split[spec["split"]]["id"])
+                if spec["split"] in existing_by_split
+                else new_id("dataset")
+                for spec in split_specs
+            }
+            created_any = any(
+                spec["split"] not in existing_by_split for spec in split_specs
+            )
+            split_labels = {
+                "train": "训练集",
+                "validation": "验证集",
+                "test": "测试集",
+            }
+            dataset_types = {"train": "train", "validation": "val", "test": "test"}
+            published_datasets: list[dict[str, Any]] = []
+            try:
+                for spec in split_specs:
+                    split_name = str(spec["split"])
+                    dataset_id = dataset_ids[split_name]
+                    storage_object_id = str(spec["storage_object_id"])
+                    if use_minio:
+                        file_name = f"{base_dataset_name}.{split_name}.jsonl"
+                        object_key = f"datasets/{dataset_id}/versions/{spec['version_id']}/{file_name}"
+                        uploaded = get_object_storage().put_bytes(
+                            object_key,
+                            spec["raw"],
+                            "application/jsonl",
+                        )
+                        conn.execute(
+                            """
+                            INSERT INTO storage_objects
+                            (id, resource_type, resource_id, version_id, bucket, object_key,
+                             file_name, content_type, checksum_sha256, byte_size, status,
+                             created_by, create_time)
+                            VALUES (%s, 'dataset', %s, %s, %s, %s, %s, %s, %s, %s,
+                                    'available', %s, %s)
+                            ON CONFLICT (resource_type, resource_id, version_id, object_key)
+                            DO UPDATE SET file_name=EXCLUDED.file_name,
+                                content_type=EXCLUDED.content_type,
+                                checksum_sha256=EXCLUDED.checksum_sha256,
+                                byte_size=EXCLUDED.byte_size,
+                                status='available',
+                                created_by=EXCLUDED.created_by
+                            """,
+                            (
+                                new_id("object"),
+                                dataset_id,
+                                spec["version_id"],
+                                uploaded["bucket"],
+                                object_key,
+                                file_name,
+                                "application/jsonl",
+                                spec["checksum"],
+                                len(spec["raw"]),
+                                payload.get("created_by") or task.get("created_by"),
+                                now,
+                            ),
+                        )
+                        storage_object_id = conn.execute(
+                            """
+                            SELECT id FROM storage_objects
+                            WHERE resource_type='dataset' AND resource_id=%s
+                              AND version_id=%s AND object_key=%s
+                            """,
+                            (dataset_id, spec["version_id"], object_key),
+                        ).fetchone()["id"]
+                    existing_dataset = existing_by_split.get(split_name)
+                    dataset_metadata = {
+                        **common_metadata,
+                        "base_dataset_name": base_dataset_name,
+                        "dataset_split": split_name,
+                        "split_group_id": split_group_id,
+                        "split_dataset_ids": dataset_ids,
+                        "split_counts": {
+                            name: split_counts[name] if name == split_name else 0
+                            for name in split_order
+                        },
+                    }
+                    dataset_name = f"{base_dataset_name}-{split_labels[split_name]}"
+                    if existing_dataset:
+                        conn.execute(
+                            "DELETE FROM dataset_records WHERE dataset_id=%s", (dataset_id,)
+                        )
+                        conn.execute(
+                            """DELETE FROM dataset_file_versions
+                               WHERE dataset_file_id IN
+                               (SELECT id FROM dataset_files WHERE dataset_id=%s)""",
+                            (dataset_id,),
+                        )
+                        conn.execute(
+                            "DELETE FROM dataset_files WHERE dataset_id=%s", (dataset_id,)
+                        )
+                        dataset = conn.execute(
+                            """
+                            UPDATE datasets
+                            SET name=%s, type=%s, storage_type=%s, size=%s, size_bytes=%s,
+                                count=%s, record_count=%s, description=%s, metadata=%s,
+                                updated_at=%s
+                            WHERE id=%s RETURNING *
+                            """,
+                            (
+                                dataset_name,
+                                dataset_types[split_name],
+                                "minio" if use_minio else (payload.get("storage_type") or "local"),
+                                f"{len(spec['raw'])} B",
+                                len(spec["raw"]),
+                                len(spec["records"]),
+                                len(spec["records"]),
+                                payload.get("description") or task.get("description") or "",
+                                json_dumps(dataset_metadata),
+                                now,
+                                dataset_id,
+                            ),
+                        ).fetchone()
+                    else:
+                        dataset = conn.execute(
+                            """
+                            INSERT INTO datasets
+                            (id, name, type, storage_type, source, task_id, source_task_id,
+                             size, size_bytes, count, record_count, description, metadata,
+                             tenant_id, project_id, owner_id, created_by, create_time,
+                             created_at, updated_at)
+                            VALUES (
+                                %s, %s, %s, %s, 'task', %s, %s,
+                                %s, %s, %s, %s, %s, %s,
+                                %s, %s, %s, %s, %s, %s, %s
+                            )
+                            RETURNING *
+                            """,
+                            (
+                                dataset_id,
+                                dataset_name,
+                                dataset_types[split_name],
+                                "minio" if use_minio else (payload.get("storage_type") or "local"),
+                                task_id,
+                                task_id,
+                                f"{len(spec['raw'])} B",
+                                len(spec["raw"]),
+                                len(spec["records"]),
+                                len(spec["records"]),
+                                payload.get("description") or task.get("description") or "",
+                                json_dumps(dataset_metadata),
+                                task.get("tenant_id"),
+                                task.get("project_id"),
+                                task.get("owner_id"),
+                                payload.get("created_by") or task.get("created_by"),
+                                now,
+                                now,
+                                now,
+                            ),
+                        ).fetchone()
+
+                    file_metadata = {**dataset_metadata, "file_split": split_name}
+                    version = {
+                        "id": spec["version_id"],
+                        "version_no": 1,
+                        "version": 1,
+                        "description": f"data process {split_name} publish",
+                        "checksum_sha256": spec["checksum"],
+                        "size_bytes": len(spec["raw"]),
+                        "record_count": len(spec["records"]),
+                        "created_at": now,
+                        "create_time": now,
+                        "source_task_id": task_id,
+                        "storage_object_id": storage_object_id,
+                    }
+                    conn.execute(
+                        """
+                        INSERT INTO dataset_files
+                        (id, dataset_id, name, storage_object_id, size, content,
+                         active_version_id, versions, create_time, current_version_id,
+                         size_bytes, record_count, file_format, checksum_sha256, version_no,
+                         source_task_id, tenant_id, project_id, created_by, metadata,
+                         created_at, updated_at)
+                        VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s,
+                                %s, %s, 1, %s, %s, %s, %s, %s, %s, %s)
+                        """,
+                        (
+                            spec["file_id"],
+                            dataset_id,
+                            f"{base_dataset_name}.{split_name}.jsonl",
+                            storage_object_id,
+                            f"{len(spec['raw'])} B",
+                            spec["content"],
+                            spec["version_id"],
+                            json_dumps([version]),
+                            now,
+                            spec["version_id"],
+                            len(spec["raw"]),
+                            len(spec["records"]),
+                            "jsonl",
+                            spec["checksum"],
+                            task_id,
+                            task.get("tenant_id"),
+                            task.get("project_id"),
+                            payload.get("created_by") or task.get("created_by"),
+                            json_dumps(file_metadata),
+                            now,
+                            now,
+                        ),
+                    )
+                    conn.execute(
+                        """
+                        INSERT INTO dataset_file_versions
+                        (id, dataset_file_id, version_no, storage_object_id, content_preview,
+                         description, size_bytes, record_count, checksum_sha256,
+                         source_task_id, metadata, created_by, created_at)
+                        VALUES (%s, %s, 1, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
+                        """,
+                        (
+                            spec["version_id"],
+                            spec["file_id"],
+                            storage_object_id,
+                            spec["content"][:2000],
+                            f"data process {split_name} publish",
+                            len(spec["raw"]),
+                            len(spec["records"]),
+                            spec["checksum"],
+                            task_id,
+                            json_dumps(file_metadata),
+                            payload.get("created_by") or task.get("created_by"),
+                            now,
+                        ),
+                    )
+                    for line_number, (source_row, record) in enumerate(
+                        spec["records"], start=1
+                    ):
+                        conn.execute(
+                            """
+                            INSERT INTO dataset_records
+                            (id, dataset_id, dataset_file_id, version_id, line_no, split,
+                             instruction, input, output, raw, status, source_task_id,
+                             source_result_id, preview_item_id, created_at)
+                            VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s,
+                                    %s, %s, %s, %s)
+                            """,
+                            (
+                                new_id("drec"),
+                                dataset_id,
+                                spec["file_id"],
+                                spec["version_id"],
+                                line_number,
+                                record["split"],
+                                record["instruction"],
+                                record["input"],
+                                record.get("output") or record.get("chosen") or "",
+                                json_dumps(
+                                    {
+                                        **record,
+                                        "source_task_id": task_id,
+                                        "source_result_id": source_row["id"],
+                                        "preview_item_id": source_row.get("preview_item_id"),
+                                    }
+                                ),
+                                source_row["status"],
+                                task_id,
+                                source_row["id"],
+                                source_row.get("preview_item_id"),
+                                now,
+                            ),
+                        )
+                    published_datasets.append(_decode_row(dataset) or {})
+
+            except psycopg.errors.UniqueViolation as exc:
+                raise ConflictError("dataset name already exists") from exc
+            train_dataset_id = dataset_ids.get("train")
+            if not train_dataset_id:
+                raise InvalidStateError("published split does not contain training data")
+            conn.execute(
+                """
+                UPDATE data_process_tasks
+                SET output_dataset_id=%s, updated_at=%s, updated_by=%s
+                WHERE id=%s
+                """,
+                (train_dataset_id, now, payload.get("created_by"), task_id),
+            )
+        train_dataset = next(
+            item
+            for item in published_datasets
+            if _json_value(item.get("metadata"), {}).get("dataset_split") == "train"
+        )
+        return {
+            "dataset": train_dataset,
+            "datasets": published_datasets,
+            "output_datasets": published_datasets,
+            "created": created_any,
+            "split_counts": split_counts,
+        }
+
+    @staticmethod
+    def _source_ids(
+        conn: psycopg.Connection[dict[str, Any]], task_id: str
+    ) -> list[dict[str, Any]]:
+        return conn.execute(
+            """
+            SELECT id FROM data_process_source_files
+            WHERE task_id=%s AND deleted_at IS NULL ORDER BY created_at, id
+            """,
+            (task_id,),
+        ).fetchall()

backend/app/modules/data_process/store/generation.py

@@ -0,0 +1,275 @@
+"""数据处理存储层 - 生成管理。"""
+
+from __future__ import annotations
+
+from typing import Any
+from collections.abc import Sequence
+
+import psycopg
+
+from .base import (
+    StoreBase,
+    utcnow,
+    new_id,
+    repeat_task_id,
+    json_dumps,
+    _json_value,
+    _decode_row,
+    _public_task,
+    _business_config,
+    _preview_config_value,
+    _preview_config_changed,
+    _preview_config_projection,
+    _normalized_preprocess_options,
+    _regeneration_marker,
+    _is_regeneration_prepared,
+    _task_output_type,
+    _task_reasoning_detail,
+    _reasoning_output_is_valid,
+    _dpo_fields_are_valid,
+    _source_storage_descriptor,
+    DataProcessStoreError,
+    NotFoundError,
+    ConflictError,
+    InvalidStateError,
+    EDITABLE_STATUSES,
+    ACTIVE_PREVIEW_STATUSES,
+    WORKFLOW_STEPS,
+    _REGENERATION_MARKER_KEY,
+    _REPEAT_SOURCE_TASK_KEY,
+    _REPEAT_REQUEST_KEY,
+)
+
+
+class GenerationMixin:
+    """生成管理 Mixin。"""
+
+    def _invalidate_results(
+        self,
+        conn: psycopg.Connection[dict[str, Any]],
+        task: dict[str, Any],
+        task_id: str,
+        now: str,
+    ) -> None:
+        if _is_regeneration_prepared(task):
+            conn.execute(
+                "UPDATE data_process_tasks SET updated_at=%s WHERE id=%s",
+                (now, task_id),
+            )
+            return
+        conn.execute("DELETE FROM data_process_results WHERE task_id=%s", (task_id,))
+        conn.execute(
+            """
+            UPDATE data_process_tasks
+            SET status='pending', progress=20, output_count=0, filtered_count=0,
+                duplicate_count=0, error_count=0, failure_reason=NULL,
+                generation_run_id=NULL, results_confirmed=FALSE, updated_at=%s
+            WHERE id=%s
+            """,
+            (now, task_id),
+        )
+
+    def start_generation(self, task_id: str, *, replace_existing: bool = True) -> dict[str, Any]:
+        if not replace_existing:
+            raise DataProcessStoreError("incremental generation is not supported")
+        with self.connect() as conn:
+            task = self._task_in_connection(conn, task_id, for_update=True)
+            regeneration_prepared = _is_regeneration_prepared(task)
+            if task.get("output_dataset_id") and not regeneration_prepared:
+                raise InvalidStateError("published task cannot be regenerated")
+            if task["status"] == "running":
+                raise ConflictError("data process task is already running")
+            if task.get("preview_status") in ACTIVE_PREVIEW_STATUSES:
+                raise ConflictError("preview is still running")
+            preview_count = conn.execute(
+                "SELECT COUNT(*) AS count FROM data_process_preview_items WHERE task_id=%s",
+                (task_id,),
+            ).fetchone()["count"]
+            if not preview_count:
+                raise InvalidStateError("preview must be built before generation")
+            conn.execute("DELETE FROM data_process_results WHERE task_id=%s", (task_id,))
+            now = utcnow()
+            generation_run_id = new_id("dprun")
+            next_config = dict(task.get("config") or {})
+            next_config.pop(_REGENERATION_MARKER_KEY, None)
+            row = conn.execute(
+                """
+                UPDATE data_process_tasks
+                SET config=%s, status='running', progress=30, failure_reason=NULL,
+                    started_at=%s, completed_at=NULL, output_dataset_id=NULL,
+                    output_count=0, filtered_count=0, duplicate_count=0, error_count=0,
+                    generation_run_id=%s, results_confirmed=FALSE,
+                    workflow_step='generate', updated_at=%s
+                WHERE id=%s
+                RETURNING *
+                """,
+                (json_dumps(next_config), now, generation_run_id, now, task_id),
+            ).fetchone()
+        return _decode_row(row) or {}
+
+    def stop_task(self, task_id: str) -> dict[str, Any]:
+        with self.connect() as conn:
+            task = self._task_in_connection(conn, task_id, for_update=True)
+            if task["status"] != "running":
+                raise InvalidStateError("only a running task can be stopped")
+            now = utcnow()
+            row = conn.execute(
+                """
+                UPDATE data_process_tasks
+                SET status='stopped', failure_reason=NULL, generation_run_id=NULL,
+                    updated_at=%s
+                WHERE id=%s RETURNING *
+                """,
+                (now, task_id),
+            ).fetchone()
+        return _decode_row(row) or {}
+
+    def generation_is_running(self, task_id: str, generation_run_id: str) -> bool:
+        with self.connect() as conn:
+            row = conn.execute(
+                """
+                SELECT status, generation_run_id
+                FROM data_process_tasks
+                WHERE id=%s AND deleted_at IS NULL
+                """,
+                (task_id,),
+            ).fetchone()
+        return bool(
+            row
+            and row.get("status") == "running"
+            and row.get("generation_run_id") == generation_run_id
+        )
+
+    def update_generation_progress(
+        self,
+        task_id: str,
+        generation_run_id: str,
+        processed_count: int,
+        total_count: int,
+    ) -> bool:
+        ratio = processed_count / max(1, total_count)
+        progress = min(95.0, 30.0 + ratio * 65.0)
+        with self.connect() as conn:
+            row = conn.execute(
+                """
+                UPDATE data_process_tasks
+                SET progress=%s, updated_at=%s
+                WHERE id=%s AND status='running' AND generation_run_id=%s
+                RETURNING id
+                """,
+                (progress, utcnow(), task_id, generation_run_id),
+            ).fetchone()
+        return row is not None
+
+    def complete_generation(
+        self,
+        task_id: str,
+        results: Sequence[dict[str, Any]],
+        *,
+        generation_run_id: str,
+        filtered_count: int = 0,
+        duplicate_count: int = 0,
+        error_count: int = 0,
+    ) -> dict[str, Any]:
+        now = utcnow()
+        with self.connect() as conn:
+            task = self._task_in_connection(conn, task_id, for_update=True)
+            if (
+                task["status"] != "running"
+                or task.get("generation_run_id") != generation_run_id
+            ):
+                return task
+            conn.execute("DELETE FROM data_process_results WHERE task_id=%s", (task_id,))
+            for result in results:
+                conn.execute(
+                    """
+                    INSERT INTO data_process_results
+                    (id, task_id, preview_item_id, instruction, input, output,
+                     chosen, rejected, original_instruction, original_input,
+                     original_output, original_chosen, original_rejected, status, error,
+                     split, quality_score, created_at, updated_at)
+                    VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s,
+                            %s, %s, %s, %s, %s, %s, %s)
+                    """,
+                    (
+                        result.get("id") or new_id("dpr"),
+                        task_id,
+                        result.get("preview_item_id"),
+                        result.get("instruction") or "",
+                        result.get("input") or "",
+                        result.get("output") or "",
+                        result.get("chosen") or "",
+                        result.get("rejected") or "",
+                        result.get("original_instruction", result.get("instruction") or ""),
+                        result.get("original_input", result.get("input") or ""),
+                        result.get("original_output", result.get("output") or ""),
+                        result.get("original_chosen", result.get("chosen") or ""),
+                        result.get("original_rejected", result.get("rejected") or ""),
+                        result.get("status") or "valid",
+                        result.get("error"),
+                        result.get("split"),
+                        json_dumps(result.get("quality_score") or {}),
+                        now,
+                        now,
+                    ),
+                )
+            row = conn.execute(
+                """
+                UPDATE data_process_tasks
+                SET status='completed', progress=100, output_count=%s, filtered_count=%s,
+                    duplicate_count=%s, error_count=%s, failure_reason=NULL,
+                    completed_at=%s, generation_run_id=NULL, results_confirmed=FALSE,
+                    updated_at=%s
+                WHERE id=%s AND generation_run_id=%s RETURNING *
+                """,
+                (
+                    len(results),
+                    filtered_count,
+                    duplicate_count,
+                    error_count,
+                    now,
+                    now,
+                    task_id,
+                    generation_run_id,
+                ),
+            ).fetchone()
+        return _decode_row(row) or {}
+
+    def mark_failed(
+        self, task_id: str, reason: str, *, generation_run_id: str
+    ) -> dict[str, Any]:
+        with self.connect() as conn:
+            task = self._task_in_connection(conn, task_id, for_update=True)
+            if (
+                task["status"] != "running"
+                or task.get("generation_run_id") != generation_run_id
+            ):
+                return task
+            now = utcnow()
+            row = conn.execute(
+                """
+                UPDATE data_process_tasks
+                SET status='failed', failure_reason=%s, completed_at=%s,
+                    generation_run_id=NULL, updated_at=%s
+                WHERE id=%s AND generation_run_id=%s RETURNING *
+                """,
+                (reason[:4000], now, now, task_id, generation_run_id),
+            ).fetchone()
+        return _decode_row(row) or {}
+
+    def progress(self, task_id: str) -> dict[str, Any]:
+        task = self.get_task(task_id)
+        return {
+            "task_id": task["id"],
+            "status": task["status"],
+            "progress": float(task.get("progress") or 0),
+            "input_count": int(task.get("input_count") or 0),
+            "output_count": int(task.get("output_count") or 0),
+            "filtered_count": int(task.get("filtered_count") or 0),
+            "duplicate_count": int(task.get("duplicate_count") or 0),
+            "error_count": int(task.get("error_count") or 0),
+            "failure_reason": task.get("failure_reason"),
+            "results_confirmed": bool(task.get("results_confirmed")),
+            "started_at": task.get("started_at"),
+            "completed_at": task.get("completed_at"),
+        }

backend/app/modules/data_process/store/preview.py

@@ -0,0 +1,539 @@
+"""数据处理存储层 - 预览管理。"""
+
+from __future__ import annotations
+
+from typing import Any
+from collections.abc import Sequence
+
+import psycopg
+
+from .base import (
+    StoreBase,
+    utcnow,
+    new_id,
+    repeat_task_id,
+    json_dumps,
+    _json_value,
+    _decode_row,
+    _public_task,
+    _business_config,
+    _preview_config_value,
+    _preview_config_changed,
+    _preview_config_projection,
+    _normalized_preprocess_options,
+    _regeneration_marker,
+    _is_regeneration_prepared,
+    _task_output_type,
+    _task_reasoning_detail,
+    _reasoning_output_is_valid,
+    _dpo_fields_are_valid,
+    _source_storage_descriptor,
+    _serialize_value,
+    NotFoundError,
+    ConflictError,
+    InvalidStateError,
+    EDITABLE_STATUSES,
+    ACTIVE_PREVIEW_STATUSES,
+    WORKFLOW_STEPS,
+    _REGENERATION_MARKER_KEY,
+    _REPEAT_SOURCE_TASK_KEY,
+    _REPEAT_REQUEST_KEY,
+)
+from ..algorithms import estimate_token_count  # noqa: E402
+
+
+class PreviewMixin:
+    """预览管理 Mixin。"""
+
+    def replace_preview_items(
+        self,
+        task_id: str,
+        items: Sequence[dict[str, Any]],
+        *,
+        source_file_ids: Sequence[str] | None = None,
+        preview_run_id: str | None = None,
+    ) -> list[dict[str, Any]]:
+        selected_ids = (
+            list(dict.fromkeys(str(file_id) for file_id in source_file_ids))
+            if source_file_ids is not None
+            else None
+        )
+        if selected_ids is not None:
+            if not selected_ids or any(not file_id for file_id in selected_ids):
+                raise ValueError("source_file_ids must contain non-empty ids")
+            selected_set = set(selected_ids)
+            unexpected = {
+                str(item.get("source_file_id") or "")
+                for item in items
+                if str(item.get("source_file_id") or "") not in selected_set
+            }
+            if unexpected:
+                raise ValueError("preview items contain an unselected source file")
+        preview_file_count = len(selected_ids) if selected_ids is not None else len(
+            {str(item.get("source_file_id") or "") for item in items}
+        )
+        is_direct_build = preview_run_id is None
+
+        now = utcnow()
+        with self.connect() as conn:
+            task = self._task_in_connection(conn, task_id, for_update=True)
+            if is_direct_build:
+                self._ensure_editable(task)
+            elif (
+                task.get("preview_run_id") != preview_run_id
+                or task.get("preview_status") != "running"
+            ):
+                raise InvalidStateError("preview run is no longer active")
+            regeneration_prepared = _is_regeneration_prepared(task)
+            if not regeneration_prepared:
+                conn.execute("DELETE FROM data_process_results WHERE task_id=%s", (task_id,))
+            if selected_ids is None:
+                conn.execute(
+                    "DELETE FROM data_process_preview_items WHERE task_id=%s", (task_id,)
+                )
+            else:
+                rows = conn.execute(
+                    """
+                    SELECT id FROM data_process_source_files
+                    WHERE task_id=%s AND deleted_at IS NULL AND id=ANY(%s)
+                    """,
+                    (task_id, selected_ids),
+                ).fetchall()
+                found = {str(row["id"]) for row in rows}
+                missing = set(selected_ids) - found
+                if missing:
+                    raise NotFoundError(
+                        f"source files not found: {', '.join(sorted(missing))}"
+                    )
+                conn.execute(
+                    """
+                    DELETE FROM data_process_preview_items
+                    WHERE task_id=%s AND source_file_id=ANY(%s)
+                    """,
+                    (task_id, selected_ids),
+                )
+            created: list[dict[str, Any]] = []
+            for item in items:
+                row = conn.execute(
+                    """
+                    INSERT INTO data_process_preview_items
+                    (id, task_id, source_file_id, original_content, edited_content,
+                     source_start, source_end, source_start_line, source_end_line,
+                     token_count, status, quality_score, created_at, updated_at)
+                    VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
+                    RETURNING *
+                    """,
+                    (
+                        item.get("id") or new_id("dpp"),
+                        task_id,
+                        item.get("source_file_id"),
+                        item.get("original_content") or "",
+                        item.get("edited_content", item.get("original_content") or ""),
+                        item.get("source_start"),
+                        item.get("source_end"),
+                        item.get("source_start_line"),
+                        item.get("source_end_line"),
+                        max(0, int(item.get("token_count") or 0)),
+                        item.get("status") or "original",
+                        json_dumps(item.get("quality_score") or {}),
+                        now,
+                        now,
+                    ),
+                ).fetchone()
+                created.append(_decode_row(row) or {})
+            if regeneration_prepared:
+                if is_direct_build:
+                    conn.execute(
+                        """
+                        UPDATE data_process_tasks
+                        SET workflow_step='preview', preview_status='completed',
+                            preview_progress=100, preview_run_id=NULL,
+                            preview_failure_reason=NULL, preview_total_files=%s,
+                            preview_completed_files=%s, updated_at=%s
+                        WHERE id=%s
+                        """,
+                        (preview_file_count, preview_file_count, now, task_id),
+                    )
+                else:
+                    conn.execute(
+                        "UPDATE data_process_tasks SET updated_at=%s WHERE id=%s",
+                        (now, task_id),
+                    )
+            else:
+                conn.execute(
+                    """
+                    UPDATE data_process_tasks
+                    SET status='pending', progress=20, output_count=0, filtered_count=0,
+                        duplicate_count=0, error_count=0, failure_reason=NULL,
+                        results_confirmed=FALSE,
+                        workflow_step=CASE WHEN %s THEN 'preview' ELSE workflow_step END,
+                        preview_status=CASE WHEN %s THEN 'completed' ELSE preview_status END,
+                        preview_progress=CASE WHEN %s THEN 100 ELSE preview_progress END,
+                        preview_run_id=CASE WHEN %s THEN NULL ELSE preview_run_id END,
+                        preview_failure_reason=CASE WHEN %s THEN NULL ELSE preview_failure_reason END,
+                        preview_total_files=CASE WHEN %s THEN %s ELSE preview_total_files END,
+                        preview_completed_files=CASE WHEN %s THEN %s ELSE preview_completed_files END,
+                        updated_at=%s
+                    WHERE id=%s
+                    """,
+                    (
+                        is_direct_build,
+                        is_direct_build,
+                        is_direct_build,
+                        is_direct_build,
+                        is_direct_build,
+                        is_direct_build,
+                        preview_file_count,
+                        is_direct_build,
+                        preview_file_count,
+                        now,
+                        task_id,
+                    ),
+                )
+        return created
+
+    def start_preview(
+        self,
+        task_id: str,
+        *,
+        source_file_ids: Sequence[str] | None = None,
+    ) -> tuple[dict[str, Any], list[str]]:
+        """创建一轮持久化切分任务，并返回本轮固定的源文件集合。"""
+
+        requested_ids = (
+            list(dict.fromkeys(str(file_id) for file_id in source_file_ids))
+            if source_file_ids is not None
+            else None
+        )
+        if requested_ids is not None and (
+            not requested_ids or any(not file_id for file_id in requested_ids)
+        ):
+            raise ValueError("source_file_ids must contain non-empty ids")
+
+        with self.connect() as conn:
+            task = self._task_in_connection(conn, task_id, for_update=True)
+            self._ensure_editable(task)
+            if requested_ids is None:
+                rows = conn.execute(
+                    """
+                    SELECT id FROM data_process_source_files
+                    WHERE task_id=%s AND deleted_at IS NULL
+                    ORDER BY created_at, id
+                    """,
+                    (task_id,),
+                ).fetchall()
+            else:
+                rows = conn.execute(
+                    """
+                    SELECT id FROM data_process_source_files
+                    WHERE task_id=%s AND deleted_at IS NULL AND id=ANY(%s)
+                    ORDER BY created_at, id
+                    """,
+                    (task_id, requested_ids),
+                ).fetchall()
+            selected_ids = [str(row["id"]) for row in rows]
+            if not selected_ids:
+                raise InvalidStateError("at least one source file is required")
+            if requested_ids is not None:
+                missing = set(requested_ids) - set(selected_ids)
+                if missing:
+                    raise NotFoundError(
+                        f"source files not found: {', '.join(sorted(missing))}"
+                    )
+
+            regeneration_prepared = _is_regeneration_prepared(task)
+            if not regeneration_prepared:
+                conn.execute("DELETE FROM data_process_results WHERE task_id=%s", (task_id,))
+            preview_run_id = new_id("dpprun")
+            now = utcnow()
+            row = conn.execute(
+                """
+                UPDATE data_process_tasks
+                SET status=CASE WHEN %s THEN status ELSE 'pending' END,
+                    progress=CASE WHEN %s THEN progress ELSE 0 END,
+                    output_count=CASE WHEN %s THEN output_count ELSE 0 END,
+                    filtered_count=CASE WHEN %s THEN filtered_count ELSE 0 END,
+                    duplicate_count=CASE WHEN %s THEN duplicate_count ELSE 0 END,
+                    error_count=CASE WHEN %s THEN error_count ELSE 0 END,
+                    failure_reason=CASE WHEN %s THEN failure_reason ELSE NULL END,
+                    results_confirmed=CASE WHEN %s THEN results_confirmed ELSE FALSE END,
+                    workflow_step='upload', preview_status='queued', preview_progress=0,
+                    preview_run_id=%s, preview_failure_reason=NULL,
+                    preview_total_files=%s, preview_completed_files=0,
+                    updated_at=%s
+                WHERE id=%s AND deleted_at IS NULL
+                RETURNING *
+                """,
+                (
+                    regeneration_prepared,
+                    regeneration_prepared,
+                    regeneration_prepared,
+                    regeneration_prepared,
+                    regeneration_prepared,
+                    regeneration_prepared,
+                    regeneration_prepared,
+                    regeneration_prepared,
+                    preview_run_id,
+                    len(selected_ids),
+                    now,
+                    task_id,
+                ),
+            ).fetchone()
+        return _public_task(_decode_row(row)) or {}, selected_ids
+
+    def mark_preview_running(self, task_id: str, preview_run_id: str) -> bool:
+        with self.connect() as conn:
+            row = conn.execute(
+                """
+                UPDATE data_process_tasks
+                SET preview_status='running', updated_at=%s
+                WHERE id=%s AND deleted_at IS NULL
+                  AND preview_status='queued' AND preview_run_id=%s
+                RETURNING id
+                """,
+                (utcnow(), task_id, preview_run_id),
+            ).fetchone()
+        return row is not None
+
+    def preview_is_running(self, task_id: str, preview_run_id: str) -> bool:
+        with self.connect() as conn:
+            row = conn.execute(
+                """
+                SELECT preview_status, preview_run_id
+                FROM data_process_tasks
+                WHERE id=%s AND deleted_at IS NULL
+                """,
+                (task_id,),
+            ).fetchone()
+        return bool(
+            row
+            and row.get("preview_status") in ACTIVE_PREVIEW_STATUSES
+            and row.get("preview_run_id") == preview_run_id
+        )
+
+    def update_preview_progress(
+        self,
+        task_id: str,
+        preview_run_id: str,
+        completed_files: int,
+        total_files: int,
+    ) -> bool:
+        total = max(1, total_files)
+        completed = min(max(0, completed_files), total)
+        progress = completed / total * 100
+        with self.connect() as conn:
+            row = conn.execute(
+                """
+                UPDATE data_process_tasks
+                SET preview_progress=%s, preview_completed_files=%s, updated_at=%s
+                WHERE id=%s AND deleted_at IS NULL
+                  AND preview_status='running' AND preview_run_id=%s
+                RETURNING id
+                """,
+                (progress, completed, utcnow(), task_id, preview_run_id),
+            ).fetchone()
+        return row is not None
+
+    def complete_preview(self, task_id: str, preview_run_id: str) -> bool:
+        with self.connect() as conn:
+            row = conn.execute(
+                """
+                UPDATE data_process_tasks
+                SET workflow_step='preview', preview_status='completed',
+                    preview_progress=100, preview_run_id=NULL,
+                    preview_failure_reason=NULL,
+                    preview_completed_files=preview_total_files, updated_at=%s
+                WHERE id=%s AND deleted_at IS NULL
+                  AND preview_status='running' AND preview_run_id=%s
+                RETURNING id
+                """,
+                (utcnow(), task_id, preview_run_id),
+            ).fetchone()
+        return row is not None
+
+    def mark_preview_failed(
+        self,
+        task_id: str,
+        reason: str,
+        *,
+        preview_run_id: str,
+    ) -> bool:
+        with self.connect() as conn:
+            row = conn.execute(
+                """
+                UPDATE data_process_tasks
+                SET preview_status='failed', preview_run_id=NULL,
+                    preview_failure_reason=%s, updated_at=%s
+                WHERE id=%s AND deleted_at IS NULL
+                  AND preview_status IN ('queued', 'running') AND preview_run_id=%s
+                RETURNING id
+                """,
+                (reason[:4000], utcnow(), task_id, preview_run_id),
+            ).fetchone()
+        return row is not None
+
+    def preview_progress(self, task_id: str) -> dict[str, Any]:
+        task = self.get_task(task_id)
+        return {
+            "task_id": task["id"],
+            "workflow_step": task.get("workflow_step") or "create",
+            "preview_status": task.get("preview_status") or "idle",
+            "preview_progress": float(task.get("preview_progress") or 0),
+            "preview_run_id": task.get("preview_run_id"),
+            "preview_failure_reason": task.get("preview_failure_reason"),
+            "preview_total_files": int(task.get("preview_total_files") or 0),
+            "preview_completed_files": int(task.get("preview_completed_files") or 0),
+        }
+
+    def list_preview_items(
+        self,
+        task_id: str,
+        *,
+        source_file_id: str | None = None,
+        page: int = 1,
+        page_size: int = 200,
+        keyword: str | None = None,
+    ) -> dict[str, Any]:
+        self.get_task(task_id)
+        clauses = ["task_id=%s"]
+        params: list[Any] = [task_id]
+        if source_file_id:
+            clauses.append("source_file_id=%s")
+            params.append(source_file_id)
+        if keyword:
+            clauses.append("(original_content ILIKE %s OR edited_content ILIKE %s)")
+            pattern = f"%{keyword.strip()}%"
+            params.extend([pattern, pattern])
+        where = " AND ".join(clauses)
+        with self.connect() as conn:
+            total = conn.execute(
+                f"SELECT COUNT(*) AS count FROM data_process_preview_items WHERE {where}", params
+            ).fetchone()["count"]
+            rows = conn.execute(
+                f"""
+                SELECT * FROM data_process_preview_items
+                WHERE {where}
+                ORDER BY source_file_id NULLS LAST, source_start NULLS LAST, created_at, id
+                LIMIT %s OFFSET %s
+                """,
+                [*params, page_size, (page - 1) * page_size],
+            ).fetchall()
+        return {
+            "items": [_decode_row(row) for row in rows],
+            "total": int(total),
+            "page": page,
+            "page_size": page_size,
+        }
+
+    def get_preview_item(self, task_id: str, preview_id: str) -> dict[str, Any]:
+        with self.connect() as conn:
+            row = conn.execute(
+                "SELECT * FROM data_process_preview_items WHERE id=%s AND task_id=%s",
+                (preview_id, task_id),
+            ).fetchone()
+        if not row:
+            raise NotFoundError("preview item not found")
+        return _decode_row(row) or {}
+
+    def create_preview_item(self, task_id: str, item: dict[str, Any]) -> dict[str, Any]:
+        now = utcnow()
+        with self.connect() as conn:
+            task = self._task_in_connection(conn, task_id, for_update=True)
+            self._ensure_editable(task)
+            if item.get("source_file_id"):
+                source = conn.execute(
+                    """
+                    SELECT id FROM data_process_source_files
+                    WHERE id=%s AND task_id=%s AND deleted_at IS NULL
+                    """,
+                    (item["source_file_id"], task_id),
+                ).fetchone()
+                if not source:
+                    raise NotFoundError("source file not found")
+            row = conn.execute(
+                """
+                INSERT INTO data_process_preview_items
+                (id, task_id, source_file_id, original_content, edited_content,
+                 source_start, source_end, source_start_line, source_end_line,
+                 token_count, status, quality_score, created_at, updated_at)
+                VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
+                RETURNING *
+                """,
+                (
+                    new_id("dpp"),
+                    task_id,
+                    item.get("source_file_id"),
+                    item.get("original_content") or "",
+                    item.get("edited_content") or "",
+                    item.get("source_start"),
+                    item.get("source_end"),
+                    item.get("source_start_line"),
+                    item.get("source_end_line"),
+                    max(0, int(item.get("token_count") or 0)),
+                    item.get("status") or "manual",
+                    json_dumps(item.get("quality_score") or {}),
+                    now,
+                    now,
+                ),
+            ).fetchone()
+            self._invalidate_results(conn, task, task_id, now)
+        return _decode_row(row) or {}
+
+    def update_preview_item(
+        self, task_id: str, preview_id: str, payload: dict[str, Any]
+    ) -> dict[str, Any]:
+        with self.connect() as conn:
+            task = self._task_in_connection(conn, task_id, for_update=True)
+            self._ensure_editable(task)
+            existing = conn.execute(
+                "SELECT * FROM data_process_preview_items WHERE id=%s AND task_id=%s",
+                (preview_id, task_id),
+            ).fetchone()
+            if not existing:
+                raise NotFoundError("preview item not found")
+            expected_updated_at = payload.get("expected_updated_at")
+            current_updated_at = _serialize_value(existing.get("updated_at"))
+            if expected_updated_at and expected_updated_at != current_updated_at:
+                raise ConflictError("preview item was modified by another request")
+            edited = payload["edited_content"]
+            status = payload.get("status")
+            if not status:
+                if not edited.strip():
+                    status = "invalid"
+                elif edited == existing["original_content"]:
+                    status = "original"
+                else:
+                    status = "modified"
+            now = utcnow()
+            row = conn.execute(
+                """
+                UPDATE data_process_preview_items
+                SET edited_content=%s, token_count=%s, status=%s, quality_score=%s,
+                    updated_at=%s
+                WHERE id=%s AND task_id=%s
+                RETURNING *
+                """,
+                (
+                    edited,
+                    estimate_token_count(edited),
+                    status,
+                    json_dumps(payload.get("quality_score") or {}),
+                    now,
+                    preview_id,
+                    task_id,
+                ),
+            ).fetchone()
+            self._invalidate_results(conn, task, task_id, now)
+        return _decode_row(row) or {}
+
+    def delete_preview_item(self, task_id: str, preview_id: str) -> None:
+        with self.connect() as conn:
+            task = self._task_in_connection(conn, task_id, for_update=True)
+            self._ensure_editable(task)
+            row = conn.execute(
+                "DELETE FROM data_process_preview_items WHERE id=%s AND task_id=%s RETURNING id",
+                (preview_id, task_id),
+            ).fetchone()
+            if not row:
+                raise NotFoundError("preview item not found")
+            self._invalidate_results(conn, task, task_id, utcnow())

backend/app/modules/data_process/store/results.py

@@ -0,0 +1,324 @@
+"""数据处理存储层 - 结果管理。"""
+
+from __future__ import annotations
+
+from typing import Any
+from collections.abc import Sequence
+
+import psycopg
+
+from .base import (
+    StoreBase,
+    utcnow,
+    new_id,
+    repeat_task_id,
+    json_dumps,
+    _json_value,
+    _decode_row,
+    _public_task,
+    _business_config,
+    _preview_config_value,
+    _preview_config_changed,
+    _preview_config_projection,
+    _normalized_preprocess_options,
+    _regeneration_marker,
+    _is_regeneration_prepared,
+    _task_output_type,
+    _task_reasoning_detail,
+    _reasoning_output_is_valid,
+    _dpo_fields_are_valid,
+    _source_storage_descriptor,
+    _serialize_value,
+    DataProcessStoreError,
+    NotFoundError,
+    ConflictError,
+    InvalidStateError,
+    EDITABLE_STATUSES,
+    ACTIVE_PREVIEW_STATUSES,
+    WORKFLOW_STEPS,
+    _REGENERATION_MARKER_KEY,
+    _REPEAT_SOURCE_TASK_KEY,
+    _REPEAT_REQUEST_KEY,
+)
+
+
+class ResultsMixin:
+    """结果管理 Mixin。"""
+
+    def confirm_results(self, task_id: str) -> dict[str, Any]:
+        """确认第六步结果，确认前再次校验所有生成记录。"""
+
+        with self.connect() as conn:
+            task = self._task_in_connection(conn, task_id, for_update=True)
+            if task["status"] != "completed":
+                raise InvalidStateError("only a completed task can confirm results")
+            if task.get("workflow_step") != "results":
+                raise InvalidStateError("workflow must be on results before confirmation")
+            if task.get("results_confirmed"):
+                return task
+
+            rows = conn.execute(
+                """
+                SELECT status, instruction, output, chosen, rejected
+                FROM data_process_results
+                WHERE task_id=%s
+                """,
+                (task_id,),
+            ).fetchall()
+            if not rows:
+                raise InvalidStateError("task has no results to confirm")
+            invalid_count = sum(
+                1
+                for row in rows
+                if row["status"] == "invalid"
+                or not str(row.get("instruction") or "").strip()
+                or not str(row.get("output") or "").strip()
+                or (
+                    _task_output_type(task) == "reasoning"
+                    and not _reasoning_output_is_valid(row.get("output"))
+                )
+                or (
+                    _task_output_type(task) == "dpo"
+                    and not _dpo_fields_are_valid(row)
+                )
+            )
+            if invalid_count:
+                raise InvalidStateError(
+                    f"task contains {invalid_count} invalid results"
+                )
+
+            row = conn.execute(
+                """
+                UPDATE data_process_tasks
+                SET results_confirmed=TRUE, updated_at=%s
+                WHERE id=%s RETURNING *
+                """,
+                (utcnow(), task_id),
+            ).fetchone()
+        return _decode_row(row) or {}
+
+    def list_results(
+        self,
+        task_id: str,
+        *,
+        page: int = 1,
+        page_size: int = 100,
+        status: str | None = None,
+        split: str | None = None,
+        keyword: str | None = None,
+    ) -> dict[str, Any]:
+        self.get_task(task_id)
+        clauses = ["task_id=%s"]
+        params: list[Any] = [task_id]
+        if status:
+            clauses.append("status=%s")
+            params.append(status)
+        if split:
+            clauses.append("split=%s")
+            params.append(split)
+        if keyword:
+            clauses.append("(instruction ILIKE %s OR input ILIKE %s OR output ILIKE %s)")
+            pattern = f"%{keyword.strip()}%"
+            params.extend([pattern, pattern, pattern])
+        where = " AND ".join(clauses)
+        with self.connect() as conn:
+            total = conn.execute(
+                f"SELECT COUNT(*) AS count FROM data_process_results WHERE {where}", params
+            ).fetchone()["count"]
+            rows = conn.execute(
+                f"""
+                SELECT * FROM data_process_results WHERE {where}
+                ORDER BY created_at, id LIMIT %s OFFSET %s
+                """,
+                [*params, page_size, (page - 1) * page_size],
+            ).fetchall()
+        return {
+            "items": [_decode_row(row) for row in rows],
+            "total": int(total),
+            "page": page,
+            "page_size": page_size,
+        }
+
+    def get_result(self, task_id: str, result_id: str) -> dict[str, Any]:
+        with self.connect() as conn:
+            row = conn.execute(
+                "SELECT * FROM data_process_results WHERE id=%s AND task_id=%s",
+                (result_id, task_id),
+            ).fetchone()
+        if not row:
+            raise NotFoundError("data process result not found")
+        return _decode_row(row) or {}
+
+    def update_result(
+        self, task_id: str, result_id: str, payload: dict[str, Any]
+    ) -> dict[str, Any]:
+        allowed = {
+            "instruction", "input", "output", "chosen", "rejected", "quality_score"
+        }
+        values = {key: value for key, value in payload.items() if key in allowed}
+        if "quality_score" in values:
+            values["quality_score"] = json_dumps(values["quality_score"])
+        if not values:
+            raise DataProcessStoreError("no result fields supplied")
+        with self.connect() as conn:
+            task = self._task_in_connection(conn, task_id, for_update=True)
+            if task["status"] == "running":
+                raise InvalidStateError("results cannot be edited while generation is running")
+            if task.get("output_dataset_id"):
+                raise InvalidStateError("published results cannot be edited")
+            current = conn.execute(
+                "SELECT * FROM data_process_results WHERE id=%s AND task_id=%s",
+                (result_id, task_id),
+            ).fetchone()
+            if not current:
+                raise NotFoundError("data process result not found")
+            expected_updated_at = payload.get("expected_updated_at")
+            current_updated_at = _serialize_value(current.get("updated_at"))
+            if expected_updated_at and expected_updated_at != current_updated_at:
+                raise ConflictError("data process result was modified by another request")
+            output_type = _task_output_type(task)
+            if output_type == "dpo" and "chosen" in values:
+                values["output"] = values["chosen"]
+            merged = {**current, **values}
+            quality = payload.get("quality_score") or {}
+            instruction_valid = bool(str(merged.get("instruction") or "").strip())
+            output_valid = bool(str(merged.get("output") or "").strip())
+            reasoning_valid = (
+                output_type != "reasoning"
+                or _reasoning_output_is_valid(merged.get("output"))
+            )
+            dpo_valid = output_type != "dpo" or _dpo_fields_are_valid(merged)
+            hard_valid = instruction_valid and output_valid and reasoning_valid and dpo_valid
+            quality_valid = bool(quality.get("is_valid", hard_valid))
+            changed = any(
+                str(merged.get(field) or "")
+                != str(merged.get(f"original_{field}") or "")
+                for field in (
+                    ("instruction", "input", "chosen", "rejected")
+                    if output_type == "dpo"
+                    else ("instruction", "input", "output")
+                )
+            )
+            status = "invalid" if not hard_valid or not quality_valid else (
+                "modified" if changed else "valid"
+            )
+            values["status"] = status
+            flags = quality.get("flags") if isinstance(quality, dict) else None
+            format_error = (
+                "思维链输出必须包含非空的 <think>...</think> 推理过程和最终答案"
+                if instruction_valid and output_valid and not reasoning_valid
+                else "DPO 输出必须包含不同的非空 Chosen 和 Rejected 回答"
+                if instruction_valid and not dpo_valid
+                else "Instruction 和 Output 不能为空"
+                if not instruction_valid or not output_valid
+                else None
+            )
+            values["error"] = ", ".join(str(flag) for flag in flags or []) or (
+                format_error
+                or ("quality validation failed" if status == "invalid" else None)
+            )
+            values["updated_at"] = utcnow()
+            assignments = ", ".join(f"{key}=%s" for key in values)
+            row = conn.execute(
+                f"""UPDATE data_process_results SET {assignments}
+                    WHERE id=%s AND task_id=%s RETURNING *""",
+                [*values.values(), result_id, task_id],
+            ).fetchone()
+            conn.execute(
+                """
+                UPDATE data_process_tasks
+                SET error_count=(
+                    SELECT COUNT(*) FROM data_process_results
+                    WHERE task_id=%s AND status='invalid'
+                ), updated_at=%s
+                WHERE id=%s
+                """,
+                (task_id, utcnow(), task_id),
+            )
+        return _decode_row(row) or {}
+
+    def replace_generated_result(
+        self,
+        task_id: str,
+        result_id: str,
+        replacement: dict[str, Any],
+        *,
+        expected_updated_at: str,
+    ) -> dict[str, Any]:
+        """用新模型结果原位替换失败项，并将新内容设为恢复基线。"""
+
+        with self.connect() as conn:
+            task = self._task_in_connection(conn, task_id, for_update=True)
+            if task["status"] != "completed" or task.get("workflow_step") != "results":
+                raise InvalidStateError("task is not editing generation results")
+            if task.get("results_confirmed"):
+                raise InvalidStateError("confirmed results cannot be regenerated")
+            if task.get("output_dataset_id"):
+                raise InvalidStateError("published results cannot be regenerated")
+
+            current = conn.execute(
+                """SELECT * FROM data_process_results
+                   WHERE id=%s AND task_id=%s FOR UPDATE""",
+                (result_id, task_id),
+            ).fetchone()
+            if not current:
+                raise NotFoundError("data process result not found")
+            if current.get("status") != "invalid":
+                raise InvalidStateError("only an invalid result can be regenerated")
+            current_updated_at = _serialize_value(current.get("updated_at"))
+            if expected_updated_at != current_updated_at:
+                raise ConflictError("data process result was modified by another request")
+
+            instruction = str(replacement.get("instruction") or "").strip()
+            input_text = str(replacement.get("input") or "").strip()
+            output = str(replacement.get("output") or "").strip()
+            chosen = str(replacement.get("chosen") or "").strip()
+            rejected = str(replacement.get("rejected") or "").strip()
+            quality_score = replacement.get("quality_score") or {}
+            if not instruction or not output or not bool(quality_score.get("is_valid")):
+                raise InvalidStateError("regenerated result did not pass quality validation")
+            if _task_output_type(task) == "reasoning" and not _reasoning_output_is_valid(output):
+                raise InvalidStateError("regenerated reasoning result has an invalid output format")
+            if _task_output_type(task) == "dpo" and not _dpo_fields_are_valid(replacement):
+                raise InvalidStateError("regenerated DPO result has invalid preference fields")
+
+            now = utcnow()
+            row = conn.execute(
+                """
+                UPDATE data_process_results
+                SET instruction=%s, input=%s, output=%s, chosen=%s, rejected=%s,
+                    original_instruction=%s, original_input=%s, original_output=%s,
+                    original_chosen=%s, original_rejected=%s,
+                    status='valid', error=NULL, quality_score=%s, updated_at=%s
+                WHERE id=%s AND task_id=%s
+                RETURNING *
+                """,
+                (
+                    instruction,
+                    input_text,
+                    output,
+                    chosen,
+                    rejected,
+                    instruction,
+                    input_text,
+                    output,
+                    chosen,
+                    rejected,
+                    json_dumps(quality_score),
+                    now,
+                    result_id,
+                    task_id,
+                ),
+            ).fetchone()
+            conn.execute(
+                """
+                UPDATE data_process_tasks
+                SET error_count=(
+                    SELECT COUNT(*) FROM data_process_results
+                    WHERE task_id=%s AND status='invalid'
+                ), updated_at=%s
+                WHERE id=%s
+                """,
+                (task_id, now, task_id),
+            )
+        return _decode_row(row) or {}

backend/app/modules/data_process/store/source_files.py

@@ -0,0 +1,321 @@
+"""数据处理存储层 - 源文件管理。"""
+
+from __future__ import annotations
+
+from typing import Any
+from collections.abc import Sequence
+
+import psycopg
+
+from .base import (
+    StoreBase,
+    utcnow,
+    new_id,
+    repeat_task_id,
+    json_dumps,
+    _json_value,
+    _decode_row,
+    _public_task,
+    _business_config,
+    _preview_config_value,
+    _preview_config_changed,
+    _preview_config_projection,
+    _normalized_preprocess_options,
+    _regeneration_marker,
+    _is_regeneration_prepared,
+    _task_output_type,
+    _task_reasoning_detail,
+    _reasoning_output_is_valid,
+    _dpo_fields_are_valid,
+    _source_storage_descriptor,
+    DataProcessStoreError,
+    NotFoundError,
+    ConflictError,
+    InvalidStateError,
+    EDITABLE_STATUSES,
+    ACTIVE_PREVIEW_STATUSES,
+    WORKFLOW_STEPS,
+    _REGENERATION_MARKER_KEY,
+    _REPEAT_SOURCE_TASK_KEY,
+    _REPEAT_REQUEST_KEY,
+)
+
+
+class SourceFilesMixin:
+    """源文件管理 Mixin。"""
+
+    def list_source_files(self, task_id: str) -> list[dict[str, Any]]:
+        self.get_task(task_id)
+        with self.connect() as conn:
+            rows = conn.execute(
+                """
+                SELECT id, task_id, storage_object_id, name, size_bytes, record_count,
+                       file_format, checksum_sha256, version_no, content_preview, metadata,
+                       tenant_id, project_id, created_by, created_at, updated_at
+                FROM data_process_source_files
+                WHERE task_id=%s AND deleted_at IS NULL
+                ORDER BY created_at, id
+                """,
+                (task_id,),
+            ).fetchall()
+        return [_decode_row(row) or {} for row in rows]
+
+    def add_source_file(
+        self,
+        task_id: str,
+        *,
+        name: str,
+        content: str,
+        raw_size: int,
+        checksum_sha256: str,
+        file_format: str,
+        record_count: int,
+        metadata: dict[str, Any] | None = None,
+        created_by: str | None = None,
+        source_file_id: str | None = None,
+        storage_object_id: str | None = None,
+    ) -> dict[str, Any]:
+        return self.add_source_files(
+            task_id,
+            [
+                {
+                    "name": name,
+                    "content": content,
+                    "raw_size": raw_size,
+                    "checksum_sha256": checksum_sha256,
+                    "file_format": file_format,
+                    "record_count": record_count,
+                    "metadata": metadata or {},
+                    "created_by": created_by,
+                    "id": source_file_id,
+                    "storage_object_id": storage_object_id,
+                }
+            ],
+        )[0]
+
+    def add_source_files(
+        self,
+        task_id: str,
+        files: Sequence[dict[str, Any]],
+    ) -> list[dict[str, Any]]:
+        """在同一事务中登记一个上传批次，任一文件失败则全部回滚。"""
+
+        if not files:
+            raise DataProcessStoreError("at least one source file is required")
+        now = utcnow()
+        created: list[dict[str, Any]] = []
+        try:
+            with self.connect() as conn:
+                task = self._task_in_connection(conn, task_id, for_update=True)
+                self._ensure_editable(task)
+                for payload in files:
+                    file_id = str(payload.get("id") or new_id("dpsf"))
+                    storage_object_id, metadata_payload = _source_storage_descriptor(
+                        payload,
+                        task_id,
+                        file_id,
+                    )
+                    row = conn.execute(
+                        """
+                        INSERT INTO data_process_source_files
+                        (id, task_id, storage_object_id, name, size_bytes, record_count,
+                         file_format, checksum_sha256, version_no, content, content_preview,
+                         metadata, tenant_id, project_id,
+                         created_by, created_at, updated_at)
+                        VALUES (%s, %s, %s, %s, %s, %s, %s, %s, 1, %s, %s, %s, %s,
+                                %s, %s, %s, %s)
+                        RETURNING id, task_id, storage_object_id, name, size_bytes, record_count,
+                                  file_format, checksum_sha256, version_no, content_preview, metadata,
+                                  tenant_id, project_id, created_by, created_at, updated_at
+                        """,
+                        (
+                            file_id,
+                            task_id,
+                            storage_object_id,
+                            payload["name"],
+                            payload["raw_size"],
+                            payload["record_count"],
+                            payload["file_format"],
+                            payload["checksum_sha256"],
+                            payload["content"],
+                            str(payload["content"])[:2000],
+                            json_dumps(metadata_payload),
+                            task.get("tenant_id"),
+                            task.get("project_id"),
+                            payload.get("created_by") or task.get("created_by"),
+                            now,
+                            now,
+                        ),
+                    ).fetchone()
+                    created.append(_decode_row(row) or {})
+                preview_row = conn.execute(
+                    """
+                    SELECT COUNT(*) AS count FROM data_process_preview_items
+                    WHERE task_id=%s
+                    """,
+                    (task_id,),
+                ).fetchone()
+                preview_count = int((preview_row or {}).get("count") or 0)
+                if _is_regeneration_prepared(task):
+                    conn.execute(
+                        """
+                        UPDATE data_process_tasks
+                        SET input_count=(
+                            SELECT COALESCE(SUM(record_count), 0)
+                            FROM data_process_source_files
+                            WHERE task_id=%s AND deleted_at IS NULL
+                        ), workflow_step='upload', preview_status='idle',
+                           preview_progress=0, preview_run_id=NULL,
+                           preview_failure_reason=NULL, preview_total_files=0,
+                           preview_completed_files=0, updated_at=%s
+                        WHERE id=%s
+                        """,
+                        (task_id, now, task_id),
+                    )
+                else:
+                    # 前端会只对本次新增的源文件构建预览，因此保留旧文件切片，
+                    # 但普通未发布任务的旧生成结果已经不再有效。
+                    conn.execute(
+                        "DELETE FROM data_process_results WHERE task_id=%s", (task_id,)
+                    )
+                    conn.execute(
+                        """
+                        UPDATE data_process_tasks
+                        SET status='pending', progress=%s, output_count=0, filtered_count=0,
+                            duplicate_count=0, error_count=0, failure_reason=NULL,
+                            generation_run_id=NULL, results_confirmed=FALSE,
+                            workflow_step='upload', preview_status='idle',
+                            preview_progress=0, preview_run_id=NULL,
+                            preview_failure_reason=NULL, preview_total_files=0,
+                            preview_completed_files=0,
+                            started_at=NULL, completed_at=NULL,
+                            input_count=(
+                                SELECT COALESCE(SUM(record_count), 0)
+                                FROM data_process_source_files
+                                WHERE task_id=%s AND deleted_at IS NULL
+                            ), updated_at=%s
+                        WHERE id=%s
+                        """,
+                        (20 if preview_count else 0, task_id, now, task_id),
+                    )
+        except psycopg.errors.UniqueViolation as exc:
+            raise ConflictError(
+                "the same source file content is already attached to this task"
+            ) from exc
+        return created
+
+    def get_source_file(
+        self, task_id: str, file_id: str, *, include_content: bool = True
+    ) -> dict[str, Any]:
+        # 先验证父任务仍然可见，避免软删除任务后通过已知文件 ID 读取正文。
+        self.get_task(task_id)
+        content_column = ", content" if include_content else ""
+        with self.connect() as conn:
+            row = conn.execute(
+                f"""
+                SELECT id, task_id, storage_object_id, name, size_bytes, record_count,
+                       file_format, checksum_sha256, version_no, content_preview, metadata,
+                       tenant_id, project_id, created_by, created_at, updated_at{content_column}
+                FROM data_process_source_files
+                WHERE id=%s AND task_id=%s AND deleted_at IS NULL
+                """,
+                (file_id, task_id),
+            ).fetchone()
+        if not row:
+            raise NotFoundError("source file not found")
+        return _decode_row(row) or {}
+
+    def source_content_window(
+        self, task_id: str, file_id: str, offset: int, limit: int
+    ) -> dict[str, Any]:
+        source_file = self.get_source_file(task_id, file_id, include_content=True)
+        content = str(source_file.pop("content", ""))
+        window = content[offset : offset + limit]
+        return {
+            "file": source_file,
+            "content": window,
+            "offset": offset,
+            "limit": limit,
+            "total_chars": len(content),
+            "has_more": offset + len(window) < len(content),
+        }
+
+    def source_content_lines(
+        self,
+        task_id: str,
+        file_id: str,
+        start_line: int,
+        line_count: int,
+    ) -> dict[str, Any]:
+        source_file = self.get_source_file(task_id, file_id, include_content=True)
+        content = str(source_file.pop("content", ""))
+        lines = content.splitlines(keepends=True)
+        start_index = min(len(lines), start_line - 1)
+        selected = lines[start_index : start_index + line_count]
+        end_line = start_index + len(selected)
+        return {
+            "file": source_file,
+            "content": "".join(selected),
+            "start_line": start_line,
+            "end_line": end_line,
+            "line_count": len(selected),
+            "total_lines": len(lines),
+            "has_more": end_line < len(lines),
+        }
+
+    def delete_source_file(self, task_id: str, file_id: str) -> None:
+        with self.connect() as conn:
+            task = self._task_in_connection(conn, task_id, for_update=True)
+            self._ensure_editable(task)
+            row = conn.execute(
+                """
+                UPDATE data_process_source_files
+                SET deleted_at=%s, updated_at=%s
+                WHERE id=%s AND task_id=%s AND deleted_at IS NULL
+                RETURNING id
+                """,
+                (utcnow(), utcnow(), file_id, task_id),
+            ).fetchone()
+            if not row:
+                raise NotFoundError("source file not found")
+            conn.execute(
+                "DELETE FROM data_process_preview_items WHERE source_file_id=%s", (file_id,)
+            )
+            now = utcnow()
+            if _is_regeneration_prepared(task):
+                conn.execute(
+                    """
+                    UPDATE data_process_tasks
+                    SET input_count=(SELECT COALESCE(SUM(record_count), 0)
+                                     FROM data_process_source_files
+                                     WHERE task_id=%s AND deleted_at IS NULL),
+                        workflow_step='upload', preview_status='idle',
+                        preview_progress=0, preview_run_id=NULL,
+                        preview_failure_reason=NULL, preview_total_files=0,
+                        preview_completed_files=0, updated_at=%s
+                    WHERE id=%s
+                    """,
+                    (task_id, now, task_id),
+                )
+            else:
+                conn.execute(
+                    "DELETE FROM data_process_results WHERE task_id=%s", (task_id,)
+                )
+                conn.execute(
+                    """
+                    UPDATE data_process_tasks
+                    SET status='pending', progress=0, output_count=0, filtered_count=0,
+                        duplicate_count=0, error_count=0, failure_reason=NULL,
+                        results_confirmed=FALSE,
+                        workflow_step='upload', preview_status='idle',
+                        preview_progress=0, preview_run_id=NULL,
+                        preview_failure_reason=NULL, preview_total_files=0,
+                        preview_completed_files=0,
+                        input_count=(SELECT COALESCE(SUM(record_count), 0)
+                                     FROM data_process_source_files
+                                     WHERE task_id=%s AND deleted_at IS NULL),
+                        updated_at=%s
+                    WHERE id=%s
+                    """,
+                    (task_id, now, task_id),
+                )

backend/app/modules/data_process/store/tasks.py

@@ -0,0 +1,871 @@
+"""数据处理存储层 - 任务管理。"""
+
+from __future__ import annotations
+
+from typing import Any
+from collections.abc import Sequence
+
+import psycopg
+
+from .base import (
+    StoreBase,
+    utcnow,
+    new_id,
+    repeat_task_id,
+    json_dumps,
+    _json_value,
+    _decode_row,
+    _public_task,
+    _business_config,
+    _preview_config_value,
+    _preview_config_changed,
+    _preview_config_projection,
+    _normalized_preprocess_options,
+    _regeneration_marker,
+    _is_regeneration_prepared,
+    _task_output_type,
+    _task_reasoning_detail,
+    _reasoning_output_is_valid,
+    _dpo_fields_are_valid,
+    _source_storage_descriptor,
+    _serialize_value,
+    NotFoundError,
+    ConflictError,
+    InvalidStateError,
+    EDITABLE_STATUSES,
+    ACTIVE_PREVIEW_STATUSES,
+    WORKFLOW_STEPS,
+    _REGENERATION_MARKER_KEY,
+    _REPEAT_SOURCE_TASK_KEY,
+    _REPEAT_REQUEST_KEY,
+    _INTERNAL_CONFIG_KEYS,
+)
+
+from ..algorithms import estimate_token_count
+
+class TasksMixin:
+    """任务管理 Mixin。"""
+
+    def list_tasks(
+        self,
+        *,
+        page: int = 1,
+        page_size: int = 20,
+        keyword: str | None = None,
+        status: str | None = None,
+        process_type: str | None = None,
+        tenant_id: str | None = None,
+        project_id: str | None = None,
+    ) -> dict[str, Any]:
+        clauses = ["task.deleted_at IS NULL"]
+        params: list[Any] = []
+        if keyword:
+            clauses.append("(task.name ILIKE %s OR COALESCE(task.description, '') ILIKE %s)")
+            pattern = f"%{keyword.strip()}%"
+            params.extend([pattern, pattern])
+        if status:
+            clauses.append("task.status = %s")
+            params.append(status)
+        if process_type:
+            clauses.append("task.process_type = %s")
+            params.append(process_type)
+        if tenant_id:
+            clauses.append("task.tenant_id = %s")
+            params.append(tenant_id)
+        if project_id:
+            clauses.append("task.project_id = %s")
+            params.append(project_id)
+        where = " AND ".join(clauses)
+        with self.connect() as conn:
+            total = conn.execute(
+                f"SELECT COUNT(*) AS count FROM data_process_tasks task WHERE {where}",
+                params,
+            ).fetchone()["count"]
+            rows = conn.execute(
+                f"""
+                SELECT task.*,
+                       (SELECT COUNT(*) FROM data_process_source_files source_file
+                        WHERE source_file.task_id=task.id
+                          AND source_file.deleted_at IS NULL) AS source_file_count
+                FROM data_process_tasks task
+                WHERE {where}
+                ORDER BY task.created_at DESC, task.id DESC
+                LIMIT %s OFFSET %s
+                """,
+                [*params, page_size, (page - 1) * page_size],
+            ).fetchall()
+        return {
+            "items": [_public_task(_decode_row(row)) for row in rows],
+            "total": int(total),
+            "page": page,
+            "page_size": page_size,
+        }
+
+    def create_task(self, payload: dict[str, Any]) -> dict[str, Any]:
+        task_id = new_id("dpt")
+        now = utcnow()
+        try:
+            with self.connect() as conn:
+                row = conn.execute(
+                    """
+                    INSERT INTO data_process_tasks
+                    (id, name, description, status, process_type, source_dataset_id, config,
+                     progress, results_confirmed, tenant_id, project_id, owner_id, created_by, updated_by,
+                     created_at, updated_at)
+                    VALUES (%s, %s, %s, 'pending', %s, %s, %s, 0, FALSE,
+                            %s, %s, %s, %s, %s, %s, %s)
+                    RETURNING *
+                    """,
+                    (
+                        task_id,
+                        payload["name"],
+                        payload.get("description") or "",
+                        payload["process_type"],
+                        payload.get("source_dataset_id"),
+                        json_dumps(_business_config(payload.get("config"))),
+                        payload.get("tenant_id"),
+                        payload.get("project_id"),
+                        payload.get("owner_id"),
+                        payload.get("created_by"),
+                        payload.get("created_by"),
+                        now,
+                        now,
+                    ),
+                ).fetchone()
+        except psycopg.errors.UniqueViolation as exc:
+            raise ConflictError("data process task name already exists") from exc
+        return _public_task(_decode_row(row)) or {}
+
+    @staticmethod
+    def _repeat_response(
+        conn: psycopg.Connection[dict[str, Any]],
+        row: dict[str, Any],
+        *,
+        source_task_id: str,
+        created: bool,
+    ) -> dict[str, Any]:
+        task_id = str(row["id"])
+        counts = conn.execute(
+            """
+            SELECT
+              (SELECT COUNT(*) FROM data_process_source_files
+               WHERE task_id=%s AND deleted_at IS NULL) AS source_file_count,
+              (SELECT COUNT(*) FROM data_process_preview_items
+               WHERE task_id=%s) AS preview_count
+            """,
+            (task_id, task_id),
+        ).fetchone() or {}
+        task = _public_task(_decode_row(row)) or {}
+        task["source_file_count"] = int(counts.get("source_file_count") or 0)
+        task["preview_count"] = int(counts.get("preview_count") or 0)
+        return {
+            "task": task,
+            "source_task_id": source_task_id,
+            "created": created,
+            "copied_source_file_count": task["source_file_count"],
+            "copied_preview_count": task["preview_count"],
+        }
+
+    def find_repeated_task(
+        self,
+        source_task_id: str,
+        request_id: str,
+    ) -> dict[str, Any] | None:
+        """查找同一幂等请求已创建的新任务。"""
+
+        task_id = repeat_task_id(source_task_id, request_id)
+        with self.connect() as conn:
+            row = conn.execute(
+                "SELECT * FROM data_process_tasks WHERE id=%s",
+                (task_id,),
+            ).fetchone()
+            if row is None:
+                return None
+            decoded = _decode_row(row) or {}
+            config = decoded.get("config") or {}
+            if (
+                config.get(_REPEAT_SOURCE_TASK_KEY) != source_task_id
+                or config.get(_REPEAT_REQUEST_KEY) != request_id
+            ):
+                raise ConflictError("再次生成请求与现有任务冲突")
+            if decoded.get("deleted_at"):
+                raise ConflictError("此次再次生成创建的任务已被删除，请重新发起")
+            return self._repeat_response(
+                conn,
+                row,
+                source_task_id=source_task_id,
+                created=False,
+            )
+
+    def repeat_task(
+        self,
+        source_task_id: str,
+        *,
+        expected_updated_at: str,
+        request_id: str,
+        file_copies: dict[str, dict[str, str]],
+    ) -> dict[str, Any]:
+        """复制已确认任务的配置、源文件和预览，结果与发布数据保持独立。"""
+
+        task_id = repeat_task_id(source_task_id, request_id)
+        now = utcnow()
+        try:
+            with self.connect() as conn:
+                existing = conn.execute(
+                    "SELECT * FROM data_process_tasks WHERE id=%s FOR UPDATE",
+                    (task_id,),
+                ).fetchone()
+                if existing is not None:
+                    decoded = _decode_row(existing) or {}
+                    config = decoded.get("config") or {}
+                    if (
+                        config.get(_REPEAT_SOURCE_TASK_KEY) != source_task_id
+                        or config.get(_REPEAT_REQUEST_KEY) != request_id
+                    ):
+                        raise ConflictError("再次生成请求与现有任务冲突")
+                    if decoded.get("deleted_at"):
+                        raise ConflictError("此次再次生成创建的任务已被删除，请重新发起")
+                    return self._repeat_response(
+                        conn,
+                        existing,
+                        source_task_id=source_task_id,
+                        created=False,
+                    )
+
+                source_task = self._task_in_connection(
+                    conn,
+                    source_task_id,
+                    for_update=True,
+                )
+                if (
+                    source_task.get("status") != "completed"
+                    or source_task.get("results_confirmed") is False
+                ):
+                    raise InvalidStateError("只有已完成并确认结果的任务可以再次生成")
+                if source_task.get("preview_status") in ACTIVE_PREVIEW_STATUSES:
+                    raise ConflictError("源任务仍在处理切分，暂时不能再次生成")
+                if expected_updated_at != _serialize_value(source_task.get("updated_at")):
+                    raise ConflictError("源任务已被其他操作修改，请刷新后重试")
+
+                source_files = conn.execute(
+                    """
+                    SELECT * FROM data_process_source_files
+                    WHERE task_id=%s AND deleted_at IS NULL
+                    ORDER BY created_at, id
+                    """,
+                    (source_task_id,),
+                ).fetchall()
+                source_file_ids = {str(row["id"]) for row in source_files}
+                if source_file_ids != set(file_copies):
+                    raise ConflictError("源文件快照已变化，请刷新后重试")
+                previews = conn.execute(
+                    """
+                    SELECT * FROM data_process_preview_items
+                    WHERE task_id=%s
+                    ORDER BY source_file_id NULLS LAST, source_start NULLS LAST,
+                             created_at, id
+                    """,
+                    (source_task_id,),
+                ).fetchall()
+                if not previews:
+                    raise InvalidStateError("源任务没有可用于再次生成的切分结果")
+
+                suffix = f"（再次生成-{task_id[-6:]}）"
+                base_name = str(source_task.get("name") or "数据处理任务")
+                repeated_name = f"{base_name[: max(1, 150 - len(suffix))]}{suffix}"
+                repeated_config = _business_config(source_task.get("config") or {})
+                repeated_config[_REPEAT_SOURCE_TASK_KEY] = source_task_id
+                repeated_config[_REPEAT_REQUEST_KEY] = request_id
+                input_count = sum(int(row.get("record_count") or 0) for row in source_files)
+                task_row = conn.execute(
+                    """
+                    INSERT INTO data_process_tasks
+                    (id, name, description, status, process_type, source_dataset_id,
+                     output_dataset_id, config, progress, input_count, output_count,
+                     filtered_count, duplicate_count, error_count, failure_reason,
+                     generation_run_id, results_confirmed, workflow_step,
+                     preview_status, preview_progress, preview_run_id,
+                     preview_failure_reason, preview_total_files,
+                     preview_completed_files, tenant_id, project_id, owner_id,
+                     approval_status, created_by, updated_by, created_at, updated_at)
+                    VALUES
+                    (%s, %s, %s, 'pending', %s, %s, NULL, %s, 20, %s, 0,
+                     0, 0, 0, NULL, NULL, FALSE, 'preview', 'completed', 100,
+                     NULL, NULL, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
+                    RETURNING *
+                    """,
+                    (
+                        task_id,
+                        repeated_name,
+                        source_task.get("description") or "",
+                        source_task["process_type"],
+                        source_task.get("source_dataset_id"),
+                        json_dumps(repeated_config),
+                        input_count,
+                        len(source_files),
+                        len(source_files),
+                        source_task.get("tenant_id"),
+                        source_task.get("project_id"),
+                        source_task.get("owner_id"),
+                        source_task.get("approval_status") or "not_required",
+                        source_task.get("created_by"),
+                        source_task.get("created_by"),
+                        now,
+                        now,
+                    ),
+                ).fetchone()
+
+                file_id_map: dict[str, str] = {}
+                for source in source_files:
+                    old_file_id = str(source["id"])
+                    copy = file_copies[old_file_id]
+                    new_file_id = str(copy["id"])
+                    storage_object_id, metadata = _source_storage_descriptor(
+                        {
+                            "storage_object_id": copy["storage_object_id"],
+                            "metadata": _json_value(source.get("metadata"), {}),
+                        },
+                        task_id,
+                        new_file_id,
+                    )
+                    file_id_map[old_file_id] = new_file_id
+                    conn.execute(
+                        """
+                        INSERT INTO data_process_source_files
+                        (id, task_id, storage_object_id, name, size_bytes, record_count,
+                         file_format, checksum_sha256, version_no, content,
+                         content_preview, metadata, tenant_id, project_id, created_by,
+                         created_at, updated_at)
+                        VALUES (%s, %s, %s, %s, %s, %s, %s, %s, 1, %s, %s, %s,
+                                %s, %s, %s, %s, %s)
+                        """,
+                        (
+                            new_file_id,
+                            task_id,
+                            storage_object_id,
+                            source["name"],
+                            source.get("size_bytes") or 0,
+                            source.get("record_count") or 0,
+                            source.get("file_format"),
+                            source["checksum_sha256"],
+                            source.get("content") or "",
+                            source.get("content_preview"),
+                            json_dumps(metadata),
+                            source_task.get("tenant_id"),
+                            source_task.get("project_id"),
+                            source.get("created_by") or source_task.get("created_by"),
+                            now,
+                            now,
+                        ),
+                    )
+
+                for preview in previews:
+                    old_source_file_id = preview.get("source_file_id")
+                    conn.execute(
+                        """
+                        INSERT INTO data_process_preview_items
+                        (id, task_id, source_file_id, original_content, edited_content,
+                         source_start, source_end, source_start_line, source_end_line,
+                         token_count, status, quality_score, created_at, updated_at)
+                        VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s,
+                                %s, %s)
+                        """,
+                        (
+                            new_id("dpp"),
+                            task_id,
+                            file_id_map.get(str(old_source_file_id))
+                            if old_source_file_id
+                            else None,
+                            preview.get("original_content") or "",
+                            preview.get("edited_content") or "",
+                            preview.get("source_start"),
+                            preview.get("source_end"),
+                            preview.get("source_start_line"),
+                            preview.get("source_end_line"),
+                            max(0, int(preview.get("token_count") or 0)),
+                            preview.get("status") or "original",
+                            json_dumps(_json_value(preview.get("quality_score"), {})),
+                            now,
+                            now,
+                        ),
+                    )
+                return self._repeat_response(
+                    conn,
+                    task_row or {},
+                    source_task_id=source_task_id,
+                    created=True,
+                )
+        except psycopg.errors.UniqueViolation as exc:
+            raise ConflictError("再次生成任务名称或请求发生冲突，请重试") from exc
+
+    def get_task(self, task_id: str, *, for_update: bool = False) -> dict[str, Any]:
+        lock = " FOR UPDATE" if for_update else ""
+        with self.connect() as conn:
+            if for_update:
+                row = conn.execute(
+                    f"SELECT * FROM data_process_tasks WHERE id=%s AND deleted_at IS NULL{lock}",
+                    (task_id,),
+                ).fetchone()
+            else:
+                row = conn.execute(
+                    """
+                    SELECT task.*,
+                           (SELECT COUNT(*) FROM data_process_source_files source
+                            WHERE source.task_id=task.id AND source.deleted_at IS NULL)
+                             AS source_file_count,
+                           (SELECT COUNT(*) FROM data_process_preview_items preview
+                            WHERE preview.task_id=task.id) AS preview_count,
+                           (SELECT COALESCE(json_agg(json_build_object(
+                               'id', dataset.id,
+                               'name', dataset.name,
+                               'type', dataset.type,
+                               'count', dataset.count,
+                               'dataset_split', CASE dataset.type
+                                   WHEN 'train' THEN 'train'
+                                   WHEN 'val' THEN 'validation'
+                                   WHEN 'test' THEN 'test'
+                                   ELSE NULL
+                               END
+                           ) ORDER BY CASE dataset.type
+                               WHEN 'train' THEN 1 WHEN 'val' THEN 2 WHEN 'test' THEN 3 ELSE 4 END), '[]'::json)
+                            FROM datasets dataset
+                            WHERE dataset.source='task'
+                              AND dataset.deleted_at IS NULL
+                              AND (
+                                  dataset.source_task_id=task.id
+                                  OR (dataset.source_task_id IS NULL AND dataset.task_id=task.id)
+                              ))
+                             AS output_datasets,
+                           CASE
+                             WHEN task.started_at IS NOT NULL AND task.completed_at IS NOT NULL
+                             THEN EXTRACT(EPOCH FROM (task.completed_at - task.started_at))
+                             ELSE NULL
+                           END AS duration_seconds
+                    FROM data_process_tasks task
+                    WHERE task.id=%s AND task.deleted_at IS NULL
+                    """,
+                    (task_id,),
+                ).fetchone()
+        if not row:
+            raise NotFoundError("data process task not found")
+        return _public_task(_decode_row(row)) or {}
+
+    def _task_in_connection(
+        self,
+        conn: psycopg.Connection[dict[str, Any]],
+        task_id: str,
+        *,
+        for_update: bool = False,
+    ) -> dict[str, Any]:
+        lock = " FOR UPDATE" if for_update else ""
+        row = conn.execute(
+            f"SELECT * FROM data_process_tasks WHERE id=%s AND deleted_at IS NULL{lock}",
+            (task_id,),
+        ).fetchone()
+        if not row:
+            raise NotFoundError("data process task not found")
+        return _decode_row(row) or {}
+
+    @staticmethod
+    def _ensure_editable(task: dict[str, Any]) -> None:
+        if task["status"] not in EDITABLE_STATUSES:
+            raise InvalidStateError(f"task cannot be edited while status is {task['status']}")
+        if task.get("preview_status") in ACTIVE_PREVIEW_STATUSES:
+            raise InvalidStateError("task cannot be edited while preview is running")
+        if task.get("output_dataset_id") and not _is_regeneration_prepared(task):
+            raise InvalidStateError("published task cannot be edited")
+
+    def update_workflow_step(self, task_id: str, workflow_step: str) -> dict[str, Any]:
+        """独立保存向导位置，不触发配置或结果失效逻辑。"""
+
+        if workflow_step not in WORKFLOW_STEPS:
+            raise ValueError("invalid data process workflow step")
+        with self.connect() as conn:
+            row = conn.execute(
+                """
+                UPDATE data_process_tasks
+                SET workflow_step=%s, updated_at=%s
+                WHERE id=%s AND deleted_at IS NULL
+                RETURNING *
+                """,
+                (workflow_step, utcnow(), task_id),
+            ).fetchone()
+        if not row:
+            raise NotFoundError("data process task not found")
+        return _public_task(_decode_row(row)) or {}
+
+    def recover_legacy_aborted_regeneration(self, task_id: str) -> dict[str, Any]:
+        """恢复旧版在真正开始生成前误删的上一轮结果。
+
+        旧实现会在 ``POST /regenerate`` 时立即把已发布任务置为 pending、
+        清空结果并解除输出指针。三个已发布数据集仍是独立完整产物，因此只在
+        这个特征完全匹配时，使用其记录恢复结果和任务状态。该操作幂等，不会
+        触碰正常的新建待生成任务或已经开始的新一轮生成。
+        """
+
+        with self.connect() as conn:
+            task = self._task_in_connection(conn, task_id, for_update=True)
+            if (
+                task.get("status") != "pending"
+                or task.get("generation_run_id")
+                or task.get("output_dataset_id")
+                or int(task.get("output_count") or 0) != 0
+            ):
+                return {"recovered": False, "result_count": 0}
+
+            result_count = int(
+                (
+                    conn.execute(
+                        "SELECT COUNT(*) AS count FROM data_process_results WHERE task_id=%s",
+                        (task_id,),
+                    ).fetchone()
+                    or {}
+                ).get("count")
+                or 0
+            )
+            if result_count:
+                return {"recovered": False, "result_count": result_count}
+
+            datasets = conn.execute(
+                """
+                SELECT id, type, count, created_at
+                FROM datasets
+                WHERE source='task' AND deleted_at IS NULL
+                  AND (
+                      source_task_id=%s
+                      OR (source_task_id IS NULL AND task_id=%s)
+                  )
+                ORDER BY CASE type
+                    WHEN 'train' THEN 1 WHEN 'val' THEN 2 WHEN 'test' THEN 3 ELSE 4
+                END, created_at, id
+                """,
+                (task_id, task_id),
+            ).fetchall()
+            train_dataset = next(
+                (dataset for dataset in datasets if dataset.get("type") == "train"),
+                None,
+            )
+            if not train_dataset:
+                return {"recovered": False, "result_count": 0}
+
+            dataset_ids = [str(dataset["id"]) for dataset in datasets]
+            records = conn.execute(
+                """
+                SELECT id, dataset_id, line_no, split, instruction, input, output,
+                       raw, status, source_result_id, preview_item_id, created_at
+                FROM dataset_records
+                WHERE dataset_id = ANY(%s)
+                ORDER BY created_at, dataset_id, line_no NULLS LAST, id
+                """,
+                (dataset_ids,),
+            ).fetchall()
+            if not records:
+                return {"recovered": False, "result_count": 0}
+
+            preview_rows = conn.execute(
+                "SELECT id FROM data_process_preview_items WHERE task_id=%s",
+                (task_id,),
+            ).fetchall()
+            preview_ids = {str(row["id"]) for row in preview_rows}
+            used_result_ids: set[str] = set()
+            recovered_count = 0
+            for record in records:
+                raw = _json_value(record.get("raw"), {})
+                raw = raw if isinstance(raw, dict) else {}
+                candidate_id = str(
+                    record.get("source_result_id")
+                    or raw.get("source_result_id")
+                    or ""
+                )
+                result_id = (
+                    candidate_id
+                    if candidate_id and candidate_id not in used_result_ids
+                    else new_id("dpr")
+                )
+                used_result_ids.add(result_id)
+                candidate_preview_id = str(
+                    record.get("preview_item_id")
+                    or raw.get("preview_item_id")
+                    or ""
+                )
+                preview_item_id = (
+                    candidate_preview_id if candidate_preview_id in preview_ids else None
+                )
+                instruction = str(record.get("instruction") or raw.get("instruction") or "")
+                input_text = str(record.get("input") or raw.get("input") or "")
+                chosen = str(raw.get("chosen") or "")
+                rejected = str(raw.get("rejected") or "")
+                output = str(
+                    record.get("output") or raw.get("output") or chosen or ""
+                )
+                split = str(record.get("split") or raw.get("split") or "") or None
+                status = str(record.get("status") or "valid")
+                if status not in {"valid", "modified", "invalid"}:
+                    status = "valid"
+                created_at = record.get("created_at") or utcnow()
+                conn.execute(
+                    """
+                    INSERT INTO data_process_results
+                    (id, task_id, preview_item_id, instruction, input, output,
+                     chosen, rejected, original_instruction, original_input,
+                     original_output, original_chosen, original_rejected, status,
+                     error, split, quality_score, created_at, updated_at)
+                    VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s,
+                            %s, %s, NULL, %s, '{}', %s, %s)
+                    """,
+                    (
+                        result_id,
+                        task_id,
+                        preview_item_id,
+                        instruction,
+                        input_text,
+                        output,
+                        chosen,
+                        rejected,
+                        instruction,
+                        input_text,
+                        output,
+                        chosen,
+                        rejected,
+                        status,
+                        split,
+                        created_at,
+                        created_at,
+                    ),
+                )
+                conn.execute(
+                    """
+                    UPDATE dataset_records
+                    SET source_result_id=%s, preview_item_id=%s
+                    WHERE id=%s
+                    """,
+                    (result_id, preview_item_id, record["id"]),
+                )
+                recovered_count += 1
+
+            now = utcnow()
+            conn.execute(
+                """
+                UPDATE data_process_tasks
+                SET status='completed', progress=100, output_dataset_id=%s,
+                    output_count=%s, filtered_count=0, duplicate_count=0,
+                    error_count=(SELECT COUNT(*) FROM data_process_results
+                                 WHERE task_id=%s AND status='invalid'),
+                    failure_reason=NULL, results_confirmed=TRUE,
+                    workflow_step='results', updated_at=%s
+                WHERE id=%s
+                """,
+                (
+                    train_dataset["id"],
+                    recovered_count,
+                    task_id,
+                    now,
+                    task_id,
+                ),
+            )
+            return {"recovered": True, "result_count": recovered_count}
+
+    def update_task(self, task_id: str, payload: dict[str, Any]) -> dict[str, Any]:
+        allowed = {
+            "name",
+            "description",
+            "process_type",
+            "source_dataset_id",
+        }
+        values: dict[str, Any] = {key: value for key, value in payload.items() if key in allowed}
+        if not values and payload.get("config") is None:
+            return self.get_task(task_id)
+        try:
+            with self.connect() as conn:
+                task = self._task_in_connection(conn, task_id, for_update=True)
+                self._ensure_editable(task)
+                regeneration_prepared = _is_regeneration_prepared(task)
+                if regeneration_prepared and any(
+                    key in payload and payload.get(key) != task.get(key)
+                    for key in ("process_type", "source_dataset_id")
+                ):
+                    raise InvalidStateError(
+                        "process type and source dataset cannot change during regeneration"
+                    )
+                if payload.get("config") is not None:
+                    next_config = _business_config(payload["config"])
+                    current_config = dict(task.get("config") or {})
+                    for key in _INTERNAL_CONFIG_KEYS:
+                        if key in current_config:
+                            next_config[key] = current_config[key]
+                    values["config"] = json_dumps(next_config)
+                invalidates_results = (
+                    ("config" in payload and payload.get("config") != task.get("config"))
+                    or (
+                        "process_type" in payload
+                        and payload.get("process_type") != task.get("process_type")
+                    )
+                    or (
+                        "source_dataset_id" in payload
+                        and payload.get("source_dataset_id") != task.get("source_dataset_id")
+                    )
+                )
+                if invalidates_results and not regeneration_prepared:
+                    values.update(
+                        {
+                            "status": "pending",
+                            "progress": 0,
+                            "output_count": 0,
+                            "filtered_count": 0,
+                            "duplicate_count": 0,
+                            "error_count": 0,
+                            "failure_reason": None,
+                            "generation_run_id": None,
+                            "results_confirmed": False,
+                            "preview_status": "idle",
+                            "preview_progress": 0,
+                            "preview_run_id": None,
+                            "preview_failure_reason": None,
+                            "preview_total_files": 0,
+                            "preview_completed_files": 0,
+                            "started_at": None,
+                            "completed_at": None,
+                        }
+                    )
+                    conn.execute("DELETE FROM data_process_results WHERE task_id=%s", (task_id,))
+                    conn.execute(
+                        "DELETE FROM data_process_preview_items WHERE task_id=%s", (task_id,)
+                    )
+                if (
+                    "process_type" in payload
+                    and payload.get("process_type") != task.get("process_type")
+                ):
+                    conn.execute(
+                        """
+                        UPDATE data_process_source_files
+                        SET deleted_at=%s, updated_at=%s
+                        WHERE task_id=%s AND deleted_at IS NULL
+                        """,
+                        (utcnow(), utcnow(), task_id),
+                    )
+                    values["input_count"] = 0
+                values["updated_at"] = utcnow()
+                assignments = ", ".join(f"{key}=%s" for key in values)
+                row = conn.execute(
+                    f"UPDATE data_process_tasks SET {assignments} WHERE id=%s RETURNING *",
+                    [*values.values(), task_id],
+                ).fetchone()
+        except psycopg.errors.UniqueViolation as exc:
+            raise ConflictError("data process task name already exists") from exc
+        return _public_task(_decode_row(row)) or {}
+
+    def prepare_regeneration(
+        self,
+        task_id: str,
+        payload: dict[str, Any],
+    ) -> dict[str, Any]:
+        """非破坏性地保存重新生成配置。
+
+        准备阶段保留任务当前状态、结果、切片及已发布数据集。真正开始
+        生成时，才在同一事务内切换运行状态并清理上一轮结果。
+        """
+
+        # 先修复曾被旧版 prepare 提前清空的任务，再建立新的非破坏性草稿标记。
+        self.recover_legacy_aborted_regeneration(task_id)
+        try:
+            with self.connect() as conn:
+                task = self._task_in_connection(conn, task_id, for_update=True)
+                if task["status"] == "running":
+                    raise ConflictError("running task cannot be prepared for regeneration")
+                if task.get("preview_status") in ACTIVE_PREVIEW_STATUSES:
+                    raise ConflictError("running preview cannot be prepared for regeneration")
+
+                current_updated_at = _serialize_value(task.get("updated_at"))
+                if payload["expected_updated_at"] != current_updated_at:
+                    raise ConflictError("data process task was modified by another request")
+
+                process_type = str(payload["process_type"])
+                if process_type != str(task["process_type"]):
+                    raise InvalidStateError("process_type cannot be changed during regeneration")
+
+                current_config = dict(task.get("config") or {})
+                next_config = _business_config(payload.get("config"))
+                for key in (_REPEAT_SOURCE_TASK_KEY, _REPEAT_REQUEST_KEY):
+                    if key in current_config:
+                        next_config[key] = current_config[key]
+                preview_invalidated = _preview_config_changed(
+                    process_type,
+                    current_config,
+                    next_config,
+                )
+                now = utcnow()
+                next_config[_REGENERATION_MARKER_KEY] = {
+                    "prepared": True,
+                    "preview_invalidated": preview_invalidated,
+                    "prepared_at": now,
+                }
+                # 002 迁移前发布的数据集只有 task_id。先补齐新关联字段，保证
+                # 解除任务输出指针后，详情和后续重新发布仍能定位原来的三份数据集。
+                conn.execute(
+                    """
+                    UPDATE datasets
+                    SET source_task_id=%s, updated_at=%s
+                    WHERE source='task' AND source_task_id IS NULL AND task_id=%s
+                      AND deleted_at IS NULL
+                    """,
+                    (task_id, now, task_id),
+                )
+                published_row = conn.execute(
+                    """
+                    SELECT EXISTS(
+                        SELECT 1 FROM datasets
+                        WHERE source='task' AND deleted_at IS NULL
+                          AND (
+                              source_task_id=%s
+                              OR (source_task_id IS NULL AND task_id=%s)
+                          )
+                    ) AS exists
+                    """,
+                    (task_id, task_id),
+                ).fetchone()
+                published_outputs_preserved = bool(task.get("output_dataset_id")) or bool(
+                    published_row and published_row.get("exists")
+                )
+
+                row = conn.execute(
+                    """
+                    UPDATE data_process_tasks
+                    SET name=%s, description=%s, config=%s, updated_at=%s
+                    WHERE id=%s
+                    RETURNING *
+                    """,
+                    (
+                        payload["name"],
+                        payload.get("description") or "",
+                        json_dumps(next_config),
+                        now,
+                        task_id,
+                    ),
+                ).fetchone()
+        except psycopg.errors.UniqueViolation as exc:
+            raise ConflictError("data process task name already exists") from exc
+        return {
+            "task": _public_task(_decode_row(row)) or {},
+            "preview_invalidated": preview_invalidated,
+            "published_outputs_preserved": published_outputs_preserved,
+        }
+
+    def delete_task(self, task_id: str, *, deleted_by: str | None = None) -> None:
+        with self.connect() as conn:
+            self._task_in_connection(conn, task_id, for_update=True)
+            now = utcnow()
+            conn.execute(
+                """
+                UPDATE data_process_tasks
+                SET status=CASE WHEN status='running' THEN 'stopped' ELSE status END,
+                    generation_run_id=NULL,
+                    preview_status=CASE
+                        WHEN preview_status IN ('queued', 'running') THEN 'cancelled'
+                        ELSE preview_status
+                    END,
+                    preview_run_id=NULL,
+                    deleted_at=%s, deleted_by=%s, updated_at=%s
+                WHERE id=%s
+                """,
+                (now, deleted_by, now, task_id),
+            )

backend/app/modules/gpu/__init__.py

@@ -0,0 +1 @@
+"""GPU assignment management module."""

backend/app/modules/gpu/router.py

@@ -0,0 +1,77 @@
+"""GPU 算力分配管理路由。"""
+from __future__ import annotations
+
+from typing import Any
+
+from fastapi import APIRouter, Body, Depends, Request
+
+from app.api.v1.endpoints.platform import ok, fail
+from app.core.auth import get_current_user, is_admin
+from app.db.platform_store import get_platform_store
+
+router = APIRouter(prefix="/compute", tags=["gpu-assignment"])
+
+
+def _actor_id(request: Request) -> str | None:
+    auth = request.headers.get("Authorization", "")
+    token = auth.replace("Bearer ", "").strip()
+    if token.startswith("platform-token-"):
+        return token[len("platform-token-"):]
+    return None
+
+
+@router.get("/gpu-assignments")
+def list_assignments(current_user: dict = Depends(get_current_user)) -> dict[str, Any]:
+    """查看全部分配关系（仅 admin）。"""
+    if not is_admin(current_user):
+        raise fail(403, "admin permission required")
+    return ok(get_platform_store().gpu_assignments())
+
+
+@router.post("/gpu-assignments")
+def assign_gpus(
+    payload: dict[str, Any] = Body(...),
+    request: Request = None,
+    current_user: dict = Depends(get_current_user),
+) -> dict[str, Any]:
+    """批量分配 GPU（仅 admin）。body: { assignments: [{ node_id, gpu_index, user_id }] }"""
+    if not is_admin(current_user):
+        raise fail(403, "admin permission required")
+    assignments = payload.get("assignments") or []
+    if not assignments:
+        raise fail(400, "assignments 不能为空")
+    actor = _actor_id(request) if request else None
+    result = get_platform_store().assign_gpus(assignments, assigned_by=actor)
+    get_platform_store().record_audit(
+        action="gpu.assign",
+        actor_id=actor,
+        target_type="gpu",
+        detail=f"count={len(assignments)}",
+    )
+    return ok(result)
+
+
+@router.delete("/gpu-assignments/{assignment_id}")
+def unassign_gpu(
+    assignment_id: str,
+    request: Request = None,
+    current_user: dict = Depends(get_current_user),
+) -> dict[str, Any]:
+    """撤销 GPU 分配（仅 admin）。"""
+    if not is_admin(current_user):
+        raise fail(403, "admin permission required")
+    get_platform_store().unassign_gpu(assignment_id)
+    actor = _actor_id(request) if request else None
+    get_platform_store().record_audit(
+        action="gpu.unassign",
+        actor_id=actor,
+        target_type="gpu",
+        target_id=assignment_id,
+    )
+    return ok({"deleted": assignment_id})
+
+
+@router.get("/my-gpus")
+def my_gpus(current_user: dict = Depends(get_current_user)) -> dict[str, Any]:
+    """查看当前用户可用的 GPU 列表。"""
+    return ok(get_platform_store().gpu_assignments_for_user(current_user["id"]))

backend/app/modules/resource/router.py

@@ -1,10 +1,12 @@
 from __future__ import annotations
 
-from fastapi import APIRouter, Body, Request
+from fastapi import APIRouter, Body, Request, Depends
 from typing import Any
 
 from app.api.v1.endpoints.platform import ok, fail
 from app.db.platform_store import get_platform_store
+from app.core.auth import get_current_user, has_resource_access, is_admin
+from app.core.audit import audit_log, AuditActions
 
 router = APIRouter(prefix="/resources", tags=["resource"])
 
@@ -16,20 +18,36 @@ def _actor(request: Request) -> str | None:
 
 
 @router.get("/{resource_type}/{resource_id}/acl")
-def get_acl(resource_type: str, resource_id: str) -> dict[str, Any]:
+def get_acl(resource_type: str, resource_id: str, current_user: dict = Depends(get_current_user)) -> dict[str, Any]:
     """查询资源 ACL，返回按主体分组的权限列表。"""
+    if not has_resource_access(resource_type, resource_id, current_user, "read"):
+        raise fail(403, "no permission to access resource ACL")
     return ok(get_platform_store().resource_acl(resource_type, resource_id))
 
 
 @router.put("/{resource_type}/{resource_id}/acl")
+@audit_log(
+    action=AuditActions.GRANT_ACL,
+    target_type="",
+    detail_template="设置资源授权: {resource_type}/{resource_id}",
+)
 def set_acl(
     resource_type: str,
     resource_id: str,
     payload: dict[str, Any] = Body(...),
     request: Request = None,
+    current_user: dict = Depends(get_current_user),
 ) -> dict[str, Any]:
     """设置资源 ACL，body: { entries: [{ subject_type, subject_id, permissions: [] }] }"""
+    if not is_admin(current_user) and not has_resource_access(resource_type, resource_id, current_user, "write"):
+        raise fail(403, "only resource owner or admin can update ACL")
     entries = payload.get("entries") or []
+    allowed = {"read", "write", "execute", "download", "delete", "admin"}
+    for entry in entries:
+        if entry.get("principal_type") not in {"user", "role"} or not entry.get("principal_id"):
+            raise fail(400, "invalid ACL principal")
+        if any(permission not in allowed for permission in entry.get("permissions") or []):
+            raise fail(400, "invalid ACL permission")
     result = get_platform_store().set_resource_acl(resource_type, resource_id, entries)
     get_platform_store().record_audit(
         action="resource.acl.set",

backend/app/modules/storage/__init__.py

@@ -0,0 +1 @@
+"""Central object storage integration."""

backend/app/modules/storage/minio_store.py

@@ -0,0 +1,68 @@
+from __future__ import annotations
+
+from datetime import timedelta
+from functools import lru_cache
+from io import BytesIO
+from typing import Any
+
+from minio import Minio
+from minio.error import S3Error
+
+from app.core.config import get_settings
+
+
+class ObjectStorageError(RuntimeError):
+    pass
+
+
+class MinioObjectStorage:
+    def __init__(self) -> None:
+        settings = get_settings()
+        endpoint = settings.minio_endpoint.replace("http://", "").replace("https://", "").rstrip("/")
+        self.client = Minio(endpoint, access_key=settings.minio_access_key, secret_key=settings.minio_secret_key, secure=settings.minio_secure)
+        self.bucket = settings.minio_bucket
+
+    def _ensure_enabled(self) -> None:
+        if not get_settings().minio_enabled:
+            raise ObjectStorageError("MinIO object storage is disabled")
+
+    def ensure_bucket(self) -> None:
+        self._ensure_enabled()
+        try:
+            if not self.client.bucket_exists(self.bucket):
+                self.client.make_bucket(self.bucket)
+        except S3Error as exc:
+            raise ObjectStorageError(str(exc)) from exc
+
+    def presigned_put(self, object_key: str, expires_seconds: int = 3600) -> str:
+        self._ensure_enabled()
+        self.ensure_bucket()
+        return self.client.presigned_put_object(self.bucket, object_key, expires=timedelta(seconds=expires_seconds))
+
+    def presigned_get(self, object_key: str, expires_seconds: int = 3600) -> str:
+        self._ensure_enabled()
+        self.ensure_bucket()
+        return self.client.presigned_get_object(self.bucket, object_key, expires=timedelta(seconds=expires_seconds))
+
+    def stat(self, object_key: str) -> dict[str, Any]:
+        self._ensure_enabled()
+        self.ensure_bucket()
+        try:
+            result = self.client.stat_object(self.bucket, object_key)
+            return {"object_key": object_key, "byte_size": result.size, "etag": result.etag, "last_modified": result.last_modified.isoformat() if result.last_modified else None}
+        except S3Error as exc:
+            raise ObjectStorageError(str(exc)) from exc
+
+    def put_bytes(self, object_key: str, content: bytes, content_type: str = "application/octet-stream") -> dict[str, Any]:
+        self._ensure_enabled()
+        self.ensure_bucket()
+        try:
+            result = self.client.put_object(self.bucket, object_key, BytesIO(content), len(content), content_type=content_type)
+            return {"bucket": self.bucket, "object_key": object_key, "etag": result.etag, "byte_size": len(content)}
+        except S3Error as exc:
+            raise ObjectStorageError(str(exc)) from exc
+
+
+@lru_cache
+def get_object_storage() -> MinioObjectStorage:
+    return MinioObjectStorage()

backend/app/modules/system/router.py

@@ -1,9 +1,10 @@
 from __future__ import annotations
 
-from fastapi import APIRouter, Body, Query, Request
+from fastapi import APIRouter, Body, Query, Request, Depends
 from fastapi.responses import StreamingResponse
 
 from app.db.platform_store import ALL_PERMISSIONS, get_platform_store
+from app.core.auth import get_current_user, is_admin
 
 
 router = APIRouter(prefix="/system", tags=["system"])
@@ -32,13 +33,13 @@ def record_visit(payload: dict = Body(...), request: Request = None) -> dict:
 
 
 @router.get("/permissions/codes")
-def permission_codes() -> dict:
+def permission_codes(current_user: dict = Depends(get_current_user)) -> dict:
     """返回平台权限码清单（权限码接口）。"""
     return {"code": 0, "message": "ok", "data": {"codes": ALL_PERMISSIONS}}
 
 
 @router.get("/permissions")
-def permissions_overview() -> dict:
+def permissions_overview(current_user: dict = Depends(get_current_user)) -> dict:
     """返回权限码清单与角色定义。"""
     store = get_platform_store()
     return {
@@ -59,8 +60,12 @@ def audit_logs(
     end_time: str | None = Query(default=None, description="ISO8601 结束时间"),
     limit: int = Query(default=50, ge=1, le=200),
     offset: int = Query(default=0, ge=0),
+    current_user: dict = Depends(get_current_user),
 ) -> dict:
     """审计日志查询：按租户/项目/操作人/动作/目标类型/时间范围分页过滤。"""
+    if not is_admin(current_user):
+        from app.api.v1.endpoints.platform import fail
+        raise fail(403, "admin permission required")
     store = get_platform_store()
     result = store.audit_logs(
         tenant_id=tenant_id,
@@ -85,8 +90,12 @@ def audit_logs_export(
     target_type: str | None = Query(default=None, description="目标类型"),
     start_time: str | None = Query(default=None, description="ISO8601 起始时间"),
     end_time: str | None = Query(default=None, description="ISO8601 结束时间"),
+    current_user: dict = Depends(get_current_user),
 ) -> StreamingResponse:
     """审计日志导出：返回 CSV 流，与应用查询相同的过滤条件。"""
+    if not is_admin(current_user):
+        from app.api.v1.endpoints.platform import fail
+        raise fail(403, "admin permission required")
     store = get_platform_store()
     result = store.audit_logs(
         tenant_id=tenant_id,
@@ -113,3 +122,143 @@ def audit_logs_export(
         media_type="text/csv",
         headers={"Content-Disposition": "attachment; filename=audit_logs.csv"},
     )
+
+
+# ===================== 操作日志 =====================
+
+@router.get("/operation-logs")
+def operation_logs(
+    user_id: str | None = Query(default=None, description="按用户 ID 筛选"),
+    module: str | None = Query(default=None, description="按模块筛选: fine-tune/model-eval/model-inference/dataset/data-convert/model-manage"),
+    action: str | None = Query(default=None, description="按动作筛选: create/start/stop/delete/upload/convert/merge"),
+    status: str | None = Query(default=None, description="按状态筛选: success/failure（不传则查全部）"),
+    keyword: str | None = Query(default=None, description="关键字搜索报错信息(error_message)"),
+    start_time: str | None = Query(default=None, description="ISO8601 起始时间"),
+    end_time: str | None = Query(default=None, description="ISO8601 结束时间"),
+    limit: int = Query(default=50, ge=1, le=200),
+    offset: int = Query(default=0, ge=0),
+    current_user: dict = Depends(get_current_user),
+) -> dict:
+    """操作日志查询：按用户/模块/动作/状态/关键字/时间范围分页过滤。"""
+    if not is_admin(current_user):
+        from app.api.v1.endpoints.platform import fail
+        raise fail(403, "admin permission required")
+    store = get_platform_store()
+    conditions = []
+    params: list = []
+    if user_id:
+        conditions.append("user_id = %s")
+        params.append(user_id)
+    if module:
+        conditions.append("module = %s")
+        params.append(module)
+    if action:
+        conditions.append("action = %s")
+        params.append(action)
+    if status:
+        conditions.append("status = %s")
+        params.append(status)
+    if keyword:
+        conditions.append("(error_message ILIKE %s OR error_type ILIKE %s)")
+        params.append(f"%{keyword}%")
+        params.append(f"%{keyword}%")
+    if start_time:
+        conditions.append("create_time >= %s")
+        params.append(start_time)
+    if end_time:
+        conditions.append("create_time <= %s")
+        params.append(end_time)
+    where = " WHERE " + " AND ".join(conditions) if conditions else ""
+    with store.connect() as conn:
+        rows = conn.execute(
+            f"SELECT * FROM operation_logs{where} ORDER BY create_time DESC LIMIT %s OFFSET %s",
+            tuple(params + [limit, offset]),
+        ).fetchall()
+        total = conn.execute(f"SELECT COUNT(*) FROM operation_logs{where}", tuple(params)).fetchone()[0]
+    return {"code": 0, "message": "ok", "data": {"items": [dict(r) for r in rows], "total": total}}
+
+
+@router.get("/operation-logs/stats")
+def operation_logs_stats(
+    start_time: str | None = Query(default=None, description="ISO8601 起始时间"),
+    end_time: str | None = Query(default=None, description="ISO8601 结束时间"),
+    current_user: dict = Depends(get_current_user),
+) -> dict:
+    """操作日志统计：总操作数、成功数、失败数、失败率、各模块失败分布、最近错误列表。"""
+    if not is_admin(current_user):
+        from app.api.v1.endpoints.platform import fail
+        raise fail(403, "admin permission required")
+    store = get_platform_store()
+    conditions = []
+    params: list = []
+    if start_time:
+        conditions.append("create_time >= %s")
+        params.append(start_time)
+    if end_time:
+        conditions.append("create_time <= %s")
+        params.append(end_time)
+    where = " WHERE " + " AND ".join(conditions) if conditions else ""
+    failure_where = where + " AND status = 'failure'" if where else " WHERE status = 'failure'"
+
+    with store.connect() as conn:
+        # 总计
+        row = conn.execute(
+            f"SELECT status, COUNT(*) as cnt FROM operation_logs{where} GROUP BY status", tuple(params)
+        ).fetchall()
+        total_count = 0
+        success_count = 0
+        failure_count = 0
+        for r in row:
+            total_count += r["cnt"]
+            if r["status"] == "success":
+                success_count = r["cnt"]
+            elif r["status"] == "failure":
+                failure_count = r["cnt"]
+        failure_rate = round(failure_count / total_count * 100, 2) if total_count > 0 else 0
+
+        # 各模块失败数
+        module_stats = conn.execute(
+            f"SELECT module, COUNT(*) as cnt FROM operation_logs{failure_where} GROUP BY module ORDER BY cnt DESC",
+            tuple(params),
+        ).fetchall()
+
+        # 各异常类型分布
+        error_type_stats = conn.execute(
+            f"SELECT error_type, COUNT(*) as cnt FROM operation_logs{failure_where} AND error_type IS NOT NULL GROUP BY error_type ORDER BY cnt DESC LIMIT 10",
+            tuple(params),
+        ).fetchall()
+
+        # 最近的 10 条错误
+        recent_errors = conn.execute(
+            f"SELECT * FROM operation_logs{failure_where} ORDER BY create_time DESC LIMIT 10",
+            tuple(params),
+        ).fetchall()
+
+    return {
+        "code": 0,
+        "message": "ok",
+        "data": {
+            "total": total_count,
+            "success": success_count,
+            "failure": failure_count,
+            "failure_rate": failure_rate,
+            "module_failures": [{"module": r["module"], "count": r["cnt"]} for r in module_stats],
+            "error_types": [{"type": r["error_type"], "count": r["cnt"]} for r in error_type_stats],
+            "recent_errors": [dict(r) for r in recent_errors],
+        },
+    }
+
+
+@router.get("/operation-logs/modules")
+def operation_log_modules(current_user: dict = Depends(get_current_user)) -> dict:
+    """返回操作日志中出现的模块列表（用于筛选下拉框）。"""
+    if not is_admin(current_user):
+        from app.api.v1.endpoints.platform import fail
+        raise fail(403, "admin permission required")
+    store = get_platform_store()
+    with store.connect() as conn:
+        rows = conn.execute(
+            "SELECT DISTINCT module FROM operation_logs WHERE module IS NOT NULL ORDER BY module"
+        ).fetchall()
+    modules = [{"value": r["module"], "label": r["module"]} for r in rows]
+    return {"code": 0, "message": "ok", "data": modules}

backend/app/modules/tenant/router.py

@@ -68,7 +68,7 @@ def update_tenant(tenant_id: str, payload: dict[str, Any] = Body(...), request:
 def set_quota(tenant_id: str, payload: dict[str, Any] = Body(...), request: Request = None) -> dict[str, Any]:
     store = get_platform_store()
     try:
-        tenant = store.set_tenant_quota(tenant_id, payload.get("quota", {}))
+        tenant = store.set_tenant_quota(tenant_id, payload)
     except KeyError:
         raise fail(404, "tenant not found")
     store.record_audit(

backend/app/schemas/data_process.py

@@ -2,6 +2,7 @@ from __future__ import annotations
 
 from enum import StrEnum
 from typing import Any, Literal
+from urllib.parse import parse_qs, urlsplit
 
 from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
 
@@ -15,6 +16,31 @@ def _config_value(config: dict[str, Any], snake_name: str, camel_name: str, defa
 
 
 def _validate_process_config(config: dict[str, Any]) -> None:
+    output_type = _config_value(config, "output_type", "outputType", "standard")
+    if output_type not in {"standard", "reasoning", "dpo"}:
+        raise ValueError("output_type must be one of: standard, reasoning, dpo")
+
+    source_mode = _config_value(config, "source_mode", "sourceMode", "local")
+    if source_mode not in {"local", "external"}:
+        raise ValueError("source_mode must be one of: local, external")
+    external_source = _config_value(config, "external_source", "externalSource", None)
+    if external_source is not None:
+        if not isinstance(external_source, dict):
+            raise ValueError("external_source must be an object")
+        if any(
+            key.lower() in {"password", "secret", "token", "api_key"}
+            for key in external_source
+        ):
+            raise ValueError("external_source must not persist credentials")
+        external_url = str(external_source.get("url") or "").strip()
+        if external_url:
+            parsed_external_url = urlsplit(external_url)
+            sensitive_query_keys = {"password", "secret", "token", "api_key", "user", "username"}
+            if parsed_external_url.username or parsed_external_url.password or (
+                set(parse_qs(parsed_external_url.query)) & sensitive_query_keys
+            ):
+                raise ValueError("external_source URL must not contain credentials")
+
     chunk_method = _config_value(config, "chunk_method", "chunkMethod", "layout_hybrid")
     if not isinstance(chunk_method, str) or chunk_method not in {
         "layout_hybrid",
@@ -303,6 +329,9 @@ class ExternalSourceRequest(BaseModel):
     username: str | None = Field(default=None, max_length=150)
     password: str | None = Field(default=None, max_length=500)
     limit: int = Field(default=1000, ge=1, le=100_000)
+    connect_timeout_seconds: int = Field(default=5, ge=1, le=30)
+    statement_timeout_seconds: int = Field(default=30, ge=1, le=300)
+    ssl_mode: Literal["disable", "prefer", "require", "verify-ca", "verify-full"] = "prefer"
 
 
 class ExternalPullRequest(ExternalSourceRequest):
@@ -324,6 +353,8 @@ class ResultUpdate(BaseModel):
     instruction: str | None = None
     input: str | None = None
     output: str | None = None
+    chosen: str | None = None
+    rejected: str | None = None
     expected_updated_at: str | None = None
 
 
@@ -374,7 +405,7 @@ class PublishRequest(BaseModel):
     dataset_type: Literal["train", "test", "eval", "val", "other"] = "train"
     storage_type: Literal["local"] = "local"
     split: DatasetSplit = Field(default_factory=DatasetSplit)
-    format: Literal["alpaca_jsonl", "jsonl"] = "alpaca_jsonl"
+    format: Literal["alpaca_jsonl", "jsonl", "dpo"] = "alpaca_jsonl"
     description: str = ""
 
     @field_validator("dataset_name")

backend/app/workers/compute_poller.py

@@ -21,8 +21,10 @@ async def run_compute_poller() -> None:
     while True:
         try:
             result = await poll_compute_jobs_once()
-            if result["synced"] or result["failed"]:
+            if result["failed"]:
-                logger.info("compute jobs polled", extra={"result": result})
+                logger.warning("compute polling reported failures", extra={"result": result})
+            elif result["synced"]:
+                logger.debug("compute jobs synchronized", extra={"result": result})
         except asyncio.CancelledError:
             logger.info("compute poller stopped")
             raise

backend/requirements.txt

@@ -8,6 +8,7 @@ psycopg-pool>=3.2.1
 alembic>=1.13.1
 redis>=5.0.4
 httpx>=0.27.0
+minio>=7.2.7
 PyJWT>=2.8.0
 passlib[bcrypt]>=1.7.4
 python-dotenv>=1.0.1

backend/tests/test_data_convert_security.py

@@ -0,0 +1,57 @@
+"""data_convert 模块安全回归测试：输出文件名路径穿越与鉴权。
+
+- ``output_filename`` 必须通过白名单校验，阻断 ``../``、``/``、``\\`` 及控制字符，
+  否则转换结果可被写出到存储根目录之外（任意文件读写/删除）。
+- 所有 data_convert 路由必须挂载 ``get_current_user`` 鉴权依赖。
+"""
+from __future__ import annotations
+
+import pytest
+from fastapi import HTTPException
+
+from app.core.auth import get_current_user
+from app.modules.data_convert.router import _safe_output_filename, router
+
+
+def test_safe_output_filename_defaults() -> None:
+    assert _safe_output_filename(None) == "converted-data.jsonl"
+    assert _safe_output_filename("") == "converted-data.jsonl"
+
+
+def test_safe_output_filename_valid() -> None:
+    assert _safe_output_filename("converted-data.jsonl") == "converted-data.jsonl"
+    assert _safe_output_filename("my-data.v1.jsonl") == "my-data.v1.jsonl"
+    assert _safe_output_filename("  报告.jsonl  ") == "报告.jsonl"
+
+
+@pytest.mark.parametrize(
+    "bad",
+    [
+        "../../etc/passwd",
+        "../x.jsonl",
+        "a/b.jsonl",
+        r"a\b.jsonl",
+        "a\\b.jsonl",
+        "..",
+        ".",
+        "x\x00.jsonl",
+        "x\n.jsonl",
+        "x\t.jsonl",
+    ],
+)
+def test_safe_output_filename_rejects_traversal(bad: str) -> None:
+    with pytest.raises(HTTPException):
+        _safe_output_filename(bad)
+
+
+def test_all_data_convert_routes_require_auth() -> None:
+    for route in router.routes:
+        node = getattr(route, "dependant", None)
+        assert node is not None, f"route {route.path} has no dependency graph"
+        stack = list(node.dependencies)
+        calls: list = []
+        while stack:
+            dep = stack.pop()
+            stack.extend(getattr(dep, "dependencies", []))
+            calls.append(getattr(dep, "call", None))
+        assert get_current_user in calls, f"route {route.path} is missing get_current_user auth"

backend/tests/test_data_process_algorithms.py

@@ -654,7 +654,10 @@ def test_office_zip_bomb_and_invalid_pdf_are_rejected_before_parsing() -> None:
     blank_writer = PdfWriter()
     blank_writer.add_blank_page(width=612, height=792)
     blank_writer.write(blank_pdf)
-    with pytest.raises(ValueError, match="scanned PDF requires OCR"):
+    with pytest.raises(
+        ValueError,
+        match="scanned or image-only PDF files are not supported",
+    ):
         parse_text_content(blank_pdf.getvalue(), filename="scanned.pdf")
 
     aes_pdf_without_open_password = parse_text_content(

backend/tests/test_data_process_api.py

@@ -1840,6 +1840,43 @@ def test_external_source_never_returns_fake_success(tmp_path: Path) -> None:
     assert response.json()["detail"]["code"] == 501
 
 
+def test_external_source_mode_belongs_to_step_three_structured_task(tmp_path: Path) -> None:
+    client, _, _ = make_client(tmp_path)
+    local_task_id = client.post(
+        "/modelTF/data-process",
+        json={
+            "name": "本地结构化任务",
+            "process_type": "structured",
+            "config": {"source_mode": "local"},
+        },
+    ).json()["data"]["id"]
+    rejected = client.post(
+        f"/modelTF/data-process/{local_task_id}/external/test",
+        json={"type": "mysql", "url": "mysql://db.example/test"},
+    )
+    assert rejected.status_code == 409
+
+    external_task_id = client.post(
+        "/modelTF/data-process",
+        json={
+            "name": "外部结构化任务",
+            "process_type": "structured",
+            "config": {"source_mode": "external"},
+        },
+    ).json()["data"]["id"]
+    accepted_as_external = client.post(
+        f"/modelTF/data-process/{external_task_id}/external/test",
+        json={"type": "mysql", "url": "mysql://db.example/test"},
+    )
+    assert accepted_as_external.status_code == 501
+
+    local_upload = client.post(
+        f"/modelTF/data-process/{external_task_id}/source-files",
+        files={"files": ("records.jsonl", b'{"id":1}\n', "application/jsonl")},
+    )
+    assert local_upload.status_code == 409
+
+
 def test_regenerate_endpoint_prepares_an_existing_published_task(tmp_path: Path) -> None:
     client, store, _ = make_client(tmp_path)
     task_id = client.post(

backend/tests/test_data_process_generation.py

@@ -35,6 +35,8 @@ def test_generate_model_records_uses_prompt_auth_and_stable_split() -> None:
         assert "客户反馈页面加载慢" in payload["messages"][1]["content"]
         assert "你正在生成标准监督微调问答数据" in payload["messages"][0]["content"]
         assert "禁止输出分析、推理过程" in payload["messages"][0]["content"]
+        assert "真实用户自然提出的问题" in payload["messages"][0]["content"]
+        assert "模板化开头" in payload["messages"][0]["content"]
         return httpx.Response(
             200,
             json={
@@ -88,6 +90,127 @@ def test_generate_model_records_uses_prompt_auth_and_stable_split() -> None:
     assert progress_updates == [(1, 1)]
 
 
+def test_generate_model_records_falls_back_to_rich_default_prompt() -> None:
+    captured: dict[str, str] = {}
+
+    def handler(request: httpx.Request) -> httpx.Response:
+        payload = json.loads(request.content)
+        captured["system"] = payload["messages"][0]["content"]
+        captured["user"] = payload["messages"][1]["content"]
+        return httpx.Response(
+            200,
+            json={
+                "choices": [
+                    {
+                        "message": {
+                            "content": json.dumps(
+                                {
+                                    "items": [
+                                        {
+                                            "instruction": "平台如何控制不同角色的菜单可见性？",
+                                            "input": "",
+                                            "output": "按角色分配权限。",
+                                        }
+                                    ]
+                                },
+                                ensure_ascii=False,
+                            )
+                        }
+                    }
+                ]
+            },
+        )
+
+    client = httpx.Client(transport=httpx.MockTransport(handler))
+    records = generate_model_records(
+        [{"id": "preview-1", "edited_content": "平台按角色分配菜单权限。"}],
+        model={
+            "name": "Qwen",
+            "online_model_name": "qwen-plus",
+            "api_url": "model.example",
+        },
+        config={},
+        task_id="task-fallback",
+        split={"train": 100, "validation": 0, "test": 0},
+        qa_pairs_per_item=1,
+        client=client,
+    )
+
+    assert len(records) == 1
+    assert records[0]["status"] == "valid"
+    assert "数据生成专家" in captured["system"]
+    assert "真实用户自然提出的问题" in captured["system"]
+    assert "平台按角色分配菜单权限。" in captured["user"]
+
+
+def test_generate_model_records_builds_native_dpo_pair() -> None:
+    def handler(request: httpx.Request) -> httpx.Response:
+        payload = json.loads(request.content)
+        system_prompt = payload["messages"][0]["content"]
+        assert '"chosen"' in system_prompt
+        assert '"rejected"' in system_prompt
+        assert "直接偏好优化" in system_prompt
+        return httpx.Response(
+            200,
+            json={
+                "choices": [{
+                    "message": {
+                        "content": json.dumps({
+                            "items": [{
+                                "instruction": "系统如何处理扫描 PDF？",
+                                "input": "",
+                                "chosen": "仅在没有文本层时调用 OCR，并保留页码。",
+                                "rejected": "所有 PDF 都重复执行 OCR。",
+                            }],
+                        }, ensure_ascii=False),
+                    },
+                }],
+            },
+        )
+
+    records = generate_model_records(
+        [{"id": "preview-dpo", "edited_content": "扫描 PDF 缺少文本层时执行 OCR。"}],
+        model={"name": "model", "api_url": "https://model.example/v1"},
+        config={"output_type": "dpo", "generation_retries": 0},
+        task_id="task-dpo",
+        split={"train": 100, "validation": 0, "test": 0},
+        qa_pairs_per_item=1,
+        client=httpx.Client(transport=httpx.MockTransport(handler)),
+    )
+
+    assert records[0]["status"] == "valid"
+    assert records[0]["chosen"] == "仅在没有文本层时调用 OCR,并保留页码。"
+    assert records[0]["rejected"] == "所有 PDF 都重复执行 OCR。"
+    assert records[0]["output"] == records[0]["chosen"]
+
+
+def test_generate_model_records_rejects_equal_dpo_pair() -> None:
+    def handler(_request: httpx.Request) -> httpx.Response:
+        return httpx.Response(
+            200,
+            json={"choices": [{"message": {"content": json.dumps({
+                "items": [{
+                    "instruction": "问题",
+                    "chosen": "相同回答",
+                    "rejected": "相同回答",
+                }],
+            }, ensure_ascii=False)}}]},
+        )
+
+    records = generate_model_records(
+        [{"id": "preview-dpo-invalid", "edited_content": "来源"}],
+        model={"name": "model", "api_url": "https://model.example/v1"},
+        config={"output_type": "dpo", "generation_retries": 0},
+        task_id="task-dpo-invalid",
+        split={"train": 100, "validation": 0, "test": 0},
+        qa_pairs_per_item=1,
+        client=httpx.Client(transport=httpx.MockTransport(handler)),
+    )
+
+    assert records[0]["status"] == "invalid"
+    assert "chosen equals rejected" in records[0]["error"]
+
+
 def test_minimax_m3_uses_split_reasoning_and_completion_token_budget() -> None:
     requests: list[dict[str, object]] = []
 

backend/tests/test_data_process_migration.py

@@ -37,6 +37,10 @@ def test_runtime_migration_fails_fast_on_incompatible_schema() -> None:
     for value in ("idle", "queued", "running", "completed", "failed", "cancelled"):
         assert f"'{value}'" in sql
     assert "CREATE TABLE IF NOT EXISTS data_process_results" in sql
+    assert "ADD COLUMN IF NOT EXISTS chosen TEXT NOT NULL DEFAULT ''" in sql
+    assert "ADD COLUMN IF NOT EXISTS rejected TEXT NOT NULL DEFAULT ''" in sql
+    assert "ADD COLUMN IF NOT EXISTS original_chosen TEXT" in sql
+    assert "ADD COLUMN IF NOT EXISTS original_rejected TEXT" in sql
     assert sql.count("BEGIN;") == 1
     assert sql.rstrip().endswith("COMMIT;")
 

backend/tests/test_data_process_store.py

@@ -1335,6 +1335,39 @@ def test_publish_rejects_invalid_reasoning_output_format() -> None:
         )
 
 
+def test_publish_dpo_writes_chosen_and_rejected_jsonl() -> None:
+    conn = _PublishConnection(
+        [
+            {
+                "id": "result-dpo",
+                "status": "valid",
+                "instruction": "如何处理扫描 PDF？",
+                "input": "",
+                "output": "仅在无文本层时执行 OCR。",
+                "chosen": "仅在无文本层时执行 OCR。",
+                "rejected": "所有 PDF 都执行 OCR。",
+                "preview_item_id": "preview-dpo",
+            }
+        ]
+    )
+
+    published = _PublishStore(conn, {"output_type": "dpo"}).publish(
+        "task-dpo",
+        {
+            "dataset_name": "偏好数据",
+            "storage_type": "local",
+            "format": "dpo",
+            "split": {"train": 100, "validation": 0, "test": 0},
+        },
+    )
+
+    record = conn.records[0]["raw"]
+    assert record["chosen"] == "仅在无文本层时执行 OCR。"
+    assert record["rejected"] == "所有 PDF 都执行 OCR。"
+    assert "output" not in record
+    assert published["datasets"][0]["metadata"]["format"] == "dpo"
+
+
 def test_source_storage_descriptor_accepts_owned_local_and_legacy_db_references() -> None:
     task_id = "dpt_task"
     source_file_id = "dpsf_source"

backend/tests/test_docs_security.py

@@ -0,0 +1,76 @@
+"""Swagger / ReDoc / OpenAPI 文档路由安全开关测试。
+
+生产环境（APP_ENV=prod）默认关闭 /docs、/redoc、/openapi.json，
+避免未授权访问泄露 API 结构；本地开发环境默认开放，可用 ENABLE_DOCS 覆盖。
+"""
+from __future__ import annotations
+
+import pytest
+
+from app.core.config import docs_kwargs, get_settings
+
+
+@pytest.fixture(autouse=True)
+def _reset_settings_cache():
+    """每次测试前后清空 get_settings 的 lru_cache，避免环境变量互相污染。"""
+    get_settings.cache_clear()
+    yield
+    get_settings.cache_clear()
+
+
+def test_docs_kwargs_enabled() -> None:
+    assert docs_kwargs(True) == {}
+
+
+def test_docs_kwargs_disabled() -> None:
+    assert docs_kwargs(False) == {"docs_url": None, "redoc_url": None, "openapi_url": None}
+
+
+def test_docs_disabled_by_default_in_prod(monkeypatch) -> None:
+    monkeypatch.delenv("ENABLE_DOCS", raising=False)
+    monkeypatch.setenv("APP_ENV", "prod")
+    assert get_settings().enable_docs is False
+
+
+def test_docs_enabled_by_default_outside_prod(monkeypatch) -> None:
+    monkeypatch.delenv("ENABLE_DOCS", raising=False)
+    monkeypatch.setenv("APP_ENV", "local")
+    assert get_settings().enable_docs is True
+
+
+def test_docs_env_override_enables_in_prod(monkeypatch) -> None:
+    monkeypatch.setenv("ENABLE_DOCS", "true")
+    monkeypatch.setenv("APP_ENV", "prod")
+    assert get_settings().enable_docs is True
+
+
+def test_docs_env_override_disables_outside_prod(monkeypatch) -> None:
+    monkeypatch.setenv("ENABLE_DOCS", "false")
+    monkeypatch.setenv("APP_ENV", "local")
+    assert get_settings().enable_docs is False
+
+
+def test_create_app_disables_docs_in_prod(monkeypatch, tmp_path) -> None:
+    pytest.importorskip("fastapi")
+    from app.main import create_app
+
+    monkeypatch.delenv("ENABLE_DOCS", raising=False)
+    monkeypatch.setenv("APP_ENV", "prod")
+    monkeypatch.setenv("LOG_DIR", str(tmp_path))
+    app = create_app()
+    assert app.docs_url is None
+    assert app.redoc_url is None
+    assert app.openapi_url is None
+
+
+def test_create_app_enables_docs_outside_prod(monkeypatch, tmp_path) -> None:
+    pytest.importorskip("fastapi")
+    from app.main import create_app
+
+    monkeypatch.delenv("ENABLE_DOCS", raising=False)
+    monkeypatch.setenv("APP_ENV", "local")
+    monkeypatch.setenv("LOG_DIR", str(tmp_path))
+    app = create_app()
+    assert app.docs_url == "/docs"
+    assert app.redoc_url == "/redoc"
+    assert app.openapi_url == "/openapi.json"

backend/tests/test_document_chunking.py

@@ -1,10 +1,14 @@
 from __future__ import annotations
 
+import sys
+from types import ModuleType, SimpleNamespace
+
 from llama_index.core.embeddings import MockEmbedding
 
 from app.modules.data_process.document_chunking import (
     DocumentChunk,
     _compact_with_offsets,
+    _document_converter,
     _project_layout_span,
     chunk_fixed_text,
     chunk_semantic_text,
@@ -42,6 +46,48 @@ def test_semantic_splitter_uses_llamaindex_and_reapplies_maximum_size() -> None:
     assert "".join(chunk.original_content for chunk in chunks) == text
 
 
+def test_layout_converter_disables_ocr(monkeypatch) -> None:
+    class FakePipelineOptions:
+        def __init__(self) -> None:
+            self.do_ocr = True
+
+    class FakePdfFormatOption:
+        def __init__(self, *, pipeline_options) -> None:
+            self.pipeline_options = pipeline_options
+
+    class FakeDocumentConverter:
+        def __init__(self, *, format_options) -> None:
+            self.format_options = format_options
+
+    docling_module = ModuleType("docling")
+    docling_module.__path__ = []
+    document_converter_module = ModuleType("docling.document_converter")
+    document_converter_module.DocumentConverter = FakeDocumentConverter
+    document_converter_module.PdfFormatOption = FakePdfFormatOption
+    datamodel_module = ModuleType("docling.datamodel")
+    datamodel_module.__path__ = []
+    base_models_module = ModuleType("docling.datamodel.base_models")
+    base_models_module.InputFormat = SimpleNamespace(PDF="pdf")
+    pipeline_options_module = ModuleType("docling.datamodel.pipeline_options")
+    pipeline_options_module.PdfPipelineOptions = FakePipelineOptions
+    for name, module in {
+        "docling": docling_module,
+        "docling.document_converter": document_converter_module,
+        "docling.datamodel": datamodel_module,
+        "docling.datamodel.base_models": base_models_module,
+        "docling.datamodel.pipeline_options": pipeline_options_module,
+    }.items():
+        monkeypatch.setitem(sys.modules, name, module)
+
+    _document_converter.cache_clear()
+    try:
+        converter = _document_converter()
+        options = converter.format_options["pdf"].pipeline_options
+        assert options.do_ocr is False
+    finally:
+        _document_converter.cache_clear()
+
+
 def test_layout_projection_ignores_layout_whitespace_but_keeps_source_lines() -> None:
     source = "标题\n第一条 这是正文。\n第二条 后续正文。"
     compact_source, offsets = _compact_with_offsets(source)

backend/tests/test_governance.py

@@ -138,6 +138,14 @@ class FakePlatformStore:
                     return dict(u)
         return None
 
+    def create_session(self, user_id: str) -> dict[str, Any]:
+        import secrets
+        sid = secrets.token_hex(16)
+        return {"session_id": sid, "user_id": user_id}
+
+    def finish_session(self, session_id: str) -> None:
+        pass
+
     def users(self) -> list[dict[str, Any]]:
         return [dict(u) for u in self._users]
 
@@ -392,12 +400,21 @@ class FakePlatformStore:
     def tasks(self) -> list[dict[str, Any]]:
         return self._tasks
 
+    def eval_tasks(self) -> list[dict[str, Any]]:
+        return []
+
     def compute_nodes(self) -> list[dict[str, Any]]:
         return self._compute_nodes
 
     def gpus(self) -> list[dict[str, Any]]:
         return self._gpus
 
+    def compare_tasks(self) -> list[dict[str, Any]]:
+        return []
+
+    def trained_models(self) -> list[dict[str, Any]]:
+        return []
+
     def system_info(self) -> dict[str, Any]:
         return {"cpu": {}, "memory": {}}
 

compute/api/main.py

@@ -11,17 +11,21 @@ import time
 from pathlib import Path
 from typing import Any
 
+import httpx
+
 from fastapi import FastAPI, File, Form, HTTPException, Query, Request, UploadFile
 from fastapi.responses import FileResponse, JSONResponse, StreamingResponse
 
 from compute.agent.process_manager import ProcessManager
+from compute.api.security import docs_kwargs
 from compute.engines.llama_factory.adapter import build_command, parse_log_line, prepare_runtime_files
 from compute.engines.llama_factory.inference import get_inference_session
 
 
 def create_app() -> FastAPI:
-    app = FastAPI(title="YG Fine-Tune Compute API")
+    app = FastAPI(title="YG Fine-Tune Compute API", **docs_kwargs())
     jobs: dict[str, dict[str, Any]] = {}
+    cache_locks: dict[str, asyncio.Lock] = {}
     route_prefix = os.getenv("MODELTF_ROUTE_PREFIX", "/modelTF").rstrip("/") or "/modelTF"
     process_manager = ProcessManager(os.getenv("TRAINING_LOG_ROOT", "/opt/yg-ft/logs/training"))
 
@@ -510,13 +514,27 @@ def create_app() -> FastAPI:
         llama_factory_home = Path(os.getenv("LLAMA_FACTORY_HOME", "/app/LLaMA-Factory"))
         gpu_items = gpu_resources()
         torch_cuda = torch_cuda_status()
+        storage_available = False
+        storage_error = ""
+        try:
+            data_root.mkdir(parents=True, exist_ok=True)
+            probe = data_root / ".yg-ft-storage-healthcheck"
+            probe.write_text(host_id(), encoding="utf-8")
+            storage_available = probe.read_text(encoding="utf-8").strip() == host_id()
+            probe.unlink(missing_ok=True)
+        except Exception as exc:  # noqa: BLE001 - health endpoint must remain available
+            storage_error = str(exc)
         return {
-            "status": "ok",
+            "status": "ok" if storage_available else "storage_unavailable",
             "api_version": "v1",
             "compute_host_id": os.getenv("COMPUTE_HOST_ID", "unknown"),
             "app_callback_enabled": os.getenv("ENABLE_APP_CALLBACK", "false").lower() == "true",
             "data_root": str(data_root),
             "data_root_exists": data_root.exists(),
+            "storage_mode": os.getenv("STORAGE_MODE", "minio-cache"),
+            "storage_available": storage_available,
+            "storage_error": storage_error,
+            "storage_root": str(data_root),
             "model_root": os.getenv("YG_FT_MODEL_ROOT", str(data_root / "models")),
             "dataset_root": str(dataset_root),
             "dataset_root_exists": dataset_root.exists(),
@@ -531,7 +549,7 @@ def create_app() -> FastAPI:
             "nvidia_gpu_count": len(gpu_items),
             "torch_cuda": torch_cuda,
             "gpu_discovery_endpoint": f"{route_prefix}/compute/resources/gpus",
-            "capabilities": ["gpu_discovery", "torch_cuda_diagnostics", "llama_factory", "file_gateway", "job_polling"],
+            "capabilities": ["gpu_discovery", "torch_cuda_diagnostics", "llama_factory", "file_gateway", "job_polling", "storage_health"],
         }
 
     @app.get(f"{route_prefix}/v1/compute/jobs")
@@ -603,6 +621,30 @@ def create_app() -> FastAPI:
             )
         return {"root": root, "base_path": str(base), "relative_path": relative_path, "items": items}
 
+    @app.post(f"{route_prefix}/compute/files/upload-to-url")
+    async def upload_file_to_url(payload: dict[str, Any]) -> dict[str, Any]:
+        """Upload one node-local artifact to a Backend-issued presigned URL."""
+        source = Path(str(payload.get("path") or "")).resolve()
+        data_root = Path(os.getenv("YG_FT_DATA_ROOT", "/data/yg-ft")).resolve()
+        if not _path_inside(data_root, source) or not source.is_file():
+            raise HTTPException(status_code=400, detail="artifact path must be an existing file inside YG_FT_DATA_ROOT")
+        upload_url = str(payload.get("upload_url") or "")
+        if not upload_url:
+            raise HTTPException(status_code=400, detail="upload_url is required")
+        digest = hashlib.sha256()
+        byte_size = 0
+        try:
+            async with httpx.AsyncClient(timeout=httpx.Timeout(900, connect=30)) as client:
+                with source.open("rb") as handle:
+                    content = handle.read()
+                    digest.update(content)
+                    byte_size = len(content)
+                response = await client.put(upload_url, content=content, headers={"Content-Type": str(payload.get("content_type") or "application/octet-stream")})
+                response.raise_for_status()
+        except Exception as exc:
+            raise HTTPException(status_code=502, detail=f"artifact upload failed: {exc}") from exc
+        return {"status": "available", "path": str(source), "byte_size": byte_size, "checksum_sha256": digest.hexdigest(), "object_key": payload.get("object_key")}
+
     @app.post(f"{route_prefix}/compute/jobs")
     async def create_job(payload: dict[str, Any]) -> dict[str, Any]:
         payload = {**payload, "require_dataset_files": True}
@@ -628,7 +670,7 @@ def create_app() -> FastAPI:
             "status": "queued",
             "progress": 10,
             "pid": int(52000 + now() % 10000),
-            "gpus": payload.get("gpus") or [0],
+            "gpus": payload.get("gpus") or [],
             "created_at": now(),
             "command": command.command,
             "work_dir": command.work_dir,
@@ -809,6 +851,107 @@ def create_app() -> FastAPI:
             "checksum_sha256": hashlib.sha256(target.read_bytes()).hexdigest() if target.is_file() else "",
         }
 
+    @app.post(f"{route_prefix}/compute/cache/prepare")
+    async def prepare_cache(payload: dict[str, Any]) -> dict[str, Any]:
+        """Download one MinIO object into the node-local cache atomically."""
+        download_url = str(payload.get("download_url") or "")
+        resource_id = str(payload.get("resource_id") or "")
+        version_id = str(payload.get("version_id") or "latest")
+        if not download_url or not resource_id:
+            raise HTTPException(status_code=400, detail="download_url and resource_id are required")
+        data_root = Path(os.getenv("YG_FT_DATA_ROOT", "/data/yg-ft"))
+        cache_root = Path(os.getenv("YG_FT_CACHE_ROOT", str(data_root)))
+        relative_path = str(payload.get("relative_path") or f"resources/{resource_id}/{version_id}/resource")
+        target = (cache_root / relative_path.lstrip("/\\")).resolve()
+        if not _path_inside(cache_root, target):
+            raise HTTPException(status_code=400, detail="cache path must stay inside cache root")
+        target.parent.mkdir(parents=True, exist_ok=True)
+        temp_target = target.with_name(f".{target.name}.part")
+        expected_checksum = str(payload.get("checksum_sha256") or "").lower()
+        expected_size = int(payload.get("byte_size") or 0)
+        lock = cache_locks.setdefault(str(target), asyncio.Lock())
+        async with lock:
+            if target.is_file() and expected_checksum:
+                existing_digest = hashlib.sha256()
+                with target.open("rb") as existing:
+                    while chunk := existing.read(1024 * 1024):
+                        existing_digest.update(chunk)
+                if existing_digest.hexdigest().lower() == expected_checksum and (not expected_size or target.stat().st_size == expected_size):
+                    return {"resource_id": resource_id, "version_id": version_id, "status": "ready", "local_path": str(target), "byte_size": target.stat().st_size, "checksum_sha256": existing_digest.hexdigest(), "reused": True}
+        digest = hashlib.sha256()
+        byte_size = 0
+        try:
+            async with lock:
+                for attempt in range(3):
+                    try:
+                        digest = hashlib.sha256()
+                        byte_size = 0
+                        temp_target.unlink(missing_ok=True)
+                        async with httpx.AsyncClient(timeout=httpx.Timeout(900, connect=30), follow_redirects=True) as client:
+                            async with client.stream("GET", download_url) as response:
+                                response.raise_for_status()
+                                with temp_target.open("wb") as output:
+                                    async for chunk in response.aiter_bytes(1024 * 1024):
+                                        output.write(chunk)
+                                        digest.update(chunk)
+                                        byte_size += len(chunk)
+                        break
+                    except Exception:
+                        temp_target.unlink(missing_ok=True)
+                        if attempt == 2:
+                            raise
+                        await asyncio.sleep(2 ** attempt)
+            checksum = digest.hexdigest()
+            if expected_checksum and checksum != expected_checksum:
+                temp_target.unlink(missing_ok=True)
+                raise HTTPException(status_code=502, detail="cache checksum mismatch")
+            if expected_size and byte_size != expected_size:
+                temp_target.unlink(missing_ok=True)
+                raise HTTPException(status_code=502, detail="cache byte size mismatch")
+            temp_target.replace(target)
+        except HTTPException:
+            raise
+        except Exception as exc:
+            temp_target.unlink(missing_ok=True)
+            raise HTTPException(status_code=502, detail=f"cache download failed: {exc}") from exc
+        return {
+            "resource_id": resource_id,
+            "version_id": version_id,
+            "status": "ready",
+            "local_path": str(target),
+            "byte_size": byte_size,
+            "checksum_sha256": checksum,
+        }
+
+    @app.get(f"{route_prefix}/compute/cache/status")
+    async def cache_status(resource_id: str = Query(...), version_id: str = Query(default="latest")) -> dict[str, Any]:
+        data_root = Path(os.getenv("YG_FT_DATA_ROOT", "/data/yg-ft"))
+        cache_root = Path(os.getenv("YG_FT_CACHE_ROOT", str(data_root)))
+        target = cache_root / "resources" / resource_id / version_id / "resource"
+        return {
+            "resource_id": resource_id,
+            "version_id": version_id,
+            "status": "ready" if target.is_file() else "missing",
+            "local_path": str(target),
+            "byte_size": target.stat().st_size if target.is_file() else 0,
+        }
+
+    @app.delete(f"{route_prefix}/compute/cache")
+    async def clear_cache(resource_id: str | None = Query(default=None), version_id: str | None = Query(default=None)) -> dict[str, Any]:
+        data_root = Path(os.getenv("YG_FT_DATA_ROOT", "/data/yg-ft"))
+        cache_root = Path(os.getenv("YG_FT_CACHE_ROOT", str(data_root)))
+        target = cache_root / "resources"
+        if resource_id:
+            target = target / resource_id
+            if version_id:
+                target = target / version_id
+        target = target.resolve()
+        if not _path_inside(cache_root, target):
+            raise HTTPException(status_code=400, detail="cache path must stay inside cache root")
+        if target.exists():
+            shutil.rmtree(target)
+        return {"status": "cleared", "resource_id": resource_id, "version_id": version_id}
+
     @app.post(f"{route_prefix}/compute/files/import-local")
     async def import_local_file(payload: dict[str, Any]) -> dict[str, Any]:
         source = Path(str(payload.get("source_path") or ""))
@@ -860,10 +1003,17 @@ def create_app() -> FastAPI:
     @app.get(f"{route_prefix}/compute/files/{{file_id}}/download")
     async def download_file(file_id: str) -> FileResponse:
         upload_root = Path(os.getenv("YG_FT_DATA_ROOT", "/data/yg-ft")) / "uploads"
+        # file_id 仅允许普通标识符，拒绝 ../、/、\ 等路径穿越字符。
+        if not file_id or not all(character.isalnum() or character in {"_", "-"} for character in file_id):
+            raise HTTPException(status_code=400, detail="invalid file id")
         matches = list(upload_root.glob(f"{file_id}_*"))
         if not matches:
             raise HTTPException(status_code=404, detail="file not found")
-        return FileResponse(matches[0])
+        # 解析符号链接后仍必须位于 upload 根目录内，防止符号链接指向目录外文件。
+        resolved = matches[0].resolve()
+        if not _path_inside(upload_root, resolved):
+            raise HTTPException(status_code=404, detail="file not found")
+        return FileResponse(resolved)
 
     return app
 

compute/api/security.py

@@ -0,0 +1,29 @@
+"""计算节点 API 安全配置：Swagger / ReDoc / OpenAPI 文档路由开关。"""
+from __future__ import annotations
+
+import os
+from typing import Any
+
+
+def docs_enabled() -> bool:
+    """判断 FastAPI 文档路由（/docs、/redoc、/openapi.json）是否开放。
+
+    显式配置 ENABLE_DOCS 时以之为准；否则仅在关闭 token 鉴权
+    （COMPUTE_AUTH_ENABLED=false，本地开发）时开放，生产环境默认关闭，
+    避免未授权访问泄露 API 结构。
+    """
+    raw = os.getenv("ENABLE_DOCS", "").strip().lower()
+    if raw in {"true", "false"}:
+        return raw == "true"
+    auth_enabled = os.getenv("COMPUTE_AUTH_ENABLED", "true").lower() == "true"
+    return not auth_enabled
+
+
+def docs_kwargs() -> dict[str, Any]:
+    """返回传入 FastAPI 的文档路由参数。
+
+    关闭时 FastAPI 不注册 /docs、/redoc、/openapi.json，访问一律返回 404。
+    """
+    if docs_enabled():
+        return {}
+    return {"docs_url": None, "redoc_url": None, "openapi_url": None}

compute/engines/llama_factory/adapter.py

@@ -15,27 +15,58 @@ class LlamaFactoryCommand:
 
 
 def _load_dataset_preview(path: Path) -> list[dict[str, Any]]:
+    """Load a preview of JSON/JSONL records from a dataset file.
+
+    Content-sniffs instead of trusting the extension so that BOM-prefixed files,
+    JSONL files containing a single JSON array, and mislabeled extensions all work.
+    """
     if not path.exists():
         return []
-    text = path.read_text(encoding="utf-8", errors="replace").strip()
+    text = path.read_text(encoding="utf-8-sig", errors="replace").strip()
     if not text:
         return []
-    if path.suffix.lower() == ".jsonl":
+    try:
-        items: list[dict[str, Any]] = []
+        value = json.loads(text)
-        for line in text.splitlines()[:20]:
+    except json.JSONDecodeError:
-            line = line.strip()
+        value = None
-            if not line:
-                continue
-            value = json.loads(line)
-            if isinstance(value, dict):
-                items.append(value)
-        return items
-    value = json.loads(text)
     if isinstance(value, list):
         return [item for item in value[:20] if isinstance(item, dict)]
     if isinstance(value, dict):
         return [value]
-    return []
+    items: list[dict[str, Any]] = []
+    for line in text.splitlines()[:20]:
+        line = line.strip()
+        if not line:
+            continue
+        try:
+            parsed = json.loads(line)
+        except json.JSONDecodeError:
+            continue
+        if isinstance(parsed, list):
+            items.extend(item for item in parsed[:20] if isinstance(item, dict))
+        elif isinstance(parsed, dict):
+            items.append(parsed)
+        if len(items) >= 20:
+            break
+    return items[:20]
+
+
+def _required_columns_for(formatting: str, columns: dict[str, Any]) -> list[str]:
+    """Required data columns per dataset format.
+
+    Mirrors LLaMA-Factory's leniency: optional columns (e.g. ``input`` / ``query``
+    in Alpaca) are never required, only fields the format structurally needs.
+    """
+    fmt = str(formatting or "").lower()
+    if fmt == "sharegpt":
+        return [str(columns.get("messages") or "messages")]
+    if fmt in {"dpo", "rm", "kto", "ppo"}:
+        return [str(columns[key]) for key in ("chosen", "rejected") if columns.get(key)]
+    if fmt in {"cpt", "pt", "pretrain"}:
+        return [str(columns.get("prompt") or columns.get("text") or "text")]
+    # alpaca family: prompt (instruction) + response (output) required,
+    # query (input) / history are optional and common to omit in jsonl datasets.
+    return [str(columns[key]) for key in ("prompt", "response") if columns.get(key)]
 
 
 def _validate_dataset_columns(config: dict[str, Any]) -> list[str]:
@@ -51,7 +82,7 @@ def _validate_dataset_columns(config: dict[str, Any]) -> list[str]:
         file_name = item.get("file_name")
         file_names = file_name if isinstance(file_name, list) else [file_name]
         columns = item.get("columns") if isinstance(item.get("columns"), dict) else {}
-        required_columns = [str(value) for value in columns.values() if value]
+        required_columns = _required_columns_for(str(item.get("formatting") or ""), columns)
         for name in file_names:
             if not name:
                 continue

compute/engines/llama_factory/eval_runner.py

@@ -22,7 +22,10 @@ from typing import Any
 
 
 def _load_dataset(path: str) -> list[dict[str, Any]]:
-    """Load a JSON or JSONL dataset file.
+    """Load a JSON or JSONL dataset file (jsonl-compatible).
+
+    Content-sniffs instead of trusting the extension so jsonl files with a BOM,
+    a single JSON array on one line, or mislabeled extensions all load correctly.
 
     Supports common field names used across the platform:
     * ``instruction`` + ``input`` + ``output``  (Alpaca-style)
@@ -30,14 +33,17 @@ def _load_dataset(path: str) -> list[dict[str, Any]]:
     * ``messages``  (ShareGPT-style – the last assistant message is treated as reference)
     """
     file_path = Path(path)
-    text = file_path.read_text(encoding="utf-8", errors="replace").strip()
+    text = file_path.read_text(encoding="utf-8-sig", errors="replace").strip()
     if not text:
         return []
-    if file_path.suffix.lower() == ".json":
+    try:
         value = json.loads(text)
-        if isinstance(value, list):
+    except json.JSONDecodeError:
-            return [item for item in value if isinstance(item, dict)]
+        value = None
-        return [value] if isinstance(value, dict) else []
+    if isinstance(value, list):
+        return [item for item in value if isinstance(item, dict)]
+    if isinstance(value, dict):
+        return [value]
 
     samples: list[dict[str, Any]] = []
     for line in text.splitlines():

compute/requirements.txt

@@ -4,6 +4,7 @@ python-multipart>=0.0.9
 pydantic>=2.7.0
 python-dotenv>=1.0.1
 httpx>=0.27.0
+# Compute Agent downloads MinIO objects through presigned HTTP URLs; no MinIO SDK is required.
 # 模型评测指标
 sacrebleu>=2.4.0
 rouge-score>=0.1.2

compute/tests/test_eval_runner.py

@@ -0,0 +1,42 @@
+from __future__ import annotations
+
+import json
+
+from compute.engines.llama_factory.eval_runner import _load_dataset
+
+
+def _write(tmp_path, name: str, text: str) -> str:
+    path = tmp_path / name
+    path.write_text(text, encoding="utf-8")
+    return str(path)
+
+
+def test_load_jsonl_multiline(tmp_path) -> None:
+    path = _write(
+        tmp_path,
+        "eval.jsonl",
+        '{"question": "q1", "answer": "a1"}\n{"question": "q2", "answer": "a2"}\n',
+    )
+    assert _load_dataset(path) == [
+        {"question": "q1", "answer": "a1"},
+        {"question": "q2", "answer": "a2"},
+    ]
+
+
+def test_load_json_array(tmp_path) -> None:
+    path = _write(
+        tmp_path,
+        "eval.json",
+        json.dumps([{"question": "x", "answer": "y"}]),
+    )
+    assert _load_dataset(path) == [{"question": "x", "answer": "y"}]
+
+
+def test_load_jsonl_with_bom_and_embedded_array(tmp_path) -> None:
+    """jsonl 带 BOM 且单行内嵌 JSON 数组，都应正常加载。"""
+    path = _write(
+        tmp_path,
+        "eval.jsonl",
+        "" + json.dumps([{"question": "a", "answer": "b"}, {"question": "c", "answer": "d"}]),
+    )
+    assert len(_load_dataset(path)) == 2

compute/tests/test_file_download_security.py

@@ -0,0 +1,63 @@
+"""compute ``download_file`` 端点安全回归测试。
+
+修复前 ``file_id`` 直接拼进 glob 模式且不校验路径包含关系，可通过 ``../``
+穿越出 upload 目录，并在 Linux 上跟随符号链接读取任意文件。
+修复后：file_id 仅允许字母/数字/下划线/连字符，返回前对解析后的路径
+做 upload 根目录包含性校验。
+"""
+from __future__ import annotations
+
+from pathlib import Path
+
+import pytest
+from fastapi.testclient import TestClient
+
+
+def _make_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> TestClient:
+    monkeypatch.setenv("TRAINING_LOG_ROOT", str(tmp_path / "logs"))
+    monkeypatch.setenv("YG_FT_DATA_ROOT", str(tmp_path / "data"))
+    monkeypatch.setenv("COMPUTE_EXECUTION_MODE", "simulator")
+    monkeypatch.setenv("COMPUTE_AUTH_ENABLED", "false")
+    monkeypatch.delenv("ENABLE_DOCS", raising=False)
+    from compute.api.main import create_app
+
+    return TestClient(create_app())
+
+
+def _upload_root(tmp_path: Path) -> Path:
+    root = tmp_path / "data" / "uploads"
+    root.mkdir(parents=True, exist_ok=True)
+    return root
+
+
+def test_download_legit_file(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+    client = _make_client(tmp_path, monkeypatch)
+    (_upload_root(tmp_path) / "file_123456_hello.txt").write_text("HELLO-DOWNLOAD", encoding="utf-8")
+    response = client.get("/modelTF/compute/files/file_123456/download")
+    assert response.status_code == 200
+    assert response.content == b"HELLO-DOWNLOAD"
+
+
+def test_download_rejects_traversal_file_id(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+    client = _make_client(tmp_path, monkeypatch)
+    outside = tmp_path / "secret" / "passwd_1.txt"
+    outside.parent.mkdir(parents=True, exist_ok=True)
+    outside.write_text("TOP-SECRET", encoding="utf-8")
+    for file_id in ["..", "file.123", "..%2F..%2Fsecret%2Fpasswd", "file%20name"]:
+        response = client.get(f"/modelTF/compute/files/{file_id}/download")
+        assert response.status_code in (400, 404), f"file_id={file_id!r} -> {response.status_code}"
+        assert b"TOP-SECRET" not in response.content
+
+
+def test_download_blocks_symlink_escape(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+    client = _make_client(tmp_path, monkeypatch)
+    upload_root = _upload_root(tmp_path)
+    outside = tmp_path / "secret.txt"
+    outside.write_text("TOP-SECRET", encoding="utf-8")
+    try:
+        (upload_root / "file_999999_link.txt").symlink_to(outside)
+    except OSError:
+        pytest.skip("symlink creation not permitted on this platform")
+    response = client.get("/modelTF/compute/files/file_999999/download")
+    assert response.status_code == 404
+    assert b"TOP-SECRET" not in response.content

compute/tests/test_llama_factory_adapter.py

@@ -1,6 +1,8 @@
 from __future__ import annotations
 
-from compute.engines.llama_factory.adapter import build_command
+import json
+
+from compute.engines.llama_factory.adapter import _validate_dataset_columns, build_command
 
 
 def test_build_command_uses_explicit_validation_dataset_without_resplitting() -> None:
@@ -21,3 +23,70 @@ def test_build_command_uses_explicit_validation_dataset_without_resplitting() ->
     )
     assert "--do_eval" in result.command
     assert "--val_size" not in result.command
+
+
+def _write(tmp_path, name: str, lines: list[dict]) -> object:
+    path = tmp_path / name
+    path.write_text(
+        "".join(json.dumps(line, ensure_ascii=False) + "\n" for line in lines),
+        encoding="utf-8",
+    )
+    return path
+
+
+def test_jsonl_alpaca_without_input_column_passes_validation(tmp_path) -> None:
+    """纯 jsonl Alpaca 数据缺省 input 字段（常见），不应被校验拦截。"""
+    _write(tmp_path, "train.jsonl", [{"instruction": "hi", "output": "hello"}])
+    errors = _validate_dataset_columns(
+        {
+            "dataset_dir": str(tmp_path),
+            "dataset_info": {
+                "ygft_a": {
+                    "file_name": "train.jsonl",
+                    "formatting": "alpaca",
+                    "columns": {"prompt": "instruction", "query": "input", "response": "output"},
+                }
+            },
+        }
+    )
+    assert errors == []
+
+
+def test_jsonl_sharegpt_passes_validation(tmp_path) -> None:
+    """ShareGPT 格式 jsonl（messages）应通过校验。"""
+    _write(
+        tmp_path,
+        "msg.jsonl",
+        [{"messages": [{"role": "user", "content": "hi"}, {"role": "assistant", "content": "hello"}]}],
+    )
+    errors = _validate_dataset_columns(
+        {
+            "dataset_dir": str(tmp_path),
+            "dataset_info": {
+                "ygft_m": {
+                    "file_name": "msg.jsonl",
+                    "formatting": "sharegpt",
+                    "columns": {"messages": "messages"},
+                }
+            },
+        }
+    )
+    assert errors == []
+
+
+def test_jsonl_missing_response_still_rejected(tmp_path) -> None:
+    """缺 output（response）仍应报错——没有答案无法做有监督微调。"""
+    _write(tmp_path, "train.jsonl", [{"instruction": "hi"}])
+    errors = _validate_dataset_columns(
+        {
+            "dataset_dir": str(tmp_path),
+            "dataset_info": {
+                "ygft_a": {
+                    "file_name": "train.jsonl",
+                    "formatting": "alpaca",
+                    "columns": {"prompt": "instruction", "query": "input", "response": "output"},
+                }
+            },
+        }
+    )
+    assert errors and "output" in errors[0]

compute/tests/test_security.py

@@ -0,0 +1,40 @@
+"""计算节点文档路由（/docs、/redoc、/openapi.json）安全开关测试。
+
+生产默认（COMPUTE_AUTH_ENABLED=true）关闭文档路由，避免未授权泄露 API 结构；
+显式配置 ENABLE_DOCS 可覆盖默认行为。
+"""
+from __future__ import annotations
+
+from compute.api.security import docs_enabled, docs_kwargs
+
+
+def test_docs_disabled_when_auth_enabled(monkeypatch) -> None:
+    monkeypatch.delenv("ENABLE_DOCS", raising=False)
+    monkeypatch.setenv("COMPUTE_AUTH_ENABLED", "true")
+    assert docs_enabled() is False
+    assert docs_kwargs() == {"docs_url": None, "redoc_url": None, "openapi_url": None}
+
+
+def test_docs_enabled_when_auth_disabled(monkeypatch) -> None:
+    monkeypatch.delenv("ENABLE_DOCS", raising=False)
+    monkeypatch.setenv("COMPUTE_AUTH_ENABLED", "false")
+    assert docs_enabled() is True
+    assert docs_kwargs() == {}
+
+
+def test_docs_env_override_enables_with_auth(monkeypatch) -> None:
+    monkeypatch.setenv("ENABLE_DOCS", "true")
+    monkeypatch.setenv("COMPUTE_AUTH_ENABLED", "true")
+    assert docs_enabled() is True
+
+
+def test_docs_env_override_disables_without_auth(monkeypatch) -> None:
+    monkeypatch.setenv("ENABLE_DOCS", "false")
+    monkeypatch.setenv("COMPUTE_AUTH_ENABLED", "false")
+    assert docs_enabled() is False
+
+
+def test_docs_default_when_auth_env_missing(monkeypatch) -> None:
+    monkeypatch.delenv("ENABLE_DOCS", raising=False)
+    monkeypatch.delenv("COMPUTE_AUTH_ENABLED", raising=False)
+    assert docs_enabled() is False

docker/README.md

@@ -1,299 +1,58 @@
 # Docker 部署说明
 
-本目录按应用服务器和算力服务器拆分 Dockerfile 与 Docker Compose 文件。Compose 文件不包含 `build:`，不会在 `docker compose up` 时自动构建业务镜像。所有业务镜像需要先通过手动 `docker build` 构建，再由 Compose 启动。
+## 部署边界
 
-## 基础镜像
+生产规划分为应用服务器、存储服务器和算力服务器。应用服务器运行 Frontend Nginx、Backend API、Redis；存储服务器运行 MinIO；算力服务器运行 Compute API、Compute Agent、GPU 和训练引擎。三类服务器不共享 Docker 网络，通过可路由地址通信。
 
-| 镜像 | 用途 |
+当前 WSL 联调地址为 `172.25.179.69`，拆分部署时必须替换为真实服务器地址。
-| --- | --- |
-| `python:3.12-slim` | 应用后端基础镜像，后端运行环境要求 Python 3.12 及以上 |
-| `nginx:1.27-alpine` | 前端静态资源与 `/modelTF` 反向代理运行镜像 |
-| `hiyouga/llamafactory:latest` | 算力服务基础镜像，基于 LLaMA-Factory 官方镜像扩展 Compute API |
-| `postgres:16-alpine` | 开发阶段内置 PostgreSQL |
-| `redis:7-alpine` | 开发阶段内置 Redis |
 
-一键拉取基础镜像：
+## 配置
 
-```bash
+应用服务器 `docker/app/.env`：
-docker pull python:3.12-slim && \
-docker pull nginx:1.27-alpine && \
-docker pull hiyouga/llamafactory:latest && \
-docker pull postgres:16-alpine && \
-docker pull redis:7-alpine
-```
-
-Windows PowerShell：
-
-```powershell
-$images = @(
-  "python:3.12-slim",
-  "nginx:1.27-alpine",
-  "hiyouga/llamafactory:latest",
-  "postgres:16-alpine",
-  "redis:7-alpine"
-)
-$images | ForEach-Object { docker pull $_ }
-```
-
-如果部署环境不能访问外网，需要提前在可联网环境执行上述拉取命令，再用 `docker save` / `docker load` 导出导入。
-
-## 业务镜像
-
-| 镜像 | Dockerfile | 构建命令 |
-| --- | --- | --- |
-| `yg-ft-backend-api:latest` | `docker/app/Dockerfile.backend` | `docker build -f docker/app/Dockerfile.backend -t yg-ft-backend-api:latest .` |
-| `yg-ft-frontend-runtime:latest` | `docker/app/Dockerfile.frontend` | `docker build -f docker/app/Dockerfile.frontend -t yg-ft-frontend-runtime:latest .` |
-| `yg-ft-compute-api:latest` | `docker/compute/Dockerfile.compute` | `docker build -f docker/compute/Dockerfile.compute -t yg-ft-compute-api:latest .` |
-
-## 对外端口
-
-所有宿主机对外端口统一使用 5 位端口。容器内部端口保持镜像默认端口，便于容器内服务和健康检查稳定。
-
-| 服务 | 宿主机对外端口 | 容器内部端口 | 说明 |
-| --- | --- | --- | --- |
-| 前端 Nginx | `16801` | `80` | 前端页面入口 |
-| 后端 API | `17861` | `8000` | FastAPI 服务 |
-| PostgreSQL | `15432` | `5432` | 开发阶段内置数据库 |
-| Redis | `16379` | `6379` | 开发阶段内置缓存 |
-| Compute API | `19100` | `9100` | 算力服务器 API |
-| File Gateway | `19101` | `9100` | 当前由 Compute API 暴露文件网关契约，后续可拆为独立服务 |
-
-注意：`8000` 是后端容器内部端口，不作为宿主机对外访问端口。宿主机或浏览器应访问 `http://<app-server-ip>:17861/modelTF/health`；前端 Nginx 容器在 Docker 网络内部访问 `http://backend-api:8000/modelTF/...`。
-
-对应配置文件：
-
-- `docker/app/.env.example`
-  - `FRONTEND_PORT=16801`
-  - `BACKEND_API_PORT=17861`
-  - `POSTGRES_PORT=15432`
-  - `REDIS_PORT=16379`
-- `docker/compute/.env.example`
-  - `COMPUTE_API_PORT=19100`
-  - `FILE_GATEWAY_PORT=19101`
-
-## 运行模式
-
-- 应用侧默认 `COMPUTE_MODE=real`，任务状态必须由真实算力同步逻辑更新。
-- 算力侧默认 `COMPUTE_EXECUTION_MODE=real`，真实执行器未完成前不会伪造训练作业。
-- 仅隔离联调时可显式设置 `COMPUTE_MODE=simulator` 或 `COMPUTE_EXECUTION_MODE=simulator`，该模式不得用于测试环境、生产环境或生产升级基线。
-
-## 应用服务器部署
-
-应用服务器包含前端 Nginx、Backend API、PostgreSQL、Redis。
-
-当前 Compose 内置 PostgreSQL 使用 `backend/app/db/sql/001_platform_runtime.sql` 初始化运行库。`docs/postgres-schema.sql` 是完整目标架构设计，不应直接挂载为当前运行库初始化脚本，否则会与当前后端代码的运行表结构不兼容。
-
-首次部署：
-
-```bash
-cd <repo-root>
-
-# 1. 使用当前 Windows/宿主机 npm 构建前端静态产物
-cd frontend
-npm ci
-npm run build
-cd ..
-
-# 2. 手动构建业务镜像
-docker build -f docker/app/Dockerfile.backend -t yg-ft-backend-api:latest .
-docker build -f docker/app/Dockerfile.frontend -t yg-ft-frontend-runtime:latest .
-
-# 3. 启动应用服务
-cd docker/app
-cp .env.example .env
-docker compose up -d
-```
-
-后端镜像构建过程中会执行依赖导入自检，确认 `fastapi`、`uvicorn`、`psycopg`、`sqlalchemy`、`redis` 等运行依赖已安装。构建后也可以手动检查：
-
-```bash
-docker run --rm yg-ft-backend-api:latest python -c "import psycopg; print(psycopg.__version__)"
-```
-
-默认访问地址：
-
-```text
-http://<app-server-ip>:16801
-```
-
-应用侧代码和数据外挂：
-
-```text
-../../backend -> /app
-../../frontend/dist -> /usr/share/nginx/html
-../../runtime/app/logs/backend -> /opt/yg-ft/logs/backend
-../../runtime/app/data -> /data/yg-ft
-```
-
-前端容器启动前必须确保 `../../frontend/dist/index.html` 已存在。若前端 Nginx 日志出现 `directory index of "/usr/share/nginx/html/" is forbidden` 或 `rewrite or internal redirection cycle while internally redirecting to "/index.html"`，通常表示当前执行 `docker compose` 的项目目录下没有构建好的 `frontend/dist`，或挂载路径不是同一份代码目录。
-
-```bash
-# 在执行 docker compose 的同一份代码目录中检查
-cd <repo-root>/frontend
-npm run build
-test -f dist/index.html && ls -lh dist/index.html
-
-cd ../docker/app
-docker compose up -d --force-recreate frontend
-docker compose logs --tail=80 frontend
-```
-
-如果使用 Windows npm 构建、WSL 中运行 Docker Compose，需要确认 Windows 路径和 WSL 路径指向同一份仓库。例如在 `D:\...\YG_FT\frontend` 构建不会自动生成 `/mnt/d/wuyongtao/Code/YG_FT/frontend/dist` 下的产物，除非二者本就是同一个目录。
-
-如果使用企业统一 PostgreSQL/Redis，修改 `docker/app/.env`：
-
-如果前端 Nginx 日志出现 `open() "/usr/share/nginx/html/modelTF/login" failed` 或 `open() "/usr/share/nginx/html/login" failed`，说明当前容器没有加载项目的 Nginx 代理配置，`/modelTF/*` 被当成静态文件查找。处理方式：
-
-```bash
-cd <repo-root>/docker/app
-docker compose up -d --force-recreate frontend
-docker compose exec frontend nginx -T | grep -n "location.*modelTF" -A12
-```
-
-正常配置中应存在 `location ^~ /modelTF/`，并代理到 `BACKEND_PROXY_PASS`，默认是 `http://backend-api:8000`。
 
 ```env
-DATABASE_URL=postgresql+psycopg://<user>:<password>@<postgres-host>:15432/<db>
+MINIO_ENABLED=true
-REDIS_URL=redis://<redis-host>:16379/0
+MINIO_ENDPOINT=http://172.25.179.69:19000
-USE_BUILTIN_POSTGRES=false
+COMPUTE_API_BASE_URL=http://172.25.179.69:19100
-USE_BUILTIN_REDIS=false
+FILE_GATEWAY_BASE_URL=http://172.25.179.69:19101
-```
-
-生产环境如完全使用外部基础设施，可以删除或注释 Compose 中的 `postgres`、`redis` 服务及 `backend-api.depends_on` 中对应依赖。
-
-## 算力服务器部署
-
-算力服务器包含 Compute API、后续 Compute Agent、File Gateway、GPU runtime、本地训练数据目录和 LLaMA-Factory。`Dockerfile.compute` 基于 LLaMA-Factory 官方镜像：
-
-```dockerfile
-FROM hiyouga/llamafactory:latest
-```
-
-部署前需要安装：
-
-- NVIDIA Driver
-- NVIDIA Container Toolkit
-- Docker Engine 和 Docker Compose Plugin
-- 本地训练数据目录，默认 `/data/yg-ft`
-
-首次部署：
-
-```bash
-cd <repo-root>
-
-# 手动构建算力业务镜像
-docker build -f docker/compute/Dockerfile.compute -t yg-ft-compute-api:latest .
-
-# 启动算力服务
-cd docker/compute
-cp .env.example .env
-docker compose up -d
-```
-
-健康检查：
-
-```text
-GET http://<compute-server-ip>:19100/modelTF/health
-GET http://<compute-server-ip>:19100/modelTF/v1/compute/health
-```
-
-算力侧代码和数据外挂：
-
-```text
-../../compute -> /app/compute
-${YG_FT_DATA_ROOT_HOST} -> /data/yg-ft
-${YG_FT_MODEL_ROOT_HOST} -> /data/yg-ft/models
-${YG_FT_DATASET_ROOT_HOST} -> /data/yg-ft/datasets
-${YG_FT_OUTPUT_ROOT_HOST} -> /data/yg-ft/outputs
-${COMPUTE_LOG_ROOT_HOST} -> /opt/yg-ft/logs/compute
-${TRAINING_LOG_ROOT_HOST} -> /opt/yg-ft/logs/training
-```
-
-算力服务器启动前必须先在宿主机创建持久化目录，基座模型、训练数据、训练产物和训练日志都应落在宿主机磁盘上，不能只写入容器层。推荐默认目录：
-
-```bash
-cd <repo-root>/docker/compute
-mkdir -p data/yg-ft/models \
-  data/yg-ft/datasets \
-  data/yg-ft/outputs \
-  data/yg-ft/logs/compute \
-  data/yg-ft/logs/training
-```
-
-默认 `docker/compute/.env.example` 使用 `./data/yg-ft`，该相对路径以 `docker/compute/docker-compose.yml` 所在目录为基准，因此实际宿主机目录是 `<repo-root>/docker/compute/data/yg-ft`。如企业环境模型盘、数据盘、产物盘分盘挂载，可在 `docker/compute/.env` 中分别调整 `YG_FT_MODEL_ROOT_HOST`、`YG_FT_DATASET_ROOT_HOST`、`YG_FT_OUTPUT_ROOT_HOST`、`COMPUTE_LOG_ROOT_HOST`、`TRAINING_LOG_ROOT_HOST`，容器内路径建议保持 `/data/yg-ft/models`、`/data/yg-ft/datasets`、`/data/yg-ft/outputs`，避免训练参数和节点配置复杂化。
-
-页面上传数据集时，文件先进入 Backend API，再由 Backend API 调用目标算力节点的 `POST /modelTF/compute/files/upload`，写入容器内 `/data/yg-ft/datasets/{dataset_id}/`。在默认开发配置下，宿主机可在 `<repo-root>/docker/compute/data/yg-ft/datasets/{dataset_id}/` 看到对应文件。仅创建 bind mount 不会自动让应用侧上传文件出现在算力目录，必须通过这条 File Gateway 链路同步。
-
-## 应用与算力分离部署
-
-应用服务器只需要主动访问算力服务器，不要求算力服务器回调应用服务器。
-
-在 `docker/app/.env` 中配置：
-
-```env
-COMPUTE_API_BASE_URL=http://<compute-server-ip>:19100
-FILE_GATEWAY_BASE_URL=http://<compute-server-ip>:19101
-COMPUTE_SERVICE_TOKEN=change_me
 COMPUTE_STATUS_SYNC_MODE=polling
 COMPUTE_POLL_INTERVAL_SECONDS=3
-COMPUTE_POLL_BATCH_SIZE=100
 ```
 
-交互链路：
+MinIO 不需要额外安装 Python 包；Backend 的 MinIO 兼容访问依赖随 `backend/requirements.txt` 安装。生产环境应使用固定 DNS/IP、内网访问和防火墙白名单。
 
-```text
+## 构建和启动
-Frontend
-  -> Backend API
-    -> Compute API
-      -> Compute Agent / LLaMA-Factory
-      -> 本地数据目录 / 模型目录 / 训练产物
-    <- Backend Worker 定时轮询 Compute API
-```
-
-算力服务默认开启服务间鉴权。`docker/compute/.env` 中保持 `COMPUTE_AUTH_ENABLED=true`，并确保 `COMPUTE_SERVICE_TOKEN` 与 `docker/app/.env` 一致；健康检查路径仍可用于容器探活。
-
-## 多算力节点部署
-
-多算力节点仍按“单机多 GPU 节点”部署。每台 GPU 服务器都独立部署一套 `docker/compute`：
-
-```text
-gpu-node-01: docker/compute + /data/yg-ft + 19100/19101
-gpu-node-02: docker/compute + /data/yg-ft + 19100/19101
-gpu-node-03: docker/compute + /data/yg-ft + 19100/19101
-```
-
-节点之间默认不互访。应用平台主动访问每个节点的 Compute API/File Gateway，并通过 `compute_nodes`、`resource_replicas`、`resource_sync_jobs` 统一调度和同步。
-
-节点地址、权重、标签、启用状态和本地路径在前端“算力节点”页面动态维护。新增或编辑节点后，点击“测试”会由 Backend API 主动访问该节点的 `GET /modelTF/v1/compute/health` 和 `GET /modelTF/compute/resources/gpus`，并把健康信息与 GPU 清单同步到 PostgreSQL。
-
-## 常用命令
-
-重新构建应用镜像：
 
 ```bash
 docker build -f docker/app/Dockerfile.backend -t yg-ft-backend-api:latest .
 docker build -f docker/app/Dockerfile.frontend -t yg-ft-frontend-runtime:latest .
-```
-
-重新构建算力镜像：
-
-```bash
 docker build -f docker/compute/Dockerfile.compute -t yg-ft-compute-api:latest .
+cd docker/minio && docker compose up -d
+cd ../app && docker compose up -d
+cd ../compute && docker compose up -d
 ```
 
-启动服务：
+验证命令：`docker compose ps`、`curl http://<app-server>:17861/modelTF/health`、`curl http://<compute-server>:19100/modelTF/health`、`curl http://<minio-server>:19000/minio/health/live`。
 
-```bash
+## 端口
-cd docker/app
-docker compose up -d
 
-cd ../compute
+| 服务 | 主机端口 | 容器端口 |
-docker compose up -d
+| --- | ---: | ---: |
-```
+| Frontend | 16801 | 80 |
+| Backend API | 17861 | 8000 |
+| Redis | 16379 | 6379 |
+| MinIO API/Console | 19000/19001 | 9000/9001 |
+| Compute API/File Gateway | 19100/19101 | 9100 |
 
-查看服务：
+## 数据和安全
 
-```bash
+算力主机应持久化 `/data/yg-ft/models`、`datasets`、`trained_models`、`outputs`、`cache` 以及训练日志目录。上述目录是 Compute Agent 的本地缓存和运行目录，权威对象必须归档 MinIO。
-docker compose ps
+
-docker compose logs -f
+- Compute API 和 File Gateway 开启 token 认证。
-```
+- MinIO 不直接暴露公网，使用内网或安全组限制访问。
+- 不在镜像和 Git 中提交数据库、Redis、MinIO 密码或服务 token。
+- 当前 Compute API 默认 root 仅适用于开发阶段，生产环境需评估非 root 和 GPU/挂载目录权限改造。
+
+## 拆分验证
+
+从 Backend 容器验证 MinIO 和每个 Compute API 的 health；上传数据集、训练、权重合并、推理和节点断网重试均需完成一次联调。

docker/app/.env

@@ -1,6 +1,8 @@
 APP_ENV=prod
 APP_NAME=YG Fine-Tune Platform API
 MODELTF_ROUTE_PREFIX=/modelTF
+# 禁止未授权访问 Swagger/ReDoc/OpenAPI 文档；调试时可临时改为 true
+ENABLE_DOCS=false
 CORS_ALLOW_ORIGINS=http://localhost:16801,http://127.0.0.1:16801
 
 FRONTEND_IMAGE=yg-ft-frontend-runtime:latest
@@ -17,7 +19,9 @@ POSTGRES_USER=root
 POSTGRES_PASSWORD=8811614287327Leo
 DATABASE_URL=postgresql+psycopg://root:8811614287327Leo@www.caoxiaozhu.com:5432/yg_ft
 
-REDIS_URL=redis://redis:6379/0
+# Redis 访问鉴权：requirepass 密码；REDIS_URL 已内嵌密码（redis://:<密码>@redis:6379/0）
+REDIS_PASSWORD=Tvhrf659WaX-S1B8FG6c2kSZK07XTv82
+REDIS_URL=redis://:Tvhrf659WaX-S1B8FG6c2kSZK07XTv82@redis:6379/0
 
 # PostgreSQL uses the shared external database. The local postgres service is disabled in docker-compose.yml.
 # Redis still uses the built-in service during current development.
@@ -44,3 +48,17 @@ COMPUTE_STATUS_SYNC_MODE=polling
 COMPUTE_POLL_INTERVAL_SECONDS=10
 COMPUTE_POLL_BATCH_SIZE=100
 COMPUTE_REQUEST_TIMEOUT_SECONDS=5
+
+# MinIO object storage. Enable after the MinIO service is reachable.
+MINIO_API_PORT=19000
+MINIO_CONSOLE_PORT=19001
+MINIO_ENABLED=true
+# For split-server deployment, replace host.docker.internal with the MinIO
+# server address, for example http://10.10.20.30:19000.
+MINIO_ENDPOINT=http://172.25.179.69:19000
+MINIO_ACCESS_KEY=minioadmin
+MINIO_SECRET_KEY=change_me_minio_secret
+MINIO_BUCKET=yg-ft-resources
+MINIO_SECURE=false
+STORAGE_WAIT_SECONDS=300
+STORAGE_CHECK_INTERVAL_SECONDS=10

docker/app/Dockerfile.backend

@@ -2,7 +2,8 @@ FROM python:3.12-slim
 
 ENV PYTHONDONTWRITEBYTECODE=1 \
     PYTHONUNBUFFERED=1 \
-    PIP_NO_CACHE_DIR=1
+    PIP_NO_CACHE_DIR=1 \
+    TIKTOKEN_CACHE_DIR=/opt/tiktoken_cache
 
 WORKDIR /app
 
@@ -11,11 +12,14 @@ RUN pip install --upgrade pip -i https://pypi.tuna.tsinghua.edu.cn/simple \
     && pip install -r /tmp/requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple \
     && rm -f /tmp/requirements.txt
 
-RUN python -c "import fastapi, uvicorn, psycopg, psycopg_pool, sqlalchemy, redis, jwt, passlib, httpx, alembic; print('backend dependency check ok')"
+RUN python -c "import fastapi, uvicorn, psycopg, psycopg_pool, sqlalchemy, redis, jwt, passlib, httpx, minio, alembic; print('backend dependency check ok')"
 
 RUN mkdir -p /opt/yg-ft/logs/backend /data/yg-ft \
     && chmod -R 0775 /opt/yg-ft /data/yg-ft
 
+# 离线打包 tiktoken cl100k_base 词表，避免无网环境下运行时联网下载
+COPY docker/app/tiktoken /opt/tiktoken_cache
+
 EXPOSE 8000
 
 CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

docker/app/docker-compose.yml

@@ -42,9 +42,10 @@ services:
       APP_ENV: ${APP_ENV:-prod}
       APP_NAME: ${APP_NAME:-YG Fine-Tune Platform API}
       MODELTF_ROUTE_PREFIX: ${MODELTF_ROUTE_PREFIX:-/modelTF}
+      ENABLE_DOCS: ${ENABLE_DOCS:-false}
       CORS_ALLOW_ORIGINS: ${CORS_ALLOW_ORIGINS:-http://localhost:16801,http://127.0.0.1:16801}
       DATABASE_URL: ${DATABASE_URL:-postgresql+psycopg://root:8811614287327Leo@www.caoxiaozhu.com:5432/yg_ft}
-      REDIS_URL: ${REDIS_URL:-redis://redis:6379/0}
+      REDIS_URL: ${REDIS_URL:-redis://:${REDIS_PASSWORD:-change_me}@redis:6379/0}
       USE_BUILTIN_POSTGRES: ${USE_BUILTIN_POSTGRES:-false}
       USE_BUILTIN_REDIS: ${USE_BUILTIN_REDIS:-true}
       LOG_LEVEL: ${LOG_LEVEL:-INFO}
@@ -61,11 +62,25 @@ services:
       COMPUTE_POLL_INTERVAL_SECONDS: ${COMPUTE_POLL_INTERVAL_SECONDS:-3}
       COMPUTE_POLL_BATCH_SIZE: ${COMPUTE_POLL_BATCH_SIZE:-100}
       COMPUTE_REQUEST_TIMEOUT_SECONDS: ${COMPUTE_REQUEST_TIMEOUT_SECONDS:-5}
+      MINIO_ENABLED: ${MINIO_ENABLED:-false}
+      # Use the storage server's externally reachable address. Do not use a
+      # MinIO container name because storage is deployed independently.
+      MINIO_ENDPOINT: ${MINIO_ENDPOINT:-http://host.docker.internal:19000}
+      MINIO_ACCESS_KEY: ${MINIO_ACCESS_KEY:-minioadmin}
+      MINIO_SECRET_KEY: ${MINIO_SECRET_KEY:-minioadmin}
+      MINIO_BUCKET: ${MINIO_BUCKET:-yg-ft-resources}
+      MINIO_SECURE: ${MINIO_SECURE:-false}
+      STORAGE_WAIT_SECONDS: ${STORAGE_WAIT_SECONDS:-300}
+      STORAGE_CHECK_INTERVAL_SECONDS: ${STORAGE_CHECK_INTERVAL_SECONDS:-10}
+      DATA_PROCESS_STORAGE_DIR: ${DATA_PROCESS_STORAGE_DIR:-/data/yg-ft/data-process}
+      YG_FT_DATA_ROOT: ${YG_FT_DATA_ROOT:-/data/yg-ft}
       PYTHONPATH: /app
     volumes:
       - ../../backend:/app:ro
       - ../../runtime/app/logs/backend:/opt/yg-ft/logs/backend
       - ../../runtime/app/data:/data/yg-ft
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
     networks:
       - yg-ft-app
     healthcheck:
@@ -103,7 +118,10 @@ services:
   redis:
     image: redis:7-alpine
     container_name: yg-ft-redis
-    command: ["redis-server", "--appendonly", "yes"]
+    command: ["redis-server", "--appendonly", "yes", "--requirepass", "${REDIS_PASSWORD:-change_me}"]
+    environment:
+      # redis-cli 健康检查免命令行传密码（避免 -a 泄露进程参数）
+      REDISCLI_AUTH: ${REDIS_PASSWORD:-change_me}
     volumes:
       - redis_data:/data
     ports:

docker/compute/.env

@@ -2,6 +2,8 @@ COMPUTE_ENV=prod
 COMPUTE_HOST_ID=gpu-node-01
 COMPUTE_EXECUTION_MODE=real
 MODELTF_ROUTE_PREFIX=/modelTF
+# 禁止未授权访问 Swagger/ReDoc/OpenAPI 文档；调试时可临时改为 true
+ENABLE_DOCS=false
 # Five-digit host ports exposed outside the compute server.
 COMPUTE_API_PORT=19100
 FILE_GATEWAY_PORT=19101

docker/compute/.env.example

@@ -2,10 +2,13 @@ COMPUTE_ENV=prod
 COMPUTE_HOST_ID=gpu-node-01
 COMPUTE_EXECUTION_MODE=real
 MODELTF_ROUTE_PREFIX=/modelTF
+# 禁止未授权访问 Swagger/ReDoc/OpenAPI 文档；调试时可临时改为 true
+ENABLE_DOCS=false
 # Five-digit host ports exposed outside the compute server.
 COMPUTE_API_PORT=19100
 FILE_GATEWAY_PORT=19101
 COMPUTE_API_IMAGE=yg-ft-compute-api:latest
+YG_FT_CACHE_ROOT=/data/yg-ft
 
 # The application server actively polls Compute API; compute server does not need reverse access.
 COMPUTE_AUTH_ENABLED=true

docker/compute/docker-compose.yml

@@ -11,6 +11,7 @@ services:
       COMPUTE_HOST_ID: ${COMPUTE_HOST_ID:-gpu-node-01}
       COMPUTE_EXECUTION_MODE: ${COMPUTE_EXECUTION_MODE:-real}
       MODELTF_ROUTE_PREFIX: ${MODELTF_ROUTE_PREFIX:-/modelTF}
+      ENABLE_DOCS: ${ENABLE_DOCS:-false}
       COMPUTE_AUTH_ENABLED: ${COMPUTE_AUTH_ENABLED:-true}
       COMPUTE_SERVICE_TOKEN: ${COMPUTE_SERVICE_TOKEN:-change_me}
       ENABLE_APP_CALLBACK: ${ENABLE_APP_CALLBACK:-false}
@@ -29,6 +30,7 @@ services:
       CUDA_VISIBLE_DEVICES: ${CUDA_VISIBLE_DEVICES:-all}
       NVIDIA_VISIBLE_DEVICES: ${NVIDIA_VISIBLE_DEVICES:-all}
       NVIDIA_DRIVER_CAPABILITIES: ${NVIDIA_DRIVER_CAPABILITIES:-compute,utility}
+      YG_FT_CACHE_ROOT: ${YG_FT_CACHE_ROOT:-/data/yg-ft}
       PYTHONPATH: /app
     volumes:
       - ../../compute:/app/compute:ro

docker/minio/.env.example

@@ -0,0 +1,9 @@
+MINIO_ACCESS_KEY=minioadmin
+MINIO_SECRET_KEY=change_me_minio_secret
+MINIO_API_PORT=19000
+MINIO_CONSOLE_PORT=19001
+MINIO_DATA_ROOT_HOST=./data
+
+# Backend and Compute API use this externally reachable endpoint when MinIO
+# runs on a separate server. Keep the container's internal API port at 9000.
+MINIO_EXTERNAL_ENDPOINT=http://<storage-server-ip>:19000

docker/minio/docker-compose.yml

@@ -0,0 +1,19 @@
+services:
+  minio:
+    image: minio/minio:RELEASE.2025-02-28T09-55-16Z
+    container_name: yg-ft-minio
+    command: server /data --console-address ":9001"
+    environment:
+      MINIO_ROOT_USER: ${MINIO_ACCESS_KEY:-minioadmin}
+      MINIO_ROOT_PASSWORD: ${MINIO_SECRET_KEY:-change_me_minio_secret}
+    ports:
+      - "${MINIO_API_PORT:-19000}:9000"
+      - "${MINIO_CONSOLE_PORT:-19001}:9001"
+    volumes:
+      - ${MINIO_DATA_ROOT_HOST:-./data}:/data
+    healthcheck:
+      test: ["CMD-SHELL", "curl -fsS http://127.0.0.1:9000/minio/health/live || exit 1"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+    restart: unless-stopped

docs/20260812/README.md

@@ -0,0 +1,59 @@
+# YG_FT 模型微调平台
+
+YG_FT 是面向多用户、多租户和多算力节点的模型训练与推理平台，提供数据集、模型、训练、权重合并、推理、评测、算力节点、项目隔离、权限和审计能力。
+
+## 架构
+
+```text
+浏览器 -> Frontend Nginx:16801 -> Backend API:17861
+                                      |-> PostgreSQL（元数据、权限、审计、任务状态）
+                                      |-> Redis（缓存及任务辅助状态）
+                                      |-> MinIO:19000（模型和数据唯一对象源）
+                                      |-> Compute API:19100
+                                                |-> Compute Agent/GPU/LLaMA-Factory
+                                      `-> File Gateway:19101
+```
+
+Backend、MinIO 和 Compute 节点可以部署在不同服务器，不依赖跨服务器 Docker 网络，通过 IP、DNS 或负载均衡地址通信。Compute 节点只保存按需准备的本地缓存，MinIO 是模型、数据集、权重、评测报告和训练产物的唯一数据源。
+
+## 目录
+
+| 目录 | 作用 |
+| --- | --- |
+| `frontend/` | Vue 3、TypeScript、Element Plus 控制台 |
+| `backend/app/api/v1/` | 平台 REST API |
+| `backend/app/core/` | 配置、认证、权限和日志 |
+| `backend/app/db/` | PostgreSQL 访问和初始化 SQL |
+| `backend/app/modules/` | 系统、资源、审批、数据处理和存储模块 |
+| `backend/app/workers/` | 节点轮询、任务对账和资源同步 |
+| `compute/` | Compute API、Agent、GPU 和训练引擎 |
+| `docker/` | 应用、MinIO、算力服务部署文件 |
+| `docs/` | 架构、权限、部署和测试文档 |
+
+## 端口
+
+| 服务 | 主机端口 | 容器端口 |
+| --- | ---: | ---: |
+| Frontend | 16801 | 80 |
+| Backend API | 17861 | 8000 |
+| Redis | 16379 | 6379 |
+| MinIO API/Console | 19000/19001 | 9000/9001 |
+| Compute API/File Gateway | 19100/19101 | 9100 |
+
+## 启动
+
+```bash
+cd frontend && npm ci && npm run build && cd ..
+docker build -f docker/app/Dockerfile.backend -t yg-ft-backend-api:latest .
+docker build -f docker/app/Dockerfile.frontend -t yg-ft-frontend-runtime:latest .
+docker build -f docker/compute/Dockerfile.compute -t yg-ft-compute-api:latest .
+cd docker/minio && docker compose up -d
+cd ../app && docker compose up -d
+cd ../compute && docker compose up -d
+```
+
+## 权限与性能
+
+系统使用角色权限、资源 ACL、用户/项目/租户归属联合校验；删除为软删除，关键操作写入审计。模型合并前准备 base model 和 adapter，结果归档 MinIO；推理前按选择节点准备缓存。远程 PostgreSQL 延迟会影响全量列表和看板，页面慢时应检查浏览器 Network、Nginx、Backend 日志、连接池和节点可达性。
+
+详细部署见 `docker/README.md`，测试见 `测试用例.md`，本次快照见 `docs/20260812/`。

docs/20260812/docker-readme.md

@@ -0,0 +1,58 @@
+# Docker 部署说明
+
+## 部署边界
+
+生产规划分为应用服务器、存储服务器和算力服务器。应用服务器运行 Frontend Nginx、Backend API、Redis；存储服务器运行 MinIO；算力服务器运行 Compute API、Compute Agent、GPU 和训练引擎。三类服务器不共享 Docker 网络，通过可路由地址通信。
+
+当前 WSL 联调地址为 `172.25.179.69`，拆分部署时必须替换为真实服务器地址。
+
+## 配置
+
+应用服务器 `docker/app/.env`：
+
+```env
+MINIO_ENABLED=true
+MINIO_ENDPOINT=http://172.25.179.69:19000
+COMPUTE_API_BASE_URL=http://172.25.179.69:19100
+FILE_GATEWAY_BASE_URL=http://172.25.179.69:19101
+COMPUTE_STATUS_SYNC_MODE=polling
+COMPUTE_POLL_INTERVAL_SECONDS=3
+```
+
+MinIO 不需要额外安装 Python 包；Backend 的 MinIO 兼容访问依赖随 `backend/requirements.txt` 安装。生产环境应使用固定 DNS/IP、内网访问和防火墙白名单。
+
+## 构建和启动
+
+```bash
+docker build -f docker/app/Dockerfile.backend -t yg-ft-backend-api:latest .
+docker build -f docker/app/Dockerfile.frontend -t yg-ft-frontend-runtime:latest .
+docker build -f docker/compute/Dockerfile.compute -t yg-ft-compute-api:latest .
+cd docker/minio && docker compose up -d
+cd ../app && docker compose up -d
+cd ../compute && docker compose up -d
+```
+
+验证命令：`docker compose ps`、`curl http://<app-server>:17861/modelTF/health`、`curl http://<compute-server>:19100/modelTF/health`、`curl http://<minio-server>:19000/minio/health/live`。
+
+## 端口
+
+| 服务 | 主机端口 | 容器端口 |
+| --- | ---: | ---: |
+| Frontend | 16801 | 80 |
+| Backend API | 17861 | 8000 |
+| Redis | 16379 | 6379 |
+| MinIO API/Console | 19000/19001 | 9000/9001 |
+| Compute API/File Gateway | 19100/19101 | 9100 |
+
+## 数据和安全
+
+算力主机应持久化 `/data/yg-ft/models`、`datasets`、`trained_models`、`outputs`、`cache` 以及训练日志目录。上述目录是 Compute Agent 的本地缓存和运行目录，权威对象必须归档 MinIO。
+
+- Compute API 和 File Gateway 开启 token 认证。
+- MinIO 不直接暴露公网，使用内网或安全组限制访问。
+- 不在镜像和 Git 中提交数据库、Redis、MinIO 密码或服务 token。
+- 当前 Compute API 默认 root 仅适用于开发阶段，生产环境需评估非 root 和 GPU/挂载目录权限改造。
+
+## 拆分验证
+
+从 Backend 容器验证 MinIO 和每个 Compute API 的 health；上传数据集、训练、权重合并、推理和节点断网重试均需完成一次联调。

docs/20260812/测试用例.md

@@ -0,0 +1,99 @@
+# YG_FT 平台测试用例
+
+## 测试范围
+
+覆盖部署、登录会话、权限、租户与项目隔离、算力节点、GPU 分配、数据集、MinIO、训练、权重合并、模型推理、模型评测、审计、日志、异常重试和性能。
+
+测试地址：前端 `http://localhost:16801`，Backend `http://localhost:17861/modelTF`，Compute `http://localhost:19100/modelTF`，File Gateway `http://localhost:19101/modelTF`，MinIO `http://localhost:19000`。
+
+## 前置条件
+
+1. 前端已执行 `npm run build`。
+2. Backend、Frontend、Redis、MinIO、Compute 容器均为 healthy。
+3. PostgreSQL 表结构与 `backend/app/db/sql/000_full_init.sql` 一致。
+4. 准备管理员、普通用户、不同项目和租户测试账号。
+5. 准备 JSON、JSONL、空文件和非法格式数据集。
+6. 准备 base model、adapter 和可推理模型。
+
+## 用例
+
+| 编号 | 场景 | 操作 | 预期 |
+| --- | --- | --- | --- |
+| DEP-001 | 容器启动 | 执行各 compose 的 `up -d` | 所有服务启动并 healthy |
+| DEP-002 | Backend 健康 | 访问 `/health` | HTTP 200，依赖状态正常 |
+| DEP-003 | Compute 健康 | 访问 Compute health | 返回节点服务正常 |
+| DEP-004 | MinIO 健康 | 访问 `/minio/health/live` | 返回存活状态 |
+| DEP-005 | 前端入口 | 打开 `:16801` | 无白屏、无外部 CDN 请求 |
+| DEP-006 | 跨服务器 | 配置远程 MinIO/Compute 地址 | Backend 可访问远程服务 |
+| AUTH-001 | 正常登录 | 输入正确账号密码 | 登录成功并保存 token |
+| AUTH-002 | 错误密码 | 连续输入错误密码 | 返回 401，达到阈值后限流 |
+| AUTH-003 | 过期会话 | 使用过期 token 请求接口 | 返回 401 并回登录页 |
+| AUTH-004 | 退出登录 | 退出后再次请求业务接口 | token 失效 |
+| AUTH-005 | 管理员 | 访问用户、节点、审计功能 | 可执行授权操作 |
+| AUTH-006 | 普通用户 | 访问管理员功能 | 按钮隐藏，后端返回 403 |
+| AUTH-007 | 项目隔离 | 用户 A 访问用户 B 项目 | 列表不显示，接口拒绝 |
+| AUTH-008 | 租户隔离 | 租户 A 请求租户 B 数据 | 不返回跨租户数据 |
+| AUTH-009 | ACL | 授予模型 read/execute | read 只能看，execute 才能运行 |
+| AUTH-010 | 软删除 | 删除模型或数据集 | 列表隐藏，保留删除审计字段 |
+| NODE-001 | 新增节点 | 填写 API、文件网关和标签 | 节点保存并显示 |
+| NODE-002 | 节点测试 | 点击测试 | health/GPU 信息同步 |
+| NODE-003 | 不可达节点 | 使用错误地址测试 | 快速失败并显示原因 |
+| NODE-004 | 删除节点 | 点击删除 | 节点从可用列表消失 |
+| NODE-005 | 多 GPU | 节点有多张卡 | 显示编号、显存和状态 |
+| NODE-006 | 指定 GPU | 训练选择 GPU 0 | 只占用 GPU 0 |
+| NODE-007 | GPU 冲突 | 两任务申请同卡 | 后者排队或拒绝，不抢占 |
+| NODE-008 | 剩余 GPU | 节点有空闲卡 | 其他任务可继续选择该节点 |
+| NODE-009 | 释放 GPU | 停止训练/卸载推理 | GPU 恢复可用 |
+| DATA-001 | JSON 统计 | 上传 3 条 JSON 数据 | 列表和详情均为 3 条 |
+| DATA-002 | JSONL 统计 | 上传 3 行 JSONL | 列表和详情均为 3 条 |
+| DATA-003 | 非法文件 | 上传空或错误格式 | 返回明确错误 |
+| DATA-004 | MinIO 归档 | 上传数据集 | 产生对象和 checksum |
+| DATA-005 | 节点同步 | 选择算力节点上传 | 文件进入目标节点缓存 |
+| DATA-006 | 同步断网 | 同步时阻断节点 | 进入重试或失败，不无限等待 |
+| DATA-007 | 数据权限 | 用户查看数据集 | 只显示有权限的数据 |
+| TRAIN-001 | 创建训练 | 选择项目、数据集、节点和 GPU | 任务关联完整 |
+| TRAIN-002 | 启动训练 | 启动任务 | 进入 queued/running |
+| TRAIN-003 | 日志轮询 | 打开训练日志 | 约 3 秒更新，不刷屏 |
+| TRAIN-004 | 训练曲线 | 产生 loss/metric | 页面显示曲线 |
+| TRAIN-005 | 停止训练 | 点击停止 | 进程停止且资源释放 |
+| TRAIN-006 | 训练失败 | 模拟引擎失败 | 显示原因和日志 |
+| MERGE-001 | 自动准备 | 执行权重合并 | 自动准备 base model/adapter |
+| MERGE-002 | 节点一致 | 权重在训练节点 | 合并请求发往训练节点 |
+| MERGE-003 | 合并归档 | 合并成功 | 结果上传 MinIO 并登记 |
+| MERGE-004 | 合并权限 | 无 execute 用户操作 | 返回 403 |
+| INF-001 | 列表加载 | 点击模型推理 | 列表快速显示，不长时间等待 |
+| INF-002 | 指定节点 | 多节点时选择节点 B | 模型只在 B 加载 |
+| INF-003 | 训练节点优先 | 未重新指定节点 | 优先使用训练节点 |
+| INF-004 | 推理缓存 | 启动未缓存模型 | 从 MinIO 下载到目标节点 |
+| INF-005 | 加载超时 | 模拟加载超过 15 分钟 | 失败并显示原因 |
+| INF-006 | 对话推理 | 发送消息 | 返回推理结果 |
+| INF-007 | 释放推理 | 点击释放 | 卸载模型并释放 GPU |
+| INF-008 | 删除推理 | 删除任务记录 | 记录删除成功并释放资源 |
+| EVAL-001 | 创建评测 | 选择模型、数据集、指标 | 任务创建成功 |
+| EVAL-002 | 数据集权限 | 选择无权数据集 | 不出现在选择列表 |
+| EVAL-003 | 指标保存 | 选择具体指标 | 结果不错误显示 custom |
+| EVAL-004 | 评测报告 | 等待任务完成 | 返回非空报告和明细 |
+| EVAL-005 | 页面轮询 | 打开评测页面 | 不刷屏，loading 可结束 |
+| OPS-001 | 审计 | 登录、创建、删除、执行资源 | 记录 actor/action/resource/time |
+| OPS-002 | 轮询日志 | 观察 Backend 日志 | 成功轮询不高频输出 INFO |
+| OPS-003 | 错误日志 | 模拟依赖异常 | 保留 WARNING/ERROR 和 request_id |
+| OPS-004 | 推理接口耗时 | 请求 `/model-compare` | 正常环境目标小于 1 秒 |
+| OPS-005 | 看板耗时 | 请求 `/dashboard/stats` | 有短缓存且不无限等待 |
+| OPS-006 | 并发访问 | 10 用户同时打开列表 | 无连接池耗尽和 5xx |
+| OPS-007 | 数据库断开 | 临时阻断 PostgreSQL | 页面明确显示依赖异常 |
+
+## 回归命令
+
+```bash
+cd frontend
+npm run build
+cd ..
+python -m compileall backend/app compute
+git diff --check
+docker compose -f docker/app/docker-compose.yml ps
+docker compose -f docker/compute/docker-compose.yml ps
+curl http://localhost:17861/modelTF/health
+curl http://localhost:19100/modelTF/health
+```
+
+失败用例必须附接口响应、容器日志、request_id 和复现步骤。

docs/data-process-design.md

@@ -118,7 +118,7 @@ pending ──start/generate──> running ──success──> completed
   抽取幻灯片文本与表格，随后统一进入切片算法。
 - 旧版二进制 DOC、XLS、PPT 不直接解析，返回 415 并提示分别转换为
   DOCX、XLSX、PPTX。
-- 扫描 PDF 没有文本层时明确提示需要 OCR；当前流程不执行 OCR。加密、损坏或
+- 扫描 PDF 没有文本层时明确提示扫描版或图片型 PDF 不支持。加密、损坏或
   超出页数/工作表/行列/解压规模限制的文件整批拒绝。
 
 现代 Office 文件在交给解析库前检查 ZIP 成员路径、重复成员、加密标记、活动

docs/database-config.md

@@ -0,0 +1,164 @@
+# 数据库配置与初始化说明（PostgreSQL / Redis）
+
+> 记录平台的 **PostgreSQL 账号密码**、**Redis 账号密码**、**数据库地址在代码中的配置位置**，
+> 以及**切换 PG 数据集时如何执行完整初始化 SQL**。
+
+---
+
+## 1. 账号密码速查表
+
+### 1.1 PostgreSQL
+
+| 环境 | 地址 | 用户 | 密码 | 数据库 | 来源 |
+|------|------|------|------|--------|------|
+| 代码默认值 | `localhost:15432` | `yg_ft` | `change_me` | `yg_ft` | `config.py` / `session.py` 的 `DATABASE_URL` 兜底 |
+| Docker 部署 | `www.caoxiaozhu.com:5432` | `root` | `8811614287327Leo` | `yg_ft` | `docker/app/.env` 的 `DATABASE_URL` |
+| Docker 内置 Postgres（已注释） | `localhost:15432` | `root` | `8811614287327Leo` | `yg_ft` | `docker/app/docker-compose.yml` 注释掉的 postgres 服务 |
+
+> ⚠️ `change_me` 与 `8811614287327Leo` 均为默认/示例凭据，生产环境务必更换。
+
+### 1.2 Redis
+
+| 项 | 值 | 说明 |
+|----|-----|------|
+| 连接串 | `redis://:<REDIS_PASSWORD>@redis:6379/0` | 已内嵌密码；docker 网络内服务名 `redis`，端口 6379，db 0 |
+| 对外端口 | `16379`（`REDIS_PORT`） | 宿主机映射 |
+| 密码 | `docker/app/.env` 的 `REDIS_PASSWORD` | 已启用 `requirepass` 鉴权 |
+| 镜像 | `redis:7-alpine` | 已开启 AOF（`--appendonly yes`）+ `requirepass` |
+
+> **当前后端代码未使用 Redis**：`redis` 包已列入 `requirements.txt`，`REDIS_URL` 通过
+> docker-compose 注入容器，但全仓库 `backend/`、`compute/` 没有任何 `import redis` /
+> `Redis(...)` 连接代码。Redis 为后续功能预留；**即便如此仍已配置鉴权**，
+> 避免无密码实例对外暴露（纵深防御）。未来启用时按 `REDIS_URL` 连接即可。
+>
+> 健康检查通过容器环境变量 `REDISCLI_AUTH` 认证，不在进程参数中泄露密码。
+
+---
+
+## 2. 数据库地址在代码中的配置位置
+
+### 2.1 后端（backend）
+
+| 文件 | 作用 | 取值 |
+|------|------|------|
+| `backend/app/core/config.py` | **唯一权威配置**，`Settings.database_url` | `os.getenv("DATABASE_URL", "postgresql+psycopg://yg_ft:change_me@localhost:15432/yg_ft")` |
+| `backend/app/db/session.py` | SQLAlchemy 引擎（`get_db` / `session_scope`） | `os.getenv("DATABASE_URL", ...)` 同样兜底 |
+| `backend/app/db/platform_store.py` | **平台主存储**，直接用 psycopg 连接池 | 取 `settings.database_url`，`_psycopg_url()` 把 `postgresql+psycopg://` 转成 `postgresql://` |
+| `backend/app/modules/data_process/store.py` | 数据处理存储 | `get_settings().database_url` |
+
+> **环境变量加载顺序**：`config.py` 导入时会 `load_dotenv(backend/.env, override=True)`，
+> 即 **`backend/.env` 会覆盖系统环境变量**；docker 部署则直接由 compose 注入 `DATABASE_URL`。
+> 最终优先级：`backend/.env` / compose 注入的环境变量 > 代码内默认值。
+
+### 2.2 部署配置（docker）
+
+| 文件 | 关键项 |
+|------|--------|
+| `docker/app/.env` | `DATABASE_URL`、`POSTGRES_USER`、`POSTGRES_PASSWORD`、`REDIS_URL`、`REDIS_PORT` |
+| `docker/app/docker-compose.yml` | `backend-api` 环境透传上述变量；`redis` 服务定义 |
+
+```ini
+# docker/app/.env（节选）
+DATABASE_URL=postgresql+psycopg://root:8811614287327Leo@www.caoxiaozhu.com:5432/yg_ft
+POSTGRES_USER=root
+POSTGRES_PASSWORD=8811614287327Leo
+REDIS_PASSWORD=<强密码>                    # Redis requirepass（新增鉴权）
+REDIS_URL=redis://:<REDIS_PASSWORD>@redis:6379/0   # 连接串内嵌密码
+REDIS_PORT=16379
+USE_BUILTIN_POSTGRES=false   # 当前用共享外部库，内置 postgres 服务被注释
+USE_BUILTIN_REDIS=true       # Redis 用内置服务
+```
+
+---
+
+## 3. 完整初始化 SQL
+
+### 3.1 脚本位置
+
+| 脚本 | 用途 |
+|------|------|
+| **`backend/app/db/sql/000_full_init.sql`** | **一键初始化脚本（新增）**：建库表 + 索引 + 种子数据，幂等，覆盖全部 39 张运行表 |
+| `backend/app/db/sql/001_platform_runtime.sql` | 平台核心表（应用启动自动执行） |
+| `backend/app/db/sql/002_governance.sql` | 治理表（应用启动自动执行） |
+| `backend/app/db/sql/003_tenant_quota.sql` | 租户配额列（应用启动自动执行） |
+| `backend/app/db/sql/003_model_path_governance.sql` | 模型可训练列（**应用不自动执行**，已并入完整脚本） |
+| `backend/app/db/sql/002_data_process.sql` | 数据处理表（**应用不自动执行**，已并入完整脚本） |
+| `docs/postgres-schema.sql` | ⚠️ **目标设计稿**（UUID/JSONB），与运行时代码不兼容，**不要用于初始化** |
+
+> **重要**：`docs/postgres-schema.sql` 是规划中的“目标 schema”（UUID 主键、`ft_platform` schema 等），
+> 运行时代码明确拒绝该结构（`002_data_process.sql` 检测到 `datasets.id` 非 TEXT 会直接报错）。
+> 初始化请使用 **`000_full_init.sql`**。
+
+### 3.2 执行步骤（全新 PG 环境）
+
+**第 1 步：创建角色与数据库**（必须单独执行，不能放进事务）
+
+```sql
+-- 以超级用户（如 postgres）连接：
+CREATE ROLE  yg_ft LOGIN PASSWORD '请改为强密码';
+CREATE DATABASE yg_ft OWNER yg_ft;
+-- 如需应用执行 CREATE EXTENSION 等，可再授予超级用户（按需）：
+-- ALTER ROLE yg_ft SUPERUSER;
+```
+
+**第 2 步：执行完整初始化脚本**
+
+```bash
+psql "postgresql://yg_ft:密码@<host>:5432/yg_ft" \
+     -f backend/app/db/sql/000_full_init.sql
+```
+
+脚本特点：
+- 全程一个事务（`BEGIN; ... COMMIT;`），失败自动回滚
+- 所有 DDL 使用 `IF NOT EXISTS` / `ADD COLUMN IF NOT EXISTS`，**可重复执行**
+- 含 `DO $$...$$` 语句块，必须用 `psql` 执行（应用内部的按分号切分 `executescript()` 不适用）
+- 自动写入种子用户：`admin / admin123`、`operator / operator123`（登录后请改密）
+
+**第 3 步：校验**
+
+```sql
+SELECT count(*) FROM pg_tables WHERE schemaname = 'public';          -- 应 ≥ 39
+SELECT username, role, status FROM users;                            -- 应有 admin / operator
+```
+
+### 3.3 执行方式对比（三种途径）
+
+| 方式 | 覆盖范围 | 命令 |
+|------|----------|------|
+| **A. 完整脚本（推荐，切换新库）** | 全部 39 表 + 索引 + 种子 | `psql ... -f 000_full_init.sql` |
+| B. 应用自动初始化 | 001 + 002_governance + 003_tenant_quota + 种子用户；**不含**数据处理表、`models.can_train`、`data_convert_tasks` | 应用首次调用 `get_platform_store()` 时 `ensure_schema()` 自动执行 |
+| C. 数据处理表单独安装 | `002_data_process.sql` 全部内容 | 在 `backend/` 目录下：`python -m app.modules.data_process.schema_cli --apply --yes`（或 `--check` 只读检查） |
+
+> **缺口说明**：
+> - `models.can_train`（训练预检用）只在 `003_model_path_governance.sql` 中创建，应用启动**不会**自动执行；
+> - `data_convert_tasks`（数据转换任务表）运行时代码引用但**原 SQL 脚本缺失**；
+>   已统一并入 `000_full_init.sql` 补齐。若现有库缺这两项，执行一次完整脚本即可幂等补上。
+
+### 3.4 完整脚本包含的表（39 张）
+
+**核心**：users、models、trained_models、model_lineage、model_artifacts、model_export_jobs、
+datasets、dataset_files、compute_nodes、gpus、fine_tune_tasks、fine_tune_metrics、
+fine_tune_checkpoints、compute_jobs、gpu_allocations、scheduler_locks、resource_replicas、
+resource_sync_jobs、eval_tasks、eval_dimensions、compare_tasks、projects、project_members、
+roles、sessions、acls
+
+**治理**：tenants、approval_templates、approval_instances、approval_steps、audit_logs、retention_policies
+
+**数据处理**：data_process_tasks、data_process_source_files、data_process_preview_items、
+data_process_results、dataset_file_versions、dataset_records
+
+**数据转换**：data_convert_tasks（新增补齐）
+
+---
+
+## 4. 安全注意事项
+
+1. **更换默认密码**：`change_me`（代码兜底）、`8811614287327Leo`（部署）、`admin123`/`operator123`（种子用户）、`REDIS_PASSWORD` 上线前必须更换。
+2. **Redis 已加鉴权**：已配置 `requirepass` + `REDISCLI_AUTH` 健康检查；`REDIS_URL` 内嵌密码。若端口需暴露公网，仍建议用防火墙/安全组限制来源。
+3. **`docker/*/.env` 已入库，含明文凭据**：
+   - `backend/.env` 已被 `.gitignore` 排除；
+   - 但 `docker/app/.env`、`docker/compute/.env` 目前被 git 跟踪（`git ls-files` 可见），
+     其中的 `DATABASE_URL`、`POSTGRES_PASSWORD`、`COMPUTE_SERVICE_TOKEN` 等均为明文。
+   - **建议**：轮换这些凭据，将 `docker/*/.env` 移出版本库（`git rm --cached`）并改用
+     部署侧机密注入（如 docker secrets / CI 变量 / 环境变量模板），保留 `.env.example` 作为模板。
+4. **最小权限**：应用角色只需对业务库的 DML/DDL 权限，尽量避免 SUPERUSER。

docs/governance-user-guide.md

@@ -0,0 +1,436 @@
+# 平台治理功能使用指南
+
+> 版本：v1.1
+> 日期：2026-08-13
+> 适用版本：YG Fine-Tune Platform v1.0+
+> 更新说明：移除页面权限码设计，改为基于角色的简化权限模型
+
+---
+
+## 目录
+
+1. [快速入门](#1-快速入门)
+2. [用户与权限管理](#2-用户与权限管理)
+3. [GPU 算力分配](#3-gpu-算力分配)
+4. [资源可见性与隔离](#4-资源可见性与隔离)
+5. [审批流程管理](#5-审批流程管理)
+6. [审计日志查询](#6-审计日志查询)
+7. [常见问题与排查](#7-常见问题与排查)
+
+---
+
+## 1. 快速入门
+
+### 1.1 平台治理是什么？
+
+平台治理是一套**多租户、多角色、细粒度权限控制**体系，用于在多人协作使用 AI 微调平台时，确保：
+
+- 每个用户只能看到和操作自己有权限的资源
+- GPU 算力按需分配，避免资源争抢
+- 高风险操作（删除、停止任务）有审批记录可追溯
+- 所有操作都有审计日志
+
+### 1.2 三种内置角色
+
+| 角色 | 能做什么 | 不能做什么 |
+|---|---|---|
+| **超级管理员 (admin)** | 全部操作；管理用户、分配 GPU、审批、查看全部资源 | — |
+| **操作员 (operator)** | 创建数据集/模型、训练/评测/推理任务 | 管理用户、分配 GPU、修改他人权限 |
+| **观察员 (viewer)** | 查看被授权的资源 | 创建或修改任何资源 |
+
+### 1.3 入口在哪里？
+
+所有治理功能集中在左侧导航栏的 **「系统设置」** 和 **「平台治理」** 分组下：
+
+```
+系统设置
+├── 用户设置        ← 用户 CRUD + 角色权限 + 密码管理（仅 admin）
+├── 平台性能        ← 系统监控
+└── 查看日志        ← 日志查看
+
+平台治理
+├── 租户管理        ← 组织/团队（仅 admin）
+├── 项目空间        ← 项目级资源隔离（仅 admin）
+├── 审批模板        ← 定义哪些操作需要审批（仅 admin）
+├── 审批中心        ← 处理待审批请求（仅 admin）
+└── 审计日志        ← 查看所有操作记录（仅 admin）
+
+算力资源
+└── 算力节点        ← GPU 分配与管理（仅 admin）
+```
+
+> ⚠️ 以上菜单**只有 admin 用户能看到**。普通用户登录后不会出现这些入口。
+>
+> **重要变更（v1.1）**：非 admin 用户**默认可以访问所有业务功能菜单**（模型训练、评测、推理、数据集、数据处理等），无需管理员单独分配权限。
+
+---
+
+## 2. 用户与权限管理
+
+### 2.1 权限模型概述（v1.1 更新）
+
+平台采用**基于角色的简化权限模型**：
+
+| 用户类型 | 可见菜单 | 说明 |
+|---------|---------|------|
+| **admin（管理员）** | **全部菜单** | 包括用户设置、平台治理、算力节点等管理功能 |
+| **非 admin 用户** | **除管理功能外的所有业务菜单** | 模型训练/评测/推理、数据集、数据处理、日志等 |
+
+> **核心原则**：
+> - 非 admin 用户**默认拥有所有业务功能的访问权限**，无需单独分配
+> - 仅以下功能**仅管理员可见**：
+>   - `用户设置`（用户 CRUD、角色管理）
+>   - `平台治理`（租户管理、项目空间、审批模板/中心、审计日志）
+>   - `算力节点`（GPU 分配）
+>
+> 资源级别的访问控制通过 **ACL（访问控制列表）** 实现，详见第 4 章。
+
+### 2.2 创建用户
+
+**路径**：`用户设置` → `创建用户`
+
+1. 以 admin 身份登录平台
+2. 进入「用户设置」页面
+3. 点击右上角「创建用户」按钮
+4. 填写信息：
+   - **账号**：登录用户名（如 `zhangsan`）
+   - **显示名称**：如 `张三`
+   - **密码**：初始密码（默认 `Platform@123`）
+   - **角色**：选择 `admin` / `operator` / `viewer`
+5. 点击保存
+
+创建后用户可以立即用该账号登录，**无需额外分配页面权限**。
+
+### 2.3 管理员专属功能
+
+以下功能**仅 admin 角色可见**，对其他用户隐藏：
+
+| 功能分组 | 包含菜单 | 路由前缀 |
+|---------|---------|----------|
+| 系统设置 - 用户设置 | 用户列表、创建用户、重置密码 | `/user-settings` |
+| 平台治理 - 租户管理 | 租户列表、配额设置 | `/tenants` |
+| 平台治理 - 项目空间 | 项目列表、成员管理、ACL | `/projects` |
+| 平台治理 - 审批模板 | 审批流程定义 | `/approval-templates` |
+| 平台治理 - 审批中心 | 待审批请求处理 | `/approval-instances` |
+| 平台治理 - 审计日志 | 操作记录查询与导出 | `/audit-logs` |
+| 算力资源 - 算力节点 | GPU 分配与管理 | `/compute` |
+
+### 2.4 重置用户密码
+
+**两种方式**：
+
+**方式一：管理员重置**
+1. 在用户列表中找到目标用户
+2. 点击「重置密码」
+3. 输入新密码，确认
+
+**方式二：用户自行修改**
+1. 用户登录后在「用户设置」页面点击「修改密码」按钮
+2. 输入旧密码 + 新密码（至少 6 位）
+3. 确认修改
+
+### 2.5 删除用户
+
+**路径**：`用户设置` → 用户列表 → 操作列「删除」
+
+> ⚠️ 删除用户时会**级联清理**其所有关联数据：
+> - 该用户创建的数据集、基座模型、微调产物、评测任务
+> - 该用户的 ACL 授权记录、GPU 分配记录
+> - 该用户的审批实例、审计日志、项目成员关系、登录会话
+> - **训练任务保留不删**（避免算力节点上的物理任务数据不一致）
+
+---
+
+## 3. GPU 算力分配
+
+### 3.1 为什么需要 GPU 分配？
+
+当服务器有多张 GPU 卡（如 8×A800）时，需要指定**哪个用户能用哪张卡**：
+
+- 避免两个人同时选同一张卡导致训练冲突
+- 按团队/项目隔离算力资源
+- 控制每个用户的 GPU 配额
+
+### 3.2 分配 GPU（仅 admin）
+
+**路径**：`算力节点` → `GPU 分配` 标签页
+
+1. 以 admin 登录，进入「算力节点」页面
+2. 点击顶部的 **「GPU 分配」** 标签（只有 admin 可见）
+3. 点击 **「分配 GPU」** 按钮
+4. 填写：
+   - **算力节点**：选择节点（如 `gpu-node-01`）
+   - **GPU 序号**：卡号（0, 1, 2, ... 7）
+   - **用户**：选择要分配给谁
+5. 点「确认分配」
+
+示例：把节点 `gpu-node-01` 的第 0、1 号卡分配给用户 `zhangsan`：
+
+```
+┌─────────────────────────────────────┐
+│  分配 GPU                    [×]  │
+├─────────────────────────────────────┤
+│  算力节点:  [gpu-node-01 ▼]         │
+│  GPU 序号:  [0          ▲]         │
+│  用户:      [zhangsan    ▼]         │
+│                                     │
+│       [取消]     [确认分配]         │
+└─────────────────────────────────────┘
+```
+
+分配后的效果：
+
+| 用户 | 可用 GPU |
+|---|---|
+| admin | 全部 GPU（不需要显式分配） |
+| zhangsan | gpu-node-01 的 0、1 号卡 |
+| lisi | （未分配，不可用） |
+
+### 3.3 撤销分配
+
+在 GPU 分配列表中，每条记录右侧有「撤销」按钮，点击后确认即可移除该分配。
+
+### 3.4 用户视角：创建训练任务时的 GPU 选择
+
+- **admin**：下拉列表显示全部可用 GPU
+- **被分配了 GPU 的用户**：只显示被分配给自己的卡
+- **未分配任何 GPU 的用户**：显示提示「未分配 GPU，请联系管理员」，无法提交训练任务
+
+---
+
+## 4. 资源可见性与隔离
+
+### 4.1 自动生效的隔离规则
+
+无需手动配置，以下规则自动生效：
+
+| 资源类型 | admin 看到 | 普通用户看到 |
+|---|---|---|
+| **基座模型**（容器内注册的本地模型） | 全部 | **全部**（共享资源） |
+| **数据集** | 全部 | **自己创建的** + 被 ACL 授权的 |
+| **微调产物**（训练输出的模型） | 全部 | **自己训练的** + 被 ACL 授权的 |
+| **评测任务** | 全部 | **自己创建的** + 被 ACL 授权的 |
+| **推理/对比任务** | 全部 | **自己创建的** + 被 ACL 授权的 |
+
+### 4.2 实际场景示例
+
+假设有三个用户：**admin**、**zhangsan**（算法工程师）、**lisi**（标注员）
+
+```
+zhangsan 上传了数据集 ds_alpaca、ds_sharegpt
+zhangsan 训练出了模型 ft_qwen_001
+lisi 上传了数据集 ds_label
+admin 注册了基座模型 Qwen3-1.7B
+```
+
+各用户看到的资源：
+
+| 用户 | 数据集 | 基座模型 | 微调产物 |
+|---|---|---|---|
+| **admin** | ds_alpaca, ds_sharegpt, ds_label (3个) | Qwen3-1.7B | ft_qwen_001 |
+| **zhangsan** | ds_alpaca, ds_sharegpt (2个) | Qwen3-1.7B | ft_qwen_001 |
+| **lisi** | ds_label (1个) | Qwen3-1.7B | (无) |
+
+### 4.3 ACL 资源授权（高级用法）
+
+如果 zhangsan 想让 lisi 也能看到自己的数据集 `ds_alpaca`：
+
+> 此功能需要在资源详情页提供「资源授权」按钮（前端已预留接口），当前可通过 API 直接操作：
+
+```bash
+# 授予 lisi 对 ds_alpaca 的读权限
+curl -X PUT /modelTF/resources/dataset/ds_alpaca_id/acl \
+  -H "Authorization: Bearer platform-token-admin" \
+  -d '{
+    "acls": [
+      {"principal_type": "user", "principal_id": "lisi_id", "permission": "read"}
+    ]
+  }'
+```
+
+---
+
+## 5. 审批流程管理
+
+### 5.1 哪些操作会触发审批？
+
+| 操作 | 触发条件 | 处理方式 |
+|---|---|---|
+| 删除他人的数据集 | 非 admin 删除别人创建的数据集 | 创建审批实例 或 admin 直接执行 |
+| 删除他人的模型 | 非 admin 删除别人创建的模型 | 同上 |
+| 停止他人的训练任务 | 非 admin 停止别人发起的任务 | 同上 |
+| 归档/删除项目空间 | 存在待审批变更时 | 拒绝执行 |
+
+**核心规则**：admin 做任何操作都直接执行（旁路）；普通用户操作他人资源时进入审批流程。
+
+### 5.2 审批流程示意
+
+```
+普通用户 lisi 尝试删除 zhangsan 的数据集
+                │
+                ▼
+    ┌───────────────────────┐
+    │ 后端检查：是 admin 吗？ │
+    └──────┬────────────────┘
+           │ 否
+           ▼
+    ┌───────────────────────┐
+    │ 创建审批实例            │
+    │ status = pending       │
+    │ 返回 202（待审批）     │
+    └──────────┬────────────┘
+               │
+               ▼
+    ┌───────────────────────┐
+    │ admin 在「审批中心」看到 │
+    │ 这条待审批请求           │
+    │ 点击「通过」或「拒绝」   │
+    └──────────┬────────────┘
+           │
+     ┌─────┴─────┐
+     │           │
+  通过        拒绝
+     │           │
+     ▼           ▼
+ 执行删除    不执行
++审计日志   +审计日志
+```
+
+### 5.3 管理审批
+
+**路径**：`平台治理` → `审批中心`
+
+1. 查看待审批列表（status=pending）
+2. 点击某条记录查看详情
+3. 决策：「通过」或「拒绝」
+4. 决策结果自动执行对应操作并记录审计日志
+
+**审批模板**（`平台治理` → `审批模板`）：定义每种操作需要几步审批、每步谁来审。默认模板都是单步（admin 审批即可）。
+
+---
+
+## 6. 审计日志查询
+
+### 6.1 什么是审计日志？
+
+平台上所有**写操作**和**敏感操作**都会自动记录审计日志，包括：
+
+- 用户创建/删除/修改
+- 权限变更
+- GPU 分配/撤销
+- 资源上传/删除
+- 训练任务启动/停止
+- 审批决策
+
+### 6.2 查询审计日志
+
+**路径**：`平台治理` → `审计日志`
+
+支持筛选条件：
+
+| 筛选项 | 说明 |
+|---|---|
+| 操作人 | 按用户 ID 过滤 |
+| 动作类型 | 如 `user.create`, `dataset.delete`, `gpu.assign` 等 |
+| 目标资源类型 | dataset / model / fine_tune_task 等 |
+| 时间范围 | 开始时间 ~ 结束时间 |
+
+### 6.3 导出审计日志
+
+审计日志页面底部有「导出 CSV」按钮，导出的文件包含当前筛选条件下的全部记录，可用于合规审计或问题追溯。
+
+### 6.4 日志保留策略
+
+审计日志受**留存策略**控制（`平台治理` → 租户管理 → 绑定留存策略）。默认保留 30 天，超期自动清理。
+
+---
+
+## 7. 常见问题与排查
+
+### Q1: 普通用户看不到某个菜单？
+
+根据 v1.1 权限模型：
+1. **业务菜单**（训练、评测、推理、数据集等）：普通用户**默认全部可见**，无需分配
+2. **管理菜单**（用户设置、租户管理、算力节点等）：**仅 admin 可见**，这是设计如此
+
+如果普通用户看不到业务菜单，请检查：
+- 用户是否正常登录（token 是否有效）
+- 用户状态是否为 `active`（未被停用）
+
+### Q2: 用户创建训练任务时报错"无权使用所选 GPU"
+
+说明该用户没有被分配所选择的 GPU 卡。解决方法：
+1. admin 进入「算力节点」→「GPU 分配」标签页
+2. 为该用户分配对应的 GPU
+3. 用户刷新页面重新选择 GPU
+
+### Q3: 删除用户后看板还显示残留数据？
+
+正常情况下 `delete_user` 会级联清理关联数据。如果仍有残留：
+- **训练任务**：设计上保留不删（避免算力节点物理数据不一致），这是预期行为
+- **登录时长排行**：可能来自旧的 session 记录（已修复：改用 INNER JOIN 过滤已删除用户）
+
+### Q4: 审批实例一直 pending 没人处理？
+
+审批实例需要 admin 在「审批中心」手动处理。如果长时间无人处理：
+- 可以在数据库中直接将 `approval_instances.status` 改为 `rejected`
+- 或者由 admin 直接以自身身份执行该操作（admin 有旁路权限）
+
+### Q5: 如何查看当前所有 GPU 分配情况？
+
+```bash
+# admin 调用接口
+curl -H "Authorization: Bearer platform-token-admin" \
+  /modelTF/compute/gpu-assignments
+```
+
+返回格式：
+```json
+{
+  "data": [
+    {
+      "id": "ga_xxx",
+      "node_name": "A800 训练节点",
+      "gpu_index": 0,
+      "display_name": "张三",
+      "assigned_at": "2026-08-10T10:00:00"
+    }
+  ]
+}
+```
+
+### Q6: 用户忘记密码怎么办？
+
+两种方案：
+1. **admin 重置**：在「用户设置」→ 用户列表 →「重置密码」
+2. **用户自助修改**：用户登录后点击「修改密码」（需知道旧密码）
+
+如果是完全忘记且不是 admin，只能由 admin 重置。
+
+### Q7: 为什么移除了「页面权限」功能？（v1.1 变更说明）
+
+旧版本要求管理员为每个用户单独分配页面权限码，这导致：
+- 新用户创建后需要额外操作才能使用系统
+- 权限配置复杂，容易出错
+- 与实际使用场景不匹配（大多数用户需要访问大部分功能）
+
+**新模型（v1.1）**简化为：
+- 非 admin 用户**默认拥有所有业务功能的访问权限**
+- 仅管理员专属功能（用户管理、平台治理、算力节点）受角色限制
+- 资源级别控制通过 ACL 实现，更灵活
+
+---
+
+## 附录：API 快速参考
+
+| 功能 | 方法 | 路径 | 鉴权 |
+|---|---|---|---|
+| 查看我的 GPU | GET | `/compute/my-gpus` | 登录用户 |
+| 查看 GPU 分配 | GET | `/compute/gpu-assignments` | admin |
+| 分配 GPU | POST | `/compute/gpu-assignments` | admin |
+| 撤销 GPU 分配 | DELETE | `/compute/gpu-assignments/{id}` | admin |
+| 修改自己的密码 | POST | `/users/me/password` | 登录用户 |
+| 查看审计日志 | GET | `/system/audit-logs` | admin |
+| 导出审计日志 | GET | `/system/audit-logs/export` | admin |
+| 查看审批列表 | GET | `/approvals` | 登录用户 |
+| 审批决策 | POST | `/approvals/:id/steps/:idx/decision` | 审批人 |

docs/minio-compute-cache-plan.md

@@ -0,0 +1,287 @@
+# MinIO + Compute Agent 本地缓存方案与开发计划
+
+## 1. 方案结论
+
+本方案用 MinIO 作为模型、数据集、checkpoint、评测结果和训练产物的唯一正式存储，算力节点不挂载 NFS，也不要求安装 NFS 客户端。
+
+算力节点只保留任务运行所需的本地缓存：
+
+```text
+MinIO
+  -> Backend API 生成资源授权和版本信息
+  -> Compute Agent 按任务下载到本地缓存
+  -> LLaMA-Factory / 推理 / 合并任务使用本地路径
+  -> 任务产物上传回 MinIO
+  -> PostgreSQL 更新资源和任务状态
+```
+
+MinIO 是唯一正式数据源，本地缓存不是正式资源，节点失效或缓存被清理不会丢失模型和数据。
+
+## 2. 为什么适合当前项目
+
+当前项目的 Compute API 已经负责：
+
+- 训练、评测、推理和权重合并任务启动；
+- 本地文件网关；
+- GPU 和任务状态管理；
+- 训练日志和产物路径管理。
+
+因此不需要把 LLaMA-Factory 改成直接读取远程对象，只需要在 Compute Agent 启动任务前准备本地路径，继续把原来的 `model_name_or_path`、`dataset_dir`、`output_dir` 传给训练引擎。
+
+## 3. MinIO 部署
+
+MinIO 独立部署在 Linux 存储服务器或专用存储节点：
+
+```text
+9000  S3 API
+9001  MinIO Console（仅管理员网络开放）
+```
+
+建议创建 bucket：
+
+```text
+yg-ft-resources
+```
+
+对象前缀建议：
+
+```text
+models/{model_id}/versions/{version_id}/...
+datasets/{dataset_id}/versions/{version_id}/...
+outputs/{task_id}/...
+evaluations/{task_id}/...
+logs/{task_id}/...
+```
+
+MinIO 可以使用官方 Docker 镜像，不需要在 Linux 主机安装 MinIO 软件包。需要持久化挂载 MinIO 的数据目录。
+
+## 4. 是否需要额外安装包
+
+### MinIO 服务端
+
+不需要安装额外系统包，使用 Docker 镜像即可：
+
+```text
+minio/minio
+```
+
+### Backend API
+
+建议增加 Python SDK：
+
+```text
+minio>=7.2.0
+```
+
+Backend 用 SDK 生成预签名上传、下载 URL，并负责 bucket、对象元数据和权限控制。
+
+### Compute API / Compute Agent
+
+推荐第一版只使用现有 `httpx` 访问预签名 URL，不额外安装 MinIO SDK。流程是：
+
+```text
+Backend -> 返回预签名 URL
+Compute Agent -> httpx 下载/上传
+```
+
+这样算力节点不需要 MinIO 客户端、AWS CLI 或 NFS 客户端。
+
+如果后续需要 Agent 直接操作 bucket、列目录或分片上传，再增加：
+
+```text
+minio>=7.2.0
+```
+
+但不建议第一阶段让算力节点持有 MinIO 管理密钥。
+
+## 5. 权限模型
+
+继续沿用当前项目的：
+
+- `projects`；
+- `project_members`；
+- `acls`。
+
+MinIO 只负责对象访问凭证，Backend 负责业务授权：
+
+1. 用户请求模型或数据集；
+2. Backend 校验项目成员关系和资源 ACL；
+3. 校验通过后生成短时预签名 URL；
+4. Compute Agent 使用 URL 下载；
+5. URL 过期后自动失效。
+
+MinIO bucket 不直接向前端或普通算力节点开放长期 Access Key。
+
+## 6. 本地缓存目录
+
+Compute API 容器继续以 root 运行，本地缓存挂载到：
+
+```text
+/data/yg-ft/cache/models
+/data/yg-ft/cache/datasets
+/data/yg-ft/cache/adapters
+/data/yg-ft/cache/outputs
+```
+
+每个缓存资源必须包含：
+
+```text
+resource_id
+version_id
+sha256
+byte_size
+last_used_at
+status
+```
+
+缓存状态：
+
+- `missing`：本地不存在；
+- `downloading`：正在下载；
+- `ready`：校验成功；
+- `corrupted`：校验失败；
+- `evicting`：正在清理。
+
+任务只能使用 `ready` 状态的缓存。
+
+## 7. 任务流程
+
+### 7.1 训练
+
+```text
+校验项目/用户权限
+  -> 获取基座模型版本和数据集版本
+  -> 检查本地缓存
+  -> 缺失则下载并校验 SHA256
+  -> 启动 LLaMA-Factory
+  -> checkpoint 写入本地临时目录
+  -> 任务完成后上传 outputs 到 MinIO
+  -> MinIO 上传完成并校验后更新数据库
+```
+
+### 7.2 推理
+
+```text
+校验模型权限
+  -> 下载或复用本地模型缓存
+  -> 使用本地模型路径加载
+  -> 推理服务只绑定当前节点缓存
+```
+
+### 7.3 权重合并
+
+```text
+下载 base model 和 adapter/checkpoint
+  -> 在指定节点执行 CPU 合并
+  -> 上传 merged model 到 MinIO
+  -> 数据库记录新的模型版本
+```
+
+### 7.4 NFS 故障规则对应关系
+
+MinIO 不可达时，节点不再依赖本地残留文件直接启动新任务：
+
+- 已有完整缓存且资源版本仍有效：允许继续执行当前任务；
+- 新任务无法确认资源版本：等待 MinIO 恢复；
+- 等待超过配置窗口：任务失败；
+- 产物无法上传：任务不得标记为最终成功，进入 `storage_error`。
+
+如果严格执行“共享存储故障时节点不能正常工作”，则即使本地缓存完整，也应禁止启动新任务。建议当前项目采用这一严格规则。
+
+## 8. 数据库建议
+
+现有 `resource_replicas` 可扩展为缓存索引，建议增加：
+
+```text
+storage_backend       -- minio
+storage_bucket
+storage_object_key
+version_id
+cache_path
+cache_status
+last_used_at
+download_progress
+```
+
+`resource_sync_jobs` 可继续用于下载和上传任务，但建议增加方向字段：
+
+```text
+direction -- download / upload
+```
+
+模型、数据集、checkpoint 和评测结果均使用 `resource_id + version_id`，不再把节点本地路径作为唯一资源标识。
+
+## 9. 开发计划
+
+### 阶段一：MinIO 服务和配置
+
+- 增加 `docker/minio/docker-compose.yml`；
+- 配置 MinIO endpoint、bucket、Access Key 和 Secret Key；
+- 增加 Backend `minio` 依赖；
+- 增加 MinIO 健康检查；
+- 创建统一 bucket 和对象前缀规则。
+
+### 阶段二：Backend 资源服务
+
+- 实现对象上传、下载、删除和 HEAD 校验；
+- 生成短时预签名 URL；
+- 接入项目成员和 ACL 校验；
+- 建立资源版本、SHA256 和大小记录；
+- 上传成功后再更新数据库资源状态。
+
+### 阶段三：Compute Agent 缓存服务
+
+- 增加缓存目录管理器；
+- 实现预签名 URL 下载；
+- 支持临时文件下载和原子改名；
+- 实现 SHA256 校验、失败重试和断点续传；
+- 增加缓存状态查询和清理接口。
+
+### 阶段四：接入业务任务
+
+- 训练前准备基座模型和数据集；
+- 推理前准备模型和 adapter；
+- 权重合并前准备 base model 和 checkpoint；
+- 评测前准备模型和数据集；
+- 训练产物、合并模型和评测结果上传 MinIO；
+- MinIO 故障时统一等待并超时失败。
+
+### 阶段五：前端和管理员功能
+
+- 显示资源版本和对象存储状态；
+- 显示节点缓存状态；
+- 支持手动预热模型；
+- 支持缓存清理；
+- 显示下载、上传和校验失败原因。
+
+### 阶段六：测试和切换
+
+- 单节点下载和缓存复用测试；
+- 多节点同时下载同一模型测试；
+- MinIO 重启和网络中断测试；
+- SHA256 损坏文件测试；
+- 训练、推理、权重合并全流程测试；
+- 关闭旧的逐节点上传逻辑。
+
+## 10. 预计工作量
+
+```text
+MinIO 部署和配置             1～2 人日
+Backend 对象存储服务          4～7 人日
+Compute Agent 缓存            6～10 人日
+训练/推理/合并/评测接入        8～15 人日
+权限、数据库和前端             5～10 人日
+故障和回归测试                 5～8 人日
+总计                          29～52 人日
+```
+
+## 11. 推荐结论
+
+当前项目建议采用 MinIO + HTTP 预签名 URL + Compute Agent 本地缓存：
+
+- MinIO 服务端使用 Docker，不安装主机软件包；
+- Backend 增加 `minio` Python SDK；
+- Compute Agent 第一阶段继续使用现有 `httpx`，不增加 MinIO SDK；
+- 算力节点不安装 NFS 客户端；
+- Compute API 继续以 root 运行；
+- 训练、推理和权重合并继续使用本地路径，改造风险低于直接让训练框架读取对象存储。

docs/permissions-and-logging-test-cases.md

@@ -0,0 +1,423 @@
+# 权限与日志改造功能测试用例
+
+## 1. 测试范围
+
+本文档用于验证本次权限和日志相关开发内容：
+
+- 训练、评测、推理资源权限校验。
+- 模型合并和训练模型导出权限。
+- 算力节点及 GPU 分配权限。
+- 资源所有者、ACL、租户/项目归属字段。
+- 资源软删除和审计记录。
+- 登录失败限流与 Token 会话有效期。
+- 后端轮询日志降噪。
+- 数据库初始化和运行时字段迁移。
+
+## 2. 测试环境
+
+| 项目 | 配置 |
+|---|---|
+| 前端地址 | `http://172.25.179.69:16801` |
+| Backend 地址 | `http://172.25.179.69:17861/modelTF` |
+| MinIO 地址 | `http://172.25.179.69:19000` |
+| WSL IP | `172.25.179.69` |
+| 运行方式 | Docker Compose |
+| 数据库 | 远程 PostgreSQL |
+
+## 3. 前置数据
+
+准备以下账号和资源：
+
+| 数据 | 要求 |
+|---|---|
+| 管理员账号 | 具有 `admin` 角色 |
+| 普通操作员 | 具有训练、评测或推理页面权限 |
+| 只读用户 | 具有 `dashboard` 或 `logs` 权限，不具有业务写权限 |
+| 测试模型 | 一个基座模型和一个训练模型 |
+| 测试数据集 | 一个训练数据集和一个评测数据集 |
+| 算力节点 | 至少一个在线节点，最好有 2 张及以上 GPU |
+| ACL 资源 | 将测试数据集授权给普通用户进行验证 |
+
+## 4. 鉴权与会话测试
+
+### AUTH-001 登录成功生成会话 Token
+
+**步骤**
+
+1. 使用有效账号调用登录接口或登录页面。
+2. 查看响应中的 `token` 和 `session_id`。
+3. 使用 Token 调用 `/modelTF/me`。
+
+**预期**
+
+- 登录返回 HTTP 200。
+- Token 格式包含用户 ID 和 session ID。
+- `/me` 能返回当前用户信息。
+- `sessions` 表生成一条未注销记录。
+
+### AUTH-002 无 Token 访问受保护接口
+
+**步骤**
+
+1. 不携带 `Authorization` 调用数据集、模型或训练列表接口。
+
+**预期**
+
+- 返回 HTTP 401。
+- 不返回资源数据。
+
+### AUTH-003 注销后 Token 失效
+
+**步骤**
+
+1. 登录并记录 Token、`session_id`。
+2. 调用注销接口。
+3. 使用原 Token 调用 `/me` 或资源接口。
+
+**预期**
+
+- 注销成功。
+- `sessions.logout_at` 已写入。
+- 原 Token 返回 HTTP 401。
+
+### AUTH-004 登录失败限流
+
+**步骤**
+
+1. 同一客户端 IP 连续输入错误密码 5 次。
+2. 第 6 次继续登录。
+
+**预期**
+
+- 前 5 次返回登录失败。
+- 第 6 次返回 HTTP 429。
+- 使用正确密码也应在冷却窗口内被限制。
+- 登录成功后失败计数清除。
+
+## 5. 资源所有权与 ACL
+
+### ACL-001 资源所有者访问自己的资源
+
+**步骤**
+
+1. 普通用户创建数据集或模型。
+2. 使用该用户查看列表和详情。
+3. 修改资源元数据。
+
+**预期**
+
+- 资源出现在自己的列表中。
+- 详情访问返回 HTTP 200。
+- 资源所有者可以执行允许的写操作。
+
+### ACL-002 未授权用户访问资源
+
+**步骤**
+
+1. 用户 A 创建数据集。
+2. 用户 B 未获得 ACL 授权时查看该数据集详情。
+
+**预期**
+
+- 用户 B 不应在列表中看到该资源。
+- 直接访问详情返回 HTTP 403。
+
+### ACL-003 ACL 授权后访问资源
+
+**步骤**
+
+1. 用户 A 或管理员给用户 B 授予 `read` 权限。
+2. 用户 B 刷新资源列表并访问详情。
+
+**预期**
+
+- 用户 B 可以看到并读取资源。
+- 用户 B 不能执行 `write`、`delete` 或 `execute` 操作。
+
+### ACL-004 非所有者修改 ACL
+
+**步骤**
+
+1. 用户 B 仅拥有资源 `read` 权限。
+2. 用户 B 调用 ACL 修改接口。
+
+**预期**
+
+- 返回 HTTP 403。
+- ACL 内容不发生变化。
+
+### ACL-005 ACL 修改审计
+
+**步骤**
+
+1. 管理员或资源所有者修改 ACL。
+2. 查询审计日志。
+
+**预期**
+
+- 出现 `resource.acl.set` 操作记录。
+- 记录操作者、资源类型、资源 ID 和变更详情。
+
+## 6. 训练权限测试
+
+### TRAIN-001 训练创建校验模型和数据集权限
+
+**步骤**
+
+1. 普通用户选择无权限的基座模型或数据集创建训练任务。
+2. 再使用已授权模型和数据集创建训练任务。
+
+**预期**
+
+- 无权限资源返回 HTTP 403。
+- 已授权资源允许创建任务。
+- 任务记录包含 `created_by`。
+
+### TRAIN-002 GPU 授权校验
+
+**步骤**
+
+1. 管理员将 GPU 0 分配给用户 A。
+2. 用户 A 选择 GPU 0 创建训练任务。
+3. 用户 A 选择未分配的 GPU 1 创建训练任务。
+
+**预期**
+
+- GPU 0 可以提交。
+- GPU 1 返回 HTTP 403。
+- 未指定 GPU 时，仅从用户已授权的空闲 GPU 中自动分配。
+
+### TRAIN-003 训练任务停止、重试和删除权限
+
+**步骤**
+
+1. 用户 A 创建训练任务。
+2. 用户 B 尝试停止、重试或删除该任务。
+3. 管理员执行相同操作。
+
+**预期**
+
+- 用户 B 无资源权限时返回 HTTP 403。
+- 删除和停止等高风险操作按配置进入审批流程。
+- 管理员可以旁路审批执行。
+- GPU 占用在停止、失败和删除后释放。
+
+## 7. 评测权限测试
+
+### EVAL-001 评测创建联合权限
+
+**步骤**
+
+1. 普通用户选择无权使用的模型创建评测。
+2. 普通用户选择无权使用的数据集创建评测。
+3. 使用同时拥有权限的模型和数据集创建评测。
+
+**预期**
+
+- 模型无权限返回 HTTP 403。
+- 数据集无权限返回 HTTP 403。
+- 两个资源均有 `execute` 权限时允许创建。
+- 评测任务包含 `created_by`、模型、数据集和算力节点信息。
+
+### EVAL-002 评测详情和删除权限
+
+**预期**
+
+- 无权限用户不能查看评测详情。
+- 评测删除需要 `delete` 权限。
+- 非管理员删除高风险评测任务时触发审批。
+
+## 8. 推理权限测试
+
+### INFER-001 推理创建模型权限
+
+**步骤**
+
+1. 普通用户选择无权模型创建推理任务。
+2. 选择已授权模型创建推理任务。
+
+**预期**
+
+- 无权模型返回 HTTP 403。
+- 有权模型允许创建。
+- 推理任务包含 `created_by`。
+
+### INFER-002 推理加载和卸载权限
+
+**步骤**
+
+1. 用户 A 创建推理任务。
+2. 用户 B 调用加载、卸载接口。
+3. 用户 A 执行加载和卸载。
+
+**预期**
+
+- 用户 B 返回 HTTP 403。
+- 用户 A 可以执行授权范围内的加载和卸载。
+- 卸载后 GPU 和节点状态恢复为空闲。
+
+### INFER-003 推理任务删除审批
+
+**预期**
+
+- 非管理员删除他人推理任务被拒绝或进入审批。
+- 管理员可以直接删除。
+- 删除操作有审计日志。
+
+## 9. 模型合并与导出测试
+
+### MODEL-001 权重合并权限
+
+**步骤**
+
+1. 普通用户选择无权限训练模型进行合并。
+2. 选择已授权训练模型进行合并。
+
+**预期**
+
+- 无权限返回 HTTP 403。
+- 已授权训练模型允许合并。
+- 如指定基座模型，还必须拥有基座模型 `execute` 权限。
+
+### MODEL-002 训练模型删除权限
+
+**预期**
+
+- 只有资源所有者、ACL 授权用户或管理员可操作。
+- 删除采用软删除。
+- `deleted_at`、`deleted_by` 被写入。
+
+### MODEL-003 导出任务访问权限
+
+**预期**
+
+- 无权用户不能查看训练模型导出任务。
+- 有权用户可以查看导出状态。
+- 导出动作应记录 `trained_model.export` 审计日志。
+
+## 10. 算力节点与 GPU 管理测试
+
+### GPU-001 普通用户节点可见范围
+
+**预期**
+
+- 普通用户只能看到被分配 GPU 所在节点。
+- 普通用户只能看到已授权 GPU。
+- 管理员可看到所有节点和 GPU。
+
+### GPU-002 节点管理接口权限
+
+验证节点创建、修改、删除、启用、禁用、排空、连接测试和健康检查。
+
+**预期**
+
+- 普通用户全部返回 HTTP 403。
+- 管理员操作成功。
+
+### GPU-003 多卡自动分配
+
+**步骤**
+
+1. 节点配置多张 GPU。
+2. 启动一个任务占用其中一张卡。
+3. 再启动任务并选择剩余卡。
+
+**预期**
+
+- 已占用 GPU 不再出现在可选列表。
+- 剩余 GPU 可以被其他任务使用。
+- 任务失败、停止或完成后 GPU 释放。
+
+## 11. 软删除与数据库测试
+
+### DB-001 初始化字段检查
+
+执行：
+
+```sql
+SELECT table_name, column_name
+FROM information_schema.columns
+WHERE table_name IN ('models', 'datasets', 'trained_models', 'eval_tasks', 'sessions')
+ORDER BY table_name, ordinal_position;
+```
+
+**预期字段**
+
+- 资源表存在 `created_by`、`tenant_id`、`project_id`、`deleted_at`、`deleted_by`。
+- `sessions` 存在 `issued_at`、`expires_at`、`logout_at`。
+
+### DB-002 软删除列表过滤
+
+**步骤**
+
+1. 删除模型、数据集或评测任务。
+2. 查询列表。
+3. 直接查询数据库记录。
+
+**预期**
+
+- 前端列表不再显示已删除资源。
+- 数据库记录仍存在。
+- `deleted_at` 和 `deleted_by` 有值。
+
+## 12. 日志降噪测试
+
+### LOG-001 正常轮询日志
+
+**步骤**
+
+1. 重启 Backend 容器。
+2. 连续观察 1 分钟日志。
+
+```bash
+docker logs -f --tail=200 yg-ft-backend-api
+```
+
+**预期**
+
+- 不再每次以 `INFO` 输出 `compute jobs polled`。
+- 健康检查成功请求不再以 `INFO` 输出应用日志。
+- 正常任务同步日志仅在 `DEBUG` 级别出现。
+
+### LOG-002 异常轮询日志
+
+**步骤**
+
+1. 临时停止算力节点或断开节点网络。
+2. 观察 Backend 日志。
+
+**预期**
+
+- 轮询失败以 `WARNING` 或 `ERROR` 输出。
+- 异常包含节点、任务或错误原因。
+- 恢复节点后轮询继续工作。
+
+## 13. 容器验证命令
+
+```bash
+docker compose ps
+docker logs --tail=200 yg-ft-backend-api
+curl -i http://172.25.179.69:17861/modelTF/health
+curl -i http://172.25.179.69:16801/
+```
+
+预期 Backend 和 Frontend 均为 `healthy`，健康接口返回 HTTP 200。
+
+## 14. 测试结果记录
+
+| 用例编号 | 测试结果 | 实际结果 | 缺陷编号 | 测试人 | 日期 |
+|---|---|---|---|---|---|
+| AUTH-001 | □通过 □失败 |  |  |  |  |
+| AUTH-002 | □通过 □失败 |  |  |  |  |
+| AUTH-003 | □通过 □失败 |  |  |  |  |
+| AUTH-004 | □通过 □失败 |  |  |  |  |
+| ACL-001 | □通过 □失败 |  |  |  |  |
+| TRAIN-001 | □通过 □失败 |  |  |  |  |
+| TRAIN-002 | □通过 □失败 |  |  |  |  |
+| EVAL-001 | □通过 □失败 |  |  |  |  |
+| INFER-001 | □通过 □失败 |  |  |  |  |
+| MODEL-001 | □通过 □失败 |  |  |  |  |
+| GPU-001 | □通过 □失败 |  |  |  |  |
+| DB-001 | □通过 □失败 |  |  |  |  |
+| LOG-001 | □通过 □失败 |  |  |  |  |
+| LOG-002 | □通过 □失败 |  |  |  |  |

docs/permissions-and-logging-test-results.md

@@ -0,0 +1,122 @@
+# 权限与日志改造自动化测试结果
+
+## 1. 测试时间
+
+2026-08-12
+
+## 2. 测试环境
+
+| 项目 | 地址/状态 |
+|---|---|
+| Frontend | `http://172.25.179.69:16801` |
+| Backend | `http://172.25.179.69:17861/modelTF` |
+| WSL | `172.25.179.69` |
+| Docker | 已运行 |
+
+## 3. 自动执行结果
+
+| 用例 | 结果 | 实际结果 |
+|---|---|---|
+| 容器状态 | 通过 | Compute、Frontend、Backend、Redis、MinIO 均为 `healthy` |
+| Backend 健康检查 | 通过 | `/modelTF/health` 返回 HTTP 200 |
+| Frontend 首页 | 通过 | `/` 返回 HTTP 200 |
+| 未登录访问数据集接口 | 通过 | `/modelTF/dataset-manage` 返回 HTTP 401 |
+| Redis 容器状态 | 通过 | 容器状态为 healthy |
+| 轮询正常日志降噪 | 通过 | 最近 120 秒未发现 `compute jobs polled` 或 `health check requested` 高频日志 |
+| 轮询异常日志保留 | 未触发 | 当前未人为停止算力节点，未产生轮询失败日志 |
+| 前端构建 | 通过 | `npm run build` 成功 |
+| 后端编译 | 通过 | 相关 Python 模块 `py_compile` 成功 |
+| Git 差异检查 | 通过 | `git diff --check` 无格式错误 |
+
+## 4. 数据库字段迁移
+
+### 静态检查结果
+
+初始化 SQL 和运行时迁移逻辑已包含以下字段：
+
+- `models.deleted_at`
+- `models.deleted_by`
+- `models.tenant_id`
+- `models.project_id`
+- `datasets.deleted_at`
+- `datasets.deleted_by`
+- `datasets.tenant_id`
+- `datasets.project_id`
+- `trained_models.deleted_at`
+- `trained_models.deleted_by`
+- `trained_models.tenant_id`
+- `trained_models.project_id`
+- `eval_tasks.deleted_at`
+- `eval_tasks.deleted_by`
+- `sessions.issued_at`
+- `sessions.expires_at`
+
+### 远程数据库检查状态
+
+已通过当前 Backend 容器使用一次性 PostgreSQL 连接检查远程数据库，字段迁移已完成。
+
+确认存在：
+
+- `models`: `created_by`、`tenant_id`、`project_id`、`deleted_at`、`deleted_by`
+- `datasets`: `created_by`、`tenant_id`、`project_id`、`deleted_at`、`deleted_by`
+- `trained_models`: `created_by`、`tenant_id`、`project_id`、`deleted_at`、`deleted_by`
+- `eval_tasks`: `created_by`、`tenant_id`、`project_id`、`deleted_at`、`deleted_by`
+- `sessions`: `issued_at`、`expires_at`、`logout_at`
+
+使用的检查 SQL：
+
+```sql
+SELECT table_name, column_name
+FROM information_schema.columns
+WHERE table_name IN ('models', 'datasets', 'trained_models', 'eval_tasks', 'sessions')
+ORDER BY table_name, ordinal_position;
+```
+
+## 5. Redis 说明
+
+已从当前环境确认 Redis 实际密码配置为 `Tvhrf659WaX-S1B8FG6c2kSZK07XTv82`，使用该密码验证返回 `PONG`。
+
+项目配置已同步为：
+
+```env
+REDIS_PASSWORD=Tvhrf659WaX-S1B8FG6c2kSZK07XTv82
+REDIS_URL=redis://:Tvhrf659WaX-S1B8FG6c2kSZK07XTv82@redis:6379/0
+```
+
+后续验证应从当前 Docker 环境读取实际密码后执行：
+
+```bash
+docker compose config
+docker exec yg-ft-redis printenv REDISCLI_AUTH
+docker exec yg-ft-redis redis-cli -a "$REDISCLI_AUTH" ping
+```
+
+预期返回：
+
+```text
+PONG
+```
+
+## 6. 未自动执行的破坏性用例
+
+以下用例需要测试账号、测试资源或人工确认，未自动执行，以避免影响现有数据：
+
+- ACL 授权和撤销。
+- 训练任务创建、停止、重试和删除。
+- 评测任务启动和删除。
+- 推理模型加载、卸载和删除。
+- 模型权重合并和导出。
+- GPU 分配、占用和释放。
+- 软删除后资源恢复和数据完整性。
+- 人为停止算力节点验证轮询失败恢复。
+- 登录失败 5 次后的限流验证。
+- Token 过期和注销后的访问验证。
+
+## 7. 当前结论
+
+当前服务基础可用，页面和 Backend 健康接口正常，未登录鉴权正常，日志降噪逻辑生效，前后端构建通过。
+
+当前仍需人工或使用专用测试数据验证：
+
+1. 权限矩阵中的创建、执行、删除、审批和 GPU 占用场景。
+2. Token 过期和登录失败限流场景。

docs/permissions-design.md

@@ -0,0 +1,890 @@
+# 平台权限设计文档
+
+> 版本：v1.0
+> 日期：2026-08-02
+> 状态：设计基线，供后端实现和前端联调参照
+
+---
+
+## 目录
+
+1. [设计目标](#1-设计目标)
+2. [整体架构](#2-整体架构)
+3. [角色体系](#3-角色体系)
+4. [页面权限码](#4-页面权限码)
+5. [资源所有权与可见性](#5-资源所有权与可见性)
+6. [资源级 ACL（访问控制列表）](#6-资源级-acl访问控制列表)
+7. [GPU 算力分配与隔离](#7-gpu-算力分配与隔离)
+8. [审批拦截机制](#8-审批拦截机制)
+9. [审计日志](#9-审计日志)
+10. [接口鉴权流程](#10-接口鉴权流程)
+11. [数据库表结构](#11-数据库表结构)
+12. [API 接口清单](#12-api-接口清单)
+13. [前端权限控制](#13-前端权限控制)
+14. [安全设计补充](#14-安全设计补充)
+15. [实施计划](#15-实施计划)
+
+---
+
+## 1. 设计目标
+
+| 目标 | 说明 |
+|---|---|
+| **数据隔离** | 用户自己创建的数据集、模型、训练任务默认只有自己可见可操作；管理员可见全部 |
+| **权限分层** | 页面级（菜单/路由可见性）+ 资源级（单条数据的读/写/删）两层控制 |
+| **GPU 管控** | 多卡服务器上，管理员可指定哪些用户能使用哪些 GPU 卡 |
+| **审批拦截** | 删除他人资源、停止他人任务、发布模型等高风险操作需审批或管理员旁路 |
+| **审计可追溯** | 所有写操作和敏感操作产生审计日志，可按用户、动作、资源、时间筛选 |
+| **权限最小变更** | 只有管理员可修改用户角色和权限码；普通用户无法提权 |
+
+---
+
+## 2. 整体架构
+
+```
+┌──────────────────────────────────────────────────────┐
+│                     前端（Vue3）                       │
+│  ┌────────────┐  ┌────────────┐  ┌───────────────┐ │
+│  │ 路由守卫    │  │ 侧边栏过滤  │  │ 页面内按钮控制  │ │
+│  │ (permission)│  │ (permission)│  │ (ACL/owner)   │ │
+│  └──────┬─────┘  └──────┬─────┘  └───────┬───────┘ │
+│         └───────────────┴─────────────────┘          │
+│                         │ HTTP (Bearer token)        │
+└─────────────────────────┼────────────────────────────┘
+                          │
+┌─────────────────────────┼────────────────────────────┐
+│                    后端（FastAPI）                     │
+│  ┌──────────────┐  ┌────┴───────┐  ┌──────────────┐  │
+│  │ get_current  │  │ 资源可见性  │  │ GPU 分配校验  │  │
+│  │ _user (鉴权) │  │ 过滤器      │  │              │  │
+│  └──────┬───────┘  └────┬───────┘  └──────┬───────┘  │
+│         │               │                  │          │
+│  ┌──────┴───────────────┴──────────────────┘          │
+│  │              PlatformStore                        │  │
+│  │  users | acls | roles | gpu_assignments |         │  │
+│  │  datasets | models | fine_tune_tasks | ...        │  │
+│  └───────────────────────────────────────────────────│  │
+│  │              audit_logs (审计日志)                │  │
+│  └───────────────────────────────────────────────────│  │
+└─────────────────────────────────────────────────────────┘
+```
+
+**鉴权链路**：
+1. 请求到达 → `get_current_user` 从 `Authorization: Bearer platform-token-{user_id}` 解析当前用户
+2. 页面级权限 → 检查 `user.permissions` 是否包含路由对应的权限码
+3. 资源级权限 → 检查资源的 `created_by` 字段（所有权）或 `acls` 表（ACL 授权）
+4. GPU 权限 → 检查 `gpu_assignments` 表确认用户是否被分配了请求的 GPU
+
+---
+
+## 3. 角色体系
+
+### 3.1 内置角色
+
+| 角色 code | 中文名 | 说明 |
+|---|---|---|
+| `admin` | 超级管理员 | 拥有全部权限码；可见全部资源；可管理用户和 GPU 分配 |
+| `operator` | 操作员 | 可创建/操作自己的数据集、模型、训练任务；不可管理用户 |
+| `viewer` | 观察员 | 只读权限；可查看被授权的资源；不可创建或修改 |
+| `guest` | 访客 | 仅登录和看板；无业务操作权限（扩展预留） |
+
+### 3.2 角色与权限码映射
+
+| 权限码 | admin | operator | viewer |
+|---|---|---|---|
+| `dashboard` | ✅ | ✅ | ✅ |
+| `fine-tune` | ✅ | ✅ | — |
+| `model-eval` | ✅ | ✅ | — |
+| `model-inference` | ✅ | ✅ | — |
+| `model-manage` | ✅ | ✅ | — |
+| `dataset` | ✅ | ✅ | — |
+| `data-process` | ✅ | ✅ | — |
+| `data-convert` | ✅ | ✅ | — |
+| `compute` | ✅ | ✅ | — |
+| `hardware` | ✅ | ✅ | ✅ |
+| `logs` | ✅ | ✅ | ✅ |
+| `user-settings` | ✅ | — | — |
+
+### 3.3 权限修改规则
+
+- **只有 admin 角色的用户**可以修改其他用户的角色和权限码
+- admin 用户的 `protected=True` 标记，防止被删除或降级
+- 权限修改操作产生审计日志：`action=user.permission.update`
+- 用户可以查看自己的权限，不能修改自己的权限
+
+---
+
+## 4. 页面权限码
+
+| 权限码 | 对应路由 | 功能 |
+|---|---|---|
+| `dashboard` | `/dashboard` | 服务看板 |
+| `fine-tune` | `/fine-tune`, `/fine-tune/create`, `/training-log/:id` | 模型训练 |
+| `model-eval` | `/model-eval`, `/model-eval/create`, `/model-eval/:id` | 模型评测 |
+| `model-inference` | `/model-inference`, `/model-inference/create`, `/model-inference/chat/:id` | 模型推理 |
+| `model-manage` | `/model-manage`, `/model-manage/create`, `/model-manage/:id/edit`, `/model-manage/merge` | 模型管理 |
+| `dataset` | `/dataset`, `/dataset/create`, `/dataset/:id/preview` | 数据集管理 |
+| `data-process` | `/data-process`, `/data-process/create`, `/data-process/:id` | 数据处理 |
+| `data-convert` | `/data-convert`, `/tools` | 数据转换与工具 |
+| `compute` | `/compute` | 算力节点 |
+| `hardware` | `/hardware` | 平台性能 |
+| `logs` | `/logs`, `/training-log/:id` | 查看日志 |
+| `user-settings` | `/user-settings`, `/tenants`, `/projects`, `/approvals`, `/audit-logs` | 系统设置与平台治理 |
+
+---
+
+## 5. 资源所有权与可见性
+
+### 5.1 所有权模型
+
+每个用户可创建的资源都携带 `created_by`（或 `owner_id`）字段，标识资源所有者。
+
+| 资源类型 | 表 | 所有者字段 | 说明 |
+|---|---|---|---|
+| 数据集 | `datasets` | `created_by` | 用户上传/创建的数据集 |
+| 基座模型 | `models` | `created_by` | 登记的本地/API 模型 |
+| 训练产物 | `trained_models` | `created_by` | 微调产出的模型 |
+| 训练任务 | `fine_tune_tasks` | `payload.created_by` | 微调任务 |
+| 评测任务 | `eval_tasks` | `created_by` | 评测任务 |
+| 推理任务 | `compare_tasks` (payload) | `created_by` | 推理/对比任务 |
+| 数据处理任务 | `data_process_tasks` | `created_by` | 数据处理任务 |
+| 数据转换任务 | `data_convert_jobs` | `created_by` | 数据转换任务 |
+
+### 5.2 可见性规则
+
+```
+资源列表查询过滤逻辑：
+
+if user.role == "admin":
+    返回全部资源
+elif resource.created_by == user.id:
+    返回（资源所有者可见自己的资源）
+elif acl 中存在 (principal_type="user", principal_id=user.id, permission 包含 "read"):
+    返回（被 ACL 显式授权的资源）
+elif acl 中存在 (principal_type="role", principal_id=user.role, permission 包含 "read"):
+    返回（被角色级 ACL 授权的资源）
+else:
+    不可见
+```
+
+### 5.3 所有权操作矩阵
+
+| 操作 | admin | 资源所有者 | 其他被授权用户 | 其他用户 |
+|---|---|---|---|---|
+| 查看资源 | ✅ 全部 | ✅ 自己的 | ✅ ACL 授权范围内 | ❌ |
+| 编辑资源 | ✅ | ✅ 自己的 | ✅ ACL 含 write 时 | ❌ |
+| 删除资源 | ✅ | ✅ 自己的（需审批） | ❌ | ❌ |
+| 分享/授权 | ✅ | ✅ 自己的 | ❌ | ❌ |
+| 使用资源（训练/推理/评测） | ✅ | ✅ 自己的 | ✅ ACL 含 execute 时 | ❌ |
+
+### 5.4 数据集可见性示例
+
+```
+用户 A 创建了数据集 ds_A1 → 只有 A 和 admin 可见
+用户 A 通过 ACL 把 ds_A1 的 read 权限授给用户 B → B 也可见
+用户 A 通过 ACL 把 ds_A1 的 write 权限授给 operator 角色 → 所有 operator 可编辑
+管理员可在任何数据集上设置 ACL
+```
+
+---
+
+## 6. 资源级 ACL（访问控制列表）
+
+### 6.1 ACL 表结构
+
+```sql
+CREATE TABLE IF NOT EXISTS acls (
+    id              TEXT PRIMARY KEY,
+    resource_type   TEXT NOT NULL,      -- 资源类型: dataset / model / trained_model / fine_tune_task / ...
+    resource_id     TEXT NOT NULL,      -- 资源 ID
+    principal_type  TEXT NOT NULL,      -- 授权主体类型: user / role
+    principal_id    TEXT NOT NULL,      -- 授权主体 ID: user_id 或 role name
+    permission      TEXT NOT NULL,      -- 权限: read / write / execute / download / delete / admin
+    create_time     TEXT
+);
+```
+
+### 6.2 权限粒度
+
+| 权限值 | 含义 | 覆盖关系 |
+|---|---|---|
+| `read` | 查看资源详情、列表 | — |
+| `write` | 编辑资源内容/元数据 | 覆盖 `read` |
+| `execute` | 使用资源（如用数据集训练、用模型推理） | 覆盖 `read` |
+| `download` | 下载资源文件 | 独立权限 |
+| `delete` | 删除资源 | 独立权限（通常需审批） |
+| `admin` | 完全控制（含 ACL 管理） | 覆盖以上全部 |
+
+### 6.3 ACL 管理接口
+
+| 接口 | 方法 | 权限要求 | 说明 |
+|---|---|---|---|
+| `/resources/{type}/{id}/acl` | GET | admin 或资源所有者 | 查询资源 ACL |
+| `/resources/{type}/{id}/acl` | PUT | admin 或资源所有者 | 设置资源 ACL（全量替换） |
+
+### 6.4 ACL 管理规则
+
+- **admin** 可以在任何资源上设置 ACL
+- **资源所有者** 可以在自己的资源上设置 ACL
+- **被授权用户** 不能转授自己获得的权限
+- ACL 变更产生审计日志：`action=resource.acl.set`
+- 设置 ACL 时全量替换该资源的所有 ACL 条目
+
+### 6.5 前端 ACL 管理入口
+
+在数据集详情、模型详情、训练任务详情页面提供「资源授权」按钮，弹出 ACL 管理对话框：
+- 显示当前 ACL 列表（主体类型 + 主体名称 + 权限勾选）
+- 支持按用户或按角色添加授权
+- 权限以多选框形式展示（read / write / execute / download / delete）
+
+---
+
+## 7. GPU 算力分配与隔离
+
+### 7.1 设计背景
+
+服务器可能安装多张 GPU 卡（如 8×A100），需要精细化管控：
+- 管理员指定哪些用户可以使用哪些 GPU 卡
+- 未被分配的 GPU 卡对用户不可见或不可选
+- admin 可以使用全部 GPU
+
+### 7.2 GPU 分配表
+
+```sql
+CREATE TABLE IF NOT EXISTS gpu_assignments (
+    id          TEXT PRIMARY KEY,
+    gpu_id      TEXT NOT NULL,          -- gpus 表的外键（node_id + gpu_index 组合）
+    node_id     TEXT NOT NULL,          -- 算力节点 ID
+    gpu_index   INTEGER NOT NULL,       -- GPU 卡序号
+    user_id     TEXT NOT NULL,          -- 被分配的用户 ID
+    assigned_by TEXT,                  -- 分配操作人 ID（admin）
+    assigned_at TEXT NOT NULL,          -- 分配时间
+    UNIQUE (node_id, gpu_index, user_id)  -- 一张卡可分配给多个用户，但每对 (卡, 用户) 唯一
+);
+```
+
+### 7.3 分配规则
+
+| 规则 | 说明 |
+|---|---|
+| 谁可分配 | 只有 `admin` 角色可以分配 GPU |
+| admin 使用 | admin 可使用全部 GPU，不需要显式分配 |
+| 普通用户 | 只能使用 `gpu_assignments` 中分配给自己的 GPU |
+| 共享分配 | 一张 GPU 可分配给多个用户（非独占），但同时只能被一个任务占用 |
+| 默认策略 | 新用户默认不分配任何 GPU，由管理员显式分配 |
+
+### 7.4 GPU 分配接口
+
+| 接口 | 方法 | 权限 | 说明 |
+|---|---|---|---|
+| `/compute/gpu-assignments` | GET | admin | 查看全部分配关系 |
+| `/compute/gpu-assignments` | POST | admin | 批量分配（body: `{ assignments: [{ node_id, gpu_index, user_id }] }`） |
+| `/compute/gpu-assignments/{id}` | DELETE | admin | 撤销某条分配 |
+| `/compute/my-gpus` | GET | 登录用户 | 查看自己可用的 GPU 列表 |
+
+### 7.5 训练/评测/推理 GPU 选择校验
+
+当普通用户创建训练任务、评测任务、推理任务并选择 GPU 时：
+1. 后端检查 `gpu_assignments` 表，确认用户被分配了所选 GPU
+2. 未被分配的 GPU → 返回 403 `"无权使用 GPU {node}:{index}"`
+3. admin 用户跳过此校验
+
+### 7.6 前端 GPU 选择交互
+
+- 普通用户在创建任务选择 GPU 时，下拉列表只显示自己被分配的 GPU
+- admin 用户在下拉列表中可看到全部 GPU
+- 未分配任何 GPU 的用户，GPU 选择区域显示提示："未分配 GPU，请联系管理员"
+
+---
+
+## 8. 审批拦截机制
+
+### 8.1 需要审批的操作
+
+| 操作 | 触发条件 | 审批动作 code |
+|---|---|---|
+| 删除他人数据集 | 非 admin 删除 `created_by != user.id` 的数据集 | `dataset.delete` |
+| 删除他人模型 | 非 admin 删除 `created_by != user.id` 的模型 | `model.delete` |
+| 停止他人训练任务 | 非 admin 停止 `created_by != user.id` 的任务 | `fine_tune.stop` |
+| 发布模型到推理服务 | 任何用户（含 admin）发布到生产环境 | `model_service.publish` |
+| 删除项目空间 | 存在待审批变更时拒绝 | `project.delete` |
+| 归档项目空间 | 存在待审批变更时拒绝 | `project.archive` |
+| 导出训练产物 | 非 admin 导出他人训练的模型 | `trained_model.export` |
+
+### 8.2 审批流程
+
+```
+普通用户发起高风险操作
+         │
+         ▼
+  ┌──────────────┐        ┌──────────────────────┐
+  │ admin 旁路？ │───是──▶│ 直接执行 + 审计日志   │
+  └──────┬───────┘        └──────────────────────┘
+         │ 否
+         ▼
+  ┌──────────────────────┐
+  │ 创建审批实例          │
+  │ status=pending       │
+  │ 返回 202（待审批）    │
+  └──────────┬───────────┘
+             │
+             ▼
+  ┌──────────────────────┐
+  │ 管理员审批            │
+  │ POST /approvals/:id  │
+  │ /steps/:idx/decision │
+  └──────────┬───────────┘
+             │
+      ┌──────┴──────┐
+      │             │
+   approved      rejected
+      │             │
+      ▼             ▼
+  执行操作      不执行
+  +审计日志    +审计日志
+```
+
+### 8.3 审批模板
+
+审批模板定义了特定操作需要几步审批、每步的审批人是谁：
+
+```json
+{
+  "id": "tpl_001",
+  "name": "删除数据集审批",
+  "action": "dataset.delete",
+  "steps": [
+    { "approver_id": "u_admin", "step_name": "管理员审核" }
+  ]
+}
+```
+
+### 8.4 审批拦截点
+
+在项目模块的 `_require_no_pending_approval` 函数中，当存在待审批实例时拒绝执行新操作。其他模块通过 `_require_approval_or_admin` 函数实现 admin 旁路或创建审批实例。
+
+---
+
+## 9. 审计日志
+
+### 9.1 审计范围
+
+所有写操作和敏感操作必须产生审计日志：
+
+| 动作分类 | action 示例 |
+|---|---|
+| 用户管理 | `user.create`, `user.update`, `user.delete`, `user.permission.update` |
+| 租户管理 | `tenant.create`, `tenant.update`, `tenant.quota.set`, `tenant.retention.set` |
+| 项目管理 | `project.create`, `project.update`, `project.archive`, `project.delete`, `project.member.add`, `project.member.update`, `project.member.remove` |
+| 资源 ACL | `resource.acl.set` |
+| 模型管理 | `model.create`, `model.update`, `model.delete`, `model.merge` |
+| 数据集 | `dataset.create`, `dataset.update`, `dataset.delete`, `dataset.upload` |
+| 训练任务 | `fine_tune.create`, `fine_tune.start`, `fine_tune.stop`, `fine_tune.delete` |
+| 评测任务 | `eval.create`, `eval.start`, `eval.stop` |
+| 推理任务 | `inference.create`, `inference.start`, `inference.stop` |
+| 审批 | `approval.create`, `approval.decide` |
+| 留存策略 | `retention.create`, `retention.update`, `retention.delete` |
+| GPU 分配 | `gpu.assign`, `gpu.unassign` |
+
+### 9.2 审计日志字段
+
+```sql
+CREATE TABLE IF NOT EXISTS audit_logs (
+    id          TEXT PRIMARY KEY,
+    time        TEXT NOT NULL,         -- ISO8601 时间戳
+    tenant_id   TEXT,                 -- 租户 ID（可选）
+    project_id  TEXT,                 -- 项目 ID（可选）
+    actor_id    TEXT,                 -- 操作人 ID
+    action      TEXT NOT NULL,         -- 动作类型
+    target_type TEXT NOT NULL,         -- 目标资源类型
+    target_id   TEXT,                 -- 目标资源 ID
+    detail      TEXT,                 -- 详情摘要
+    client_ip   TEXT                  -- 客户端 IP
+);
+```
+
+### 9.3 查询与导出
+
+| 接口 | 方法 | 说明 |
+|---|---|---|
+| `/system/audit-logs` | GET | 分页查询，支持按 tenant_id / project_id / actor_id / action / target_type / start_time / end_time 筛选 |
+| `/system/audit-logs/export` | GET | CSV 导出，与应用查询相同的过滤条件 |
+
+---
+
+## 10. 接口鉴权流程
+
+### 10.1 Token 格式
+
+```
+Authorization: Bearer platform-token-{user_id}
+```
+
+登录成功后返回 `token` 和 `user` 信息。Token 中编码了 `user_id`，后端通过 `get_current_user` 解析。
+
+### 10.2 鉴权层级
+
+```
+请求到达
+  │
+  ├─ 1. 公开路径检查（/health, /login, /system-info）→ 直接放行
+  │
+  ├─ 2. Token 解析 → get_current_user
+  │     ├─ 无 token / token 无效 → 401
+  │     └─ 用户不存在 / 状态 disabled → 401
+  │
+  ├─ 3. 页面级权限码检查（路由守卫 / Depends）
+  │     └─ user.permissions 不含所需权限码 → 403
+  │
+  ├─ 4. 资源级权限检查（路由函数内）
+  │     ├─ admin → 全部放行
+  │     ├─ resource.created_by == user.id → 放行
+  │     ├─ ACL 检查 has_resource_access() → 有授权则放行
+  │     └─ 否则 → 403
+  │
+  ├─ 5. GPU 权限检查（训练/评测/推理创建时）
+  │     ├─ admin → 全部放行
+  │     ├─ gpu_assignments 检查 → 有分配则放行
+  │     └─ 否则 → 403
+  │
+  └─ 6. 审批拦截检查（高风险操作）
+        ├─ admin → 旁路，直接执行
+        ├─ 无待审批实例 → 可执行
+        ├─ 有待审批实例 → 409 "存在待审批的变更"
+        └─ 需要审批 → 202 "已创建审批实例"
+```
+
+### 10.3 FastAPI 依赖注入
+
+```python
+# 任何需要登录的接口
+@router.get("/datasets")
+async def list_datasets(user: dict = Depends(get_current_user)):
+    ...
+
+# 需要管理员权限的接口
+@router.post("/users")
+async def create_user(user: dict = Depends(require_admin)):
+    ...
+
+# 需要资源级权限检查的接口
+@router.delete("/datasets/{dataset_id}")
+async def delete_dataset(
+    dataset_id: str,
+    user: dict = Depends(get_current_user),
+):
+    if not has_resource_access("dataset", dataset_id, user, "delete"):
+        raise HTTPException(403, "forbidden")
+    ...
+```
+
+---
+
+## 11. 数据库表结构
+
+### 11.1 现有表（已实现）
+
+| 表名 | 用途 |
+|---|---|
+| `users` | 用户表（id, username, password_hash, role, status, permissions, protected） |
+| `roles` | 角色定义（name, permissions） |
+| `sessions` | 登录会话（user_id, issued_at, expires_at, ip） |
+| `acls` | 资源访问控制列表（resource_type, resource_id, principal_type, principal_id, permission） |
+| `audit_logs` | 审计日志（actor_id, action, target_type, target_id, time） |
+| `datasets` | 数据集（需补充 `created_by` 字段） |
+| `models` | 基座模型（需补充 `created_by` 字段） |
+| `trained_models` | 训练产物（需补充 `created_by` 字段） |
+| `fine_tune_tasks` | 训练任务（payload 中存储 `created_by`） |
+| `gpus` | GPU 设备（node_id, gpu_index, uuid, name, memory） |
+| `compute_nodes` | 算力节点 |
+| `tenants` | 租户 |
+| `projects` | 项目空间 |
+| `project_members` | 项目成员 |
+| `approval_templates` | 审批模板 |
+| `approval_instances` | 审批实例 |
+| `retention_policies` | 留存策略 |
+
+### 11.2 需新增/补充的表和字段
+
+#### 新增 `gpu_assignments` 表
+
+```sql
+CREATE TABLE IF NOT EXISTS gpu_assignments (
+    id           TEXT PRIMARY KEY,
+    node_id      TEXT NOT NULL REFERENCES compute_nodes(id) ON DELETE CASCADE,
+    gpu_index    INTEGER NOT NULL,
+    user_id      TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
+    assigned_by  TEXT,
+    assigned_at  TEXT NOT NULL,
+    UNIQUE (node_id, gpu_index, user_id)
+);
+CREATE INDEX IF NOT EXISTS idx_gpu_assignments_user ON gpu_assignments(user_id);
+CREATE INDEX IF NOT EXISTS idx_gpu_assignments_gpu ON gpu_assignments(node_id, gpu_index);
+```
+
+#### 补充 `created_by` 字段
+
+```sql
+-- 数据集表补充所有者
+ALTER TABLE datasets ADD COLUMN IF NOT EXISTS created_by TEXT;
+
+-- 基座模型表补充所有者
+ALTER TABLE models ADD COLUMN IF NOT EXISTS created_by TEXT;
+
+-- 训练产物表补充所有者
+ALTER TABLE trained_models ADD COLUMN IF NOT EXISTS created_by TEXT;
+
+-- 评测任务表补充所有者
+ALTER TABLE eval_tasks ADD COLUMN IF NOT EXISTS created_by TEXT;
+
+-- 推理任务表补充所有者
+-- 注意：inference_tasks 表尚未创建，后续建表时直接包含 created_by 字段
+```
+
+---
+
+## 12. API 接口清单
+
+### 12.1 鉴权接口
+
+| 接口 | 方法 | 鉴权 | 说明 |
+|---|---|---|---|
+| `/modelTF/login` | POST | 公开 | 登录，返回 token + user |
+| `/modelTF/me` | GET | Bearer token | 获取当前用户信息 |
+| `/modelTF/users` | GET | admin | 用户列表 |
+| `/modelTF/users` | POST | admin | 创建用户 |
+| `/modelTF/users/:id` | PUT | admin | 更新用户（角色/状态/权限） |
+| `/modelTF/users/:id` | DELETE | admin | 删除用户（protected 用户不可删） |
+| `/modelTF/users/:id/reset-password` | POST | admin | 重置密码 |
+| `/modelTF/system/permissions/codes` | GET | 登录 | 权限码清单 |
+| `/modelTF/system/permissions` | GET | 登录 | 权限码 + 角色定义 |
+
+### 12.2 资源 ACL 接口
+
+| 接口 | 方法 | 鉴权 | 说明 |
+|---|---|---|---|
+| `/modelTF/resources/:type/:id/acl` | GET | admin 或所有者 | 查询资源 ACL |
+| `/modelTF/resources/:type/:id/acl` | PUT | admin 或所有者 | 设置资源 ACL |
+
+### 12.3 GPU 分配接口
+
+| 接口 | 方法 | 鉴权 | 说明 |
+|---|---|---|---|
+| `/modelTF/compute/gpu-assignments` | GET | admin | 查看全部分配 |
+| `/modelTF/compute/gpu-assignments` | POST | admin | 批量分配 |
+| `/modelTF/compute/gpu-assignments/:id` | DELETE | admin | 撤销分配 |
+| `/modelTF/compute/my-gpus` | GET | 登录 | 查看自己可用 GPU |
+
+### 12.4 审批接口
+
+| 接口 | 方法 | 鉴权 | 说明 |
+|---|---|---|---|
+| `/modelTF/approvals/templates` | GET/POST | admin | 审批模板列表/创建 |
+| `/modelTF/approvals` | GET/POST | 登录 | 审批实例列表/创建 |
+| `/modelTF/approvals/:id` | GET | 登录 | 审批实例详情 |
+| `/modelTF/approvals/:id/steps/:idx/decision` | POST | 审批人 | 审批决策 |
+
+### 12.5 审计接口
+
+| 接口 | 方法 | 鉴权 | 说明 |
+|---|---|---|---|
+| `/modelTF/system/audit-logs` | GET | admin | 审计日志分页查询 |
+| `/modelTF/system/audit-logs/export` | GET | admin | CSV 导出 |
+
+### 12.6 租户/项目接口
+
+| 接口 | 方法 | 鉴权 | 说明 |
+|---|---|---|---|
+| `/modelTF/tenants` | GET/POST | admin | 租户列表/创建 |
+| `/modelTF/tenants/:id` | GET/PUT | admin | 租户详情/更新 |
+| `/modelTF/tenants/:id/quota` | PUT | admin | 设置配额 |
+| `/modelTF/tenants/:id/retention-policy` | PUT | admin | 绑定留存策略 |
+| `/modelTF/projects` | GET/POST | 登录 | 项目列表/创建 |
+| `/modelTF/projects/:id` | GET/PUT | 登录 | 项目详情/更新 |
+| `/modelTF/projects/:id/archive` | POST | admin 或所有者 | 归档（审批拦截） |
+| `/modelTF/projects/:id/members` | GET/POST | 登录 | 成员列表/添加 |
+| `/modelTF/projects/:id/members/:uid` | PUT/DELETE | admin 或所有者 | 改角色/移除 |
+
+### 12.7 留存策略接口
+
+| 接口 | 方法 | 鉴权 | 说明 |
+|---|---|---|---|
+| `/modelTF/retention-policies` | GET/POST | admin | 策略列表/创建 |
+| `/modelTF/retention-policies/:id` | GET/PUT/DELETE | admin | 策略详情/更新/删除 |
+
+---
+
+## 13. 前端权限控制
+
+### 13.1 路由守卫
+
+```typescript
+// router/index.ts
+router.beforeEach((to, _from, next) => {
+  const auth = useAuthStore()
+  auth.syncSession()
+
+  if (to.meta.public) {
+    next()
+    return
+  }
+
+  if (!auth.isLoggedIn) {
+    next({ name: 'login' })
+    return
+  }
+
+  if (!to.meta.skipPermission) {
+    const permission = requiredPermission(to.path, to.meta.permission)
+    if (permission && !auth.hasPermission(permission)) {
+      next({ name: 'permission-denied', replace: true })
+      return
+    }
+  }
+
+  next()
+})
+```
+
+### 13.2 侧边栏过滤
+
+```typescript
+// layouts/MainLayout.vue
+const visibleMenus = computed(() =>
+  allMenus.filter(menu => {
+    if (!menu.permission) return true
+    return auth.hasPermission(menu.permission)
+  })
+)
+```
+
+### 13.3 资源级按钮控制
+
+```vue
+<!-- 数据集详情页 -->
+<template>
+  <el-button v-if="canEdit" @click="handleEdit">编辑</el-button>
+  <el-button v-if="canDelete" @click="handleDelete">删除</el-button>
+  <el-button v-if="canManageAcl" @click="showAclDialog = true">资源授权</el-button>
+</template>
+
+<script setup>
+const canEdit = computed(() =>
+  isAdmin.value || resource.value.created_by === userId.value
+)
+const canDelete = computed(() =>
+  isAdmin.value || resource.value.created_by === userId.value
+)
+const canManageAcl = computed(() =>
+  isAdmin.value || resource.value.created_by === userId.value
+)
+</script>
+```
+
+### 13.4 GPU 选择过滤
+
+```vue
+<!-- 创建训练任务页 -->
+<template>
+  <el-select v-model="selectedGpus" multiple>
+    <el-option
+      v-for="gpu in availableGpus"
+      :key="gpu.id"
+      :label="`${gpu.node_name} GPU ${gpu.gpu_index}`"
+      :value="gpu.id"
+    />
+  </el-select>
+  <el-alert v-if="availableGpus.length === 0 && !isAdmin" type="warning">
+    未分配 GPU，请联系管理员
+  </el-alert>
+</template>
+
+<script setup>
+// 普通用户只看到 my-gpus 返回的列表
+// admin 看到全部 GPU
+const availableGpus = ref([])
+async function loadGpus() {
+  if (isAdmin.value) {
+    availableGpus.value = await getAllGpus()
+  } else {
+    availableGpus.value = await getMyGpus()
+  }
+}
+</script>
+```
+
+---
+
+## 14. 安全设计补充
+
+### 14.1 密码安全
+
+- 密码使用 PBKDF2-SHA256 存储（salt + 390000 次迭代）
+- 旧系统明文密码在首次登录时自动升级为哈希
+- 管理员可重置用户密码，用户不可自行修改密码（本期设计）
+- 默认密码：`platform123`（创建用户时由管理员设定）
+
+### 14.2 会话安全
+
+| 规则 | 说明 |
+|---|---|
+| Token 格式 | `platform-token-{user_id}` |
+| 会话超时 | 默认 30 分钟无操作自动过期 |
+| 并发会话 | 同一用户可有多会话，各自独立计时 |
+| 会话续期 | 前端定时调用 `auth.refresh()` 续期 |
+| 强制下线 | admin 可通过修改用户 status=disabled 使其 token 失效 |
+
+### 14.3 操作限流
+
+| 接口 | 限制 |
+|---|---|
+| `/login` | 同一 IP 5 次/分钟，失败后 30 秒冷却 |
+| 文件上传 | 单文件最大由配置控制，默认 2GB |
+| 训练任务创建 | 同一用户并发运行任务数受 GPU 分配限制 |
+
+### 14.4 数据安全
+
+| 规则 | 说明 |
+|---|---|
+| 软删除 | 数据集、模型、任务使用 `deleted_at` 标记，保留审计可追溯 |
+| 敏感字段 | API 密钥（`api_key`）在列表接口不返回明文 |
+| 下载审计 | 数据集下载产生审计日志，记录下载人和时间 |
+| 导出审计 | 训练产物导出产生审计日志 |
+
+### 14.5 多租户隔离
+
+| 规则 | 说明 |
+|---|---|
+| 租户隔离 | 同一租户内的资源相互可见；跨租户默认不可见 |
+| 项目隔离 | 项目内资源受项目 ACL 控制；项目间默认不可见 |
+| admin 旁路 | admin 可跨租户/项目访问全部资源 |
+| 配额管控 | 租户级配额限制 GPU 并发数、存储容量、最大项目数 |
+
+---
+
+## 15. 实施计划
+
+### 15.1 已实现
+
+| 功能 | 状态 |
+|---|---|
+| 登录/会话/Token | ✅ 已实现 |
+| 用户 CRUD + 权限码 | ✅ 已实现 |
+| 角色定义 | ✅ 已实现 |
+| 资源 ACL（acls 表 + 接口） | ✅ 已实现 |
+| 审计日志（查询 + 导出） | ✅ 已实现 |
+| 审批模板/实例 | ✅ 已实现 |
+| 项目空间 + 成员 | ✅ 已实现 |
+| 租户 + 配额 + 留存 | ✅ 已实现 |
+| 审批拦截（项目归档/删除） | ✅ 已实现 |
+| 资源所有权 ACL 字段适配（subject_type/permissions[]） | ✅ 已实现 |
+
+### 15.2 待实现
+
+| 功能 | 优先级 | 涉及表/接口 |
+|---|---|---|
+| GPU 分配表 + 接口 | P0 | `gpu_assignments` 表 + `/compute/gpu-assignments` + `/compute/my-gpus` |
+| 资源 `created_by` 字段补充 | P0 | `datasets` / `models` / `trained_models` / `eval_tasks` 表 ALTER |
+| 资源列表按 `created_by` + ACL 过滤 | P0 | `platform_store.py` 中 datasets/models/tasks 列表方法 |
+| GPU 选择校验（训练/评测/推理创建时） | P0 | `platform.py` 中 create_task/eval/inference |
+| 前端 GPU 下拉过滤 | P1 | 前端创建任务页面 |
+| 前端资源授权按钮 | P1 | 前端数据集/模型/任务详情页 |
+| 前端权限管理页面优化 | P1 | 前端用户设置页面 |
+| 审批拦截扩展（删除数据集/模型/停止任务） | P1 | `platform.py` 中 delete/stop 接口 |
+| 密码安全策略（用户自行修改） | P2 | 新增 `/users/me/password` 接口 |
+| 操作限流（login 限流） | P2 | 中间件或 SlowAPI |
+| 多租户隔离（按 tenant_id 过滤） | P2 | 各列表接口增加 tenant_id 过滤 |
+
+### 15.3 实施步骤
+
+1. **数据库迁移**：创建 `gpu_assignments` 表，为资源表补充 `created_by` 字段
+2. **后端接口**：实现 GPU 分配 CRUD + `my-gpus` + 创建任务时的 GPU 权限校验
+3. **资源过滤**：在 `datasets()` / `models()` / `tasks()` 等列表方法中按 `created_by` + ACL 过滤
+4. **前端适配**：GPU 下拉过滤、资源授权按钮、权限管理页面优化
+5. **审批扩展**：在删除/停止接口中接入 `_require_approval_or_admin`
+6. **测试补充**：扩展 `test_governance.py` 覆盖 GPU 分配、资源过滤、审批扩展场景
+## 16. 2.0 权限增强基线（MinIO、缓存与跨节点场景）
+
+### 16.1 默认拒绝
+
+所有受保护接口采用 deny by default。权限判定依次执行：认证、租户边界、项目成员关系、页面权限、资源动作权限、审批校验和审计记录。任何一步无法确定时返回 403，不得因为字段缺失、资源不存在或 ACL 查询异常而自动放行。
+
+### 16.2 资源归属和继承
+
+资源统一使用 `tenant_id`、`project_id`、`created_by`、`visibility` 表达边界。训练任务继承模型、数据集和项目边界；训练模型继承训练任务边界；合并模型继承被合并模型边界；评测必须同时校验模型和数据集；推理必须校验模型、项目和算力节点。
+
+### 16.3 MinIO 对象权限
+
+- 所有对象访问必须经过 Backend 鉴权，前端不得持有 MinIO 密钥。
+- Compute API 只使用 Backend 签发的预签名 URL。
+- 预签名 URL 默认有效期不超过 15 分钟，上传和下载分别签发。
+- 生成 URL 前必须校验资源权限、对象状态和版本归属。
+- bucket 由服务端配置，禁止客户端提交任意 bucket。
+- 禁止通过修改 `object_key`、`version_id` 或文件名越权访问对象。
+- 删除对象使用 `deleting -> deleted` 状态，失败时保留错误信息并可重试。
+
+### 16.4 缓存和算力节点权限
+
+- 缓存是资源副本，不产生新的资源所有权。
+- 只有拥有源模型或数据集 `execute` 权限的用户才能触发缓存。
+- 用户不能直接调用 Compute API 的缓存、文件、上传和推理管理接口。
+- Compute API 只接受 Backend 服务令牌，不能转发用户 Token。
+- 产物归档必须校验任务、节点、资源和项目关联关系。
+- 任务运行期间缓存引用不可被普通用户清理。
+- 缓存清理只能删除节点副本，不得删除 MinIO 正式对象。
+
+### 16.5 训练、合并、推理和评测动作矩阵
+
+| 动作 | 必要权限 | 额外约束 |
+|---|---|---|
+| 创建训练 | 项目 `write` + 模型/数据集 `execute` | GPU、配额和节点权限同时通过 |
+| 查看训练 | 任务 `read` | 日志、曲线、checkpoint 继承任务权限 |
+| 停止训练 | 任务 `write` 或 `admin` | 停止他人任务需要审批或管理员权限 |
+| 权重合并 | 训练模型 `execute` | 使用绑定节点或有权限的指定节点 |
+| 归档训练产物 | 任务 `write` | 只能归档任务输出目录内文件 |
+| 创建推理 | 模型 `execute` | 节点、GPU 配额和项目权限通过 |
+| 删除推理 | 推理服务 `delete` 或管理员 | 释放 GPU 和缓存引用 |
+| 创建评测 | 模型/数据集 `execute` | 两个资源必须在允许范围内 |
+| 下载报告 | 评测任务 `read` + 报告 `download` | 预签名 URL 短时有效 |
+
+### 16.6 服务身份和密钥边界
+
+| 身份 | 用途 | 禁止事项 |
+|---|---|---|
+| 用户 Token | 调用 Backend 业务接口 | 直接调用 Compute 或 MinIO |
+| Backend 服务令牌 | 调用 Compute API | 返回前端或写入任务参数 |
+| MinIO 管理密钥 | Backend 对象操作 | 注入浏览器、Compute 容器或日志 |
+| Compute 节点身份 | 节点心跳和任务执行 | 访问其他节点本地路径 |
+
+生产环境禁止使用默认凭据；服务令牌必须从环境变量或密钥管理系统读取并脱敏记录。
+
+### 16.7 必须补充的接口保护
+
+```text
+POST /storage/objects/presign
+POST /storage/resources/{type}/{id}/prepare/{node_id}
+POST /storage/resources/{type}/{id}/archive-node/{node_id}
+GET  /storage/cache/jobs/{node_id}
+POST /model-manage/merge
+POST /model-chat/local/preload
+POST /model-chat/trained/preload
+POST /model-chat/local/unload
+POST /model-compare/{task_id}/load
+POST /model-compare/{task_id}/unload
+```
+
+请求体中的所有资源 ID 都必须校验。`node_id`、`model_id`、`dataset_id`、`task_id` 不一致时返回 403 或 409。
+
+### 16.8 权限审计验收用例
+
+必须覆盖：用户 A 不能读取用户 B 资源；项目 A 不能使用项目 B 数据集；评测必须同时拥有模型和数据集 `execute` 权限；修改对象 key、版本或资源 ID 不能获取预签名 URL；普通用户不能调用 Compute API；产物路径不能越出任务数据根目录；用户不能清理其他项目正在使用的缓存；无节点权限时推理返回 403；删除、停止、合并和归档他人资源按规则进入审批。
+
+## 17. 2.0 实施顺序
+
+1. 建立统一 `authorize_resource_action()` 和 `authorize_task_resources()` 后端辅助函数。
+2. 为模型、数据集、训练、推理、评测和对象接口补齐租户/项目过滤。
+3. 统一预签名 URL 权限校验、有效期和审计日志。
+4. 为 Compute API 增加服务令牌、节点归属和路径范围校验。
+5. 将 GPU、缓存、配额和节点选择校验合并到任务创建事务中。
+6. 增加跨资源权限测试和越权回归测试。
+7. 前端仅负责隐藏操作按钮，最终权限以 Backend 返回为准。

docs/security-hardening.md

@@ -0,0 +1,290 @@
+# 安全加固总结（前端 / 后端 / 算力节点）
+
+> 记录 2026-08-06 对本平台的漏洞修复。核心目标：修复 **FastAPI 文档接口未授权访问**、
+> **Swagger 泄露 API 结构**、以及两类**任意文件读取**漏洞（路径穿越 + 符号链接跟随），
+> 修复过程不改变正常业务流程。
+>
+> 其中 **FastAPI 文档开关（`ENABLE_DOCS`）** 的详细用法见
+> [§4 FastAPI 文档开关使用说明](#4-fastapi-文档开关使用说明enabledocs)。
+
+---
+
+## 1. 漏洞总览
+
+| # | 影响面 | 漏洞 | 风险等级 | 修复 |
+|---|--------|------|----------|------|
+| 1 | 后端 + 算力节点 | FastAPI 默认暴露 `/docs`、`/redoc`、`/openapi.json`，**未授权**泄露全部 API 结构、参数、内部路由 | 中 | 生产环境关闭文档路由，访问返回 404（`ENABLE_DOCS` 可覆盖） |
+| 2 | 后端 | `data-convert` 模块 `output_filename` **路径穿越**：可任意文件读 / 写 / 删，且整个模块**无鉴权** | **严重** | 输出文件名白名单校验 + 全部端点补鉴权 |
+| 3 | 算力节点 | `compute/files/{file_id}/download`：`file_id` 直接拼进 glob 模式可 `../` **穿越出上传目录**，`FileResponse` 在 Linux 上**跟随符号链接**读取任意文件 | 中高 | `file_id` 字符白名单 + 解析后路径包含性二次校验 |
+| 4 | 前端（Vue + nginx） | 无文件服务代码；nginx 仅服务受控静态目录，无 `alias` | 无 | 审计确认，无需修复 |
+
+---
+
+## 2. 前端（Vue 3 + nginx）
+
+**审计结论：不构成 ComfyUI `follow_symlinks` 类文件读取漏洞。**
+
+- nginx（`docker/nginx.conf.template`）只服务受控的 `dist/` 静态目录，使用 `try_files`，
+  无 `alias` 指令、无用户可控文件路径，不存在路径穿越面。
+- Vue SPA 自身没有任何文件服务逻辑；文件下载全部走后端/算力节点 API。
+- 前端 axios 拦截器（`frontend/src/api/request.ts`）对**每个请求**自动附加
+  `Authorization: Bearer platform-token-{user_id}`，因此给后端接口补鉴权不会影响页面功能。
+
+---
+
+## 3. 后端（FastAPI）
+
+### 3.1 data_convert 输出文件名路径穿越（严重）
+
+**问题**：`backend/app/modules/data_convert/router.py`
+
+- `output_filename` 由请求体传入后**原样入库**，随后拼进
+  `output_dir / output_filename` 用于写/读/删：
+  - `if output_path.exists(): output_path.unlink()` → 任意文件删除
+  - `open(output_path, "a")` → 任意文件追加写
+  - `download_result` 用 `FileResponse(output_path)` → 任意文件读取
+- 整个 router **无任何鉴权依赖**（后端无全局鉴权中间件），任意网络访问者可利用。
+
+**修复**：
+
+```python
+def _safe_output_filename(value: Any) -> str:
+    """输出文件名白名单：拒绝 ../、/、\ 及控制字符，仅允许普通文件名。"""
+    name = str(value or "converted-data.jsonl").strip()
+    if (
+        not name
+        or name in {".", ".."}
+        or name != Path(name).name
+        or "/" in name
+        or "\\" in name
+        or any(ord(c) < 32 or ord(c) == 127 for c in name)
+    ):
+        raise fail(400, "output filename must be a plain file name")
+    return name
+
+def _task_output_path(task: dict[str, Any]) -> Path:
+    """统一构造转换输出路径，始终位于任务 output 目录内。"""
+    return _output_dir(task["id"]) / _safe_output_filename(task.get("output_filename"))
+```
+
+- `create_task` 创建时即校验（恶意值直接 400）
+- 全部 4 处使用点（`upload_source_files` 自动转换、`run_convert`、`download_result`、
+  `import_as_dataset`）统一改用 `_task_output_path()`，历史任务同样受保护
+- 全部 **8 个** `/data-convert` 端点补充 `current_user: dict = Depends(get_current_user)` 鉴权
+
+**功能影响**：正常转换流程（前端 `outputName + '.jsonl'` 这类纯文件名）不受影响；
+接口现在要求登录态，未登录调用返回 401。
+
+---
+
+## 4. FastAPI 文档开关使用说明（`ENABLE_DOCS`）
+
+### 4.1 为什么需要这个开关
+
+FastAPI 默认注册 3 个**无需鉴权**的路由，直接泄露全部 API 结构：
+
+| 路由 | 说明 |
+|------|------|
+| `/docs` | Swagger UI 交互文档 |
+| `/redoc` | ReDoc 文档 |
+| `/openapi.json` | OpenAPI Schema（含全部接口、参数、模型定义） |
+
+修复方式是：**关闭时让 FastAPI 不注册这 3 个路由**，访问一律返回 404，而不是返回空页面。
+
+### 4.2 核心实现
+
+关闭的本质是向 `FastAPI(...)` 传入三个 `None` 参数：
+
+```python
+# docs 关闭时等价于：
+FastAPI(
+    title=...,
+    docs_url=None,     # /docs      → 404
+    redoc_url=None,    # /redoc     → 404
+    openapi_url=None,  # /openapi.json → 404
+)
+```
+
+### 4.3 后端开关逻辑（`backend/app/core/config.py` + `backend/app/main.py`）
+
+```python
+# config.py —— Settings.enable_docs 在 __post_init__ 中计算
+object.__setattr__(
+    self,
+    "enable_docs",
+    _bool_env("ENABLE_DOCS", os.getenv("APP_ENV", "local") != "prod"),
+)
+
+# config.py —— 返回传给 FastAPI 的文档参数
+def docs_kwargs(enabled: bool) -> dict[str, Any]:
+    if enabled:
+        return {}
+    return {"docs_url": None, "redoc_url": None, "openapi_url": None}
+
+# main.py —— 接入
+app = FastAPI(title=settings.app_name, **docs_kwargs(settings.enable_docs))
+```
+
+**判定顺序（优先级从高到低）**：
+
+1. 显式设置 `ENABLE_DOCS=true/false` → 以显式值为准
+2. 未设置 → `APP_ENV != "prod"` 时开放，`APP_ENV=prod` 时**关闭**
+
+> 注意：`enable_docs` 从**运行时环境**读取 `APP_ENV`（而非类定义时缓存的默认值），
+> 确保生产环境默认关闭始终生效且便于测试。
+
+### 4.4 算力节点开关逻辑（`compute/api/security.py` + `compute/api/main.py`）
+
+```python
+# security.py
+def docs_enabled() -> bool:
+    raw = os.getenv("ENABLE_DOCS", "").strip().lower()
+    if raw in {"true", "false"}:
+        return raw == "true"
+    auth_enabled = os.getenv("COMPUTE_AUTH_ENABLED", "true").lower() == "true"
+    return not auth_enabled  # 开启 token 鉴权（生产）时默认关闭文档
+
+def docs_kwargs() -> dict[str, Any]:
+    if docs_enabled():
+        return {}
+    return {"docs_url": None, "redoc_url": None, "openapi_url": None}
+
+# main.py
+app = FastAPI(title="YG Fine-Tune Compute API", **docs_kwargs())
+```
+
+**判定顺序（优先级从高到低）**：
+
+1. 显式设置 `ENABLE_DOCS=true/false` → 以显式值为准
+2. 未设置 → `COMPUTE_AUTH_ENABLED=true`（生产默认）时**关闭**；
+   `COMPUTE_AUTH_ENABLED=false`（本地开发）时开放
+
+### 4.5 环境变量速查表
+
+| 服务 | 环境变量 | 取值 | 默认行为 |
+|------|----------|------|----------|
+| 后端 | `ENABLE_DOCS` | `true` / `false` | 未设置时按 `APP_ENV != "prod"` 判定 |
+| 后端 | `APP_ENV` | `local` / `prod` 等 | `prod` 时关闭文档 |
+| 算力节点 | `ENABLE_DOCS` | `true` / `false` | 未设置时按 `COMPUTE_AUTH_ENABLED` 判定 |
+| 算力节点 | `COMPUTE_AUTH_ENABLED` | `true` / `false` | `true` 时关闭文档 |
+
+### 4.6 Docker 部署配置
+
+已在以下文件加入 `ENABLE_DOCS=false`，并通过 docker-compose 透传（默认 `false`）：
+
+```
+docker/app/.env                     → ENABLE_DOCS=false
+docker/compute/.env                 → ENABLE_DOCS=false
+docker/compute/.env.example         → ENABLE_DOCS=false
+docker/app/docker-compose.yml       → ENABLE_DOCS: ${ENABLE_DOCS:-false}
+docker/compute/docker-compose.yml   → ENABLE_DOCS: ${ENABLE_DOCS:-false}
+```
+
+### 4.7 如何临时开启（排查/调试）
+
+```bash
+# 后端：非 prod 环境默认已开启；prod 环境临时开启
+ENABLE_DOCS=true docker compose -f docker/app/docker-compose.yml up -d backend-api
+
+# 算力节点：临时开启（生产默认关闭）
+ENABLE_DOCS=true docker compose -f docker/compute/docker-compose.yml up -d compute-api
+```
+
+> ⚠️ 仅在可信内网调试时开启，用毕改回 `false`。
+
+### 4.8 验证方法
+
+```bash
+# 关闭状态下三个地址均应返回 404
+curl -s -o /dev/null -w "%{http_code}\n" http://<host>/docs           # 404
+curl -s -o /dev/null -w "%{http_code}\n" http://<host>/redoc          # 404
+curl -s -o /dev/null -w "%{http_code}\n" http://<host>/openapi.json   # 404
+
+# 健康检查不受影响
+curl -s http://<host>/modelTF/health
+```
+
+---
+
+## 5. 算力节点（FastAPI）
+
+### 5.1 文档开关
+
+见 [§4.4](#44-算力节点开关逻辑computeapisecuritypy--computeapimainpy)，逻辑与后端一致，
+生产（`COMPUTE_AUTH_ENABLED=true`）默认关闭。
+
+> 补充：原 token 鉴权中间件已覆盖全部非 health 路径；现在文档路由同时被 FastAPI 层关闭，
+> 属于**纵深防御**（双重保护）。
+
+### 5.2 `download_file` glob 穿越 + 符号链接跟随（`compute/api/main.py`）
+
+**问题**：
+
+```python
+matches = list(upload_root.glob(f"{file_id}_*"))   # file_id 来自 URL，直接拼进 glob
+return FileResponse(matches[0])                     # 跟随符号链接
+```
+
+- `Path.glob` 支持 `..` 段，`file_id` 注入 `../` 可**穿越出 upload 目录**（已实测确认）
+- Linux 上目录内符号链接可被 `FileResponse` 跟随 → 读取任意文件
+
+**修复**：
+
+```python
+# 1) file_id 字符白名单：仅字母/数字/_/-，含 .、/、% 等一律 400
+if not file_id or not all(c.isalnum() or c in {"_", "-"} for c in file_id):
+    raise HTTPException(status_code=400, detail="invalid file id")
+matches = list(upload_root.glob(f"{file_id}_*"))
+if not matches:
+    raise HTTPException(status_code=404, detail="file not found")
+# 2) 解析符号链接后必须仍位于 upload 根目录内
+resolved = matches[0].resolve()
+if not _path_inside(upload_root, resolved):
+    raise HTTPException(status_code=404, detail="file not found")
+return FileResponse(resolved)
+```
+
+**功能影响**：服务端生成的 `file_<时间戳>` 格式完全兼容；外部工具使用正常 `file_id` 下载不受影响。
+
+### 5.3 已确认安全的同类文件端点（无需改动）
+
+| 端点 | 保护机制 |
+|------|----------|
+| `compute/files/list` | `_path_inside()` + `.resolve()`，符号链接逃逸被阻断 |
+| `compute/files/read` | 同上 |
+| `compute/files/upload` | 同上 + 文件名取 `.name` |
+| `compute/files/import-local` | 目标路径 `_path_inside()` 校验 |
+| 后端 `data-process` 存储 `LocalDataProcessStorage` | `lstat` + `S_ISLNK` + `O_NOFOLLOW` + 规范化引用校验，彻底防符号链接 |
+
+---
+
+## 6. 测试与验证结果
+
+| 验证项 | 结果 |
+|--------|------|
+| 计算节点全量测试 | **22 passed, 1 skipped**（跳过项为 Windows 无权限建符号链接，Linux 生产环境会执行） |
+| 新增 compute 下载安全测试 | 正常下载 200；穿越样本 400/404；符号链接逃逸 404 |
+| 新增后端 data_convert 安全测试 | **13 passed**（穿越样本 9 项全拦截 + 鉴权覆盖检查） |
+| 后端文档开关测试 | 6 passed（2 项 `create_app` 集成测试需完整依赖，在 WSL 下运行） |
+| 实时验证 | 生产环境 `/docs` `/redoc` `/openapi.json` 均返回 **404**；`download_file` 合法 `file_123456` 返回 200 |
+
+---
+
+## 7. 变更文件清单
+
+**后端**
+- `backend/app/core/config.py` — 新增 `_bool_env`、`docs_kwargs()`、`Settings.enable_docs`
+- `backend/app/main.py` — `FastAPI(...)` 接入 `docs_kwargs`
+- `backend/app/modules/data_convert/router.py` — 输出文件名白名单 + 全部端点补鉴权
+
+**算力节点**
+- `compute/api/security.py` — 新增文档开关模块（`docs_enabled` / `docs_kwargs`）
+- `compute/api/main.py` — 文档开关接入 + `download_file` 加固
+
+**Docker 配置**
+- `docker/app/.env`、`docker/compute/.env`、`docker/compute/.env.example` — `ENABLE_DOCS=false`
+- `docker/app/docker-compose.yml`、`docker/compute/docker-compose.yml` — 透传 `ENABLE_DOCS`
+
+**测试**
+- `backend/tests/test_docs_security.py`、`backend/tests/test_data_convert_security.py`
+- `compute/tests/test_security.py`、`compute/tests/test_file_download_security.py`