YG-Soft/YG_FT
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: YG-Soft:a12f80492d18b98dac078dbf0e18f8aafbe3aff2
Branches Tags
YG-Soft:main YG-Soft:ft_wyt YG-Soft:ft_wjm YG-Soft:dev YG-Soft:server
YG-Soft:baseline/ft_wyt-20260804
..
compare: YG-Soft:main
Branches Tags
YG-Soft:ft_wyt YG-Soft:ft_wjm YG-Soft:dev YG-Soft:main YG-Soft:server
YG-Soft:baseline/ft_wyt-20260804

2 Commits
a12f80492d ... main

Author SHA1 Message Date
caoxiaozhu
c9058a27b7 Merge pull request 'ft_wyt' (#6) from ft_wyt into main 
Reviewed-on: #6
 2026-07-21 14:23:17 +08:00
caoxiaozhu
1468834116 Merge pull request 'ft_wyt' (#5) from ft_wyt into main 
Reviewed-on: #5
 2026-07-16 13:54:08 +08:00
505 changed files with 4052 additions and 51869 deletions
Expand all files Collapse all files

28
.gitignore vendored
View File

@@ -12,6 +12,8 @@ __pycache__/
build/
develop-eggs/
dist/
!frontend/dist/
!frontend/dist/**
node_modules/
*.tsbuildinfo
downloads/
@@ -42,7 +44,6 @@ pip-delete-this-directory.txt
# Runtime data and logs
runtime/
backend/runtime/
backend/storage/
logs/
backend/logs/
*.db
@@ -142,7 +143,6 @@ celerybeat.pid
# Environments
.env
!.env.example
.venv
env/
venv/
@@ -150,16 +150,6 @@ ENV/
env.bak/
venv.bak/
# Local backend config (含数据库账号密码等敏感信息，勿提交)
backend/config.yaml
# Agent / IDE 工具产物，不应进版本库
.codex-backups/
.pnpm-store/
.zcode/
.claude/
CLAUDE.md
# Spyder project settings
.spyderproject
.spyproject
@@ -197,17 +187,3 @@ cython_debug/
# PyPI configuration file
.pypirc
docker/llamafactory-latest.tar.gz
# Compute data - 保留目录结构和 README，忽略子目录内容（日志、模型、数据集等）
!docker/compute/data/yg-ft/logs/
docker/compute/data/yg-ft/datasets/*
docker/compute/data/yg-ft/models/*
docker/compute/data/yg-ft/outputs/*
docker/compute/data/yg-ft/logs/**
!docker/compute/data/yg-ft/logs/compute/
!docker/compute/data/yg-ft/logs/training/
!docker/compute/data/yg-ft/**/.gitkeep
!docker/compute/data/yg-ft/**/README.md
# Offline deployment bundle - 离线部署包（镜像、运行时等大文件，不提交）
docker/offline/

129
README.md
View File

@@ -48,41 +48,6 @@ YG_FT/
- 前端新增 `/compute` 算力节点页面，展示节点地址、权重、标签、启用状态、GPU、队列和资源副本。
- `compute/engines/llama_factory/adapter.py` 提供 LLaMA-Factory 参数校验、命令生成和日志解析基础能力。
## 前后端一键启动
首次使用前请确保前端依赖已安装、PostgreSQL 已可用。之后在项目根目录执行：
```bash
bash ./start.sh
```
脚本会自动补装后端 `requirements.txt`，然后同时启动前端 `http://localhost:16801` 和后端
`http://127.0.0.1:17861`，按 `Ctrl+C` 会同时停止两个服务。脚本不会自动安装
前端依赖，也不会启动 PostgreSQL、Redis 或算力服务。
仅检查依赖和端口而不启动服务：
```bash
bash ./start.sh --check
```
本地启动推荐只配置数据库主机。脚本会复用 `docker/app/.env` 中已有的
`POSTGRES_USER`、`POSTGRES_PASSWORD` 和 `POSTGRES_DB`，端口默认使用
PostgreSQL 标准端口 `5432`：
```bash
DATABASE_HOST='www.caoxiaozhu.com' bash ./start.sh
```
也可以在 `docker/app/.env` 中增加：
```env
DATABASE_HOST=www.caoxiaozhu.com
```
需要使用非标准端口时再设置 `DATABASE_PORT`。`DATABASE_URL` 仍可作为完整连接串
高级覆盖项；终端环境变量优先级最高。脚本不会输出数据库密码。
## 后端启动
```bash
@@ -102,37 +67,15 @@ GET /modelTF/model-manage
GET /modelTF/dataset-manage
GET /modelTF/fine-tune
GET /modelTF/compute/nodes
GET /modelTF/data-convert
```
### 数据库配置
本地运行时默认 PostgreSQL 连接：
后端通过 `backend/.env` 文件配置数据库连接（自动加载，`override=True`）：
```env
DATABASE_URL=postgresql+psycopg://用户:密码@数据库地址:端口/库名
COMPUTE_SERVICE_TOKEN=change_me
```text
DATABASE_URL=postgresql+psycopg://yg_ft:change_me@localhost:15432/yg_ft
```
支持远程数据库。连接池参数已针对远程库优化（`connect_timeout=30`、`max_size=20`、`max_waiting=50`）。
### 环境变量
| 变量 | 默认值 | 说明 |
|---|---|---|
| `DATABASE_URL` | `postgresql+psycopg://yg_ft:change_me@localhost:15432/yg_ft` | 数据库连接串 |
| `COMPUTE_SERVICE_TOKEN` | `""` | 算力服务认证 token，需与 compute 一致 |
| `COMPUTE_STATUS_SYNC_MODE` | `polling` | `off` 禁用轮询（远程库慢时推荐） |
| `COMPUTE_POLL_INTERVAL_SECONDS` | `3` | 轮询间隔秒数 |
| `COMPUTE_REQUEST_TIMEOUT_SECONDS` | `5` | 调 compute 的超时秒数 |
### 推荐启动命令（远程数据库）
```cmd
cd /d E:\yg_ft\backend
set COMPUTE_STATUS_SYNC_MODE=off
.\.venv\Scripts\python.exe -m uvicorn app.main:app --reload --port 17861
```
本地启动前需要确保 PostgreSQL 已监听 `localhost:15432`，并已创建 `yg_ft` 数据库和 `yg_ft` 用户。后端启动后会自动创建当前运行表并写入内置管理员账号，运行数据统一写入 PostgreSQL。
开发阶段内置登录账号：
@@ -155,72 +98,12 @@ npm run dev
## 算力服务启动
算力服务是一个 FastAPI 应用，同时承载 Compute API（模型训练/推理/GPU 管理）和 File Gateway（文件上传下载）路由。Docker 部署时对外暴露两个端口（19100 和 19101）均指向同一服务，方便应用平台分别配置 `api_base_url` 和 `file_gateway_url`。本地开发只需启动一个进程。
### 方式一：Docker 启动（推荐）
**1. 构建镜像**（首次或依赖变更后）：
```cmd
cd /d E:\yg_ft
docker build -f docker/compute/Dockerfile.compute -t yg-ft-compute-api:latest .
```
**2. 启动容器**：
```cmd
cd docker/compute
cp .env.example .env
docker compose up -d
```
**3. 验证**：
```cmd
curl http://localhost:19100/health
```
> 注意：构建上下文必须是项目根目录 `E:\yg_ft`（`docker build` 最后的 `.`），因为 Dockerfile 需要 `COPY compute/requirements.txt`。
### 方式二：本地开发启动
**Windows (cmd)：**
```cmd
cd /d E:\yg_ft\compute
set PYTHONPATH=E:\yg_ft
.\.venv\Scripts\python.exe -m uvicorn api.main:app --reload --host 0.0.0.0 --port 19100
```
> `PYTHONPATH=E:\yg_ft` 是必需的，因为代码使用 `from compute.agent...` 绝对导入。
> `--host 0.0.0.0` 让其他机器可以通过 IP 访问（算力节点测试需要）。
**Linux / macOS：**
```bash
cd compute
PYTHONPATH=.. uvicorn api.main:app --reload --host 0.0.0.0 --port 19100
uvicorn api.main:app --reload --port 19100
```
### 算力节点配置
在平台的「算力节点」页面新增节点，填入：
- **Compute API**：`http://你的IP:19100`
- **File Gateway**：`http://你的IP:19101`
本机测试用 `http://localhost:19100`。
### 环境变量说明
| 变量 | 默认值 | 说明 |
|---|---|---|
| `COMPUTE_MODE` | `real` | `real` / `simulator`，仅隔离联调用 simulator |
| `COMPUTE_EXECUTION_MODE` | `real` | 训练执行模式 |
| `COMPUTE_SERVICE_TOKEN` | `change_me` | 服务间认证 token，需与 backend 一致 |
| `COMPUTE_AUTH_ENABLED` | `true` | 是否开启 token 认证 |
| `MODELTF_ROUTE_PREFIX` | `/modelTF` | API 路由前缀 |
应用平台通过数据库 `compute_nodes` 表中的 `api_base_url` 和 `file_gateway_url` 主动轮询算力节点状态。
默认 `COMPUTE_MODE=real`。真实 GPU 接入时，在每台算力服务器上部署 Compute API、Agent、File Gateway 和 LLaMA-Factory，应用平台通过 `compute_nodes.api_base_url` 和 `compute_nodes.file_gateway_url` 主动轮询。仅在隔离联调环境可显式设置 `COMPUTE_MODE=simulator` 或 `COMPUTE_EXECUTION_MODE=simulator`。
## 日志

9
backend/.env.example Normal file
View File

@@ -0,0 +1,9 @@
APP_NAME=YG Fine-Tune Platform API
APP_ENV=local
API_PREFIX=/api
LOG_LEVEL=INFO
LOG_DIR=./logs
LOG_FILE_PREFIX=backend
LOG_ERROR_FILE_PREFIX=error
LOG_MAX_BYTES=20971520
LOG_RETENTION_DAYS=10

10
backend/_check_sessions.py
View File

@@ -1,10 +0,0 @@
from app.db.platform_store import get_platform_store
store = get_platform_store()
with store.connect() as conn:
rows = conn.execute(
"SELECT id, user_id, login_at, logout_at, duration_seconds FROM sessions ORDER BY login_at DESC LIMIT 10"
).fetchall()
print(f"sessions count: {len(rows)}")
for r in rows:
print(f" user={r['user_id'][:25]}... login={r['login_at']} logout={r['logout_at']} dur={r['duration_seconds']}")

2503
backend/app/api/v1/endpoints/data_process.py
View File

File diff suppressed because it is too large Load Diff

6
backend/app/api/v1/endpoints/health.py
View File

@@ -10,9 +10,5 @@ logger = get_logger(__name__)
@router.get("/health")
async def health_check() -> dict[str, object]:
logger.info("health check requested")
return {
"code": 0,
"message": "ok",
"data": get_platform_store().health_metrics(),
}
return {"code": 0, "message": "ok", "data": get_platform_store().health_metrics()}

2130
backend/app/api/v1/endpoints/platform.py
View File

File diff suppressed because it is too large Load Diff

21
backend/app/api/v1/router.py
View File

@@ -1,26 +1,9 @@
from fastapi import APIRouter
from fastapi import APIRouter
from app.api.v1.endpoints.data_process import router as data_process_router
from app.api.v1.endpoints.platform import router as platform_router
from app.api.v1.endpoints.health import router as health_router
from app.modules.tenant.router import router as tenant_router
from app.modules.project.router import router as project_router
from app.modules.approval.router import router as approval_router
from app.modules.system.router import router as system_router
from app.modules.retention.router import router as retention_router
from app.modules.resource.router import router as resource_router
from app.modules.gpu.router import router as gpu_router
from app.modules.data_convert.router import router as data_convert_router
api_router = APIRouter()
api_router.include_router(health_router, tags=["health"])
api_router.include_router(data_process_router, tags=["data-process"])
api_router.include_router(platform_router, tags=["platform"])
api_router.include_router(system_router, tags=["system"])
api_router.include_router(tenant_router, tags=["tenant"])
api_router.include_router(project_router, tags=["project"])
api_router.include_router(approval_router, tags=["approval"])
api_router.include_router(retention_router, tags=["retention"])
api_router.include_router(resource_router, tags=["resource"])
api_router.include_router(gpu_router, tags=["gpu-assignment"])
api_router.include_router(data_convert_router, tags=["data-convert"])

138
backend/app/core/auth.py
View File

@@ -1,138 +0,0 @@
"""鉴权依赖：从 Authorization header 解析当前用户，提供权限校验。"""
from __future__ import annotations
from typing import Any
from fastapi import Depends, HTTPException, Query, Request, status
from app.db.platform_store import get_platform_store
# 无需鉴权的路径前缀（健康检查、登录等）
PUBLIC_PATHS = ("/health", "/login", "/system-info")
def _extract_token(request: Request) -> str | None:
"""从 Authorization header 提取 token（格式: Bearer platform-token-{user_id}）。"""
auth = request.headers.get("Authorization", "")
token = auth.replace("Bearer ", "").strip()
if token.startswith("platform-token-"):
return token[len("platform-token-"):]
return None
def get_current_user(request: Request) -> dict[str, Any]:
"""
FastAPI 依赖：解析当前登录用户。
- 公开路径（/health, /login 等）直接放行，返回匿名用户。
- 无 token 或 token 无效时抛 401。
- admin 用户标记为超级管理员，拥有全部权限。
"""
path = request.url.path
# 去掉路由前缀后判断
for prefix in PUBLIC_PATHS:
if path.endswith(prefix):
return {"id": None, "username": "anonymous", "role": "viewer", "permissions": [], "protected": False}
user_id = _extract_token(request)
if not user_id:
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="missing or invalid token")
store = get_platform_store()
for u in store.users():
if u.get("id") == user_id:
return u
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="user not found")
def require_admin(current_user: dict[str, Any] = Depends(get_current_user)) -> dict[str, Any]:
"""FastAPI 依赖：要求当前用户是管理员（role=admin 或 protected）。"""
if current_user.get("role") == "admin" or current_user.get("protected"):
return current_user
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="admin permission required")
def is_admin(user: dict[str, Any]) -> bool:
"""判断用户是否为管理员（admin 角色或 protected 标记）。"""
return user.get("role") == "admin" or user.get("protected", False)
def has_resource_access(
resource_type: str,
resource_id: str,
user: dict[str, Any],
permission: str = "read",
) -> bool:
"""
检查用户对某资源是否有指定权限。
- admin/protected 用户直接放行（旁路）。
- 其他用户检查 acls 表中是否有对应授权。
"""
if user.get("role") == "admin" or user.get("protected"):
return True
store = get_platform_store()
acls = store.get_acl(resource_type, resource_id)
user_id = user.get("id")
user_role = user.get("role")
for entry in acls:
# 按 user 授权
if entry.get("principal_type") == "user" and entry.get("principal_id") == user_id:
if _permission_covers(entry.get("permission"), permission):
return True
# 按 role 授权
if entry.get("principal_type") == "role" and entry.get("principal_id") == user_role:
if _permission_covers(entry.get("permission"), permission):
return True
return False
def _permission_covers(granted: str | None, required: str) -> bool:
"""权限覆盖判断：write/execute 覆盖 read；admin 覆盖一切。"""
if not granted:
return False
if granted == "admin":
return True
if granted == required:
return True
# write 覆盖 read
if required == "read" and granted in ("write", "execute"):
return True
return False
def filter_accessible_resource_ids(
resource_type: str,
all_ids: list[str],
user: dict[str, Any],
) -> list[str]:
"""
从全部资源 ID 中过滤出当前用户可访问的 ID 列表。
- admin 直接返回全部。
- 普通用户查 acls 表取交集。
"""
if user.get("role") == "admin" or user.get("protected"):
return all_ids
if not all_ids:
return []
store = get_platform_store()
user_id = user.get("id")
user_role = user.get("role")
# 查询该用户在该资源类型下有 read 权限的所有 resource_id
with store.connect() as conn:
rows = conn.execute(
"""
SELECT DISTINCT resource_id FROM acls
WHERE resource_type=? AND (
(principal_type='user' AND principal_id=?)
OR (principal_type='role' AND principal_id=?)
)
""",
(resource_type, user_id, user_role),
).fetchall()
accessible = {r["resource_id"] for r in rows}
return [rid for rid in all_ids if rid in accessible]

43
backend/app/core/config.py
View File

@@ -1,19 +1,6 @@
from dataclasses import dataclass
from functools import lru_cache
import os
from typing import Any
try:
from pathlib import Path as _Path
from dotenv import load_dotenv
# 显式指定 backend 目录下的 .env，并强制覆盖已有环境变量，
# 确保远程数据库配置生效，不被本地默认值或残留环境变量影响。
_env_path = _Path(__file__).resolve().parent.parent.parent / ".env"
load_dotenv(dotenv_path=_env_path, override=True)
except ImportError:
pass
def _int_env(name: str, default: int) -> int:
@@ -30,24 +17,6 @@ def _list_env(name: str, default: list[str]) -> list[str]:
return [item.strip() for item in raw.split(",") if item.strip()]
def _bool_env(name: str, default: bool) -> bool:
raw = os.getenv(name)
if raw is None or raw.strip() == "":
return default
return raw.strip().lower() in {"1", "true", "yes", "on"}
def docs_kwargs(enabled: bool) -> dict[str, Any]:
"""Swagger UI / ReDoc / OpenAPI schema 路由开关。
关闭时 FastAPI 不注册 /docs、/redoc、/openapi.json，访问一律返回 404，
避免未授权访问泄露 API 结构。
"""
if enabled:
return {}
return {"docs_url": None, "redoc_url": None, "openapi_url": None}
@dataclass(frozen=True)
class Settings:
app_name: str = os.getenv("APP_NAME", "YG Fine-Tune Platform API")
@@ -59,15 +28,12 @@ class Settings:
compute_mode: str = os.getenv("COMPUTE_MODE", "real")
compute_status_sync_mode: str = os.getenv("COMPUTE_STATUS_SYNC_MODE", "polling")
compute_poll_interval_seconds: int = _int_env("COMPUTE_POLL_INTERVAL_SECONDS", 3)
compute_request_timeout_seconds: int = _int_env("COMPUTE_REQUEST_TIMEOUT_SECONDS", 5)
compute_service_token: str = os.getenv("COMPUTE_SERVICE_TOKEN", "")
log_level: str = os.getenv("LOG_LEVEL", "INFO")
log_dir: str = os.getenv("LOG_DIR", "./logs")
log_file_prefix: str = os.getenv("LOG_FILE_PREFIX", "backend")
log_error_file_prefix: str = os.getenv("LOG_ERROR_FILE_PREFIX", "error")
log_max_bytes: int = _int_env("LOG_MAX_BYTES", 20 * 1024 * 1024)
log_retention_days: int = _int_env("LOG_RETENTION_DAYS", 10)
enable_docs: bool = None # type: ignore[assignment]
def __post_init__(self) -> None:
object.__setattr__(
@@ -83,15 +49,6 @@ class Settings:
],
),
)
# Swagger UI / ReDoc / OpenAPI 文档路由开关：
# 未显式配置 ENABLE_DOCS 时，仅本地/开发环境开放，生产环境默认关闭，
# 避免未授权访问泄露 API 结构。从运行时环境读取 APP_ENV，而非类定义时
# 缓存的默认值，保证生产默认关闭始终生效且便于测试。
object.__setattr__(
self,
"enable_docs",
_bool_env("ENABLE_DOCS", os.getenv("APP_ENV", "local") != "prod"),
)
@lru_cache

3313
backend/app/db/platform_store.py
View File

File diff suppressed because it is too large Load Diff

755
backend/app/db/sql/000_full_init.sql
View File

@@ -1,755 +0,0 @@
-- ============================================================================
-- YG Fine-Tune Platform — PostgreSQL 完整初始化脚本（一键建库建表）
-- ============================================================================
-- 用途：切换到新的 PG 数据集时，一次性创建平台运行所需的全部数据库对象与
-- 基础种子数据（幂等，可重复执行）。
--
-- 覆盖范围（与运行时代码实际使用的表一致）：
-- 001_platform_runtime.sql 平台核心表
-- 002_governance.sql 治理表（租户 / 审批 / 审计 / 留存）
-- 003_tenant_quota.sql 租户配额列
-- 003_model_path_governance.sql 模型可训练标识列
-- 002_data_process.sql 数据处理表 + 数据集扩展列
-- 本文件补充：data_convert_tasks（数据转换任务，运行时代码引用但原脚本缺失）
-- 种子数据：admin / operator 两个初始用户
--
-- 说明：
-- * 本脚本通过 psql 执行，包含 DO $$ ... $$ 块与事务，不能用应用的
-- executescript()（按分号切分）执行。
-- * 应用启动时 PlatformStore.ensure_schema() 只会自动执行
-- 001 / 002_governance / 003_tenant_quota；数据处理表需另跑
-- 002_data_process.sql（本脚本已包含）。应用首次启动还会自动补充
-- admin/operator 种子用户（本脚本已包含，二选一即可）。
-- * 脚本内所有 DDL 均使用 IF NOT EXISTS / ADD COLUMN IF NOT EXISTS，
-- 可在已初始化的库上安全重复执行。
--
-- 执行步骤（详见 docs/database-config.md）：
-- 1. 以超级用户创建角色与数据库（必须单独执行，不能放进事务）：
-- CREATE ROLE yg_ft LOGIN PASSWORD '请改为强密码';
-- CREATE DATABASE yg_ft OWNER yg_ft;
-- 2. 连接目标库执行本脚本：
-- psql "postgresql://yg_ft:密码@<host>:5432/yg_ft" -f backend/app/db/sql/000_full_init.sql
-- 3. 可选：为 superuser 授权
-- ALTER ROLE yg_ft SUPERUSER; -- 仅当需要执行 CREATE EXTENSION 等
-- ============================================================================
BEGIN;
-- ============================================================================
-- 一、平台核心表（来源：001_platform_runtime.sql）
-- ============================================================================
CREATE TABLE IF NOT EXISTS users (
id TEXT PRIMARY KEY,
username TEXT NOT NULL UNIQUE,
password_hash TEXT NOT NULL,
display_name TEXT NOT NULL,
role TEXT NOT NULL,
status TEXT NOT NULL,
permissions TEXT NOT NULL,
create_time TEXT NOT NULL,
last_login TEXT,
protected INTEGER NOT NULL DEFAULT 0
);
CREATE TABLE IF NOT EXISTS models (
id TEXT PRIMARY KEY,
name TEXT NOT NULL UNIQUE,
type TEXT NOT NULL,
purpose TEXT NOT NULL,
model_source TEXT NOT NULL,
description TEXT,
path TEXT,
api_url TEXT,
api_key TEXT,
online_model_name TEXT,
can_train INTEGER NOT NULL DEFAULT 0,
create_time TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS trained_models (
id TEXT PRIMARY KEY,
name TEXT NOT NULL UNIQUE,
train_methods TEXT NOT NULL,
base_model_path TEXT,
create_time TEXT NOT NULL,
merged INTEGER NOT NULL DEFAULT 0,
merging INTEGER NOT NULL DEFAULT 0,
merged_path TEXT,
artifact_dir TEXT,
compute_node_id TEXT,
compute_node_name TEXT
);
CREATE TABLE IF NOT EXISTS model_lineage (
id TEXT PRIMARY KEY,
child_resource_type TEXT NOT NULL,
child_resource_id TEXT NOT NULL,
parent_resource_type TEXT NOT NULL,
parent_resource_id TEXT NOT NULL,
relation_type TEXT NOT NULL,
compute_job_id TEXT,
payload TEXT NOT NULL,
create_time TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS model_artifacts (
id TEXT PRIMARY KEY,
model_id TEXT NOT NULL,
model_kind TEXT NOT NULL,
artifact_type TEXT NOT NULL,
path TEXT NOT NULL,
size_bytes BIGINT NOT NULL DEFAULT 0,
checksum_sha256 TEXT,
metadata TEXT NOT NULL,
compute_job_id TEXT,
create_time TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS model_export_jobs (
id TEXT PRIMARY KEY,
trained_model_id TEXT,
compute_job_id TEXT NOT NULL,
node_id TEXT,
export_type TEXT NOT NULL,
quantization_bit INTEGER NOT NULL DEFAULT 0,
status TEXT NOT NULL,
output_dir TEXT,
payload TEXT NOT NULL,
create_time TEXT NOT NULL,
completed_at TEXT
);
CREATE TABLE IF NOT EXISTS datasets (
id TEXT PRIMARY KEY,
name TEXT NOT NULL UNIQUE,
type TEXT NOT NULL,
storage_type TEXT NOT NULL,
source TEXT NOT NULL,
task_id TEXT,
size TEXT,
count INTEGER NOT NULL DEFAULT 0,
description TEXT,
create_time TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS dataset_files (
id TEXT PRIMARY KEY,
dataset_id TEXT NOT NULL REFERENCES datasets(id) ON DELETE CASCADE,
name TEXT NOT NULL,
size TEXT,
content TEXT NOT NULL,
active_version_id TEXT NOT NULL,
versions TEXT NOT NULL,
create_time TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS compute_nodes (
id TEXT PRIMARY KEY,
code TEXT NOT NULL UNIQUE,
name TEXT NOT NULL,
api_base_url TEXT NOT NULL,
file_gateway_url TEXT NOT NULL,
enabled INTEGER NOT NULL DEFAULT 1,
scheduler_status TEXT NOT NULL,
scheduler_weight INTEGER NOT NULL DEFAULT 100,
tags TEXT NOT NULL,
gpu_count INTEGER NOT NULL DEFAULT 0,
current_running_jobs INTEGER NOT NULL DEFAULT 0,
max_parallel_jobs INTEGER NOT NULL DEFAULT 2,
data_root TEXT NOT NULL,
model_root TEXT NOT NULL,
log_root TEXT NOT NULL,
api_version TEXT NOT NULL DEFAULT 'v1',
capabilities TEXT NOT NULL DEFAULT '[]',
description TEXT,
last_health_check_at TEXT,
health_detail TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS gpus (
id TEXT PRIMARY KEY,
node_id TEXT NOT NULL REFERENCES compute_nodes(id) ON DELETE CASCADE,
gpu_index INTEGER NOT NULL,
uuid TEXT NOT NULL,
name TEXT NOT NULL,
memory_total_gb DOUBLE PRECISION NOT NULL,
power_limit_w DOUBLE PRECISION NOT NULL,
base_temperature INTEGER NOT NULL,
last_seen_at TEXT
);
CREATE TABLE IF NOT EXISTS fine_tune_tasks (
id TEXT PRIMARY KEY,
name TEXT NOT NULL UNIQUE,
payload TEXT NOT NULL,
status TEXT NOT NULL,
progress INTEGER NOT NULL DEFAULT 0,
process_id INTEGER,
create_time TEXT NOT NULL,
start_time TEXT,
completed_at TEXT,
compute_node_id TEXT REFERENCES compute_nodes(id) ON DELETE SET NULL,
gpus TEXT NOT NULL,
sync_job_id TEXT,
compute_job_id TEXT
);
CREATE TABLE IF NOT EXISTS fine_tune_metrics (
id TEXT PRIMARY KEY,
task_id TEXT NOT NULL REFERENCES fine_tune_tasks(id) ON DELETE CASCADE,
step INTEGER NOT NULL,
epoch DOUBLE PRECISION,
loss DOUBLE PRECISION,
grad_norm DOUBLE PRECISION,
learning_rate DOUBLE PRECISION,
raw TEXT NOT NULL,
create_time TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS fine_tune_checkpoints (
id TEXT PRIMARY KEY,
task_id TEXT NOT NULL REFERENCES fine_tune_tasks(id) ON DELETE CASCADE,
step INTEGER NOT NULL,
name TEXT NOT NULL,
path TEXT NOT NULL,
size_bytes BIGINT NOT NULL DEFAULT 0,
create_time TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS compute_jobs (
id TEXT PRIMARY KEY,
task_id TEXT REFERENCES fine_tune_tasks(id) ON DELETE SET NULL,
node_id TEXT REFERENCES compute_nodes(id) ON DELETE SET NULL,
engine TEXT NOT NULL,
status TEXT NOT NULL,
command TEXT NOT NULL,
output_dir TEXT,
log_file TEXT,
payload TEXT NOT NULL,
create_time TEXT NOT NULL,
update_time TEXT NOT NULL,
completed_at TEXT
);
CREATE TABLE IF NOT EXISTS gpu_allocations (
id TEXT PRIMARY KEY,
task_id TEXT REFERENCES fine_tune_tasks(id) ON DELETE CASCADE,
compute_job_id TEXT,
node_id TEXT REFERENCES compute_nodes(id) ON DELETE CASCADE,
gpu_index INTEGER NOT NULL,
status TEXT NOT NULL,
create_time TEXT NOT NULL,
released_at TEXT
);
CREATE TABLE IF NOT EXISTS scheduler_locks (
lock_key TEXT PRIMARY KEY,
owner TEXT NOT NULL,
expires_at TEXT NOT NULL,
create_time TEXT NOT NULL,
update_time TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS resource_replicas (
id TEXT PRIMARY KEY,
node_id TEXT NOT NULL REFERENCES compute_nodes(id) ON DELETE CASCADE,
resource_type TEXT NOT NULL,
resource_id TEXT NOT NULL,
local_path TEXT NOT NULL,
status TEXT NOT NULL,
sync_status TEXT NOT NULL,
checksum_sha256 TEXT,
byte_size BIGINT NOT NULL DEFAULT 0,
last_checked_at TEXT,
last_error TEXT,
create_time TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS resource_sync_jobs (
id TEXT PRIMARY KEY,
target_node_id TEXT NOT NULL,
resources TEXT NOT NULL,
status TEXT NOT NULL,
progress INTEGER NOT NULL DEFAULT 0,
create_time TEXT NOT NULL,
completed_at TEXT
);
CREATE TABLE IF NOT EXISTS eval_tasks (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
payload TEXT NOT NULL,
status TEXT NOT NULL,
create_time TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS eval_dimensions (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
payload TEXT NOT NULL,
is_active INTEGER NOT NULL DEFAULT 1,
is_default INTEGER NOT NULL DEFAULT 0,
create_time TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS compare_tasks (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
payload TEXT NOT NULL,
status TEXT NOT NULL,
create_time TEXT NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_fine_tune_status ON fine_tune_tasks(status);
CREATE INDEX IF NOT EXISTS idx_fine_tune_compute_job ON fine_tune_tasks(compute_job_id);
CREATE INDEX IF NOT EXISTS idx_fine_tune_compute_node_status ON fine_tune_tasks(compute_node_id, status);
CREATE INDEX IF NOT EXISTS idx_model_lineage_child ON model_lineage(child_resource_type, child_resource_id);
CREATE INDEX IF NOT EXISTS idx_model_lineage_parent ON model_lineage(parent_resource_type, parent_resource_id);
CREATE INDEX IF NOT EXISTS idx_model_artifacts_model ON model_artifacts(model_kind, model_id, artifact_type);
CREATE INDEX IF NOT EXISTS idx_model_export_jobs_model ON model_export_jobs(trained_model_id, create_time DESC);
CREATE INDEX IF NOT EXISTS idx_model_export_jobs_compute ON model_export_jobs(compute_job_id);
CREATE INDEX IF NOT EXISTS idx_fine_tune_metrics_task_step ON fine_tune_metrics(task_id, step);
CREATE UNIQUE INDEX IF NOT EXISTS uq_fine_tune_metrics_task_step_epoch ON fine_tune_metrics(task_id, step, epoch);
CREATE INDEX IF NOT EXISTS idx_fine_tune_checkpoints_task_step ON fine_tune_checkpoints(task_id, step);
CREATE UNIQUE INDEX IF NOT EXISTS uq_fine_tune_checkpoints_task_path ON fine_tune_checkpoints(task_id, path);
CREATE INDEX IF NOT EXISTS idx_compute_jobs_task ON compute_jobs(task_id);
CREATE INDEX IF NOT EXISTS idx_compute_jobs_node_status ON compute_jobs(node_id, status);
CREATE INDEX IF NOT EXISTS idx_gpu_allocations_node_status ON gpu_allocations(node_id, status);
CREATE UNIQUE INDEX IF NOT EXISTS uq_gpu_allocations_active ON gpu_allocations(node_id, gpu_index) WHERE status IN ('allocated','running');
CREATE INDEX IF NOT EXISTS idx_scheduler_locks_expires ON scheduler_locks(expires_at);
CREATE INDEX IF NOT EXISTS idx_dataset_files_dataset ON dataset_files(dataset_id);
CREATE INDEX IF NOT EXISTS idx_gpus_node ON gpus(node_id);
CREATE UNIQUE INDEX IF NOT EXISTS uq_gpus_node_index ON gpus(node_id, gpu_index);
CREATE INDEX IF NOT EXISTS idx_replicas_resource ON resource_replicas(resource_type, resource_id);
CREATE UNIQUE INDEX IF NOT EXISTS uq_replicas_node_resource ON resource_replicas(node_id, resource_type, resource_id);
CREATE INDEX IF NOT EXISTS idx_sync_jobs_node_status ON resource_sync_jobs(target_node_id, status);
CREATE INDEX IF NOT EXISTS idx_eval_tasks_status ON eval_tasks(status);
CREATE INDEX IF NOT EXISTS idx_eval_dimensions_active ON eval_dimensions(is_active);
CREATE INDEX IF NOT EXISTS idx_compare_tasks_status ON compare_tasks(status);
-- ---- 项目 / 租户 ----
CREATE TABLE IF NOT EXISTS projects (
id TEXT PRIMARY KEY,
tenant_id TEXT NOT NULL DEFAULT 'default',
name TEXT NOT NULL,
code TEXT NOT NULL,
description TEXT,
quota TEXT,
status TEXT NOT NULL DEFAULT 'active',
create_time TEXT NOT NULL,
create_by TEXT,
updated_at TEXT
);
CREATE TABLE IF NOT EXISTS project_members (
project_id TEXT NOT NULL REFERENCES projects(id) ON DELETE CASCADE,
user_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
role TEXT NOT NULL DEFAULT 'member',
create_time TEXT NOT NULL,
PRIMARY KEY (project_id, user_id)
);
CREATE TABLE IF NOT EXISTS roles (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
permissions TEXT NOT NULL DEFAULT '[]',
create_time TEXT
);
CREATE TABLE IF NOT EXISTS sessions (
id TEXT PRIMARY KEY,
user_id TEXT NOT NULL,
issued_at TEXT NOT NULL,
expires_at TEXT NOT NULL,
ip TEXT
);
CREATE TABLE IF NOT EXISTS acls (
id TEXT PRIMARY KEY,
resource_type TEXT NOT NULL,
resource_id TEXT NOT NULL,
principal_type TEXT NOT NULL,
principal_id TEXT NOT NULL,
permission TEXT NOT NULL,
create_time TEXT
);
-- ============================================================================
-- 二、治理表（来源：002_governance.sql）
-- ============================================================================
CREATE TABLE IF NOT EXISTS tenants (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
code TEXT,
status TEXT DEFAULT 'active',
owner_user_id TEXT,
quota TEXT,
retention_policy_id TEXT,
create_time TEXT
);
CREATE TABLE IF NOT EXISTS approval_templates (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
steps TEXT,
create_time TEXT
);
CREATE TABLE IF NOT EXISTS approval_instances (
id TEXT PRIMARY KEY,
template_id TEXT,
resource_type TEXT,
resource_id TEXT,
applicant_id TEXT,
status TEXT DEFAULT 'pending',
current_step INTEGER DEFAULT 0,
create_time TEXT
);
CREATE TABLE IF NOT EXISTS approval_steps (
id TEXT PRIMARY KEY,
instance_id TEXT,
step_index INTEGER,
approver_id TEXT,
status TEXT DEFAULT 'pending',
comment TEXT,
time TEXT
);
CREATE TABLE IF NOT EXISTS audit_logs (
id TEXT PRIMARY KEY,
tenant_id TEXT,
project_id TEXT,
actor_id TEXT,
action TEXT,
target_type TEXT,
target_id TEXT,
detail TEXT,
client_ip TEXT,
time TEXT
);
CREATE INDEX IF NOT EXISTS idx_audit_tenant ON audit_logs(tenant_id);
CREATE INDEX IF NOT EXISTS idx_audit_project ON audit_logs(project_id);
CREATE INDEX IF NOT EXISTS idx_audit_action ON audit_logs(action);
CREATE INDEX IF NOT EXISTS idx_audit_time ON audit_logs(time);
CREATE TABLE IF NOT EXISTS retention_policies (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
scope TEXT,
rule TEXT,
status TEXT DEFAULT 'active',
create_time TEXT,
create_by TEXT,
updated_at TEXT
);
-- ============================================================================
-- 三、租户配额扩展（来源：003_tenant_quota.sql）
-- ============================================================================
ALTER TABLE tenants ADD COLUMN IF NOT EXISTS gpu_quota TEXT;
ALTER TABLE tenants ADD COLUMN IF NOT EXISTS storage_quota TEXT;
-- ============================================================================
-- 四、模型路径治理（来源：003_model_path_governance.sql）
-- models.can_train 已在建表语句中声明；以下为兼容旧库的幂等语句。
-- ============================================================================
ALTER TABLE models ADD COLUMN IF NOT EXISTS can_train INTEGER NOT NULL DEFAULT 0;
ALTER TABLE trained_models ADD COLUMN IF NOT EXISTS artifact_dir TEXT;
-- 按规则推定已有模型的 can_train（新库为空表，此语句为 no-op）
UPDATE models
SET can_train = CASE
WHEN path IS NOT NULL AND path != '' AND model_source IS NOT NULL AND model_source != 'api' THEN 1
ELSE 0
END;
-- ============================================================================
-- 五、数据处理（来源：002_data_process.sql，去掉其外层 BEGIN/COMMIT）
-- 数据集扩展列
-- ============================================================================
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS source_task_id TEXT;
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS size_bytes BIGINT NOT NULL DEFAULT 0;
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS record_count BIGINT NOT NULL DEFAULT 0;
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS metadata TEXT NOT NULL DEFAULT '{}';
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS tenant_id TEXT;
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS project_id TEXT;
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS owner_id TEXT;
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS created_by TEXT;
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS created_at TIMESTAMPTZ NOT NULL DEFAULT now();
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS updated_at TIMESTAMPTZ NOT NULL DEFAULT now();
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS deleted_at TIMESTAMPTZ;
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS storage_object_id TEXT;
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS current_version_id TEXT;
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS size_bytes BIGINT NOT NULL DEFAULT 0;
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS record_count BIGINT NOT NULL DEFAULT 0;
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS file_format VARCHAR(40);
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS checksum_sha256 CHAR(64);
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS version_no INTEGER NOT NULL DEFAULT 1;
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS source_task_id TEXT;
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS tenant_id TEXT;
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS project_id TEXT;
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS created_by TEXT;
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS metadata TEXT NOT NULL DEFAULT '{}';
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS created_at TIMESTAMPTZ NOT NULL DEFAULT now();
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS updated_at TIMESTAMPTZ NOT NULL DEFAULT now();
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS deleted_at TIMESTAMPTZ;
-- ---- 数据处理任务 / 源文件 / 预览 / 结果 ----
CREATE TABLE IF NOT EXISTS data_process_tasks (
id TEXT PRIMARY KEY,
name VARCHAR(150) NOT NULL,
description TEXT,
status VARCHAR(20) NOT NULL DEFAULT 'pending'
CHECK (status IN ('pending', 'running', 'completed', 'failed', 'stopped')),
process_type VARCHAR(20) NOT NULL
CHECK (process_type IN ('structured', 'unstructured', 'external')),
source_dataset_id TEXT REFERENCES datasets(id) ON DELETE SET NULL,
output_dataset_id TEXT REFERENCES datasets(id) ON DELETE SET NULL,
config TEXT NOT NULL DEFAULT '{}',
progress NUMERIC(5,2) NOT NULL DEFAULT 0 CHECK (progress >= 0 AND progress <= 100),
input_count BIGINT NOT NULL DEFAULT 0 CHECK (input_count >= 0),
output_count BIGINT NOT NULL DEFAULT 0 CHECK (output_count >= 0),
filtered_count BIGINT NOT NULL DEFAULT 0 CHECK (filtered_count >= 0),
duplicate_count BIGINT NOT NULL DEFAULT 0 CHECK (duplicate_count >= 0),
error_count BIGINT NOT NULL DEFAULT 0 CHECK (error_count >= 0),
failure_reason TEXT,
generation_run_id TEXT,
results_confirmed BOOLEAN NOT NULL DEFAULT TRUE,
workflow_step VARCHAR(20) NOT NULL DEFAULT 'create'
CHECK (workflow_step IN ('create', 'model', 'upload', 'preview', 'generate', 'results')),
preview_status VARCHAR(20) NOT NULL DEFAULT 'idle'
CHECK (preview_status IN ('idle', 'queued', 'running', 'completed', 'failed', 'cancelled')),
preview_progress NUMERIC(5,2) NOT NULL DEFAULT 0
CHECK (preview_progress >= 0 AND preview_progress <= 100),
preview_run_id TEXT,
preview_failure_reason TEXT,
preview_total_files INTEGER NOT NULL DEFAULT 0 CHECK (preview_total_files >= 0),
preview_completed_files INTEGER NOT NULL DEFAULT 0 CHECK (preview_completed_files >= 0),
tenant_id TEXT,
project_id TEXT,
owner_id TEXT,
approval_status VARCHAR(30) NOT NULL DEFAULT 'not_required',
created_by TEXT,
updated_by TEXT,
deleted_by TEXT,
started_at TIMESTAMPTZ,
completed_at TIMESTAMPTZ,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
deleted_at TIMESTAMPTZ
);
ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS generation_run_id TEXT;
ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS results_confirmed BOOLEAN NOT NULL DEFAULT TRUE;
ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS workflow_step VARCHAR(20);
ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_status VARCHAR(20);
ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_progress NUMERIC(5,2);
ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_run_id TEXT;
ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_failure_reason TEXT;
ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_total_files INTEGER;
ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_completed_files INTEGER;
ALTER TABLE data_process_tasks ALTER COLUMN workflow_step SET DEFAULT 'create';
ALTER TABLE data_process_tasks ALTER COLUMN workflow_step SET NOT NULL;
ALTER TABLE data_process_tasks ALTER COLUMN preview_status SET DEFAULT 'idle';
ALTER TABLE data_process_tasks ALTER COLUMN preview_status SET NOT NULL;
ALTER TABLE data_process_tasks ALTER COLUMN preview_progress SET DEFAULT 0;
ALTER TABLE data_process_tasks ALTER COLUMN preview_progress SET NOT NULL;
ALTER TABLE data_process_tasks ALTER COLUMN preview_total_files SET DEFAULT 0;
ALTER TABLE data_process_tasks ALTER COLUMN preview_total_files SET NOT NULL;
ALTER TABLE data_process_tasks ALTER COLUMN preview_completed_files SET DEFAULT 0;
ALTER TABLE data_process_tasks ALTER COLUMN preview_completed_files SET NOT NULL;
CREATE UNIQUE INDEX IF NOT EXISTS uq_data_process_tasks_name_alive
ON data_process_tasks(name) WHERE deleted_at IS NULL;
CREATE INDEX IF NOT EXISTS idx_data_process_tasks_scope_status
ON data_process_tasks(tenant_id, project_id, status, created_at DESC)
WHERE deleted_at IS NULL;
CREATE INDEX IF NOT EXISTS idx_data_process_tasks_creator_created
ON data_process_tasks(created_by, created_at DESC) WHERE deleted_at IS NULL;
CREATE TABLE IF NOT EXISTS data_process_source_files (
id TEXT PRIMARY KEY,
task_id TEXT NOT NULL REFERENCES data_process_tasks(id) ON DELETE CASCADE,
storage_object_id TEXT,
name TEXT NOT NULL,
size_bytes BIGINT NOT NULL DEFAULT 0 CHECK (size_bytes >= 0),
record_count BIGINT NOT NULL DEFAULT 0 CHECK (record_count >= 0),
file_format VARCHAR(40),
checksum_sha256 CHAR(64) NOT NULL,
version_no INTEGER NOT NULL DEFAULT 1 CHECK (version_no > 0),
content TEXT NOT NULL,
content_preview TEXT,
metadata TEXT NOT NULL DEFAULT '{}',
tenant_id TEXT,
project_id TEXT,
created_by TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
deleted_at TIMESTAMPTZ
);
CREATE INDEX IF NOT EXISTS idx_data_process_source_files_task
ON data_process_source_files(task_id, created_at) WHERE deleted_at IS NULL;
CREATE UNIQUE INDEX IF NOT EXISTS uq_data_process_source_checksum_alive
ON data_process_source_files(task_id, checksum_sha256) WHERE deleted_at IS NULL;
CREATE TABLE IF NOT EXISTS data_process_preview_items (
id TEXT PRIMARY KEY,
task_id TEXT NOT NULL REFERENCES data_process_tasks(id) ON DELETE CASCADE,
source_file_id TEXT REFERENCES data_process_source_files(id) ON DELETE CASCADE,
original_content TEXT NOT NULL DEFAULT '',
edited_content TEXT NOT NULL DEFAULT '',
source_start INTEGER CHECK (source_start IS NULL OR source_start >= 0),
source_end INTEGER CHECK (source_end IS NULL OR source_end >= 0),
source_start_line INTEGER CHECK (source_start_line IS NULL OR source_start_line > 0),
source_end_line INTEGER CHECK (source_end_line IS NULL OR source_end_line > 0),
token_count INTEGER NOT NULL DEFAULT 0 CHECK (token_count >= 0),
status VARCHAR(20) NOT NULL DEFAULT 'original'
CHECK (status IN ('original', 'modified', 'manual', 'invalid')),
quality_score TEXT NOT NULL DEFAULT '{}',
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CHECK (source_start IS NULL OR source_end IS NULL OR source_end >= source_start),
CHECK (source_start_line IS NULL OR source_end_line IS NULL OR source_end_line >= source_start_line)
);
CREATE INDEX IF NOT EXISTS idx_data_process_preview_task_file
ON data_process_preview_items(task_id, source_file_id, created_at);
CREATE TABLE IF NOT EXISTS data_process_results (
id TEXT PRIMARY KEY,
task_id TEXT NOT NULL REFERENCES data_process_tasks(id) ON DELETE CASCADE,
preview_item_id TEXT REFERENCES data_process_preview_items(id) ON DELETE SET NULL,
instruction TEXT NOT NULL,
input TEXT NOT NULL DEFAULT '',
output TEXT NOT NULL,
original_instruction TEXT,
original_input TEXT,
original_output TEXT,
status VARCHAR(20) NOT NULL DEFAULT 'valid'
CHECK (status IN ('valid', 'modified', 'invalid')),
error TEXT,
split VARCHAR(20) CHECK (split IS NULL OR split IN ('train', 'validation', 'test')),
quality_score TEXT NOT NULL DEFAULT '{}',
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS idx_data_process_results_task_status
ON data_process_results(task_id, status, id);
CREATE INDEX IF NOT EXISTS idx_data_process_results_task_split
ON data_process_results(task_id, split);
-- ---- 数据集版本 / 记录 ----
CREATE TABLE IF NOT EXISTS dataset_file_versions (
id TEXT PRIMARY KEY,
dataset_file_id TEXT NOT NULL REFERENCES dataset_files(id) ON DELETE CASCADE,
version_no INTEGER NOT NULL CHECK (version_no > 0),
storage_object_id TEXT NOT NULL,
content_preview TEXT,
description TEXT,
base_version_id TEXT REFERENCES dataset_file_versions(id) ON DELETE SET NULL,
size_bytes BIGINT NOT NULL DEFAULT 0 CHECK (size_bytes >= 0),
record_count BIGINT NOT NULL DEFAULT 0 CHECK (record_count >= 0),
checksum_sha256 CHAR(64) NOT NULL,
source_task_id TEXT REFERENCES data_process_tasks(id) ON DELETE SET NULL,
metadata TEXT NOT NULL DEFAULT '{}',
created_by TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
ALTER TABLE dataset_file_versions ADD COLUMN IF NOT EXISTS source_task_id TEXT;
ALTER TABLE dataset_file_versions ADD COLUMN IF NOT EXISTS metadata TEXT NOT NULL DEFAULT '{}';
CREATE UNIQUE INDEX IF NOT EXISTS uq_dataset_file_versions_no_002
ON dataset_file_versions(dataset_file_id, version_no);
CREATE INDEX IF NOT EXISTS idx_dataset_file_versions_source_task_002
ON dataset_file_versions(source_task_id) WHERE source_task_id IS NOT NULL;
CREATE TABLE IF NOT EXISTS dataset_records (
id TEXT PRIMARY KEY,
dataset_id TEXT NOT NULL REFERENCES datasets(id) ON DELETE CASCADE,
dataset_file_id TEXT REFERENCES dataset_files(id) ON DELETE CASCADE,
version_id TEXT REFERENCES dataset_file_versions(id) ON DELETE CASCADE,
line_no INTEGER,
split VARCHAR(20) CHECK (split IS NULL OR split IN ('train', 'validation', 'test')),
instruction TEXT,
input TEXT,
output TEXT,
raw TEXT NOT NULL DEFAULT '{}',
status VARCHAR(20) NOT NULL DEFAULT 'valid'
CHECK (status IN ('valid', 'modified', 'invalid')),
source_task_id TEXT REFERENCES data_process_tasks(id) ON DELETE SET NULL,
source_result_id TEXT REFERENCES data_process_results(id) ON DELETE SET NULL,
preview_item_id TEXT REFERENCES data_process_preview_items(id) ON DELETE SET NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
ALTER TABLE dataset_records ADD COLUMN IF NOT EXISTS source_task_id TEXT;
ALTER TABLE dataset_records ADD COLUMN IF NOT EXISTS source_result_id TEXT;
ALTER TABLE dataset_records ADD COLUMN IF NOT EXISTS preview_item_id TEXT;
CREATE INDEX IF NOT EXISTS idx_dataset_records_dataset_002
ON dataset_records(dataset_id, id);
CREATE INDEX IF NOT EXISTS idx_dataset_records_source_task_002
ON dataset_records(source_task_id, source_result_id);
CREATE INDEX IF NOT EXISTS idx_datasets_source_task_002
ON datasets(source_task_id) WHERE source_task_id IS NOT NULL;
CREATE INDEX IF NOT EXISTS idx_dataset_files_source_task_002
ON dataset_files(source_task_id) WHERE source_task_id IS NOT NULL;
-- ============================================================================
-- 六、数据转换任务（data_convert_tasks）
-- 运行时 router（app/modules/data_convert/router.py）引用但原脚本缺失，本文件补齐。
-- ============================================================================
CREATE TABLE IF NOT EXISTS data_convert_tasks (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
description TEXT,
output_filename TEXT DEFAULT 'converted-data.jsonl',
status TEXT NOT NULL DEFAULT 'pending',
input_count INTEGER NOT NULL DEFAULT 0,
output_count INTEGER NOT NULL DEFAULT 0,
error_message TEXT,
create_time TEXT NOT NULL DEFAULT (to_char(now(), 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"')),
update_time TEXT NOT NULL DEFAULT (to_char(now(), 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"')),
deleted_at TIMESTAMPTZ
);
CREATE INDEX IF NOT EXISTS idx_data_convert_tasks_status ON data_convert_tasks(status);
CREATE INDEX IF NOT EXISTS idx_data_convert_tasks_create_time ON data_convert_tasks(create_time DESC);
-- ============================================================================
-- 七、种子数据：初始管理员 / 操作员
-- 应用首次启动（ensure_seed_data）也会自动创建；此处提供以便脱离应用直接初始化。
-- 密码：admin / admin123，operator / operator123（上线前请改密）。
-- ============================================================================
INSERT INTO users
(id, username, password_hash, display_name, role, status, permissions, create_time, protected)
VALUES
(
'u_admin', 'admin', 'pbkdf2_sha256$390000$ygft_init_salt_admin$2b6f31f22968c4f5a30bcf0acf066b7a0f58d4773d15c5ab898ba715ea87b5bd',
'Platform Admin', 'admin', 'active',
'["dashboard","fine-tune","model-eval","model-inference","model-manage","dataset","data-process","data-convert","compute","hardware","logs","user-settings"]',
to_char(now(), 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"'), 1
),
(
'u_operator', 'operator', 'pbkdf2_sha256$390000$ygft_init_salt_op$525bf35d02ed26f37952cbd6862b0ae358b9d1a7fa0cbbf0217aa2b5dd544125',
'Platform Operator', 'operator', 'active',
'["dashboard","fine-tune","model-eval","model-inference","model-manage","dataset","data-process","data-convert","compute","hardware","logs"]',
to_char(now(), 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"'), 0
)
ON CONFLICT (username) DO NOTHING;
COMMIT;

208
backend/app/db/sql/001_platform_runtime.sql
View File

@@ -33,49 +33,7 @@ CREATE TABLE IF NOT EXISTS trained_models (
create_time TEXT NOT NULL,
merged INTEGER NOT NULL DEFAULT 0,
merging INTEGER NOT NULL DEFAULT 0,
merged_path TEXT,
artifact_dir TEXT,
compute_node_id TEXT,
compute_node_name TEXT
);
CREATE TABLE IF NOT EXISTS model_lineage (
id TEXT PRIMARY KEY,
child_resource_type TEXT NOT NULL,
child_resource_id TEXT NOT NULL,
parent_resource_type TEXT NOT NULL,
parent_resource_id TEXT NOT NULL,
relation_type TEXT NOT NULL,
compute_job_id TEXT,
payload TEXT NOT NULL,
create_time TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS model_artifacts (
id TEXT PRIMARY KEY,
model_id TEXT NOT NULL,
model_kind TEXT NOT NULL,
artifact_type TEXT NOT NULL,
path TEXT NOT NULL,
size_bytes BIGINT NOT NULL DEFAULT 0,
checksum_sha256 TEXT,
metadata TEXT NOT NULL,
compute_job_id TEXT,
create_time TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS model_export_jobs (
id TEXT PRIMARY KEY,
trained_model_id TEXT,
compute_job_id TEXT NOT NULL,
node_id TEXT,
export_type TEXT NOT NULL,
quantization_bit INTEGER NOT NULL DEFAULT 0,
status TEXT NOT NULL,
output_dir TEXT,
payload TEXT NOT NULL,
create_time TEXT NOT NULL,
completed_at TEXT
merged_path TEXT
);
CREATE TABLE IF NOT EXISTS datasets (
@@ -118,9 +76,6 @@ CREATE TABLE IF NOT EXISTS compute_nodes (
data_root TEXT NOT NULL,
model_root TEXT NOT NULL,
log_root TEXT NOT NULL,
api_version TEXT NOT NULL DEFAULT 'v1',
capabilities TEXT NOT NULL DEFAULT '[]',
description TEXT,
last_health_check_at TEXT,
health_detail TEXT NOT NULL
);
@@ -133,8 +88,7 @@ CREATE TABLE IF NOT EXISTS gpus (
name TEXT NOT NULL,
memory_total_gb DOUBLE PRECISION NOT NULL,
power_limit_w DOUBLE PRECISION NOT NULL,
base_temperature INTEGER NOT NULL,
last_seen_at TEXT
base_temperature INTEGER NOT NULL
);
CREATE TABLE IF NOT EXISTS fine_tune_tasks (
@@ -149,64 +103,7 @@ CREATE TABLE IF NOT EXISTS fine_tune_tasks (
completed_at TEXT,
compute_node_id TEXT REFERENCES compute_nodes(id) ON DELETE SET NULL,
gpus TEXT NOT NULL,
sync_job_id TEXT,
compute_job_id TEXT
);
CREATE TABLE IF NOT EXISTS fine_tune_metrics (
id TEXT PRIMARY KEY,
task_id TEXT NOT NULL REFERENCES fine_tune_tasks(id) ON DELETE CASCADE,
step INTEGER NOT NULL,
epoch DOUBLE PRECISION,
loss DOUBLE PRECISION,
grad_norm DOUBLE PRECISION,
learning_rate DOUBLE PRECISION,
raw TEXT NOT NULL,
create_time TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS fine_tune_checkpoints (
id TEXT PRIMARY KEY,
task_id TEXT NOT NULL REFERENCES fine_tune_tasks(id) ON DELETE CASCADE,
step INTEGER NOT NULL,
name TEXT NOT NULL,
path TEXT NOT NULL,
size_bytes BIGINT NOT NULL DEFAULT 0,
create_time TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS compute_jobs (
id TEXT PRIMARY KEY,
task_id TEXT REFERENCES fine_tune_tasks(id) ON DELETE SET NULL,
node_id TEXT REFERENCES compute_nodes(id) ON DELETE SET NULL,
engine TEXT NOT NULL,
status TEXT NOT NULL,
command TEXT NOT NULL,
output_dir TEXT,
log_file TEXT,
payload TEXT NOT NULL,
create_time TEXT NOT NULL,
update_time TEXT NOT NULL,
completed_at TEXT
);
CREATE TABLE IF NOT EXISTS gpu_allocations (
id TEXT PRIMARY KEY,
task_id TEXT REFERENCES fine_tune_tasks(id) ON DELETE CASCADE,
compute_job_id TEXT,
node_id TEXT REFERENCES compute_nodes(id) ON DELETE CASCADE,
gpu_index INTEGER NOT NULL,
status TEXT NOT NULL,
create_time TEXT NOT NULL,
released_at TEXT
);
CREATE TABLE IF NOT EXISTS scheduler_locks (
lock_key TEXT PRIMARY KEY,
owner TEXT NOT NULL,
expires_at TEXT NOT NULL,
create_time TEXT NOT NULL,
update_time TEXT NOT NULL
sync_job_id TEXT
);
CREATE TABLE IF NOT EXISTS resource_replicas (
@@ -217,10 +114,6 @@ CREATE TABLE IF NOT EXISTS resource_replicas (
local_path TEXT NOT NULL,
status TEXT NOT NULL,
sync_status TEXT NOT NULL,
checksum_sha256 TEXT,
byte_size BIGINT NOT NULL DEFAULT 0,
last_checked_at TEXT,
last_error TEXT,
create_time TEXT NOT NULL
);
@@ -234,102 +127,7 @@ CREATE TABLE IF NOT EXISTS resource_sync_jobs (
completed_at TEXT
);
CREATE TABLE IF NOT EXISTS eval_tasks (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
payload TEXT NOT NULL,
status TEXT NOT NULL,
create_time TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS eval_dimensions (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
payload TEXT NOT NULL,
is_active INTEGER NOT NULL DEFAULT 1,
is_default INTEGER NOT NULL DEFAULT 0,
create_time TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS compare_tasks (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
payload TEXT NOT NULL,
status TEXT NOT NULL,
create_time TEXT NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_fine_tune_status ON fine_tune_tasks(status);
CREATE INDEX IF NOT EXISTS idx_fine_tune_compute_job ON fine_tune_tasks(compute_job_id);
CREATE INDEX IF NOT EXISTS idx_fine_tune_compute_node_status ON fine_tune_tasks(compute_node_id, status);
CREATE INDEX IF NOT EXISTS idx_model_lineage_child ON model_lineage(child_resource_type, child_resource_id);
CREATE INDEX IF NOT EXISTS idx_model_lineage_parent ON model_lineage(parent_resource_type, parent_resource_id);
CREATE INDEX IF NOT EXISTS idx_model_artifacts_model ON model_artifacts(model_kind, model_id, artifact_type);
CREATE INDEX IF NOT EXISTS idx_model_export_jobs_model ON model_export_jobs(trained_model_id, create_time DESC);
CREATE INDEX IF NOT EXISTS idx_model_export_jobs_compute ON model_export_jobs(compute_job_id);
CREATE INDEX IF NOT EXISTS idx_fine_tune_metrics_task_step ON fine_tune_metrics(task_id, step);
CREATE UNIQUE INDEX IF NOT EXISTS uq_fine_tune_metrics_task_step_epoch ON fine_tune_metrics(task_id, step, epoch);
CREATE INDEX IF NOT EXISTS idx_fine_tune_checkpoints_task_step ON fine_tune_checkpoints(task_id, step);
CREATE UNIQUE INDEX IF NOT EXISTS uq_fine_tune_checkpoints_task_path ON fine_tune_checkpoints(task_id, path);
CREATE INDEX IF NOT EXISTS idx_compute_jobs_task ON compute_jobs(task_id);
CREATE INDEX IF NOT EXISTS idx_compute_jobs_node_status ON compute_jobs(node_id, status);
CREATE INDEX IF NOT EXISTS idx_gpu_allocations_node_status ON gpu_allocations(node_id, status);
CREATE UNIQUE INDEX IF NOT EXISTS uq_gpu_allocations_active ON gpu_allocations(node_id, gpu_index) WHERE status IN ('allocated','running');
CREATE INDEX IF NOT EXISTS idx_scheduler_locks_expires ON scheduler_locks(expires_at);
CREATE INDEX IF NOT EXISTS idx_dataset_files_dataset ON dataset_files(dataset_id);
CREATE INDEX IF NOT EXISTS idx_gpus_node ON gpus(node_id);
CREATE UNIQUE INDEX IF NOT EXISTS uq_gpus_node_index ON gpus(node_id, gpu_index);
CREATE INDEX IF NOT EXISTS idx_replicas_resource ON resource_replicas(resource_type, resource_id);
CREATE UNIQUE INDEX IF NOT EXISTS uq_replicas_node_resource ON resource_replicas(node_id, resource_type, resource_id);
CREATE INDEX IF NOT EXISTS idx_sync_jobs_node_status ON resource_sync_jobs(target_node_id, status);
CREATE INDEX IF NOT EXISTS idx_eval_tasks_status ON eval_tasks(status);
CREATE INDEX IF NOT EXISTS idx_eval_dimensions_active ON eval_dimensions(is_active);
CREATE INDEX IF NOT EXISTS idx_compare_tasks_status ON compare_tasks(status);
-- ===================== Project / Tenant =====================
CREATE TABLE IF NOT EXISTS projects (
id TEXT PRIMARY KEY,
tenant_id TEXT NOT NULL DEFAULT 'default',
name TEXT NOT NULL,
code TEXT NOT NULL,
description TEXT,
quota TEXT,
status TEXT NOT NULL DEFAULT 'active',
create_time TEXT NOT NULL,
create_by TEXT,
updated_at TEXT
);
CREATE TABLE IF NOT EXISTS project_members (
project_id TEXT NOT NULL REFERENCES projects(id) ON DELETE CASCADE,
user_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
role TEXT NOT NULL DEFAULT 'member',
create_time TEXT NOT NULL,
PRIMARY KEY (project_id, user_id)
);
CREATE TABLE IF NOT EXISTS roles (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
permissions TEXT NOT NULL DEFAULT '[]',
create_time TEXT
);
CREATE TABLE IF NOT EXISTS sessions (
id TEXT PRIMARY KEY,
user_id TEXT NOT NULL,
issued_at TEXT NOT NULL,
expires_at TEXT NOT NULL,
ip TEXT
);
CREATE TABLE IF NOT EXISTS acls (
id TEXT PRIMARY KEY,
resource_type TEXT NOT NULL,
resource_id TEXT NOT NULL,
principal_type TEXT NOT NULL,
principal_id TEXT NOT NULL,
permission TEXT NOT NULL,
create_time TEXT
);

352
backend/app/db/sql/002_data_process.sql
View File

@@ -1,352 +0,0 @@
-- Data processing migration.
--
-- IMPORTANT: This file is intentionally NOT wired into application startup.
-- Apply it explicitly in a controlled deployment, or call
-- DataProcessStore.ensure_schema() from an administrative command.
BEGIN;
-- This migration targets the current runtime schema created by
-- 001_platform_runtime.sql. Refuse the UUID/JSONB target-design schema instead
-- of partially altering it with incompatible TEXT foreign keys.
DO $$
DECLARE
datasets_id_type TEXT;
BEGIN
SELECT format_type(a.atttypid, a.atttypmod)
INTO datasets_id_type
FROM pg_attribute a
JOIN pg_class c ON c.oid = a.attrelid
JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE n.nspname = current_schema()
AND c.relname = 'datasets'
AND a.attname = 'id'
AND a.attnum > 0
AND NOT a.attisdropped;
IF datasets_id_type IS NULL THEN
RAISE EXCEPTION '002_data_process.sql requires 001_platform_runtime.sql first';
END IF;
IF datasets_id_type <> 'text' THEN
RAISE EXCEPTION
'002_data_process.sql supports only the current TEXT runtime schema; found datasets.id type %',
datasets_id_type;
END IF;
END $$;
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS source_task_id TEXT;
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS size_bytes BIGINT NOT NULL DEFAULT 0;
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS record_count BIGINT NOT NULL DEFAULT 0;
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS metadata TEXT NOT NULL DEFAULT '{}';
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS tenant_id TEXT;
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS project_id TEXT;
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS owner_id TEXT;
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS created_by TEXT;
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS created_at TIMESTAMPTZ NOT NULL DEFAULT now();
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS updated_at TIMESTAMPTZ NOT NULL DEFAULT now();
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS deleted_at TIMESTAMPTZ;
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS storage_object_id TEXT;
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS current_version_id TEXT;
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS size_bytes BIGINT NOT NULL DEFAULT 0;
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS record_count BIGINT NOT NULL DEFAULT 0;
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS file_format VARCHAR(40);
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS checksum_sha256 CHAR(64);
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS version_no INTEGER NOT NULL DEFAULT 1;
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS source_task_id TEXT;
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS tenant_id TEXT;
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS project_id TEXT;
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS created_by TEXT;
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS metadata TEXT NOT NULL DEFAULT '{}';
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS created_at TIMESTAMPTZ NOT NULL DEFAULT now();
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS updated_at TIMESTAMPTZ NOT NULL DEFAULT now();
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS deleted_at TIMESTAMPTZ;
CREATE TABLE IF NOT EXISTS data_process_tasks (
id TEXT PRIMARY KEY,
name VARCHAR(150) NOT NULL,
description TEXT,
status VARCHAR(20) NOT NULL DEFAULT 'pending'
CHECK (status IN ('pending', 'running', 'completed', 'failed', 'stopped')),
process_type VARCHAR(20) NOT NULL
CHECK (process_type IN ('structured', 'unstructured', 'external')),
source_dataset_id TEXT REFERENCES datasets(id) ON DELETE SET NULL,
output_dataset_id TEXT REFERENCES datasets(id) ON DELETE SET NULL,
config TEXT NOT NULL DEFAULT '{}',
progress NUMERIC(5,2) NOT NULL DEFAULT 0 CHECK (progress >= 0 AND progress <= 100),
input_count BIGINT NOT NULL DEFAULT 0 CHECK (input_count >= 0),
output_count BIGINT NOT NULL DEFAULT 0 CHECK (output_count >= 0),
filtered_count BIGINT NOT NULL DEFAULT 0 CHECK (filtered_count >= 0),
duplicate_count BIGINT NOT NULL DEFAULT 0 CHECK (duplicate_count >= 0),
error_count BIGINT NOT NULL DEFAULT 0 CHECK (error_count >= 0),
failure_reason TEXT,
generation_run_id TEXT,
results_confirmed BOOLEAN NOT NULL DEFAULT TRUE,
workflow_step VARCHAR(20) NOT NULL DEFAULT 'create'
CHECK (workflow_step IN ('create', 'model', 'upload', 'preview', 'generate', 'results')),
preview_status VARCHAR(20) NOT NULL DEFAULT 'idle'
CHECK (preview_status IN ('idle', 'queued', 'running', 'completed', 'failed', 'cancelled')),
preview_progress NUMERIC(5,2) NOT NULL DEFAULT 0
CHECK (preview_progress >= 0 AND preview_progress <= 100),
preview_run_id TEXT,
preview_failure_reason TEXT,
preview_total_files INTEGER NOT NULL DEFAULT 0 CHECK (preview_total_files >= 0),
preview_completed_files INTEGER NOT NULL DEFAULT 0 CHECK (preview_completed_files >= 0),
tenant_id TEXT,
project_id TEXT,
owner_id TEXT,
approval_status VARCHAR(30) NOT NULL DEFAULT 'not_required',
created_by TEXT,
updated_by TEXT,
deleted_by TEXT,
started_at TIMESTAMPTZ,
completed_at TIMESTAMPTZ,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
deleted_at TIMESTAMPTZ
);
ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS generation_run_id TEXT;
-- 历史任务在引入六步确认流程前已经完成审核，默认保留为已确认；
-- 新任务由创建接口显式写入 FALSE，并在第六步确认后转为 TRUE。
ALTER TABLE data_process_tasks
ADD COLUMN IF NOT EXISTS results_confirmed BOOLEAN NOT NULL DEFAULT TRUE;
UPDATE data_process_tasks
SET results_confirmed=FALSE
WHERE status <> 'completed' AND results_confirmed=TRUE;
-- 先以可空列接入旧库，才能只回填历史行；随后再收紧默认值与约束。
ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS workflow_step VARCHAR(20);
ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_status VARCHAR(20);
ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_progress NUMERIC(5,2);
ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_run_id TEXT;
ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_failure_reason TEXT;
ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_total_files INTEGER;
ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_completed_files INTEGER;
CREATE TEMP TABLE data_process_workflow_backfill_ids ON COMMIT DROP AS
SELECT id FROM data_process_tasks WHERE workflow_step IS NULL;
UPDATE data_process_tasks task
SET workflow_step = CASE
WHEN task.status IN ('running', 'failed', 'stopped') THEN 'generate'
WHEN task.status = 'completed' AND task.results_confirmed=FALSE THEN 'generate'
WHEN task.status = 'completed' THEN 'results'
ELSE 'create'
END
WHERE task.workflow_step IS NULL;
UPDATE data_process_tasks
SET preview_status='idle', preview_progress=0,
preview_total_files=0, preview_completed_files=0
WHERE preview_status IS NULL OR preview_progress IS NULL
OR preview_total_files IS NULL OR preview_completed_files IS NULL;
ALTER TABLE data_process_tasks ALTER COLUMN workflow_step SET DEFAULT 'create';
ALTER TABLE data_process_tasks ALTER COLUMN workflow_step SET NOT NULL;
ALTER TABLE data_process_tasks ALTER COLUMN preview_status SET DEFAULT 'idle';
ALTER TABLE data_process_tasks ALTER COLUMN preview_status SET NOT NULL;
ALTER TABLE data_process_tasks ALTER COLUMN preview_progress SET DEFAULT 0;
ALTER TABLE data_process_tasks ALTER COLUMN preview_progress SET NOT NULL;
ALTER TABLE data_process_tasks ALTER COLUMN preview_total_files SET DEFAULT 0;
ALTER TABLE data_process_tasks ALTER COLUMN preview_total_files SET NOT NULL;
ALTER TABLE data_process_tasks ALTER COLUMN preview_completed_files SET DEFAULT 0;
ALTER TABLE data_process_tasks ALTER COLUMN preview_completed_files SET NOT NULL;
DO $$
BEGIN
IF NOT EXISTS (
SELECT 1 FROM pg_constraint
WHERE conrelid='data_process_tasks'::regclass
AND conname='ck_data_process_tasks_workflow_step'
) THEN
ALTER TABLE data_process_tasks ADD CONSTRAINT ck_data_process_tasks_workflow_step
CHECK (workflow_step IN ('create', 'model', 'upload', 'preview', 'generate', 'results'));
END IF;
IF NOT EXISTS (
SELECT 1 FROM pg_constraint
WHERE conrelid='data_process_tasks'::regclass
AND conname='ck_data_process_tasks_preview_status'
) THEN
ALTER TABLE data_process_tasks ADD CONSTRAINT ck_data_process_tasks_preview_status
CHECK (preview_status IN ('idle', 'queued', 'running', 'completed', 'failed', 'cancelled'));
END IF;
IF NOT EXISTS (
SELECT 1 FROM pg_constraint
WHERE conrelid='data_process_tasks'::regclass
AND conname='ck_data_process_tasks_preview_progress'
) THEN
ALTER TABLE data_process_tasks ADD CONSTRAINT ck_data_process_tasks_preview_progress
CHECK (preview_progress >= 0 AND preview_progress <= 100);
END IF;
IF NOT EXISTS (
SELECT 1 FROM pg_constraint
WHERE conrelid='data_process_tasks'::regclass
AND conname='ck_data_process_tasks_preview_file_counts'
) THEN
ALTER TABLE data_process_tasks ADD CONSTRAINT ck_data_process_tasks_preview_file_counts
CHECK (preview_total_files >= 0 AND preview_completed_files >= 0
AND preview_completed_files <= preview_total_files);
END IF;
END $$;
CREATE UNIQUE INDEX IF NOT EXISTS uq_data_process_tasks_name_alive
ON data_process_tasks(name) WHERE deleted_at IS NULL;
CREATE INDEX IF NOT EXISTS idx_data_process_tasks_scope_status
ON data_process_tasks(tenant_id, project_id, status, created_at DESC)
WHERE deleted_at IS NULL;
CREATE INDEX IF NOT EXISTS idx_data_process_tasks_creator_created
ON data_process_tasks(created_by, created_at DESC) WHERE deleted_at IS NULL;
CREATE TABLE IF NOT EXISTS data_process_source_files (
id TEXT PRIMARY KEY,
task_id TEXT NOT NULL REFERENCES data_process_tasks(id) ON DELETE CASCADE,
storage_object_id TEXT,
name TEXT NOT NULL,
size_bytes BIGINT NOT NULL DEFAULT 0 CHECK (size_bytes >= 0),
record_count BIGINT NOT NULL DEFAULT 0 CHECK (record_count >= 0),
file_format VARCHAR(40),
checksum_sha256 CHAR(64) NOT NULL,
version_no INTEGER NOT NULL DEFAULT 1 CHECK (version_no > 0),
content TEXT NOT NULL,
content_preview TEXT,
metadata TEXT NOT NULL DEFAULT '{}',
tenant_id TEXT,
project_id TEXT,
created_by TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
deleted_at TIMESTAMPTZ
);
CREATE INDEX IF NOT EXISTS idx_data_process_source_files_task
ON data_process_source_files(task_id, created_at) WHERE deleted_at IS NULL;
CREATE UNIQUE INDEX IF NOT EXISTS uq_data_process_source_checksum_alive
ON data_process_source_files(task_id, checksum_sha256) WHERE deleted_at IS NULL;
CREATE TABLE IF NOT EXISTS data_process_preview_items (
id TEXT PRIMARY KEY,
task_id TEXT NOT NULL REFERENCES data_process_tasks(id) ON DELETE CASCADE,
source_file_id TEXT REFERENCES data_process_source_files(id) ON DELETE CASCADE,
original_content TEXT NOT NULL DEFAULT '',
edited_content TEXT NOT NULL DEFAULT '',
source_start INTEGER CHECK (source_start IS NULL OR source_start >= 0),
source_end INTEGER CHECK (source_end IS NULL OR source_end >= 0),
source_start_line INTEGER CHECK (source_start_line IS NULL OR source_start_line > 0),
source_end_line INTEGER CHECK (source_end_line IS NULL OR source_end_line > 0),
token_count INTEGER NOT NULL DEFAULT 0 CHECK (token_count >= 0),
status VARCHAR(20) NOT NULL DEFAULT 'original'
CHECK (status IN ('original', 'modified', 'manual', 'invalid')),
quality_score TEXT NOT NULL DEFAULT '{}',
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CHECK (source_start IS NULL OR source_end IS NULL OR source_end >= source_start),
CHECK (source_start_line IS NULL OR source_end_line IS NULL OR source_end_line >= source_start_line)
);
CREATE INDEX IF NOT EXISTS idx_data_process_preview_task_file
ON data_process_preview_items(task_id, source_file_id, created_at);
-- 子表在新库中到这里才存在；只修复本次新增 workflow_step 前的历史任务。
UPDATE data_process_tasks task
SET workflow_step = CASE
WHEN EXISTS (
SELECT 1 FROM data_process_preview_items preview
WHERE preview.task_id=task.id
) THEN 'preview'
WHEN EXISTS (
SELECT 1 FROM data_process_source_files source_file
WHERE source_file.task_id=task.id AND source_file.deleted_at IS NULL
) THEN 'upload'
ELSE task.workflow_step
END
WHERE task.id IN (SELECT id FROM data_process_workflow_backfill_ids)
AND task.status='pending';
CREATE TABLE IF NOT EXISTS data_process_results (
id TEXT PRIMARY KEY,
task_id TEXT NOT NULL REFERENCES data_process_tasks(id) ON DELETE CASCADE,
preview_item_id TEXT REFERENCES data_process_preview_items(id) ON DELETE SET NULL,
instruction TEXT NOT NULL,
input TEXT NOT NULL DEFAULT '',
output TEXT NOT NULL,
chosen TEXT NOT NULL DEFAULT '',
rejected TEXT NOT NULL DEFAULT '',
original_instruction TEXT,
original_input TEXT,
original_output TEXT,
original_chosen TEXT,
original_rejected TEXT,
status VARCHAR(20) NOT NULL DEFAULT 'valid'
CHECK (status IN ('valid', 'modified', 'invalid')),
error TEXT,
split VARCHAR(20) CHECK (split IS NULL OR split IN ('train', 'validation', 'test')),
quality_score TEXT NOT NULL DEFAULT '{}',
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
ALTER TABLE data_process_results ADD COLUMN IF NOT EXISTS chosen TEXT NOT NULL DEFAULT '';
ALTER TABLE data_process_results ADD COLUMN IF NOT EXISTS rejected TEXT NOT NULL DEFAULT '';
ALTER TABLE data_process_results ADD COLUMN IF NOT EXISTS original_chosen TEXT;
ALTER TABLE data_process_results ADD COLUMN IF NOT EXISTS original_rejected TEXT;
CREATE INDEX IF NOT EXISTS idx_data_process_results_task_status
ON data_process_results(task_id, status, id);
CREATE INDEX IF NOT EXISTS idx_data_process_results_task_split
ON data_process_results(task_id, split);
CREATE TABLE IF NOT EXISTS dataset_file_versions (
id TEXT PRIMARY KEY,
dataset_file_id TEXT NOT NULL REFERENCES dataset_files(id) ON DELETE CASCADE,
version_no INTEGER NOT NULL CHECK (version_no > 0),
storage_object_id TEXT NOT NULL,
content_preview TEXT,
description TEXT,
base_version_id TEXT REFERENCES dataset_file_versions(id) ON DELETE SET NULL,
size_bytes BIGINT NOT NULL DEFAULT 0 CHECK (size_bytes >= 0),
record_count BIGINT NOT NULL DEFAULT 0 CHECK (record_count >= 0),
checksum_sha256 CHAR(64) NOT NULL,
source_task_id TEXT REFERENCES data_process_tasks(id) ON DELETE SET NULL,
metadata TEXT NOT NULL DEFAULT '{}',
created_by TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
ALTER TABLE dataset_file_versions ADD COLUMN IF NOT EXISTS source_task_id TEXT;
ALTER TABLE dataset_file_versions ADD COLUMN IF NOT EXISTS metadata TEXT NOT NULL DEFAULT '{}';
CREATE UNIQUE INDEX IF NOT EXISTS uq_dataset_file_versions_no_002
ON dataset_file_versions(dataset_file_id, version_no);
CREATE INDEX IF NOT EXISTS idx_dataset_file_versions_source_task_002
ON dataset_file_versions(source_task_id) WHERE source_task_id IS NOT NULL;
CREATE TABLE IF NOT EXISTS dataset_records (
id TEXT PRIMARY KEY,
dataset_id TEXT NOT NULL REFERENCES datasets(id) ON DELETE CASCADE,
dataset_file_id TEXT REFERENCES dataset_files(id) ON DELETE CASCADE,
version_id TEXT REFERENCES dataset_file_versions(id) ON DELETE CASCADE,
line_no INTEGER,
split VARCHAR(20) CHECK (split IS NULL OR split IN ('train', 'validation', 'test')),
instruction TEXT,
input TEXT,
output TEXT,
raw TEXT NOT NULL DEFAULT '{}',
status VARCHAR(20) NOT NULL DEFAULT 'valid'
CHECK (status IN ('valid', 'modified', 'invalid')),
source_task_id TEXT REFERENCES data_process_tasks(id) ON DELETE SET NULL,
source_result_id TEXT REFERENCES data_process_results(id) ON DELETE SET NULL,
preview_item_id TEXT REFERENCES data_process_preview_items(id) ON DELETE SET NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
ALTER TABLE dataset_records ADD COLUMN IF NOT EXISTS source_task_id TEXT;
ALTER TABLE dataset_records ADD COLUMN IF NOT EXISTS source_result_id TEXT;
ALTER TABLE dataset_records ADD COLUMN IF NOT EXISTS preview_item_id TEXT;
CREATE INDEX IF NOT EXISTS idx_dataset_records_dataset_002
ON dataset_records(dataset_id, id);
CREATE INDEX IF NOT EXISTS idx_dataset_records_source_task_002
ON dataset_records(source_task_id, source_result_id);
CREATE INDEX IF NOT EXISTS idx_datasets_source_task_002
ON datasets(source_task_id) WHERE source_task_id IS NOT NULL;
CREATE INDEX IF NOT EXISTS idx_dataset_files_source_task_002
ON dataset_files(source_task_id) WHERE source_task_id IS NOT NULL;
COMMIT;

68
backend/app/db/sql/002_governance.sql
View File

@@ -1,68 +0,0 @@
-- 平台治理：租户 / 审批 / 审计（字段以 platform_store 实际写入为准）
CREATE TABLE IF NOT EXISTS tenants (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
code TEXT,
status TEXT DEFAULT 'active',
owner_user_id TEXT,
quota TEXT,
retention_policy_id TEXT,
create_time TEXT
);
CREATE TABLE IF NOT EXISTS approval_templates (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
steps TEXT,
create_time TEXT
);
CREATE TABLE IF NOT EXISTS approval_instances (
id TEXT PRIMARY KEY,
template_id TEXT,
resource_type TEXT,
resource_id TEXT,
applicant_id TEXT,
status TEXT DEFAULT 'pending',
current_step INTEGER DEFAULT 0,
create_time TEXT
);
CREATE TABLE IF NOT EXISTS approval_steps (
id TEXT PRIMARY KEY,
instance_id TEXT,
step_index INTEGER,
approver_id TEXT,
status TEXT DEFAULT 'pending',
comment TEXT,
time TEXT
);
CREATE TABLE IF NOT EXISTS audit_logs (
id TEXT PRIMARY KEY,
tenant_id TEXT,
project_id TEXT,
actor_id TEXT,
action TEXT,
target_type TEXT,
target_id TEXT,
detail TEXT,
client_ip TEXT,
time TEXT
);
CREATE INDEX IF NOT EXISTS idx_audit_tenant ON audit_logs(tenant_id);
CREATE INDEX IF NOT EXISTS idx_audit_project ON audit_logs(project_id);
CREATE INDEX IF NOT EXISTS idx_audit_action ON audit_logs(action);
CREATE INDEX IF NOT EXISTS idx_audit_time ON audit_logs(time);
CREATE TABLE IF NOT EXISTS retention_policies (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
scope TEXT,
rule TEXT,
status TEXT DEFAULT 'active',
create_time TEXT,
create_by TEXT,
updated_at TEXT
);

18
backend/app/db/sql/003_model_path_governance.sql
View File

@@ -1,18 +0,0 @@
-- 003_model_path_governance
-- 模型路径治理：增加 can_train 标识，区分本地可训练模型与 API / 远程模型。
-- 训练预检阶段依赖该字段拦截不适合 LLaMA-Factory 本地训练的基座模型。
-- 1. models 表增加 can_train（默认 0，后设搬迁为 1 的规则如下）
ALTER TABLE models ADD COLUMN IF NOT EXISTS can_train INTEGER NOT NULL DEFAULT 0;
-- 2. 将已有模型按规则推定 can_train：
-- - path 非空 且 model_source != 'api' → 可训练 (1)
-- - 其余 → 不可训练 (0)
UPDATE models
SET can_train = CASE
WHEN path IS NOT NULL AND path != '' AND model_source IS NOT NULL AND model_source != 'api' THEN 1
ELSE 0
END;
-- 3. 给 trained_models 增加 artifact_dir（训练产物目录扫描结果目录）
ALTER TABLE trained_models ADD COLUMN IF NOT EXISTS artifact_dir TEXT;

3
backend/app/db/sql/003_tenant_quota.sql
View File

@@ -1,3 +0,0 @@
-- 租户配额与保留策略扩展（如后续治理表需补列，可在此追加）
ALTER TABLE tenants ADD COLUMN IF NOT EXISTS gpu_quota TEXT;
ALTER TABLE tenants ADD COLUMN IF NOT EXISTS storage_quota TEXT;

22
backend/app/db/sql/004_permissions.sql
View File

@@ -1,22 +0,0 @@
-- ============================================================
-- 权限体系扩展：GPU 分配表 + 资源所有权字段
-- ============================================================
-- GPU 分配表：管理员指定哪些用户可以使用哪些 GPU 卡
CREATE TABLE IF NOT EXISTS gpu_assignments (
id TEXT PRIMARY KEY,
node_id TEXT NOT NULL REFERENCES compute_nodes(id) ON DELETE CASCADE,
gpu_index INTEGER NOT NULL,
user_id TEXT NOT NULL,
assigned_by TEXT,
assigned_at TEXT NOT NULL,
UNIQUE (node_id, gpu_index, user_id)
);
CREATE INDEX IF NOT EXISTS idx_gpu_assignments_user ON gpu_assignments(user_id);
CREATE INDEX IF NOT EXISTS idx_gpu_assignments_gpu ON gpu_assignments(node_id, gpu_index);
-- 资源所有权字段：用户创建的数据集/模型/训练产物/评测任务
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS created_by TEXT;
ALTER TABLE models ADD COLUMN IF NOT EXISTS created_by TEXT;
ALTER TABLE trained_models ADD COLUMN IF NOT EXISTS created_by TEXT;
ALTER TABLE eval_tasks ADD COLUMN IF NOT EXISTS created_by TEXT;

21
backend/app/main.py
View File

@@ -1,20 +1,16 @@
import asyncio
from contextlib import suppress
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from app.api.v1.router import api_router
from app.core.config import docs_kwargs, get_settings
from app.core.config import get_settings
from app.core.logging import configure_logging, setup_request_logging
from app.workers.compute_poller import run_compute_poller
def create_app() -> FastAPI:
settings = get_settings()
configure_logging(settings)
app = FastAPI(title=settings.app_name, **docs_kwargs(settings.enable_docs))
app = FastAPI(title=settings.app_name)
app.add_middleware(
CORSMiddleware,
allow_origins=settings.cors_allow_origins,
@@ -24,19 +20,6 @@ def create_app() -> FastAPI:
)
setup_request_logging(app)
app.include_router(api_router, prefix=settings.route_prefix)
@app.on_event("startup")
async def start_workers() -> None:
app.state.compute_poller_task = asyncio.create_task(run_compute_poller())
@app.on_event("shutdown")
async def stop_workers() -> None:
task = getattr(app.state, "compute_poller_task", None)
if task:
task.cancel()
with suppress(asyncio.CancelledError):
await task
return app

91
backend/app/modules/approval/router.py
View File

@@ -1,91 +0,0 @@
from __future__ import annotations
from fastapi import APIRouter, Body
from typing import Any
from app.api.v1.endpoints.platform import ok, fail
from app.db.platform_store import get_platform_store
router = APIRouter(prefix="/approvals", tags=["approval"])
@router.get("/templates")
def list_templates() -> dict[str, Any]:
return ok(get_platform_store().approval_templates())
@router.post("/templates")
def create_template(payload: dict[str, Any] = Body(...)) -> dict[str, Any]:
if not payload.get("name"):
raise fail(400, "name 必填")
return ok(get_platform_store().create_approval_template(payload))
@router.get("/templates/{template_id}")
def get_template(template_id: str) -> dict[str, Any]:
try:
return ok(get_platform_store().approval_template(template_id))
except KeyError:
raise fail(404, "template not found")
@router.put("/templates/{template_id}")
def update_template(template_id: str, payload: dict[str, Any] = Body(...)) -> dict[str, Any]:
try:
return ok(get_platform_store().update_approval_template(template_id, payload))
except KeyError:
raise fail(404, "template not found")
@router.delete("/templates/{template_id}")
def delete_template(template_id: str) -> dict[str, Any]:
try:
return ok(get_platform_store().delete_approval_template(template_id))
except KeyError:
raise fail(404, "template not found")
@router.get("")
def list_instances(status: str | None = None) -> dict[str, Any]:
return ok(get_platform_store().approval_instances(status=status))
@router.post("")
def create_instance(payload: dict[str, Any] = Body(...)) -> dict[str, Any]:
for field in ("resource_type", "resource_id", "applicant_id"):
if not payload.get(field):
raise fail(400, f"{field} 必填")
try:
return ok(get_platform_store().create_approval_instance(payload))
except KeyError:
raise fail(404, "template not found")
@router.get("/{instance_id}")
def get_instance(instance_id: str) -> dict[str, Any]:
try:
return ok(get_platform_store().approval_instance(instance_id))
except KeyError:
raise fail(404, "instance not found")
@router.post("/{instance_id}/steps/{step_index}/decision")
def decide(
instance_id: str,
step_index: int,
payload: dict[str, Any] = Body(...),
) -> dict[str, Any]:
if not payload.get("approver_id"):
raise fail(400, "approver_id 必填")
try:
return ok(
get_platform_store().decide_approval_step(
instance_id,
step_index,
approver_id=payload["approver_id"],
approved=bool(payload.get("approved", False)),
comment=payload.get("comment"),
)
)
except (KeyError, ValueError) as e:
raise fail(400, str(e))

247
backend/app/modules/compute_gateway/client.py
View File

@@ -1,247 +0,0 @@
from __future__ import annotations
import time
from typing import Any
from urllib.parse import urljoin
import httpx
from app.core.config import get_settings
def _join_url(base_url: str, path: str) -> str:
return urljoin(base_url.rstrip("/") + "/", path.lstrip("/"))
def _unwrap_items(payload: Any) -> list[dict[str, Any]]:
if isinstance(payload, list):
return [item for item in payload if isinstance(item, dict)]
if isinstance(payload, dict):
data = payload.get("data")
if isinstance(data, dict) and isinstance(data.get("items"), list):
return [item for item in data["items"] if isinstance(item, dict)]
if isinstance(payload.get("items"), list):
return [item for item in payload["items"] if isinstance(item, dict)]
if isinstance(data, list):
return [item for item in data if isinstance(item, dict)]
return []
def _unwrap_dict(payload: Any) -> dict[str, Any]:
if isinstance(payload, dict) and isinstance(payload.get("data"), dict):
return payload["data"]
return payload if isinstance(payload, dict) else {}
# Inference calls are intentionally short-timeout:
# - load dispatch only confirms the compute node accepted the request
# (the actual model load now runs asynchronously on the node).
# - status/unload must never block the platform for long when a node is
# unreachable but still marked online.
INFERENCE_LOAD_TIMEOUT = httpx.Timeout(30, connect=10)
INFERENCE_STATUS_TIMEOUT = httpx.Timeout(30, connect=5)
INFERENCE_UNLOAD_TIMEOUT = httpx.Timeout(30, connect=5)
class ComputeNodeClient:
"""Application-side client for one compute node.
The client accepts both current YG Compute API responses and common
wrapper shapes such as `{code,message,data}` to make future engine/node
adapters less brittle.
"""
def __init__(self, api_base_url: str, token: str | None = None, timeout: float | None = None) -> None:
settings = get_settings()
self.api_base_url = api_base_url.rstrip("/")
self.token = token or settings.compute_service_token
self.timeout = timeout or settings.compute_request_timeout_seconds
self.route_prefix = settings.route_prefix.rstrip("/") or "/modelTF"
def headers(self) -> dict[str, str]:
if not self.token:
return {}
return {"X-Compute-Token": self.token}
async def test_connection(self) -> dict[str, Any]:
started = time.perf_counter()
health = await self.health()
gpus = await self.gpus()
return {
"success": True,
"latency_ms": int((time.perf_counter() - started) * 1000),
"health": health,
"gpus": gpus,
}
async def health(self) -> dict[str, Any]:
paths = [f"{self.route_prefix}/v1/compute/health", f"{self.route_prefix}/health", "/health"]
last_error = ""
async with httpx.AsyncClient(timeout=self.timeout, headers=self.headers()) as client:
for path in paths:
try:
response = await client.get(_join_url(self.api_base_url, path))
response.raise_for_status()
return _unwrap_dict(response.json())
except Exception as exc: # noqa: BLE001 - keep endpoint compatibility fallback broad
last_error = str(exc)
raise RuntimeError(last_error or "compute health check failed")
async def gpus(self) -> list[dict[str, Any]]:
paths = [
f"{self.route_prefix}/compute/resources/gpus",
f"{self.route_prefix}/v1/compute/resources/gpus",
"/compute/resources/gpus",
]
last_error = ""
async with httpx.AsyncClient(timeout=self.timeout, headers=self.headers()) as client:
for path in paths:
try:
response = await client.get(_join_url(self.api_base_url, path))
response.raise_for_status()
return _unwrap_items(response.json())
except Exception as exc: # noqa: BLE001
last_error = str(exc)
raise RuntimeError(last_error or "compute gpu discovery failed")
async def create_job(self, payload: dict[str, Any]) -> dict[str, Any]:
async with httpx.AsyncClient(timeout=self.timeout, headers=self.headers()) as client:
response = await client.post(_join_url(self.api_base_url, f"{self.route_prefix}/compute/jobs"), json=payload)
response.raise_for_status()
return _unwrap_dict(response.json())
async def preview_job(self, payload: dict[str, Any]) -> dict[str, Any]:
async with httpx.AsyncClient(timeout=self.timeout, headers=self.headers()) as client:
response = await client.post(
_join_url(self.api_base_url, f"{self.route_prefix}/compute/jobs/preview"),
json=payload,
)
response.raise_for_status()
return _unwrap_dict(response.json())
async def validate_job(self, payload: dict[str, Any]) -> dict[str, Any]:
async with httpx.AsyncClient(timeout=self.timeout, headers=self.headers()) as client:
response = await client.post(
_join_url(self.api_base_url, f"{self.route_prefix}/compute/jobs/validate"),
json=payload,
)
response.raise_for_status()
return _unwrap_dict(response.json())
async def check_paths(self, paths: list[dict[str, Any]]) -> dict[str, Any]:
async with httpx.AsyncClient(timeout=self.timeout, headers=self.headers()) as client:
response = await client.post(
_join_url(self.api_base_url, f"{self.route_prefix}/compute/files/check-paths"),
json={"paths": paths},
)
response.raise_for_status()
return _unwrap_dict(response.json())
async def list_files(
self,
root: str = "data",
relative_path: str = "",
directories_only: bool = False,
) -> dict[str, Any]:
async with httpx.AsyncClient(timeout=self.timeout, headers=self.headers()) as client:
response = await client.get(
_join_url(self.api_base_url, f"{self.route_prefix}/compute/files/list"),
params={"root": root, "relative_path": relative_path, "directories_only": directories_only},
)
response.raise_for_status()
return _unwrap_dict(response.json())
async def get_job(self, job_id: str) -> dict[str, Any]:
async with httpx.AsyncClient(timeout=self.timeout, headers=self.headers()) as client:
response = await client.get(_join_url(self.api_base_url, f"{self.route_prefix}/compute/jobs/{job_id}"))
response.raise_for_status()
return _unwrap_dict(response.json())
async def stop_job(self, job_id: str) -> dict[str, Any]:
async with httpx.AsyncClient(timeout=self.timeout, headers=self.headers()) as client:
response = await client.post(_join_url(self.api_base_url, f"{self.route_prefix}/compute/jobs/{job_id}/stop"))
response.raise_for_status()
return _unwrap_dict(response.json())
async def job_logs(
self,
job_id: str,
tail_lines: int | None = None,
offset: int | None = None,
limit: int | None = None,
) -> dict[str, Any]:
params = {
key: value
for key, value in {"tail_lines": tail_lines, "offset": offset, "limit": limit}.items()
if value is not None
}
async with httpx.AsyncClient(timeout=self.timeout, headers=self.headers()) as client:
response = await client.get(
_join_url(self.api_base_url, f"{self.route_prefix}/compute/jobs/{job_id}/logs"),
params=params,
)
response.raise_for_status()
return _unwrap_dict(response.json())
async def import_local_file(self, payload: dict[str, Any]) -> dict[str, Any]:
async with httpx.AsyncClient(timeout=self.timeout, headers=self.headers()) as client:
response = await client.post(
_join_url(self.api_base_url, f"{self.route_prefix}/compute/files/import-local"),
json=payload,
)
response.raise_for_status()
return _unwrap_dict(response.json())
async def _request(
self,
method: str,
path: str,
json_data: dict[str, Any] | None = None,
timeout: float | None = None,
) -> dict[str, Any]:
"""Generic request method for compute API endpoints."""
url = _join_url(self.api_base_url, f"{self.route_prefix}{path}")
async with httpx.AsyncClient(timeout=timeout or 300, headers=self.headers()) as client:
if method.upper() == "GET":
response = await client.get(url)
else:
response = await client.post(url, json=json_data)
response.raise_for_status()
return _unwrap_dict(response.json())
# ── Inference helpers (short timeouts — see module constants) ──────────
async def inference_load(self, payload: dict[str, Any]) -> dict[str, Any]:
"""Dispatch a model load. Returns as soon as the node accepts the
request; the node now loads asynchronously (status goes 'loading')."""
return await self._request("POST", "/inference/load", json_data=payload, timeout=INFERENCE_LOAD_TIMEOUT)
async def inference_status(self) -> dict[str, Any]:
return await self._request("GET", "/inference/status", timeout=INFERENCE_STATUS_TIMEOUT)
async def inference_unload(self) -> dict[str, Any]:
return await self._request("POST", "/inference/unload", json_data={}, timeout=INFERENCE_UNLOAD_TIMEOUT)
async def upload_file(
self,
filename: str,
content: bytes,
target_relative_path: str,
resource_type: str | None = None,
resource_id: str | None = None,
) -> dict[str, Any]:
data = {
"target_relative_path": target_relative_path,
"resource_type": resource_type or "",
"resource_id": resource_id or "",
}
files = {"file": (filename, content)}
timeout = httpx.Timeout(max(self.timeout, 60), connect=self.timeout)
async with httpx.AsyncClient(timeout=timeout, headers=self.headers()) as client:
response = await client.post(
_join_url(self.api_base_url, f"{self.route_prefix}/compute/files/upload"),
data=data,
files=files,
)
response.raise_for_status()
return _unwrap_dict(response.json())

191
backend/app/modules/compute_gateway/sync.py
View File

@@ -1,191 +0,0 @@
from __future__ import annotations
import json
import time
from typing import Any
from app.db.platform_store import get_platform_store
from app.modules.compute_gateway.client import ComputeNodeClient
# starting 状态允许的最大轮询次数（约 40 * 3s ≈ 2 分钟），超过即判定节点不可达
MAX_STARTING_ATTEMPTS = 40
def _node_for_task(task: dict[str, Any]) -> dict[str, Any] | None:
return next((node for node in get_platform_store().compute_nodes() if node["id"] == task.get("compute_node_id")), None)
def _parse_inference_load_status(task: dict[str, Any]) -> tuple[list[dict[str, Any]], dict[str, Any]]:
load_status = task.get("load_status") or {}
if isinstance(load_status, str):
try:
load_status = json.loads(load_status)
except (json.JSONDecodeError, TypeError):
load_status = {}
return load_status.get("loaded_models") or [], load_status
async def reconcile_inference_loads(store: Any) -> list[dict[str, Any]]:
"""推进处于 starting 状态的推理加载。
模型加载已改为异步派发：/model-compare/{id}/load 立即返回，这里在每次
轮询时查询对应计算节点的 /inference/status，把任务从 starting 推进到
ready/error。使用短超时，单节点不可达不会阻塞整轮轮询。
"""
reconciled: list[dict[str, Any]] = []
now = time.time()
for task in store.compare_tasks():
items, _ = _parse_inference_load_status(task)
if not any(item.get("status") == "starting" for item in items):
continue
# dirty 只要处理过任一 starting 项就置位：load_attempts / last_polled_at
# 必须落库，否则节点不可达时计数不会累积，封顶逻辑永远触发不了
dirty = False
for item in items:
if item.get("status") != "starting":
continue
# 节流：同一 item 每 3s 只查询一次
if now - float(item.get("last_polled_at") or 0) < 3:
continue
item["last_polled_at"] = now
item["load_attempts"] = int(item.get("load_attempts") or 0) + 1
dirty = True
node = next((n for n in store.compute_nodes() if n["id"] == item.get("node_id")), None)
if not node:
item["status"] = "error"
item["error"] = "compute node deleted"
store.mark_inference_unloaded(item.get("node_id") or "")
continue
if not node.get("enabled") or node.get("scheduler_status") != "online":
item["status"] = "error"
item["error"] = "compute node offline"
store.mark_inference_unloaded(node["id"])
continue
try:
status = await ComputeNodeClient(node["api_base_url"]).inference_status()
except Exception as exc: # noqa: BLE001 - node unreachable; keep retrying until cap
if int(item.get("load_attempts") or 0) >= MAX_STARTING_ATTEMPTS:
item["status"] = "error"
item["error"] = f"compute node unreachable: {exc}"
store.mark_inference_unloaded(node["id"])
continue
node_status = status.get("status")
if node_status == "ready":
item["status"] = "ready"
item.pop("error", None)
store.mark_inference_loaded(node["id"])
elif node_status == "error":
item["status"] = "error"
item["error"] = status.get("error") or "model load failed on compute node"
store.mark_inference_unloaded(node["id"])
elif node_status == "idle":
# 节点重启导致已加载模型丢失
item["status"] = "error"
item["error"] = "model disappeared from compute node (node may have restarted)"
store.mark_inference_unloaded(node["id"])
# node_status == "loading" -> 保持 starting，下轮再查
if dirty:
if any(i.get("status") in {"ready", "running"} for i in items):
new_status = "loaded"
elif any(i.get("status") == "starting" for i in items):
new_status = "starting" # 仍在加载中，保持 starting
else:
new_status = "failed"
store.update_compare_task(task["id"], {"status": new_status, "load_status": {"loaded_models": items}})
reconciled.append({"task_id": task["id"], "status": new_status})
return reconciled
async def fetch_eval_result_content(client: ComputeNodeClient, node: dict[str, Any], job: dict[str, Any]) -> dict[str, Any] | None:
output_dir = job.get("output_dir")
if not output_dir:
return None
full_path = f"{str(output_dir).rstrip('/')}/eval_results.json"
data_root = "/data/yg-ft/"
if full_path.startswith(data_root):
full_path = full_path[len(data_root):]
rel_path = full_path.lstrip("/")
import httpx
url = f"{node['api_base_url'].rstrip('/')}/modelTF/compute/files/read"
async with httpx.AsyncClient(timeout=30, headers=client.headers()) as http:
response = await http.get(url, params={"path": rel_path})
response.raise_for_status()
payload = response.json()
return payload if isinstance(payload, dict) else None
async def poll_compute_jobs_once() -> dict[str, Any]:
store = get_platform_store()
synced: list[dict[str, Any]] = []
failed: list[dict[str, str]] = []
for task in store.running_compute_tasks():
node = _node_for_task(task)
if not node:
failed.append({"task_id": task["id"], "error": "compute node not found"})
continue
try:
client = ComputeNodeClient(node["api_base_url"])
job = await client.get_job(task["compute_job_id"])
try:
logs = await client.job_logs(task["compute_job_id"], tail_lines=5000)
store.record_training_log_metrics(task["id"], str(logs.get("content") or ""))
except Exception:
pass
# P0-4: Force-fetch last log snippet when job reaches terminal state
if job.get("status") in {"failed", "stopped"}:
try:
last_logs = await client.job_logs(task["compute_job_id"], tail_lines=200)
job["log_snippet"] = str(last_logs.get("content") or "")[:8192]
except Exception:
pass
synced.append(store.apply_compute_job(task["id"], job))
except Exception as exc: # noqa: BLE001 - keep polling other jobs
failed.append({"task_id": task["id"], "error": str(exc)})
standalone_synced: list[dict[str, Any]] = []
for record in store.active_standalone_compute_jobs():
node = next((item for item in store.compute_nodes() if item["id"] == record.get("node_id")), None)
if not node:
failed.append({"job_id": record["id"], "error": "compute node not found"})
continue
try:
job = await ComputeNodeClient(node["api_base_url"]).get_job(record["id"])
standalone_synced.append(store.sync_model_merge_job(record["id"], job))
except Exception as exc: # noqa: BLE001 - keep polling other jobs
failed.append({"job_id": record["id"], "error": str(exc)})
# ── Eval job sync ────────────────────────────────────────────────
eval_synced = 0
for eval_task in store.running_eval_tasks():
node = next(
(item for item in store.compute_nodes() if item["id"] == eval_task.get("compute_node_id")),
None,
)
if not node:
failed.append({"eval_task_id": eval_task["id"], "error": "compute node not found"})
continue
try:
client = ComputeNodeClient(node["api_base_url"])
job = await client.get_job(eval_task["compute_job_id"])
result_content = None
# Try to read eval_results.json from the job output directory
if job.get("status") == "completed" and job.get("output_dir"):
try:
result_content = await fetch_eval_result_content(client, node, job)
except Exception:
pass
store.apply_eval_job_result(eval_task["id"], job, result_content)
# 评测 GPU 占用由 eval_tasks 状态派生，无需维护推理内存标记
eval_synced += 1
except Exception as exc: # noqa: BLE001
failed.append({"eval_task_id": eval_task["id"], "error": str(exc)})
# ── Inference load reconciliation ─────────────────────────────────────
try:
inference_reconciled = await reconcile_inference_loads(store)
except Exception as exc: # noqa: BLE001 - keep polling alive
failed.append({"inference_reconcile": str(exc)})
inference_reconciled = []
return {"synced": len(synced) + len(standalone_synced) + eval_synced, "failed": failed,
"items": synced, "standalone": standalone_synced, "eval_synced": eval_synced,
"inference_reconciled": inference_reconciled}

3
backend/app/modules/data_convert/__init__.py
View File

@@ -1,3 +0,0 @@
from .router import router
__all__ = ["router"]

356
backend/app/modules/data_convert/router.py
View File

@@ -1,356 +0,0 @@
from __future__ import annotations
import json
import os
from pathlib import Path
from typing import Any
from fastapi import APIRouter, Body, Depends, File, UploadFile
from fastapi.responses import FileResponse
from app.api.v1.endpoints.platform import ok, fail
from app.core.auth import get_current_user
from app.db.platform_store import get_platform_store, new_id
router = APIRouter(prefix="/data-convert", tags=["data-convert"])
# 存储根目录
STORAGE_ROOT = Path(__file__).resolve().parents[3] / "storage" / "data-convert"
def _safe_output_filename(value: Any) -> str:
"""输出文件名白名单校验：仅允许普通文件名，阻断 ``../``、``/``、``\\`` 等路径穿越。
转换结果始终写入 ``STORAGE_ROOT/<task_id>/output/<output_filename>``，
若文件名可被注入路径分隔符，将导致任意文件读写/删除。
"""
name = str(value or "converted-data.jsonl").strip()
if (
not name
or name in {".", ".."}
or name != Path(name).name
or "/" in name
or "\\" in name
or any(ord(character) < 32 or ord(character) == 127 for character in name)
):
raise fail(400, "output filename must be a plain file name")
return name
def _task_output_path(task: dict[str, Any]) -> Path:
"""返回经过白名单校验的转换输出文件路径（始终位于任务 output 目录内）。"""
return _output_dir(task["id"]) / _safe_output_filename(task.get("output_filename"))
def _task_dir(task_id: str) -> Path:
return STORAGE_ROOT / task_id
def _input_dir(task_id: str) -> Path:
return _task_dir(task_id) / "input"
def _output_dir(task_id: str) -> Path:
return _task_dir(task_id) / "output"
@router.get("")
def list_tasks(
page: int = 1,
page_size: int = 20,
current_user: dict = Depends(get_current_user),
) -> dict[str, Any]:
store = get_platform_store()
with store.connect() as conn:
rows = conn.execute(
"SELECT * FROM data_convert_tasks WHERE deleted_at IS NULL "
"ORDER BY create_time DESC LIMIT %s OFFSET %s",
(page_size, (page - 1) * page_size),
).fetchall()
total = conn.execute(
"SELECT COUNT(*) FROM data_convert_tasks WHERE deleted_at IS NULL"
).fetchone()[0]
return ok({"items": [dict(r) for r in rows], "total": total})
@router.post("")
def create_task(
payload: dict[str, Any] = Body(...),
current_user: dict = Depends(get_current_user),
) -> dict[str, Any]:
name = str(payload.get("name") or "").strip()
if not name:
raise fail(400, "name is required")
task_id = new_id("dct")
output_filename = _safe_output_filename(payload.get("output_filename"))
description = str(payload.get("description") or "").strip()
store = get_platform_store()
with store.connect() as conn:
conn.execute(
"INSERT INTO data_convert_tasks (id, name, description, output_filename) "
"VALUES (%s, %s, %s, %s)",
(task_id, name, description, output_filename),
)
# 创建目录
_input_dir(task_id).mkdir(parents=True, exist_ok=True)
_output_dir(task_id).mkdir(parents=True, exist_ok=True)
return ok(_get_task(task_id))
@router.get("/{task_id}")
def get_task(
task_id: str,
current_user: dict = Depends(get_current_user),
) -> dict[str, Any]:
task = _get_task(task_id)
if not task:
raise fail(404, "task not found")
# 附加输入文件列表
input_dir = _input_dir(task_id)
files = []
if input_dir.exists():
for f in sorted(input_dir.iterdir()):
if f.is_file():
files.append({"name": f.name, "size": f.stat().st_size})
task["input_files"] = files
return ok(task)
@router.post("/{task_id}/source-files")
async def upload_source_files(
task_id: str,
files: list[UploadFile] = File(...),
current_user: dict = Depends(get_current_user),
) -> dict[str, Any]:
task = _get_task(task_id)
if not task:
raise fail(404, "task not found")
if task["status"] not in ("pending", "uploaded"):
raise fail(400, "task is not editable")
input_dir = _input_dir(task_id)
input_dir.mkdir(parents=True, exist_ok=True)
staged = []
for upload in files:
name = Path(upload.filename or "input.json").name
if not name.lower().endswith(".json"):
raise fail(415, f"only JSON files are supported: {name}")
target = input_dir / name
content = await upload.read()
target.write_bytes(content)
staged.append({"name": name, "size": len(content)})
store = get_platform_store()
# 标记上传完成
with store.connect() as conn:
conn.execute(
"UPDATE data_convert_tasks SET status='uploaded', update_time=NOW() WHERE id=%s",
(task_id,),
)
# 自动转换并导入数据集
try:
output_dir = _output_dir(task_id)
output_dir.mkdir(parents=True, exist_ok=True)
output_path = _task_output_path(task)
# 清空旧输出（如果重新上传）
if output_path.exists():
output_path.unlink()
input_count = 0
output_count = 0
for json_file in sorted(input_dir.iterdir()):
if not json_file.is_file() or not json_file.name.lower().endswith(".json"):
continue
input_count += 1
with open(json_file, "r", encoding="utf-8") as f:
data = json.load(f)
if isinstance(data, list):
records = data
elif isinstance(data, dict):
records = [data]
else:
raise ValueError(f"JSON must be object or array: {json_file.name}")
with open(output_path, "a", encoding="utf-8") as f:
for record in records:
f.write(json.dumps(record, ensure_ascii=False) + "\n")
output_count += 1
with store.connect() as conn:
conn.execute(
"UPDATE data_convert_tasks SET status='completed', "
"input_count=%s, output_count=%s, update_time=NOW() WHERE id=%s",
(input_count, output_count, task_id),
)
# 自动导入数据集
content = output_path.read_text(encoding="utf-8")
size_bytes = len(content.encode("utf-8"))
dataset = store.create_dataset({
"name": task["name"],
"type": "train",
"storage_type": "local",
"source": "upload",
"task_id": task_id,
"size": f"{size_bytes} B",
"count": output_count,
"description": f"由数据类型转换任务 {task_id} 自动导入",
})
dataset_id = dataset["id"]
with store.connect() as conn:
store.add_dataset_file(conn, dataset_id, _safe_output_filename(task.get("output_filename")), content)
return ok({
"staged_files": staged,
"auto_converted": True,
"dataset_id": dataset_id,
"input_count": input_count,
"output_count": output_count,
})
except Exception as exc:
with store.connect() as conn:
conn.execute(
"UPDATE data_convert_tasks SET status='failed', error_message=%s, update_time=NOW() WHERE id=%s",
(str(exc)[:500], task_id),
)
return ok({"staged_files": staged, "auto_converted": False, "error": str(exc)[:500]})
@router.post("/{task_id}/run")
def run_convert(
task_id: str,
current_user: dict = Depends(get_current_user),
) -> dict[str, Any]:
task = _get_task(task_id)
if not task:
raise fail(404, "task not found")
if task["status"] not in ("uploaded", "completed", "failed"):
raise fail(400, "please upload source files first")
# 标记运行中
store = get_platform_store()
with store.connect() as conn:
conn.execute(
"UPDATE data_convert_tasks SET status='running', error_message='', update_time=NOW() WHERE id=%s",
(task_id,),
)
try:
input_dir = _input_dir(task_id)
output_dir = _output_dir(task_id)
output_dir.mkdir(parents=True, exist_ok=True)
output_path = _task_output_path(task)
input_count = 0
output_count = 0
for json_file in sorted(input_dir.iterdir()):
if not json_file.is_file() or not json_file.name.lower().endswith(".json"):
continue
input_count += 1
with open(json_file, "r", encoding="utf-8") as f:
data = json.load(f)
if isinstance(data, list):
records = data
elif isinstance(data, dict):
records = [data]
else:
raise ValueError(f"JSON must be object or array: {json_file.name}")
with open(output_path, "a", encoding="utf-8") as f:
for record in records:
f.write(json.dumps(record, ensure_ascii=False) + "\n")
output_count += 1
# 更新任务状态
with store.connect() as conn:
conn.execute(
"UPDATE data_convert_tasks SET status='completed', "
"input_count=%s, output_count=%s, update_time=NOW() WHERE id=%s",
(input_count, output_count, task_id),
)
except Exception as exc:
with store.connect() as conn:
conn.execute(
"UPDATE data_convert_tasks SET status='failed', error_message=%s, update_time=NOW() WHERE id=%s",
(str(exc)[:500], task_id),
)
raise fail(500, f"convert failed: {exc}")
return ok(_get_task(task_id))
@router.get("/{task_id}/download")
def download_result(
task_id: str,
current_user: dict = Depends(get_current_user),
):
task = _get_task(task_id)
if not task:
raise fail(404, "task not found")
if task["status"] != "completed":
raise fail(400, "task is not completed")
output_path = _task_output_path(task)
if not output_path.exists():
raise fail(404, "output file not found")
return FileResponse(
str(output_path),
media_type="application/octet-stream",
filename=_safe_output_filename(task.get("output_filename")),
)
@router.post("/{task_id}/import-as-dataset")
def import_as_dataset(
task_id: str,
payload: dict[str, Any] = Body(default={}),
current_user: dict = Depends(get_current_user),
) -> dict[str, Any]:
"""把已转换的 JSONL 文件导入为数据集管理中的上传任务记录（source='task'）。"""
task = _get_task(task_id)
if not task:
raise fail(404, "task not found")
if task["status"] != "completed":
raise fail(400, "task is not completed")
output_path = _task_output_path(task)
if not output_path.exists():
raise fail(404, "output file not found")
content = output_path.read_text(encoding="utf-8")
dataset_name = str(payload.get("name") or task["name"]).strip()
description = str(payload.get("description") or f"由数据类型转换任务 {task_id} 导入").strip()
size_bytes = len(content.encode("utf-8"))
store = get_platform_store()
# 用 store 提供的接口创建数据集与文件
dataset = store.create_dataset({
"name": dataset_name,
"type": "train",
"storage_type": "local",
"source": "upload",
"task_id": task_id,
"size": f"{size_bytes} B",
"count": task["output_count"],
"description": description,
})
dataset_id = dataset["id"]
with store.connect() as conn:
store.add_dataset_file(conn, dataset_id, _safe_output_filename(task.get("output_filename")), content)
return ok({"dataset_id": dataset_id, "name": dataset_name})
@router.delete("/{task_id}")
def delete_task(
task_id: str,
current_user: dict = Depends(get_current_user),
) -> dict[str, Any]:
task = _get_task(task_id)
if not task:
raise fail(404, "task not found")
store = get_platform_store()
with store.connect() as conn:
conn.execute(
"UPDATE data_convert_tasks SET deleted_at=NOW() WHERE id=%s",
(task_id,),
)
# 清理文件
import shutil
task_dir = _task_dir(task_id)
if task_dir.exists():
shutil.rmtree(task_dir, ignore_errors=True)
return ok({"deleted": task_id})
def _get_task(task_id: str) -> dict[str, Any] | None:
store = get_platform_store()
with store.connect() as conn:
row = conn.execute(
"SELECT * FROM data_convert_tasks WHERE id=%s AND deleted_at IS NULL",
(task_id,),
).fetchone()
return dict(row) if row else None

3330
backend/app/modules/data_process/algorithms.py
View File

File diff suppressed because it is too large Load Diff

7
backend/app/modules/data_process/constants.py
View File

@@ -1,7 +0,0 @@
"""数据处理模块的共享限制。"""
MAX_QA_PAIRS_PER_ITEM = 50
MODEL_GENERATION_BATCH_SIZE = 10
__all__ = ["MAX_QA_PAIRS_PER_ITEM", "MODEL_GENERATION_BATCH_SIZE"]

158
backend/app/modules/data_process/dataset_format.py
View File

@@ -1,158 +0,0 @@
"""Dataset format validation for Alpaca, ShareGPT, DPO, CPT formats.
Used by the training preflight flow to validate that uploaded dataset files
conform to the declared format before submitting to the compute node.
"""
from __future__ import annotations
import json
from typing import Any
def _load_sample(path: str | None, content: str | None = None, max_samples: int = 20) -> list[dict[str, Any]]:
"""Load up to max_samples records from JSONL file path or raw content string."""
try:
if content is not None:
text = content.strip()
elif path:
with open(path, "r", encoding="utf-8") as fh:
text = fh.read().strip()
else:
return []
except Exception:
return []
if not text:
return []
# 先按整文件 JSON（数组/单对象）解析，兼容 .json；失败再按 jsonl 逐行解析
try:
value = json.loads(text)
except json.JSONDecodeError:
value = None
if isinstance(value, list):
return [item for item in value[:max_samples] if isinstance(item, dict)]
if isinstance(value, dict):
return [value]
lines = text.splitlines()[:max_samples]
records: list[dict[str, Any]] = []
for line in lines:
line = line.strip()
if not line:
continue
try:
record = json.loads(line)
except json.JSONDecodeError:
continue
if isinstance(record, dict):
records.append(record)
return records
def _check_alpaca(records: list[dict[str, Any]]) -> list[str]:
"""Validate Alpaca format: requires 'instruction' field."""
errors: list[str] = []
if not records:
errors.append("Alpaca 格式数据集无有效记录")
return errors
missing_instruction = sum(1 for r in records if not r.get("instruction"))
if missing_instruction:
errors.append(
f"Alpaca 格式要求每条记录包含 instruction 字段，"
f"前{len(records)}条中有{missing_instruction}条缺失"
)
return errors
def _check_sharegpt(records: list[dict[str, Any]]) -> list[str]:
"""Validate ShareGPT format: requires 'messages' (list of dicts with role/content)."""
errors: list[str] = []
if not records:
errors.append("ShareGPT 格式数据集无有效记录")
return errors
bad = 0
for r in records:
messages = r.get("messages")
if not isinstance(messages, list) or not messages:
bad += 1
continue
for msg in messages:
if not isinstance(msg, dict) or "role" not in msg or "content" not in msg:
bad += 1
break
if bad:
errors.append(
f"ShareGPT 格式要求每条记录包含 messages 列表，"
f"每条消息需有 role 和 content 字段，前{len(records)}条中有{bad}条不符合"
)
return errors
def _check_dpo(records: list[dict[str, Any]]) -> list[str]:
"""Validate DPO format: requires 'chosen' and 'rejected' fields."""
errors: list[str] = []
if not records:
errors.append("DPO 格式数据集无有效记录")
return errors
missing_chosen = sum(1 for r in records if not r.get("chosen"))
missing_rejected = sum(1 for r in records if not r.get("rejected"))
if missing_chosen:
errors.append(f"DPO 格式要求 chosen 字段，前{len(records)}条中有{missing_chosen}条缺失")
if missing_rejected:
errors.append(f"DPO 格式要求 rejected 字段，前{len(records)}条中有{missing_rejected}条缺失")
return errors
def _check_cpt(records: list[dict[str, Any]]) -> list[str]:
"""Validate CPT format: requires 'text' field, should NOT have instruction/output."""
errors: list[str] = []
if not records:
errors.append("CPT 格式数据集无有效记录")
return errors
missing_text = sum(1 for r in records if not r.get("text"))
has_instruction = sum(1 for r in records if r.get("instruction") or r.get("output"))
if missing_text:
errors.append(f"CPT 格式要求 text 字段，前{len(records)}条中有{missing_text}条缺失")
if has_instruction:
errors.append(
f"CPT 格式不应包含 instruction/output 字段（疑似 Alpaca 格式），"
f"前{len(records)}条中有{has_instruction}条包含此类字段"
)
return errors
FORMAT_VALIDATORS = {
"alpaca": _check_alpaca,
"alpaca_jsonl": _check_alpaca,
"sharegpt": _check_sharegpt,
"dpo": _check_dpo,
"cpt": _check_cpt,
"pt": _check_cpt,
}
def validate_dataset_format(
dataset_format: str,
content: str | None = None,
path: str | None = None,
max_samples: int = 20,
) -> list[str]:
"""Validate dataset content against expected format.
Args:
dataset_format: One of 'alpaca', 'sharegpt', 'dpo', 'cpt'.
content: Raw file content (JSONL text). Mutually exclusive with path.
path: File path to read content from.
max_samples: Maximum records to sample for validation.
Returns:
List of error messages (empty if valid).
"""
fmt = str(dataset_format).lower().strip()
validator = FORMAT_VALIDATORS.get(fmt)
if not validator:
return [f"不支持的数据集格式: {dataset_format}，支持的格式: {', '.join(sorted(FORMAT_VALIDATORS))}"]
records = _load_sample(path=path, content=content, max_samples=max_samples)
return validator(records)

499
backend/app/modules/data_process/document_chunking.py
View File

@@ -1,499 +0,0 @@
"""基于 Docling 与 LlamaIndex 的文档切分实现。"""
from __future__ import annotations
import os
import re
import threading
import unicodedata
from dataclasses import dataclass
from functools import lru_cache
from io import BytesIO
from typing import Any, Literal
import tiktoken
from docling_core.transforms.chunker.hierarchical_chunker import ChunkingSerializerProvider
from llama_index.core import Document
from llama_index.core.base.embeddings.base import BaseEmbedding
from llama_index.core.node_parser import SemanticSplitterNodeParser, SentenceSplitter
from app.modules.data_process.algorithms import normalize_text
ChunkMethod = Literal["layout_hybrid", "semantic", "fixed"]
_PAGE_FURNITURE = re.compile(
r"(?m)^\s*(?:第\s*\d+\s*页\s*共\s*\d+\s*页|[-—–]?\s*\d+\s*[/／]\s*\d+\s*[-—–]?)\s*$"
)
_COMPACT_CHARACTER = re.compile(r"[\w\u3400-\u4dbf\u4e00-\u9fff]", re.UNICODE)
_CONVERTER_LOCK = threading.Lock()
@dataclass(frozen=True, slots=True)
class DocumentChunk:
"""切片正文及其在原文件中的可追溯信息。"""
original_content: str
contextualized_content: str
source_start: int | None
source_end: int | None
source_start_line: int | None
source_end_line: int | None
token_count: int
heading_path: tuple[str, ...] = ()
source_pages: tuple[int, ...] = ()
doc_item_refs: tuple[str, ...] = ()
source_bboxes: tuple[dict[str, Any], ...] = ()
def _sentence_chunks(text: str) -> list[str]:
"""提供稳定的中英文句界，避免 LlamaIndex 默认分词器下载额外资源。"""
boundary = re.compile(
r".*?(?:\n\s*\n|[。！？!?；;](?:[\"'”’）】》]*)|\.(?:\s+|$)|$)",
re.DOTALL,
)
return [part for part in boundary.findall(text) if part]
@lru_cache(maxsize=1)
def _tokenizer() -> tiktoken.Encoding:
"""加载 cl100k_base 编码器，优先在线下载，失败时使用本地缓存以支持离线环境。"""
import os
import base64
# 先设置缓存目录环境变量
offline_cache = os.path.expanduser("~/.cache/tiktoken")
os.environ.setdefault("TIKTOKEN_CACHE_DIR", offline_cache)
try:
# 尝试标准方式加载
return tiktoken.get_encoding("cl100k_base")
except Exception:
# 如果失败，尝试手动从本地文件构造
try:
from pathlib import Path
local_file = Path(offline_cache) / "9b5ad71b2ce5302211f9c61530b329a4922fc6a4"
if not local_file.exists():
# 尝试另一个可能的文件名
local_file = Path(offline_cache) / "cl100k_base.tiktoken"
if local_file.exists():
# 读取 BPE 文件内容
with open(local_file, "rb") as f:
contents = f.read()
# 解析 BPE 文件
mergeable_ranks = {}
for line in contents.splitlines():
if line:
token, rank = line.split()
mergeable_ranks[base64.b64decode(token)] = int(rank)
# 构造 Encoding 对象
import tiktoken.core
return tiktoken.core.Encoding(
name="cl100k_base",
pat_str=r"""'(?i:[sdmt]|ll|ve|re)|[^\r\n\p{L}\p{N}]?+\p{L}+|\p{N}{1,3}| ?[^\s\p{L}\p{N}]++[\r\n]*|\s*[\r\n]|\s+(?!\S)|\s+""",
mergeable_ranks=mergeable_ranks,
special_tokens={
"<|endoftext|>": 100257,
"<|fim_prefix|>": 100258,
"<|fim_middle|>": 100259,
"<|fim_suffix|>": 100260,
"<|endofprompt|>": 100276,
},
)
except Exception:
pass
raise RuntimeError(
f"无法加载 cl100k_base 编码器\n"
f"请确保以下任一条件满足:\n"
f"1. 服务器可以访问网络\n"
f"2. 本地存在缓存文件: {offline_cache}/9b5ad71b2ce5302211f9c61530b329a4922fc6a4"
)
def _text_chunks(
text: str,
*,
chunk_size: int,
chunk_overlap: int,
) -> list[DocumentChunk]:
normalized = normalize_text(text)
if not normalized:
return []
splitter = SentenceSplitter(
chunk_size=chunk_size,
chunk_overlap=chunk_overlap,
tokenizer=_tokenizer().encode,
chunking_tokenizer_fn=_sentence_chunks,
include_metadata=False,
include_prev_next_rel=False,
)
nodes = splitter.get_nodes_from_documents([Document(text=normalized)])
return _nodes_to_chunks(nodes, normalized)
def chunk_fixed_text(
text: str,
*,
chunk_size: int,
chunk_overlap: int,
) -> list[DocumentChunk]:
"""使用 LlamaIndex SentenceSplitter 按句界控制固定 Token 长度。"""
return _text_chunks(text, chunk_size=chunk_size, chunk_overlap=chunk_overlap)
@lru_cache(maxsize=1)
def _semantic_embedding_model() -> BaseEmbedding:
# 模型可在部署环境覆盖；默认模型体积较小且适合中英文语义边界判断。
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
return HuggingFaceEmbedding(
model_name=os.getenv("DATA_PROCESS_EMBEDDING_MODEL", "BAAI/bge-small-zh-v1.5"),
device=os.getenv("DATA_PROCESS_EMBEDDING_DEVICE", "cpu"),
trust_remote_code=False,
)
def chunk_semantic_text(
text: str,
*,
chunk_size: int,
chunk_overlap: int,
breakpoint_percentile_threshold: int,
embed_model: BaseEmbedding | None = None,
) -> list[DocumentChunk]:
"""使用 LlamaIndex SemanticSplitter 识别主题跳变，再限制最大长度。"""
normalized = normalize_text(text)
if not normalized:
return []
splitter = SemanticSplitterNodeParser.from_defaults(
embed_model=embed_model or _semantic_embedding_model(),
breakpoint_percentile_threshold=breakpoint_percentile_threshold,
buffer_size=1,
sentence_splitter=_sentence_chunks,
include_metadata=False,
include_prev_next_rel=False,
)
semantic_nodes = splitter.get_nodes_from_documents([Document(text=normalized)])
result: list[DocumentChunk] = []
search_from = 0
for node in semantic_nodes:
content = node.get_content().strip()
if not content:
continue
start = _locate_text(normalized, content, search_from)
if start is None:
start = _locate_text(normalized, content, 0)
if start is None:
continue
if len(_tokenizer().encode(content)) <= chunk_size:
result.append(_make_text_chunk(normalized, start, start + len(content)))
else:
for child in _text_chunks(
content,
chunk_size=chunk_size,
chunk_overlap=chunk_overlap,
):
if child.source_start is None or child.source_end is None:
continue
result.append(
_make_text_chunk(
normalized,
start + child.source_start,
start + child.source_end,
)
)
search_from = start + len(content)
return result
def _nodes_to_chunks(nodes: list[Any], source_text: str) -> list[DocumentChunk]:
chunks: list[DocumentChunk] = []
search_from = 0
for node in nodes:
content = node.get_content().strip()
if not content:
continue
raw_start = getattr(node, "start_char_idx", None)
raw_end = getattr(node, "end_char_idx", None)
if (
isinstance(raw_start, int)
and isinstance(raw_end, int)
and source_text[raw_start:raw_end].strip() == content
):
start = raw_start + len(source_text[raw_start:raw_end]) - len(source_text[raw_start:raw_end].lstrip())
else:
start = _locate_text(source_text, content, search_from)
if start is None:
start = _locate_text(source_text, content, 0)
if start is None:
continue
end = start + len(content)
chunks.append(_make_text_chunk(source_text, start, end))
search_from = max(search_from, end)
return chunks
def _locate_text(source: str, content: str, start: int) -> int | None:
position = source.find(content, start)
return position if position >= 0 else None
def _make_text_chunk(source: str, start: int, end: int) -> DocumentChunk:
content = source[start:end]
return DocumentChunk(
original_content=content,
contextualized_content=content,
source_start=start,
source_end=end,
source_start_line=source.count("\n", 0, start) + 1,
source_end_line=source.count("\n", 0, max(start, end - 1)) + 1,
token_count=len(_tokenizer().encode(content)),
)
@lru_cache(maxsize=1)
def _document_converter():
from docling.document_converter import DocumentConverter
return DocumentConverter()
class _MarkdownSerializerProvider(ChunkingSerializerProvider):
def get_serializer(self, doc: Any):
from docling_core.transforms.chunker.hierarchical_chunker import ChunkingDocSerializer
from docling_core.transforms.serializer.markdown import (
MarkdownParams,
MarkdownTableSerializer,
)
from docling_core.types.doc import DocItemLabel
excluded = {
DocItemLabel.DOCUMENT_INDEX,
DocItemLabel.PAGE_HEADER,
DocItemLabel.PAGE_FOOTER,
}
return ChunkingDocSerializer(
doc=doc,
table_serializer=MarkdownTableSerializer(),
params=MarkdownParams(
labels=set(DocItemLabel) - excluded,
compact_tables=True,
image_placeholder="",
escape_html=False,
escape_underscores=False,
),
)
def _clean_layout_text(value: str) -> str:
return normalize_text(_PAGE_FURNITURE.sub("", value)).strip()
def _compact_with_offsets(value: str) -> tuple[str, list[int]]:
compact: list[str] = []
offsets: list[int] = []
for index, character in enumerate(unicodedata.normalize("NFKC", value)):
if _COMPACT_CHARACTER.fullmatch(character):
compact.append(character.casefold())
offsets.append(index)
return "".join(compact), offsets
def _project_layout_span(
source_text: str,
content: str,
*,
compact_source: str,
source_offsets: list[int],
compact_start: int,
) -> tuple[int | None, int | None, int]:
compact_content, _ = _compact_with_offsets(content)
if len(compact_content) < 4:
return None, None, compact_start
position = compact_source.find(compact_content, compact_start)
if position < 0:
position = compact_source.find(compact_content)
if position < 0:
return None, None, compact_start
start = source_offsets[position]
end = source_offsets[position + len(compact_content) - 1] + 1
while start > 0 and source_text[start - 1] not in "\r\n":
start -= 1
while end < len(source_text) and source_text[end] not in "\r\n":
end += 1
return start, end, position + len(compact_content)
def chunk_layout_document(
raw: bytes,
*,
filename: str,
source_text: str,
chunk_size: int,
) -> list[DocumentChunk]:
"""使用 Docling HybridChunker 按版面层级、列表与表格边界切分。"""
from docling.chunking import HybridChunker
from docling.datamodel.base_models import DocumentStream
from docling.exceptions import BaseError as DoclingError
from docling_core.transforms.chunker.tokenizer.openai import OpenAITokenizer
from docling_core.types.doc import DocItemLabel
try:
with _CONVERTER_LOCK:
conversion = _document_converter().convert(
DocumentStream(name=filename, stream=BytesIO(raw))
)
except DoclingError as exc:
raise ValueError(f"文档版面解析失败: {exc}") from exc
chunker = HybridChunker(
tokenizer=OpenAITokenizer(tokenizer=_tokenizer(), max_tokens=chunk_size),
serializer_provider=_MarkdownSerializerProvider(),
merge_peers=True,
repeat_table_header=True,
)
compact_source, source_offsets = _compact_with_offsets(source_text)
compact_start = 0
result: list[DocumentChunk] = []
excluded = {
DocItemLabel.DOCUMENT_INDEX,
DocItemLabel.PAGE_HEADER,
DocItemLabel.PAGE_FOOTER,
}
for raw_chunk in chunker.chunk(conversion.document):
doc_items = tuple(raw_chunk.meta.doc_items or ())
if doc_items and all(item.label in excluded for item in doc_items):
continue
content = _clean_layout_text(raw_chunk.text)
if not content:
continue
contextualized = _clean_layout_text(chunker.contextualize(raw_chunk)) or content
start, end, compact_start = _project_layout_span(
source_text,
content,
compact_source=compact_source,
source_offsets=source_offsets,
compact_start=compact_start,
)
original = source_text[start:end] if start is not None and end is not None else content
pages: set[int] = set()
refs: list[str] = []
bboxes: list[dict[str, Any]] = []
for item in doc_items:
refs.append(str(item.self_ref))
for provenance in item.prov or ():
pages.add(int(provenance.page_no))
bbox = provenance.bbox
bboxes.append(
{
"page": int(provenance.page_no),
"left": float(bbox.l),
"top": float(bbox.t),
"right": float(bbox.r),
"bottom": float(bbox.b),
"origin": str(bbox.coord_origin.value),
}
)
result.append(
DocumentChunk(
original_content=original,
contextualized_content=contextualized,
source_start=start,
source_end=end,
source_start_line=(source_text.count("\n", 0, start) + 1 if start is not None else None),
source_end_line=(
source_text.count("\n", 0, max(start or 0, (end or 1) - 1)) + 1
if end is not None
else None
),
token_count=len(_tokenizer().encode(contextualized)),
heading_path=tuple(str(item) for item in (raw_chunk.meta.headings or ())),
source_pages=tuple(sorted(pages)),
doc_item_refs=tuple(refs),
source_bboxes=tuple(bboxes),
)
)
return result
def merge_short_chunks(
chunks: list[DocumentChunk],
*,
source_text: str,
min_token_count: int,
max_token_count: int,
) -> list[DocumentChunk]:
"""在不突破长度上限的前提下，把过短块并入相邻内容。"""
result: list[DocumentChunk] = []
index = 0
while index < len(chunks):
current = chunks[index]
if current.token_count >= min_token_count:
result.append(current)
index += 1
continue
if index + 1 < len(chunks):
combined = _combine_chunks(current, chunks[index + 1], source_text)
if combined.token_count <= max_token_count:
result.append(combined)
index += 2
continue
if result:
combined = _combine_chunks(result[-1], current, source_text)
if combined.token_count <= max_token_count:
result[-1] = combined
index += 1
continue
result.append(current)
index += 1
return result
def _combine_chunks(
left: DocumentChunk,
right: DocumentChunk,
source_text: str,
) -> DocumentChunk:
contextualized = "\n\n".join(
part for part in (left.contextualized_content, right.contextualized_content) if part
)
start = left.source_start
end = right.source_end
has_contiguous_source = (
start is not None
and left.source_end is not None
and right.source_start is not None
and end is not None
and left.source_end <= right.source_start
)
original = (
source_text[start:end]
if has_contiguous_source and start is not None and end is not None
else "\n\n".join(
part for part in (left.original_content, right.original_content) if part
)
)
if not has_contiguous_source:
start = None
end = None
return DocumentChunk(
original_content=original,
contextualized_content=contextualized,
source_start=start,
source_end=end,
source_start_line=left.source_start_line if start is not None else None,
source_end_line=right.source_end_line if end is not None else None,
token_count=len(_tokenizer().encode(contextualized)),
heading_path=left.heading_path or right.heading_path,
source_pages=tuple(sorted(set(left.source_pages) | set(right.source_pages))),
doc_item_refs=left.doc_item_refs + right.doc_item_refs,
source_bboxes=left.source_bboxes + right.source_bboxes,
)

BIN
backend/app/modules/data_process/document_chunking.txt
View File

Binary file not shown.

622
backend/app/modules/data_process/generation.py
View File

@@ -1,622 +0,0 @@
"""数据处理任务的大模型生成适配器。"""
from __future__ import annotations
import hashlib
import json
import logging
import re
from collections.abc import Callable, Iterable, Mapping
from typing import Any
from urllib.parse import urlsplit, urlunsplit
import httpx
from app.modules.data_process.algorithms import normalize_text, stable_split_assignments
from app.modules.data_process.constants import (
MAX_QA_PAIRS_PER_ITEM,
MODEL_GENERATION_BATCH_SIZE,
)
class ModelGenerationError(ValueError):
"""模型配置、响应或调用失败。"""
class _TerminalModelGenerationError(ModelGenerationError):
"""使用相同参数重试也无法恢复的模型响应错误。"""
OUTPUT_TYPE_STANDARD = "standard"
OUTPUT_TYPE_REASONING = "reasoning"
OUTPUT_TYPE_DPO = "dpo"
SUPPORTED_OUTPUT_TYPES = {
OUTPUT_TYPE_STANDARD,
OUTPUT_TYPE_REASONING,
OUTPUT_TYPE_DPO,
}
REASONING_DETAIL_NORMAL = "normal"
REASONING_DETAIL_DETAILED = "detailed"
SUPPORTED_REASONING_DETAILS = {
REASONING_DETAIL_NORMAL,
REASONING_DETAIL_DETAILED,
}
MINIMAX_M3_API_HOSTS = {"api.minimax.io", "api.minimaxi.com"}
MINIMAX_M3_MIN_COMPLETION_TOKENS = 4096
logger = logging.getLogger(__name__)
def _is_retryable_generation_error(exc: Exception) -> bool:
if isinstance(exc, _TerminalModelGenerationError):
return False
if isinstance(exc, httpx.HTTPStatusError):
status_code = exc.response.status_code
return status_code in {408, 425, 429} or status_code >= 500
if isinstance(exc, httpx.RequestError):
return True
return isinstance(exc, (json.JSONDecodeError, ModelGenerationError))
def _is_official_minimax_m3(endpoint: str, model_name: str) -> bool:
host = (urlsplit(endpoint).hostname or "").casefold()
return host in MINIMAX_M3_API_HOSTS and model_name.casefold() == "minimax-m3"
def chat_completions_url(value: str) -> str:
"""把域名、基础 URL 或完整地址统一为 chat completions 地址。"""
raw = (value or "").strip()
if not raw:
raise ModelGenerationError("generation model api_url is required")
if "://" not in raw:
raw = f"https://{raw}"
parsed = urlsplit(raw)
if parsed.scheme not in {"http", "https"} or not parsed.hostname:
raise ModelGenerationError("generation model api_url must be an HTTP(S) host or URL")
if parsed.username or parsed.password:
raise ModelGenerationError("generation model api_url must not contain credentials")
path = parsed.path.rstrip("/")
if path.endswith("/chat/completions"):
target_path = path
elif path.endswith("/v1"):
target_path = f"{path}/chat/completions"
elif not path:
target_path = "/v1/chat/completions"
else:
target_path = f"{path}/v1/chat/completions"
return urlunsplit((parsed.scheme, parsed.netloc, target_path, "", ""))
def _response_choice(payload: Mapping[str, Any]) -> Mapping[str, Any]:
try:
choice = payload["choices"][0]
except (KeyError, IndexError, TypeError) as exc:
raise ModelGenerationError("模型响应缺少 choices[0]") from exc
if not isinstance(choice, Mapping):
raise ModelGenerationError("模型响应 choices[0] 不是对象")
return choice
def _response_finish_reason(payload: Mapping[str, Any]) -> str:
try:
return str(_response_choice(payload).get("finish_reason") or "").strip().lower()
except ModelGenerationError:
return ""
def _response_content_length(payload: Mapping[str, Any]) -> int:
try:
message = _response_choice(payload).get("message")
if not isinstance(message, Mapping):
return 0
content = message.get("content")
if isinstance(content, str):
return len(content)
if isinstance(content, list):
return sum(
len(str(item.get("text") or ""))
for item in content
if isinstance(item, Mapping)
)
except ModelGenerationError:
pass
return 0
def _raise_for_terminal_response(payload: Mapping[str, Any]) -> Mapping[str, Any]:
choice = _response_choice(payload)
base_response = payload.get("base_resp")
status_code: Any = None
status_message = ""
if isinstance(base_response, Mapping):
status_code = base_response.get("status_code")
status_message = re.sub(
r"\s+", " ", str(base_response.get("status_msg") or "")
).strip()[:200]
if bool(payload.get("input_sensitive")) or status_code in {1026, "1026"}:
raise _TerminalModelGenerationError(
f"模型输入触发内容安全拦截（code={status_code or 1026}）"
)
if bool(payload.get("output_sensitive")) or status_code in {1027, "1027"}:
raise _TerminalModelGenerationError(
f"模型输出触发内容安全拦截（code={status_code or 1027}）"
)
finish_reason = str(choice.get("finish_reason") or "").strip().lower()
if finish_reason == "length":
raise _TerminalModelGenerationError(
"模型输出因达到 Token 上限被截断（finish_reason=length），"
"请提高最大输出长度后重试"
)
if finish_reason == "content_filter":
raise _TerminalModelGenerationError(
"模型输出被内容安全策略拦截（finish_reason=content_filter）"
)
if finish_reason in {"tool_calls", "function_call"}:
raise _TerminalModelGenerationError(
f"模型返回了当前生成任务不支持的工具调用（finish_reason={finish_reason}）"
)
if status_code not in {None, "", 0, "0"}:
detail = f"：{status_message}" if status_message else ""
raise _TerminalModelGenerationError(
f"模型服务返回业务错误（code={status_code}）{detail}"
)
return choice
def _message_content(payload: Mapping[str, Any]) -> str:
choice = _raise_for_terminal_response(payload)
message = choice.get("message")
if not isinstance(message, Mapping):
raise ModelGenerationError("模型响应缺少 choices[0].message")
content = message.get("content")
if isinstance(content, str):
result = content
elif isinstance(content, list):
parts = [
str(item.get("text") or "")
for item in content
if isinstance(item, Mapping) and item.get("type") in {None, "text", "output_text"}
]
result = "".join(parts)
elif content is None:
result = ""
else:
raise ModelGenerationError("模型响应 content 必须是文本")
if not result.strip():
raise ModelGenerationError("模型返回的最终内容为空，未生成可解析的 JSON")
return result
def _json_documents(content: str) -> list[Any]:
decoder = json.JSONDecoder()
documents: list[Any] = []
cursor = 0
while cursor < len(content):
match = re.search(r"[\[{]", content[cursor:])
if not match:
break
start = cursor + match.start()
try:
value, end = decoder.raw_decode(content[start:])
except json.JSONDecodeError:
cursor = start + 1
continue
if isinstance(value, (Mapping, list)):
documents.append(value)
cursor = start + max(end, 1)
return documents
def _json_payload(content: str) -> Any:
# 只移除模型在 JSON 之前自行输出的思考过程，不能破坏 JSON 字段中的训练内容。
cleaned = content.strip()
if re.match(r"^\s*<think>", cleaned, flags=re.IGNORECASE) and not re.match(
r"^\s*<think>[\s\S]*?</think>", cleaned, flags=re.IGNORECASE
):
raise ModelGenerationError("模型思考内容未闭合，响应可能已被截断")
cleaned = re.sub(
r"^\s*(?:<think>[\s\S]*?</think>\s*)+",
"",
cleaned,
count=1,
flags=re.IGNORECASE,
).strip()
fenced = re.fullmatch(r"```(?:json)?\s*([\s\S]*?)\s*```", cleaned, flags=re.IGNORECASE)
if fenced:
cleaned = fenced.group(1).strip()
try:
return json.loads(cleaned)
except json.JSONDecodeError as direct_error:
documents = _json_documents(cleaned)
if len(documents) == 1:
return documents[0]
if len(documents) > 1:
raise ModelGenerationError("模型响应包含多个 JSON 对象，无法确定应使用哪一个")
raise ModelGenerationError(
"模型响应中没有找到唯一且完整的 JSON 对象"
f"（第 {direct_error.lineno} 行，第 {direct_error.colno} 列）"
) from direct_error
def _result_items(payload: Any) -> list[Mapping[str, Any]]:
if isinstance(payload, list):
values = payload
elif isinstance(payload, Mapping):
nested = next(
(
payload[key]
for key in ("items", "results", "data", "records")
if isinstance(payload.get(key), list)
),
None,
)
values = nested if isinstance(nested, list) else [payload]
else:
raise ModelGenerationError("model JSON must be an object or array")
items = [item for item in values if isinstance(item, Mapping)]
if not items:
raise ModelGenerationError("model JSON does not contain result objects")
return items
def _prompt_messages(
prompt: str,
content: str,
count: int,
*,
start_index: int,
total_count: int,
output_type: str,
reasoning_detail: str,
) -> list[dict[str, str]]:
end_index = start_index + count - 1
if output_type == OUTPUT_TYPE_REASONING:
schema = '{"items":[{"instruction":"...","input":"...","reasoning":"...","answer":"..."}]}'
detail_rule = (
"推理详细程度为“详细”：完整展开问题条件、来源依据、中间计算或推导，"
"并在得出答案前核对结论；每一步都必须能从来源内容中验证。"
if reasoning_detail == REASONING_DETAIL_DETAILED
else
"推理详细程度为“普通”：只保留得出答案所需的关键依据和必要步骤，"
"避免冗长复述、套话和无依据扩展。"
)
output_rule = (
"你正在生成用于训练推理模型的思维链数据，而不是普通问答数据。"
"instruction、reasoning 和 answer 均不得为空；reasoning 必须是基于来源内容、"
f"可核对的推理过程，answer 只写最终答案。{detail_rule}"
"这是思维链输出模式，即使其他提示语要求省略分析，也不得省略 reasoning。"
"不要自行添加 <think> 标签，系统会在保存时统一组装。"
)
elif output_type == OUTPUT_TYPE_DPO:
schema = (
'{"items":[{"instruction":"...","input":"...",'
'"chosen":"...","rejected":"..."}]}'
)
output_rule = (
"你正在生成用于直接偏好优化（DPO）的成对偏好数据。"
"instruction、chosen 和 rejected 均不得为空；chosen 必须是忠于来源、"
"准确完整的优选回答，rejected 必须是表面合理但存在明确质量缺陷的拒选回答。"
"两者不得相同；rejected 不得包含违法危险内容，也不得用空白、乱码或无关文本凑数。"
"不要输出分析过程或 <think> 标签。"
)
else:
schema = '{"items":[{"instruction":"...","input":"...","output":"..."}]}'
output_rule = (
"你正在生成标准监督微调问答数据。instruction 和 output 不得为空；"
"output 只写最终答案，禁止输出分析、推理过程或 <think> 标签。"
)
schema_instruction = (
f"必须只返回 JSON 对象，格式为 {schema}；items 必须包含 {count} 条。"
f"这是总计 {total_count} 条中的第 {start_index}-{end_index} 条，"
"各条必须使用不同的提问角度和表述，避免重复。"
f"{output_rule}不要输出 Markdown 代码围栏或 JSON 之外的说明。"
)
base_prompt = normalize_text(prompt) or "请根据来源内容生成可用于监督微调的问答数据。"
if "{{ content }}" in base_prompt:
user_prompt = base_prompt.replace("{{ content }}", content)
return [
{"role": "system", "content": schema_instruction},
{"role": "user", "content": user_prompt},
]
return [
{"role": "system", "content": f"{base_prompt}\n{schema_instruction}"},
{"role": "user", "content": f"来源内容：\n{content}"},
]
def generate_model_records(
preview_items: Iterable[Mapping[str, Any]],
*,
model: Mapping[str, Any],
config: Mapping[str, Any],
task_id: str,
split: Mapping[str, int],
qa_pairs_per_item: int,
client: httpx.Client | None = None,
on_progress: Callable[[int, int], None] | None = None,
) -> list[dict[str, Any]]:
"""调用 OpenAI 兼容接口，将预览切片生成标准训练记录。
每个切片按安全批次调用模型；失败批次会产生一条可人工修复的
invalid 结果，已经成功的批次不会丢失。
"""
if not 1 <= qa_pairs_per_item <= MAX_QA_PAIRS_PER_ITEM:
raise ModelGenerationError(f"qa_pairs_per_item must be in [1, {MAX_QA_PAIRS_PER_ITEM}]")
output_type = str(config.get("output_type") or OUTPUT_TYPE_STANDARD).strip().lower()
if output_type not in SUPPORTED_OUTPUT_TYPES:
raise ModelGenerationError(f"output_type must be one of {sorted(SUPPORTED_OUTPUT_TYPES)}")
reasoning_detail = str(
config.get("reasoning_detail") or REASONING_DETAIL_NORMAL
).strip().lower()
if reasoning_detail not in SUPPORTED_REASONING_DETAILS:
raise ModelGenerationError(
f"reasoning_detail must be one of {sorted(SUPPORTED_REASONING_DETAILS)}"
)
endpoint = chat_completions_url(str(model.get("api_url") or ""))
model_name = str(model.get("online_model_name") or model.get("name") or "").strip()
if not model_name:
raise ModelGenerationError("generation model name is required")
is_minimax_m3 = _is_official_minimax_m3(endpoint, model_name)
temperature = float(config.get("temperature", 0.7))
max_tokens = int(config.get("max_tokens", 1024))
timeout = max(1.0, min(120.0, float(config.get("request_timeout_seconds", 60))))
retries = max(0, min(5, int(config.get("generation_retries", 2))))
headers = {"Content-Type": "application/json"}
api_key = str(model.get("api_key") or "").strip()
if api_key:
headers["Authorization"] = f"Bearer {api_key}"
owns_client = client is None
http_client = client or httpx.Client(timeout=timeout)
results: list[dict[str, Any]] = []
try:
preview_list = list(preview_items)
total_items = len(preview_list)
for item_index, item in enumerate(preview_list):
preview_id = str(item.get("id") or f"preview-{item_index + 1}")
content = normalize_text(
str(item.get("edited_content") or item.get("original_content") or "")
)
for batch_offset in range(0, qa_pairs_per_item, MODEL_GENERATION_BATCH_SIZE):
batch_count = min(
MODEL_GENERATION_BATCH_SIZE,
qa_pairs_per_item - batch_offset,
)
batch_start = batch_offset + 1
batch_end = batch_offset + batch_count
request_payload: dict[str, Any] = {
"model": model_name,
"messages": _prompt_messages(
str(config.get("generation_prompt") or ""),
content,
batch_count,
start_index=batch_start,
total_count=qa_pairs_per_item,
output_type=output_type,
reasoning_detail=reasoning_detail,
),
"temperature": temperature,
}
if is_minimax_m3:
request_payload.update(
reasoning_split=True,
max_completion_tokens=max(
max_tokens,
MINIMAX_M3_MIN_COMPLETION_TOKENS,
),
)
else:
request_payload["max_tokens"] = max_tokens
if bool(config.get("json_mode", False)) and not is_minimax_m3:
request_payload["response_format"] = {"type": "json_object"}
last_error: Exception | None = None
generated_items: list[Mapping[str, Any]] | None = None
for _ in range(retries + 1):
try:
response = http_client.post(
endpoint,
headers=headers,
json=request_payload,
)
response.raise_for_status()
body = response.json()
if not isinstance(body, Mapping):
raise ModelGenerationError("model response body must be a JSON object")
try:
candidate_items = _result_items(
_json_payload(_message_content(body))
)
except ModelGenerationError as exc:
logger.warning(
"data process model response rejected task_id=%s model=%s "
"finish_reason=%s response_chars=%s input_sensitive=%s "
"output_sensitive=%s reason=%s",
task_id,
model_name,
_response_finish_reason(body) or "missing",
_response_content_length(body),
bool(body.get("input_sensitive")),
bool(body.get("output_sensitive")),
str(exc),
)
raise
if len(candidate_items) < batch_count:
raise ModelGenerationError(
"model response contains fewer result objects than requested: "
f"expected {batch_count}, got {len(candidate_items)}"
)
generated_items = candidate_items
break
except (
httpx.HTTPError,
json.JSONDecodeError,
ModelGenerationError,
) as exc:
last_error = exc
if not _is_retryable_generation_error(exc):
break
if generated_items is None:
error_message = str(last_error or "model generation failed")[:2000]
failure_instruction = (
f"模型生成失败，请人工补充（第 {batch_start}-{batch_end} 条）"
)
result_id = (
"result_"
f"{hashlib.sha256(f'{preview_id}:error:{batch_start}'.encode()).hexdigest()[:16]}"
)
results.append(
{
"id": result_id,
"preview_item_id": preview_id,
"instruction": failure_instruction,
"input": content,
"output": "",
"chosen": "",
"rejected": "",
"original_instruction": failure_instruction,
"original_input": content,
"original_output": "",
"original_chosen": "",
"original_rejected": "",
"status": "invalid",
"error": error_message,
"split": "train",
}
)
continue
for batch_index, value in enumerate(generated_items[:batch_count]):
variant_index = batch_offset + batch_index
instruction = normalize_text(
str(value.get("instruction") or value.get("question") or "")
)
input_text = normalize_text(
str(value.get("input") or value.get("context") or "")
)
chosen = ""
rejected = ""
if output_type == OUTPUT_TYPE_REASONING:
reasoning = normalize_text(
re.sub(
r"</?think>",
"",
str(value.get("reasoning") or value.get("analysis") or ""),
flags=re.IGNORECASE,
)
)
answer = normalize_text(
re.sub(
r"</?think>",
"",
str(
value.get("answer")
or value.get("final_answer")
or value.get("output")
or ""
),
flags=re.IGNORECASE,
)
)
output = (
f"<think>\n{reasoning}\n</think>\n{answer}"
if reasoning and answer
else answer or (f"<think>\n{reasoning}\n</think>" if reasoning else "")
)
valid = bool(instruction and reasoning and answer)
missing_error = "model result is missing instruction, reasoning or answer"
elif output_type == OUTPUT_TYPE_DPO:
chosen = normalize_text(str(value.get("chosen") or ""))
rejected = normalize_text(str(value.get("rejected") or ""))
chosen = normalize_text(
re.sub(
r"<think>[\s\S]*?(?:</think>|$)",
"",
chosen,
flags=re.IGNORECASE,
)
)
rejected = normalize_text(
re.sub(
r"<think>[\s\S]*?(?:</think>|$)",
"",
rejected,
flags=re.IGNORECASE,
)
)
output = chosen
valid = bool(
instruction
and chosen
and rejected
and chosen.strip() != rejected.strip()
)
missing_error = (
"model result is missing instruction, chosen or rejected, "
"or chosen equals rejected"
)
else:
output = normalize_text(
str(
value.get("output")
or value.get("answer")
or value.get("response")
or ""
)
)
output = normalize_text(
re.sub(
r"<think>[\s\S]*?(?:</think>|$)",
"",
output,
flags=re.IGNORECASE,
)
)
valid = bool(instruction and output)
missing_error = "model result is missing instruction or output"
raw_id = (
f"{preview_id}:{variant_index + 1}:{instruction}:"
f"{output}:{rejected}"
)
result_id = f"result_{hashlib.sha256(raw_id.encode()).hexdigest()[:16]}"
results.append(
{
"id": result_id,
"preview_item_id": preview_id,
"instruction": instruction,
"input": input_text,
"output": output,
"chosen": chosen,
"rejected": rejected,
"original_instruction": instruction,
"original_input": input_text,
"original_output": output,
"original_chosen": chosen,
"original_rejected": rejected,
"status": "valid" if valid else "invalid",
"error": (None if valid else missing_error),
"split": "train",
}
)
if on_progress:
on_progress(item_index + 1, total_items)
finally:
if owns_client:
http_client.close()
assignments = stable_split_assignments(
[str(result["id"]) for result in results],
split,
seed=task_id,
)
for result, assignment in zip(results, assignments, strict=True):
result["split"] = assignment
return results
__all__ = ["ModelGenerationError", "chat_completions_url", "generate_model_records"]

308
backend/app/modules/data_process/office_preview.py
View File

@@ -1,308 +0,0 @@
"""Word 与 Excel 原文件的安全、受限预览模型。
预览只返回浏览器绘制所需的结构化数据，不返回或执行 Office 包中的活动内容。
DOCX 的字符偏移与上传时的正文抽取规则保持一致，供前端定位当前切片。
"""
from __future__ import annotations
import io
import re
from typing import Any
from docx import Document
from docx.oxml.table import CT_Tbl
from docx.oxml.text.paragraph import CT_P
from docx.table import Table
from docx.text.paragraph import Paragraph
from openpyxl import load_workbook
from app.modules.data_process.algorithms import (
_MAX_WORKBOOK_COLUMNS,
_MAX_WORKBOOK_HEADER_SCAN_ROWS,
_infer_xlsx_header_region,
_normalize_spreadsheet_value,
_rewrite_xlsx_workbook_relationships,
_validate_office_archive,
_xlsx_sheet_merge_ranges,
normalize_text,
)
MAX_DOCX_PREVIEW_BLOCKS = 2_000
MAX_XLSX_PREVIEW_ROWS = 200
def _docx_alignment(paragraph: Paragraph) -> str:
value = paragraph.alignment
return {
0: "left",
1: "center",
2: "right",
3: "justify",
4: "distribute",
5: "justify",
7: "justify",
8: "distribute",
9: "distribute",
}.get(int(value) if value is not None else -1, "left")
def _docx_heading_level(paragraph: Paragraph) -> int | None:
style = paragraph.style
if style is None:
return None
style_name = str(style.name or "")
style_id = str(style.style_id or "")
match = re.search(r"(?:heading|标题)\s*([1-6])", f"{style_name} {style_id}", re.IGNORECASE)
return int(match.group(1)) if match else None
def build_docx_preview(raw: bytes) -> dict[str, Any]:
"""把 DOCX 转为保留标题、段落和表格顺序的浏览器预览模型。"""
_validate_office_archive(raw, "docx")
try:
document = Document(io.BytesIO(raw))
except Exception as exc:
raise ValueError(f"invalid DOCX file: {exc}") from exc
blocks: list[dict[str, Any]] = []
source_cursor = 0
has_source_content = False
rendered_blocks = 0
truncated = False
def source_range(value: str) -> tuple[str, int, int] | None:
nonlocal source_cursor, has_source_content
text = normalize_text(value)
if not text:
return None
if has_source_content:
source_cursor += 2
start = source_cursor
source_cursor += len(text)
has_source_content = True
return text, start, source_cursor
for child in document.element.body.iterchildren():
if rendered_blocks >= MAX_DOCX_PREVIEW_BLOCKS:
truncated = True
break
if isinstance(child, CT_P):
paragraph = Paragraph(child, document)
located = source_range(paragraph.text)
if located is None:
continue
text, start, end = located
style_name = str(paragraph.style.name or "") if paragraph.style else ""
blocks.append(
{
"type": "paragraph",
"text": text,
"style": style_name,
"heading_level": _docx_heading_level(paragraph),
"alignment": _docx_alignment(paragraph),
"is_list": "list" in style_name.casefold() or "列表" in style_name,
"source_start": start,
"source_end": end,
}
)
rendered_blocks += 1
continue
if not isinstance(child, CT_Tbl):
continue
table = Table(child, document)
preview_rows: list[dict[str, Any]] = []
for row in table.rows:
if rendered_blocks >= MAX_DOCX_PREVIEW_BLOCKS:
truncated = True
break
cell_values = [normalize_text(cell.text) for cell in row.cells]
located = source_range("\t".join(cell_values))
if located is None:
continue
_, start, end = located
preview_rows.append(
{
"cells": cell_values,
"source_start": start,
"source_end": end,
}
)
rendered_blocks += 1
if preview_rows:
blocks.append({"type": "table", "rows": preview_rows})
if truncated:
break
return {
"format": "docx",
"blocks": blocks,
"truncated": truncated,
}
def build_xlsx_preview(
raw: bytes,
*,
sheet_index: int = 0,
offset: int = 0,
limit: int = 100,
) -> dict[str, Any]:
"""按工作表分页返回 XLSX 的表头和记录网格。"""
if sheet_index < 0 or offset < 0:
raise ValueError("sheet_index and offset must be non-negative")
if limit < 1 or limit > MAX_XLSX_PREVIEW_ROWS:
raise ValueError(
f"XLSX preview limit must be in [1, {MAX_XLSX_PREVIEW_ROWS}]"
)
_validate_office_archive(raw, "xlsx")
merged_by_sheet, normalized_targets = _xlsx_sheet_merge_ranges(raw)
workbook_raw = (
_rewrite_xlsx_workbook_relationships(raw, normalized_targets)
if normalized_targets
else raw
)
try:
workbook = load_workbook(
io.BytesIO(workbook_raw),
read_only=True,
data_only=True,
keep_links=False,
)
except Exception as exc:
raise ValueError(f"invalid XLSX file: {exc}") from exc
try:
sheets = [
{
"index": index,
"name": worksheet.title,
"state": worksheet.sheet_state,
}
for index, worksheet in enumerate(workbook.worksheets)
]
if not sheets:
raise ValueError("XLSX workbook contains no worksheets")
if sheet_index >= len(sheets):
raise ValueError("XLSX worksheet index is out of range")
worksheet = workbook.worksheets[sheet_index]
reset_dimensions = getattr(worksheet, "reset_dimensions", None)
if callable(reset_dimensions):
reset_dimensions()
row_iterator = enumerate(worksheet.iter_rows(values_only=True), start=1)
buffered_rows: dict[int, tuple[Any, ...]] = {}
def normalized_values(row: tuple[Any, ...]) -> list[Any]:
values = list(row)
while values and values[-1] in {None, ""}:
values.pop()
if len(values) > _MAX_WORKBOOK_COLUMNS:
raise ValueError(
f"XLSX worksheet {worksheet.title!r} exceeds "
f"{_MAX_WORKBOOK_COLUMNS} columns"
)
return values
for row_number, row in row_iterator:
values = normalized_values(row)
if not values or all(value in {None, ""} for value in values):
continue
buffered_rows[row_number] = tuple(values)
if len(buffered_rows) >= _MAX_WORKBOOK_HEADER_SCAN_ROWS:
break
if not buffered_rows:
return {
"format": "xlsx",
"sheets": sheets,
"active_sheet": {
"index": sheet_index,
"name": worksheet.title,
"columns": [],
"rows": [],
"offset": offset,
"limit": limit,
"has_more": False,
},
}
_, header_end_row, headers = _infer_xlsx_header_region(
worksheet.title,
buffered_rows,
merged_by_sheet.get(worksheet.title, ()),
)
preview_rows: list[dict[str, Any]] = []
record_index = 0
has_more = False
def append_row(row_number: int, values: tuple[Any, ...] | list[Any]) -> bool:
nonlocal record_index, has_more
row_values = list(values)
if len(row_values) > len(headers):
raise ValueError(
f"XLSX worksheet {worksheet.title!r} has a row wider than its header"
)
row_values.extend([None] * (len(headers) - len(row_values)))
record = {
header: _normalize_spreadsheet_value(value)
for header, value in zip(headers, row_values, strict=True)
}
if not any(value not in {"", None} for value in record.values()):
return False
current_index = record_index
record_index += 1
if current_index < offset:
return False
if len(preview_rows) >= limit:
has_more = True
return True
preview_rows.append(
{
"row_number": row_number,
"record_index": current_index,
"values": [record[header] for header in headers],
"record": record,
}
)
return False
for row_number, values in buffered_rows.items():
if row_number > header_end_row and append_row(row_number, values):
break
else:
for row_number, row in row_iterator:
values = normalized_values(row)
if not values or all(value in {None, ""} for value in values):
continue
if append_row(row_number, values):
break
return {
"format": "xlsx",
"sheets": sheets,
"active_sheet": {
"index": sheet_index,
"name": worksheet.title,
"columns": headers,
"rows": preview_rows,
"offset": offset,
"limit": limit,
"has_more": has_more,
},
}
finally:
workbook.close()
__all__ = [
"MAX_DOCX_PREVIEW_BLOCKS",
"MAX_XLSX_PREVIEW_ROWS",
"build_docx_preview",
"build_xlsx_preview",
]

76
backend/app/modules/data_process/schema_cli.py
View File

@@ -1,76 +0,0 @@
"""数据处理运行表的显式检查与安装命令。"""
from __future__ import annotations
import argparse
from urllib.parse import urlsplit
from app.modules.data_process.store import DataProcessStore
REQUIRED_TASK_COLUMNS = (
"generation_run_id",
"results_confirmed",
"workflow_step",
"preview_status",
"preview_progress",
"preview_run_id",
"preview_failure_reason",
"preview_total_files",
"preview_completed_files",
)
def _target_label(database_url: str) -> str:
parsed = urlsplit(database_url)
database = parsed.path.strip("/") or "(unknown)"
return f"{parsed.hostname or '(unknown)'}:{parsed.port or 5432}/{database}"
def _schema_ready(store: DataProcessStore) -> bool:
with store.connect() as conn:
row = conn.execute(
"""
SELECT COUNT(*) = %s AS ready
FROM information_schema.columns
WHERE table_schema=current_schema()
AND table_name='data_process_tasks'
AND column_name = ANY(%s)
""",
(len(REQUIRED_TASK_COLUMNS), list(REQUIRED_TASK_COLUMNS)),
).fetchone()
return bool(row and row["ready"])
def main() -> int:
parser = argparse.ArgumentParser(
description="检查或显式安装数据处理运行表（不会由应用启动自动执行）"
)
action = parser.add_mutually_exclusive_group(required=True)
action.add_argument("--check", action="store_true", help="只读检查迁移是否已安装")
action.add_argument("--apply", action="store_true", help="执行 002 数据处理迁移")
parser.add_argument(
"--yes",
action="store_true",
help="确认允许修改 DATABASE_URL 指向的数据库；与 --apply 同时使用",
)
args = parser.parse_args()
store = DataProcessStore()
target = _target_label(store.database_url)
if args.check:
ready = _schema_ready(store)
print(f"数据处理 schema：{'已安装' if ready else '未安装'}；目标：{target}")
return 0 if ready else 1
if not args.yes:
parser.error("--apply 必须同时提供 --yes，确认修改目标数据库")
print(f"正在安装数据处理 schema；目标：{target}")
store.ensure_schema()
if not _schema_ready(store):
raise RuntimeError("迁移执行后仍未检测到 generation_run_id")
print("数据处理 schema 安装完成")
return 0
if __name__ == "__main__":
raise SystemExit(main())

564
backend/app/modules/data_process/storage.py
View File

@@ -1,564 +0,0 @@
"""数据处理原始源文件的受控本地对象存储。"""
from __future__ import annotations
import os
import re
import stat
import unicodedata
import uuid
from dataclasses import dataclass
from functools import lru_cache
from pathlib import Path, PurePosixPath
from typing import Iterable, Iterator
from urllib.parse import quote, unquote, urlsplit
class DataProcessStorageError(ValueError):
"""本地对象引用或文件系统状态不安全。"""
@dataclass(frozen=True, slots=True)
class StagedSourceObject:
"""尚未发布的原始文件；绝对路径仅在存储模块内部流转。"""
reference: str
_temporary_path: Path
_relative_path: PurePosixPath
def _default_storage_root() -> Path:
return Path(__file__).resolve().parents[3] / "storage" / "data-process"
def _configured_storage_root() -> Path:
configured = os.getenv("DATA_PROCESS_STORAGE_DIR", "").strip()
if not configured:
return _default_storage_root()
path = Path(configured).expanduser()
# 相对配置固定以 backend 目录为基准，
# 避免从不同 cwd 启动时写入不同位置。
return path if path.is_absolute() else Path(__file__).resolve().parents[3] / path
def _safe_component(value: str, label: str) -> str:
if not value or value in {".", ".."} or len(value) > 128:
raise DataProcessStorageError(f"invalid {label}")
if not value[0].isalnum() or any(
not (character.isalnum() or character in {"-", "_", "."})
for character in value
):
raise DataProcessStorageError(f"invalid {label}")
return value
def _safe_basename(value: str) -> str:
if not value or len(value.encode("utf-8")) > 255:
raise DataProcessStorageError("invalid source file name")
if value != Path(value).name or "/" in value or "\\" in value or "\x00" in value:
raise DataProcessStorageError("invalid source file name")
if value in {".", ".."} or any(
unicodedata.category(character).startswith("C") for character in value
):
raise DataProcessStorageError("invalid source file name")
return value
class LocalDataProcessStorage:
"""只允许访问配置根目录下的版本化原始文件。"""
def __init__(self, root: str | os.PathLike[str] | Path | None = None) -> None:
configured = Path(root) if root is not None else _configured_storage_root()
configured = configured.expanduser()
if configured.exists() and configured.is_symlink():
raise DataProcessStorageError("data process storage root must not be a symlink")
configured.mkdir(parents=True, exist_ok=True, mode=0o700)
self._root = configured.resolve(strict=True)
# StagedSourceObject 本身是普通 dataclass，不能只依赖其中的路径字段判断
# 来源；只接受由当前存储实例实际签发的对象，
# 避免调用方伪造暂存路径。
self._issued_staged_objects: dict[Path, StagedSourceObject] = {}
self._ensure_directory(self._root / ".staging")
@property
def root(self) -> Path:
"""仅供运维和测试检查；API 响应不得序列化该属性。"""
return self._root
def new_batch_id(self) -> str:
return f"batch-{uuid.uuid4().hex}"
def stage_bytes(
self,
*,
batch_id: str,
task_id: str,
source_file_id: str,
version: int,
name: str,
content: bytes,
) -> StagedSourceObject:
batch_id = _safe_component(batch_id, "batch id")
task_id = _safe_component(task_id, "task id")
source_file_id = _safe_component(source_file_id, "source file id")
if isinstance(version, bool) or not isinstance(version, int) or version < 1:
raise DataProcessStorageError("invalid source file version")
basename = _safe_basename(name)
if not isinstance(content, bytes):
raise TypeError("content must be bytes")
batch_directory = self._ensure_directory(self._root / ".staging" / batch_id)
temporary_path = batch_directory / f"{source_file_id}-{uuid.uuid4().hex}.tmp"
flags = os.O_CREAT | os.O_EXCL | os.O_WRONLY
if hasattr(os, "O_NOFOLLOW"):
flags |= os.O_NOFOLLOW
descriptor = os.open(temporary_path, flags, 0o600)
try:
with os.fdopen(descriptor, "wb", closefd=True) as stream:
stream.write(content)
stream.flush()
os.fsync(stream.fileno())
except Exception:
temporary_path.unlink(missing_ok=True)
raise
relative_path = PurePosixPath(
task_id,
source_file_id,
f"v{version}",
basename,
)
reference = (
"local://data-process/"
f"{task_id}/{source_file_id}/v{version}/{quote(basename, safe='')}"
)
staged = StagedSourceObject(reference, temporary_path, relative_path)
self._issued_staged_objects[temporary_path] = staged
return staged
def stage_copy(
self,
*,
batch_id: str,
source_reference: str,
expected_source_task_id: str,
expected_source_file_id: str,
task_id: str,
source_file_id: str,
version: int,
name: str,
) -> StagedSourceObject:
"""为不可变源对象创建独立目录项，不把大文件重新读入内存。"""
batch_id = _safe_component(batch_id, "batch id")
task_id = _safe_component(task_id, "task id")
source_file_id = _safe_component(source_file_id, "source file id")
if isinstance(version, bool) or not isinstance(version, int) or version < 1:
raise DataProcessStorageError("invalid source file version")
basename = _safe_basename(name)
source_relative = self._relative_from_reference(source_reference)
if source_relative is None:
raise DataProcessStorageError("original source object is not available")
self._assert_expected_owner(
source_relative,
expected_task_id=expected_source_task_id,
expected_source_file_id=expected_source_file_id,
)
descriptor, source_info = self._open_read_descriptor(source_relative)
os.close(descriptor)
batch_directory = self._ensure_directory(self._root / ".staging" / batch_id)
temporary_path = batch_directory / f"{source_file_id}-{uuid.uuid4().hex}.tmp"
source_path = self._path_for_relative(source_relative)
try:
os.link(source_path, temporary_path, follow_symlinks=False)
copy_info = temporary_path.lstat()
if (
not stat.S_ISREG(copy_info.st_mode)
or source_info.st_dev != copy_info.st_dev
or source_info.st_ino != copy_info.st_ino
):
raise DataProcessStorageError("source storage object changed while copying")
except Exception:
temporary_path.unlink(missing_ok=True)
raise
relative_path = PurePosixPath(
task_id,
source_file_id,
f"v{version}",
basename,
)
reference = (
"local://data-process/"
f"{task_id}/{source_file_id}/v{version}/{quote(basename, safe='')}"
)
staged = StagedSourceObject(reference, temporary_path, relative_path)
self._issued_staged_objects[temporary_path] = staged
return staged
def publish(self, objects: Iterable[StagedSourceObject]) -> None:
staged = list(objects)
published: list[StagedSourceObject] = []
try:
seen_temporary_paths: set[Path] = set()
for item in staged:
self._validate_staged_object(item, require_file=True)
if item._temporary_path in seen_temporary_paths:
raise DataProcessStorageError("duplicate staged source object")
seen_temporary_paths.add(item._temporary_path)
for item in staged:
final_path = self._path_for_relative(item._relative_path)
self._ensure_directory(final_path.parent)
if final_path.exists() or final_path.is_symlink():
raise DataProcessStorageError("source storage object already exists")
os.link(item._temporary_path, final_path, follow_symlinks=False)
published.append(item)
item._temporary_path.unlink()
self._fsync_directory(final_path.parent)
except Exception:
for item in reversed(published):
try:
self.delete(item.reference)
except Exception:
# 回滚必须尽量处理其余对象，并保留真正的发布异常。
pass
for item in staged:
try:
self.discard([item])
except Exception:
pass
raise
self.discard(staged)
def discard(self, objects: Iterable[StagedSourceObject]) -> None:
staged = list(objects)
for item in staged:
self._validate_staged_object(item, require_file=False)
batch_directories: set[Path] = set()
first_error: Exception | None = None
for item in staged:
temporary_path = item._temporary_path
try:
temporary_path.unlink(missing_ok=True)
except Exception as exc:
if first_error is None:
first_error = exc
else:
self._issued_staged_objects.pop(temporary_path, None)
batch_directories.add(temporary_path.parent)
for directory in batch_directories:
self._remove_empty_directory(directory)
if first_error is not None:
raise first_error
def read(self, reference: str) -> bytes | None:
"""读取 local 引用；旧 ``db://`` 对象返回 ``None`` 由数据库正文兜底。"""
relative_path = self._relative_from_reference(reference)
if relative_path is None:
return None
descriptor, _ = self._open_read_descriptor(relative_path)
with os.fdopen(descriptor, "rb", closefd=True) as stream:
return stream.read()
def file_size(
self,
reference: str,
*,
expected_task_id: str,
expected_source_file_id: str,
) -> int | None:
"""返回受控 local 对象大小；旧 ``db://`` 对象没有原始文件。"""
relative_path = self._relative_from_reference(reference)
if relative_path is None:
return None
self._assert_expected_owner(
relative_path,
expected_task_id=expected_task_id,
expected_source_file_id=expected_source_file_id,
)
descriptor, info = self._open_read_descriptor(relative_path)
os.close(descriptor)
return info.st_size
def iter_bytes(
self,
reference: str,
*,
expected_task_id: str,
expected_source_file_id: str,
expected_size: int,
start: int = 0,
length: int | None = None,
chunk_size: int = 256 * 1024,
) -> Iterator[bytes]:
"""按范围流式读取原始文件，避免 PDF 预览把大文件整体载入内存。"""
relative_path = self._relative_from_reference(reference)
if relative_path is None:
raise DataProcessStorageError("original source object is not available")
self._assert_expected_owner(
relative_path,
expected_task_id=expected_task_id,
expected_source_file_id=expected_source_file_id,
)
if start < 0 or expected_size < 0 or chunk_size < 1:
raise DataProcessStorageError("invalid source byte range")
descriptor, info = self._open_read_descriptor(relative_path)
if info.st_size != expected_size:
os.close(descriptor)
raise DataProcessStorageError("source object size does not match metadata")
remaining = expected_size - start if length is None else length
if remaining < 0 or start + remaining > expected_size:
os.close(descriptor)
raise DataProcessStorageError("invalid source byte range")
with os.fdopen(descriptor, "rb", closefd=True) as stream:
stream.seek(start)
while remaining:
chunk = stream.read(min(chunk_size, remaining))
if not chunk:
raise DataProcessStorageError("source object ended unexpectedly")
remaining -= len(chunk)
yield chunk
def validate_owner(
self,
reference: str,
*,
expected_task_id: str,
expected_source_file_id: str,
) -> bool:
"""校验 local 引用归属；旧 ``db://`` 引用无需文件系统处理。"""
relative_path = self._relative_from_reference(reference)
if relative_path is None:
return False
self._assert_expected_owner(
relative_path,
expected_task_id=expected_task_id,
expected_source_file_id=expected_source_file_id,
)
return True
def _open_read_descriptor(
self,
relative_path: PurePosixPath,
) -> tuple[int, os.stat_result]:
path = self._path_for_relative(relative_path)
self._assert_controlled_parent(path)
try:
before_open = path.lstat()
except FileNotFoundError as exc:
raise DataProcessStorageError("source storage object does not exist") from exc
if stat.S_ISLNK(before_open.st_mode) or not stat.S_ISREG(before_open.st_mode):
raise DataProcessStorageError("source storage object is not a regular file")
flags = os.O_RDONLY
if hasattr(os, "O_NOFOLLOW"):
flags |= os.O_NOFOLLOW
descriptor = os.open(path, flags)
after_open = os.fstat(descriptor)
if (
not stat.S_ISREG(after_open.st_mode)
or before_open.st_dev != after_open.st_dev
or before_open.st_ino != after_open.st_ino
):
os.close(descriptor)
raise DataProcessStorageError("source storage object changed while opening")
return descriptor, after_open
def delete(
self,
reference: str,
*,
expected_task_id: str | None = None,
expected_source_file_id: str | None = None,
) -> bool:
"""删除受控 local 对象；旧 ``db://`` 引用保持不变。"""
relative_path = self._relative_from_reference(reference)
if relative_path is None:
return False
if (expected_task_id is None) != (expected_source_file_id is None):
raise DataProcessStorageError("both expected storage owner fields are required")
if expected_task_id is not None and expected_source_file_id is not None:
self._assert_expected_owner(
relative_path,
expected_task_id=expected_task_id,
expected_source_file_id=expected_source_file_id,
)
path = self._path_for_relative(relative_path)
self._assert_controlled_parent(path)
try:
info = path.lstat()
except FileNotFoundError:
return False
if stat.S_ISLNK(info.st_mode) or not stat.S_ISREG(info.st_mode):
raise DataProcessStorageError("refusing to delete a non-regular storage object")
path.unlink()
self._fsync_directory(path.parent)
for directory in (path.parent, path.parent.parent, path.parent.parent.parent):
self._remove_empty_directory(directory)
return True
@staticmethod
def _assert_expected_owner(
relative_path: PurePosixPath,
*,
expected_task_id: str,
expected_source_file_id: str,
) -> None:
task_id = _safe_component(expected_task_id, "expected task id")
source_file_id = _safe_component(
expected_source_file_id,
"expected source file id",
)
if relative_path.parts[:2] != (task_id, source_file_id):
raise DataProcessStorageError("source storage object owner mismatch")
def _relative_from_reference(self, reference: str) -> PurePosixPath | None:
if reference.startswith("db://"):
return None
parsed = urlsplit(reference)
if parsed.scheme != "local" or parsed.netloc != "data-process":
raise DataProcessStorageError("unsupported source storage reference")
if parsed.query or parsed.fragment or "\\" in parsed.path:
raise DataProcessStorageError("unsafe source storage reference")
raw_parts = parsed.path.lstrip("/").split("/")
if len(raw_parts) != 4:
raise DataProcessStorageError("unsafe source storage reference")
if any(re.search(r"%(?![0-9A-Fa-f]{2})", part) for part in raw_parts):
raise DataProcessStorageError("unsafe source storage reference")
try:
decoded = [unquote(part, encoding="utf-8", errors="strict") for part in raw_parts]
except UnicodeDecodeError as exc:
raise DataProcessStorageError("unsafe source storage reference") from exc
if any("/" in part or "\\" in part for part in decoded):
raise DataProcessStorageError("unsafe source storage reference")
canonical_parts = [
quote(decoded[0], safe="-_."),
quote(decoded[1], safe="-_."),
quote(decoded[2], safe="-_."),
quote(decoded[3], safe=""),
]
if canonical_parts != raw_parts:
raise DataProcessStorageError("source storage reference is not canonical")
task_id = _safe_component(decoded[0], "task id")
source_file_id = _safe_component(decoded[1], "source file id")
version_component = decoded[2]
if not version_component.startswith("v") or not version_component[1:].isdigit():
raise DataProcessStorageError("invalid source file version")
version = int(version_component[1:])
if version < 1:
raise DataProcessStorageError("invalid source file version")
basename = _safe_basename(decoded[3])
return PurePosixPath(task_id, source_file_id, f"v{version}", basename)
def _path_for_relative(self, relative_path: PurePosixPath) -> Path:
if relative_path.is_absolute() or any(
part in {"", ".", ".."} for part in relative_path.parts
):
raise DataProcessStorageError("storage path escapes the configured root")
path = self._root.joinpath(*relative_path.parts)
self._assert_controlled_parent(path)
return path
def _validate_staged_object(
self,
item: StagedSourceObject,
*,
require_file: bool,
) -> None:
if not isinstance(item, StagedSourceObject):
raise DataProcessStorageError("invalid staged source object")
if self._issued_staged_objects.get(item._temporary_path) is not item:
raise DataProcessStorageError("staged source object was not issued by this storage")
expected_relative = self._relative_from_reference(item.reference)
if expected_relative is None or expected_relative != item._relative_path:
raise DataProcessStorageError("staged source object reference mismatch")
staging_root = self._root / ".staging"
try:
relative_temporary = item._temporary_path.relative_to(staging_root)
except ValueError as exc:
raise DataProcessStorageError("staged source object escapes staging") from exc
if len(relative_temporary.parts) != 2:
raise DataProcessStorageError("invalid staged source object path")
_safe_component(relative_temporary.parts[0], "batch id")
_safe_basename(relative_temporary.parts[1])
self._assert_controlled_parent(item._temporary_path)
try:
info = item._temporary_path.lstat()
except FileNotFoundError:
if require_file:
raise DataProcessStorageError("staged source object does not exist") from None
return
if stat.S_ISLNK(info.st_mode) or not stat.S_ISREG(info.st_mode):
raise DataProcessStorageError("staged source object is not a regular file")
def _ensure_directory(self, directory: Path) -> Path:
try:
relative = directory.relative_to(self._root)
except ValueError as exc:
raise DataProcessStorageError("storage path escapes the configured root") from exc
current = self._root
for component in relative.parts:
current = current / component
try:
current.mkdir(mode=0o700)
except FileExistsError:
pass
info = current.lstat()
if stat.S_ISLNK(info.st_mode) or not stat.S_ISDIR(info.st_mode):
raise DataProcessStorageError("storage path contains a symlink or non-directory")
return directory
def _assert_controlled_parent(self, path: Path) -> None:
try:
relative_parent = path.parent.relative_to(self._root)
except ValueError as exc:
raise DataProcessStorageError("storage path escapes the configured root") from exc
current = self._root
for component in relative_parent.parts:
current = current / component
if not current.exists():
continue
info = current.lstat()
if stat.S_ISLNK(info.st_mode) or not stat.S_ISDIR(info.st_mode):
raise DataProcessStorageError("storage path contains a symlink or non-directory")
@staticmethod
def _fsync_directory(directory: Path) -> None:
# Windows 不支持以 O_RDONLY 打开目录做 fsync，跳过即可。
# 数据完整性在 Linux 生产环境保障，Windows 开发环境忽略。
if os.name == "nt":
return
descriptor = os.open(directory, os.O_RDONLY)
try:
os.fsync(descriptor)
finally:
os.close(descriptor)
def _remove_empty_directory(self, directory: Path) -> None:
if directory in {self._root, self._root / ".staging"}:
return
self._assert_controlled_parent(directory / "placeholder")
try:
directory.rmdir()
except (FileNotFoundError, OSError):
return
@lru_cache
def get_data_process_storage() -> LocalDataProcessStorage:
return LocalDataProcessStorage()
__all__ = [
"DataProcessStorageError",
"LocalDataProcessStorage",
"StagedSourceObject",
"get_data_process_storage",
]

2869
backend/app/modules/data_process/store.py
View File

File diff suppressed because it is too large Load Diff

1
backend/app/modules/gpu/__init__.py
View File

@@ -1 +0,0 @@
"""GPU assignment management module."""

77
backend/app/modules/gpu/router.py
View File

@@ -1,77 +0,0 @@
"""GPU 算力分配管理路由。"""
from __future__ import annotations
from typing import Any
from fastapi import APIRouter, Body, Depends, Request
from app.api.v1.endpoints.platform import ok, fail
from app.core.auth import get_current_user, is_admin
from app.db.platform_store import get_platform_store
router = APIRouter(prefix="/compute", tags=["gpu-assignment"])
def _actor_id(request: Request) -> str | None:
auth = request.headers.get("Authorization", "")
token = auth.replace("Bearer ", "").strip()
if token.startswith("platform-token-"):
return token[len("platform-token-"):]
return None
@router.get("/gpu-assignments")
def list_assignments(current_user: dict = Depends(get_current_user)) -> dict[str, Any]:
"""查看全部分配关系（仅 admin）。"""
if not is_admin(current_user):
raise fail(403, "admin permission required")
return ok(get_platform_store().gpu_assignments())
@router.post("/gpu-assignments")
def assign_gpus(
payload: dict[str, Any] = Body(...),
request: Request = None,
current_user: dict = Depends(get_current_user),
) -> dict[str, Any]:
"""批量分配 GPU（仅 admin）。body: { assignments: [{ node_id, gpu_index, user_id }] }"""
if not is_admin(current_user):
raise fail(403, "admin permission required")
assignments = payload.get("assignments") or []
if not assignments:
raise fail(400, "assignments 不能为空")
actor = _actor_id(request) if request else None
result = get_platform_store().assign_gpus(assignments, assigned_by=actor)
get_platform_store().record_audit(
action="gpu.assign",
actor_id=actor,
target_type="gpu",
detail=f"count={len(assignments)}",
)
return ok(result)
@router.delete("/gpu-assignments/{assignment_id}")
def unassign_gpu(
assignment_id: str,
request: Request = None,
current_user: dict = Depends(get_current_user),
) -> dict[str, Any]:
"""撤销 GPU 分配（仅 admin）。"""
if not is_admin(current_user):
raise fail(403, "admin permission required")
get_platform_store().unassign_gpu(assignment_id)
actor = _actor_id(request) if request else None
get_platform_store().record_audit(
action="gpu.unassign",
actor_id=actor,
target_type="gpu",
target_id=assignment_id,
)
return ok({"deleted": assignment_id})
@router.get("/my-gpus")
def my_gpus(current_user: dict = Depends(get_current_user)) -> dict[str, Any]:
"""查看当前用户可用的 GPU 列表。"""
return ok(get_platform_store().gpu_assignments_for_user(current_user["id"]))

244
backend/app/modules/project/router.py
View File

@@ -1,244 +0,0 @@
from __future__ import annotations
from fastapi import APIRouter, Body, Depends, Request
from typing import Any
from app.api.v1.endpoints.platform import ok, fail
from app.core.auth import filter_accessible_resource_ids, get_current_user, has_resource_access, is_admin
from app.db.platform_store import get_platform_store
router = APIRouter(prefix="/projects", tags=["project"])
def _actor(request: Request) -> str | None:
auth = request.headers.get("Authorization", "")
token = auth.replace("Bearer ", "").strip()
return token or None
def _require_no_pending_approval(resource_type: str, resource_id: str) -> None:
"""第 4 周：写操作审批拦截——存在待审批实例时拒绝执行。"""
store = get_platform_store()
pending = [
i for i in store.approval_instances(status="pending")
if i["resource_type"] == resource_type and i["resource_id"] == resource_id
]
if pending:
raise fail(409, "存在待审批的变更，请先完成审批")
def _require_approval_or_admin(
resource_type: str,
resource_id: str,
current_user: dict[str, Any],
action_desc: str = "",
) -> dict[str, Any] | None:
"""高风险操作审批旁路：admin 直接放行，普通用户创建审批实例（code=202）。"""
if is_admin(current_user):
return None
store = get_platform_store()
instance = store.create_approval_instance({
"resource_type": resource_type,
"resource_id": resource_id,
"applicant_id": current_user.get("id"),
"template_id": None,
})
return {
"code": 202,
"message": f"操作已提交审批，等待管理员批准：{action_desc}",
"data": {"approval_required": True, "approval_id": instance["id"]},
}
@router.get("")
def list_projects(
tenant_id: str = "default",
status: str | None = None,
keyword: str | None = None,
current_user: dict = Depends(get_current_user),
) -> dict[str, Any]:
store = get_platform_store()
projects = store.projects(tenant_id=tenant_id, status=status, keyword=keyword)
# #1 ACL 过滤：admin 直接放行，普通用户只能看到自己被授权的项目
accessible_ids = set(
filter_accessible_resource_ids("project", [p["id"] for p in projects], current_user)
)
filtered = [p for p in projects if p["id"] in accessible_ids]
return ok(filtered)
@router.post("")
def create_project(payload: dict[str, Any] = Body(...), request: Request = None) -> dict[str, Any]:
store = get_platform_store()
proj = store.create_project(payload)
store.record_audit(
action="project.create",
actor_id=_actor(request) if request else None,
target_type="project",
target_id=proj["id"],
tenant_id=proj.get("tenant_id"),
detail=f"name={proj.get('name')}",
)
return ok(proj)
@router.get("/{project_id}")
def get_project(project_id: str, current_user: dict = Depends(get_current_user)) -> dict[str, Any]:
# #2 访问控制：普通用户无 read 权限则拒绝
if not has_resource_access("project", project_id, current_user, "read"):
raise fail(403, "no permission to access this project")
try:
return ok(get_platform_store().project(project_id))
except KeyError:
raise fail(404, "project not found")
@router.put("/{project_id}")
def update_project(
project_id: str,
payload: dict[str, Any] = Body(...),
request: Request = None,
current_user: dict = Depends(get_current_user),
) -> dict[str, Any]:
if not has_resource_access("project", project_id, current_user, "write"):
raise fail(403, "no permission to update this project")
store = get_platform_store()
try:
proj = store.update_project(project_id, payload)
except KeyError:
raise fail(404, "project not found")
store.record_audit(
action="project.update",
actor_id=_actor(request) if request else None,
target_type="project",
target_id=project_id,
tenant_id=proj.get("tenant_id"),
detail=f"fields={','.join(payload.keys())}",
)
return ok(proj)
@router.post("/{project_id}/archive")
def archive_project(
project_id: str,
request: Request = None,
current_user: dict = Depends(get_current_user),
) -> dict[str, Any]:
_require_no_pending_approval("project", project_id)
pending = _require_approval_or_admin("project", project_id, current_user, f"归档项目 {project_id}")
if pending:
return pending
store = get_platform_store()
try:
proj = store.archive_project(project_id)
except KeyError:
raise fail(404, "project not found")
store.record_audit(
action="project.archive",
actor_id=_actor(request) if request else None,
target_type="project",
target_id=project_id,
tenant_id=proj.get("tenant_id"),
)
return ok(proj)
@router.delete("/{project_id}")
def delete_project(
project_id: str,
request: Request = None,
current_user: dict = Depends(get_current_user),
) -> dict[str, Any]:
_require_no_pending_approval("project", project_id)
pending = _require_approval_or_admin("project", project_id, current_user, f"删除项目 {project_id}")
if pending:
return pending
store = get_platform_store()
store.delete_project(project_id)
store.record_audit(
action="project.delete",
actor_id=_actor(request) if request else None,
target_type="project",
target_id=project_id,
)
return ok(None)
@router.get("/{project_id}/members")
def list_members(project_id: str, current_user: dict = Depends(get_current_user)) -> dict[str, Any]:
if not has_resource_access("project", project_id, current_user, "read"):
raise fail(403, "no permission to access this project")
try:
return ok(get_platform_store().project_members(project_id))
except KeyError:
raise fail(404, "project not found")
@router.post("/{project_id}/members")
def add_member(
project_id: str,
payload: dict[str, Any] = Body(...),
request: Request = None,
current_user: dict = Depends(get_current_user),
) -> dict[str, Any]:
if not has_resource_access("project", project_id, current_user, "write"):
raise fail(403, "no permission to manage members of this project")
store = get_platform_store()
try:
member = store.add_project_member(project_id, payload)
except KeyError:
raise fail(404, "project not found")
store.record_audit(
action="project.member.add",
actor_id=_actor(request) if request else None,
target_type="project.member",
target_id=project_id,
detail=f"user_id={payload.get('user_id')},role={payload.get('role')}",
)
return ok(member)
@router.put("/{project_id}/members/{user_id}")
def update_member(
project_id: str,
user_id: str,
payload: dict[str, Any] = Body(...),
request: Request = None,
current_user: dict = Depends(get_current_user),
) -> dict[str, Any]:
if not has_resource_access("project", project_id, current_user, "write"):
raise fail(403, "no permission to manage members of this project")
store = get_platform_store()
try:
member = store.update_project_member_role(project_id, user_id, payload)
except KeyError:
raise fail(404, "project or member not found")
store.record_audit(
action="project.member.update",
actor_id=_actor(request) if request else None,
target_type="project.member",
target_id=project_id,
detail=f"user_id={user_id},role={payload.get('role')}",
)
return ok(member)
@router.delete("/{project_id}/members/{user_id}")
def remove_member(
project_id: str,
user_id: str,
request: Request = None,
current_user: dict = Depends(get_current_user),
) -> dict[str, Any]:
if not has_resource_access("project", project_id, current_user, "write"):
raise fail(403, "no permission to manage members of this project")
store = get_platform_store()
store.remove_project_member(project_id, user_id)
store.record_audit(
action="project.member.remove",
actor_id=_actor(request) if request else None,
target_type="project.member",
target_id=project_id,
detail=f"user_id={user_id}",
)
return ok(None)

1
backend/app/modules/resource/__init__.py
View File

@@ -1 +0,0 @@
"""Resource access control list (ACL) module."""

41
backend/app/modules/resource/router.py
View File

@@ -1,41 +0,0 @@
from __future__ import annotations
from fastapi import APIRouter, Body, Request
from typing import Any
from app.api.v1.endpoints.platform import ok, fail
from app.db.platform_store import get_platform_store
router = APIRouter(prefix="/resources", tags=["resource"])
def _actor(request: Request) -> str | None:
auth = request.headers.get("Authorization", "")
token = auth.replace("Bearer ", "").strip()
return token or None
@router.get("/{resource_type}/{resource_id}/acl")
def get_acl(resource_type: str, resource_id: str) -> dict[str, Any]:
"""查询资源 ACL，返回按主体分组的权限列表。"""
return ok(get_platform_store().resource_acl(resource_type, resource_id))
@router.put("/{resource_type}/{resource_id}/acl")
def set_acl(
resource_type: str,
resource_id: str,
payload: dict[str, Any] = Body(...),
request: Request = None,
) -> dict[str, Any]:
"""设置资源 ACL，body: { entries: [{ subject_type, subject_id, permissions: [] }] }"""
entries = payload.get("entries") or []
result = get_platform_store().set_resource_acl(resource_type, resource_id, entries)
get_platform_store().record_audit(
action="resource.acl.set",
actor_id=_actor(request) if request else None,
target_type=resource_type,
target_id=resource_id,
detail=f"entries={len(entries)}",
)
return ok(result)

75
backend/app/modules/retention/router.py
View File

@@ -1,75 +0,0 @@
from __future__ import annotations
from fastapi import APIRouter, Body, Request
from typing import Any
from app.api.v1.endpoints.platform import ok, fail
from app.db.platform_store import get_platform_store
router = APIRouter(prefix="/retention-policies", tags=["retention"])
def _actor(request: Request) -> str | None:
auth = request.headers.get("Authorization", "")
token = auth.replace("Bearer ", "").strip()
return token or None
@router.get("")
def list_policies() -> dict[str, Any]:
return ok(get_platform_store().retention_policies())
@router.post("")
def create_policy(payload: dict[str, Any] = Body(...), request: Request = None) -> dict[str, Any]:
if not payload.get("name"):
raise fail(400, "name 必填")
policy = get_platform_store().create_retention_policy(payload)
get_platform_store().record_audit(
action="retention.create",
actor_id=_actor(request) if request else None,
target_type="retention_policy",
target_id=policy["id"],
detail=f"name={policy.get('name')}",
)
return ok(policy)
@router.get("/{policy_id}")
def get_policy(policy_id: str) -> dict[str, Any]:
try:
return ok(get_platform_store().retention_policy(policy_id))
except KeyError:
raise fail(404, "retention policy not found")
@router.put("/{policy_id}")
def update_policy(
policy_id: str, payload: dict[str, Any] = Body(...), request: Request = None
) -> dict[str, Any]:
store = get_platform_store()
try:
policy = store.update_retention_policy(policy_id, payload)
except KeyError:
raise fail(404, "retention policy not found")
store.record_audit(
action="retention.update",
actor_id=_actor(request) if request else None,
target_type="retention_policy",
target_id=policy_id,
detail=f"fields={','.join(payload.keys())}",
)
return ok(policy)
@router.delete("/{policy_id}")
def delete_policy(policy_id: str, request: Request = None) -> dict[str, Any]:
store = get_platform_store()
store.delete_retention_policy(policy_id)
store.record_audit(
action="retention.delete",
actor_id=_actor(request) if request else None,
target_type="retention_policy",
target_id=policy_id,
)
return ok({"deleted": policy_id})

115
backend/app/modules/system/router.py
View File

@@ -1,115 +0,0 @@
from __future__ import annotations
from fastapi import APIRouter, Body, Query, Request
from fastapi.responses import StreamingResponse
from app.db.platform_store import ALL_PERMISSIONS, get_platform_store
router = APIRouter(prefix="/system", tags=["system"])
@router.post("/audit/visit")
def record_visit(payload: dict = Body(...), request: Request = None) -> dict:
"""记录用户访问业务模块的行为，用于看板用户操作分布统计。"""
action = str(payload.get("action") or payload.get("module") or "").strip()
if not action:
return {"code": 0, "message": "ok", "data": {"recorded": False}}
actor_id = ""
if request is not None:
auth = request.headers.get("Authorization", "")
token = auth.replace("Bearer ", "").strip()
if token.startswith("platform-token-"):
actor_id = token[len("platform-token-"):]
get_platform_store().record_audit(
action=action,
actor_id=actor_id or None,
target_type="module",
target_id=action,
detail=str(payload.get("detail") or ""),
)
return {"code": 0, "message": "ok", "data": {"recorded": True}}
@router.get("/permissions/codes")
def permission_codes() -> dict:
"""返回平台权限码清单（权限码接口）。"""
return {"code": 0, "message": "ok", "data": {"codes": ALL_PERMISSIONS}}
@router.get("/permissions")
def permissions_overview() -> dict:
"""返回权限码清单与角色定义。"""
store = get_platform_store()
return {
"code": 0,
"message": "ok",
"data": {"codes": ALL_PERMISSIONS, "roles": store.roles()},
}
@router.get("/audit-logs")
def audit_logs(
tenant_id: str | None = Query(default=None, description="租户 ID"),
project_id: str | None = Query(default=None, description="项目 ID"),
actor_id: str | None = Query(default=None, description="操作人 ID"),
action: str | None = Query(default=None, description="动作类型"),
target_type: str | None = Query(default=None, description="目标类型"),
start_time: str | None = Query(default=None, description="ISO8601 起始时间"),
end_time: str | None = Query(default=None, description="ISO8601 结束时间"),
limit: int = Query(default=50, ge=1, le=200),
offset: int = Query(default=0, ge=0),
) -> dict:
"""审计日志查询：按租户/项目/操作人/动作/目标类型/时间范围分页过滤。"""
store = get_platform_store()
result = store.audit_logs(
tenant_id=tenant_id,
project_id=project_id,
actor_id=actor_id,
action=action,
target_type=target_type,
start_time=start_time,
end_time=end_time,
limit=limit,
offset=offset,
)
return {"code": 0, "message": "ok", "data": result}
@router.get("/audit-logs/export")
def audit_logs_export(
tenant_id: str | None = Query(default=None, description="租户 ID"),
project_id: str | None = Query(default=None, description="项目 ID"),
actor_id: str | None = Query(default=None, description="操作人 ID"),
action: str | None = Query(default=None, description="动作类型"),
target_type: str | None = Query(default=None, description="目标类型"),
start_time: str | None = Query(default=None, description="ISO8601 起始时间"),
end_time: str | None = Query(default=None, description="ISO8601 结束时间"),
) -> StreamingResponse:
"""审计日志导出：返回 CSV 流，与应用查询相同的过滤条件。"""
store = get_platform_store()
result = store.audit_logs(
tenant_id=tenant_id,
project_id=project_id,
actor_id=actor_id,
action=action,
target_type=target_type,
start_time=start_time,
end_time=end_time,
limit=10000,
offset=0,
)
items = result["items"]
columns = ["time", "tenant_id", "project_id", "actor_id", "action", "target_type", "target_id", "detail", "client_ip"]
header = ",".join(columns) + "\n"
def iter_rows():
yield header
for row in items:
yield ",".join(f'"{str(row.get(c, "") or "")}"' for c in columns) + "\n"
return StreamingResponse(
iter_rows(),
media_type="text/csv",
headers={"Content-Disposition": "attachment; filename=audit_logs.csv"},
)

116
backend/app/modules/tenant/router.py
View File

@@ -1,116 +0,0 @@
from __future__ import annotations
from fastapi import APIRouter, Body, Request
from typing import Any
from app.api.v1.endpoints.platform import ok, fail
from app.db.platform_store import get_platform_store
router = APIRouter(prefix="/tenants", tags=["tenant"])
def _actor(request: Request) -> str | None:
auth = request.headers.get("Authorization", "")
token = auth.replace("Bearer ", "").strip()
return token or None
@router.get("")
def list_tenants() -> dict[str, Any]:
return ok(get_platform_store().tenants())
@router.post("")
def create_tenant(payload: dict[str, Any] = Body(...), request: Request = None) -> dict[str, Any]:
store = get_platform_store()
try:
tenant = store.create_tenant(payload)
except KeyError as e:
raise fail(400, f"missing field: {e}")
store.record_audit(
action="tenant.create",
actor_id=_actor(request) if request else None,
target_type="tenant",
target_id=tenant["id"],
tenant_id=tenant["id"],
detail=f"name={tenant.get('name')}",
)
return ok(tenant)
@router.get("/{tenant_id}")
def get_tenant(tenant_id: str) -> dict[str, Any]:
try:
return ok(get_platform_store().tenant(tenant_id))
except KeyError:
raise fail(404, "tenant not found")
@router.put("/{tenant_id}")
def update_tenant(tenant_id: str, payload: dict[str, Any] = Body(...), request: Request = None) -> dict[str, Any]:
store = get_platform_store()
try:
tenant = store.update_tenant(tenant_id, payload)
except KeyError:
raise fail(404, "tenant not found")
store.record_audit(
action="tenant.update",
actor_id=_actor(request) if request else None,
target_type="tenant",
target_id=tenant_id,
tenant_id=tenant_id,
detail=f"fields={','.join(payload.keys())}",
)
return ok(tenant)
@router.put("/{tenant_id}/quota")
def set_quota(tenant_id: str, payload: dict[str, Any] = Body(...), request: Request = None) -> dict[str, Any]:
store = get_platform_store()
try:
tenant = store.set_tenant_quota(tenant_id, payload.get("quota", {}))
except KeyError:
raise fail(404, "tenant not found")
store.record_audit(
action="tenant.quota.set",
actor_id=_actor(request) if request else None,
target_type="tenant",
target_id=tenant_id,
tenant_id=tenant_id,
)
return ok(tenant)
@router.put("/{tenant_id}/retention-policy")
def set_retention(tenant_id: str, payload: dict[str, Any] = Body(...), request: Request = None) -> dict[str, Any]:
store = get_platform_store()
try:
tenant = store.set_tenant_retention(tenant_id, payload.get("retention_policy_id"))
except KeyError:
raise fail(404, "tenant not found")
store.record_audit(
action="tenant.retention.set",
actor_id=_actor(request) if request else None,
target_type="tenant",
target_id=tenant_id,
tenant_id=tenant_id,
)
return ok(tenant)
@router.delete("/{tenant_id}")
def delete_tenant(tenant_id: str, request: Request = None) -> dict[str, Any]:
store = get_platform_store()
try:
tenant = store.delete_tenant(tenant_id)
except KeyError:
raise fail(404, "tenant not found")
store.record_audit(
action="tenant.delete",
actor_id=_actor(request) if request else None,
target_type="tenant",
target_id=tenant_id,
tenant_id=tenant_id,
detail=f"name={tenant.get('name')}",
)
return ok(tenant)

417
backend/app/schemas/data_process.py
View File

@@ -1,417 +0,0 @@
from __future__ import annotations
from enum import StrEnum
from typing import Any, Literal
from urllib.parse import parse_qs, urlsplit
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
from app.modules.data_process.constants import MAX_QA_PAIRS_PER_ITEM
def _config_value(config: dict[str, Any], snake_name: str, camel_name: str, default: Any) -> Any:
if snake_name in config:
return config[snake_name]
return config.get(camel_name, default)
def _validate_process_config(config: dict[str, Any]) -> None:
output_type = _config_value(config, "output_type", "outputType", "standard")
if output_type not in {"standard", "reasoning", "dpo"}:
raise ValueError("output_type must be one of: standard, reasoning, dpo")
source_mode = _config_value(config, "source_mode", "sourceMode", "local")
if source_mode not in {"local", "external"}:
raise ValueError("source_mode must be one of: local, external")
external_source = _config_value(config, "external_source", "externalSource", None)
if external_source is not None:
if not isinstance(external_source, dict):
raise ValueError("external_source must be an object")
if any(
key.lower() in {"password", "secret", "token", "api_key"}
for key in external_source
):
raise ValueError("external_source must not persist credentials")
external_url = str(external_source.get("url") or "").strip()
if external_url:
parsed_external_url = urlsplit(external_url)
sensitive_query_keys = {"password", "secret", "token", "api_key", "user", "username"}
if parsed_external_url.username or parsed_external_url.password or (
set(parse_qs(parsed_external_url.query)) & sensitive_query_keys
):
raise ValueError("external_source URL must not contain credentials")
chunk_method = _config_value(config, "chunk_method", "chunkMethod", "layout_hybrid")
if not isinstance(chunk_method, str) or chunk_method not in {
"layout_hybrid",
"semantic",
"fixed",
}:
raise ValueError("chunk_method must be one of: layout_hybrid, semantic, fixed")
semantic_percentile = _config_value(
config,
"semantic_breakpoint_percentile",
"semanticBreakpointPercentile",
95,
)
if (
isinstance(semantic_percentile, bool)
or not isinstance(semantic_percentile, int)
or not 1 <= semantic_percentile <= 99
):
raise ValueError("semantic_breakpoint_percentile must be an integer in [1, 99]")
split = _config_value(config, "dataset_split", "datasetSplit", None)
if split is not None:
if not isinstance(split, dict) or set(split) != {"train", "validation", "test"}:
raise ValueError("dataset_split must contain train, validation and test")
values = list(split.values())
if any(isinstance(value, bool) or not isinstance(value, int) for value in values):
raise ValueError("dataset_split values must be integers")
if any(value < 0 or value > 100 for value in values) or sum(values) != 100:
raise ValueError("dataset_split values must be in [0, 100] and total 100")
chunk_fields = {
"chunk_size",
"chunkSize",
"chunk_overlap",
"chunkOverlap",
"min_chunk_size",
"minChunkSize",
}
if chunk_fields.intersection(config):
chunk_size = _config_value(config, "chunk_size", "chunkSize", 800)
overlap = _config_value(config, "chunk_overlap", "chunkOverlap", 100)
minimum = _config_value(config, "min_chunk_size", "minChunkSize", 100)
if any(
isinstance(value, bool) or not isinstance(value, int)
for value in (chunk_size, overlap, minimum)
):
raise ValueError("chunk_size, chunk_overlap and min_chunk_size must be integers")
if not 16 <= chunk_size <= 32_768:
raise ValueError("chunk_size must be in [16, 32768]")
if overlap < 0 or overlap >= chunk_size:
raise ValueError("chunk_overlap must be in [0, chunk_size)")
if minimum <= 0 or minimum > chunk_size or overlap + minimum > chunk_size:
raise ValueError("min_chunk_size and chunk_overlap exceed chunk_size")
temperature = _config_value(config, "temperature", "temperature", None)
if temperature is not None:
if isinstance(temperature, bool) or not isinstance(temperature, (int, float)):
raise ValueError("temperature must be a number")
if not 0 <= float(temperature) <= 2:
raise ValueError("temperature must be in [0, 2]")
max_tokens = _config_value(config, "max_tokens", "maxTokens", None)
if max_tokens is not None:
if isinstance(max_tokens, bool) or not isinstance(max_tokens, int):
raise ValueError("max_tokens must be an integer")
if not 1 <= max_tokens <= 32_768:
raise ValueError("max_tokens must be in [1, 32768]")
for snake_name, camel_name in (
("qa_pairs_per_row", "qaPairsPerRow"),
("qa_pairs_per_chunk", "qaPairsPerChunk"),
):
pairs = _config_value(config, snake_name, camel_name, None)
if pairs is None:
continue
if (
isinstance(pairs, bool)
or not isinstance(pairs, int)
or not 1 <= pairs <= MAX_QA_PAIRS_PER_ITEM
):
raise ValueError(
f"{snake_name} must be an integer in [1, {MAX_QA_PAIRS_PER_ITEM}]"
)
class DataProcessStatus(StrEnum):
pending = "pending"
running = "running"
completed = "completed"
failed = "failed"
stopped = "stopped"
class DataProcessWorkflowStep(StrEnum):
create = "create"
model = "model"
upload = "upload"
preview = "preview"
generate = "generate"
results = "results"
class DataProcessPreviewStatus(StrEnum):
idle = "idle"
queued = "queued"
running = "running"
completed = "completed"
failed = "failed"
cancelled = "cancelled"
class ProcessType(StrEnum):
structured = "structured"
unstructured = "unstructured"
external = "external"
class DataProcessTaskCreate(BaseModel):
model_config = ConfigDict(extra="forbid")
name: str = Field(min_length=1, max_length=150)
description: str = ""
process_type: ProcessType
source_dataset_id: str | None = None
config: dict[str, Any] = Field(default_factory=dict)
@field_validator("name")
@classmethod
def normalize_name(cls, value: str) -> str:
value = value.strip()
if not value:
raise ValueError("task name cannot be empty")
return value
@model_validator(mode="after")
def validate_config(self) -> "DataProcessTaskCreate":
_validate_process_config(self.config)
return self
class DataProcessTaskUpdate(BaseModel):
model_config = ConfigDict(extra="forbid")
name: str | None = Field(default=None, min_length=1, max_length=150)
description: str | None = None
process_type: ProcessType | None = None
source_dataset_id: str | None = None
config: dict[str, Any] | None = None
@field_validator("name")
@classmethod
def normalize_name(cls, value: str | None) -> str | None:
if value is None:
return None
value = value.strip()
if not value:
raise ValueError("task name cannot be empty")
return value
@model_validator(mode="after")
def validate_config(self) -> "DataProcessTaskUpdate":
if self.config is not None:
_validate_process_config(self.config)
return self
class DataProcessWorkflowStepUpdate(BaseModel):
"""仅保存创建向导位置，不修改配置或使下游产物失效。"""
model_config = ConfigDict(extra="forbid")
workflow_step: DataProcessWorkflowStep
class DataProcessRegenerateRequest(BaseModel):
"""以一份完整配置准备任务重新生成。
``expected_updated_at`` 用于防止详情页的旧快照覆盖其他人刚刚
保存的配置。重新生成不允许改变处理类型，避免旧源文件在新解析
规则下被静默误用。
"""
model_config = ConfigDict(extra="forbid")
name: str = Field(min_length=1, max_length=150)
description: str
process_type: ProcessType
config: dict[str, Any]
expected_updated_at: str = Field(min_length=1)
@field_validator("name")
@classmethod
def normalize_name(cls, value: str) -> str:
value = value.strip()
if not value:
raise ValueError("task name cannot be empty")
return value
@model_validator(mode="after")
def validate_config(self) -> "DataProcessRegenerateRequest":
_validate_process_config(self.config)
return self
class DataProcessRepeatRequest(BaseModel):
"""按已确认任务的完整快照创建一批独立的新生成结果。"""
model_config = ConfigDict(extra="forbid")
expected_updated_at: str = Field(min_length=1)
request_id: str = Field(
min_length=8,
max_length=80,
pattern=r"^[A-Za-z0-9_-]+$",
)
class PreviewBuildRequest(BaseModel):
model_config = ConfigDict(extra="forbid")
replace_existing: Literal[True] = True
source_file_ids: list[str] | None = None
source_file_id: str | None = None
@model_validator(mode="after")
def validate_source_file_selection(self) -> "PreviewBuildRequest":
if self.source_file_ids is not None and self.source_file_id is not None:
raise ValueError("source_file_id and source_file_ids cannot be used together")
values = self.source_file_ids
if values is None and self.source_file_id is not None:
values = [self.source_file_id]
if values is None:
return self
normalized = list(dict.fromkeys(str(value).strip() for value in values))
if not normalized or any(not value for value in normalized):
raise ValueError("at least one non-empty source file id is required")
self.source_file_ids = normalized
self.source_file_id = None
return self
class PreviewItemCreate(BaseModel):
model_config = ConfigDict(extra="forbid")
source_file_id: str | None = None
original_content: str = ""
edited_content: str = ""
source_start: int | None = Field(default=None, ge=0)
source_end: int | None = Field(default=None, ge=0)
source_start_line: int | None = Field(default=None, ge=1)
source_end_line: int | None = Field(default=None, ge=1)
@model_validator(mode="after")
def validate_ranges(self) -> "PreviewItemCreate":
if self.source_start is not None and self.source_end is not None:
if self.source_end < self.source_start:
raise ValueError("source_end must be greater than or equal to source_start")
if self.source_start_line is not None and self.source_end_line is not None:
if self.source_end_line < self.source_start_line:
raise ValueError(
"source_end_line must be greater than or equal to source_start_line"
)
return self
class PreviewItemUpdate(BaseModel):
model_config = ConfigDict(extra="forbid")
edited_content: str
expected_updated_at: str | None = None
class GenerateRequest(BaseModel):
model_config = ConfigDict(extra="forbid")
replace_existing: Literal[True] = True
class ExternalSourceRequest(BaseModel):
model_config = ConfigDict(extra="forbid")
type: str = Field(min_length=1, max_length=30)
url: str = Field(min_length=1, max_length=2048)
auth_mode: Literal["none", "basic"] = "none"
username: str | None = Field(default=None, max_length=150)
password: str | None = Field(default=None, max_length=500)
limit: int = Field(default=1000, ge=1, le=100_000)
connect_timeout_seconds: int = Field(default=5, ge=1, le=30)
statement_timeout_seconds: int = Field(default=30, ge=1, le=300)
ssl_mode: Literal["disable", "prefer", "require", "verify-ca", "verify-full"] = "prefer"
class ExternalPullRequest(ExternalSourceRequest):
query: str | None = Field(default=None, max_length=20_000)
file_name: str = Field(default="external-data.jsonl", min_length=1, max_length=255)
@field_validator("file_name")
@classmethod
def validate_file_name(cls, value: str) -> str:
name = value.strip()
if not name.lower().endswith((".jsonl", ".ndjson")):
raise ValueError("external pull file_name must end with .jsonl or .ndjson")
return name
class ResultUpdate(BaseModel):
model_config = ConfigDict(extra="forbid")
instruction: str | None = None
input: str | None = None
output: str | None = None
chosen: str | None = None
rejected: str | None = None
expected_updated_at: str | None = None
class ResultRegenerateRequest(BaseModel):
model_config = ConfigDict(extra="forbid")
expected_updated_at: str = Field(min_length=1, max_length=100)
class ResultBatchRegenerateItem(BaseModel):
model_config = ConfigDict(extra="forbid")
result_id: str = Field(min_length=1, max_length=100)
expected_updated_at: str = Field(min_length=1, max_length=100)
class ResultBatchRegenerateRequest(BaseModel):
model_config = ConfigDict(extra="forbid")
items: list[ResultBatchRegenerateItem] = Field(min_length=1, max_length=100)
@model_validator(mode="after")
def validate_unique_results(self) -> "ResultBatchRegenerateRequest":
result_ids = [item.result_id for item in self.items]
if len(result_ids) != len(set(result_ids)):
raise ValueError("result_id values must be unique")
return self
class DatasetSplit(BaseModel):
model_config = ConfigDict(extra="forbid")
train: int = Field(default=80, ge=0, le=100)
validation: int = Field(default=10, ge=0, le=100)
test: int = Field(default=10, ge=0, le=100)
@model_validator(mode="after")
def validate_total(self) -> "DatasetSplit":
if self.train + self.validation + self.test != 100:
raise ValueError("dataset split must total 100")
return self
class PublishRequest(BaseModel):
model_config = ConfigDict(extra="forbid")
dataset_name: str = Field(min_length=1, max_length=150)
dataset_type: Literal["train", "test", "eval", "val", "other"] = "train"
storage_type: Literal["local"] = "local"
split: DatasetSplit = Field(default_factory=DatasetSplit)
format: Literal["alpaca_jsonl", "jsonl", "dpo"] = "alpaca_jsonl"
description: str = ""
@field_validator("dataset_name")
@classmethod
def normalize_dataset_name(cls, value: str) -> str:
value = value.strip()
if not value:
raise ValueError("dataset name cannot be empty")
return value

31
backend/app/workers/compute_poller.py
View File

@@ -1,31 +0,0 @@
from __future__ import annotations
import asyncio
from app.core.config import get_settings
from app.core.logging import get_logger
from app.modules.compute_gateway.sync import poll_compute_jobs_once
logger = get_logger(__name__)
async def run_compute_poller() -> None:
settings = get_settings()
if settings.compute_mode == "simulator" or settings.compute_status_sync_mode != "polling":
logger.info("compute poller disabled", extra={"compute_mode": settings.compute_mode})
return
interval = max(3, settings.compute_poll_interval_seconds)
logger.info("compute poller started", extra={"interval_seconds": interval})
while True:
try:
result = await poll_compute_jobs_once()
if result["synced"] or result["failed"]:
logger.info("compute jobs polled", extra={"result": result})
except asyncio.CancelledError:
logger.info("compute poller stopped")
raise
except Exception as exc: # noqa: BLE001 - keep background polling alive
logger.exception("compute poller failed", extra={"error": str(exc)})
await asyncio.sleep(interval)

9
backend/pyproject.toml
View File

@@ -10,21 +10,12 @@ dependencies = [
"pydantic>=2.7.0",
"sqlalchemy>=2.0.30",
"psycopg[binary]>=3.2.1",
"psycopg-pool>=3.2.1",
"alembic>=1.13.1",
"redis>=5.0.4",
"httpx>=0.27.0",
"PyJWT>=2.8.0",
"passlib[bcrypt]>=1.7.4",
"python-dotenv>=1.0.1",
"pypdf[crypto]>=5.0.0",
"python-docx>=1.1.2",
"openpyxl>=3.1.5",
"python-pptx>=1.0.2",
"llama-index-core==0.14.23",
"llama-index-embeddings-huggingface==0.6.1",
"docling==2.115.0",
"tiktoken>=0.7.0",
]
[project.optional-dependencies]

13
backend/requirements.txt
View File

@@ -4,22 +4,9 @@ python-multipart>=0.0.9
pydantic>=2.7.0
sqlalchemy>=2.0.30
psycopg[binary]>=3.2.1
psycopg-pool>=3.2.1
alembic>=1.13.1
redis>=5.0.4
httpx>=0.27.0
PyJWT>=2.8.0
passlib[bcrypt]>=1.7.4
python-dotenv>=1.0.1
pypdf[crypto]>=5.0.0
python-docx>=1.1.2
openpyxl>=3.1.5
python-pptx>=1.0.2
llama-index-core==0.14.23
llama-index-embeddings-huggingface==0.6.1
docling==2.115.0
tiktoken>=0.7.0
# 测试与代码检查
pytest>=8.2.0
ruff>=0.5.0

276
backend/tests/test_compare_inference_async.py
View File

@@ -1,276 +0,0 @@
"""
模型推理异步加载改造的单元测试。
覆盖：
- model_compare_load：异步派发，立即返回 starting + 节点信息（不等待加载完成）
- model_compare_delete：先删记录，卸载失败也不阻塞删除
- reconcile_inference_loads：starting -> ready/error/idle/不可达的状态迁移与封顶
- _unload_from_compute_node：任务感知，只命中记录中的节点
"""
from __future__ import annotations
import asyncio
from types import SimpleNamespace
from typing import Any
from app.api.v1.endpoints.platform import model_compare_delete, model_compare_load
import app.api.v1.endpoints.platform as platform
from app.modules.compute_gateway.client import ComputeNodeClient
from app.modules.compute_gateway.sync import MAX_STARTING_ATTEMPTS, reconcile_inference_loads
class FakeInferenceStore:
"""内存 store，仅实现推理加载/对账用到的接口。"""
def __init__(self, tasks: list[dict[str, Any]] | None = None, nodes: list[dict[str, Any]] | None = None) -> None:
self._tasks: dict[str, dict[str, Any]] = {t["id"]: dict(t) for t in (tasks or [])}
self._nodes = nodes or []
self._inference_nodes: set[str] = set()
def compare_task(self, task_id: str) -> dict[str, Any]:
if task_id not in self._tasks:
raise KeyError(task_id)
return dict(self._tasks[task_id])
def compare_tasks(self) -> list[dict[str, Any]]:
return [dict(t) for t in self._tasks.values()]
def update_compare_task(self, task_id: str, payload: dict[str, Any]) -> dict[str, Any]:
current = self._tasks[task_id]
merged = {**current, **payload, "id": task_id}
self._tasks[task_id] = merged
return dict(merged)
def delete_compare_task(self, task_id: str) -> None:
self._tasks.pop(task_id, None)
def compute_nodes(self) -> list[dict[str, Any]]:
return [dict(n) for n in self._nodes]
def model(self, model_id: str) -> dict[str, Any]:
raise KeyError(model_id)
def trained_models(self) -> list[dict[str, Any]]:
return []
def mark_inference_loaded(self, node_id: str) -> None:
self._inference_nodes.add(node_id)
def mark_inference_unloaded(self, node_id: str) -> None:
self._inference_nodes.discard(node_id)
def is_inference_loaded(self, node_id: str) -> bool:
return node_id in self._inference_nodes
def _node(node_id: str, code: str = "") -> dict[str, Any]:
return {
"id": node_id,
"code": code or node_id,
"name": code or node_id,
"api_base_url": f"http://{code or node_id}:19100",
"enabled": True,
"scheduler_status": "online",
}
def _task(task_id: str, *, node_id: str | None = None, load_status: dict[str, Any] | None = None) -> dict[str, Any]:
return {
"id": task_id,
"name": f"task-{task_id}",
"status": "pending",
"models": [
{"model_id": "m_1", "model_name": "qwen", "model_path": "/models/qwen", "node_id": node_id}
],
"load_status": load_status or {"loaded_models": []},
}
async def _fake_inference_load(self, payload: dict[str, Any]) -> dict[str, Any]:
return {"loaded": False, "status": "loading", "request_id": "req-1"}
async def _fake_inference_unload(self) -> dict[str, Any]:
return {"unloaded": True, "status": "idle"}
def _patch_store(monkeypatch, store: FakeInferenceStore) -> None:
monkeypatch.setattr(platform, "get_platform_store", lambda: store)
monkeypatch.setattr(platform, "get_settings", lambda: SimpleNamespace(compute_mode="real"))
def test_select_eval_node_prefers_model_node(monkeypatch) -> None:
from app.api.v1.endpoints.platform import _select_eval_node
store = FakeInferenceStore(nodes=[_node("n1"), _node("n2")])
# 指定模型所在节点时优先返回该节点
assert _select_eval_node(store, "n2")["id"] == "n2"
# 无指定节点时回退到第一个在线节点
assert _select_eval_node(store, None)["id"] == "n1"
def test_select_eval_node_returns_none_when_model_node_offline(monkeypatch) -> None:
from app.api.v1.endpoints.platform import _select_eval_node
nodes = [_node("n1"), _node("n2")]
nodes[1]["enabled"] = False
store = FakeInferenceStore(nodes=nodes)
# 模型所在节点不可用 → 明确失败，不派发到其它节点
assert _select_eval_node(store, "n2") is None
# 无指定节点时仍回退第一个在线节点
assert _select_eval_node(store, None)["id"] == "n1"
def test_model_compare_load_dispatches_and_returns_starting(monkeypatch) -> None:
store = FakeInferenceStore(tasks=[_task("t1", node_id="n1")], nodes=[_node("n1")])
_patch_store(monkeypatch, store)
monkeypatch.setattr(ComputeNodeClient, "inference_load", _fake_inference_load)
result = asyncio.run(model_compare_load("t1"))
assert result["code"] == 0
updated = result["data"]
assert updated["status"] == "starting"
items = updated["load_status"]["loaded_models"]
assert items[0]["status"] == "starting"
assert items[0]["node_id"] == "n1"
assert "n1" in store._inference_nodes
def test_model_compare_load_marks_error_when_all_nodes_fail(monkeypatch) -> None:
store = FakeInferenceStore(tasks=[_task("t1", node_id="n1")], nodes=[_node("n1")])
_patch_store(monkeypatch, store)
async def _raise(self, payload: dict[str, Any]) -> dict[str, Any]:
raise RuntimeError("conn refused")
monkeypatch.setattr(ComputeNodeClient, "inference_load", _raise)
result = asyncio.run(model_compare_load("t1"))
updated = result["data"]
assert updated["status"] == "failed"
assert updated["load_status"]["loaded_models"][0]["status"] == "error"
assert "conn refused" in updated["load_status"]["loaded_models"][0]["error"]
def test_model_compare_delete_removes_record_even_if_unload_raises(monkeypatch) -> None:
task = _task(
"t1",
node_id="n1",
load_status={"loaded_models": [{"model_id": "m_1", "status": "ready", "node_id": "n1"}]},
)
store = FakeInferenceStore(tasks=[task], nodes=[_node("n1")])
_patch_store(monkeypatch, store)
async def _raise(self) -> dict[str, Any]:
raise RuntimeError("unload boom")
monkeypatch.setattr(ComputeNodeClient, "inference_unload", _raise)
result = asyncio.run(model_compare_delete("t1"))
assert result["data"] == {"deleted": "t1"}
assert "t1" not in store._tasks
# finally 中仍清掉了节点标记
assert "n1" not in store._inference_nodes
def test_unload_from_compute_node_only_hits_recorded_node(monkeypatch) -> None:
task = _task(
"t1",
load_status={"loaded_models": [{"model_id": "m_1", "status": "ready", "node_id": "n1"}]},
)
store = FakeInferenceStore(tasks=[task], nodes=[_node("n1"), _node("n2")])
_patch_store(monkeypatch, store)
monkeypatch.setattr(ComputeNodeClient, "inference_unload", _fake_inference_unload)
from app.api.v1.endpoints.platform import _unload_from_compute_node
result = asyncio.run(_unload_from_compute_node(store, task=task))
assert result["unloaded"] is True
# 只命中任务记录中的节点 n1，n2 未被卸载
assert [r["node_id"] for r in result["nodes"]] == ["n1"]
assert "n1" not in store._inference_nodes
async def _status_ready(self) -> dict[str, Any]:
return {"loaded": True, "status": "ready", "model_name": "qwen"}
def test_reconcile_transitions_starting_to_ready(monkeypatch) -> None:
task = _task(
"t1",
node_id="n1",
load_status={"loaded_models": [{"model_id": "m_1", "status": "starting", "node_id": "n1"}]},
)
store = FakeInferenceStore(tasks=[task], nodes=[_node("n1")])
monkeypatch.setattr(ComputeNodeClient, "inference_status", _status_ready)
reconciled = asyncio.run(reconcile_inference_loads(store))
assert reconciled == [{"task_id": "t1", "status": "loaded"}]
updated = store._tasks["t1"]
assert updated["status"] == "loaded"
assert updated["load_status"]["loaded_models"][0]["status"] == "ready"
assert "n1" in store._inference_nodes
def test_reconcile_transitions_to_error_and_failed(monkeypatch) -> None:
async def _status_error(self) -> dict[str, Any]:
return {"loaded": False, "status": "error", "error": "CUDA out of memory"}
task = _task(
"t1",
node_id="n1",
load_status={"loaded_models": [{"model_id": "m_1", "status": "starting", "node_id": "n1"}]},
)
store = FakeInferenceStore(tasks=[task], nodes=[_node("n1")])
monkeypatch.setattr(ComputeNodeClient, "inference_status", _status_error)
reconciled = asyncio.run(reconcile_inference_loads(store))
assert reconciled == [{"task_id": "t1", "status": "failed"}]
item = store._tasks["t1"]["load_status"]["loaded_models"][0]
assert item["status"] == "error"
assert "CUDA out of memory" in item["error"]
assert "n1" not in store._inference_nodes
def test_reconcile_idle_marks_model_disappeared(monkeypatch) -> None:
async def _status_idle(self) -> dict[str, Any]:
return {"loaded": False, "status": "idle"}
task = _task(
"t1",
node_id="n1",
load_status={"loaded_models": [{"model_id": "m_1", "status": "starting", "node_id": "n1"}]},
)
store = FakeInferenceStore(tasks=[task], nodes=[_node("n1")])
monkeypatch.setattr(ComputeNodeClient, "inference_status", _status_idle)
asyncio.run(reconcile_inference_loads(store))
item = store._tasks["t1"]["load_status"]["loaded_models"][0]
assert item["status"] == "error"
assert "disappeared" in item["error"]
assert store._tasks["t1"]["status"] == "failed"
def test_reconcile_unreachable_node_flips_to_error_after_cap(monkeypatch) -> None:
async def _raise(self) -> dict[str, Any]:
raise RuntimeError("conn refused")
task = _task(
"t1",
node_id="n1",
load_status={"loaded_models": [{"model_id": "m_1", "status": "starting", "node_id": "n1"}]},
)
store = FakeInferenceStore(tasks=[task], nodes=[_node("n1")])
monkeypatch.setattr(ComputeNodeClient, "inference_status", _raise)
# 每次轮询前重置节流时间戳，逐次推进 load_attempts 到封顶
for _ in range(MAX_STARTING_ATTEMPTS):
item = store._tasks["t1"]["load_status"]["loaded_models"][0]
item["last_polled_at"] = 0
asyncio.run(reconcile_inference_loads(store))
item = store._tasks["t1"]["load_status"]["loaded_models"][0]
assert item["status"] == "error"
assert "unreachable" in item["error"]
assert store._tasks["t1"]["status"] == "failed"

57
backend/tests/test_data_convert_security.py
View File

@@ -1,57 +0,0 @@
"""data_convert 模块安全回归测试：输出文件名路径穿越与鉴权。
- ``output_filename`` 必须通过白名单校验，阻断 ``../``、``/``、``\\`` 及控制字符，
否则转换结果可被写出到存储根目录之外（任意文件读写/删除）。
- 所有 data_convert 路由必须挂载 ``get_current_user`` 鉴权依赖。
"""
from __future__ import annotations
import pytest
from fastapi import HTTPException
from app.core.auth import get_current_user
from app.modules.data_convert.router import _safe_output_filename, router
def test_safe_output_filename_defaults() -> None:
assert _safe_output_filename(None) == "converted-data.jsonl"
assert _safe_output_filename("") == "converted-data.jsonl"
def test_safe_output_filename_valid() -> None:
assert _safe_output_filename("converted-data.jsonl") == "converted-data.jsonl"
assert _safe_output_filename("my-data.v1.jsonl") == "my-data.v1.jsonl"
assert _safe_output_filename(" 报告.jsonl ") == "报告.jsonl"
@pytest.mark.parametrize(
"bad",
[
"../../etc/passwd",
"../x.jsonl",
"a/b.jsonl",
r"a\b.jsonl",
"a\\b.jsonl",
"..",
".",
"x\x00.jsonl",
"x\n.jsonl",
"x\t.jsonl",
],
)
def test_safe_output_filename_rejects_traversal(bad: str) -> None:
with pytest.raises(HTTPException):
_safe_output_filename(bad)
def test_all_data_convert_routes_require_auth() -> None:
for route in router.routes:
node = getattr(route, "dependant", None)
assert node is not None, f"route {route.path} has no dependency graph"
stack = list(node.dependencies)
calls: list = []
while stack:
dep = stack.pop()
stack.extend(getattr(dep, "dependencies", []))
calls.append(getattr(dep, "call", None))
assert get_current_user in calls, f"route {route.path} is missing get_current_user auth"

1071
backend/tests/test_data_process_algorithms.py
View File

File diff suppressed because it is too large Load Diff

3528
backend/tests/test_data_process_api.py
View File

File diff suppressed because it is too large Load Diff

965
backend/tests/test_data_process_generation.py
View File

@@ -1,965 +0,0 @@
from __future__ import annotations
import json
import httpx
import pytest
from app.modules.data_process.generation import (
ModelGenerationError,
chat_completions_url,
generate_model_records,
)
def test_chat_completions_url_accepts_host_base_and_complete_url() -> None:
assert chat_completions_url("www.caoxiaozhu.com") == (
"https://www.caoxiaozhu.com/v1/chat/completions"
)
assert chat_completions_url("https://model.example/v1") == (
"https://model.example/v1/chat/completions"
)
complete = "https://model.example/openai/v1/chat/completions"
assert chat_completions_url(complete) == complete
def test_generate_model_records_uses_prompt_auth_and_stable_split() -> None:
requests: list[httpx.Request] = []
progress_updates: list[tuple[int, int]] = []
def handler(request: httpx.Request) -> httpx.Response:
requests.append(request)
payload = json.loads(request.content)
assert payload["model"] == "qwen-plus"
assert payload["response_format"] == {"type": "json_object"}
assert "客户反馈页面加载慢" in payload["messages"][1]["content"]
assert "你正在生成标准监督微调问答数据" in payload["messages"][0]["content"]
assert "禁止输出分析、推理过程" in payload["messages"][0]["content"]
return httpx.Response(
200,
json={
"choices": [
{
"message": {
"content": json.dumps(
{
"items": [
{
"instruction": "请生成简洁客服回复",
"input": "客户反馈页面加载慢",
"output": "已收到反馈，我们正在排查。",
}
]
},
ensure_ascii=False,
)
}
}
]
},
)
client = httpx.Client(transport=httpx.MockTransport(handler))
records = generate_model_records(
[{"id": "preview-1", "edited_content": "客户反馈页面加载慢"}],
model={
"name": "Qwen",
"online_model_name": "qwen-plus",
"api_url": "model.example",
"api_key": "test-secret",
},
config={
"generation_prompt": "请处理：{{ content }}",
"json_mode": True,
"temperature": 0.2,
"max_tokens": 512,
},
task_id="task-1",
split={"train": 100, "validation": 0, "test": 0},
qa_pairs_per_item=1,
client=client,
on_progress=lambda processed, total: progress_updates.append((processed, total)),
)
assert len(records) == 1
assert records[0]["status"] == "valid"
assert records[0]["split"] == "train"
assert requests[0].headers["Authorization"] == "Bearer test-secret"
assert progress_updates == [(1, 1)]
def test_generate_model_records_builds_native_dpo_pair() -> None:
def handler(request: httpx.Request) -> httpx.Response:
payload = json.loads(request.content)
system_prompt = payload["messages"][0]["content"]
assert '"chosen"' in system_prompt
assert '"rejected"' in system_prompt
assert "直接偏好优化" in system_prompt
return httpx.Response(
200,
json={
"choices": [{
"message": {
"content": json.dumps({
"items": [{
"instruction": "系统如何处理扫描 PDF？",
"input": "",
"chosen": "仅在没有文本层时调用 OCR，并保留页码。",
"rejected": "所有 PDF 都重复执行 OCR。",
}],
}, ensure_ascii=False),
},
}],
},
)
records = generate_model_records(
[{"id": "preview-dpo", "edited_content": "扫描 PDF 缺少文本层时执行 OCR。"}],
model={"name": "model", "api_url": "https://model.example/v1"},
config={"output_type": "dpo", "generation_retries": 0},
task_id="task-dpo",
split={"train": 100, "validation": 0, "test": 0},
qa_pairs_per_item=1,
client=httpx.Client(transport=httpx.MockTransport(handler)),
)
assert records[0]["status"] == "valid"
assert records[0]["chosen"] == "仅在没有文本层时调用 OCR,并保留页码。"
assert records[0]["rejected"] == "所有 PDF 都重复执行 OCR。"
assert records[0]["output"] == records[0]["chosen"]
def test_generate_model_records_rejects_equal_dpo_pair() -> None:
def handler(_request: httpx.Request) -> httpx.Response:
return httpx.Response(
200,
json={"choices": [{"message": {"content": json.dumps({
"items": [{
"instruction": "问题",
"chosen": "相同回答",
"rejected": "相同回答",
}],
}, ensure_ascii=False)}}]},
)
records = generate_model_records(
[{"id": "preview-dpo-invalid", "edited_content": "来源"}],
model={"name": "model", "api_url": "https://model.example/v1"},
config={"output_type": "dpo", "generation_retries": 0},
task_id="task-dpo-invalid",
split={"train": 100, "validation": 0, "test": 0},
qa_pairs_per_item=1,
client=httpx.Client(transport=httpx.MockTransport(handler)),
)
assert records[0]["status"] == "invalid"
assert "chosen equals rejected" in records[0]["error"]
def test_minimax_m3_uses_split_reasoning_and_completion_token_budget() -> None:
requests: list[dict[str, object]] = []
def handler(request: httpx.Request) -> httpx.Response:
payload = json.loads(request.content)
requests.append(payload)
return httpx.Response(
200,
json={
"choices": [{
"finish_reason": "stop",
"message": {
"reasoning_content": "模型内部思考不应混入业务 JSON",
"content": json.dumps({
"items": [{
"instruction": "申请编号有什么作用？",
"reasoning": "来源说明它用于标识报销申请。",
"answer": "它用于唯一标识一笔报销申请。",
}],
}, ensure_ascii=False),
},
}],
"output_sensitive": False,
"base_resp": {"status_code": 0, "status_msg": ""},
},
)
records = generate_model_records(
[{"id": "preview-minimax", "edited_content": "申请编号用于标识报销申请。"}],
model={
"name": "MiniMax",
"online_model_name": "MiniMax-M3",
"api_url": "https://api.minimaxi.com/v1",
},
config={
"output_type": "reasoning",
"json_mode": True,
"max_tokens": 1024,
"generation_retries": 0,
},
task_id="task-minimax",
split={"train": 100, "validation": 0, "test": 0},
qa_pairs_per_item=1,
client=httpx.Client(transport=httpx.MockTransport(handler)),
)
assert records[0]["status"] == "valid"
assert len(requests) == 1
assert requests[0]["reasoning_split"] is True
assert requests[0]["max_completion_tokens"] >= 4096
assert "max_tokens" not in requests[0]
assert "response_format" not in requests[0]
def test_minimax_m3_keeps_larger_configured_completion_budget() -> None:
requests: list[dict[str, object]] = []
def handler(request: httpx.Request) -> httpx.Response:
requests.append(json.loads(request.content))
return httpx.Response(
200,
json={
"choices": [{
"message": {
"content": json.dumps({
"items": [{
"instruction": "问题",
"output": "这是满足测试要求的完整答案。",
}],
}, ensure_ascii=False),
},
}],
},
)
generate_model_records(
[{"id": "preview-minimax-budget", "edited_content": "来源正文"}],
model={
"online_model_name": "MiniMax-M3",
"api_url": "https://api.minimax.io/v1",
},
config={"max_tokens": 8192, "generation_retries": 0},
task_id="task-minimax-budget",
split={"train": 100, "validation": 0, "test": 0},
qa_pairs_per_item=1,
client=httpx.Client(transport=httpx.MockTransport(handler)),
)
assert requests[0]["max_completion_tokens"] == 8192
def test_minimax_m3_name_on_custom_proxy_keeps_generic_openai_parameters() -> None:
requests: list[dict[str, object]] = []
def handler(request: httpx.Request) -> httpx.Response:
requests.append(json.loads(request.content))
return httpx.Response(
200,
json={
"choices": [{
"message": {
"content": json.dumps({
"items": [{
"instruction": "问题",
"output": "这是代理服务返回的完整答案。",
}],
}, ensure_ascii=False),
},
}],
},
)
generate_model_records(
[{"id": "preview-minimax-proxy", "edited_content": "来源正文"}],
model={
"online_model_name": "MiniMax-M3",
"api_url": "https://model-proxy.example/v1",
},
config={"max_tokens": 1024, "json_mode": True, "generation_retries": 0},
task_id="task-minimax-proxy",
split={"train": 100, "validation": 0, "test": 0},
qa_pairs_per_item=1,
client=httpx.Client(transport=httpx.MockTransport(handler)),
)
assert requests[0]["max_tokens"] == 1024
assert requests[0]["response_format"] == {"type": "json_object"}
assert "reasoning_split" not in requests[0]
assert "max_completion_tokens" not in requests[0]
def test_generate_model_records_extracts_json_surrounded_by_model_explanation() -> None:
content = "模型结果如下：\n```json\n" + json.dumps(
{
"items": [{
"instruction": "字段有什么作用？",
"output": "该字段用于唯一标识记录。",
}],
},
ensure_ascii=False,
) + "\n```\n生成完毕。"
client = httpx.Client(
transport=httpx.MockTransport(
lambda _: httpx.Response(
200,
json={"choices": [{"message": {"content": content}}]},
)
)
)
records = generate_model_records(
[{"id": "preview-explanation", "edited_content": "字段用于唯一标识记录。"}],
model={"name": "model", "api_url": "https://model.example/v1"},
config={"generation_retries": 0},
task_id="task-explanation",
split={"train": 100, "validation": 0, "test": 0},
qa_pairs_per_item=1,
client=client,
)
assert records[0]["status"] == "valid"
assert records[0]["output"] == "该字段用于唯一标识记录。"
def test_generate_model_records_reports_token_truncation_instead_of_json_error() -> None:
client = httpx.Client(
transport=httpx.MockTransport(
lambda _: httpx.Response(
200,
json={
"choices": [{
"finish_reason": "length",
"message": {"content": ""},
}],
"output_sensitive": False,
},
)
)
)
records = generate_model_records(
[{"id": "preview-truncated", "edited_content": "来源正文"}],
model={"name": "model", "api_url": "https://model.example/v1"},
config={"generation_retries": 0},
task_id="task-truncated",
split={"train": 100, "validation": 0, "test": 0},
qa_pairs_per_item=1,
client=client,
)
assert records[0]["status"] == "invalid"
assert "Token" in records[0]["error"]
assert "截断" in records[0]["error"]
def test_token_truncation_is_not_retried_even_when_json_looks_complete() -> None:
request_count = 0
content = json.dumps({
"items": [{
"instruction": "问题",
"output": "表面完整但服务端已声明截断。",
}],
}, ensure_ascii=False)
def handler(_: httpx.Request) -> httpx.Response:
nonlocal request_count
request_count += 1
return httpx.Response(
200,
json={
"choices": [{
"finish_reason": "length",
"message": {"content": content},
}],
},
)
records = generate_model_records(
[{"id": "preview-length", "edited_content": "来源正文"}],
model={"name": "model", "api_url": "https://model.example/v1"},
config={"generation_retries": 5},
task_id="task-length",
split={"train": 100, "validation": 0, "test": 0},
qa_pairs_per_item=1,
client=httpx.Client(transport=httpx.MockTransport(handler)),
)
assert request_count == 1
assert records[0]["status"] == "invalid"
assert "finish_reason=length" in records[0]["error"]
def test_sensitive_model_response_is_not_retried_or_saved() -> None:
request_count = 0
def handler(_: httpx.Request) -> httpx.Response:
nonlocal request_count
request_count += 1
return httpx.Response(
200,
json={
"choices": [{
"finish_reason": "stop",
"message": {"content": "{}"},
}],
"output_sensitive": True,
"base_resp": {"status_code": 1027, "status_msg": "output sensitive"},
},
)
records = generate_model_records(
[{"id": "preview-sensitive", "edited_content": "来源正文"}],
model={"name": "model", "api_url": "https://model.example/v1"},
config={"generation_retries": 5},
task_id="task-sensitive",
split={"train": 100, "validation": 0, "test": 0},
qa_pairs_per_item=1,
client=httpx.Client(transport=httpx.MockTransport(handler)),
)
assert request_count == 1
assert records[0]["status"] == "invalid"
assert "安全拦截" in records[0]["error"]
assert "1027" in records[0]["error"]
def test_empty_model_content_can_retry_then_succeed() -> None:
request_count = 0
def handler(_: httpx.Request) -> httpx.Response:
nonlocal request_count
request_count += 1
if request_count == 1:
return httpx.Response(
200,
json={"choices": [{"finish_reason": "stop", "message": {"content": ""}}]},
)
return httpx.Response(
200,
json={
"choices": [{
"finish_reason": "stop",
"message": {
"content": json.dumps({
"items": [{
"instruction": "问题",
"output": "第二次请求返回了完整答案。",
}],
}, ensure_ascii=False),
},
}],
},
)
records = generate_model_records(
[{"id": "preview-empty-retry", "edited_content": "来源正文"}],
model={"name": "model", "api_url": "https://model.example/v1"},
config={"generation_retries": 1},
task_id="task-empty-retry",
split={"train": 100, "validation": 0, "test": 0},
qa_pairs_per_item=1,
client=httpx.Client(transport=httpx.MockTransport(handler)),
)
assert request_count == 2
assert records[0]["status"] == "valid"
def test_multiple_top_level_json_documents_are_rejected_as_ambiguous() -> None:
first = json.dumps({
"items": [{"instruction": "问题一", "output": "答案一"}],
}, ensure_ascii=False)
second = json.dumps({
"items": [{"instruction": "问题二", "output": "答案二"}],
}, ensure_ascii=False)
client = httpx.Client(
transport=httpx.MockTransport(
lambda _: httpx.Response(
200,
json={"choices": [{"message": {"content": f"{first}\n{second}"}}]},
)
)
)
records = generate_model_records(
[{"id": "preview-ambiguous", "edited_content": "来源正文"}],
model={"name": "model", "api_url": "https://model.example/v1"},
config={"generation_retries": 0},
task_id="task-ambiguous",
split={"train": 100, "validation": 0, "test": 0},
qa_pairs_per_item=1,
client=client,
)
assert records[0]["status"] == "invalid"
assert "多个 JSON" in records[0]["error"]
def test_generate_model_records_builds_reasoning_output_with_think_tags() -> None:
def handler(request: httpx.Request) -> httpx.Response:
payload = json.loads(request.content)
system_prompt = payload["messages"][0]["content"]
assert '"reasoning":"...","answer":"..."' in system_prompt
assert "你正在生成用于训练推理模型的思维链数据" in system_prompt
assert "推理详细程度为“普通”" in system_prompt
assert "系统会在保存时统一组装" in system_prompt
content = "<think>模型接口自己的分析</think>" + json.dumps(
{
"items": [
{
"instruction": "计算两项费用合计",
"input": "交通费 30 元，餐费 20 元",
"reasoning": "先识别两项费用，再计算 30 + 20。",
"answer": "合计 50 元。",
}
]
},
ensure_ascii=False,
)
return httpx.Response(
200,
json={"choices": [{"message": {"content": content}}]},
)
records = generate_model_records(
[{"id": "preview-reasoning", "edited_content": "交通费 30 元，餐费 20 元"}],
model={"name": "model", "api_url": "https://model.example/v1"},
config={"output_type": "reasoning"},
task_id="task-reasoning",
split={"train": 100, "validation": 0, "test": 0},
qa_pairs_per_item=1,
client=httpx.Client(transport=httpx.MockTransport(handler)),
)
assert records[0]["status"] == "valid"
assert records[0]["output"] == (
"<think>\n先识别两项费用,再计算 30 + 20。\n</think>\n合计 50 元。"
)
def test_generate_model_records_uses_detailed_reasoning_instruction() -> None:
def handler(request: httpx.Request) -> httpx.Response:
system_prompt = json.loads(request.content)["messages"][0]["content"]
assert "推理详细程度为“详细”" in system_prompt
assert "完整展开问题条件、来源依据、中间计算或推导" in system_prompt
return httpx.Response(
200,
json={
"choices": [
{
"message": {
"content": json.dumps(
{
"items": [
{
"instruction": "计算报销总额",
"reasoning": "条件为交通费 30 元和餐费 20 元。分别核对后相加，30 + 20 = 50。",
"answer": "报销总额为 50 元。",
}
]
},
ensure_ascii=False,
)
}
}
]
},
)
records = generate_model_records(
[{"id": "preview-detailed", "edited_content": "交通费 30 元，餐费 20 元"}],
model={"name": "model", "api_url": "https://model.example/v1"},
config={"output_type": "reasoning", "reasoning_detail": "detailed"},
task_id="task-detailed",
split={"train": 100, "validation": 0, "test": 0},
qa_pairs_per_item=1,
client=httpx.Client(transport=httpx.MockTransport(handler)),
)
assert records[0]["status"] == "valid"
assert "分别核对后相加" in records[0]["output"]
def test_generate_model_records_marks_reasoning_without_reasoning_field_invalid() -> None:
response = {
"choices": [
{
"message": {
"content": json.dumps(
{
"items": [
{
"instruction": "问题",
"answer": "只有最终答案",
}
]
},
ensure_ascii=False,
)
}
}
]
}
client = httpx.Client(
transport=httpx.MockTransport(lambda _: httpx.Response(200, json=response))
)
records = generate_model_records(
[{"id": "preview-missing-reasoning", "edited_content": "来源正文"}],
model={"name": "model", "api_url": "https://model.example/v1"},
config={"output_type": "reasoning"},
task_id="task-missing-reasoning",
split={"train": 100, "validation": 0, "test": 0},
qa_pairs_per_item=1,
client=client,
)
assert records[0]["status"] == "invalid"
assert records[0]["output"] == "只有最终答案"
assert "reasoning" in records[0]["error"]
def test_standard_output_removes_model_think_block() -> None:
content = json.dumps(
{
"items": [
{
"instruction": "问题",
"output": "<think>不应保存的分析</think>最终答案",
}
]
},
ensure_ascii=False,
)
client = httpx.Client(
transport=httpx.MockTransport(
lambda _: httpx.Response(
200,
json={"choices": [{"message": {"content": content}}]},
)
)
)
records = generate_model_records(
[{"id": "preview-standard", "edited_content": "来源正文"}],
model={"name": "model", "api_url": "https://model.example/v1"},
config={"output_type": "standard"},
task_id="task-standard",
split={"train": 100, "validation": 0, "test": 0},
qa_pairs_per_item=1,
client=client,
)
assert records[0]["output"] == "最终答案"
def test_generate_model_records_keeps_partial_failure_for_manual_repair() -> None:
client = httpx.Client(
transport=httpx.MockTransport(
lambda _: httpx.Response(200, json={"choices": [{"message": {"content": "not-json"}}]})
)
)
records = generate_model_records(
[{"id": "preview-1", "edited_content": "来源正文"}],
model={"name": "model", "api_url": "https://model.example/v1"},
config={"generation_retries": 1},
task_id="task-1",
split={"train": 80, "validation": 10, "test": 10},
qa_pairs_per_item=1,
client=client,
)
assert len(records) == 1
assert records[0]["status"] == "invalid"
assert records[0]["error"]
def test_generate_model_records_batches_fifty_results_with_unique_ids() -> None:
requests: list[httpx.Request] = []
def handler(request: httpx.Request) -> httpx.Response:
requests.append(request)
batch_start = (len(requests) - 1) * 10 + 1
batch_end = batch_start + 9
payload = json.loads(request.content)
system_prompt = payload["messages"][0]["content"]
assert "items 必须包含 10 条" in system_prompt
assert f"第 {batch_start}-{batch_end} 条" in system_prompt
return httpx.Response(
200,
json={
"choices": [
{
"message": {
"content": json.dumps(
{
"items": [
{
"instruction": "同一问题",
"input": "来源正文",
"output": "同一答案",
}
for _ in range(batch_start, batch_end + 1)
]
},
ensure_ascii=False,
)
}
}
]
},
)
records = generate_model_records(
[{"id": "preview-50", "edited_content": "来源正文"}],
model={"name": "model", "api_url": "https://model.example/v1"},
config={},
task_id="task-50",
split={"train": 100, "validation": 0, "test": 0},
qa_pairs_per_item=50,
client=httpx.Client(transport=httpx.MockTransport(handler)),
)
assert len(requests) == 5
assert len(records) == 50
assert len({record["id"] for record in records}) == 50
assert {record["instruction"] for record in records} == {"同一问题"}
assert all(record["status"] == "valid" for record in records)
def test_generate_model_records_preserves_successful_batches_when_one_fails() -> None:
request_count = 0
def handler(_: httpx.Request) -> httpx.Response:
nonlocal request_count
request_count += 1
if request_count == 2:
return httpx.Response(500)
return httpx.Response(
200,
json={
"choices": [
{
"message": {
"content": json.dumps(
{
"items": [
{
"instruction": f"问题 {index}",
"output": f"答案 {index}",
}
for index in range(1, 11)
]
},
ensure_ascii=False,
)
}
}
]
},
)
records = generate_model_records(
[{"id": "preview-partial", "edited_content": "来源正文"}],
model={"name": "model", "api_url": "https://model.example/v1"},
config={"generation_retries": 0},
task_id="task-partial",
split={"train": 100, "validation": 0, "test": 0},
qa_pairs_per_item=20,
client=httpx.Client(transport=httpx.MockTransport(handler)),
)
assert len(records) == 11
assert sum(record["status"] == "valid" for record in records) == 10
failed = next(record for record in records if record["status"] == "invalid")
assert "第 11-20 条" in failed["instruction"]
assert len({record["id"] for record in records}) == len(records)
def test_generate_model_records_retries_short_batch_then_marks_it_invalid() -> None:
request_count = 0
def handler(_: httpx.Request) -> httpx.Response:
nonlocal request_count
request_count += 1
return httpx.Response(
200,
json={
"choices": [
{
"message": {
"content": json.dumps(
{
"items": [
{
"instruction": "只有一条",
"output": "不足本批要求数量",
}
]
},
ensure_ascii=False,
)
}
}
]
},
)
records = generate_model_records(
[{"id": "preview-short", "edited_content": "来源正文"}],
model={"name": "model", "api_url": "https://model.example/v1"},
config={"generation_retries": 1},
task_id="task-short",
split={"train": 100, "validation": 0, "test": 0},
qa_pairs_per_item=10,
client=httpx.Client(transport=httpx.MockTransport(handler)),
)
assert request_count == 2
assert len(records) == 1
assert records[0]["status"] == "invalid"
assert "expected 10, got 1" in records[0]["error"]
def test_generate_model_records_does_not_retry_non_retryable_http_errors() -> None:
request_count = 0
def handler(_: httpx.Request) -> httpx.Response:
nonlocal request_count
request_count += 1
return httpx.Response(401, json={"error": {"message": "unauthorized"}})
records = generate_model_records(
[{"id": "preview-auth", "edited_content": "来源内容"}],
model={
"api_url": "https://model.example/v1",
"online_model_name": "test-model",
"api_key": "invalid",
},
config={"generation_retries": 5},
task_id="task-auth",
split={"train": 100, "validation": 0, "test": 0},
qa_pairs_per_item=1,
client=httpx.Client(transport=httpx.MockTransport(handler)),
)
assert request_count == 1
assert records[0]["status"] == "invalid"
assert "401" in records[0]["error"]
@pytest.mark.parametrize("status_code", [408, 425, 429, 500])
def test_generate_model_records_retries_retryable_http_statuses(
status_code: int,
) -> None:
request_count = 0
def handler(_: httpx.Request) -> httpx.Response:
nonlocal request_count
request_count += 1
if request_count == 1:
return httpx.Response(status_code)
return httpx.Response(
200,
json={
"choices": [{
"message": {
"content": json.dumps({
"items": [{
"instruction": "来源内容是什么？",
"output": "这是用于验证可重试错误的来源内容。",
}],
}, ensure_ascii=False),
},
}],
},
)
records = generate_model_records(
[{"id": "preview-retryable", "edited_content": "来源内容"}],
model={"name": "model", "api_url": "https://model.example/v1"},
config={"generation_retries": 1},
task_id="task-retryable",
split={"train": 100, "validation": 0, "test": 0},
qa_pairs_per_item=1,
client=httpx.Client(transport=httpx.MockTransport(handler)),
)
assert request_count == 2
assert records[0]["status"] == "valid"
def test_generate_model_records_retries_transient_network_errors() -> None:
request_count = 0
def handler(request: httpx.Request) -> httpx.Response:
nonlocal request_count
request_count += 1
if request_count == 1:
raise httpx.ConnectError("temporary connection failure", request=request)
return httpx.Response(
200,
json={
"choices": [{
"message": {
"content": json.dumps({
"items": [{
"instruction": "网络恢复了吗？",
"output": "临时连接错误后，第二次模型请求已经成功。",
}],
}, ensure_ascii=False),
},
}],
},
)
records = generate_model_records(
[{"id": "preview-network", "edited_content": "网络重试来源"}],
model={"name": "model", "api_url": "https://model.example/v1"},
config={"generation_retries": 1},
task_id="task-network",
split={"train": 100, "validation": 0, "test": 0},
qa_pairs_per_item=1,
client=httpx.Client(transport=httpx.MockTransport(handler)),
)
assert request_count == 2
assert records[0]["status"] == "valid"
@pytest.mark.parametrize("qa_pairs_per_item", [0, 51])
def test_generate_model_records_rejects_out_of_range_count(
qa_pairs_per_item: int,
) -> None:
with pytest.raises(ModelGenerationError, match=r"\[1, 50\]"):
generate_model_records(
[],
model={"name": "model", "api_url": "https://model.example/v1"},
config={},
task_id="task-invalid",
split={"train": 100, "validation": 0, "test": 0},
qa_pairs_per_item=qa_pairs_per_item,
)
def test_generate_model_records_rejects_unknown_output_type() -> None:
with pytest.raises(ModelGenerationError, match="output_type"):
generate_model_records(
[],
model={"name": "model", "api_url": "https://model.example/v1"},
config={"output_type": "unknown"},
task_id="task-invalid-output",
split={"train": 100, "validation": 0, "test": 0},
qa_pairs_per_item=1,
)
def test_generate_model_records_rejects_unknown_reasoning_detail() -> None:
with pytest.raises(ModelGenerationError, match="reasoning_detail"):
generate_model_records(
[],
model={"name": "model", "api_url": "https://model.example/v1"},
config={"output_type": "reasoning", "reasoning_detail": "verbose"},
task_id="task-invalid-reasoning-detail",
split={"train": 100, "validation": 0, "test": 0},
qa_pairs_per_item=1,
)

65
backend/tests/test_data_process_migration.py
View File

@@ -1,65 +0,0 @@
from __future__ import annotations
from pathlib import Path
from app.modules.data_process.schema_cli import REQUIRED_TASK_COLUMNS, _target_label
def test_runtime_migration_fails_fast_on_incompatible_schema() -> None:
sql_path = (
Path(__file__).resolve().parents[1]
/ "app"
/ "db"
/ "sql"
/ "002_data_process.sql"
)
sql = sql_path.read_text(encoding="utf-8")
assert "requires 001_platform_runtime.sql first" in sql
assert "supports only the current TEXT runtime schema" in sql
assert "generation_run_id" in sql
assert "results_confirmed BOOLEAN NOT NULL DEFAULT TRUE" in sql
assert "WHERE status <> 'completed' AND results_confirmed=TRUE" in sql
assert "ADD COLUMN IF NOT EXISTS workflow_step VARCHAR(20)" in sql
assert "ADD COLUMN IF NOT EXISTS preview_status VARCHAR(20)" in sql
assert "ADD COLUMN IF NOT EXISTS preview_progress NUMERIC(5,2)" in sql
assert "ADD COLUMN IF NOT EXISTS preview_run_id TEXT" in sql
assert "ADD COLUMN IF NOT EXISTS preview_failure_reason TEXT" in sql
assert "ADD COLUMN IF NOT EXISTS preview_total_files INTEGER" in sql
assert "ADD COLUMN IF NOT EXISTS preview_completed_files INTEGER" in sql
assert "data_process_workflow_backfill_ids" in sql
assert "ck_data_process_tasks_workflow_step" in sql
assert "ck_data_process_tasks_preview_status" in sql
assert "ck_data_process_tasks_preview_progress" in sql
assert "ck_data_process_tasks_preview_file_counts" in sql
for value in ("create", "model", "upload", "preview", "generate", "results"):
assert f"'{value}'" in sql
for value in ("idle", "queued", "running", "completed", "failed", "cancelled"):
assert f"'{value}'" in sql
assert "CREATE TABLE IF NOT EXISTS data_process_results" in sql
assert "ADD COLUMN IF NOT EXISTS chosen TEXT NOT NULL DEFAULT ''" in sql
assert "ADD COLUMN IF NOT EXISTS rejected TEXT NOT NULL DEFAULT ''" in sql
assert "ADD COLUMN IF NOT EXISTS original_chosen TEXT" in sql
assert "ADD COLUMN IF NOT EXISTS original_rejected TEXT" in sql
assert sql.count("BEGIN;") == 1
assert sql.rstrip().endswith("COMMIT;")
def test_schema_cli_target_label_never_contains_credentials() -> None:
label = _target_label("postgresql://secret-user:secret-password@db.example:5433/yg_ft")
assert label == "db.example:5433/yg_ft"
assert "secret" not in label
def test_schema_check_requires_current_runtime_columns() -> None:
assert REQUIRED_TASK_COLUMNS == (
"generation_run_id",
"results_confirmed",
"workflow_step",
"preview_status",
"preview_progress",
"preview_run_id",
"preview_failure_reason",
"preview_total_files",
"preview_completed_files",
)

276
backend/tests/test_data_process_storage.py
View File

@@ -1,276 +0,0 @@
from __future__ import annotations
from pathlib import Path, PurePosixPath
import pytest
from app.modules.data_process import storage as storage_module
from app.modules.data_process.storage import (
DataProcessStorageError,
LocalDataProcessStorage,
StagedSourceObject,
)
def _stage(
storage: LocalDataProcessStorage,
*,
batch_id: str = "batch-main",
task_id: str = "task-1",
source_file_id: str = "source-1",
version: int = 1,
name: str = "source.txt",
content: bytes = b"payload",
) -> StagedSourceObject:
return storage.stage_bytes(
batch_id=batch_id,
task_id=task_id,
source_file_id=source_file_id,
version=version,
name=name,
content=content,
)
def _create_symlink(link: Path, target: Path, *, target_is_directory: bool = False) -> None:
try:
link.symlink_to(target, target_is_directory=target_is_directory)
except (NotImplementedError, OSError) as exc:
pytest.skip(f"当前平台不支持创建测试所需的符号链接: {exc}")
def _assert_staging_empty(storage: LocalDataProcessStorage) -> None:
assert list((storage.root / ".staging").iterdir()) == []
def test_stage_publish_read_delete_roundtrip_with_unicode_filename(tmp_path: Path) -> None:
storage = LocalDataProcessStorage(tmp_path / "storage")
content = "第一行\n第二行，100% 完成".encode()
staged = _stage(
storage,
name="中文 数据 100%.csv",
content=content,
)
assert "%20" in staged.reference
assert "%25" in staged.reference
storage.publish([staged])
assert storage.read(staged.reference) == content
assert storage.delete(staged.reference) is True
assert storage.delete(staged.reference) is False
_assert_staging_empty(storage)
def test_stage_copy_creates_an_independently_deletable_source_object(
tmp_path: Path,
) -> None:
storage = LocalDataProcessStorage(tmp_path / "storage")
original = _stage(storage, content=b"immutable source")
storage.publish([original])
copied = storage.stage_copy(
batch_id="batch-copy",
source_reference=original.reference,
expected_source_task_id="task-1",
expected_source_file_id="source-1",
task_id="task-2",
source_file_id="source-2",
version=1,
name="source.txt",
)
storage.publish([copied])
assert storage.read(copied.reference) == b"immutable source"
assert storage.delete(
original.reference,
expected_task_id="task-1",
expected_source_file_id="source-1",
) is True
assert storage.read(copied.reference) == b"immutable source"
assert storage.delete(
copied.reference,
expected_task_id="task-2",
expected_source_file_id="source-2",
) is True
_assert_staging_empty(storage)
def test_db_reference_is_left_to_database_storage(tmp_path: Path) -> None:
storage = LocalDataProcessStorage(tmp_path / "storage")
assert storage.read("db://source-files/source-1") is None
assert storage.delete("db://source-files/source-1") is False
def test_owned_source_can_be_streamed_by_byte_range(tmp_path: Path) -> None:
storage = LocalDataProcessStorage(tmp_path / "storage")
content = b"0123456789abcdef"
staged = _stage(storage, content=content)
storage.publish([staged])
assert storage.file_size(
staged.reference,
expected_task_id="task-1",
expected_source_file_id="source-1",
) == len(content)
assert b"".join(storage.iter_bytes(
staged.reference,
expected_task_id="task-1",
expected_source_file_id="source-1",
expected_size=len(content),
start=4,
length=6,
chunk_size=2,
)) == b"456789"
with pytest.raises(DataProcessStorageError, match="owner mismatch"):
storage.file_size(
staged.reference,
expected_task_id="another-task",
expected_source_file_id="source-1",
)
with pytest.raises(DataProcessStorageError, match="does not match metadata"):
b"".join(storage.iter_bytes(
staged.reference,
expected_task_id="task-1",
expected_source_file_id="source-1",
expected_size=len(content) + 1,
))
@pytest.mark.parametrize(
"reference",
[
"local://data-process/../source-1/v1/file.txt",
"local://data-process/task-1/source-1/v1/file%2Fname.txt",
"local://data-process/task-1/source-1/v1/file.txt?download=1",
"local://data-process/task-1/source-1/v1/file.txt#fragment",
"https://data-process/task-1/source-1/v1/file.txt",
],
ids=[
"parent-traversal",
"percent-encoded-slash",
"query",
"fragment",
"wrong-scheme",
],
)
def test_unsafe_references_are_rejected(tmp_path: Path, reference: str) -> None:
storage = LocalDataProcessStorage(tmp_path / "storage")
with pytest.raises(DataProcessStorageError):
storage.read(reference)
with pytest.raises(DataProcessStorageError):
storage.delete(reference)
def test_publish_rejects_intermediate_directory_symlink(tmp_path: Path) -> None:
storage = LocalDataProcessStorage(tmp_path / "storage")
outside = tmp_path / "outside"
outside.mkdir()
staged = _stage(storage, task_id="linked-task")
_create_symlink(
storage.root / "linked-task",
outside,
target_is_directory=True,
)
with pytest.raises(DataProcessStorageError, match="symlink|non-directory"):
storage.publish([staged])
assert list(outside.iterdir()) == []
_assert_staging_empty(storage)
def test_target_symlink_is_never_followed_or_deleted(tmp_path: Path) -> None:
storage = LocalDataProcessStorage(tmp_path / "storage")
staged = _stage(storage, task_id="task-link", source_file_id="source-link")
outside_file = tmp_path / "outside.txt"
outside_file.write_bytes(b"outside sentinel")
final_path = storage.root.joinpath(*staged._relative_path.parts)
final_path.parent.mkdir(parents=True)
_create_symlink(final_path, outside_file)
with pytest.raises(DataProcessStorageError, match="already exists"):
storage.publish([staged])
with pytest.raises(DataProcessStorageError, match="regular file"):
storage.read(staged.reference)
with pytest.raises(DataProcessStorageError, match="non-regular"):
storage.delete(staged.reference)
assert final_path.is_symlink()
assert outside_file.read_bytes() == b"outside sentinel"
_assert_staging_empty(storage)
def test_publish_rolls_back_first_object_when_second_target_collides(tmp_path: Path) -> None:
storage = LocalDataProcessStorage(tmp_path / "storage")
existing = _stage(
storage,
batch_id="batch-existing",
source_file_id="source-existing",
content=b"existing content",
)
storage.publish([existing])
first = _stage(
storage,
batch_id="batch-new",
source_file_id="source-new",
content=b"must be rolled back",
)
colliding_second = _stage(
storage,
batch_id="batch-new",
source_file_id="source-existing",
content=b"must not replace existing content",
)
with pytest.raises(DataProcessStorageError, match="already exists"):
storage.publish([first, colliding_second])
with pytest.raises(DataProcessStorageError, match="does not exist"):
storage.read(first.reference)
assert storage.read(existing.reference) == b"existing content"
_assert_staging_empty(storage)
def test_publish_rejects_manually_forged_staged_object(tmp_path: Path) -> None:
storage = LocalDataProcessStorage(tmp_path / "storage")
temporary_path = storage.root / ".staging" / "batch-forged" / "forged.tmp"
temporary_path.parent.mkdir()
temporary_path.write_bytes(b"forged content")
relative_path = PurePosixPath("task-forged", "source-forged", "v1", "forged.txt")
forged = StagedSourceObject(
reference="local://data-process/task-forged/source-forged/v1/forged.txt",
_temporary_path=temporary_path,
_relative_path=relative_path,
)
with pytest.raises(DataProcessStorageError, match="was not issued"):
storage.publish([forged])
with pytest.raises(DataProcessStorageError, match="was not issued"):
storage.discard([forged])
assert temporary_path.read_bytes() == b"forged content"
assert not storage.root.joinpath(*relative_path.parts).exists()
def test_relative_storage_configuration_is_anchored_to_backend_root(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
relative_configuration = Path("relative-storage") / tmp_path.name
backend_root = Path(storage_module.__file__).resolve().parents[3]
monkeypatch.chdir(tmp_path)
monkeypatch.setenv("DATA_PROCESS_STORAGE_DIR", str(relative_configuration))
storage_module.get_data_process_storage.cache_clear()
try:
configured_root = storage_module._configured_storage_root()
assert configured_root == backend_root / relative_configuration
assert not configured_root.exists()
finally:
storage_module.get_data_process_storage.cache_clear()

1507
backend/tests/test_data_process_store.py
View File

File diff suppressed because it is too large Load Diff

76
backend/tests/test_docs_security.py
View File

@@ -1,76 +0,0 @@
"""Swagger / ReDoc / OpenAPI 文档路由安全开关测试。
生产环境（APP_ENV=prod）默认关闭 /docs、/redoc、/openapi.json，
避免未授权访问泄露 API 结构；本地开发环境默认开放，可用 ENABLE_DOCS 覆盖。
"""
from __future__ import annotations
import pytest
from app.core.config import docs_kwargs, get_settings
@pytest.fixture(autouse=True)
def _reset_settings_cache():
"""每次测试前后清空 get_settings 的 lru_cache，避免环境变量互相污染。"""
get_settings.cache_clear()
yield
get_settings.cache_clear()
def test_docs_kwargs_enabled() -> None:
assert docs_kwargs(True) == {}
def test_docs_kwargs_disabled() -> None:
assert docs_kwargs(False) == {"docs_url": None, "redoc_url": None, "openapi_url": None}
def test_docs_disabled_by_default_in_prod(monkeypatch) -> None:
monkeypatch.delenv("ENABLE_DOCS", raising=False)
monkeypatch.setenv("APP_ENV", "prod")
assert get_settings().enable_docs is False
def test_docs_enabled_by_default_outside_prod(monkeypatch) -> None:
monkeypatch.delenv("ENABLE_DOCS", raising=False)
monkeypatch.setenv("APP_ENV", "local")
assert get_settings().enable_docs is True
def test_docs_env_override_enables_in_prod(monkeypatch) -> None:
monkeypatch.setenv("ENABLE_DOCS", "true")
monkeypatch.setenv("APP_ENV", "prod")
assert get_settings().enable_docs is True
def test_docs_env_override_disables_outside_prod(monkeypatch) -> None:
monkeypatch.setenv("ENABLE_DOCS", "false")
monkeypatch.setenv("APP_ENV", "local")
assert get_settings().enable_docs is False
def test_create_app_disables_docs_in_prod(monkeypatch, tmp_path) -> None:
pytest.importorskip("fastapi")
from app.main import create_app
monkeypatch.delenv("ENABLE_DOCS", raising=False)
monkeypatch.setenv("APP_ENV", "prod")
monkeypatch.setenv("LOG_DIR", str(tmp_path))
app = create_app()
assert app.docs_url is None
assert app.redoc_url is None
assert app.openapi_url is None
def test_create_app_enables_docs_outside_prod(monkeypatch, tmp_path) -> None:
pytest.importorskip("fastapi")
from app.main import create_app
monkeypatch.delenv("ENABLE_DOCS", raising=False)
monkeypatch.setenv("APP_ENV", "local")
monkeypatch.setenv("LOG_DIR", str(tmp_path))
app = create_app()
assert app.docs_url == "/docs"
assert app.redoc_url == "/redoc"
assert app.openapi_url == "/openapi.json"

85
backend/tests/test_document_chunking.py
View File

@@ -1,85 +0,0 @@
from __future__ import annotations
from llama_index.core.embeddings import MockEmbedding
from app.modules.data_process.document_chunking import (
DocumentChunk,
_compact_with_offsets,
_project_layout_span,
chunk_fixed_text,
chunk_semantic_text,
merge_short_chunks,
)
def test_fixed_splitter_preserves_offsets_and_token_limit() -> None:
text = "第一段说明苹果。第二段说明香蕉。\n第三段说明数据库。第四段说明索引。"
chunks = chunk_fixed_text(text, chunk_size=20, chunk_overlap=0)
assert len(chunks) > 1
assert all(chunk.source_start is not None for chunk in chunks)
assert all(chunk.source_end is not None for chunk in chunks)
assert all(
chunk.original_content == text[chunk.source_start : chunk.source_end]
for chunk in chunks
if chunk.source_start is not None and chunk.source_end is not None
)
assert all(chunk.token_count <= 20 for chunk in chunks)
def test_semantic_splitter_uses_llamaindex_and_reapplies_maximum_size() -> None:
text = "第一段讨论水果。第二段继续讨论香蕉。第三段讨论数据库。第四段讨论索引。"
chunks = chunk_semantic_text(
text,
chunk_size=30,
chunk_overlap=0,
breakpoint_percentile_threshold=95,
embed_model=MockEmbedding(embed_dim=8),
)
assert len(chunks) >= 2
assert all(chunk.token_count <= 30 for chunk in chunks)
assert "".join(chunk.original_content for chunk in chunks) == text
def test_layout_projection_ignores_layout_whitespace_but_keeps_source_lines() -> None:
source = "标题\n第一条 这是正文。\n第二条 后续正文。"
compact_source, offsets = _compact_with_offsets(source)
start, end, cursor = _project_layout_span(
source,
"第一条\n这是正文。",
compact_source=compact_source,
source_offsets=offsets,
compact_start=0,
)
assert source[start:end] == "第一条 这是正文。"
assert cursor > 0
def test_short_layout_chunk_merges_with_neighbor_and_keeps_page_provenance() -> None:
source = "短标题\n这是一段足够长的正文内容，用于测试相邻切片合并。"
chunks = [
DocumentChunk("短标题", "短标题", 0, 3, 1, 1, 2, source_pages=(1,)),
DocumentChunk(
"这是一段足够长的正文内容，用于测试相邻切片合并。",
"这是一段足够长的正文内容，用于测试相邻切片合并。",
4,
len(source),
2,
2,
20,
source_pages=(1, 2),
),
]
merged = merge_short_chunks(
chunks,
source_text=source,
min_token_count=10,
max_token_count=100,
)
assert len(merged) == 1
assert merged[0].original_content == source
assert merged[0].source_pages == (1, 2)

791
backend/tests/test_governance.py
View File

@@ -1,791 +0,0 @@
"""
平台治理功能集成测试 —— 覆盖第 1-4 周交付内容。
测试策略：
- 在导入 app 模块前 mock psycopg / psycopg_pool，避免依赖真实数据库驱动
- 使用 FastAPI TestClient 对真实路由栈发起请求
- 通过 mock.get_platform_store 替换为内存 FakeStore
- 每周交付内容对应一组 test class，方便分阶段验收
覆盖范围：
第 1 周 — 登录、当前用户、用户列表、权限码、日志查询
第 2 周 — 租户、项目、项目成员、资源 ACL
第 3 周 — 审批实例、审批模板、审计日志查询和导出
第 4 周 — 写操作审计、审批拦截、权限校验
"""
from __future__ import annotations
import json
import sys
import types
from contextlib import contextmanager
from typing import Any, Iterator
from unittest.mock import MagicMock, patch
import pytest
from fastapi import FastAPI
from fastapi.testclient import TestClient
# ============================================================
# 在导入 app 之前 mock psycopg / psycopg_pool
# ============================================================
_psycopg_mock = types.ModuleType("psycopg")
_psycopg_mock.PgConn = type("PgConn", (), {})
_psycopg_mock.PostgresConnectionPool = MagicMock()
_psycopg_mock.connection = MagicMock()
sys.modules.setdefault("psycopg", _psycopg_mock)
_psycopg_pool_mock = types.ModuleType("psycopg_pool")
_psycopg_pool_mock.ConnectionPool = MagicMock()
sys.modules.setdefault("psycopg_pool", _psycopg_pool_mock)
# 现在安全导入 app 模块
from app.api.v1.endpoints.platform import ok, fail # noqa: E402
from app.modules.tenant.router import router as tenant_router # noqa: E402
from app.modules.project.router import router as project_router # noqa: E402
from app.modules.approval.router import router as approval_router # noqa: E402
from app.modules.system.router import router as system_router # noqa: E402
from app.modules.retention.router import router as retention_router # noqa: E402
from app.modules.resource.router import router as resource_router # noqa: E402
from app.api.v1.endpoints.platform import router as platform_router # noqa: E402
PREFIX = "/modelTF"
ADMIN_TOKEN = "platform-token-u_admin"
OP_TOKEN = "platform-token-u_op"
# ============================================================
# FakePlatformStore —— 内存实现，模拟 PlatformStore 全部治理接口
# ============================================================
class FakePlatformStore:
"""平台治理测试专用内存 store，确保测试不连接真实数据库。"""
def __init__(self) -> None:
self._users: list[dict[str, Any]] = [
{
"id": "u_admin",
"username": "admin",
"display_name": "Admin",
"role": "admin",
"status": "active",
"permissions": [
"dashboard", "fine-tune", "model-eval", "model-inference",
"model-manage", "dataset", "data-process", "data-convert",
"compute", "hardware", "logs", "user-settings",
],
"last_login": "2026-08-01T10:00:00Z",
"protected": True,
},
{
"id": "u_op",
"username": "operator",
"display_name": "Operator",
"role": "operator",
"status": "active",
"permissions": ["dashboard", "fine-tune"],
"last_login": "2026-08-01T11:00:00Z",
"protected": False,
},
]
self._tenants: dict[str, dict[str, Any]] = {}
self._projects: dict[str, dict[str, Any]] = {}
self._members: dict[str, list[dict[str, Any]]] = {}
self._acl: dict[str, list[dict[str, Any]]] = {}
self._audit_logs: list[dict[str, Any]] = []
self._approval_templates: dict[str, dict[str, Any]] = {}
self._approval_instances: dict[str, dict[str, Any]] = {}
self._retention_policies: dict[str, dict[str, Any]] = {}
self._models: list[dict[str, Any]] = []
self._datasets: list[dict[str, Any]] = []
self._tasks: list[dict[str, Any]] = []
self._compute_nodes: list[dict[str, Any]] = []
self._gpus: list[dict[str, Any]] = []
self._sessions: list[dict[str, Any]] = []
self._seq = 0
@contextmanager
def connect(self) -> Iterator[Any]:
class FakeConn:
def execute(self, *a, **kw):
return []
def commit(self):
pass
def rollback(self):
pass
def close(self):
pass
yield FakeConn()
# ---- helpers ----
def _next_id(self, prefix: str) -> str:
self._seq += 1
return f"{prefix}_{self._seq}"
# ==================== 第1周：登录 / 用户 / 权限码 / 日志 ====================
def login(self, username: str, password: str) -> dict[str, Any] | None:
for u in self._users:
if u["username"] == username and u["status"] == "active":
if password in ("admin123", "operator123", "test123"):
return dict(u)
return None
def create_session(self, user_id: str) -> dict[str, Any]:
import secrets
sid = secrets.token_hex(16)
return {"session_id": sid, "user_id": user_id}
def finish_session(self, session_id: str) -> None:
pass
def users(self) -> list[dict[str, Any]]:
return [dict(u) for u in self._users]
def create_user(self, payload: dict[str, Any]) -> dict[str, Any]:
u = {"id": self._next_id("u"), "protected": False, **payload}
self._users.append(u)
return u
def update_user(self, user_id: str, payload: dict[str, Any]) -> dict[str, Any]:
for u in self._users:
if u["id"] == user_id:
u.update(payload)
return u
raise KeyError(user_id)
def delete_user(self, user_id: str) -> None:
self._users = [u for u in self._users if u["id"] != user_id]
def roles(self) -> list[dict[str, Any]]:
return [
{"name": "admin", "display_name": "管理员"},
{"name": "operator", "display_name": "操作员"},
{"name": "viewer", "display_name": "访客"},
]
def log_files(self, date: str | None = None) -> list[dict[str, Any]]:
return [{"name": "backend-2026-08-01.log", "size": "1 KB", "date": "2026-08-01"}]
def log_content(self, file: str) -> dict[str, Any]:
return {"file": file, "content": "[INFO] test line", "size": "1 KB"}
def training_log_files(self) -> list[dict[str, Any]]:
return [{"task_id": "ft_001", "name": "ft_001.log", "size": "2 KB"}]
def training_log_content(self, file: str) -> dict[str, Any]:
return {"file": file, "content": "epoch 0 loss 1.0", "size": "2 KB"}
# ==================== 第2周：租户 / 项目 / 成员 / ACL ====================
def tenants(self) -> list[dict[str, Any]]:
return list(self._tenants.values())
def tenant(self, tenant_id: str) -> dict[str, Any]:
if tenant_id not in self._tenants:
raise KeyError(tenant_id)
return dict(self._tenants[tenant_id])
def create_tenant(self, payload: dict[str, Any]) -> dict[str, Any]:
tid = self._next_id("tnt")
t = {"id": tid, "status": "active", "quota": "{}", "retention_policy_id": None,
"create_time": "2026-08-01T00:00:00Z", **payload}
self._tenants[tid] = t
return dict(t)
def update_tenant(self, tenant_id: str, payload: dict[str, Any]) -> dict[str, Any]:
self._tenants[tenant_id].update(payload)
return dict(self._tenants[tenant_id])
def set_tenant_quota(self, tenant_id: str, quota: dict[str, Any]) -> dict[str, Any]:
self._tenants[tenant_id]["quota"] = json.dumps(quota)
return dict(self._tenants[tenant_id])
def set_tenant_retention(self, tenant_id: str, retention_policy_id: str | None) -> dict[str, Any]:
self._tenants[tenant_id]["retention_policy_id"] = retention_policy_id
return dict(self._tenants[tenant_id])
def projects(self, *, tenant_id: str = "default", status: str | None = None, keyword: str | None = None) -> list[dict[str, Any]]:
result = []
for p in self._projects.values():
if p.get("tenant_id") != tenant_id:
continue
if status and p.get("status") != status:
continue
if keyword and keyword.lower() not in p.get("name", "").lower():
continue
result.append(dict(p))
return result
def project(self, project_id: str) -> dict[str, Any]:
if project_id not in self._projects:
raise KeyError(project_id)
return dict(self._projects[project_id])
def create_project(self, payload: dict[str, Any]) -> dict[str, Any]:
pid = self._next_id("prj")
p = {"id": pid, "status": "active", "quota": "{}", "create_time": "2026-08-01T00:00:00Z", **payload}
self._projects[pid] = p
self._members[pid] = []
return dict(p)
def update_project(self, project_id: str, payload: dict[str, Any]) -> dict[str, Any]:
self._projects[project_id].update(payload)
return dict(self._projects[project_id])
def archive_project(self, project_id: str) -> dict[str, Any]:
self._projects[project_id]["status"] = "archived"
return dict(self._projects[project_id])
def delete_project(self, project_id: str) -> None:
self._projects.pop(project_id, None)
self._members.pop(project_id, None)
def project_members(self, project_id: str) -> list[dict[str, Any]]:
return [dict(m) for m in self._members.get(project_id, [])]
def add_project_member(self, project_id: str, payload: dict[str, Any]) -> dict[str, Any]:
m = {"joined_at": "2026-08-01T00:00:00Z", **payload}
self._members.setdefault(project_id, []).append(m)
return m
def update_project_member_role(self, project_id: str, user_id: str, role: str) -> dict[str, Any]:
for m in self._members.get(project_id, []):
if m["user_id"] == user_id:
m["role"] = role
return m
raise KeyError(user_id)
def remove_project_member(self, project_id: str, user_id: str) -> None:
self._members[project_id] = [m for m in self._members.get(project_id, []) if m["user_id"] != user_id]
# ---- ACL ----
def get_acl(self, resource_type: str, resource_id: str) -> list[dict[str, Any]]:
key = f"{resource_type}:{resource_id}"
return [dict(a) for a in self._acl.get(key, [])]
def set_acl(self, resource_type: str, resource_id: str, entries: list[dict[str, Any]]) -> list[dict[str, Any]]:
key = f"{resource_type}:{resource_id}"
self._acl[key] = [dict(e) for e in entries]
return self.get_acl(resource_type, resource_id)
def resource_acl(self, resource_type: str, resource_id: str) -> list[dict[str, Any]]:
rows = self.get_acl(resource_type, resource_id)
grouped: dict[str, dict[str, Any]] = {}
for r in rows:
k = f"{r.get('principal_type')}:{r.get('principal_id')}"
bucket = grouped.setdefault(k, {
"subject_type": r.get("principal_type"),
"subject_id": r.get("principal_id"),
"permissions": [],
})
perm = r.get("permission")
if perm and perm not in bucket["permissions"]:
bucket["permissions"].append(perm)
return list(grouped.values())
def set_resource_acl(self, resource_type: str, resource_id: str, entries: list[dict[str, Any]]) -> list[dict[str, Any]]:
flat: list[dict[str, Any]] = []
for e in entries:
for perm in e.get("permissions") or []:
flat.append({
"principal_type": e.get("subject_type"),
"principal_id": e.get("subject_id"),
"permission": perm,
})
self.set_acl(resource_type, resource_id, flat)
return self.resource_acl(resource_type, resource_id)
# ==================== 第3周：审批 / 审计 / 留存 ====================
def approval_templates(self) -> list[dict[str, Any]]:
return list(self._approval_templates.values())
def create_approval_template(self, payload: dict[str, Any]) -> dict[str, Any]:
tid = payload.get("id") or self._next_id("tpl")
t = {"id": tid, "steps": [], "create_time": "2026-08-01T00:00:00Z", **payload}
self._approval_templates[tid] = t
return dict(t)
def approval_instances(self, *, status: str | None = None) -> list[dict[str, Any]]:
result = []
for i in self._approval_instances.values():
if status and i.get("status") != status:
continue
result.append(dict(i))
return result
def approval_instance(self, instance_id: str) -> dict[str, Any]:
if instance_id not in self._approval_instances:
raise KeyError(instance_id)
return dict(self._approval_instances[instance_id])
def create_approval_instance(self, payload: dict[str, Any]) -> dict[str, Any]:
iid = self._next_id("appr")
inst = {
"id": iid,
"status": "pending",
"current_step": 0,
"steps": [],
"create_time": "2026-08-01T00:00:00Z",
**payload,
}
self._approval_instances[iid] = inst
return dict(inst)
def decide_approval_step(self, instance_id: str, step_index: int, *, approver_id: str, approved: bool, comment: str | None = None) -> dict[str, Any]:
inst = self._approval_instances[instance_id]
inst["status"] = "approved" if approved else "rejected"
inst["current_step"] = step_index + 1
return dict(inst)
def audit_logs(self, **kw) -> dict[str, Any]:
items = [dict(l) for l in self._audit_logs]
for filter_key in ("tenant_id", "project_id", "actor_id", "action", "target_type"):
val = kw.get(filter_key)
if val:
items = [l for l in items if l.get(filter_key) == val]
limit = kw.get("limit", 50)
offset = kw.get("offset", 0)
total = len(items)
items = items[offset:offset + limit]
return {"items": items, "total": total}
def record_audit(self, **kw) -> None:
log = {"id": self._next_id("log"), "time": "2026-08-01T12:00:00Z", **kw}
self._audit_logs.append(log)
# ---- 留存策略 ----
def retention_policies(self) -> list[dict[str, Any]]:
return list(self._retention_policies.values())
def retention_policy(self, policy_id: str) -> dict[str, Any]:
if policy_id not in self._retention_policies:
raise KeyError(policy_id)
return dict(self._retention_policies[policy_id])
def create_retention_policy(self, payload: dict[str, Any]) -> dict[str, Any]:
pid = payload.get("id") or self._next_id("rpol")
p = {"id": pid, "status": "active", "create_time": "2026-08-01T00:00:00Z", **payload}
self._retention_policies[pid] = p
return dict(p)
def update_retention_policy(self, policy_id: str, payload: dict[str, Any]) -> dict[str, Any]:
self._retention_policies[policy_id].update(payload)
return dict(self._retention_policies[policy_id])
def delete_retention_policy(self, policy_id: str) -> None:
self._retention_policies.pop(policy_id, None)
# ---- dashboard & other stubs ----
def login_duration_rank(self, limit: int = 8, days: int = 30) -> list[dict[str, Any]]:
return [{"user": "admin", "role": "admin", "duration": 10.0}]
def models(self) -> list[dict[str, Any]]:
return self._models
def datasets(self) -> list[dict[str, Any]]:
return self._datasets
def tasks(self) -> list[dict[str, Any]]:
return self._tasks
def eval_tasks(self) -> list[dict[str, Any]]:
return []
def compute_nodes(self) -> list[dict[str, Any]]:
return self._compute_nodes
def gpus(self) -> list[dict[str, Any]]:
return self._gpus
def compare_tasks(self) -> list[dict[str, Any]]:
return []
def trained_models(self) -> list[dict[str, Any]]:
return []
def system_info(self) -> dict[str, Any]:
return {"cpu": {}, "memory": {}}
# ============================================================
# 测试 fixtures
# ============================================================
@pytest.fixture(scope="module")
def fake_store() -> FakePlatformStore:
return FakePlatformStore()
def _build_client(store: FakePlatformStore) -> TestClient:
"""构建 TestClient，patch 所有治理模块的 get_platform_store。"""
app = FastAPI()
app.include_router(platform_router, prefix=PREFIX)
app.include_router(system_router, prefix=PREFIX)
app.include_router(tenant_router, prefix=PREFIX)
app.include_router(project_router, prefix=PREFIX)
app.include_router(approval_router, prefix=PREFIX)
app.include_router(retention_router, prefix=PREFIX)
app.include_router(resource_router, prefix=PREFIX)
patches = [
patch("app.db.platform_store.get_platform_store", return_value=store),
patch("app.core.auth.get_platform_store", return_value=store),
patch("app.api.v1.endpoints.platform.get_platform_store", return_value=store),
patch("app.modules.system.router.get_platform_store", return_value=store),
patch("app.modules.tenant.router.get_platform_store", return_value=store),
patch("app.modules.project.router.get_platform_store", return_value=store),
patch("app.modules.approval.router.get_platform_store", return_value=store),
patch("app.modules.retention.router.get_platform_store", return_value=store),
patch("app.modules.resource.router.get_platform_store", return_value=store),
]
for p in patches:
p.start()
client = TestClient(app, raise_server_exceptions=False)
client._fake_store = store # type: ignore[attr-defined]
return client
@pytest.fixture(scope="module")
def client(fake_store: FakePlatformStore) -> TestClient:
c = _build_client(fake_store)
yield c
def _admin_headers() -> dict[str, str]:
return {"Authorization": f"Bearer {ADMIN_TOKEN}"}
def _op_headers() -> dict[str, str]:
return {"Authorization": f"Bearer {OP_TOKEN}"}
# ============================================================
# 第 1 周测试：登录、当前用户、用户列表、权限码、日志查询
# ============================================================
class TestWeek1AuthUserPermissionsLogs:
"""第 1 周：登录、当前用户、用户列表、权限码、日志查询接口。"""
def test_login_success(self, client: TestClient):
resp = client.post(f"{PREFIX}/login", json={"username": "admin", "password": "admin123"})
assert resp.status_code == 200
data = resp.json()["data"]
assert data["token"] == ADMIN_TOKEN
assert data["user"]["username"] == "admin"
def test_login_invalid(self, client: TestClient):
resp = client.post(f"{PREFIX}/login", json={"username": "admin", "password": "wrong"})
assert resp.status_code == 401
def test_me_with_valid_token(self, client: TestClient):
resp = client.get(f"{PREFIX}/me", headers=_admin_headers())
assert resp.status_code == 200
assert resp.json()["data"]["username"] == "admin"
def test_me_without_token(self, client: TestClient):
resp = client.get(f"{PREFIX}/me")
assert resp.status_code == 401
def test_users_list(self, client: TestClient):
resp = client.get(f"{PREFIX}/users", headers=_admin_headers())
assert resp.status_code == 200
users = resp.json()["data"]
assert len(users) >= 2
assert any(u["username"] == "admin" for u in users)
def test_create_user(self, client: TestClient):
resp = client.post(
f"{PREFIX}/users",
json={"username": "tester", "display_name": "Tester", "role": "viewer", "password": "test123"},
headers=_admin_headers(),
)
assert resp.status_code == 200
assert resp.json()["data"]["username"] == "tester"
def test_permission_codes(self, client: TestClient):
resp = client.get(f"{PREFIX}/system/permissions/codes")
assert resp.status_code == 200
codes = resp.json()["data"]["codes"]
assert "dashboard" in codes
assert "user-settings" in codes
def test_permissions_overview(self, client: TestClient):
resp = client.get(f"{PREFIX}/system/permissions")
assert resp.status_code == 200
data = resp.json()["data"]
assert "codes" in data
assert "roles" in data
def test_log_files(self, client: TestClient):
resp = client.get(f"{PREFIX}/log-files", headers=_admin_headers())
assert resp.status_code == 200
files = resp.json()["data"]
assert len(files) >= 1
def test_log_content(self, client: TestClient):
resp = client.get(f"{PREFIX}/log-content", params={"file": "backend.log"}, headers=_admin_headers())
assert resp.status_code == 200
assert "content" in resp.json()["data"]
def test_training_log_files(self, client: TestClient):
resp = client.get(f"{PREFIX}/training-log-files", headers=_admin_headers())
assert resp.status_code == 200
assert len(resp.json()["data"]) >= 1
def test_training_log_content(self, client: TestClient):
resp = client.get(f"{PREFIX}/training-log-content", params={"file": "ft_001.log"}, headers=_admin_headers())
assert resp.status_code == 200
assert "content" in resp.json()["data"]
# ============================================================
# 第 2 周测试：租户、项目、项目成员、资源 ACL
# ============================================================
class TestWeek2TenantProjectACL:
"""第 2 周：租户、项目、项目成员、资源 ACL。"""
def test_tenant_crud(self, client: TestClient):
# 创建
resp = client.post(f"{PREFIX}/tenants", json={"name": "Tenant-A", "code": "ta"}, headers=_admin_headers())
assert resp.status_code == 200
tid = resp.json()["data"]["id"]
# 查列表
resp = client.get(f"{PREFIX}/tenants", headers=_admin_headers())
assert resp.status_code == 200
assert any(t["id"] == tid for t in resp.json()["data"])
# 查详情
resp = client.get(f"{PREFIX}/tenants/{tid}", headers=_admin_headers())
assert resp.status_code == 200
assert resp.json()["data"]["name"] == "Tenant-A"
# 更新
resp = client.put(f"{PREFIX}/tenants/{tid}", json={"name": "Tenant-A2"}, headers=_admin_headers())
assert resp.status_code == 200
assert resp.json()["data"]["name"] == "Tenant-A2"
def test_tenant_quota(self, client: TestClient):
resp = client.post(f"{PREFIX}/tenants", json={"name": "Q-Tenant", "code": "qt"}, headers=_admin_headers())
tid = resp.json()["data"]["id"]
resp = client.put(f"{PREFIX}/tenants/{tid}/quota", json={"quota": {"gpu": 4}}, headers=_admin_headers())
assert resp.status_code == 200
def test_tenant_retention(self, client: TestClient):
resp = client.post(f"{PREFIX}/tenants", json={"name": "R-Tenant", "code": "rt"}, headers=_admin_headers())
tid = resp.json()["data"]["id"]
resp = client.put(f"{PREFIX}/tenants/{tid}/retention-policy", json={"retention_policy_id": "rpol_1"}, headers=_admin_headers())
assert resp.status_code == 200
def test_project_crud(self, client: TestClient):
# 创建项目
resp = client.post(f"{PREFIX}/projects", json={"name": "Proj-1", "code": "p1", "tenant_id": "default"}, headers=_admin_headers())
assert resp.status_code == 200
pid = resp.json()["data"]["id"]
# 查列表
resp = client.get(f"{PREFIX}/projects", params={"tenant_id": "default"}, headers=_admin_headers())
assert resp.status_code == 200
assert any(p["id"] == pid for p in resp.json()["data"])
# 查详情
resp = client.get(f"{PREFIX}/projects/{pid}", headers=_admin_headers())
assert resp.status_code == 200
assert resp.json()["data"]["name"] == "Proj-1"
# 更新
resp = client.put(f"{PREFIX}/projects/{pid}", json={"description": "updated"}, headers=_admin_headers())
assert resp.status_code == 200
# 归档
resp = client.post(f"{PREFIX}/projects/{pid}/archive", headers=_admin_headers())
assert resp.status_code == 200
assert resp.json()["data"]["status"] == "archived"
def test_project_members(self, client: TestClient):
resp = client.post(f"{PREFIX}/projects", json={"name": "Proj-M", "code": "pm", "tenant_id": "default"}, headers=_admin_headers())
pid = resp.json()["data"]["id"]
# 加成员
resp = client.post(f"{PREFIX}/projects/{pid}/members", json={"user_id": "u_op", "role": "developer"}, headers=_admin_headers())
assert resp.status_code == 200
# 列成员
resp = client.get(f"{PREFIX}/projects/{pid}/members", headers=_admin_headers())
assert resp.status_code == 200
assert len(resp.json()["data"]) >= 1
# 改角色
resp = client.put(f"{PREFIX}/projects/{pid}/members/u_op", json={"role": "maintainer"}, headers=_admin_headers())
assert resp.status_code == 200
# 删成员
resp = client.delete(f"{PREFIX}/projects/{pid}/members/u_op", headers=_admin_headers())
assert resp.status_code == 200
def test_resource_acl(self, client: TestClient):
# 设置 ACL
resp = client.put(
f"{PREFIX}/resources/model/m001/acl",
json={"entries": [{"subject_type": "user", "subject_id": "u_op", "permissions": ["read", "write"]}]},
headers=_admin_headers(),
)
assert resp.status_code == 200
result = resp.json()["data"]
assert len(result) == 1
assert set(result[0]["permissions"]) == {"read", "write"}
# 查询 ACL
resp = client.get(f"{PREFIX}/resources/model/m001/acl", headers=_admin_headers())
assert resp.status_code == 200
assert len(resp.json()["data"]) == 1
# ============================================================
# 第 3 周测试：审批实例、审批模板、审计日志查询和导出
# ============================================================
class TestWeek3ApprovalAudit:
"""第 3 周：审批实例、审批模板、审计日志查询和导出。"""
def test_approval_template_crud(self, client: TestClient):
# 创建模板
resp = client.post(f"{PREFIX}/approvals/templates", json={"name": "delete-approval", "steps": [{"approver_id": "u_admin", "status": "pending"}]}, headers=_admin_headers())
assert resp.status_code == 200
tpl_id = resp.json()["data"]["id"]
# 查列表
resp = client.get(f"{PREFIX}/approvals/templates", headers=_admin_headers())
assert resp.status_code == 200
assert any(t["id"] == tpl_id for t in resp.json()["data"])
def test_approval_instance_flow(self, client: TestClient):
# 创建审批实例
resp = client.post(f"{PREFIX}/approvals", json={
"resource_type": "dataset", "resource_id": "ds_001",
"applicant_id": "u_op",
}, headers=_admin_headers())
assert resp.status_code == 200
iid = resp.json()["data"]["id"]
# 查详情
resp = client.get(f"{PREFIX}/approvals/{iid}", headers=_admin_headers())
assert resp.status_code == 200
assert resp.json()["data"]["status"] == "pending"
# 审批决策
resp = client.post(f"{PREFIX}/approvals/{iid}/steps/0/decision", json={
"approver_id": "u_admin", "approved": True, "comment": "ok",
}, headers=_admin_headers())
assert resp.status_code == 200
assert resp.json()["data"]["status"] == "approved"
def test_approval_instance_reject(self, client: TestClient):
resp = client.post(f"{PREFIX}/approvals", json={
"resource_type": "model", "resource_id": "m_002",
"applicant_id": "u_op",
}, headers=_admin_headers())
iid = resp.json()["data"]["id"]
resp = client.post(f"{PREFIX}/approvals/{iid}/steps/0/decision", json={
"approver_id": "u_admin", "approved": False, "comment": "no",
}, headers=_admin_headers())
assert resp.status_code == 200
assert resp.json()["data"]["status"] == "rejected"
def test_approval_missing_field(self, client: TestClient):
resp = client.post(f"{PREFIX}/approvals", json={"resource_type": "dataset"}, headers=_admin_headers())
assert resp.status_code == 400
def test_audit_logs_query(self, client: TestClient):
# 通过 API 写操作触发审计
client.post(f"{PREFIX}/tenants", json={"name": "Audit-Tenant", "code": "at"}, headers=_admin_headers())
# 查询
resp = client.get(f"{PREFIX}/system/audit-logs", params={"limit": 50}, headers=_admin_headers())
assert resp.status_code == 200
data = resp.json()["data"]
assert "items" in data
assert "total" in data
assert data["total"] >= 1
def test_audit_logs_filter_by_action(self, client: TestClient):
resp = client.get(f"{PREFIX}/system/audit-logs", params={"action": "tenant.create"}, headers=_admin_headers())
assert resp.status_code == 200
items = resp.json()["data"]["items"]
assert all(i.get("action") == "tenant.create" for i in items)
def test_audit_logs_export_csv(self, client: TestClient):
resp = client.get(f"{PREFIX}/system/audit-logs/export", headers=_admin_headers())
assert resp.status_code == 200
assert "text/csv" in resp.headers.get("content-type", "")
# CSV 首行是表头
lines = resp.text.strip().split("\n")
assert "time" in lines[0]
# ============================================================
# 第 4 周测试：写操作审计、审批拦截、权限校验
# ============================================================
class TestWeek4AuditInterceptPermission:
"""第 4 周：写操作审计、审批拦截、权限校验。"""
def test_write_operation_produces_audit(self, client: TestClient, fake_store: FakePlatformStore):
# 清空审计日志便于断言
fake_store._audit_logs.clear()
# 创建租户 → 应产生 tenant.create 审计
client.post(f"{PREFIX}/tenants", json={"name": "W-Tenant", "code": "wt"}, headers=_admin_headers())
assert any(l["action"] == "tenant.create" for l in fake_store._audit_logs)
# 创建项目 → 应产生 project.create 审计
client.post(f"{PREFIX}/projects", json={"name": "W-Proj", "code": "wp", "tenant_id": "default"}, headers=_admin_headers())
assert any(l["action"] == "project.create" for l in fake_store._audit_logs)
# 设置 ACL → 应产生 resource.acl.set 审计
client.put(f"{PREFIX}/resources/model/w001/acl", json={"entries": []}, headers=_admin_headers())
assert any(l["action"] == "resource.acl.set" for l in fake_store._audit_logs)
def test_approval_intercept_on_project_archive(self, client: TestClient, fake_store: FakePlatformStore):
# 创建项目
resp = client.post(f"{PREFIX}/projects", json={"name": "I-Proj", "code": "ip", "tenant_id": "default"}, headers=_admin_headers())
pid = resp.json()["data"]["id"]
# 无待审批 → 可归档
resp = client.post(f"{PREFIX}/projects/{pid}/archive", headers=_admin_headers())
assert resp.status_code == 200
def test_approval_intercept_blocks_when_pending(self, client: TestClient, fake_store: FakePlatformStore):
# 创建项目
resp = client.post(f"{PREFIX}/projects", json={"name": "B-Proj", "code": "bp", "tenant_id": "default"}, headers=_admin_headers())
pid = resp.json()["data"]["id"]
# 注入一条待审批实例
fake_store.create_approval_instance({
"resource_type": "project",
"resource_id": pid,
"applicant_id": "u_op",
})
# 有待审批 → 归档应被拒绝
resp = client.post(f"{PREFIX}/projects/{pid}/archive", headers=_admin_headers())
assert resp.status_code == 409
def test_retention_policy_crud_with_audit(self, client: TestClient, fake_store: FakePlatformStore):
fake_store._audit_logs.clear()
# 创建
resp = client.post(f"{PREFIX}/retention-policies", json={"name": "30d-keep", "scope": "tenant"}, headers=_admin_headers())
assert resp.status_code == 200
rpid = resp.json()["data"]["id"]
assert any(l["action"] == "retention.create" for l in fake_store._audit_logs)
# 查列表
resp = client.get(f"{PREFIX}/retention-policies", headers=_admin_headers())
assert resp.status_code == 200
assert any(p["id"] == rpid for p in resp.json()["data"])
# 更新
resp = client.put(f"{PREFIX}/retention-policies/{rpid}", json={"status": "inactive"}, headers=_admin_headers())
assert resp.status_code == 200
assert resp.json()["data"]["status"] == "inactive"
# 删除
resp = client.delete(f"{PREFIX}/retention-policies/{rpid}", headers=_admin_headers())
assert resp.status_code == 200
def test_login_duration_rank_in_dashboard(self, client: TestClient):
resp = client.get(f"{PREFIX}/dashboard/stats", headers=_admin_headers())
assert resp.status_code == 200
data = resp.json()["data"]
assert "login_duration_rank" in data
assert "recent_login_users" in data
assert "service_status" in data
assert "training_7d" in data

103
backend/tests/test_platform_dataset_metadata.py
View File

@@ -1,103 +0,0 @@
from __future__ import annotations
from contextlib import contextmanager
from typing import Any, Iterator
from app.db.platform_store import PlatformStore, dataset_file_version_summary, parse_size_bytes
class _DatasetCursor:
def __init__(self, rows: list[dict[str, Any]]) -> None:
self.rows = rows
def fetchall(self) -> list[dict[str, Any]]:
return self.rows
class _DatasetConnection:
def __init__(self) -> None:
self.queries: list[str] = []
def execute(self, sql: str, params: tuple[Any, ...] | None = None) -> _DatasetCursor:
self.queries.append(sql)
if "FROM datasets dataset" in sql:
return _DatasetCursor(
[
{
"id": "dataset-train",
"name": "cash-数据集-训练集",
"type": "train",
"storage_type": "local",
"source": "task",
"task_id": "task-cash",
"source_task_id": "task-cash",
"task_name": "cash",
"size": "0 B",
"size_bytes": 0,
"metadata": "{}",
}
]
)
if "FROM dataset_files" in sql or "FROM dataset_records" in sql:
return _DatasetCursor([])
raise AssertionError(f"unexpected query: {sql}")
def test_parse_size_bytes_supports_legacy_units() -> None:
assert parse_size_bytes("21563 B") == 21563
assert parse_size_bytes("1.5 KB") == 1536
assert parse_size_bytes("2 MB") == 2 * 1024**2
assert parse_size_bytes(4096) == 4096
assert parse_size_bytes("unknown") == 0
def test_dataset_file_version_summary_uses_active_version_metadata() -> None:
summary = dataset_file_version_summary(
{
"active_version_id": "file-1-v3",
"current_version_id": "file-1-v1",
"version_no": 1,
"versions": (
'[{"id":"file-1-v1","version":1},'
'{"id":"file-1-v3","version_no":3}]'
),
}
)
assert summary == {
"active_version_id": "file-1-v3",
"current_version_id": "file-1-v3",
"current_version_no": 3,
"version_count": 2,
}
def test_dataset_file_version_summary_uses_normalized_version_number_as_fallback() -> None:
summary = dataset_file_version_summary(
{
"active_version_id": "",
"current_version_id": None,
"version_no": 1,
"versions": "[]",
}
)
assert summary["current_version_no"] == 1
assert summary["version_count"] == 0
def test_dataset_list_exposes_source_task_name() -> None:
store = PlatformStore.__new__(PlatformStore)
conn = _DatasetConnection()
@contextmanager
def connect() -> Iterator[_DatasetConnection]:
yield conn
store.connect = connect # type: ignore[method-assign]
[dataset] = store.datasets()
assert dataset["task_name"] == "cash"
assert dataset["name"] == "cash-数据集-训练集"
assert any("task.name AS task_name" in query for query in conn.queries)

54
compute/README.md
View File

@@ -25,57 +25,5 @@ compute/
## 运行模式
- 默认 `COMPUTE_EXECUTION_MODE=real`，Compute API 会通过 `compute.agent.process_manager.ProcessManager` 启动真实 `llamafactory-cli train` 子进程，并将日志写入 `TRAINING_LOG_ROOT`。
- 真实模式下 GPU 发现优先使用宿主机 `nvidia-smi`。如果部署环境暂时无法调用 `nvidia-smi`，可通过 `COMPUTE_GPU_COUNT`、`COMPUTE_GPU_NAME`、`COMPUTE_GPU_MEMORY_GB`、`COMPUTE_GPU_POWER_LIMIT_W` 声明兼容 GPU 清单，便于应用侧先完成节点登记和联调。
- 默认 `COMPUTE_EXECUTION_MODE=real`，Compute API 只暴露健康检查和接口契约；真实训练执行器完成前，创建作业会返回未实现错误。
- 仅隔离联调时可设置 `COMPUTE_EXECUTION_MODE=simulator`，启用内存状态机和合成 GPU/日志数据。该模式不得作为生产运行路径。
- 服务间鉴权默认开启：设置 `COMPUTE_AUTH_ENABLED=true` 和一致的 `COMPUTE_SERVICE_TOKEN`，应用侧会通过 `X-Compute-Token` 调用 Compute API。
- 真实训练作业会登记到 `TRAINING_LOG_ROOT/compute-jobs.json`。Compute API 重启后会恢复作业索引，继续提供状态、停止和日志查询。
- 同一算力节点内按 GPU ID 做轻量锁定；已有运行中作业占用的 GPU 不允许再次提交，避免同机多 GPU 场景下误复用。
真实执行前提：
- 镜像或宿主机环境中 `llamafactory-cli` 可执行。
- `LLAMA_FACTORY_HOME` 指向 LLaMA-Factory 工作目录。
- 基座模型路径和数据集名称/目录已经在算力服务器本地可访问。
- 应用侧训练任务中的 GPU、模型、数据集配置能映射到当前节点本地路径。
## 应用侧接入
应用平台通过“算力节点”页面维护每台 GPU 服务器的 `Compute API` 和 `File Gateway` 地址。点击连接测试时，Backend API 会主动调用：
```text
GET /modelTF/v1/compute/health
GET /modelTF/compute/resources/gpus
```
连接成功后，应用侧会同步节点健康信息、能力标签和 GPU 清单到 PostgreSQL。多节点阶段仍按“每台算力服务器 = 单机多 GPU 节点”管理，每台服务器都部署 Compute API、Agent、File Gateway 契约和 LLaMA-Factory。
训练闭环：
```text
Frontend 创建/启动训练
-> Backend API 选择 compute_nodes 节点
-> Backend API POST /modelTF/compute/jobs 到目标 Compute API
-> Compute API 启动 llamafactory-cli 子进程
-> Backend Worker 定时 GET /modelTF/compute/jobs/{id}
-> Backend API 同步 fine_tune_tasks 状态、进度、PID、日志路径和产物索引
```
## 当前接口能力
日志接口：
```text
GET /modelTF/compute/jobs/{job_id}/logs?tail_lines=200
GET /modelTF/compute/jobs/{job_id}/logs?offset=0&limit=500
```
返回 `content`、`metrics`、`total_lines`、`offset`、`limit`、`has_more`、`next_offset`，用于前端增量刷新和日志平台采集。
文件导入：
```text
POST /modelTF/compute/files/import-local
```
该接口用于应用侧调度前把算力服务器本地可访问的模型/数据集路径导入到 `YG_FT_DATA_ROOT` 内部。目标路径会校验不能逃逸出 `YG_FT_DATA_ROOT`，源路径必须已存在于算力服务器本地或挂载目录。

281
compute/agent/process_manager.py
View File

@@ -1,281 +0,0 @@
from __future__ import annotations
import os
import json
import contextlib
import hashlib
import signal
import subprocess
import time
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any
TERMINAL_STATUSES = {"completed", "failed", "stopped"}
@dataclass
class ManagedProcess:
id: str
name: str
command: list[str]
work_dir: str
log_path: Path
output_dir: str
gpus: list[int]
process: subprocess.Popen[Any] | None
created_at: float
pid: int | None = None
status: str = "running"
progress: int = 5
artifacts: list[dict[str, Any]] = field(default_factory=list)
class ProcessManager:
def __init__(self, log_root: str) -> None:
self.log_root = Path(log_root)
self.log_root.mkdir(parents=True, exist_ok=True)
self.registry_path = self.log_root / "compute-jobs.json"
self.jobs: dict[str, ManagedProcess] = {}
self._load_registry()
def create_job(self, payload: dict[str, Any], command: list[str], work_dir: str) -> dict[str, Any]:
job_id = str(payload.get("id") or f"job_{int(time.time() * 1000)}")
if job_id in self.jobs and self.jobs[job_id].status not in TERMINAL_STATUSES:
raise ValueError(f"job {job_id} is already running")
output_dir = str(payload.get("output_dir") or f"/data/yg-ft/outputs/{payload.get('name', job_id)}")
Path(output_dir).mkdir(parents=True, exist_ok=True)
log_path = self.log_root / f"{job_id}.log"
env = os.environ.copy()
gpus = [int(item) for item in payload.get("gpus") or []]
locked = self.locked_gpus()
conflict = sorted(set(gpus).intersection(locked))
if conflict:
raise ValueError(f"gpu already locked: {conflict}")
if gpus:
env["CUDA_VISIBLE_DEVICES"] = ",".join(str(item) for item in gpus)
env.update({str(k): str(v) for k, v in payload.get("env", {}).items()})
cwd = work_dir if Path(work_dir).exists() else None
with log_path.open("ab") as log_file:
log_file.write(f"[INFO] starting job_id={job_id} command={' '.join(command)}\n".encode("utf-8"))
process = subprocess.Popen(
command,
cwd=cwd,
env=env,
stdout=log_file,
stderr=subprocess.STDOUT,
)
managed = ManagedProcess(
id=job_id,
name=str(payload.get("name") or job_id),
command=command,
work_dir=work_dir,
log_path=log_path,
output_dir=output_dir,
gpus=gpus,
process=process,
created_at=time.time(),
pid=process.pid,
progress=10,
)
self.jobs[job_id] = managed
data = self.serialize(managed)
self._save_registry()
return data
def get_job(self, job_id: str) -> dict[str, Any] | None:
job = self.jobs.get(job_id)
if not job:
return None
return self.serialize(job)
def list_jobs(self) -> list[dict[str, Any]]:
return [self.serialize(job) for job in self.jobs.values()]
def stop_job(self, job_id: str) -> dict[str, Any] | None:
job = self.jobs.get(job_id)
if not job:
return None
if job.status not in TERMINAL_STATUSES:
try:
if job.process is not None and os.name == "nt":
job.process.terminate()
elif job.pid is not None:
os.kill(job.pid, signal.SIGTERM)
if job.process is not None:
job.process.wait(timeout=10)
except Exception:
if job.process is not None:
job.process.kill()
elif job.pid is not None:
with contextlib.suppress(Exception):
os.kill(job.pid, signal.SIGKILL)
job.status = "stopped"
job.progress = min(job.progress, 99)
data = self.serialize(job)
self._save_registry()
return data
def logs(self, job_id: str) -> str:
job = self.jobs.get(job_id)
if not job or not job.log_path.exists():
return ""
return job.log_path.read_text(encoding="utf-8", errors="replace")
def serialize(self, job: ManagedProcess) -> dict[str, Any]:
code = job.process.poll() if job.process is not None else None
checkpoints = self._collect_checkpoints(job.output_dir)
if job.status not in TERMINAL_STATUSES:
if job.process is None and job.pid is not None and not self._pid_alive(job.pid):
job.status = "failed"
job.progress = min(job.progress, 99)
code = -1
elif code is None:
job.status = "running"
elapsed = max(0, int(time.time() - job.created_at))
job.progress = min(95, max(job.progress, 10 + elapsed // 6))
elif code == 0:
job.status = "completed"
job.progress = 100
job.artifacts = self._collect_artifacts(job.output_dir)
else:
job.status = "failed"
job.progress = min(job.progress, 99)
self._save_registry()
return {
"id": job.id,
"name": job.name,
"status": job.status,
"progress": job.progress,
"pid": job.pid,
"gpus": job.gpus,
"created_at": job.created_at,
"command": job.command,
"work_dir": job.work_dir,
"output_dir": job.output_dir,
"log_file": str(job.log_path),
"artifacts": job.artifacts,
"checkpoints": checkpoints,
"return_code": code,
}
def locked_gpus(self) -> set[int]:
locked: set[int] = set()
for job in self.jobs.values():
status = self.serialize(job)["status"]
if status in {"queued", "running"}:
locked.update(job.gpus)
return locked
def _collect_artifacts(self, output_dir: str) -> list[dict[str, Any]]:
root = Path(output_dir)
if not root.exists():
return []
artifacts: list[dict[str, Any]] = []
for path in root.rglob("*"):
if path.is_file():
digest = hashlib.sha256()
with path.open("rb") as handle:
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
digest.update(chunk)
size = path.stat().st_size
artifacts.append(
{
"path": str(path),
"name": path.name,
"size": size,
"size_bytes": size,
"checksum_sha256": digest.hexdigest(),
}
)
return artifacts[:200]
def _collect_checkpoints(self, output_dir: str) -> list[dict[str, Any]]:
root = Path(output_dir)
if not root.exists():
return []
checkpoints: list[dict[str, Any]] = []
for path in root.glob("checkpoint-*"):
if not path.is_dir():
continue
step = 0
try:
step = int(path.name.rsplit("-", 1)[-1])
except ValueError:
step = 0
size_bytes = sum(item.stat().st_size for item in path.rglob("*") if item.is_file())
checkpoints.append(
{
"step": step,
"name": path.name,
"path": str(path),
"size_bytes": size_bytes,
"create_time": path.stat().st_mtime,
}
)
return sorted(checkpoints, key=lambda item: (int(item.get("step") or 0), str(item.get("name") or "")))
def _save_registry(self) -> None:
items = []
for job in self.jobs.values():
items.append(
{
"id": job.id,
"name": job.name,
"command": job.command,
"work_dir": job.work_dir,
"log_path": str(job.log_path),
"output_dir": job.output_dir,
"gpus": job.gpus,
"pid": job.pid,
"created_at": job.created_at,
"status": job.status,
"progress": job.progress,
"artifacts": job.artifacts,
}
)
self.registry_path.write_text(json.dumps(items, ensure_ascii=False, indent=2), encoding="utf-8")
def _load_registry(self) -> None:
if not self.registry_path.exists():
return
try:
items = json.loads(self.registry_path.read_text(encoding="utf-8"))
except json.JSONDecodeError:
return
for item in items if isinstance(items, list) else []:
if not isinstance(item, dict):
continue
pid = item.get("pid")
status = item.get("status", "failed")
if status not in TERMINAL_STATUSES and pid and not self._pid_alive(int(pid)):
status = "failed"
job = ManagedProcess(
id=str(item["id"]),
name=str(item.get("name") or item["id"]),
command=[str(part) for part in item.get("command") or []],
work_dir=str(item.get("work_dir") or ""),
log_path=Path(item.get("log_path") or self.log_root / f"{item['id']}.log"),
output_dir=str(item.get("output_dir") or ""),
gpus=[int(gpu) for gpu in item.get("gpus") or []],
process=None,
pid=int(pid) if pid else None,
created_at=float(item.get("created_at") or time.time()),
status=status,
progress=int(item.get("progress") or 0),
artifacts=item.get("artifacts") or [],
)
self.jobs[job.id] = job
def _pid_alive(self, pid: int) -> bool:
if pid <= 0:
return False
try:
os.kill(pid, 0)
return True
except OSError:
return False

716
compute/api/main.py
View File

@@ -1,43 +1,20 @@
from __future__ import annotations
import asyncio
import json
import os
import math
import hashlib
import shutil
import subprocess
import time
from pathlib import Path
from typing import Any
from fastapi import FastAPI, File, Form, HTTPException, Query, Request, UploadFile
from fastapi.responses import FileResponse, JSONResponse, StreamingResponse
from fastapi import FastAPI, HTTPException
from compute.agent.process_manager import ProcessManager
from compute.api.security import docs_kwargs
from compute.engines.llama_factory.adapter import build_command, parse_log_line, prepare_runtime_files
from compute.engines.llama_factory.inference import get_inference_session
from compute.engines.llama_factory.adapter import build_command, parse_log_line
def create_app() -> FastAPI:
app = FastAPI(title="YG Fine-Tune Compute API", **docs_kwargs())
app = FastAPI(title="YG Fine-Tune Compute API")
jobs: dict[str, dict[str, Any]] = {}
route_prefix = os.getenv("MODELTF_ROUTE_PREFIX", "/modelTF").rstrip("/") or "/modelTF"
process_manager = ProcessManager(os.getenv("TRAINING_LOG_ROOT", "/opt/yg-ft/logs/training"))
@app.middleware("http")
async def compute_token_auth(request: Request, call_next):
token = os.getenv("COMPUTE_SERVICE_TOKEN", "")
auth_enabled = os.getenv("COMPUTE_AUTH_ENABLED", "true").lower() == "true"
public_paths = {f"{route_prefix}/health", "/health"}
if auth_enabled and token and request.url.path not in public_paths:
header_token = request.headers.get("x-compute-token", "")
auth_header = request.headers.get("authorization", "")
bearer_token = auth_header.removeprefix("Bearer ").strip() if auth_header.startswith("Bearer ") else ""
if header_token != token and bearer_token != token:
return JSONResponse({"detail": "invalid compute service token"}, status_code=401)
return await call_next(request)
def now() -> float:
return time.time()
@@ -48,110 +25,6 @@ def create_app() -> FastAPI:
def execution_mode() -> str:
return os.getenv("COMPUTE_EXECUTION_MODE", os.getenv("COMPUTE_MODE", "real")).lower()
def _int_env(name: str, default: int) -> int:
raw = os.getenv(name)
if raw is None or raw == "":
return default
return int(raw)
def _float_env(name: str, default: float) -> float:
raw = os.getenv(name)
if raw is None or raw == "":
return default
return float(raw)
def _path_inside(root: Path, candidate: Path) -> bool:
try:
candidate.resolve().relative_to(root.resolve())
return True
except ValueError:
return False
def _llama_factory_version() -> str:
for command in (["llamafactory-cli", "version"], ["llamafactory-cli", "--version"]):
try:
result = subprocess.run(command, capture_output=True, text=True, timeout=5)
except Exception:
continue
output = (result.stdout or result.stderr).strip()
if result.returncode == 0 and output:
return output.splitlines()[0][:120]
return ""
def torch_cuda_status() -> dict[str, Any]:
try:
import torch # type: ignore[import-not-found]
except Exception as exc: # noqa: BLE001 - keep health endpoint resilient
return {
"available": False,
"device_count": 0,
"torch_version": "",
"torch_cuda_version": "",
"error": f"torch import failed: {exc}",
}
try:
available = bool(torch.cuda.is_available())
device_count = int(torch.cuda.device_count())
devices = []
for index in range(device_count):
props = torch.cuda.get_device_properties(index)
devices.append(
{
"index": index,
"name": props.name,
"memory_total_gb": round(props.total_memory / 1024 / 1024 / 1024, 2),
}
)
return {
"available": available,
"device_count": device_count,
"torch_version": str(torch.__version__),
"torch_cuda_version": str(torch.version.cuda or ""),
"devices": devices,
"error": "" if available else "torch cuda is not available",
}
except Exception as exc: # noqa: BLE001 - expose CUDA initialization failures
return {
"available": False,
"device_count": 0,
"torch_version": str(getattr(torch, "__version__", "")),
"torch_cuda_version": str(getattr(torch.version, "cuda", "") or ""),
"devices": [],
"error": str(exc),
}
def _slice_log_content(
content: str,
tail_lines: int | None = None,
offset: int | None = None,
limit: int | None = None,
) -> dict[str, Any]:
lines = content.splitlines()
total = len(lines)
if offset is not None or limit is not None:
start = max(0, offset or 0)
end = start + limit if limit else total
selected = lines[start:end]
else:
tail = tail_lines or 200
start = max(0, total - tail)
selected = lines[start:]
next_offset = start + len(selected)
return {
"content": "\n".join(selected),
"total_lines": total,
"offset": start,
"limit": len(selected),
"has_more": next_offset < total,
"next_offset": next_offset if next_offset < total else None,
}
def _safe_float(value: Any, default: float = 0) -> float:
try:
return float(str(value).replace("[N/A]", "").strip() or default)
except (TypeError, ValueError):
return default
def job_status(job: dict[str, Any]) -> dict[str, Any]:
if execution_mode() != "simulator":
return job
@@ -201,80 +74,9 @@ def create_app() -> FastAPI:
)
return "\n".join(lines)
def real_gpu_resources() -> list[dict[str, Any]]:
query = (
"index,uuid,name,memory.total,memory.used,utilization.gpu,"
"temperature.gpu,power.draw,power.limit"
)
try:
result = subprocess.run(
["nvidia-smi", f"--query-gpu={query}", "--format=csv,noheader,nounits"],
check=True,
capture_output=True,
text=True,
timeout=5,
)
except Exception:
return fallback_gpu_resources()
items: list[dict[str, Any]] = []
for line in result.stdout.splitlines():
parts = [part.strip() for part in line.split(",")]
if len(parts) < 9:
continue
idx, uuid, name, mem_total, mem_used, util, temp, power, power_limit = parts[:9]
total_gb = round(_safe_float(mem_total) / 1024, 2)
used_gb = round(_safe_float(mem_used) / 1024, 2)
memory_percent = round(used_gb / total_gb * 100, 1) if total_gb else 0
gpu_percent = int(_safe_float(util))
items.append(
{
"id": int(idx),
"gpu_index": int(idx),
"uuid": uuid,
"name": name,
"status": "busy" if gpu_percent >= 5 or used_gb > 1 else "idle",
"gpu_percent": gpu_percent,
"memory_used_gb": used_gb,
"memory_total_gb": total_gb,
"memory_percent": memory_percent,
"temperature": int(_safe_float(temp)),
"power_w": round(_safe_float(power), 1),
"power_limit_w": round(_safe_float(power_limit), 1),
"processes": [],
}
)
return items
def fallback_gpu_resources() -> list[dict[str, Any]]:
count = _int_env("COMPUTE_GPU_COUNT", 0)
if count <= 0:
return []
name = os.getenv("COMPUTE_GPU_NAME", "Configured GPU")
memory_total = _float_env("COMPUTE_GPU_MEMORY_GB", 80.0)
power_limit = _float_env("COMPUTE_GPU_POWER_LIMIT_W", 300.0)
return [
{
"id": idx,
"gpu_index": idx,
"uuid": f"GPU-{host_id().upper()}-{idx}",
"name": name,
"status": "idle",
"gpu_percent": 0,
"memory_used_gb": 0,
"memory_total_gb": memory_total,
"memory_percent": 0,
"temperature": _int_env("COMPUTE_GPU_BASE_TEMPERATURE", 35),
"power_w": 0,
"power_limit_w": power_limit,
"processes": [],
}
for idx in range(count)
]
def gpu_resources() -> list[dict[str, Any]]:
if execution_mode() != "simulator":
return real_gpu_resources()
return []
active_jobs = [job_status(job) for job in jobs.values() if job["status"] in {"queued", "running"}]
gpus: list[dict[str, Any]] = []
for idx in range(4):
@@ -307,191 +109,6 @@ def create_app() -> FastAPI:
)
return gpus
def _validate_training_accelerator(payload: dict[str, Any]) -> tuple[list[str], list[str], dict[str, Any]]:
errors: list[str] = []
warnings: list[str] = []
if str(payload.get("engine") or payload.get("training_engine") or "llama_factory") == "smoke":
return errors, warnings, {}
requested_gpus = [int(item) for item in payload.get("gpus") or []]
if not requested_gpus:
warnings.append("no gpu selected; training will run on CPU")
return errors, warnings, {}
cuda = torch_cuda_status()
if not cuda.get("available"):
errors.append(f"torch cuda unavailable on compute node: {cuda.get('error') or 'unknown error'}")
device_count = int(cuda.get("device_count") or 0)
if device_count and max(requested_gpus) >= device_count:
errors.append(f"requested gpu index out of torch device range: requested={requested_gpus}, device_count={device_count}")
min_memory_gb = _float_env("MIN_TRAINING_GPU_MEMORY_GB", 4.0)
gpus = {int(item["gpu_index"]): item for item in gpu_resources() if "gpu_index" in item}
for gpu_index in requested_gpus:
gpu = gpus.get(gpu_index)
if not gpu:
errors.append(f"requested gpu not found by nvidia-smi: {gpu_index}")
continue
memory_total = float(gpu.get("memory_total_gb") or 0)
if memory_total and memory_total < min_memory_gb:
errors.append(
f"gpu {gpu_index} memory too small: {memory_total}GB < required {min_memory_gb}GB"
)
return errors, warnings, cuda
def _check_path_item(item: dict[str, Any]) -> dict[str, Any]:
path = Path(str(item.get("path") or ""))
exists = path.exists()
expected_type = str(item.get("type") or "any")
ok = exists
if exists and expected_type == "dir":
ok = path.is_dir()
if exists and expected_type == "file":
ok = path.is_file()
return {
"name": item.get("name") or "",
"path": str(path),
"type": expected_type,
"required": bool(item.get("required", True)),
"exists": exists,
"is_dir": path.is_dir() if exists else False,
"is_file": path.is_file() if exists else False,
"byte_size": sum(child.stat().st_size for child in path.rglob("*") if child.is_file()) if exists and path.is_dir() else path.stat().st_size if exists and path.is_file() else 0,
"ok": ok or not item.get("required", True),
}
def _job_preview(payload: dict[str, Any], check_paths: bool) -> dict[str, Any]:
warnings: list[str] = []
runtime_files: list[dict[str, str]] = []
command_payload = {**payload, "require_dataset_files": check_paths}
if check_paths:
try:
runtime_files = prepare_runtime_files(command_payload)
except OSError as exc:
return {
"valid": False,
"errors": [f"prepare runtime files failed: {exc}"],
"warnings": warnings,
"engine": str(payload.get("engine") or payload.get("training_engine") or "llama_factory"),
"command": [],
"command_text": "",
"work_dir": os.getenv("LLAMA_FACTORY_HOME", "/app/LLaMA-Factory"),
"env": {},
"runtime_files": [],
"path_checks": [],
}
try:
command = build_command(command_payload, os.getenv("LLAMA_FACTORY_HOME", "/app/LLaMA-Factory"))
except ValueError as exc:
return {
"valid": False,
"errors": [part.strip() for part in str(exc).split(";") if part.strip()],
"warnings": warnings,
"engine": str(payload.get("engine") or payload.get("training_engine") or "llama_factory"),
"command": [],
"command_text": "",
"work_dir": os.getenv("LLAMA_FACTORY_HOME", "/app/LLaMA-Factory"),
"env": {},
"runtime_files": runtime_files,
"path_checks": [],
}
errors: list[str] = []
engine = str(payload.get("engine") or payload.get("training_engine") or "llama_factory")
path_checks: list[dict[str, Any]] = []
accelerator: dict[str, Any] = {}
if check_paths and engine != "smoke":
path_checks = [
_check_path_item(
{
"name": "model_name_or_path",
"path": payload.get("model_name_or_path") or payload.get("base_model") or payload.get("base_model_path") or "",
"type": "any",
"required": True,
}
)
]
if engine in {"merge", "export", "llama_factory_export"} and payload.get("adapter_name_or_path"):
path_checks.append(
_check_path_item(
{
"name": "adapter_name_or_path",
"path": payload.get("adapter_name_or_path"),
"type": "any",
"required": True,
}
)
)
if payload.get("dataset_dir"):
path_checks.append(
_check_path_item(
{
"name": "dataset_dir",
"path": payload.get("dataset_dir"),
"type": "dir",
"required": True,
}
)
)
output_dir = Path(str(payload.get("output_dir") or "/data/yg-ft/outputs/training-job"))
path_checks.append(
_check_path_item(
{
"name": "output_parent",
"path": str(output_dir.parent),
"type": "dir",
"required": False,
}
)
)
errors.extend(
[f"{item['name']} path not available: {item['path']}" for item in path_checks if not item["ok"] and item["required"]]
)
if shutil.which(command.command[0]) is None:
errors.append(f"training command not found: {command.command[0]}")
if not Path(command.work_dir).exists():
errors.append(f"llama_factory_home not found: {command.work_dir}")
if engine not in {"merge", "export", "llama_factory_export"}:
accelerator_errors, accelerator_warnings, accelerator = _validate_training_accelerator(payload)
errors.extend(accelerator_errors)
warnings.extend(accelerator_warnings)
elif engine == "eval":
# Eval engine: validate model path and dataset path
if not payload.get("model_name_or_path"):
errors.append("model_name_or_path is required for eval")
else:
path_checks.append(_check_path_item({
"name": "model_name_or_path",
"path": payload.get("model_name_or_path", ""),
"type": "any",
"required": True,
}))
if payload.get("dataset_path"):
path_checks.append(_check_path_item({
"name": "dataset_path",
"path": payload.get("dataset_path", ""),
"type": "file",
"required": True,
}))
else:
errors.append("dataset_path is required for eval")
if shutil.which("python") is None:
errors.append("python runtime not found")
elif engine == "smoke":
warnings.append("smoke engine skips model and dataset path checks")
return {
"valid": not errors,
"errors": errors,
"warnings": warnings,
"engine": engine,
"command": command.command,
"command_text": " ".join(command.command),
"work_dir": command.work_dir,
"env": command.env,
"runtime_files": runtime_files,
"accelerator": accelerator,
"path_checks": path_checks,
}
@app.get(f"{route_prefix}/health")
async def health_check() -> dict[str, str]:
return {
@@ -499,130 +116,41 @@ def create_app() -> FastAPI:
"compute_host_id": os.getenv("COMPUTE_HOST_ID", "unknown"),
}
@app.get("/health")
async def health_check_root() -> dict[str, str]:
return await health_check()
@app.get(f"{route_prefix}/v1/compute/health")
async def compute_health_check() -> dict[str, Any]:
async def compute_health_check() -> dict[str, str | bool]:
data_root = Path(os.getenv("YG_FT_DATA_ROOT", "/data/yg-ft"))
dataset_root = Path(os.getenv("YG_FT_DATASET_ROOT", str(data_root / "datasets")))
output_root = Path(os.getenv("YG_FT_OUTPUT_ROOT", str(data_root / "outputs")))
llama_factory_home = Path(os.getenv("LLAMA_FACTORY_HOME", "/app/LLaMA-Factory"))
gpu_items = gpu_resources()
torch_cuda = torch_cuda_status()
return {
"status": "ok",
"api_version": "v1",
"compute_host_id": os.getenv("COMPUTE_HOST_ID", "unknown"),
"app_callback_enabled": os.getenv("ENABLE_APP_CALLBACK", "false").lower() == "true",
"data_root": str(data_root),
"data_root_exists": data_root.exists(),
"model_root": os.getenv("YG_FT_MODEL_ROOT", str(data_root / "models")),
"dataset_root": str(dataset_root),
"dataset_root_exists": dataset_root.exists(),
"output_root": str(output_root),
"output_root_exists": output_root.exists(),
"log_root": os.getenv("TRAINING_LOG_ROOT", "/opt/yg-ft/logs/training"),
"llama_factory_home": str(llama_factory_home),
"llama_factory_home_exists": llama_factory_home.exists(),
"llama_factory_version": os.getenv("LLAMA_FACTORY_VERSION", ""),
"execution_mode": execution_mode(),
"gpu_count": _int_env("COMPUTE_GPU_COUNT", 0),
"nvidia_gpu_count": len(gpu_items),
"torch_cuda": torch_cuda,
"gpu_discovery_endpoint": f"{route_prefix}/compute/resources/gpus",
"capabilities": ["gpu_discovery", "torch_cuda_diagnostics", "llama_factory", "file_gateway", "job_polling"],
}
@app.get(f"{route_prefix}/v1/compute/jobs")
async def list_jobs_alias() -> dict[str, list[dict[str, Any]]]:
items = process_manager.list_jobs() if execution_mode() != "simulator" else [job_status(job) for job in jobs.values()]
return {"items": items}
return {"items": [job_status(job) for job in jobs.values()]}
@app.get(f"{route_prefix}/compute/resources/gpus")
async def list_gpus() -> dict[str, Any]:
return {"items": gpu_resources(), "compute_host_id": host_id()}
@app.get(f"{route_prefix}/v1/compute/resources/gpus")
async def list_gpus_v1() -> dict[str, Any]:
return {"items": gpu_resources(), "compute_host_id": host_id()}
@app.post(f"{route_prefix}/compute/jobs/preview")
async def preview_job(payload: dict[str, Any]) -> dict[str, Any]:
return _job_preview(payload, check_paths=False)
@app.post(f"{route_prefix}/compute/jobs/validate")
async def validate_job(payload: dict[str, Any]) -> dict[str, Any]:
return _job_preview(payload, check_paths=True)
@app.post(f"{route_prefix}/v1/compute/jobs/preview")
async def preview_job_v1(payload: dict[str, Any]) -> dict[str, Any]:
return await preview_job(payload)
@app.post(f"{route_prefix}/v1/compute/jobs/validate")
async def validate_job_v1(payload: dict[str, Any]) -> dict[str, Any]:
return await validate_job(payload)
@app.post(f"{route_prefix}/compute/files/check-paths")
async def check_paths(payload: dict[str, Any]) -> dict[str, Any]:
items = [_check_path_item(item) for item in payload.get("paths", []) if isinstance(item, dict)]
return {"valid": all(item["ok"] for item in items), "items": items}
@app.get(f"{route_prefix}/compute/files/list")
async def list_files(
root: str = Query(default="data"),
relative_path: str = Query(default=""),
directories_only: bool = Query(default=False),
) -> dict[str, Any]:
roots = {
"data": Path(os.getenv("YG_FT_DATA_ROOT", "/data/yg-ft")),
"models": Path(os.getenv("YG_FT_MODEL_ROOT", os.getenv("YG_FT_DATA_ROOT", "/data/yg-ft") + "/models")),
"datasets": Path(os.getenv("YG_FT_DATASET_ROOT", os.getenv("YG_FT_DATA_ROOT", "/data/yg-ft") + "/datasets")),
"outputs": Path(os.getenv("YG_FT_OUTPUT_ROOT", os.getenv("YG_FT_DATA_ROOT", "/data/yg-ft") + "/outputs")),
}
base = roots.get(root)
if base is None:
raise HTTPException(status_code=400, detail="invalid root")
target = (base / relative_path.lstrip("/\\")).resolve()
if not _path_inside(base, target):
raise HTTPException(status_code=400, detail="path must stay inside selected root")
if not target.exists():
return {"root": root, "base_path": str(base), "relative_path": relative_path, "items": []}
items = []
for child in sorted(target.iterdir(), key=lambda path: (not path.is_dir(), path.name.lower())):
if directories_only and not child.is_dir():
continue
items.append(
{
"name": child.name,
"path": str(child),
"relative_path": str(child.relative_to(base)).replace("\\", "/"),
"type": "directory" if child.is_dir() else "file",
"byte_size": child.stat().st_size if child.is_file() else 0,
}
)
return {"root": root, "base_path": str(base), "relative_path": relative_path, "items": items}
@app.post(f"{route_prefix}/compute/jobs")
async def create_job(payload: dict[str, Any]) -> dict[str, Any]:
payload = {**payload, "require_dataset_files": True}
try:
prepare_runtime_files(payload)
except OSError as exc:
raise HTTPException(status_code=400, detail=f"prepare runtime files failed: {exc}")
try:
command = build_command(payload, os.getenv("LLAMA_FACTORY_HOME", "/app/LLaMA-Factory"))
except ValueError as exc:
raise HTTPException(status_code=400, detail=str(exc))
job_id = str(payload.get("id") or f"job_{int(now() * 1000)}")
if execution_mode() != "simulator":
try:
return process_manager.create_job({**payload, "id": job_id}, command.command, command.work_dir)
except FileNotFoundError as exc:
raise HTTPException(status_code=500, detail=f"training command not found: {exc.filename}")
except ValueError as exc:
raise HTTPException(status_code=409, detail=str(exc))
raise HTTPException(
status_code=501,
detail="real compute executor is not implemented yet; set COMPUTE_EXECUTION_MODE=simulator only for isolated development",
)
job_id = str(payload.get("id") or f"job_{int(now() * 1000)}")
job = {
"id": job_id,
"name": payload.get("name", job_id),
@@ -641,29 +169,17 @@ def create_app() -> FastAPI:
@app.get(f"{route_prefix}/compute/jobs")
async def list_jobs() -> dict[str, Any]:
items = process_manager.list_jobs() if execution_mode() != "simulator" else [job_status(job) for job in jobs.values()]
return {"items": items}
return {"items": [job_status(job) for job in jobs.values()]}
@app.get(f"{route_prefix}/compute/jobs/{{job_id}}")
async def get_job(job_id: str) -> dict[str, Any]:
job = jobs.get(job_id)
if execution_mode() != "simulator":
job = process_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="job not found")
return job
job = jobs.get(job_id)
if not job:
raise HTTPException(status_code=404, detail="job not found")
return job_status(job)
@app.post(f"{route_prefix}/compute/jobs/{{job_id}}/stop")
async def stop_job(job_id: str) -> dict[str, Any]:
if execution_mode() != "simulator":
job = process_manager.stop_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="job not found")
return job
job = jobs.get(job_id)
if not job:
raise HTTPException(status_code=404, detail="job not found")
@@ -672,206 +188,22 @@ def create_app() -> FastAPI:
return job
@app.get(f"{route_prefix}/compute/jobs/{{job_id}}/logs")
async def job_logs(
job_id: str,
tail_lines: int | None = Query(default=200, ge=1, le=5000),
offset: int | None = Query(default=None, ge=0),
limit: int | None = Query(default=None, ge=1, le=5000),
) -> dict[str, Any]:
if execution_mode() != "simulator":
job = process_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="job not found")
content = process_manager.logs(job_id)
else:
job = jobs.get(job_id)
if not job:
raise HTTPException(status_code=404, detail="job not found")
job = job_status(job)
content = job["logs"]
window = _slice_log_content(content, tail_lines, offset, limit)
metrics = [parse_log_line(line) for line in window["content"].splitlines()]
return {"job_id": job_id, **window, "metrics": [m for m in metrics if m]}
# ── Inference Endpoints ───────────────────────────────────────────
@app.post(f"{route_prefix}/inference/load")
async def inference_load(payload: dict[str, Any]) -> dict[str, Any]:
"""Load a model for inference using LLaMA-Factory ChatModel.
Expected payload:
model_name_or_path: str (required)
adapter_name_or_path: str (optional, for LoRA adapters)
template: str (default: "qwen")
infer_backend: str (default: "huggingface")
infer_dtype: str (default: "auto")
"""
session = get_inference_session()
result = session.load(
model_name_or_path=payload.get("model_name_or_path", ""),
adapter_name_or_path=payload.get("adapter_name_or_path", ""),
template=payload.get("template", "qwen"),
infer_backend=payload.get("infer_backend", "huggingface"),
infer_dtype=payload.get("infer_dtype", "auto"),
)
return result
@app.post(f"{route_prefix}/inference/unload")
async def inference_unload() -> dict[str, Any]:
"""Unload the currently loaded model and free GPU memory."""
# Teardown (gc.collect + cuda.empty_cache) can take a while; run it off
# the event loop so /health and /inference/status stay responsive.
return await asyncio.to_thread(get_inference_session().unload)
@app.get(f"{route_prefix}/inference/status")
async def inference_status() -> dict[str, Any]:
"""Get the current inference session status."""
return get_inference_session().info()
@app.post(f"{route_prefix}/inference/chat")
async def inference_chat(payload: dict[str, Any]) -> dict[str, Any]:
"""Chat with the loaded model (non-streaming).
Expected payload:
messages: list[dict] (OpenAI format)
temperature: float (default 0.95)
top_p: float (default 0.7)
max_new_tokens: int (default 1024)
"""
messages = payload.get("messages") or []
if not messages:
raise HTTPException(status_code=400, detail="messages is required")
# Generation is long-running; run it in a thread so the event loop keeps
# serving /inference/status and /health during inference.
result = await asyncio.to_thread(
get_inference_session().chat,
messages=messages,
temperature=float(payload.get("temperature", 0.95)),
top_p=float(payload.get("top_p", 0.7)),
max_new_tokens=int(payload.get("max_new_tokens", 1024)),
do_sample=bool(payload.get("do_sample", True)),
)
if result.get("error"):
raise HTTPException(status_code=500, detail=result["error"])
return {"response": result["response"]}
@app.post(f"{route_prefix}/inference/chat/stream")
async def inference_chat_stream(payload: dict[str, Any]) -> StreamingResponse:
"""Chat with streaming response (Server-Sent Events)."""
messages = payload.get("messages") or []
if not messages:
raise HTTPException(status_code=400, detail="messages is required")
def generate():
session = get_inference_session()
for chunk in session.chat_stream(
messages=messages,
temperature=float(payload.get("temperature", 0.95)),
top_p=float(payload.get("top_p", 0.7)),
max_new_tokens=int(payload.get("max_new_tokens", 1024)),
do_sample=bool(payload.get("do_sample", True)),
):
yield chunk
return StreamingResponse(generate(), media_type="text/event-stream")
async def job_logs(job_id: str) -> dict[str, Any]:
job = jobs.get(job_id)
if not job:
raise HTTPException(status_code=404, detail="job not found")
job = job_status(job)
metrics = [parse_log_line(line) for line in job["logs"].splitlines()]
return {"job_id": job_id, "content": job["logs"], "metrics": [m for m in metrics if m]}
@app.post(f"{route_prefix}/compute/files/upload")
async def upload_file(
file: UploadFile | None = File(default=None),
target_relative_path: str | None = Form(default=None),
resource_type: str | None = Form(default=None),
resource_id: str | None = Form(default=None),
) -> dict[str, Any]:
file_id = f"file_{int(now() * 1000)}"
data_root = Path(os.getenv("YG_FT_DATA_ROOT", "/data/yg-ft"))
data_root.mkdir(parents=True, exist_ok=True)
filename = Path(file.filename if file else file_id).name
if target_relative_path:
target = (data_root / target_relative_path.lstrip("/\\")).resolve()
if not _path_inside(data_root, target):
raise HTTPException(status_code=400, detail="target path must stay inside YG_FT_DATA_ROOT")
else:
target = data_root / "uploads" / f"{file_id}_{filename}"
if file:
target.parent.mkdir(parents=True, exist_ok=True)
with target.open("wb") as output:
while chunk := await file.read(1024 * 1024):
output.write(chunk)
else:
target.parent.mkdir(parents=True, exist_ok=True)
target.write_text("", encoding="utf-8")
return {
"id": file_id,
"resource_type": resource_type,
"resource_id": resource_id,
"status": "available",
"local_path": str(target),
"byte_size": target.stat().st_size,
"checksum_sha256": hashlib.sha256(target.read_bytes()).hexdigest() if target.is_file() else "",
}
@app.post(f"{route_prefix}/compute/files/import-local")
async def import_local_file(payload: dict[str, Any]) -> dict[str, Any]:
source = Path(str(payload.get("source_path") or ""))
if not source.exists():
raise HTTPException(status_code=404, detail="source path not found")
data_root = Path(os.getenv("YG_FT_DATA_ROOT", "/data/yg-ft"))
data_root.mkdir(parents=True, exist_ok=True)
relative = str(payload.get("target_relative_path") or f"imports/{source.name}").lstrip("/\\")
target = (data_root / relative).resolve()
if not _path_inside(data_root, target):
raise HTTPException(status_code=400, detail="target path must stay inside YG_FT_DATA_ROOT")
target.parent.mkdir(parents=True, exist_ok=True)
if source.is_dir():
if target.exists():
shutil.rmtree(target)
shutil.copytree(source, target)
byte_size = sum(path.stat().st_size for path in target.rglob("*") if path.is_file())
checksum = ""
else:
shutil.copy2(source, target)
byte_size = target.stat().st_size
checksum = hashlib.sha256(target.read_bytes()).hexdigest()
return {
"id": str(payload.get("id") or f"file_{int(now() * 1000)}"),
"resource_type": payload.get("resource_type"),
"resource_id": payload.get("resource_id"),
"status": "available",
"local_path": str(target),
"byte_size": byte_size,
"checksum_sha256": checksum,
}
@app.get(f"{route_prefix}/compute/files/read")
async def read_file(path: str = Query(...)) -> JSONResponse:
"""Read a text file from within YG_FT_DATA_ROOT. Used by the backend
to fetch eval results and other job outputs."""
data_root = Path(os.getenv("YG_FT_DATA_ROOT", "/data/yg-ft"))
target = (data_root / path.lstrip("/\\")).resolve()
if not _path_inside(data_root, target):
raise HTTPException(status_code=400, detail="path must stay inside YG_FT_DATA_ROOT")
if not target.is_file():
raise HTTPException(status_code=404, detail="file not found")
try:
content = target.read_text(encoding="utf-8")
return JSONResponse(json.loads(content) if content.strip().startswith("{") else {"content": content})
except Exception as exc:
raise HTTPException(status_code=500, detail=str(exc))
async def upload_file(payload: dict[str, Any]) -> dict[str, Any]:
file_id = str(payload.get("id") or f"file_{int(now() * 1000)}")
return {"id": file_id, "status": "available", "local_path": f"/data/yg-ft/uploads/{file_id}"}
@app.get(f"{route_prefix}/compute/files/{{file_id}}/download")
async def download_file(file_id: str) -> FileResponse:
upload_root = Path(os.getenv("YG_FT_DATA_ROOT", "/data/yg-ft")) / "uploads"
# file_id 仅允许普通标识符，拒绝 ../、/、\ 等路径穿越字符。
if not file_id or not all(character.isalnum() or character in {"_", "-"} for character in file_id):
raise HTTPException(status_code=400, detail="invalid file id")
matches = list(upload_root.glob(f"{file_id}_*"))
if not matches:
raise HTTPException(status_code=404, detail="file not found")
# 解析符号链接后仍必须位于 upload 根目录内，防止符号链接指向目录外文件。
resolved = matches[0].resolve()
if not _path_inside(upload_root, resolved):
raise HTTPException(status_code=404, detail="file not found")
return FileResponse(resolved)
async def download_file(file_id: str) -> dict[str, Any]:
return {"id": file_id, "status": "ready", "download_url": f"{route_prefix}/compute/files/{file_id}/download"}
return app

29
compute/api/security.py
View File

@@ -1,29 +0,0 @@
"""计算节点 API 安全配置：Swagger / ReDoc / OpenAPI 文档路由开关。"""
from __future__ import annotations
import os
from typing import Any
def docs_enabled() -> bool:
"""判断 FastAPI 文档路由（/docs、/redoc、/openapi.json）是否开放。
显式配置 ENABLE_DOCS 时以之为准；否则仅在关闭 token 鉴权
（COMPUTE_AUTH_ENABLED=false，本地开发）时开放，生产环境默认关闭，
避免未授权访问泄露 API 结构。
"""
raw = os.getenv("ENABLE_DOCS", "").strip().lower()
if raw in {"true", "false"}:
return raw == "true"
auth_enabled = os.getenv("COMPUTE_AUTH_ENABLED", "true").lower() == "true"
return not auth_enabled
def docs_kwargs() -> dict[str, Any]:
"""返回传入 FastAPI 的文档路由参数。
关闭时 FastAPI 不注册 /docs、/redoc、/openapi.json，访问一律返回 404。
"""
if docs_enabled():
return {}
return {"docs_url": None, "redoc_url": None, "openapi_url": None}

287
compute/engines/llama_factory/adapter.py
View File

@@ -1,6 +1,5 @@
from __future__ import annotations
import json
import re
from dataclasses import dataclass
from pathlib import Path
@@ -14,290 +13,40 @@ class LlamaFactoryCommand:
env: dict[str, str]
def _load_dataset_preview(path: Path) -> list[dict[str, Any]]:
"""Load a preview of JSON/JSONL records from a dataset file.
Content-sniffs instead of trusting the extension so that BOM-prefixed files,
JSONL files containing a single JSON array, and mislabeled extensions all work.
"""
if not path.exists():
return []
text = path.read_text(encoding="utf-8-sig", errors="replace").strip()
if not text:
return []
try:
value = json.loads(text)
except json.JSONDecodeError:
value = None
if isinstance(value, list):
return [item for item in value[:20] if isinstance(item, dict)]
if isinstance(value, dict):
return [value]
items: list[dict[str, Any]] = []
for line in text.splitlines()[:20]:
line = line.strip()
if not line:
continue
try:
parsed = json.loads(line)
except json.JSONDecodeError:
continue
if isinstance(parsed, list):
items.extend(item for item in parsed[:20] if isinstance(item, dict))
elif isinstance(parsed, dict):
items.append(parsed)
if len(items) >= 20:
break
return items[:20]
def _required_columns_for(formatting: str, columns: dict[str, Any]) -> list[str]:
"""Required data columns per dataset format.
Mirrors LLaMA-Factory's leniency: optional columns (e.g. ``input`` / ``query``
in Alpaca) are never required, only fields the format structurally needs.
"""
fmt = str(formatting or "").lower()
if fmt == "sharegpt":
return [str(columns.get("messages") or "messages")]
if fmt in {"dpo", "rm", "kto", "ppo"}:
return [str(columns[key]) for key in ("chosen", "rejected") if columns.get(key)]
if fmt in {"cpt", "pt", "pretrain"}:
return [str(columns.get("prompt") or columns.get("text") or "text")]
# alpaca family: prompt (instruction) + response (output) required,
# query (input) / history are optional and common to omit in jsonl datasets.
return [str(columns[key]) for key in ("prompt", "response") if columns.get(key)]
def _validate_dataset_columns(config: dict[str, Any]) -> list[str]:
dataset_dir = config.get("dataset_dir")
dataset_info = config.get("dataset_info")
if not dataset_dir or not isinstance(dataset_info, dict):
return []
root = Path(str(dataset_dir))
errors: list[str] = []
for dataset_key, item in dataset_info.items():
if not isinstance(item, dict):
continue
file_name = item.get("file_name")
file_names = file_name if isinstance(file_name, list) else [file_name]
columns = item.get("columns") if isinstance(item.get("columns"), dict) else {}
required_columns = _required_columns_for(str(item.get("formatting") or ""), columns)
for name in file_names:
if not name:
continue
path = root / str(name).lstrip("/\\")
if not path.exists():
continue
try:
preview_rows = _load_dataset_preview(path)
except Exception as exc: # noqa: BLE001 - expose malformed data as validation error
errors.append(f"dataset file parse failed: {path}: {exc}")
continue
if not preview_rows:
errors.append(f"dataset file has no valid object records: {path}")
continue
available = set().union(*(row.keys() for row in preview_rows))
missing = [column for column in required_columns if column not in available]
if missing:
errors.append(
f"dataset columns missing in {path.name} for {dataset_key}: {', '.join(sorted(set(missing)))}"
)
return errors
def validate_config(config: dict[str, Any]) -> list[str]:
errors: list[str] = []
if not config.get("base_model") and not config.get("model_name_or_path"):
errors.append("base_model or model_name_or_path is required")
if not config.get("dataset") and not config.get("dataset_dir"):
errors.append("dataset or dataset_dir is required")
try:
learning_rate = float(config.get("learning_rate", 0.0002))
except (TypeError, ValueError):
learning_rate = 0
learning_rate = float(config.get("learning_rate", 0.0002))
if learning_rate <= 0:
errors.append("learning_rate must be greater than zero")
try:
epochs = int(config.get("n_epochs", config.get("num_train_epochs", 1)))
except (TypeError, ValueError):
epochs = 0
epochs = int(config.get("n_epochs", config.get("num_train_epochs", 1)))
if epochs <= 0:
errors.append("n_epochs must be greater than zero")
dataset_dir = config.get("dataset_dir")
dataset_info = config.get("dataset_info")
if config.get("require_dataset_files") and dataset_dir and isinstance(dataset_info, dict):
root = Path(str(dataset_dir))
for dataset_key, item in dataset_info.items():
if not isinstance(item, dict):
errors.append(f"dataset_info entry must be object: {dataset_key}")
continue
file_name = item.get("file_name")
file_names = file_name if isinstance(file_name, list) else [file_name]
for name in file_names:
if not name:
errors.append(f"dataset_info file_name is required: {dataset_key}")
continue
path = root / str(name).lstrip("/\\")
if not path.exists():
errors.append(f"dataset file not found: {path}")
errors.extend(_validate_dataset_columns(config))
return errors
def _optional_arg(config: dict[str, Any], command: list[str], option: str, *keys: str) -> None:
for key in keys:
value = config.get(key)
if value is not None and value != "":
command.extend([option, str(value)])
return
def _optional_bool_arg(config: dict[str, Any], command: list[str], option: str, *keys: str) -> None:
for key in keys:
value = config.get(key)
if value is True or str(value).lower() == "true":
command.extend([option, "true"])
return
def _normalize_stage(config: dict[str, Any]) -> str:
raw = str(config.get("stage") or config.get("train_type") or "sft").strip().lower()
return {
"sft": "sft",
"dpo": "dpo",
"cpt": "pt",
"pt": "pt",
"pretrain": "pt",
"rm": "rm",
"ppo": "ppo",
"kto": "kto",
}.get(raw, raw or "sft")
def prepare_runtime_files(config: dict[str, Any]) -> list[dict[str, str]]:
dataset_dir = config.get("dataset_dir")
dataset_info = config.get("dataset_info")
if not dataset_dir or not isinstance(dataset_info, dict):
return []
root = Path(str(dataset_dir))
root.mkdir(parents=True, exist_ok=True)
path = root / "dataset_info.json"
existing: dict[str, Any] = {}
if path.exists():
try:
loaded = json.loads(path.read_text(encoding="utf-8"))
existing = loaded if isinstance(loaded, dict) else {}
except json.JSONDecodeError:
existing = {}
existing.update(dataset_info)
path.write_text(json.dumps(existing, ensure_ascii=False, indent=2), encoding="utf-8")
return [{"name": "dataset_info", "path": str(path)}]
def build_command(config: dict[str, Any], llama_factory_home: str = "/app/LLaMA-Factory") -> LlamaFactoryCommand:
engine = str(config.get("engine") or config.get("training_engine") or "llama_factory")
if engine in {"merge", "export", "llama_factory_export"}:
model_path = config.get("base_model") or config.get("model_name_or_path") or config.get("base_model_path")
adapter_path = config.get("adapter_name_or_path") or config.get("adapter_path") or config.get("lora_path")
output_dir = config.get("output_dir") or config.get("export_dir")
errors: list[str] = []
if not model_path:
errors.append("base_model or model_name_or_path is required")
if not adapter_path and engine == "merge":
errors.append("adapter_name_or_path or adapter_path is required")
if not output_dir:
errors.append("output_dir or export_dir is required")
if errors:
raise ValueError("; ".join(errors))
command = [
"llamafactory-cli",
"export",
"--model_name_or_path",
str(model_path),
"--template",
str(config.get("template", "qwen")),
"--finetuning_type",
str(config.get("train_method", config.get("finetuning_type", "lora"))),
"--export_dir",
str(output_dir),
"--export_size",
str(config.get("export_size", 2)),
"--export_device",
str(config.get("export_device", "cpu")),
"--export_legacy_format",
str(config.get("export_legacy_format", False)).lower(),
]
if adapter_path:
command.extend(["--adapter_name_or_path", str(adapter_path)])
quantization_bit = int(config.get("export_quantization_bit", config.get("quantization_bit", 0)) or 0)
if quantization_bit in {4, 8}:
command.extend(["--quantization_bit", str(quantization_bit)])
return LlamaFactoryCommand(command=command, work_dir=str(Path(llama_factory_home)), env={})
if engine == "eval":
output_dir = config.get("output_dir") or f"/data/yg-ft/outputs/{config.get('name', 'eval-job')}"
eval_config_path = str(Path(output_dir) / "eval_config.json")
eval_config = {
"model_name_or_path": config.get("model_name_or_path", ""),
"adapter_name_or_path": config.get("adapter_name_or_path", ""),
"template": config.get("template", "qwen"),
"dataset_path": config.get("dataset_path", ""),
"output_dir": output_dir,
"basic_metrics": config.get("basic_metrics", {}),
"dimension": config.get("dimension", {}),
"temperature": config.get("temperature", 0.1),
"top_p": config.get("top_p", 0.95),
"max_new_tokens": config.get("max_new_tokens", 512),
"infer_backend": config.get("infer_backend", "huggingface"),
"infer_dtype": config.get("infer_dtype", "auto"),
}
Path(output_dir).mkdir(parents=True, exist_ok=True)
Path(eval_config_path).write_text(json.dumps(eval_config, ensure_ascii=False, indent=2), encoding="utf-8")
return LlamaFactoryCommand(
command=["python", "-u", "-m", "compute.engines.llama_factory.eval_runner", "--config", eval_config_path],
work_dir="/app",
env={},
)
errors = validate_config(config)
if errors:
raise ValueError("; ".join(errors))
if engine == "smoke":
output_dir = config.get("output_dir") or f"/data/yg-ft/outputs/{config.get('name', 'training-smoke')}"
script = (
"import json, os, time; "
f"out={str(output_dir)!r}; "
"os.makedirs(out, exist_ok=True); "
"print('[INFO] smoke training started', flush=True); "
"\nfor step in range(1, 7):\n"
" loss=round(1.8/(step+1), 4)\n"
" lr=round(0.0002*(1-step/10), 8)\n"
" print({'loss': loss, 'grad_norm': round(0.4 + step*0.03, 4), 'learning_rate': lr, 'epoch': round(step/6, 4)}, flush=True)\n"
" time.sleep(0.4)\n"
"\nopen(os.path.join(out, 'adapter_config.json'), 'w', encoding='utf-8').write(json.dumps({'engine':'smoke','status':'completed'})); "
"print('***** train metrics *****', flush=True); "
"print('train_loss = 0.12', flush=True); "
"print('***** train metrics end *****', flush=True)"
)
return LlamaFactoryCommand(command=["python", "-u", "-c", script], work_dir="/app", env={})
model_path = config.get("base_model") or config.get("model_name_or_path")
dataset = config.get("dataset") or config.get("dataset_name")
dataset_dir = config.get("dataset_dir")
dataset = config.get("dataset") or config.get("dataset_dir")
output_dir = config.get("output_dir") or f"/data/yg-ft/outputs/{config.get('name', 'training-job')}"
command = [
"llamafactory-cli",
"train",
"--stage",
_normalize_stage(config),
str(config.get("stage", "sft")).lower(),
"--do_train",
"true",
"--model_name_or_path",
str(model_path),
"--dataset",
str(dataset or "default"),
str(dataset),
"--template",
str(config.get("template", "qwen")),
"--finetuning_type",
@@ -312,32 +61,7 @@ def build_command(config: dict[str, Any], llama_factory_home: str = "/app/LLaMA-
str(config.get("n_epochs", 3)),
"--save_steps",
str(config.get("save_steps", 50)),
"--logging_steps",
str(config.get("logging_steps", 10)),
"--overwrite_output_dir",
"true",
"--plot_loss",
"true",
]
if dataset_dir:
command.extend(["--dataset_dir", str(dataset_dir)])
eval_dataset = config.get("eval_dataset")
if eval_dataset:
command.extend(["--eval_dataset", str(eval_dataset), "--do_eval", "true"])
_optional_arg(config, command, "--cutoff_len", "max_length", "cutoff_len")
_optional_arg(config, command, "--lr_scheduler_type", "lr_scheduler_type")
_optional_arg(config, command, "--warmup_ratio", "warmup_ratio")
_optional_arg(config, command, "--weight_decay", "weight_decay")
_optional_arg(config, command, "--lora_rank", "lora_rank", "rank")
_optional_arg(config, command, "--lora_alpha", "lora_alpha")
_optional_arg(config, command, "--lora_dropout", "lora_dropout")
_optional_arg(config, command, "--gradient_accumulation_steps", "gradient_accumulation_steps")
if not eval_dataset:
_optional_arg(config, command, "--val_size", "val_size")
_optional_arg(config, command, "--max_samples", "max_samples")
_optional_arg(config, command, "--preprocessing_num_workers", "preprocessing_num_workers")
_optional_bool_arg(config, command, "--fp16", "fp16")
_optional_bool_arg(config, command, "--bf16", "bf16")
quantization_bit = int(config.get("quantization_bit", 0) or 0)
if quantization_bit in {4, 8}:
command.extend(["--quantization_bit", str(quantization_bit)])
@@ -353,3 +77,4 @@ def parse_log_line(line: str) -> dict[str, float] | None:
if match:
result[key] = float(match.group(1))
return result or None

491
compute/engines/llama_factory/eval_runner.py
View File

@@ -1,491 +0,0 @@
"""
Evaluation runner — executes model evaluation as a subprocess job.
Usage:
python -m compute.engines.llama_factory.eval_runner --config <config_json_path>
The config JSON is written by the compute API before spawning this subprocess.
Results are written to ``output_dir/eval_results.json`` and progress is printed
to stdout (captured as job logs).
"""
from __future__ import annotations
import json
import math
import re
import sys
import time
from difflib import SequenceMatcher
from pathlib import Path
from typing import Any
def _load_dataset(path: str) -> list[dict[str, Any]]:
"""Load a JSON or JSONL dataset file (jsonl-compatible).
Content-sniffs instead of trusting the extension so jsonl files with a BOM,
a single JSON array on one line, or mislabeled extensions all load correctly.
Supports common field names used across the platform:
* ``instruction`` + ``input`` + ``output`` (Alpaca-style)
* ``question`` + ``answer``
* ``messages`` (ShareGPT-style – the last assistant message is treated as reference)
"""
file_path = Path(path)
text = file_path.read_text(encoding="utf-8-sig", errors="replace").strip()
if not text:
return []
try:
value = json.loads(text)
except json.JSONDecodeError:
value = None
if isinstance(value, list):
return [item for item in value if isinstance(item, dict)]
if isinstance(value, dict):
return [value]
samples: list[dict[str, Any]] = []
for line in text.splitlines():
line = line.strip()
if not line:
continue
try:
obj = json.loads(line)
except json.JSONDecodeError:
continue
if isinstance(obj, dict):
samples.append(obj)
return samples
def _sample_question(sample: dict[str, Any]) -> str:
"""Extract the user-facing question / instruction from a sample."""
if sample.get("instruction"):
text = sample["instruction"]
if sample.get("input"):
text += "\n" + sample["input"]
return text
if sample.get("question"):
return sample["question"]
# ShareGPT-style: use the last user message as question
messages = sample.get("messages") or []
user_msgs = [m["content"] for m in messages if m.get("role") == "user"]
return user_msgs[-1] if user_msgs else ""
def _sample_reference(sample: dict[str, Any]) -> str:
"""Extract the reference answer from a sample."""
if sample.get("output"):
return sample["output"]
if sample.get("answer"):
return sample["answer"]
messages = sample.get("messages") or []
assistant_msgs = [m["content"] for m in messages if m.get("role") == "assistant"]
return assistant_msgs[-1] if assistant_msgs else ""
# ---------------------------------------------------------------------------
# Basic metrics
# ---------------------------------------------------------------------------
def _compute_bleu(references: list[str], predictions: list[str], ngram: int = 4) -> dict[str, Any]:
"""Compute BLEU score via sacrebleu (corpus-level)."""
try:
from sacrebleu.metrics import BLEU
except ImportError:
return {"enabled": False, "error": "sacrebleu not installed", "score": 0}
bleu = BLEU(max_ngram_order=ngram)
# sacrebleu expects list-of-strings; we have one reference per prediction
score = bleu.corpus_score(predictions, [references])
return {
"enabled": True,
"score": round(score.score, 2),
"bleu": round(score.score, 2),
}
def _compute_rouge(references: list[str], predictions: list[str], methods: list[str] | None = None) -> dict[str, Any]:
"""Compute ROUGE scores via rouge-score."""
try:
from rouge_score import rouge_scorer
except ImportError:
return {"enabled": False, "error": "rouge-score not installed", "score": 0}
methods = methods or ["rouge1", "rouge2", "rougeL"]
# Normalize: map "rouge_1"/"rouge1" → "rouge1", "rouge_l"/"rougeL" → "rougeL"
_rouge_aliases = {"rouge_1": "rouge1", "rouge_2": "rouge2", "rouge_l": "rougeL"}
methods = [_rouge_aliases.get(m, m.replace("_", "")) for m in methods]
scorer = rouge_scorer.RougeScorer(methods, use_stemmer=True)
totals: dict[str, float] = {}
n = max(len(predictions), 1)
for ref, pred in zip(references, predictions):
result = scorer.score(ref, pred)
for key in methods:
totals[key] = totals.get(key, 0) + result[key].fmeasure
avg = {k: round(v / n, 4) for k, v in totals.items()}
return {"enabled": True, "score": round(avg.get("rougeL", avg.get("rouge1", 0)) * 100, 2), **avg}
def _compute_cosine(references: list[str], predictions: list[str]) -> dict[str, Any]:
"""Compute average cosine similarity via sklearn."""
try:
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
except ImportError:
return {"enabled": False, "error": "scikit-learn not installed", "score": 0}
try:
vectorizer = TfidfVectorizer()
tfidf = vectorizer.fit_transform(references + predictions)
n = len(references)
ref_vec = tfidf[:n]
pred_vec = tfidf[n:]
sims = cosine_similarity(ref_vec, pred_vec).diagonal()
return {"enabled": True, "score": round(float(sims.mean()) * 100, 2)}
except ValueError:
return {"enabled": True, "score": 0, "error": "insufficient text for vectorization"}
def _normalize_text(value: str) -> str:
return re.sub(r"\s+", " ", str(value or "").strip().lower())
def _compute_exact_match(references: list[str], predictions: list[str]) -> dict[str, Any]:
total = len(predictions)
if not total:
return {"enabled": True, "score": 0, "matched": 0, "total": 0}
matched = sum(
1
for ref, pred in zip(references, predictions)
if _normalize_text(ref) == _normalize_text(pred)
)
return {"enabled": True, "score": round(matched / total * 100, 2), "matched": matched, "total": total}
def _compute_text_similarity(references: list[str], predictions: list[str]) -> dict[str, Any]:
if not predictions:
return {"enabled": True, "score": 0}
scores = [
SequenceMatcher(None, _normalize_text(ref), _normalize_text(pred)).ratio()
for ref, pred in zip(references, predictions)
]
return {"enabled": True, "score": round(sum(scores) / max(len(scores), 1) * 100, 2)}
# ---------------------------------------------------------------------------
# LLM Judge
# ---------------------------------------------------------------------------
def _judge_sample(
question: str,
reference: str,
prediction: str,
config: dict[str, Any],
) -> dict[str, Any]:
"""Call an OpenAI-compatible LLM to judge a single sample.
Returns a dict with keys:
score, max_score, passed, judgement, evaluation_reason, error_type
"""
api_url = (config.get("api_url") or "").strip().rstrip("/")
api_key = (config.get("api_key") or "").strip()
eval_model = (config.get("eval_model") or "").strip()
# 优先使用模型记录里配置的真实 API 模型名（如 deepseek-chat），
# 否则回退到平台内部模型名
api_model = (config.get("api_model") or "").strip() or eval_model
eval_prompt = (config.get("eval_prompt") or "").strip()
score_min = float(config.get("score_min", 0))
score_max = float(config.get("score_max", 5))
pass_threshold = float(config.get("pass_threshold", 3))
if not api_url or not eval_model:
return {"score": 0, "max_score": score_max, "passed": False, "judgement": "未配置",
"evaluation_reason": "未配置评测模型", "error_type": "其他"}
system_msg = (
eval_prompt
or "你是一个专业的评测专家。请根据参考答-案对被测模型的输出进行评分。"
)
user_msg = (
f"## 问题\n{question}\n\n"
f"## 参考答案\n{reference}\n\n"
f"## 模型输出\n{prediction}\n\n"
f"请给出 {score_min}-{score_max} 分的评分，并说明理由。"
)
try:
import urllib.request
import urllib.error
body = json.dumps({
"model": api_model,
"messages": [
{"role": "system", "content": system_msg},
{"role": "user", "content": user_msg},
],
"temperature": 0.3,
"max_tokens": 512,
}).encode("utf-8")
req = urllib.request.Request(
f"{api_url}/v1/chat/completions",
data=body,
headers={
"Content-Type": "application/json",
"Authorization": f"Bearer {api_key}",
},
)
resp = urllib.request.urlopen(req, timeout=120)
data = json.loads(resp.read().decode("utf-8"))
reply = data["choices"][0]["message"]["content"]
except Exception as exc:
return {"score": 0, "max_score": score_max, "passed": False,
"judgement": "错误", "evaluation_reason": f"评测模型调用失败: {exc}",
"error_type": "其他"}
# Parse score from reply — look for patterns like "4分" or "Score: 4"
score = 0
import re
score_patterns = [
r'(?:得分|分数|评分|score)[^\d]*(\d+(?:\.\d+)?)',
r'(\d+(?:\.\d+)?)\s*分',
r'(\d+(?:\.\d+)?)\s*/\s*\d+',
]
for pat in score_patterns:
m = re.search(pat, reply, re.IGNORECASE)
if m:
try:
score = float(m.group(1))
except ValueError:
continue
break
score = max(score_min, min(score_max, score))
passed = score >= pass_threshold
# Determine judgement label
if score >= pass_threshold + 1:
judgement = "正确"
elif score >= pass_threshold:
judgement = "部分正确"
else:
judgement = "错误"
# Guess error type from reply
reply_lower = reply.lower()
if any(w in reply_lower for w in ["幻觉", "hallucination", "编造"]):
error_type = "幻觉"
elif any(w in reply_lower for w in ["不完整", "incomplete", "遗漏"]):
error_type = "不完整"
elif any(w in reply_lower for w in ["格式", "format"]):
error_type = "格式偏差"
elif any(w in reply_lower for w in ["混淆", "confusion", "错误"]):
error_type = "混淆"
else:
error_type = "其他"
return {
"score": score,
"max_score": score_max,
"passed": passed,
"judgement": judgement,
"evaluation_reason": reply[:2000],
"error_type": error_type,
}
# ---------------------------------------------------------------------------
# Main entry point
# ---------------------------------------------------------------------------
def run_eval(config: dict[str, Any]) -> dict[str, Any]:
"""Execute a full evaluation run. Returns the result dict (also written to file)."""
model_path = config["model_name_or_path"]
adapter_path = config.get("adapter_name_or_path", "")
template = config.get("template", "qwen")
dataset_path = config["dataset_path"]
output_dir = Path(config["output_dir"])
output_dir.mkdir(parents=True, exist_ok=True)
basic_cfg = config.get("basic_metrics", {})
dimension_cfg = config.get("dimension", {}) or {}
output_precision = int(basic_cfg.get("output_precision", 2))
# ---- 1. Load dataset ----
print(f"[eval] loading dataset: {dataset_path}")
raw_samples = _load_dataset(dataset_path)
print(f"[eval] loaded {len(raw_samples)} samples")
# ---- 2. Load model ----
print(f"[eval] loading model: {model_path}")
from compute.engines.llama_factory.inference import InferenceSession
session = InferenceSession()
session.load(
model_name_or_path=model_path,
adapter_name_or_path=adapter_path,
template=template,
infer_backend=config.get("infer_backend", "huggingface"),
infer_dtype=config.get("infer_dtype", "auto"),
)
# load() 为异步加载（立即返回 loading），必须等待后台线程完成后再进行推理
load_result = session.wait_until_loaded(timeout=float(config.get("load_timeout", 1800)))
if not load_result.get("loaded"):
raise RuntimeError(f"model load failed: {load_result.get('error', 'unknown')}")
print(f"[eval] model loaded OK")
# ---- 3. Run inference on each sample ----
samples: list[dict[str, Any]] = []
predictions: list[str] = []
references: list[str] = []
questions: list[str] = []
total = len(raw_samples)
judge_enabled = bool(dimension_cfg.get("eval_model") and dimension_cfg.get("api_url"))
print(f"[eval] starting inference on {total} samples, judge={'enabled' if judge_enabled else 'disabled'}")
for idx, raw in enumerate(raw_samples, start=1):
question = _sample_question(raw)
reference = _sample_reference(raw)
if not question:
print(f"[eval] sample {idx}/{total}: skipped (no question)")
continue
# Inference
chat_msgs = [{"role": "user", "content": question}]
result = session.chat(
chat_msgs,
temperature=float(config.get("temperature", 0.1)),
top_p=float(config.get("top_p", 0.95)),
max_new_tokens=int(config.get("max_new_tokens", 512)),
do_sample=False,
)
prediction = result.get("response", "") if not result.get("error") else f"[ERROR] {result['error']}"
predictions.append(prediction)
references.append(reference)
questions.append(question)
# LLM Judge
judge_result: dict[str, Any] = {}
if judge_enabled:
judge_result = _judge_sample(question, reference, prediction, dimension_cfg)
samples.append({
"index": idx,
"input": question,
"reference_answer": reference,
"model_output": prediction,
"score": judge_result.get("score"),
"max_score": judge_result.get("max_score", dimension_cfg.get("score_max", 5)),
"passed": judge_result.get("passed"),
"judgement": judge_result.get("judgement"),
"evaluation_reason": judge_result.get("evaluation_reason", ""),
"error_type": judge_result.get("error_type"),
"dimension_scores": [
{"name": "judge_score", "score": judge_result.get("score", 0),
"max_score": judge_result.get("max_score", dimension_cfg.get("score_max", 5))},
] if judge_result else [],
"status": "completed",
})
progress_pct = int(idx / max(total, 1) * 100)
print(f"[eval] sample {idx}/{total} ({progress_pct}%) done")
# ---- 4. Compute basic metrics ----
print(f"[eval] computing basic metrics on {len(predictions)} predictions")
metrics_result: dict[str, Any] = {}
bleu_cfg = basic_cfg.get("bleu", {})
if bleu_cfg.get("enabled"):
metrics_result["bleu"] = _compute_bleu(references, predictions, int(bleu_cfg.get("ngram", 4)))
rouge_cfg = basic_cfg.get("rouge", {})
if rouge_cfg.get("enabled"):
metrics_result["rouge"] = _compute_rouge(references, predictions, rouge_cfg.get("methods"))
cosine_cfg = basic_cfg.get("cosine", {})
if cosine_cfg.get("enabled"):
metrics_result["cosine"] = _compute_cosine(references, predictions)
metrics_result["exact_match"] = _compute_exact_match(references, predictions)
metrics_result["text_similarity"] = _compute_text_similarity(references, predictions)
# ---- 5. Summarise ----
completed = len(samples)
if judge_enabled:
scored = [s for s in samples if s.get("score") is not None]
passed_count = len([s for s in scored if s.get("passed")])
avg_score = round(sum(s["score"] for s in scored) / max(len(scored), 1), output_precision)
max_score = dimension_cfg.get("score_max", 5)
overall_score = round(avg_score / max_score * 100, output_precision)
overall_score_max = 100
dimension_summary = [{
"name": "综合评分",
"score": overall_score,
"max_score": 100,
"pass_rate": round(passed_count / max(completed, 1) * 100, 1),
}]
overall_evaluation = f"评测完成：{completed} 样本，{passed_count} 通过，平均 {avg_score}/{max_score} 分"
else:
passed_count = 0
enabled_scores = [
float(item.get("score") or 0)
for item in metrics_result.values()
if isinstance(item, dict) and item.get("enabled", True) and item.get("score") is not None
]
overall_score = round(sum(enabled_scores) / len(enabled_scores), output_precision) if enabled_scores else 0
overall_score_max = 100
dimension_summary = [
{
"name": name,
"score": float(item.get("score") or 0),
"max_score": 100,
"pass_rate": float(item.get("score") or 0),
}
for name, item in metrics_result.items()
if isinstance(item, dict) and item.get("enabled", True) and item.get("score") is not None
]
overall_evaluation = f"评测完成：{completed} 样本（未配置 LLM 评委）"
result = {
"overall_score": overall_score,
"overall_score_max": overall_score_max,
"overall_evaluation": overall_evaluation,
"improvement_suggestions": [],
"dimension_summary": dimension_summary,
"samples": samples,
"sample_count": total,
"completed_count": completed,
"passed_count": passed_count,
"basic_metrics": metrics_result,
}
# ---- 6. Write results ----
result_path = output_dir / "eval_results.json"
result_path.write_text(json.dumps(result, ensure_ascii=False, indent=2), encoding="utf-8")
print(f"[eval] results written to {result_path}")
return result
def main() -> None:
import argparse
parser = argparse.ArgumentParser(description="YG-FT Evaluation Runner")
parser.add_argument("--config", required=True, help="Path to eval config JSON file")
args = parser.parse_args()
config_path = Path(args.config)
if not config_path.exists():
print(f"FATAL: config file not found: {args.config}", file=sys.stderr)
sys.exit(1)
config = json.loads(config_path.read_text(encoding="utf-8"))
start = time.time()
try:
run_eval(config)
elapsed = time.time() - start
print(f"[eval] DONE in {elapsed:.1f}s")
except Exception as exc:
print(f"[eval] FAILED: {exc}", file=sys.stderr)
import traceback
traceback.print_exc()
sys.exit(1)
if __name__ == "__main__":
main()

272
compute/engines/llama_factory/inference.py
View File

@@ -1,272 +0,0 @@
from __future__ import annotations
import threading
import time
import uuid
from typing import Any, Iterator
class InferenceSession:
"""Manages a loaded model for inference with LLaMA-Factory ChatModel.
Model loading is asynchronous: ``load()`` spawns a background daemon thread
and returns immediately with ``status == "loading"``. ``info()`` (served by
``/inference/status``) is always responsive, so the platform backend can
poll loading progress without being blocked by a minutes-long model load —
which previously froze the whole compute node event loop.
State machine: idle -> loading -> ready | error, ready -> idle (unload),
loading -> idle (cancelled). Long operations (ChatModel build, teardown,
generation) never run while holding ``_state_lock``; they either run in the
worker thread or under ``_chat_lock`` only.
"""
def __init__(self) -> None:
self._state_lock = threading.Lock() # brief state transitions only
self._chat_lock = threading.Lock() # serialize chat/teardown
self._status: str = "idle"
self._error: str = ""
self._request_id: str = ""
self._load_args: dict[str, Any] = {}
self._teardown_old = False # load-while-ready: unload old before loading new
self._cancel_requested = False # unload-while-loading: tear down after load finishes
self._load_thread: threading.Thread | None = None
self._model: Any = None
self._tokenizer: Any = None
self._generating_args: dict[str, Any] = {}
self._model_name: str = ""
self._adapter_path: str = ""
self._loaded_at: float = 0.0
@property
def status(self) -> str:
with self._state_lock:
return self._status
def info(self) -> dict[str, Any]:
with self._state_lock:
return {
"loaded": self._status == "ready",
"status": self._status,
"model_name": self._model_name,
"adapter_path": self._adapter_path,
"loaded_at": self._loaded_at,
"request_id": self._request_id,
"error": self._error,
}
def wait_until_loaded(self, timeout: float | None = None) -> dict[str, Any]:
"""Wait for an in-flight async load to finish and return its outcome.
供同步消费方（如 eval_runner 子进程）使用：``load()`` 立即返回 loading 后，
调用本方法等待后台加载线程完成，拿到最终的 loaded/error 结果。
若在 timeout 秒内仍未加载完成，返回 ``status == "loading"`` 并附上超时提示。
"""
with self._state_lock:
thread = self._load_thread
if thread is not None and thread.is_alive():
thread.join(timeout=timeout)
with self._state_lock:
loaded = self._status == "ready"
status = self._status
error = self._error
if not loaded and status == "loading":
error = error or f"model load timed out after {timeout or 'N/A'}s"
return {
"loaded": loaded,
"status": status,
"model_name": self._model_name,
"adapter_path": self._adapter_path,
"error": error,
}
def load(
self,
model_name_or_path,
adapter_name_or_path="",
template="qwen",
infer_backend="huggingface",
infer_dtype="auto",
**kwargs,
) -> dict[str, Any]:
with self._state_lock:
if self._status == "loading":
# A model is already loading — dedupe, reuse the same request id.
return {"loaded": False, "status": "loading", "request_id": self._request_id}
self._teardown_old = self._status == "ready"
self._status = "loading"
self._error = ""
self._request_id = uuid.uuid4().hex[:12]
self._cancel_requested = False
self._load_args = {
"model_name_or_path": model_name_or_path,
"template": template,
"infer_backend": infer_backend,
"infer_dtype": infer_dtype,
}
if adapter_name_or_path:
self._load_args["adapter_name_or_path"] = adapter_name_or_path
self._load_args.update(kwargs)
self._model_name = model_name_or_path
self._adapter_path = adapter_name_or_path
self._load_thread = threading.Thread(target=self._load_worker, daemon=True)
self._load_thread.start()
return {"loaded": False, "status": "loading", "request_id": self._request_id}
def _load_worker(self) -> None:
"""Build the ChatModel off the state lock so info() never blocks."""
model = None
tokenizer = None
generating_args: dict[str, Any] = {}
error = ""
try:
if self._teardown_old:
self._release_model()
from llamafactory.chat import ChatModel
from llamafactory.hparams import get_infer_args
args = dict(self._load_args)
infer_result = get_infer_args(args)
model = ChatModel(args)
tokenizer = getattr(model, "tokenizer", None) or model.engine.tokenizer
generating_args = infer_result[-1]
if hasattr(generating_args, "__dataclass_fields__"):
generating_args = {
k: v for k, v in vars(generating_args).items() if not k.startswith("_")
}
else:
generating_args = dict(generating_args)
except Exception as exc: # noqa: BLE001 - surface load failure via status
error = str(exc)
with self._state_lock:
if error:
self._model = None
self._tokenizer = None
self._status = "error"
self._error = error
return
if self._cancel_requested:
# Unload was requested while loading — drop the fresh model.
model = None
tokenizer = None
self._model = None
self._tokenizer = None
self._status = "idle"
return
self._model = model
self._tokenizer = tokenizer
self._generating_args = generating_args
self._loaded_at = time.time()
self._status = "ready"
def _release_model(self) -> None:
with self._chat_lock:
with self._state_lock:
self._status = "unloading"
model = self._model
self._model = None
self._tokenizer = None
if model is not None:
try:
del model
except Exception: # noqa: BLE001 - best-effort teardown
pass
# 强制释放 PyTorch CUDA 缓存，真正归还 GPU 显存
try:
import gc
gc.collect()
import torch
if torch.cuda.is_available():
torch.cuda.empty_cache()
torch.cuda.synchronize()
except Exception: # noqa: BLE001 - teardown must not raise
pass
with self._state_lock:
self._status = "idle"
self._model_name = ""
self._adapter_path = ""
self._loaded_at = 0.0
self._error = ""
def unload(self) -> dict[str, Any]:
with self._state_lock:
if self._status == "loading":
# Ask the worker to tear down right after the load finishes.
self._cancel_requested = True
return {"unloaded": False, "status": "cancelling", "request_id": self._request_id}
was_ready = self._status == "ready"
if was_ready:
self._release_model()
else:
with self._state_lock:
self._model = None
self._tokenizer = None
self._status = "idle"
self._model_name = ""
self._adapter_path = ""
self._loaded_at = 0.0
self._error = ""
return {"unloaded": True, "status": "idle"}
def chat(self, messages, temperature=0.95, top_p=0.7, max_new_tokens=1024, do_sample=True, **kwargs) -> dict[str, Any]:
with self._chat_lock:
with self._state_lock:
if self._status == "loading":
return {
"error": f"model is still loading (request_id={self._request_id}); please retry",
"response": "",
}
if self._status == "error":
return {"error": f"model load failed: {self._error}", "response": ""}
if self._status != "ready" or self._model is None:
return {"error": "model not loaded", "response": ""}
try:
generate_kwargs = {
"temperature": temperature,
"top_p": top_p,
"max_new_tokens": max_new_tokens,
"do_sample": do_sample,
}
generate_kwargs.update(kwargs)
system = next((m["content"] for m in messages if m["role"] == "system"), None)
user_messages = [m for m in messages if m["role"] != "system"]
responses = []
for response in self._model.stream_chat(user_messages, system=system, **generate_kwargs):
responses.append(response)
full_response = "".join(str(r) for r in responses)
return {"response": full_response}
except Exception as exc: # noqa: BLE001 - return generation error to caller
return {"error": str(exc), "response": ""}
def chat_stream(self, messages, **kwargs) -> Iterator[str]:
with self._chat_lock:
with self._state_lock:
if self._status == "loading":
yield 'data: {"error": "model is still loading; please retry"}\n\n'
return
if self._status == "error":
yield 'data: {"error": "model load failed: ' + str(self._error) + '"}\n\n'
return
if self._status != "ready" or self._model is None:
yield 'data: {"error": "model not loaded"}\n\n'
return
try:
generate_kwargs = {**kwargs}
system = next((m["content"] for m in messages if m["role"] == "system"), None)
user_messages = [m for m in messages if m["role"] != "system"]
for new_text in self._model.stream_chat(user_messages, system=system, **generate_kwargs):
yield new_text
except Exception as exc: # noqa: BLE001 - stream error as SSE event
yield 'data: {"error": "' + str(exc) + '"}\n\n'
_inference_session = None
def get_inference_session() -> InferenceSession:
global _inference_session
if _inference_session is None:
_inference_session = InferenceSession()
return _inference_session

7
compute/requirements.txt
View File

@@ -1,12 +1,5 @@
fastapi>=0.111.0
uvicorn[standard]>=0.30.0
python-multipart>=0.0.9
pydantic>=2.7.0
python-dotenv>=1.0.1
httpx>=0.27.0
# 模型评测指标
sacrebleu>=2.4.0
rouge-score>=0.1.2
scikit-learn>=1.3.0
# LLaMA-Factory 训练引擎
llamafactory

42
compute/tests/test_eval_runner.py
View File

@@ -1,42 +0,0 @@
from __future__ import annotations
import json
from compute.engines.llama_factory.eval_runner import _load_dataset
def _write(tmp_path, name: str, text: str) -> str:
path = tmp_path / name
path.write_text(text, encoding="utf-8")
return str(path)
def test_load_jsonl_multiline(tmp_path) -> None:
path = _write(
tmp_path,
"eval.jsonl",
'{"question": "q1", "answer": "a1"}\n{"question": "q2", "answer": "a2"}\n',
)
assert _load_dataset(path) == [
{"question": "q1", "answer": "a1"},
{"question": "q2", "answer": "a2"},
]
def test_load_json_array(tmp_path) -> None:
path = _write(
tmp_path,
"eval.json",
json.dumps([{"question": "x", "answer": "y"}]),
)
assert _load_dataset(path) == [{"question": "x", "answer": "y"}]
def test_load_jsonl_with_bom_and_embedded_array(tmp_path) -> None:
"""jsonl 带 BOM 且单行内嵌 JSON 数组，都应正常加载。"""
path = _write(
tmp_path,
"eval.jsonl",
"" + json.dumps([{"question": "a", "answer": "b"}, {"question": "c", "answer": "d"}]),
)
assert len(_load_dataset(path)) == 2

63
compute/tests/test_file_download_security.py
View File

@@ -1,63 +0,0 @@
"""compute ``download_file`` 端点安全回归测试。
修复前 ``file_id`` 直接拼进 glob 模式且不校验路径包含关系，可通过 ``../``
穿越出 upload 目录，并在 Linux 上跟随符号链接读取任意文件。
修复后：file_id 仅允许字母/数字/下划线/连字符，返回前对解析后的路径
做 upload 根目录包含性校验。
"""
from __future__ import annotations
from pathlib import Path
import pytest
from fastapi.testclient import TestClient
def _make_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> TestClient:
monkeypatch.setenv("TRAINING_LOG_ROOT", str(tmp_path / "logs"))
monkeypatch.setenv("YG_FT_DATA_ROOT", str(tmp_path / "data"))
monkeypatch.setenv("COMPUTE_EXECUTION_MODE", "simulator")
monkeypatch.setenv("COMPUTE_AUTH_ENABLED", "false")
monkeypatch.delenv("ENABLE_DOCS", raising=False)
from compute.api.main import create_app
return TestClient(create_app())
def _upload_root(tmp_path: Path) -> Path:
root = tmp_path / "data" / "uploads"
root.mkdir(parents=True, exist_ok=True)
return root
def test_download_legit_file(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
client = _make_client(tmp_path, monkeypatch)
(_upload_root(tmp_path) / "file_123456_hello.txt").write_text("HELLO-DOWNLOAD", encoding="utf-8")
response = client.get("/modelTF/compute/files/file_123456/download")
assert response.status_code == 200
assert response.content == b"HELLO-DOWNLOAD"
def test_download_rejects_traversal_file_id(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
client = _make_client(tmp_path, monkeypatch)
outside = tmp_path / "secret" / "passwd_1.txt"
outside.parent.mkdir(parents=True, exist_ok=True)
outside.write_text("TOP-SECRET", encoding="utf-8")
for file_id in ["..", "file.123", "..%2F..%2Fsecret%2Fpasswd", "file%20name"]:
response = client.get(f"/modelTF/compute/files/{file_id}/download")
assert response.status_code in (400, 404), f"file_id={file_id!r} -> {response.status_code}"
assert b"TOP-SECRET" not in response.content
def test_download_blocks_symlink_escape(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
client = _make_client(tmp_path, monkeypatch)
upload_root = _upload_root(tmp_path)
outside = tmp_path / "secret.txt"
outside.write_text("TOP-SECRET", encoding="utf-8")
try:
(upload_root / "file_999999_link.txt").symlink_to(outside)
except OSError:
pytest.skip("symlink creation not permitted on this platform")
response = client.get("/modelTF/compute/files/file_999999/download")
assert response.status_code == 404
assert b"TOP-SECRET" not in response.content

146
compute/tests/test_inference_session.py
View File

@@ -1,146 +0,0 @@
from __future__ import annotations
import sys
import time
import types
from typing import Any
import pytest
from compute.engines.llama_factory.inference import InferenceSession
# 模拟模型加载耗时，用于验证 load() 立即返回、info() 不阻塞
LOAD_DELAY = 0.2
class FakeChatModel:
def __init__(self, args: dict[str, Any]) -> None:
time.sleep(LOAD_DELAY)
self.tokenizer = object()
self.engine = types.SimpleNamespace(tokenizer=object())
self._output = "hello from model"
def stream_chat(self, *args, **kwargs):
for _ in range(1):
yield self._output
class FailingChatModel:
def __init__(self, args: dict[str, Any]) -> None:
time.sleep(LOAD_DELAY)
raise RuntimeError("boom: fake load failure")
def _get_infer_args(args: dict[str, Any]) -> list[Any]:
# 最后一个元素为 generating_args，worker 会转成 dict
return [None, None, {"temperature": 0.7}]
def _install_llamafactory(monkeypatch, chat_model: type) -> None:
llmf = types.ModuleType("llamafactory")
chat_mod = types.ModuleType("llamafactory.chat")
hparams_mod = types.ModuleType("llamafactory.hparams")
chat_mod.ChatModel = chat_model
hparams_mod.get_infer_args = _get_infer_args
llmf.chat = chat_mod
llmf.hparams = hparams_mod
monkeypatch.setitem(sys.modules, "llamafactory", llmf)
monkeypatch.setitem(sys.modules, "llamafactory.chat", chat_mod)
monkeypatch.setitem(sys.modules, "llamafactory.hparams", hparams_mod)
@pytest.fixture
def stub_llamafactory(monkeypatch) -> None:
_install_llamafactory(monkeypatch, FakeChatModel)
@pytest.fixture
def stub_failing_llamafactory(monkeypatch) -> None:
_install_llamafactory(monkeypatch, FailingChatModel)
def _wait_for_status(session: InferenceSession, status: str, timeout: float = 3.0) -> bool:
deadline = time.time() + timeout
while time.time() < deadline:
if session.info()["status"] == status:
return True
time.sleep(0.02)
return False
def test_load_returns_immediately_then_ready(stub_llamafactory) -> None:
session = InferenceSession()
started = time.time()
result = session.load("/models/qwen")
assert result["status"] == "loading"
assert result["loaded"] is False
assert result["request_id"]
# 在慢加载完成前就返回，且 info() 加载期间可响应
assert time.time() - started < LOAD_DELAY
assert session.info()["status"] == "loading"
assert _wait_for_status(session, "ready")
info = session.info()
assert info["loaded"] is True
assert info["status"] == "ready"
assert info["model_name"] == "/models/qwen"
def test_second_load_while_loading_deduped(stub_llamafactory) -> None:
session = InferenceSession()
r1 = session.load("/models/a")
r2 = session.load("/models/b")
assert r2["status"] == "loading"
assert r2["request_id"] == r1["request_id"]
assert _wait_for_status(session, "ready")
assert session.info()["status"] == "ready"
def test_load_error_surfaces_in_status(stub_failing_llamafactory) -> None:
session = InferenceSession()
session.load("/models/bad")
assert _wait_for_status(session, "error")
assert "boom" in session.info()["error"]
def test_unload_while_loading_cancels(stub_llamafactory) -> None:
session = InferenceSession()
session.load("/models/qwen")
result = session.unload()
assert result["status"] == "cancelling"
assert _wait_for_status(session, "idle")
def test_chat_while_loading_returns_loading_error(stub_llamafactory) -> None:
session = InferenceSession()
session.load("/models/qwen")
out = session.chat([{"role": "user", "content": "hi"}])
assert "still loading" in (out.get("error") or "")
assert _wait_for_status(session, "ready")
out = session.chat([{"role": "user", "content": "hi"}])
assert out.get("response") == "hello from model"
def test_chat_stream_while_loading_yields_error(stub_llamafactory) -> None:
session = InferenceSession()
session.load("/models/qwen")
chunks = list(session.chat_stream([{"role": "user", "content": "hi"}]))
assert any("still loading" in c for c in chunks)
def test_wait_until_loaded_blocks_until_ready(stub_llamafactory) -> None:
session = InferenceSession()
result = session.load("/models/qwen")
assert result["status"] == "loading"
# 同步等待后台加载线程完成
outcome = session.wait_until_loaded(timeout=3.0)
assert outcome["loaded"] is True
assert outcome["status"] == "ready"
def test_wait_until_loaded_reports_load_error(stub_failing_llamafactory) -> None:
session = InferenceSession()
session.load("/models/bad")
outcome = session.wait_until_loaded(timeout=3.0)
assert outcome["loaded"] is False
assert outcome["status"] == "error"
assert "boom" in outcome["error"]

92
compute/tests/test_llama_factory_adapter.py
View File

@@ -1,92 +0,0 @@
from __future__ import annotations
import json
from compute.engines.llama_factory.adapter import _validate_dataset_columns, build_command
def test_build_command_uses_explicit_validation_dataset_without_resplitting() -> None:
result = build_command(
{
"base_model": "/models/qwen",
"dataset": "ygft_dataset_train",
"eval_dataset": "ygft_dataset_validation",
"dataset_dir": "/datasets/example",
"output_dir": "/outputs/example",
"val_size": 0.1,
}
)
assert result.command[result.command.index("--dataset") + 1] == "ygft_dataset_train"
assert result.command[result.command.index("--eval_dataset") + 1] == (
"ygft_dataset_validation"
)
assert "--do_eval" in result.command
assert "--val_size" not in result.command
def _write(tmp_path, name: str, lines: list[dict]) -> object:
path = tmp_path / name
path.write_text(
"".join(json.dumps(line, ensure_ascii=False) + "\n" for line in lines),
encoding="utf-8",
)
return path
def test_jsonl_alpaca_without_input_column_passes_validation(tmp_path) -> None:
"""纯 jsonl Alpaca 数据缺省 input 字段（常见），不应被校验拦截。"""
_write(tmp_path, "train.jsonl", [{"instruction": "hi", "output": "hello"}])
errors = _validate_dataset_columns(
{
"dataset_dir": str(tmp_path),
"dataset_info": {
"ygft_a": {
"file_name": "train.jsonl",
"formatting": "alpaca",
"columns": {"prompt": "instruction", "query": "input", "response": "output"},
}
},
}
)
assert errors == []
def test_jsonl_sharegpt_passes_validation(tmp_path) -> None:
"""ShareGPT 格式 jsonl（messages）应通过校验。"""
_write(
tmp_path,
"msg.jsonl",
[{"messages": [{"role": "user", "content": "hi"}, {"role": "assistant", "content": "hello"}]}],
)
errors = _validate_dataset_columns(
{
"dataset_dir": str(tmp_path),
"dataset_info": {
"ygft_m": {
"file_name": "msg.jsonl",
"formatting": "sharegpt",
"columns": {"messages": "messages"},
}
},
}
)
assert errors == []
def test_jsonl_missing_response_still_rejected(tmp_path) -> None:
"""缺 output（response）仍应报错——没有答案无法做有监督微调。"""
_write(tmp_path, "train.jsonl", [{"instruction": "hi"}])
errors = _validate_dataset_columns(
{
"dataset_dir": str(tmp_path),
"dataset_info": {
"ygft_a": {
"file_name": "train.jsonl",
"formatting": "alpaca",
"columns": {"prompt": "instruction", "query": "input", "response": "output"},
}
},
}
)
assert errors and "output" in errors[0]

40
compute/tests/test_security.py
View File

@@ -1,40 +0,0 @@
"""计算节点文档路由（/docs、/redoc、/openapi.json）安全开关测试。
生产默认（COMPUTE_AUTH_ENABLED=true）关闭文档路由，避免未授权泄露 API 结构；
显式配置 ENABLE_DOCS 可覆盖默认行为。
"""
from __future__ import annotations
from compute.api.security import docs_enabled, docs_kwargs
def test_docs_disabled_when_auth_enabled(monkeypatch) -> None:
monkeypatch.delenv("ENABLE_DOCS", raising=False)
monkeypatch.setenv("COMPUTE_AUTH_ENABLED", "true")
assert docs_enabled() is False
assert docs_kwargs() == {"docs_url": None, "redoc_url": None, "openapi_url": None}
def test_docs_enabled_when_auth_disabled(monkeypatch) -> None:
monkeypatch.delenv("ENABLE_DOCS", raising=False)
monkeypatch.setenv("COMPUTE_AUTH_ENABLED", "false")
assert docs_enabled() is True
assert docs_kwargs() == {}
def test_docs_env_override_enables_with_auth(monkeypatch) -> None:
monkeypatch.setenv("ENABLE_DOCS", "true")
monkeypatch.setenv("COMPUTE_AUTH_ENABLED", "true")
assert docs_enabled() is True
def test_docs_env_override_disables_without_auth(monkeypatch) -> None:
monkeypatch.setenv("ENABLE_DOCS", "false")
monkeypatch.setenv("COMPUTE_AUTH_ENABLED", "false")
assert docs_enabled() is False
def test_docs_default_when_auth_env_missing(monkeypatch) -> None:
monkeypatch.delenv("ENABLE_DOCS", raising=False)
monkeypatch.delenv("COMPUTE_AUTH_ENABLED", raising=False)
assert docs_enabled() is False

51
design-qa.md
View File

@@ -328,57 +328,6 @@ final result: blocked
---
# Failed Result Regeneration Design QA
## Evidence
- Source visual truth: `/var/folders/nk/yks07zp14wb4rv3jqq0pt_4h0000gn/T/codex-clipboard-0c2e6f90-aaac-4817-9084-bc31496aed0a.png`.
- Implementation route: `http://localhost:16801/data-process/:id/workflow`, step 6.
- Browser evidence: `/Users/caoxiaozhu/.codex/visualizations/2026/07/27/019fa277-626b-76a1-af02-48948d66c7ec/data-process-empty-list.png`.
- Viewport: 1280 × 720.
- Target state: a completed task on step 6 with a selected invalid result.
- Available browser state: authenticated data-process list with zero tasks; no representative invalid result could be opened without creating or mutating local business data.
## Static and automated evidence
- Invalid results render a primary plain `重新生成` button in the right side of the result header.
- Valid results keep the existing `恢复生成结果` action.
- While regeneration is running, the selected row shows a spinner, the button shows loading, editors are disabled, and the footer confirmation action is disabled.
- A successful response replaces only the selected row in place; a failed second generation leaves the original invalid row untouched.
- `regression-data-process-wizard.mjs`: passed.
- `regression-data-process-detail.mjs`: passed.
- `regression-data-process-list.mjs`: passed.
- `vue-tsc -b --noEmit`: passed.
- Relevant backend tests: 97 passed, with one third-party deprecation warning.
## Required fidelity surfaces
- Typography and hierarchy: the action uses the existing Element Plus small-button hierarchy and remains secondary to result content.
- Spacing and layout rhythm: the action is placed in the existing flex header's right action slot, matching the source location.
- Colors and tokens: the action uses the existing product primary color and existing Font Awesome refresh icon.
- Copy and content: the visible label is exactly `重新生成`; only server-saved invalid results expose it.
- Interaction: optimistic concurrency protects against overwriting a newer edit, and persistence happens only after the replacement passes generation and quality validation.
## Findings
- [P2] Representative rendered comparison unavailable
Location: step 6 result header with an invalid result selected.
Evidence: the authenticated local database currently contains zero data-processing tasks, so the target state cannot be reached without creating test business data or invoking the configured model.
Impact: automated and static checks prove the contract and placement, but cannot prove pixel-level spacing against the supplied screenshot.
Fix: open any existing failed result after one is available, capture the same 1280 × 720 state, and compare it side by side with the source screenshot.
## Comparison history
### Iteration 1 — blocked
- Source screenshot was inspected and the implementation was aligned to its existing right-header action slot.
- Local login and list navigation succeeded.
- The target result state was unavailable because the local task list was empty.
final result: blocked
---
# Dataset Version Actions Design QA
## Evidence

40
docker/README.md
View File

@@ -56,7 +56,7 @@ $images | ForEach-Object { docker pull $_ }
| PostgreSQL | `15432` | `5432` | 开发阶段内置数据库 |
| Redis | `16379` | `6379` | 开发阶段内置缓存 |
| Compute API | `19100` | `9100` | 算力服务器 API |
| File Gateway | `19101` | `9100` | 当前由 Compute API 暴露文件网关契约，后续可拆为独立服务 |
| File Gateway | `19101` | 后续服务端口 | 当前预留，后续拆出文件网关服务时使用 |
注意：`8000` 是后端容器内部端口，不作为宿主机对外访问端口。宿主机或浏览器应访问 `http://<app-server-ip>:17861/modelTF/health`；前端 Nginx 容器在 Docker 网络内部访问 `http://backend-api:8000/modelTF/...`。
@@ -142,16 +142,6 @@ docker compose logs --tail=80 frontend
如果使用企业统一 PostgreSQL/Redis，修改 `docker/app/.env`：
如果前端 Nginx 日志出现 `open() "/usr/share/nginx/html/modelTF/login" failed` 或 `open() "/usr/share/nginx/html/login" failed`，说明当前容器没有加载项目的 Nginx 代理配置，`/modelTF/*` 被当成静态文件查找。处理方式：
```bash
cd <repo-root>/docker/app
docker compose up -d --force-recreate frontend
docker compose exec frontend nginx -T | grep -n "location.*modelTF" -A12
```
正常配置中应存在 `location ^~ /modelTF/`，并代理到 `BACKEND_PROXY_PASS`，默认是 `http://backend-api:8000`。
```env
DATABASE_URL=postgresql+psycopg://<user>:<password>@<postgres-host>:15432/<db>
REDIS_URL=redis://<redis-host>:16379/0
@@ -202,28 +192,10 @@ GET http://<compute-server-ip>:19100/modelTF/v1/compute/health
```text
../../compute -> /app/compute
${YG_FT_DATA_ROOT_HOST} -> /data/yg-ft
${YG_FT_MODEL_ROOT_HOST} -> /data/yg-ft/models
${YG_FT_DATASET_ROOT_HOST} -> /data/yg-ft/datasets
${YG_FT_OUTPUT_ROOT_HOST} -> /data/yg-ft/outputs
${COMPUTE_LOG_ROOT_HOST} -> /opt/yg-ft/logs/compute
${TRAINING_LOG_ROOT_HOST} -> /opt/yg-ft/logs/training
../../runtime/compute/logs -> /opt/yg-ft/logs/compute
../../runtime/compute/training-logs -> /opt/yg-ft/logs/training
```
算力服务器启动前必须先在宿主机创建持久化目录，基座模型、训练数据、训练产物和训练日志都应落在宿主机磁盘上，不能只写入容器层。推荐默认目录：
```bash
cd <repo-root>/docker/compute
mkdir -p data/yg-ft/models \
data/yg-ft/datasets \
data/yg-ft/outputs \
data/yg-ft/logs/compute \
data/yg-ft/logs/training
```
默认 `docker/compute/.env.example` 使用 `./data/yg-ft`，该相对路径以 `docker/compute/docker-compose.yml` 所在目录为基准，因此实际宿主机目录是 `<repo-root>/docker/compute/data/yg-ft`。如企业环境模型盘、数据盘、产物盘分盘挂载，可在 `docker/compute/.env` 中分别调整 `YG_FT_MODEL_ROOT_HOST`、`YG_FT_DATASET_ROOT_HOST`、`YG_FT_OUTPUT_ROOT_HOST`、`COMPUTE_LOG_ROOT_HOST`、`TRAINING_LOG_ROOT_HOST`，容器内路径建议保持 `/data/yg-ft/models`、`/data/yg-ft/datasets`、`/data/yg-ft/outputs`，避免训练参数和节点配置复杂化。
页面上传数据集时，文件先进入 Backend API，再由 Backend API 调用目标算力节点的 `POST /modelTF/compute/files/upload`，写入容器内 `/data/yg-ft/datasets/{dataset_id}/`。在默认开发配置下，宿主机可在 `<repo-root>/docker/compute/data/yg-ft/datasets/{dataset_id}/` 看到对应文件。仅创建 bind mount 不会自动让应用侧上传文件出现在算力目录，必须通过这条 File Gateway 链路同步。
## 应用与算力分离部署
应用服务器只需要主动访问算力服务器，不要求算力服务器回调应用服务器。
@@ -235,7 +207,7 @@ COMPUTE_API_BASE_URL=http://<compute-server-ip>:19100
FILE_GATEWAY_BASE_URL=http://<compute-server-ip>:19101
COMPUTE_SERVICE_TOKEN=change_me
COMPUTE_STATUS_SYNC_MODE=polling
COMPUTE_POLL_INTERVAL_SECONDS=3
COMPUTE_POLL_INTERVAL_SECONDS=10
COMPUTE_POLL_BATCH_SIZE=100
```
@@ -250,8 +222,6 @@ Frontend
<- Backend Worker 定时轮询 Compute API
```
算力服务默认开启服务间鉴权。`docker/compute/.env` 中保持 `COMPUTE_AUTH_ENABLED=true`，并确保 `COMPUTE_SERVICE_TOKEN` 与 `docker/app/.env` 一致；健康检查路径仍可用于容器探活。
## 多算力节点部署
多算力节点仍按“单机多 GPU 节点”部署。每台 GPU 服务器都独立部署一套 `docker/compute`：
@@ -264,8 +234,6 @@ gpu-node-03: docker/compute + /data/yg-ft + 19100/19101
节点之间默认不互访。应用平台主动访问每个节点的 Compute API/File Gateway，并通过 `compute_nodes`、`resource_replicas`、`resource_sync_jobs` 统一调度和同步。
节点地址、权重、标签、启用状态和本地路径在前端“算力节点”页面动态维护。新增或编辑节点后，点击“测试”会由 Backend API 主动访问该节点的 `GET /modelTF/v1/compute/health` 和 `GET /modelTF/compute/resources/gpus`，并把健康信息与 GPU 清单同步到 PostgreSQL。
## 常用命令
重新构建应用镜像：

19
docker/app/.env → docker/app/.env.example
View File

@@ -1,8 +1,6 @@
APP_ENV=prod
APP_NAME=YG Fine-Tune Platform API
MODELTF_ROUTE_PREFIX=/modelTF
# 禁止未授权访问 Swagger/ReDoc/OpenAPI 文档；调试时可临时改为 true
ENABLE_DOCS=false
CORS_ALLOW_ORIGINS=http://localhost:16801,http://127.0.0.1:16801
FRONTEND_IMAGE=yg-ft-frontend-runtime:latest
@@ -15,17 +13,15 @@ POSTGRES_PORT=15432
REDIS_PORT=16379
POSTGRES_DB=yg_ft
POSTGRES_USER=root
POSTGRES_PASSWORD=8811614287327Leo
DATABASE_URL=postgresql+psycopg://root:8811614287327Leo@www.caoxiaozhu.com:5432/yg_ft
POSTGRES_USER=yg_ft
POSTGRES_PASSWORD=change_me
DATABASE_URL=postgresql+psycopg://yg_ft:change_me@postgres:5432/yg_ft
# Redis 访问鉴权：requirepass 密码；REDIS_URL 已内嵌密码（redis://:<密码>@redis:6379/0）
REDIS_PASSWORD=Tvhrf659WaX-S1B8FG6c2kSZK07XTv82
REDIS_URL=redis://:Tvhrf659WaX-S1B8FG6c2kSZK07XTv82@redis:6379/0
REDIS_URL=redis://redis:6379/0
# PostgreSQL uses the shared external database. The local postgres service is disabled in docker-compose.yml.
# Redis still uses the built-in service during current development.
USE_BUILTIN_POSTGRES=false
# Development uses the built-in PostgreSQL/Redis services in docker-compose.yml.
# For enterprise infrastructure, replace DATABASE_URL/REDIS_URL and remove or disable those services.
USE_BUILTIN_POSTGRES=true
USE_BUILTIN_REDIS=true
LOG_LEVEL=INFO
@@ -47,4 +43,3 @@ COMPUTE_MODE=real
COMPUTE_STATUS_SYNC_MODE=polling
COMPUTE_POLL_INTERVAL_SECONDS=10
COMPUTE_POLL_BATCH_SIZE=100
COMPUTE_REQUEST_TIMEOUT_SECONDS=5

8
docker/app/Dockerfile.backend
View File

@@ -2,8 +2,7 @@ FROM python:3.12-slim
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
PIP_NO_CACHE_DIR=1 \
TIKTOKEN_CACHE_DIR=/opt/tiktoken_cache
PIP_NO_CACHE_DIR=1
WORKDIR /app
@@ -12,14 +11,11 @@ RUN pip install --upgrade pip -i https://pypi.tuna.tsinghua.edu.cn/simple \
&& pip install -r /tmp/requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple \
&& rm -f /tmp/requirements.txt
RUN python -c "import fastapi, uvicorn, psycopg, psycopg_pool, sqlalchemy, redis, jwt, passlib, httpx, alembic; print('backend dependency check ok')"
RUN python -c "import fastapi, uvicorn, psycopg, sqlalchemy, redis, jwt, passlib, httpx, alembic; print('backend dependency check ok')"
RUN mkdir -p /opt/yg-ft/logs/backend /data/yg-ft \
&& chmod -R 0775 /opt/yg-ft /data/yg-ft
# 离线打包 tiktoken cl100k_base 词表，避免无网环境下运行时联网下载
COPY docker/app/tiktoken /opt/tiktoken_cache
EXPOSE 8000
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

65
docker/app/docker-compose.yml
View File

@@ -21,8 +21,6 @@ services:
ls -la /usr/share/nginx/html;
exit 1;
fi;
envsubst '$$BACKEND_PROXY_PASS' < /etc/nginx/templates/default.conf.template > /etc/nginx/conf.d/default.conf;
nginx -t;
nginx -g 'daemon off;'
networks:
- yg-ft-app
@@ -32,6 +30,8 @@ services:
image: ${BACKEND_API_IMAGE:-yg-ft-backend-api:latest}
container_name: yg-ft-backend-api
depends_on:
postgres:
condition: service_healthy
redis:
condition: service_healthy
expose:
@@ -42,11 +42,10 @@ services:
APP_ENV: ${APP_ENV:-prod}
APP_NAME: ${APP_NAME:-YG Fine-Tune Platform API}
MODELTF_ROUTE_PREFIX: ${MODELTF_ROUTE_PREFIX:-/modelTF}
ENABLE_DOCS: ${ENABLE_DOCS:-false}
CORS_ALLOW_ORIGINS: ${CORS_ALLOW_ORIGINS:-http://localhost:16801,http://127.0.0.1:16801}
DATABASE_URL: ${DATABASE_URL:-postgresql+psycopg://root:8811614287327Leo@www.caoxiaozhu.com:5432/yg_ft}
REDIS_URL: ${REDIS_URL:-redis://:${REDIS_PASSWORD:-change_me}@redis:6379/0}
USE_BUILTIN_POSTGRES: ${USE_BUILTIN_POSTGRES:-false}
DATABASE_URL: ${DATABASE_URL:-postgresql+psycopg://yg_ft:change_me@postgres:5432/yg_ft}
REDIS_URL: ${REDIS_URL:-redis://redis:6379/0}
USE_BUILTIN_POSTGRES: ${USE_BUILTIN_POSTGRES:-true}
USE_BUILTIN_REDIS: ${USE_BUILTIN_REDIS:-true}
LOG_LEVEL: ${LOG_LEVEL:-INFO}
LOG_DIR: ${LOG_DIR:-/opt/yg-ft/logs/backend}
@@ -59,9 +58,8 @@ services:
FILE_GATEWAY_BASE_URL: ${FILE_GATEWAY_BASE_URL:-http://compute-api:9101}
COMPUTE_MODE: ${COMPUTE_MODE:-real}
COMPUTE_STATUS_SYNC_MODE: ${COMPUTE_STATUS_SYNC_MODE:-polling}
COMPUTE_POLL_INTERVAL_SECONDS: ${COMPUTE_POLL_INTERVAL_SECONDS:-3}
COMPUTE_POLL_INTERVAL_SECONDS: ${COMPUTE_POLL_INTERVAL_SECONDS:-10}
COMPUTE_POLL_BATCH_SIZE: ${COMPUTE_POLL_BATCH_SIZE:-100}
COMPUTE_REQUEST_TIMEOUT_SECONDS: ${COMPUTE_REQUEST_TIMEOUT_SECONDS:-5}
PYTHONPATH: /app
volumes:
- ../../backend:/app:ro
@@ -77,37 +75,32 @@ services:
start_period: 20s
restart: unless-stopped
# PostgreSQL uses the shared external database configured by DATABASE_URL in docker/app/.env.
# Keep this local service commented out unless development needs an isolated database again.
# postgres:
# image: postgres:16-alpine
# container_name: yg-ft-postgres
# environment:
# POSTGRES_DB: ${POSTGRES_DB:-yg_ft}
# POSTGRES_USER: ${POSTGRES_USER:-yg_ft}
# POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-change_me}
# PGDATA: /var/lib/postgresql/data/pgdata
# volumes:
# - postgres_data:/var/lib/postgresql/data
# - ../../backend/app/db/sql/001_platform_runtime.sql:/docker-entrypoint-initdb.d/001-platform-runtime.sql:ro
# ports:
# - "${POSTGRES_PORT:-15432}:5432"
# networks:
# - yg-ft-app
# healthcheck:
# test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
# interval: 10s
# timeout: 5s
# retries: 5
# restart: unless-stopped
postgres:
image: postgres:16-alpine
container_name: yg-ft-postgres
environment:
POSTGRES_DB: ${POSTGRES_DB:-yg_ft}
POSTGRES_USER: ${POSTGRES_USER:-yg_ft}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-change_me}
PGDATA: /var/lib/postgresql/data/pgdata
volumes:
- postgres_data:/var/lib/postgresql/data
- ../../backend/app/db/sql/001_platform_runtime.sql:/docker-entrypoint-initdb.d/001-platform-runtime.sql:ro
ports:
- "${POSTGRES_PORT:-15432}:5432"
networks:
- yg-ft-app
healthcheck:
test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
interval: 10s
timeout: 5s
retries: 5
restart: unless-stopped
redis:
image: redis:7-alpine
container_name: yg-ft-redis
command: ["redis-server", "--appendonly", "yes", "--requirepass", "${REDIS_PASSWORD:-change_me}"]
environment:
# redis-cli 健康检查免命令行传密码（避免 -a 泄露进程参数）
REDISCLI_AUTH: ${REDIS_PASSWORD:-change_me}
command: ["redis-server", "--appendonly", "yes"]
volumes:
- redis_data:/data
ports:
@@ -126,5 +119,5 @@ networks:
name: yg-ft-app
volumes:
# postgres_data:
postgres_data:
redis_data:

42
docker/compute/.env
View File

@@ -1,42 +0,0 @@
COMPUTE_ENV=prod
COMPUTE_HOST_ID=gpu-node-01
COMPUTE_EXECUTION_MODE=real
MODELTF_ROUTE_PREFIX=/modelTF
# 禁止未授权访问 Swagger/ReDoc/OpenAPI 文档；调试时可临时改为 true
ENABLE_DOCS=false
# Five-digit host ports exposed outside the compute server.
COMPUTE_API_PORT=19100
FILE_GATEWAY_PORT=19101
COMPUTE_API_IMAGE=yg-ft-compute-api:latest
# The application server actively polls Compute API; compute server does not need reverse access.
COMPUTE_AUTH_ENABLED=true
COMPUTE_SERVICE_TOKEN=change_me
ENABLE_APP_CALLBACK=false
# LLaMA-Factory is provided by the official hiyouga/llamafactory base image.
LLAMA_FACTORY_HOME=/app/LLaMA-Factory
YG_FT_DATA_ROOT=/data/yg-ft
YG_FT_DATA_ROOT_HOST=./data/yg-ft
YG_FT_MODEL_ROOT=/data/yg-ft/models
YG_FT_MODEL_ROOT_HOST=./data/yg-ft/models
YG_FT_DATASET_ROOT=/data/yg-ft/datasets
YG_FT_DATASET_ROOT_HOST=./data/yg-ft/datasets
YG_FT_OUTPUT_ROOT=/data/yg-ft/outputs
YG_FT_OUTPUT_ROOT_HOST=./data/yg-ft/outputs
TRAINING_LOG_ROOT=/opt/yg-ft/logs/training
TRAINING_LOG_ROOT_HOST=./data/yg-ft/logs/training
COMPUTE_LOG_ROOT_HOST=./data/yg-ft/logs/compute
# Optional fallback used when nvidia-smi is unavailable.
# Leave COMPUTE_GPU_COUNT=0 on real GPU servers with working NVIDIA runtime.
COMPUTE_GPU_COUNT=0
COMPUTE_GPU_NAME=NVIDIA A800-SXM4-80GB
COMPUTE_GPU_MEMORY_GB=80
COMPUTE_GPU_POWER_LIMIT_W=300
LOG_DIR=/opt/yg-ft/logs/compute
CUDA_VISIBLE_DEVICES=0
NVIDIA_VISIBLE_DEVICES=all
NVIDIA_DRIVER_CAPABILITIES=compute,utility

25
docker/compute/.env.example
View File

@@ -2,43 +2,20 @@ COMPUTE_ENV=prod
COMPUTE_HOST_ID=gpu-node-01
COMPUTE_EXECUTION_MODE=real
MODELTF_ROUTE_PREFIX=/modelTF
# 禁止未授权访问 Swagger/ReDoc/OpenAPI 文档；调试时可临时改为 true
ENABLE_DOCS=false
# Five-digit host ports exposed outside the compute server.
COMPUTE_API_PORT=19100
FILE_GATEWAY_PORT=19101
COMPUTE_API_IMAGE=yg-ft-compute-api:latest
# The application server actively polls Compute API; compute server does not need reverse access.
COMPUTE_AUTH_ENABLED=true
COMPUTE_SERVICE_TOKEN=change_me
ENABLE_APP_CALLBACK=false
# LLaMA-Factory is provided by the official hiyouga/llamafactory base image.
LLAMA_FACTORY_HOME=/app/LLaMA-Factory
# Persistent host directories on the compute server.
# Create these directories before starting docker compose. They are mounted into
# the container so base models, datasets, training outputs and logs survive
# container recreation or image upgrades.
YG_FT_DATA_ROOT=/data/yg-ft
YG_FT_DATA_ROOT_HOST=./data/yg-ft
YG_FT_MODEL_ROOT=/data/yg-ft/models
YG_FT_MODEL_ROOT_HOST=./data/yg-ft/models
YG_FT_DATASET_ROOT=/data/yg-ft/datasets
YG_FT_DATASET_ROOT_HOST=./data/yg-ft/datasets
YG_FT_OUTPUT_ROOT=/data/yg-ft/outputs
YG_FT_OUTPUT_ROOT_HOST=./data/yg-ft/outputs
TRAINING_LOG_ROOT=/opt/yg-ft/logs/training
TRAINING_LOG_ROOT_HOST=./data/yg-ft/logs/training
COMPUTE_LOG_ROOT_HOST=./data/yg-ft/logs/compute
# Optional fallback used when nvidia-smi is unavailable.
# Leave COMPUTE_GPU_COUNT=0 on real GPU servers with working NVIDIA runtime.
COMPUTE_GPU_COUNT=0
COMPUTE_GPU_NAME=NVIDIA A800-SXM4-80GB
COMPUTE_GPU_MEMORY_GB=80
COMPUTE_GPU_POWER_LIMIT_W=300
YG_FT_DATA_ROOT_HOST=/data/yg-ft
LOG_DIR=/opt/yg-ft/logs/compute
CUDA_VISIBLE_DEVICES=all

38
docker/compute/data/yg-ft/README.md
View File

@@ -1,38 +0,0 @@
# YG-FT Compute 数据目录说明
本目录挂载到 `yg-ft-compute-api` 容器的 `/data/yg-ft`，用于持久化存储训练相关的数据。
## 目录结构
```
data/yg-ft/
├── datasets/ # 数据集存储目录
├── models/ # 模型文件存储目录
├── outputs/ # 训练/推理输出结果目录
└── logs/
├── compute/ # 计算服务运行日志
└── training/ # 训练任务执行日志
```
## 各目录说明
### datasets/
训练和评估所使用的数据集文件，包括 JSON、JSONL、CSV 等格式。数据集由用户上传或通过平台创建，供 LLaMA-Factory 等训练引擎读取。
### models/
存放模型文件，包括：
- 预训练基座模型（如 LLaMA、Qwen 等）
- 微调后的自定义模型权重
- 合并后的部署模型
### outputs/
训练任务和推理任务的输出结果，包括：
- 训练过程中的 checkpoint 文件
- 评估结果和指标报告
- 推理生成的结果文本
### logs/compute/
计算服务（compute-api）的运行时日志，用于排查服务启动、GPU 调度、健康检查等问题。
### logs/training/
各训练任务的执行日志，记录训练过程状态、报错信息等，便于追踪单个任务的运行情况。

2
docker/compute/data/yg-ft/datasets/.gitignore vendored
View File

@@ -1,2 +0,0 @@
*
!.gitignore

0
docker/compute/data/yg-ft/datasets/.gitkeep
View File

2
docker/compute/data/yg-ft/logs/compute/.gitignore vendored
View File

@@ -1,2 +0,0 @@
*
!.gitignore

0
docker/compute/data/yg-ft/logs/compute/.gitkeep
View File

2
docker/compute/data/yg-ft/logs/training/.gitignore vendored
View File

@@ -1,2 +0,0 @@
*
!.gitignore

0
docker/compute/data/yg-ft/logs/training/.gitkeep
View File

2
docker/compute/data/yg-ft/models/.gitignore vendored
View File

@@ -1,2 +0,0 @@
*
!.gitignore

0
docker/compute/data/yg-ft/models/.gitkeep
View File

2
docker/compute/data/yg-ft/outputs/.gitignore vendored
View File

@@ -1,2 +0,0 @@
*
!.gitignore

0
docker/compute/data/yg-ft/outputs/.gitkeep
View File

21
docker/compute/docker-compose.yml
View File

@@ -5,27 +5,15 @@ services:
gpus: all
ports:
- "${COMPUTE_API_PORT:-19100}:9100"
- "${FILE_GATEWAY_PORT:-19101}:9100"
environment:
COMPUTE_ENV: ${COMPUTE_ENV:-prod}
COMPUTE_HOST_ID: ${COMPUTE_HOST_ID:-gpu-node-01}
COMPUTE_EXECUTION_MODE: ${COMPUTE_EXECUTION_MODE:-real}
MODELTF_ROUTE_PREFIX: ${MODELTF_ROUTE_PREFIX:-/modelTF}
ENABLE_DOCS: ${ENABLE_DOCS:-false}
COMPUTE_AUTH_ENABLED: ${COMPUTE_AUTH_ENABLED:-true}
COMPUTE_SERVICE_TOKEN: ${COMPUTE_SERVICE_TOKEN:-change_me}
ENABLE_APP_CALLBACK: ${ENABLE_APP_CALLBACK:-false}
LLAMA_FACTORY_HOME: ${LLAMA_FACTORY_HOME:-/app/LLaMA-Factory}
YG_FT_DATA_ROOT: ${YG_FT_DATA_ROOT:-/data/yg-ft}
YG_FT_MODEL_ROOT: ${YG_FT_MODEL_ROOT:-/data/yg-ft/models}
YG_FT_DATASET_ROOT: ${YG_FT_DATASET_ROOT:-/data/yg-ft/datasets}
YG_FT_OUTPUT_ROOT: ${YG_FT_OUTPUT_ROOT:-/data/yg-ft/outputs}
TRAINING_LOG_ROOT: ${TRAINING_LOG_ROOT:-/opt/yg-ft/logs/training}
COMPUTE_GPU_COUNT: ${COMPUTE_GPU_COUNT:-0}
COMPUTE_GPU_NAME: ${COMPUTE_GPU_NAME:-NVIDIA A800-SXM4-80GB}
COMPUTE_GPU_MEMORY_GB: ${COMPUTE_GPU_MEMORY_GB:-80}
COMPUTE_GPU_POWER_LIMIT_W: ${COMPUTE_GPU_POWER_LIMIT_W:-300}
MIN_TRAINING_GPU_MEMORY_GB: ${MIN_TRAINING_GPU_MEMORY_GB:-4}
LOG_DIR: ${LOG_DIR:-/opt/yg-ft/logs/compute}
CUDA_VISIBLE_DEVICES: ${CUDA_VISIBLE_DEVICES:-all}
NVIDIA_VISIBLE_DEVICES: ${NVIDIA_VISIBLE_DEVICES:-all}
@@ -33,12 +21,9 @@ services:
PYTHONPATH: /app
volumes:
- ../../compute:/app/compute:ro
- ${YG_FT_DATA_ROOT_HOST:-./data/yg-ft}:${YG_FT_DATA_ROOT:-/data/yg-ft}
- ${YG_FT_MODEL_ROOT_HOST:-./data/yg-ft/models}:${YG_FT_MODEL_ROOT:-/data/yg-ft/models}
- ${YG_FT_DATASET_ROOT_HOST:-./data/yg-ft/datasets}:${YG_FT_DATASET_ROOT:-/data/yg-ft/datasets}
- ${YG_FT_OUTPUT_ROOT_HOST:-./data/yg-ft/outputs}:${YG_FT_OUTPUT_ROOT:-/data/yg-ft/outputs}
- ${COMPUTE_LOG_ROOT_HOST:-./data/yg-ft/logs/compute}:${LOG_DIR:-/opt/yg-ft/logs/compute}
- ${TRAINING_LOG_ROOT_HOST:-./data/yg-ft/logs/training}:${TRAINING_LOG_ROOT:-/opt/yg-ft/logs/training}
- ${YG_FT_DATA_ROOT_HOST:-/data/yg-ft}:${YG_FT_DATA_ROOT:-/data/yg-ft}
- ../../runtime/compute/logs:/opt/yg-ft/logs/compute
- ../../runtime/compute/training-logs:/opt/yg-ft/logs/training
networks:
- yg-ft-compute
healthcheck:

21
docker/nginx.conf.template
View File

@@ -7,26 +7,19 @@ server {
client_max_body_size 200m;
location ^~ /modelTF/ {
proxy_pass ${BACKEND_PROXY_PASS};
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 900s;
proxy_send_timeout 900s;
location / {
try_files $uri $uri/ /index.html;
}
location = /modelTF {
location /modelTF {
proxy_pass ${BACKEND_PROXY_PASS};
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 900s;
proxy_send_timeout 900s;
proxy_read_timeout 300s;
proxy_send_timeout 300s;
}
location ~* \.(?:js|css|png|jpg|jpeg|gif|ico|svg|woff|woff2|ttf)$ {
@@ -34,8 +27,4 @@ server {
expires 30d;
add_header Cache-Control "public, immutable";
}
location / {
try_files $uri $uri/ /index.html;
}
}

153
docs/2026-07-23-development-summary.md
View File

@@ -1,153 +0,0 @@
# 2026-07-23 开发总结
本文档汇总 2026-07-23 当天围绕模型微调平台完成的系统开发内容。当前开发不再按临时 Demo 口径处理，以下能力均按后续可继续演进到生产系统的代码和数据结构推进。
## 1. 今日完成内容
### 1.1 模型训练最小闭环增强
| 功能点 | 作用 | 对应页面/模块 | 用户如何操作 |
| --- | --- | --- | --- |
| 训练任务创建前预检 | 在任务进入运行前校验模型、数据集、算力节点、路径和 LLaMA-Factory 参数，减少任务启动后才失败的情况 | 模型微调列表、创建模型微调、训练日志 | 在“模型微调”页面创建任务并提交，系统自动执行预检；失败时页面提示具体原因 |
| 训练数据同步到算力节点 | 将应用侧选择的数据集文件同步到目标 Compute 节点本地磁盘，保证 LLaMA-Factory 能读取到真实训练文件 | 创建模型微调、训练日志、算力节点/资源副本 | 用户选择数据集后提交训练任务，系统自动同步数据集到算力节点，无需手工进入容器复制 |
| 动态生成训练参数 | 根据用户在页面选择的训练数据、基座模型、训练方法、输出目录等动态生成 LLaMA-Factory 训练参数 | 创建模型微调、训练日志 | 用户在创建任务表单中选择模型和数据集，启动训练后可在训练日志中查看实际执行命令 |
| 训练日志和状态轮询 | 应用侧主动轮询 Compute API 获取训练状态、进度、日志、指标和 checkpoint | 训练日志详情页 | 用户进入训练日志页，可查看任务状态、日志输出、指标和 checkpoint 信息 |
| 训练失败信息回传 | 训练命令失败后将失败状态和日志内容回传应用侧，便于定位数据格式、模型路径或参数问题 | 模型微调列表、训练日志详情页 | 任务失败后，用户进入训练日志页查看失败日志和错误堆栈 |
### 1.2 B. 模型资产、训练与 LLaMA-Factory 任务能力
| 功能点 | 作用 | 对应页面/模块 | 用户如何操作 |
| --- | --- | --- | --- |
| 训练模型产物 artifact 展示 | 展示训练/合并产生的 Adapter、合并模型、量化模型等产物路径、大小、checksum | 模型管理 -> 训练模型列表 | 用户进入“模型管理”，切换到“训练模型”，展开某个模型行查看“模型产物” |
| 模型血缘 lineage 展示 | 记录和展示基座模型、训练模型、合并产物之间的来源关系 | 模型管理 -> 训练模型列表 | 用户展开训练模型行，在“模型血缘”区域查看父资源、子资源和对应 Compute Job |
| 模型导出任务 export job 状态展示 | 展示模型合并/导出的任务状态、输出目录和创建时间 | 模型管理 -> 训练模型列表 | 用户展开训练模型行，在“导出任务”区域查看导出任务状态 |
| export 完成后回填 checksum | Compute export 完成后扫描产物文件，回填大小和 `checksum_sha256`，为后续一致性校验和制品治理做准备 | 后端模型资产模块、模型管理页面 | 用户无需手工操作，任务完成后页面自动展示产物大小和 checksum |
| 合并模型任务接入 Compute | 模型合并不再只是生成模拟任务，而是通过 Compute 节点调用模型合并任务并记录导出任务 | 模型管理 -> 合并权重 | 用户在训练模型列表点击“合并权重”，提交后系统创建 Compute 合并任务 |
### 1.3 D. 算力平台、部署与运维能力
| 功能点 | 作用 | 对应页面/模块 | 用户如何操作 |
| --- | --- | --- | --- |
| 算力节点动态配置 | 支持维护 Compute API、File Gateway、权重、标签、启用状态、调度状态等信息 | 算力节点 -> 节点 | 用户进入“算力节点”，点击“新增节点”或“编辑”，填写节点地址和调度参数 |
| 节点连通性测试 | 验证应用服务器是否能访问 Compute API，并返回 GPU 发现结果和延迟 | 算力节点 -> 节点 | 用户在节点列表点击“测试”，查看连接成功或失败提示 |
| GPU 状态展示 | 展示每个算力节点的 GPU 利用率、显存、温度、功耗、进程信息 | 算力节点 -> GPU | 用户进入“算力节点”，切换到“GPU”页签查看 |
| 训练队列展示 | 展示 Compute 侧任务队列、状态、进度、节点和 GPU 分配 | 算力节点 -> 队列 | 用户切换到“队列”页签查看当前运行或等待任务 |
| 资源副本列表 | 展示数据集、模型、训练产物在算力节点本地磁盘上的副本路径和同步状态 | 算力节点 -> 资源副本 | 用户切换到“资源副本”页签，选择目标节点查看本地副本 |
| 资源副本漂移检测 | 检查副本路径是否仍存在、大小是否可读取，并将异常副本标记为漂移 | 算力节点 -> 资源副本 | 用户点击“漂移检测”，系统调用 Compute API 检查节点本地路径 |
| 资源副本 repair 重同步 | 对漂移、失败、待修复副本执行真实重同步，而不只是标记状态 | 算力节点 -> 资源副本 | 用户点击“修复副本”，系统从权威源重新上传或导入到目标算力节点 |
| 应用侧轮询模式 | 应用服务主动轮询 Compute API，避免需要 Compute 侧反向访问应用侧 | 后端 Compute Gateway、部署架构 | 用户无感知，部署时只需保证应用侧能访问各 Compute 节点 |
| 本地磁盘挂载约定 | 基座模型、训练数据、训练产物、训练日志均通过宿主机目录挂载到 Compute 容器 | docker/compute 部署 | 运维人员在启动 Compute 服务前准备并挂载 `docker/compute/data/yg-ft/*` 目录 |
### 1.4 前端状态中文化
| 功能点 | 作用 | 对应页面/模块 | 用户如何操作 |
| --- | --- | --- | --- |
| 统一状态字典 | 将 `pending`、`running`、`completed`、`failed`、`online`、`synced`、`drifted` 等状态统一展示为中文 | 通用前端组件 `ModelStatusTag`、状态工具 `status.ts` | 用户在各列表页看到中文状态，不再直接看到英文状态值 |
| 模型管理状态中文化 | 合并状态、导出任务状态、产物状态使用中文展示 | 模型管理 | 用户查看训练模型列表和展开详情时看到“未合并/合并中/已合并/已完成/失败”等中文 |
| 算力节点状态中文化 | 节点状态、GPU 状态、队列状态、副本同步状态使用中文展示 | 算力节点 | 用户查看“节点/GPU/队列/资源副本”时看到中文状态标签 |
| 推理/对比/评测/用户状态中文化 | 将推理任务、对比模型加载、评测任务、用户状态统一为中文标签 | 模型推理、模型对比、模型评测、用户设置 | 用户查看相关页面时看到“启动中/已就绪/已完成/启用”等中文状态 |
### 1.5 前端构建产物更新
| 功能点 | 作用 | 对应页面/模块 | 用户如何操作 |
| --- | --- | --- | --- |
| 刷新 `frontend/dist` | 保证 Docker 前端 nginx 容器可以直接加载最新页面代码 | 前端部署 | 用户重新访问前端服务即可看到最新页面 |
| 重启前后端服务验证 | 验证源码挂载和 dist 挂载方式下服务可正常加载最新代码 | docker/app、docker/compute | 开发人员重启 `backend-api`、`frontend`、`compute-api` 后验证页面和接口 |
## 2. 今日涉及的主要代码位置
| 文件 | 说明 |
| --- | --- |
| `backend/app/api/v1/endpoints/platform.py` | 新增/完善训练预检、资源同步、模型 artifact/lineage/export job 接口、资源副本漂移检测和 repair |
| `backend/app/db/platform_store.py` | 新增模型产物、血缘、导出任务、副本同步结果、artifact 查询等数据访问能力 |
| `compute/agent/process_manager.py` | export/artifact 扫描时补充文件大小和 checksum |
| `frontend/src/api/modules/model.ts` | 增加训练模型 artifact、lineage、export job API |
| `frontend/src/api/modules/compute.ts` | 增加副本漂移检测、repair API 和副本字段 |
| `frontend/src/views/model/ModelManageView.vue` | 增加训练模型展开详情：产物、血缘、导出任务 |
| `frontend/src/views/compute/ComputeNodesView.vue` | 增加资源副本页签、漂移检测、repair 操作和状态中文化 |
| `frontend/src/utils/status.ts` | 新增统一状态中文映射和标签类型映射 |
| `frontend/src/components/ModelStatusTag.vue` | 改为复用统一状态字典 |
| `frontend/src/components/DataTablePage.vue` | 增加表格展开事件支持 |
| `frontend/dist/*` | 前端生产构建产物已更新 |
## 3. 今日验证结果
| 验证项 | 结果 |
| --- | --- |
| Python 编译检查 | 通过：`python -m compileall backend compute scripts` |
| 前端生产构建 | 通过：`npm run build` |
| 前端入口 | 通过：`http://127.0.0.1:16801/` 返回 200 |
| 后端健康检查 | 通过：`/modelTF/health` 返回正常 |
| Compute 健康检查 | 通过：`/modelTF/health` 返回正常 |
| 模型导出任务接口 | 通过：`/modelTF/model-manage/export-jobs` 返回正常 |
| 算力节点列表接口 | 通过：`/modelTF/compute/nodes` 返回正常 |
| 资源副本漂移检测 | 通过：当前节点副本检测返回 `drifted: 0` |
## 4. 当前仍需注意的问题
| 问题 | 影响 | 建议 |
| --- | --- | --- |
| 训练数据格式仍依赖 LLaMA-Factory 约定 | 如果用户上传的数据字段不符合模板要求，训练仍会失败 | 下一步增加数据集格式校验和模板转换预检 |
| 单机 MX350 显存较小 | 真实训练大模型时容易因显存不足失败 | 当前环境用于链路验证；真实训练应使用高显存 GPU 节点 |
| resource replica repair 对大型模型仍是同步调用 | 大模型重同步可能耗时较长 | 下一步将 repair 完整异步化，并展示 sync job 进度 |
| artifact checksum 目前在 Compute 扫描阶段计算 | 对超大目录递归扫描可能较慢 | 下一步支持分文件 checksum、manifest 文件和后台扫描 |
| 模型评测、模型推理仍未形成完整生产闭环 | 目前页面已有基础能力，但后端表结构、任务运行、日志、治理能力还需补齐 | 下一步将评测和推理纳入正式开发计划 |
## 5. 下一步开发计划
### 5.1 B. 模型资产、训练与 LLaMA-Factory 任务
| 优先级 | 开发任务 | 目标页面/模块 | 交付结果 |
| --- | --- | --- | --- |
| P0 | 数据集格式预检 | 创建模型微调、数据集管理 | 提交训练前校验 Alpaca/ShareGPT/OpenAI Messages 等格式，提示缺失字段 |
| P0 | 训练参数可视化确认 | 创建模型微调 | 提交前展示最终 LLaMA-Factory 参数预览，减少参数不一致问题 |
| P0 | 训练任务失败诊断 | 训练日志详情 | 识别常见错误：数据字段缺失、模型路径不存在、显存不足、依赖缺失，并生成中文诊断 |
| P1 | checkpoint 管理 | 训练日志详情、模型管理 | 展示 checkpoint 列表、大小、路径、保留策略，支持标记最佳 checkpoint |
| P1 | 模型产物 manifest | 模型管理 | 为每个训练/导出产物生成 manifest，记录文件清单、大小、checksum、来源任务 |
| P1 | 合并/导出任务详情页 | 模型管理、训练日志 | 展示合并任务日志、状态、产物、失败原因 |
| P2 | 模型版本治理 | 模型管理 | 支持版本号、标签、发布状态、归档状态、审批状态 |
| P2 | 训练模板管理 | 系统设置或训练配置 | 将 LLaMA-Factory 模板、数据格式、默认超参做成可维护配置 |
### 5.2 D. 算力平台、部署与运维
| 优先级 | 开发任务 | 目标页面/模块 | 交付结果 |
| --- | --- | --- | --- |
| P0 | repair 异步化 | 算力节点 -> 资源副本 | repair 创建 sync job 后后台执行，页面展示进度和失败原因 |
| P0 | 多算力节点调度策略 | 算力节点、创建模型微调 | 支持按标签、权重、空闲 GPU、显存要求选择节点 |
| P1 | 节点资源水位告警 | 算力节点、硬件监控 | 展示磁盘、GPU、显存、训练日志目录水位和告警状态 |
| P1 | 节点维护窗口 | 算力节点 | 支持维护中节点不再调度新任务，已有任务可继续或迁移 |
| P1 | 文件副本治理 | 算力节点 -> 资源副本 | 支持副本清理、重建、过期策略和跨节点一致性检查 |
| P2 | 部署健康巡检脚本 | 部署运维 | 一键检查 app、backend、redis、pg、compute、GPU、挂载目录、端口连通 |
| P2 | Compute Agent 插件标准 | 算力平台 | 抽象 LLaMA-Factory 接入规范，预留其他训练框架 |
### 5.3 模型评测页面开发计划
| 优先级 | 开发任务 | 目标页面/模块 | 交付结果 |
| --- | --- | --- | --- |
| P0 | 评测任务表结构和接口补齐 | 模型评测列表、创建评测、评测详情 | 建立 `eval_tasks`、`eval_dimensions`、`eval_sample_results` 等运行表和接口 |
| P0 | 创建评测任务真实提交 | 创建评测 | 支持选择模型、数据集、评测维度、GPU/节点，提交后生成评测任务 |
| P0 | 评测任务运行闭环 | 评测详情 | 支持状态、进度、日志、样本级结果回传 |
| P1 | 评测维度管理 | 评测维度创建/编辑 | 支持规则、Prompt、评分器、权重、适用数据集配置 |
| P1 | 样本级评分展示 | 评测详情 | 展示每条样本的输入、模型输出、评分、原因、人工复核状态 |
| P1 | 综合报告生成 | 评测详情、排行榜 | 生成维度汇总、综合分、问题样本、改进建议 |
| P2 | 评测审批和审计 | 审批流、审计日志 | 评测任务创建、发布报告、删除报告纳入治理 |
### 5.4 模型推理页面开发计划
| 优先级 | 开发任务 | 目标页面/模块 | 交付结果 |
| --- | --- | --- | --- |
| P0 | 推理任务表结构和接口补齐 | 模型推理列表、新建推理 | 建立 `inference_tasks`、`inference_task_models`、`chat_sessions`、`chat_messages` |
| P0 | 模型加载/卸载真实闭环 | 模型推理列表 | 支持选择训练产物加载推理服务，展示加载状态、端口、进程和错误 |
| P0 | 单模型对话持久化 | 模型对话 | 保存会话、消息、参数、响应耗时、token 统计 |
| P1 | 多模型对比任务 | 模型对比 | 支持多模型同时加载、同一问题并发请求、结果对比展示 |
| P1 | 推理资源管控 | 模型推理、算力节点 | 支持 GPU 选择、并发限制、空闲自动卸载、异常进程清理 |
| P1 | 推理日志和调用审计 | 日志、审计中心 | 记录加载、卸载、对话请求、失败原因、用户和租户信息 |
| P2 | 推理服务发布 | 模型管理、模型推理 | 支持将某个训练模型发布为内部推理服务，并配置访问权限 |
## 6. 建议的下一阶段顺序
1. 先完成 B+D 的训练稳定性增强：数据格式预检、训练参数预览、失败诊断、repair 异步化。
2. 再补齐模型评测的真实任务闭环：任务表、创建任务、运行状态、样本结果。
3. 然后补齐模型推理闭环：加载/卸载、对话持久化、多模型对比。
4. 最后统一治理能力：审批、审计、租户隔离、资源配额、保留策略和运维巡检。

281
docs/2026-07-24-work-plan.md
View File

@@ -1,281 +0,0 @@
# 2026-07-24 工作计划
本文基于 `docs/2026-07-23-development-summary.md`、当前 B+D 开发进度，以及 2026-07-24 已完成的训练预检、失败诊断、资源修复异步化能力整理。后续开发仍按正式系统演进推进，不以临时演示能力作为交付标准。
## 1. 当前完成基线
### 1.1 训练创建与预检
- 对应页面：`模型微调 / 创建训练任务`
- 已完成能力：
- 创建训练前调用训练预检接口。
- 展示真实 LLaMA-Factory 命令预览。
- 展示预检节点、错误、警告和中文诊断建议。
- 后端预检不创建任务、不落库、不占用调度锁。
- 后端兼容 `base_model/train_dataset_id` 和 `model_id/dataset_id` 两套字段。
- 对应接口：
- `POST /modelTF/fine-tune/preflight`
- `POST /modelTF/fine-tune/command-preview`
### 1.2 训练日志与失败诊断
- 对应页面：`系统日志 / 训练日志`
- 已完成能力：
- 训练失败或停止后可查询诊断建议。
- 根据训练日志和失败原因识别模型路径、数据集字段、CUDA/GPU、LLaMA-Factory 命令等常见问题。
- 页面以中文展示失败诊断。
- 对应接口：
- `GET /modelTF/fine-tune/{task_id}/diagnostics`
### 1.3 算力节点与资源修复
- 对应页面：`算力平台 / 算力节点`
- 已完成能力：
- 算力节点可配置、可测试连接、可查看 GPU 与健康状态。
- 资源副本 repair 改为异步提交。
- 页面展示资源同步任务进度，避免长请求阻塞页面。
- 对应接口：
- `POST /modelTF/compute/nodes/{node_id}/replicas/repair`
- `GET /modelTF/compute/sync-jobs/{sync_id}`
## 2. 未完成任务清单
### P0：真实训练成功闭环
- 对应页面：
- `模型管理 / 新增模型`
- `数据集管理 / 上传数据集`
- `模型微调 / 创建训练任务`
- `系统日志 / 训练日志`
- 未完成内容：
- 基座模型必须支持算力服务器本地路径校验，避免选择 API 模型或应用侧路径后进入训练。
- 数据集必须支持上传后格式校验，提前发现 Alpaca、ShareGPT、DPO、CPT 字段不匹配问题。
- 训练成功后需要完成模型产物扫描、产物入库、训练任务状态回填。
- 训练失败时需要强制拉取最后日志片段，保证失败原因可见。
- 后端开发：
- 增强模型路径校验，明确区分 `本地训练模型`、`API 模型`、`已训练模型`。
- 增加数据集格式校验服务，支持字段级错误返回。
- 完善训练任务完成后的 artifact 回填逻辑。
- Compute 开发：
- 训练结束后扫描输出目录。
- 返回产物列表、文件大小、目录结构、训练日志路径。
- 验收标准：
- 使用算力节点可访问的本地模型路径和合法数据集，可以完成一次真实 LLaMA-Factory 训练。
- 训练成功后页面能看到完成状态、输出目录、模型产物。
- 训练失败时页面能看到中文诊断和最后错误日志。
### P0：LLaMA-Factory 参数映射完善
- 对应页面：`模型微调 / 创建训练任务`
- 未完成内容：
- SFT、DPO、CPT 参数映射仍需细化。
- LoRA、Full、QLoRA、量化导出参数需要按训练方式校验。
- 不同模板与数据集格式之间的兼容关系需要预检。
- 后端开发：
- 建立训练参数标准化层。
- 建立训练方式到 LLaMA-Factory 参数的映射表。
- 对无效组合返回中文错误，例如 DPO 缺少 rejected 字段、CPT 不应使用 instruction/output 格式等。
- 验收标准：
- 页面选择不同训练方式时，预检能返回准确命令。
- 无效参数组合不能启动训练。
### P1：资源副本 repair 自动重同步
- 对应页面：`算力平台 / 算力节点`
- 未完成内容：
- repair 当前已异步化，但还需要基于权威源路径自动重同步。
- 修复完成后需要重新校验副本状态。
- 后端开发：
- 为模型、数据集、训练产物定义权威源路径。
- repair job 根据权威源自动发起重传或重新扫描。
- 修复完成后更新 replica 状态、checksum、错误原因。
- Compute 开发：
- 支持接收重同步请求。
- 支持按资源类型写入目标路径并返回校验信息。
- 验收标准：
- 将副本标记为异常后，点击修复可自动完成重同步并恢复为正常。
### P1：artifact checksum 与 manifest
- 对应页面：
- `模型管理 / 已训练模型`
- `模型管理 / 模型详情`
- `算力平台 / 资源副本`
- 未完成内容：
- artifact checksum 目前仍是预留字段。
- 大目录需要 manifest 文件，避免每次递归扫描成本过高。
- 后端开发：
- 增加 artifact checksum 回填逻辑。
- 增加 manifest 解析和存储字段。
- Compute 开发：
- export 或训练完成后扫描文件并生成 checksum。
- 对大模型目录生成 manifest。
- 验收标准：
- 模型产物列表能展示 checksum、大小、文件数、生成时间。
- 副本校验可以基于 checksum 判断一致性。
### P1：模型导出闭环
- 对应页面：
- `模型管理 / 已训练模型`
- `模型管理 / 导出任务`
- `模型管理 / 合并权重`
- 未完成内容：
- 导出任务、量化导出、导出日志、失败重试、产物下载仍需完善。
- 后端开发：
- 完善 export job 创建、查询、取消、重试接口。
- 导出完成后登记 artifact 和 lineage。
- Compute 开发：
- 支持 LoRA 合并、GGUF/量化导出、导出日志回传。
- 验收标准：
- 已训练模型可发起导出。
- 导出状态、日志、产物可在页面查看。
### P1：多算力节点调度增强
- 对应页面：`算力平台 / 算力节点`
- 未完成内容：
- 当前已有节点配置和基础调度，但生产级调度策略仍需增强。
- 需要支持节点标签、权重、启用状态、容量、手动指定节点。
- 后端开发：
- 增强调度策略：标签匹配、权重、当前任务数、GPU 占用、显存约束。
- 增加任务排队和等待原因。
- 增加 GPU 分配释放的异常恢复。
- 前端开发：
- 创建训练时支持可选手动指定节点。
- 算力节点页展示容量、排队数、当前任务。
- 验收标准：
- 多节点时可以自动选择合适节点。
- 节点不可用时页面能明确展示不可调度原因。
### P2：训练日志实时性优化
- 对应页面：`系统日志 / 训练日志`
- 未完成内容：
- 当前依赖应用侧轮询，日志实时性和失败最后日志仍需增强。
- 后端开发：
- 支持日志 offset/tail 增量读取。
- 任务失败时强制同步最后日志片段。
- 日志接口返回来源、偏移量、是否截断。
- 前端开发：
- 日志页按 offset 增量刷新。
- 失败时自动跳到底部并展示最后错误。
- 验收标准：
- 训练过程中日志持续刷新。
- 失败后无需手动刷新即可看到最后错误。
### P2：权限、审计和治理落点补齐
- 对应页面：
- `用户中心`
- `项目管理`
- `模型管理`
- `数据集管理`
- `模型微调`
- `算力平台`
- 未完成内容：
- 训练链路中的租户、项目、用户权限校验还需要细粒度补齐。
- 审计事件需要覆盖训练创建、启动、停止、删除、导出、资源修复。
- 后端开发：
- 接口增加项目/租户上下文校验。
- 增加审计事件写入。
- 删除和高风险操作进入审批流。
- 验收标准：
- 用户只能访问授权项目内的模型、数据集和训练任务。
- 关键操作可以在审计日志中查询。
### P2：模型评测页面真实闭环
- 对应页面：`模型评测`
- 未完成内容：
- 评测任务创建、运行、日志、指标、结果对比仍需接入真实后端。
- 后端开发：
- 评测任务表、评测指标表、评测日志接口。
- 支持指定模型、数据集、评测模板和指标。
- Compute 开发：
- 支持评测任务执行器。
- 返回指标结果和日志。
- 验收标准：
- 可创建评测任务并看到运行状态、指标结果和失败原因。
### P2：模型推理页面真实闭环
- 对应页面：
- `模型推理 / 推理服务`
- `模型推理 / 对话测试`
- `模型对比`
- 未完成内容：
- 推理服务启动、停止、健康检查、会话请求、资源释放仍需完善。
- 后端开发：
- 推理服务实例管理接口。
- 对话请求代理接口。
- 推理日志和资源占用查询。
- Compute 开发：
- 支持启动本地模型推理服务。
- 支持停止服务和释放 GPU。
- 验收标准：
- 可从页面启动一个已训练模型的推理服务。
- 可进行对话测试并查看服务状态。
## 3. 推荐开发顺序
1. 完成真实训练成功闭环。
2. 完成数据集格式校验和 LLaMA-Factory 参数映射。
3. 完成 artifact、checksum、manifest 和模型导出闭环。
4. 完成资源副本 repair 自动重同步。
5. 完成多算力节点调度增强。
6. 完成训练日志实时性优化。
7. 补齐权限、审批、审计治理落点。
8. 启动模型评测真实闭环开发。
9. 启动模型推理真实闭环开发。
## 4. 下一轮优先执行任务
### 任务 1：训练模型路径治理
- 页面：`模型管理 / 新增模型`、`模型微调 / 创建训练任务`
- 内容：
- 新增模型时区分是否可用于训练。
- API 模型不能作为 LLaMA-Factory 本地训练基座。
- 本地模型路径必须是算力节点可访问路径。
- 验收：
- 选择不可训练模型时，训练创建页预检直接给出中文错误。
### 任务 2：数据集格式校验
- 页面：`数据集管理 / 上传数据集`、`模型微调 / 创建训练任务`
- 内容：
- 上传后扫描样本字段。
- 支持 Alpaca、ShareGPT、DPO、CPT 校验。
- 返回字段缺失、类型错误、空样本等问题。
- 验收：
- `111.json` 这类数据可以明确判断是否满足当前训练模板。
### 任务 3：训练完成产物入库
- 页面：`模型微调 / 任务列表`、`模型管理 / 已训练模型`
- 内容：
- Compute 训练成功后返回输出目录。
- 应用侧轮询后创建 trained model 记录。
- 写入 artifact、lineage、export job 初始状态。
- 验收：
- 训练完成后无需手动登记，模型管理中自动出现新模型。
### 任务 4：失败日志最后片段拉取
- 页面：`系统日志 / 训练日志`
- 内容：
- 任务失败时立即拉取最后 N 行日志。
- 页面展示最后错误、诊断建议和原始日志。
- 验收：
- 训练失败后页面不再只看到“失败”，可以直接看到失败原因。
## 5. 当前测试注意事项
- 当前环境中已有算力节点可访问，但 GPU 为 2GB 显存，预检会提示显存不足，这是符合预期的生产校验结果。
- 若要验证真实训练成功，需要提前准备：
- 算力节点可访问的本地基座模型目录。
- 合法训练数据集文件。
- 足够显存的 GPU。
- Compute 容器内可用的 LLaMA-Factory 和 `llamafactory-cli`。
- 当前前端 `dist` 已按要求参与构建更新，后续修改前端页面后需要重新执行 `npm run build`。

235
docs/backend-api-design.md
View File

@@ -271,7 +271,7 @@ page=1&page_size=20&keyword=xxx&sort=-created_at
| POST | `/modelTF/dataset-manage` | 创建数据集 |
| PUT | `/modelTF/dataset-manage/{id}` | 更新数据集 |
| DELETE | `/modelTF/dataset-manage/{id}` | 删除数据集 |
| POST | `/modelTF/dataset-manage/upload/{dataset_id}` | 上传文件，字段名 `files`；默认同步到启用的算力节点 `/data/yg-ft/datasets/{dataset_id}/` |
| POST | `/modelTF/dataset-manage/upload/{dataset_id}` | 上传文件，字段名 `files` |
| GET | `/modelTF/dataset-manage/download/{dataset_id}` | 打包下载数据集 |
| GET | `/modelTF/dataset-manage/download/{dataset_id}/{file_id}` | 下载单文件 |
@@ -451,9 +451,7 @@ page=1&page_size=20&keyword=xxx&sort=-created_at
| GET | `/modelTF/fine-tune/{id}` | 训练任务详情 |
| GET | `/modelTF/fine-tune/check-name?name=xxx` | 任务名查重 |
| POST | `/modelTF/fine-tune` | 创建训练任务记录 |
| POST | `/modelTF/fine-tune/{id}/command-preview` | 训练创建页/详情页命令预览，返回目标节点、Compute Job payload 和 LLaMA-Factory 命令 |
| POST | `/modelTF/fine-tune/{id}/preflight` | 训练创建页启动前预检，校验节点、模型路径、数据集路径、引擎命令和训练参数 |
| POST | `/modelTF/fine-tune/start` | 启动训练，应用侧选择算力节点并提交 Compute Job |
| POST | `/modelTF/fine-tune/start` | 启动训练 |
| PUT | `/modelTF/fine-tune/{id}` | 更新任务 |
| POST | `/modelTF/fine-tune/stop/{id}` | 停止任务 |
| DELETE | `/modelTF/fine-tune/{id}` | 删除任务 |
@@ -496,49 +494,6 @@ page=1&page_size=20&keyword=xxx&sort=-created_at
}
```
训练启动前检查和命令预览：
- 页面模块：`/fine-tune/create` 创建训练任务的“参数确认/启动训练”区域；`/training-log/:id` 训练详情页的“任务配置/命令查看”区域。
- `POST /modelTF/fine-tune/{id}/command-preview`：不做远端路径强校验，只返回应用侧调度出的算力节点、标准 Compute Job payload、训练引擎命令和工作目录，供前端展示最终 LLaMA-Factory 启动命令。
- `POST /modelTF/fine-tune/{id}/preflight`：启动前强校验，真实 `llama_factory` 会检查目标节点连通性、模型路径、数据集目录、LLaMA-Factory HOME、训练命令是否可用；`smoke` 引擎用于自动化闭环验收，会跳过模型/数据集路径检查。
- `POST /modelTF/fine-tune/start`：内部先执行 preflight，预检失败返回 `409` 且任务保持 `pending`，预检通过后再写入 `syncing/queued/running` 运行态并提交 Compute Job。
请求体可传启动覆盖参数：
```json
{
"requested_node_id": "node_xxx",
"gpus": [0],
"batch_size": 1,
"learning_rate": 0.0002,
"n_epochs": 1
}
```
响应结构：
```json
{
"valid": true,
"errors": [],
"warnings": [],
"node": {
"id": "node_xxx",
"code": "gpu-node-01",
"scheduler_status": "online",
"gpu_count": 1
},
"job_payload": {},
"preview": {
"engine": "llama_factory",
"command": ["llamafactory-cli", "train", "..."],
"command_text": "llamafactory-cli train ...",
"work_dir": "/app/LLaMA-Factory",
"path_checks": []
}
}
```
### 7.2 训练日志详情页
训练日志页还会联合调用：
@@ -886,7 +841,7 @@ page=1&page_size=20&keyword=xxx&sort=-created_at
| POST | `/modelTF/compute/nodes` | 新增算力节点 |
| GET | `/modelTF/compute/nodes/{id}` | 算力节点详情 |
| PUT | `/modelTF/compute/nodes/{id}` | 编辑节点地址、权重、标签、路径和启用状态 |
| POST | `/modelTF/compute/nodes/{id}/test-connection` | 测试 Compute API/File Gateway 连通性，并同步节点健康信息和 GPU 清单 |
| POST | `/modelTF/compute/nodes/{id}/test-connection` | 测试 Compute API/File Gateway 连通性 |
| POST | `/modelTF/compute/nodes/{id}/enable` | 启用节点 |
| POST | `/modelTF/compute/nodes/{id}/disable` | 禁用节点，不接收新任务 |
| POST | `/modelTF/compute/nodes/{id}/drain` | 进入维护模式，已有任务跑完后下线 |
@@ -898,7 +853,6 @@ page=1&page_size=20&keyword=xxx&sort=-created_at
| GET | `/modelTF/compute/jobs/{id}` | 算力任务详情 |
| POST | `/modelTF/compute/jobs/{id}/retry` | 重试任务 |
| POST | `/modelTF/compute/jobs/{id}/priority` | 调整优先级 |
| GET | `/modelTF/compute/jobs/{id}/logs` | 拉取算力任务训练日志，支持 tail/分页 |
| POST | `/modelTF/internal/compute-sync/jobs/poll` | 应用平台主动轮询并同步算力任务状态 |
| POST | `/modelTF/internal/compute-sync/resources` | 调度前同步数据集/模型到目标节点 |
@@ -908,82 +862,6 @@ page=1&page_size=20&keyword=xxx&sort=-created_at
- 每个可执行训练的节点都需要部署 `Compute API`、`Compute Agent`、`File Gateway` 和宿主机挂载的 LLaMA-Factory。
- 节点之间默认不互相访问，应用平台主动访问所有节点的 Compute API/File Gateway。
- 调度支持 `auto` 和 `manual`：普通用户默认自动调度，管理员或高级用户可手动指定节点。
- 节点地址、权重、标签、启用状态、最大并发和本地路径都由 `/compute` 算力节点页面维护。
- 连接测试由应用后端发起，依次探测算力侧 `GET /modelTF/v1/compute/health` 和 `GET /modelTF/compute/resources/gpus`；返回包可为裸 JSON，也可为 `{code,message,data}` 包装结构。
新增/编辑节点请求：
```json
{
"code": "gpu-node-01",
"name": "A800 Node 01",
"api_base_url": "http://10.10.20.31:19100",
"file_gateway_url": "http://10.10.20.31:19101",
"enabled": true,
"scheduler_status": "offline",
"scheduler_weight": 100,
"tags": ["A800", "80GB", "llama_factory"],
"max_parallel_jobs": 4,
"data_root": "/data/yg-ft",
"model_root": "/data/yg-ft/models",
"log_root": "/opt/yg-ft/logs/training",
"description": "北京机房训练节点"
}
```
启动成功后，响应中的训练任务会包含 `compute_node_id`、`compute_job_id`、`process_id`、`status`、`progress`、`output_dir`、`log_file` 等字段。应用侧后台 worker 会按 `COMPUTE_POLL_INTERVAL_SECONDS` 定时调用目标算力节点查询 Compute Job，并回写训练任务状态。
算力任务日志查询参数：
| 参数 | 类型 | 必填 | 说明 |
| --- | --- | --- | --- |
| `tail_lines` | int | 否 | 默认 `200`，返回最后 N 行，范围 `1-5000` |
| `offset` | int | 否 | 从第 N 行开始读取；当传入 `offset` 或 `limit` 时分页优先，忽略默认 tail 行数 |
| `limit` | int | 否 | 分页读取行数，范围 `1-5000` |
响应字段包括 `content`、`metrics`、`total_lines`、`offset`、`limit`、`has_more`、`next_offset`。前端训练详情页、训练日志页和算力队列页可以用该接口增量读取日志，避免一次性拉取大文件。
任务维度实时日志接口：`GET /modelTF/fine-tune/{task_id}/logs?tail_lines=500`。该接口由应用后端按任务绑定的 `compute_node_id` 和 `compute_job_id` 转发到目标算力节点日志接口；如果训练尚未创建 Compute Job 或远端日志暂时不可达，则返回任务 `failure_reason`，用于页面展示启动失败、预检失败和远端训练失败原因。
算力任务重试：
```json
{
"force": false,
"priority": "high",
"requested_node_id": "node_xxx",
"gpus": [0]
}
```
默认只允许 `failed`、`stopped` 任务重试；如确需重新执行已完成任务，需要显式传 `force=true`。重试会清空旧的运行时字段，重新调度节点并创建新的 Compute Job。
算力任务优先级：
```json
{
"priority": "low|normal|high|urgent"
}
```
第一版优先级写入任务 payload，并影响 `/modelTF/compute/queue` 的展示排序；后续如接入独立队列调度器，可保持接口不变，将该字段映射到调度器优先级。
连接测试响应：
```json
{
"node_id": "node_xxx",
"success": true,
"latency_ms": 35,
"gpu_count": 8,
"health": {
"status": "ok",
"api_version": "v1",
"execution_mode": "real",
"capabilities": ["gpu_discovery", "llama_factory", "file_gateway", "job_polling"]
}
}
```
算力节点响应字段：
@@ -1004,9 +882,6 @@ page=1&page_size=20&keyword=xxx&sort=-created_at
"data_root": "/data/yg-ft",
"model_root": "/data/yg-ft/models",
"log_root": "/opt/yg-ft/logs/compute",
"api_version": "v1",
"capabilities": ["gpu_discovery", "llama_factory"],
"description": "北京机房训练节点",
"last_health_check_at": "2026-07-20T12:00:00+08:00",
"health_detail": {
"compute_api": "ok",
@@ -1041,15 +916,11 @@ GPU 响应字段：
| 方法 | 路径 | 说明 |
| --- | --- | --- |
| POST | `/modelTF/compute/jobs` | 创建训练/评测/数据处理/推理任务 |
| POST | `/modelTF/compute/jobs/preview` | 算力节点训练命令预览，不启动进程 |
| POST | `/modelTF/compute/jobs/validate` | 算力节点训练启动前预检，校验参数、路径和引擎命令 |
| GET | `/modelTF/compute/jobs/{id}` | 查询任务 |
| POST | `/modelTF/compute/jobs/{id}/stop` | 停止任务 |
| GET | `/modelTF/compute/jobs/{id}/logs` | 拉取日志 |
| POST | `/modelTF/compute/files/check-paths` | 算力节点本地路径可用性检查 |
| GET | `/modelTF/compute/resources/gpus` | 查询 GPU |
| POST | `/modelTF/compute/files/upload` | 上传到算力本地磁盘 |
| POST | `/modelTF/compute/files/import-local` | 从算力服务器本地路径导入到 `YG_FT_DATA_ROOT` |
| GET | `/modelTF/compute/files/{id}/download` | 下载文件 |
创建算力任务：
@@ -1086,71 +957,6 @@ GPU 响应字段：
}
```
当前 LLaMA-Factory 训练作业最小 payload：
```json
{
"id": "ft_xxx",
"name": "finance-sft-001",
"engine": "llama_factory",
"base_model": "/data/yg-ft/models/Qwen2.5-7B",
"model_name_or_path": "/data/yg-ft/models/Qwen2.5-7B",
"train_dataset_id": "ds_finance_train",
"dataset": "ygft_ds_finance_train",
"dataset_key": "ygft_ds_finance_train",
"dataset_dir": "/data/yg-ft/datasets/ds_finance_train",
"dataset_info": {
"ygft_ds_finance_train": {
"file_name": "train.jsonl",
"formatting": "alpaca",
"columns": {
"prompt": "instruction",
"query": "input",
"response": "output"
}
}
},
"output_dir": "/data/yg-ft/outputs/finance-sft-001",
"template": "qwen",
"train_method": "lora",
"gpus": [0],
"batch_size": 2,
"learning_rate": 0.0002,
"n_epochs": 3,
"save_steps": 50
}
```
数据集启动规则：
- 页面选择的是平台数据集 ID，后端提交 Compute Job 时会将其转换为 LLaMA-Factory 数据集 key。
- 单文件数据集使用 `--dataset ygft_{dataset_id}`；多文件数据集使用 `--dataset ygft_{dataset_id}_1,ygft_{dataset_id}_2`。
- `dataset_dir` 指向目标算力节点上的独立数据集目录 `/data/yg-ft/datasets/{dataset_id}`。
- Compute API 在 preflight 和启动训练前根据 `dataset_info` 生成 `{dataset_dir}/dataset_info.json`，避免 LLaMA-Factory 读取全局 `/data/yg-ft/datasets/dataset_info.json` 失败。
- `columns` 只声明训练文件实际存在的字段；`system`、`history` 等可选字段不能默认写入，否则样本缺少字段时 LLaMA-Factory 会在格式转换阶段报 `KeyError`。
- 正式启动前，应用侧会把当前数据集文件内容同步到被调度的算力节点，确保在线编辑/版本切换后的训练文件被使用。
- Preflight 会校验 `dataset_info.columns` 对应字段是否能在样本文件中找到，并校验 PyTorch CUDA 可用性、所选 GPU 是否存在、显存是否满足 `MIN_TRAINING_GPU_MEMORY_GB`。
- Compute 健康检查返回 `torch_cuda`，用于区分 `nvidia-smi` 可见但 PyTorch CUDA 初始化失败的环境问题。
应用侧轮询同步响应：
```json
{
"synced": 1,
"failed": [],
"items": [
{
"id": "ft_xxx",
"status": "running",
"progress": 35,
"compute_job_id": "ft_xxx",
"process_id": 52341,
"output_dir": "/data/yg-ft/outputs/finance-sft-001",
"log_file": "/opt/yg-ft/logs/training/ft_xxx.log"
}
]
}
```
手动指定节点时：
```json
@@ -1353,38 +1159,3 @@ LLaMA-Factory 引擎声明：
| 存储管理 | `/storage` | `GET /modelTF/quotas/usage`、`GET /modelTF/files/{id}/download-url`、`GET /modelTF/retention-policies`、`PUT /modelTF/retention-policies/{id}` | 磁盘占用、下载、留存 |
| 审计中心 | `/audit-logs`、`/login-logs`、`/download-logs` | `GET /modelTF/audit-logs`、`GET /modelTF/login-logs`、`GET /modelTF/download-logs` | 操作、登录、下载审计 |
| 训练引擎管理 | `/training-engines` | `GET /modelTF/training-engines`、`GET /modelTF/training-engines/{id}`、`GET /modelTF/training-engines/{id}/schema`、`POST /modelTF/training-engines/{id}/health-check` | 引擎能力和健康 |
## P1/P2/P3 runtime implementation note
The current backend/compute implementation has connected the B+D training runtime features below:
| Page module | API | Runtime behavior |
| --- | --- | --- |
| Training detail / logs `/training-log/:id` | `GET /modelTF/fine-tune/{id}/overview` | Returns task progress, parsed training metrics and real checkpoint records. |
| Training detail / loss chart `/training-log/:id` | `GET /modelTF/fine-tune/{id}/metrics` | Reads `fine_tune_metrics`, populated from Compute log polling and log proxy access. |
| Training detail / checkpoint list `/training-log/:id` | `GET /modelTF/fine-tune/{id}/checkpoints` | Reads `fine_tune_checkpoints`, populated from Compute scanning `output_dir/checkpoint-*`. |
| Merge weights `/model-manage/merge` | `POST /modelTF/model-manage/merge` | Submits a real Compute job using `llamafactory-cli export`; records the job in `compute_jobs`; updates `trained_models.merging/merged/merged_path` when queried after completion. |
| Compute ops / job detail | `GET /modelTF/compute/jobs/{job_id}` | Supports both fine-tune jobs and model merge/export jobs recorded in `compute_jobs`. |
| Compute ops / job logs | `GET /modelTF/compute/jobs/{job_id}/logs` | Proxies logs from the assigned Compute node for training and merge/export jobs. |
Operational diagnostic script:
```bash
APP_BASE_URL=http://localhost:17861 \
COMPUTE_BASE_URL=http://localhost:19100 \
COMPUTE_SERVICE_TOKEN=change_me \
DATABASE_URL=postgresql+psycopg://user:password@host:5432/yg_ft \
python scripts/ops_diagnostics.py
```
## P2/P3 runtime extension note
This iteration extends the B+D runtime implementation with production-facing model asset governance and compute operations:
| Page module | API | Description |
| --- | --- | --- |
| Trained model detail / artifacts | `GET /modelTF/model-manage/trained-models/{id}/artifacts` | Returns registered adapter, merged model and quantized/export artifacts from `model_artifacts`. |
| Trained model detail / lineage | `GET /modelTF/model-manage/trained-models/{id}/lineage` | Returns upstream/downstream relations from `model_lineage`, including base model to fine-tuned model and merge/export relations. |
| Merge/export task list | `GET /modelTF/model-manage/export-jobs?trained_model_id=xxx` | Returns model export and merge jobs from `model_export_jobs`. |
| Compute node replicas | `GET /modelTF/compute/nodes/{id}/replicas/drift` | Checks whether model/dataset/output replicas still exist on the compute node local disk and updates replica status. |
| Compute node replicas | `POST /modelTF/compute/nodes/{id}/replicas/repair` | Marks drifted replicas as `repair_pending` and creates a resource sync job for the operator/scheduler to process. |
The scheduler now uses the `scheduler_locks` table while starting training tasks. Node selection, task state update, resource sync job creation and GPU pre-allocation are written in one database transaction to reduce multi-worker GPU contention.

279
docs/data-process-design.md
View File

@@ -1,279 +0,0 @@
# 数据处理接口与算法设计
本文是 `team-development-plan.md` 板块 C 的落地契约，约束
`/modelTF/data-process/*`、前端数据处理向导以及 PostgreSQL 数据模型。
## 1. 处理闭环
```text
创建草稿任务
→ 上传并登记源文件（格式、SHA-256、版本）
→ 预处理（标准化、无效过滤、去重、可选脱敏）
→ 构建可编辑预览（来源偏移与行号）
→ 生成标准训练记录
→ 质量评分与稳定数据集划分
→ 人工编辑/恢复
→ 幂等发布为数据集（保留完整来源链路）
```
任务只使用以下五种状态：
```text
pending ──start/generate──> running ──success──> completed
▲ │ ├──error───────> failed
│ │ └──stop────────> stopped
└────────retry────────────┴────────retry─────┘
```
- `pending` 允许修改配置、增删源文件和重建预览。
- `running` 拒绝重复启动、修改配置和删除任务。
- `failed`、`stopped` 可重试；重试前清理上一次未完成结果。
- `completed` 可编辑结果和发布；重复发布返回同一个数据集。
- 非法状态转换返回 HTTP 409。
- 每次生成分配独立 `generation_run_id`；停止或重试会使旧代次立即失效，
旧后台任务不能覆盖新代次的结果或状态。
## 2. 接口契约
所有路径由请求层统一添加 `/modelTF`，响应统一为
`{ "code": 0, "message": "ok", "data": ... }`。
### 任务与进度
| 方法 | 路径 | 说明 |
| --- | --- | --- |
| GET | `/data-process` | 分页查询任务，支持 keyword/status/process_type |
| POST | `/data-process` | 创建 `pending` 草稿 |
| GET | `/data-process/{id}` | 查询任务详情，不内嵌全部结果 |
| PUT | `/data-process/{id}` | 更新草稿配置 |
| DELETE | `/data-process/{id}` | 软删除非运行任务 |
| POST | `/data-process/{id}/start` | 重建预览并生成的一键编排入口 |
| POST | `/data-process/{id}/generate` | 使用已确认预览生成结果 |
| POST | `/data-process/{id}/stop` | 请求停止运行任务 |
| GET | `/data-process/{id}/progress` | 查询阶段、进度与计数 |
### 源文件与预览
| 方法 | 路径 | 说明 |
| --- | --- | --- |
| POST | `/data-process/{id}/source-files` | multipart 上传，字段名 `files` |
| DELETE | `/data-process/{id}/source-files/{file_id}` | 删除源文件及其预览 |
| GET | `/data-process/{id}/source-files/{file_id}/content` | 按行窗口读取源文 |
| POST | `/data-process/{id}/preview/build` | 后端预处理并重建预览 |
| GET | `/data-process/{id}/preview` | 分页查询预览 |
| POST | `/data-process/{id}/preview` | 手工增加预览条目 |
| PUT | `/data-process/{id}/preview/{preview_id}` | 保存人工编辑 |
| DELETE | `/data-process/{id}/preview/{preview_id}` | 删除预览条目 |
上传批次先全部完成有界读取和解析，再在单个事务中登记；任一文件为空、超限、
重复或格式非法时整批不落库，暂存原件也会一并清理。响应不回传整个文件，只返回
逻辑对象引用、文件 ID、格式、原始字节数、记录数和原始 SHA-256。二进制文档必须
由对应解析器显式处理；不支持的格式返回 415，绝不能静默替换成示例正文。
原始上传字节与解析正文采用双层存储：原件默认保存在
`backend/storage/data-process/<task_id>/<file_id>/v<version>/<安全文件名>`，数据库的
`storage_object_id` 只保存 `local://data-process/...` 逻辑引用，不保存或返回宿主机
绝对路径；完整解析正文继续保存在 `data_process_source_files.content`，列表摘要使用
`content_preview`，因此 PDF、Office 等文件的预览无需反复解析原始二进制。可通过
`DATA_PROCESS_STORAGE_DIR` 指定其他本地根目录；从 `start.sh` 启动时，该变量应在
当前终端导出。历史 `db://data-process/...` 记录继续从数据库正文预览。
单独删除源文件时先提交数据库软删除，再立即删除受控目录中的原件；若物理删除
失败，接口仍按数据库结果返回成功并标记 `storage_cleanup_pending=true`，软删除记录
中的逻辑引用可供运维补偿清理。任务软删除以及修改 `process_type` 导致的源文件
软删除按留存数据处理，当前版本不自动物理清除。
### 结果与发布
| 方法 | 路径 | 说明 |
| --- | --- | --- |
| GET | `/data-process/{id}/results` | 分页查询，支持 keyword/status/split |
| PUT | `/data-process/{id}/results/{result_id}` | 保存人工编辑并重评分 |
| POST | `/data-process/{id}/results/{result_id}/restore` | 恢复生成时的原值 |
| POST | `/data-process/{id}/publish` | 幂等发布为数据集 |
## 3. 配置校验
- `process_type`：`structured | unstructured | external`。
- 数据集划分的 `train + validation + test` 必须等于 100，各项为 0～100。
- `chunk_size` 为 16～32768 token；`chunk_overlap` 必须小于
`chunk_size`；`min_chunk_size` 不得大于 `chunk_size`。
- `temperature` 为 0～2，`max_tokens` 为 1～32768。
- 任务名称在未删除任务中唯一。
- 选择 `generation_model_id` 后，启动生成时校验模型是否存在，并保存不含密钥的
模型版本快照。
- 当前运行库沿用平台现有的单租户模式，不接受客户端提交 tenant/owner/operator
字段，避免伪造隔离上下文；接入平台可信认证上下文后再启用数据库中预留的
tenant/project 字段。
## 4. 格式解析与标准化
上传格式按处理类型约束：
- 结构化数据支持 JSON、JSONL/NDJSON、CSV/TSV 和 XLSX。XLSX 能识别纵向、
横向合并单元格组成的多级表头，并稳定展平为 `销售.Q1` 一类字段；公式只读取
文件中已缓存的计算结果，不在服务端执行。
- 非结构化数据支持 UTF-8/UTF-8 BOM 的 TXT、Markdown、JSON/JSONL，以及
文本型 PDF、DOCX 和 PPTX。PDF 按页抽取文本，DOCX 抽取段落与表格，PPTX
抽取幻灯片文本与表格，随后统一进入切片算法。
- 旧版二进制 DOC、XLS、PPT 不直接解析，返回 415 并提示分别转换为
DOCX、XLSX、PPTX。
- 扫描 PDF 没有文本层时明确提示需要 OCR；当前流程不执行 OCR。加密、损坏或
超出页数/工作表/行列/解压规模限制的文件整批拒绝。
现代 Office 文件在交给解析库前检查 ZIP 成员路径、重复成员、加密标记、活动
XML、单成员大小、总解压大小和压缩比，避免路径穿越、实体扩展与 ZIP bomb。
结构化选项按固定顺序执行，关闭某项时不会隐式执行对应业务变换：
1. `detect_structure`：展平嵌套对象；XLSX 上传解析阶段识别合并单元格和多级表头。
2. `normalize_format`：字段名转 snake_case，执行 Unicode NFKC、换行和容器值规范化，
输出键顺序稳定的 canonical JSON；账号、邮编等字符串不会转成数值。
3. `clean_invalid`：删除全空列和全空记录；存在 `id/uuid/key/code/*_id` 身份字段时，
删除身份字段残缺的行，但不会因备注等可选字段为空误删有效记录。
4. `filter_anomaly`：仅对不少于 8 个样本的非身份数值字段使用 Tukey IQR 过滤离群行，
同时过滤明确乱码、不可打印或极端超长文本；小样本和 ID 字段不参与统计过滤。
5. `deduplicate`：先按整行 canonical JSON 精确去重，再按非空
`id/uuid/key/code/*_id` 字段稳定保留首条；空关键值互不视为重复。
6. `desensitize`：对结构化姓名字段和正文中的高置信上下文姓名、邮箱、手机号、
身份证号进行不可逆掩码，并分别记录命中数。
非结构化“智能预处理”由六个可独立执行的底层选项组成：
- `clean_invalid_content` 删除确定为空、不可读或纯重复符号的无效块。
- `detect_document_structure` 识别 Markdown、中文章节和数字标题，切片不跨章节，
并在预览质量详情中保存 `heading_path`。
- `merge_short_content` 在同一章节中合并短块，合并后不突破 `chunk_size`。
- `filter_low_quality` 在生成前过滤乱码、不可打印、重复或极端超长内容。
- `deduplicate_content` 先精确去重，再对足够长的内容进行保守近重复判断；数字或
否定含义变化时始终保留。
- `preserve_context` 才启用相邻切片 overlap；关闭时切片不共享正文上下文，且上下文
永不跨文件或章节。
表格、围栏代码块和连续列表保护是三个独立参数。启用时切点避开相应 Markdown
块，关闭时允许按正常长度切分。
脱敏是不可逆掩码：
- 邮箱：`[EMAIL]`
- 中国大陆手机号：`[PHONE]`
- 18 位身份证号：`[ID_CARD]`
- 高置信姓名：`[NAME]`
源文件原文与脱敏后的预览分开保存，结果不得反向覆盖源文件。
## 5. 切片算法
首阶段只提供三种切片策略：
- `structure` 先识别 Markdown、中文章节及编号标题，再由 LlamaIndex
`SentenceSplitter` 在章节内按段落和中英文句界限长；章节之间不共享 overlap。
- `fixed` 使用 LlamaIndex `TokenTextSplitter` 按目标 token 窗口切分。
- `custom` 使用用户给定分隔符，在找不到合适分隔点时回退到固定窗口。
不提供 `semantic` 和旧 `heading` 配置；创建或更新任务时传入这些值会直接拒绝。
LlamaIndex 只负责通用切分，原文 offset、行号、标题路径和 Markdown 保护块仍由
项目适配层统一维护。
首版使用可替换的确定性 token 估算器，中文字符、标点和英文词分别计数；
所有偏移以 Python/JavaScript 都能稳定表达的 Unicode 文本偏移为准。
算法必须满足：
- 每轮游标严格前进，异常分隔符不能产生死循环。
- overlap 是最大重叠量，尾部过短切片合并到上一片。
- 代码块、Markdown 表格和连续列表在启用保护时不从中间切开。
- 每个预览条目记录 `source_file_id`、字符偏移、起止行、token 数和算法版本。
## 6. 生成与质量评分
结构化记录优先识别以下字段：
1. `instruction/input/output`
2. `question/context/answer`
3. `prompt/input/response`
已有标准字段时只做标准化；需要语义生成时调用所选模型的 OpenAI 兼容接口，
并固化模型 ID、模型版本、prompt、temperature、max_tokens 和 JSON mode 快照。
模型地址可输入域名、`/v1` 基础地址或完整地址：例如输入
`www.caoxiaozhu.com` 会规范为
`https://www.caoxiaozhu.com/v1/chat/completions`，无需用户手工拼接路径。
单条失败记录为 `invalid`，有限重试耗尽后继续处理下一条，避免整批丢失。
每条结果总分为 0～100：
```text
总分 = 完整性 35% + 长度合理性 20% + 可读性 20%
+ 来源相关性 15% + 非重复性 10%
```
- instruction 或 output 为空时格式硬失败并标记 `invalid`。
- 开启短文本过滤且 output 低于 `min_output_length` 时标记过滤原因。
- 评分详情、命中规则与过滤原因必须落库并返回前端，不只返回一个总分。
## 7. 稳定划分
划分不能依赖结果插入顺序。对每条记录计算：
```text
bucket = SHA256(task_id + ":" + result_id) mod 10000
```
按万分位阈值映射为 `train/validation/test`。同一任务重试、分页或进程重启后，
同一结果仍落入相同 split。
## 8. 发布与来源链路
发布在一个数据库事务中完成：
```text
source_file
→ data_process_task
→ data_process_result
→ dataset
→ dataset_file + dataset_file_version
→ dataset_record
```
只发布 `valid/modified` 且满足质量门槛的结果。输出 JSONL 先计算 checksum，
再登记文件版本和记录。发布请求中的 split 会重新进行稳定划分。任务的
`output_dataset_id` 是幂等键；重复调用返回已有数据集，目标数据集若已被外部
删除则解除断链并重新发布。当前运行库只开放 `local` 存储类型，正文保存在
当前平台的 `dataset_files.content`，不虚假宣称已上传 MinIO 或云存储。
## 9. 安全边界
- 文件名只保留 basename，响应不返回宿主机绝对路径。
- 上传限制单文件、批次文件数与批次总大小，解析采用有界读取。
- 外部数据源凭据不写日志、不进入 localStorage、不在详情接口回显。
- 外部 PostgreSQL 只允许单条 `SELECT/WITH`、只读事务、5 秒连接超时、
30 秒语句超时和 50 MiB 响应上限；默认阻止回环、链路本地及私网地址。
可信内网部署必须显式设置 `DATA_PROCESS_ALLOW_PRIVATE_EXTERNAL_DB=true`。
- SQL 迁移独立存放，应用启动不会隐式修改当前远程数据库。
## 10. 迁移边界
`backend/app/db/sql/002_data_process.sql` 只面向当前运行脚本
`001_platform_runtime.sql` 的 TEXT/最小表模型。它会在执行前检查
`datasets.id` 类型；若检测到 `docs/postgres-schema.sql` 的 UUID/JSONB 目标模型，
会直接失败而不是进行一半成功、一半失败的危险迁移。目标模型后续应由独立
Alembic 迁移和对应存储实现承接。
`DataProcessStore.ensure_schema()` 仅供受控管理命令显式调用，API 路由和应用启动
均不会自动执行该迁移。本次开发和测试没有修改任何远程数据库。
在已加载 `DATABASE_URL` 的终端中可先只读检查：
```bash
cd backend
.venv/bin/python -m app.modules.data_process.schema_cli --check
```
确认目标主机和数据库名称无误后，才显式执行：
```bash
cd backend
.venv/bin/python -m app.modules.data_process.schema_cli --apply --yes
```
命令输出只显示主机、端口和数据库名，不显示用户名或密码。

164
docs/database-config.md
View File

@@ -1,164 +0,0 @@
# 数据库配置与初始化说明（PostgreSQL / Redis）
> 记录平台的 **PostgreSQL 账号密码**、**Redis 账号密码**、**数据库地址在代码中的配置位置**，
> 以及**切换 PG 数据集时如何执行完整初始化 SQL**。
---
## 1. 账号密码速查表
### 1.1 PostgreSQL
| 环境 | 地址 | 用户 | 密码 | 数据库 | 来源 |
|------|------|------|------|--------|------|
| 代码默认值 | `localhost:15432` | `yg_ft` | `change_me` | `yg_ft` | `config.py` / `session.py` 的 `DATABASE_URL` 兜底 |
| Docker 部署 | `www.caoxiaozhu.com:5432` | `root` | `8811614287327Leo` | `yg_ft` | `docker/app/.env` 的 `DATABASE_URL` |
| Docker 内置 Postgres（已注释） | `localhost:15432` | `root` | `8811614287327Leo` | `yg_ft` | `docker/app/docker-compose.yml` 注释掉的 postgres 服务 |
> ⚠️ `change_me` 与 `8811614287327Leo` 均为默认/示例凭据，生产环境务必更换。
### 1.2 Redis
| 项 | 值 | 说明 |
|----|-----|------|
| 连接串 | `redis://:<REDIS_PASSWORD>@redis:6379/0` | 已内嵌密码；docker 网络内服务名 `redis`，端口 6379，db 0 |
| 对外端口 | `16379`（`REDIS_PORT`） | 宿主机映射 |
| 密码 | `docker/app/.env` 的 `REDIS_PASSWORD` | 已启用 `requirepass` 鉴权 |
| 镜像 | `redis:7-alpine` | 已开启 AOF（`--appendonly yes`）+ `requirepass` |
> **当前后端代码未使用 Redis**：`redis` 包已列入 `requirements.txt`，`REDIS_URL` 通过
> docker-compose 注入容器，但全仓库 `backend/`、`compute/` 没有任何 `import redis` /
> `Redis(...)` 连接代码。Redis 为后续功能预留；**即便如此仍已配置鉴权**，
> 避免无密码实例对外暴露（纵深防御）。未来启用时按 `REDIS_URL` 连接即可。
>
> 健康检查通过容器环境变量 `REDISCLI_AUTH` 认证，不在进程参数中泄露密码。
---
## 2. 数据库地址在代码中的配置位置
### 2.1 后端（backend）
| 文件 | 作用 | 取值 |
|------|------|------|
| `backend/app/core/config.py` | **唯一权威配置**，`Settings.database_url` | `os.getenv("DATABASE_URL", "postgresql+psycopg://yg_ft:change_me@localhost:15432/yg_ft")` |
| `backend/app/db/session.py` | SQLAlchemy 引擎（`get_db` / `session_scope`） | `os.getenv("DATABASE_URL", ...)` 同样兜底 |
| `backend/app/db/platform_store.py` | **平台主存储**，直接用 psycopg 连接池 | 取 `settings.database_url`，`_psycopg_url()` 把 `postgresql+psycopg://` 转成 `postgresql://` |
| `backend/app/modules/data_process/store.py` | 数据处理存储 | `get_settings().database_url` |
> **环境变量加载顺序**：`config.py` 导入时会 `load_dotenv(backend/.env, override=True)`，
> 即 **`backend/.env` 会覆盖系统环境变量**；docker 部署则直接由 compose 注入 `DATABASE_URL`。
> 最终优先级：`backend/.env` / compose 注入的环境变量 > 代码内默认值。
### 2.2 部署配置（docker）
| 文件 | 关键项 |
|------|--------|
| `docker/app/.env` | `DATABASE_URL`、`POSTGRES_USER`、`POSTGRES_PASSWORD`、`REDIS_URL`、`REDIS_PORT` |
| `docker/app/docker-compose.yml` | `backend-api` 环境透传上述变量；`redis` 服务定义 |
```ini
# docker/app/.env（节选）
DATABASE_URL=postgresql+psycopg://root:8811614287327Leo@www.caoxiaozhu.com:5432/yg_ft
POSTGRES_USER=root
POSTGRES_PASSWORD=8811614287327Leo
REDIS_PASSWORD=<强密码> # Redis requirepass（新增鉴权）
REDIS_URL=redis://:<REDIS_PASSWORD>@redis:6379/0 # 连接串内嵌密码
REDIS_PORT=16379
USE_BUILTIN_POSTGRES=false # 当前用共享外部库，内置 postgres 服务被注释
USE_BUILTIN_REDIS=true # Redis 用内置服务
```
---
## 3. 完整初始化 SQL
### 3.1 脚本位置
| 脚本 | 用途 |
|------|------|
| **`backend/app/db/sql/000_full_init.sql`** | **一键初始化脚本（新增）**：建库表 + 索引 + 种子数据，幂等，覆盖全部 39 张运行表 |
| `backend/app/db/sql/001_platform_runtime.sql` | 平台核心表（应用启动自动执行） |
| `backend/app/db/sql/002_governance.sql` | 治理表（应用启动自动执行） |
| `backend/app/db/sql/003_tenant_quota.sql` | 租户配额列（应用启动自动执行） |
| `backend/app/db/sql/003_model_path_governance.sql` | 模型可训练列（**应用不自动执行**，已并入完整脚本） |
| `backend/app/db/sql/002_data_process.sql` | 数据处理表（**应用不自动执行**，已并入完整脚本） |
| `docs/postgres-schema.sql` | ⚠️ **目标设计稿**（UUID/JSONB），与运行时代码不兼容，**不要用于初始化** |
> **重要**：`docs/postgres-schema.sql` 是规划中的“目标 schema”（UUID 主键、`ft_platform` schema 等），
> 运行时代码明确拒绝该结构（`002_data_process.sql` 检测到 `datasets.id` 非 TEXT 会直接报错）。
> 初始化请使用 **`000_full_init.sql`**。
### 3.2 执行步骤（全新 PG 环境）
**第 1 步：创建角色与数据库**（必须单独执行，不能放进事务）
```sql
-- 以超级用户（如 postgres）连接：
CREATE ROLE yg_ft LOGIN PASSWORD '请改为强密码';
CREATE DATABASE yg_ft OWNER yg_ft;
-- 如需应用执行 CREATE EXTENSION 等，可再授予超级用户（按需）：
-- ALTER ROLE yg_ft SUPERUSER;
```
**第 2 步：执行完整初始化脚本**
```bash
psql "postgresql://yg_ft:密码@<host>:5432/yg_ft" \
-f backend/app/db/sql/000_full_init.sql
```
脚本特点：
- 全程一个事务（`BEGIN; ... COMMIT;`），失败自动回滚
- 所有 DDL 使用 `IF NOT EXISTS` / `ADD COLUMN IF NOT EXISTS`，**可重复执行**
- 含 `DO $$...$$` 语句块，必须用 `psql` 执行（应用内部的按分号切分 `executescript()` 不适用）
- 自动写入种子用户：`admin / admin123`、`operator / operator123`（登录后请改密）
**第 3 步：校验**
```sql
SELECT count(*) FROM pg_tables WHERE schemaname = 'public'; -- 应 ≥ 39
SELECT username, role, status FROM users; -- 应有 admin / operator
```
### 3.3 执行方式对比（三种途径）
| 方式 | 覆盖范围 | 命令 |
|------|----------|------|
| **A. 完整脚本（推荐，切换新库）** | 全部 39 表 + 索引 + 种子 | `psql ... -f 000_full_init.sql` |
| B. 应用自动初始化 | 001 + 002_governance + 003_tenant_quota + 种子用户；**不含**数据处理表、`models.can_train`、`data_convert_tasks` | 应用首次调用 `get_platform_store()` 时 `ensure_schema()` 自动执行 |
| C. 数据处理表单独安装 | `002_data_process.sql` 全部内容 | 在 `backend/` 目录下：`python -m app.modules.data_process.schema_cli --apply --yes`（或 `--check` 只读检查） |
> **缺口说明**：
> - `models.can_train`（训练预检用）只在 `003_model_path_governance.sql` 中创建，应用启动**不会**自动执行；
> - `data_convert_tasks`（数据转换任务表）运行时代码引用但**原 SQL 脚本缺失**；
> 已统一并入 `000_full_init.sql` 补齐。若现有库缺这两项，执行一次完整脚本即可幂等补上。
### 3.4 完整脚本包含的表（39 张）
**核心**：users、models、trained_models、model_lineage、model_artifacts、model_export_jobs、
datasets、dataset_files、compute_nodes、gpus、fine_tune_tasks、fine_tune_metrics、
fine_tune_checkpoints、compute_jobs、gpu_allocations、scheduler_locks、resource_replicas、
resource_sync_jobs、eval_tasks、eval_dimensions、compare_tasks、projects、project_members、
roles、sessions、acls
**治理**：tenants、approval_templates、approval_instances、approval_steps、audit_logs、retention_policies
**数据处理**：data_process_tasks、data_process_source_files、data_process_preview_items、
data_process_results、dataset_file_versions、dataset_records
**数据转换**：data_convert_tasks（新增补齐）
---
## 4. 安全注意事项
1. **更换默认密码**：`change_me`（代码兜底）、`8811614287327Leo`（部署）、`admin123`/`operator123`（种子用户）、`REDIS_PASSWORD` 上线前必须更换。
2. **Redis 已加鉴权**：已配置 `requirepass` + `REDISCLI_AUTH` 健康检查；`REDIS_URL` 内嵌密码。若端口需暴露公网，仍建议用防火墙/安全组限制来源。
3. **`docker/*/.env` 已入库，含明文凭据**：
- `backend/.env` 已被 `.gitignore` 排除；
- 但 `docker/app/.env`、`docker/compute/.env` 目前被 git 跟踪（`git ls-files` 可见），
其中的 `DATABASE_URL`、`POSTGRES_PASSWORD`、`COMPUTE_SERVICE_TOKEN` 等均为明文。
- **建议**：轮换这些凭据，将 `docker/*/.env` 移出版本库（`git rm --cached`）并改用
部署侧机密注入（如 docker secrets / CI 变量 / 环境变量模板），保留 `.env.example` 作为模板。
4. **最小权限**：应用角色只需对业务库的 DML/DDL 权限，尽量避免 SUPERUSER。

29
docs/deployment-plan.md
View File

@@ -228,20 +228,6 @@ GPU 算力服务器部署：
多节点任务调度由应用平台统一完成。应用平台从 `compute_nodes` 读取节点地址、权重、标签、启用状态、维护状态和健康检查结果；从 `resource_replicas` 判断目标节点是否已有所需数据集/模型副本；缺失时创建 `resource_sync_jobs`，通过目标节点 File Gateway 同步资源。
当前实现已支持在 `/compute` 算力节点页面新增和编辑节点。运维人员维护 `Compute API` 地址、`File Gateway` 地址、权重、标签、启用状态、最大并发和本地路径后，点击连接测试会由应用后端主动访问目标节点健康检查和 GPU 清单接口，并将 `health_detail`、`gpu_count`、`gpu_devices/gpus` 同步到 PostgreSQL。真实 GPU 服务器优先通过 `nvidia-smi` 发现 GPU；特殊环境可用 `COMPUTE_GPU_COUNT` 等环境变量声明兼容清单。
训练运行闭环：
- 前端启动训练后，Backend API 按 `compute_nodes` 的启用状态、调度状态、权重和并行任务数选择节点。
- Backend API 向目标节点 `POST /modelTF/compute/jobs` 提交 LLaMA-Factory 训练作业，并在 `fine_tune_tasks.compute_job_id` 记录算力任务 ID。
- Compute API 在真实模式下启动 `llamafactory-cli train` 子进程，训练日志写入 `TRAINING_LOG_ROOT/{job_id}.log`。
- Backend API 启动后会运行应用侧轮询 worker，按 `COMPUTE_POLL_INTERVAL_SECONDS` 主动查询目标节点 `GET /modelTF/compute/jobs/{id}`，同步任务状态、进度、PID、输出目录、日志路径和产物索引。
- 停止训练时，Backend API 优先调用目标节点 `POST /modelTF/compute/jobs/{id}/stop`，再回写应用任务状态。
- 失败或停止任务可以通过 `POST /modelTF/compute/jobs/{id}/retry` 重试；重试会清空旧运行态，重新调度节点并创建 Compute Job。
- 训练日志通过 `GET /modelTF/compute/jobs/{id}/logs` 读取，支持 `tail_lines`、`offset`、`limit`，用于训练详情页、训练日志页和日志平台采集。
- Compute API 使用 `COMPUTE_SERVICE_TOKEN` 做服务间鉴权，应用侧请求携带 `X-Compute-Token`；健康检查接口保持可公开探活。
- Compute API 会把本机训练作业登记到 `TRAINING_LOG_ROOT/compute-jobs.json`，服务重启后可恢复任务索引并继续暴露状态和日志。
调度策略：
- 默认自动调度，按节点健康、标签、GPU 空闲、队列长度、节点权重和资源副本命中率排序。
@@ -293,7 +279,7 @@ COMPUTE_API_BASE_URL=https://compute.internal:19100
COMPUTE_SERVICE_TOKEN=***
FILE_GATEWAY_BASE_URL=https://compute.internal:19101
COMPUTE_STATUS_SYNC_MODE=polling
COMPUTE_POLL_INTERVAL_SECONDS=3
COMPUTE_POLL_INTERVAL_SECONDS=10
COMPUTE_POLL_BATCH_SIZE=100
```
@@ -304,21 +290,10 @@ COMPUTE_ENV=prod
COMPUTE_HOST_ID=gpu-node-01
COMPUTE_API_PORT=19100
FILE_GATEWAY_PORT=19101
COMPUTE_AUTH_ENABLED=true
COMPUTE_SERVICE_TOKEN=***
ENABLE_APP_CALLBACK=false
LLAMA_FACTORY_HOME=/app/LLaMA-Factory
YG_FT_DATA_ROOT=/data/yg-ft
YG_FT_DATA_ROOT_HOST=./data/yg-ft
YG_FT_MODEL_ROOT=/data/yg-ft/models
YG_FT_MODEL_ROOT_HOST=./data/yg-ft/models
YG_FT_DATASET_ROOT=/data/yg-ft/datasets
YG_FT_DATASET_ROOT_HOST=./data/yg-ft/datasets
YG_FT_OUTPUT_ROOT=/data/yg-ft/outputs
YG_FT_OUTPUT_ROOT_HOST=./data/yg-ft/outputs
TRAINING_LOG_ROOT=/opt/yg-ft/logs/training
TRAINING_LOG_ROOT_HOST=./data/yg-ft/logs/training
COMPUTE_LOG_ROOT_HOST=./data/yg-ft/logs/compute
LOG_DIR=/opt/yg-ft/logs/compute
CUDA_VISIBLE_DEVICES=0,1,2,3
```
@@ -418,7 +393,7 @@ gpu-node-03 -> http://10.10.20.33:19100 / http://10.10.20.33:19101
```env
ENABLE_APP_CALLBACK=false
COMPUTE_SERVICE_TOKEN=change_me
YG_FT_DATA_ROOT_HOST=./data/yg-ft
YG_FT_DATA_ROOT_HOST=/data/yg-ft
```
## 12. 仍需确认的问题

411
docs/governance-user-guide.md
View File

@@ -1,411 +0,0 @@
# 平台治理功能使用指南
> 版本：v1.0
> 日期：2026-08-10
> 适用版本：YG Fine-Tune Platform v1.0+
---
## 目录
1. [快速入门](#1-快速入门)
2. [用户与权限管理](#2-用户与权限管理)
3. [GPU 算力分配](#3-gpu-算力分配)
4. [资源可见性与隔离](#4-资源可见性与隔离)
5. [审批流程管理](#5-审批流程管理)
6. [审计日志查询](#6-审计日志查询)
7. [常见问题与排查](#7-常见问题与排查)
---
## 1. 快速入门
### 1.1 平台治理是什么？
平台治理是一套**多租户、多角色、细粒度权限控制**体系，用于在多人协作使用 AI 微调平台时，确保：
- 每个用户只能看到和操作自己有权限的资源
- GPU 算力按需分配，避免资源争抢
- 高风险操作（删除、停止任务）有审批记录可追溯
- 所有操作都有审计日志
### 1.2 三种内置角色
| 角色 | 能做什么 | 不能做什么 |
|---|---|---|
| **超级管理员 (admin)** | 全部操作；管理用户、分配 GPU、审批、查看全部资源 | — |
| **操作员 (operator)** | 创建数据集/模型、训练/评测/推理任务 | 管理用户、分配 GPU、修改他人权限 |
| **观察员 (viewer)** | 查看被授权的资源 | 创建或修改任何资源 |
### 1.3 入口在哪里？
所有治理功能集中在左侧导航栏的 **「系统设置」** 分组下：
```
系统设置
├── 用户设置 ← 用户 CRUD + 角色权限 + 密码管理
├── 租户管理 ← 组织/团队（可选）
├── 项目空间 ← 项目级资源隔离（可选）
├── 审批模板 ← 定义哪些操作需要审批
├── 审批中心 ← 处理待审批请求
└── 审计日志 ← 查看所有操作记录
```
> ⚠️ 以上菜单**只有 admin 用户能看到**。普通用户登录后不会出现这些入口。
---
## 2. 用户与权限管理
### 2.1 创建用户
**路径**：`用户设置` → `创建用户`
1. 以 admin 身份登录平台
2. 进入「用户设置」页面
3. 点击右上角「创建用户」按钮
4. 填写信息：
- **账号**：登录用户名（如 `zhangsan`）
- **显示名称**：如 `张三`
- **密码**：初始密码（默认 `Platform@123`）
- **角色**：选择 `admin` / `operator` / `viewer`
5. 点击保存
创建后用户可以立即用该账号登录。
### 2.2 管理用户权限
**路径**：`用户设置` → 用户列表 → 操作列「页面权限」
#### 给普通用户分配业务模块权限
点击某用户的「页面权限」按钮，弹出对话框：
```
为 张三 分配可访问的页面模块：
☑ 服务看板 ☑ 模型训练 ☑ 模型评测
☑ 模型推理 ☑ 模型管理 ☑ 数据集管理
☐ 数据处理 ☐ 数据类型转换 ☐ 算力节点 ← 不勾选则不可见
☑ 平台性能 ☑ 查看日志
[取消] [保存]
```
勾选需要的模块，点「保存」即可。
> **注意**：
> - 「用户与权限」这个选项**只有 admin 能看到**，其他用户即使被赋权也不会显示
> - admin 用户的权限**不可更改**，始终是全选状态且只读
#### 限制说明
| 权限码 | 说明 | 谁能拥有 |
|---|---|---|
| `user-settings` | 用户设置、租户管理、项目空间、审批、审计日志 | **仅 admin** |
| `compute` | 算力节点、GPU 分配 | **仅 admin** |
| 其他业务权限 | 训练、评测、推理、模型、数据集等 | admin 可分配给任何人 |
### 2.3 重置用户密码
**两种方式**：
**方式一：管理员重置**
1. 在用户列表中找到目标用户
2. 点击「重置密码」
3. 输入新密码，确认
**方式二：用户自行修改**
1. 用户登录后在「用户设置」页面点击「修改密码」按钮
2. 输入旧密码 + 新密码（至少 6 位）
3. 确认修改
### 2.4 删除用户
**路径**：`用户设置` → 用户列表 → 操作列「删除」
> ⚠️ 删除用户时会**级联清理**其所有关联数据：
> - 该用户创建的数据集、基座模型、微调产物、评测任务
> - 该用户的 ACL 授权记录、GPU 分配记录
> - 该用户的审批实例、审计日志、项目成员关系
> - **训练任务保留不删**（避免算力节点上的物理任务数据不一致）
---
## 3. GPU 算力分配
### 3.1 为什么需要 GPU 分配？
当服务器有多张 GPU 卡（如 8×A800）时，需要指定**哪个用户能用哪张卡**：
- 避免两个人同时选同一张卡导致训练冲突
- 按团队/项目隔离算力资源
- 控制每个用户的 GPU 配额
### 3.2 分配 GPU（仅 admin）
**路径**：`算力节点` → `GPU 分配` 标签页
1. 以 admin 登录，进入「算力节点」页面
2. 点击顶部的 **「GPU 分配」** 标签（只有 admin 可见）
3. 点击 **「分配 GPU」** 按钮
4. 填写：
- **算力节点**：选择节点（如 `gpu-node-01`）
- **GPU 序号**：卡号（0, 1, 2, ... 7）
- **用户**：选择要分配给谁
5. 点「确认分配」
示例：把节点 `gpu-node-01` 的第 0、1 号卡分配给用户 `zhangsan`：
```
┌─────────────────────────────────────┐
│ 分配 GPU [×] │
├─────────────────────────────────────┤
│ 算力节点: [gpu-node-01 ▼] │
│ GPU 序号: [0 ▲] │
│ 用户: [zhangsan ▼] │
│ │
│ [取消] [确认分配] │
└─────────────────────────────────────┘
```
分配后的效果：
| 用户 | 可用 GPU |
|---|---|
| admin | 全部 GPU（不需要显式分配） |
| zhangsan | gpu-node-01 的 0、1 号卡 |
| lisi | （未分配，不可用） |
### 3.3 撤销分配
在 GPU 分配列表中，每条记录右侧有「撤销」按钮，点击后确认即可移除该分配。
### 3.4 用户视角：创建训练任务时的 GPU 选择
- **admin**：下拉列表显示全部可用 GPU
- **被分配了 GPU 的用户**：只显示被分配给自己的卡
- **未分配任何 GPU 的用户**：显示提示「未分配 GPU，请联系管理员」，无法提交训练任务
---
## 4. 资源可见性与隔离
### 4.1 自动生效的隔离规则
无需手动配置，以下规则自动生效：
| 资源类型 | admin 看到 | 普通用户看到 |
|---|---|---|
| **基座模型**（容器内注册的本地模型） | 全部 | **全部**（共享资源，有 model-manage 权限即可见） |
| **数据集** | 全部 | **自己创建的** + 被 ACL 授权的 |
| **微调产物**（训练输出的模型） | 全部 | **自己训练的** + 被 ACL 授权的 |
| **评测任务** | 全部 | **自己创建的** + 被 ACL 授权的 |
| **推理/对比任务** | 全部 | **自己创建的** + 被 ACL 授权的 |
### 4.2 实际场景示例
假设有三个用户：**admin**、**zhangsan**（算法工程师）、**lisi**（标注员）
```
zhangsan 上传了数据集 ds_alpaca、ds_sharegpt
zhangsan 训练出了模型 ft_qwen_001
lisi 上传了数据集 ds_label
admin 注册了基座模型 Qwen3-1.7B
```
各用户看到的资源：
| 用户 | 数据集 | 基座模型 | 微调产物 |
|---|---|---|---|
| **admin** | ds_alpaca, ds_sharegpt, ds_label (3个) | Qwen3-1.7B | ft_qwen_001 |
| **zhangsan** | ds_alpaca, ds_sharegpt (2个) | Qwen3-1.7B | ft_qwen_001 |
| **lisi** | ds_label (1个) | Qwen3-1.7B | (无) |
### 4.3 ACL 资源授权（高级用法）
如果 zhangsan 想让 lisi 也能看到自己的数据集 `ds_alpaca`：
> 此功能需要在资源详情页提供「资源授权」按钮（前端已预留接口），当前可通过 API 直接操作：
```bash
# 授予 lisi 对 ds_alpaca 的读权限
curl -X PUT /modelTF/resources/dataset/ds_alpaca_id/acl \
-H "Authorization: Bearer platform-token-admin" \
-d '{
"acls": [
{"principal_type": "user", "principal_id": "lisi_id", "permission": "read"}
]
}'
```
---
## 5. 审批流程管理
### 5.1 哪些操作会触发审批？
| 操作 | 触发条件 | 处理方式 |
|---|---|---|
| 删除他人的数据集 | 非 admin 删除别人创建的数据集 | 创建审批实例 或 admin 直接执行 |
| 删除他人的模型 | 非 admin 删除别人创建的模型 | 同上 |
| 停止他人的训练任务 | 非 admin 停止别人发起的任务 | 同上 |
| 归档/删除项目空间 | 存在待审批变更时 | 拒绝执行 |
**核心规则**：admin 做任何操作都直接执行（旁路）；普通用户操作他人资源时进入审批流程。
### 5.2 审批流程示意
```
普通用户 lisi 尝试删除 zhangsan 的数据集
│
▼
┌───────────────────────┐
│ 后端检查：是 admin 吗？ │
└──────┬────────────────┘
│ 否
▼
┌───────────────────────┐
│ 创建审批实例 │
│ status = pending │
│ 返回 202（待审批） │
└──────────┬────────────┘
│
▼
┌───────────────────────┐
│ admin 在「审批中心」看到 │
│ 这条待审批请求 │
│ 点击「通过」或「拒绝」 │
└──────────┬────────────┘
│
┌─────┴─────┐
│ │
通过 拒绝
│ │
▼ ▼
执行删除 不执行
+审计日志 +审计日志
```
### 5.3 管理审批
**路径**：`系统设置` → `审批中心`
1. 查看待审批列表（status=pending）
2. 点击某条记录查看详情
3. 决策：「通过」或「拒绝」
4. 决策结果自动执行对应操作并记录审计日志
**审批模板**（`系统设置` → `审批模板`）：定义每种操作需要几步审批、每步谁来审。默认模板都是单步（admin 审批即可）。
---
## 6. 审计日志查询
### 6.1 什么是审计日志？
平台上所有**写操作**和**敏感操作**都会自动记录审计日志，包括：
- 用户创建/删除/修改
- 权限变更
- GPU 分配/撤销
- 资源上传/删除
- 训练任务启动/停止
- 审批决策
### 6.2 查询审计日志
**路径**：`系统设置` → `审计日志`
支持筛选条件：
| 筛选项 | 说明 |
|---|---|
| 操作人 | 按用户 ID 过滤 |
| 动作类型 | 如 `user.create`, `dataset.delete`, `gpu.assign` 等 |
| 目标资源类型 | dataset / model / fine_tune_task 等 |
| 时间范围 | 开始时间 ~ 结束时间 |
### 6.3 导出审计日志
审计日志页面底部有「导出 CSV」按钮，导出的文件包含当前筛选条件下的全部记录，可用于合规审计或问题追溯。
### 6.4 日志保留策略
审计日志受**留存策略**控制（`系统设置` → 租户管理 → 绑定留存策略）。默认保留 30 天，超期自动清理。
---
## 7. 常见问题与排查
### Q1: 普通用户看不到某个菜单？
检查两件事：
1. 该用户是否有对应的**权限码**（admin 在「用户设置」→「页面权限」中分配）
2. 该菜单是否属于 **admin 专属**（如「用户设置」「算力节点」——这些对非 admin 永远不可见）
### Q2: 用户创建训练任务时报错"无权使用所选 GPU"
说明该用户没有被分配所选择的 GPU 卡。解决方法：
1. admin 进入「算力节点」→「GPU 分配」标签页
2. 为该用户分配对应的 GPU
3. 用户刷新页面重新选择 GPU
### Q3: 删除用户后看板还显示残留数据？
正常情况下 `delete_user` 会级联清理关联数据。如果仍有残留：
- **训练任务**：设计上保留不删（避免算力节点物理数据不一致），这是预期行为
- **登录时长排行**：可能来自旧的 session 记录，不影响功能，新登录后会更新
### Q4: 审批实例一直 pending 没人处理？
审批实例需要 admin 在「审批中心」手动处理。如果长时间无人处理：
- 可以在数据库中直接将 `approval_instances.status` 改为 `rejected`
- 或者由 admin 直接以自身身份执行该操作（admin 有旁路权限）
### Q5: 如何查看当前所有 GPU 分配情况？
```bash
# admin 调用接口
curl -H "Authorization: Bearer platform-token-admin" \
/modelTF/compute/gpu-assignments
```
返回格式：
```json
{
"data": [
{
"id": "ga_xxx",
"node_name": "A800 训练节点",
"gpu_index": 0,
"display_name": "张三",
"assigned_at": "2026-08-10T10:00:00"
}
]
}
```
### Q6: 用户忘记密码怎么办？
两种方案：
1. **admin 重置**：在「用户设置」→ 用户列表 →「重置密码」
2. **用户自助修改**：用户登录后点击「修改密码」（需知道旧密码）
如果是完全忘记且不是 admin，只能由 admin 重置。
---
## 附录：API 快速参考
| 功能 | 方法 | 路径 | 鉴权 |
|---|---|---|---|
| 查看我的 GPU | GET | `/compute/my-gpus` | 登录用户 |
| 查看 GPU 分配 | GET | `/compute/gpu-assignments` | admin |
| 分配 GPU | POST | `/compute/gpu-assignments` | admin |
| 撤销 GPU 分配 | DELETE | `/compute/gpu-assignments/{id}` | admin |
| 修改自己的密码 | POST | `/users/me/password` | 登录用户 |
| 查看审计日志 | GET | `/system/audit-logs` | admin |
| 导出审计日志 | GET | `/system/audit-logs/export` | admin |
| 查看审批列表 | GET | `/approvals` | 登录用户 |
| 审批决策 | POST | `/approvals/:id/steps/:idx/decision` | 审批人 |

805
docs/permissions-design.md
View File

@@ -1,805 +0,0 @@
# 平台权限设计文档
> 版本：v1.0
> 日期：2026-08-02
> 状态：设计基线，供后端实现和前端联调参照
---
## 目录
1. [设计目标](#1-设计目标)
2. [整体架构](#2-整体架构)
3. [角色体系](#3-角色体系)
4. [页面权限码](#4-页面权限码)
5. [资源所有权与可见性](#5-资源所有权与可见性)
6. [资源级 ACL（访问控制列表）](#6-资源级-acl访问控制列表)
7. [GPU 算力分配与隔离](#7-gpu-算力分配与隔离)
8. [审批拦截机制](#8-审批拦截机制)
9. [审计日志](#9-审计日志)
10. [接口鉴权流程](#10-接口鉴权流程)
11. [数据库表结构](#11-数据库表结构)
12. [API 接口清单](#12-api-接口清单)
13. [前端权限控制](#13-前端权限控制)
14. [安全设计补充](#14-安全设计补充)
15. [实施计划](#15-实施计划)
---
## 1. 设计目标
| 目标 | 说明 |
|---|---|
| **数据隔离** | 用户自己创建的数据集、模型、训练任务默认只有自己可见可操作；管理员可见全部 |
| **权限分层** | 页面级（菜单/路由可见性）+ 资源级（单条数据的读/写/删）两层控制 |
| **GPU 管控** | 多卡服务器上，管理员可指定哪些用户能使用哪些 GPU 卡 |
| **审批拦截** | 删除他人资源、停止他人任务、发布模型等高风险操作需审批或管理员旁路 |
| **审计可追溯** | 所有写操作和敏感操作产生审计日志，可按用户、动作、资源、时间筛选 |
| **权限最小变更** | 只有管理员可修改用户角色和权限码；普通用户无法提权 |
---
## 2. 整体架构
```
┌──────────────────────────────────────────────────────┐
│ 前端（Vue3） │
│ ┌────────────┐ ┌────────────┐ ┌───────────────┐ │
│ │ 路由守卫 │ │ 侧边栏过滤 │ │ 页面内按钮控制 │ │
│ │ (permission)│ │ (permission)│ │ (ACL/owner) │ │
│ └──────┬─────┘ └──────┬─────┘ └───────┬───────┘ │
│ └───────────────┴─────────────────┘ │
│ │ HTTP (Bearer token) │
└─────────────────────────┼────────────────────────────┘
│
┌─────────────────────────┼────────────────────────────┐
│ 后端（FastAPI） │
│ ┌──────────────┐ ┌────┴───────┐ ┌──────────────┐ │
│ │ get_current │ │ 资源可见性 │ │ GPU 分配校验 │ │
│ │ _user (鉴权) │ │ 过滤器 │ │ │ │
│ └──────┬───────┘ └────┬───────┘ └──────┬───────┘ │
│ │ │ │ │
│ ┌──────┴───────────────┴──────────────────┘ │
│ │ PlatformStore │ │
│ │ users | acls | roles | gpu_assignments | │ │
│ │ datasets | models | fine_tune_tasks | ... │ │
│ └───────────────────────────────────────────────────│ │
│ │ audit_logs (审计日志) │ │
│ └───────────────────────────────────────────────────│ │
└─────────────────────────────────────────────────────────┘
```
**鉴权链路**：
1. 请求到达 → `get_current_user` 从 `Authorization: Bearer platform-token-{user_id}` 解析当前用户
2. 页面级权限 → 检查 `user.permissions` 是否包含路由对应的权限码
3. 资源级权限 → 检查资源的 `created_by` 字段（所有权）或 `acls` 表（ACL 授权）
4. GPU 权限 → 检查 `gpu_assignments` 表确认用户是否被分配了请求的 GPU
---
## 3. 角色体系
### 3.1 内置角色
| 角色 code | 中文名 | 说明 |
|---|---|---|
| `admin` | 超级管理员 | 拥有全部权限码；可见全部资源；可管理用户和 GPU 分配 |
| `operator` | 操作员 | 可创建/操作自己的数据集、模型、训练任务；不可管理用户 |
| `viewer` | 观察员 | 只读权限；可查看被授权的资源；不可创建或修改 |
| `guest` | 访客 | 仅登录和看板；无业务操作权限（扩展预留） |
### 3.2 角色与权限码映射
| 权限码 | admin | operator | viewer |
|---|---|---|---|
| `dashboard` | ✅ | ✅ | ✅ |
| `fine-tune` | ✅ | ✅ | — |
| `model-eval` | ✅ | ✅ | — |
| `model-inference` | ✅ | ✅ | — |
| `model-manage` | ✅ | ✅ | — |
| `dataset` | ✅ | ✅ | — |
| `data-process` | ✅ | ✅ | — |
| `data-convert` | ✅ | ✅ | — |
| `compute` | ✅ | ✅ | — |
| `hardware` | ✅ | ✅ | ✅ |
| `logs` | ✅ | ✅ | ✅ |
| `user-settings` | ✅ | — | — |
### 3.3 权限修改规则
- **只有 admin 角色的用户**可以修改其他用户的角色和权限码
- admin 用户的 `protected=True` 标记，防止被删除或降级
- 权限修改操作产生审计日志：`action=user.permission.update`
- 用户可以查看自己的权限，不能修改自己的权限
---
## 4. 页面权限码
| 权限码 | 对应路由 | 功能 |
|---|---|---|
| `dashboard` | `/dashboard` | 服务看板 |
| `fine-tune` | `/fine-tune`, `/fine-tune/create`, `/training-log/:id` | 模型训练 |
| `model-eval` | `/model-eval`, `/model-eval/create`, `/model-eval/:id` | 模型评测 |
| `model-inference` | `/model-inference`, `/model-inference/create`, `/model-inference/chat/:id` | 模型推理 |
| `model-manage` | `/model-manage`, `/model-manage/create`, `/model-manage/:id/edit`, `/model-manage/merge` | 模型管理 |
| `dataset` | `/dataset`, `/dataset/create`, `/dataset/:id/preview` | 数据集管理 |
| `data-process` | `/data-process`, `/data-process/create`, `/data-process/:id` | 数据处理 |
| `data-convert` | `/data-convert`, `/tools` | 数据转换与工具 |
| `compute` | `/compute` | 算力节点 |
| `hardware` | `/hardware` | 平台性能 |
| `logs` | `/logs`, `/training-log/:id` | 查看日志 |
| `user-settings` | `/user-settings`, `/tenants`, `/projects`, `/approvals`, `/audit-logs` | 系统设置与平台治理 |
---
## 5. 资源所有权与可见性
### 5.1 所有权模型
每个用户可创建的资源都携带 `created_by`（或 `owner_id`）字段，标识资源所有者。
| 资源类型 | 表 | 所有者字段 | 说明 |
|---|---|---|---|
| 数据集 | `datasets` | `created_by` | 用户上传/创建的数据集 |
| 基座模型 | `models` | `created_by` | 登记的本地/API 模型 |
| 训练产物 | `trained_models` | `created_by` | 微调产出的模型 |
| 训练任务 | `fine_tune_tasks` | `payload.created_by` | 微调任务 |
| 评测任务 | `eval_tasks` | `created_by` | 评测任务 |
| 推理任务 | `compare_tasks` (payload) | `created_by` | 推理/对比任务 |
| 数据处理任务 | `data_process_tasks` | `created_by` | 数据处理任务 |
| 数据转换任务 | `data_convert_jobs` | `created_by` | 数据转换任务 |
### 5.2 可见性规则
```
资源列表查询过滤逻辑：
if user.role == "admin":
返回全部资源
elif resource.created_by == user.id:
返回（资源所有者可见自己的资源）
elif acl 中存在 (principal_type="user", principal_id=user.id, permission 包含 "read"):
返回（被 ACL 显式授权的资源）
elif acl 中存在 (principal_type="role", principal_id=user.role, permission 包含 "read"):
返回（被角色级 ACL 授权的资源）
else:
不可见
```
### 5.3 所有权操作矩阵
| 操作 | admin | 资源所有者 | 其他被授权用户 | 其他用户 |
|---|---|---|---|---|
| 查看资源 | ✅ 全部 | ✅ 自己的 | ✅ ACL 授权范围内 | ❌ |
| 编辑资源 | ✅ | ✅ 自己的 | ✅ ACL 含 write 时 | ❌ |
| 删除资源 | ✅ | ✅ 自己的（需审批） | ❌ | ❌ |
| 分享/授权 | ✅ | ✅ 自己的 | ❌ | ❌ |
| 使用资源（训练/推理/评测） | ✅ | ✅ 自己的 | ✅ ACL 含 execute 时 | ❌ |
### 5.4 数据集可见性示例
```
用户 A 创建了数据集 ds_A1 → 只有 A 和 admin 可见
用户 A 通过 ACL 把 ds_A1 的 read 权限授给用户 B → B 也可见
用户 A 通过 ACL 把 ds_A1 的 write 权限授给 operator 角色 → 所有 operator 可编辑
管理员可在任何数据集上设置 ACL
```
---
## 6. 资源级 ACL（访问控制列表）
### 6.1 ACL 表结构
```sql
CREATE TABLE IF NOT EXISTS acls (
id TEXT PRIMARY KEY,
resource_type TEXT NOT NULL, -- 资源类型: dataset / model / trained_model / fine_tune_task / ...
resource_id TEXT NOT NULL, -- 资源 ID
principal_type TEXT NOT NULL, -- 授权主体类型: user / role
principal_id TEXT NOT NULL, -- 授权主体 ID: user_id 或 role name
permission TEXT NOT NULL, -- 权限: read / write / execute / download / delete / admin
create_time TEXT
);
```
### 6.2 权限粒度
| 权限值 | 含义 | 覆盖关系 |
|---|---|---|
| `read` | 查看资源详情、列表 | — |
| `write` | 编辑资源内容/元数据 | 覆盖 `read` |
| `execute` | 使用资源（如用数据集训练、用模型推理） | 覆盖 `read` |
| `download` | 下载资源文件 | 独立权限 |
| `delete` | 删除资源 | 独立权限（通常需审批） |
| `admin` | 完全控制（含 ACL 管理） | 覆盖以上全部 |
### 6.3 ACL 管理接口
| 接口 | 方法 | 权限要求 | 说明 |
|---|---|---|---|
| `/resources/{type}/{id}/acl` | GET | admin 或资源所有者 | 查询资源 ACL |
| `/resources/{type}/{id}/acl` | PUT | admin 或资源所有者 | 设置资源 ACL（全量替换） |
### 6.4 ACL 管理规则
- **admin** 可以在任何资源上设置 ACL
- **资源所有者** 可以在自己的资源上设置 ACL
- **被授权用户** 不能转授自己获得的权限
- ACL 变更产生审计日志：`action=resource.acl.set`
- 设置 ACL 时全量替换该资源的所有 ACL 条目
### 6.5 前端 ACL 管理入口
在数据集详情、模型详情、训练任务详情页面提供「资源授权」按钮，弹出 ACL 管理对话框：
- 显示当前 ACL 列表（主体类型 + 主体名称 + 权限勾选）
- 支持按用户或按角色添加授权
- 权限以多选框形式展示（read / write / execute / download / delete）
---
## 7. GPU 算力分配与隔离
### 7.1 设计背景
服务器可能安装多张 GPU 卡（如 8×A100），需要精细化管控：
- 管理员指定哪些用户可以使用哪些 GPU 卡
- 未被分配的 GPU 卡对用户不可见或不可选
- admin 可以使用全部 GPU
### 7.2 GPU 分配表
```sql
CREATE TABLE IF NOT EXISTS gpu_assignments (
id TEXT PRIMARY KEY,
gpu_id TEXT NOT NULL, -- gpus 表的外键（node_id + gpu_index 组合）
node_id TEXT NOT NULL, -- 算力节点 ID
gpu_index INTEGER NOT NULL, -- GPU 卡序号
user_id TEXT NOT NULL, -- 被分配的用户 ID
assigned_by TEXT, -- 分配操作人 ID（admin）
assigned_at TEXT NOT NULL, -- 分配时间
UNIQUE (node_id, gpu_index, user_id) -- 一张卡可分配给多个用户，但每对 (卡, 用户) 唯一
);
```
### 7.3 分配规则
| 规则 | 说明 |
|---|---|
| 谁可分配 | 只有 `admin` 角色可以分配 GPU |
| admin 使用 | admin 可使用全部 GPU，不需要显式分配 |
| 普通用户 | 只能使用 `gpu_assignments` 中分配给自己的 GPU |
| 共享分配 | 一张 GPU 可分配给多个用户（非独占），但同时只能被一个任务占用 |
| 默认策略 | 新用户默认不分配任何 GPU，由管理员显式分配 |
### 7.4 GPU 分配接口
| 接口 | 方法 | 权限 | 说明 |
|---|---|---|---|
| `/compute/gpu-assignments` | GET | admin | 查看全部分配关系 |
| `/compute/gpu-assignments` | POST | admin | 批量分配（body: `{ assignments: [{ node_id, gpu_index, user_id }] }`） |
| `/compute/gpu-assignments/{id}` | DELETE | admin | 撤销某条分配 |
| `/compute/my-gpus` | GET | 登录用户 | 查看自己可用的 GPU 列表 |
### 7.5 训练/评测/推理 GPU 选择校验
当普通用户创建训练任务、评测任务、推理任务并选择 GPU 时：
1. 后端检查 `gpu_assignments` 表，确认用户被分配了所选 GPU
2. 未被分配的 GPU → 返回 403 `"无权使用 GPU {node}:{index}"`
3. admin 用户跳过此校验
### 7.6 前端 GPU 选择交互
- 普通用户在创建任务选择 GPU 时，下拉列表只显示自己被分配的 GPU
- admin 用户在下拉列表中可看到全部 GPU
- 未分配任何 GPU 的用户，GPU 选择区域显示提示："未分配 GPU，请联系管理员"
---
## 8. 审批拦截机制
### 8.1 需要审批的操作
| 操作 | 触发条件 | 审批动作 code |
|---|---|---|
| 删除他人数据集 | 非 admin 删除 `created_by != user.id` 的数据集 | `dataset.delete` |
| 删除他人模型 | 非 admin 删除 `created_by != user.id` 的模型 | `model.delete` |
| 停止他人训练任务 | 非 admin 停止 `created_by != user.id` 的任务 | `fine_tune.stop` |
| 发布模型到推理服务 | 任何用户（含 admin）发布到生产环境 | `model_service.publish` |
| 删除项目空间 | 存在待审批变更时拒绝 | `project.delete` |
| 归档项目空间 | 存在待审批变更时拒绝 | `project.archive` |
| 导出训练产物 | 非 admin 导出他人训练的模型 | `trained_model.export` |
### 8.2 审批流程
```
普通用户发起高风险操作
│
▼
┌──────────────┐ ┌──────────────────────┐
│ admin 旁路？ │───是──▶│ 直接执行 + 审计日志 │
└──────┬───────┘ └──────────────────────┘
│ 否
▼
┌──────────────────────┐
│ 创建审批实例 │
│ status=pending │
│ 返回 202（待审批） │
└──────────┬───────────┘
│
▼
┌──────────────────────┐
│ 管理员审批 │
│ POST /approvals/:id │
│ /steps/:idx/decision │
└──────────┬───────────┘
│
┌──────┴──────┐
│ │
approved rejected
│ │
▼ ▼
执行操作 不执行
+审计日志 +审计日志
```
### 8.3 审批模板
审批模板定义了特定操作需要几步审批、每步的审批人是谁：
```json
{
"id": "tpl_001",
"name": "删除数据集审批",
"action": "dataset.delete",
"steps": [
{ "approver_id": "u_admin", "step_name": "管理员审核" }
]
}
```
### 8.4 审批拦截点
在项目模块的 `_require_no_pending_approval` 函数中，当存在待审批实例时拒绝执行新操作。其他模块通过 `_require_approval_or_admin` 函数实现 admin 旁路或创建审批实例。
---
## 9. 审计日志
### 9.1 审计范围
所有写操作和敏感操作必须产生审计日志：
| 动作分类 | action 示例 |
|---|---|
| 用户管理 | `user.create`, `user.update`, `user.delete`, `user.permission.update` |
| 租户管理 | `tenant.create`, `tenant.update`, `tenant.quota.set`, `tenant.retention.set` |
| 项目管理 | `project.create`, `project.update`, `project.archive`, `project.delete`, `project.member.add`, `project.member.update`, `project.member.remove` |
| 资源 ACL | `resource.acl.set` |
| 模型管理 | `model.create`, `model.update`, `model.delete`, `model.merge` |
| 数据集 | `dataset.create`, `dataset.update`, `dataset.delete`, `dataset.upload` |
| 训练任务 | `fine_tune.create`, `fine_tune.start`, `fine_tune.stop`, `fine_tune.delete` |
| 评测任务 | `eval.create`, `eval.start`, `eval.stop` |
| 推理任务 | `inference.create`, `inference.start`, `inference.stop` |
| 审批 | `approval.create`, `approval.decide` |
| 留存策略 | `retention.create`, `retention.update`, `retention.delete` |
| GPU 分配 | `gpu.assign`, `gpu.unassign` |
### 9.2 审计日志字段
```sql
CREATE TABLE IF NOT EXISTS audit_logs (
id TEXT PRIMARY KEY,
time TEXT NOT NULL, -- ISO8601 时间戳
tenant_id TEXT, -- 租户 ID（可选）
project_id TEXT, -- 项目 ID（可选）
actor_id TEXT, -- 操作人 ID
action TEXT NOT NULL, -- 动作类型
target_type TEXT NOT NULL, -- 目标资源类型
target_id TEXT, -- 目标资源 ID
detail TEXT, -- 详情摘要
client_ip TEXT -- 客户端 IP
);
```
### 9.3 查询与导出
| 接口 | 方法 | 说明 |
|---|---|---|
| `/system/audit-logs` | GET | 分页查询，支持按 tenant_id / project_id / actor_id / action / target_type / start_time / end_time 筛选 |
| `/system/audit-logs/export` | GET | CSV 导出，与应用查询相同的过滤条件 |
---
## 10. 接口鉴权流程
### 10.1 Token 格式
```
Authorization: Bearer platform-token-{user_id}
```
登录成功后返回 `token` 和 `user` 信息。Token 中编码了 `user_id`，后端通过 `get_current_user` 解析。
### 10.2 鉴权层级
```
请求到达
│
├─ 1. 公开路径检查（/health, /login, /system-info）→ 直接放行
│
├─ 2. Token 解析 → get_current_user
│ ├─ 无 token / token 无效 → 401
│ └─ 用户不存在 / 状态 disabled → 401
│
├─ 3. 页面级权限码检查（路由守卫 / Depends）
│ └─ user.permissions 不含所需权限码 → 403
│
├─ 4. 资源级权限检查（路由函数内）
│ ├─ admin → 全部放行
│ ├─ resource.created_by == user.id → 放行
│ ├─ ACL 检查 has_resource_access() → 有授权则放行
│ └─ 否则 → 403
│
├─ 5. GPU 权限检查（训练/评测/推理创建时）
│ ├─ admin → 全部放行
│ ├─ gpu_assignments 检查 → 有分配则放行
│ └─ 否则 → 403
│
└─ 6. 审批拦截检查（高风险操作）
├─ admin → 旁路，直接执行
├─ 无待审批实例 → 可执行
├─ 有待审批实例 → 409 "存在待审批的变更"
└─ 需要审批 → 202 "已创建审批实例"
```
### 10.3 FastAPI 依赖注入
```python
# 任何需要登录的接口
@router.get("/datasets")
async def list_datasets(user: dict = Depends(get_current_user)):
...
# 需要管理员权限的接口
@router.post("/users")
async def create_user(user: dict = Depends(require_admin)):
...
# 需要资源级权限检查的接口
@router.delete("/datasets/{dataset_id}")
async def delete_dataset(
dataset_id: str,
user: dict = Depends(get_current_user),
):
if not has_resource_access("dataset", dataset_id, user, "delete"):
raise HTTPException(403, "forbidden")
...
```
---
## 11. 数据库表结构
### 11.1 现有表（已实现）
| 表名 | 用途 |
|---|---|
| `users` | 用户表（id, username, password_hash, role, status, permissions, protected） |
| `roles` | 角色定义（name, permissions） |
| `sessions` | 登录会话（user_id, issued_at, expires_at, ip） |
| `acls` | 资源访问控制列表（resource_type, resource_id, principal_type, principal_id, permission） |
| `audit_logs` | 审计日志（actor_id, action, target_type, target_id, time） |
| `datasets` | 数据集（需补充 `created_by` 字段） |
| `models` | 基座模型（需补充 `created_by` 字段） |
| `trained_models` | 训练产物（需补充 `created_by` 字段） |
| `fine_tune_tasks` | 训练任务（payload 中存储 `created_by`） |
| `gpus` | GPU 设备（node_id, gpu_index, uuid, name, memory） |
| `compute_nodes` | 算力节点 |
| `tenants` | 租户 |
| `projects` | 项目空间 |
| `project_members` | 项目成员 |
| `approval_templates` | 审批模板 |
| `approval_instances` | 审批实例 |
| `retention_policies` | 留存策略 |
### 11.2 需新增/补充的表和字段
#### 新增 `gpu_assignments` 表
```sql
CREATE TABLE IF NOT EXISTS gpu_assignments (
id TEXT PRIMARY KEY,
node_id TEXT NOT NULL REFERENCES compute_nodes(id) ON DELETE CASCADE,
gpu_index INTEGER NOT NULL,
user_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
assigned_by TEXT,
assigned_at TEXT NOT NULL,
UNIQUE (node_id, gpu_index, user_id)
);
CREATE INDEX IF NOT EXISTS idx_gpu_assignments_user ON gpu_assignments(user_id);
CREATE INDEX IF NOT EXISTS idx_gpu_assignments_gpu ON gpu_assignments(node_id, gpu_index);
```
#### 补充 `created_by` 字段
```sql
-- 数据集表补充所有者
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS created_by TEXT;
-- 基座模型表补充所有者
ALTER TABLE models ADD COLUMN IF NOT EXISTS created_by TEXT;
-- 训练产物表补充所有者
ALTER TABLE trained_models ADD COLUMN IF NOT EXISTS created_by TEXT;
-- 评测任务表补充所有者
ALTER TABLE eval_tasks ADD COLUMN IF NOT EXISTS created_by TEXT;
-- 推理任务表补充所有者
-- 注意：inference_tasks 表尚未创建，后续建表时直接包含 created_by 字段
```
---
## 12. API 接口清单
### 12.1 鉴权接口
| 接口 | 方法 | 鉴权 | 说明 |
|---|---|---|---|
| `/modelTF/login` | POST | 公开 | 登录，返回 token + user |
| `/modelTF/me` | GET | Bearer token | 获取当前用户信息 |
| `/modelTF/users` | GET | admin | 用户列表 |
| `/modelTF/users` | POST | admin | 创建用户 |
| `/modelTF/users/:id` | PUT | admin | 更新用户（角色/状态/权限） |
| `/modelTF/users/:id` | DELETE | admin | 删除用户（protected 用户不可删） |
| `/modelTF/users/:id/reset-password` | POST | admin | 重置密码 |
| `/modelTF/system/permissions/codes` | GET | 登录 | 权限码清单 |
| `/modelTF/system/permissions` | GET | 登录 | 权限码 + 角色定义 |
### 12.2 资源 ACL 接口
| 接口 | 方法 | 鉴权 | 说明 |
|---|---|---|---|
| `/modelTF/resources/:type/:id/acl` | GET | admin 或所有者 | 查询资源 ACL |
| `/modelTF/resources/:type/:id/acl` | PUT | admin 或所有者 | 设置资源 ACL |
### 12.3 GPU 分配接口
| 接口 | 方法 | 鉴权 | 说明 |
|---|---|---|---|
| `/modelTF/compute/gpu-assignments` | GET | admin | 查看全部分配 |
| `/modelTF/compute/gpu-assignments` | POST | admin | 批量分配 |
| `/modelTF/compute/gpu-assignments/:id` | DELETE | admin | 撤销分配 |
| `/modelTF/compute/my-gpus` | GET | 登录 | 查看自己可用 GPU |
### 12.4 审批接口
| 接口 | 方法 | 鉴权 | 说明 |
|---|---|---|---|
| `/modelTF/approvals/templates` | GET/POST | admin | 审批模板列表/创建 |
| `/modelTF/approvals` | GET/POST | 登录 | 审批实例列表/创建 |
| `/modelTF/approvals/:id` | GET | 登录 | 审批实例详情 |
| `/modelTF/approvals/:id/steps/:idx/decision` | POST | 审批人 | 审批决策 |
### 12.5 审计接口
| 接口 | 方法 | 鉴权 | 说明 |
|---|---|---|---|
| `/modelTF/system/audit-logs` | GET | admin | 审计日志分页查询 |
| `/modelTF/system/audit-logs/export` | GET | admin | CSV 导出 |
### 12.6 租户/项目接口
| 接口 | 方法 | 鉴权 | 说明 |
|---|---|---|---|
| `/modelTF/tenants` | GET/POST | admin | 租户列表/创建 |
| `/modelTF/tenants/:id` | GET/PUT | admin | 租户详情/更新 |
| `/modelTF/tenants/:id/quota` | PUT | admin | 设置配额 |
| `/modelTF/tenants/:id/retention-policy` | PUT | admin | 绑定留存策略 |
| `/modelTF/projects` | GET/POST | 登录 | 项目列表/创建 |
| `/modelTF/projects/:id` | GET/PUT | 登录 | 项目详情/更新 |
| `/modelTF/projects/:id/archive` | POST | admin 或所有者 | 归档（审批拦截） |
| `/modelTF/projects/:id/members` | GET/POST | 登录 | 成员列表/添加 |
| `/modelTF/projects/:id/members/:uid` | PUT/DELETE | admin 或所有者 | 改角色/移除 |
### 12.7 留存策略接口
| 接口 | 方法 | 鉴权 | 说明 |
|---|---|---|---|
| `/modelTF/retention-policies` | GET/POST | admin | 策略列表/创建 |
| `/modelTF/retention-policies/:id` | GET/PUT/DELETE | admin | 策略详情/更新/删除 |
---
## 13. 前端权限控制
### 13.1 路由守卫
```typescript
// router/index.ts
router.beforeEach((to, _from, next) => {
const auth = useAuthStore()
auth.syncSession()
if (to.meta.public) {
next()
return
}
if (!auth.isLoggedIn) {
next({ name: 'login' })
return
}
if (!to.meta.skipPermission) {
const permission = requiredPermission(to.path, to.meta.permission)
if (permission && !auth.hasPermission(permission)) {
next({ name: 'permission-denied', replace: true })
return
}
}
next()
})
```
### 13.2 侧边栏过滤
```typescript
// layouts/MainLayout.vue
const visibleMenus = computed(() =>
allMenus.filter(menu => {
if (!menu.permission) return true
return auth.hasPermission(menu.permission)
})
)
```
### 13.3 资源级按钮控制
```vue
<!-- 数据集详情页 -->
<template>
<el-button v-if="canEdit" @click="handleEdit">编辑</el-button>
<el-button v-if="canDelete" @click="handleDelete">删除</el-button>
<el-button v-if="canManageAcl" @click="showAclDialog = true">资源授权</el-button>
</template>
<script setup>
const canEdit = computed(() =>
isAdmin.value || resource.value.created_by === userId.value
)
const canDelete = computed(() =>
isAdmin.value || resource.value.created_by === userId.value
)
const canManageAcl = computed(() =>
isAdmin.value || resource.value.created_by === userId.value
)
</script>
```
### 13.4 GPU 选择过滤
```vue
<!-- 创建训练任务页 -->
<template>
<el-select v-model="selectedGpus" multiple>
<el-option
v-for="gpu in availableGpus"
:key="gpu.id"
:label="`${gpu.node_name} GPU ${gpu.gpu_index}`"
:value="gpu.id"
/>
</el-select>
<el-alert v-if="availableGpus.length === 0 && !isAdmin" type="warning">
未分配 GPU，请联系管理员
</el-alert>
</template>
<script setup>
// 普通用户只看到 my-gpus 返回的列表
// admin 看到全部 GPU
const availableGpus = ref([])
async function loadGpus() {
if (isAdmin.value) {
availableGpus.value = await getAllGpus()
} else {
availableGpus.value = await getMyGpus()
}
}
</script>
```
---
## 14. 安全设计补充
### 14.1 密码安全
- 密码使用 PBKDF2-SHA256 存储（salt + 390000 次迭代）
- 旧系统明文密码在首次登录时自动升级为哈希
- 管理员可重置用户密码，用户不可自行修改密码（本期设计）
- 默认密码：`platform123`（创建用户时由管理员设定）
### 14.2 会话安全
| 规则 | 说明 |
|---|---|
| Token 格式 | `platform-token-{user_id}` |
| 会话超时 | 默认 30 分钟无操作自动过期 |
| 并发会话 | 同一用户可有多会话，各自独立计时 |
| 会话续期 | 前端定时调用 `auth.refresh()` 续期 |
| 强制下线 | admin 可通过修改用户 status=disabled 使其 token 失效 |
### 14.3 操作限流
| 接口 | 限制 |
|---|---|
| `/login` | 同一 IP 5 次/分钟，失败后 30 秒冷却 |
| 文件上传 | 单文件最大由配置控制，默认 2GB |
| 训练任务创建 | 同一用户并发运行任务数受 GPU 分配限制 |
### 14.4 数据安全
| 规则 | 说明 |
|---|---|
| 软删除 | 数据集、模型、任务使用 `deleted_at` 标记，保留审计可追溯 |
| 敏感字段 | API 密钥（`api_key`）在列表接口不返回明文 |
| 下载审计 | 数据集下载产生审计日志，记录下载人和时间 |
| 导出审计 | 训练产物导出产生审计日志 |
### 14.5 多租户隔离
| 规则 | 说明 |
|---|---|
| 租户隔离 | 同一租户内的资源相互可见；跨租户默认不可见 |
| 项目隔离 | 项目内资源受项目 ACL 控制；项目间默认不可见 |
| admin 旁路 | admin 可跨租户/项目访问全部资源 |
| 配额管控 | 租户级配额限制 GPU 并发数、存储容量、最大项目数 |
---
## 15. 实施计划
### 15.1 已实现
| 功能 | 状态 |
|---|---|
| 登录/会话/Token | ✅ 已实现 |
| 用户 CRUD + 权限码 | ✅ 已实现 |
| 角色定义 | ✅ 已实现 |
| 资源 ACL（acls 表 + 接口） | ✅ 已实现 |
| 审计日志（查询 + 导出） | ✅ 已实现 |
| 审批模板/实例 | ✅ 已实现 |
| 项目空间 + 成员 | ✅ 已实现 |
| 租户 + 配额 + 留存 | ✅ 已实现 |
| 审批拦截（项目归档/删除） | ✅ 已实现 |
| 资源所有权 ACL 字段适配（subject_type/permissions[]） | ✅ 已实现 |
### 15.2 待实现
| 功能 | 优先级 | 涉及表/接口 |
|---|---|---|
| GPU 分配表 + 接口 | P0 | `gpu_assignments` 表 + `/compute/gpu-assignments` + `/compute/my-gpus` |
| 资源 `created_by` 字段补充 | P0 | `datasets` / `models` / `trained_models` / `eval_tasks` 表 ALTER |
| 资源列表按 `created_by` + ACL 过滤 | P0 | `platform_store.py` 中 datasets/models/tasks 列表方法 |
| GPU 选择校验（训练/评测/推理创建时） | P0 | `platform.py` 中 create_task/eval/inference |
| 前端 GPU 下拉过滤 | P1 | 前端创建任务页面 |
| 前端资源授权按钮 | P1 | 前端数据集/模型/任务详情页 |
| 前端权限管理页面优化 | P1 | 前端用户设置页面 |
| 审批拦截扩展（删除数据集/模型/停止任务） | P1 | `platform.py` 中 delete/stop 接口 |
| 密码安全策略（用户自行修改） | P2 | 新增 `/users/me/password` 接口 |
| 操作限流（login 限流） | P2 | 中间件或 SlowAPI |
| 多租户隔离（按 tenant_id 过滤） | P2 | 各列表接口增加 tenant_id 过滤 |
### 15.3 实施步骤
1. **数据库迁移**：创建 `gpu_assignments` 表，为资源表补充 `created_by` 字段
2. **后端接口**：实现 GPU 分配 CRUD + `my-gpus` + 创建任务时的 GPU 权限校验
3. **资源过滤**：在 `datasets()` / `models()` / `tasks()` 等列表方法中按 `created_by` + ACL 过滤
4. **前端适配**：GPU 下拉过滤、资源授权按钮、权限管理页面优化
5. **审批扩展**：在删除/停止接口中接入 `_require_approval_or_admin`
6. **测试补充**：扩展 `test_governance.py` 覆盖 GPU 分配、资源过滤、审批扩展场景

17
docs/postgres-schema.sql
View File

@@ -1061,20 +1061,8 @@ CREATE TABLE IF NOT EXISTS compute_nodes (
name varchar(150) NOT NULL,
host varchar(200) NOT NULL,
api_base_url text NOT NULL,
file_gateway_url text NOT NULL DEFAULT '',
storage_node_id uuid REFERENCES storage_nodes(id) ON DELETE SET NULL,
status varchar(40) NOT NULL DEFAULT 'online',
scheduler_status varchar(40) NOT NULL DEFAULT 'online',
scheduler_weight integer NOT NULL DEFAULT 100,
enabled boolean NOT NULL DEFAULT true,
max_parallel_jobs integer NOT NULL DEFAULT 1,
data_root text NOT NULL DEFAULT '/data/yg-ft',
model_root text NOT NULL DEFAULT '/data/yg-ft/models',
log_root text NOT NULL DEFAULT '/opt/yg-ft/logs/training',
api_version varchar(40) NOT NULL DEFAULT 'v1',
capabilities jsonb NOT NULL DEFAULT '[]'::jsonb,
description text,
health_detail jsonb NOT NULL DEFAULT '{}'::jsonb,
agent_version varchar(80),
gpu_count integer NOT NULL DEFAULT 0,
last_heartbeat_at timestamptz,
@@ -1446,15 +1434,10 @@ ALTER TABLE fine_tune_tasks ADD COLUMN IF NOT EXISTS tenant_id uuid REFERENCES t
ALTER TABLE fine_tune_tasks ADD COLUMN IF NOT EXISTS project_id uuid REFERENCES projects(id) ON DELETE SET NULL;
ALTER TABLE fine_tune_tasks ADD COLUMN IF NOT EXISTS owner_id uuid REFERENCES users(id) ON DELETE SET NULL;
ALTER TABLE fine_tune_tasks ADD COLUMN IF NOT EXISTS approval_status approval_status NOT NULL DEFAULT 'not_required';
ALTER TABLE fine_tune_tasks ADD COLUMN IF NOT EXISTS compute_node_id uuid REFERENCES compute_nodes(id) ON DELETE SET NULL;
ALTER TABLE fine_tune_tasks ADD COLUMN IF NOT EXISTS compute_job_id uuid REFERENCES compute_jobs(id) ON DELETE SET NULL;
ALTER TABLE fine_tune_tasks ADD COLUMN IF NOT EXISTS resume_checkpoint_id uuid REFERENCES fine_tune_checkpoints(id) ON DELETE SET NULL;
CREATE INDEX IF NOT EXISTS idx_fine_tune_tasks_scope_status
ON fine_tune_tasks(tenant_id, project_id, status, created_at DESC) WHERE deleted_at IS NULL;
CREATE INDEX IF NOT EXISTS idx_fine_tune_tasks_compute_job
ON fine_tune_tasks(compute_job_id) WHERE deleted_at IS NULL;
CREATE INDEX IF NOT EXISTS idx_fine_tune_tasks_node_status
ON fine_tune_tasks(compute_node_id, status, created_at DESC) WHERE deleted_at IS NULL;
ALTER TABLE inference_tasks ADD COLUMN IF NOT EXISTS tenant_id uuid REFERENCES tenants(id) ON DELETE SET NULL;
ALTER TABLE inference_tasks ADD COLUMN IF NOT EXISTS project_id uuid REFERENCES projects(id) ON DELETE SET NULL;

Some files were not shown because too many files have changed in this diff Show More