Compare commits
1 Commits
baseline/f
...
1468834116
| Author | SHA1 | Date | |
|---|---|---|---|
| 1468834116 |
34
.gitignore
vendored
34
.gitignore
vendored
@@ -12,8 +12,6 @@ __pycache__/
|
||||
build/
|
||||
develop-eggs/
|
||||
dist/
|
||||
node_modules/
|
||||
*.tsbuildinfo
|
||||
downloads/
|
||||
eggs/
|
||||
.eggs/
|
||||
@@ -39,16 +37,6 @@ MANIFEST
|
||||
pip-log.txt
|
||||
pip-delete-this-directory.txt
|
||||
|
||||
# Runtime data and logs
|
||||
runtime/
|
||||
backend/runtime/
|
||||
backend/storage/
|
||||
logs/
|
||||
backend/logs/
|
||||
*.db
|
||||
*.sqlite
|
||||
*.sqlite3
|
||||
|
||||
# Unit test / coverage reports
|
||||
htmlcov/
|
||||
.tox/
|
||||
@@ -142,7 +130,6 @@ celerybeat.pid
|
||||
|
||||
# Environments
|
||||
.env
|
||||
!.env.example
|
||||
.venv
|
||||
env/
|
||||
venv/
|
||||
@@ -150,16 +137,6 @@ ENV/
|
||||
env.bak/
|
||||
venv.bak/
|
||||
|
||||
# Local backend config (含数据库账号密码等敏感信息，勿提交)
|
||||
backend/config.yaml
|
||||
|
||||
# Agent / IDE 工具产物，不应进版本库
|
||||
.codex-backups/
|
||||
.pnpm-store/
|
||||
.zcode/
|
||||
.claude/
|
||||
CLAUDE.md
|
||||
|
||||
# Spyder project settings
|
||||
.spyderproject
|
||||
.spyproject
|
||||
@@ -197,14 +174,3 @@ cython_debug/
|
||||
# PyPI configuration file
|
||||
.pypirc
|
||||
|
||||
docker/llamafactory-latest.tar.gz
|
||||
# Compute data - 保留目录结构和 README，忽略子目录内容（日志、模型、数据集等）
|
||||
!docker/compute/data/yg-ft/logs/
|
||||
docker/compute/data/yg-ft/datasets/*
|
||||
docker/compute/data/yg-ft/models/*
|
||||
docker/compute/data/yg-ft/outputs/*
|
||||
docker/compute/data/yg-ft/logs/**
|
||||
!docker/compute/data/yg-ft/logs/compute/
|
||||
!docker/compute/data/yg-ft/logs/training/
|
||||
!docker/compute/data/yg-ft/**/.gitkeep
|
||||
!docker/compute/data/yg-ft/**/README.md
|
||||
|
||||
19
Dockerfile
Normal file
19
Dockerfile
Normal file
@@ -0,0 +1,19 @@
|
||||
# syntax=docker/dockerfile:1
|
||||
|
||||
FROM node:20-alpine AS build
|
||||
WORKDIR /app/frontend
|
||||
|
||||
COPY frontend/package*.json ./
|
||||
RUN npm ci
|
||||
|
||||
COPY frontend/ ./
|
||||
RUN npm run build
|
||||
|
||||
FROM nginx:1.27-alpine
|
||||
COPY docker/nginx.conf.template /etc/nginx/templates/default.conf.template
|
||||
COPY --from=build /app/frontend/dist /usr/share/nginx/html
|
||||
|
||||
EXPOSE 80
|
||||
|
||||
HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
|
||||
CMD wget -qO- http://127.0.0.1/ >/dev/null || exit 1
|
||||
168
README.md
168
README.md
@@ -1,14 +1,12 @@
|
||||
# YG_FT 模型微调平台
|
||||
|
||||
YG_FT 是一个面向企业治理场景的模型微调平台，覆盖用户中心、多租户、项目隔离、数据集管理、模型管理、训练任务、评测、推理、审批流、审计留存、算力调度和训练引擎适配。
|
||||
|
||||
当前前端已有基础页面，后端与算力平台已按多人协作开发方式建立工程骨架，并开始实现正式系统主链路能力。当前代码和 SQL 均作为后续生产演进基线维护，不再以一次性演示或静态 Mock 为开发准则。
|
||||
YG_FT 是一个面向企业治理场景的完整模型微调平台，覆盖用户中心、多租户、项目隔离、数据集管理、模型管理、训练任务、评测、推理、审批流、审计留存、算力调度和训练引擎适配。当前前端已存在基础页面，后端与算力平台已按多人协作开发方式建立工程骨架。
|
||||
|
||||
## 总体架构
|
||||
|
||||
```text
|
||||
YG_FT/
|
||||
frontend/ # 前端控制台
|
||||
frontend/ # 前端应用，承载训练平台控制台页面
|
||||
backend/ # FastAPI 应用平台后端
|
||||
app/
|
||||
api/v1/ # 对前端暴露的 REST API
|
||||
@@ -25,7 +23,7 @@ YG_FT/
|
||||
engines/llama_factory/ # LLaMA-Factory 适配器
|
||||
file_gateway/ # 本地文件上传、下载、导入、产物管理
|
||||
docs/ # 需求、接口、数据库、开发计划和部署文档
|
||||
docker/ # 容器化配置
|
||||
docker/ # Nginx 等容器化配置
|
||||
```
|
||||
|
||||
## 平台分层
|
||||
@@ -34,55 +32,20 @@ YG_FT/
|
||||
| --- | --- | --- |
|
||||
| 前端控制台 | 用户操作入口、任务看板、项目/模型/数据集/训练/审批/审计页面 | `frontend/` |
|
||||
| 应用平台后端 | 用户中心、多租户、RBAC/ABAC、项目隔离、元数据、审批流、审计、API 编排 | `backend/` |
|
||||
| 算力平台 | GPU 发现、资源锁定、训练进程管理、日志采集、产物归档、任务状态同步 | `compute/` |
|
||||
| 算力平台 | GPU 发现、资源锁定、训练进程管理、日志采集、产物归档、任务状态回传 | `compute/` |
|
||||
| 训练引擎 | 当前固定接入 LLaMA-Factory，预留其他训练平台适配标准 | `compute/engines/` |
|
||||
| 数据层 | PostgreSQL、Redis、本地文件存储、日志归档 | `docs/postgres-schema.sql` |
|
||||
|
||||
## 当前开发基线
|
||||
## 关键能力
|
||||
|
||||
- 使用 FastAPI 提供统一 API 响应结构 `{ code, message, data }`。
|
||||
- 本地运行阶段统一使用 PostgreSQL，后端启动时会在 PG 中初始化当前运行表和系统内置账号；模型、数据集、算力节点、GPU、微调任务等业务数据必须通过页面、接口或正式导入流程产生。
|
||||
- 支持登录、模型管理、数据集管理、微调任务创建/启动/停止/进度轮询。
|
||||
- 支持训练日志、loss 指标、checkpoint 和训练产物接口；真实训练执行器接入前，联调状态机必须通过显式环境变量开启。
|
||||
- 支持多算力节点、GPU、任务队列、资源副本和资源同步状态接口。
|
||||
- 前端新增 `/compute` 算力节点页面，展示节点地址、权重、标签、启用状态、GPU、队列和资源副本。
|
||||
- `compute/engines/llama_factory/adapter.py` 提供 LLaMA-Factory 参数校验、命令生成和日志解析基础能力。
|
||||
|
||||
## 前后端一键启动
|
||||
|
||||
首次使用前，请先按下方“后端启动”和“前端启动”说明安装依赖，并确保
|
||||
PostgreSQL 已可用。之后在项目根目录执行：
|
||||
|
||||
```bash
|
||||
bash ./start.sh
|
||||
```
|
||||
|
||||
脚本会同时启动前端 `http://localhost:16801` 和后端
|
||||
`http://127.0.0.1:17861`，按 `Ctrl+C` 会同时停止两个服务。脚本只负责
|
||||
启动前后端，不会自动安装依赖，也不会启动 PostgreSQL、Redis 或算力服务。
|
||||
|
||||
仅检查依赖和端口而不启动服务：
|
||||
|
||||
```bash
|
||||
bash ./start.sh --check
|
||||
```
|
||||
|
||||
本地启动推荐只配置数据库主机。脚本会复用 `docker/app/.env` 中已有的
|
||||
`POSTGRES_USER`、`POSTGRES_PASSWORD` 和 `POSTGRES_DB`，端口默认使用
|
||||
PostgreSQL 标准端口 `5432`：
|
||||
|
||||
```bash
|
||||
DATABASE_HOST='www.caoxiaozhu.com' bash ./start.sh
|
||||
```
|
||||
|
||||
也可以在 `docker/app/.env` 中增加：
|
||||
|
||||
```env
|
||||
DATABASE_HOST=www.caoxiaozhu.com
|
||||
```
|
||||
|
||||
需要使用非标准端口时再设置 `DATABASE_PORT`。`DATABASE_URL` 仍可作为完整连接串
|
||||
高级覆盖项；终端环境变量优先级最高。脚本不会输出数据库密码。
|
||||
- 多租户：租户级数据隔离、租户配置、租户成员和角色。
|
||||
- 权限控制：支持项目、模型、数据集级隔离，后续可扩展到字段级和操作级策略。
|
||||
- 审批流：覆盖数据集发布、模型发布、训练资源申请、推理服务上线等企业流程。
|
||||
- 审计留存：操作审计、安全审计、审批审计、任务审计，支持留存周期策略。
|
||||
- 训练任务：训练参数管理、单机多 GPU 调度、任务状态同步、训练日志、产物管理。
|
||||
- 引擎适配：默认 LLaMA-Factory，预留统一 Engine Adapter 接口接入其他微调框架。
|
||||
- 文件存储：当前使用本地磁盘，按租户/项目/数据集/任务分区。
|
||||
- 日志采集：后端 JSON Lines 日志，主日志和错误日志拆分，便于 ELK/日志平台采集。
|
||||
|
||||
## 后端启动
|
||||
|
||||
@@ -91,89 +54,15 @@ cd backend
|
||||
python -m venv .venv
|
||||
.venv\Scripts\activate
|
||||
pip install -r requirements.txt
|
||||
uvicorn app.main:app --reload --port 17861
|
||||
uvicorn app.main:app --reload
|
||||
```
|
||||
|
||||
默认接口前缀为 `/modelTF`，例如：
|
||||
默认健康检查：
|
||||
|
||||
```text
|
||||
GET /modelTF/health
|
||||
POST /modelTF/login
|
||||
GET /modelTF/model-manage
|
||||
GET /modelTF/dataset-manage
|
||||
GET /modelTF/fine-tune
|
||||
GET /modelTF/compute/nodes
|
||||
GET /api/v1/health
|
||||
```
|
||||
|
||||
本地运行时默认 PostgreSQL 连接：
|
||||
|
||||
```text
|
||||
DATABASE_URL=postgresql+psycopg://yg_ft:change_me@localhost:15432/yg_ft
|
||||
```
|
||||
|
||||
本地启动前需要确保 PostgreSQL 已监听 `localhost:15432`，并已创建 `yg_ft` 数据库和 `yg_ft` 用户。后端启动后会自动创建当前运行表并写入内置管理员账号，运行数据统一写入 PostgreSQL。
|
||||
|
||||
开发阶段内置登录账号：
|
||||
|
||||
| 角色 | 账号 | 密码 | 说明 |
|
||||
| --- | --- | --- | --- |
|
||||
| 超级管理员 | `admin` | `admin123` | 拥有当前全部页面权限 |
|
||||
| 操作员 | `operator` | `operator123` | 拥有业务操作相关页面权限 |
|
||||
|
||||
以上账号仅用于本地开发和联调。生产环境初始化后应立即修改密码，或改为企业统一身份认证/管理员初始化流程。
|
||||
|
||||
## 前端启动
|
||||
|
||||
```bash
|
||||
cd frontend
|
||||
npm install
|
||||
npm run dev
|
||||
```
|
||||
|
||||
前端开发服务默认运行在 `http://localhost:16801`，并通过 Vite proxy 将 `/modelTF` 转发到 `http://localhost:17861`。
|
||||
|
||||
## 算力服务启动
|
||||
|
||||
算力服务是一个 FastAPI 应用，同时承载 Compute API（模型训练/推理/GPU 管理）和 File Gateway（文件上传下载）路由。Docker 部署时对外暴露两个端口（19100 和 19101）均指向同一服务，方便应用平台分别配置 `api_base_url` 和 `file_gateway_url`。本地开发只需启动一个进程。
|
||||
|
||||
### 方式一：Docker 启动（推荐）
|
||||
|
||||
```bash
|
||||
cd docker/compute
|
||||
cp .env.example .env
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
### 方式二：本地开发启动
|
||||
|
||||
**Windows (cmd)：**
|
||||
|
||||
```cmd
|
||||
cd /d E:\yg_ft\compute
|
||||
set PYTHONPATH=E:\yg_ft
|
||||
.\.venv\Scripts\python.exe -m uvicorn api.main:app --reload --port 19100
|
||||
```
|
||||
|
||||
> `PYTHONPATH=E:\yg_ft` 是必需的，因为代码使用 `from compute.agent...` 绝对导入。
|
||||
|
||||
**Linux / macOS：**
|
||||
|
||||
```bash
|
||||
cd compute
|
||||
PYTHONPATH=.. uvicorn api.main:app --reload --port 19100
|
||||
```
|
||||
|
||||
### 环境变量说明
|
||||
|
||||
| 变量 | 默认值 | 说明 |
|
||||
|---|---|---|
|
||||
| `COMPUTE_MODE` | `real` | `real` / `simulator`，仅隔离联调用 simulator |
|
||||
| `COMPUTE_EXECUTION_MODE` | `real` | 训练执行模式 |
|
||||
| `COMPUTE_SERVICE_TOKEN` | `change_me` | 服务间认证 token |
|
||||
| `MODELTF_ROUTE_PREFIX` | `/modelTF` | API 路由前缀 |
|
||||
|
||||
应用平台通过数据库 `compute_nodes` 表中的 `api_base_url` 和 `file_gateway_url` 主动轮询算力节点状态。
|
||||
|
||||
## 日志
|
||||
|
||||
后端日志模块位于 `backend/app/core/logging.py`，说明文档见：
|
||||
@@ -192,35 +81,22 @@ logs/error-YYYY-MM-DD.log
|
||||
## 主要文档
|
||||
|
||||
- `docs/platform-architecture-requirements.md`：平台需求、功能模块、页面补全建议。
|
||||
- `docs/menu-functional-requirements.md`：当前菜单、二级路由、规划菜单、功能需求、接口和数据库映射。
|
||||
- `docs/backend-api-design.md`：FastAPI 接口分组、参数定义、权限说明。
|
||||
- `docs/postgres-schema.sql`：PostgreSQL 数据库脚本，包含权限、用户中心、多租户、审批、审计等模型。
|
||||
- `docs/system-development-plan.md`：多人协作开发计划，按前端、后端、DB、部署拆分。
|
||||
- `docs/team-development-plan.md`：3-4 人并行开发分工计划，按人员边界标注页面、接口、数据库和交付节奏。
|
||||
- `docs/first-version-development-plan.md`：当前系统主链路开发计划，覆盖前端、后端、DB、Compute API、GPU 和 LLaMA-Factory 适配。
|
||||
- `docs/backend-logging.md`：后端日志模块使用说明。
|
||||
- `docs/deployment-plan.md`：后期部署方案，覆盖单机算力服务器部署与应用/算力分离部署。
|
||||
- `docker/README.md`：Docker 部署入口，包含应用服务器和算力服务器两套 Compose 使用方式。
|
||||
|
||||
## Docker 部署入口
|
||||
## 部署模式
|
||||
|
||||
应用服务器：
|
||||
平台支持两种主要部署模式：
|
||||
|
||||
```bash
|
||||
cd docker/app
|
||||
cp .env.example .env
|
||||
docker compose up -d
|
||||
```
|
||||
1. 所有服务部署在算力服务器：适合 PoC、内网试点、小团队单机多 GPU 使用。
|
||||
2. 应用服务和算力/训练服务独立部署：适合企业生产环境，应用平台部署在业务服务区，算力平台和 LLaMA-Factory 部署在 GPU 服务器。
|
||||
|
||||
算力服务器：
|
||||
生产环境建议采用第二种模式。算力平台与训练框架应部署在 GPU 算力服务器上，应用平台不直接控制 GPU 进程，而是通过内部 Compute API 调度训练任务。
|
||||
|
||||
```bash
|
||||
cd docker/compute
|
||||
cp .env.example .env
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
两套 Compose 均采用代码外挂方式运行，镜像只包含运行时环境和第三方依赖。项目根目录不再保留 `Dockerfile` 和 `docker-compose.yml`，部署时统一进入 `docker/app` 或 `docker/compute` 目录执行。
|
||||
详细方案见 `docs/deployment-plan.md`。
|
||||
|
||||
## 后续开发原则
|
||||
|
||||
|
||||
9
backend/.env.example
Normal file
9
backend/.env.example
Normal file
@@ -0,0 +1,9 @@
|
||||
APP_NAME=YG Fine-Tune Platform API
|
||||
APP_ENV=local
|
||||
API_PREFIX=/api
|
||||
LOG_LEVEL=INFO
|
||||
LOG_DIR=./logs
|
||||
LOG_FILE_PREFIX=backend
|
||||
LOG_ERROR_FILE_PREFIX=error
|
||||
LOG_MAX_BYTES=20971520
|
||||
LOG_RETENTION_DAYS=10
|
||||
@@ -8,7 +8,7 @@
|
||||
backend/
|
||||
app/
|
||||
main.py # FastAPI 应用入口
|
||||
api/v1/ # 对前端暴露的 接口路由
|
||||
api/v1/ # 对前端暴露的 API 路由
|
||||
core/ # 配置、日志、中间件、权限等基础能力
|
||||
db/ # 数据库连接、迁移集成、事务工具
|
||||
modules/ # 业务模块
|
||||
@@ -48,7 +48,7 @@ uvicorn app.main:app --reload
|
||||
健康检查：
|
||||
|
||||
```text
|
||||
GET /modelTF/health
|
||||
GET /api/v1/health
|
||||
```
|
||||
|
||||
## 日志
|
||||
|
||||
@@ -1,10 +0,0 @@
|
||||
from app.db.platform_store import get_platform_store
|
||||
|
||||
store = get_platform_store()
|
||||
with store.connect() as conn:
|
||||
rows = conn.execute(
|
||||
"SELECT id, user_id, login_at, logout_at, duration_seconds FROM sessions ORDER BY login_at DESC LIMIT 10"
|
||||
).fetchall()
|
||||
print(f"sessions count: {len(rows)}")
|
||||
for r in rows:
|
||||
print(f" user={r['user_id'][:25]}... login={r['login_at']} logout={r['logout_at']} dur={r['duration_seconds']}")
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,18 +1,12 @@
|
||||
from fastapi import APIRouter
|
||||
from fastapi import APIRouter
|
||||
|
||||
from app.core.logging import get_logger
|
||||
from app.db.platform_store import get_platform_store
|
||||
|
||||
router = APIRouter()
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
@router.get("/health")
|
||||
async def health_check() -> dict[str, object]:
|
||||
async def health_check() -> dict[str, str]:
|
||||
logger.info("health check requested")
|
||||
return {
|
||||
"code": 0,
|
||||
"message": "ok",
|
||||
"data": get_platform_store().health_metrics(),
|
||||
}
|
||||
|
||||
return {"status": "ok"}
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,22 +1,6 @@
|
||||
from fastapi import APIRouter
|
||||
from fastapi import APIRouter
|
||||
|
||||
from app.api.v1.endpoints.data_process import router as data_process_router
|
||||
from app.api.v1.endpoints.platform import router as platform_router
|
||||
from app.api.v1.endpoints.health import router as health_router
|
||||
from app.modules.tenant.router import router as tenant_router
|
||||
from app.modules.project.router import router as project_router
|
||||
from app.modules.approval.router import router as approval_router
|
||||
from app.modules.system.router import router as system_router
|
||||
from app.modules.retention.router import router as retention_router
|
||||
from app.modules.resource.router import router as resource_router
|
||||
|
||||
api_router = APIRouter()
|
||||
api_router.include_router(health_router, tags=["health"])
|
||||
api_router.include_router(data_process_router, tags=["data-process"])
|
||||
api_router.include_router(platform_router, tags=["platform"])
|
||||
api_router.include_router(system_router, tags=["system"])
|
||||
api_router.include_router(tenant_router, tags=["tenant"])
|
||||
api_router.include_router(project_router, tags=["project"])
|
||||
api_router.include_router(approval_router, tags=["approval"])
|
||||
api_router.include_router(retention_router, tags=["retention"])
|
||||
api_router.include_router(resource_router, tags=["resource"])
|
||||
|
||||
@@ -1,138 +0,0 @@
|
||||
"""鉴权依赖：从 Authorization header 解析当前用户，提供权限校验。"""
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from fastapi import Depends, HTTPException, Query, Request, status
|
||||
|
||||
from app.db.platform_store import get_platform_store
|
||||
|
||||
# 无需鉴权的路径前缀（健康检查、登录等）
|
||||
PUBLIC_PATHS = ("/health", "/login", "/system-info")
|
||||
|
||||
|
||||
def _extract_token(request: Request) -> str | None:
|
||||
"""从 Authorization header 提取 token（格式: Bearer platform-token-{user_id}）。"""
|
||||
auth = request.headers.get("Authorization", "")
|
||||
token = auth.replace("Bearer ", "").strip()
|
||||
if token.startswith("platform-token-"):
|
||||
return token[len("platform-token-"):]
|
||||
return None
|
||||
|
||||
|
||||
def get_current_user(request: Request) -> dict[str, Any]:
|
||||
"""
|
||||
FastAPI 依赖：解析当前登录用户。
|
||||
- 公开路径（/health, /login 等）直接放行，返回匿名用户。
|
||||
- 无 token 或 token 无效时抛 401。
|
||||
- admin 用户标记为超级管理员，拥有全部权限。
|
||||
"""
|
||||
path = request.url.path
|
||||
# 去掉路由前缀后判断
|
||||
for prefix in PUBLIC_PATHS:
|
||||
if path.endswith(prefix):
|
||||
return {"id": None, "username": "anonymous", "role": "viewer", "permissions": [], "protected": False}
|
||||
|
||||
user_id = _extract_token(request)
|
||||
if not user_id:
|
||||
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="missing or invalid token")
|
||||
|
||||
store = get_platform_store()
|
||||
for u in store.users():
|
||||
if u.get("id") == user_id:
|
||||
return u
|
||||
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="user not found")
|
||||
|
||||
|
||||
def require_admin(current_user: dict[str, Any] = Depends(get_current_user)) -> dict[str, Any]:
|
||||
"""FastAPI 依赖：要求当前用户是管理员（role=admin 或 protected）。"""
|
||||
if current_user.get("role") == "admin" or current_user.get("protected"):
|
||||
return current_user
|
||||
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="admin permission required")
|
||||
|
||||
|
||||
def is_admin(user: dict[str, Any]) -> bool:
|
||||
"""判断用户是否为管理员（admin 角色或 protected 标记）。"""
|
||||
return user.get("role") == "admin" or user.get("protected", False)
|
||||
|
||||
|
||||
def has_resource_access(
|
||||
resource_type: str,
|
||||
resource_id: str,
|
||||
user: dict[str, Any],
|
||||
permission: str = "read",
|
||||
) -> bool:
|
||||
"""
|
||||
检查用户对某资源是否有指定权限。
|
||||
- admin/protected 用户直接放行（旁路）。
|
||||
- 其他用户检查 acls 表中是否有对应授权。
|
||||
"""
|
||||
if user.get("role") == "admin" or user.get("protected"):
|
||||
return True
|
||||
|
||||
store = get_platform_store()
|
||||
acls = store.get_acl(resource_type, resource_id)
|
||||
user_id = user.get("id")
|
||||
user_role = user.get("role")
|
||||
|
||||
for entry in acls:
|
||||
# 按 user 授权
|
||||
if entry.get("principal_type") == "user" and entry.get("principal_id") == user_id:
|
||||
if _permission_covers(entry.get("permission"), permission):
|
||||
return True
|
||||
# 按 role 授权
|
||||
if entry.get("principal_type") == "role" and entry.get("principal_id") == user_role:
|
||||
if _permission_covers(entry.get("permission"), permission):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _permission_covers(granted: str | None, required: str) -> bool:
|
||||
"""权限覆盖判断：write/execute 覆盖 read；admin 覆盖一切。"""
|
||||
if not granted:
|
||||
return False
|
||||
if granted == "admin":
|
||||
return True
|
||||
if granted == required:
|
||||
return True
|
||||
# write 覆盖 read
|
||||
if required == "read" and granted in ("write", "execute"):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def filter_accessible_resource_ids(
|
||||
resource_type: str,
|
||||
all_ids: list[str],
|
||||
user: dict[str, Any],
|
||||
) -> list[str]:
|
||||
"""
|
||||
从全部资源 ID 中过滤出当前用户可访问的 ID 列表。
|
||||
- admin 直接返回全部。
|
||||
- 普通用户查 acls 表取交集。
|
||||
"""
|
||||
if user.get("role") == "admin" or user.get("protected"):
|
||||
return all_ids
|
||||
|
||||
if not all_ids:
|
||||
return []
|
||||
|
||||
store = get_platform_store()
|
||||
user_id = user.get("id")
|
||||
user_role = user.get("role")
|
||||
|
||||
# 查询该用户在该资源类型下有 read 权限的所有 resource_id
|
||||
with store.connect() as conn:
|
||||
rows = conn.execute(
|
||||
"""
|
||||
SELECT DISTINCT resource_id FROM acls
|
||||
WHERE resource_type=? AND (
|
||||
(principal_type='user' AND principal_id=?)
|
||||
OR (principal_type='role' AND principal_id=?)
|
||||
)
|
||||
""",
|
||||
(resource_type, user_id, user_role),
|
||||
).fetchall()
|
||||
|
||||
accessible = {r["resource_id"] for r in rows}
|
||||
return [rid for rid in all_ids if rid in accessible]
|
||||
@@ -1,19 +1,7 @@
|
||||
from dataclasses import dataclass
|
||||
from dataclasses import dataclass
|
||||
from functools import lru_cache
|
||||
import os
|
||||
|
||||
try:
|
||||
from pathlib import Path as _Path
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
# 显式指定 backend 目录下的 .env，并强制覆盖已有环境变量，
|
||||
# 确保远程数据库配置生效，不被本地默认值或残留环境变量影响。
|
||||
_env_path = _Path(__file__).resolve().parent.parent.parent / ".env"
|
||||
load_dotenv(dotenv_path=_env_path, override=True)
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
|
||||
def _int_env(name: str, default: int) -> int:
|
||||
raw = os.getenv(name)
|
||||
@@ -22,26 +10,11 @@ def _int_env(name: str, default: int) -> int:
|
||||
return int(raw)
|
||||
|
||||
|
||||
def _list_env(name: str, default: list[str]) -> list[str]:
|
||||
raw = os.getenv(name)
|
||||
if raw is None or raw.strip() == "":
|
||||
return default
|
||||
return [item.strip() for item in raw.split(",") if item.strip()]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Settings:
|
||||
app_name: str = os.getenv("APP_NAME", "YG Fine-Tune Platform API")
|
||||
app_env: str = os.getenv("APP_ENV", "local")
|
||||
route_prefix: str = os.getenv("MODELTF_ROUTE_PREFIX", "/modelTF")
|
||||
app_mode: str = os.getenv("APP_MODE", "local")
|
||||
database_url: str = os.getenv("DATABASE_URL", "postgresql+psycopg://yg_ft:change_me@localhost:15432/yg_ft")
|
||||
cors_allow_origins: list[str] = None # type: ignore[assignment]
|
||||
compute_mode: str = os.getenv("COMPUTE_MODE", "real")
|
||||
compute_status_sync_mode: str = os.getenv("COMPUTE_STATUS_SYNC_MODE", "polling")
|
||||
compute_poll_interval_seconds: int = _int_env("COMPUTE_POLL_INTERVAL_SECONDS", 3)
|
||||
compute_request_timeout_seconds: int = _int_env("COMPUTE_REQUEST_TIMEOUT_SECONDS", 5)
|
||||
compute_service_token: str = os.getenv("COMPUTE_SERVICE_TOKEN", "")
|
||||
api_prefix: str = os.getenv("API_PREFIX", "/api")
|
||||
log_level: str = os.getenv("LOG_LEVEL", "INFO")
|
||||
log_dir: str = os.getenv("LOG_DIR", "./logs")
|
||||
log_file_prefix: str = os.getenv("LOG_FILE_PREFIX", "backend")
|
||||
@@ -49,23 +22,7 @@ class Settings:
|
||||
log_max_bytes: int = _int_env("LOG_MAX_BYTES", 20 * 1024 * 1024)
|
||||
log_retention_days: int = _int_env("LOG_RETENTION_DAYS", 10)
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
object.__setattr__(
|
||||
self,
|
||||
"cors_allow_origins",
|
||||
_list_env(
|
||||
"CORS_ALLOW_ORIGINS",
|
||||
[
|
||||
"http://localhost:16801",
|
||||
"http://127.0.0.1:16801",
|
||||
"http://localhost:17861",
|
||||
"http://127.0.0.1:17861",
|
||||
],
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
@lru_cache
|
||||
def get_settings() -> Settings:
|
||||
return Settings()
|
||||
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,40 +1,4 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from collections.abc import Generator
|
||||
from contextlib import contextmanager
|
||||
|
||||
from sqlalchemy import create_engine
|
||||
from sqlalchemy.orm import Session, sessionmaker
|
||||
|
||||
|
||||
DATABASE_URL = os.getenv("DATABASE_URL", "postgresql+psycopg://yg_ft:change_me@localhost:15432/yg_ft")
|
||||
|
||||
engine = create_engine(
|
||||
DATABASE_URL,
|
||||
pool_pre_ping=True,
|
||||
future=True,
|
||||
)
|
||||
SessionLocal = sessionmaker(bind=engine, autoflush=False, autocommit=False, expire_on_commit=False, future=True)
|
||||
|
||||
|
||||
def get_db() -> Generator[Session, None, None]:
|
||||
db = SessionLocal()
|
||||
try:
|
||||
yield db
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
@contextmanager
|
||||
def session_scope() -> Generator[Session, None, None]:
|
||||
db = SessionLocal()
|
||||
try:
|
||||
yield db
|
||||
db.commit()
|
||||
except Exception:
|
||||
db.rollback()
|
||||
raise
|
||||
finally:
|
||||
db.close()
|
||||
"""Database session factory placeholder.
|
||||
|
||||
Implement SQLAlchemy/SQLModel session management here when database development starts.
|
||||
"""
|
||||
|
||||
@@ -1,335 +0,0 @@
|
||||
CREATE TABLE IF NOT EXISTS users (
|
||||
id TEXT PRIMARY KEY,
|
||||
username TEXT NOT NULL UNIQUE,
|
||||
password_hash TEXT NOT NULL,
|
||||
display_name TEXT NOT NULL,
|
||||
role TEXT NOT NULL,
|
||||
status TEXT NOT NULL,
|
||||
permissions TEXT NOT NULL,
|
||||
create_time TEXT NOT NULL,
|
||||
last_login TEXT,
|
||||
protected INTEGER NOT NULL DEFAULT 0
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS models (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL UNIQUE,
|
||||
type TEXT NOT NULL,
|
||||
purpose TEXT NOT NULL,
|
||||
model_source TEXT NOT NULL,
|
||||
description TEXT,
|
||||
path TEXT,
|
||||
api_url TEXT,
|
||||
api_key TEXT,
|
||||
online_model_name TEXT,
|
||||
create_time TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS trained_models (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL UNIQUE,
|
||||
train_methods TEXT NOT NULL,
|
||||
base_model_path TEXT,
|
||||
create_time TEXT NOT NULL,
|
||||
merged INTEGER NOT NULL DEFAULT 0,
|
||||
merging INTEGER NOT NULL DEFAULT 0,
|
||||
merged_path TEXT,
|
||||
artifact_dir TEXT,
|
||||
compute_node_id TEXT,
|
||||
compute_node_name TEXT
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS model_lineage (
|
||||
id TEXT PRIMARY KEY,
|
||||
child_resource_type TEXT NOT NULL,
|
||||
child_resource_id TEXT NOT NULL,
|
||||
parent_resource_type TEXT NOT NULL,
|
||||
parent_resource_id TEXT NOT NULL,
|
||||
relation_type TEXT NOT NULL,
|
||||
compute_job_id TEXT,
|
||||
payload TEXT NOT NULL,
|
||||
create_time TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS model_artifacts (
|
||||
id TEXT PRIMARY KEY,
|
||||
model_id TEXT NOT NULL,
|
||||
model_kind TEXT NOT NULL,
|
||||
artifact_type TEXT NOT NULL,
|
||||
path TEXT NOT NULL,
|
||||
size_bytes BIGINT NOT NULL DEFAULT 0,
|
||||
checksum_sha256 TEXT,
|
||||
metadata TEXT NOT NULL,
|
||||
compute_job_id TEXT,
|
||||
create_time TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS model_export_jobs (
|
||||
id TEXT PRIMARY KEY,
|
||||
trained_model_id TEXT,
|
||||
compute_job_id TEXT NOT NULL,
|
||||
node_id TEXT,
|
||||
export_type TEXT NOT NULL,
|
||||
quantization_bit INTEGER NOT NULL DEFAULT 0,
|
||||
status TEXT NOT NULL,
|
||||
output_dir TEXT,
|
||||
payload TEXT NOT NULL,
|
||||
create_time TEXT NOT NULL,
|
||||
completed_at TEXT
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS datasets (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL UNIQUE,
|
||||
type TEXT NOT NULL,
|
||||
storage_type TEXT NOT NULL,
|
||||
source TEXT NOT NULL,
|
||||
task_id TEXT,
|
||||
size TEXT,
|
||||
count INTEGER NOT NULL DEFAULT 0,
|
||||
description TEXT,
|
||||
create_time TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS dataset_files (
|
||||
id TEXT PRIMARY KEY,
|
||||
dataset_id TEXT NOT NULL REFERENCES datasets(id) ON DELETE CASCADE,
|
||||
name TEXT NOT NULL,
|
||||
size TEXT,
|
||||
content TEXT NOT NULL,
|
||||
active_version_id TEXT NOT NULL,
|
||||
versions TEXT NOT NULL,
|
||||
create_time TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS compute_nodes (
|
||||
id TEXT PRIMARY KEY,
|
||||
code TEXT NOT NULL UNIQUE,
|
||||
name TEXT NOT NULL,
|
||||
api_base_url TEXT NOT NULL,
|
||||
file_gateway_url TEXT NOT NULL,
|
||||
enabled INTEGER NOT NULL DEFAULT 1,
|
||||
scheduler_status TEXT NOT NULL,
|
||||
scheduler_weight INTEGER NOT NULL DEFAULT 100,
|
||||
tags TEXT NOT NULL,
|
||||
gpu_count INTEGER NOT NULL DEFAULT 0,
|
||||
current_running_jobs INTEGER NOT NULL DEFAULT 0,
|
||||
max_parallel_jobs INTEGER NOT NULL DEFAULT 2,
|
||||
data_root TEXT NOT NULL,
|
||||
model_root TEXT NOT NULL,
|
||||
log_root TEXT NOT NULL,
|
||||
api_version TEXT NOT NULL DEFAULT 'v1',
|
||||
capabilities TEXT NOT NULL DEFAULT '[]',
|
||||
description TEXT,
|
||||
last_health_check_at TEXT,
|
||||
health_detail TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS gpus (
|
||||
id TEXT PRIMARY KEY,
|
||||
node_id TEXT NOT NULL REFERENCES compute_nodes(id) ON DELETE CASCADE,
|
||||
gpu_index INTEGER NOT NULL,
|
||||
uuid TEXT NOT NULL,
|
||||
name TEXT NOT NULL,
|
||||
memory_total_gb DOUBLE PRECISION NOT NULL,
|
||||
power_limit_w DOUBLE PRECISION NOT NULL,
|
||||
base_temperature INTEGER NOT NULL,
|
||||
last_seen_at TEXT
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS fine_tune_tasks (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL UNIQUE,
|
||||
payload TEXT NOT NULL,
|
||||
status TEXT NOT NULL,
|
||||
progress INTEGER NOT NULL DEFAULT 0,
|
||||
process_id INTEGER,
|
||||
create_time TEXT NOT NULL,
|
||||
start_time TEXT,
|
||||
completed_at TEXT,
|
||||
compute_node_id TEXT REFERENCES compute_nodes(id) ON DELETE SET NULL,
|
||||
gpus TEXT NOT NULL,
|
||||
sync_job_id TEXT,
|
||||
compute_job_id TEXT
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS fine_tune_metrics (
|
||||
id TEXT PRIMARY KEY,
|
||||
task_id TEXT NOT NULL REFERENCES fine_tune_tasks(id) ON DELETE CASCADE,
|
||||
step INTEGER NOT NULL,
|
||||
epoch DOUBLE PRECISION,
|
||||
loss DOUBLE PRECISION,
|
||||
grad_norm DOUBLE PRECISION,
|
||||
learning_rate DOUBLE PRECISION,
|
||||
raw TEXT NOT NULL,
|
||||
create_time TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS fine_tune_checkpoints (
|
||||
id TEXT PRIMARY KEY,
|
||||
task_id TEXT NOT NULL REFERENCES fine_tune_tasks(id) ON DELETE CASCADE,
|
||||
step INTEGER NOT NULL,
|
||||
name TEXT NOT NULL,
|
||||
path TEXT NOT NULL,
|
||||
size_bytes BIGINT NOT NULL DEFAULT 0,
|
||||
create_time TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS compute_jobs (
|
||||
id TEXT PRIMARY KEY,
|
||||
task_id TEXT REFERENCES fine_tune_tasks(id) ON DELETE SET NULL,
|
||||
node_id TEXT REFERENCES compute_nodes(id) ON DELETE SET NULL,
|
||||
engine TEXT NOT NULL,
|
||||
status TEXT NOT NULL,
|
||||
command TEXT NOT NULL,
|
||||
output_dir TEXT,
|
||||
log_file TEXT,
|
||||
payload TEXT NOT NULL,
|
||||
create_time TEXT NOT NULL,
|
||||
update_time TEXT NOT NULL,
|
||||
completed_at TEXT
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS gpu_allocations (
|
||||
id TEXT PRIMARY KEY,
|
||||
task_id TEXT REFERENCES fine_tune_tasks(id) ON DELETE CASCADE,
|
||||
compute_job_id TEXT,
|
||||
node_id TEXT REFERENCES compute_nodes(id) ON DELETE CASCADE,
|
||||
gpu_index INTEGER NOT NULL,
|
||||
status TEXT NOT NULL,
|
||||
create_time TEXT NOT NULL,
|
||||
released_at TEXT
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS scheduler_locks (
|
||||
lock_key TEXT PRIMARY KEY,
|
||||
owner TEXT NOT NULL,
|
||||
expires_at TEXT NOT NULL,
|
||||
create_time TEXT NOT NULL,
|
||||
update_time TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS resource_replicas (
|
||||
id TEXT PRIMARY KEY,
|
||||
node_id TEXT NOT NULL REFERENCES compute_nodes(id) ON DELETE CASCADE,
|
||||
resource_type TEXT NOT NULL,
|
||||
resource_id TEXT NOT NULL,
|
||||
local_path TEXT NOT NULL,
|
||||
status TEXT NOT NULL,
|
||||
sync_status TEXT NOT NULL,
|
||||
checksum_sha256 TEXT,
|
||||
byte_size BIGINT NOT NULL DEFAULT 0,
|
||||
last_checked_at TEXT,
|
||||
last_error TEXT,
|
||||
create_time TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS resource_sync_jobs (
|
||||
id TEXT PRIMARY KEY,
|
||||
target_node_id TEXT NOT NULL,
|
||||
resources TEXT NOT NULL,
|
||||
status TEXT NOT NULL,
|
||||
progress INTEGER NOT NULL DEFAULT 0,
|
||||
create_time TEXT NOT NULL,
|
||||
completed_at TEXT
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS eval_tasks (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
payload TEXT NOT NULL,
|
||||
status TEXT NOT NULL,
|
||||
create_time TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS eval_dimensions (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
payload TEXT NOT NULL,
|
||||
is_active INTEGER NOT NULL DEFAULT 1,
|
||||
is_default INTEGER NOT NULL DEFAULT 0,
|
||||
create_time TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS compare_tasks (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
payload TEXT NOT NULL,
|
||||
status TEXT NOT NULL,
|
||||
create_time TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_fine_tune_status ON fine_tune_tasks(status);
|
||||
CREATE INDEX IF NOT EXISTS idx_fine_tune_compute_job ON fine_tune_tasks(compute_job_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_fine_tune_compute_node_status ON fine_tune_tasks(compute_node_id, status);
|
||||
CREATE INDEX IF NOT EXISTS idx_model_lineage_child ON model_lineage(child_resource_type, child_resource_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_model_lineage_parent ON model_lineage(parent_resource_type, parent_resource_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_model_artifacts_model ON model_artifacts(model_kind, model_id, artifact_type);
|
||||
CREATE INDEX IF NOT EXISTS idx_model_export_jobs_model ON model_export_jobs(trained_model_id, create_time DESC);
|
||||
CREATE INDEX IF NOT EXISTS idx_model_export_jobs_compute ON model_export_jobs(compute_job_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_fine_tune_metrics_task_step ON fine_tune_metrics(task_id, step);
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS uq_fine_tune_metrics_task_step_epoch ON fine_tune_metrics(task_id, step, epoch);
|
||||
CREATE INDEX IF NOT EXISTS idx_fine_tune_checkpoints_task_step ON fine_tune_checkpoints(task_id, step);
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS uq_fine_tune_checkpoints_task_path ON fine_tune_checkpoints(task_id, path);
|
||||
CREATE INDEX IF NOT EXISTS idx_compute_jobs_task ON compute_jobs(task_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_compute_jobs_node_status ON compute_jobs(node_id, status);
|
||||
CREATE INDEX IF NOT EXISTS idx_gpu_allocations_node_status ON gpu_allocations(node_id, status);
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS uq_gpu_allocations_active ON gpu_allocations(node_id, gpu_index) WHERE status IN ('allocated','running');
|
||||
CREATE INDEX IF NOT EXISTS idx_scheduler_locks_expires ON scheduler_locks(expires_at);
|
||||
CREATE INDEX IF NOT EXISTS idx_dataset_files_dataset ON dataset_files(dataset_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_gpus_node ON gpus(node_id);
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS uq_gpus_node_index ON gpus(node_id, gpu_index);
|
||||
CREATE INDEX IF NOT EXISTS idx_replicas_resource ON resource_replicas(resource_type, resource_id);
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS uq_replicas_node_resource ON resource_replicas(node_id, resource_type, resource_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_sync_jobs_node_status ON resource_sync_jobs(target_node_id, status);
|
||||
CREATE INDEX IF NOT EXISTS idx_eval_tasks_status ON eval_tasks(status);
|
||||
CREATE INDEX IF NOT EXISTS idx_eval_dimensions_active ON eval_dimensions(is_active);
|
||||
CREATE INDEX IF NOT EXISTS idx_compare_tasks_status ON compare_tasks(status);
|
||||
|
||||
-- ===================== Project / Tenant =====================
|
||||
|
||||
CREATE TABLE IF NOT EXISTS projects (
|
||||
id TEXT PRIMARY KEY,
|
||||
tenant_id TEXT NOT NULL DEFAULT 'default',
|
||||
name TEXT NOT NULL,
|
||||
code TEXT NOT NULL,
|
||||
description TEXT,
|
||||
quota TEXT,
|
||||
status TEXT NOT NULL DEFAULT 'active',
|
||||
create_time TEXT NOT NULL,
|
||||
create_by TEXT,
|
||||
updated_at TEXT
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS project_members (
|
||||
project_id TEXT NOT NULL REFERENCES projects(id) ON DELETE CASCADE,
|
||||
user_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||
role TEXT NOT NULL DEFAULT 'member',
|
||||
create_time TEXT NOT NULL,
|
||||
PRIMARY KEY (project_id, user_id)
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS roles (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
permissions TEXT NOT NULL DEFAULT '[]',
|
||||
create_time TEXT
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS sessions (
|
||||
id TEXT PRIMARY KEY,
|
||||
user_id TEXT NOT NULL,
|
||||
issued_at TEXT NOT NULL,
|
||||
expires_at TEXT NOT NULL,
|
||||
ip TEXT
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS acls (
|
||||
id TEXT PRIMARY KEY,
|
||||
resource_type TEXT NOT NULL,
|
||||
resource_id TEXT NOT NULL,
|
||||
principal_type TEXT NOT NULL,
|
||||
principal_id TEXT NOT NULL,
|
||||
permission TEXT NOT NULL,
|
||||
create_time TEXT
|
||||
);
|
||||
@@ -1,343 +0,0 @@
|
||||
-- Data processing migration.
|
||||
--
|
||||
-- IMPORTANT: This file is intentionally NOT wired into application startup.
|
||||
-- Apply it explicitly in a controlled deployment, or call
|
||||
-- DataProcessStore.ensure_schema() from an administrative command.
|
||||
|
||||
BEGIN;
|
||||
|
||||
-- This migration targets the current runtime schema created by
|
||||
-- 001_platform_runtime.sql. Refuse the UUID/JSONB target-design schema instead
|
||||
-- of partially altering it with incompatible TEXT foreign keys.
|
||||
DO $$
|
||||
DECLARE
|
||||
datasets_id_type TEXT;
|
||||
BEGIN
|
||||
SELECT format_type(a.atttypid, a.atttypmod)
|
||||
INTO datasets_id_type
|
||||
FROM pg_attribute a
|
||||
JOIN pg_class c ON c.oid = a.attrelid
|
||||
JOIN pg_namespace n ON n.oid = c.relnamespace
|
||||
WHERE n.nspname = current_schema()
|
||||
AND c.relname = 'datasets'
|
||||
AND a.attname = 'id'
|
||||
AND a.attnum > 0
|
||||
AND NOT a.attisdropped;
|
||||
IF datasets_id_type IS NULL THEN
|
||||
RAISE EXCEPTION '002_data_process.sql requires 001_platform_runtime.sql first';
|
||||
END IF;
|
||||
IF datasets_id_type <> 'text' THEN
|
||||
RAISE EXCEPTION
|
||||
'002_data_process.sql supports only the current TEXT runtime schema; found datasets.id type %',
|
||||
datasets_id_type;
|
||||
END IF;
|
||||
END $$;
|
||||
|
||||
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS source_task_id TEXT;
|
||||
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS size_bytes BIGINT NOT NULL DEFAULT 0;
|
||||
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS record_count BIGINT NOT NULL DEFAULT 0;
|
||||
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS metadata TEXT NOT NULL DEFAULT '{}';
|
||||
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS tenant_id TEXT;
|
||||
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS project_id TEXT;
|
||||
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS owner_id TEXT;
|
||||
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS created_by TEXT;
|
||||
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS created_at TIMESTAMPTZ NOT NULL DEFAULT now();
|
||||
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS updated_at TIMESTAMPTZ NOT NULL DEFAULT now();
|
||||
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS deleted_at TIMESTAMPTZ;
|
||||
|
||||
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS storage_object_id TEXT;
|
||||
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS current_version_id TEXT;
|
||||
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS size_bytes BIGINT NOT NULL DEFAULT 0;
|
||||
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS record_count BIGINT NOT NULL DEFAULT 0;
|
||||
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS file_format VARCHAR(40);
|
||||
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS checksum_sha256 CHAR(64);
|
||||
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS version_no INTEGER NOT NULL DEFAULT 1;
|
||||
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS source_task_id TEXT;
|
||||
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS tenant_id TEXT;
|
||||
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS project_id TEXT;
|
||||
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS created_by TEXT;
|
||||
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS metadata TEXT NOT NULL DEFAULT '{}';
|
||||
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS created_at TIMESTAMPTZ NOT NULL DEFAULT now();
|
||||
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS updated_at TIMESTAMPTZ NOT NULL DEFAULT now();
|
||||
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS deleted_at TIMESTAMPTZ;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS data_process_tasks (
|
||||
id TEXT PRIMARY KEY,
|
||||
name VARCHAR(150) NOT NULL,
|
||||
description TEXT,
|
||||
status VARCHAR(20) NOT NULL DEFAULT 'pending'
|
||||
CHECK (status IN ('pending', 'running', 'completed', 'failed', 'stopped')),
|
||||
process_type VARCHAR(20) NOT NULL
|
||||
CHECK (process_type IN ('structured', 'unstructured', 'external')),
|
||||
source_dataset_id TEXT REFERENCES datasets(id) ON DELETE SET NULL,
|
||||
output_dataset_id TEXT REFERENCES datasets(id) ON DELETE SET NULL,
|
||||
config TEXT NOT NULL DEFAULT '{}',
|
||||
progress NUMERIC(5,2) NOT NULL DEFAULT 0 CHECK (progress >= 0 AND progress <= 100),
|
||||
input_count BIGINT NOT NULL DEFAULT 0 CHECK (input_count >= 0),
|
||||
output_count BIGINT NOT NULL DEFAULT 0 CHECK (output_count >= 0),
|
||||
filtered_count BIGINT NOT NULL DEFAULT 0 CHECK (filtered_count >= 0),
|
||||
duplicate_count BIGINT NOT NULL DEFAULT 0 CHECK (duplicate_count >= 0),
|
||||
error_count BIGINT NOT NULL DEFAULT 0 CHECK (error_count >= 0),
|
||||
failure_reason TEXT,
|
||||
generation_run_id TEXT,
|
||||
results_confirmed BOOLEAN NOT NULL DEFAULT TRUE,
|
||||
workflow_step VARCHAR(20) NOT NULL DEFAULT 'create'
|
||||
CHECK (workflow_step IN ('create', 'model', 'upload', 'preview', 'generate', 'results')),
|
||||
preview_status VARCHAR(20) NOT NULL DEFAULT 'idle'
|
||||
CHECK (preview_status IN ('idle', 'queued', 'running', 'completed', 'failed', 'cancelled')),
|
||||
preview_progress NUMERIC(5,2) NOT NULL DEFAULT 0
|
||||
CHECK (preview_progress >= 0 AND preview_progress <= 100),
|
||||
preview_run_id TEXT,
|
||||
preview_failure_reason TEXT,
|
||||
preview_total_files INTEGER NOT NULL DEFAULT 0 CHECK (preview_total_files >= 0),
|
||||
preview_completed_files INTEGER NOT NULL DEFAULT 0 CHECK (preview_completed_files >= 0),
|
||||
tenant_id TEXT,
|
||||
project_id TEXT,
|
||||
owner_id TEXT,
|
||||
approval_status VARCHAR(30) NOT NULL DEFAULT 'not_required',
|
||||
created_by TEXT,
|
||||
updated_by TEXT,
|
||||
deleted_by TEXT,
|
||||
started_at TIMESTAMPTZ,
|
||||
completed_at TIMESTAMPTZ,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
deleted_at TIMESTAMPTZ
|
||||
);
|
||||
|
||||
ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS generation_run_id TEXT;
|
||||
-- 历史任务在引入六步确认流程前已经完成审核，默认保留为已确认；
|
||||
-- 新任务由创建接口显式写入 FALSE，并在第六步确认后转为 TRUE。
|
||||
ALTER TABLE data_process_tasks
|
||||
ADD COLUMN IF NOT EXISTS results_confirmed BOOLEAN NOT NULL DEFAULT TRUE;
|
||||
UPDATE data_process_tasks
|
||||
SET results_confirmed=FALSE
|
||||
WHERE status <> 'completed' AND results_confirmed=TRUE;
|
||||
|
||||
-- 先以可空列接入旧库，才能只回填历史行；随后再收紧默认值与约束。
|
||||
ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS workflow_step VARCHAR(20);
|
||||
ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_status VARCHAR(20);
|
||||
ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_progress NUMERIC(5,2);
|
||||
ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_run_id TEXT;
|
||||
ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_failure_reason TEXT;
|
||||
ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_total_files INTEGER;
|
||||
ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_completed_files INTEGER;
|
||||
|
||||
CREATE TEMP TABLE data_process_workflow_backfill_ids ON COMMIT DROP AS
|
||||
SELECT id FROM data_process_tasks WHERE workflow_step IS NULL;
|
||||
|
||||
UPDATE data_process_tasks task
|
||||
SET workflow_step = CASE
|
||||
WHEN task.status IN ('running', 'failed', 'stopped') THEN 'generate'
|
||||
WHEN task.status = 'completed' AND task.results_confirmed=FALSE THEN 'generate'
|
||||
WHEN task.status = 'completed' THEN 'results'
|
||||
ELSE 'create'
|
||||
END
|
||||
WHERE task.workflow_step IS NULL;
|
||||
UPDATE data_process_tasks
|
||||
SET preview_status='idle', preview_progress=0,
|
||||
preview_total_files=0, preview_completed_files=0
|
||||
WHERE preview_status IS NULL OR preview_progress IS NULL
|
||||
OR preview_total_files IS NULL OR preview_completed_files IS NULL;
|
||||
|
||||
ALTER TABLE data_process_tasks ALTER COLUMN workflow_step SET DEFAULT 'create';
|
||||
ALTER TABLE data_process_tasks ALTER COLUMN workflow_step SET NOT NULL;
|
||||
ALTER TABLE data_process_tasks ALTER COLUMN preview_status SET DEFAULT 'idle';
|
||||
ALTER TABLE data_process_tasks ALTER COLUMN preview_status SET NOT NULL;
|
||||
ALTER TABLE data_process_tasks ALTER COLUMN preview_progress SET DEFAULT 0;
|
||||
ALTER TABLE data_process_tasks ALTER COLUMN preview_progress SET NOT NULL;
|
||||
ALTER TABLE data_process_tasks ALTER COLUMN preview_total_files SET DEFAULT 0;
|
||||
ALTER TABLE data_process_tasks ALTER COLUMN preview_total_files SET NOT NULL;
|
||||
ALTER TABLE data_process_tasks ALTER COLUMN preview_completed_files SET DEFAULT 0;
|
||||
ALTER TABLE data_process_tasks ALTER COLUMN preview_completed_files SET NOT NULL;
|
||||
|
||||
DO $$
|
||||
BEGIN
|
||||
IF NOT EXISTS (
|
||||
SELECT 1 FROM pg_constraint
|
||||
WHERE conrelid='data_process_tasks'::regclass
|
||||
AND conname='ck_data_process_tasks_workflow_step'
|
||||
) THEN
|
||||
ALTER TABLE data_process_tasks ADD CONSTRAINT ck_data_process_tasks_workflow_step
|
||||
CHECK (workflow_step IN ('create', 'model', 'upload', 'preview', 'generate', 'results'));
|
||||
END IF;
|
||||
IF NOT EXISTS (
|
||||
SELECT 1 FROM pg_constraint
|
||||
WHERE conrelid='data_process_tasks'::regclass
|
||||
AND conname='ck_data_process_tasks_preview_status'
|
||||
) THEN
|
||||
ALTER TABLE data_process_tasks ADD CONSTRAINT ck_data_process_tasks_preview_status
|
||||
CHECK (preview_status IN ('idle', 'queued', 'running', 'completed', 'failed', 'cancelled'));
|
||||
END IF;
|
||||
IF NOT EXISTS (
|
||||
SELECT 1 FROM pg_constraint
|
||||
WHERE conrelid='data_process_tasks'::regclass
|
||||
AND conname='ck_data_process_tasks_preview_progress'
|
||||
) THEN
|
||||
ALTER TABLE data_process_tasks ADD CONSTRAINT ck_data_process_tasks_preview_progress
|
||||
CHECK (preview_progress >= 0 AND preview_progress <= 100);
|
||||
END IF;
|
||||
IF NOT EXISTS (
|
||||
SELECT 1 FROM pg_constraint
|
||||
WHERE conrelid='data_process_tasks'::regclass
|
||||
AND conname='ck_data_process_tasks_preview_file_counts'
|
||||
) THEN
|
||||
ALTER TABLE data_process_tasks ADD CONSTRAINT ck_data_process_tasks_preview_file_counts
|
||||
CHECK (preview_total_files >= 0 AND preview_completed_files >= 0
|
||||
AND preview_completed_files <= preview_total_files);
|
||||
END IF;
|
||||
END $$;
|
||||
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS uq_data_process_tasks_name_alive
|
||||
ON data_process_tasks(name) WHERE deleted_at IS NULL;
|
||||
CREATE INDEX IF NOT EXISTS idx_data_process_tasks_scope_status
|
||||
ON data_process_tasks(tenant_id, project_id, status, created_at DESC)
|
||||
WHERE deleted_at IS NULL;
|
||||
CREATE INDEX IF NOT EXISTS idx_data_process_tasks_creator_created
|
||||
ON data_process_tasks(created_by, created_at DESC) WHERE deleted_at IS NULL;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS data_process_source_files (
|
||||
id TEXT PRIMARY KEY,
|
||||
task_id TEXT NOT NULL REFERENCES data_process_tasks(id) ON DELETE CASCADE,
|
||||
storage_object_id TEXT,
|
||||
name TEXT NOT NULL,
|
||||
size_bytes BIGINT NOT NULL DEFAULT 0 CHECK (size_bytes >= 0),
|
||||
record_count BIGINT NOT NULL DEFAULT 0 CHECK (record_count >= 0),
|
||||
file_format VARCHAR(40),
|
||||
checksum_sha256 CHAR(64) NOT NULL,
|
||||
version_no INTEGER NOT NULL DEFAULT 1 CHECK (version_no > 0),
|
||||
content TEXT NOT NULL,
|
||||
content_preview TEXT,
|
||||
metadata TEXT NOT NULL DEFAULT '{}',
|
||||
tenant_id TEXT,
|
||||
project_id TEXT,
|
||||
created_by TEXT,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
deleted_at TIMESTAMPTZ
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_data_process_source_files_task
|
||||
ON data_process_source_files(task_id, created_at) WHERE deleted_at IS NULL;
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS uq_data_process_source_checksum_alive
|
||||
ON data_process_source_files(task_id, checksum_sha256) WHERE deleted_at IS NULL;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS data_process_preview_items (
|
||||
id TEXT PRIMARY KEY,
|
||||
task_id TEXT NOT NULL REFERENCES data_process_tasks(id) ON DELETE CASCADE,
|
||||
source_file_id TEXT REFERENCES data_process_source_files(id) ON DELETE CASCADE,
|
||||
original_content TEXT NOT NULL DEFAULT '',
|
||||
edited_content TEXT NOT NULL DEFAULT '',
|
||||
source_start INTEGER CHECK (source_start IS NULL OR source_start >= 0),
|
||||
source_end INTEGER CHECK (source_end IS NULL OR source_end >= 0),
|
||||
source_start_line INTEGER CHECK (source_start_line IS NULL OR source_start_line > 0),
|
||||
source_end_line INTEGER CHECK (source_end_line IS NULL OR source_end_line > 0),
|
||||
token_count INTEGER NOT NULL DEFAULT 0 CHECK (token_count >= 0),
|
||||
status VARCHAR(20) NOT NULL DEFAULT 'original'
|
||||
CHECK (status IN ('original', 'modified', 'manual', 'invalid')),
|
||||
quality_score TEXT NOT NULL DEFAULT '{}',
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
CHECK (source_start IS NULL OR source_end IS NULL OR source_end >= source_start),
|
||||
CHECK (source_start_line IS NULL OR source_end_line IS NULL OR source_end_line >= source_start_line)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_data_process_preview_task_file
|
||||
ON data_process_preview_items(task_id, source_file_id, created_at);
|
||||
|
||||
-- 子表在新库中到这里才存在；只修复本次新增 workflow_step 前的历史任务。
|
||||
UPDATE data_process_tasks task
|
||||
SET workflow_step = CASE
|
||||
WHEN EXISTS (
|
||||
SELECT 1 FROM data_process_preview_items preview
|
||||
WHERE preview.task_id=task.id
|
||||
) THEN 'preview'
|
||||
WHEN EXISTS (
|
||||
SELECT 1 FROM data_process_source_files source_file
|
||||
WHERE source_file.task_id=task.id AND source_file.deleted_at IS NULL
|
||||
) THEN 'upload'
|
||||
ELSE task.workflow_step
|
||||
END
|
||||
WHERE task.id IN (SELECT id FROM data_process_workflow_backfill_ids)
|
||||
AND task.status='pending';
|
||||
|
||||
CREATE TABLE IF NOT EXISTS data_process_results (
|
||||
id TEXT PRIMARY KEY,
|
||||
task_id TEXT NOT NULL REFERENCES data_process_tasks(id) ON DELETE CASCADE,
|
||||
preview_item_id TEXT REFERENCES data_process_preview_items(id) ON DELETE SET NULL,
|
||||
instruction TEXT NOT NULL,
|
||||
input TEXT NOT NULL DEFAULT '',
|
||||
output TEXT NOT NULL,
|
||||
original_instruction TEXT,
|
||||
original_input TEXT,
|
||||
original_output TEXT,
|
||||
status VARCHAR(20) NOT NULL DEFAULT 'valid'
|
||||
CHECK (status IN ('valid', 'modified', 'invalid')),
|
||||
error TEXT,
|
||||
split VARCHAR(20) CHECK (split IS NULL OR split IN ('train', 'validation', 'test')),
|
||||
quality_score TEXT NOT NULL DEFAULT '{}',
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_data_process_results_task_status
|
||||
ON data_process_results(task_id, status, id);
|
||||
CREATE INDEX IF NOT EXISTS idx_data_process_results_task_split
|
||||
ON data_process_results(task_id, split);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS dataset_file_versions (
|
||||
id TEXT PRIMARY KEY,
|
||||
dataset_file_id TEXT NOT NULL REFERENCES dataset_files(id) ON DELETE CASCADE,
|
||||
version_no INTEGER NOT NULL CHECK (version_no > 0),
|
||||
storage_object_id TEXT NOT NULL,
|
||||
content_preview TEXT,
|
||||
description TEXT,
|
||||
base_version_id TEXT REFERENCES dataset_file_versions(id) ON DELETE SET NULL,
|
||||
size_bytes BIGINT NOT NULL DEFAULT 0 CHECK (size_bytes >= 0),
|
||||
record_count BIGINT NOT NULL DEFAULT 0 CHECK (record_count >= 0),
|
||||
checksum_sha256 CHAR(64) NOT NULL,
|
||||
source_task_id TEXT REFERENCES data_process_tasks(id) ON DELETE SET NULL,
|
||||
metadata TEXT NOT NULL DEFAULT '{}',
|
||||
created_by TEXT,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
ALTER TABLE dataset_file_versions ADD COLUMN IF NOT EXISTS source_task_id TEXT;
|
||||
ALTER TABLE dataset_file_versions ADD COLUMN IF NOT EXISTS metadata TEXT NOT NULL DEFAULT '{}';
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS uq_dataset_file_versions_no_002
|
||||
ON dataset_file_versions(dataset_file_id, version_no);
|
||||
CREATE INDEX IF NOT EXISTS idx_dataset_file_versions_source_task_002
|
||||
ON dataset_file_versions(source_task_id) WHERE source_task_id IS NOT NULL;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS dataset_records (
|
||||
id TEXT PRIMARY KEY,
|
||||
dataset_id TEXT NOT NULL REFERENCES datasets(id) ON DELETE CASCADE,
|
||||
dataset_file_id TEXT REFERENCES dataset_files(id) ON DELETE CASCADE,
|
||||
version_id TEXT REFERENCES dataset_file_versions(id) ON DELETE CASCADE,
|
||||
line_no INTEGER,
|
||||
split VARCHAR(20) CHECK (split IS NULL OR split IN ('train', 'validation', 'test')),
|
||||
instruction TEXT,
|
||||
input TEXT,
|
||||
output TEXT,
|
||||
raw TEXT NOT NULL DEFAULT '{}',
|
||||
status VARCHAR(20) NOT NULL DEFAULT 'valid'
|
||||
CHECK (status IN ('valid', 'modified', 'invalid')),
|
||||
source_task_id TEXT REFERENCES data_process_tasks(id) ON DELETE SET NULL,
|
||||
source_result_id TEXT REFERENCES data_process_results(id) ON DELETE SET NULL,
|
||||
preview_item_id TEXT REFERENCES data_process_preview_items(id) ON DELETE SET NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
ALTER TABLE dataset_records ADD COLUMN IF NOT EXISTS source_task_id TEXT;
|
||||
ALTER TABLE dataset_records ADD COLUMN IF NOT EXISTS source_result_id TEXT;
|
||||
ALTER TABLE dataset_records ADD COLUMN IF NOT EXISTS preview_item_id TEXT;
|
||||
CREATE INDEX IF NOT EXISTS idx_dataset_records_dataset_002
|
||||
ON dataset_records(dataset_id, id);
|
||||
CREATE INDEX IF NOT EXISTS idx_dataset_records_source_task_002
|
||||
ON dataset_records(source_task_id, source_result_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_datasets_source_task_002
|
||||
ON datasets(source_task_id) WHERE source_task_id IS NOT NULL;
|
||||
CREATE INDEX IF NOT EXISTS idx_dataset_files_source_task_002
|
||||
ON dataset_files(source_task_id) WHERE source_task_id IS NOT NULL;
|
||||
|
||||
COMMIT;
|
||||
@@ -1,68 +0,0 @@
|
||||
-- 平台治理：租户 / 审批 / 审计（字段以 platform_store 实际写入为准）
|
||||
CREATE TABLE IF NOT EXISTS tenants (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
code TEXT,
|
||||
status TEXT DEFAULT 'active',
|
||||
owner_user_id TEXT,
|
||||
quota TEXT,
|
||||
retention_policy_id TEXT,
|
||||
create_time TEXT
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS approval_templates (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
steps TEXT,
|
||||
create_time TEXT
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS approval_instances (
|
||||
id TEXT PRIMARY KEY,
|
||||
template_id TEXT,
|
||||
resource_type TEXT,
|
||||
resource_id TEXT,
|
||||
applicant_id TEXT,
|
||||
status TEXT DEFAULT 'pending',
|
||||
current_step INTEGER DEFAULT 0,
|
||||
create_time TEXT
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS approval_steps (
|
||||
id TEXT PRIMARY KEY,
|
||||
instance_id TEXT,
|
||||
step_index INTEGER,
|
||||
approver_id TEXT,
|
||||
status TEXT DEFAULT 'pending',
|
||||
comment TEXT,
|
||||
time TEXT
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS audit_logs (
|
||||
id TEXT PRIMARY KEY,
|
||||
tenant_id TEXT,
|
||||
project_id TEXT,
|
||||
actor_id TEXT,
|
||||
action TEXT,
|
||||
target_type TEXT,
|
||||
target_id TEXT,
|
||||
detail TEXT,
|
||||
client_ip TEXT,
|
||||
time TEXT
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_audit_tenant ON audit_logs(tenant_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_audit_project ON audit_logs(project_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_audit_action ON audit_logs(action);
|
||||
CREATE INDEX IF NOT EXISTS idx_audit_time ON audit_logs(time);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS retention_policies (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
scope TEXT,
|
||||
rule TEXT,
|
||||
status TEXT DEFAULT 'active',
|
||||
create_time TEXT,
|
||||
create_by TEXT,
|
||||
updated_at TEXT
|
||||
);
|
||||
@@ -1,18 +0,0 @@
|
||||
-- 003_model_path_governance
|
||||
-- 模型路径治理：增加 can_train 标识，区分本地可训练模型与 API / 远程模型。
|
||||
-- 训练预检阶段依赖该字段拦截不适合 LLaMA-Factory 本地训练的基座模型。
|
||||
|
||||
-- 1. models 表增加 can_train（默认 0，后设搬迁为 1 的规则如下）
|
||||
ALTER TABLE models ADD COLUMN IF NOT EXISTS can_train INTEGER NOT NULL DEFAULT 0;
|
||||
|
||||
-- 2. 将已有模型按规则推定 can_train：
|
||||
-- - path 非空 且 model_source != 'api' → 可训练 (1)
|
||||
-- - 其余 → 不可训练 (0)
|
||||
UPDATE models
|
||||
SET can_train = CASE
|
||||
WHEN path IS NOT NULL AND path != '' AND model_source IS NOT NULL AND model_source != 'api' THEN 1
|
||||
ELSE 0
|
||||
END;
|
||||
|
||||
-- 3. 给 trained_models 增加 artifact_dir（训练产物目录扫描结果目录）
|
||||
ALTER TABLE trained_models ADD COLUMN IF NOT EXISTS artifact_dir TEXT;
|
||||
@@ -1,3 +0,0 @@
|
||||
-- 租户配额与保留策略扩展（如后续治理表需补列，可在此追加）
|
||||
ALTER TABLE tenants ADD COLUMN IF NOT EXISTS gpu_quota TEXT;
|
||||
ALTER TABLE tenants ADD COLUMN IF NOT EXISTS storage_quota TEXT;
|
||||
@@ -1,13 +1,8 @@
|
||||
import asyncio
|
||||
from contextlib import suppress
|
||||
|
||||
from fastapi import FastAPI
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
|
||||
from app.api.v1.router import api_router
|
||||
from app.core.config import get_settings
|
||||
from app.core.logging import configure_logging, setup_request_logging
|
||||
from app.workers.compute_poller import run_compute_poller
|
||||
|
||||
|
||||
def create_app() -> FastAPI:
|
||||
@@ -15,28 +10,8 @@ def create_app() -> FastAPI:
|
||||
configure_logging(settings)
|
||||
|
||||
app = FastAPI(title=settings.app_name)
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=settings.cors_allow_origins,
|
||||
allow_credentials=True,
|
||||
allow_methods=["*"],
|
||||
allow_headers=["*"],
|
||||
)
|
||||
setup_request_logging(app)
|
||||
app.include_router(api_router, prefix=settings.route_prefix)
|
||||
|
||||
@app.on_event("startup")
|
||||
async def start_workers() -> None:
|
||||
app.state.compute_poller_task = asyncio.create_task(run_compute_poller())
|
||||
|
||||
@app.on_event("shutdown")
|
||||
async def stop_workers() -> None:
|
||||
task = getattr(app.state, "compute_poller_task", None)
|
||||
if task:
|
||||
task.cancel()
|
||||
with suppress(asyncio.CancelledError):
|
||||
await task
|
||||
|
||||
app.include_router(api_router, prefix=settings.api_prefix)
|
||||
return app
|
||||
|
||||
|
||||
|
||||
@@ -1,91 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from fastapi import APIRouter, Body
|
||||
from typing import Any
|
||||
|
||||
from app.api.v1.endpoints.platform import ok, fail
|
||||
from app.db.platform_store import get_platform_store
|
||||
|
||||
router = APIRouter(prefix="/approvals", tags=["approval"])
|
||||
|
||||
|
||||
@router.get("/templates")
|
||||
def list_templates() -> dict[str, Any]:
|
||||
return ok(get_platform_store().approval_templates())
|
||||
|
||||
|
||||
@router.post("/templates")
|
||||
def create_template(payload: dict[str, Any] = Body(...)) -> dict[str, Any]:
|
||||
if not payload.get("name"):
|
||||
raise fail(400, "name 必填")
|
||||
return ok(get_platform_store().create_approval_template(payload))
|
||||
|
||||
|
||||
@router.get("/templates/{template_id}")
|
||||
def get_template(template_id: str) -> dict[str, Any]:
|
||||
try:
|
||||
return ok(get_platform_store().approval_template(template_id))
|
||||
except KeyError:
|
||||
raise fail(404, "template not found")
|
||||
|
||||
|
||||
@router.put("/templates/{template_id}")
|
||||
def update_template(template_id: str, payload: dict[str, Any] = Body(...)) -> dict[str, Any]:
|
||||
try:
|
||||
return ok(get_platform_store().update_approval_template(template_id, payload))
|
||||
except KeyError:
|
||||
raise fail(404, "template not found")
|
||||
|
||||
|
||||
@router.delete("/templates/{template_id}")
|
||||
def delete_template(template_id: str) -> dict[str, Any]:
|
||||
try:
|
||||
return ok(get_platform_store().delete_approval_template(template_id))
|
||||
except KeyError:
|
||||
raise fail(404, "template not found")
|
||||
|
||||
|
||||
@router.get("")
|
||||
def list_instances(status: str | None = None) -> dict[str, Any]:
|
||||
return ok(get_platform_store().approval_instances(status=status))
|
||||
|
||||
|
||||
@router.post("")
|
||||
def create_instance(payload: dict[str, Any] = Body(...)) -> dict[str, Any]:
|
||||
for field in ("resource_type", "resource_id", "applicant_id"):
|
||||
if not payload.get(field):
|
||||
raise fail(400, f"{field} 必填")
|
||||
try:
|
||||
return ok(get_platform_store().create_approval_instance(payload))
|
||||
except KeyError:
|
||||
raise fail(404, "template not found")
|
||||
|
||||
|
||||
@router.get("/{instance_id}")
|
||||
def get_instance(instance_id: str) -> dict[str, Any]:
|
||||
try:
|
||||
return ok(get_platform_store().approval_instance(instance_id))
|
||||
except KeyError:
|
||||
raise fail(404, "instance not found")
|
||||
|
||||
|
||||
@router.post("/{instance_id}/steps/{step_index}/decision")
|
||||
def decide(
|
||||
instance_id: str,
|
||||
step_index: int,
|
||||
payload: dict[str, Any] = Body(...),
|
||||
) -> dict[str, Any]:
|
||||
if not payload.get("approver_id"):
|
||||
raise fail(400, "approver_id 必填")
|
||||
try:
|
||||
return ok(
|
||||
get_platform_store().decide_approval_step(
|
||||
instance_id,
|
||||
step_index,
|
||||
approver_id=payload["approver_id"],
|
||||
approved=bool(payload.get("approved", False)),
|
||||
comment=payload.get("comment"),
|
||||
)
|
||||
)
|
||||
except (KeyError, ValueError) as e:
|
||||
raise fail(400, str(e))
|
||||
@@ -1,247 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from typing import Any
|
||||
from urllib.parse import urljoin
|
||||
|
||||
import httpx
|
||||
|
||||
from app.core.config import get_settings
|
||||
|
||||
|
||||
def _join_url(base_url: str, path: str) -> str:
|
||||
return urljoin(base_url.rstrip("/") + "/", path.lstrip("/"))
|
||||
|
||||
|
||||
def _unwrap_items(payload: Any) -> list[dict[str, Any]]:
|
||||
if isinstance(payload, list):
|
||||
return [item for item in payload if isinstance(item, dict)]
|
||||
if isinstance(payload, dict):
|
||||
data = payload.get("data")
|
||||
if isinstance(data, dict) and isinstance(data.get("items"), list):
|
||||
return [item for item in data["items"] if isinstance(item, dict)]
|
||||
if isinstance(payload.get("items"), list):
|
||||
return [item for item in payload["items"] if isinstance(item, dict)]
|
||||
if isinstance(data, list):
|
||||
return [item for item in data if isinstance(item, dict)]
|
||||
return []
|
||||
|
||||
|
||||
def _unwrap_dict(payload: Any) -> dict[str, Any]:
|
||||
if isinstance(payload, dict) and isinstance(payload.get("data"), dict):
|
||||
return payload["data"]
|
||||
return payload if isinstance(payload, dict) else {}
|
||||
|
||||
|
||||
# Inference calls are intentionally short-timeout:
|
||||
# - load dispatch only confirms the compute node accepted the request
|
||||
# (the actual model load now runs asynchronously on the node).
|
||||
# - status/unload must never block the platform for long when a node is
|
||||
# unreachable but still marked online.
|
||||
INFERENCE_LOAD_TIMEOUT = httpx.Timeout(30, connect=10)
|
||||
INFERENCE_STATUS_TIMEOUT = httpx.Timeout(30, connect=5)
|
||||
INFERENCE_UNLOAD_TIMEOUT = httpx.Timeout(30, connect=5)
|
||||
|
||||
|
||||
class ComputeNodeClient:
|
||||
"""Application-side client for one compute node.
|
||||
|
||||
The client accepts both current YG Compute API responses and common
|
||||
wrapper shapes such as `{code,message,data}` to make future engine/node
|
||||
adapters less brittle.
|
||||
"""
|
||||
|
||||
def __init__(self, api_base_url: str, token: str | None = None, timeout: float | None = None) -> None:
|
||||
settings = get_settings()
|
||||
self.api_base_url = api_base_url.rstrip("/")
|
||||
self.token = token or settings.compute_service_token
|
||||
self.timeout = timeout or settings.compute_request_timeout_seconds
|
||||
self.route_prefix = settings.route_prefix.rstrip("/") or "/modelTF"
|
||||
|
||||
def headers(self) -> dict[str, str]:
|
||||
if not self.token:
|
||||
return {}
|
||||
return {"X-Compute-Token": self.token}
|
||||
|
||||
async def test_connection(self) -> dict[str, Any]:
|
||||
started = time.perf_counter()
|
||||
health = await self.health()
|
||||
gpus = await self.gpus()
|
||||
return {
|
||||
"success": True,
|
||||
"latency_ms": int((time.perf_counter() - started) * 1000),
|
||||
"health": health,
|
||||
"gpus": gpus,
|
||||
}
|
||||
|
||||
async def health(self) -> dict[str, Any]:
|
||||
paths = [f"{self.route_prefix}/v1/compute/health", f"{self.route_prefix}/health", "/health"]
|
||||
last_error = ""
|
||||
async with httpx.AsyncClient(timeout=self.timeout, headers=self.headers()) as client:
|
||||
for path in paths:
|
||||
try:
|
||||
response = await client.get(_join_url(self.api_base_url, path))
|
||||
response.raise_for_status()
|
||||
return _unwrap_dict(response.json())
|
||||
except Exception as exc: # noqa: BLE001 - keep endpoint compatibility fallback broad
|
||||
last_error = str(exc)
|
||||
raise RuntimeError(last_error or "compute health check failed")
|
||||
|
||||
async def gpus(self) -> list[dict[str, Any]]:
|
||||
paths = [
|
||||
f"{self.route_prefix}/compute/resources/gpus",
|
||||
f"{self.route_prefix}/v1/compute/resources/gpus",
|
||||
"/compute/resources/gpus",
|
||||
]
|
||||
last_error = ""
|
||||
async with httpx.AsyncClient(timeout=self.timeout, headers=self.headers()) as client:
|
||||
for path in paths:
|
||||
try:
|
||||
response = await client.get(_join_url(self.api_base_url, path))
|
||||
response.raise_for_status()
|
||||
return _unwrap_items(response.json())
|
||||
except Exception as exc: # noqa: BLE001
|
||||
last_error = str(exc)
|
||||
raise RuntimeError(last_error or "compute gpu discovery failed")
|
||||
|
||||
async def create_job(self, payload: dict[str, Any]) -> dict[str, Any]:
|
||||
async with httpx.AsyncClient(timeout=self.timeout, headers=self.headers()) as client:
|
||||
response = await client.post(_join_url(self.api_base_url, f"{self.route_prefix}/compute/jobs"), json=payload)
|
||||
response.raise_for_status()
|
||||
return _unwrap_dict(response.json())
|
||||
|
||||
async def preview_job(self, payload: dict[str, Any]) -> dict[str, Any]:
|
||||
async with httpx.AsyncClient(timeout=self.timeout, headers=self.headers()) as client:
|
||||
response = await client.post(
|
||||
_join_url(self.api_base_url, f"{self.route_prefix}/compute/jobs/preview"),
|
||||
json=payload,
|
||||
)
|
||||
response.raise_for_status()
|
||||
return _unwrap_dict(response.json())
|
||||
|
||||
async def validate_job(self, payload: dict[str, Any]) -> dict[str, Any]:
|
||||
async with httpx.AsyncClient(timeout=self.timeout, headers=self.headers()) as client:
|
||||
response = await client.post(
|
||||
_join_url(self.api_base_url, f"{self.route_prefix}/compute/jobs/validate"),
|
||||
json=payload,
|
||||
)
|
||||
response.raise_for_status()
|
||||
return _unwrap_dict(response.json())
|
||||
|
||||
async def check_paths(self, paths: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
async with httpx.AsyncClient(timeout=self.timeout, headers=self.headers()) as client:
|
||||
response = await client.post(
|
||||
_join_url(self.api_base_url, f"{self.route_prefix}/compute/files/check-paths"),
|
||||
json={"paths": paths},
|
||||
)
|
||||
response.raise_for_status()
|
||||
return _unwrap_dict(response.json())
|
||||
|
||||
async def list_files(
|
||||
self,
|
||||
root: str = "data",
|
||||
relative_path: str = "",
|
||||
directories_only: bool = False,
|
||||
) -> dict[str, Any]:
|
||||
async with httpx.AsyncClient(timeout=self.timeout, headers=self.headers()) as client:
|
||||
response = await client.get(
|
||||
_join_url(self.api_base_url, f"{self.route_prefix}/compute/files/list"),
|
||||
params={"root": root, "relative_path": relative_path, "directories_only": directories_only},
|
||||
)
|
||||
response.raise_for_status()
|
||||
return _unwrap_dict(response.json())
|
||||
|
||||
async def get_job(self, job_id: str) -> dict[str, Any]:
|
||||
async with httpx.AsyncClient(timeout=self.timeout, headers=self.headers()) as client:
|
||||
response = await client.get(_join_url(self.api_base_url, f"{self.route_prefix}/compute/jobs/{job_id}"))
|
||||
response.raise_for_status()
|
||||
return _unwrap_dict(response.json())
|
||||
|
||||
async def stop_job(self, job_id: str) -> dict[str, Any]:
|
||||
async with httpx.AsyncClient(timeout=self.timeout, headers=self.headers()) as client:
|
||||
response = await client.post(_join_url(self.api_base_url, f"{self.route_prefix}/compute/jobs/{job_id}/stop"))
|
||||
response.raise_for_status()
|
||||
return _unwrap_dict(response.json())
|
||||
|
||||
async def job_logs(
|
||||
self,
|
||||
job_id: str,
|
||||
tail_lines: int | None = None,
|
||||
offset: int | None = None,
|
||||
limit: int | None = None,
|
||||
) -> dict[str, Any]:
|
||||
params = {
|
||||
key: value
|
||||
for key, value in {"tail_lines": tail_lines, "offset": offset, "limit": limit}.items()
|
||||
if value is not None
|
||||
}
|
||||
async with httpx.AsyncClient(timeout=self.timeout, headers=self.headers()) as client:
|
||||
response = await client.get(
|
||||
_join_url(self.api_base_url, f"{self.route_prefix}/compute/jobs/{job_id}/logs"),
|
||||
params=params,
|
||||
)
|
||||
response.raise_for_status()
|
||||
return _unwrap_dict(response.json())
|
||||
|
||||
async def import_local_file(self, payload: dict[str, Any]) -> dict[str, Any]:
|
||||
async with httpx.AsyncClient(timeout=self.timeout, headers=self.headers()) as client:
|
||||
response = await client.post(
|
||||
_join_url(self.api_base_url, f"{self.route_prefix}/compute/files/import-local"),
|
||||
json=payload,
|
||||
)
|
||||
response.raise_for_status()
|
||||
return _unwrap_dict(response.json())
|
||||
|
||||
async def _request(
|
||||
self,
|
||||
method: str,
|
||||
path: str,
|
||||
json_data: dict[str, Any] | None = None,
|
||||
timeout: float | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Generic request method for compute API endpoints."""
|
||||
url = _join_url(self.api_base_url, f"{self.route_prefix}{path}")
|
||||
async with httpx.AsyncClient(timeout=timeout or 300, headers=self.headers()) as client:
|
||||
if method.upper() == "GET":
|
||||
response = await client.get(url)
|
||||
else:
|
||||
response = await client.post(url, json=json_data)
|
||||
response.raise_for_status()
|
||||
return _unwrap_dict(response.json())
|
||||
|
||||
# ── Inference helpers (short timeouts — see module constants) ──────────
|
||||
|
||||
async def inference_load(self, payload: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Dispatch a model load. Returns as soon as the node accepts the
|
||||
request; the node now loads asynchronously (status goes 'loading')."""
|
||||
return await self._request("POST", "/inference/load", json_data=payload, timeout=INFERENCE_LOAD_TIMEOUT)
|
||||
|
||||
async def inference_status(self) -> dict[str, Any]:
|
||||
return await self._request("GET", "/inference/status", timeout=INFERENCE_STATUS_TIMEOUT)
|
||||
|
||||
async def inference_unload(self) -> dict[str, Any]:
|
||||
return await self._request("POST", "/inference/unload", json_data={}, timeout=INFERENCE_UNLOAD_TIMEOUT)
|
||||
|
||||
async def upload_file(
|
||||
self,
|
||||
filename: str,
|
||||
content: bytes,
|
||||
target_relative_path: str,
|
||||
resource_type: str | None = None,
|
||||
resource_id: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
data = {
|
||||
"target_relative_path": target_relative_path,
|
||||
"resource_type": resource_type or "",
|
||||
"resource_id": resource_id or "",
|
||||
}
|
||||
files = {"file": (filename, content)}
|
||||
timeout = httpx.Timeout(max(self.timeout, 60), connect=self.timeout)
|
||||
async with httpx.AsyncClient(timeout=timeout, headers=self.headers()) as client:
|
||||
response = await client.post(
|
||||
_join_url(self.api_base_url, f"{self.route_prefix}/compute/files/upload"),
|
||||
data=data,
|
||||
files=files,
|
||||
)
|
||||
response.raise_for_status()
|
||||
return _unwrap_dict(response.json())
|
||||
@@ -1,191 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import time
|
||||
from typing import Any
|
||||
|
||||
from app.db.platform_store import get_platform_store
|
||||
from app.modules.compute_gateway.client import ComputeNodeClient
|
||||
|
||||
# starting 状态允许的最大轮询次数（约 40 * 3s ≈ 2 分钟），超过即判定节点不可达
|
||||
MAX_STARTING_ATTEMPTS = 40
|
||||
|
||||
|
||||
def _node_for_task(task: dict[str, Any]) -> dict[str, Any] | None:
|
||||
return next((node for node in get_platform_store().compute_nodes() if node["id"] == task.get("compute_node_id")), None)
|
||||
|
||||
|
||||
def _parse_inference_load_status(task: dict[str, Any]) -> tuple[list[dict[str, Any]], dict[str, Any]]:
|
||||
load_status = task.get("load_status") or {}
|
||||
if isinstance(load_status, str):
|
||||
try:
|
||||
load_status = json.loads(load_status)
|
||||
except (json.JSONDecodeError, TypeError):
|
||||
load_status = {}
|
||||
return load_status.get("loaded_models") or [], load_status
|
||||
|
||||
|
||||
async def reconcile_inference_loads(store: Any) -> list[dict[str, Any]]:
|
||||
"""推进处于 starting 状态的推理加载。
|
||||
|
||||
模型加载已改为异步派发：/model-compare/{id}/load 立即返回，这里在每次
|
||||
轮询时查询对应计算节点的 /inference/status，把任务从 starting 推进到
|
||||
ready/error。使用短超时，单节点不可达不会阻塞整轮轮询。
|
||||
"""
|
||||
reconciled: list[dict[str, Any]] = []
|
||||
now = time.time()
|
||||
for task in store.compare_tasks():
|
||||
items, _ = _parse_inference_load_status(task)
|
||||
if not any(item.get("status") == "starting" for item in items):
|
||||
continue
|
||||
# dirty 只要处理过任一 starting 项就置位：load_attempts / last_polled_at
|
||||
# 必须落库，否则节点不可达时计数不会累积，封顶逻辑永远触发不了
|
||||
dirty = False
|
||||
for item in items:
|
||||
if item.get("status") != "starting":
|
||||
continue
|
||||
# 节流：同一 item 每 3s 只查询一次
|
||||
if now - float(item.get("last_polled_at") or 0) < 3:
|
||||
continue
|
||||
item["last_polled_at"] = now
|
||||
item["load_attempts"] = int(item.get("load_attempts") or 0) + 1
|
||||
dirty = True
|
||||
node = next((n for n in store.compute_nodes() if n["id"] == item.get("node_id")), None)
|
||||
if not node:
|
||||
item["status"] = "error"
|
||||
item["error"] = "compute node deleted"
|
||||
store.mark_inference_unloaded(item.get("node_id") or "")
|
||||
continue
|
||||
if not node.get("enabled") or node.get("scheduler_status") != "online":
|
||||
item["status"] = "error"
|
||||
item["error"] = "compute node offline"
|
||||
store.mark_inference_unloaded(node["id"])
|
||||
continue
|
||||
try:
|
||||
status = await ComputeNodeClient(node["api_base_url"]).inference_status()
|
||||
except Exception as exc: # noqa: BLE001 - node unreachable; keep retrying until cap
|
||||
if int(item.get("load_attempts") or 0) >= MAX_STARTING_ATTEMPTS:
|
||||
item["status"] = "error"
|
||||
item["error"] = f"compute node unreachable: {exc}"
|
||||
store.mark_inference_unloaded(node["id"])
|
||||
continue
|
||||
node_status = status.get("status")
|
||||
if node_status == "ready":
|
||||
item["status"] = "ready"
|
||||
item.pop("error", None)
|
||||
store.mark_inference_loaded(node["id"])
|
||||
elif node_status == "error":
|
||||
item["status"] = "error"
|
||||
item["error"] = status.get("error") or "model load failed on compute node"
|
||||
store.mark_inference_unloaded(node["id"])
|
||||
elif node_status == "idle":
|
||||
# 节点重启导致已加载模型丢失
|
||||
item["status"] = "error"
|
||||
item["error"] = "model disappeared from compute node (node may have restarted)"
|
||||
store.mark_inference_unloaded(node["id"])
|
||||
# node_status == "loading" -> 保持 starting，下轮再查
|
||||
if dirty:
|
||||
if any(i.get("status") in {"ready", "running"} for i in items):
|
||||
new_status = "loaded"
|
||||
elif any(i.get("status") == "starting" for i in items):
|
||||
new_status = "starting" # 仍在加载中，保持 starting
|
||||
else:
|
||||
new_status = "failed"
|
||||
store.update_compare_task(task["id"], {"status": new_status, "load_status": {"loaded_models": items}})
|
||||
reconciled.append({"task_id": task["id"], "status": new_status})
|
||||
return reconciled
|
||||
|
||||
|
||||
async def fetch_eval_result_content(client: ComputeNodeClient, node: dict[str, Any], job: dict[str, Any]) -> dict[str, Any] | None:
|
||||
output_dir = job.get("output_dir")
|
||||
if not output_dir:
|
||||
return None
|
||||
full_path = f"{str(output_dir).rstrip('/')}/eval_results.json"
|
||||
data_root = "/data/yg-ft/"
|
||||
if full_path.startswith(data_root):
|
||||
full_path = full_path[len(data_root):]
|
||||
rel_path = full_path.lstrip("/")
|
||||
import httpx
|
||||
url = f"{node['api_base_url'].rstrip('/')}/modelTF/compute/files/read"
|
||||
async with httpx.AsyncClient(timeout=30, headers=client.headers()) as http:
|
||||
response = await http.get(url, params={"path": rel_path})
|
||||
response.raise_for_status()
|
||||
payload = response.json()
|
||||
return payload if isinstance(payload, dict) else None
|
||||
|
||||
|
||||
async def poll_compute_jobs_once() -> dict[str, Any]:
|
||||
store = get_platform_store()
|
||||
synced: list[dict[str, Any]] = []
|
||||
failed: list[dict[str, str]] = []
|
||||
for task in store.running_compute_tasks():
|
||||
node = _node_for_task(task)
|
||||
if not node:
|
||||
failed.append({"task_id": task["id"], "error": "compute node not found"})
|
||||
continue
|
||||
try:
|
||||
client = ComputeNodeClient(node["api_base_url"])
|
||||
job = await client.get_job(task["compute_job_id"])
|
||||
try:
|
||||
logs = await client.job_logs(task["compute_job_id"], tail_lines=5000)
|
||||
store.record_training_log_metrics(task["id"], str(logs.get("content") or ""))
|
||||
except Exception:
|
||||
pass
|
||||
# P0-4: Force-fetch last log snippet when job reaches terminal state
|
||||
if job.get("status") in {"failed", "stopped"}:
|
||||
try:
|
||||
last_logs = await client.job_logs(task["compute_job_id"], tail_lines=200)
|
||||
job["log_snippet"] = str(last_logs.get("content") or "")[:8192]
|
||||
except Exception:
|
||||
pass
|
||||
synced.append(store.apply_compute_job(task["id"], job))
|
||||
except Exception as exc: # noqa: BLE001 - keep polling other jobs
|
||||
failed.append({"task_id": task["id"], "error": str(exc)})
|
||||
standalone_synced: list[dict[str, Any]] = []
|
||||
for record in store.active_standalone_compute_jobs():
|
||||
node = next((item for item in store.compute_nodes() if item["id"] == record.get("node_id")), None)
|
||||
if not node:
|
||||
failed.append({"job_id": record["id"], "error": "compute node not found"})
|
||||
continue
|
||||
try:
|
||||
job = await ComputeNodeClient(node["api_base_url"]).get_job(record["id"])
|
||||
standalone_synced.append(store.sync_model_merge_job(record["id"], job))
|
||||
except Exception as exc: # noqa: BLE001 - keep polling other jobs
|
||||
failed.append({"job_id": record["id"], "error": str(exc)})
|
||||
|
||||
# ── Eval job sync ────────────────────────────────────────────────
|
||||
eval_synced = 0
|
||||
for eval_task in store.running_eval_tasks():
|
||||
node = next(
|
||||
(item for item in store.compute_nodes() if item["id"] == eval_task.get("compute_node_id")),
|
||||
None,
|
||||
)
|
||||
if not node:
|
||||
failed.append({"eval_task_id": eval_task["id"], "error": "compute node not found"})
|
||||
continue
|
||||
try:
|
||||
client = ComputeNodeClient(node["api_base_url"])
|
||||
job = await client.get_job(eval_task["compute_job_id"])
|
||||
result_content = None
|
||||
# Try to read eval_results.json from the job output directory
|
||||
if job.get("status") == "completed" and job.get("output_dir"):
|
||||
try:
|
||||
result_content = await fetch_eval_result_content(client, node, job)
|
||||
except Exception:
|
||||
pass
|
||||
store.apply_eval_job_result(eval_task["id"], job, result_content)
|
||||
# 评测 GPU 占用由 eval_tasks 状态派生，无需维护推理内存标记
|
||||
eval_synced += 1
|
||||
except Exception as exc: # noqa: BLE001
|
||||
failed.append({"eval_task_id": eval_task["id"], "error": str(exc)})
|
||||
|
||||
# ── Inference load reconciliation ─────────────────────────────────────
|
||||
try:
|
||||
inference_reconciled = await reconcile_inference_loads(store)
|
||||
except Exception as exc: # noqa: BLE001 - keep polling alive
|
||||
failed.append({"inference_reconcile": str(exc)})
|
||||
inference_reconciled = []
|
||||
|
||||
return {"synced": len(synced) + len(standalone_synced) + eval_synced, "failed": failed,
|
||||
"items": synced, "standalone": standalone_synced, "eval_synced": eval_synced,
|
||||
"inference_reconciled": inference_reconciled}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,7 +0,0 @@
|
||||
"""数据处理模块的共享限制。"""
|
||||
|
||||
MAX_QA_PAIRS_PER_ITEM = 50
|
||||
MODEL_GENERATION_BATCH_SIZE = 10
|
||||
|
||||
|
||||
__all__ = ["MAX_QA_PAIRS_PER_ITEM", "MODEL_GENERATION_BATCH_SIZE"]
|
||||
@@ -1,148 +0,0 @@
|
||||
"""Dataset format validation for Alpaca, ShareGPT, DPO, CPT formats.
|
||||
|
||||
Used by the training preflight flow to validate that uploaded dataset files
|
||||
conform to the declared format before submitting to the compute node.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import Any
|
||||
|
||||
|
||||
def _load_sample(path: str | None, content: str | None = None, max_samples: int = 20) -> list[dict[str, Any]]:
|
||||
"""Load up to max_samples records from JSONL file path or raw content string."""
|
||||
try:
|
||||
if content is not None:
|
||||
text = content.strip()
|
||||
elif path:
|
||||
with open(path, "r", encoding="utf-8") as fh:
|
||||
text = fh.read().strip()
|
||||
else:
|
||||
return []
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
if not text:
|
||||
return []
|
||||
|
||||
lines = text.splitlines()[:max_samples]
|
||||
records: list[dict[str, Any]] = []
|
||||
for line in lines:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
record = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
if isinstance(record, dict):
|
||||
records.append(record)
|
||||
return records
|
||||
|
||||
|
||||
def _check_alpaca(records: list[dict[str, Any]]) -> list[str]:
|
||||
"""Validate Alpaca format: requires 'instruction' field."""
|
||||
errors: list[str] = []
|
||||
if not records:
|
||||
errors.append("Alpaca 格式数据集无有效记录")
|
||||
return errors
|
||||
missing_instruction = sum(1 for r in records if not r.get("instruction"))
|
||||
if missing_instruction:
|
||||
errors.append(
|
||||
f"Alpaca 格式要求每条记录包含 instruction 字段，"
|
||||
f"前{len(records)}条中有{missing_instruction}条缺失"
|
||||
)
|
||||
return errors
|
||||
|
||||
|
||||
def _check_sharegpt(records: list[dict[str, Any]]) -> list[str]:
|
||||
"""Validate ShareGPT format: requires 'messages' (list of dicts with role/content)."""
|
||||
errors: list[str] = []
|
||||
if not records:
|
||||
errors.append("ShareGPT 格式数据集无有效记录")
|
||||
return errors
|
||||
bad = 0
|
||||
for r in records:
|
||||
messages = r.get("messages")
|
||||
if not isinstance(messages, list) or not messages:
|
||||
bad += 1
|
||||
continue
|
||||
for msg in messages:
|
||||
if not isinstance(msg, dict) or "role" not in msg or "content" not in msg:
|
||||
bad += 1
|
||||
break
|
||||
if bad:
|
||||
errors.append(
|
||||
f"ShareGPT 格式要求每条记录包含 messages 列表，"
|
||||
f"每条消息需有 role 和 content 字段，前{len(records)}条中有{bad}条不符合"
|
||||
)
|
||||
return errors
|
||||
|
||||
|
||||
def _check_dpo(records: list[dict[str, Any]]) -> list[str]:
|
||||
"""Validate DPO format: requires 'chosen' and 'rejected' fields."""
|
||||
errors: list[str] = []
|
||||
if not records:
|
||||
errors.append("DPO 格式数据集无有效记录")
|
||||
return errors
|
||||
missing_chosen = sum(1 for r in records if not r.get("chosen"))
|
||||
missing_rejected = sum(1 for r in records if not r.get("rejected"))
|
||||
if missing_chosen:
|
||||
errors.append(f"DPO 格式要求 chosen 字段，前{len(records)}条中有{missing_chosen}条缺失")
|
||||
if missing_rejected:
|
||||
errors.append(f"DPO 格式要求 rejected 字段，前{len(records)}条中有{missing_rejected}条缺失")
|
||||
return errors
|
||||
|
||||
|
||||
def _check_cpt(records: list[dict[str, Any]]) -> list[str]:
|
||||
"""Validate CPT format: requires 'text' field, should NOT have instruction/output."""
|
||||
errors: list[str] = []
|
||||
if not records:
|
||||
errors.append("CPT 格式数据集无有效记录")
|
||||
return errors
|
||||
missing_text = sum(1 for r in records if not r.get("text"))
|
||||
has_instruction = sum(1 for r in records if r.get("instruction") or r.get("output"))
|
||||
if missing_text:
|
||||
errors.append(f"CPT 格式要求 text 字段，前{len(records)}条中有{missing_text}条缺失")
|
||||
if has_instruction:
|
||||
errors.append(
|
||||
f"CPT 格式不应包含 instruction/output 字段（疑似 Alpaca 格式），"
|
||||
f"前{len(records)}条中有{has_instruction}条包含此类字段"
|
||||
)
|
||||
return errors
|
||||
|
||||
|
||||
FORMAT_VALIDATORS = {
|
||||
"alpaca": _check_alpaca,
|
||||
"alpaca_jsonl": _check_alpaca,
|
||||
"sharegpt": _check_sharegpt,
|
||||
"dpo": _check_dpo,
|
||||
"cpt": _check_cpt,
|
||||
"pt": _check_cpt,
|
||||
}
|
||||
|
||||
|
||||
def validate_dataset_format(
|
||||
dataset_format: str,
|
||||
content: str | None = None,
|
||||
path: str | None = None,
|
||||
max_samples: int = 20,
|
||||
) -> list[str]:
|
||||
"""Validate dataset content against expected format.
|
||||
|
||||
Args:
|
||||
dataset_format: One of 'alpaca', 'sharegpt', 'dpo', 'cpt'.
|
||||
content: Raw file content (JSONL text). Mutually exclusive with path.
|
||||
path: File path to read content from.
|
||||
max_samples: Maximum records to sample for validation.
|
||||
|
||||
Returns:
|
||||
List of error messages (empty if valid).
|
||||
"""
|
||||
fmt = str(dataset_format).lower().strip()
|
||||
validator = FORMAT_VALIDATORS.get(fmt)
|
||||
if not validator:
|
||||
return [f"不支持的数据集格式: {dataset_format}，支持的格式: {', '.join(sorted(FORMAT_VALIDATORS))}"]
|
||||
records = _load_sample(path=path, content=content, max_samples=max_samples)
|
||||
return validator(records)
|
||||
@@ -1,444 +0,0 @@
|
||||
"""基于 Docling 与 LlamaIndex 的文档切分实现。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import re
|
||||
import threading
|
||||
import unicodedata
|
||||
from dataclasses import dataclass
|
||||
from functools import lru_cache
|
||||
from io import BytesIO
|
||||
from typing import Any, Literal
|
||||
|
||||
import tiktoken
|
||||
from docling_core.transforms.chunker.hierarchical_chunker import ChunkingSerializerProvider
|
||||
from llama_index.core import Document
|
||||
from llama_index.core.base.embeddings.base import BaseEmbedding
|
||||
from llama_index.core.node_parser import SemanticSplitterNodeParser, SentenceSplitter
|
||||
|
||||
from app.modules.data_process.algorithms import normalize_text
|
||||
|
||||
ChunkMethod = Literal["layout_hybrid", "semantic", "fixed"]
|
||||
|
||||
_PAGE_FURNITURE = re.compile(
|
||||
r"(?m)^\s*(?:第\s*\d+\s*页\s*共\s*\d+\s*页|[-—–]?\s*\d+\s*[/／]\s*\d+\s*[-—–]?)\s*$"
|
||||
)
|
||||
_COMPACT_CHARACTER = re.compile(r"[\w\u3400-\u4dbf\u4e00-\u9fff]", re.UNICODE)
|
||||
_CONVERTER_LOCK = threading.Lock()
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class DocumentChunk:
|
||||
"""切片正文及其在原文件中的可追溯信息。"""
|
||||
|
||||
original_content: str
|
||||
contextualized_content: str
|
||||
source_start: int | None
|
||||
source_end: int | None
|
||||
source_start_line: int | None
|
||||
source_end_line: int | None
|
||||
token_count: int
|
||||
heading_path: tuple[str, ...] = ()
|
||||
source_pages: tuple[int, ...] = ()
|
||||
doc_item_refs: tuple[str, ...] = ()
|
||||
source_bboxes: tuple[dict[str, Any], ...] = ()
|
||||
|
||||
|
||||
def _sentence_chunks(text: str) -> list[str]:
|
||||
"""提供稳定的中英文句界，避免 LlamaIndex 默认分词器下载额外资源。"""
|
||||
|
||||
boundary = re.compile(
|
||||
r".*?(?:\n\s*\n|[。！？!?；;](?:[\"'”’）】》]*)|\.(?:\s+|$)|$)",
|
||||
re.DOTALL,
|
||||
)
|
||||
return [part for part in boundary.findall(text) if part]
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def _tokenizer() -> tiktoken.Encoding:
|
||||
return tiktoken.get_encoding("cl100k_base")
|
||||
|
||||
|
||||
def _text_chunks(
|
||||
text: str,
|
||||
*,
|
||||
chunk_size: int,
|
||||
chunk_overlap: int,
|
||||
) -> list[DocumentChunk]:
|
||||
normalized = normalize_text(text)
|
||||
if not normalized:
|
||||
return []
|
||||
splitter = SentenceSplitter(
|
||||
chunk_size=chunk_size,
|
||||
chunk_overlap=chunk_overlap,
|
||||
tokenizer=_tokenizer().encode,
|
||||
chunking_tokenizer_fn=_sentence_chunks,
|
||||
include_metadata=False,
|
||||
include_prev_next_rel=False,
|
||||
)
|
||||
nodes = splitter.get_nodes_from_documents([Document(text=normalized)])
|
||||
return _nodes_to_chunks(nodes, normalized)
|
||||
|
||||
|
||||
def chunk_fixed_text(
|
||||
text: str,
|
||||
*,
|
||||
chunk_size: int,
|
||||
chunk_overlap: int,
|
||||
) -> list[DocumentChunk]:
|
||||
"""使用 LlamaIndex SentenceSplitter 按句界控制固定 Token 长度。"""
|
||||
|
||||
return _text_chunks(text, chunk_size=chunk_size, chunk_overlap=chunk_overlap)
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def _semantic_embedding_model() -> BaseEmbedding:
|
||||
# 模型可在部署环境覆盖；默认模型体积较小且适合中英文语义边界判断。
|
||||
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
|
||||
|
||||
return HuggingFaceEmbedding(
|
||||
model_name=os.getenv("DATA_PROCESS_EMBEDDING_MODEL", "BAAI/bge-small-zh-v1.5"),
|
||||
device=os.getenv("DATA_PROCESS_EMBEDDING_DEVICE", "cpu"),
|
||||
trust_remote_code=False,
|
||||
)
|
||||
|
||||
|
||||
def chunk_semantic_text(
|
||||
text: str,
|
||||
*,
|
||||
chunk_size: int,
|
||||
chunk_overlap: int,
|
||||
breakpoint_percentile_threshold: int,
|
||||
embed_model: BaseEmbedding | None = None,
|
||||
) -> list[DocumentChunk]:
|
||||
"""使用 LlamaIndex SemanticSplitter 识别主题跳变，再限制最大长度。"""
|
||||
|
||||
normalized = normalize_text(text)
|
||||
if not normalized:
|
||||
return []
|
||||
splitter = SemanticSplitterNodeParser.from_defaults(
|
||||
embed_model=embed_model or _semantic_embedding_model(),
|
||||
breakpoint_percentile_threshold=breakpoint_percentile_threshold,
|
||||
buffer_size=1,
|
||||
sentence_splitter=_sentence_chunks,
|
||||
include_metadata=False,
|
||||
include_prev_next_rel=False,
|
||||
)
|
||||
semantic_nodes = splitter.get_nodes_from_documents([Document(text=normalized)])
|
||||
result: list[DocumentChunk] = []
|
||||
search_from = 0
|
||||
for node in semantic_nodes:
|
||||
content = node.get_content().strip()
|
||||
if not content:
|
||||
continue
|
||||
start = _locate_text(normalized, content, search_from)
|
||||
if start is None:
|
||||
start = _locate_text(normalized, content, 0)
|
||||
if start is None:
|
||||
continue
|
||||
if len(_tokenizer().encode(content)) <= chunk_size:
|
||||
result.append(_make_text_chunk(normalized, start, start + len(content)))
|
||||
else:
|
||||
for child in _text_chunks(
|
||||
content,
|
||||
chunk_size=chunk_size,
|
||||
chunk_overlap=chunk_overlap,
|
||||
):
|
||||
if child.source_start is None or child.source_end is None:
|
||||
continue
|
||||
result.append(
|
||||
_make_text_chunk(
|
||||
normalized,
|
||||
start + child.source_start,
|
||||
start + child.source_end,
|
||||
)
|
||||
)
|
||||
search_from = start + len(content)
|
||||
return result
|
||||
|
||||
|
||||
def _nodes_to_chunks(nodes: list[Any], source_text: str) -> list[DocumentChunk]:
|
||||
chunks: list[DocumentChunk] = []
|
||||
search_from = 0
|
||||
for node in nodes:
|
||||
content = node.get_content().strip()
|
||||
if not content:
|
||||
continue
|
||||
raw_start = getattr(node, "start_char_idx", None)
|
||||
raw_end = getattr(node, "end_char_idx", None)
|
||||
if (
|
||||
isinstance(raw_start, int)
|
||||
and isinstance(raw_end, int)
|
||||
and source_text[raw_start:raw_end].strip() == content
|
||||
):
|
||||
start = raw_start + len(source_text[raw_start:raw_end]) - len(source_text[raw_start:raw_end].lstrip())
|
||||
else:
|
||||
start = _locate_text(source_text, content, search_from)
|
||||
if start is None:
|
||||
start = _locate_text(source_text, content, 0)
|
||||
if start is None:
|
||||
continue
|
||||
end = start + len(content)
|
||||
chunks.append(_make_text_chunk(source_text, start, end))
|
||||
search_from = max(search_from, end)
|
||||
return chunks
|
||||
|
||||
|
||||
def _locate_text(source: str, content: str, start: int) -> int | None:
|
||||
position = source.find(content, start)
|
||||
return position if position >= 0 else None
|
||||
|
||||
|
||||
def _make_text_chunk(source: str, start: int, end: int) -> DocumentChunk:
|
||||
content = source[start:end]
|
||||
return DocumentChunk(
|
||||
original_content=content,
|
||||
contextualized_content=content,
|
||||
source_start=start,
|
||||
source_end=end,
|
||||
source_start_line=source.count("\n", 0, start) + 1,
|
||||
source_end_line=source.count("\n", 0, max(start, end - 1)) + 1,
|
||||
token_count=len(_tokenizer().encode(content)),
|
||||
)
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def _document_converter():
|
||||
from docling.document_converter import DocumentConverter
|
||||
|
||||
return DocumentConverter()
|
||||
|
||||
|
||||
class _MarkdownSerializerProvider(ChunkingSerializerProvider):
|
||||
def get_serializer(self, doc: Any):
|
||||
from docling_core.transforms.chunker.hierarchical_chunker import ChunkingDocSerializer
|
||||
from docling_core.transforms.serializer.markdown import (
|
||||
MarkdownParams,
|
||||
MarkdownTableSerializer,
|
||||
)
|
||||
from docling_core.types.doc import DocItemLabel
|
||||
|
||||
excluded = {
|
||||
DocItemLabel.DOCUMENT_INDEX,
|
||||
DocItemLabel.PAGE_HEADER,
|
||||
DocItemLabel.PAGE_FOOTER,
|
||||
}
|
||||
return ChunkingDocSerializer(
|
||||
doc=doc,
|
||||
table_serializer=MarkdownTableSerializer(),
|
||||
params=MarkdownParams(
|
||||
labels=set(DocItemLabel) - excluded,
|
||||
compact_tables=True,
|
||||
image_placeholder="",
|
||||
escape_html=False,
|
||||
escape_underscores=False,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _clean_layout_text(value: str) -> str:
|
||||
return normalize_text(_PAGE_FURNITURE.sub("", value)).strip()
|
||||
|
||||
|
||||
def _compact_with_offsets(value: str) -> tuple[str, list[int]]:
|
||||
compact: list[str] = []
|
||||
offsets: list[int] = []
|
||||
for index, character in enumerate(unicodedata.normalize("NFKC", value)):
|
||||
if _COMPACT_CHARACTER.fullmatch(character):
|
||||
compact.append(character.casefold())
|
||||
offsets.append(index)
|
||||
return "".join(compact), offsets
|
||||
|
||||
|
||||
def _project_layout_span(
|
||||
source_text: str,
|
||||
content: str,
|
||||
*,
|
||||
compact_source: str,
|
||||
source_offsets: list[int],
|
||||
compact_start: int,
|
||||
) -> tuple[int | None, int | None, int]:
|
||||
compact_content, _ = _compact_with_offsets(content)
|
||||
if len(compact_content) < 4:
|
||||
return None, None, compact_start
|
||||
position = compact_source.find(compact_content, compact_start)
|
||||
if position < 0:
|
||||
position = compact_source.find(compact_content)
|
||||
if position < 0:
|
||||
return None, None, compact_start
|
||||
start = source_offsets[position]
|
||||
end = source_offsets[position + len(compact_content) - 1] + 1
|
||||
while start > 0 and source_text[start - 1] not in "\r\n":
|
||||
start -= 1
|
||||
while end < len(source_text) and source_text[end] not in "\r\n":
|
||||
end += 1
|
||||
return start, end, position + len(compact_content)
|
||||
|
||||
|
||||
def chunk_layout_document(
|
||||
raw: bytes,
|
||||
*,
|
||||
filename: str,
|
||||
source_text: str,
|
||||
chunk_size: int,
|
||||
) -> list[DocumentChunk]:
|
||||
"""使用 Docling HybridChunker 按版面层级、列表与表格边界切分。"""
|
||||
|
||||
from docling.chunking import HybridChunker
|
||||
from docling.datamodel.base_models import DocumentStream
|
||||
from docling.exceptions import BaseError as DoclingError
|
||||
from docling_core.transforms.chunker.tokenizer.openai import OpenAITokenizer
|
||||
from docling_core.types.doc import DocItemLabel
|
||||
|
||||
try:
|
||||
with _CONVERTER_LOCK:
|
||||
conversion = _document_converter().convert(
|
||||
DocumentStream(name=filename, stream=BytesIO(raw))
|
||||
)
|
||||
except DoclingError as exc:
|
||||
raise ValueError(f"文档版面解析失败: {exc}") from exc
|
||||
chunker = HybridChunker(
|
||||
tokenizer=OpenAITokenizer(tokenizer=_tokenizer(), max_tokens=chunk_size),
|
||||
serializer_provider=_MarkdownSerializerProvider(),
|
||||
merge_peers=True,
|
||||
repeat_table_header=True,
|
||||
)
|
||||
compact_source, source_offsets = _compact_with_offsets(source_text)
|
||||
compact_start = 0
|
||||
result: list[DocumentChunk] = []
|
||||
excluded = {
|
||||
DocItemLabel.DOCUMENT_INDEX,
|
||||
DocItemLabel.PAGE_HEADER,
|
||||
DocItemLabel.PAGE_FOOTER,
|
||||
}
|
||||
for raw_chunk in chunker.chunk(conversion.document):
|
||||
doc_items = tuple(raw_chunk.meta.doc_items or ())
|
||||
if doc_items and all(item.label in excluded for item in doc_items):
|
||||
continue
|
||||
content = _clean_layout_text(raw_chunk.text)
|
||||
if not content:
|
||||
continue
|
||||
contextualized = _clean_layout_text(chunker.contextualize(raw_chunk)) or content
|
||||
start, end, compact_start = _project_layout_span(
|
||||
source_text,
|
||||
content,
|
||||
compact_source=compact_source,
|
||||
source_offsets=source_offsets,
|
||||
compact_start=compact_start,
|
||||
)
|
||||
original = source_text[start:end] if start is not None and end is not None else content
|
||||
pages: set[int] = set()
|
||||
refs: list[str] = []
|
||||
bboxes: list[dict[str, Any]] = []
|
||||
for item in doc_items:
|
||||
refs.append(str(item.self_ref))
|
||||
for provenance in item.prov or ():
|
||||
pages.add(int(provenance.page_no))
|
||||
bbox = provenance.bbox
|
||||
bboxes.append(
|
||||
{
|
||||
"page": int(provenance.page_no),
|
||||
"left": float(bbox.l),
|
||||
"top": float(bbox.t),
|
||||
"right": float(bbox.r),
|
||||
"bottom": float(bbox.b),
|
||||
"origin": str(bbox.coord_origin.value),
|
||||
}
|
||||
)
|
||||
result.append(
|
||||
DocumentChunk(
|
||||
original_content=original,
|
||||
contextualized_content=contextualized,
|
||||
source_start=start,
|
||||
source_end=end,
|
||||
source_start_line=(source_text.count("\n", 0, start) + 1 if start is not None else None),
|
||||
source_end_line=(
|
||||
source_text.count("\n", 0, max(start or 0, (end or 1) - 1)) + 1
|
||||
if end is not None
|
||||
else None
|
||||
),
|
||||
token_count=len(_tokenizer().encode(contextualized)),
|
||||
heading_path=tuple(str(item) for item in (raw_chunk.meta.headings or ())),
|
||||
source_pages=tuple(sorted(pages)),
|
||||
doc_item_refs=tuple(refs),
|
||||
source_bboxes=tuple(bboxes),
|
||||
)
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
def merge_short_chunks(
|
||||
chunks: list[DocumentChunk],
|
||||
*,
|
||||
source_text: str,
|
||||
min_token_count: int,
|
||||
max_token_count: int,
|
||||
) -> list[DocumentChunk]:
|
||||
"""在不突破长度上限的前提下，把过短块并入相邻内容。"""
|
||||
|
||||
result: list[DocumentChunk] = []
|
||||
index = 0
|
||||
while index < len(chunks):
|
||||
current = chunks[index]
|
||||
if current.token_count >= min_token_count:
|
||||
result.append(current)
|
||||
index += 1
|
||||
continue
|
||||
if index + 1 < len(chunks):
|
||||
combined = _combine_chunks(current, chunks[index + 1], source_text)
|
||||
if combined.token_count <= max_token_count:
|
||||
result.append(combined)
|
||||
index += 2
|
||||
continue
|
||||
if result:
|
||||
combined = _combine_chunks(result[-1], current, source_text)
|
||||
if combined.token_count <= max_token_count:
|
||||
result[-1] = combined
|
||||
index += 1
|
||||
continue
|
||||
result.append(current)
|
||||
index += 1
|
||||
return result
|
||||
|
||||
|
||||
def _combine_chunks(
|
||||
left: DocumentChunk,
|
||||
right: DocumentChunk,
|
||||
source_text: str,
|
||||
) -> DocumentChunk:
|
||||
contextualized = "\n\n".join(
|
||||
part for part in (left.contextualized_content, right.contextualized_content) if part
|
||||
)
|
||||
start = left.source_start
|
||||
end = right.source_end
|
||||
has_contiguous_source = (
|
||||
start is not None
|
||||
and left.source_end is not None
|
||||
and right.source_start is not None
|
||||
and end is not None
|
||||
and left.source_end <= right.source_start
|
||||
)
|
||||
original = (
|
||||
source_text[start:end]
|
||||
if has_contiguous_source and start is not None and end is not None
|
||||
else "\n\n".join(
|
||||
part for part in (left.original_content, right.original_content) if part
|
||||
)
|
||||
)
|
||||
if not has_contiguous_source:
|
||||
start = None
|
||||
end = None
|
||||
return DocumentChunk(
|
||||
original_content=original,
|
||||
contextualized_content=contextualized,
|
||||
source_start=start,
|
||||
source_end=end,
|
||||
source_start_line=left.source_start_line if start is not None else None,
|
||||
source_end_line=right.source_end_line if end is not None else None,
|
||||
token_count=len(_tokenizer().encode(contextualized)),
|
||||
heading_path=left.heading_path or right.heading_path,
|
||||
source_pages=tuple(sorted(set(left.source_pages) | set(right.source_pages))),
|
||||
doc_item_refs=left.doc_item_refs + right.doc_item_refs,
|
||||
source_bboxes=left.source_bboxes + right.source_bboxes,
|
||||
)
|
||||
@@ -1,562 +0,0 @@
|
||||
"""数据处理任务的大模型生成适配器。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
from collections.abc import Callable, Iterable, Mapping
|
||||
from typing import Any
|
||||
from urllib.parse import urlsplit, urlunsplit
|
||||
|
||||
import httpx
|
||||
|
||||
from app.modules.data_process.algorithms import normalize_text, stable_split_assignments
|
||||
from app.modules.data_process.constants import (
|
||||
MAX_QA_PAIRS_PER_ITEM,
|
||||
MODEL_GENERATION_BATCH_SIZE,
|
||||
)
|
||||
|
||||
|
||||
class ModelGenerationError(ValueError):
|
||||
"""模型配置、响应或调用失败。"""
|
||||
|
||||
|
||||
class _TerminalModelGenerationError(ModelGenerationError):
|
||||
"""使用相同参数重试也无法恢复的模型响应错误。"""
|
||||
|
||||
|
||||
OUTPUT_TYPE_STANDARD = "standard"
|
||||
OUTPUT_TYPE_REASONING = "reasoning"
|
||||
SUPPORTED_OUTPUT_TYPES = {OUTPUT_TYPE_STANDARD, OUTPUT_TYPE_REASONING}
|
||||
REASONING_DETAIL_NORMAL = "normal"
|
||||
REASONING_DETAIL_DETAILED = "detailed"
|
||||
SUPPORTED_REASONING_DETAILS = {
|
||||
REASONING_DETAIL_NORMAL,
|
||||
REASONING_DETAIL_DETAILED,
|
||||
}
|
||||
MINIMAX_M3_API_HOSTS = {"api.minimax.io", "api.minimaxi.com"}
|
||||
MINIMAX_M3_MIN_COMPLETION_TOKENS = 4096
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _is_retryable_generation_error(exc: Exception) -> bool:
|
||||
if isinstance(exc, _TerminalModelGenerationError):
|
||||
return False
|
||||
if isinstance(exc, httpx.HTTPStatusError):
|
||||
status_code = exc.response.status_code
|
||||
return status_code in {408, 425, 429} or status_code >= 500
|
||||
if isinstance(exc, httpx.RequestError):
|
||||
return True
|
||||
return isinstance(exc, (json.JSONDecodeError, ModelGenerationError))
|
||||
|
||||
|
||||
def _is_official_minimax_m3(endpoint: str, model_name: str) -> bool:
|
||||
host = (urlsplit(endpoint).hostname or "").casefold()
|
||||
return host in MINIMAX_M3_API_HOSTS and model_name.casefold() == "minimax-m3"
|
||||
|
||||
|
||||
def chat_completions_url(value: str) -> str:
|
||||
"""把域名、基础 URL 或完整地址统一为 chat completions 地址。"""
|
||||
|
||||
raw = (value or "").strip()
|
||||
if not raw:
|
||||
raise ModelGenerationError("generation model api_url is required")
|
||||
if "://" not in raw:
|
||||
raw = f"https://{raw}"
|
||||
parsed = urlsplit(raw)
|
||||
if parsed.scheme not in {"http", "https"} or not parsed.hostname:
|
||||
raise ModelGenerationError("generation model api_url must be an HTTP(S) host or URL")
|
||||
if parsed.username or parsed.password:
|
||||
raise ModelGenerationError("generation model api_url must not contain credentials")
|
||||
|
||||
path = parsed.path.rstrip("/")
|
||||
if path.endswith("/chat/completions"):
|
||||
target_path = path
|
||||
elif path.endswith("/v1"):
|
||||
target_path = f"{path}/chat/completions"
|
||||
elif not path:
|
||||
target_path = "/v1/chat/completions"
|
||||
else:
|
||||
target_path = f"{path}/v1/chat/completions"
|
||||
return urlunsplit((parsed.scheme, parsed.netloc, target_path, "", ""))
|
||||
|
||||
|
||||
def _response_choice(payload: Mapping[str, Any]) -> Mapping[str, Any]:
|
||||
try:
|
||||
choice = payload["choices"][0]
|
||||
except (KeyError, IndexError, TypeError) as exc:
|
||||
raise ModelGenerationError("模型响应缺少 choices[0]") from exc
|
||||
if not isinstance(choice, Mapping):
|
||||
raise ModelGenerationError("模型响应 choices[0] 不是对象")
|
||||
return choice
|
||||
|
||||
|
||||
def _response_finish_reason(payload: Mapping[str, Any]) -> str:
|
||||
try:
|
||||
return str(_response_choice(payload).get("finish_reason") or "").strip().lower()
|
||||
except ModelGenerationError:
|
||||
return ""
|
||||
|
||||
|
||||
def _response_content_length(payload: Mapping[str, Any]) -> int:
|
||||
try:
|
||||
message = _response_choice(payload).get("message")
|
||||
if not isinstance(message, Mapping):
|
||||
return 0
|
||||
content = message.get("content")
|
||||
if isinstance(content, str):
|
||||
return len(content)
|
||||
if isinstance(content, list):
|
||||
return sum(
|
||||
len(str(item.get("text") or ""))
|
||||
for item in content
|
||||
if isinstance(item, Mapping)
|
||||
)
|
||||
except ModelGenerationError:
|
||||
pass
|
||||
return 0
|
||||
|
||||
|
||||
def _raise_for_terminal_response(payload: Mapping[str, Any]) -> Mapping[str, Any]:
|
||||
choice = _response_choice(payload)
|
||||
base_response = payload.get("base_resp")
|
||||
status_code: Any = None
|
||||
status_message = ""
|
||||
if isinstance(base_response, Mapping):
|
||||
status_code = base_response.get("status_code")
|
||||
status_message = re.sub(
|
||||
r"\s+", " ", str(base_response.get("status_msg") or "")
|
||||
).strip()[:200]
|
||||
|
||||
if bool(payload.get("input_sensitive")) or status_code in {1026, "1026"}:
|
||||
raise _TerminalModelGenerationError(
|
||||
f"模型输入触发内容安全拦截（code={status_code or 1026}）"
|
||||
)
|
||||
if bool(payload.get("output_sensitive")) or status_code in {1027, "1027"}:
|
||||
raise _TerminalModelGenerationError(
|
||||
f"模型输出触发内容安全拦截（code={status_code or 1027}）"
|
||||
)
|
||||
|
||||
finish_reason = str(choice.get("finish_reason") or "").strip().lower()
|
||||
if finish_reason == "length":
|
||||
raise _TerminalModelGenerationError(
|
||||
"模型输出因达到 Token 上限被截断（finish_reason=length），"
|
||||
"请提高最大输出长度后重试"
|
||||
)
|
||||
if finish_reason == "content_filter":
|
||||
raise _TerminalModelGenerationError(
|
||||
"模型输出被内容安全策略拦截（finish_reason=content_filter）"
|
||||
)
|
||||
if finish_reason in {"tool_calls", "function_call"}:
|
||||
raise _TerminalModelGenerationError(
|
||||
f"模型返回了当前生成任务不支持的工具调用（finish_reason={finish_reason}）"
|
||||
)
|
||||
if status_code not in {None, "", 0, "0"}:
|
||||
detail = f"：{status_message}" if status_message else ""
|
||||
raise _TerminalModelGenerationError(
|
||||
f"模型服务返回业务错误（code={status_code}）{detail}"
|
||||
)
|
||||
return choice
|
||||
|
||||
|
||||
def _message_content(payload: Mapping[str, Any]) -> str:
|
||||
choice = _raise_for_terminal_response(payload)
|
||||
message = choice.get("message")
|
||||
if not isinstance(message, Mapping):
|
||||
raise ModelGenerationError("模型响应缺少 choices[0].message")
|
||||
content = message.get("content")
|
||||
if isinstance(content, str):
|
||||
result = content
|
||||
elif isinstance(content, list):
|
||||
parts = [
|
||||
str(item.get("text") or "")
|
||||
for item in content
|
||||
if isinstance(item, Mapping) and item.get("type") in {None, "text", "output_text"}
|
||||
]
|
||||
result = "".join(parts)
|
||||
elif content is None:
|
||||
result = ""
|
||||
else:
|
||||
raise ModelGenerationError("模型响应 content 必须是文本")
|
||||
if not result.strip():
|
||||
raise ModelGenerationError("模型返回的最终内容为空，未生成可解析的 JSON")
|
||||
return result
|
||||
|
||||
|
||||
def _json_documents(content: str) -> list[Any]:
|
||||
decoder = json.JSONDecoder()
|
||||
documents: list[Any] = []
|
||||
cursor = 0
|
||||
while cursor < len(content):
|
||||
match = re.search(r"[\[{]", content[cursor:])
|
||||
if not match:
|
||||
break
|
||||
start = cursor + match.start()
|
||||
try:
|
||||
value, end = decoder.raw_decode(content[start:])
|
||||
except json.JSONDecodeError:
|
||||
cursor = start + 1
|
||||
continue
|
||||
if isinstance(value, (Mapping, list)):
|
||||
documents.append(value)
|
||||
cursor = start + max(end, 1)
|
||||
return documents
|
||||
|
||||
|
||||
def _json_payload(content: str) -> Any:
|
||||
# 只移除模型在 JSON 之前自行输出的思考过程，不能破坏 JSON 字段中的训练内容。
|
||||
cleaned = content.strip()
|
||||
if re.match(r"^\s*<think>", cleaned, flags=re.IGNORECASE) and not re.match(
|
||||
r"^\s*<think>[\s\S]*?</think>", cleaned, flags=re.IGNORECASE
|
||||
):
|
||||
raise ModelGenerationError("模型思考内容未闭合，响应可能已被截断")
|
||||
cleaned = re.sub(
|
||||
r"^\s*(?:<think>[\s\S]*?</think>\s*)+",
|
||||
"",
|
||||
cleaned,
|
||||
count=1,
|
||||
flags=re.IGNORECASE,
|
||||
).strip()
|
||||
fenced = re.fullmatch(r"```(?:json)?\s*([\s\S]*?)\s*```", cleaned, flags=re.IGNORECASE)
|
||||
if fenced:
|
||||
cleaned = fenced.group(1).strip()
|
||||
try:
|
||||
return json.loads(cleaned)
|
||||
except json.JSONDecodeError as direct_error:
|
||||
documents = _json_documents(cleaned)
|
||||
if len(documents) == 1:
|
||||
return documents[0]
|
||||
if len(documents) > 1:
|
||||
raise ModelGenerationError("模型响应包含多个 JSON 对象，无法确定应使用哪一个")
|
||||
raise ModelGenerationError(
|
||||
"模型响应中没有找到唯一且完整的 JSON 对象"
|
||||
f"（第 {direct_error.lineno} 行，第 {direct_error.colno} 列）"
|
||||
) from direct_error
|
||||
|
||||
|
||||
def _result_items(payload: Any) -> list[Mapping[str, Any]]:
|
||||
if isinstance(payload, list):
|
||||
values = payload
|
||||
elif isinstance(payload, Mapping):
|
||||
nested = next(
|
||||
(
|
||||
payload[key]
|
||||
for key in ("items", "results", "data", "records")
|
||||
if isinstance(payload.get(key), list)
|
||||
),
|
||||
None,
|
||||
)
|
||||
values = nested if isinstance(nested, list) else [payload]
|
||||
else:
|
||||
raise ModelGenerationError("model JSON must be an object or array")
|
||||
items = [item for item in values if isinstance(item, Mapping)]
|
||||
if not items:
|
||||
raise ModelGenerationError("model JSON does not contain result objects")
|
||||
return items
|
||||
|
||||
|
||||
def _prompt_messages(
|
||||
prompt: str,
|
||||
content: str,
|
||||
count: int,
|
||||
*,
|
||||
start_index: int,
|
||||
total_count: int,
|
||||
output_type: str,
|
||||
reasoning_detail: str,
|
||||
) -> list[dict[str, str]]:
|
||||
end_index = start_index + count - 1
|
||||
if output_type == OUTPUT_TYPE_REASONING:
|
||||
schema = '{"items":[{"instruction":"...","input":"...","reasoning":"...","answer":"..."}]}'
|
||||
detail_rule = (
|
||||
"推理详细程度为“详细”：完整展开问题条件、来源依据、中间计算或推导，"
|
||||
"并在得出答案前核对结论；每一步都必须能从来源内容中验证。"
|
||||
if reasoning_detail == REASONING_DETAIL_DETAILED
|
||||
else
|
||||
"推理详细程度为“普通”：只保留得出答案所需的关键依据和必要步骤，"
|
||||
"避免冗长复述、套话和无依据扩展。"
|
||||
)
|
||||
output_rule = (
|
||||
"你正在生成用于训练推理模型的思维链数据，而不是普通问答数据。"
|
||||
"instruction、reasoning 和 answer 均不得为空；reasoning 必须是基于来源内容、"
|
||||
f"可核对的推理过程，answer 只写最终答案。{detail_rule}"
|
||||
"这是思维链输出模式，即使其他提示语要求省略分析，也不得省略 reasoning。"
|
||||
"不要自行添加 <think> 标签，系统会在保存时统一组装。"
|
||||
)
|
||||
else:
|
||||
schema = '{"items":[{"instruction":"...","input":"...","output":"..."}]}'
|
||||
output_rule = (
|
||||
"你正在生成标准监督微调问答数据。instruction 和 output 不得为空；"
|
||||
"output 只写最终答案，禁止输出分析、推理过程或 <think> 标签。"
|
||||
)
|
||||
schema_instruction = (
|
||||
f"必须只返回 JSON 对象，格式为 {schema}；items 必须包含 {count} 条。"
|
||||
f"这是总计 {total_count} 条中的第 {start_index}-{end_index} 条，"
|
||||
"各条必须使用不同的提问角度和表述，避免重复。"
|
||||
f"{output_rule}不要输出 Markdown 代码围栏或 JSON 之外的说明。"
|
||||
)
|
||||
base_prompt = normalize_text(prompt) or "请根据来源内容生成可用于监督微调的问答数据。"
|
||||
if "{{ content }}" in base_prompt:
|
||||
user_prompt = base_prompt.replace("{{ content }}", content)
|
||||
return [
|
||||
{"role": "system", "content": schema_instruction},
|
||||
{"role": "user", "content": user_prompt},
|
||||
]
|
||||
return [
|
||||
{"role": "system", "content": f"{base_prompt}\n{schema_instruction}"},
|
||||
{"role": "user", "content": f"来源内容：\n{content}"},
|
||||
]
|
||||
|
||||
|
||||
def generate_model_records(
|
||||
preview_items: Iterable[Mapping[str, Any]],
|
||||
*,
|
||||
model: Mapping[str, Any],
|
||||
config: Mapping[str, Any],
|
||||
task_id: str,
|
||||
split: Mapping[str, int],
|
||||
qa_pairs_per_item: int,
|
||||
client: httpx.Client | None = None,
|
||||
on_progress: Callable[[int, int], None] | None = None,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""调用 OpenAI 兼容接口，将预览切片生成标准训练记录。
|
||||
|
||||
每个切片按安全批次调用模型；失败批次会产生一条可人工修复的
|
||||
invalid 结果，已经成功的批次不会丢失。
|
||||
"""
|
||||
|
||||
if not 1 <= qa_pairs_per_item <= MAX_QA_PAIRS_PER_ITEM:
|
||||
raise ModelGenerationError(f"qa_pairs_per_item must be in [1, {MAX_QA_PAIRS_PER_ITEM}]")
|
||||
output_type = str(config.get("output_type") or OUTPUT_TYPE_STANDARD).strip().lower()
|
||||
if output_type not in SUPPORTED_OUTPUT_TYPES:
|
||||
raise ModelGenerationError(f"output_type must be one of {sorted(SUPPORTED_OUTPUT_TYPES)}")
|
||||
reasoning_detail = str(
|
||||
config.get("reasoning_detail") or REASONING_DETAIL_NORMAL
|
||||
).strip().lower()
|
||||
if reasoning_detail not in SUPPORTED_REASONING_DETAILS:
|
||||
raise ModelGenerationError(
|
||||
f"reasoning_detail must be one of {sorted(SUPPORTED_REASONING_DETAILS)}"
|
||||
)
|
||||
endpoint = chat_completions_url(str(model.get("api_url") or ""))
|
||||
model_name = str(model.get("online_model_name") or model.get("name") or "").strip()
|
||||
if not model_name:
|
||||
raise ModelGenerationError("generation model name is required")
|
||||
is_minimax_m3 = _is_official_minimax_m3(endpoint, model_name)
|
||||
|
||||
temperature = float(config.get("temperature", 0.7))
|
||||
max_tokens = int(config.get("max_tokens", 1024))
|
||||
timeout = max(1.0, min(120.0, float(config.get("request_timeout_seconds", 60))))
|
||||
retries = max(0, min(5, int(config.get("generation_retries", 2))))
|
||||
headers = {"Content-Type": "application/json"}
|
||||
api_key = str(model.get("api_key") or "").strip()
|
||||
if api_key:
|
||||
headers["Authorization"] = f"Bearer {api_key}"
|
||||
|
||||
owns_client = client is None
|
||||
http_client = client or httpx.Client(timeout=timeout)
|
||||
results: list[dict[str, Any]] = []
|
||||
try:
|
||||
preview_list = list(preview_items)
|
||||
total_items = len(preview_list)
|
||||
for item_index, item in enumerate(preview_list):
|
||||
preview_id = str(item.get("id") or f"preview-{item_index + 1}")
|
||||
content = normalize_text(
|
||||
str(item.get("edited_content") or item.get("original_content") or "")
|
||||
)
|
||||
for batch_offset in range(0, qa_pairs_per_item, MODEL_GENERATION_BATCH_SIZE):
|
||||
batch_count = min(
|
||||
MODEL_GENERATION_BATCH_SIZE,
|
||||
qa_pairs_per_item - batch_offset,
|
||||
)
|
||||
batch_start = batch_offset + 1
|
||||
batch_end = batch_offset + batch_count
|
||||
request_payload: dict[str, Any] = {
|
||||
"model": model_name,
|
||||
"messages": _prompt_messages(
|
||||
str(config.get("generation_prompt") or ""),
|
||||
content,
|
||||
batch_count,
|
||||
start_index=batch_start,
|
||||
total_count=qa_pairs_per_item,
|
||||
output_type=output_type,
|
||||
reasoning_detail=reasoning_detail,
|
||||
),
|
||||
"temperature": temperature,
|
||||
}
|
||||
if is_minimax_m3:
|
||||
request_payload.update(
|
||||
reasoning_split=True,
|
||||
max_completion_tokens=max(
|
||||
max_tokens,
|
||||
MINIMAX_M3_MIN_COMPLETION_TOKENS,
|
||||
),
|
||||
)
|
||||
else:
|
||||
request_payload["max_tokens"] = max_tokens
|
||||
if bool(config.get("json_mode", False)) and not is_minimax_m3:
|
||||
request_payload["response_format"] = {"type": "json_object"}
|
||||
|
||||
last_error: Exception | None = None
|
||||
generated_items: list[Mapping[str, Any]] | None = None
|
||||
for _ in range(retries + 1):
|
||||
try:
|
||||
response = http_client.post(
|
||||
endpoint,
|
||||
headers=headers,
|
||||
json=request_payload,
|
||||
)
|
||||
response.raise_for_status()
|
||||
body = response.json()
|
||||
if not isinstance(body, Mapping):
|
||||
raise ModelGenerationError("model response body must be a JSON object")
|
||||
try:
|
||||
candidate_items = _result_items(
|
||||
_json_payload(_message_content(body))
|
||||
)
|
||||
except ModelGenerationError as exc:
|
||||
logger.warning(
|
||||
"data process model response rejected task_id=%s model=%s "
|
||||
"finish_reason=%s response_chars=%s input_sensitive=%s "
|
||||
"output_sensitive=%s reason=%s",
|
||||
task_id,
|
||||
model_name,
|
||||
_response_finish_reason(body) or "missing",
|
||||
_response_content_length(body),
|
||||
bool(body.get("input_sensitive")),
|
||||
bool(body.get("output_sensitive")),
|
||||
str(exc),
|
||||
)
|
||||
raise
|
||||
if len(candidate_items) < batch_count:
|
||||
raise ModelGenerationError(
|
||||
"model response contains fewer result objects than requested: "
|
||||
f"expected {batch_count}, got {len(candidate_items)}"
|
||||
)
|
||||
generated_items = candidate_items
|
||||
break
|
||||
except (
|
||||
httpx.HTTPError,
|
||||
json.JSONDecodeError,
|
||||
ModelGenerationError,
|
||||
) as exc:
|
||||
last_error = exc
|
||||
if not _is_retryable_generation_error(exc):
|
||||
break
|
||||
|
||||
if generated_items is None:
|
||||
error_message = str(last_error or "model generation failed")[:2000]
|
||||
failure_instruction = (
|
||||
f"模型生成失败，请人工补充（第 {batch_start}-{batch_end} 条）"
|
||||
)
|
||||
result_id = (
|
||||
"result_"
|
||||
f"{hashlib.sha256(f'{preview_id}:error:{batch_start}'.encode()).hexdigest()[:16]}"
|
||||
)
|
||||
results.append(
|
||||
{
|
||||
"id": result_id,
|
||||
"preview_item_id": preview_id,
|
||||
"instruction": failure_instruction,
|
||||
"input": content,
|
||||
"output": "",
|
||||
"original_instruction": failure_instruction,
|
||||
"original_input": content,
|
||||
"original_output": "",
|
||||
"status": "invalid",
|
||||
"error": error_message,
|
||||
"split": "train",
|
||||
}
|
||||
)
|
||||
continue
|
||||
|
||||
for batch_index, value in enumerate(generated_items[:batch_count]):
|
||||
variant_index = batch_offset + batch_index
|
||||
instruction = normalize_text(
|
||||
str(value.get("instruction") or value.get("question") or "")
|
||||
)
|
||||
input_text = normalize_text(
|
||||
str(value.get("input") or value.get("context") or "")
|
||||
)
|
||||
if output_type == OUTPUT_TYPE_REASONING:
|
||||
reasoning = normalize_text(
|
||||
re.sub(
|
||||
r"</?think>",
|
||||
"",
|
||||
str(value.get("reasoning") or value.get("analysis") or ""),
|
||||
flags=re.IGNORECASE,
|
||||
)
|
||||
)
|
||||
answer = normalize_text(
|
||||
re.sub(
|
||||
r"</?think>",
|
||||
"",
|
||||
str(
|
||||
value.get("answer")
|
||||
or value.get("final_answer")
|
||||
or value.get("output")
|
||||
or ""
|
||||
),
|
||||
flags=re.IGNORECASE,
|
||||
)
|
||||
)
|
||||
output = (
|
||||
f"<think>\n{reasoning}\n</think>\n{answer}"
|
||||
if reasoning and answer
|
||||
else answer or (f"<think>\n{reasoning}\n</think>" if reasoning else "")
|
||||
)
|
||||
valid = bool(instruction and reasoning and answer)
|
||||
missing_error = "model result is missing instruction, reasoning or answer"
|
||||
else:
|
||||
output = normalize_text(
|
||||
str(
|
||||
value.get("output")
|
||||
or value.get("answer")
|
||||
or value.get("response")
|
||||
or ""
|
||||
)
|
||||
)
|
||||
output = normalize_text(
|
||||
re.sub(
|
||||
r"<think>[\s\S]*?(?:</think>|$)",
|
||||
"",
|
||||
output,
|
||||
flags=re.IGNORECASE,
|
||||
)
|
||||
)
|
||||
valid = bool(instruction and output)
|
||||
missing_error = "model result is missing instruction or output"
|
||||
raw_id = f"{preview_id}:{variant_index + 1}:{instruction}:{output}"
|
||||
result_id = f"result_{hashlib.sha256(raw_id.encode()).hexdigest()[:16]}"
|
||||
results.append(
|
||||
{
|
||||
"id": result_id,
|
||||
"preview_item_id": preview_id,
|
||||
"instruction": instruction,
|
||||
"input": input_text,
|
||||
"output": output,
|
||||
"original_instruction": instruction,
|
||||
"original_input": input_text,
|
||||
"original_output": output,
|
||||
"status": "valid" if valid else "invalid",
|
||||
"error": (None if valid else missing_error),
|
||||
"split": "train",
|
||||
}
|
||||
)
|
||||
if on_progress:
|
||||
on_progress(item_index + 1, total_items)
|
||||
finally:
|
||||
if owns_client:
|
||||
http_client.close()
|
||||
assignments = stable_split_assignments(
|
||||
[str(result["id"]) for result in results],
|
||||
split,
|
||||
seed=task_id,
|
||||
)
|
||||
for result, assignment in zip(results, assignments, strict=True):
|
||||
result["split"] = assignment
|
||||
return results
|
||||
|
||||
|
||||
__all__ = ["ModelGenerationError", "chat_completions_url", "generate_model_records"]
|
||||
@@ -1,308 +0,0 @@
|
||||
"""Word 与 Excel 原文件的安全、受限预览模型。
|
||||
|
||||
预览只返回浏览器绘制所需的结构化数据，不返回或执行 Office 包中的活动内容。
|
||||
DOCX 的字符偏移与上传时的正文抽取规则保持一致，供前端定位当前切片。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import io
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
from docx import Document
|
||||
from docx.oxml.table import CT_Tbl
|
||||
from docx.oxml.text.paragraph import CT_P
|
||||
from docx.table import Table
|
||||
from docx.text.paragraph import Paragraph
|
||||
from openpyxl import load_workbook
|
||||
|
||||
from app.modules.data_process.algorithms import (
|
||||
_MAX_WORKBOOK_COLUMNS,
|
||||
_MAX_WORKBOOK_HEADER_SCAN_ROWS,
|
||||
_infer_xlsx_header_region,
|
||||
_normalize_spreadsheet_value,
|
||||
_rewrite_xlsx_workbook_relationships,
|
||||
_validate_office_archive,
|
||||
_xlsx_sheet_merge_ranges,
|
||||
normalize_text,
|
||||
)
|
||||
|
||||
MAX_DOCX_PREVIEW_BLOCKS = 2_000
|
||||
MAX_XLSX_PREVIEW_ROWS = 200
|
||||
|
||||
|
||||
def _docx_alignment(paragraph: Paragraph) -> str:
|
||||
value = paragraph.alignment
|
||||
return {
|
||||
0: "left",
|
||||
1: "center",
|
||||
2: "right",
|
||||
3: "justify",
|
||||
4: "distribute",
|
||||
5: "justify",
|
||||
7: "justify",
|
||||
8: "distribute",
|
||||
9: "distribute",
|
||||
}.get(int(value) if value is not None else -1, "left")
|
||||
|
||||
|
||||
def _docx_heading_level(paragraph: Paragraph) -> int | None:
|
||||
style = paragraph.style
|
||||
if style is None:
|
||||
return None
|
||||
style_name = str(style.name or "")
|
||||
style_id = str(style.style_id or "")
|
||||
match = re.search(r"(?:heading|标题)\s*([1-6])", f"{style_name} {style_id}", re.IGNORECASE)
|
||||
return int(match.group(1)) if match else None
|
||||
|
||||
|
||||
def build_docx_preview(raw: bytes) -> dict[str, Any]:
|
||||
"""把 DOCX 转为保留标题、段落和表格顺序的浏览器预览模型。"""
|
||||
|
||||
_validate_office_archive(raw, "docx")
|
||||
try:
|
||||
document = Document(io.BytesIO(raw))
|
||||
except Exception as exc:
|
||||
raise ValueError(f"invalid DOCX file: {exc}") from exc
|
||||
|
||||
blocks: list[dict[str, Any]] = []
|
||||
source_cursor = 0
|
||||
has_source_content = False
|
||||
rendered_blocks = 0
|
||||
truncated = False
|
||||
|
||||
def source_range(value: str) -> tuple[str, int, int] | None:
|
||||
nonlocal source_cursor, has_source_content
|
||||
text = normalize_text(value)
|
||||
if not text:
|
||||
return None
|
||||
if has_source_content:
|
||||
source_cursor += 2
|
||||
start = source_cursor
|
||||
source_cursor += len(text)
|
||||
has_source_content = True
|
||||
return text, start, source_cursor
|
||||
|
||||
for child in document.element.body.iterchildren():
|
||||
if rendered_blocks >= MAX_DOCX_PREVIEW_BLOCKS:
|
||||
truncated = True
|
||||
break
|
||||
|
||||
if isinstance(child, CT_P):
|
||||
paragraph = Paragraph(child, document)
|
||||
located = source_range(paragraph.text)
|
||||
if located is None:
|
||||
continue
|
||||
text, start, end = located
|
||||
style_name = str(paragraph.style.name or "") if paragraph.style else ""
|
||||
blocks.append(
|
||||
{
|
||||
"type": "paragraph",
|
||||
"text": text,
|
||||
"style": style_name,
|
||||
"heading_level": _docx_heading_level(paragraph),
|
||||
"alignment": _docx_alignment(paragraph),
|
||||
"is_list": "list" in style_name.casefold() or "列表" in style_name,
|
||||
"source_start": start,
|
||||
"source_end": end,
|
||||
}
|
||||
)
|
||||
rendered_blocks += 1
|
||||
continue
|
||||
|
||||
if not isinstance(child, CT_Tbl):
|
||||
continue
|
||||
table = Table(child, document)
|
||||
preview_rows: list[dict[str, Any]] = []
|
||||
for row in table.rows:
|
||||
if rendered_blocks >= MAX_DOCX_PREVIEW_BLOCKS:
|
||||
truncated = True
|
||||
break
|
||||
cell_values = [normalize_text(cell.text) for cell in row.cells]
|
||||
located = source_range("\t".join(cell_values))
|
||||
if located is None:
|
||||
continue
|
||||
_, start, end = located
|
||||
preview_rows.append(
|
||||
{
|
||||
"cells": cell_values,
|
||||
"source_start": start,
|
||||
"source_end": end,
|
||||
}
|
||||
)
|
||||
rendered_blocks += 1
|
||||
if preview_rows:
|
||||
blocks.append({"type": "table", "rows": preview_rows})
|
||||
if truncated:
|
||||
break
|
||||
|
||||
return {
|
||||
"format": "docx",
|
||||
"blocks": blocks,
|
||||
"truncated": truncated,
|
||||
}
|
||||
|
||||
|
||||
def build_xlsx_preview(
|
||||
raw: bytes,
|
||||
*,
|
||||
sheet_index: int = 0,
|
||||
offset: int = 0,
|
||||
limit: int = 100,
|
||||
) -> dict[str, Any]:
|
||||
"""按工作表分页返回 XLSX 的表头和记录网格。"""
|
||||
|
||||
if sheet_index < 0 or offset < 0:
|
||||
raise ValueError("sheet_index and offset must be non-negative")
|
||||
if limit < 1 or limit > MAX_XLSX_PREVIEW_ROWS:
|
||||
raise ValueError(
|
||||
f"XLSX preview limit must be in [1, {MAX_XLSX_PREVIEW_ROWS}]"
|
||||
)
|
||||
|
||||
_validate_office_archive(raw, "xlsx")
|
||||
merged_by_sheet, normalized_targets = _xlsx_sheet_merge_ranges(raw)
|
||||
workbook_raw = (
|
||||
_rewrite_xlsx_workbook_relationships(raw, normalized_targets)
|
||||
if normalized_targets
|
||||
else raw
|
||||
)
|
||||
try:
|
||||
workbook = load_workbook(
|
||||
io.BytesIO(workbook_raw),
|
||||
read_only=True,
|
||||
data_only=True,
|
||||
keep_links=False,
|
||||
)
|
||||
except Exception as exc:
|
||||
raise ValueError(f"invalid XLSX file: {exc}") from exc
|
||||
|
||||
try:
|
||||
sheets = [
|
||||
{
|
||||
"index": index,
|
||||
"name": worksheet.title,
|
||||
"state": worksheet.sheet_state,
|
||||
}
|
||||
for index, worksheet in enumerate(workbook.worksheets)
|
||||
]
|
||||
if not sheets:
|
||||
raise ValueError("XLSX workbook contains no worksheets")
|
||||
if sheet_index >= len(sheets):
|
||||
raise ValueError("XLSX worksheet index is out of range")
|
||||
|
||||
worksheet = workbook.worksheets[sheet_index]
|
||||
reset_dimensions = getattr(worksheet, "reset_dimensions", None)
|
||||
if callable(reset_dimensions):
|
||||
reset_dimensions()
|
||||
row_iterator = enumerate(worksheet.iter_rows(values_only=True), start=1)
|
||||
buffered_rows: dict[int, tuple[Any, ...]] = {}
|
||||
|
||||
def normalized_values(row: tuple[Any, ...]) -> list[Any]:
|
||||
values = list(row)
|
||||
while values and values[-1] in {None, ""}:
|
||||
values.pop()
|
||||
if len(values) > _MAX_WORKBOOK_COLUMNS:
|
||||
raise ValueError(
|
||||
f"XLSX worksheet {worksheet.title!r} exceeds "
|
||||
f"{_MAX_WORKBOOK_COLUMNS} columns"
|
||||
)
|
||||
return values
|
||||
|
||||
for row_number, row in row_iterator:
|
||||
values = normalized_values(row)
|
||||
if not values or all(value in {None, ""} for value in values):
|
||||
continue
|
||||
buffered_rows[row_number] = tuple(values)
|
||||
if len(buffered_rows) >= _MAX_WORKBOOK_HEADER_SCAN_ROWS:
|
||||
break
|
||||
|
||||
if not buffered_rows:
|
||||
return {
|
||||
"format": "xlsx",
|
||||
"sheets": sheets,
|
||||
"active_sheet": {
|
||||
"index": sheet_index,
|
||||
"name": worksheet.title,
|
||||
"columns": [],
|
||||
"rows": [],
|
||||
"offset": offset,
|
||||
"limit": limit,
|
||||
"has_more": False,
|
||||
},
|
||||
}
|
||||
|
||||
_, header_end_row, headers = _infer_xlsx_header_region(
|
||||
worksheet.title,
|
||||
buffered_rows,
|
||||
merged_by_sheet.get(worksheet.title, ()),
|
||||
)
|
||||
preview_rows: list[dict[str, Any]] = []
|
||||
record_index = 0
|
||||
has_more = False
|
||||
|
||||
def append_row(row_number: int, values: tuple[Any, ...] | list[Any]) -> bool:
|
||||
nonlocal record_index, has_more
|
||||
row_values = list(values)
|
||||
if len(row_values) > len(headers):
|
||||
raise ValueError(
|
||||
f"XLSX worksheet {worksheet.title!r} has a row wider than its header"
|
||||
)
|
||||
row_values.extend([None] * (len(headers) - len(row_values)))
|
||||
record = {
|
||||
header: _normalize_spreadsheet_value(value)
|
||||
for header, value in zip(headers, row_values, strict=True)
|
||||
}
|
||||
if not any(value not in {"", None} for value in record.values()):
|
||||
return False
|
||||
current_index = record_index
|
||||
record_index += 1
|
||||
if current_index < offset:
|
||||
return False
|
||||
if len(preview_rows) >= limit:
|
||||
has_more = True
|
||||
return True
|
||||
preview_rows.append(
|
||||
{
|
||||
"row_number": row_number,
|
||||
"record_index": current_index,
|
||||
"values": [record[header] for header in headers],
|
||||
"record": record,
|
||||
}
|
||||
)
|
||||
return False
|
||||
|
||||
for row_number, values in buffered_rows.items():
|
||||
if row_number > header_end_row and append_row(row_number, values):
|
||||
break
|
||||
else:
|
||||
for row_number, row in row_iterator:
|
||||
values = normalized_values(row)
|
||||
if not values or all(value in {None, ""} for value in values):
|
||||
continue
|
||||
if append_row(row_number, values):
|
||||
break
|
||||
|
||||
return {
|
||||
"format": "xlsx",
|
||||
"sheets": sheets,
|
||||
"active_sheet": {
|
||||
"index": sheet_index,
|
||||
"name": worksheet.title,
|
||||
"columns": headers,
|
||||
"rows": preview_rows,
|
||||
"offset": offset,
|
||||
"limit": limit,
|
||||
"has_more": has_more,
|
||||
},
|
||||
}
|
||||
finally:
|
||||
workbook.close()
|
||||
|
||||
|
||||
__all__ = [
|
||||
"MAX_DOCX_PREVIEW_BLOCKS",
|
||||
"MAX_XLSX_PREVIEW_ROWS",
|
||||
"build_docx_preview",
|
||||
"build_xlsx_preview",
|
||||
]
|
||||
@@ -1,76 +0,0 @@
|
||||
"""数据处理运行表的显式检查与安装命令。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from app.modules.data_process.store import DataProcessStore
|
||||
|
||||
REQUIRED_TASK_COLUMNS = (
|
||||
"generation_run_id",
|
||||
"results_confirmed",
|
||||
"workflow_step",
|
||||
"preview_status",
|
||||
"preview_progress",
|
||||
"preview_run_id",
|
||||
"preview_failure_reason",
|
||||
"preview_total_files",
|
||||
"preview_completed_files",
|
||||
)
|
||||
|
||||
|
||||
def _target_label(database_url: str) -> str:
|
||||
parsed = urlsplit(database_url)
|
||||
database = parsed.path.strip("/") or "(unknown)"
|
||||
return f"{parsed.hostname or '(unknown)'}:{parsed.port or 5432}/{database}"
|
||||
|
||||
|
||||
def _schema_ready(store: DataProcessStore) -> bool:
|
||||
with store.connect() as conn:
|
||||
row = conn.execute(
|
||||
"""
|
||||
SELECT COUNT(*) = %s AS ready
|
||||
FROM information_schema.columns
|
||||
WHERE table_schema=current_schema()
|
||||
AND table_name='data_process_tasks'
|
||||
AND column_name = ANY(%s)
|
||||
""",
|
||||
(len(REQUIRED_TASK_COLUMNS), list(REQUIRED_TASK_COLUMNS)),
|
||||
).fetchone()
|
||||
return bool(row and row["ready"])
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="检查或显式安装数据处理运行表（不会由应用启动自动执行）"
|
||||
)
|
||||
action = parser.add_mutually_exclusive_group(required=True)
|
||||
action.add_argument("--check", action="store_true", help="只读检查迁移是否已安装")
|
||||
action.add_argument("--apply", action="store_true", help="执行 002 数据处理迁移")
|
||||
parser.add_argument(
|
||||
"--yes",
|
||||
action="store_true",
|
||||
help="确认允许修改 DATABASE_URL 指向的数据库；与 --apply 同时使用",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
store = DataProcessStore()
|
||||
target = _target_label(store.database_url)
|
||||
if args.check:
|
||||
ready = _schema_ready(store)
|
||||
print(f"数据处理 schema：{'已安装' if ready else '未安装'}；目标：{target}")
|
||||
return 0 if ready else 1
|
||||
if not args.yes:
|
||||
parser.error("--apply 必须同时提供 --yes，确认修改目标数据库")
|
||||
|
||||
print(f"正在安装数据处理 schema；目标：{target}")
|
||||
store.ensure_schema()
|
||||
if not _schema_ready(store):
|
||||
raise RuntimeError("迁移执行后仍未检测到 generation_run_id")
|
||||
print("数据处理 schema 安装完成")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -1,560 +0,0 @@
|
||||
"""数据处理原始源文件的受控本地对象存储。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import re
|
||||
import stat
|
||||
import unicodedata
|
||||
import uuid
|
||||
from dataclasses import dataclass
|
||||
from functools import lru_cache
|
||||
from pathlib import Path, PurePosixPath
|
||||
from typing import Iterable, Iterator
|
||||
from urllib.parse import quote, unquote, urlsplit
|
||||
|
||||
|
||||
class DataProcessStorageError(ValueError):
|
||||
"""本地对象引用或文件系统状态不安全。"""
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class StagedSourceObject:
|
||||
"""尚未发布的原始文件；绝对路径仅在存储模块内部流转。"""
|
||||
|
||||
reference: str
|
||||
_temporary_path: Path
|
||||
_relative_path: PurePosixPath
|
||||
|
||||
|
||||
def _default_storage_root() -> Path:
|
||||
return Path(__file__).resolve().parents[3] / "storage" / "data-process"
|
||||
|
||||
|
||||
def _configured_storage_root() -> Path:
|
||||
configured = os.getenv("DATA_PROCESS_STORAGE_DIR", "").strip()
|
||||
if not configured:
|
||||
return _default_storage_root()
|
||||
path = Path(configured).expanduser()
|
||||
# 相对配置固定以 backend 目录为基准，
|
||||
# 避免从不同 cwd 启动时写入不同位置。
|
||||
return path if path.is_absolute() else Path(__file__).resolve().parents[3] / path
|
||||
|
||||
|
||||
def _safe_component(value: str, label: str) -> str:
|
||||
if not value or value in {".", ".."} or len(value) > 128:
|
||||
raise DataProcessStorageError(f"invalid {label}")
|
||||
if not value[0].isalnum() or any(
|
||||
not (character.isalnum() or character in {"-", "_", "."})
|
||||
for character in value
|
||||
):
|
||||
raise DataProcessStorageError(f"invalid {label}")
|
||||
return value
|
||||
|
||||
|
||||
def _safe_basename(value: str) -> str:
|
||||
if not value or len(value.encode("utf-8")) > 255:
|
||||
raise DataProcessStorageError("invalid source file name")
|
||||
if value != Path(value).name or "/" in value or "\\" in value or "\x00" in value:
|
||||
raise DataProcessStorageError("invalid source file name")
|
||||
if value in {".", ".."} or any(
|
||||
unicodedata.category(character).startswith("C") for character in value
|
||||
):
|
||||
raise DataProcessStorageError("invalid source file name")
|
||||
return value
|
||||
|
||||
|
||||
class LocalDataProcessStorage:
|
||||
"""只允许访问配置根目录下的版本化原始文件。"""
|
||||
|
||||
def __init__(self, root: str | os.PathLike[str] | Path | None = None) -> None:
|
||||
configured = Path(root) if root is not None else _configured_storage_root()
|
||||
configured = configured.expanduser()
|
||||
if configured.exists() and configured.is_symlink():
|
||||
raise DataProcessStorageError("data process storage root must not be a symlink")
|
||||
configured.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
self._root = configured.resolve(strict=True)
|
||||
# StagedSourceObject 本身是普通 dataclass，不能只依赖其中的路径字段判断
|
||||
# 来源；只接受由当前存储实例实际签发的对象，
|
||||
# 避免调用方伪造暂存路径。
|
||||
self._issued_staged_objects: dict[Path, StagedSourceObject] = {}
|
||||
self._ensure_directory(self._root / ".staging")
|
||||
|
||||
@property
|
||||
def root(self) -> Path:
|
||||
"""仅供运维和测试检查；API 响应不得序列化该属性。"""
|
||||
|
||||
return self._root
|
||||
|
||||
def new_batch_id(self) -> str:
|
||||
return f"batch-{uuid.uuid4().hex}"
|
||||
|
||||
def stage_bytes(
|
||||
self,
|
||||
*,
|
||||
batch_id: str,
|
||||
task_id: str,
|
||||
source_file_id: str,
|
||||
version: int,
|
||||
name: str,
|
||||
content: bytes,
|
||||
) -> StagedSourceObject:
|
||||
batch_id = _safe_component(batch_id, "batch id")
|
||||
task_id = _safe_component(task_id, "task id")
|
||||
source_file_id = _safe_component(source_file_id, "source file id")
|
||||
if isinstance(version, bool) or not isinstance(version, int) or version < 1:
|
||||
raise DataProcessStorageError("invalid source file version")
|
||||
basename = _safe_basename(name)
|
||||
if not isinstance(content, bytes):
|
||||
raise TypeError("content must be bytes")
|
||||
|
||||
batch_directory = self._ensure_directory(self._root / ".staging" / batch_id)
|
||||
temporary_path = batch_directory / f"{source_file_id}-{uuid.uuid4().hex}.tmp"
|
||||
flags = os.O_CREAT | os.O_EXCL | os.O_WRONLY
|
||||
if hasattr(os, "O_NOFOLLOW"):
|
||||
flags |= os.O_NOFOLLOW
|
||||
descriptor = os.open(temporary_path, flags, 0o600)
|
||||
try:
|
||||
with os.fdopen(descriptor, "wb", closefd=True) as stream:
|
||||
stream.write(content)
|
||||
stream.flush()
|
||||
os.fsync(stream.fileno())
|
||||
except Exception:
|
||||
temporary_path.unlink(missing_ok=True)
|
||||
raise
|
||||
|
||||
relative_path = PurePosixPath(
|
||||
task_id,
|
||||
source_file_id,
|
||||
f"v{version}",
|
||||
basename,
|
||||
)
|
||||
reference = (
|
||||
"local://data-process/"
|
||||
f"{task_id}/{source_file_id}/v{version}/{quote(basename, safe='')}"
|
||||
)
|
||||
staged = StagedSourceObject(reference, temporary_path, relative_path)
|
||||
self._issued_staged_objects[temporary_path] = staged
|
||||
return staged
|
||||
|
||||
def stage_copy(
|
||||
self,
|
||||
*,
|
||||
batch_id: str,
|
||||
source_reference: str,
|
||||
expected_source_task_id: str,
|
||||
expected_source_file_id: str,
|
||||
task_id: str,
|
||||
source_file_id: str,
|
||||
version: int,
|
||||
name: str,
|
||||
) -> StagedSourceObject:
|
||||
"""为不可变源对象创建独立目录项，不把大文件重新读入内存。"""
|
||||
|
||||
batch_id = _safe_component(batch_id, "batch id")
|
||||
task_id = _safe_component(task_id, "task id")
|
||||
source_file_id = _safe_component(source_file_id, "source file id")
|
||||
if isinstance(version, bool) or not isinstance(version, int) or version < 1:
|
||||
raise DataProcessStorageError("invalid source file version")
|
||||
basename = _safe_basename(name)
|
||||
source_relative = self._relative_from_reference(source_reference)
|
||||
if source_relative is None:
|
||||
raise DataProcessStorageError("original source object is not available")
|
||||
self._assert_expected_owner(
|
||||
source_relative,
|
||||
expected_task_id=expected_source_task_id,
|
||||
expected_source_file_id=expected_source_file_id,
|
||||
)
|
||||
descriptor, source_info = self._open_read_descriptor(source_relative)
|
||||
os.close(descriptor)
|
||||
|
||||
batch_directory = self._ensure_directory(self._root / ".staging" / batch_id)
|
||||
temporary_path = batch_directory / f"{source_file_id}-{uuid.uuid4().hex}.tmp"
|
||||
source_path = self._path_for_relative(source_relative)
|
||||
try:
|
||||
os.link(source_path, temporary_path, follow_symlinks=False)
|
||||
copy_info = temporary_path.lstat()
|
||||
if (
|
||||
not stat.S_ISREG(copy_info.st_mode)
|
||||
or source_info.st_dev != copy_info.st_dev
|
||||
or source_info.st_ino != copy_info.st_ino
|
||||
):
|
||||
raise DataProcessStorageError("source storage object changed while copying")
|
||||
except Exception:
|
||||
temporary_path.unlink(missing_ok=True)
|
||||
raise
|
||||
|
||||
relative_path = PurePosixPath(
|
||||
task_id,
|
||||
source_file_id,
|
||||
f"v{version}",
|
||||
basename,
|
||||
)
|
||||
reference = (
|
||||
"local://data-process/"
|
||||
f"{task_id}/{source_file_id}/v{version}/{quote(basename, safe='')}"
|
||||
)
|
||||
staged = StagedSourceObject(reference, temporary_path, relative_path)
|
||||
self._issued_staged_objects[temporary_path] = staged
|
||||
return staged
|
||||
|
||||
def publish(self, objects: Iterable[StagedSourceObject]) -> None:
|
||||
staged = list(objects)
|
||||
published: list[StagedSourceObject] = []
|
||||
try:
|
||||
seen_temporary_paths: set[Path] = set()
|
||||
for item in staged:
|
||||
self._validate_staged_object(item, require_file=True)
|
||||
if item._temporary_path in seen_temporary_paths:
|
||||
raise DataProcessStorageError("duplicate staged source object")
|
||||
seen_temporary_paths.add(item._temporary_path)
|
||||
for item in staged:
|
||||
final_path = self._path_for_relative(item._relative_path)
|
||||
self._ensure_directory(final_path.parent)
|
||||
if final_path.exists() or final_path.is_symlink():
|
||||
raise DataProcessStorageError("source storage object already exists")
|
||||
os.link(item._temporary_path, final_path, follow_symlinks=False)
|
||||
published.append(item)
|
||||
item._temporary_path.unlink()
|
||||
self._fsync_directory(final_path.parent)
|
||||
except Exception:
|
||||
for item in reversed(published):
|
||||
try:
|
||||
self.delete(item.reference)
|
||||
except Exception:
|
||||
# 回滚必须尽量处理其余对象，并保留真正的发布异常。
|
||||
pass
|
||||
for item in staged:
|
||||
try:
|
||||
self.discard([item])
|
||||
except Exception:
|
||||
pass
|
||||
raise
|
||||
self.discard(staged)
|
||||
|
||||
def discard(self, objects: Iterable[StagedSourceObject]) -> None:
|
||||
staged = list(objects)
|
||||
for item in staged:
|
||||
self._validate_staged_object(item, require_file=False)
|
||||
|
||||
batch_directories: set[Path] = set()
|
||||
first_error: Exception | None = None
|
||||
for item in staged:
|
||||
temporary_path = item._temporary_path
|
||||
try:
|
||||
temporary_path.unlink(missing_ok=True)
|
||||
except Exception as exc:
|
||||
if first_error is None:
|
||||
first_error = exc
|
||||
else:
|
||||
self._issued_staged_objects.pop(temporary_path, None)
|
||||
batch_directories.add(temporary_path.parent)
|
||||
for directory in batch_directories:
|
||||
self._remove_empty_directory(directory)
|
||||
if first_error is not None:
|
||||
raise first_error
|
||||
|
||||
def read(self, reference: str) -> bytes | None:
|
||||
"""读取 local 引用；旧 ``db://`` 对象返回 ``None`` 由数据库正文兜底。"""
|
||||
|
||||
relative_path = self._relative_from_reference(reference)
|
||||
if relative_path is None:
|
||||
return None
|
||||
descriptor, _ = self._open_read_descriptor(relative_path)
|
||||
with os.fdopen(descriptor, "rb", closefd=True) as stream:
|
||||
return stream.read()
|
||||
|
||||
def file_size(
|
||||
self,
|
||||
reference: str,
|
||||
*,
|
||||
expected_task_id: str,
|
||||
expected_source_file_id: str,
|
||||
) -> int | None:
|
||||
"""返回受控 local 对象大小；旧 ``db://`` 对象没有原始文件。"""
|
||||
|
||||
relative_path = self._relative_from_reference(reference)
|
||||
if relative_path is None:
|
||||
return None
|
||||
self._assert_expected_owner(
|
||||
relative_path,
|
||||
expected_task_id=expected_task_id,
|
||||
expected_source_file_id=expected_source_file_id,
|
||||
)
|
||||
descriptor, info = self._open_read_descriptor(relative_path)
|
||||
os.close(descriptor)
|
||||
return info.st_size
|
||||
|
||||
def iter_bytes(
|
||||
self,
|
||||
reference: str,
|
||||
*,
|
||||
expected_task_id: str,
|
||||
expected_source_file_id: str,
|
||||
expected_size: int,
|
||||
start: int = 0,
|
||||
length: int | None = None,
|
||||
chunk_size: int = 256 * 1024,
|
||||
) -> Iterator[bytes]:
|
||||
"""按范围流式读取原始文件，避免 PDF 预览把大文件整体载入内存。"""
|
||||
|
||||
relative_path = self._relative_from_reference(reference)
|
||||
if relative_path is None:
|
||||
raise DataProcessStorageError("original source object is not available")
|
||||
self._assert_expected_owner(
|
||||
relative_path,
|
||||
expected_task_id=expected_task_id,
|
||||
expected_source_file_id=expected_source_file_id,
|
||||
)
|
||||
if start < 0 or expected_size < 0 or chunk_size < 1:
|
||||
raise DataProcessStorageError("invalid source byte range")
|
||||
descriptor, info = self._open_read_descriptor(relative_path)
|
||||
if info.st_size != expected_size:
|
||||
os.close(descriptor)
|
||||
raise DataProcessStorageError("source object size does not match metadata")
|
||||
remaining = expected_size - start if length is None else length
|
||||
if remaining < 0 or start + remaining > expected_size:
|
||||
os.close(descriptor)
|
||||
raise DataProcessStorageError("invalid source byte range")
|
||||
with os.fdopen(descriptor, "rb", closefd=True) as stream:
|
||||
stream.seek(start)
|
||||
while remaining:
|
||||
chunk = stream.read(min(chunk_size, remaining))
|
||||
if not chunk:
|
||||
raise DataProcessStorageError("source object ended unexpectedly")
|
||||
remaining -= len(chunk)
|
||||
yield chunk
|
||||
|
||||
def validate_owner(
|
||||
self,
|
||||
reference: str,
|
||||
*,
|
||||
expected_task_id: str,
|
||||
expected_source_file_id: str,
|
||||
) -> bool:
|
||||
"""校验 local 引用归属；旧 ``db://`` 引用无需文件系统处理。"""
|
||||
|
||||
relative_path = self._relative_from_reference(reference)
|
||||
if relative_path is None:
|
||||
return False
|
||||
self._assert_expected_owner(
|
||||
relative_path,
|
||||
expected_task_id=expected_task_id,
|
||||
expected_source_file_id=expected_source_file_id,
|
||||
)
|
||||
return True
|
||||
|
||||
def _open_read_descriptor(
|
||||
self,
|
||||
relative_path: PurePosixPath,
|
||||
) -> tuple[int, os.stat_result]:
|
||||
path = self._path_for_relative(relative_path)
|
||||
self._assert_controlled_parent(path)
|
||||
try:
|
||||
before_open = path.lstat()
|
||||
except FileNotFoundError as exc:
|
||||
raise DataProcessStorageError("source storage object does not exist") from exc
|
||||
if stat.S_ISLNK(before_open.st_mode) or not stat.S_ISREG(before_open.st_mode):
|
||||
raise DataProcessStorageError("source storage object is not a regular file")
|
||||
flags = os.O_RDONLY
|
||||
if hasattr(os, "O_NOFOLLOW"):
|
||||
flags |= os.O_NOFOLLOW
|
||||
descriptor = os.open(path, flags)
|
||||
after_open = os.fstat(descriptor)
|
||||
if (
|
||||
not stat.S_ISREG(after_open.st_mode)
|
||||
or before_open.st_dev != after_open.st_dev
|
||||
or before_open.st_ino != after_open.st_ino
|
||||
):
|
||||
os.close(descriptor)
|
||||
raise DataProcessStorageError("source storage object changed while opening")
|
||||
return descriptor, after_open
|
||||
|
||||
def delete(
|
||||
self,
|
||||
reference: str,
|
||||
*,
|
||||
expected_task_id: str | None = None,
|
||||
expected_source_file_id: str | None = None,
|
||||
) -> bool:
|
||||
"""删除受控 local 对象；旧 ``db://`` 引用保持不变。"""
|
||||
|
||||
relative_path = self._relative_from_reference(reference)
|
||||
if relative_path is None:
|
||||
return False
|
||||
if (expected_task_id is None) != (expected_source_file_id is None):
|
||||
raise DataProcessStorageError("both expected storage owner fields are required")
|
||||
if expected_task_id is not None and expected_source_file_id is not None:
|
||||
self._assert_expected_owner(
|
||||
relative_path,
|
||||
expected_task_id=expected_task_id,
|
||||
expected_source_file_id=expected_source_file_id,
|
||||
)
|
||||
path = self._path_for_relative(relative_path)
|
||||
self._assert_controlled_parent(path)
|
||||
try:
|
||||
info = path.lstat()
|
||||
except FileNotFoundError:
|
||||
return False
|
||||
if stat.S_ISLNK(info.st_mode) or not stat.S_ISREG(info.st_mode):
|
||||
raise DataProcessStorageError("refusing to delete a non-regular storage object")
|
||||
path.unlink()
|
||||
self._fsync_directory(path.parent)
|
||||
for directory in (path.parent, path.parent.parent, path.parent.parent.parent):
|
||||
self._remove_empty_directory(directory)
|
||||
return True
|
||||
|
||||
@staticmethod
|
||||
def _assert_expected_owner(
|
||||
relative_path: PurePosixPath,
|
||||
*,
|
||||
expected_task_id: str,
|
||||
expected_source_file_id: str,
|
||||
) -> None:
|
||||
task_id = _safe_component(expected_task_id, "expected task id")
|
||||
source_file_id = _safe_component(
|
||||
expected_source_file_id,
|
||||
"expected source file id",
|
||||
)
|
||||
if relative_path.parts[:2] != (task_id, source_file_id):
|
||||
raise DataProcessStorageError("source storage object owner mismatch")
|
||||
|
||||
def _relative_from_reference(self, reference: str) -> PurePosixPath | None:
|
||||
if reference.startswith("db://"):
|
||||
return None
|
||||
parsed = urlsplit(reference)
|
||||
if parsed.scheme != "local" or parsed.netloc != "data-process":
|
||||
raise DataProcessStorageError("unsupported source storage reference")
|
||||
if parsed.query or parsed.fragment or "\\" in parsed.path:
|
||||
raise DataProcessStorageError("unsafe source storage reference")
|
||||
raw_parts = parsed.path.lstrip("/").split("/")
|
||||
if len(raw_parts) != 4:
|
||||
raise DataProcessStorageError("unsafe source storage reference")
|
||||
if any(re.search(r"%(?![0-9A-Fa-f]{2})", part) for part in raw_parts):
|
||||
raise DataProcessStorageError("unsafe source storage reference")
|
||||
try:
|
||||
decoded = [unquote(part, encoding="utf-8", errors="strict") for part in raw_parts]
|
||||
except UnicodeDecodeError as exc:
|
||||
raise DataProcessStorageError("unsafe source storage reference") from exc
|
||||
if any("/" in part or "\\" in part for part in decoded):
|
||||
raise DataProcessStorageError("unsafe source storage reference")
|
||||
canonical_parts = [
|
||||
quote(decoded[0], safe="-_."),
|
||||
quote(decoded[1], safe="-_."),
|
||||
quote(decoded[2], safe="-_."),
|
||||
quote(decoded[3], safe=""),
|
||||
]
|
||||
if canonical_parts != raw_parts:
|
||||
raise DataProcessStorageError("source storage reference is not canonical")
|
||||
task_id = _safe_component(decoded[0], "task id")
|
||||
source_file_id = _safe_component(decoded[1], "source file id")
|
||||
version_component = decoded[2]
|
||||
if not version_component.startswith("v") or not version_component[1:].isdigit():
|
||||
raise DataProcessStorageError("invalid source file version")
|
||||
version = int(version_component[1:])
|
||||
if version < 1:
|
||||
raise DataProcessStorageError("invalid source file version")
|
||||
basename = _safe_basename(decoded[3])
|
||||
return PurePosixPath(task_id, source_file_id, f"v{version}", basename)
|
||||
|
||||
def _path_for_relative(self, relative_path: PurePosixPath) -> Path:
|
||||
if relative_path.is_absolute() or any(
|
||||
part in {"", ".", ".."} for part in relative_path.parts
|
||||
):
|
||||
raise DataProcessStorageError("storage path escapes the configured root")
|
||||
path = self._root.joinpath(*relative_path.parts)
|
||||
self._assert_controlled_parent(path)
|
||||
return path
|
||||
|
||||
def _validate_staged_object(
|
||||
self,
|
||||
item: StagedSourceObject,
|
||||
*,
|
||||
require_file: bool,
|
||||
) -> None:
|
||||
if not isinstance(item, StagedSourceObject):
|
||||
raise DataProcessStorageError("invalid staged source object")
|
||||
if self._issued_staged_objects.get(item._temporary_path) is not item:
|
||||
raise DataProcessStorageError("staged source object was not issued by this storage")
|
||||
expected_relative = self._relative_from_reference(item.reference)
|
||||
if expected_relative is None or expected_relative != item._relative_path:
|
||||
raise DataProcessStorageError("staged source object reference mismatch")
|
||||
staging_root = self._root / ".staging"
|
||||
try:
|
||||
relative_temporary = item._temporary_path.relative_to(staging_root)
|
||||
except ValueError as exc:
|
||||
raise DataProcessStorageError("staged source object escapes staging") from exc
|
||||
if len(relative_temporary.parts) != 2:
|
||||
raise DataProcessStorageError("invalid staged source object path")
|
||||
_safe_component(relative_temporary.parts[0], "batch id")
|
||||
_safe_basename(relative_temporary.parts[1])
|
||||
self._assert_controlled_parent(item._temporary_path)
|
||||
try:
|
||||
info = item._temporary_path.lstat()
|
||||
except FileNotFoundError:
|
||||
if require_file:
|
||||
raise DataProcessStorageError("staged source object does not exist") from None
|
||||
return
|
||||
if stat.S_ISLNK(info.st_mode) or not stat.S_ISREG(info.st_mode):
|
||||
raise DataProcessStorageError("staged source object is not a regular file")
|
||||
|
||||
def _ensure_directory(self, directory: Path) -> Path:
|
||||
try:
|
||||
relative = directory.relative_to(self._root)
|
||||
except ValueError as exc:
|
||||
raise DataProcessStorageError("storage path escapes the configured root") from exc
|
||||
current = self._root
|
||||
for component in relative.parts:
|
||||
current = current / component
|
||||
try:
|
||||
current.mkdir(mode=0o700)
|
||||
except FileExistsError:
|
||||
pass
|
||||
info = current.lstat()
|
||||
if stat.S_ISLNK(info.st_mode) or not stat.S_ISDIR(info.st_mode):
|
||||
raise DataProcessStorageError("storage path contains a symlink or non-directory")
|
||||
return directory
|
||||
|
||||
def _assert_controlled_parent(self, path: Path) -> None:
|
||||
try:
|
||||
relative_parent = path.parent.relative_to(self._root)
|
||||
except ValueError as exc:
|
||||
raise DataProcessStorageError("storage path escapes the configured root") from exc
|
||||
current = self._root
|
||||
for component in relative_parent.parts:
|
||||
current = current / component
|
||||
if not current.exists():
|
||||
continue
|
||||
info = current.lstat()
|
||||
if stat.S_ISLNK(info.st_mode) or not stat.S_ISDIR(info.st_mode):
|
||||
raise DataProcessStorageError("storage path contains a symlink or non-directory")
|
||||
|
||||
@staticmethod
|
||||
def _fsync_directory(directory: Path) -> None:
|
||||
descriptor = os.open(directory, os.O_RDONLY)
|
||||
try:
|
||||
os.fsync(descriptor)
|
||||
finally:
|
||||
os.close(descriptor)
|
||||
|
||||
def _remove_empty_directory(self, directory: Path) -> None:
|
||||
if directory in {self._root, self._root / ".staging"}:
|
||||
return
|
||||
self._assert_controlled_parent(directory / "placeholder")
|
||||
try:
|
||||
directory.rmdir()
|
||||
except (FileNotFoundError, OSError):
|
||||
return
|
||||
|
||||
|
||||
@lru_cache
|
||||
def get_data_process_storage() -> LocalDataProcessStorage:
|
||||
return LocalDataProcessStorage()
|
||||
|
||||
|
||||
__all__ = [
|
||||
"DataProcessStorageError",
|
||||
"LocalDataProcessStorage",
|
||||
"StagedSourceObject",
|
||||
"get_data_process_storage",
|
||||
]
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,244 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from fastapi import APIRouter, Body, Depends, Request
|
||||
from typing import Any
|
||||
|
||||
from app.api.v1.endpoints.platform import ok, fail
|
||||
from app.core.auth import filter_accessible_resource_ids, get_current_user, has_resource_access, is_admin
|
||||
from app.db.platform_store import get_platform_store
|
||||
|
||||
router = APIRouter(prefix="/projects", tags=["project"])
|
||||
|
||||
|
||||
def _actor(request: Request) -> str | None:
|
||||
auth = request.headers.get("Authorization", "")
|
||||
token = auth.replace("Bearer ", "").strip()
|
||||
return token or None
|
||||
|
||||
|
||||
def _require_no_pending_approval(resource_type: str, resource_id: str) -> None:
|
||||
"""第 4 周：写操作审批拦截——存在待审批实例时拒绝执行。"""
|
||||
store = get_platform_store()
|
||||
pending = [
|
||||
i for i in store.approval_instances(status="pending")
|
||||
if i["resource_type"] == resource_type and i["resource_id"] == resource_id
|
||||
]
|
||||
if pending:
|
||||
raise fail(409, "存在待审批的变更，请先完成审批")
|
||||
|
||||
|
||||
def _require_approval_or_admin(
|
||||
resource_type: str,
|
||||
resource_id: str,
|
||||
current_user: dict[str, Any],
|
||||
action_desc: str = "",
|
||||
) -> dict[str, Any] | None:
|
||||
"""高风险操作审批旁路：admin 直接放行，普通用户创建审批实例（code=202）。"""
|
||||
if is_admin(current_user):
|
||||
return None
|
||||
store = get_platform_store()
|
||||
instance = store.create_approval_instance({
|
||||
"resource_type": resource_type,
|
||||
"resource_id": resource_id,
|
||||
"applicant_id": current_user.get("id"),
|
||||
"template_id": None,
|
||||
})
|
||||
return {
|
||||
"code": 202,
|
||||
"message": f"操作已提交审批，等待管理员批准：{action_desc}",
|
||||
"data": {"approval_required": True, "approval_id": instance["id"]},
|
||||
}
|
||||
|
||||
|
||||
@router.get("")
|
||||
def list_projects(
|
||||
tenant_id: str = "default",
|
||||
status: str | None = None,
|
||||
keyword: str | None = None,
|
||||
current_user: dict = Depends(get_current_user),
|
||||
) -> dict[str, Any]:
|
||||
store = get_platform_store()
|
||||
projects = store.projects(tenant_id=tenant_id, status=status, keyword=keyword)
|
||||
# #1 ACL 过滤：admin 直接放行，普通用户只能看到自己被授权的项目
|
||||
accessible_ids = set(
|
||||
filter_accessible_resource_ids("project", [p["id"] for p in projects], current_user)
|
||||
)
|
||||
filtered = [p for p in projects if p["id"] in accessible_ids]
|
||||
return ok(filtered)
|
||||
|
||||
|
||||
@router.post("")
|
||||
def create_project(payload: dict[str, Any] = Body(...), request: Request = None) -> dict[str, Any]:
|
||||
store = get_platform_store()
|
||||
proj = store.create_project(payload)
|
||||
store.record_audit(
|
||||
action="project.create",
|
||||
actor_id=_actor(request) if request else None,
|
||||
target_type="project",
|
||||
target_id=proj["id"],
|
||||
tenant_id=proj.get("tenant_id"),
|
||||
detail=f"name={proj.get('name')}",
|
||||
)
|
||||
return ok(proj)
|
||||
|
||||
|
||||
@router.get("/{project_id}")
|
||||
def get_project(project_id: str, current_user: dict = Depends(get_current_user)) -> dict[str, Any]:
|
||||
# #2 访问控制：普通用户无 read 权限则拒绝
|
||||
if not has_resource_access("project", project_id, current_user, "read"):
|
||||
raise fail(403, "no permission to access this project")
|
||||
try:
|
||||
return ok(get_platform_store().project(project_id))
|
||||
except KeyError:
|
||||
raise fail(404, "project not found")
|
||||
|
||||
|
||||
@router.put("/{project_id}")
|
||||
def update_project(
|
||||
project_id: str,
|
||||
payload: dict[str, Any] = Body(...),
|
||||
request: Request = None,
|
||||
current_user: dict = Depends(get_current_user),
|
||||
) -> dict[str, Any]:
|
||||
if not has_resource_access("project", project_id, current_user, "write"):
|
||||
raise fail(403, "no permission to update this project")
|
||||
store = get_platform_store()
|
||||
try:
|
||||
proj = store.update_project(project_id, payload)
|
||||
except KeyError:
|
||||
raise fail(404, "project not found")
|
||||
store.record_audit(
|
||||
action="project.update",
|
||||
actor_id=_actor(request) if request else None,
|
||||
target_type="project",
|
||||
target_id=project_id,
|
||||
tenant_id=proj.get("tenant_id"),
|
||||
detail=f"fields={','.join(payload.keys())}",
|
||||
)
|
||||
return ok(proj)
|
||||
|
||||
|
||||
@router.post("/{project_id}/archive")
|
||||
def archive_project(
|
||||
project_id: str,
|
||||
request: Request = None,
|
||||
current_user: dict = Depends(get_current_user),
|
||||
) -> dict[str, Any]:
|
||||
_require_no_pending_approval("project", project_id)
|
||||
pending = _require_approval_or_admin("project", project_id, current_user, f"归档项目 {project_id}")
|
||||
if pending:
|
||||
return pending
|
||||
store = get_platform_store()
|
||||
try:
|
||||
proj = store.archive_project(project_id)
|
||||
except KeyError:
|
||||
raise fail(404, "project not found")
|
||||
store.record_audit(
|
||||
action="project.archive",
|
||||
actor_id=_actor(request) if request else None,
|
||||
target_type="project",
|
||||
target_id=project_id,
|
||||
tenant_id=proj.get("tenant_id"),
|
||||
)
|
||||
return ok(proj)
|
||||
|
||||
|
||||
@router.delete("/{project_id}")
|
||||
def delete_project(
|
||||
project_id: str,
|
||||
request: Request = None,
|
||||
current_user: dict = Depends(get_current_user),
|
||||
) -> dict[str, Any]:
|
||||
_require_no_pending_approval("project", project_id)
|
||||
pending = _require_approval_or_admin("project", project_id, current_user, f"删除项目 {project_id}")
|
||||
if pending:
|
||||
return pending
|
||||
store = get_platform_store()
|
||||
store.delete_project(project_id)
|
||||
store.record_audit(
|
||||
action="project.delete",
|
||||
actor_id=_actor(request) if request else None,
|
||||
target_type="project",
|
||||
target_id=project_id,
|
||||
)
|
||||
return ok(None)
|
||||
|
||||
|
||||
@router.get("/{project_id}/members")
|
||||
def list_members(project_id: str, current_user: dict = Depends(get_current_user)) -> dict[str, Any]:
|
||||
if not has_resource_access("project", project_id, current_user, "read"):
|
||||
raise fail(403, "no permission to access this project")
|
||||
try:
|
||||
return ok(get_platform_store().project_members(project_id))
|
||||
except KeyError:
|
||||
raise fail(404, "project not found")
|
||||
|
||||
|
||||
@router.post("/{project_id}/members")
|
||||
def add_member(
|
||||
project_id: str,
|
||||
payload: dict[str, Any] = Body(...),
|
||||
request: Request = None,
|
||||
current_user: dict = Depends(get_current_user),
|
||||
) -> dict[str, Any]:
|
||||
if not has_resource_access("project", project_id, current_user, "write"):
|
||||
raise fail(403, "no permission to manage members of this project")
|
||||
store = get_platform_store()
|
||||
try:
|
||||
member = store.add_project_member(project_id, payload)
|
||||
except KeyError:
|
||||
raise fail(404, "project not found")
|
||||
store.record_audit(
|
||||
action="project.member.add",
|
||||
actor_id=_actor(request) if request else None,
|
||||
target_type="project.member",
|
||||
target_id=project_id,
|
||||
detail=f"user_id={payload.get('user_id')},role={payload.get('role')}",
|
||||
)
|
||||
return ok(member)
|
||||
|
||||
|
||||
@router.put("/{project_id}/members/{user_id}")
|
||||
def update_member(
|
||||
project_id: str,
|
||||
user_id: str,
|
||||
payload: dict[str, Any] = Body(...),
|
||||
request: Request = None,
|
||||
current_user: dict = Depends(get_current_user),
|
||||
) -> dict[str, Any]:
|
||||
if not has_resource_access("project", project_id, current_user, "write"):
|
||||
raise fail(403, "no permission to manage members of this project")
|
||||
store = get_platform_store()
|
||||
try:
|
||||
member = store.update_project_member_role(project_id, user_id, payload)
|
||||
except KeyError:
|
||||
raise fail(404, "project or member not found")
|
||||
store.record_audit(
|
||||
action="project.member.update",
|
||||
actor_id=_actor(request) if request else None,
|
||||
target_type="project.member",
|
||||
target_id=project_id,
|
||||
detail=f"user_id={user_id},role={payload.get('role')}",
|
||||
)
|
||||
return ok(member)
|
||||
|
||||
|
||||
@router.delete("/{project_id}/members/{user_id}")
|
||||
def remove_member(
|
||||
project_id: str,
|
||||
user_id: str,
|
||||
request: Request = None,
|
||||
current_user: dict = Depends(get_current_user),
|
||||
) -> dict[str, Any]:
|
||||
if not has_resource_access("project", project_id, current_user, "write"):
|
||||
raise fail(403, "no permission to manage members of this project")
|
||||
store = get_platform_store()
|
||||
store.remove_project_member(project_id, user_id)
|
||||
store.record_audit(
|
||||
action="project.member.remove",
|
||||
actor_id=_actor(request) if request else None,
|
||||
target_type="project.member",
|
||||
target_id=project_id,
|
||||
detail=f"user_id={user_id}",
|
||||
)
|
||||
return ok(None)
|
||||
@@ -1 +0,0 @@
|
||||
"""Resource access control list (ACL) module."""
|
||||
@@ -1,41 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from fastapi import APIRouter, Body, Request
|
||||
from typing import Any
|
||||
|
||||
from app.api.v1.endpoints.platform import ok, fail
|
||||
from app.db.platform_store import get_platform_store
|
||||
|
||||
router = APIRouter(prefix="/resources", tags=["resource"])
|
||||
|
||||
|
||||
def _actor(request: Request) -> str | None:
|
||||
auth = request.headers.get("Authorization", "")
|
||||
token = auth.replace("Bearer ", "").strip()
|
||||
return token or None
|
||||
|
||||
|
||||
@router.get("/{resource_type}/{resource_id}/acl")
|
||||
def get_acl(resource_type: str, resource_id: str) -> dict[str, Any]:
|
||||
"""查询资源 ACL，返回按主体分组的权限列表。"""
|
||||
return ok(get_platform_store().resource_acl(resource_type, resource_id))
|
||||
|
||||
|
||||
@router.put("/{resource_type}/{resource_id}/acl")
|
||||
def set_acl(
|
||||
resource_type: str,
|
||||
resource_id: str,
|
||||
payload: dict[str, Any] = Body(...),
|
||||
request: Request = None,
|
||||
) -> dict[str, Any]:
|
||||
"""设置资源 ACL，body: { entries: [{ subject_type, subject_id, permissions: [] }] }"""
|
||||
entries = payload.get("entries") or []
|
||||
result = get_platform_store().set_resource_acl(resource_type, resource_id, entries)
|
||||
get_platform_store().record_audit(
|
||||
action="resource.acl.set",
|
||||
actor_id=_actor(request) if request else None,
|
||||
target_type=resource_type,
|
||||
target_id=resource_id,
|
||||
detail=f"entries={len(entries)}",
|
||||
)
|
||||
return ok(result)
|
||||
@@ -1,75 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from fastapi import APIRouter, Body, Request
|
||||
from typing import Any
|
||||
|
||||
from app.api.v1.endpoints.platform import ok, fail
|
||||
from app.db.platform_store import get_platform_store
|
||||
|
||||
router = APIRouter(prefix="/retention-policies", tags=["retention"])
|
||||
|
||||
|
||||
def _actor(request: Request) -> str | None:
|
||||
auth = request.headers.get("Authorization", "")
|
||||
token = auth.replace("Bearer ", "").strip()
|
||||
return token or None
|
||||
|
||||
|
||||
@router.get("")
|
||||
def list_policies() -> dict[str, Any]:
|
||||
return ok(get_platform_store().retention_policies())
|
||||
|
||||
|
||||
@router.post("")
|
||||
def create_policy(payload: dict[str, Any] = Body(...), request: Request = None) -> dict[str, Any]:
|
||||
if not payload.get("name"):
|
||||
raise fail(400, "name 必填")
|
||||
policy = get_platform_store().create_retention_policy(payload)
|
||||
get_platform_store().record_audit(
|
||||
action="retention.create",
|
||||
actor_id=_actor(request) if request else None,
|
||||
target_type="retention_policy",
|
||||
target_id=policy["id"],
|
||||
detail=f"name={policy.get('name')}",
|
||||
)
|
||||
return ok(policy)
|
||||
|
||||
|
||||
@router.get("/{policy_id}")
|
||||
def get_policy(policy_id: str) -> dict[str, Any]:
|
||||
try:
|
||||
return ok(get_platform_store().retention_policy(policy_id))
|
||||
except KeyError:
|
||||
raise fail(404, "retention policy not found")
|
||||
|
||||
|
||||
@router.put("/{policy_id}")
|
||||
def update_policy(
|
||||
policy_id: str, payload: dict[str, Any] = Body(...), request: Request = None
|
||||
) -> dict[str, Any]:
|
||||
store = get_platform_store()
|
||||
try:
|
||||
policy = store.update_retention_policy(policy_id, payload)
|
||||
except KeyError:
|
||||
raise fail(404, "retention policy not found")
|
||||
store.record_audit(
|
||||
action="retention.update",
|
||||
actor_id=_actor(request) if request else None,
|
||||
target_type="retention_policy",
|
||||
target_id=policy_id,
|
||||
detail=f"fields={','.join(payload.keys())}",
|
||||
)
|
||||
return ok(policy)
|
||||
|
||||
|
||||
@router.delete("/{policy_id}")
|
||||
def delete_policy(policy_id: str, request: Request = None) -> dict[str, Any]:
|
||||
store = get_platform_store()
|
||||
store.delete_retention_policy(policy_id)
|
||||
store.record_audit(
|
||||
action="retention.delete",
|
||||
actor_id=_actor(request) if request else None,
|
||||
target_type="retention_policy",
|
||||
target_id=policy_id,
|
||||
)
|
||||
return ok({"deleted": policy_id})
|
||||
@@ -1,115 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from fastapi import APIRouter, Body, Query, Request
|
||||
from fastapi.responses import StreamingResponse
|
||||
|
||||
from app.db.platform_store import ALL_PERMISSIONS, get_platform_store
|
||||
|
||||
|
||||
router = APIRouter(prefix="/system", tags=["system"])
|
||||
|
||||
|
||||
@router.post("/audit/visit")
|
||||
def record_visit(payload: dict = Body(...), request: Request = None) -> dict:
|
||||
"""记录用户访问业务模块的行为，用于看板用户操作分布统计。"""
|
||||
action = str(payload.get("action") or payload.get("module") or "").strip()
|
||||
if not action:
|
||||
return {"code": 0, "message": "ok", "data": {"recorded": False}}
|
||||
actor_id = ""
|
||||
if request is not None:
|
||||
auth = request.headers.get("Authorization", "")
|
||||
token = auth.replace("Bearer ", "").strip()
|
||||
if token.startswith("platform-token-"):
|
||||
actor_id = token[len("platform-token-"):]
|
||||
get_platform_store().record_audit(
|
||||
action=action,
|
||||
actor_id=actor_id or None,
|
||||
target_type="module",
|
||||
target_id=action,
|
||||
detail=str(payload.get("detail") or ""),
|
||||
)
|
||||
return {"code": 0, "message": "ok", "data": {"recorded": True}}
|
||||
|
||||
|
||||
@router.get("/permissions/codes")
|
||||
def permission_codes() -> dict:
|
||||
"""返回平台权限码清单（权限码接口）。"""
|
||||
return {"code": 0, "message": "ok", "data": {"codes": ALL_PERMISSIONS}}
|
||||
|
||||
|
||||
@router.get("/permissions")
|
||||
def permissions_overview() -> dict:
|
||||
"""返回权限码清单与角色定义。"""
|
||||
store = get_platform_store()
|
||||
return {
|
||||
"code": 0,
|
||||
"message": "ok",
|
||||
"data": {"codes": ALL_PERMISSIONS, "roles": store.roles()},
|
||||
}
|
||||
|
||||
|
||||
@router.get("/audit-logs")
|
||||
def audit_logs(
|
||||
tenant_id: str | None = Query(default=None, description="租户 ID"),
|
||||
project_id: str | None = Query(default=None, description="项目 ID"),
|
||||
actor_id: str | None = Query(default=None, description="操作人 ID"),
|
||||
action: str | None = Query(default=None, description="动作类型"),
|
||||
target_type: str | None = Query(default=None, description="目标类型"),
|
||||
start_time: str | None = Query(default=None, description="ISO8601 起始时间"),
|
||||
end_time: str | None = Query(default=None, description="ISO8601 结束时间"),
|
||||
limit: int = Query(default=50, ge=1, le=200),
|
||||
offset: int = Query(default=0, ge=0),
|
||||
) -> dict:
|
||||
"""审计日志查询：按租户/项目/操作人/动作/目标类型/时间范围分页过滤。"""
|
||||
store = get_platform_store()
|
||||
result = store.audit_logs(
|
||||
tenant_id=tenant_id,
|
||||
project_id=project_id,
|
||||
actor_id=actor_id,
|
||||
action=action,
|
||||
target_type=target_type,
|
||||
start_time=start_time,
|
||||
end_time=end_time,
|
||||
limit=limit,
|
||||
offset=offset,
|
||||
)
|
||||
return {"code": 0, "message": "ok", "data": result}
|
||||
|
||||
|
||||
@router.get("/audit-logs/export")
|
||||
def audit_logs_export(
|
||||
tenant_id: str | None = Query(default=None, description="租户 ID"),
|
||||
project_id: str | None = Query(default=None, description="项目 ID"),
|
||||
actor_id: str | None = Query(default=None, description="操作人 ID"),
|
||||
action: str | None = Query(default=None, description="动作类型"),
|
||||
target_type: str | None = Query(default=None, description="目标类型"),
|
||||
start_time: str | None = Query(default=None, description="ISO8601 起始时间"),
|
||||
end_time: str | None = Query(default=None, description="ISO8601 结束时间"),
|
||||
) -> StreamingResponse:
|
||||
"""审计日志导出：返回 CSV 流，与应用查询相同的过滤条件。"""
|
||||
store = get_platform_store()
|
||||
result = store.audit_logs(
|
||||
tenant_id=tenant_id,
|
||||
project_id=project_id,
|
||||
actor_id=actor_id,
|
||||
action=action,
|
||||
target_type=target_type,
|
||||
start_time=start_time,
|
||||
end_time=end_time,
|
||||
limit=10000,
|
||||
offset=0,
|
||||
)
|
||||
items = result["items"]
|
||||
columns = ["time", "tenant_id", "project_id", "actor_id", "action", "target_type", "target_id", "detail", "client_ip"]
|
||||
header = ",".join(columns) + "\n"
|
||||
|
||||
def iter_rows():
|
||||
yield header
|
||||
for row in items:
|
||||
yield ",".join(f'"{str(row.get(c, "") or "")}"' for c in columns) + "\n"
|
||||
|
||||
return StreamingResponse(
|
||||
iter_rows(),
|
||||
media_type="text/csv",
|
||||
headers={"Content-Disposition": "attachment; filename=audit_logs.csv"},
|
||||
)
|
||||
@@ -1,116 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from fastapi import APIRouter, Body, Request
|
||||
from typing import Any
|
||||
|
||||
from app.api.v1.endpoints.platform import ok, fail
|
||||
from app.db.platform_store import get_platform_store
|
||||
|
||||
router = APIRouter(prefix="/tenants", tags=["tenant"])
|
||||
|
||||
|
||||
def _actor(request: Request) -> str | None:
|
||||
auth = request.headers.get("Authorization", "")
|
||||
token = auth.replace("Bearer ", "").strip()
|
||||
return token or None
|
||||
|
||||
|
||||
@router.get("")
|
||||
def list_tenants() -> dict[str, Any]:
|
||||
return ok(get_platform_store().tenants())
|
||||
|
||||
|
||||
@router.post("")
|
||||
def create_tenant(payload: dict[str, Any] = Body(...), request: Request = None) -> dict[str, Any]:
|
||||
store = get_platform_store()
|
||||
try:
|
||||
tenant = store.create_tenant(payload)
|
||||
except KeyError as e:
|
||||
raise fail(400, f"missing field: {e}")
|
||||
store.record_audit(
|
||||
action="tenant.create",
|
||||
actor_id=_actor(request) if request else None,
|
||||
target_type="tenant",
|
||||
target_id=tenant["id"],
|
||||
tenant_id=tenant["id"],
|
||||
detail=f"name={tenant.get('name')}",
|
||||
)
|
||||
return ok(tenant)
|
||||
|
||||
|
||||
@router.get("/{tenant_id}")
|
||||
def get_tenant(tenant_id: str) -> dict[str, Any]:
|
||||
try:
|
||||
return ok(get_platform_store().tenant(tenant_id))
|
||||
except KeyError:
|
||||
raise fail(404, "tenant not found")
|
||||
|
||||
|
||||
@router.put("/{tenant_id}")
|
||||
def update_tenant(tenant_id: str, payload: dict[str, Any] = Body(...), request: Request = None) -> dict[str, Any]:
|
||||
store = get_platform_store()
|
||||
try:
|
||||
tenant = store.update_tenant(tenant_id, payload)
|
||||
except KeyError:
|
||||
raise fail(404, "tenant not found")
|
||||
store.record_audit(
|
||||
action="tenant.update",
|
||||
actor_id=_actor(request) if request else None,
|
||||
target_type="tenant",
|
||||
target_id=tenant_id,
|
||||
tenant_id=tenant_id,
|
||||
detail=f"fields={','.join(payload.keys())}",
|
||||
)
|
||||
return ok(tenant)
|
||||
|
||||
|
||||
@router.put("/{tenant_id}/quota")
|
||||
def set_quota(tenant_id: str, payload: dict[str, Any] = Body(...), request: Request = None) -> dict[str, Any]:
|
||||
store = get_platform_store()
|
||||
try:
|
||||
tenant = store.set_tenant_quota(tenant_id, payload.get("quota", {}))
|
||||
except KeyError:
|
||||
raise fail(404, "tenant not found")
|
||||
store.record_audit(
|
||||
action="tenant.quota.set",
|
||||
actor_id=_actor(request) if request else None,
|
||||
target_type="tenant",
|
||||
target_id=tenant_id,
|
||||
tenant_id=tenant_id,
|
||||
)
|
||||
return ok(tenant)
|
||||
|
||||
|
||||
@router.put("/{tenant_id}/retention-policy")
|
||||
def set_retention(tenant_id: str, payload: dict[str, Any] = Body(...), request: Request = None) -> dict[str, Any]:
|
||||
store = get_platform_store()
|
||||
try:
|
||||
tenant = store.set_tenant_retention(tenant_id, payload.get("retention_policy_id"))
|
||||
except KeyError:
|
||||
raise fail(404, "tenant not found")
|
||||
store.record_audit(
|
||||
action="tenant.retention.set",
|
||||
actor_id=_actor(request) if request else None,
|
||||
target_type="tenant",
|
||||
target_id=tenant_id,
|
||||
tenant_id=tenant_id,
|
||||
)
|
||||
return ok(tenant)
|
||||
|
||||
|
||||
@router.delete("/{tenant_id}")
|
||||
def delete_tenant(tenant_id: str, request: Request = None) -> dict[str, Any]:
|
||||
store = get_platform_store()
|
||||
try:
|
||||
tenant = store.delete_tenant(tenant_id)
|
||||
except KeyError:
|
||||
raise fail(404, "tenant not found")
|
||||
store.record_audit(
|
||||
action="tenant.delete",
|
||||
actor_id=_actor(request) if request else None,
|
||||
target_type="tenant",
|
||||
target_id=tenant_id,
|
||||
tenant_id=tenant_id,
|
||||
detail=f"name={tenant.get('name')}",
|
||||
)
|
||||
return ok(tenant)
|
||||
@@ -1,386 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from enum import StrEnum
|
||||
from typing import Any, Literal
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
|
||||
|
||||
from app.modules.data_process.constants import MAX_QA_PAIRS_PER_ITEM
|
||||
|
||||
|
||||
def _config_value(config: dict[str, Any], snake_name: str, camel_name: str, default: Any) -> Any:
|
||||
if snake_name in config:
|
||||
return config[snake_name]
|
||||
return config.get(camel_name, default)
|
||||
|
||||
|
||||
def _validate_process_config(config: dict[str, Any]) -> None:
|
||||
chunk_method = _config_value(config, "chunk_method", "chunkMethod", "layout_hybrid")
|
||||
if not isinstance(chunk_method, str) or chunk_method not in {
|
||||
"layout_hybrid",
|
||||
"semantic",
|
||||
"fixed",
|
||||
}:
|
||||
raise ValueError("chunk_method must be one of: layout_hybrid, semantic, fixed")
|
||||
|
||||
semantic_percentile = _config_value(
|
||||
config,
|
||||
"semantic_breakpoint_percentile",
|
||||
"semanticBreakpointPercentile",
|
||||
95,
|
||||
)
|
||||
if (
|
||||
isinstance(semantic_percentile, bool)
|
||||
or not isinstance(semantic_percentile, int)
|
||||
or not 1 <= semantic_percentile <= 99
|
||||
):
|
||||
raise ValueError("semantic_breakpoint_percentile must be an integer in [1, 99]")
|
||||
|
||||
split = _config_value(config, "dataset_split", "datasetSplit", None)
|
||||
if split is not None:
|
||||
if not isinstance(split, dict) or set(split) != {"train", "validation", "test"}:
|
||||
raise ValueError("dataset_split must contain train, validation and test")
|
||||
values = list(split.values())
|
||||
if any(isinstance(value, bool) or not isinstance(value, int) for value in values):
|
||||
raise ValueError("dataset_split values must be integers")
|
||||
if any(value < 0 or value > 100 for value in values) or sum(values) != 100:
|
||||
raise ValueError("dataset_split values must be in [0, 100] and total 100")
|
||||
|
||||
chunk_fields = {
|
||||
"chunk_size",
|
||||
"chunkSize",
|
||||
"chunk_overlap",
|
||||
"chunkOverlap",
|
||||
"min_chunk_size",
|
||||
"minChunkSize",
|
||||
}
|
||||
if chunk_fields.intersection(config):
|
||||
chunk_size = _config_value(config, "chunk_size", "chunkSize", 800)
|
||||
overlap = _config_value(config, "chunk_overlap", "chunkOverlap", 100)
|
||||
minimum = _config_value(config, "min_chunk_size", "minChunkSize", 100)
|
||||
if any(
|
||||
isinstance(value, bool) or not isinstance(value, int)
|
||||
for value in (chunk_size, overlap, minimum)
|
||||
):
|
||||
raise ValueError("chunk_size, chunk_overlap and min_chunk_size must be integers")
|
||||
if not 16 <= chunk_size <= 32_768:
|
||||
raise ValueError("chunk_size must be in [16, 32768]")
|
||||
if overlap < 0 or overlap >= chunk_size:
|
||||
raise ValueError("chunk_overlap must be in [0, chunk_size)")
|
||||
if minimum <= 0 or minimum > chunk_size or overlap + minimum > chunk_size:
|
||||
raise ValueError("min_chunk_size and chunk_overlap exceed chunk_size")
|
||||
|
||||
temperature = _config_value(config, "temperature", "temperature", None)
|
||||
if temperature is not None:
|
||||
if isinstance(temperature, bool) or not isinstance(temperature, (int, float)):
|
||||
raise ValueError("temperature must be a number")
|
||||
if not 0 <= float(temperature) <= 2:
|
||||
raise ValueError("temperature must be in [0, 2]")
|
||||
|
||||
max_tokens = _config_value(config, "max_tokens", "maxTokens", None)
|
||||
if max_tokens is not None:
|
||||
if isinstance(max_tokens, bool) or not isinstance(max_tokens, int):
|
||||
raise ValueError("max_tokens must be an integer")
|
||||
if not 1 <= max_tokens <= 32_768:
|
||||
raise ValueError("max_tokens must be in [1, 32768]")
|
||||
|
||||
for snake_name, camel_name in (
|
||||
("qa_pairs_per_row", "qaPairsPerRow"),
|
||||
("qa_pairs_per_chunk", "qaPairsPerChunk"),
|
||||
):
|
||||
pairs = _config_value(config, snake_name, camel_name, None)
|
||||
if pairs is None:
|
||||
continue
|
||||
if (
|
||||
isinstance(pairs, bool)
|
||||
or not isinstance(pairs, int)
|
||||
or not 1 <= pairs <= MAX_QA_PAIRS_PER_ITEM
|
||||
):
|
||||
raise ValueError(
|
||||
f"{snake_name} must be an integer in [1, {MAX_QA_PAIRS_PER_ITEM}]"
|
||||
)
|
||||
|
||||
|
||||
class DataProcessStatus(StrEnum):
|
||||
pending = "pending"
|
||||
running = "running"
|
||||
completed = "completed"
|
||||
failed = "failed"
|
||||
stopped = "stopped"
|
||||
|
||||
|
||||
class DataProcessWorkflowStep(StrEnum):
|
||||
create = "create"
|
||||
model = "model"
|
||||
upload = "upload"
|
||||
preview = "preview"
|
||||
generate = "generate"
|
||||
results = "results"
|
||||
|
||||
|
||||
class DataProcessPreviewStatus(StrEnum):
|
||||
idle = "idle"
|
||||
queued = "queued"
|
||||
running = "running"
|
||||
completed = "completed"
|
||||
failed = "failed"
|
||||
cancelled = "cancelled"
|
||||
|
||||
|
||||
class ProcessType(StrEnum):
|
||||
structured = "structured"
|
||||
unstructured = "unstructured"
|
||||
external = "external"
|
||||
|
||||
|
||||
class DataProcessTaskCreate(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
name: str = Field(min_length=1, max_length=150)
|
||||
description: str = ""
|
||||
process_type: ProcessType
|
||||
source_dataset_id: str | None = None
|
||||
config: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
@field_validator("name")
|
||||
@classmethod
|
||||
def normalize_name(cls, value: str) -> str:
|
||||
value = value.strip()
|
||||
if not value:
|
||||
raise ValueError("task name cannot be empty")
|
||||
return value
|
||||
|
||||
@model_validator(mode="after")
|
||||
def validate_config(self) -> "DataProcessTaskCreate":
|
||||
_validate_process_config(self.config)
|
||||
return self
|
||||
|
||||
|
||||
class DataProcessTaskUpdate(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
name: str | None = Field(default=None, min_length=1, max_length=150)
|
||||
description: str | None = None
|
||||
process_type: ProcessType | None = None
|
||||
source_dataset_id: str | None = None
|
||||
config: dict[str, Any] | None = None
|
||||
|
||||
@field_validator("name")
|
||||
@classmethod
|
||||
def normalize_name(cls, value: str | None) -> str | None:
|
||||
if value is None:
|
||||
return None
|
||||
value = value.strip()
|
||||
if not value:
|
||||
raise ValueError("task name cannot be empty")
|
||||
return value
|
||||
|
||||
@model_validator(mode="after")
|
||||
def validate_config(self) -> "DataProcessTaskUpdate":
|
||||
if self.config is not None:
|
||||
_validate_process_config(self.config)
|
||||
return self
|
||||
|
||||
|
||||
class DataProcessWorkflowStepUpdate(BaseModel):
|
||||
"""仅保存创建向导位置，不修改配置或使下游产物失效。"""
|
||||
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
workflow_step: DataProcessWorkflowStep
|
||||
|
||||
|
||||
class DataProcessRegenerateRequest(BaseModel):
|
||||
"""以一份完整配置准备任务重新生成。
|
||||
|
||||
``expected_updated_at`` 用于防止详情页的旧快照覆盖其他人刚刚
|
||||
保存的配置。重新生成不允许改变处理类型，避免旧源文件在新解析
|
||||
规则下被静默误用。
|
||||
"""
|
||||
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
name: str = Field(min_length=1, max_length=150)
|
||||
description: str
|
||||
process_type: ProcessType
|
||||
config: dict[str, Any]
|
||||
expected_updated_at: str = Field(min_length=1)
|
||||
|
||||
@field_validator("name")
|
||||
@classmethod
|
||||
def normalize_name(cls, value: str) -> str:
|
||||
value = value.strip()
|
||||
if not value:
|
||||
raise ValueError("task name cannot be empty")
|
||||
return value
|
||||
|
||||
@model_validator(mode="after")
|
||||
def validate_config(self) -> "DataProcessRegenerateRequest":
|
||||
_validate_process_config(self.config)
|
||||
return self
|
||||
|
||||
|
||||
class DataProcessRepeatRequest(BaseModel):
|
||||
"""按已确认任务的完整快照创建一批独立的新生成结果。"""
|
||||
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
expected_updated_at: str = Field(min_length=1)
|
||||
request_id: str = Field(
|
||||
min_length=8,
|
||||
max_length=80,
|
||||
pattern=r"^[A-Za-z0-9_-]+$",
|
||||
)
|
||||
|
||||
|
||||
class PreviewBuildRequest(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
replace_existing: Literal[True] = True
|
||||
source_file_ids: list[str] | None = None
|
||||
source_file_id: str | None = None
|
||||
|
||||
@model_validator(mode="after")
|
||||
def validate_source_file_selection(self) -> "PreviewBuildRequest":
|
||||
if self.source_file_ids is not None and self.source_file_id is not None:
|
||||
raise ValueError("source_file_id and source_file_ids cannot be used together")
|
||||
values = self.source_file_ids
|
||||
if values is None and self.source_file_id is not None:
|
||||
values = [self.source_file_id]
|
||||
if values is None:
|
||||
return self
|
||||
normalized = list(dict.fromkeys(str(value).strip() for value in values))
|
||||
if not normalized or any(not value for value in normalized):
|
||||
raise ValueError("at least one non-empty source file id is required")
|
||||
self.source_file_ids = normalized
|
||||
self.source_file_id = None
|
||||
return self
|
||||
|
||||
|
||||
class PreviewItemCreate(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
source_file_id: str | None = None
|
||||
original_content: str = ""
|
||||
edited_content: str = ""
|
||||
source_start: int | None = Field(default=None, ge=0)
|
||||
source_end: int | None = Field(default=None, ge=0)
|
||||
source_start_line: int | None = Field(default=None, ge=1)
|
||||
source_end_line: int | None = Field(default=None, ge=1)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def validate_ranges(self) -> "PreviewItemCreate":
|
||||
if self.source_start is not None and self.source_end is not None:
|
||||
if self.source_end < self.source_start:
|
||||
raise ValueError("source_end must be greater than or equal to source_start")
|
||||
if self.source_start_line is not None and self.source_end_line is not None:
|
||||
if self.source_end_line < self.source_start_line:
|
||||
raise ValueError(
|
||||
"source_end_line must be greater than or equal to source_start_line"
|
||||
)
|
||||
return self
|
||||
|
||||
|
||||
class PreviewItemUpdate(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
edited_content: str
|
||||
expected_updated_at: str | None = None
|
||||
|
||||
|
||||
class GenerateRequest(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
replace_existing: Literal[True] = True
|
||||
|
||||
|
||||
class ExternalSourceRequest(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
type: str = Field(min_length=1, max_length=30)
|
||||
url: str = Field(min_length=1, max_length=2048)
|
||||
auth_mode: Literal["none", "basic"] = "none"
|
||||
username: str | None = Field(default=None, max_length=150)
|
||||
password: str | None = Field(default=None, max_length=500)
|
||||
limit: int = Field(default=1000, ge=1, le=100_000)
|
||||
|
||||
|
||||
class ExternalPullRequest(ExternalSourceRequest):
|
||||
query: str | None = Field(default=None, max_length=20_000)
|
||||
file_name: str = Field(default="external-data.jsonl", min_length=1, max_length=255)
|
||||
|
||||
@field_validator("file_name")
|
||||
@classmethod
|
||||
def validate_file_name(cls, value: str) -> str:
|
||||
name = value.strip()
|
||||
if not name.lower().endswith((".jsonl", ".ndjson")):
|
||||
raise ValueError("external pull file_name must end with .jsonl or .ndjson")
|
||||
return name
|
||||
|
||||
|
||||
class ResultUpdate(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
instruction: str | None = None
|
||||
input: str | None = None
|
||||
output: str | None = None
|
||||
expected_updated_at: str | None = None
|
||||
|
||||
|
||||
class ResultRegenerateRequest(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
expected_updated_at: str = Field(min_length=1, max_length=100)
|
||||
|
||||
|
||||
class ResultBatchRegenerateItem(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
result_id: str = Field(min_length=1, max_length=100)
|
||||
expected_updated_at: str = Field(min_length=1, max_length=100)
|
||||
|
||||
|
||||
class ResultBatchRegenerateRequest(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
items: list[ResultBatchRegenerateItem] = Field(min_length=1, max_length=100)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def validate_unique_results(self) -> "ResultBatchRegenerateRequest":
|
||||
result_ids = [item.result_id for item in self.items]
|
||||
if len(result_ids) != len(set(result_ids)):
|
||||
raise ValueError("result_id values must be unique")
|
||||
return self
|
||||
|
||||
|
||||
class DatasetSplit(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
train: int = Field(default=80, ge=0, le=100)
|
||||
validation: int = Field(default=10, ge=0, le=100)
|
||||
test: int = Field(default=10, ge=0, le=100)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def validate_total(self) -> "DatasetSplit":
|
||||
if self.train + self.validation + self.test != 100:
|
||||
raise ValueError("dataset split must total 100")
|
||||
return self
|
||||
|
||||
|
||||
class PublishRequest(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
dataset_name: str = Field(min_length=1, max_length=150)
|
||||
dataset_type: Literal["train", "test", "eval", "val", "other"] = "train"
|
||||
storage_type: Literal["local"] = "local"
|
||||
split: DatasetSplit = Field(default_factory=DatasetSplit)
|
||||
format: Literal["alpaca_jsonl", "jsonl"] = "alpaca_jsonl"
|
||||
description: str = ""
|
||||
|
||||
@field_validator("dataset_name")
|
||||
@classmethod
|
||||
def normalize_dataset_name(cls, value: str) -> str:
|
||||
value = value.strip()
|
||||
if not value:
|
||||
raise ValueError("dataset name cannot be empty")
|
||||
return value
|
||||
@@ -1,31 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
|
||||
from app.core.config import get_settings
|
||||
from app.core.logging import get_logger
|
||||
from app.modules.compute_gateway.sync import poll_compute_jobs_once
|
||||
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
async def run_compute_poller() -> None:
|
||||
settings = get_settings()
|
||||
if settings.compute_mode == "simulator" or settings.compute_status_sync_mode != "polling":
|
||||
logger.info("compute poller disabled", extra={"compute_mode": settings.compute_mode})
|
||||
return
|
||||
|
||||
interval = max(3, settings.compute_poll_interval_seconds)
|
||||
logger.info("compute poller started", extra={"interval_seconds": interval})
|
||||
while True:
|
||||
try:
|
||||
result = await poll_compute_jobs_once()
|
||||
if result["synced"] or result["failed"]:
|
||||
logger.info("compute jobs polled", extra={"result": result})
|
||||
except asyncio.CancelledError:
|
||||
logger.info("compute poller stopped")
|
||||
raise
|
||||
except Exception as exc: # noqa: BLE001 - keep background polling alive
|
||||
logger.exception("compute poller failed", extra={"error": str(exc)})
|
||||
await asyncio.sleep(interval)
|
||||
@@ -2,29 +2,20 @@
|
||||
name = "yg-ft-backend"
|
||||
version = "0.1.0"
|
||||
description = "Backend service for the model fine-tuning platform"
|
||||
requires-python = ">=3.12"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = [
|
||||
"fastapi>=0.111.0",
|
||||
"uvicorn[standard]>=0.30.0",
|
||||
"python-multipart>=0.0.9",
|
||||
"pydantic>=2.7.0",
|
||||
"sqlalchemy>=2.0.30",
|
||||
"psycopg[binary]>=3.2.1",
|
||||
"psycopg-pool>=3.2.1",
|
||||
"asyncpg>=0.29.0",
|
||||
"alembic>=1.13.1",
|
||||
"redis>=5.0.4",
|
||||
"httpx>=0.27.0",
|
||||
"PyJWT>=2.8.0",
|
||||
"passlib[bcrypt]>=1.7.4",
|
||||
"python-dotenv>=1.0.1",
|
||||
"pypdf[crypto]>=5.0.0",
|
||||
"python-docx>=1.1.2",
|
||||
"openpyxl>=3.1.5",
|
||||
"python-pptx>=1.0.2",
|
||||
"llama-index-core==0.14.23",
|
||||
"llama-index-embeddings-huggingface==0.6.1",
|
||||
"docling==2.115.0",
|
||||
"tiktoken>=0.7.0",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
@@ -35,7 +26,7 @@ dev = [
|
||||
|
||||
[tool.ruff]
|
||||
line-length = 100
|
||||
target-version = "py312"
|
||||
target-version = "py311"
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
|
||||
@@ -3,23 +3,10 @@ uvicorn[standard]>=0.30.0
|
||||
python-multipart>=0.0.9
|
||||
pydantic>=2.7.0
|
||||
sqlalchemy>=2.0.30
|
||||
psycopg[binary]>=3.2.1
|
||||
psycopg-pool>=3.2.1
|
||||
asyncpg>=0.29.0
|
||||
alembic>=1.13.1
|
||||
redis>=5.0.4
|
||||
httpx>=0.27.0
|
||||
PyJWT>=2.8.0
|
||||
passlib[bcrypt]>=1.7.4
|
||||
python-dotenv>=1.0.1
|
||||
pypdf[crypto]>=5.0.0
|
||||
python-docx>=1.1.2
|
||||
openpyxl>=3.1.5
|
||||
python-pptx>=1.0.2
|
||||
llama-index-core==0.14.23
|
||||
llama-index-embeddings-huggingface==0.6.1
|
||||
docling==2.115.0
|
||||
tiktoken>=0.7.0
|
||||
|
||||
# 测试与代码检查
|
||||
pytest>=8.2.0
|
||||
ruff>=0.5.0
|
||||
|
||||
@@ -1,276 +0,0 @@
|
||||
"""
|
||||
模型推理异步加载改造的单元测试。
|
||||
|
||||
覆盖：
|
||||
- model_compare_load：异步派发，立即返回 starting + 节点信息（不等待加载完成）
|
||||
- model_compare_delete：先删记录，卸载失败也不阻塞删除
|
||||
- reconcile_inference_loads：starting -> ready/error/idle/不可达的状态迁移与封顶
|
||||
- _unload_from_compute_node：任务感知，只命中记录中的节点
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from types import SimpleNamespace
|
||||
from typing import Any
|
||||
|
||||
from app.api.v1.endpoints.platform import model_compare_delete, model_compare_load
|
||||
import app.api.v1.endpoints.platform as platform
|
||||
from app.modules.compute_gateway.client import ComputeNodeClient
|
||||
from app.modules.compute_gateway.sync import MAX_STARTING_ATTEMPTS, reconcile_inference_loads
|
||||
|
||||
|
||||
class FakeInferenceStore:
|
||||
"""内存 store，仅实现推理加载/对账用到的接口。"""
|
||||
|
||||
def __init__(self, tasks: list[dict[str, Any]] | None = None, nodes: list[dict[str, Any]] | None = None) -> None:
|
||||
self._tasks: dict[str, dict[str, Any]] = {t["id"]: dict(t) for t in (tasks or [])}
|
||||
self._nodes = nodes or []
|
||||
self._inference_nodes: set[str] = set()
|
||||
|
||||
def compare_task(self, task_id: str) -> dict[str, Any]:
|
||||
if task_id not in self._tasks:
|
||||
raise KeyError(task_id)
|
||||
return dict(self._tasks[task_id])
|
||||
|
||||
def compare_tasks(self) -> list[dict[str, Any]]:
|
||||
return [dict(t) for t in self._tasks.values()]
|
||||
|
||||
def update_compare_task(self, task_id: str, payload: dict[str, Any]) -> dict[str, Any]:
|
||||
current = self._tasks[task_id]
|
||||
merged = {**current, **payload, "id": task_id}
|
||||
self._tasks[task_id] = merged
|
||||
return dict(merged)
|
||||
|
||||
def delete_compare_task(self, task_id: str) -> None:
|
||||
self._tasks.pop(task_id, None)
|
||||
|
||||
def compute_nodes(self) -> list[dict[str, Any]]:
|
||||
return [dict(n) for n in self._nodes]
|
||||
|
||||
def model(self, model_id: str) -> dict[str, Any]:
|
||||
raise KeyError(model_id)
|
||||
|
||||
def trained_models(self) -> list[dict[str, Any]]:
|
||||
return []
|
||||
|
||||
def mark_inference_loaded(self, node_id: str) -> None:
|
||||
self._inference_nodes.add(node_id)
|
||||
|
||||
def mark_inference_unloaded(self, node_id: str) -> None:
|
||||
self._inference_nodes.discard(node_id)
|
||||
|
||||
def is_inference_loaded(self, node_id: str) -> bool:
|
||||
return node_id in self._inference_nodes
|
||||
|
||||
|
||||
def _node(node_id: str, code: str = "") -> dict[str, Any]:
|
||||
return {
|
||||
"id": node_id,
|
||||
"code": code or node_id,
|
||||
"name": code or node_id,
|
||||
"api_base_url": f"http://{code or node_id}:19100",
|
||||
"enabled": True,
|
||||
"scheduler_status": "online",
|
||||
}
|
||||
|
||||
|
||||
def _task(task_id: str, *, node_id: str | None = None, load_status: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||
return {
|
||||
"id": task_id,
|
||||
"name": f"task-{task_id}",
|
||||
"status": "pending",
|
||||
"models": [
|
||||
{"model_id": "m_1", "model_name": "qwen", "model_path": "/models/qwen", "node_id": node_id}
|
||||
],
|
||||
"load_status": load_status or {"loaded_models": []},
|
||||
}
|
||||
|
||||
|
||||
async def _fake_inference_load(self, payload: dict[str, Any]) -> dict[str, Any]:
|
||||
return {"loaded": False, "status": "loading", "request_id": "req-1"}
|
||||
|
||||
|
||||
async def _fake_inference_unload(self) -> dict[str, Any]:
|
||||
return {"unloaded": True, "status": "idle"}
|
||||
|
||||
|
||||
def _patch_store(monkeypatch, store: FakeInferenceStore) -> None:
|
||||
monkeypatch.setattr(platform, "get_platform_store", lambda: store)
|
||||
monkeypatch.setattr(platform, "get_settings", lambda: SimpleNamespace(compute_mode="real"))
|
||||
|
||||
|
||||
def test_select_eval_node_prefers_model_node(monkeypatch) -> None:
|
||||
from app.api.v1.endpoints.platform import _select_eval_node
|
||||
|
||||
store = FakeInferenceStore(nodes=[_node("n1"), _node("n2")])
|
||||
# 指定模型所在节点时优先返回该节点
|
||||
assert _select_eval_node(store, "n2")["id"] == "n2"
|
||||
# 无指定节点时回退到第一个在线节点
|
||||
assert _select_eval_node(store, None)["id"] == "n1"
|
||||
|
||||
|
||||
def test_select_eval_node_returns_none_when_model_node_offline(monkeypatch) -> None:
|
||||
from app.api.v1.endpoints.platform import _select_eval_node
|
||||
|
||||
nodes = [_node("n1"), _node("n2")]
|
||||
nodes[1]["enabled"] = False
|
||||
store = FakeInferenceStore(nodes=nodes)
|
||||
# 模型所在节点不可用 → 明确失败，不派发到其它节点
|
||||
assert _select_eval_node(store, "n2") is None
|
||||
# 无指定节点时仍回退第一个在线节点
|
||||
assert _select_eval_node(store, None)["id"] == "n1"
|
||||
|
||||
|
||||
def test_model_compare_load_dispatches_and_returns_starting(monkeypatch) -> None:
|
||||
store = FakeInferenceStore(tasks=[_task("t1", node_id="n1")], nodes=[_node("n1")])
|
||||
_patch_store(monkeypatch, store)
|
||||
monkeypatch.setattr(ComputeNodeClient, "inference_load", _fake_inference_load)
|
||||
|
||||
result = asyncio.run(model_compare_load("t1"))
|
||||
assert result["code"] == 0
|
||||
updated = result["data"]
|
||||
assert updated["status"] == "starting"
|
||||
items = updated["load_status"]["loaded_models"]
|
||||
assert items[0]["status"] == "starting"
|
||||
assert items[0]["node_id"] == "n1"
|
||||
assert "n1" in store._inference_nodes
|
||||
|
||||
|
||||
def test_model_compare_load_marks_error_when_all_nodes_fail(monkeypatch) -> None:
|
||||
store = FakeInferenceStore(tasks=[_task("t1", node_id="n1")], nodes=[_node("n1")])
|
||||
_patch_store(monkeypatch, store)
|
||||
|
||||
async def _raise(self, payload: dict[str, Any]) -> dict[str, Any]:
|
||||
raise RuntimeError("conn refused")
|
||||
|
||||
monkeypatch.setattr(ComputeNodeClient, "inference_load", _raise)
|
||||
|
||||
result = asyncio.run(model_compare_load("t1"))
|
||||
updated = result["data"]
|
||||
assert updated["status"] == "failed"
|
||||
assert updated["load_status"]["loaded_models"][0]["status"] == "error"
|
||||
assert "conn refused" in updated["load_status"]["loaded_models"][0]["error"]
|
||||
|
||||
|
||||
def test_model_compare_delete_removes_record_even_if_unload_raises(monkeypatch) -> None:
|
||||
task = _task(
|
||||
"t1",
|
||||
node_id="n1",
|
||||
load_status={"loaded_models": [{"model_id": "m_1", "status": "ready", "node_id": "n1"}]},
|
||||
)
|
||||
store = FakeInferenceStore(tasks=[task], nodes=[_node("n1")])
|
||||
_patch_store(monkeypatch, store)
|
||||
|
||||
async def _raise(self) -> dict[str, Any]:
|
||||
raise RuntimeError("unload boom")
|
||||
|
||||
monkeypatch.setattr(ComputeNodeClient, "inference_unload", _raise)
|
||||
|
||||
result = asyncio.run(model_compare_delete("t1"))
|
||||
assert result["data"] == {"deleted": "t1"}
|
||||
assert "t1" not in store._tasks
|
||||
# finally 中仍清掉了节点标记
|
||||
assert "n1" not in store._inference_nodes
|
||||
|
||||
|
||||
def test_unload_from_compute_node_only_hits_recorded_node(monkeypatch) -> None:
|
||||
task = _task(
|
||||
"t1",
|
||||
load_status={"loaded_models": [{"model_id": "m_1", "status": "ready", "node_id": "n1"}]},
|
||||
)
|
||||
store = FakeInferenceStore(tasks=[task], nodes=[_node("n1"), _node("n2")])
|
||||
_patch_store(monkeypatch, store)
|
||||
monkeypatch.setattr(ComputeNodeClient, "inference_unload", _fake_inference_unload)
|
||||
|
||||
from app.api.v1.endpoints.platform import _unload_from_compute_node
|
||||
|
||||
result = asyncio.run(_unload_from_compute_node(store, task=task))
|
||||
assert result["unloaded"] is True
|
||||
# 只命中任务记录中的节点 n1，n2 未被卸载
|
||||
assert [r["node_id"] for r in result["nodes"]] == ["n1"]
|
||||
assert "n1" not in store._inference_nodes
|
||||
|
||||
|
||||
async def _status_ready(self) -> dict[str, Any]:
|
||||
return {"loaded": True, "status": "ready", "model_name": "qwen"}
|
||||
|
||||
|
||||
def test_reconcile_transitions_starting_to_ready(monkeypatch) -> None:
|
||||
task = _task(
|
||||
"t1",
|
||||
node_id="n1",
|
||||
load_status={"loaded_models": [{"model_id": "m_1", "status": "starting", "node_id": "n1"}]},
|
||||
)
|
||||
store = FakeInferenceStore(tasks=[task], nodes=[_node("n1")])
|
||||
monkeypatch.setattr(ComputeNodeClient, "inference_status", _status_ready)
|
||||
|
||||
reconciled = asyncio.run(reconcile_inference_loads(store))
|
||||
assert reconciled == [{"task_id": "t1", "status": "loaded"}]
|
||||
updated = store._tasks["t1"]
|
||||
assert updated["status"] == "loaded"
|
||||
assert updated["load_status"]["loaded_models"][0]["status"] == "ready"
|
||||
assert "n1" in store._inference_nodes
|
||||
|
||||
|
||||
def test_reconcile_transitions_to_error_and_failed(monkeypatch) -> None:
|
||||
async def _status_error(self) -> dict[str, Any]:
|
||||
return {"loaded": False, "status": "error", "error": "CUDA out of memory"}
|
||||
|
||||
task = _task(
|
||||
"t1",
|
||||
node_id="n1",
|
||||
load_status={"loaded_models": [{"model_id": "m_1", "status": "starting", "node_id": "n1"}]},
|
||||
)
|
||||
store = FakeInferenceStore(tasks=[task], nodes=[_node("n1")])
|
||||
monkeypatch.setattr(ComputeNodeClient, "inference_status", _status_error)
|
||||
|
||||
reconciled = asyncio.run(reconcile_inference_loads(store))
|
||||
assert reconciled == [{"task_id": "t1", "status": "failed"}]
|
||||
item = store._tasks["t1"]["load_status"]["loaded_models"][0]
|
||||
assert item["status"] == "error"
|
||||
assert "CUDA out of memory" in item["error"]
|
||||
assert "n1" not in store._inference_nodes
|
||||
|
||||
|
||||
def test_reconcile_idle_marks_model_disappeared(monkeypatch) -> None:
|
||||
async def _status_idle(self) -> dict[str, Any]:
|
||||
return {"loaded": False, "status": "idle"}
|
||||
|
||||
task = _task(
|
||||
"t1",
|
||||
node_id="n1",
|
||||
load_status={"loaded_models": [{"model_id": "m_1", "status": "starting", "node_id": "n1"}]},
|
||||
)
|
||||
store = FakeInferenceStore(tasks=[task], nodes=[_node("n1")])
|
||||
monkeypatch.setattr(ComputeNodeClient, "inference_status", _status_idle)
|
||||
|
||||
asyncio.run(reconcile_inference_loads(store))
|
||||
item = store._tasks["t1"]["load_status"]["loaded_models"][0]
|
||||
assert item["status"] == "error"
|
||||
assert "disappeared" in item["error"]
|
||||
assert store._tasks["t1"]["status"] == "failed"
|
||||
|
||||
|
||||
def test_reconcile_unreachable_node_flips_to_error_after_cap(monkeypatch) -> None:
|
||||
async def _raise(self) -> dict[str, Any]:
|
||||
raise RuntimeError("conn refused")
|
||||
|
||||
task = _task(
|
||||
"t1",
|
||||
node_id="n1",
|
||||
load_status={"loaded_models": [{"model_id": "m_1", "status": "starting", "node_id": "n1"}]},
|
||||
)
|
||||
store = FakeInferenceStore(tasks=[task], nodes=[_node("n1")])
|
||||
monkeypatch.setattr(ComputeNodeClient, "inference_status", _raise)
|
||||
|
||||
# 每次轮询前重置节流时间戳，逐次推进 load_attempts 到封顶
|
||||
for _ in range(MAX_STARTING_ATTEMPTS):
|
||||
item = store._tasks["t1"]["load_status"]["loaded_models"][0]
|
||||
item["last_polled_at"] = 0
|
||||
asyncio.run(reconcile_inference_loads(store))
|
||||
|
||||
item = store._tasks["t1"]["load_status"]["loaded_models"][0]
|
||||
assert item["status"] == "error"
|
||||
assert "unreachable" in item["error"]
|
||||
assert store._tasks["t1"]["status"] == "failed"
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -1,897 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from app.modules.data_process.generation import (
|
||||
ModelGenerationError,
|
||||
chat_completions_url,
|
||||
generate_model_records,
|
||||
)
|
||||
|
||||
|
||||
def test_chat_completions_url_accepts_host_base_and_complete_url() -> None:
|
||||
assert chat_completions_url("www.caoxiaozhu.com") == (
|
||||
"https://www.caoxiaozhu.com/v1/chat/completions"
|
||||
)
|
||||
assert chat_completions_url("https://model.example/v1") == (
|
||||
"https://model.example/v1/chat/completions"
|
||||
)
|
||||
complete = "https://model.example/openai/v1/chat/completions"
|
||||
assert chat_completions_url(complete) == complete
|
||||
|
||||
|
||||
def test_generate_model_records_uses_prompt_auth_and_stable_split() -> None:
|
||||
requests: list[httpx.Request] = []
|
||||
progress_updates: list[tuple[int, int]] = []
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
requests.append(request)
|
||||
payload = json.loads(request.content)
|
||||
assert payload["model"] == "qwen-plus"
|
||||
assert payload["response_format"] == {"type": "json_object"}
|
||||
assert "客户反馈页面加载慢" in payload["messages"][1]["content"]
|
||||
assert "你正在生成标准监督微调问答数据" in payload["messages"][0]["content"]
|
||||
assert "禁止输出分析、推理过程" in payload["messages"][0]["content"]
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"choices": [
|
||||
{
|
||||
"message": {
|
||||
"content": json.dumps(
|
||||
{
|
||||
"items": [
|
||||
{
|
||||
"instruction": "请生成简洁客服回复",
|
||||
"input": "客户反馈页面加载慢",
|
||||
"output": "已收到反馈，我们正在排查。",
|
||||
}
|
||||
]
|
||||
},
|
||||
ensure_ascii=False,
|
||||
)
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
)
|
||||
|
||||
client = httpx.Client(transport=httpx.MockTransport(handler))
|
||||
records = generate_model_records(
|
||||
[{"id": "preview-1", "edited_content": "客户反馈页面加载慢"}],
|
||||
model={
|
||||
"name": "Qwen",
|
||||
"online_model_name": "qwen-plus",
|
||||
"api_url": "model.example",
|
||||
"api_key": "test-secret",
|
||||
},
|
||||
config={
|
||||
"generation_prompt": "请处理：{{ content }}",
|
||||
"json_mode": True,
|
||||
"temperature": 0.2,
|
||||
"max_tokens": 512,
|
||||
},
|
||||
task_id="task-1",
|
||||
split={"train": 100, "validation": 0, "test": 0},
|
||||
qa_pairs_per_item=1,
|
||||
client=client,
|
||||
on_progress=lambda processed, total: progress_updates.append((processed, total)),
|
||||
)
|
||||
|
||||
assert len(records) == 1
|
||||
assert records[0]["status"] == "valid"
|
||||
assert records[0]["split"] == "train"
|
||||
assert requests[0].headers["Authorization"] == "Bearer test-secret"
|
||||
assert progress_updates == [(1, 1)]
|
||||
|
||||
|
||||
def test_minimax_m3_uses_split_reasoning_and_completion_token_budget() -> None:
|
||||
requests: list[dict[str, object]] = []
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
payload = json.loads(request.content)
|
||||
requests.append(payload)
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"choices": [{
|
||||
"finish_reason": "stop",
|
||||
"message": {
|
||||
"reasoning_content": "模型内部思考不应混入业务 JSON",
|
||||
"content": json.dumps({
|
||||
"items": [{
|
||||
"instruction": "申请编号有什么作用？",
|
||||
"reasoning": "来源说明它用于标识报销申请。",
|
||||
"answer": "它用于唯一标识一笔报销申请。",
|
||||
}],
|
||||
}, ensure_ascii=False),
|
||||
},
|
||||
}],
|
||||
"output_sensitive": False,
|
||||
"base_resp": {"status_code": 0, "status_msg": ""},
|
||||
},
|
||||
)
|
||||
|
||||
records = generate_model_records(
|
||||
[{"id": "preview-minimax", "edited_content": "申请编号用于标识报销申请。"}],
|
||||
model={
|
||||
"name": "MiniMax",
|
||||
"online_model_name": "MiniMax-M3",
|
||||
"api_url": "https://api.minimaxi.com/v1",
|
||||
},
|
||||
config={
|
||||
"output_type": "reasoning",
|
||||
"json_mode": True,
|
||||
"max_tokens": 1024,
|
||||
"generation_retries": 0,
|
||||
},
|
||||
task_id="task-minimax",
|
||||
split={"train": 100, "validation": 0, "test": 0},
|
||||
qa_pairs_per_item=1,
|
||||
client=httpx.Client(transport=httpx.MockTransport(handler)),
|
||||
)
|
||||
|
||||
assert records[0]["status"] == "valid"
|
||||
assert len(requests) == 1
|
||||
assert requests[0]["reasoning_split"] is True
|
||||
assert requests[0]["max_completion_tokens"] >= 4096
|
||||
assert "max_tokens" not in requests[0]
|
||||
assert "response_format" not in requests[0]
|
||||
|
||||
|
||||
def test_minimax_m3_keeps_larger_configured_completion_budget() -> None:
|
||||
requests: list[dict[str, object]] = []
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
requests.append(json.loads(request.content))
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"choices": [{
|
||||
"message": {
|
||||
"content": json.dumps({
|
||||
"items": [{
|
||||
"instruction": "问题",
|
||||
"output": "这是满足测试要求的完整答案。",
|
||||
}],
|
||||
}, ensure_ascii=False),
|
||||
},
|
||||
}],
|
||||
},
|
||||
)
|
||||
|
||||
generate_model_records(
|
||||
[{"id": "preview-minimax-budget", "edited_content": "来源正文"}],
|
||||
model={
|
||||
"online_model_name": "MiniMax-M3",
|
||||
"api_url": "https://api.minimax.io/v1",
|
||||
},
|
||||
config={"max_tokens": 8192, "generation_retries": 0},
|
||||
task_id="task-minimax-budget",
|
||||
split={"train": 100, "validation": 0, "test": 0},
|
||||
qa_pairs_per_item=1,
|
||||
client=httpx.Client(transport=httpx.MockTransport(handler)),
|
||||
)
|
||||
|
||||
assert requests[0]["max_completion_tokens"] == 8192
|
||||
|
||||
|
||||
def test_minimax_m3_name_on_custom_proxy_keeps_generic_openai_parameters() -> None:
|
||||
requests: list[dict[str, object]] = []
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
requests.append(json.loads(request.content))
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"choices": [{
|
||||
"message": {
|
||||
"content": json.dumps({
|
||||
"items": [{
|
||||
"instruction": "问题",
|
||||
"output": "这是代理服务返回的完整答案。",
|
||||
}],
|
||||
}, ensure_ascii=False),
|
||||
},
|
||||
}],
|
||||
},
|
||||
)
|
||||
|
||||
generate_model_records(
|
||||
[{"id": "preview-minimax-proxy", "edited_content": "来源正文"}],
|
||||
model={
|
||||
"online_model_name": "MiniMax-M3",
|
||||
"api_url": "https://model-proxy.example/v1",
|
||||
},
|
||||
config={"max_tokens": 1024, "json_mode": True, "generation_retries": 0},
|
||||
task_id="task-minimax-proxy",
|
||||
split={"train": 100, "validation": 0, "test": 0},
|
||||
qa_pairs_per_item=1,
|
||||
client=httpx.Client(transport=httpx.MockTransport(handler)),
|
||||
)
|
||||
|
||||
assert requests[0]["max_tokens"] == 1024
|
||||
assert requests[0]["response_format"] == {"type": "json_object"}
|
||||
assert "reasoning_split" not in requests[0]
|
||||
assert "max_completion_tokens" not in requests[0]
|
||||
|
||||
|
||||
def test_generate_model_records_extracts_json_surrounded_by_model_explanation() -> None:
|
||||
content = "模型结果如下：\n```json\n" + json.dumps(
|
||||
{
|
||||
"items": [{
|
||||
"instruction": "字段有什么作用？",
|
||||
"output": "该字段用于唯一标识记录。",
|
||||
}],
|
||||
},
|
||||
ensure_ascii=False,
|
||||
) + "\n```\n生成完毕。"
|
||||
client = httpx.Client(
|
||||
transport=httpx.MockTransport(
|
||||
lambda _: httpx.Response(
|
||||
200,
|
||||
json={"choices": [{"message": {"content": content}}]},
|
||||
)
|
||||
)
|
||||
)
|
||||
|
||||
records = generate_model_records(
|
||||
[{"id": "preview-explanation", "edited_content": "字段用于唯一标识记录。"}],
|
||||
model={"name": "model", "api_url": "https://model.example/v1"},
|
||||
config={"generation_retries": 0},
|
||||
task_id="task-explanation",
|
||||
split={"train": 100, "validation": 0, "test": 0},
|
||||
qa_pairs_per_item=1,
|
||||
client=client,
|
||||
)
|
||||
|
||||
assert records[0]["status"] == "valid"
|
||||
assert records[0]["output"] == "该字段用于唯一标识记录。"
|
||||
|
||||
|
||||
def test_generate_model_records_reports_token_truncation_instead_of_json_error() -> None:
|
||||
client = httpx.Client(
|
||||
transport=httpx.MockTransport(
|
||||
lambda _: httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"choices": [{
|
||||
"finish_reason": "length",
|
||||
"message": {"content": ""},
|
||||
}],
|
||||
"output_sensitive": False,
|
||||
},
|
||||
)
|
||||
)
|
||||
)
|
||||
|
||||
records = generate_model_records(
|
||||
[{"id": "preview-truncated", "edited_content": "来源正文"}],
|
||||
model={"name": "model", "api_url": "https://model.example/v1"},
|
||||
config={"generation_retries": 0},
|
||||
task_id="task-truncated",
|
||||
split={"train": 100, "validation": 0, "test": 0},
|
||||
qa_pairs_per_item=1,
|
||||
client=client,
|
||||
)
|
||||
|
||||
assert records[0]["status"] == "invalid"
|
||||
assert "Token" in records[0]["error"]
|
||||
assert "截断" in records[0]["error"]
|
||||
|
||||
|
||||
def test_token_truncation_is_not_retried_even_when_json_looks_complete() -> None:
|
||||
request_count = 0
|
||||
content = json.dumps({
|
||||
"items": [{
|
||||
"instruction": "问题",
|
||||
"output": "表面完整但服务端已声明截断。",
|
||||
}],
|
||||
}, ensure_ascii=False)
|
||||
|
||||
def handler(_: httpx.Request) -> httpx.Response:
|
||||
nonlocal request_count
|
||||
request_count += 1
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"choices": [{
|
||||
"finish_reason": "length",
|
||||
"message": {"content": content},
|
||||
}],
|
||||
},
|
||||
)
|
||||
|
||||
records = generate_model_records(
|
||||
[{"id": "preview-length", "edited_content": "来源正文"}],
|
||||
model={"name": "model", "api_url": "https://model.example/v1"},
|
||||
config={"generation_retries": 5},
|
||||
task_id="task-length",
|
||||
split={"train": 100, "validation": 0, "test": 0},
|
||||
qa_pairs_per_item=1,
|
||||
client=httpx.Client(transport=httpx.MockTransport(handler)),
|
||||
)
|
||||
|
||||
assert request_count == 1
|
||||
assert records[0]["status"] == "invalid"
|
||||
assert "finish_reason=length" in records[0]["error"]
|
||||
|
||||
|
||||
def test_sensitive_model_response_is_not_retried_or_saved() -> None:
|
||||
request_count = 0
|
||||
|
||||
def handler(_: httpx.Request) -> httpx.Response:
|
||||
nonlocal request_count
|
||||
request_count += 1
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"choices": [{
|
||||
"finish_reason": "stop",
|
||||
"message": {"content": "{}"},
|
||||
}],
|
||||
"output_sensitive": True,
|
||||
"base_resp": {"status_code": 1027, "status_msg": "output sensitive"},
|
||||
},
|
||||
)
|
||||
|
||||
records = generate_model_records(
|
||||
[{"id": "preview-sensitive", "edited_content": "来源正文"}],
|
||||
model={"name": "model", "api_url": "https://model.example/v1"},
|
||||
config={"generation_retries": 5},
|
||||
task_id="task-sensitive",
|
||||
split={"train": 100, "validation": 0, "test": 0},
|
||||
qa_pairs_per_item=1,
|
||||
client=httpx.Client(transport=httpx.MockTransport(handler)),
|
||||
)
|
||||
|
||||
assert request_count == 1
|
||||
assert records[0]["status"] == "invalid"
|
||||
assert "安全拦截" in records[0]["error"]
|
||||
assert "1027" in records[0]["error"]
|
||||
|
||||
|
||||
def test_empty_model_content_can_retry_then_succeed() -> None:
|
||||
request_count = 0
|
||||
|
||||
def handler(_: httpx.Request) -> httpx.Response:
|
||||
nonlocal request_count
|
||||
request_count += 1
|
||||
if request_count == 1:
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={"choices": [{"finish_reason": "stop", "message": {"content": ""}}]},
|
||||
)
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"choices": [{
|
||||
"finish_reason": "stop",
|
||||
"message": {
|
||||
"content": json.dumps({
|
||||
"items": [{
|
||||
"instruction": "问题",
|
||||
"output": "第二次请求返回了完整答案。",
|
||||
}],
|
||||
}, ensure_ascii=False),
|
||||
},
|
||||
}],
|
||||
},
|
||||
)
|
||||
|
||||
records = generate_model_records(
|
||||
[{"id": "preview-empty-retry", "edited_content": "来源正文"}],
|
||||
model={"name": "model", "api_url": "https://model.example/v1"},
|
||||
config={"generation_retries": 1},
|
||||
task_id="task-empty-retry",
|
||||
split={"train": 100, "validation": 0, "test": 0},
|
||||
qa_pairs_per_item=1,
|
||||
client=httpx.Client(transport=httpx.MockTransport(handler)),
|
||||
)
|
||||
|
||||
assert request_count == 2
|
||||
assert records[0]["status"] == "valid"
|
||||
|
||||
|
||||
def test_multiple_top_level_json_documents_are_rejected_as_ambiguous() -> None:
|
||||
first = json.dumps({
|
||||
"items": [{"instruction": "问题一", "output": "答案一"}],
|
||||
}, ensure_ascii=False)
|
||||
second = json.dumps({
|
||||
"items": [{"instruction": "问题二", "output": "答案二"}],
|
||||
}, ensure_ascii=False)
|
||||
client = httpx.Client(
|
||||
transport=httpx.MockTransport(
|
||||
lambda _: httpx.Response(
|
||||
200,
|
||||
json={"choices": [{"message": {"content": f"{first}\n{second}"}}]},
|
||||
)
|
||||
)
|
||||
)
|
||||
|
||||
records = generate_model_records(
|
||||
[{"id": "preview-ambiguous", "edited_content": "来源正文"}],
|
||||
model={"name": "model", "api_url": "https://model.example/v1"},
|
||||
config={"generation_retries": 0},
|
||||
task_id="task-ambiguous",
|
||||
split={"train": 100, "validation": 0, "test": 0},
|
||||
qa_pairs_per_item=1,
|
||||
client=client,
|
||||
)
|
||||
|
||||
assert records[0]["status"] == "invalid"
|
||||
assert "多个 JSON" in records[0]["error"]
|
||||
|
||||
|
||||
def test_generate_model_records_builds_reasoning_output_with_think_tags() -> None:
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
payload = json.loads(request.content)
|
||||
system_prompt = payload["messages"][0]["content"]
|
||||
assert '"reasoning":"...","answer":"..."' in system_prompt
|
||||
assert "你正在生成用于训练推理模型的思维链数据" in system_prompt
|
||||
assert "推理详细程度为“普通”" in system_prompt
|
||||
assert "系统会在保存时统一组装" in system_prompt
|
||||
content = "<think>模型接口自己的分析</think>" + json.dumps(
|
||||
{
|
||||
"items": [
|
||||
{
|
||||
"instruction": "计算两项费用合计",
|
||||
"input": "交通费 30 元，餐费 20 元",
|
||||
"reasoning": "先识别两项费用，再计算 30 + 20。",
|
||||
"answer": "合计 50 元。",
|
||||
}
|
||||
]
|
||||
},
|
||||
ensure_ascii=False,
|
||||
)
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={"choices": [{"message": {"content": content}}]},
|
||||
)
|
||||
|
||||
records = generate_model_records(
|
||||
[{"id": "preview-reasoning", "edited_content": "交通费 30 元，餐费 20 元"}],
|
||||
model={"name": "model", "api_url": "https://model.example/v1"},
|
||||
config={"output_type": "reasoning"},
|
||||
task_id="task-reasoning",
|
||||
split={"train": 100, "validation": 0, "test": 0},
|
||||
qa_pairs_per_item=1,
|
||||
client=httpx.Client(transport=httpx.MockTransport(handler)),
|
||||
)
|
||||
|
||||
assert records[0]["status"] == "valid"
|
||||
assert records[0]["output"] == (
|
||||
"<think>\n先识别两项费用,再计算 30 + 20。\n</think>\n合计 50 元。"
|
||||
)
|
||||
|
||||
|
||||
def test_generate_model_records_uses_detailed_reasoning_instruction() -> None:
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
system_prompt = json.loads(request.content)["messages"][0]["content"]
|
||||
assert "推理详细程度为“详细”" in system_prompt
|
||||
assert "完整展开问题条件、来源依据、中间计算或推导" in system_prompt
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"choices": [
|
||||
{
|
||||
"message": {
|
||||
"content": json.dumps(
|
||||
{
|
||||
"items": [
|
||||
{
|
||||
"instruction": "计算报销总额",
|
||||
"reasoning": "条件为交通费 30 元和餐费 20 元。分别核对后相加，30 + 20 = 50。",
|
||||
"answer": "报销总额为 50 元。",
|
||||
}
|
||||
]
|
||||
},
|
||||
ensure_ascii=False,
|
||||
)
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
)
|
||||
|
||||
records = generate_model_records(
|
||||
[{"id": "preview-detailed", "edited_content": "交通费 30 元，餐费 20 元"}],
|
||||
model={"name": "model", "api_url": "https://model.example/v1"},
|
||||
config={"output_type": "reasoning", "reasoning_detail": "detailed"},
|
||||
task_id="task-detailed",
|
||||
split={"train": 100, "validation": 0, "test": 0},
|
||||
qa_pairs_per_item=1,
|
||||
client=httpx.Client(transport=httpx.MockTransport(handler)),
|
||||
)
|
||||
|
||||
assert records[0]["status"] == "valid"
|
||||
assert "分别核对后相加" in records[0]["output"]
|
||||
|
||||
|
||||
def test_generate_model_records_marks_reasoning_without_reasoning_field_invalid() -> None:
|
||||
response = {
|
||||
"choices": [
|
||||
{
|
||||
"message": {
|
||||
"content": json.dumps(
|
||||
{
|
||||
"items": [
|
||||
{
|
||||
"instruction": "问题",
|
||||
"answer": "只有最终答案",
|
||||
}
|
||||
]
|
||||
},
|
||||
ensure_ascii=False,
|
||||
)
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
client = httpx.Client(
|
||||
transport=httpx.MockTransport(lambda _: httpx.Response(200, json=response))
|
||||
)
|
||||
|
||||
records = generate_model_records(
|
||||
[{"id": "preview-missing-reasoning", "edited_content": "来源正文"}],
|
||||
model={"name": "model", "api_url": "https://model.example/v1"},
|
||||
config={"output_type": "reasoning"},
|
||||
task_id="task-missing-reasoning",
|
||||
split={"train": 100, "validation": 0, "test": 0},
|
||||
qa_pairs_per_item=1,
|
||||
client=client,
|
||||
)
|
||||
|
||||
assert records[0]["status"] == "invalid"
|
||||
assert records[0]["output"] == "只有最终答案"
|
||||
assert "reasoning" in records[0]["error"]
|
||||
|
||||
|
||||
def test_standard_output_removes_model_think_block() -> None:
|
||||
content = json.dumps(
|
||||
{
|
||||
"items": [
|
||||
{
|
||||
"instruction": "问题",
|
||||
"output": "<think>不应保存的分析</think>最终答案",
|
||||
}
|
||||
]
|
||||
},
|
||||
ensure_ascii=False,
|
||||
)
|
||||
client = httpx.Client(
|
||||
transport=httpx.MockTransport(
|
||||
lambda _: httpx.Response(
|
||||
200,
|
||||
json={"choices": [{"message": {"content": content}}]},
|
||||
)
|
||||
)
|
||||
)
|
||||
|
||||
records = generate_model_records(
|
||||
[{"id": "preview-standard", "edited_content": "来源正文"}],
|
||||
model={"name": "model", "api_url": "https://model.example/v1"},
|
||||
config={"output_type": "standard"},
|
||||
task_id="task-standard",
|
||||
split={"train": 100, "validation": 0, "test": 0},
|
||||
qa_pairs_per_item=1,
|
||||
client=client,
|
||||
)
|
||||
|
||||
assert records[0]["output"] == "最终答案"
|
||||
|
||||
|
||||
def test_generate_model_records_keeps_partial_failure_for_manual_repair() -> None:
|
||||
client = httpx.Client(
|
||||
transport=httpx.MockTransport(
|
||||
lambda _: httpx.Response(200, json={"choices": [{"message": {"content": "not-json"}}]})
|
||||
)
|
||||
)
|
||||
records = generate_model_records(
|
||||
[{"id": "preview-1", "edited_content": "来源正文"}],
|
||||
model={"name": "model", "api_url": "https://model.example/v1"},
|
||||
config={"generation_retries": 1},
|
||||
task_id="task-1",
|
||||
split={"train": 80, "validation": 10, "test": 10},
|
||||
qa_pairs_per_item=1,
|
||||
client=client,
|
||||
)
|
||||
|
||||
assert len(records) == 1
|
||||
assert records[0]["status"] == "invalid"
|
||||
assert records[0]["error"]
|
||||
|
||||
|
||||
def test_generate_model_records_batches_fifty_results_with_unique_ids() -> None:
|
||||
requests: list[httpx.Request] = []
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
requests.append(request)
|
||||
batch_start = (len(requests) - 1) * 10 + 1
|
||||
batch_end = batch_start + 9
|
||||
payload = json.loads(request.content)
|
||||
system_prompt = payload["messages"][0]["content"]
|
||||
assert "items 必须包含 10 条" in system_prompt
|
||||
assert f"第 {batch_start}-{batch_end} 条" in system_prompt
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"choices": [
|
||||
{
|
||||
"message": {
|
||||
"content": json.dumps(
|
||||
{
|
||||
"items": [
|
||||
{
|
||||
"instruction": "同一问题",
|
||||
"input": "来源正文",
|
||||
"output": "同一答案",
|
||||
}
|
||||
for _ in range(batch_start, batch_end + 1)
|
||||
]
|
||||
},
|
||||
ensure_ascii=False,
|
||||
)
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
)
|
||||
|
||||
records = generate_model_records(
|
||||
[{"id": "preview-50", "edited_content": "来源正文"}],
|
||||
model={"name": "model", "api_url": "https://model.example/v1"},
|
||||
config={},
|
||||
task_id="task-50",
|
||||
split={"train": 100, "validation": 0, "test": 0},
|
||||
qa_pairs_per_item=50,
|
||||
client=httpx.Client(transport=httpx.MockTransport(handler)),
|
||||
)
|
||||
|
||||
assert len(requests) == 5
|
||||
assert len(records) == 50
|
||||
assert len({record["id"] for record in records}) == 50
|
||||
assert {record["instruction"] for record in records} == {"同一问题"}
|
||||
assert all(record["status"] == "valid" for record in records)
|
||||
|
||||
|
||||
def test_generate_model_records_preserves_successful_batches_when_one_fails() -> None:
|
||||
request_count = 0
|
||||
|
||||
def handler(_: httpx.Request) -> httpx.Response:
|
||||
nonlocal request_count
|
||||
request_count += 1
|
||||
if request_count == 2:
|
||||
return httpx.Response(500)
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"choices": [
|
||||
{
|
||||
"message": {
|
||||
"content": json.dumps(
|
||||
{
|
||||
"items": [
|
||||
{
|
||||
"instruction": f"问题 {index}",
|
||||
"output": f"答案 {index}",
|
||||
}
|
||||
for index in range(1, 11)
|
||||
]
|
||||
},
|
||||
ensure_ascii=False,
|
||||
)
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
)
|
||||
|
||||
records = generate_model_records(
|
||||
[{"id": "preview-partial", "edited_content": "来源正文"}],
|
||||
model={"name": "model", "api_url": "https://model.example/v1"},
|
||||
config={"generation_retries": 0},
|
||||
task_id="task-partial",
|
||||
split={"train": 100, "validation": 0, "test": 0},
|
||||
qa_pairs_per_item=20,
|
||||
client=httpx.Client(transport=httpx.MockTransport(handler)),
|
||||
)
|
||||
|
||||
assert len(records) == 11
|
||||
assert sum(record["status"] == "valid" for record in records) == 10
|
||||
failed = next(record for record in records if record["status"] == "invalid")
|
||||
assert "第 11-20 条" in failed["instruction"]
|
||||
assert len({record["id"] for record in records}) == len(records)
|
||||
|
||||
|
||||
def test_generate_model_records_retries_short_batch_then_marks_it_invalid() -> None:
|
||||
request_count = 0
|
||||
|
||||
def handler(_: httpx.Request) -> httpx.Response:
|
||||
nonlocal request_count
|
||||
request_count += 1
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"choices": [
|
||||
{
|
||||
"message": {
|
||||
"content": json.dumps(
|
||||
{
|
||||
"items": [
|
||||
{
|
||||
"instruction": "只有一条",
|
||||
"output": "不足本批要求数量",
|
||||
}
|
||||
]
|
||||
},
|
||||
ensure_ascii=False,
|
||||
)
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
)
|
||||
|
||||
records = generate_model_records(
|
||||
[{"id": "preview-short", "edited_content": "来源正文"}],
|
||||
model={"name": "model", "api_url": "https://model.example/v1"},
|
||||
config={"generation_retries": 1},
|
||||
task_id="task-short",
|
||||
split={"train": 100, "validation": 0, "test": 0},
|
||||
qa_pairs_per_item=10,
|
||||
client=httpx.Client(transport=httpx.MockTransport(handler)),
|
||||
)
|
||||
|
||||
assert request_count == 2
|
||||
assert len(records) == 1
|
||||
assert records[0]["status"] == "invalid"
|
||||
assert "expected 10, got 1" in records[0]["error"]
|
||||
|
||||
|
||||
def test_generate_model_records_does_not_retry_non_retryable_http_errors() -> None:
|
||||
request_count = 0
|
||||
|
||||
def handler(_: httpx.Request) -> httpx.Response:
|
||||
nonlocal request_count
|
||||
request_count += 1
|
||||
return httpx.Response(401, json={"error": {"message": "unauthorized"}})
|
||||
|
||||
records = generate_model_records(
|
||||
[{"id": "preview-auth", "edited_content": "来源内容"}],
|
||||
model={
|
||||
"api_url": "https://model.example/v1",
|
||||
"online_model_name": "test-model",
|
||||
"api_key": "invalid",
|
||||
},
|
||||
config={"generation_retries": 5},
|
||||
task_id="task-auth",
|
||||
split={"train": 100, "validation": 0, "test": 0},
|
||||
qa_pairs_per_item=1,
|
||||
client=httpx.Client(transport=httpx.MockTransport(handler)),
|
||||
)
|
||||
|
||||
assert request_count == 1
|
||||
assert records[0]["status"] == "invalid"
|
||||
assert "401" in records[0]["error"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("status_code", [408, 425, 429, 500])
|
||||
def test_generate_model_records_retries_retryable_http_statuses(
|
||||
status_code: int,
|
||||
) -> None:
|
||||
request_count = 0
|
||||
|
||||
def handler(_: httpx.Request) -> httpx.Response:
|
||||
nonlocal request_count
|
||||
request_count += 1
|
||||
if request_count == 1:
|
||||
return httpx.Response(status_code)
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"choices": [{
|
||||
"message": {
|
||||
"content": json.dumps({
|
||||
"items": [{
|
||||
"instruction": "来源内容是什么？",
|
||||
"output": "这是用于验证可重试错误的来源内容。",
|
||||
}],
|
||||
}, ensure_ascii=False),
|
||||
},
|
||||
}],
|
||||
},
|
||||
)
|
||||
|
||||
records = generate_model_records(
|
||||
[{"id": "preview-retryable", "edited_content": "来源内容"}],
|
||||
model={"name": "model", "api_url": "https://model.example/v1"},
|
||||
config={"generation_retries": 1},
|
||||
task_id="task-retryable",
|
||||
split={"train": 100, "validation": 0, "test": 0},
|
||||
qa_pairs_per_item=1,
|
||||
client=httpx.Client(transport=httpx.MockTransport(handler)),
|
||||
)
|
||||
|
||||
assert request_count == 2
|
||||
assert records[0]["status"] == "valid"
|
||||
|
||||
|
||||
def test_generate_model_records_retries_transient_network_errors() -> None:
|
||||
request_count = 0
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
nonlocal request_count
|
||||
request_count += 1
|
||||
if request_count == 1:
|
||||
raise httpx.ConnectError("temporary connection failure", request=request)
|
||||
return httpx.Response(
|
||||
200,
|
||||
json={
|
||||
"choices": [{
|
||||
"message": {
|
||||
"content": json.dumps({
|
||||
"items": [{
|
||||
"instruction": "网络恢复了吗？",
|
||||
"output": "临时连接错误后，第二次模型请求已经成功。",
|
||||
}],
|
||||
}, ensure_ascii=False),
|
||||
},
|
||||
}],
|
||||
},
|
||||
)
|
||||
|
||||
records = generate_model_records(
|
||||
[{"id": "preview-network", "edited_content": "网络重试来源"}],
|
||||
model={"name": "model", "api_url": "https://model.example/v1"},
|
||||
config={"generation_retries": 1},
|
||||
task_id="task-network",
|
||||
split={"train": 100, "validation": 0, "test": 0},
|
||||
qa_pairs_per_item=1,
|
||||
client=httpx.Client(transport=httpx.MockTransport(handler)),
|
||||
)
|
||||
|
||||
assert request_count == 2
|
||||
assert records[0]["status"] == "valid"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("qa_pairs_per_item", [0, 51])
|
||||
def test_generate_model_records_rejects_out_of_range_count(
|
||||
qa_pairs_per_item: int,
|
||||
) -> None:
|
||||
with pytest.raises(ModelGenerationError, match=r"\[1, 50\]"):
|
||||
generate_model_records(
|
||||
[],
|
||||
model={"name": "model", "api_url": "https://model.example/v1"},
|
||||
config={},
|
||||
task_id="task-invalid",
|
||||
split={"train": 100, "validation": 0, "test": 0},
|
||||
qa_pairs_per_item=qa_pairs_per_item,
|
||||
)
|
||||
|
||||
|
||||
def test_generate_model_records_rejects_unknown_output_type() -> None:
|
||||
with pytest.raises(ModelGenerationError, match="output_type"):
|
||||
generate_model_records(
|
||||
[],
|
||||
model={"name": "model", "api_url": "https://model.example/v1"},
|
||||
config={"output_type": "unknown"},
|
||||
task_id="task-invalid-output",
|
||||
split={"train": 100, "validation": 0, "test": 0},
|
||||
qa_pairs_per_item=1,
|
||||
)
|
||||
|
||||
|
||||
def test_generate_model_records_rejects_unknown_reasoning_detail() -> None:
|
||||
with pytest.raises(ModelGenerationError, match="reasoning_detail"):
|
||||
generate_model_records(
|
||||
[],
|
||||
model={"name": "model", "api_url": "https://model.example/v1"},
|
||||
config={"output_type": "reasoning", "reasoning_detail": "verbose"},
|
||||
task_id="task-invalid-reasoning-detail",
|
||||
split={"train": 100, "validation": 0, "test": 0},
|
||||
qa_pairs_per_item=1,
|
||||
)
|
||||
@@ -1,61 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from app.modules.data_process.schema_cli import REQUIRED_TASK_COLUMNS, _target_label
|
||||
|
||||
|
||||
def test_runtime_migration_fails_fast_on_incompatible_schema() -> None:
|
||||
sql_path = (
|
||||
Path(__file__).resolve().parents[1]
|
||||
/ "app"
|
||||
/ "db"
|
||||
/ "sql"
|
||||
/ "002_data_process.sql"
|
||||
)
|
||||
sql = sql_path.read_text(encoding="utf-8")
|
||||
|
||||
assert "requires 001_platform_runtime.sql first" in sql
|
||||
assert "supports only the current TEXT runtime schema" in sql
|
||||
assert "generation_run_id" in sql
|
||||
assert "results_confirmed BOOLEAN NOT NULL DEFAULT TRUE" in sql
|
||||
assert "WHERE status <> 'completed' AND results_confirmed=TRUE" in sql
|
||||
assert "ADD COLUMN IF NOT EXISTS workflow_step VARCHAR(20)" in sql
|
||||
assert "ADD COLUMN IF NOT EXISTS preview_status VARCHAR(20)" in sql
|
||||
assert "ADD COLUMN IF NOT EXISTS preview_progress NUMERIC(5,2)" in sql
|
||||
assert "ADD COLUMN IF NOT EXISTS preview_run_id TEXT" in sql
|
||||
assert "ADD COLUMN IF NOT EXISTS preview_failure_reason TEXT" in sql
|
||||
assert "ADD COLUMN IF NOT EXISTS preview_total_files INTEGER" in sql
|
||||
assert "ADD COLUMN IF NOT EXISTS preview_completed_files INTEGER" in sql
|
||||
assert "data_process_workflow_backfill_ids" in sql
|
||||
assert "ck_data_process_tasks_workflow_step" in sql
|
||||
assert "ck_data_process_tasks_preview_status" in sql
|
||||
assert "ck_data_process_tasks_preview_progress" in sql
|
||||
assert "ck_data_process_tasks_preview_file_counts" in sql
|
||||
for value in ("create", "model", "upload", "preview", "generate", "results"):
|
||||
assert f"'{value}'" in sql
|
||||
for value in ("idle", "queued", "running", "completed", "failed", "cancelled"):
|
||||
assert f"'{value}'" in sql
|
||||
assert "CREATE TABLE IF NOT EXISTS data_process_results" in sql
|
||||
assert sql.count("BEGIN;") == 1
|
||||
assert sql.rstrip().endswith("COMMIT;")
|
||||
|
||||
|
||||
def test_schema_cli_target_label_never_contains_credentials() -> None:
|
||||
label = _target_label("postgresql://secret-user:secret-password@db.example:5433/yg_ft")
|
||||
assert label == "db.example:5433/yg_ft"
|
||||
assert "secret" not in label
|
||||
|
||||
|
||||
def test_schema_check_requires_current_runtime_columns() -> None:
|
||||
assert REQUIRED_TASK_COLUMNS == (
|
||||
"generation_run_id",
|
||||
"results_confirmed",
|
||||
"workflow_step",
|
||||
"preview_status",
|
||||
"preview_progress",
|
||||
"preview_run_id",
|
||||
"preview_failure_reason",
|
||||
"preview_total_files",
|
||||
"preview_completed_files",
|
||||
)
|
||||
@@ -1,276 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path, PurePosixPath
|
||||
|
||||
import pytest
|
||||
|
||||
from app.modules.data_process import storage as storage_module
|
||||
from app.modules.data_process.storage import (
|
||||
DataProcessStorageError,
|
||||
LocalDataProcessStorage,
|
||||
StagedSourceObject,
|
||||
)
|
||||
|
||||
|
||||
def _stage(
|
||||
storage: LocalDataProcessStorage,
|
||||
*,
|
||||
batch_id: str = "batch-main",
|
||||
task_id: str = "task-1",
|
||||
source_file_id: str = "source-1",
|
||||
version: int = 1,
|
||||
name: str = "source.txt",
|
||||
content: bytes = b"payload",
|
||||
) -> StagedSourceObject:
|
||||
return storage.stage_bytes(
|
||||
batch_id=batch_id,
|
||||
task_id=task_id,
|
||||
source_file_id=source_file_id,
|
||||
version=version,
|
||||
name=name,
|
||||
content=content,
|
||||
)
|
||||
|
||||
|
||||
def _create_symlink(link: Path, target: Path, *, target_is_directory: bool = False) -> None:
|
||||
try:
|
||||
link.symlink_to(target, target_is_directory=target_is_directory)
|
||||
except (NotImplementedError, OSError) as exc:
|
||||
pytest.skip(f"当前平台不支持创建测试所需的符号链接: {exc}")
|
||||
|
||||
|
||||
def _assert_staging_empty(storage: LocalDataProcessStorage) -> None:
|
||||
assert list((storage.root / ".staging").iterdir()) == []
|
||||
|
||||
|
||||
def test_stage_publish_read_delete_roundtrip_with_unicode_filename(tmp_path: Path) -> None:
|
||||
storage = LocalDataProcessStorage(tmp_path / "storage")
|
||||
content = "第一行\n第二行，100% 完成".encode()
|
||||
|
||||
staged = _stage(
|
||||
storage,
|
||||
name="中文 数据 100%.csv",
|
||||
content=content,
|
||||
)
|
||||
|
||||
assert "%20" in staged.reference
|
||||
assert "%25" in staged.reference
|
||||
storage.publish([staged])
|
||||
|
||||
assert storage.read(staged.reference) == content
|
||||
assert storage.delete(staged.reference) is True
|
||||
assert storage.delete(staged.reference) is False
|
||||
_assert_staging_empty(storage)
|
||||
|
||||
|
||||
def test_stage_copy_creates_an_independently_deletable_source_object(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
storage = LocalDataProcessStorage(tmp_path / "storage")
|
||||
original = _stage(storage, content=b"immutable source")
|
||||
storage.publish([original])
|
||||
|
||||
copied = storage.stage_copy(
|
||||
batch_id="batch-copy",
|
||||
source_reference=original.reference,
|
||||
expected_source_task_id="task-1",
|
||||
expected_source_file_id="source-1",
|
||||
task_id="task-2",
|
||||
source_file_id="source-2",
|
||||
version=1,
|
||||
name="source.txt",
|
||||
)
|
||||
storage.publish([copied])
|
||||
|
||||
assert storage.read(copied.reference) == b"immutable source"
|
||||
assert storage.delete(
|
||||
original.reference,
|
||||
expected_task_id="task-1",
|
||||
expected_source_file_id="source-1",
|
||||
) is True
|
||||
assert storage.read(copied.reference) == b"immutable source"
|
||||
assert storage.delete(
|
||||
copied.reference,
|
||||
expected_task_id="task-2",
|
||||
expected_source_file_id="source-2",
|
||||
) is True
|
||||
_assert_staging_empty(storage)
|
||||
|
||||
|
||||
def test_db_reference_is_left_to_database_storage(tmp_path: Path) -> None:
|
||||
storage = LocalDataProcessStorage(tmp_path / "storage")
|
||||
|
||||
assert storage.read("db://source-files/source-1") is None
|
||||
assert storage.delete("db://source-files/source-1") is False
|
||||
|
||||
|
||||
def test_owned_source_can_be_streamed_by_byte_range(tmp_path: Path) -> None:
|
||||
storage = LocalDataProcessStorage(tmp_path / "storage")
|
||||
content = b"0123456789abcdef"
|
||||
staged = _stage(storage, content=content)
|
||||
storage.publish([staged])
|
||||
|
||||
assert storage.file_size(
|
||||
staged.reference,
|
||||
expected_task_id="task-1",
|
||||
expected_source_file_id="source-1",
|
||||
) == len(content)
|
||||
assert b"".join(storage.iter_bytes(
|
||||
staged.reference,
|
||||
expected_task_id="task-1",
|
||||
expected_source_file_id="source-1",
|
||||
expected_size=len(content),
|
||||
start=4,
|
||||
length=6,
|
||||
chunk_size=2,
|
||||
)) == b"456789"
|
||||
|
||||
with pytest.raises(DataProcessStorageError, match="owner mismatch"):
|
||||
storage.file_size(
|
||||
staged.reference,
|
||||
expected_task_id="another-task",
|
||||
expected_source_file_id="source-1",
|
||||
)
|
||||
with pytest.raises(DataProcessStorageError, match="does not match metadata"):
|
||||
b"".join(storage.iter_bytes(
|
||||
staged.reference,
|
||||
expected_task_id="task-1",
|
||||
expected_source_file_id="source-1",
|
||||
expected_size=len(content) + 1,
|
||||
))
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"reference",
|
||||
[
|
||||
"local://data-process/../source-1/v1/file.txt",
|
||||
"local://data-process/task-1/source-1/v1/file%2Fname.txt",
|
||||
"local://data-process/task-1/source-1/v1/file.txt?download=1",
|
||||
"local://data-process/task-1/source-1/v1/file.txt#fragment",
|
||||
"https://data-process/task-1/source-1/v1/file.txt",
|
||||
],
|
||||
ids=[
|
||||
"parent-traversal",
|
||||
"percent-encoded-slash",
|
||||
"query",
|
||||
"fragment",
|
||||
"wrong-scheme",
|
||||
],
|
||||
)
|
||||
def test_unsafe_references_are_rejected(tmp_path: Path, reference: str) -> None:
|
||||
storage = LocalDataProcessStorage(tmp_path / "storage")
|
||||
|
||||
with pytest.raises(DataProcessStorageError):
|
||||
storage.read(reference)
|
||||
with pytest.raises(DataProcessStorageError):
|
||||
storage.delete(reference)
|
||||
|
||||
|
||||
def test_publish_rejects_intermediate_directory_symlink(tmp_path: Path) -> None:
|
||||
storage = LocalDataProcessStorage(tmp_path / "storage")
|
||||
outside = tmp_path / "outside"
|
||||
outside.mkdir()
|
||||
staged = _stage(storage, task_id="linked-task")
|
||||
_create_symlink(
|
||||
storage.root / "linked-task",
|
||||
outside,
|
||||
target_is_directory=True,
|
||||
)
|
||||
|
||||
with pytest.raises(DataProcessStorageError, match="symlink|non-directory"):
|
||||
storage.publish([staged])
|
||||
|
||||
assert list(outside.iterdir()) == []
|
||||
_assert_staging_empty(storage)
|
||||
|
||||
|
||||
def test_target_symlink_is_never_followed_or_deleted(tmp_path: Path) -> None:
|
||||
storage = LocalDataProcessStorage(tmp_path / "storage")
|
||||
staged = _stage(storage, task_id="task-link", source_file_id="source-link")
|
||||
outside_file = tmp_path / "outside.txt"
|
||||
outside_file.write_bytes(b"outside sentinel")
|
||||
final_path = storage.root.joinpath(*staged._relative_path.parts)
|
||||
final_path.parent.mkdir(parents=True)
|
||||
_create_symlink(final_path, outside_file)
|
||||
|
||||
with pytest.raises(DataProcessStorageError, match="already exists"):
|
||||
storage.publish([staged])
|
||||
with pytest.raises(DataProcessStorageError, match="regular file"):
|
||||
storage.read(staged.reference)
|
||||
with pytest.raises(DataProcessStorageError, match="non-regular"):
|
||||
storage.delete(staged.reference)
|
||||
|
||||
assert final_path.is_symlink()
|
||||
assert outside_file.read_bytes() == b"outside sentinel"
|
||||
_assert_staging_empty(storage)
|
||||
|
||||
|
||||
def test_publish_rolls_back_first_object_when_second_target_collides(tmp_path: Path) -> None:
|
||||
storage = LocalDataProcessStorage(tmp_path / "storage")
|
||||
existing = _stage(
|
||||
storage,
|
||||
batch_id="batch-existing",
|
||||
source_file_id="source-existing",
|
||||
content=b"existing content",
|
||||
)
|
||||
storage.publish([existing])
|
||||
|
||||
first = _stage(
|
||||
storage,
|
||||
batch_id="batch-new",
|
||||
source_file_id="source-new",
|
||||
content=b"must be rolled back",
|
||||
)
|
||||
colliding_second = _stage(
|
||||
storage,
|
||||
batch_id="batch-new",
|
||||
source_file_id="source-existing",
|
||||
content=b"must not replace existing content",
|
||||
)
|
||||
|
||||
with pytest.raises(DataProcessStorageError, match="already exists"):
|
||||
storage.publish([first, colliding_second])
|
||||
|
||||
with pytest.raises(DataProcessStorageError, match="does not exist"):
|
||||
storage.read(first.reference)
|
||||
assert storage.read(existing.reference) == b"existing content"
|
||||
_assert_staging_empty(storage)
|
||||
|
||||
|
||||
def test_publish_rejects_manually_forged_staged_object(tmp_path: Path) -> None:
|
||||
storage = LocalDataProcessStorage(tmp_path / "storage")
|
||||
temporary_path = storage.root / ".staging" / "batch-forged" / "forged.tmp"
|
||||
temporary_path.parent.mkdir()
|
||||
temporary_path.write_bytes(b"forged content")
|
||||
relative_path = PurePosixPath("task-forged", "source-forged", "v1", "forged.txt")
|
||||
forged = StagedSourceObject(
|
||||
reference="local://data-process/task-forged/source-forged/v1/forged.txt",
|
||||
_temporary_path=temporary_path,
|
||||
_relative_path=relative_path,
|
||||
)
|
||||
|
||||
with pytest.raises(DataProcessStorageError, match="was not issued"):
|
||||
storage.publish([forged])
|
||||
with pytest.raises(DataProcessStorageError, match="was not issued"):
|
||||
storage.discard([forged])
|
||||
|
||||
assert temporary_path.read_bytes() == b"forged content"
|
||||
assert not storage.root.joinpath(*relative_path.parts).exists()
|
||||
|
||||
|
||||
def test_relative_storage_configuration_is_anchored_to_backend_root(
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
relative_configuration = Path("relative-storage") / tmp_path.name
|
||||
backend_root = Path(storage_module.__file__).resolve().parents[3]
|
||||
monkeypatch.chdir(tmp_path)
|
||||
monkeypatch.setenv("DATA_PROCESS_STORAGE_DIR", str(relative_configuration))
|
||||
storage_module.get_data_process_storage.cache_clear()
|
||||
|
||||
try:
|
||||
configured_root = storage_module._configured_storage_root()
|
||||
assert configured_root == backend_root / relative_configuration
|
||||
assert not configured_root.exists()
|
||||
finally:
|
||||
storage_module.get_data_process_storage.cache_clear()
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,85 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from llama_index.core.embeddings import MockEmbedding
|
||||
|
||||
from app.modules.data_process.document_chunking import (
|
||||
DocumentChunk,
|
||||
_compact_with_offsets,
|
||||
_project_layout_span,
|
||||
chunk_fixed_text,
|
||||
chunk_semantic_text,
|
||||
merge_short_chunks,
|
||||
)
|
||||
|
||||
|
||||
def test_fixed_splitter_preserves_offsets_and_token_limit() -> None:
|
||||
text = "第一段说明苹果。第二段说明香蕉。\n第三段说明数据库。第四段说明索引。"
|
||||
chunks = chunk_fixed_text(text, chunk_size=20, chunk_overlap=0)
|
||||
|
||||
assert len(chunks) > 1
|
||||
assert all(chunk.source_start is not None for chunk in chunks)
|
||||
assert all(chunk.source_end is not None for chunk in chunks)
|
||||
assert all(
|
||||
chunk.original_content == text[chunk.source_start : chunk.source_end]
|
||||
for chunk in chunks
|
||||
if chunk.source_start is not None and chunk.source_end is not None
|
||||
)
|
||||
assert all(chunk.token_count <= 20 for chunk in chunks)
|
||||
|
||||
|
||||
def test_semantic_splitter_uses_llamaindex_and_reapplies_maximum_size() -> None:
|
||||
text = "第一段讨论水果。第二段继续讨论香蕉。第三段讨论数据库。第四段讨论索引。"
|
||||
chunks = chunk_semantic_text(
|
||||
text,
|
||||
chunk_size=30,
|
||||
chunk_overlap=0,
|
||||
breakpoint_percentile_threshold=95,
|
||||
embed_model=MockEmbedding(embed_dim=8),
|
||||
)
|
||||
|
||||
assert len(chunks) >= 2
|
||||
assert all(chunk.token_count <= 30 for chunk in chunks)
|
||||
assert "".join(chunk.original_content for chunk in chunks) == text
|
||||
|
||||
|
||||
def test_layout_projection_ignores_layout_whitespace_but_keeps_source_lines() -> None:
|
||||
source = "标题\n第一条 这是正文。\n第二条 后续正文。"
|
||||
compact_source, offsets = _compact_with_offsets(source)
|
||||
start, end, cursor = _project_layout_span(
|
||||
source,
|
||||
"第一条\n这是正文。",
|
||||
compact_source=compact_source,
|
||||
source_offsets=offsets,
|
||||
compact_start=0,
|
||||
)
|
||||
|
||||
assert source[start:end] == "第一条 这是正文。"
|
||||
assert cursor > 0
|
||||
|
||||
|
||||
def test_short_layout_chunk_merges_with_neighbor_and_keeps_page_provenance() -> None:
|
||||
source = "短标题\n这是一段足够长的正文内容，用于测试相邻切片合并。"
|
||||
chunks = [
|
||||
DocumentChunk("短标题", "短标题", 0, 3, 1, 1, 2, source_pages=(1,)),
|
||||
DocumentChunk(
|
||||
"这是一段足够长的正文内容，用于测试相邻切片合并。",
|
||||
"这是一段足够长的正文内容，用于测试相邻切片合并。",
|
||||
4,
|
||||
len(source),
|
||||
2,
|
||||
2,
|
||||
20,
|
||||
source_pages=(1, 2),
|
||||
),
|
||||
]
|
||||
|
||||
merged = merge_short_chunks(
|
||||
chunks,
|
||||
source_text=source,
|
||||
min_token_count=10,
|
||||
max_token_count=100,
|
||||
)
|
||||
|
||||
assert len(merged) == 1
|
||||
assert merged[0].original_content == source
|
||||
assert merged[0].source_pages == (1, 2)
|
||||
@@ -1,774 +0,0 @@
|
||||
"""
|
||||
平台治理功能集成测试 —— 覆盖第 1-4 周交付内容。
|
||||
|
||||
测试策略：
|
||||
- 在导入 app 模块前 mock psycopg / psycopg_pool，避免依赖真实数据库驱动
|
||||
- 使用 FastAPI TestClient 对真实路由栈发起请求
|
||||
- 通过 mock.get_platform_store 替换为内存 FakeStore
|
||||
- 每周交付内容对应一组 test class，方便分阶段验收
|
||||
|
||||
覆盖范围：
|
||||
第 1 周 — 登录、当前用户、用户列表、权限码、日志查询
|
||||
第 2 周 — 租户、项目、项目成员、资源 ACL
|
||||
第 3 周 — 审批实例、审批模板、审计日志查询和导出
|
||||
第 4 周 — 写操作审计、审批拦截、权限校验
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sys
|
||||
import types
|
||||
from contextlib import contextmanager
|
||||
from typing import Any, Iterator
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import pytest
|
||||
from fastapi import FastAPI
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
# ============================================================
|
||||
# 在导入 app 之前 mock psycopg / psycopg_pool
|
||||
# ============================================================
|
||||
|
||||
_psycopg_mock = types.ModuleType("psycopg")
|
||||
_psycopg_mock.PgConn = type("PgConn", (), {})
|
||||
_psycopg_mock.PostgresConnectionPool = MagicMock()
|
||||
_psycopg_mock.connection = MagicMock()
|
||||
sys.modules.setdefault("psycopg", _psycopg_mock)
|
||||
|
||||
_psycopg_pool_mock = types.ModuleType("psycopg_pool")
|
||||
_psycopg_pool_mock.ConnectionPool = MagicMock()
|
||||
sys.modules.setdefault("psycopg_pool", _psycopg_pool_mock)
|
||||
|
||||
# 现在安全导入 app 模块
|
||||
from app.api.v1.endpoints.platform import ok, fail # noqa: E402
|
||||
from app.modules.tenant.router import router as tenant_router # noqa: E402
|
||||
from app.modules.project.router import router as project_router # noqa: E402
|
||||
from app.modules.approval.router import router as approval_router # noqa: E402
|
||||
from app.modules.system.router import router as system_router # noqa: E402
|
||||
from app.modules.retention.router import router as retention_router # noqa: E402
|
||||
from app.modules.resource.router import router as resource_router # noqa: E402
|
||||
from app.api.v1.endpoints.platform import router as platform_router # noqa: E402
|
||||
|
||||
PREFIX = "/modelTF"
|
||||
ADMIN_TOKEN = "platform-token-u_admin"
|
||||
OP_TOKEN = "platform-token-u_op"
|
||||
|
||||
|
||||
# ============================================================
|
||||
# FakePlatformStore —— 内存实现，模拟 PlatformStore 全部治理接口
|
||||
# ============================================================
|
||||
|
||||
class FakePlatformStore:
|
||||
"""平台治理测试专用内存 store，确保测试不连接真实数据库。"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._users: list[dict[str, Any]] = [
|
||||
{
|
||||
"id": "u_admin",
|
||||
"username": "admin",
|
||||
"display_name": "Admin",
|
||||
"role": "admin",
|
||||
"status": "active",
|
||||
"permissions": [
|
||||
"dashboard", "fine-tune", "model-eval", "model-inference",
|
||||
"model-manage", "dataset", "data-process", "data-convert",
|
||||
"compute", "hardware", "logs", "user-settings",
|
||||
],
|
||||
"last_login": "2026-08-01T10:00:00Z",
|
||||
"protected": True,
|
||||
},
|
||||
{
|
||||
"id": "u_op",
|
||||
"username": "operator",
|
||||
"display_name": "Operator",
|
||||
"role": "operator",
|
||||
"status": "active",
|
||||
"permissions": ["dashboard", "fine-tune"],
|
||||
"last_login": "2026-08-01T11:00:00Z",
|
||||
"protected": False,
|
||||
},
|
||||
]
|
||||
self._tenants: dict[str, dict[str, Any]] = {}
|
||||
self._projects: dict[str, dict[str, Any]] = {}
|
||||
self._members: dict[str, list[dict[str, Any]]] = {}
|
||||
self._acl: dict[str, list[dict[str, Any]]] = {}
|
||||
self._audit_logs: list[dict[str, Any]] = []
|
||||
self._approval_templates: dict[str, dict[str, Any]] = {}
|
||||
self._approval_instances: dict[str, dict[str, Any]] = {}
|
||||
self._retention_policies: dict[str, dict[str, Any]] = {}
|
||||
self._models: list[dict[str, Any]] = []
|
||||
self._datasets: list[dict[str, Any]] = []
|
||||
self._tasks: list[dict[str, Any]] = []
|
||||
self._compute_nodes: list[dict[str, Any]] = []
|
||||
self._gpus: list[dict[str, Any]] = []
|
||||
self._sessions: list[dict[str, Any]] = []
|
||||
self._seq = 0
|
||||
|
||||
@contextmanager
|
||||
def connect(self) -> Iterator[Any]:
|
||||
class FakeConn:
|
||||
def execute(self, *a, **kw):
|
||||
return []
|
||||
|
||||
def commit(self):
|
||||
pass
|
||||
|
||||
def rollback(self):
|
||||
pass
|
||||
|
||||
def close(self):
|
||||
pass
|
||||
|
||||
yield FakeConn()
|
||||
|
||||
# ---- helpers ----
|
||||
|
||||
def _next_id(self, prefix: str) -> str:
|
||||
self._seq += 1
|
||||
return f"{prefix}_{self._seq}"
|
||||
|
||||
# ==================== 第1周：登录 / 用户 / 权限码 / 日志 ====================
|
||||
|
||||
def login(self, username: str, password: str) -> dict[str, Any] | None:
|
||||
for u in self._users:
|
||||
if u["username"] == username and u["status"] == "active":
|
||||
if password in ("admin123", "operator123", "test123"):
|
||||
return dict(u)
|
||||
return None
|
||||
|
||||
def users(self) -> list[dict[str, Any]]:
|
||||
return [dict(u) for u in self._users]
|
||||
|
||||
def create_user(self, payload: dict[str, Any]) -> dict[str, Any]:
|
||||
u = {"id": self._next_id("u"), "protected": False, **payload}
|
||||
self._users.append(u)
|
||||
return u
|
||||
|
||||
def update_user(self, user_id: str, payload: dict[str, Any]) -> dict[str, Any]:
|
||||
for u in self._users:
|
||||
if u["id"] == user_id:
|
||||
u.update(payload)
|
||||
return u
|
||||
raise KeyError(user_id)
|
||||
|
||||
def delete_user(self, user_id: str) -> None:
|
||||
self._users = [u for u in self._users if u["id"] != user_id]
|
||||
|
||||
def roles(self) -> list[dict[str, Any]]:
|
||||
return [
|
||||
{"name": "admin", "display_name": "管理员"},
|
||||
{"name": "operator", "display_name": "操作员"},
|
||||
{"name": "viewer", "display_name": "访客"},
|
||||
]
|
||||
|
||||
def log_files(self, date: str | None = None) -> list[dict[str, Any]]:
|
||||
return [{"name": "backend-2026-08-01.log", "size": "1 KB", "date": "2026-08-01"}]
|
||||
|
||||
def log_content(self, file: str) -> dict[str, Any]:
|
||||
return {"file": file, "content": "[INFO] test line", "size": "1 KB"}
|
||||
|
||||
def training_log_files(self) -> list[dict[str, Any]]:
|
||||
return [{"task_id": "ft_001", "name": "ft_001.log", "size": "2 KB"}]
|
||||
|
||||
def training_log_content(self, file: str) -> dict[str, Any]:
|
||||
return {"file": file, "content": "epoch 0 loss 1.0", "size": "2 KB"}
|
||||
|
||||
# ==================== 第2周：租户 / 项目 / 成员 / ACL ====================
|
||||
|
||||
def tenants(self) -> list[dict[str, Any]]:
|
||||
return list(self._tenants.values())
|
||||
|
||||
def tenant(self, tenant_id: str) -> dict[str, Any]:
|
||||
if tenant_id not in self._tenants:
|
||||
raise KeyError(tenant_id)
|
||||
return dict(self._tenants[tenant_id])
|
||||
|
||||
def create_tenant(self, payload: dict[str, Any]) -> dict[str, Any]:
|
||||
tid = self._next_id("tnt")
|
||||
t = {"id": tid, "status": "active", "quota": "{}", "retention_policy_id": None,
|
||||
"create_time": "2026-08-01T00:00:00Z", **payload}
|
||||
self._tenants[tid] = t
|
||||
return dict(t)
|
||||
|
||||
def update_tenant(self, tenant_id: str, payload: dict[str, Any]) -> dict[str, Any]:
|
||||
self._tenants[tenant_id].update(payload)
|
||||
return dict(self._tenants[tenant_id])
|
||||
|
||||
def set_tenant_quota(self, tenant_id: str, quota: dict[str, Any]) -> dict[str, Any]:
|
||||
self._tenants[tenant_id]["quota"] = json.dumps(quota)
|
||||
return dict(self._tenants[tenant_id])
|
||||
|
||||
def set_tenant_retention(self, tenant_id: str, retention_policy_id: str | None) -> dict[str, Any]:
|
||||
self._tenants[tenant_id]["retention_policy_id"] = retention_policy_id
|
||||
return dict(self._tenants[tenant_id])
|
||||
|
||||
def projects(self, *, tenant_id: str = "default", status: str | None = None, keyword: str | None = None) -> list[dict[str, Any]]:
|
||||
result = []
|
||||
for p in self._projects.values():
|
||||
if p.get("tenant_id") != tenant_id:
|
||||
continue
|
||||
if status and p.get("status") != status:
|
||||
continue
|
||||
if keyword and keyword.lower() not in p.get("name", "").lower():
|
||||
continue
|
||||
result.append(dict(p))
|
||||
return result
|
||||
|
||||
def project(self, project_id: str) -> dict[str, Any]:
|
||||
if project_id not in self._projects:
|
||||
raise KeyError(project_id)
|
||||
return dict(self._projects[project_id])
|
||||
|
||||
def create_project(self, payload: dict[str, Any]) -> dict[str, Any]:
|
||||
pid = self._next_id("prj")
|
||||
p = {"id": pid, "status": "active", "quota": "{}", "create_time": "2026-08-01T00:00:00Z", **payload}
|
||||
self._projects[pid] = p
|
||||
self._members[pid] = []
|
||||
return dict(p)
|
||||
|
||||
def update_project(self, project_id: str, payload: dict[str, Any]) -> dict[str, Any]:
|
||||
self._projects[project_id].update(payload)
|
||||
return dict(self._projects[project_id])
|
||||
|
||||
def archive_project(self, project_id: str) -> dict[str, Any]:
|
||||
self._projects[project_id]["status"] = "archived"
|
||||
return dict(self._projects[project_id])
|
||||
|
||||
def delete_project(self, project_id: str) -> None:
|
||||
self._projects.pop(project_id, None)
|
||||
self._members.pop(project_id, None)
|
||||
|
||||
def project_members(self, project_id: str) -> list[dict[str, Any]]:
|
||||
return [dict(m) for m in self._members.get(project_id, [])]
|
||||
|
||||
def add_project_member(self, project_id: str, payload: dict[str, Any]) -> dict[str, Any]:
|
||||
m = {"joined_at": "2026-08-01T00:00:00Z", **payload}
|
||||
self._members.setdefault(project_id, []).append(m)
|
||||
return m
|
||||
|
||||
def update_project_member_role(self, project_id: str, user_id: str, role: str) -> dict[str, Any]:
|
||||
for m in self._members.get(project_id, []):
|
||||
if m["user_id"] == user_id:
|
||||
m["role"] = role
|
||||
return m
|
||||
raise KeyError(user_id)
|
||||
|
||||
def remove_project_member(self, project_id: str, user_id: str) -> None:
|
||||
self._members[project_id] = [m for m in self._members.get(project_id, []) if m["user_id"] != user_id]
|
||||
|
||||
# ---- ACL ----
|
||||
|
||||
def get_acl(self, resource_type: str, resource_id: str) -> list[dict[str, Any]]:
|
||||
key = f"{resource_type}:{resource_id}"
|
||||
return [dict(a) for a in self._acl.get(key, [])]
|
||||
|
||||
def set_acl(self, resource_type: str, resource_id: str, entries: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
key = f"{resource_type}:{resource_id}"
|
||||
self._acl[key] = [dict(e) for e in entries]
|
||||
return self.get_acl(resource_type, resource_id)
|
||||
|
||||
def resource_acl(self, resource_type: str, resource_id: str) -> list[dict[str, Any]]:
|
||||
rows = self.get_acl(resource_type, resource_id)
|
||||
grouped: dict[str, dict[str, Any]] = {}
|
||||
for r in rows:
|
||||
k = f"{r.get('principal_type')}:{r.get('principal_id')}"
|
||||
bucket = grouped.setdefault(k, {
|
||||
"subject_type": r.get("principal_type"),
|
||||
"subject_id": r.get("principal_id"),
|
||||
"permissions": [],
|
||||
})
|
||||
perm = r.get("permission")
|
||||
if perm and perm not in bucket["permissions"]:
|
||||
bucket["permissions"].append(perm)
|
||||
return list(grouped.values())
|
||||
|
||||
def set_resource_acl(self, resource_type: str, resource_id: str, entries: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
flat: list[dict[str, Any]] = []
|
||||
for e in entries:
|
||||
for perm in e.get("permissions") or []:
|
||||
flat.append({
|
||||
"principal_type": e.get("subject_type"),
|
||||
"principal_id": e.get("subject_id"),
|
||||
"permission": perm,
|
||||
})
|
||||
self.set_acl(resource_type, resource_id, flat)
|
||||
return self.resource_acl(resource_type, resource_id)
|
||||
|
||||
# ==================== 第3周：审批 / 审计 / 留存 ====================
|
||||
|
||||
def approval_templates(self) -> list[dict[str, Any]]:
|
||||
return list(self._approval_templates.values())
|
||||
|
||||
def create_approval_template(self, payload: dict[str, Any]) -> dict[str, Any]:
|
||||
tid = payload.get("id") or self._next_id("tpl")
|
||||
t = {"id": tid, "steps": [], "create_time": "2026-08-01T00:00:00Z", **payload}
|
||||
self._approval_templates[tid] = t
|
||||
return dict(t)
|
||||
|
||||
def approval_instances(self, *, status: str | None = None) -> list[dict[str, Any]]:
|
||||
result = []
|
||||
for i in self._approval_instances.values():
|
||||
if status and i.get("status") != status:
|
||||
continue
|
||||
result.append(dict(i))
|
||||
return result
|
||||
|
||||
def approval_instance(self, instance_id: str) -> dict[str, Any]:
|
||||
if instance_id not in self._approval_instances:
|
||||
raise KeyError(instance_id)
|
||||
return dict(self._approval_instances[instance_id])
|
||||
|
||||
def create_approval_instance(self, payload: dict[str, Any]) -> dict[str, Any]:
|
||||
iid = self._next_id("appr")
|
||||
inst = {
|
||||
"id": iid,
|
||||
"status": "pending",
|
||||
"current_step": 0,
|
||||
"steps": [],
|
||||
"create_time": "2026-08-01T00:00:00Z",
|
||||
**payload,
|
||||
}
|
||||
self._approval_instances[iid] = inst
|
||||
return dict(inst)
|
||||
|
||||
def decide_approval_step(self, instance_id: str, step_index: int, *, approver_id: str, approved: bool, comment: str | None = None) -> dict[str, Any]:
|
||||
inst = self._approval_instances[instance_id]
|
||||
inst["status"] = "approved" if approved else "rejected"
|
||||
inst["current_step"] = step_index + 1
|
||||
return dict(inst)
|
||||
|
||||
def audit_logs(self, **kw) -> dict[str, Any]:
|
||||
items = [dict(l) for l in self._audit_logs]
|
||||
for filter_key in ("tenant_id", "project_id", "actor_id", "action", "target_type"):
|
||||
val = kw.get(filter_key)
|
||||
if val:
|
||||
items = [l for l in items if l.get(filter_key) == val]
|
||||
limit = kw.get("limit", 50)
|
||||
offset = kw.get("offset", 0)
|
||||
total = len(items)
|
||||
items = items[offset:offset + limit]
|
||||
return {"items": items, "total": total}
|
||||
|
||||
def record_audit(self, **kw) -> None:
|
||||
log = {"id": self._next_id("log"), "time": "2026-08-01T12:00:00Z", **kw}
|
||||
self._audit_logs.append(log)
|
||||
|
||||
# ---- 留存策略 ----
|
||||
|
||||
def retention_policies(self) -> list[dict[str, Any]]:
|
||||
return list(self._retention_policies.values())
|
||||
|
||||
def retention_policy(self, policy_id: str) -> dict[str, Any]:
|
||||
if policy_id not in self._retention_policies:
|
||||
raise KeyError(policy_id)
|
||||
return dict(self._retention_policies[policy_id])
|
||||
|
||||
def create_retention_policy(self, payload: dict[str, Any]) -> dict[str, Any]:
|
||||
pid = payload.get("id") or self._next_id("rpol")
|
||||
p = {"id": pid, "status": "active", "create_time": "2026-08-01T00:00:00Z", **payload}
|
||||
self._retention_policies[pid] = p
|
||||
return dict(p)
|
||||
|
||||
def update_retention_policy(self, policy_id: str, payload: dict[str, Any]) -> dict[str, Any]:
|
||||
self._retention_policies[policy_id].update(payload)
|
||||
return dict(self._retention_policies[policy_id])
|
||||
|
||||
def delete_retention_policy(self, policy_id: str) -> None:
|
||||
self._retention_policies.pop(policy_id, None)
|
||||
|
||||
# ---- dashboard & other stubs ----
|
||||
|
||||
def login_duration_rank(self, limit: int = 8, days: int = 30) -> list[dict[str, Any]]:
|
||||
return [{"user": "admin", "role": "admin", "duration": 10.0}]
|
||||
|
||||
def models(self) -> list[dict[str, Any]]:
|
||||
return self._models
|
||||
|
||||
def datasets(self) -> list[dict[str, Any]]:
|
||||
return self._datasets
|
||||
|
||||
def tasks(self) -> list[dict[str, Any]]:
|
||||
return self._tasks
|
||||
|
||||
def compute_nodes(self) -> list[dict[str, Any]]:
|
||||
return self._compute_nodes
|
||||
|
||||
def gpus(self) -> list[dict[str, Any]]:
|
||||
return self._gpus
|
||||
|
||||
def system_info(self) -> dict[str, Any]:
|
||||
return {"cpu": {}, "memory": {}}
|
||||
|
||||
|
||||
# ============================================================
|
||||
# 测试 fixtures
|
||||
# ============================================================
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def fake_store() -> FakePlatformStore:
|
||||
return FakePlatformStore()
|
||||
|
||||
|
||||
def _build_client(store: FakePlatformStore) -> TestClient:
|
||||
"""构建 TestClient，patch 所有治理模块的 get_platform_store。"""
|
||||
app = FastAPI()
|
||||
app.include_router(platform_router, prefix=PREFIX)
|
||||
app.include_router(system_router, prefix=PREFIX)
|
||||
app.include_router(tenant_router, prefix=PREFIX)
|
||||
app.include_router(project_router, prefix=PREFIX)
|
||||
app.include_router(approval_router, prefix=PREFIX)
|
||||
app.include_router(retention_router, prefix=PREFIX)
|
||||
app.include_router(resource_router, prefix=PREFIX)
|
||||
|
||||
patches = [
|
||||
patch("app.db.platform_store.get_platform_store", return_value=store),
|
||||
patch("app.core.auth.get_platform_store", return_value=store),
|
||||
patch("app.api.v1.endpoints.platform.get_platform_store", return_value=store),
|
||||
patch("app.modules.system.router.get_platform_store", return_value=store),
|
||||
patch("app.modules.tenant.router.get_platform_store", return_value=store),
|
||||
patch("app.modules.project.router.get_platform_store", return_value=store),
|
||||
patch("app.modules.approval.router.get_platform_store", return_value=store),
|
||||
patch("app.modules.retention.router.get_platform_store", return_value=store),
|
||||
patch("app.modules.resource.router.get_platform_store", return_value=store),
|
||||
]
|
||||
for p in patches:
|
||||
p.start()
|
||||
client = TestClient(app, raise_server_exceptions=False)
|
||||
client._fake_store = store # type: ignore[attr-defined]
|
||||
return client
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def client(fake_store: FakePlatformStore) -> TestClient:
|
||||
c = _build_client(fake_store)
|
||||
yield c
|
||||
|
||||
|
||||
def _admin_headers() -> dict[str, str]:
|
||||
return {"Authorization": f"Bearer {ADMIN_TOKEN}"}
|
||||
|
||||
|
||||
def _op_headers() -> dict[str, str]:
|
||||
return {"Authorization": f"Bearer {OP_TOKEN}"}
|
||||
|
||||
|
||||
# ============================================================
|
||||
# 第 1 周测试：登录、当前用户、用户列表、权限码、日志查询
|
||||
# ============================================================
|
||||
|
||||
class TestWeek1AuthUserPermissionsLogs:
|
||||
"""第 1 周：登录、当前用户、用户列表、权限码、日志查询接口。"""
|
||||
|
||||
def test_login_success(self, client: TestClient):
|
||||
resp = client.post(f"{PREFIX}/login", json={"username": "admin", "password": "admin123"})
|
||||
assert resp.status_code == 200
|
||||
data = resp.json()["data"]
|
||||
assert data["token"] == ADMIN_TOKEN
|
||||
assert data["user"]["username"] == "admin"
|
||||
|
||||
def test_login_invalid(self, client: TestClient):
|
||||
resp = client.post(f"{PREFIX}/login", json={"username": "admin", "password": "wrong"})
|
||||
assert resp.status_code == 401
|
||||
|
||||
def test_me_with_valid_token(self, client: TestClient):
|
||||
resp = client.get(f"{PREFIX}/me", headers=_admin_headers())
|
||||
assert resp.status_code == 200
|
||||
assert resp.json()["data"]["username"] == "admin"
|
||||
|
||||
def test_me_without_token(self, client: TestClient):
|
||||
resp = client.get(f"{PREFIX}/me")
|
||||
assert resp.status_code == 401
|
||||
|
||||
def test_users_list(self, client: TestClient):
|
||||
resp = client.get(f"{PREFIX}/users", headers=_admin_headers())
|
||||
assert resp.status_code == 200
|
||||
users = resp.json()["data"]
|
||||
assert len(users) >= 2
|
||||
assert any(u["username"] == "admin" for u in users)
|
||||
|
||||
def test_create_user(self, client: TestClient):
|
||||
resp = client.post(
|
||||
f"{PREFIX}/users",
|
||||
json={"username": "tester", "display_name": "Tester", "role": "viewer", "password": "test123"},
|
||||
headers=_admin_headers(),
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
assert resp.json()["data"]["username"] == "tester"
|
||||
|
||||
def test_permission_codes(self, client: TestClient):
|
||||
resp = client.get(f"{PREFIX}/system/permissions/codes")
|
||||
assert resp.status_code == 200
|
||||
codes = resp.json()["data"]["codes"]
|
||||
assert "dashboard" in codes
|
||||
assert "user-settings" in codes
|
||||
|
||||
def test_permissions_overview(self, client: TestClient):
|
||||
resp = client.get(f"{PREFIX}/system/permissions")
|
||||
assert resp.status_code == 200
|
||||
data = resp.json()["data"]
|
||||
assert "codes" in data
|
||||
assert "roles" in data
|
||||
|
||||
def test_log_files(self, client: TestClient):
|
||||
resp = client.get(f"{PREFIX}/log-files", headers=_admin_headers())
|
||||
assert resp.status_code == 200
|
||||
files = resp.json()["data"]
|
||||
assert len(files) >= 1
|
||||
|
||||
def test_log_content(self, client: TestClient):
|
||||
resp = client.get(f"{PREFIX}/log-content", params={"file": "backend.log"}, headers=_admin_headers())
|
||||
assert resp.status_code == 200
|
||||
assert "content" in resp.json()["data"]
|
||||
|
||||
def test_training_log_files(self, client: TestClient):
|
||||
resp = client.get(f"{PREFIX}/training-log-files", headers=_admin_headers())
|
||||
assert resp.status_code == 200
|
||||
assert len(resp.json()["data"]) >= 1
|
||||
|
||||
def test_training_log_content(self, client: TestClient):
|
||||
resp = client.get(f"{PREFIX}/training-log-content", params={"file": "ft_001.log"}, headers=_admin_headers())
|
||||
assert resp.status_code == 200
|
||||
assert "content" in resp.json()["data"]
|
||||
|
||||
|
||||
# ============================================================
|
||||
# 第 2 周测试：租户、项目、项目成员、资源 ACL
|
||||
# ============================================================
|
||||
|
||||
class TestWeek2TenantProjectACL:
|
||||
"""第 2 周：租户、项目、项目成员、资源 ACL。"""
|
||||
|
||||
def test_tenant_crud(self, client: TestClient):
|
||||
# 创建
|
||||
resp = client.post(f"{PREFIX}/tenants", json={"name": "Tenant-A", "code": "ta"}, headers=_admin_headers())
|
||||
assert resp.status_code == 200
|
||||
tid = resp.json()["data"]["id"]
|
||||
# 查列表
|
||||
resp = client.get(f"{PREFIX}/tenants", headers=_admin_headers())
|
||||
assert resp.status_code == 200
|
||||
assert any(t["id"] == tid for t in resp.json()["data"])
|
||||
# 查详情
|
||||
resp = client.get(f"{PREFIX}/tenants/{tid}", headers=_admin_headers())
|
||||
assert resp.status_code == 200
|
||||
assert resp.json()["data"]["name"] == "Tenant-A"
|
||||
# 更新
|
||||
resp = client.put(f"{PREFIX}/tenants/{tid}", json={"name": "Tenant-A2"}, headers=_admin_headers())
|
||||
assert resp.status_code == 200
|
||||
assert resp.json()["data"]["name"] == "Tenant-A2"
|
||||
|
||||
def test_tenant_quota(self, client: TestClient):
|
||||
resp = client.post(f"{PREFIX}/tenants", json={"name": "Q-Tenant", "code": "qt"}, headers=_admin_headers())
|
||||
tid = resp.json()["data"]["id"]
|
||||
resp = client.put(f"{PREFIX}/tenants/{tid}/quota", json={"quota": {"gpu": 4}}, headers=_admin_headers())
|
||||
assert resp.status_code == 200
|
||||
|
||||
def test_tenant_retention(self, client: TestClient):
|
||||
resp = client.post(f"{PREFIX}/tenants", json={"name": "R-Tenant", "code": "rt"}, headers=_admin_headers())
|
||||
tid = resp.json()["data"]["id"]
|
||||
resp = client.put(f"{PREFIX}/tenants/{tid}/retention-policy", json={"retention_policy_id": "rpol_1"}, headers=_admin_headers())
|
||||
assert resp.status_code == 200
|
||||
|
||||
def test_project_crud(self, client: TestClient):
|
||||
# 创建项目
|
||||
resp = client.post(f"{PREFIX}/projects", json={"name": "Proj-1", "code": "p1", "tenant_id": "default"}, headers=_admin_headers())
|
||||
assert resp.status_code == 200
|
||||
pid = resp.json()["data"]["id"]
|
||||
# 查列表
|
||||
resp = client.get(f"{PREFIX}/projects", params={"tenant_id": "default"}, headers=_admin_headers())
|
||||
assert resp.status_code == 200
|
||||
assert any(p["id"] == pid for p in resp.json()["data"])
|
||||
# 查详情
|
||||
resp = client.get(f"{PREFIX}/projects/{pid}", headers=_admin_headers())
|
||||
assert resp.status_code == 200
|
||||
assert resp.json()["data"]["name"] == "Proj-1"
|
||||
# 更新
|
||||
resp = client.put(f"{PREFIX}/projects/{pid}", json={"description": "updated"}, headers=_admin_headers())
|
||||
assert resp.status_code == 200
|
||||
# 归档
|
||||
resp = client.post(f"{PREFIX}/projects/{pid}/archive", headers=_admin_headers())
|
||||
assert resp.status_code == 200
|
||||
assert resp.json()["data"]["status"] == "archived"
|
||||
|
||||
def test_project_members(self, client: TestClient):
|
||||
resp = client.post(f"{PREFIX}/projects", json={"name": "Proj-M", "code": "pm", "tenant_id": "default"}, headers=_admin_headers())
|
||||
pid = resp.json()["data"]["id"]
|
||||
# 加成员
|
||||
resp = client.post(f"{PREFIX}/projects/{pid}/members", json={"user_id": "u_op", "role": "developer"}, headers=_admin_headers())
|
||||
assert resp.status_code == 200
|
||||
# 列成员
|
||||
resp = client.get(f"{PREFIX}/projects/{pid}/members", headers=_admin_headers())
|
||||
assert resp.status_code == 200
|
||||
assert len(resp.json()["data"]) >= 1
|
||||
# 改角色
|
||||
resp = client.put(f"{PREFIX}/projects/{pid}/members/u_op", json={"role": "maintainer"}, headers=_admin_headers())
|
||||
assert resp.status_code == 200
|
||||
# 删成员
|
||||
resp = client.delete(f"{PREFIX}/projects/{pid}/members/u_op", headers=_admin_headers())
|
||||
assert resp.status_code == 200
|
||||
|
||||
def test_resource_acl(self, client: TestClient):
|
||||
# 设置 ACL
|
||||
resp = client.put(
|
||||
f"{PREFIX}/resources/model/m001/acl",
|
||||
json={"entries": [{"subject_type": "user", "subject_id": "u_op", "permissions": ["read", "write"]}]},
|
||||
headers=_admin_headers(),
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
result = resp.json()["data"]
|
||||
assert len(result) == 1
|
||||
assert set(result[0]["permissions"]) == {"read", "write"}
|
||||
# 查询 ACL
|
||||
resp = client.get(f"{PREFIX}/resources/model/m001/acl", headers=_admin_headers())
|
||||
assert resp.status_code == 200
|
||||
assert len(resp.json()["data"]) == 1
|
||||
|
||||
|
||||
# ============================================================
|
||||
# 第 3 周测试：审批实例、审批模板、审计日志查询和导出
|
||||
# ============================================================
|
||||
|
||||
class TestWeek3ApprovalAudit:
|
||||
"""第 3 周：审批实例、审批模板、审计日志查询和导出。"""
|
||||
|
||||
def test_approval_template_crud(self, client: TestClient):
|
||||
# 创建模板
|
||||
resp = client.post(f"{PREFIX}/approvals/templates", json={"name": "delete-approval", "steps": [{"approver_id": "u_admin", "status": "pending"}]}, headers=_admin_headers())
|
||||
assert resp.status_code == 200
|
||||
tpl_id = resp.json()["data"]["id"]
|
||||
# 查列表
|
||||
resp = client.get(f"{PREFIX}/approvals/templates", headers=_admin_headers())
|
||||
assert resp.status_code == 200
|
||||
assert any(t["id"] == tpl_id for t in resp.json()["data"])
|
||||
|
||||
def test_approval_instance_flow(self, client: TestClient):
|
||||
# 创建审批实例
|
||||
resp = client.post(f"{PREFIX}/approvals", json={
|
||||
"resource_type": "dataset", "resource_id": "ds_001",
|
||||
"applicant_id": "u_op",
|
||||
}, headers=_admin_headers())
|
||||
assert resp.status_code == 200
|
||||
iid = resp.json()["data"]["id"]
|
||||
# 查详情
|
||||
resp = client.get(f"{PREFIX}/approvals/{iid}", headers=_admin_headers())
|
||||
assert resp.status_code == 200
|
||||
assert resp.json()["data"]["status"] == "pending"
|
||||
# 审批决策
|
||||
resp = client.post(f"{PREFIX}/approvals/{iid}/steps/0/decision", json={
|
||||
"approver_id": "u_admin", "approved": True, "comment": "ok",
|
||||
}, headers=_admin_headers())
|
||||
assert resp.status_code == 200
|
||||
assert resp.json()["data"]["status"] == "approved"
|
||||
|
||||
def test_approval_instance_reject(self, client: TestClient):
|
||||
resp = client.post(f"{PREFIX}/approvals", json={
|
||||
"resource_type": "model", "resource_id": "m_002",
|
||||
"applicant_id": "u_op",
|
||||
}, headers=_admin_headers())
|
||||
iid = resp.json()["data"]["id"]
|
||||
resp = client.post(f"{PREFIX}/approvals/{iid}/steps/0/decision", json={
|
||||
"approver_id": "u_admin", "approved": False, "comment": "no",
|
||||
}, headers=_admin_headers())
|
||||
assert resp.status_code == 200
|
||||
assert resp.json()["data"]["status"] == "rejected"
|
||||
|
||||
def test_approval_missing_field(self, client: TestClient):
|
||||
resp = client.post(f"{PREFIX}/approvals", json={"resource_type": "dataset"}, headers=_admin_headers())
|
||||
assert resp.status_code == 400
|
||||
|
||||
def test_audit_logs_query(self, client: TestClient):
|
||||
# 通过 API 写操作触发审计
|
||||
client.post(f"{PREFIX}/tenants", json={"name": "Audit-Tenant", "code": "at"}, headers=_admin_headers())
|
||||
# 查询
|
||||
resp = client.get(f"{PREFIX}/system/audit-logs", params={"limit": 50}, headers=_admin_headers())
|
||||
assert resp.status_code == 200
|
||||
data = resp.json()["data"]
|
||||
assert "items" in data
|
||||
assert "total" in data
|
||||
assert data["total"] >= 1
|
||||
|
||||
def test_audit_logs_filter_by_action(self, client: TestClient):
|
||||
resp = client.get(f"{PREFIX}/system/audit-logs", params={"action": "tenant.create"}, headers=_admin_headers())
|
||||
assert resp.status_code == 200
|
||||
items = resp.json()["data"]["items"]
|
||||
assert all(i.get("action") == "tenant.create" for i in items)
|
||||
|
||||
def test_audit_logs_export_csv(self, client: TestClient):
|
||||
resp = client.get(f"{PREFIX}/system/audit-logs/export", headers=_admin_headers())
|
||||
assert resp.status_code == 200
|
||||
assert "text/csv" in resp.headers.get("content-type", "")
|
||||
# CSV 首行是表头
|
||||
lines = resp.text.strip().split("\n")
|
||||
assert "time" in lines[0]
|
||||
|
||||
|
||||
# ============================================================
|
||||
# 第 4 周测试：写操作审计、审批拦截、权限校验
|
||||
# ============================================================
|
||||
|
||||
class TestWeek4AuditInterceptPermission:
|
||||
"""第 4 周：写操作审计、审批拦截、权限校验。"""
|
||||
|
||||
def test_write_operation_produces_audit(self, client: TestClient, fake_store: FakePlatformStore):
|
||||
# 清空审计日志便于断言
|
||||
fake_store._audit_logs.clear()
|
||||
# 创建租户 → 应产生 tenant.create 审计
|
||||
client.post(f"{PREFIX}/tenants", json={"name": "W-Tenant", "code": "wt"}, headers=_admin_headers())
|
||||
assert any(l["action"] == "tenant.create" for l in fake_store._audit_logs)
|
||||
# 创建项目 → 应产生 project.create 审计
|
||||
client.post(f"{PREFIX}/projects", json={"name": "W-Proj", "code": "wp", "tenant_id": "default"}, headers=_admin_headers())
|
||||
assert any(l["action"] == "project.create" for l in fake_store._audit_logs)
|
||||
# 设置 ACL → 应产生 resource.acl.set 审计
|
||||
client.put(f"{PREFIX}/resources/model/w001/acl", json={"entries": []}, headers=_admin_headers())
|
||||
assert any(l["action"] == "resource.acl.set" for l in fake_store._audit_logs)
|
||||
|
||||
def test_approval_intercept_on_project_archive(self, client: TestClient, fake_store: FakePlatformStore):
|
||||
# 创建项目
|
||||
resp = client.post(f"{PREFIX}/projects", json={"name": "I-Proj", "code": "ip", "tenant_id": "default"}, headers=_admin_headers())
|
||||
pid = resp.json()["data"]["id"]
|
||||
# 无待审批 → 可归档
|
||||
resp = client.post(f"{PREFIX}/projects/{pid}/archive", headers=_admin_headers())
|
||||
assert resp.status_code == 200
|
||||
|
||||
def test_approval_intercept_blocks_when_pending(self, client: TestClient, fake_store: FakePlatformStore):
|
||||
# 创建项目
|
||||
resp = client.post(f"{PREFIX}/projects", json={"name": "B-Proj", "code": "bp", "tenant_id": "default"}, headers=_admin_headers())
|
||||
pid = resp.json()["data"]["id"]
|
||||
# 注入一条待审批实例
|
||||
fake_store.create_approval_instance({
|
||||
"resource_type": "project",
|
||||
"resource_id": pid,
|
||||
"applicant_id": "u_op",
|
||||
})
|
||||
# 有待审批 → 归档应被拒绝
|
||||
resp = client.post(f"{PREFIX}/projects/{pid}/archive", headers=_admin_headers())
|
||||
assert resp.status_code == 409
|
||||
|
||||
def test_retention_policy_crud_with_audit(self, client: TestClient, fake_store: FakePlatformStore):
|
||||
fake_store._audit_logs.clear()
|
||||
# 创建
|
||||
resp = client.post(f"{PREFIX}/retention-policies", json={"name": "30d-keep", "scope": "tenant"}, headers=_admin_headers())
|
||||
assert resp.status_code == 200
|
||||
rpid = resp.json()["data"]["id"]
|
||||
assert any(l["action"] == "retention.create" for l in fake_store._audit_logs)
|
||||
# 查列表
|
||||
resp = client.get(f"{PREFIX}/retention-policies", headers=_admin_headers())
|
||||
assert resp.status_code == 200
|
||||
assert any(p["id"] == rpid for p in resp.json()["data"])
|
||||
# 更新
|
||||
resp = client.put(f"{PREFIX}/retention-policies/{rpid}", json={"status": "inactive"}, headers=_admin_headers())
|
||||
assert resp.status_code == 200
|
||||
assert resp.json()["data"]["status"] == "inactive"
|
||||
# 删除
|
||||
resp = client.delete(f"{PREFIX}/retention-policies/{rpid}", headers=_admin_headers())
|
||||
assert resp.status_code == 200
|
||||
|
||||
def test_login_duration_rank_in_dashboard(self, client: TestClient):
|
||||
resp = client.get(f"{PREFIX}/dashboard/stats", headers=_admin_headers())
|
||||
assert resp.status_code == 200
|
||||
data = resp.json()["data"]
|
||||
assert "login_duration_rank" in data
|
||||
assert "recent_login_users" in data
|
||||
assert "service_status" in data
|
||||
assert "training_7d" in data
|
||||
@@ -1,103 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from contextlib import contextmanager
|
||||
from typing import Any, Iterator
|
||||
|
||||
from app.db.platform_store import PlatformStore, dataset_file_version_summary, parse_size_bytes
|
||||
|
||||
|
||||
class _DatasetCursor:
|
||||
def __init__(self, rows: list[dict[str, Any]]) -> None:
|
||||
self.rows = rows
|
||||
|
||||
def fetchall(self) -> list[dict[str, Any]]:
|
||||
return self.rows
|
||||
|
||||
|
||||
class _DatasetConnection:
|
||||
def __init__(self) -> None:
|
||||
self.queries: list[str] = []
|
||||
|
||||
def execute(self, sql: str, params: tuple[Any, ...] | None = None) -> _DatasetCursor:
|
||||
self.queries.append(sql)
|
||||
if "FROM datasets dataset" in sql:
|
||||
return _DatasetCursor(
|
||||
[
|
||||
{
|
||||
"id": "dataset-train",
|
||||
"name": "cash-数据集-训练集",
|
||||
"type": "train",
|
||||
"storage_type": "local",
|
||||
"source": "task",
|
||||
"task_id": "task-cash",
|
||||
"source_task_id": "task-cash",
|
||||
"task_name": "cash",
|
||||
"size": "0 B",
|
||||
"size_bytes": 0,
|
||||
"metadata": "{}",
|
||||
}
|
||||
]
|
||||
)
|
||||
if "FROM dataset_files" in sql or "FROM dataset_records" in sql:
|
||||
return _DatasetCursor([])
|
||||
raise AssertionError(f"unexpected query: {sql}")
|
||||
|
||||
|
||||
def test_parse_size_bytes_supports_legacy_units() -> None:
|
||||
assert parse_size_bytes("21563 B") == 21563
|
||||
assert parse_size_bytes("1.5 KB") == 1536
|
||||
assert parse_size_bytes("2 MB") == 2 * 1024**2
|
||||
assert parse_size_bytes(4096) == 4096
|
||||
assert parse_size_bytes("unknown") == 0
|
||||
|
||||
|
||||
def test_dataset_file_version_summary_uses_active_version_metadata() -> None:
|
||||
summary = dataset_file_version_summary(
|
||||
{
|
||||
"active_version_id": "file-1-v3",
|
||||
"current_version_id": "file-1-v1",
|
||||
"version_no": 1,
|
||||
"versions": (
|
||||
'[{"id":"file-1-v1","version":1},'
|
||||
'{"id":"file-1-v3","version_no":3}]'
|
||||
),
|
||||
}
|
||||
)
|
||||
|
||||
assert summary == {
|
||||
"active_version_id": "file-1-v3",
|
||||
"current_version_id": "file-1-v3",
|
||||
"current_version_no": 3,
|
||||
"version_count": 2,
|
||||
}
|
||||
|
||||
|
||||
def test_dataset_file_version_summary_uses_normalized_version_number_as_fallback() -> None:
|
||||
summary = dataset_file_version_summary(
|
||||
{
|
||||
"active_version_id": "",
|
||||
"current_version_id": None,
|
||||
"version_no": 1,
|
||||
"versions": "[]",
|
||||
}
|
||||
)
|
||||
|
||||
assert summary["current_version_no"] == 1
|
||||
assert summary["version_count"] == 0
|
||||
|
||||
|
||||
def test_dataset_list_exposes_source_task_name() -> None:
|
||||
store = PlatformStore.__new__(PlatformStore)
|
||||
conn = _DatasetConnection()
|
||||
|
||||
@contextmanager
|
||||
def connect() -> Iterator[_DatasetConnection]:
|
||||
yield conn
|
||||
|
||||
store.connect = connect # type: ignore[method-assign]
|
||||
|
||||
[dataset] = store.datasets()
|
||||
|
||||
assert dataset["task_name"] == "cash"
|
||||
assert dataset["name"] == "cash-数据集-训练集"
|
||||
assert any("task.name AS task_name" in query for query in conn.queries)
|
||||
@@ -14,7 +14,7 @@ compute/
|
||||
tests/
|
||||
```
|
||||
|
||||
## 开发职责
|
||||
## 第一版职责
|
||||
|
||||
- GPU 发现、状态上报、锁定和释放。
|
||||
- 本地磁盘工作区管理。
|
||||
@@ -22,60 +22,3 @@ compute/
|
||||
- LLaMA-Factory 命令生成、日志解析、产物收集。
|
||||
- 分片上传、短时下载、离线导入。
|
||||
- 通过服务间 token 接受应用平台调用。
|
||||
|
||||
## 运行模式
|
||||
|
||||
- 默认 `COMPUTE_EXECUTION_MODE=real`，Compute API 会通过 `compute.agent.process_manager.ProcessManager` 启动真实 `llamafactory-cli train` 子进程，并将日志写入 `TRAINING_LOG_ROOT`。
|
||||
- 真实模式下 GPU 发现优先使用宿主机 `nvidia-smi`。如果部署环境暂时无法调用 `nvidia-smi`，可通过 `COMPUTE_GPU_COUNT`、`COMPUTE_GPU_NAME`、`COMPUTE_GPU_MEMORY_GB`、`COMPUTE_GPU_POWER_LIMIT_W` 声明兼容 GPU 清单，便于应用侧先完成节点登记和联调。
|
||||
- 仅隔离联调时可设置 `COMPUTE_EXECUTION_MODE=simulator`，启用内存状态机和合成 GPU/日志数据。该模式不得作为生产运行路径。
|
||||
- 服务间鉴权默认开启：设置 `COMPUTE_AUTH_ENABLED=true` 和一致的 `COMPUTE_SERVICE_TOKEN`，应用侧会通过 `X-Compute-Token` 调用 Compute API。
|
||||
- 真实训练作业会登记到 `TRAINING_LOG_ROOT/compute-jobs.json`。Compute API 重启后会恢复作业索引，继续提供状态、停止和日志查询。
|
||||
- 同一算力节点内按 GPU ID 做轻量锁定；已有运行中作业占用的 GPU 不允许再次提交，避免同机多 GPU 场景下误复用。
|
||||
|
||||
真实执行前提：
|
||||
|
||||
- 镜像或宿主机环境中 `llamafactory-cli` 可执行。
|
||||
- `LLAMA_FACTORY_HOME` 指向 LLaMA-Factory 工作目录。
|
||||
- 基座模型路径和数据集名称/目录已经在算力服务器本地可访问。
|
||||
- 应用侧训练任务中的 GPU、模型、数据集配置能映射到当前节点本地路径。
|
||||
|
||||
## 应用侧接入
|
||||
|
||||
应用平台通过“算力节点”页面维护每台 GPU 服务器的 `Compute API` 和 `File Gateway` 地址。点击连接测试时，Backend API 会主动调用：
|
||||
|
||||
```text
|
||||
GET /modelTF/v1/compute/health
|
||||
GET /modelTF/compute/resources/gpus
|
||||
```
|
||||
|
||||
连接成功后，应用侧会同步节点健康信息、能力标签和 GPU 清单到 PostgreSQL。多节点阶段仍按“每台算力服务器 = 单机多 GPU 节点”管理，每台服务器都部署 Compute API、Agent、File Gateway 契约和 LLaMA-Factory。
|
||||
|
||||
训练闭环：
|
||||
|
||||
```text
|
||||
Frontend 创建/启动训练
|
||||
-> Backend API 选择 compute_nodes 节点
|
||||
-> Backend API POST /modelTF/compute/jobs 到目标 Compute API
|
||||
-> Compute API 启动 llamafactory-cli 子进程
|
||||
-> Backend Worker 定时 GET /modelTF/compute/jobs/{id}
|
||||
-> Backend API 同步 fine_tune_tasks 状态、进度、PID、日志路径和产物索引
|
||||
```
|
||||
|
||||
## 当前接口能力
|
||||
|
||||
日志接口：
|
||||
|
||||
```text
|
||||
GET /modelTF/compute/jobs/{job_id}/logs?tail_lines=200
|
||||
GET /modelTF/compute/jobs/{job_id}/logs?offset=0&limit=500
|
||||
```
|
||||
|
||||
返回 `content`、`metrics`、`total_lines`、`offset`、`limit`、`has_more`、`next_offset`，用于前端增量刷新和日志平台采集。
|
||||
|
||||
文件导入：
|
||||
|
||||
```text
|
||||
POST /modelTF/compute/files/import-local
|
||||
```
|
||||
|
||||
该接口用于应用侧调度前把算力服务器本地可访问的模型/数据集路径导入到 `YG_FT_DATA_ROOT` 内部。目标路径会校验不能逃逸出 `YG_FT_DATA_ROOT`，源路径必须已存在于算力服务器本地或挂载目录。
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
"""Compute platform package."""
|
||||
@@ -1,281 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import json
|
||||
import contextlib
|
||||
import hashlib
|
||||
import signal
|
||||
import subprocess
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
TERMINAL_STATUSES = {"completed", "failed", "stopped"}
|
||||
|
||||
|
||||
@dataclass
|
||||
class ManagedProcess:
|
||||
id: str
|
||||
name: str
|
||||
command: list[str]
|
||||
work_dir: str
|
||||
log_path: Path
|
||||
output_dir: str
|
||||
gpus: list[int]
|
||||
process: subprocess.Popen[Any] | None
|
||||
created_at: float
|
||||
pid: int | None = None
|
||||
status: str = "running"
|
||||
progress: int = 5
|
||||
artifacts: list[dict[str, Any]] = field(default_factory=list)
|
||||
|
||||
|
||||
class ProcessManager:
|
||||
def __init__(self, log_root: str) -> None:
|
||||
self.log_root = Path(log_root)
|
||||
self.log_root.mkdir(parents=True, exist_ok=True)
|
||||
self.registry_path = self.log_root / "compute-jobs.json"
|
||||
self.jobs: dict[str, ManagedProcess] = {}
|
||||
self._load_registry()
|
||||
|
||||
def create_job(self, payload: dict[str, Any], command: list[str], work_dir: str) -> dict[str, Any]:
|
||||
job_id = str(payload.get("id") or f"job_{int(time.time() * 1000)}")
|
||||
if job_id in self.jobs and self.jobs[job_id].status not in TERMINAL_STATUSES:
|
||||
raise ValueError(f"job {job_id} is already running")
|
||||
|
||||
output_dir = str(payload.get("output_dir") or f"/data/yg-ft/outputs/{payload.get('name', job_id)}")
|
||||
Path(output_dir).mkdir(parents=True, exist_ok=True)
|
||||
log_path = self.log_root / f"{job_id}.log"
|
||||
env = os.environ.copy()
|
||||
gpus = [int(item) for item in payload.get("gpus") or []]
|
||||
locked = self.locked_gpus()
|
||||
conflict = sorted(set(gpus).intersection(locked))
|
||||
if conflict:
|
||||
raise ValueError(f"gpu already locked: {conflict}")
|
||||
if gpus:
|
||||
env["CUDA_VISIBLE_DEVICES"] = ",".join(str(item) for item in gpus)
|
||||
env.update({str(k): str(v) for k, v in payload.get("env", {}).items()})
|
||||
|
||||
cwd = work_dir if Path(work_dir).exists() else None
|
||||
with log_path.open("ab") as log_file:
|
||||
log_file.write(f"[INFO] starting job_id={job_id} command={' '.join(command)}\n".encode("utf-8"))
|
||||
process = subprocess.Popen(
|
||||
command,
|
||||
cwd=cwd,
|
||||
env=env,
|
||||
stdout=log_file,
|
||||
stderr=subprocess.STDOUT,
|
||||
)
|
||||
|
||||
managed = ManagedProcess(
|
||||
id=job_id,
|
||||
name=str(payload.get("name") or job_id),
|
||||
command=command,
|
||||
work_dir=work_dir,
|
||||
log_path=log_path,
|
||||
output_dir=output_dir,
|
||||
gpus=gpus,
|
||||
process=process,
|
||||
created_at=time.time(),
|
||||
pid=process.pid,
|
||||
progress=10,
|
||||
)
|
||||
self.jobs[job_id] = managed
|
||||
data = self.serialize(managed)
|
||||
self._save_registry()
|
||||
return data
|
||||
|
||||
def get_job(self, job_id: str) -> dict[str, Any] | None:
|
||||
job = self.jobs.get(job_id)
|
||||
if not job:
|
||||
return None
|
||||
return self.serialize(job)
|
||||
|
||||
def list_jobs(self) -> list[dict[str, Any]]:
|
||||
return [self.serialize(job) for job in self.jobs.values()]
|
||||
|
||||
def stop_job(self, job_id: str) -> dict[str, Any] | None:
|
||||
job = self.jobs.get(job_id)
|
||||
if not job:
|
||||
return None
|
||||
if job.status not in TERMINAL_STATUSES:
|
||||
try:
|
||||
if job.process is not None and os.name == "nt":
|
||||
job.process.terminate()
|
||||
elif job.pid is not None:
|
||||
os.kill(job.pid, signal.SIGTERM)
|
||||
if job.process is not None:
|
||||
job.process.wait(timeout=10)
|
||||
except Exception:
|
||||
if job.process is not None:
|
||||
job.process.kill()
|
||||
elif job.pid is not None:
|
||||
with contextlib.suppress(Exception):
|
||||
os.kill(job.pid, signal.SIGKILL)
|
||||
job.status = "stopped"
|
||||
job.progress = min(job.progress, 99)
|
||||
data = self.serialize(job)
|
||||
self._save_registry()
|
||||
return data
|
||||
|
||||
def logs(self, job_id: str) -> str:
|
||||
job = self.jobs.get(job_id)
|
||||
if not job or not job.log_path.exists():
|
||||
return ""
|
||||
return job.log_path.read_text(encoding="utf-8", errors="replace")
|
||||
|
||||
def serialize(self, job: ManagedProcess) -> dict[str, Any]:
|
||||
code = job.process.poll() if job.process is not None else None
|
||||
checkpoints = self._collect_checkpoints(job.output_dir)
|
||||
if job.status not in TERMINAL_STATUSES:
|
||||
if job.process is None and job.pid is not None and not self._pid_alive(job.pid):
|
||||
job.status = "failed"
|
||||
job.progress = min(job.progress, 99)
|
||||
code = -1
|
||||
elif code is None:
|
||||
job.status = "running"
|
||||
elapsed = max(0, int(time.time() - job.created_at))
|
||||
job.progress = min(95, max(job.progress, 10 + elapsed // 6))
|
||||
elif code == 0:
|
||||
job.status = "completed"
|
||||
job.progress = 100
|
||||
job.artifacts = self._collect_artifacts(job.output_dir)
|
||||
else:
|
||||
job.status = "failed"
|
||||
job.progress = min(job.progress, 99)
|
||||
self._save_registry()
|
||||
return {
|
||||
"id": job.id,
|
||||
"name": job.name,
|
||||
"status": job.status,
|
||||
"progress": job.progress,
|
||||
"pid": job.pid,
|
||||
"gpus": job.gpus,
|
||||
"created_at": job.created_at,
|
||||
"command": job.command,
|
||||
"work_dir": job.work_dir,
|
||||
"output_dir": job.output_dir,
|
||||
"log_file": str(job.log_path),
|
||||
"artifacts": job.artifacts,
|
||||
"checkpoints": checkpoints,
|
||||
"return_code": code,
|
||||
}
|
||||
|
||||
def locked_gpus(self) -> set[int]:
|
||||
locked: set[int] = set()
|
||||
for job in self.jobs.values():
|
||||
status = self.serialize(job)["status"]
|
||||
if status in {"queued", "running"}:
|
||||
locked.update(job.gpus)
|
||||
return locked
|
||||
|
||||
def _collect_artifacts(self, output_dir: str) -> list[dict[str, Any]]:
|
||||
root = Path(output_dir)
|
||||
if not root.exists():
|
||||
return []
|
||||
artifacts: list[dict[str, Any]] = []
|
||||
for path in root.rglob("*"):
|
||||
if path.is_file():
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
size = path.stat().st_size
|
||||
artifacts.append(
|
||||
{
|
||||
"path": str(path),
|
||||
"name": path.name,
|
||||
"size": size,
|
||||
"size_bytes": size,
|
||||
"checksum_sha256": digest.hexdigest(),
|
||||
}
|
||||
)
|
||||
return artifacts[:200]
|
||||
|
||||
def _collect_checkpoints(self, output_dir: str) -> list[dict[str, Any]]:
|
||||
root = Path(output_dir)
|
||||
if not root.exists():
|
||||
return []
|
||||
checkpoints: list[dict[str, Any]] = []
|
||||
for path in root.glob("checkpoint-*"):
|
||||
if not path.is_dir():
|
||||
continue
|
||||
step = 0
|
||||
try:
|
||||
step = int(path.name.rsplit("-", 1)[-1])
|
||||
except ValueError:
|
||||
step = 0
|
||||
size_bytes = sum(item.stat().st_size for item in path.rglob("*") if item.is_file())
|
||||
checkpoints.append(
|
||||
{
|
||||
"step": step,
|
||||
"name": path.name,
|
||||
"path": str(path),
|
||||
"size_bytes": size_bytes,
|
||||
"create_time": path.stat().st_mtime,
|
||||
}
|
||||
)
|
||||
return sorted(checkpoints, key=lambda item: (int(item.get("step") or 0), str(item.get("name") or "")))
|
||||
|
||||
def _save_registry(self) -> None:
|
||||
items = []
|
||||
for job in self.jobs.values():
|
||||
items.append(
|
||||
{
|
||||
"id": job.id,
|
||||
"name": job.name,
|
||||
"command": job.command,
|
||||
"work_dir": job.work_dir,
|
||||
"log_path": str(job.log_path),
|
||||
"output_dir": job.output_dir,
|
||||
"gpus": job.gpus,
|
||||
"pid": job.pid,
|
||||
"created_at": job.created_at,
|
||||
"status": job.status,
|
||||
"progress": job.progress,
|
||||
"artifacts": job.artifacts,
|
||||
}
|
||||
)
|
||||
self.registry_path.write_text(json.dumps(items, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
|
||||
def _load_registry(self) -> None:
|
||||
if not self.registry_path.exists():
|
||||
return
|
||||
try:
|
||||
items = json.loads(self.registry_path.read_text(encoding="utf-8"))
|
||||
except json.JSONDecodeError:
|
||||
return
|
||||
for item in items if isinstance(items, list) else []:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
pid = item.get("pid")
|
||||
status = item.get("status", "failed")
|
||||
if status not in TERMINAL_STATUSES and pid and not self._pid_alive(int(pid)):
|
||||
status = "failed"
|
||||
job = ManagedProcess(
|
||||
id=str(item["id"]),
|
||||
name=str(item.get("name") or item["id"]),
|
||||
command=[str(part) for part in item.get("command") or []],
|
||||
work_dir=str(item.get("work_dir") or ""),
|
||||
log_path=Path(item.get("log_path") or self.log_root / f"{item['id']}.log"),
|
||||
output_dir=str(item.get("output_dir") or ""),
|
||||
gpus=[int(gpu) for gpu in item.get("gpus") or []],
|
||||
process=None,
|
||||
pid=int(pid) if pid else None,
|
||||
created_at=float(item.get("created_at") or time.time()),
|
||||
status=status,
|
||||
progress=int(item.get("progress") or 0),
|
||||
artifacts=item.get("artifacts") or [],
|
||||
)
|
||||
self.jobs[job.id] = job
|
||||
|
||||
def _pid_alive(self, pid: int) -> bool:
|
||||
if pid <= 0:
|
||||
return False
|
||||
try:
|
||||
os.kill(pid, 0)
|
||||
return True
|
||||
except OSError:
|
||||
return False
|
||||
@@ -1,871 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
import math
|
||||
import hashlib
|
||||
import shutil
|
||||
import subprocess
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from fastapi import FastAPI, File, Form, HTTPException, Query, Request, UploadFile
|
||||
from fastapi.responses import FileResponse, JSONResponse, StreamingResponse
|
||||
|
||||
from compute.agent.process_manager import ProcessManager
|
||||
from compute.engines.llama_factory.adapter import build_command, parse_log_line, prepare_runtime_files
|
||||
from compute.engines.llama_factory.inference import get_inference_session
|
||||
|
||||
|
||||
def create_app() -> FastAPI:
|
||||
app = FastAPI(title="YG Fine-Tune Compute API")
|
||||
jobs: dict[str, dict[str, Any]] = {}
|
||||
route_prefix = os.getenv("MODELTF_ROUTE_PREFIX", "/modelTF").rstrip("/") or "/modelTF"
|
||||
process_manager = ProcessManager(os.getenv("TRAINING_LOG_ROOT", "/opt/yg-ft/logs/training"))
|
||||
|
||||
@app.middleware("http")
|
||||
async def compute_token_auth(request: Request, call_next):
|
||||
token = os.getenv("COMPUTE_SERVICE_TOKEN", "")
|
||||
auth_enabled = os.getenv("COMPUTE_AUTH_ENABLED", "true").lower() == "true"
|
||||
public_paths = {f"{route_prefix}/health", "/health"}
|
||||
if auth_enabled and token and request.url.path not in public_paths:
|
||||
header_token = request.headers.get("x-compute-token", "")
|
||||
auth_header = request.headers.get("authorization", "")
|
||||
bearer_token = auth_header.removeprefix("Bearer ").strip() if auth_header.startswith("Bearer ") else ""
|
||||
if header_token != token and bearer_token != token:
|
||||
return JSONResponse({"detail": "invalid compute service token"}, status_code=401)
|
||||
return await call_next(request)
|
||||
|
||||
def now() -> float:
|
||||
return time.time()
|
||||
|
||||
def host_id() -> str:
|
||||
return os.getenv("COMPUTE_HOST_ID", "gpu-node-01")
|
||||
|
||||
def execution_mode() -> str:
|
||||
return os.getenv("COMPUTE_EXECUTION_MODE", os.getenv("COMPUTE_MODE", "real")).lower()
|
||||
|
||||
def _int_env(name: str, default: int) -> int:
|
||||
raw = os.getenv(name)
|
||||
if raw is None or raw == "":
|
||||
return default
|
||||
return int(raw)
|
||||
|
||||
def _float_env(name: str, default: float) -> float:
|
||||
raw = os.getenv(name)
|
||||
if raw is None or raw == "":
|
||||
return default
|
||||
return float(raw)
|
||||
|
||||
def _path_inside(root: Path, candidate: Path) -> bool:
|
||||
try:
|
||||
candidate.resolve().relative_to(root.resolve())
|
||||
return True
|
||||
except ValueError:
|
||||
return False
|
||||
|
||||
def _llama_factory_version() -> str:
|
||||
for command in (["llamafactory-cli", "version"], ["llamafactory-cli", "--version"]):
|
||||
try:
|
||||
result = subprocess.run(command, capture_output=True, text=True, timeout=5)
|
||||
except Exception:
|
||||
continue
|
||||
output = (result.stdout or result.stderr).strip()
|
||||
if result.returncode == 0 and output:
|
||||
return output.splitlines()[0][:120]
|
||||
return ""
|
||||
|
||||
def torch_cuda_status() -> dict[str, Any]:
|
||||
try:
|
||||
import torch # type: ignore[import-not-found]
|
||||
except Exception as exc: # noqa: BLE001 - keep health endpoint resilient
|
||||
return {
|
||||
"available": False,
|
||||
"device_count": 0,
|
||||
"torch_version": "",
|
||||
"torch_cuda_version": "",
|
||||
"error": f"torch import failed: {exc}",
|
||||
}
|
||||
try:
|
||||
available = bool(torch.cuda.is_available())
|
||||
device_count = int(torch.cuda.device_count())
|
||||
devices = []
|
||||
for index in range(device_count):
|
||||
props = torch.cuda.get_device_properties(index)
|
||||
devices.append(
|
||||
{
|
||||
"index": index,
|
||||
"name": props.name,
|
||||
"memory_total_gb": round(props.total_memory / 1024 / 1024 / 1024, 2),
|
||||
}
|
||||
)
|
||||
return {
|
||||
"available": available,
|
||||
"device_count": device_count,
|
||||
"torch_version": str(torch.__version__),
|
||||
"torch_cuda_version": str(torch.version.cuda or ""),
|
||||
"devices": devices,
|
||||
"error": "" if available else "torch cuda is not available",
|
||||
}
|
||||
except Exception as exc: # noqa: BLE001 - expose CUDA initialization failures
|
||||
return {
|
||||
"available": False,
|
||||
"device_count": 0,
|
||||
"torch_version": str(getattr(torch, "__version__", "")),
|
||||
"torch_cuda_version": str(getattr(torch.version, "cuda", "") or ""),
|
||||
"devices": [],
|
||||
"error": str(exc),
|
||||
}
|
||||
|
||||
def _slice_log_content(
|
||||
content: str,
|
||||
tail_lines: int | None = None,
|
||||
offset: int | None = None,
|
||||
limit: int | None = None,
|
||||
) -> dict[str, Any]:
|
||||
lines = content.splitlines()
|
||||
total = len(lines)
|
||||
if offset is not None or limit is not None:
|
||||
start = max(0, offset or 0)
|
||||
end = start + limit if limit else total
|
||||
selected = lines[start:end]
|
||||
else:
|
||||
tail = tail_lines or 200
|
||||
start = max(0, total - tail)
|
||||
selected = lines[start:]
|
||||
next_offset = start + len(selected)
|
||||
return {
|
||||
"content": "\n".join(selected),
|
||||
"total_lines": total,
|
||||
"offset": start,
|
||||
"limit": len(selected),
|
||||
"has_more": next_offset < total,
|
||||
"next_offset": next_offset if next_offset < total else None,
|
||||
}
|
||||
|
||||
def _safe_float(value: Any, default: float = 0) -> float:
|
||||
try:
|
||||
return float(str(value).replace("[N/A]", "").strip() or default)
|
||||
except (TypeError, ValueError):
|
||||
return default
|
||||
|
||||
def job_status(job: dict[str, Any]) -> dict[str, Any]:
|
||||
if execution_mode() != "simulator":
|
||||
return job
|
||||
elapsed = max(0, int(now() - job["created_at"]))
|
||||
if job["status"] not in {"stopped", "failed", "completed"}:
|
||||
if elapsed < 5:
|
||||
job["status"] = "queued"
|
||||
job["progress"] = 12 + elapsed * 3
|
||||
elif elapsed < 60:
|
||||
job["status"] = "running"
|
||||
job["progress"] = min(96, 25 + int((elapsed - 5) / 55 * 70))
|
||||
else:
|
||||
job["status"] = "completed"
|
||||
job["progress"] = 100
|
||||
job["logs"] = generate_logs(job)
|
||||
return job
|
||||
|
||||
def generate_logs(job: dict[str, Any]) -> str:
|
||||
progress = int(job.get("progress", 0) or 0)
|
||||
points = max(1, min(80, progress))
|
||||
lines = [
|
||||
f"[INFO] compute_host_id={host_id()} job_id={job['id']} engine=llama_factory",
|
||||
f"[INFO] command={' '.join(job['command'])}",
|
||||
]
|
||||
for step in range(1, points + 1):
|
||||
if step % 4 != 0 and step != points:
|
||||
continue
|
||||
loss = max(0.11, 2.5 * math.exp(-step / 40))
|
||||
grad_norm = 0.4 + (step % 5) * 0.04
|
||||
lr = 0.0002 * max(0.05, 1 - step / 100)
|
||||
epoch = round(step / points * 3, 4)
|
||||
lines.append(
|
||||
"{"
|
||||
f"'loss': {loss:.4f}, 'grad_norm': {grad_norm:.4f}, "
|
||||
f"'learning_rate': {lr:.8f}, 'epoch': {epoch:.4f}"
|
||||
"}"
|
||||
)
|
||||
if job.get("status") == "completed":
|
||||
lines.extend(
|
||||
[
|
||||
"***** train metrics *****",
|
||||
"epoch = 3",
|
||||
"train_loss = 0.1181",
|
||||
"train_runtime = 1m 0s",
|
||||
"***** train metrics end *****",
|
||||
]
|
||||
)
|
||||
return "\n".join(lines)
|
||||
|
||||
def real_gpu_resources() -> list[dict[str, Any]]:
|
||||
query = (
|
||||
"index,uuid,name,memory.total,memory.used,utilization.gpu,"
|
||||
"temperature.gpu,power.draw,power.limit"
|
||||
)
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["nvidia-smi", f"--query-gpu={query}", "--format=csv,noheader,nounits"],
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=5,
|
||||
)
|
||||
except Exception:
|
||||
return fallback_gpu_resources()
|
||||
|
||||
items: list[dict[str, Any]] = []
|
||||
for line in result.stdout.splitlines():
|
||||
parts = [part.strip() for part in line.split(",")]
|
||||
if len(parts) < 9:
|
||||
continue
|
||||
idx, uuid, name, mem_total, mem_used, util, temp, power, power_limit = parts[:9]
|
||||
total_gb = round(_safe_float(mem_total) / 1024, 2)
|
||||
used_gb = round(_safe_float(mem_used) / 1024, 2)
|
||||
memory_percent = round(used_gb / total_gb * 100, 1) if total_gb else 0
|
||||
gpu_percent = int(_safe_float(util))
|
||||
items.append(
|
||||
{
|
||||
"id": int(idx),
|
||||
"gpu_index": int(idx),
|
||||
"uuid": uuid,
|
||||
"name": name,
|
||||
"status": "busy" if gpu_percent >= 5 or used_gb > 1 else "idle",
|
||||
"gpu_percent": gpu_percent,
|
||||
"memory_used_gb": used_gb,
|
||||
"memory_total_gb": total_gb,
|
||||
"memory_percent": memory_percent,
|
||||
"temperature": int(_safe_float(temp)),
|
||||
"power_w": round(_safe_float(power), 1),
|
||||
"power_limit_w": round(_safe_float(power_limit), 1),
|
||||
"processes": [],
|
||||
}
|
||||
)
|
||||
return items
|
||||
|
||||
def fallback_gpu_resources() -> list[dict[str, Any]]:
|
||||
count = _int_env("COMPUTE_GPU_COUNT", 0)
|
||||
if count <= 0:
|
||||
return []
|
||||
name = os.getenv("COMPUTE_GPU_NAME", "Configured GPU")
|
||||
memory_total = _float_env("COMPUTE_GPU_MEMORY_GB", 80.0)
|
||||
power_limit = _float_env("COMPUTE_GPU_POWER_LIMIT_W", 300.0)
|
||||
return [
|
||||
{
|
||||
"id": idx,
|
||||
"gpu_index": idx,
|
||||
"uuid": f"GPU-{host_id().upper()}-{idx}",
|
||||
"name": name,
|
||||
"status": "idle",
|
||||
"gpu_percent": 0,
|
||||
"memory_used_gb": 0,
|
||||
"memory_total_gb": memory_total,
|
||||
"memory_percent": 0,
|
||||
"temperature": _int_env("COMPUTE_GPU_BASE_TEMPERATURE", 35),
|
||||
"power_w": 0,
|
||||
"power_limit_w": power_limit,
|
||||
"processes": [],
|
||||
}
|
||||
for idx in range(count)
|
||||
]
|
||||
|
||||
def gpu_resources() -> list[dict[str, Any]]:
|
||||
if execution_mode() != "simulator":
|
||||
return real_gpu_resources()
|
||||
active_jobs = [job_status(job) for job in jobs.values() if job["status"] in {"queued", "running"}]
|
||||
gpus: list[dict[str, Any]] = []
|
||||
for idx in range(4):
|
||||
task = next((job for job in active_jobs if idx in job.get("gpus", [])), None)
|
||||
busy = task is not None and task["status"] == "running"
|
||||
reserved = task is not None and task["status"] == "queued"
|
||||
gpus.append(
|
||||
{
|
||||
"id": idx,
|
||||
"uuid": f"GPU-{host_id().upper()}-{idx}",
|
||||
"name": os.getenv("COMPUTE_GPU_NAME", "NVIDIA A800-SXM4-80GB"),
|
||||
"status": "busy" if busy else "reserved" if reserved else "idle",
|
||||
"gpu_percent": 88 if busy else 25 if reserved else 4,
|
||||
"memory_used_gb": 58 if busy else 12 if reserved else 2,
|
||||
"memory_total_gb": 80,
|
||||
"temperature": 61 if busy else 45 if reserved else 36,
|
||||
"power_w": 215 if busy else 80 if reserved else 25,
|
||||
"power_limit_w": 300,
|
||||
"processes": [
|
||||
{
|
||||
"pid": task["pid"],
|
||||
"name": "llamafactory-cli",
|
||||
"task_name": task["name"],
|
||||
"memory_used_gb": 58 if busy else 12,
|
||||
}
|
||||
]
|
||||
if task
|
||||
else [],
|
||||
}
|
||||
)
|
||||
return gpus
|
||||
|
||||
def _validate_training_accelerator(payload: dict[str, Any]) -> tuple[list[str], list[str], dict[str, Any]]:
|
||||
errors: list[str] = []
|
||||
warnings: list[str] = []
|
||||
if str(payload.get("engine") or payload.get("training_engine") or "llama_factory") == "smoke":
|
||||
return errors, warnings, {}
|
||||
requested_gpus = [int(item) for item in payload.get("gpus") or []]
|
||||
if not requested_gpus:
|
||||
warnings.append("no gpu selected; training will run on CPU")
|
||||
return errors, warnings, {}
|
||||
cuda = torch_cuda_status()
|
||||
if not cuda.get("available"):
|
||||
errors.append(f"torch cuda unavailable on compute node: {cuda.get('error') or 'unknown error'}")
|
||||
device_count = int(cuda.get("device_count") or 0)
|
||||
if device_count and max(requested_gpus) >= device_count:
|
||||
errors.append(f"requested gpu index out of torch device range: requested={requested_gpus}, device_count={device_count}")
|
||||
min_memory_gb = _float_env("MIN_TRAINING_GPU_MEMORY_GB", 4.0)
|
||||
gpus = {int(item["gpu_index"]): item for item in gpu_resources() if "gpu_index" in item}
|
||||
for gpu_index in requested_gpus:
|
||||
gpu = gpus.get(gpu_index)
|
||||
if not gpu:
|
||||
errors.append(f"requested gpu not found by nvidia-smi: {gpu_index}")
|
||||
continue
|
||||
memory_total = float(gpu.get("memory_total_gb") or 0)
|
||||
if memory_total and memory_total < min_memory_gb:
|
||||
errors.append(
|
||||
f"gpu {gpu_index} memory too small: {memory_total}GB < required {min_memory_gb}GB"
|
||||
)
|
||||
return errors, warnings, cuda
|
||||
|
||||
def _check_path_item(item: dict[str, Any]) -> dict[str, Any]:
|
||||
path = Path(str(item.get("path") or ""))
|
||||
exists = path.exists()
|
||||
expected_type = str(item.get("type") or "any")
|
||||
ok = exists
|
||||
if exists and expected_type == "dir":
|
||||
ok = path.is_dir()
|
||||
if exists and expected_type == "file":
|
||||
ok = path.is_file()
|
||||
return {
|
||||
"name": item.get("name") or "",
|
||||
"path": str(path),
|
||||
"type": expected_type,
|
||||
"required": bool(item.get("required", True)),
|
||||
"exists": exists,
|
||||
"is_dir": path.is_dir() if exists else False,
|
||||
"is_file": path.is_file() if exists else False,
|
||||
"byte_size": sum(child.stat().st_size for child in path.rglob("*") if child.is_file()) if exists and path.is_dir() else path.stat().st_size if exists and path.is_file() else 0,
|
||||
"ok": ok or not item.get("required", True),
|
||||
}
|
||||
|
||||
def _job_preview(payload: dict[str, Any], check_paths: bool) -> dict[str, Any]:
|
||||
warnings: list[str] = []
|
||||
runtime_files: list[dict[str, str]] = []
|
||||
command_payload = {**payload, "require_dataset_files": check_paths}
|
||||
if check_paths:
|
||||
try:
|
||||
runtime_files = prepare_runtime_files(command_payload)
|
||||
except OSError as exc:
|
||||
return {
|
||||
"valid": False,
|
||||
"errors": [f"prepare runtime files failed: {exc}"],
|
||||
"warnings": warnings,
|
||||
"engine": str(payload.get("engine") or payload.get("training_engine") or "llama_factory"),
|
||||
"command": [],
|
||||
"command_text": "",
|
||||
"work_dir": os.getenv("LLAMA_FACTORY_HOME", "/app/LLaMA-Factory"),
|
||||
"env": {},
|
||||
"runtime_files": [],
|
||||
"path_checks": [],
|
||||
}
|
||||
try:
|
||||
command = build_command(command_payload, os.getenv("LLAMA_FACTORY_HOME", "/app/LLaMA-Factory"))
|
||||
except ValueError as exc:
|
||||
return {
|
||||
"valid": False,
|
||||
"errors": [part.strip() for part in str(exc).split(";") if part.strip()],
|
||||
"warnings": warnings,
|
||||
"engine": str(payload.get("engine") or payload.get("training_engine") or "llama_factory"),
|
||||
"command": [],
|
||||
"command_text": "",
|
||||
"work_dir": os.getenv("LLAMA_FACTORY_HOME", "/app/LLaMA-Factory"),
|
||||
"env": {},
|
||||
"runtime_files": runtime_files,
|
||||
"path_checks": [],
|
||||
}
|
||||
|
||||
errors: list[str] = []
|
||||
engine = str(payload.get("engine") or payload.get("training_engine") or "llama_factory")
|
||||
path_checks: list[dict[str, Any]] = []
|
||||
accelerator: dict[str, Any] = {}
|
||||
if check_paths and engine != "smoke":
|
||||
path_checks = [
|
||||
_check_path_item(
|
||||
{
|
||||
"name": "model_name_or_path",
|
||||
"path": payload.get("model_name_or_path") or payload.get("base_model") or payload.get("base_model_path") or "",
|
||||
"type": "any",
|
||||
"required": True,
|
||||
}
|
||||
)
|
||||
]
|
||||
if engine in {"merge", "export", "llama_factory_export"} and payload.get("adapter_name_or_path"):
|
||||
path_checks.append(
|
||||
_check_path_item(
|
||||
{
|
||||
"name": "adapter_name_or_path",
|
||||
"path": payload.get("adapter_name_or_path"),
|
||||
"type": "any",
|
||||
"required": True,
|
||||
}
|
||||
)
|
||||
)
|
||||
if payload.get("dataset_dir"):
|
||||
path_checks.append(
|
||||
_check_path_item(
|
||||
{
|
||||
"name": "dataset_dir",
|
||||
"path": payload.get("dataset_dir"),
|
||||
"type": "dir",
|
||||
"required": True,
|
||||
}
|
||||
)
|
||||
)
|
||||
output_dir = Path(str(payload.get("output_dir") or "/data/yg-ft/outputs/training-job"))
|
||||
path_checks.append(
|
||||
_check_path_item(
|
||||
{
|
||||
"name": "output_parent",
|
||||
"path": str(output_dir.parent),
|
||||
"type": "dir",
|
||||
"required": False,
|
||||
}
|
||||
)
|
||||
)
|
||||
errors.extend(
|
||||
[f"{item['name']} path not available: {item['path']}" for item in path_checks if not item["ok"] and item["required"]]
|
||||
)
|
||||
if shutil.which(command.command[0]) is None:
|
||||
errors.append(f"training command not found: {command.command[0]}")
|
||||
if not Path(command.work_dir).exists():
|
||||
errors.append(f"llama_factory_home not found: {command.work_dir}")
|
||||
if engine not in {"merge", "export", "llama_factory_export"}:
|
||||
accelerator_errors, accelerator_warnings, accelerator = _validate_training_accelerator(payload)
|
||||
errors.extend(accelerator_errors)
|
||||
warnings.extend(accelerator_warnings)
|
||||
elif engine == "eval":
|
||||
# Eval engine: validate model path and dataset path
|
||||
if not payload.get("model_name_or_path"):
|
||||
errors.append("model_name_or_path is required for eval")
|
||||
else:
|
||||
path_checks.append(_check_path_item({
|
||||
"name": "model_name_or_path",
|
||||
"path": payload.get("model_name_or_path", ""),
|
||||
"type": "any",
|
||||
"required": True,
|
||||
}))
|
||||
if payload.get("dataset_path"):
|
||||
path_checks.append(_check_path_item({
|
||||
"name": "dataset_path",
|
||||
"path": payload.get("dataset_path", ""),
|
||||
"type": "file",
|
||||
"required": True,
|
||||
}))
|
||||
else:
|
||||
errors.append("dataset_path is required for eval")
|
||||
if shutil.which("python") is None:
|
||||
errors.append("python runtime not found")
|
||||
|
||||
elif engine == "smoke":
|
||||
warnings.append("smoke engine skips model and dataset path checks")
|
||||
|
||||
return {
|
||||
"valid": not errors,
|
||||
"errors": errors,
|
||||
"warnings": warnings,
|
||||
"engine": engine,
|
||||
"command": command.command,
|
||||
"command_text": " ".join(command.command),
|
||||
"work_dir": command.work_dir,
|
||||
"env": command.env,
|
||||
"runtime_files": runtime_files,
|
||||
"accelerator": accelerator,
|
||||
"path_checks": path_checks,
|
||||
}
|
||||
|
||||
@app.get(f"{route_prefix}/health")
|
||||
async def health_check() -> dict[str, str]:
|
||||
return {
|
||||
"status": "ok",
|
||||
"compute_host_id": os.getenv("COMPUTE_HOST_ID", "unknown"),
|
||||
}
|
||||
|
||||
@app.get("/health")
|
||||
async def health_check_root() -> dict[str, str]:
|
||||
return await health_check()
|
||||
|
||||
@app.get(f"{route_prefix}/v1/compute/health")
|
||||
async def compute_health_check() -> dict[str, Any]:
|
||||
data_root = Path(os.getenv("YG_FT_DATA_ROOT", "/data/yg-ft"))
|
||||
dataset_root = Path(os.getenv("YG_FT_DATASET_ROOT", str(data_root / "datasets")))
|
||||
output_root = Path(os.getenv("YG_FT_OUTPUT_ROOT", str(data_root / "outputs")))
|
||||
llama_factory_home = Path(os.getenv("LLAMA_FACTORY_HOME", "/app/LLaMA-Factory"))
|
||||
gpu_items = gpu_resources()
|
||||
torch_cuda = torch_cuda_status()
|
||||
return {
|
||||
"status": "ok",
|
||||
"api_version": "v1",
|
||||
"compute_host_id": os.getenv("COMPUTE_HOST_ID", "unknown"),
|
||||
"app_callback_enabled": os.getenv("ENABLE_APP_CALLBACK", "false").lower() == "true",
|
||||
"data_root": str(data_root),
|
||||
"data_root_exists": data_root.exists(),
|
||||
"model_root": os.getenv("YG_FT_MODEL_ROOT", str(data_root / "models")),
|
||||
"dataset_root": str(dataset_root),
|
||||
"dataset_root_exists": dataset_root.exists(),
|
||||
"output_root": str(output_root),
|
||||
"output_root_exists": output_root.exists(),
|
||||
"log_root": os.getenv("TRAINING_LOG_ROOT", "/opt/yg-ft/logs/training"),
|
||||
"llama_factory_home": str(llama_factory_home),
|
||||
"llama_factory_home_exists": llama_factory_home.exists(),
|
||||
"llama_factory_version": os.getenv("LLAMA_FACTORY_VERSION", ""),
|
||||
"execution_mode": execution_mode(),
|
||||
"gpu_count": _int_env("COMPUTE_GPU_COUNT", 0),
|
||||
"nvidia_gpu_count": len(gpu_items),
|
||||
"torch_cuda": torch_cuda,
|
||||
"gpu_discovery_endpoint": f"{route_prefix}/compute/resources/gpus",
|
||||
"capabilities": ["gpu_discovery", "torch_cuda_diagnostics", "llama_factory", "file_gateway", "job_polling"],
|
||||
}
|
||||
|
||||
@app.get(f"{route_prefix}/v1/compute/jobs")
|
||||
async def list_jobs_alias() -> dict[str, list[dict[str, Any]]]:
|
||||
items = process_manager.list_jobs() if execution_mode() != "simulator" else [job_status(job) for job in jobs.values()]
|
||||
return {"items": items}
|
||||
|
||||
@app.get(f"{route_prefix}/compute/resources/gpus")
|
||||
async def list_gpus() -> dict[str, Any]:
|
||||
return {"items": gpu_resources(), "compute_host_id": host_id()}
|
||||
|
||||
@app.get(f"{route_prefix}/v1/compute/resources/gpus")
|
||||
async def list_gpus_v1() -> dict[str, Any]:
|
||||
return {"items": gpu_resources(), "compute_host_id": host_id()}
|
||||
|
||||
@app.post(f"{route_prefix}/compute/jobs/preview")
|
||||
async def preview_job(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
return _job_preview(payload, check_paths=False)
|
||||
|
||||
@app.post(f"{route_prefix}/compute/jobs/validate")
|
||||
async def validate_job(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
return _job_preview(payload, check_paths=True)
|
||||
|
||||
@app.post(f"{route_prefix}/v1/compute/jobs/preview")
|
||||
async def preview_job_v1(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
return await preview_job(payload)
|
||||
|
||||
@app.post(f"{route_prefix}/v1/compute/jobs/validate")
|
||||
async def validate_job_v1(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
return await validate_job(payload)
|
||||
|
||||
@app.post(f"{route_prefix}/compute/files/check-paths")
|
||||
async def check_paths(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
items = [_check_path_item(item) for item in payload.get("paths", []) if isinstance(item, dict)]
|
||||
return {"valid": all(item["ok"] for item in items), "items": items}
|
||||
|
||||
@app.get(f"{route_prefix}/compute/files/list")
|
||||
async def list_files(
|
||||
root: str = Query(default="data"),
|
||||
relative_path: str = Query(default=""),
|
||||
directories_only: bool = Query(default=False),
|
||||
) -> dict[str, Any]:
|
||||
roots = {
|
||||
"data": Path(os.getenv("YG_FT_DATA_ROOT", "/data/yg-ft")),
|
||||
"models": Path(os.getenv("YG_FT_MODEL_ROOT", os.getenv("YG_FT_DATA_ROOT", "/data/yg-ft") + "/models")),
|
||||
"datasets": Path(os.getenv("YG_FT_DATASET_ROOT", os.getenv("YG_FT_DATA_ROOT", "/data/yg-ft") + "/datasets")),
|
||||
"outputs": Path(os.getenv("YG_FT_OUTPUT_ROOT", os.getenv("YG_FT_DATA_ROOT", "/data/yg-ft") + "/outputs")),
|
||||
}
|
||||
base = roots.get(root)
|
||||
if base is None:
|
||||
raise HTTPException(status_code=400, detail="invalid root")
|
||||
target = (base / relative_path.lstrip("/\\")).resolve()
|
||||
if not _path_inside(base, target):
|
||||
raise HTTPException(status_code=400, detail="path must stay inside selected root")
|
||||
if not target.exists():
|
||||
return {"root": root, "base_path": str(base), "relative_path": relative_path, "items": []}
|
||||
items = []
|
||||
for child in sorted(target.iterdir(), key=lambda path: (not path.is_dir(), path.name.lower())):
|
||||
if directories_only and not child.is_dir():
|
||||
continue
|
||||
items.append(
|
||||
{
|
||||
"name": child.name,
|
||||
"path": str(child),
|
||||
"relative_path": str(child.relative_to(base)).replace("\\", "/"),
|
||||
"type": "directory" if child.is_dir() else "file",
|
||||
"byte_size": child.stat().st_size if child.is_file() else 0,
|
||||
}
|
||||
)
|
||||
return {"root": root, "base_path": str(base), "relative_path": relative_path, "items": items}
|
||||
|
||||
@app.post(f"{route_prefix}/compute/jobs")
|
||||
async def create_job(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
payload = {**payload, "require_dataset_files": True}
|
||||
try:
|
||||
prepare_runtime_files(payload)
|
||||
except OSError as exc:
|
||||
raise HTTPException(status_code=400, detail=f"prepare runtime files failed: {exc}")
|
||||
try:
|
||||
command = build_command(payload, os.getenv("LLAMA_FACTORY_HOME", "/app/LLaMA-Factory"))
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc))
|
||||
job_id = str(payload.get("id") or f"job_{int(now() * 1000)}")
|
||||
if execution_mode() != "simulator":
|
||||
try:
|
||||
return process_manager.create_job({**payload, "id": job_id}, command.command, command.work_dir)
|
||||
except FileNotFoundError as exc:
|
||||
raise HTTPException(status_code=500, detail=f"training command not found: {exc.filename}")
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=409, detail=str(exc))
|
||||
job = {
|
||||
"id": job_id,
|
||||
"name": payload.get("name", job_id),
|
||||
"status": "queued",
|
||||
"progress": 10,
|
||||
"pid": int(52000 + now() % 10000),
|
||||
"gpus": payload.get("gpus") or [0],
|
||||
"created_at": now(),
|
||||
"command": command.command,
|
||||
"work_dir": command.work_dir,
|
||||
"artifacts": [],
|
||||
"logs": "",
|
||||
}
|
||||
jobs[job_id] = job
|
||||
return job_status(job)
|
||||
|
||||
@app.get(f"{route_prefix}/compute/jobs")
|
||||
async def list_jobs() -> dict[str, Any]:
|
||||
items = process_manager.list_jobs() if execution_mode() != "simulator" else [job_status(job) for job in jobs.values()]
|
||||
return {"items": items}
|
||||
|
||||
@app.get(f"{route_prefix}/compute/jobs/{{job_id}}")
|
||||
async def get_job(job_id: str) -> dict[str, Any]:
|
||||
job = jobs.get(job_id)
|
||||
if execution_mode() != "simulator":
|
||||
job = process_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="job not found")
|
||||
return job
|
||||
job = jobs.get(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="job not found")
|
||||
return job_status(job)
|
||||
|
||||
@app.post(f"{route_prefix}/compute/jobs/{{job_id}}/stop")
|
||||
async def stop_job(job_id: str) -> dict[str, Any]:
|
||||
if execution_mode() != "simulator":
|
||||
job = process_manager.stop_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="job not found")
|
||||
return job
|
||||
job = jobs.get(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="job not found")
|
||||
job["status"] = "stopped"
|
||||
job["progress"] = min(job.get("progress", 0), 99)
|
||||
return job
|
||||
|
||||
@app.get(f"{route_prefix}/compute/jobs/{{job_id}}/logs")
|
||||
async def job_logs(
|
||||
job_id: str,
|
||||
tail_lines: int | None = Query(default=200, ge=1, le=5000),
|
||||
offset: int | None = Query(default=None, ge=0),
|
||||
limit: int | None = Query(default=None, ge=1, le=5000),
|
||||
) -> dict[str, Any]:
|
||||
if execution_mode() != "simulator":
|
||||
job = process_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="job not found")
|
||||
content = process_manager.logs(job_id)
|
||||
else:
|
||||
job = jobs.get(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="job not found")
|
||||
job = job_status(job)
|
||||
content = job["logs"]
|
||||
window = _slice_log_content(content, tail_lines, offset, limit)
|
||||
metrics = [parse_log_line(line) for line in window["content"].splitlines()]
|
||||
return {"job_id": job_id, **window, "metrics": [m for m in metrics if m]}
|
||||
|
||||
# ── Inference Endpoints ───────────────────────────────────────────
|
||||
|
||||
@app.post(f"{route_prefix}/inference/load")
|
||||
async def inference_load(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Load a model for inference using LLaMA-Factory ChatModel.
|
||||
|
||||
Expected payload:
|
||||
model_name_or_path: str (required)
|
||||
adapter_name_or_path: str (optional, for LoRA adapters)
|
||||
template: str (default: "qwen")
|
||||
infer_backend: str (default: "huggingface")
|
||||
infer_dtype: str (default: "auto")
|
||||
"""
|
||||
session = get_inference_session()
|
||||
result = session.load(
|
||||
model_name_or_path=payload.get("model_name_or_path", ""),
|
||||
adapter_name_or_path=payload.get("adapter_name_or_path", ""),
|
||||
template=payload.get("template", "qwen"),
|
||||
infer_backend=payload.get("infer_backend", "huggingface"),
|
||||
infer_dtype=payload.get("infer_dtype", "auto"),
|
||||
)
|
||||
return result
|
||||
|
||||
@app.post(f"{route_prefix}/inference/unload")
|
||||
async def inference_unload() -> dict[str, Any]:
|
||||
"""Unload the currently loaded model and free GPU memory."""
|
||||
# Teardown (gc.collect + cuda.empty_cache) can take a while; run it off
|
||||
# the event loop so /health and /inference/status stay responsive.
|
||||
return await asyncio.to_thread(get_inference_session().unload)
|
||||
|
||||
@app.get(f"{route_prefix}/inference/status")
|
||||
async def inference_status() -> dict[str, Any]:
|
||||
"""Get the current inference session status."""
|
||||
return get_inference_session().info()
|
||||
|
||||
@app.post(f"{route_prefix}/inference/chat")
|
||||
async def inference_chat(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Chat with the loaded model (non-streaming).
|
||||
|
||||
Expected payload:
|
||||
messages: list[dict] (OpenAI format)
|
||||
temperature: float (default 0.95)
|
||||
top_p: float (default 0.7)
|
||||
max_new_tokens: int (default 1024)
|
||||
"""
|
||||
messages = payload.get("messages") or []
|
||||
if not messages:
|
||||
raise HTTPException(status_code=400, detail="messages is required")
|
||||
# Generation is long-running; run it in a thread so the event loop keeps
|
||||
# serving /inference/status and /health during inference.
|
||||
result = await asyncio.to_thread(
|
||||
get_inference_session().chat,
|
||||
messages=messages,
|
||||
temperature=float(payload.get("temperature", 0.95)),
|
||||
top_p=float(payload.get("top_p", 0.7)),
|
||||
max_new_tokens=int(payload.get("max_new_tokens", 1024)),
|
||||
do_sample=bool(payload.get("do_sample", True)),
|
||||
)
|
||||
if result.get("error"):
|
||||
raise HTTPException(status_code=500, detail=result["error"])
|
||||
return {"response": result["response"]}
|
||||
|
||||
@app.post(f"{route_prefix}/inference/chat/stream")
|
||||
async def inference_chat_stream(payload: dict[str, Any]) -> StreamingResponse:
|
||||
"""Chat with streaming response (Server-Sent Events)."""
|
||||
messages = payload.get("messages") or []
|
||||
if not messages:
|
||||
raise HTTPException(status_code=400, detail="messages is required")
|
||||
|
||||
def generate():
|
||||
session = get_inference_session()
|
||||
for chunk in session.chat_stream(
|
||||
messages=messages,
|
||||
temperature=float(payload.get("temperature", 0.95)),
|
||||
top_p=float(payload.get("top_p", 0.7)),
|
||||
max_new_tokens=int(payload.get("max_new_tokens", 1024)),
|
||||
do_sample=bool(payload.get("do_sample", True)),
|
||||
):
|
||||
yield chunk
|
||||
|
||||
return StreamingResponse(generate(), media_type="text/event-stream")
|
||||
|
||||
@app.post(f"{route_prefix}/compute/files/upload")
|
||||
async def upload_file(
|
||||
file: UploadFile | None = File(default=None),
|
||||
target_relative_path: str | None = Form(default=None),
|
||||
resource_type: str | None = Form(default=None),
|
||||
resource_id: str | None = Form(default=None),
|
||||
) -> dict[str, Any]:
|
||||
file_id = f"file_{int(now() * 1000)}"
|
||||
data_root = Path(os.getenv("YG_FT_DATA_ROOT", "/data/yg-ft"))
|
||||
data_root.mkdir(parents=True, exist_ok=True)
|
||||
filename = Path(file.filename if file else file_id).name
|
||||
if target_relative_path:
|
||||
target = (data_root / target_relative_path.lstrip("/\\")).resolve()
|
||||
if not _path_inside(data_root, target):
|
||||
raise HTTPException(status_code=400, detail="target path must stay inside YG_FT_DATA_ROOT")
|
||||
else:
|
||||
target = data_root / "uploads" / f"{file_id}_{filename}"
|
||||
if file:
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
with target.open("wb") as output:
|
||||
while chunk := await file.read(1024 * 1024):
|
||||
output.write(chunk)
|
||||
else:
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.write_text("", encoding="utf-8")
|
||||
return {
|
||||
"id": file_id,
|
||||
"resource_type": resource_type,
|
||||
"resource_id": resource_id,
|
||||
"status": "available",
|
||||
"local_path": str(target),
|
||||
"byte_size": target.stat().st_size,
|
||||
"checksum_sha256": hashlib.sha256(target.read_bytes()).hexdigest() if target.is_file() else "",
|
||||
}
|
||||
|
||||
@app.post(f"{route_prefix}/compute/files/import-local")
|
||||
async def import_local_file(payload: dict[str, Any]) -> dict[str, Any]:
|
||||
source = Path(str(payload.get("source_path") or ""))
|
||||
if not source.exists():
|
||||
raise HTTPException(status_code=404, detail="source path not found")
|
||||
data_root = Path(os.getenv("YG_FT_DATA_ROOT", "/data/yg-ft"))
|
||||
data_root.mkdir(parents=True, exist_ok=True)
|
||||
relative = str(payload.get("target_relative_path") or f"imports/{source.name}").lstrip("/\\")
|
||||
target = (data_root / relative).resolve()
|
||||
if not _path_inside(data_root, target):
|
||||
raise HTTPException(status_code=400, detail="target path must stay inside YG_FT_DATA_ROOT")
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
if source.is_dir():
|
||||
if target.exists():
|
||||
shutil.rmtree(target)
|
||||
shutil.copytree(source, target)
|
||||
byte_size = sum(path.stat().st_size for path in target.rglob("*") if path.is_file())
|
||||
checksum = ""
|
||||
else:
|
||||
shutil.copy2(source, target)
|
||||
byte_size = target.stat().st_size
|
||||
checksum = hashlib.sha256(target.read_bytes()).hexdigest()
|
||||
return {
|
||||
"id": str(payload.get("id") or f"file_{int(now() * 1000)}"),
|
||||
"resource_type": payload.get("resource_type"),
|
||||
"resource_id": payload.get("resource_id"),
|
||||
"status": "available",
|
||||
"local_path": str(target),
|
||||
"byte_size": byte_size,
|
||||
"checksum_sha256": checksum,
|
||||
}
|
||||
|
||||
@app.get(f"{route_prefix}/compute/files/read")
|
||||
async def read_file(path: str = Query(...)) -> JSONResponse:
|
||||
"""Read a text file from within YG_FT_DATA_ROOT. Used by the backend
|
||||
to fetch eval results and other job outputs."""
|
||||
data_root = Path(os.getenv("YG_FT_DATA_ROOT", "/data/yg-ft"))
|
||||
target = (data_root / path.lstrip("/\\")).resolve()
|
||||
if not _path_inside(data_root, target):
|
||||
raise HTTPException(status_code=400, detail="path must stay inside YG_FT_DATA_ROOT")
|
||||
if not target.is_file():
|
||||
raise HTTPException(status_code=404, detail="file not found")
|
||||
try:
|
||||
content = target.read_text(encoding="utf-8")
|
||||
return JSONResponse(json.loads(content) if content.strip().startswith("{") else {"content": content})
|
||||
except Exception as exc:
|
||||
raise HTTPException(status_code=500, detail=str(exc))
|
||||
|
||||
@app.get(f"{route_prefix}/compute/files/{{file_id}}/download")
|
||||
async def download_file(file_id: str) -> FileResponse:
|
||||
upload_root = Path(os.getenv("YG_FT_DATA_ROOT", "/data/yg-ft")) / "uploads"
|
||||
matches = list(upload_root.glob(f"{file_id}_*"))
|
||||
if not matches:
|
||||
raise HTTPException(status_code=404, detail="file not found")
|
||||
return FileResponse(matches[0])
|
||||
|
||||
return app
|
||||
|
||||
|
||||
app = create_app()
|
||||
@@ -1,324 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class LlamaFactoryCommand:
|
||||
command: list[str]
|
||||
work_dir: str
|
||||
env: dict[str, str]
|
||||
|
||||
|
||||
def _load_dataset_preview(path: Path) -> list[dict[str, Any]]:
|
||||
if not path.exists():
|
||||
return []
|
||||
text = path.read_text(encoding="utf-8", errors="replace").strip()
|
||||
if not text:
|
||||
return []
|
||||
if path.suffix.lower() == ".jsonl":
|
||||
items: list[dict[str, Any]] = []
|
||||
for line in text.splitlines()[:20]:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
value = json.loads(line)
|
||||
if isinstance(value, dict):
|
||||
items.append(value)
|
||||
return items
|
||||
value = json.loads(text)
|
||||
if isinstance(value, list):
|
||||
return [item for item in value[:20] if isinstance(item, dict)]
|
||||
if isinstance(value, dict):
|
||||
return [value]
|
||||
return []
|
||||
|
||||
|
||||
def _validate_dataset_columns(config: dict[str, Any]) -> list[str]:
|
||||
dataset_dir = config.get("dataset_dir")
|
||||
dataset_info = config.get("dataset_info")
|
||||
if not dataset_dir or not isinstance(dataset_info, dict):
|
||||
return []
|
||||
root = Path(str(dataset_dir))
|
||||
errors: list[str] = []
|
||||
for dataset_key, item in dataset_info.items():
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
file_name = item.get("file_name")
|
||||
file_names = file_name if isinstance(file_name, list) else [file_name]
|
||||
columns = item.get("columns") if isinstance(item.get("columns"), dict) else {}
|
||||
required_columns = [str(value) for value in columns.values() if value]
|
||||
for name in file_names:
|
||||
if not name:
|
||||
continue
|
||||
path = root / str(name).lstrip("/\\")
|
||||
if not path.exists():
|
||||
continue
|
||||
try:
|
||||
preview_rows = _load_dataset_preview(path)
|
||||
except Exception as exc: # noqa: BLE001 - expose malformed data as validation error
|
||||
errors.append(f"dataset file parse failed: {path}: {exc}")
|
||||
continue
|
||||
if not preview_rows:
|
||||
errors.append(f"dataset file has no valid object records: {path}")
|
||||
continue
|
||||
available = set().union(*(row.keys() for row in preview_rows))
|
||||
missing = [column for column in required_columns if column not in available]
|
||||
if missing:
|
||||
errors.append(
|
||||
f"dataset columns missing in {path.name} for {dataset_key}: {', '.join(sorted(set(missing)))}"
|
||||
)
|
||||
return errors
|
||||
|
||||
|
||||
def validate_config(config: dict[str, Any]) -> list[str]:
|
||||
errors: list[str] = []
|
||||
if not config.get("base_model") and not config.get("model_name_or_path"):
|
||||
errors.append("base_model or model_name_or_path is required")
|
||||
if not config.get("dataset") and not config.get("dataset_dir"):
|
||||
errors.append("dataset or dataset_dir is required")
|
||||
try:
|
||||
learning_rate = float(config.get("learning_rate", 0.0002))
|
||||
except (TypeError, ValueError):
|
||||
learning_rate = 0
|
||||
if learning_rate <= 0:
|
||||
errors.append("learning_rate must be greater than zero")
|
||||
try:
|
||||
epochs = int(config.get("n_epochs", config.get("num_train_epochs", 1)))
|
||||
except (TypeError, ValueError):
|
||||
epochs = 0
|
||||
if epochs <= 0:
|
||||
errors.append("n_epochs must be greater than zero")
|
||||
dataset_dir = config.get("dataset_dir")
|
||||
dataset_info = config.get("dataset_info")
|
||||
if config.get("require_dataset_files") and dataset_dir and isinstance(dataset_info, dict):
|
||||
root = Path(str(dataset_dir))
|
||||
for dataset_key, item in dataset_info.items():
|
||||
if not isinstance(item, dict):
|
||||
errors.append(f"dataset_info entry must be object: {dataset_key}")
|
||||
continue
|
||||
file_name = item.get("file_name")
|
||||
file_names = file_name if isinstance(file_name, list) else [file_name]
|
||||
for name in file_names:
|
||||
if not name:
|
||||
errors.append(f"dataset_info file_name is required: {dataset_key}")
|
||||
continue
|
||||
path = root / str(name).lstrip("/\\")
|
||||
if not path.exists():
|
||||
errors.append(f"dataset file not found: {path}")
|
||||
errors.extend(_validate_dataset_columns(config))
|
||||
return errors
|
||||
|
||||
|
||||
def _optional_arg(config: dict[str, Any], command: list[str], option: str, *keys: str) -> None:
|
||||
for key in keys:
|
||||
value = config.get(key)
|
||||
if value is not None and value != "":
|
||||
command.extend([option, str(value)])
|
||||
return
|
||||
|
||||
|
||||
def _optional_bool_arg(config: dict[str, Any], command: list[str], option: str, *keys: str) -> None:
|
||||
for key in keys:
|
||||
value = config.get(key)
|
||||
if value is True or str(value).lower() == "true":
|
||||
command.extend([option, "true"])
|
||||
return
|
||||
|
||||
|
||||
def _normalize_stage(config: dict[str, Any]) -> str:
|
||||
raw = str(config.get("stage") or config.get("train_type") or "sft").strip().lower()
|
||||
return {
|
||||
"sft": "sft",
|
||||
"dpo": "dpo",
|
||||
"cpt": "pt",
|
||||
"pt": "pt",
|
||||
"pretrain": "pt",
|
||||
"rm": "rm",
|
||||
"ppo": "ppo",
|
||||
"kto": "kto",
|
||||
}.get(raw, raw or "sft")
|
||||
|
||||
|
||||
def prepare_runtime_files(config: dict[str, Any]) -> list[dict[str, str]]:
|
||||
dataset_dir = config.get("dataset_dir")
|
||||
dataset_info = config.get("dataset_info")
|
||||
if not dataset_dir or not isinstance(dataset_info, dict):
|
||||
return []
|
||||
root = Path(str(dataset_dir))
|
||||
root.mkdir(parents=True, exist_ok=True)
|
||||
path = root / "dataset_info.json"
|
||||
existing: dict[str, Any] = {}
|
||||
if path.exists():
|
||||
try:
|
||||
loaded = json.loads(path.read_text(encoding="utf-8"))
|
||||
existing = loaded if isinstance(loaded, dict) else {}
|
||||
except json.JSONDecodeError:
|
||||
existing = {}
|
||||
existing.update(dataset_info)
|
||||
path.write_text(json.dumps(existing, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
return [{"name": "dataset_info", "path": str(path)}]
|
||||
|
||||
|
||||
def build_command(config: dict[str, Any], llama_factory_home: str = "/app/LLaMA-Factory") -> LlamaFactoryCommand:
|
||||
engine = str(config.get("engine") or config.get("training_engine") or "llama_factory")
|
||||
if engine in {"merge", "export", "llama_factory_export"}:
|
||||
model_path = config.get("base_model") or config.get("model_name_or_path") or config.get("base_model_path")
|
||||
adapter_path = config.get("adapter_name_or_path") or config.get("adapter_path") or config.get("lora_path")
|
||||
output_dir = config.get("output_dir") or config.get("export_dir")
|
||||
errors: list[str] = []
|
||||
if not model_path:
|
||||
errors.append("base_model or model_name_or_path is required")
|
||||
if not adapter_path and engine == "merge":
|
||||
errors.append("adapter_name_or_path or adapter_path is required")
|
||||
if not output_dir:
|
||||
errors.append("output_dir or export_dir is required")
|
||||
if errors:
|
||||
raise ValueError("; ".join(errors))
|
||||
command = [
|
||||
"llamafactory-cli",
|
||||
"export",
|
||||
"--model_name_or_path",
|
||||
str(model_path),
|
||||
"--template",
|
||||
str(config.get("template", "qwen")),
|
||||
"--finetuning_type",
|
||||
str(config.get("train_method", config.get("finetuning_type", "lora"))),
|
||||
"--export_dir",
|
||||
str(output_dir),
|
||||
"--export_size",
|
||||
str(config.get("export_size", 2)),
|
||||
"--export_device",
|
||||
str(config.get("export_device", "cpu")),
|
||||
"--export_legacy_format",
|
||||
str(config.get("export_legacy_format", False)).lower(),
|
||||
]
|
||||
if adapter_path:
|
||||
command.extend(["--adapter_name_or_path", str(adapter_path)])
|
||||
quantization_bit = int(config.get("export_quantization_bit", config.get("quantization_bit", 0)) or 0)
|
||||
if quantization_bit in {4, 8}:
|
||||
command.extend(["--quantization_bit", str(quantization_bit)])
|
||||
return LlamaFactoryCommand(command=command, work_dir=str(Path(llama_factory_home)), env={})
|
||||
|
||||
if engine == "eval":
|
||||
output_dir = config.get("output_dir") or f"/data/yg-ft/outputs/{config.get('name', 'eval-job')}"
|
||||
eval_config_path = str(Path(output_dir) / "eval_config.json")
|
||||
eval_config = {
|
||||
"model_name_or_path": config.get("model_name_or_path", ""),
|
||||
"adapter_name_or_path": config.get("adapter_name_or_path", ""),
|
||||
"template": config.get("template", "qwen"),
|
||||
"dataset_path": config.get("dataset_path", ""),
|
||||
"output_dir": output_dir,
|
||||
"basic_metrics": config.get("basic_metrics", {}),
|
||||
"dimension": config.get("dimension", {}),
|
||||
"temperature": config.get("temperature", 0.1),
|
||||
"top_p": config.get("top_p", 0.95),
|
||||
"max_new_tokens": config.get("max_new_tokens", 512),
|
||||
"infer_backend": config.get("infer_backend", "huggingface"),
|
||||
"infer_dtype": config.get("infer_dtype", "auto"),
|
||||
}
|
||||
Path(output_dir).mkdir(parents=True, exist_ok=True)
|
||||
Path(eval_config_path).write_text(json.dumps(eval_config, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
return LlamaFactoryCommand(
|
||||
command=["python", "-u", "-m", "compute.engines.llama_factory.eval_runner", "--config", eval_config_path],
|
||||
work_dir="/app",
|
||||
env={},
|
||||
)
|
||||
|
||||
errors = validate_config(config)
|
||||
if errors:
|
||||
raise ValueError("; ".join(errors))
|
||||
|
||||
if engine == "smoke":
|
||||
output_dir = config.get("output_dir") or f"/data/yg-ft/outputs/{config.get('name', 'training-smoke')}"
|
||||
script = (
|
||||
"import json, os, time; "
|
||||
f"out={str(output_dir)!r}; "
|
||||
"os.makedirs(out, exist_ok=True); "
|
||||
"print('[INFO] smoke training started', flush=True); "
|
||||
"\nfor step in range(1, 7):\n"
|
||||
" loss=round(1.8/(step+1), 4)\n"
|
||||
" lr=round(0.0002*(1-step/10), 8)\n"
|
||||
" print({'loss': loss, 'grad_norm': round(0.4 + step*0.03, 4), 'learning_rate': lr, 'epoch': round(step/6, 4)}, flush=True)\n"
|
||||
" time.sleep(0.4)\n"
|
||||
"\nopen(os.path.join(out, 'adapter_config.json'), 'w', encoding='utf-8').write(json.dumps({'engine':'smoke','status':'completed'})); "
|
||||
"print('***** train metrics *****', flush=True); "
|
||||
"print('train_loss = 0.12', flush=True); "
|
||||
"print('***** train metrics end *****', flush=True)"
|
||||
)
|
||||
return LlamaFactoryCommand(command=["python", "-u", "-c", script], work_dir="/app", env={})
|
||||
|
||||
model_path = config.get("base_model") or config.get("model_name_or_path")
|
||||
dataset = config.get("dataset") or config.get("dataset_name")
|
||||
dataset_dir = config.get("dataset_dir")
|
||||
output_dir = config.get("output_dir") or f"/data/yg-ft/outputs/{config.get('name', 'training-job')}"
|
||||
command = [
|
||||
"llamafactory-cli",
|
||||
"train",
|
||||
"--stage",
|
||||
_normalize_stage(config),
|
||||
"--do_train",
|
||||
"true",
|
||||
"--model_name_or_path",
|
||||
str(model_path),
|
||||
"--dataset",
|
||||
str(dataset or "default"),
|
||||
"--template",
|
||||
str(config.get("template", "qwen")),
|
||||
"--finetuning_type",
|
||||
str(config.get("train_method", config.get("finetuning_type", "lora"))),
|
||||
"--output_dir",
|
||||
str(output_dir),
|
||||
"--per_device_train_batch_size",
|
||||
str(config.get("batch_size", 2)),
|
||||
"--learning_rate",
|
||||
str(config.get("learning_rate", 0.0002)),
|
||||
"--num_train_epochs",
|
||||
str(config.get("n_epochs", 3)),
|
||||
"--save_steps",
|
||||
str(config.get("save_steps", 50)),
|
||||
"--logging_steps",
|
||||
str(config.get("logging_steps", 10)),
|
||||
"--overwrite_output_dir",
|
||||
"true",
|
||||
"--plot_loss",
|
||||
"true",
|
||||
]
|
||||
if dataset_dir:
|
||||
command.extend(["--dataset_dir", str(dataset_dir)])
|
||||
eval_dataset = config.get("eval_dataset")
|
||||
if eval_dataset:
|
||||
command.extend(["--eval_dataset", str(eval_dataset), "--do_eval", "true"])
|
||||
_optional_arg(config, command, "--cutoff_len", "max_length", "cutoff_len")
|
||||
_optional_arg(config, command, "--lr_scheduler_type", "lr_scheduler_type")
|
||||
_optional_arg(config, command, "--warmup_ratio", "warmup_ratio")
|
||||
_optional_arg(config, command, "--weight_decay", "weight_decay")
|
||||
_optional_arg(config, command, "--lora_rank", "lora_rank", "rank")
|
||||
_optional_arg(config, command, "--lora_alpha", "lora_alpha")
|
||||
_optional_arg(config, command, "--lora_dropout", "lora_dropout")
|
||||
_optional_arg(config, command, "--gradient_accumulation_steps", "gradient_accumulation_steps")
|
||||
if not eval_dataset:
|
||||
_optional_arg(config, command, "--val_size", "val_size")
|
||||
_optional_arg(config, command, "--max_samples", "max_samples")
|
||||
_optional_arg(config, command, "--preprocessing_num_workers", "preprocessing_num_workers")
|
||||
_optional_bool_arg(config, command, "--fp16", "fp16")
|
||||
_optional_bool_arg(config, command, "--bf16", "bf16")
|
||||
quantization_bit = int(config.get("quantization_bit", 0) or 0)
|
||||
if quantization_bit in {4, 8}:
|
||||
command.extend(["--quantization_bit", str(quantization_bit)])
|
||||
return LlamaFactoryCommand(command=command, work_dir=str(Path(llama_factory_home)), env={})
|
||||
|
||||
|
||||
def parse_log_line(line: str) -> dict[str, float] | None:
|
||||
if "loss" not in line or "learning_rate" not in line:
|
||||
return None
|
||||
result: dict[str, float] = {}
|
||||
for key in ["loss", "grad_norm", "learning_rate", "epoch"]:
|
||||
match = re.search(rf"['\"]?{key}['\"]?\s*:\s*([-+]?\d+(?:\.\d+)?(?:[eE][-+]?\d+)?)", line)
|
||||
if match:
|
||||
result[key] = float(match.group(1))
|
||||
return result or None
|
||||
@@ -1,485 +0,0 @@
|
||||
"""
|
||||
Evaluation runner — executes model evaluation as a subprocess job.
|
||||
|
||||
Usage:
|
||||
python -m compute.engines.llama_factory.eval_runner --config <config_json_path>
|
||||
|
||||
The config JSON is written by the compute API before spawning this subprocess.
|
||||
Results are written to ``output_dir/eval_results.json`` and progress is printed
|
||||
to stdout (captured as job logs).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import math
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
from difflib import SequenceMatcher
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
def _load_dataset(path: str) -> list[dict[str, Any]]:
|
||||
"""Load a JSON or JSONL dataset file.
|
||||
|
||||
Supports common field names used across the platform:
|
||||
* ``instruction`` + ``input`` + ``output`` (Alpaca-style)
|
||||
* ``question`` + ``answer``
|
||||
* ``messages`` (ShareGPT-style – the last assistant message is treated as reference)
|
||||
"""
|
||||
file_path = Path(path)
|
||||
text = file_path.read_text(encoding="utf-8", errors="replace").strip()
|
||||
if not text:
|
||||
return []
|
||||
if file_path.suffix.lower() == ".json":
|
||||
value = json.loads(text)
|
||||
if isinstance(value, list):
|
||||
return [item for item in value if isinstance(item, dict)]
|
||||
return [value] if isinstance(value, dict) else []
|
||||
|
||||
samples: list[dict[str, Any]] = []
|
||||
for line in text.splitlines():
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
obj = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
if isinstance(obj, dict):
|
||||
samples.append(obj)
|
||||
return samples
|
||||
|
||||
|
||||
def _sample_question(sample: dict[str, Any]) -> str:
|
||||
"""Extract the user-facing question / instruction from a sample."""
|
||||
if sample.get("instruction"):
|
||||
text = sample["instruction"]
|
||||
if sample.get("input"):
|
||||
text += "\n" + sample["input"]
|
||||
return text
|
||||
if sample.get("question"):
|
||||
return sample["question"]
|
||||
# ShareGPT-style: use the last user message as question
|
||||
messages = sample.get("messages") or []
|
||||
user_msgs = [m["content"] for m in messages if m.get("role") == "user"]
|
||||
return user_msgs[-1] if user_msgs else ""
|
||||
|
||||
|
||||
def _sample_reference(sample: dict[str, Any]) -> str:
|
||||
"""Extract the reference answer from a sample."""
|
||||
if sample.get("output"):
|
||||
return sample["output"]
|
||||
if sample.get("answer"):
|
||||
return sample["answer"]
|
||||
messages = sample.get("messages") or []
|
||||
assistant_msgs = [m["content"] for m in messages if m.get("role") == "assistant"]
|
||||
return assistant_msgs[-1] if assistant_msgs else ""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Basic metrics
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _compute_bleu(references: list[str], predictions: list[str], ngram: int = 4) -> dict[str, Any]:
|
||||
"""Compute BLEU score via sacrebleu (corpus-level)."""
|
||||
try:
|
||||
from sacrebleu.metrics import BLEU
|
||||
except ImportError:
|
||||
return {"enabled": False, "error": "sacrebleu not installed", "score": 0}
|
||||
bleu = BLEU(max_ngram_order=ngram)
|
||||
# sacrebleu expects list-of-strings; we have one reference per prediction
|
||||
score = bleu.corpus_score(predictions, [references])
|
||||
return {
|
||||
"enabled": True,
|
||||
"score": round(score.score, 2),
|
||||
"bleu": round(score.score, 2),
|
||||
}
|
||||
|
||||
|
||||
def _compute_rouge(references: list[str], predictions: list[str], methods: list[str] | None = None) -> dict[str, Any]:
|
||||
"""Compute ROUGE scores via rouge-score."""
|
||||
try:
|
||||
from rouge_score import rouge_scorer
|
||||
except ImportError:
|
||||
return {"enabled": False, "error": "rouge-score not installed", "score": 0}
|
||||
methods = methods or ["rouge1", "rouge2", "rougeL"]
|
||||
# Normalize: map "rouge_1"/"rouge1" → "rouge1", "rouge_l"/"rougeL" → "rougeL"
|
||||
_rouge_aliases = {"rouge_1": "rouge1", "rouge_2": "rouge2", "rouge_l": "rougeL"}
|
||||
methods = [_rouge_aliases.get(m, m.replace("_", "")) for m in methods]
|
||||
scorer = rouge_scorer.RougeScorer(methods, use_stemmer=True)
|
||||
totals: dict[str, float] = {}
|
||||
n = max(len(predictions), 1)
|
||||
for ref, pred in zip(references, predictions):
|
||||
result = scorer.score(ref, pred)
|
||||
for key in methods:
|
||||
totals[key] = totals.get(key, 0) + result[key].fmeasure
|
||||
avg = {k: round(v / n, 4) for k, v in totals.items()}
|
||||
return {"enabled": True, "score": round(avg.get("rougeL", avg.get("rouge1", 0)) * 100, 2), **avg}
|
||||
|
||||
|
||||
def _compute_cosine(references: list[str], predictions: list[str]) -> dict[str, Any]:
|
||||
"""Compute average cosine similarity via sklearn."""
|
||||
try:
|
||||
from sklearn.feature_extraction.text import TfidfVectorizer
|
||||
from sklearn.metrics.pairwise import cosine_similarity
|
||||
except ImportError:
|
||||
return {"enabled": False, "error": "scikit-learn not installed", "score": 0}
|
||||
try:
|
||||
vectorizer = TfidfVectorizer()
|
||||
tfidf = vectorizer.fit_transform(references + predictions)
|
||||
n = len(references)
|
||||
ref_vec = tfidf[:n]
|
||||
pred_vec = tfidf[n:]
|
||||
sims = cosine_similarity(ref_vec, pred_vec).diagonal()
|
||||
return {"enabled": True, "score": round(float(sims.mean()) * 100, 2)}
|
||||
except ValueError:
|
||||
return {"enabled": True, "score": 0, "error": "insufficient text for vectorization"}
|
||||
|
||||
|
||||
def _normalize_text(value: str) -> str:
|
||||
return re.sub(r"\s+", " ", str(value or "").strip().lower())
|
||||
|
||||
|
||||
def _compute_exact_match(references: list[str], predictions: list[str]) -> dict[str, Any]:
|
||||
total = len(predictions)
|
||||
if not total:
|
||||
return {"enabled": True, "score": 0, "matched": 0, "total": 0}
|
||||
matched = sum(
|
||||
1
|
||||
for ref, pred in zip(references, predictions)
|
||||
if _normalize_text(ref) == _normalize_text(pred)
|
||||
)
|
||||
return {"enabled": True, "score": round(matched / total * 100, 2), "matched": matched, "total": total}
|
||||
|
||||
|
||||
def _compute_text_similarity(references: list[str], predictions: list[str]) -> dict[str, Any]:
|
||||
if not predictions:
|
||||
return {"enabled": True, "score": 0}
|
||||
scores = [
|
||||
SequenceMatcher(None, _normalize_text(ref), _normalize_text(pred)).ratio()
|
||||
for ref, pred in zip(references, predictions)
|
||||
]
|
||||
return {"enabled": True, "score": round(sum(scores) / max(len(scores), 1) * 100, 2)}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# LLM Judge
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _judge_sample(
|
||||
question: str,
|
||||
reference: str,
|
||||
prediction: str,
|
||||
config: dict[str, Any],
|
||||
) -> dict[str, Any]:
|
||||
"""Call an OpenAI-compatible LLM to judge a single sample.
|
||||
|
||||
Returns a dict with keys:
|
||||
score, max_score, passed, judgement, evaluation_reason, error_type
|
||||
"""
|
||||
api_url = (config.get("api_url") or "").strip().rstrip("/")
|
||||
api_key = (config.get("api_key") or "").strip()
|
||||
eval_model = (config.get("eval_model") or "").strip()
|
||||
# 优先使用模型记录里配置的真实 API 模型名（如 deepseek-chat），
|
||||
# 否则回退到平台内部模型名
|
||||
api_model = (config.get("api_model") or "").strip() or eval_model
|
||||
eval_prompt = (config.get("eval_prompt") or "").strip()
|
||||
score_min = float(config.get("score_min", 0))
|
||||
score_max = float(config.get("score_max", 5))
|
||||
pass_threshold = float(config.get("pass_threshold", 3))
|
||||
|
||||
if not api_url or not eval_model:
|
||||
return {"score": 0, "max_score": score_max, "passed": False, "judgement": "未配置",
|
||||
"evaluation_reason": "未配置评测模型", "error_type": "其他"}
|
||||
|
||||
system_msg = (
|
||||
eval_prompt
|
||||
or "你是一个专业的评测专家。请根据参考答-案对被测模型的输出进行评分。"
|
||||
)
|
||||
user_msg = (
|
||||
f"## 问题\n{question}\n\n"
|
||||
f"## 参考答案\n{reference}\n\n"
|
||||
f"## 模型输出\n{prediction}\n\n"
|
||||
f"请给出 {score_min}-{score_max} 分的评分，并说明理由。"
|
||||
)
|
||||
|
||||
try:
|
||||
import urllib.request
|
||||
import urllib.error
|
||||
|
||||
body = json.dumps({
|
||||
"model": api_model,
|
||||
"messages": [
|
||||
{"role": "system", "content": system_msg},
|
||||
{"role": "user", "content": user_msg},
|
||||
],
|
||||
"temperature": 0.3,
|
||||
"max_tokens": 512,
|
||||
}).encode("utf-8")
|
||||
|
||||
req = urllib.request.Request(
|
||||
f"{api_url}/v1/chat/completions",
|
||||
data=body,
|
||||
headers={
|
||||
"Content-Type": "application/json",
|
||||
"Authorization": f"Bearer {api_key}",
|
||||
},
|
||||
)
|
||||
resp = urllib.request.urlopen(req, timeout=120)
|
||||
data = json.loads(resp.read().decode("utf-8"))
|
||||
reply = data["choices"][0]["message"]["content"]
|
||||
except Exception as exc:
|
||||
return {"score": 0, "max_score": score_max, "passed": False,
|
||||
"judgement": "错误", "evaluation_reason": f"评测模型调用失败: {exc}",
|
||||
"error_type": "其他"}
|
||||
|
||||
# Parse score from reply — look for patterns like "4分" or "Score: 4"
|
||||
score = 0
|
||||
import re
|
||||
score_patterns = [
|
||||
r'(?:得分|分数|评分|score)[^\d]*(\d+(?:\.\d+)?)',
|
||||
r'(\d+(?:\.\d+)?)\s*分',
|
||||
r'(\d+(?:\.\d+)?)\s*/\s*\d+',
|
||||
]
|
||||
for pat in score_patterns:
|
||||
m = re.search(pat, reply, re.IGNORECASE)
|
||||
if m:
|
||||
try:
|
||||
score = float(m.group(1))
|
||||
except ValueError:
|
||||
continue
|
||||
break
|
||||
score = max(score_min, min(score_max, score))
|
||||
passed = score >= pass_threshold
|
||||
|
||||
# Determine judgement label
|
||||
if score >= pass_threshold + 1:
|
||||
judgement = "正确"
|
||||
elif score >= pass_threshold:
|
||||
judgement = "部分正确"
|
||||
else:
|
||||
judgement = "错误"
|
||||
|
||||
# Guess error type from reply
|
||||
reply_lower = reply.lower()
|
||||
if any(w in reply_lower for w in ["幻觉", "hallucination", "编造"]):
|
||||
error_type = "幻觉"
|
||||
elif any(w in reply_lower for w in ["不完整", "incomplete", "遗漏"]):
|
||||
error_type = "不完整"
|
||||
elif any(w in reply_lower for w in ["格式", "format"]):
|
||||
error_type = "格式偏差"
|
||||
elif any(w in reply_lower for w in ["混淆", "confusion", "错误"]):
|
||||
error_type = "混淆"
|
||||
else:
|
||||
error_type = "其他"
|
||||
|
||||
return {
|
||||
"score": score,
|
||||
"max_score": score_max,
|
||||
"passed": passed,
|
||||
"judgement": judgement,
|
||||
"evaluation_reason": reply[:2000],
|
||||
"error_type": error_type,
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Main entry point
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def run_eval(config: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Execute a full evaluation run. Returns the result dict (also written to file)."""
|
||||
model_path = config["model_name_or_path"]
|
||||
adapter_path = config.get("adapter_name_or_path", "")
|
||||
template = config.get("template", "qwen")
|
||||
dataset_path = config["dataset_path"]
|
||||
output_dir = Path(config["output_dir"])
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
basic_cfg = config.get("basic_metrics", {})
|
||||
dimension_cfg = config.get("dimension", {}) or {}
|
||||
output_precision = int(basic_cfg.get("output_precision", 2))
|
||||
|
||||
# ---- 1. Load dataset ----
|
||||
print(f"[eval] loading dataset: {dataset_path}")
|
||||
raw_samples = _load_dataset(dataset_path)
|
||||
print(f"[eval] loaded {len(raw_samples)} samples")
|
||||
|
||||
# ---- 2. Load model ----
|
||||
print(f"[eval] loading model: {model_path}")
|
||||
from compute.engines.llama_factory.inference import InferenceSession
|
||||
session = InferenceSession()
|
||||
session.load(
|
||||
model_name_or_path=model_path,
|
||||
adapter_name_or_path=adapter_path,
|
||||
template=template,
|
||||
infer_backend=config.get("infer_backend", "huggingface"),
|
||||
infer_dtype=config.get("infer_dtype", "auto"),
|
||||
)
|
||||
# load() 为异步加载（立即返回 loading），必须等待后台线程完成后再进行推理
|
||||
load_result = session.wait_until_loaded(timeout=float(config.get("load_timeout", 1800)))
|
||||
if not load_result.get("loaded"):
|
||||
raise RuntimeError(f"model load failed: {load_result.get('error', 'unknown')}")
|
||||
print(f"[eval] model loaded OK")
|
||||
|
||||
# ---- 3. Run inference on each sample ----
|
||||
samples: list[dict[str, Any]] = []
|
||||
predictions: list[str] = []
|
||||
references: list[str] = []
|
||||
questions: list[str] = []
|
||||
|
||||
total = len(raw_samples)
|
||||
judge_enabled = bool(dimension_cfg.get("eval_model") and dimension_cfg.get("api_url"))
|
||||
print(f"[eval] starting inference on {total} samples, judge={'enabled' if judge_enabled else 'disabled'}")
|
||||
|
||||
for idx, raw in enumerate(raw_samples, start=1):
|
||||
question = _sample_question(raw)
|
||||
reference = _sample_reference(raw)
|
||||
if not question:
|
||||
print(f"[eval] sample {idx}/{total}: skipped (no question)")
|
||||
continue
|
||||
|
||||
# Inference
|
||||
chat_msgs = [{"role": "user", "content": question}]
|
||||
result = session.chat(
|
||||
chat_msgs,
|
||||
temperature=float(config.get("temperature", 0.1)),
|
||||
top_p=float(config.get("top_p", 0.95)),
|
||||
max_new_tokens=int(config.get("max_new_tokens", 512)),
|
||||
do_sample=False,
|
||||
)
|
||||
prediction = result.get("response", "") if not result.get("error") else f"[ERROR] {result['error']}"
|
||||
|
||||
predictions.append(prediction)
|
||||
references.append(reference)
|
||||
questions.append(question)
|
||||
|
||||
# LLM Judge
|
||||
judge_result: dict[str, Any] = {}
|
||||
if judge_enabled:
|
||||
judge_result = _judge_sample(question, reference, prediction, dimension_cfg)
|
||||
|
||||
samples.append({
|
||||
"index": idx,
|
||||
"input": question,
|
||||
"reference_answer": reference,
|
||||
"model_output": prediction,
|
||||
"score": judge_result.get("score"),
|
||||
"max_score": judge_result.get("max_score", dimension_cfg.get("score_max", 5)),
|
||||
"passed": judge_result.get("passed"),
|
||||
"judgement": judge_result.get("judgement"),
|
||||
"evaluation_reason": judge_result.get("evaluation_reason", ""),
|
||||
"error_type": judge_result.get("error_type"),
|
||||
"dimension_scores": [
|
||||
{"name": "judge_score", "score": judge_result.get("score", 0),
|
||||
"max_score": judge_result.get("max_score", dimension_cfg.get("score_max", 5))},
|
||||
] if judge_result else [],
|
||||
"status": "completed",
|
||||
})
|
||||
|
||||
progress_pct = int(idx / max(total, 1) * 100)
|
||||
print(f"[eval] sample {idx}/{total} ({progress_pct}%) done")
|
||||
|
||||
# ---- 4. Compute basic metrics ----
|
||||
print(f"[eval] computing basic metrics on {len(predictions)} predictions")
|
||||
metrics_result: dict[str, Any] = {}
|
||||
|
||||
bleu_cfg = basic_cfg.get("bleu", {})
|
||||
if bleu_cfg.get("enabled"):
|
||||
metrics_result["bleu"] = _compute_bleu(references, predictions, int(bleu_cfg.get("ngram", 4)))
|
||||
|
||||
rouge_cfg = basic_cfg.get("rouge", {})
|
||||
if rouge_cfg.get("enabled"):
|
||||
metrics_result["rouge"] = _compute_rouge(references, predictions, rouge_cfg.get("methods"))
|
||||
|
||||
cosine_cfg = basic_cfg.get("cosine", {})
|
||||
if cosine_cfg.get("enabled"):
|
||||
metrics_result["cosine"] = _compute_cosine(references, predictions)
|
||||
metrics_result["exact_match"] = _compute_exact_match(references, predictions)
|
||||
metrics_result["text_similarity"] = _compute_text_similarity(references, predictions)
|
||||
|
||||
# ---- 5. Summarise ----
|
||||
completed = len(samples)
|
||||
if judge_enabled:
|
||||
scored = [s for s in samples if s.get("score") is not None]
|
||||
passed_count = len([s for s in scored if s.get("passed")])
|
||||
avg_score = round(sum(s["score"] for s in scored) / max(len(scored), 1), output_precision)
|
||||
max_score = dimension_cfg.get("score_max", 5)
|
||||
overall_score = round(avg_score / max_score * 100, output_precision)
|
||||
overall_score_max = 100
|
||||
dimension_summary = [{
|
||||
"name": "综合评分",
|
||||
"score": overall_score,
|
||||
"max_score": 100,
|
||||
"pass_rate": round(passed_count / max(completed, 1) * 100, 1),
|
||||
}]
|
||||
overall_evaluation = f"评测完成：{completed} 样本，{passed_count} 通过，平均 {avg_score}/{max_score} 分"
|
||||
else:
|
||||
passed_count = 0
|
||||
enabled_scores = [
|
||||
float(item.get("score") or 0)
|
||||
for item in metrics_result.values()
|
||||
if isinstance(item, dict) and item.get("enabled", True) and item.get("score") is not None
|
||||
]
|
||||
overall_score = round(sum(enabled_scores) / len(enabled_scores), output_precision) if enabled_scores else 0
|
||||
overall_score_max = 100
|
||||
dimension_summary = [
|
||||
{
|
||||
"name": name,
|
||||
"score": float(item.get("score") or 0),
|
||||
"max_score": 100,
|
||||
"pass_rate": float(item.get("score") or 0),
|
||||
}
|
||||
for name, item in metrics_result.items()
|
||||
if isinstance(item, dict) and item.get("enabled", True) and item.get("score") is not None
|
||||
]
|
||||
overall_evaluation = f"评测完成：{completed} 样本（未配置 LLM 评委）"
|
||||
|
||||
result = {
|
||||
"overall_score": overall_score,
|
||||
"overall_score_max": overall_score_max,
|
||||
"overall_evaluation": overall_evaluation,
|
||||
"improvement_suggestions": [],
|
||||
"dimension_summary": dimension_summary,
|
||||
"samples": samples,
|
||||
"sample_count": total,
|
||||
"completed_count": completed,
|
||||
"passed_count": passed_count,
|
||||
"basic_metrics": metrics_result,
|
||||
}
|
||||
|
||||
# ---- 6. Write results ----
|
||||
result_path = output_dir / "eval_results.json"
|
||||
result_path.write_text(json.dumps(result, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
print(f"[eval] results written to {result_path}")
|
||||
return result
|
||||
|
||||
|
||||
def main() -> None:
|
||||
import argparse
|
||||
parser = argparse.ArgumentParser(description="YG-FT Evaluation Runner")
|
||||
parser.add_argument("--config", required=True, help="Path to eval config JSON file")
|
||||
args = parser.parse_args()
|
||||
|
||||
config_path = Path(args.config)
|
||||
if not config_path.exists():
|
||||
print(f"FATAL: config file not found: {args.config}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
config = json.loads(config_path.read_text(encoding="utf-8"))
|
||||
start = time.time()
|
||||
try:
|
||||
run_eval(config)
|
||||
elapsed = time.time() - start
|
||||
print(f"[eval] DONE in {elapsed:.1f}s")
|
||||
except Exception as exc:
|
||||
print(f"[eval] FAILED: {exc}", file=sys.stderr)
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,272 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
import time
|
||||
import uuid
|
||||
from typing import Any, Iterator
|
||||
|
||||
|
||||
class InferenceSession:
|
||||
"""Manages a loaded model for inference with LLaMA-Factory ChatModel.
|
||||
|
||||
Model loading is asynchronous: ``load()`` spawns a background daemon thread
|
||||
and returns immediately with ``status == "loading"``. ``info()`` (served by
|
||||
``/inference/status``) is always responsive, so the platform backend can
|
||||
poll loading progress without being blocked by a minutes-long model load —
|
||||
which previously froze the whole compute node event loop.
|
||||
|
||||
State machine: idle -> loading -> ready | error, ready -> idle (unload),
|
||||
loading -> idle (cancelled). Long operations (ChatModel build, teardown,
|
||||
generation) never run while holding ``_state_lock``; they either run in the
|
||||
worker thread or under ``_chat_lock`` only.
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._state_lock = threading.Lock() # brief state transitions only
|
||||
self._chat_lock = threading.Lock() # serialize chat/teardown
|
||||
self._status: str = "idle"
|
||||
self._error: str = ""
|
||||
self._request_id: str = ""
|
||||
self._load_args: dict[str, Any] = {}
|
||||
self._teardown_old = False # load-while-ready: unload old before loading new
|
||||
self._cancel_requested = False # unload-while-loading: tear down after load finishes
|
||||
self._load_thread: threading.Thread | None = None
|
||||
self._model: Any = None
|
||||
self._tokenizer: Any = None
|
||||
self._generating_args: dict[str, Any] = {}
|
||||
self._model_name: str = ""
|
||||
self._adapter_path: str = ""
|
||||
self._loaded_at: float = 0.0
|
||||
|
||||
@property
|
||||
def status(self) -> str:
|
||||
with self._state_lock:
|
||||
return self._status
|
||||
|
||||
def info(self) -> dict[str, Any]:
|
||||
with self._state_lock:
|
||||
return {
|
||||
"loaded": self._status == "ready",
|
||||
"status": self._status,
|
||||
"model_name": self._model_name,
|
||||
"adapter_path": self._adapter_path,
|
||||
"loaded_at": self._loaded_at,
|
||||
"request_id": self._request_id,
|
||||
"error": self._error,
|
||||
}
|
||||
|
||||
def wait_until_loaded(self, timeout: float | None = None) -> dict[str, Any]:
|
||||
"""Wait for an in-flight async load to finish and return its outcome.
|
||||
|
||||
供同步消费方（如 eval_runner 子进程）使用：``load()`` 立即返回 loading 后，
|
||||
调用本方法等待后台加载线程完成，拿到最终的 loaded/error 结果。
|
||||
若在 timeout 秒内仍未加载完成，返回 ``status == "loading"`` 并附上超时提示。
|
||||
"""
|
||||
with self._state_lock:
|
||||
thread = self._load_thread
|
||||
if thread is not None and thread.is_alive():
|
||||
thread.join(timeout=timeout)
|
||||
with self._state_lock:
|
||||
loaded = self._status == "ready"
|
||||
status = self._status
|
||||
error = self._error
|
||||
if not loaded and status == "loading":
|
||||
error = error or f"model load timed out after {timeout or 'N/A'}s"
|
||||
return {
|
||||
"loaded": loaded,
|
||||
"status": status,
|
||||
"model_name": self._model_name,
|
||||
"adapter_path": self._adapter_path,
|
||||
"error": error,
|
||||
}
|
||||
|
||||
def load(
|
||||
self,
|
||||
model_name_or_path,
|
||||
adapter_name_or_path="",
|
||||
template="qwen",
|
||||
infer_backend="huggingface",
|
||||
infer_dtype="auto",
|
||||
**kwargs,
|
||||
) -> dict[str, Any]:
|
||||
with self._state_lock:
|
||||
if self._status == "loading":
|
||||
# A model is already loading — dedupe, reuse the same request id.
|
||||
return {"loaded": False, "status": "loading", "request_id": self._request_id}
|
||||
self._teardown_old = self._status == "ready"
|
||||
self._status = "loading"
|
||||
self._error = ""
|
||||
self._request_id = uuid.uuid4().hex[:12]
|
||||
self._cancel_requested = False
|
||||
self._load_args = {
|
||||
"model_name_or_path": model_name_or_path,
|
||||
"template": template,
|
||||
"infer_backend": infer_backend,
|
||||
"infer_dtype": infer_dtype,
|
||||
}
|
||||
if adapter_name_or_path:
|
||||
self._load_args["adapter_name_or_path"] = adapter_name_or_path
|
||||
self._load_args.update(kwargs)
|
||||
self._model_name = model_name_or_path
|
||||
self._adapter_path = adapter_name_or_path
|
||||
self._load_thread = threading.Thread(target=self._load_worker, daemon=True)
|
||||
self._load_thread.start()
|
||||
return {"loaded": False, "status": "loading", "request_id": self._request_id}
|
||||
|
||||
def _load_worker(self) -> None:
|
||||
"""Build the ChatModel off the state lock so info() never blocks."""
|
||||
model = None
|
||||
tokenizer = None
|
||||
generating_args: dict[str, Any] = {}
|
||||
error = ""
|
||||
try:
|
||||
if self._teardown_old:
|
||||
self._release_model()
|
||||
from llamafactory.chat import ChatModel
|
||||
from llamafactory.hparams import get_infer_args
|
||||
|
||||
args = dict(self._load_args)
|
||||
infer_result = get_infer_args(args)
|
||||
model = ChatModel(args)
|
||||
tokenizer = getattr(model, "tokenizer", None) or model.engine.tokenizer
|
||||
generating_args = infer_result[-1]
|
||||
if hasattr(generating_args, "__dataclass_fields__"):
|
||||
generating_args = {
|
||||
k: v for k, v in vars(generating_args).items() if not k.startswith("_")
|
||||
}
|
||||
else:
|
||||
generating_args = dict(generating_args)
|
||||
except Exception as exc: # noqa: BLE001 - surface load failure via status
|
||||
error = str(exc)
|
||||
with self._state_lock:
|
||||
if error:
|
||||
self._model = None
|
||||
self._tokenizer = None
|
||||
self._status = "error"
|
||||
self._error = error
|
||||
return
|
||||
if self._cancel_requested:
|
||||
# Unload was requested while loading — drop the fresh model.
|
||||
model = None
|
||||
tokenizer = None
|
||||
self._model = None
|
||||
self._tokenizer = None
|
||||
self._status = "idle"
|
||||
return
|
||||
self._model = model
|
||||
self._tokenizer = tokenizer
|
||||
self._generating_args = generating_args
|
||||
self._loaded_at = time.time()
|
||||
self._status = "ready"
|
||||
|
||||
def _release_model(self) -> None:
|
||||
with self._chat_lock:
|
||||
with self._state_lock:
|
||||
self._status = "unloading"
|
||||
model = self._model
|
||||
self._model = None
|
||||
self._tokenizer = None
|
||||
if model is not None:
|
||||
try:
|
||||
del model
|
||||
except Exception: # noqa: BLE001 - best-effort teardown
|
||||
pass
|
||||
# 强制释放 PyTorch CUDA 缓存，真正归还 GPU 显存
|
||||
try:
|
||||
import gc
|
||||
|
||||
gc.collect()
|
||||
import torch
|
||||
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.empty_cache()
|
||||
torch.cuda.synchronize()
|
||||
except Exception: # noqa: BLE001 - teardown must not raise
|
||||
pass
|
||||
with self._state_lock:
|
||||
self._status = "idle"
|
||||
self._model_name = ""
|
||||
self._adapter_path = ""
|
||||
self._loaded_at = 0.0
|
||||
self._error = ""
|
||||
|
||||
def unload(self) -> dict[str, Any]:
|
||||
with self._state_lock:
|
||||
if self._status == "loading":
|
||||
# Ask the worker to tear down right after the load finishes.
|
||||
self._cancel_requested = True
|
||||
return {"unloaded": False, "status": "cancelling", "request_id": self._request_id}
|
||||
was_ready = self._status == "ready"
|
||||
if was_ready:
|
||||
self._release_model()
|
||||
else:
|
||||
with self._state_lock:
|
||||
self._model = None
|
||||
self._tokenizer = None
|
||||
self._status = "idle"
|
||||
self._model_name = ""
|
||||
self._adapter_path = ""
|
||||
self._loaded_at = 0.0
|
||||
self._error = ""
|
||||
return {"unloaded": True, "status": "idle"}
|
||||
|
||||
def chat(self, messages, temperature=0.95, top_p=0.7, max_new_tokens=1024, do_sample=True, **kwargs) -> dict[str, Any]:
|
||||
with self._chat_lock:
|
||||
with self._state_lock:
|
||||
if self._status == "loading":
|
||||
return {
|
||||
"error": f"model is still loading (request_id={self._request_id}); please retry",
|
||||
"response": "",
|
||||
}
|
||||
if self._status == "error":
|
||||
return {"error": f"model load failed: {self._error}", "response": ""}
|
||||
if self._status != "ready" or self._model is None:
|
||||
return {"error": "model not loaded", "response": ""}
|
||||
try:
|
||||
generate_kwargs = {
|
||||
"temperature": temperature,
|
||||
"top_p": top_p,
|
||||
"max_new_tokens": max_new_tokens,
|
||||
"do_sample": do_sample,
|
||||
}
|
||||
generate_kwargs.update(kwargs)
|
||||
system = next((m["content"] for m in messages if m["role"] == "system"), None)
|
||||
user_messages = [m for m in messages if m["role"] != "system"]
|
||||
responses = []
|
||||
for response in self._model.stream_chat(user_messages, system=system, **generate_kwargs):
|
||||
responses.append(response)
|
||||
full_response = "".join(str(r) for r in responses)
|
||||
return {"response": full_response}
|
||||
except Exception as exc: # noqa: BLE001 - return generation error to caller
|
||||
return {"error": str(exc), "response": ""}
|
||||
|
||||
def chat_stream(self, messages, **kwargs) -> Iterator[str]:
|
||||
with self._chat_lock:
|
||||
with self._state_lock:
|
||||
if self._status == "loading":
|
||||
yield 'data: {"error": "model is still loading; please retry"}\n\n'
|
||||
return
|
||||
if self._status == "error":
|
||||
yield 'data: {"error": "model load failed: ' + str(self._error) + '"}\n\n'
|
||||
return
|
||||
if self._status != "ready" or self._model is None:
|
||||
yield 'data: {"error": "model not loaded"}\n\n'
|
||||
return
|
||||
try:
|
||||
generate_kwargs = {**kwargs}
|
||||
system = next((m["content"] for m in messages if m["role"] == "system"), None)
|
||||
user_messages = [m for m in messages if m["role"] != "system"]
|
||||
for new_text in self._model.stream_chat(user_messages, system=system, **generate_kwargs):
|
||||
yield new_text
|
||||
except Exception as exc: # noqa: BLE001 - stream error as SSE event
|
||||
yield 'data: {"error": "' + str(exc) + '"}\n\n'
|
||||
|
||||
|
||||
_inference_session = None
|
||||
|
||||
|
||||
def get_inference_session() -> InferenceSession:
|
||||
global _inference_session
|
||||
if _inference_session is None:
|
||||
_inference_session = InferenceSession()
|
||||
return _inference_session
|
||||
@@ -1,12 +0,0 @@
|
||||
fastapi>=0.111.0
|
||||
uvicorn[standard]>=0.30.0
|
||||
python-multipart>=0.0.9
|
||||
pydantic>=2.7.0
|
||||
python-dotenv>=1.0.1
|
||||
httpx>=0.27.0
|
||||
# 模型评测指标
|
||||
sacrebleu>=2.4.0
|
||||
rouge-score>=0.1.2
|
||||
scikit-learn>=1.3.0
|
||||
# LLaMA-Factory 训练引擎
|
||||
llamafactory
|
||||
@@ -1,146 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
import time
|
||||
import types
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
from compute.engines.llama_factory.inference import InferenceSession
|
||||
|
||||
# 模拟模型加载耗时，用于验证 load() 立即返回、info() 不阻塞
|
||||
LOAD_DELAY = 0.2
|
||||
|
||||
|
||||
class FakeChatModel:
|
||||
def __init__(self, args: dict[str, Any]) -> None:
|
||||
time.sleep(LOAD_DELAY)
|
||||
self.tokenizer = object()
|
||||
self.engine = types.SimpleNamespace(tokenizer=object())
|
||||
self._output = "hello from model"
|
||||
|
||||
def stream_chat(self, *args, **kwargs):
|
||||
for _ in range(1):
|
||||
yield self._output
|
||||
|
||||
|
||||
class FailingChatModel:
|
||||
def __init__(self, args: dict[str, Any]) -> None:
|
||||
time.sleep(LOAD_DELAY)
|
||||
raise RuntimeError("boom: fake load failure")
|
||||
|
||||
|
||||
def _get_infer_args(args: dict[str, Any]) -> list[Any]:
|
||||
# 最后一个元素为 generating_args，worker 会转成 dict
|
||||
return [None, None, {"temperature": 0.7}]
|
||||
|
||||
|
||||
def _install_llamafactory(monkeypatch, chat_model: type) -> None:
|
||||
llmf = types.ModuleType("llamafactory")
|
||||
chat_mod = types.ModuleType("llamafactory.chat")
|
||||
hparams_mod = types.ModuleType("llamafactory.hparams")
|
||||
chat_mod.ChatModel = chat_model
|
||||
hparams_mod.get_infer_args = _get_infer_args
|
||||
llmf.chat = chat_mod
|
||||
llmf.hparams = hparams_mod
|
||||
monkeypatch.setitem(sys.modules, "llamafactory", llmf)
|
||||
monkeypatch.setitem(sys.modules, "llamafactory.chat", chat_mod)
|
||||
monkeypatch.setitem(sys.modules, "llamafactory.hparams", hparams_mod)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def stub_llamafactory(monkeypatch) -> None:
|
||||
_install_llamafactory(monkeypatch, FakeChatModel)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def stub_failing_llamafactory(monkeypatch) -> None:
|
||||
_install_llamafactory(monkeypatch, FailingChatModel)
|
||||
|
||||
|
||||
def _wait_for_status(session: InferenceSession, status: str, timeout: float = 3.0) -> bool:
|
||||
deadline = time.time() + timeout
|
||||
while time.time() < deadline:
|
||||
if session.info()["status"] == status:
|
||||
return True
|
||||
time.sleep(0.02)
|
||||
return False
|
||||
|
||||
|
||||
def test_load_returns_immediately_then_ready(stub_llamafactory) -> None:
|
||||
session = InferenceSession()
|
||||
started = time.time()
|
||||
result = session.load("/models/qwen")
|
||||
assert result["status"] == "loading"
|
||||
assert result["loaded"] is False
|
||||
assert result["request_id"]
|
||||
# 在慢加载完成前就返回，且 info() 加载期间可响应
|
||||
assert time.time() - started < LOAD_DELAY
|
||||
assert session.info()["status"] == "loading"
|
||||
assert _wait_for_status(session, "ready")
|
||||
info = session.info()
|
||||
assert info["loaded"] is True
|
||||
assert info["status"] == "ready"
|
||||
assert info["model_name"] == "/models/qwen"
|
||||
|
||||
|
||||
def test_second_load_while_loading_deduped(stub_llamafactory) -> None:
|
||||
session = InferenceSession()
|
||||
r1 = session.load("/models/a")
|
||||
r2 = session.load("/models/b")
|
||||
assert r2["status"] == "loading"
|
||||
assert r2["request_id"] == r1["request_id"]
|
||||
assert _wait_for_status(session, "ready")
|
||||
assert session.info()["status"] == "ready"
|
||||
|
||||
|
||||
def test_load_error_surfaces_in_status(stub_failing_llamafactory) -> None:
|
||||
session = InferenceSession()
|
||||
session.load("/models/bad")
|
||||
assert _wait_for_status(session, "error")
|
||||
assert "boom" in session.info()["error"]
|
||||
|
||||
|
||||
def test_unload_while_loading_cancels(stub_llamafactory) -> None:
|
||||
session = InferenceSession()
|
||||
session.load("/models/qwen")
|
||||
result = session.unload()
|
||||
assert result["status"] == "cancelling"
|
||||
assert _wait_for_status(session, "idle")
|
||||
|
||||
|
||||
def test_chat_while_loading_returns_loading_error(stub_llamafactory) -> None:
|
||||
session = InferenceSession()
|
||||
session.load("/models/qwen")
|
||||
out = session.chat([{"role": "user", "content": "hi"}])
|
||||
assert "still loading" in (out.get("error") or "")
|
||||
assert _wait_for_status(session, "ready")
|
||||
out = session.chat([{"role": "user", "content": "hi"}])
|
||||
assert out.get("response") == "hello from model"
|
||||
|
||||
|
||||
def test_chat_stream_while_loading_yields_error(stub_llamafactory) -> None:
|
||||
session = InferenceSession()
|
||||
session.load("/models/qwen")
|
||||
chunks = list(session.chat_stream([{"role": "user", "content": "hi"}]))
|
||||
assert any("still loading" in c for c in chunks)
|
||||
|
||||
|
||||
def test_wait_until_loaded_blocks_until_ready(stub_llamafactory) -> None:
|
||||
session = InferenceSession()
|
||||
result = session.load("/models/qwen")
|
||||
assert result["status"] == "loading"
|
||||
# 同步等待后台加载线程完成
|
||||
outcome = session.wait_until_loaded(timeout=3.0)
|
||||
assert outcome["loaded"] is True
|
||||
assert outcome["status"] == "ready"
|
||||
|
||||
|
||||
def test_wait_until_loaded_reports_load_error(stub_failing_llamafactory) -> None:
|
||||
session = InferenceSession()
|
||||
session.load("/models/bad")
|
||||
outcome = session.wait_until_loaded(timeout=3.0)
|
||||
assert outcome["loaded"] is False
|
||||
assert outcome["status"] == "error"
|
||||
assert "boom" in outcome["error"]
|
||||
@@ -1,23 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from compute.engines.llama_factory.adapter import build_command
|
||||
|
||||
|
||||
def test_build_command_uses_explicit_validation_dataset_without_resplitting() -> None:
|
||||
result = build_command(
|
||||
{
|
||||
"base_model": "/models/qwen",
|
||||
"dataset": "ygft_dataset_train",
|
||||
"eval_dataset": "ygft_dataset_validation",
|
||||
"dataset_dir": "/datasets/example",
|
||||
"output_dir": "/outputs/example",
|
||||
"val_size": 0.1,
|
||||
}
|
||||
)
|
||||
|
||||
assert result.command[result.command.index("--dataset") + 1] == "ygft_dataset_train"
|
||||
assert result.command[result.command.index("--eval_dataset") + 1] == (
|
||||
"ygft_dataset_validation"
|
||||
)
|
||||
assert "--do_eval" in result.command
|
||||
assert "--val_size" not in result.command
|
||||
51
design-qa.md
51
design-qa.md
@@ -328,57 +328,6 @@ final result: blocked
|
||||
|
||||
---
|
||||
|
||||
# Failed Result Regeneration Design QA
|
||||
|
||||
## Evidence
|
||||
|
||||
- Source visual truth: `/var/folders/nk/yks07zp14wb4rv3jqq0pt_4h0000gn/T/codex-clipboard-0c2e6f90-aaac-4817-9084-bc31496aed0a.png`.
|
||||
- Implementation route: `http://localhost:16801/data-process/:id/workflow`, step 6.
|
||||
- Browser evidence: `/Users/caoxiaozhu/.codex/visualizations/2026/07/27/019fa277-626b-76a1-af02-48948d66c7ec/data-process-empty-list.png`.
|
||||
- Viewport: 1280 × 720.
|
||||
- Target state: a completed task on step 6 with a selected invalid result.
|
||||
- Available browser state: authenticated data-process list with zero tasks; no representative invalid result could be opened without creating or mutating local business data.
|
||||
|
||||
## Static and automated evidence
|
||||
|
||||
- Invalid results render a primary plain `重新生成` button in the right side of the result header.
|
||||
- Valid results keep the existing `恢复生成结果` action.
|
||||
- While regeneration is running, the selected row shows a spinner, the button shows loading, editors are disabled, and the footer confirmation action is disabled.
|
||||
- A successful response replaces only the selected row in place; a failed second generation leaves the original invalid row untouched.
|
||||
- `regression-data-process-wizard.mjs`: passed.
|
||||
- `regression-data-process-detail.mjs`: passed.
|
||||
- `regression-data-process-list.mjs`: passed.
|
||||
- `vue-tsc -b --noEmit`: passed.
|
||||
- Relevant backend tests: 97 passed, with one third-party deprecation warning.
|
||||
|
||||
## Required fidelity surfaces
|
||||
|
||||
- Typography and hierarchy: the action uses the existing Element Plus small-button hierarchy and remains secondary to result content.
|
||||
- Spacing and layout rhythm: the action is placed in the existing flex header's right action slot, matching the source location.
|
||||
- Colors and tokens: the action uses the existing product primary color and existing Font Awesome refresh icon.
|
||||
- Copy and content: the visible label is exactly `重新生成`; only server-saved invalid results expose it.
|
||||
- Interaction: optimistic concurrency protects against overwriting a newer edit, and persistence happens only after the replacement passes generation and quality validation.
|
||||
|
||||
## Findings
|
||||
|
||||
- [P2] Representative rendered comparison unavailable
|
||||
Location: step 6 result header with an invalid result selected.
|
||||
Evidence: the authenticated local database currently contains zero data-processing tasks, so the target state cannot be reached without creating test business data or invoking the configured model.
|
||||
Impact: automated and static checks prove the contract and placement, but cannot prove pixel-level spacing against the supplied screenshot.
|
||||
Fix: open any existing failed result after one is available, capture the same 1280 × 720 state, and compare it side by side with the source screenshot.
|
||||
|
||||
## Comparison history
|
||||
|
||||
### Iteration 1 — blocked
|
||||
|
||||
- Source screenshot was inspected and the implementation was aligned to its existing right-header action slot.
|
||||
- Local login and list navigation succeeded.
|
||||
- The target result state was unavailable because the local task list was empty.
|
||||
|
||||
final result: blocked
|
||||
|
||||
---
|
||||
|
||||
# Dataset Version Actions Design QA
|
||||
|
||||
## Evidence
|
||||
|
||||
15
docker-compose.yml
Normal file
15
docker-compose.yml
Normal file
@@ -0,0 +1,15 @@
|
||||
services:
|
||||
yg-ft-frontend:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile
|
||||
image: yg-ft-frontend:latest
|
||||
container_name: yg-ft-frontend
|
||||
ports:
|
||||
- "6801:80"
|
||||
environment:
|
||||
# Change this if the backend is deployed somewhere else.
|
||||
API_PROXY_PASS: "http://host.docker.internal:7861"
|
||||
extra_hosts:
|
||||
- "host.docker.internal:host-gateway"
|
||||
restart: unless-stopped
|
||||
299
docker/README.md
299
docker/README.md
@@ -1,299 +0,0 @@
|
||||
# Docker 部署说明
|
||||
|
||||
本目录按应用服务器和算力服务器拆分 Dockerfile 与 Docker Compose 文件。Compose 文件不包含 `build:`，不会在 `docker compose up` 时自动构建业务镜像。所有业务镜像需要先通过手动 `docker build` 构建，再由 Compose 启动。
|
||||
|
||||
## 基础镜像
|
||||
|
||||
| 镜像 | 用途 |
|
||||
| --- | --- |
|
||||
| `python:3.12-slim` | 应用后端基础镜像，后端运行环境要求 Python 3.12 及以上 |
|
||||
| `nginx:1.27-alpine` | 前端静态资源与 `/modelTF` 反向代理运行镜像 |
|
||||
| `hiyouga/llamafactory:latest` | 算力服务基础镜像，基于 LLaMA-Factory 官方镜像扩展 Compute API |
|
||||
| `postgres:16-alpine` | 开发阶段内置 PostgreSQL |
|
||||
| `redis:7-alpine` | 开发阶段内置 Redis |
|
||||
|
||||
一键拉取基础镜像：
|
||||
|
||||
```bash
|
||||
docker pull python:3.12-slim && \
|
||||
docker pull nginx:1.27-alpine && \
|
||||
docker pull hiyouga/llamafactory:latest && \
|
||||
docker pull postgres:16-alpine && \
|
||||
docker pull redis:7-alpine
|
||||
```
|
||||
|
||||
Windows PowerShell：
|
||||
|
||||
```powershell
|
||||
$images = @(
|
||||
"python:3.12-slim",
|
||||
"nginx:1.27-alpine",
|
||||
"hiyouga/llamafactory:latest",
|
||||
"postgres:16-alpine",
|
||||
"redis:7-alpine"
|
||||
)
|
||||
$images | ForEach-Object { docker pull $_ }
|
||||
```
|
||||
|
||||
如果部署环境不能访问外网，需要提前在可联网环境执行上述拉取命令，再用 `docker save` / `docker load` 导出导入。
|
||||
|
||||
## 业务镜像
|
||||
|
||||
| 镜像 | Dockerfile | 构建命令 |
|
||||
| --- | --- | --- |
|
||||
| `yg-ft-backend-api:latest` | `docker/app/Dockerfile.backend` | `docker build -f docker/app/Dockerfile.backend -t yg-ft-backend-api:latest .` |
|
||||
| `yg-ft-frontend-runtime:latest` | `docker/app/Dockerfile.frontend` | `docker build -f docker/app/Dockerfile.frontend -t yg-ft-frontend-runtime:latest .` |
|
||||
| `yg-ft-compute-api:latest` | `docker/compute/Dockerfile.compute` | `docker build -f docker/compute/Dockerfile.compute -t yg-ft-compute-api:latest .` |
|
||||
|
||||
## 对外端口
|
||||
|
||||
所有宿主机对外端口统一使用 5 位端口。容器内部端口保持镜像默认端口，便于容器内服务和健康检查稳定。
|
||||
|
||||
| 服务 | 宿主机对外端口 | 容器内部端口 | 说明 |
|
||||
| --- | --- | --- | --- |
|
||||
| 前端 Nginx | `16801` | `80` | 前端页面入口 |
|
||||
| 后端 API | `17861` | `8000` | FastAPI 服务 |
|
||||
| PostgreSQL | `15432` | `5432` | 开发阶段内置数据库 |
|
||||
| Redis | `16379` | `6379` | 开发阶段内置缓存 |
|
||||
| Compute API | `19100` | `9100` | 算力服务器 API |
|
||||
| File Gateway | `19101` | `9100` | 当前由 Compute API 暴露文件网关契约，后续可拆为独立服务 |
|
||||
|
||||
注意：`8000` 是后端容器内部端口，不作为宿主机对外访问端口。宿主机或浏览器应访问 `http://<app-server-ip>:17861/modelTF/health`；前端 Nginx 容器在 Docker 网络内部访问 `http://backend-api:8000/modelTF/...`。
|
||||
|
||||
对应配置文件：
|
||||
|
||||
- `docker/app/.env.example`
|
||||
- `FRONTEND_PORT=16801`
|
||||
- `BACKEND_API_PORT=17861`
|
||||
- `POSTGRES_PORT=15432`
|
||||
- `REDIS_PORT=16379`
|
||||
- `docker/compute/.env.example`
|
||||
- `COMPUTE_API_PORT=19100`
|
||||
- `FILE_GATEWAY_PORT=19101`
|
||||
|
||||
## 运行模式
|
||||
|
||||
- 应用侧默认 `COMPUTE_MODE=real`，任务状态必须由真实算力同步逻辑更新。
|
||||
- 算力侧默认 `COMPUTE_EXECUTION_MODE=real`，真实执行器未完成前不会伪造训练作业。
|
||||
- 仅隔离联调时可显式设置 `COMPUTE_MODE=simulator` 或 `COMPUTE_EXECUTION_MODE=simulator`，该模式不得用于测试环境、生产环境或生产升级基线。
|
||||
|
||||
## 应用服务器部署
|
||||
|
||||
应用服务器包含前端 Nginx、Backend API、PostgreSQL、Redis。
|
||||
|
||||
当前 Compose 内置 PostgreSQL 使用 `backend/app/db/sql/001_platform_runtime.sql` 初始化运行库。`docs/postgres-schema.sql` 是完整目标架构设计，不应直接挂载为当前运行库初始化脚本，否则会与当前后端代码的运行表结构不兼容。
|
||||
|
||||
首次部署：
|
||||
|
||||
```bash
|
||||
cd <repo-root>
|
||||
|
||||
# 1. 使用当前 Windows/宿主机 npm 构建前端静态产物
|
||||
cd frontend
|
||||
npm ci
|
||||
npm run build
|
||||
cd ..
|
||||
|
||||
# 2. 手动构建业务镜像
|
||||
docker build -f docker/app/Dockerfile.backend -t yg-ft-backend-api:latest .
|
||||
docker build -f docker/app/Dockerfile.frontend -t yg-ft-frontend-runtime:latest .
|
||||
|
||||
# 3. 启动应用服务
|
||||
cd docker/app
|
||||
cp .env.example .env
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
后端镜像构建过程中会执行依赖导入自检，确认 `fastapi`、`uvicorn`、`psycopg`、`sqlalchemy`、`redis` 等运行依赖已安装。构建后也可以手动检查：
|
||||
|
||||
```bash
|
||||
docker run --rm yg-ft-backend-api:latest python -c "import psycopg; print(psycopg.__version__)"
|
||||
```
|
||||
|
||||
默认访问地址：
|
||||
|
||||
```text
|
||||
http://<app-server-ip>:16801
|
||||
```
|
||||
|
||||
应用侧代码和数据外挂：
|
||||
|
||||
```text
|
||||
../../backend -> /app
|
||||
../../frontend/dist -> /usr/share/nginx/html
|
||||
../../runtime/app/logs/backend -> /opt/yg-ft/logs/backend
|
||||
../../runtime/app/data -> /data/yg-ft
|
||||
```
|
||||
|
||||
前端容器启动前必须确保 `../../frontend/dist/index.html` 已存在。若前端 Nginx 日志出现 `directory index of "/usr/share/nginx/html/" is forbidden` 或 `rewrite or internal redirection cycle while internally redirecting to "/index.html"`，通常表示当前执行 `docker compose` 的项目目录下没有构建好的 `frontend/dist`，或挂载路径不是同一份代码目录。
|
||||
|
||||
```bash
|
||||
# 在执行 docker compose 的同一份代码目录中检查
|
||||
cd <repo-root>/frontend
|
||||
npm run build
|
||||
test -f dist/index.html && ls -lh dist/index.html
|
||||
|
||||
cd ../docker/app
|
||||
docker compose up -d --force-recreate frontend
|
||||
docker compose logs --tail=80 frontend
|
||||
```
|
||||
|
||||
如果使用 Windows npm 构建、WSL 中运行 Docker Compose，需要确认 Windows 路径和 WSL 路径指向同一份仓库。例如在 `D:\...\YG_FT\frontend` 构建不会自动生成 `/mnt/d/wuyongtao/Code/YG_FT/frontend/dist` 下的产物，除非二者本就是同一个目录。
|
||||
|
||||
如果使用企业统一 PostgreSQL/Redis，修改 `docker/app/.env`：
|
||||
|
||||
如果前端 Nginx 日志出现 `open() "/usr/share/nginx/html/modelTF/login" failed` 或 `open() "/usr/share/nginx/html/login" failed`，说明当前容器没有加载项目的 Nginx 代理配置，`/modelTF/*` 被当成静态文件查找。处理方式：
|
||||
|
||||
```bash
|
||||
cd <repo-root>/docker/app
|
||||
docker compose up -d --force-recreate frontend
|
||||
docker compose exec frontend nginx -T | grep -n "location.*modelTF" -A12
|
||||
```
|
||||
|
||||
正常配置中应存在 `location ^~ /modelTF/`，并代理到 `BACKEND_PROXY_PASS`，默认是 `http://backend-api:8000`。
|
||||
|
||||
```env
|
||||
DATABASE_URL=postgresql+psycopg://<user>:<password>@<postgres-host>:15432/<db>
|
||||
REDIS_URL=redis://<redis-host>:16379/0
|
||||
USE_BUILTIN_POSTGRES=false
|
||||
USE_BUILTIN_REDIS=false
|
||||
```
|
||||
|
||||
生产环境如完全使用外部基础设施，可以删除或注释 Compose 中的 `postgres`、`redis` 服务及 `backend-api.depends_on` 中对应依赖。
|
||||
|
||||
## 算力服务器部署
|
||||
|
||||
算力服务器包含 Compute API、后续 Compute Agent、File Gateway、GPU runtime、本地训练数据目录和 LLaMA-Factory。`Dockerfile.compute` 基于 LLaMA-Factory 官方镜像：
|
||||
|
||||
```dockerfile
|
||||
FROM hiyouga/llamafactory:latest
|
||||
```
|
||||
|
||||
部署前需要安装：
|
||||
|
||||
- NVIDIA Driver
|
||||
- NVIDIA Container Toolkit
|
||||
- Docker Engine 和 Docker Compose Plugin
|
||||
- 本地训练数据目录，默认 `/data/yg-ft`
|
||||
|
||||
首次部署：
|
||||
|
||||
```bash
|
||||
cd <repo-root>
|
||||
|
||||
# 手动构建算力业务镜像
|
||||
docker build -f docker/compute/Dockerfile.compute -t yg-ft-compute-api:latest .
|
||||
|
||||
# 启动算力服务
|
||||
cd docker/compute
|
||||
cp .env.example .env
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
健康检查：
|
||||
|
||||
```text
|
||||
GET http://<compute-server-ip>:19100/modelTF/health
|
||||
GET http://<compute-server-ip>:19100/modelTF/v1/compute/health
|
||||
```
|
||||
|
||||
算力侧代码和数据外挂：
|
||||
|
||||
```text
|
||||
../../compute -> /app/compute
|
||||
${YG_FT_DATA_ROOT_HOST} -> /data/yg-ft
|
||||
${YG_FT_MODEL_ROOT_HOST} -> /data/yg-ft/models
|
||||
${YG_FT_DATASET_ROOT_HOST} -> /data/yg-ft/datasets
|
||||
${YG_FT_OUTPUT_ROOT_HOST} -> /data/yg-ft/outputs
|
||||
${COMPUTE_LOG_ROOT_HOST} -> /opt/yg-ft/logs/compute
|
||||
${TRAINING_LOG_ROOT_HOST} -> /opt/yg-ft/logs/training
|
||||
```
|
||||
|
||||
算力服务器启动前必须先在宿主机创建持久化目录，基座模型、训练数据、训练产物和训练日志都应落在宿主机磁盘上，不能只写入容器层。推荐默认目录：
|
||||
|
||||
```bash
|
||||
cd <repo-root>/docker/compute
|
||||
mkdir -p data/yg-ft/models \
|
||||
data/yg-ft/datasets \
|
||||
data/yg-ft/outputs \
|
||||
data/yg-ft/logs/compute \
|
||||
data/yg-ft/logs/training
|
||||
```
|
||||
|
||||
默认 `docker/compute/.env.example` 使用 `./data/yg-ft`，该相对路径以 `docker/compute/docker-compose.yml` 所在目录为基准，因此实际宿主机目录是 `<repo-root>/docker/compute/data/yg-ft`。如企业环境模型盘、数据盘、产物盘分盘挂载，可在 `docker/compute/.env` 中分别调整 `YG_FT_MODEL_ROOT_HOST`、`YG_FT_DATASET_ROOT_HOST`、`YG_FT_OUTPUT_ROOT_HOST`、`COMPUTE_LOG_ROOT_HOST`、`TRAINING_LOG_ROOT_HOST`，容器内路径建议保持 `/data/yg-ft/models`、`/data/yg-ft/datasets`、`/data/yg-ft/outputs`，避免训练参数和节点配置复杂化。
|
||||
|
||||
页面上传数据集时，文件先进入 Backend API，再由 Backend API 调用目标算力节点的 `POST /modelTF/compute/files/upload`，写入容器内 `/data/yg-ft/datasets/{dataset_id}/`。在默认开发配置下，宿主机可在 `<repo-root>/docker/compute/data/yg-ft/datasets/{dataset_id}/` 看到对应文件。仅创建 bind mount 不会自动让应用侧上传文件出现在算力目录，必须通过这条 File Gateway 链路同步。
|
||||
|
||||
## 应用与算力分离部署
|
||||
|
||||
应用服务器只需要主动访问算力服务器，不要求算力服务器回调应用服务器。
|
||||
|
||||
在 `docker/app/.env` 中配置：
|
||||
|
||||
```env
|
||||
COMPUTE_API_BASE_URL=http://<compute-server-ip>:19100
|
||||
FILE_GATEWAY_BASE_URL=http://<compute-server-ip>:19101
|
||||
COMPUTE_SERVICE_TOKEN=change_me
|
||||
COMPUTE_STATUS_SYNC_MODE=polling
|
||||
COMPUTE_POLL_INTERVAL_SECONDS=3
|
||||
COMPUTE_POLL_BATCH_SIZE=100
|
||||
```
|
||||
|
||||
交互链路：
|
||||
|
||||
```text
|
||||
Frontend
|
||||
-> Backend API
|
||||
-> Compute API
|
||||
-> Compute Agent / LLaMA-Factory
|
||||
-> 本地数据目录 / 模型目录 / 训练产物
|
||||
<- Backend Worker 定时轮询 Compute API
|
||||
```
|
||||
|
||||
算力服务默认开启服务间鉴权。`docker/compute/.env` 中保持 `COMPUTE_AUTH_ENABLED=true`，并确保 `COMPUTE_SERVICE_TOKEN` 与 `docker/app/.env` 一致；健康检查路径仍可用于容器探活。
|
||||
|
||||
## 多算力节点部署
|
||||
|
||||
多算力节点仍按“单机多 GPU 节点”部署。每台 GPU 服务器都独立部署一套 `docker/compute`：
|
||||
|
||||
```text
|
||||
gpu-node-01: docker/compute + /data/yg-ft + 19100/19101
|
||||
gpu-node-02: docker/compute + /data/yg-ft + 19100/19101
|
||||
gpu-node-03: docker/compute + /data/yg-ft + 19100/19101
|
||||
```
|
||||
|
||||
节点之间默认不互访。应用平台主动访问每个节点的 Compute API/File Gateway，并通过 `compute_nodes`、`resource_replicas`、`resource_sync_jobs` 统一调度和同步。
|
||||
|
||||
节点地址、权重、标签、启用状态和本地路径在前端“算力节点”页面动态维护。新增或编辑节点后，点击“测试”会由 Backend API 主动访问该节点的 `GET /modelTF/v1/compute/health` 和 `GET /modelTF/compute/resources/gpus`，并把健康信息与 GPU 清单同步到 PostgreSQL。
|
||||
|
||||
## 常用命令
|
||||
|
||||
重新构建应用镜像：
|
||||
|
||||
```bash
|
||||
docker build -f docker/app/Dockerfile.backend -t yg-ft-backend-api:latest .
|
||||
docker build -f docker/app/Dockerfile.frontend -t yg-ft-frontend-runtime:latest .
|
||||
```
|
||||
|
||||
重新构建算力镜像：
|
||||
|
||||
```bash
|
||||
docker build -f docker/compute/Dockerfile.compute -t yg-ft-compute-api:latest .
|
||||
```
|
||||
|
||||
启动服务：
|
||||
|
||||
```bash
|
||||
cd docker/app
|
||||
docker compose up -d
|
||||
|
||||
cd ../compute
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
查看服务：
|
||||
|
||||
```bash
|
||||
docker compose ps
|
||||
docker compose logs -f
|
||||
```
|
||||
@@ -1,46 +0,0 @@
|
||||
APP_ENV=prod
|
||||
APP_NAME=YG Fine-Tune Platform API
|
||||
MODELTF_ROUTE_PREFIX=/modelTF
|
||||
CORS_ALLOW_ORIGINS=http://localhost:16801,http://127.0.0.1:16801
|
||||
|
||||
FRONTEND_IMAGE=yg-ft-frontend-runtime:latest
|
||||
BACKEND_API_IMAGE=yg-ft-backend-api:latest
|
||||
|
||||
# Five-digit host ports exposed outside the application server.
|
||||
FRONTEND_PORT=16801
|
||||
BACKEND_API_PORT=17861
|
||||
POSTGRES_PORT=15432
|
||||
REDIS_PORT=16379
|
||||
|
||||
POSTGRES_DB=yg_ft
|
||||
POSTGRES_USER=root
|
||||
POSTGRES_PASSWORD=8811614287327Leo
|
||||
DATABASE_URL=postgresql+psycopg://root:8811614287327Leo@www.caoxiaozhu.com:5432/yg_ft
|
||||
|
||||
REDIS_URL=redis://redis:6379/0
|
||||
|
||||
# PostgreSQL uses the shared external database. The local postgres service is disabled in docker-compose.yml.
|
||||
# Redis still uses the built-in service during current development.
|
||||
USE_BUILTIN_POSTGRES=false
|
||||
USE_BUILTIN_REDIS=true
|
||||
|
||||
LOG_LEVEL=INFO
|
||||
LOG_DIR=/opt/yg-ft/logs/backend
|
||||
LOG_FILE_PREFIX=backend
|
||||
LOG_ERROR_FILE_PREFIX=error
|
||||
LOG_MAX_BYTES=20971520
|
||||
LOG_RETENTION_DAYS=10
|
||||
|
||||
BACKEND_PROXY_PASS=http://backend-api:8000
|
||||
|
||||
# Split deployment: set these to the compute server address, for example http://10.10.20.31:19100.
|
||||
COMPUTE_API_BASE_URL=http://compute-api:9100
|
||||
COMPUTE_SERVICE_TOKEN=change_me
|
||||
FILE_GATEWAY_BASE_URL=http://compute-api:9101
|
||||
|
||||
# The application side polls Compute API for job state to avoid opening reverse network access.
|
||||
COMPUTE_MODE=real
|
||||
COMPUTE_STATUS_SYNC_MODE=polling
|
||||
COMPUTE_POLL_INTERVAL_SECONDS=10
|
||||
COMPUTE_POLL_BATCH_SIZE=100
|
||||
COMPUTE_REQUEST_TIMEOUT_SECONDS=5
|
||||
@@ -1,21 +0,0 @@
|
||||
FROM python:3.12-slim
|
||||
|
||||
ENV PYTHONDONTWRITEBYTECODE=1 \
|
||||
PYTHONUNBUFFERED=1 \
|
||||
PIP_NO_CACHE_DIR=1
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY backend/requirements.txt /tmp/requirements.txt
|
||||
RUN pip install --upgrade pip -i https://pypi.tuna.tsinghua.edu.cn/simple \
|
||||
&& pip install -r /tmp/requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple \
|
||||
&& rm -f /tmp/requirements.txt
|
||||
|
||||
RUN python -c "import fastapi, uvicorn, psycopg, psycopg_pool, sqlalchemy, redis, jwt, passlib, httpx, alembic; print('backend dependency check ok')"
|
||||
|
||||
RUN mkdir -p /opt/yg-ft/logs/backend /data/yg-ft \
|
||||
&& chmod -R 0775 /opt/yg-ft /data/yg-ft
|
||||
|
||||
EXPOSE 8000
|
||||
|
||||
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
@@ -1,9 +0,0 @@
|
||||
|
||||
FROM nginx:1.27-alpine
|
||||
|
||||
RUN mkdir -p /usr/share/nginx/html
|
||||
|
||||
EXPOSE 80
|
||||
|
||||
HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
|
||||
CMD test -f /usr/share/nginx/html/index.html && wget -qO- http://127.0.0.1/index.html >/dev/null || exit 1
|
||||
@@ -1,126 +0,0 @@
|
||||
services:
|
||||
frontend:
|
||||
image: ${FRONTEND_IMAGE:-yg-ft-frontend-runtime:latest}
|
||||
container_name: yg-ft-frontend
|
||||
depends_on:
|
||||
backend-api:
|
||||
condition: service_started
|
||||
ports:
|
||||
- "${FRONTEND_PORT:-16801}:80"
|
||||
environment:
|
||||
BACKEND_PROXY_PASS: ${BACKEND_PROXY_PASS:-http://backend-api:8000}
|
||||
volumes:
|
||||
- ../../frontend/dist:/usr/share/nginx/html:ro
|
||||
- ../nginx.conf.template:/etc/nginx/templates/default.conf.template:ro
|
||||
command:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- |
|
||||
if [ ! -f /usr/share/nginx/html/index.html ]; then
|
||||
echo "frontend dist is missing: build frontend first and ensure ../../frontend/dist is mounted";
|
||||
ls -la /usr/share/nginx/html;
|
||||
exit 1;
|
||||
fi;
|
||||
envsubst '$$BACKEND_PROXY_PASS' < /etc/nginx/templates/default.conf.template > /etc/nginx/conf.d/default.conf;
|
||||
nginx -t;
|
||||
nginx -g 'daemon off;'
|
||||
networks:
|
||||
- yg-ft-app
|
||||
restart: unless-stopped
|
||||
|
||||
backend-api:
|
||||
image: ${BACKEND_API_IMAGE:-yg-ft-backend-api:latest}
|
||||
container_name: yg-ft-backend-api
|
||||
depends_on:
|
||||
redis:
|
||||
condition: service_healthy
|
||||
expose:
|
||||
- "8000"
|
||||
ports:
|
||||
- "${BACKEND_API_PORT:-17861}:8000"
|
||||
environment:
|
||||
APP_ENV: ${APP_ENV:-prod}
|
||||
APP_NAME: ${APP_NAME:-YG Fine-Tune Platform API}
|
||||
MODELTF_ROUTE_PREFIX: ${MODELTF_ROUTE_PREFIX:-/modelTF}
|
||||
CORS_ALLOW_ORIGINS: ${CORS_ALLOW_ORIGINS:-http://localhost:16801,http://127.0.0.1:16801}
|
||||
DATABASE_URL: ${DATABASE_URL:-postgresql+psycopg://root:8811614287327Leo@www.caoxiaozhu.com:5432/yg_ft}
|
||||
REDIS_URL: ${REDIS_URL:-redis://redis:6379/0}
|
||||
USE_BUILTIN_POSTGRES: ${USE_BUILTIN_POSTGRES:-false}
|
||||
USE_BUILTIN_REDIS: ${USE_BUILTIN_REDIS:-true}
|
||||
LOG_LEVEL: ${LOG_LEVEL:-INFO}
|
||||
LOG_DIR: ${LOG_DIR:-/opt/yg-ft/logs/backend}
|
||||
LOG_FILE_PREFIX: ${LOG_FILE_PREFIX:-backend}
|
||||
LOG_ERROR_FILE_PREFIX: ${LOG_ERROR_FILE_PREFIX:-error}
|
||||
LOG_MAX_BYTES: ${LOG_MAX_BYTES:-20971520}
|
||||
LOG_RETENTION_DAYS: ${LOG_RETENTION_DAYS:-10}
|
||||
COMPUTE_API_BASE_URL: ${COMPUTE_API_BASE_URL:-http://compute-api:9100}
|
||||
COMPUTE_SERVICE_TOKEN: ${COMPUTE_SERVICE_TOKEN:-change_me}
|
||||
FILE_GATEWAY_BASE_URL: ${FILE_GATEWAY_BASE_URL:-http://compute-api:9101}
|
||||
COMPUTE_MODE: ${COMPUTE_MODE:-real}
|
||||
COMPUTE_STATUS_SYNC_MODE: ${COMPUTE_STATUS_SYNC_MODE:-polling}
|
||||
COMPUTE_POLL_INTERVAL_SECONDS: ${COMPUTE_POLL_INTERVAL_SECONDS:-3}
|
||||
COMPUTE_POLL_BATCH_SIZE: ${COMPUTE_POLL_BATCH_SIZE:-100}
|
||||
COMPUTE_REQUEST_TIMEOUT_SECONDS: ${COMPUTE_REQUEST_TIMEOUT_SECONDS:-5}
|
||||
PYTHONPATH: /app
|
||||
volumes:
|
||||
- ../../backend:/app:ro
|
||||
- ../../runtime/app/logs/backend:/opt/yg-ft/logs/backend
|
||||
- ../../runtime/app/data:/data/yg-ft
|
||||
networks:
|
||||
- yg-ft-app
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "python -c \"import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/modelTF/health', timeout=3).read()\""]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 20s
|
||||
restart: unless-stopped
|
||||
|
||||
# PostgreSQL uses the shared external database configured by DATABASE_URL in docker/app/.env.
|
||||
# Keep this local service commented out unless development needs an isolated database again.
|
||||
# postgres:
|
||||
# image: postgres:16-alpine
|
||||
# container_name: yg-ft-postgres
|
||||
# environment:
|
||||
# POSTGRES_DB: ${POSTGRES_DB:-yg_ft}
|
||||
# POSTGRES_USER: ${POSTGRES_USER:-yg_ft}
|
||||
# POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-change_me}
|
||||
# PGDATA: /var/lib/postgresql/data/pgdata
|
||||
# volumes:
|
||||
# - postgres_data:/var/lib/postgresql/data
|
||||
# - ../../backend/app/db/sql/001_platform_runtime.sql:/docker-entrypoint-initdb.d/001-platform-runtime.sql:ro
|
||||
# ports:
|
||||
# - "${POSTGRES_PORT:-15432}:5432"
|
||||
# networks:
|
||||
# - yg-ft-app
|
||||
# healthcheck:
|
||||
# test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
|
||||
# interval: 10s
|
||||
# timeout: 5s
|
||||
# retries: 5
|
||||
# restart: unless-stopped
|
||||
|
||||
redis:
|
||||
image: redis:7-alpine
|
||||
container_name: yg-ft-redis
|
||||
command: ["redis-server", "--appendonly", "yes"]
|
||||
volumes:
|
||||
- redis_data:/data
|
||||
ports:
|
||||
- "${REDIS_PORT:-16379}:6379"
|
||||
networks:
|
||||
- yg-ft-app
|
||||
healthcheck:
|
||||
test: ["CMD", "redis-cli", "ping"]
|
||||
interval: 10s
|
||||
timeout: 3s
|
||||
retries: 5
|
||||
restart: unless-stopped
|
||||
|
||||
networks:
|
||||
yg-ft-app:
|
||||
name: yg-ft-app
|
||||
|
||||
volumes:
|
||||
# postgres_data:
|
||||
redis_data:
|
||||
@@ -1,40 +0,0 @@
|
||||
COMPUTE_ENV=prod
|
||||
COMPUTE_HOST_ID=gpu-node-01
|
||||
COMPUTE_EXECUTION_MODE=real
|
||||
MODELTF_ROUTE_PREFIX=/modelTF
|
||||
# Five-digit host ports exposed outside the compute server.
|
||||
COMPUTE_API_PORT=19100
|
||||
FILE_GATEWAY_PORT=19101
|
||||
COMPUTE_API_IMAGE=yg-ft-compute-api:latest
|
||||
|
||||
# The application server actively polls Compute API; compute server does not need reverse access.
|
||||
COMPUTE_AUTH_ENABLED=true
|
||||
COMPUTE_SERVICE_TOKEN=change_me
|
||||
ENABLE_APP_CALLBACK=false
|
||||
|
||||
# LLaMA-Factory is provided by the official hiyouga/llamafactory base image.
|
||||
LLAMA_FACTORY_HOME=/app/LLaMA-Factory
|
||||
|
||||
YG_FT_DATA_ROOT=/data/yg-ft
|
||||
YG_FT_DATA_ROOT_HOST=./data/yg-ft
|
||||
YG_FT_MODEL_ROOT=/data/yg-ft/models
|
||||
YG_FT_MODEL_ROOT_HOST=./data/yg-ft/models
|
||||
YG_FT_DATASET_ROOT=/data/yg-ft/datasets
|
||||
YG_FT_DATASET_ROOT_HOST=./data/yg-ft/datasets
|
||||
YG_FT_OUTPUT_ROOT=/data/yg-ft/outputs
|
||||
YG_FT_OUTPUT_ROOT_HOST=./data/yg-ft/outputs
|
||||
TRAINING_LOG_ROOT=/opt/yg-ft/logs/training
|
||||
TRAINING_LOG_ROOT_HOST=./data/yg-ft/logs/training
|
||||
COMPUTE_LOG_ROOT_HOST=./data/yg-ft/logs/compute
|
||||
|
||||
# Optional fallback used when nvidia-smi is unavailable.
|
||||
# Leave COMPUTE_GPU_COUNT=0 on real GPU servers with working NVIDIA runtime.
|
||||
COMPUTE_GPU_COUNT=0
|
||||
COMPUTE_GPU_NAME=NVIDIA A800-SXM4-80GB
|
||||
COMPUTE_GPU_MEMORY_GB=80
|
||||
COMPUTE_GPU_POWER_LIMIT_W=300
|
||||
|
||||
LOG_DIR=/opt/yg-ft/logs/compute
|
||||
CUDA_VISIBLE_DEVICES=0
|
||||
NVIDIA_VISIBLE_DEVICES=all
|
||||
NVIDIA_DRIVER_CAPABILITIES=compute,utility
|
||||
@@ -1,44 +0,0 @@
|
||||
COMPUTE_ENV=prod
|
||||
COMPUTE_HOST_ID=gpu-node-01
|
||||
COMPUTE_EXECUTION_MODE=real
|
||||
MODELTF_ROUTE_PREFIX=/modelTF
|
||||
# Five-digit host ports exposed outside the compute server.
|
||||
COMPUTE_API_PORT=19100
|
||||
FILE_GATEWAY_PORT=19101
|
||||
COMPUTE_API_IMAGE=yg-ft-compute-api:latest
|
||||
|
||||
# The application server actively polls Compute API; compute server does not need reverse access.
|
||||
COMPUTE_AUTH_ENABLED=true
|
||||
COMPUTE_SERVICE_TOKEN=change_me
|
||||
ENABLE_APP_CALLBACK=false
|
||||
|
||||
# LLaMA-Factory is provided by the official hiyouga/llamafactory base image.
|
||||
LLAMA_FACTORY_HOME=/app/LLaMA-Factory
|
||||
|
||||
# Persistent host directories on the compute server.
|
||||
# Create these directories before starting docker compose. They are mounted into
|
||||
# the container so base models, datasets, training outputs and logs survive
|
||||
# container recreation or image upgrades.
|
||||
YG_FT_DATA_ROOT=/data/yg-ft
|
||||
YG_FT_DATA_ROOT_HOST=./data/yg-ft
|
||||
YG_FT_MODEL_ROOT=/data/yg-ft/models
|
||||
YG_FT_MODEL_ROOT_HOST=./data/yg-ft/models
|
||||
YG_FT_DATASET_ROOT=/data/yg-ft/datasets
|
||||
YG_FT_DATASET_ROOT_HOST=./data/yg-ft/datasets
|
||||
YG_FT_OUTPUT_ROOT=/data/yg-ft/outputs
|
||||
YG_FT_OUTPUT_ROOT_HOST=./data/yg-ft/outputs
|
||||
TRAINING_LOG_ROOT=/opt/yg-ft/logs/training
|
||||
TRAINING_LOG_ROOT_HOST=./data/yg-ft/logs/training
|
||||
COMPUTE_LOG_ROOT_HOST=./data/yg-ft/logs/compute
|
||||
|
||||
# Optional fallback used when nvidia-smi is unavailable.
|
||||
# Leave COMPUTE_GPU_COUNT=0 on real GPU servers with working NVIDIA runtime.
|
||||
COMPUTE_GPU_COUNT=0
|
||||
COMPUTE_GPU_NAME=NVIDIA A800-SXM4-80GB
|
||||
COMPUTE_GPU_MEMORY_GB=80
|
||||
COMPUTE_GPU_POWER_LIMIT_W=300
|
||||
|
||||
LOG_DIR=/opt/yg-ft/logs/compute
|
||||
CUDA_VISIBLE_DEVICES=all
|
||||
NVIDIA_VISIBLE_DEVICES=all
|
||||
NVIDIA_DRIVER_CAPABILITIES=compute,utility
|
||||
@@ -1,27 +0,0 @@
|
||||
|
||||
FROM hiyouga/llamafactory:latest
|
||||
|
||||
ENV DEBIAN_FRONTEND=noninteractive \
|
||||
PYTHONDONTWRITEBYTECODE=1 \
|
||||
PYTHONUNBUFFERED=1 \
|
||||
PIP_NO_CACHE_DIR=1
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
RUN apt-get update \
|
||||
&& apt-get install -y --no-install-recommends tini \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
COPY compute/requirements.txt /tmp/requirements.txt
|
||||
RUN pip install --upgrade pip -i https://pypi.tuna.tsinghua.edu.cn/simple \
|
||||
&& pip install -r /tmp/requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple \
|
||||
&& rm -f /tmp/requirements.txt
|
||||
|
||||
RUN mkdir -p /opt/yg-ft/logs/compute /opt/yg-ft/logs/training /data/yg-ft /app/LLaMA-Factory \
|
||||
&& chmod -R 0775 /opt/yg-ft /data/yg-ft /app/LLaMA-Factory
|
||||
|
||||
ENTRYPOINT ["/usr/bin/tini", "--"]
|
||||
|
||||
EXPOSE 9100
|
||||
|
||||
CMD ["uvicorn", "compute.api.main:app", "--host", "0.0.0.0", "--port", "9100"]
|
||||
@@ -1,38 +0,0 @@
|
||||
# YG-FT Compute 数据目录说明
|
||||
|
||||
本目录挂载到 `yg-ft-compute-api` 容器的 `/data/yg-ft`，用于持久化存储训练相关的数据。
|
||||
|
||||
## 目录结构
|
||||
|
||||
```
|
||||
data/yg-ft/
|
||||
├── datasets/ # 数据集存储目录
|
||||
├── models/ # 模型文件存储目录
|
||||
├── outputs/ # 训练/推理输出结果目录
|
||||
└── logs/
|
||||
├── compute/ # 计算服务运行日志
|
||||
└── training/ # 训练任务执行日志
|
||||
```
|
||||
|
||||
## 各目录说明
|
||||
|
||||
### datasets/
|
||||
训练和评估所使用的数据集文件，包括 JSON、JSONL、CSV 等格式。数据集由用户上传或通过平台创建，供 LLaMA-Factory 等训练引擎读取。
|
||||
|
||||
### models/
|
||||
存放模型文件，包括：
|
||||
- 预训练基座模型（如 LLaMA、Qwen 等）
|
||||
- 微调后的自定义模型权重
|
||||
- 合并后的部署模型
|
||||
|
||||
### outputs/
|
||||
训练任务和推理任务的输出结果，包括：
|
||||
- 训练过程中的 checkpoint 文件
|
||||
- 评估结果和指标报告
|
||||
- 推理生成的结果文本
|
||||
|
||||
### logs/compute/
|
||||
计算服务（compute-api）的运行时日志，用于排查服务启动、GPU 调度、健康检查等问题。
|
||||
|
||||
### logs/training/
|
||||
各训练任务的执行日志，记录训练过程状态、报错信息等，便于追踪单个任务的运行情况。
|
||||
@@ -1,2 +0,0 @@
|
||||
*
|
||||
!.gitignore
|
||||
@@ -1,2 +0,0 @@
|
||||
*
|
||||
!.gitignore
|
||||
@@ -1,2 +0,0 @@
|
||||
*
|
||||
!.gitignore
|
||||
2
docker/compute/data/yg-ft/models/.gitignore
vendored
2
docker/compute/data/yg-ft/models/.gitignore
vendored
@@ -1,2 +0,0 @@
|
||||
*
|
||||
!.gitignore
|
||||
2
docker/compute/data/yg-ft/outputs/.gitignore
vendored
2
docker/compute/data/yg-ft/outputs/.gitignore
vendored
@@ -1,2 +0,0 @@
|
||||
*
|
||||
!.gitignore
|
||||
@@ -1,53 +0,0 @@
|
||||
services:
|
||||
compute-api:
|
||||
image: ${COMPUTE_API_IMAGE:-yg-ft-compute-api:latest}
|
||||
container_name: yg-ft-compute-api
|
||||
gpus: all
|
||||
ports:
|
||||
- "${COMPUTE_API_PORT:-19100}:9100"
|
||||
- "${FILE_GATEWAY_PORT:-19101}:9100"
|
||||
environment:
|
||||
COMPUTE_ENV: ${COMPUTE_ENV:-prod}
|
||||
COMPUTE_HOST_ID: ${COMPUTE_HOST_ID:-gpu-node-01}
|
||||
COMPUTE_EXECUTION_MODE: ${COMPUTE_EXECUTION_MODE:-real}
|
||||
MODELTF_ROUTE_PREFIX: ${MODELTF_ROUTE_PREFIX:-/modelTF}
|
||||
COMPUTE_AUTH_ENABLED: ${COMPUTE_AUTH_ENABLED:-true}
|
||||
COMPUTE_SERVICE_TOKEN: ${COMPUTE_SERVICE_TOKEN:-change_me}
|
||||
ENABLE_APP_CALLBACK: ${ENABLE_APP_CALLBACK:-false}
|
||||
LLAMA_FACTORY_HOME: ${LLAMA_FACTORY_HOME:-/app/LLaMA-Factory}
|
||||
YG_FT_DATA_ROOT: ${YG_FT_DATA_ROOT:-/data/yg-ft}
|
||||
YG_FT_MODEL_ROOT: ${YG_FT_MODEL_ROOT:-/data/yg-ft/models}
|
||||
YG_FT_DATASET_ROOT: ${YG_FT_DATASET_ROOT:-/data/yg-ft/datasets}
|
||||
YG_FT_OUTPUT_ROOT: ${YG_FT_OUTPUT_ROOT:-/data/yg-ft/outputs}
|
||||
TRAINING_LOG_ROOT: ${TRAINING_LOG_ROOT:-/opt/yg-ft/logs/training}
|
||||
COMPUTE_GPU_COUNT: ${COMPUTE_GPU_COUNT:-0}
|
||||
COMPUTE_GPU_NAME: ${COMPUTE_GPU_NAME:-NVIDIA A800-SXM4-80GB}
|
||||
COMPUTE_GPU_MEMORY_GB: ${COMPUTE_GPU_MEMORY_GB:-80}
|
||||
COMPUTE_GPU_POWER_LIMIT_W: ${COMPUTE_GPU_POWER_LIMIT_W:-300}
|
||||
MIN_TRAINING_GPU_MEMORY_GB: ${MIN_TRAINING_GPU_MEMORY_GB:-4}
|
||||
LOG_DIR: ${LOG_DIR:-/opt/yg-ft/logs/compute}
|
||||
CUDA_VISIBLE_DEVICES: ${CUDA_VISIBLE_DEVICES:-all}
|
||||
NVIDIA_VISIBLE_DEVICES: ${NVIDIA_VISIBLE_DEVICES:-all}
|
||||
NVIDIA_DRIVER_CAPABILITIES: ${NVIDIA_DRIVER_CAPABILITIES:-compute,utility}
|
||||
PYTHONPATH: /app
|
||||
volumes:
|
||||
- ../../compute:/app/compute:ro
|
||||
- ${YG_FT_DATA_ROOT_HOST:-./data/yg-ft}:${YG_FT_DATA_ROOT:-/data/yg-ft}
|
||||
- ${YG_FT_MODEL_ROOT_HOST:-./data/yg-ft/models}:${YG_FT_MODEL_ROOT:-/data/yg-ft/models}
|
||||
- ${YG_FT_DATASET_ROOT_HOST:-./data/yg-ft/datasets}:${YG_FT_DATASET_ROOT:-/data/yg-ft/datasets}
|
||||
- ${YG_FT_OUTPUT_ROOT_HOST:-./data/yg-ft/outputs}:${YG_FT_OUTPUT_ROOT:-/data/yg-ft/outputs}
|
||||
- ${COMPUTE_LOG_ROOT_HOST:-./data/yg-ft/logs/compute}:${LOG_DIR:-/opt/yg-ft/logs/compute}
|
||||
- ${TRAINING_LOG_ROOT_HOST:-./data/yg-ft/logs/training}:${TRAINING_LOG_ROOT:-/opt/yg-ft/logs/training}
|
||||
networks:
|
||||
- yg-ft-compute
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "python -c \"import urllib.request; urllib.request.urlopen('http://127.0.0.1:9100/modelTF/health', timeout=3).read()\""]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 20s
|
||||
restart: unless-stopped
|
||||
|
||||
networks:
|
||||
yg-ft-compute:
|
||||
name: yg-ft-compute
|
||||
@@ -7,26 +7,19 @@ server {
|
||||
|
||||
client_max_body_size 200m;
|
||||
|
||||
location ^~ /modelTF/ {
|
||||
proxy_pass ${BACKEND_PROXY_PASS};
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_read_timeout 900s;
|
||||
proxy_send_timeout 900s;
|
||||
location / {
|
||||
try_files $uri $uri/ /index.html;
|
||||
}
|
||||
|
||||
location = /modelTF {
|
||||
proxy_pass ${BACKEND_PROXY_PASS};
|
||||
location /api {
|
||||
proxy_pass ${API_PROXY_PASS};
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_read_timeout 900s;
|
||||
proxy_send_timeout 900s;
|
||||
proxy_read_timeout 300s;
|
||||
proxy_send_timeout 300s;
|
||||
}
|
||||
|
||||
location ~* \.(?:js|css|png|jpg|jpeg|gif|ico|svg|woff|woff2|ttf)$ {
|
||||
@@ -34,8 +27,4 @@ server {
|
||||
expires 30d;
|
||||
add_header Cache-Control "public, immutable";
|
||||
}
|
||||
|
||||
location / {
|
||||
try_files $uri $uri/ /index.html;
|
||||
}
|
||||
}
|
||||
@@ -1,153 +0,0 @@
|
||||
# 2026-07-23 开发总结
|
||||
|
||||
本文档汇总 2026-07-23 当天围绕模型微调平台完成的系统开发内容。当前开发不再按临时 Demo 口径处理，以下能力均按后续可继续演进到生产系统的代码和数据结构推进。
|
||||
|
||||
## 1. 今日完成内容
|
||||
|
||||
### 1.1 模型训练最小闭环增强
|
||||
|
||||
| 功能点 | 作用 | 对应页面/模块 | 用户如何操作 |
|
||||
| --- | --- | --- | --- |
|
||||
| 训练任务创建前预检 | 在任务进入运行前校验模型、数据集、算力节点、路径和 LLaMA-Factory 参数，减少任务启动后才失败的情况 | 模型微调列表、创建模型微调、训练日志 | 在“模型微调”页面创建任务并提交，系统自动执行预检；失败时页面提示具体原因 |
|
||||
| 训练数据同步到算力节点 | 将应用侧选择的数据集文件同步到目标 Compute 节点本地磁盘，保证 LLaMA-Factory 能读取到真实训练文件 | 创建模型微调、训练日志、算力节点/资源副本 | 用户选择数据集后提交训练任务，系统自动同步数据集到算力节点，无需手工进入容器复制 |
|
||||
| 动态生成训练参数 | 根据用户在页面选择的训练数据、基座模型、训练方法、输出目录等动态生成 LLaMA-Factory 训练参数 | 创建模型微调、训练日志 | 用户在创建任务表单中选择模型和数据集，启动训练后可在训练日志中查看实际执行命令 |
|
||||
| 训练日志和状态轮询 | 应用侧主动轮询 Compute API 获取训练状态、进度、日志、指标和 checkpoint | 训练日志详情页 | 用户进入训练日志页，可查看任务状态、日志输出、指标和 checkpoint 信息 |
|
||||
| 训练失败信息回传 | 训练命令失败后将失败状态和日志内容回传应用侧，便于定位数据格式、模型路径或参数问题 | 模型微调列表、训练日志详情页 | 任务失败后，用户进入训练日志页查看失败日志和错误堆栈 |
|
||||
|
||||
### 1.2 B. 模型资产、训练与 LLaMA-Factory 任务能力
|
||||
|
||||
| 功能点 | 作用 | 对应页面/模块 | 用户如何操作 |
|
||||
| --- | --- | --- | --- |
|
||||
| 训练模型产物 artifact 展示 | 展示训练/合并产生的 Adapter、合并模型、量化模型等产物路径、大小、checksum | 模型管理 -> 训练模型列表 | 用户进入“模型管理”，切换到“训练模型”，展开某个模型行查看“模型产物” |
|
||||
| 模型血缘 lineage 展示 | 记录和展示基座模型、训练模型、合并产物之间的来源关系 | 模型管理 -> 训练模型列表 | 用户展开训练模型行，在“模型血缘”区域查看父资源、子资源和对应 Compute Job |
|
||||
| 模型导出任务 export job 状态展示 | 展示模型合并/导出的任务状态、输出目录和创建时间 | 模型管理 -> 训练模型列表 | 用户展开训练模型行，在“导出任务”区域查看导出任务状态 |
|
||||
| export 完成后回填 checksum | Compute export 完成后扫描产物文件，回填大小和 `checksum_sha256`，为后续一致性校验和制品治理做准备 | 后端模型资产模块、模型管理页面 | 用户无需手工操作，任务完成后页面自动展示产物大小和 checksum |
|
||||
| 合并模型任务接入 Compute | 模型合并不再只是生成模拟任务，而是通过 Compute 节点调用模型合并任务并记录导出任务 | 模型管理 -> 合并权重 | 用户在训练模型列表点击“合并权重”，提交后系统创建 Compute 合并任务 |
|
||||
|
||||
### 1.3 D. 算力平台、部署与运维能力
|
||||
|
||||
| 功能点 | 作用 | 对应页面/模块 | 用户如何操作 |
|
||||
| --- | --- | --- | --- |
|
||||
| 算力节点动态配置 | 支持维护 Compute API、File Gateway、权重、标签、启用状态、调度状态等信息 | 算力节点 -> 节点 | 用户进入“算力节点”，点击“新增节点”或“编辑”，填写节点地址和调度参数 |
|
||||
| 节点连通性测试 | 验证应用服务器是否能访问 Compute API，并返回 GPU 发现结果和延迟 | 算力节点 -> 节点 | 用户在节点列表点击“测试”，查看连接成功或失败提示 |
|
||||
| GPU 状态展示 | 展示每个算力节点的 GPU 利用率、显存、温度、功耗、进程信息 | 算力节点 -> GPU | 用户进入“算力节点”，切换到“GPU”页签查看 |
|
||||
| 训练队列展示 | 展示 Compute 侧任务队列、状态、进度、节点和 GPU 分配 | 算力节点 -> 队列 | 用户切换到“队列”页签查看当前运行或等待任务 |
|
||||
| 资源副本列表 | 展示数据集、模型、训练产物在算力节点本地磁盘上的副本路径和同步状态 | 算力节点 -> 资源副本 | 用户切换到“资源副本”页签，选择目标节点查看本地副本 |
|
||||
| 资源副本漂移检测 | 检查副本路径是否仍存在、大小是否可读取，并将异常副本标记为漂移 | 算力节点 -> 资源副本 | 用户点击“漂移检测”，系统调用 Compute API 检查节点本地路径 |
|
||||
| 资源副本 repair 重同步 | 对漂移、失败、待修复副本执行真实重同步，而不只是标记状态 | 算力节点 -> 资源副本 | 用户点击“修复副本”，系统从权威源重新上传或导入到目标算力节点 |
|
||||
| 应用侧轮询模式 | 应用服务主动轮询 Compute API，避免需要 Compute 侧反向访问应用侧 | 后端 Compute Gateway、部署架构 | 用户无感知，部署时只需保证应用侧能访问各 Compute 节点 |
|
||||
| 本地磁盘挂载约定 | 基座模型、训练数据、训练产物、训练日志均通过宿主机目录挂载到 Compute 容器 | docker/compute 部署 | 运维人员在启动 Compute 服务前准备并挂载 `docker/compute/data/yg-ft/*` 目录 |
|
||||
|
||||
### 1.4 前端状态中文化
|
||||
|
||||
| 功能点 | 作用 | 对应页面/模块 | 用户如何操作 |
|
||||
| --- | --- | --- | --- |
|
||||
| 统一状态字典 | 将 `pending`、`running`、`completed`、`failed`、`online`、`synced`、`drifted` 等状态统一展示为中文 | 通用前端组件 `ModelStatusTag`、状态工具 `status.ts` | 用户在各列表页看到中文状态，不再直接看到英文状态值 |
|
||||
| 模型管理状态中文化 | 合并状态、导出任务状态、产物状态使用中文展示 | 模型管理 | 用户查看训练模型列表和展开详情时看到“未合并/合并中/已合并/已完成/失败”等中文 |
|
||||
| 算力节点状态中文化 | 节点状态、GPU 状态、队列状态、副本同步状态使用中文展示 | 算力节点 | 用户查看“节点/GPU/队列/资源副本”时看到中文状态标签 |
|
||||
| 推理/对比/评测/用户状态中文化 | 将推理任务、对比模型加载、评测任务、用户状态统一为中文标签 | 模型推理、模型对比、模型评测、用户设置 | 用户查看相关页面时看到“启动中/已就绪/已完成/启用”等中文状态 |
|
||||
|
||||
### 1.5 前端构建产物更新
|
||||
|
||||
| 功能点 | 作用 | 对应页面/模块 | 用户如何操作 |
|
||||
| --- | --- | --- | --- |
|
||||
| 刷新 `frontend/dist` | 保证 Docker 前端 nginx 容器可以直接加载最新页面代码 | 前端部署 | 用户重新访问前端服务即可看到最新页面 |
|
||||
| 重启前后端服务验证 | 验证源码挂载和 dist 挂载方式下服务可正常加载最新代码 | docker/app、docker/compute | 开发人员重启 `backend-api`、`frontend`、`compute-api` 后验证页面和接口 |
|
||||
|
||||
## 2. 今日涉及的主要代码位置
|
||||
|
||||
| 文件 | 说明 |
|
||||
| --- | --- |
|
||||
| `backend/app/api/v1/endpoints/platform.py` | 新增/完善训练预检、资源同步、模型 artifact/lineage/export job 接口、资源副本漂移检测和 repair |
|
||||
| `backend/app/db/platform_store.py` | 新增模型产物、血缘、导出任务、副本同步结果、artifact 查询等数据访问能力 |
|
||||
| `compute/agent/process_manager.py` | export/artifact 扫描时补充文件大小和 checksum |
|
||||
| `frontend/src/api/modules/model.ts` | 增加训练模型 artifact、lineage、export job API |
|
||||
| `frontend/src/api/modules/compute.ts` | 增加副本漂移检测、repair API 和副本字段 |
|
||||
| `frontend/src/views/model/ModelManageView.vue` | 增加训练模型展开详情：产物、血缘、导出任务 |
|
||||
| `frontend/src/views/compute/ComputeNodesView.vue` | 增加资源副本页签、漂移检测、repair 操作和状态中文化 |
|
||||
| `frontend/src/utils/status.ts` | 新增统一状态中文映射和标签类型映射 |
|
||||
| `frontend/src/components/ModelStatusTag.vue` | 改为复用统一状态字典 |
|
||||
| `frontend/src/components/DataTablePage.vue` | 增加表格展开事件支持 |
|
||||
| `frontend/dist/*` | 前端生产构建产物已更新 |
|
||||
|
||||
## 3. 今日验证结果
|
||||
|
||||
| 验证项 | 结果 |
|
||||
| --- | --- |
|
||||
| Python 编译检查 | 通过：`python -m compileall backend compute scripts` |
|
||||
| 前端生产构建 | 通过：`npm run build` |
|
||||
| 前端入口 | 通过：`http://127.0.0.1:16801/` 返回 200 |
|
||||
| 后端健康检查 | 通过：`/modelTF/health` 返回正常 |
|
||||
| Compute 健康检查 | 通过：`/modelTF/health` 返回正常 |
|
||||
| 模型导出任务接口 | 通过：`/modelTF/model-manage/export-jobs` 返回正常 |
|
||||
| 算力节点列表接口 | 通过：`/modelTF/compute/nodes` 返回正常 |
|
||||
| 资源副本漂移检测 | 通过：当前节点副本检测返回 `drifted: 0` |
|
||||
|
||||
## 4. 当前仍需注意的问题
|
||||
|
||||
| 问题 | 影响 | 建议 |
|
||||
| --- | --- | --- |
|
||||
| 训练数据格式仍依赖 LLaMA-Factory 约定 | 如果用户上传的数据字段不符合模板要求，训练仍会失败 | 下一步增加数据集格式校验和模板转换预检 |
|
||||
| 单机 MX350 显存较小 | 真实训练大模型时容易因显存不足失败 | 当前环境用于链路验证；真实训练应使用高显存 GPU 节点 |
|
||||
| resource replica repair 对大型模型仍是同步调用 | 大模型重同步可能耗时较长 | 下一步将 repair 完整异步化，并展示 sync job 进度 |
|
||||
| artifact checksum 目前在 Compute 扫描阶段计算 | 对超大目录递归扫描可能较慢 | 下一步支持分文件 checksum、manifest 文件和后台扫描 |
|
||||
| 模型评测、模型推理仍未形成完整生产闭环 | 目前页面已有基础能力，但后端表结构、任务运行、日志、治理能力还需补齐 | 下一步将评测和推理纳入正式开发计划 |
|
||||
|
||||
## 5. 下一步开发计划
|
||||
|
||||
### 5.1 B. 模型资产、训练与 LLaMA-Factory 任务
|
||||
|
||||
| 优先级 | 开发任务 | 目标页面/模块 | 交付结果 |
|
||||
| --- | --- | --- | --- |
|
||||
| P0 | 数据集格式预检 | 创建模型微调、数据集管理 | 提交训练前校验 Alpaca/ShareGPT/OpenAI Messages 等格式，提示缺失字段 |
|
||||
| P0 | 训练参数可视化确认 | 创建模型微调 | 提交前展示最终 LLaMA-Factory 参数预览，减少参数不一致问题 |
|
||||
| P0 | 训练任务失败诊断 | 训练日志详情 | 识别常见错误：数据字段缺失、模型路径不存在、显存不足、依赖缺失，并生成中文诊断 |
|
||||
| P1 | checkpoint 管理 | 训练日志详情、模型管理 | 展示 checkpoint 列表、大小、路径、保留策略，支持标记最佳 checkpoint |
|
||||
| P1 | 模型产物 manifest | 模型管理 | 为每个训练/导出产物生成 manifest，记录文件清单、大小、checksum、来源任务 |
|
||||
| P1 | 合并/导出任务详情页 | 模型管理、训练日志 | 展示合并任务日志、状态、产物、失败原因 |
|
||||
| P2 | 模型版本治理 | 模型管理 | 支持版本号、标签、发布状态、归档状态、审批状态 |
|
||||
| P2 | 训练模板管理 | 系统设置或训练配置 | 将 LLaMA-Factory 模板、数据格式、默认超参做成可维护配置 |
|
||||
|
||||
### 5.2 D. 算力平台、部署与运维
|
||||
|
||||
| 优先级 | 开发任务 | 目标页面/模块 | 交付结果 |
|
||||
| --- | --- | --- | --- |
|
||||
| P0 | repair 异步化 | 算力节点 -> 资源副本 | repair 创建 sync job 后后台执行，页面展示进度和失败原因 |
|
||||
| P0 | 多算力节点调度策略 | 算力节点、创建模型微调 | 支持按标签、权重、空闲 GPU、显存要求选择节点 |
|
||||
| P1 | 节点资源水位告警 | 算力节点、硬件监控 | 展示磁盘、GPU、显存、训练日志目录水位和告警状态 |
|
||||
| P1 | 节点维护窗口 | 算力节点 | 支持维护中节点不再调度新任务，已有任务可继续或迁移 |
|
||||
| P1 | 文件副本治理 | 算力节点 -> 资源副本 | 支持副本清理、重建、过期策略和跨节点一致性检查 |
|
||||
| P2 | 部署健康巡检脚本 | 部署运维 | 一键检查 app、backend、redis、pg、compute、GPU、挂载目录、端口连通 |
|
||||
| P2 | Compute Agent 插件标准 | 算力平台 | 抽象 LLaMA-Factory 接入规范，预留其他训练框架 |
|
||||
|
||||
### 5.3 模型评测页面开发计划
|
||||
|
||||
| 优先级 | 开发任务 | 目标页面/模块 | 交付结果 |
|
||||
| --- | --- | --- | --- |
|
||||
| P0 | 评测任务表结构和接口补齐 | 模型评测列表、创建评测、评测详情 | 建立 `eval_tasks`、`eval_dimensions`、`eval_sample_results` 等运行表和接口 |
|
||||
| P0 | 创建评测任务真实提交 | 创建评测 | 支持选择模型、数据集、评测维度、GPU/节点，提交后生成评测任务 |
|
||||
| P0 | 评测任务运行闭环 | 评测详情 | 支持状态、进度、日志、样本级结果回传 |
|
||||
| P1 | 评测维度管理 | 评测维度创建/编辑 | 支持规则、Prompt、评分器、权重、适用数据集配置 |
|
||||
| P1 | 样本级评分展示 | 评测详情 | 展示每条样本的输入、模型输出、评分、原因、人工复核状态 |
|
||||
| P1 | 综合报告生成 | 评测详情、排行榜 | 生成维度汇总、综合分、问题样本、改进建议 |
|
||||
| P2 | 评测审批和审计 | 审批流、审计日志 | 评测任务创建、发布报告、删除报告纳入治理 |
|
||||
|
||||
### 5.4 模型推理页面开发计划
|
||||
|
||||
| 优先级 | 开发任务 | 目标页面/模块 | 交付结果 |
|
||||
| --- | --- | --- | --- |
|
||||
| P0 | 推理任务表结构和接口补齐 | 模型推理列表、新建推理 | 建立 `inference_tasks`、`inference_task_models`、`chat_sessions`、`chat_messages` |
|
||||
| P0 | 模型加载/卸载真实闭环 | 模型推理列表 | 支持选择训练产物加载推理服务，展示加载状态、端口、进程和错误 |
|
||||
| P0 | 单模型对话持久化 | 模型对话 | 保存会话、消息、参数、响应耗时、token 统计 |
|
||||
| P1 | 多模型对比任务 | 模型对比 | 支持多模型同时加载、同一问题并发请求、结果对比展示 |
|
||||
| P1 | 推理资源管控 | 模型推理、算力节点 | 支持 GPU 选择、并发限制、空闲自动卸载、异常进程清理 |
|
||||
| P1 | 推理日志和调用审计 | 日志、审计中心 | 记录加载、卸载、对话请求、失败原因、用户和租户信息 |
|
||||
| P2 | 推理服务发布 | 模型管理、模型推理 | 支持将某个训练模型发布为内部推理服务，并配置访问权限 |
|
||||
|
||||
## 6. 建议的下一阶段顺序
|
||||
|
||||
1. 先完成 B+D 的训练稳定性增强：数据格式预检、训练参数预览、失败诊断、repair 异步化。
|
||||
2. 再补齐模型评测的真实任务闭环：任务表、创建任务、运行状态、样本结果。
|
||||
3. 然后补齐模型推理闭环：加载/卸载、对话持久化、多模型对比。
|
||||
4. 最后统一治理能力：审批、审计、租户隔离、资源配额、保留策略和运维巡检。
|
||||
|
||||
@@ -1,281 +0,0 @@
|
||||
# 2026-07-24 工作计划
|
||||
|
||||
本文基于 `docs/2026-07-23-development-summary.md`、当前 B+D 开发进度，以及 2026-07-24 已完成的训练预检、失败诊断、资源修复异步化能力整理。后续开发仍按正式系统演进推进，不以临时演示能力作为交付标准。
|
||||
|
||||
## 1. 当前完成基线
|
||||
|
||||
### 1.1 训练创建与预检
|
||||
|
||||
- 对应页面：`模型微调 / 创建训练任务`
|
||||
- 已完成能力：
|
||||
- 创建训练前调用训练预检接口。
|
||||
- 展示真实 LLaMA-Factory 命令预览。
|
||||
- 展示预检节点、错误、警告和中文诊断建议。
|
||||
- 后端预检不创建任务、不落库、不占用调度锁。
|
||||
- 后端兼容 `base_model/train_dataset_id` 和 `model_id/dataset_id` 两套字段。
|
||||
- 对应接口：
|
||||
- `POST /modelTF/fine-tune/preflight`
|
||||
- `POST /modelTF/fine-tune/command-preview`
|
||||
|
||||
### 1.2 训练日志与失败诊断
|
||||
|
||||
- 对应页面：`系统日志 / 训练日志`
|
||||
- 已完成能力：
|
||||
- 训练失败或停止后可查询诊断建议。
|
||||
- 根据训练日志和失败原因识别模型路径、数据集字段、CUDA/GPU、LLaMA-Factory 命令等常见问题。
|
||||
- 页面以中文展示失败诊断。
|
||||
- 对应接口：
|
||||
- `GET /modelTF/fine-tune/{task_id}/diagnostics`
|
||||
|
||||
### 1.3 算力节点与资源修复
|
||||
|
||||
- 对应页面：`算力平台 / 算力节点`
|
||||
- 已完成能力：
|
||||
- 算力节点可配置、可测试连接、可查看 GPU 与健康状态。
|
||||
- 资源副本 repair 改为异步提交。
|
||||
- 页面展示资源同步任务进度，避免长请求阻塞页面。
|
||||
- 对应接口：
|
||||
- `POST /modelTF/compute/nodes/{node_id}/replicas/repair`
|
||||
- `GET /modelTF/compute/sync-jobs/{sync_id}`
|
||||
|
||||
## 2. 未完成任务清单
|
||||
|
||||
### P0：真实训练成功闭环
|
||||
|
||||
- 对应页面：
|
||||
- `模型管理 / 新增模型`
|
||||
- `数据集管理 / 上传数据集`
|
||||
- `模型微调 / 创建训练任务`
|
||||
- `系统日志 / 训练日志`
|
||||
- 未完成内容：
|
||||
- 基座模型必须支持算力服务器本地路径校验，避免选择 API 模型或应用侧路径后进入训练。
|
||||
- 数据集必须支持上传后格式校验，提前发现 Alpaca、ShareGPT、DPO、CPT 字段不匹配问题。
|
||||
- 训练成功后需要完成模型产物扫描、产物入库、训练任务状态回填。
|
||||
- 训练失败时需要强制拉取最后日志片段，保证失败原因可见。
|
||||
- 后端开发：
|
||||
- 增强模型路径校验，明确区分 `本地训练模型`、`API 模型`、`已训练模型`。
|
||||
- 增加数据集格式校验服务，支持字段级错误返回。
|
||||
- 完善训练任务完成后的 artifact 回填逻辑。
|
||||
- Compute 开发：
|
||||
- 训练结束后扫描输出目录。
|
||||
- 返回产物列表、文件大小、目录结构、训练日志路径。
|
||||
- 验收标准：
|
||||
- 使用算力节点可访问的本地模型路径和合法数据集，可以完成一次真实 LLaMA-Factory 训练。
|
||||
- 训练成功后页面能看到完成状态、输出目录、模型产物。
|
||||
- 训练失败时页面能看到中文诊断和最后错误日志。
|
||||
|
||||
### P0：LLaMA-Factory 参数映射完善
|
||||
|
||||
- 对应页面：`模型微调 / 创建训练任务`
|
||||
- 未完成内容：
|
||||
- SFT、DPO、CPT 参数映射仍需细化。
|
||||
- LoRA、Full、QLoRA、量化导出参数需要按训练方式校验。
|
||||
- 不同模板与数据集格式之间的兼容关系需要预检。
|
||||
- 后端开发：
|
||||
- 建立训练参数标准化层。
|
||||
- 建立训练方式到 LLaMA-Factory 参数的映射表。
|
||||
- 对无效组合返回中文错误，例如 DPO 缺少 rejected 字段、CPT 不应使用 instruction/output 格式等。
|
||||
- 验收标准：
|
||||
- 页面选择不同训练方式时，预检能返回准确命令。
|
||||
- 无效参数组合不能启动训练。
|
||||
|
||||
### P1：资源副本 repair 自动重同步
|
||||
|
||||
- 对应页面：`算力平台 / 算力节点`
|
||||
- 未完成内容：
|
||||
- repair 当前已异步化，但还需要基于权威源路径自动重同步。
|
||||
- 修复完成后需要重新校验副本状态。
|
||||
- 后端开发：
|
||||
- 为模型、数据集、训练产物定义权威源路径。
|
||||
- repair job 根据权威源自动发起重传或重新扫描。
|
||||
- 修复完成后更新 replica 状态、checksum、错误原因。
|
||||
- Compute 开发：
|
||||
- 支持接收重同步请求。
|
||||
- 支持按资源类型写入目标路径并返回校验信息。
|
||||
- 验收标准：
|
||||
- 将副本标记为异常后，点击修复可自动完成重同步并恢复为正常。
|
||||
|
||||
### P1：artifact checksum 与 manifest
|
||||
|
||||
- 对应页面：
|
||||
- `模型管理 / 已训练模型`
|
||||
- `模型管理 / 模型详情`
|
||||
- `算力平台 / 资源副本`
|
||||
- 未完成内容：
|
||||
- artifact checksum 目前仍是预留字段。
|
||||
- 大目录需要 manifest 文件，避免每次递归扫描成本过高。
|
||||
- 后端开发：
|
||||
- 增加 artifact checksum 回填逻辑。
|
||||
- 增加 manifest 解析和存储字段。
|
||||
- Compute 开发：
|
||||
- export 或训练完成后扫描文件并生成 checksum。
|
||||
- 对大模型目录生成 manifest。
|
||||
- 验收标准：
|
||||
- 模型产物列表能展示 checksum、大小、文件数、生成时间。
|
||||
- 副本校验可以基于 checksum 判断一致性。
|
||||
|
||||
### P1：模型导出闭环
|
||||
|
||||
- 对应页面：
|
||||
- `模型管理 / 已训练模型`
|
||||
- `模型管理 / 导出任务`
|
||||
- `模型管理 / 合并权重`
|
||||
- 未完成内容：
|
||||
- 导出任务、量化导出、导出日志、失败重试、产物下载仍需完善。
|
||||
- 后端开发：
|
||||
- 完善 export job 创建、查询、取消、重试接口。
|
||||
- 导出完成后登记 artifact 和 lineage。
|
||||
- Compute 开发：
|
||||
- 支持 LoRA 合并、GGUF/量化导出、导出日志回传。
|
||||
- 验收标准：
|
||||
- 已训练模型可发起导出。
|
||||
- 导出状态、日志、产物可在页面查看。
|
||||
|
||||
### P1：多算力节点调度增强
|
||||
|
||||
- 对应页面：`算力平台 / 算力节点`
|
||||
- 未完成内容：
|
||||
- 当前已有节点配置和基础调度，但生产级调度策略仍需增强。
|
||||
- 需要支持节点标签、权重、启用状态、容量、手动指定节点。
|
||||
- 后端开发：
|
||||
- 增强调度策略：标签匹配、权重、当前任务数、GPU 占用、显存约束。
|
||||
- 增加任务排队和等待原因。
|
||||
- 增加 GPU 分配释放的异常恢复。
|
||||
- 前端开发：
|
||||
- 创建训练时支持可选手动指定节点。
|
||||
- 算力节点页展示容量、排队数、当前任务。
|
||||
- 验收标准：
|
||||
- 多节点时可以自动选择合适节点。
|
||||
- 节点不可用时页面能明确展示不可调度原因。
|
||||
|
||||
### P2：训练日志实时性优化
|
||||
|
||||
- 对应页面：`系统日志 / 训练日志`
|
||||
- 未完成内容：
|
||||
- 当前依赖应用侧轮询，日志实时性和失败最后日志仍需增强。
|
||||
- 后端开发：
|
||||
- 支持日志 offset/tail 增量读取。
|
||||
- 任务失败时强制同步最后日志片段。
|
||||
- 日志接口返回来源、偏移量、是否截断。
|
||||
- 前端开发：
|
||||
- 日志页按 offset 增量刷新。
|
||||
- 失败时自动跳到底部并展示最后错误。
|
||||
- 验收标准：
|
||||
- 训练过程中日志持续刷新。
|
||||
- 失败后无需手动刷新即可看到最后错误。
|
||||
|
||||
### P2：权限、审计和治理落点补齐
|
||||
|
||||
- 对应页面：
|
||||
- `用户中心`
|
||||
- `项目管理`
|
||||
- `模型管理`
|
||||
- `数据集管理`
|
||||
- `模型微调`
|
||||
- `算力平台`
|
||||
- 未完成内容：
|
||||
- 训练链路中的租户、项目、用户权限校验还需要细粒度补齐。
|
||||
- 审计事件需要覆盖训练创建、启动、停止、删除、导出、资源修复。
|
||||
- 后端开发：
|
||||
- 接口增加项目/租户上下文校验。
|
||||
- 增加审计事件写入。
|
||||
- 删除和高风险操作进入审批流。
|
||||
- 验收标准：
|
||||
- 用户只能访问授权项目内的模型、数据集和训练任务。
|
||||
- 关键操作可以在审计日志中查询。
|
||||
|
||||
### P2：模型评测页面真实闭环
|
||||
|
||||
- 对应页面：`模型评测`
|
||||
- 未完成内容：
|
||||
- 评测任务创建、运行、日志、指标、结果对比仍需接入真实后端。
|
||||
- 后端开发：
|
||||
- 评测任务表、评测指标表、评测日志接口。
|
||||
- 支持指定模型、数据集、评测模板和指标。
|
||||
- Compute 开发：
|
||||
- 支持评测任务执行器。
|
||||
- 返回指标结果和日志。
|
||||
- 验收标准：
|
||||
- 可创建评测任务并看到运行状态、指标结果和失败原因。
|
||||
|
||||
### P2：模型推理页面真实闭环
|
||||
|
||||
- 对应页面：
|
||||
- `模型推理 / 推理服务`
|
||||
- `模型推理 / 对话测试`
|
||||
- `模型对比`
|
||||
- 未完成内容：
|
||||
- 推理服务启动、停止、健康检查、会话请求、资源释放仍需完善。
|
||||
- 后端开发：
|
||||
- 推理服务实例管理接口。
|
||||
- 对话请求代理接口。
|
||||
- 推理日志和资源占用查询。
|
||||
- Compute 开发：
|
||||
- 支持启动本地模型推理服务。
|
||||
- 支持停止服务和释放 GPU。
|
||||
- 验收标准：
|
||||
- 可从页面启动一个已训练模型的推理服务。
|
||||
- 可进行对话测试并查看服务状态。
|
||||
|
||||
## 3. 推荐开发顺序
|
||||
|
||||
1. 完成真实训练成功闭环。
|
||||
2. 完成数据集格式校验和 LLaMA-Factory 参数映射。
|
||||
3. 完成 artifact、checksum、manifest 和模型导出闭环。
|
||||
4. 完成资源副本 repair 自动重同步。
|
||||
5. 完成多算力节点调度增强。
|
||||
6. 完成训练日志实时性优化。
|
||||
7. 补齐权限、审批、审计治理落点。
|
||||
8. 启动模型评测真实闭环开发。
|
||||
9. 启动模型推理真实闭环开发。
|
||||
|
||||
## 4. 下一轮优先执行任务
|
||||
|
||||
### 任务 1：训练模型路径治理
|
||||
|
||||
- 页面：`模型管理 / 新增模型`、`模型微调 / 创建训练任务`
|
||||
- 内容：
|
||||
- 新增模型时区分是否可用于训练。
|
||||
- API 模型不能作为 LLaMA-Factory 本地训练基座。
|
||||
- 本地模型路径必须是算力节点可访问路径。
|
||||
- 验收：
|
||||
- 选择不可训练模型时，训练创建页预检直接给出中文错误。
|
||||
|
||||
### 任务 2：数据集格式校验
|
||||
|
||||
- 页面：`数据集管理 / 上传数据集`、`模型微调 / 创建训练任务`
|
||||
- 内容：
|
||||
- 上传后扫描样本字段。
|
||||
- 支持 Alpaca、ShareGPT、DPO、CPT 校验。
|
||||
- 返回字段缺失、类型错误、空样本等问题。
|
||||
- 验收：
|
||||
- `111.json` 这类数据可以明确判断是否满足当前训练模板。
|
||||
|
||||
### 任务 3：训练完成产物入库
|
||||
|
||||
- 页面：`模型微调 / 任务列表`、`模型管理 / 已训练模型`
|
||||
- 内容：
|
||||
- Compute 训练成功后返回输出目录。
|
||||
- 应用侧轮询后创建 trained model 记录。
|
||||
- 写入 artifact、lineage、export job 初始状态。
|
||||
- 验收：
|
||||
- 训练完成后无需手动登记，模型管理中自动出现新模型。
|
||||
|
||||
### 任务 4：失败日志最后片段拉取
|
||||
|
||||
- 页面：`系统日志 / 训练日志`
|
||||
- 内容：
|
||||
- 任务失败时立即拉取最后 N 行日志。
|
||||
- 页面展示最后错误、诊断建议和原始日志。
|
||||
- 验收：
|
||||
- 训练失败后页面不再只看到“失败”，可以直接看到失败原因。
|
||||
|
||||
## 5. 当前测试注意事项
|
||||
|
||||
- 当前环境中已有算力节点可访问，但 GPU 为 2GB 显存，预检会提示显存不足，这是符合预期的生产校验结果。
|
||||
- 若要验证真实训练成功，需要提前准备：
|
||||
- 算力节点可访问的本地基座模型目录。
|
||||
- 合法训练数据集文件。
|
||||
- 足够显存的 GPU。
|
||||
- Compute 容器内可用的 LLaMA-Factory 和 `llamafactory-cli`。
|
||||
- 当前前端 `dist` 已按要求参与构建更新，后续修改前端页面后需要重新执行 `npm run build`。
|
||||
File diff suppressed because it is too large
Load Diff
@@ -43,7 +43,7 @@ LOG_RETENTION_DAYS=10
|
||||
"@timestamp": "2026-07-16T13:20:10.123",
|
||||
"level": "INFO",
|
||||
"logger": "app.access",
|
||||
"message": "request completed method=GET path=/modelTF/health status_code=200 duration_ms=3.12 client=127.0.0.1",
|
||||
"message": "request completed method=GET path=/api/v1/health status_code=200 duration_ms=3.12 client=127.0.0.1",
|
||||
"module": "logging",
|
||||
"function": "request_logging_middleware",
|
||||
"file": "D:\\AI\\codex-code\\YG_FT\\backend\\app\\core\\logging.py",
|
||||
|
||||
@@ -1,279 +0,0 @@
|
||||
# 数据处理接口与算法设计
|
||||
|
||||
本文是 `team-development-plan.md` 板块 C 的落地契约，约束
|
||||
`/modelTF/data-process/*`、前端数据处理向导以及 PostgreSQL 数据模型。
|
||||
|
||||
## 1. 处理闭环
|
||||
|
||||
```text
|
||||
创建草稿任务
|
||||
→ 上传并登记源文件（格式、SHA-256、版本）
|
||||
→ 预处理（标准化、无效过滤、去重、可选脱敏）
|
||||
→ 构建可编辑预览（来源偏移与行号）
|
||||
→ 生成标准训练记录
|
||||
→ 质量评分与稳定数据集划分
|
||||
→ 人工编辑/恢复
|
||||
→ 幂等发布为数据集（保留完整来源链路）
|
||||
```
|
||||
|
||||
任务只使用以下五种状态：
|
||||
|
||||
```text
|
||||
pending ──start/generate──> running ──success──> completed
|
||||
▲ │ ├──error───────> failed
|
||||
│ │ └──stop────────> stopped
|
||||
└────────retry────────────┴────────retry─────┘
|
||||
```
|
||||
|
||||
- `pending` 允许修改配置、增删源文件和重建预览。
|
||||
- `running` 拒绝重复启动、修改配置和删除任务。
|
||||
- `failed`、`stopped` 可重试；重试前清理上一次未完成结果。
|
||||
- `completed` 可编辑结果和发布；重复发布返回同一个数据集。
|
||||
- 非法状态转换返回 HTTP 409。
|
||||
- 每次生成分配独立 `generation_run_id`；停止或重试会使旧代次立即失效，
|
||||
旧后台任务不能覆盖新代次的结果或状态。
|
||||
|
||||
## 2. 接口契约
|
||||
|
||||
所有路径由请求层统一添加 `/modelTF`，响应统一为
|
||||
`{ "code": 0, "message": "ok", "data": ... }`。
|
||||
|
||||
### 任务与进度
|
||||
|
||||
| 方法 | 路径 | 说明 |
|
||||
| --- | --- | --- |
|
||||
| GET | `/data-process` | 分页查询任务，支持 keyword/status/process_type |
|
||||
| POST | `/data-process` | 创建 `pending` 草稿 |
|
||||
| GET | `/data-process/{id}` | 查询任务详情，不内嵌全部结果 |
|
||||
| PUT | `/data-process/{id}` | 更新草稿配置 |
|
||||
| DELETE | `/data-process/{id}` | 软删除非运行任务 |
|
||||
| POST | `/data-process/{id}/start` | 重建预览并生成的一键编排入口 |
|
||||
| POST | `/data-process/{id}/generate` | 使用已确认预览生成结果 |
|
||||
| POST | `/data-process/{id}/stop` | 请求停止运行任务 |
|
||||
| GET | `/data-process/{id}/progress` | 查询阶段、进度与计数 |
|
||||
|
||||
### 源文件与预览
|
||||
|
||||
| 方法 | 路径 | 说明 |
|
||||
| --- | --- | --- |
|
||||
| POST | `/data-process/{id}/source-files` | multipart 上传，字段名 `files` |
|
||||
| DELETE | `/data-process/{id}/source-files/{file_id}` | 删除源文件及其预览 |
|
||||
| GET | `/data-process/{id}/source-files/{file_id}/content` | 按行窗口读取源文 |
|
||||
| POST | `/data-process/{id}/preview/build` | 后端预处理并重建预览 |
|
||||
| GET | `/data-process/{id}/preview` | 分页查询预览 |
|
||||
| POST | `/data-process/{id}/preview` | 手工增加预览条目 |
|
||||
| PUT | `/data-process/{id}/preview/{preview_id}` | 保存人工编辑 |
|
||||
| DELETE | `/data-process/{id}/preview/{preview_id}` | 删除预览条目 |
|
||||
|
||||
上传批次先全部完成有界读取和解析，再在单个事务中登记；任一文件为空、超限、
|
||||
重复或格式非法时整批不落库，暂存原件也会一并清理。响应不回传整个文件，只返回
|
||||
逻辑对象引用、文件 ID、格式、原始字节数、记录数和原始 SHA-256。二进制文档必须
|
||||
由对应解析器显式处理；不支持的格式返回 415，绝不能静默替换成示例正文。
|
||||
|
||||
原始上传字节与解析正文采用双层存储：原件默认保存在
|
||||
`backend/storage/data-process/<task_id>/<file_id>/v<version>/<安全文件名>`，数据库的
|
||||
`storage_object_id` 只保存 `local://data-process/...` 逻辑引用，不保存或返回宿主机
|
||||
绝对路径；完整解析正文继续保存在 `data_process_source_files.content`，列表摘要使用
|
||||
`content_preview`，因此 PDF、Office 等文件的预览无需反复解析原始二进制。可通过
|
||||
`DATA_PROCESS_STORAGE_DIR` 指定其他本地根目录；从 `start.sh` 启动时，该变量应在
|
||||
当前终端导出。历史 `db://data-process/...` 记录继续从数据库正文预览。
|
||||
|
||||
单独删除源文件时先提交数据库软删除，再立即删除受控目录中的原件；若物理删除
|
||||
失败，接口仍按数据库结果返回成功并标记 `storage_cleanup_pending=true`，软删除记录
|
||||
中的逻辑引用可供运维补偿清理。任务软删除以及修改 `process_type` 导致的源文件
|
||||
软删除按留存数据处理，当前版本不自动物理清除。
|
||||
|
||||
### 结果与发布
|
||||
|
||||
| 方法 | 路径 | 说明 |
|
||||
| --- | --- | --- |
|
||||
| GET | `/data-process/{id}/results` | 分页查询，支持 keyword/status/split |
|
||||
| PUT | `/data-process/{id}/results/{result_id}` | 保存人工编辑并重评分 |
|
||||
| POST | `/data-process/{id}/results/{result_id}/restore` | 恢复生成时的原值 |
|
||||
| POST | `/data-process/{id}/publish` | 幂等发布为数据集 |
|
||||
|
||||
## 3. 配置校验
|
||||
|
||||
- `process_type`：`structured | unstructured | external`。
|
||||
- 数据集划分的 `train + validation + test` 必须等于 100，各项为 0～100。
|
||||
- `chunk_size` 为 16～32768 token；`chunk_overlap` 必须小于
|
||||
`chunk_size`；`min_chunk_size` 不得大于 `chunk_size`。
|
||||
- `temperature` 为 0～2，`max_tokens` 为 1～32768。
|
||||
- 任务名称在未删除任务中唯一。
|
||||
- 选择 `generation_model_id` 后，启动生成时校验模型是否存在，并保存不含密钥的
|
||||
模型版本快照。
|
||||
- 当前运行库沿用平台现有的单租户模式，不接受客户端提交 tenant/owner/operator
|
||||
字段，避免伪造隔离上下文；接入平台可信认证上下文后再启用数据库中预留的
|
||||
tenant/project 字段。
|
||||
|
||||
## 4. 格式解析与标准化
|
||||
|
||||
上传格式按处理类型约束：
|
||||
|
||||
- 结构化数据支持 JSON、JSONL/NDJSON、CSV/TSV 和 XLSX。XLSX 能识别纵向、
|
||||
横向合并单元格组成的多级表头，并稳定展平为 `销售.Q1` 一类字段；公式只读取
|
||||
文件中已缓存的计算结果，不在服务端执行。
|
||||
- 非结构化数据支持 UTF-8/UTF-8 BOM 的 TXT、Markdown、JSON/JSONL，以及
|
||||
文本型 PDF、DOCX 和 PPTX。PDF 按页抽取文本，DOCX 抽取段落与表格，PPTX
|
||||
抽取幻灯片文本与表格，随后统一进入切片算法。
|
||||
- 旧版二进制 DOC、XLS、PPT 不直接解析，返回 415 并提示分别转换为
|
||||
DOCX、XLSX、PPTX。
|
||||
- 扫描 PDF 没有文本层时明确提示需要 OCR；当前流程不执行 OCR。加密、损坏或
|
||||
超出页数/工作表/行列/解压规模限制的文件整批拒绝。
|
||||
|
||||
现代 Office 文件在交给解析库前检查 ZIP 成员路径、重复成员、加密标记、活动
|
||||
XML、单成员大小、总解压大小和压缩比，避免路径穿越、实体扩展与 ZIP bomb。
|
||||
|
||||
结构化选项按固定顺序执行，关闭某项时不会隐式执行对应业务变换：
|
||||
|
||||
1. `detect_structure`：展平嵌套对象；XLSX 上传解析阶段识别合并单元格和多级表头。
|
||||
2. `normalize_format`：字段名转 snake_case，执行 Unicode NFKC、换行和容器值规范化，
|
||||
输出键顺序稳定的 canonical JSON；账号、邮编等字符串不会转成数值。
|
||||
3. `clean_invalid`：删除全空列和全空记录；存在 `id/uuid/key/code/*_id` 身份字段时，
|
||||
删除身份字段残缺的行，但不会因备注等可选字段为空误删有效记录。
|
||||
4. `filter_anomaly`：仅对不少于 8 个样本的非身份数值字段使用 Tukey IQR 过滤离群行，
|
||||
同时过滤明确乱码、不可打印或极端超长文本；小样本和 ID 字段不参与统计过滤。
|
||||
5. `deduplicate`：先按整行 canonical JSON 精确去重，再按非空
|
||||
`id/uuid/key/code/*_id` 字段稳定保留首条；空关键值互不视为重复。
|
||||
6. `desensitize`：对结构化姓名字段和正文中的高置信上下文姓名、邮箱、手机号、
|
||||
身份证号进行不可逆掩码，并分别记录命中数。
|
||||
|
||||
非结构化“智能预处理”由六个可独立执行的底层选项组成：
|
||||
|
||||
- `clean_invalid_content` 删除确定为空、不可读或纯重复符号的无效块。
|
||||
- `detect_document_structure` 识别 Markdown、中文章节和数字标题，切片不跨章节，
|
||||
并在预览质量详情中保存 `heading_path`。
|
||||
- `merge_short_content` 在同一章节中合并短块，合并后不突破 `chunk_size`。
|
||||
- `filter_low_quality` 在生成前过滤乱码、不可打印、重复或极端超长内容。
|
||||
- `deduplicate_content` 先精确去重，再对足够长的内容进行保守近重复判断；数字或
|
||||
否定含义变化时始终保留。
|
||||
- `preserve_context` 才启用相邻切片 overlap；关闭时切片不共享正文上下文，且上下文
|
||||
永不跨文件或章节。
|
||||
|
||||
表格、围栏代码块和连续列表保护是三个独立参数。启用时切点避开相应 Markdown
|
||||
块，关闭时允许按正常长度切分。
|
||||
|
||||
脱敏是不可逆掩码：
|
||||
|
||||
- 邮箱：`[EMAIL]`
|
||||
- 中国大陆手机号：`[PHONE]`
|
||||
- 18 位身份证号：`[ID_CARD]`
|
||||
- 高置信姓名：`[NAME]`
|
||||
|
||||
源文件原文与脱敏后的预览分开保存，结果不得反向覆盖源文件。
|
||||
|
||||
## 5. 切片算法
|
||||
|
||||
首阶段只提供三种切片策略：
|
||||
|
||||
- `structure` 先识别 Markdown、中文章节及编号标题，再由 LlamaIndex
|
||||
`SentenceSplitter` 在章节内按段落和中英文句界限长；章节之间不共享 overlap。
|
||||
- `fixed` 使用 LlamaIndex `TokenTextSplitter` 按目标 token 窗口切分。
|
||||
- `custom` 使用用户给定分隔符，在找不到合适分隔点时回退到固定窗口。
|
||||
|
||||
不提供 `semantic` 和旧 `heading` 配置；创建或更新任务时传入这些值会直接拒绝。
|
||||
LlamaIndex 只负责通用切分，原文 offset、行号、标题路径和 Markdown 保护块仍由
|
||||
项目适配层统一维护。
|
||||
|
||||
首版使用可替换的确定性 token 估算器，中文字符、标点和英文词分别计数；
|
||||
所有偏移以 Python/JavaScript 都能稳定表达的 Unicode 文本偏移为准。
|
||||
|
||||
算法必须满足：
|
||||
|
||||
- 每轮游标严格前进，异常分隔符不能产生死循环。
|
||||
- overlap 是最大重叠量，尾部过短切片合并到上一片。
|
||||
- 代码块、Markdown 表格和连续列表在启用保护时不从中间切开。
|
||||
- 每个预览条目记录 `source_file_id`、字符偏移、起止行、token 数和算法版本。
|
||||
|
||||
## 6. 生成与质量评分
|
||||
|
||||
结构化记录优先识别以下字段：
|
||||
|
||||
1. `instruction/input/output`
|
||||
2. `question/context/answer`
|
||||
3. `prompt/input/response`
|
||||
|
||||
已有标准字段时只做标准化；需要语义生成时调用所选模型的 OpenAI 兼容接口，
|
||||
并固化模型 ID、模型版本、prompt、temperature、max_tokens 和 JSON mode 快照。
|
||||
模型地址可输入域名、`/v1` 基础地址或完整地址：例如输入
|
||||
`www.caoxiaozhu.com` 会规范为
|
||||
`https://www.caoxiaozhu.com/v1/chat/completions`，无需用户手工拼接路径。
|
||||
单条失败记录为 `invalid`，有限重试耗尽后继续处理下一条，避免整批丢失。
|
||||
|
||||
每条结果总分为 0～100：
|
||||
|
||||
```text
|
||||
总分 = 完整性 35% + 长度合理性 20% + 可读性 20%
|
||||
+ 来源相关性 15% + 非重复性 10%
|
||||
```
|
||||
|
||||
- instruction 或 output 为空时格式硬失败并标记 `invalid`。
|
||||
- 开启短文本过滤且 output 低于 `min_output_length` 时标记过滤原因。
|
||||
- 评分详情、命中规则与过滤原因必须落库并返回前端，不只返回一个总分。
|
||||
|
||||
## 7. 稳定划分
|
||||
|
||||
划分不能依赖结果插入顺序。对每条记录计算：
|
||||
|
||||
```text
|
||||
bucket = SHA256(task_id + ":" + result_id) mod 10000
|
||||
```
|
||||
|
||||
按万分位阈值映射为 `train/validation/test`。同一任务重试、分页或进程重启后，
|
||||
同一结果仍落入相同 split。
|
||||
|
||||
## 8. 发布与来源链路
|
||||
|
||||
发布在一个数据库事务中完成：
|
||||
|
||||
```text
|
||||
source_file
|
||||
→ data_process_task
|
||||
→ data_process_result
|
||||
→ dataset
|
||||
→ dataset_file + dataset_file_version
|
||||
→ dataset_record
|
||||
```
|
||||
|
||||
只发布 `valid/modified` 且满足质量门槛的结果。输出 JSONL 先计算 checksum，
|
||||
再登记文件版本和记录。发布请求中的 split 会重新进行稳定划分。任务的
|
||||
`output_dataset_id` 是幂等键；重复调用返回已有数据集，目标数据集若已被外部
|
||||
删除则解除断链并重新发布。当前运行库只开放 `local` 存储类型，正文保存在
|
||||
当前平台的 `dataset_files.content`，不虚假宣称已上传 MinIO 或云存储。
|
||||
|
||||
## 9. 安全边界
|
||||
|
||||
- 文件名只保留 basename，响应不返回宿主机绝对路径。
|
||||
- 上传限制单文件、批次文件数与批次总大小，解析采用有界读取。
|
||||
- 外部数据源凭据不写日志、不进入 localStorage、不在详情接口回显。
|
||||
- 外部 PostgreSQL 只允许单条 `SELECT/WITH`、只读事务、5 秒连接超时、
|
||||
30 秒语句超时和 50 MiB 响应上限；默认阻止回环、链路本地及私网地址。
|
||||
可信内网部署必须显式设置 `DATA_PROCESS_ALLOW_PRIVATE_EXTERNAL_DB=true`。
|
||||
- SQL 迁移独立存放，应用启动不会隐式修改当前远程数据库。
|
||||
|
||||
## 10. 迁移边界
|
||||
|
||||
`backend/app/db/sql/002_data_process.sql` 只面向当前运行脚本
|
||||
`001_platform_runtime.sql` 的 TEXT/最小表模型。它会在执行前检查
|
||||
`datasets.id` 类型；若检测到 `docs/postgres-schema.sql` 的 UUID/JSONB 目标模型，
|
||||
会直接失败而不是进行一半成功、一半失败的危险迁移。目标模型后续应由独立
|
||||
Alembic 迁移和对应存储实现承接。
|
||||
|
||||
`DataProcessStore.ensure_schema()` 仅供受控管理命令显式调用，API 路由和应用启动
|
||||
均不会自动执行该迁移。本次开发和测试没有修改任何远程数据库。
|
||||
|
||||
在已加载 `DATABASE_URL` 的终端中可先只读检查：
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
.venv/bin/python -m app.modules.data_process.schema_cli --check
|
||||
```
|
||||
|
||||
确认目标主机和数据库名称无误后，才显式执行：
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
.venv/bin/python -m app.modules.data_process.schema_cli --apply --yes
|
||||
```
|
||||
|
||||
命令输出只显示主机、端口和数据库名，不显示用户名或密码。
|
||||
@@ -63,7 +63,7 @@
|
||||
|
||||
### 4.1 适用场景
|
||||
|
||||
- 开发联调、单机试运行、资源受限的早期上线环境。
|
||||
- PoC、试点环境、演示环境。
|
||||
- 小团队共用一台单机多 GPU 服务器。
|
||||
- 网络隔离要求不高，部署资源有限。
|
||||
|
||||
@@ -111,11 +111,11 @@ flowchart LR
|
||||
| 服务 | 端口 | 暴露范围 |
|
||||
| --- | --- | --- |
|
||||
| Nginx | 80/443 | 用户网段 |
|
||||
| Backend API | 17861 | 仅 Nginx、本机 |
|
||||
| Compute API | 19100 | 仅 Backend API、本机 |
|
||||
| File Gateway | 19101 | 仅 Backend API、本机 |
|
||||
| PostgreSQL | 15432 | 本机或内网 |
|
||||
| Redis | 16379 | 本机或内网 |
|
||||
| Backend API | 8000 | 仅 Nginx、本机 |
|
||||
| Compute API | 9100 | 仅 Backend API、本机 |
|
||||
| File Gateway | 9101 | 仅 Backend API、本机 |
|
||||
| PostgreSQL | 5432 | 本机或内网 |
|
||||
| Redis | 6379 | 本机或内网 |
|
||||
|
||||
## 5. 方案二：应用服务与算力/训练服务独立部署
|
||||
|
||||
@@ -139,7 +139,7 @@ flowchart LR
|
||||
A --> L["LLaMA-Factory"]
|
||||
A --> G["GPU/CUDA"]
|
||||
A --> FS["算力服务器本地磁盘"]
|
||||
B -- "定时轮询任务状态/指标/产物索引" --> C
|
||||
A -- "状态回调/日志摘要" --> B
|
||||
```
|
||||
|
||||
### 5.3 部署边界
|
||||
@@ -170,8 +170,8 @@ GPU 算力服务器部署：
|
||||
- 协议：内部 HTTPS REST，后续可扩展 gRPC。
|
||||
- 鉴权：服务间 Token，生产建议 mTLS + IP 白名单。
|
||||
- 幂等：训练任务提交使用 `Idempotency-Key` 或 `job_id`。
|
||||
- 状态同步：默认由应用平台定时轮询 Compute API，拉取任务状态、指标摘要和产物索引。
|
||||
- 回调策略：第一阶段关闭算力侧回调，避免算力服务器访问应用服务器，减少双向网络策略开通。
|
||||
- 回调：算力平台向应用平台回调任务状态、指标摘要、产物索引。
|
||||
- 拉取：应用平台也可以定时轮询 Compute API，避免回调失败导致状态丢失。
|
||||
|
||||
文件互通：
|
||||
|
||||
@@ -196,60 +196,9 @@ GPU 算力服务器部署：
|
||||
### 5.6 风险
|
||||
|
||||
- 文件传输链路比单机部署复杂。
|
||||
- 需要处理跨服务器网络失败、轮询延迟、任务状态对账。
|
||||
- 需要处理跨服务器网络失败、回调失败、任务状态对账。
|
||||
- 需要明确模型、数据集、产物在应用侧和算力侧的索引关系。
|
||||
|
||||
### 5.7 多算力节点部署约定
|
||||
|
||||
多算力节点阶段仍然按“单机多 GPU 节点”部署，每台 GPU 服务器都是一个独立算力节点。每个参与调度的节点都必须部署：
|
||||
|
||||
- Compute API。
|
||||
- Compute Agent。
|
||||
- File Gateway。
|
||||
- LLaMA-Factory 宿主机目录和训练依赖。
|
||||
- CUDA、NVIDIA Driver、NCCL、PyTorch。
|
||||
- 本地数据盘 `/data/yg-ft`。
|
||||
- 本地日志和训练产物目录。
|
||||
|
||||
网络策略保持单向：
|
||||
|
||||
```text
|
||||
应用服务器 -> 算力节点 A Compute API/File Gateway
|
||||
应用服务器 -> 算力节点 B Compute API/File Gateway
|
||||
应用服务器 -> 算力节点 C Compute API/File Gateway
|
||||
```
|
||||
|
||||
默认不要求：
|
||||
|
||||
```text
|
||||
算力节点 -> 应用服务器
|
||||
算力节点 A -> 算力节点 B
|
||||
```
|
||||
|
||||
多节点任务调度由应用平台统一完成。应用平台从 `compute_nodes` 读取节点地址、权重、标签、启用状态、维护状态和健康检查结果；从 `resource_replicas` 判断目标节点是否已有所需数据集/模型副本；缺失时创建 `resource_sync_jobs`，通过目标节点 File Gateway 同步资源。
|
||||
|
||||
当前实现已支持在 `/compute` 算力节点页面新增和编辑节点。运维人员维护 `Compute API` 地址、`File Gateway` 地址、权重、标签、启用状态、最大并发和本地路径后，点击连接测试会由应用后端主动访问目标节点健康检查和 GPU 清单接口，并将 `health_detail`、`gpu_count`、`gpu_devices/gpus` 同步到 PostgreSQL。真实 GPU 服务器优先通过 `nvidia-smi` 发现 GPU；特殊环境可用 `COMPUTE_GPU_COUNT` 等环境变量声明兼容清单。
|
||||
|
||||
训练运行闭环：
|
||||
|
||||
- 前端启动训练后，Backend API 按 `compute_nodes` 的启用状态、调度状态、权重和并行任务数选择节点。
|
||||
- Backend API 向目标节点 `POST /modelTF/compute/jobs` 提交 LLaMA-Factory 训练作业，并在 `fine_tune_tasks.compute_job_id` 记录算力任务 ID。
|
||||
- Compute API 在真实模式下启动 `llamafactory-cli train` 子进程，训练日志写入 `TRAINING_LOG_ROOT/{job_id}.log`。
|
||||
- Backend API 启动后会运行应用侧轮询 worker，按 `COMPUTE_POLL_INTERVAL_SECONDS` 主动查询目标节点 `GET /modelTF/compute/jobs/{id}`，同步任务状态、进度、PID、输出目录、日志路径和产物索引。
|
||||
- 停止训练时，Backend API 优先调用目标节点 `POST /modelTF/compute/jobs/{id}/stop`，再回写应用任务状态。
|
||||
- 失败或停止任务可以通过 `POST /modelTF/compute/jobs/{id}/retry` 重试；重试会清空旧运行态，重新调度节点并创建 Compute Job。
|
||||
- 训练日志通过 `GET /modelTF/compute/jobs/{id}/logs` 读取，支持 `tail_lines`、`offset`、`limit`，用于训练详情页、训练日志页和日志平台采集。
|
||||
- Compute API 使用 `COMPUTE_SERVICE_TOKEN` 做服务间鉴权，应用侧请求携带 `X-Compute-Token`；健康检查接口保持可公开探活。
|
||||
- Compute API 会把本机训练作业登记到 `TRAINING_LOG_ROOT/compute-jobs.json`，服务重启后可恢复任务索引并继续暴露状态和日志。
|
||||
|
||||
调度策略：
|
||||
|
||||
- 默认自动调度，按节点健康、标签、GPU 空闲、队列长度、节点权重和资源副本命中率排序。
|
||||
- 支持管理员/高级用户手动指定节点或 GPU。
|
||||
- `disabled` 节点不参与调度。
|
||||
- `draining` 节点不接收新任务，但允许已有任务跑完。
|
||||
- `maintenance/offline` 节点只允许查看和清理，不允许提交训练任务。
|
||||
|
||||
## 6. Compute API 接入标准
|
||||
|
||||
为预留其他训练平台，应用平台只依赖统一算力接口，不直接依赖 LLaMA-Factory 命令。
|
||||
@@ -285,16 +234,13 @@ compute/engines/openrlhf/
|
||||
|
||||
```env
|
||||
APP_ENV=prod
|
||||
MODELTF_ROUTE_PREFIX=/modelTF
|
||||
DATABASE_URL=postgresql+psycopg://yg_ft:***@postgres:5432/yg_ft
|
||||
API_PREFIX=/api
|
||||
DATABASE_URL=postgresql+asyncpg://yg_ft:***@postgres:5432/yg_ft
|
||||
REDIS_URL=redis://redis:6379/0
|
||||
LOG_DIR=/opt/yg-ft/logs/backend
|
||||
COMPUTE_API_BASE_URL=https://compute.internal:19100
|
||||
COMPUTE_API_BASE_URL=https://compute.internal:9100
|
||||
COMPUTE_SERVICE_TOKEN=***
|
||||
FILE_GATEWAY_BASE_URL=https://compute.internal:19101
|
||||
COMPUTE_STATUS_SYNC_MODE=polling
|
||||
COMPUTE_POLL_INTERVAL_SECONDS=3
|
||||
COMPUTE_POLL_BATCH_SIZE=100
|
||||
FILE_GATEWAY_BASE_URL=https://compute.internal:9101
|
||||
```
|
||||
|
||||
算力平台：
|
||||
@@ -302,23 +248,12 @@ COMPUTE_POLL_BATCH_SIZE=100
|
||||
```env
|
||||
COMPUTE_ENV=prod
|
||||
COMPUTE_HOST_ID=gpu-node-01
|
||||
COMPUTE_API_PORT=19100
|
||||
FILE_GATEWAY_PORT=19101
|
||||
COMPUTE_AUTH_ENABLED=true
|
||||
COMPUTE_SERVICE_TOKEN=***
|
||||
ENABLE_APP_CALLBACK=false
|
||||
LLAMA_FACTORY_HOME=/app/LLaMA-Factory
|
||||
COMPUTE_API_PORT=9100
|
||||
FILE_GATEWAY_PORT=9101
|
||||
APP_CALLBACK_BASE_URL=https://app.internal/api/v1/compute/callbacks
|
||||
APP_SERVICE_TOKEN=***
|
||||
LLAMA_FACTORY_HOME=/opt/LLaMA-Factory
|
||||
YG_FT_DATA_ROOT=/data/yg-ft
|
||||
YG_FT_DATA_ROOT_HOST=./data/yg-ft
|
||||
YG_FT_MODEL_ROOT=/data/yg-ft/models
|
||||
YG_FT_MODEL_ROOT_HOST=./data/yg-ft/models
|
||||
YG_FT_DATASET_ROOT=/data/yg-ft/datasets
|
||||
YG_FT_DATASET_ROOT_HOST=./data/yg-ft/datasets
|
||||
YG_FT_OUTPUT_ROOT=/data/yg-ft/outputs
|
||||
YG_FT_OUTPUT_ROOT_HOST=./data/yg-ft/outputs
|
||||
TRAINING_LOG_ROOT=/opt/yg-ft/logs/training
|
||||
TRAINING_LOG_ROOT_HOST=./data/yg-ft/logs/training
|
||||
COMPUTE_LOG_ROOT_HOST=./data/yg-ft/logs/compute
|
||||
LOG_DIR=/opt/yg-ft/logs/compute
|
||||
CUDA_VISIBLE_DEVICES=0,1,2,3
|
||||
```
|
||||
@@ -347,12 +282,12 @@ CUDA_VISIBLE_DEVICES=0,1,2,3
|
||||
|
||||
## 10. 部署检查清单
|
||||
|
||||
- PostgreSQL 已初始化当前运行脚本 `backend/app/db/sql/001_platform_runtime.sql`；`docs/postgres-schema.sql` 作为目标架构设计，后续通过迁移体系逐步收敛。
|
||||
- PostgreSQL 已初始化 `docs/postgres-schema.sql`。
|
||||
- Redis 可连通。
|
||||
- 后端 `GET /modelTF/health` 正常。
|
||||
- 后端 `GET /api/v1/health` 正常。
|
||||
- Compute API 健康检查正常。
|
||||
- Compute Agent 能识别 GPU、显存、CUDA 版本。
|
||||
- LLaMA-Factory 能在命令行完成最小训练作业。
|
||||
- LLaMA-Factory 能在命令行完成最小训练样例。
|
||||
- 应用平台能提交训练任务到 Compute API。
|
||||
- 任务状态能从算力平台同步回应用平台。
|
||||
- 数据集上传、离线导入、产物下载路径权限正确。
|
||||
@@ -360,72 +295,11 @@ CUDA_VISIBLE_DEVICES=0,1,2,3
|
||||
- ERROR 日志能触发告警。
|
||||
- 日志、数据集、模型、产物所在磁盘容量有监控和告警。
|
||||
|
||||
## 11. Docker Compose 文件规划
|
||||
|
||||
当前项目按应用服务器和算力服务器拆分了两套 Docker 部署文件，均采用代码外挂方式运行：
|
||||
|
||||
```text
|
||||
docker/
|
||||
app/
|
||||
Dockerfile.backend # Backend API 运行时镜像，代码通过 volume 挂载到 /app
|
||||
Dockerfile.frontend # Nginx 前端运行时镜像，frontend/dist 通过 volume 挂载
|
||||
docker-compose.yml # 应用服务器：frontend、backend-api、postgres、redis
|
||||
.env.example
|
||||
compute/
|
||||
Dockerfile.compute # CUDA + Python + Compute API 运行时镜像
|
||||
docker-compose.yml # 算力服务器：compute-api，预留 agent/file gateway 拆分
|
||||
.env.example
|
||||
```
|
||||
|
||||
项目根目录不再保留 `Dockerfile` 和 `docker-compose.yml`，避免与拆分部署入口混淆。
|
||||
|
||||
应用服务器启动：
|
||||
|
||||
```bash
|
||||
cd docker/app
|
||||
cp .env.example .env
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
算力服务器启动：
|
||||
|
||||
```bash
|
||||
cd docker/compute
|
||||
cp .env.example .env
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
应用服务器与算力服务器独立部署时，需要在 `docker/app/.env` 中配置：
|
||||
|
||||
```env
|
||||
COMPUTE_API_BASE_URL=http://<compute-server-ip>:19100
|
||||
FILE_GATEWAY_BASE_URL=http://<compute-server-ip>:19101
|
||||
COMPUTE_SERVICE_TOKEN=change_me
|
||||
```
|
||||
|
||||
这些地址在当前 Docker 阶段通过环境变量动态配置。后续多算力节点阶段建议升级为数据库配置，由应用平台从 `compute_nodes` 表读取节点地址、权重、标签、健康状态和启用状态，并在“算力节点管理”页面维护。
|
||||
|
||||
多节点后，每台算力服务器各自进入 `docker/compute` 启动一套算力服务，并在应用平台中登记为一条 `compute_nodes` 记录：
|
||||
|
||||
```text
|
||||
gpu-node-01 -> http://10.10.20.31:19100 / http://10.10.20.31:19101
|
||||
gpu-node-02 -> http://10.10.20.32:19100 / http://10.10.20.32:19101
|
||||
gpu-node-03 -> http://10.10.20.33:19100 / http://10.10.20.33:19101
|
||||
```
|
||||
|
||||
算力服务器需要在 `docker/compute/.env` 中配置：
|
||||
|
||||
```env
|
||||
ENABLE_APP_CALLBACK=false
|
||||
COMPUTE_SERVICE_TOKEN=change_me
|
||||
YG_FT_DATA_ROOT_HOST=./data/yg-ft
|
||||
```
|
||||
|
||||
## 12. 仍需确认的问题
|
||||
## 11. 仍需确认的问题
|
||||
|
||||
- 生产环境是否已有统一 ELK/OpenSearch、Filebeat/Vector 标准配置。
|
||||
- PostgreSQL/Redis 开发阶段采用项目自带部署；生产阶段是否切换企业统一基础设施，以及对应 SLA 仍需确认。
|
||||
- 数据库和 Redis 是否由企业基础设施统一提供，还是由项目自行部署。
|
||||
- 是否需要 PostgreSQL 主备、备份恢复、审计日志长期归档的明确 SLA。
|
||||
- 大文件上传是否需要断点续传、限速、病毒扫描或 DLP 检测。
|
||||
- 应用服务器与算力服务器默认只开通应用侧主动访问算力侧；如后续需要实时回调，再单独评估双向网络策略。
|
||||
- 多算力节点已按单机多 GPU 节点扩展设计；仍需确认是否需要节点组、租户绑定节点、同步限速和资源副本清理审批。
|
||||
- 应用服务器与算力服务器之间是否允许双向访问，还是只能应用侧主动访问算力侧。
|
||||
- 是否需要未来支持多台 GPU 节点调度；如果需要，Compute API 需要提前设计节点注册和调度策略。
|
||||
|
||||
@@ -1,145 +0,0 @@
|
||||
# 当前系统主链路开发计划
|
||||
|
||||
> 说明：本计划描述当前正在开发的系统能力。代码、接口和 SQL 均按后续生产演进基线维护，不以一次性演示、静态 Mock 或样例数据作为开发准则。联调辅助能力必须显式配置启用，并不得成为默认运行路径。
|
||||
|
||||
## 1. 阶段目标
|
||||
|
||||
当前阶段需要完成模型微调平台的主链路工程基础：
|
||||
|
||||
```text
|
||||
登录
|
||||
-> 模型管理
|
||||
-> 数据集管理
|
||||
-> 创建微调任务
|
||||
-> 调度算力节点与 GPU
|
||||
-> 检查并同步模型/数据集资源
|
||||
-> 启动训练任务
|
||||
-> 轮询任务状态、GPU 占用、训练日志、loss 曲线
|
||||
-> 训练完成后登记训练产物
|
||||
```
|
||||
|
||||
该阶段是正式系统的第一批可运行能力，不再初始化业务样例数据。系统只允许初始化内置管理员/运维账号，模型、数据集、算力节点、GPU、训练任务和资源副本必须通过页面、接口、算力 Agent 扫描或正式导入流程产生。
|
||||
|
||||
## 2. 运行模式
|
||||
|
||||
| 模式 | 说明 | 当前要求 |
|
||||
| --- | --- | --- |
|
||||
| `real` | 面向真实部署，等待 Compute API、Agent、File Gateway 和 LLaMA-Factory 执行器回写状态 | 默认模式 |
|
||||
| `simulator` | 仅用于隔离联调，无真实 GPU 时临时推进任务状态、GPU 状态和训练日志 | 必须显式开启，不得用于生产基线 |
|
||||
|
||||
后端默认 `COMPUTE_MODE=real`。在该模式下，任务状态不再按时间自动推进，必须由后续真实算力同步逻辑更新。算力服务默认 `COMPUTE_EXECUTION_MODE=real`，真实训练执行器未完成前，创建训练作业会返回明确的未实现错误，避免误认为已经完成生产训练能力。
|
||||
|
||||
训练相关能力必须沉淀在 `compute/engines/` 适配层，不允许在应用平台后端直接拼接或执行训练命令。
|
||||
|
||||
## 3. 当前开发范围
|
||||
|
||||
### 3.1 应用平台后端
|
||||
|
||||
对应目录：
|
||||
|
||||
```text
|
||||
backend/app/
|
||||
api/v1/endpoints/platform.py
|
||||
core/
|
||||
db/
|
||||
```
|
||||
|
||||
已建立能力：
|
||||
- 统一 API 响应结构 `{ code, message, data }`。
|
||||
- PostgreSQL 运行表初始化，当前执行脚本位于 `backend/app/db/sql/001_platform_runtime.sql`。
|
||||
- 内置管理员账号初始化，业务数据不再自动写入样例记录。
|
||||
- 登录、当前用户、用户列表与权限页面接口。
|
||||
- 模型管理、训练产物列表、权重合并任务入口。
|
||||
- 数据集管理、文件上传、预览、版本管理和下载。
|
||||
- 微调任务创建、启动、停止、删除、进度查询、checkpoint 查询。
|
||||
- 系统健康指标、系统信息、训练日志、系统日志接口。
|
||||
- 算力节点、GPU、队列、资源副本、资源同步任务接口。
|
||||
|
||||
待继续开发：
|
||||
- 接入正式 ORM/Repository/Service 分层和 Alembic 迁移。
|
||||
- 将任务状态更新改为应用侧定时轮询 Compute API/File Gateway 后落库。
|
||||
- 完成项目/模型/数据集级权限隔离校验。
|
||||
- 完成审批流、审计留存、配额、资源申请和多租户上下文。
|
||||
- 增加正式异常码、接口鉴权、中间件、幂等控制和分页规范。
|
||||
|
||||
### 3.2 算力平台服务
|
||||
|
||||
对应目录：
|
||||
|
||||
```text
|
||||
compute/
|
||||
api/main.py
|
||||
agent/
|
||||
engines/llama_factory/
|
||||
file_gateway/
|
||||
```
|
||||
|
||||
已建立能力：
|
||||
- `/modelTF/health` 与 `/modelTF/v1/compute/health` 节点健康检查。
|
||||
- LLaMA-Factory 参数校验、命令生成和训练日志指标解析。
|
||||
- Compute API 作业、GPU、文件网关接口壳。
|
||||
- 显式 `simulator` 模式下的内存状态机，用于隔离联调。
|
||||
|
||||
待继续开发：
|
||||
- 真实 GPU 发现：接入 `nvidia-smi` 或 NVML。
|
||||
- GPU 锁定与释放：一张 GPU 同一时间只分配给一个训练或推理任务。
|
||||
- LLaMA-Factory 真实执行器：生成 YAML/命令、启动进程、停止进程、采集 PID。
|
||||
- 训练日志采集：读取宿主机挂载日志文件，解析 loss、learning rate、epoch 等指标。
|
||||
- Checkpoint/adapter/merged model 扫描与产物登记。
|
||||
- File Gateway：本地磁盘文件上传、下载、校验、导入和跨节点资源同步。
|
||||
|
||||
### 3.3 前端页面
|
||||
|
||||
已接入页面：
|
||||
- `/login`：登录接口。
|
||||
- `/model-manage`：模型列表、模型来源、训练产物。
|
||||
- `/dataset`、`/dataset/:id/preview`：数据集列表、预览、版本。
|
||||
- `/fine-tune`、`/fine-tune/create`：微调任务创建、启动、状态轮询。
|
||||
- `/training-log/:id`：训练日志和 loss 曲线。
|
||||
- `/hardware`：平台 GPU 与系统性能。
|
||||
- `/compute`：算力节点、GPU、队列、资源副本。
|
||||
|
||||
前端 Mock 默认关闭。仅在隔离前端开发时可设置 `VITE_ENABLE_MOCK=true`，真实联调和后续生产演进均以 `/modelTF` 后端接口为准。
|
||||
|
||||
待继续开发：
|
||||
- 补齐多租户、项目管理、审批中心、审计中心、配额管理页面。
|
||||
- 完成算力节点管理表单，包括节点地址、权重、标签、启用状态和健康检查结果。
|
||||
- 完成模型/数据集导入页面，支持本地路径扫描和归属项目选择。
|
||||
- 推理服务页面需接入真实后端任务接口，移除页面内本地假对话路径。
|
||||
|
||||
### 3.4 数据库
|
||||
|
||||
当前运行 SQL：
|
||||
|
||||
```text
|
||||
backend/app/db/sql/001_platform_runtime.sql
|
||||
```
|
||||
|
||||
架构目标 SQL：
|
||||
|
||||
```text
|
||||
docs/postgres-schema.sql
|
||||
```
|
||||
|
||||
当前运行 SQL 用于支持已开发接口落库；架构目标 SQL 包含用户中心、多租户、项目隔离、审批、审计、配额、评测等完整模型。后续需要通过 Alembic 将二者收敛为统一迁移体系，生产升级只走迁移脚本，不依赖手工改表。
|
||||
|
||||
## 4. 验收标准
|
||||
|
||||
- 启动后端必须连接 PostgreSQL，不允许回退到 SQLite。
|
||||
- 后端启动只初始化系统内置账号，不初始化模型、数据集、算力节点、GPU、训练任务等业务样例数据。
|
||||
- 前端默认请求真实 `/modelTF` 接口，除非显式设置 `VITE_ENABLE_MOCK=true`。
|
||||
- 默认 `real` 模式下任务状态不自动伪造完成，必须等待真实算力同步。
|
||||
- 显式 `simulator` 模式只能用于隔离联调，部署文档必须标注不得用于生产。
|
||||
- 登录后可以进入主界面，并可通过页面/API 创建真实业务记录。
|
||||
- 代码、接口路由、配置项、数据库表名不得使用 `demo` 命名。
|
||||
|
||||
## 5. 后续开发计划
|
||||
|
||||
| 阶段 | 重点 | 说明 |
|
||||
| --- | --- | --- |
|
||||
| 阶段 1 | 数据库迁移体系 | 将当前运行 SQL 与架构 SQL 收敛到 Alembic 迁移 |
|
||||
| 阶段 2 | 后端领域分层 | 拆分用户、模型、数据集、训练、算力、审计等模块 |
|
||||
| 阶段 3 | 真实 Compute Agent | GPU 发现、资源锁定、进程管理、日志采集 |
|
||||
| 阶段 4 | LLaMA-Factory 训练执行 | YAML/命令生成、进程启动/停止、checkpoint 和 adapter 扫描 |
|
||||
| 阶段 5 | 企业治理 | 多租户、项目隔离、审批流、审计留存、配额和资源申请 |
|
||||
| 阶段 6 | 多算力节点调度 | 基于 `compute_nodes`、标签、权重、资源副本和节点健康实现调度策略 |
|
||||
@@ -1,98 +0,0 @@
|
||||
# 菜单与功能需求总览
|
||||
|
||||
> 本文根据当前前端侧边栏、路由、需求文档、接口文档、部署文档和 SQL 脚本整理。当前代码和 SQL 均按正式系统开发基线维护；Mock、Simulator 只能作为显式联调能力，不作为默认开发准则。
|
||||
|
||||
## 1. 菜单分层
|
||||
|
||||
### 1.1 当前侧边栏菜单
|
||||
|
||||
| 一级分组 | 菜单 | 路由 | 权限码 | 当前状态 | 主要功能 |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 服务看板 | 服务看板 | `/dashboard` | `dashboard` | 已有页面，接口需继续完善 | 总览指标、服务状态、训练统计、最近任务、健康入口 |
|
||||
| 模型服务 | 模型训练 | `/fine-tune` | `fine-tune` | 已接入主链路 | 训练任务列表、创建训练、启动/停止、进度、训练日志、checkpoint |
|
||||
| 模型服务 | 模型评测 | `/model-eval` | `model-eval` | 前端页面已有，后端待完整实现 | 评测任务、评测维度、样本评分、综合结果 |
|
||||
| 模型服务 | 模型推理 | `/model-inference` | `model-inference` | 前端页面已有，后端待完整实现 | 推理任务、模型加载、单模型对话、模型对比入口 |
|
||||
| 模型服务 | 模型管理 | `/model-manage` | `model-manage` | 已接入主链路 | 基座模型登记、本地/API 模型、训练产物、权重合并、模型导出 |
|
||||
| 数据治理 | 数据集管理 | `/dataset` | `dataset` | 已接入主链路 | 数据集列表、上传、预览、在线编辑、版本、下载、删除审批入口 |
|
||||
| 数据治理 | 数据处理 | `/data-process` | `data-process` | 前端页面已有，后端待完整实现 | 文档上传、切片预览、LLM 生成、结果编辑、发布数据集 |
|
||||
| 其他工具 | 数据类型转换 | `/data-convert` | `data-convert` | 前端页面已有，后端待实现 | JSON/JSONL/Markdown 等格式转换任务 |
|
||||
| 算力资源 | 算力节点 | `/compute` | `compute` | 已接入节点管理接口 | 节点地址、权重、标签、启用状态、GPU、队列、资源副本 |
|
||||
| 系统设置 | 用户设置 | `/user-settings` | `user-settings` | 已接入基础用户接口 | 用户列表、创建用户、启停、页面权限 |
|
||||
| 系统设置 | 平台性能 | `/hardware` | `hardware` | 已有接口，需接真实采集 | CPU、内存、磁盘、GPU、进程、网络监控 |
|
||||
| 系统设置 | 查看日志 | `/logs` | `logs` | 已有接口，需接真实日志文件 | 后端日志、error 日志、训练日志索引、日志内容查看 |
|
||||
|
||||
### 1.2 当前二级和隐藏路由
|
||||
|
||||
| 页面 | 路由 | 归属菜单 | 说明 |
|
||||
| --- | --- | --- | --- |
|
||||
| 登录 | `/login` | 独立入口 | 登录后进入主界面 |
|
||||
| 使用文档 | `/guide` | 独立入口 | 当前系统使用说明 |
|
||||
| 创建训练任务 | `/fine-tune/create` | 模型训练 | 训练参数、模型/数据集/GPU 选择 |
|
||||
| 训练日志 | `/training-log/:id` | 模型训练 | 日志、指标、checkpoint、任务概览 |
|
||||
| 新建评测 | `/model-eval/create` | 模型评测 | 模型、数据集、维度、GPU 选择 |
|
||||
| 评测详情 | `/model-eval/:id` | 模型评测 | 维度汇总、样本结果、人工复核预留 |
|
||||
| 评测维度创建/编辑 | `/model-eval/dimension/create`、`/model-eval/dimension/:id/edit` | 模型评测 | 评测规则、Prompt、评分器配置 |
|
||||
| 新建推理 | `/model-inference/create` | 模型推理 | 推理任务和模型加载配置 |
|
||||
| 模型对话 | `/model-inference/chat/:id` | 模型推理 | 单模型对话 |
|
||||
| 模型对比 | `/model-compare/chat/:id`、`/model-compare/result` | 模型推理 | 多模型对比和结果页 |
|
||||
| 添加/编辑模型 | `/model-manage/create`、`/model-manage/:id/edit` | 模型管理 | 模型登记、用途、来源、路径/API 配置 |
|
||||
| 合并权重 | `/model-manage/merge` | 模型管理 | LoRA/Adapter 合并任务 |
|
||||
| 数据处理创建/详情 | `/data-process/create`、`/data-process/:id` | 数据处理 | 数据处理向导和任务详情 |
|
||||
| 数据集创建/编辑/预览 | `/dataset/create`、`/dataset/:id/edit`、`/dataset/:id/preview` | 数据集管理 | 数据集元数据、文件、版本与内容 |
|
||||
| 自定义工具 | `/tools`、`/tools/create`、`/tools/:id/edit` | 规划入口 | 路由存在，当前侧边栏未展示，后续可归入“其他工具” |
|
||||
| 算力子页 | `/compute/gpus`、`/compute/queue`、`/compute/nodes` | 算力节点 | 当前可作为页签或深链 |
|
||||
| 创建用户/权限设置 | `/user-settings/create`、`/user-settings/:id/permission` | 用户设置 | 用户创建和页面权限 |
|
||||
| 无权限页 | `/permission-denied` | 系统页 | 路由守卫无权限跳转 |
|
||||
|
||||
### 1.3 企业治理待补菜单
|
||||
|
||||
| 建议菜单分组 | 菜单 | 建议路由 | 优先级 | 必要性 |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| 组织与项目 | 租户管理 | `/tenants`、`/tenants/:id` | P0 | 多租户隔离、配额、留存策略入口 |
|
||||
| 组织与项目 | 项目空间 | `/projects`、`/projects/:id`、`/projects/:id/members` | P0 | 项目级模型/数据集/任务隔离 |
|
||||
| 组织与项目 | 资源授权 | `/projects/:id/permissions` 或资源详情弹窗 | P0 | 模型/数据集/任务级 ACL |
|
||||
| 治理中心 | 审批中心 | `/approvals`、`/approvals/:id` | P0 | 删除、发布、导出、停止他人任务等高风险动作 |
|
||||
| 治理中心 | 审批设置 | `/approval-settings` | P1 | 审批模板、审批人规则、超时策略 |
|
||||
| 治理中心 | 审计中心 | `/audit-logs`、`/login-logs`、`/download-logs` | P1 | 操作审计、登录审计、下载审计、导出 |
|
||||
| 运维中心 | 存储管理 | `/storage` | P1 | 本地磁盘占用、临时文件、checkpoint 清理、留存 |
|
||||
| 运维中心 | 训练引擎管理 | `/training-engines` | P2 | LLaMA-Factory 和后续引擎能力 schema、健康检查 |
|
||||
| 模型服务 | 模型服务治理 | `/model-services`、`/model-services/:id` | P1 | 测试/生产服务发布、调用统计、下线审批 |
|
||||
|
||||
## 2. 菜单对应接口和数据库
|
||||
|
||||
| 菜单/模块 | 主要接口 | 当前运行 SQL | 目标 SQL |
|
||||
| --- | --- | --- | --- |
|
||||
| 登录、用户设置 | `/modelTF/login`、`/modelTF/me`、`/modelTF/users` | `users` | `users`、`login_sessions`、`permissions`、`role_permissions`、`user_permission_overrides` |
|
||||
| 服务看板 | `/modelTF/dashboard/overview`、`/modelTF/health` | 复用模型/数据集/任务/算力表 | `system_metric_snapshots`、`web_logs`、各业务表聚合 |
|
||||
| 模型管理 | `/modelTF/model-manage`、`/modelTF/model-manage/trained-models`、`/modelTF/model-manage/merge` | `models`、`trained_models` | `models`、`trained_models`、`storage_objects`、`local_import_jobs`、`resource_acl` |
|
||||
| 数据集管理 | `/modelTF/dataset-manage`、`/modelTF/dataset-manage/upload/{id}`、`/preview`、`/versions` | `datasets`、`dataset_files` | `datasets`、`dataset_files`、`dataset_file_versions`、`dataset_records`、`storage_objects` |
|
||||
| 模型训练 | `/modelTF/fine-tune`、`/start`、`/progress`、`/checkpoints` | `fine_tune_tasks`、`trained_models` | `fine_tune_tasks`、`fine_tune_metrics`、`fine_tune_checkpoints`、`compute_jobs`、`gpu_allocations` |
|
||||
| 训练日志 | `/modelTF/training-log-files`、`/modelTF/training-log-content` | 由任务表生成索引 | 日志文件元数据、`fine_tune_metrics`、`audit_logs` |
|
||||
| 算力节点 | `/modelTF/compute/nodes`、`/compute/gpus`、`/compute/queue`、`/compute/nodes/{id}/replicas` | `compute_nodes`、`gpus`、`resource_replicas`、`resource_sync_jobs` | `compute_nodes`、`gpu_devices`、`compute_node_engines`、`compute_jobs`、`resource_replicas`、`resource_sync_jobs` |
|
||||
| 平台性能 | `/modelTF/system-info`、`/modelTF/compute/gpus` | `gpus`、任务表 | `system_metric_snapshots`、`gpu_devices`、`compute_jobs` |
|
||||
| 查看日志 | `/modelTF/log-files`、`/modelTF/log-content`、`/modelTF/web-log` | 文件日志 | `web_logs`、`audit_logs`，大日志进入日志平台 |
|
||||
| 模型评测 | `/modelTF/model-eval`、`/modelTF/dimension` | 当前运行 SQL 未覆盖 | `eval_tasks`、`eval_dimensions`、`eval_sample_results`、`eval_dimension_summaries` |
|
||||
| 模型推理/对比 | `/modelTF/model-compare`、`/modelTF/model-chat/*` | 当前运行 SQL 未覆盖 | `inference_tasks`、`inference_task_models`、`chat_sessions`、`chat_messages` |
|
||||
| 数据处理 | `/modelTF/data-process/*` | 当前运行 SQL 未覆盖 | `data_process_tasks`、`data_process_source_files`、`data_process_preview_items`、`data_process_results` |
|
||||
| 数据转换/自定义工具 | `/modelTF/data-convert/jobs`、`/modelTF/tools` | 当前运行 SQL 未覆盖 | `data_convert_jobs`、`custom_tools` |
|
||||
| 租户/项目/资源授权 | `/modelTF/tenants`、`/modelTF/projects`、`/modelTF/resources/{type}/{id}/acl` | 当前运行 SQL 未覆盖 | `tenants`、`tenant_users`、`projects`、`project_members`、`resource_acl` |
|
||||
| 审批/审计/留存/配额 | `/modelTF/approvals`、`/modelTF/audit-logs`、`/modelTF/retention-policies`、`/modelTF/quotas/usage` | 当前运行 SQL 未覆盖 | `approval_templates`、`approval_instances`、`approval_steps`、`audit_logs`、`retention_policies`、`quotas`、`quota_usage` |
|
||||
|
||||
## 3. 文档和脚本检查结论
|
||||
|
||||
| 对象 | 当前结论 | 本次补充 |
|
||||
| --- | --- | --- |
|
||||
| 需求文档 | `docs/platform-architecture-requirements.md` 和 `docs/system-development-plan.md` 已覆盖多租户、项目隔离、审批、审计、多算力节点、应用/算力分离部署；缺少一份按当前菜单组织的总览 | 新增本文作为菜单和功能需求总览 |
|
||||
| 接口文档 | `docs/backend-api-design.md` 已统一 `/modelTF`，并已有页面/接口映射；需要明确引用菜单总览，避免开发只看接口不看页面入口 | 在接口文档增加菜单总览引用 |
|
||||
| 开发计划 | `docs/system-development-plan.md` 已按工作包列出页面、接口和 DB；需要把本文作为任务认领入口 | 在开发计划增加菜单总览引用 |
|
||||
| 部署文档 | `docs/deployment-plan.md`、`docker/README.md` 已覆盖应用/算力分离、单机多 GPU、本地磁盘、真实模式默认、Docker 拆分 | 暂无新增部署配置要求 |
|
||||
| 目标 SQL | `docs/postgres-schema.sql` 覆盖完整目标模型，包含用户、权限、多租户、项目、审批、审计、模型、数据集、训练、评测、推理、算力、存储、导入、服务治理 | 暂不需要新增目标表 |
|
||||
| 当前运行 SQL | `backend/app/db/sql/001_platform_runtime.sql` 只覆盖已接入运行接口的最小表集 | 后续每实现一个 P0/P1 菜单模块，应同步补运行 SQL 或迁移脚本；不能再以样例数据补功能 |
|
||||
|
||||
## 4. 后续补充原则
|
||||
|
||||
- 新增侧边栏菜单时，必须同步补齐：路由、权限码、接口文档、DB 表/迁移、审计动作、部署依赖。
|
||||
- 新增后端接口时，必须在 `docs/backend-api-design.md` 标注对应页面/功能模块。
|
||||
- 新增表结构时，目标模型写入 `docs/postgres-schema.sql`，当前可执行落库写入 `backend/app/db/sql/` 或 Alembic 迁移。
|
||||
- 与训练、评测、推理、数据处理相关的异步任务必须落库，不能依赖前端本地状态。
|
||||
- 与算力相关的功能默认走真实模式；Simulator 只能显式开启用于隔离联调。
|
||||
@@ -80,20 +80,20 @@ flowchart LR
|
||||
|
||||
- 应用平台调用算力平台必须携带 `X-Service-Token` 或 mTLS 证书。
|
||||
- 算力平台不信任前端用户身份，只信任应用平台下发的租户、项目、任务和资源上下文。
|
||||
- 第一阶段不默认启用算力侧回调，应用平台通过定时轮询 Compute API 同步任务状态；如未来启用回调，必须带签名，避免伪造状态。
|
||||
- 所有任务回调必须带签名，避免伪造状态。
|
||||
|
||||
核心通信接口：
|
||||
|
||||
| 方向 | 接口 | 说明 |
|
||||
| --- | --- | --- |
|
||||
| 应用 -> 算力 | `POST /modelTF/compute/jobs` | 创建训练/评测/数据处理/推理任务 |
|
||||
| 应用 -> 算力 | `POST /modelTF/compute/jobs/{id}/stop` | 停止任务 |
|
||||
| 应用 -> 算力 | `GET /modelTF/compute/jobs/{id}` | 查询任务状态 |
|
||||
| 应用 -> 算力 | `GET /modelTF/compute/jobs/{id}/logs` | 拉取日志 |
|
||||
| 应用 -> 算力 | `GET /modelTF/compute/resources/gpus` | 查询 GPU 状态 |
|
||||
| 应用 -> 算力 | `POST /modelTF/compute/files/upload` | 上传文件到算力本地磁盘 |
|
||||
| 应用 -> 算力 | `GET /modelTF/compute/files/{object_id}/download` | 下载文件 |
|
||||
| 应用 -> 算力 | `GET /modelTF/compute/jobs?status=running` | 定时轮询任务状态、指标、产物索引 |
|
||||
| 应用 -> 算力 | `POST /compute/jobs` | 创建训练/评测/数据处理/推理任务 |
|
||||
| 应用 -> 算力 | `POST /compute/jobs/{id}/stop` | 停止任务 |
|
||||
| 应用 -> 算力 | `GET /compute/jobs/{id}` | 查询任务状态 |
|
||||
| 应用 -> 算力 | `GET /compute/jobs/{id}/logs` | 拉取日志 |
|
||||
| 应用 -> 算力 | `GET /compute/resources/gpus` | 查询 GPU 状态 |
|
||||
| 应用 -> 算力 | `POST /compute/files/upload` | 上传文件到算力本地磁盘 |
|
||||
| 应用 -> 算力 | `GET /compute/files/{object_id}/download` | 下载文件 |
|
||||
| 算力 -> 应用 | `POST /api/internal/compute-callbacks/jobs` | 回调任务状态、指标、产物 |
|
||||
|
||||
## 3. 本地磁盘存储设计
|
||||
|
||||
@@ -149,8 +149,6 @@ flowchart LR
|
||||
|
||||
## 4. 单机多 GPU 资源调度
|
||||
|
||||
第一版可以先以单个算力节点跑通主链路，但数据模型、接口和页面需要按多算力节点预留。多节点阶段仍然采用“每台 GPU 服务器 = 一个单机多 GPU 节点”的模式，不引入 Kubernetes。
|
||||
|
||||
### 4.1 GPU 资源模型
|
||||
|
||||
每张 GPU 需要记录：
|
||||
@@ -185,25 +183,6 @@ flowchart LR
|
||||
- 支持任务队列优先级：`low`、`normal`、`high`、`urgent`。
|
||||
- 高优任务是否可抢占低优任务，需要审批或管理员权限。
|
||||
|
||||
### 4.4 多算力节点升级策略
|
||||
|
||||
多算力节点阶段的推荐决策：
|
||||
|
||||
- 每个可执行训练任务的 GPU 节点都部署 `Compute API`、`Compute Agent`、`File Gateway`、LLaMA-Factory、CUDA/PyTorch 训练环境和本地数据盘。
|
||||
- 应用服务器可以主动访问所有算力节点的 `Compute API/File Gateway`。
|
||||
- 算力节点之间默认不互相访问，不做节点间点对点同步；所有调度、状态同步和资源分发由应用平台统一编排。
|
||||
- 长期坚持每台算力服务器本地磁盘，因此需要 `resource_replicas` 记录数据集、基座模型、checkpoint、adapter、导出模型在哪些节点已有本地副本。
|
||||
- 调度前必须检查目标节点是否已有模型和数据集副本；缺失时由应用平台通过目标节点 File Gateway 创建同步任务，完成后再启动训练。
|
||||
- 调度支持自动和手动两种模式：普通用户默认自动调度，管理员/高级用户可手动指定节点、GPU、标签或节点组。
|
||||
|
||||
多节点自动调度建议：
|
||||
|
||||
1. 过滤 `enabled = true` 且 `scheduler_status = online` 的节点。
|
||||
2. 按训练引擎、GPU 型号、显存、节点标签、租户/项目配额过滤。
|
||||
3. 优先选择已存在所需模型/数据集副本的节点，减少跨节点复制。
|
||||
4. 同等条件下按空闲 GPU、队列长度、节点权重和最近健康检查排序。
|
||||
5. `draining` 节点不接收新任务，但允许已有任务完成。
|
||||
|
||||
## 5. 训练引擎接入标准
|
||||
|
||||
### 5.1 引擎抽象
|
||||
@@ -442,9 +421,6 @@ LLaMA-Factory 适配器负责：
|
||||
- 队列中的任务。
|
||||
- 资源配额：租户/项目/用户维度。
|
||||
- 算力节点 Agent 状态。
|
||||
- 算力节点新增/编辑、连接测试、启用/禁用、维护模式。
|
||||
- 节点权重、标签、训练引擎版本、LLaMA-Factory 健康状态。
|
||||
- 节点本地资源副本：数据集、模型、checkpoint、adapter 和导出模型缓存。
|
||||
- 训练引擎健康状态。
|
||||
|
||||
### 8.5 文件与存储管理
|
||||
@@ -556,54 +532,54 @@ LLaMA-Factory 适配器负责：
|
||||
|
||||
租户：
|
||||
|
||||
- `GET /modelTF/tenants`
|
||||
- `POST /modelTF/tenants`
|
||||
- `GET /modelTF/tenants/{id}`
|
||||
- `PUT /modelTF/tenants/{id}`
|
||||
- `PUT /modelTF/tenants/{id}/quota`
|
||||
- `PUT /modelTF/tenants/{id}/retention-policy`
|
||||
- `GET /api/tenants`
|
||||
- `POST /api/tenants`
|
||||
- `GET /api/tenants/{id}`
|
||||
- `PUT /api/tenants/{id}`
|
||||
- `PUT /api/tenants/{id}/quota`
|
||||
- `PUT /api/tenants/{id}/retention-policy`
|
||||
|
||||
项目：
|
||||
|
||||
- `GET /modelTF/projects`
|
||||
- `POST /modelTF/projects`
|
||||
- `GET /modelTF/projects/{id}`
|
||||
- `PUT /modelTF/projects/{id}`
|
||||
- `POST /modelTF/projects/{id}/archive`
|
||||
- `GET /modelTF/projects/{id}/members`
|
||||
- `POST /modelTF/projects/{id}/members`
|
||||
- `PUT /modelTF/projects/{id}/members/{user_id}`
|
||||
- `DELETE /modelTF/projects/{id}/members/{user_id}`
|
||||
- `GET /api/projects`
|
||||
- `POST /api/projects`
|
||||
- `GET /api/projects/{id}`
|
||||
- `PUT /api/projects/{id}`
|
||||
- `POST /api/projects/{id}/archive`
|
||||
- `GET /api/projects/{id}/members`
|
||||
- `POST /api/projects/{id}/members`
|
||||
- `PUT /api/projects/{id}/members/{user_id}`
|
||||
- `DELETE /api/projects/{id}/members/{user_id}`
|
||||
|
||||
资源授权：
|
||||
|
||||
- `GET /modelTF/resources/{resource_type}/{resource_id}/acl`
|
||||
- `PUT /modelTF/resources/{resource_type}/{resource_id}/acl`
|
||||
- `POST /modelTF/resources/{resource_type}/{resource_id}/share`
|
||||
- `GET /api/resources/{resource_type}/{resource_id}/acl`
|
||||
- `PUT /api/resources/{resource_type}/{resource_id}/acl`
|
||||
- `POST /api/resources/{resource_type}/{resource_id}/share`
|
||||
|
||||
审批：
|
||||
|
||||
- `GET /modelTF/approvals`
|
||||
- `POST /modelTF/approvals`
|
||||
- `GET /modelTF/approvals/{id}`
|
||||
- `POST /modelTF/approvals/{id}/approve`
|
||||
- `POST /modelTF/approvals/{id}/reject`
|
||||
- `POST /modelTF/approvals/{id}/cancel`
|
||||
- `GET /api/approvals`
|
||||
- `POST /api/approvals`
|
||||
- `GET /api/approvals/{id}`
|
||||
- `POST /api/approvals/{id}/approve`
|
||||
- `POST /api/approvals/{id}/reject`
|
||||
- `POST /api/approvals/{id}/cancel`
|
||||
|
||||
算力：
|
||||
|
||||
- `GET /modelTF/compute/nodes`
|
||||
- `GET /modelTF/compute/gpus`
|
||||
- `GET /modelTF/compute/queue`
|
||||
- `POST /modelTF/compute/jobs/{id}/retry`
|
||||
- `POST /modelTF/compute/jobs/{id}/priority`
|
||||
- `GET /api/compute/nodes`
|
||||
- `GET /api/compute/gpus`
|
||||
- `GET /api/compute/queue`
|
||||
- `POST /api/compute/jobs/{id}/retry`
|
||||
- `POST /api/compute/jobs/{id}/priority`
|
||||
|
||||
训练引擎：
|
||||
|
||||
- `GET /modelTF/training-engines`
|
||||
- `GET /modelTF/training-engines/{id}`
|
||||
- `POST /modelTF/training-engines/{id}/health-check`
|
||||
- `GET /modelTF/training-engines/{id}/schema`
|
||||
- `GET /api/training-engines`
|
||||
- `GET /api/training-engines/{id}`
|
||||
- `POST /api/training-engines/{id}/health-check`
|
||||
- `GET /api/training-engines/{id}/schema`
|
||||
|
||||
## 10. 端到端业务流程
|
||||
|
||||
@@ -674,8 +650,6 @@ LLaMA-Factory 适配器负责：
|
||||
10. 是否需要对外提供标准 API 给其他系统调用训练、评测、推理能力？
|
||||
11. 是否需要接入企业统一身份认证，例如 LDAP、OIDC、企业微信、钉钉？
|
||||
12. 是否需要成本核算：按租户/项目统计 GPU 小时、磁盘占用、模型调用量？
|
||||
13. 多算力节点是否需要节点组、租户绑定节点或项目绑定节点策略？
|
||||
14. 跨节点资源同步是否需要限速、同步窗口和管理员审批？
|
||||
|
||||
## 13. 推荐决策补充
|
||||
|
||||
@@ -684,17 +658,15 @@ LLaMA-Factory 适配器负责：
|
||||
1. 本地磁盘主存储放在算力服务器，应用服务器只保留上传临时文件。临时文件默认保留 24 小时，成功转发到算力文件网关后可立即进入清理队列。
|
||||
2. 第一版不支持 MIG、GPU 分片和多任务共享同一张 GPU。一张 GPU 同一时间只分配给一个训练任务或一个推理服务。GPU 数据模型预留 `partition_type`、`parent_gpu_uuid`、`memory_total_mb`，便于后续扩展 MIG。
|
||||
3. 第一版不做自动抢占。支持任务优先级和排队；停止他人任务需要审批或平台管理员权限。
|
||||
4. 多算力节点仍按“单机多 GPU 节点”管理，每个节点独立部署算力服务和 LLaMA-Factory；节点之间不互相访问，由应用平台统一调度和资源同步。
|
||||
5. 多节点调度默认自动选择节点，同时支持管理员/高级用户手动指定节点；调度优先考虑节点健康、标签、权重、空闲 GPU、队列长度和资源副本是否已存在。
|
||||
6. 第一版必须支持离线导入已有模型和数据集目录。导入由算力 Agent 扫描、校验、登记，并归属指定租户和项目。
|
||||
7. 模型发布区分测试服务和生产服务。测试服务项目内可启动并默认限流；生产服务必须审批。
|
||||
8. 数据脱敏和数据质量评分作为数据处理模块的一等能力进入第一期，先实现规则版脱敏、格式校验、重复率、完整性、长度分布等指标。
|
||||
9. 人工评测/复核作为第二期功能，但第一期需在数据库和页面入口预留人工复核状态与修订字段。
|
||||
10. 第一版支持从 checkpoint 手动恢复训练，不做自动失败续训。失败任务可选择 checkpoint 重试。
|
||||
11. 第一版必须支持 checkpoint 自动清理策略：默认保留最近 3 个、最优 2 个；已发布模型关联 checkpoint 不自动删除；失败任务 checkpoint 默认保留 14 天。
|
||||
12. 第一版提供内部 API，第二期再开放面向其他系统的标准 API、API Key、限流和 Webhook。
|
||||
13. 第一版使用本地账号，预留 OIDC/LDAP 字段和认证 provider 抽象；第二期接入企业统一身份认证。
|
||||
14. 第一版做 GPU 小时、磁盘占用、任务时长、推理调用量等用量统计；第二期再做成本单价和账单核算。
|
||||
4. 第一版必须支持离线导入已有模型和数据集目录。导入由算力 Agent 扫描、校验、登记，并归属指定租户和项目。
|
||||
5. 模型发布区分测试服务和生产服务。测试服务项目内可启动并默认限流；生产服务必须审批。
|
||||
6. 数据脱敏和数据质量评分作为数据处理模块的一等能力进入第一期，先实现规则版脱敏、格式校验、重复率、完整性、长度分布等指标。
|
||||
7. 人工评测/复核作为第二期功能，但第一期需在数据库和页面入口预留人工复核状态与修订字段。
|
||||
8. 第一版支持从 checkpoint 手动恢复训练，不做自动失败续训。失败任务可选择 checkpoint 重试。
|
||||
9. 第一版必须支持 checkpoint 自动清理策略：默认保留最近 3 个、最优 2 个；已发布模型关联 checkpoint 不自动删除；失败任务 checkpoint 默认保留 14 天。
|
||||
10. 第一版提供内部 API，第二期再开放面向其他系统的标准 API、API Key、限流和 Webhook。
|
||||
11. 第一版使用本地账号，预留 OIDC/LDAP 字段和认证 provider 抽象；第二期接入企业统一身份认证。
|
||||
12. 第一版做 GPU 小时、磁盘占用、任务时长、推理调用量等用量统计；第二期再做成本单价和账单核算。
|
||||
|
||||
以上决策需要同步反映在接口文档、数据库 SQL、前端页面和部署方案中。第一版实现不再阻塞于这些问题的反复确认，除非实际部署环境与假设明显冲突。
|
||||
|
||||
|
||||
@@ -1061,20 +1061,8 @@ CREATE TABLE IF NOT EXISTS compute_nodes (
|
||||
name varchar(150) NOT NULL,
|
||||
host varchar(200) NOT NULL,
|
||||
api_base_url text NOT NULL,
|
||||
file_gateway_url text NOT NULL DEFAULT '',
|
||||
storage_node_id uuid REFERENCES storage_nodes(id) ON DELETE SET NULL,
|
||||
status varchar(40) NOT NULL DEFAULT 'online',
|
||||
scheduler_status varchar(40) NOT NULL DEFAULT 'online',
|
||||
scheduler_weight integer NOT NULL DEFAULT 100,
|
||||
enabled boolean NOT NULL DEFAULT true,
|
||||
max_parallel_jobs integer NOT NULL DEFAULT 1,
|
||||
data_root text NOT NULL DEFAULT '/data/yg-ft',
|
||||
model_root text NOT NULL DEFAULT '/data/yg-ft/models',
|
||||
log_root text NOT NULL DEFAULT '/opt/yg-ft/logs/training',
|
||||
api_version varchar(40) NOT NULL DEFAULT 'v1',
|
||||
capabilities jsonb NOT NULL DEFAULT '[]'::jsonb,
|
||||
description text,
|
||||
health_detail jsonb NOT NULL DEFAULT '{}'::jsonb,
|
||||
agent_version varchar(80),
|
||||
gpu_count integer NOT NULL DEFAULT 0,
|
||||
last_heartbeat_at timestamptz,
|
||||
@@ -1159,116 +1147,6 @@ CREATE INDEX IF NOT EXISTS idx_gpu_allocations_scope ON gpu_allocations(tenant_i
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS uq_gpu_allocations_active_gpu
|
||||
ON gpu_allocations(gpu_device_id) WHERE released_at IS NULL;
|
||||
|
||||
-- Multi compute-node scheduling and local-cache metadata.
|
||||
-- Each GPU server is modeled as one compute node. Nodes do not call each other;
|
||||
-- the application platform schedules jobs and syncs resources through each node's File Gateway.
|
||||
ALTER TABLE compute_nodes ADD COLUMN IF NOT EXISTS file_gateway_url text;
|
||||
ALTER TABLE compute_nodes ADD COLUMN IF NOT EXISTS enabled boolean NOT NULL DEFAULT true;
|
||||
ALTER TABLE compute_nodes ADD COLUMN IF NOT EXISTS scheduler_status varchar(40) NOT NULL DEFAULT 'online';
|
||||
ALTER TABLE compute_nodes ADD COLUMN IF NOT EXISTS scheduler_weight integer NOT NULL DEFAULT 100 CHECK (scheduler_weight >= 0);
|
||||
ALTER TABLE compute_nodes ADD COLUMN IF NOT EXISTS tags text[] NOT NULL DEFAULT '{}';
|
||||
ALTER TABLE compute_nodes ADD COLUMN IF NOT EXISTS service_token_encrypted text;
|
||||
ALTER TABLE compute_nodes ADD COLUMN IF NOT EXISTS data_root text NOT NULL DEFAULT '/data/yg-ft';
|
||||
ALTER TABLE compute_nodes ADD COLUMN IF NOT EXISTS model_root text NOT NULL DEFAULT '/data/yg-ft/models';
|
||||
ALTER TABLE compute_nodes ADD COLUMN IF NOT EXISTS log_root text NOT NULL DEFAULT '/opt/yg-ft/logs/compute';
|
||||
ALTER TABLE compute_nodes ADD COLUMN IF NOT EXISTS max_parallel_jobs integer NOT NULL DEFAULT 1 CHECK (max_parallel_jobs >= 0);
|
||||
ALTER TABLE compute_nodes ADD COLUMN IF NOT EXISTS current_running_jobs integer NOT NULL DEFAULT 0 CHECK (current_running_jobs >= 0);
|
||||
ALTER TABLE compute_nodes ADD COLUMN IF NOT EXISTS last_health_check_at timestamptz;
|
||||
ALTER TABLE compute_nodes ADD COLUMN IF NOT EXISTS health_detail jsonb NOT NULL DEFAULT '{}'::jsonb;
|
||||
ALTER TABLE compute_nodes ADD COLUMN IF NOT EXISTS drain_reason text;
|
||||
CREATE INDEX IF NOT EXISTS idx_compute_nodes_scheduler
|
||||
ON compute_nodes(enabled, scheduler_status, scheduler_weight DESC, last_health_check_at DESC);
|
||||
CREATE INDEX IF NOT EXISTS idx_compute_nodes_tags_gin
|
||||
ON compute_nodes USING gin(tags);
|
||||
|
||||
ALTER TABLE compute_jobs ADD COLUMN IF NOT EXISTS scheduler_mode varchar(40) NOT NULL DEFAULT 'auto';
|
||||
ALTER TABLE compute_jobs ADD COLUMN IF NOT EXISTS requested_node_id uuid REFERENCES compute_nodes(id) ON DELETE SET NULL;
|
||||
ALTER TABLE compute_jobs ADD COLUMN IF NOT EXISTS assigned_at timestamptz;
|
||||
ALTER TABLE compute_jobs ADD COLUMN IF NOT EXISTS scheduler_reason text;
|
||||
CREATE INDEX IF NOT EXISTS idx_compute_jobs_requested_node
|
||||
ON compute_jobs(requested_node_id, status, created_at DESC);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS compute_node_engines (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
compute_node_id uuid NOT NULL REFERENCES compute_nodes(id) ON DELETE CASCADE,
|
||||
engine_id uuid REFERENCES training_engines(id) ON DELETE SET NULL,
|
||||
engine_code varchar(80) NOT NULL,
|
||||
engine_version varchar(80),
|
||||
home_path text,
|
||||
status varchar(40) NOT NULL DEFAULT 'available',
|
||||
capability jsonb NOT NULL DEFAULT '{}'::jsonb,
|
||||
last_health_check_at timestamptz,
|
||||
health_detail jsonb NOT NULL DEFAULT '{}'::jsonb,
|
||||
created_at timestamptz NOT NULL DEFAULT now(),
|
||||
updated_at timestamptz NOT NULL DEFAULT now(),
|
||||
UNIQUE (compute_node_id, engine_code)
|
||||
);
|
||||
SELECT touch_updated_at('compute_node_engines');
|
||||
CREATE INDEX IF NOT EXISTS idx_compute_node_engines_node_status
|
||||
ON compute_node_engines(compute_node_id, status, engine_code);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS resource_replicas (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
tenant_id uuid REFERENCES tenants(id) ON DELETE SET NULL,
|
||||
project_id uuid REFERENCES projects(id) ON DELETE SET NULL,
|
||||
resource_type varchar(80) NOT NULL,
|
||||
resource_id uuid NOT NULL,
|
||||
storage_object_id uuid REFERENCES storage_objects(id) ON DELETE SET NULL,
|
||||
compute_node_id uuid NOT NULL REFERENCES compute_nodes(id) ON DELETE CASCADE,
|
||||
local_path text NOT NULL,
|
||||
status varchar(40) NOT NULL DEFAULT 'available',
|
||||
sync_status varchar(40) NOT NULL DEFAULT 'synced',
|
||||
checksum_sha256 char(64),
|
||||
byte_size bigint NOT NULL DEFAULT 0 CHECK (byte_size >= 0),
|
||||
version varchar(120),
|
||||
pinned boolean NOT NULL DEFAULT false,
|
||||
last_verified_at timestamptz,
|
||||
expires_at timestamptz,
|
||||
failure_reason text,
|
||||
metadata jsonb NOT NULL DEFAULT '{}'::jsonb,
|
||||
created_at timestamptz NOT NULL DEFAULT now(),
|
||||
updated_at timestamptz NOT NULL DEFAULT now(),
|
||||
UNIQUE (resource_type, resource_id, compute_node_id)
|
||||
);
|
||||
SELECT touch_updated_at('resource_replicas');
|
||||
CREATE INDEX IF NOT EXISTS idx_resource_replicas_resource
|
||||
ON resource_replicas(resource_type, resource_id, status);
|
||||
CREATE INDEX IF NOT EXISTS idx_resource_replicas_node_status
|
||||
ON resource_replicas(compute_node_id, status, sync_status, updated_at DESC);
|
||||
CREATE INDEX IF NOT EXISTS idx_resource_replicas_scope
|
||||
ON resource_replicas(tenant_id, project_id, resource_type, updated_at DESC);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS resource_sync_jobs (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
tenant_id uuid REFERENCES tenants(id) ON DELETE SET NULL,
|
||||
project_id uuid REFERENCES projects(id) ON DELETE SET NULL,
|
||||
target_compute_node_id uuid NOT NULL REFERENCES compute_nodes(id) ON DELETE CASCADE,
|
||||
source_compute_node_id uuid REFERENCES compute_nodes(id) ON DELETE SET NULL,
|
||||
resource_type varchar(80) NOT NULL,
|
||||
resource_id uuid NOT NULL,
|
||||
storage_object_id uuid REFERENCES storage_objects(id) ON DELETE SET NULL,
|
||||
status task_status NOT NULL DEFAULT 'pending',
|
||||
transfer_mode varchar(40) NOT NULL DEFAULT 'app_proxy',
|
||||
source_uri text,
|
||||
target_path text NOT NULL,
|
||||
byte_size bigint NOT NULL DEFAULT 0 CHECK (byte_size >= 0),
|
||||
checksum_sha256 char(64),
|
||||
progress numeric(5,2) NOT NULL DEFAULT 0 CHECK (progress >= 0 AND progress <= 100),
|
||||
failure_reason text,
|
||||
requested_by uuid REFERENCES users(id) ON DELETE SET NULL,
|
||||
started_at timestamptz,
|
||||
completed_at timestamptz,
|
||||
created_at timestamptz NOT NULL DEFAULT now(),
|
||||
updated_at timestamptz NOT NULL DEFAULT now()
|
||||
);
|
||||
SELECT touch_updated_at('resource_sync_jobs');
|
||||
CREATE INDEX IF NOT EXISTS idx_resource_sync_jobs_status
|
||||
ON resource_sync_jobs(status, created_at DESC);
|
||||
CREATE INDEX IF NOT EXISTS idx_resource_sync_jobs_target
|
||||
ON resource_sync_jobs(target_compute_node_id, status, created_at DESC);
|
||||
CREATE INDEX IF NOT EXISTS idx_resource_sync_jobs_resource
|
||||
ON resource_sync_jobs(resource_type, resource_id, status);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS resource_acl (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
tenant_id uuid REFERENCES tenants(id) ON DELETE CASCADE,
|
||||
@@ -1446,15 +1324,10 @@ ALTER TABLE fine_tune_tasks ADD COLUMN IF NOT EXISTS tenant_id uuid REFERENCES t
|
||||
ALTER TABLE fine_tune_tasks ADD COLUMN IF NOT EXISTS project_id uuid REFERENCES projects(id) ON DELETE SET NULL;
|
||||
ALTER TABLE fine_tune_tasks ADD COLUMN IF NOT EXISTS owner_id uuid REFERENCES users(id) ON DELETE SET NULL;
|
||||
ALTER TABLE fine_tune_tasks ADD COLUMN IF NOT EXISTS approval_status approval_status NOT NULL DEFAULT 'not_required';
|
||||
ALTER TABLE fine_tune_tasks ADD COLUMN IF NOT EXISTS compute_node_id uuid REFERENCES compute_nodes(id) ON DELETE SET NULL;
|
||||
ALTER TABLE fine_tune_tasks ADD COLUMN IF NOT EXISTS compute_job_id uuid REFERENCES compute_jobs(id) ON DELETE SET NULL;
|
||||
ALTER TABLE fine_tune_tasks ADD COLUMN IF NOT EXISTS resume_checkpoint_id uuid REFERENCES fine_tune_checkpoints(id) ON DELETE SET NULL;
|
||||
CREATE INDEX IF NOT EXISTS idx_fine_tune_tasks_scope_status
|
||||
ON fine_tune_tasks(tenant_id, project_id, status, created_at DESC) WHERE deleted_at IS NULL;
|
||||
CREATE INDEX IF NOT EXISTS idx_fine_tune_tasks_compute_job
|
||||
ON fine_tune_tasks(compute_job_id) WHERE deleted_at IS NULL;
|
||||
CREATE INDEX IF NOT EXISTS idx_fine_tune_tasks_node_status
|
||||
ON fine_tune_tasks(compute_node_id, status, created_at DESC) WHERE deleted_at IS NULL;
|
||||
|
||||
ALTER TABLE inference_tasks ADD COLUMN IF NOT EXISTS tenant_id uuid REFERENCES tenants(id) ON DELETE SET NULL;
|
||||
ALTER TABLE inference_tasks ADD COLUMN IF NOT EXISTS project_id uuid REFERENCES projects(id) ON DELETE SET NULL;
|
||||
|
||||
@@ -431,9 +431,9 @@ Expected: 三项检查全部 PASS。
|
||||
|
||||
- [ ] **Step 3: 启动页面并逐步验证四步交互**
|
||||
|
||||
Run: `cd frontend && npm run dev -- --host 0.0.0.0 --port 16801`
|
||||
Run: `cd frontend && npm run dev -- --host 0.0.0.0 --port 6801`
|
||||
|
||||
Browser checks at `http://localhost:16801/data-process/create`:
|
||||
Browser checks at `http://localhost:6801/data-process/create`:
|
||||
|
||||
1. 第一步上传文本并选择非结构化数据。
|
||||
2. 第二步点击至少三个右侧切片，确认左侧滚动目标和高亮范围变化。
|
||||
|
||||
@@ -156,7 +156,7 @@
|
||||
- API Key 管理。
|
||||
- OpenAPI 文档。
|
||||
- 请求限流。
|
||||
- Webhook 回调作为外部系统集成的可选能力，不作为算力状态同步默认方案。
|
||||
- Webhook 回调。
|
||||
- 外部系统发起训练、评测、推理。
|
||||
|
||||
### 1.11 企业统一身份认证
|
||||
@@ -412,11 +412,11 @@ YG_FT/
|
||||
|
||||
联调接口：
|
||||
|
||||
- `GET /modelTF/users`
|
||||
- `POST /modelTF/users`
|
||||
- `PUT /modelTF/users/{id}`
|
||||
- `DELETE /modelTF/users/{id}`
|
||||
- `PUT /modelTF/users/{id}/password`
|
||||
- `GET /api/users`
|
||||
- `POST /api/users`
|
||||
- `PUT /api/users/{id}`
|
||||
- `DELETE /api/users/{id}`
|
||||
- `PUT /api/users/{id}/password`
|
||||
|
||||
#### 项目空间
|
||||
|
||||
@@ -439,12 +439,12 @@ YG_FT/
|
||||
|
||||
联调接口：
|
||||
|
||||
- `GET /modelTF/projects`
|
||||
- `POST /modelTF/projects`
|
||||
- `GET /modelTF/projects/{id}`
|
||||
- `PUT /modelTF/projects/{id}`
|
||||
- `GET /modelTF/projects/{id}/members`
|
||||
- `POST /modelTF/projects/{id}/members`
|
||||
- `GET /api/projects`
|
||||
- `POST /api/projects`
|
||||
- `GET /api/projects/{id}`
|
||||
- `PUT /api/projects/{id}`
|
||||
- `GET /api/projects/{id}/members`
|
||||
- `POST /api/projects/{id}/members`
|
||||
|
||||
#### 审批中心
|
||||
|
||||
@@ -466,10 +466,10 @@ YG_FT/
|
||||
|
||||
联调接口：
|
||||
|
||||
- `GET /modelTF/approvals`
|
||||
- `POST /modelTF/approvals/{id}/approve`
|
||||
- `POST /modelTF/approvals/{id}/reject`
|
||||
- `POST /modelTF/approvals/{id}/cancel`
|
||||
- `GET /api/approvals`
|
||||
- `POST /api/approvals/{id}/approve`
|
||||
- `POST /api/approvals/{id}/reject`
|
||||
- `POST /api/approvals/{id}/cancel`
|
||||
|
||||
#### 算力资源中心
|
||||
|
||||
@@ -487,16 +487,13 @@ YG_FT/
|
||||
- 任务队列。
|
||||
- Agent 健康状态。
|
||||
- 租户/项目配额。
|
||||
- 算力节点新增、编辑、连接测试、启用、禁用、维护模式。
|
||||
- 节点权重、标签、训练引擎版本和本地资源副本。
|
||||
- 自动调度和手动指定节点入口。
|
||||
|
||||
联调接口：
|
||||
|
||||
- `GET /modelTF/compute/gpus`
|
||||
- `GET /modelTF/compute/queue`
|
||||
- `GET /modelTF/compute/nodes`
|
||||
- `GET /modelTF/quotas/usage`
|
||||
- `GET /api/compute/gpus`
|
||||
- `GET /api/compute/queue`
|
||||
- `GET /api/compute/nodes`
|
||||
- `GET /api/quotas/usage`
|
||||
|
||||
### 5.2 现有页面改造
|
||||
|
||||
@@ -532,9 +529,9 @@ YG_FT/
|
||||
|
||||
交付接口：
|
||||
|
||||
- `POST /modelTF/login`
|
||||
- `POST /modelTF/logout`
|
||||
- `GET /modelTF/me`
|
||||
- `POST /api/login`
|
||||
- `POST /api/logout`
|
||||
- `GET /api/me`
|
||||
|
||||
#### Tenant / Project 模块
|
||||
|
||||
@@ -548,9 +545,9 @@ YG_FT/
|
||||
|
||||
交付接口：
|
||||
|
||||
- `/modelTF/tenants`
|
||||
- `/modelTF/projects`
|
||||
- `/modelTF/projects/{id}/members`
|
||||
- `/api/tenants`
|
||||
- `/api/projects`
|
||||
- `/api/projects/{id}/members`
|
||||
|
||||
#### Resource ACL 模块
|
||||
|
||||
@@ -562,35 +559,24 @@ YG_FT/
|
||||
|
||||
交付接口：
|
||||
|
||||
- `GET /modelTF/resources/{resource_type}/{resource_id}/acl`
|
||||
- `PUT /modelTF/resources/{resource_type}/{resource_id}/acl`
|
||||
- `GET /api/resources/{resource_type}/{resource_id}/acl`
|
||||
- `PUT /api/resources/{resource_type}/{resource_id}/acl`
|
||||
|
||||
#### Compute Gateway 模块
|
||||
|
||||
开发内容：
|
||||
|
||||
- 算力平台客户端。
|
||||
- 算力节点管理：地址、File Gateway、权重、标签、启用状态、维护状态。
|
||||
- 自动/手动调度策略。
|
||||
- 调度前检查模型/数据集在目标节点的资源副本。
|
||||
- 缺失资源时创建同步任务，通过目标节点 File Gateway 写入本地磁盘。
|
||||
- 创建 compute job。
|
||||
- 查询状态和日志。
|
||||
- 应用侧定时轮询 Compute API，同步任务状态、日志摘要和产物索引。
|
||||
- 状态回调验签。
|
||||
- 任务状态映射。
|
||||
|
||||
交付接口：
|
||||
|
||||
- `GET /modelTF/compute/gpus`
|
||||
- `GET /modelTF/compute/queue`
|
||||
- `GET/POST/PUT /modelTF/compute/nodes`
|
||||
- `POST /modelTF/compute/nodes/{id}/test-connection`
|
||||
- `POST /modelTF/compute/nodes/{id}/enable`
|
||||
- `POST /modelTF/compute/nodes/{id}/disable`
|
||||
- `POST /modelTF/compute/nodes/{id}/drain`
|
||||
- `GET /modelTF/compute/nodes/{id}/replicas`
|
||||
- `POST /modelTF/internal/compute-sync/jobs/poll`
|
||||
- `POST /modelTF/internal/compute-sync/resources`
|
||||
- `GET /api/compute/gpus`
|
||||
- `GET /api/compute/queue`
|
||||
- `POST /api/internal/compute-callbacks/jobs`
|
||||
|
||||
#### Fine Tune 模块
|
||||
|
||||
@@ -606,7 +592,7 @@ YG_FT/
|
||||
|
||||
交付接口：
|
||||
|
||||
- 沿用 `docs/backend-api-design.md` 的 `/modelTF/fine-tune` 系列。
|
||||
- 沿用 `docs/backend-api-design.md` 的 `/api/fine-tune` 系列。
|
||||
|
||||
#### Approval 模块
|
||||
|
||||
@@ -619,8 +605,8 @@ YG_FT/
|
||||
|
||||
交付接口：
|
||||
|
||||
- `/modelTF/approvals`
|
||||
- `/modelTF/approval-templates`
|
||||
- `/api/approvals`
|
||||
- `/api/approval-templates`
|
||||
|
||||
#### Audit 模块
|
||||
|
||||
@@ -634,8 +620,8 @@ YG_FT/
|
||||
|
||||
交付接口：
|
||||
|
||||
- `GET /modelTF/audit-logs`
|
||||
- `GET /modelTF/login-logs`
|
||||
- `GET /api/audit-logs`
|
||||
- `GET /api/login-logs`
|
||||
|
||||
### 6.2 P1 模块
|
||||
|
||||
@@ -660,13 +646,13 @@ YG_FT/
|
||||
|
||||
接口：
|
||||
|
||||
- `POST /modelTF/compute/jobs`
|
||||
- `GET /modelTF/compute/jobs/{id}`
|
||||
- `POST /modelTF/compute/jobs/{id}/stop`
|
||||
- `GET /modelTF/compute/jobs/{id}/logs`
|
||||
- `GET /modelTF/compute/resources/gpus`
|
||||
- `POST /modelTF/compute/files/upload`
|
||||
- `GET /modelTF/compute/files/{id}/download`
|
||||
- `POST /compute/jobs`
|
||||
- `GET /compute/jobs/{id}`
|
||||
- `POST /compute/jobs/{id}/stop`
|
||||
- `GET /compute/jobs/{id}/logs`
|
||||
- `GET /compute/resources/gpus`
|
||||
- `POST /compute/files/upload`
|
||||
- `GET /compute/files/{id}/download`
|
||||
|
||||
职责：
|
||||
|
||||
@@ -674,9 +660,7 @@ YG_FT/
|
||||
- 校验服务 token。
|
||||
- 调用 Agent。
|
||||
- 聚合状态。
|
||||
- 提供任务状态查询接口，供应用平台定时轮询。
|
||||
- 每个单机多 GPU 算力节点都部署一套 Compute API，不依赖其他算力节点。
|
||||
- 暴露节点健康、GPU、训练引擎、资源副本和文件网关状态。
|
||||
- 回调应用平台。
|
||||
|
||||
### 7.2 Compute Agent
|
||||
|
||||
@@ -701,9 +685,7 @@ YG_FT/
|
||||
职责：
|
||||
|
||||
- 生成训练 YAML 或 CLI。
|
||||
- 根据平台数据集 ID 生成 LLaMA-Factory dataset key 和 `{dataset_dir}/dataset_info.json`。
|
||||
- 单文件数据集使用一个 dataset key，多文件数据集拆分为多个 key 并通过逗号组合传入 `--dataset`。
|
||||
- 正式启动前由应用侧同步当前数据集文件到目标算力节点，确保训练使用最新激活版本。
|
||||
- 生成 dataset_info。
|
||||
- 设置 `CUDA_VISIBLE_DEVICES`。
|
||||
- 启动训练。
|
||||
- 解析日志。
|
||||
@@ -753,12 +735,9 @@ YG_FT/
|
||||
| `quota_usage` | 配额使用 |
|
||||
| `storage_nodes` | 存储节点 |
|
||||
| `compute_nodes` | 算力节点 |
|
||||
| `compute_node_engines` | 算力节点训练引擎能力 |
|
||||
| `gpu_devices` | GPU 设备 |
|
||||
| `gpu_allocations` | GPU 分配记录 |
|
||||
| `compute_jobs` | 算力任务 |
|
||||
| `resource_replicas` | 数据集/模型/产物在算力节点的本地副本 |
|
||||
| `resource_sync_jobs` | 应用平台编排的资源同步任务 |
|
||||
| `training_engines` | 训练引擎 |
|
||||
| `retention_policies` | 保留策略 |
|
||||
| `cleanup_jobs` | 清理任务 |
|
||||
@@ -803,14 +782,12 @@ YG_FT/
|
||||
- `file-gateway`
|
||||
- `llama-factory-env`
|
||||
|
||||
多算力节点阶段，每台单机多 GPU 服务器都部署以上组件和宿主机挂载的 LLaMA-Factory。算力节点之间默认不互相访问，由应用平台统一调度和同步资源。
|
||||
|
||||
配置：
|
||||
|
||||
- `COMPUTE_NODE_ID`
|
||||
- `SERVICE_TOKEN`
|
||||
- `ENABLE_APP_CALLBACK=false`
|
||||
- `DATA_ROOT=/data/yg-ft`
|
||||
- `APP_CALLBACK_URL`
|
||||
- `DATA_ROOT=/data/ft-platform`
|
||||
- `LLAMA_FACTORY_PATH`
|
||||
- `PYTHON_ENV_PATH`
|
||||
- `GPU_VISIBLE_DEVICES`
|
||||
@@ -821,7 +798,7 @@ YG_FT/
|
||||
|
||||
- 前端只访问应用平台。
|
||||
- 应用平台可访问算力平台内部 API。
|
||||
- 第一阶段只开通应用平台主动访问算力平台内部 API，状态同步采用应用侧轮询。
|
||||
- 算力平台可回调应用平台 internal callback。
|
||||
- 算力平台不直接暴露给公网。
|
||||
- 文件下载通过应用平台签发令牌。
|
||||
|
||||
@@ -962,9 +939,9 @@ YG_FT/
|
||||
|
||||
### 算力组
|
||||
|
||||
- CE-A：Compute API、Agent、GPU 调度、多节点健康检查。
|
||||
- CE-A：Compute API、Agent、GPU 调度。
|
||||
- CE-B：LLaMA-Factory Adapter、日志解析、产物管理。
|
||||
- CE-C：File Gateway、本地磁盘、离线导入、资源副本同步。
|
||||
- CE-C：File Gateway、本地磁盘、离线导入。
|
||||
|
||||
### 数据库/部署组
|
||||
|
||||
@@ -977,8 +954,7 @@ YG_FT/
|
||||
| --- | --- | --- |
|
||||
| M1 基础治理 | 第 1-2 周 | 登录、用户、租户、项目、权限 |
|
||||
| M2 资源管理 | 第 3-4 周 | 模型、数据集、文件网关、离线导入 |
|
||||
| M3 训练主链路 | 第 5-7 周 | GPU 调度、LLaMA-Factory 训练、日志指标、产物、资源副本检查 |
|
||||
| M3.5 多节点预留 | 第 7-8 周 | compute_nodes 管理、节点权重/标签、自动/手动调度、资源同步任务 |
|
||||
| M3 训练主链路 | 第 5-7 周 | GPU 调度、LLaMA-Factory 训练、日志指标、产物 |
|
||||
| M4 数据处理和评测 | 第 8-9 周 | 数据处理、质量评分、自动评测 |
|
||||
| M5 推理发布 | 第 10-11 周 | 推理服务、模型对比、生产发布审批 |
|
||||
| M6 企业治理收口 | 第 12 周 | 审批、审计、配额、清理、部署文档 |
|
||||
@@ -991,8 +967,7 @@ YG_FT/
|
||||
- 项目成员只能访问授权项目资源。
|
||||
- 模型和数据集支持资源级授权。
|
||||
- 上传数据集后可预览、编辑版本、用于训练。
|
||||
- 训练任务可自动调度节点/GPU，也可由管理员手动指定节点/GPU 并成功运行。
|
||||
- 调度前可识别目标节点是否已有数据集和模型副本，缺失时能创建同步任务。
|
||||
- 训练任务可指定 GPU 并成功运行。
|
||||
- 训练日志和指标实时可见。
|
||||
- 训练产物可登记、合并、发布测试服务。
|
||||
- 评测任务可生成样本级结果。
|
||||
@@ -1032,7 +1007,6 @@ YG_FT/
|
||||
- `docs/postgres-schema.sql`：初始数据库模型。
|
||||
- `docs/platform-architecture-requirements.md`：架构与功能需求补充。
|
||||
- `docs/system-development-plan.md`：多人开发拆分和实施计划。
|
||||
- `docs/team-development-plan.md`：3-4 人并行开发人员分工、模块边界、接口范围和里程碑。
|
||||
|
||||
后续建议再补两份执行文档：
|
||||
|
||||
@@ -1041,72 +1015,72 @@ YG_FT/
|
||||
|
||||
## 16. 页面模块开发工作包
|
||||
|
||||
本节用于多人并行开发时认领任务。每个工作包都标明对应页面、前端内容、后端接口、DB 表和部署/算力依赖。当前菜单、二级路由、规划菜单和接口/数据库映射总览见 `docs/menu-functional-requirements.md`；按 3-4 人落地的具体人员边界和开发节奏见 `docs/team-development-plan.md`。
|
||||
本节用于多人并行开发时认领任务。每个工作包都标明对应页面、前端内容、后端接口、DB 表和部署/算力依赖。
|
||||
|
||||
### 16.1 基础入口与用户权限
|
||||
|
||||
| 工作包 | 页面/路由 | 前端开发 | 后端开发 | DB 关联 | 依赖 |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 登录和会话 | `/login` | 登录表单、错误提示、登录后跳转、会话过期处理 | `POST /modelTF/login`、`GET /modelTF/me`、JWT 中间件 | `users`、`login_sessions`、`v_user_effective_permissions` | 无 |
|
||||
| 登录和会话 | `/login` | 登录表单、错误提示、登录后跳转、会话过期处理 | `POST /api/login`、`GET /api/me`、JWT 中间件 | `users`、`login_sessions`、`v_user_effective_permissions` | 无 |
|
||||
| 主布局和权限菜单 | `/` | 菜单按权限过滤、用户信息、项目切换器 | 当前用户权限、当前项目上下文 | `permissions`、`role_permissions`、`project_members` | 登录 |
|
||||
| 用户中心 | `/user-settings`、`/user-settings/create`、`/user-settings/:id/permission` | 用户列表、创建、禁用、重置密码、页面权限 | `/modelTF/users` 系列 | `users`、`user_permissions`、`tenant_users` | 租户/项目 |
|
||||
| 用户中心 | `/user-settings`、`/user-settings/create`、`/user-settings/:id/permission` | 用户列表、创建、禁用、重置密码、页面权限 | `/api/users` 系列 | `users`、`user_permissions`、`tenant_users` | 租户/项目 |
|
||||
| 无权限页 | `/permission-denied` | 无权限说明、返回入口 | 权限异常返回统一错误码 | 无新增 | 主布局 |
|
||||
|
||||
### 16.2 租户和项目空间
|
||||
|
||||
| 工作包 | 页面/路由 | 前端开发 | 后端开发 | DB 关联 | 依赖 |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 租户管理 | `/tenants`、`/tenants/:id` | 租户列表、创建/禁用、配额、留存策略 | `/modelTF/tenants` 系列 | `tenants`、`tenant_users`、`quotas`、`retention_policies` | 用户中心 |
|
||||
| 项目列表和详情 | `/projects`、`/projects/:id` | 项目列表、创建、归档、资源概览 | `/modelTF/projects` 系列 | `projects`、`quota_usage` | 租户 |
|
||||
| 项目成员 | `/projects/:id/members` | 成员列表、添加成员、角色选择 | `/modelTF/projects/{id}/members` 系列 | `project_members` | 项目 |
|
||||
| 资源授权 | `/projects/:id/permissions`、资源详情弹窗 | ACL 表格、用户/角色授权 | `/modelTF/resources/{type}/{id}/acl` | `resource_acl` | 项目、资源表 |
|
||||
| 租户管理 | `/tenants`、`/tenants/:id` | 租户列表、创建/禁用、配额、留存策略 | `/api/tenants` 系列 | `tenants`、`tenant_users`、`quotas`、`retention_policies` | 用户中心 |
|
||||
| 项目列表和详情 | `/projects`、`/projects/:id` | 项目列表、创建、归档、资源概览 | `/api/projects` 系列 | `projects`、`quota_usage` | 租户 |
|
||||
| 项目成员 | `/projects/:id/members` | 成员列表、添加成员、角色选择 | `/api/projects/{id}/members` 系列 | `project_members` | 项目 |
|
||||
| 资源授权 | `/projects/:id/permissions`、资源详情弹窗 | ACL 表格、用户/角色授权 | `/api/resources/{type}/{id}/acl` | `resource_acl` | 项目、资源表 |
|
||||
|
||||
### 16.3 数据集和数据处理
|
||||
|
||||
| 工作包 | 页面/路由 | 前端开发 | 后端开发 | DB 关联 | 依赖 |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 数据集列表 | `/dataset` | 列表、搜索、项目过滤、下载、删除审批入口 | `GET/DELETE /modelTF/dataset-manage` | `datasets`、`resource_acl`、`approval_instances` | 项目上下文 |
|
||||
| 数据集创建/上传 | `/dataset/create`、`/dataset/:id/edit` | 表单、分片上传、离线导入弹窗 | `POST /modelTF/dataset-manage`、`POST /modelTF/files/upload-session`、`POST /modelTF/import/local-dataset` | `datasets`、`dataset_files`、`storage_objects`、`file_upload_sessions`、`local_import_jobs` | 文件网关 |
|
||||
| 数据集预览/版本 | `/dataset/:id/preview` | 文件列表、内容预览、版本栏、在线编辑 | `/modelTF/dataset-manage/preview`、`/versions` 系列 | `dataset_file_versions`、`dataset_records` | 数据集上传 |
|
||||
| 数据处理列表 | `/data-process` | 任务列表、状态、输出数据集跳转 | `/modelTF/data-process` 系列 | `data_process_tasks` | 数据集 |
|
||||
| 数据处理创建向导 | `/data-process/create` | 任务配置、模型选择、源文件、预览切片、生成、结果编辑、发布 | `/modelTF/data-process/{id}/source-files`、`preview/build`、`generate`、`publish` | `data_process_source_files`、`data_process_preview_items`、`data_process_results` | 数据集、模型、文件网关 |
|
||||
| 数据处理详情 | `/data-process/:id` | 运行统计、失败原因、结果表格 | `GET /modelTF/data-process/{id}`、`results`、`events` | `data_process_tasks`、`data_process_results` | 数据处理任务 |
|
||||
| 数据转换 | `/data-convert` | 文件上传、转换状态、下载 | `/modelTF/data-convert/jobs` 系列 | `data_convert_jobs`、`storage_objects` | 文件网关 |
|
||||
| 数据集列表 | `/dataset` | 列表、搜索、项目过滤、下载、删除审批入口 | `GET/DELETE /api/dataset-manage` | `datasets`、`resource_acl`、`approval_instances` | 项目上下文 |
|
||||
| 数据集创建/上传 | `/dataset/create`、`/dataset/:id/edit` | 表单、分片上传、离线导入弹窗 | `POST /api/dataset-manage`、`POST /api/files/upload-session`、`POST /api/import/local-dataset` | `datasets`、`dataset_files`、`storage_objects`、`file_upload_sessions`、`local_import_jobs` | 文件网关 |
|
||||
| 数据集预览/版本 | `/dataset/:id/preview` | 文件列表、内容预览、版本栏、在线编辑 | `/api/dataset-manage/preview`、`/versions` 系列 | `dataset_file_versions`、`dataset_records` | 数据集上传 |
|
||||
| 数据处理列表 | `/data-process` | 任务列表、状态、输出数据集跳转 | `/api/data-process` 系列 | `data_process_tasks` | 数据集 |
|
||||
| 数据处理创建向导 | `/data-process/create` | 任务配置、模型选择、源文件、预览切片、生成、结果编辑、发布 | `/api/data-process/{id}/source-files`、`preview/build`、`generate`、`publish` | `data_process_source_files`、`data_process_preview_items`、`data_process_results` | 数据集、模型、文件网关 |
|
||||
| 数据处理详情 | `/data-process/:id` | 运行统计、失败原因、结果表格 | `GET /api/data-process/{id}`、`results`、`events` | `data_process_tasks`、`data_process_results` | 数据处理任务 |
|
||||
| 数据转换 | `/data-convert` | 文件上传、转换状态、下载 | `/api/data-convert/jobs` 系列 | `data_convert_jobs`、`storage_objects` | 文件网关 |
|
||||
|
||||
### 16.4 模型管理和训练
|
||||
|
||||
| 工作包 | 页面/路由 | 前端开发 | 后端开发 | DB 关联 | 依赖 |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 模型列表 | `/model-manage` | 列表、用途标签、授权、导出、删除审批 | `/modelTF/model-manage` 系列 | `models`、`trained_models`、`resource_acl` | 项目上下文 |
|
||||
| 模型创建/离线导入 | `/model-manage/create`、`/model-manage/:id/edit` | 本地/API 模型表单、离线导入 | `POST/PUT /modelTF/model-manage`、`POST /modelTF/import/local-model` | `models`、`storage_objects`、`local_import_jobs` | 文件网关 |
|
||||
| 权重合并 | `/model-manage/merge` | 选择训练产物、合并状态 | `POST /modelTF/model-manage/merge`、`GET /modelTF/compute/jobs/{id}` | `trained_models`、`compute_jobs` | 算力平台 |
|
||||
| 微调列表 | `/fine-tune` | 任务列表、停止、删除、日志入口 | `/modelTF/fine-tune` 系列 | `fine_tune_tasks` | 模型、数据集 |
|
||||
| 微调创建 | `/fine-tune/create` | 训练参数、GPU 选择、命令预览、审批提示 | `POST /modelTF/fine-tune`、`POST /modelTF/fine-tune/start`、`GET /modelTF/compute/gpus` | `fine_tune_tasks`、`compute_jobs`、`gpu_allocations` | 算力平台、审批 |
|
||||
| 训练日志和 checkpoint | `/training-log/:id` | 日志 tail、指标曲线、checkpoint、恢复/重试 | `GET /modelTF/fine-tune/{id}/overview`、`checkpoints`、`resume`、`retry` | `fine_tune_metrics`、`fine_tune_checkpoints` | 训练任务 |
|
||||
| 训练引擎管理 | `/training-engines` | 引擎列表、schema、健康检查 | `/modelTF/training-engines` 系列 | `training_engines` | 算力 Agent |
|
||||
| 模型列表 | `/model-manage` | 列表、用途标签、授权、导出、删除审批 | `/api/model-manage` 系列 | `models`、`trained_models`、`resource_acl` | 项目上下文 |
|
||||
| 模型创建/离线导入 | `/model-manage/create`、`/model-manage/:id/edit` | 本地/API 模型表单、离线导入 | `POST/PUT /api/model-manage`、`POST /api/import/local-model` | `models`、`storage_objects`、`local_import_jobs` | 文件网关 |
|
||||
| 权重合并 | `/model-manage/merge` | 选择训练产物、合并状态 | `POST /api/model-manage/merge`、`GET /api/compute/jobs/{id}` | `trained_models`、`compute_jobs` | 算力平台 |
|
||||
| 微调列表 | `/fine-tune` | 任务列表、停止、删除、日志入口 | `/api/fine-tune` 系列 | `fine_tune_tasks` | 模型、数据集 |
|
||||
| 微调创建 | `/fine-tune/create` | 训练参数、GPU 选择、命令预览、审批提示 | `POST /api/fine-tune`、`POST /api/fine-tune/start`、`GET /api/compute/gpus` | `fine_tune_tasks`、`compute_jobs`、`gpu_allocations` | 算力平台、审批 |
|
||||
| 训练日志和 checkpoint | `/training-log/:id` | 日志 tail、指标曲线、checkpoint、恢复/重试 | `GET /api/fine-tune/{id}/overview`、`checkpoints`、`resume`、`retry` | `fine_tune_metrics`、`fine_tune_checkpoints` | 训练任务 |
|
||||
| 训练引擎管理 | `/training-engines` | 引擎列表、schema、健康检查 | `/api/training-engines` 系列 | `training_engines` | 算力 Agent |
|
||||
|
||||
### 16.5 评测、推理和发布
|
||||
|
||||
| 工作包 | 页面/路由 | 前端开发 | 后端开发 | DB 关联 | 依赖 |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 评测列表 | `/model-eval` | 列表、分数、状态、删除 | `/modelTF/model-eval` 系列 | `eval_tasks` | 模型、数据集 |
|
||||
| 评测创建 | `/model-eval/create` | 模型/数据集/维度/GPU 选择、指标配置 | `POST /modelTF/model-eval/start`、`GET /modelTF/dimension` | `eval_tasks`、`eval_dimensions`、`compute_jobs` | 算力平台 |
|
||||
| 评测详情 | `/model-eval/:id` | 综合评价、维度汇总、样本结果、人工复核预留 | `GET /modelTF/model-eval/{id}`、`events` | `eval_sample_results`、`eval_dimension_summaries` | 评测任务 |
|
||||
| 评测维度 | `/model-eval/dimension/:id/edit` | 维度表单、Prompt 编辑 | `/modelTF/dimension` 系列 | `eval_dimensions` | 模型管理 |
|
||||
| 推理列表/创建 | `/model-inference`、`/model-inference/create` | 任务列表、模型选择、GPU 选择、加载状态 | `/modelTF/model-compare`、`load/unload` | `inference_tasks`、`inference_task_models` | 算力平台 |
|
||||
| 评测列表 | `/model-eval` | 列表、分数、状态、删除 | `/api/model-eval` 系列 | `eval_tasks` | 模型、数据集 |
|
||||
| 评测创建 | `/model-eval/create` | 模型/数据集/维度/GPU 选择、指标配置 | `POST /api/model-eval/start`、`GET /api/dimension` | `eval_tasks`、`eval_dimensions`、`compute_jobs` | 算力平台 |
|
||||
| 评测详情 | `/model-eval/:id` | 综合评价、维度汇总、样本结果、人工复核预留 | `GET /api/model-eval/{id}`、`events` | `eval_sample_results`、`eval_dimension_summaries` | 评测任务 |
|
||||
| 评测维度 | `/model-eval/dimension/:id/edit` | 维度表单、Prompt 编辑 | `/api/dimension` 系列 | `eval_dimensions` | 模型管理 |
|
||||
| 推理列表/创建 | `/model-inference`、`/model-inference/create` | 任务列表、模型选择、GPU 选择、加载状态 | `/api/model-compare`、`load/unload` | `inference_tasks`、`inference_task_models` | 算力平台 |
|
||||
| 推理对话 | `/model-inference/chat/:id` | 单模型流式对话、历史消息 | `stream-chat`、`chat-with-port` | `chat_sessions`、`chat_messages` | 推理服务 |
|
||||
| 模型对比 | `/model-compare/chat/:id`、`/model-compare/result` | 多模型对话、结果对比 | `/modelTF/model-chat/*` | `chat_sessions`、`chat_messages` | 推理任务 |
|
||||
| 模型服务治理 | `/model-services`、`/model-services/:id` | 测试/生产服务、发布申请、下线、调用统计 | `POST /modelTF/approvals`、`GET /modelTF/usage/summary` | `model_services`、`approval_instances` | 审批、算力平台 |
|
||||
| 模型对比 | `/model-compare/chat/:id`、`/model-compare/result` | 多模型对话、结果对比 | `/api/model-chat/*` | `chat_sessions`、`chat_messages` | 推理任务 |
|
||||
| 模型服务治理 | `/model-services`、`/model-services/:id` | 测试/生产服务、发布申请、下线、调用统计 | `POST /api/approvals`、`GET /api/usage/summary` | `model_services`、`approval_instances` | 审批、算力平台 |
|
||||
|
||||
### 16.6 审批、审计、算力和存储运维
|
||||
|
||||
| 工作包 | 页面/路由 | 前端开发 | 后端开发 | DB 关联 | 依赖 |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| 审批中心 | `/approvals`、`/approvals/pending`、`/approvals/mine`、`/approvals/:id` | 审批列表、详情、通过/驳回/撤回 | `/modelTF/approvals` 系列 | `approval_instances`、`approval_steps` | 用户、项目 |
|
||||
| 审批模板 | `/approval-settings` | 动作策略、审批人规则、超时配置 | `/modelTF/approval-templates` 系列 | `approval_templates` | 租户/项目 |
|
||||
| 算力资源中心 | `/compute`、`/compute/gpus`、`/compute/queue`、`/compute/nodes` | GPU 卡片、节点状态、队列、优先级、节点新增/编辑、连接测试、启用/禁用、维护模式、节点权重/标签、本地资源副本 | `/modelTF/compute/*`、`/modelTF/compute/*` 内部接口 | `compute_nodes`、`compute_node_engines`、`gpu_devices`、`compute_jobs`、`gpu_allocations`、`resource_replicas`、`resource_sync_jobs` | Compute API/Agent/File Gateway |
|
||||
| 存储管理 | `/storage` | 磁盘占用、大文件、临时文件、checkpoint 清理 | `GET /modelTF/quotas/usage`、`GET/PUT /modelTF/retention-policies` | `storage_nodes`、`storage_objects`、`cleanup_jobs`、`retention_policies` | 文件网关 |
|
||||
| 审计中心 | `/audit-logs`、`/login-logs`、`/download-logs` | 筛选、详情、导出 | `GET /modelTF/audit-logs`、`GET /modelTF/login-logs`、`GET /modelTF/download-logs` | `audit_logs`、`login_sessions` | 审计中间件 |
|
||||
| 审批中心 | `/approvals`、`/approvals/pending`、`/approvals/mine`、`/approvals/:id` | 审批列表、详情、通过/驳回/撤回 | `/api/approvals` 系列 | `approval_instances`、`approval_steps` | 用户、项目 |
|
||||
| 审批模板 | `/approval-settings` | 动作策略、审批人规则、超时配置 | `/api/approval-templates` 系列 | `approval_templates` | 租户/项目 |
|
||||
| 算力资源中心 | `/compute`、`/compute/gpus`、`/compute/queue`、`/compute/nodes` | GPU 卡片、节点状态、队列、优先级 | `/api/compute/*`、`/compute/*` 内部接口 | `compute_nodes`、`gpu_devices`、`compute_jobs`、`gpu_allocations` | Compute API/Agent |
|
||||
| 存储管理 | `/storage` | 磁盘占用、大文件、临时文件、checkpoint 清理 | `GET /api/quotas/usage`、`GET/PUT /api/retention-policies` | `storage_nodes`、`storage_objects`、`cleanup_jobs`、`retention_policies` | 文件网关 |
|
||||
| 审计中心 | `/audit-logs`、`/login-logs`、`/download-logs` | 筛选、详情、导出 | `GET /api/audit-logs`、`GET /api/login-logs`、`GET /api/download-logs` | `audit_logs`、`login_sessions` | 审计中间件 |
|
||||
|
||||
### 16.7 页面开发优先级建议
|
||||
|
||||
@@ -1118,10 +1092,3 @@ YG_FT/
|
||||
| P1 | 数据处理、评测、推理、模型服务治理 | 形成训练前后闭环 |
|
||||
| P1 | 存储管理、checkpoint、审计中心 | 企业治理和运维收口 |
|
||||
| P2 | 训练引擎管理、外部 API、OIDC/LDAP、成本核算、人工评测 | 扩展能力，第一版可预留 |
|
||||
## P1/P2/P3 Development Status
|
||||
|
||||
| Priority | Scope | Implemented content | Remaining production hardening |
|
||||
| --- | --- | --- | --- |
|
||||
| P1 | Training runtime assets | Compute scans real `checkpoint-*` directories; backend persists checkpoints to `fine_tune_checkpoints`; backend parses log metrics into `fine_tune_metrics`; completed training auto-registers a trained model with its real output directory. | Add full TensorBoard event parsing and retention/cleanup execution policies. |
|
||||
| P2 | Model merge/export | `/modelTF/model-manage/merge` submits `llamafactory-cli export` through Compute; merge/export jobs are recorded in `compute_jobs` and `model_export_jobs`; trained model artifacts are recorded in `model_artifacts`; base->trained and merge/export relations are recorded in `model_lineage`; APIs expose artifacts, lineage and export jobs. | Add frontend polling/status panels for export jobs and richer artifact checksum generation during long-running export. |
|
||||
| P3 | Compute ops | `compute_jobs` and `gpu_allocations` are updated from runtime polling; training start uses `scheduler_locks` to serialize node selection and GPU pre-allocation; terminal task states release GPU allocations; resource replica drift check and repair-pending APIs are available; `scripts/ops_diagnostics.py` checks backend, PostgreSQL, Compute health, GPU discovery and Compute jobs. | Add dedicated scheduler worker, automatic replica repair execution from authoritative source metadata, and lock timeout alarms. |
|
||||
|
||||
@@ -1,208 +0,0 @@
|
||||
# 多人并行开发分工计划
|
||||
|
||||
> 本文基于 `docs/menu-functional-requirements.md`、`docs/backend-api-design.md`、`docs/postgres-schema.sql`、`docs/system-development-plan.md` 和当前前端路由整理，用于 3-4 人并行开发。开发口径以正式系统演进为准，不以临时演示或静态 Mock 作为交付标准。
|
||||
|
||||
## 1. 分工原则
|
||||
|
||||
- 每位开发尽量独立负责一组页面、后端模块、数据库表和联调脚本，避免多人同时改同一个业务文件。
|
||||
- 公共接口契约先在 `docs/backend-api-design.md` 更新，再进入代码实现。
|
||||
- 前端 API 模块按业务域维护：`system.ts`、`model.ts`、`dataset.ts`、`fineTune.ts`、`compute.ts`、`eval.ts`、`log.ts`。
|
||||
- 后端业务代码按 `backend/app/modules/<domain>/` 拆分，路由统一挂载 `/modelTF`。
|
||||
- 数据库按迁移脚本推进，目标模型以 `docs/postgres-schema.sql` 为准，当前运行库脚本以 `backend/app/db/sql/` 为准。
|
||||
- 所有写操作必须预留审计；涉及删除、导出、发布、停止他人任务等高风险动作必须预留审批入口。
|
||||
|
||||
## 2. 4 人开发拆分
|
||||
|
||||
### A. 平台基础与企业治理
|
||||
|
||||
负责人边界：
|
||||
|
||||
- 前端目录：`frontend/src/views/login/`、`frontend/src/views/system/`，后续新增 `tenants`、`projects`、`approvals`、`audit` 页面目录。
|
||||
- 前端 API：`frontend/src/api/modules/system.ts`、`frontend/src/api/modules/log.ts`。
|
||||
- 后端模块：`auth`、`tenant`、`project`、`approval`、`audit`、`retention`、`system`。
|
||||
- 数据库表：`users`、`permissions`、`roles`、`role_permissions`、`user_permission_overrides`、`tenants`、`tenant_users`、`projects`、`project_members`、`resource_acl`、`approval_templates`、`approval_instances`、`approval_steps`、`audit_logs`、`retention_policies`。
|
||||
|
||||
对应页面和功能：
|
||||
|
||||
| 页面/模块 | 路由 | 功能点 | 接口 |
|
||||
| --- | --- | --- | --- |
|
||||
| 登录 | `/login` | 登录、Token 写入、登录失败提示、会话恢复 | `POST /modelTF/login`、`GET /modelTF/me` |
|
||||
| 用户设置 | `/user-settings`、`/user-settings/create`、`/user-settings/:id/permission` | 用户列表、创建、启停、重置密码、页面权限 | `/modelTF/users/*`、`/modelTF/permissions` |
|
||||
| 平台性能 | `/hardware` | 系统资源、进程、GPU 摘要 | `/modelTF/system-info`、`/modelTF/compute/gpus` |
|
||||
| 查看日志 | `/logs`、`/training-log/:id` | 后端日志、error 日志、训练日志索引和内容 | `/modelTF/log-files`、`/modelTF/log-content`、`/modelTF/training-log-*` |
|
||||
| 租户管理 | `/tenants`、`/tenants/:id` | 租户、配额、留存策略 | `/modelTF/tenants/*`、`/modelTF/retention-policies/*` |
|
||||
| 项目空间 | `/projects`、`/projects/:id`、`/projects/:id/members` | 项目、成员、项目角色 | `/modelTF/projects/*` |
|
||||
| 资源授权 | `/resources/:type/:id/acl` 或弹窗 | 模型/数据集/任务 ACL | `/modelTF/resources/{type}/{id}/acl` |
|
||||
| 审批中心 | `/approvals`、`/approval-settings` | 审批待办、审批历史、审批模板 | `/modelTF/approvals/*`、`/modelTF/approval-templates/*` |
|
||||
| 审计中心 | `/audit-logs` | 操作审计、登录审计、导出 | `/modelTF/audit-logs` |
|
||||
|
||||
开发计划：
|
||||
|
||||
| 阶段 | 交付内容 |
|
||||
| --- | --- |
|
||||
| 第 1 周 | 完成登录、当前用户、用户列表、权限码、日志查询接口；完善当前运行 SQL。 |
|
||||
| 第 2 周 | 完成租户、项目、项目成员、资源 ACL 后端和基础页面。 |
|
||||
| 第 3 周 | 完成审批实例、审批模板、审计日志查询和导出。 |
|
||||
| 第 4 周 | 接入其他模块写操作审计和审批拦截，补充权限测试。 |
|
||||
|
||||
验收标准：
|
||||
|
||||
- 所有业务列表按租户、项目、资源 ACL 过滤。
|
||||
- 普通用户无法访问未授权项目、模型和数据集。
|
||||
- 高风险动作有审批或管理员旁路规则。
|
||||
- 日志文件和审计日志可按时间、用户、动作、资源筛选。
|
||||
|
||||
### B. 模型资产、训练与 LLaMA-Factory 任务
|
||||
|
||||
负责人边界：
|
||||
|
||||
- 前端目录：`frontend/src/views/model/`、`frontend/src/views/fine-tune/`。
|
||||
- 前端 API：`frontend/src/api/modules/model.ts`、`frontend/src/api/modules/fineTune.ts`。
|
||||
- 后端模块：`model`、`fine_tune`、`engine_registry`、`compute_gateway` 中的训练编排部分。
|
||||
- 数据库表：`models`、`trained_models`、`fine_tune_tasks`、`fine_tune_metrics`、`fine_tune_checkpoints`、`training_engines`、`training_engine_capabilities`、`compute_jobs`、`gpu_allocations`。
|
||||
|
||||
对应页面和功能：
|
||||
|
||||
| 页面/模块 | 路由 | 功能点 | 接口 |
|
||||
| --- | --- | --- | --- |
|
||||
| 模型管理 | `/model-manage` | 基座模型、API 模型、训练产物列表、筛选、删除审批入口 | `/modelTF/model-manage`、`/modelTF/model-manage/trained-models` |
|
||||
| 添加/编辑模型 | `/model-manage/create`、`/model-manage/:id/edit` | 模型登记、本地路径/API 配置、能力标签 | `/modelTF/model-manage`、`/modelTF/model-manage/{id}` |
|
||||
| 合并权重 | `/model-manage/merge` | LoRA/Adapter 合并、产物登记 | `/modelTF/model-manage/merge` |
|
||||
| 模型训练 | `/fine-tune` | 训练任务列表、状态、启动、停止、删除审批入口 | `/modelTF/fine-tune`、`/modelTF/fine-tune/{id}/start`、`/stop` |
|
||||
| 创建训练任务 | `/fine-tune/create` | 选择模型、数据集、超参、GPU、节点策略 | `/modelTF/fine-tune`、`/modelTF/model-manage`、`/modelTF/dataset-manage`、`/modelTF/compute/*` |
|
||||
| 训练日志 | `/training-log/:id` | 实时日志、loss 曲线、checkpoint、产物 | `/modelTF/fine-tune/{id}/progress`、`/metrics`、`/checkpoints`、`/modelTF/training-log-*` |
|
||||
|
||||
开发计划：
|
||||
|
||||
| 阶段 | 交付内容 |
|
||||
| --- | --- |
|
||||
| 第 1 周 | 完成模型 CRUD、训练任务 CRUD、训练参数校验和接口联调。 |
|
||||
| 第 2 周 | 完成训练启动、停止、状态轮询、日志和指标落库。 |
|
||||
| 第 3 周 | 完成 LLaMA-Factory 参数映射、checkpoint 列表、训练产物登记。 |
|
||||
| 第 4 周 | 完成权重合并、失败恢复、权限隔离和审计接入。 |
|
||||
|
||||
验收标准：
|
||||
|
||||
- 训练任务不能绕过项目、模型、数据集权限。
|
||||
- 训练任务状态以应用侧轮询 Compute API 为主。
|
||||
- 训练命令只能由训练引擎适配层生成，不在页面或应用 API 中拼命令。
|
||||
- checkpoint、日志、产物均可追溯到任务、节点、GPU 和项目。
|
||||
|
||||
### C. 数据集、数据处理、评测与推理
|
||||
|
||||
负责人边界：
|
||||
|
||||
- 前端目录：`frontend/src/views/dataset/`、`frontend/src/views/data-process/`、`frontend/src/views/data-convert/`、`frontend/src/views/eval/`、`frontend/src/views/inference/`、`frontend/src/views/compare/`、`frontend/src/views/tools/`。
|
||||
- 前端 API：`dataset.ts`、`eval.ts`、`compare.ts`，必要时新增 `dataProcess.ts`、`dataConvert.ts`、`inference.ts`。
|
||||
- 后端模块：`dataset`、`data_process`、`eval`、`inference`、`file_gateway` 中的数据资产登记部分。
|
||||
- 数据库表：`datasets`、`dataset_files`、`dataset_file_versions`、`dataset_records`、`data_process_tasks`、`data_process_source_files`、`data_process_preview_items`、`data_process_results`、`data_convert_jobs`、`eval_tasks`、`eval_dimensions`、`eval_sample_results`、`inference_tasks`、`chat_sessions`、`chat_messages`、`custom_tools`。
|
||||
|
||||
对应页面和功能：
|
||||
|
||||
| 页面/模块 | 路由 | 功能点 | 接口 |
|
||||
| --- | --- | --- | --- |
|
||||
| 数据集管理 | `/dataset` | 数据集列表、搜索、版本、下载、删除审批入口 | `/modelTF/dataset-manage` |
|
||||
| 数据集创建/编辑 | `/dataset/create`、`/dataset/:id/edit` | 元数据、文件上传、格式识别、项目归属 | `/modelTF/dataset-manage`、`/upload/{id}` |
|
||||
| 数据集预览 | `/dataset/:id/preview` | 分页预览、在线编辑、版本对比 | `/modelTF/dataset-manage/{id}/preview`、`/versions` |
|
||||
| 数据处理 | `/data-process`、`/data-process/create`、`/data-process/:id` | 文档上传、切片、脱敏、质量评分、发布数据集 | `/modelTF/data-process/*` |
|
||||
| 数据类型转换 | `/data-convert` | JSON/JSONL/Markdown 转换任务 | `/modelTF/data-convert/jobs/*` |
|
||||
| 模型评测 | `/model-eval`、`/model-eval/create`、`/model-eval/:id` | 评测任务、维度、样本级结果、人工复核预留 | `/modelTF/model-eval/*`、`/modelTF/dimension/*` |
|
||||
| 模型推理/对比 | `/model-inference/*`、`/model-compare/*` | 模型加载、对话、对比、结果沉淀 | `/modelTF/model-chat/*`、`/modelTF/model-compare/*` |
|
||||
| 自定义工具 | `/tools`、`/tools/create`、`/tools/:id/edit` | 工具登记、参数 schema、启停 | `/modelTF/tools/*` |
|
||||
|
||||
开发计划：
|
||||
|
||||
| 阶段 | 交付内容 |
|
||||
| --- | --- |
|
||||
| 第 1 周 | 完成数据集 CRUD、上传、预览、版本接口和页面联调。 |
|
||||
| 第 2 周 | 完成数据处理任务、切片预览、质量评分和发布数据集。 |
|
||||
| 第 3 周 | 完成评测任务、评测维度、样本结果查询。 |
|
||||
| 第 4 周 | 完成推理会话、模型对比、数据转换、自定义工具基础能力。 |
|
||||
|
||||
验收标准：
|
||||
|
||||
- 数据文件必须登记存储对象、checksum、版本和项目归属。
|
||||
- 数据处理产物发布为数据集时保留来源链路。
|
||||
- 评测和推理必须记录使用的模型版本、数据集版本和参数快照。
|
||||
- 下载、删除、导出等动作必须接入审计和审批策略。
|
||||
|
||||
### D. 算力平台、部署与运维
|
||||
|
||||
负责人边界：
|
||||
|
||||
- 前端目录：`frontend/src/views/compute/`，协助 `system/HardwareView.vue`。
|
||||
- 前端 API：`frontend/src/api/modules/compute.ts`。
|
||||
- 后端模块：`compute_gateway`、`engine_registry`、`file_gateway`、`system` 中的资源采集部分。
|
||||
- 算力目录：`compute/api/`、`compute/agent/`、`compute/engines/llama_factory/`、`compute/file_gateway/`。
|
||||
- 部署目录：`docker/app/`、`docker/compute/`、`docker/README.md`、`docs/deployment-plan.md`。
|
||||
- 数据库表：`compute_nodes`、`gpu_devices`、`compute_node_engines`、`compute_jobs`、`gpu_allocations`、`resource_replicas`、`resource_sync_jobs`、`system_metric_snapshots`、`storage_objects`。
|
||||
|
||||
对应页面和功能：
|
||||
|
||||
| 页面/模块 | 路由 | 功能点 | 接口 |
|
||||
| --- | --- | --- | --- |
|
||||
| 算力节点 | `/compute`、`/compute?tab=nodes` | 节点地址、File Gateway 地址、权重、标签、启用状态、连接测试 | `/modelTF/compute/nodes/*` |
|
||||
| GPU 资源 | `/compute?tab=gpus`、`/hardware` | GPU 显存、利用率、温度、分配状态、节点归属 | `/modelTF/compute/gpus`、`/modelTF/system-info` |
|
||||
| 任务队列 | `/compute?tab=queue` | 队列、优先级、占用 GPU、任务状态 | `/modelTF/compute/queue` |
|
||||
| 资源副本 | `/compute` 节点详情 | 模型/数据集在算力节点上的同步状态 | `/modelTF/compute/nodes/{id}/replicas` |
|
||||
| 文件网关 | 无独立页面，供模型/数据/训练调用 | 上传、下载、离线导入、产物归档 | 应用侧 `/modelTF/*` 编排，算力侧内部 File Gateway API |
|
||||
| 部署运维 | 文档和 Compose | 应用/算力分离部署、端口、镜像、日志、健康检查 | Docker Compose、健康检查接口 |
|
||||
|
||||
开发计划：
|
||||
|
||||
| 阶段 | 交付内容 |
|
||||
| --- | --- |
|
||||
| 第 1 周 | 完成 Compute API 健康检查、GPU 发现、节点登记和连接测试。 |
|
||||
| 第 2 周 | 完成任务状态查询、应用侧轮询、资源副本状态同步。 |
|
||||
| 第 3 周 | 完成 LLaMA-Factory 容器/宿主机路径适配、日志采集、训练进程管理。 |
|
||||
| 第 4 周 | 完成应用/算力两套 Docker Compose、部署文档、故障排查脚本。 |
|
||||
|
||||
验收标准：
|
||||
|
||||
- 多算力节点阶段仍按“每台算力服务器 = 单机多 GPU 节点”设计。
|
||||
- 每台算力服务器都部署 Compute API、Agent、File Gateway 和 LLaMA-Factory。
|
||||
- 应用服务器只需主动访问所有算力节点，不要求算力节点反向访问应用服务器。
|
||||
- 节点地址、权重、标签、启用状态必须可动态维护。
|
||||
|
||||
## 3. 3 人开发合并方案
|
||||
|
||||
如果团队只有 3 人，建议合并为：
|
||||
|
||||
| 开发人员 | 合并内容 | 不建议合并的原因 |
|
||||
| --- | --- | --- |
|
||||
| A | 平台基础与企业治理 | 该部分是所有模块的权限和隔离底座，不宜再叠加训练或数据主链路。 |
|
||||
| B | 模型资产、训练与 LLaMA-Factory 任务 | 模型和训练强耦合，适合一人端到端打通。 |
|
||||
| C | 数据集、数据处理、评测、推理、算力部署协同 | 数据链路和评测推理使用相同数据/模型资产；算力底层可先由 C 搭骨架，后续扩人拆出 D。 |
|
||||
|
||||
若进入真实 GPU 联调阶段，必须优先把 D 独立出来，否则训练问题、部署问题和业务问题会混在一起，排障效率会明显下降。
|
||||
|
||||
## 4. 公共契约和协作节奏
|
||||
|
||||
公共契约负责人建议由 A 兼任，所有人遵守：
|
||||
|
||||
| 契约 | 文件 | 变更规则 |
|
||||
| --- | --- | --- |
|
||||
| 路由前缀 | `backend/app/core/config.py`、`backend/app/api/v1/router.py`、接口文档 | 统一 `/modelTF`，不得新增 `/api` 前缀 |
|
||||
| 响应结构 | `frontend/src/api/request.ts`、后端 schema | 统一 `{ code, message, data }` |
|
||||
| 权限码 | `frontend/src/types/index.ts`、`permissions` 表、接口文档 | 新菜单先登记权限码再开发 |
|
||||
| 项目隔离 | `project_id`、`tenant_id`、`resource_acl` | 所有模型、数据集、任务必须带项目归属 |
|
||||
| 审计动作 | `audit_logs`、后端审计中间件/服务 | 写操作默认审计 |
|
||||
| 异步状态 | 任务表、`compute_jobs` | 统一 `pending/running/completed/failed/stopped` |
|
||||
| 文件存储 | `storage_objects`、File Gateway | 不暴露宿主机绝对路径给前端 |
|
||||
|
||||
建议节奏：
|
||||
|
||||
- 每周一上午同步接口契约和数据库迁移计划。
|
||||
- 每天下午固定一次跨模块联调窗口，优先处理阻塞其他人的接口。
|
||||
- 每个模块 PR 必须包含页面入口、接口说明、SQL/迁移、最小验证步骤。
|
||||
- 公共文件如 `frontend/src/types/index.ts`、`backend/app/core/*`、`docs/backend-api-design.md` 由对应 owner 统一合并，其他人通过小 PR 提交变更。
|
||||
|
||||
## 5. 里程碑
|
||||
|
||||
| 里程碑 | 目标 | 必须完成 |
|
||||
| --- | --- | --- |
|
||||
| M1 基础可用 | 用户登录、模型/数据集/训练主链路可运行 | A 登录权限，B 模型训练，C 数据集，D 单节点 GPU 状态 |
|
||||
| M2 企业隔离 | 多租户、项目、资源 ACL 接入主链路 | 所有资源按租户/项目过滤，审计落库 |
|
||||
| M3 训练闭环 | LLaMA-Factory 真实训练、日志、checkpoint、产物登记 | 应用侧轮询 Compute API，训练产物可在模型管理查看 |
|
||||
| M4 治理闭环 | 审批、审计、留存、导出、部署文档完善 | 高风险动作审批，审计可检索，应用/算力分离部署可复现 |
|
||||
|
||||
121
docs/模型评测功能总结.md
121
docs/模型评测功能总结.md
@@ -1,121 +0,0 @@
|
||||
# 模型评测功能总结
|
||||
|
||||
本项目（基于 LLaMA-Factory 的微调训练平台）包含 **4 套相对独立** 的模型评测能力，分别面向不同的使用场景：
|
||||
|
||||
| 能力 | 入口/目录 | 评测类型 | 打分方式 |
|
||||
| --- | --- | --- | --- |
|
||||
| 1. 学术 Benchmark 评测 | `llamafactory/eval/` | 选择题式基准（类 MMLU/C-Eval） | 选项匹配 + few-shot |
|
||||
| 2. 评估工作台 | `backend/app/api/v1/eval/` | 生成式问答（指令跟随） | BLEU / ROUGE / ExactMatch + 可选 LLM 评审 |
|
||||
| 3. 平台评估系统 | `backend/app/api/v1/evaluation/` | 基于评估数据集的问答 | 判卷模型（judge model）打分（0–5 分） |
|
||||
| 4. 训练时验证评估 | `backend/app/services/task_runner.py` | 训练验证集 | loss 指标 |
|
||||
|
||||
下面分别说明。
|
||||
|
||||
---
|
||||
|
||||
## 1. 学术 Benchmark 评测（LLaMA-Factory 原生）
|
||||
|
||||
面向标准学术选择题基准（如 MMLU、C-Eval 等），复用 LLaMA-Factory 原生的评测框架。
|
||||
|
||||
**核心文件**
|
||||
- `llamafactory/eval/evaluator.py`：`Evaluator` 类 + `run_eval()` 入口
|
||||
- `llamafactory/eval/template.py`：评测 prompt 模板（中/英，含 few-shot 示例构建）
|
||||
- `llamafactory/hparams/evaluation_args.py`：`EvaluationArguments` 配置类
|
||||
|
||||
**工作流程**
|
||||
1. 按 `task`（benchmark 名称）加载数据集，按科目（subject）拆分。
|
||||
2. 每个样本构造 few-shot 提示词（`n_shot` 控制示例数，由 `lang` 决定中/英模板），将题干与候选选项拼入 prompt。
|
||||
3. 调用模型推理得到预测，与标准答案比对，统计每个科目及整体的 `accuracy`。
|
||||
4. 结果写入 `save_dir`，打印各科目与平均准确率。
|
||||
|
||||
**关键参数（`EvaluationArguments`）**
|
||||
- `task`：基准数据集名
|
||||
- `batch_size` / `n_shot` / `lang` / `save_dir` / `seed`
|
||||
- `model_name_or_path`、`template`、`trust_remote_code` 等模型相关参数
|
||||
|
||||
> 该能力属于框架底层，本平台前端未直接提供操作入口，主要通过配置文件/脚本调用。
|
||||
|
||||
---
|
||||
|
||||
## 2. 评估工作台（生成式评测 + 指标计算）
|
||||
|
||||
后端路由位于 `backend/app/api/v1/eval/__init__.py`，前端称为「评估工作台」。**适用于评测模型的指令跟随与生成质量**，并支持 LLM 作为裁判（LLM-as-a-Judge）。
|
||||
|
||||
**API 端点**
|
||||
- `GET /evaluation/tasks`：列出评测任务（`frontend/src/api/evaluation.ts:listTasks`）
|
||||
- `POST /evaluation/run`：提交一次评测（`runEval`）
|
||||
- `GET /evaluation/report/{task_id}`：拉取评测报告（`getReport`）
|
||||
- `DELETE /evaluation/tasks/{task_id}`：删除任务（`deleteTask`）
|
||||
|
||||
**评测流程（`run_eval`）**
|
||||
1. 通过 **LLaMA-Factory 数据管道**（`get_dataset`) 加载数据集，支持 `subset` 与抽样（`eval_sample`）。
|
||||
2. 用 **原生 transformers** 加载模型在本地做生成推理（单进程顺序生成，便于展示样本）。
|
||||
3. 计算客观指标（`compute_score`）：
|
||||
- `BLEU`（sacrebleu）
|
||||
- `ROUGE-1 / ROUGE-2 / ROUGE-L`（rouge-score）
|
||||
- `Exact Match`
|
||||
4. **可选 LLM 评审**（judge）：当配置了 `judge_model` / `judge_api_base` / `judge_api_key` 时，调用 OpenAI 兼容接口对每条样本打分（10 分制），并输出 4 个维度与理由：
|
||||
- 核心事实正确性 `factual`
|
||||
- 信息完整性 `completeness`
|
||||
- 无幻觉 `no_hallucination`
|
||||
- 格式合规性 `format`
|
||||
- 综合分 `score` + `reason`
|
||||
5. 任务状态持久化在后端 `eval_tasks.json`（支持 running/completed/failed/stopped），前端轮询进度。
|
||||
|
||||
**前端页面**
|
||||
- `frontend/src/views/evaluation/EvaluateTask.vue`：任务列表、创建评测对话框（选模型、数据集、指标、可选 judge 配置）
|
||||
- `frontend/src/views/evaluation/EvaluateReport.vue`：报告页，展示综合得分、BLEU、ROUGE-L、各维度指标及「参考答案 vs 模型预测 vs LLM 评审」对比样例
|
||||
|
||||
---
|
||||
|
||||
## 3. 平台评估系统（基于评估数据集 + 判卷模型）
|
||||
|
||||
后端路由位于 `backend/app/api/v1/evaluation/__init__.py`，是平台业务层自研的评测体系。通过「评估数据集」组织题目，可一次性对 **多个被测模型 + 指定判卷模型** 进行批量评分。
|
||||
|
||||
**核心概念（数据模型 `backend/app/models/models.py`）**
|
||||
- `EvalDataset`（`models.py:131`）：评估数据集，从项目问答对（`Question`/`Chunk`）中按 `question_type`（mixed/fact/reasoning）选题构建，状态 `pending/running/completed/failed`。
|
||||
- `EvalResult`（`models.py:147`）：单条评测结果，含 `judge_score`（0–5 分）、`is_correct`（true/false/partial）、`feedback`、`expected_answer` 等。
|
||||
- `Task`（`models.py:184`）：后台任务，`task_type="model-evaluation"`，记录进度与 `model_info`（存放平均分等汇总）。
|
||||
|
||||
**评测流程（`process_evaluation_task`，`backend/app/services/task_processor.py:336` 起）**
|
||||
1. 加载评估数据集关联的题目，可选带入 `chunk` 上下文（RAG 场景）。
|
||||
2. 对每道题，先用 `build_eval_prompt` 组合「上下文 + 题目 + 参考答案」，调用 **判卷模型**（`call_model`，temperature=0.3）生成评分。
|
||||
3. `parse_eval_result` 解析出 `score`(0–5)、`is_correct`、`feedback`，写入 `EvalResult`。
|
||||
4. 逐题提交进度（`completed_count` / `progress`），支持中途 `stopped`。
|
||||
5. 汇总：`avg_score = 总分/有效数 × 20`（换算百分制），`avg_score_5 = 总分/有效数`（5 分制），存入 `task.model_info`。判定规则：得分 **≥3 视为正确**。
|
||||
|
||||
**特点**
|
||||
- 判卷与被测模型解耦：被测模型给出答案，判卷模型（judge）独立评分，降低自评偏差。
|
||||
- 支持失败隔离：单题异常写入 `evaluation_status: failed` 记录而不中断整体任务。
|
||||
|
||||
---
|
||||
|
||||
## 4. 训练时验证评估
|
||||
|
||||
在微调训练任务执行期间，由 `backend/app/services/task_runner.py` 的 `do_eval` 触发：
|
||||
|
||||
- 在训练过程中对验证集（validation set）计算 `eval_loss`，用于监控过拟合。
|
||||
- 结果回填到 `Task` 的 `loss_info` / `detail`，前端绘制 loss 曲线。
|
||||
- 属于训练配套的轻量评估，不参与上述 1–3 的业务评测。
|
||||
|
||||
---
|
||||
|
||||
## 附属：前端评测相关页面
|
||||
|
||||
| 文件 | 作用 |
|
||||
| --- | --- |
|
||||
| `frontend/src/views/evaluation/EvaluateTask.vue` | 评估工作台：任务列表 + 创建评测 |
|
||||
| `frontend/src/views/evaluation/EvaluateReport.vue` | 评估报告：指标卡 + 维度标签 + 对比样例 |
|
||||
| `frontend/src/api/evaluation.ts` | 评估工作台接口封装 |
|
||||
| 平台评估系统入口 | 评估数据集管理 + 评估任务（model-evaluation）创建与结果查看 |
|
||||
|
||||
---
|
||||
|
||||
## 小结
|
||||
|
||||
- **想要学术榜单式准确率** → 用能力 1（LLaMA-Factory `eval/`）。
|
||||
- **想要开放式生成质量（BLEU/ROUGE + LLM 评审）** → 用能力 2（评估工作台 `/evaluation/run`）。
|
||||
- **想要基于自有问答数据、用判卷模型批量打分** → 用能力 3（平台评估系统 `model-evaluation` 任务）。
|
||||
- **训练过程监控** → 能力 4（`do_eval` 验证集 loss）。
|
||||
|
||||
三种业务评测（1/2/3）相互独立，可并存于同一平台；数据模型（`EvalDataset`/`EvalResult`/`Task`）主要服务于能力 3，而能力 2 使用独立的 `eval_tasks.json` 文件持久化。
|
||||
@@ -1,22 +0,0 @@
|
||||
4.2.3 评估工作台
|
||||
负责训练后模型的质量评测和问题诊断。这是平台闭环的核心环节。
|
||||
|
||||
|
||||
评估流程图
|
||||
1.评估方式
|
||||
评估默认使用本地部署的 LLM 作为评审模型，不依赖外部 API。评审模型对每条测试数据从四个子维度打分：核心事实正确性、信息完整性、无幻觉、格式合规性（可选）。汇总为三档判定：正确、部分正确、错误。
|
||||
用户可以选择额外启动人工复核——平台按错误类型分层抽样建议五十到两百条题目，用户在界面上逐条确认或修改 LLM 评审的判定。人工复核的结果用于校准 LLM 评审——一致率超过百分之八十时 LLM 评审结果标记为"可信"，低于百分之六十时标记为"以人工为准"。
|
||||
平台同时提供自动指标作为参考——如关键字段匹配率。这些指标不单独作为判定依据，仅作为快速参考。
|
||||
|
||||
2.错误诊断
|
||||
评审模型在完成评分后，额外输出一个错误分类标签。标签从五种固定类型中选择：混淆（模型回答的值像是另一个实体的属性值）、不完整（事实正确但缺少部分信息）、格式偏差（语义正确但措辞与预期不符）、幻觉（回答中存在标准答案没有的内容）、其他（不属于以上任何类型）。
|
||||
这种分类方式简单可落地——它是一个固定枚举的分类任务，评审模型的prompt 中已包含每种类型的定义和判别示例，不需要额外的自然语言聚类或机器学习算法。
|
||||
|
||||
3.评估报告
|
||||
评估完成后自动生成报告，分为四个部分：总览面板：整体得分和各维度通过率。如果做了人工复核，展示 LLM 评审和人工判定的一致率及可信度标记。
|
||||
错误分类面板：按五种分类标签分组的错误列表，每组展示数量和占比。点击展开可查看具体错误样例（问题、标准答案、模型回答、评审模型的原因描述）。
|
||||
修复建议面板：根据错误分类的统计分布，自动生成方向性建议。如"混淆"类错误占比最高时，建议检查训练数据中指令相似但答案不同的样本对；"不完整"类占比最高时，建议统一同类问题的答案详略标准。某类错误的绝对数量不足百分之三时不单独给建议，"其他"类占比最高时提示用户人工分析错误样例。
|
||||
迭代对比面板：如果存在上轮评估记录，展示两轮各分类标签的数量变化。标注"混淆类错误从35条降到14条，下降60%，修复可能生效"。
|
||||
|
||||
4.建议有效性的验证
|
||||
平台在每次评估完成后将分类标签的统计数据（每种标签的数量和占比）存入评估记录。下一轮评估的对应数据与之对比，计算差值和变化百分比。某类错误数量下降超过百分之二十，标注"调整可能生效"；变化不足百分之十，标注"调整可能未生效或生效不显著"；反向上升则标注"建议检查本次调整方向是否正确"。
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user