10 Commits

Author SHA1 Message Date
wangjiming
a12f80492d Merge branch 'ft_wyt' of http://www.caoxiaozhu.com:13001/YG-Soft/YG_FT into ft_wyt 2026-08-11 15:45:23 +08:00
wangjiming
71405def14 新增配置模型测试按钮 2026-08-11 15:41:51 +08:00
caoxiaozhu
5f6e7523cf feat: 新增外部数据源拉取与 DPO 输出格式支持
- 支持从 PostgreSQL 数据库拉取结构化数据作为训练来源
- 新增 DPO (Direct Preference Optimization) 输出类型
- 支持 chosen/rejected 字段的编辑、校验和发布
- 完善数据预处理切分逻辑和元数据管理
- 移除 OCR 扫描 PDF 功能,保持基础文本解析能力

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-11 14:17:45 +08:00
wangjiming
f809825a7d 完善部分平台治理功能,及修改看板缺陷 2026-08-10 11:41:16 +08:00
wuyongtao
75cc105ebc chore: 忽略离线部署包,提交安全加固、数据库初始化与文档
- .gitignore: 忽略 docker/offline 离线部署包(镜像/运行时等大文件)
- 安全加固: 新增 compute/api/security.py 及各端安全测试,补充 docs/security-hardening.md
- 数据库: 新增完整初始化 SQL 与 docs/database-config.md
- 数据转换与评测: 修复类型检查、增强校验并补充测试
- Docker 配置与环境变量更新

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-07 09:24:35 +08:00
wuyongtao
e397bcc2ca fix: 修复数据转换页面类型检查错误
- statusTag 返回类型改用 TagProps['type'],修复 el-tag type 属性 TS 报错
- handleDelete 参数加 DataConvertTask 类型断言,修复 DefaultRow 类型不匹配

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-05 16:47:53 +08:00
wangjiming
c64fa1cd61 新增数据类型转换 2026-08-05 16:23:00 +08:00
wangjiming
b5c6557341 Merge branch 'ft_wyt' of http://www.caoxiaozhu.com:13001/YG-Soft/YG_FT into ft_wyt 2026-08-05 08:34:55 +08:00
wangjiming
7f93ed6d09 feat: 更新 request.ts 2026-08-04 17:18:50 +08:00
wangjiming
5ec950cc9f feat: 更新 request.ts 2026-08-04 17:02:57 +08:00
76 changed files with 5825 additions and 856 deletions

3
.gitignore vendored
View File

@@ -208,3 +208,6 @@ docker/compute/data/yg-ft/logs/**
!docker/compute/data/yg-ft/logs/training/ !docker/compute/data/yg-ft/logs/training/
!docker/compute/data/yg-ft/**/.gitkeep !docker/compute/data/yg-ft/**/.gitkeep
!docker/compute/data/yg-ft/**/README.md !docker/compute/data/yg-ft/**/README.md
# Offline deployment bundle - 离线部署包(镜像、运行时等大文件,不提交)
docker/offline/

View File

@@ -50,16 +50,15 @@ YG_FT/
## 前后端一键启动 ## 前后端一键启动
首次使用前,请先按下方“后端启动”和“前端启动”说明安装依赖,并确保 首次使用前请确保前端依赖已安装、PostgreSQL 已可用。之后在项目根目录执行:
PostgreSQL 已可用。之后在项目根目录执行:
```bash ```bash
bash ./start.sh bash ./start.sh
``` ```
脚本会同时启动前端 `http://localhost:16801` 和后端 脚本会自动补装后端 `requirements.txt`,然后同时启动前端 `http://localhost:16801` 和后端
`http://127.0.0.1:17861`,按 `Ctrl+C` 会同时停止两个服务。脚本只负责 `http://127.0.0.1:17861`,按 `Ctrl+C` 会同时停止两个服务。脚本不会自动安装
启动前后端,不会自动安装依赖,也不会启动 PostgreSQL、Redis 或算力服务。 前端依赖,也不会启动 PostgreSQL、Redis 或算力服务。
仅检查依赖和端口而不启动服务: 仅检查依赖和端口而不启动服务:
@@ -103,15 +102,37 @@ GET /modelTF/model-manage
GET /modelTF/dataset-manage GET /modelTF/dataset-manage
GET /modelTF/fine-tune GET /modelTF/fine-tune
GET /modelTF/compute/nodes GET /modelTF/compute/nodes
GET /modelTF/data-convert
``` ```
本地运行时默认 PostgreSQL 连接: ### 数据库配置
```text 后端通过 `backend/.env` 文件配置数据库连接(自动加载,`override=True`
DATABASE_URL=postgresql+psycopg://yg_ft:change_me@localhost:15432/yg_ft
```env
DATABASE_URL=postgresql+psycopg://用户:密码@数据库地址:端口/库名
COMPUTE_SERVICE_TOKEN=change_me
``` ```
本地启动前需要确保 PostgreSQL 已监听 `localhost:15432`,并已创建 `yg_ft` 数据库和 `yg_ft` 用户。后端启动后会自动创建当前运行表并写入内置管理员账号,运行数据统一写入 PostgreSQL 支持远程数据库。连接池参数已针对远程库优化(`connect_timeout=30``max_size=20``max_waiting=50`
### 环境变量
| 变量 | 默认值 | 说明 |
|---|---|---|
| `DATABASE_URL` | `postgresql+psycopg://yg_ft:change_me@localhost:15432/yg_ft` | 数据库连接串 |
| `COMPUTE_SERVICE_TOKEN` | `""` | 算力服务认证 token需与 compute 一致 |
| `COMPUTE_STATUS_SYNC_MODE` | `polling` | `off` 禁用轮询(远程库慢时推荐) |
| `COMPUTE_POLL_INTERVAL_SECONDS` | `3` | 轮询间隔秒数 |
| `COMPUTE_REQUEST_TIMEOUT_SECONDS` | `5` | 调 compute 的超时秒数 |
### 推荐启动命令(远程数据库)
```cmd
cd /d E:\yg_ft\backend
set COMPUTE_STATUS_SYNC_MODE=off
.\.venv\Scripts\python.exe -m uvicorn app.main:app --reload --port 17861
```
开发阶段内置登录账号: 开发阶段内置登录账号:
@@ -138,12 +159,29 @@ npm run dev
### 方式一Docker 启动(推荐) ### 方式一Docker 启动(推荐)
```bash **1. 构建镜像**(首次或依赖变更后):
```cmd
cd /d E:\yg_ft
docker build -f docker/compute/Dockerfile.compute -t yg-ft-compute-api:latest .
```
**2. 启动容器**
```cmd
cd docker/compute cd docker/compute
cp .env.example .env cp .env.example .env
docker compose up -d docker compose up -d
``` ```
**3. 验证**
```cmd
curl http://localhost:19100/health
```
> 注意:构建上下文必须是项目根目录 `E:\yg_ft``docker build` 最后的 `.`),因为 Dockerfile 需要 `COPY compute/requirements.txt`。
### 方式二:本地开发启动 ### 方式二:本地开发启动
**Windows (cmd)** **Windows (cmd)**
@@ -151,25 +189,35 @@ docker compose up -d
```cmd ```cmd
cd /d E:\yg_ft\compute cd /d E:\yg_ft\compute
set PYTHONPATH=E:\yg_ft set PYTHONPATH=E:\yg_ft
.\.venv\Scripts\python.exe -m uvicorn api.main:app --reload --port 19100 .\.venv\Scripts\python.exe -m uvicorn api.main:app --reload --host 0.0.0.0 --port 19100
``` ```
> `PYTHONPATH=E:\yg_ft` 是必需的,因为代码使用 `from compute.agent...` 绝对导入。 > `PYTHONPATH=E:\yg_ft` 是必需的,因为代码使用 `from compute.agent...` 绝对导入。
> `--host 0.0.0.0` 让其他机器可以通过 IP 访问(算力节点测试需要)。
**Linux / macOS** **Linux / macOS**
```bash ```bash
cd compute cd compute
PYTHONPATH=.. uvicorn api.main:app --reload --port 19100 PYTHONPATH=.. uvicorn api.main:app --reload --host 0.0.0.0 --port 19100
``` ```
### 算力节点配置
在平台的「算力节点」页面新增节点,填入:
- **Compute API**`http://你的IP:19100`
- **File Gateway**`http://你的IP:19101`
本机测试用 `http://localhost:19100`
### 环境变量说明 ### 环境变量说明
| 变量 | 默认值 | 说明 | | 变量 | 默认值 | 说明 |
|---|---|---| |---|---|---|
| `COMPUTE_MODE` | `real` | `real` / `simulator`,仅隔离联调用 simulator | | `COMPUTE_MODE` | `real` | `real` / `simulator`,仅隔离联调用 simulator |
| `COMPUTE_EXECUTION_MODE` | `real` | 训练执行模式 | | `COMPUTE_EXECUTION_MODE` | `real` | 训练执行模式 |
| `COMPUTE_SERVICE_TOKEN` | `change_me` | 服务间认证 token | | `COMPUTE_SERVICE_TOKEN` | `change_me` | 服务间认证 token,需与 backend 一致 |
| `COMPUTE_AUTH_ENABLED` | `true` | 是否开启 token 认证 |
| `MODELTF_ROUTE_PREFIX` | `/modelTF` | API 路由前缀 | | `MODELTF_ROUTE_PREFIX` | `/modelTF` | API 路由前缀 |
应用平台通过数据库 `compute_nodes` 表中的 `api_base_url``file_gateway_url` 主动轮询算力节点状态。 应用平台通过数据库 `compute_nodes` 表中的 `api_base_url``file_gateway_url` 主动轮询算力节点状态。

View File

@@ -16,7 +16,7 @@ from dataclasses import asdict
from pathlib import Path from pathlib import Path
from threading import BoundedSemaphore, Lock from threading import BoundedSemaphore, Lock
from typing import Any, Literal from typing import Any, Literal
from urllib.parse import quote, urlsplit from urllib.parse import parse_qs, quote, urlsplit
import httpx import httpx
import psycopg import psycopg
@@ -676,8 +676,10 @@ def _run_generation(
qa_pairs_per_item=int(pairs or 1), qa_pairs_per_item=int(pairs or 1),
on_progress=report_progress, on_progress=report_progress,
) )
elif output_type == "reasoning": elif output_type in {"reasoning", "dpo"}:
if output_type == "reasoning":
raise InvalidStateError("思维链输出必须配置可用的数据生成模型") raise InvalidStateError("思维链输出必须配置可用的数据生成模型")
raise InvalidStateError("DPO 输出必须配置可用的数据生成模型")
else: else:
generated = generate_standard_records( generated = generate_standard_records(
preview_items, preview_items,
@@ -1077,7 +1079,10 @@ async def upload_source_files(
with api_errors(): with api_errors():
task = store.get_task(task_id) task = store.get_task(task_id)
process_type = str(task["process_type"]) process_type = str(task["process_type"])
if process_type == "external": source_mode = str(
_value(task.get("config") or {}, "source_mode", "sourceMode", "local")
)
if process_type == "external" or source_mode == "external":
raise InvalidStateError( raise InvalidStateError(
"external tasks must import data through the external source endpoint" "external tasks must import data through the external source endpoint"
) )
@@ -1375,6 +1380,10 @@ def _external_postgres_connection(payload: ExternalSourceRequest) -> psycopg.Con
raise fail(501, f"external data source type is not supported: {payload.type}") raise fail(501, f"external data source type is not supported: {payload.type}")
if parsed_url.username or parsed_url.password: if parsed_url.username or parsed_url.password:
raise fail(400, "database credentials must use the account and password fields") raise fail(400, "database credentials must use the account and password fields")
if set(parse_qs(parsed_url.query)) & {
"password", "secret", "token", "api_key", "user", "username"
}:
raise fail(400, "database URL query must not contain credentials")
if payload.auth_mode not in {"none", "basic"}: if payload.auth_mode not in {"none", "basic"}:
raise fail(400, "PostgreSQL supports only none or basic authentication") raise fail(400, "PostgreSQL supports only none or basic authentication")
if payload.auth_mode == "basic" and not payload.username: if payload.auth_mode == "basic" and not payload.username:
@@ -1413,10 +1422,14 @@ def _external_postgres_connection(payload: ExternalSourceRequest) -> psycopg.Con
"set DATA_PROCESS_ALLOW_PRIVATE_EXTERNAL_DB=true only in a trusted deployment", "set DATA_PROCESS_ALLOW_PRIVATE_EXTERNAL_DB=true only in a trusted deployment",
) )
kwargs: dict[str, Any] = { kwargs: dict[str, Any] = {
"connect_timeout": 5, "connect_timeout": payload.connect_timeout_seconds,
"row_factory": dict_row, "row_factory": dict_row,
"application_name": "yg-ft-data-process-readonly", "application_name": "yg-ft-data-process-readonly",
"options": "-c default_transaction_read_only=on -c statement_timeout=30000", "options": (
"-c default_transaction_read_only=on "
f"-c statement_timeout={payload.statement_timeout_seconds * 1000}"
),
"sslmode": payload.ssl_mode,
} }
if payload.auth_mode == "basic" and payload.username: if payload.auth_mode == "basic" and payload.username:
kwargs["user"] = payload.username kwargs["user"] = payload.username
@@ -1425,6 +1438,17 @@ def _external_postgres_connection(payload: ExternalSourceRequest) -> psycopg.Con
return psycopg.connect(payload.url, **kwargs) return psycopg.connect(payload.url, **kwargs)
def _assert_external_source_task(task: dict[str, Any]) -> None:
if str(task.get("process_type")) == "external":
return
config = task.get("config") or {}
source_mode = str(_value(config, "source_mode", "sourceMode", "local"))
if str(task.get("process_type")) != "structured" or source_mode != "external":
raise InvalidStateError(
"external source access requires a structured task with source_mode=external"
)
@router.post("/{task_id}/external/test") @router.post("/{task_id}/external/test")
def test_external_source( def test_external_source(
task_id: str, task_id: str,
@@ -1433,10 +1457,7 @@ def test_external_source(
) -> dict[str, Any]: ) -> dict[str, Any]:
with api_errors(): with api_errors():
task = store.get_task(task_id) task = store.get_task(task_id)
if str(task.get("process_type")) != "external": _assert_external_source_task(task)
raise InvalidStateError(
"external source access requires an external data processing task"
)
try: try:
with _external_postgres_connection(payload) as conn: with _external_postgres_connection(payload) as conn:
conn.execute("SELECT 1 AS ok").fetchone() conn.execute("SELECT 1 AS ok").fetchone()
@@ -1462,12 +1483,14 @@ def pull_external_source(
raise fail(400, "a read-only SELECT or WITH query is required for external pull") raise fail(400, "a read-only SELECT or WITH query is required for external pull")
with api_errors(): with api_errors():
task = store.get_task(task_id) task = store.get_task(task_id)
if str(task.get("process_type")) != "external": _assert_external_source_task(task)
raise InvalidStateError("external pull requires an external data processing task")
try: try:
with _external_postgres_connection(payload) as conn: with _external_postgres_connection(payload) as conn:
conn.execute("SET TRANSACTION READ ONLY") conn.execute("SET TRANSACTION READ ONLY")
conn.execute("SET LOCAL statement_timeout = '30s'") conn.execute(
"SELECT set_config('statement_timeout', %s, true)",
(f"{payload.statement_timeout_seconds}s",),
)
cursor = conn.execute(query) cursor = conn.execute(query)
rows: list[dict[str, Any]] = [] rows: list[dict[str, Any]] = []
content_parts: list[str] = [] content_parts: list[str] = []
@@ -1518,6 +1541,9 @@ def pull_external_source(
"external_type": payload.type, "external_type": payload.type,
"external_host": urlsplit(payload.url).hostname, "external_host": urlsplit(payload.url).hostname,
"external_limit": payload.limit, "external_limit": payload.limit,
"external_ssl_mode": payload.ssl_mode,
"external_connect_timeout_seconds": payload.connect_timeout_seconds,
"external_statement_timeout_seconds": payload.statement_timeout_seconds,
}, },
} }
], ],
@@ -1566,7 +1592,10 @@ def _prepare_preview_items(
for index, source in enumerate(sources): for index, source in enumerate(sources):
source_format = str(source.get("file_format") or "").lower() source_format = str(source.get("file_format") or "").lower()
needs_structured_xlsx = not is_unstructured and source_format == "xlsx" needs_structured_xlsx = not is_unstructured and source_format == "xlsx"
needs_layout_raw = is_unstructured and chunk_method == "layout_hybrid" needs_layout_raw = (
is_unstructured
and chunk_method == "layout_hybrid"
)
needs_pdf_noise = ( needs_pdf_noise = (
is_unstructured is_unstructured
and not needs_layout_raw and not needs_layout_raw
@@ -2038,6 +2067,8 @@ def restore_result(
"instruction": current.get("original_instruction") or current.get("instruction") or "", "instruction": current.get("original_instruction") or current.get("instruction") or "",
"input": current.get("original_input") or current.get("input") or "", "input": current.get("original_input") or current.get("input") or "",
"output": current.get("original_output") or current.get("output") or "", "output": current.get("original_output") or current.get("output") or "",
"chosen": current.get("original_chosen") or current.get("chosen") or "",
"rejected": current.get("original_rejected") or current.get("rejected") or "",
} }
preview_id = current.get("preview_item_id") preview_id = current.get("preview_item_id")
source_content = "" source_content = ""
@@ -2070,6 +2101,8 @@ def restore_result(
"instruction": restored["instruction"], "instruction": restored["instruction"],
"input": restored["input"], "input": restored["input"],
"output": restored["output"], "output": restored["output"],
"chosen": restored["chosen"],
"rejected": restored["rejected"],
"quality_score": asdict(quality), "quality_score": asdict(quality),
"expected_updated_at": current.get("updated_at"), "expected_updated_at": current.get("updated_at"),
}, },
@@ -2141,13 +2174,15 @@ def _generate_result_replacement(
).strip().lower() ).strip().lower()
previous_instruction = str(current.get("instruction") or "")[:1000] previous_instruction = str(current.get("instruction") or "")[:1000]
previous_output = str(current.get("output") or "")[:1000] previous_output = str(current.get("output") or "")[:1000]
previous_rejected = str(current.get("rejected") or "")[:1000]
base_prompt = str( base_prompt = str(
_value(config, "generation_prompt", "generationPrompt", "") or "" _value(config, "generation_prompt", "generationPrompt", "") or ""
) )
regeneration_instruction = ( regeneration_instruction = (
"这是一次失败结果的重新生成。请使用新的提问角度和表达," "这是一次失败结果的重新生成。请使用新的提问角度和表达,"
"不要复述旧结果。旧问题:" "不要复述旧结果。旧问题:"
f"{previous_instruction or ''};旧答案:{previous_output or ''}" f"{previous_instruction or ''};旧优选答案:{previous_output or ''}"
f"旧拒选答案:{previous_rejected or ''}"
) )
runtime_config = { runtime_config = {
**config, **config,

View File

@@ -1,4 +1,4 @@
from __future__ import annotations from __future__ import annotations
import json import json
import uuid import uuid
@@ -442,16 +442,14 @@ async def dashboard_stats() -> dict[str, Any]:
running_statuses = {"syncing", "queued", "running"} running_statuses = {"syncing", "queued", "running"}
running_ft = [t for t in tasks if t.get("status") in running_statuses] running_ft = [t for t in tasks if t.get("status") in running_statuses]
failed_ft = [t for t in tasks if t.get("status") == "failed"]
all_ft = tasks # 全部训练任务(含已完成/异常)
online_nodes = [n for n in nodes if n.get("scheduler_status") == "online"] online_nodes = [n for n in nodes if n.get("scheduler_status") == "online"]
# 评测中运行的任务 # 评测中运行的任务
running_eval = [e for e in eval_tasks if e.get("status") in running_statuses] eval_running = 0
# 数据处理中运行的任务
try: try:
dp_running = int(dp_store.list_tasks(page=1, page_size=1, status="running").get("total", 0)) eval_tasks = store.eval_tasks()
eval_running = len([e for e in eval_tasks if e.get("status") in running_statuses])
except Exception: except Exception:
dp_running = 0 eval_running = 0
# 近 7 天训练统计(按创建日期分桶) # 近 7 天训练统计(按创建日期分桶)
now = datetime.now(timezone.utc) now = datetime.now(timezone.utc)
@@ -472,45 +470,30 @@ async def dashboard_stats() -> dict[str, Any]:
} }
) )
# 服务状态 —— 通过对应接口连通性判断是否正常 # 服务状态 —— 每个服务的"实例数"含义:
# 模型训练 → 训练任务总数
# 模型评测 → 评测任务总数
# 模型推理 → 推理/对比任务实例数
# 模型管理 → 基座模型注册总数
# 数据集管理 → 数据集总数
# 数据处理 → 数据处理任务总数
# 数据类型转换 → 数据转换任务总数
service_checks = [ service_checks = [
("模型训练", "/fine-tune", "模型训练"), ("模型训练", "fine-tune", len(tasks)),
("模型评测", "/model-eval", "模型评测"), ("模型评测", "model-eval", len(eval_tasks)),
("模型推理", "/model-inference", "模型推理"), ("模型推理", "model-inference", len(store.compare_tasks())),
("模型管理", "/model-manage", "模型管理"), ("模型管理", "model-manage", len(store.models())),
("数据集管理", "/dataset-manage", "数据集管理"), ("数据集管理", "dataset-manage", len(datasets)),
("数据处理", "/data-process", "数据处理"), ("数据处理", "data-process", dp_count),
("数据类型转换", "/data-convert", "数据类型转换"), ("数据类型转换", "data-convert", dp_count),
] ]
service_status = [] service_status = []
for svc_type, _path, _label in service_checks: for svc_type, _path, svc_count in service_checks:
try:
svc_count = 0
if svc_type == "模型训练":
svc_count = len(tasks)
elif svc_type == "模型评测":
svc_count = len(eval_tasks)
elif svc_type == "模型推理":
svc_count = len(online_nodes)
elif svc_type == "模型管理":
svc_count = len(store.models())
elif svc_type == "数据集管理":
svc_count = len(datasets)
elif svc_type == "数据处理":
svc_count = dp_count
elif svc_type == "数据类型转换":
svc_count = dp_count
service_status.append({ service_status.append({
"type": svc_type, "type": svc_type,
"status": "normal", "status": "normal",
"count": svc_count, "count": svc_count,
}) })
except Exception:
service_status.append({
"type": svc_type,
"status": "error",
"count": 0,
})
# 训练任务状态归一化 # 训练任务状态归一化
status_map = { status_map = {
@@ -590,12 +573,16 @@ async def dashboard_stats() -> dict[str, Any]:
] ]
# 登录时长排行(本月),只取 top 5 # 登录时长排行(本月),只取 top 5
login_duration_rank = []
try:
login_duration_rank = store.login_duration_rank(limit=5) login_duration_rank = store.login_duration_rank(limit=5)
except Exception:
pass
return ok( return ok(
{ {
"online_services": sum(s["count"] for s in service_status), "online_services": sum(s["count"] for s in service_status),
"running_tasks": len(running_ft) + len(running_eval) + dp_running, "running_tasks": len(running_ft) + eval_running,
"pending_alerts": 0, "pending_alerts": 0,
"training_7d": training_7d, "training_7d": training_7d,
"service_status": service_status, "service_status": service_status,
@@ -656,6 +643,29 @@ async def reset_user_password(
raise fail(400, str(exc)) raise fail(400, str(exc))
@router.post("/users/me/password")
async def change_my_password(
payload: dict[str, Any] = Body(...),
current_user: dict = Depends(get_current_user),
) -> dict[str, Any]:
"""用户自行修改密码:验证旧密码后设置新密码。"""
old_password = payload.get("old_password") or ""
new_password = payload.get("new_password") or ""
if not old_password or not new_password:
raise fail(400, "old_password and new_password are required")
if len(new_password) < 6:
raise fail(400, "new password must be at least 6 characters")
try:
success = get_platform_store().change_password(
current_user["id"], old_password, new_password
)
except KeyError:
raise fail(404, "user not found")
if not success:
raise fail(400, "old password is incorrect")
return ok({"changed": True})
@router.get("/model-manage/local-models") @router.get("/model-manage/local-models")
async def local_models() -> dict[str, Any]: async def local_models() -> dict[str, Any]:
store = get_platform_store() store = get_platform_store()
@@ -685,8 +695,15 @@ async def local_models() -> dict[str, Any]:
@router.get("/model-manage/trained-models") @router.get("/model-manage/trained-models")
async def trained_models() -> dict[str, Any]: async def trained_models(current_user: dict = Depends(get_current_user)) -> dict[str, Any]:
return ok({"models": get_platform_store().trained_models()}) all_models = get_platform_store().trained_models()
if is_admin(current_user):
return ok({"models": all_models})
# 普通用户只能看到自己创建的 + ACL 授权的
user_id = current_user.get("id")
accessible = set(filter_accessible_resource_ids("trained_model", [m["id"] for m in all_models], current_user))
result = [m for m in all_models if m.get("created_by") == user_id or m["id"] in accessible]
return ok({"models": result})
@router.delete("/model-manage/trained-models/{model_id}") @router.delete("/model-manage/trained-models/{model_id}")
@@ -720,16 +737,74 @@ async def model_by_name(name: str) -> dict[str, Any]:
@router.get("/model-manage") @router.get("/model-manage")
async def model_list(current_user: dict = Depends(get_current_user)) -> dict[str, Any]: async def model_list(current_user: dict = Depends(get_current_user)) -> dict[str, Any]:
models = get_platform_store().models() # 基座模型是平台共享资源,所有登录用户均可查看
if current_user.get("role") == "admin" or current_user.get("protected"): return ok(get_platform_store().models())
return ok(models)
# 普通用户只返回有 ACL 授权的模型
accessible = set(filter_accessible_resource_ids("model", [m["id"] for m in models], current_user)) @router.post("/model-manage/test-online")
return ok([m for m in models if m["id"] in accessible]) async def test_online_model(payload: dict[str, Any] = Body(...)) -> dict[str, Any]:
"""测试在线模型 API 是否可用:发送一个简单的 chat/completions 请求验证连通性。"""
api_url = (payload.get("api_url") or "").rstrip("/")
api_key = payload.get("api_key") or ""
model_name = payload.get("online_model_name") or ""
if not api_url:
raise fail(400, "api_url is required")
if not model_name:
raise fail(400, "online_model_name is required")
import httpx
try:
async with httpx.AsyncClient(timeout=15) as client:
headers = {"Content-Type": "application/json"}
if api_key:
headers["Authorization"] = f"Bearer {api_key}"
# 尝试多种 OpenAI 兼容路径
chat_paths = [
f"{api_url}/chat/completions",
f"{api_url}/v1/chat/completions",
f"{api_url}/modelTF/v1/chat/completions",
]
resp = None
for path in chat_paths:
try:
r = await client.post(
path,
json={
"model": model_name,
"messages": [{"role": "user", "content": "Hi"}],
"max_tokens": 5,
"temperature": 0,
},
headers=headers,
)
if r.status_code in (200, 201):
resp = r
break
except Exception:
continue
if resp is None:
return ok({"success": False, "error": f"无法连接到 {api_url},请检查地址和端口"})
body = resp.json()
usage = body.get("usage", {})
return ok({
"success": True,
"model": body.get("model", model_name),
"provider": body.get("object", ""),
"usage": {
"prompt_tokens": usage.get("prompt_tokens", 0),
"completion_tokens": usage.get("completion_tokens", 0),
"total_tokens": usage.get("total_tokens", 0),
},
"latency_ms": None, # 由前端计算
})
except httpx.TimeoutException:
return ok({"success": False, "error": "连接超时15s请检查网络或 API 地址是否正确"})
except Exception as exc:
return ok({"success": False, "error": str(exc)})
@router.post("/model-manage") @router.post("/model-manage")
async def create_model(payload: dict[str, Any] = Body(...)) -> dict[str, Any]: async def create_model(payload: dict[str, Any] = Body(...), current_user: dict = Depends(get_current_user)) -> dict[str, Any]:
payload.setdefault("created_by", current_user.get("id"))
try: try:
return ok(get_platform_store().create_model(payload)) return ok(get_platform_store().create_model(payload))
except KeyError as exc: except KeyError as exc:
@@ -1047,14 +1122,18 @@ async def download_dataset_file(dataset_id: str, file_id: str, version_id: str |
@router.get("/dataset-manage") @router.get("/dataset-manage")
async def dataset_list(current_user: dict = Depends(get_current_user)) -> dict[str, Any]: async def dataset_list(current_user: dict = Depends(get_current_user)) -> dict[str, Any]:
datasets = get_platform_store().datasets() datasets = get_platform_store().datasets()
if current_user.get("role") == "admin" or current_user.get("protected"): if is_admin(current_user):
return ok(datasets) return ok(datasets)
# 普通用户可见:自己创建的 + ACL 授权的
user_id = current_user.get("id")
accessible = set(filter_accessible_resource_ids("dataset", [d["id"] for d in datasets], current_user)) accessible = set(filter_accessible_resource_ids("dataset", [d["id"] for d in datasets], current_user))
return ok([d for d in datasets if d["id"] in accessible]) result = [d for d in datasets if d.get("created_by") == user_id or d["id"] in accessible]
return ok(result)
@router.post("/dataset-manage") @router.post("/dataset-manage")
async def create_dataset(payload: dict[str, Any] = Body(...)) -> dict[str, Any]: async def create_dataset(payload: dict[str, Any] = Body(...), current_user: dict = Depends(get_current_user)) -> dict[str, Any]:
payload.setdefault("created_by", current_user.get("id"))
dataset = get_platform_store().create_dataset(payload) dataset = get_platform_store().create_dataset(payload)
return ok({"id": dataset["id"]}) return ok({"id": dataset["id"]})
@@ -1127,8 +1206,20 @@ async def create_fine_tune(payload: dict[str, Any] = Body(...)) -> dict[str, Any
@router.post("/fine-tune/start") @router.post("/fine-tune/start")
async def start_fine_tune(payload: dict[str, Any] = Body(...)) -> dict[str, Any]: async def start_fine_tune(
payload: dict[str, Any] = Body(...),
current_user: dict = Depends(get_current_user),
) -> dict[str, Any]:
store = get_platform_store() store = get_platform_store()
# GPU 权限校验:普通用户只能使用被分配的 GPU
if not is_admin(current_user):
node_id = payload.get("compute_node_id") or payload.get("node_id")
gpu_indices = payload.get("gpus") or []
if node_id and gpu_indices:
if not store.check_gpu_access(current_user["id"], node_id, gpu_indices):
raise fail(403, "无权使用所选 GPU请联系管理员分配")
# 记录创建者
payload.setdefault("created_by", current_user.get("id"))
try: try:
return ok(await _submit_fine_tune_task(store, payload)) return ok(await _submit_fine_tune_task(store, payload))
except KeyError: except KeyError:
@@ -1267,10 +1358,14 @@ async def update_fine_tune(task_id: str, payload: dict[str, Any] = Body(...)) ->
@router.post("/fine-tune/stop/{task_id}") @router.post("/fine-tune/stop/{task_id}")
async def stop_fine_tune(task_id: str) -> dict[str, Any]: async def stop_fine_tune(task_id: str, current_user: dict = Depends(get_current_user)) -> dict[str, Any]:
store = get_platform_store() store = get_platform_store()
try: try:
task = store.task(task_id) task = store.task(task_id)
# 审批拦截:非 admin 停止他人任务需审批
pending = _require_approval_or_admin("fine_tune_task", task_id, current_user, f"停止训练任务 {task_id}")
if pending:
return pending
node = _node_for_task(task) node = _node_for_task(task)
if task.get("compute_job_id") and node and get_settings().compute_mode != "simulator": if task.get("compute_job_id") and node and get_settings().compute_mode != "simulator":
job = await ComputeNodeClient(node["api_base_url"]).stop_job(task["compute_job_id"]) job = await ComputeNodeClient(node["api_base_url"]).stop_job(task["compute_job_id"])
@@ -1281,8 +1376,8 @@ async def stop_fine_tune(task_id: str) -> dict[str, Any]:
@router.post("/fine-tune/{task_id}/stop") @router.post("/fine-tune/{task_id}/stop")
async def stop_fine_tune_alt(task_id: str) -> dict[str, Any]: async def stop_fine_tune_alt(task_id: str, current_user: dict = Depends(get_current_user)) -> dict[str, Any]:
return await stop_fine_tune(task_id) return await stop_fine_tune(task_id, current_user)
@router.post("/fine-tune/{task_id}/retry") @router.post("/fine-tune/{task_id}/retry")

View File

@@ -1,4 +1,4 @@
from fastapi import APIRouter from fastapi import APIRouter
from app.api.v1.endpoints.data_process import router as data_process_router from app.api.v1.endpoints.data_process import router as data_process_router
from app.api.v1.endpoints.platform import router as platform_router from app.api.v1.endpoints.platform import router as platform_router
@@ -9,6 +9,8 @@ from app.modules.approval.router import router as approval_router
from app.modules.system.router import router as system_router from app.modules.system.router import router as system_router
from app.modules.retention.router import router as retention_router from app.modules.retention.router import router as retention_router
from app.modules.resource.router import router as resource_router from app.modules.resource.router import router as resource_router
from app.modules.gpu.router import router as gpu_router
from app.modules.data_convert.router import router as data_convert_router
api_router = APIRouter() api_router = APIRouter()
api_router.include_router(health_router, tags=["health"]) api_router.include_router(health_router, tags=["health"])
@@ -20,3 +22,5 @@ api_router.include_router(project_router, tags=["project"])
api_router.include_router(approval_router, tags=["approval"]) api_router.include_router(approval_router, tags=["approval"])
api_router.include_router(retention_router, tags=["retention"]) api_router.include_router(retention_router, tags=["retention"])
api_router.include_router(resource_router, tags=["resource"]) api_router.include_router(resource_router, tags=["resource"])
api_router.include_router(gpu_router, tags=["gpu-assignment"])
api_router.include_router(data_convert_router, tags=["data-convert"])

View File

@@ -1,6 +1,7 @@
from dataclasses import dataclass from dataclasses import dataclass
from functools import lru_cache from functools import lru_cache
import os import os
from typing import Any
try: try:
from pathlib import Path as _Path from pathlib import Path as _Path
@@ -29,6 +30,24 @@ def _list_env(name: str, default: list[str]) -> list[str]:
return [item.strip() for item in raw.split(",") if item.strip()] return [item.strip() for item in raw.split(",") if item.strip()]
def _bool_env(name: str, default: bool) -> bool:
raw = os.getenv(name)
if raw is None or raw.strip() == "":
return default
return raw.strip().lower() in {"1", "true", "yes", "on"}
def docs_kwargs(enabled: bool) -> dict[str, Any]:
"""Swagger UI / ReDoc / OpenAPI schema 路由开关。
关闭时 FastAPI 不注册 /docs、/redoc、/openapi.json访问一律返回 404
避免未授权访问泄露 API 结构。
"""
if enabled:
return {}
return {"docs_url": None, "redoc_url": None, "openapi_url": None}
@dataclass(frozen=True) @dataclass(frozen=True)
class Settings: class Settings:
app_name: str = os.getenv("APP_NAME", "YG Fine-Tune Platform API") app_name: str = os.getenv("APP_NAME", "YG Fine-Tune Platform API")
@@ -48,6 +67,7 @@ class Settings:
log_error_file_prefix: str = os.getenv("LOG_ERROR_FILE_PREFIX", "error") log_error_file_prefix: str = os.getenv("LOG_ERROR_FILE_PREFIX", "error")
log_max_bytes: int = _int_env("LOG_MAX_BYTES", 20 * 1024 * 1024) log_max_bytes: int = _int_env("LOG_MAX_BYTES", 20 * 1024 * 1024)
log_retention_days: int = _int_env("LOG_RETENTION_DAYS", 10) log_retention_days: int = _int_env("LOG_RETENTION_DAYS", 10)
enable_docs: bool = None # type: ignore[assignment]
def __post_init__(self) -> None: def __post_init__(self) -> None:
object.__setattr__( object.__setattr__(
@@ -63,6 +83,15 @@ class Settings:
], ],
), ),
) )
# Swagger UI / ReDoc / OpenAPI 文档路由开关:
# 未显式配置 ENABLE_DOCS 时,仅本地/开发环境开放,生产环境默认关闭,
# 避免未授权访问泄露 API 结构。从运行时环境读取 APP_ENV而非类定义时
# 缓存的默认值,保证生产默认关闭始终生效且便于测试。
object.__setattr__(
self,
"enable_docs",
_bool_env("ENABLE_DOCS", os.getenv("APP_ENV", "local") != "prod"),
)
@lru_cache @lru_cache

View File

@@ -239,10 +239,18 @@ def llama_dataset_info(dataset_key: str, file_names: list[str], formatting: str
fmt = str(formatting).lower() fmt = str(formatting).lower()
for key, file_name in zip(llama_dataset_keys(dataset_key, file_names), file_names): for key, file_name in zip(llama_dataset_keys(dataset_key, file_names), file_names):
if fmt == "sharegpt": if fmt == "sharegpt":
# 平台校验按 OpenAI 风格消息role/content故 tags 用 role/content
# 与 LLaMA-Factory 默认的 from/value 不同,需显式声明避免解析失败。
result[key] = { result[key] = {
"file_name": file_name, "file_name": file_name,
"formatting": "sharegpt", "formatting": "sharegpt",
"columns": {"messages": "messages"}, "columns": {"messages": "messages"},
"tags": {
"role_tag": "role",
"content_tag": "content",
"user_tag": "user",
"assistant_tag": "assistant",
},
} }
elif fmt == "dpo": elif fmt == "dpo":
result[key] = { result[key] = {
@@ -273,6 +281,47 @@ def llama_dataset_info(dataset_key: str, file_names: list[str], formatting: str
return result return result
def _sniff_dataset_format(sample_text: str, max_samples: int = 20) -> str:
"""嗅探数据集内容格式(兼容 jsonl返回 sharegpt / dpo / cpt / alpaca。
按内容而非文件名判断,纯 jsonl 数据集(如 ShareGPT messages、缺省 input 的
Alpaca都能被正确识别避免训练任务误按 alpaca 解析而失败。
"""
text = (sample_text or "").strip()
if not text:
return ""
records: list[dict[str, Any]] = []
try:
value = json.loads(text)
except (TypeError, ValueError, json.JSONDecodeError):
value = None
if isinstance(value, list):
records = [item for item in value[:max_samples] if isinstance(item, dict)]
elif isinstance(value, dict):
records = [value]
else:
for line in text.splitlines()[:max_samples]:
line = line.strip()
if not line:
continue
try:
obj = json.loads(line)
except json.JSONDecodeError:
continue
if isinstance(obj, dict):
records.append(obj)
records = records[:max_samples]
if not records:
return ""
if all("messages" in record for record in records):
return "sharegpt"
if all(record.get("chosen") and record.get("rejected") for record in records):
return "dpo"
if all(record.get("text") and not (record.get("instruction") or record.get("output")) for record in records):
return "cpt"
return "alpaca"
PASSWORD_HASH_ITERATIONS = 390_000 PASSWORD_HASH_ITERATIONS = 390_000
@@ -378,7 +427,7 @@ class PlatformStore:
# request (notably expensive against the remote PostgreSQL instance). # request (notably expensive against the remote PostgreSQL instance).
# TCP keepalive 让操作系统持续保活连接,抵抗远程库空闲静默断连。 # TCP keepalive 让操作系统持续保活连接,抵抗远程库空闲静默断连。
pool_kwargs = { pool_kwargs = {
"connect_timeout": 5, "connect_timeout": 30,
"keepalives": 1, "keepalives": 1,
"keepalives_idle": 10, "keepalives_idle": 10,
"keepalives_interval": 5, "keepalives_interval": 5,
@@ -388,14 +437,14 @@ class PlatformStore:
conninfo=self.database_url, conninfo=self.database_url,
kwargs=pool_kwargs, kwargs=pool_kwargs,
min_size=2, min_size=2,
max_size=10, max_size=20,
# 借出前校验连接可用性,避免执行 SQL 时才发现 [BAD] 再重建。 # 借出前校验连接可用性,避免执行 SQL 时才发现 [BAD] 再重建。
check=ConnectionPool.check_connection, check=ConnectionPool.check_connection,
# 不主动回收空闲连接(远程库约 10s 断,由 keepalive 维持), # 不主动回收空闲连接(远程库约 10s 断,由 keepalive 维持),
# 减少无谓的重建握手。 # 减少无谓的重建握手。
max_idle=0, max_idle=0,
# 请求最多排队等待 5s避免雪崩时无限堆积 # 请求最多排队等待,调大以适应远程库慢查询
max_waiting=16, max_waiting=50,
open=False, open=False,
) )
# 注意:不要在此调用 pool.wait(),它会阻塞等待 min_size 个连接就绪, # 注意:不要在此调用 pool.wait(),它会阻塞等待 min_size 个连接就绪,
@@ -476,7 +525,7 @@ class PlatformStore:
}, },
) )
schema_dir = Path(__file__).with_name("sql") schema_dir = Path(__file__).with_name("sql")
for extra in ("002_governance.sql", "003_tenant_quota.sql"): for extra in ("002_governance.sql", "003_tenant_quota.sql", "004_permissions.sql"):
extra_path = schema_dir / extra extra_path = schema_dir / extra
if extra_path.exists(): if extra_path.exists():
conn.executescript(extra_path.read_text(encoding="utf-8")) conn.executescript(extra_path.read_text(encoding="utf-8"))
@@ -618,8 +667,8 @@ class PlatformStore:
conn.execute( conn.execute(
""" """
INSERT INTO trained_models INSERT INTO trained_models
(id, name, train_methods, base_model_path, create_time, merged, merging, merged_path, artifact_dir, compute_node_id, compute_node_name) (id, name, train_methods, base_model_path, create_time, merged, merging, merged_path, artifact_dir, compute_node_id, compute_node_name, created_by)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""", """,
( (
trained_model_id, trained_model_id,
@@ -633,6 +682,7 @@ class PlatformStore:
output_dir, output_dir,
task.get("compute_node_id"), task.get("compute_node_id"),
task.get("compute_node_code") or task.get("compute_node_name"), task.get("compute_node_code") or task.get("compute_node_name"),
task.get("created_by"),
), ),
) )
# Use real artifact data from compute node when available # Use real artifact data from compute node when available
@@ -1235,10 +1285,22 @@ class PlatformStore:
row = conn.execute("SELECT * FROM users WHERE id=?", (user_id,)).fetchone() row = conn.execute("SELECT * FROM users WHERE id=?", (user_id,)).fetchone()
if not row: if not row:
raise KeyError(user_id) raise KeyError(user_id)
is_admin = row["role"] == "admin" or bool(row["protected"])
if "permissions" in payload:
perms = payload["permissions"]
if is_admin:
# 管理员权限不可更改,必须是全部
perms = ALL_PERMISSIONS
else:
# 非 admin 用户不能拥有 user-settings 权限
perms = [p for p in (perms or []) if p != "user-settings"]
payload = {**payload, "permissions": perms}
values = { values = {
"role": payload.get("role", row["role"]), "role": payload.get("role", row["role"]),
"status": payload.get("status", row["status"]), "status": payload.get("status", row["status"]),
"permissions": json_dumps(payload.get("permissions", json_loads(row["permissions"], []))), "permissions": json_dumps(
payload.get("permissions", json_loads(row["permissions"], []))
),
} }
conn.execute( conn.execute(
"UPDATE users SET role=?, status=?, permissions=? WHERE id=?", "UPDATE users SET role=?, status=?, permissions=? WHERE id=?",
@@ -1253,6 +1315,34 @@ class PlatformStore:
raise KeyError(user_id) raise KeyError(user_id)
if row["protected"]: if row["protected"]:
raise ValueError("protected user cannot be deleted") raise ValueError("protected user cannot be deleted")
# 级联删除该用户关联的数据
tables_to_clean = [
# ACL 授权
("acls", "principal_type='user' AND principal_id=?", [user_id]),
# 审批实例(申请人)
("approval_instances", "applicant_id=?", [user_id]),
# 审计日志
("audit_logs", "actor_id=?", [user_id]),
# 项目成员
("project_members", "user_id=?", [user_id]),
# GPU 分配
("gpu_assignments", "user_id=?", [user_id]),
# 数据集
("datasets", "created_by=?", [user_id]),
# 基座模型
("models", "created_by=?", [user_id]),
# 微调产物
("trained_models", "created_by=?", [user_id]),
# 评测任务
("eval_tasks", "created_by=?", [user_id]),
# 对比/推理任务payload 中 creator
# 训练任务:仅标记为已删除或保留(有 compute_job_id 关联),不清物理数据
]
for table_name, where_clause, params in tables_to_clean:
try:
conn.execute(f"DELETE FROM {table_name} WHERE {where_clause}", params)
except Exception:
pass # 表可能不存在或字段不存在,跳过
conn.execute("DELETE FROM users WHERE id=?", (user_id,)) conn.execute("DELETE FROM users WHERE id=?", (user_id,))
def reset_password(self, user_id: str, new_password: str) -> None: def reset_password(self, user_id: str, new_password: str) -> None:
@@ -1308,8 +1398,8 @@ class PlatformStore:
conn.execute( conn.execute(
""" """
INSERT INTO models INSERT INTO models
(id, name, type, purpose, model_source, description, path, api_url, api_key, online_model_name, can_train, create_time) (id, name, type, purpose, model_source, description, path, api_url, api_key, online_model_name, can_train, create_time, created_by)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""", """,
( (
model_id, model_id,
@@ -1324,6 +1414,7 @@ class PlatformStore:
payload.get("online_model_name"), payload.get("online_model_name"),
can_train, can_train,
utcnow(), utcnow(),
payload.get("created_by"),
), ),
) )
return dict(conn.execute("SELECT * FROM models WHERE id=?", (model_id,)).fetchone()) return dict(conn.execute("SELECT * FROM models WHERE id=?", (model_id,)).fetchone())
@@ -1491,8 +1582,8 @@ class PlatformStore:
conn.execute( conn.execute(
""" """
INSERT INTO datasets INSERT INTO datasets
(id, name, type, storage_type, source, task_id, size, count, description, create_time) (id, name, type, storage_type, source, task_id, size, count, description, create_time, created_by)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""", """,
( (
dataset_id, dataset_id,
@@ -1505,6 +1596,7 @@ class PlatformStore:
payload.get("count", 0), payload.get("count", 0),
payload.get("description"), payload.get("description"),
utcnow(), utcnow(),
payload.get("created_by"),
), ),
) )
return self._dataset(conn, conn.execute("SELECT * FROM datasets WHERE id=?", (dataset_id,)).fetchone()) return self._dataset(conn, conn.execute("SELECT * FROM datasets WHERE id=?", (dataset_id,)).fetchone())
@@ -2076,18 +2168,38 @@ class PlatformStore:
runtime_keys = llama_dataset_keys(dataset_key, runtime_file_names) runtime_keys = llama_dataset_keys(dataset_key, runtime_file_names)
training_keys = runtime_keys[: len(training_files)] training_keys = runtime_keys[: len(training_files)]
validation_keys = runtime_keys[len(training_files) :] validation_keys = runtime_keys[len(training_files) :]
dataset_format = str(task.get("dataset_format") or (dataset and dataset.get("formatting")) or "alpaca").lower() # P0-2: 推导数据集格式并校验内容(兼容 jsonl按内容嗅探 ShareGPT/DPO/CPT/Alpaca
# P0-2: Validate dataset content against declared format
train_type = str(task.get("train_type", task.get("train_method", ""))).upper() train_type = str(task.get("train_type", task.get("train_method", ""))).upper()
expected_format = { expected_format = {"DPO": "dpo", "CPT": "cpt"}.get(train_type)
"DPO": "dpo", raw_format = str(
"CPT": "cpt", task.get("dataset_format")
}.get(train_type) or dataset_metadata.get("format")
or (dataset and dataset.get("formatting"))
or "alpaca"
).lower()
sniffed_format = ""
content_samples: dict[str, str] = {}
if training_files:
with self.connect() as conn:
for file_entry in training_files:
sample_row = conn.execute(
"SELECT substr(content, 1, 400000) AS sample FROM dataset_files WHERE id=?",
(str(file_entry["id"]),),
).fetchone()
sample = (sample_row or {}).get("sample") or ""
content_samples[str(file_entry["id"])] = sample
if not sniffed_format:
sniffed_format = _sniff_dataset_format(sample)
known_formats = {"sharegpt", "dpo", "cpt", "pt", "pretrain"}
if expected_format: if expected_format:
dataset_format = expected_format dataset_format = expected_format
elif raw_format in known_formats:
dataset_format = raw_format
else:
dataset_format = sniffed_format or raw_format or "alpaca"
format_errors: list[str] = [] format_errors: list[str] = []
for file_entry in training_files: for file_entry in training_files:
content = file_entry.get("content") or "" content = content_samples.get(str(file_entry["id"])) or ""
if content: if content:
from app.modules.data_process.dataset_format import validate_dataset_format from app.modules.data_process.dataset_format import validate_dataset_format
file_errors = validate_dataset_format(dataset_format, content=content) file_errors = validate_dataset_format(dataset_format, content=content)
@@ -4005,6 +4117,130 @@ class PlatformStore:
"DELETE FROM retention_policies WHERE id=?", (policy_id,) "DELETE FROM retention_policies WHERE id=?", (policy_id,)
) )
# ===================== 平台治理GPU 算力分配 =====================
def gpu_assignments(self) -> list[dict[str, Any]]:
"""查询全部分配关系。"""
with self.connect() as conn:
rows = conn.execute(
"""
SELECT ga.*, u.username, u.display_name,
n.code AS node_code, n.name AS node_name, g.name AS gpu_name
FROM gpu_assignments ga
LEFT JOIN users u ON u.id = ga.user_id
LEFT JOIN compute_nodes n ON n.id = ga.node_id
LEFT JOIN gpus g ON g.node_id = ga.node_id AND g.gpu_index = ga.gpu_index
ORDER BY ga.assigned_at DESC
"""
).fetchall()
return [dict(r) for r in rows]
def gpu_assignments_for_user(self, user_id: str) -> list[dict[str, Any]]:
"""查询某用户被分配的 GPU 列表。"""
with self.connect() as conn:
rows = conn.execute(
"""
SELECT ga.node_id, ga.gpu_index,
n.code AS node_code, n.name AS node_name,
g.name AS gpu_name, g.uuid, g.memory_total_gb
FROM gpu_assignments ga
JOIN compute_nodes n ON n.id = ga.node_id
LEFT JOIN gpus g ON g.node_id = ga.node_id AND g.gpu_index = ga.gpu_index
WHERE ga.user_id = ?
ORDER BY n.code, ga.gpu_index
""",
(user_id,),
).fetchall()
return [dict(r) for r in rows]
def assign_gpus(self, assignments: list[dict[str, Any]], assigned_by: str | None = None) -> list[dict[str, Any]]:
"""批量分配 GPU幂等已存在的分配跳过"""
now = utcnow()
with self.connect() as conn:
for a in assignments:
node_id = a["node_id"]
gpu_index = a["gpu_index"]
user_id = a["user_id"]
existing = conn.execute(
"SELECT id FROM gpu_assignments WHERE node_id=? AND gpu_index=? AND user_id=?",
(node_id, gpu_index, user_id),
).fetchone()
if existing:
continue
aid = new_id("ga")
conn.execute(
"""
INSERT INTO gpu_assignments (id, node_id, gpu_index, user_id, assigned_by, assigned_at)
VALUES (?, ?, ?, ?, ?, ?)
""",
(aid, node_id, gpu_index, user_id, assigned_by, now),
)
return self.gpu_assignments()
def unassign_gpu(self, assignment_id: str) -> None:
with self.connect() as conn:
conn.execute("DELETE FROM gpu_assignments WHERE id=?", (assignment_id,))
def check_gpu_access(self, user_id: str, node_id: str, gpu_indices: list[int]) -> bool:
"""检查用户是否被分配了指定节点的指定 GPU 卡。"""
if not gpu_indices:
return True
with self.connect() as conn:
rows = conn.execute(
"""
SELECT gpu_index FROM gpu_assignments
WHERE user_id=? AND node_id=?
""",
(user_id, node_id),
).fetchall()
assigned = {r["gpu_index"] for r in rows}
return all(idx in assigned for idx in gpu_indices)
# ===================== 平台治理:资源可见性过滤 =====================
def _filter_accessible_ids(
self, resource_type: str, all_ids: list[str], user: dict[str, Any]
) -> list[str]:
"""从全部资源 ID 中过滤出当前用户可访问的 ID 列表。
- admin 直接返回全部。
- 资源所有者可见(需调用方在 all_ids 中提供 owned ids
- ACL 授权的用户/角色可见。
"""
if user.get("role") == "admin" or user.get("protected"):
return all_ids
if not all_ids:
return []
user_id = user.get("id")
user_role = user.get("role")
with self.connect() as conn:
rows = conn.execute(
"""
SELECT DISTINCT resource_id FROM acls
WHERE resource_type=? AND (
(principal_type='user' AND principal_id=?)
OR (principal_type='role' AND principal_id=?)
)
""",
(resource_type, user_id, user_role),
).fetchall()
accessible = {r["resource_id"] for r in rows}
return [rid for rid in all_ids if rid in accessible]
def change_password(self, user_id: str, old_password: str, new_password: str) -> bool:
"""用户自行修改密码:验证旧密码后设置新密码。"""
with self.connect() as conn:
row = conn.execute("SELECT * FROM users WHERE id=?", (user_id,)).fetchone()
if not row:
raise KeyError(user_id)
matched, _ = verify_password(old_password, row["password_hash"])
if not matched:
return False
conn.execute(
"UPDATE users SET password_hash=? WHERE id=?",
(hash_password(new_password), user_id),
)
return True
_store: PlatformStore | None = None _store: PlatformStore | None = None

View File

@@ -0,0 +1,755 @@
-- ============================================================================
-- YG Fine-Tune Platform — PostgreSQL 完整初始化脚本(一键建库建表)
-- ============================================================================
-- 用途:切换到新的 PG 数据集时,一次性创建平台运行所需的全部数据库对象与
-- 基础种子数据(幂等,可重复执行)。
--
-- 覆盖范围(与运行时代码实际使用的表一致):
-- 001_platform_runtime.sql 平台核心表
-- 002_governance.sql 治理表(租户 / 审批 / 审计 / 留存)
-- 003_tenant_quota.sql 租户配额列
-- 003_model_path_governance.sql 模型可训练标识列
-- 002_data_process.sql 数据处理表 + 数据集扩展列
-- 本文件补充data_convert_tasks数据转换任务运行时代码引用但原脚本缺失
-- 种子数据admin / operator 两个初始用户
--
-- 说明:
-- * 本脚本通过 psql 执行,包含 DO $$ ... $$ 块与事务,不能用应用的
-- executescript()(按分号切分)执行。
-- * 应用启动时 PlatformStore.ensure_schema() 只会自动执行
-- 001 / 002_governance / 003_tenant_quota数据处理表需另跑
-- 002_data_process.sql本脚本已包含。应用首次启动还会自动补充
-- admin/operator 种子用户(本脚本已包含,二选一即可)。
-- * 脚本内所有 DDL 均使用 IF NOT EXISTS / ADD COLUMN IF NOT EXISTS
-- 可在已初始化的库上安全重复执行。
--
-- 执行步骤(详见 docs/database-config.md
-- 1. 以超级用户创建角色与数据库(必须单独执行,不能放进事务):
-- CREATE ROLE yg_ft LOGIN PASSWORD '请改为强密码';
-- CREATE DATABASE yg_ft OWNER yg_ft;
-- 2. 连接目标库执行本脚本:
-- psql "postgresql://yg_ft:密码@<host>:5432/yg_ft" -f backend/app/db/sql/000_full_init.sql
-- 3. 可选:为 superuser 授权
-- ALTER ROLE yg_ft SUPERUSER; -- 仅当需要执行 CREATE EXTENSION 等
-- ============================================================================
BEGIN;
-- ============================================================================
-- 一、平台核心表来源001_platform_runtime.sql
-- ============================================================================
CREATE TABLE IF NOT EXISTS users (
id TEXT PRIMARY KEY,
username TEXT NOT NULL UNIQUE,
password_hash TEXT NOT NULL,
display_name TEXT NOT NULL,
role TEXT NOT NULL,
status TEXT NOT NULL,
permissions TEXT NOT NULL,
create_time TEXT NOT NULL,
last_login TEXT,
protected INTEGER NOT NULL DEFAULT 0
);
CREATE TABLE IF NOT EXISTS models (
id TEXT PRIMARY KEY,
name TEXT NOT NULL UNIQUE,
type TEXT NOT NULL,
purpose TEXT NOT NULL,
model_source TEXT NOT NULL,
description TEXT,
path TEXT,
api_url TEXT,
api_key TEXT,
online_model_name TEXT,
can_train INTEGER NOT NULL DEFAULT 0,
create_time TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS trained_models (
id TEXT PRIMARY KEY,
name TEXT NOT NULL UNIQUE,
train_methods TEXT NOT NULL,
base_model_path TEXT,
create_time TEXT NOT NULL,
merged INTEGER NOT NULL DEFAULT 0,
merging INTEGER NOT NULL DEFAULT 0,
merged_path TEXT,
artifact_dir TEXT,
compute_node_id TEXT,
compute_node_name TEXT
);
CREATE TABLE IF NOT EXISTS model_lineage (
id TEXT PRIMARY KEY,
child_resource_type TEXT NOT NULL,
child_resource_id TEXT NOT NULL,
parent_resource_type TEXT NOT NULL,
parent_resource_id TEXT NOT NULL,
relation_type TEXT NOT NULL,
compute_job_id TEXT,
payload TEXT NOT NULL,
create_time TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS model_artifacts (
id TEXT PRIMARY KEY,
model_id TEXT NOT NULL,
model_kind TEXT NOT NULL,
artifact_type TEXT NOT NULL,
path TEXT NOT NULL,
size_bytes BIGINT NOT NULL DEFAULT 0,
checksum_sha256 TEXT,
metadata TEXT NOT NULL,
compute_job_id TEXT,
create_time TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS model_export_jobs (
id TEXT PRIMARY KEY,
trained_model_id TEXT,
compute_job_id TEXT NOT NULL,
node_id TEXT,
export_type TEXT NOT NULL,
quantization_bit INTEGER NOT NULL DEFAULT 0,
status TEXT NOT NULL,
output_dir TEXT,
payload TEXT NOT NULL,
create_time TEXT NOT NULL,
completed_at TEXT
);
CREATE TABLE IF NOT EXISTS datasets (
id TEXT PRIMARY KEY,
name TEXT NOT NULL UNIQUE,
type TEXT NOT NULL,
storage_type TEXT NOT NULL,
source TEXT NOT NULL,
task_id TEXT,
size TEXT,
count INTEGER NOT NULL DEFAULT 0,
description TEXT,
create_time TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS dataset_files (
id TEXT PRIMARY KEY,
dataset_id TEXT NOT NULL REFERENCES datasets(id) ON DELETE CASCADE,
name TEXT NOT NULL,
size TEXT,
content TEXT NOT NULL,
active_version_id TEXT NOT NULL,
versions TEXT NOT NULL,
create_time TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS compute_nodes (
id TEXT PRIMARY KEY,
code TEXT NOT NULL UNIQUE,
name TEXT NOT NULL,
api_base_url TEXT NOT NULL,
file_gateway_url TEXT NOT NULL,
enabled INTEGER NOT NULL DEFAULT 1,
scheduler_status TEXT NOT NULL,
scheduler_weight INTEGER NOT NULL DEFAULT 100,
tags TEXT NOT NULL,
gpu_count INTEGER NOT NULL DEFAULT 0,
current_running_jobs INTEGER NOT NULL DEFAULT 0,
max_parallel_jobs INTEGER NOT NULL DEFAULT 2,
data_root TEXT NOT NULL,
model_root TEXT NOT NULL,
log_root TEXT NOT NULL,
api_version TEXT NOT NULL DEFAULT 'v1',
capabilities TEXT NOT NULL DEFAULT '[]',
description TEXT,
last_health_check_at TEXT,
health_detail TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS gpus (
id TEXT PRIMARY KEY,
node_id TEXT NOT NULL REFERENCES compute_nodes(id) ON DELETE CASCADE,
gpu_index INTEGER NOT NULL,
uuid TEXT NOT NULL,
name TEXT NOT NULL,
memory_total_gb DOUBLE PRECISION NOT NULL,
power_limit_w DOUBLE PRECISION NOT NULL,
base_temperature INTEGER NOT NULL,
last_seen_at TEXT
);
CREATE TABLE IF NOT EXISTS fine_tune_tasks (
id TEXT PRIMARY KEY,
name TEXT NOT NULL UNIQUE,
payload TEXT NOT NULL,
status TEXT NOT NULL,
progress INTEGER NOT NULL DEFAULT 0,
process_id INTEGER,
create_time TEXT NOT NULL,
start_time TEXT,
completed_at TEXT,
compute_node_id TEXT REFERENCES compute_nodes(id) ON DELETE SET NULL,
gpus TEXT NOT NULL,
sync_job_id TEXT,
compute_job_id TEXT
);
CREATE TABLE IF NOT EXISTS fine_tune_metrics (
id TEXT PRIMARY KEY,
task_id TEXT NOT NULL REFERENCES fine_tune_tasks(id) ON DELETE CASCADE,
step INTEGER NOT NULL,
epoch DOUBLE PRECISION,
loss DOUBLE PRECISION,
grad_norm DOUBLE PRECISION,
learning_rate DOUBLE PRECISION,
raw TEXT NOT NULL,
create_time TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS fine_tune_checkpoints (
id TEXT PRIMARY KEY,
task_id TEXT NOT NULL REFERENCES fine_tune_tasks(id) ON DELETE CASCADE,
step INTEGER NOT NULL,
name TEXT NOT NULL,
path TEXT NOT NULL,
size_bytes BIGINT NOT NULL DEFAULT 0,
create_time TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS compute_jobs (
id TEXT PRIMARY KEY,
task_id TEXT REFERENCES fine_tune_tasks(id) ON DELETE SET NULL,
node_id TEXT REFERENCES compute_nodes(id) ON DELETE SET NULL,
engine TEXT NOT NULL,
status TEXT NOT NULL,
command TEXT NOT NULL,
output_dir TEXT,
log_file TEXT,
payload TEXT NOT NULL,
create_time TEXT NOT NULL,
update_time TEXT NOT NULL,
completed_at TEXT
);
CREATE TABLE IF NOT EXISTS gpu_allocations (
id TEXT PRIMARY KEY,
task_id TEXT REFERENCES fine_tune_tasks(id) ON DELETE CASCADE,
compute_job_id TEXT,
node_id TEXT REFERENCES compute_nodes(id) ON DELETE CASCADE,
gpu_index INTEGER NOT NULL,
status TEXT NOT NULL,
create_time TEXT NOT NULL,
released_at TEXT
);
CREATE TABLE IF NOT EXISTS scheduler_locks (
lock_key TEXT PRIMARY KEY,
owner TEXT NOT NULL,
expires_at TEXT NOT NULL,
create_time TEXT NOT NULL,
update_time TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS resource_replicas (
id TEXT PRIMARY KEY,
node_id TEXT NOT NULL REFERENCES compute_nodes(id) ON DELETE CASCADE,
resource_type TEXT NOT NULL,
resource_id TEXT NOT NULL,
local_path TEXT NOT NULL,
status TEXT NOT NULL,
sync_status TEXT NOT NULL,
checksum_sha256 TEXT,
byte_size BIGINT NOT NULL DEFAULT 0,
last_checked_at TEXT,
last_error TEXT,
create_time TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS resource_sync_jobs (
id TEXT PRIMARY KEY,
target_node_id TEXT NOT NULL,
resources TEXT NOT NULL,
status TEXT NOT NULL,
progress INTEGER NOT NULL DEFAULT 0,
create_time TEXT NOT NULL,
completed_at TEXT
);
CREATE TABLE IF NOT EXISTS eval_tasks (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
payload TEXT NOT NULL,
status TEXT NOT NULL,
create_time TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS eval_dimensions (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
payload TEXT NOT NULL,
is_active INTEGER NOT NULL DEFAULT 1,
is_default INTEGER NOT NULL DEFAULT 0,
create_time TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS compare_tasks (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
payload TEXT NOT NULL,
status TEXT NOT NULL,
create_time TEXT NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_fine_tune_status ON fine_tune_tasks(status);
CREATE INDEX IF NOT EXISTS idx_fine_tune_compute_job ON fine_tune_tasks(compute_job_id);
CREATE INDEX IF NOT EXISTS idx_fine_tune_compute_node_status ON fine_tune_tasks(compute_node_id, status);
CREATE INDEX IF NOT EXISTS idx_model_lineage_child ON model_lineage(child_resource_type, child_resource_id);
CREATE INDEX IF NOT EXISTS idx_model_lineage_parent ON model_lineage(parent_resource_type, parent_resource_id);
CREATE INDEX IF NOT EXISTS idx_model_artifacts_model ON model_artifacts(model_kind, model_id, artifact_type);
CREATE INDEX IF NOT EXISTS idx_model_export_jobs_model ON model_export_jobs(trained_model_id, create_time DESC);
CREATE INDEX IF NOT EXISTS idx_model_export_jobs_compute ON model_export_jobs(compute_job_id);
CREATE INDEX IF NOT EXISTS idx_fine_tune_metrics_task_step ON fine_tune_metrics(task_id, step);
CREATE UNIQUE INDEX IF NOT EXISTS uq_fine_tune_metrics_task_step_epoch ON fine_tune_metrics(task_id, step, epoch);
CREATE INDEX IF NOT EXISTS idx_fine_tune_checkpoints_task_step ON fine_tune_checkpoints(task_id, step);
CREATE UNIQUE INDEX IF NOT EXISTS uq_fine_tune_checkpoints_task_path ON fine_tune_checkpoints(task_id, path);
CREATE INDEX IF NOT EXISTS idx_compute_jobs_task ON compute_jobs(task_id);
CREATE INDEX IF NOT EXISTS idx_compute_jobs_node_status ON compute_jobs(node_id, status);
CREATE INDEX IF NOT EXISTS idx_gpu_allocations_node_status ON gpu_allocations(node_id, status);
CREATE UNIQUE INDEX IF NOT EXISTS uq_gpu_allocations_active ON gpu_allocations(node_id, gpu_index) WHERE status IN ('allocated','running');
CREATE INDEX IF NOT EXISTS idx_scheduler_locks_expires ON scheduler_locks(expires_at);
CREATE INDEX IF NOT EXISTS idx_dataset_files_dataset ON dataset_files(dataset_id);
CREATE INDEX IF NOT EXISTS idx_gpus_node ON gpus(node_id);
CREATE UNIQUE INDEX IF NOT EXISTS uq_gpus_node_index ON gpus(node_id, gpu_index);
CREATE INDEX IF NOT EXISTS idx_replicas_resource ON resource_replicas(resource_type, resource_id);
CREATE UNIQUE INDEX IF NOT EXISTS uq_replicas_node_resource ON resource_replicas(node_id, resource_type, resource_id);
CREATE INDEX IF NOT EXISTS idx_sync_jobs_node_status ON resource_sync_jobs(target_node_id, status);
CREATE INDEX IF NOT EXISTS idx_eval_tasks_status ON eval_tasks(status);
CREATE INDEX IF NOT EXISTS idx_eval_dimensions_active ON eval_dimensions(is_active);
CREATE INDEX IF NOT EXISTS idx_compare_tasks_status ON compare_tasks(status);
-- ---- 项目 / 租户 ----
CREATE TABLE IF NOT EXISTS projects (
id TEXT PRIMARY KEY,
tenant_id TEXT NOT NULL DEFAULT 'default',
name TEXT NOT NULL,
code TEXT NOT NULL,
description TEXT,
quota TEXT,
status TEXT NOT NULL DEFAULT 'active',
create_time TEXT NOT NULL,
create_by TEXT,
updated_at TEXT
);
CREATE TABLE IF NOT EXISTS project_members (
project_id TEXT NOT NULL REFERENCES projects(id) ON DELETE CASCADE,
user_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
role TEXT NOT NULL DEFAULT 'member',
create_time TEXT NOT NULL,
PRIMARY KEY (project_id, user_id)
);
CREATE TABLE IF NOT EXISTS roles (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
permissions TEXT NOT NULL DEFAULT '[]',
create_time TEXT
);
CREATE TABLE IF NOT EXISTS sessions (
id TEXT PRIMARY KEY,
user_id TEXT NOT NULL,
issued_at TEXT NOT NULL,
expires_at TEXT NOT NULL,
ip TEXT
);
CREATE TABLE IF NOT EXISTS acls (
id TEXT PRIMARY KEY,
resource_type TEXT NOT NULL,
resource_id TEXT NOT NULL,
principal_type TEXT NOT NULL,
principal_id TEXT NOT NULL,
permission TEXT NOT NULL,
create_time TEXT
);
-- ============================================================================
-- 二、治理表来源002_governance.sql
-- ============================================================================
CREATE TABLE IF NOT EXISTS tenants (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
code TEXT,
status TEXT DEFAULT 'active',
owner_user_id TEXT,
quota TEXT,
retention_policy_id TEXT,
create_time TEXT
);
CREATE TABLE IF NOT EXISTS approval_templates (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
steps TEXT,
create_time TEXT
);
CREATE TABLE IF NOT EXISTS approval_instances (
id TEXT PRIMARY KEY,
template_id TEXT,
resource_type TEXT,
resource_id TEXT,
applicant_id TEXT,
status TEXT DEFAULT 'pending',
current_step INTEGER DEFAULT 0,
create_time TEXT
);
CREATE TABLE IF NOT EXISTS approval_steps (
id TEXT PRIMARY KEY,
instance_id TEXT,
step_index INTEGER,
approver_id TEXT,
status TEXT DEFAULT 'pending',
comment TEXT,
time TEXT
);
CREATE TABLE IF NOT EXISTS audit_logs (
id TEXT PRIMARY KEY,
tenant_id TEXT,
project_id TEXT,
actor_id TEXT,
action TEXT,
target_type TEXT,
target_id TEXT,
detail TEXT,
client_ip TEXT,
time TEXT
);
CREATE INDEX IF NOT EXISTS idx_audit_tenant ON audit_logs(tenant_id);
CREATE INDEX IF NOT EXISTS idx_audit_project ON audit_logs(project_id);
CREATE INDEX IF NOT EXISTS idx_audit_action ON audit_logs(action);
CREATE INDEX IF NOT EXISTS idx_audit_time ON audit_logs(time);
CREATE TABLE IF NOT EXISTS retention_policies (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
scope TEXT,
rule TEXT,
status TEXT DEFAULT 'active',
create_time TEXT,
create_by TEXT,
updated_at TEXT
);
-- ============================================================================
-- 三、租户配额扩展来源003_tenant_quota.sql
-- ============================================================================
ALTER TABLE tenants ADD COLUMN IF NOT EXISTS gpu_quota TEXT;
ALTER TABLE tenants ADD COLUMN IF NOT EXISTS storage_quota TEXT;
-- ============================================================================
-- 四、模型路径治理来源003_model_path_governance.sql
-- models.can_train 已在建表语句中声明;以下为兼容旧库的幂等语句。
-- ============================================================================
ALTER TABLE models ADD COLUMN IF NOT EXISTS can_train INTEGER NOT NULL DEFAULT 0;
ALTER TABLE trained_models ADD COLUMN IF NOT EXISTS artifact_dir TEXT;
-- 按规则推定已有模型的 can_train新库为空表此语句为 no-op
UPDATE models
SET can_train = CASE
WHEN path IS NOT NULL AND path != '' AND model_source IS NOT NULL AND model_source != 'api' THEN 1
ELSE 0
END;
-- ============================================================================
-- 五、数据处理来源002_data_process.sql去掉其外层 BEGIN/COMMIT
-- 数据集扩展列
-- ============================================================================
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS source_task_id TEXT;
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS size_bytes BIGINT NOT NULL DEFAULT 0;
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS record_count BIGINT NOT NULL DEFAULT 0;
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS metadata TEXT NOT NULL DEFAULT '{}';
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS tenant_id TEXT;
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS project_id TEXT;
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS owner_id TEXT;
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS created_by TEXT;
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS created_at TIMESTAMPTZ NOT NULL DEFAULT now();
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS updated_at TIMESTAMPTZ NOT NULL DEFAULT now();
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS deleted_at TIMESTAMPTZ;
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS storage_object_id TEXT;
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS current_version_id TEXT;
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS size_bytes BIGINT NOT NULL DEFAULT 0;
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS record_count BIGINT NOT NULL DEFAULT 0;
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS file_format VARCHAR(40);
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS checksum_sha256 CHAR(64);
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS version_no INTEGER NOT NULL DEFAULT 1;
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS source_task_id TEXT;
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS tenant_id TEXT;
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS project_id TEXT;
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS created_by TEXT;
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS metadata TEXT NOT NULL DEFAULT '{}';
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS created_at TIMESTAMPTZ NOT NULL DEFAULT now();
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS updated_at TIMESTAMPTZ NOT NULL DEFAULT now();
ALTER TABLE dataset_files ADD COLUMN IF NOT EXISTS deleted_at TIMESTAMPTZ;
-- ---- 数据处理任务 / 源文件 / 预览 / 结果 ----
CREATE TABLE IF NOT EXISTS data_process_tasks (
id TEXT PRIMARY KEY,
name VARCHAR(150) NOT NULL,
description TEXT,
status VARCHAR(20) NOT NULL DEFAULT 'pending'
CHECK (status IN ('pending', 'running', 'completed', 'failed', 'stopped')),
process_type VARCHAR(20) NOT NULL
CHECK (process_type IN ('structured', 'unstructured', 'external')),
source_dataset_id TEXT REFERENCES datasets(id) ON DELETE SET NULL,
output_dataset_id TEXT REFERENCES datasets(id) ON DELETE SET NULL,
config TEXT NOT NULL DEFAULT '{}',
progress NUMERIC(5,2) NOT NULL DEFAULT 0 CHECK (progress >= 0 AND progress <= 100),
input_count BIGINT NOT NULL DEFAULT 0 CHECK (input_count >= 0),
output_count BIGINT NOT NULL DEFAULT 0 CHECK (output_count >= 0),
filtered_count BIGINT NOT NULL DEFAULT 0 CHECK (filtered_count >= 0),
duplicate_count BIGINT NOT NULL DEFAULT 0 CHECK (duplicate_count >= 0),
error_count BIGINT NOT NULL DEFAULT 0 CHECK (error_count >= 0),
failure_reason TEXT,
generation_run_id TEXT,
results_confirmed BOOLEAN NOT NULL DEFAULT TRUE,
workflow_step VARCHAR(20) NOT NULL DEFAULT 'create'
CHECK (workflow_step IN ('create', 'model', 'upload', 'preview', 'generate', 'results')),
preview_status VARCHAR(20) NOT NULL DEFAULT 'idle'
CHECK (preview_status IN ('idle', 'queued', 'running', 'completed', 'failed', 'cancelled')),
preview_progress NUMERIC(5,2) NOT NULL DEFAULT 0
CHECK (preview_progress >= 0 AND preview_progress <= 100),
preview_run_id TEXT,
preview_failure_reason TEXT,
preview_total_files INTEGER NOT NULL DEFAULT 0 CHECK (preview_total_files >= 0),
preview_completed_files INTEGER NOT NULL DEFAULT 0 CHECK (preview_completed_files >= 0),
tenant_id TEXT,
project_id TEXT,
owner_id TEXT,
approval_status VARCHAR(30) NOT NULL DEFAULT 'not_required',
created_by TEXT,
updated_by TEXT,
deleted_by TEXT,
started_at TIMESTAMPTZ,
completed_at TIMESTAMPTZ,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
deleted_at TIMESTAMPTZ
);
ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS generation_run_id TEXT;
ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS results_confirmed BOOLEAN NOT NULL DEFAULT TRUE;
ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS workflow_step VARCHAR(20);
ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_status VARCHAR(20);
ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_progress NUMERIC(5,2);
ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_run_id TEXT;
ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_failure_reason TEXT;
ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_total_files INTEGER;
ALTER TABLE data_process_tasks ADD COLUMN IF NOT EXISTS preview_completed_files INTEGER;
ALTER TABLE data_process_tasks ALTER COLUMN workflow_step SET DEFAULT 'create';
ALTER TABLE data_process_tasks ALTER COLUMN workflow_step SET NOT NULL;
ALTER TABLE data_process_tasks ALTER COLUMN preview_status SET DEFAULT 'idle';
ALTER TABLE data_process_tasks ALTER COLUMN preview_status SET NOT NULL;
ALTER TABLE data_process_tasks ALTER COLUMN preview_progress SET DEFAULT 0;
ALTER TABLE data_process_tasks ALTER COLUMN preview_progress SET NOT NULL;
ALTER TABLE data_process_tasks ALTER COLUMN preview_total_files SET DEFAULT 0;
ALTER TABLE data_process_tasks ALTER COLUMN preview_total_files SET NOT NULL;
ALTER TABLE data_process_tasks ALTER COLUMN preview_completed_files SET DEFAULT 0;
ALTER TABLE data_process_tasks ALTER COLUMN preview_completed_files SET NOT NULL;
CREATE UNIQUE INDEX IF NOT EXISTS uq_data_process_tasks_name_alive
ON data_process_tasks(name) WHERE deleted_at IS NULL;
CREATE INDEX IF NOT EXISTS idx_data_process_tasks_scope_status
ON data_process_tasks(tenant_id, project_id, status, created_at DESC)
WHERE deleted_at IS NULL;
CREATE INDEX IF NOT EXISTS idx_data_process_tasks_creator_created
ON data_process_tasks(created_by, created_at DESC) WHERE deleted_at IS NULL;
CREATE TABLE IF NOT EXISTS data_process_source_files (
id TEXT PRIMARY KEY,
task_id TEXT NOT NULL REFERENCES data_process_tasks(id) ON DELETE CASCADE,
storage_object_id TEXT,
name TEXT NOT NULL,
size_bytes BIGINT NOT NULL DEFAULT 0 CHECK (size_bytes >= 0),
record_count BIGINT NOT NULL DEFAULT 0 CHECK (record_count >= 0),
file_format VARCHAR(40),
checksum_sha256 CHAR(64) NOT NULL,
version_no INTEGER NOT NULL DEFAULT 1 CHECK (version_no > 0),
content TEXT NOT NULL,
content_preview TEXT,
metadata TEXT NOT NULL DEFAULT '{}',
tenant_id TEXT,
project_id TEXT,
created_by TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
deleted_at TIMESTAMPTZ
);
CREATE INDEX IF NOT EXISTS idx_data_process_source_files_task
ON data_process_source_files(task_id, created_at) WHERE deleted_at IS NULL;
CREATE UNIQUE INDEX IF NOT EXISTS uq_data_process_source_checksum_alive
ON data_process_source_files(task_id, checksum_sha256) WHERE deleted_at IS NULL;
CREATE TABLE IF NOT EXISTS data_process_preview_items (
id TEXT PRIMARY KEY,
task_id TEXT NOT NULL REFERENCES data_process_tasks(id) ON DELETE CASCADE,
source_file_id TEXT REFERENCES data_process_source_files(id) ON DELETE CASCADE,
original_content TEXT NOT NULL DEFAULT '',
edited_content TEXT NOT NULL DEFAULT '',
source_start INTEGER CHECK (source_start IS NULL OR source_start >= 0),
source_end INTEGER CHECK (source_end IS NULL OR source_end >= 0),
source_start_line INTEGER CHECK (source_start_line IS NULL OR source_start_line > 0),
source_end_line INTEGER CHECK (source_end_line IS NULL OR source_end_line > 0),
token_count INTEGER NOT NULL DEFAULT 0 CHECK (token_count >= 0),
status VARCHAR(20) NOT NULL DEFAULT 'original'
CHECK (status IN ('original', 'modified', 'manual', 'invalid')),
quality_score TEXT NOT NULL DEFAULT '{}',
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CHECK (source_start IS NULL OR source_end IS NULL OR source_end >= source_start),
CHECK (source_start_line IS NULL OR source_end_line IS NULL OR source_end_line >= source_start_line)
);
CREATE INDEX IF NOT EXISTS idx_data_process_preview_task_file
ON data_process_preview_items(task_id, source_file_id, created_at);
CREATE TABLE IF NOT EXISTS data_process_results (
id TEXT PRIMARY KEY,
task_id TEXT NOT NULL REFERENCES data_process_tasks(id) ON DELETE CASCADE,
preview_item_id TEXT REFERENCES data_process_preview_items(id) ON DELETE SET NULL,
instruction TEXT NOT NULL,
input TEXT NOT NULL DEFAULT '',
output TEXT NOT NULL,
original_instruction TEXT,
original_input TEXT,
original_output TEXT,
status VARCHAR(20) NOT NULL DEFAULT 'valid'
CHECK (status IN ('valid', 'modified', 'invalid')),
error TEXT,
split VARCHAR(20) CHECK (split IS NULL OR split IN ('train', 'validation', 'test')),
quality_score TEXT NOT NULL DEFAULT '{}',
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS idx_data_process_results_task_status
ON data_process_results(task_id, status, id);
CREATE INDEX IF NOT EXISTS idx_data_process_results_task_split
ON data_process_results(task_id, split);
-- ---- 数据集版本 / 记录 ----
CREATE TABLE IF NOT EXISTS dataset_file_versions (
id TEXT PRIMARY KEY,
dataset_file_id TEXT NOT NULL REFERENCES dataset_files(id) ON DELETE CASCADE,
version_no INTEGER NOT NULL CHECK (version_no > 0),
storage_object_id TEXT NOT NULL,
content_preview TEXT,
description TEXT,
base_version_id TEXT REFERENCES dataset_file_versions(id) ON DELETE SET NULL,
size_bytes BIGINT NOT NULL DEFAULT 0 CHECK (size_bytes >= 0),
record_count BIGINT NOT NULL DEFAULT 0 CHECK (record_count >= 0),
checksum_sha256 CHAR(64) NOT NULL,
source_task_id TEXT REFERENCES data_process_tasks(id) ON DELETE SET NULL,
metadata TEXT NOT NULL DEFAULT '{}',
created_by TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
ALTER TABLE dataset_file_versions ADD COLUMN IF NOT EXISTS source_task_id TEXT;
ALTER TABLE dataset_file_versions ADD COLUMN IF NOT EXISTS metadata TEXT NOT NULL DEFAULT '{}';
CREATE UNIQUE INDEX IF NOT EXISTS uq_dataset_file_versions_no_002
ON dataset_file_versions(dataset_file_id, version_no);
CREATE INDEX IF NOT EXISTS idx_dataset_file_versions_source_task_002
ON dataset_file_versions(source_task_id) WHERE source_task_id IS NOT NULL;
CREATE TABLE IF NOT EXISTS dataset_records (
id TEXT PRIMARY KEY,
dataset_id TEXT NOT NULL REFERENCES datasets(id) ON DELETE CASCADE,
dataset_file_id TEXT REFERENCES dataset_files(id) ON DELETE CASCADE,
version_id TEXT REFERENCES dataset_file_versions(id) ON DELETE CASCADE,
line_no INTEGER,
split VARCHAR(20) CHECK (split IS NULL OR split IN ('train', 'validation', 'test')),
instruction TEXT,
input TEXT,
output TEXT,
raw TEXT NOT NULL DEFAULT '{}',
status VARCHAR(20) NOT NULL DEFAULT 'valid'
CHECK (status IN ('valid', 'modified', 'invalid')),
source_task_id TEXT REFERENCES data_process_tasks(id) ON DELETE SET NULL,
source_result_id TEXT REFERENCES data_process_results(id) ON DELETE SET NULL,
preview_item_id TEXT REFERENCES data_process_preview_items(id) ON DELETE SET NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
ALTER TABLE dataset_records ADD COLUMN IF NOT EXISTS source_task_id TEXT;
ALTER TABLE dataset_records ADD COLUMN IF NOT EXISTS source_result_id TEXT;
ALTER TABLE dataset_records ADD COLUMN IF NOT EXISTS preview_item_id TEXT;
CREATE INDEX IF NOT EXISTS idx_dataset_records_dataset_002
ON dataset_records(dataset_id, id);
CREATE INDEX IF NOT EXISTS idx_dataset_records_source_task_002
ON dataset_records(source_task_id, source_result_id);
CREATE INDEX IF NOT EXISTS idx_datasets_source_task_002
ON datasets(source_task_id) WHERE source_task_id IS NOT NULL;
CREATE INDEX IF NOT EXISTS idx_dataset_files_source_task_002
ON dataset_files(source_task_id) WHERE source_task_id IS NOT NULL;
-- ============================================================================
-- 六、数据转换任务data_convert_tasks
-- 运行时 routerapp/modules/data_convert/router.py引用但原脚本缺失本文件补齐。
-- ============================================================================
CREATE TABLE IF NOT EXISTS data_convert_tasks (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
description TEXT,
output_filename TEXT DEFAULT 'converted-data.jsonl',
status TEXT NOT NULL DEFAULT 'pending',
input_count INTEGER NOT NULL DEFAULT 0,
output_count INTEGER NOT NULL DEFAULT 0,
error_message TEXT,
create_time TEXT NOT NULL DEFAULT (to_char(now(), 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"')),
update_time TEXT NOT NULL DEFAULT (to_char(now(), 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"')),
deleted_at TIMESTAMPTZ
);
CREATE INDEX IF NOT EXISTS idx_data_convert_tasks_status ON data_convert_tasks(status);
CREATE INDEX IF NOT EXISTS idx_data_convert_tasks_create_time ON data_convert_tasks(create_time DESC);
-- ============================================================================
-- 七、种子数据:初始管理员 / 操作员
-- 应用首次启动ensure_seed_data也会自动创建此处提供以便脱离应用直接初始化。
-- 密码admin / admin123operator / operator123上线前请改密
-- ============================================================================
INSERT INTO users
(id, username, password_hash, display_name, role, status, permissions, create_time, protected)
VALUES
(
'u_admin', 'admin', 'pbkdf2_sha256$390000$ygft_init_salt_admin$2b6f31f22968c4f5a30bcf0acf066b7a0f58d4773d15c5ab898ba715ea87b5bd',
'Platform Admin', 'admin', 'active',
'["dashboard","fine-tune","model-eval","model-inference","model-manage","dataset","data-process","data-convert","compute","hardware","logs","user-settings"]',
to_char(now(), 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"'), 1
),
(
'u_operator', 'operator', 'pbkdf2_sha256$390000$ygft_init_salt_op$525bf35d02ed26f37952cbd6862b0ae358b9d1a7fa0cbbf0217aa2b5dd544125',
'Platform Operator', 'operator', 'active',
'["dashboard","fine-tune","model-eval","model-inference","model-manage","dataset","data-process","data-convert","compute","hardware","logs"]',
to_char(now(), 'YYYY-MM-DD"T"HH24:MI:SS.MS"Z"'), 0
)
ON CONFLICT (username) DO NOTHING;
COMMIT;

View File

@@ -268,9 +268,13 @@ CREATE TABLE IF NOT EXISTS data_process_results (
instruction TEXT NOT NULL, instruction TEXT NOT NULL,
input TEXT NOT NULL DEFAULT '', input TEXT NOT NULL DEFAULT '',
output TEXT NOT NULL, output TEXT NOT NULL,
chosen TEXT NOT NULL DEFAULT '',
rejected TEXT NOT NULL DEFAULT '',
original_instruction TEXT, original_instruction TEXT,
original_input TEXT, original_input TEXT,
original_output TEXT, original_output TEXT,
original_chosen TEXT,
original_rejected TEXT,
status VARCHAR(20) NOT NULL DEFAULT 'valid' status VARCHAR(20) NOT NULL DEFAULT 'valid'
CHECK (status IN ('valid', 'modified', 'invalid')), CHECK (status IN ('valid', 'modified', 'invalid')),
error TEXT, error TEXT,
@@ -280,6 +284,11 @@ CREATE TABLE IF NOT EXISTS data_process_results (
updated_at TIMESTAMPTZ NOT NULL DEFAULT now() updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
); );
ALTER TABLE data_process_results ADD COLUMN IF NOT EXISTS chosen TEXT NOT NULL DEFAULT '';
ALTER TABLE data_process_results ADD COLUMN IF NOT EXISTS rejected TEXT NOT NULL DEFAULT '';
ALTER TABLE data_process_results ADD COLUMN IF NOT EXISTS original_chosen TEXT;
ALTER TABLE data_process_results ADD COLUMN IF NOT EXISTS original_rejected TEXT;
CREATE INDEX IF NOT EXISTS idx_data_process_results_task_status CREATE INDEX IF NOT EXISTS idx_data_process_results_task_status
ON data_process_results(task_id, status, id); ON data_process_results(task_id, status, id);
CREATE INDEX IF NOT EXISTS idx_data_process_results_task_split CREATE INDEX IF NOT EXISTS idx_data_process_results_task_split

View File

@@ -0,0 +1,22 @@
-- ============================================================
-- 权限体系扩展GPU 分配表 + 资源所有权字段
-- ============================================================
-- GPU 分配表:管理员指定哪些用户可以使用哪些 GPU 卡
CREATE TABLE IF NOT EXISTS gpu_assignments (
id TEXT PRIMARY KEY,
node_id TEXT NOT NULL REFERENCES compute_nodes(id) ON DELETE CASCADE,
gpu_index INTEGER NOT NULL,
user_id TEXT NOT NULL,
assigned_by TEXT,
assigned_at TEXT NOT NULL,
UNIQUE (node_id, gpu_index, user_id)
);
CREATE INDEX IF NOT EXISTS idx_gpu_assignments_user ON gpu_assignments(user_id);
CREATE INDEX IF NOT EXISTS idx_gpu_assignments_gpu ON gpu_assignments(node_id, gpu_index);
-- 资源所有权字段:用户创建的数据集/模型/训练产物/评测任务
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS created_by TEXT;
ALTER TABLE models ADD COLUMN IF NOT EXISTS created_by TEXT;
ALTER TABLE trained_models ADD COLUMN IF NOT EXISTS created_by TEXT;
ALTER TABLE eval_tasks ADD COLUMN IF NOT EXISTS created_by TEXT;

View File

@@ -5,7 +5,7 @@ from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware from fastapi.middleware.cors import CORSMiddleware
from app.api.v1.router import api_router from app.api.v1.router import api_router
from app.core.config import get_settings from app.core.config import docs_kwargs, get_settings
from app.core.logging import configure_logging, setup_request_logging from app.core.logging import configure_logging, setup_request_logging
from app.workers.compute_poller import run_compute_poller from app.workers.compute_poller import run_compute_poller
@@ -14,7 +14,7 @@ def create_app() -> FastAPI:
settings = get_settings() settings = get_settings()
configure_logging(settings) configure_logging(settings)
app = FastAPI(title=settings.app_name) app = FastAPI(title=settings.app_name, **docs_kwargs(settings.enable_docs))
app.add_middleware( app.add_middleware(
CORSMiddleware, CORSMiddleware,
allow_origins=settings.cors_allow_origins, allow_origins=settings.cors_allow_origins,

View File

@@ -0,0 +1,3 @@
from .router import router
__all__ = ["router"]

View File

@@ -0,0 +1,356 @@
from __future__ import annotations
import json
import os
from pathlib import Path
from typing import Any
from fastapi import APIRouter, Body, Depends, File, UploadFile
from fastapi.responses import FileResponse
from app.api.v1.endpoints.platform import ok, fail
from app.core.auth import get_current_user
from app.db.platform_store import get_platform_store, new_id
router = APIRouter(prefix="/data-convert", tags=["data-convert"])
# 存储根目录
STORAGE_ROOT = Path(__file__).resolve().parents[3] / "storage" / "data-convert"
def _safe_output_filename(value: Any) -> str:
"""输出文件名白名单校验:仅允许普通文件名,阻断 ``../``、``/``、``\\`` 等路径穿越。
转换结果始终写入 ``STORAGE_ROOT/<task_id>/output/<output_filename>``
若文件名可被注入路径分隔符,将导致任意文件读写/删除。
"""
name = str(value or "converted-data.jsonl").strip()
if (
not name
or name in {".", ".."}
or name != Path(name).name
or "/" in name
or "\\" in name
or any(ord(character) < 32 or ord(character) == 127 for character in name)
):
raise fail(400, "output filename must be a plain file name")
return name
def _task_output_path(task: dict[str, Any]) -> Path:
"""返回经过白名单校验的转换输出文件路径(始终位于任务 output 目录内)。"""
return _output_dir(task["id"]) / _safe_output_filename(task.get("output_filename"))
def _task_dir(task_id: str) -> Path:
return STORAGE_ROOT / task_id
def _input_dir(task_id: str) -> Path:
return _task_dir(task_id) / "input"
def _output_dir(task_id: str) -> Path:
return _task_dir(task_id) / "output"
@router.get("")
def list_tasks(
page: int = 1,
page_size: int = 20,
current_user: dict = Depends(get_current_user),
) -> dict[str, Any]:
store = get_platform_store()
with store.connect() as conn:
rows = conn.execute(
"SELECT * FROM data_convert_tasks WHERE deleted_at IS NULL "
"ORDER BY create_time DESC LIMIT %s OFFSET %s",
(page_size, (page - 1) * page_size),
).fetchall()
total = conn.execute(
"SELECT COUNT(*) FROM data_convert_tasks WHERE deleted_at IS NULL"
).fetchone()[0]
return ok({"items": [dict(r) for r in rows], "total": total})
@router.post("")
def create_task(
payload: dict[str, Any] = Body(...),
current_user: dict = Depends(get_current_user),
) -> dict[str, Any]:
name = str(payload.get("name") or "").strip()
if not name:
raise fail(400, "name is required")
task_id = new_id("dct")
output_filename = _safe_output_filename(payload.get("output_filename"))
description = str(payload.get("description") or "").strip()
store = get_platform_store()
with store.connect() as conn:
conn.execute(
"INSERT INTO data_convert_tasks (id, name, description, output_filename) "
"VALUES (%s, %s, %s, %s)",
(task_id, name, description, output_filename),
)
# 创建目录
_input_dir(task_id).mkdir(parents=True, exist_ok=True)
_output_dir(task_id).mkdir(parents=True, exist_ok=True)
return ok(_get_task(task_id))
@router.get("/{task_id}")
def get_task(
task_id: str,
current_user: dict = Depends(get_current_user),
) -> dict[str, Any]:
task = _get_task(task_id)
if not task:
raise fail(404, "task not found")
# 附加输入文件列表
input_dir = _input_dir(task_id)
files = []
if input_dir.exists():
for f in sorted(input_dir.iterdir()):
if f.is_file():
files.append({"name": f.name, "size": f.stat().st_size})
task["input_files"] = files
return ok(task)
@router.post("/{task_id}/source-files")
async def upload_source_files(
task_id: str,
files: list[UploadFile] = File(...),
current_user: dict = Depends(get_current_user),
) -> dict[str, Any]:
task = _get_task(task_id)
if not task:
raise fail(404, "task not found")
if task["status"] not in ("pending", "uploaded"):
raise fail(400, "task is not editable")
input_dir = _input_dir(task_id)
input_dir.mkdir(parents=True, exist_ok=True)
staged = []
for upload in files:
name = Path(upload.filename or "input.json").name
if not name.lower().endswith(".json"):
raise fail(415, f"only JSON files are supported: {name}")
target = input_dir / name
content = await upload.read()
target.write_bytes(content)
staged.append({"name": name, "size": len(content)})
store = get_platform_store()
# 标记上传完成
with store.connect() as conn:
conn.execute(
"UPDATE data_convert_tasks SET status='uploaded', update_time=NOW() WHERE id=%s",
(task_id,),
)
# 自动转换并导入数据集
try:
output_dir = _output_dir(task_id)
output_dir.mkdir(parents=True, exist_ok=True)
output_path = _task_output_path(task)
# 清空旧输出(如果重新上传)
if output_path.exists():
output_path.unlink()
input_count = 0
output_count = 0
for json_file in sorted(input_dir.iterdir()):
if not json_file.is_file() or not json_file.name.lower().endswith(".json"):
continue
input_count += 1
with open(json_file, "r", encoding="utf-8") as f:
data = json.load(f)
if isinstance(data, list):
records = data
elif isinstance(data, dict):
records = [data]
else:
raise ValueError(f"JSON must be object or array: {json_file.name}")
with open(output_path, "a", encoding="utf-8") as f:
for record in records:
f.write(json.dumps(record, ensure_ascii=False) + "\n")
output_count += 1
with store.connect() as conn:
conn.execute(
"UPDATE data_convert_tasks SET status='completed', "
"input_count=%s, output_count=%s, update_time=NOW() WHERE id=%s",
(input_count, output_count, task_id),
)
# 自动导入数据集
content = output_path.read_text(encoding="utf-8")
size_bytes = len(content.encode("utf-8"))
dataset = store.create_dataset({
"name": task["name"],
"type": "train",
"storage_type": "local",
"source": "upload",
"task_id": task_id,
"size": f"{size_bytes} B",
"count": output_count,
"description": f"由数据类型转换任务 {task_id} 自动导入",
})
dataset_id = dataset["id"]
with store.connect() as conn:
store.add_dataset_file(conn, dataset_id, _safe_output_filename(task.get("output_filename")), content)
return ok({
"staged_files": staged,
"auto_converted": True,
"dataset_id": dataset_id,
"input_count": input_count,
"output_count": output_count,
})
except Exception as exc:
with store.connect() as conn:
conn.execute(
"UPDATE data_convert_tasks SET status='failed', error_message=%s, update_time=NOW() WHERE id=%s",
(str(exc)[:500], task_id),
)
return ok({"staged_files": staged, "auto_converted": False, "error": str(exc)[:500]})
@router.post("/{task_id}/run")
def run_convert(
task_id: str,
current_user: dict = Depends(get_current_user),
) -> dict[str, Any]:
task = _get_task(task_id)
if not task:
raise fail(404, "task not found")
if task["status"] not in ("uploaded", "completed", "failed"):
raise fail(400, "please upload source files first")
# 标记运行中
store = get_platform_store()
with store.connect() as conn:
conn.execute(
"UPDATE data_convert_tasks SET status='running', error_message='', update_time=NOW() WHERE id=%s",
(task_id,),
)
try:
input_dir = _input_dir(task_id)
output_dir = _output_dir(task_id)
output_dir.mkdir(parents=True, exist_ok=True)
output_path = _task_output_path(task)
input_count = 0
output_count = 0
for json_file in sorted(input_dir.iterdir()):
if not json_file.is_file() or not json_file.name.lower().endswith(".json"):
continue
input_count += 1
with open(json_file, "r", encoding="utf-8") as f:
data = json.load(f)
if isinstance(data, list):
records = data
elif isinstance(data, dict):
records = [data]
else:
raise ValueError(f"JSON must be object or array: {json_file.name}")
with open(output_path, "a", encoding="utf-8") as f:
for record in records:
f.write(json.dumps(record, ensure_ascii=False) + "\n")
output_count += 1
# 更新任务状态
with store.connect() as conn:
conn.execute(
"UPDATE data_convert_tasks SET status='completed', "
"input_count=%s, output_count=%s, update_time=NOW() WHERE id=%s",
(input_count, output_count, task_id),
)
except Exception as exc:
with store.connect() as conn:
conn.execute(
"UPDATE data_convert_tasks SET status='failed', error_message=%s, update_time=NOW() WHERE id=%s",
(str(exc)[:500], task_id),
)
raise fail(500, f"convert failed: {exc}")
return ok(_get_task(task_id))
@router.get("/{task_id}/download")
def download_result(
task_id: str,
current_user: dict = Depends(get_current_user),
):
task = _get_task(task_id)
if not task:
raise fail(404, "task not found")
if task["status"] != "completed":
raise fail(400, "task is not completed")
output_path = _task_output_path(task)
if not output_path.exists():
raise fail(404, "output file not found")
return FileResponse(
str(output_path),
media_type="application/octet-stream",
filename=_safe_output_filename(task.get("output_filename")),
)
@router.post("/{task_id}/import-as-dataset")
def import_as_dataset(
task_id: str,
payload: dict[str, Any] = Body(default={}),
current_user: dict = Depends(get_current_user),
) -> dict[str, Any]:
"""把已转换的 JSONL 文件导入为数据集管理中的上传任务记录source='task')。"""
task = _get_task(task_id)
if not task:
raise fail(404, "task not found")
if task["status"] != "completed":
raise fail(400, "task is not completed")
output_path = _task_output_path(task)
if not output_path.exists():
raise fail(404, "output file not found")
content = output_path.read_text(encoding="utf-8")
dataset_name = str(payload.get("name") or task["name"]).strip()
description = str(payload.get("description") or f"由数据类型转换任务 {task_id} 导入").strip()
size_bytes = len(content.encode("utf-8"))
store = get_platform_store()
# 用 store 提供的接口创建数据集与文件
dataset = store.create_dataset({
"name": dataset_name,
"type": "train",
"storage_type": "local",
"source": "upload",
"task_id": task_id,
"size": f"{size_bytes} B",
"count": task["output_count"],
"description": description,
})
dataset_id = dataset["id"]
with store.connect() as conn:
store.add_dataset_file(conn, dataset_id, _safe_output_filename(task.get("output_filename")), content)
return ok({"dataset_id": dataset_id, "name": dataset_name})
@router.delete("/{task_id}")
def delete_task(
task_id: str,
current_user: dict = Depends(get_current_user),
) -> dict[str, Any]:
task = _get_task(task_id)
if not task:
raise fail(404, "task not found")
store = get_platform_store()
with store.connect() as conn:
conn.execute(
"UPDATE data_convert_tasks SET deleted_at=NOW() WHERE id=%s",
(task_id,),
)
# 清理文件
import shutil
task_dir = _task_dir(task_id)
if task_dir.exists():
shutil.rmtree(task_dir, ignore_errors=True)
return ok({"deleted": task_id})
def _get_task(task_id: str) -> dict[str, Any] | None:
store = get_platform_store()
with store.connect() as conn:
row = conn.execute(
"SELECT * FROM data_convert_tasks WHERE id=%s AND deleted_at IS NULL",
(task_id,),
).fetchone()
return dict(row) if row else None

View File

@@ -26,6 +26,16 @@ def _load_sample(path: str | None, content: str | None = None, max_samples: int
if not text: if not text:
return [] return []
# 先按整文件 JSON数组/单对象)解析,兼容 .json失败再按 jsonl 逐行解析
try:
value = json.loads(text)
except json.JSONDecodeError:
value = None
if isinstance(value, list):
return [item for item in value[:max_samples] if isinstance(item, dict)]
if isinstance(value, dict):
return [value]
lines = text.splitlines()[:max_samples] lines = text.splitlines()[:max_samples]
records: list[dict[str, Any]] = [] records: list[dict[str, Any]] = []
for line in lines: for line in lines:

View File

@@ -57,7 +57,62 @@ def _sentence_chunks(text: str) -> list[str]:
@lru_cache(maxsize=1) @lru_cache(maxsize=1)
def _tokenizer() -> tiktoken.Encoding: def _tokenizer() -> tiktoken.Encoding:
"""加载 cl100k_base 编码器,优先在线下载,失败时使用本地缓存以支持离线环境。"""
import os
import base64
# 先设置缓存目录环境变量
offline_cache = os.path.expanduser("~/.cache/tiktoken")
os.environ.setdefault("TIKTOKEN_CACHE_DIR", offline_cache)
try:
# 尝试标准方式加载
return tiktoken.get_encoding("cl100k_base") return tiktoken.get_encoding("cl100k_base")
except Exception:
# 如果失败,尝试手动从本地文件构造
try:
from pathlib import Path
local_file = Path(offline_cache) / "9b5ad71b2ce5302211f9c61530b329a4922fc6a4"
if not local_file.exists():
# 尝试另一个可能的文件名
local_file = Path(offline_cache) / "cl100k_base.tiktoken"
if local_file.exists():
# 读取 BPE 文件内容
with open(local_file, "rb") as f:
contents = f.read()
# 解析 BPE 文件
mergeable_ranks = {}
for line in contents.splitlines():
if line:
token, rank = line.split()
mergeable_ranks[base64.b64decode(token)] = int(rank)
# 构造 Encoding 对象
import tiktoken.core
return tiktoken.core.Encoding(
name="cl100k_base",
pat_str=r"""'(?i:[sdmt]|ll|ve|re)|[^\r\n\p{L}\p{N}]?+\p{L}+|\p{N}{1,3}| ?[^\s\p{L}\p{N}]++[\r\n]*|\s*[\r\n]|\s+(?!\S)|\s+""",
mergeable_ranks=mergeable_ranks,
special_tokens={
"<|endoftext|>": 100257,
"<|fim_prefix|>": 100258,
"<|fim_middle|>": 100259,
"<|fim_suffix|>": 100260,
"<|endofprompt|>": 100276,
},
)
except Exception:
pass
raise RuntimeError(
f"无法加载 cl100k_base 编码器\n"
f"请确保以下任一条件满足:\n"
f"1. 服务器可以访问网络\n"
f"2. 本地存在缓存文件: {offline_cache}/9b5ad71b2ce5302211f9c61530b329a4922fc6a4"
)
def _text_chunks( def _text_chunks(

Binary file not shown.

View File

@@ -29,7 +29,12 @@ class _TerminalModelGenerationError(ModelGenerationError):
OUTPUT_TYPE_STANDARD = "standard" OUTPUT_TYPE_STANDARD = "standard"
OUTPUT_TYPE_REASONING = "reasoning" OUTPUT_TYPE_REASONING = "reasoning"
SUPPORTED_OUTPUT_TYPES = {OUTPUT_TYPE_STANDARD, OUTPUT_TYPE_REASONING} OUTPUT_TYPE_DPO = "dpo"
SUPPORTED_OUTPUT_TYPES = {
OUTPUT_TYPE_STANDARD,
OUTPUT_TYPE_REASONING,
OUTPUT_TYPE_DPO,
}
REASONING_DETAIL_NORMAL = "normal" REASONING_DETAIL_NORMAL = "normal"
REASONING_DETAIL_DETAILED = "detailed" REASONING_DETAIL_DETAILED = "detailed"
SUPPORTED_REASONING_DETAILS = { SUPPORTED_REASONING_DETAILS = {
@@ -285,6 +290,18 @@ def _prompt_messages(
"这是思维链输出模式,即使其他提示语要求省略分析,也不得省略 reasoning。" "这是思维链输出模式,即使其他提示语要求省略分析,也不得省略 reasoning。"
"不要自行添加 <think> 标签,系统会在保存时统一组装。" "不要自行添加 <think> 标签,系统会在保存时统一组装。"
) )
elif output_type == OUTPUT_TYPE_DPO:
schema = (
'{"items":[{"instruction":"...","input":"...",'
'"chosen":"...","rejected":"..."}]}'
)
output_rule = (
"你正在生成用于直接偏好优化DPO的成对偏好数据。"
"instruction、chosen 和 rejected 均不得为空chosen 必须是忠于来源、"
"准确完整的优选回答rejected 必须是表面合理但存在明确质量缺陷的拒选回答。"
"两者不得相同rejected 不得包含违法危险内容,也不得用空白、乱码或无关文本凑数。"
"不要输出分析过程或 <think> 标签。"
)
else: else:
schema = '{"items":[{"instruction":"...","input":"...","output":"..."}]}' schema = '{"items":[{"instruction":"...","input":"...","output":"..."}]}'
output_rule = ( output_rule = (
@@ -461,9 +478,13 @@ def generate_model_records(
"instruction": failure_instruction, "instruction": failure_instruction,
"input": content, "input": content,
"output": "", "output": "",
"chosen": "",
"rejected": "",
"original_instruction": failure_instruction, "original_instruction": failure_instruction,
"original_input": content, "original_input": content,
"original_output": "", "original_output": "",
"original_chosen": "",
"original_rejected": "",
"status": "invalid", "status": "invalid",
"error": error_message, "error": error_message,
"split": "train", "split": "train",
@@ -479,6 +500,8 @@ def generate_model_records(
input_text = normalize_text( input_text = normalize_text(
str(value.get("input") or value.get("context") or "") str(value.get("input") or value.get("context") or "")
) )
chosen = ""
rejected = ""
if output_type == OUTPUT_TYPE_REASONING: if output_type == OUTPUT_TYPE_REASONING:
reasoning = normalize_text( reasoning = normalize_text(
re.sub( re.sub(
@@ -508,6 +531,36 @@ def generate_model_records(
) )
valid = bool(instruction and reasoning and answer) valid = bool(instruction and reasoning and answer)
missing_error = "model result is missing instruction, reasoning or answer" missing_error = "model result is missing instruction, reasoning or answer"
elif output_type == OUTPUT_TYPE_DPO:
chosen = normalize_text(str(value.get("chosen") or ""))
rejected = normalize_text(str(value.get("rejected") or ""))
chosen = normalize_text(
re.sub(
r"<think>[\s\S]*?(?:</think>|$)",
"",
chosen,
flags=re.IGNORECASE,
)
)
rejected = normalize_text(
re.sub(
r"<think>[\s\S]*?(?:</think>|$)",
"",
rejected,
flags=re.IGNORECASE,
)
)
output = chosen
valid = bool(
instruction
and chosen
and rejected
and chosen.strip() != rejected.strip()
)
missing_error = (
"model result is missing instruction, chosen or rejected, "
"or chosen equals rejected"
)
else: else:
output = normalize_text( output = normalize_text(
str( str(
@@ -527,7 +580,10 @@ def generate_model_records(
) )
valid = bool(instruction and output) valid = bool(instruction and output)
missing_error = "model result is missing instruction or output" missing_error = "model result is missing instruction or output"
raw_id = f"{preview_id}:{variant_index + 1}:{instruction}:{output}" raw_id = (
f"{preview_id}:{variant_index + 1}:{instruction}:"
f"{output}:{rejected}"
)
result_id = f"result_{hashlib.sha256(raw_id.encode()).hexdigest()[:16]}" result_id = f"result_{hashlib.sha256(raw_id.encode()).hexdigest()[:16]}"
results.append( results.append(
{ {
@@ -536,9 +592,13 @@ def generate_model_records(
"instruction": instruction, "instruction": instruction,
"input": input_text, "input": input_text,
"output": output, "output": output,
"chosen": chosen,
"rejected": rejected,
"original_instruction": instruction, "original_instruction": instruction,
"original_input": input_text, "original_input": input_text,
"original_output": output, "original_output": output,
"original_chosen": chosen,
"original_rejected": rejected,
"status": "valid" if valid else "invalid", "status": "valid" if valid else "invalid",
"error": (None if valid else missing_error), "error": (None if valid else missing_error),
"split": "train", "split": "train",

View File

@@ -531,6 +531,10 @@ class LocalDataProcessStorage:
@staticmethod @staticmethod
def _fsync_directory(directory: Path) -> None: def _fsync_directory(directory: Path) -> None:
# Windows 不支持以 O_RDONLY 打开目录做 fsync跳过即可。
# 数据完整性在 Linux 生产环境保障Windows 开发环境忽略。
if os.name == "nt":
return
descriptor = os.open(directory, os.O_RDONLY) descriptor = os.open(directory, os.O_RDONLY)
try: try:
os.fsync(descriptor) os.fsync(descriptor)

View File

@@ -140,6 +140,12 @@ def _reasoning_output_is_valid(value: Any) -> bool:
) )
def _dpo_fields_are_valid(row: dict[str, Any]) -> bool:
chosen = str(row.get("chosen") or "").strip()
rejected = str(row.get("rejected") or "").strip()
return bool(chosen and rejected and chosen != rejected)
def _preview_config_value(config: dict[str, Any], key: str, default: Any) -> Any: def _preview_config_value(config: dict[str, Any], key: str, default: Any) -> Any:
if key in config: if key in config:
return config[key] return config[key]
@@ -846,7 +852,11 @@ class DataProcessStore:
) )
instruction = str(record.get("instruction") or raw.get("instruction") or "") instruction = str(record.get("instruction") or raw.get("instruction") or "")
input_text = str(record.get("input") or raw.get("input") or "") input_text = str(record.get("input") or raw.get("input") or "")
output = str(record.get("output") or raw.get("output") or "") chosen = str(raw.get("chosen") or "")
rejected = str(raw.get("rejected") or "")
output = str(
record.get("output") or raw.get("output") or chosen or ""
)
split = str(record.get("split") or raw.get("split") or "") or None split = str(record.get("split") or raw.get("split") or "") or None
status = str(record.get("status") or "valid") status = str(record.get("status") or "valid")
if status not in {"valid", "modified", "invalid"}: if status not in {"valid", "modified", "invalid"}:
@@ -856,10 +866,11 @@ class DataProcessStore:
""" """
INSERT INTO data_process_results INSERT INTO data_process_results
(id, task_id, preview_item_id, instruction, input, output, (id, task_id, preview_item_id, instruction, input, output,
original_instruction, original_input, original_output, status, chosen, rejected, original_instruction, original_input,
original_output, original_chosen, original_rejected, status,
error, split, quality_score, created_at, updated_at) error, split, quality_score, created_at, updated_at)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s,
NULL, %s, '{}', %s, %s) %s, %s, NULL, %s, '{}', %s, %s)
""", """,
( (
result_id, result_id,
@@ -868,9 +879,13 @@ class DataProcessStore:
instruction, instruction,
input_text, input_text,
output, output,
chosen,
rejected,
instruction, instruction,
input_text, input_text,
output, output,
chosen,
rejected,
status, status,
split, split,
created_at, created_at,
@@ -2024,9 +2039,11 @@ class DataProcessStore:
""" """
INSERT INTO data_process_results INSERT INTO data_process_results
(id, task_id, preview_item_id, instruction, input, output, (id, task_id, preview_item_id, instruction, input, output,
original_instruction, original_input, original_output, status, error, chosen, rejected, original_instruction, original_input,
original_output, original_chosen, original_rejected, status, error,
split, quality_score, created_at, updated_at) split, quality_score, created_at, updated_at)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s,
%s, %s, %s, %s, %s, %s, %s)
""", """,
( (
result.get("id") or new_id("dpr"), result.get("id") or new_id("dpr"),
@@ -2035,9 +2052,13 @@ class DataProcessStore:
result.get("instruction") or "", result.get("instruction") or "",
result.get("input") or "", result.get("input") or "",
result.get("output") or "", result.get("output") or "",
result.get("chosen") or "",
result.get("rejected") or "",
result.get("original_instruction", result.get("instruction") or ""), result.get("original_instruction", result.get("instruction") or ""),
result.get("original_input", result.get("input") or ""), result.get("original_input", result.get("input") or ""),
result.get("original_output", result.get("output") or ""), result.get("original_output", result.get("output") or ""),
result.get("original_chosen", result.get("chosen") or ""),
result.get("original_rejected", result.get("rejected") or ""),
result.get("status") or "valid", result.get("status") or "valid",
result.get("error"), result.get("error"),
result.get("split"), result.get("split"),
@@ -2121,7 +2142,7 @@ class DataProcessStore:
rows = conn.execute( rows = conn.execute(
""" """
SELECT status, instruction, output SELECT status, instruction, output, chosen, rejected
FROM data_process_results FROM data_process_results
WHERE task_id=%s WHERE task_id=%s
""", """,
@@ -2139,6 +2160,10 @@ class DataProcessStore:
_task_output_type(task) == "reasoning" _task_output_type(task) == "reasoning"
and not _reasoning_output_is_valid(row.get("output")) and not _reasoning_output_is_valid(row.get("output"))
) )
or (
_task_output_type(task) == "dpo"
and not _dpo_fields_are_valid(row)
)
) )
if invalid_count: if invalid_count:
raise InvalidStateError( raise InvalidStateError(
@@ -2210,7 +2235,9 @@ class DataProcessStore:
def update_result( def update_result(
self, task_id: str, result_id: str, payload: dict[str, Any] self, task_id: str, result_id: str, payload: dict[str, Any]
) -> dict[str, Any]: ) -> dict[str, Any]:
allowed = {"instruction", "input", "output", "quality_score"} allowed = {
"instruction", "input", "output", "chosen", "rejected", "quality_score"
}
values = {key: value for key, value in payload.items() if key in allowed} values = {key: value for key, value in payload.items() if key in allowed}
if "quality_score" in values: if "quality_score" in values:
values["quality_score"] = json_dumps(values["quality_score"]) values["quality_score"] = json_dumps(values["quality_score"])
@@ -2232,20 +2259,28 @@ class DataProcessStore:
current_updated_at = _serialize_value(current.get("updated_at")) current_updated_at = _serialize_value(current.get("updated_at"))
if expected_updated_at and expected_updated_at != current_updated_at: if expected_updated_at and expected_updated_at != current_updated_at:
raise ConflictError("data process result was modified by another request") raise ConflictError("data process result was modified by another request")
output_type = _task_output_type(task)
if output_type == "dpo" and "chosen" in values:
values["output"] = values["chosen"]
merged = {**current, **values} merged = {**current, **values}
quality = payload.get("quality_score") or {} quality = payload.get("quality_score") or {}
instruction_valid = bool(str(merged.get("instruction") or "").strip()) instruction_valid = bool(str(merged.get("instruction") or "").strip())
output_valid = bool(str(merged.get("output") or "").strip()) output_valid = bool(str(merged.get("output") or "").strip())
reasoning_valid = ( reasoning_valid = (
_task_output_type(task) != "reasoning" output_type != "reasoning"
or _reasoning_output_is_valid(merged.get("output")) or _reasoning_output_is_valid(merged.get("output"))
) )
hard_valid = instruction_valid and output_valid and reasoning_valid dpo_valid = output_type != "dpo" or _dpo_fields_are_valid(merged)
hard_valid = instruction_valid and output_valid and reasoning_valid and dpo_valid
quality_valid = bool(quality.get("is_valid", hard_valid)) quality_valid = bool(quality.get("is_valid", hard_valid))
changed = any( changed = any(
str(merged.get(field) or "") str(merged.get(field) or "")
!= str(merged.get(f"original_{field}") or "") != str(merged.get(f"original_{field}") or "")
for field in ("instruction", "input", "output") for field in (
("instruction", "input", "chosen", "rejected")
if output_type == "dpo"
else ("instruction", "input", "output")
)
) )
status = "invalid" if not hard_valid or not quality_valid else ( status = "invalid" if not hard_valid or not quality_valid else (
"modified" if changed else "valid" "modified" if changed else "valid"
@@ -2255,6 +2290,8 @@ class DataProcessStore:
format_error = ( format_error = (
"思维链输出必须包含非空的 <think>...</think> 推理过程和最终答案" "思维链输出必须包含非空的 <think>...</think> 推理过程和最终答案"
if instruction_valid and output_valid and not reasoning_valid if instruction_valid and output_valid and not reasoning_valid
else "DPO 输出必须包含不同的非空 Chosen 和 Rejected 回答"
if instruction_valid and not dpo_valid
else "Instruction 和 Output 不能为空" else "Instruction 和 Output 不能为空"
if not instruction_valid or not output_valid if not instruction_valid or not output_valid
else None else None
@@ -2318,18 +2355,23 @@ class DataProcessStore:
instruction = str(replacement.get("instruction") or "").strip() instruction = str(replacement.get("instruction") or "").strip()
input_text = str(replacement.get("input") or "").strip() input_text = str(replacement.get("input") or "").strip()
output = str(replacement.get("output") or "").strip() output = str(replacement.get("output") or "").strip()
chosen = str(replacement.get("chosen") or "").strip()
rejected = str(replacement.get("rejected") or "").strip()
quality_score = replacement.get("quality_score") or {} quality_score = replacement.get("quality_score") or {}
if not instruction or not output or not bool(quality_score.get("is_valid")): if not instruction or not output or not bool(quality_score.get("is_valid")):
raise InvalidStateError("regenerated result did not pass quality validation") raise InvalidStateError("regenerated result did not pass quality validation")
if _task_output_type(task) == "reasoning" and not _reasoning_output_is_valid(output): if _task_output_type(task) == "reasoning" and not _reasoning_output_is_valid(output):
raise InvalidStateError("regenerated reasoning result has an invalid output format") raise InvalidStateError("regenerated reasoning result has an invalid output format")
if _task_output_type(task) == "dpo" and not _dpo_fields_are_valid(replacement):
raise InvalidStateError("regenerated DPO result has invalid preference fields")
now = utcnow() now = utcnow()
row = conn.execute( row = conn.execute(
""" """
UPDATE data_process_results UPDATE data_process_results
SET instruction=%s, input=%s, output=%s, SET instruction=%s, input=%s, output=%s, chosen=%s, rejected=%s,
original_instruction=%s, original_input=%s, original_output=%s, original_instruction=%s, original_input=%s, original_output=%s,
original_chosen=%s, original_rejected=%s,
status='valid', error=NULL, quality_score=%s, updated_at=%s status='valid', error=NULL, quality_score=%s, updated_at=%s
WHERE id=%s AND task_id=%s WHERE id=%s AND task_id=%s
RETURNING * RETURNING *
@@ -2338,9 +2380,13 @@ class DataProcessStore:
instruction, instruction,
input_text, input_text,
output, output,
chosen,
rejected,
instruction, instruction,
input_text, input_text,
output, output,
chosen,
rejected,
json_dumps(quality_score), json_dumps(quality_score),
now, now,
result_id, result_id,
@@ -2434,6 +2480,10 @@ class DataProcessStore:
_task_output_type(task) == "reasoning" _task_output_type(task) == "reasoning"
and not _reasoning_output_is_valid(row.get("output")) and not _reasoning_output_is_valid(row.get("output"))
) )
or (
_task_output_type(task) == "dpo"
and not _dpo_fields_are_valid(row)
)
) )
if invalid_count: if invalid_count:
raise InvalidStateError(f"task contains {invalid_count} invalid results") raise InvalidStateError(f"task contains {invalid_count} invalid results")
@@ -2449,6 +2499,18 @@ class DataProcessStore:
requested_split, requested_split,
seed=task_id, seed=task_id,
) )
if _task_output_type(task) == "dpo":
records = [
{
"instruction": row["instruction"],
"input": row["input"],
"chosen": row["chosen"],
"rejected": row["rejected"],
"split": assignment,
}
for row, assignment in zip(rows, assignments, strict=True)
]
else:
records = [ records = [
{ {
"instruction": row["instruction"], "instruction": row["instruction"],
@@ -2498,7 +2560,11 @@ class DataProcessStore:
"reasoning_detail": _task_reasoning_detail(task), "reasoning_detail": _task_reasoning_detail(task),
"source_file_ids": [item["id"] for item in self._source_ids(conn, task_id)], "source_file_ids": [item["id"] for item in self._source_ids(conn, task_id)],
"source_result_ids": source_result_ids, "source_result_ids": source_result_ids,
"format": payload.get("format") or "alpaca_jsonl", "format": (
"dpo"
if _task_output_type(task) == "dpo"
else payload.get("format") or "alpaca_jsonl"
),
"split": requested_split, "split": requested_split,
} }
@@ -2741,7 +2807,7 @@ class DataProcessStore:
record["split"], record["split"],
record["instruction"], record["instruction"],
record["input"], record["input"],
record["output"], record.get("output") or record.get("chosen") or "",
json_dumps( json_dumps(
{ {
**record, **record,

View File

@@ -0,0 +1 @@
"""GPU assignment management module."""

View File

@@ -0,0 +1,77 @@
"""GPU 算力分配管理路由。"""
from __future__ import annotations
from typing import Any
from fastapi import APIRouter, Body, Depends, Request
from app.api.v1.endpoints.platform import ok, fail
from app.core.auth import get_current_user, is_admin
from app.db.platform_store import get_platform_store
router = APIRouter(prefix="/compute", tags=["gpu-assignment"])
def _actor_id(request: Request) -> str | None:
auth = request.headers.get("Authorization", "")
token = auth.replace("Bearer ", "").strip()
if token.startswith("platform-token-"):
return token[len("platform-token-"):]
return None
@router.get("/gpu-assignments")
def list_assignments(current_user: dict = Depends(get_current_user)) -> dict[str, Any]:
"""查看全部分配关系(仅 admin"""
if not is_admin(current_user):
raise fail(403, "admin permission required")
return ok(get_platform_store().gpu_assignments())
@router.post("/gpu-assignments")
def assign_gpus(
payload: dict[str, Any] = Body(...),
request: Request = None,
current_user: dict = Depends(get_current_user),
) -> dict[str, Any]:
"""批量分配 GPU仅 admin。body: { assignments: [{ node_id, gpu_index, user_id }] }"""
if not is_admin(current_user):
raise fail(403, "admin permission required")
assignments = payload.get("assignments") or []
if not assignments:
raise fail(400, "assignments 不能为空")
actor = _actor_id(request) if request else None
result = get_platform_store().assign_gpus(assignments, assigned_by=actor)
get_platform_store().record_audit(
action="gpu.assign",
actor_id=actor,
target_type="gpu",
detail=f"count={len(assignments)}",
)
return ok(result)
@router.delete("/gpu-assignments/{assignment_id}")
def unassign_gpu(
assignment_id: str,
request: Request = None,
current_user: dict = Depends(get_current_user),
) -> dict[str, Any]:
"""撤销 GPU 分配(仅 admin"""
if not is_admin(current_user):
raise fail(403, "admin permission required")
get_platform_store().unassign_gpu(assignment_id)
actor = _actor_id(request) if request else None
get_platform_store().record_audit(
action="gpu.unassign",
actor_id=actor,
target_type="gpu",
target_id=assignment_id,
)
return ok({"deleted": assignment_id})
@router.get("/my-gpus")
def my_gpus(current_user: dict = Depends(get_current_user)) -> dict[str, Any]:
"""查看当前用户可用的 GPU 列表。"""
return ok(get_platform_store().gpu_assignments_for_user(current_user["id"]))

View File

@@ -2,6 +2,7 @@ from __future__ import annotations
from enum import StrEnum from enum import StrEnum
from typing import Any, Literal from typing import Any, Literal
from urllib.parse import parse_qs, urlsplit
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
@@ -15,6 +16,31 @@ def _config_value(config: dict[str, Any], snake_name: str, camel_name: str, defa
def _validate_process_config(config: dict[str, Any]) -> None: def _validate_process_config(config: dict[str, Any]) -> None:
output_type = _config_value(config, "output_type", "outputType", "standard")
if output_type not in {"standard", "reasoning", "dpo"}:
raise ValueError("output_type must be one of: standard, reasoning, dpo")
source_mode = _config_value(config, "source_mode", "sourceMode", "local")
if source_mode not in {"local", "external"}:
raise ValueError("source_mode must be one of: local, external")
external_source = _config_value(config, "external_source", "externalSource", None)
if external_source is not None:
if not isinstance(external_source, dict):
raise ValueError("external_source must be an object")
if any(
key.lower() in {"password", "secret", "token", "api_key"}
for key in external_source
):
raise ValueError("external_source must not persist credentials")
external_url = str(external_source.get("url") or "").strip()
if external_url:
parsed_external_url = urlsplit(external_url)
sensitive_query_keys = {"password", "secret", "token", "api_key", "user", "username"}
if parsed_external_url.username or parsed_external_url.password or (
set(parse_qs(parsed_external_url.query)) & sensitive_query_keys
):
raise ValueError("external_source URL must not contain credentials")
chunk_method = _config_value(config, "chunk_method", "chunkMethod", "layout_hybrid") chunk_method = _config_value(config, "chunk_method", "chunkMethod", "layout_hybrid")
if not isinstance(chunk_method, str) or chunk_method not in { if not isinstance(chunk_method, str) or chunk_method not in {
"layout_hybrid", "layout_hybrid",
@@ -303,6 +329,9 @@ class ExternalSourceRequest(BaseModel):
username: str | None = Field(default=None, max_length=150) username: str | None = Field(default=None, max_length=150)
password: str | None = Field(default=None, max_length=500) password: str | None = Field(default=None, max_length=500)
limit: int = Field(default=1000, ge=1, le=100_000) limit: int = Field(default=1000, ge=1, le=100_000)
connect_timeout_seconds: int = Field(default=5, ge=1, le=30)
statement_timeout_seconds: int = Field(default=30, ge=1, le=300)
ssl_mode: Literal["disable", "prefer", "require", "verify-ca", "verify-full"] = "prefer"
class ExternalPullRequest(ExternalSourceRequest): class ExternalPullRequest(ExternalSourceRequest):
@@ -324,6 +353,8 @@ class ResultUpdate(BaseModel):
instruction: str | None = None instruction: str | None = None
input: str | None = None input: str | None = None
output: str | None = None output: str | None = None
chosen: str | None = None
rejected: str | None = None
expected_updated_at: str | None = None expected_updated_at: str | None = None
@@ -374,7 +405,7 @@ class PublishRequest(BaseModel):
dataset_type: Literal["train", "test", "eval", "val", "other"] = "train" dataset_type: Literal["train", "test", "eval", "val", "other"] = "train"
storage_type: Literal["local"] = "local" storage_type: Literal["local"] = "local"
split: DatasetSplit = Field(default_factory=DatasetSplit) split: DatasetSplit = Field(default_factory=DatasetSplit)
format: Literal["alpaca_jsonl", "jsonl"] = "alpaca_jsonl" format: Literal["alpaca_jsonl", "jsonl", "dpo"] = "alpaca_jsonl"
description: str = "" description: str = ""
@field_validator("dataset_name") @field_validator("dataset_name")

View File

@@ -0,0 +1,57 @@
"""data_convert 模块安全回归测试:输出文件名路径穿越与鉴权。
- ``output_filename`` 必须通过白名单校验,阻断 ``../``、``/``、``\\`` 及控制字符,
否则转换结果可被写出到存储根目录之外(任意文件读写/删除)。
- 所有 data_convert 路由必须挂载 ``get_current_user`` 鉴权依赖。
"""
from __future__ import annotations
import pytest
from fastapi import HTTPException
from app.core.auth import get_current_user
from app.modules.data_convert.router import _safe_output_filename, router
def test_safe_output_filename_defaults() -> None:
assert _safe_output_filename(None) == "converted-data.jsonl"
assert _safe_output_filename("") == "converted-data.jsonl"
def test_safe_output_filename_valid() -> None:
assert _safe_output_filename("converted-data.jsonl") == "converted-data.jsonl"
assert _safe_output_filename("my-data.v1.jsonl") == "my-data.v1.jsonl"
assert _safe_output_filename(" 报告.jsonl ") == "报告.jsonl"
@pytest.mark.parametrize(
"bad",
[
"../../etc/passwd",
"../x.jsonl",
"a/b.jsonl",
r"a\b.jsonl",
"a\\b.jsonl",
"..",
".",
"x\x00.jsonl",
"x\n.jsonl",
"x\t.jsonl",
],
)
def test_safe_output_filename_rejects_traversal(bad: str) -> None:
with pytest.raises(HTTPException):
_safe_output_filename(bad)
def test_all_data_convert_routes_require_auth() -> None:
for route in router.routes:
node = getattr(route, "dependant", None)
assert node is not None, f"route {route.path} has no dependency graph"
stack = list(node.dependencies)
calls: list = []
while stack:
dep = stack.pop()
stack.extend(getattr(dep, "dependencies", []))
calls.append(getattr(dep, "call", None))
assert get_current_user in calls, f"route {route.path} is missing get_current_user auth"

View File

@@ -1840,6 +1840,43 @@ def test_external_source_never_returns_fake_success(tmp_path: Path) -> None:
assert response.json()["detail"]["code"] == 501 assert response.json()["detail"]["code"] == 501
def test_external_source_mode_belongs_to_step_three_structured_task(tmp_path: Path) -> None:
client, _, _ = make_client(tmp_path)
local_task_id = client.post(
"/modelTF/data-process",
json={
"name": "本地结构化任务",
"process_type": "structured",
"config": {"source_mode": "local"},
},
).json()["data"]["id"]
rejected = client.post(
f"/modelTF/data-process/{local_task_id}/external/test",
json={"type": "mysql", "url": "mysql://db.example/test"},
)
assert rejected.status_code == 409
external_task_id = client.post(
"/modelTF/data-process",
json={
"name": "外部结构化任务",
"process_type": "structured",
"config": {"source_mode": "external"},
},
).json()["data"]["id"]
accepted_as_external = client.post(
f"/modelTF/data-process/{external_task_id}/external/test",
json={"type": "mysql", "url": "mysql://db.example/test"},
)
assert accepted_as_external.status_code == 501
local_upload = client.post(
f"/modelTF/data-process/{external_task_id}/source-files",
files={"files": ("records.jsonl", b'{"id":1}\n', "application/jsonl")},
)
assert local_upload.status_code == 409
def test_regenerate_endpoint_prepares_an_existing_published_task(tmp_path: Path) -> None: def test_regenerate_endpoint_prepares_an_existing_published_task(tmp_path: Path) -> None:
client, store, _ = make_client(tmp_path) client, store, _ = make_client(tmp_path)
task_id = client.post( task_id = client.post(

View File

@@ -88,6 +88,74 @@ def test_generate_model_records_uses_prompt_auth_and_stable_split() -> None:
assert progress_updates == [(1, 1)] assert progress_updates == [(1, 1)]
def test_generate_model_records_builds_native_dpo_pair() -> None:
def handler(request: httpx.Request) -> httpx.Response:
payload = json.loads(request.content)
system_prompt = payload["messages"][0]["content"]
assert '"chosen"' in system_prompt
assert '"rejected"' in system_prompt
assert "直接偏好优化" in system_prompt
return httpx.Response(
200,
json={
"choices": [{
"message": {
"content": json.dumps({
"items": [{
"instruction": "系统如何处理扫描 PDF",
"input": "",
"chosen": "仅在没有文本层时调用 OCR并保留页码。",
"rejected": "所有 PDF 都重复执行 OCR。",
}],
}, ensure_ascii=False),
},
}],
},
)
records = generate_model_records(
[{"id": "preview-dpo", "edited_content": "扫描 PDF 缺少文本层时执行 OCR。"}],
model={"name": "model", "api_url": "https://model.example/v1"},
config={"output_type": "dpo", "generation_retries": 0},
task_id="task-dpo",
split={"train": 100, "validation": 0, "test": 0},
qa_pairs_per_item=1,
client=httpx.Client(transport=httpx.MockTransport(handler)),
)
assert records[0]["status"] == "valid"
assert records[0]["chosen"] == "仅在没有文本层时调用 OCR,并保留页码。"
assert records[0]["rejected"] == "所有 PDF 都重复执行 OCR。"
assert records[0]["output"] == records[0]["chosen"]
def test_generate_model_records_rejects_equal_dpo_pair() -> None:
def handler(_request: httpx.Request) -> httpx.Response:
return httpx.Response(
200,
json={"choices": [{"message": {"content": json.dumps({
"items": [{
"instruction": "问题",
"chosen": "相同回答",
"rejected": "相同回答",
}],
}, ensure_ascii=False)}}]},
)
records = generate_model_records(
[{"id": "preview-dpo-invalid", "edited_content": "来源"}],
model={"name": "model", "api_url": "https://model.example/v1"},
config={"output_type": "dpo", "generation_retries": 0},
task_id="task-dpo-invalid",
split={"train": 100, "validation": 0, "test": 0},
qa_pairs_per_item=1,
client=httpx.Client(transport=httpx.MockTransport(handler)),
)
assert records[0]["status"] == "invalid"
assert "chosen equals rejected" in records[0]["error"]
def test_minimax_m3_uses_split_reasoning_and_completion_token_budget() -> None: def test_minimax_m3_uses_split_reasoning_and_completion_token_budget() -> None:
requests: list[dict[str, object]] = [] requests: list[dict[str, object]] = []

View File

@@ -37,6 +37,10 @@ def test_runtime_migration_fails_fast_on_incompatible_schema() -> None:
for value in ("idle", "queued", "running", "completed", "failed", "cancelled"): for value in ("idle", "queued", "running", "completed", "failed", "cancelled"):
assert f"'{value}'" in sql assert f"'{value}'" in sql
assert "CREATE TABLE IF NOT EXISTS data_process_results" in sql assert "CREATE TABLE IF NOT EXISTS data_process_results" in sql
assert "ADD COLUMN IF NOT EXISTS chosen TEXT NOT NULL DEFAULT ''" in sql
assert "ADD COLUMN IF NOT EXISTS rejected TEXT NOT NULL DEFAULT ''" in sql
assert "ADD COLUMN IF NOT EXISTS original_chosen TEXT" in sql
assert "ADD COLUMN IF NOT EXISTS original_rejected TEXT" in sql
assert sql.count("BEGIN;") == 1 assert sql.count("BEGIN;") == 1
assert sql.rstrip().endswith("COMMIT;") assert sql.rstrip().endswith("COMMIT;")

View File

@@ -1335,6 +1335,39 @@ def test_publish_rejects_invalid_reasoning_output_format() -> None:
) )
def test_publish_dpo_writes_chosen_and_rejected_jsonl() -> None:
conn = _PublishConnection(
[
{
"id": "result-dpo",
"status": "valid",
"instruction": "如何处理扫描 PDF",
"input": "",
"output": "仅在无文本层时执行 OCR。",
"chosen": "仅在无文本层时执行 OCR。",
"rejected": "所有 PDF 都执行 OCR。",
"preview_item_id": "preview-dpo",
}
]
)
published = _PublishStore(conn, {"output_type": "dpo"}).publish(
"task-dpo",
{
"dataset_name": "偏好数据",
"storage_type": "local",
"format": "dpo",
"split": {"train": 100, "validation": 0, "test": 0},
},
)
record = conn.records[0]["raw"]
assert record["chosen"] == "仅在无文本层时执行 OCR。"
assert record["rejected"] == "所有 PDF 都执行 OCR。"
assert "output" not in record
assert published["datasets"][0]["metadata"]["format"] == "dpo"
def test_source_storage_descriptor_accepts_owned_local_and_legacy_db_references() -> None: def test_source_storage_descriptor_accepts_owned_local_and_legacy_db_references() -> None:
task_id = "dpt_task" task_id = "dpt_task"
source_file_id = "dpsf_source" source_file_id = "dpsf_source"

View File

@@ -0,0 +1,76 @@
"""Swagger / ReDoc / OpenAPI 文档路由安全开关测试。
生产环境APP_ENV=prod默认关闭 /docs、/redoc、/openapi.json
避免未授权访问泄露 API 结构;本地开发环境默认开放,可用 ENABLE_DOCS 覆盖。
"""
from __future__ import annotations
import pytest
from app.core.config import docs_kwargs, get_settings
@pytest.fixture(autouse=True)
def _reset_settings_cache():
"""每次测试前后清空 get_settings 的 lru_cache避免环境变量互相污染。"""
get_settings.cache_clear()
yield
get_settings.cache_clear()
def test_docs_kwargs_enabled() -> None:
assert docs_kwargs(True) == {}
def test_docs_kwargs_disabled() -> None:
assert docs_kwargs(False) == {"docs_url": None, "redoc_url": None, "openapi_url": None}
def test_docs_disabled_by_default_in_prod(monkeypatch) -> None:
monkeypatch.delenv("ENABLE_DOCS", raising=False)
monkeypatch.setenv("APP_ENV", "prod")
assert get_settings().enable_docs is False
def test_docs_enabled_by_default_outside_prod(monkeypatch) -> None:
monkeypatch.delenv("ENABLE_DOCS", raising=False)
monkeypatch.setenv("APP_ENV", "local")
assert get_settings().enable_docs is True
def test_docs_env_override_enables_in_prod(monkeypatch) -> None:
monkeypatch.setenv("ENABLE_DOCS", "true")
monkeypatch.setenv("APP_ENV", "prod")
assert get_settings().enable_docs is True
def test_docs_env_override_disables_outside_prod(monkeypatch) -> None:
monkeypatch.setenv("ENABLE_DOCS", "false")
monkeypatch.setenv("APP_ENV", "local")
assert get_settings().enable_docs is False
def test_create_app_disables_docs_in_prod(monkeypatch, tmp_path) -> None:
pytest.importorskip("fastapi")
from app.main import create_app
monkeypatch.delenv("ENABLE_DOCS", raising=False)
monkeypatch.setenv("APP_ENV", "prod")
monkeypatch.setenv("LOG_DIR", str(tmp_path))
app = create_app()
assert app.docs_url is None
assert app.redoc_url is None
assert app.openapi_url is None
def test_create_app_enables_docs_outside_prod(monkeypatch, tmp_path) -> None:
pytest.importorskip("fastapi")
from app.main import create_app
monkeypatch.delenv("ENABLE_DOCS", raising=False)
monkeypatch.setenv("APP_ENV", "local")
monkeypatch.setenv("LOG_DIR", str(tmp_path))
app = create_app()
assert app.docs_url == "/docs"
assert app.redoc_url == "/redoc"
assert app.openapi_url == "/openapi.json"

View File

@@ -138,6 +138,14 @@ class FakePlatformStore:
return dict(u) return dict(u)
return None return None
def create_session(self, user_id: str) -> dict[str, Any]:
import secrets
sid = secrets.token_hex(16)
return {"session_id": sid, "user_id": user_id}
def finish_session(self, session_id: str) -> None:
pass
def users(self) -> list[dict[str, Any]]: def users(self) -> list[dict[str, Any]]:
return [dict(u) for u in self._users] return [dict(u) for u in self._users]
@@ -392,12 +400,21 @@ class FakePlatformStore:
def tasks(self) -> list[dict[str, Any]]: def tasks(self) -> list[dict[str, Any]]:
return self._tasks return self._tasks
def eval_tasks(self) -> list[dict[str, Any]]:
return []
def compute_nodes(self) -> list[dict[str, Any]]: def compute_nodes(self) -> list[dict[str, Any]]:
return self._compute_nodes return self._compute_nodes
def gpus(self) -> list[dict[str, Any]]: def gpus(self) -> list[dict[str, Any]]:
return self._gpus return self._gpus
def compare_tasks(self) -> list[dict[str, Any]]:
return []
def trained_models(self) -> list[dict[str, Any]]:
return []
def system_info(self) -> dict[str, Any]: def system_info(self) -> dict[str, Any]:
return {"cpu": {}, "memory": {}} return {"cpu": {}, "memory": {}}

View File

@@ -15,12 +15,13 @@ from fastapi import FastAPI, File, Form, HTTPException, Query, Request, UploadFi
from fastapi.responses import FileResponse, JSONResponse, StreamingResponse from fastapi.responses import FileResponse, JSONResponse, StreamingResponse
from compute.agent.process_manager import ProcessManager from compute.agent.process_manager import ProcessManager
from compute.api.security import docs_kwargs
from compute.engines.llama_factory.adapter import build_command, parse_log_line, prepare_runtime_files from compute.engines.llama_factory.adapter import build_command, parse_log_line, prepare_runtime_files
from compute.engines.llama_factory.inference import get_inference_session from compute.engines.llama_factory.inference import get_inference_session
def create_app() -> FastAPI: def create_app() -> FastAPI:
app = FastAPI(title="YG Fine-Tune Compute API") app = FastAPI(title="YG Fine-Tune Compute API", **docs_kwargs())
jobs: dict[str, dict[str, Any]] = {} jobs: dict[str, dict[str, Any]] = {}
route_prefix = os.getenv("MODELTF_ROUTE_PREFIX", "/modelTF").rstrip("/") or "/modelTF" route_prefix = os.getenv("MODELTF_ROUTE_PREFIX", "/modelTF").rstrip("/") or "/modelTF"
process_manager = ProcessManager(os.getenv("TRAINING_LOG_ROOT", "/opt/yg-ft/logs/training")) process_manager = ProcessManager(os.getenv("TRAINING_LOG_ROOT", "/opt/yg-ft/logs/training"))
@@ -860,10 +861,17 @@ def create_app() -> FastAPI:
@app.get(f"{route_prefix}/compute/files/{{file_id}}/download") @app.get(f"{route_prefix}/compute/files/{{file_id}}/download")
async def download_file(file_id: str) -> FileResponse: async def download_file(file_id: str) -> FileResponse:
upload_root = Path(os.getenv("YG_FT_DATA_ROOT", "/data/yg-ft")) / "uploads" upload_root = Path(os.getenv("YG_FT_DATA_ROOT", "/data/yg-ft")) / "uploads"
# file_id 仅允许普通标识符,拒绝 ../、/、\ 等路径穿越字符。
if not file_id or not all(character.isalnum() or character in {"_", "-"} for character in file_id):
raise HTTPException(status_code=400, detail="invalid file id")
matches = list(upload_root.glob(f"{file_id}_*")) matches = list(upload_root.glob(f"{file_id}_*"))
if not matches: if not matches:
raise HTTPException(status_code=404, detail="file not found") raise HTTPException(status_code=404, detail="file not found")
return FileResponse(matches[0]) # 解析符号链接后仍必须位于 upload 根目录内,防止符号链接指向目录外文件。
resolved = matches[0].resolve()
if not _path_inside(upload_root, resolved):
raise HTTPException(status_code=404, detail="file not found")
return FileResponse(resolved)
return app return app

29
compute/api/security.py Normal file
View File

@@ -0,0 +1,29 @@
"""计算节点 API 安全配置Swagger / ReDoc / OpenAPI 文档路由开关。"""
from __future__ import annotations
import os
from typing import Any
def docs_enabled() -> bool:
"""判断 FastAPI 文档路由(/docs、/redoc、/openapi.json是否开放。
显式配置 ENABLE_DOCS 时以之为准;否则仅在关闭 token 鉴权
COMPUTE_AUTH_ENABLED=false本地开发时开放生产环境默认关闭
避免未授权访问泄露 API 结构。
"""
raw = os.getenv("ENABLE_DOCS", "").strip().lower()
if raw in {"true", "false"}:
return raw == "true"
auth_enabled = os.getenv("COMPUTE_AUTH_ENABLED", "true").lower() == "true"
return not auth_enabled
def docs_kwargs() -> dict[str, Any]:
"""返回传入 FastAPI 的文档路由参数。
关闭时 FastAPI 不注册 /docs、/redoc、/openapi.json访问一律返回 404。
"""
if docs_enabled():
return {}
return {"docs_url": None, "redoc_url": None, "openapi_url": None}

View File

@@ -15,27 +15,58 @@ class LlamaFactoryCommand:
def _load_dataset_preview(path: Path) -> list[dict[str, Any]]: def _load_dataset_preview(path: Path) -> list[dict[str, Any]]:
"""Load a preview of JSON/JSONL records from a dataset file.
Content-sniffs instead of trusting the extension so that BOM-prefixed files,
JSONL files containing a single JSON array, and mislabeled extensions all work.
"""
if not path.exists(): if not path.exists():
return [] return []
text = path.read_text(encoding="utf-8", errors="replace").strip() text = path.read_text(encoding="utf-8-sig", errors="replace").strip()
if not text: if not text:
return [] return []
if path.suffix.lower() == ".jsonl": try:
value = json.loads(text)
except json.JSONDecodeError:
value = None
if isinstance(value, list):
return [item for item in value[:20] if isinstance(item, dict)]
if isinstance(value, dict):
return [value]
items: list[dict[str, Any]] = [] items: list[dict[str, Any]] = []
for line in text.splitlines()[:20]: for line in text.splitlines()[:20]:
line = line.strip() line = line.strip()
if not line: if not line:
continue continue
value = json.loads(line) try:
if isinstance(value, dict): parsed = json.loads(line)
items.append(value) except json.JSONDecodeError:
return items continue
value = json.loads(text) if isinstance(parsed, list):
if isinstance(value, list): items.extend(item for item in parsed[:20] if isinstance(item, dict))
return [item for item in value[:20] if isinstance(item, dict)] elif isinstance(parsed, dict):
if isinstance(value, dict): items.append(parsed)
return [value] if len(items) >= 20:
return [] break
return items[:20]
def _required_columns_for(formatting: str, columns: dict[str, Any]) -> list[str]:
"""Required data columns per dataset format.
Mirrors LLaMA-Factory's leniency: optional columns (e.g. ``input`` / ``query``
in Alpaca) are never required, only fields the format structurally needs.
"""
fmt = str(formatting or "").lower()
if fmt == "sharegpt":
return [str(columns.get("messages") or "messages")]
if fmt in {"dpo", "rm", "kto", "ppo"}:
return [str(columns[key]) for key in ("chosen", "rejected") if columns.get(key)]
if fmt in {"cpt", "pt", "pretrain"}:
return [str(columns.get("prompt") or columns.get("text") or "text")]
# alpaca family: prompt (instruction) + response (output) required,
# query (input) / history are optional and common to omit in jsonl datasets.
return [str(columns[key]) for key in ("prompt", "response") if columns.get(key)]
def _validate_dataset_columns(config: dict[str, Any]) -> list[str]: def _validate_dataset_columns(config: dict[str, Any]) -> list[str]:
@@ -51,7 +82,7 @@ def _validate_dataset_columns(config: dict[str, Any]) -> list[str]:
file_name = item.get("file_name") file_name = item.get("file_name")
file_names = file_name if isinstance(file_name, list) else [file_name] file_names = file_name if isinstance(file_name, list) else [file_name]
columns = item.get("columns") if isinstance(item.get("columns"), dict) else {} columns = item.get("columns") if isinstance(item.get("columns"), dict) else {}
required_columns = [str(value) for value in columns.values() if value] required_columns = _required_columns_for(str(item.get("formatting") or ""), columns)
for name in file_names: for name in file_names:
if not name: if not name:
continue continue

View File

@@ -22,7 +22,10 @@ from typing import Any
def _load_dataset(path: str) -> list[dict[str, Any]]: def _load_dataset(path: str) -> list[dict[str, Any]]:
"""Load a JSON or JSONL dataset file. """Load a JSON or JSONL dataset file (jsonl-compatible).
Content-sniffs instead of trusting the extension so jsonl files with a BOM,
a single JSON array on one line, or mislabeled extensions all load correctly.
Supports common field names used across the platform: Supports common field names used across the platform:
* ``instruction`` + ``input`` + ``output`` (Alpaca-style) * ``instruction`` + ``input`` + ``output`` (Alpaca-style)
@@ -30,14 +33,17 @@ def _load_dataset(path: str) -> list[dict[str, Any]]:
* ``messages`` (ShareGPT-style the last assistant message is treated as reference) * ``messages`` (ShareGPT-style the last assistant message is treated as reference)
""" """
file_path = Path(path) file_path = Path(path)
text = file_path.read_text(encoding="utf-8", errors="replace").strip() text = file_path.read_text(encoding="utf-8-sig", errors="replace").strip()
if not text: if not text:
return [] return []
if file_path.suffix.lower() == ".json": try:
value = json.loads(text) value = json.loads(text)
except json.JSONDecodeError:
value = None
if isinstance(value, list): if isinstance(value, list):
return [item for item in value if isinstance(item, dict)] return [item for item in value if isinstance(item, dict)]
return [value] if isinstance(value, dict) else [] if isinstance(value, dict):
return [value]
samples: list[dict[str, Any]] = [] samples: list[dict[str, Any]] = []
for line in text.splitlines(): for line in text.splitlines():

View File

@@ -0,0 +1,42 @@
from __future__ import annotations
import json
from compute.engines.llama_factory.eval_runner import _load_dataset
def _write(tmp_path, name: str, text: str) -> str:
path = tmp_path / name
path.write_text(text, encoding="utf-8")
return str(path)
def test_load_jsonl_multiline(tmp_path) -> None:
path = _write(
tmp_path,
"eval.jsonl",
'{"question": "q1", "answer": "a1"}\n{"question": "q2", "answer": "a2"}\n',
)
assert _load_dataset(path) == [
{"question": "q1", "answer": "a1"},
{"question": "q2", "answer": "a2"},
]
def test_load_json_array(tmp_path) -> None:
path = _write(
tmp_path,
"eval.json",
json.dumps([{"question": "x", "answer": "y"}]),
)
assert _load_dataset(path) == [{"question": "x", "answer": "y"}]
def test_load_jsonl_with_bom_and_embedded_array(tmp_path) -> None:
"""jsonl 带 BOM 且单行内嵌 JSON 数组,都应正常加载。"""
path = _write(
tmp_path,
"eval.jsonl",
"" + json.dumps([{"question": "a", "answer": "b"}, {"question": "c", "answer": "d"}]),
)
assert len(_load_dataset(path)) == 2

View File

@@ -0,0 +1,63 @@
"""compute ``download_file`` 端点安全回归测试。
修复前 ``file_id`` 直接拼进 glob 模式且不校验路径包含关系,可通过 ``../``
穿越出 upload 目录,并在 Linux 上跟随符号链接读取任意文件。
修复后file_id 仅允许字母/数字/下划线/连字符,返回前对解析后的路径
做 upload 根目录包含性校验。
"""
from __future__ import annotations
from pathlib import Path
import pytest
from fastapi.testclient import TestClient
def _make_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> TestClient:
monkeypatch.setenv("TRAINING_LOG_ROOT", str(tmp_path / "logs"))
monkeypatch.setenv("YG_FT_DATA_ROOT", str(tmp_path / "data"))
monkeypatch.setenv("COMPUTE_EXECUTION_MODE", "simulator")
monkeypatch.setenv("COMPUTE_AUTH_ENABLED", "false")
monkeypatch.delenv("ENABLE_DOCS", raising=False)
from compute.api.main import create_app
return TestClient(create_app())
def _upload_root(tmp_path: Path) -> Path:
root = tmp_path / "data" / "uploads"
root.mkdir(parents=True, exist_ok=True)
return root
def test_download_legit_file(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
client = _make_client(tmp_path, monkeypatch)
(_upload_root(tmp_path) / "file_123456_hello.txt").write_text("HELLO-DOWNLOAD", encoding="utf-8")
response = client.get("/modelTF/compute/files/file_123456/download")
assert response.status_code == 200
assert response.content == b"HELLO-DOWNLOAD"
def test_download_rejects_traversal_file_id(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
client = _make_client(tmp_path, monkeypatch)
outside = tmp_path / "secret" / "passwd_1.txt"
outside.parent.mkdir(parents=True, exist_ok=True)
outside.write_text("TOP-SECRET", encoding="utf-8")
for file_id in ["..", "file.123", "..%2F..%2Fsecret%2Fpasswd", "file%20name"]:
response = client.get(f"/modelTF/compute/files/{file_id}/download")
assert response.status_code in (400, 404), f"file_id={file_id!r} -> {response.status_code}"
assert b"TOP-SECRET" not in response.content
def test_download_blocks_symlink_escape(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
client = _make_client(tmp_path, monkeypatch)
upload_root = _upload_root(tmp_path)
outside = tmp_path / "secret.txt"
outside.write_text("TOP-SECRET", encoding="utf-8")
try:
(upload_root / "file_999999_link.txt").symlink_to(outside)
except OSError:
pytest.skip("symlink creation not permitted on this platform")
response = client.get("/modelTF/compute/files/file_999999/download")
assert response.status_code == 404
assert b"TOP-SECRET" not in response.content

View File

@@ -1,6 +1,8 @@
from __future__ import annotations from __future__ import annotations
from compute.engines.llama_factory.adapter import build_command import json
from compute.engines.llama_factory.adapter import _validate_dataset_columns, build_command
def test_build_command_uses_explicit_validation_dataset_without_resplitting() -> None: def test_build_command_uses_explicit_validation_dataset_without_resplitting() -> None:
@@ -21,3 +23,70 @@ def test_build_command_uses_explicit_validation_dataset_without_resplitting() ->
) )
assert "--do_eval" in result.command assert "--do_eval" in result.command
assert "--val_size" not in result.command assert "--val_size" not in result.command
def _write(tmp_path, name: str, lines: list[dict]) -> object:
path = tmp_path / name
path.write_text(
"".join(json.dumps(line, ensure_ascii=False) + "\n" for line in lines),
encoding="utf-8",
)
return path
def test_jsonl_alpaca_without_input_column_passes_validation(tmp_path) -> None:
"""纯 jsonl Alpaca 数据缺省 input 字段(常见),不应被校验拦截。"""
_write(tmp_path, "train.jsonl", [{"instruction": "hi", "output": "hello"}])
errors = _validate_dataset_columns(
{
"dataset_dir": str(tmp_path),
"dataset_info": {
"ygft_a": {
"file_name": "train.jsonl",
"formatting": "alpaca",
"columns": {"prompt": "instruction", "query": "input", "response": "output"},
}
},
}
)
assert errors == []
def test_jsonl_sharegpt_passes_validation(tmp_path) -> None:
"""ShareGPT 格式 jsonlmessages应通过校验。"""
_write(
tmp_path,
"msg.jsonl",
[{"messages": [{"role": "user", "content": "hi"}, {"role": "assistant", "content": "hello"}]}],
)
errors = _validate_dataset_columns(
{
"dataset_dir": str(tmp_path),
"dataset_info": {
"ygft_m": {
"file_name": "msg.jsonl",
"formatting": "sharegpt",
"columns": {"messages": "messages"},
}
},
}
)
assert errors == []
def test_jsonl_missing_response_still_rejected(tmp_path) -> None:
"""缺 outputresponse仍应报错——没有答案无法做有监督微调。"""
_write(tmp_path, "train.jsonl", [{"instruction": "hi"}])
errors = _validate_dataset_columns(
{
"dataset_dir": str(tmp_path),
"dataset_info": {
"ygft_a": {
"file_name": "train.jsonl",
"formatting": "alpaca",
"columns": {"prompt": "instruction", "query": "input", "response": "output"},
}
},
}
)
assert errors and "output" in errors[0]

View File

@@ -0,0 +1,40 @@
"""计算节点文档路由(/docs、/redoc、/openapi.json安全开关测试。
生产默认COMPUTE_AUTH_ENABLED=true关闭文档路由避免未授权泄露 API 结构;
显式配置 ENABLE_DOCS 可覆盖默认行为。
"""
from __future__ import annotations
from compute.api.security import docs_enabled, docs_kwargs
def test_docs_disabled_when_auth_enabled(monkeypatch) -> None:
monkeypatch.delenv("ENABLE_DOCS", raising=False)
monkeypatch.setenv("COMPUTE_AUTH_ENABLED", "true")
assert docs_enabled() is False
assert docs_kwargs() == {"docs_url": None, "redoc_url": None, "openapi_url": None}
def test_docs_enabled_when_auth_disabled(monkeypatch) -> None:
monkeypatch.delenv("ENABLE_DOCS", raising=False)
monkeypatch.setenv("COMPUTE_AUTH_ENABLED", "false")
assert docs_enabled() is True
assert docs_kwargs() == {}
def test_docs_env_override_enables_with_auth(monkeypatch) -> None:
monkeypatch.setenv("ENABLE_DOCS", "true")
monkeypatch.setenv("COMPUTE_AUTH_ENABLED", "true")
assert docs_enabled() is True
def test_docs_env_override_disables_without_auth(monkeypatch) -> None:
monkeypatch.setenv("ENABLE_DOCS", "false")
monkeypatch.setenv("COMPUTE_AUTH_ENABLED", "false")
assert docs_enabled() is False
def test_docs_default_when_auth_env_missing(monkeypatch) -> None:
monkeypatch.delenv("ENABLE_DOCS", raising=False)
monkeypatch.delenv("COMPUTE_AUTH_ENABLED", raising=False)
assert docs_enabled() is False

View File

@@ -1,6 +1,8 @@
APP_ENV=prod APP_ENV=prod
APP_NAME=YG Fine-Tune Platform API APP_NAME=YG Fine-Tune Platform API
MODELTF_ROUTE_PREFIX=/modelTF MODELTF_ROUTE_PREFIX=/modelTF
# 禁止未授权访问 Swagger/ReDoc/OpenAPI 文档;调试时可临时改为 true
ENABLE_DOCS=false
CORS_ALLOW_ORIGINS=http://localhost:16801,http://127.0.0.1:16801 CORS_ALLOW_ORIGINS=http://localhost:16801,http://127.0.0.1:16801
FRONTEND_IMAGE=yg-ft-frontend-runtime:latest FRONTEND_IMAGE=yg-ft-frontend-runtime:latest
@@ -17,7 +19,9 @@ POSTGRES_USER=root
POSTGRES_PASSWORD=8811614287327Leo POSTGRES_PASSWORD=8811614287327Leo
DATABASE_URL=postgresql+psycopg://root:8811614287327Leo@www.caoxiaozhu.com:5432/yg_ft DATABASE_URL=postgresql+psycopg://root:8811614287327Leo@www.caoxiaozhu.com:5432/yg_ft
REDIS_URL=redis://redis:6379/0 # Redis 访问鉴权requirepass 密码;REDIS_URL 已内嵌密码(redis://:<密码>@redis:6379/0
REDIS_PASSWORD=Tvhrf659WaX-S1B8FG6c2kSZK07XTv82
REDIS_URL=redis://:Tvhrf659WaX-S1B8FG6c2kSZK07XTv82@redis:6379/0
# PostgreSQL uses the shared external database. The local postgres service is disabled in docker-compose.yml. # PostgreSQL uses the shared external database. The local postgres service is disabled in docker-compose.yml.
# Redis still uses the built-in service during current development. # Redis still uses the built-in service during current development.

View File

@@ -2,7 +2,8 @@ FROM python:3.12-slim
ENV PYTHONDONTWRITEBYTECODE=1 \ ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \ PYTHONUNBUFFERED=1 \
PIP_NO_CACHE_DIR=1 PIP_NO_CACHE_DIR=1 \
TIKTOKEN_CACHE_DIR=/opt/tiktoken_cache
WORKDIR /app WORKDIR /app
@@ -16,6 +17,9 @@ RUN python -c "import fastapi, uvicorn, psycopg, psycopg_pool, sqlalchemy, redis
RUN mkdir -p /opt/yg-ft/logs/backend /data/yg-ft \ RUN mkdir -p /opt/yg-ft/logs/backend /data/yg-ft \
&& chmod -R 0775 /opt/yg-ft /data/yg-ft && chmod -R 0775 /opt/yg-ft /data/yg-ft
# 离线打包 tiktoken cl100k_base 词表,避免无网环境下运行时联网下载
COPY docker/app/tiktoken /opt/tiktoken_cache
EXPOSE 8000 EXPOSE 8000
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"] CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

View File

@@ -42,9 +42,10 @@ services:
APP_ENV: ${APP_ENV:-prod} APP_ENV: ${APP_ENV:-prod}
APP_NAME: ${APP_NAME:-YG Fine-Tune Platform API} APP_NAME: ${APP_NAME:-YG Fine-Tune Platform API}
MODELTF_ROUTE_PREFIX: ${MODELTF_ROUTE_PREFIX:-/modelTF} MODELTF_ROUTE_PREFIX: ${MODELTF_ROUTE_PREFIX:-/modelTF}
ENABLE_DOCS: ${ENABLE_DOCS:-false}
CORS_ALLOW_ORIGINS: ${CORS_ALLOW_ORIGINS:-http://localhost:16801,http://127.0.0.1:16801} CORS_ALLOW_ORIGINS: ${CORS_ALLOW_ORIGINS:-http://localhost:16801,http://127.0.0.1:16801}
DATABASE_URL: ${DATABASE_URL:-postgresql+psycopg://root:8811614287327Leo@www.caoxiaozhu.com:5432/yg_ft} DATABASE_URL: ${DATABASE_URL:-postgresql+psycopg://root:8811614287327Leo@www.caoxiaozhu.com:5432/yg_ft}
REDIS_URL: ${REDIS_URL:-redis://redis:6379/0} REDIS_URL: ${REDIS_URL:-redis://:${REDIS_PASSWORD:-change_me}@redis:6379/0}
USE_BUILTIN_POSTGRES: ${USE_BUILTIN_POSTGRES:-false} USE_BUILTIN_POSTGRES: ${USE_BUILTIN_POSTGRES:-false}
USE_BUILTIN_REDIS: ${USE_BUILTIN_REDIS:-true} USE_BUILTIN_REDIS: ${USE_BUILTIN_REDIS:-true}
LOG_LEVEL: ${LOG_LEVEL:-INFO} LOG_LEVEL: ${LOG_LEVEL:-INFO}
@@ -103,7 +104,10 @@ services:
redis: redis:
image: redis:7-alpine image: redis:7-alpine
container_name: yg-ft-redis container_name: yg-ft-redis
command: ["redis-server", "--appendonly", "yes"] command: ["redis-server", "--appendonly", "yes", "--requirepass", "${REDIS_PASSWORD:-change_me}"]
environment:
# redis-cli 健康检查免命令行传密码(避免 -a 泄露进程参数)
REDISCLI_AUTH: ${REDIS_PASSWORD:-change_me}
volumes: volumes:
- redis_data:/data - redis_data:/data
ports: ports:

View File

@@ -2,6 +2,8 @@ COMPUTE_ENV=prod
COMPUTE_HOST_ID=gpu-node-01 COMPUTE_HOST_ID=gpu-node-01
COMPUTE_EXECUTION_MODE=real COMPUTE_EXECUTION_MODE=real
MODELTF_ROUTE_PREFIX=/modelTF MODELTF_ROUTE_PREFIX=/modelTF
# 禁止未授权访问 Swagger/ReDoc/OpenAPI 文档;调试时可临时改为 true
ENABLE_DOCS=false
# Five-digit host ports exposed outside the compute server. # Five-digit host ports exposed outside the compute server.
COMPUTE_API_PORT=19100 COMPUTE_API_PORT=19100
FILE_GATEWAY_PORT=19101 FILE_GATEWAY_PORT=19101

View File

@@ -2,6 +2,8 @@ COMPUTE_ENV=prod
COMPUTE_HOST_ID=gpu-node-01 COMPUTE_HOST_ID=gpu-node-01
COMPUTE_EXECUTION_MODE=real COMPUTE_EXECUTION_MODE=real
MODELTF_ROUTE_PREFIX=/modelTF MODELTF_ROUTE_PREFIX=/modelTF
# 禁止未授权访问 Swagger/ReDoc/OpenAPI 文档;调试时可临时改为 true
ENABLE_DOCS=false
# Five-digit host ports exposed outside the compute server. # Five-digit host ports exposed outside the compute server.
COMPUTE_API_PORT=19100 COMPUTE_API_PORT=19100
FILE_GATEWAY_PORT=19101 FILE_GATEWAY_PORT=19101

View File

@@ -11,6 +11,7 @@ services:
COMPUTE_HOST_ID: ${COMPUTE_HOST_ID:-gpu-node-01} COMPUTE_HOST_ID: ${COMPUTE_HOST_ID:-gpu-node-01}
COMPUTE_EXECUTION_MODE: ${COMPUTE_EXECUTION_MODE:-real} COMPUTE_EXECUTION_MODE: ${COMPUTE_EXECUTION_MODE:-real}
MODELTF_ROUTE_PREFIX: ${MODELTF_ROUTE_PREFIX:-/modelTF} MODELTF_ROUTE_PREFIX: ${MODELTF_ROUTE_PREFIX:-/modelTF}
ENABLE_DOCS: ${ENABLE_DOCS:-false}
COMPUTE_AUTH_ENABLED: ${COMPUTE_AUTH_ENABLED:-true} COMPUTE_AUTH_ENABLED: ${COMPUTE_AUTH_ENABLED:-true}
COMPUTE_SERVICE_TOKEN: ${COMPUTE_SERVICE_TOKEN:-change_me} COMPUTE_SERVICE_TOKEN: ${COMPUTE_SERVICE_TOKEN:-change_me}
ENABLE_APP_CALLBACK: ${ENABLE_APP_CALLBACK:-false} ENABLE_APP_CALLBACK: ${ENABLE_APP_CALLBACK:-false}

164
docs/database-config.md Normal file
View File

@@ -0,0 +1,164 @@
# 数据库配置与初始化说明PostgreSQL / Redis
> 记录平台的 **PostgreSQL 账号密码**、**Redis 账号密码**、**数据库地址在代码中的配置位置**
> 以及**切换 PG 数据集时如何执行完整初始化 SQL**。
---
## 1. 账号密码速查表
### 1.1 PostgreSQL
| 环境 | 地址 | 用户 | 密码 | 数据库 | 来源 |
|------|------|------|------|--------|------|
| 代码默认值 | `localhost:15432` | `yg_ft` | `change_me` | `yg_ft` | `config.py` / `session.py``DATABASE_URL` 兜底 |
| Docker 部署 | `www.caoxiaozhu.com:5432` | `root` | `8811614287327Leo` | `yg_ft` | `docker/app/.env``DATABASE_URL` |
| Docker 内置 Postgres已注释 | `localhost:15432` | `root` | `8811614287327Leo` | `yg_ft` | `docker/app/docker-compose.yml` 注释掉的 postgres 服务 |
> ⚠️ `change_me` 与 `8811614287327Leo` 均为默认/示例凭据,生产环境务必更换。
### 1.2 Redis
| 项 | 值 | 说明 |
|----|-----|------|
| 连接串 | `redis://:<REDIS_PASSWORD>@redis:6379/0` | 已内嵌密码docker 网络内服务名 `redis`,端口 6379db 0 |
| 对外端口 | `16379``REDIS_PORT` | 宿主机映射 |
| 密码 | `docker/app/.env``REDIS_PASSWORD` | 已启用 `requirepass` 鉴权 |
| 镜像 | `redis:7-alpine` | 已开启 AOF`--appendonly yes`+ `requirepass` |
> **当前后端代码未使用 Redis**`redis` 包已列入 `requirements.txt``REDIS_URL` 通过
> docker-compose 注入容器,但全仓库 `backend/`、`compute/` 没有任何 `import redis` /
> `Redis(...)` 连接代码。Redis 为后续功能预留;**即便如此仍已配置鉴权**
> 避免无密码实例对外暴露(纵深防御)。未来启用时按 `REDIS_URL` 连接即可。
>
> 健康检查通过容器环境变量 `REDISCLI_AUTH` 认证,不在进程参数中泄露密码。
---
## 2. 数据库地址在代码中的配置位置
### 2.1 后端backend
| 文件 | 作用 | 取值 |
|------|------|------|
| `backend/app/core/config.py` | **唯一权威配置**`Settings.database_url` | `os.getenv("DATABASE_URL", "postgresql+psycopg://yg_ft:change_me@localhost:15432/yg_ft")` |
| `backend/app/db/session.py` | SQLAlchemy 引擎(`get_db` / `session_scope` | `os.getenv("DATABASE_URL", ...)` 同样兜底 |
| `backend/app/db/platform_store.py` | **平台主存储**,直接用 psycopg 连接池 | 取 `settings.database_url``_psycopg_url()``postgresql+psycopg://` 转成 `postgresql://` |
| `backend/app/modules/data_process/store.py` | 数据处理存储 | `get_settings().database_url` |
> **环境变量加载顺序**`config.py` 导入时会 `load_dotenv(backend/.env, override=True)`
> 即 **`backend/.env` 会覆盖系统环境变量**docker 部署则直接由 compose 注入 `DATABASE_URL`。
> 最终优先级:`backend/.env` / compose 注入的环境变量 > 代码内默认值。
### 2.2 部署配置docker
| 文件 | 关键项 |
|------|--------|
| `docker/app/.env` | `DATABASE_URL``POSTGRES_USER``POSTGRES_PASSWORD``REDIS_URL``REDIS_PORT` |
| `docker/app/docker-compose.yml` | `backend-api` 环境透传上述变量;`redis` 服务定义 |
```ini
# docker/app/.env节选
DATABASE_URL=postgresql+psycopg://root:8811614287327Leo@www.caoxiaozhu.com:5432/yg_ft
POSTGRES_USER=root
POSTGRES_PASSWORD=8811614287327Leo
REDIS_PASSWORD=<强密码> # Redis requirepass新增鉴权
REDIS_URL=redis://:<REDIS_PASSWORD>@redis:6379/0 # 连接串内嵌密码
REDIS_PORT=16379
USE_BUILTIN_POSTGRES=false # 当前用共享外部库,内置 postgres 服务被注释
USE_BUILTIN_REDIS=true # Redis 用内置服务
```
---
## 3. 完整初始化 SQL
### 3.1 脚本位置
| 脚本 | 用途 |
|------|------|
| **`backend/app/db/sql/000_full_init.sql`** | **一键初始化脚本(新增)**:建库表 + 索引 + 种子数据,幂等,覆盖全部 39 张运行表 |
| `backend/app/db/sql/001_platform_runtime.sql` | 平台核心表(应用启动自动执行) |
| `backend/app/db/sql/002_governance.sql` | 治理表(应用启动自动执行) |
| `backend/app/db/sql/003_tenant_quota.sql` | 租户配额列(应用启动自动执行) |
| `backend/app/db/sql/003_model_path_governance.sql` | 模型可训练列(**应用不自动执行**,已并入完整脚本) |
| `backend/app/db/sql/002_data_process.sql` | 数据处理表(**应用不自动执行**,已并入完整脚本) |
| `docs/postgres-schema.sql` | ⚠️ **目标设计稿**UUID/JSONB与运行时代码不兼容**不要用于初始化** |
> **重要**`docs/postgres-schema.sql` 是规划中的“目标 schema”UUID 主键、`ft_platform` schema 等),
> 运行时代码明确拒绝该结构(`002_data_process.sql` 检测到 `datasets.id` 非 TEXT 会直接报错)。
> 初始化请使用 **`000_full_init.sql`**。
### 3.2 执行步骤(全新 PG 环境)
**第 1 步:创建角色与数据库**(必须单独执行,不能放进事务)
```sql
-- 以超级用户(如 postgres连接
CREATE ROLE yg_ft LOGIN PASSWORD '请改为强密码';
CREATE DATABASE yg_ft OWNER yg_ft;
-- 如需应用执行 CREATE EXTENSION 等,可再授予超级用户(按需):
-- ALTER ROLE yg_ft SUPERUSER;
```
**第 2 步:执行完整初始化脚本**
```bash
psql "postgresql://yg_ft:密码@<host>:5432/yg_ft" \
-f backend/app/db/sql/000_full_init.sql
```
脚本特点:
- 全程一个事务(`BEGIN; ... COMMIT;`),失败自动回滚
- 所有 DDL 使用 `IF NOT EXISTS` / `ADD COLUMN IF NOT EXISTS`**可重复执行**
-`DO $$...$$` 语句块,必须用 `psql` 执行(应用内部的按分号切分 `executescript()` 不适用)
- 自动写入种子用户:`admin / admin123``operator / operator123`(登录后请改密)
**第 3 步:校验**
```sql
SELECT count(*) FROM pg_tables WHERE schemaname = 'public'; -- 应 ≥ 39
SELECT username, role, status FROM users; -- 应有 admin / operator
```
### 3.3 执行方式对比(三种途径)
| 方式 | 覆盖范围 | 命令 |
|------|----------|------|
| **A. 完整脚本(推荐,切换新库)** | 全部 39 表 + 索引 + 种子 | `psql ... -f 000_full_init.sql` |
| B. 应用自动初始化 | 001 + 002_governance + 003_tenant_quota + 种子用户;**不含**数据处理表、`models.can_train``data_convert_tasks` | 应用首次调用 `get_platform_store()``ensure_schema()` 自动执行 |
| C. 数据处理表单独安装 | `002_data_process.sql` 全部内容 | 在 `backend/` 目录下:`python -m app.modules.data_process.schema_cli --apply --yes`(或 `--check` 只读检查) |
> **缺口说明**
> - `models.can_train`(训练预检用)只在 `003_model_path_governance.sql` 中创建,应用启动**不会**自动执行;
> - `data_convert_tasks`(数据转换任务表)运行时代码引用但**原 SQL 脚本缺失**
> 已统一并入 `000_full_init.sql` 补齐。若现有库缺这两项,执行一次完整脚本即可幂等补上。
### 3.4 完整脚本包含的表39 张)
**核心**users、models、trained_models、model_lineage、model_artifacts、model_export_jobs、
datasets、dataset_files、compute_nodes、gpus、fine_tune_tasks、fine_tune_metrics、
fine_tune_checkpoints、compute_jobs、gpu_allocations、scheduler_locks、resource_replicas、
resource_sync_jobs、eval_tasks、eval_dimensions、compare_tasks、projects、project_members、
roles、sessions、acls
**治理**tenants、approval_templates、approval_instances、approval_steps、audit_logs、retention_policies
**数据处理**data_process_tasks、data_process_source_files、data_process_preview_items、
data_process_results、dataset_file_versions、dataset_records
**数据转换**data_convert_tasks新增补齐
---
## 4. 安全注意事项
1. **更换默认密码**`change_me`(代码兜底)、`8811614287327Leo`(部署)、`admin123`/`operator123`(种子用户)、`REDIS_PASSWORD` 上线前必须更换。
2. **Redis 已加鉴权**:已配置 `requirepass` + `REDISCLI_AUTH` 健康检查;`REDIS_URL` 内嵌密码。若端口需暴露公网,仍建议用防火墙/安全组限制来源。
3. **`docker/*/.env` 已入库,含明文凭据**
- `backend/.env` 已被 `.gitignore` 排除;
-`docker/app/.env``docker/compute/.env` 目前被 git 跟踪(`git ls-files` 可见),
其中的 `DATABASE_URL``POSTGRES_PASSWORD``COMPUTE_SERVICE_TOKEN` 等均为明文。
- **建议**:轮换这些凭据,将 `docker/*/.env` 移出版本库(`git rm --cached`)并改用
部署侧机密注入(如 docker secrets / CI 变量 / 环境变量模板),保留 `.env.example` 作为模板。
4. **最小权限**:应用角色只需对业务库的 DML/DDL 权限,尽量避免 SUPERUSER。

View File

@@ -0,0 +1,411 @@
# 平台治理功能使用指南
> 版本v1.0
> 日期2026-08-10
> 适用版本YG Fine-Tune Platform v1.0+
---
## 目录
1. [快速入门](#1-快速入门)
2. [用户与权限管理](#2-用户与权限管理)
3. [GPU 算力分配](#3-gpu-算力分配)
4. [资源可见性与隔离](#4-资源可见性与隔离)
5. [审批流程管理](#5-审批流程管理)
6. [审计日志查询](#6-审计日志查询)
7. [常见问题与排查](#7-常见问题与排查)
---
## 1. 快速入门
### 1.1 平台治理是什么?
平台治理是一套**多租户、多角色、细粒度权限控制**体系,用于在多人协作使用 AI 微调平台时,确保:
- 每个用户只能看到和操作自己有权限的资源
- GPU 算力按需分配,避免资源争抢
- 高风险操作(删除、停止任务)有审批记录可追溯
- 所有操作都有审计日志
### 1.2 三种内置角色
| 角色 | 能做什么 | 不能做什么 |
|---|---|---|
| **超级管理员 (admin)** | 全部操作;管理用户、分配 GPU、审批、查看全部资源 | — |
| **操作员 (operator)** | 创建数据集/模型、训练/评测/推理任务 | 管理用户、分配 GPU、修改他人权限 |
| **观察员 (viewer)** | 查看被授权的资源 | 创建或修改任何资源 |
### 1.3 入口在哪里?
所有治理功能集中在左侧导航栏的 **「系统设置」** 分组下:
```
系统设置
├── 用户设置 ← 用户 CRUD + 角色权限 + 密码管理
├── 租户管理 ← 组织/团队(可选)
├── 项目空间 ← 项目级资源隔离(可选)
├── 审批模板 ← 定义哪些操作需要审批
├── 审批中心 ← 处理待审批请求
└── 审计日志 ← 查看所有操作记录
```
> ⚠️ 以上菜单**只有 admin 用户能看到**。普通用户登录后不会出现这些入口。
---
## 2. 用户与权限管理
### 2.1 创建用户
**路径**`用户设置``创建用户`
1. 以 admin 身份登录平台
2. 进入「用户设置」页面
3. 点击右上角「创建用户」按钮
4. 填写信息:
- **账号**:登录用户名(如 `zhangsan`
- **显示名称**:如 `张三`
- **密码**:初始密码(默认 `Platform@123`
- **角色**:选择 `admin` / `operator` / `viewer`
5. 点击保存
创建后用户可以立即用该账号登录。
### 2.2 管理用户权限
**路径**`用户设置` → 用户列表 → 操作列「页面权限」
#### 给普通用户分配业务模块权限
点击某用户的「页面权限」按钮,弹出对话框:
```
为 张三 分配可访问的页面模块:
☑ 服务看板 ☑ 模型训练 ☑ 模型评测
☑ 模型推理 ☑ 模型管理 ☑ 数据集管理
☐ 数据处理 ☐ 数据类型转换 ☐ 算力节点 ← 不勾选则不可见
☑ 平台性能 ☑ 查看日志
[取消] [保存]
```
勾选需要的模块,点「保存」即可。
> **注意**
> - 「用户与权限」这个选项**只有 admin 能看到**,其他用户即使被赋权也不会显示
> - admin 用户的权限**不可更改**,始终是全选状态且只读
#### 限制说明
| 权限码 | 说明 | 谁能拥有 |
|---|---|---|
| `user-settings` | 用户设置、租户管理、项目空间、审批、审计日志 | **仅 admin** |
| `compute` | 算力节点、GPU 分配 | **仅 admin** |
| 其他业务权限 | 训练、评测、推理、模型、数据集等 | admin 可分配给任何人 |
### 2.3 重置用户密码
**两种方式**
**方式一:管理员重置**
1. 在用户列表中找到目标用户
2. 点击「重置密码」
3. 输入新密码,确认
**方式二:用户自行修改**
1. 用户登录后在「用户设置」页面点击「修改密码」按钮
2. 输入旧密码 + 新密码(至少 6 位)
3. 确认修改
### 2.4 删除用户
**路径**`用户设置` → 用户列表 → 操作列「删除」
> ⚠️ 删除用户时会**级联清理**其所有关联数据:
> - 该用户创建的数据集、基座模型、微调产物、评测任务
> - 该用户的 ACL 授权记录、GPU 分配记录
> - 该用户的审批实例、审计日志、项目成员关系
> - **训练任务保留不删**(避免算力节点上的物理任务数据不一致)
---
## 3. GPU 算力分配
### 3.1 为什么需要 GPU 分配?
当服务器有多张 GPU 卡(如 8×A800需要指定**哪个用户能用哪张卡**
- 避免两个人同时选同一张卡导致训练冲突
- 按团队/项目隔离算力资源
- 控制每个用户的 GPU 配额
### 3.2 分配 GPU仅 admin
**路径**`算力节点``GPU 分配` 标签页
1. 以 admin 登录,进入「算力节点」页面
2. 点击顶部的 **「GPU 分配」** 标签(只有 admin 可见)
3. 点击 **「分配 GPU」** 按钮
4. 填写:
- **算力节点**:选择节点(如 `gpu-node-01`
- **GPU 序号**卡号0, 1, 2, ... 7
- **用户**:选择要分配给谁
5. 点「确认分配」
示例:把节点 `gpu-node-01` 的第 0、1 号卡分配给用户 `zhangsan`
```
┌─────────────────────────────────────┐
│ 分配 GPU [×] │
├─────────────────────────────────────┤
│ 算力节点: [gpu-node-01 ▼] │
│ GPU 序号: [0 ▲] │
│ 用户: [zhangsan ▼] │
│ │
│ [取消] [确认分配] │
└─────────────────────────────────────┘
```
分配后的效果:
| 用户 | 可用 GPU |
|---|---|
| admin | 全部 GPU不需要显式分配 |
| zhangsan | gpu-node-01 的 0、1 号卡 |
| lisi | (未分配,不可用) |
### 3.3 撤销分配
在 GPU 分配列表中,每条记录右侧有「撤销」按钮,点击后确认即可移除该分配。
### 3.4 用户视角:创建训练任务时的 GPU 选择
- **admin**:下拉列表显示全部可用 GPU
- **被分配了 GPU 的用户**:只显示被分配给自己的卡
- **未分配任何 GPU 的用户**:显示提示「未分配 GPU请联系管理员」无法提交训练任务
---
## 4. 资源可见性与隔离
### 4.1 自动生效的隔离规则
无需手动配置,以下规则自动生效:
| 资源类型 | admin 看到 | 普通用户看到 |
|---|---|---|
| **基座模型**(容器内注册的本地模型) | 全部 | **全部**(共享资源,有 model-manage 权限即可见) |
| **数据集** | 全部 | **自己创建的** + 被 ACL 授权的 |
| **微调产物**(训练输出的模型) | 全部 | **自己训练的** + 被 ACL 授权的 |
| **评测任务** | 全部 | **自己创建的** + 被 ACL 授权的 |
| **推理/对比任务** | 全部 | **自己创建的** + 被 ACL 授权的 |
### 4.2 实际场景示例
假设有三个用户:**admin**、**zhangsan**(算法工程师)、**lisi**(标注员)
```
zhangsan 上传了数据集 ds_alpaca、ds_sharegpt
zhangsan 训练出了模型 ft_qwen_001
lisi 上传了数据集 ds_label
admin 注册了基座模型 Qwen3-1.7B
```
各用户看到的资源:
| 用户 | 数据集 | 基座模型 | 微调产物 |
|---|---|---|---|
| **admin** | ds_alpaca, ds_sharegpt, ds_label (3个) | Qwen3-1.7B | ft_qwen_001 |
| **zhangsan** | ds_alpaca, ds_sharegpt (2个) | Qwen3-1.7B | ft_qwen_001 |
| **lisi** | ds_label (1个) | Qwen3-1.7B | (无) |
### 4.3 ACL 资源授权(高级用法)
如果 zhangsan 想让 lisi 也能看到自己的数据集 `ds_alpaca`
> 此功能需要在资源详情页提供「资源授权」按钮(前端已预留接口),当前可通过 API 直接操作:
```bash
# 授予 lisi 对 ds_alpaca 的读权限
curl -X PUT /modelTF/resources/dataset/ds_alpaca_id/acl \
-H "Authorization: Bearer platform-token-admin" \
-d '{
"acls": [
{"principal_type": "user", "principal_id": "lisi_id", "permission": "read"}
]
}'
```
---
## 5. 审批流程管理
### 5.1 哪些操作会触发审批?
| 操作 | 触发条件 | 处理方式 |
|---|---|---|
| 删除他人的数据集 | 非 admin 删除别人创建的数据集 | 创建审批实例 或 admin 直接执行 |
| 删除他人的模型 | 非 admin 删除别人创建的模型 | 同上 |
| 停止他人的训练任务 | 非 admin 停止别人发起的任务 | 同上 |
| 归档/删除项目空间 | 存在待审批变更时 | 拒绝执行 |
**核心规则**admin 做任何操作都直接执行(旁路);普通用户操作他人资源时进入审批流程。
### 5.2 审批流程示意
```
普通用户 lisi 尝试删除 zhangsan 的数据集
┌───────────────────────┐
│ 后端检查:是 admin 吗? │
└──────┬────────────────┘
│ 否
┌───────────────────────┐
│ 创建审批实例 │
│ status = pending │
│ 返回 202待审批
└──────────┬────────────┘
┌───────────────────────┐
│ admin 在「审批中心」看到 │
│ 这条待审批请求 │
│ 点击「通过」或「拒绝」 │
└──────────┬────────────┘
┌─────┴─────┐
│ │
通过 拒绝
│ │
▼ ▼
执行删除 不执行
+审计日志 +审计日志
```
### 5.3 管理审批
**路径**`系统设置``审批中心`
1. 查看待审批列表status=pending
2. 点击某条记录查看详情
3. 决策:「通过」或「拒绝」
4. 决策结果自动执行对应操作并记录审计日志
**审批模板**`系统设置``审批模板`定义每种操作需要几步审批、每步谁来审。默认模板都是单步admin 审批即可)。
---
## 6. 审计日志查询
### 6.1 什么是审计日志?
平台上所有**写操作**和**敏感操作**都会自动记录审计日志,包括:
- 用户创建/删除/修改
- 权限变更
- GPU 分配/撤销
- 资源上传/删除
- 训练任务启动/停止
- 审批决策
### 6.2 查询审计日志
**路径**`系统设置``审计日志`
支持筛选条件:
| 筛选项 | 说明 |
|---|---|
| 操作人 | 按用户 ID 过滤 |
| 动作类型 | 如 `user.create`, `dataset.delete`, `gpu.assign` 等 |
| 目标资源类型 | dataset / model / fine_tune_task 等 |
| 时间范围 | 开始时间 ~ 结束时间 |
### 6.3 导出审计日志
审计日志页面底部有「导出 CSV」按钮导出的文件包含当前筛选条件下的全部记录可用于合规审计或问题追溯。
### 6.4 日志保留策略
审计日志受**留存策略**控制(`系统设置` → 租户管理 → 绑定留存策略)。默认保留 30 天,超期自动清理。
---
## 7. 常见问题与排查
### Q1: 普通用户看不到某个菜单?
检查两件事:
1. 该用户是否有对应的**权限码**admin 在「用户设置」→「页面权限」中分配)
2. 该菜单是否属于 **admin 专属**(如「用户设置」「算力节点」——这些对非 admin 永远不可见)
### Q2: 用户创建训练任务时报错"无权使用所选 GPU"
说明该用户没有被分配所选择的 GPU 卡。解决方法:
1. admin 进入「算力节点」→「GPU 分配」标签页
2. 为该用户分配对应的 GPU
3. 用户刷新页面重新选择 GPU
### Q3: 删除用户后看板还显示残留数据?
正常情况下 `delete_user` 会级联清理关联数据。如果仍有残留:
- **训练任务**:设计上保留不删(避免算力节点物理数据不一致),这是预期行为
- **登录时长排行**:可能来自旧的 session 记录,不影响功能,新登录后会更新
### Q4: 审批实例一直 pending 没人处理?
审批实例需要 admin 在「审批中心」手动处理。如果长时间无人处理:
- 可以在数据库中直接将 `approval_instances.status` 改为 `rejected`
- 或者由 admin 直接以自身身份执行该操作admin 有旁路权限)
### Q5: 如何查看当前所有 GPU 分配情况?
```bash
# admin 调用接口
curl -H "Authorization: Bearer platform-token-admin" \
/modelTF/compute/gpu-assignments
```
返回格式:
```json
{
"data": [
{
"id": "ga_xxx",
"node_name": "A800 训练节点",
"gpu_index": 0,
"display_name": "张三",
"assigned_at": "2026-08-10T10:00:00"
}
]
}
```
### Q6: 用户忘记密码怎么办?
两种方案:
1. **admin 重置**:在「用户设置」→ 用户列表 →「重置密码」
2. **用户自助修改**:用户登录后点击「修改密码」(需知道旧密码)
如果是完全忘记且不是 admin只能由 admin 重置。
---
## 附录API 快速参考
| 功能 | 方法 | 路径 | 鉴权 |
|---|---|---|---|
| 查看我的 GPU | GET | `/compute/my-gpus` | 登录用户 |
| 查看 GPU 分配 | GET | `/compute/gpu-assignments` | admin |
| 分配 GPU | POST | `/compute/gpu-assignments` | admin |
| 撤销 GPU 分配 | DELETE | `/compute/gpu-assignments/{id}` | admin |
| 修改自己的密码 | POST | `/users/me/password` | 登录用户 |
| 查看审计日志 | GET | `/system/audit-logs` | admin |
| 导出审计日志 | GET | `/system/audit-logs/export` | admin |
| 查看审批列表 | GET | `/approvals` | 登录用户 |
| 审批决策 | POST | `/approvals/:id/steps/:idx/decision` | 审批人 |

805
docs/permissions-design.md Normal file
View File

@@ -0,0 +1,805 @@
# 平台权限设计文档
> 版本v1.0
> 日期2026-08-02
> 状态:设计基线,供后端实现和前端联调参照
---
## 目录
1. [设计目标](#1-设计目标)
2. [整体架构](#2-整体架构)
3. [角色体系](#3-角色体系)
4. [页面权限码](#4-页面权限码)
5. [资源所有权与可见性](#5-资源所有权与可见性)
6. [资源级 ACL访问控制列表](#6-资源级-acl访问控制列表)
7. [GPU 算力分配与隔离](#7-gpu-算力分配与隔离)
8. [审批拦截机制](#8-审批拦截机制)
9. [审计日志](#9-审计日志)
10. [接口鉴权流程](#10-接口鉴权流程)
11. [数据库表结构](#11-数据库表结构)
12. [API 接口清单](#12-api-接口清单)
13. [前端权限控制](#13-前端权限控制)
14. [安全设计补充](#14-安全设计补充)
15. [实施计划](#15-实施计划)
---
## 1. 设计目标
| 目标 | 说明 |
|---|---|
| **数据隔离** | 用户自己创建的数据集、模型、训练任务默认只有自己可见可操作;管理员可见全部 |
| **权限分层** | 页面级(菜单/路由可见性)+ 资源级(单条数据的读/写/删)两层控制 |
| **GPU 管控** | 多卡服务器上,管理员可指定哪些用户能使用哪些 GPU 卡 |
| **审批拦截** | 删除他人资源、停止他人任务、发布模型等高风险操作需审批或管理员旁路 |
| **审计可追溯** | 所有写操作和敏感操作产生审计日志,可按用户、动作、资源、时间筛选 |
| **权限最小变更** | 只有管理员可修改用户角色和权限码;普通用户无法提权 |
---
## 2. 整体架构
```
┌──────────────────────────────────────────────────────┐
│ 前端Vue3
│ ┌────────────┐ ┌────────────┐ ┌───────────────┐ │
│ │ 路由守卫 │ │ 侧边栏过滤 │ │ 页面内按钮控制 │ │
│ │ (permission)│ │ (permission)│ │ (ACL/owner) │ │
│ └──────┬─────┘ └──────┬─────┘ └───────┬───────┘ │
│ └───────────────┴─────────────────┘ │
│ │ HTTP (Bearer token) │
└─────────────────────────┼────────────────────────────┘
┌─────────────────────────┼────────────────────────────┐
│ 后端FastAPI
│ ┌──────────────┐ ┌────┴───────┐ ┌──────────────┐ │
│ │ get_current │ │ 资源可见性 │ │ GPU 分配校验 │ │
│ │ _user (鉴权) │ │ 过滤器 │ │ │ │
│ └──────┬───────┘ └────┬───────┘ └──────┬───────┘ │
│ │ │ │ │
│ ┌──────┴───────────────┴──────────────────┘ │
│ │ PlatformStore │ │
│ │ users | acls | roles | gpu_assignments | │ │
│ │ datasets | models | fine_tune_tasks | ... │ │
│ └───────────────────────────────────────────────────│ │
│ │ audit_logs (审计日志) │ │
│ └───────────────────────────────────────────────────│ │
└─────────────────────────────────────────────────────────┘
```
**鉴权链路**
1. 请求到达 → `get_current_user``Authorization: Bearer platform-token-{user_id}` 解析当前用户
2. 页面级权限 → 检查 `user.permissions` 是否包含路由对应的权限码
3. 资源级权限 → 检查资源的 `created_by` 字段(所有权)或 `acls`ACL 授权)
4. GPU 权限 → 检查 `gpu_assignments` 表确认用户是否被分配了请求的 GPU
---
## 3. 角色体系
### 3.1 内置角色
| 角色 code | 中文名 | 说明 |
|---|---|---|
| `admin` | 超级管理员 | 拥有全部权限码;可见全部资源;可管理用户和 GPU 分配 |
| `operator` | 操作员 | 可创建/操作自己的数据集、模型、训练任务;不可管理用户 |
| `viewer` | 观察员 | 只读权限;可查看被授权的资源;不可创建或修改 |
| `guest` | 访客 | 仅登录和看板;无业务操作权限(扩展预留) |
### 3.2 角色与权限码映射
| 权限码 | admin | operator | viewer |
|---|---|---|---|
| `dashboard` | ✅ | ✅ | ✅ |
| `fine-tune` | ✅ | ✅ | — |
| `model-eval` | ✅ | ✅ | — |
| `model-inference` | ✅ | ✅ | — |
| `model-manage` | ✅ | ✅ | — |
| `dataset` | ✅ | ✅ | — |
| `data-process` | ✅ | ✅ | — |
| `data-convert` | ✅ | ✅ | — |
| `compute` | ✅ | ✅ | — |
| `hardware` | ✅ | ✅ | ✅ |
| `logs` | ✅ | ✅ | ✅ |
| `user-settings` | ✅ | — | — |
### 3.3 权限修改规则
- **只有 admin 角色的用户**可以修改其他用户的角色和权限码
- admin 用户的 `protected=True` 标记,防止被删除或降级
- 权限修改操作产生审计日志:`action=user.permission.update`
- 用户可以查看自己的权限,不能修改自己的权限
---
## 4. 页面权限码
| 权限码 | 对应路由 | 功能 |
|---|---|---|
| `dashboard` | `/dashboard` | 服务看板 |
| `fine-tune` | `/fine-tune`, `/fine-tune/create`, `/training-log/:id` | 模型训练 |
| `model-eval` | `/model-eval`, `/model-eval/create`, `/model-eval/:id` | 模型评测 |
| `model-inference` | `/model-inference`, `/model-inference/create`, `/model-inference/chat/:id` | 模型推理 |
| `model-manage` | `/model-manage`, `/model-manage/create`, `/model-manage/:id/edit`, `/model-manage/merge` | 模型管理 |
| `dataset` | `/dataset`, `/dataset/create`, `/dataset/:id/preview` | 数据集管理 |
| `data-process` | `/data-process`, `/data-process/create`, `/data-process/:id` | 数据处理 |
| `data-convert` | `/data-convert`, `/tools` | 数据转换与工具 |
| `compute` | `/compute` | 算力节点 |
| `hardware` | `/hardware` | 平台性能 |
| `logs` | `/logs`, `/training-log/:id` | 查看日志 |
| `user-settings` | `/user-settings`, `/tenants`, `/projects`, `/approvals`, `/audit-logs` | 系统设置与平台治理 |
---
## 5. 资源所有权与可见性
### 5.1 所有权模型
每个用户可创建的资源都携带 `created_by`(或 `owner_id`)字段,标识资源所有者。
| 资源类型 | 表 | 所有者字段 | 说明 |
|---|---|---|---|
| 数据集 | `datasets` | `created_by` | 用户上传/创建的数据集 |
| 基座模型 | `models` | `created_by` | 登记的本地/API 模型 |
| 训练产物 | `trained_models` | `created_by` | 微调产出的模型 |
| 训练任务 | `fine_tune_tasks` | `payload.created_by` | 微调任务 |
| 评测任务 | `eval_tasks` | `created_by` | 评测任务 |
| 推理任务 | `compare_tasks` (payload) | `created_by` | 推理/对比任务 |
| 数据处理任务 | `data_process_tasks` | `created_by` | 数据处理任务 |
| 数据转换任务 | `data_convert_jobs` | `created_by` | 数据转换任务 |
### 5.2 可见性规则
```
资源列表查询过滤逻辑:
if user.role == "admin":
返回全部资源
elif resource.created_by == user.id:
返回(资源所有者可见自己的资源)
elif acl 中存在 (principal_type="user", principal_id=user.id, permission 包含 "read"):
返回(被 ACL 显式授权的资源)
elif acl 中存在 (principal_type="role", principal_id=user.role, permission 包含 "read"):
返回(被角色级 ACL 授权的资源)
else:
不可见
```
### 5.3 所有权操作矩阵
| 操作 | admin | 资源所有者 | 其他被授权用户 | 其他用户 |
|---|---|---|---|---|
| 查看资源 | ✅ 全部 | ✅ 自己的 | ✅ ACL 授权范围内 | ❌ |
| 编辑资源 | ✅ | ✅ 自己的 | ✅ ACL 含 write 时 | ❌ |
| 删除资源 | ✅ | ✅ 自己的(需审批) | ❌ | ❌ |
| 分享/授权 | ✅ | ✅ 自己的 | ❌ | ❌ |
| 使用资源(训练/推理/评测) | ✅ | ✅ 自己的 | ✅ ACL 含 execute 时 | ❌ |
### 5.4 数据集可见性示例
```
用户 A 创建了数据集 ds_A1 → 只有 A 和 admin 可见
用户 A 通过 ACL 把 ds_A1 的 read 权限授给用户 B → B 也可见
用户 A 通过 ACL 把 ds_A1 的 write 权限授给 operator 角色 → 所有 operator 可编辑
管理员可在任何数据集上设置 ACL
```
---
## 6. 资源级 ACL访问控制列表
### 6.1 ACL 表结构
```sql
CREATE TABLE IF NOT EXISTS acls (
id TEXT PRIMARY KEY,
resource_type TEXT NOT NULL, -- 资源类型: dataset / model / trained_model / fine_tune_task / ...
resource_id TEXT NOT NULL, -- 资源 ID
principal_type TEXT NOT NULL, -- 授权主体类型: user / role
principal_id TEXT NOT NULL, -- 授权主体 ID: user_id 或 role name
permission TEXT NOT NULL, -- 权限: read / write / execute / download / delete / admin
create_time TEXT
);
```
### 6.2 权限粒度
| 权限值 | 含义 | 覆盖关系 |
|---|---|---|
| `read` | 查看资源详情、列表 | — |
| `write` | 编辑资源内容/元数据 | 覆盖 `read` |
| `execute` | 使用资源(如用数据集训练、用模型推理) | 覆盖 `read` |
| `download` | 下载资源文件 | 独立权限 |
| `delete` | 删除资源 | 独立权限(通常需审批) |
| `admin` | 完全控制(含 ACL 管理) | 覆盖以上全部 |
### 6.3 ACL 管理接口
| 接口 | 方法 | 权限要求 | 说明 |
|---|---|---|---|
| `/resources/{type}/{id}/acl` | GET | admin 或资源所有者 | 查询资源 ACL |
| `/resources/{type}/{id}/acl` | PUT | admin 或资源所有者 | 设置资源 ACL全量替换 |
### 6.4 ACL 管理规则
- **admin** 可以在任何资源上设置 ACL
- **资源所有者** 可以在自己的资源上设置 ACL
- **被授权用户** 不能转授自己获得的权限
- ACL 变更产生审计日志:`action=resource.acl.set`
- 设置 ACL 时全量替换该资源的所有 ACL 条目
### 6.5 前端 ACL 管理入口
在数据集详情、模型详情、训练任务详情页面提供「资源授权」按钮,弹出 ACL 管理对话框:
- 显示当前 ACL 列表(主体类型 + 主体名称 + 权限勾选)
- 支持按用户或按角色添加授权
- 权限以多选框形式展示read / write / execute / download / delete
---
## 7. GPU 算力分配与隔离
### 7.1 设计背景
服务器可能安装多张 GPU 卡(如 8×A100需要精细化管控
- 管理员指定哪些用户可以使用哪些 GPU 卡
- 未被分配的 GPU 卡对用户不可见或不可选
- admin 可以使用全部 GPU
### 7.2 GPU 分配表
```sql
CREATE TABLE IF NOT EXISTS gpu_assignments (
id TEXT PRIMARY KEY,
gpu_id TEXT NOT NULL, -- gpus 表的外键node_id + gpu_index 组合)
node_id TEXT NOT NULL, -- 算力节点 ID
gpu_index INTEGER NOT NULL, -- GPU 卡序号
user_id TEXT NOT NULL, -- 被分配的用户 ID
assigned_by TEXT, -- 分配操作人 IDadmin
assigned_at TEXT NOT NULL, -- 分配时间
UNIQUE (node_id, gpu_index, user_id) -- 一张卡可分配给多个用户,但每对 (卡, 用户) 唯一
);
```
### 7.3 分配规则
| 规则 | 说明 |
|---|---|
| 谁可分配 | 只有 `admin` 角色可以分配 GPU |
| admin 使用 | admin 可使用全部 GPU不需要显式分配 |
| 普通用户 | 只能使用 `gpu_assignments` 中分配给自己的 GPU |
| 共享分配 | 一张 GPU 可分配给多个用户(非独占),但同时只能被一个任务占用 |
| 默认策略 | 新用户默认不分配任何 GPU由管理员显式分配 |
### 7.4 GPU 分配接口
| 接口 | 方法 | 权限 | 说明 |
|---|---|---|---|
| `/compute/gpu-assignments` | GET | admin | 查看全部分配关系 |
| `/compute/gpu-assignments` | POST | admin | 批量分配body: `{ assignments: [{ node_id, gpu_index, user_id }] }` |
| `/compute/gpu-assignments/{id}` | DELETE | admin | 撤销某条分配 |
| `/compute/my-gpus` | GET | 登录用户 | 查看自己可用的 GPU 列表 |
### 7.5 训练/评测/推理 GPU 选择校验
当普通用户创建训练任务、评测任务、推理任务并选择 GPU 时:
1. 后端检查 `gpu_assignments` 表,确认用户被分配了所选 GPU
2. 未被分配的 GPU → 返回 403 `"无权使用 GPU {node}:{index}"`
3. admin 用户跳过此校验
### 7.6 前端 GPU 选择交互
- 普通用户在创建任务选择 GPU 时,下拉列表只显示自己被分配的 GPU
- admin 用户在下拉列表中可看到全部 GPU
- 未分配任何 GPU 的用户GPU 选择区域显示提示:"未分配 GPU请联系管理员"
---
## 8. 审批拦截机制
### 8.1 需要审批的操作
| 操作 | 触发条件 | 审批动作 code |
|---|---|---|
| 删除他人数据集 | 非 admin 删除 `created_by != user.id` 的数据集 | `dataset.delete` |
| 删除他人模型 | 非 admin 删除 `created_by != user.id` 的模型 | `model.delete` |
| 停止他人训练任务 | 非 admin 停止 `created_by != user.id` 的任务 | `fine_tune.stop` |
| 发布模型到推理服务 | 任何用户(含 admin发布到生产环境 | `model_service.publish` |
| 删除项目空间 | 存在待审批变更时拒绝 | `project.delete` |
| 归档项目空间 | 存在待审批变更时拒绝 | `project.archive` |
| 导出训练产物 | 非 admin 导出他人训练的模型 | `trained_model.export` |
### 8.2 审批流程
```
普通用户发起高风险操作
┌──────────────┐ ┌──────────────────────┐
│ admin 旁路? │───是──▶│ 直接执行 + 审计日志 │
└──────┬───────┘ └──────────────────────┘
│ 否
┌──────────────────────┐
│ 创建审批实例 │
│ status=pending │
│ 返回 202待审批
└──────────┬───────────┘
┌──────────────────────┐
│ 管理员审批 │
│ POST /approvals/:id │
│ /steps/:idx/decision │
└──────────┬───────────┘
┌──────┴──────┐
│ │
approved rejected
│ │
▼ ▼
执行操作 不执行
+审计日志 +审计日志
```
### 8.3 审批模板
审批模板定义了特定操作需要几步审批、每步的审批人是谁:
```json
{
"id": "tpl_001",
"name": "删除数据集审批",
"action": "dataset.delete",
"steps": [
{ "approver_id": "u_admin", "step_name": "管理员审核" }
]
}
```
### 8.4 审批拦截点
在项目模块的 `_require_no_pending_approval` 函数中,当存在待审批实例时拒绝执行新操作。其他模块通过 `_require_approval_or_admin` 函数实现 admin 旁路或创建审批实例。
---
## 9. 审计日志
### 9.1 审计范围
所有写操作和敏感操作必须产生审计日志:
| 动作分类 | action 示例 |
|---|---|
| 用户管理 | `user.create`, `user.update`, `user.delete`, `user.permission.update` |
| 租户管理 | `tenant.create`, `tenant.update`, `tenant.quota.set`, `tenant.retention.set` |
| 项目管理 | `project.create`, `project.update`, `project.archive`, `project.delete`, `project.member.add`, `project.member.update`, `project.member.remove` |
| 资源 ACL | `resource.acl.set` |
| 模型管理 | `model.create`, `model.update`, `model.delete`, `model.merge` |
| 数据集 | `dataset.create`, `dataset.update`, `dataset.delete`, `dataset.upload` |
| 训练任务 | `fine_tune.create`, `fine_tune.start`, `fine_tune.stop`, `fine_tune.delete` |
| 评测任务 | `eval.create`, `eval.start`, `eval.stop` |
| 推理任务 | `inference.create`, `inference.start`, `inference.stop` |
| 审批 | `approval.create`, `approval.decide` |
| 留存策略 | `retention.create`, `retention.update`, `retention.delete` |
| GPU 分配 | `gpu.assign`, `gpu.unassign` |
### 9.2 审计日志字段
```sql
CREATE TABLE IF NOT EXISTS audit_logs (
id TEXT PRIMARY KEY,
time TEXT NOT NULL, -- ISO8601 时间戳
tenant_id TEXT, -- 租户 ID可选
project_id TEXT, -- 项目 ID可选
actor_id TEXT, -- 操作人 ID
action TEXT NOT NULL, -- 动作类型
target_type TEXT NOT NULL, -- 目标资源类型
target_id TEXT, -- 目标资源 ID
detail TEXT, -- 详情摘要
client_ip TEXT -- 客户端 IP
);
```
### 9.3 查询与导出
| 接口 | 方法 | 说明 |
|---|---|---|
| `/system/audit-logs` | GET | 分页查询,支持按 tenant_id / project_id / actor_id / action / target_type / start_time / end_time 筛选 |
| `/system/audit-logs/export` | GET | CSV 导出,与应用查询相同的过滤条件 |
---
## 10. 接口鉴权流程
### 10.1 Token 格式
```
Authorization: Bearer platform-token-{user_id}
```
登录成功后返回 `token``user` 信息。Token 中编码了 `user_id`,后端通过 `get_current_user` 解析。
### 10.2 鉴权层级
```
请求到达
├─ 1. 公开路径检查(/health, /login, /system-info→ 直接放行
├─ 2. Token 解析 → get_current_user
│ ├─ 无 token / token 无效 → 401
│ └─ 用户不存在 / 状态 disabled → 401
├─ 3. 页面级权限码检查(路由守卫 / Depends
│ └─ user.permissions 不含所需权限码 → 403
├─ 4. 资源级权限检查(路由函数内)
│ ├─ admin → 全部放行
│ ├─ resource.created_by == user.id → 放行
│ ├─ ACL 检查 has_resource_access() → 有授权则放行
│ └─ 否则 → 403
├─ 5. GPU 权限检查(训练/评测/推理创建时)
│ ├─ admin → 全部放行
│ ├─ gpu_assignments 检查 → 有分配则放行
│ └─ 否则 → 403
└─ 6. 审批拦截检查(高风险操作)
├─ admin → 旁路,直接执行
├─ 无待审批实例 → 可执行
├─ 有待审批实例 → 409 "存在待审批的变更"
└─ 需要审批 → 202 "已创建审批实例"
```
### 10.3 FastAPI 依赖注入
```python
# 任何需要登录的接口
@router.get("/datasets")
async def list_datasets(user: dict = Depends(get_current_user)):
...
# 需要管理员权限的接口
@router.post("/users")
async def create_user(user: dict = Depends(require_admin)):
...
# 需要资源级权限检查的接口
@router.delete("/datasets/{dataset_id}")
async def delete_dataset(
dataset_id: str,
user: dict = Depends(get_current_user),
):
if not has_resource_access("dataset", dataset_id, user, "delete"):
raise HTTPException(403, "forbidden")
...
```
---
## 11. 数据库表结构
### 11.1 现有表(已实现)
| 表名 | 用途 |
|---|---|
| `users` | 用户表id, username, password_hash, role, status, permissions, protected |
| `roles` | 角色定义name, permissions |
| `sessions` | 登录会话user_id, issued_at, expires_at, ip |
| `acls` | 资源访问控制列表resource_type, resource_id, principal_type, principal_id, permission |
| `audit_logs` | 审计日志actor_id, action, target_type, target_id, time |
| `datasets` | 数据集(需补充 `created_by` 字段) |
| `models` | 基座模型(需补充 `created_by` 字段) |
| `trained_models` | 训练产物(需补充 `created_by` 字段) |
| `fine_tune_tasks` | 训练任务payload 中存储 `created_by` |
| `gpus` | GPU 设备node_id, gpu_index, uuid, name, memory |
| `compute_nodes` | 算力节点 |
| `tenants` | 租户 |
| `projects` | 项目空间 |
| `project_members` | 项目成员 |
| `approval_templates` | 审批模板 |
| `approval_instances` | 审批实例 |
| `retention_policies` | 留存策略 |
### 11.2 需新增/补充的表和字段
#### 新增 `gpu_assignments` 表
```sql
CREATE TABLE IF NOT EXISTS gpu_assignments (
id TEXT PRIMARY KEY,
node_id TEXT NOT NULL REFERENCES compute_nodes(id) ON DELETE CASCADE,
gpu_index INTEGER NOT NULL,
user_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
assigned_by TEXT,
assigned_at TEXT NOT NULL,
UNIQUE (node_id, gpu_index, user_id)
);
CREATE INDEX IF NOT EXISTS idx_gpu_assignments_user ON gpu_assignments(user_id);
CREATE INDEX IF NOT EXISTS idx_gpu_assignments_gpu ON gpu_assignments(node_id, gpu_index);
```
#### 补充 `created_by` 字段
```sql
-- 数据集表补充所有者
ALTER TABLE datasets ADD COLUMN IF NOT EXISTS created_by TEXT;
-- 基座模型表补充所有者
ALTER TABLE models ADD COLUMN IF NOT EXISTS created_by TEXT;
-- 训练产物表补充所有者
ALTER TABLE trained_models ADD COLUMN IF NOT EXISTS created_by TEXT;
-- 评测任务表补充所有者
ALTER TABLE eval_tasks ADD COLUMN IF NOT EXISTS created_by TEXT;
-- 推理任务表补充所有者
-- 注意inference_tasks 表尚未创建,后续建表时直接包含 created_by 字段
```
---
## 12. API 接口清单
### 12.1 鉴权接口
| 接口 | 方法 | 鉴权 | 说明 |
|---|---|---|---|
| `/modelTF/login` | POST | 公开 | 登录,返回 token + user |
| `/modelTF/me` | GET | Bearer token | 获取当前用户信息 |
| `/modelTF/users` | GET | admin | 用户列表 |
| `/modelTF/users` | POST | admin | 创建用户 |
| `/modelTF/users/:id` | PUT | admin | 更新用户(角色/状态/权限) |
| `/modelTF/users/:id` | DELETE | admin | 删除用户protected 用户不可删) |
| `/modelTF/users/:id/reset-password` | POST | admin | 重置密码 |
| `/modelTF/system/permissions/codes` | GET | 登录 | 权限码清单 |
| `/modelTF/system/permissions` | GET | 登录 | 权限码 + 角色定义 |
### 12.2 资源 ACL 接口
| 接口 | 方法 | 鉴权 | 说明 |
|---|---|---|---|
| `/modelTF/resources/:type/:id/acl` | GET | admin 或所有者 | 查询资源 ACL |
| `/modelTF/resources/:type/:id/acl` | PUT | admin 或所有者 | 设置资源 ACL |
### 12.3 GPU 分配接口
| 接口 | 方法 | 鉴权 | 说明 |
|---|---|---|---|
| `/modelTF/compute/gpu-assignments` | GET | admin | 查看全部分配 |
| `/modelTF/compute/gpu-assignments` | POST | admin | 批量分配 |
| `/modelTF/compute/gpu-assignments/:id` | DELETE | admin | 撤销分配 |
| `/modelTF/compute/my-gpus` | GET | 登录 | 查看自己可用 GPU |
### 12.4 审批接口
| 接口 | 方法 | 鉴权 | 说明 |
|---|---|---|---|
| `/modelTF/approvals/templates` | GET/POST | admin | 审批模板列表/创建 |
| `/modelTF/approvals` | GET/POST | 登录 | 审批实例列表/创建 |
| `/modelTF/approvals/:id` | GET | 登录 | 审批实例详情 |
| `/modelTF/approvals/:id/steps/:idx/decision` | POST | 审批人 | 审批决策 |
### 12.5 审计接口
| 接口 | 方法 | 鉴权 | 说明 |
|---|---|---|---|
| `/modelTF/system/audit-logs` | GET | admin | 审计日志分页查询 |
| `/modelTF/system/audit-logs/export` | GET | admin | CSV 导出 |
### 12.6 租户/项目接口
| 接口 | 方法 | 鉴权 | 说明 |
|---|---|---|---|
| `/modelTF/tenants` | GET/POST | admin | 租户列表/创建 |
| `/modelTF/tenants/:id` | GET/PUT | admin | 租户详情/更新 |
| `/modelTF/tenants/:id/quota` | PUT | admin | 设置配额 |
| `/modelTF/tenants/:id/retention-policy` | PUT | admin | 绑定留存策略 |
| `/modelTF/projects` | GET/POST | 登录 | 项目列表/创建 |
| `/modelTF/projects/:id` | GET/PUT | 登录 | 项目详情/更新 |
| `/modelTF/projects/:id/archive` | POST | admin 或所有者 | 归档(审批拦截) |
| `/modelTF/projects/:id/members` | GET/POST | 登录 | 成员列表/添加 |
| `/modelTF/projects/:id/members/:uid` | PUT/DELETE | admin 或所有者 | 改角色/移除 |
### 12.7 留存策略接口
| 接口 | 方法 | 鉴权 | 说明 |
|---|---|---|---|
| `/modelTF/retention-policies` | GET/POST | admin | 策略列表/创建 |
| `/modelTF/retention-policies/:id` | GET/PUT/DELETE | admin | 策略详情/更新/删除 |
---
## 13. 前端权限控制
### 13.1 路由守卫
```typescript
// router/index.ts
router.beforeEach((to, _from, next) => {
const auth = useAuthStore()
auth.syncSession()
if (to.meta.public) {
next()
return
}
if (!auth.isLoggedIn) {
next({ name: 'login' })
return
}
if (!to.meta.skipPermission) {
const permission = requiredPermission(to.path, to.meta.permission)
if (permission && !auth.hasPermission(permission)) {
next({ name: 'permission-denied', replace: true })
return
}
}
next()
})
```
### 13.2 侧边栏过滤
```typescript
// layouts/MainLayout.vue
const visibleMenus = computed(() =>
allMenus.filter(menu => {
if (!menu.permission) return true
return auth.hasPermission(menu.permission)
})
)
```
### 13.3 资源级按钮控制
```vue
<!-- 数据集详情页 -->
<template>
<el-button v-if="canEdit" @click="handleEdit">编辑</el-button>
<el-button v-if="canDelete" @click="handleDelete">删除</el-button>
<el-button v-if="canManageAcl" @click="showAclDialog = true">资源授权</el-button>
</template>
<script setup>
const canEdit = computed(() =>
isAdmin.value || resource.value.created_by === userId.value
)
const canDelete = computed(() =>
isAdmin.value || resource.value.created_by === userId.value
)
const canManageAcl = computed(() =>
isAdmin.value || resource.value.created_by === userId.value
)
</script>
```
### 13.4 GPU 选择过滤
```vue
<!-- 创建训练任务页 -->
<template>
<el-select v-model="selectedGpus" multiple>
<el-option
v-for="gpu in availableGpus"
:key="gpu.id"
:label="`${gpu.node_name} GPU ${gpu.gpu_index}`"
:value="gpu.id"
/>
</el-select>
<el-alert v-if="availableGpus.length === 0 && !isAdmin" type="warning">
未分配 GPU请联系管理员
</el-alert>
</template>
<script setup>
// 普通用户只看到 my-gpus 返回的列表
// admin 看到全部 GPU
const availableGpus = ref([])
async function loadGpus() {
if (isAdmin.value) {
availableGpus.value = await getAllGpus()
} else {
availableGpus.value = await getMyGpus()
}
}
</script>
```
---
## 14. 安全设计补充
### 14.1 密码安全
- 密码使用 PBKDF2-SHA256 存储salt + 390000 次迭代)
- 旧系统明文密码在首次登录时自动升级为哈希
- 管理员可重置用户密码,用户不可自行修改密码(本期设计)
- 默认密码:`platform123`(创建用户时由管理员设定)
### 14.2 会话安全
| 规则 | 说明 |
|---|---|
| Token 格式 | `platform-token-{user_id}` |
| 会话超时 | 默认 30 分钟无操作自动过期 |
| 并发会话 | 同一用户可有多会话,各自独立计时 |
| 会话续期 | 前端定时调用 `auth.refresh()` 续期 |
| 强制下线 | admin 可通过修改用户 status=disabled 使其 token 失效 |
### 14.3 操作限流
| 接口 | 限制 |
|---|---|
| `/login` | 同一 IP 5 次/分钟,失败后 30 秒冷却 |
| 文件上传 | 单文件最大由配置控制,默认 2GB |
| 训练任务创建 | 同一用户并发运行任务数受 GPU 分配限制 |
### 14.4 数据安全
| 规则 | 说明 |
|---|---|
| 软删除 | 数据集、模型、任务使用 `deleted_at` 标记,保留审计可追溯 |
| 敏感字段 | API 密钥(`api_key`)在列表接口不返回明文 |
| 下载审计 | 数据集下载产生审计日志,记录下载人和时间 |
| 导出审计 | 训练产物导出产生审计日志 |
### 14.5 多租户隔离
| 规则 | 说明 |
|---|---|
| 租户隔离 | 同一租户内的资源相互可见;跨租户默认不可见 |
| 项目隔离 | 项目内资源受项目 ACL 控制;项目间默认不可见 |
| admin 旁路 | admin 可跨租户/项目访问全部资源 |
| 配额管控 | 租户级配额限制 GPU 并发数、存储容量、最大项目数 |
---
## 15. 实施计划
### 15.1 已实现
| 功能 | 状态 |
|---|---|
| 登录/会话/Token | ✅ 已实现 |
| 用户 CRUD + 权限码 | ✅ 已实现 |
| 角色定义 | ✅ 已实现 |
| 资源 ACLacls 表 + 接口) | ✅ 已实现 |
| 审计日志(查询 + 导出) | ✅ 已实现 |
| 审批模板/实例 | ✅ 已实现 |
| 项目空间 + 成员 | ✅ 已实现 |
| 租户 + 配额 + 留存 | ✅ 已实现 |
| 审批拦截(项目归档/删除) | ✅ 已实现 |
| 资源所有权 ACL 字段适配subject_type/permissions[] | ✅ 已实现 |
### 15.2 待实现
| 功能 | 优先级 | 涉及表/接口 |
|---|---|---|
| GPU 分配表 + 接口 | P0 | `gpu_assignments` 表 + `/compute/gpu-assignments` + `/compute/my-gpus` |
| 资源 `created_by` 字段补充 | P0 | `datasets` / `models` / `trained_models` / `eval_tasks` 表 ALTER |
| 资源列表按 `created_by` + ACL 过滤 | P0 | `platform_store.py` 中 datasets/models/tasks 列表方法 |
| GPU 选择校验(训练/评测/推理创建时) | P0 | `platform.py` 中 create_task/eval/inference |
| 前端 GPU 下拉过滤 | P1 | 前端创建任务页面 |
| 前端资源授权按钮 | P1 | 前端数据集/模型/任务详情页 |
| 前端权限管理页面优化 | P1 | 前端用户设置页面 |
| 审批拦截扩展(删除数据集/模型/停止任务) | P1 | `platform.py` 中 delete/stop 接口 |
| 密码安全策略(用户自行修改) | P2 | 新增 `/users/me/password` 接口 |
| 操作限流login 限流) | P2 | 中间件或 SlowAPI |
| 多租户隔离(按 tenant_id 过滤) | P2 | 各列表接口增加 tenant_id 过滤 |
### 15.3 实施步骤
1. **数据库迁移**:创建 `gpu_assignments` 表,为资源表补充 `created_by` 字段
2. **后端接口**:实现 GPU 分配 CRUD + `my-gpus` + 创建任务时的 GPU 权限校验
3. **资源过滤**:在 `datasets()` / `models()` / `tasks()` 等列表方法中按 `created_by` + ACL 过滤
4. **前端适配**GPU 下拉过滤、资源授权按钮、权限管理页面优化
5. **审批扩展**:在删除/停止接口中接入 `_require_approval_or_admin`
6. **测试补充**:扩展 `test_governance.py` 覆盖 GPU 分配、资源过滤、审批扩展场景

290
docs/security-hardening.md Normal file
View File

@@ -0,0 +1,290 @@
# 安全加固总结(前端 / 后端 / 算力节点)
> 记录 2026-08-06 对本平台的漏洞修复。核心目标:修复 **FastAPI 文档接口未授权访问**、
> **Swagger 泄露 API 结构**、以及两类**任意文件读取**漏洞(路径穿越 + 符号链接跟随),
> 修复过程不改变正常业务流程。
>
> 其中 **FastAPI 文档开关(`ENABLE_DOCS`** 的详细用法见
> [§4 FastAPI 文档开关使用说明](#4-fastapi-文档开关使用说明enabledocs)。
---
## 1. 漏洞总览
| # | 影响面 | 漏洞 | 风险等级 | 修复 |
|---|--------|------|----------|------|
| 1 | 后端 + 算力节点 | FastAPI 默认暴露 `/docs``/redoc``/openapi.json`**未授权**泄露全部 API 结构、参数、内部路由 | 中 | 生产环境关闭文档路由,访问返回 404`ENABLE_DOCS` 可覆盖) |
| 2 | 后端 | `data-convert` 模块 `output_filename` **路径穿越**:可任意文件读 / 写 / 删,且整个模块**无鉴权** | **严重** | 输出文件名白名单校验 + 全部端点补鉴权 |
| 3 | 算力节点 | `compute/files/{file_id}/download``file_id` 直接拼进 glob 模式可 `../` **穿越出上传目录**`FileResponse` 在 Linux 上**跟随符号链接**读取任意文件 | 中高 | `file_id` 字符白名单 + 解析后路径包含性二次校验 |
| 4 | 前端Vue + nginx | 无文件服务代码nginx 仅服务受控静态目录,无 `alias` | 无 | 审计确认,无需修复 |
---
## 2. 前端Vue 3 + nginx
**审计结论:不构成 ComfyUI `follow_symlinks` 类文件读取漏洞。**
- nginx`docker/nginx.conf.template`)只服务受控的 `dist/` 静态目录,使用 `try_files`
`alias` 指令、无用户可控文件路径,不存在路径穿越面。
- Vue SPA 自身没有任何文件服务逻辑;文件下载全部走后端/算力节点 API。
- 前端 axios 拦截器(`frontend/src/api/request.ts`)对**每个请求**自动附加
`Authorization: Bearer platform-token-{user_id}`,因此给后端接口补鉴权不会影响页面功能。
---
## 3. 后端FastAPI
### 3.1 data_convert 输出文件名路径穿越(严重)
**问题**`backend/app/modules/data_convert/router.py`
- `output_filename` 由请求体传入后**原样入库**,随后拼进
`output_dir / output_filename` 用于写/读/删:
- `if output_path.exists(): output_path.unlink()` → 任意文件删除
- `open(output_path, "a")` → 任意文件追加写
- `download_result``FileResponse(output_path)` → 任意文件读取
- 整个 router **无任何鉴权依赖**(后端无全局鉴权中间件),任意网络访问者可利用。
**修复**
```python
def _safe_output_filename(value: Any) -> str:
"""输出文件名白名单:拒绝 ../、/、\ 及控制字符,仅允许普通文件名。"""
name = str(value or "converted-data.jsonl").strip()
if (
not name
or name in {".", ".."}
or name != Path(name).name
or "/" in name
or "\\" in name
or any(ord(c) < 32 or ord(c) == 127 for c in name)
):
raise fail(400, "output filename must be a plain file name")
return name
def _task_output_path(task: dict[str, Any]) -> Path:
"""统一构造转换输出路径,始终位于任务 output 目录内。"""
return _output_dir(task["id"]) / _safe_output_filename(task.get("output_filename"))
```
- `create_task` 创建时即校验(恶意值直接 400
- 全部 4 处使用点(`upload_source_files` 自动转换、`run_convert``download_result`
`import_as_dataset`)统一改用 `_task_output_path()`,历史任务同样受保护
- 全部 **8 个** `/data-convert` 端点补充 `current_user: dict = Depends(get_current_user)` 鉴权
**功能影响**:正常转换流程(前端 `outputName + '.jsonl'` 这类纯文件名)不受影响;
接口现在要求登录态,未登录调用返回 401。
---
## 4. FastAPI 文档开关使用说明(`ENABLE_DOCS`
### 4.1 为什么需要这个开关
FastAPI 默认注册 3 个**无需鉴权**的路由,直接泄露全部 API 结构:
| 路由 | 说明 |
|------|------|
| `/docs` | Swagger UI 交互文档 |
| `/redoc` | ReDoc 文档 |
| `/openapi.json` | OpenAPI Schema含全部接口、参数、模型定义 |
修复方式是:**关闭时让 FastAPI 不注册这 3 个路由**,访问一律返回 404而不是返回空页面。
### 4.2 核心实现
关闭的本质是向 `FastAPI(...)` 传入三个 `None` 参数:
```python
# docs 关闭时等价于:
FastAPI(
title=...,
docs_url=None, # /docs → 404
redoc_url=None, # /redoc → 404
openapi_url=None, # /openapi.json → 404
)
```
### 4.3 后端开关逻辑(`backend/app/core/config.py` + `backend/app/main.py`
```python
# config.py —— Settings.enable_docs 在 __post_init__ 中计算
object.__setattr__(
self,
"enable_docs",
_bool_env("ENABLE_DOCS", os.getenv("APP_ENV", "local") != "prod"),
)
# config.py —— 返回传给 FastAPI 的文档参数
def docs_kwargs(enabled: bool) -> dict[str, Any]:
if enabled:
return {}
return {"docs_url": None, "redoc_url": None, "openapi_url": None}
# main.py —— 接入
app = FastAPI(title=settings.app_name, **docs_kwargs(settings.enable_docs))
```
**判定顺序(优先级从高到低)**
1. 显式设置 `ENABLE_DOCS=true/false` → 以显式值为准
2. 未设置 → `APP_ENV != "prod"` 时开放,`APP_ENV=prod` 时**关闭**
> 注意:`enable_docs` 从**运行时环境**读取 `APP_ENV`(而非类定义时缓存的默认值),
> 确保生产环境默认关闭始终生效且便于测试。
### 4.4 算力节点开关逻辑(`compute/api/security.py` + `compute/api/main.py`
```python
# security.py
def docs_enabled() -> bool:
raw = os.getenv("ENABLE_DOCS", "").strip().lower()
if raw in {"true", "false"}:
return raw == "true"
auth_enabled = os.getenv("COMPUTE_AUTH_ENABLED", "true").lower() == "true"
return not auth_enabled # 开启 token 鉴权(生产)时默认关闭文档
def docs_kwargs() -> dict[str, Any]:
if docs_enabled():
return {}
return {"docs_url": None, "redoc_url": None, "openapi_url": None}
# main.py
app = FastAPI(title="YG Fine-Tune Compute API", **docs_kwargs())
```
**判定顺序(优先级从高到低)**
1. 显式设置 `ENABLE_DOCS=true/false` → 以显式值为准
2. 未设置 → `COMPUTE_AUTH_ENABLED=true`(生产默认)时**关闭**
`COMPUTE_AUTH_ENABLED=false`(本地开发)时开放
### 4.5 环境变量速查表
| 服务 | 环境变量 | 取值 | 默认行为 |
|------|----------|------|----------|
| 后端 | `ENABLE_DOCS` | `true` / `false` | 未设置时按 `APP_ENV != "prod"` 判定 |
| 后端 | `APP_ENV` | `local` / `prod` 等 | `prod` 时关闭文档 |
| 算力节点 | `ENABLE_DOCS` | `true` / `false` | 未设置时按 `COMPUTE_AUTH_ENABLED` 判定 |
| 算力节点 | `COMPUTE_AUTH_ENABLED` | `true` / `false` | `true` 时关闭文档 |
### 4.6 Docker 部署配置
已在以下文件加入 `ENABLE_DOCS=false`,并通过 docker-compose 透传(默认 `false`
```
docker/app/.env → ENABLE_DOCS=false
docker/compute/.env → ENABLE_DOCS=false
docker/compute/.env.example → ENABLE_DOCS=false
docker/app/docker-compose.yml → ENABLE_DOCS: ${ENABLE_DOCS:-false}
docker/compute/docker-compose.yml → ENABLE_DOCS: ${ENABLE_DOCS:-false}
```
### 4.7 如何临时开启(排查/调试)
```bash
# 后端:非 prod 环境默认已开启prod 环境临时开启
ENABLE_DOCS=true docker compose -f docker/app/docker-compose.yml up -d backend-api
# 算力节点:临时开启(生产默认关闭)
ENABLE_DOCS=true docker compose -f docker/compute/docker-compose.yml up -d compute-api
```
> ⚠️ 仅在可信内网调试时开启,用毕改回 `false`。
### 4.8 验证方法
```bash
# 关闭状态下三个地址均应返回 404
curl -s -o /dev/null -w "%{http_code}\n" http://<host>/docs # 404
curl -s -o /dev/null -w "%{http_code}\n" http://<host>/redoc # 404
curl -s -o /dev/null -w "%{http_code}\n" http://<host>/openapi.json # 404
# 健康检查不受影响
curl -s http://<host>/modelTF/health
```
---
## 5. 算力节点FastAPI
### 5.1 文档开关
见 [§4.4](#44-算力节点开关逻辑computeapisecuritypy--computeapimainpy),逻辑与后端一致,
生产(`COMPUTE_AUTH_ENABLED=true`)默认关闭。
> 补充:原 token 鉴权中间件已覆盖全部非 health 路径;现在文档路由同时被 FastAPI 层关闭,
> 属于**纵深防御**(双重保护)。
### 5.2 `download_file` glob 穿越 + 符号链接跟随(`compute/api/main.py`
**问题**
```python
matches = list(upload_root.glob(f"{file_id}_*")) # file_id 来自 URL直接拼进 glob
return FileResponse(matches[0]) # 跟随符号链接
```
- `Path.glob` 支持 `..` 段,`file_id` 注入 `../` 可**穿越出 upload 目录**(已实测确认)
- Linux 上目录内符号链接可被 `FileResponse` 跟随 → 读取任意文件
**修复**
```python
# 1) file_id 字符白名单:仅字母/数字/_/-,含 .、/、% 等一律 400
if not file_id or not all(c.isalnum() or c in {"_", "-"} for c in file_id):
raise HTTPException(status_code=400, detail="invalid file id")
matches = list(upload_root.glob(f"{file_id}_*"))
if not matches:
raise HTTPException(status_code=404, detail="file not found")
# 2) 解析符号链接后必须仍位于 upload 根目录内
resolved = matches[0].resolve()
if not _path_inside(upload_root, resolved):
raise HTTPException(status_code=404, detail="file not found")
return FileResponse(resolved)
```
**功能影响**:服务端生成的 `file_<时间戳>` 格式完全兼容;外部工具使用正常 `file_id` 下载不受影响。
### 5.3 已确认安全的同类文件端点(无需改动)
| 端点 | 保护机制 |
|------|----------|
| `compute/files/list` | `_path_inside()` + `.resolve()`,符号链接逃逸被阻断 |
| `compute/files/read` | 同上 |
| `compute/files/upload` | 同上 + 文件名取 `.name` |
| `compute/files/import-local` | 目标路径 `_path_inside()` 校验 |
| 后端 `data-process` 存储 `LocalDataProcessStorage` | `lstat` + `S_ISLNK` + `O_NOFOLLOW` + 规范化引用校验,彻底防符号链接 |
---
## 6. 测试与验证结果
| 验证项 | 结果 |
|--------|------|
| 计算节点全量测试 | **22 passed, 1 skipped**(跳过项为 Windows 无权限建符号链接Linux 生产环境会执行) |
| 新增 compute 下载安全测试 | 正常下载 200穿越样本 400/404符号链接逃逸 404 |
| 新增后端 data_convert 安全测试 | **13 passed**(穿越样本 9 项全拦截 + 鉴权覆盖检查) |
| 后端文档开关测试 | 6 passed2 项 `create_app` 集成测试需完整依赖,在 WSL 下运行) |
| 实时验证 | 生产环境 `/docs` `/redoc` `/openapi.json` 均返回 **404**`download_file` 合法 `file_123456` 返回 200 |
---
## 7. 变更文件清单
**后端**
- `backend/app/core/config.py` — 新增 `_bool_env``docs_kwargs()``Settings.enable_docs`
- `backend/app/main.py``FastAPI(...)` 接入 `docs_kwargs`
- `backend/app/modules/data_convert/router.py` — 输出文件名白名单 + 全部端点补鉴权
**算力节点**
- `compute/api/security.py` — 新增文档开关模块(`docs_enabled` / `docs_kwargs`
- `compute/api/main.py` — 文档开关接入 + `download_file` 加固
**Docker 配置**
- `docker/app/.env``docker/compute/.env``docker/compute/.env.example``ENABLE_DOCS=false`
- `docker/app/docker-compose.yml``docker/compute/docker-compose.yml` — 透传 `ENABLE_DOCS`
**测试**
- `backend/tests/test_docs_security.py``backend/tests/test_data_convert_security.py`
- `compute/tests/test_security.py``compute/tests/test_file_download_security.py`

View File

@@ -101,7 +101,9 @@ assert.match(detailSource, /!\/\(\?:password\|secret\|token\|api_key\)\/i\.test\
assert.match(detailSource, /key !== 'generation_model_snapshot'/, '处理配置仍直接展示内部模型快照') assert.match(detailSource, /key !== 'generation_model_snapshot'/, '处理配置仍直接展示内部模型快照')
assert.match(detailSource, /preprocessOptionLabelMap/, '处理配置没有把预处理内部枚举转换为中文') assert.match(detailSource, /preprocessOptionLabelMap/, '处理配置没有把预处理内部枚举转换为中文')
assert.match(detailSource, /output_type:\s*'输出类型'/, '处理配置没有显示输出类型名称') assert.match(detailSource, /output_type:\s*'输出类型'/, '处理配置没有显示输出类型名称')
assert.match(detailSource, /value === 'reasoning' \? '思维链回答' : '标准回答'/, '处理配置没有转换输出类型枚举') assert.match(detailSource, /value === 'reasoning' \? '思维链回答' : value === 'dpo' \? 'DPO 偏好对' : '标准回答'/, '处理配置没有转换输出类型枚举')
assert.match(detailSource, /prop="chosen"[\s\S]*?prop="rejected"/, 'DPO 结果没有展示 Chosen 与 Rejected')
assert.match(detailSource, /publishForm\.format = isDpoOutput\.value \? 'dpo'/, 'DPO 发布没有锁定原生格式')
assert.match(detailSource, /reasoning_detail:\s*'推理详细程度'/, '处理配置没有显示推理详细程度名称') assert.match(detailSource, /reasoning_detail:\s*'推理详细程度'/, '处理配置没有显示推理详细程度名称')
assert.match(detailSource, /value === 'detailed' \? '详细推理' : '普通推理'/, '处理配置没有转换推理详细程度枚举') assert.match(detailSource, /value === 'detailed' \? '详细推理' : '普通推理'/, '处理配置没有转换推理详细程度枚举')
assert.match(detailSource, /key !== 'reasoning_detail' \|\| config\.output_type === 'reasoning'/, '标准回答任务不应展示无关的推理详细程度') assert.match(detailSource, /key !== 'reasoning_detail' \|\| config\.output_type === 'reasoning'/, '标准回答任务不应展示无关的推理详细程度')

View File

@@ -122,13 +122,13 @@ assert.match(regenerationSource, /route\.name === 'data-process-regenerate'[\s\S
assert.match(viewSource, /const currentStep = ref\(0\)/, '重新生成必须从向导第一步开始') assert.match(viewSource, /const currentStep = ref\(0\)/, '重新生成必须从向导第一步开始')
assert.match(viewSource, /const WIZARD_STEPS = \[/, '向导步骤尚未改为固定常量') assert.match(viewSource, /const WIZARD_STEPS = \[/, '向导步骤尚未改为固定常量')
for (const title of ['创建任务', '大模型选择', '上传文件', '数据预览', '开始生成', '结果编辑与保存']) { for (const title of ['创建任务', '大模型选择', '数据来源', '数据预览', '开始生成', '结果编辑与保存']) {
assert.ok(viewSource.includes(`title: '${title}'`), `缺少固定步骤:${title}`) assert.ok(viewSource.includes(`title: '${title}'`), `缺少固定步骤:${title}`)
} }
assert.match( assert.match(
viewSource, viewSource,
/\{ id: 'create',[\s\S]*?\{ id: 'model',[\s\S]*?\{ id: 'upload',[\s\S]*?\{ id: 'preview',[\s\S]*?\{ id: 'generate',[\s\S]*?\{ id: 'results'/, /\{ id: 'create',[\s\S]*?\{ id: 'model',[\s\S]*?\{ id: 'upload',[\s\S]*?\{ id: 'preview',[\s\S]*?\{ id: 'generate',[\s\S]*?\{ id: 'results'/,
'六步向导顺序必须为创建任务、大模型选择、上传文件、数据预览、开始生成、结果编辑与保存', '六步向导顺序必须为创建任务、大模型选择、数据来源、数据预览、开始生成、结果编辑与保存',
) )
assert.doesNotMatch(viewSource, /steps\s*=\s*computed|all\.filter/, '步骤仍根据处理类型动态增减') assert.doesNotMatch(viewSource, /steps\s*=\s*computed|all\.filter/, '步骤仍根据处理类型动态增减')
assert.match( assert.match(
@@ -380,7 +380,7 @@ for (const marker of ['<el-upload', '源数据上传', '数据源配置', 'uploa
assert.match(viewSource, /<ModelSelectionStep\s+[\s\S]*?v-else-if="currentStepId === 'model'"/, '第二步没有挂载独立大模型选择组件') assert.match(viewSource, /<ModelSelectionStep\s+[\s\S]*?v-else-if="currentStepId === 'model'"/, '第二步没有挂载独立大模型选择组件')
assert.match(viewSource, /<SourceUploadStep\s+[\s\S]*?v-else-if="currentStepId === 'upload'"/, '第三步没有挂载独立上传组件') assert.match(viewSource, /<SourceUploadStep\s+[\s\S]*?v-else-if="currentStepId === 'upload'"/, '第三步没有挂载独立上传组件')
assert.match(viewSource, /if \(currentStepId\.value === 'create'\) return '继续:选择大模型'/, '第一步主按钮没有指向大模型选择') assert.match(viewSource, /if \(currentStepId\.value === 'create'\) return '继续:选择大模型'/, '第一步主按钮没有指向大模型选择')
assert.match(viewSource, /if \(currentStepId\.value === 'model'\) return '继续:上传文件'/, '第二步主按钮没有指向上传文件') assert.match(viewSource, /if \(currentStepId\.value === 'model'\) return '继续:选择数据来源'/, '第二步主按钮没有指向数据来源选择')
assert.match( assert.match(
viewSource, viewSource,
/if \(currentStepId\.value === 'upload'\) \{[\s\S]*?sourceUploading\.value[\s\S]*?'正在上传'[\s\S]*?previewBuilding\.value \? '正在切分' : '继续:数据预览'/, /if \(currentStepId\.value === 'upload'\) \{[\s\S]*?sourceUploading\.value[\s\S]*?'正在上传'[\s\S]*?previewBuilding\.value \? '正在切分' : '继续:数据预览'/,
@@ -1032,7 +1032,7 @@ assert.match(stateSource, /Object\.prototype\.hasOwnProperty\.call\(config, key\
assert.match(stateSource, /Number\.isFinite\(value\) \? value : fallback/, '配置反向映射没有保留合法数字 0') assert.match(stateSource, /Number\.isFinite\(value\) \? value : fallback/, '配置反向映射没有保留合法数字 0')
assert.match(stateSource, /qaPairsPerRow:\s*normalizeQaPairsGenerationCount\([\s\S]*?qa_pairs_per_row[\s\S]*?defaults\.qaPairsPerRow/, '结构化生成数量回填没有按 1 到 50 归一化') assert.match(stateSource, /qaPairsPerRow:\s*normalizeQaPairsGenerationCount\([\s\S]*?qa_pairs_per_row[\s\S]*?defaults\.qaPairsPerRow/, '结构化生成数量回填没有按 1 到 50 归一化')
assert.match(stateSource, /qaPairsPerChunk:\s*normalizeQaPairsGenerationCount\([\s\S]*?qa_pairs_per_chunk[\s\S]*?defaults\.qaPairsPerChunk/, '非结构化生成数量回填没有按 1 到 50 归一化') assert.match(stateSource, /qaPairsPerChunk:\s*normalizeQaPairsGenerationCount\([\s\S]*?qa_pairs_per_chunk[\s\S]*?defaults\.qaPairsPerChunk/, '非结构化生成数量回填没有按 1 到 50 归一化')
assert.match(stateSource, /const outputType = configValue\(config, 'output_type', defaults\.outputType\) === 'reasoning'[\s\S]*?\? 'reasoning'[\s\S]*?: 'standard'/, '输出类型没有从任务配置安全回填') assert.match(stateSource, /configuredOutputType === 'reasoning' \|\| configuredOutputType === 'dpo'[\s\S]*?\? configuredOutputType[\s\S]*?: 'standard'/, '输出类型没有从任务配置安全回填')
assert.match(stateSource, /reasoningDetail:\s*configValue\(config, 'reasoning_detail', defaults\.reasoningDetail\) === 'detailed'[\s\S]*?\? 'detailed'[\s\S]*?: 'normal'/, '推理详细程度没有从任务配置安全回填') assert.match(stateSource, /reasoningDetail:\s*configValue\(config, 'reasoning_detail', defaults\.reasoningDetail\) === 'detailed'[\s\S]*?\? 'detailed'[\s\S]*?: 'normal'/, '推理详细程度没有从任务配置安全回填')
assert.match(stateSource, /isBuiltInGenerationPrompt\(configuredPrompt\)[\s\S]*?defaultGenerationPrompt\(outputType\)/, '旧版内置提示语没有按输出类型迁移') assert.match(stateSource, /isBuiltInGenerationPrompt\(configuredPrompt\)[\s\S]*?defaultGenerationPrompt\(outputType\)/, '旧版内置提示语没有按输出类型迁移')
assert.match(stateSource, /createStructuredOptionsFromConfig/, '结构化配置缺少后端到表单的反向映射') assert.match(stateSource, /createStructuredOptionsFromConfig/, '结构化配置缺少后端到表单的反向映射')
@@ -1412,7 +1412,7 @@ assert.match(
) )
assert.match( assert.match(
sourceUploadSource, sourceUploadSource,
/<el-upload\s+v-if="uploadedFiles\.length === 0"\s+drag\s+multiple\s+:accept="uploadAccept"\s+:disabled="previewBuilding"\s+:auto-upload="false"\s+:show-file-list="false"\s+:on-change="\(file: UploadFile\) => emit\('file-change', file\)"[^>]*>/, /<el-upload\s+v-if="uploadedFiles\.length === 0"\s+drag\s+multiple\s+:accept="uploadAccept"\s+:disabled="previewBuilding \|\| externalPulling"\s+:auto-upload="false"\s+:show-file-list="false"\s+:on-change="\(file: UploadFile\) => emit\('file-change', file\)"[^>]*>/,
'无文件时未保留原有大拖拽上传区或上传配置', '无文件时未保留原有大拖拽上传区或上传配置',
) )
assert.match( assert.match(
@@ -1432,7 +1432,7 @@ assert.ok(sourceUploadSource.includes('旧版 DOC/PPT 请先转换'), '非结构
assert.ok(sourceUploadSource.includes('旧版 XLS 请先转换'), '结构化格式提示没有说明旧版 XLS 需转换') assert.ok(sourceUploadSource.includes('旧版 XLS 请先转换'), '结构化格式提示没有说明旧版 XLS 需转换')
assert.match( assert.match(
sourceUploadSource, sourceUploadSource,
/<div\s+class="uploaded-file-list-header">[\s\S]*?已选择 \{\{ uploadedFiles\.length \}\} 个文件[\s\S]*?<div\s+class="continue-upload">\s*<el-upload\s+multiple\s+:accept="uploadAccept"\s+:disabled="previewBuilding"\s+:auto-upload="false"\s+:show-file-list="false"\s+:on-change="\(file: UploadFile\) => emit\('file-change', file\)"[^>]*>\s*<el-button\s+size="small"\s+type="primary"\s+:disabled="previewBuilding">继续上传<\/el-button>\s*<\/el-upload>\s*<\/div>\s*<\/div>/, /<div\s+class="uploaded-file-list-header">[\s\S]*?已选择 \{\{ uploadedFiles\.length \}\} 个文件[\s\S]*?<div\s+class="continue-upload">\s*<el-upload\s+multiple\s+:accept="uploadAccept"\s+:disabled="previewBuilding \|\| externalPulling"\s+:auto-upload="false"\s+:show-file-list="false"\s+:on-change="\(file: UploadFile\) => emit\('file-change', file\)"[^>]*>\s*<el-button\s+size="small"\s+type="primary"\s+:disabled="previewBuilding \|\| externalPulling">继续上传<\/el-button>\s*<\/el-upload>\s*<\/div>\s*<\/div>/,
'有文件时缺少标题右侧的继续上传触发器或上传配置', '有文件时缺少标题右侧的继续上传触发器或上传配置',
) )
assert.match( assert.match(
@@ -1476,4 +1476,13 @@ assert.match(
) )
assert.match(previewSource, /height:\s*clamp\(560px,\s*calc\(100vh - 370px\),\s*720px\)/, '对照预览高度不足以展示切片正文') assert.match(previewSource, /height:\s*clamp\(560px,\s*calc\(100vh - 370px\),\s*720px\)/, '对照预览高度不足以展示切片正文')
assert.doesNotMatch(taskSetupSource, /外来数据源拉取/, '外部数据源仍错误地放在第一步处理类型中')
assert.match(sourceUploadSource, /<h3>数据来源<\/h3>[\s\S]*?本地上传[\s\S]*?外部数据源/, '第三步缺少本地与外部数据来源选择')
for (const field of ['地址 / URL', '鉴权方式', 'SSL 模式', '连接超时', '查询超时', '只读查询语句', '拉取条数', '落地文件名']) {
assert.ok(sourceUploadSource.includes(field), `外部数据源标准配置缺少:${field}`)
}
assert.match(generationControlSource, /label="DPO 偏好对" value="dpo"/, '输出类型缺少 DPO 偏好对')
assert.match(resultEditorSource, /Chosen[\s\S]*?Rejected/, '结果编辑器缺少 DPO 成对字段')
assert.match(viewSource, /sourceConfigForBackend\(sourceMode\.value, externalSource\)/, '任务配置没有保存第三步数据来源模式')
console.log('数据处理六步向导回归检查通过') console.log('数据处理六步向导回归检查通过')

View File

@@ -0,0 +1,5 @@
import { post } from '../request'
/** 记录用户访问某个业务模块(用于看板用户操作分布统计) */
export const recordModuleVisit = (module: string, detail?: string) =>
post('/system/audit/visit', { action: module, detail: detail || '' })

View File

@@ -0,0 +1,43 @@
import { del, get, post } from '../request'
export interface DataConvertTask {
id: string
name: string
description: string
status: string
output_filename: string
input_count: number
output_count: number
error_message: string
create_time: string
update_time?: string
input_files?: Array<{ name: string; size: number }>
}
export const getDataConvertTasks = (page = 1, pageSize = 20) =>
get<{ items: DataConvertTask[]; total: number }>('/data-convert', { params: { page, page_size: pageSize } })
export const getDataConvertTask = (id: string) =>
get<DataConvertTask>(`/data-convert/${id}`)
export const createDataConvertTask = (payload: Partial<DataConvertTask>) =>
post<DataConvertTask>('/data-convert', payload)
export const uploadSourceFiles = (id: string, files: File[]) => {
const formData = new FormData()
files.forEach(f => formData.append('files', f))
return post(`/data-convert/${id}/source-files`, formData)
}
export const runConvert = (id: string) =>
post<DataConvertTask>(`/data-convert/${id}/run`)
export const downloadResult = (id: string) =>
get(`/data-convert/${id}/download`, { responseType: 'blob' })
export const deleteDataConvertTask = (id: string) =>
del(`/data-convert/${id}`)
/** 把转换结果导入为数据集管理中的上传任务记录 */
export const importAsDataset = (id: string, payload: { name?: string; description?: string } = {}) =>
post(`/data-convert/${id}/import-as-dataset`, payload)

View File

@@ -0,0 +1,38 @@
import { del, get, post } from '../request'
export interface GpuAssignment {
id: string
node_id: string
gpu_index: number
user_id: string
assigned_by?: string | null
assigned_at?: string
username?: string
display_name?: string
node_code?: string
node_name?: string
gpu_name?: string
}
export interface MyGpu {
node_id: string
gpu_index: number
node_code: string
node_name: string
gpu_name?: string
uuid?: string
memory_total_gb?: number
}
/** 查看全部分配关系admin */
export const getGpuAssignments = () => get<GpuAssignment[]>('/compute/gpu-assignments')
/** 批量分配 GPUadmin */
export const assignGpus = (assignments: Array<{ node_id: string; gpu_index: number; user_id: string }>) =>
post<GpuAssignment[]>('/compute/gpu-assignments', { assignments })
/** 撤销 GPU 分配admin */
export const unassignGpu = (id: string) => del(`/compute/gpu-assignments/${id}`)
/** 查看当前用户可用的 GPU 列表 */
export const getMyGpus = () => get<MyGpu[]>('/compute/my-gpus')

View File

@@ -100,3 +100,14 @@ export const mergeModel = (data: {
/** 导出已训练模型权重 */ /** 导出已训练模型权重 */
export const exportModelUrl = (modelName: string) => export const exportModelUrl = (modelName: string) =>
`/modelTF/model-manage/trained-models/${encodeURIComponent(modelName)}/export` `/modelTF/model-manage/trained-models/${encodeURIComponent(modelName)}/export`
/** 测试在线模型连通性 */
export const testOnlineModel = (data: {
api_url: string
api_key: string
online_model_name: string
}) =>
post<{ success: boolean; error?: string; model?: string; usage?: object }>(
'/model-manage/test-online',
data,
)

View File

@@ -42,3 +42,10 @@ export const resetUserPassword = (id: string, password?: string) =>
/** 删除用户protected 管理员账号不允许删除) */ /** 删除用户protected 管理员账号不允许删除) */
export const deleteUser = (id: string) => export const deleteUser = (id: string) =>
del<{ deleted: string }>(`/users/${encodeURIComponent(id)}`) del<{ deleted: string }>(`/users/${encodeURIComponent(id)}`)
/** 用户自行修改密码 */
export const changeMyPassword = (oldPassword: string, newPassword: string) =>
post<{ changed: boolean }>('/users/me/password', {
old_password: oldPassword,
new_password: newPassword,
})

View File

@@ -1,6 +1,38 @@
import axios, { type AxiosInstance, type AxiosRequestConfig } from 'axios' import axios, { type AxiosInstance, type AxiosRequestConfig } from 'axios'
import { ElMessage } from 'element-plus' import { ElMessage } from 'element-plus'
/**
* 用户操作分布:哪些模块路径算"业务操作"(用于看板统计)
* 请求命中这些路径时,会自动调用 record_visit 记录一次(同一模块 60 秒内去重)
*/
const VISIT_TRACKED_PREFIXES: Array<[string, string]> = [
['/fine-tune', 'fine-tune'],
['/model-eval', 'model-eval'],
['/model-inference', 'model-inference'],
['/model-compare', 'model-inference'],
['/data-process', 'data-process'],
['/data-convert', 'data-convert'],
['/model-manage', 'model-manage'],
['/dataset-manage', 'dataset'],
]
function trackVisit(url: string | undefined) {
if (!url) return
for (const [prefix, module] of VISIT_TRACKED_PREFIXES) {
if (url.includes(prefix)) {
const key = `visit:${module}`
const last = Number(sessionStorage.getItem(key) || 0)
if (Date.now() - last < 60000) return // 60 秒内去重
sessionStorage.setItem(key, String(Date.now()))
// fire-and-forget 调用后端记录接口
import('./modules/audit-visit').then(({ recordModuleVisit }) => {
recordModuleVisit(module, url).catch(() => { /* ignore */ })
}).catch(() => { /* ignore */ })
return
}
}
}
/** /**
* 后端统一响应格式 * 后端统一响应格式
* code === 0 表示成功data 为业务数据 * code === 0 表示成功data 为业务数据

View File

@@ -98,7 +98,14 @@ const visibleMenuGroups = computed(() =>
menuGroups menuGroups
.map((group) => ({ .map((group) => ({
...group, ...group,
items: group.items.filter((item) => auth.hasPermission(item.permission)), items: group.items.filter((item) => {
if (!auth.hasPermission(item.permission)) return false
// user-settings 权限对应的菜单仅管理员可见
if (item.permission === 'user-settings' && !auth.isAdmin) return false
// 算力节点仅管理员可见
if (item.permission === 'compute' && !auth.isAdmin) return false
return true
}),
})) }))
.filter((group) => group.items.length > 0), .filter((group) => group.items.length > 0),
) )

View File

@@ -388,6 +388,41 @@ router.beforeEach((to, _from, next) => {
next({ name: 'permission-denied', replace: true }) next({ name: 'permission-denied', replace: true })
return return
} }
// user-settings 权限对应的页面仅管理员可访问
if (permission === 'user-settings' && !auth.isAdmin) {
next({ name: 'permission-denied', replace: true })
return
}
// 算力节点仅管理员可访问
if (permission === 'compute' && !auth.isAdmin) {
next({ name: 'permission-denied', replace: true })
return
}
}
// 路由切换时记录业务模块访问(用于看板用户操作分布统计)
const ROUTE_TO_MODULE: Record<string, string> = {
'/fine-tune': 'fine-tune',
'/model-eval': 'model-eval',
'/model-inference': 'model-inference',
'/model-compare': 'model-inference',
'/data-process': 'data-process',
'/data-convert': 'data-convert',
'/model-manage': 'model-manage',
'/dataset-manage': 'dataset',
}
for (const [prefix, module] of Object.entries(ROUTE_TO_MODULE)) {
if (to.path.startsWith(prefix)) {
const key = `route-visit:${module}`
const last = Number(sessionStorage.getItem(key) || 0)
if (Date.now() - last >= 60000) {
sessionStorage.setItem(key, String(Date.now()))
import('@/api/modules/audit-visit').then(({ recordModuleVisit }) => {
recordModuleVisit(module, to.fullPath).catch(() => {})
}).catch(() => {})
}
break
}
} }
next() next()

View File

@@ -5,6 +5,7 @@ import type { PermissionCode, SystemUser } from '@/types'
const USER_STORAGE_KEY = 'currentUser' const USER_STORAGE_KEY = 'currentUser'
const SESSION_STORAGE_KEY = 'sessionId' const SESSION_STORAGE_KEY = 'sessionId'
const AUTHED_FLAG_KEY = 'hasAuthed'
const allPermissions: PermissionCode[] = [ const allPermissions: PermissionCode[] = [
'dashboard', 'dashboard',
@@ -21,25 +22,24 @@ const allPermissions: PermissionCode[] = [
'user-settings', 'user-settings',
] ]
function restoreUser(): SystemUser | null { /**
const persisted = localStorage.getItem(USER_STORAGE_KEY) * 认证 store
if (persisted) { * - 用户信息持久化到 localStorage刷新页面不丢失
* - 但每次新开标签页/窗口必须重新点击登录sessionStorage 标记本次会话已登录)
*/
export const useAuthStore = defineStore('auth', () => {
// 仅当 sessionStorage 里标记了"已登录"时,才从 localStorage 恢复用户
const hasAuthed = sessionStorage.getItem(AUTHED_FLAG_KEY) === '1'
const persistedUser = hasAuthed ? localStorage.getItem(USER_STORAGE_KEY) : null
let initialUser: SystemUser | null = null
if (persistedUser) {
try { try {
return JSON.parse(persisted) as SystemUser initialUser = JSON.parse(persistedUser) as SystemUser
} catch { } catch {
localStorage.removeItem(USER_STORAGE_KEY) localStorage.removeItem(USER_STORAGE_KEY)
} }
} }
return null const currentUser = ref<SystemUser | null>(initialUser)
}
/**
* 认证 store
* 登录态管理:有 currentUser 即视为已登录。
* 离开页面超时由 App.vue 的 visibilitychange 监听接管。
*/
export const useAuthStore = defineStore('auth', () => {
const currentUser = ref<SystemUser | null>(restoreUser())
const username = computed(() => currentUser.value?.username || '') const username = computed(() => currentUser.value?.username || '')
const displayName = computed(() => currentUser.value?.display_name || username.value) const displayName = computed(() => currentUser.value?.display_name || username.value)
const roleLabel = computed(() => { const roleLabel = computed(() => {
@@ -49,15 +49,16 @@ export const useAuthStore = defineStore('auth', () => {
}) })
const isLoggedIn = computed(() => currentUser.value !== null) const isLoggedIn = computed(() => currentUser.value !== null)
const isAdmin = computed(() => currentUser.value?.role === 'admin')
/** 登录 */ /** 登录 */
async function login(user: string, password: string) { async function login(user: string, password: string) {
const response = await loginApi(user, password) const response = await loginApi(user, password)
currentUser.value = response.user currentUser.value = response.user
localStorage.setItem('username', response.user.username)
localStorage.setItem(USER_STORAGE_KEY, JSON.stringify(response.user)) localStorage.setItem(USER_STORAGE_KEY, JSON.stringify(response.user))
sessionStorage.setItem(AUTHED_FLAG_KEY, '1')
if (response.session_id) { if (response.session_id) {
localStorage.setItem(SESSION_STORAGE_KEY, response.session_id) sessionStorage.setItem(SESSION_STORAGE_KEY, response.session_id)
} }
} }
@@ -69,14 +70,14 @@ export const useAuthStore = defineStore('auth', () => {
/** 退出 */ /** 退出 */
async function logout() { async function logout() {
const sessionId = localStorage.getItem(SESSION_STORAGE_KEY) const sessionId = sessionStorage.getItem(SESSION_STORAGE_KEY)
if (sessionId) { if (sessionId) {
try { await logoutApi(sessionId) } catch { /* 静默 */ } try { await logoutApi(sessionId) } catch { /* 静默 */ }
} }
currentUser.value = null currentUser.value = null
localStorage.removeItem('username')
localStorage.removeItem(USER_STORAGE_KEY) localStorage.removeItem(USER_STORAGE_KEY)
localStorage.removeItem(SESSION_STORAGE_KEY) sessionStorage.removeItem(AUTHED_FLAG_KEY)
sessionStorage.removeItem(SESSION_STORAGE_KEY)
} }
return { return {
@@ -85,6 +86,7 @@ export const useAuthStore = defineStore('auth', () => {
displayName, displayName,
roleLabel, roleLabel,
isLoggedIn, isLoggedIn,
isAdmin,
hasPermission, hasPermission,
login, login,
logout, logout,

View File

@@ -4,7 +4,7 @@ export type DataProcessStatus = 'pending' | 'running' | 'completed' | 'failed' |
export type DataProcessType = 'structured' | 'unstructured' | 'external' export type DataProcessType = 'structured' | 'unstructured' | 'external'
export type DataProcessResultStatus = 'valid' | 'modified' | 'invalid' export type DataProcessResultStatus = 'valid' | 'modified' | 'invalid'
export type DataProcessSplit = 'train' | 'validation' | 'test' export type DataProcessSplit = 'train' | 'validation' | 'test'
export type DataProcessOutputType = 'standard' | 'reasoning' export type DataProcessOutputType = 'standard' | 'reasoning' | 'dpo'
export type DataProcessReasoningDetail = 'normal' | 'detailed' export type DataProcessReasoningDetail = 'normal' | 'detailed'
export type DataProcessWorkflowStep = 'create' | 'model' | 'upload' | 'preview' | 'generate' | 'results' export type DataProcessWorkflowStep = 'create' | 'model' | 'upload' | 'preview' | 'generate' | 'results'
export type DataProcessPreviewStatus = 'idle' | 'queued' | 'running' | 'completed' | 'failed' | 'cancelled' export type DataProcessPreviewStatus = 'idle' | 'queued' | 'running' | 'completed' | 'failed' | 'cancelled'
@@ -221,6 +221,9 @@ export interface DataProcessExternalSourcePayload {
username?: string username?: string
password?: string password?: string
limit: number limit: number
connect_timeout_seconds: number
statement_timeout_seconds: number
ssl_mode: 'disable' | 'prefer' | 'require' | 'verify-ca' | 'verify-full'
query?: string query?: string
file_name?: string file_name?: string
} }
@@ -342,9 +345,13 @@ export interface DataProcessResult {
instruction: string instruction: string
input: string input: string
output: string output: string
chosen?: string
rejected?: string
original_instruction?: string | null original_instruction?: string | null
original_input?: string | null original_input?: string | null
original_output?: string | null original_output?: string | null
original_chosen?: string | null
original_rejected?: string | null
status: DataProcessResultStatus status: DataProcessResultStatus
error?: string | null error?: string | null
split?: DataProcessSplit | null split?: DataProcessSplit | null
@@ -356,6 +363,8 @@ export interface DataProcessResultUpdatePayload {
instruction: string instruction: string
input: string input: string
output: string output: string
chosen?: string
rejected?: string
expected_updated_at?: string expected_updated_at?: string
} }
@@ -410,7 +419,7 @@ export interface DataProcessPublishPayload {
dataset_type: 'train' | 'test' | 'eval' | 'val' | 'other' dataset_type: 'train' | 'test' | 'eval' | 'val' | 'other'
storage_type: 'local' storage_type: 'local'
split: DataProcessDatasetSplit split: DataProcessDatasetSplit
format: 'alpaca_jsonl' | 'jsonl' format: 'alpaca_jsonl' | 'jsonl' | 'dpo'
} }
export interface DataProcessPublishResult { export interface DataProcessPublishResult {

View File

@@ -22,6 +22,9 @@ import {
type ResourceReplica, type ResourceReplica,
type ResourceSyncJob, type ResourceSyncJob,
} from '@/api/modules/compute' } from '@/api/modules/compute'
import { getGpuAssignments, assignGpus, unassignGpu, type GpuAssignment } from '@/api/modules/gpu'
import { getUsers, type SystemUser } from '@/api/modules/system'
import { useAuthStore } from '@/stores/auth'
import { statusLabel, statusTagType } from '@/utils/status' import { statusLabel, statusTagType } from '@/utils/status'
const route = useRoute() const route = useRoute()
@@ -37,6 +40,52 @@ const gpus = ref<ComputeGpu[]>([])
const queue = ref<ComputeQueueItem[]>([]) const queue = ref<ComputeQueueItem[]>([])
const replicas = ref<ResourceReplica[]>([]) const replicas = ref<ResourceReplica[]>([])
const activeSyncJob = ref<ResourceSyncJob | null>(null) const activeSyncJob = ref<ResourceSyncJob | null>(null)
// GPU 分配管理
const auth = useAuthStore()
const gpuAssignments = ref<GpuAssignment[]>([])
const assignmentLoading = ref(false)
const showAssignDialog = ref(false)
const assignForm = ref({ node_id: '', gpu_index: 0, user_id: '' })
const allUsers = ref<SystemUser[]>([])
async function loadGpuAssignments() {
if (!auth.isAdmin) return
assignmentLoading.value = true
try {
gpuAssignments.value = await getGpuAssignments()
} catch {
gpuAssignments.value = []
} finally {
assignmentLoading.value = false
}
}
async function handleAssign() {
if (!assignForm.value.node_id || !assignForm.value.user_id) {
ElMessage.warning('请选择节点和用户')
return
}
try {
await assignGpus([{ node_id: assignForm.value.node_id, gpu_index: assignForm.value.gpu_index, user_id: assignForm.value.user_id }])
ElMessage.success('GPU 分配成功')
showAssignDialog.value = false
await loadGpuAssignments()
} catch {
ElMessage.error('GPU 分配失败')
}
}
async function handleUnassign(id: string) {
try {
await ElMessageBox.confirm('确认撤销此 GPU 分配?', '提示', { type: 'warning' })
await unassignGpu(id)
ElMessage.success('已撤销分配')
await loadGpuAssignments()
} catch {
// 用户取消
}
}
const selectedNodeId = ref('') const selectedNodeId = ref('')
const lastUpdated = ref('') const lastUpdated = ref('')
const nodeDialogVisible = ref(false) const nodeDialogVisible = ref(false)
@@ -127,6 +176,10 @@ async function loadReplicas() {
async function changeTab(name: string | number) { async function changeTab(name: string | number) {
await router.replace({ path: '/compute', query: { tab: String(name) } }) await router.replace({ path: '/compute', query: { tab: String(name) } })
if (name === 'assignments') {
loadGpuAssignments()
allUsers.value = await getUsers().catch(() => [])
}
} }
async function handleNodeAction(action: 'enable' | 'disable' | 'test', node: ComputeNode) { async function handleNodeAction(action: 'enable' | 'disable' | 'test', node: ComputeNode) {
@@ -309,6 +362,7 @@ function formatTime(value?: string) {
onMounted(() => { onMounted(() => {
load({ showLoading: true }) load({ showLoading: true })
timer = setInterval(() => load(), 5000) timer = setInterval(() => load(), 5000)
if (auth.isAdmin) loadGpuAssignments()
}) })
onUnmounted(() => { onUnmounted(() => {
@@ -476,6 +530,24 @@ onUnmounted(() => {
</el-table-column> </el-table-column>
</el-table> </el-table>
</el-tab-pane> </el-tab-pane>
<el-tab-pane v-if="auth.isAdmin" label="GPU 分配" name="assignments">
<div style="margin-bottom: 16px">
<el-button type="primary" @click="showAssignDialog = true">分配 GPU</el-button>
</div>
<el-table :data="gpuAssignments" border v-loading="assignmentLoading">
<el-table-column prop="node_name" label="节点" min-width="140" />
<el-table-column prop="gpu_index" label="GPU 序号" width="100" />
<el-table-column prop="gpu_name" label="GPU 名称" min-width="160" />
<el-table-column prop="display_name" label="被分配用户" min-width="140" />
<el-table-column prop="assigned_at" label="分配时间" min-width="180" />
<el-table-column label="操作" width="100">
<template #default="{ row }">
<el-button type="danger" size="small" @click="handleUnassign(row.id)">撤销</el-button>
</template>
</el-table-column>
</el-table>
</el-tab-pane>
</el-tabs> </el-tabs>
<el-dialog <el-dialog
@@ -526,6 +598,29 @@ onUnmounted(() => {
<el-button type="primary" :loading="savingNode" @click="saveNode">保存</el-button> <el-button type="primary" :loading="savingNode" @click="saveNode">保存</el-button>
</template> </template>
</el-dialog> </el-dialog>
<!-- GPU 分配对话框 -->
<el-dialog v-model="showAssignDialog" title="分配 GPU" width="480px">
<el-form label-width="100px">
<el-form-item label="算力节点">
<el-select v-model="assignForm.node_id" placeholder="选择节点">
<el-option v-for="n in nodes" :key="n.id" :label="n.name || n.code" :value="n.id" />
</el-select>
</el-form-item>
<el-form-item label="GPU 序号">
<el-input-number v-model="assignForm.gpu_index" :min="0" :max="15" />
</el-form-item>
<el-form-item label="用户">
<el-select v-model="assignForm.user_id" placeholder="选择用户" filterable>
<el-option v-for="u in allUsers" :key="u.id" :label="u.display_name || u.username" :value="u.id" />
</el-select>
</el-form-item>
</el-form>
<template #footer>
<el-button @click="showAssignDialog = false">取消</el-button>
<el-button type="primary" @click="handleAssign">确认分配</el-button>
</template>
</el-dialog>
</section> </section>
</template> </template>

View File

@@ -1,305 +1,165 @@
<script setup lang="ts"> <script setup lang="ts">
import { reactive } from 'vue' import { onMounted, ref } from 'vue'
import { ElMessage } from 'element-plus' import { ElMessage, ElMessageBox } from 'element-plus'
import { Plus, Delete, Refresh } from '@element-plus/icons-vue'
import type { TagProps, UploadRequestOptions } from 'element-plus'
import PageCard from '@/components/PageCard.vue' import PageCard from '@/components/PageCard.vue'
import {
getDataConvertTasks,
createDataConvertTask,
uploadSourceFiles,
deleteDataConvertTask,
type DataConvertTask,
} from '@/api/modules/data-convert'
const settings = reactive({ const loading = ref(false)
outputName: 'converted-data', const tasks = ref<DataConvertTask[]>([])
encoding: 'UTF-8', const showCreate = ref(false)
}) const form = ref({ name: '', outputName: 'converted-data' })
function showPrototypeNotice() { async function load() {
ElMessage.info('当前仅完成界面设计,转换功能将在后续接入') loading.value = true
try {
const res = await getDataConvertTasks()
tasks.value = res.items || []
} finally {
loading.value = false
}
} }
function resetSettings() { async function submitCreate() {
Object.assign(settings, { if (!form.value.name) {
outputName: 'converted-data', ElMessage.warning('请填写任务名称')
encoding: 'UTF-8', return
})
} }
await createDataConvertTask({
name: form.value.name,
output_filename: form.value.outputName + '.jsonl',
})
ElMessage.success('任务创建成功')
showCreate.value = false
form.value = { name: '', outputName: 'converted-data' }
load()
}
async function customUpload(options: UploadRequestOptions) {
const taskId = options.data?.taskId as string
if (!taskId) {
ElMessage.error('任务 ID 缺失')
return
}
try {
const res = await uploadSourceFiles(taskId, [options.file])
const data = (res as any)?.data || res
if (data?.auto_converted) {
ElMessage.success(
`上传并自动转换完成,输入 ${data.input_count} / 输出 ${data.output_count},已自动导入数据集`,
)
} else {
ElMessage.warning('上传完成,但转换失败:' + (data?.error || '未知错误'))
}
load()
} catch {
ElMessage.error('上传失败')
}
}
async function handleDelete(task: DataConvertTask) {
try {
await ElMessageBox.confirm(
`确定要删除任务「${task.name}」吗?`,
'删除确认',
{ confirmButtonText: '确定删除', cancelButtonText: '取消', type: 'warning' },
)
} catch {
return
}
await deleteDataConvertTask(task.id)
ElMessage.success('任务已删除')
load()
}
function statusTag(status: string) {
const map: Record<string, { type: TagProps['type']; label: string }> = {
pending: { type: 'info', label: '待上传' },
uploaded: { type: 'warning', label: '已上传' },
running: { type: 'warning', label: '转换中' },
completed: { type: 'success', label: '已完成' },
failed: { type: 'danger', label: '失败' },
}
return map[status] || { type: 'info', label: status }
}
onMounted(load)
</script> </script>
<template> <template>
<PageCard <PageCard
class="data-convert-page" class="data-convert-page"
title="数据类型转换" title="数据类型转换"
subtitle="将 JSON 文件转换为便于训练和评测使用的 JSONL 格式" subtitle="将 JSON 文件转换为 JSONL 格式,转换结果自动导入到数据集管理"
> >
<div class="converter-panel"> <div class="toolbar">
<div class="panel-header"> <el-button type="primary" :icon="Plus" @click="showCreate = true">新建转换任务</el-button>
<div class="tool-icon" aria-hidden="true"> <el-button :icon="Refresh" @click="load">刷新</el-button>
<i class="fa fa-exchange" />
</div>
<div>
<h3>JSON JSONL</h3>
<p>每条 JSON 数据将输出为 JSONL 文件中的一行记录</p>
</div>
</div> </div>
<el-form class="converter-form" label-position="top"> <el-table :data="tasks" v-loading="loading" border stripe>
<el-form-item label="转换类型"> <el-table-column prop="name" label="任务名称" min-width="140" />
<div class="format-field" aria-label="JSON JSONL"> <el-table-column label="状态" min-width="100">
<span>JSON</span> <template #default="{ row }">
<i class="fa fa-long-arrow-right" aria-hidden="true" /> <el-tag :type="statusTag(row.status).type">{{ statusTag(row.status).label }}</el-tag>
<span>JSONL</span> </template>
</div> </el-table-column>
<el-table-column label="数据量" min-width="120">
<template #default="{ row }">
{{ row.output_count }}
</template>
</el-table-column>
<el-table-column prop="output_filename" label="输出文件名" min-width="160" />
<el-table-column prop="create_time" label="创建时间" min-width="180" />
<el-table-column label="操作" width="240" fixed="right">
<template #default="{ row }">
<el-upload
v-if="row.status === 'pending'"
:auto-upload="true"
:show-file-list="false"
accept=".json"
:http-request="customUpload"
:data="{ taskId: row.id }"
style="display: inline-block; margin-right: 8px"
>
<el-button link type="primary">上传文件</el-button>
</el-upload>
<el-button link type="danger" :icon="Delete" @click="handleDelete(row as DataConvertTask)">删除</el-button>
</template>
</el-table-column>
</el-table>
<!-- 新建任务弹窗 -->
<el-dialog v-model="showCreate" title="新建转换任务" width="480px">
<el-form label-width="100px">
<el-form-item label="任务名称" required>
<el-input v-model="form.name" placeholder="请输入任务名称" />
</el-form-item> </el-form-item>
<el-form-item label="源文件" required>
<button class="upload-zone" type="button" @click="showPrototypeNotice">
<i class="fa fa-cloud-upload" aria-hidden="true" />
<span class="upload-content">
<strong>点击选择或拖拽 JSON 文件到此处</strong>
<small>仅支持 .json 格式单文件不超过 200 MB</small>
</span>
<span class="select-button">选择文件</span>
</button>
</el-form-item>
<div class="form-row">
<el-form-item label="输出文件名"> <el-form-item label="输出文件名">
<el-input v-model="settings.outputName"> <el-input v-model="form.outputName" placeholder="converted-data">
<template #append>.jsonl</template> <template #append>.jsonl</template>
</el-input> </el-input>
</el-form-item> </el-form-item>
<el-form-item label="字符编码">
<el-select v-model="settings.encoding" style="width: 100%">
<el-option label="UTF-8" value="UTF-8" />
</el-select>
</el-form-item>
</div>
<div class="format-tip">
<i class="fa fa-info-circle" aria-hidden="true" />
<span>支持由 JSON 数组转换为 JSONL每个数组元素输出为一行</span>
</div>
</el-form> </el-form>
<template #footer>
<div class="panel-footer"> <el-button @click="showCreate = false">取消</el-button>
<span class="prototype-label">当前为 UI 原型暂不执行实际转换</span> <el-button type="primary" @click="submitCreate">创建</el-button>
<div class="actions"> </template>
<el-button @click="resetSettings">重置</el-button> </el-dialog>
<el-tooltip content="转换功能将在后续开发中接入" placement="top">
<span><el-button type="primary" disabled>开始转换</el-button></span>
</el-tooltip>
</div>
</div>
</div>
</PageCard> </PageCard>
</template> </template>
<style scoped lang="scss"> <style scoped lang="scss">
.converter-panel { .toolbar {
width: 100%;
min-height: calc(100vh - 220px);
border: 1px solid #e4e7ed;
border-radius: 8px;
background: #fff;
display: flex;
flex-direction: column;
}
.panel-header {
min-height: 72px;
padding: 16px 20px;
border-bottom: 1px solid #ebeef5;
background: #fafafa;
display: flex;
align-items: center;
gap: 12px;
box-sizing: border-box;
.tool-icon {
width: 38px;
height: 38px;
flex: 0 0 auto;
border-radius: 6px;
background: var(--el-color-primary-light-9);
color: var(--primary-color);
display: flex;
align-items: center;
justify-content: center;
}
h3 {
margin: 0;
color: #303133;
font-size: 15px;
font-weight: 600;
}
p {
margin: 4px 0 0;
color: #909399;
font-size: 12px;
}
}
.converter-form {
flex: 1;
padding: 22px 24px 6px;
:deep(.el-form-item) {
margin-bottom: 20px;
}
:deep(.el-form-item__label) {
padding-bottom: 8px;
color: #606266;
font-size: 13px;
}
}
.format-field {
width: 100%;
min-height: 40px;
padding: 0 14px;
border: 1px solid #dcdfe6;
border-radius: 4px;
background: #f5f7fa;
color: #303133;
display: flex;
align-items: center;
gap: 14px;
box-sizing: border-box;
font-size: 13px;
font-weight: 500;
i {
color: #909399;
}
}
.upload-zone {
width: 100%;
min-height: 112px;
padding: 20px;
border: 1px dashed #b8c4d1;
border-radius: 6px;
background: #fafcff;
color: #606266;
cursor: pointer;
font: inherit;
display: flex;
align-items: center;
gap: 14px;
text-align: left;
transition: border-color 0.2s ease, background 0.2s ease;
&:hover,
&:focus-visible {
border-color: var(--primary-color);
background: var(--el-color-primary-light-9);
outline: none;
}
> i {
color: var(--primary-color);
font-size: 24px;
}
.upload-content {
min-width: 0;
display: flex;
flex: 1;
flex-direction: column;
gap: 5px;
}
strong {
color: #303133;
font-size: 13px;
font-weight: 500;
}
small {
color: #909399;
font-size: 12px;
}
.select-button {
min-height: 32px;
padding: 0 14px;
border: 1px solid #dcdfe6;
border-radius: 4px;
background: #fff;
color: #606266;
display: inline-flex;
align-items: center;
white-space: nowrap;
}
}
.form-row {
display: grid;
grid-template-columns: minmax(0, 2fr) minmax(180px, 1fr);
gap: 16px;
}
.format-tip {
min-height: 38px;
padding: 9px 12px;
border-radius: 4px;
background: var(--el-color-primary-light-9);
color: #606266;
display: flex;
align-items: center;
gap: 8px;
box-sizing: border-box;
font-size: 12px;
i {
color: var(--primary-color);
}
}
.panel-footer {
min-height: 64px;
padding: 12px 24px;
border-top: 1px solid #ebeef5;
background: #fafafa;
display: flex;
align-items: center;
justify-content: space-between;
gap: 20px;
box-sizing: border-box;
.prototype-label {
color: #909399;
font-size: 12px;
}
.actions {
display: flex; display: flex;
gap: 10px; gap: 10px;
} margin-bottom: 16px;
}
@media (max-width: 640px) {
.converter-form {
padding: 18px 16px 4px;
}
.form-row {
grid-template-columns: 1fr;
gap: 0;
}
.upload-zone {
align-items: flex-start;
flex-wrap: wrap;
.select-button {
margin-left: 38px;
}
}
.panel-footer {
padding: 12px 16px;
align-items: flex-end;
flex-direction: column;
}
} }
</style> </style>

View File

@@ -20,6 +20,7 @@ import {
import { useDataProcessGeneration } from './create/useDataProcessGeneration' import { useDataProcessGeneration } from './create/useDataProcessGeneration'
import { useDataProcessPreviewBuild } from './create/useDataProcessPreviewBuild' import { useDataProcessPreviewBuild } from './create/useDataProcessPreviewBuild'
import { useDataProcessRegeneration } from './create/useDataProcessRegeneration' import { useDataProcessRegeneration } from './create/useDataProcessRegeneration'
import { createDefaultExternalSource, externalSourcePayload, restoreExternalSourceConfig, sourceConfigForBackend } from './create/externalSourceConfig'
import { import {
loadCanonicalSourceContent, loadCanonicalSourceContent,
mapDataProcessSourceFile, mapDataProcessSourceFile,
@@ -39,26 +40,20 @@ import {
updateDataProcessPreview, updateDataProcessPreview,
updateDataProcessTask, updateDataProcessTask,
updateDataProcessWorkflowStep, updateDataProcessWorkflowStep,
type DataProcessExternalSourcePayload,
type DataProcessPreviewItem, type DataProcessPreviewItem,
} from '@/api/modules/dataProcess' } from '@/api/modules/dataProcess'
import type { import type { DataProcessConfig, DataProcessPreviewProgress, DataProcessTask, DataProcessWorkflowStep } from '@/types/dataProcess'
DataProcessConfig,
DataProcessPreviewProgress,
DataProcessTask,
DataProcessWorkflowStep,
} from '@/types/dataProcess'
import type { import type {
ExternalDataSource, ExternalDataSource,
GenerationControlOptions, GenerationControlOptions,
PreviewItem, PreviewItem,
ProcessType, ProcessType,
SourceMode,
StepId, StepId,
StructuredProcessOptions, StructuredProcessOptions,
UnstructuredProcessOptions, UnstructuredProcessOptions,
UploadedDataFile, UploadedDataFile,
} from './create/types' } from './create/types'
const router = useRouter() const router = useRouter()
const modelsStore = useModelsStore() const modelsStore = useModelsStore()
const { list: modelList, loaded: modelsLoaded } = storeToRefs(modelsStore) const { list: modelList, loaded: modelsLoaded } = storeToRefs(modelsStore)
@@ -71,7 +66,7 @@ const PREVIEW_MODEL_VERSION = 'backend-pipeline-v4'
const WIZARD_STEPS = [ const WIZARD_STEPS = [
{ id: 'create', title: '创建任务', desc: '填写任务信息与处理配置' }, { id: 'create', title: '创建任务', desc: '填写任务信息与处理配置' },
{ id: 'model', title: '大模型选择', desc: '选择生成模型并设置输出要求' }, { id: 'model', title: '大模型选择', desc: '选择生成模型并设置输出要求' },
{ id: 'upload', title: '上传文件', desc: '上传或接入待处理的源数据' }, { id: 'upload', title: '数据来源', desc: '选择本地上传或外部数据源拉取' },
{ id: 'preview', title: '数据预览', desc: '核对源文件与预览内容' }, { id: 'preview', title: '数据预览', desc: '核对源文件与预览内容' },
{ id: 'generate', title: '开始生成', desc: '确认摘要并启动处理' }, { id: 'generate', title: '开始生成', desc: '确认摘要并启动处理' },
{ id: 'results', title: '结果编辑与保存', desc: '检查、修改并保存结果' }, { id: 'results', title: '结果编辑与保存', desc: '检查、修改并保存结果' },
@@ -81,11 +76,13 @@ const currentStepId = computed<StepId>(() => WIZARD_STEPS[currentStep.value]?.id
const task = reactive({ name: '', description: '' }) const task = reactive({ name: '', description: '' })
const taskId = ref<string | null>(null) const taskId = ref<string | null>(null)
const processType = ref<ProcessType>('structured') const processType = ref<ProcessType>('structured')
const sourceMode = ref<SourceMode>('local')
const structuredOptions = ref<StructuredProcessOptions>(createDefaultStructuredOptions()) const structuredOptions = ref<StructuredProcessOptions>(createDefaultStructuredOptions())
const unstructuredOptions = ref<UnstructuredProcessOptions>(createDefaultUnstructuredOptions()) const unstructuredOptions = ref<UnstructuredProcessOptions>(createDefaultUnstructuredOptions())
const modelSelectionOptions = computed<GenerationControlOptions>(() => ( const modelSelectionOptions = computed<GenerationControlOptions>(() => (
processType.value === 'unstructured' ? unstructuredOptions.value : structuredOptions.value processType.value === 'unstructured' ? unstructuredOptions.value : structuredOptions.value
)) ))
const activeOutputType = computed(() => modelSelectionOptions.value.outputType)
const uploadedFiles = ref<UploadedDataFile[]>([]) const uploadedFiles = ref<UploadedDataFile[]>([])
const previewBuilding = ref(false) const previewBuilding = ref(false)
const { const {
@@ -93,16 +90,7 @@ const {
startPreviewBuild, startPreviewBuild,
stopPreviewPolling, stopPreviewPolling,
} = useDataProcessPreviewBuild() } = useDataProcessPreviewBuild()
const externalSource = reactive<ExternalDataSource>({ const externalSource = reactive<ExternalDataSource>(createDefaultExternalSource())
type: 'postgresql',
url: '',
authMode: 'none',
username: '',
password: '',
limit: 1000,
query: '',
fileName: 'external-data.jsonl',
})
const externalPulling = ref(false) const externalPulling = ref(false)
const externalConnected = ref(false) const externalConnected = ref(false)
const fileName = computed(() => uploadedFiles.value.map(f => f.name).join(', ')) const fileName = computed(() => uploadedFiles.value.map(f => f.name).join(', '))
@@ -135,6 +123,7 @@ const {
} = useDataProcessGeneration({ } = useDataProcessGeneration({
taskId, taskId,
dirty, dirty,
outputType: activeOutputType,
beforeGenerate: beforeStartGeneration, beforeGenerate: beforeStartGeneration,
}) })
const { enqueueSourceUpload, sourceUploading } = useDataProcessSourceUpload({ const { enqueueSourceUpload, sourceUploading } = useDataProcessSourceUpload({
@@ -174,7 +163,7 @@ const previewFiles = computed(() => uploadedFiles.value.map((file) => {
})) }))
const primaryActionLabel = computed(() => { const primaryActionLabel = computed(() => {
if (currentStepId.value === 'create') return '继续:选择大模型' if (currentStepId.value === 'create') return '继续:选择大模型'
if (currentStepId.value === 'model') return '继续:上传文件' if (currentStepId.value === 'model') return '继续:选择数据来源'
if (currentStepId.value === 'upload') { if (currentStepId.value === 'upload') {
if (sourceUploading.value) return '正在上传' if (sourceUploading.value) return '正在上传'
return previewBuilding.value ? '正在切分' : '继续:数据预览' return previewBuilding.value ? '正在切分' : '继续:数据预览'
@@ -198,7 +187,6 @@ function goToStep(stepId: StepId) {
const nextStepIndex = WIZARD_STEPS.findIndex((step) => step.id === stepId) const nextStepIndex = WIZARD_STEPS.findIndex((step) => step.id === stepId)
if (nextStepIndex >= 0) currentStep.value = nextStepIndex if (nextStepIndex >= 0) currentStep.value = nextStepIndex
} }
function updateModelSelectionOptions(value: GenerationControlOptions) { function updateModelSelectionOptions(value: GenerationControlOptions) {
if (processType.value === 'unstructured') { if (processType.value === 'unstructured') {
unstructuredOptions.value = { ...unstructuredOptions.value, ...value } unstructuredOptions.value = { ...unstructuredOptions.value, ...value }
@@ -206,12 +194,12 @@ function updateModelSelectionOptions(value: GenerationControlOptions) {
} }
structuredOptions.value = { ...structuredOptions.value, ...value } structuredOptions.value = { ...structuredOptions.value, ...value }
} }
function toBackendConfig(): DataProcessConfig { function toBackendConfig(): DataProcessConfig {
const options = processType.value === 'unstructured' const options = processType.value === 'unstructured'
? unstructuredOptions.value ? unstructuredOptions.value
: structuredOptions.value : structuredOptions.value
const common = { const common = {
...sourceConfigForBackend(sourceMode.value, externalSource),
preprocess_options: [...options.preprocessOptions], preprocess_options: [...options.preprocessOptions],
semantic_enrichment: options.semanticEnrichment, semantic_enrichment: options.semanticEnrichment,
dataset_split: { ...options.datasetSplit }, dataset_split: { ...options.datasetSplit },
@@ -247,7 +235,6 @@ function toBackendConfig(): DataProcessConfig {
qa_pairs_per_row: structuredOptions.value.qaPairsPerRow, qa_pairs_per_row: structuredOptions.value.qaPairsPerRow,
} }
} }
function taskPayload() { function taskPayload() {
return { return {
name: task.name.trim(), name: task.name.trim(),
@@ -256,12 +243,10 @@ function taskPayload() {
config: toBackendConfig(), config: toBackendConfig(),
} }
} }
async function persistWorkflowStep(step: StepId) { async function persistWorkflowStep(step: StepId) {
if (!taskId.value) return if (!taskId.value) return
await updateDataProcessWorkflowStep(taskId.value, step as DataProcessWorkflowStep) await updateDataProcessWorkflowStep(taskId.value, step as DataProcessWorkflowStep)
} }
async function saveTaskConfiguration() { async function saveTaskConfiguration() {
if (isRegeneration.value) { if (isRegeneration.value) {
const regenerated = await prepareRegeneration(taskPayload()) const regenerated = await prepareRegeneration(taskPayload())
@@ -277,19 +262,6 @@ async function saveTaskConfiguration() {
return saved return saved
} }
function externalPayload(): DataProcessExternalSourcePayload {
return {
type: 'postgresql',
url: externalSource.url.trim(),
auth_mode: externalSource.authMode,
username: externalSource.username || undefined,
password: externalSource.password || undefined,
limit: externalSource.limit,
query: externalSource.query?.trim() || undefined,
file_name: externalSource.fileName || 'external-data.jsonl',
}
}
function mapPreviewItem(item: DataProcessPreviewItem): PreviewItem { function mapPreviewItem(item: DataProcessPreviewItem): PreviewItem {
const sourceLocator = item.quality_score?.source_locator const sourceLocator = item.quality_score?.source_locator
return { return {
@@ -378,7 +350,7 @@ async function beforeStartGeneration() {
) )
} }
watch( watch(
[() => task.name, () => task.description, processType, structuredOptions, unstructuredOptions, externalSource], [() => task.name, () => task.description, processType, sourceMode, structuredOptions, unstructuredOptions, externalSource],
() => { () => {
if (!hydrating.value) dirty.value = true if (!hydrating.value) dirty.value = true
}, },
@@ -396,6 +368,13 @@ watch(processType, (nextType, previousType) => {
ElMessage.info('处理类型已变更,请重新上传或拉取匹配的源数据') ElMessage.info('处理类型已变更,请重新上传或拉取匹配的源数据')
}) })
watch(processType, (nextType) => {
if (nextType === 'unstructured' && sourceMode.value === 'external') {
sourceMode.value = 'local'
externalConnected.value = false
}
})
watch(generationOptionsSignature, (currentSignature, previousSignature) => { watch(generationOptionsSignature, (currentSignature, previousSignature) => {
if (hydrating.value) return if (hydrating.value) return
if (currentSignature === previousSignature) return if (currentSignature === previousSignature) return
@@ -453,6 +432,16 @@ function updateExternalSource(value: ExternalDataSource) {
externalConnected.value = false externalConnected.value = false
} }
async function updateSourceMode(value: SourceMode) {
const previous = sourceMode.value
sourceMode.value = value
externalConnected.value = false
dirty.value = true
externalPulling.value = true
try { await saveTaskConfiguration() } catch { sourceMode.value = previous }
finally { externalPulling.value = false }
}
async function handleTestConnection() { async function handleTestConnection() {
if (!taskId.value) { if (!taskId.value) {
ElMessage.error('任务尚未创建,请返回模型选择步骤后重试') ElMessage.error('任务尚未创建,请返回模型选择步骤后重试')
@@ -464,7 +453,8 @@ async function handleTestConnection() {
} }
externalPulling.value = true externalPulling.value = true
try { try {
const result = await testDataProcessExternalSource(taskId.value, externalPayload()) await saveTaskConfiguration()
const result = await testDataProcessExternalSource(taskId.value, externalSourcePayload(externalSource))
externalConnected.value = result.connected externalConnected.value = result.connected
if (result.connected) ElMessage.success(result.message || '数据源连接测试成功') if (result.connected) ElMessage.success(result.message || '数据源连接测试成功')
else ElMessage.warning(result.message || '数据源连接失败') else ElMessage.warning(result.message || '数据源连接失败')
@@ -490,7 +480,8 @@ async function handlePullData() {
} }
externalPulling.value = true externalPulling.value = true
try { try {
const response = await pullDataProcessExternalSource(taskId.value, externalPayload()) await saveTaskConfiguration()
const response = await pullDataProcessExternalSource(taskId.value, externalSourcePayload(externalSource))
const newFiles: UploadedDataFile[] = [] const newFiles: UploadedDataFile[] = []
for (const file of response.files) { for (const file of response.files) {
const content = await loadCanonicalSourceContent(taskId.value, file.id) const content = await loadCanonicalSourceContent(taskId.value, file.id)
@@ -620,6 +611,12 @@ function applyPreviewProgress(progress: DataProcessPreviewProgress) {
async function completePreviewWorkspace() { async function completePreviewWorkspace() {
previewItems.value = await loadAllPreviewItems() previewItems.value = await loadAllPreviewItems()
for (const file of uploadedFiles.value) {
if (!file.content && file.sourceFileId && file.fileFormat?.replace('.', '') === 'pdf') {
file.content = await loadCanonicalSourceContent(taskId.value as string, file.sourceFileId)
.catch(() => file.content)
}
}
const configSignature = buildPreviewConfigSignature() const configSignature = buildPreviewConfigSignature()
const previewCounts = new Map<string, number>() const previewCounts = new Map<string, number>()
for (const item of previewItems.value) { for (const item of previewItems.value) {
@@ -678,7 +675,7 @@ async function nextFromUpload() {
return return
} }
if (uploadedFiles.value.length === 0) { if (uploadedFiles.value.length === 0) {
ElMessage.warning(processType.value === 'external' ? '请先拉取至少一个数据源' : '请上传至少一个源数据文件') ElMessage.warning(sourceMode.value === 'external' ? '请先拉取至少一个数据源' : '请上传至少一个源数据文件')
return return
} }
const failedUploads = uploadedFiles.value.filter((file) => file.status === 'failed') const failedUploads = uploadedFiles.value.filter((file) => file.status === 'failed')
@@ -992,6 +989,9 @@ async function initializeExistingWorkflow() {
const sourceTask = await loadRegenerationSource() const sourceTask = await loadRegenerationSource()
if (!sourceTask) return if (!sourceTask) return
taskId.value = String(sourceTask.id) taskId.value = String(sourceTask.id)
const restoredSource = restoreExternalSourceConfig(sourceTask.config || {})
sourceMode.value = restoredSource.mode
Object.assign(externalSource, restoredSource.source)
if (!isWorkflowResume.value) return if (!isWorkflowResume.value) return
if (sourceTask.status === 'completed' && sourceTask.results_confirmed !== false) { if (sourceTask.status === 'completed' && sourceTask.results_confirmed !== false) {
allowLeave = true allowLeave = true
@@ -1102,6 +1102,7 @@ onMounted(() => {
<SourceUploadStep <SourceUploadStep
v-else-if="currentStepId === 'upload'" v-else-if="currentStepId === 'upload'"
:process-type="processType" :process-type="processType"
:source-mode="sourceMode"
:uploaded-files="uploadedFiles" :uploaded-files="uploadedFiles"
:external-source="externalSource" :external-source="externalSource"
:external-pulling="externalPulling" :external-pulling="externalPulling"
@@ -1109,6 +1110,7 @@ onMounted(() => {
:preview-building="previewBuilding" :preview-building="previewBuilding"
:source-uploading="sourceUploading" :source-uploading="sourceUploading"
@update:external-source="updateExternalSource" @update:external-source="updateExternalSource"
@update:source-mode="updateSourceMode"
@file-change="handleFileChange" @file-change="handleFileChange"
@remove-file="handleRemoveFile" @remove-file="handleRemoveFile"
@use-sample="useSampleFile" @use-sample="useSampleFile"
@@ -1154,6 +1156,7 @@ onMounted(() => {
:preview-items="previewItems" :preview-items="previewItems"
:regenerating-result-id="regeneratingResultId" :regenerating-result-id="regeneratingResultId"
:bulk-regeneration="bulkRegeneration" :bulk-regeneration="bulkRegeneration"
:output-type="activeOutputType"
@update:field="updateResultField" @update:field="updateResultField"
@regenerate:all="regenerateAllResults" @regenerate:all="regenerateAllResults"
@regenerate:item="regenerateResult" @regenerate:item="regenerateResult"
@@ -1179,8 +1182,8 @@ onMounted(() => {
<el-button <el-button
class="wizard-primary-action" class="wizard-primary-action"
type="primary" type="primary"
:loading="modelSubmitLoading || generationStarting || generation.status === 'running' || resultRegenerationBusy || (currentStepId === 'upload' && (sourceUploading || previewBuilding))" :loading="externalPulling || modelSubmitLoading || generationStarting || generation.status === 'running' || resultRegenerationBusy || (currentStepId === 'upload' && (sourceUploading || previewBuilding))"
:disabled="hydrating || modelSubmitLoading || generationStarting || Boolean(initializationError) || resultRegenerationBusy || (currentStepId === 'generate' && generation.status === 'running') || previewBuilding || sourceUploading || (currentStepId === 'upload' && hasUnfinishedUploads)" :disabled="externalPulling || hydrating || modelSubmitLoading || generationStarting || Boolean(initializationError) || resultRegenerationBusy || (currentStepId === 'generate' && generation.status === 'running') || previewBuilding || sourceUploading || (currentStepId === 'upload' && hasUnfinishedUploads)"
@click="handlePrimaryAction" @click="handlePrimaryAction"
> >
{{ primaryActionLabel }} <i class="fa" :class="primaryActionIcon" style="margin-left: 6px;" /> {{ primaryActionLabel }} <i class="fa" :class="primaryActionIcon" style="margin-left: 6px;" />

View File

@@ -53,7 +53,7 @@ const resultCellTooltipOptions = {
} as const } as const
let resultFilterTimer: ReturnType<typeof setTimeout> | null = null let resultFilterTimer: ReturnType<typeof setTimeout> | null = null
const editForm = reactive({ instruction: '', input: '', output: '' }) const editForm = reactive({ instruction: '', input: '', output: '', chosen: '', rejected: '' })
const publishForm = reactive<DataProcessPublishPayload>({ const publishForm = reactive<DataProcessPublishPayload>({
dataset_name: '', dataset_name: '',
dataset_type: 'train', dataset_type: 'train',
@@ -69,6 +69,8 @@ const processTypeMap: Record<DataProcessType, string> = {
} }
const configLabelMap: Record<string, string> = { const configLabelMap: Record<string, string> = {
source_mode: '数据来源',
external_source: '外部数据源配置',
preprocess_options: '预处理规则', preprocess_options: '预处理规则',
dataset_split: '数据集划分', dataset_split: '数据集划分',
generation_model_id: '数据生成模型', generation_model_id: '数据生成模型',
@@ -212,6 +214,7 @@ const outputDatasetName = computed(() => (
)) ))
const outputDatasetId = computed(() => detail.value?.output_dataset_id || null) const outputDatasetId = computed(() => detail.value?.output_dataset_id || null)
const hasPublishedOutputs = computed(() => outputDatasets.value.length > 0) const hasPublishedOutputs = computed(() => outputDatasets.value.length > 0)
const isDpoOutput = computed(() => detail.value?.config?.output_type === 'dpo')
const hasCurrentPublishedDataset = computed(() => ( const hasCurrentPublishedDataset = computed(() => (
Boolean(outputDatasetId.value) Boolean(outputDatasetId.value)
&& outputDatasets.value.some((dataset) => String(dataset.id) === String(outputDatasetId.value)) && outputDatasets.value.some((dataset) => String(dataset.id) === String(outputDatasetId.value))
@@ -294,6 +297,18 @@ const configRows = computed(() => {
}) })
function formatConfigValue(key: string, value: unknown) { function formatConfigValue(key: string, value: unknown) {
if (key === 'source_mode') return value === 'external' ? '外部数据源' : '本地上传'
if (key === 'external_source' && value && typeof value === 'object') {
const source = value as Record<string, unknown>
return [
source.type,
source.url,
source.ssl_mode ? `SSL ${source.ssl_mode}` : '',
source.limit ? `上限 ${source.limit}` : '',
source.connect_timeout_seconds ? `连接 ${source.connect_timeout_seconds}s` : '',
source.statement_timeout_seconds ? `查询 ${source.statement_timeout_seconds}s` : '',
].filter(Boolean).join(' · ')
}
if (key === 'generation_model_id') { if (key === 'generation_model_id') {
const snapshot = detail.value?.config?.generation_model_snapshot const snapshot = detail.value?.config?.generation_model_snapshot
if (snapshot && typeof snapshot === 'object' && !Array.isArray(snapshot)) { if (snapshot && typeof snapshot === 'object' && !Array.isArray(snapshot)) {
@@ -305,7 +320,7 @@ function formatConfigValue(key: string, value: unknown) {
return chunkMethodLabelMap[value] || value return chunkMethodLabelMap[value] || value
} }
if (key === 'output_type') { if (key === 'output_type') {
return value === 'reasoning' ? '思维链回答' : '标准回答' return value === 'reasoning' ? '思维链回答' : value === 'dpo' ? 'DPO 偏好对' : '标准回答'
} }
if (key === 'reasoning_detail') { if (key === 'reasoning_detail') {
return value === 'detailed' ? '详细推理' : '普通推理' return value === 'detailed' ? '详细推理' : '普通推理'
@@ -461,14 +476,26 @@ function openResultEditor(result: DataProcessResult) {
instruction: result.instruction, instruction: result.instruction,
input: result.input, input: result.input,
output: result.output, output: result.output,
chosen: result.chosen || result.output || '',
rejected: result.rejected || '',
}) })
editDialogVisible.value = true editDialogVisible.value = true
} }
async function saveResult() { async function saveResult() {
if (!editingResult.value) return if (!editingResult.value) return
if (!editForm.instruction.trim() || !editForm.output.trim()) { const invalid = isDpoOutput.value
ElMessage.warning('Instruction 和 Output 不能为空') ? !editForm.instruction.trim()
|| !editForm.chosen.trim()
|| !editForm.rejected.trim()
|| editForm.chosen.trim() === editForm.rejected.trim()
: !editForm.instruction.trim() || !editForm.output.trim()
if (invalid) {
ElMessage.warning(
isDpoOutput.value
? 'Instruction、Chosen、Rejected 均不能为空,且两个回答不能相同'
: 'Instruction 和 Output 不能为空',
)
return return
} }
@@ -478,6 +505,8 @@ async function saveResult() {
instruction: editForm.instruction, instruction: editForm.instruction,
input: editForm.input, input: editForm.input,
output: editForm.output, output: editForm.output,
chosen: editForm.chosen,
rejected: editForm.rejected,
expected_updated_at: editingResult.value.updated_at, expected_updated_at: editingResult.value.updated_at,
}) })
replaceResult(updated) replaceResult(updated)
@@ -530,6 +559,7 @@ function openPublishDialog() {
if (!detail.value) return if (!detail.value) return
publishForm.dataset_name = `${detail.value.name}-数据集` publishForm.dataset_name = `${detail.value.name}-数据集`
publishForm.split = configuredSplit() publishForm.split = configuredSplit()
publishForm.format = isDpoOutput.value ? 'dpo' : 'alpaca_jsonl'
publishDialogVisible.value = true publishDialogVisible.value = true
} }
@@ -808,7 +838,11 @@ onBeforeUnmount(() => {
<el-table-column type="index" label="#" width="56" align="center" /> <el-table-column type="index" label="#" width="56" align="center" />
<el-table-column label="指令" min-width="190" show-overflow-tooltip prop="instruction" /> <el-table-column label="指令" min-width="190" show-overflow-tooltip prop="instruction" />
<el-table-column label="输入" min-width="160" show-overflow-tooltip prop="input" /> <el-table-column label="输入" min-width="160" show-overflow-tooltip prop="input" />
<el-table-column label="输出" min-width="230" show-overflow-tooltip prop="output" /> <el-table-column v-if="!isDpoOutput" label="输出" min-width="230" show-overflow-tooltip prop="output" />
<template v-else>
<el-table-column label="Chosen" min-width="210" show-overflow-tooltip prop="chosen" />
<el-table-column label="Rejected" min-width="210" show-overflow-tooltip prop="rejected" />
</template>
<el-table-column label="质量分" width="88" align="center"> <el-table-column label="质量分" width="88" align="center">
<template #default="{ row }"> <template #default="{ row }">
<el-tooltip :content="qualityFlagsLabel((row as DataProcessResult).quality_score)"> <el-tooltip :content="qualityFlagsLabel((row as DataProcessResult).quality_score)">
@@ -862,9 +896,17 @@ onBeforeUnmount(() => {
<el-form-item label="Input"> <el-form-item label="Input">
<el-input v-model="editForm.input" type="textarea" :rows="3" maxlength="10000" show-word-limit /> <el-input v-model="editForm.input" type="textarea" :rows="3" maxlength="10000" show-word-limit />
</el-form-item> </el-form-item>
<el-form-item label="Output" required> <el-form-item v-if="!isDpoOutput" label="Output" required>
<el-input v-model="editForm.output" type="textarea" :rows="6" maxlength="20000" show-word-limit /> <el-input v-model="editForm.output" type="textarea" :rows="6" maxlength="20000" show-word-limit />
</el-form-item> </el-form-item>
<template v-else>
<el-form-item label="Chosen优选回答" required>
<el-input v-model="editForm.chosen" type="textarea" :rows="6" maxlength="20000" show-word-limit />
</el-form-item>
<el-form-item label="Rejected拒选回答" required>
<el-input v-model="editForm.rejected" type="textarea" :rows="6" maxlength="20000" show-word-limit />
</el-form-item>
</template>
</el-form> </el-form>
<template #footer> <template #footer>
<el-button @click="editDialogVisible = false">取消</el-button> <el-button @click="editDialogVisible = false">取消</el-button>
@@ -884,7 +926,8 @@ onBeforeUnmount(() => {
</el-select> </el-select>
</el-form-item> </el-form-item>
<el-form-item label="输出格式"> <el-form-item label="输出格式">
<el-select v-model="publishForm.format"> <el-select v-model="publishForm.format" :disabled="isDpoOutput">
<el-option v-if="isDpoOutput" label="DPO JSONL" value="dpo" />
<el-option label="Alpaca JSONL" value="alpaca_jsonl" /> <el-option label="Alpaca JSONL" value="alpaca_jsonl" />
<el-option label="JSONL" value="jsonl" /> <el-option label="JSONL" value="jsonl" />
</el-select> </el-select>

View File

@@ -76,7 +76,7 @@ function updateQualityRules(value: Array<string | number>) {
} }
function updateOutputType(value: string | number | boolean | undefined) { function updateOutputType(value: string | number | boolean | undefined) {
const outputType = value === 'reasoning' ? 'reasoning' : 'standard' const outputType = value === 'reasoning' || value === 'dpo' ? value : 'standard'
emit('update:options', { emit('update:options', {
...props.options, ...props.options,
outputType, outputType,
@@ -153,6 +153,9 @@ function modelMeta(model: ModelItem) {
<small v-if="options.outputType === 'reasoning'"> <small v-if="options.outputType === 'reasoning'">
当前使用思维链专用提示语系统还会按所选详细程度约束推理结构 当前使用思维链专用提示语系统还会按所选详细程度约束推理结构
</small> </small>
<small v-else-if="options.outputType === 'dpo'">
当前使用 DPO 专用提示语模型会同时生成 Chosen Rejected 偏好回答
</small>
<small v-else>当前使用标准回答提示语只要求问题和最终答案</small> <small v-else>当前使用标准回答提示语只要求问题和最终答案</small>
</div> </div>
<div class="prompt-input-wrapper"> <div class="prompt-input-wrapper">
@@ -237,7 +240,7 @@ function modelMeta(model: ModelItem) {
<div class="output-type-row"> <div class="output-type-row">
<div class="field-copy"> <div class="field-copy">
<strong>输出类型</strong> <strong>输出类型</strong>
<small>控制答案是否包含可用于推理模型训练的思维链内容</small> <small>选择标准监督思维链或 DPO 成对偏好数据</small>
</div> </div>
<el-select <el-select
class="output-type-select" class="output-type-select"
@@ -247,6 +250,7 @@ function modelMeta(model: ModelItem) {
> >
<el-option label="标准回答" value="standard" /> <el-option label="标准回答" value="standard" />
<el-option label="思维链回答" value="reasoning" /> <el-option label="思维链回答" value="reasoning" />
<el-option label="DPO 偏好对" value="dpo" />
</el-select> </el-select>
</div> </div>
<div v-if="options.outputType === 'reasoning'" class="output-type-row"> <div v-if="options.outputType === 'reasoning'" class="output-type-row">
@@ -272,6 +276,9 @@ function modelMeta(model: ModelItem) {
<template v-else>保留关键依据与必要步骤</template> <template v-else>保留关键依据与必要步骤</template>
最终按 <code>&lt;think&gt;推理过程&lt;/think&gt;</code> 最终按 <code>&lt;think&gt;推理过程&lt;/think&gt;</code>
</template> </template>
<template v-else-if="options.outputType === 'dpo'">
保存同一问题的 <code>chosen</code> 优选回答和 <code>rejected</code> 拒选回答用于直接偏好优化训练
</template>
<template v-else>仅保存最终答案不包含推理过程</template> <template v-else>仅保存最终答案不包含推理过程</template>
</p> </p>
</div> </div>

View File

@@ -1,6 +1,7 @@
<script setup lang="ts"> <script setup lang="ts">
import { computed, ref } from 'vue' import { computed, ref } from 'vue'
import type { BulkResultRegenerationState, PreviewItem, ResultItem } from './types' import type { BulkResultRegenerationState, PreviewItem, ResultItem } from './types'
import type { DataProcessOutputType } from '@/types/dataProcess'
const props = defineProps<{ const props = defineProps<{
items: ResultItem[] items: ResultItem[]
@@ -8,11 +9,12 @@ const props = defineProps<{
selectedId: string | null selectedId: string | null
regeneratingResultId: string | null regeneratingResultId: string | null
bulkRegeneration: BulkResultRegenerationState bulkRegeneration: BulkResultRegenerationState
outputType: DataProcessOutputType
}>() }>()
const emit = defineEmits<{ const emit = defineEmits<{
'update:selectedId': [value: string] 'update:selectedId': [value: string]
'update:field': [id: string, field: 'instruction' | 'input' | 'output', value: string] 'update:field': [id: string, field: 'instruction' | 'input' | 'output' | 'chosen' | 'rejected', value: string]
'regenerate:item': [id: string] 'regenerate:item': [id: string]
'regenerate:all': [] 'regenerate:all': []
}>() }>()
@@ -59,6 +61,11 @@ const selectedSourceMeta = computed(() => {
const source = selectedSource.value const source = selectedSource.value
if (!source) return '' if (!source) return ''
const parts: string[] = [] const parts: string[] = []
if (source.sourcePages?.length) {
const first = source.sourcePages[0]
const last = source.sourcePages[source.sourcePages.length - 1]
parts.push(first === last ? `${first}` : `${first}${last}`)
}
if (source.sourceStartLine != null) { if (source.sourceStartLine != null) {
parts.push( parts.push(
source.sourceEndLine != null && source.sourceEndLine !== source.sourceStartLine source.sourceEndLine != null && source.sourceEndLine !== source.sourceStartLine
@@ -75,6 +82,8 @@ const filteredItems = computed(() => props.items.filter((item, index) => {
const matchesSearch = !keyword const matchesSearch = !keyword
|| item.instruction.toLowerCase().includes(keyword) || item.instruction.toLowerCase().includes(keyword)
|| item.output.toLowerCase().includes(keyword) || item.output.toLowerCase().includes(keyword)
|| item.chosen.toLowerCase().includes(keyword)
|| item.rejected.toLowerCase().includes(keyword)
|| String(index + 1).includes(keyword) || String(index + 1).includes(keyword)
return matchesSearch && (!invalidOnly.value || item.status === 'invalid') return matchesSearch && (!invalidOnly.value || item.status === 'invalid')
})) }))
@@ -135,7 +144,7 @@ function selectRelative(offset: number) {
<span class="result-index">#{{ String(items.findIndex((entry) => entry.id === item.id) + 1).padStart(3, '0') }}</span> <span class="result-index">#{{ String(items.findIndex((entry) => entry.id === item.id) + 1).padStart(3, '0') }}</span>
<span class="result-copy"> <span class="result-copy">
<strong>{{ item.instruction || '未填写指令' }}</strong> <strong>{{ item.instruction || '未填写指令' }}</strong>
<small>{{ item.output || '未填写输出' }}</small> <small>{{ outputType === 'dpo' ? (item.chosen || '未填写 Chosen') : (item.output || '未填写输出') }}</small>
</span> </span>
<i v-if="itemRegenerating(item.id)" class="css-spinner" /> <i v-if="itemRegenerating(item.id)" class="css-spinner" />
<i <i
@@ -207,7 +216,7 @@ function selectRelative(offset: number) {
@update:model-value="emit('update:field', selectedItem.id, 'input', $event)" @update:model-value="emit('update:field', selectedItem.id, 'input', $event)"
/> />
</div> </div>
<div class="field-editor"> <div v-if="outputType !== 'dpo'" class="field-editor">
<label>Output <em>必填</em></label> <label>Output <em>必填</em></label>
<el-input <el-input
:model-value="selectedItem.output" :model-value="selectedItem.output"
@@ -217,6 +226,28 @@ function selectRelative(offset: number) {
@update:model-value="emit('update:field', selectedItem.id, 'output', $event)" @update:model-value="emit('update:field', selectedItem.id, 'output', $event)"
/> />
</div> </div>
<template v-else>
<div class="field-editor dpo-field is-chosen">
<label>Chosen <em>优选回答必填</em></label>
<el-input
:model-value="selectedItem.chosen"
:disabled="selectedItemRegenerating"
type="textarea"
:rows="6"
@update:model-value="emit('update:field', selectedItem.id, 'chosen', $event)"
/>
</div>
<div class="field-editor dpo-field is-rejected">
<label>Rejected <em>拒选回答必填</em></label>
<el-input
:model-value="selectedItem.rejected"
:disabled="selectedItemRegenerating"
type="textarea"
:rows="6"
@update:model-value="emit('update:field', selectedItem.id, 'rejected', $event)"
/>
</div>
</template>
<div v-if="selectedItem.error" class="validation-error"> <div v-if="selectedItem.error" class="validation-error">
<i class="fa fa-exclamation-circle" /> {{ selectedItem.error }} <i class="fa fa-exclamation-circle" /> {{ selectedItem.error }}
</div> </div>

View File

@@ -1,10 +1,11 @@
<script setup lang="ts"> <script setup lang="ts">
import { computed, ref, watch } from 'vue' import { computed, ref, watch } from 'vue'
import type { UploadFile } from 'element-plus' import type { UploadFile } from 'element-plus'
import type { ExternalDataSource, ProcessType, UploadedDataFile } from './types' import type { ExternalDataSource, ProcessType, SourceMode, UploadedDataFile } from './types'
const props = defineProps<{ const props = defineProps<{
processType: ProcessType processType: ProcessType
sourceMode: SourceMode
uploadedFiles: UploadedDataFile[] uploadedFiles: UploadedDataFile[]
externalSource: ExternalDataSource externalSource: ExternalDataSource
externalPulling: boolean externalPulling: boolean
@@ -14,6 +15,7 @@ const props = defineProps<{
}>() }>()
const emit = defineEmits<{ const emit = defineEmits<{
'update:sourceMode': [value: SourceMode]
'update:externalSource': [value: ExternalDataSource] 'update:externalSource': [value: ExternalDataSource]
'file-change': [file: UploadFile] 'file-change': [file: UploadFile]
'remove-file': [uid: string | number] 'remove-file': [uid: string | number]
@@ -31,6 +33,14 @@ const AUTH_MODES = [
{ value: 'basic', label: '账号密码' }, { value: 'basic', label: '账号密码' },
] ]
const SSL_MODES = [
{ value: 'disable', label: '关闭' },
{ value: 'prefer', label: '优先使用(默认)' },
{ value: 'require', label: '必须加密' },
{ value: 'verify-ca', label: '验证 CA' },
{ value: 'verify-full', label: '完整验证' },
]
const FILE_PAGE_SIZE = 10 const FILE_PAGE_SIZE = 10
const currentFilePage = ref(1) const currentFilePage = ref(1)
type FileStage = 'queued' | 'uploading' | 'waiting' | 'processing' | 'success' | 'upload-failed' | 'preview-failed' type FileStage = 'queued' | 'uploading' | 'waiting' | 'processing' | 'success' | 'upload-failed' | 'preview-failed'
@@ -45,7 +55,10 @@ const FILE_STAGE_META: Record<FileStage, { label: string; icon: string }> = {
'preview-failed': { label: '切分失败', icon: 'fa-exclamation-circle' }, 'preview-failed': { label: '切分失败', icon: 'fa-exclamation-circle' },
} }
const isExternal = computed(() => props.processType === 'external') const isExternal = computed(() => props.sourceMode === 'external')
const sourceModeLocked = computed(() => (
props.uploadedFiles.length > 0 || props.sourceUploading || props.previewBuilding || props.externalPulling
))
const uploadAccept = computed(() => props.processType === 'unstructured' const uploadAccept = computed(() => props.processType === 'unstructured'
? '.txt,.md,.markdown,.pdf,.docx,.pptx,.json,.jsonl,.ndjson' ? '.txt,.md,.markdown,.pdf,.docx,.pptx,.json,.jsonl,.ndjson'
@@ -124,6 +137,44 @@ function getFileError(file: UploadedDataFile) {
<template> <template>
<section class="source-upload-step" aria-labelledby="source-upload-title"> <section class="source-upload-step" aria-labelledby="source-upload-title">
<div class="source-mode-section">
<div class="section-title-row">
<div>
<h3>数据来源</h3>
<p>在当前步骤选择本地文件上传或从外部系统拉取处理类型保持不变</p>
</div>
</div>
<div class="source-mode-options">
<button
type="button"
class="source-mode-option"
:class="{ 'is-active': sourceMode === 'local' }"
:disabled="sourceModeLocked"
@click="emit('update:sourceMode', 'local')"
>
<i class="fa fa-upload" aria-hidden="true" />
<span><strong>本地上传</strong><small>从当前设备选择文件</small></span>
<i class="fa fa-check-circle selection-mark" aria-hidden="true" />
</button>
<el-tooltip
:content="processType === 'structured' ? '从标准化外部数据源配置拉取' : '当前仅支持把外部数据库拉取为结构化数据'"
>
<button
type="button"
class="source-mode-option"
:class="{ 'is-active': sourceMode === 'external' }"
:disabled="sourceModeLocked || processType !== 'structured'"
@click="emit('update:sourceMode', 'external')"
>
<i class="fa fa-cloud-download" aria-hidden="true" />
<span><strong>外部数据源</strong><small>连接数据库并执行只读拉取</small></span>
<i class="fa fa-check-circle selection-mark" aria-hidden="true" />
</button>
</el-tooltip>
</div>
<small v-if="sourceModeLocked" class="source-mode-lock-hint">已有来源文件时不可切换删除现有文件后可重新选择</small>
</div>
<div v-if="isExternal" class="form-section external-section"> <div v-if="isExternal" class="form-section external-section">
<div class="section-title-row"> <div class="section-title-row">
<div> <div>
@@ -202,6 +253,45 @@ function getFileError(file: UploadedDataFile) {
@update:model-value="updateExternalField('limit', Number($event) || 0)" @update:model-value="updateExternalField('limit', Number($event) || 0)"
/> />
</el-form-item> </el-form-item>
<el-form-item label="SSL 模式">
<el-select
:model-value="externalSource.sslMode"
aria-label="SSL 模式"
@update:model-value="updateExternalField('sslMode', $event)"
>
<el-option v-for="item in SSL_MODES" :key="item.value" :label="item.label" :value="item.value" />
</el-select>
</el-form-item>
<el-form-item label="连接超时">
<el-input-number
:model-value="externalSource.connectTimeoutSeconds"
:min="1"
:max="30"
controls-position="right"
aria-label="连接超时秒数"
@update:model-value="updateExternalField('connectTimeoutSeconds', Number($event) || 5)"
/>
<small>单位</small>
</el-form-item>
<el-form-item label="查询超时">
<el-input-number
:model-value="externalSource.statementTimeoutSeconds"
:min="1"
:max="300"
controls-position="right"
aria-label="查询超时秒数"
@update:model-value="updateExternalField('statementTimeoutSeconds', Number($event) || 30)"
/>
<small>单位</small>
</el-form-item>
<el-form-item label="落地文件名">
<el-input
:model-value="externalSource.fileName"
placeholder="external-data.jsonl"
aria-label="外部数据落地文件名"
@update:model-value="updateExternalField('fileName', $event)"
/>
</el-form-item>
<el-form-item label="只读查询语句" class="external-query-field"> <el-form-item label="只读查询语句" class="external-query-field">
<el-input <el-input
:model-value="externalSource.query" :model-value="externalSource.query"
@@ -314,7 +404,7 @@ function getFileError(file: UploadedDataFile) {
v-if="uploadedFiles.length === 0" v-if="uploadedFiles.length === 0"
link link
type="primary" type="primary"
:disabled="previewBuilding" :disabled="previewBuilding || externalPulling"
@click="emit('use-sample')" @click="emit('use-sample')"
> >
使用示例数据 使用示例数据
@@ -326,7 +416,7 @@ function getFileError(file: UploadedDataFile) {
drag drag
multiple multiple
:accept="uploadAccept" :accept="uploadAccept"
:disabled="previewBuilding" :disabled="previewBuilding || externalPulling"
:auto-upload="false" :auto-upload="false"
:show-file-list="false" :show-file-list="false"
:on-change="(file: UploadFile) => emit('file-change', file)" :on-change="(file: UploadFile) => emit('file-change', file)"
@@ -353,13 +443,13 @@ function getFileError(file: UploadedDataFile) {
<el-upload <el-upload
multiple multiple
:accept="uploadAccept" :accept="uploadAccept"
:disabled="previewBuilding" :disabled="previewBuilding || externalPulling"
:auto-upload="false" :auto-upload="false"
:show-file-list="false" :show-file-list="false"
:on-change="(file: UploadFile) => emit('file-change', file)" :on-change="(file: UploadFile) => emit('file-change', file)"
aria-label="继续添加源数据文件" aria-label="继续添加源数据文件"
> >
<el-button size="small" type="primary" :disabled="previewBuilding">继续上传</el-button> <el-button size="small" type="primary" :disabled="previewBuilding || externalPulling">继续上传</el-button>
</el-upload> </el-upload>
</div> </div>
</div> </div>
@@ -430,6 +520,68 @@ function getFileError(file: UploadedDataFile) {
width: 100%; width: 100%;
} }
.source-mode-section {
padding-bottom: 22px;
margin-bottom: 22px;
border-bottom: 1px solid #edf0f5;
}
.source-mode-options {
display: grid;
grid-template-columns: repeat(2, minmax(0, 1fr));
gap: 14px;
margin-top: 14px;
}
.source-mode-option {
position: relative;
display: flex;
align-items: center;
gap: 12px;
min-height: 76px;
padding: 14px 16px;
color: #667085;
text-align: left;
background: #fff;
border: 1px solid #dfe3ea;
border-radius: 8px;
cursor: pointer;
> .fa:first-child {
width: 28px;
color: #5b50f2;
font-size: 20px;
text-align: center;
}
span {
display: flex;
flex: 1;
flex-direction: column;
gap: 5px;
}
strong { color: #344054; font-size: 14px; }
small { color: #8a93a3; font-size: 12px; }
.selection-mark { color: transparent; }
&.is-active {
background: #fafaff;
border-color: #5b50f2;
box-shadow: 0 0 0 1px rgba(91, 80, 242, 0.08);
}
&.is-active .selection-mark { color: #5b50f2; }
&:disabled { cursor: not-allowed; opacity: 0.55; }
}
.source-mode-lock-hint {
display: block;
margin-top: 9px;
color: #8a93a3;
font-size: 11px;
}
.form-section { .form-section {
padding: 0; padding: 0;
@@ -672,6 +824,7 @@ function getFileError(file: UploadedDataFile) {
} }
@media (max-width: 900px) { @media (max-width: 900px) {
.source-mode-options,
.external-section .external-grid { .external-section .external-grid {
grid-template-columns: minmax(0, 1fr); grid-template-columns: minmax(0, 1fr);
} }

View File

@@ -188,20 +188,6 @@ defineExpose({ validate })
</span> </span>
<i class="fa fa-check-circle selection-mark" /> <i class="fa fa-check-circle selection-mark" />
</button> </button>
<button
type="button"
class="type-option"
:class="{ 'is-active': processType === 'external' }"
:disabled="processTypeLocked"
@click="emit('update:processType', 'external')"
>
<span class="type-icon"><i class="fa fa-cloud-download" /></span>
<span>
<strong>外来数据源拉取</strong>
<small>适用于数据库API 接口等需远程拉取的外部数据</small>
</span>
<i class="fa fa-check-circle selection-mark" />
</button>
</div> </div>
</el-form-item> </el-form-item>
</div> </div>
@@ -274,7 +260,7 @@ defineExpose({ validate })
.type-options { .type-options {
display: grid; display: grid;
grid-template-columns: repeat(3, minmax(0, 1fr)); grid-template-columns: repeat(2, minmax(0, 1fr));
gap: 16px; gap: 16px;
width: 100%; width: 100%;
} }

View File

@@ -73,10 +73,21 @@ export const DEFAULT_REASONING_GENERATION_PROMPT = `你是一名具有深厚专
3. 信息不足时如实处理:遇到来源信息缺失或相互矛盾时,需在推理中明确指出,并基于来源做出合理假设,但不得编造与来源无关的内容。 3. 信息不足时如实处理:遇到来源信息缺失或相互矛盾时,需在推理中明确指出,并基于来源做出合理假设,但不得编造与来源无关的内容。
4. 最终答案须能从前面的推理过程中自然得出,语言简洁、结构清晰。` 4. 最终答案须能从前面的推理过程中自然得出,语言简洁、结构清晰。`
export function defaultGenerationPrompt(outputType: 'standard' | 'reasoning') { export const DEFAULT_DPO_GENERATION_PROMPT = `你是一名偏好数据构造专家。请基于下方来源内容生成可用于直接偏好优化DPO的成对问答数据。
return outputType === 'reasoning'
? DEFAULT_REASONING_GENERATION_PROMPT 来源内容:
: DEFAULT_STANDARD_GENERATION_PROMPT {{ content }}
任务要求:
1. 问题应聚焦来源内容的核心信息,并具有实际训练价值。
2. Chosen 必须准确、完整、清晰且严格忠于来源内容。
3. Rejected 应当表面合理但包含可辨认的缺陷,例如遗漏关键条件、事实偏差、逻辑不完整或表达含混;不得用乱码、空话或危险内容凑数。
4. Chosen 与 Rejected 必须明显不同,且都直接回答同一个问题。`
export function defaultGenerationPrompt(outputType: 'standard' | 'reasoning' | 'dpo') {
if (outputType === 'reasoning') return DEFAULT_REASONING_GENERATION_PROMPT
if (outputType === 'dpo') return DEFAULT_DPO_GENERATION_PROMPT
return DEFAULT_STANDARD_GENERATION_PROMPT
} }
export function isBuiltInGenerationPrompt(value: string) { export function isBuiltInGenerationPrompt(value: string) {
@@ -92,6 +103,7 @@ export function isBuiltInGenerationPrompt(value: string) {
PREVIOUS_DEFAULT_REASONING_GENERATION_PROMPT_3, PREVIOUS_DEFAULT_REASONING_GENERATION_PROMPT_3,
DEFAULT_STANDARD_GENERATION_PROMPT, DEFAULT_STANDARD_GENERATION_PROMPT,
DEFAULT_REASONING_GENERATION_PROMPT, DEFAULT_REASONING_GENERATION_PROMPT,
DEFAULT_DPO_GENERATION_PROMPT,
].map(normalize).includes(normalizedValue) ].map(normalize).includes(normalizedValue)
} }
@@ -171,8 +183,9 @@ function generationOptionsFromConfig(
config: DataProcessConfig, config: DataProcessConfig,
defaults: GenerationControlOptions, defaults: GenerationControlOptions,
): GenerationControlOptions { ): GenerationControlOptions {
const outputType = configValue(config, 'output_type', defaults.outputType) === 'reasoning' const configuredOutputType = configValue(config, 'output_type', defaults.outputType)
? 'reasoning' const outputType = configuredOutputType === 'reasoning' || configuredOutputType === 'dpo'
? configuredOutputType
: 'standard' : 'standard'
const configuredPrompt = String( const configuredPrompt = String(
configValue(config, 'generation_prompt', defaults.generationPrompt), configValue(config, 'generation_prompt', defaults.generationPrompt),

View File

@@ -0,0 +1,77 @@
import type { DataProcessExternalSourcePayload } from '@/api/modules/dataProcess'
import type { DataProcessConfig } from '@/types/dataProcess'
import type { ExternalDataSource, SourceMode } from './types'
export function createDefaultExternalSource(): ExternalDataSource {
return {
type: 'postgresql',
url: '',
authMode: 'none',
username: '',
password: '',
limit: 1000,
connectTimeoutSeconds: 5,
statementTimeoutSeconds: 30,
sslMode: 'prefer',
query: '',
fileName: 'external-data.jsonl',
}
}
export function externalSourcePayload(source: ExternalDataSource): DataProcessExternalSourcePayload {
return {
type: source.type,
url: source.url.trim(),
auth_mode: source.authMode,
username: source.username?.trim() || undefined,
password: source.password || undefined,
limit: source.limit,
connect_timeout_seconds: source.connectTimeoutSeconds,
statement_timeout_seconds: source.statementTimeoutSeconds,
ssl_mode: source.sslMode,
query: source.query?.trim() || undefined,
file_name: source.fileName || 'external-data.jsonl',
}
}
export function sourceConfigForBackend(mode: SourceMode, source: ExternalDataSource) {
const payload = externalSourcePayload(source)
const { password: _password, ...safeSource } = payload
return {
source_mode: mode,
external_source: mode === 'external' ? safeSource : undefined,
}
}
export function restoreExternalSourceConfig(config: DataProcessConfig): {
mode: SourceMode
source: ExternalDataSource
} {
const defaults = createDefaultExternalSource()
const value = config.external_source
if (!value || typeof value !== 'object') {
return {
mode: config.source_mode === 'external' ? 'external' : 'local',
source: defaults,
}
}
const source = value as Record<string, unknown>
const sslMode = String(source.ssl_mode || '')
return {
mode: config.source_mode === 'external' ? 'external' : 'local',
source: {
...defaults,
url: String(source.url || ''),
authMode: source.auth_mode === 'basic' ? 'basic' : 'none',
username: String(source.username || ''),
limit: Number(source.limit) || defaults.limit,
connectTimeoutSeconds: Number(source.connect_timeout_seconds) || defaults.connectTimeoutSeconds,
statementTimeoutSeconds: Number(source.statement_timeout_seconds) || defaults.statementTimeoutSeconds,
sslMode: ['disable', 'prefer', 'require', 'verify-ca', 'verify-full'].includes(sslMode)
? sslMode as ExternalDataSource['sslMode']
: defaults.sslMode,
query: String(source.query || ''),
fileName: String(source.file_name || defaults.fileName),
},
}
}

View File

@@ -5,6 +5,7 @@ import type {
} from '@/types/dataProcess' } from '@/types/dataProcess'
export type ProcessType = 'structured' | 'unstructured' | 'external' export type ProcessType = 'structured' | 'unstructured' | 'external'
export type SourceMode = 'local' | 'external'
export type StepId = 'create' | 'model' | 'upload' | 'preview' | 'generate' | 'results' export type StepId = 'create' | 'model' | 'upload' | 'preview' | 'generate' | 'results'
@@ -88,6 +89,9 @@ export interface ExternalDataSource {
username?: string username?: string
password?: string password?: string
limit: number limit: number
connectTimeoutSeconds: number
statementTimeoutSeconds: number
sslMode: 'disable' | 'prefer' | 'require' | 'verify-ca' | 'verify-full'
query?: string query?: string
fileName?: string fileName?: string
} }
@@ -167,12 +171,18 @@ export interface ResultItem {
instruction: string instruction: string
input: string input: string
output: string output: string
chosen: string
rejected: string
originalInstruction: string originalInstruction: string
originalInput: string originalInput: string
originalOutput: string originalOutput: string
originalChosen: string
originalRejected: string
savedInstruction: string savedInstruction: string
savedInput: string savedInput: string
savedOutput: string savedOutput: string
savedChosen: string
savedRejected: string
savedStatus: 'valid' | 'modified' | 'invalid' savedStatus: 'valid' | 'modified' | 'invalid'
status: 'valid' | 'modified' | 'invalid' status: 'valid' | 'modified' | 'invalid'
error?: string error?: string

View File

@@ -12,10 +12,12 @@ import {
type DataProcessResult, type DataProcessResult,
} from '@/api/modules/dataProcess' } from '@/api/modules/dataProcess'
import type { BulkResultRegenerationState, GenerationState, ResultItem } from './types' import type { BulkResultRegenerationState, GenerationState, ResultItem } from './types'
import type { DataProcessOutputType } from '@/types/dataProcess'
interface GenerationBindings { interface GenerationBindings {
taskId: Ref<string | null> taskId: Ref<string | null>
dirty: Ref<boolean> dirty: Ref<boolean>
outputType: Ref<DataProcessOutputType>
beforeGenerate?: () => Promise<boolean | void> beforeGenerate?: () => Promise<boolean | void>
} }
@@ -32,12 +34,18 @@ function mapResult(item: DataProcessResult): ResultItem {
instruction: item.instruction, instruction: item.instruction,
input: item.input || '', input: item.input || '',
output: item.output, output: item.output,
chosen: item.chosen || item.output || '',
rejected: item.rejected || '',
originalInstruction: item.original_instruction ?? item.instruction, originalInstruction: item.original_instruction ?? item.instruction,
originalInput: item.original_input ?? item.input ?? '', originalInput: item.original_input ?? item.input ?? '',
originalOutput: item.original_output ?? item.output, originalOutput: item.original_output ?? item.output,
originalChosen: item.original_chosen ?? item.chosen ?? item.output ?? '',
originalRejected: item.original_rejected ?? item.rejected ?? '',
savedInstruction: item.instruction, savedInstruction: item.instruction,
savedInput: item.input || '', savedInput: item.input || '',
savedOutput: item.output, savedOutput: item.output,
savedChosen: item.chosen || item.output || '',
savedRejected: item.rejected || '',
savedStatus: item.status, savedStatus: item.status,
status: item.status, status: item.status,
error: item.error || undefined, error: item.error || undefined,
@@ -246,15 +254,34 @@ export function useDataProcessGeneration(bindings: GenerationBindings) {
} }
} }
function updateResultField(id: string, field: 'instruction' | 'input' | 'output', value: string) { function updateResultField(
id: string,
field: 'instruction' | 'input' | 'output' | 'chosen' | 'rejected',
value: string,
) {
const item = results.value.find((entry) => entry.id === id) const item = results.value.find((entry) => entry.id === id)
if (!item) return if (!item) return
item[field] = value item[field] = value
const valid = item.instruction.trim() && item.output.trim() if (field === 'chosen') item.output = value
item.error = valid ? undefined : 'Instruction 和 Output 不能为空' const isDpo = bindings.outputType.value === 'dpo'
const valid = isDpo
? Boolean(
item.instruction.trim()
&& item.chosen.trim()
&& item.rejected.trim()
&& item.chosen.trim() !== item.rejected.trim()
)
: Boolean(item.instruction.trim() && item.output.trim())
item.error = valid
? undefined
: isDpo
? 'Instruction、Chosen、Rejected 均不能为空,且两个回答不能相同'
: 'Instruction 和 Output 不能为空'
const changed = item.instruction !== item.originalInstruction const changed = item.instruction !== item.originalInstruction
|| item.input !== item.originalInput || item.input !== item.originalInput
|| item.output !== item.originalOutput || item.output !== item.originalOutput
|| item.chosen !== item.originalChosen
|| item.rejected !== item.originalRejected
item.status = item.error ? 'invalid' : changed ? 'modified' : 'valid' item.status = item.error ? 'invalid' : changed ? 'modified' : 'valid'
bindings.dirty.value = true bindings.dirty.value = true
} }
@@ -311,6 +338,8 @@ export function useDataProcessGeneration(bindings: GenerationBindings) {
item.instruction !== item.savedInstruction item.instruction !== item.savedInstruction
|| item.input !== item.savedInput || item.input !== item.savedInput
|| item.output !== item.savedOutput || item.output !== item.savedOutput
|| item.chosen !== item.savedChosen
|| item.rejected !== item.savedRejected
)) ))
if (unsaved) { if (unsaved) {
selectedResultId.value = unsaved.id selectedResultId.value = unsaved.id
@@ -409,12 +438,16 @@ export function useDataProcessGeneration(bindings: GenerationBindings) {
item.instruction !== item.savedInstruction item.instruction !== item.savedInstruction
|| item.input !== item.savedInput || item.input !== item.savedInput
|| item.output !== item.savedOutput || item.output !== item.savedOutput
|| item.chosen !== item.savedChosen
|| item.rejected !== item.savedRejected
)) ))
for (const item of changed) { for (const item of changed) {
const saved = await updateDataProcessResult(taskId, item.id, { const saved = await updateDataProcessResult(taskId, item.id, {
instruction: item.instruction, instruction: item.instruction,
input: item.input, input: item.input,
output: item.output, output: item.output,
chosen: item.chosen,
rejected: item.rejected,
expected_updated_at: item.updatedAt, expected_updated_at: item.updatedAt,
}) })
const index = results.value.findIndex((entry) => entry.id === item.id) const index = results.value.findIndex((entry) => entry.id === item.id)
@@ -425,7 +458,13 @@ export function useDataProcessGeneration(bindings: GenerationBindings) {
function validateResults() { function validateResults() {
let firstInvalidId: string | null = null let firstInvalidId: string | null = null
for (const item of results.value) { for (const item of results.value) {
if (!item.instruction.trim() || !item.output.trim() || item.status === 'invalid') { const requiredFieldsInvalid = bindings.outputType.value === 'dpo'
? !item.instruction.trim()
|| !item.chosen.trim()
|| !item.rejected.trim()
|| item.chosen.trim() === item.rejected.trim()
: !item.instruction.trim() || !item.output.trim()
if (requiredFieldsInvalid || item.status === 'invalid') {
item.error ||= '结果未通过后端质量校验,请修改后重新保存' item.error ||= '结果未通过后端质量校验,请修改后重新保存'
item.status = 'invalid' item.status = 'invalid'
firstInvalidId ??= item.id firstInvalidId ??= item.id

View File

@@ -16,6 +16,8 @@ import { getModelList } from '@/api/modules/model'
import { getDatasetList } from '@/api/modules/dataset' import { getDatasetList } from '@/api/modules/dataset'
import { getSystemInfo } from '@/api/modules/system' import { getSystemInfo } from '@/api/modules/system'
import { getComputeNodes } from '@/api/modules/compute' import { getComputeNodes } from '@/api/modules/compute'
import { getMyGpus } from '@/api/modules/gpu'
import { useAuthStore } from '@/stores/auth'
import { TEMPLATE_GROUPS, LR_SCHEDULER_OPTIONS, QUANTIZATION_BIT_OPTIONS, QUANT_METHOD_OPTIONS, GGUF_FORMAT_OPTIONS } from '@/constants' import { TEMPLATE_GROUPS, LR_SCHEDULER_OPTIONS, QUANTIZATION_BIT_OPTIONS, QUANT_METHOD_OPTIONS, GGUF_FORMAT_OPTIONS } from '@/constants'
import { import {
DEFAULT_TRAINING_PARAMS, DEFAULT_TRAINING_PARAMS,
@@ -46,8 +48,21 @@ const availableGpus = computed(() => {
.filter((n) => n.scheduler_status === 'online' || n.scheduler_status === 'draining') .filter((n) => n.scheduler_status === 'online' || n.scheduler_status === 'draining')
.map((n) => n.id), .map((n) => n.id),
) )
return gpus.value.filter((gpu) => !gpu.node_id || onlineNodeIds.has(gpu.node_id)) let result = gpus.value.filter((gpu) => !gpu.node_id || onlineNodeIds.has(gpu.node_id))
// 普通用户只能看到被分配的 GPU
if (!auth.isAdmin) {
const assignedKeys = new Set(
myAssignedGpus.value.map((g) => `${g.node_id}:${g.gpu_index}`),
)
result = result.filter((gpu) => {
const key = `${gpu.node_id}:${gpu.id ?? gpu.uuid ?? gpu.name}`
return assignedKeys.has(key) || myAssignedGpus.value.length === 0
}) })
}
return result
})
const auth = useAuthStore()
const myAssignedGpus = ref<Array<{ node_id: string; gpu_index: number }>>([])
const modelDialogVisible = ref(false) const modelDialogVisible = ref(false)
const form = reactive(createDefaultFineTuneForm()) const form = reactive(createDefaultFineTuneForm())
@@ -207,6 +222,14 @@ async function loadGpus() {
const [sys, nodes] = await Promise.all([getSystemInfo(), getComputeNodes().catch(() => [])]) const [sys, nodes] = await Promise.all([getSystemInfo(), getComputeNodes().catch(() => [])])
gpus.value = sys?.gpu || [] gpus.value = sys?.gpu || []
computeNodes.value = nodes || [] computeNodes.value = nodes || []
// 普通用户加载被分配的 GPU
if (!auth.isAdmin) {
try {
myAssignedGpus.value = await getMyGpus()
} catch {
myAssignedGpus.value = []
}
}
const firstIdle = availableGpus.value.find((gpu) => !isGpuUnavailable(gpu) && gpu.id != null) const firstIdle = availableGpus.value.find((gpu) => !isGpuUnavailable(gpu) && gpu.id != null)
if (firstIdle) selectedGpuKeys.value = [gpuKey(firstIdle)] if (firstIdle) selectedGpuKeys.value = [gpuKey(firstIdle)]
} catch { } catch {

View File

@@ -8,6 +8,7 @@ import {
createModel, createModel,
updateModel, updateModel,
getLocalModels, getLocalModels,
testOnlineModel,
} from '@/api/modules/model' } from '@/api/modules/model'
import { MODEL_TYPE_MAP } from '@/constants' import { MODEL_TYPE_MAP } from '@/constants'
import type { ModelForm, ModelSource } from '@/types' import type { ModelForm, ModelSource } from '@/types'
@@ -35,6 +36,10 @@ const form = reactive<ModelForm>({
online_model_name: '', online_model_name: '',
}) })
// 在线模型测试
const testLoading = ref(false)
const testResult = ref<{ success: boolean; error?: string; model?: string; usage?: object } | null>(null)
const rules: FormRules = { const rules: FormRules = {
name: [ name: [
{ required: true, message: '请输入模型名称', trigger: 'blur' }, { required: true, message: '请输入模型名称', trigger: 'blur' },
@@ -74,7 +79,7 @@ async function loadEditData() {
path: model.path || '', path: model.path || '',
api_url: model.api_url || '', api_url: model.api_url || '',
api_key: model.api_key || '', api_key: model.api_key || '',
online_model_name: model.model_name || '', online_model_name: model.online_model_name || '',
}) })
} catch { } catch {
// ignore // ignore
@@ -133,6 +138,36 @@ async function handleSubmit() {
}) })
} }
async function handleTestOnline() {
if (!form.api_url || !form.online_model_name) {
ElMessage.warning('请先填写 API 地址和模型名称')
return
}
testLoading.value = true
testResult.value = null
try {
const start = Date.now()
const res = await testOnlineModel({
api_url: form.api_url,
api_key: form.api_key,
online_model_name: form.online_model_name,
})
const data = res as any
if (data.success) {
data.latency_ms = Date.now() - start
ElMessage.success(`模型 ${data.model || form.online_model_name} 连接成功 (${data.latency_ms}ms)`)
} else {
ElMessage.error(data.error || '连接失败')
}
testResult.value = data
} catch (e: any) {
ElMessage.error(e?.message || e?.response?.data?.message || '测试请求失败')
testResult.value = { success: false, error: String(e) }
} finally {
testLoading.value = false
}
}
function handleCancel() { function handleCancel() {
router.back() router.back()
} }
@@ -217,7 +252,25 @@ onMounted(() => {
<el-input v-model="form.api_key" type="password" show-password placeholder="请输入 API Key" /> <el-input v-model="form.api_key" type="password" show-password placeholder="请输入 API Key" />
</el-form-item> </el-form-item>
<el-form-item label="模型名称" prop="online_model_name"> <el-form-item label="模型名称" prop="online_model_name">
<div class="model-test-row">
<el-input v-model="form.online_model_name" placeholder="如gpt-4、qwen-turbo" /> <el-input v-model="form.online_model_name" placeholder="如gpt-4、qwen-turbo" />
<el-button
type="primary"
:loading="testLoading"
:disabled="!form.api_url || !form.online_model_name"
@click="handleTestOnline"
>测试连接</el-button>
</div>
<!-- 测试结果 -->
<div v-if="testResult" class="test-result" :class="{ success: testResult.success, error: !testResult.success }">
<template v-if="testResult.success">
<i class="fa fa-check-circle" /> 连接成功 · 模型{{ testResult.model }} · 耗时{{ testResult.latency_ms }}ms
<span v-if="testResult.usage" class="usage-info">tokens: {{ (testResult.usage as any)?.total_tokens }}</span>
</template>
<template v-else>
<i class="fa fa-exclamation-circle" /> {{ testResult.error }}
</template>
</div>
</el-form-item> </el-form-item>
</template> </template>
@@ -255,4 +308,41 @@ onMounted(() => {
color: #64748b; color: #64748b;
cursor: help; cursor: help;
} }
.model-test-row {
display: grid;
grid-template-columns: minmax(0, 1fr) auto;
align-items: center;
gap: 8px;
width: 100%;
}
.test-result {
margin-top: 8px;
padding: 10px 14px;
border-radius: 6px;
font-size: 13px;
&.success {
background: #f0fdf4;
color: #166534;
border: 1px solid #bbf7d0;
}
&.error {
background: #fef2f2;
color: #991b1b;
border: 1px solid #fecaca;
}
i {
margin-right: 6px;
}
.usage-info {
margin-left: 12px;
color: #64748b;
font-size: 12px;
}
}
</style> </style>

View File

@@ -2,6 +2,7 @@
import { computed, onMounted, reactive, ref } from 'vue' import { computed, onMounted, reactive, ref } from 'vue'
import { ElMessage, ElMessageBox } from 'element-plus' import { ElMessage, ElMessageBox } from 'element-plus'
import { import {
changeMyPassword,
deleteUser, deleteUser,
getUsers, getUsers,
resetUserPassword, resetUserPassword,
@@ -106,6 +107,34 @@ async function confirmResetPwd() {
} }
} }
// ---------- 用户自改密码 ----------
const myPwdDialog = reactive({ visible: false, oldPassword: '', newPassword: '', saving: false })
function openChangeMyPwd() {
myPwdDialog.oldPassword = ''
myPwdDialog.newPassword = ''
myPwdDialog.visible = true
}
async function confirmChangeMyPwd() {
if (!myPwdDialog.oldPassword.trim() || !myPwdDialog.newPassword.trim()) {
ElMessage.warning('请填写旧密码和新密码')
return
}
if (myPwdDialog.newPassword.length < 6) {
ElMessage.warning('新密码至少 6 位')
return
}
myPwdDialog.saving = true
try {
await changeMyPassword(myPwdDialog.oldPassword.trim(), myPwdDialog.newPassword.trim())
ElMessage.success('密码修改成功')
myPwdDialog.visible = false
} catch {
ElMessage.error('密码修改失败,请检查旧密码是否正确')
} finally {
myPwdDialog.saving = false
}
}
// ---------- 页面权限 ---------- // ---------- 页面权限 ----------
const permDialog = reactive({ const permDialog = reactive({
visible: false, visible: false,
@@ -117,10 +146,24 @@ const permDialog = reactive({
function openPerms(row: SystemUser) { function openPerms(row: SystemUser) {
permDialog.id = row.id permDialog.id = row.id
permDialog.name = row.display_name permDialog.name = row.display_name
permDialog.checked = [...(row.permissions || [])] // admin 用户强制全选且只读
if (row.role === 'admin' || row.protected) {
permDialog.checked = [...ALL_PERMISSIONS]
} else {
// 非 admin 用户去掉 user-settings
permDialog.checked = (row.permissions || []).filter((p) => p !== 'user-settings')
}
permDialog.visible = true permDialog.visible = true
} }
async function confirmPerms() { async function confirmPerms() {
if (permReadonly.value) {
permDialog.visible = false
return
}
// 双重保险:非 admin 用户不允许勾选 user-settings
if (!isTargetAdmin.value) {
permDialog.checked = permDialog.checked.filter((p) => p !== 'user-settings')
}
permDialog.saving = true permDialog.saving = true
try { try {
await updateUserAccess(permDialog.id, { permissions: permDialog.checked }) await updateUserAccess(permDialog.id, { permissions: permDialog.checked })
@@ -134,7 +177,17 @@ async function confirmPerms() {
} }
} }
const permColumns = computed(() => ALL_PERMISSIONS) /** 权限列admin 用户全选且只读,非 admin 用户不显示 user-settings */
const isTargetAdmin = computed(() => {
const u = users.value.find((u) => u.id === permDialog.id)
return u?.role === 'admin' || u?.protected === true
})
const permColumns = computed(() => {
if (isTargetAdmin.value) return ALL_PERMISSIONS
// 非 admin 用户不能拥有 user-settings 权限
return ALL_PERMISSIONS.filter((c) => c !== 'user-settings')
})
const permReadonly = computed(() => isTargetAdmin.value)
// ---------- 删除 ---------- // ---------- 删除 ----------
async function removeUser(row: SystemUser) { async function removeUser(row: SystemUser) {
@@ -165,7 +218,10 @@ async function removeUser(row: SystemUser) {
<h1>用户设置</h1> <h1>用户设置</h1>
<p>管理平台账号角色状态登录密码与页面权限</p> <p>管理平台账号角色状态登录密码与页面权限</p>
</div> </div>
<div>
<el-button @click="openChangeMyPwd">修改密码</el-button>
<el-button type="primary" @click="$router.push('/user-settings/create')">创建用户</el-button> <el-button type="primary" @click="$router.push('/user-settings/create')">创建用户</el-button>
</div>
</header> </header>
<el-table :data="users" border> <el-table :data="users" border>
@@ -236,8 +292,11 @@ async function removeUser(row: SystemUser) {
<!-- 页面权限 --> <!-- 页面权限 -->
<el-dialog v-model="permDialog.visible" title="页面权限" width="540px"> <el-dialog v-model="permDialog.visible" title="页面权限" width="540px">
<p class="dlg-tip"> <b>{{ permDialog.name }}</b> 分配可访问的页面模块</p> <p class="dlg-tip">
<el-checkbox-group v-model="permDialog.checked" class="perm-group"> <b>{{ permDialog.name }}</b> 分配可访问的页面模块
<el-tag v-if="permReadonly" type="warning" size="small" style="margin-left: 8px">管理员权限不可更改</el-tag>
</p>
<el-checkbox-group v-model="permDialog.checked" class="perm-group" :disabled="permReadonly">
<el-checkbox <el-checkbox
v-for="code in permColumns" v-for="code in permColumns"
:key="code" :key="code"
@@ -246,8 +305,24 @@ async function removeUser(row: SystemUser) {
/> />
</el-checkbox-group> </el-checkbox-group>
<template #footer> <template #footer>
<el-button @click="permDialog.visible = false">取消</el-button> <el-button @click="permDialog.visible = false">{{ permReadonly ? '关闭' : '取消' }}</el-button>
<el-button type="primary" :loading="permDialog.saving" @click="confirmPerms">保存</el-button> <el-button v-if="!permReadonly" type="primary" :loading="permDialog.saving" @click="confirmPerms">保存</el-button>
</template>
</el-dialog>
<!-- 修改自己的密码 -->
<el-dialog v-model="myPwdDialog.visible" title="修改密码" width="420px">
<el-form label-width="80px">
<el-form-item label="旧密码">
<el-input v-model="myPwdDialog.oldPassword" placeholder="请输入当前密码" show-password />
</el-form-item>
<el-form-item label="新密码">
<el-input v-model="myPwdDialog.newPassword" placeholder="至少 6 位" show-password />
</el-form-item>
</el-form>
<template #footer>
<el-button @click="myPwdDialog.visible = false">取消</el-button>
<el-button type="primary" :loading="myPwdDialog.saving" @click="confirmChangeMyPwd">确认修改</el-button>
</template> </template>
</el-dialog> </el-dialog>
</section> </section>

View File

@@ -1,282 +0,0 @@
# 平台治理功能说明
平台治理涵盖用户管理、租户管理、项目管理、审批管理、审计日志、资源授权(ACL)、留存策略等企业管理能力。功能入口位于侧边栏「平台治理」和「系统设置」两个分组下。
本平台核心业务是**模型微调**:用户上传数据 → 数据处理 → 模型训练 → 模型评测 → 模型推理。平台治理负责管理**谁**能访问**哪个租户/项目**的**哪些资源**。
---
## 侧边栏菜单结构
```
平台治理
├── 租户管理 /tenants
├── 项目空间 /projects
├── 审计日志 /audit-logs
├── 审批模板 /approval-templates
└── 审批中心 /approval-instances
系统设置
├── 用户设置 /user-settings
├── 平台性能 /hardware
└── 查看日志 /logs
```
---
## 1. 租户管理
**对应页面**:列表页 `TenantListView.vue` + 详情页 `TenantDetailView.vue`
**后端接口**`GET/POST /tenants``GET/PUT/DELETE /tenants/:id``PUT /tenants/:id/quota``PUT /tenants/:id/retention-policy`
### 列表页
**展示列**租户名称、用户ID、状态、创建时间
**行操作**
| 按钮 | 功能 |
|------|------|
| 详情 | 跳转详情页 `/tenants/:id` |
| 删除 | 确认弹窗后调用 `DELETE /tenants/:id` |
**新建租户弹窗**
| 字段 | 控件 | 说明 |
|------|------|------|
| 名称 | 文本输入 | 必填 |
| 用户ID | 文本输入 | 可选 |
| GPU 数量 | 数字输入 | 配额0=不限制 |
| 存储配额(GB) | 数字输入 | 配额0=不限制 |
| 最大项目数 | 数字输入 | 配额0=不限制 |
### 详情页
**基本信息**名称、用户ID、状态、创建时间、配额友好格式显示`GPU 8 | 存储 100GB | 项目 5`
**配额编辑**GPU 数量 / 存储配额(GB) / 最大项目数,三个数字输入 + 保存按钮
**已移除**:详情页中不再内嵌项目空间列表(项目有独立页面)
### 当前缺失
- [ ] 编辑租户名称和用户ID`updateTenant` API 已有UI 缺失)
- [ ] 留存策略关联设置(`setTenantRetention` API 已有UI 缺失)
- [ ] 列表页无搜索/筛选
- [ ] 状态列无 Tag 着色
---
## 2. 项目空间
**对应页面**:列表页 `ProjectListView.vue` + 详情页 `ProjectDetailView.vue`
**后端接口**`GET/POST /projects``GET/PUT/DELETE /projects/:id``GET/POST/DELETE /projects/:id/members``PUT /projects/:id/archive`
### 列表页
**展示列**项目名、编码ID、状态、描述、创建时间
**工具栏**
- 租户选择器(下拉过滤,按租户编码选择)
- 搜索框(按项目名/编码搜索)
- 新建项目按钮
**新建项目弹窗**
| 字段 | 控件 | 说明 |
|------|------|------|
| 名称 | 文本输入 | 必填 |
| 编码ID | 下拉选择 | 选择已有租户的编码(如 `default`),关联到该租户 |
| 描述 | 文本域 | 可选3行 |
**行操作**:详情 + 删除
### 详情页
**基本信息**:名称、编码、状态、租户、描述、创建时间
**项目成员管理**
- 成员列表(用户名、角色、添加时间)
- 添加成员:选择用户 + 角色member/admin/viewer
- 移除成员
**资源授权(ACL)**
- 弹窗编辑器,逐条配置授权规则
- 主体类型用户下拉选已有用户或项目角色member/admin/viewer
- 权限read / write / execute / download / delete / share多选 checkbox
- 增删行后统一保存
### 当前缺失
- [ ] 编辑项目基本信息(名称、描述)
- [ ] 项目启停/归档操作(`archiveProject` API 已有UI 缺失)
- [ ] 项目下的资源使用统计(模型数、数据集数、任务数)
---
## 3. 审批管理
**对应页面**`ApprovalTemplateView.vue`(模板)+ `ApprovalInstanceView.vue`(实例)
**后端接口**:模板 CRUD、实例列表+决策
### 审批模板
**展示列**:模板名、步骤数、创建时间
**新建模板**
| 字段 | 控件 | 说明 |
|------|------|------|
| 模板名称 | 文本输入 | 必填 |
| 审批步骤 | 文本域 | **需手写 JSON 字符串**,体验差 |
**当前缺失**
- [ ] 可视化步骤编辑(拖拽添加步骤、选审批人)
- [ ] 模板编辑/删除API 已有UI 缺失)
- [ ] 模板详情页
### 审批中心
**展示列**资源类型、资源ID、状态Tag着色、发起人、创建时间
**工具栏**:状态筛选(待审批/已通过/已拒绝)、资源类型/ID 搜索
**行操作**:通过/拒绝(弹窗填写审批意见)
**当前缺失**
- [ ] "我发起的"审批视角
- [ ] 审批流转详情(谁审批了、什么时间)
- [ ] 撤回功能
---
## 4. 审计日志
**对应页面**`AuditLogView.vue`
**后端接口**`GET /system/audit-logs`(多条件筛选+分页)、`GET /system/audit-logs/export`CSV导出
### 列表页
**展示列**时间、租户ID、项目ID、操作人ID、动作、目标类型、目标ID、详情、IP
**筛选条件**租户ID、项目ID、操作人ID、动作、目标类型、开始时间、结束时间
**工具栏**CSV 导出按钮(最多 10000 条)
### 当前缺失
- [ ] 分页控件(数据量大时需要翻页)
- [ ] 筛选字段改为下拉选择(当前全是文本输入,不知道有哪些可选值)
- [ ] 详情弹窗(点击某条记录查看完整信息)
---
## 5. 用户设置
**对应页面**`UserSettingsView.vue`(列表)+ `UserCreateView.vue`(创建)+ `UserPermissionView.vue`(权限弹窗)
**后端接口**`GET/POST/PUT/DELETE /users``POST /users/:id/reset-password`
### 列表页
**展示列**:账号、显示名、角色、状态、页面权限、创建时间
**行操作**
| 操作 | 说明 |
|------|------|
| 启停开关 | protected 用户和自己不可操作 |
| 重置密码 | 弹窗确认,默认密码 `Platform@123` |
| 页面权限 | 弹窗 checkbox 组12 个模块可选:看板/模型训练/模型评测/模型推理/模型管理/数据集/数据处理/数据转换/算力/平台性能/查看日志/用户设置 |
| 删除 | 确认弹窗protected 用户和自己不可操作 |
### 创建页
**表单字段**:账号、显示名、初始密码、角色(超级管理员/操作员/观察员)、状态、页面权限
### 当前缺失
- [ ] 用户编辑页面(修改显示名、角色等,`updateUser` API 已有UI 缺失)
- [ ] 权限码显示为英文(如 `fine-tune`),无中文翻译
---
## 6. 平台性能
**对应页面**`HardwareView.vue`
**后端接口**`GET /system-info``GET /health`(顶部栏实时指标)
### 页面内容
**概览卡片**CPU型号/核心数/使用率)、内存(已用/总量/使用率)、磁盘(已用/总量/使用率)、网络吞吐
**趋势图**CPU/内存/磁盘/GPU 利用率折线图(最近 60 次采样1/3/5/10秒自动刷新
**GPU 资源池**:每张卡展示名称、状态、利用率、显存、温度、功耗、风扇转速;点击卡片查看详情抽屉(设备属性、趋势图、进程列表)
**主机信息**OS、运行时长、进程数、GPU 驱动版本
### 当前缺失
- [ ] 历史数据持久化(刷新后采样清空)
- [ ] 告警阈值设置
- [ ] 网络吞吐数据显示为空
---
## 7. 查看日志
**对应页面**`LogsView.vue`(系统日志 + 训练日志双 Tab
### 系统日志
- 按日期选择日志文件
- 关键词搜索
- 日志级别筛选INFO/WARN/ERROR/DEBUG
- 自动刷新5/10/30/60 秒可调)
### 训练日志
- 按 PID 选择训练日志文件
- 同样支持搜索和级别筛选
### 当前缺失
- [ ] 日志下载/导出
- [ ] 分页或虚拟滚动(大文件加载慢)
- [ ] 行号显示
---
## 8. 留存策略 ⚠️ 前端完全缺失
**后端 API 已完整实现**CRUD 5 个端点),**前端 `retention.ts` 模块已封装**,但:
- 侧边栏无菜单入口
- 路由未配置
- 无任何 Vue 页面
---
## 数据流转关系
```
用户登录 → 分配角色(admin/operator/viewer) + 页面权限
├─ 创建租户配额GPU/存储/项目数)
│ └─ 关联用户
├─ 创建项目关联租户选择编码ID
│ ├─ 项目成员角色member/admin/viewer
│ └─ 资源授权(ACL):谁对什么资源有什么权限
├─ 创建审批模板(定义审批流程)
│ └─ 审批实例:敏感操作需要审批(通过/拒绝)
├─ 审计日志:所有操作自动记录(谁在什么时间做了什么)
└─ 留存策略:定义数据保留周期,自动清理过期数据
```
---
## 完成度总表
| 模块 | 列表 | 新建 | 编辑 | 删除 | 搜索 | 筛选 | 导出 | 完成度 |
|------|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
| 租户管理 | ✅ | ✅ | 仅配额 | ✅ | ❌ | ❌ | ❌ | 75% |
| 项目空间 | ✅ | ✅ | ❌ | ✅ | ✅ | 按租户 | ❌ | 80% |
| 审批模板 | ✅ | ✅(JSON) | ❌ | ❌ | ❌ | ❌ | ❌ | 45% |
| 审批中心 | ✅ | N/A | N/A | ❌ | ✅ | 按状态 | ❌ | 65% |
| 审计日志 | ✅ | N/A | N/A | N/A | ❌ | 多字段 | CSV | 70% |
| 用户设置 | ✅ | ✅ | 仅权限 | ✅ | ❌ | ❌ | ❌ | 75% |
| 平台性能 | ✅ | N/A | N/A | N/A | N/A | N/A | ❌ | 80% |
| 查看日志 | ✅ | N/A | N/A | N/A | 关键词 | 按级别 | ❌ | 70% |
| **留存策略** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | **0%** |

380
测试脚本.md Normal file
View File

@@ -0,0 +1,380 @@
# YG_FT 模型微调平台测试脚本
## 环境准备
### 启动顺序
1. **数据库**:远程 PostgreSQL `www.caoxiaozhu.com:5432`(已就绪)
2. **后端**
```cmd
cd /d E:\yg_ft\backend
set COMPUTE_STATUS_SYNC_MODE=off
.\.venv\Scripts\python.exe -m uvicorn app.main:app --reload --port 17861
```
3. **算力服务**Docker
```cmd
cd /d E:\yg_ft
docker build -f docker/compute/Dockerfile.compute -t yg-ft-compute-api:latest .
cd docker/compute
docker compose up -d
```
4. **前端**
```cmd
cd /d E:\yg_ft\frontend
npm run dev
```
### 验证服务可用性
| 服务 | 验证命令 | 期望结果 |
|------|---------|---------|
| 后端 | `curl http://localhost:17861/modelTF/health` | `{"code":0,...,"data":{"cpu_percent":x,...}}` |
| 算力 | `curl http://localhost:19100/health` | `{"status":"ok","compute_host_id":"gpu-node-01"}` |
| 前端 | 浏览器打开 `http://localhost:16801` | 跳转登录页 |
---
## 一、登录与认证
### 1.1 登录
| 步骤 | 操作 | 预期结果 |
|------|------|---------|
| 1 | 打开 `http://localhost:16801` | 显示登录页,不自动登录 |
| 2 | 输入账号 `admin`,密码 `admin123`,点击登录 | 提示"登录成功",跳转看板页 |
| 3 | F12 控制台执行 `sessionStorage.getItem('sessionId')` | 有值(非 null |
| 4 | 浏览器 Network 查看 `/modelTF/login` 响应 | 返回 `token`、`user`、`session_id` |
### 1.2 退出登录
| 步骤 | 操作 | 预期结果 |
|------|------|---------|
| 1 | 点击侧边栏底部"退出登录"按钮 | 跳转登录页 |
| 2 | F12 控制台执行 `sessionStorage.getItem('hasAuthed')` | 返回 null |
| 3 | 浏览器 Network 查看 `/modelTF/logout` 请求 | 状态 200 |
### 1.3 刷新页面保持登录
| 步骤 | 操作 | 预期结果 |
|------|------|---------|
| 1 | 登录后按 F5 刷新页面 | 仍在功能页,不跳登录页 |
### 1.4 关闭标签页再打开
| 步骤 | 操作 | 预期结果 |
|------|------|---------|
| 1 | 登录后关闭浏览器标签页 | — |
| 2 | 重新打开 `http://localhost:16801` | 跳转登录页sessionStorage 已清空) |
### 1.5 会话超时
| 步骤 | 操作 | 预期结果 |
|------|------|---------|
| 1 | 登录后切到其他软件,等待 5 分钟以上 | — |
| 2 | 切回浏览器 | 提示"登录已过期",跳转登录页 |
---
## 二、看板
### 2.1 服务状态
| 步骤 | 操作 | 预期结果 |
|------|------|---------|
| 1 | 登录后进入看板页 | 显示 7 个服务:模型训练/模型评测/模型推理/模型管理/数据集管理/数据处理/数据类型转换 |
| 2 | 检查每个服务状态 | 全部显示"正常" |
| 3 | 检查实例数 | 模型管理有数量、数据集管理有数量等 |
### 2.2 运行中任务
| 步骤 | 操作 | 预期结果 |
|------|------|---------|
| 1 | 查看顶部"运行中任务"数字 | 显示训练+评测+数据处理中 running 状态的任务总数 |
| 2 | 无运行任务时 | 显示 0 |
### 2.3 用户操作分布
| 步骤 | 操作 | 预期结果 |
|------|------|---------|
| 1 | 查看饼图 | 4 个分类(数据处理/模型训练/模型评测/模型推理)各一色 |
| 2 | 无数据时 | 4 等分(各 25% |
| 3 | 点击侧边栏"模型训练"菜单,回到看板 | 模型训练权重增加 |
| 4 | 点击"模型推理"菜单,回到看板 | 模型推理权重增加 |
| 5 | 浏览器控制台执行 `sessionStorage.clear()` 后重试 | 去重缓存清除,可再次记录 |
### 2.4 登录时长排行
| 步骤 | 操作 | 预期结果 |
|------|------|---------|
| 1 | 查看柱状图 | 显示当前登录用户,时长 > 0 |
| 2 | 条形图上无文字标签 | 只在鼠标悬浮时显示 tooltip |
| 3 | Y 轴用户名清晰可读 | 字体颜色 `#1f2937`,非模糊 |
### 2.5 顶部状态栏
| 步骤 | 操作 | 预期结果 |
|------|------|---------|
| 1 | 查看顶部 CPU/内存/磁盘 | 显示真实数值(非 0 |
| 2 | 等待 30 秒 | 数值自动刷新 |
---
## 三、租户管理
### 3.1 列表
| 步骤 | 操作 | 预期结果 |
|------|------|---------|
| 1 | 侧边栏 → 平台治理 → 租户管理 | 显示列表(名称/用户ID/状态/创建时间) |
| 2 | 检查列名 | "编码"已改为"用户ID" |
| 3 | 检查操作列 | 有"详情"和"删除"按钮,无"配额"按钮 |
### 3.2 新建租户
| 步骤 | 操作 | 预期结果 |
|------|------|---------|
| 1 | 点击"新建租户" | 弹窗显示名称、用户ID、GPU数量、存储配额、最大项目数 |
| 2 | 填写名称"测试租户"GPU=2存储=100项目=5点击创建 | 提示"租户创建成功",列表新增一行 |
### 3.3 租户详情与配额
| 步骤 | 操作 | 预期结果 |
|------|------|---------|
| 1 | 点击"详情" | 跳转详情页,显示基本信息 + 配额 |
| 2 | 配额显示格式 | `GPU 2 | 存储 100GB | 项目 5` |
| 3 | 修改配额数字,点击"保存配额" | 提示"配额已保存" |
| 4 | 检查详情页无"项目空间"卡片 | 已移除 |
### 3.4 删除租户
| 步骤 | 操作 | 预期结果 |
|------|------|---------|
| 1 | 在列表页点击"删除" | 弹出确认框 |
| 2 | 点击"确定删除" | 提示"租户已删除",列表刷新 |
---
## 四、项目空间
### 4.1 列表
| 步骤 | 操作 | 预期结果 |
|------|------|---------|
| 1 | 侧边栏 → 项目空间 | 显示列表(项目名/编码ID/状态/描述/创建时间) |
| 2 | 检查列名 | "编码"已改为"编码ID" |
| 3 | 顶部有租户筛选下拉框 | 可按租户过滤 |
### 4.2 新建项目
| 步骤 | 操作 | 预期结果 |
|------|------|---------|
| 1 | 点击"新建项目" | 弹窗显示名称、编码ID下拉、描述 |
| 2 | 编码ID下拉选项 | 显示已有租户的编码 |
| 3 | 选择编码ID后提交 | 提示"项目创建成功" |
### 4.3 项目详情
| 步骤 | 操作 | 预期结果 |
|------|------|---------|
| 1 | 点击"详情" | 显示基本信息 |
| 2 | 查看成员管理区域 | 显示成员列表 + 添加成员按钮 |
| 3 | 点击"资源授权(ACL)" | 弹窗显示 ACL 编辑器 |
### 4.4 ACL 授权
| 步骤 | 操作 | 预期结果 |
|------|------|---------|
| 1 | 点击"添加授权项" | 新增一行 |
| 2 | 主体类型选"用户" | 显示用户下拉框(非文本输入) |
| 3 | 主体类型选"项目角色" | 显示角色下拉框member/admin/viewer |
| 4 | 勾选权限后保存 | 提示"ACL 已保存" |
### 4.5 删除项目
| 步骤 | 操作 | 预期结果 |
|------|------|---------|
| 1 | 点击"删除" | 确认弹窗 |
| 2 | 确认删除 | 提示"项目已删除" |
---
## 五、算力节点
### 5.1 节点列表
| 步骤 | 操作 | 预期结果 |
|------|------|---------|
| 1 | 侧边栏 → 算力节点 | 显示节点列表 |
| 2 | 检查 gpu-node-01 | 状态为启用 |
### 5.2 测试连接
| 步骤 | 操作 | 预期结果 |
|------|------|---------|
| 1 | 点击 gpu-node-01 的"测试"按钮 | 提示"连接成功,发现 X 张 GPU延迟 Xms" |
| 2 | 检查 GPU Tab | 显示 GPU 资源列表 |
### 5.3 节点启停
| 步骤 | 操作 | 预期结果 |
|------|------|---------|
| 1 | 点击"停用" | 节点状态变为停用 |
| 2 | 点击"启用" | 节点状态变为启用 |
---
## 六、用户设置
### 6.1 用户列表
| 步骤 | 操作 | 预期结果 |
|------|------|---------|
| 1 | 侧边栏 → 系统设置 → 用户设置 | 显示用户列表 |
| 2 | admin 用户 | 不可停用、不可删除 |
### 6.2 创建用户
| 步骤 | 操作 | 预期结果 |
|------|------|---------|
| 1 | 点击"创建用户" | 跳转创建页 |
| 2 | 填写账号/显示名/密码/角色/权限 | 表单正常 |
| 3 | 提交 | 提示创建成功,列表新增 |
### 6.3 权限分配
| 步骤 | 操作 | 预期结果 |
|------|------|---------|
| 1 | 点击某用户"页面权限" | 弹窗显示 12 个权限 checkbox |
| 2 | 取消某权限,保存 | 该用户侧边栏不再显示对应菜单 |
### 6.4 重置密码
| 步骤 | 操作 | 预期结果 |
|------|------|---------|
| 1 | 点击"重置密码" | 弹窗显示默认密码 `Platform@123` |
| 2 | 确认 | 提示重置成功 |
---
## 七、审批管理
### 7.1 审批模板
| 步骤 | 操作 | 预期结果 |
|------|------|---------|
| 1 | 侧边栏 → 审批模板 | 显示模板列表 |
| 2 | 点击新建 | 弹窗显示名称 + 步骤 JSON |
| 3 | 填写并提交 | 列表新增 |
### 7.2 审批中心
| 步骤 | 操作 | 预期结果 |
|------|------|---------|
| 1 | 侧边栏 → 审批中心 | 显示实例列表 |
| 2 | 按状态筛选 | 列表过滤正常 |
| 3 | 对 pending 实例点"通过/拒绝" | 弹窗填写意见,提交后状态变更 |
---
## 八、审计日志
| 步骤 | 操作 | 预期结果 |
|------|------|---------|
| 1 | 侧边栏 → 审计日志 | 显示日志列表 |
| 2 | 填写筛选条件查询 | 列表过滤 |
| 3 | 点击"导出" | 下载 CSV 文件 |
---
## 九、平台性能
| 步骤 | 操作 | 预期结果 |
|------|------|---------|
| 1 | 侧边栏 → 平台性能 | 显示 CPU/内存/磁盘/GPU 卡片 |
| 2 | 数据非 0 | 与本机实际使用率一致 |
| 3 | 开启自动刷新3秒 | 趋势图实时更新 |
| 4 | 点击 GPU 卡片 | 弹出详情抽屉 |
---
## 十、查看日志
| 步骤 | 操作 | 预期结果 |
|------|------|---------|
| 1 | 侧边栏 → 查看日志 | 显示系统日志 Tab |
| 2 | 选择日期和文件 | 显示日志内容 |
| 3 | 输入关键词搜索 | 高亮匹配 |
| 4 | 切换到训练日志 Tab | 显示训练日志列表 |
---
## 十一、数据流主链路
### 11.1 数据集管理
| 步骤 | 操作 | 预期结果 |
|------|------|---------|
| 1 | 进入数据集管理 | 显示列表 |
| 2 | 新建数据集 | 创建成功 |
| 3 | 上传文件 | 文件上传成功 |
| 4 | 预览文件 | 显示内容 |
### 11.2 数据类型转换
| 步骤 | 操作 | 预期结果 |
|------|------|---------|
| 1 | 侧边栏 → 数据类型转换 | 显示任务列表 |
| 2 | 点击"新建转换任务",填写名称,点击创建 | 列表新增一行,状态为"待上传" |
| 3 | 点击"上传文件",选择 `.json` 文件 | 上传成功后自动转换为 JSONL 并导入数据集 |
| 4 | 检查任务状态 | 变为"已完成",显示输入/输出数 |
| 5 | 进入数据集管理 → 上传任务 Tab | 可看到自动导入的数据集 |
| 6 | 上传非 `.json` 文件 | 提示"仅支持 .json 文件" |
| 7 | 点击"删除" | 确认后任务删除 |
### 11.3 数据处理
| 步骤 | 操作 | 预期结果 |
|------|------|---------|
| 1 | 进入数据处理 | 显示任务列表 |
| 2 | 创建数据处理任务 | 创建成功 |
| 3 | 启动任务 | 状态变为 running |
| 4 | 回到看板 | "运行中任务"数字 +1 |
### 11.4 模型训练
| 步骤 | 操作 | 预期结果 |
|------|------|---------|
| 1 | 进入模型训练 | 显示训练任务列表 |
| 2 | 创建训练任务 | 选择数据集 + 模型 + GPU |
| 3 | 启动训练 | 状态变为 running |
| 4 | 查看训练日志 | 显示实时日志 |
| 5 | 训练完成 | 产出模型 |
### 11.5 模型评测
| 步骤 | 操作 | 预期结果 |
|------|------|---------|
| 1 | 进入模型评测 | 显示评测任务列表 |
| 2 | 创建评测任务 | 选择模型 + 数据集 |
| 3 | 启动评测 | 状态变为 running |
| 4 | 评测完成 | 显示指标结果 |
### 11.6 模型推理
| 步骤 | 操作 | 预期结果 |
|------|------|---------|
| 1 | 进入模型推理 | 显示推理会话列表 |
| 2 | 创建推理任务 | 选择模型 + 加载 |
| 3 | 开始对话 | 返回模型回复 |
---
## 十二、异常场景
| 场景 | 操作 | 预期结果 |
|------|------|---------|
| 数据库断开 | 停止远程数据库连接 | 所有接口返回 500 |
| 算力服务停掉 | `docker compose down` | 算力测试返回失败 |
| 未登录访问 | 直接访问 `/dashboard` | 跳转登录页 |
| 权限不足 | 普通用户访问用户设置 | 跳转 403 页面 |
| Token 过期 | 关闭标签页再打开 | 跳转登录页sessionStorage 清空) |