Compare commits
15 Commits
15c4223f2c
...
baseline/f
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
ec7d8c0a3d | ||
|
|
0292bf5138 | ||
|
|
0271942ba5 | ||
|
|
250e060271 | ||
|
|
7b36bc774e | ||
|
|
4e5c43fad5 | ||
|
|
62a1d03eac | ||
|
|
94230cad16 | ||
|
|
0c601934a0 | ||
|
|
5cc306eb0a | ||
|
|
cc08b164d0 | ||
|
|
24c77a990a | ||
|
|
46d343fb63 | ||
|
|
0c39f2f5b9 | ||
|
|
c7c9ed925b |
39
README.md
39
README.md
@@ -134,12 +134,45 @@ npm run dev
|
|||||||
|
|
||||||
## 算力服务启动
|
## 算力服务启动
|
||||||
|
|
||||||
|
算力服务是一个 FastAPI 应用,同时承载 Compute API(模型训练/推理/GPU 管理)和 File Gateway(文件上传下载)路由。Docker 部署时对外暴露两个端口(19100 和 19101)均指向同一服务,方便应用平台分别配置 `api_base_url` 和 `file_gateway_url`。本地开发只需启动一个进程。
|
||||||
|
|
||||||
|
### 方式一:Docker 启动(推荐)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
cd compute
|
cd docker/compute
|
||||||
uvicorn api.main:app --reload --port 19100
|
cp .env.example .env
|
||||||
|
docker compose up -d
|
||||||
```
|
```
|
||||||
|
|
||||||
默认 `COMPUTE_MODE=real`。真实 GPU 接入时,在每台算力服务器上部署 Compute API、Agent、File Gateway 和 LLaMA-Factory,应用平台通过 `compute_nodes.api_base_url` 和 `compute_nodes.file_gateway_url` 主动轮询。仅在隔离联调环境可显式设置 `COMPUTE_MODE=simulator` 或 `COMPUTE_EXECUTION_MODE=simulator`。
|
### 方式二:本地开发启动
|
||||||
|
|
||||||
|
**Windows (cmd):**
|
||||||
|
|
||||||
|
```cmd
|
||||||
|
cd /d E:\yg_ft\compute
|
||||||
|
set PYTHONPATH=E:\yg_ft
|
||||||
|
.\.venv\Scripts\python.exe -m uvicorn api.main:app --reload --port 19100
|
||||||
|
```
|
||||||
|
|
||||||
|
> `PYTHONPATH=E:\yg_ft` 是必需的,因为代码使用 `from compute.agent...` 绝对导入。
|
||||||
|
|
||||||
|
**Linux / macOS:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd compute
|
||||||
|
PYTHONPATH=.. uvicorn api.main:app --reload --port 19100
|
||||||
|
```
|
||||||
|
|
||||||
|
### 环境变量说明
|
||||||
|
|
||||||
|
| 变量 | 默认值 | 说明 |
|
||||||
|
|---|---|---|
|
||||||
|
| `COMPUTE_MODE` | `real` | `real` / `simulator`,仅隔离联调用 simulator |
|
||||||
|
| `COMPUTE_EXECUTION_MODE` | `real` | 训练执行模式 |
|
||||||
|
| `COMPUTE_SERVICE_TOKEN` | `change_me` | 服务间认证 token |
|
||||||
|
| `MODELTF_ROUTE_PREFIX` | `/modelTF` | API 路由前缀 |
|
||||||
|
|
||||||
|
应用平台通过数据库 `compute_nodes` 表中的 `api_base_url` 和 `file_gateway_url` 主动轮询算力节点状态。
|
||||||
|
|
||||||
## 日志
|
## 日志
|
||||||
|
|
||||||
|
|||||||
10
backend/_check_sessions.py
Normal file
10
backend/_check_sessions.py
Normal file
@@ -0,0 +1,10 @@
|
|||||||
|
from app.db.platform_store import get_platform_store
|
||||||
|
|
||||||
|
store = get_platform_store()
|
||||||
|
with store.connect() as conn:
|
||||||
|
rows = conn.execute(
|
||||||
|
"SELECT id, user_id, login_at, logout_at, duration_seconds FROM sessions ORDER BY login_at DESC LIMIT 10"
|
||||||
|
).fetchall()
|
||||||
|
print(f"sessions count: {len(rows)}")
|
||||||
|
for r in rows:
|
||||||
|
print(f" user={r['user_id'][:25]}... login={r['login_at']} logout={r['logout_at']} dur={r['duration_seconds']}")
|
||||||
@@ -10,5 +10,9 @@ logger = get_logger(__name__)
|
|||||||
@router.get("/health")
|
@router.get("/health")
|
||||||
async def health_check() -> dict[str, object]:
|
async def health_check() -> dict[str, object]:
|
||||||
logger.info("health check requested")
|
logger.info("health check requested")
|
||||||
return {"code": 0, "message": "ok", "data": get_platform_store().health_metrics()}
|
return {
|
||||||
|
"code": 0,
|
||||||
|
"message": "ok",
|
||||||
|
"data": get_platform_store().health_metrics(),
|
||||||
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -15,7 +15,7 @@ from app.core.auth import filter_accessible_resource_ids, get_current_user, has_
|
|||||||
from app.core.config import get_settings
|
from app.core.config import get_settings
|
||||||
from app.db.platform_store import get_platform_store
|
from app.db.platform_store import get_platform_store
|
||||||
from app.modules.compute_gateway.client import ComputeNodeClient
|
from app.modules.compute_gateway.client import ComputeNodeClient
|
||||||
from app.modules.compute_gateway.sync import poll_compute_jobs_once
|
from app.modules.compute_gateway.sync import fetch_eval_result_content, poll_compute_jobs_once
|
||||||
|
|
||||||
router = APIRouter()
|
router = APIRouter()
|
||||||
|
|
||||||
@@ -33,6 +33,124 @@ def _select_first_online_node(store: Any) -> dict[str, Any] | None:
|
|||||||
return None
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _select_eval_node(store: Any, preferred_node_id: str | None = None) -> dict[str, Any] | None:
|
||||||
|
"""Select the compute node for an eval job.
|
||||||
|
|
||||||
|
被评测模型是节点相关的(训练/合并产物只存在于对应算力节点),因此优先使用
|
||||||
|
页面选择的节点或模型所在节点;若该节点不可用则明确失败,绝不派发到其它
|
||||||
|
可能没有模型路径的节点(多算力节点场景下这是评测失败的主因)。
|
||||||
|
"""
|
||||||
|
if preferred_node_id:
|
||||||
|
node = next((n for n in store.compute_nodes() if n.get("id") == preferred_node_id), None)
|
||||||
|
if node:
|
||||||
|
if node.get("enabled") and node.get("scheduler_status") == "online":
|
||||||
|
return node
|
||||||
|
return None
|
||||||
|
return _select_first_online_node(store)
|
||||||
|
|
||||||
|
|
||||||
|
def _candidate_online_nodes(store: Any, preferred_node_id: str | None = None) -> list[dict[str, Any]]:
|
||||||
|
nodes = [node for node in store.compute_nodes() if node.get("enabled") and node.get("scheduler_status") == "online"]
|
||||||
|
if not preferred_node_id:
|
||||||
|
return nodes
|
||||||
|
preferred = [node for node in nodes if node.get("id") == preferred_node_id]
|
||||||
|
others = [node for node in nodes if node.get("id") != preferred_node_id]
|
||||||
|
return preferred + others
|
||||||
|
|
||||||
|
|
||||||
|
def _build_messages_payload(payload: dict[str, Any]) -> dict[str, Any]:
|
||||||
|
"""Convert frontend inference payload to compute API messages format.
|
||||||
|
|
||||||
|
Accepts both:
|
||||||
|
- OpenAI-style: {messages: [{role, content}, ...], temperature, ...}
|
||||||
|
- Frontend-style: {user_question, system_prompt, temperature, ...}
|
||||||
|
"""
|
||||||
|
if payload.get("messages"):
|
||||||
|
messages = payload["messages"]
|
||||||
|
# messages already in OpenAI format; pass through with optional system prompt
|
||||||
|
if payload.get("system_prompt") and not any(m.get("role") == "system" for m in messages):
|
||||||
|
messages = [{"role": "system", "content": payload["system_prompt"]}] + list(messages)
|
||||||
|
else:
|
||||||
|
messages = []
|
||||||
|
if payload.get("system_prompt"):
|
||||||
|
messages.append({"role": "system", "content": payload["system_prompt"]})
|
||||||
|
question = payload.get("user_question") or payload.get("question") or ""
|
||||||
|
if question:
|
||||||
|
messages.append({"role": "user", "content": question})
|
||||||
|
return {
|
||||||
|
"messages": messages,
|
||||||
|
"temperature": float(payload.get("temperature", 0.7)),
|
||||||
|
"top_p": float(payload.get("top_p", 0.95)),
|
||||||
|
"max_new_tokens": int(payload.get("max_tokens", 2048)),
|
||||||
|
"do_sample": bool(payload.get("do_sample", True)),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _node_for_inference_payload(store: Any, payload: dict[str, Any]) -> dict[str, Any] | None:
|
||||||
|
node_id = payload.get("node_id") or payload.get("compute_node_id")
|
||||||
|
task_id = payload.get("task_id") or payload.get("compare_task_id")
|
||||||
|
if task_id and not node_id:
|
||||||
|
try:
|
||||||
|
task = store.compare_task(str(task_id))
|
||||||
|
load_status = task.get("load_status") or {}
|
||||||
|
if isinstance(load_status, str):
|
||||||
|
load_status = json.loads(load_status)
|
||||||
|
loaded_models = load_status.get("loaded_models") or []
|
||||||
|
ready_model = next((item for item in loaded_models if item.get("status") in {"ready", "running"} and item.get("node_id")), None)
|
||||||
|
if ready_model:
|
||||||
|
node_id = ready_model.get("node_id")
|
||||||
|
except Exception:
|
||||||
|
node_id = None
|
||||||
|
if node_id:
|
||||||
|
return next((node for node in store.compute_nodes() if node.get("id") == node_id), None)
|
||||||
|
return _select_first_online_node(store)
|
||||||
|
|
||||||
|
|
||||||
|
async def _stream_chat_proxy(payload: dict[str, Any]) -> StreamingResponse:
|
||||||
|
"""Common SSE streaming proxy: convert payload → forward to compute node → stream back."""
|
||||||
|
store = get_platform_store()
|
||||||
|
# 任务仍在加载中时,直接返回明确的加载中提示,避免转发到尚未就绪的节点
|
||||||
|
task_id = payload.get("task_id") or payload.get("compare_task_id")
|
||||||
|
if task_id:
|
||||||
|
try:
|
||||||
|
task = store.compare_task(str(task_id))
|
||||||
|
load_status = task.get("load_status") or {}
|
||||||
|
if isinstance(load_status, str):
|
||||||
|
load_status = json.loads(load_status)
|
||||||
|
items = load_status.get("loaded_models") or []
|
||||||
|
if items and not any(item.get("status") in {"ready", "running"} for item in items):
|
||||||
|
if any(item.get("status") == "starting" for item in items):
|
||||||
|
return StreamingResponse(
|
||||||
|
iter(['data: {"error": "模型加载中,请稍候再试"}\n\n']),
|
||||||
|
media_type="text/event-stream",
|
||||||
|
)
|
||||||
|
except Exception: # noqa: BLE001 - fall through to normal routing on lookup errors
|
||||||
|
pass
|
||||||
|
node = _node_for_inference_payload(store, payload)
|
||||||
|
if not node:
|
||||||
|
return StreamingResponse(
|
||||||
|
iter(['data: {"error": "no online compute node available for inference"}\n\n']),
|
||||||
|
media_type="text/event-stream",
|
||||||
|
)
|
||||||
|
client = ComputeNodeClient(node["api_base_url"])
|
||||||
|
compute_payload = _build_messages_payload(payload)
|
||||||
|
|
||||||
|
async def stream_proxy():
|
||||||
|
async with httpx.AsyncClient(timeout=300) as http:
|
||||||
|
url = f"{node['api_base_url'].rstrip('/')}{client.route_prefix}/inference/chat/stream"
|
||||||
|
try:
|
||||||
|
async with http.stream("POST", url, json=compute_payload, headers=client.headers()) as resp:
|
||||||
|
if resp.status_code >= 400:
|
||||||
|
yield f'data: {{"error": "compute node returned {resp.status_code}"}}\n\n'.encode()
|
||||||
|
return
|
||||||
|
async for chunk in resp.aiter_bytes():
|
||||||
|
yield chunk
|
||||||
|
except Exception as exc:
|
||||||
|
yield f'data: {{"error": "stream proxy failed: {exc}"}}\n\n'.encode()
|
||||||
|
|
||||||
|
return StreamingResponse(stream_proxy(), media_type="text/event-stream")
|
||||||
|
|
||||||
|
|
||||||
def fail(status_code: int, message: str) -> HTTPException:
|
def fail(status_code: int, message: str) -> HTTPException:
|
||||||
return HTTPException(status_code=status_code, detail={"code": status_code, "message": message, "data": None})
|
return HTTPException(status_code=status_code, detail={"code": status_code, "message": message, "data": None})
|
||||||
|
|
||||||
@@ -255,10 +373,21 @@ async def _fine_tune_preflight_with_job_payload(
|
|||||||
|
|
||||||
@router.post("/login")
|
@router.post("/login")
|
||||||
async def login(payload: dict[str, Any] = Body(...)) -> dict[str, Any]:
|
async def login(payload: dict[str, Any] = Body(...)) -> dict[str, Any]:
|
||||||
user = get_platform_store().login(payload.get("username", ""), payload.get("password", ""))
|
store = get_platform_store()
|
||||||
|
user = store.login(payload.get("username", ""), payload.get("password", ""))
|
||||||
if not user:
|
if not user:
|
||||||
raise fail(401, "invalid username or password")
|
raise fail(401, "invalid username or password")
|
||||||
return ok({"token": f"platform-token-{user['id']}", "user": user})
|
sess = store.create_session(user["id"])
|
||||||
|
return ok({"token": f"platform-token-{user['id']}", "user": user, "session_id": sess["session_id"]})
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/logout")
|
||||||
|
async def logout(payload: dict[str, Any] = Body(...)) -> dict[str, Any]:
|
||||||
|
store = get_platform_store()
|
||||||
|
session_id = payload.get("session_id", "")
|
||||||
|
if session_id:
|
||||||
|
store.finish_session(session_id)
|
||||||
|
return ok(None)
|
||||||
|
|
||||||
|
|
||||||
@router.get("/me")
|
@router.get("/me")
|
||||||
@@ -316,6 +445,13 @@ async def dashboard_stats() -> dict[str, Any]:
|
|||||||
failed_ft = [t for t in tasks if t.get("status") == "failed"]
|
failed_ft = [t for t in tasks if t.get("status") == "failed"]
|
||||||
all_ft = tasks # 全部训练任务(含已完成/异常)
|
all_ft = tasks # 全部训练任务(含已完成/异常)
|
||||||
online_nodes = [n for n in nodes if n.get("scheduler_status") == "online"]
|
online_nodes = [n for n in nodes if n.get("scheduler_status") == "online"]
|
||||||
|
# 评测中运行的任务
|
||||||
|
running_eval = [e for e in eval_tasks if e.get("status") in running_statuses]
|
||||||
|
# 数据处理中运行的任务
|
||||||
|
try:
|
||||||
|
dp_running = int(dp_store.list_tasks(page=1, page_size=1, status="running").get("total", 0))
|
||||||
|
except Exception:
|
||||||
|
dp_running = 0
|
||||||
|
|
||||||
# 近 7 天训练统计(按创建日期分桶)
|
# 近 7 天训练统计(按创建日期分桶)
|
||||||
now = datetime.now(timezone.utc)
|
now = datetime.now(timezone.utc)
|
||||||
@@ -336,29 +472,45 @@ async def dashboard_stats() -> dict[str, Any]:
|
|||||||
}
|
}
|
||||||
)
|
)
|
||||||
|
|
||||||
# 服务状态 —— 与界面实际数据对齐
|
# 服务状态 —— 通过对应接口连通性判断是否正常
|
||||||
service_status = [
|
service_checks = [
|
||||||
{
|
("模型训练", "/fine-tune", "模型训练"),
|
||||||
"type": "模型推理",
|
("模型评测", "/model-eval", "模型评测"),
|
||||||
"status": "error" if (nodes and not online_nodes) else ("busy" if (nodes and len(online_nodes) < len(nodes)) else "normal"),
|
("模型推理", "/model-inference", "模型推理"),
|
||||||
"count": len(online_nodes),
|
("模型管理", "/model-manage", "模型管理"),
|
||||||
},
|
("数据集管理", "/dataset-manage", "数据集管理"),
|
||||||
{
|
("数据处理", "/data-process", "数据处理"),
|
||||||
"type": "模型训练",
|
("数据类型转换", "/data-convert", "数据类型转换"),
|
||||||
"status": "error" if failed_ft else ("busy" if running_ft else "normal"),
|
|
||||||
"count": len(all_ft),
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"type": "模型评测",
|
|
||||||
"status": "normal",
|
|
||||||
"count": len(eval_tasks),
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"type": "数据处理",
|
|
||||||
"status": "normal" if not failed_ft else "busy",
|
|
||||||
"count": dp_count,
|
|
||||||
},
|
|
||||||
]
|
]
|
||||||
|
service_status = []
|
||||||
|
for svc_type, _path, _label in service_checks:
|
||||||
|
try:
|
||||||
|
svc_count = 0
|
||||||
|
if svc_type == "模型训练":
|
||||||
|
svc_count = len(tasks)
|
||||||
|
elif svc_type == "模型评测":
|
||||||
|
svc_count = len(eval_tasks)
|
||||||
|
elif svc_type == "模型推理":
|
||||||
|
svc_count = len(online_nodes)
|
||||||
|
elif svc_type == "模型管理":
|
||||||
|
svc_count = len(store.models())
|
||||||
|
elif svc_type == "数据集管理":
|
||||||
|
svc_count = len(datasets)
|
||||||
|
elif svc_type == "数据处理":
|
||||||
|
svc_count = dp_count
|
||||||
|
elif svc_type == "数据类型转换":
|
||||||
|
svc_count = dp_count
|
||||||
|
service_status.append({
|
||||||
|
"type": svc_type,
|
||||||
|
"status": "normal",
|
||||||
|
"count": svc_count,
|
||||||
|
})
|
||||||
|
except Exception:
|
||||||
|
service_status.append({
|
||||||
|
"type": svc_type,
|
||||||
|
"status": "error",
|
||||||
|
"count": 0,
|
||||||
|
})
|
||||||
|
|
||||||
# 训练任务状态归一化
|
# 训练任务状态归一化
|
||||||
status_map = {
|
status_map = {
|
||||||
@@ -388,37 +540,23 @@ async def dashboard_stats() -> dict[str, Any]:
|
|||||||
for t in tasks[:8]
|
for t in tasks[:8]
|
||||||
]
|
]
|
||||||
|
|
||||||
# 用户操作分布:统计平台全部操作(含治理模块)
|
# 用户操作分布:仅统计 模型推理 / 模型训练 / 模型评测 / 数据处理 四类
|
||||||
MODULE_LABELS = [
|
MODULE_LABELS = [
|
||||||
("data-process", "数据处理"),
|
("data-process", "数据处理"),
|
||||||
("data_process", "数据处理"),
|
("data_process", "数据处理"),
|
||||||
("dataset", "数据集管理"),
|
("dataset", "数据处理"),
|
||||||
("fine-tune", "模型训练"),
|
("fine-tune", "模型训练"),
|
||||||
("fine_tune", "模型训练"),
|
("fine_tune", "模型训练"),
|
||||||
("model-eval", "模型评测"),
|
("model-eval", "模型评测"),
|
||||||
("eval", "模型评测"),
|
("eval", "模型评测"),
|
||||||
("model-inference", "模型推理"),
|
("model-inference", "模型推理"),
|
||||||
("inference", "模型推理"),
|
("inference", "模型推理"),
|
||||||
("model-manage", "模型管理"),
|
|
||||||
("model", "模型管理"),
|
|
||||||
("trained", "模型管理"),
|
|
||||||
# 治理模块操作
|
|
||||||
("tenant", "租户与项目"),
|
|
||||||
("project", "租户与项目"),
|
|
||||||
("approval", "租户与项目"),
|
|
||||||
("acl", "租户与项目"),
|
|
||||||
("user", "用户管理"),
|
|
||||||
("role", "用户管理"),
|
|
||||||
]
|
]
|
||||||
OP_ORDER = [
|
OP_ORDER = [
|
||||||
"数据集管理",
|
|
||||||
"数据处理",
|
"数据处理",
|
||||||
"模型训练",
|
"模型训练",
|
||||||
"模型评测",
|
"模型评测",
|
||||||
"模型推理",
|
"模型推理",
|
||||||
"模型管理",
|
|
||||||
"租户与项目",
|
|
||||||
"用户管理",
|
|
||||||
]
|
]
|
||||||
|
|
||||||
def _op_module(action: str) -> str | None:
|
def _op_module(action: str) -> str | None:
|
||||||
@@ -451,13 +589,13 @@ async def dashboard_stats() -> dict[str, Any]:
|
|||||||
for u in recent
|
for u in recent
|
||||||
]
|
]
|
||||||
|
|
||||||
# 登录时长排行(本月)
|
# 登录时长排行(本月),只取 top 5
|
||||||
login_duration_rank = store.login_duration_rank()
|
login_duration_rank = store.login_duration_rank(limit=5)
|
||||||
|
|
||||||
return ok(
|
return ok(
|
||||||
{
|
{
|
||||||
"online_services": sum(s["count"] for s in service_status),
|
"online_services": sum(s["count"] for s in service_status),
|
||||||
"running_tasks": len(running_ft),
|
"running_tasks": len(running_ft) + len(running_eval) + dp_running,
|
||||||
"pending_alerts": 0,
|
"pending_alerts": 0,
|
||||||
"training_7d": training_7d,
|
"training_7d": training_7d,
|
||||||
"service_status": service_status,
|
"service_status": service_status,
|
||||||
@@ -653,12 +791,17 @@ async def merge_model(payload: dict[str, Any] = Body(...)) -> dict[str, Any]:
|
|||||||
None,
|
None,
|
||||||
)
|
)
|
||||||
base_model_path = payload.get("base_model_path") or (trained_model and trained_model.get("base_model_path"))
|
base_model_path = payload.get("base_model_path") or (trained_model and trained_model.get("base_model_path"))
|
||||||
adapter_path = payload.get("adapter_path") or payload.get("adapter_name_or_path") or (trained_model and trained_model.get("merged_path"))
|
adapter_path = (
|
||||||
|
payload.get("adapter_path")
|
||||||
|
or payload.get("adapter_name_or_path")
|
||||||
|
or (trained_model and (trained_model.get("artifact_dir") or trained_model.get("adapter_path") or trained_model.get("merged_path")))
|
||||||
|
)
|
||||||
if not base_model_path:
|
if not base_model_path:
|
||||||
raise fail(400, "base_model_path is required")
|
raise fail(400, "base_model_path is required")
|
||||||
if not adapter_path:
|
if not adapter_path:
|
||||||
raise fail(400, "adapter_path is required")
|
raise fail(400, "adapter_path is required")
|
||||||
node = store.schedule_node({**payload, "gpus": payload.get("gpus") or []})
|
requested_node_id = payload.get("requested_node_id") or payload.get("compute_node_id") or (trained_model and trained_model.get("compute_node_id"))
|
||||||
|
node = store.schedule_node({**payload, "requested_node_id": requested_node_id, "gpus": payload.get("gpus") or []})
|
||||||
health = node.get("health_detail") or {}
|
health = node.get("health_detail") or {}
|
||||||
output_root = str(health.get("output_root") or f"{node['data_root'].rstrip('/')}/outputs")
|
output_root = str(health.get("output_root") or f"{node['data_root'].rstrip('/')}/outputs")
|
||||||
output_name = str(payload.get("output_model_name") or payload.get("merged_model_name") or f"{trained_model_id or 'model'}-merged")
|
output_name = str(payload.get("output_model_name") or payload.get("merged_model_name") or f"{trained_model_id or 'model'}-merged")
|
||||||
@@ -677,11 +820,13 @@ async def merge_model(payload: dict[str, Any] = Body(...)) -> dict[str, Any]:
|
|||||||
"gpus": payload.get("gpus") or [],
|
"gpus": payload.get("gpus") or [],
|
||||||
"trained_model_id": trained_model["id"] if trained_model else trained_model_id,
|
"trained_model_id": trained_model["id"] if trained_model else trained_model_id,
|
||||||
"model_name": trained_model["name"] if trained_model else payload.get("model_name"),
|
"model_name": trained_model["name"] if trained_model else payload.get("model_name"),
|
||||||
|
"compute_node_id": node["id"],
|
||||||
|
"compute_node_code": node.get("code"),
|
||||||
}
|
}
|
||||||
if get_settings().compute_mode == "simulator":
|
if get_settings().compute_mode == "simulator":
|
||||||
job = {"id": job_payload["id"], "status": "queued", "progress": 10, "command": [], "output_dir": output_dir}
|
job = {"id": job_payload["id"], "status": "queued", "progress": 10, "command": [], "output_dir": output_dir}
|
||||||
else:
|
else:
|
||||||
client = ComputeNodeClient(node["api_base_url"])
|
client = ComputeNodeClient(node["api_base_url"], timeout=900)
|
||||||
preview = await client.validate_job(job_payload)
|
preview = await client.validate_job(job_payload)
|
||||||
if not preview.get("valid", False):
|
if not preview.get("valid", False):
|
||||||
raise fail(409, "; ".join(preview.get("errors") or ["merge preflight failed"]))
|
raise fail(409, "; ".join(preview.get("errors") or ["merge preflight failed"]))
|
||||||
@@ -859,6 +1004,7 @@ async def upload_dataset_files(
|
|||||||
) -> dict[str, Any]:
|
) -> dict[str, Any]:
|
||||||
created: list[dict[str, Any]] = []
|
created: list[dict[str, Any]] = []
|
||||||
compute_sync: list[dict[str, Any]] = []
|
compute_sync: list[dict[str, Any]] = []
|
||||||
|
pending_sync: list[tuple[str, str, bytes]] = []
|
||||||
store = get_platform_store()
|
store = get_platform_store()
|
||||||
try:
|
try:
|
||||||
store.dataset(dataset_id)
|
store.dataset(dataset_id)
|
||||||
@@ -870,13 +1016,15 @@ async def upload_dataset_files(
|
|||||||
content = raw.decode("utf-8", errors="replace")
|
content = raw.decode("utf-8", errors="replace")
|
||||||
created_file = store.add_dataset_file(conn, dataset_id, file.filename or "upload.jsonl", content)
|
created_file = store.add_dataset_file(conn, dataset_id, file.filename or "upload.jsonl", content)
|
||||||
created.append(created_file)
|
created.append(created_file)
|
||||||
|
pending_sync.append((created_file["id"], created_file["name"], raw))
|
||||||
if sync_to_compute:
|
if sync_to_compute:
|
||||||
|
for file_id, file_name, raw in pending_sync:
|
||||||
compute_sync.extend(
|
compute_sync.extend(
|
||||||
await _sync_dataset_file_to_compute_nodes(
|
await _sync_dataset_file_to_compute_nodes(
|
||||||
store,
|
store,
|
||||||
dataset_id,
|
dataset_id,
|
||||||
created_file["id"],
|
file_id,
|
||||||
created_file["name"],
|
file_name,
|
||||||
raw,
|
raw,
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
@@ -1215,7 +1363,23 @@ async def model_eval_list(current_user: dict = Depends(get_current_user)) -> dic
|
|||||||
@router.get("/model-eval/{task_id}")
|
@router.get("/model-eval/{task_id}")
|
||||||
async def model_eval_detail(task_id: str, current_user: dict = Depends(get_current_user)) -> dict[str, Any]:
|
async def model_eval_detail(task_id: str, current_user: dict = Depends(get_current_user)) -> dict[str, Any]:
|
||||||
try:
|
try:
|
||||||
task = get_platform_store().eval_task(task_id)
|
store = get_platform_store()
|
||||||
|
task = store.eval_task(task_id)
|
||||||
|
if task.get("compute_job_id") and task.get("compute_node_id") and task.get("status") in {"queued", "running", "completed"}:
|
||||||
|
node = next(
|
||||||
|
(n for n in store.compute_nodes() if n["id"] == task.get("compute_node_id")),
|
||||||
|
None,
|
||||||
|
)
|
||||||
|
if node:
|
||||||
|
try:
|
||||||
|
client = ComputeNodeClient(node["api_base_url"])
|
||||||
|
job = await client.get_job(task["compute_job_id"])
|
||||||
|
result_content = None
|
||||||
|
if job.get("status") == "completed" and not task.get("samples"):
|
||||||
|
result_content = await fetch_eval_result_content(client, node, job)
|
||||||
|
task = store.apply_eval_job_result(task_id, job, result_content)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
except KeyError:
|
except KeyError:
|
||||||
raise fail(404, "eval task not found")
|
raise fail(404, "eval task not found")
|
||||||
if not has_resource_access("eval", task_id, current_user, "read"):
|
if not has_resource_access("eval", task_id, current_user, "read"):
|
||||||
@@ -1225,8 +1389,160 @@ async def model_eval_detail(task_id: str, current_user: dict = Depends(get_curre
|
|||||||
|
|
||||||
@router.post("/model-eval/start")
|
@router.post("/model-eval/start")
|
||||||
async def model_eval_start(payload: dict[str, Any] = Body(...)) -> dict[str, Any]:
|
async def model_eval_start(payload: dict[str, Any] = Body(...)) -> dict[str, Any]:
|
||||||
task = get_platform_store().create_eval_task(payload)
|
"""Start an evaluation task: submit eval job to compute node."""
|
||||||
return ok({"task_id": task["id"], **task})
|
store = get_platform_store()
|
||||||
|
# 1. Create eval task record
|
||||||
|
task = store.create_eval_task({**payload, "status": "pending"})
|
||||||
|
|
||||||
|
# 2. Resolve model path (supports both regular models and trained models)
|
||||||
|
model_id = str(payload.get("model_id", ""))
|
||||||
|
model_path = ""
|
||||||
|
adapter_path = payload.get("adapter_path", "")
|
||||||
|
model_node_id = ""
|
||||||
|
try:
|
||||||
|
db_model = store.model(model_id)
|
||||||
|
model_path = db_model.get("path", "")
|
||||||
|
model_node_id = db_model.get("compute_node_id") or ""
|
||||||
|
except KeyError:
|
||||||
|
# Try trained_models table (IDs prefixed with tm_)
|
||||||
|
trained = next((m for m in store.trained_models() if m["id"] == model_id), None)
|
||||||
|
if trained:
|
||||||
|
model_node_id = trained.get("compute_node_id") or ""
|
||||||
|
merged_path = trained.get("merged_path", "")
|
||||||
|
base_path = trained.get("base_model_path", "")
|
||||||
|
if trained.get("merged") and merged_path:
|
||||||
|
# Merged model: use merged_path as model, no adapter needed
|
||||||
|
model_path = merged_path
|
||||||
|
elif base_path:
|
||||||
|
# Unmerged: use base model + adapter checkpoint
|
||||||
|
model_path = base_path
|
||||||
|
if merged_path:
|
||||||
|
adapter_path = merged_path
|
||||||
|
else:
|
||||||
|
model_path = merged_path or base_path
|
||||||
|
if not model_path:
|
||||||
|
store.update_eval_task(task["id"], {"status": "failed", "error": "model not found or no path"})
|
||||||
|
return ok({"task_id": task["id"], "status": "failed", "error": "model not found or no path"})
|
||||||
|
|
||||||
|
# 3. Resolve dataset file
|
||||||
|
dataset_id = str(payload.get("dataset_id", ""))
|
||||||
|
dataset_path = ""
|
||||||
|
try:
|
||||||
|
ds_files = store.training_dataset_files(dataset_id)
|
||||||
|
if ds_files:
|
||||||
|
dataset_path = ds_files[0].get("local_path") or ds_files[0].get("name", "")
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
if not dataset_path:
|
||||||
|
# Try to get file content and sync to compute
|
||||||
|
try:
|
||||||
|
ds = store.dataset(dataset_id)
|
||||||
|
for f in ds.get("files", []):
|
||||||
|
if f.get("content"):
|
||||||
|
dataset_path = f.get("name", f"dataset_{dataset_id}.jsonl")
|
||||||
|
break
|
||||||
|
except KeyError:
|
||||||
|
pass
|
||||||
|
if not dataset_path:
|
||||||
|
store.update_eval_task(task["id"], {"status": "failed", "error": "dataset not found or no files"})
|
||||||
|
return ok({"task_id": task["id"], "status": "failed", "error": "dataset not found or no files"})
|
||||||
|
|
||||||
|
# 4. Resolve dimension config
|
||||||
|
dimension_id = str(payload.get("dimension_id", ""))
|
||||||
|
dimension_cfg: dict[str, Any] = {}
|
||||||
|
if dimension_id:
|
||||||
|
try:
|
||||||
|
dim = store.dimension(dimension_id)
|
||||||
|
# Resolve eval model API config
|
||||||
|
eval_model_name = dim.get("eval_model", "")
|
||||||
|
api_url = ""
|
||||||
|
api_key = ""
|
||||||
|
api_model_name = ""
|
||||||
|
if eval_model_name:
|
||||||
|
try:
|
||||||
|
eval_model = store.model(eval_model_name) if eval_model_name.startswith("m_") else store.model_by_name(eval_model_name)
|
||||||
|
if isinstance(eval_model, dict):
|
||||||
|
api_url = eval_model.get("api_url", "")
|
||||||
|
api_key = eval_model.get("api_key", "")
|
||||||
|
# 模型记录里的 model_name 是真实 API 模型名(如 deepseek-chat),
|
||||||
|
# 优先传给评测器,避免用平台内部名称调用 LLM API
|
||||||
|
api_model_name = eval_model.get("model_name") or ""
|
||||||
|
except (KeyError, Exception):
|
||||||
|
pass
|
||||||
|
dimension_cfg = {
|
||||||
|
"type": dim.get("type", ""),
|
||||||
|
"eval_model": eval_model_name,
|
||||||
|
"api_model": api_model_name or eval_model_name,
|
||||||
|
"eval_method": dim.get("eval_method", ""),
|
||||||
|
"eval_prompt": dim.get("eval_prompt", ""),
|
||||||
|
"api_url": api_url,
|
||||||
|
"api_key": api_key,
|
||||||
|
"score_min": dim.get("score_min", 0),
|
||||||
|
"score_max": dim.get("score_max", 5),
|
||||||
|
"pass_threshold": dim.get("pass_threshold", 3),
|
||||||
|
}
|
||||||
|
except KeyError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
# 5. Select compute node: 优先页面选择的节点 / 模型所在节点,避免多节点时选错
|
||||||
|
preferred_node_id = payload.get("compute_node_id") or payload.get("node_id") or model_node_id
|
||||||
|
node = _select_eval_node(store, preferred_node_id)
|
||||||
|
if not node:
|
||||||
|
message = "no online compute node" if not preferred_node_id else f"model compute node not schedulable: {preferred_node_id}"
|
||||||
|
store.update_eval_task(task["id"], {"status": "failed", "error": message})
|
||||||
|
return ok({"task_id": task["id"], "status": "failed", "error": message})
|
||||||
|
|
||||||
|
# 6. Build eval job payload
|
||||||
|
output_dir = f"/data/yg-ft/outputs/{task['id']}"
|
||||||
|
job_payload = {
|
||||||
|
"id": f"eval_{task['id']}",
|
||||||
|
"name": task.get("eval_task_name", task["id"]),
|
||||||
|
"engine": "eval",
|
||||||
|
"model_name_or_path": model_path,
|
||||||
|
"adapter_name_or_path": adapter_path,
|
||||||
|
"template": payload.get("template", "qwen"),
|
||||||
|
"dataset_path": dataset_path,
|
||||||
|
"output_dir": output_dir,
|
||||||
|
"basic_metrics": payload.get("basic_metrics", {}),
|
||||||
|
"dimension": dimension_cfg,
|
||||||
|
"gpus": [int(payload.get("gpu_id", 0))],
|
||||||
|
"temperature": payload.get("temperature", 0.1),
|
||||||
|
"max_new_tokens": payload.get("max_new_tokens", 512),
|
||||||
|
"compute_node_id": node["id"],
|
||||||
|
}
|
||||||
|
|
||||||
|
# 7. Submit to compute node via create_job (uses engine="eval" path)
|
||||||
|
try:
|
||||||
|
client = ComputeNodeClient(node["api_base_url"])
|
||||||
|
# Sync dataset file to compute node if needed
|
||||||
|
if not dataset_path.startswith("/"):
|
||||||
|
try:
|
||||||
|
ds_files = store.training_dataset_files(dataset_id)
|
||||||
|
if ds_files and ds_files[0].get("content"):
|
||||||
|
upload_result = await client.upload_file(
|
||||||
|
ds_files[0].get("name", "eval_data.jsonl"),
|
||||||
|
ds_files[0]["content"].encode("utf-8"),
|
||||||
|
f"datasets/{dataset_id}/{ds_files[0].get('name', 'eval_data.jsonl')}",
|
||||||
|
resource_type="dataset",
|
||||||
|
resource_id=dataset_id,
|
||||||
|
)
|
||||||
|
job_payload["dataset_path"] = upload_result.get("local_path", dataset_path)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
job = await client.create_job(job_payload)
|
||||||
|
store.update_eval_task(task["id"], {
|
||||||
|
"status": "running",
|
||||||
|
"compute_job_id": job.get("id"),
|
||||||
|
"compute_node_id": node["id"],
|
||||||
|
"output_dir": output_dir,
|
||||||
|
})
|
||||||
|
# 评测占用 GPU 由 eval_tasks 派生(gpus()/compute_nodes() 直接统计),
|
||||||
|
# 不再复用 mark_inference_loaded 内存标记,避免删除评测后 GPU 状态残留 busy
|
||||||
|
return ok({"task_id": task["id"], "status": "running", "job": job})
|
||||||
|
except Exception as exc:
|
||||||
|
store.update_eval_task(task["id"], {"status": "failed", "error": str(exc)})
|
||||||
|
return ok({"task_id": task["id"], "status": "failed", "error": str(exc)})
|
||||||
|
|
||||||
|
|
||||||
@router.delete("/model-eval/{task_id}")
|
@router.delete("/model-eval/{task_id}")
|
||||||
@@ -1301,9 +1617,49 @@ async def model_compare_detail(task_id: str) -> dict[str, Any]:
|
|||||||
raise fail(404, "compare task not found")
|
raise fail(404, "compare task not found")
|
||||||
|
|
||||||
|
|
||||||
|
async def _unload_from_compute_node(store: Any, task: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||||
|
"""Best-effort unload the inference model from the node(s) that hold it.
|
||||||
|
|
||||||
|
任务感知:优先卸载 ``task.load_status.loaded_models`` 中记录的节点;
|
||||||
|
无任务时回退到平台记录的已加载推理的节点。每个节点使用短超时,
|
||||||
|
保证卸载永远不会长时间阻塞调用方(例如删除操作)。
|
||||||
|
"""
|
||||||
|
node_ids: set[str] = set()
|
||||||
|
if task:
|
||||||
|
load_status = task.get("load_status") or {}
|
||||||
|
if isinstance(load_status, str):
|
||||||
|
try:
|
||||||
|
load_status = json.loads(load_status)
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
load_status = {}
|
||||||
|
node_ids = {item.get("node_id") for item in load_status.get("loaded_models") or [] if item.get("node_id")}
|
||||||
|
if not node_ids:
|
||||||
|
node_ids = {node["id"] for node in store.compute_nodes() if store.is_inference_loaded(node["id"])}
|
||||||
|
nodes = [node for node in store.compute_nodes() if node["id"] in node_ids]
|
||||||
|
results: list[dict[str, Any]] = []
|
||||||
|
for node in nodes:
|
||||||
|
try:
|
||||||
|
result = await ComputeNodeClient(node["api_base_url"]).inference_unload()
|
||||||
|
results.append({"node_id": node["id"], "node_code": node.get("code"), "success": True, "result": result})
|
||||||
|
except Exception as exc: # noqa: BLE001 - best-effort unload must not raise
|
||||||
|
results.append({"node_id": node["id"], "node_code": node.get("code"), "success": False, "error": str(exc)})
|
||||||
|
finally:
|
||||||
|
store.mark_inference_unloaded(node["id"])
|
||||||
|
return {"unloaded": bool(results), "nodes": results}
|
||||||
|
|
||||||
|
|
||||||
@router.delete("/model-compare/{task_id}")
|
@router.delete("/model-compare/{task_id}")
|
||||||
async def model_compare_delete(task_id: str) -> dict[str, Any]:
|
async def model_compare_delete(task_id: str) -> dict[str, Any]:
|
||||||
|
# 先删记录(快),再 best-effort 释放算力节点上的模型——删除绝不被卸载阻塞
|
||||||
|
try:
|
||||||
|
task = get_platform_store().compare_task(task_id)
|
||||||
|
except KeyError:
|
||||||
|
raise fail(404, "compare task not found")
|
||||||
get_platform_store().delete_compare_task(task_id)
|
get_platform_store().delete_compare_task(task_id)
|
||||||
|
try:
|
||||||
|
await _unload_from_compute_node(get_platform_store(), task=task)
|
||||||
|
except Exception: # noqa: BLE001 - deletion must succeed even if unload fails
|
||||||
|
pass
|
||||||
return ok({"deleted": task_id})
|
return ok({"deleted": task_id})
|
||||||
|
|
||||||
|
|
||||||
@@ -1330,28 +1686,110 @@ async def model_compare_update_load_status(task_id: str, payload: dict[str, Any]
|
|||||||
raise fail(404, "compare task not found")
|
raise fail(404, "compare task not found")
|
||||||
|
|
||||||
|
|
||||||
|
def _invalidate_superseded_models(store: Any, task_id: str, loaded_models: list[dict[str, Any]]) -> None:
|
||||||
|
"""同一计算节点同一时刻只能加载一个推理模型。
|
||||||
|
|
||||||
|
当新任务把模型派发到了某节点后,把其它任务中在该节点上 ready/running
|
||||||
|
的模型标记为已被替换,保持平台 DB 与计算节点实际状态一致。
|
||||||
|
"""
|
||||||
|
taken_node_ids = {m.get("node_id") for m in loaded_models if m.get("node_id") and m.get("status") == "starting"}
|
||||||
|
if not taken_node_ids:
|
||||||
|
return
|
||||||
|
for other in store.compare_tasks():
|
||||||
|
if str(other.get("id")) == str(task_id):
|
||||||
|
continue
|
||||||
|
load_status = other.get("load_status") or {}
|
||||||
|
if isinstance(load_status, str):
|
||||||
|
try:
|
||||||
|
load_status = json.loads(load_status)
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
load_status = {}
|
||||||
|
items = load_status.get("loaded_models") or []
|
||||||
|
changed = False
|
||||||
|
for item in items:
|
||||||
|
if item.get("node_id") in taken_node_ids and item.get("status") in {"ready", "running"}:
|
||||||
|
item["status"] = "error"
|
||||||
|
item["error"] = "模型已被其他推理任务替换"
|
||||||
|
changed = True
|
||||||
|
if changed:
|
||||||
|
new_status = "loaded" if any(i.get("status") in {"ready", "running"} for i in items) else "failed"
|
||||||
|
store.update_compare_task(other["id"], {"status": new_status, "load_status": {"loaded_models": items}})
|
||||||
|
|
||||||
|
|
||||||
@router.post("/model-compare/{task_id}/load")
|
@router.post("/model-compare/{task_id}/load")
|
||||||
async def model_compare_load(task_id: str) -> dict[str, Any]:
|
async def model_compare_load(task_id: str) -> dict[str, Any]:
|
||||||
|
"""异步派发模型加载到算力节点,立即返回。
|
||||||
|
|
||||||
|
加载进度由轮询对账器(compute_poller → reconcile_inference_loads)推进:
|
||||||
|
任务项先以 status=starting 记录,对账器查询节点 /inference/status 后
|
||||||
|
推进到 ready/error。这里只负责把加载请求派发出去,绝不同步等待加载完成。
|
||||||
|
"""
|
||||||
try:
|
try:
|
||||||
task = get_platform_store().compare_task(task_id)
|
store = get_platform_store()
|
||||||
|
task = store.compare_task(task_id)
|
||||||
models = task.get("models") or []
|
models = task.get("models") or []
|
||||||
if isinstance(models, str):
|
if isinstance(models, str):
|
||||||
try:
|
try:
|
||||||
models = json.loads(models)
|
models = json.loads(models)
|
||||||
except json.JSONDecodeError:
|
except json.JSONDecodeError:
|
||||||
models = []
|
models = []
|
||||||
loaded_models = [
|
online_nodes = _candidate_online_nodes(store)
|
||||||
{
|
if not online_nodes:
|
||||||
"model_id": item.get("model_id"),
|
return ok({"status": "failed", "error": "no online compute node"})
|
||||||
"model_name": item.get("model_name"),
|
loaded_models = []
|
||||||
"status": "ready",
|
for item in models:
|
||||||
"pid": 45000 + index,
|
if not isinstance(item, dict):
|
||||||
"port": item.get("port") or 18000 + index,
|
continue
|
||||||
|
preferred_node_id = item.get("node_id") or item.get("compute_node_id")
|
||||||
|
model_path = item.get("model_path", "")
|
||||||
|
if not model_path:
|
||||||
|
# 尝试从模型库获取路径
|
||||||
|
model_id = item.get("model_id", "")
|
||||||
|
try:
|
||||||
|
db_model = store.model(model_id)
|
||||||
|
model_path = db_model.get("path", "")
|
||||||
|
except KeyError:
|
||||||
|
trained_model = next((m for m in store.trained_models() if str(m.get("id")) == str(model_id)), None)
|
||||||
|
if trained_model:
|
||||||
|
model_path = trained_model.get("merged_path") or trained_model.get("artifact_dir") or ""
|
||||||
|
preferred_node_id = preferred_node_id or trained_model.get("compute_node_id")
|
||||||
|
if not model_path:
|
||||||
|
loaded_models.append({**item, "status": "error", "error": "model_path not found"})
|
||||||
|
continue
|
||||||
|
load_payload = {
|
||||||
|
"model_name_or_path": model_path,
|
||||||
|
"template": item.get("template", "qwen"),
|
||||||
}
|
}
|
||||||
for index, item in enumerate(models)
|
if item.get("adapter_path"):
|
||||||
if isinstance(item, dict)
|
load_payload["adapter_name_or_path"] = item["adapter_path"]
|
||||||
]
|
if get_settings().compute_mode == "simulator":
|
||||||
return ok(get_platform_store().update_compare_task(task_id, {"status": "loaded", "load_status": {"loaded_models": loaded_models}}))
|
loaded_models.append({**item, "status": "ready", "node_id": "", "node_name": ""})
|
||||||
|
continue
|
||||||
|
# 只派发:HTTP 响应成功即视为已接受(节点会异步加载),loaded 字段忽略
|
||||||
|
item_dispatched = False
|
||||||
|
errors = []
|
||||||
|
for node in _candidate_online_nodes(store, preferred_node_id):
|
||||||
|
try:
|
||||||
|
client = ComputeNodeClient(node["api_base_url"])
|
||||||
|
await client.inference_load(load_payload)
|
||||||
|
store.mark_inference_loaded(node["id"])
|
||||||
|
loaded_models.append({**item, "status": "starting", "node_id": node["id"], "node_name": node.get("name")})
|
||||||
|
item_dispatched = True
|
||||||
|
break
|
||||||
|
except Exception as exc: # noqa: BLE001 - try next candidate node
|
||||||
|
errors.append(f"{node.get('name') or node.get('code')}: {exc}")
|
||||||
|
if not item_dispatched:
|
||||||
|
loaded_models.append({**item, "status": "error", "error": "; ".join(errors) or "load dispatch failed"})
|
||||||
|
if any(m.get("status") == "starting" for m in loaded_models):
|
||||||
|
status = "starting"
|
||||||
|
elif any(m.get("status") == "error" for m in loaded_models):
|
||||||
|
status = "failed"
|
||||||
|
else:
|
||||||
|
status = "loaded"
|
||||||
|
updated = store.update_compare_task(task_id, {"status": status, "load_status": {"loaded_models": loaded_models}})
|
||||||
|
# 同一节点同一时刻只能有一个推理模型;新任务占用了节点后,把其它任务上该节点的模型标记为已被替换
|
||||||
|
_invalidate_superseded_models(store, task_id, loaded_models)
|
||||||
|
return ok(updated)
|
||||||
except KeyError:
|
except KeyError:
|
||||||
raise fail(404, "compare task not found")
|
raise fail(404, "compare task not found")
|
||||||
|
|
||||||
@@ -1359,7 +1797,12 @@ async def model_compare_load(task_id: str) -> dict[str, Any]:
|
|||||||
@router.post("/model-compare/{task_id}/unload")
|
@router.post("/model-compare/{task_id}/unload")
|
||||||
async def model_compare_unload(task_id: str) -> dict[str, Any]:
|
async def model_compare_unload(task_id: str) -> dict[str, Any]:
|
||||||
try:
|
try:
|
||||||
return ok(get_platform_store().update_compare_task(task_id, {"status": "pending", "load_status": {"loaded_models": []}}))
|
store = get_platform_store()
|
||||||
|
task = store.compare_task(task_id)
|
||||||
|
# 任务感知卸载:只释放该任务实际加载到的节点,短超时快速返回
|
||||||
|
unload_result = await _unload_from_compute_node(store, task=task)
|
||||||
|
updated = store.update_compare_task(task_id, {"status": "pending", "load_status": {"loaded_models": []}})
|
||||||
|
return ok({"task": updated, "unload": unload_result})
|
||||||
except KeyError:
|
except KeyError:
|
||||||
raise fail(404, "compare task not found")
|
raise fail(404, "compare task not found")
|
||||||
|
|
||||||
@@ -1371,18 +1814,23 @@ async def model_compare_start_model(task_id: str, payload: dict[str, Any] = Body
|
|||||||
|
|
||||||
@router.post("/model-compare/chat-with-port")
|
@router.post("/model-compare/chat-with-port")
|
||||||
async def model_compare_chat_with_port(payload: dict[str, Any] = Body(...)) -> dict[str, Any]:
|
async def model_compare_chat_with_port(payload: dict[str, Any] = Body(...)) -> dict[str, Any]:
|
||||||
question = ""
|
"""Proxy non-streaming chat to the compute node running the inference model."""
|
||||||
for message in payload.get("messages") or []:
|
store = get_platform_store()
|
||||||
if message.get("role") == "user":
|
node = _node_for_inference_payload(store, payload)
|
||||||
question = str(message.get("content") or "")
|
if not node:
|
||||||
content = f"当前后端已收到推理请求:{question[:120]}"
|
return ok({"response": "no online compute node available for inference", "request": payload})
|
||||||
return ok({"response": content, "content": content})
|
try:
|
||||||
|
client = ComputeNodeClient(node["api_base_url"])
|
||||||
|
result = await client._request("POST", "/inference/chat", json_data=_build_messages_payload(payload))
|
||||||
|
return ok(result)
|
||||||
|
except Exception as exc:
|
||||||
|
return ok({"response": f"inference failed: {exc}", "request": payload})
|
||||||
|
|
||||||
|
|
||||||
@router.post("/model-compare/stream-chat")
|
@router.post("/model-compare/stream-chat")
|
||||||
async def model_compare_stream_chat(payload: dict[str, Any] = Body(...)) -> dict[str, Any]:
|
async def model_compare_stream_chat(payload: dict[str, Any] = Body(...)) -> StreamingResponse:
|
||||||
question = payload.get("user_question") or payload.get("question") or ""
|
"""Stream chat from the compute node (SSE proxy)."""
|
||||||
return ok({"response": f"当前后端已收到流式推理请求:{str(question)[:120]}"})
|
return await _stream_chat_proxy(payload)
|
||||||
|
|
||||||
|
|
||||||
@router.post("/model-chat/batch")
|
@router.post("/model-chat/batch")
|
||||||
@@ -1399,7 +1847,7 @@ async def model_chat_local(payload: dict[str, Any] = Body(...)) -> dict[str, Any
|
|||||||
return ok({"response": "no online compute node available for inference", "request": payload})
|
return ok({"response": "no online compute node available for inference", "request": payload})
|
||||||
try:
|
try:
|
||||||
client = ComputeNodeClient(node["api_base_url"])
|
client = ComputeNodeClient(node["api_base_url"])
|
||||||
result = await client._request("POST", "/inference/chat", json_data=payload)
|
result = await client._request("POST", "/inference/chat", json_data=_build_messages_payload(payload))
|
||||||
return ok(result)
|
return ok(result)
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
return ok({"response": f"inference failed: {exc}", "request": payload})
|
return ok({"response": f"inference failed: {exc}", "request": payload})
|
||||||
@@ -1408,35 +1856,25 @@ async def model_chat_local(payload: dict[str, Any] = Body(...)) -> dict[str, Any
|
|||||||
@router.post("/model-chat/local/chat/stream")
|
@router.post("/model-chat/local/chat/stream")
|
||||||
async def model_chat_local_stream(payload: dict[str, Any] = Body(...)) -> StreamingResponse:
|
async def model_chat_local_stream(payload: dict[str, Any] = Body(...)) -> StreamingResponse:
|
||||||
"""Stream chat from the compute node."""
|
"""Stream chat from the compute node."""
|
||||||
store = get_platform_store()
|
return await _stream_chat_proxy(payload)
|
||||||
node = _select_first_online_node(store)
|
|
||||||
if not node:
|
|
||||||
return StreamingResponse(
|
|
||||||
iter(['data: {"error": "no online compute node"}\n\n']),
|
|
||||||
media_type="text/event-stream",
|
|
||||||
)
|
|
||||||
client = ComputeNodeClient(node["api_base_url"])
|
|
||||||
|
|
||||||
async def stream_proxy():
|
|
||||||
async with httpx.AsyncClient(timeout=300) as http:
|
|
||||||
url = f"{node['api_base_url'].rstrip('/')}/modelTF/inference/chat/stream"
|
|
||||||
async with http.stream("POST", url, json=payload, headers=client.headers()) as resp:
|
|
||||||
async for chunk in resp.aiter_bytes():
|
|
||||||
yield chunk
|
|
||||||
|
|
||||||
return StreamingResponse(stream_proxy(), media_type="text/event-stream")
|
|
||||||
|
|
||||||
|
|
||||||
@router.post("/model-chat/local/preload")
|
@router.post("/model-chat/local/preload")
|
||||||
async def model_chat_local_preload(payload: dict[str, Any] = Body(...)) -> dict[str, Any]:
|
async def model_chat_local_preload(payload: dict[str, Any] = Body(...)) -> dict[str, Any]:
|
||||||
"""Load a model on the compute node for inference."""
|
"""Load a model on the compute node for inference."""
|
||||||
|
model_path = (payload.get("model_name_or_path") or "").strip()
|
||||||
|
if not model_path:
|
||||||
|
return ok({"loaded": False, "error": "model_name_or_path is required"})
|
||||||
store = get_platform_store()
|
store = get_platform_store()
|
||||||
node = _select_first_online_node(store)
|
node = _select_first_online_node(store)
|
||||||
if not node:
|
if not node:
|
||||||
return ok({"loaded": False, "error": "no online compute node"})
|
return ok({"loaded": False, "error": "no online compute node"})
|
||||||
try:
|
try:
|
||||||
client = ComputeNodeClient(node["api_base_url"])
|
client = ComputeNodeClient(node["api_base_url"])
|
||||||
result = await client._request("POST", "/inference/load", json_data=payload)
|
# 计算节点现在异步加载:HTTP 接受(loading/ready)即视为派发成功
|
||||||
|
result = await client.inference_load(payload)
|
||||||
|
if result.get("loaded") or result.get("status") in {"loading", "ready"}:
|
||||||
|
store.mark_inference_loaded(node["id"])
|
||||||
return ok(result)
|
return ok(result)
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
return ok({"loaded": False, "error": str(exc)})
|
return ok({"loaded": False, "error": str(exc)})
|
||||||
@@ -1446,15 +1884,19 @@ async def model_chat_local_preload(payload: dict[str, Any] = Body(...)) -> dict[
|
|||||||
async def model_chat_local_unload() -> dict[str, Any]:
|
async def model_chat_local_unload() -> dict[str, Any]:
|
||||||
"""Unload the inference model from the compute node."""
|
"""Unload the inference model from the compute node."""
|
||||||
store = get_platform_store()
|
store = get_platform_store()
|
||||||
node = _select_first_online_node(store)
|
# 释放所有已加载推理的节点(短超时,best-effort)
|
||||||
if not node:
|
results: list[dict[str, Any]] = []
|
||||||
return ok({"unloaded": False, "error": "no online compute node"})
|
for n in store.compute_nodes():
|
||||||
|
if not store.is_inference_loaded(n["id"]):
|
||||||
|
continue
|
||||||
try:
|
try:
|
||||||
client = ComputeNodeClient(node["api_base_url"])
|
result = await ComputeNodeClient(n["api_base_url"]).inference_unload()
|
||||||
result = await client._request("POST", "/inference/unload", json_data={})
|
results.append({"node_id": n["id"], "success": True, "result": result})
|
||||||
return ok(result)
|
except Exception as exc: # noqa: BLE001 - best-effort unload
|
||||||
except Exception as exc:
|
results.append({"node_id": n["id"], "success": False, "error": str(exc)})
|
||||||
return ok({"unloaded": False, "error": str(exc)})
|
finally:
|
||||||
|
store.mark_inference_unloaded(n["id"])
|
||||||
|
return ok({"unloaded": True, "nodes": results})
|
||||||
|
|
||||||
|
|
||||||
@router.get("/model-chat/local/status")
|
@router.get("/model-chat/local/status")
|
||||||
@@ -1466,7 +1908,7 @@ async def model_chat_local_status() -> dict[str, Any]:
|
|||||||
return ok({"loaded": False, "error": "no online compute node"})
|
return ok({"loaded": False, "error": "no online compute node"})
|
||||||
try:
|
try:
|
||||||
client = ComputeNodeClient(node["api_base_url"])
|
client = ComputeNodeClient(node["api_base_url"])
|
||||||
result = await client._request("GET", "/inference/status")
|
result = await client.inference_status()
|
||||||
return ok(result)
|
return ok(result)
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
return ok({"loaded": False, "error": str(exc)})
|
return ok({"loaded": False, "error": str(exc)})
|
||||||
@@ -1475,13 +1917,19 @@ async def model_chat_local_status() -> dict[str, Any]:
|
|||||||
@router.post("/model-chat/trained/preload")
|
@router.post("/model-chat/trained/preload")
|
||||||
async def model_chat_trained_preload(payload: dict[str, Any] = Body(...)) -> dict[str, Any]:
|
async def model_chat_trained_preload(payload: dict[str, Any] = Body(...)) -> dict[str, Any]:
|
||||||
"""Load a trained model (base + adapter) on the compute node for inference."""
|
"""Load a trained model (base + adapter) on the compute node for inference."""
|
||||||
|
model_path = (payload.get("model_name_or_path") or "").strip()
|
||||||
|
if not model_path:
|
||||||
|
return ok({"loaded": False, "error": "model_name_or_path is required"})
|
||||||
store = get_platform_store()
|
store = get_platform_store()
|
||||||
node = _select_first_online_node(store)
|
node = _select_first_online_node(store)
|
||||||
if not node:
|
if not node:
|
||||||
return ok({"loaded": False, "error": "no online compute node"})
|
return ok({"loaded": False, "error": "no online compute node"})
|
||||||
try:
|
try:
|
||||||
client = ComputeNodeClient(node["api_base_url"])
|
client = ComputeNodeClient(node["api_base_url"])
|
||||||
result = await client._request("POST", "/inference/load", json_data=payload)
|
# 计算节点现在异步加载:HTTP 接受(loading/ready)即视为派发成功
|
||||||
|
result = await client.inference_load(payload)
|
||||||
|
if result.get("loaded") or result.get("status") in {"loading", "ready"}:
|
||||||
|
store.mark_inference_loaded(node["id"])
|
||||||
return ok(result)
|
return ok(result)
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
return ok({"loaded": False, "error": str(exc)})
|
return ok({"loaded": False, "error": str(exc)})
|
||||||
@@ -1520,6 +1968,16 @@ async def update_compute_node(node_id: str, payload: dict[str, Any] = Body(...))
|
|||||||
raise fail(400, str(exc))
|
raise fail(400, str(exc))
|
||||||
|
|
||||||
|
|
||||||
|
@router.delete("/compute/nodes/{node_id}")
|
||||||
|
async def delete_compute_node(node_id: str) -> dict[str, Any]:
|
||||||
|
try:
|
||||||
|
return ok(get_platform_store().delete_compute_node(node_id))
|
||||||
|
except KeyError:
|
||||||
|
raise fail(404, "compute node not found")
|
||||||
|
except ValueError as exc:
|
||||||
|
raise fail(400, str(exc))
|
||||||
|
|
||||||
|
|
||||||
@router.post("/compute/nodes/{node_id}/test-connection")
|
@router.post("/compute/nodes/{node_id}/test-connection")
|
||||||
async def test_compute_node(node_id: str) -> dict[str, Any]:
|
async def test_compute_node(node_id: str) -> dict[str, Any]:
|
||||||
store = get_platform_store()
|
store = get_platform_store()
|
||||||
|
|||||||
@@ -88,6 +88,84 @@ def parse_size_bytes(value: Any) -> int:
|
|||||||
return max(0, round(amount * _SIZE_UNIT_BYTES[unit]))
|
return max(0, round(amount * _SIZE_UNIT_BYTES[unit]))
|
||||||
|
|
||||||
|
|
||||||
|
def count_dataset_records(content: str) -> int:
|
||||||
|
text = (content or "").strip()
|
||||||
|
if not text:
|
||||||
|
return 0
|
||||||
|
|
||||||
|
try:
|
||||||
|
value = json.loads(text)
|
||||||
|
if isinstance(value, list):
|
||||||
|
return len(value)
|
||||||
|
return 1
|
||||||
|
except (TypeError, ValueError, json.JSONDecodeError):
|
||||||
|
pass
|
||||||
|
|
||||||
|
return len([line for line in text.splitlines() if line.strip()])
|
||||||
|
|
||||||
|
|
||||||
|
_EVAL_METHOD_LABELS = {
|
||||||
|
"standard": "标准匹配",
|
||||||
|
"metric_standard": "综合评测",
|
||||||
|
"semantic": "语义相似度",
|
||||||
|
"sentiment": "情感分析",
|
||||||
|
"accuracy": "准确性评估",
|
||||||
|
"safety": "安全性评估",
|
||||||
|
"relevance": "相关性评估",
|
||||||
|
"fluency": "流畅性评估",
|
||||||
|
"factuality": "事实性评估",
|
||||||
|
"custom": "自定义评估",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _eval_method_label(value: Any) -> str:
|
||||||
|
if isinstance(value, list):
|
||||||
|
return "、".join(_eval_method_label(item) for item in value if item)
|
||||||
|
text = str(value or "").strip()
|
||||||
|
return _EVAL_METHOD_LABELS.get(text, text)
|
||||||
|
|
||||||
|
|
||||||
|
def _basic_metric_labels(config: dict[str, Any] | None) -> list[str]:
|
||||||
|
cfg = config or {}
|
||||||
|
labels: list[str] = []
|
||||||
|
bleu = cfg.get("bleu") or {}
|
||||||
|
if bleu.get("enabled"):
|
||||||
|
labels.append(f"BLEU-{int(bleu.get('ngram') or 4)}")
|
||||||
|
rouge = cfg.get("rouge") or {}
|
||||||
|
if rouge.get("enabled"):
|
||||||
|
methods = rouge.get("methods") or []
|
||||||
|
method_labels = {
|
||||||
|
"rouge1": "ROUGE-1",
|
||||||
|
"rouge2": "ROUGE-2",
|
||||||
|
"rougeL": "ROUGE-L",
|
||||||
|
"rouge_1": "ROUGE-1",
|
||||||
|
"rouge_2": "ROUGE-2",
|
||||||
|
"rouge_l": "ROUGE-L",
|
||||||
|
}
|
||||||
|
labels.extend(method_labels.get(str(item), str(item)) for item in methods)
|
||||||
|
cosine = cfg.get("cosine") or {}
|
||||||
|
if cosine.get("enabled"):
|
||||||
|
labels.append("Cosine")
|
||||||
|
return labels
|
||||||
|
|
||||||
|
|
||||||
|
def _build_eval_metric_label(payload: dict[str, Any], dimension: dict[str, Any] | None = None) -> str:
|
||||||
|
parts: list[str] = []
|
||||||
|
dim = dimension or {}
|
||||||
|
dim_type = str(dim.get("type") or payload.get("dimension_type") or "").strip()
|
||||||
|
method_label = _eval_method_label(dim.get("eval_method") or payload.get("eval_method"))
|
||||||
|
if dim_type in {"classification", "metric"} and method_label:
|
||||||
|
parts.append(f"LLM:{method_label}")
|
||||||
|
elif dim_type == "text_similarity" and method_label:
|
||||||
|
parts.append(method_label)
|
||||||
|
|
||||||
|
parts.extend(_basic_metric_labels(payload.get("basic_metrics") or {}))
|
||||||
|
if not parts:
|
||||||
|
metric = str(payload.get("metric") or payload.get("eval_type") or "").strip()
|
||||||
|
return "自定义评测" if metric == "custom" else (metric or "-")
|
||||||
|
return " + ".join(parts)
|
||||||
|
|
||||||
|
|
||||||
def version_number(value: Any, default: int = 0) -> int:
|
def version_number(value: Any, default: int = 0) -> int:
|
||||||
try:
|
try:
|
||||||
number = int(value)
|
number = int(value)
|
||||||
@@ -130,8 +208,12 @@ def parse_training_metric_line(line: str) -> dict[str, float] | None:
|
|||||||
if "loss" not in line and "learning_rate" not in line:
|
if "loss" not in line and "learning_rate" not in line:
|
||||||
return None
|
return None
|
||||||
result: dict[str, float] = {}
|
result: dict[str, float] = {}
|
||||||
|
number_pattern = r"([-+]?(?:\d+(?:\.\d*)?|\.\d+)(?:[eE][-+]?\d+)?)"
|
||||||
|
step_match = re.search(rf"(?:^|[\s,{{])['\"]?step['\"]?\s*(?:=|:)\s*{number_pattern}", line, re.I)
|
||||||
|
if step_match:
|
||||||
|
result["step"] = float(step_match.group(1))
|
||||||
for key in ["loss", "grad_norm", "learning_rate", "epoch"]:
|
for key in ["loss", "grad_norm", "learning_rate", "epoch"]:
|
||||||
match = re.search(rf"['\"]?{key}['\"]?\s*:\s*([-+]?\d+(?:\.\d+)?(?:[eE][-+]?\d+)?)", line)
|
match = re.search(rf"['\"]?{key}['\"]?\s*(?:=|:)\s*{number_pattern}", line, re.I)
|
||||||
if match:
|
if match:
|
||||||
result[key] = float(match.group(1))
|
result[key] = float(match.group(1))
|
||||||
return result or None
|
return result or None
|
||||||
@@ -296,10 +378,11 @@ class PlatformStore:
|
|||||||
# request (notably expensive against the remote PostgreSQL instance).
|
# request (notably expensive against the remote PostgreSQL instance).
|
||||||
# TCP keepalive 让操作系统持续保活连接,抵抗远程库空闲静默断连。
|
# TCP keepalive 让操作系统持续保活连接,抵抗远程库空闲静默断连。
|
||||||
pool_kwargs = {
|
pool_kwargs = {
|
||||||
|
"connect_timeout": 5,
|
||||||
"keepalives": 1,
|
"keepalives": 1,
|
||||||
"keepalives_idle": 30,
|
"keepalives_idle": 10,
|
||||||
"keepalives_interval": 10,
|
"keepalives_interval": 5,
|
||||||
"keepalives_count": 5,
|
"keepalives_count": 3,
|
||||||
}
|
}
|
||||||
self._pool = ConnectionPool(
|
self._pool = ConnectionPool(
|
||||||
conninfo=self.database_url,
|
conninfo=self.database_url,
|
||||||
@@ -320,6 +403,20 @@ class PlatformStore:
|
|||||||
self._pool.open()
|
self._pool.open()
|
||||||
self.ensure_schema()
|
self.ensure_schema()
|
||||||
self.ensure_seed_data()
|
self.ensure_seed_data()
|
||||||
|
# Track which compute nodes have an active inference model loaded
|
||||||
|
self._inference_nodes: set[str] = set()
|
||||||
|
self._last_runtime_refresh = 0.0
|
||||||
|
|
||||||
|
# ── inference node tracking ────────────────────────────────────
|
||||||
|
|
||||||
|
def mark_inference_loaded(self, node_id: str) -> None:
|
||||||
|
self._inference_nodes.add(node_id)
|
||||||
|
|
||||||
|
def mark_inference_unloaded(self, node_id: str) -> None:
|
||||||
|
self._inference_nodes.discard(node_id)
|
||||||
|
|
||||||
|
def is_inference_loaded(self, node_id: str) -> bool:
|
||||||
|
return node_id in self._inference_nodes
|
||||||
|
|
||||||
@contextmanager
|
@contextmanager
|
||||||
def connect(self) -> Iterator["PgConnection"]:
|
def connect(self) -> Iterator["PgConnection"]:
|
||||||
@@ -359,6 +456,15 @@ class PlatformStore:
|
|||||||
)
|
)
|
||||||
self._ensure_columns(conn, "gpus", {"last_seen_at": "TEXT"})
|
self._ensure_columns(conn, "gpus", {"last_seen_at": "TEXT"})
|
||||||
self._ensure_columns(conn, "fine_tune_tasks", {"compute_job_id": "TEXT"})
|
self._ensure_columns(conn, "fine_tune_tasks", {"compute_job_id": "TEXT"})
|
||||||
|
self._ensure_columns(
|
||||||
|
conn,
|
||||||
|
"trained_models",
|
||||||
|
{
|
||||||
|
"artifact_dir": "TEXT",
|
||||||
|
"compute_node_id": "TEXT",
|
||||||
|
"compute_node_name": "TEXT",
|
||||||
|
},
|
||||||
|
)
|
||||||
self._ensure_columns(
|
self._ensure_columns(
|
||||||
conn,
|
conn,
|
||||||
"resource_replicas",
|
"resource_replicas",
|
||||||
@@ -433,6 +539,10 @@ class PlatformStore:
|
|||||||
def refresh_runtime_state(self) -> None:
|
def refresh_runtime_state(self) -> None:
|
||||||
if get_settings().compute_mode != "simulator":
|
if get_settings().compute_mode != "simulator":
|
||||||
return
|
return
|
||||||
|
now_ts = time.monotonic()
|
||||||
|
if now_ts - self._last_runtime_refresh < 2:
|
||||||
|
return
|
||||||
|
self._last_runtime_refresh = now_ts
|
||||||
|
|
||||||
with self.connect() as conn:
|
with self.connect() as conn:
|
||||||
rows = conn.execute(
|
rows = conn.execute(
|
||||||
@@ -492,6 +602,15 @@ class PlatformStore:
|
|||||||
name = task.get("output_model_name") or f"{task['name']}-lora"
|
name = task.get("output_model_name") or f"{task['name']}-lora"
|
||||||
exists = conn.execute("SELECT id FROM trained_models WHERE name=?", (name,)).fetchone()
|
exists = conn.execute("SELECT id FROM trained_models WHERE name=?", (name,)).fetchone()
|
||||||
if exists:
|
if exists:
|
||||||
|
conn.execute(
|
||||||
|
"""
|
||||||
|
UPDATE trained_models
|
||||||
|
SET compute_node_id=COALESCE(compute_node_id, ?),
|
||||||
|
compute_node_name=COALESCE(compute_node_name, ?)
|
||||||
|
WHERE id=?
|
||||||
|
""",
|
||||||
|
(task.get("compute_node_id"), task.get("compute_node_code") or task.get("compute_node_name"), exists["id"]),
|
||||||
|
)
|
||||||
return
|
return
|
||||||
model = conn.execute("SELECT path FROM models WHERE id=?", (task.get("base_model"),)).fetchone()
|
model = conn.execute("SELECT path FROM models WHERE id=?", (task.get("base_model"),)).fetchone()
|
||||||
output_dir = task.get("output_dir") or f"/data/yg-ft/outputs/{task['name']}"
|
output_dir = task.get("output_dir") or f"/data/yg-ft/outputs/{task['name']}"
|
||||||
@@ -499,8 +618,8 @@ class PlatformStore:
|
|||||||
conn.execute(
|
conn.execute(
|
||||||
"""
|
"""
|
||||||
INSERT INTO trained_models
|
INSERT INTO trained_models
|
||||||
(id, name, train_methods, base_model_path, create_time, merged, merging, merged_path, artifact_dir)
|
(id, name, train_methods, base_model_path, create_time, merged, merging, merged_path, artifact_dir, compute_node_id, compute_node_name)
|
||||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
|
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||||
""",
|
""",
|
||||||
(
|
(
|
||||||
trained_model_id,
|
trained_model_id,
|
||||||
@@ -512,6 +631,8 @@ class PlatformStore:
|
|||||||
0,
|
0,
|
||||||
output_dir,
|
output_dir,
|
||||||
output_dir,
|
output_dir,
|
||||||
|
task.get("compute_node_id"),
|
||||||
|
task.get("compute_node_code") or task.get("compute_node_name"),
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
# Use real artifact data from compute node when available
|
# Use real artifact data from compute node when available
|
||||||
@@ -778,7 +899,7 @@ class PlatformStore:
|
|||||||
(
|
(
|
||||||
new_id("metric"),
|
new_id("metric"),
|
||||||
task_id,
|
task_id,
|
||||||
line_number,
|
int(metric.get("step") or line_number),
|
||||||
metric.get("epoch"),
|
metric.get("epoch"),
|
||||||
metric.get("loss"),
|
metric.get("loss"),
|
||||||
metric.get("grad_norm"),
|
metric.get("grad_norm"),
|
||||||
@@ -1245,15 +1366,25 @@ class PlatformStore:
|
|||||||
self.refresh_runtime_state()
|
self.refresh_runtime_state()
|
||||||
with self.connect() as conn:
|
with self.connect() as conn:
|
||||||
rows = conn.execute("SELECT * FROM trained_models ORDER BY create_time DESC").fetchall()
|
rows = conn.execute("SELECT * FROM trained_models ORDER BY create_time DESC").fetchall()
|
||||||
return [
|
items = []
|
||||||
{
|
for row in rows:
|
||||||
|
item = {
|
||||||
**dict(row),
|
**dict(row),
|
||||||
"train_methods": json_loads(row["train_methods"], []),
|
"train_methods": json_loads(row["train_methods"], []),
|
||||||
"merged": bool(row["merged"]),
|
"merged": bool(row["merged"]),
|
||||||
"merging": bool(row["merging"]),
|
"merging": bool(row["merging"]),
|
||||||
}
|
}
|
||||||
for row in rows
|
if not item.get("compute_node_id"):
|
||||||
]
|
task_rows = conn.execute("SELECT payload FROM fine_tune_tasks ORDER BY create_time DESC").fetchall()
|
||||||
|
for task in task_rows:
|
||||||
|
task_payload = json_loads(task["payload"], {})
|
||||||
|
output_name = task_payload.get("output_model_name") or f"{task_payload.get('name')}-lora"
|
||||||
|
if output_name == item["name"]:
|
||||||
|
item["compute_node_id"] = task_payload.get("compute_node_id")
|
||||||
|
item["compute_node_name"] = task_payload.get("compute_node_code") or task_payload.get("compute_node_name")
|
||||||
|
break
|
||||||
|
items.append(item)
|
||||||
|
return items
|
||||||
|
|
||||||
def delete_trained_model(self, model_id: str) -> None:
|
def delete_trained_model(self, model_id: str) -> None:
|
||||||
with self.connect() as conn:
|
with self.connect() as conn:
|
||||||
@@ -1298,6 +1429,7 @@ class PlatformStore:
|
|||||||
file_size_bytes = int(file_row.get("size_bytes") or 0)
|
file_size_bytes = int(file_row.get("size_bytes") or 0)
|
||||||
if file_size_bytes <= 0:
|
if file_size_bytes <= 0:
|
||||||
file_size_bytes = parse_size_bytes(file_row.get("size"))
|
file_size_bytes = parse_size_bytes(file_row.get("size"))
|
||||||
|
file_record_count = int(file_row.get("record_count") or 0)
|
||||||
decoded_files.append(
|
decoded_files.append(
|
||||||
{
|
{
|
||||||
"id": file_row["id"],
|
"id": file_row["id"],
|
||||||
@@ -1306,7 +1438,7 @@ class PlatformStore:
|
|||||||
"size_bytes": file_size_bytes,
|
"size_bytes": file_size_bytes,
|
||||||
**dataset_file_version_summary(file_row),
|
**dataset_file_version_summary(file_row),
|
||||||
"create_time": file_row["create_time"],
|
"create_time": file_row["create_time"],
|
||||||
"record_count": int(file_row.get("record_count") or 0),
|
"record_count": file_record_count,
|
||||||
"split": metadata.get("file_split"),
|
"split": metadata.get("file_split"),
|
||||||
}
|
}
|
||||||
)
|
)
|
||||||
@@ -1324,6 +1456,9 @@ class PlatformStore:
|
|||||||
total_size_bytes = int(row.get("size_bytes") or 0)
|
total_size_bytes = int(row.get("size_bytes") or 0)
|
||||||
if total_size_bytes <= 0:
|
if total_size_bytes <= 0:
|
||||||
total_size_bytes = parse_size_bytes(row.get("size"))
|
total_size_bytes = parse_size_bytes(row.get("size"))
|
||||||
|
total_record_count = sum(int(item.get("record_count") or 0) for item in decoded_files)
|
||||||
|
if not decoded_files:
|
||||||
|
total_record_count = int(row.get("record_count") or row.get("count") or 0)
|
||||||
current_version_nos = sorted(
|
current_version_nos = sorted(
|
||||||
{
|
{
|
||||||
int(item["current_version_no"])
|
int(item["current_version_no"])
|
||||||
@@ -1333,6 +1468,8 @@ class PlatformStore:
|
|||||||
)
|
)
|
||||||
return {
|
return {
|
||||||
**dict(row),
|
**dict(row),
|
||||||
|
"count": total_record_count,
|
||||||
|
"record_count": total_record_count,
|
||||||
"size_bytes": total_size_bytes,
|
"size_bytes": total_size_bytes,
|
||||||
"current_version_no": (
|
"current_version_no": (
|
||||||
current_version_nos[0] if len(current_version_nos) == 1 else None
|
current_version_nos[0] if len(current_version_nos) == 1 else None
|
||||||
@@ -1407,7 +1544,7 @@ class PlatformStore:
|
|||||||
version_id = f"{file_id}_v1"
|
version_id = f"{file_id}_v1"
|
||||||
size_bytes = len(content.encode("utf-8"))
|
size_bytes = len(content.encode("utf-8"))
|
||||||
size = f"{size_bytes} B"
|
size = f"{size_bytes} B"
|
||||||
record_count = len([line for line in content.splitlines() if line.strip()])
|
record_count = count_dataset_records(content)
|
||||||
version = {
|
version = {
|
||||||
"id": version_id,
|
"id": version_id,
|
||||||
"version": 1,
|
"version": 1,
|
||||||
@@ -1440,10 +1577,18 @@ class PlatformStore:
|
|||||||
)
|
)
|
||||||
conn.execute(
|
conn.execute(
|
||||||
"""UPDATE datasets
|
"""UPDATE datasets
|
||||||
SET count=count+?, record_count=record_count+?,
|
SET count=stats.record_count,
|
||||||
size_bytes=size_bytes+?, size=((size_bytes+?)::text || ' B')
|
record_count=stats.record_count,
|
||||||
|
size_bytes=stats.size_bytes,
|
||||||
|
size=(stats.size_bytes::text || ' B')
|
||||||
|
FROM (
|
||||||
|
SELECT COALESCE(SUM(record_count), 0) AS record_count,
|
||||||
|
COALESCE(SUM(size_bytes), 0) AS size_bytes
|
||||||
|
FROM dataset_files
|
||||||
|
WHERE dataset_id=?
|
||||||
|
) stats
|
||||||
WHERE id=?""",
|
WHERE id=?""",
|
||||||
(record_count, record_count, size_bytes, size_bytes, dataset_id),
|
(dataset_id, dataset_id),
|
||||||
)
|
)
|
||||||
return {
|
return {
|
||||||
"id": file_id,
|
"id": file_id,
|
||||||
@@ -1548,19 +1693,57 @@ class PlatformStore:
|
|||||||
row = conn.execute("SELECT * FROM dataset_files WHERE id=?", (file_id,)).fetchone()
|
row = conn.execute("SELECT * FROM dataset_files WHERE id=?", (file_id,)).fetchone()
|
||||||
if not row:
|
if not row:
|
||||||
raise KeyError(file_id)
|
raise KeyError(file_id)
|
||||||
|
content = payload.get("content", "")
|
||||||
|
size_bytes = len(content.encode("utf-8"))
|
||||||
|
record_count = count_dataset_records(content)
|
||||||
versions = json_loads(row["versions"], [])
|
versions = json_loads(row["versions"], [])
|
||||||
version = {
|
version = {
|
||||||
"id": f"{file_id}_v{len(versions) + 1}",
|
"id": f"{file_id}_v{len(versions) + 1}",
|
||||||
"version": len(versions) + 1,
|
"version": len(versions) + 1,
|
||||||
|
"version_no": len(versions) + 1,
|
||||||
"create_time": utcnow(),
|
"create_time": utcnow(),
|
||||||
"description": payload.get("description", "online edit"),
|
"description": payload.get("description", "online edit"),
|
||||||
|
"size_bytes": size_bytes,
|
||||||
|
"record_count": record_count,
|
||||||
}
|
}
|
||||||
versions.append(version)
|
versions.append(version)
|
||||||
conn.execute(
|
conn.execute(
|
||||||
"UPDATE dataset_files SET content=?, active_version_id=?, versions=? WHERE id=?",
|
"""
|
||||||
(payload.get("content", ""), version["id"], json_dumps(versions), file_id),
|
UPDATE dataset_files
|
||||||
|
SET content=?, active_version_id=?, current_version_id=?, versions=?,
|
||||||
|
size_bytes=?, size=?, record_count=?, version_no=?
|
||||||
|
WHERE id=?
|
||||||
|
""",
|
||||||
|
(
|
||||||
|
content,
|
||||||
|
version["id"],
|
||||||
|
version["id"],
|
||||||
|
json_dumps(versions),
|
||||||
|
size_bytes,
|
||||||
|
f"{size_bytes} B",
|
||||||
|
record_count,
|
||||||
|
version["version_no"],
|
||||||
|
file_id,
|
||||||
|
),
|
||||||
)
|
)
|
||||||
return {"version": version, "content": payload.get("content", "")}
|
conn.execute(
|
||||||
|
"""UPDATE datasets
|
||||||
|
SET count=stats.record_count,
|
||||||
|
record_count=stats.record_count,
|
||||||
|
size_bytes=stats.size_bytes,
|
||||||
|
size=(stats.size_bytes::text || ' B')
|
||||||
|
FROM (
|
||||||
|
SELECT dataset_id,
|
||||||
|
COALESCE(SUM(record_count), 0) AS record_count,
|
||||||
|
COALESCE(SUM(size_bytes), 0) AS size_bytes
|
||||||
|
FROM dataset_files
|
||||||
|
WHERE dataset_id=(SELECT dataset_id FROM dataset_files WHERE id=?)
|
||||||
|
GROUP BY dataset_id
|
||||||
|
) stats
|
||||||
|
WHERE datasets.id=stats.dataset_id""",
|
||||||
|
(file_id,),
|
||||||
|
)
|
||||||
|
return {"version": version, "content": content}
|
||||||
|
|
||||||
def activate_file_version(self, file_id: str, version_id: str) -> dict[str, Any]:
|
def activate_file_version(self, file_id: str, version_id: str) -> dict[str, Any]:
|
||||||
with self.connect() as conn:
|
with self.connect() as conn:
|
||||||
@@ -2064,19 +2247,54 @@ class PlatformStore:
|
|||||||
def _json_payload_row(self, row: PgRow) -> dict[str, Any]:
|
def _json_payload_row(self, row: PgRow) -> dict[str, Any]:
|
||||||
payload = json_loads(row["payload"], {})
|
payload = json_loads(row["payload"], {})
|
||||||
payload.update({"id": row["id"], "status": row.get("status"), "create_time": row["create_time"]})
|
payload.update({"id": row["id"], "status": row.get("status"), "create_time": row["create_time"]})
|
||||||
|
if not payload.get("metric_label"):
|
||||||
|
payload["metric_label"] = _build_eval_metric_label(payload)
|
||||||
|
if not payload.get("metric") or payload.get("metric") == "custom":
|
||||||
|
payload["metric"] = payload["metric_label"]
|
||||||
return payload
|
return payload
|
||||||
|
|
||||||
|
def _enrich_eval_payload(self, conn: PgConnection, payload: dict[str, Any]) -> dict[str, Any]:
|
||||||
|
data = dict(payload)
|
||||||
|
model_id = str(data.get("model_id") or "")
|
||||||
|
if model_id and not data.get("model_name"):
|
||||||
|
model = conn.execute("SELECT name FROM models WHERE id=?", (model_id,)).fetchone()
|
||||||
|
if not model:
|
||||||
|
model = conn.execute("SELECT name FROM trained_models WHERE id=? OR name=?", (model_id, model_id)).fetchone()
|
||||||
|
if model:
|
||||||
|
data["model_name"] = model["name"]
|
||||||
|
|
||||||
|
dataset_id = str(data.get("dataset_id") or "")
|
||||||
|
if dataset_id and not data.get("dataset"):
|
||||||
|
dataset = conn.execute("SELECT name FROM datasets WHERE id=?", (dataset_id,)).fetchone()
|
||||||
|
if dataset:
|
||||||
|
data["dataset"] = dataset["name"]
|
||||||
|
|
||||||
|
dimension = None
|
||||||
|
dimension_id = str(data.get("dimension_id") or "")
|
||||||
|
if dimension_id:
|
||||||
|
dimension_row = conn.execute("SELECT payload FROM eval_dimensions WHERE id=?", (dimension_id,)).fetchone()
|
||||||
|
if dimension_row:
|
||||||
|
dimension = json_loads(dimension_row["payload"], {})
|
||||||
|
data.setdefault("dimension_type", dimension.get("type"))
|
||||||
|
data.setdefault("eval_method", dimension.get("eval_method"))
|
||||||
|
data.setdefault("evaluator_model", dimension.get("eval_model"))
|
||||||
|
|
||||||
|
data["metric_label"] = _build_eval_metric_label(data, dimension)
|
||||||
|
if not data.get("metric") or data.get("metric") in {"custom", "自定义评测"}:
|
||||||
|
data["metric"] = data["metric_label"]
|
||||||
|
return data
|
||||||
|
|
||||||
def eval_tasks(self) -> list[dict[str, Any]]:
|
def eval_tasks(self) -> list[dict[str, Any]]:
|
||||||
with self.connect() as conn:
|
with self.connect() as conn:
|
||||||
rows = conn.execute("SELECT * FROM eval_tasks ORDER BY create_time DESC").fetchall()
|
rows = conn.execute("SELECT * FROM eval_tasks ORDER BY create_time DESC").fetchall()
|
||||||
return [self._json_payload_row(row) for row in rows]
|
return [self._enrich_eval_payload(conn, self._json_payload_row(row)) for row in rows]
|
||||||
|
|
||||||
def eval_task(self, task_id: str) -> dict[str, Any]:
|
def eval_task(self, task_id: str) -> dict[str, Any]:
|
||||||
with self.connect() as conn:
|
with self.connect() as conn:
|
||||||
row = conn.execute("SELECT * FROM eval_tasks WHERE id=?", (task_id,)).fetchone()
|
row = conn.execute("SELECT * FROM eval_tasks WHERE id=?", (task_id,)).fetchone()
|
||||||
if not row:
|
if not row:
|
||||||
raise KeyError(task_id)
|
raise KeyError(task_id)
|
||||||
payload = self._json_payload_row(row)
|
payload = self._enrich_eval_payload(conn, self._json_payload_row(row))
|
||||||
payload.setdefault("sample_count", 0)
|
payload.setdefault("sample_count", 0)
|
||||||
payload.setdefault("completed_count", 0)
|
payload.setdefault("completed_count", 0)
|
||||||
payload.setdefault("passed_count", 0)
|
payload.setdefault("passed_count", 0)
|
||||||
@@ -2102,22 +2320,92 @@ class PlatformStore:
|
|||||||
"metric": payload.get("metric") or "custom",
|
"metric": payload.get("metric") or "custom",
|
||||||
}
|
}
|
||||||
with self.connect() as conn:
|
with self.connect() as conn:
|
||||||
model = conn.execute("SELECT name FROM models WHERE id=?", (str(payload.get("model_id")),)).fetchone()
|
model_id = str(payload.get("model_id") or "")
|
||||||
|
model = conn.execute("SELECT name FROM models WHERE id=?", (model_id,)).fetchone()
|
||||||
|
trained_model = conn.execute("SELECT name FROM trained_models WHERE id=? OR name=?", (model_id, model_id)).fetchone()
|
||||||
dataset = conn.execute("SELECT name FROM datasets WHERE id=?", (str(payload.get("dataset_id")),)).fetchone()
|
dataset = conn.execute("SELECT name FROM datasets WHERE id=?", (str(payload.get("dataset_id")),)).fetchone()
|
||||||
|
dimension = None
|
||||||
|
dimension_id = str(payload.get("dimension_id") or "")
|
||||||
|
if dimension_id:
|
||||||
|
dimension_row = conn.execute("SELECT payload FROM eval_dimensions WHERE id=?", (dimension_id,)).fetchone()
|
||||||
|
if dimension_row:
|
||||||
|
dimension = json_loads(dimension_row["payload"], {})
|
||||||
if model:
|
if model:
|
||||||
data.setdefault("model_name", model["name"])
|
data.setdefault("model_name", model["name"])
|
||||||
|
elif trained_model:
|
||||||
|
data.setdefault("model_name", trained_model["name"])
|
||||||
if dataset:
|
if dataset:
|
||||||
data.setdefault("dataset", dataset["name"])
|
data.setdefault("dataset", dataset["name"])
|
||||||
|
if dimension:
|
||||||
|
data.setdefault("dimension_type", dimension.get("type"))
|
||||||
|
data.setdefault("eval_method", dimension.get("eval_method"))
|
||||||
|
data.setdefault("evaluator_model", dimension.get("eval_model"))
|
||||||
|
data["metric_label"] = _build_eval_metric_label(data, dimension)
|
||||||
|
if data.get("metric") == "custom":
|
||||||
|
data["metric"] = data["metric_label"]
|
||||||
conn.execute(
|
conn.execute(
|
||||||
"INSERT INTO eval_tasks (id, name, payload, status, create_time) VALUES (?, ?, ?, ?, ?)",
|
"INSERT INTO eval_tasks (id, name, payload, status, create_time) VALUES (?, ?, ?, ?, ?)",
|
||||||
(task_id, name, json_dumps(data), status, now),
|
(task_id, name, json_dumps(data), status, now),
|
||||||
)
|
)
|
||||||
return self.eval_task(task_id)
|
return self.eval_task(task_id)
|
||||||
|
|
||||||
|
def update_eval_task(self, task_id: str, updates: dict[str, Any]) -> dict[str, Any]:
|
||||||
|
"""Update fields in an eval task's payload without replacing the whole record."""
|
||||||
|
task = self.eval_task(task_id)
|
||||||
|
merged = {**task, **updates}
|
||||||
|
with self.connect() as conn:
|
||||||
|
conn.execute(
|
||||||
|
"UPDATE eval_tasks SET payload=?, status=? WHERE id=?",
|
||||||
|
(json_dumps(merged), merged.get("status", task.get("status", "pending")), task_id),
|
||||||
|
)
|
||||||
|
return self.eval_task(task_id)
|
||||||
|
|
||||||
def delete_eval_task(self, task_id: str) -> None:
|
def delete_eval_task(self, task_id: str) -> None:
|
||||||
with self.connect() as conn:
|
with self.connect() as conn:
|
||||||
conn.execute("DELETE FROM eval_tasks WHERE id=?", (task_id,))
|
conn.execute("DELETE FROM eval_tasks WHERE id=?", (task_id,))
|
||||||
|
|
||||||
|
def running_eval_tasks(self) -> list[dict[str, Any]]:
|
||||||
|
"""Return eval tasks that have been submitted to a compute node and are still running."""
|
||||||
|
return [
|
||||||
|
task for task in self.eval_tasks()
|
||||||
|
if task.get("compute_job_id") and task.get("status") in {"queued", "running"}
|
||||||
|
]
|
||||||
|
|
||||||
|
def apply_eval_job_result(self, task_id: str, job: dict[str, Any], result_content: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||||
|
"""Sync a compute job status/result back to an eval task."""
|
||||||
|
task = self.eval_task(task_id)
|
||||||
|
job_status = str(job.get("status", ""))
|
||||||
|
status_map = {"queued": "running", "running": "running", "completed": "completed",
|
||||||
|
"failed": "failed", "stopped": "stopped"}
|
||||||
|
new_status = status_map.get(job_status, job_status or task.get("status", "pending"))
|
||||||
|
updates: dict[str, Any] = {
|
||||||
|
"status": new_status,
|
||||||
|
"progress": int(job.get("progress", 0)),
|
||||||
|
"output_dir": job.get("output_dir", task.get("output_dir", "")),
|
||||||
|
}
|
||||||
|
# On completion, populate results from eval_results.json content
|
||||||
|
if new_status == "completed" and result_content:
|
||||||
|
updates.update({
|
||||||
|
"overall_score": result_content.get("overall_score", 0),
|
||||||
|
"overall_score_max": result_content.get("overall_score_max", 100),
|
||||||
|
"overall_evaluation": result_content.get("overall_evaluation", ""),
|
||||||
|
"improvement_suggestions": result_content.get("improvement_suggestions", []),
|
||||||
|
"dimension_summary": result_content.get("dimension_summary", []),
|
||||||
|
"samples": result_content.get("samples", []),
|
||||||
|
"sample_count": result_content.get("sample_count", 0),
|
||||||
|
"completed_count": result_content.get("completed_count", 0),
|
||||||
|
"passed_count": result_content.get("passed_count", 0),
|
||||||
|
"basic_metrics": result_content.get("basic_metrics", {}),
|
||||||
|
"score": result_content.get("overall_score", 0),
|
||||||
|
"completed_time": utcnow(),
|
||||||
|
})
|
||||||
|
elif new_status in {"failed", "stopped"}:
|
||||||
|
updates.update({
|
||||||
|
"error": job.get("error") or task.get("error") or "",
|
||||||
|
"completed_time": utcnow(),
|
||||||
|
})
|
||||||
|
return self.update_eval_task(task_id, updates)
|
||||||
|
|
||||||
def dimensions(self) -> list[dict[str, Any]]:
|
def dimensions(self) -> list[dict[str, Any]]:
|
||||||
with self.connect() as conn:
|
with self.connect() as conn:
|
||||||
rows = conn.execute("SELECT * FROM eval_dimensions ORDER BY create_time DESC").fetchall()
|
rows = conn.execute("SELECT * FROM eval_dimensions ORDER BY create_time DESC").fetchall()
|
||||||
@@ -2284,6 +2572,15 @@ class PlatformStore:
|
|||||||
).fetchall()
|
).fetchall()
|
||||||
return {int(row["gpu_index"]) for row in rows}
|
return {int(row["gpu_index"]) for row in rows}
|
||||||
|
|
||||||
|
def _node_gpu_indexes(self, conn: PgConnection, node: dict[str, Any]) -> set[int]:
|
||||||
|
rows = conn.execute("SELECT gpu_index FROM gpus WHERE node_id=?", (node["id"],)).fetchall()
|
||||||
|
if rows:
|
||||||
|
return {int(row["gpu_index"]) for row in rows}
|
||||||
|
return set(range(max(0, int(node.get("gpu_count") or 0))))
|
||||||
|
|
||||||
|
def _node_capacity(self, node: dict[str, Any]) -> int:
|
||||||
|
return max(1, int(node.get("max_parallel_jobs") or 1), int(node.get("gpu_count") or 0))
|
||||||
|
|
||||||
def _schedule_node_locked(self, conn: PgConnection, payload: dict[str, Any]) -> dict[str, Any]:
|
def _schedule_node_locked(self, conn: PgConnection, payload: dict[str, Any]) -> dict[str, Any]:
|
||||||
requested = payload.get("requested_node_id") or payload.get("compute_node_id")
|
requested = payload.get("requested_node_id") or payload.get("compute_node_id")
|
||||||
requested_gpus = [int(item) for item in payload.get("gpus") or []]
|
requested_gpus = [int(item) for item in payload.get("gpus") or []]
|
||||||
@@ -2291,13 +2588,15 @@ class PlatformStore:
|
|||||||
candidates = [
|
candidates = [
|
||||||
n
|
n
|
||||||
for n in nodes
|
for n in nodes
|
||||||
if n["enabled"] and n["scheduler_status"] == "online" and n["current_running_jobs"] < n["max_parallel_jobs"]
|
if n["enabled"] and n["scheduler_status"] == "online" and n["current_running_jobs"] < self._node_capacity(n)
|
||||||
]
|
]
|
||||||
if requested_gpus:
|
if requested_gpus:
|
||||||
|
requested_gpu_set = set(requested_gpus)
|
||||||
candidates = [
|
candidates = [
|
||||||
node
|
node
|
||||||
for node in candidates
|
for node in candidates
|
||||||
if not set(requested_gpus).intersection(self._active_gpu_indexes(conn, node["id"]))
|
if requested_gpu_set.issubset(self._node_gpu_indexes(conn, node))
|
||||||
|
and not requested_gpu_set.intersection(self._active_gpu_indexes(conn, node["id"]))
|
||||||
]
|
]
|
||||||
if requested:
|
if requested:
|
||||||
selected = next((n for n in candidates if n["id"] == requested), None)
|
selected = next((n for n in candidates if n["id"] == requested), None)
|
||||||
@@ -2312,8 +2611,8 @@ class PlatformStore:
|
|||||||
reason = "disabled"
|
reason = "disabled"
|
||||||
elif node["scheduler_status"] != "online":
|
elif node["scheduler_status"] != "online":
|
||||||
reason = f"status={node['scheduler_status']}"
|
reason = f"status={node['scheduler_status']}"
|
||||||
elif node["current_running_jobs"] >= node["max_parallel_jobs"]:
|
elif node["current_running_jobs"] >= self._node_capacity(node):
|
||||||
reason = f"capacity full {node['current_running_jobs']}/{node['max_parallel_jobs']}"
|
reason = f"capacity full {node['current_running_jobs']}/{self._node_capacity(node)}"
|
||||||
else:
|
else:
|
||||||
reason = "not selected"
|
reason = "not selected"
|
||||||
reasons.append(f"{node['code']}({reason})")
|
reasons.append(f"{node['code']}({reason})")
|
||||||
@@ -2538,6 +2837,28 @@ class PlatformStore:
|
|||||||
"SELECT compute_node_id, COUNT(*) AS cnt FROM fine_tune_tasks WHERE status IN ('syncing','queued','running') GROUP BY compute_node_id"
|
"SELECT compute_node_id, COUNT(*) AS cnt FROM fine_tune_tasks WHERE status IN ('syncing','queued','running') GROUP BY compute_node_id"
|
||||||
).fetchall()
|
).fetchall()
|
||||||
running_map = {r["compute_node_id"]: r["cnt"] for r in running}
|
running_map = {r["compute_node_id"]: r["cnt"] for r in running}
|
||||||
|
# 评测任务同样占用算力节点,纳入运行任务统计
|
||||||
|
for row in conn.execute(
|
||||||
|
"SELECT payload FROM eval_tasks WHERE status IN ('syncing','queued','running')"
|
||||||
|
).fetchall():
|
||||||
|
node_id = json_loads(row["payload"], {}).get("compute_node_id")
|
||||||
|
if node_id:
|
||||||
|
running_map[node_id] = running_map.get(node_id, 0) + 1
|
||||||
|
# 推理模型占用算力节点同样计入:优先从 compare_tasks 持久化状态派生
|
||||||
|
# (重启后仍准确),并用内存标记兜底(直接 preload 的模型无 compare 记录)
|
||||||
|
inference_node_ids = set(self._inference_nodes)
|
||||||
|
for ctr in conn.execute("SELECT payload FROM compare_tasks").fetchall():
|
||||||
|
ls = json_loads(ctr["payload"], {}).get("load_status") or {}
|
||||||
|
if isinstance(ls, str):
|
||||||
|
try:
|
||||||
|
ls = json.loads(ls)
|
||||||
|
except (json.JSONDecodeError, TypeError):
|
||||||
|
ls = {}
|
||||||
|
for m in ls.get("loaded_models") or []:
|
||||||
|
if m.get("status") in {"ready", "running"} and m.get("node_id"):
|
||||||
|
inference_node_ids.add(m["node_id"])
|
||||||
|
for nid in inference_node_ids:
|
||||||
|
running_map[nid] = running_map.get(nid, 0) + 1
|
||||||
rows = conn.execute("SELECT * FROM compute_nodes ORDER BY scheduler_weight DESC, code").fetchall()
|
rows = conn.execute("SELECT * FROM compute_nodes ORDER BY scheduler_weight DESC, code").fetchall()
|
||||||
return [
|
return [
|
||||||
{
|
{
|
||||||
@@ -2664,6 +2985,24 @@ class PlatformStore:
|
|||||||
)
|
)
|
||||||
return next(node for node in self.compute_nodes() if node["id"] == node_id)
|
return next(node for node in self.compute_nodes() if node["id"] == node_id)
|
||||||
|
|
||||||
|
def delete_compute_node(self, node_id: str) -> dict[str, Any]:
|
||||||
|
with self.connect() as conn:
|
||||||
|
node = conn.execute("SELECT * FROM compute_nodes WHERE id=?", (node_id,)).fetchone()
|
||||||
|
if not node:
|
||||||
|
raise KeyError(node_id)
|
||||||
|
active = conn.execute(
|
||||||
|
"""
|
||||||
|
SELECT COUNT(*) AS cnt
|
||||||
|
FROM fine_tune_tasks
|
||||||
|
WHERE compute_node_id=? AND status IN ('syncing','queued','running')
|
||||||
|
""",
|
||||||
|
(node_id,),
|
||||||
|
).fetchone()
|
||||||
|
if active and int(active["cnt"] or 0) > 0:
|
||||||
|
raise ValueError("compute node has active training tasks")
|
||||||
|
conn.execute("DELETE FROM compute_nodes WHERE id=?", (node_id,))
|
||||||
|
return {"deleted": node_id}
|
||||||
|
|
||||||
def update_compute_node_health(self, node_id: str, health: dict[str, Any], success: bool, error: str | None = None) -> dict[str, Any]:
|
def update_compute_node_health(self, node_id: str, health: dict[str, Any], success: bool, error: str | None = None) -> dict[str, Any]:
|
||||||
current = next((n for n in self.compute_nodes() if n["id"] == node_id), None)
|
current = next((n for n in self.compute_nodes() if n["id"] == node_id), None)
|
||||||
if not current:
|
if not current:
|
||||||
@@ -2737,6 +3076,26 @@ class PlatformStore:
|
|||||||
"SELECT * FROM fine_tune_tasks WHERE status IN ('syncing','queued','running')"
|
"SELECT * FROM fine_tune_tasks WHERE status IN ('syncing','queued','running')"
|
||||||
).fetchall()
|
).fetchall()
|
||||||
]
|
]
|
||||||
|
# 评测任务同样占用节点 GPU
|
||||||
|
eval_running = [
|
||||||
|
json_loads(row["payload"], {})
|
||||||
|
for row in conn.execute(
|
||||||
|
"SELECT payload FROM eval_tasks WHERE status IN ('syncing','queued','running')"
|
||||||
|
).fetchall()
|
||||||
|
]
|
||||||
|
# 推理模型占用的节点:优先从 compare_tasks 持久化状态派生(重启后仍准确),
|
||||||
|
# 内存标记兜底(直接 preload 的模型无 compare 记录)
|
||||||
|
inference_node_ids = set(self._inference_nodes)
|
||||||
|
for ctr in conn.execute("SELECT payload FROM compare_tasks").fetchall():
|
||||||
|
ls = json_loads(ctr["payload"], {}).get("load_status") or {}
|
||||||
|
if isinstance(ls, str):
|
||||||
|
try:
|
||||||
|
ls = json.loads(ls)
|
||||||
|
except (json.JSONDecodeError, TypeError):
|
||||||
|
ls = {}
|
||||||
|
for m in ls.get("loaded_models") or []:
|
||||||
|
if m.get("status") in {"ready", "running"} and m.get("node_id"):
|
||||||
|
inference_node_ids.add(m["node_id"])
|
||||||
items = []
|
items = []
|
||||||
for row in rows:
|
for row in rows:
|
||||||
task = next(
|
task = next(
|
||||||
@@ -2747,8 +3106,23 @@ class PlatformStore:
|
|||||||
),
|
),
|
||||||
None,
|
None,
|
||||||
)
|
)
|
||||||
busy = task is not None and task.get("status") == "running"
|
eval_task = next(
|
||||||
reserved = task is not None and task.get("status") in {"syncing", "queued"}
|
(
|
||||||
|
t
|
||||||
|
for t in eval_running
|
||||||
|
if t.get("compute_node_id") == row["node_id"]
|
||||||
|
and row["gpu_index"] == (int(t["gpu_id"]) if t.get("gpu_id") is not None else -1)
|
||||||
|
),
|
||||||
|
None,
|
||||||
|
)
|
||||||
|
busy = (task is not None and task.get("status") == "running") or eval_task is not None
|
||||||
|
reserved = (task is not None and task.get("status") in {"syncing", "queued"}) or (
|
||||||
|
eval_task is not None and eval_task.get("status") in {"syncing", "queued"}
|
||||||
|
)
|
||||||
|
# Also mark GPU as busy if an inference model is loaded on this node
|
||||||
|
if row["node_id"] in inference_node_ids and not busy:
|
||||||
|
busy = True
|
||||||
|
reserved = False
|
||||||
memory_used = round(row["memory_total_gb"] * (0.72 if busy else 0.18 if reserved else 0.04), 1)
|
memory_used = round(row["memory_total_gb"] * (0.72 if busy else 0.18 if reserved else 0.04), 1)
|
||||||
gpu_percent = 86 if busy else 22 if reserved else 3
|
gpu_percent = 86 if busy else 22 if reserved else 3
|
||||||
memory_total = float(row["memory_total_gb"] or 0)
|
memory_total = float(row["memory_total_gb"] or 0)
|
||||||
@@ -2778,6 +3152,16 @@ class PlatformStore:
|
|||||||
}
|
}
|
||||||
]
|
]
|
||||||
if task
|
if task
|
||||||
|
else [
|
||||||
|
{
|
||||||
|
"pid": int(eval_task.get("process_id") or 0),
|
||||||
|
"name": "eval_runner",
|
||||||
|
"memory_used_gb": memory_used,
|
||||||
|
"task_name": eval_task.get("eval_task_name") or eval_task.get("name") or "评测任务",
|
||||||
|
"user": "admin",
|
||||||
|
}
|
||||||
|
]
|
||||||
|
if eval_task
|
||||||
else [],
|
else [],
|
||||||
}
|
}
|
||||||
)
|
)
|
||||||
@@ -2827,11 +3211,24 @@ class PlatformStore:
|
|||||||
}
|
}
|
||||||
|
|
||||||
def health_metrics(self) -> dict[str, float]:
|
def health_metrics(self) -> dict[str, float]:
|
||||||
info = self.system_info()
|
# 轻量健康检查:采集真实 CPU/内存/磁盘使用率
|
||||||
|
# 用于顶部栏快速展示与 Docker 健康检查。
|
||||||
|
try:
|
||||||
|
import psutil
|
||||||
|
# cpu_percent(interval=None) 首次调用返回 0,需要短暂采样
|
||||||
|
cpu_percent = float(psutil.cpu_percent(interval=0.1))
|
||||||
|
memory_percent = float(psutil.virtual_memory().percent)
|
||||||
|
# Windows 兼容:尝试当前盘符
|
||||||
|
try:
|
||||||
|
disk_percent = float(psutil.disk_usage('/').percent)
|
||||||
|
except Exception:
|
||||||
|
disk_percent = float(psutil.disk_usage('C:\\').percent)
|
||||||
|
except Exception:
|
||||||
|
cpu_percent = memory_percent = disk_percent = 0.0
|
||||||
return {
|
return {
|
||||||
"cpu_percent": info["cpu"]["percent"],
|
"cpu_percent": round(cpu_percent, 1),
|
||||||
"memory_percent": info["memory"]["percent"],
|
"memory_percent": round(memory_percent, 1),
|
||||||
"disk_percent": info["disk"]["percent"],
|
"disk_percent": round(disk_percent, 1),
|
||||||
}
|
}
|
||||||
|
|
||||||
def queue(self) -> list[dict[str, Any]]:
|
def queue(self) -> list[dict[str, Any]]:
|
||||||
|
|||||||
@@ -33,7 +33,10 @@ CREATE TABLE IF NOT EXISTS trained_models (
|
|||||||
create_time TEXT NOT NULL,
|
create_time TEXT NOT NULL,
|
||||||
merged INTEGER NOT NULL DEFAULT 0,
|
merged INTEGER NOT NULL DEFAULT 0,
|
||||||
merging INTEGER NOT NULL DEFAULT 0,
|
merging INTEGER NOT NULL DEFAULT 0,
|
||||||
merged_path TEXT
|
merged_path TEXT,
|
||||||
|
artifact_dir TEXT,
|
||||||
|
compute_node_id TEXT,
|
||||||
|
compute_node_name TEXT
|
||||||
);
|
);
|
||||||
|
|
||||||
CREATE TABLE IF NOT EXISTS model_lineage (
|
CREATE TABLE IF NOT EXISTS model_lineage (
|
||||||
|
|||||||
@@ -33,6 +33,16 @@ def _unwrap_dict(payload: Any) -> dict[str, Any]:
|
|||||||
return payload if isinstance(payload, dict) else {}
|
return payload if isinstance(payload, dict) else {}
|
||||||
|
|
||||||
|
|
||||||
|
# Inference calls are intentionally short-timeout:
|
||||||
|
# - load dispatch only confirms the compute node accepted the request
|
||||||
|
# (the actual model load now runs asynchronously on the node).
|
||||||
|
# - status/unload must never block the platform for long when a node is
|
||||||
|
# unreachable but still marked online.
|
||||||
|
INFERENCE_LOAD_TIMEOUT = httpx.Timeout(30, connect=10)
|
||||||
|
INFERENCE_STATUS_TIMEOUT = httpx.Timeout(30, connect=5)
|
||||||
|
INFERENCE_UNLOAD_TIMEOUT = httpx.Timeout(30, connect=5)
|
||||||
|
|
||||||
|
|
||||||
class ComputeNodeClient:
|
class ComputeNodeClient:
|
||||||
"""Application-side client for one compute node.
|
"""Application-side client for one compute node.
|
||||||
|
|
||||||
@@ -182,10 +192,16 @@ class ComputeNodeClient:
|
|||||||
response.raise_for_status()
|
response.raise_for_status()
|
||||||
return _unwrap_dict(response.json())
|
return _unwrap_dict(response.json())
|
||||||
|
|
||||||
async def _request(self, method: str, path: str, json_data: dict[str, Any] | None = None) -> dict[str, Any]:
|
async def _request(
|
||||||
|
self,
|
||||||
|
method: str,
|
||||||
|
path: str,
|
||||||
|
json_data: dict[str, Any] | None = None,
|
||||||
|
timeout: float | None = None,
|
||||||
|
) -> dict[str, Any]:
|
||||||
"""Generic request method for compute API endpoints."""
|
"""Generic request method for compute API endpoints."""
|
||||||
url = _join_url(self.api_base_url, f"{self.route_prefix}{path}")
|
url = _join_url(self.api_base_url, f"{self.route_prefix}{path}")
|
||||||
async with httpx.AsyncClient(timeout=300, headers=self.headers()) as client:
|
async with httpx.AsyncClient(timeout=timeout or 300, headers=self.headers()) as client:
|
||||||
if method.upper() == "GET":
|
if method.upper() == "GET":
|
||||||
response = await client.get(url)
|
response = await client.get(url)
|
||||||
else:
|
else:
|
||||||
@@ -193,6 +209,19 @@ class ComputeNodeClient:
|
|||||||
response.raise_for_status()
|
response.raise_for_status()
|
||||||
return _unwrap_dict(response.json())
|
return _unwrap_dict(response.json())
|
||||||
|
|
||||||
|
# ── Inference helpers (short timeouts — see module constants) ──────────
|
||||||
|
|
||||||
|
async def inference_load(self, payload: dict[str, Any]) -> dict[str, Any]:
|
||||||
|
"""Dispatch a model load. Returns as soon as the node accepts the
|
||||||
|
request; the node now loads asynchronously (status goes 'loading')."""
|
||||||
|
return await self._request("POST", "/inference/load", json_data=payload, timeout=INFERENCE_LOAD_TIMEOUT)
|
||||||
|
|
||||||
|
async def inference_status(self) -> dict[str, Any]:
|
||||||
|
return await self._request("GET", "/inference/status", timeout=INFERENCE_STATUS_TIMEOUT)
|
||||||
|
|
||||||
|
async def inference_unload(self) -> dict[str, Any]:
|
||||||
|
return await self._request("POST", "/inference/unload", json_data={}, timeout=INFERENCE_UNLOAD_TIMEOUT)
|
||||||
|
|
||||||
async def upload_file(
|
async def upload_file(
|
||||||
self,
|
self,
|
||||||
filename: str,
|
filename: str,
|
||||||
@@ -207,7 +236,8 @@ class ComputeNodeClient:
|
|||||||
"resource_id": resource_id or "",
|
"resource_id": resource_id or "",
|
||||||
}
|
}
|
||||||
files = {"file": (filename, content)}
|
files = {"file": (filename, content)}
|
||||||
async with httpx.AsyncClient(timeout=max(self.timeout, 60), headers=self.headers()) as client:
|
timeout = httpx.Timeout(max(self.timeout, 60), connect=self.timeout)
|
||||||
|
async with httpx.AsyncClient(timeout=timeout, headers=self.headers()) as client:
|
||||||
response = await client.post(
|
response = await client.post(
|
||||||
_join_url(self.api_base_url, f"{self.route_prefix}/compute/files/upload"),
|
_join_url(self.api_base_url, f"{self.route_prefix}/compute/files/upload"),
|
||||||
data=data,
|
data=data,
|
||||||
|
|||||||
@@ -1,15 +1,119 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import time
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
from app.db.platform_store import get_platform_store
|
from app.db.platform_store import get_platform_store
|
||||||
from app.modules.compute_gateway.client import ComputeNodeClient
|
from app.modules.compute_gateway.client import ComputeNodeClient
|
||||||
|
|
||||||
|
# starting 状态允许的最大轮询次数(约 40 * 3s ≈ 2 分钟),超过即判定节点不可达
|
||||||
|
MAX_STARTING_ATTEMPTS = 40
|
||||||
|
|
||||||
|
|
||||||
def _node_for_task(task: dict[str, Any]) -> dict[str, Any] | None:
|
def _node_for_task(task: dict[str, Any]) -> dict[str, Any] | None:
|
||||||
return next((node for node in get_platform_store().compute_nodes() if node["id"] == task.get("compute_node_id")), None)
|
return next((node for node in get_platform_store().compute_nodes() if node["id"] == task.get("compute_node_id")), None)
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_inference_load_status(task: dict[str, Any]) -> tuple[list[dict[str, Any]], dict[str, Any]]:
|
||||||
|
load_status = task.get("load_status") or {}
|
||||||
|
if isinstance(load_status, str):
|
||||||
|
try:
|
||||||
|
load_status = json.loads(load_status)
|
||||||
|
except (json.JSONDecodeError, TypeError):
|
||||||
|
load_status = {}
|
||||||
|
return load_status.get("loaded_models") or [], load_status
|
||||||
|
|
||||||
|
|
||||||
|
async def reconcile_inference_loads(store: Any) -> list[dict[str, Any]]:
|
||||||
|
"""推进处于 starting 状态的推理加载。
|
||||||
|
|
||||||
|
模型加载已改为异步派发:/model-compare/{id}/load 立即返回,这里在每次
|
||||||
|
轮询时查询对应计算节点的 /inference/status,把任务从 starting 推进到
|
||||||
|
ready/error。使用短超时,单节点不可达不会阻塞整轮轮询。
|
||||||
|
"""
|
||||||
|
reconciled: list[dict[str, Any]] = []
|
||||||
|
now = time.time()
|
||||||
|
for task in store.compare_tasks():
|
||||||
|
items, _ = _parse_inference_load_status(task)
|
||||||
|
if not any(item.get("status") == "starting" for item in items):
|
||||||
|
continue
|
||||||
|
# dirty 只要处理过任一 starting 项就置位:load_attempts / last_polled_at
|
||||||
|
# 必须落库,否则节点不可达时计数不会累积,封顶逻辑永远触发不了
|
||||||
|
dirty = False
|
||||||
|
for item in items:
|
||||||
|
if item.get("status") != "starting":
|
||||||
|
continue
|
||||||
|
# 节流:同一 item 每 3s 只查询一次
|
||||||
|
if now - float(item.get("last_polled_at") or 0) < 3:
|
||||||
|
continue
|
||||||
|
item["last_polled_at"] = now
|
||||||
|
item["load_attempts"] = int(item.get("load_attempts") or 0) + 1
|
||||||
|
dirty = True
|
||||||
|
node = next((n for n in store.compute_nodes() if n["id"] == item.get("node_id")), None)
|
||||||
|
if not node:
|
||||||
|
item["status"] = "error"
|
||||||
|
item["error"] = "compute node deleted"
|
||||||
|
store.mark_inference_unloaded(item.get("node_id") or "")
|
||||||
|
continue
|
||||||
|
if not node.get("enabled") or node.get("scheduler_status") != "online":
|
||||||
|
item["status"] = "error"
|
||||||
|
item["error"] = "compute node offline"
|
||||||
|
store.mark_inference_unloaded(node["id"])
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
status = await ComputeNodeClient(node["api_base_url"]).inference_status()
|
||||||
|
except Exception as exc: # noqa: BLE001 - node unreachable; keep retrying until cap
|
||||||
|
if int(item.get("load_attempts") or 0) >= MAX_STARTING_ATTEMPTS:
|
||||||
|
item["status"] = "error"
|
||||||
|
item["error"] = f"compute node unreachable: {exc}"
|
||||||
|
store.mark_inference_unloaded(node["id"])
|
||||||
|
continue
|
||||||
|
node_status = status.get("status")
|
||||||
|
if node_status == "ready":
|
||||||
|
item["status"] = "ready"
|
||||||
|
item.pop("error", None)
|
||||||
|
store.mark_inference_loaded(node["id"])
|
||||||
|
elif node_status == "error":
|
||||||
|
item["status"] = "error"
|
||||||
|
item["error"] = status.get("error") or "model load failed on compute node"
|
||||||
|
store.mark_inference_unloaded(node["id"])
|
||||||
|
elif node_status == "idle":
|
||||||
|
# 节点重启导致已加载模型丢失
|
||||||
|
item["status"] = "error"
|
||||||
|
item["error"] = "model disappeared from compute node (node may have restarted)"
|
||||||
|
store.mark_inference_unloaded(node["id"])
|
||||||
|
# node_status == "loading" -> 保持 starting,下轮再查
|
||||||
|
if dirty:
|
||||||
|
if any(i.get("status") in {"ready", "running"} for i in items):
|
||||||
|
new_status = "loaded"
|
||||||
|
elif any(i.get("status") == "starting" for i in items):
|
||||||
|
new_status = "starting" # 仍在加载中,保持 starting
|
||||||
|
else:
|
||||||
|
new_status = "failed"
|
||||||
|
store.update_compare_task(task["id"], {"status": new_status, "load_status": {"loaded_models": items}})
|
||||||
|
reconciled.append({"task_id": task["id"], "status": new_status})
|
||||||
|
return reconciled
|
||||||
|
|
||||||
|
|
||||||
|
async def fetch_eval_result_content(client: ComputeNodeClient, node: dict[str, Any], job: dict[str, Any]) -> dict[str, Any] | None:
|
||||||
|
output_dir = job.get("output_dir")
|
||||||
|
if not output_dir:
|
||||||
|
return None
|
||||||
|
full_path = f"{str(output_dir).rstrip('/')}/eval_results.json"
|
||||||
|
data_root = "/data/yg-ft/"
|
||||||
|
if full_path.startswith(data_root):
|
||||||
|
full_path = full_path[len(data_root):]
|
||||||
|
rel_path = full_path.lstrip("/")
|
||||||
|
import httpx
|
||||||
|
url = f"{node['api_base_url'].rstrip('/')}/modelTF/compute/files/read"
|
||||||
|
async with httpx.AsyncClient(timeout=30, headers=client.headers()) as http:
|
||||||
|
response = await http.get(url, params={"path": rel_path})
|
||||||
|
response.raise_for_status()
|
||||||
|
payload = response.json()
|
||||||
|
return payload if isinstance(payload, dict) else None
|
||||||
|
|
||||||
|
|
||||||
async def poll_compute_jobs_once() -> dict[str, Any]:
|
async def poll_compute_jobs_once() -> dict[str, Any]:
|
||||||
store = get_platform_store()
|
store = get_platform_store()
|
||||||
synced: list[dict[str, Any]] = []
|
synced: list[dict[str, Any]] = []
|
||||||
@@ -48,4 +152,40 @@ async def poll_compute_jobs_once() -> dict[str, Any]:
|
|||||||
standalone_synced.append(store.sync_model_merge_job(record["id"], job))
|
standalone_synced.append(store.sync_model_merge_job(record["id"], job))
|
||||||
except Exception as exc: # noqa: BLE001 - keep polling other jobs
|
except Exception as exc: # noqa: BLE001 - keep polling other jobs
|
||||||
failed.append({"job_id": record["id"], "error": str(exc)})
|
failed.append({"job_id": record["id"], "error": str(exc)})
|
||||||
return {"synced": len(synced) + len(standalone_synced), "failed": failed, "items": synced, "standalone": standalone_synced}
|
|
||||||
|
# ── Eval job sync ────────────────────────────────────────────────
|
||||||
|
eval_synced = 0
|
||||||
|
for eval_task in store.running_eval_tasks():
|
||||||
|
node = next(
|
||||||
|
(item for item in store.compute_nodes() if item["id"] == eval_task.get("compute_node_id")),
|
||||||
|
None,
|
||||||
|
)
|
||||||
|
if not node:
|
||||||
|
failed.append({"eval_task_id": eval_task["id"], "error": "compute node not found"})
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
client = ComputeNodeClient(node["api_base_url"])
|
||||||
|
job = await client.get_job(eval_task["compute_job_id"])
|
||||||
|
result_content = None
|
||||||
|
# Try to read eval_results.json from the job output directory
|
||||||
|
if job.get("status") == "completed" and job.get("output_dir"):
|
||||||
|
try:
|
||||||
|
result_content = await fetch_eval_result_content(client, node, job)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
store.apply_eval_job_result(eval_task["id"], job, result_content)
|
||||||
|
# 评测 GPU 占用由 eval_tasks 状态派生,无需维护推理内存标记
|
||||||
|
eval_synced += 1
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
failed.append({"eval_task_id": eval_task["id"], "error": str(exc)})
|
||||||
|
|
||||||
|
# ── Inference load reconciliation ─────────────────────────────────────
|
||||||
|
try:
|
||||||
|
inference_reconciled = await reconcile_inference_loads(store)
|
||||||
|
except Exception as exc: # noqa: BLE001 - keep polling alive
|
||||||
|
failed.append({"inference_reconcile": str(exc)})
|
||||||
|
inference_reconciled = []
|
||||||
|
|
||||||
|
return {"synced": len(synced) + len(standalone_synced) + eval_synced, "failed": failed,
|
||||||
|
"items": synced, "standalone": standalone_synced, "eval_synced": eval_synced,
|
||||||
|
"inference_reconciled": inference_reconciled}
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
from fastapi import APIRouter, Query
|
from fastapi import APIRouter, Body, Query, Request
|
||||||
from fastapi.responses import StreamingResponse
|
from fastapi.responses import StreamingResponse
|
||||||
|
|
||||||
from app.db.platform_store import ALL_PERMISSIONS, get_platform_store
|
from app.db.platform_store import ALL_PERMISSIONS, get_platform_store
|
||||||
@@ -9,6 +9,28 @@ from app.db.platform_store import ALL_PERMISSIONS, get_platform_store
|
|||||||
router = APIRouter(prefix="/system", tags=["system"])
|
router = APIRouter(prefix="/system", tags=["system"])
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/audit/visit")
|
||||||
|
def record_visit(payload: dict = Body(...), request: Request = None) -> dict:
|
||||||
|
"""记录用户访问业务模块的行为,用于看板用户操作分布统计。"""
|
||||||
|
action = str(payload.get("action") or payload.get("module") or "").strip()
|
||||||
|
if not action:
|
||||||
|
return {"code": 0, "message": "ok", "data": {"recorded": False}}
|
||||||
|
actor_id = ""
|
||||||
|
if request is not None:
|
||||||
|
auth = request.headers.get("Authorization", "")
|
||||||
|
token = auth.replace("Bearer ", "").strip()
|
||||||
|
if token.startswith("platform-token-"):
|
||||||
|
actor_id = token[len("platform-token-"):]
|
||||||
|
get_platform_store().record_audit(
|
||||||
|
action=action,
|
||||||
|
actor_id=actor_id or None,
|
||||||
|
target_type="module",
|
||||||
|
target_id=action,
|
||||||
|
detail=str(payload.get("detail") or ""),
|
||||||
|
)
|
||||||
|
return {"code": 0, "message": "ok", "data": {"recorded": True}}
|
||||||
|
|
||||||
|
|
||||||
@router.get("/permissions/codes")
|
@router.get("/permissions/codes")
|
||||||
def permission_codes() -> dict:
|
def permission_codes() -> dict:
|
||||||
"""返回平台权限码清单(权限码接口)。"""
|
"""返回平台权限码清单(权限码接口)。"""
|
||||||
|
|||||||
@@ -10,6 +10,7 @@ dependencies = [
|
|||||||
"pydantic>=2.7.0",
|
"pydantic>=2.7.0",
|
||||||
"sqlalchemy>=2.0.30",
|
"sqlalchemy>=2.0.30",
|
||||||
"psycopg[binary]>=3.2.1",
|
"psycopg[binary]>=3.2.1",
|
||||||
|
"psycopg-pool>=3.2.1",
|
||||||
"alembic>=1.13.1",
|
"alembic>=1.13.1",
|
||||||
"redis>=5.0.4",
|
"redis>=5.0.4",
|
||||||
"httpx>=0.27.0",
|
"httpx>=0.27.0",
|
||||||
|
|||||||
@@ -4,6 +4,7 @@ python-multipart>=0.0.9
|
|||||||
pydantic>=2.7.0
|
pydantic>=2.7.0
|
||||||
sqlalchemy>=2.0.30
|
sqlalchemy>=2.0.30
|
||||||
psycopg[binary]>=3.2.1
|
psycopg[binary]>=3.2.1
|
||||||
|
psycopg-pool>=3.2.1
|
||||||
alembic>=1.13.1
|
alembic>=1.13.1
|
||||||
redis>=5.0.4
|
redis>=5.0.4
|
||||||
httpx>=0.27.0
|
httpx>=0.27.0
|
||||||
@@ -18,3 +19,7 @@ llama-index-core==0.14.23
|
|||||||
llama-index-embeddings-huggingface==0.6.1
|
llama-index-embeddings-huggingface==0.6.1
|
||||||
docling==2.115.0
|
docling==2.115.0
|
||||||
tiktoken>=0.7.0
|
tiktoken>=0.7.0
|
||||||
|
|
||||||
|
# 测试与代码检查
|
||||||
|
pytest>=8.2.0
|
||||||
|
ruff>=0.5.0
|
||||||
|
|||||||
276
backend/tests/test_compare_inference_async.py
Normal file
276
backend/tests/test_compare_inference_async.py
Normal file
@@ -0,0 +1,276 @@
|
|||||||
|
"""
|
||||||
|
模型推理异步加载改造的单元测试。
|
||||||
|
|
||||||
|
覆盖:
|
||||||
|
- model_compare_load:异步派发,立即返回 starting + 节点信息(不等待加载完成)
|
||||||
|
- model_compare_delete:先删记录,卸载失败也不阻塞删除
|
||||||
|
- reconcile_inference_loads:starting -> ready/error/idle/不可达的状态迁移与封顶
|
||||||
|
- _unload_from_compute_node:任务感知,只命中记录中的节点
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
from types import SimpleNamespace
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from app.api.v1.endpoints.platform import model_compare_delete, model_compare_load
|
||||||
|
import app.api.v1.endpoints.platform as platform
|
||||||
|
from app.modules.compute_gateway.client import ComputeNodeClient
|
||||||
|
from app.modules.compute_gateway.sync import MAX_STARTING_ATTEMPTS, reconcile_inference_loads
|
||||||
|
|
||||||
|
|
||||||
|
class FakeInferenceStore:
|
||||||
|
"""内存 store,仅实现推理加载/对账用到的接口。"""
|
||||||
|
|
||||||
|
def __init__(self, tasks: list[dict[str, Any]] | None = None, nodes: list[dict[str, Any]] | None = None) -> None:
|
||||||
|
self._tasks: dict[str, dict[str, Any]] = {t["id"]: dict(t) for t in (tasks or [])}
|
||||||
|
self._nodes = nodes or []
|
||||||
|
self._inference_nodes: set[str] = set()
|
||||||
|
|
||||||
|
def compare_task(self, task_id: str) -> dict[str, Any]:
|
||||||
|
if task_id not in self._tasks:
|
||||||
|
raise KeyError(task_id)
|
||||||
|
return dict(self._tasks[task_id])
|
||||||
|
|
||||||
|
def compare_tasks(self) -> list[dict[str, Any]]:
|
||||||
|
return [dict(t) for t in self._tasks.values()]
|
||||||
|
|
||||||
|
def update_compare_task(self, task_id: str, payload: dict[str, Any]) -> dict[str, Any]:
|
||||||
|
current = self._tasks[task_id]
|
||||||
|
merged = {**current, **payload, "id": task_id}
|
||||||
|
self._tasks[task_id] = merged
|
||||||
|
return dict(merged)
|
||||||
|
|
||||||
|
def delete_compare_task(self, task_id: str) -> None:
|
||||||
|
self._tasks.pop(task_id, None)
|
||||||
|
|
||||||
|
def compute_nodes(self) -> list[dict[str, Any]]:
|
||||||
|
return [dict(n) for n in self._nodes]
|
||||||
|
|
||||||
|
def model(self, model_id: str) -> dict[str, Any]:
|
||||||
|
raise KeyError(model_id)
|
||||||
|
|
||||||
|
def trained_models(self) -> list[dict[str, Any]]:
|
||||||
|
return []
|
||||||
|
|
||||||
|
def mark_inference_loaded(self, node_id: str) -> None:
|
||||||
|
self._inference_nodes.add(node_id)
|
||||||
|
|
||||||
|
def mark_inference_unloaded(self, node_id: str) -> None:
|
||||||
|
self._inference_nodes.discard(node_id)
|
||||||
|
|
||||||
|
def is_inference_loaded(self, node_id: str) -> bool:
|
||||||
|
return node_id in self._inference_nodes
|
||||||
|
|
||||||
|
|
||||||
|
def _node(node_id: str, code: str = "") -> dict[str, Any]:
|
||||||
|
return {
|
||||||
|
"id": node_id,
|
||||||
|
"code": code or node_id,
|
||||||
|
"name": code or node_id,
|
||||||
|
"api_base_url": f"http://{code or node_id}:19100",
|
||||||
|
"enabled": True,
|
||||||
|
"scheduler_status": "online",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _task(task_id: str, *, node_id: str | None = None, load_status: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||||
|
return {
|
||||||
|
"id": task_id,
|
||||||
|
"name": f"task-{task_id}",
|
||||||
|
"status": "pending",
|
||||||
|
"models": [
|
||||||
|
{"model_id": "m_1", "model_name": "qwen", "model_path": "/models/qwen", "node_id": node_id}
|
||||||
|
],
|
||||||
|
"load_status": load_status or {"loaded_models": []},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
async def _fake_inference_load(self, payload: dict[str, Any]) -> dict[str, Any]:
|
||||||
|
return {"loaded": False, "status": "loading", "request_id": "req-1"}
|
||||||
|
|
||||||
|
|
||||||
|
async def _fake_inference_unload(self) -> dict[str, Any]:
|
||||||
|
return {"unloaded": True, "status": "idle"}
|
||||||
|
|
||||||
|
|
||||||
|
def _patch_store(monkeypatch, store: FakeInferenceStore) -> None:
|
||||||
|
monkeypatch.setattr(platform, "get_platform_store", lambda: store)
|
||||||
|
monkeypatch.setattr(platform, "get_settings", lambda: SimpleNamespace(compute_mode="real"))
|
||||||
|
|
||||||
|
|
||||||
|
def test_select_eval_node_prefers_model_node(monkeypatch) -> None:
|
||||||
|
from app.api.v1.endpoints.platform import _select_eval_node
|
||||||
|
|
||||||
|
store = FakeInferenceStore(nodes=[_node("n1"), _node("n2")])
|
||||||
|
# 指定模型所在节点时优先返回该节点
|
||||||
|
assert _select_eval_node(store, "n2")["id"] == "n2"
|
||||||
|
# 无指定节点时回退到第一个在线节点
|
||||||
|
assert _select_eval_node(store, None)["id"] == "n1"
|
||||||
|
|
||||||
|
|
||||||
|
def test_select_eval_node_returns_none_when_model_node_offline(monkeypatch) -> None:
|
||||||
|
from app.api.v1.endpoints.platform import _select_eval_node
|
||||||
|
|
||||||
|
nodes = [_node("n1"), _node("n2")]
|
||||||
|
nodes[1]["enabled"] = False
|
||||||
|
store = FakeInferenceStore(nodes=nodes)
|
||||||
|
# 模型所在节点不可用 → 明确失败,不派发到其它节点
|
||||||
|
assert _select_eval_node(store, "n2") is None
|
||||||
|
# 无指定节点时仍回退第一个在线节点
|
||||||
|
assert _select_eval_node(store, None)["id"] == "n1"
|
||||||
|
|
||||||
|
|
||||||
|
def test_model_compare_load_dispatches_and_returns_starting(monkeypatch) -> None:
|
||||||
|
store = FakeInferenceStore(tasks=[_task("t1", node_id="n1")], nodes=[_node("n1")])
|
||||||
|
_patch_store(monkeypatch, store)
|
||||||
|
monkeypatch.setattr(ComputeNodeClient, "inference_load", _fake_inference_load)
|
||||||
|
|
||||||
|
result = asyncio.run(model_compare_load("t1"))
|
||||||
|
assert result["code"] == 0
|
||||||
|
updated = result["data"]
|
||||||
|
assert updated["status"] == "starting"
|
||||||
|
items = updated["load_status"]["loaded_models"]
|
||||||
|
assert items[0]["status"] == "starting"
|
||||||
|
assert items[0]["node_id"] == "n1"
|
||||||
|
assert "n1" in store._inference_nodes
|
||||||
|
|
||||||
|
|
||||||
|
def test_model_compare_load_marks_error_when_all_nodes_fail(monkeypatch) -> None:
|
||||||
|
store = FakeInferenceStore(tasks=[_task("t1", node_id="n1")], nodes=[_node("n1")])
|
||||||
|
_patch_store(monkeypatch, store)
|
||||||
|
|
||||||
|
async def _raise(self, payload: dict[str, Any]) -> dict[str, Any]:
|
||||||
|
raise RuntimeError("conn refused")
|
||||||
|
|
||||||
|
monkeypatch.setattr(ComputeNodeClient, "inference_load", _raise)
|
||||||
|
|
||||||
|
result = asyncio.run(model_compare_load("t1"))
|
||||||
|
updated = result["data"]
|
||||||
|
assert updated["status"] == "failed"
|
||||||
|
assert updated["load_status"]["loaded_models"][0]["status"] == "error"
|
||||||
|
assert "conn refused" in updated["load_status"]["loaded_models"][0]["error"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_model_compare_delete_removes_record_even_if_unload_raises(monkeypatch) -> None:
|
||||||
|
task = _task(
|
||||||
|
"t1",
|
||||||
|
node_id="n1",
|
||||||
|
load_status={"loaded_models": [{"model_id": "m_1", "status": "ready", "node_id": "n1"}]},
|
||||||
|
)
|
||||||
|
store = FakeInferenceStore(tasks=[task], nodes=[_node("n1")])
|
||||||
|
_patch_store(monkeypatch, store)
|
||||||
|
|
||||||
|
async def _raise(self) -> dict[str, Any]:
|
||||||
|
raise RuntimeError("unload boom")
|
||||||
|
|
||||||
|
monkeypatch.setattr(ComputeNodeClient, "inference_unload", _raise)
|
||||||
|
|
||||||
|
result = asyncio.run(model_compare_delete("t1"))
|
||||||
|
assert result["data"] == {"deleted": "t1"}
|
||||||
|
assert "t1" not in store._tasks
|
||||||
|
# finally 中仍清掉了节点标记
|
||||||
|
assert "n1" not in store._inference_nodes
|
||||||
|
|
||||||
|
|
||||||
|
def test_unload_from_compute_node_only_hits_recorded_node(monkeypatch) -> None:
|
||||||
|
task = _task(
|
||||||
|
"t1",
|
||||||
|
load_status={"loaded_models": [{"model_id": "m_1", "status": "ready", "node_id": "n1"}]},
|
||||||
|
)
|
||||||
|
store = FakeInferenceStore(tasks=[task], nodes=[_node("n1"), _node("n2")])
|
||||||
|
_patch_store(monkeypatch, store)
|
||||||
|
monkeypatch.setattr(ComputeNodeClient, "inference_unload", _fake_inference_unload)
|
||||||
|
|
||||||
|
from app.api.v1.endpoints.platform import _unload_from_compute_node
|
||||||
|
|
||||||
|
result = asyncio.run(_unload_from_compute_node(store, task=task))
|
||||||
|
assert result["unloaded"] is True
|
||||||
|
# 只命中任务记录中的节点 n1,n2 未被卸载
|
||||||
|
assert [r["node_id"] for r in result["nodes"]] == ["n1"]
|
||||||
|
assert "n1" not in store._inference_nodes
|
||||||
|
|
||||||
|
|
||||||
|
async def _status_ready(self) -> dict[str, Any]:
|
||||||
|
return {"loaded": True, "status": "ready", "model_name": "qwen"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_reconcile_transitions_starting_to_ready(monkeypatch) -> None:
|
||||||
|
task = _task(
|
||||||
|
"t1",
|
||||||
|
node_id="n1",
|
||||||
|
load_status={"loaded_models": [{"model_id": "m_1", "status": "starting", "node_id": "n1"}]},
|
||||||
|
)
|
||||||
|
store = FakeInferenceStore(tasks=[task], nodes=[_node("n1")])
|
||||||
|
monkeypatch.setattr(ComputeNodeClient, "inference_status", _status_ready)
|
||||||
|
|
||||||
|
reconciled = asyncio.run(reconcile_inference_loads(store))
|
||||||
|
assert reconciled == [{"task_id": "t1", "status": "loaded"}]
|
||||||
|
updated = store._tasks["t1"]
|
||||||
|
assert updated["status"] == "loaded"
|
||||||
|
assert updated["load_status"]["loaded_models"][0]["status"] == "ready"
|
||||||
|
assert "n1" in store._inference_nodes
|
||||||
|
|
||||||
|
|
||||||
|
def test_reconcile_transitions_to_error_and_failed(monkeypatch) -> None:
|
||||||
|
async def _status_error(self) -> dict[str, Any]:
|
||||||
|
return {"loaded": False, "status": "error", "error": "CUDA out of memory"}
|
||||||
|
|
||||||
|
task = _task(
|
||||||
|
"t1",
|
||||||
|
node_id="n1",
|
||||||
|
load_status={"loaded_models": [{"model_id": "m_1", "status": "starting", "node_id": "n1"}]},
|
||||||
|
)
|
||||||
|
store = FakeInferenceStore(tasks=[task], nodes=[_node("n1")])
|
||||||
|
monkeypatch.setattr(ComputeNodeClient, "inference_status", _status_error)
|
||||||
|
|
||||||
|
reconciled = asyncio.run(reconcile_inference_loads(store))
|
||||||
|
assert reconciled == [{"task_id": "t1", "status": "failed"}]
|
||||||
|
item = store._tasks["t1"]["load_status"]["loaded_models"][0]
|
||||||
|
assert item["status"] == "error"
|
||||||
|
assert "CUDA out of memory" in item["error"]
|
||||||
|
assert "n1" not in store._inference_nodes
|
||||||
|
|
||||||
|
|
||||||
|
def test_reconcile_idle_marks_model_disappeared(monkeypatch) -> None:
|
||||||
|
async def _status_idle(self) -> dict[str, Any]:
|
||||||
|
return {"loaded": False, "status": "idle"}
|
||||||
|
|
||||||
|
task = _task(
|
||||||
|
"t1",
|
||||||
|
node_id="n1",
|
||||||
|
load_status={"loaded_models": [{"model_id": "m_1", "status": "starting", "node_id": "n1"}]},
|
||||||
|
)
|
||||||
|
store = FakeInferenceStore(tasks=[task], nodes=[_node("n1")])
|
||||||
|
monkeypatch.setattr(ComputeNodeClient, "inference_status", _status_idle)
|
||||||
|
|
||||||
|
asyncio.run(reconcile_inference_loads(store))
|
||||||
|
item = store._tasks["t1"]["load_status"]["loaded_models"][0]
|
||||||
|
assert item["status"] == "error"
|
||||||
|
assert "disappeared" in item["error"]
|
||||||
|
assert store._tasks["t1"]["status"] == "failed"
|
||||||
|
|
||||||
|
|
||||||
|
def test_reconcile_unreachable_node_flips_to_error_after_cap(monkeypatch) -> None:
|
||||||
|
async def _raise(self) -> dict[str, Any]:
|
||||||
|
raise RuntimeError("conn refused")
|
||||||
|
|
||||||
|
task = _task(
|
||||||
|
"t1",
|
||||||
|
node_id="n1",
|
||||||
|
load_status={"loaded_models": [{"model_id": "m_1", "status": "starting", "node_id": "n1"}]},
|
||||||
|
)
|
||||||
|
store = FakeInferenceStore(tasks=[task], nodes=[_node("n1")])
|
||||||
|
monkeypatch.setattr(ComputeNodeClient, "inference_status", _raise)
|
||||||
|
|
||||||
|
# 每次轮询前重置节流时间戳,逐次推进 load_attempts 到封顶
|
||||||
|
for _ in range(MAX_STARTING_ATTEMPTS):
|
||||||
|
item = store._tasks["t1"]["load_status"]["loaded_models"][0]
|
||||||
|
item["last_polled_at"] = 0
|
||||||
|
asyncio.run(reconcile_inference_loads(store))
|
||||||
|
|
||||||
|
item = store._tasks["t1"]["load_status"]["loaded_models"][0]
|
||||||
|
assert item["status"] == "error"
|
||||||
|
assert "unreachable" in item["error"]
|
||||||
|
assert store._tasks["t1"]["status"] == "failed"
|
||||||
@@ -1,5 +1,7 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import json
|
||||||
import os
|
import os
|
||||||
import math
|
import math
|
||||||
import hashlib
|
import hashlib
|
||||||
@@ -449,6 +451,29 @@ def create_app() -> FastAPI:
|
|||||||
accelerator_errors, accelerator_warnings, accelerator = _validate_training_accelerator(payload)
|
accelerator_errors, accelerator_warnings, accelerator = _validate_training_accelerator(payload)
|
||||||
errors.extend(accelerator_errors)
|
errors.extend(accelerator_errors)
|
||||||
warnings.extend(accelerator_warnings)
|
warnings.extend(accelerator_warnings)
|
||||||
|
elif engine == "eval":
|
||||||
|
# Eval engine: validate model path and dataset path
|
||||||
|
if not payload.get("model_name_or_path"):
|
||||||
|
errors.append("model_name_or_path is required for eval")
|
||||||
|
else:
|
||||||
|
path_checks.append(_check_path_item({
|
||||||
|
"name": "model_name_or_path",
|
||||||
|
"path": payload.get("model_name_or_path", ""),
|
||||||
|
"type": "any",
|
||||||
|
"required": True,
|
||||||
|
}))
|
||||||
|
if payload.get("dataset_path"):
|
||||||
|
path_checks.append(_check_path_item({
|
||||||
|
"name": "dataset_path",
|
||||||
|
"path": payload.get("dataset_path", ""),
|
||||||
|
"type": "file",
|
||||||
|
"required": True,
|
||||||
|
}))
|
||||||
|
else:
|
||||||
|
errors.append("dataset_path is required for eval")
|
||||||
|
if shutil.which("python") is None:
|
||||||
|
errors.append("python runtime not found")
|
||||||
|
|
||||||
elif engine == "smoke":
|
elif engine == "smoke":
|
||||||
warnings.append("smoke engine skips model and dataset path checks")
|
warnings.append("smoke engine skips model and dataset path checks")
|
||||||
|
|
||||||
@@ -688,14 +713,14 @@ def create_app() -> FastAPI:
|
|||||||
infer_backend=payload.get("infer_backend", "huggingface"),
|
infer_backend=payload.get("infer_backend", "huggingface"),
|
||||||
infer_dtype=payload.get("infer_dtype", "auto"),
|
infer_dtype=payload.get("infer_dtype", "auto"),
|
||||||
)
|
)
|
||||||
if not result.get("loaded"):
|
|
||||||
raise HTTPException(status_code=500, detail=result.get("error", "model load failed"))
|
|
||||||
return result
|
return result
|
||||||
|
|
||||||
@app.post(f"{route_prefix}/inference/unload")
|
@app.post(f"{route_prefix}/inference/unload")
|
||||||
async def inference_unload() -> dict[str, Any]:
|
async def inference_unload() -> dict[str, Any]:
|
||||||
"""Unload the currently loaded model and free GPU memory."""
|
"""Unload the currently loaded model and free GPU memory."""
|
||||||
return get_inference_session().unload()
|
# Teardown (gc.collect + cuda.empty_cache) can take a while; run it off
|
||||||
|
# the event loop so /health and /inference/status stay responsive.
|
||||||
|
return await asyncio.to_thread(get_inference_session().unload)
|
||||||
|
|
||||||
@app.get(f"{route_prefix}/inference/status")
|
@app.get(f"{route_prefix}/inference/status")
|
||||||
async def inference_status() -> dict[str, Any]:
|
async def inference_status() -> dict[str, Any]:
|
||||||
@@ -715,7 +740,10 @@ def create_app() -> FastAPI:
|
|||||||
messages = payload.get("messages") or []
|
messages = payload.get("messages") or []
|
||||||
if not messages:
|
if not messages:
|
||||||
raise HTTPException(status_code=400, detail="messages is required")
|
raise HTTPException(status_code=400, detail="messages is required")
|
||||||
result = get_inference_session().chat(
|
# Generation is long-running; run it in a thread so the event loop keeps
|
||||||
|
# serving /inference/status and /health during inference.
|
||||||
|
result = await asyncio.to_thread(
|
||||||
|
get_inference_session().chat,
|
||||||
messages=messages,
|
messages=messages,
|
||||||
temperature=float(payload.get("temperature", 0.95)),
|
temperature=float(payload.get("temperature", 0.95)),
|
||||||
top_p=float(payload.get("top_p", 0.7)),
|
top_p=float(payload.get("top_p", 0.7)),
|
||||||
@@ -813,6 +841,22 @@ def create_app() -> FastAPI:
|
|||||||
"checksum_sha256": checksum,
|
"checksum_sha256": checksum,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@app.get(f"{route_prefix}/compute/files/read")
|
||||||
|
async def read_file(path: str = Query(...)) -> JSONResponse:
|
||||||
|
"""Read a text file from within YG_FT_DATA_ROOT. Used by the backend
|
||||||
|
to fetch eval results and other job outputs."""
|
||||||
|
data_root = Path(os.getenv("YG_FT_DATA_ROOT", "/data/yg-ft"))
|
||||||
|
target = (data_root / path.lstrip("/\\")).resolve()
|
||||||
|
if not _path_inside(data_root, target):
|
||||||
|
raise HTTPException(status_code=400, detail="path must stay inside YG_FT_DATA_ROOT")
|
||||||
|
if not target.is_file():
|
||||||
|
raise HTTPException(status_code=404, detail="file not found")
|
||||||
|
try:
|
||||||
|
content = target.read_text(encoding="utf-8")
|
||||||
|
return JSONResponse(json.loads(content) if content.strip().startswith("{") else {"content": content})
|
||||||
|
except Exception as exc:
|
||||||
|
raise HTTPException(status_code=500, detail=str(exc))
|
||||||
|
|
||||||
@app.get(f"{route_prefix}/compute/files/{{file_id}}/download")
|
@app.get(f"{route_prefix}/compute/files/{{file_id}}/download")
|
||||||
async def download_file(file_id: str) -> FileResponse:
|
async def download_file(file_id: str) -> FileResponse:
|
||||||
upload_root = Path(os.getenv("YG_FT_DATA_ROOT", "/data/yg-ft")) / "uploads"
|
upload_root = Path(os.getenv("YG_FT_DATA_ROOT", "/data/yg-ft")) / "uploads"
|
||||||
|
|||||||
@@ -204,6 +204,31 @@ def build_command(config: dict[str, Any], llama_factory_home: str = "/app/LLaMA-
|
|||||||
command.extend(["--quantization_bit", str(quantization_bit)])
|
command.extend(["--quantization_bit", str(quantization_bit)])
|
||||||
return LlamaFactoryCommand(command=command, work_dir=str(Path(llama_factory_home)), env={})
|
return LlamaFactoryCommand(command=command, work_dir=str(Path(llama_factory_home)), env={})
|
||||||
|
|
||||||
|
if engine == "eval":
|
||||||
|
output_dir = config.get("output_dir") or f"/data/yg-ft/outputs/{config.get('name', 'eval-job')}"
|
||||||
|
eval_config_path = str(Path(output_dir) / "eval_config.json")
|
||||||
|
eval_config = {
|
||||||
|
"model_name_or_path": config.get("model_name_or_path", ""),
|
||||||
|
"adapter_name_or_path": config.get("adapter_name_or_path", ""),
|
||||||
|
"template": config.get("template", "qwen"),
|
||||||
|
"dataset_path": config.get("dataset_path", ""),
|
||||||
|
"output_dir": output_dir,
|
||||||
|
"basic_metrics": config.get("basic_metrics", {}),
|
||||||
|
"dimension": config.get("dimension", {}),
|
||||||
|
"temperature": config.get("temperature", 0.1),
|
||||||
|
"top_p": config.get("top_p", 0.95),
|
||||||
|
"max_new_tokens": config.get("max_new_tokens", 512),
|
||||||
|
"infer_backend": config.get("infer_backend", "huggingface"),
|
||||||
|
"infer_dtype": config.get("infer_dtype", "auto"),
|
||||||
|
}
|
||||||
|
Path(output_dir).mkdir(parents=True, exist_ok=True)
|
||||||
|
Path(eval_config_path).write_text(json.dumps(eval_config, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||||
|
return LlamaFactoryCommand(
|
||||||
|
command=["python", "-u", "-m", "compute.engines.llama_factory.eval_runner", "--config", eval_config_path],
|
||||||
|
work_dir="/app",
|
||||||
|
env={},
|
||||||
|
)
|
||||||
|
|
||||||
errors = validate_config(config)
|
errors = validate_config(config)
|
||||||
if errors:
|
if errors:
|
||||||
raise ValueError("; ".join(errors))
|
raise ValueError("; ".join(errors))
|
||||||
|
|||||||
485
compute/engines/llama_factory/eval_runner.py
Normal file
485
compute/engines/llama_factory/eval_runner.py
Normal file
@@ -0,0 +1,485 @@
|
|||||||
|
"""
|
||||||
|
Evaluation runner — executes model evaluation as a subprocess job.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
python -m compute.engines.llama_factory.eval_runner --config <config_json_path>
|
||||||
|
|
||||||
|
The config JSON is written by the compute API before spawning this subprocess.
|
||||||
|
Results are written to ``output_dir/eval_results.json`` and progress is printed
|
||||||
|
to stdout (captured as job logs).
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import math
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
from difflib import SequenceMatcher
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
|
||||||
|
def _load_dataset(path: str) -> list[dict[str, Any]]:
|
||||||
|
"""Load a JSON or JSONL dataset file.
|
||||||
|
|
||||||
|
Supports common field names used across the platform:
|
||||||
|
* ``instruction`` + ``input`` + ``output`` (Alpaca-style)
|
||||||
|
* ``question`` + ``answer``
|
||||||
|
* ``messages`` (ShareGPT-style – the last assistant message is treated as reference)
|
||||||
|
"""
|
||||||
|
file_path = Path(path)
|
||||||
|
text = file_path.read_text(encoding="utf-8", errors="replace").strip()
|
||||||
|
if not text:
|
||||||
|
return []
|
||||||
|
if file_path.suffix.lower() == ".json":
|
||||||
|
value = json.loads(text)
|
||||||
|
if isinstance(value, list):
|
||||||
|
return [item for item in value if isinstance(item, dict)]
|
||||||
|
return [value] if isinstance(value, dict) else []
|
||||||
|
|
||||||
|
samples: list[dict[str, Any]] = []
|
||||||
|
for line in text.splitlines():
|
||||||
|
line = line.strip()
|
||||||
|
if not line:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
obj = json.loads(line)
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
continue
|
||||||
|
if isinstance(obj, dict):
|
||||||
|
samples.append(obj)
|
||||||
|
return samples
|
||||||
|
|
||||||
|
|
||||||
|
def _sample_question(sample: dict[str, Any]) -> str:
|
||||||
|
"""Extract the user-facing question / instruction from a sample."""
|
||||||
|
if sample.get("instruction"):
|
||||||
|
text = sample["instruction"]
|
||||||
|
if sample.get("input"):
|
||||||
|
text += "\n" + sample["input"]
|
||||||
|
return text
|
||||||
|
if sample.get("question"):
|
||||||
|
return sample["question"]
|
||||||
|
# ShareGPT-style: use the last user message as question
|
||||||
|
messages = sample.get("messages") or []
|
||||||
|
user_msgs = [m["content"] for m in messages if m.get("role") == "user"]
|
||||||
|
return user_msgs[-1] if user_msgs else ""
|
||||||
|
|
||||||
|
|
||||||
|
def _sample_reference(sample: dict[str, Any]) -> str:
|
||||||
|
"""Extract the reference answer from a sample."""
|
||||||
|
if sample.get("output"):
|
||||||
|
return sample["output"]
|
||||||
|
if sample.get("answer"):
|
||||||
|
return sample["answer"]
|
||||||
|
messages = sample.get("messages") or []
|
||||||
|
assistant_msgs = [m["content"] for m in messages if m.get("role") == "assistant"]
|
||||||
|
return assistant_msgs[-1] if assistant_msgs else ""
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Basic metrics
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def _compute_bleu(references: list[str], predictions: list[str], ngram: int = 4) -> dict[str, Any]:
|
||||||
|
"""Compute BLEU score via sacrebleu (corpus-level)."""
|
||||||
|
try:
|
||||||
|
from sacrebleu.metrics import BLEU
|
||||||
|
except ImportError:
|
||||||
|
return {"enabled": False, "error": "sacrebleu not installed", "score": 0}
|
||||||
|
bleu = BLEU(max_ngram_order=ngram)
|
||||||
|
# sacrebleu expects list-of-strings; we have one reference per prediction
|
||||||
|
score = bleu.corpus_score(predictions, [references])
|
||||||
|
return {
|
||||||
|
"enabled": True,
|
||||||
|
"score": round(score.score, 2),
|
||||||
|
"bleu": round(score.score, 2),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _compute_rouge(references: list[str], predictions: list[str], methods: list[str] | None = None) -> dict[str, Any]:
|
||||||
|
"""Compute ROUGE scores via rouge-score."""
|
||||||
|
try:
|
||||||
|
from rouge_score import rouge_scorer
|
||||||
|
except ImportError:
|
||||||
|
return {"enabled": False, "error": "rouge-score not installed", "score": 0}
|
||||||
|
methods = methods or ["rouge1", "rouge2", "rougeL"]
|
||||||
|
# Normalize: map "rouge_1"/"rouge1" → "rouge1", "rouge_l"/"rougeL" → "rougeL"
|
||||||
|
_rouge_aliases = {"rouge_1": "rouge1", "rouge_2": "rouge2", "rouge_l": "rougeL"}
|
||||||
|
methods = [_rouge_aliases.get(m, m.replace("_", "")) for m in methods]
|
||||||
|
scorer = rouge_scorer.RougeScorer(methods, use_stemmer=True)
|
||||||
|
totals: dict[str, float] = {}
|
||||||
|
n = max(len(predictions), 1)
|
||||||
|
for ref, pred in zip(references, predictions):
|
||||||
|
result = scorer.score(ref, pred)
|
||||||
|
for key in methods:
|
||||||
|
totals[key] = totals.get(key, 0) + result[key].fmeasure
|
||||||
|
avg = {k: round(v / n, 4) for k, v in totals.items()}
|
||||||
|
return {"enabled": True, "score": round(avg.get("rougeL", avg.get("rouge1", 0)) * 100, 2), **avg}
|
||||||
|
|
||||||
|
|
||||||
|
def _compute_cosine(references: list[str], predictions: list[str]) -> dict[str, Any]:
|
||||||
|
"""Compute average cosine similarity via sklearn."""
|
||||||
|
try:
|
||||||
|
from sklearn.feature_extraction.text import TfidfVectorizer
|
||||||
|
from sklearn.metrics.pairwise import cosine_similarity
|
||||||
|
except ImportError:
|
||||||
|
return {"enabled": False, "error": "scikit-learn not installed", "score": 0}
|
||||||
|
try:
|
||||||
|
vectorizer = TfidfVectorizer()
|
||||||
|
tfidf = vectorizer.fit_transform(references + predictions)
|
||||||
|
n = len(references)
|
||||||
|
ref_vec = tfidf[:n]
|
||||||
|
pred_vec = tfidf[n:]
|
||||||
|
sims = cosine_similarity(ref_vec, pred_vec).diagonal()
|
||||||
|
return {"enabled": True, "score": round(float(sims.mean()) * 100, 2)}
|
||||||
|
except ValueError:
|
||||||
|
return {"enabled": True, "score": 0, "error": "insufficient text for vectorization"}
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_text(value: str) -> str:
|
||||||
|
return re.sub(r"\s+", " ", str(value or "").strip().lower())
|
||||||
|
|
||||||
|
|
||||||
|
def _compute_exact_match(references: list[str], predictions: list[str]) -> dict[str, Any]:
|
||||||
|
total = len(predictions)
|
||||||
|
if not total:
|
||||||
|
return {"enabled": True, "score": 0, "matched": 0, "total": 0}
|
||||||
|
matched = sum(
|
||||||
|
1
|
||||||
|
for ref, pred in zip(references, predictions)
|
||||||
|
if _normalize_text(ref) == _normalize_text(pred)
|
||||||
|
)
|
||||||
|
return {"enabled": True, "score": round(matched / total * 100, 2), "matched": matched, "total": total}
|
||||||
|
|
||||||
|
|
||||||
|
def _compute_text_similarity(references: list[str], predictions: list[str]) -> dict[str, Any]:
|
||||||
|
if not predictions:
|
||||||
|
return {"enabled": True, "score": 0}
|
||||||
|
scores = [
|
||||||
|
SequenceMatcher(None, _normalize_text(ref), _normalize_text(pred)).ratio()
|
||||||
|
for ref, pred in zip(references, predictions)
|
||||||
|
]
|
||||||
|
return {"enabled": True, "score": round(sum(scores) / max(len(scores), 1) * 100, 2)}
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# LLM Judge
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def _judge_sample(
|
||||||
|
question: str,
|
||||||
|
reference: str,
|
||||||
|
prediction: str,
|
||||||
|
config: dict[str, Any],
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
"""Call an OpenAI-compatible LLM to judge a single sample.
|
||||||
|
|
||||||
|
Returns a dict with keys:
|
||||||
|
score, max_score, passed, judgement, evaluation_reason, error_type
|
||||||
|
"""
|
||||||
|
api_url = (config.get("api_url") or "").strip().rstrip("/")
|
||||||
|
api_key = (config.get("api_key") or "").strip()
|
||||||
|
eval_model = (config.get("eval_model") or "").strip()
|
||||||
|
# 优先使用模型记录里配置的真实 API 模型名(如 deepseek-chat),
|
||||||
|
# 否则回退到平台内部模型名
|
||||||
|
api_model = (config.get("api_model") or "").strip() or eval_model
|
||||||
|
eval_prompt = (config.get("eval_prompt") or "").strip()
|
||||||
|
score_min = float(config.get("score_min", 0))
|
||||||
|
score_max = float(config.get("score_max", 5))
|
||||||
|
pass_threshold = float(config.get("pass_threshold", 3))
|
||||||
|
|
||||||
|
if not api_url or not eval_model:
|
||||||
|
return {"score": 0, "max_score": score_max, "passed": False, "judgement": "未配置",
|
||||||
|
"evaluation_reason": "未配置评测模型", "error_type": "其他"}
|
||||||
|
|
||||||
|
system_msg = (
|
||||||
|
eval_prompt
|
||||||
|
or "你是一个专业的评测专家。请根据参考答-案对被测模型的输出进行评分。"
|
||||||
|
)
|
||||||
|
user_msg = (
|
||||||
|
f"## 问题\n{question}\n\n"
|
||||||
|
f"## 参考答案\n{reference}\n\n"
|
||||||
|
f"## 模型输出\n{prediction}\n\n"
|
||||||
|
f"请给出 {score_min}-{score_max} 分的评分,并说明理由。"
|
||||||
|
)
|
||||||
|
|
||||||
|
try:
|
||||||
|
import urllib.request
|
||||||
|
import urllib.error
|
||||||
|
|
||||||
|
body = json.dumps({
|
||||||
|
"model": api_model,
|
||||||
|
"messages": [
|
||||||
|
{"role": "system", "content": system_msg},
|
||||||
|
{"role": "user", "content": user_msg},
|
||||||
|
],
|
||||||
|
"temperature": 0.3,
|
||||||
|
"max_tokens": 512,
|
||||||
|
}).encode("utf-8")
|
||||||
|
|
||||||
|
req = urllib.request.Request(
|
||||||
|
f"{api_url}/v1/chat/completions",
|
||||||
|
data=body,
|
||||||
|
headers={
|
||||||
|
"Content-Type": "application/json",
|
||||||
|
"Authorization": f"Bearer {api_key}",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
resp = urllib.request.urlopen(req, timeout=120)
|
||||||
|
data = json.loads(resp.read().decode("utf-8"))
|
||||||
|
reply = data["choices"][0]["message"]["content"]
|
||||||
|
except Exception as exc:
|
||||||
|
return {"score": 0, "max_score": score_max, "passed": False,
|
||||||
|
"judgement": "错误", "evaluation_reason": f"评测模型调用失败: {exc}",
|
||||||
|
"error_type": "其他"}
|
||||||
|
|
||||||
|
# Parse score from reply — look for patterns like "4分" or "Score: 4"
|
||||||
|
score = 0
|
||||||
|
import re
|
||||||
|
score_patterns = [
|
||||||
|
r'(?:得分|分数|评分|score)[^\d]*(\d+(?:\.\d+)?)',
|
||||||
|
r'(\d+(?:\.\d+)?)\s*分',
|
||||||
|
r'(\d+(?:\.\d+)?)\s*/\s*\d+',
|
||||||
|
]
|
||||||
|
for pat in score_patterns:
|
||||||
|
m = re.search(pat, reply, re.IGNORECASE)
|
||||||
|
if m:
|
||||||
|
try:
|
||||||
|
score = float(m.group(1))
|
||||||
|
except ValueError:
|
||||||
|
continue
|
||||||
|
break
|
||||||
|
score = max(score_min, min(score_max, score))
|
||||||
|
passed = score >= pass_threshold
|
||||||
|
|
||||||
|
# Determine judgement label
|
||||||
|
if score >= pass_threshold + 1:
|
||||||
|
judgement = "正确"
|
||||||
|
elif score >= pass_threshold:
|
||||||
|
judgement = "部分正确"
|
||||||
|
else:
|
||||||
|
judgement = "错误"
|
||||||
|
|
||||||
|
# Guess error type from reply
|
||||||
|
reply_lower = reply.lower()
|
||||||
|
if any(w in reply_lower for w in ["幻觉", "hallucination", "编造"]):
|
||||||
|
error_type = "幻觉"
|
||||||
|
elif any(w in reply_lower for w in ["不完整", "incomplete", "遗漏"]):
|
||||||
|
error_type = "不完整"
|
||||||
|
elif any(w in reply_lower for w in ["格式", "format"]):
|
||||||
|
error_type = "格式偏差"
|
||||||
|
elif any(w in reply_lower for w in ["混淆", "confusion", "错误"]):
|
||||||
|
error_type = "混淆"
|
||||||
|
else:
|
||||||
|
error_type = "其他"
|
||||||
|
|
||||||
|
return {
|
||||||
|
"score": score,
|
||||||
|
"max_score": score_max,
|
||||||
|
"passed": passed,
|
||||||
|
"judgement": judgement,
|
||||||
|
"evaluation_reason": reply[:2000],
|
||||||
|
"error_type": error_type,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Main entry point
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def run_eval(config: dict[str, Any]) -> dict[str, Any]:
|
||||||
|
"""Execute a full evaluation run. Returns the result dict (also written to file)."""
|
||||||
|
model_path = config["model_name_or_path"]
|
||||||
|
adapter_path = config.get("adapter_name_or_path", "")
|
||||||
|
template = config.get("template", "qwen")
|
||||||
|
dataset_path = config["dataset_path"]
|
||||||
|
output_dir = Path(config["output_dir"])
|
||||||
|
output_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
basic_cfg = config.get("basic_metrics", {})
|
||||||
|
dimension_cfg = config.get("dimension", {}) or {}
|
||||||
|
output_precision = int(basic_cfg.get("output_precision", 2))
|
||||||
|
|
||||||
|
# ---- 1. Load dataset ----
|
||||||
|
print(f"[eval] loading dataset: {dataset_path}")
|
||||||
|
raw_samples = _load_dataset(dataset_path)
|
||||||
|
print(f"[eval] loaded {len(raw_samples)} samples")
|
||||||
|
|
||||||
|
# ---- 2. Load model ----
|
||||||
|
print(f"[eval] loading model: {model_path}")
|
||||||
|
from compute.engines.llama_factory.inference import InferenceSession
|
||||||
|
session = InferenceSession()
|
||||||
|
session.load(
|
||||||
|
model_name_or_path=model_path,
|
||||||
|
adapter_name_or_path=adapter_path,
|
||||||
|
template=template,
|
||||||
|
infer_backend=config.get("infer_backend", "huggingface"),
|
||||||
|
infer_dtype=config.get("infer_dtype", "auto"),
|
||||||
|
)
|
||||||
|
# load() 为异步加载(立即返回 loading),必须等待后台线程完成后再进行推理
|
||||||
|
load_result = session.wait_until_loaded(timeout=float(config.get("load_timeout", 1800)))
|
||||||
|
if not load_result.get("loaded"):
|
||||||
|
raise RuntimeError(f"model load failed: {load_result.get('error', 'unknown')}")
|
||||||
|
print(f"[eval] model loaded OK")
|
||||||
|
|
||||||
|
# ---- 3. Run inference on each sample ----
|
||||||
|
samples: list[dict[str, Any]] = []
|
||||||
|
predictions: list[str] = []
|
||||||
|
references: list[str] = []
|
||||||
|
questions: list[str] = []
|
||||||
|
|
||||||
|
total = len(raw_samples)
|
||||||
|
judge_enabled = bool(dimension_cfg.get("eval_model") and dimension_cfg.get("api_url"))
|
||||||
|
print(f"[eval] starting inference on {total} samples, judge={'enabled' if judge_enabled else 'disabled'}")
|
||||||
|
|
||||||
|
for idx, raw in enumerate(raw_samples, start=1):
|
||||||
|
question = _sample_question(raw)
|
||||||
|
reference = _sample_reference(raw)
|
||||||
|
if not question:
|
||||||
|
print(f"[eval] sample {idx}/{total}: skipped (no question)")
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Inference
|
||||||
|
chat_msgs = [{"role": "user", "content": question}]
|
||||||
|
result = session.chat(
|
||||||
|
chat_msgs,
|
||||||
|
temperature=float(config.get("temperature", 0.1)),
|
||||||
|
top_p=float(config.get("top_p", 0.95)),
|
||||||
|
max_new_tokens=int(config.get("max_new_tokens", 512)),
|
||||||
|
do_sample=False,
|
||||||
|
)
|
||||||
|
prediction = result.get("response", "") if not result.get("error") else f"[ERROR] {result['error']}"
|
||||||
|
|
||||||
|
predictions.append(prediction)
|
||||||
|
references.append(reference)
|
||||||
|
questions.append(question)
|
||||||
|
|
||||||
|
# LLM Judge
|
||||||
|
judge_result: dict[str, Any] = {}
|
||||||
|
if judge_enabled:
|
||||||
|
judge_result = _judge_sample(question, reference, prediction, dimension_cfg)
|
||||||
|
|
||||||
|
samples.append({
|
||||||
|
"index": idx,
|
||||||
|
"input": question,
|
||||||
|
"reference_answer": reference,
|
||||||
|
"model_output": prediction,
|
||||||
|
"score": judge_result.get("score"),
|
||||||
|
"max_score": judge_result.get("max_score", dimension_cfg.get("score_max", 5)),
|
||||||
|
"passed": judge_result.get("passed"),
|
||||||
|
"judgement": judge_result.get("judgement"),
|
||||||
|
"evaluation_reason": judge_result.get("evaluation_reason", ""),
|
||||||
|
"error_type": judge_result.get("error_type"),
|
||||||
|
"dimension_scores": [
|
||||||
|
{"name": "judge_score", "score": judge_result.get("score", 0),
|
||||||
|
"max_score": judge_result.get("max_score", dimension_cfg.get("score_max", 5))},
|
||||||
|
] if judge_result else [],
|
||||||
|
"status": "completed",
|
||||||
|
})
|
||||||
|
|
||||||
|
progress_pct = int(idx / max(total, 1) * 100)
|
||||||
|
print(f"[eval] sample {idx}/{total} ({progress_pct}%) done")
|
||||||
|
|
||||||
|
# ---- 4. Compute basic metrics ----
|
||||||
|
print(f"[eval] computing basic metrics on {len(predictions)} predictions")
|
||||||
|
metrics_result: dict[str, Any] = {}
|
||||||
|
|
||||||
|
bleu_cfg = basic_cfg.get("bleu", {})
|
||||||
|
if bleu_cfg.get("enabled"):
|
||||||
|
metrics_result["bleu"] = _compute_bleu(references, predictions, int(bleu_cfg.get("ngram", 4)))
|
||||||
|
|
||||||
|
rouge_cfg = basic_cfg.get("rouge", {})
|
||||||
|
if rouge_cfg.get("enabled"):
|
||||||
|
metrics_result["rouge"] = _compute_rouge(references, predictions, rouge_cfg.get("methods"))
|
||||||
|
|
||||||
|
cosine_cfg = basic_cfg.get("cosine", {})
|
||||||
|
if cosine_cfg.get("enabled"):
|
||||||
|
metrics_result["cosine"] = _compute_cosine(references, predictions)
|
||||||
|
metrics_result["exact_match"] = _compute_exact_match(references, predictions)
|
||||||
|
metrics_result["text_similarity"] = _compute_text_similarity(references, predictions)
|
||||||
|
|
||||||
|
# ---- 5. Summarise ----
|
||||||
|
completed = len(samples)
|
||||||
|
if judge_enabled:
|
||||||
|
scored = [s for s in samples if s.get("score") is not None]
|
||||||
|
passed_count = len([s for s in scored if s.get("passed")])
|
||||||
|
avg_score = round(sum(s["score"] for s in scored) / max(len(scored), 1), output_precision)
|
||||||
|
max_score = dimension_cfg.get("score_max", 5)
|
||||||
|
overall_score = round(avg_score / max_score * 100, output_precision)
|
||||||
|
overall_score_max = 100
|
||||||
|
dimension_summary = [{
|
||||||
|
"name": "综合评分",
|
||||||
|
"score": overall_score,
|
||||||
|
"max_score": 100,
|
||||||
|
"pass_rate": round(passed_count / max(completed, 1) * 100, 1),
|
||||||
|
}]
|
||||||
|
overall_evaluation = f"评测完成:{completed} 样本,{passed_count} 通过,平均 {avg_score}/{max_score} 分"
|
||||||
|
else:
|
||||||
|
passed_count = 0
|
||||||
|
enabled_scores = [
|
||||||
|
float(item.get("score") or 0)
|
||||||
|
for item in metrics_result.values()
|
||||||
|
if isinstance(item, dict) and item.get("enabled", True) and item.get("score") is not None
|
||||||
|
]
|
||||||
|
overall_score = round(sum(enabled_scores) / len(enabled_scores), output_precision) if enabled_scores else 0
|
||||||
|
overall_score_max = 100
|
||||||
|
dimension_summary = [
|
||||||
|
{
|
||||||
|
"name": name,
|
||||||
|
"score": float(item.get("score") or 0),
|
||||||
|
"max_score": 100,
|
||||||
|
"pass_rate": float(item.get("score") or 0),
|
||||||
|
}
|
||||||
|
for name, item in metrics_result.items()
|
||||||
|
if isinstance(item, dict) and item.get("enabled", True) and item.get("score") is not None
|
||||||
|
]
|
||||||
|
overall_evaluation = f"评测完成:{completed} 样本(未配置 LLM 评委)"
|
||||||
|
|
||||||
|
result = {
|
||||||
|
"overall_score": overall_score,
|
||||||
|
"overall_score_max": overall_score_max,
|
||||||
|
"overall_evaluation": overall_evaluation,
|
||||||
|
"improvement_suggestions": [],
|
||||||
|
"dimension_summary": dimension_summary,
|
||||||
|
"samples": samples,
|
||||||
|
"sample_count": total,
|
||||||
|
"completed_count": completed,
|
||||||
|
"passed_count": passed_count,
|
||||||
|
"basic_metrics": metrics_result,
|
||||||
|
}
|
||||||
|
|
||||||
|
# ---- 6. Write results ----
|
||||||
|
result_path = output_dir / "eval_results.json"
|
||||||
|
result_path.write_text(json.dumps(result, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||||
|
print(f"[eval] results written to {result_path}")
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
import argparse
|
||||||
|
parser = argparse.ArgumentParser(description="YG-FT Evaluation Runner")
|
||||||
|
parser.add_argument("--config", required=True, help="Path to eval config JSON file")
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
config_path = Path(args.config)
|
||||||
|
if not config_path.exists():
|
||||||
|
print(f"FATAL: config file not found: {args.config}", file=sys.stderr)
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
config = json.loads(config_path.read_text(encoding="utf-8"))
|
||||||
|
start = time.time()
|
||||||
|
try:
|
||||||
|
run_eval(config)
|
||||||
|
elapsed = time.time() - start
|
||||||
|
print(f"[eval] DONE in {elapsed:.1f}s")
|
||||||
|
except Exception as exc:
|
||||||
|
print(f"[eval] FAILED: {exc}", file=sys.stderr)
|
||||||
|
import traceback
|
||||||
|
traceback.print_exc()
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -2,123 +2,270 @@ from __future__ import annotations
|
|||||||
|
|
||||||
import threading
|
import threading
|
||||||
import time
|
import time
|
||||||
from typing import Any
|
import uuid
|
||||||
|
from typing import Any, Iterator
|
||||||
|
|
||||||
|
|
||||||
class InferenceSession:
|
class InferenceSession:
|
||||||
"""Manages a loaded model for inference with LLaMA-Factory ChatModel."""
|
"""Manages a loaded model for inference with LLaMA-Factory ChatModel.
|
||||||
|
|
||||||
|
Model loading is asynchronous: ``load()`` spawns a background daemon thread
|
||||||
|
and returns immediately with ``status == "loading"``. ``info()`` (served by
|
||||||
|
``/inference/status``) is always responsive, so the platform backend can
|
||||||
|
poll loading progress without being blocked by a minutes-long model load —
|
||||||
|
which previously froze the whole compute node event loop.
|
||||||
|
|
||||||
|
State machine: idle -> loading -> ready | error, ready -> idle (unload),
|
||||||
|
loading -> idle (cancelled). Long operations (ChatModel build, teardown,
|
||||||
|
generation) never run while holding ``_state_lock``; they either run in the
|
||||||
|
worker thread or under ``_chat_lock`` only.
|
||||||
|
"""
|
||||||
|
|
||||||
def __init__(self) -> None:
|
def __init__(self) -> None:
|
||||||
|
self._state_lock = threading.Lock() # brief state transitions only
|
||||||
|
self._chat_lock = threading.Lock() # serialize chat/teardown
|
||||||
|
self._status: str = "idle"
|
||||||
|
self._error: str = ""
|
||||||
|
self._request_id: str = ""
|
||||||
|
self._load_args: dict[str, Any] = {}
|
||||||
|
self._teardown_old = False # load-while-ready: unload old before loading new
|
||||||
|
self._cancel_requested = False # unload-while-loading: tear down after load finishes
|
||||||
|
self._load_thread: threading.Thread | None = None
|
||||||
self._model: Any = None
|
self._model: Any = None
|
||||||
self._tokenizer: Any = None
|
self._tokenizer: Any = None
|
||||||
self._generating_args: dict[str, Any] = {}
|
self._generating_args: dict[str, Any] = {}
|
||||||
self._model_name: str = ""
|
self._model_name: str = ""
|
||||||
self._adapter_path: str = ""
|
self._adapter_path: str = ""
|
||||||
self._lock = threading.Lock()
|
|
||||||
self._loaded_at: float = 0.0
|
self._loaded_at: float = 0.0
|
||||||
self._status: str = "idle"
|
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def status(self) -> str:
|
def status(self) -> str:
|
||||||
|
with self._state_lock:
|
||||||
return self._status
|
return self._status
|
||||||
|
|
||||||
@property
|
|
||||||
def model_name(self) -> str:
|
|
||||||
return self._model_name
|
|
||||||
|
|
||||||
@property
|
|
||||||
def adapter_path(self) -> str:
|
|
||||||
return self._adapter_path
|
|
||||||
|
|
||||||
@property
|
|
||||||
def loaded_at(self) -> float:
|
|
||||||
return self._loaded_at
|
|
||||||
|
|
||||||
def info(self) -> dict[str, Any]:
|
def info(self) -> dict[str, Any]:
|
||||||
|
with self._state_lock:
|
||||||
return {
|
return {
|
||||||
"loaded": self._status == "ready",
|
"loaded": self._status == "ready",
|
||||||
"status": self._status,
|
"status": self._status,
|
||||||
"model_name": self._model_name,
|
"model_name": self._model_name,
|
||||||
"adapter_path": self._adapter_path,
|
"adapter_path": self._adapter_path,
|
||||||
"loaded_at": self._loaded_at,
|
"loaded_at": self._loaded_at,
|
||||||
|
"request_id": self._request_id,
|
||||||
|
"error": self._error,
|
||||||
}
|
}
|
||||||
|
|
||||||
def load(self, model_name_or_path, adapter_name_or_path="", template="qwen", infer_backend="huggingface", infer_dtype="auto", **kwargs):
|
def wait_until_loaded(self, timeout: float | None = None) -> dict[str, Any]:
|
||||||
with self._lock:
|
"""Wait for an in-flight async load to finish and return its outcome.
|
||||||
|
|
||||||
|
供同步消费方(如 eval_runner 子进程)使用:``load()`` 立即返回 loading 后,
|
||||||
|
调用本方法等待后台加载线程完成,拿到最终的 loaded/error 结果。
|
||||||
|
若在 timeout 秒内仍未加载完成,返回 ``status == "loading"`` 并附上超时提示。
|
||||||
|
"""
|
||||||
|
with self._state_lock:
|
||||||
|
thread = self._load_thread
|
||||||
|
if thread is not None and thread.is_alive():
|
||||||
|
thread.join(timeout=timeout)
|
||||||
|
with self._state_lock:
|
||||||
|
loaded = self._status == "ready"
|
||||||
|
status = self._status
|
||||||
|
error = self._error
|
||||||
|
if not loaded and status == "loading":
|
||||||
|
error = error or f"model load timed out after {timeout or 'N/A'}s"
|
||||||
|
return {
|
||||||
|
"loaded": loaded,
|
||||||
|
"status": status,
|
||||||
|
"model_name": self._model_name,
|
||||||
|
"adapter_path": self._adapter_path,
|
||||||
|
"error": error,
|
||||||
|
}
|
||||||
|
|
||||||
|
def load(
|
||||||
|
self,
|
||||||
|
model_name_or_path,
|
||||||
|
adapter_name_or_path="",
|
||||||
|
template="qwen",
|
||||||
|
infer_backend="huggingface",
|
||||||
|
infer_dtype="auto",
|
||||||
|
**kwargs,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
with self._state_lock:
|
||||||
if self._status == "loading":
|
if self._status == "loading":
|
||||||
return {"loaded": False, "error": "model is already loading"}
|
# A model is already loading — dedupe, reuse the same request id.
|
||||||
if self._status == "ready":
|
return {"loaded": False, "status": "loading", "request_id": self._request_id}
|
||||||
self.unload()
|
self._teardown_old = self._status == "ready"
|
||||||
self._status = "loading"
|
self._status = "loading"
|
||||||
|
self._error = ""
|
||||||
|
self._request_id = uuid.uuid4().hex[:12]
|
||||||
|
self._cancel_requested = False
|
||||||
|
self._load_args = {
|
||||||
|
"model_name_or_path": model_name_or_path,
|
||||||
|
"template": template,
|
||||||
|
"infer_backend": infer_backend,
|
||||||
|
"infer_dtype": infer_dtype,
|
||||||
|
}
|
||||||
|
if adapter_name_or_path:
|
||||||
|
self._load_args["adapter_name_or_path"] = adapter_name_or_path
|
||||||
|
self._load_args.update(kwargs)
|
||||||
self._model_name = model_name_or_path
|
self._model_name = model_name_or_path
|
||||||
self._adapter_path = adapter_name_or_path
|
self._adapter_path = adapter_name_or_path
|
||||||
|
self._load_thread = threading.Thread(target=self._load_worker, daemon=True)
|
||||||
|
self._load_thread.start()
|
||||||
|
return {"loaded": False, "status": "loading", "request_id": self._request_id}
|
||||||
|
|
||||||
|
def _load_worker(self) -> None:
|
||||||
|
"""Build the ChatModel off the state lock so info() never blocks."""
|
||||||
|
model = None
|
||||||
|
tokenizer = None
|
||||||
|
generating_args: dict[str, Any] = {}
|
||||||
|
error = ""
|
||||||
try:
|
try:
|
||||||
|
if self._teardown_old:
|
||||||
|
self._release_model()
|
||||||
from llamafactory.chat import ChatModel
|
from llamafactory.chat import ChatModel
|
||||||
from llamafactory.hparams import get_infer_args
|
from llamafactory.hparams import get_infer_args
|
||||||
args = {"model_name_or_path": model_name_or_path, "template": template, "infer_backend": infer_backend, "infer_dtype": infer_dtype}
|
|
||||||
if adapter_name_or_path:
|
args = dict(self._load_args)
|
||||||
args["adapter_name_or_path"] = adapter_name_or_path
|
infer_result = get_infer_args(args)
|
||||||
args.update(kwargs)
|
model = ChatModel(args)
|
||||||
model_args, generating_args = get_infer_args(args)
|
tokenizer = getattr(model, "tokenizer", None) or model.engine.tokenizer
|
||||||
self._model = ChatModel(model_args)
|
generating_args = infer_result[-1]
|
||||||
self._tokenizer = self._model.tokenizer
|
if hasattr(generating_args, "__dataclass_fields__"):
|
||||||
|
generating_args = {
|
||||||
|
k: v for k, v in vars(generating_args).items() if not k.startswith("_")
|
||||||
|
}
|
||||||
|
else:
|
||||||
|
generating_args = dict(generating_args)
|
||||||
|
except Exception as exc: # noqa: BLE001 - surface load failure via status
|
||||||
|
error = str(exc)
|
||||||
|
with self._state_lock:
|
||||||
|
if error:
|
||||||
|
self._model = None
|
||||||
|
self._tokenizer = None
|
||||||
|
self._status = "error"
|
||||||
|
self._error = error
|
||||||
|
return
|
||||||
|
if self._cancel_requested:
|
||||||
|
# Unload was requested while loading — drop the fresh model.
|
||||||
|
model = None
|
||||||
|
tokenizer = None
|
||||||
|
self._model = None
|
||||||
|
self._tokenizer = None
|
||||||
|
self._status = "idle"
|
||||||
|
return
|
||||||
|
self._model = model
|
||||||
|
self._tokenizer = tokenizer
|
||||||
self._generating_args = generating_args
|
self._generating_args = generating_args
|
||||||
self._loaded_at = time.time()
|
self._loaded_at = time.time()
|
||||||
self._status = "ready"
|
self._status = "ready"
|
||||||
return {"loaded": True, "status": "ready"}
|
|
||||||
except Exception as exc:
|
|
||||||
self._status = "error"
|
|
||||||
self._model = None
|
|
||||||
return {"loaded": False, "status": "error", "error": str(exc)}
|
|
||||||
|
|
||||||
def unload(self):
|
def _release_model(self) -> None:
|
||||||
with self._lock:
|
with self._chat_lock:
|
||||||
if self._model is not None:
|
with self._state_lock:
|
||||||
|
self._status = "unloading"
|
||||||
|
model = self._model
|
||||||
|
self._model = None
|
||||||
|
self._tokenizer = None
|
||||||
|
if model is not None:
|
||||||
try:
|
try:
|
||||||
del self._model
|
del model
|
||||||
except Exception:
|
except Exception: # noqa: BLE001 - best-effort teardown
|
||||||
pass
|
pass
|
||||||
|
# 强制释放 PyTorch CUDA 缓存,真正归还 GPU 显存
|
||||||
|
try:
|
||||||
|
import gc
|
||||||
|
|
||||||
|
gc.collect()
|
||||||
|
import torch
|
||||||
|
|
||||||
|
if torch.cuda.is_available():
|
||||||
|
torch.cuda.empty_cache()
|
||||||
|
torch.cuda.synchronize()
|
||||||
|
except Exception: # noqa: BLE001 - teardown must not raise
|
||||||
|
pass
|
||||||
|
with self._state_lock:
|
||||||
|
self._status = "idle"
|
||||||
|
self._model_name = ""
|
||||||
|
self._adapter_path = ""
|
||||||
|
self._loaded_at = 0.0
|
||||||
|
self._error = ""
|
||||||
|
|
||||||
|
def unload(self) -> dict[str, Any]:
|
||||||
|
with self._state_lock:
|
||||||
|
if self._status == "loading":
|
||||||
|
# Ask the worker to tear down right after the load finishes.
|
||||||
|
self._cancel_requested = True
|
||||||
|
return {"unloaded": False, "status": "cancelling", "request_id": self._request_id}
|
||||||
|
was_ready = self._status == "ready"
|
||||||
|
if was_ready:
|
||||||
|
self._release_model()
|
||||||
|
else:
|
||||||
|
with self._state_lock:
|
||||||
self._model = None
|
self._model = None
|
||||||
self._tokenizer = None
|
self._tokenizer = None
|
||||||
self._status = "idle"
|
self._status = "idle"
|
||||||
self._model_name = ""
|
self._model_name = ""
|
||||||
self._adapter_path = ""
|
self._adapter_path = ""
|
||||||
self._loaded_at = 0.0
|
self._loaded_at = 0.0
|
||||||
return {"unloaded": True}
|
self._error = ""
|
||||||
|
return {"unloaded": True, "status": "idle"}
|
||||||
|
|
||||||
def chat(self, messages, temperature=0.95, top_p=0.7, max_new_tokens=1024, do_sample=True, **kwargs):
|
def chat(self, messages, temperature=0.95, top_p=0.7, max_new_tokens=1024, do_sample=True, **kwargs) -> dict[str, Any]:
|
||||||
with self._lock:
|
with self._chat_lock:
|
||||||
|
with self._state_lock:
|
||||||
|
if self._status == "loading":
|
||||||
|
return {
|
||||||
|
"error": f"model is still loading (request_id={self._request_id}); please retry",
|
||||||
|
"response": "",
|
||||||
|
}
|
||||||
|
if self._status == "error":
|
||||||
|
return {"error": f"model load failed: {self._error}", "response": ""}
|
||||||
if self._status != "ready" or self._model is None:
|
if self._status != "ready" or self._model is None:
|
||||||
return {"error": "model not loaded", "response": ""}
|
return {"error": "model not loaded", "response": ""}
|
||||||
try:
|
try:
|
||||||
generate_kwargs = {**self._generating_args, "temperature": temperature, "top_p": top_p, "max_new_tokens": max_new_tokens, "do_sample": do_sample}
|
generate_kwargs = {
|
||||||
|
"temperature": temperature,
|
||||||
|
"top_p": top_p,
|
||||||
|
"max_new_tokens": max_new_tokens,
|
||||||
|
"do_sample": do_sample,
|
||||||
|
}
|
||||||
generate_kwargs.update(kwargs)
|
generate_kwargs.update(kwargs)
|
||||||
formatted = self._model.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
system = next((m["content"] for m in messages if m["role"] == "system"), None)
|
||||||
|
user_messages = [m for m in messages if m["role"] != "system"]
|
||||||
responses = []
|
responses = []
|
||||||
for response in self._model.stream_chat(formatted, generate_kwargs):
|
for response in self._model.stream_chat(user_messages, system=system, **generate_kwargs):
|
||||||
responses.append(response)
|
responses.append(response)
|
||||||
full_response = "".join(str(r) for r in responses)
|
full_response = "".join(str(r) for r in responses)
|
||||||
return {"response": full_response}
|
return {"response": full_response}
|
||||||
except Exception as exc:
|
except Exception as exc: # noqa: BLE001 - return generation error to caller
|
||||||
return {"error": str(exc), "response": ""}
|
return {"error": str(exc), "response": ""}
|
||||||
|
|
||||||
def chat_stream(self, messages, **kwargs):
|
def chat_stream(self, messages, **kwargs) -> Iterator[str]:
|
||||||
with self._lock:
|
with self._chat_lock:
|
||||||
|
with self._state_lock:
|
||||||
|
if self._status == "loading":
|
||||||
|
yield 'data: {"error": "model is still loading; please retry"}\n\n'
|
||||||
|
return
|
||||||
|
if self._status == "error":
|
||||||
|
yield 'data: {"error": "model load failed: ' + str(self._error) + '"}\n\n'
|
||||||
|
return
|
||||||
if self._status != "ready" or self._model is None:
|
if self._status != "ready" or self._model is None:
|
||||||
yield 'data: {"error": "model not loaded"}\n\n'
|
yield 'data: {"error": "model not loaded"}\n\n'
|
||||||
return
|
return
|
||||||
try:
|
try:
|
||||||
generate_kwargs = {**self._generating_args, **kwargs}
|
generate_kwargs = {**kwargs}
|
||||||
formatted = self._model.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
system = next((m["content"] for m in messages if m["role"] == "system"), None)
|
||||||
for new_text in self._model.stream_chat(formatted, generate_kwargs):
|
user_messages = [m for m in messages if m["role"] != "system"]
|
||||||
|
for new_text in self._model.stream_chat(user_messages, system=system, **generate_kwargs):
|
||||||
yield new_text
|
yield new_text
|
||||||
except Exception as exc:
|
except Exception as exc: # noqa: BLE001 - stream error as SSE event
|
||||||
yield 'data: {"error": "' + str(exc) + '"}\n\n'
|
yield 'data: {"error": "' + str(exc) + '"}\n\n'
|
||||||
|
|
||||||
|
|
||||||
_inference_session = None
|
_inference_session = None
|
||||||
|
|
||||||
def get_inference_session():
|
|
||||||
|
def get_inference_session() -> InferenceSession:
|
||||||
global _inference_session
|
global _inference_session
|
||||||
if _inference_session is None:
|
if _inference_session is None:
|
||||||
_inference_session = InferenceSession()
|
_inference_session = InferenceSession()
|
||||||
|
|||||||
@@ -4,4 +4,9 @@ python-multipart>=0.0.9
|
|||||||
pydantic>=2.7.0
|
pydantic>=2.7.0
|
||||||
python-dotenv>=1.0.1
|
python-dotenv>=1.0.1
|
||||||
httpx>=0.27.0
|
httpx>=0.27.0
|
||||||
|
# 模型评测指标
|
||||||
|
sacrebleu>=2.4.0
|
||||||
|
rouge-score>=0.1.2
|
||||||
|
scikit-learn>=1.3.0
|
||||||
|
# LLaMA-Factory 训练引擎
|
||||||
llamafactory
|
llamafactory
|
||||||
146
compute/tests/test_inference_session.py
Normal file
146
compute/tests/test_inference_session.py
Normal file
@@ -0,0 +1,146 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
import types
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from compute.engines.llama_factory.inference import InferenceSession
|
||||||
|
|
||||||
|
# 模拟模型加载耗时,用于验证 load() 立即返回、info() 不阻塞
|
||||||
|
LOAD_DELAY = 0.2
|
||||||
|
|
||||||
|
|
||||||
|
class FakeChatModel:
|
||||||
|
def __init__(self, args: dict[str, Any]) -> None:
|
||||||
|
time.sleep(LOAD_DELAY)
|
||||||
|
self.tokenizer = object()
|
||||||
|
self.engine = types.SimpleNamespace(tokenizer=object())
|
||||||
|
self._output = "hello from model"
|
||||||
|
|
||||||
|
def stream_chat(self, *args, **kwargs):
|
||||||
|
for _ in range(1):
|
||||||
|
yield self._output
|
||||||
|
|
||||||
|
|
||||||
|
class FailingChatModel:
|
||||||
|
def __init__(self, args: dict[str, Any]) -> None:
|
||||||
|
time.sleep(LOAD_DELAY)
|
||||||
|
raise RuntimeError("boom: fake load failure")
|
||||||
|
|
||||||
|
|
||||||
|
def _get_infer_args(args: dict[str, Any]) -> list[Any]:
|
||||||
|
# 最后一个元素为 generating_args,worker 会转成 dict
|
||||||
|
return [None, None, {"temperature": 0.7}]
|
||||||
|
|
||||||
|
|
||||||
|
def _install_llamafactory(monkeypatch, chat_model: type) -> None:
|
||||||
|
llmf = types.ModuleType("llamafactory")
|
||||||
|
chat_mod = types.ModuleType("llamafactory.chat")
|
||||||
|
hparams_mod = types.ModuleType("llamafactory.hparams")
|
||||||
|
chat_mod.ChatModel = chat_model
|
||||||
|
hparams_mod.get_infer_args = _get_infer_args
|
||||||
|
llmf.chat = chat_mod
|
||||||
|
llmf.hparams = hparams_mod
|
||||||
|
monkeypatch.setitem(sys.modules, "llamafactory", llmf)
|
||||||
|
monkeypatch.setitem(sys.modules, "llamafactory.chat", chat_mod)
|
||||||
|
monkeypatch.setitem(sys.modules, "llamafactory.hparams", hparams_mod)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def stub_llamafactory(monkeypatch) -> None:
|
||||||
|
_install_llamafactory(monkeypatch, FakeChatModel)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def stub_failing_llamafactory(monkeypatch) -> None:
|
||||||
|
_install_llamafactory(monkeypatch, FailingChatModel)
|
||||||
|
|
||||||
|
|
||||||
|
def _wait_for_status(session: InferenceSession, status: str, timeout: float = 3.0) -> bool:
|
||||||
|
deadline = time.time() + timeout
|
||||||
|
while time.time() < deadline:
|
||||||
|
if session.info()["status"] == status:
|
||||||
|
return True
|
||||||
|
time.sleep(0.02)
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def test_load_returns_immediately_then_ready(stub_llamafactory) -> None:
|
||||||
|
session = InferenceSession()
|
||||||
|
started = time.time()
|
||||||
|
result = session.load("/models/qwen")
|
||||||
|
assert result["status"] == "loading"
|
||||||
|
assert result["loaded"] is False
|
||||||
|
assert result["request_id"]
|
||||||
|
# 在慢加载完成前就返回,且 info() 加载期间可响应
|
||||||
|
assert time.time() - started < LOAD_DELAY
|
||||||
|
assert session.info()["status"] == "loading"
|
||||||
|
assert _wait_for_status(session, "ready")
|
||||||
|
info = session.info()
|
||||||
|
assert info["loaded"] is True
|
||||||
|
assert info["status"] == "ready"
|
||||||
|
assert info["model_name"] == "/models/qwen"
|
||||||
|
|
||||||
|
|
||||||
|
def test_second_load_while_loading_deduped(stub_llamafactory) -> None:
|
||||||
|
session = InferenceSession()
|
||||||
|
r1 = session.load("/models/a")
|
||||||
|
r2 = session.load("/models/b")
|
||||||
|
assert r2["status"] == "loading"
|
||||||
|
assert r2["request_id"] == r1["request_id"]
|
||||||
|
assert _wait_for_status(session, "ready")
|
||||||
|
assert session.info()["status"] == "ready"
|
||||||
|
|
||||||
|
|
||||||
|
def test_load_error_surfaces_in_status(stub_failing_llamafactory) -> None:
|
||||||
|
session = InferenceSession()
|
||||||
|
session.load("/models/bad")
|
||||||
|
assert _wait_for_status(session, "error")
|
||||||
|
assert "boom" in session.info()["error"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_unload_while_loading_cancels(stub_llamafactory) -> None:
|
||||||
|
session = InferenceSession()
|
||||||
|
session.load("/models/qwen")
|
||||||
|
result = session.unload()
|
||||||
|
assert result["status"] == "cancelling"
|
||||||
|
assert _wait_for_status(session, "idle")
|
||||||
|
|
||||||
|
|
||||||
|
def test_chat_while_loading_returns_loading_error(stub_llamafactory) -> None:
|
||||||
|
session = InferenceSession()
|
||||||
|
session.load("/models/qwen")
|
||||||
|
out = session.chat([{"role": "user", "content": "hi"}])
|
||||||
|
assert "still loading" in (out.get("error") or "")
|
||||||
|
assert _wait_for_status(session, "ready")
|
||||||
|
out = session.chat([{"role": "user", "content": "hi"}])
|
||||||
|
assert out.get("response") == "hello from model"
|
||||||
|
|
||||||
|
|
||||||
|
def test_chat_stream_while_loading_yields_error(stub_llamafactory) -> None:
|
||||||
|
session = InferenceSession()
|
||||||
|
session.load("/models/qwen")
|
||||||
|
chunks = list(session.chat_stream([{"role": "user", "content": "hi"}]))
|
||||||
|
assert any("still loading" in c for c in chunks)
|
||||||
|
|
||||||
|
|
||||||
|
def test_wait_until_loaded_blocks_until_ready(stub_llamafactory) -> None:
|
||||||
|
session = InferenceSession()
|
||||||
|
result = session.load("/models/qwen")
|
||||||
|
assert result["status"] == "loading"
|
||||||
|
# 同步等待后台加载线程完成
|
||||||
|
outcome = session.wait_until_loaded(timeout=3.0)
|
||||||
|
assert outcome["loaded"] is True
|
||||||
|
assert outcome["status"] == "ready"
|
||||||
|
|
||||||
|
|
||||||
|
def test_wait_until_loaded_reports_load_error(stub_failing_llamafactory) -> None:
|
||||||
|
session = InferenceSession()
|
||||||
|
session.load("/models/bad")
|
||||||
|
outcome = session.wait_until_loaded(timeout=3.0)
|
||||||
|
assert outcome["loaded"] is False
|
||||||
|
assert outcome["status"] == "error"
|
||||||
|
assert "boom" in outcome["error"]
|
||||||
@@ -11,7 +11,7 @@ RUN pip install --upgrade pip -i https://pypi.tuna.tsinghua.edu.cn/simple \
|
|||||||
&& pip install -r /tmp/requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple \
|
&& pip install -r /tmp/requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple \
|
||||||
&& rm -f /tmp/requirements.txt
|
&& rm -f /tmp/requirements.txt
|
||||||
|
|
||||||
RUN python -c "import fastapi, uvicorn, psycopg, sqlalchemy, redis, jwt, passlib, httpx, alembic; print('backend dependency check ok')"
|
RUN python -c "import fastapi, uvicorn, psycopg, psycopg_pool, sqlalchemy, redis, jwt, passlib, httpx, alembic; print('backend dependency check ok')"
|
||||||
|
|
||||||
RUN mkdir -p /opt/yg-ft/logs/backend /data/yg-ft \
|
RUN mkdir -p /opt/yg-ft/logs/backend /data/yg-ft \
|
||||||
&& chmod -R 0775 /opt/yg-ft /data/yg-ft
|
&& chmod -R 0775 /opt/yg-ft /data/yg-ft
|
||||||
|
|||||||
@@ -14,8 +14,8 @@ server {
|
|||||||
proxy_set_header X-Real-IP $remote_addr;
|
proxy_set_header X-Real-IP $remote_addr;
|
||||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||||
proxy_set_header X-Forwarded-Proto $scheme;
|
proxy_set_header X-Forwarded-Proto $scheme;
|
||||||
proxy_read_timeout 300s;
|
proxy_read_timeout 900s;
|
||||||
proxy_send_timeout 300s;
|
proxy_send_timeout 900s;
|
||||||
}
|
}
|
||||||
|
|
||||||
location = /modelTF {
|
location = /modelTF {
|
||||||
@@ -25,8 +25,8 @@ server {
|
|||||||
proxy_set_header X-Real-IP $remote_addr;
|
proxy_set_header X-Real-IP $remote_addr;
|
||||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||||
proxy_set_header X-Forwarded-Proto $scheme;
|
proxy_set_header X-Forwarded-Proto $scheme;
|
||||||
proxy_read_timeout 300s;
|
proxy_read_timeout 900s;
|
||||||
proxy_send_timeout 300s;
|
proxy_send_timeout 900s;
|
||||||
}
|
}
|
||||||
|
|
||||||
location ~* \.(?:js|css|png|jpg|jpeg|gif|ico|svg|woff|woff2|ttf)$ {
|
location ~* \.(?:js|css|png|jpg|jpeg|gif|ico|svg|woff|woff2|ttf)$ {
|
||||||
|
|||||||
121
docs/模型评测功能总结.md
Normal file
121
docs/模型评测功能总结.md
Normal file
@@ -0,0 +1,121 @@
|
|||||||
|
# 模型评测功能总结
|
||||||
|
|
||||||
|
本项目(基于 LLaMA-Factory 的微调训练平台)包含 **4 套相对独立** 的模型评测能力,分别面向不同的使用场景:
|
||||||
|
|
||||||
|
| 能力 | 入口/目录 | 评测类型 | 打分方式 |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| 1. 学术 Benchmark 评测 | `llamafactory/eval/` | 选择题式基准(类 MMLU/C-Eval) | 选项匹配 + few-shot |
|
||||||
|
| 2. 评估工作台 | `backend/app/api/v1/eval/` | 生成式问答(指令跟随) | BLEU / ROUGE / ExactMatch + 可选 LLM 评审 |
|
||||||
|
| 3. 平台评估系统 | `backend/app/api/v1/evaluation/` | 基于评估数据集的问答 | 判卷模型(judge model)打分(0–5 分) |
|
||||||
|
| 4. 训练时验证评估 | `backend/app/services/task_runner.py` | 训练验证集 | loss 指标 |
|
||||||
|
|
||||||
|
下面分别说明。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. 学术 Benchmark 评测(LLaMA-Factory 原生)
|
||||||
|
|
||||||
|
面向标准学术选择题基准(如 MMLU、C-Eval 等),复用 LLaMA-Factory 原生的评测框架。
|
||||||
|
|
||||||
|
**核心文件**
|
||||||
|
- `llamafactory/eval/evaluator.py`:`Evaluator` 类 + `run_eval()` 入口
|
||||||
|
- `llamafactory/eval/template.py`:评测 prompt 模板(中/英,含 few-shot 示例构建)
|
||||||
|
- `llamafactory/hparams/evaluation_args.py`:`EvaluationArguments` 配置类
|
||||||
|
|
||||||
|
**工作流程**
|
||||||
|
1. 按 `task`(benchmark 名称)加载数据集,按科目(subject)拆分。
|
||||||
|
2. 每个样本构造 few-shot 提示词(`n_shot` 控制示例数,由 `lang` 决定中/英模板),将题干与候选选项拼入 prompt。
|
||||||
|
3. 调用模型推理得到预测,与标准答案比对,统计每个科目及整体的 `accuracy`。
|
||||||
|
4. 结果写入 `save_dir`,打印各科目与平均准确率。
|
||||||
|
|
||||||
|
**关键参数(`EvaluationArguments`)**
|
||||||
|
- `task`:基准数据集名
|
||||||
|
- `batch_size` / `n_shot` / `lang` / `save_dir` / `seed`
|
||||||
|
- `model_name_or_path`、`template`、`trust_remote_code` 等模型相关参数
|
||||||
|
|
||||||
|
> 该能力属于框架底层,本平台前端未直接提供操作入口,主要通过配置文件/脚本调用。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. 评估工作台(生成式评测 + 指标计算)
|
||||||
|
|
||||||
|
后端路由位于 `backend/app/api/v1/eval/__init__.py`,前端称为「评估工作台」。**适用于评测模型的指令跟随与生成质量**,并支持 LLM 作为裁判(LLM-as-a-Judge)。
|
||||||
|
|
||||||
|
**API 端点**
|
||||||
|
- `GET /evaluation/tasks`:列出评测任务(`frontend/src/api/evaluation.ts:listTasks`)
|
||||||
|
- `POST /evaluation/run`:提交一次评测(`runEval`)
|
||||||
|
- `GET /evaluation/report/{task_id}`:拉取评测报告(`getReport`)
|
||||||
|
- `DELETE /evaluation/tasks/{task_id}`:删除任务(`deleteTask`)
|
||||||
|
|
||||||
|
**评测流程(`run_eval`)**
|
||||||
|
1. 通过 **LLaMA-Factory 数据管道**(`get_dataset`) 加载数据集,支持 `subset` 与抽样(`eval_sample`)。
|
||||||
|
2. 用 **原生 transformers** 加载模型在本地做生成推理(单进程顺序生成,便于展示样本)。
|
||||||
|
3. 计算客观指标(`compute_score`):
|
||||||
|
- `BLEU`(sacrebleu)
|
||||||
|
- `ROUGE-1 / ROUGE-2 / ROUGE-L`(rouge-score)
|
||||||
|
- `Exact Match`
|
||||||
|
4. **可选 LLM 评审**(judge):当配置了 `judge_model` / `judge_api_base` / `judge_api_key` 时,调用 OpenAI 兼容接口对每条样本打分(10 分制),并输出 4 个维度与理由:
|
||||||
|
- 核心事实正确性 `factual`
|
||||||
|
- 信息完整性 `completeness`
|
||||||
|
- 无幻觉 `no_hallucination`
|
||||||
|
- 格式合规性 `format`
|
||||||
|
- 综合分 `score` + `reason`
|
||||||
|
5. 任务状态持久化在后端 `eval_tasks.json`(支持 running/completed/failed/stopped),前端轮询进度。
|
||||||
|
|
||||||
|
**前端页面**
|
||||||
|
- `frontend/src/views/evaluation/EvaluateTask.vue`:任务列表、创建评测对话框(选模型、数据集、指标、可选 judge 配置)
|
||||||
|
- `frontend/src/views/evaluation/EvaluateReport.vue`:报告页,展示综合得分、BLEU、ROUGE-L、各维度指标及「参考答案 vs 模型预测 vs LLM 评审」对比样例
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. 平台评估系统(基于评估数据集 + 判卷模型)
|
||||||
|
|
||||||
|
后端路由位于 `backend/app/api/v1/evaluation/__init__.py`,是平台业务层自研的评测体系。通过「评估数据集」组织题目,可一次性对 **多个被测模型 + 指定判卷模型** 进行批量评分。
|
||||||
|
|
||||||
|
**核心概念(数据模型 `backend/app/models/models.py`)**
|
||||||
|
- `EvalDataset`(`models.py:131`):评估数据集,从项目问答对(`Question`/`Chunk`)中按 `question_type`(mixed/fact/reasoning)选题构建,状态 `pending/running/completed/failed`。
|
||||||
|
- `EvalResult`(`models.py:147`):单条评测结果,含 `judge_score`(0–5 分)、`is_correct`(true/false/partial)、`feedback`、`expected_answer` 等。
|
||||||
|
- `Task`(`models.py:184`):后台任务,`task_type="model-evaluation"`,记录进度与 `model_info`(存放平均分等汇总)。
|
||||||
|
|
||||||
|
**评测流程(`process_evaluation_task`,`backend/app/services/task_processor.py:336` 起)**
|
||||||
|
1. 加载评估数据集关联的题目,可选带入 `chunk` 上下文(RAG 场景)。
|
||||||
|
2. 对每道题,先用 `build_eval_prompt` 组合「上下文 + 题目 + 参考答案」,调用 **判卷模型**(`call_model`,temperature=0.3)生成评分。
|
||||||
|
3. `parse_eval_result` 解析出 `score`(0–5)、`is_correct`、`feedback`,写入 `EvalResult`。
|
||||||
|
4. 逐题提交进度(`completed_count` / `progress`),支持中途 `stopped`。
|
||||||
|
5. 汇总:`avg_score = 总分/有效数 × 20`(换算百分制),`avg_score_5 = 总分/有效数`(5 分制),存入 `task.model_info`。判定规则:得分 **≥3 视为正确**。
|
||||||
|
|
||||||
|
**特点**
|
||||||
|
- 判卷与被测模型解耦:被测模型给出答案,判卷模型(judge)独立评分,降低自评偏差。
|
||||||
|
- 支持失败隔离:单题异常写入 `evaluation_status: failed` 记录而不中断整体任务。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. 训练时验证评估
|
||||||
|
|
||||||
|
在微调训练任务执行期间,由 `backend/app/services/task_runner.py` 的 `do_eval` 触发:
|
||||||
|
|
||||||
|
- 在训练过程中对验证集(validation set)计算 `eval_loss`,用于监控过拟合。
|
||||||
|
- 结果回填到 `Task` 的 `loss_info` / `detail`,前端绘制 loss 曲线。
|
||||||
|
- 属于训练配套的轻量评估,不参与上述 1–3 的业务评测。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 附属:前端评测相关页面
|
||||||
|
|
||||||
|
| 文件 | 作用 |
|
||||||
|
| --- | --- |
|
||||||
|
| `frontend/src/views/evaluation/EvaluateTask.vue` | 评估工作台:任务列表 + 创建评测 |
|
||||||
|
| `frontend/src/views/evaluation/EvaluateReport.vue` | 评估报告:指标卡 + 维度标签 + 对比样例 |
|
||||||
|
| `frontend/src/api/evaluation.ts` | 评估工作台接口封装 |
|
||||||
|
| 平台评估系统入口 | 评估数据集管理 + 评估任务(model-evaluation)创建与结果查看 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 小结
|
||||||
|
|
||||||
|
- **想要学术榜单式准确率** → 用能力 1(LLaMA-Factory `eval/`)。
|
||||||
|
- **想要开放式生成质量(BLEU/ROUGE + LLM 评审)** → 用能力 2(评估工作台 `/evaluation/run`)。
|
||||||
|
- **想要基于自有问答数据、用判卷模型批量打分** → 用能力 3(平台评估系统 `model-evaluation` 任务)。
|
||||||
|
- **训练过程监控** → 能力 4(`do_eval` 验证集 loss)。
|
||||||
|
|
||||||
|
三种业务评测(1/2/3)相互独立,可并存于同一平台;数据模型(`EvalDataset`/`EvalResult`/`Task`)主要服务于能力 3,而能力 2 使用独立的 `eval_tasks.json` 文件持久化。
|
||||||
@@ -18,12 +18,12 @@ const auth = useAuthStore()
|
|||||||
*/
|
*/
|
||||||
let hiddenAt = 0
|
let hiddenAt = 0
|
||||||
|
|
||||||
function handleVisibility() {
|
async function handleVisibility() {
|
||||||
if (document.hidden) {
|
if (document.hidden) {
|
||||||
hiddenAt = Date.now()
|
hiddenAt = Date.now()
|
||||||
} else {
|
} else {
|
||||||
if (hiddenAt > 0 && Date.now() - hiddenAt >= SESSION_TIMEOUT) {
|
if (hiddenAt > 0 && Date.now() - hiddenAt >= SESSION_TIMEOUT) {
|
||||||
auth.logout()
|
await auth.logout()
|
||||||
ElMessage.warning('登录已过期,请重新登录')
|
ElMessage.warning('登录已过期,请重新登录')
|
||||||
router.push('/login')
|
router.push('/login')
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,6 +1,8 @@
|
|||||||
import { get, post, del } from '../request'
|
import { get, post, del } from '../request'
|
||||||
import type { CompareTask, CompareModelRef } from '@/types'
|
import type { CompareTask, CompareModelRef } from '@/types'
|
||||||
|
|
||||||
|
const INFERENCE_START_TIMEOUT_MS = 15 * 60 * 1000
|
||||||
|
|
||||||
/** 推理/对比任务列表 */
|
/** 推理/对比任务列表 */
|
||||||
export const getCompareList = () => get<CompareTask[]>('/model-compare')
|
export const getCompareList = () => get<CompareTask[]>('/model-compare')
|
||||||
|
|
||||||
@@ -12,7 +14,7 @@ export const createCompare = (data: Partial<CompareTask>) =>
|
|||||||
post<{ id: string | number }>('/model-compare', data)
|
post<{ id: string | number }>('/model-compare', data)
|
||||||
|
|
||||||
/** 删除任务 */
|
/** 删除任务 */
|
||||||
export const deleteCompare = (id: string | number) => del(`/model-compare/${id}`)
|
export const deleteCompare = (id: string | number) => del(`/model-compare/${id}`, undefined, { timeout: 60_000 })
|
||||||
|
|
||||||
/** 更新任务加载状态 */
|
/** 更新任务加载状态 */
|
||||||
export const updateLoadStatus = (id: string | number, load_status: any) =>
|
export const updateLoadStatus = (id: string | number, load_status: any) =>
|
||||||
@@ -34,7 +36,8 @@ export const stopModelByPid = (pid: number) =>
|
|||||||
post('/model-compare/stop-by-pid', { pid })
|
post('/model-compare/stop-by-pid', { pid })
|
||||||
|
|
||||||
/** 加载任务 */
|
/** 加载任务 */
|
||||||
export const loadCompare = (id: string | number) => post(`/model-compare/${id}/load`)
|
export const loadCompare = (id: string | number) =>
|
||||||
|
post(`/model-compare/${id}/load`, undefined, { timeout: INFERENCE_START_TIMEOUT_MS })
|
||||||
|
|
||||||
/** 卸载任务 */
|
/** 卸载任务 */
|
||||||
export const unloadCompare = (id: string | number) => post(`/model-compare/${id}/unload`)
|
export const unloadCompare = (id: string | number) => post(`/model-compare/${id}/unload`)
|
||||||
@@ -64,6 +67,31 @@ export const streamChat = async (data: any): Promise<any> => {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** 真实流式对话 — 使用 fetch 调用后端 SSE 端点,返回 Response 供 ReadableStream 消费 */
|
||||||
|
export const streamChatReal = (data: any): Promise<Response> => {
|
||||||
|
const messages = data.messages || []
|
||||||
|
if (!messages.length && data.user_question) {
|
||||||
|
if (data.system_prompt) {
|
||||||
|
messages.push({ role: 'system', content: data.system_prompt })
|
||||||
|
}
|
||||||
|
messages.push({ role: 'user', content: data.user_question })
|
||||||
|
}
|
||||||
|
return fetch('/modelTF/model-compare/stream-chat', {
|
||||||
|
method: 'POST',
|
||||||
|
headers: { 'Content-Type': 'application/json' },
|
||||||
|
body: JSON.stringify({
|
||||||
|
messages,
|
||||||
|
temperature: data.temperature ?? 0.7,
|
||||||
|
top_p: data.top_p ?? 0.95,
|
||||||
|
max_tokens: data.max_tokens ?? 2048,
|
||||||
|
// 透传 task_id/node_id,让后端按 load_status 路由到真正加载了模型的算力节点,
|
||||||
|
// 避免在多节点时回退到“第一个在线节点”导致连接失败
|
||||||
|
task_id: data.task_id,
|
||||||
|
node_id: data.node_id,
|
||||||
|
}),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
/** 非流式对话(按端口代理) */
|
/** 非流式对话(按端口代理) */
|
||||||
export const chatWithPort = (data: any) => post('/model-compare/chat-with-port', data)
|
export const chatWithPort = (data: any) => post('/model-compare/chat-with-port', data)
|
||||||
|
|
||||||
@@ -73,8 +101,8 @@ export const batchChat = (data: any) => post('/model-chat/batch', data)
|
|||||||
/** 本地 transformers 模型对话 */
|
/** 本地 transformers 模型对话 */
|
||||||
export const localChat = (data: any) => post('/model-chat/local/chat', data)
|
export const localChat = (data: any) => post('/model-chat/local/chat', data)
|
||||||
|
|
||||||
/** 预加载本地模型 */
|
/** 预加载本地模型(模型加载耗时长,超时 15 分钟) */
|
||||||
export const preloadLocalModel = (data: any) => post('/model-chat/local/preload', data)
|
export const preloadLocalModel = (data: any) => post('/model-chat/local/preload', data, { timeout: INFERENCE_START_TIMEOUT_MS })
|
||||||
|
|
||||||
/** 预加载已训练模型 */
|
/** 预加载已训练模型(超时 15 分钟) */
|
||||||
export const preloadTrainedModel = (data: any) => post('/model-chat/trained/preload', data)
|
export const preloadTrainedModel = (data: any) => post('/model-chat/trained/preload', data, { timeout: INFERENCE_START_TIMEOUT_MS })
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
import { get, post, put } from '../request'
|
import { del, get, post, put } from '../request'
|
||||||
|
|
||||||
export interface ComputeNode {
|
export interface ComputeNode {
|
||||||
id: string
|
id: string
|
||||||
@@ -95,6 +95,9 @@ export const createComputeNode = (data: ComputeNodePayload) =>
|
|||||||
export const updateComputeNode = (id: string, data: Partial<ComputeNode>) =>
|
export const updateComputeNode = (id: string, data: Partial<ComputeNode>) =>
|
||||||
put<ComputeNode>(`/compute/nodes/${id}`, data)
|
put<ComputeNode>(`/compute/nodes/${id}`, data)
|
||||||
|
|
||||||
|
export const deleteComputeNode = (id: string) =>
|
||||||
|
del<{ deleted: string }>(`/compute/nodes/${id}`)
|
||||||
|
|
||||||
export const testComputeNode = (id: string) =>
|
export const testComputeNode = (id: string) =>
|
||||||
post<{ node_id: string; success: boolean; latency_ms: number; gpu_count: number; error?: string }>(`/compute/nodes/${id}/test-connection`)
|
post<{ node_id: string; success: boolean; latency_ms: number; gpu_count: number; error?: string }>(`/compute/nodes/${id}/test-connection`)
|
||||||
|
|
||||||
|
|||||||
@@ -29,6 +29,7 @@ export const uploadDatasetFiles = (datasetId: string | number, files: File[]) =>
|
|||||||
files.forEach((f) => formData.append('files', f))
|
files.forEach((f) => formData.append('files', f))
|
||||||
return post(`/dataset-manage/upload/${datasetId}`, formData, {
|
return post(`/dataset-manage/upload/${datasetId}`, formData, {
|
||||||
headers: { 'Content-Type': 'multipart/form-data' },
|
headers: { 'Content-Type': 'multipart/form-data' },
|
||||||
|
timeout: 120000,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,16 @@
|
|||||||
import { get, post, put, del } from '../request'
|
import { get, post, put, del } from '../request'
|
||||||
import type { FineTuneStartPayload, FineTuneTask, TrainingProgress, LogContent } from '@/types'
|
import type { FineTuneStartPayload, FineTuneTask, TrainingProgress, LogContent } from '@/types'
|
||||||
|
|
||||||
|
export interface FineTuneMetricPoint {
|
||||||
|
step: number
|
||||||
|
epoch?: number | null
|
||||||
|
loss?: number | null
|
||||||
|
grad_norm?: number | null
|
||||||
|
learning_rate?: number | null
|
||||||
|
raw?: string
|
||||||
|
create_time?: string
|
||||||
|
}
|
||||||
|
|
||||||
export interface TrainingDiagnostic {
|
export interface TrainingDiagnostic {
|
||||||
level: string
|
level: string
|
||||||
title: string
|
title: string
|
||||||
@@ -80,6 +90,10 @@ export const getFineTuneLogs = (
|
|||||||
params: { tail_lines?: number; offset?: number; limit?: number } = {},
|
params: { tail_lines?: number; offset?: number; limit?: number } = {},
|
||||||
) => get<LogContent & { job_id?: string; source?: string }>(`/fine-tune/${id}/logs`, params)
|
) => get<LogContent & { job_id?: string; source?: string }>(`/fine-tune/${id}/logs`, params)
|
||||||
|
|
||||||
|
/** 获取训练指标曲线数据 */
|
||||||
|
export const getFineTuneMetrics = (id: string | number) =>
|
||||||
|
get<FineTuneMetricPoint[]>(`/fine-tune/${id}/metrics`)
|
||||||
|
|
||||||
/** 启动 TensorBoard */
|
/** 启动 TensorBoard */
|
||||||
export const startTensorboard = () => post('/fine-tune/tensorboard/start')
|
export const startTensorboard = () => post('/fine-tune/tensorboard/start')
|
||||||
|
|
||||||
|
|||||||
@@ -88,10 +88,14 @@ export const updateModelPurpose = (id: string | number, purpose: string) =>
|
|||||||
|
|
||||||
/** 合并 LoRA 权重 */
|
/** 合并 LoRA 权重 */
|
||||||
export const mergeModel = (data: {
|
export const mergeModel = (data: {
|
||||||
|
trained_model_id?: string | number
|
||||||
model_name: string
|
model_name: string
|
||||||
train_method: string
|
train_method: string
|
||||||
base_model_path: string
|
base_model_path: string
|
||||||
}) => post('/model-manage/merge', data)
|
adapter_path?: string
|
||||||
|
compute_node_id?: string
|
||||||
|
output_model_name?: string
|
||||||
|
}) => post('/model-manage/merge', data, { timeout: 15 * 60 * 1000 })
|
||||||
|
|
||||||
/** 导出已训练模型权重 */
|
/** 导出已训练模型权重 */
|
||||||
export const exportModelUrl = (modelName: string) =>
|
export const exportModelUrl = (modelName: string) =>
|
||||||
|
|||||||
@@ -53,3 +53,6 @@ export const updateProjectMember = (id: string, userId: string, role: string) =>
|
|||||||
/** 移除成员 */
|
/** 移除成员 */
|
||||||
export const removeProjectMember = (id: string, userId: string) =>
|
export const removeProjectMember = (id: string, userId: string) =>
|
||||||
del(`/projects/${id}/members/${userId}`)
|
del(`/projects/${id}/members/${userId}`)
|
||||||
|
|
||||||
|
/** 删除项目 */
|
||||||
|
export const deleteProject = (id: string) => del(`/projects/${id}`)
|
||||||
|
|||||||
@@ -8,6 +8,8 @@ import type {
|
|||||||
UpdateUserAccessPayload,
|
UpdateUserAccessPayload,
|
||||||
} from '@/types'
|
} from '@/types'
|
||||||
|
|
||||||
|
export type { SystemUser } from '@/types'
|
||||||
|
|
||||||
/** 系统信息(CPU/内存/磁盘/GPU/网络/系统) */
|
/** 系统信息(CPU/内存/磁盘/GPU/网络/系统) */
|
||||||
export const getSystemInfo = () => get<SystemInfo>('/system-info')
|
export const getSystemInfo = () => get<SystemInfo>('/system-info')
|
||||||
|
|
||||||
@@ -18,6 +20,10 @@ export const getHealth = () => get<HealthMetrics>('/health')
|
|||||||
export const login = (username: string, password: string) =>
|
export const login = (username: string, password: string) =>
|
||||||
post<LoginResponse>('/login', { username, password })
|
post<LoginResponse>('/login', { username, password })
|
||||||
|
|
||||||
|
/** 登出 */
|
||||||
|
export const logout = (sessionId?: string) =>
|
||||||
|
post('/logout', { session_id: sessionId || '' })
|
||||||
|
|
||||||
/** 用户列表 */
|
/** 用户列表 */
|
||||||
export const getUsers = () => get<SystemUser[]>('/users')
|
export const getUsers = () => get<SystemUser[]>('/users')
|
||||||
|
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
import { get, post, put } from '../request'
|
import { del, get, post, put } from '../request'
|
||||||
|
|
||||||
export interface Tenant {
|
export interface Tenant {
|
||||||
id: string
|
id: string
|
||||||
@@ -25,6 +25,9 @@ export const createTenant = (payload: Partial<Tenant>) =>
|
|||||||
export const updateTenant = (id: string, payload: Partial<Tenant>) =>
|
export const updateTenant = (id: string, payload: Partial<Tenant>) =>
|
||||||
put<Tenant>(`/tenants/${id}`, payload)
|
put<Tenant>(`/tenants/${id}`, payload)
|
||||||
|
|
||||||
|
/** 删除租户 */
|
||||||
|
export const deleteTenant = (id: string) => del(`/tenants/${id}`)
|
||||||
|
|
||||||
/** 设置租户配额 */
|
/** 设置租户配额 */
|
||||||
export const setTenantQuota = (id: string, quota: Record<string, unknown>) =>
|
export const setTenantQuota = (id: string, quota: Record<string, unknown>) =>
|
||||||
put<Tenant>(`/tenants/${id}/quota`, { quota })
|
put<Tenant>(`/tenants/${id}/quota`, { quota })
|
||||||
|
|||||||
@@ -14,7 +14,7 @@ export interface ApiResult<T = any> {
|
|||||||
const service: AxiosInstance = axios.create({
|
const service: AxiosInstance = axios.create({
|
||||||
// Use a relative path; Vite proxies /modelTF to http://localhost:17861 in local development.
|
// Use a relative path; Vite proxies /modelTF to http://localhost:17861 in local development.
|
||||||
baseURL: '/modelTF',
|
baseURL: '/modelTF',
|
||||||
timeout: 30000,
|
timeout: 120000,
|
||||||
})
|
})
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|||||||
@@ -2,6 +2,7 @@
|
|||||||
import { computed, ref, watch } from 'vue'
|
import { computed, ref, watch } from 'vue'
|
||||||
import { ElMessage } from 'element-plus'
|
import { ElMessage } from 'element-plus'
|
||||||
import { getAcl, setAcl, type AclEntry } from '@/api/modules/acl'
|
import { getAcl, setAcl, type AclEntry } from '@/api/modules/acl'
|
||||||
|
import { getUsers, type SystemUser } from '@/api/modules/system'
|
||||||
|
|
||||||
const props = defineProps<{
|
const props = defineProps<{
|
||||||
modelValue: boolean
|
modelValue: boolean
|
||||||
@@ -15,13 +16,20 @@ const visible = computed({
|
|||||||
set: (v) => emit('update:modelValue', v),
|
set: (v) => emit('update:modelValue', v),
|
||||||
})
|
})
|
||||||
const entries = ref<AclEntry[]>([])
|
const entries = ref<AclEntry[]>([])
|
||||||
|
const users = ref<SystemUser[]>([])
|
||||||
const loading = ref(false)
|
const loading = ref(false)
|
||||||
const ALL_PERMS = ['read', 'write', 'execute', 'download', 'delete', 'share']
|
const ALL_PERMS = ['read', 'write', 'execute', 'download', 'delete', 'share']
|
||||||
|
const PROJECT_ROLES = ['member', 'admin', 'viewer']
|
||||||
|
|
||||||
async function load() {
|
async function load() {
|
||||||
loading.value = true
|
loading.value = true
|
||||||
try {
|
try {
|
||||||
entries.value = await getAcl(props.resourceType, props.resourceId)
|
const [acl, us] = await Promise.all([
|
||||||
|
getAcl(props.resourceType, props.resourceId),
|
||||||
|
getUsers().catch(() => [] as SystemUser[]),
|
||||||
|
])
|
||||||
|
entries.value = acl
|
||||||
|
users.value = us
|
||||||
} finally {
|
} finally {
|
||||||
loading.value = false
|
loading.value = false
|
||||||
}
|
}
|
||||||
@@ -53,7 +61,12 @@ async function save() {
|
|||||||
<el-option label="用户" value="user" />
|
<el-option label="用户" value="user" />
|
||||||
<el-option label="项目角色" value="project_role" />
|
<el-option label="项目角色" value="project_role" />
|
||||||
</el-select>
|
</el-select>
|
||||||
<el-input v-model="entry.subject_id" placeholder="subject ID" style="width: 200px" />
|
<el-select v-if="entry.subject_type === 'user'" v-model="entry.subject_id" placeholder="选择用户" style="width: 200px" filterable>
|
||||||
|
<el-option v-for="u in users" :key="u.id" :label="`${u.username} (${u.id})`" :value="u.id" />
|
||||||
|
</el-select>
|
||||||
|
<el-select v-else v-model="entry.subject_id" placeholder="选择角色" style="width: 200px">
|
||||||
|
<el-option v-for="r in PROJECT_ROLES" :key="r" :label="r" :value="r" />
|
||||||
|
</el-select>
|
||||||
<el-checkbox-group v-model="entry.permissions">
|
<el-checkbox-group v-model="entry.permissions">
|
||||||
<el-checkbox v-for="p in ALL_PERMS" :key="p" :value="p">{{ p }}</el-checkbox>
|
<el-checkbox v-for="p in ALL_PERMS" :key="p" :value="p">{{ p }}</el-checkbox>
|
||||||
</el-checkbox-group>
|
</el-checkbox-group>
|
||||||
|
|||||||
@@ -132,8 +132,8 @@ async function handleSelect(key: string) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
function handleLogout() {
|
async function handleLogout() {
|
||||||
auth.logout()
|
await auth.logout()
|
||||||
router.push('/login')
|
router.push('/login')
|
||||||
}
|
}
|
||||||
</script>
|
</script>
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
import { ref } from 'vue'
|
import { ref } from 'vue'
|
||||||
import { streamChat } from '@/api/modules/compare'
|
import { streamChat, streamChatReal } from '@/api/modules/compare'
|
||||||
|
|
||||||
export interface StreamMessage {
|
export interface StreamMessage {
|
||||||
/** 用户问题 */
|
/** 用户问题 */
|
||||||
@@ -20,6 +20,11 @@ export interface StreamMessage {
|
|||||||
error?: string
|
error?: string
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export interface SendOptions {
|
||||||
|
/** 是否使用 mock 模式(默认 true,向后兼容) */
|
||||||
|
useMock?: boolean
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* 流式对话 composable
|
* 流式对话 composable
|
||||||
* 移植自原 model-chat.html:
|
* 移植自原 model-chat.html:
|
||||||
@@ -39,6 +44,24 @@ export function useStreamChat() {
|
|||||||
})
|
})
|
||||||
const loading = ref(false)
|
const loading = ref(false)
|
||||||
|
|
||||||
|
/** 从 SSE 帧中提取错误信息(后端/计算节点错误以 data: {"error": "..."} 形式下发) */
|
||||||
|
function extractSseError(buffer: string): string | null {
|
||||||
|
const trimmed = buffer.trim()
|
||||||
|
if (!trimmed.startsWith('data: ')) return null
|
||||||
|
const lines = trimmed.split(/\r?\n/)
|
||||||
|
for (let i = lines.length - 1; i >= 0; i--) {
|
||||||
|
const line = lines[i].trim()
|
||||||
|
if (!line.startsWith('data: ')) continue
|
||||||
|
try {
|
||||||
|
const obj = JSON.parse(line.slice(6))
|
||||||
|
if (obj && typeof obj.error === 'string' && obj.error) return obj.error
|
||||||
|
} catch {
|
||||||
|
/* 非 JSON 的 data 行忽略 */
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return trimmed
|
||||||
|
}
|
||||||
|
|
||||||
/** 从内容中解析 think 标签 */
|
/** 从内容中解析 think 标签 */
|
||||||
function parseContent(content: string) {
|
function parseContent(content: string) {
|
||||||
const thinkRegex = /<think>([\s\S]*?)(<\/think>)?/g
|
const thinkRegex = /<think>([\s\S]*?)(<\/think>)?/g
|
||||||
@@ -65,8 +88,10 @@ export function useStreamChat() {
|
|||||||
/**
|
/**
|
||||||
* 发起流式对话
|
* 发起流式对话
|
||||||
* @param payload 后端请求体 { port, model_name, model_path, system_prompt, user_question, ... }
|
* @param payload 后端请求体 { port, model_name, model_path, system_prompt, user_question, ... }
|
||||||
|
* @param options 可选配置 { useMock?: boolean }
|
||||||
*/
|
*/
|
||||||
async function send(payload: any) {
|
async function send(payload: any, options?: SendOptions) {
|
||||||
|
const useMock = options?.useMock ?? true
|
||||||
loading.value = true
|
loading.value = true
|
||||||
message.value = {
|
message.value = {
|
||||||
question: payload.user_question || '',
|
question: payload.user_question || '',
|
||||||
@@ -82,7 +107,10 @@ export function useStreamChat() {
|
|||||||
const UPDATE_INTERVAL = 50 // 50ms 节流
|
const UPDATE_INTERVAL = 50 // 50ms 节流
|
||||||
|
|
||||||
try {
|
try {
|
||||||
const response = await streamChat(payload)
|
const response = useMock
|
||||||
|
? await streamChat(payload)
|
||||||
|
: await streamChatReal(payload)
|
||||||
|
|
||||||
if (!response.ok) {
|
if (!response.ok) {
|
||||||
throw new Error(`HTTP ${response.status}`)
|
throw new Error(`HTTP ${response.status}`)
|
||||||
}
|
}
|
||||||
@@ -111,6 +139,16 @@ export function useStreamChat() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// 最终更新
|
// 最终更新
|
||||||
|
// 若整段响应是 SSE 错误帧,提取 error 字段以干净文案展示
|
||||||
|
const sseError = extractSseError(buffer)
|
||||||
|
if (sseError) {
|
||||||
|
message.value.isThinking = false
|
||||||
|
message.value.isStreaming = false
|
||||||
|
message.value.done = true
|
||||||
|
message.value.error = sseError
|
||||||
|
message.value.displayContent = sseError
|
||||||
|
return
|
||||||
|
}
|
||||||
const parsed = parseContent(buffer)
|
const parsed = parseContent(buffer)
|
||||||
message.value.thinkContent = parsed.think
|
message.value.thinkContent = parsed.think
|
||||||
message.value.displayContent = parsed.display
|
message.value.displayContent = parsed.display
|
||||||
|
|||||||
@@ -377,7 +377,7 @@ router.beforeEach((to, _from, next) => {
|
|||||||
}
|
}
|
||||||
|
|
||||||
if (!auth.isLoggedIn) {
|
if (!auth.isLoggedIn) {
|
||||||
auth.logout()
|
auth.logout() // fire-and-forget,无需阻塞跳转
|
||||||
next({ name: 'login' })
|
next({ name: 'login' })
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,9 +1,10 @@
|
|||||||
import { defineStore } from 'pinia'
|
import { defineStore } from 'pinia'
|
||||||
import { ref, computed } from 'vue'
|
import { ref, computed } from 'vue'
|
||||||
import { login as loginApi } from '@/api/modules/system'
|
import { login as loginApi, logout as logoutApi } from '@/api/modules/system'
|
||||||
import type { PermissionCode, SystemUser } from '@/types'
|
import type { PermissionCode, SystemUser } from '@/types'
|
||||||
|
|
||||||
const USER_STORAGE_KEY = 'currentUser'
|
const USER_STORAGE_KEY = 'currentUser'
|
||||||
|
const SESSION_STORAGE_KEY = 'sessionId'
|
||||||
|
|
||||||
const allPermissions: PermissionCode[] = [
|
const allPermissions: PermissionCode[] = [
|
||||||
'dashboard',
|
'dashboard',
|
||||||
@@ -29,20 +30,6 @@ function restoreUser(): SystemUser | null {
|
|||||||
localStorage.removeItem(USER_STORAGE_KEY)
|
localStorage.removeItem(USER_STORAGE_KEY)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// 兼容改造前已经登录的 admin 会话。
|
|
||||||
if (localStorage.getItem('username') === 'admin') {
|
|
||||||
return {
|
|
||||||
id: 'USR-0001',
|
|
||||||
username: 'admin',
|
|
||||||
display_name: '系统管理员',
|
|
||||||
role: 'admin',
|
|
||||||
status: 'active',
|
|
||||||
permissions: allPermissions,
|
|
||||||
create_time: '2026-01-01T08:00:00+08:00',
|
|
||||||
protected: true,
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return null
|
return null
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -69,6 +56,9 @@ export const useAuthStore = defineStore('auth', () => {
|
|||||||
currentUser.value = response.user
|
currentUser.value = response.user
|
||||||
localStorage.setItem('username', response.user.username)
|
localStorage.setItem('username', response.user.username)
|
||||||
localStorage.setItem(USER_STORAGE_KEY, JSON.stringify(response.user))
|
localStorage.setItem(USER_STORAGE_KEY, JSON.stringify(response.user))
|
||||||
|
if (response.session_id) {
|
||||||
|
localStorage.setItem(SESSION_STORAGE_KEY, response.session_id)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/** 检查当前账号是否拥有指定模块权限。 */
|
/** 检查当前账号是否拥有指定模块权限。 */
|
||||||
@@ -78,10 +68,15 @@ export const useAuthStore = defineStore('auth', () => {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/** 退出 */
|
/** 退出 */
|
||||||
function logout() {
|
async function logout() {
|
||||||
|
const sessionId = localStorage.getItem(SESSION_STORAGE_KEY)
|
||||||
|
if (sessionId) {
|
||||||
|
try { await logoutApi(sessionId) } catch { /* 静默 */ }
|
||||||
|
}
|
||||||
currentUser.value = null
|
currentUser.value = null
|
||||||
localStorage.removeItem('username')
|
localStorage.removeItem('username')
|
||||||
localStorage.removeItem(USER_STORAGE_KEY)
|
localStorage.removeItem(USER_STORAGE_KEY)
|
||||||
|
localStorage.removeItem(SESSION_STORAGE_KEY)
|
||||||
}
|
}
|
||||||
|
|
||||||
return {
|
return {
|
||||||
|
|||||||
@@ -34,6 +34,10 @@ export interface TrainedModel {
|
|||||||
name: string
|
name: string
|
||||||
train_methods?: TrainMethod[]
|
train_methods?: TrainMethod[]
|
||||||
base_model_path?: string
|
base_model_path?: string
|
||||||
|
artifact_dir?: string
|
||||||
|
adapter_path?: string
|
||||||
|
compute_node_id?: string
|
||||||
|
compute_node_name?: string
|
||||||
create_time?: string
|
create_time?: string
|
||||||
merged?: boolean
|
merged?: boolean
|
||||||
merging?: boolean
|
merging?: boolean
|
||||||
@@ -132,6 +136,7 @@ export interface FineTuneTask {
|
|||||||
train_dataset_id?: number | string
|
train_dataset_id?: number | string
|
||||||
auto_merge?: boolean
|
auto_merge?: boolean
|
||||||
output_model_name?: string
|
output_model_name?: string
|
||||||
|
compute_node_id?: string
|
||||||
gpus?: number[]
|
gpus?: number[]
|
||||||
batch_size?: number
|
batch_size?: number
|
||||||
learning_rate?: number
|
learning_rate?: number
|
||||||
@@ -211,6 +216,9 @@ export interface LoadedModel {
|
|||||||
status?: string
|
status?: string
|
||||||
pid?: number
|
pid?: number
|
||||||
port?: number
|
port?: number
|
||||||
|
node_id?: string
|
||||||
|
node_name?: string
|
||||||
|
error?: string
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface CompareTask {
|
export interface CompareTask {
|
||||||
@@ -229,6 +237,8 @@ export interface CompareModelRef {
|
|||||||
model_name: string
|
model_name: string
|
||||||
model_path: string
|
model_path: string
|
||||||
gpu_id: number
|
gpu_id: number
|
||||||
|
node_id?: string
|
||||||
|
node_name?: string
|
||||||
source?: string
|
source?: string
|
||||||
port?: number
|
port?: number
|
||||||
}
|
}
|
||||||
@@ -244,7 +254,9 @@ export interface EvalTask {
|
|||||||
model_name?: string
|
model_name?: string
|
||||||
model_id?: number | string
|
model_id?: number | string
|
||||||
dataset?: string
|
dataset?: string
|
||||||
|
dataset_id?: number | string
|
||||||
metric?: string
|
metric?: string
|
||||||
|
metric_label?: string
|
||||||
score?: number
|
score?: number
|
||||||
status?: string
|
status?: string
|
||||||
create_time?: string
|
create_time?: string
|
||||||
@@ -271,6 +283,7 @@ export interface StartEvalPayload {
|
|||||||
eval_type: EvalType
|
eval_type: EvalType
|
||||||
model_id: string | number
|
model_id: string | number
|
||||||
gpu_id: string | number
|
gpu_id: string | number
|
||||||
|
compute_node_id?: string
|
||||||
dataset_id: string | number
|
dataset_id: string | number
|
||||||
dimension_id: string | number
|
dimension_id: string | number
|
||||||
data_source: 'dataset' | 'inference'
|
data_source: 'dataset' | 'inference'
|
||||||
@@ -355,16 +368,16 @@ export interface GpuInfo {
|
|||||||
power_w: number
|
power_w: number
|
||||||
id?: number
|
id?: number
|
||||||
uuid?: string
|
uuid?: string
|
||||||
status?: 'idle' | 'busy' | 'warning' | 'offline'
|
status?: 'idle' | 'busy' | 'reserved' | 'warning' | 'offline'
|
||||||
memory_percent?: number
|
memory_percent?: number
|
||||||
power_limit_w?: number
|
power_limit_w?: number
|
||||||
processes?: GpuProcess[]
|
processes?: GpuProcess[]
|
||||||
fan_speed?: number
|
fan_speed?: number
|
||||||
clock_mhz?: number
|
clock_mhz?: number
|
||||||
driver_version?: string
|
|
||||||
node_id?: string
|
node_id?: string
|
||||||
node_code?: string
|
node_code?: string
|
||||||
node_name?: string
|
node_name?: string
|
||||||
|
driver_version?: string
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface SystemInfo {
|
export interface SystemInfo {
|
||||||
@@ -446,6 +459,7 @@ export interface SystemUser {
|
|||||||
export interface LoginResponse {
|
export interface LoginResponse {
|
||||||
token: string
|
token: string
|
||||||
user: SystemUser
|
user: SystemUser
|
||||||
|
session_id?: string
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface CreateUserPayload {
|
export interface CreateUserPayload {
|
||||||
|
|||||||
@@ -3,7 +3,8 @@ import { onMounted, reactive, ref } from 'vue'
|
|||||||
import { ElMessage } from 'element-plus'
|
import { ElMessage } from 'element-plus'
|
||||||
import DataTablePage from '@/components/DataTablePage.vue'
|
import DataTablePage from '@/components/DataTablePage.vue'
|
||||||
import { getApprovalInstances, decideApproval, type ApprovalInstance } from '@/api/modules/approval'
|
import { getApprovalInstances, decideApproval, type ApprovalInstance } from '@/api/modules/approval'
|
||||||
import { getUsers, type SystemUser } from '@/api/modules/system'
|
import { getUsers } from '@/api/modules/system'
|
||||||
|
import type { SystemUser } from '@/types'
|
||||||
|
|
||||||
const loading = ref(false)
|
const loading = ref(false)
|
||||||
const instances = ref<ApprovalInstance[]>([])
|
const instances = ref<ApprovalInstance[]>([])
|
||||||
@@ -48,6 +49,10 @@ function openDecide(inst: ApprovalInstance) {
|
|||||||
showDecide.value = true
|
showDecide.value = true
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function asApprovalInstance(row: unknown): ApprovalInstance {
|
||||||
|
return row as ApprovalInstance
|
||||||
|
}
|
||||||
|
|
||||||
async function submitDecision() {
|
async function submitDecision() {
|
||||||
if (!current.value) return
|
if (!current.value) return
|
||||||
if (!decision.value.approver_id) {
|
if (!decision.value.approver_id) {
|
||||||
@@ -72,7 +77,7 @@ onMounted(() => {
|
|||||||
|
|
||||||
<template>
|
<template>
|
||||||
<div class="page">
|
<div class="page">
|
||||||
<DataTablePage title="审批实例" :data="instances" :loading="loading" searchable search-fields="resource_type,resource_id">
|
<DataTablePage title="审批实例" :data="instances" :loading="loading" searchable :search-fields="['resource_type', 'resource_id']">
|
||||||
<template #toolbar-extra>
|
<template #toolbar-extra>
|
||||||
<el-select v-model="statusFilter" placeholder="状态" clearable style="width: 140px" @change="load">
|
<el-select v-model="statusFilter" placeholder="状态" clearable style="width: 140px" @change="load">
|
||||||
<el-option v-for="s in statusOptions" :key="s.value" :label="s.label" :value="s.value" />
|
<el-option v-for="s in statusOptions" :key="s.value" :label="s.label" :value="s.value" />
|
||||||
@@ -82,14 +87,14 @@ onMounted(() => {
|
|||||||
<el-table-column prop="resource_type" label="资源类型" min-width="120" />
|
<el-table-column prop="resource_type" label="资源类型" min-width="120" />
|
||||||
<el-table-column prop="resource_id" label="资源 ID" min-width="160" show-overflow-tooltip />
|
<el-table-column prop="resource_id" label="资源 ID" min-width="160" show-overflow-tooltip />
|
||||||
<el-table-column prop="applicant_id" label="申请人" min-width="120">
|
<el-table-column prop="applicant_id" label="申请人" min-width="120">
|
||||||
<template #default="{ row }">{{ userName(row.applicant_id) }}</template>
|
<template #default="{ row }">{{ userName(asApprovalInstance(row).applicant_id) }}</template>
|
||||||
</el-table-column>
|
</el-table-column>
|
||||||
<el-table-column prop="status" label="状态" min-width="100" />
|
<el-table-column prop="status" label="状态" min-width="100" />
|
||||||
<el-table-column prop="current_step" label="当前步骤" min-width="100" />
|
<el-table-column prop="current_step" label="当前步骤" min-width="100" />
|
||||||
<el-table-column prop="create_time" label="创建时间" min-width="180" />
|
<el-table-column prop="create_time" label="创建时间" min-width="180" />
|
||||||
</template>
|
</template>
|
||||||
<template #actions="{ row }">
|
<template #actions="{ row }">
|
||||||
<el-button v-if="row.status === 'pending'" link type="primary" @click="openDecide(row)">审批</el-button>
|
<el-button v-if="asApprovalInstance(row).status === 'pending'" link type="primary" @click="openDecide(asApprovalInstance(row))">审批</el-button>
|
||||||
</template>
|
</template>
|
||||||
</DataTablePage>
|
</DataTablePage>
|
||||||
<el-dialog v-model="showDecide" title="审批决策" width="480px">
|
<el-dialog v-model="showDecide" title="审批决策" width="480px">
|
||||||
|
|||||||
@@ -1,12 +1,12 @@
|
|||||||
<script setup lang="ts">
|
<script setup lang="ts">
|
||||||
import { computed, onMounted, onUnmounted, reactive, ref, watch } from 'vue'
|
import { computed, onMounted, onUnmounted, reactive, ref, watch } from 'vue'
|
||||||
import { useRoute, useRouter } from 'vue-router'
|
import { useRoute, useRouter } from 'vue-router'
|
||||||
import { ElMessage } from 'element-plus'
|
import { ElMessage, ElMessageBox } from 'element-plus'
|
||||||
import {
|
import {
|
||||||
checkNodeReplicaDrift,
|
checkNodeReplicaDrift,
|
||||||
createComputeNode,
|
createComputeNode,
|
||||||
|
deleteComputeNode,
|
||||||
disableComputeNode,
|
disableComputeNode,
|
||||||
drainComputeNode,
|
|
||||||
enableComputeNode,
|
enableComputeNode,
|
||||||
getComputeGpus,
|
getComputeGpus,
|
||||||
getComputeNodes,
|
getComputeNodes,
|
||||||
@@ -129,11 +129,10 @@ async function changeTab(name: string | number) {
|
|||||||
await router.replace({ path: '/compute', query: { tab: String(name) } })
|
await router.replace({ path: '/compute', query: { tab: String(name) } })
|
||||||
}
|
}
|
||||||
|
|
||||||
async function handleNodeAction(action: 'enable' | 'disable' | 'drain' | 'test', node: ComputeNode) {
|
async function handleNodeAction(action: 'enable' | 'disable' | 'test', node: ComputeNode) {
|
||||||
const nodeId = String(node.id)
|
const nodeId = String(node.id)
|
||||||
if (action === 'enable') await enableComputeNode(nodeId)
|
if (action === 'enable') await enableComputeNode(nodeId)
|
||||||
if (action === 'disable') await disableComputeNode(nodeId)
|
if (action === 'disable') await disableComputeNode(nodeId)
|
||||||
if (action === 'drain') await drainComputeNode(nodeId)
|
|
||||||
if (action === 'test') {
|
if (action === 'test') {
|
||||||
const result = await testComputeNode(nodeId)
|
const result = await testComputeNode(nodeId)
|
||||||
if (result.success) {
|
if (result.success) {
|
||||||
@@ -145,6 +144,27 @@ async function handleNodeAction(action: 'enable' | 'disable' | 'drain' | 'test',
|
|||||||
await load()
|
await load()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async function handleDeleteNode(node: ComputeNode) {
|
||||||
|
try {
|
||||||
|
await ElMessageBox.confirm(
|
||||||
|
`确定删除算力节点「${node.name || node.code}」吗?节点删除后,其 GPU 设备和资源副本记录也会一并移除。`,
|
||||||
|
'删除算力节点',
|
||||||
|
{ type: 'warning', confirmButtonText: '删除', cancelButtonText: '取消' },
|
||||||
|
)
|
||||||
|
} catch {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
await deleteComputeNode(String(node.id))
|
||||||
|
ElMessage.success('算力节点已删除')
|
||||||
|
if (selectedNodeId.value === node.id) selectedNodeId.value = ''
|
||||||
|
await load({ showButtonLoading: true })
|
||||||
|
} catch (err: any) {
|
||||||
|
const message = err?.response?.data?.detail?.message || err?.response?.data?.message || '删除算力节点失败'
|
||||||
|
ElMessage.error(message)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
async function handleReplicaDriftCheck() {
|
async function handleReplicaDriftCheck() {
|
||||||
if (!selectedNodeId.value) return
|
if (!selectedNodeId.value) return
|
||||||
checkingReplicas.value = true
|
checkingReplicas.value = true
|
||||||
@@ -355,7 +375,7 @@ onUnmounted(() => {
|
|||||||
<el-button size="small" @click="handleNodeAction('test', asComputeNode(row))">测试</el-button>
|
<el-button size="small" @click="handleNodeAction('test', asComputeNode(row))">测试</el-button>
|
||||||
<el-button v-if="row.enabled" size="small" @click="handleNodeAction('disable', asComputeNode(row))">停用</el-button>
|
<el-button v-if="row.enabled" size="small" @click="handleNodeAction('disable', asComputeNode(row))">停用</el-button>
|
||||||
<el-button v-else size="small" type="primary" @click="handleNodeAction('enable', asComputeNode(row))">启用</el-button>
|
<el-button v-else size="small" type="primary" @click="handleNodeAction('enable', asComputeNode(row))">启用</el-button>
|
||||||
<el-button size="small" type="warning" plain @click="handleNodeAction('drain', asComputeNode(row))">维护</el-button>
|
<el-button size="small" type="danger" plain @click="handleDeleteNode(asComputeNode(row))">删除</el-button>
|
||||||
</template>
|
</template>
|
||||||
</el-table-column>
|
</el-table-column>
|
||||||
</el-table>
|
</el-table>
|
||||||
|
|||||||
@@ -66,8 +66,12 @@ const operationDistribution = ref<{ name: string; value: number }[]>([])
|
|||||||
const serviceIcon: Record<string, string> = {
|
const serviceIcon: Record<string, string> = {
|
||||||
'模型推理': 'fa-cube',
|
'模型推理': 'fa-cube',
|
||||||
'模型微调': 'fa-sliders',
|
'模型微调': 'fa-sliders',
|
||||||
|
'模型训练': 'fa-sliders',
|
||||||
'模型评测': 'fa-bar-chart',
|
'模型评测': 'fa-bar-chart',
|
||||||
|
'模型管理': 'fa-cubes',
|
||||||
|
'数据集管理': 'fa-file-text',
|
||||||
'数据处理': 'fa-filter',
|
'数据处理': 'fa-filter',
|
||||||
|
'数据类型转换': 'fa-exchange',
|
||||||
}
|
}
|
||||||
const roleLabel: Record<string, string> = {
|
const roleLabel: Record<string, string> = {
|
||||||
admin: '超级管理员',
|
admin: '超级管理员',
|
||||||
@@ -115,7 +119,7 @@ const chartOption = computed<EChartsOption>(() => ({
|
|||||||
borderWidth: 0,
|
borderWidth: 0,
|
||||||
padding: [10, 12],
|
padding: [10, 12],
|
||||||
textStyle: { color: '#ffffff', fontSize: 12 },
|
textStyle: { color: '#ffffff', fontSize: 12 },
|
||||||
valueFormatter: (value) => `${value}`,
|
valueFormatter: (value) => String(value ?? ''),
|
||||||
},
|
},
|
||||||
xAxis: {
|
xAxis: {
|
||||||
type: 'category',
|
type: 'category',
|
||||||
@@ -179,20 +183,30 @@ const chartOption = computed<EChartsOption>(() => ({
|
|||||||
],
|
],
|
||||||
}))
|
}))
|
||||||
|
|
||||||
// 模块固定配色,保证每个模块颜色不同
|
// 模块固定配色,按顺序循环分配颜色(与后端 OP_ORDER 一致:数据处理/模型训练/模型评测/模型推理)
|
||||||
const OPERATION_COLORS = ['#4f46e5', '#10b981', '#f59e0b', '#3b82f6', '#ec4899', '#8b5cf6', '#ef4444', '#14b8a6']
|
const OPERATION_COLORS = ['#4f46e5', '#10b981', '#f59e0b', '#3b82f6']
|
||||||
const operationChartOption = computed<EChartsOption>(() => {
|
const operationChartOption = computed<EChartsOption>(() => {
|
||||||
const items = operationDistribution.value
|
const items = operationDistribution.value
|
||||||
const total = items.reduce((s, d) => s + (d.value || 0), 0)
|
const total = items.reduce((s, d) => s + (d.value || 0), 0)
|
||||||
// 完全没有操作数据时,用等分灰色占位扇区,保证 6 个模块都可见
|
// 按数据项顺序显式分配颜色,避免依赖 name 匹配或全局 color 数组;
|
||||||
const data =
|
// value=0 的项给一个极小值(0.001)让扇区可见,从而显示各自颜色,
|
||||||
total > 0
|
// 但占比几乎为 0 不影响有数据项的百分比展示。
|
||||||
? items.map((d) => ({ value: d.value || 0, name: d.name }))
|
const data = items.map((d, idx) => {
|
||||||
: items.map((d) => ({ value: 1, name: d.name, itemStyle: { color: '#e2e8f0' } }))
|
const raw = d.value || 0
|
||||||
|
return {
|
||||||
|
value: total > 0 ? (raw > 0 ? raw : 0.001) : 1,
|
||||||
|
name: d.name,
|
||||||
|
itemStyle: {
|
||||||
|
color: OPERATION_COLORS[idx % OPERATION_COLORS.length] || '#94a3b8',
|
||||||
|
borderRadius: 6,
|
||||||
|
borderColor: '#fff',
|
||||||
|
borderWidth: 2,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
})
|
||||||
return {
|
return {
|
||||||
animationDuration: 500,
|
animationDuration: 500,
|
||||||
tooltip: { trigger: 'item', formatter: '{b}: {c} ({d}%)' },
|
tooltip: { trigger: 'item', formatter: '{b}: {c} ({d}%)' },
|
||||||
color: OPERATION_COLORS,
|
|
||||||
legend: {
|
legend: {
|
||||||
type: 'scroll',
|
type: 'scroll',
|
||||||
bottom: 0,
|
bottom: 0,
|
||||||
@@ -207,11 +221,6 @@ const operationChartOption = computed<EChartsOption>(() => {
|
|||||||
radius: ['38%', '60%'],
|
radius: ['38%', '60%'],
|
||||||
center: ['50%', '42%'],
|
center: ['50%', '42%'],
|
||||||
avoidLabelOverlap: true,
|
avoidLabelOverlap: true,
|
||||||
itemStyle: {
|
|
||||||
borderRadius: 6,
|
|
||||||
borderColor: '#fff',
|
|
||||||
borderWidth: 2,
|
|
||||||
},
|
|
||||||
label: {
|
label: {
|
||||||
show: true,
|
show: true,
|
||||||
position: 'outside',
|
position: 'outside',
|
||||||
@@ -235,17 +244,23 @@ const operationChartOption = computed<EChartsOption>(() => {
|
|||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
||||||
const loginDurationChartOption = computed<EChartsOption>(() => ({
|
const loginDurationChartOption = computed<EChartsOption>(() => {
|
||||||
|
const stats = loginDurationStats.value
|
||||||
|
const data = stats.map((u) => ({ name: u.username, value: u.duration }))
|
||||||
|
const maxVal = data.length
|
||||||
|
? Math.max(10, Math.ceil(Math.max(...data.map((d) => d.value), 0) * 1.15 / 10) * 10)
|
||||||
|
: 10
|
||||||
|
return {
|
||||||
animationDuration: 500,
|
animationDuration: 500,
|
||||||
grid: { top: 8, right: 12, bottom: 6, left: 8, containLabel: true },
|
grid: { top: 8, right: 12, bottom: 6, left: 8, containLabel: true },
|
||||||
tooltip: {
|
tooltip: {
|
||||||
trigger: 'axis',
|
trigger: 'axis',
|
||||||
axisPointer: { type: 'shadow' },
|
axisPointer: { type: 'shadow' },
|
||||||
valueFormatter: (value) => `${value} 小时`,
|
valueFormatter: (value: unknown) => String(Number(Array.isArray(value) ? value[0] : value) || 0) + ' 小时',
|
||||||
},
|
},
|
||||||
xAxis: {
|
xAxis: {
|
||||||
type: 'value',
|
type: 'value',
|
||||||
max: Math.max(10, Math.ceil(Math.max(...loginDurationStats.value.map((user) => user.duration), 0) * 1.15 / 10) * 10),
|
max: maxVal,
|
||||||
splitNumber: 4,
|
splitNumber: 4,
|
||||||
axisLabel: { color: '#94a3b8', fontSize: 11, formatter: '{value}h' },
|
axisLabel: { color: '#94a3b8', fontSize: 11, formatter: '{value}h' },
|
||||||
axisLine: { show: false },
|
axisLine: { show: false },
|
||||||
@@ -255,8 +270,13 @@ const loginDurationChartOption = computed<EChartsOption>(() => ({
|
|||||||
yAxis: {
|
yAxis: {
|
||||||
type: 'category',
|
type: 'category',
|
||||||
inverse: true,
|
inverse: true,
|
||||||
data: loginDurationStats.value.map((user) => user.username),
|
data: data.map((d) => d.name),
|
||||||
axisLabel: { color: '#475569', fontSize: 12 },
|
axisLabel: {
|
||||||
|
color: '#1f2937',
|
||||||
|
fontSize: 14,
|
||||||
|
fontFamily: '"PingFang SC", "Microsoft YaHei", system-ui, -apple-system, sans-serif',
|
||||||
|
margin: 12,
|
||||||
|
},
|
||||||
axisLine: { show: false },
|
axisLine: { show: false },
|
||||||
axisTick: { show: false },
|
axisTick: { show: false },
|
||||||
},
|
},
|
||||||
@@ -264,14 +284,14 @@ const loginDurationChartOption = computed<EChartsOption>(() => ({
|
|||||||
{
|
{
|
||||||
name: '登录时长',
|
name: '登录时长',
|
||||||
type: 'bar',
|
type: 'bar',
|
||||||
data: loginDurationStats.value.map((user) => user.duration),
|
data: data.map((d) => d.value),
|
||||||
barMaxWidth: 18,
|
barMaxWidth: 18,
|
||||||
barCategoryGap: '34%',
|
barCategoryGap: '34%',
|
||||||
itemStyle: { color: '#4f46e5', borderRadius: [0, 4, 4, 0] },
|
itemStyle: { color: '#4f46e5', borderRadius: [0, 4, 4, 0] },
|
||||||
label: { show: true, position: 'insideRight', distance: 6, color: '#ffffff', fontSize: 11, formatter: '{c} 小时' },
|
|
||||||
},
|
},
|
||||||
],
|
],
|
||||||
}))
|
}
|
||||||
|
})
|
||||||
|
|
||||||
const roleTagType: Record<string, 'danger' | 'primary' | 'info'> = {
|
const roleTagType: Record<string, 'danger' | 'primary' | 'info'> = {
|
||||||
'超级管理员': 'danger',
|
'超级管理员': 'danger',
|
||||||
@@ -405,10 +425,9 @@ function viewTask(task: DashboardTask) {
|
|||||||
|
|
||||||
<section class="stat-card" aria-labelledby="login-dur-title">
|
<section class="stat-card" aria-labelledby="login-dur-title">
|
||||||
<h2 id="login-dur-title" class="section-title">登录时长排行 (本月)</h2>
|
<h2 id="login-dur-title" class="section-title">登录时长排行 (本月)</h2>
|
||||||
<div v-if="loginDurationStats.length" class="chart-container">
|
<div class="chart-container">
|
||||||
<VChart class="duration-chart" :option="loginDurationChartOption" autoresize />
|
<VChart class="duration-chart" :option="loginDurationChartOption" autoresize />
|
||||||
</div>
|
</div>
|
||||||
<div v-else class="empty-hint">暂无数据</div>
|
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
<section class="stat-card" aria-labelledby="recent-login-title">
|
<section class="stat-card" aria-labelledby="recent-login-title">
|
||||||
@@ -723,10 +742,11 @@ function viewTask(task: DashboardTask) {
|
|||||||
|
|
||||||
.service-table {
|
.service-table {
|
||||||
display: grid;
|
display: grid;
|
||||||
grid-template-rows: 36px repeat(4, minmax(48px, 1fr));
|
grid-auto-rows: minmax(44px, auto);
|
||||||
flex: 1 1 auto;
|
flex: 1 1 auto;
|
||||||
margin-top: 12px;
|
margin-top: 12px;
|
||||||
min-height: 0;
|
min-height: 0;
|
||||||
|
overflow-y: auto;
|
||||||
}
|
}
|
||||||
|
|
||||||
.service-row {
|
.service-row {
|
||||||
@@ -920,7 +940,7 @@ function viewTask(task: DashboardTask) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
.service-table {
|
.service-table {
|
||||||
grid-template-rows: 32px repeat(4, minmax(40px, 1fr));
|
grid-auto-rows: minmax(38px, auto);
|
||||||
margin-top: 8px;
|
margin-top: 8px;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -50,6 +50,20 @@ const rules: FormRules = {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/** 处理文件选择(替换模式:新文件覆盖旧文件) */
|
/** 处理文件选择(替换模式:新文件覆盖旧文件) */
|
||||||
|
function parseDatasetRecordValues(text: string, fileName: string): unknown[] {
|
||||||
|
const content = text.trim()
|
||||||
|
if (!content) return []
|
||||||
|
if (fileName.toLowerCase().endsWith('.json')) {
|
||||||
|
const parsed = JSON.parse(content)
|
||||||
|
return Array.isArray(parsed) ? parsed : [parsed]
|
||||||
|
}
|
||||||
|
return content
|
||||||
|
.split('\n')
|
||||||
|
.map((line) => line.trim())
|
||||||
|
.filter(Boolean)
|
||||||
|
.map((line) => JSON.parse(line))
|
||||||
|
}
|
||||||
|
|
||||||
async function handleFileChange(uploadFile: UploadFile) {
|
async function handleFileChange(uploadFile: UploadFile) {
|
||||||
const raw = uploadFile.raw
|
const raw = uploadFile.raw
|
||||||
if (!raw) return
|
if (!raw) return
|
||||||
@@ -70,25 +84,19 @@ async function handleFileChange(uploadFile: UploadFile) {
|
|||||||
async function analyzeFile(file: File) {
|
async function analyzeFile(file: File) {
|
||||||
try {
|
try {
|
||||||
const text = await file.text()
|
const text = await file.text()
|
||||||
const lines = text.trim().split('\n').filter(Boolean)
|
const records = parseDatasetRecordValues(text, file.name)
|
||||||
fileCount.value = lines.length
|
fileCount.value = records.length
|
||||||
|
|
||||||
// Alpaca 格式校验:每行 JSON 须含 instruction 字段
|
// Alpaca 格式校验:每行 JSON 须含 instruction 字段
|
||||||
let validCount = 0
|
const validCount = records.filter(
|
||||||
for (const line of lines) {
|
(obj) => obj && typeof obj === 'object' && 'instruction' in obj,
|
||||||
try {
|
).length
|
||||||
const obj = JSON.parse(line)
|
if (validCount > 0 && validCount === records.length) {
|
||||||
if (obj.instruction !== undefined) validCount++
|
|
||||||
} catch {
|
|
||||||
// 非 JSON 行(如纯 JSONL 多行结构)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if (validCount > 0 && validCount === lines.length) {
|
|
||||||
formatValid.value = true
|
formatValid.value = true
|
||||||
formatMessage.value = `符合 Alpaca 格式(含 instruction 字段)`
|
formatMessage.value = `符合 Alpaca 格式(含 instruction 字段)`
|
||||||
} else if (validCount > 0) {
|
} else if (validCount > 0) {
|
||||||
formatValid.value = true
|
formatValid.value = true
|
||||||
formatMessage.value = `部分符合 Alpaca 格式(${validCount}/${lines.length})`
|
formatMessage.value = `部分符合 Alpaca 格式(${validCount}/${records.length})`
|
||||||
} else {
|
} else {
|
||||||
formatValid.value = false
|
formatValid.value = false
|
||||||
formatMessage.value = '未检测到标准 Alpaca 格式(缺少 instruction 字段),仍可上传'
|
formatMessage.value = '未检测到标准 Alpaca 格式(缺少 instruction 字段),仍可上传'
|
||||||
|
|||||||
@@ -79,7 +79,9 @@ async function loadEditData() {
|
|||||||
async function loadModels() {
|
async function loadModels() {
|
||||||
try {
|
try {
|
||||||
const all = (await getModelList()) || []
|
const all = (await getModelList()) || []
|
||||||
evalModels.value = all.filter((m) => m.purpose === 'evaluation')
|
evalModels.value = all.filter(
|
||||||
|
(m) => m.purpose === 'evaluation' || (m.model_source === 'api' && !!m.api_url),
|
||||||
|
)
|
||||||
} catch {
|
} catch {
|
||||||
evalModels.value = []
|
evalModels.value = []
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
<script setup lang="ts">
|
<script setup lang="ts">
|
||||||
import { onMounted, ref, watch } from 'vue'
|
import { onMounted, ref, watch } from 'vue'
|
||||||
import { useRouter } from 'vue-router'
|
import { useRouter } from 'vue-router'
|
||||||
import { ElMessage } from 'element-plus'
|
import { ElMessage } from 'element-plus'
|
||||||
@@ -10,7 +10,7 @@ import StartEvalStep from './create/StartEvalStep.vue'
|
|||||||
import { createDimension, startEval } from '@/api/modules/eval'
|
import { createDimension, startEval } from '@/api/modules/eval'
|
||||||
import { getTrainedModels, getModelList } from '@/api/modules/model'
|
import { getTrainedModels, getModelList } from '@/api/modules/model'
|
||||||
import { getDatasetList } from '@/api/modules/dataset'
|
import { getDatasetList } from '@/api/modules/dataset'
|
||||||
import { getSystemInfo } from '@/api/modules/system'
|
import { getComputeGpus } from '@/api/modules/compute'
|
||||||
import type { DatasetItem, Dimension, GpuInfo, ModelItem, TrainedModel } from '@/types'
|
import type { DatasetItem, Dimension, GpuInfo, ModelItem, TrainedModel } from '@/types'
|
||||||
|
|
||||||
type StepExposed = { validate: () => Promise<boolean> }
|
type StepExposed = { validate: () => Promise<boolean> }
|
||||||
@@ -82,17 +82,21 @@ async function loadData() {
|
|||||||
const results = await Promise.allSettled([
|
const results = await Promise.allSettled([
|
||||||
getTrainedModels(),
|
getTrainedModels(),
|
||||||
getDatasetList(),
|
getDatasetList(),
|
||||||
getSystemInfo(),
|
|
||||||
getModelList(),
|
getModelList(),
|
||||||
|
getComputeGpus(),
|
||||||
])
|
])
|
||||||
|
|
||||||
if (results[0].status === 'fulfilled') trainedModels.value = results[0].value?.models || []
|
if (results[0].status === 'fulfilled') trainedModels.value = results[0].value?.models || []
|
||||||
if (results[1].status === 'fulfilled') {
|
if (results[1].status === 'fulfilled') {
|
||||||
evalDatasets.value = (results[1].value || []).filter((dataset) => dataset.type === 'eval')
|
evalDatasets.value = (results[1].value || []).filter((dataset) => dataset.type === 'eval')
|
||||||
}
|
}
|
||||||
if (results[2].status === 'fulfilled') gpus.value = results[2].value?.gpu || []
|
if (results[2].status === 'fulfilled') {
|
||||||
|
evalModels.value = (results[2].value || []).filter(
|
||||||
|
(model) => model.purpose === 'evaluation' || (model.model_source === 'api' && !!model.api_url),
|
||||||
|
)
|
||||||
|
}
|
||||||
if (results[3].status === 'fulfilled') {
|
if (results[3].status === 'fulfilled') {
|
||||||
evalModels.value = (results[3].value || []).filter((model) => model.purpose === 'evaluation')
|
gpus.value = ((results[3].value || []) as unknown as GpuInfo[]).filter((g) => g.status === 'idle')
|
||||||
}
|
}
|
||||||
|
|
||||||
const failedCount = results.filter((result) => result.status === 'rejected').length
|
const failedCount = results.filter((result) => result.status === 'rejected').length
|
||||||
@@ -139,11 +143,15 @@ async function handleSubmit() {
|
|||||||
submitting.value = true
|
submitting.value = true
|
||||||
try {
|
try {
|
||||||
const dimensionId = await resolveDimensionId()
|
const dimensionId = await resolveDimensionId()
|
||||||
await startEval({
|
// GPU 选择为「节点:GPU序号」复合值,解析出节点与 GPU 序号,
|
||||||
|
// 多算力节点时必须把节点信息传给后端,否则会派发到错误的算力节点
|
||||||
|
const [gpuNodeId, gpuIndex] = String(taskForm.value.gpu_id).split(':')
|
||||||
|
const evalResult: any = await startEval({
|
||||||
eval_task_name: taskForm.value.eval_task_name,
|
eval_task_name: taskForm.value.eval_task_name,
|
||||||
eval_type: 'custom',
|
eval_type: 'custom',
|
||||||
model_id: taskForm.value.model_id,
|
model_id: taskForm.value.model_id,
|
||||||
gpu_id: taskForm.value.gpu_id,
|
gpu_id: Number(gpuIndex) || 0,
|
||||||
|
compute_node_id: gpuNodeId || '',
|
||||||
dataset_id: taskForm.value.data_source === 'dataset' ? taskForm.value.dataset_id : '',
|
dataset_id: taskForm.value.data_source === 'dataset' ? taskForm.value.dataset_id : '',
|
||||||
dimension_id: dimensionId,
|
dimension_id: dimensionId,
|
||||||
data_source: taskForm.value.data_source,
|
data_source: taskForm.value.data_source,
|
||||||
@@ -163,6 +171,10 @@ async function handleSubmit() {
|
|||||||
output_precision: basicMetricForm.value.output_precision,
|
output_precision: basicMetricForm.value.output_precision,
|
||||||
},
|
},
|
||||||
})
|
})
|
||||||
|
if (evalResult?.status === 'failed' || evalResult?.error) {
|
||||||
|
ElMessage.error(`评测启动失败:${evalResult?.error || '请检查算力节点与模型路径'}`)
|
||||||
|
return
|
||||||
|
}
|
||||||
ElMessage.success('评测任务已创建并启动')
|
ElMessage.success('评测任务已创建并启动')
|
||||||
router.push('/model-eval')
|
router.push('/model-eval')
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
|
|||||||
@@ -1,9 +1,10 @@
|
|||||||
<script setup lang="ts">
|
<script setup lang="ts">
|
||||||
import { computed, onMounted, ref } from 'vue'
|
import { computed, onMounted, onUnmounted, ref } from 'vue'
|
||||||
import { useRoute } from 'vue-router'
|
import { useRoute } from 'vue-router'
|
||||||
import PageCard from '@/components/PageCard.vue'
|
import PageCard from '@/components/PageCard.vue'
|
||||||
import ModelStatusTag from '@/components/ModelStatusTag.vue'
|
import ModelStatusTag from '@/components/ModelStatusTag.vue'
|
||||||
import { getEvalDetail } from '@/api/modules/eval'
|
import { getEvalDetail } from '@/api/modules/eval'
|
||||||
|
import { usePolling } from '@/composables/usePolling'
|
||||||
import type { EvalSampleResult, EvalTaskDetail } from '@/types'
|
import type { EvalSampleResult, EvalTaskDetail } from '@/types'
|
||||||
|
|
||||||
const route = useRoute()
|
const route = useRoute()
|
||||||
@@ -16,6 +17,7 @@ const keyword = ref('')
|
|||||||
const judgementFilter = ref('')
|
const judgementFilter = ref('')
|
||||||
const currentPage = ref(1)
|
const currentPage = ref(1)
|
||||||
const pageSize = ref(10)
|
const pageSize = ref(10)
|
||||||
|
const ACTIVE_STATUSES = new Set(['pending', 'queued', 'running'])
|
||||||
|
|
||||||
const filteredSamples = computed(() => {
|
const filteredSamples = computed(() => {
|
||||||
const normalizedKeyword = keyword.value.trim().toLowerCase()
|
const normalizedKeyword = keyword.value.trim().toLowerCase()
|
||||||
@@ -48,6 +50,8 @@ const passRate = computed(() => {
|
|||||||
})
|
})
|
||||||
|
|
||||||
const overallScore = computed(() => formatScore(detail.value?.overall_score, detail.value?.overall_score_max))
|
const overallScore = computed(() => formatScore(detail.value?.overall_score, detail.value?.overall_score_max))
|
||||||
|
const displayModelName = computed(() => detail.value?.model_name || String(detail.value?.model_id || '-'))
|
||||||
|
const displayMetric = computed(() => detail.value?.metric_label || detail.value?.metric || '-')
|
||||||
|
|
||||||
function formatDateTime(value?: string) {
|
function formatDateTime(value?: string) {
|
||||||
if (!value) return '-'
|
if (!value) return '-'
|
||||||
@@ -74,8 +78,8 @@ function resetPage() {
|
|||||||
currentPage.value = 1
|
currentPage.value = 1
|
||||||
}
|
}
|
||||||
|
|
||||||
async function loadDetail() {
|
async function loadDetail(options: { silent?: boolean } = {}) {
|
||||||
loading.value = true
|
if (!options.silent) loading.value = true
|
||||||
loadError.value = ''
|
loadError.value = ''
|
||||||
try {
|
try {
|
||||||
detail.value = await getEvalDetail(taskId)
|
detail.value = await getEvalDetail(taskId)
|
||||||
@@ -83,11 +87,29 @@ async function loadDetail() {
|
|||||||
detail.value = null
|
detail.value = null
|
||||||
loadError.value = '评测详情加载失败,请稍后重试。'
|
loadError.value = '评测详情加载失败,请稍后重试。'
|
||||||
} finally {
|
} finally {
|
||||||
loading.value = false
|
if (!options.silent) loading.value = false
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
onMounted(loadDetail)
|
const { start: startPolling, stop: stopPolling } = usePolling(
|
||||||
|
async () => {
|
||||||
|
await loadDetail({ silent: true })
|
||||||
|
if (!ACTIVE_STATUSES.has(String(detail.value?.status || ''))) {
|
||||||
|
stopPolling()
|
||||||
|
}
|
||||||
|
},
|
||||||
|
5000,
|
||||||
|
{ immediate: false },
|
||||||
|
)
|
||||||
|
|
||||||
|
onMounted(async () => {
|
||||||
|
await loadDetail()
|
||||||
|
if (ACTIVE_STATUSES.has(String(detail.value?.status || ''))) {
|
||||||
|
startPolling()
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
onUnmounted(stopPolling)
|
||||||
</script>
|
</script>
|
||||||
|
|
||||||
<template>
|
<template>
|
||||||
@@ -101,9 +123,9 @@ onMounted(loadDetail)
|
|||||||
</div>
|
</div>
|
||||||
<dl class="task-meta">
|
<dl class="task-meta">
|
||||||
<div><dt>任务 ID</dt><dd>{{ detail?.id || taskId }}</dd></div>
|
<div><dt>任务 ID</dt><dd>{{ detail?.id || taskId }}</dd></div>
|
||||||
<div><dt>评测模型</dt><dd>{{ detail?.model_name || '-' }}</dd></div>
|
<div><dt>评测模型</dt><dd>{{ displayModelName }}</dd></div>
|
||||||
<div><dt>测试集</dt><dd>{{ detail?.dataset || '-' }}</dd></div>
|
<div><dt>测试集</dt><dd>{{ detail?.dataset || '-' }}</dd></div>
|
||||||
<div><dt>评测指标</dt><dd>{{ detail?.metric || '-' }}</dd></div>
|
<div><dt>评测指标</dt><dd>{{ displayMetric }}</dd></div>
|
||||||
</dl>
|
</dl>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
@@ -113,7 +135,7 @@ onMounted(loadDetail)
|
|||||||
<i class="fa fa-exclamation-circle" aria-hidden="true" />
|
<i class="fa fa-exclamation-circle" aria-hidden="true" />
|
||||||
<h2>无法加载评测详情</h2>
|
<h2>无法加载评测详情</h2>
|
||||||
<p>{{ loadError }}</p>
|
<p>{{ loadError }}</p>
|
||||||
<el-button type="primary" @click="loadDetail">重新加载</el-button>
|
<el-button type="primary" @click="() => loadDetail()">重新加载</el-button>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<template v-else-if="detail">
|
<template v-else-if="detail">
|
||||||
@@ -121,7 +143,7 @@ onMounted(loadDetail)
|
|||||||
<div class="overview-item score-hero">
|
<div class="overview-item score-hero">
|
||||||
<span>综合得分</span>
|
<span>综合得分</span>
|
||||||
<strong>{{ overallScore }}</strong>
|
<strong>{{ overallScore }}</strong>
|
||||||
<small>大模型综合评分</small>
|
<small>模型综合评分</small>
|
||||||
</div>
|
</div>
|
||||||
<div class="overview-item">
|
<div class="overview-item">
|
||||||
<span>样本通过率</span>
|
<span>样本通过率</span>
|
||||||
@@ -144,7 +166,7 @@ onMounted(loadDetail)
|
|||||||
<div class="review-copy">
|
<div class="review-copy">
|
||||||
<div class="section-heading">
|
<div class="section-heading">
|
||||||
<div>
|
<div>
|
||||||
<h2 id="overall-review-title">大模型综合评价</h2>
|
<h2 id="overall-review-title">综合评价</h2>
|
||||||
<p>基于全部已评测样本生成的总体结论</p>
|
<p>基于全部已评测样本生成的总体结论</p>
|
||||||
</div>
|
</div>
|
||||||
<el-tag v-if="detail.evaluator_model" type="primary" size="small">
|
<el-tag v-if="detail.evaluator_model" type="primary" size="small">
|
||||||
@@ -152,7 +174,7 @@ onMounted(loadDetail)
|
|||||||
</el-tag>
|
</el-tag>
|
||||||
</div>
|
</div>
|
||||||
<p class="review-text">
|
<p class="review-text">
|
||||||
{{ detail.overall_evaluation || (detail.status === 'running' ? '评测仍在进行,综合评价将在样本评分完成后生成。' : '暂无综合评价。') }}
|
{{ detail.overall_evaluation || (detail.status === 'running' ? '评测正在进行,综合评价将在样本完成后生成。' : '暂无综合评价。') }}
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
<div class="suggestion-block">
|
<div class="suggestion-block">
|
||||||
@@ -170,8 +192,8 @@ onMounted(loadDetail)
|
|||||||
<section v-if="detail.dimension_summary?.length" class="dimension-summary" aria-labelledby="dimension-title">
|
<section v-if="detail.dimension_summary?.length" class="dimension-summary" aria-labelledby="dimension-title">
|
||||||
<div class="section-heading compact-heading">
|
<div class="section-heading compact-heading">
|
||||||
<div>
|
<div>
|
||||||
<h2 id="dimension-title">维度表现</h2>
|
<h2 id="dimension-title">指标表现</h2>
|
||||||
<p>查看各评测维度的得分与样本通过率</p>
|
<p>查看各评测指标的得分与通过率</p>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
<div class="dimension-grid">
|
<div class="dimension-grid">
|
||||||
@@ -192,22 +214,10 @@ onMounted(loadDetail)
|
|||||||
<p>共 {{ filteredSamples.length }} 条结果,展开行可查看评分依据与子维度分数</p>
|
<p>共 {{ filteredSamples.length }} 条结果,展开行可查看评分依据与子维度分数</p>
|
||||||
</div>
|
</div>
|
||||||
<div class="sample-filters" aria-label="样本筛选">
|
<div class="sample-filters" aria-label="样本筛选">
|
||||||
<el-input
|
<el-input v-model="keyword" clearable placeholder="搜索问题、回答或评价" aria-label="搜索样本" @input="resetPage">
|
||||||
v-model="keyword"
|
|
||||||
clearable
|
|
||||||
placeholder="搜索问题、回答或评价"
|
|
||||||
aria-label="搜索样本"
|
|
||||||
@input="resetPage"
|
|
||||||
>
|
|
||||||
<template #prefix><i class="fa fa-search" aria-hidden="true" /></template>
|
<template #prefix><i class="fa fa-search" aria-hidden="true" /></template>
|
||||||
</el-input>
|
</el-input>
|
||||||
<el-select
|
<el-select v-model="judgementFilter" clearable placeholder="全部判定" aria-label="按判定筛选" @change="resetPage">
|
||||||
v-model="judgementFilter"
|
|
||||||
clearable
|
|
||||||
placeholder="全部判定"
|
|
||||||
aria-label="按判定筛选"
|
|
||||||
@change="resetPage"
|
|
||||||
>
|
|
||||||
<el-option label="正确" value="正确" />
|
<el-option label="正确" value="正确" />
|
||||||
<el-option label="部分正确" value="部分正确" />
|
<el-option label="部分正确" value="部分正确" />
|
||||||
<el-option label="错误" value="错误" />
|
<el-option label="错误" value="错误" />
|
||||||
@@ -215,18 +225,12 @@ onMounted(loadDetail)
|
|||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<el-table
|
<el-table v-if="filteredSamples.length" class="sample-results-table" :data="paginatedSamples" row-key="id" table-layout="fixed">
|
||||||
v-if="filteredSamples.length"
|
|
||||||
class="sample-results-table"
|
|
||||||
:data="paginatedSamples"
|
|
||||||
row-key="id"
|
|
||||||
table-layout="fixed"
|
|
||||||
>
|
|
||||||
<el-table-column type="expand" width="48">
|
<el-table-column type="expand" width="48">
|
||||||
<template #default="{ row }">
|
<template #default="{ row }">
|
||||||
<div class="sample-detail-grid">
|
<div class="sample-detail-grid">
|
||||||
<div class="evaluation-reason">
|
<div class="evaluation-reason">
|
||||||
<span>大模型评分依据</span>
|
<span>评分依据</span>
|
||||||
<p>{{ row.evaluation_reason || '暂无评分依据。' }}</p>
|
<p>{{ row.evaluation_reason || '暂无评分依据。' }}</p>
|
||||||
</div>
|
</div>
|
||||||
<div v-if="row.error_type" class="error-type">
|
<div v-if="row.error_type" class="error-type">
|
||||||
@@ -255,9 +259,7 @@ onMounted(loadDetail)
|
|||||||
<template #default="{ row }"><p class="cell-copy">{{ row.model_output || '等待生成' }}</p></template>
|
<template #default="{ row }"><p class="cell-copy">{{ row.model_output || '等待生成' }}</p></template>
|
||||||
</el-table-column>
|
</el-table-column>
|
||||||
<el-table-column label="得分" width="90" align="center">
|
<el-table-column label="得分" width="90" align="center">
|
||||||
<template #default="{ row }">
|
<template #default="{ row }"><span class="sample-score">{{ formatScore(row.score, row.max_score) }}</span></template>
|
||||||
<span class="sample-score">{{ formatScore(row.score, row.max_score) }}</span>
|
|
||||||
</template>
|
|
||||||
</el-table-column>
|
</el-table-column>
|
||||||
<el-table-column label="判定" width="96" align="center">
|
<el-table-column label="判定" width="96" align="center">
|
||||||
<template #default="{ row }">
|
<template #default="{ row }">
|
||||||
@@ -274,21 +276,11 @@ onMounted(loadDetail)
|
|||||||
<p>{{ detail.status === 'running' ? '任务正在运行,结果生成后会显示在这里。' : '请调整筛选条件或稍后重试。' }}</p>
|
<p>{{ detail.status === 'running' ? '任务正在运行,结果生成后会显示在这里。' : '请调整筛选条件或稍后重试。' }}</p>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<el-pagination
|
<el-pagination v-if="filteredSamples.length > pageSize" v-model:current-page="currentPage" v-model:page-size="pageSize" background layout="total, sizes, prev, pager, next" :page-sizes="[10, 20, 50]" :total="filteredSamples.length" aria-label="样本结果分页" />
|
||||||
v-if="filteredSamples.length > pageSize"
|
|
||||||
v-model:current-page="currentPage"
|
|
||||||
v-model:page-size="pageSize"
|
|
||||||
background
|
|
||||||
layout="total, sizes, prev, pager, next"
|
|
||||||
:page-sizes="[10, 20, 50]"
|
|
||||||
:total="filteredSamples.length"
|
|
||||||
aria-label="样本结果分页"
|
|
||||||
/>
|
|
||||||
</section>
|
</section>
|
||||||
</template>
|
</template>
|
||||||
</PageCard>
|
</PageCard>
|
||||||
</template>
|
</template>
|
||||||
|
|
||||||
<style scoped lang="scss">
|
<style scoped lang="scss">
|
||||||
.eval-detail-page {
|
.eval-detail-page {
|
||||||
min-width: 0;
|
min-width: 0;
|
||||||
@@ -747,3 +739,6 @@ onMounted(loadDetail)
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
</style>
|
</style>
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1,9 +1,10 @@
|
|||||||
<script setup lang="ts">
|
<script setup lang="ts">
|
||||||
import { ref, onMounted } from 'vue'
|
import { ref, onMounted, onUnmounted } from 'vue'
|
||||||
import { useRouter } from 'vue-router'
|
import { useRouter } from 'vue-router'
|
||||||
import { ElMessage, ElMessageBox } from 'element-plus'
|
import { ElMessage, ElMessageBox } from 'element-plus'
|
||||||
import DataTablePage from '@/components/DataTablePage.vue'
|
import DataTablePage from '@/components/DataTablePage.vue'
|
||||||
import ModelStatusTag from '@/components/ModelStatusTag.vue'
|
import ModelStatusTag from '@/components/ModelStatusTag.vue'
|
||||||
|
import { usePolling } from '@/composables/usePolling'
|
||||||
import {
|
import {
|
||||||
getEvalList,
|
getEvalList,
|
||||||
deleteEval,
|
deleteEval,
|
||||||
@@ -23,14 +24,16 @@ const leaderboard = ref([
|
|||||||
{ rank: 3, name: 'Qwen-Max', score: 85.3 },
|
{ rank: 3, name: 'Qwen-Max', score: 85.3 },
|
||||||
])
|
])
|
||||||
|
|
||||||
async function loadEvalList() {
|
const ACTIVE_STATUSES = new Set(['pending', 'queued', 'running'])
|
||||||
evalLoading.value = true
|
|
||||||
|
async function loadEvalList(options: { silent?: boolean } = {}) {
|
||||||
|
if (!options.silent) evalLoading.value = true
|
||||||
try {
|
try {
|
||||||
evalList.value = (await getEvalList()) || []
|
evalList.value = (await getEvalList()) || []
|
||||||
} catch {
|
} catch {
|
||||||
evalList.value = []
|
evalList.value = []
|
||||||
} finally {
|
} finally {
|
||||||
evalLoading.value = false
|
if (!options.silent) evalLoading.value = false
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -54,8 +57,34 @@ function handleViewDetail(row: any) {
|
|||||||
router.push({ name: 'model-eval-detail', params: { id: row.id } })
|
router.push({ name: 'model-eval-detail', params: { id: row.id } })
|
||||||
}
|
}
|
||||||
|
|
||||||
onMounted(() => {
|
function displayModelName(row: Partial<EvalTask>) {
|
||||||
loadEvalList()
|
return row.model_name || String(row.model_id || '-')
|
||||||
|
}
|
||||||
|
|
||||||
|
function displayMetric(row: Partial<EvalTask>) {
|
||||||
|
return row.metric_label || row.metric || '-'
|
||||||
|
}
|
||||||
|
|
||||||
|
const { start: startPolling, stop: stopPolling } = usePolling(
|
||||||
|
async () => {
|
||||||
|
await loadEvalList({ silent: true })
|
||||||
|
if (!evalList.value.some((item) => ACTIVE_STATUSES.has(String(item.status || '')))) {
|
||||||
|
stopPolling()
|
||||||
|
}
|
||||||
|
},
|
||||||
|
5000,
|
||||||
|
{ immediate: false },
|
||||||
|
)
|
||||||
|
|
||||||
|
onMounted(async () => {
|
||||||
|
await loadEvalList()
|
||||||
|
if (evalList.value.some((item) => ACTIVE_STATUSES.has(String(item.status || '')))) {
|
||||||
|
startPolling()
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
onUnmounted(() => {
|
||||||
|
stopPolling()
|
||||||
})
|
})
|
||||||
</script>
|
</script>
|
||||||
|
|
||||||
@@ -81,9 +110,21 @@ onMounted(() => {
|
|||||||
</template>
|
</template>
|
||||||
<template #columns>
|
<template #columns>
|
||||||
<el-table-column label="任务名称" prop="eval_task_name" align="center" />
|
<el-table-column label="任务名称" prop="eval_task_name" align="center" />
|
||||||
<el-table-column label="评测模型" prop="model_name" align="center" />
|
<el-table-column label="评测模型" align="center" min-width="160">
|
||||||
|
<template #default="{ row }">
|
||||||
|
<el-tooltip :content="displayModelName(row)" placement="top" :disabled="displayModelName(row).length < 18">
|
||||||
|
<span class="cell-ellipsis">{{ displayModelName(row) }}</span>
|
||||||
|
</el-tooltip>
|
||||||
|
</template>
|
||||||
|
</el-table-column>
|
||||||
<el-table-column label="数据集" prop="dataset" align="center" />
|
<el-table-column label="数据集" prop="dataset" align="center" />
|
||||||
<el-table-column label="指标" prop="metric" align="center" />
|
<el-table-column label="指标" align="center" min-width="220">
|
||||||
|
<template #default="{ row }">
|
||||||
|
<el-tooltip :content="displayMetric(row)" placement="top" :disabled="displayMetric(row).length < 24">
|
||||||
|
<span class="cell-ellipsis">{{ displayMetric(row) }}</span>
|
||||||
|
</el-tooltip>
|
||||||
|
</template>
|
||||||
|
</el-table-column>
|
||||||
<el-table-column label="评分" prop="score" width="100" align="center" />
|
<el-table-column label="评分" prop="score" width="100" align="center" />
|
||||||
<el-table-column label="状态" width="100" align="center">
|
<el-table-column label="状态" width="100" align="center">
|
||||||
<template #default="{ row }">
|
<template #default="{ row }">
|
||||||
@@ -138,7 +179,7 @@ onMounted(() => {
|
|||||||
min-height: 0;
|
min-height: 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
/* 胶囊切换栏样式 */
|
/* 胶囊切换栏 */
|
||||||
.capsule-tabs {
|
.capsule-tabs {
|
||||||
display: flex;
|
display: flex;
|
||||||
background: #f1f5f9;
|
background: #f1f5f9;
|
||||||
@@ -172,4 +213,13 @@ onMounted(() => {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
.cell-ellipsis {
|
||||||
|
display: inline-block;
|
||||||
|
max-width: 100%;
|
||||||
|
overflow: hidden;
|
||||||
|
text-overflow: ellipsis;
|
||||||
|
vertical-align: middle;
|
||||||
|
white-space: nowrap;
|
||||||
|
}
|
||||||
|
|
||||||
</style>
|
</style>
|
||||||
|
|||||||
@@ -56,9 +56,9 @@ defineExpose({ validate })
|
|||||||
</el-form-item>
|
</el-form-item>
|
||||||
<el-form-item v-if="form.rouge_enabled" label="ROUGE methods">
|
<el-form-item v-if="form.rouge_enabled" label="ROUGE methods">
|
||||||
<el-checkbox-group v-model="form.rouge_methods">
|
<el-checkbox-group v-model="form.rouge_methods">
|
||||||
<el-checkbox value="rouge_1">ROUGE-1</el-checkbox>
|
<el-checkbox value="rouge1">ROUGE-1</el-checkbox>
|
||||||
<el-checkbox value="rouge_2">ROUGE-2</el-checkbox>
|
<el-checkbox value="rouge2">ROUGE-2</el-checkbox>
|
||||||
<el-checkbox value="rouge_l">ROUGE-L</el-checkbox>
|
<el-checkbox value="rougeL">ROUGE-L</el-checkbox>
|
||||||
</el-checkbox-group>
|
</el-checkbox-group>
|
||||||
</el-form-item>
|
</el-form-item>
|
||||||
|
|
||||||
|
|||||||
@@ -103,10 +103,10 @@ defineExpose({ validate })
|
|||||||
<el-form-item label="选择 GPU" prop="gpu_id">
|
<el-form-item label="选择 GPU" prop="gpu_id">
|
||||||
<el-select v-model="form.gpu_id" placeholder="请选择 GPU" style="width: 100%" :loading="loading">
|
<el-select v-model="form.gpu_id" placeholder="请选择 GPU" style="width: 100%" :loading="loading">
|
||||||
<el-option
|
<el-option
|
||||||
v-for="(gpu, index) in gpus"
|
v-for="gpu in gpus"
|
||||||
:key="index"
|
:key="`${gpu.node_id || ''}:${gpu.id ?? 0}`"
|
||||||
:label="`${gpu.name} (GPU ${index})`"
|
:label="`${gpu.node_name || gpu.node_code || '算力节点'} / ${gpu.name} (GPU ${gpu.id ?? 0})`"
|
||||||
:value="index"
|
:value="`${gpu.node_id || ''}:${gpu.id ?? 0}`"
|
||||||
/>
|
/>
|
||||||
</el-select>
|
</el-select>
|
||||||
</el-form-item>
|
</el-form-item>
|
||||||
|
|||||||
@@ -1,4 +1,5 @@
|
|||||||
<script setup lang="ts">
|
<script setup lang="ts">
|
||||||
|
import { computed } from 'vue'
|
||||||
import type { DatasetItem, GpuInfo, ModelItem, TrainedModel } from '@/types'
|
import type { DatasetItem, GpuInfo, ModelItem, TrainedModel } from '@/types'
|
||||||
import type { BasicMetricSetupDraft } from './BasicMetricSetupStep.vue'
|
import type { BasicMetricSetupDraft } from './BasicMetricSetupStep.vue'
|
||||||
import type { EvalRuleSetupDraft } from './EvalRuleSetupStep.vue'
|
import type { EvalRuleSetupDraft } from './EvalRuleSetupStep.vue'
|
||||||
@@ -17,6 +18,15 @@ const props = defineProps<{
|
|||||||
function nameOf<T extends { id: string | number; name?: string }>(items: T[], id: string | number) {
|
function nameOf<T extends { id: string | number; name?: string }>(items: T[], id: string | number) {
|
||||||
return items.find((item) => item.id === id)?.name || String(id || '-')
|
return items.find((item) => item.id === id)?.name || String(id || '-')
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** GPU 选择为「节点:GPU序号」复合值,解析并展示为可读标签 */
|
||||||
|
const gpuLabel = computed(() => {
|
||||||
|
const key = String(props.task.gpu_id || '')
|
||||||
|
const gpu = props.gpus.find((g) => `${g.node_id || ''}:${g.id ?? 0}` === key)
|
||||||
|
if (gpu) return `${gpu.node_name || gpu.node_code || '算力节点'} / GPU ${gpu.id ?? 0}`
|
||||||
|
const [nodeId, idx] = key.split(':')
|
||||||
|
return nodeId ? `节点 ${nodeId} / GPU ${idx || 0}` : `GPU ${key || 0}`
|
||||||
|
})
|
||||||
</script>
|
</script>
|
||||||
|
|
||||||
<template>
|
<template>
|
||||||
@@ -24,7 +34,7 @@ function nameOf<T extends { id: string | number; name?: string }>(items: T[], id
|
|||||||
<el-descriptions :column="2" border>
|
<el-descriptions :column="2" border>
|
||||||
<el-descriptions-item label="Task">{{ props.task.eval_task_name || '-' }}</el-descriptions-item>
|
<el-descriptions-item label="Task">{{ props.task.eval_task_name || '-' }}</el-descriptions-item>
|
||||||
<el-descriptions-item label="Model">{{ nameOf(props.trainedModels, props.task.model_id) }}</el-descriptions-item>
|
<el-descriptions-item label="Model">{{ nameOf(props.trainedModels, props.task.model_id) }}</el-descriptions-item>
|
||||||
<el-descriptions-item label="GPU">GPU {{ props.task.gpu_id || 0 }}</el-descriptions-item>
|
<el-descriptions-item label="GPU">{{ gpuLabel }}</el-descriptions-item>
|
||||||
<el-descriptions-item label="Dataset">
|
<el-descriptions-item label="Dataset">
|
||||||
{{ props.task.data_source === 'dataset' ? nameOf(props.evalDatasets, props.task.dataset_id) : 'Inference results' }}
|
{{ props.task.data_source === 'dataset' ? nameOf(props.evalDatasets, props.task.dataset_id) : 'Inference results' }}
|
||||||
</el-descriptions-item>
|
</el-descriptions-item>
|
||||||
|
|||||||
@@ -37,7 +37,7 @@ const models = ref<ModelItem[]>([])
|
|||||||
const datasets = ref<DatasetItem[]>([])
|
const datasets = ref<DatasetItem[]>([])
|
||||||
const gpus = ref<GpuInfo[]>([])
|
const gpus = ref<GpuInfo[]>([])
|
||||||
const computeNodes = ref<Array<{ id: string; scheduler_status?: string }>>([])
|
const computeNodes = ref<Array<{ id: string; scheduler_status?: string }>>([])
|
||||||
const selectedGpuId = ref<number | null>(null)
|
const selectedGpuKeys = ref<string[]>([])
|
||||||
|
|
||||||
/** Only show GPUs from nodes that are online or draining */
|
/** Only show GPUs from nodes that are online or draining */
|
||||||
const availableGpus = computed(() => {
|
const availableGpus = computed(() => {
|
||||||
@@ -74,7 +74,13 @@ const selectedModel = computed(() => models.value.find((model) => model.id === f
|
|||||||
const modelDialogTitle = computed(() => selectedModel.value?.name || '')
|
const modelDialogTitle = computed(() => selectedModel.value?.name || '')
|
||||||
|
|
||||||
/** 训练命令与提交载荷共用同一份表单模型。 */
|
/** 训练命令与提交载荷共用同一份表单模型。 */
|
||||||
const selectedGpuIds = computed(() => (selectedGpuId.value != null ? [selectedGpuId.value] : []))
|
const selectedGpus = computed(() =>
|
||||||
|
selectedGpuKeys.value
|
||||||
|
.map((key) => availableGpus.value.find((gpu) => gpuKey(gpu) === key))
|
||||||
|
.filter((gpu): gpu is GpuInfo => Boolean(gpu)),
|
||||||
|
)
|
||||||
|
const selectedComputeNodeId = computed(() => selectedGpus.value[0]?.node_id)
|
||||||
|
const selectedGpuIds = computed(() => selectedGpus.value.map((gpu) => Number(gpu.id)))
|
||||||
const commandPreview = computed(() => buildFineTuneCommand(form, selectedGpuIds.value))
|
const commandPreview = computed(() => buildFineTuneCommand(form, selectedGpuIds.value))
|
||||||
|
|
||||||
const remoteCommandPreview = computed(() => {
|
const remoteCommandPreview = computed(() => {
|
||||||
@@ -83,9 +89,32 @@ const remoteCommandPreview = computed(() => {
|
|||||||
return preflightResult.value?.preview?.command_text || ''
|
return preflightResult.value?.preview?.command_text || ''
|
||||||
})
|
})
|
||||||
|
|
||||||
/** GPU 单选切换(每次只选中一张 GPU) */
|
function gpuKey(gpu: GpuInfo) {
|
||||||
function toggleGpu(gpuId: number) {
|
return `${gpu.node_id || 'local'}:${gpu.id ?? gpu.uuid ?? gpu.name}`
|
||||||
selectedGpuId.value = selectedGpuId.value === gpuId ? null : gpuId
|
}
|
||||||
|
|
||||||
|
function isGpuUnavailable(gpu: GpuInfo) {
|
||||||
|
return gpu.status === 'busy' || gpu.status === 'reserved' || gpu.status === 'offline'
|
||||||
|
}
|
||||||
|
|
||||||
|
function isGpuSelected(gpu: GpuInfo) {
|
||||||
|
return selectedGpuKeys.value.includes(gpuKey(gpu))
|
||||||
|
}
|
||||||
|
|
||||||
|
/** GPU 多选切换:单个任务只允许选择同一算力节点内的空闲卡。 */
|
||||||
|
function toggleGpu(gpu: GpuInfo) {
|
||||||
|
if (isGpuUnavailable(gpu) || gpu.id == null) return
|
||||||
|
const key = gpuKey(gpu)
|
||||||
|
if (isGpuSelected(gpu)) {
|
||||||
|
selectedGpuKeys.value = selectedGpuKeys.value.filter((item) => item !== key)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if (selectedComputeNodeId.value && gpu.node_id && selectedComputeNodeId.value !== gpu.node_id) {
|
||||||
|
selectedGpuKeys.value = [key]
|
||||||
|
ElMessage.info('已切换到新的算力节点,之前选择的 GPU 已清空')
|
||||||
|
return
|
||||||
|
}
|
||||||
|
selectedGpuKeys.value = [...selectedGpuKeys.value, key]
|
||||||
}
|
}
|
||||||
|
|
||||||
function gpuUsageWidth(percent: number) {
|
function gpuUsageWidth(percent: number) {
|
||||||
@@ -178,8 +207,8 @@ async function loadGpus() {
|
|||||||
const [sys, nodes] = await Promise.all([getSystemInfo(), getComputeNodes().catch(() => [])])
|
const [sys, nodes] = await Promise.all([getSystemInfo(), getComputeNodes().catch(() => [])])
|
||||||
gpus.value = sys?.gpu || []
|
gpus.value = sys?.gpu || []
|
||||||
computeNodes.value = nodes || []
|
computeNodes.value = nodes || []
|
||||||
// Default select first available GPU
|
const firstIdle = availableGpus.value.find((gpu) => !isGpuUnavailable(gpu) && gpu.id != null)
|
||||||
if (availableGpus.value.length > 0) selectedGpuId.value = availableGpus.value[0].id ?? null
|
if (firstIdle) selectedGpuKeys.value = [gpuKey(firstIdle)]
|
||||||
} catch {
|
} catch {
|
||||||
gpus.value = []
|
gpus.value = []
|
||||||
}
|
}
|
||||||
@@ -189,8 +218,8 @@ async function handleSubmit() {
|
|||||||
if (!formRef.value) return
|
if (!formRef.value) return
|
||||||
await formRef.value.validate(async (valid) => {
|
await formRef.value.validate(async (valid) => {
|
||||||
if (!valid) return
|
if (!valid) return
|
||||||
if (selectedGpuId.value == null) {
|
if (!selectedGpuIds.value.length) {
|
||||||
ElMessage.warning('请选择一个 GPU')
|
ElMessage.warning('请至少选择一张空闲 GPU')
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
submitting.value = true
|
submitting.value = true
|
||||||
@@ -207,7 +236,7 @@ async function handleSubmit() {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
const payload = buildFineTunePayload(form, selectedGpuIds.value)
|
const payload = buildFineTunePayload(form, selectedGpuIds.value, selectedComputeNodeId.value)
|
||||||
const preflight = await runPreflight(payload)
|
const preflight = await runPreflight(payload)
|
||||||
if (!preflight?.valid) {
|
if (!preflight?.valid) {
|
||||||
ElMessage.error('训练预检未通过,请先处理预检问题')
|
ElMessage.error('训练预检未通过,请先处理预检问题')
|
||||||
@@ -232,7 +261,7 @@ async function handleSubmit() {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
async function runPreflight(payload = buildFineTunePayload(form, selectedGpuIds.value)) {
|
async function runPreflight(payload = buildFineTunePayload(form, selectedGpuIds.value, selectedComputeNodeId.value)) {
|
||||||
preflightLoading.value = true
|
preflightLoading.value = true
|
||||||
try {
|
try {
|
||||||
const result = await preflightFineTune(payload)
|
const result = await preflightFineTune(payload)
|
||||||
@@ -261,8 +290,8 @@ async function handlePreflightClick() {
|
|||||||
if (!formRef.value) return
|
if (!formRef.value) return
|
||||||
await formRef.value.validate(async (valid) => {
|
await formRef.value.validate(async (valid) => {
|
||||||
if (!valid) return
|
if (!valid) return
|
||||||
if (selectedGpuId.value == null) {
|
if (!selectedGpuIds.value.length) {
|
||||||
ElMessage.warning('请选择一个 GPU')
|
ElMessage.warning('请至少选择一张空闲 GPU')
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
await runPreflight()
|
await runPreflight()
|
||||||
@@ -297,12 +326,16 @@ onMounted(() => {
|
|||||||
<el-divider content-position="left">训练配置</el-divider>
|
<el-divider content-position="left">训练配置</el-divider>
|
||||||
<el-form-item label="GPU 硬件">
|
<el-form-item label="GPU 硬件">
|
||||||
<div class="gpu-list">
|
<div class="gpu-list">
|
||||||
|
<div class="gpu-selection-summary">
|
||||||
|
已选择 {{ selectedGpuIds.length }} 张 GPU
|
||||||
|
<template v-if="selectedGpus[0]?.node_code"> · {{ selectedGpus[0].node_code }}</template>
|
||||||
|
</div>
|
||||||
<div
|
<div
|
||||||
v-for="gpu in availableGpus"
|
v-for="gpu in availableGpus"
|
||||||
:key="gpu.id"
|
:key="gpuKey(gpu)"
|
||||||
class="gpu-card"
|
class="gpu-card"
|
||||||
:class="{ active: selectedGpuId === gpu.id, 'is-busy': gpu.gpu_percent > 80 }"
|
:class="{ active: isGpuSelected(gpu), 'is-busy': isGpuUnavailable(gpu), 'is-disabled': isGpuUnavailable(gpu) }"
|
||||||
@click="toggleGpu(gpu.id!)"
|
@click="toggleGpu(gpu)"
|
||||||
>
|
>
|
||||||
<div class="gpu-card-top">
|
<div class="gpu-card-top">
|
||||||
<div class="gpu-title">
|
<div class="gpu-title">
|
||||||
@@ -312,7 +345,7 @@ onMounted(() => {
|
|||||||
</span>
|
</span>
|
||||||
<span class="gpu-name">{{ gpu.name }}</span>
|
<span class="gpu-name">{{ gpu.name }}</span>
|
||||||
</div>
|
</div>
|
||||||
<span class="gpu-usage">{{ gpu.gpu_percent }}%</span>
|
<span class="gpu-usage">{{ isGpuUnavailable(gpu) ? gpu.status : `${gpu.gpu_percent}%` }}</span>
|
||||||
</div>
|
</div>
|
||||||
<div class="gpu-usage-bar">
|
<div class="gpu-usage-bar">
|
||||||
<span :style="{ width: gpuUsageWidth(gpu.gpu_percent) }" />
|
<span :style="{ width: gpuUsageWidth(gpu.gpu_percent) }" />
|
||||||
@@ -578,6 +611,13 @@ onMounted(() => {
|
|||||||
width: 100%;
|
width: 100%;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
.gpu-selection-summary {
|
||||||
|
grid-column: 1 / -1;
|
||||||
|
color: #64748b;
|
||||||
|
font-size: 12px;
|
||||||
|
line-height: 20px;
|
||||||
|
}
|
||||||
|
|
||||||
.gpu-card {
|
.gpu-card {
|
||||||
border: 1px solid #e5e7eb;
|
border: 1px solid #e5e7eb;
|
||||||
border-radius: 6px;
|
border-radius: 6px;
|
||||||
@@ -627,6 +667,11 @@ onMounted(() => {
|
|||||||
background: #dc2626;
|
background: #dc2626;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
&.is-disabled {
|
||||||
|
cursor: not-allowed;
|
||||||
|
opacity: 0.72;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
.gpu-card-top {
|
.gpu-card-top {
|
||||||
|
|||||||
@@ -65,6 +65,7 @@ export function createDefaultFineTuneForm(): FineTuneFormModel {
|
|||||||
export function buildFineTunePayload(
|
export function buildFineTunePayload(
|
||||||
form: FineTuneFormModel,
|
form: FineTuneFormModel,
|
||||||
gpus: number[],
|
gpus: number[],
|
||||||
|
computeNodeId?: string,
|
||||||
): Omit<FineTuneStartPayload, 'task_id'> {
|
): Omit<FineTuneStartPayload, 'task_id'> {
|
||||||
return {
|
return {
|
||||||
name: form.name,
|
name: form.name,
|
||||||
@@ -77,6 +78,7 @@ export function buildFineTunePayload(
|
|||||||
train_dataset_id: form.train_dataset_id,
|
train_dataset_id: form.train_dataset_id,
|
||||||
auto_merge: form.train_type === 'SFT' && form.auto_merge,
|
auto_merge: form.train_type === 'SFT' && form.auto_merge,
|
||||||
output_model_name: form.name,
|
output_model_name: form.name,
|
||||||
|
compute_node_id: computeNodeId,
|
||||||
batch_size: form.batch_size,
|
batch_size: form.batch_size,
|
||||||
learning_rate: form.learning_rate,
|
learning_rate: form.learning_rate,
|
||||||
n_epochs: form.n_epochs,
|
n_epochs: form.n_epochs,
|
||||||
|
|||||||
@@ -1,17 +1,17 @@
|
|||||||
<script setup lang="ts">
|
<script setup lang="ts">
|
||||||
import { ref, reactive, nextTick, onMounted, watch } from 'vue'
|
import { ref, reactive, nextTick, onMounted, onUnmounted, watch } from 'vue'
|
||||||
import { useRoute, useRouter } from 'vue-router'
|
import { useRoute, useRouter } from 'vue-router'
|
||||||
import { ElMessage } from 'element-plus'
|
import { ElMessage } from 'element-plus'
|
||||||
import MarkdownView from '@/components/MarkdownView.vue'
|
import MarkdownView from '@/components/MarkdownView.vue'
|
||||||
import { useStreamChat } from '@/composables/useStreamChat'
|
import { useStreamChat } from '@/composables/useStreamChat'
|
||||||
import { getCompare } from '@/api/modules/compare'
|
import { getCompare, getLoadStatus } from '@/api/modules/compare'
|
||||||
import type { CompareTask, LoadedModel } from '@/types'
|
import type { CompareTask, LoadedModel } from '@/types'
|
||||||
|
|
||||||
const route = useRoute()
|
const route = useRoute()
|
||||||
const router = useRouter()
|
const router = useRouter()
|
||||||
const taskId = route.params.id as string
|
const taskId = route.params.id as string
|
||||||
/** 是否为 mock 直通模式(新建推理假数据进入,不走真实任务接口) */
|
/** 是否为 mock 模式(新建推理无真实 taskId 或明确为 mock 时进入 mock 模式) */
|
||||||
const isMock = taskId === 'mock'
|
const isMock = taskId === 'mock' || !taskId || taskId === 'unknown'
|
||||||
/** 当前对话使用的模型名 */
|
/** 当前对话使用的模型名 */
|
||||||
const modelName = ref(route.query.model as string || '')
|
const modelName = ref(route.query.model as string || '')
|
||||||
|
|
||||||
@@ -37,6 +37,10 @@ const contentRef = ref<HTMLElement>()
|
|||||||
let activeAssistant: ChatMessage | null = null
|
let activeAssistant: ChatMessage | null = null
|
||||||
/** 设置面板抽屉 */
|
/** 设置面板抽屉 */
|
||||||
const showSettings = ref(false)
|
const showSettings = ref(false)
|
||||||
|
/** 模型仍在加载中(直接 URL 进入 chat 时兜底轮询就绪状态) */
|
||||||
|
const taskLoading = ref(false)
|
||||||
|
const taskError = ref('')
|
||||||
|
let statusTimer: ReturnType<typeof setInterval> | null = null
|
||||||
|
|
||||||
/** 获取任务信息,定位已启动的模型(mock 模式跳过) */
|
/** 获取任务信息,定位已启动的模型(mock 模式跳过) */
|
||||||
async function loadTask() {
|
async function loadTask() {
|
||||||
@@ -45,6 +49,14 @@ async function loadTask() {
|
|||||||
task.value = await getCompare(taskId)
|
task.value = await getCompare(taskId)
|
||||||
const models = parseLoadedModels(task.value)
|
const models = parseLoadedModels(task.value)
|
||||||
if (models[0]?.model_name) modelName.value = models[0].model_name
|
if (models[0]?.model_name) modelName.value = models[0].model_name
|
||||||
|
// 恢复本地保存的历史对话
|
||||||
|
restoreHistory()
|
||||||
|
// 模型仍在上次加载中:启动轮询等待就绪
|
||||||
|
if (models.some((m) => m.status === 'starting')) {
|
||||||
|
taskLoading.value = true
|
||||||
|
await pollTaskStatus()
|
||||||
|
statusTimer = setInterval(pollTaskStatus, 3000)
|
||||||
|
}
|
||||||
} catch {
|
} catch {
|
||||||
// ignore
|
// ignore
|
||||||
}
|
}
|
||||||
@@ -60,6 +72,80 @@ function parseLoadedModels(t: CompareTask | null): LoadedModel[] {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** 对话历史本地持久化(按任务 id 存储,退出重进可恢复) */
|
||||||
|
const STORAGE_PREFIX = 'ygft_chat_history_'
|
||||||
|
|
||||||
|
function historyKey(id: string | number): string {
|
||||||
|
return `${STORAGE_PREFIX}${id}`
|
||||||
|
}
|
||||||
|
|
||||||
|
function saveHistory() {
|
||||||
|
if (isMock) return
|
||||||
|
try {
|
||||||
|
const snapshot = messages.value.map((m) => ({
|
||||||
|
role: m.role,
|
||||||
|
content: m.content,
|
||||||
|
think: m.think,
|
||||||
|
done: true,
|
||||||
|
}))
|
||||||
|
localStorage.setItem(historyKey(taskId), JSON.stringify(snapshot))
|
||||||
|
} catch {
|
||||||
|
// 存储失败忽略
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function restoreHistory() {
|
||||||
|
if (isMock) return
|
||||||
|
try {
|
||||||
|
const raw = localStorage.getItem(historyKey(taskId))
|
||||||
|
if (!raw) return
|
||||||
|
const parsed = JSON.parse(raw)
|
||||||
|
if (Array.isArray(parsed)) {
|
||||||
|
messages.value = parsed.map((m) => ({
|
||||||
|
role: m.role === 'user' ? 'user' : 'assistant',
|
||||||
|
content: m.content || '',
|
||||||
|
think: m.think || '',
|
||||||
|
isThinking: false,
|
||||||
|
isStreaming: false,
|
||||||
|
done: true,
|
||||||
|
}))
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
// 恢复失败忽略
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 停止就绪状态轮询 */
|
||||||
|
function stopStatusPolling() {
|
||||||
|
if (statusTimer) {
|
||||||
|
clearInterval(statusTimer)
|
||||||
|
statusTimer = null
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 轮询任务加载状态:starting → ready/error */
|
||||||
|
async function pollTaskStatus() {
|
||||||
|
try {
|
||||||
|
const st = await getLoadStatus(taskId)
|
||||||
|
const items = st.loaded_models || []
|
||||||
|
const anyReady = items.some((m) => m.status === 'ready' || m.status === 'running')
|
||||||
|
const anyError = items.some((m) => m.status === 'error')
|
||||||
|
if (anyReady) {
|
||||||
|
taskLoading.value = false
|
||||||
|
taskError.value = ''
|
||||||
|
stopStatusPolling()
|
||||||
|
} else if (anyError) {
|
||||||
|
taskLoading.value = false
|
||||||
|
taskError.value = items.find((m) => m.status === 'error')?.error || '模型加载失败'
|
||||||
|
stopStatusPolling()
|
||||||
|
} else {
|
||||||
|
taskLoading.value = true
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
// 轮询失败忽略,下次再试
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
async function handleSend() {
|
async function handleSend() {
|
||||||
const question = inputQuestion.value.trim()
|
const question = inputQuestion.value.trim()
|
||||||
if (!question || loading.value) return
|
if (!question || loading.value) return
|
||||||
@@ -76,6 +162,7 @@ async function handleSend() {
|
|||||||
done: false,
|
done: false,
|
||||||
})
|
})
|
||||||
messages.value.push(assistantMsg)
|
messages.value.push(assistantMsg)
|
||||||
|
saveHistory()
|
||||||
|
|
||||||
inputQuestion.value = ''
|
inputQuestion.value = ''
|
||||||
await nextTick()
|
await nextTick()
|
||||||
@@ -85,33 +172,25 @@ async function handleSend() {
|
|||||||
// mock 模式:直接用假数据逐字填充
|
// mock 模式:直接用假数据逐字填充
|
||||||
if (isMock) {
|
if (isMock) {
|
||||||
await mockReply(assistantMsg, question)
|
await mockReply(assistantMsg, question)
|
||||||
|
saveHistory()
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
// 真实模式:获取已启动模型的端口/路径
|
// 真实模式:通过后端 SSE 流式代理到算力节点进行推理
|
||||||
const models = parseLoadedModels(task.value)
|
|
||||||
const target = models[0]
|
|
||||||
if (!target) {
|
|
||||||
ElMessage.error('未找到已启动的模型')
|
|
||||||
assistantMsg.content = '未找到已启动的模型,请先返回列表加载模型'
|
|
||||||
assistantMsg.done = true
|
|
||||||
assistantMsg.isStreaming = false
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
// 流式状态变化时只同步当前回复,避免固定定时器空转。
|
|
||||||
activeAssistant = assistantMsg
|
activeAssistant = assistantMsg
|
||||||
|
|
||||||
await send({
|
await send(
|
||||||
port: target.port,
|
{
|
||||||
model_name: target.model_name,
|
model_path: route.query.model_path as string || '',
|
||||||
model_path: '',
|
task_id: taskId,
|
||||||
system_prompt: systemPrompt.value,
|
system_prompt: systemPrompt.value,
|
||||||
user_question: question,
|
user_question: question,
|
||||||
temperature: temperature.value,
|
temperature: temperature.value,
|
||||||
top_p: top_p.value,
|
top_p: top_p.value,
|
||||||
max_tokens: maxTokens.value,
|
max_tokens: maxTokens.value,
|
||||||
})
|
},
|
||||||
|
{ useMock: false },
|
||||||
|
)
|
||||||
|
|
||||||
// 完成后同步最终内容
|
// 完成后同步最终内容
|
||||||
assistantMsg.content = message.value.displayContent || message.value.error || '(无回复)'
|
assistantMsg.content = message.value.displayContent || message.value.error || '(无回复)'
|
||||||
@@ -121,6 +200,7 @@ async function handleSend() {
|
|||||||
assistantMsg.done = true
|
assistantMsg.done = true
|
||||||
activeAssistant = null
|
activeAssistant = null
|
||||||
reset()
|
reset()
|
||||||
|
saveHistory()
|
||||||
await nextTick()
|
await nextTick()
|
||||||
scrollToBottom()
|
scrollToBottom()
|
||||||
}
|
}
|
||||||
@@ -179,6 +259,11 @@ function handleNewChat() {
|
|||||||
activeAssistant = null
|
activeAssistant = null
|
||||||
messages.value = []
|
messages.value = []
|
||||||
reset()
|
reset()
|
||||||
|
try {
|
||||||
|
localStorage.removeItem(historyKey(taskId))
|
||||||
|
} catch {
|
||||||
|
// 忽略
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/** 输入框自适应高度 */
|
/** 输入框自适应高度 */
|
||||||
@@ -195,6 +280,7 @@ function resetInputHeight() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
onMounted(loadTask)
|
onMounted(loadTask)
|
||||||
|
onUnmounted(stopStatusPolling)
|
||||||
</script>
|
</script>
|
||||||
|
|
||||||
<template>
|
<template>
|
||||||
@@ -262,6 +348,12 @@ onMounted(loadTask)
|
|||||||
|
|
||||||
<!-- 输入栏 -->
|
<!-- 输入栏 -->
|
||||||
<footer class="chat-input-container">
|
<footer class="chat-input-container">
|
||||||
|
<div v-if="taskLoading" class="loading-hint">
|
||||||
|
<i class="fa fa-spinner fa-spin" style="margin-right: 6px" />模型加载中,就绪后即可对话...
|
||||||
|
</div>
|
||||||
|
<div v-else-if="taskError" class="loading-hint error">
|
||||||
|
<i class="fa fa-exclamation-circle" style="margin-right: 6px" />{{ taskError }}
|
||||||
|
</div>
|
||||||
<div class="chat-input-inner">
|
<div class="chat-input-inner">
|
||||||
<button class="clear-btn" title="清空对话" @click="handleNewChat">
|
<button class="clear-btn" title="清空对话" @click="handleNewChat">
|
||||||
<i class="fa fa-eraser" />
|
<i class="fa fa-eraser" />
|
||||||
@@ -271,22 +363,21 @@ onMounted(loadTask)
|
|||||||
v-model="inputQuestion"
|
v-model="inputQuestion"
|
||||||
class="input-box"
|
class="input-box"
|
||||||
rows="1"
|
rows="1"
|
||||||
:disabled="loading"
|
:disabled="loading || taskLoading"
|
||||||
placeholder="给模型发送消息..."
|
placeholder="给模型发送消息..."
|
||||||
@keydown.enter.exact.prevent="handleSend"
|
@keydown.enter.exact.prevent="handleSend"
|
||||||
@input="autoResize"
|
@input="autoResize"
|
||||||
/>
|
/>
|
||||||
<button
|
<button
|
||||||
class="send-btn"
|
class="send-btn"
|
||||||
:class="{ active: inputQuestion.trim() && !loading }"
|
:class="{ active: inputQuestion.trim() && !loading && !taskLoading }"
|
||||||
:disabled="!inputQuestion.trim() || loading"
|
:disabled="!inputQuestion.trim() || loading || taskLoading"
|
||||||
@click="handleSend"
|
@click="handleSend"
|
||||||
>
|
>
|
||||||
<i class="fa fa-arrow-up" />
|
<i class="fa fa-arrow-up" />
|
||||||
</button>
|
</button>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
<div class="footer-hint">内容由 AI 生成,请仔细甄别。</div>
|
|
||||||
</footer>
|
</footer>
|
||||||
|
|
||||||
<!-- 设置抽屉(系统提示词等) -->
|
<!-- 设置抽屉(系统提示词等) -->
|
||||||
@@ -713,9 +804,18 @@ onMounted(loadTask)
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
.footer-hint {
|
.loading-hint {
|
||||||
margin-top: 12px;
|
margin-bottom: 10px;
|
||||||
font-size: 12px;
|
padding: 6px 14px;
|
||||||
color: #9ca3af;
|
font-size: 13px;
|
||||||
|
color: #b45309;
|
||||||
|
background: #fef3c7;
|
||||||
|
border-radius: 8px;
|
||||||
|
text-align: center;
|
||||||
|
|
||||||
|
&.error {
|
||||||
|
color: #b91c1c;
|
||||||
|
background: #fee2e2;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
</style>
|
</style>
|
||||||
|
|||||||
@@ -1,10 +1,12 @@
|
|||||||
<script setup lang="ts">
|
<script setup lang="ts">
|
||||||
import { ref, reactive, computed, onMounted } from 'vue'
|
import { ref, reactive, computed, onMounted, watch } from 'vue'
|
||||||
import { useRouter } from 'vue-router'
|
import { useRouter } from 'vue-router'
|
||||||
import { ElMessage, type FormInstance, type FormRules } from 'element-plus'
|
import { ElMessage, type FormInstance, type FormRules } from 'element-plus'
|
||||||
import PageCard from '@/components/PageCard.vue'
|
import PageCard from '@/components/PageCard.vue'
|
||||||
import { getModelList, getTrainedModels } from '@/api/modules/model'
|
import { getModelList, getTrainedModels } from '@/api/modules/model'
|
||||||
import { getSystemInfo } from '@/api/modules/system'
|
import { getSystemInfo } from '@/api/modules/system'
|
||||||
|
import { getComputeNodes, type ComputeNode } from '@/api/modules/compute'
|
||||||
|
import { createCompare, loadCompare } from '@/api/modules/compare'
|
||||||
import type { ModelItem, TrainedModel, GpuInfo } from '@/types'
|
import type { ModelItem, TrainedModel, GpuInfo } from '@/types'
|
||||||
|
|
||||||
const router = useRouter()
|
const router = useRouter()
|
||||||
@@ -15,6 +17,7 @@ const startupStatus = ref('')
|
|||||||
const dbModels = ref<ModelItem[]>([])
|
const dbModels = ref<ModelItem[]>([])
|
||||||
const trainedModels = ref<TrainedModel[]>([])
|
const trainedModels = ref<TrainedModel[]>([])
|
||||||
const gpus = ref<GpuInfo[]>([])
|
const gpus = ref<GpuInfo[]>([])
|
||||||
|
const computeNodes = ref<ComputeNode[]>([])
|
||||||
|
|
||||||
/** 可选模型(下拉用,区分本地/已训练两类) */
|
/** 可选模型(下拉用,区分本地/已训练两类) */
|
||||||
interface SelectableModel {
|
interface SelectableModel {
|
||||||
@@ -24,6 +27,8 @@ interface SelectableModel {
|
|||||||
name: string
|
name: string
|
||||||
source: 'database' | 'trained'
|
source: 'database' | 'trained'
|
||||||
model_path: string
|
model_path: string
|
||||||
|
compute_node_id?: string
|
||||||
|
compute_node_name?: string
|
||||||
merged?: boolean
|
merged?: boolean
|
||||||
merging?: boolean
|
merging?: boolean
|
||||||
disabled?: boolean
|
disabled?: boolean
|
||||||
@@ -48,13 +53,25 @@ const trainedOptions = computed<SelectableModel[]>(() =>
|
|||||||
name: m.name,
|
name: m.name,
|
||||||
source: 'trained',
|
source: 'trained',
|
||||||
model_path: m.merged_path || m.base_model_path || '',
|
model_path: m.merged_path || m.base_model_path || '',
|
||||||
|
compute_node_id: m.compute_node_id,
|
||||||
|
compute_node_name: m.compute_node_name,
|
||||||
merged: m.merged,
|
merged: m.merged,
|
||||||
merging: m.merging,
|
merging: m.merging,
|
||||||
disabled: m.merged === false,
|
disabled: m.merged === false,
|
||||||
})),
|
})),
|
||||||
)
|
)
|
||||||
|
|
||||||
/** key → 模型映射,便于取选中项 */
|
/** 仅显示在线算力节点上的空闲 GPU */
|
||||||
|
const onlineNodeIds = computed(() => new Set(
|
||||||
|
computeNodes.value
|
||||||
|
.filter((n) => n.enabled && n.scheduler_status === 'online')
|
||||||
|
.map((n) => n.id),
|
||||||
|
))
|
||||||
|
const idleGpus = computed(() =>
|
||||||
|
gpus.value.filter(
|
||||||
|
(g) => g.status === 'idle' && (!g.node_id || onlineNodeIds.value.has(g.node_id)),
|
||||||
|
),
|
||||||
|
)
|
||||||
const modelMap = computed<Record<string, SelectableModel>>(() => {
|
const modelMap = computed<Record<string, SelectableModel>>(() => {
|
||||||
const map: Record<string, SelectableModel> = {}
|
const map: Record<string, SelectableModel> = {}
|
||||||
for (const m of [...dbOptions.value, ...trainedOptions.value]) map[m.key] = m
|
for (const m of [...dbOptions.value, ...trainedOptions.value]) map[m.key] = m
|
||||||
@@ -67,7 +84,7 @@ const form = reactive({
|
|||||||
/** 选中的模型 key(单选) */
|
/** 选中的模型 key(单选) */
|
||||||
model_key: '',
|
model_key: '',
|
||||||
/** 使用的 GPU */
|
/** 使用的 GPU */
|
||||||
gpu_id: 0,
|
gpu_key: '',
|
||||||
})
|
})
|
||||||
|
|
||||||
const rules: FormRules = {
|
const rules: FormRules = {
|
||||||
@@ -77,6 +94,13 @@ const rules: FormRules = {
|
|||||||
|
|
||||||
/** 当前选中的模型对象 */
|
/** 当前选中的模型对象 */
|
||||||
const selectedModel = computed(() => modelMap.value[form.model_key])
|
const selectedModel = computed(() => modelMap.value[form.model_key])
|
||||||
|
const selectedGpu = computed(() => idleGpus.value.find((g) => `${g.node_id || ''}:${g.id ?? 0}` === form.gpu_key))
|
||||||
|
|
||||||
|
watch(selectedModel, (model) => {
|
||||||
|
if (!model?.compute_node_id) return
|
||||||
|
const gpu = idleGpus.value.find((item) => item.node_id === model.compute_node_id)
|
||||||
|
if (gpu) form.gpu_key = `${gpu.node_id || ''}:${gpu.id ?? 0}`
|
||||||
|
})
|
||||||
|
|
||||||
async function handleSubmit() {
|
async function handleSubmit() {
|
||||||
if (!formRef.value) return
|
if (!formRef.value) return
|
||||||
@@ -88,18 +112,47 @@ async function handleSubmit() {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
submitting.value = true
|
submitting.value = true
|
||||||
startupStatus.value = '正在启动模型服务...'
|
startupStatus.value = '正在创建推理任务...'
|
||||||
try {
|
try {
|
||||||
// 当前为 mock 环境:不创建任务、不启动后端服务,
|
if (!m.model_path) {
|
||||||
// 用假数据直通进入对话界面(模型名通过 query 传递)。
|
ElMessage.warning('当前模型未配置算力节点可访问路径,请先在模型管理中维护模型路径')
|
||||||
// 接入真实后端后,可在此恢复 createCompare / startModelsInBackground / monitorStartup 流程。
|
return
|
||||||
await new Promise((resolve) => setTimeout(resolve, 1200))
|
}
|
||||||
|
|
||||||
ElMessage.success('模型已启动')
|
// Step 1: 创建推理任务记录
|
||||||
router.push({
|
const taskResult = await createCompare({
|
||||||
path: '/model-inference/chat/mock',
|
name: form.name || m.name,
|
||||||
query: { model: m.name },
|
description: form.description,
|
||||||
|
status: 'pending',
|
||||||
|
models: [
|
||||||
|
{
|
||||||
|
model_id: String(m.id),
|
||||||
|
model_name: m.name,
|
||||||
|
model_path: m.model_path,
|
||||||
|
source: m.source,
|
||||||
|
gpu_id: selectedGpu.value?.id ?? 0,
|
||||||
|
node_id: selectedGpu.value?.node_id || m.compute_node_id,
|
||||||
|
node_name: selectedGpu.value?.node_name || m.compute_node_name,
|
||||||
|
},
|
||||||
|
],
|
||||||
})
|
})
|
||||||
|
const taskId = taskResult?.id || 'unknown'
|
||||||
|
|
||||||
|
// Step 2: 统一通过推理任务加载接口异步派发模型加载,状态会落到列表记录中。
|
||||||
|
startupStatus.value = '正在启动模型服务,首次加载可能需要数分钟...'
|
||||||
|
const loadResult: any = await loadCompare(taskId)
|
||||||
|
if (loadResult?.status === 'failed' || loadResult?.error) {
|
||||||
|
ElMessage.warning(`模型加载失败:${loadResult?.error || '请检查算力节点日志'}`)
|
||||||
|
router.push('/model-inference')
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// 加载为异步派发,回到列表页可看到“启动中 → 已就绪”的状态流转
|
||||||
|
ElMessage.success('模型加载中,就绪后即可对话')
|
||||||
|
router.push('/model-inference')
|
||||||
|
} catch (e: any) {
|
||||||
|
const reason = e?.message || e?.toString() || '未知错误'
|
||||||
|
ElMessage.warning(`推理服务启动失败:${reason}`)
|
||||||
} finally {
|
} finally {
|
||||||
submitting.value = false
|
submitting.value = false
|
||||||
startupStatus.value = ''
|
startupStatus.value = ''
|
||||||
@@ -113,16 +166,21 @@ function handleCancel() {
|
|||||||
|
|
||||||
async function loadData() {
|
async function loadData() {
|
||||||
try {
|
try {
|
||||||
const [db, trained, sys] = await Promise.all([
|
const [db, trained, sys, nodes] = await Promise.all([
|
||||||
getModelList(),
|
getModelList(),
|
||||||
getTrainedModels(),
|
getTrainedModels(),
|
||||||
getSystemInfo(),
|
getSystemInfo(),
|
||||||
|
getComputeNodes(),
|
||||||
])
|
])
|
||||||
dbModels.value = db || []
|
dbModels.value = db || []
|
||||||
trainedModels.value = trained?.models || []
|
trainedModels.value = trained?.models || []
|
||||||
gpus.value = sys?.gpu || []
|
gpus.value = sys?.gpu || []
|
||||||
// 默认选中第一个 GPU
|
computeNodes.value = nodes || []
|
||||||
if (gpus.value.length > 0) form.gpu_id = 0
|
// 默认选中第一个空闲 GPU
|
||||||
|
if (idleGpus.value.length > 0) {
|
||||||
|
const firstGpu = idleGpus.value[0]
|
||||||
|
form.gpu_key = `${firstGpu.node_id || ''}:${firstGpu.id ?? 0}`
|
||||||
|
}
|
||||||
} catch {
|
} catch {
|
||||||
// ignore
|
// ignore
|
||||||
}
|
}
|
||||||
@@ -170,12 +228,12 @@ onMounted(loadData)
|
|||||||
</el-form-item>
|
</el-form-item>
|
||||||
|
|
||||||
<el-form-item label="GPU">
|
<el-form-item label="GPU">
|
||||||
<el-select v-model="form.gpu_id" style="width: 400px">
|
<el-select v-model="form.gpu_key" style="width: 400px">
|
||||||
<el-option
|
<el-option
|
||||||
v-for="(g, idx) in gpus"
|
v-for="g in idleGpus"
|
||||||
:key="idx"
|
:key="`${g.node_id || ''}:${g.id ?? 0}`"
|
||||||
:label="`${g.name} (GPU${idx})`"
|
:label="`${g.node_name || g.node_code || '算力节点'} / ${g.name} (GPU${g.id ?? 0}) [空闲]`"
|
||||||
:value="idx"
|
:value="`${g.node_id || ''}:${g.id ?? 0}`"
|
||||||
/>
|
/>
|
||||||
</el-select>
|
</el-select>
|
||||||
</el-form-item>
|
</el-form-item>
|
||||||
|
|||||||
@@ -7,10 +7,8 @@ import { usePolling } from '@/composables/usePolling'
|
|||||||
import {
|
import {
|
||||||
getCompareList,
|
getCompareList,
|
||||||
deleteCompare,
|
deleteCompare,
|
||||||
getCompare,
|
|
||||||
loadCompare,
|
loadCompare,
|
||||||
unloadCompare,
|
unloadCompare,
|
||||||
stopModelByPid,
|
|
||||||
} from '@/api/modules/compare'
|
} from '@/api/modules/compare'
|
||||||
import type { CompareTask, LoadedModel } from '@/types'
|
import type { CompareTask, LoadedModel } from '@/types'
|
||||||
import { statusLabel, statusTagType } from '@/utils/status'
|
import { statusLabel, statusTagType } from '@/utils/status'
|
||||||
@@ -86,26 +84,17 @@ async function handleLoad(row: any) {
|
|||||||
delayedRefreshTimer = setTimeout(loadData, 1000)
|
delayedRefreshTimer = setTimeout(loadData, 1000)
|
||||||
}
|
}
|
||||||
|
|
||||||
/** 卸载推理任务 */
|
/** 释放推理任务(停止模型服务,释放算力节点 GPU 显存) */
|
||||||
async function handleUnload(row: any) {
|
async function handleUnload(row: any) {
|
||||||
await ElMessageBox.confirm('确定要停止模型服务吗?', '确认停止', { type: 'warning' })
|
await ElMessageBox.confirm('确定要释放模型服务吗?将停止模型进程并释放 GPU 显存。', '确认释放', { type: 'warning' })
|
||||||
await unloadCompare(row.id)
|
await unloadCompare(row.id)
|
||||||
ElMessage.success('已停止模型服务')
|
ElMessage.success('已释放模型服务')
|
||||||
loadData()
|
loadData()
|
||||||
}
|
}
|
||||||
|
|
||||||
/** 删除(先停止进程) */
|
/** 删除(后端删除内部会 best-effort 释放算力节点,这里直接删记录) */
|
||||||
async function handleDelete(row: any) {
|
async function handleDelete(row: any) {
|
||||||
// 先尝试停止已加载的模型进程
|
await ElMessageBox.confirm('确定要删除该推理记录吗?将先释放算力节点再删除。', '确认删除', { type: 'warning' })
|
||||||
const task = await getCompare(row.id).catch(() => null)
|
|
||||||
if (task?.load_status) {
|
|
||||||
const models = parseLoadedModels(task as CompareTask)
|
|
||||||
for (const m of models) {
|
|
||||||
if (m.pid) {
|
|
||||||
await stopModelByPid(m.pid).catch(() => {})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
await deleteCompare(row.id)
|
await deleteCompare(row.id)
|
||||||
dataList.value = dataList.value.filter((item) => item.id !== row.id)
|
dataList.value = dataList.value.filter((item) => item.id !== row.id)
|
||||||
await loadData(true)
|
await loadData(true)
|
||||||
@@ -180,7 +169,7 @@ onUnmounted(() => {
|
|||||||
<i class="fa fa-comments-o" style="margin-right: 4px" />对话
|
<i class="fa fa-comments-o" style="margin-right: 4px" />对话
|
||||||
</el-button>
|
</el-button>
|
||||||
<el-button type="warning" link size="small" @click="handleUnload(row)">
|
<el-button type="warning" link size="small" @click="handleUnload(row)">
|
||||||
<i class="fa fa-stop-circle-o" style="margin-right: 4px" />停止
|
<i class="fa fa-stop-circle-o" style="margin-right: 4px" />释放
|
||||||
</el-button>
|
</el-button>
|
||||||
</template>
|
</template>
|
||||||
<template v-else>
|
<template v-else>
|
||||||
|
|||||||
@@ -18,9 +18,12 @@ const trainedModels = ref<TrainedModel[]>([])
|
|||||||
const currentModel = computed(() => trainedModels.value.find((m) => m.name === modelName.value))
|
const currentModel = computed(() => trainedModels.value.find((m) => m.name === modelName.value))
|
||||||
|
|
||||||
const form = reactive({
|
const form = reactive({
|
||||||
|
trained_model_id: '',
|
||||||
model_name: modelName.value,
|
model_name: modelName.value,
|
||||||
train_method: method.value,
|
train_method: method.value,
|
||||||
base_model_path: '',
|
base_model_path: '',
|
||||||
|
adapter_path: '',
|
||||||
|
compute_node_id: '',
|
||||||
})
|
})
|
||||||
|
|
||||||
async function loadModel() {
|
async function loadModel() {
|
||||||
@@ -28,23 +31,30 @@ async function loadModel() {
|
|||||||
const res = await getTrainedModels()
|
const res = await getTrainedModels()
|
||||||
trainedModels.value = res?.models || []
|
trainedModels.value = res?.models || []
|
||||||
const target = trainedModels.value.find((m) => m.name === modelName.value)
|
const target = trainedModels.value.find((m) => m.name === modelName.value)
|
||||||
|
form.trained_model_id = target?.id == null ? '' : String(target.id)
|
||||||
form.base_model_path = target?.base_model_path || ''
|
form.base_model_path = target?.base_model_path || ''
|
||||||
|
form.adapter_path = target?.artifact_dir || target?.adapter_path || target?.merged_path || ''
|
||||||
|
form.compute_node_id = target?.compute_node_id || ''
|
||||||
} catch {
|
} catch {
|
||||||
// ignore
|
// ignore
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
async function handleMerge() {
|
async function handleMerge() {
|
||||||
if (!form.model_name || !form.base_model_path) {
|
if (!form.model_name || !form.base_model_path || !form.adapter_path) {
|
||||||
ElMessage.warning('缺少模型信息')
|
ElMessage.warning('缺少模型信息')
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
merging.value = true
|
merging.value = true
|
||||||
try {
|
try {
|
||||||
await mergeModel({
|
await mergeModel({
|
||||||
|
trained_model_id: form.trained_model_id || form.model_name,
|
||||||
model_name: form.model_name,
|
model_name: form.model_name,
|
||||||
train_method: form.train_method,
|
train_method: form.train_method,
|
||||||
base_model_path: form.base_model_path,
|
base_model_path: form.base_model_path,
|
||||||
|
adapter_path: form.adapter_path,
|
||||||
|
compute_node_id: form.compute_node_id,
|
||||||
|
output_model_name: `${form.model_name}-merged`,
|
||||||
})
|
})
|
||||||
ElMessage.success('合并成功')
|
ElMessage.success('合并成功')
|
||||||
router.push('/model-manage')
|
router.push('/model-manage')
|
||||||
@@ -82,6 +92,9 @@ onMounted(loadModel)
|
|||||||
<el-form-item label="基座模型路径">
|
<el-form-item label="基座模型路径">
|
||||||
<el-input v-model="form.base_model_path" placeholder="基座模型路径" />
|
<el-input v-model="form.base_model_path" placeholder="基座模型路径" />
|
||||||
</el-form-item>
|
</el-form-item>
|
||||||
|
<el-form-item label="Adapter 路径">
|
||||||
|
<el-input v-model="form.adapter_path" placeholder="LoRA Adapter 权重目录" />
|
||||||
|
</el-form-item>
|
||||||
|
|
||||||
<el-form-item>
|
<el-form-item>
|
||||||
<el-button type="primary" :loading="merging" @click="handleMerge">
|
<el-button type="primary" :loading="merging" @click="handleMerge">
|
||||||
|
|||||||
@@ -5,7 +5,8 @@ import { ElMessage } from 'element-plus'
|
|||||||
import DataTablePage from '@/components/DataTablePage.vue'
|
import DataTablePage from '@/components/DataTablePage.vue'
|
||||||
import AclDialog from '@/components/AclDialog.vue'
|
import AclDialog from '@/components/AclDialog.vue'
|
||||||
import { getProject, getProjectMembers, addProjectMember, removeProjectMember, type Project, type ProjectMember } from '@/api/modules/project'
|
import { getProject, getProjectMembers, addProjectMember, removeProjectMember, type Project, type ProjectMember } from '@/api/modules/project'
|
||||||
import { getUsers, type SystemUser } from '@/api/modules/system'
|
import { getUsers } from '@/api/modules/system'
|
||||||
|
import type { SystemUser } from '@/types'
|
||||||
|
|
||||||
const route = useRoute()
|
const route = useRoute()
|
||||||
const router = useRouter()
|
const router = useRouter()
|
||||||
@@ -56,6 +57,10 @@ async function removeMember(userId: string) {
|
|||||||
load()
|
load()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function asProjectMember(row: unknown): ProjectMember {
|
||||||
|
return row as ProjectMember
|
||||||
|
}
|
||||||
|
|
||||||
onMounted(() => {
|
onMounted(() => {
|
||||||
loadUsers()
|
loadUsers()
|
||||||
load()
|
load()
|
||||||
@@ -94,7 +99,7 @@ onMounted(() => {
|
|||||||
<el-table-column prop="create_time" label="加入时间" min-width="180" />
|
<el-table-column prop="create_time" label="加入时间" min-width="180" />
|
||||||
</template>
|
</template>
|
||||||
<template #actions="{ row }">
|
<template #actions="{ row }">
|
||||||
<el-button link type="danger" @click="removeMember(row.user_id)">移除</el-button>
|
<el-button link type="danger" @click="removeMember(asProjectMember(row).user_id)">移除</el-button>
|
||||||
</template>
|
</template>
|
||||||
</DataTablePage>
|
</DataTablePage>
|
||||||
</el-card>
|
</el-card>
|
||||||
|
|||||||
@@ -1,10 +1,10 @@
|
|||||||
<script setup lang="ts">
|
<script setup lang="ts">
|
||||||
import { computed, onMounted, ref } from 'vue'
|
import { computed, onMounted, ref } from 'vue'
|
||||||
import { useRouter } from 'vue-router'
|
import { useRouter } from 'vue-router'
|
||||||
import { ElMessage } from 'element-plus'
|
import { ElMessage, ElMessageBox } from 'element-plus'
|
||||||
import { Plus } from '@element-plus/icons-vue'
|
import { Plus } from '@element-plus/icons-vue'
|
||||||
import DataTablePage from '@/components/DataTablePage.vue'
|
import DataTablePage from '@/components/DataTablePage.vue'
|
||||||
import { createProject, getProjects, type Project } from '@/api/modules/project'
|
import { createProject, deleteProject, getProjects, type Project } from '@/api/modules/project'
|
||||||
import { getTenants, type Tenant } from '@/api/modules/tenant'
|
import { getTenants, type Tenant } from '@/api/modules/tenant'
|
||||||
|
|
||||||
const router = useRouter()
|
const router = useRouter()
|
||||||
@@ -13,13 +13,24 @@ const projects = ref<Project[]>([])
|
|||||||
const tenants = ref<Tenant[]>([])
|
const tenants = ref<Tenant[]>([])
|
||||||
const tenantId = ref('default')
|
const tenantId = ref('default')
|
||||||
const showCreate = ref(false)
|
const showCreate = ref(false)
|
||||||
const form = ref({ name: '', code: '', description: '', tenant_id: 'default' })
|
const form = ref({ name: '', code: '', description: '', tenant_id: '' })
|
||||||
|
|
||||||
const tenantOptions = computed(() => [
|
const tenantOptions = computed(() => [
|
||||||
{ label: 'default', value: 'default' },
|
{ label: 'default', value: 'default' },
|
||||||
...tenants.value.map((t) => ({ label: t.name, value: t.id })),
|
...tenants.value.map((t) => ({ label: t.name, value: t.id })),
|
||||||
])
|
])
|
||||||
|
|
||||||
|
const tenantCodeOptions = computed(() =>
|
||||||
|
tenants.value.map((t) => ({ label: t.code, value: t.code, tenantId: t.id }))
|
||||||
|
)
|
||||||
|
|
||||||
|
function onTenantCodeChange(code: string) {
|
||||||
|
const tenant = tenants.value.find((t) => t.code === code)
|
||||||
|
if (tenant) {
|
||||||
|
form.value.tenant_id = tenant.id
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
async function load() {
|
async function load() {
|
||||||
loading.value = true
|
loading.value = true
|
||||||
try {
|
try {
|
||||||
@@ -41,15 +52,34 @@ function openDetail(id: string) {
|
|||||||
router.push(`/projects/${id}`)
|
router.push(`/projects/${id}`)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function asProject(row: unknown): Project {
|
||||||
|
return row as Project
|
||||||
|
}
|
||||||
|
|
||||||
async function submitCreate() {
|
async function submitCreate() {
|
||||||
if (!form.value.name || !form.value.code) {
|
if (!form.value.name || !form.value.tenant_id) {
|
||||||
ElMessage.warning('请填写项目名与编码')
|
ElMessage.warning('请填写项目名与编码ID')
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
await createProject({ ...form.value })
|
await createProject({ ...form.value })
|
||||||
ElMessage.success('项目创建成功')
|
ElMessage.success('项目创建成功')
|
||||||
showCreate.value = false
|
showCreate.value = false
|
||||||
form.value = { name: '', code: '', description: '', tenant_id: 'default' }
|
form.value = { name: '', code: '', description: '', tenant_id: '' }
|
||||||
|
load()
|
||||||
|
}
|
||||||
|
|
||||||
|
async function handleDelete(row: Project) {
|
||||||
|
try {
|
||||||
|
await ElMessageBox.confirm(
|
||||||
|
`确定要删除项目「${row.name}」吗?删除后相关数据将无法恢复。`,
|
||||||
|
'删除确认',
|
||||||
|
{ confirmButtonText: '确定删除', cancelButtonText: '取消', type: 'warning' },
|
||||||
|
)
|
||||||
|
} catch {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
await deleteProject(row.id)
|
||||||
|
ElMessage.success('项目已删除')
|
||||||
load()
|
load()
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -61,7 +91,7 @@ onMounted(() => {
|
|||||||
|
|
||||||
<template>
|
<template>
|
||||||
<div class="page">
|
<div class="page">
|
||||||
<DataTablePage title="项目空间" :data="projects" :loading="loading" searchable search-fields="name,code">
|
<DataTablePage title="项目空间" :data="projects" :loading="loading" searchable :search-fields="['name', 'code']">
|
||||||
<template #toolbar-extra>
|
<template #toolbar-extra>
|
||||||
<el-select v-model="tenantId" placeholder="租户" style="width: 160px" @change="load">
|
<el-select v-model="tenantId" placeholder="租户" style="width: 160px" @change="load">
|
||||||
<el-option v-for="t in tenantOptions" :key="t.value" :label="t.label" :value="t.value" />
|
<el-option v-for="t in tenantOptions" :key="t.value" :label="t.label" :value="t.value" />
|
||||||
@@ -69,27 +99,25 @@ onMounted(() => {
|
|||||||
<el-button type="primary" :icon="Plus" @click="showCreate = true">新建项目</el-button>
|
<el-button type="primary" :icon="Plus" @click="showCreate = true">新建项目</el-button>
|
||||||
</template>
|
</template>
|
||||||
<template #columns>
|
<template #columns>
|
||||||
<el-table-column prop="name" label="项目名" min-width="140" />
|
<el-table-column prop="name" label="项目名称" min-width="140" />
|
||||||
<el-table-column prop="code" label="编码" min-width="100" />
|
<el-table-column prop="code" label="编码 ID" min-width="100" />
|
||||||
<el-table-column prop="status" label="状态" min-width="100" />
|
<el-table-column prop="status" label="状态" min-width="100" />
|
||||||
<el-table-column prop="description" label="描述" min-width="200" show-overflow-tooltip />
|
<el-table-column prop="description" label="描述" min-width="200" show-overflow-tooltip />
|
||||||
<el-table-column prop="create_time" label="创建时间" min-width="180" />
|
<el-table-column prop="create_time" label="创建时间" min-width="180" />
|
||||||
</template>
|
</template>
|
||||||
<template #actions="{ row }">
|
<template #actions="{ row }">
|
||||||
<el-button link type="primary" @click="openDetail(row.id)">详情</el-button>
|
<el-button link type="primary" @click="openDetail(row.id)">详情</el-button>
|
||||||
|
<el-button link type="danger" @click="handleDelete(asProject(row))">删除</el-button>
|
||||||
</template>
|
</template>
|
||||||
</DataTablePage>
|
</DataTablePage>
|
||||||
<el-dialog v-model="showCreate" title="新建项目" width="520px">
|
<el-dialog v-model="showCreate" title="新建项目" width="520px">
|
||||||
<el-form label-width="90px">
|
<el-form label-width="90px">
|
||||||
<el-form-item label="名称" required>
|
<el-form-item label="名称" required>
|
||||||
<el-input v-model="form.name" placeholder="项目名" />
|
<el-input v-model="form.name" placeholder="项目名称" />
|
||||||
</el-form-item>
|
</el-form-item>
|
||||||
<el-form-item label="编码" required>
|
<el-form-item label="编码 ID" required>
|
||||||
<el-input v-model="form.code" placeholder="project code" />
|
<el-select v-model="form.tenant_id" style="width: 100%" placeholder="选择租户编码">
|
||||||
</el-form-item>
|
<el-option v-for="t in tenantCodeOptions" :key="t.value" :label="t.label" :value="t.tenantId" />
|
||||||
<el-form-item label="租户">
|
|
||||||
<el-select v-model="form.tenant_id" style="width: 100%">
|
|
||||||
<el-option v-for="t in tenantOptions" :key="t.value" :label="t.label" :value="t.value" />
|
|
||||||
</el-select>
|
</el-select>
|
||||||
</el-form-item>
|
</el-form-item>
|
||||||
<el-form-item label="描述">
|
<el-form-item label="描述">
|
||||||
|
|||||||
@@ -8,13 +8,14 @@ import TrainingTaskOverview from './training-log/TrainingTaskOverview.vue'
|
|||||||
import { usePolling } from '@/composables/usePolling'
|
import { usePolling } from '@/composables/usePolling'
|
||||||
import '@/plugins/echarts-training-log'
|
import '@/plugins/echarts-training-log'
|
||||||
import { useModelsStore } from '@/stores/models'
|
import { useModelsStore } from '@/stores/models'
|
||||||
import { getFineTune, getFineTuneDiagnostics, getFineTuneLogs, type TrainingDiagnostic } from '@/api/modules/fineTune'
|
import { getFineTune, getFineTuneDiagnostics, getFineTuneLogs, getFineTuneMetrics, type TrainingDiagnostic } from '@/api/modules/fineTune'
|
||||||
import { getTrainingLogFiles, getTrainingLogContent } from '@/api/modules/log'
|
import { getTrainingLogFiles, getTrainingLogContent } from '@/api/modules/log'
|
||||||
import { getDataset } from '@/api/modules/dataset'
|
import { getDataset } from '@/api/modules/dataset'
|
||||||
import { getSystemInfo } from '@/api/modules/system'
|
import { getSystemInfo } from '@/api/modules/system'
|
||||||
import { TRAIN_TYPE_MAP, TRAIN_METHOD_MAP } from '@/constants'
|
import { TRAIN_TYPE_MAP, TRAIN_METHOD_MAP } from '@/constants'
|
||||||
import {
|
import {
|
||||||
buildMetricChartOption,
|
buildMetricChartOption,
|
||||||
|
metricsFromApi,
|
||||||
parseTrainingLog,
|
parseTrainingLog,
|
||||||
resolveTrainingLogFile,
|
resolveTrainingLogFile,
|
||||||
} from './training-log/trainingLogModel'
|
} from './training-log/trainingLogModel'
|
||||||
@@ -42,6 +43,7 @@ const loading = ref(true)
|
|||||||
|
|
||||||
// 训练指标数据(ECharts 接收 number[],下标即 step)
|
// 训练指标数据(ECharts 接收 number[],下标即 step)
|
||||||
const metricData = reactive({
|
const metricData = reactive({
|
||||||
|
steps: [] as number[],
|
||||||
loss: [] as number[],
|
loss: [] as number[],
|
||||||
gradNorm: [] as number[],
|
gradNorm: [] as number[],
|
||||||
lr: [] as number[],
|
lr: [] as number[],
|
||||||
@@ -62,9 +64,9 @@ const gpuExpanded = ref(false)
|
|||||||
let refreshInFlight = false
|
let refreshInFlight = false
|
||||||
|
|
||||||
/** 三个曲线的 ECharts 配置(响应式,数据变化自动重绘) */
|
/** 三个曲线的 ECharts 配置(响应式,数据变化自动重绘) */
|
||||||
const lossChartOption = computed(() => buildMetricChartOption('Loss', metricData.loss, '#4f46e5'))
|
const lossChartOption = computed(() => buildMetricChartOption('Loss', metricData.loss, metricData.steps, '#4f46e5'))
|
||||||
const gradChartOption = computed(() => buildMetricChartOption('Grad Norm', metricData.gradNorm, '#3b82f6'))
|
const gradChartOption = computed(() => buildMetricChartOption('Grad Norm', metricData.gradNorm, metricData.steps, '#3b82f6'))
|
||||||
const lrChartOption = computed(() => buildMetricChartOption('Learning Rate', metricData.lr, '#14b8a6', true))
|
const lrChartOption = computed(() => buildMetricChartOption('Learning Rate', metricData.lr, metricData.steps, '#14b8a6', true))
|
||||||
const baseModelName = computed(() => task.value?.base_model != null
|
const baseModelName = computed(() => task.value?.base_model != null
|
||||||
? modelsStore.getModelName(task.value.base_model)
|
? modelsStore.getModelName(task.value.base_model)
|
||||||
: '未配置')
|
: '未配置')
|
||||||
@@ -77,10 +79,10 @@ const trainingMethodName = computed(() => task.value?.train_method
|
|||||||
const taskGpuLabel = computed(() => task.value?.gpus?.length
|
const taskGpuLabel = computed(() => task.value?.gpus?.length
|
||||||
? task.value.gpus.map((gpuId) => `GPU ${gpuId}`).join('、')
|
? task.value.gpus.map((gpuId) => `GPU ${gpuId}`).join('、')
|
||||||
: '未配置')
|
: '未配置')
|
||||||
const latestLoss = computed(() => metricData.loss[metricData.loss.length - 1])
|
const latestLoss = computed(() => lastFinite(metricData.loss))
|
||||||
const latestGradNorm = computed(() => metricData.gradNorm[metricData.gradNorm.length - 1])
|
const latestGradNorm = computed(() => lastFinite(metricData.gradNorm))
|
||||||
const latestLearningRate = computed(() => metricData.lr[metricData.lr.length - 1])
|
const latestLearningRate = computed(() => lastFinite(metricData.lr))
|
||||||
const latestEpoch = computed(() => metricData.epoch[metricData.epoch.length - 1])
|
const latestEpoch = computed(() => lastFinite(metricData.epoch))
|
||||||
const logLineCount = computed(() => logContent.value ? logContent.value.split(/\r?\n/).length : 0)
|
const logLineCount = computed(() => logContent.value ? logContent.value.split(/\r?\n/).length : 0)
|
||||||
const taskGpuItems = computed<TaskGpuItem[]>(() => (task.value?.gpus ?? []).map((gpuId) => {
|
const taskGpuItems = computed<TaskGpuItem[]>(() => (task.value?.gpus ?? []).map((gpuId) => {
|
||||||
const index = Number(gpuId)
|
const index = Number(gpuId)
|
||||||
@@ -123,7 +125,7 @@ const gpuRefreshState = computed(() => {
|
|||||||
})
|
})
|
||||||
return gpuLoadError.value
|
return gpuLoadError.value
|
||||||
? `更新失败 · 最后更新 ${updateTime}`
|
? `更新失败 · 最后更新 ${updateTime}`
|
||||||
: `${updateTime} 更新 · 每 5 秒刷新`
|
: `${updateTime} 更新 · 每 3 秒刷新`
|
||||||
})
|
})
|
||||||
|
|
||||||
function formatMetric(value?: number, scientific = false) {
|
function formatMetric(value?: number, scientific = false) {
|
||||||
@@ -131,6 +133,13 @@ function formatMetric(value?: number, scientific = false) {
|
|||||||
return scientific ? value.toExponential(2) : value.toFixed(4).replace(/0+$/, '').replace(/\.$/, '')
|
return scientific ? value.toExponential(2) : value.toFixed(4).replace(/0+$/, '').replace(/\.$/, '')
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function lastFinite(values: number[]) {
|
||||||
|
for (let index = values.length - 1; index >= 0; index -= 1) {
|
||||||
|
if (Number.isFinite(values[index])) return values[index]
|
||||||
|
}
|
||||||
|
return undefined
|
||||||
|
}
|
||||||
|
|
||||||
function safePercent(value?: number) {
|
function safePercent(value?: number) {
|
||||||
return Math.round(Math.min(100, Math.max(0, Number(value || 0))))
|
return Math.round(Math.min(100, Math.max(0, Number(value || 0))))
|
||||||
}
|
}
|
||||||
@@ -216,6 +225,7 @@ const isLoraMethod = computed(() =>
|
|||||||
function applyLogContent(content: string) {
|
function applyLogContent(content: string) {
|
||||||
const parsed = parseTrainingLog(content)
|
const parsed = parseTrainingLog(content)
|
||||||
logContent.value = content
|
logContent.value = content
|
||||||
|
metricData.steps = parsed.metrics.steps
|
||||||
metricData.loss = parsed.metrics.loss
|
metricData.loss = parsed.metrics.loss
|
||||||
metricData.gradNorm = parsed.metrics.gradNorm
|
metricData.gradNorm = parsed.metrics.gradNorm
|
||||||
metricData.lr = parsed.metrics.lr
|
metricData.lr = parsed.metrics.lr
|
||||||
@@ -223,6 +233,26 @@ function applyLogContent(content: string) {
|
|||||||
Object.assign(summary, parsed.summary)
|
Object.assign(summary, parsed.summary)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function applyMetricData(metrics = { steps: [] as number[], loss: [] as number[], gradNorm: [] as number[], lr: [] as number[], epoch: [] as number[] }) {
|
||||||
|
metricData.steps = metrics.steps
|
||||||
|
metricData.loss = metrics.loss
|
||||||
|
metricData.gradNorm = metrics.gradNorm
|
||||||
|
metricData.lr = metrics.lr
|
||||||
|
metricData.epoch = metrics.epoch
|
||||||
|
}
|
||||||
|
|
||||||
|
async function loadMetrics(currentTask: FineTuneTask) {
|
||||||
|
try {
|
||||||
|
const points = await getFineTuneMetrics(currentTask.id)
|
||||||
|
const parsed = metricsFromApi(points || [])
|
||||||
|
if (parsed.loss.length || parsed.gradNorm.length || parsed.lr.length) {
|
||||||
|
applyMetricData(parsed)
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
// 日志解析结果会作为兜底曲线数据。
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
async function loadLog(currentTask: FineTuneTask) {
|
async function loadLog(currentTask: FineTuneTask) {
|
||||||
try {
|
try {
|
||||||
const runtime = await getFineTuneLogs(currentTask.id, { tail_lines: 800 })
|
const runtime = await getFineTuneLogs(currentTask.id, { tail_lines: 800 })
|
||||||
@@ -277,6 +307,7 @@ async function refreshAll() {
|
|||||||
? loadDataset(currentTask.train_dataset_id)
|
? loadDataset(currentTask.train_dataset_id)
|
||||||
: Promise.resolve()
|
: Promise.resolve()
|
||||||
await Promise.all([datasetPromise, loadLog(currentTask), loadGpuStatus(), loadDiagnostics(currentTask)])
|
await Promise.all([datasetPromise, loadLog(currentTask), loadGpuStatus(), loadDiagnostics(currentTask)])
|
||||||
|
await loadMetrics(currentTask)
|
||||||
} finally {
|
} finally {
|
||||||
loading.value = false
|
loading.value = false
|
||||||
refreshInFlight = false
|
refreshInFlight = false
|
||||||
@@ -523,7 +554,7 @@ onMounted(async () => {
|
|||||||
|
|
||||||
<!-- 训练曲线 -->
|
<!-- 训练曲线 -->
|
||||||
<PageCard class="metrics-panel" title="训练曲线" subtitle="持续监控模型收敛情况与学习率变化">
|
<PageCard class="metrics-panel" title="训练曲线" subtitle="持续监控模型收敛情况与学习率变化">
|
||||||
<template #extra><span class="refresh-state">每 5 秒刷新</span></template>
|
<template #extra><span class="refresh-state">每 3 秒刷新</span></template>
|
||||||
<div class="chart-list" aria-label="训练指标曲线">
|
<div class="chart-list" aria-label="训练指标曲线">
|
||||||
<section class="chart-section">
|
<section class="chart-section">
|
||||||
<div class="chart-section-header">
|
<div class="chart-section-header">
|
||||||
@@ -560,7 +591,7 @@ onMounted(async () => {
|
|||||||
|
|
||||||
<!-- 原始日志 -->
|
<!-- 原始日志 -->
|
||||||
<PageCard class="log-card" title="训练日志" subtitle="查看训练任务的原始运行输出">
|
<PageCard class="log-card" title="训练日志" subtitle="查看训练任务的原始运行输出">
|
||||||
<template #extra><span class="log-meta">{{ logLineCount }} 行 · 每 5 秒刷新</span></template>
|
<template #extra><span class="log-meta">{{ logLineCount }} 行 · 每 3 秒刷新</span></template>
|
||||||
<pre class="log-pre">{{ logContent || '暂无日志' }}</pre>
|
<pre class="log-pre">{{ logContent || '暂无日志' }}</pre>
|
||||||
</PageCard>
|
</PageCard>
|
||||||
</template>
|
</template>
|
||||||
|
|||||||
@@ -54,6 +54,18 @@ function isSelf(row: SystemUser) {
|
|||||||
return row.username === currentUsername.value
|
return row.username === currentUsername.value
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function asSystemUser(row: unknown): SystemUser {
|
||||||
|
return row as SystemUser
|
||||||
|
}
|
||||||
|
|
||||||
|
function userPermissions(row: unknown): PermissionCode[] {
|
||||||
|
return (asSystemUser(row).permissions || []) as PermissionCode[]
|
||||||
|
}
|
||||||
|
|
||||||
|
function permissionLabel(code: PermissionCode): string {
|
||||||
|
return PERMISSION_LABELS[code] || code
|
||||||
|
}
|
||||||
|
|
||||||
// ---------- 启停 ----------
|
// ---------- 启停 ----------
|
||||||
async function toggleStatus(row: SystemUser, next: boolean) {
|
async function toggleStatus(row: SystemUser, next: boolean) {
|
||||||
const nextStatus: UserStatus = next ? 'active' : 'disabled'
|
const nextStatus: UserStatus = next ? 'active' : 'disabled'
|
||||||
@@ -162,31 +174,31 @@ async function removeUser(row: SystemUser) {
|
|||||||
<el-table-column prop="role" label="角色" width="120" />
|
<el-table-column prop="role" label="角色" width="120" />
|
||||||
<el-table-column label="状态" width="130">
|
<el-table-column label="状态" width="130">
|
||||||
<template #default="{ row }">
|
<template #default="{ row }">
|
||||||
<el-tag :type="statusTagType(row.status)" size="small">{{ statusLabel(row.status) }}</el-tag>
|
<el-tag :type="statusTagType(asSystemUser(row).status)" size="small">{{ statusLabel(asSystemUser(row).status) }}</el-tag>
|
||||||
</template>
|
</template>
|
||||||
</el-table-column>
|
</el-table-column>
|
||||||
<el-table-column label="页面权限" min-width="160">
|
<el-table-column label="页面权限" min-width="160">
|
||||||
<template #default="{ row }">
|
<template #default="{ row }">
|
||||||
<el-tag
|
<el-tag
|
||||||
v-for="p in (row.permissions || []).slice(0, 3)"
|
v-for="p in userPermissions(row).slice(0, 3)"
|
||||||
:key="p"
|
:key="p"
|
||||||
size="small"
|
size="small"
|
||||||
type="info"
|
type="info"
|
||||||
class="perm-tag"
|
class="perm-tag"
|
||||||
>{{ PERMISSION_LABELS[p] || p }}</el-tag>
|
>{{ permissionLabel(p) }}</el-tag>
|
||||||
<span v-if="(row.permissions || []).length > 3" class="perm-more">
|
<span v-if="userPermissions(row).length > 3" class="perm-more">
|
||||||
+{{ (row.permissions || []).length - 3 }}
|
+{{ userPermissions(row).length - 3 }}
|
||||||
</span>
|
</span>
|
||||||
<span v-if="!(row.permissions || []).length" class="perm-more">无</span>
|
<span v-if="!userPermissions(row).length" class="perm-more">无</span>
|
||||||
</template>
|
</template>
|
||||||
</el-table-column>
|
</el-table-column>
|
||||||
<el-table-column prop="create_time" label="创建时间" min-width="180" />
|
<el-table-column prop="create_time" label="创建时间" min-width="180" />
|
||||||
<el-table-column label="操作" width="260" fixed="right">
|
<el-table-column label="操作" width="260" fixed="right">
|
||||||
<template #default="{ row }">
|
<template #default="{ row }">
|
||||||
<el-switch
|
<el-switch
|
||||||
:model-value="row.status === 'active'"
|
:model-value="asSystemUser(row).status === 'active'"
|
||||||
:disabled="row.protected || isSelf(row)"
|
:disabled="asSystemUser(row).protected || isSelf(asSystemUser(row))"
|
||||||
@change="(v: any) => toggleStatus(row, v)"
|
@change="(v: any) => toggleStatus(asSystemUser(row), v)"
|
||||||
inline-prompt
|
inline-prompt
|
||||||
active-text="启用"
|
active-text="启用"
|
||||||
inactive-text="停用"
|
inactive-text="停用"
|
||||||
@@ -194,19 +206,19 @@ async function removeUser(row: SystemUser) {
|
|||||||
<el-button
|
<el-button
|
||||||
link
|
link
|
||||||
type="primary"
|
type="primary"
|
||||||
:disabled="row.protected"
|
:disabled="asSystemUser(row).protected"
|
||||||
@click="openResetPwd(row)"
|
@click="openResetPwd(asSystemUser(row))"
|
||||||
>重置密码</el-button>
|
>重置密码</el-button>
|
||||||
<el-button
|
<el-button
|
||||||
link
|
link
|
||||||
type="primary"
|
type="primary"
|
||||||
@click="openPerms(row)"
|
@click="openPerms(asSystemUser(row))"
|
||||||
>页面权限</el-button>
|
>页面权限</el-button>
|
||||||
<el-button
|
<el-button
|
||||||
link
|
link
|
||||||
type="danger"
|
type="danger"
|
||||||
:disabled="row.protected || isSelf(row)"
|
:disabled="asSystemUser(row).protected || isSelf(asSystemUser(row))"
|
||||||
@click="removeUser(row)"
|
@click="removeUser(asSystemUser(row))"
|
||||||
>删除</el-button>
|
>删除</el-button>
|
||||||
</template>
|
</template>
|
||||||
</el-table-column>
|
</el-table-column>
|
||||||
|
|||||||
@@ -1,7 +1,9 @@
|
|||||||
import type { EChartsOption } from 'echarts'
|
import type { EChartsOption } from 'echarts'
|
||||||
import type { FineTuneTask, TrainingLogFile } from '@/types'
|
import type { FineTuneTask, TrainingLogFile } from '@/types'
|
||||||
|
import type { FineTuneMetricPoint } from '@/api/modules/fineTune'
|
||||||
|
|
||||||
export interface TrainingMetricData {
|
export interface TrainingMetricData {
|
||||||
|
steps: number[]
|
||||||
loss: number[]
|
loss: number[]
|
||||||
gradNorm: number[]
|
gradNorm: number[]
|
||||||
lr: number[]
|
lr: number[]
|
||||||
@@ -26,7 +28,7 @@ function escapeRegExp(value: string) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
function extractNumber(source: string, key: string) {
|
function extractNumber(source: string, key: string) {
|
||||||
const match = source.match(new RegExp(`['"]?${escapeRegExp(key)}['"]?\\s*:\\s*(${NUMBER_SOURCE})`, 'i'))
|
const match = source.match(new RegExp(`['"]?${escapeRegExp(key)}['"]?\\s*(?:=|:)\\s*(${NUMBER_SOURCE})`, 'i'))
|
||||||
return match ? Number(match[1]) : undefined
|
return match ? Number(match[1]) : undefined
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -57,24 +59,44 @@ export function resolveTrainingLogFile(
|
|||||||
|
|
||||||
/** 解析日志中的逐步指标。字段顺序和常见数值格式均不受限制。 */
|
/** 解析日志中的逐步指标。字段顺序和常见数值格式均不受限制。 */
|
||||||
export function parseTrainingMetrics(text: string): TrainingMetricData {
|
export function parseTrainingMetrics(text: string): TrainingMetricData {
|
||||||
const metrics: TrainingMetricData = { loss: [], gradNorm: [], lr: [], epoch: [] }
|
const metrics: TrainingMetricData = { steps: [], loss: [], gradNorm: [], lr: [], epoch: [] }
|
||||||
const blocks = text.match(/\{[^{}\r\n]*\}/g) || []
|
const candidates = text
|
||||||
|
.split(/\r?\n/)
|
||||||
|
.flatMap((line) => {
|
||||||
|
const blocks = line.match(/\{[^{}\r\n]*\}/g)
|
||||||
|
return blocks?.length ? blocks.map((block) => `${line} ${block}`) : [line]
|
||||||
|
})
|
||||||
|
|
||||||
for (const block of blocks) {
|
for (const [index, line] of candidates.entries()) {
|
||||||
const loss = extractNumber(block, 'loss')
|
const loss = extractNumber(line, 'loss')
|
||||||
const gradNorm = extractNumber(block, 'grad_norm')
|
const gradNorm = extractNumber(line, 'grad_norm')
|
||||||
const learningRate = extractNumber(block, 'learning_rate')
|
const learningRate = extractNumber(line, 'learning_rate')
|
||||||
const epoch = extractNumber(block, 'epoch')
|
const epoch = extractNumber(line, 'epoch')
|
||||||
if (loss == null || gradNorm == null || learningRate == null) continue
|
if (loss == null && gradNorm == null && learningRate == null) continue
|
||||||
metrics.loss.push(loss)
|
metrics.steps.push(extractNumber(line, 'step') ?? metrics.steps.length + index + 1)
|
||||||
metrics.gradNorm.push(gradNorm)
|
metrics.loss.push(loss ?? Number.NaN)
|
||||||
metrics.lr.push(learningRate)
|
metrics.gradNorm.push(gradNorm ?? Number.NaN)
|
||||||
if (epoch != null) metrics.epoch.push(epoch)
|
metrics.lr.push(learningRate ?? Number.NaN)
|
||||||
|
metrics.epoch.push(epoch ?? Number.NaN)
|
||||||
}
|
}
|
||||||
|
|
||||||
return metrics
|
return metrics
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export function metricsFromApi(points: FineTuneMetricPoint[]): TrainingMetricData {
|
||||||
|
const metrics: TrainingMetricData = { steps: [], loss: [], gradNorm: [], lr: [], epoch: [] }
|
||||||
|
for (const [index, point] of points.entries()) {
|
||||||
|
const hasMetric = point.loss != null || point.grad_norm != null || point.learning_rate != null
|
||||||
|
if (!hasMetric) continue
|
||||||
|
metrics.steps.push(Number(point.step || index + 1))
|
||||||
|
metrics.loss.push(point.loss == null ? Number.NaN : Number(point.loss))
|
||||||
|
metrics.gradNorm.push(point.grad_norm == null ? Number.NaN : Number(point.grad_norm))
|
||||||
|
metrics.lr.push(point.learning_rate == null ? Number.NaN : Number(point.learning_rate))
|
||||||
|
metrics.epoch.push(point.epoch == null ? Number.NaN : Number(point.epoch))
|
||||||
|
}
|
||||||
|
return metrics
|
||||||
|
}
|
||||||
|
|
||||||
/** 每次都返回新对象,日志截断或切换时不会残留上一轮汇总。 */
|
/** 每次都返回新对象,日志截断或切换时不会残留上一轮汇总。 */
|
||||||
export function parseTrainingSummary(text: string): TrainingSummary {
|
export function parseTrainingSummary(text: string): TrainingSummary {
|
||||||
const emptySummary: TrainingSummary = { epoch: '', trainLoss: '', runtime: '' }
|
const emptySummary: TrainingSummary = { epoch: '', trainLoss: '', runtime: '' }
|
||||||
@@ -102,11 +124,23 @@ export function parseTrainingLog(text: string): ParsedTrainingLog {
|
|||||||
export function buildMetricChartOption(
|
export function buildMetricChartOption(
|
||||||
label: string,
|
label: string,
|
||||||
data: number[],
|
data: number[],
|
||||||
|
steps: number[],
|
||||||
color: string,
|
color: string,
|
||||||
logScale = false,
|
logScale = false,
|
||||||
): EChartsOption {
|
): EChartsOption {
|
||||||
|
const visibleData = data.map((value) => (Number.isFinite(value) ? value : null))
|
||||||
return {
|
return {
|
||||||
grid: { top: 24, right: 20, bottom: 56, left: 56 },
|
grid: { top: 24, right: 20, bottom: 56, left: 56 },
|
||||||
|
graphic: visibleData.some((value) => value != null)
|
||||||
|
? []
|
||||||
|
: [
|
||||||
|
{
|
||||||
|
type: 'text',
|
||||||
|
left: 'center',
|
||||||
|
top: 'middle',
|
||||||
|
style: { text: '暂无训练指标数据', fill: '#94a3b8', fontSize: 13 },
|
||||||
|
},
|
||||||
|
],
|
||||||
tooltip: {
|
tooltip: {
|
||||||
trigger: 'axis',
|
trigger: 'axis',
|
||||||
axisPointer: { type: 'cross' },
|
axisPointer: { type: 'cross' },
|
||||||
@@ -116,6 +150,7 @@ export function buildMetricChartOption(
|
|||||||
},
|
},
|
||||||
xAxis: {
|
xAxis: {
|
||||||
type: 'category',
|
type: 'category',
|
||||||
|
data: steps.map((step, index) => (Number.isFinite(step) ? String(step) : String(index + 1))),
|
||||||
boundaryGap: false,
|
boundaryGap: false,
|
||||||
name: 'Step',
|
name: 'Step',
|
||||||
nameTextStyle: { color: '#94a3b8', fontSize: 11 },
|
nameTextStyle: { color: '#94a3b8', fontSize: 11 },
|
||||||
@@ -142,7 +177,7 @@ export function buildMetricChartOption(
|
|||||||
{
|
{
|
||||||
name: label,
|
name: label,
|
||||||
type: 'line',
|
type: 'line',
|
||||||
data,
|
data: visibleData,
|
||||||
smooth: true,
|
smooth: true,
|
||||||
symbol: 'none',
|
symbol: 'none',
|
||||||
lineStyle: { width: 2, color },
|
lineStyle: { width: 2, color },
|
||||||
|
|||||||
@@ -1,25 +1,42 @@
|
|||||||
<script setup lang="ts">
|
<script setup lang="ts">
|
||||||
import { onMounted, ref } from 'vue'
|
import { onMounted, reactive, ref } from 'vue'
|
||||||
import { useRoute, useRouter } from 'vue-router'
|
import { useRoute, useRouter } from 'vue-router'
|
||||||
import { ElMessage } from 'element-plus'
|
import { ElMessage } from 'element-plus'
|
||||||
import DataTablePage from '@/components/DataTablePage.vue'
|
|
||||||
import { getTenant, setTenantQuota, type Tenant } from '@/api/modules/tenant'
|
import { getTenant, setTenantQuota, type Tenant } from '@/api/modules/tenant'
|
||||||
import { getProjects, type Project } from '@/api/modules/project'
|
|
||||||
|
|
||||||
const route = useRoute()
|
const route = useRoute()
|
||||||
const router = useRouter()
|
const router = useRouter()
|
||||||
const tenant = ref<Tenant | null>(null)
|
const tenant = ref<Tenant | null>(null)
|
||||||
const projects = ref<Project[]>([])
|
|
||||||
const loading = ref(false)
|
const loading = ref(false)
|
||||||
const quotaText = ref('')
|
const quotaForm = reactive({ gpu: 0, storage: 0, maxProjects: 0 })
|
||||||
|
|
||||||
|
function parseQuota(quota: Record<string, unknown> | undefined | null) {
|
||||||
|
const q = quota || {}
|
||||||
|
return {
|
||||||
|
gpu: Number(q.gpu || q.gpu_quota || 0),
|
||||||
|
storage: Number(q.storage || q.storage_quota || 0),
|
||||||
|
maxProjects: Number(q.max_projects || 0),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function formatQuota(quota: Record<string, unknown> | undefined | null) {
|
||||||
|
const q = parseQuota(quota)
|
||||||
|
const parts: string[] = []
|
||||||
|
if (q.gpu > 0) parts.push(`GPU ${q.gpu}`)
|
||||||
|
if (q.storage > 0) parts.push(`存储 ${q.storage}GB`)
|
||||||
|
if (q.maxProjects > 0) parts.push(`项目 ${q.maxProjects}`)
|
||||||
|
return parts.length ? parts.join(' | ') : '—'
|
||||||
|
}
|
||||||
|
|
||||||
async function load() {
|
async function load() {
|
||||||
const id = route.params.id as string
|
const id = route.params.id as string
|
||||||
loading.value = true
|
loading.value = true
|
||||||
try {
|
try {
|
||||||
tenant.value = await getTenant(id)
|
tenant.value = await getTenant(id)
|
||||||
projects.value = await getProjects(id)
|
const q = parseQuota(tenant.value?.quota)
|
||||||
quotaText.value = JSON.stringify(tenant.value?.quota || {})
|
quotaForm.gpu = q.gpu
|
||||||
|
quotaForm.storage = q.storage
|
||||||
|
quotaForm.maxProjects = q.maxProjects
|
||||||
} finally {
|
} finally {
|
||||||
loading.value = false
|
loading.value = false
|
||||||
}
|
}
|
||||||
@@ -27,18 +44,13 @@ async function load() {
|
|||||||
|
|
||||||
async function saveQuota() {
|
async function saveQuota() {
|
||||||
if (!tenant.value) return
|
if (!tenant.value) return
|
||||||
try {
|
const quota: Record<string, unknown> = {}
|
||||||
const q = JSON.parse(quotaText.value || '{}')
|
if (quotaForm.gpu > 0) quota.gpu = quotaForm.gpu
|
||||||
await setTenantQuota(tenant.value.id, q)
|
if (quotaForm.storage > 0) quota.storage = quotaForm.storage
|
||||||
|
if (quotaForm.maxProjects > 0) quota.max_projects = quotaForm.maxProjects
|
||||||
|
await setTenantQuota(tenant.value.id, quota)
|
||||||
ElMessage.success('配额已保存')
|
ElMessage.success('配额已保存')
|
||||||
load()
|
load()
|
||||||
} catch {
|
|
||||||
ElMessage.error('配额需为合法 JSON')
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
function openProject(id: string) {
|
|
||||||
router.push(`/projects/${id}`)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
onMounted(load)
|
onMounted(load)
|
||||||
@@ -55,31 +67,30 @@ onMounted(load)
|
|||||||
<template #header>基本信息</template>
|
<template #header>基本信息</template>
|
||||||
<el-descriptions :column="2" border>
|
<el-descriptions :column="2" border>
|
||||||
<el-descriptions-item label="名称">{{ tenant?.name }}</el-descriptions-item>
|
<el-descriptions-item label="名称">{{ tenant?.name }}</el-descriptions-item>
|
||||||
<el-descriptions-item label="编码">{{ tenant?.code }}</el-descriptions-item>
|
<el-descriptions-item label="用户ID">{{ tenant?.code }}</el-descriptions-item>
|
||||||
<el-descriptions-item label="状态">{{ tenant?.status }}</el-descriptions-item>
|
<el-descriptions-item label="状态">{{ tenant?.status }}</el-descriptions-item>
|
||||||
<el-descriptions-item label="创建时间">{{ tenant?.create_time }}</el-descriptions-item>
|
<el-descriptions-item label="创建时间">{{ tenant?.create_time }}</el-descriptions-item>
|
||||||
|
<el-descriptions-item label="配额">{{ formatQuota(tenant?.quota) }}</el-descriptions-item>
|
||||||
</el-descriptions>
|
</el-descriptions>
|
||||||
<el-divider />
|
<el-divider />
|
||||||
<div class="quota-edit">
|
<div class="quota-edit">
|
||||||
<span class="label">配额 JSON</span>
|
<span class="label">配额设置(0 表示不限制)</span>
|
||||||
<el-input v-model="quotaText" type="textarea" :rows="3" />
|
<el-form label-width="100px">
|
||||||
|
<el-form-item label="GPU 数量">
|
||||||
|
<el-input-number v-model="quotaForm.gpu" :min="0" :step="1" />
|
||||||
|
</el-form-item>
|
||||||
|
<el-form-item label="存储配额(GB)">
|
||||||
|
<el-input-number v-model="quotaForm.storage" :min="0" :step="10" />
|
||||||
|
</el-form-item>
|
||||||
|
<el-form-item label="最大项目数">
|
||||||
|
<el-input-number v-model="quotaForm.maxProjects" :min="0" :step="1" />
|
||||||
|
</el-form-item>
|
||||||
|
<el-form-item>
|
||||||
<el-button type="primary" @click="saveQuota">保存配额</el-button>
|
<el-button type="primary" @click="saveQuota">保存配额</el-button>
|
||||||
|
</el-form-item>
|
||||||
|
</el-form>
|
||||||
</div>
|
</div>
|
||||||
</el-card>
|
</el-card>
|
||||||
<el-card class="section">
|
|
||||||
<template #header>项目空间</template>
|
|
||||||
<DataTablePage title="项目空间" :data="projects">
|
|
||||||
<template #columns>
|
|
||||||
<el-table-column prop="name" label="项目名" min-width="140" />
|
|
||||||
<el-table-column prop="code" label="编码" min-width="100" />
|
|
||||||
<el-table-column prop="status" label="状态" min-width="100" />
|
|
||||||
<el-table-column prop="create_time" label="创建时间" min-width="180" />
|
|
||||||
</template>
|
|
||||||
<template #actions="{ row }">
|
|
||||||
<el-button link type="primary" @click="openProject(row.id)">打开</el-button>
|
|
||||||
</template>
|
|
||||||
</DataTablePage>
|
|
||||||
</el-card>
|
|
||||||
</div>
|
</div>
|
||||||
</template>
|
</template>
|
||||||
|
|
||||||
|
|||||||
@@ -1,16 +1,37 @@
|
|||||||
<script setup lang="ts">
|
<script setup lang="ts">
|
||||||
import { onMounted, ref } from 'vue'
|
import { onMounted, reactive, ref } from 'vue'
|
||||||
import { useRouter } from 'vue-router'
|
import { useRouter } from 'vue-router'
|
||||||
import { ElMessage, ElMessageBox } from 'element-plus'
|
import { ElMessage, ElMessageBox } from 'element-plus'
|
||||||
import { Plus } from '@element-plus/icons-vue'
|
import { Plus } from '@element-plus/icons-vue'
|
||||||
import DataTablePage from '@/components/DataTablePage.vue'
|
import DataTablePage from '@/components/DataTablePage.vue'
|
||||||
import { createTenant, getTenants, setTenantQuota, type Tenant } from '@/api/modules/tenant'
|
import { createTenant, deleteTenant, getTenants, setTenantQuota, type Tenant } from '@/api/modules/tenant'
|
||||||
|
|
||||||
const router = useRouter()
|
const router = useRouter()
|
||||||
const loading = ref(false)
|
const loading = ref(false)
|
||||||
const tenants = ref<Tenant[]>([])
|
const tenants = ref<Tenant[]>([])
|
||||||
const showCreate = ref(false)
|
const showCreate = ref(false)
|
||||||
const form = ref({ name: '', code: '', quota: '' as string })
|
const showQuota = ref(false)
|
||||||
|
const currentTenant = ref<Tenant | null>(null)
|
||||||
|
const form = ref({ name: '', code: '', gpu: 0, storage: 0, maxProjects: 0 })
|
||||||
|
const quotaForm = reactive({ gpu: 0, storage: 0, maxProjects: 0 })
|
||||||
|
|
||||||
|
function parseQuota(quota: Record<string, unknown> | undefined | null) {
|
||||||
|
const q = quota || {}
|
||||||
|
return {
|
||||||
|
gpu: Number(q.gpu || q.gpu_quota || 0),
|
||||||
|
storage: Number(q.storage || q.storage_quota || 0),
|
||||||
|
maxProjects: Number(q.max_projects || 0),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function formatQuota(quota: Record<string, unknown> | undefined | null) {
|
||||||
|
const q = parseQuota(quota)
|
||||||
|
const parts: string[] = []
|
||||||
|
if (q.gpu > 0) parts.push(`GPU ${q.gpu}`)
|
||||||
|
if (q.storage > 0) parts.push(`存储 ${q.storage}GB`)
|
||||||
|
if (q.maxProjects > 0) parts.push(`项目 ${q.maxProjects}`)
|
||||||
|
return parts.length ? parts.join(' | ') : '-'
|
||||||
|
}
|
||||||
|
|
||||||
async function load() {
|
async function load() {
|
||||||
loading.value = true
|
loading.value = true
|
||||||
@@ -25,40 +46,65 @@ function openDetail(id: string) {
|
|||||||
router.push(`/tenants/${id}`)
|
router.push(`/tenants/${id}`)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function asTenant(row: unknown): Tenant {
|
||||||
|
return row as Tenant
|
||||||
|
}
|
||||||
|
|
||||||
|
function quotaText(row: unknown): string {
|
||||||
|
const quota = asTenant(row).quota || {}
|
||||||
|
return Object.keys(quota).length ? JSON.stringify(quota) : '-'
|
||||||
|
}
|
||||||
|
|
||||||
async function submitCreate() {
|
async function submitCreate() {
|
||||||
if (!form.value.name) {
|
if (!form.value.name) {
|
||||||
ElMessage.warning('请填写租户名称')
|
ElMessage.warning('请填写租户名称')
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
let quota: Record<string, unknown> = {}
|
const quota: Record<string, unknown> = {}
|
||||||
if (form.value.quota) {
|
if (form.value.gpu > 0) quota.gpu = form.value.gpu
|
||||||
try {
|
if (form.value.storage > 0) quota.storage = form.value.storage
|
||||||
quota = JSON.parse(form.value.quota)
|
if (form.value.maxProjects > 0) quota.max_projects = form.value.maxProjects
|
||||||
} catch {
|
|
||||||
ElMessage.error('配额需为合法 JSON')
|
|
||||||
return
|
|
||||||
}
|
|
||||||
}
|
|
||||||
await createTenant({ name: form.value.name, code: form.value.code, quota })
|
await createTenant({ name: form.value.name, code: form.value.code, quota })
|
||||||
ElMessage.success('租户创建成功')
|
ElMessage.success('租户创建成功')
|
||||||
showCreate.value = false
|
showCreate.value = false
|
||||||
form.value = { name: '', code: '', quota: '' }
|
form.value = { name: '', code: '', gpu: 0, storage: 0, maxProjects: 0 }
|
||||||
load()
|
load()
|
||||||
}
|
}
|
||||||
|
|
||||||
async function setQuota(row: Tenant) {
|
function openQuotaDialog(row: Tenant) {
|
||||||
const input = await ElMessageBox.prompt('输入租户配额 JSON', '设置配额', {
|
currentTenant.value = row
|
||||||
inputValue: JSON.stringify(row.quota || {}),
|
const q = parseQuota(row.quota)
|
||||||
}).catch(() => null)
|
quotaForm.gpu = q.gpu
|
||||||
if (!input) return
|
quotaForm.storage = q.storage
|
||||||
try {
|
quotaForm.maxProjects = q.maxProjects
|
||||||
const q = JSON.parse(input.value)
|
showQuota.value = true
|
||||||
await setTenantQuota(row.id, q)
|
}
|
||||||
|
|
||||||
|
async function submitQuota() {
|
||||||
|
if (!currentTenant.value) return
|
||||||
|
const quota: Record<string, unknown> = {}
|
||||||
|
if (quotaForm.gpu > 0) quota.gpu = quotaForm.gpu
|
||||||
|
if (quotaForm.storage > 0) quota.storage = quotaForm.storage
|
||||||
|
if (quotaForm.maxProjects > 0) quota.max_projects = quotaForm.maxProjects
|
||||||
|
await setTenantQuota(currentTenant.value.id, quota)
|
||||||
ElMessage.success('配额已更新')
|
ElMessage.success('配额已更新')
|
||||||
|
showQuota.value = false
|
||||||
load()
|
load()
|
||||||
|
}
|
||||||
|
|
||||||
|
async function handleDelete(row: Tenant) {
|
||||||
|
try {
|
||||||
|
await ElMessageBox.confirm(
|
||||||
|
`确定要删除租户「${row.name}」吗?删除后相关数据将无法恢复。`,
|
||||||
|
'删除确认',
|
||||||
|
{ confirmButtonText: '确定删除', cancelButtonText: '取消', type: 'warning' },
|
||||||
|
)
|
||||||
} catch {
|
} catch {
|
||||||
ElMessage.error('无效的 JSON')
|
return
|
||||||
}
|
}
|
||||||
|
await deleteTenant(row.id)
|
||||||
|
ElMessage.success('租户已删除')
|
||||||
|
load()
|
||||||
}
|
}
|
||||||
|
|
||||||
onMounted(load)
|
onMounted(load)
|
||||||
@@ -72,30 +118,34 @@ onMounted(load)
|
|||||||
</template>
|
</template>
|
||||||
<template #columns>
|
<template #columns>
|
||||||
<el-table-column prop="name" label="租户名称" min-width="140" />
|
<el-table-column prop="name" label="租户名称" min-width="140" />
|
||||||
<el-table-column prop="code" label="编码" min-width="100" />
|
<el-table-column prop="code" label="租户 ID" min-width="100" />
|
||||||
<el-table-column label="配额" min-width="160">
|
|
||||||
<template #default="{ row }">
|
|
||||||
{{ Object.keys(row.quota || {}).length ? JSON.stringify(row.quota) : '—' }}
|
|
||||||
</template>
|
|
||||||
</el-table-column>
|
|
||||||
<el-table-column prop="status" label="状态" min-width="100" />
|
<el-table-column prop="status" label="状态" min-width="100" />
|
||||||
<el-table-column prop="create_time" label="创建时间" min-width="180" />
|
<el-table-column prop="create_time" label="创建时间" min-width="180" />
|
||||||
</template>
|
</template>
|
||||||
<template #actions="{ row }">
|
<template #actions="{ row }">
|
||||||
<el-button link type="primary" @click="openDetail(row.id)">详情</el-button>
|
<el-button link type="primary" @click="openDetail(row.id)">详情</el-button>
|
||||||
<el-button link type="primary" @click="setQuota(row)">配额</el-button>
|
<el-button link type="danger" @click="handleDelete(asTenant(row))">删除</el-button>
|
||||||
</template>
|
</template>
|
||||||
</DataTablePage>
|
</DataTablePage>
|
||||||
|
|
||||||
|
<!-- 新建租户弹窗 -->
|
||||||
<el-dialog v-model="showCreate" title="新建租户" width="520px">
|
<el-dialog v-model="showCreate" title="新建租户" width="520px">
|
||||||
<el-form label-width="90px">
|
<el-form label-width="100px">
|
||||||
<el-form-item label="名称" required>
|
<el-form-item label="名称" required>
|
||||||
<el-input v-model="form.name" placeholder="租户名称" />
|
<el-input v-model="form.name" placeholder="租户名称" />
|
||||||
</el-form-item>
|
</el-form-item>
|
||||||
<el-form-item label="编码">
|
<el-form-item label="用户ID">
|
||||||
<el-input v-model="form.code" placeholder="tenant code" />
|
<el-input v-model="form.code" placeholder="用户ID" />
|
||||||
</el-form-item>
|
</el-form-item>
|
||||||
<el-form-item label="配额 JSON">
|
<el-divider content-position="left">配额设置(可选,0 表示不限制)</el-divider>
|
||||||
<el-input v-model="form.quota" type="textarea" :rows="3" placeholder='{"gpu": 8}' />
|
<el-form-item label="GPU 数量">
|
||||||
|
<el-input-number v-model="form.gpu" :min="0" :step="1" placeholder="GPU 卡数" />
|
||||||
|
</el-form-item>
|
||||||
|
<el-form-item label="存储配额(GB)">
|
||||||
|
<el-input-number v-model="form.storage" :min="0" :step="10" placeholder="存储大小" />
|
||||||
|
</el-form-item>
|
||||||
|
<el-form-item label="最大项目数">
|
||||||
|
<el-input-number v-model="form.maxProjects" :min="0" :step="1" placeholder="项目上限" />
|
||||||
</el-form-item>
|
</el-form-item>
|
||||||
</el-form>
|
</el-form>
|
||||||
<template #footer>
|
<template #footer>
|
||||||
@@ -103,6 +153,25 @@ onMounted(load)
|
|||||||
<el-button type="primary" @click="submitCreate">创建</el-button>
|
<el-button type="primary" @click="submitCreate">创建</el-button>
|
||||||
</template>
|
</template>
|
||||||
</el-dialog>
|
</el-dialog>
|
||||||
|
|
||||||
|
<!-- 设置配额弹窗 -->
|
||||||
|
<el-dialog v-model="showQuota" title="设置配额" width="480px">
|
||||||
|
<el-form label-width="100px">
|
||||||
|
<el-form-item label="GPU 数量">
|
||||||
|
<el-input-number v-model="quotaForm.gpu" :min="0" :step="1" placeholder="GPU 卡数" />
|
||||||
|
</el-form-item>
|
||||||
|
<el-form-item label="存储配额(GB)">
|
||||||
|
<el-input-number v-model="quotaForm.storage" :min="0" :step="10" placeholder="存储大小" />
|
||||||
|
</el-form-item>
|
||||||
|
<el-form-item label="最大项目数">
|
||||||
|
<el-input-number v-model="quotaForm.maxProjects" :min="0" :step="1" placeholder="项目上限" />
|
||||||
|
</el-form-item>
|
||||||
|
</el-form>
|
||||||
|
<template #footer>
|
||||||
|
<el-button @click="showQuota = false">取消</el-button>
|
||||||
|
<el-button type="primary" @click="submitQuota">保存</el-button>
|
||||||
|
</template>
|
||||||
|
</el-dialog>
|
||||||
</div>
|
</div>
|
||||||
</template>
|
</template>
|
||||||
|
|
||||||
|
|||||||
282
平台治理.md
Normal file
282
平台治理.md
Normal file
@@ -0,0 +1,282 @@
|
|||||||
|
# 平台治理功能说明
|
||||||
|
|
||||||
|
平台治理涵盖用户管理、租户管理、项目管理、审批管理、审计日志、资源授权(ACL)、留存策略等企业管理能力。功能入口位于侧边栏「平台治理」和「系统设置」两个分组下。
|
||||||
|
|
||||||
|
本平台核心业务是**模型微调**:用户上传数据 → 数据处理 → 模型训练 → 模型评测 → 模型推理。平台治理负责管理**谁**能访问**哪个租户/项目**的**哪些资源**。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 侧边栏菜单结构
|
||||||
|
|
||||||
|
```
|
||||||
|
平台治理
|
||||||
|
├── 租户管理 /tenants
|
||||||
|
├── 项目空间 /projects
|
||||||
|
├── 审计日志 /audit-logs
|
||||||
|
├── 审批模板 /approval-templates
|
||||||
|
└── 审批中心 /approval-instances
|
||||||
|
|
||||||
|
系统设置
|
||||||
|
├── 用户设置 /user-settings
|
||||||
|
├── 平台性能 /hardware
|
||||||
|
└── 查看日志 /logs
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. 租户管理
|
||||||
|
|
||||||
|
**对应页面**:列表页 `TenantListView.vue` + 详情页 `TenantDetailView.vue`
|
||||||
|
**后端接口**:`GET/POST /tenants`、`GET/PUT/DELETE /tenants/:id`、`PUT /tenants/:id/quota`、`PUT /tenants/:id/retention-policy`
|
||||||
|
|
||||||
|
### 列表页
|
||||||
|
|
||||||
|
**展示列**:租户名称、用户ID、状态、创建时间
|
||||||
|
|
||||||
|
**行操作**:
|
||||||
|
| 按钮 | 功能 |
|
||||||
|
|------|------|
|
||||||
|
| 详情 | 跳转详情页 `/tenants/:id` |
|
||||||
|
| 删除 | 确认弹窗后调用 `DELETE /tenants/:id` |
|
||||||
|
|
||||||
|
**新建租户弹窗**:
|
||||||
|
| 字段 | 控件 | 说明 |
|
||||||
|
|------|------|------|
|
||||||
|
| 名称 | 文本输入 | 必填 |
|
||||||
|
| 用户ID | 文本输入 | 可选 |
|
||||||
|
| GPU 数量 | 数字输入 | 配额,0=不限制 |
|
||||||
|
| 存储配额(GB) | 数字输入 | 配额,0=不限制 |
|
||||||
|
| 最大项目数 | 数字输入 | 配额,0=不限制 |
|
||||||
|
|
||||||
|
### 详情页
|
||||||
|
|
||||||
|
**基本信息**:名称、用户ID、状态、创建时间、配额(友好格式显示,如 `GPU 8 | 存储 100GB | 项目 5`)
|
||||||
|
|
||||||
|
**配额编辑**:GPU 数量 / 存储配额(GB) / 最大项目数,三个数字输入 + 保存按钮
|
||||||
|
|
||||||
|
**已移除**:详情页中不再内嵌项目空间列表(项目有独立页面)
|
||||||
|
|
||||||
|
### 当前缺失
|
||||||
|
- [ ] 编辑租户名称和用户ID(`updateTenant` API 已有,UI 缺失)
|
||||||
|
- [ ] 留存策略关联设置(`setTenantRetention` API 已有,UI 缺失)
|
||||||
|
- [ ] 列表页无搜索/筛选
|
||||||
|
- [ ] 状态列无 Tag 着色
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. 项目空间
|
||||||
|
|
||||||
|
**对应页面**:列表页 `ProjectListView.vue` + 详情页 `ProjectDetailView.vue`
|
||||||
|
**后端接口**:`GET/POST /projects`、`GET/PUT/DELETE /projects/:id`、`GET/POST/DELETE /projects/:id/members`、`PUT /projects/:id/archive`
|
||||||
|
|
||||||
|
### 列表页
|
||||||
|
|
||||||
|
**展示列**:项目名、编码ID、状态、描述、创建时间
|
||||||
|
|
||||||
|
**工具栏**:
|
||||||
|
- 租户选择器(下拉过滤,按租户编码选择)
|
||||||
|
- 搜索框(按项目名/编码搜索)
|
||||||
|
- 新建项目按钮
|
||||||
|
|
||||||
|
**新建项目弹窗**:
|
||||||
|
| 字段 | 控件 | 说明 |
|
||||||
|
|------|------|------|
|
||||||
|
| 名称 | 文本输入 | 必填 |
|
||||||
|
| 编码ID | 下拉选择 | 选择已有租户的编码(如 `default`),关联到该租户 |
|
||||||
|
| 描述 | 文本域 | 可选,3行 |
|
||||||
|
|
||||||
|
**行操作**:详情 + 删除
|
||||||
|
|
||||||
|
### 详情页
|
||||||
|
|
||||||
|
**基本信息**:名称、编码、状态、租户、描述、创建时间
|
||||||
|
|
||||||
|
**项目成员管理**:
|
||||||
|
- 成员列表(用户名、角色、添加时间)
|
||||||
|
- 添加成员:选择用户 + 角色(member/admin/viewer)
|
||||||
|
- 移除成员
|
||||||
|
|
||||||
|
**资源授权(ACL)**:
|
||||||
|
- 弹窗编辑器,逐条配置授权规则
|
||||||
|
- 主体类型:用户(下拉选已有用户)或项目角色(member/admin/viewer)
|
||||||
|
- 权限:read / write / execute / download / delete / share(多选 checkbox)
|
||||||
|
- 增删行后统一保存
|
||||||
|
|
||||||
|
### 当前缺失
|
||||||
|
- [ ] 编辑项目基本信息(名称、描述)
|
||||||
|
- [ ] 项目启停/归档操作(`archiveProject` API 已有,UI 缺失)
|
||||||
|
- [ ] 项目下的资源使用统计(模型数、数据集数、任务数)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. 审批管理
|
||||||
|
|
||||||
|
**对应页面**:`ApprovalTemplateView.vue`(模板)+ `ApprovalInstanceView.vue`(实例)
|
||||||
|
**后端接口**:模板 CRUD、实例列表+决策
|
||||||
|
|
||||||
|
### 审批模板
|
||||||
|
|
||||||
|
**展示列**:模板名、步骤数、创建时间
|
||||||
|
|
||||||
|
**新建模板**:
|
||||||
|
| 字段 | 控件 | 说明 |
|
||||||
|
|------|------|------|
|
||||||
|
| 模板名称 | 文本输入 | 必填 |
|
||||||
|
| 审批步骤 | 文本域 | **需手写 JSON 字符串**,体验差 |
|
||||||
|
|
||||||
|
**当前缺失**:
|
||||||
|
- [ ] 可视化步骤编辑(拖拽添加步骤、选审批人)
|
||||||
|
- [ ] 模板编辑/删除(API 已有,UI 缺失)
|
||||||
|
- [ ] 模板详情页
|
||||||
|
|
||||||
|
### 审批中心
|
||||||
|
|
||||||
|
**展示列**:资源类型、资源ID、状态(Tag着色)、发起人、创建时间
|
||||||
|
|
||||||
|
**工具栏**:状态筛选(待审批/已通过/已拒绝)、资源类型/ID 搜索
|
||||||
|
|
||||||
|
**行操作**:通过/拒绝(弹窗填写审批意见)
|
||||||
|
|
||||||
|
**当前缺失**:
|
||||||
|
- [ ] "我发起的"审批视角
|
||||||
|
- [ ] 审批流转详情(谁审批了、什么时间)
|
||||||
|
- [ ] 撤回功能
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. 审计日志
|
||||||
|
|
||||||
|
**对应页面**:`AuditLogView.vue`
|
||||||
|
**后端接口**:`GET /system/audit-logs`(多条件筛选+分页)、`GET /system/audit-logs/export`(CSV导出)
|
||||||
|
|
||||||
|
### 列表页
|
||||||
|
|
||||||
|
**展示列**:时间、租户ID、项目ID、操作人ID、动作、目标类型、目标ID、详情、IP
|
||||||
|
|
||||||
|
**筛选条件**:租户ID、项目ID、操作人ID、动作、目标类型、开始时间、结束时间
|
||||||
|
|
||||||
|
**工具栏**:CSV 导出按钮(最多 10000 条)
|
||||||
|
|
||||||
|
### 当前缺失
|
||||||
|
- [ ] 分页控件(数据量大时需要翻页)
|
||||||
|
- [ ] 筛选字段改为下拉选择(当前全是文本输入,不知道有哪些可选值)
|
||||||
|
- [ ] 详情弹窗(点击某条记录查看完整信息)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. 用户设置
|
||||||
|
|
||||||
|
**对应页面**:`UserSettingsView.vue`(列表)+ `UserCreateView.vue`(创建)+ `UserPermissionView.vue`(权限弹窗)
|
||||||
|
**后端接口**:`GET/POST/PUT/DELETE /users`、`POST /users/:id/reset-password`
|
||||||
|
|
||||||
|
### 列表页
|
||||||
|
|
||||||
|
**展示列**:账号、显示名、角色、状态、页面权限、创建时间
|
||||||
|
|
||||||
|
**行操作**:
|
||||||
|
| 操作 | 说明 |
|
||||||
|
|------|------|
|
||||||
|
| 启停开关 | protected 用户和自己不可操作 |
|
||||||
|
| 重置密码 | 弹窗确认,默认密码 `Platform@123` |
|
||||||
|
| 页面权限 | 弹窗 checkbox 组,12 个模块可选:看板/模型训练/模型评测/模型推理/模型管理/数据集/数据处理/数据转换/算力/平台性能/查看日志/用户设置 |
|
||||||
|
| 删除 | 确认弹窗,protected 用户和自己不可操作 |
|
||||||
|
|
||||||
|
### 创建页
|
||||||
|
|
||||||
|
**表单字段**:账号、显示名、初始密码、角色(超级管理员/操作员/观察员)、状态、页面权限
|
||||||
|
|
||||||
|
### 当前缺失
|
||||||
|
- [ ] 用户编辑页面(修改显示名、角色等,`updateUser` API 已有,UI 缺失)
|
||||||
|
- [ ] 权限码显示为英文(如 `fine-tune`),无中文翻译
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. 平台性能
|
||||||
|
|
||||||
|
**对应页面**:`HardwareView.vue`
|
||||||
|
**后端接口**:`GET /system-info`、`GET /health`(顶部栏实时指标)
|
||||||
|
|
||||||
|
### 页面内容
|
||||||
|
|
||||||
|
**概览卡片**:CPU(型号/核心数/使用率)、内存(已用/总量/使用率)、磁盘(已用/总量/使用率)、网络吞吐
|
||||||
|
|
||||||
|
**趋势图**:CPU/内存/磁盘/GPU 利用率折线图(最近 60 次采样,1/3/5/10秒自动刷新)
|
||||||
|
|
||||||
|
**GPU 资源池**:每张卡展示名称、状态、利用率、显存、温度、功耗、风扇转速;点击卡片查看详情抽屉(设备属性、趋势图、进程列表)
|
||||||
|
|
||||||
|
**主机信息**:OS、运行时长、进程数、GPU 驱动版本
|
||||||
|
|
||||||
|
### 当前缺失
|
||||||
|
- [ ] 历史数据持久化(刷新后采样清空)
|
||||||
|
- [ ] 告警阈值设置
|
||||||
|
- [ ] 网络吞吐数据显示为空
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. 查看日志
|
||||||
|
|
||||||
|
**对应页面**:`LogsView.vue`(系统日志 + 训练日志双 Tab)
|
||||||
|
|
||||||
|
### 系统日志
|
||||||
|
|
||||||
|
- 按日期选择日志文件
|
||||||
|
- 关键词搜索
|
||||||
|
- 日志级别筛选(INFO/WARN/ERROR/DEBUG)
|
||||||
|
- 自动刷新(5/10/30/60 秒可调)
|
||||||
|
|
||||||
|
### 训练日志
|
||||||
|
|
||||||
|
- 按 PID 选择训练日志文件
|
||||||
|
- 同样支持搜索和级别筛选
|
||||||
|
|
||||||
|
### 当前缺失
|
||||||
|
- [ ] 日志下载/导出
|
||||||
|
- [ ] 分页或虚拟滚动(大文件加载慢)
|
||||||
|
- [ ] 行号显示
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. 留存策略 ⚠️ 前端完全缺失
|
||||||
|
|
||||||
|
**后端 API 已完整实现**(CRUD 5 个端点),**前端 `retention.ts` 模块已封装**,但:
|
||||||
|
- 侧边栏无菜单入口
|
||||||
|
- 路由未配置
|
||||||
|
- 无任何 Vue 页面
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 数据流转关系
|
||||||
|
|
||||||
|
```
|
||||||
|
用户登录 → 分配角色(admin/operator/viewer) + 页面权限
|
||||||
|
│
|
||||||
|
├─ 创建租户(配额:GPU/存储/项目数)
|
||||||
|
│ └─ 关联用户
|
||||||
|
│
|
||||||
|
├─ 创建项目(关联租户,选择编码ID)
|
||||||
|
│ ├─ 项目成员(角色:member/admin/viewer)
|
||||||
|
│ └─ 资源授权(ACL):谁对什么资源有什么权限
|
||||||
|
│
|
||||||
|
├─ 创建审批模板(定义审批流程)
|
||||||
|
│ └─ 审批实例:敏感操作需要审批(通过/拒绝)
|
||||||
|
│
|
||||||
|
├─ 审计日志:所有操作自动记录(谁在什么时间做了什么)
|
||||||
|
│
|
||||||
|
└─ 留存策略:定义数据保留周期,自动清理过期数据
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 完成度总表
|
||||||
|
|
||||||
|
| 模块 | 列表 | 新建 | 编辑 | 删除 | 搜索 | 筛选 | 导出 | 完成度 |
|
||||||
|
|------|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
|
||||||
|
| 租户管理 | ✅ | ✅ | 仅配额 | ✅ | ❌ | ❌ | ❌ | 75% |
|
||||||
|
| 项目空间 | ✅ | ✅ | ❌ | ✅ | ✅ | 按租户 | ❌ | 80% |
|
||||||
|
| 审批模板 | ✅ | ✅(JSON) | ❌ | ❌ | ❌ | ❌ | ❌ | 45% |
|
||||||
|
| 审批中心 | ✅ | N/A | N/A | ❌ | ✅ | 按状态 | ❌ | 65% |
|
||||||
|
| 审计日志 | ✅ | N/A | N/A | N/A | ❌ | 多字段 | CSV | 70% |
|
||||||
|
| 用户设置 | ✅ | ✅ | 仅权限 | ✅ | ❌ | ❌ | ❌ | 75% |
|
||||||
|
| 平台性能 | ✅ | N/A | N/A | N/A | N/A | N/A | ❌ | 80% |
|
||||||
|
| 查看日志 | ✅ | N/A | N/A | N/A | 关键词 | 按级别 | ❌ | 70% |
|
||||||
|
| **留存策略** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | **0%** |
|
||||||
Reference in New Issue
Block a user