fix(migrations): enforce schema ownership safety

This commit is contained in:
caoxiaozhu
2026-07-14 09:23:34 +08:00
parent 1347366b95
commit 11275e4ba6
18 changed files with 755 additions and 57 deletions

View File

@@ -1,6 +1,6 @@
# AI 费用闭环与价值证明 概念文档
更新时间2026-07-13
更新时间2026-07-14
文档路径document/development/2026-07-13/feature/ai-expense-closed-loop-and-value-proof/CONCEPT.md
@@ -246,6 +246,7 @@
-`ReimbursementRequest` 进入只读兼容和迁移状态,停止新增第二套业务编排。
- 现有 `risk_flags_json` 保持读取兼容,新审批、付款、归档和关系事件写入结构化表。
- API 新字段优先追加,不在同一阶段破坏现有前端契约。
- 迁移桥接期集中维护 migration-owned 表集合;所有 legacy bootstrap 只能创建集合之外的表。标准启动在 `alembic upgrade head` 前只读核对 revision 与自有表集合发现漂移时拒绝启动不自动猜测、stamp 或修复。
- 所有表结构通过 Alembic 迁移,不继续在请求路径执行 DDL。
#### 版本与审计
@@ -405,7 +406,7 @@ customer_contribution_margin
```bash
docker exec -w /app -e SERVER_VENV_DIR=/tmp/x-financial-server-venv \
local-x-financial-linux \
x-financial-local-linux \
/tmp/x-financial-server-venv/bin/pytest -q server/tests/test_expense_case_service.py
```
@@ -500,3 +501,6 @@ docker exec -w /app -e SERVER_VENV_DIR=/tmp/x-financial-server-venv \
- 2026-07-13AI 申请身份边界):申请预览快速入口不再接受请求体中的 `user_id`、管理员标记、角色、员工编号或其他身份字段作为授权事实,全部强制绑定服务端会话;伪造管理员身份编辑他人退回申请会返回 400且原申请与费用事件不发生变化。
- 2026-07-13安全与事务验证容器内受影响后端定向回归 36 项、Expense Case 前端兼容测试 9 项和 Python `ruff --select F,I` 通过;覆盖本人、当前审批人、财务、管理员、无权限、跨租户、整 Case 安全摘要、草稿新建/更新、失败回滚、HTTP 创建重放、相同快照去重、不同快照留痕、Steward 重放及伪造身份越权。未修改数据库结构,未执行持久化开发数据库迁移。
- 2026-07-13联调边界当前持久化开发数据库尚无 `auth_sessions``expense_cases``expense_case_links``business_events` 表,浏览器登录无法获得有效认证凭证,因此本轮未声称完成真实页面端到端联调,也未擅自执行数据库迁移。计划、消费/票据、入账、对账和复盘事件仍待后续补齐。
- 2026-07-14迁移所有权加固新增统一 `schema_ownership.py`,七个运行时初始化入口只创建 legacy 表;标准启动在 Alembic upgrade 前执行只读漂移预检revision 与 migration-owned 表集合不一致时 fail-fast且不会自动 stamp 或修改数据库。
- 2026-07-14真实迁移验证在主应用容器连接的一次性 tmpfs PostgreSQL 17 中完成空库升级、重复升级、关键约束/索引、真实外键级联、降级到 base、legacy 哨兵保留、无版本自有表漂移拒绝和再次升级,`test_alembic_migrations.py` 4 项通过,最终 revision 为 `20260713_0002`;临时容器已自动清理,持久化开发数据库复查仍未迁移。
- 2026-07-14剩余边界当前两条 revision 只覆盖 Expense Case、Business Event 和 Auth Session完整 legacy schema baseline 及停止其余运行时 DDL 仍未完成;本轮受影响服务回归 46 项通过,既有员工目录历史部门归一化用例仍单独失败,未混入本次迁移安全范围。

View File

@@ -1,6 +1,6 @@
# AI 费用闭环与价值证明 开发 TODO
更新时间2026-07-13
更新时间2026-07-14
文档路径document/development/2026-07-13/feature/ai-expense-closed-loop-and-value-proof/TODO.md
@@ -58,9 +58,12 @@
证据:`expense_case.py``test_expense_case_endpoints.py`;用户态响应只保留安全流程摘要,关联资源 ID 和未知嵌套 payload 被递归过滤,本人、当前审批人、财务、管理员、无权限及跨租户边界测试通过。
- [ ] [CONCEPT: 权限与安全] 为 Qdrant collection/namespace、对象存储前缀和缓存键补齐租户隔离回归测试。
- [ ] [CONCEPT: 数据与契约] 建立 Alembic baseline 和正式迁移链,停止请求路径运行 DDL。
- [x] [CONCEPT: 兼容策略] 集中 migration-owned 表所有权并在标准启动迁移前执行只读漂移预检,禁止 legacy bootstrap 越权建表。
证据:`schema_ownership.py``migration_preflight.py``server_start.sh``test_migration_preflight.py``test_schema_ownership.py`;无版本自有表、缺表、多表、未知/多 revision 均 fail-fast不自动 stamp 或修改数据库。
- [x] [CONCEPT: 费用领域与编排] 新增 `ExpenseCaseService` 和费用事件查询接口,保持编排与具体职责分离。
证据:`server/src/app/services/expense_cases.py``server/src/app/api/v1/endpoints/expense_cases.py``GET /api/v1/expense-cases/by-claim/{claim_id}`;容器 OpenAPI 校验通过。
- [ ] [CONCEPT: 数据与契约] 新增 `expense_cases``expense_case_links``business_events` 表及迁移。
- [x] [CONCEPT: 数据与契约] 新增 `expense_cases``expense_case_links``business_events` 表及迁移。
证据:`20260713_0001_expense_case_business_events.py`;一次性 PostgreSQL 17 已验证表、唯一约束、复合索引、外键级联、降级和再次升级。
- [ ] [CONCEPT: 业务事件与 AI 决策] 新增 `ai_decisions``ai_decision_feedback``workflow_outcomes` 表及迁移。
- [ ] [CONCEPT: 数据与契约] 为申请、票据、草稿、提交、退回、审批、付款和归档接入统一 correlation ID。
- [ ] [CONCEPT: 业务事件与 AI 决策] 建立事务 Outbox申请、提交、退回、审批、支付和入账状态与事件同事务提交消费端按事件 ID 幂等处理。
@@ -73,7 +76,7 @@
- [x] [CONCEPT: 权限与安全] 将 AI 申请预览快速入口的用户、租户、角色与管理员身份强制绑定到服务端会话,拒绝请求体伪造身份编辑他人申请。
证据:`reimbursements.py``test_reimbursement_endpoints.py`;对抗用例修复前返回 200修复后返回 400且目标申请和费用事件保持不变。
- [x] [CONCEPT: 兼容策略] 建立迁移桥接:服务启动先执行 Alembic旧 metadata bootstrap 排除 migration-owned 表。
证据:`server_start.sh``agent_foundation.py``20260713_0001_expense_case_business_events.py`;容器 `sh -n`、离线 upgrade/downgrade SQL 和 legacy bootstrap 排除测试通过。
证据:`server_start.sh``schema_ownership.py``migration_preflight.py``20260713_0001_expense_case_business_events.py``20260713_0002_auth_sessions.py`;容器 Shell/静态检查及一次性 PostgreSQL 完整 upgrade/downgrade/re-upgrade 通过。
- [ ] [CONCEPT: 兼容策略] 正式切换前以 shadow 事件校验现有 `ExpenseClaim` 映射;切换后禁止关键事件可丢弃写入。
- [ ] [CONCEPT: 兼容策略] 制定旧 `ReimbursementRequest` 只读兼容、迁移和停止新增编排的计划。
- [ ] [CONCEPT: 兼容策略] 把新增审批、付款、归档和关系事件移出 `risk_flags_json`,保留旧数据读取兼容。
@@ -155,6 +158,8 @@
- [x] [CONCEPT: 测试方案] 为 Expense Case GET 接口补充 owner、审批人、财务、管理员、无权限用户和整 Case 关联事件可见范围的 HTTP 权限测试。
证据:`test_expense_case_endpoints.py` 容器内 8 项通过,覆盖跨租户、无 Case、申请与报销关联摘要以及内部字段递归过滤。
- [ ] [CONCEPT: 测试方案] 为 Alembic baseline、升级、旧数据迁移和回滚边界补充 Postgres 集成测试。
- [x] [CONCEPT: 测试方案] 为当前 migration-owned schema 切片补充一次性 PostgreSQL 集成测试和危险 URL 防误连门禁。
证据:`test_alembic_migrations.py` 默认无显式 URL 时跳过,主机和库名必须带 disposable 标记tmpfs PostgreSQL 17 中 4 项通过,覆盖空库升级、重复升级、关键约束/索引、外键级联、base 降级、legacy 哨兵保留、漂移拒绝和再次升级;持久化开发库未被修改。完整 legacy baseline 仍保留在上一条未完成项中。
- [ ] [CONCEPT: 测试方案] 为连接器幂等、重试、回执、失败恢复、重复付款和对账补充测试。
- [ ] [CONCEPT: 测试方案] 跑通申请 → 票据 → 报销 → 预审 → 审批 → 付款 → 入账 → 归档端到端。
- [ ] [CONCEPT: 测试方案] 跑通 AI 建议 → 用户修改 → 工作流结果 → 记忆激活 → 下次建议变化闭环。

View File

@@ -0,0 +1,12 @@
# 迁移自有表被运行时 bootstrap 越权创建
日期2026-07-14
文档路径document/development/2026-07-14/dev-logs/bugs/migration-owned-table-bootstrap-drift.md
## 修复记录
- 09:21记录 bug 修复:迁移自有表被运行时 bootstrap 越权创建。bug-log:1347366b
- Git 提交检查fetch 成功upstream `origin/main`upstream 新提交:未发现;本地 ahead 提交1347366b (HEAD -> main) feat(expenses): secure timeline and draft events22669a90 feat(expenses): show unified expense event timelinea616b30c fix(expenses): unify AI application submission transaction653eda05 feat(auth): add opaque bearer sessions661990b2 feat(expenses): add transactional expense case events。
- 修改:新增 `schema_ownership.py` 统一四张 migration-owned 表的所有权边界,把 Agent Foundation、员工、预算、设置、系统看板、数字员工看板和演示数据初始化中的全量 `create_all` 改为只创建 legacy 表;新增 `migration_preflight.py`,在启动迁移前只读核对 Alembic revision 与自有表集合,遇到无版本表、缺表、多表、未知版本或多版本时直接拒绝,不自动 stamp 或修复。
- 操作:调整 `server_start.sh` 的启动顺序和 `alembic.ini` 的脚本绝对定位方式在无端口、无持久卷、tmpfs 数据目录的一次性 PostgreSQL 17 容器中执行空库升级、重复升级、降级到 base、漂移拦截和再次升级结束后自动删除临时容器没有执行持久化开发数据库迁移。
- 验证:迁移与预检定向测试 `19 passed, 1 skipped`;一次性 PostgreSQL 真实迁移测试 `4 passed`,最终 revision 为 `20260713_0002`关键唯一约束、复合索引和外键级联均通过legacy 哨兵跨降级/再升级保留;受影响服务回归 `46 passed, 1 failed`,唯一失败为既有员工目录历史部门归一化用例,单独运行同样失败,与本次建表 helper 无关;新增文件 Ruff 全规则、旧服务 F/I/B/UP、Shell 语法、Alembic heads 和 `git diff --check` 均通过。
- 影响:运行时 bootstrap 不再可能越权重建 Alembic 管理的表,启动会在可能破坏数据前暴露版本/表漂移;完整 legacy schema baseline 仍未建立,后续迁移旧表时必须继续小步验证。

View File

@@ -58,14 +58,33 @@ pip install -e .[dev]
copy ..\\.env.example ..\\.env
```
3. 启动服务
3. 使用标准入口启动服务
```bash
uvicorn app.main:app --reload --app-dir src
cd ..
./start.sh server
```
标准入口在启动新的 FastAPI 进程前会自动执行 `alembic upgrade head`,迁移成功后才会
启动 Uvicorn。`./start.sh all` 在需要启动后端时也会复用同一流程。不要把直接运行
`uvicorn` 当作标准启动方式;手工调试 Uvicorn 时,需要先自行完成迁移。
## 迁移
```bash
alembic upgrade head
cd server
alembic -c alembic.ini upgrade head
```
一次性 PostgreSQL 迁移测试默认跳过,只有显式提供
`MIGRATION_TEST_DATABASE_URL` 时才会执行。为避免误操作开发库,测试会同时要求主机名
和数据库名使用 `migration-probe``disposable-probe` 安全前缀,并要求数据库初始为空。
```bash
cd server
MIGRATION_TEST_DATABASE_URL='postgresql+psycopg://migration_probe:migration_probe_pw@x-financial-migration-probe-123:5432/migration_probe' \
pytest -q tests/test_alembic_migrations.py
```
该测试覆盖首次升级、重复升级、关键表/约束/索引、外键级联、降级到 `base`、无关旧表
及哨兵数据保留,以及再次升级。请只把它指向无持久卷的一次性 PostgreSQL 容器。

View File

@@ -1,6 +1,7 @@
[alembic]
script_location = alembic
script_location = %(here)s/alembic
prepend_sys_path = .
path_separator = os
sqlalchemy.url = postgresql+psycopg://postgres:postgres@127.0.0.1:5432/x_financial
[loggers]

View File

@@ -379,6 +379,9 @@ ensure_dependencies() {
}
run_database_migrations() {
info "Checking database migration state..."
PYTHONPATH="$SCRIPT_DIR/src${PYTHONPATH:+:$PYTHONPATH}" \
"$PYTHON_BIN" -m app.db.migration_preflight
info "Applying database migrations..."
"$PYTHON_BIN" -m alembic -c "$SCRIPT_DIR/alembic.ini" upgrade head
info "Database migrations are up to date."

View File

@@ -0,0 +1,129 @@
from __future__ import annotations
import sys
from dataclasses import dataclass
from sqlalchemy import create_engine, inspect, text
from sqlalchemy.engine import Connection, Engine
from sqlalchemy.exc import SQLAlchemyError
from app.core.config import get_settings
from app.db.schema_ownership import MIGRATION_OWNED_TABLES
# 迁移脚本新增或删除自有表时,必须同步更新该映射并补充对应测试。
MIGRATION_OWNED_TABLES_BY_REVISION: dict[str, frozenset[str]] = {
"20260713_0001": frozenset(
{
"expense_cases",
"expense_case_links",
"business_events",
}
),
"20260713_0002": frozenset(
{
"expense_cases",
"expense_case_links",
"business_events",
"auth_sessions",
}
),
}
if MIGRATION_OWNED_TABLES_BY_REVISION["20260713_0002"] != MIGRATION_OWNED_TABLES:
raise RuntimeError("latest Alembic revision must own the centralized migration table set")
class MigrationPreflightError(RuntimeError):
"""Raised when the database schema cannot be safely advanced by Alembic."""
@dataclass(frozen=True)
class MigrationPreflightState:
revision: str | None
owned_tables: frozenset[str]
def _format_tables(table_names: frozenset[str]) -> str:
return ", ".join(sorted(table_names)) or "none"
def _validate_connection(connection: Connection) -> MigrationPreflightState:
table_names = frozenset(inspect(connection).get_table_names())
owned_tables = table_names & MIGRATION_OWNED_TABLES
if "alembic_version" not in table_names:
if owned_tables:
raise MigrationPreflightError(
"unversioned database contains migration-owned tables "
f"({_format_tables(owned_tables)}); refusing to guess, stamp, or repair"
)
return MigrationPreflightState(revision=None, owned_tables=owned_tables)
revisions = tuple(
str(revision)
for revision in connection.execute(
text("SELECT version_num FROM alembic_version")
).scalars()
)
if not revisions:
if owned_tables:
raise MigrationPreflightError(
"alembic_version has no recorded revision but migration-owned tables exist "
f"({_format_tables(owned_tables)}); refusing to guess, stamp, or repair"
)
return MigrationPreflightState(revision=None, owned_tables=owned_tables)
if len(revisions) > 1:
raise MigrationPreflightError(
"multiple Alembic revisions are recorded "
f"({', '.join(sorted(revisions))}); branch state is unsupported"
)
revision = revisions[0]
expected_tables = MIGRATION_OWNED_TABLES_BY_REVISION.get(revision)
if expected_tables is None:
raise MigrationPreflightError(
f"unknown Alembic revision {revision!r}; refusing to run migrations"
)
if owned_tables != expected_tables:
missing_tables = expected_tables - owned_tables
unexpected_tables = owned_tables - expected_tables
raise MigrationPreflightError(
f"migration-owned table set does not match revision {revision}: "
f"missing={_format_tables(missing_tables)}; "
f"unexpected={_format_tables(unexpected_tables)}"
)
return MigrationPreflightState(revision=revision, owned_tables=owned_tables)
def validate_migration_state(bind: Engine | Connection) -> MigrationPreflightState:
"""Inspect the database without changing its schema or Alembic revision state."""
if isinstance(bind, Engine):
with bind.connect() as connection:
return _validate_connection(connection)
return _validate_connection(bind)
def main() -> int:
settings = get_settings()
engine = create_engine(settings.resolved_database_url, pool_pre_ping=True)
try:
state = validate_migration_state(engine)
except (MigrationPreflightError, SQLAlchemyError) as exc:
print(f"Database migration preflight failed: {exc}", file=sys.stderr)
return 1
finally:
engine.dispose()
revision = state.revision or "unversioned/base"
print(
"Database migration preflight passed: "
f"revision={revision}; owned_tables={_format_tables(state.owned_tables)}"
)
return 0
if __name__ == "__main__":
raise SystemExit(main())

View File

@@ -0,0 +1,22 @@
from __future__ import annotations
from sqlalchemy.engine import Connection, Engine
from app.db.base import Base
MIGRATION_OWNED_TABLES: frozenset[str] = frozenset(
{
"auth_sessions",
"expense_cases",
"expense_case_links",
"business_events",
}
)
def create_legacy_schema(bind: Engine | Connection) -> None:
"""创建仍由旧 bootstrap 管理的表,不越过 Alembic 的表所有权边界。"""
legacy_tables = [
table for table in Base.metadata.sorted_tables if table.name not in MIGRATION_OWNED_TABLES
]
Base.metadata.create_all(bind=bind, tables=legacy_tables)

View File

@@ -2,31 +2,16 @@ from __future__ import annotations
import threading
from sqlalchemy import inspect, select, text
from sqlalchemy import inspect, text
from sqlalchemy.orm import Session
from app.core.config import get_settings
from app.core.logging import get_logger
from app.db.base import Base
from app.db.schema_ownership import create_legacy_schema
from app.db.session import get_session_factory
from app.models.agent_asset import AgentAsset
from app.services.agent_foundation_asset_helpers import AgentFoundationAssetHelperMixin
from app.services.agent_foundation_asset_seed import AgentFoundationAssetSeedMixin
from app.services.agent_foundation_asset_topup import AgentFoundationAssetTopUpMixin
from app.services.agent_foundation_constants import (
ATTACHMENT_RULE_ASSET_CODE,
ATTACHMENT_RULE_RUNTIME_CONFIG,
COMPANY_COMMUNICATION_RULE_SCENARIO_JSON,
COMPANY_COMMUNICATION_RULE_VERSION,
COMPANY_TRAVEL_RULE_SCENARIO_JSON,
COMPANY_TRAVEL_RULE_VERSION,
DEMO_EXPENSE_CLAIM_SIGNATURES,
DEMO_PAYABLE_SIGNATURES,
DEMO_RECEIVABLE_SIGNATURES,
LEGACY_RULE_CODES,
PLATFORM_DESTINATION_LOCATION_RULE_CODE,
PLATFORM_DESTINATION_LOCATION_RULE_FILENAME,
)
from app.services.agent_foundation_digital_employee_tasks import (
AgentFoundationDigitalEmployeeTaskMixin,
)
@@ -38,12 +23,6 @@ from app.services.agent_foundation_spreadsheets import AgentFoundationSpreadshee
logger = get_logger("app.services.agent_foundation")
_foundation_ready_lock = threading.RLock()
_foundation_ready_keys: set[str] = set()
MIGRATION_OWNED_TABLES = {
"auth_sessions",
"expense_cases",
"expense_case_links",
"business_events",
}
def prepare_agent_foundation() -> None:
@@ -83,12 +62,7 @@ class AgentFoundationService(
def _prepare_foundation(self) -> None:
try:
legacy_bootstrap_tables = [
table
for table in Base.metadata.sorted_tables
if table.name not in MIGRATION_OWNED_TABLES
]
Base.metadata.create_all(bind=self.db.get_bind(), tables=legacy_bootstrap_tables)
create_legacy_schema(self.db.get_bind())
self._ensure_agent_asset_schema()
self._ensure_financial_record_schema()
self._seed_agent_assets()

View File

@@ -7,7 +7,7 @@ from typing import Any
from sqlalchemy import select
from sqlalchemy.orm import Session
from app.db.base import Base
from app.db.schema_ownership import create_legacy_schema
from app.models.budget import BudgetAllocation, BudgetReservation
from app.models.financial_record import ExpenseClaim
from app.schemas.budget import (
@@ -32,7 +32,7 @@ class BudgetService(BudgetPaginationMixin, BudgetSupportMixin):
def ensure_budget_ready(self) -> None:
# 复用当前 Session 连接,避免在业务事务中通过 Engine 隐式提交已 flush 的单据。
Base.metadata.create_all(bind=self.db.connection())
create_legacy_schema(self.db.connection())
exists = self.db.scalar(select(BudgetAllocation.id).limit(1))
if exists:
return

View File

@@ -10,7 +10,7 @@ from sqlalchemy import or_, select
from sqlalchemy.orm import Session, selectinload
from app.core.security import hash_password
from app.db.base import Base
from app.db.schema_ownership import create_legacy_schema
from app.models.budget import BudgetAllocation, BudgetReservation, BudgetTransaction
from app.models.employee import Employee
from app.models.financial_record import ExpenseClaim, ExpenseClaimItem
@@ -78,7 +78,7 @@ class HalfYearExpenseSimulationSeeder:
return self._run(apply=True)
def _run(self, *, apply: bool) -> SimulationSummary:
Base.metadata.create_all(bind=self.db.get_bind())
create_legacy_schema(self.db.get_bind())
departments = self._department_refs(apply=apply)
current_employee_count = self._employee_count()
planned_employees = self._build_new_employee_refs(departments, current_employee_count)

View File

@@ -7,7 +7,7 @@ from sqlalchemy import or_, select
from sqlalchemy.orm import Session, selectinload
from app.core.agent_enums import AgentName, AgentRunSource
from app.db.base import Base
from app.db.schema_ownership import create_legacy_schema
from app.models.agent_run import AgentRun, AgentToolCall
from app.schemas.digital_employee_dashboard import DigitalEmployeeDashboardRead
@@ -186,7 +186,7 @@ class DigitalEmployeeDashboardService:
)
def _ensure_storage_ready(self) -> None:
Base.metadata.create_all(bind=self.db.get_bind())
create_legacy_schema(self.db.get_bind())
def _fetch_runs(self, *, start: datetime, limit: int) -> list[AgentRun]:
stmt = (

View File

@@ -1,8 +1,8 @@
from __future__ import annotations
from collections import Counter
from datetime import UTC, date, datetime
import threading
from collections import Counter
from datetime import UTC, datetime
from typing import Any
from sqlalchemy import select
@@ -11,7 +11,7 @@ from sqlalchemy.orm import Session
from app.core.config import get_settings
from app.core.logging import get_logger
from app.core.security import hash_password
from app.db.base import Base
from app.db.schema_ownership import create_legacy_schema
from app.db.session import get_session_factory
from app.models.employee import Employee
from app.models.employee_change_log import EmployeeChangeLog
@@ -28,11 +28,9 @@ from app.schemas.employee import (
EmployeeStatusSummaryRead,
EmployeeUpdate,
)
from app.services.employee_import import EmployeeImportCoordinator
from app.services.employee_bank_info import apply_default_bank_info
from app.services.employee_import import EmployeeImportCoordinator
from app.services.employee_schema import ensure_employee_schema
from app.services.employee_serialization import serialize_employee
from app.services.employee_spreadsheet import build_import_template_bytes
from app.services.employee_seed import (
CANONICAL_DEPARTMENT_CODES,
EMPLOYEE_DEFINITIONS,
@@ -44,6 +42,8 @@ from app.services.employee_seed import (
ROLE_PERMISSION_MAP,
normalize_organization_unit_code,
)
from app.services.employee_serialization import serialize_employee
from app.services.employee_spreadsheet import build_import_template_bytes
from app.services.employee_time import (
format_date,
format_datetime,
@@ -108,7 +108,7 @@ class EmployeeService:
def _ensure_directory_ready_uncached(self) -> None:
try:
Base.metadata.create_all(bind=self.db.get_bind())
create_legacy_schema(self.db.get_bind())
ensure_employee_schema(self.db)
self._prune_extra_seed_employees()
self._seed_roles()

View File

@@ -8,16 +8,20 @@ from datetime import datetime
from sqlalchemy import inspect, text
from sqlalchemy.orm import Session
from app.core.admin_secret import legacy_admin_secret_to_password_hash, read_admin_secret, verify_admin_secret
from app.core.admin_secret import (
legacy_admin_secret_to_password_hash,
read_admin_secret,
verify_admin_secret,
)
from app.core.config import get_settings
from app.core.secret_box import decrypt_secret, encrypt_secret
from app.core.security import hash_password, verify_password
from app.db.base import Base
from app.db.schema_ownership import create_legacy_schema
from app.db.session import get_session_factory
from app.models.hermes_config import HermesTaskConfig
from app.models.system_model_setting import SystemModelSetting
from app.models.system_setting import SystemSetting
from app.models.system_setting_secret import SystemSettingSecret
from app.models.hermes_config import HermesTaskConfig
from app.repositories.settings import SETTINGS_ROW_ID, SettingsRepository
from app.schemas.settings import SettingsRead, SettingsWrite
from app.services.hermes_sync import (
@@ -161,7 +165,7 @@ class SettingsService:
if cache_key not in self._schema_ready_keys:
with self._schema_ready_lock:
if cache_key not in self._schema_ready_keys:
Base.metadata.create_all(bind=self.db.get_bind())
create_legacy_schema(self.db.get_bind())
self._ensure_settings_schema()
self._schema_ready_keys.add(cache_key)

View File

@@ -8,7 +8,7 @@ from typing import Any
from sqlalchemy import or_, select
from sqlalchemy.orm import Session
from app.db.base import Base
from app.db.schema_ownership import create_legacy_schema
from app.models.agent_feedback import AgentOperationFeedback
from app.models.agent_run import AgentRun, AgentToolCall
from app.models.user_session_metric import UserSessionMetric
@@ -143,7 +143,7 @@ class SystemDashboardService:
)
def _ensure_storage_ready(self) -> None:
Base.metadata.create_all(bind=self.db.get_bind())
create_legacy_schema(self.db.get_bind())
def _fetch_runs(self, start: datetime, *, before: datetime | None = None) -> list[_DashboardRun]:
stmt = (

View File

@@ -0,0 +1,364 @@
from __future__ import annotations
import os
import re
from collections.abc import Iterator
from pathlib import Path
import pytest
from alembic.config import Config
from sqlalchemy import create_engine, inspect, text
from sqlalchemy.engine import Engine, make_url
from sqlalchemy.pool import NullPool
from alembic import command
from app.core.config import get_settings
from app.db.migration_preflight import MigrationPreflightError, validate_migration_state
from app.db.schema_ownership import MIGRATION_OWNED_TABLES
MIGRATION_TEST_DATABASE_URL = os.getenv("MIGRATION_TEST_DATABASE_URL", "").strip()
LEGACY_PROBE_TABLE = "legacy_migration_probe_records"
HEAD_REVISION = "20260713_0002"
SERVER_DIR = Path(__file__).resolve().parents[1]
ALEMBIC_INI_PATH = SERVER_DIR / "alembic.ini"
def _normalize_probe_component(value: str) -> str:
return re.sub(r"[^a-z0-9]+", "-", value.lower()).strip("-")
def _is_disposable_probe_host(value: str) -> bool:
markers = ("migration-probe", "disposable-probe")
return value in markers or any(
value.startswith(f"{marker}-") or value.startswith(f"x-financial-{marker}-")
for marker in markers
)
def _is_disposable_probe_database(value: str) -> bool:
markers = ("migration-probe", "disposable-probe")
return value in markers or any(value.startswith(f"{marker}-") for marker in markers)
def _require_disposable_probe_url(raw_url: str) -> str:
try:
parsed = make_url(raw_url)
except Exception as exc: # pragma: no cover - SQLAlchemy 提供具体解析异常
raise RuntimeError("MIGRATION_TEST_DATABASE_URL 不是有效的数据库 URL") from exc
if parsed.get_backend_name() != "postgresql":
raise RuntimeError("迁移测试只允许连接 PostgreSQL 一次性数据库")
host = _normalize_probe_component(parsed.host or "")
database = _normalize_probe_component(parsed.database or "")
if not _is_disposable_probe_host(host):
raise RuntimeError(
"迁移测试数据库主机名必须使用 migration-probe 或 disposable-probe 前缀"
)
if not _is_disposable_probe_database(database):
raise RuntimeError(
"迁移测试数据库名必须使用 migration-probe 或 disposable-probe 前缀"
)
return raw_url
@pytest.fixture(scope="module")
def migration_database_url() -> Iterator[str]:
if not MIGRATION_TEST_DATABASE_URL:
pytest.skip("仅在显式配置 MIGRATION_TEST_DATABASE_URL 时运行一次性 PostgreSQL 迁移测试")
database_url = _require_disposable_probe_url(MIGRATION_TEST_DATABASE_URL)
previous_database_url = os.environ.get("DATABASE_URL")
os.environ["DATABASE_URL"] = database_url
get_settings.cache_clear()
try:
resolved_url = get_settings().resolved_database_url
if make_url(resolved_url) != make_url(database_url):
raise RuntimeError("运行时数据库 URL 未解析到 MIGRATION_TEST_DATABASE_URL")
yield database_url
finally:
if previous_database_url is None:
os.environ.pop("DATABASE_URL", None)
else:
os.environ["DATABASE_URL"] = previous_database_url
get_settings.cache_clear()
def _alembic_config(database_url: str) -> Config:
config = Config(str(ALEMBIC_INI_PATH))
config.set_main_option("sqlalchemy.url", database_url.replace("%", "%%"))
return config
def _upgrade_head(database_url: str) -> None:
get_settings.cache_clear()
command.upgrade(_alembic_config(database_url), "head")
def _downgrade_base(database_url: str) -> None:
get_settings.cache_clear()
command.downgrade(_alembic_config(database_url), "base")
def _table_names(engine: Engine) -> set[str]:
return set(inspect(engine).get_table_names(schema="public"))
def _assert_unique_constraint(
engine: Engine,
table_name: str,
constraint_name: str,
expected_columns: tuple[str, ...],
) -> None:
constraints = {
str(item["name"]): tuple(item["column_names"])
for item in inspect(engine).get_unique_constraints(table_name, schema="public")
}
assert constraints.get(constraint_name) == expected_columns
def _assert_indexes(
engine: Engine,
table_name: str,
expected_indexes: dict[str, tuple[str, ...]],
) -> None:
indexes = {
str(item["name"]): tuple(item["column_names"])
for item in inspect(engine).get_indexes(table_name, schema="public")
}
for index_name, expected_columns in expected_indexes.items():
assert indexes.get(index_name) == expected_columns
def _assert_cascade_foreign_key(engine: Engine, table_name: str) -> None:
foreign_keys = inspect(engine).get_foreign_keys(table_name, schema="public")
matching = [
item
for item in foreign_keys
if item["constrained_columns"] == ["expense_case_id"]
and item["referred_table"] == "expense_cases"
and item["referred_columns"] == ["id"]
]
assert len(matching) == 1
assert str(matching[0].get("options", {}).get("ondelete", "")).upper() == "CASCADE"
def _assert_head_schema(engine: Engine) -> None:
names = _table_names(engine)
assert MIGRATION_OWNED_TABLES.issubset(names)
assert "alembic_version" in names
with engine.connect() as connection:
assert connection.scalar(text("SELECT version_num FROM alembic_version")) == HEAD_REVISION
_assert_unique_constraint(
engine,
"expense_cases",
"uq_expense_cases_tenant_case_no",
("tenant_id", "case_no"),
)
_assert_unique_constraint(
engine,
"expense_case_links",
"uq_expense_case_links_resource",
("resource_type", "resource_id"),
)
_assert_unique_constraint(
engine,
"business_events",
"uq_business_event_idempotency",
("tenant_id", "aggregate_type", "aggregate_id", "event_type", "idempotency_key"),
)
_assert_unique_constraint(
engine,
"auth_sessions",
"uq_auth_sessions_token_hash",
("token_hash",),
)
_assert_indexes(
engine,
"expense_cases",
{
"ix_expense_cases_tenant_stage": ("tenant_id", "current_stage"),
"ix_expense_cases_tenant_status": ("tenant_id", "status"),
},
)
_assert_indexes(
engine,
"expense_case_links",
{"ix_expense_case_links_tenant_case": ("tenant_id", "expense_case_id")},
)
_assert_indexes(
engine,
"business_events",
{
"ix_business_events_aggregate": ("aggregate_type", "aggregate_id"),
"ix_business_events_outbox": ("delivery_status", "occurred_at"),
"ix_business_events_tenant_case_time": (
"tenant_id",
"expense_case_id",
"occurred_at",
),
},
)
_assert_indexes(
engine,
"auth_sessions",
{
"ix_auth_sessions_principal_active": (
"principal_type",
"revoked_at",
"expires_at",
),
"ix_auth_sessions_tenant_username": ("tenant_id", "username"),
},
)
_assert_cascade_foreign_key(engine, "expense_case_links")
_assert_cascade_foreign_key(engine, "business_events")
def _assert_runtime_cascade(engine: Engine) -> None:
with engine.begin() as connection:
connection.execute(
text(
"""
INSERT INTO expense_cases (
id, tenant_id, case_no, scene_code, title, current_stage, status
) VALUES (
'migration-probe-case', 'migration-probe', 'CASE-MIGRATION-PROBE',
'reimbursement', '迁移级联验证', 'claiming', 'active'
)
"""
)
)
connection.execute(
text(
"""
INSERT INTO expense_case_links (
id, tenant_id, expense_case_id, resource_type, resource_id, relation_type
) VALUES (
'migration-probe-link', 'migration-probe', 'migration-probe-case',
'expense_claim', 'migration-probe-claim', 'claim'
)
"""
)
)
connection.execute(
text(
"""
INSERT INTO business_events (
id, tenant_id, expense_case_id, aggregate_type, aggregate_id,
event_type, event_version, idempotency_key, correlation_id,
actor_id, actor_type, payload_json, delivery_status, delivery_attempts
) VALUES (
'migration-probe-event', 'migration-probe', 'migration-probe-case',
'expense_claim', 'migration-probe-claim', 'claim_draft_created', 1,
'migration-probe-idempotency', 'migration-probe-correlation',
'migration-probe-user', 'user', '{}', 'pending', 0
)
"""
)
)
connection.execute(
text("DELETE FROM expense_cases WHERE id = 'migration-probe-case'")
)
assert connection.scalar(
text("SELECT COUNT(*) FROM expense_case_links WHERE id = 'migration-probe-link'")
) == 0
assert connection.scalar(
text("SELECT COUNT(*) FROM business_events WHERE id = 'migration-probe-event'")
) == 0
def _create_legacy_sentinel(engine: Engine) -> None:
with engine.begin() as connection:
connection.execute(
text(
f"""
CREATE TABLE {LEGACY_PROBE_TABLE} (
id VARCHAR(64) PRIMARY KEY,
payload VARCHAR(255) NOT NULL
)
"""
)
)
connection.execute(
text(
f"""
INSERT INTO {LEGACY_PROBE_TABLE} (id, payload)
VALUES ('legacy-sentinel', 'must-survive-migration-cycle')
"""
)
)
def _assert_legacy_sentinel(engine: Engine) -> None:
assert LEGACY_PROBE_TABLE in _table_names(engine)
with engine.connect() as connection:
payload = connection.scalar(
text(
f"SELECT payload FROM {LEGACY_PROBE_TABLE} "
"WHERE id = 'legacy-sentinel'"
)
)
assert payload == "must-survive-migration-cycle"
def _assert_base_schema(engine: Engine) -> None:
names = _table_names(engine)
assert not MIGRATION_OWNED_TABLES.intersection(names)
assert "alembic_version" in names
with engine.connect() as connection:
assert connection.scalar(text("SELECT COUNT(*) FROM alembic_version")) == 0
@pytest.mark.parametrize(
"database_url",
[
"sqlite+pysqlite:///:memory:",
"postgresql+psycopg://postgres:postgres@x-financial-local-postgres:5432/x_financial",
"postgresql+psycopg://probe:probe@migration-probe-123:5432/x_financial",
],
)
def test_disposable_database_guard_rejects_unsafe_urls(database_url: str) -> None:
with pytest.raises(RuntimeError):
_require_disposable_probe_url(database_url)
def test_alembic_migration_cycle_on_disposable_postgres(
migration_database_url: str,
) -> None:
engine = create_engine(migration_database_url, poolclass=NullPool)
try:
assert _table_names(engine) == set(), "迁移测试必须从全新空库开始"
assert validate_migration_state(engine).revision is None
_upgrade_head(migration_database_url)
_assert_head_schema(engine)
assert validate_migration_state(engine).revision == HEAD_REVISION
_upgrade_head(migration_database_url)
_assert_head_schema(engine)
_assert_runtime_cascade(engine)
_create_legacy_sentinel(engine)
_downgrade_base(migration_database_url)
_assert_base_schema(engine)
_assert_legacy_sentinel(engine)
assert validate_migration_state(engine).revision is None
with engine.begin() as connection:
connection.execute(text("CREATE TABLE expense_cases (id VARCHAR(36) PRIMARY KEY)"))
with pytest.raises(MigrationPreflightError, match="migration-owned tables exist"):
validate_migration_state(engine)
with engine.begin() as connection:
connection.execute(text("DROP TABLE expense_cases"))
assert validate_migration_state(engine).revision is None
_upgrade_head(migration_database_url)
_assert_head_schema(engine)
_assert_legacy_sentinel(engine)
finally:
engine.dispose()

View File

@@ -0,0 +1,135 @@
from __future__ import annotations
from pathlib import Path
import pytest
from sqlalchemy import Column, Integer, MetaData, Table, create_engine, text
from sqlalchemy.engine import Engine
from app.db.migration_preflight import (
MIGRATION_OWNED_TABLES_BY_REVISION,
MigrationPreflightError,
validate_migration_state,
)
@pytest.fixture
def engine() -> Engine:
database = create_engine("sqlite+pysqlite:///:memory:")
try:
yield database
finally:
database.dispose()
def _create_tables(engine: Engine, table_names: set[str] | frozenset[str]) -> None:
metadata = MetaData()
for table_name in table_names:
Table(table_name, metadata, Column("id", Integer, primary_key=True))
metadata.create_all(engine)
def _create_version_table(engine: Engine, *revisions: str) -> None:
with engine.begin() as connection:
connection.execute(text("CREATE TABLE alembic_version (version_num VARCHAR(32) NOT NULL)"))
for revision in revisions:
connection.execute(
text("INSERT INTO alembic_version (version_num) VALUES (:revision)"),
{"revision": revision},
)
def test_unversioned_database_without_migration_owned_tables_is_safe(engine: Engine) -> None:
state = validate_migration_state(engine)
assert state.revision is None
assert state.owned_tables == frozenset()
@pytest.mark.parametrize(
"owned_table",
sorted(MIGRATION_OWNED_TABLES_BY_REVISION["20260713_0002"]),
)
def test_unversioned_database_with_any_migration_owned_table_is_rejected(
engine: Engine,
owned_table: str,
) -> None:
_create_tables(engine, {owned_table})
with pytest.raises(MigrationPreflightError, match="unversioned database contains"):
validate_migration_state(engine)
@pytest.mark.parametrize(
("revision", "expected_tables"),
list(MIGRATION_OWNED_TABLES_BY_REVISION.items()),
)
def test_known_revision_requires_and_accepts_its_exact_owned_table_set(
engine: Engine,
revision: str,
expected_tables: frozenset[str],
) -> None:
_create_tables(engine, expected_tables)
_create_version_table(engine, revision)
state = validate_migration_state(engine)
assert state.revision == revision
assert state.owned_tables == expected_tables
@pytest.mark.parametrize(
("revision", "actual_tables"),
[
("20260713_0001", frozenset({"expense_cases", "expense_case_links"})),
("20260713_0001", MIGRATION_OWNED_TABLES_BY_REVISION["20260713_0002"]),
(
"20260713_0002",
MIGRATION_OWNED_TABLES_BY_REVISION["20260713_0002"] - {"auth_sessions"},
),
],
)
def test_known_revision_with_missing_or_unexpected_owned_tables_is_rejected(
engine: Engine,
revision: str,
actual_tables: frozenset[str],
) -> None:
_create_tables(engine, actual_tables)
_create_version_table(engine, revision)
with pytest.raises(MigrationPreflightError, match="does not match revision"):
validate_migration_state(engine)
def test_unknown_revision_is_rejected(engine: Engine) -> None:
_create_version_table(engine, "20990101_unknown")
with pytest.raises(MigrationPreflightError, match="unknown Alembic revision"):
validate_migration_state(engine)
def test_multiple_revisions_are_rejected(engine: Engine) -> None:
_create_version_table(engine, "20260713_0001", "20260713_0002")
with pytest.raises(MigrationPreflightError, match="multiple Alembic revisions"):
validate_migration_state(engine)
def test_empty_version_table_is_safe_only_when_owned_tables_are_absent(engine: Engine) -> None:
_create_version_table(engine)
assert validate_migration_state(engine).revision is None
_create_tables(engine, {"expense_cases"})
with pytest.raises(MigrationPreflightError, match="no recorded revision"):
validate_migration_state(engine)
def test_server_start_runs_preflight_before_alembic_upgrade() -> None:
script_path = Path(__file__).resolve().parents[1] / "server_start.sh"
script = script_path.read_text(encoding="utf-8")
preflight = '"$PYTHON_BIN" -m app.db.migration_preflight'
upgrade = '"$PYTHON_BIN" -m alembic -c "$SCRIPT_DIR/alembic.ini" upgrade head'
assert 'PYTHONPATH="$SCRIPT_DIR/src${PYTHONPATH:+:$PYTHONPATH}"' in script
assert script.index(preflight) < script.index(upgrade)

View File

@@ -0,0 +1,26 @@
from __future__ import annotations
from sqlalchemy import create_engine, inspect
from app.db.schema_ownership import MIGRATION_OWNED_TABLES, create_legacy_schema
def test_create_legacy_schema_never_creates_migration_owned_tables() -> None:
engine = create_engine("sqlite+pysqlite:///:memory:")
try:
create_legacy_schema(engine)
table_names = set(inspect(engine).get_table_names())
finally:
engine.dispose()
assert MIGRATION_OWNED_TABLES == frozenset(
{
"auth_sessions",
"business_events",
"expense_case_links",
"expense_cases",
}
)
assert table_names
assert table_names.isdisjoint(MIGRATION_OWNED_TABLES)