diff --git a/.gitignore b/.gitignore index afaf213..0bb535b 100644 --- a/.gitignore +++ b/.gitignore @@ -187,3 +187,5 @@ cython_debug/ # PyPI configuration file .pypirc +# Local backend configuration may contain database credentials. +backend/config.yaml diff --git a/README.md b/README.md index df42faa..d321f9d 100644 --- a/README.md +++ b/README.md @@ -48,12 +48,36 @@ YG_FT/ - 前端新增 `/compute` 算力节点页面,展示节点地址、权重、标签、启用状态、GPU、队列和资源副本。 - `compute/engines/llama_factory/adapter.py` 提供 LLaMA-Factory 参数校验、命令生成和日志解析基础能力。 +## 一键启动前端和后端 + +```bash +./scripts/start-dev.sh --setup # 首次运行,初始化本地依赖 +./scripts/start-dev.sh # 后续直接启动 +./scripts/start-dev.sh --check # 仅检查依赖和 PostgreSQL 连接 +``` + +首次运行前可将 `backend/config.example.yaml` 复制为 `backend/config.yaml` 并填写 +本地数据库凭据;`config.yaml` 已加入 Git 忽略。脚本会检查 PostgreSQL 和两个服务 +的健康状态,并在退出时同时停止前端和后端。也可以按下面步骤分别启动服务。 + +一键启动使用的前后端端口也统一配置在同一个 YAML 中: + +```yaml +server: + frontend_port: 16801 + backend_port: 17861 +``` + +端口被占用时修改这两项后重新启动即可。未显式配置 `cors_allow_origins` 时,后端会 +根据 `frontend_port` 自动允许本机前端来源。环境变量 `FRONTEND_PORT` 和 +`BACKEND_PORT` 可以临时覆盖 YAML。 + ## 后端启动 ```bash cd backend -python -m venv .venv -.venv\Scripts\activate +/opt/miniconda3/bin/python3 -m venv .venv +source .venv/bin/activate pip install -r requirements.txt uvicorn app.main:app --reload --port 17861 ``` @@ -69,13 +93,22 @@ GET /modelTF/fine-tune GET /modelTF/compute/nodes ``` -本地运行时默认 PostgreSQL 连接: +本地运行时默认通过 `backend/config.yaml` 连接 PostgreSQL: -```text -DATABASE_URL=postgresql+psycopg://yg_ft:change_me@localhost:15432/yg_ft +```yaml +server: + frontend_port: 16801 + backend_port: 17861 + +database: + url: postgresql+psycopg://localhost:5432/yg_ft + username: yg_ft + password: "change_me" ``` -本地启动前需要确保 PostgreSQL 已监听 `localhost:15432`,并已创建 `yg_ft` 数据库和 `yg_ft` 用户。后端启动后会自动创建当前运行表并写入内置管理员账号,运行数据统一写入 PostgreSQL。 +本地启动前需要确保 PostgreSQL 已监听 YAML 配置的地址(默认 `localhost:5432`), +并已创建 `yg_ft` 数据库和 `yg_ft` 用户。后端启动后会自动创建当前运行表并写入 +内置管理员账号,运行数据统一写入 PostgreSQL。 开发阶段内置登录账号: @@ -94,7 +127,9 @@ npm install npm run dev ``` -前端开发服务默认运行在 `http://localhost:16801`,并通过 Vite proxy 将 `/modelTF` 转发到 `http://localhost:17861`。 +通过一键脚本启动时,前端端口和 Vite proxy 的后端端口均来自 +`backend/config.yaml`。单独运行 `npm run dev` 时,默认使用前端 `16801`、后端 +`17861`,也可以通过 `FRONTEND_PORT` 和 `BACKEND_PORT` 覆盖。 ## 算力服务启动 diff --git a/backend/.env.example b/backend/.env.example index 436987d..422ca04 100644 --- a/backend/.env.example +++ b/backend/.env.example @@ -1,9 +1,18 @@ -APP_NAME=YG Fine-Tune Platform API -APP_ENV=local -API_PREFIX=/api -LOG_LEVEL=INFO -LOG_DIR=./logs -LOG_FILE_PREFIX=backend -LOG_ERROR_FILE_PREFIX=error -LOG_MAX_BYTES=20971520 -LOG_RETENTION_DAYS=10 +# 默认自动读取 backend/config.yaml。设置此变量可切换到其他 YAML 文件。 +BACKEND_CONFIG_FILE=./config.yaml + +# 下列环境变量按需启用,并优先于 YAML 中的配置。 +# DATABASE_BASE_URL=postgresql+psycopg://localhost:5432/yg_ft +# DATABASE_USERNAME=yg_ft +# DATABASE_PASSWORD=change_me +# DATABASE_URL=postgresql+psycopg://yg_ft:change_me@localhost:5432/yg_ft +# DATABASE_URL 是完整连接串覆盖项,优先级高于上面三个分离项。 +# MODELTF_ROUTE_PREFIX=/modelTF +# FRONTEND_PORT=16801 +# BACKEND_PORT=17861 +# CORS_ALLOW_ORIGINS=http://localhost:16801,http://127.0.0.1:16801 +# LOG_LEVEL=INFO +# LOG_DIR=./logs +# COMPUTE_MODE=real +# COMPUTE_STATUS_SYNC_MODE=polling +# COMPUTE_POLL_INTERVAL_SECONDS=3 diff --git a/backend/README.md b/backend/README.md index a328320..2a0d936 100644 --- a/backend/README.md +++ b/backend/README.md @@ -32,17 +32,65 @@ backend/ services/ # 跨模块应用服务 workers/ # 后台任务入口 requirements.txt # 后端第三方依赖 + config.example.yaml # 可提交的脱敏配置模板 + config.yaml # 本地配置,已忽略,环境变量可覆盖 logs/ # 本地开发日志目录,生产环境建议挂载到独立日志盘 ``` +## 配置 + +先复制本地配置并填写数据库凭据: + +```bash +cp config.example.yaml config.yaml +``` + +后端默认读取 `config.yaml`,配置优先级为: + +```text +环境变量 > BACKEND_CONFIG_FILE 指定的 YAML > backend/config.yaml > 代码默认值 +``` + +前后端端口、数据库地址、跨域来源、日志路径和计算状态轮询参数均可在 YAML 中 +配置。敏感信息或不同环境的差异建议通过环境变量覆盖,不要写入仓库。YAML 中的 +相对日志路径以该 YAML 文件所在目录为基准。使用其他配置文件时: + +```bash +BACKEND_CONFIG_FILE=./config.prod.yaml uvicorn app.main:app --reload --port 17861 +``` + +本机直连 PostgreSQL 时,数据库配置拆分为地址、用户名和密码: + +```yaml +server: + frontend_port: 16801 + backend_port: 17861 + +database: + url: postgresql+psycopg://localhost:5432/yg_ft + username: yg_ft + password: "change_me" +``` + +服务启动时会安全拼装完整连接串;用户名或密码中的特殊字符会自动编码。Docker +和生产环境仍可用完整的 `DATABASE_URL` 环境变量覆盖以上三项。 + +这里的 `5432` 是宿主机 PostgreSQL 默认端口;如果数据库容器映射到 `15432`, +将 YAML 中的端口改为 `15432` 即可。 + +`server.frontend_port` 和 `server.backend_port` 由根目录的 +`scripts/start-dev.sh` 读取,并分别传给 Vite 和 Uvicorn。未配置 +`app.cors_allow_origins` 时,本机 CORS 来源会跟随前端端口。直接手工执行 +`uvicorn` 时仍需通过 `--port` 指定监听端口。 + ## 本地启动 ```bash cd backend -python -m venv .venv -.venv\Scripts\activate +/opt/miniconda3/bin/python3 -m venv .venv +source .venv/bin/activate pip install -r requirements.txt -uvicorn app.main:app --reload +uvicorn app.main:app --reload --port 17861 ``` 健康检查: diff --git a/backend/app/core/config.py b/backend/app/core/config.py index 926d13e..5abe72e 100644 --- a/backend/app/core/config.py +++ b/backend/app/core/config.py @@ -1,57 +1,326 @@ -from dataclasses import dataclass +from __future__ import annotations + +from dataclasses import dataclass, field from functools import lru_cache import os +from pathlib import Path +from typing import Any +from urllib.parse import quote, urlsplit, urlunsplit + +import yaml -def _int_env(name: str, default: int) -> int: - raw = os.getenv(name) - if raw is None or raw == "": +BACKEND_ROOT = Path(__file__).resolve().parents[2] +DEFAULT_CONFIG_PATH = BACKEND_ROOT / "config.yaml" +DEFAULT_DATABASE_BASE_URL = "postgresql+psycopg://localhost:5432/yg_ft" +DEFAULT_DATABASE_USERNAME = "yg_ft" +DEFAULT_DATABASE_PASSWORD = "change_me" + + +class ConfigurationError(ValueError): + """Raised when the backend configuration cannot be parsed or validated.""" + + +def _resolve_config_path(config_path: str | Path | None = None) -> tuple[Path, bool]: + configured_path = config_path or os.getenv("BACKEND_CONFIG_FILE") + is_explicit = configured_path is not None + path = Path(configured_path).expanduser() if configured_path else DEFAULT_CONFIG_PATH + if not path.is_absolute(): + path = BACKEND_ROOT / path + return path.resolve(), is_explicit + + +def _load_yaml(config_path: str | Path | None = None) -> tuple[dict[str, Any], Path]: + path, is_explicit = _resolve_config_path(config_path) + if not path.exists(): + if is_explicit: + raise ConfigurationError(f"Backend config file does not exist: {path}") + return {}, path + + try: + loaded = yaml.safe_load(path.read_text(encoding="utf-8")) + except yaml.YAMLError as exc: + raise ConfigurationError(f"Invalid YAML in backend config file {path}: {exc}") from exc + + if loaded is None: + return {}, path + if not isinstance(loaded, dict): + raise ConfigurationError(f"Backend config root must be a mapping: {path}") + return loaded, path + + +def _yaml_value(config: dict[str, Any], section: str, key: str, default: Any) -> Any: + section_value = config.get(section, {}) + if section_value is None: return default - return int(raw) + if not isinstance(section_value, dict): + raise ConfigurationError(f"Config section '{section}' must be a mapping") + return section_value.get(key, default) -def _list_env(name: str, default: list[str]) -> list[str]: - raw = os.getenv(name) - if raw is None or raw.strip() == "": - return default - return [item.strip() for item in raw.split(",") if item.strip()] +def _string_setting( + env_name: str, + config: dict[str, Any], + section: str, + key: str, + default: str, +) -> str: + env_value = os.getenv(env_name) + if env_value is not None: + return env_value + value = _yaml_value(config, section, key, default) + return default if value is None else str(value) + + +def _int_setting( + env_name: str, + config: dict[str, Any], + section: str, + key: str, + default: int, +) -> int: + env_value = os.getenv(env_name) + value = ( + env_value + if env_value not in (None, "") + else _yaml_value(config, section, key, default) + ) + if isinstance(value, (bool, float)): + raise ConfigurationError(f"Config value {section}.{key} must be an integer") + try: + return int(value) + except (TypeError, ValueError) as exc: + raise ConfigurationError(f"Config value {section}.{key} must be an integer") from exc + + +def _port_setting( + env_name: str, + config: dict[str, Any], + key: str, + default: int, +) -> int: + port = _int_setting(env_name, config, "server", key, default) + if not 1 <= port <= 65535: + raise ConfigurationError( + f"Config value server.{key} must be between 1 and 65535" + ) + return port + + +def _list_setting( + env_name: str, + config: dict[str, Any], + section: str, + key: str, + default: list[str], +) -> list[str]: + env_value = os.getenv(env_name) + value = ( + env_value + if env_value and env_value.strip() + else _yaml_value(config, section, key, default) + ) + if isinstance(value, str): + return [item.strip() for item in value.split(",") if item.strip()] + if isinstance(value, list): + return [str(item).strip() for item in value if str(item).strip()] + raise ConfigurationError( + f"Config value {section}.{key} must be a list or comma-separated string" + ) + + +def _path_setting( + env_name: str, + config: dict[str, Any], + section: str, + key: str, + default: str, + config_dir: Path, +) -> str: + env_value = os.getenv(env_name) + if env_value is not None: + return env_value + value = Path(str(_yaml_value(config, section, key, default))).expanduser() + if not value.is_absolute(): + value = config_dir / value + return str(value.resolve()) + + +def _build_database_url(base_url: str, username: str, password: str) -> str: + try: + parsed = urlsplit(base_url) + host = parsed.hostname + port = parsed.port + except ValueError as exc: + raise ConfigurationError("Config value database.url is not a valid URL") from exc + + if not parsed.scheme or not host: + raise ConfigurationError( + "Config value database.url must include a scheme and host" + ) + if parsed.scheme not in {"postgresql", "postgresql+psycopg"}: + raise ConfigurationError( + "Config value database.url must use postgresql or postgresql+psycopg" + ) + if parsed.path in {"", "/"}: + raise ConfigurationError("Config value database.url must include a database name") + if not username: + raise ConfigurationError("Config value database.username cannot be empty") + if not password: + raise ConfigurationError("Config value database.password cannot be empty") + + formatted_host = f"[{host}]" if ":" in host and not host.startswith("[") else host + host_and_port = f"{formatted_host}:{port}" if port is not None else formatted_host + credentials = f"{quote(username, safe='')}:{quote(password, safe='')}" + parts = ( + parsed.scheme, + f"{credentials}@{host_and_port}", + parsed.path, + parsed.query, + parsed.fragment, + ) + return urlunsplit(parts) + + +def _database_url(config: dict[str, Any]) -> str: + # 保留原有完整连接串环境变量,便于 Docker 和生产环境注入密钥。 + complete_url_override = os.getenv("DATABASE_URL") + if complete_url_override is not None: + return complete_url_override + + base_url = _string_setting( + "DATABASE_BASE_URL", + config, + "database", + "url", + DEFAULT_DATABASE_BASE_URL, + ) + username = _string_setting( + "DATABASE_USERNAME", + config, + "database", + "username", + DEFAULT_DATABASE_USERNAME, + ) + password = _string_setting( + "DATABASE_PASSWORD", + config, + "database", + "password", + DEFAULT_DATABASE_PASSWORD, + ) + return _build_database_url(base_url, username, password) @dataclass(frozen=True) class Settings: - app_name: str = os.getenv("APP_NAME", "YG Fine-Tune Platform API") - app_env: str = os.getenv("APP_ENV", "local") - route_prefix: str = os.getenv("MODELTF_ROUTE_PREFIX", "/modelTF") - app_mode: str = os.getenv("APP_MODE", "local") - database_url: str = os.getenv("DATABASE_URL", "postgresql+psycopg://yg_ft:change_me@localhost:15432/yg_ft") - cors_allow_origins: list[str] = None # type: ignore[assignment] - compute_mode: str = os.getenv("COMPUTE_MODE", "real") - compute_status_sync_mode: str = os.getenv("COMPUTE_STATUS_SYNC_MODE", "polling") - compute_poll_interval_seconds: int = _int_env("COMPUTE_POLL_INTERVAL_SECONDS", 3) - log_level: str = os.getenv("LOG_LEVEL", "INFO") - log_dir: str = os.getenv("LOG_DIR", "./logs") - log_file_prefix: str = os.getenv("LOG_FILE_PREFIX", "backend") - log_error_file_prefix: str = os.getenv("LOG_ERROR_FILE_PREFIX", "error") - log_max_bytes: int = _int_env("LOG_MAX_BYTES", 20 * 1024 * 1024) - log_retention_days: int = _int_env("LOG_RETENTION_DAYS", 10) + app_name: str = "YG Fine-Tune Platform API" + app_env: str = "local" + route_prefix: str = "/modelTF" + app_mode: str = "local" + frontend_port: int = 16801 + backend_port: int = 17861 + database_url: str = field( + default="postgresql+psycopg://yg_ft:change_me@localhost:5432/yg_ft", + repr=False, + ) + cors_allow_origins: list[str] = field( + default_factory=lambda: [ + "http://localhost:16801", + "http://127.0.0.1:16801", + ] + ) + compute_mode: str = "real" + compute_status_sync_mode: str = "polling" + compute_poll_interval_seconds: int = 3 + log_level: str = "INFO" + log_dir: str = "./logs" + log_file_prefix: str = "backend" + log_error_file_prefix: str = "error" + log_max_bytes: int = 20 * 1024 * 1024 + log_retention_days: int = 10 - def __post_init__(self) -> None: - object.__setattr__( - self, - "cors_allow_origins", - _list_env( - "CORS_ALLOW_ORIGINS", - [ - "http://localhost:16801", - "http://127.0.0.1:16801", - "http://localhost:17861", - "http://127.0.0.1:17861", - ], - ), + +def load_settings(config_path: str | Path | None = None) -> Settings: + config, resolved_config_path = _load_yaml(config_path) + frontend_port = _port_setting( + "FRONTEND_PORT", config, "frontend_port", Settings.frontend_port + ) + backend_port = _port_setting( + "BACKEND_PORT", config, "backend_port", Settings.backend_port + ) + if frontend_port == backend_port: + raise ConfigurationError( + "Config values server.frontend_port and server.backend_port must be different" ) + default_cors = [ + f"http://localhost:{frontend_port}", + f"http://127.0.0.1:{frontend_port}", + ] + return Settings( + app_name=_string_setting("APP_NAME", config, "app", "name", Settings.app_name), + app_env=_string_setting("APP_ENV", config, "app", "env", Settings.app_env), + route_prefix=_string_setting( + "MODELTF_ROUTE_PREFIX", config, "app", "route_prefix", Settings.route_prefix + ), + app_mode=_string_setting("APP_MODE", config, "app", "mode", Settings.app_mode), + frontend_port=frontend_port, + backend_port=backend_port, + database_url=_database_url(config), + cors_allow_origins=_list_setting( + "CORS_ALLOW_ORIGINS", config, "app", "cors_allow_origins", default_cors + ), + compute_mode=_string_setting( + "COMPUTE_MODE", config, "compute", "mode", Settings.compute_mode + ), + compute_status_sync_mode=_string_setting( + "COMPUTE_STATUS_SYNC_MODE", + config, + "compute", + "status_sync_mode", + Settings.compute_status_sync_mode, + ), + compute_poll_interval_seconds=_int_setting( + "COMPUTE_POLL_INTERVAL_SECONDS", + config, + "compute", + "poll_interval_seconds", + Settings.compute_poll_interval_seconds, + ), + log_level=_string_setting( + "LOG_LEVEL", config, "logging", "level", Settings.log_level + ), + log_dir=_path_setting( + "LOG_DIR", + config, + "logging", + "directory", + Settings.log_dir, + resolved_config_path.parent, + ), + log_file_prefix=_string_setting( + "LOG_FILE_PREFIX", config, "logging", "file_prefix", Settings.log_file_prefix + ), + log_error_file_prefix=_string_setting( + "LOG_ERROR_FILE_PREFIX", + config, + "logging", + "error_file_prefix", + Settings.log_error_file_prefix, + ), + log_max_bytes=_int_setting( + "LOG_MAX_BYTES", config, "logging", "max_bytes", Settings.log_max_bytes + ), + log_retention_days=_int_setting( + "LOG_RETENTION_DAYS", + config, + "logging", + "retention_days", + Settings.log_retention_days, + ), + ) @lru_cache def get_settings() -> Settings: - return Settings() - + return load_settings() diff --git a/backend/app/db/platform_store.py b/backend/app/db/platform_store.py index 8f87d8f..883601c 100644 --- a/backend/app/db/platform_store.py +++ b/backend/app/db/platform_store.py @@ -78,7 +78,10 @@ def verify_password(password: str, stored: str) -> tuple[bool, bool]: def _psycopg_url(database_url: str) -> str: - return database_url.replace("postgresql+psycopg://", "postgresql://") + sqlalchemy_prefix = "postgresql+psycopg://" + if database_url.startswith(sqlalchemy_prefix): + return f"postgresql://{database_url[len(sqlalchemy_prefix):]}" + return database_url def _pg_sql(sql: str) -> str: @@ -1147,4 +1150,3 @@ def get_platform_store() -> PlatformStore: if _store is None: _store = PlatformStore() return _store - diff --git a/backend/app/db/session.py b/backend/app/db/session.py index befe080..49cf713 100644 --- a/backend/app/db/session.py +++ b/backend/app/db/session.py @@ -1,14 +1,15 @@ from __future__ import annotations -import os from collections.abc import Generator from contextlib import contextmanager from sqlalchemy import create_engine from sqlalchemy.orm import Session, sessionmaker +from app.core.config import get_settings -DATABASE_URL = os.getenv("DATABASE_URL", "postgresql+psycopg://yg_ft:change_me@localhost:15432/yg_ft") + +DATABASE_URL = get_settings().database_url engine = create_engine( DATABASE_URL, @@ -37,4 +38,3 @@ def session_scope() -> Generator[Session, None, None]: raise finally: db.close() - diff --git a/backend/config.example.yaml b/backend/config.example.yaml new file mode 100644 index 0000000..74125fa --- /dev/null +++ b/backend/config.example.yaml @@ -0,0 +1,31 @@ +# 后端配置模板。复制为 config.yaml 后填写本地数据库凭据。 +# config.yaml 已加入 Git 忽略,不会提交真实密码。 + +app: + name: YG Fine-Tune Platform API + env: local + mode: local + route_prefix: /modelTF + +server: + # 一键启动脚本统一读取这里的端口;端口被占用时只需修改这两项。 + frontend_port: 16801 + backend_port: 17861 + +database: + url: postgresql+psycopg://localhost:5432/yg_ft + username: yg_ft + password: "change_me" + +compute: + mode: real + status_sync_mode: polling + poll_interval_seconds: 3 + +logging: + level: INFO + directory: ./logs + file_prefix: backend + error_file_prefix: error + max_bytes: 20971520 + retention_days: 10 diff --git a/backend/pyproject.toml b/backend/pyproject.toml index 042a8ab..aaac9a3 100644 --- a/backend/pyproject.toml +++ b/backend/pyproject.toml @@ -16,6 +16,7 @@ dependencies = [ "PyJWT>=2.8.0", "passlib[bcrypt]>=1.7.4", "python-dotenv>=1.0.1", + "PyYAML>=6.0.2", ] [project.optional-dependencies] diff --git a/backend/requirements.txt b/backend/requirements.txt index 6ef0933..a55defe 100644 --- a/backend/requirements.txt +++ b/backend/requirements.txt @@ -10,3 +10,4 @@ httpx>=0.27.0 PyJWT>=2.8.0 passlib[bcrypt]>=1.7.4 python-dotenv>=1.0.1 +PyYAML>=6.0.2 diff --git a/backend/tests/test_config.py b/backend/tests/test_config.py new file mode 100644 index 0000000..9a7f08a --- /dev/null +++ b/backend/tests/test_config.py @@ -0,0 +1,196 @@ +from __future__ import annotations + +from pathlib import Path + +import pytest + +from app.core.config import ConfigurationError, load_settings + + +CONFIG_ENV_NAMES = ( + "APP_NAME", + "APP_ENV", + "APP_MODE", + "MODELTF_ROUTE_PREFIX", + "FRONTEND_PORT", + "BACKEND_PORT", + "DATABASE_URL", + "DATABASE_BASE_URL", + "DATABASE_USERNAME", + "DATABASE_PASSWORD", + "CORS_ALLOW_ORIGINS", + "COMPUTE_MODE", + "COMPUTE_STATUS_SYNC_MODE", + "COMPUTE_POLL_INTERVAL_SECONDS", + "LOG_LEVEL", + "LOG_DIR", + "LOG_FILE_PREFIX", + "LOG_ERROR_FILE_PREFIX", + "LOG_MAX_BYTES", + "LOG_RETENTION_DAYS", +) + + +def clear_config_env(monkeypatch: pytest.MonkeyPatch) -> None: + for env_name in CONFIG_ENV_NAMES: + monkeypatch.delenv(env_name, raising=False) + + +def write_config(path: Path) -> None: + path.write_text( + """ +app: + name: YAML API + route_prefix: /yaml-api + cors_allow_origins: + - http://yaml.example +server: + frontend_port: 18001 + backend_port: 18002 +database: + url: postgresql+psycopg://db:5432/yaml + username: yaml-user + password: yaml-secret +compute: + mode: simulated + poll_interval_seconds: 9 +logging: + directory: ./yaml-logs + max_bytes: 1024 + retention_days: 2 +""".strip(), + encoding="utf-8", + ) + + +def test_load_settings_from_yaml(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + clear_config_env(monkeypatch) + config_path = tmp_path / "config.yaml" + write_config(config_path) + + settings = load_settings(config_path) + + assert settings.app_name == "YAML API" + assert settings.route_prefix == "/yaml-api" + assert settings.frontend_port == 18001 + assert settings.backend_port == 18002 + assert settings.database_url == "postgresql+psycopg://yaml-user:yaml-secret@db:5432/yaml" + assert settings.cors_allow_origins == ["http://yaml.example"] + assert settings.compute_mode == "simulated" + assert settings.compute_poll_interval_seconds == 9 + assert settings.log_dir == str(tmp_path / "yaml-logs") + assert settings.log_max_bytes == 1024 + assert settings.log_retention_days == 2 + + +def test_environment_overrides_yaml(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + clear_config_env(monkeypatch) + config_path = tmp_path / "config.yaml" + write_config(config_path) + monkeypatch.setenv("DATABASE_URL", "postgresql+psycopg://env:secret@db:5432/env") + monkeypatch.setenv("CORS_ALLOW_ORIGINS", "http://one.example,http://two.example") + monkeypatch.setenv("COMPUTE_POLL_INTERVAL_SECONDS", "15") + monkeypatch.setenv("FRONTEND_PORT", "19001") + monkeypatch.setenv("BACKEND_PORT", "19002") + + settings = load_settings(config_path) + + assert settings.database_url == "postgresql+psycopg://env:secret@db:5432/env" + assert settings.cors_allow_origins == ["http://one.example", "http://two.example"] + assert settings.compute_poll_interval_seconds == 15 + assert settings.frontend_port == 19001 + assert settings.backend_port == 19002 + + +def test_separate_database_credentials_are_encoded( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + clear_config_env(monkeypatch) + config_path = tmp_path / "config.yaml" + write_config(config_path) + monkeypatch.setenv("DATABASE_USERNAME", "user@example.com") + monkeypatch.setenv("DATABASE_PASSWORD", "secret:/?#[]@") + + settings = load_settings(config_path) + + assert settings.database_url == ( + "postgresql+psycopg://user%40example.com:secret%3A%2F%3F%23%5B%5D%40@db:5432/yaml" + ) + + +def test_explicit_missing_config_is_rejected(tmp_path: Path) -> None: + with pytest.raises(ConfigurationError, match="does not exist"): + load_settings(tmp_path / "missing.yaml") + + +def test_default_cors_follows_frontend_port( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + clear_config_env(monkeypatch) + config_path = tmp_path / "config.yaml" + config_path.write_text( + "server:\n frontend_port: 28001\n backend_port: 28002\n", + encoding="utf-8", + ) + + settings = load_settings(config_path) + + assert settings.cors_allow_origins == [ + "http://localhost:28001", + "http://127.0.0.1:28001", + ] + + +@pytest.mark.parametrize("frontend_port", [0, 65536, "invalid", True]) +def test_invalid_server_port_is_rejected( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + frontend_port: object, +) -> None: + clear_config_env(monkeypatch) + config_path = tmp_path / "config.yaml" + config_path.write_text( + "server:\n" + f" frontend_port: {str(frontend_port).lower()}\n" + " backend_port: 28002\n", + encoding="utf-8", + ) + + with pytest.raises(ConfigurationError, match="server.frontend_port"): + load_settings(config_path) + + +def test_frontend_and_backend_ports_must_differ( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + clear_config_env(monkeypatch) + config_path = tmp_path / "config.yaml" + config_path.write_text( + "server:\n frontend_port: 28001\n backend_port: 28001\n", + encoding="utf-8", + ) + + with pytest.raises(ConfigurationError, match="must be different"): + load_settings(config_path) + + +@pytest.mark.parametrize( + ("database_base_url", "message"), + [ + ("mysql://localhost:3306/yg_ft", "must use postgresql"), + ("postgresql+psycopg://localhost:5432", "must include a database name"), + ], +) +def test_invalid_database_url_is_rejected( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + database_base_url: str, + message: str, +) -> None: + clear_config_env(monkeypatch) + config_path = tmp_path / "config.yaml" + write_config(config_path) + monkeypatch.setenv("DATABASE_BASE_URL", database_base_url) + + with pytest.raises(ConfigurationError, match=message): + load_settings(config_path) diff --git a/docker/app/Dockerfile.backend b/docker/app/Dockerfile.backend index 1f03dca..f842eb2 100644 --- a/docker/app/Dockerfile.backend +++ b/docker/app/Dockerfile.backend @@ -11,7 +11,7 @@ RUN pip install --upgrade pip -i https://pypi.tuna.tsinghua.edu.cn/simple \ && pip install -r /tmp/requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple \ && rm -f /tmp/requirements.txt -RUN python -c "import fastapi, uvicorn, psycopg, sqlalchemy, redis, jwt, passlib, httpx, alembic; print('backend dependency check ok')" +RUN python -c "import fastapi, uvicorn, psycopg, sqlalchemy, redis, yaml, jwt, passlib, httpx, alembic; print('backend dependency check ok')" RUN mkdir -p /opt/yg-ft/logs/backend /data/yg-ft \ && chmod -R 0775 /opt/yg-ft /data/yg-ft diff --git a/frontend/README.md b/frontend/README.md index 7220265..3a9bcf0 100644 --- a/frontend/README.md +++ b/frontend/README.md @@ -22,9 +22,9 @@ npm install npm run dev ``` -开发服务器默认运行在 `http://localhost:16801`。 - -后端 API 默认通过 Vite 代理转发到 `http://localhost:17861`(见 `vite.config.ts`)。 +通过根目录 `scripts/start-dev.sh` 启动时,开发服务器端口和后端代理端口来自 +`backend/config.yaml` 的 `server` 配置。单独运行 `npm run dev` 时默认使用前端 +`16801` 和后端 `17861`;可用 `FRONTEND_PORT`、`BACKEND_PORT` 环境变量覆盖。 开发环境默认联调真实后端接口。如需进行隔离前端开发,可显式启用 Mock: diff --git a/frontend/vite.config.ts b/frontend/vite.config.ts index 0dc2a57..a0a13cb 100644 --- a/frontend/vite.config.ts +++ b/frontend/vite.config.ts @@ -5,6 +5,17 @@ import Components from 'unplugin-vue-components/vite' import { ElementPlusResolver } from 'unplugin-vue-components/resolvers' import path from 'node:path' +function configuredPort(name: 'FRONTEND_PORT' | 'BACKEND_PORT', fallback: number) { + const port = Number(process.env[name] ?? fallback) + if (!Number.isInteger(port) || port < 1 || port > 65535) { + throw new Error(`${name} must be an integer between 1 and 65535`) + } + return port +} + +const frontendPort = configuredPort('FRONTEND_PORT', 16801) +const backendPort = configuredPort('BACKEND_PORT', 17861) + // https://vite.dev/config/ export default defineConfig({ plugins: [ @@ -22,11 +33,12 @@ export default defineConfig({ }, }, server: { - port: 16801, + port: frontendPort, + strictPort: true, proxy: { - // Frontend uses /modelTF and proxies to the local five-digit backend port. + // 一键启动脚本会将 YAML 中的后端端口注入当前 Vite 进程。 '/modelTF': { - target: 'http://localhost:17861', + target: `http://127.0.0.1:${backendPort}`, changeOrigin: true, }, }, diff --git a/scripts/start-dev.sh b/scripts/start-dev.sh new file mode 100755 index 0000000..2fece73 --- /dev/null +++ b/scripts/start-dev.sh @@ -0,0 +1,277 @@ +#!/usr/bin/env bash + +set -Eeuo pipefail +set -m + +ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +BACKEND_DIR="$ROOT_DIR/backend" +FRONTEND_DIR="$ROOT_DIR/frontend" +BACKEND_VENV="$BACKEND_DIR/.venv" +BACKEND_PYTHON="$BACKEND_VENV/bin/python" + +DO_SETUP=false +CHECK_ONLY=false +BACKEND_PID="" +FRONTEND_PID="" +CONFIG_PATH="" +FRONTEND_PORT="" +BACKEND_PORT="" + +usage() { + cat <<'EOF' +用法: + ./scripts/start-dev.sh 启动前端和后端 + ./scripts/start-dev.sh --setup 同步本地依赖后启动 + ./scripts/start-dev.sh --check 只检查依赖、配置和数据库连接 + +环境变量: + PYTHON_BIN 创建虚拟环境时使用的 Python 3.12+ 可执行文件 + BACKEND_CONFIG_FILE 指定其他后端 YAML 配置文件 + FRONTEND_PORT 覆盖 YAML 中的前端端口 + BACKEND_PORT 覆盖 YAML 中的后端端口 +EOF +} + +fail() { + echo "❌ $*" >&2 + exit 1 +} + +process_group_alive() { + local pid="$1" + [[ -n "$pid" ]] && kill -0 -- "-$pid" 2>/dev/null +} + +stop_process_group() { + local pid="$1" + local attempt=0 + [[ -n "$pid" ]] || return 0 + + if process_group_alive "$pid"; then + kill -TERM -- "-$pid" 2>/dev/null || true + while process_group_alive "$pid" && [[ $attempt -lt 50 ]]; do + sleep 0.1 + attempt=$((attempt + 1)) + done + if process_group_alive "$pid"; then + kill -KILL -- "-$pid" 2>/dev/null || true + fi + elif kill -0 "$pid" 2>/dev/null; then + kill -TERM "$pid" 2>/dev/null || true + fi + wait "$pid" 2>/dev/null || true +} + +cleanup() { + trap - EXIT + stop_process_group "$FRONTEND_PID" + stop_process_group "$BACKEND_PID" +} + +handle_signal() { + echo + echo "🛑 正在停止前端和后端……" + exit 130 +} + +port_in_use() { + local port="$1" + lsof -nP -iTCP:"$port" -sTCP:LISTEN >/dev/null 2>&1 +} + +select_base_python() { + if [[ -n "${PYTHON_BIN:-}" ]]; then + printf '%s\n' "$PYTHON_BIN" + elif [[ -x /opt/miniconda3/bin/python3 ]]; then + printf '%s\n' /opt/miniconda3/bin/python3 + else + command -v python3 || true + fi +} + +setup_dependencies() { + local base_python + base_python="$(select_base_python)" + [[ -n "$base_python" ]] || fail "未找到 Python 3,请通过 PYTHON_BIN 指定 Python 3.12+。" + + "$base_python" -c \ + 'import sys; raise SystemExit(0 if sys.version_info >= (3, 12) else 1)' \ + || fail "后端要求 Python 3.12+,当前为 $($base_python --version 2>&1)。" + + if [[ ! -x "$BACKEND_PYTHON" ]]; then + echo "📦 创建后端虚拟环境……" + "$base_python" -m venv "$BACKEND_VENV" + fi + + echo "📦 同步后端依赖……" + "$BACKEND_PYTHON" -m pip install -r "$BACKEND_DIR/requirements.txt" + + echo "📦 同步前端依赖……" + (cd "$FRONTEND_DIR" && npm ci) + + if [[ -z "${BACKEND_CONFIG_FILE:-}" && ! -f "$BACKEND_DIR/config.yaml" ]]; then + cp "$BACKEND_DIR/config.example.yaml" "$BACKEND_DIR/config.yaml" + echo "📝 已根据 config.example.yaml 创建本地 config.yaml,请填写数据库密码。" + fi +} + +resolve_and_check_config() { + local port_values + + CONFIG_PATH="$( + cd "$BACKEND_DIR" + "$BACKEND_PYTHON" -c \ + 'from app.core.config import _resolve_config_path; print(_resolve_config_path()[0])' + )" || fail "无法解析后端配置文件路径。" + + [[ -f "$CONFIG_PATH" ]] || fail "配置文件不存在:$CONFIG_PATH" + port_values="$( + cd "$BACKEND_DIR" + "$BACKEND_PYTHON" -c \ + 'from app.core.config import get_settings; settings = get_settings(); print(settings.frontend_port, settings.backend_port)' + )" || fail "配置文件无法加载:$CONFIG_PATH" + + read -r FRONTEND_PORT BACKEND_PORT <<< "$port_values" + [[ -n "$FRONTEND_PORT" && -n "$BACKEND_PORT" ]] \ + || fail "无法从配置文件读取前端和后端端口:$CONFIG_PATH" +} + +check_dependencies() { + command -v node >/dev/null 2>&1 || fail "未找到 Node.js。" + command -v npm >/dev/null 2>&1 || fail "未找到 npm。" + command -v curl >/dev/null 2>&1 || fail "未找到 curl。" + command -v lsof >/dev/null 2>&1 || fail "未找到 lsof。" + [[ -x "$FRONTEND_DIR/node_modules/.bin/vite" ]] \ + || fail "前端依赖不完整,请先运行 ./scripts/start-dev.sh --setup。" + [[ -x "$BACKEND_PYTHON" ]] \ + || fail "后端虚拟环境不存在,请先运行 ./scripts/start-dev.sh --setup。" + + "$BACKEND_PYTHON" -c 'import fastapi, psycopg, sqlalchemy, uvicorn, yaml' \ + || fail "后端依赖不完整,请先运行 ./scripts/start-dev.sh --setup。" + resolve_and_check_config +} + +check_database() { + echo "🔍 检查 PostgreSQL 连接:$CONFIG_PATH" + ( + cd "$BACKEND_DIR" + "$BACKEND_PYTHON" -c ' +import psycopg +from app.core.config import get_settings + +url = get_settings().database_url +prefix = "postgresql+psycopg://" +if url.startswith(prefix): + url = "postgresql://" + url[len(prefix):] +connection = psycopg.connect(url, connect_timeout=3) +connection.close() +' + ) || fail "PostgreSQL 无法连接,请检查配置文件中的地址、用户名和密码:$CONFIG_PATH" +} + +wait_for_url() { + local url="$1" + local service_name="$2" + local pid="$3" + local attempt=0 + local max_attempts=30 + + while [[ $attempt -lt $max_attempts ]]; do + if curl -fsS --max-time 1 "$url" >/dev/null 2>&1; then + return 0 + fi + if ! kill -0 "$pid" 2>/dev/null; then + echo "❌ $service_name 进程已提前退出。" >&2 + return 1 + fi + sleep 1 + attempt=$((attempt + 1)) + done + + echo "❌ 等待 $service_name 就绪超时:$url" >&2 + return 1 +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --setup) + DO_SETUP=true + ;; + --check) + CHECK_ONLY=true + ;; + -h|--help) + usage + exit 0 + ;; + *) + usage >&2 + fail "未知参数:$1" + ;; + esac + shift +done + +if [[ "$DO_SETUP" == true ]]; then + setup_dependencies +fi + +check_dependencies +check_database + +if [[ "$CHECK_ONLY" == true ]]; then + echo "✅ 前端、后端依赖、配置和 PostgreSQL 连接均正常。" + exit 0 +fi + +port_in_use "$BACKEND_PORT" \ + && fail "后端端口 $BACKEND_PORT 已被占用,后端可能已经启动。" +port_in_use "$FRONTEND_PORT" \ + && fail "前端端口 $FRONTEND_PORT 已被占用,前端可能已经启动。" + +trap cleanup EXIT +trap handle_signal INT TERM + +echo "🚀 启动后端:http://127.0.0.1:$BACKEND_PORT" +( + cd "$BACKEND_DIR" + exec "$BACKEND_PYTHON" -m uvicorn app.main:app \ + --host 0.0.0.0 \ + --port "$BACKEND_PORT" \ + --reload +) & +BACKEND_PID=$! + +wait_for_url "http://127.0.0.1:$BACKEND_PORT/modelTF/health" "后端" "$BACKEND_PID" \ + || fail "后端启动失败。" + +echo "🚀 启动前端:http://127.0.0.1:$FRONTEND_PORT" +( + cd "$FRONTEND_DIR" + export FRONTEND_PORT BACKEND_PORT + exec npm run dev -- --host 0.0.0.0 --port "$FRONTEND_PORT" --strictPort +) & +FRONTEND_PID=$! + +wait_for_url "http://127.0.0.1:$FRONTEND_PORT" "前端" "$FRONTEND_PID" \ + || fail "前端启动失败。" + +echo "✅ 前端和后端均已就绪,按 Ctrl+C 可同时停止。" + +while kill -0 "$BACKEND_PID" 2>/dev/null && kill -0 "$FRONTEND_PID" 2>/dev/null; do + sleep 1 +done + +set +e +if ! kill -0 "$BACKEND_PID" 2>/dev/null; then + wait "$BACKEND_PID" + EXIT_CODE=$? + echo "❌ 后端已退出,正在停止前端。" >&2 +else + wait "$FRONTEND_PID" + EXIT_CODE=$? + echo "❌ 前端已退出,正在停止后端。" >&2 +fi +set -e + +exit "$EXIT_CODE"