feat: 新增外部数据源拉取与 DPO 输出格式支持
- 支持从 PostgreSQL 数据库拉取结构化数据作为训练来源 - 新增 DPO (Direct Preference Optimization) 输出类型 - 支持 chosen/rejected 字段的编辑、校验和发布 - 完善数据预处理切分逻辑和元数据管理 - 移除 OCR 扫描 PDF 功能,保持基础文本解析能力 Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -2,7 +2,8 @@ FROM python:3.12-slim
|
||||
|
||||
ENV PYTHONDONTWRITEBYTECODE=1 \
|
||||
PYTHONUNBUFFERED=1 \
|
||||
PIP_NO_CACHE_DIR=1
|
||||
PIP_NO_CACHE_DIR=1 \
|
||||
TIKTOKEN_CACHE_DIR=/opt/tiktoken_cache
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
@@ -16,6 +17,9 @@ RUN python -c "import fastapi, uvicorn, psycopg, psycopg_pool, sqlalchemy, redis
|
||||
RUN mkdir -p /opt/yg-ft/logs/backend /data/yg-ft \
|
||||
&& chmod -R 0775 /opt/yg-ft /data/yg-ft
|
||||
|
||||
# 离线打包 tiktoken cl100k_base 词表,避免无网环境下运行时联网下载
|
||||
COPY docker/app/tiktoken /opt/tiktoken_cache
|
||||
|
||||
EXPOSE 8000
|
||||
|
||||
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]
|
||||
|
||||
Reference in New Issue
Block a user