feat: 新增外部数据源拉取与 DPO 输出格式支持

Browse Source

- 支持从 PostgreSQL 数据库拉取结构化数据作为训练来源
- 新增 DPO (Direct Preference Optimization) 输出类型
- 支持 chosen/rejected 字段的编辑、校验和发布
- 完善数据预处理切分逻辑和元数据管理
- 移除 OCR 扫描 PDF 功能，保持基础文本解析能力

Co-Authored-By: Claude <noreply@anthropic.com>

...

This commit is contained in:

caoxiaozhu

2026-08-11 14:17:45 +08:00

parent f809825a7d

commit 5f6e7523cf

26 changed files with 927 additions and 144 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

BIN

backend/app/modules/data_process/document_chunking.txt Normal file

Binary file not shown.