# 屏蔽数据处理 OCR 能力实施计划 > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** 在保留 Docling 版面分析和 `layout_hybrid` 的前提下，显式关闭 PDF pipeline 的 OCR，并拒绝无文本层 PDF。 **Architecture:** 只修改 `document_chunking.py` 中的 Docling converter 构造，使 PDF 使用 `PdfPipelineOptions(do_ocr=False)`；不修改切分方式、依赖、前端配置或版面 chunker。pypdf 继续负责上传阶段文本提取和扫描 PDF 边界判断。 **Tech Stack:** Python 3.12、FastAPI、pypdf、Docling 2.115.0、LlamaIndex、pytest、Ruff、Vue 3、Node 回归脚本。 ## Global Constraints - 必须保留 `layout_hybrid`、Docling 版面分析和现有前端切分选项。 - 只能关闭 OCR，不删除 Docling、版面/表格分析、PDF 页码映射或普通切分。 - 纯扫描/图片型 PDF 明确拒绝；混合 PDF 保留，图片页不产生文字。 - 不新增 OCR 开关，不提供重新开启 OCR 的参数或环境变量。 - 不修改 `.codex-tmp/` 和 `outputs/` 等既有未跟踪用户文件。 - 不执行 `git commit`；提交前必须取得用户明确确认。 --- ### Task 1: 显式关闭 Docling PDF OCR **Files:** - Modify: `backend/app/modules/data_process/document_chunking.py:261-265` - Test: `backend/tests/test_document_chunking.py` **Interfaces:** - `_document_converter()` 保持无参数、缓存和返回 `DocumentConverter` 的现有接口。 - PDF 的 `PdfFormatOption` 必须收到 `pipeline_options.do_ocr is False`。 - [ ] **Step 1: Add a failing configuration test** 在 `backend/tests/test_document_chunking.py` 导入 `_document_converter`、`sys`、`ModuleType` 和 `SimpleNamespace`，添加使用假的 Docling 模块的测试： ```python def test_layout_converter_disables_ocr(monkeypatch) -> None: class FakePipelineOptions: def __init__(self) -> None: self.do_ocr = True class FakePdfFormatOption: def __init__(self, *, pipeline_options) -> None: self.pipeline_options = pipeline_options class FakeDocumentConverter: def __init__(self, *, format_options) -> None: self.format_options = format_options # 将这些 fake 类注册到 docling.document_converter、 # docling.datamodel.base_models 和 docling.datamodel.pipeline_options。 # 调用 _document_converter 后断言： _document_converter.cache_clear() converter = _document_converter() assert converter.format_options["pdf"].pipeline_options.do_ocr is False _document_converter.cache_clear() ``` 测试保留现有版面投影和 provenance 测试，证明本任务不是删除版面分析。 - [ ] **Step 2: Run the new test before implementation** Run: ```bash cd backend && PYTHONPATH=. .venv/bin/pytest tests/test_document_chunking.py::test_layout_converter_disables_ocr -q ``` Expected: FAIL，因为当前 `_document_converter()` 没有 PDF pipeline options，fake converter 不会收到 `format_options`。 - [ ] **Step 3: Implement the explicit PDF options** 将 `_document_converter()` 改为： ```python @lru_cache(maxsize=1) def _document_converter(): from docling.datamodel.base_models import InputFormat from docling.datamodel.pipeline_options import PdfPipelineOptions from docling.document_converter import DocumentConverter, PdfFormatOption pipeline_options = PdfPipelineOptions() pipeline_options.do_ocr = False return DocumentConverter( format_options={ InputFormat.PDF: PdfFormatOption(pipeline_options=pipeline_options), } ) ``` 不要删除 `_MarkdownSerializerProvider`、`chunk_layout_document`、`HybridChunker`、来源页码或 bbox 代码。 - [ ] **Step 4: Run OCR and layout regression tests** Run: ```bash cd backend && PYTHONPATH=. .venv/bin/pytest tests/test_document_chunking.py::test_layout_converter_disables_ocr tests/test_document_chunking.py::test_layout_projection_ignores_layout_whitespace_but_keeps_source_lines -q ``` Expected: PASS；OCR 配置关闭，版面文本投影仍通过。 --- ### Task 2: 更新无文本 PDF 的错误和文档 **Files:** - Modify: `backend/app/modules/data_process/algorithms/parsers/pdf.py:106-108` - Modify: `backend/tests/test_data_process_algorithms.py:650-658` - Modify: `docs/data-process-design.md:121-122` **Interfaces:** - `extract_pdf_page_texts()` 的判定逻辑不变，只更新无文本 PDF 的错误文案。 - 混合 PDF 仍由 `has_text` 判定为可处理。 - [ ] **Step 1: Update the expected error** 将测试断言改为： ```python with pytest.raises( ValueError, match="scanned or image-only PDF files are not supported", ): parse_text_content(blank_pdf.getvalue(), filename="scanned.pdf") ``` - [ ] **Step 2: Change only the error message** 将解析器分支改为： ```python if not has_text: raise ValueError( "PDF contains no extractable text; scanned or image-only PDF files are not supported" ) ``` 不要修改 `has_text` 逻辑，以保留混合 PDF。 - [ ] **Step 3: Update the design documentation** 将 `docs/data-process-design.md` 中“扫描 PDF 没有文本层时明确提示需要 OCR；当前流程不执行 OCR”改为“扫描版或图片型 PDF 不支持”，保留其余格式限制说明。 - [ ] **Step 4: Run PDF boundary tests** Run: ```bash cd backend && PYTHONPATH=. .venv/bin/pytest tests/test_data_process_algorithms.py::test_office_zip_bomb_and_invalid_pdf_are_rejected_before_parsing -q ``` Expected: PASS；纯扫描 PDF 明确拒绝。 --- ### Task 3: 运行受影响验证并检查范围 **Files:** - Test: `backend/tests/test_document_chunking.py` - Test: `backend/tests/test_data_process_algorithms.py` - Test: `frontend/scripts/regression-data-process-wizard.mjs` - [ ] **Step 1: Run targeted backend tests and Ruff** Run: ```bash cd backend && PYTHONPATH=. .venv/bin/pytest tests/test_document_chunking.py::test_layout_converter_disables_ocr tests/test_document_chunking.py::test_layout_projection_ignores_layout_whitespace_but_keeps_source_lines tests/test_data_process_algorithms.py::test_office_zip_bomb_and_invalid_pdf_are_rejected_before_parsing -q .venv/bin/ruff check app/modules/data_process/document_chunking.py tests/test_document_chunking.py app/modules/data_process/algorithms/parsers/pdf.py ``` Expected: targeted tests pass. Any pre-existing Ruff findings outside the changed OCR configuration must be reported separately, not fixed as unrelated refactoring. - [ ] **Step 2: Run the frontend regression and type check** Run: ```bash cd frontend && node scripts/regression-data-process-wizard.mjs npm run type-check ``` Expected: existing `layout_hybrid`/semantic/fixed assertions remain unchanged; any unrelated pre-existing regression failure is reported separately. - [ ] **Step 3: Check OCR references and final diff** Run from the repository root: ```bash rg -n -i --hidden \ --glob '!.git/**' \ --glob '!node_modules/**' \ --glob '!dist/**' \ --glob '!outputs/**' \ --glob '!.codex-tmp/**' \ '(tesseract|paddleocr|easyocr|ocrmypdf|pytesseract)' \ backend frontend docs/data-process-design.md git diff --check git status --short ``` Expected: no OCR engine dependency or executable OCR integration; the only intended OCR-related runtime setting is `do_ocr = False`, and the diff does not touch layout options, frontend method choices, Docling dependencies, `.codex-tmp/` or `outputs/`.