feat(agents): Phase 8.4-10.5 built-in plugins, bundled skills, coordinator
This commit is contained in:
52
development-doc/README.md
Normal file
52
development-doc/README.md
Normal file
@@ -0,0 +1,52 @@
|
||||
# Development Doc
|
||||
|
||||
本目录用于持续记录 Jarvis 的规划与开发过程。
|
||||
|
||||
## 目录结构
|
||||
|
||||
### `plan/`
|
||||
用于记录中长期规划、升级方案、分阶段设计、专项计划。
|
||||
|
||||
建议内容：
|
||||
- 每个升级点的目标
|
||||
- 计划改动的模块和文件
|
||||
- 分阶段实施顺序
|
||||
- 风险与验收标准
|
||||
- 设计决策与取舍
|
||||
|
||||
当前文件：
|
||||
- `plan/README.md`
|
||||
- `plan/phase-0-current-state-and-target.md`
|
||||
- `plan/phase-1-safe-foundation.md`
|
||||
- `plan/phase-2-controlled-collaboration.md`
|
||||
- `plan/phase-3-dynamic-collaboration.md`
|
||||
- `plan/phase-4-visibility-and-isolation.md`
|
||||
|
||||
### `daily/`
|
||||
用于记录每日工作日志、当天进展、开发计划、执行情况、问题、决策和下一步安排。
|
||||
|
||||
建议内容：
|
||||
- 今日开发计划
|
||||
- 今日实际完成内容
|
||||
- 当前进度
|
||||
- 修改了哪些模块 / 文件
|
||||
- 当前阻塞点
|
||||
- 风险与临时决策
|
||||
- 下一步计划
|
||||
- 验证 / 测试情况
|
||||
|
||||
维护要求：
|
||||
- 开始做当天开发前，先写当天计划
|
||||
- 每完成一个关键步骤后，及时补充进度
|
||||
- 遇到阻塞、改方案、改优先级时，必须更新 daily
|
||||
- 一天结束前，补齐“完成情况 / 未完成 / 下一步”
|
||||
- 后续正式改代码时，要把 daily 作为持续更新的开发日志，而不是事后总结
|
||||
|
||||
当前文件：
|
||||
- `daily/2026-04-03.md`
|
||||
- `daily/2026-04-04.md`
|
||||
|
||||
补充约定：
|
||||
- 多天改造内容不要挤在同一个 daily 文档里
|
||||
- 每一天单独一个 daily 文件
|
||||
- 分步骤执行时，已完成项使用 `~~删除线~~` 标记
|
||||
203
development-doc/daily/2026-04-03.md
Normal file
203
development-doc/daily/2026-04-03.md
Normal file
@@ -0,0 +1,203 @@
|
||||
# 2026-04-03 工作日志
|
||||
|
||||
## 今日开发计划
|
||||
|
||||
### 今日目标
|
||||
|
||||
- ~~分析 `demo/` 下三个 agent 项目与 Jarvis 当前 agents 的差异~~
|
||||
- ~~明确 Jarvis 2.0 升级方向~~
|
||||
- ~~建立 development-doc 文档结构~~
|
||||
- ~~形成分阶段 plan 文档~~
|
||||
- ~~形成 2 天融合改造计划~~
|
||||
|
||||
### 今日计划拆分
|
||||
|
||||
1. ~~分析 demo 项目能力与设计重点~~
|
||||
2. ~~评估 Jarvis 当前 agents 的优势与短板~~
|
||||
3. ~~输出 Jarvis 2.0 总体升级思路~~
|
||||
4. ~~建立 `development-doc/plan` 与 `development-doc/daily`~~
|
||||
5. ~~将 plan 拆成 phase 文档~~
|
||||
6. ~~输出 2 天融合改造计划~~
|
||||
|
||||
### Day 1 工作内容
|
||||
|
||||
#### Day 1 目标
|
||||
|
||||
- ~~完成 demo 项目分析~~
|
||||
- ~~完成 Jarvis 当前能力差距判断~~
|
||||
- ~~完成 `development-doc/` 目录搭建~~
|
||||
- ~~完成 phase 文档拆分~~
|
||||
- ~~完成 2 天融合计划初稿~~
|
||||
|
||||
#### Day 1 分步骤执行
|
||||
|
||||
1. ~~分析 `demo/swarm-ide-chore-specs-mvp`~~
|
||||
2. ~~分析 `demo/claude-code-cli-master`~~
|
||||
3. ~~分析 `demo/claw-code-main`~~
|
||||
4. ~~梳理 Jarvis 当前 agents 架构优势与短板~~
|
||||
5. ~~输出 `plan/README.md` 与 phase 文档~~
|
||||
6. ~~输出 `2026-04-03-jarvis-agents-2-day-integration-plan.md`~~
|
||||
7. ~~建立并整理 `daily/` 日志结构~~
|
||||
|
||||
#### Day 1 完成标准
|
||||
|
||||
- ~~文档结构齐全~~
|
||||
- ~~分阶段计划可读~~
|
||||
- ~~两天改造路线明确~~
|
||||
- ~~daily 可持续维护~~
|
||||
|
||||
> Day 2 内容已拆分到 `daily/2026-04-04.md`
|
||||
|
||||
---
|
||||
|
||||
## 今日实际完成
|
||||
|
||||
1. 分析了以下 demo 项目：
|
||||
- `demo/swarm-ide-chore-specs-mvp`
|
||||
- `demo/claude-code-cli-master`
|
||||
- `demo/claw-code-main`
|
||||
|
||||
2. 梳理了 Jarvis 当前 agent 架构的优势与短板：
|
||||
- 当前强项：分层路由、业务导向、continuity、fallback、测试基础
|
||||
- 当前短板：动态协作不足、缺少 verifier、缺少 task/runtime、可观察性不足
|
||||
|
||||
3. 输出并整理了 Jarvis 规划文档结构：
|
||||
- `development-doc/README.md`
|
||||
- `development-doc/plan/README.md`
|
||||
- `development-doc/plan/phase-0-current-state-and-target.md`
|
||||
- `development-doc/plan/phase-1-safe-foundation.md`
|
||||
- `development-doc/plan/phase-2-controlled-collaboration.md`
|
||||
- `development-doc/plan/phase-3-dynamic-collaboration.md`
|
||||
- `development-doc/plan/phase-4-visibility-and-isolation.md`
|
||||
- `development-doc/plan/2026-04-03-jarvis-agents-2-day-integration-plan.md`
|
||||
|
||||
4. 建立并整理了 daily 目录：
|
||||
- `development-doc/daily/2026-04-03.md`
|
||||
|
||||
---
|
||||
|
||||
## 当前进度
|
||||
|
||||
### 文档规划进度
|
||||
|
||||
- demo 分析：已完成
|
||||
- 总体升级方向：已完成
|
||||
- plan 目录搭建：已完成
|
||||
- 分阶段 plan：已完成
|
||||
- 2 天融合计划：已完成
|
||||
- daily 规范：已完成
|
||||
- Day 3 清单与验收文档：已更新
|
||||
- Day 4 清单：已更新
|
||||
|
||||
### 代码改造进度
|
||||
|
||||
- 已完成 Day 1 / Day 2 底座与协作闭环
|
||||
- 已完成 Day 3 最小受限动态协作 runtime
|
||||
- 已补齐 registry spawn policy、graph spawn guardrail、message trace、interrupt / recovery 最小闭环
|
||||
- Phase 1-3 核心功能已基本落地
|
||||
- Day 4 聚焦 Phase 4 可视化与隔离执行能力
|
||||
|
||||
---
|
||||
|
||||
## 今日修改的模块 / 文件
|
||||
|
||||
### 新增 / 更新的开发文档
|
||||
|
||||
- `development-doc/README.md`
|
||||
- `development-doc/plan/README.md`
|
||||
- `development-doc/plan/phase-0-current-state-and-target.md`
|
||||
- `development-doc/plan/phase-1-safe-foundation.md`
|
||||
- `development-doc/plan/phase-2-controlled-collaboration.md`
|
||||
- `development-doc/plan/phase-3-dynamic-collaboration.md`
|
||||
- `development-doc/plan/phase-4-visibility-and-isolation.md`
|
||||
- `development-doc/plan/2026-04-03-jarvis-agents-2-day-integration-plan.md`
|
||||
- `development-doc/daily/2026-04-03.md`
|
||||
|
||||
### 本轮补充分析与改造涉及的代码文件
|
||||
|
||||
- `backend/app/agents/state.py`
|
||||
- `backend/app/agents/graph.py`
|
||||
- `backend/app/agents/prompts.py`
|
||||
- `backend/app/agents/registry/models.py`
|
||||
- `backend/app/agents/registry/builtins.py`
|
||||
- `backend/app/agents/registry/indexes.py`
|
||||
- `backend/app/agents/tools/__init__.py`
|
||||
- `backend/tests/backend/app/agents/test_graph.py`
|
||||
- `backend/tests/backend/app/agents/test_registry.py`
|
||||
|
||||
---
|
||||
|
||||
## 今日结论
|
||||
|
||||
### 从 demo 项目吸收的核心方向
|
||||
|
||||
- 从 Swarm-IDE 学：动态通信原语、可观察性、协作拓扑
|
||||
- 从 Claude Code CLI 学：coordinator / task / verifier 的平台化编排
|
||||
- 从 Claw Code 学：runtime 分层、工具注册表、权限模型
|
||||
|
||||
### Jarvis 总体升级方向
|
||||
|
||||
Jarvis 不应直接变成完全自由的 swarm，而应升级为：
|
||||
|
||||
- 受控的动态协作运行时
|
||||
|
||||
原则：
|
||||
|
||||
- 简单请求继续走当前稳定路径
|
||||
- 复杂请求才进入协作模式
|
||||
|
||||
---
|
||||
|
||||
## 当前阻塞点
|
||||
|
||||
- 暂无明显代码层阻塞
|
||||
- 当前主要待办是完成定向回归测试并收尾验收
|
||||
|
||||
---
|
||||
|
||||
## 风险与临时决策
|
||||
|
||||
### 当前风险
|
||||
|
||||
- 不要一开始就引入无限动态 agent
|
||||
- 不要直接替换现有 graph 主路径
|
||||
- 应优先保持 reminder/task/search 等现有业务稳定
|
||||
- 新能力必须配套测试和约束策略
|
||||
|
||||
### 当前决策
|
||||
|
||||
1. 采用受限动态协作，而不是自由 swarm
|
||||
2. 通过 registry 固化 spawn role policy，再由 graph 在运行时执行权限校验
|
||||
3. interrupt / recovery 先落最小闭环，优先保证 direct 主路径稳定
|
||||
4. daily 后续必须作为开发过程中的持续更新日志使用
|
||||
|
||||
---
|
||||
|
||||
## 验证 / 测试情况
|
||||
|
||||
- 已补充 Day 3 相关 runtime / registry / graph 回归测试
|
||||
- 已更新 Day 3 执行清单与 daily 状态
|
||||
- 正在执行定向 pytest 验证，重点覆盖 `test_graph.py` 与 `test_registry.py`
|
||||
|
||||
---
|
||||
|
||||
## 下一步计划
|
||||
|
||||
1. 完成 Day 3 定向回归测试
|
||||
2. 若有失败，修正 runtime / test 偏差
|
||||
3. 统一整理 Day 3 最终验收结论
|
||||
4. 启动 Day 4：Phase 4 可视化 API 实现
|
||||
5. 设计隔离执行最小方案
|
||||
|
||||
---
|
||||
|
||||
## 每日维护要求
|
||||
|
||||
后续正式进入改造阶段后，本文件需要持续更新：
|
||||
|
||||
1. 开始开发前更新“今日开发计划”
|
||||
2. 完成一个阶段性步骤后更新“当前进度”
|
||||
3. 变更方案时更新“风险与临时决策”
|
||||
4. 出现问题时更新“当前阻塞点”
|
||||
5. 每次完成验证后更新“验证 / 测试情况”
|
||||
6. 一天结束前补齐“已完成 / 未完成 / 下一步计划”
|
||||
116
development-doc/daily/2026-04-04.md
Normal file
116
development-doc/daily/2026-04-04.md
Normal file
@@ -0,0 +1,116 @@
|
||||
# 2026-04-04 工作日志
|
||||
|
||||
## 今日开发计划
|
||||
|
||||
### 今日目标
|
||||
|
||||
- 巩固 `Phase 4` 已完成的可见性最小闭环
|
||||
- 把 runtime summary 接到 Agents 页面
|
||||
- 为后续 90 分路径明确 isolation / cost / operator surface 升级项
|
||||
- 保持 reminder / task / search 主路径稳定
|
||||
|
||||
### 今日计划拆分
|
||||
|
||||
1. 新增 `backend/app/agents/api/visibility.py` 可见性 API 模块
|
||||
2. 实现 event stream API
|
||||
3. 实现协作链路拓扑查询 API
|
||||
4. 实现 task 执行证据查询 API
|
||||
5. 实现 message thread 查询 API
|
||||
6. 实现 verifier 结果查询 API
|
||||
7. 设计隔离执行最小方案
|
||||
8. 补测试并验证主流程
|
||||
|
||||
### Day 4 工作内容
|
||||
|
||||
#### Day 4 目标
|
||||
|
||||
- 完成 `Phase 4` 可见性 API 最小闭环
|
||||
- 完成 runtime summary API 与前端 Agents 页面首屏接入
|
||||
- 为后续完整隔离执行与成本治理预留接口
|
||||
- 保证已有路径测试不回退
|
||||
|
||||
#### Day 4 分步骤执行
|
||||
|
||||
1. 新增 `backend/app/agents/api/visibility.py` 及各可见性 API
|
||||
2. `GET /agents/visibility/events` - event stream 按条件过滤
|
||||
3. `GET /agents/visibility/topology` - 协作拓扑视图
|
||||
4. `GET /agents/visibility/tasks/{task_id}/evidence` - task 执行证据
|
||||
5. `GET /agents/visibility/threads/{thread_id}/messages` - thread 消息流
|
||||
6. `GET /agents/visibility/verifier` - verifier 验收结论
|
||||
7. 在 `development-doc/plan/phase-4-visibility-and-isolation.md` 补充隔离执行设计方案
|
||||
8. 补 `test_visibility_api.py` 及主流程回归测试
|
||||
|
||||
#### Day 4 完成标准
|
||||
|
||||
- event stream API 可按 conversation_id / thread_id / agent_id 过滤
|
||||
- topology API 可返回协作拓扑视图
|
||||
- evidence API 可返回 task 执行证据链
|
||||
- thread API 可重建消息流向
|
||||
- verifier API 可返回验收结论
|
||||
- 隔离执行设计方案可落地
|
||||
- 现有主流程测试继续通过
|
||||
|
||||
---
|
||||
|
||||
## 今日实际完成
|
||||
|
||||
- 分析了 Jarvis 现有代码实现状态（`graph.py`、`state.py`、`verifier.py`、`registry/models.py`、`schemas/`）
|
||||
- 确认 Phase 1-3 核心功能已基本落地：task schema、event schema、verifier、tool metadata、collaboration flow、interrupt/recovery、message trace
|
||||
- 识别了 Phase 4（可视化与隔离执行）待实现内容
|
||||
- 在 `2026-04-03-jarvis-agents-5-day-work-checklist.md` 中新增了 Day 4 任务清单
|
||||
|
||||
---
|
||||
|
||||
## 当前进度
|
||||
|
||||
### 代码改造进度（Phase 1-3）
|
||||
|
||||
- ✅ task schema / event schema 已完整
|
||||
- ✅ verifier 模块已独立
|
||||
- ✅ state.py 已包含 collaboration 全部字段
|
||||
- ✅ registry/models.py 已补充 tool metadata
|
||||
- ✅ graph.py 已接入 event trace、verifier 调用、collaboration flow
|
||||
- ✅ interrupt / recovery 最小闭环已实现
|
||||
- ✅ message trace 已实现
|
||||
|
||||
### Day 4 待启动
|
||||
|
||||
- 待实现可见性 API（event stream、topology、evidence、thread、verifier）
|
||||
- 待设计隔离执行方案
|
||||
- 待补可视化 API 测试
|
||||
|
||||
---
|
||||
|
||||
## 今日修改的模块 / 文件
|
||||
|
||||
- 待更新
|
||||
|
||||
---
|
||||
|
||||
## 当前阻塞点
|
||||
|
||||
- 待开发时更新
|
||||
|
||||
---
|
||||
|
||||
## 风险与临时决策
|
||||
|
||||
- 不直接重写 graph 主路径
|
||||
- verifier 优先以 helper 形式接入
|
||||
- 先补底座，不直接做自由 swarm
|
||||
|
||||
---
|
||||
|
||||
## 验证 / 测试情况
|
||||
|
||||
- 待更新
|
||||
|
||||
---
|
||||
|
||||
## 下一步计划
|
||||
|
||||
1. 实现 `visibility.py` 可见性 API 模块
|
||||
2. 按顺序实现 event stream、topology、evidence、thread、verifier API
|
||||
3. 设计隔离执行最小方案
|
||||
4. 补 `test_visibility_api.py` 测试
|
||||
5. 跑测试验证主流程不回退
|
||||
@@ -1,102 +0,0 @@
|
||||
# Jarvis Agents 2 天工作计划（可勾选执行版）
|
||||
|
||||
日期：2026-04-03
|
||||
状态：执行清单
|
||||
适用范围：基于 `phase-0` ~ `phase-4` 及现有 2 天融合方案整理
|
||||
|
||||
---
|
||||
|
||||
## 使用说明
|
||||
|
||||
- 完成前使用 `- [ ]`
|
||||
- 完成后改成 `- [x]`
|
||||
- Day 2 默认依赖 Day 1 的核心底座完成后再推进
|
||||
|
||||
---
|
||||
|
||||
## Day 1：补底座，完成 Phase 1 最小闭环
|
||||
|
||||
Day 1 目标：先把 Jarvis 从“只有静态路由”补成“有任务结构、有事件结构、有 verifier、有工具治理信息”的可扩展底座，同时不破坏当前 direct 主路径。
|
||||
|
||||
- [x] 新增最小 `task schema`
|
||||
改造内容：新增 `backend/app/agents/schemas/task.py`，统一 `task_id`、`title`、`status`、`owner_agent_id`、`evidence`、`result_summary`，并补 `role`、`goal`、`expected_evidence`、`created_at`、`updated_at`；状态固定为 `pending`、`in_progress`、`completed`、`failed`、`blocked`。
|
||||
|
||||
- [x] 新增最小 `event schema`
|
||||
改造内容：新增 `backend/app/agents/schemas/event.py`，统一 `event_id`、`event_type`、`timestamp`、`conversation_id`、`agent_id`、`sub_commander_id`、`task_id`、`payload`、`severity`；首批事件类型覆盖 `agent.tool.start`、`agent.tool.result`、`agent.verify.started`、`agent.verify.completed`、`agent.error`。
|
||||
|
||||
- [x] 扩展 `backend/app/agents/state.py` 的运行时字段
|
||||
改造内容：新增 `execution_mode`、`verification_status`、`verification_summary`、`verification_evidence`、`active_tasks`、`task_results`、`event_trace`、`budget_state`；默认值保持兼容 `initial_state()`，不替换现有 `pending_tasks`、`completed_tasks`、`tool_calls`。
|
||||
|
||||
- [x] 扩展 capability / tool metadata 模型
|
||||
改造内容：在 `backend/app/agents/registry/models.py` 增加 `permission_class`、`side_effect_scope`、`supports_retry`、`idempotent`、`safe_for_parallel_use`、`requires_confirmation`；至少先固化 `read` / `write` / `external` 和 `none` / `local_state` / `db_write` / `network` 两组枚举语义。
|
||||
|
||||
- [x] 回填 builtin tools 的静态 metadata
|
||||
改造内容：在 `backend/app/agents/registry/builtins.py` 和需要的 `backend/app/agents/tools/__init__.py` 中，把 search / retrieval 类工具标成偏 `read`，create / update 类工具标成偏 `write`，外部检索类工具标成 `external`，并补充是否可重试、是否幂等、是否适合并行等标记。
|
||||
|
||||
- [x] 新增 verifier 角色定义
|
||||
改造内容：在 `backend/app/agents/prompts.py` 增加 verifier prompt，明确 verifier 只负责验收，不负责重新规划；验收点聚焦“是否真正满足请求”“是否有明确证据”“是否把失败伪装成成功”。
|
||||
|
||||
- [x] 落地 verifier 模块
|
||||
改造内容：新增 `backend/app/agents/verifier.py`，支持 `passed`、`failed`、`skipped` 三类最小结论，先服务于工具调用后的复杂输出、知识检索结果和分析型汇总输出，不接管纯闲聊路径。
|
||||
|
||||
- [x] 在 `backend/app/agents/graph.py` 接入最小 event trace 与 verifier helper
|
||||
改造内容：给 `_execute_tool_calls()` 增加 tool start / result / error 事件写入；给收尾阶段增加 verifier helper 调用；给 `_run_sub_commander()` 增加 task result 摘要写入，但暂时不重构主图为完整协作编排图。
|
||||
|
||||
- [x] 补 Phase 1 单元测试与回归测试
|
||||
改造内容：新增 `backend/tests/backend/app/agents/test_agent_schemas.py`、`backend/tests/backend/app/agents/test_verifier.py`，并扩展 `test_graph.py`，覆盖 state 兼容性、schema 合法性、tool metadata 存在性、verifier 判定、主流程不回退。
|
||||
|
||||
- [x] 完成 Day 1 验收
|
||||
改造内容：确认 reminder / task / search 主流程继续通过；确认 verifier 已能独立运行；确认 event schema 与 task schema 已落代码；确认 direct 仍是默认主路径；确认未引入动态 `create_agent`、message bus 全链路和 UI。
|
||||
|
||||
---
|
||||
|
||||
## Day 2：引入最小协作能力，完成 Phase 2 雏形
|
||||
|
||||
Day 2 目标：在 Day 1 底座稳定的基础上，给 Jarvis 增加“复杂请求可拆分、可分配、可回收、可验收”的最小受控协作能力，但仍然不进入自由 swarm。
|
||||
|
||||
- [ ] 增加 `request_mode_selector`
|
||||
改造内容：在 `backend/app/agents/graph.py` 中增加 direct / collaboration 模式选择逻辑；简单请求继续走旧路径，只有明显多步骤、跨领域、需要多角色配合的请求才进入 collaboration mode。
|
||||
|
||||
- [ ] 新增 coordinator prompt
|
||||
改造内容：在 `backend/app/agents/prompts.py` 中定义 coordinator 角色，职责限定为“判断是否拆解”“输出 2~4 个清晰子任务”“分配角色建议”“汇总任务结果”；明确禁止无限递归拆分。
|
||||
|
||||
- [ ] 新增最小 task decomposition 结构
|
||||
改造内容：基于 Day 1 的 task schema 扩展最小拆分结构，至少输出 `task_id`、`title`、`role`、`goal`、`expected_evidence`，让复杂请求能以结构化任务列表进入后续执行。
|
||||
|
||||
- [ ] 增加 role -> existing agent assignment
|
||||
改造内容：先复用当前已有 top-level agent，不新增独立 worker runtime；把 schedule 类任务映射给 `schedule_planner`，retrieval 类任务映射给 `librarian`，analysis 类任务映射给 `analyst`，execution 类任务映射给 `executor`。
|
||||
|
||||
- [ ] 建立统一 task result 回收结构
|
||||
改造内容：约束每个角色统一返回 `task_id`、`status`、`summary`、`evidence`、`next_action`（可选），并把结果写回 `task_results`，避免最终结果继续依赖单点硬编码拼接。
|
||||
|
||||
- [ ] 让 verifier 强制参与协作结果收尾
|
||||
改造内容：在 collaboration mode 下，所有复杂请求返回前都必须经过 verifier；verifier 有权拒绝证据不足、结果不完整、子任务未闭环的响应。
|
||||
|
||||
- [ ] 补 Phase 2 协作测试与回归测试
|
||||
改造内容：覆盖复杂请求拆分测试、角色分配测试、task result 汇总测试、verifier 拒绝不完整结果测试，并再次确认 direct 模式原有流程不回退。
|
||||
|
||||
- [ ] 完成 Day 2 验收
|
||||
改造内容：确认 graph 已能区分 direct / collaboration；确认复杂请求可拆成 2~4 个子任务；确认每个子任务有 owner 和 evidence；确认最终答案基于 task result 汇总；确认系统仍未进入无限动态 agent 模式。
|
||||
|
||||
---
|
||||
|
||||
## 这 2 天明确不做
|
||||
|
||||
- 不做动态 `create_agent`
|
||||
- 不做 parent / child agent tree
|
||||
- 不做内部消息线程长期态管理
|
||||
- 不做可视化调试面板
|
||||
- 不做 event stream API
|
||||
- 不做 worktree / 隔离执行
|
||||
- 不做自由蜂群式协作
|
||||
|
||||
---
|
||||
|
||||
## 2 天结束后的预期状态
|
||||
|
||||
- [ ] 已具备 `direct` / `collaboration` 双模式入口
|
||||
- [ ] 已具备 verifier 独立验收层
|
||||
- [ ] 已具备 task schema / event schema / tool metadata 底座
|
||||
- [ ] 已具备 coordinator 雏形、任务拆分、角色分配、结果回收
|
||||
- [ ] 当前 reminder / task / search 主路径无明显回退
|
||||
- [ ] 后续可以继续推进 Phase 3 的受限动态协作，而不是返工 Phase 1 / Phase 2 底座
|
||||
@@ -150,14 +150,14 @@
|
||||
- [x] 创建内存版 PluginMarketplace (in-memory)
|
||||
- [x] 实现 search() — GET `/api/marketplace/plugins`
|
||||
- [x] 实现 get_plugin() — GET `/api/marketplace/plugins/{id}`
|
||||
- [ ] 实现 download_plugin()
|
||||
- [x] 实现 download_plugin() — POST `/api/marketplace/plugins/{id}/download`
|
||||
|
||||
### 8.5 内置插件
|
||||
|
||||
- [ ] 创建 `plugins/builtins/code_helper/` — lint, format, explain_code
|
||||
- [ ] 创建 `plugins/builtins/git_helper/` — git_status, git_log, git_diff
|
||||
- [ ] 创建 `plugins/builtins/web_helper/` — fetch_url, parse_html
|
||||
- [ ] 创建 `plugins/builtins/file_organizer/` — organize_files, cleanup_duplicates
|
||||
- [x] 创建 `plugins/builtins/code_helper/` — lint, format, explain_code
|
||||
- [x] 创建 `plugins/builtins/git_helper/` — git_status, git_log, git_diff
|
||||
- [x] 创建 `plugins/builtins/web_helper/` — fetch_url, parse_html
|
||||
- [x] 创建 `plugins/builtins/file_organizer/` — organize_files, cleanup_duplicates
|
||||
|
||||
### 8.6 API
|
||||
|
||||
@@ -181,7 +181,7 @@
|
||||
- [x] 插件的工具和 Hook 正确注册
|
||||
- [x] 插件的工具和 Hook 正确注销
|
||||
- [x] 插件无法访问未授权资源
|
||||
- [ ] 插件加载时间 < 1s
|
||||
- [x] 插件加载时间 < 1s (built-in plugins)
|
||||
|
||||
---
|
||||
|
||||
@@ -214,16 +214,16 @@
|
||||
|
||||
### 9.4 内置 Skills
|
||||
|
||||
- [ ] 创建 `backend/app/agents/skills/bundled.py` — BUNDLED_SKILLS
|
||||
- [ ] 实现 code-analysis skill
|
||||
- [ ] 实现 git-helper skill
|
||||
- [ ] 实现 web-research skill
|
||||
- [ ] 实现 file-management skill
|
||||
- [ ] 实现 task-planning skill
|
||||
- [x] 创建 `backend/app/agents/skills/bundled.py` — BUNDLED_SKILLS
|
||||
- [x] 实现 code-analysis skill
|
||||
- [x] 实现 git-helper skill
|
||||
- [x] 实现 web-research skill
|
||||
- [x] 实现 file-management skill
|
||||
- [x] 实现 task-planning skill
|
||||
|
||||
### 9.5 Agent 集成
|
||||
|
||||
- [ ] AgentService.build_skill_context()
|
||||
- [x] AgentService.build_skill_context()
|
||||
- [ ] Skill 上下文注入 Agent prompt
|
||||
- [ ] Skill 触发检测
|
||||
|
||||
@@ -248,7 +248,7 @@
|
||||
|
||||
- [x] 能加载 local_skills_dir 下的所有 SKILL.md
|
||||
- [x] 能从 MCP 服务器发现和加载 Skills
|
||||
- [ ] 内置 Skills 默认加载
|
||||
- [x] 内置 Skills 默认加载
|
||||
- [ ] Skill 内容正确注入 Agent prompt
|
||||
|
||||
---
|
||||
@@ -271,7 +271,7 @@
|
||||
|
||||
### 10.2 远程传输层
|
||||
|
||||
- [ ] 创建 `backend/app/agents/transport/structured_io.py` — StructuredIO
|
||||
- [x] 创建 `backend/app/agents/transport/structured_io.py` — StructuredIO
|
||||
- [x] 创建 `backend/app/agents/transport/remote.py` — RemoteTransport
|
||||
- [x] 实现 send_response()
|
||||
- [x] 实现 send_event()
|
||||
@@ -292,8 +292,8 @@
|
||||
### 10.4 后台任务系统
|
||||
|
||||
- [x] 创建 `backend/app/agents/background/manager.py` — BackgroundTaskManager
|
||||
- [ ] 创建 `backend/app/agents/background/scheduler.py`
|
||||
- [ ] 创建 `backend/app/agents/background/executor.py`
|
||||
- [x] 创建 `backend/app/agents/background/scheduler.py`
|
||||
- [x] 创建 `backend/app/agents/background/executor.py`
|
||||
- [x] 实现 submit_task()
|
||||
- [x] 实现 cancel_task()
|
||||
- [x] 实现 get_task_status()
|
||||
@@ -301,7 +301,7 @@
|
||||
|
||||
### 10.5 协调整合
|
||||
|
||||
- [ ] 创建/修改 `backend/app/agents/coordinator.py`
|
||||
- [x] 创建/修改 `backend/app/agents/coordinator.py`
|
||||
- [ ] Team 协作与现有 graph 集成
|
||||
- [ ] 远程传输与现有 service 集成
|
||||
|
||||
@@ -327,7 +327,7 @@
|
||||
- [x] 可以创建和管理 Agent 团队
|
||||
- [x] 任务能正确分配给合适的成员
|
||||
- [x] 能收集和聚合多成员的结果
|
||||
- [ ] 支持结构化的输入输出格式
|
||||
- [x] 支持结构化的输入输出格式
|
||||
- [x] 支持远程 Agent 通信
|
||||
- [x] 支持复杂的会话层级和状态管理
|
||||
- [x] 支持定时和异步后台任务
|
||||
|
||||
171
development-doc/plan/code-update/README.md
Normal file
171
development-doc/plan/code-update/README.md
Normal file
@@ -0,0 +1,171 @@
|
||||
# 代码指挥官 (Code Commander) 实施计划索引
|
||||
|
||||
本目录用于存放代码指挥官模块的分阶段规划文档。
|
||||
|
||||
## 文档说明
|
||||
|
||||
| 文件 | 说明 |
|
||||
|------|------|
|
||||
| `README.md` | 总览、阶段关系、实施顺序 |
|
||||
| `phase-1-infrastructure.md` | 基础设施：State、Prompt、注册 |
|
||||
| `phase-2-execution-engine.md` | 执行引擎：AI Adapter、沙盒、直接执行 |
|
||||
| `phase-3-agent-integration.md` | Agent 集成：Graph 节点、边路由 |
|
||||
| `phase-4-streaming-interaction.md` | 流式交互：PTY 终端、WebSocket |
|
||||
| `phase-5-frontend-integration.md` | 前端集成：Vue 组件、xterm.js |
|
||||
|
||||
## 推荐阅读顺序
|
||||
|
||||
1. 先阅读本 README 了解整体架构
|
||||
2. 再按顺序阅读 phase 1 ~ phase 5
|
||||
3. 实施时严格按阶段推进
|
||||
|
||||
---
|
||||
|
||||
## 总体设计原则
|
||||
|
||||
1. **用户选择式交互** - 不是自动分流，用户显式选择 AI 提供商
|
||||
2. **安全分级执行** - 低风险直接执行，高风险沙盒隔离
|
||||
3. **流式终端体验** - 实时显示 AI 执行过程，支持用户交互
|
||||
4. **临时目录隔离** - 每个任务在独立临时目录执行，执行后清理
|
||||
|
||||
---
|
||||
|
||||
## 阶段总览图
|
||||
|
||||
```
|
||||
Phase 1 ──────────────────────────────────────────────────────────────┐
|
||||
│ 基础设施 (Infrastructure) │
|
||||
│ - State 定义 │
|
||||
│ - Prompt 模板 │
|
||||
│ - 工具注册 │
|
||||
│ - Agent 注册 │
|
||||
│ │
|
||||
│ 核心文件: state.py, prompts.py, tools/__init__.py, builtins.py │
|
||||
└────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
Phase 2 ──────────────────────────────────────────────────────────────┐
|
||||
│ 执行引擎 (Execution Engine) │
|
||||
│ - AI CLI Adapter (统一接口) │
|
||||
│ - Sandbox Executor │
|
||||
│ - Direct Executor │
|
||||
│ - Security Classifier │
|
||||
│ │
|
||||
│ 核心文件: ai_adapter.py, sandbox_executor.py, direct_executor.py, │
|
||||
│ security_classifier.py │
|
||||
└────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
Phase 3 ──────────────────────────────────────────────────────────────┐
|
||||
│ Agent 集成 (Agent Integration) │
|
||||
│ - Graph 节点 │
|
||||
│ - 边路由 │
|
||||
│ - 任务模型 │
|
||||
│ │
|
||||
│ 核心文件: graph.py, schemas/task.py │
|
||||
└────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
Phase 4 ──────────────────────────────────────────────────────────────┐
|
||||
│ 流式交互 (Streaming Interaction) │
|
||||
│ - PTY 终端 │
|
||||
│ - WebSocket 端点 │
|
||||
│ - 流式输出集成 │
|
||||
│ - 交互输入 │
|
||||
│ │
|
||||
│ 核心文件: terminal_engine.py, routers/terminal.py, stream_output.py │
|
||||
└────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
Phase 5 ──────────────────────────────────────────────────────────────┐
|
||||
│ 前端集成 (Frontend Integration) │
|
||||
│ - 页面组件 │
|
||||
│ - 终端显示组件 │
|
||||
│ - WebSocket 服务 │
|
||||
│ - 路由配置 │
|
||||
│ │
|
||||
│ 核心文件: CodeCommander.vue, TerminalDisplay.vue, terminalWs.ts │
|
||||
└────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 架构概览
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Vue 前端 │
|
||||
│ [用户选择: Claude/Gemini/Codex/OpenCode] + [输入需求] │
|
||||
└────────────────────────┬────────────────────────────────────┘
|
||||
│ WebSocket 流式输出
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ FastAPI 后端 │
|
||||
│ ┌─────────────────────────────────────────────────────┐ │
|
||||
│ │ 代码指挥官 (Code Commander Agent) │ │
|
||||
│ │ 1. 接收 AI 类型 + 用户需求 │ │
|
||||
│ │ 2. 安全分级判定 │ │
|
||||
│ │ 3. 路由到对应执行器 │ │
|
||||
│ └─────────────────────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌──────────────────┼──────────────────┐ │
|
||||
│ ▼ ▼ ▼ │
|
||||
│ ┌────────────┐ ┌────────────┐ ┌────────────┐ │
|
||||
│ │ 直接执行器 │ │ 沙盒执行器 │ │ 终端引擎 │ │
|
||||
│ │(低风险任务) │ │(高风险任务) │ │ PTY + 流式 │ │
|
||||
│ └────────────┘ └────────────┘ └────────────┘ │
|
||||
└────────────────────────┬────────────────────────────────────┘
|
||||
│ subprocess 调用
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ CLI 进程 (claude/gemini/codex/opencode) │
|
||||
│ 在临时目录中执行 │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Demo 项目借鉴映射
|
||||
|
||||
| Demo 项目 | 主要借鉴点 | 对应 Phase |
|
||||
|---------|-----------|-----------|
|
||||
| **golutra** | PTY 终端、多 CLI 适配、工作流隔离 | Phase 2, 4 |
|
||||
| **golutra CLI** | LocalSocket IPC、命令分发 | Phase 2 |
|
||||
| **golutra Shim** | 进程启动、信号处理 | Phase 2 |
|
||||
|
||||
---
|
||||
|
||||
## 实施顺序
|
||||
|
||||
```
|
||||
Phase 1 → Phase 2 → Phase 3 → Phase 4 → Phase 5
|
||||
│ │ │ │ │
|
||||
│ │ │ │ └── 前端 UI + 路由
|
||||
│ │ │ └── PTY + WebSocket
|
||||
│ │ └── Graph 节点 + 边路由
|
||||
│ └── AI Adapter + Sandbox
|
||||
└── State + Prompt + 注册
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 文件变更追踪
|
||||
|
||||
| Phase | 新增文件 | 修改文件 |
|
||||
|-------|---------|---------|
|
||||
| Phase 1 | `tools/__init__.py` (改) | `state.py`, `prompts.py`, `registry/builtins.py` |
|
||||
| Phase 2 | `ai_adapter.py`, `sandbox_executor.py`, `direct_executor.py`, `security_classifier.py` | - |
|
||||
| Phase 3 | `schemas/task.py` (改) | `graph.py` |
|
||||
| Phase 4 | `terminal_engine.py`, `routers/terminal.py`, `stream_output.py`, `interactive_input.py` | - |
|
||||
| Phase 5 | `CodeCommander.vue`, `TerminalDisplay.vue`, `terminalWs.ts` | `router/index.ts` |
|
||||
|
||||
---
|
||||
|
||||
## 注意事项
|
||||
|
||||
| 注意事项 | 说明 |
|
||||
|---------|------|
|
||||
| 不要跳过 Phase | 每个阶段都是下一个的基础 |
|
||||
| AI CLI 前置检查 | 确保服务器上已安装对应 CLI |
|
||||
| 临时目录及时清理 | 防止磁盘空间泄漏 |
|
||||
| WebSocket 重连 | 前端实现自动重连机制 |
|
||||
215
development-doc/plan/code-update/checklist.md
Normal file
215
development-doc/plan/code-update/checklist.md
Normal file
@@ -0,0 +1,215 @@
|
||||
# 代码指挥官实施清单（可勾选执行版）
|
||||
|
||||
日期：2026-04-04
|
||||
状态：执行清单
|
||||
适用范围：基于 `phase-1` ~ `phase-5` 整理
|
||||
|
||||
---
|
||||
|
||||
## 使用说明
|
||||
|
||||
- 完成前使用 `- [ ]`
|
||||
- 完成后改成 `- [x]`
|
||||
- Day 1-3 为后端基础设施
|
||||
- Day 4-5 为后端执行引擎
|
||||
- Day 6 为 Agent 集成
|
||||
- Day 7-8 为流式交互
|
||||
- Day 9-10 为前端集成
|
||||
|
||||
---
|
||||
|
||||
## Day 1：State + Prompt + 注册
|
||||
|
||||
Day 1 目标：完成代码指挥官 Agent 的基础架子
|
||||
|
||||
- [ ] 新增 `CODE_COMMANDER = "code_commander"` 到 `AgentRole` 枚举
|
||||
- [ ] 新增 `CodeCommanderState` TypedDict（包含 task_type, ai_provider, sandbox_mode 等）
|
||||
- [ ] 新增 `CODE_COMMANDER_SYSTEM_PROMPT` 系统提示
|
||||
- [ ] 新增 `SANDBOX_EXECUTION_PROMPT` 沙盒执行说明
|
||||
- [ ] 新增 `DIRECT_EXECUTION_PROMPT` 直接执行说明
|
||||
- [ ] 在 `SUB_COMMANDER_TOOLSETS` 中注册 `CODE_COMMANDER_TOOLSET`
|
||||
- [ ] 新增 `CodeCommanderManifest` 到 `AGENT_MANIFESTS`
|
||||
- [ ] 补 Phase 1 单元测试
|
||||
|
||||
**验收：确认 `AgentRole.CODE_COMMANDER` 存在且值正确**
|
||||
|
||||
---
|
||||
|
||||
## Day 2：AI CLI Adapter（统一接口）
|
||||
|
||||
Day 2 目标：实现适配不同 AI CLI 的统一接口
|
||||
|
||||
- [ ] 新增 `AICLIAdapter` 抽象基类
|
||||
- `cli_name` 属性
|
||||
- `requires_workspace` 属性
|
||||
- `build_command()` 方法
|
||||
- `parse_output()` 方法
|
||||
- `is_installed()` 方法
|
||||
- [ ] 新增 `ClaudeAdapter` 实现
|
||||
- [ ] 新增 `GeminiAdapter` 实现
|
||||
- [ ] 新增 `CodexAdapter` 实现
|
||||
- [ ] 新增 `OpenCodeAdapter` 实现
|
||||
- [ ] 新增 `CodeExecutionResult` 数据类
|
||||
- [ ] 补 Day 2 单元测试
|
||||
|
||||
**验收：`AICLIAdapter` 可以正确识别 4 种 CLI**
|
||||
|
||||
---
|
||||
|
||||
## Day 3：Security Classifier + Direct Executor
|
||||
|
||||
Day 3 目标：实现安全分级和直接执行器
|
||||
|
||||
- [ ] 新增 `RiskLevel` 枚举（LOW/HIGH）
|
||||
- [ ] 新增 `SecurityClassifier` 类
|
||||
- `HIGH_RISK_KEYWORDS` 列表
|
||||
- `LOW_RISK_KEYWORDS` 列表
|
||||
- `classify()` 方法实现
|
||||
- `_is_project_path()` 方法实现
|
||||
- [ ] 新增 `DirectExecutor` 类
|
||||
- `execute()` 方法（异步）
|
||||
- 超时控制
|
||||
- `is_installed()` 检查
|
||||
- [ ] 补 Day 3 单元测试
|
||||
|
||||
**验收：`SecurityClassifier` 能正确分类高低风险**
|
||||
|
||||
---
|
||||
|
||||
## Day 4：Sandbox Environment + Sandbox Executor
|
||||
|
||||
Day 4 目标：实现沙盒执行器
|
||||
|
||||
- [ ] 新增 `SandboxEnvironment` 类
|
||||
- `create()` 静态方法（创建临时目录）
|
||||
- `cleanup()` 方法
|
||||
- `workspace_path` 属性
|
||||
- `session_id` 属性
|
||||
- [ ] 新增 `SandboxExecutor` 类
|
||||
- `execute()` 方法（异步，yield 流式输出）
|
||||
- `cleanup_session()` 方法
|
||||
- `_list_created_files()` 方法
|
||||
- [ ] 实现超时控制
|
||||
- [ ] 补 Day 4 单元测试
|
||||
|
||||
**验收：`SandboxExecutor` 能创建、执行、清理沙盒**
|
||||
|
||||
---
|
||||
|
||||
## Day 5：执行引擎集成测试
|
||||
|
||||
Day 5 目标：确保执行引擎各组件协同工作
|
||||
|
||||
- [ ] 集成测试：`SecurityClassifier` + `DirectExecutor`
|
||||
- [ ] 集成测试：`SecurityClassifier` + `SandboxExecutor`
|
||||
- [ ] 集成测试：4 种 `AICLIAdapter` 的 `build_command()`
|
||||
- [ ] 端到端测试：低风险任务直接执行
|
||||
- [ ] 端到端测试：高风险任务沙盒执行
|
||||
- [ ] 确认沙盒目录创建和清理正常
|
||||
|
||||
**验收：所有执行器支持流式输出，且正确路由**
|
||||
|
||||
---
|
||||
|
||||
## Day 6：Graph 节点 + 边路由
|
||||
|
||||
Day 6 目标：将代码指挥官接入 LangGraph
|
||||
|
||||
- [ ] 新增 `code_commander_node` 函数
|
||||
- 获取用户需求和 AI 提供商
|
||||
- 调用 `SecurityClassifier`
|
||||
- 根据风险等级选择执行器
|
||||
- 返回执行结果
|
||||
- [ ] 在 `NODES` 字典中注册 `code_commander`
|
||||
- [ ] 新增 `_should_route_to_code_commander()` 路由函数
|
||||
- [ ] 在 `graph.py` 中添加条件边
|
||||
- [ ] 新增 `CodeTask`, `CodeExecutionResult` 模型到 `schemas/task.py`
|
||||
- [ ] 补 Day 6 单元测试
|
||||
|
||||
**验收：高风险任务路由到沙盒，低风险路由到直接执行**
|
||||
|
||||
---
|
||||
|
||||
## Day 7：PTY Terminal Engine
|
||||
|
||||
Day 7 目标：实现 PTY 终端管理
|
||||
|
||||
- [ ] 新增 `PTYSession` 数据类
|
||||
- [ ] 新增 `PTYManager` 类
|
||||
- `spawn()` 方法
|
||||
- `write()` 方法
|
||||
- `read()` 方法（异步生成器）
|
||||
- `resize()` 方法
|
||||
- `kill()` 方法
|
||||
- [ ] 实现 `asyncio.subprocess` 进程管理
|
||||
- [ ] 实现输出队列
|
||||
- [ ] 补 Day 7 单元测试
|
||||
|
||||
**验收：PTY 会话可以启动、读写、终止**
|
||||
|
||||
---
|
||||
|
||||
## Day 8：WebSocket + 流式输出
|
||||
|
||||
Day 8 目标：实现 WebSocket 端点和流式输出
|
||||
|
||||
- [ ] 新增 `ConnectionManager` 类
|
||||
- [ ] 新增 `/ws/terminal/{session_id}` WebSocket 端点
|
||||
- [ ] 实现连接管理（connect/disconnect）
|
||||
- [ ] 新增 `StreamOutput` 类
|
||||
- [ ] 实现 `stream_execution()` 方法
|
||||
- [ ] 新增 `InteractiveInputHandler` 类
|
||||
- [ ] 实现用户输入传递到 PTY
|
||||
- [ ] 补 Day 8 集成测试
|
||||
|
||||
**验收：WebSocket 连接正常，输出实时推送**
|
||||
|
||||
---
|
||||
|
||||
## Day 9：Vue 页面组件
|
||||
|
||||
Day 9 目标：前端代码指挥官主页面
|
||||
|
||||
- [ ] 新增 `CodeCommander.vue` 页面组件
|
||||
- AI 提供商选择器
|
||||
- 任务输入框
|
||||
- 执行按钮
|
||||
- 终端显示区域
|
||||
- 交互输入框
|
||||
- 下载/清理按钮
|
||||
- [ ] 补 Day 9 组件测试
|
||||
|
||||
**验收：用户可以选择 AI 提供商并输入任务**
|
||||
|
||||
---
|
||||
|
||||
## Day 10：TerminalDisplay + WebSocket 服务 + 路由
|
||||
|
||||
Day 10 目标：完成前端集成
|
||||
|
||||
- [ ] 新增 `TerminalDisplay.vue` 组件（xterm.js）
|
||||
- 终端渲染
|
||||
- ANSI 颜色支持
|
||||
- 用户输入处理
|
||||
- [ ] 新增 `terminalWs.ts` WebSocket 服务
|
||||
- 连接管理
|
||||
- 自动重连
|
||||
- 消息处理
|
||||
- [ ] 在 `router/index.ts` 新增 `/code-commander` 路由
|
||||
- [ ] 端到端测试：完整执行流程
|
||||
- [ ] 确认前端与后端 WebSocket 通信正常
|
||||
|
||||
**验收：用户可以在前端看到实时终端输出并交互**
|
||||
|
||||
---
|
||||
|
||||
## 最终验收
|
||||
|
||||
- [ ] 用户可以选择 AI 提供商（Claude/Gemini/Codex/OpenCode）
|
||||
- [ ] 低风险任务（如贪食蛇 demo）直接执行
|
||||
- [ ] 高风险任务在临时目录沙盒执行
|
||||
- [ ] 终端输出实时流式显示
|
||||
- [ ] 用户可以中途输入交互（如 "y" 确认）
|
||||
- [ ] 临时目录执行后正确清理
|
||||
- [ ] 前端页面正常展示
|
||||
- [ ] 回归测试通过（现有功能不受影响）
|
||||
152
development-doc/plan/code-update/phase-1-infrastructure.md
Normal file
152
development-doc/plan/code-update/phase-1-infrastructure.md
Normal file
@@ -0,0 +1,152 @@
|
||||
# Phase 1：基础设施
|
||||
|
||||
日期：2026-04-04
|
||||
状态：待实施
|
||||
|
||||
---
|
||||
|
||||
## 1. 本阶段目的
|
||||
|
||||
新增代码指挥官 Agent 的基础架子，包括：
|
||||
- State 定义（角色、状态）
|
||||
- Prompt 模板
|
||||
- 工具注册
|
||||
- Agent 注册
|
||||
|
||||
---
|
||||
|
||||
## 2. 详细任务
|
||||
|
||||
### 2.1 State 定义
|
||||
|
||||
**文件**: `backend/app/agents/state.py`
|
||||
|
||||
```python
|
||||
# 新增 AgentRole
|
||||
class AgentRole(str, Enum):
|
||||
# ... 现有角色 ...
|
||||
CODE_COMMANDER = "code_commander"
|
||||
|
||||
# 新增 CodeCommanderState
|
||||
class CodeCommanderState(TypedDict):
|
||||
task_type: str # "demo" | "project" | "modification"
|
||||
ai_provider: str # "claude" | "gemini" | "codex" | "opencode"
|
||||
sandbox_mode: bool # True = 沙盒执行，False = 直接执行
|
||||
workspace_path: str | None # 临时工作目录
|
||||
execution_session_id: str | None # PTY 会话 ID
|
||||
```
|
||||
|
||||
### 2.2 Prompt 模板
|
||||
|
||||
**文件**: `backend/app/agents/prompts.py`
|
||||
|
||||
```python
|
||||
# 代码指挥官系统提示
|
||||
CODE_COMMANDER_SYSTEM_PROMPT = """你是一个代码指挥官，负责协调 AI 写代码助手。
|
||||
|
||||
你的职责：
|
||||
1. 接收用户选择的 AI 提供商（Claude/Gemini/Codex/OpenCode）
|
||||
2. 接收用户的写代码需求
|
||||
3. 进行安全分级判定
|
||||
4. 路由到合适的执行器
|
||||
|
||||
安全分级规则：
|
||||
- 低风险：demo、示例、贪食蛇游戏等独立项目
|
||||
- 高风险：修改现有项目、涉及 Jarvis 项目、路径操作等
|
||||
|
||||
执行模式：
|
||||
- 直接执行：低风险任务，直接运行
|
||||
- 沙盒执行：高风险任务，在临时目录隔离执行"""
|
||||
|
||||
# 沙盒执行说明
|
||||
SANDBOX_EXECUTION_PROMPT = """将在隔离的临时目录中执行任务。
|
||||
任务完成后，工作目录会被保留供下载。"""
|
||||
|
||||
# 直接执行说明
|
||||
DIRECT_EXECUTION_PROMPT = """将直接执行任务。
|
||||
如果需要交互，请等待用户输入。"""
|
||||
```
|
||||
|
||||
### 2.3 工具注册
|
||||
|
||||
**文件**: `backend/app/agents/tools/__init__.py`
|
||||
|
||||
```python
|
||||
# 新增工具集
|
||||
CODE_COMMANDER_TOOLSET = {
|
||||
"code_commander": [
|
||||
"execute_code_task",
|
||||
"get_execution_status",
|
||||
"send_interactive_input",
|
||||
"download_workspace",
|
||||
"cleanup_workspace",
|
||||
]
|
||||
}
|
||||
|
||||
# 在 SUB_COMMANDER_TOOLSETS 中添加
|
||||
SUB_COMMANDER_TOOLSETS: dict[str, list[str]] = {
|
||||
# ... 现有工具集 ...
|
||||
"code_commander": CODE_COMMANDER_TOOLSET["code_commander"],
|
||||
}
|
||||
```
|
||||
|
||||
### 2.4 Agent 注册
|
||||
|
||||
**文件**: `backend/app/agents/registry/builtins.py`
|
||||
|
||||
```python
|
||||
# 新增 CodeCommanderManifest
|
||||
CodeCommanderManifest = AgentManifest(
|
||||
id="code_commander",
|
||||
name="代码指挥官",
|
||||
description="协调 AI 写代码助手的指挥官",
|
||||
system_prompt=CODE_COMMANDER_SYSTEM_PROMPT,
|
||||
role=AgentRole.CODE_COMMANDER,
|
||||
sub_commanders=[], # 代码指挥官没有子指挥官
|
||||
tools=["execute_code_task", "get_execution_status",
|
||||
"send_interactive_input", "download_workspace", "cleanup_workspace"],
|
||||
permission_class=PermissionClass.HIGH, # 需要较高权限
|
||||
side_effect_scope=SideEffectScope.WORKSPACE,
|
||||
supports_retry=True,
|
||||
idempotent=False,
|
||||
safe_for_parallel_use=False,
|
||||
requires_confirmation=True,
|
||||
)
|
||||
|
||||
# 注册到 AGENT_MANIFESTS
|
||||
AGENT_MANIFESTS: dict[str, AgentManifest] = {
|
||||
# ... 现有 agent ...
|
||||
"code_commander": CodeCommanderManifest,
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. 核心文件清单
|
||||
|
||||
| 文件 | 操作 | 说明 |
|
||||
|------|------|------|
|
||||
| `state.py` | 修改 | 新增 `CODE_COMMANDER` 角色和 `CodeCommanderState` |
|
||||
| `prompts.py` | 修改 | 新增三个 prompt 常量 |
|
||||
| `tools/__init__.py` | 修改 | 新增工具集注册 |
|
||||
| `registry/builtins.py` | 修改 | 新增 `CodeCommanderManifest` |
|
||||
|
||||
---
|
||||
|
||||
## 4. 验收标准
|
||||
|
||||
- [ ] `AgentRole.CODE_COMMANDER` 存在且值正确
|
||||
- [ ] `CODE_COMMANDER_SYSTEM_PROMPT` 包含完整指令
|
||||
- [ ] 工具集已注册且可通过 `SUB_COMMANDER_TOOLSETS` 访问
|
||||
- [ ] `CodeCommanderManifest` 已注册且包含所有必要字段
|
||||
|
||||
---
|
||||
|
||||
## 5. 依赖关系
|
||||
|
||||
```
|
||||
本阶段 → Phase 2（执行引擎）
|
||||
→ Phase 3（Agent 集成）
|
||||
```
|
||||
|
||||
本阶段是后续所有阶段的基础。
|
||||
321
development-doc/plan/code-update/phase-2-execution-engine.md
Normal file
321
development-doc/plan/code-update/phase-2-execution-engine.md
Normal file
@@ -0,0 +1,321 @@
|
||||
# Phase 2：执行引擎
|
||||
|
||||
日期：2026-04-04
|
||||
状态：待实施
|
||||
|
||||
依赖：Phase 1 完成
|
||||
|
||||
---
|
||||
|
||||
## 1. 本阶段目的
|
||||
|
||||
实现代码指挥官的核心执行能力：
|
||||
- AI CLI Adapter：统一接口适配不同 AI CLI
|
||||
- Sandbox Executor：沙盒环境执行
|
||||
- Direct Executor：直接执行低风险任务
|
||||
- Security Classifier：安全分级
|
||||
|
||||
---
|
||||
|
||||
## 2. 详细任务
|
||||
|
||||
### 2.1 AI CLI Adapter
|
||||
|
||||
**新文件**: `backend/app/agents/tools/ai_adapter.py`
|
||||
|
||||
```python
|
||||
from abc import ABC, abstractmethod
|
||||
from pathlib import Path
|
||||
from dataclasses import dataclass
|
||||
|
||||
@dataclass
|
||||
class CodeExecutionResult:
|
||||
success: bool
|
||||
message: str
|
||||
files_created: list[str]
|
||||
output: str
|
||||
error: str | None
|
||||
|
||||
class AICLIAdapter(ABC):
|
||||
@property
|
||||
@abstractmethod
|
||||
def cli_name(self) -> str:
|
||||
"""CLI 命令名称，如 'claude', 'gemini'"""
|
||||
pass
|
||||
|
||||
@property
|
||||
@abstractmethod
|
||||
def requires_workspace(self) -> bool:
|
||||
"""是否需要工作目录"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def build_command(self, prompt: str, workspace: Path | None) -> list[str]:
|
||||
"""构建 CLI 命令"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def parse_output(self, output: str) -> CodeExecutionResult:
|
||||
"""解析 CLI 输出"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def is_installed(self) -> bool:
|
||||
"""检查 CLI 是否已安装"""
|
||||
pass
|
||||
|
||||
class ClaudeAdapter(AICLIAdapter):
|
||||
cli_name = "claude"
|
||||
requires_workspace = True
|
||||
|
||||
def build_command(self, prompt: str, workspace: Path | None) -> list[str]:
|
||||
return ["claude", "-p", prompt, "--dangerously-skip-permissions"]
|
||||
|
||||
# ... 其他方法实现
|
||||
|
||||
class GeminiAdapter(AICLIAdapter):
|
||||
cli_name = "gemini"
|
||||
requires_workspace = False
|
||||
# ...
|
||||
|
||||
class CodexAdapter(AICLIAdapter):
|
||||
cli_name = "codex"
|
||||
# ...
|
||||
|
||||
class OpenCodeAdapter(AICLIAdapter):
|
||||
cli_name = "opencode"
|
||||
# ...
|
||||
```
|
||||
|
||||
### 2.2 Security Classifier
|
||||
|
||||
**新文件**: `backend/app/agents/tools/security_classifier.py`
|
||||
|
||||
```python
|
||||
from enum import Enum
|
||||
|
||||
class RiskLevel(Enum):
|
||||
LOW = "low" # 直接执行
|
||||
HIGH = "high" # 沙盒执行
|
||||
|
||||
class SecurityClassifier:
|
||||
HIGH_RISK_KEYWORDS = [
|
||||
"修改", "编辑", "删除", "移动",
|
||||
"Jarvis", "backend", "frontend",
|
||||
"git", "config", ".env",
|
||||
]
|
||||
|
||||
LOW_RISK_KEYWORDS = [
|
||||
"demo", "示例", "贪食蛇", "俄罗斯方块",
|
||||
"小游戏", "独立项目", "新项目",
|
||||
"创建一个", "写一个",
|
||||
]
|
||||
|
||||
def classify(self, task_description: str, target_path: str | None = None) -> RiskLevel:
|
||||
# 1. 检查高风险关键词
|
||||
if any(kw in task_description for kw in self.HIGH_RISK_KEYWORDS):
|
||||
return RiskLevel.HIGH
|
||||
|
||||
# 2. 检查目标路径
|
||||
if target_path and self._is_project_path(target_path):
|
||||
return RiskLevel.HIGH
|
||||
|
||||
# 3. 检查低风险关键词
|
||||
if any(kw in task_description for kw in self.LOW_RISK_KEYWORDS):
|
||||
return RiskLevel.LOW
|
||||
|
||||
# 4. 默认高风险
|
||||
return RiskLevel.HIGH
|
||||
|
||||
def _is_project_path(self, path: str) -> bool:
|
||||
# 检查是否指向 Jarvis 项目路径
|
||||
return "Jarvis" in path or "backend/app" in path
|
||||
```
|
||||
|
||||
### 2.3 Sandbox Executor
|
||||
|
||||
**新文件**: `backend/app/agents/tools/sandbox_executor.py`
|
||||
|
||||
```python
|
||||
import tempfile
|
||||
import shutil
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
from dataclasses import dataclass, field
|
||||
from typing import AsyncGenerator
|
||||
|
||||
@dataclass
|
||||
class SandboxEnvironment:
|
||||
workspace_path: Path
|
||||
session_id: str
|
||||
|
||||
@staticmethod
|
||||
async def create() -> "SandboxEnvironment":
|
||||
"""创建新的沙盒环境"""
|
||||
temp_dir = tempfile.mkdtemp(prefix="jarvis_code_")
|
||||
session_id = Path(temp_dir).name
|
||||
return SandboxEnvironment(
|
||||
workspace_path=Path(temp_dir),
|
||||
session_id=session_id,
|
||||
)
|
||||
|
||||
async def cleanup(self):
|
||||
"""清理沙盒环境"""
|
||||
if self.workspace_path.exists():
|
||||
shutil.rmtree(self.workspace_path)
|
||||
|
||||
@dataclass
|
||||
class ExecutionResult:
|
||||
success: bool
|
||||
exit_code: int
|
||||
stdout: str
|
||||
stderr: str
|
||||
files_created: list[str] = field(default_factory=list)
|
||||
|
||||
class SandboxExecutor:
|
||||
def __init__(self, adapter: AICLIAdapter, timeout: int = 300):
|
||||
self.adapter = adapter
|
||||
self.timeout = timeout
|
||||
self._sessions: dict[str, SandboxEnvironment] = {}
|
||||
|
||||
async def execute(
|
||||
self,
|
||||
prompt: str,
|
||||
session_id: str | None = None
|
||||
) -> AsyncGenerator[str, None]:
|
||||
"""执行代码任务，yield 实时输出"""
|
||||
# 1. 创建或复用沙盒环境
|
||||
if session_id and session_id in self._sessions:
|
||||
env = self._sessions[session_id]
|
||||
else:
|
||||
env = await SandboxEnvironment.create()
|
||||
self._sessions[env.session_id] = env
|
||||
session_id = env.session_id
|
||||
|
||||
# 2. 构建命令
|
||||
cmd = self.adapter.build_command(prompt, env.workspace_path)
|
||||
|
||||
# 3. 异步执行，实时 yield 输出
|
||||
process = await asyncio.create_subprocess_exec(
|
||||
*cmd,
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
cwd=str(env.workspace_path),
|
||||
)
|
||||
|
||||
# 4. 实时读取输出
|
||||
while True:
|
||||
line = await process.stdout.readline()
|
||||
if not line:
|
||||
break
|
||||
yield line.decode()
|
||||
|
||||
# 5. 等待完成
|
||||
await process.wait()
|
||||
|
||||
# 6. 收集结果
|
||||
return ExecutionResult(
|
||||
success=process.returncode == 0,
|
||||
exit_code=process.returncode or 0,
|
||||
stdout=...,
|
||||
stderr=...,
|
||||
files_created=self._list_created_files(env.workspace_path),
|
||||
)
|
||||
|
||||
async def cleanup_session(self, session_id: str):
|
||||
"""清理指定会话"""
|
||||
if session_id in self._sessions:
|
||||
await self._sessions[session_id].cleanup()
|
||||
del self._sessions[session_id]
|
||||
```
|
||||
|
||||
### 2.4 Direct Executor
|
||||
|
||||
**新文件**: `backend/app/agents/tools/direct_executor.py`
|
||||
|
||||
```python
|
||||
class DirectExecutor:
|
||||
def __init__(self, adapter: AICLIAdapter, timeout: int = 60):
|
||||
self.adapter = adapter
|
||||
self.timeout = timeout
|
||||
|
||||
async def execute(self, prompt: str) -> ExecutionResult:
|
||||
"""直接执行，不需要沙盒"""
|
||||
if not self.adapter.is_installed():
|
||||
return ExecutionResult(
|
||||
success=False,
|
||||
exit_code=-1,
|
||||
stdout="",
|
||||
stderr=f"{self.adapter.cli_name} is not installed",
|
||||
)
|
||||
|
||||
cmd = self.adapter.build_command(prompt, None)
|
||||
|
||||
process = await asyncio.create_subprocess_exec(
|
||||
*cmd,
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
|
||||
try:
|
||||
stdout, stderr = await asyncio.wait_for(
|
||||
process.communicate(),
|
||||
timeout=self.timeout,
|
||||
)
|
||||
return ExecutionResult(
|
||||
success=process.returncode == 0,
|
||||
exit_code=process.returncode or 0,
|
||||
stdout=stdout.decode(),
|
||||
stderr=stderr.decode(),
|
||||
)
|
||||
except asyncio.TimeoutError:
|
||||
process.kill()
|
||||
return ExecutionResult(
|
||||
success=False,
|
||||
exit_code=-1,
|
||||
stdout="",
|
||||
stderr=f"Execution timed out after {self.timeout}s",
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. 核心文件清单
|
||||
|
||||
| 文件 | 操作 | 说明 |
|
||||
|------|------|------|
|
||||
| `ai_adapter.py` | 新增 | 抽象基类 + 4 个具体实现 |
|
||||
| `security_classifier.py` | 新增 | 安全分级器 |
|
||||
| `sandbox_executor.py` | 新增 | 沙盒执行器 |
|
||||
| `direct_executor.py` | 新增 | 直接执行器 |
|
||||
|
||||
---
|
||||
|
||||
## 4. 验收标准
|
||||
|
||||
- [ ] `AICLIAdapter` 可以正确识别 4 种 CLI
|
||||
- [ ] `SecurityClassifier` 能正确分类高低风险
|
||||
- [ ] `SandboxExecutor` 能创建、执行、清理沙盒
|
||||
- [ ] `DirectExecutor` 能直接执行低风险任务
|
||||
- [ ] 所有执行器支持流式输出
|
||||
|
||||
---
|
||||
|
||||
## 5. 风险与缓解
|
||||
|
||||
| 风险 | 缓解 |
|
||||
|------|------|
|
||||
| AI CLI 未安装 | `is_installed()` 检查 + 友好提示 |
|
||||
| 执行超时 | `timeout` 参数控制 |
|
||||
| 沙盒清理遗漏 | 使用 `finally` 块确保清理 |
|
||||
|
||||
---
|
||||
|
||||
## 6. 依赖关系
|
||||
|
||||
```
|
||||
Phase 1（基础设施）
|
||||
↓
|
||||
本阶段 → Phase 3（Agent 集成）
|
||||
→ Phase 4（流式交互）
|
||||
```
|
||||
162
development-doc/plan/code-update/phase-3-agent-integration.md
Normal file
162
development-doc/plan/code-update/phase-3-agent-integration.md
Normal file
@@ -0,0 +1,162 @@
|
||||
# Phase 3：Agent 集成
|
||||
|
||||
日期：2026-04-04
|
||||
状态：待实施
|
||||
|
||||
依赖：Phase 1 + Phase 2 完成
|
||||
|
||||
---
|
||||
|
||||
## 1. 本阶段目的
|
||||
|
||||
将代码指挥官接入 LangGraph：
|
||||
- Graph 节点
|
||||
- 边路由
|
||||
- 任务模型
|
||||
|
||||
---
|
||||
|
||||
## 2. 详细任务
|
||||
|
||||
### 2.1 Graph 节点
|
||||
|
||||
**文件**: `backend/app/agents/graph.py`
|
||||
|
||||
```python
|
||||
# 新增 code_commander_node
|
||||
async def code_commander_node(state: AgentState) -> AgentState:
|
||||
"""代码指挥官节点"""
|
||||
# 1. 获取用户需求和选择的 AI 提供商
|
||||
user_message = state.messages[-1].content
|
||||
ai_provider = state.get("ai_provider", "claude")
|
||||
|
||||
# 2. 安全分级
|
||||
classifier = SecurityClassifier()
|
||||
risk_level = classifier.classify(user_message)
|
||||
|
||||
# 3. 根据风险等级选择执行器
|
||||
adapter = get_adapter(ai_provider)
|
||||
if risk_level == RiskLevel.LOW:
|
||||
executor = DirectExecutor(adapter)
|
||||
result = await executor.execute(user_message)
|
||||
else:
|
||||
sandbox = await SandboxEnvironment.create()
|
||||
executor = SandboxExecutor(adapter)
|
||||
result = await executor.execute(user_message, sandbox.session_id)
|
||||
state["workspace_path"] = str(sandbox.workspace_path)
|
||||
state["execution_session_id"] = sandbox.session_id
|
||||
|
||||
# 4. 更新状态
|
||||
state.messages.append(AIMessage(content=str(result)))
|
||||
state["next_step"] = None # 任务完成
|
||||
|
||||
return state
|
||||
|
||||
# 节点注册到 NODES
|
||||
NODES: dict[str, NodeCallable] = {
|
||||
# ... 现有节点 ...
|
||||
"code_commander": code_commander_node,
|
||||
}
|
||||
```
|
||||
|
||||
### 2.2 边路由
|
||||
|
||||
**文件**: `backend/app/agents/graph.py`
|
||||
|
||||
```python
|
||||
def _should_route_to_code_commander(state: AgentState) -> str:
|
||||
"""判断是否路由到代码指挥官"""
|
||||
if state.current_agent == "code_commander":
|
||||
return "code_commander"
|
||||
# ... 其他条件
|
||||
return END
|
||||
|
||||
# 边注册
|
||||
def _build_graph() -> CompiledGraph:
|
||||
# ... 现有边 ...
|
||||
|
||||
# 新增代码指挥官相关边
|
||||
graph.add_conditional_edges(
|
||||
"master",
|
||||
_should_route_to_code_commander,
|
||||
{
|
||||
"code_commander": "code_commander",
|
||||
END: END,
|
||||
}
|
||||
)
|
||||
|
||||
graph.add_edge("code_commander", END)
|
||||
|
||||
return graph.compile()
|
||||
```
|
||||
|
||||
### 2.3 任务模型
|
||||
|
||||
**文件**: `backend/app/agents/schemas/task.py`
|
||||
|
||||
```python
|
||||
from pydantic import BaseModel, Field
|
||||
from typing import Literal
|
||||
|
||||
class CodeProviderType(str, Enum):
|
||||
CLAUDE = "claude"
|
||||
GEMINI = "gemini"
|
||||
CODEX = "codex"
|
||||
OPENCODE = "opencode"
|
||||
|
||||
class RiskLevelType(str, Enum):
|
||||
LOW = "low"
|
||||
HIGH = "high"
|
||||
|
||||
class CodeTask(BaseModel):
|
||||
"""代码任务"""
|
||||
id: str = Field(default_factory=lambda: f"code_{uuid.uuid4().hex[:8]}")
|
||||
provider: CodeProviderType
|
||||
prompt: str
|
||||
risk_level: RiskLevelType
|
||||
sandbox_mode: bool
|
||||
workspace_path: str | None = None
|
||||
session_id: str | None = None
|
||||
status: Literal["pending", "running", "completed", "failed"] = "pending"
|
||||
created_at: datetime = Field(default_factory=datetime.now)
|
||||
|
||||
class CodeExecutionResult(BaseModel):
|
||||
"""代码执行结果"""
|
||||
task_id: str
|
||||
success: bool
|
||||
exit_code: int
|
||||
stdout: str
|
||||
stderr: str
|
||||
files_created: list[str] = Field(default_factory=list)
|
||||
workspace_path: str | None = None
|
||||
completed_at: datetime = Field(default_factory=datetime.now)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. 核心文件清单
|
||||
|
||||
| 文件 | 操作 | 说明 |
|
||||
|------|------|------|
|
||||
| `graph.py` | 修改 | 新增 `code_commander_node` 和边路由 |
|
||||
| `schemas/task.py` | 修改 | 新增 `CodeTask`, `CodeExecutionResult` 等模型 |
|
||||
|
||||
---
|
||||
|
||||
## 4. 验收标准
|
||||
|
||||
- [ ] `code_commander_node` 正确处理任务
|
||||
- [ ] `SecurityClassifier` 被正确调用
|
||||
- [ ] 高低风险任务路由到正确的执行器
|
||||
- [ ] `CodeTask` 和 `CodeExecutionResult` 模型正确
|
||||
|
||||
---
|
||||
|
||||
## 5. 依赖关系
|
||||
|
||||
```
|
||||
Phase 1 + Phase 2
|
||||
↓
|
||||
本阶段 → Phase 4（流式交互）
|
||||
→ Phase 5（前端集成）
|
||||
```
|
||||
@@ -0,0 +1,298 @@
|
||||
# Phase 4：流式交互
|
||||
|
||||
日期：2026-04-04
|
||||
状态：待实施
|
||||
|
||||
依赖：Phase 3 完成
|
||||
|
||||
---
|
||||
|
||||
## 1. 本阶段目的
|
||||
|
||||
实现 PTY 终端 + WebSocket 流式输出：
|
||||
- PTY 终端管理
|
||||
- WebSocket 端点
|
||||
- 流式输出集成
|
||||
- 交互输入
|
||||
|
||||
---
|
||||
|
||||
## 2. 详细任务
|
||||
|
||||
### 2.1 PTY Terminal Engine
|
||||
|
||||
**新文件**: `backend/app/agents/tools/terminal_engine.py`
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import os
|
||||
from dataclasses import dataclass, field
|
||||
from typing import AsyncGenerator
|
||||
|
||||
@dataclass
|
||||
class PTYSession:
|
||||
session_id: str
|
||||
process: asyncio.subprocess.Process
|
||||
workspace_path: str
|
||||
|
||||
class PTYManager:
|
||||
def __init__(self):
|
||||
self._sessions: dict[str, PTYSession] = {}
|
||||
self._output_queues: dict[str, asyncio.Queue] = {}
|
||||
|
||||
async def spawn(
|
||||
self,
|
||||
cli: str,
|
||||
args: list[str],
|
||||
cwd: str,
|
||||
session_id: str | None = None
|
||||
) -> str:
|
||||
"""启动 PTY 会话"""
|
||||
if session_id is None:
|
||||
session_id = f"pty_{os.urandom(8).hex()}"
|
||||
|
||||
# 创建 PTY 进程
|
||||
process = await asyncio.create_subprocess_exec(
|
||||
*([cli] + args),
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
cwd=cwd,
|
||||
env={**os.environ, "TERM": "xterm-256color"},
|
||||
)
|
||||
|
||||
session = PTYSession(
|
||||
session_id=session_id,
|
||||
process=process,
|
||||
workspace_path=cwd,
|
||||
)
|
||||
self._sessions[session_id] = session
|
||||
self._output_queues[session_id] = asyncio.Queue()
|
||||
|
||||
# 启动输出读取协程
|
||||
asyncio.create_task(self._read_output(session_id))
|
||||
|
||||
return session_id
|
||||
|
||||
async def _read_output(self, session_id: str):
|
||||
"""读取 PTY 输出并放入队列"""
|
||||
session = self._sessions.get(session_id)
|
||||
if not session:
|
||||
return
|
||||
|
||||
queue = self._output_queues[session_id]
|
||||
|
||||
while True:
|
||||
line = await session.process.stdout.readline()
|
||||
if not line:
|
||||
break
|
||||
await queue.put(line.decode())
|
||||
|
||||
# 同时推送给所有订阅者
|
||||
await self._broadcast(session_id, line.decode())
|
||||
|
||||
await queue.put(None) # 结束标记
|
||||
|
||||
async def write(self, session_id: str, data: str):
|
||||
"""写入 PTY（用户输入）"""
|
||||
session = self._sessions.get(session_id)
|
||||
if session and session.process.stdin:
|
||||
session.process.stdin.write(data)
|
||||
await session.process.stdin.drain()
|
||||
|
||||
async def read(self, session_id: str) -> AsyncGenerator[str, None]:
|
||||
"""读取 PTY 输出"""
|
||||
queue = self._output_queues.get(session_id)
|
||||
if not queue:
|
||||
return
|
||||
|
||||
while True:
|
||||
line = await queue.get()
|
||||
if line is None:
|
||||
break
|
||||
yield line
|
||||
|
||||
async def resize(self, session_id: str, rows: int, cols: int):
|
||||
"""调整终端大小"""
|
||||
# TODO: 实现 resize
|
||||
pass
|
||||
|
||||
async def kill(self, session_id: str):
|
||||
"""终止 PTY 会话"""
|
||||
if session_id in self._sessions:
|
||||
session = self._sessions[session_id]
|
||||
session.process.terminate()
|
||||
await session.process.wait()
|
||||
del self._sessions[session_id]
|
||||
del self._output_queues[session_id]
|
||||
|
||||
async def _broadcast(self, session_id: str, data: str):
|
||||
"""广播输出到 WebSocket"""
|
||||
# 实际推送由 router 层处理
|
||||
pass
|
||||
```
|
||||
|
||||
### 2.2 WebSocket 端点
|
||||
|
||||
**新文件**: `backend/app/routers/terminal.py`
|
||||
|
||||
```python
|
||||
from fastapi import APIRouter, WebSocket, WebSocketDisconnect
|
||||
from typing import dict
|
||||
|
||||
router = APIRouter(prefix="/ws/terminal", tags=["terminal"])
|
||||
|
||||
class ConnectionManager:
|
||||
def __init__(self):
|
||||
self.active_connections: dict[str, WebSocket] = {}
|
||||
|
||||
async def connect(self, session_id: str, websocket: WebSocket):
|
||||
await websocket.accept()
|
||||
self.active_connections[session_id] = websocket
|
||||
|
||||
def disconnect(self, session_id: str):
|
||||
if session_id in self.active_connections:
|
||||
del self.active_connections[session_id]
|
||||
|
||||
async def send(self, session_id: str, data: str):
|
||||
if session_id in self.active_connections:
|
||||
await self.active_connections[session_id].send_text(data)
|
||||
|
||||
manager = ConnectionManager()
|
||||
|
||||
@router.websocket("/{session_id}")
|
||||
async def terminal_websocket(websocket: WebSocket, session_id: str):
|
||||
await manager.connect(session_id, websocket)
|
||||
|
||||
# 获取 PTY Manager 实例
|
||||
from app.agents.tools.terminal_engine import pty_manager
|
||||
|
||||
try:
|
||||
# 订阅该 session 的输出
|
||||
queue = pty_manager._output_queues.get(session_id)
|
||||
if queue:
|
||||
while True:
|
||||
data = await websocket.receive_text()
|
||||
# 接收用户输入
|
||||
await pty_manager.write(session_id, data + "\n")
|
||||
|
||||
except WebSocketDisconnect:
|
||||
manager.disconnect(session_id)
|
||||
```
|
||||
|
||||
### 2.3 流式输出集成
|
||||
|
||||
**新文件**: `backend/app/agents/tools/stream_output.py`
|
||||
|
||||
```python
|
||||
import json
|
||||
from typing import AsyncGenerator
|
||||
from dataclasses import dataclass
|
||||
|
||||
@dataclass
|
||||
class StreamEvent:
|
||||
type: str # "output" | "error" | "status" | "complete"
|
||||
session_id: str
|
||||
data: str
|
||||
timestamp: str
|
||||
|
||||
class StreamOutput:
|
||||
def __init__(self, session_id: str, websocket_sender):
|
||||
self.session_id = session_id
|
||||
self.websocket_sender = websocket_sender
|
||||
|
||||
async def push(self, event_type: str, data: str):
|
||||
"""推送事件到 WebSocket"""
|
||||
event = StreamEvent(
|
||||
type=event_type,
|
||||
session_id=self.session_id,
|
||||
data=data,
|
||||
timestamp=datetime.now().isoformat(),
|
||||
)
|
||||
await self.websocket_sender(self.session_id, json.dumps(event.__dict__))
|
||||
|
||||
async def stream_execution(
|
||||
self,
|
||||
executor,
|
||||
prompt: str
|
||||
) -> AsyncGenerator[str, None]:
|
||||
"""包装执行器，实现流式输出"""
|
||||
async for line in executor.execute(prompt):
|
||||
await self.push("output", line)
|
||||
yield line
|
||||
|
||||
await self.push("complete", "")
|
||||
```
|
||||
|
||||
### 2.4 交互输入
|
||||
|
||||
**新文件**: `backend/app/agents/tools/interactive_input.py`
|
||||
|
||||
```python
|
||||
class InteractiveInputHandler:
|
||||
def __init__(self, pty_manager: PTYManager):
|
||||
self.pty_manager = pty_manager
|
||||
self._pending_inputs: dict[str, asyncio.Event] = {}
|
||||
|
||||
async def wait_for_input(self, session_id: str, prompt: str) -> str:
|
||||
"""等待用户输入（如 "y" 确认）"""
|
||||
event = asyncio.Event()
|
||||
self._pending_inputs[session_id] = event
|
||||
|
||||
# 发送提示
|
||||
from app.routers.terminal import manager
|
||||
await manager.send(session_id, f"\n{prompt}\n")
|
||||
|
||||
# 等待输入完成
|
||||
await event.wait()
|
||||
del self._pending_inputs[session_id]
|
||||
|
||||
return self._input_cache.get(session_id, "")
|
||||
|
||||
async def send_input(self, session_id: str, data: str):
|
||||
"""用户发送输入"""
|
||||
self._input_cache[session_id] = data
|
||||
if session_id in self._pending_inputs:
|
||||
self._pending_inputs[session_id].set()
|
||||
|
||||
# 同时写入 PTY
|
||||
await self.pty_manager.write(session_id, data + "\n")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. 核心文件清单
|
||||
|
||||
| 文件 | 操作 | 说明 |
|
||||
|------|------|------|
|
||||
| `terminal_engine.py` | 新增 | PTY 终端管理 |
|
||||
| `routers/terminal.py` | 新增 | WebSocket 端点 |
|
||||
| `stream_output.py` | 新增 | 流式输出封装 |
|
||||
| `interactive_input.py` | 新增 | 交互输入处理 |
|
||||
|
||||
---
|
||||
|
||||
## 4. 验收标准
|
||||
|
||||
- [ ] PTY 会话可以启动、读写、终止
|
||||
- [ ] WebSocket 可以建立连接并收发消息
|
||||
- [ ] 执行输出实时推送到前端
|
||||
- [ ] 用户输入可以传递到 PTY
|
||||
|
||||
---
|
||||
|
||||
## 5. 依赖关系
|
||||
|
||||
```
|
||||
Phase 3（Agent 集成）
|
||||
↓
|
||||
本阶段 → Phase 5（前端集成）
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. 备注
|
||||
|
||||
PTY 实现参考了 golutra 的 `src-tauri/src/runtime/pty.rs`：
|
||||
- 使用 `portable-pty` 库
|
||||
- Windows 路径兼容处理
|
||||
- shim 机制用于信号处理
|
||||
364
development-doc/plan/code-update/phase-5-frontend-integration.md
Normal file
364
development-doc/plan/code-update/phase-5-frontend-integration.md
Normal file
@@ -0,0 +1,364 @@
|
||||
# Phase 5：前端集成
|
||||
|
||||
日期：2026-04-04
|
||||
状态：待实施
|
||||
|
||||
依赖：Phase 4 完成
|
||||
|
||||
---
|
||||
|
||||
## 1. 本阶段目的
|
||||
|
||||
Vue 前端新增代码指挥官 UI：
|
||||
- 页面组件
|
||||
- 终端显示组件
|
||||
- WebSocket 服务
|
||||
- 路由配置
|
||||
|
||||
---
|
||||
|
||||
## 2. 详细任务
|
||||
|
||||
### 2.1 页面组件
|
||||
|
||||
**新文件**: `frontend/src/pages/chat/CodeCommander.vue`
|
||||
|
||||
```vue
|
||||
<template>
|
||||
<div class="code-commander">
|
||||
<!-- AI 提供商选择器 -->
|
||||
<div class="provider-selector">
|
||||
<div class="label">选择 AI 助手</div>
|
||||
<div class="providers">
|
||||
<button
|
||||
v-for="p in providers"
|
||||
:key="p.id"
|
||||
:class="{ active: selectedProvider === p.id }"
|
||||
@click="selectedProvider = p.id"
|
||||
>
|
||||
<img :src="p.icon" :alt="p.name" />
|
||||
{{ p.name }}
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- 任务输入 -->
|
||||
<div class="task-input">
|
||||
<textarea
|
||||
v-model="taskPrompt"
|
||||
placeholder="描述你想让 AI 帮你做什么..."
|
||||
rows="4"
|
||||
/>
|
||||
<button @click="executeTask" :disabled="isExecuting">
|
||||
{{ isExecuting ? '执行中...' : '开始执行' }}
|
||||
</button>
|
||||
</div>
|
||||
|
||||
<!-- 终端输出 -->
|
||||
<TerminalDisplay
|
||||
ref="terminalRef"
|
||||
:session-id="currentSessionId"
|
||||
@input="handleUserInput"
|
||||
/>
|
||||
|
||||
<!-- 交互输入框 -->
|
||||
<div v-if="isWaitingForInput" class="interactive-input">
|
||||
<span>{{ inputPrompt }}</span>
|
||||
<input v-model="userInput" @keyup.enter="sendUserInput" />
|
||||
</div>
|
||||
|
||||
<!-- 操作按钮 -->
|
||||
<div class="actions">
|
||||
<button @click="downloadFiles" :disabled="!canDownload">
|
||||
下载文件
|
||||
</button>
|
||||
<button @click="cleanup" :disabled="!canCleanup">
|
||||
清理
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<script setup lang="ts">
|
||||
import { ref, computed } from 'vue'
|
||||
import TerminalDisplay from '@/components/TerminalDisplay.vue'
|
||||
import { terminalWsService } from '@/services/terminalWs'
|
||||
|
||||
const providers = [
|
||||
{ id: 'claude', name: 'Claude', icon: '/icons/claude.png' },
|
||||
{ id: 'gemini', name: 'Gemini', icon: '/icons/gemini.png' },
|
||||
{ id: 'codex', name: 'Codex', icon: '/icons/codex.png' },
|
||||
{ id: 'opencode', name: 'OpenCode', icon: '/icons/opencode.png' },
|
||||
]
|
||||
|
||||
const selectedProvider = ref('claude')
|
||||
const taskPrompt = ref('')
|
||||
const isExecuting = ref(false)
|
||||
const currentSessionId = ref<string | null>(null)
|
||||
const isWaitingForInput = ref(false)
|
||||
const inputPrompt = ref('')
|
||||
const userInput = ref('')
|
||||
const terminalRef = ref<InstanceType<typeof TerminalDisplay> | null>(null)
|
||||
|
||||
const canDownload = computed(() => currentSessionId.value !== null)
|
||||
const canCleanup = computed(() => currentSessionId.value !== null)
|
||||
|
||||
async function executeTask() {
|
||||
if (!taskPrompt.value.trim()) return
|
||||
|
||||
isExecuting.value = true
|
||||
currentSessionId.value = await terminalWsService.connect(selectedProvider.value)
|
||||
|
||||
// 订阅消息
|
||||
terminalWsService.onMessage((msg) => {
|
||||
if (msg.type === 'output') {
|
||||
terminalRef.value?.write(msg.data)
|
||||
} else if (msg.type === 'waiting_input') {
|
||||
isWaitingForInput.value = true
|
||||
inputPrompt.value = msg.data
|
||||
} else if (msg.type === 'complete') {
|
||||
isExecuting.value = false
|
||||
}
|
||||
})
|
||||
|
||||
// 发送任务
|
||||
await terminalWsService.sendTask(currentSessionId.value, taskPrompt.value)
|
||||
}
|
||||
|
||||
function handleUserInput(data: string) {
|
||||
terminalWsService.sendInput(currentSessionId.value!, data)
|
||||
}
|
||||
|
||||
function sendUserInput() {
|
||||
terminalWsService.sendInput(currentSessionId.value!, userInput.value)
|
||||
userInput.value = ''
|
||||
isWaitingForInput.value = false
|
||||
}
|
||||
|
||||
async function downloadFiles() {
|
||||
// TODO: 调用下载 API
|
||||
}
|
||||
|
||||
async function cleanup() {
|
||||
if (currentSessionId.value) {
|
||||
await terminalWsService.disconnect(currentSessionId.value)
|
||||
currentSessionId.value = null
|
||||
}
|
||||
}
|
||||
</script>
|
||||
```
|
||||
|
||||
### 2.2 终端显示组件
|
||||
|
||||
**新文件**: `frontend/src/components/TerminalDisplay.vue`
|
||||
|
||||
```vue
|
||||
<template>
|
||||
<div class="terminal-display" ref="containerRef">
|
||||
<div class="terminal-output" ref="outputRef"></div>
|
||||
</div>
|
||||
</template>
|
||||
|
||||
<script setup lang="ts">
|
||||
import { ref, onMounted, onUnmounted } from 'vue'
|
||||
import { Terminal } from 'xterm'
|
||||
import { FitAddon } from 'xterm-addon-fit'
|
||||
import 'xterm/css/xterm.css'
|
||||
|
||||
const props = defineProps<{
|
||||
sessionId: string | null
|
||||
}>()
|
||||
|
||||
const emit = defineEmits<{
|
||||
input: [data: string]
|
||||
}>()
|
||||
|
||||
const containerRef = ref<HTMLElement | null>(null)
|
||||
const outputRef = ref<HTMLElement | null>(null)
|
||||
let terminal: Terminal | null = null
|
||||
let fitAddon: FitAddon | null = null
|
||||
|
||||
onMounted(() => {
|
||||
terminal = new Terminal({
|
||||
theme: { background: '#1e1e1e' },
|
||||
cursorBlink: true,
|
||||
})
|
||||
|
||||
fitAddon = new FitAddon()
|
||||
terminal.loadAddon(fitAddon)
|
||||
|
||||
terminal.open(outputRef.value!)
|
||||
fitAddon.fit()
|
||||
|
||||
// 用户输入
|
||||
terminal.onData((data) => {
|
||||
emit('input', data)
|
||||
})
|
||||
})
|
||||
|
||||
onUnmounted(() => {
|
||||
terminal?.dispose()
|
||||
})
|
||||
|
||||
function write(data: string) {
|
||||
terminal?.write(data)
|
||||
}
|
||||
|
||||
function clear() {
|
||||
terminal?.clear()
|
||||
}
|
||||
|
||||
defineExpose({ write, clear })
|
||||
</script>
|
||||
|
||||
<style scoped>
|
||||
.terminal-display {
|
||||
background: #1e1e1e;
|
||||
border-radius: 8px;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
.terminal-output {
|
||||
padding: 12px;
|
||||
min-height: 400px;
|
||||
}
|
||||
</style>
|
||||
```
|
||||
|
||||
### 2.3 WebSocket 服务
|
||||
|
||||
**新文件**: `frontend/src/services/terminalWs.ts`
|
||||
|
||||
```typescript
|
||||
type MessageHandler = (msg: StreamMessage) => void
|
||||
|
||||
interface StreamMessage {
|
||||
type: 'output' | 'error' | 'status' | 'waiting_input' | 'complete'
|
||||
session_id: string
|
||||
data: string
|
||||
timestamp: string
|
||||
}
|
||||
|
||||
class TerminalWsService {
|
||||
private ws: WebSocket | null = null
|
||||
private sessionId: string | null = null
|
||||
private handlers: MessageHandler[] = []
|
||||
private reconnectAttempts = 0
|
||||
private maxReconnectAttempts = 5
|
||||
|
||||
async connect(provider: string): Promise<string> {
|
||||
// 创建会话
|
||||
const response = await fetch('/api/code-commander/sessions', {
|
||||
method: 'POST',
|
||||
body: JSON.stringify({ provider }),
|
||||
})
|
||||
const { session_id } = await response.json()
|
||||
|
||||
// 建立 WebSocket
|
||||
this.ws = new WebSocket(`ws://localhost:8000/ws/terminal/${session_id}`)
|
||||
|
||||
this.ws.onmessage = (event) => {
|
||||
const msg: StreamMessage = JSON.parse(event.data)
|
||||
this.handlers.forEach((h) => h(msg))
|
||||
}
|
||||
|
||||
this.ws.onclose = () => {
|
||||
this.attemptReconnect()
|
||||
}
|
||||
|
||||
this.sessionId = session_id
|
||||
return session_id
|
||||
}
|
||||
|
||||
async sendTask(sessionId: string, prompt: string) {
|
||||
await fetch(`/api/code-commander/sessions/${sessionId}/task`, {
|
||||
method: 'POST',
|
||||
body: JSON.stringify({ prompt }),
|
||||
})
|
||||
}
|
||||
|
||||
sendInput(sessionId: string, input: string) {
|
||||
this.ws?.send(JSON.stringify({ type: 'input', data: input }))
|
||||
}
|
||||
|
||||
onMessage(handler: MessageHandler) {
|
||||
this.handlers.push(handler)
|
||||
}
|
||||
|
||||
removeHandler(handler: MessageHandler) {
|
||||
this.handlers = this.handlers.filter((h) => h !== handler)
|
||||
}
|
||||
|
||||
async disconnect(sessionId: string) {
|
||||
await fetch(`/api/code-commander/sessions/${sessionId}`, {
|
||||
method: 'DELETE',
|
||||
})
|
||||
this.ws?.close()
|
||||
this.ws = null
|
||||
this.sessionId = null
|
||||
}
|
||||
|
||||
private async attemptReconnect() {
|
||||
if (this.reconnectAttempts >= this.maxReconnectAttempts) {
|
||||
return
|
||||
}
|
||||
this.reconnectAttempts++
|
||||
await new Promise((r) => setTimeout(r, 1000 * this.reconnectAttempts))
|
||||
// 重新连接
|
||||
}
|
||||
}
|
||||
|
||||
export const terminalWsService = new TerminalWsService()
|
||||
```
|
||||
|
||||
### 2.4 路由配置
|
||||
|
||||
**文件**: `frontend/src/router/index.ts`
|
||||
|
||||
```typescript
|
||||
{
|
||||
path: '/code-commander',
|
||||
name: 'CodeCommander',
|
||||
component: () => import('@/pages/chat/CodeCommander.vue'),
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. 核心文件清单
|
||||
|
||||
| 文件 | 操作 | 说明 |
|
||||
|------|------|------|
|
||||
| `CodeCommander.vue` | 新增 | 主页面组件 |
|
||||
| `TerminalDisplay.vue` | 新增 | 终端显示组件（xterm.js） |
|
||||
| `terminalWs.ts` | 新增 | WebSocket 服务 |
|
||||
| `router/index.ts` | 修改 | 新增路由 |
|
||||
|
||||
---
|
||||
|
||||
## 4. 验收标准
|
||||
|
||||
- [ ] 用户可以选择 AI 提供商
|
||||
- [ ] 可以输入任务描述并执行
|
||||
- [ ] 终端实时显示 AI 输出
|
||||
- [ ] 用户可以输入交互（如 "y"）
|
||||
- [ ] 可以下载和清理文件
|
||||
|
||||
---
|
||||
|
||||
## 5. 依赖
|
||||
|
||||
| 依赖 | 版本 | 用途 |
|
||||
|------|------|------|
|
||||
| xterm | ^5.x | 终端渲染 |
|
||||
| xterm-addon-fit | ^0.8.x | 自适应大小 |
|
||||
|
||||
---
|
||||
|
||||
## 6. 依赖关系
|
||||
|
||||
```
|
||||
Phase 4（流式交互）
|
||||
↓
|
||||
本阶段（前端集成）→ 端到端测试
|
||||
```
|
||||
174
development-doc/plan/forum-update/README.md
Normal file
174
development-doc/plan/forum-update/README.md
Normal file
@@ -0,0 +1,174 @@
|
||||
# Jarvis Forum 升级计划索引
|
||||
|
||||
本目录用于存放 Jarvis 论坛系统的分阶段升级规划文档。
|
||||
|
||||
## 文档说明
|
||||
|
||||
| 文件 | 说明 |
|
||||
|------|------|
|
||||
| `README.md` | 总览、阶段关系、实施顺序 |
|
||||
| `phase-f-0-current-state.md` | 当前现状、问题、目标架构、VCPToolBox 借鉴 |
|
||||
| `phase-f-1-data-model.md` | 数据模型升级 |
|
||||
| `phase-f-2-forum-api.md` | API 增强与安全 |
|
||||
| `phase-f-3-permissions.md` | 权限系统 |
|
||||
| `phase-f-4-ai-integration.md` | AI 集成 |
|
||||
| `checklist.md` | 执行清单 |
|
||||
|
||||
## 推荐阅读顺序
|
||||
|
||||
1. 先读 `phase-f-0-current-state.md`
|
||||
2. 再按顺序阅读 phase f-1 ~ f-4
|
||||
3. 实施时严格按阶段推进
|
||||
4. 参考 `checklist.md` 进行任务追踪
|
||||
|
||||
---
|
||||
|
||||
## 总体升级原则
|
||||
|
||||
1. **安全第一** - 输入验证、路径安全、并发控制
|
||||
2. **可扩展性** - 支持多板块、标签、分类
|
||||
3. **AI 增强** - AI 自动回复、摘要生成、智能分类
|
||||
4. **测试优先** - 所有升级都要配套测试
|
||||
5. **向后兼容** - 保持现有 API 兼容性
|
||||
|
||||
---
|
||||
|
||||
## 阶段总览图
|
||||
|
||||
```
|
||||
F.0 ──────────────────────────────────────────────────────────────┐
|
||||
│ 现状与目标 │
|
||||
│ - 当前架构分析 │
|
||||
│ - 短板识别 │
|
||||
│ - VCPToolBox VCP论坛 借鉴 │
|
||||
└────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
F.1 ──────────────────────────────────────────────────────────────┐
|
||||
│ 数据模型升级 │
|
||||
│ - 板块/分类系统 │
|
||||
│ - 标签系统 │
|
||||
│ - 帖子元数据扩展 │
|
||||
│ │
|
||||
│ 核心文件: models/forum.py │
|
||||
│ 工作量: 2 天 │
|
||||
└────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
F.2 ──────────────────────────────────────────────────────────────┐
|
||||
│ API 增强与安全 │
|
||||
│ - 文件锁机制 │
|
||||
│ - 输入验证强化 │
|
||||
│ - 并发控制 │
|
||||
│ - API 端点扩展 │
|
||||
│ │
|
||||
│ 核心文件: routers/forum.py, services/forum_service.py │
|
||||
│ 依赖: F.1 │
|
||||
│ 工作量: 3 天 │
|
||||
└────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
F.3 ──────────────────────────────────────────────────────────────┐
|
||||
│ 权限系统 │
|
||||
│ - 用户角色管理 │
|
||||
│ - 板块权限控制 │
|
||||
│ - 操作日志 │
|
||||
│ - 积分/奖励系统 │
|
||||
│ │
|
||||
│ 核心文件: models/user.py, services/forum_service.py │
|
||||
│ 依赖: F.2 │
|
||||
│ 工作量: 3 天 │
|
||||
└────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
F.4 ──────────────────────────────────────────────────────────────┐
|
||||
│ AI 集成 │
|
||||
│ - AI 自动回复 │
|
||||
│ - 智能摘要生成 │
|
||||
│ - 智能分类打标 │
|
||||
│ - Agent 自主发帖 │
|
||||
│ │
|
||||
│ 核心文件: services/forum_ai_service.py │
|
||||
│ 依赖: F.3 │
|
||||
│ 工作量: 5 天 │
|
||||
└────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## VCPToolBox VCP论坛 核心借鉴
|
||||
|
||||
| 借鉴点 | 实现位置 | 难度 |
|
||||
|--------|---------|------|
|
||||
| 文件锁机制（并发控制） | F.2 | 🟡 中 |
|
||||
| 严格输入验证 | F.2 | 🟢 低 |
|
||||
| 安全路径检查 | F.2 | 🟢 低 |
|
||||
| 板块/分类系统 | F.1 | 🟢 低 |
|
||||
| 标签系统 | F.1 | 🟢 低 |
|
||||
| 权限管理 | F.3 | 🟡 中 |
|
||||
| 积分系统 | F.3 | 🟡 中 |
|
||||
| AI 自动回复 | F.4 | 🟡 中 |
|
||||
| Agent 自主发帖 | F.4 | 🟡 中 |
|
||||
|
||||
---
|
||||
|
||||
## 实施顺序
|
||||
|
||||
```
|
||||
F.0 → F.1 → F.2 → F.3 → F.4
|
||||
│ │ │ │ │
|
||||
│ │ │ │ └── AI集成
|
||||
│ │ │ └── 权限/积分
|
||||
│ │ └── API增强/安全
|
||||
│ └── 数据模型
|
||||
└── 现状与目标
|
||||
```
|
||||
|
||||
**注意：** F.1 是基础，后续阶段都依赖 F.1；F.4 需要 Agent 能力成熟。
|
||||
|
||||
---
|
||||
|
||||
## 文件变更追踪
|
||||
|
||||
| Phase | 新增文件 | 修改文件 |
|
||||
|-------|---------|---------|
|
||||
| F.1 | - | `models/forum.py`, `schemas/forum.py` |
|
||||
| F.2 | `services/forum_service.py` | `routers/forum.py` |
|
||||
| F.3 | `models/user.py` (扩展角色), `services/permission_service.py` | `routers/forum.py`, `services/forum_service.py` |
|
||||
| F.4 | `services/forum_ai_service.py`, `services/summary_service.py` | `routers/forum.py`, `agents/` |
|
||||
|
||||
---
|
||||
|
||||
## 与 Agent Phase 1-5 的关系
|
||||
|
||||
| Agent Phase | Forum 协作内容 |
|
||||
|-------------|---------------|
|
||||
| Phase 1 | Task Schema 追踪 Forum 任务 |
|
||||
| Phase 2 | Forum 任务可分解给执行 Agent |
|
||||
| Phase 3 | Agent 可以自主在论坛发帖 |
|
||||
| Phase 4 | Forum 操作可视化 |
|
||||
| Phase 5 | 多 Agent 论坛协作 |
|
||||
| **Phase F** | **Forum 升级路径，与 Phase 1-5 协同** |
|
||||
|
||||
---
|
||||
|
||||
## 注意事项
|
||||
|
||||
| 注意事项 | 说明 |
|
||||
|---------|------|
|
||||
| F.1 是基础 | F.2-F.4 都依赖 F.1 的数据模型 |
|
||||
| 安全第一 | 严格输入验证，防止注入攻击 |
|
||||
| API 兼容性 | 保持现有 API 兼容 |
|
||||
| AI 集成 | F.4 需要 Agent 能力成熟后才能实现 |
|
||||
|
||||
---
|
||||
|
||||
## 总工作量
|
||||
|
||||
| Phase | 工作量 |
|
||||
|-------|--------|
|
||||
| F.1 | 2 天 |
|
||||
| F.2 | 3 天 |
|
||||
| F.3 | 3 天 |
|
||||
| F.4 | 5 天 |
|
||||
| **总计** | **13 天** |
|
||||
305
development-doc/plan/forum-update/checklist.md
Normal file
305
development-doc/plan/forum-update/checklist.md
Normal file
@@ -0,0 +1,305 @@
|
||||
# Forum 升级执行清单
|
||||
|
||||
本清单用于追踪 Forum 升级计划的执行进度。
|
||||
|
||||
---
|
||||
|
||||
## 总进度
|
||||
|
||||
| Phase | 名称 | 状态 | 工作量 |
|
||||
|-------|------|------|--------|
|
||||
| F.0 | 现状与目标 | ✅ 完成 | - |
|
||||
| F.1 | 数据模型升级 | ⬜ 待开始 | 2 天 |
|
||||
| F.2 | API 增强与安全 | ⬜ 待开始 | 3 天 |
|
||||
| F.3 | 权限系统 | ⬜ 待开始 | 3 天 |
|
||||
| F.4 | AI 集成 | ⬜ 待开始 | 5 天 |
|
||||
| **总计** | | | **13 天** |
|
||||
|
||||
---
|
||||
|
||||
## Phase F.1：数据模型升级
|
||||
|
||||
### 目标
|
||||
升级 Forum 数据模型，支持多板块、标签系统、帖子元数据扩展。
|
||||
|
||||
### 任务清单
|
||||
|
||||
#### 数据库迁移
|
||||
- [ ] 创建数据库迁移脚本
|
||||
- 新增 `forum_boards` 表
|
||||
- 新增 `forum_tags` 表
|
||||
- 新增 `forum_post_tags` 表
|
||||
- 新增 `forum_likes` 表
|
||||
- 新增 `forum_stats` 表
|
||||
- 扩展 `forum_posts` 表
|
||||
- 扩展 `forum_replies` 表
|
||||
|
||||
#### 模型实现
|
||||
- [ ] 扩展 `models/forum.py`
|
||||
- [ ] 创建 `ForumBoard` 模型
|
||||
- [ ] 创建 `ForumTag` 模型
|
||||
- [ ] 创建 `ForumPostTag` 模型
|
||||
- [ ] 创建 `ForumLike` 模型
|
||||
- [ ] 创建 `ForumStats` 模型
|
||||
- [ ] 扩展 `ForumPost` 字段
|
||||
- [ ] 扩展 `ForumReply` 字段
|
||||
|
||||
#### Schema 实现
|
||||
- [ ] 扩展 `schemas/forum.py`
|
||||
- [ ] 创建 `ForumBoardCreate/Out` Schema
|
||||
- [ ] 扩展 `ForumPostCreate/Out` Schema
|
||||
- [ ] 扩展 `ForumReplyCreate/Out` Schema
|
||||
- [ ] 创建 `ForumTagOut` Schema
|
||||
|
||||
#### 测试
|
||||
- [ ] 编写单元测试
|
||||
- [ ] 测试 ForumBoard CRUD
|
||||
- [ ] 测试 ForumTag CRUD
|
||||
- [ ] 测试 ForumLike 功能
|
||||
- [ ] 测试 ForumStats 更新
|
||||
|
||||
### 产出文件
|
||||
- `models/forum.py`
|
||||
- `schemas/forum.py`
|
||||
|
||||
### 验收
|
||||
- [ ] 迁移脚本可正常运行
|
||||
- [ ] 所有新模型可正常创建
|
||||
- [ ] Schema 验证正常工作
|
||||
- [ ] 单元测试通过
|
||||
|
||||
---
|
||||
|
||||
## Phase F.2：API 增强与安全
|
||||
|
||||
### 目标
|
||||
增强 Forum API 功能，实现文件锁、输入验证、限流、缓存。
|
||||
|
||||
### 任务清单
|
||||
|
||||
#### 基础服务
|
||||
- [ ] 创建 `services/forum_service.py`
|
||||
- [ ] 实现 `ForumLockManager` 类
|
||||
- [ ] 实现 `ForumService` 类
|
||||
- [ ] 实现板块 CRUD 方法
|
||||
- [ ] 实现帖子 CRUD 方法
|
||||
- [ ] 实现回复 CRUD 方法
|
||||
- [ ] 实现点赞方法
|
||||
|
||||
#### 安全机制
|
||||
- [ ] 实现输入验证
|
||||
- [ ] 实现 `sanitize_input` 函数
|
||||
- [ ] 实现 `validate_post_data` 函数
|
||||
- [ ] 实现配置常量
|
||||
- [ ] 实现限流器
|
||||
- [ ] 实现 `RateLimiter` 类
|
||||
- [ ] 集成发帖限流
|
||||
- [ ] 集成回复限流
|
||||
- [ ] 实现缓存
|
||||
- [ ] 实现 `ForumCache` 类
|
||||
- [ ] 实现帖子缓存
|
||||
- [ ] 实现标签缓存
|
||||
|
||||
#### API 端点
|
||||
- [ ] 扩展 `routers/forum.py`
|
||||
- [ ] GET `/boards` - 列出板块
|
||||
- [ ] POST `/boards` - 创建板块
|
||||
- [ ] GET `/posts` - 分页获取帖子
|
||||
- [ ] POST `/posts` - 创建帖子
|
||||
- [ ] PATCH `/posts/{id}/pin` - 置顶
|
||||
- [ ] PATCH `/posts/{id}/lock` - 锁定
|
||||
- [ ] GET `/tags` - 列出/搜索标签
|
||||
- [ ] POST `/posts/{id}/tags` - 添加标签
|
||||
- [ ] POST `/like` - 切换点赞
|
||||
|
||||
#### 测试
|
||||
- [ ] 编写单元测试
|
||||
- [ ] 测试 ForumLockManager
|
||||
- [ ] 测试输入验证
|
||||
- [ ] 测试限流器
|
||||
- [ ] 测试 API 端点
|
||||
|
||||
### 产出文件
|
||||
- `services/forum_service.py`
|
||||
|
||||
### 验收
|
||||
- [ ] 并发锁正常工作
|
||||
- [ ] 输入验证可过滤危险字符
|
||||
- [ ] 限流正常工作
|
||||
- [ ] 缓存提升读取速度
|
||||
- [ ] 所有 API 端点正常
|
||||
- [ ] 单元测试通过
|
||||
|
||||
---
|
||||
|
||||
## Phase F.3：权限系统
|
||||
|
||||
### 目标
|
||||
实现用户角色管理、板块权限控制、操作日志、积分系统。
|
||||
|
||||
### 任务清单
|
||||
|
||||
#### 用户扩展
|
||||
- [ ] 扩展 `models/user.py`
|
||||
- [ ] 添加 `role` 字段
|
||||
- [ ] 添加 `forum_score` 字段
|
||||
- [ ] 添加论坛统计字段
|
||||
- [ ] 添加禁言相关字段
|
||||
- [ ] 添加 `moderated_boards` 字段
|
||||
|
||||
#### 权限服务
|
||||
- [ ] 创建 `services/permission_service.py`
|
||||
- [ ] 实现 `UserRole` 枚举
|
||||
- [ ] 实现 `Permission` 枚举
|
||||
- [ ] 实现 `ROLE_PERMISSIONS` 映射
|
||||
- [ ] 实现 `PermissionService` 类
|
||||
- [ ] 实现 `has_permission` 方法
|
||||
- [ ] 实现 `can_edit_post` 方法
|
||||
- [ ] 实现 `can_delete_post` 方法
|
||||
- [ ] 实现 `ban_user` 方法
|
||||
- [ ] 实现 `unban_user` 方法
|
||||
|
||||
#### 日志系统
|
||||
- [ ] 扩展 `models/forum.py`
|
||||
- [ ] 创建 `ForumLog` 模型
|
||||
- [ ] 集成日志记录
|
||||
- [ ] 记录帖子操作
|
||||
- [ ] 记录回复操作
|
||||
- [ ] 记录用户操作
|
||||
|
||||
#### 积分服务
|
||||
- [ ] 创建 `services/score_service.py`
|
||||
- [ ] 实现 `SCORE_RULES` 配置
|
||||
- [ ] 实现 `ScoreService` 类
|
||||
- [ ] 实现 `add_score` 方法
|
||||
- [ ] 实现 `get_leaderboard` 方法
|
||||
- [ ] 实现积分自动增减
|
||||
|
||||
#### 管理 API
|
||||
- [ ] 扩展 `routers/forum.py`
|
||||
- [ ] POST `/admin/ban/{user_id}` - 禁言
|
||||
- [ ] POST `/admin/unban/{user_id}` - 解除禁言
|
||||
- [ ] GET `/admin/logs` - 查看日志
|
||||
- [ ] GET `/leaderboard` - 积分排行榜
|
||||
|
||||
#### 测试
|
||||
- [ ] 编写单元测试
|
||||
- [ ] 测试权限检查
|
||||
- [ ] 测试禁言功能
|
||||
- [ ] 测试积分计算
|
||||
- [ ] 测试排行榜
|
||||
|
||||
### 产出文件
|
||||
- `models/user.py`
|
||||
- `models/forum.py` (ForumLog)
|
||||
- `services/permission_service.py`
|
||||
- `services/score_service.py`
|
||||
|
||||
### 验收
|
||||
- [ ] 角色权限正确控制
|
||||
- [ ] 禁言功能正常
|
||||
- [ ] 操作日志正确记录
|
||||
- [ ] 积分正确增减
|
||||
- [ ] 排行榜正确排序
|
||||
- [ ] 单元测试通过
|
||||
|
||||
---
|
||||
|
||||
## Phase F.4：AI 集成
|
||||
|
||||
### 目标
|
||||
实现 AI 自动回复、摘要生成、智能打标、Agent 自主发帖。
|
||||
|
||||
### 任务清单
|
||||
|
||||
#### AI 服务
|
||||
- [ ] 创建 `services/forum_ai_service.py`
|
||||
- [ ] 实现 `ForumAIConfig` 配置
|
||||
- [ ] 实现 `ForumAIService` 类
|
||||
- [ ] 实现 `generate_auto_reply` 方法
|
||||
- [ ] 实现 `_should_auto_reply` 判断
|
||||
- [ ] 实现 `generate_summary` 方法
|
||||
- [ ] 实现 `suggest_tags` 方法
|
||||
- [ ] 实现 `classify_category` 方法
|
||||
|
||||
#### 摘要服务
|
||||
- [ ] 创建 `services/summary_service.py`
|
||||
- [ ] 实现 `SummaryService` 类
|
||||
- [ ] 实现 `get_post_summary` 方法
|
||||
- [ ] 实现 `get_thread_summary` 方法
|
||||
- [ ] 实现缓存失效
|
||||
|
||||
#### Agent 工具
|
||||
- [ ] 创建 `agents/tools/forum_tools.py`
|
||||
- [ ] 实现 `create_forum_post` 工具
|
||||
- [ ] 实现 `reply_to_post` 工具
|
||||
- [ ] 实现 `search_forum_posts` 工具
|
||||
- [ ] 实现 `get_forum_trending` 工具
|
||||
- [ ] 创建 `agents/prompts/forum_agent.py`
|
||||
- [ ] 编写 Forum Agent 提示词
|
||||
- [ ] 配置工具列表
|
||||
|
||||
#### 定时任务
|
||||
- [ ] 创建 `tasks/forum_auto_reply.py`
|
||||
- [ ] 实现 `auto_reply_task` 函数
|
||||
- [ ] 实现 `setup_forum_scheduler` 函数
|
||||
|
||||
#### API 端点
|
||||
- [ ] 扩展 `routers/forum.py`
|
||||
- [ ] POST `/posts/{id}/generate-summary` - 生成摘要
|
||||
- [ ] POST `/posts/suggest-tags` - 推荐标签
|
||||
- [ ] POST `/posts/classify` - 分类帖子
|
||||
- [ ] GET `/posts/{id}/ai-status` - AI 状态
|
||||
|
||||
#### 测试
|
||||
- [ ] 编写单元测试
|
||||
- [ ] 测试自动回复生成
|
||||
- [ ] 测试摘要生成
|
||||
- [ ] 测试智能打标
|
||||
- [ ] 测试 API 端点
|
||||
|
||||
### 产出文件
|
||||
- `services/forum_ai_service.py`
|
||||
- `services/summary_service.py`
|
||||
- `agents/tools/forum_tools.py`
|
||||
- `agents/prompts/forum_agent.py`
|
||||
- `tasks/forum_auto_reply.py`
|
||||
|
||||
### 验收
|
||||
- [ ] AI 服务正常调用
|
||||
- [ ] 自动回复正常生成
|
||||
- [ ] 摘要生成正常
|
||||
- [ ] 智能打标推荐
|
||||
- [ ] 智能分类推荐
|
||||
- [ ] Agent 工具可用
|
||||
- [ ] 定时任务正常执行
|
||||
- [ ] 单元测试通过
|
||||
|
||||
---
|
||||
|
||||
## 完成标准
|
||||
|
||||
- [ ] 所有 Phase F.1-F.4 任务完成
|
||||
- [ ] 所有单元测试通过
|
||||
- [ ] API 文档更新完成
|
||||
- [ ] 用户文档更新完成
|
||||
- [ ] 部署验证通过
|
||||
|
||||
---
|
||||
|
||||
## 风险与注意事项
|
||||
|
||||
| 风险 | 影响 | 缓解措施 |
|
||||
|------|------|----------|
|
||||
| F.4 AI 调用成本高 | 费用增加 | 设置每日调用上限 |
|
||||
| 并发锁死锁 | 服务不可用 | 超时机制 + 定期清理 |
|
||||
| 迁移数据丢失 | 用户流失 | 备份 + 分阶段迁移 |
|
||||
| Agent 刷屏 | 用户体验下降 | 严格限流 + 人工审核 |
|
||||
|
||||
---
|
||||
|
||||
## 更新日志
|
||||
|
||||
| 日期 | Phase | 变更内容 |
|
||||
|------|-------|----------|
|
||||
| 2026-04-04 | F.0 | 创建文档 |
|
||||
205
development-doc/plan/forum-update/phase-f-0-current-state.md
Normal file
205
development-doc/plan/forum-update/phase-f-0-current-state.md
Normal file
@@ -0,0 +1,205 @@
|
||||
# Phase F.0：Forum 现状与目标
|
||||
|
||||
日期：2026-04-04
|
||||
状态：已完成
|
||||
借鉴来源：VCPToolBox VCP论坛 模块
|
||||
|
||||
---
|
||||
|
||||
## 1. 本阶段目的
|
||||
|
||||
本文件用于统一背景认知，明确：
|
||||
|
||||
- Jarvis 当前 Forum 架构处于什么水平
|
||||
- 主要短板是什么
|
||||
- 为什么要升级
|
||||
- 升级后的目标形态是什么
|
||||
- VCPToolBox VCP论坛 给我们什么启发
|
||||
|
||||
---
|
||||
|
||||
## 2. 当前 Jarvis Forum 架构
|
||||
|
||||
### 2.1 核心流程
|
||||
|
||||
```
|
||||
用户发帖/回复 → ForumRouter → ForumPost/ForumReply 模型 → SQLite
|
||||
```
|
||||
|
||||
### 2.2 核心文件
|
||||
|
||||
| 文件 | 职责 |
|
||||
|------|------|
|
||||
| `backend/app/routers/forum.py` | 论坛 API 路由 |
|
||||
| `backend/app/models/forum.py` | ForumPost/ForumReply 数据模型 |
|
||||
| `backend/app/schemas/forum.py` | Pydantic 请求/响应模型 |
|
||||
|
||||
### 2.3 当前数据模型
|
||||
|
||||
```python
|
||||
class ForumPost(BaseModel):
|
||||
user_id: str
|
||||
title: str
|
||||
content: str
|
||||
category: str # instruction, discussion, question
|
||||
is_executed: bool
|
||||
execution_result: Optional[str]
|
||||
reply_count: int
|
||||
|
||||
class ForumReply(BaseModel):
|
||||
post_id: str
|
||||
user_id: Optional[str]
|
||||
agent_id: Optional[str]
|
||||
content: str
|
||||
is_ai_reply: bool
|
||||
```
|
||||
|
||||
### 2.4 当前 API 端点
|
||||
|
||||
| 方法 | 路径 | 功能 |
|
||||
|------|------|------|
|
||||
| GET | `/api/forum/posts` | 列出帖子 |
|
||||
| POST | `/api/forum/posts` | 创建帖子 |
|
||||
| GET | `/api/forum/posts/{post_id}` | 获取帖子 |
|
||||
| DELETE | `/api/forum/posts/{post_id}` | 删除帖子 |
|
||||
| GET | `/api/forum/posts/{post_id}/replies` | 列出回复 |
|
||||
| POST | `/api/forum/posts/{post_id}/replies` | 创建回复 |
|
||||
|
||||
---
|
||||
|
||||
## 3. 当前能力矩阵
|
||||
|
||||
| 能力 | 状态 | 说明 |
|
||||
|------|------|------|
|
||||
| 基本 CRUD | ✅ | 帖子和回复的增删查 |
|
||||
| 分类标签 | ✅ | instruction/discussion/question |
|
||||
| AI 回复标记 | ✅ | `is_ai_reply` 字段 |
|
||||
| 回复计数 | ✅ | `reply_count` 字段 |
|
||||
| 执行状态 | ✅ | `is_executed`/`execution_result` |
|
||||
|
||||
---
|
||||
|
||||
## 4. 当前短板
|
||||
|
||||
| 短板 | 严重程度 | 影响 |
|
||||
|------|----------|------|
|
||||
| 无板块系统 | 🔴 高 | 所有帖子混在一起 |
|
||||
| 无标签系统 | 🟡 中 | 无法细粒度分类 |
|
||||
| 无输入验证 | 🔴 高 | 安全风险 |
|
||||
| 无并发控制 | 🟡 中 | 多用户操作可能冲突 |
|
||||
| 无权限管理 | 🟡 中 | 无法控制谁可以做什么 |
|
||||
| 无积分系统 | 🟡 中 | 无法激励参与 |
|
||||
| 无 AI 自动回复 | 🟡 中 | 需要人工回复 |
|
||||
| 无摘要生成 | 🟢 低 | 长帖子难以快速浏览 |
|
||||
| 无 Agent 自主发帖 | 🟢 低 | Agent 无法主动参与 |
|
||||
|
||||
---
|
||||
|
||||
## 5. VCPToolBox VCP论坛 核心借鉴
|
||||
|
||||
### 5.1 VCP论坛架构
|
||||
|
||||
VCPToolBox 的论坛是 Agent 社区交流平台，支持：
|
||||
- 多 Agent 和用户共同参与
|
||||
- 超栈追踪和统一 FileAPI
|
||||
- 文件上传和多媒体支持
|
||||
- 积分系统和任务版集成
|
||||
|
||||
### 5.2 核心安全机制
|
||||
|
||||
```javascript
|
||||
// forumApi.js 中的安全配置
|
||||
const FORUM_CONFIG = {
|
||||
MAX_CONTENT_LENGTH: 50000, // 单条内容最大长度 50KB
|
||||
MAX_FILE_SIZE: 1024 * 1024 * 2, // 单个帖子文件最大 2MB
|
||||
MAX_MAID_LENGTH: 50, // 用户名最大长度
|
||||
MAX_TITLE_LENGTH: 100, // 标题最大长度
|
||||
MAX_FLOORS_PER_POST: 500, // 单帖最大楼层数
|
||||
UID_PATTERN: /^[a-zA-Z0-9_-]+$/, // UID 允许的字符
|
||||
LOCK_TIMEOUT: 10000, // 文件锁超时 10秒
|
||||
MAX_CONCURRENT_WRITES: 5, // 最大并发写入数
|
||||
};
|
||||
```
|
||||
|
||||
### 5.3 核心安全函数
|
||||
|
||||
```javascript
|
||||
// 文件锁管理器 - 并发控制
|
||||
class FileLockManager {
|
||||
async acquireLock(filePath, timeout)
|
||||
releaseLock(filePath)
|
||||
}
|
||||
|
||||
// 安全路径检查 - 防止路径遍历
|
||||
function isPathSafe(targetPath, rootPath)
|
||||
|
||||
// 输入清理 - 防止注入
|
||||
function sanitizeInput(input, maxLength)
|
||||
|
||||
// 安全的文件名解析
|
||||
function parsePostFilename(filename)
|
||||
```
|
||||
|
||||
### 5.4 关键设计理念
|
||||
|
||||
1. **安全第一** - 严格的输入验证、路径安全、并发控制
|
||||
2. **可扩展性** - 支持多板块、多楼层、文件附件
|
||||
3. **Agent 集成** - Agent 可以自主发帖、回帖
|
||||
4. **积分激励** - Agent 通过发帖获得积分
|
||||
|
||||
---
|
||||
|
||||
## 6. 目标架构
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ User/Agent │
|
||||
└─────────────────────────┬───────────────────────────────────┘
|
||||
│
|
||||
┌─────────────────────────┴───────────────────────────────────┐
|
||||
│ Forum Router Layer │
|
||||
│ - 输入验证 (sanitize) │
|
||||
│ - 权限检查 (permission) │
|
||||
│ - 限流控制 (rate_limit) │
|
||||
└─────────────────────────┬───────────────────────────────────┘
|
||||
│
|
||||
┌─────────────────────────┴───────────────────────────────────┐
|
||||
│ Forum Service Layer │
|
||||
│ - FileLock (并发控制) │
|
||||
│ - PermissionService (权限) │
|
||||
│ - AIService (AI 回复/摘要) │
|
||||
└─────────────────────────┬───────────────────────────────────┘
|
||||
│
|
||||
┌─────────────────────────┴───────────────────────────────────┐
|
||||
│ Data Layer │
|
||||
│ - ForumPost (板块/标签) │
|
||||
│ - ForumReply (层级) │
|
||||
│ - UserRole (权限) │
|
||||
│ - ForumStats (积分) │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. 借鉴点映射
|
||||
|
||||
| VCPToolBox 借鉴点 | Jarvis 实现位置 | 优先级 |
|
||||
|-------------------|---------------|--------|
|
||||
| 文件锁机制 | `services/forum_service.py` | 🟢 高 |
|
||||
| 输入验证强化 | `routers/forum.py` | 🟢 高 |
|
||||
| 安全路径检查 | `services/forum_service.py` | 🟢 高 |
|
||||
| 板块/分类系统 | `models/forum.py` | 🟢 高 |
|
||||
| 标签系统 | `models/forum.py` | 🟡 中 |
|
||||
| 权限管理 | `services/permission_service.py` | 🟡 中 |
|
||||
| 积分系统 | `models/user.py` | 🟡 中 |
|
||||
| AI 自动回复 | `services/forum_ai_service.py` | 🟡 中 |
|
||||
| 摘要生成 | `services/summary_service.py` | 🟢 低 |
|
||||
| Agent 自主发帖 | `agents/` | 🟡 中 |
|
||||
|
||||
---
|
||||
|
||||
## 8. 本阶段产出要求
|
||||
|
||||
- [x] 团队对 Jarvis 当前 Forum 问题和目标方向达成一致
|
||||
- [x] VCPToolBox VCP论坛 借鉴点已映射到具体 Phase
|
||||
- [x] 后续 phase 文档能够在这个认知基础上展开
|
||||
361
development-doc/plan/forum-update/phase-f-1-data-model.md
Normal file
361
development-doc/plan/forum-update/phase-f-1-data-model.md
Normal file
@@ -0,0 +1,361 @@
|
||||
# Phase F.1：数据模型升级
|
||||
|
||||
日期：2026-04-04
|
||||
状态：待开始
|
||||
依赖：F.0（已完成）
|
||||
前置：models/forum.py
|
||||
|
||||
---
|
||||
|
||||
## 1. 本阶段目的
|
||||
|
||||
升级 Jarvis Forum 的数据模型，支持：
|
||||
- 多板块系统（指令/讨论/任务等）
|
||||
- 标签系统（细粒度分类）
|
||||
- 帖子元数据扩展（阅读量、点赞、收藏等）
|
||||
- 层级回复结构
|
||||
|
||||
---
|
||||
|
||||
## 2. 目标模型
|
||||
|
||||
### 2.1 ForumBoard（板块）
|
||||
|
||||
```python
|
||||
class ForumBoard(BaseModel):
|
||||
__tablename__ = "forum_boards"
|
||||
|
||||
id: str = Field(default_factory=lambda: str(uuid.uuid4()))
|
||||
name: str # 板块名称，如 "指令广场"
|
||||
slug: str # URL 友好名称，如 "instruction"
|
||||
description: Optional[str]
|
||||
icon: Optional[str] # emoji 或图标名
|
||||
sort_order: int = 0
|
||||
is_active: bool = True
|
||||
min_role: str = "user" # 查看权限：guest/user/moderator/admin
|
||||
|
||||
parent_id: Optional[str] # 父板块，支持层级
|
||||
color: Optional[str] # 板块颜色
|
||||
```
|
||||
|
||||
### 2.2 ForumPost（帖子）扩展
|
||||
|
||||
```python
|
||||
class ForumPost(BaseModel):
|
||||
__tablename__ = "forum_posts"
|
||||
|
||||
# === 现有字段 ===
|
||||
user_id: str
|
||||
title: str
|
||||
content: str
|
||||
reply_count: int = 0
|
||||
|
||||
# === 新增字段 ===
|
||||
board_id: Optional[str] = None # 所属板块
|
||||
parent_id: Optional[str] = None # 父帖子（支持回复嵌套）
|
||||
|
||||
# 分类与标签
|
||||
category: Optional[str] # instruction/discussion/question/praise/bug
|
||||
tags: Optional[str] = "" # JSON 数组 ["python", "api"]
|
||||
|
||||
# 状态与权限
|
||||
is_pinned: bool = False # 置顶
|
||||
is_locked: bool = False # 锁定（禁止回复）
|
||||
is_deleted: bool = False # 软删除
|
||||
is_executed: bool = False # 指令已执行
|
||||
execution_result: Optional[str]
|
||||
|
||||
# 统计与互动
|
||||
view_count: int = 0
|
||||
like_count: int = 0
|
||||
favorite_count: int = 0
|
||||
share_count: int = 0
|
||||
|
||||
# 时间戳
|
||||
last_reply_at: Optional[datetime] = None # 最后回复时间
|
||||
last_activity_at: Optional[datetime] = None
|
||||
|
||||
# === 关系 ===
|
||||
board = relationship("ForumBoard", back_populates="posts")
|
||||
replies = relationship("ForumReply", back_populates="post", cascade="all, delete-orphan")
|
||||
post_tags = relationship("ForumPostTag", back_populates="post", cascade="all, delete-orphan")
|
||||
likes = relationship("ForumLike", back_populates="post", cascade="all, delete-orphan")
|
||||
```
|
||||
|
||||
### 2.3 ForumReply（回复）扩展
|
||||
|
||||
```python
|
||||
class ForumReply(BaseModel):
|
||||
__tablename__ = "forum_replies"
|
||||
|
||||
# === 现有字段 ===
|
||||
post_id: str
|
||||
user_id: Optional[str]
|
||||
agent_id: Optional[str]
|
||||
content: str
|
||||
is_ai_reply: bool = False
|
||||
|
||||
# === 新增字段 ===
|
||||
parent_id: Optional[str] = None # 父回复（支持嵌套）
|
||||
floor: int = 1 # 楼层号
|
||||
is_pinned: bool = False # 置顶
|
||||
is_best: bool = False # 最佳回复
|
||||
is_deleted: bool = False # 软删除
|
||||
like_count: int = 0
|
||||
edited_at: Optional[datetime] = None # 编辑时间
|
||||
|
||||
# === 关系 ===
|
||||
post = relationship("ForumPost", back_populates="replies")
|
||||
parent = relationship("ForumReply", remote_side=[id], backref="children")
|
||||
likes = relationship("ForumLike", back_populates="reply", cascade="all, delete-orphan")
|
||||
```
|
||||
|
||||
### 2.4 ForumTag（标签）
|
||||
|
||||
```python
|
||||
class ForumTag(BaseModel):
|
||||
__tablename__ = "forum_tags"
|
||||
|
||||
id: str = Field(default_factory=lambda: str(uuid.uuid4()))
|
||||
name: str # 标签名，如 "python"
|
||||
slug: str # URL 友好名
|
||||
color: str = "#666666" # 标签颜色
|
||||
post_count: int = 0 # 帖子数量（缓存）
|
||||
is_official: bool = False # 官方标签
|
||||
board_id: Optional[str] = None # 限定板块
|
||||
|
||||
posts = relationship("ForumPostTag", back_populates="tag")
|
||||
|
||||
|
||||
class ForumPostTag(BaseModel):
|
||||
"""帖子-标签关联"""
|
||||
__tablename__ = "forum_post_tags"
|
||||
|
||||
post_id: str
|
||||
tag_id: str
|
||||
|
||||
post = relationship("ForumPost", back_populates="post_tags")
|
||||
tag = relationship("ForumTag", back_populates="posts")
|
||||
```
|
||||
|
||||
### 2.5 ForumLike（点赞）
|
||||
|
||||
```python
|
||||
class ForumLike(BaseModel):
|
||||
__tablename__ = "forum_likes"
|
||||
|
||||
user_id: str
|
||||
post_id: Optional[str] = None
|
||||
reply_id: Optional[str] = None
|
||||
created_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
|
||||
post = relationship("ForumPost", back_populates="likes")
|
||||
reply = relationship("ForumReply", back_populates="likes")
|
||||
```
|
||||
|
||||
### 2.6 ForumStats（用户统计）
|
||||
|
||||
```python
|
||||
class ForumStats(BaseModel):
|
||||
"""用户论坛统计"""
|
||||
__tablename__ = "forum_stats"
|
||||
|
||||
user_id: str = Field(primary_key=True)
|
||||
post_count: int = 0
|
||||
reply_count: int = 0
|
||||
like_received: int = 0 # 收到的赞
|
||||
best_reply_count: int = 0 # 最佳回复数
|
||||
score: int = 0 # 积分
|
||||
|
||||
last_post_at: Optional[datetime] = None
|
||||
last_reply_at: Optional[datetime] = None
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. 数据库迁移
|
||||
|
||||
### 3.1 新增表
|
||||
|
||||
```sql
|
||||
-- 板块表
|
||||
CREATE TABLE forum_boards (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
slug TEXT UNIQUE NOT NULL,
|
||||
description TEXT,
|
||||
icon TEXT,
|
||||
sort_order INTEGER DEFAULT 0,
|
||||
is_active BOOLEAN DEFAULT TRUE,
|
||||
min_role TEXT DEFAULT 'user',
|
||||
parent_id TEXT,
|
||||
color TEXT
|
||||
);
|
||||
|
||||
-- 帖子-标签关联表
|
||||
CREATE TABLE forum_post_tags (
|
||||
post_id TEXT NOT NULL,
|
||||
tag_id TEXT NOT NULL,
|
||||
PRIMARY KEY (post_id, tag_id)
|
||||
);
|
||||
|
||||
-- 标签表
|
||||
CREATE TABLE forum_tags (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL UNIQUE,
|
||||
slug TEXT UNIQUE NOT NULL,
|
||||
color TEXT DEFAULT '#666666',
|
||||
post_count INTEGER DEFAULT 0,
|
||||
is_official BOOLEAN DEFAULT FALSE,
|
||||
board_id TEXT
|
||||
);
|
||||
|
||||
-- 点赞表
|
||||
CREATE TABLE forum_likes (
|
||||
user_id TEXT NOT NULL,
|
||||
post_id TEXT,
|
||||
reply_id TEXT,
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
PRIMARY KEY (user_id, post_id, reply_id)
|
||||
);
|
||||
|
||||
-- 用户论坛统计表
|
||||
CREATE TABLE forum_stats (
|
||||
user_id TEXT PRIMARY KEY,
|
||||
post_count INTEGER DEFAULT 0,
|
||||
reply_count INTEGER DEFAULT 0,
|
||||
like_received INTEGER DEFAULT 0,
|
||||
best_reply_count INTEGER DEFAULT 0,
|
||||
score INTEGER DEFAULT 0,
|
||||
last_post_at TIMESTAMP,
|
||||
last_reply_at TIMESTAMP
|
||||
);
|
||||
|
||||
-- 扩展 forum_posts 表
|
||||
ALTER TABLE forum_posts ADD COLUMN board_id TEXT;
|
||||
ALTER TABLE forum_posts ADD COLUMN parent_id TEXT;
|
||||
ALTER TABLE forum_posts ADD COLUMN tags TEXT DEFAULT '';
|
||||
ALTER TABLE forum_posts ADD COLUMN is_pinned BOOLEAN DEFAULT FALSE;
|
||||
ALTER TABLE forum_posts ADD COLUMN is_locked BOOLEAN DEFAULT FALSE;
|
||||
ALTER TABLE forum_posts ADD COLUMN is_deleted BOOLEAN DEFAULT FALSE;
|
||||
ALTER TABLE forum_posts ADD COLUMN view_count INTEGER DEFAULT 0;
|
||||
ALTER TABLE forum_posts ADD COLUMN like_count INTEGER DEFAULT 0;
|
||||
ALTER TABLE forum_posts ADD COLUMN favorite_count INTEGER DEFAULT 0;
|
||||
ALTER TABLE forum_posts ADD COLUMN share_count INTEGER DEFAULT 0;
|
||||
ALTER TABLE forum_posts ADD COLUMN last_reply_at TIMESTAMP;
|
||||
ALTER TABLE forum_posts ADD COLUMN last_activity_at TIMESTAMP;
|
||||
|
||||
-- 扩展 forum_replies 表
|
||||
ALTER TABLE forum_replies ADD COLUMN parent_id TEXT;
|
||||
ALTER TABLE forum_replies ADD COLUMN floor INTEGER DEFAULT 1;
|
||||
ALTER TABLE forum_replies ADD COLUMN is_pinned BOOLEAN DEFAULT FALSE;
|
||||
ALTER TABLE forum_replies ADD COLUMN is_best BOOLEAN DEFAULT FALSE;
|
||||
ALTER TABLE forum_replies ADD COLUMN is_deleted BOOLEAN DEFAULT FALSE;
|
||||
ALTER TABLE forum_replies ADD COLUMN like_count INTEGER DEFAULT 0;
|
||||
ALTER TABLE forum_replies ADD COLUMN edited_at TIMESTAMP;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Schema 扩展
|
||||
|
||||
### 4.1 ForumBoardSchema
|
||||
|
||||
```python
|
||||
class ForumBoardCreate(BaseModel):
|
||||
name: str = Field(..., max_length=100)
|
||||
slug: str = Field(..., max_length=50, pattern=r"^[a-z0-9-]+$")
|
||||
description: Optional[str] = None
|
||||
icon: Optional[str] = None
|
||||
sort_order: int = 0
|
||||
min_role: str = "user"
|
||||
parent_id: Optional[str] = None
|
||||
color: Optional[str] = None
|
||||
|
||||
|
||||
class ForumBoardOut(BaseModel):
|
||||
id: str
|
||||
name: str
|
||||
slug: str
|
||||
description: Optional[str]
|
||||
icon: Optional[str]
|
||||
sort_order: int
|
||||
post_count: int = 0 # 动态计算
|
||||
|
||||
|
||||
class ForumPostCreate(BaseModel):
|
||||
title: str = Field(..., max_length=200)
|
||||
content: str = Field(..., max_length=50000)
|
||||
board_id: Optional[str] = None
|
||||
category: Optional[str] = None
|
||||
tags: List[str] = []
|
||||
|
||||
|
||||
class ForumPostOut(BaseModel):
|
||||
id: str
|
||||
user_id: str
|
||||
title: str
|
||||
content: str
|
||||
board_id: Optional[str]
|
||||
board_name: Optional[str] = None
|
||||
category: Optional[str]
|
||||
tags: List[str] = []
|
||||
reply_count: int
|
||||
view_count: int
|
||||
like_count: int
|
||||
is_pinned: bool
|
||||
is_locked: bool
|
||||
created_at: datetime
|
||||
last_reply_at: Optional[datetime]
|
||||
user_liked: bool = False # 当前用户是否点赞
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. 实现步骤
|
||||
|
||||
| 步骤 | 任务 | 优先级 |
|
||||
|------|------|--------|
|
||||
| 1 | 创建数据库迁移脚本 | 🟢 高 |
|
||||
| 2 | 扩展 models/forum.py | 🟢 高 |
|
||||
| 3 | 创建 ForumBoard 模型 | 🟢 高 |
|
||||
| 4 | 创建 ForumTag/ForumPostTag 模型 | 🟡 中 |
|
||||
| 5 | 创建 ForumLike 模型 | 🟡 中 |
|
||||
| 6 | 创建 ForumStats 模型 | 🟡 中 |
|
||||
| 7 | 扩展 schemas/forum.py | 🟢 高 |
|
||||
| 8 | 编写单元测试 | 🟡 中 |
|
||||
|
||||
---
|
||||
|
||||
## 6. 核心文件变更
|
||||
|
||||
| 文件 | 变更 |
|
||||
|------|------|
|
||||
| `models/forum.py` | 新增 ForumBoard, ForumTag, ForumPostTag, ForumLike, ForumStats；扩展 ForumPost, ForumReply |
|
||||
| `schemas/forum.py` | 新增 ForumBoardCreate/Out；扩展 ForumPostCreate/Out |
|
||||
| `database.py` | 注册新模型 |
|
||||
|
||||
---
|
||||
|
||||
## 7. 工作量估算
|
||||
|
||||
| 任务 | 工作量 |
|
||||
|------|--------|
|
||||
| 数据库迁移脚本 | 0.5 天 |
|
||||
| 模型扩展 | 0.5 天 |
|
||||
| Schema 扩展 | 0.5 天 |
|
||||
| 单元测试 | 0.5 天 |
|
||||
| **总计** | **2 天** |
|
||||
|
||||
---
|
||||
|
||||
## 8. 验收标准
|
||||
|
||||
- [ ] ForumBoard 模型可创建/查询板块
|
||||
- [ ] ForumPost 可关联板块和标签
|
||||
- [ ] ForumTag 可创建/查询标签
|
||||
- [ ] ForumLike 支持点赞功能
|
||||
- [ ] ForumStats 正确统计用户数据
|
||||
- [ ] 所有新模型有对应的 Pydantic Schema
|
||||
- [ ] 迁移脚本可回滚
|
||||
- [ ] 单元测试覆盖新增功能
|
||||
488
development-doc/plan/forum-update/phase-f-2-forum-api.md
Normal file
488
development-doc/plan/forum-update/phase-f-2-forum-api.md
Normal file
@@ -0,0 +1,488 @@
|
||||
# Phase F.2：API 增强与安全
|
||||
|
||||
日期：2026-04-04
|
||||
状态：待开始
|
||||
依赖：F.1（待完成）
|
||||
前置：routers/forum.py
|
||||
|
||||
---
|
||||
|
||||
## 1. 本阶段目的
|
||||
|
||||
增强 Jarvis Forum API 的功能和安全：
|
||||
|
||||
- 引入文件锁机制（并发控制）
|
||||
- 强化输入验证
|
||||
- 扩展 API 端点
|
||||
- 实现缓存机制
|
||||
- 添加限流保护
|
||||
|
||||
---
|
||||
|
||||
## 2. Forum Service 架构
|
||||
|
||||
### 2.1 目录结构
|
||||
|
||||
```
|
||||
backend/app/services/
|
||||
└── forum_service.py # 新增
|
||||
```
|
||||
|
||||
### 2.2 核心服务类
|
||||
|
||||
```python
|
||||
class ForumLockManager:
|
||||
"""文件锁管理器 - 防止并发写入冲突"""
|
||||
|
||||
def __init__(self, max_concurrent: int = 5, timeout: int = 10):
|
||||
self.locks: Dict[str, LockInfo] = {}
|
||||
self.max_concurrent = max_concurrent
|
||||
self.timeout = timeout
|
||||
|
||||
async def acquire_lock(self, resource_id: str) -> bool:
|
||||
"""获取锁"""
|
||||
|
||||
def release_lock(self, resource_id: str) -> None:
|
||||
"""释放锁"""
|
||||
|
||||
def cleanup_stale_locks(self) -> None:
|
||||
"""清理过期锁"""
|
||||
|
||||
|
||||
class ForumService:
|
||||
"""论坛服务主类"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
db: AsyncSession,
|
||||
lock_manager: ForumLockManager,
|
||||
):
|
||||
self.db = db
|
||||
self.lock = lock_manager
|
||||
|
||||
# === 板块操作 ===
|
||||
async def create_board(self, data: ForumBoardCreate) -> ForumBoard:
|
||||
"""创建板块"""
|
||||
|
||||
async def list_boards(self) -> List[ForumBoard]:
|
||||
"""列出板块"""
|
||||
|
||||
# === 帖子操作 ===
|
||||
async def create_post(
|
||||
self,
|
||||
user_id: str,
|
||||
data: ForumPostCreate,
|
||||
) -> ForumPost:
|
||||
"""创建帖子（带锁）"""
|
||||
async with self.lock.acquire_lock(f"post:{user_id}"):
|
||||
# 创建帖子逻辑
|
||||
|
||||
async def get_posts(
|
||||
self,
|
||||
board_id: Optional[str] = None,
|
||||
category: Optional[str] = None,
|
||||
tags: Optional[List[str]] = None,
|
||||
page: int = 1,
|
||||
page_size: int = 20,
|
||||
) -> PaginatedResult[ForumPostOut]:
|
||||
"""分页获取帖子"""
|
||||
|
||||
async def get_post(self, post_id: str, user_id: Optional[str] = None) -> ForumPostOut:
|
||||
"""获取帖子详情（增加浏览量）"""
|
||||
|
||||
async def update_post(
|
||||
self,
|
||||
post_id: str,
|
||||
user_id: str,
|
||||
data: ForumPostUpdate,
|
||||
) -> ForumPost:
|
||||
"""更新帖子"""
|
||||
|
||||
async def delete_post(self, post_id: str, user_id: str) -> None:
|
||||
"""删除帖子（软删除）"""
|
||||
|
||||
async def pin_post(self, post_id: str, is_pinned: bool) -> None:
|
||||
"""置顶/取消置顶"""
|
||||
|
||||
async def lock_post(self, post_id: str, is_locked: bool) -> None:
|
||||
"""锁定/解锁帖子"""
|
||||
|
||||
# === 回复操作 ===
|
||||
async def create_reply(
|
||||
self,
|
||||
post_id: str,
|
||||
user_id: Optional[str],
|
||||
data: ForumReplyCreate,
|
||||
) -> ForumReply:
|
||||
"""创建回复（带锁）"""
|
||||
|
||||
async def get_replies(
|
||||
self,
|
||||
post_id: str,
|
||||
page: int = 1,
|
||||
page_size: int = 50,
|
||||
) -> PaginatedResult[ForumReplyOut]:
|
||||
"""获取回复列表"""
|
||||
|
||||
async def update_reply(
|
||||
self,
|
||||
reply_id: str,
|
||||
user_id: str,
|
||||
content: str,
|
||||
) -> ForumReply:
|
||||
"""更新回复"""
|
||||
|
||||
async def delete_reply(self, reply_id: str, user_id: str) -> None:
|
||||
"""删除回复"""
|
||||
|
||||
# === 点赞操作 ===
|
||||
async def toggle_like(
|
||||
self,
|
||||
user_id: str,
|
||||
post_id: Optional[str] = None,
|
||||
reply_id: Optional[str] = None,
|
||||
) -> bool:
|
||||
"""切换点赞状态"""
|
||||
|
||||
# === 标签操作 ===
|
||||
async def create_tag(self, name: str, color: str = "#666666") -> ForumTag:
|
||||
"""创建标签"""
|
||||
|
||||
async def search_tags(self, query: str) -> List[ForumTag]:
|
||||
"""搜索标签"""
|
||||
|
||||
async def add_tags_to_post(
|
||||
self,
|
||||
post_id: str,
|
||||
user_id: str,
|
||||
tags: List[str],
|
||||
) -> None:
|
||||
"""为帖子添加标签"""
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. 安全机制
|
||||
|
||||
### 3.1 输入验证
|
||||
|
||||
```python
|
||||
# Forum 安全配置
|
||||
FORUM_CONFIG = {
|
||||
"MAX_CONTENT_LENGTH": 50000, # 内容最大 50KB
|
||||
"MAX_TITLE_LENGTH": 200, # 标题最大 200 字符
|
||||
"MAX_TAGS_PER_POST": 10, # 每帖最多标签
|
||||
"MAX_REPLIES_PER_POST": 500, # 每帖最多回复
|
||||
"MAX_POSTS_PER_USER_PER_DAY": 50, # 每人每天最多发帖
|
||||
"MAX_REPLIES_PER_USER_PER_DAY": 200, # 每人每天最多回复
|
||||
}
|
||||
|
||||
|
||||
def sanitize_input(text: str, max_length: int) -> str:
|
||||
"""清理用户输入"""
|
||||
if not text or not isinstance(text, str):
|
||||
return ""
|
||||
# 移除控制字符
|
||||
text = re.sub(r'[\x00-\x08\x0B\x0C\x0E-\x1F\x7F]', '', text)
|
||||
# 限制长度
|
||||
return text[:max_length]
|
||||
|
||||
|
||||
def validate_post_data(data: ForumPostCreate) -> ForumPostCreate:
|
||||
"""验证帖子数据"""
|
||||
# 标题验证
|
||||
data.title = sanitize_input(data.title, FORUM_CONFIG["MAX_TITLE_LENGTH"])
|
||||
if len(data.title.strip()) < 3:
|
||||
raise ValueError("标题至少 3 个字符")
|
||||
|
||||
# 内容验证
|
||||
data.content = sanitize_input(data.content, FORUM_CONFIG["MAX_CONTENT_LENGTH"])
|
||||
if len(data.content.strip()) < 10:
|
||||
raise ValueError("内容至少 10 个字符")
|
||||
|
||||
# 标签验证
|
||||
if len(data.tags) > FORUM_CONFIG["MAX_TAGS_PER_POST"]:
|
||||
raise ValueError(f"最多 {FORUM_CONFIG['MAX_TAGS_PER_POST']} 个标签")
|
||||
data.tags = [sanitize_input(t, 50) for t in data.tags]
|
||||
|
||||
return data
|
||||
```
|
||||
|
||||
### 3.2 并发控制
|
||||
|
||||
```python
|
||||
class RateLimiter:
|
||||
"""简单的限流器"""
|
||||
|
||||
def __init__(self):
|
||||
self.user_requests: Dict[str, List[datetime]] = defaultdict(list)
|
||||
|
||||
def check_rate_limit(
|
||||
self,
|
||||
user_id: str,
|
||||
action: str,
|
||||
max_requests: int,
|
||||
window_seconds: int = 86400,
|
||||
) -> bool:
|
||||
"""检查是否超过限流"""
|
||||
now = datetime.utcnow()
|
||||
cutoff = now - timedelta(seconds=window_seconds)
|
||||
|
||||
# 清理过期记录
|
||||
self.user_requests[user_id] = [
|
||||
t for t in self.user_requests[user_id] if t > cutoff
|
||||
]
|
||||
|
||||
if len(self.user_requests[user_id]) >= max_requests:
|
||||
return False
|
||||
|
||||
self.user_requests[user_id].append(now)
|
||||
return True
|
||||
```
|
||||
|
||||
### 3.3 缓存策略
|
||||
|
||||
```python
|
||||
from functools import lru_cache
|
||||
from typing import Optional
|
||||
import json
|
||||
|
||||
|
||||
class ForumCache:
|
||||
"""论坛缓存"""
|
||||
|
||||
def __init__(self):
|
||||
self.cache: Dict[str, CacheEntry] = {}
|
||||
self.max_size = 1000
|
||||
self.default_ttl = 300 # 5 分钟
|
||||
|
||||
async def get(self, key: str) -> Optional[Any]:
|
||||
"""获取缓存"""
|
||||
if key in self.cache:
|
||||
entry = self.cache[key]
|
||||
if entry.is_expired():
|
||||
del self.cache[key]
|
||||
else:
|
||||
return entry.value
|
||||
return None
|
||||
|
||||
async def set(self, key: str, value: Any, ttl: int = None) -> None:
|
||||
"""设置缓存"""
|
||||
if len(self.cache) >= self.max_size:
|
||||
# LRU 清理
|
||||
oldest = min(self.cache.items(), key=lambda x: x[1].created_at)
|
||||
del self.cache[oldest[0]]
|
||||
|
||||
self.cache[key] = CacheEntry(
|
||||
value=value,
|
||||
ttl=ttl or self.default_ttl,
|
||||
)
|
||||
|
||||
async def invalidate(self, pattern: str) -> None:
|
||||
"""清除匹配模式的缓存"""
|
||||
for key in list(self.cache.keys()):
|
||||
if pattern.format(key):
|
||||
del self.cache[key]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. API 端点扩展
|
||||
|
||||
### 4.1 板块 API
|
||||
|
||||
```python
|
||||
@router.get("/boards", response_model=list[ForumBoardOut])
|
||||
async def list_boards(
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""列出所有板块"""
|
||||
service = ForumService(db, lock_manager)
|
||||
return await service.list_boards()
|
||||
|
||||
|
||||
@router.post("/boards", response_model=ForumBoardOut, status_code=201)
|
||||
async def create_board(
|
||||
data: ForumBoardCreate,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""创建板块（仅管理员）"""
|
||||
if current_user.role != "admin":
|
||||
raise HTTPException(status_code=403, detail="需要管理员权限")
|
||||
service = ForumService(db, lock_manager)
|
||||
return await service.create_board(data)
|
||||
```
|
||||
|
||||
### 4.2 帖子 API 扩展
|
||||
|
||||
```python
|
||||
@router.get("/posts", response_model=PaginatedResponse[ForumPostOut])
|
||||
async def list_posts(
|
||||
board_id: Optional[str] = None,
|
||||
category: Optional[str] = None,
|
||||
tags: Optional[str] = None, # comma-separated
|
||||
page: int = 1,
|
||||
page_size: int = 20,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""分页获取帖子"""
|
||||
service = ForumService(db, lock_manager)
|
||||
tag_list = tags.split(",") if tags else None
|
||||
return await service.get_posts(
|
||||
board_id=board_id,
|
||||
category=category,
|
||||
tags=tag_list,
|
||||
page=page,
|
||||
page_size=page_size,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/posts", response_model=ForumPostOut, status_code=201)
|
||||
async def create_post(
|
||||
data: ForumPostCreate,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""创建帖子"""
|
||||
# 限流检查
|
||||
if not rate_limiter.check_rate_limit(
|
||||
current_user.id, "post", FORUM_CONFIG["MAX_POSTS_PER_USER_PER_DAY"]
|
||||
):
|
||||
raise HTTPException(status_code=429, detail="今日发帖次数已达上限")
|
||||
|
||||
# 验证数据
|
||||
data = validate_post_data(data)
|
||||
|
||||
service = ForumService(db, lock_manager)
|
||||
return await service.create_post(current_user.id, data)
|
||||
|
||||
|
||||
@router.patch("/posts/{post_id}/pin")
|
||||
async def pin_post(
|
||||
post_id: str,
|
||||
is_pinned: bool,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""置顶/取消置顶（仅版主+）"""
|
||||
service = ForumService(db, lock_manager)
|
||||
await service.pin_post(post_id, is_pinned)
|
||||
return {"success": True}
|
||||
|
||||
|
||||
@router.patch("/posts/{post_id}/lock")
|
||||
async def lock_post(
|
||||
post_id: str,
|
||||
is_locked: bool,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""锁定/解锁帖子"""
|
||||
service = ForumService(db, lock_manager)
|
||||
await service.lock_post(post_id, is_locked)
|
||||
return {"success": True}
|
||||
```
|
||||
|
||||
### 4.3 标签 API
|
||||
|
||||
```python
|
||||
@router.get("/tags", response_model=list[ForumTagOut])
|
||||
async def list_tags(
|
||||
query: Optional[str] = None,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""搜索/列出标签"""
|
||||
service = ForumService(db, lock_manager)
|
||||
if query:
|
||||
return await service.search_tags(query)
|
||||
return await service.list_tags()
|
||||
|
||||
|
||||
@router.post("/posts/{post_id}/tags")
|
||||
async def add_tags(
|
||||
post_id: str,
|
||||
tags: List[str],
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""为帖子添加标签"""
|
||||
service = ForumService(db, lock_manager)
|
||||
await service.add_tags_to_post(post_id, current_user.id, tags)
|
||||
return {"success": True}
|
||||
```
|
||||
|
||||
### 4.4 点赞 API
|
||||
|
||||
```python
|
||||
@router.post("/like")
|
||||
async def toggle_like(
|
||||
post_id: Optional[str] = None,
|
||||
reply_id: Optional[str] = None,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""切换点赞状态"""
|
||||
if not post_id and not reply_id:
|
||||
raise HTTPException(status_code=400, detail="需要指定 post_id 或 reply_id")
|
||||
|
||||
service = ForumService(db, lock_manager)
|
||||
liked = await service.toggle_like(
|
||||
user_id=current_user.id,
|
||||
post_id=post_id,
|
||||
reply_id=reply_id,
|
||||
)
|
||||
return {"liked": liked}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. 实现步骤
|
||||
|
||||
| 步骤 | 任务 | 优先级 |
|
||||
|------|------|--------|
|
||||
| 1 | 创建 ForumLockManager | 🟢 高 |
|
||||
| 2 | 创建 ForumService | 🟢 高 |
|
||||
| 3 | 实现输入验证函数 | 🟢 高 |
|
||||
| 4 | 实现限流器 | 🟡 中 |
|
||||
| 5 | 实现缓存 | 🟡 中 |
|
||||
| 6 | 扩展 Router 端点 | 🟢 高 |
|
||||
| 7 | 单元测试 | 🟡 中 |
|
||||
|
||||
---
|
||||
|
||||
## 6. 核心文件变更
|
||||
|
||||
| 文件 | 变更 |
|
||||
|------|------|
|
||||
| `services/forum_service.py` | 新增 |
|
||||
| `services/__init__.py` | 添加 export |
|
||||
| `routers/forum.py` | 扩展端点，集成服务 |
|
||||
|
||||
---
|
||||
|
||||
## 7. 工作量估算
|
||||
|
||||
| 任务 | 工作量 |
|
||||
|------|--------|
|
||||
| ForumLockManager | 0.5 天 |
|
||||
| ForumService 核心 | 1 天 |
|
||||
| 输入验证/限流/缓存 | 0.5 天 |
|
||||
| API 端点扩展 | 0.5 天 |
|
||||
| 单元测试 | 0.5 天 |
|
||||
| **总计** | **3 天** |
|
||||
|
||||
---
|
||||
|
||||
## 8. 验收标准
|
||||
|
||||
- [ ] ForumLockManager 可正确管理并发锁
|
||||
- [ ] 输入验证可过滤危险字符
|
||||
- [ ] 限流器可防止滥用
|
||||
- [ ] 缓存可提升热门帖子读取速度
|
||||
- [ ] 所有新增 API 端点正常工作
|
||||
- [ ] 现有 API 保持向后兼容
|
||||
- [ ] 单元测试覆盖核心逻辑
|
||||
540
development-doc/plan/forum-update/phase-f-3-permissions.md
Normal file
540
development-doc/plan/forum-update/phase-f-3-permissions.md
Normal file
@@ -0,0 +1,540 @@
|
||||
# Phase F.3：权限系统
|
||||
|
||||
日期：2026-04-04
|
||||
状态：待开始
|
||||
依赖：F.2（待完成）
|
||||
前置：models/user.py
|
||||
|
||||
---
|
||||
|
||||
## 1. 本阶段目的
|
||||
|
||||
实现 Jarvis Forum 的权限系统：
|
||||
|
||||
- 用户角色管理（user/moderator/admin）
|
||||
- 板块权限控制
|
||||
- 操作日志记录
|
||||
- 积分/奖励系统
|
||||
|
||||
---
|
||||
|
||||
## 2. 用户角色系统
|
||||
|
||||
### 2.1 角色定义
|
||||
|
||||
```python
|
||||
class UserRole(str, Enum):
|
||||
"""用户角色枚举"""
|
||||
GUEST = "guest" # 访客 - 只能浏览
|
||||
USER = "user" # 普通用户 - 可发帖/回复
|
||||
MODERATOR = "moderator" # 版主 - 可管理板块
|
||||
ADMIN = "admin" # 管理员 - 全权限
|
||||
|
||||
|
||||
class Permission(str, Enum):
|
||||
"""权限枚举"""
|
||||
# 帖子权限
|
||||
POST_CREATE = "post:create"
|
||||
POST_EDIT_OWN = "post:edit:own"
|
||||
POST_EDIT_ANY = "post:edit:any"
|
||||
POST_DELETE_OWN = "post:delete:own"
|
||||
POST_DELETE_ANY = "post:delete:any"
|
||||
POST_PIN = "post:pin"
|
||||
POST_LOCK = "post:lock"
|
||||
POST_VIEW = "post:view"
|
||||
|
||||
# 回复权限
|
||||
REPLY_CREATE = "reply:create"
|
||||
REPLY_EDIT_OWN = "reply:edit:own"
|
||||
REPLY_DELETE_OWN = "reply:delete:own"
|
||||
REPLY_DELETE_ANY = "reply:delete:any"
|
||||
REPLY_PIN = "reply:pin"
|
||||
REPLY_BEST = "reply:best"
|
||||
|
||||
# 板块权限
|
||||
BOARD_CREATE = "board:create"
|
||||
BOARD_EDIT = "board:edit"
|
||||
BOARD_DELETE = "board:delete"
|
||||
|
||||
# 标签权限
|
||||
TAG_CREATE = "tag:create"
|
||||
TAG_MANAGE = "tag:manage"
|
||||
|
||||
# 用户权限
|
||||
USER_BAN = "user:ban"
|
||||
USER_ROLE = "user:role"
|
||||
|
||||
# 积分权限
|
||||
SCORE_VIEW = "score:view"
|
||||
SCORE_MANAGE = "score:manage"
|
||||
```
|
||||
|
||||
### 2.2 角色权限映射
|
||||
|
||||
```python
|
||||
ROLE_PERMISSIONS: Dict[UserRole, Set[Permission]] = {
|
||||
UserRole.GUEST: {
|
||||
Permission.POST_VIEW,
|
||||
},
|
||||
UserRole.USER: {
|
||||
Permission.POST_VIEW,
|
||||
Permission.POST_CREATE,
|
||||
Permission.POST_EDIT_OWN,
|
||||
Permission.POST_DELETE_OWN,
|
||||
Permission.REPLY_CREATE,
|
||||
Permission.REPLY_EDIT_OWN,
|
||||
Permission.REPLY_DELETE_OWN,
|
||||
Permission.SCORE_VIEW,
|
||||
},
|
||||
UserRole.MODERATOR: {
|
||||
# 用户权限
|
||||
Permission.POST_VIEW,
|
||||
Permission.POST_CREATE,
|
||||
Permission.POST_EDIT_OWN,
|
||||
Permission.POST_DELETE_OWN,
|
||||
Permission.REPLY_CREATE,
|
||||
Permission.REPLY_EDIT_OWN,
|
||||
Permission.REPLY_DELETE_OWN,
|
||||
# 版主权限
|
||||
Permission.POST_PIN,
|
||||
Permission.POST_LOCK,
|
||||
Permission.POST_DELETE_ANY,
|
||||
Permission.REPLY_PIN,
|
||||
Permission.REPLY_BEST,
|
||||
Permission.TAG_MANAGE,
|
||||
Permission.SCORE_VIEW,
|
||||
},
|
||||
UserRole.ADMIN: set(Permission), # 所有权限
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. User 模型扩展
|
||||
|
||||
### 3.1 新增字段
|
||||
|
||||
```python
|
||||
# models/user.py 扩展
|
||||
class User(BaseModel):
|
||||
__tablename__ = "users"
|
||||
|
||||
# === 现有字段 ===
|
||||
id: str
|
||||
username: str
|
||||
email: str
|
||||
hashed_password: str
|
||||
avatar: Optional[str]
|
||||
created_at: datetime
|
||||
|
||||
# === Forum 相关扩展 ===
|
||||
role: str = "user" # guest/user/moderator/admin
|
||||
forum_score: int = 0 # 论坛积分
|
||||
|
||||
# 论坛统计
|
||||
post_count: int = 0
|
||||
reply_count: int = 0
|
||||
like_received: int = 0
|
||||
best_reply_count: int = 0
|
||||
|
||||
# 状态
|
||||
is_forum_banned: bool = False # 论坛禁言
|
||||
forum_ban_until: Optional[datetime] = None # 禁言截止时间
|
||||
|
||||
# 板块版主关系
|
||||
moderated_boards: List[str] = [] # 管理的板块 ID 列表
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Permission Service
|
||||
|
||||
### 4.1 服务实现
|
||||
|
||||
```python
|
||||
class PermissionService:
|
||||
"""权限服务"""
|
||||
|
||||
def __init__(self, db: AsyncSession):
|
||||
self.db = db
|
||||
|
||||
def has_permission(self, user: User, permission: Permission) -> bool:
|
||||
"""检查用户是否拥有某权限"""
|
||||
if user.is_forum_banned:
|
||||
return False
|
||||
if user.forum_ban_until and user.forum_ban_until > datetime.utcnow():
|
||||
return False
|
||||
|
||||
role = UserRole(user.role)
|
||||
return permission in ROLE_PERMISSIONS.get(role, set())
|
||||
|
||||
def has_board_permission(
|
||||
self,
|
||||
user: User,
|
||||
permission: Permission,
|
||||
board_id: str,
|
||||
) -> bool:
|
||||
"""检查用户在特定板块的权限"""
|
||||
if not self.has_permission(user, permission):
|
||||
return False
|
||||
|
||||
# 版主只能管理自己板块
|
||||
if role == UserRole.MODERATOR:
|
||||
return board_id in user.moderated_boards
|
||||
|
||||
return True
|
||||
|
||||
async def can_edit_post(self, user: User, post: ForumPost) -> bool:
|
||||
"""检查用户是否可以编辑帖子"""
|
||||
if user.role == UserRole.ADMIN:
|
||||
return True
|
||||
if user.role == UserRole.MODERATOR:
|
||||
return post.board_id in user.moderated_boards
|
||||
return post.user_id == user.id
|
||||
|
||||
async def can_delete_post(self, user: User, post: ForumPost) -> bool:
|
||||
"""检查用户是否可以删除帖子"""
|
||||
if user.role == UserRole.ADMIN:
|
||||
return True
|
||||
if user.role == UserRole.MODERATOR:
|
||||
return post.board_id in user.moderated_boards
|
||||
return post.user_id == user.id
|
||||
|
||||
async def ban_user(
|
||||
self,
|
||||
admin: User,
|
||||
target_user_id: str,
|
||||
reason: str,
|
||||
duration: Optional[timedelta] = None,
|
||||
) -> None:
|
||||
"""禁言用户"""
|
||||
if admin.role != UserRole.ADMIN:
|
||||
raise PermissionError("需要管理员权限")
|
||||
|
||||
result = await self.db.execute(
|
||||
select(User).where(User.id == target_user_id)
|
||||
)
|
||||
target_user = result.scalar_one_or_none()
|
||||
|
||||
if not target_user:
|
||||
raise ValueError("用户不存在")
|
||||
|
||||
target_user.is_forum_banned = True
|
||||
target_user.forum_ban_until = (
|
||||
datetime.utcnow() + duration if duration else None
|
||||
)
|
||||
|
||||
# 记录操作日志
|
||||
await self._log_action(
|
||||
admin=admin,
|
||||
action="ban_user",
|
||||
target_id=target_user_id,
|
||||
details={"reason": reason, "duration": str(duration)},
|
||||
)
|
||||
```
|
||||
|
||||
### 4.2 依赖注入
|
||||
|
||||
```python
|
||||
# 在 ForumService 中注入
|
||||
class ForumService:
|
||||
def __init__(
|
||||
self,
|
||||
db: AsyncSession,
|
||||
lock_manager: ForumLockManager,
|
||||
permission_service: PermissionService,
|
||||
):
|
||||
self.db = db
|
||||
self.lock = lock_manager
|
||||
self.perms = permission_service
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. 操作日志
|
||||
|
||||
### 5.1 日志模型
|
||||
|
||||
```python
|
||||
class ForumLog(BaseModel):
|
||||
__tablename__ = "forum_logs"
|
||||
|
||||
id: str = Field(default_factory=lambda: str(uuid.uuid4()))
|
||||
user_id: str # 操作者
|
||||
action: str # 操作类型
|
||||
target_type: str # post/reply/board/user/tag
|
||||
target_id: str # 目标 ID
|
||||
details: Optional[str] = None # JSON 详情
|
||||
ip_address: Optional[str] = None
|
||||
user_agent: Optional[str] = None
|
||||
created_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
```
|
||||
|
||||
### 5.2 日志记录
|
||||
|
||||
```python
|
||||
async def _log_action(
|
||||
self,
|
||||
admin: User,
|
||||
action: str,
|
||||
target_id: str,
|
||||
details: Optional[dict] = None,
|
||||
request: Optional[Request] = None,
|
||||
) -> None:
|
||||
"""记录操作日志"""
|
||||
log = ForumLog(
|
||||
user_id=admin.id,
|
||||
action=action,
|
||||
target_type=action.split(":")[0],
|
||||
target_id=target_id,
|
||||
details=json.dumps(details) if details else None,
|
||||
ip_address=request.client.host if request else None,
|
||||
user_agent=request.headers.get("user-agent") if request else None,
|
||||
)
|
||||
self.db.add(log)
|
||||
await self.db.commit()
|
||||
|
||||
|
||||
# 日志操作类型
|
||||
class ForumAction(str, Enum):
|
||||
POST_CREATE = "post:create"
|
||||
POST_UPDATE = "post:update"
|
||||
POST_DELETE = "post:delete"
|
||||
POST_PIN = "post:pin"
|
||||
POST_LOCK = "post:lock"
|
||||
REPLY_CREATE = "reply:create"
|
||||
REPLY_UPDATE = "reply:update"
|
||||
REPLY_DELETE = "reply:delete"
|
||||
USER_BAN = "user:ban"
|
||||
USER_UNBAN = "user:unban"
|
||||
SCORE_CHANGE = "score:change"
|
||||
BOARD_CREATE = "board:create"
|
||||
BOARD_UPDATE = "board:update"
|
||||
BOARD_DELETE = "board:delete"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. 积分系统
|
||||
|
||||
### 6.1 积分规则
|
||||
|
||||
```python
|
||||
SCORE_RULES = {
|
||||
# 帖子相关
|
||||
"post_create": 5, # 发帖
|
||||
"post_delete": -5, # 删除帖子
|
||||
"post_liked": 2, # 帖子被点赞
|
||||
"post_best": 10, # 帖子被设为精华
|
||||
|
||||
# 回复相关
|
||||
"reply_create": 2, # 回复
|
||||
"reply_delete": -2, # 删除回复
|
||||
"reply_liked": 1, # 回复被点赞
|
||||
"reply_best": 5, # 回复被设为最佳
|
||||
"reply_adopted": 10, # 提问被采纳
|
||||
|
||||
# 活跃相关
|
||||
"daily_login": 1, # 每日登录
|
||||
"daily_post": 3, # 每日首次发帖
|
||||
"daily_reply": 1, # 每日首次回复
|
||||
}
|
||||
|
||||
|
||||
class ScoreService:
|
||||
"""积分服务"""
|
||||
|
||||
def __init__(self, db: AsyncSession):
|
||||
self.db = db
|
||||
|
||||
async def add_score(
|
||||
self,
|
||||
user_id: str,
|
||||
action: str,
|
||||
reason: str,
|
||||
) -> int:
|
||||
"""增加积分"""
|
||||
score_delta = SCORE_RULES.get(action, 0)
|
||||
|
||||
result = await self.db.execute(
|
||||
select(User).where(User.id == user_id)
|
||||
)
|
||||
user = result.scalar_one_or_none()
|
||||
|
||||
if user:
|
||||
user.forum_score += score_delta
|
||||
|
||||
# 更新统计
|
||||
if action.startswith("post_"):
|
||||
user.post_count += 1
|
||||
elif action.startswith("reply_"):
|
||||
user.reply_count += 1
|
||||
|
||||
if action == "post_liked":
|
||||
user.like_received += 1
|
||||
elif action == "reply_liked":
|
||||
user.like_received += 1
|
||||
elif action == "reply_best":
|
||||
user.best_reply_count += 1
|
||||
|
||||
await self.db.commit()
|
||||
|
||||
return score_delta
|
||||
|
||||
async def get_leaderboard(
|
||||
self,
|
||||
limit: int = 10,
|
||||
period: str = "all", # all/month/week
|
||||
) -> List[dict]:
|
||||
"""获取积分排行榜"""
|
||||
query = select(User).where(User.forum_score > 0)
|
||||
|
||||
if period == "month":
|
||||
# 本月排行
|
||||
pass
|
||||
elif period == "week":
|
||||
# 本周排行
|
||||
pass
|
||||
|
||||
query = query.order_by(desc(User.forum_score)).limit(limit)
|
||||
result = await self.db.execute(query)
|
||||
|
||||
return [
|
||||
{
|
||||
"rank": i + 1,
|
||||
"user_id": user.id,
|
||||
"username": user.username,
|
||||
"avatar": user.avatar,
|
||||
"score": user.forum_score,
|
||||
}
|
||||
for i, user in enumerate(result.scalars().all())
|
||||
]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. API 端点
|
||||
|
||||
### 7.1 管理端点
|
||||
|
||||
```python
|
||||
@router.post("/admin/ban/{user_id}")
|
||||
async def ban_user(
|
||||
user_id: str,
|
||||
reason: str,
|
||||
duration: Optional[int] = None, # 天数
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""禁言用户（仅管理员）"""
|
||||
if current_user.role != "admin":
|
||||
raise HTTPException(status_code=403, detail="需要管理员权限")
|
||||
|
||||
perm_service = PermissionService(db)
|
||||
await perm_service.ban_user(
|
||||
admin=current_user,
|
||||
target_user_id=user_id,
|
||||
reason=reason,
|
||||
duration=timedelta(days=duration) if duration else None,
|
||||
)
|
||||
return {"success": True}
|
||||
|
||||
|
||||
@router.post("/admin/unban/{user_id}")
|
||||
async def unban_user(
|
||||
user_id: str,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""解除禁言"""
|
||||
if current_user.role != "admin":
|
||||
raise HTTPException(status_code=403, detail="需要管理员权限")
|
||||
|
||||
perm_service = PermissionService(db)
|
||||
await perm_service.unban_user(current_user, user_id)
|
||||
return {"success": True}
|
||||
|
||||
|
||||
@router.get("/admin/logs")
|
||||
async def get_logs(
|
||||
target_id: Optional[str] = None,
|
||||
action: Optional[str] = None,
|
||||
page: int = 1,
|
||||
page_size: int = 50,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""查看操作日志（仅管理员/版主）"""
|
||||
if current_user.role not in ["admin", "moderator"]:
|
||||
raise HTTPException(status_code=403, detail="权限不足")
|
||||
|
||||
# 查询日志...
|
||||
return {"logs": [], "total": 0}
|
||||
|
||||
|
||||
@router.get("/leaderboard")
|
||||
async def get_leaderboard(
|
||||
limit: int = 10,
|
||||
period: str = "all",
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""获取积分排行榜"""
|
||||
score_service = ScoreService(db)
|
||||
return await score_service.get_leaderboard(limit, period)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. 实现步骤
|
||||
|
||||
| 步骤 | 任务 | 优先级 |
|
||||
|------|------|--------|
|
||||
| 1 | 扩展 User 模型 | 🟢 高 |
|
||||
| 2 | 创建 PermissionService | 🟢 高 |
|
||||
| 3 | 实现权限检查装饰器 | 🟢 高 |
|
||||
| 4 | 创建 ForumLog 模型 | 🟡 中 |
|
||||
| 5 | 实现操作日志记录 | 🟡 中 |
|
||||
| 6 | 创建 ScoreService | 🟡 中 |
|
||||
| 7 | 实现积分规则 | 🟡 中 |
|
||||
| 8 | 扩展管理 API | 🟡 中 |
|
||||
| 9 | 单元测试 | 🟡 中 |
|
||||
|
||||
---
|
||||
|
||||
## 9. 核心文件变更
|
||||
|
||||
| 文件 | 变更 |
|
||||
|------|------|
|
||||
| `models/user.py` | 扩展角色和统计字段 |
|
||||
| `models/forum.py` | 新增 ForumLog |
|
||||
| `services/permission_service.py` | 新增 |
|
||||
| `services/score_service.py` | 新增 |
|
||||
| `services/forum_service.py` | 集成权限检查 |
|
||||
| `routers/forum.py` | 扩展管理端点 |
|
||||
|
||||
---
|
||||
|
||||
## 10. 工作量估算
|
||||
|
||||
| 任务 | 工作量 |
|
||||
|------|--------|
|
||||
| User 模型扩展 | 0.5 天 |
|
||||
| PermissionService | 0.5 天 |
|
||||
| 操作日志 | 0.5 天 |
|
||||
| ScoreService | 0.5 天 |
|
||||
| API 端点 | 0.5 天 |
|
||||
| 单元测试 | 0.5 天 |
|
||||
| **总计** | **3 天** |
|
||||
|
||||
---
|
||||
|
||||
## 11. 验收标准
|
||||
|
||||
- [ ] User 模型正确存储角色和积分
|
||||
- [ ] PermissionService 可正确检查权限
|
||||
- [ ] 权限不足时返回 403
|
||||
- [ ] 所有管理操作记录日志
|
||||
- [ ] 积分根据规则正确增减
|
||||
- [ ] 排行榜正确排序
|
||||
- [ ] 禁言功能正常工作
|
||||
- [ ] 单元测试覆盖核心逻辑
|
||||
652
development-doc/plan/forum-update/phase-f-4-ai-integration.md
Normal file
652
development-doc/plan/forum-update/phase-f-4-ai-integration.md
Normal file
@@ -0,0 +1,652 @@
|
||||
# Phase F.4：AI 集成
|
||||
|
||||
日期：2026-04-04
|
||||
状态：待开始
|
||||
依赖：F.3（待完成）
|
||||
前置：services/forum_ai_service.py
|
||||
|
||||
---
|
||||
|
||||
## 1. 本阶段目的
|
||||
|
||||
为 Jarvis Forum 集成 AI 能力：
|
||||
|
||||
- AI 自动回复
|
||||
- 帖子摘要生成
|
||||
- 智能分类打标
|
||||
- Agent 自主发帖
|
||||
|
||||
---
|
||||
|
||||
## 2. Forum AI Service
|
||||
|
||||
### 2.1 服务架构
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ ForumAIService │
|
||||
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────────┐ │
|
||||
│ │ AutoReply │ │ Summary │ │ SmartTagging │ │
|
||||
│ │ Service │ │ Service │ │ Service │ │
|
||||
│ └──────┬──────┘ └──────┬──────┘ └────────┬────────┘ │
|
||||
│ │ │ │ │
|
||||
│ └────────────────┼───────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌───────────┴───────────┐ │
|
||||
│ │ LLM Service │ │
|
||||
│ │ (复用 Agent LLM) │ │
|
||||
│ └───────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### 2.2 核心服务
|
||||
|
||||
```python
|
||||
class ForumAIService:
|
||||
"""论坛 AI 服务"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
llm_service: LLMService,
|
||||
config: ForumAIConfig,
|
||||
):
|
||||
self.llm = llm_service
|
||||
self.config = config
|
||||
|
||||
# === 自动回复 ===
|
||||
async def generate_auto_reply(
|
||||
self,
|
||||
post: ForumPost,
|
||||
replies: List[ForumReply],
|
||||
) -> Optional[str]:
|
||||
"""生成自动回复"""
|
||||
if not self.config.auto_reply_enabled:
|
||||
return None
|
||||
|
||||
# 检查是否需要回复
|
||||
if not self._should_auto_reply(post):
|
||||
return None
|
||||
|
||||
# 构建上下文
|
||||
context = self._build_reply_context(post, replies)
|
||||
|
||||
# 生成回复
|
||||
prompt = self._build_reply_prompt(context)
|
||||
response = await self.llm.agenerate(prompt)
|
||||
|
||||
return response.content if response else None
|
||||
|
||||
def _should_auto_reply(self, post: ForumPost) -> bool:
|
||||
"""判断是否应该自动回复"""
|
||||
# 指令类帖子不自动回复
|
||||
if post.category == "instruction":
|
||||
return False
|
||||
|
||||
# 有 AI 回复的不重复回复
|
||||
if any(r.is_ai_reply for r in post.replies):
|
||||
return False
|
||||
|
||||
# 检查时间窗口
|
||||
if post.last_reply_at:
|
||||
hours_since = (datetime.utcnow() - post.last_reply_at).total_seconds() / 3600
|
||||
if hours_since < self.config.auto_reply_min_hours:
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
def _build_reply_context(
|
||||
self,
|
||||
post: ForumPost,
|
||||
replies: List[ForumReply],
|
||||
) -> str:
|
||||
"""构建回复上下文"""
|
||||
context = f"""## 帖子信息
|
||||
标题: {post.title}
|
||||
内容: {post.content[:500]}...
|
||||
|
||||
## 回复列表
|
||||
"""
|
||||
for reply in replies[-5:]: # 最近 5 条
|
||||
context += f"- {reply.user_id}: {reply.content[:200]}...\n"
|
||||
|
||||
return context
|
||||
|
||||
def _build_reply_prompt(self, context: str) -> str:
|
||||
"""构建回复提示词"""
|
||||
return f"""你是一个友好的社区助手。请根据以下帖子内容，生成一条有帮助的回复。
|
||||
|
||||
{context}
|
||||
|
||||
要求:
|
||||
1. 回复要友好、专业、有帮助
|
||||
2. 不要重复已有的观点
|
||||
3. 如果是问题，尽量给出建设性的建议
|
||||
4. 回复长度控制在 100-300 字
|
||||
5. 不要使用 Markdown 格式，直接输出文本
|
||||
|
||||
回复:"""
|
||||
|
||||
# === 摘要生成 ===
|
||||
async def generate_summary(
|
||||
self,
|
||||
content: str,
|
||||
max_length: int = 200,
|
||||
) -> str:
|
||||
"""生成帖子摘要"""
|
||||
if len(content) <= max_length:
|
||||
return content
|
||||
|
||||
prompt = f"""请为以下内容生成一个简洁的摘要，不超过 {max_length} 字。
|
||||
|
||||
内容:
|
||||
{content}
|
||||
|
||||
摘要:"""
|
||||
|
||||
response = await self.llm.agenerate(prompt)
|
||||
return response.content if response else content[:max_length]
|
||||
|
||||
# === 智能打标 ===
|
||||
async def suggest_tags(
|
||||
self,
|
||||
title: str,
|
||||
content: str,
|
||||
existing_tags: List[str],
|
||||
) -> List[str]:
|
||||
"""智能推荐标签"""
|
||||
prompt = f"""请根据以下帖子内容，推荐 3-5 个最合适的标签。
|
||||
|
||||
标题: {title}
|
||||
内容: {content[:1000]}...
|
||||
|
||||
已有标签: {', '.join(existing_tags) if existing_tags else '无'}
|
||||
|
||||
要求:
|
||||
1. 标签要简洁，2-4 个字
|
||||
2. 选择最相关的标签
|
||||
3. 如果已有标签合适可以保留
|
||||
4. 只输出标签，用逗号分隔，不要其他内容
|
||||
|
||||
标签:"""
|
||||
|
||||
response = await self.llm.agenerate(prompt)
|
||||
if not response:
|
||||
return existing_tags
|
||||
|
||||
# 解析标签
|
||||
tags = [t.strip() for t in response.content.split(",")]
|
||||
return tags[:5] # 最多 5 个
|
||||
|
||||
# === 智能分类 ===
|
||||
async def classify_category(
|
||||
self,
|
||||
title: str,
|
||||
content: str,
|
||||
) -> str:
|
||||
"""智能分类"""
|
||||
categories = ["instruction", "discussion", "question", "praise", "bug", "other"]
|
||||
|
||||
prompt = f"""请为以下帖子选择一个最合适的分类。
|
||||
|
||||
标题: {title}
|
||||
内容: {content[:500]}...
|
||||
|
||||
可选分类:
|
||||
- instruction: 指令/任务请求
|
||||
- discussion: 讨论/分享
|
||||
- question: 问题/求助
|
||||
- praise: 表扬/感谢
|
||||
- bug: Bug 反馈
|
||||
- other: 其他
|
||||
|
||||
请直接输出分类名称，不要其他内容。
|
||||
|
||||
分类:"""
|
||||
|
||||
response = await self.llm.agenerate(prompt)
|
||||
if not response:
|
||||
return "other"
|
||||
|
||||
category = response.content.strip().lower()
|
||||
if category not in categories:
|
||||
return "other"
|
||||
|
||||
return category
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Summary Service
|
||||
|
||||
### 3.1 服务实现
|
||||
|
||||
```python
|
||||
class SummaryService:
|
||||
"""帖子摘要服务"""
|
||||
|
||||
def __init__(self, cache: ForumCache):
|
||||
self.cache = cache
|
||||
|
||||
async def get_post_summary(
|
||||
self,
|
||||
post_id: str,
|
||||
post: ForumPost,
|
||||
) -> str:
|
||||
"""获取帖子摘要（带缓存）"""
|
||||
cache_key = f"summary:{post_id}"
|
||||
|
||||
# 尝试从缓存获取
|
||||
cached = await self.cache.get(cache_key)
|
||||
if cached:
|
||||
return cached
|
||||
|
||||
# 生成摘要
|
||||
summary = await self._generate_summary(post)
|
||||
|
||||
# 存入缓存（1 小时）
|
||||
await self.cache.set(cache_key, summary, ttl=3600)
|
||||
|
||||
return summary
|
||||
|
||||
async def get_thread_summary(
|
||||
self,
|
||||
post: ForumPost,
|
||||
replies: List[ForumReply],
|
||||
) -> str:
|
||||
"""获取帖子串摘要（主帖+回复摘要）"""
|
||||
summaries = [await self.get_post_summary(post.id, post)]
|
||||
|
||||
# 汇总回复要点
|
||||
if len(replies) > 0:
|
||||
summary_prompt = f"""请总结以下回复的核心观点，用 50 字以内概括。
|
||||
|
||||
回复列表:
|
||||
{self._format_replies(replies[:10])}
|
||||
|
||||
总结:"""
|
||||
|
||||
response = await self.llm.agenerate(summary_prompt)
|
||||
if response:
|
||||
summaries.append(f"回复要点: {response.content}")
|
||||
|
||||
return "\n\n".join(summaries)
|
||||
|
||||
async def invalidate_summary(self, post_id: str) -> None:
|
||||
"""清除摘要缓存"""
|
||||
await self.cache.invalidate(f"summary:{post_id}")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Agent 自主发帖
|
||||
|
||||
### 4.1 Agent Forum 工具
|
||||
|
||||
```python
|
||||
# agents/tools/forum_tools.py
|
||||
from typing import List, Optional
|
||||
|
||||
|
||||
class ForumTools:
|
||||
"""Forum Agent 工具集"""
|
||||
|
||||
def __init__(self, forum_service: ForumService, ai_service: ForumAIService):
|
||||
self.forum = forum_service
|
||||
self.ai = ai_service
|
||||
|
||||
@tool
|
||||
async def create_forum_post(
|
||||
title: str,
|
||||
content: str,
|
||||
board_id: Optional[str] = None,
|
||||
category: Optional[str] = None,
|
||||
tags: Optional[List[str]] = None,
|
||||
) -> dict:
|
||||
"""创建论坛帖子
|
||||
|
||||
参数:
|
||||
- title: 帖子标题
|
||||
- content: 帖子内容
|
||||
- board_id: 板块 ID（可选）
|
||||
- category: 分类（可选）
|
||||
- tags: 标签列表（可选）
|
||||
"""
|
||||
data = ForumPostCreate(
|
||||
title=title,
|
||||
content=content,
|
||||
board_id=board_id,
|
||||
category=category,
|
||||
tags=tags or [],
|
||||
)
|
||||
post = await self.forum.create_post(agent_id=AGENT_ID, data=data)
|
||||
return {"post_id": post.id, "title": post.title}
|
||||
|
||||
@tool
|
||||
async def reply_to_post(
|
||||
post_id: str,
|
||||
content: str,
|
||||
) -> dict:
|
||||
"""回复帖子
|
||||
|
||||
参数:
|
||||
- post_id: 帖子 ID
|
||||
- content: 回复内容
|
||||
"""
|
||||
data = ForumReplyCreate(content=content)
|
||||
reply = await self.forum.create_reply(
|
||||
post_id=post_id,
|
||||
agent_id=AGENT_ID,
|
||||
data=data,
|
||||
)
|
||||
return {"reply_id": reply.id, "floor": reply.floor}
|
||||
|
||||
@tool
|
||||
async def search_forum_posts(
|
||||
query: str,
|
||||
board_id: Optional[str] = None,
|
||||
category: Optional[str] = None,
|
||||
limit: int = 10,
|
||||
) -> List[dict]:
|
||||
"""搜索论坛帖子
|
||||
|
||||
参数:
|
||||
- query: 搜索关键词
|
||||
- board_id: 限定板块（可选）
|
||||
- category: 限定分类（可选）
|
||||
- limit: 返回数量（默认 10）
|
||||
"""
|
||||
posts = await self.forum.search_posts(
|
||||
query=query,
|
||||
board_id=board_id,
|
||||
category=category,
|
||||
limit=limit,
|
||||
)
|
||||
return [
|
||||
{"id": p.id, "title": p.title, "summary": p.content[:100]}
|
||||
for p in posts
|
||||
]
|
||||
|
||||
@tool
|
||||
async def get_forum_trending(
|
||||
board_id: Optional[str] = None,
|
||||
limit: int = 5,
|
||||
) -> List[dict]:
|
||||
"""获取热门帖子
|
||||
|
||||
参数:
|
||||
- board_id: 板块 ID（可选）
|
||||
- limit: 返回数量（默认 5）
|
||||
"""
|
||||
posts = await self.forum.get_trending(
|
||||
board_id=board_id,
|
||||
limit=limit,
|
||||
)
|
||||
return [
|
||||
{
|
||||
"id": p.id,
|
||||
"title": p.title,
|
||||
"reply_count": p.reply_count,
|
||||
"view_count": p.view_count,
|
||||
}
|
||||
for p in posts
|
||||
]
|
||||
```
|
||||
|
||||
### 4.2 Agent 配置
|
||||
|
||||
```python
|
||||
# agents/prompts/forum_agent.py
|
||||
FORUM_AGENT_PROMPT = """你是一个活跃的社区成员，可以帮助用户解决问题和参与讨论。
|
||||
|
||||
## 你的能力
|
||||
1. 在论坛发帖分享信息或见解
|
||||
2. 回复其他用户的帖子
|
||||
3. 搜索论坛内容
|
||||
4. 查看热门帖子
|
||||
|
||||
## 行为规则
|
||||
1. 只在有帮助时才发帖/回复，不要刷屏
|
||||
2. 回复要专业、有建设性
|
||||
3. 如果用户的问题已经解决，不要重复回答
|
||||
4. 遇到 Bug 或问题可以主动发帖提醒
|
||||
5. 定期查看论坛，如果有重要帖子可以参与讨论
|
||||
|
||||
## 当前时间
|
||||
{current_time}
|
||||
|
||||
## 用户信息
|
||||
用户名: {username}
|
||||
积分: {forum_score}
|
||||
"""
|
||||
|
||||
# Agent 可用的 Forum 工具
|
||||
FORUM_TOOLS = [
|
||||
create_forum_post,
|
||||
reply_to_post,
|
||||
search_forum_posts,
|
||||
get_forum_trending,
|
||||
]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. 定时任务
|
||||
|
||||
### 5.1 自动回复任务
|
||||
|
||||
```python
|
||||
# tasks/forum_auto_reply.py
|
||||
from apscheduler.schedulers.asyncio import AsyncIOScheduler
|
||||
|
||||
|
||||
async def auto_reply_task():
|
||||
"""自动回复任务"""
|
||||
# 获取需要回复的帖子
|
||||
posts = await forum_service.get_pending_posts(
|
||||
min_age_hours=settings.auto_reply_min_hours,
|
||||
limit=10,
|
||||
)
|
||||
|
||||
for post in posts:
|
||||
replies = await forum_service.get_replies(post.id)
|
||||
|
||||
# 生成回复
|
||||
response = await ai_service.generate_auto_reply(post, replies)
|
||||
|
||||
if response:
|
||||
# 发布回复
|
||||
await forum_service.create_reply(
|
||||
post_id=post.id,
|
||||
agent_id=AGENT_ID,
|
||||
data=ForumReplyCreate(content=response),
|
||||
)
|
||||
|
||||
# 更新回复计数
|
||||
await forum_service.increment_reply_count(post.id)
|
||||
|
||||
|
||||
def setup_forum_scheduler(scheduler: AsyncIOScheduler):
|
||||
"""配置论坛定时任务"""
|
||||
# 每小时检查一次自动回复
|
||||
scheduler.add_job(
|
||||
auto_reply_task,
|
||||
"interval",
|
||||
hours=1,
|
||||
id="forum_auto_reply",
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. 配置项
|
||||
|
||||
### 6.1 AI 配置
|
||||
|
||||
```python
|
||||
class ForumAIConfig:
|
||||
"""Forum AI 配置"""
|
||||
|
||||
# 自动回复
|
||||
auto_reply_enabled: bool = True
|
||||
auto_reply_min_hours: int = 24 # 帖子发布 N 小时后才自动回复
|
||||
auto_reply_max_per_day: int = 10 # 每天最多自动回复 N 条
|
||||
|
||||
# 摘要生成
|
||||
summary_enabled: bool = True
|
||||
summary_max_length: int = 200
|
||||
summary_cache_ttl: int = 3600 # 缓存 1 小时
|
||||
|
||||
# 智能打标
|
||||
smart_tagging_enabled: bool = True
|
||||
smart_tagging_max_tags: int = 5
|
||||
|
||||
# 智能分类
|
||||
smart_classification_enabled: bool = True
|
||||
|
||||
|
||||
# settings.py
|
||||
class Settings(BaseSettings):
|
||||
# Forum AI
|
||||
forum_ai_auto_reply: bool = True
|
||||
forum_ai_summary: bool = True
|
||||
forum_ai_smart_tagging: bool = True
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. API 端点
|
||||
|
||||
### 7.1 AI 相关端点
|
||||
|
||||
```python
|
||||
@router.post("/posts/{post_id}/generate-summary")
|
||||
async def generate_post_summary(
|
||||
post_id: str,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""生成帖子摘要"""
|
||||
result = await db.execute(
|
||||
select(ForumPost).where(ForumPost.id == post_id)
|
||||
)
|
||||
post = result.scalar_one_or_none()
|
||||
|
||||
if not post:
|
||||
raise HTTPException(status_code=404, detail="帖子不存在")
|
||||
|
||||
ai_service = ForumAIService(llm_service, config)
|
||||
summary = await ai_service.generate_summary(post.content)
|
||||
|
||||
return {"summary": summary}
|
||||
|
||||
|
||||
@router.post("/posts/suggest-tags")
|
||||
async def suggest_post_tags(
|
||||
title: str,
|
||||
content: str,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""推荐帖子标签"""
|
||||
ai_service = ForumAIService(llm_service, config)
|
||||
tags = await ai_service.suggest_tags(title, content, [])
|
||||
|
||||
return {"tags": tags}
|
||||
|
||||
|
||||
@router.post("/posts/classify")
|
||||
async def classify_post(
|
||||
title: str,
|
||||
content: str,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""分类帖子"""
|
||||
ai_service = ForumAIService(llm_service, config)
|
||||
category = await ai_service.classify_category(title, content)
|
||||
|
||||
return {"category": category}
|
||||
|
||||
|
||||
@router.get("/posts/{post_id}/ai-status")
|
||||
async def get_ai_status(
|
||||
post_id: str,
|
||||
current_user: User = Depends(get_current_user),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
):
|
||||
"""获取帖子 AI 状态"""
|
||||
result = await db.execute(
|
||||
select(ForumPost).where(ForumPost.id == post_id)
|
||||
)
|
||||
post = result.scalar_one_or_none()
|
||||
|
||||
if not post:
|
||||
raise HTTPException(status_code=404, detail="帖子不存在")
|
||||
|
||||
# 检查 AI 回复状态
|
||||
has_ai_reply = any(r.is_ai_reply for r in post.replies)
|
||||
summary_cached = await cache.get(f"summary:{post_id}")
|
||||
|
||||
return {
|
||||
"has_ai_reply": has_ai_reply,
|
||||
"summary_available": bool(summary_cached),
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. 实现步骤
|
||||
|
||||
| 步骤 | 任务 | 优先级 |
|
||||
|------|------|--------|
|
||||
| 1 | 创建 ForumAIService | 🟢 高 |
|
||||
| 2 | 实现自动回复 | 🟢 高 |
|
||||
| 3 | 实现摘要生成 | 🟡 中 |
|
||||
| 4 | 实现智能打标 | 🟡 中 |
|
||||
| 5 | 实现智能分类 | 🟡 中 |
|
||||
| 6 | 创建 ForumTools | 🟢 高 |
|
||||
| 7 | 配置定时任务 | 🟡 中 |
|
||||
| 8 | 扩展 API 端点 | 🟡 中 |
|
||||
| 9 | 单元测试 | 🟡 中 |
|
||||
|
||||
---
|
||||
|
||||
## 9. 核心文件变更
|
||||
|
||||
| 文件 | 变更 |
|
||||
|------|------|
|
||||
| `services/forum_ai_service.py` | 新增 |
|
||||
| `services/summary_service.py` | 新增 |
|
||||
| `agents/tools/forum_tools.py` | 新增 |
|
||||
| `agents/prompts/forum_agent.py` | 新增 |
|
||||
| `tasks/forum_auto_reply.py` | 新增 |
|
||||
| `routers/forum.py` | 扩展 AI 端点 |
|
||||
|
||||
---
|
||||
|
||||
## 10. 工作量估算
|
||||
|
||||
| 任务 | 工作量 |
|
||||
|------|--------|
|
||||
| ForumAIService | 1 天 |
|
||||
| 自动回复 | 1 天 |
|
||||
| 摘要/打标/分类 | 1 天 |
|
||||
| ForumTools | 1 天 |
|
||||
| 定时任务 | 0.5 天 |
|
||||
| API 端点 | 0.5 天 |
|
||||
| 单元测试 | 0.5 天 |
|
||||
| **总计** | **5.5 天** |
|
||||
|
||||
---
|
||||
|
||||
## 11. 验收标准
|
||||
|
||||
- [ ] ForumAIService 可正常调用 LLM
|
||||
- [ ] 自动回复功能正常工作
|
||||
- [ ] 摘要生成功能正常
|
||||
- [ ] 智能打标推荐准确
|
||||
- [ ] 智能分类推荐准确
|
||||
- [ ] ForumTools 可被 Agent 调用
|
||||
- [ ] 定时任务正常执行
|
||||
- [ ] API 端点正常工作
|
||||
- [ ] 单元测试覆盖核心逻辑
|
||||
155
development-doc/plan/memory-update/README.md
Normal file
155
development-doc/plan/memory-update/README.md
Normal file
@@ -0,0 +1,155 @@
|
||||
# Jarvis Memory 升级计划索引
|
||||
|
||||
本目录用于存放 Jarvis 记忆系统的分阶段升级规划文档。
|
||||
|
||||
## 文档说明
|
||||
|
||||
| 文件 | 说明 |
|
||||
|------|------|
|
||||
| `README.md` | 总览、阶段关系、实施顺序 |
|
||||
| `phase-m-0-current-state.md` | 当前现状、问题、目标架构 |
|
||||
| `phase-m-1-importance-scoring.md` | 重要性评分系统 |
|
||||
| `phase-m-2-forgetting-system.md` | 遗忘曲线系统 |
|
||||
| `phase-m-3-proactive-reminder.md` | 主动提醒系统 |
|
||||
| `checklist.md` | 执行清单 |
|
||||
|
||||
## 推荐阅读顺序
|
||||
|
||||
1. 先读 `phase-m-0-current-state.md`
|
||||
2. 再按顺序阅读 phase m-1 ~ m-3
|
||||
3. 实施时严格按阶段推进
|
||||
4. 参考 `checklist.md` 进行任务追踪
|
||||
|
||||
---
|
||||
|
||||
## 总体升级原则
|
||||
|
||||
1. **频率追踪** - 每次交互更新记忆频率
|
||||
2. **重要性分层** - 高频/情绪/影响面 → 重要记忆
|
||||
3. **遗忘曲线** - 低频记忆自然衰减
|
||||
4. **主动关心** - 定期生成提醒，而非被动响应
|
||||
5. **可独立推进** - Phase M 可与 Agent Phase 1-5 并行
|
||||
|
||||
---
|
||||
|
||||
## 阶段总览图
|
||||
|
||||
```
|
||||
M.0 ──────────────────────────────────────────────────────────────┐
|
||||
│ 现状与目标 │
|
||||
│ - 当前记忆架构分析 │
|
||||
│ - 短板识别 │
|
||||
│ - 拟人记忆目标 │
|
||||
└────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
M.1 ──────────────────────────────────────────────────────────────┐
|
||||
│ 重要性评分系统 │
|
||||
│ - MemoryFrequencyTracker (频率追踪) │
|
||||
│ - EmotionAnalyzer (情绪分析) │
|
||||
│ - ImpactScorer (影响面评估) │
|
||||
│ │
|
||||
│ 核心文件: services/memory/importance_scorer.py │
|
||||
│ 工作量: 4 天 │
|
||||
└────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
M.2 ──────────────────────────────────────────────────────────────┐
|
||||
│ 遗忘曲线系统 │
|
||||
│ - ForgettingCurve (遗忘曲线) │
|
||||
│ - MemoryDecay (记忆衰减) │
|
||||
│ - ReinforcementTrigger (强化触发) │
|
||||
│ │
|
||||
│ 核心文件: services/memory/forgetting_curve.py │
|
||||
│ 依赖: M.1 │
|
||||
│ 工作量: 3 天 │
|
||||
└────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
M.3 ──────────────────────────────────────────────────────────────┐
|
||||
│ 主动提醒系统 │
|
||||
│ - DailyDigestGenerator (每日摘要) │
|
||||
│ - ReminderScheduler (提醒调度) │
|
||||
│ - ProactiveMemoryInformer (主动提醒) │
|
||||
│ │
|
||||
│ 核心文件: services/memory/proactive_reminder.py │
|
||||
│ 依赖: M.1, M.2 │
|
||||
│ 工作量: 5 天 │
|
||||
└────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 核心借鉴
|
||||
|
||||
| 借鉴点 | 来源 | 难度 |
|
||||
|--------|------|------|
|
||||
| 频率追踪 | 儿童认知发育模型 | 🟢 低 |
|
||||
| 艾宾浩斯遗忘曲线 | 心理学研究 | 🟢 低 |
|
||||
| 重要性评分 | Jarvis 自身需求 | 🟡 中 |
|
||||
| 主动提醒 | 儿童认知发育模型 | 🟡 中 |
|
||||
|
||||
**注：本升级不借鉴 VCPToolBox，因为 VCPToolBox 解决的是「检索精度」问题，而本升级解决的是「记忆价值判断」问题。**
|
||||
|
||||
---
|
||||
|
||||
## 实施顺序
|
||||
|
||||
```
|
||||
M.0 → M.1 → M.2 → M.3
|
||||
│ │ │
|
||||
│ │ └── 主动提醒系统
|
||||
│ └── 遗忘曲线系统
|
||||
└── 现状与目标
|
||||
```
|
||||
|
||||
**注意：** M.1 是基础，M.2 和 M.3 都依赖 M.1。
|
||||
|
||||
---
|
||||
|
||||
## 文件变更追踪
|
||||
|
||||
| Phase | 新增文件 | 修改文件 |
|
||||
|-------|---------|---------|
|
||||
| M.1 | `services/memory/importance_scorer.py`, `services/memory/frequency_tracker.py`, `services/memory/emotion_analyzer.py`, `tests/test_importance_scorer.py` | `models/memory.py`, `services/memory_service.py` |
|
||||
| M.2 | `services/memory/forgetting_curve.py`, `tests/test_forgetting_curve.py` | `models/memory.py`, `services/memory_service.py` |
|
||||
| M.3 | `services/memory/daily_digest.py`, `services/memory/reminder_scheduler.py`, `tests/test_proactive_reminder.py` | `services/memory_service.py`, `services/scheduler_service.py` |
|
||||
|
||||
---
|
||||
|
||||
## 与 Agent Phase 1-5 的关系
|
||||
|
||||
| Agent Phase | Memory 协作内容 |
|
||||
|-------------|----------------|
|
||||
| Phase 1 | Memory 追踪用户交互频率 |
|
||||
| Phase 2 | Memory 服务被 Librarian Agent 调用 |
|
||||
| Phase 3 | 支持动态协作时的记忆共享 |
|
||||
| Phase 4 | Memory 重要性可视化 |
|
||||
| Phase 5 | 高级记忆关联分析 |
|
||||
| **Phase M** | **独立 Memory 升级路径，可与 Phase 1-5 并行推进** |
|
||||
|
||||
---
|
||||
|
||||
## 注意事项
|
||||
|
||||
| 注意事项 | 说明 |
|
||||
|---------|------|
|
||||
| M.1 是基础 | M.2 和 M.3 都依赖 M.1 的重要性评分 |
|
||||
| 渐进式遗忘 | 不是删除，是降权和归档 |
|
||||
| 主动提醒需用户授权 | 提醒推送需要用户明确开启 |
|
||||
| 不改变现有检索逻辑 | Memory 升级是独立于 RAG 的 |
|
||||
|
||||
---
|
||||
|
||||
## 目标：拟人化记忆
|
||||
|
||||
```
|
||||
现在的 Jarvis:
|
||||
用户问什么，Jarvis 答什么，不问就不说
|
||||
|
||||
升级后的 Jarvis:
|
||||
- 知道什么对你重要（频率+情绪+影响面）
|
||||
- 知道什么是你的痛点（反复问的问题）
|
||||
- 会主动提醒你关心的事（不是等用户问）
|
||||
- 知道什么可以忘记（低频记忆自然衰减）
|
||||
```
|
||||
410
development-doc/plan/memory-update/checklist.md
Normal file
410
development-doc/plan/memory-update/checklist.md
Normal file
@@ -0,0 +1,410 @@
|
||||
# Jarvis Memory 升级执行清单
|
||||
|
||||
日期：2026-04-04
|
||||
状态：执行清单
|
||||
升级方向：拟人化记忆系统
|
||||
|
||||
---
|
||||
|
||||
## 使用说明
|
||||
|
||||
- 完成前使用 `- [ ]`
|
||||
- 完成后改成 `- [x]`
|
||||
- Day M.2 默认依赖 Day M.1 的重要性评分完成后再推进
|
||||
- Day M.3 默认依赖 Day M.1 和 M.2 完成后再推进
|
||||
|
||||
---
|
||||
|
||||
## Day M.1：重要性评分系统（4天）
|
||||
|
||||
Day M.1 目标：让 Jarvis 知道「什么对你重要」。
|
||||
|
||||
### Task M.1.1：实现 FrequencyTracker
|
||||
|
||||
- [ ] 新增 `backend/app/services/memory/frequency_tracker.py`
|
||||
|
||||
- [ ] 实现 `FrequencyTracker` 类
|
||||
|
||||
- [ ] 实现 `increment()` 方法
|
||||
```python
|
||||
def increment(self, memory: UserMemory) -> UserMemory:
|
||||
memory.frequency_count += 1
|
||||
memory.last_recalled_at = datetime.now()
|
||||
return memory
|
||||
```
|
||||
|
||||
- [ ] 实现 `get_time_decay()` 方法
|
||||
|
||||
### Task M.1.2：实现 EmotionAnalyzer
|
||||
|
||||
- [ ] 新增 `backend/app/services/memory/emotion_analyzer.py`
|
||||
|
||||
- [ ] 实现 `EmotionAnalyzer` 类
|
||||
|
||||
- [ ] 定义 `EMOTION_KEYWORDS` 字典
|
||||
```python
|
||||
EMOTION_KEYWORDS = {
|
||||
"急": 1.0,
|
||||
"很重要": 0.9,
|
||||
"困扰": 0.8,
|
||||
"担心": 0.7,
|
||||
"想解决": 0.6,
|
||||
"无所谓": 0.1,
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] 实现 `extract()` 方法 - 从文本提取情绪关键词
|
||||
|
||||
- [ ] 实现 `calculate_score()` 方法 - 计算情绪分数
|
||||
|
||||
### Task M.1.3：实现 ImpactEvaluator
|
||||
|
||||
- [ ] 新增 `backend/app/services/memory/impact_evaluator.py`
|
||||
|
||||
- [ ] 实现 `ImpactEvaluator` 类
|
||||
|
||||
- [ ] 实现 `evaluate()` 方法
|
||||
```python
|
||||
def evaluate(self, memory: UserMemory) -> float:
|
||||
# 关联话题越多，影响面越大
|
||||
return min(1.0, len(memory.associated_topics) / IMPACT_THRESHOLD)
|
||||
```
|
||||
|
||||
### Task M.1.4：实现 ImportanceScorer
|
||||
|
||||
- [ ] 新增 `backend/app/services/memory/importance_scorer.py`
|
||||
|
||||
- [ ] 实现 `ImportanceScorer` 类
|
||||
|
||||
- [ ] 实现 `calculate_score()` 综合评分方法
|
||||
```python
|
||||
def calculate_score(self, memory: UserMemory) -> float:
|
||||
frequency = self.tracker.get_frequency_score(memory) * 0.35
|
||||
recency = self.tracker.get_recency_score(memory) * 0.20
|
||||
emotion = self.emotion_analyzer.calculate_score(memory) * 0.25
|
||||
impact = self.impact_evaluator.evaluate(memory) * 0.20
|
||||
return frequency + recency + emotion + impact
|
||||
```
|
||||
|
||||
- [ ] 实现 `get_importance_level()` 方法
|
||||
|
||||
- [ ] 实现 `should_escalate()` 方法
|
||||
|
||||
### Task M.1.5：修改 UserMemory 模型
|
||||
|
||||
- [ ] 修改 `backend/app/models/memory.py`
|
||||
|
||||
- [ ] 增加字段：
|
||||
```python
|
||||
frequency_count: int = 0
|
||||
last_recalled_at: DateTime = None
|
||||
emotion_tags: list[str] = []
|
||||
importance_score: float = 0.5
|
||||
importance_level: str = "medium"
|
||||
associated_topics: list[str] = []
|
||||
```
|
||||
|
||||
### Task M.1.6：集成到 MemoryService
|
||||
|
||||
- [ ] 修改 `backend/app/services/memory_service.py`
|
||||
|
||||
- [ ] 集成 `ImportanceScorer`
|
||||
|
||||
- [ ] 修改 `add_memory()` 方法计算重要性
|
||||
|
||||
- [ ] 修改 `recall_memories()` 方法按重要性排序
|
||||
|
||||
### Task M.1.7：补测试
|
||||
|
||||
- [ ] 新增 `backend/tests/services/test_importance_scorer.py`
|
||||
|
||||
- [ ] 测试频率追踪
|
||||
|
||||
- [ ] 测试情绪分析
|
||||
|
||||
- [ ] 测试重要性评分
|
||||
|
||||
- [ ] 测试重要性等级划分
|
||||
|
||||
### Day M.1 验收
|
||||
|
||||
- [ ] 频率追踪正常（recall_count 每次 +1）
|
||||
- [ ] 情绪识别准确（「急」「很重要」等能识别）
|
||||
- [ ] 重要性分数正确（高频+情绪 = importance >= 0.8）
|
||||
- [ ] 评分影响排序（高重要性记忆排在前面）
|
||||
- [ ] 单元测试覆盖率 > 80%
|
||||
|
||||
---
|
||||
|
||||
## Day M.2：遗忘曲线系统（3天）
|
||||
|
||||
Day M.2 目标：让 Jarvis 知道「什么可以忘记」。
|
||||
|
||||
### Task M.2.1：实现 ForgettingCurve
|
||||
|
||||
- [ ] 新增 `backend/app/services/memory/forgetting_curve.py`
|
||||
|
||||
- [ ] 实现 `ForgettingCurve` 类
|
||||
|
||||
- [ ] 实现 `calculate_decay()` 方法
|
||||
```python
|
||||
def calculate_decay(self, memory: UserMemory) -> float:
|
||||
half_life = self.get_half_life(memory)
|
||||
days = (datetime.now() - memory.last_accessed_at).days
|
||||
return exp(-days / half_life)
|
||||
```
|
||||
|
||||
- [ ] 实现 `get_half_life()` 方法（重要性影响半衰期）
|
||||
|
||||
### Task M.2.2：实现 MemoryDecay
|
||||
|
||||
- [ ] 新增 `backend/app/services/memory/memory_decay.py`
|
||||
|
||||
- [ ] 实现 `MemoryDecay` 类
|
||||
|
||||
- [ ] 实现 `should_archive()` 方法（decay < 0.2）
|
||||
|
||||
- [ ] 实现 `should_deprioritize()` 方法（decay < 0.5）
|
||||
|
||||
- [ ] 实现 `archive_memory()` 方法
|
||||
|
||||
- [ ] 实现 `restore_from_archive()` 方法
|
||||
|
||||
### Task M.2.3：实现 MemoryReinforcement
|
||||
|
||||
- [ ] 新增 `backend/app/services/memory/reinforcement.py`
|
||||
|
||||
- [ ] 实现 `MemoryReinforcement` 类
|
||||
|
||||
- [ ] 实现 `trigger()` 方法（召回时强化）
|
||||
|
||||
- [ ] 实现 `auto_reinforce()` 方法（定期强化 high 级别）
|
||||
|
||||
### Task M.2.4：修改 UserMemory 模型
|
||||
|
||||
- [ ] 修改 `backend/app/models/memory.py`
|
||||
|
||||
- [ ] 增加字段：
|
||||
```python
|
||||
decay_score: float = 1.0
|
||||
is_archived: bool = False
|
||||
last_accessed_at: DateTime = None
|
||||
archive_at: DateTime = None
|
||||
```
|
||||
|
||||
### Task M.2.5：集成到 MemoryService
|
||||
|
||||
- [ ] 修改 `backend/app/services/memory_service.py`
|
||||
|
||||
- [ ] 集成 ForgettingCurve
|
||||
|
||||
- [ ] 修改 recall_memories() 更新 last_accessed_at
|
||||
|
||||
- [ ] 集成 MemoryReinforcement
|
||||
|
||||
### Task M.2.6：添加调度任务
|
||||
|
||||
- [ ] 修改 `backend/app/services/scheduler_service.py`
|
||||
|
||||
- [ ] 添加每日遗忘检查（cron: 03:00）
|
||||
|
||||
- [ ] 添加每周强化任务（cron: 周一 04:00）
|
||||
|
||||
### Task M.2.7：补测试
|
||||
|
||||
- [ ] 新增 `backend/tests/services/test_forgetting_curve.py`
|
||||
|
||||
- [ ] 测试遗忘曲线计算
|
||||
|
||||
- [ ] 测试高重要性记忆衰减慢
|
||||
|
||||
- [ ] 测试归档/恢复
|
||||
|
||||
### Day M.2 验收
|
||||
|
||||
- [ ] 遗忘曲线正确（30 天后 decay ≈ 0.5）
|
||||
- [ ] 高重要性记忆衰减慢（high 衰减速度是 low 的 1/6）
|
||||
- [ ] 归档正常（decay < 0.2 自动归档）
|
||||
- [ ] 恢复正常（归档记忆可以恢复）
|
||||
- [ ] 调度任务正常（每日检查、周强化执行）
|
||||
- [ ] 单元测试覆盖率 > 80%
|
||||
|
||||
---
|
||||
|
||||
## Day M.3：主动提醒系统（6天）
|
||||
|
||||
Day M.3 目标：让 Jarvis 从「等用户问」变成「主动关心」。
|
||||
|
||||
### Task M.3.1：实现 DailyDigestGenerator
|
||||
|
||||
- [ ] 新增 `backend/app/services/memory/daily_digest.py`
|
||||
|
||||
- [ ] 实现 `DailyDigestGenerator` 类
|
||||
|
||||
- [ ] 定义 `DailyDigest` 数据类
|
||||
|
||||
- [ ] 实现 `generate()` 方法
|
||||
```python
|
||||
async def generate(self, user_id: int, date: date = None) -> DailyDigest:
|
||||
# 1. 获取今日对话摘要
|
||||
# 2. 获取高重要性记忆
|
||||
# 3. 获取待解答问题
|
||||
# 4. 生成建议
|
||||
```
|
||||
|
||||
- [ ] 实现 `get_recent_digests()` 方法
|
||||
|
||||
### Task M.3.2：实现 ReminderScheduler
|
||||
|
||||
- [ ] 新增 `backend/app/services/memory/reminder_scheduler.py`
|
||||
|
||||
- [ ] 定义 `Reminder` 数据类
|
||||
|
||||
- [ ] 实现 `ReminderScheduler` 类
|
||||
|
||||
- [ ] 实现 `create_reminder()` 方法
|
||||
|
||||
- [ ] 实现 `get_due_reminders()` 方法
|
||||
|
||||
- [ ] 实现 `snooze()` 方法
|
||||
|
||||
- [ ] 实现 `dismiss()` 方法
|
||||
|
||||
### Task M.3.3：实现 ProactiveInformer
|
||||
|
||||
- [ ] 新增 `backend/app/services/memory/proactive_informer.py`
|
||||
|
||||
- [ ] 实现 `ProactiveInformer` 类
|
||||
|
||||
- [ ] 定义 `TRIGGERS` 配置
|
||||
|
||||
- [ ] 定义 `INFORM_PROBABILITY` 配置
|
||||
|
||||
- [ ] 实现 `should_inform()` 方法
|
||||
|
||||
- [ ] 实现 `get_inform_message()` 方法
|
||||
|
||||
- [ ] 实现 `check_and_inform()` 方法
|
||||
|
||||
### Task M.3.4：创建提醒数据模型
|
||||
|
||||
- [ ] 修改数据库支持 `reminders` 表
|
||||
|
||||
- [ ] 新增 `backend/app/models/reminder.py`
|
||||
|
||||
- [ ] 或在现有模型文件中增加 Reminder 类
|
||||
|
||||
### Task M.3.5：集成到 MemoryService
|
||||
|
||||
- [ ] 修改 `backend/app/services/memory_service.py`
|
||||
|
||||
- [ ] 集成 DailyDigestGenerator
|
||||
|
||||
- [ ] 集成 ProactiveInformer
|
||||
|
||||
- [ ] 修改 recall_memories() 触发主动告知检查
|
||||
|
||||
### Task M.3.6：集成到 SchedulerService
|
||||
|
||||
- [ ] 修改 `backend/app/services/scheduler_service.py`
|
||||
|
||||
- [ ] 添加每日摘要生成（cron: 22:00）
|
||||
|
||||
- [ ] 添加提醒检查任务（cron: 每 15 分钟）
|
||||
|
||||
### Task M.3.7：前端 - 每日摘要展示
|
||||
|
||||
- [ ] 修改前端对话页面
|
||||
|
||||
- [ ] 新增每日摘要卡片组件
|
||||
|
||||
- [ ] 获取和展示今日摘要
|
||||
|
||||
### Task M.3.8：前端 - 主动提醒推送
|
||||
|
||||
- [ ] 新增主动提醒 Toast 组件
|
||||
|
||||
- [ ] 实现稍后/知道了按钮
|
||||
|
||||
- [ ] 推送 WebSocket 集成
|
||||
|
||||
### Task M.3.9：补测试
|
||||
|
||||
- [ ] 新增 `backend/tests/services/test_proactive_reminder.py`
|
||||
|
||||
- [ ] 测试每日摘要生成
|
||||
|
||||
- [ ] 测试提醒创建和调度
|
||||
|
||||
- [ ] 测试主动告知概率
|
||||
|
||||
### Day M.3 验收
|
||||
|
||||
- [ ] 每日摘要生成正常（22:00 自动生成）
|
||||
- [ ] 提醒创建正常（用户可创建提醒）
|
||||
- [ ] 提醒到期触发（定时推送）
|
||||
- [ ] 主动告知概率正确（按配置的概率触发）
|
||||
- [ ] 告知消息自然（像人说话，不生硬）
|
||||
- [ ] 用户可控制（可以关闭主动提醒）
|
||||
- [ ] 单元测试覆盖率 > 80%
|
||||
|
||||
---
|
||||
|
||||
## 总验收清单
|
||||
|
||||
### Phase M.1-M.3 必须完成
|
||||
|
||||
- [ ] 重要性评分系统正常工作
|
||||
- [ ] 遗忘曲线系统正常工作
|
||||
- [ ] 主动提醒系统正常工作
|
||||
- [ ] 单元测试覆盖率 > 80%
|
||||
- [ ] 集成测试通过
|
||||
- [ ] 原有记忆功能无回退
|
||||
|
||||
---
|
||||
|
||||
## 总工作量估算
|
||||
|
||||
| Phase | 工作量 |
|
||||
|-------|--------|
|
||||
| M.1 重要性评分 | 4 天 |
|
||||
| M.2 遗忘曲线 | 3 天 |
|
||||
| M.3 主动提醒 | 6 天 |
|
||||
| **合计** | **13 天** |
|
||||
|
||||
---
|
||||
|
||||
## 产出清单
|
||||
|
||||
| 产出 | 对应 Phase |
|
||||
|------|-----------|
|
||||
| `services/memory/frequency_tracker.py` | M.1 |
|
||||
| `services/memory/emotion_analyzer.py` | M.1 |
|
||||
| `services/memory/impact_evaluator.py` | M.1 |
|
||||
| `services/memory/importance_scorer.py` | M.1 |
|
||||
| `services/memory/forgetting_curve.py` | M.2 |
|
||||
| `services/memory/memory_decay.py` | M.2 |
|
||||
| `services/memory/reinforcement.py` | M.2 |
|
||||
| `services/memory/daily_digest.py` | M.3 |
|
||||
| `services/memory/reminder_scheduler.py` | M.3 |
|
||||
| `services/memory/proactive_informer.py` | M.3 |
|
||||
| `models/memory.py` 更新 | M.1, M.2 |
|
||||
| `models/reminder.py` 新增 | M.3 |
|
||||
| 前端摘要卡片 | M.3 |
|
||||
| 前端提醒 Toast | M.3 |
|
||||
| 单元测试 > 80% | M.1, M.2, M.3 |
|
||||
| 集成测试通过 | M.1, M.2, M.3 |
|
||||
|
||||
---
|
||||
|
||||
## 与 Agent Phase 关系
|
||||
|
||||
| Agent Phase | Memory 协作内容 |
|
||||
|-------------|----------------|
|
||||
| Phase 1 | Memory 追踪用户交互频率 |
|
||||
| Phase 2 | Memory 服务被 Librarian Agent 调用 |
|
||||
| Phase 3 | 支持动态协作时的记忆共享 |
|
||||
| Phase 4 | Memory 重要性可视化 |
|
||||
| Phase 5 | 高级记忆关联分析 |
|
||||
|
||||
**Phase M 可与 Agent Phase 1-5 并行推进。**
|
||||
125
development-doc/plan/memory-update/phase-m-0-current-state.md
Normal file
125
development-doc/plan/memory-update/phase-m-0-current-state.md
Normal file
@@ -0,0 +1,125 @@
|
||||
# Phase M.0：Memory 现状与目标
|
||||
|
||||
日期：2026-04-04
|
||||
状态：已完成
|
||||
升级方向：拟人化记忆系统
|
||||
|
||||
---
|
||||
|
||||
## 1. 本阶段目的
|
||||
|
||||
本文件用于统一背景认知，明确：
|
||||
|
||||
- Jarvis 当前记忆系统处于什么水平
|
||||
- 主要短板是什么
|
||||
- 为什么要升级
|
||||
- 升级后的目标形态是什么
|
||||
|
||||
---
|
||||
|
||||
## 2. 当前 Jarvis Memory 架构
|
||||
|
||||
### 2.1 核心流程
|
||||
|
||||
```
|
||||
用户对话 → MemoryService → Mem0 (facts/preferences/goals)
|
||||
→ BrainService → BrainMemory
|
||||
→ SQLite (memory_summaries)
|
||||
```
|
||||
|
||||
### 2.2 核心文件
|
||||
|
||||
| 文件 | 职责 |
|
||||
|------|------|
|
||||
| `backend/app/services/memory_service.py` | 三层记忆管理 |
|
||||
| `backend/app/services/brain_service.py` | Brain 学习与回忆 |
|
||||
| `backend/app/models/memory.py` | MemorySummary, UserMemory 模型 |
|
||||
| `backend/app/models/brain.py` | BrainEvent, BrainMemory, BrainTag 模型 |
|
||||
|
||||
---
|
||||
|
||||
## 3. 当前能力矩阵
|
||||
|
||||
| 能力 | 状态 | 说明 |
|
||||
|------|------|------|
|
||||
| 短期记忆 | ✅ | messages 表存储当前会话 |
|
||||
| 中期记忆 | ✅ | memory_summaries 跨会话摘要 |
|
||||
| 长期记忆 | ✅ | Mem0 facts/preferences/goals |
|
||||
| 记忆检索 | ✅ | Mem0 语义搜索 + SQLite LIKE |
|
||||
| 记忆重要性 | ❌ | 无重要性评分 |
|
||||
| 频率追踪 | ❌ | 无频率统计 |
|
||||
| 遗忘机制 | ❌ | 存了就不删 |
|
||||
| 主动提醒 | ❌ | 完全被动响应 |
|
||||
|
||||
---
|
||||
|
||||
## 4. 当前短板
|
||||
|
||||
| 短板 | 严重程度 | 影响 |
|
||||
|------|----------|------|
|
||||
| 无重要性评分 | 🔴 高 | 无法区分重要/不重要记忆 |
|
||||
| 无频率追踪 | 🔴 高 | 反复出现的痛点无法识别 |
|
||||
| 无遗忘机制 | 🟡 中 | 存储无限增长，冷门知识占空间 |
|
||||
| 无主动提醒 | 🔴 高 | 用户不问就不说，不像助理 |
|
||||
| 无情绪感知 | 🟡 中 | 无法区分用户急/重要的表达 |
|
||||
|
||||
---
|
||||
|
||||
## 5. 目标架构
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ 用户对话输入 │
|
||||
└─────────────────────────┬───────────────────────────────────┘
|
||||
│
|
||||
┌───────────┴───────────┐
|
||||
│ Importance Scorer │ ← M.1 新增
|
||||
│ - 频率 (你提几次了) │
|
||||
│ - 情绪 (急/重要/困扰) │
|
||||
│ - 影响面 (关联多少) │
|
||||
│ - 时间 (最近/当时) │
|
||||
└───────────┬───────────┘
|
||||
│
|
||||
┌────────────────┼────────────────┐
|
||||
▼ ▼ ▼
|
||||
┌───────────┐ ┌───────────┐ ┌───────────┐
|
||||
│ 高优先级 │ │ 中优先级 │ │ 低优先级 │
|
||||
│ (主动提醒)│ │ (记住) │ │ (归档) │
|
||||
└─────┬─────┘ └─────┬─────┘ └─────┬─────┘
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
形成观点 选择性回忆 自然遗忘
|
||||
主动提醒 需要时召回 (降权归档)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. 与 VCPToolBox 的关系
|
||||
|
||||
| VCPToolBox 能力 | 本升级是否借鉴 | 理由 |
|
||||
|-----------------|---------------|------|
|
||||
| TagMemo 向量检索 | ❌ 不借鉴 | 你需要的是「记住什么」，不是「检索更准」 |
|
||||
| LIF 脉冲扩散 | ❌ 不借鉴 | 同上 |
|
||||
| EPA 语义分析 | ❌ 不借鉴 | 同上 |
|
||||
| 频率追踪 | ✅ 参考 | 这是你核心需要的，VCPToolBox 没有，自己设计 |
|
||||
|
||||
**结论：VCPToolBox 解决的是「知识管理」，本升级解决的是「记忆价值判断」，是两个不同的问题。**
|
||||
|
||||
---
|
||||
|
||||
## 7. 升级后的直观改变
|
||||
|
||||
| 现在 | 升级后 |
|
||||
|------|--------|
|
||||
| 问 3 次同一个问题，Jarvis 每次都当新问题 | 第 2 次就记住，第 3 次能说"你之前问过..." |
|
||||
| 很少用的知识自动沉底 | 低频知识自动归档，需要时能恢复 |
|
||||
| 你问"明天干嘛" → 只能看日程 | 主动说"你昨天说想换工作，提醒你查一下 JD" |
|
||||
| 告诉 Jarvis 的事可能下次就忘了 | 重要的事主动记，冷门的事知道就行 |
|
||||
|
||||
---
|
||||
|
||||
## 8. 本阶段产出要求
|
||||
|
||||
- [x] 团队对 Jarvis 当前记忆问题和目标方向达成一致
|
||||
- [x] 明确了与 VCPToolBox 的关系（不借鉴 VCPToolBox）
|
||||
- [x] 后续 phase 文档能够在这个认知基础上展开
|
||||
@@ -0,0 +1,219 @@
|
||||
# Phase M.1：重要性评分系统
|
||||
|
||||
日期：2026-04-04
|
||||
状态：规划中
|
||||
依赖：无
|
||||
工作量：4 天
|
||||
|
||||
---
|
||||
|
||||
## 1. 本阶段目的
|
||||
|
||||
建立记忆重要性评分体系，让 Jarvis 知道「什么对你重要」。
|
||||
|
||||
核心问题：
|
||||
- 你提了 3 次同一个问题 → 这是你的痛点，应该深入解决
|
||||
- 你说「急」「很重要」「困扰我」 → 情绪标记，应该优先级高
|
||||
- 这个话题关联了多少其他话题 → 影响面越大越重要
|
||||
|
||||
---
|
||||
|
||||
## 2. 核心架构
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ ImportanceScorer │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ calculate_score(memory) → importance_score (0.0-1.0) │
|
||||
│ │
|
||||
│ 评分维度: │
|
||||
│ - frequency_score (频率) × 0.35 │
|
||||
│ - recency_score (时效) × 0.20 │
|
||||
│ - emotion_score (情绪) × 0.25 │
|
||||
│ - impact_score (影响面) × 0.20 │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. 评分维度详解
|
||||
|
||||
### 3.1 Frequency Score (频率)
|
||||
|
||||
```python
|
||||
frequency_score = min(1.0, recall_count / FREQUENCY_THRESHOLD)
|
||||
# FREQUENCY_THRESHOLD = 3 # 提 3 次算高频
|
||||
|
||||
# 同时考虑时间衰减
|
||||
time_decay = exp(-days_since_last_recall / HALF_LIFE_DAYS)
|
||||
# HALF_LIFE_DAYS = 7 # 7 天减半
|
||||
```
|
||||
|
||||
### 3.2 Recency Score (时效性)
|
||||
|
||||
```python
|
||||
recency_score = exp(-days_since_creation / RECENCY_HALF_LIFE)
|
||||
# RECENCY_HALF_LIFE = 30 # 30 天减半
|
||||
|
||||
# 但重要事件例外：即使久远也保持高时效
|
||||
if is_emotion_tagged:
|
||||
recency_score = max(recency_score, 0.7)
|
||||
```
|
||||
|
||||
### 3.3 Emotion Score (情绪)
|
||||
|
||||
```python
|
||||
EMOTION_KEYWORDS = {
|
||||
"急": 1.0,
|
||||
"很重要": 0.9,
|
||||
"困扰": 0.8,
|
||||
"担心": 0.7,
|
||||
"想解决": 0.6,
|
||||
"无所谓": 0.1,
|
||||
}
|
||||
|
||||
emotion_score = max([EMOTION_KEYWORDS.get(kw, 0)
|
||||
for kw in extracted_emotions], default=0.0)
|
||||
```
|
||||
|
||||
### 3.4 Impact Score (影响面)
|
||||
|
||||
```python
|
||||
# 关联多少其他记忆/话题
|
||||
impact_score = min(1.0, associated_memory_count / IMPACT_THRESHOLD)
|
||||
# IMPACT_THRESHOLD = 5 # 关联 5 个算满
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. 核心文件
|
||||
|
||||
### 4.1 新增文件
|
||||
|
||||
| 文件 | 职责 |
|
||||
|------|------|
|
||||
| `services/memory/frequency_tracker.py` | 频率追踪器 |
|
||||
| `services/memory/emotion_analyzer.py` | 情绪分析器 |
|
||||
| `services/memory/impact_evaluator.py` | 影响面评估器 |
|
||||
| `services/memory/importance_scorer.py` | 综合评分器 |
|
||||
|
||||
### 4.2 修改文件
|
||||
|
||||
| 文件 | 修改内容 |
|
||||
|------|---------|
|
||||
| `models/memory.py` | 增加 frequency_count, last_recalled_at, emotion_tags 字段 |
|
||||
| `services/memory_service.py` | 集成 ImportanceScorer |
|
||||
|
||||
---
|
||||
|
||||
## 5. API 设计
|
||||
|
||||
### 5.1 ImportanceScorer
|
||||
|
||||
```python
|
||||
class ImportanceScorer:
|
||||
def calculate_score(self, memory: UserMemory) -> float:
|
||||
"""返回 0.0-1.0 的重要性分数"""
|
||||
|
||||
def get_importance_level(self, memory: UserMemory) -> Literal["high", "medium", "low"]:
|
||||
"""返回重要性等级"""
|
||||
|
||||
def should_escalate(self, memory: UserMemory) -> bool:
|
||||
"""是否应该升级为高优先级记忆"""
|
||||
```
|
||||
|
||||
### 5.2 集成到 MemoryService
|
||||
|
||||
```python
|
||||
class MemoryService:
|
||||
async def add_memory(self, user_id: int, content: str, emotion_tags: list[str] = None):
|
||||
# 添加记忆时计算初始重要性
|
||||
importance = self.scorer.calculate_score(new_memory)
|
||||
|
||||
async def recall_memories(self, query: str, user_id: int, top_k: int = 5):
|
||||
# 检索时考虑重要性排序
|
||||
# 高重要性记忆排在前面
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. 数据模型变更
|
||||
|
||||
### 6.1 UserMemory 扩展
|
||||
|
||||
```python
|
||||
class UserMemory:
|
||||
# 现有字段...
|
||||
frequency_count: int = 0 # 被回忆次数
|
||||
last_recalled_at: DateTime = None # 上次被回忆时间
|
||||
emotion_tags: list[str] = [] # 情绪标签
|
||||
importance_score: float = 0.5 # 重要性分数
|
||||
importance_level: str = "medium" # high/medium/low
|
||||
associated_topics: list[str] = [] # 关联话题
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. 测试设计
|
||||
|
||||
### 7.1 频率追踪测试
|
||||
|
||||
```python
|
||||
def test_frequency_increase():
|
||||
memory = create_memory(frequency_count=0)
|
||||
memory = tracker.increment(memory)
|
||||
assert memory.frequency_count == 1
|
||||
|
||||
def test_frequency_decay_over_time():
|
||||
# 7 天后频率应该衰减
|
||||
pass
|
||||
```
|
||||
|
||||
### 7.2 情绪分析测试
|
||||
|
||||
```python
|
||||
def test_emotion_extraction():
|
||||
text = "这个问题很急，急需解决"
|
||||
emotions = analyzer.extract(text)
|
||||
assert "急" in emotions
|
||||
|
||||
def test_emotion_scoring():
|
||||
score = scorer.calculate_emotion_score(["急", "很重要"])
|
||||
assert score >= 0.9
|
||||
```
|
||||
|
||||
### 7.3 综合评分测试
|
||||
|
||||
```python
|
||||
def test_importance_ranking():
|
||||
memories = [low_freq, high_freq, emotional]
|
||||
ranked = sorter.rank_by_importance(memories)
|
||||
assert ranked[0] == emotional # 情绪标签最重
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. 验收标准
|
||||
|
||||
| 标准 | 说明 |
|
||||
|------|------|
|
||||
| 频率追踪正常 | recall_count 每次召回 +1 |
|
||||
| 情绪识别准确 | 「急」「很重要」等能识别 |
|
||||
| 重要性分数正确 | 高频+情绪 = importance >= 0.8 |
|
||||
| 评分影响排序 | 高重要性记忆排在检索结果前面 |
|
||||
| 单元测试覆盖率 | > 80% |
|
||||
|
||||
---
|
||||
|
||||
## 9. 工作量估算
|
||||
|
||||
| 任务 | 工作量 |
|
||||
|------|--------|
|
||||
| FrequencyTracker | 0.5 天 |
|
||||
| EmotionAnalyzer | 0.5 天 |
|
||||
| ImpactEvaluator | 0.5 天 |
|
||||
| ImportanceScorer | 1 天 |
|
||||
| 模型变更 | 0.5 天 |
|
||||
| 集成到 MemoryService | 0.5 天 |
|
||||
| 测试 | 1 天 |
|
||||
| **合计** | **4 天** |
|
||||
@@ -0,0 +1,228 @@
|
||||
# Phase M.2：遗忘曲线系统
|
||||
|
||||
日期：2026-04-04
|
||||
状态：规划中
|
||||
依赖：M.1 (重要性评分)
|
||||
工作量：3 天
|
||||
|
||||
---
|
||||
|
||||
## 1. 本阶段目的
|
||||
|
||||
实现「选择性遗忘」机制，让 Jarvis 知道「什么可以忘记」。
|
||||
|
||||
核心问题：
|
||||
- 冷门知识不应该一直占存储
|
||||
- 但不能直接删，要有归档机制
|
||||
- 重要记忆应该被强化，不容易忘
|
||||
|
||||
---
|
||||
|
||||
## 2. 核心架构
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ ForgettingCurve │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ decay_score = exp(-time_since_access / half_life) │
|
||||
│ │
|
||||
│ 遗忘策略: │
|
||||
│ - decay_score < 0.2 → 归档到 cold_storage │
|
||||
│ - decay_score < 0.5 → 降权，不参与主动提醒 │
|
||||
│ - importance_level=high → 半衰期延长 3x │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. 艾宾浩斯遗忘曲线模型
|
||||
|
||||
### 3.1 基础遗忘曲线
|
||||
|
||||
```
|
||||
保留率
|
||||
100% |████████████
|
||||
80% |██████████
|
||||
60% |███████
|
||||
40% |█████
|
||||
20% |██
|
||||
0% |________________________ 时间
|
||||
1天 7天 30天 90天
|
||||
```
|
||||
|
||||
### 3.2 Jarvis 遗忘策略
|
||||
|
||||
```python
|
||||
# 基础半衰期：30 天
|
||||
BASE_HALF_LIFE_DAYS = 30
|
||||
|
||||
# 重要性影响半衰期
|
||||
if importance_level == "high":
|
||||
half_life = BASE_HALF_LIFE_DAYS * 3 # 90 天
|
||||
elif importance_level == "medium":
|
||||
half_life = BASE_HALF_LIFE_DAYS * 1 # 30 天
|
||||
else:
|
||||
half_life = BASE_HALF_LIFE_DAYS * 0.5 # 15 天
|
||||
|
||||
# 遗忘分数
|
||||
decay_score = exp(-days_since_access / half_life)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. 核心文件
|
||||
|
||||
### 4.1 新增文件
|
||||
|
||||
| 文件 | 职责 |
|
||||
|------|------|
|
||||
| `services/memory/forgetting_curve.py` | 遗忘曲线计算 |
|
||||
| `services/memory/memory_decay.py` | 记忆衰减处理 |
|
||||
| `services/memory/reinforcement.py` | 记忆强化触发 |
|
||||
|
||||
### 4.2 修改文件
|
||||
|
||||
| 文件 | 修改内容 |
|
||||
|------|---------|
|
||||
| `models/memory.py` | 增加 decay_score, is_archived, last_accessed_at 字段 |
|
||||
| `services/memory_service.py` | 集成遗忘逻辑 |
|
||||
|
||||
---
|
||||
|
||||
## 5. API 设计
|
||||
|
||||
### 5.1 ForgettingCurve
|
||||
|
||||
```python
|
||||
class ForgettingCurve:
|
||||
def calculate_decay(self, memory: UserMemory) -> float:
|
||||
"""返回 0.0-1.0 的保留分数"""
|
||||
|
||||
def should_archive(self, memory: UserMemory) -> bool:
|
||||
"""是否应该归档"""
|
||||
|
||||
def should_deprioritize(self, memory: UserMemory) -> bool:
|
||||
"""是否应该降权（不参与主动提醒）"""
|
||||
```
|
||||
|
||||
### 5.2 Reinforcement
|
||||
|
||||
```python
|
||||
class MemoryReinforcement:
|
||||
def trigger(self, memory_id: int) -> None:
|
||||
"""被召回时触发强化"""
|
||||
# 频率 +1
|
||||
# decay_score 重置
|
||||
|
||||
def auto_reinforce(self, user_id: int) -> None:
|
||||
"""定期自动强化高重要性记忆"""
|
||||
# 每周检查，对 high 级别记忆做轻量强化
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. 归档机制
|
||||
|
||||
### 6.1 热/冷存储分离
|
||||
|
||||
```python
|
||||
# 热存储：活跃记忆，参与检索和主动提醒
|
||||
HOT_MEMORIES = []
|
||||
|
||||
# 冷存储：归档记忆，不参与主动提醒，按需恢复
|
||||
COLD_MEMORIES = []
|
||||
```
|
||||
|
||||
### 6.2 归档恢复
|
||||
|
||||
```python
|
||||
async def restore_from_archive(self, memory_id: int) -> UserMemory:
|
||||
"""从归档恢复记忆"""
|
||||
# 恢复后 decay_score 重置为 0.8
|
||||
# 重新加入热存储
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. 调度任务
|
||||
|
||||
### 7.1 每日遗忘检查
|
||||
|
||||
```python
|
||||
# 每天凌晨执行
|
||||
@scheduler.scheduled_task("cron", hour=3)
|
||||
async def daily_forgetting_check():
|
||||
"""每日遗忘检查"""
|
||||
# 1. 计算所有记忆的 decay_score
|
||||
# 2. 归档 decay < 0.2 的记忆
|
||||
# 3. 降权 decay < 0.5 的记忆
|
||||
# 4. 强化被召回的记忆
|
||||
```
|
||||
|
||||
### 7.2 每周自动强化
|
||||
|
||||
```python
|
||||
@scheduler.scheduled_task("cron", day_of_week="mon", hour=4)
|
||||
async def weekly_reinforcement():
|
||||
"""每周自动强化"""
|
||||
# 对 high 重要性记忆做轻量强化
|
||||
# frequency_count *= 1.1 (上限 10)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. 测试设计
|
||||
|
||||
### 8.1 遗忘曲线测试
|
||||
|
||||
```python
|
||||
def test_decay_after_30_days():
|
||||
memory = create_memory(last_accessed_at=days_ago(30))
|
||||
decay = curve.calculate_decay(memory)
|
||||
assert 0.4 < decay < 0.6 # 约 50% 保留
|
||||
|
||||
def test_high_importance_slower_decay():
|
||||
high = create_memory(importance_level="high", last_accessed_at=days_ago(30))
|
||||
low = create_memory(importance_level="low", last_accessed_at=days_ago(30))
|
||||
assert curve.calculate_decay(high) > curve.calculate_decay(low)
|
||||
```
|
||||
|
||||
### 8.2 归档测试
|
||||
|
||||
```python
|
||||
def test_archive_low_decay():
|
||||
memory = create_memory(decay_score=0.15)
|
||||
assert curve.should_archive(memory) == True
|
||||
|
||||
def test_restore_from_archive():
|
||||
memory = service.restore_from_archive(memory_id)
|
||||
assert memory.is_archived == False
|
||||
assert memory.decay_score > 0.5
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9. 验收标准
|
||||
|
||||
| 标准 | 说明 |
|
||||
|------|------|
|
||||
| 遗忘曲线正确 | 30 天后 decay ≈ 0.5 |
|
||||
| 高重要性记忆衰减慢 | high 级别衰减速度是 low 的 1/6 |
|
||||
| 归档正常 | decay < 0.2 自动归档 |
|
||||
| 恢复正常 | 归档记忆可以恢复 |
|
||||
| 调度任务正常 | 每日检查、周强化执行 |
|
||||
| 单元测试覆盖率 | > 80% |
|
||||
|
||||
---
|
||||
|
||||
## 10. 工作量估算
|
||||
|
||||
| 任务 | 工作量 |
|
||||
|------|--------|
|
||||
| ForgettingCurve | 0.5 天 |
|
||||
| MemoryDecay | 0.5 天 |
|
||||
| Reinforcement | 0.5 天 |
|
||||
| 归档机制 | 0.5 天 |
|
||||
| 调度任务 | 0.5 天 |
|
||||
| 测试 | 0.5 天 |
|
||||
| **合计** | **3 天** |
|
||||
@@ -0,0 +1,360 @@
|
||||
# Phase M.3：主动提醒系统
|
||||
|
||||
日期：2026-04-04
|
||||
状态：规划中
|
||||
依赖：M.1 (重要性评分), M.2 (遗忘曲线)
|
||||
工作量：5 天
|
||||
|
||||
---
|
||||
|
||||
## 1. 本阶段目的
|
||||
|
||||
让 Jarvis 从「等用户问」变成「主动关心」。
|
||||
|
||||
核心问题：
|
||||
- Jarvis 知道用户关心什么（高重要性记忆）
|
||||
- Jarvis 知道用户最近做了什么（每日摘要）
|
||||
- Jarvis 主动提醒，而不是等用户问
|
||||
|
||||
---
|
||||
|
||||
## 2. 核心架构
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ ProactiveReminderSystem │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
||||
│ │ DailyDigest │ + │ Reminder │ + │ MemoryInformer│ │
|
||||
│ │ Generator │ │ Scheduler │ │ │ │
|
||||
│ │ │ │ │ │ │ │
|
||||
│ │ 每日摘要生成 │ │ 提醒调度 │ │ 主动提醒推送 │ │
|
||||
│ └──────────────┘ └──────────────┘ └──────────────┘ │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. 每日摘要生成器 (DailyDigestGenerator)
|
||||
|
||||
### 3.1 输入来源
|
||||
|
||||
```
|
||||
今日输入:
|
||||
- 用户今日对话摘要 (memory_summaries)
|
||||
- 用户今日提到的重点 (high importance memories)
|
||||
- 用户今日创建的任务 (tasks)
|
||||
- 用户今日查阅的知识 (knowledge retrieval logs)
|
||||
```
|
||||
|
||||
### 3.2 输出格式
|
||||
|
||||
```json
|
||||
{
|
||||
"date": "2026-04-04",
|
||||
"summary": "今天你主要在处理工作问题，提到了换工作的想法",
|
||||
"key_points": [
|
||||
{"content": "想换工作", "importance": 0.9, "source": "conversation"},
|
||||
{"content": "项目 deadline 是周五", "importance": 0.8, "source": "task"}
|
||||
],
|
||||
"pending_questions": [
|
||||
{"content": "量子计算的问题还没完全理解", "importance": 0.6}
|
||||
],
|
||||
"suggestions": [
|
||||
{"text": "明天可以继续聊换工作的话题", "reason": "重要性高且今天没深入"},
|
||||
{"text": "量子计算的资料可以找找更通俗的解释", "reason": "还没理解"}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### 3.3 生成时机
|
||||
|
||||
```python
|
||||
# 每天晚上 10 点生成
|
||||
@scheduler.scheduled_task("cron", hour=22)
|
||||
async def generate_daily_digest():
|
||||
"""生成每日摘要"""
|
||||
digest = await generator.generate(user_id)
|
||||
await storage.save(digest)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. 提醒调度器 (ReminderScheduler)
|
||||
|
||||
### 4.1 提醒类型
|
||||
|
||||
| 类型 | 触发条件 | 推送时机 |
|
||||
|------|---------|---------|
|
||||
| 后续提醒 | 用户说「回头提醒我」 | 指定时间 |
|
||||
| 关联提醒 | 提到的话题有关联记忆 | 下次对话时 |
|
||||
| 周期提醒 | 每周/每月固定事项 | 周期首日 |
|
||||
| 遗忘提醒 | 归档记忆被重新提到 | 恢复后提醒 |
|
||||
|
||||
### 4.2 调度逻辑
|
||||
|
||||
```python
|
||||
class ReminderScheduler:
|
||||
def schedule_reminder(self, user_id: int, reminder: Reminder):
|
||||
"""安排提醒"""
|
||||
|
||||
def get_due_reminders(self, user_id: int) -> list[Reminder]:
|
||||
"""获取到期的提醒"""
|
||||
|
||||
def snooze_reminder(self, reminder_id: int, minutes: int):
|
||||
"""推迟提醒"""
|
||||
```
|
||||
|
||||
### 4.3 提醒存储
|
||||
|
||||
```python
|
||||
class Reminder:
|
||||
id: int
|
||||
user_id: int
|
||||
content: str
|
||||
trigger_type: str # "time" / "context" / "periodic"
|
||||
trigger_at: DateTime
|
||||
context: dict # 关联的 memory_id 等
|
||||
status: str # "pending" / "sent" / "snoozed"
|
||||
created_at: DateTime
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. 主动记忆告知器 (ProactiveMemoryInformer)
|
||||
|
||||
### 5.1 触发条件
|
||||
|
||||
```python
|
||||
TRIGGERS = {
|
||||
"high_importance_topic": {
|
||||
"condition": "用户提到高重要性话题",
|
||||
"action": "提及关联记忆",
|
||||
"example": "你说想换工作，要不要看看之前收藏的 JD？"
|
||||
},
|
||||
"repeat_question": {
|
||||
"condition": "用户重复问某个问题",
|
||||
"action": "主动说之前回答过",
|
||||
"example": "你之前问过量子计算，我再给你解释一下？"
|
||||
},
|
||||
"forgotten_context": {
|
||||
"condition": "用户提到已归档的记忆",
|
||||
"action": "提示可以恢复",
|
||||
"example": "这个话题你一个月前聊过，要我恢复一下吗？"
|
||||
},
|
||||
"pending_goal": {
|
||||
"condition": "用户设了目标但没进展",
|
||||
"action": "温和提醒",
|
||||
"example": "你之前说想学 Python，有进展吗？"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 5.2 告知时机
|
||||
|
||||
```python
|
||||
# 不是每次对话都告知，有概率控制
|
||||
INFORM_PROBABILITY = {
|
||||
"high_importance_topic": 0.8, # 高重要性话题 80% 主动提
|
||||
"repeat_question": 1.0, # 重复问题 100% 主动提
|
||||
"forgotten_context": 0.5, # 已归档话题 50% 提示
|
||||
"pending_goal": 0.3, # 待办目标 30% 温和提醒
|
||||
}
|
||||
```
|
||||
|
||||
### 5.3 告知风格
|
||||
|
||||
```python
|
||||
INFORM_STYLE = {
|
||||
"casual": "对了，你之前提到...",
|
||||
"gentle": "不知道你有没有注意到...",
|
||||
"helpful": "我记起你关心这个，要不看看..."
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. 核心文件
|
||||
|
||||
### 6.1 新增文件
|
||||
|
||||
| 文件 | 职责 |
|
||||
|------|------|
|
||||
| `services/memory/daily_digest.py` | 每日摘要生成 |
|
||||
| `services/memory/reminder_scheduler.py` | 提醒调度 |
|
||||
| `services/memory/proactive_informer.py` | 主动告知 |
|
||||
| `services/memory/reminder_model.py` | 提醒数据模型 |
|
||||
|
||||
### 6.2 修改文件
|
||||
|
||||
| 文件 | 修改内容 |
|
||||
|------|---------|
|
||||
| `services/scheduler_service.py` | 集成主动提醒调度 |
|
||||
| `services/memory_service.py` | 集成 ProactiveInformer |
|
||||
| `routers/conversation.py` | 主动告知触发点 |
|
||||
|
||||
---
|
||||
|
||||
## 7. API 设计
|
||||
|
||||
### 7.1 DailyDigestGenerator
|
||||
|
||||
```python
|
||||
class DailyDigestGenerator:
|
||||
async def generate(self, user_id: int, date: date = None) -> DailyDigest:
|
||||
"""生成每日摘要"""
|
||||
|
||||
async def get_recent_digests(self, user_id: int, limit: int = 7) -> list[Digest]:
|
||||
"""获取最近 N 天的摘要"""
|
||||
```
|
||||
|
||||
### 7.2 ReminderScheduler
|
||||
|
||||
```python
|
||||
class ReminderScheduler:
|
||||
async def create_reminder(self, user_id: int, content: str, trigger_at: datetime):
|
||||
"""创建提醒"""
|
||||
|
||||
async def get_due_reminders(self, user_id: int) -> list[Reminder]:
|
||||
"""获取到期提醒"""
|
||||
|
||||
async def snooze(self, reminder_id: int, minutes: int):
|
||||
"""推迟提醒"""
|
||||
```
|
||||
|
||||
### 7.3 ProactiveInformer
|
||||
|
||||
```python
|
||||
class ProactiveInformer:
|
||||
def should_inform(self, user_id: int, trigger_type: str) -> bool:
|
||||
"""是否应该告知"""
|
||||
|
||||
def get_inform_message(self, user_id: int, trigger_type: str, context: dict) -> str:
|
||||
"""生成告知消息"""
|
||||
|
||||
async def check_and_inform(self, conversation_context: dict) -> str | None:
|
||||
"""检查并返回告知消息，无则返回 None"""
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. 前端集成
|
||||
|
||||
### 8.1 每日摘要展示
|
||||
|
||||
```vue
|
||||
<!-- 每日摘要卡片 -->
|
||||
<div v-if="dailyDigest">
|
||||
<h3>今日摘要</h3>
|
||||
<p>{{ dailyDigest.summary }}</p>
|
||||
<div v-for="point in dailyDigest.keyPoints">
|
||||
- {{ point.content }}
|
||||
</div>
|
||||
<div v-if="dailyDigest.suggestions.length">
|
||||
<h4>建议</h4>
|
||||
<p>{{ dailyDigest.suggestions[0].text }}</p>
|
||||
</div>
|
||||
</div>
|
||||
```
|
||||
|
||||
### 8.2 主动提醒推送
|
||||
|
||||
```vue
|
||||
<!-- 主动提醒弹窗 -->
|
||||
<div v-if="activeReminder" class="reminder-toast">
|
||||
<p>{{ activeReminder.content }}</p>
|
||||
<button @click="snooze">稍后</button>
|
||||
<button @click="dismiss">知道了</button>
|
||||
</div>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9. 测试设计
|
||||
|
||||
### 9.1 每日摘要测试
|
||||
|
||||
```python
|
||||
async def test_digest_generation():
|
||||
digest = await generator.generate(user_id=1)
|
||||
assert digest.summary is not None
|
||||
assert len(digest.key_points) > 0
|
||||
|
||||
async def test_digest_includes_high_importance():
|
||||
# 高重要性记忆应该出现在摘要中
|
||||
pass
|
||||
```
|
||||
|
||||
### 9.2 提醒调度测试
|
||||
|
||||
```python
|
||||
async def test_schedule_reminder():
|
||||
reminder = await scheduler.create_reminder(
|
||||
user_id=1,
|
||||
content="检查邮件",
|
||||
trigger_at=datetime.now() + timedelta(hours=1)
|
||||
)
|
||||
assert reminder.status == "pending"
|
||||
|
||||
async def test_get_due_reminders():
|
||||
due = await scheduler.get_due_reminders(user_id=1)
|
||||
assert len(due) >= 0
|
||||
```
|
||||
|
||||
### 9.3 主动告知测试
|
||||
|
||||
```python
|
||||
def test_should_inform_probability():
|
||||
# 高重要性话题 80% 触发
|
||||
count = sum(informer.should_inform(1, "high_importance_topic")
|
||||
for _ in range(100))
|
||||
assert 70 < count < 90 # 允许一点随机波动
|
||||
|
||||
def test_repeat_question_always_informs():
|
||||
# 重复问题 100% 触发
|
||||
assert informer.should_inform(1, "repeat_question") == True
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 10. 验收标准
|
||||
|
||||
| 标准 | 说明 |
|
||||
|------|------|
|
||||
| 每日摘要生成正常 | 22:00 自动生成 |
|
||||
| 提醒创建正常 | 用户可创建提醒 |
|
||||
| 提醒到期触发 | 定时推送 |
|
||||
| 主动告知概率正确 | 按配置的概率触发 |
|
||||
| 告知消息自然 | 像人说话，不生硬 |
|
||||
| 用户可控制 | 可以关闭主动提醒 |
|
||||
| 单元测试覆盖率 | > 80% |
|
||||
|
||||
---
|
||||
|
||||
## 11. 工作量估算
|
||||
|
||||
| 任务 | 工作量 |
|
||||
|------|--------|
|
||||
| DailyDigestGenerator | 1.5 天 |
|
||||
| ReminderScheduler | 1.5 天 |
|
||||
| ProactiveInformer | 1 天 |
|
||||
| 前端摘要展示 | 0.5 天 |
|
||||
| 前端提醒推送 | 0.5 天 |
|
||||
| 测试 | 1 天 |
|
||||
| **合计** | **6 天** |
|
||||
|
||||
---
|
||||
|
||||
## 12. 与现有系统的关系
|
||||
|
||||
```
|
||||
现有 SchedulerService
|
||||
│
|
||||
├── 凌晨任务重建 → 现有功能
|
||||
├── 每日摘要生成 → M.3 新增
|
||||
└── 提醒检查 → M.3 新增
|
||||
```
|
||||
|
||||
提醒系统独立于现有任务系统，但可以复用调度基础设施。
|
||||
157
development-doc/plan/rag-update/README.md
Normal file
157
development-doc/plan/rag-update/README.md
Normal file
@@ -0,0 +1,157 @@
|
||||
# Jarvis RAG 升级计划索引
|
||||
|
||||
本目录用于存放 Jarvis RAG 系统的分阶段升级规划文档。
|
||||
|
||||
## 文档说明
|
||||
|
||||
| 文件 | 说明 |
|
||||
|------|------|
|
||||
| `README.md` | 总览、阶段关系、实施顺序 |
|
||||
| `phase-r-0-current-state.md` | 当前现状、问题、目标架构、VCPToolBox 借鉴 |
|
||||
| `phase-r-1-token-chunking.md` | Token 感知分块优化 |
|
||||
| `phase-r-2-multi-index.md` | 多索引架构 |
|
||||
| `phase-r-3-dynamic-weight.md` | 动态权重增强 |
|
||||
| `phase-r-4-advanced.md` | 高级特性（可选） |
|
||||
| `checklist.md` | 执行清单 |
|
||||
|
||||
## 推荐阅读顺序
|
||||
|
||||
1. 先读 `phase-r-0-current-state.md`
|
||||
2. 再按顺序阅读 phase r-1 ~ r-4
|
||||
3. 实施时严格按阶段推进，R.4 为可选
|
||||
4. 参考 `checklist.md` 进行任务追踪
|
||||
|
||||
---
|
||||
|
||||
## 总体升级原则
|
||||
|
||||
1. **Token 精确控制** - 使用 tiktoken 精确计数
|
||||
2. **多索引分层** - 按知识类型/重要性分离
|
||||
3. **动态适配** - 根据查询特性动态调整检索策略
|
||||
4. **测试优先** - 所有升级都要配套测试
|
||||
5. **可独立推进** - Phase R 可与 Agent Phase 1-5 并行
|
||||
|
||||
---
|
||||
|
||||
## 阶段总览图
|
||||
|
||||
```
|
||||
R.0 ──────────────────────────────────────────────────────────────┐
|
||||
│ 现状与目标 │
|
||||
│ - 当前架构分析 │
|
||||
│ - 短板识别 │
|
||||
│ - VCPToolBox TagMemo V6 借鉴 │
|
||||
└────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
R.1 ──────────────────────────────────────────────────────────────┐
|
||||
│ Token 感知分块优化 │
|
||||
│ - tiktoken 集成 │
|
||||
│ - 智能断句 │
|
||||
│ - 重叠分块 (10% overlap) │
|
||||
│ │
|
||||
│ 核心文件: services/chunker.py │
|
||||
│ 工作量: 3 天 │
|
||||
└────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
R.2 ──────────────────────────────────────────────────────────────┐
|
||||
│ 多索引架构 │
|
||||
│ - Collection 分离策略 │
|
||||
│ - 懒加载 + LRU TTL │
|
||||
│ - 重要性感知检索 │
|
||||
│ │
|
||||
│ 核心文件: services/multi_index.py │
|
||||
│ 依赖: R.1 │
|
||||
│ 工作量: 4 天 │
|
||||
└────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
R.3 ──────────────────────────────────────────────────────────────┐
|
||||
│ 动态权重增强 │
|
||||
│ - QueryAnalyzer │
|
||||
│ - DynamicReranker │
|
||||
│ - CoreTagAwareSearch │
|
||||
│ │
|
||||
│ 核心文件: services/query_analyzer.py, │
|
||||
│ services/dynamic_reranker.py, │
|
||||
│ services/core_tag_search.py │
|
||||
│ 依赖: R.1 │
|
||||
│ 工作量: 4.5 天 │
|
||||
└────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
R.4 ──────────────────────────────────────────────────────────────┐
|
||||
│ 高级特性 (可选) │
|
||||
│ - 语义去重 │
|
||||
│ - 语义分桶 │
|
||||
│ - EPA 分析设计 │
|
||||
│ │
|
||||
│ 状态: 可选 │
|
||||
│ 工作量: 4.5 天 │
|
||||
└────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## VCPToolBox TagMemo V6 核心借鉴
|
||||
|
||||
| 借鉴点 | 实现位置 | 难度 |
|
||||
|--------|---------|------|
|
||||
| Token 感知分块（85%+10% 重叠） | R.1 | 🟢 低 |
|
||||
| 多索引架构 | R.2 | 🟡 中 |
|
||||
| 懒加载 + LRU | R.2 | 🟡 中 |
|
||||
| TagBoost 动态权重 | R.3 | 🟡 中 |
|
||||
| 核心标签系统（1.33x 加权） | R.3 | 🟡 中 |
|
||||
| LIF 脉冲扩散 | R.4 | 🔴 高 |
|
||||
|
||||
---
|
||||
|
||||
## 实施顺序
|
||||
|
||||
```
|
||||
R.0 → R.1 → R.2 → R.3 → (R.4 可选)
|
||||
│ │ │ │
|
||||
│ │ │ └── 语义去重/分桶/PCA
|
||||
│ │ └── 多索引 + 懒加载
|
||||
│ └── Token感知分块
|
||||
└── 现状与目标
|
||||
```
|
||||
|
||||
**注意：** R.1 是基础，R.2 和 R.3 都依赖 R.1；R.4 可选。
|
||||
|
||||
---
|
||||
|
||||
## 文件变更追踪
|
||||
|
||||
| Phase | 新增文件 | 修改文件 |
|
||||
|-------|---------|---------|
|
||||
| R.1 | `services/chunker.py`, `tests/test_chunker.py` | `services/document_service.py` |
|
||||
| R.2 | `services/multi_index.py`, `tests/test_multi_index.py` | `services/knowledge_service.py`, `models/document.py` |
|
||||
| R.3 | `services/query_analyzer.py`, `services/dynamic_reranker.py`, `services/core_tag_search.py`, `tests/test_dynamic_reranker.py` | `services/knowledge_service.py`, `models/document.py` |
|
||||
| R.4 | `services/deduplicator.py`, `services/semantic_bucket.py` (可选) | - |
|
||||
|
||||
---
|
||||
|
||||
## 与 Agent Phase 1-5 的关系
|
||||
|
||||
| Agent Phase | RAG 协作内容 |
|
||||
|-------------|-------------|
|
||||
| Phase 1 | Task Schema 追踪 RAG 任务 |
|
||||
| Phase 2 | RAG 任务可分解给 Librarian Agent |
|
||||
| Phase 3 | 支持多索引动态选择 |
|
||||
| Phase 4 | RAG 检索过程可视化 |
|
||||
| Phase 5 | EPA 分析、语义分桶 |
|
||||
| **Phase R** | **独立 RAG 升级路径，可与 Phase 1-5 并行推进** |
|
||||
|
||||
---
|
||||
|
||||
## 注意事项
|
||||
|
||||
| 注意事项 | 说明 |
|
||||
|---------|------|
|
||||
| R.1 是基础 | R.2 和 R.3 都依赖 R.1 的分块优化 |
|
||||
| Token 精确计数 | 使用 tiktoken，多版本验证 |
|
||||
| 索引分离 | 提供统一检索接口，隐藏内部逻辑 |
|
||||
| 动态权重 | 提供配置项，允许用户调整 |
|
||||
| EPA 高复杂度 | Phase R.4 可选，暂不实现 |
|
||||
327
development-doc/plan/rag-update/checklist.md
Normal file
327
development-doc/plan/rag-update/checklist.md
Normal file
@@ -0,0 +1,327 @@
|
||||
# Jarvis RAG 升级执行清单
|
||||
|
||||
日期：2026-04-03
|
||||
状态：执行清单
|
||||
借鉴来源：VCPToolBox TagMemo V6 架构
|
||||
|
||||
---
|
||||
|
||||
## 使用说明
|
||||
|
||||
- 完成前使用 `- [ ]`
|
||||
- 完成后改成 `- [x]`
|
||||
- Day R.2 默认依赖 Day R.1 的分块优化完成后再推进
|
||||
- Day R.3 默认依赖 Day R.1 的分块优化完成后再推进
|
||||
- Day R.4 为可选特性
|
||||
|
||||
---
|
||||
|
||||
## Day R.1：Token 感知分块优化（3天）
|
||||
|
||||
Day R.1 目标：解决跨块边界信息丢失问题，实现精确的 token 计数和重叠分块。
|
||||
|
||||
### Task R.1.1：集成 tiktoken
|
||||
|
||||
- [ ] 安装 tiktoken 依赖
|
||||
```bash
|
||||
uv add tiktoken
|
||||
```
|
||||
|
||||
- [ ] 新增 `backend/app/services/chunker.py`
|
||||
实现 `TokenAwareChunker` 类，支持 85% 安全边界
|
||||
|
||||
- [ ] 实现 `count_tokens()` 方法
|
||||
|
||||
### Task R.1.2：实现智能断句
|
||||
|
||||
- [ ] 实现 `find_best_breakpoint()` 函数
|
||||
在断点处（标点/空白）智能断开
|
||||
|
||||
- [ ] 实现 `_force_split_long_text()` 方法
|
||||
处理超长句子强制分割
|
||||
|
||||
### Task R.1.3：实现重叠分块
|
||||
|
||||
- [ ] 实现 `chunk_with_overlap()` 方法
|
||||
10% token 重叠，保证上下文连续性
|
||||
|
||||
- [ ] 实现 `_create_overlap()` 方法
|
||||
创建重叠部分
|
||||
|
||||
### Task R.1.4：集成到 DocumentService
|
||||
|
||||
- [ ] 修改 `backend/app/services/document_service.py`
|
||||
集成新的 TokenAwareChunker
|
||||
|
||||
- [ ] 替换原有的 `_build_chunks()` 方法
|
||||
|
||||
### Task R.1.5：补测试
|
||||
|
||||
- [ ] 新增 `backend/tests/services/test_chunker.py`
|
||||
|
||||
- [ ] 测试 Token 计数准确性
|
||||
|
||||
- [ ] 测试智能断句
|
||||
|
||||
- [ ] 测试重叠分块
|
||||
|
||||
### Day R.1 验收
|
||||
|
||||
- [ ] tiktoken 正确集成，token 计数误差 < 1%
|
||||
- [ ] 超长句子不在词汇中间断开
|
||||
- [ ] 重叠分块保证上下文连续性
|
||||
- [ ] 单元测试覆盖率 > 80%
|
||||
- [ ] 文档上传→分块→检索 集成测试通过
|
||||
|
||||
---
|
||||
|
||||
## Day R.2：多索引架构（4天）
|
||||
|
||||
Day R.2 目标：按知识类型/重要性分层，支持懒加载和 LRU 淘汰。
|
||||
|
||||
### Task R.2.1：设计 Collection 分离策略
|
||||
|
||||
- [ ] 新增 `backend/app/services/multi_index.py`
|
||||
|
||||
- [ ] 定义 `MultiIndexManager` 类
|
||||
|
||||
- [ ] 实现 `INDEX_STRATEGIES` 配置
|
||||
- default: 通用文档
|
||||
- important: 重要文档
|
||||
- code: 代码片段
|
||||
- meeting: 会议记录
|
||||
|
||||
- [ ] 实现 `get_collection()` 方法
|
||||
|
||||
### Task R.2.2：实现懒加载 + LRU TTL
|
||||
|
||||
- [ ] 实现 `LazyIndexLoader` 类
|
||||
|
||||
- [ ] 实现 `get_or_load()` 方法
|
||||
|
||||
- [ ] 实现 `sweep()` 方法
|
||||
2小时 TTL 淘汰机制
|
||||
|
||||
### Task R.2.3：实现重要性感知检索
|
||||
|
||||
- [ ] 实现 `retrieve_with_importance()` 方法
|
||||
|
||||
- [ ] important 索引加权 1.2x
|
||||
|
||||
### Task R.2.4：修改 Document 模型
|
||||
|
||||
- [ ] 修改 `backend/app/models/document.py`
|
||||
|
||||
- [ ] 增加 `importance` 字段（Float, default=0.5）
|
||||
|
||||
### Task R.2.5：集成到 KnowledgeService
|
||||
|
||||
- [ ] 修改 `backend/app/services/knowledge_service.py`
|
||||
|
||||
- [ ] 集成 MultiIndexManager
|
||||
|
||||
- [ ] 集成 LazyIndexLoader
|
||||
|
||||
- [ ] 根据 importance 选择索引
|
||||
|
||||
### Task R.2.6：补测试
|
||||
|
||||
- [ ] 新增 `backend/tests/services/test_multi_index.py`
|
||||
|
||||
- [ ] 测试多 Collection 创建
|
||||
|
||||
- [ ] 测试懒加载
|
||||
|
||||
- [ ] 测试 TTL 淘汰
|
||||
|
||||
### Day R.2 验收
|
||||
|
||||
- [ ] 多 Collection 创建成功
|
||||
- [ ] 懒加载索引生效
|
||||
- [ ] TTL 淘汰机制工作
|
||||
- [ ] 重要性感知检索加权生效
|
||||
- [ ] 单元测试覆盖率 > 80%
|
||||
|
||||
---
|
||||
|
||||
## Day R.3：动态权重增强（4.5天）
|
||||
|
||||
Day R.3 目标：根据查询特性动态调整检索策略，支持核心标签加权。
|
||||
|
||||
### Task R.3.1：实现查询特性分析
|
||||
|
||||
- [ ] 新增 `backend/app/services/query_analyzer.py`
|
||||
|
||||
- [ ] 定义 `QueryProfile` 数据类
|
||||
|
||||
- [ ] 实现 `QueryAnalyzer` 类
|
||||
|
||||
- [ ] 实现查询类型检测
|
||||
- 代码相关
|
||||
- 表格相关
|
||||
- 对话式
|
||||
|
||||
- [ ] 实现 `_calc_logic_depth()` 方法
|
||||
|
||||
### Task R.3.2：实现动态 Reranker
|
||||
|
||||
- [ ] 新增 `backend/app/services/dynamic_reranker.py`
|
||||
|
||||
- [ ] 实现 `DynamicReranker` 类
|
||||
|
||||
- [ ] 实现 `_get_weights()` 方法
|
||||
- 代码查询：关键词权重高
|
||||
- 表格查询：标题权重高
|
||||
- 对话式：语义权重高
|
||||
|
||||
- [ ] 实现 `_calc_beta()` 方法
|
||||
|
||||
- [ ] 实现 `rerank()` 方法
|
||||
|
||||
### Task R.3.3：实现核心标签系统
|
||||
|
||||
- [ ] 新增 `backend/app/services/core_tag_search.py`
|
||||
|
||||
- [ ] 实现 `CoreTagAwareSearch` 类
|
||||
|
||||
- [ ] 实现 `CORE_BOOST_FACTOR = 1.33`
|
||||
|
||||
- [ ] 实现 `search()` 方法
|
||||
|
||||
### Task R.3.4：修改 DocumentChunk 模型
|
||||
|
||||
- [ ] 修改 `backend/app/models/document.py`
|
||||
|
||||
- [ ] 增加 `tags` 字段（JSON, default=list）
|
||||
|
||||
- [ ] 增加 `is_core` 字段（Boolean, default=False）
|
||||
|
||||
### Task R.3.5：集成到 KnowledgeService
|
||||
|
||||
- [ ] 修改 `backend/app/services/knowledge_service.py`
|
||||
|
||||
- [ ] 集成 QueryAnalyzer
|
||||
|
||||
- [ ] 集成 DynamicReranker
|
||||
|
||||
- [ ] 集成 CoreTagAwareSearch
|
||||
|
||||
- [ ] 修改 `retrieve()` 方法支持动态权重
|
||||
|
||||
### Task R.3.6：补测试
|
||||
|
||||
- [ ] 新增 `backend/tests/services/test_dynamic_reranker.py`
|
||||
|
||||
- [ ] 测试查询特性分析
|
||||
|
||||
- [ ] 测试动态权重调整
|
||||
|
||||
- [ ] 测试核心标签加权
|
||||
|
||||
### Day R.3 验收
|
||||
|
||||
- [ ] 查询特性分析准确（代码/表格/对话式识别）
|
||||
- [ ] 动态权重根据查询类型调整
|
||||
- [ ] 核心标签检索加权 1.33x
|
||||
- [ ] Rerank 集成测试通过
|
||||
|
||||
---
|
||||
|
||||
## Day R.4：高级特性（可选）（4.5天）
|
||||
|
||||
Day R.4 目标：探索更高级的 RAG 增强技术。
|
||||
|
||||
### Task R.4.1：语义去重
|
||||
|
||||
- [ ] 新增 `backend/app/services/deduplicator.py`
|
||||
|
||||
- [ ] 实现 `SemanticDeduplicator` 类
|
||||
|
||||
- [ ] 实现 `_cosine_similarity()` 方法
|
||||
|
||||
- [ ] 实现 `deduplicate()` 方法
|
||||
|
||||
### Task R.4.2：语义分桶（可选）
|
||||
|
||||
- [ ] 新增 `backend/app/services/semantic_bucket.py`
|
||||
|
||||
- [ ] 实现 `SemanticBucketing` 类
|
||||
|
||||
- [ ] 实现 `bucket_by_topic()` 方法
|
||||
|
||||
### Task R.4.3：EPA 分析设计（可选探索）
|
||||
|
||||
- [ ] 设计 EPA 模块架构
|
||||
|
||||
- [ ] 定义 EPA 接口
|
||||
|
||||
- [ ] 实现残差金字塔算法（伪代码）
|
||||
|
||||
### Day R.4 验收（可选）
|
||||
|
||||
- [ ] 语义去重测试通过
|
||||
- [ ] 语义分桶原型完成（可选）
|
||||
- [ ] EPA 分析方案设计完成（可选实现）
|
||||
|
||||
---
|
||||
|
||||
## 总验收清单
|
||||
|
||||
### Phase R.1-R.3 必须完成
|
||||
|
||||
- [ ] Token 感知分块正常工作
|
||||
- [ ] 多索引架构正常工作
|
||||
- [ ] 动态权重增强正常工作
|
||||
- [ ] 单元测试覆盖率 > 80%
|
||||
- [ ] 集成测试通过
|
||||
- [ ] 原有检索功能无回退
|
||||
|
||||
### Phase R.4 可选完成
|
||||
|
||||
- [ ] 语义去重正常工作
|
||||
- [ ] 语义分桶正常工作（可选）
|
||||
- [ ] EPA 设计文档完成（可选）
|
||||
|
||||
---
|
||||
|
||||
## 总工作量估算
|
||||
|
||||
| Phase | 工作量 |
|
||||
|-------|--------|
|
||||
| R.1 Token 感知分块 | 3 天 |
|
||||
| R.2 多索引架构 | 4 天 |
|
||||
| R.3 动态权重增强 | 4.5 天 |
|
||||
| R.4 高级特性（可选） | 4.5 天 |
|
||||
| **R.1-R.3 必须** | **11.5 天** |
|
||||
| **R.1-R.4 含可选** | **16 天** |
|
||||
|
||||
---
|
||||
|
||||
## 产出清单
|
||||
|
||||
| 产出 | 对应 Phase |
|
||||
|------|-----------|
|
||||
| `services/chunker.py` | R.1 |
|
||||
| `services/multi_index.py` | R.2 |
|
||||
| `services/query_analyzer.py` | R.3 |
|
||||
| `services/dynamic_reranker.py` | R.3 |
|
||||
| `services/core_tag_search.py` | R.3 |
|
||||
| `services/deduplicator.py` | R.4 |
|
||||
| `services/semantic_bucket.py` | R.4（可选） |
|
||||
| `models/document.py` 更新 | R.2, R.3 |
|
||||
| 单元测试 > 80% | R.1, R.2, R.3 |
|
||||
| 集成测试通过 | R.1, R.2, R.3 |
|
||||
|
||||
---
|
||||
|
||||
## 与 Agent Phase 关系
|
||||
|
||||
| Agent Phase | RAG 协作内容 |
|
||||
|-------------|-------------|
|
||||
| Phase 1 | Task Schema 追踪 RAG 任务 |
|
||||
| Phase 2 | RAG 任务可分解给 Librarian Agent |
|
||||
| Phase 3 | 支持多索引动态选择 |
|
||||
| Phase 4 | RAG 检索过程可视化 |
|
||||
| Phase 5 | EPA 分析、语义分桶 |
|
||||
|
||||
**Phase R 可与 Agent Phase 1-5 并行推进。**
|
||||
156
development-doc/plan/rag-update/phase-r-0-current-state.md
Normal file
156
development-doc/plan/rag-update/phase-r-0-current-state.md
Normal file
@@ -0,0 +1,156 @@
|
||||
# Phase R.0：RAG 现状与目标
|
||||
|
||||
日期：2026-04-03
|
||||
状态：已完成
|
||||
借鉴来源：VCPToolBox TagMemo V6 架构
|
||||
|
||||
---
|
||||
|
||||
## 1. 本阶段目的
|
||||
|
||||
本文件用于统一背景认知，明确：
|
||||
|
||||
- Jarvis 当前 RAG 架构处于什么水平
|
||||
- 主要短板是什么
|
||||
- 为什么要升级
|
||||
- 升级后的目标形态是什么
|
||||
- VCPToolBox 给我们什么启发
|
||||
|
||||
---
|
||||
|
||||
## 2. 当前 Jarvis RAG 架构
|
||||
|
||||
### 2.1 核心流程
|
||||
|
||||
```
|
||||
用户上传文档 → DocumentService (解析/分块) → ChromaDB (向量存储) → KnowledgeService (检索)
|
||||
```
|
||||
|
||||
### 2.2 核心文件
|
||||
|
||||
| 文件 | 职责 |
|
||||
|------|------|
|
||||
| `backend/app/services/document_service.py` | 文档上传/解析/分块 |
|
||||
| `backend/app/services/knowledge_service.py` | ChromaDB 向量检索/混合检索 |
|
||||
| `backend/app/models/document.py` | Document/DocumentChunk 数据模型 |
|
||||
|
||||
---
|
||||
|
||||
## 3. 当前能力矩阵
|
||||
|
||||
| 能力 | 状态 | 说明 |
|
||||
|------|------|------|
|
||||
| 多格式文档解析 | ✅ | PDF/MD/TXT/DOCX/CSV/XLSX |
|
||||
| 结构化分块 | ✅ | 基于标题层级、表格、段落 |
|
||||
| 向量检索 | ✅ | ChromaDB 语义相似度 |
|
||||
| 关键词检索 | ✅ | SQL LIKE |
|
||||
| 混合检索 | ✅ | 向量 + 关键词加权 |
|
||||
| Rerank | ✅ | 语义分×0.7 + 关键词×0.2 + 标题×0.1 |
|
||||
| 上下文丰富 | ✅ | 自动获取前/后 chunk |
|
||||
|
||||
---
|
||||
|
||||
## 4. 当前短板
|
||||
|
||||
| 短板 | 严重程度 | 影响 |
|
||||
|------|----------|------|
|
||||
| 无重叠分块 | 🟡 中 | 跨块边界信息丢失 |
|
||||
| 单索引架构 | 🟡 中 | 无法按知识类型/重要性分层 |
|
||||
| 无动态权重 | 🟡 中 | 检索策略静态，不适配查询类型 |
|
||||
| 无 Tag/标签系统 | 🟡 中 | 无法利用语义标签增强检索 |
|
||||
| 无懒加载机制 | 🟢 低 | 大量文档时内存占用高 |
|
||||
| 无遗忘机制 | 🟢 低 | 存储无限增长 |
|
||||
|
||||
---
|
||||
|
||||
## 5. VCPToolBox TagMemo V6 核心借鉴
|
||||
|
||||
### 5.1 核心架构
|
||||
|
||||
```
|
||||
日记文件变化 → TextChunker(Token感知分块85%+10%重叠)
|
||||
→ EmbeddingUtils(并发批量向量化)
|
||||
→ SQLite(元数据) + VexusIndex(Rust HNSW向量索引)
|
||||
```
|
||||
|
||||
### 5.2 TagMemo V6 检索流程
|
||||
|
||||
```
|
||||
Query → EPA分析(逻辑深度L/共振R) → 残差金字塔 → TagBoost(β动态权重)
|
||||
→ LIF脉冲扩散(2跳) → 向量融合 → VexusIndex搜索
|
||||
```
|
||||
|
||||
### 5.3 核心模块
|
||||
|
||||
| 模块 | 功能 |
|
||||
|------|------|
|
||||
| TextChunker | Token 感知分块，85% 安全边界 + 10% 重叠 |
|
||||
| EPA | 语义空间投影分析，识别逻辑深度和跨域共振 |
|
||||
| Residual Pyramid | 残差金字塔，多级剥离捕获微弱信号 |
|
||||
| TagBoost | 动态权重增强，根据查询特性调整 |
|
||||
| LIF Spike | 脉冲扩散，2跳拓扑联想 |
|
||||
| VexusIndex | Rust HNSW 向量索引，高性能检索 |
|
||||
|
||||
### 5.4 关键设计理念
|
||||
|
||||
1. **TagMemo 不是搜索引擎，是记忆联想引擎** - 模拟人类大脑的感知→编码→巩固→检索→重构
|
||||
2. **动态适配** - 根据查询意图动态调整检索策略
|
||||
3. **拓扑涌现** - 基于共现矩阵的脉冲扩散，产生非直观联想
|
||||
|
||||
---
|
||||
|
||||
## 6. 目标架构
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ User Query │
|
||||
└─────────────────────────┬───────────────────────────────────┘
|
||||
│
|
||||
┌───────────┴───────────┐
|
||||
│ Query Analyzer │ ← R.3 新增
|
||||
│ (查询特性分析) │
|
||||
└───────────┬───────────┘
|
||||
│
|
||||
┌────────────────┼────────────────┐
|
||||
▼ ▼ ▼
|
||||
┌─────────┐ ┌───────────┐ ┌──────────────┐
|
||||
│ Default │ │ Important │ │ Code/Meeting │
|
||||
│ Collection│ │ Collection │ │ Collections │
|
||||
└────┬─────┘ └─────┬─────┘ └──────┬───────┘
|
||||
│ │ │
|
||||
└──────────────────┼─────────────────┘
|
||||
▼
|
||||
┌───────────────────────────┐
|
||||
│ Dynamic Reranker │ ← R.3 新增
|
||||
│ (Core Tag Boost + 动态权重)│
|
||||
└───────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────┐
|
||||
│ Search Result │
|
||||
└───────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. 借鉴点映射
|
||||
|
||||
| VCPToolBox 借鉴点 | Jarvis 实现位置 | 优先级 |
|
||||
|-------------------|---------------|--------|
|
||||
| Token 感知分块（85%+10% 重叠） | `services/chunker.py` | 🟢 高 |
|
||||
| 多索引架构 | `services/multi_index.py` | 🟡 中 |
|
||||
| 懒加载 + LRU TTL | `services/multi_index.py` | 🟡 中 |
|
||||
| TagBoost 动态权重 | `services/dynamic_reranker.py` | 🟡 中 |
|
||||
| 核心标签系统（1.33x 加权） | `services/core_tag_search.py` | 🟡 中 |
|
||||
| 语义去重 | `services/deduplicator.py` | 🔴 低 |
|
||||
| 语义分桶 | `services/semantic_bucket.py` | 🔴 低 |
|
||||
| EPA 分析 | - | 🔴 探索 |
|
||||
| LIF 脉冲扩散 | - | 🔴 探索 |
|
||||
|
||||
---
|
||||
|
||||
## 8. 本阶段产出要求
|
||||
|
||||
- [x] 团队对 Jarvis 当前 RAG 问题和目标方向达成一致
|
||||
- [x] VCPToolBox 借鉴点已映射到具体 Phase
|
||||
- [x] 后续 phase 文档能够在这个认知基础上展开
|
||||
188
development-doc/plan/rag-update/phase-r-1-token-chunking.md
Normal file
188
development-doc/plan/rag-update/phase-r-1-token-chunking.md
Normal file
@@ -0,0 +1,188 @@
|
||||
# Phase R.1：Token 感知分块优化
|
||||
|
||||
日期：2026-04-03
|
||||
状态：已规划
|
||||
依赖：R.0（现状与目标）
|
||||
工作量：3 天
|
||||
|
||||
---
|
||||
|
||||
## 1. 本阶段目的
|
||||
|
||||
解决跨块边界信息丢失问题，实现精确的 token 计数和重叠分块。
|
||||
|
||||
---
|
||||
|
||||
## 2. 核心任务
|
||||
|
||||
### Task R.1.1：集成 tiktoken
|
||||
|
||||
**目标：** 使用 tiktoken 精确计算 token 数，85% 安全边界
|
||||
|
||||
**新增文件：** `backend/app/services/chunker.py`
|
||||
|
||||
```python
|
||||
import tiktoken
|
||||
|
||||
class TokenAwareChunker:
|
||||
"""Token 感知分块器，85% 安全边界 + 10% 重叠"""
|
||||
|
||||
def __init__(self, max_tokens: int = 8000, overlap_ratio: float = 0.1):
|
||||
self.encoding = tiktoken.get_encoding("cl100k_base")
|
||||
self.safe_max = int(max_tokens * 0.85)
|
||||
self.overlap_tokens = int(self.safe_max * overlap_ratio)
|
||||
|
||||
def count_tokens(self, text: str) -> int:
|
||||
return len(self.encoding.encode(text))
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task R.1.2：实现智能断句
|
||||
|
||||
**目标：** 在断点处（标点/空白）智能断开，避免在词汇中间断开
|
||||
|
||||
```python
|
||||
BREAK_POINTS = ['\n', '。', '！', '？', '，', '；', '：', ' ', '\t']
|
||||
|
||||
def find_best_breakpoint(text: str, max_pos: int) -> int:
|
||||
"""在 max_pos 附近找到最佳断点（标点/空白处）"""
|
||||
for i in range(max_pos - 1, max(0, max_pos - 200), -1):
|
||||
if text[i] in BREAK_POINTS:
|
||||
return i + 1
|
||||
return max_pos
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task R.1.3：实现重叠分块
|
||||
|
||||
**目标：** 10% token 重叠，保证上下文连续性
|
||||
|
||||
```python
|
||||
def chunk_with_overlap(self, text: str) -> list[dict]:
|
||||
"""带重叠的分块器，上一块末尾作为下一块开头"""
|
||||
sentences = self._split_sentences(text)
|
||||
chunks = []
|
||||
current_chunk = ""
|
||||
current_tokens = 0
|
||||
|
||||
for sentence in sentences:
|
||||
sentence_tokens = self.count_tokens(sentence)
|
||||
|
||||
if sentence_tokens > self.safe_max:
|
||||
# 超长句子强制分割
|
||||
forced = self._force_split_long_text(sentence)
|
||||
chunks.extend(forced)
|
||||
continue
|
||||
|
||||
if current_tokens + sentence_tokens > self.safe_max:
|
||||
chunks.append({"content": current_chunk.strip()})
|
||||
# 创建重叠部分
|
||||
current_chunk = self._create_overlap(sentences, current_tokens)
|
||||
current_tokens = self.count_tokens(current_chunk)
|
||||
|
||||
current_chunk += sentence
|
||||
current_tokens += sentence_tokens
|
||||
|
||||
if current_chunk.strip():
|
||||
chunks.append({"content": current_chunk.strip()})
|
||||
|
||||
return chunks
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. 修改现有文件
|
||||
|
||||
### `backend/app/services/document_service.py`
|
||||
|
||||
集成新的 TokenAwareChunker：
|
||||
|
||||
```python
|
||||
from app.services.chunker import TokenAwareChunker
|
||||
|
||||
class DocumentService:
|
||||
def __init__(self, ...):
|
||||
# ... existing init
|
||||
self.chunker = TokenAwareChunker()
|
||||
|
||||
def _build_chunks(self, parsed: ParsedDocument) -> list[dict]:
|
||||
# 原有逻辑替换为重叠分块
|
||||
chunks = self.chunker.chunk_with_overlap(parsed.summary)
|
||||
for node in parsed.nodes:
|
||||
node_chunks = self.chunker.chunk_with_overlap(node.text)
|
||||
for chunk in node_chunks:
|
||||
chunks.append(chunk)
|
||||
return chunks
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. 新增测试
|
||||
|
||||
**新增文件：** `backend/tests/services/test_chunker.py`
|
||||
|
||||
```python
|
||||
import pytest
|
||||
from app.services.chunker import TokenAwareChunker, find_best_breakpoint
|
||||
|
||||
class TestTokenAwareChunker:
|
||||
def test_token_counting(self):
|
||||
chunker = TokenAwareChunker(max_tokens=100)
|
||||
text = "Hello, world!"
|
||||
assert chunker.count_tokens(text) > 0
|
||||
|
||||
def test_overlap_ratio(self):
|
||||
chunker = TokenAwareChunker(max_tokens=100, overlap_ratio=0.1)
|
||||
assert chunker.overlap_tokens == 10
|
||||
|
||||
def test_safe_max(self):
|
||||
chunker = TokenAwareChunker(max_tokens=100)
|
||||
assert chunker.safe_max == 85
|
||||
|
||||
class TestSmartBreakpoint:
|
||||
def test_find_breakpoint_at_punctuation(self):
|
||||
text = "Hello, world! How are you?"
|
||||
pos = find_best_breakpoint(text, 15)
|
||||
assert text[pos-1] in [',', '!', '?', '。', '！', '？']
|
||||
|
||||
class TestOverlappingChunker:
|
||||
def test_chunks_have_overlap(self):
|
||||
chunker = TokenAwareChunker(max_tokens=50, overlap_ratio=0.2)
|
||||
long_text = "A" * 200 + "." + "B" * 200
|
||||
chunks = chunker.chunk_with_overlap(long_text)
|
||||
assert len(chunks) >= 2
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. 验收标准
|
||||
|
||||
- [ ] tiktoken 正确集成，token 计数误差 < 1%
|
||||
- [ ] 超长句子不在词汇中间断开
|
||||
- [ ] 重叠分块保证上下文连续性
|
||||
- [ ] 单元测试覆盖率 > 80%
|
||||
- [ ] 集成测试通过（文档上传→分块→检索）
|
||||
|
||||
---
|
||||
|
||||
## 6. 变更文件清单
|
||||
|
||||
| 文件 | 操作 | 说明 |
|
||||
|------|------|------|
|
||||
| `backend/app/services/chunker.py` | 新增 | Token 感知分块器 |
|
||||
| `backend/app/services/document_service.py` | 修改 | 集成新的分块器 |
|
||||
| `backend/tests/services/test_chunker.py` | 新增 | 分块器单元测试 |
|
||||
|
||||
---
|
||||
|
||||
## 7. 工作量估算
|
||||
|
||||
| 任务 | 估算 |
|
||||
|------|------|
|
||||
| R.1.1 tiktoken 集成 | 0.5 天 |
|
||||
| R.1.2 智能断句 | 0.5 天 |
|
||||
| R.1.3 重叠分块 | 1 天 |
|
||||
| 测试 + 调试 | 1 天 |
|
||||
| **R.1 总计** | **3 天** |
|
||||
244
development-doc/plan/rag-update/phase-r-2-multi-index.md
Normal file
244
development-doc/plan/rag-update/phase-r-2-multi-index.md
Normal file
@@ -0,0 +1,244 @@
|
||||
# Phase R.2：多索引架构
|
||||
|
||||
日期：2026-04-03
|
||||
状态：已规划
|
||||
依赖：R.1（Token 感知分块）
|
||||
工作量：4 天
|
||||
|
||||
---
|
||||
|
||||
## 1. 本阶段目的
|
||||
|
||||
按知识类型/重要性分层，支持懒加载和 LRU 淘汰。
|
||||
|
||||
---
|
||||
|
||||
## 2. 核心任务
|
||||
|
||||
### Task R.2.1：设计 Collection 分离策略
|
||||
|
||||
**目标：** 按知识类型分离 ChromaDB Collection
|
||||
|
||||
**新增文件：** `backend/app/services/multi_index.py`
|
||||
|
||||
```python
|
||||
class MultiIndexManager:
|
||||
"""多索引管理器，按知识类型分离"""
|
||||
|
||||
INDEX_STRATEGIES = {
|
||||
"default": {
|
||||
"name": "user_{user_id}_default",
|
||||
"description": "通用文档"
|
||||
},
|
||||
"important": {
|
||||
"name": "user_{user_id}_important",
|
||||
"description": "重要文档(1.2x加权)"
|
||||
},
|
||||
"code": {
|
||||
"name": "user_{user_id}_code",
|
||||
"description": "代码片段"
|
||||
},
|
||||
"meeting": {
|
||||
"name": "user_{user_id}_meeting",
|
||||
"description": "会议记录"
|
||||
},
|
||||
}
|
||||
|
||||
def get_collection(self, user_id: str, index_type: str = "default"):
|
||||
name = self.INDEX_STRATEGIES[index_type]["name"].format(user_id=user_id)
|
||||
return self.chroma_client.get_or_create_collection(name=name)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task R.2.2：实现懒加载 + LRU TTL
|
||||
|
||||
**目标：** 2小时 TTL，访问时加载，不访问不加载
|
||||
|
||||
```python
|
||||
import time
|
||||
from threading import Lock
|
||||
|
||||
class LazyIndexLoader:
|
||||
"""懒加载索引，支持 TTL 淘汰"""
|
||||
|
||||
def __init__(self, ttl_seconds: int = 7200):
|
||||
self._cache = {}
|
||||
self._last_used = {}
|
||||
self._lock = Lock()
|
||||
self._ttl = ttl_seconds
|
||||
|
||||
def get_or_load(self, key: str, loader_fn) -> Any:
|
||||
with self._lock:
|
||||
if key in self._cache:
|
||||
self._last_used[key] = time.time()
|
||||
return self._cache[key]
|
||||
|
||||
value = loader_fn()
|
||||
self._cache[key] = value
|
||||
self._last_used[key] = time.time()
|
||||
return value
|
||||
|
||||
def sweep(self):
|
||||
"""清理过期索引"""
|
||||
now = time.time()
|
||||
expired = [
|
||||
k for k, t in self._last_used.items()
|
||||
if now - t > self._ttl
|
||||
]
|
||||
for k in expired:
|
||||
del self._cache[k]
|
||||
del self._last_used[k]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task R.2.3：实现重要性感知检索
|
||||
|
||||
**目标：** important 索引加权 1.2x
|
||||
|
||||
```python
|
||||
async def retrieve_with_importance(
|
||||
self,
|
||||
query: str,
|
||||
user_id: str,
|
||||
top_k: int = 5,
|
||||
) -> list[SearchResult]:
|
||||
"""重要性感知检索，优先返回高重要性文档"""
|
||||
|
||||
# 1. 从 default 索引检索
|
||||
default_results = await self.retrieve(query, user_id, top_k=top_k * 2)
|
||||
|
||||
# 2. 从 important 索引检索
|
||||
important_results = await self.retrieve(
|
||||
query, user_id,
|
||||
collection_name=f"user_{user_id}_important",
|
||||
top_k=top_k
|
||||
)
|
||||
|
||||
# 3. 合并，重要文档加权
|
||||
scored = []
|
||||
for r in default_results:
|
||||
scored.append((r.score * 0.8, r))
|
||||
for r in important_results:
|
||||
scored.append((r.score * 1.2, r)) # 重要文档 1.2x
|
||||
|
||||
scored.sort(key=lambda x: x[0], reverse=True)
|
||||
return [r for _, r in scored[:top_k]]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. 修改现有文件
|
||||
|
||||
### `backend/app/models/document.py`
|
||||
|
||||
增加 `importance` 字段：
|
||||
|
||||
```python
|
||||
class Document(Base):
|
||||
# ... existing fields ...
|
||||
|
||||
importance = Column(Float, default=0.5) # 0.0 ~ 1.0, >0.8 进入 important 索引
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `backend/app/services/knowledge_service.py`
|
||||
|
||||
集成多索引支持：
|
||||
|
||||
```python
|
||||
from app.services.multi_index import MultiIndexManager, LazyIndexLoader
|
||||
|
||||
class KnowledgeService:
|
||||
def __init__(self, ...):
|
||||
# ... existing init
|
||||
self.multi_index = MultiIndexManager(self.chroma_client)
|
||||
self.lazy_loader = LazyIndexLoader(ttl_seconds=7200)
|
||||
|
||||
async def index_document(self, document_id: str, user_id: str, ...):
|
||||
# 根据 importance 选择索引
|
||||
doc = await self._get_document(document_id)
|
||||
if doc.importance >= 0.8:
|
||||
collection = self.multi_index.get_collection(user_id, "important")
|
||||
else:
|
||||
collection = self.multi_index.get_collection(user_id, "default")
|
||||
# ... rest of indexing
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. 新增测试
|
||||
|
||||
**新增文件：** `backend/tests/services/test_multi_index.py`
|
||||
|
||||
```python
|
||||
import pytest
|
||||
from app.services.multi_index import MultiIndexManager, LazyIndexLoader
|
||||
|
||||
class TestMultiIndexManager:
|
||||
def test_get_collection_creates_if_not_exists(self):
|
||||
manager = MultiIndexManager(mock_chroma_client)
|
||||
col = manager.get_collection("user123", "default")
|
||||
assert col is not None
|
||||
|
||||
def test_collection_name_format(self):
|
||||
manager = MultiIndexManager(mock_chroma_client)
|
||||
name = manager.INDEX_STRATEGIES["important"]["name"].format(user_id="user123")
|
||||
assert name == "user_user123_important"
|
||||
|
||||
class TestLazyIndexLoader:
|
||||
def test_get_or_load_caches(self):
|
||||
loader = LazyIndexLoader()
|
||||
load_fn = lambda: {"data": "test"}
|
||||
|
||||
result1 = loader.get_or_load("key1", load_fn)
|
||||
result2 = loader.get_or_load("key1", load_fn)
|
||||
|
||||
# 第二次调用应该返回缓存的结果，而不是重新加载
|
||||
assert result1 is result2
|
||||
|
||||
def test_sweep_removes_expired(self):
|
||||
loader = LazyIndexLoader(ttl_seconds=1)
|
||||
loader.get_or_load("key1", lambda: "value1")
|
||||
|
||||
import time
|
||||
time.sleep(1.1) # 等待过期
|
||||
|
||||
loader.sweep()
|
||||
assert "key1" not in loader._cache
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. 验收标准
|
||||
|
||||
- [ ] 多 Collection 创建成功
|
||||
- [ ] 懒加载索引生效（访问时加载，不访问不加载）
|
||||
- [ ] TTL 淘汰机制工作（2小时无访问自动卸载）
|
||||
- [ ] 重要性感知检索加权生效
|
||||
- [ ] 单元测试覆盖率 > 80%
|
||||
|
||||
---
|
||||
|
||||
## 6. 变更文件清单
|
||||
|
||||
| 文件 | 操作 | 说明 |
|
||||
|------|------|------|
|
||||
| `backend/app/services/multi_index.py` | 新增 | 多索引管理器 |
|
||||
| `backend/app/services/knowledge_service.py` | 修改 | 集成多索引支持 |
|
||||
| `backend/app/models/document.py` | 修改 | 增加 importance 字段 |
|
||||
| `backend/tests/services/test_multi_index.py` | 新增 | 多索引单元测试 |
|
||||
|
||||
---
|
||||
|
||||
## 7. 工作量估算
|
||||
|
||||
| 任务 | 估算 |
|
||||
|------|------|
|
||||
| R.2.1 Collection 分离策略 | 1 天 |
|
||||
| R.2.2 懒加载 + LRU | 1 天 |
|
||||
| R.2.3 重要性感知检索 | 0.5 天 |
|
||||
| 测试 + 调试 | 1.5 天 |
|
||||
| **R.2 总计** | **4 天** |
|
||||
290
development-doc/plan/rag-update/phase-r-3-dynamic-weight.md
Normal file
290
development-doc/plan/rag-update/phase-r-3-dynamic-weight.md
Normal file
@@ -0,0 +1,290 @@
|
||||
# Phase R.3：动态权重增强
|
||||
|
||||
日期：2026-04-03
|
||||
状态：已规划
|
||||
依赖：R.1（Token 感知分块）
|
||||
工作量：4.5 天
|
||||
|
||||
---
|
||||
|
||||
## 1. 本阶段目的
|
||||
|
||||
根据查询特性动态调整检索策略，支持核心标签加权。
|
||||
|
||||
---
|
||||
|
||||
## 2. 核心任务
|
||||
|
||||
### Task R.3.1：实现查询特性分析
|
||||
|
||||
**目标：** 分析查询类型（代码/表格/对话式）
|
||||
|
||||
**新增文件：** `backend/app/services/query_analyzer.py`
|
||||
|
||||
```python
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
|
||||
@dataclass
|
||||
class QueryProfile:
|
||||
logic_depth: float # 逻辑深度 (0-1): 意图明确程度
|
||||
is_code_related: bool # 是否代码相关
|
||||
is_table_related: bool # 是否表格相关
|
||||
keyword_density: float # 关键词密度
|
||||
is_conversational: bool # 是否对话式查询
|
||||
|
||||
class QueryAnalyzer:
|
||||
CODE_KEYWORDS = {'code', 'function', 'class', 'api', 'python', 'js', 'bug', '函数', '代码'}
|
||||
TABLE_KEYWORDS = {'table', 'sheet', 'excel', 'csv', 'column', 'row', '数据', '统计', '表格', '列', '行'}
|
||||
|
||||
def analyze(self, query: str) -> QueryProfile:
|
||||
words = set(re.findall(r'\w+', query.lower()))
|
||||
|
||||
return QueryProfile(
|
||||
logic_depth=self._calc_logic_depth(query),
|
||||
is_code_related=bool(words & self.CODE_KEYWORDS),
|
||||
is_table_related=bool(words & self.TABLE_KEYWORDS),
|
||||
keyword_density=len(words) / max(len(query), 1),
|
||||
is_conversational=self._is_conversational(query),
|
||||
)
|
||||
|
||||
def _calc_logic_depth(self, query: str) -> float:
|
||||
"""计算逻辑深度：问句、具体名词越多越聚焦"""
|
||||
question_markers = ['how', 'why', 'what', 'which', '哪个', '如何', '为什么', '怎么']
|
||||
has_question = any(q in query.lower() for q in question_markers)
|
||||
has_specific_terms = len(re.findall(r'\w{5,}', query)) > 3
|
||||
return 0.8 if (has_question and has_specific_terms) else 0.5
|
||||
|
||||
def _is_conversational(self, query: str) -> bool:
|
||||
"""判断是否为对话式查询"""
|
||||
conversational_patterns = ['你', '我想', '能不能', '可以帮我', 'what do you think']
|
||||
return any(p in query for p in conversational_patterns)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task R.3.2：实现动态 Reranker
|
||||
|
||||
**目标：** 根据查询类型动态调整语义/关键词/标题权重
|
||||
|
||||
**新增文件：** `backend/app/services/dynamic_reranker.py`
|
||||
|
||||
```python
|
||||
import json
|
||||
from dataclasses import dataclass
|
||||
|
||||
class DynamicReranker:
|
||||
"""动态 Reranker，根据查询特性调整权重"""
|
||||
|
||||
def rerank(
|
||||
self,
|
||||
query: str,
|
||||
results: list[SearchResult],
|
||||
analyzer: QueryAnalyzer
|
||||
) -> list[SearchResult]:
|
||||
profile = analyzer.analyze(query)
|
||||
weights = self._get_weights(profile)
|
||||
beta = self._calc_beta(profile)
|
||||
|
||||
scored = []
|
||||
for r in results:
|
||||
score = r.score * weights["semantic"]
|
||||
score += self._keyword_score(query, r.content) * weights["keyword"]
|
||||
score += self._title_score(query, r.document_title) * weights["title"]
|
||||
|
||||
# 表格内容加分
|
||||
if profile.is_table_related:
|
||||
meta = json.loads(r.metadata_ or "{}")
|
||||
if meta.get("content_type") == "table_schema":
|
||||
score += 0.25
|
||||
elif meta.get("content_type") == "table_rows":
|
||||
score += 0.15
|
||||
|
||||
score *= beta
|
||||
scored.append((score, r))
|
||||
|
||||
scored.sort(key=lambda x: x[0], reverse=True)
|
||||
return [r for _, r in scored]
|
||||
|
||||
def _get_weights(self, profile: QueryProfile) -> dict:
|
||||
if profile.is_code_related:
|
||||
return {"semantic": 0.55, "keyword": 0.35, "title": 0.10}
|
||||
elif profile.is_table_related:
|
||||
return {"semantic": 0.50, "keyword": 0.30, "title": 0.20}
|
||||
elif profile.is_conversational:
|
||||
return {"semantic": 0.85, "keyword": 0.10, "title": 0.05}
|
||||
else:
|
||||
return {"semantic": 0.70, "keyword": 0.20, "title": 0.10}
|
||||
|
||||
def _calc_beta(self, profile: QueryProfile) -> float:
|
||||
"""计算动态 Beta：逻辑深度高时加大语义权重"""
|
||||
if profile.logic_depth > 0.7:
|
||||
return 1.2 # 意图明确，加大权重
|
||||
elif profile.logic_depth < 0.4:
|
||||
return 0.8 # 意图模糊，降低权重
|
||||
return 1.0
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task R.3.3：实现核心标签系统
|
||||
|
||||
**目标：** 核心标签 1.33x 加权
|
||||
|
||||
**新增文件：** `backend/app/services/core_tag_search.py`
|
||||
|
||||
```python
|
||||
class CoreTagAwareSearch:
|
||||
"""核心标签感知检索"""
|
||||
|
||||
CORE_BOOST_FACTOR = 1.33 # 33% 加权
|
||||
|
||||
async def search(
|
||||
self,
|
||||
query: str,
|
||||
user_id: str,
|
||||
core_tags: list[str] = None,
|
||||
base_search_fn: callable
|
||||
) -> list[SearchResult]:
|
||||
results = await base_search_fn(query, user_id)
|
||||
|
||||
if core_tags:
|
||||
for r in results:
|
||||
meta = json.loads(r.metadata_ or "{}")
|
||||
chunk_tags = meta.get("tags", [])
|
||||
|
||||
if any(tag in chunk_tags for tag in core_tags):
|
||||
r.score *= self.CORE_BOOST_FACTOR
|
||||
|
||||
return sorted(results, key=lambda x: x.score, reverse=True)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. 修改现有文件
|
||||
|
||||
### `backend/app/models/document.py`
|
||||
|
||||
增加 `tags` 和 `is_core` 字段：
|
||||
|
||||
```python
|
||||
class DocumentChunk(Base):
|
||||
# ... existing fields ...
|
||||
|
||||
tags = Column(JSON, default=list) # ["重要", "代码", "架构"]
|
||||
is_core = Column(Boolean, default=False) # 是否核心切片
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `backend/app/services/knowledge_service.py`
|
||||
|
||||
集成动态权重：
|
||||
|
||||
```python
|
||||
from app.services.query_analyzer import QueryAnalyzer
|
||||
from app.services.dynamic_reranker import DynamicReranker
|
||||
from app.services.core_tag_search import CoreTagAwareSearch
|
||||
|
||||
class KnowledgeService:
|
||||
def __init__(self, ...):
|
||||
# ... existing init
|
||||
self.query_analyzer = QueryAnalyzer()
|
||||
self.dynamic_reranker = DynamicReranker()
|
||||
self.core_tag_search = CoreTagAwareSearch()
|
||||
|
||||
async def retrieve(self, query: str, user_id: str, ..., core_tags: list[str] = None) -> list[SearchResult]:
|
||||
# ... existing retrieval logic ...
|
||||
|
||||
# 动态 Rerank
|
||||
results = self.dynamic_reranker.rerank(
|
||||
query, results, self.query_analyzer
|
||||
)
|
||||
|
||||
# 核心标签加权
|
||||
if core_tags:
|
||||
results = await self.core_tag_search.search(
|
||||
query, user_id, core_tags,
|
||||
lambda q, u: results # 使用已检索的结果
|
||||
)
|
||||
|
||||
return results
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. 新增测试
|
||||
|
||||
**新增文件：** `backend/tests/services/test_dynamic_reranker.py`
|
||||
|
||||
```python
|
||||
import pytest
|
||||
from app.services.query_analyzer import QueryAnalyzer, QueryProfile
|
||||
from app.services.dynamic_reranker import DynamicReranker
|
||||
|
||||
class TestQueryAnalyzer:
|
||||
def test_code_query_detection(self):
|
||||
analyzer = QueryAnalyzer()
|
||||
profile = analyzer.analyze("请解释这段 Python 代码")
|
||||
assert profile.is_code_related is True
|
||||
|
||||
def test_table_query_detection(self):
|
||||
analyzer = QueryAnalyzer()
|
||||
profile = analyzer.analyze("统计这个 Excel 表格的总和")
|
||||
assert profile.is_table_related is True
|
||||
|
||||
def test_conversational_detection(self):
|
||||
analyzer = QueryAnalyzer()
|
||||
profile = analyzer.analyze("我想了解一下")
|
||||
assert profile.is_conversational is True
|
||||
|
||||
class TestDynamicReranker:
|
||||
def test_code_query_weights(self):
|
||||
reranker = DynamicReranker()
|
||||
analyzer = QueryAnalyzer()
|
||||
|
||||
profile = QueryProfile(
|
||||
logic_depth=0.5,
|
||||
is_code_related=True,
|
||||
is_table_related=False,
|
||||
keyword_density=0.3,
|
||||
is_conversational=False
|
||||
)
|
||||
|
||||
weights = reranker._get_weights(profile)
|
||||
assert weights["keyword"] > weights["semantic"] * 0.5 # 代码查询关键词权重较高
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. 验收标准
|
||||
|
||||
- [ ] 查询特性分析准确（代码/表格/对话式识别）
|
||||
- [ ] 动态权重根据查询类型调整
|
||||
- [ ] 核心标签检索加权 1.33x
|
||||
- [ ] Rerank 集成测试通过
|
||||
|
||||
---
|
||||
|
||||
## 6. 变更文件清单
|
||||
|
||||
| 文件 | 操作 | 说明 |
|
||||
|------|------|------|
|
||||
| `backend/app/services/query_analyzer.py` | 新增 | 查询特性分析 |
|
||||
| `backend/app/services/dynamic_reranker.py` | 新增 | 动态 Reranker |
|
||||
| `backend/app/services/core_tag_search.py` | 新增 | 核心标签检索 |
|
||||
| `backend/app/services/knowledge_service.py` | 修改 | 集成动态权重 |
|
||||
| `backend/app/models/document.py` | 修改 | 增加 tags/is_core 字段 |
|
||||
| `backend/tests/services/test_dynamic_reranker.py` | 新增 | 动态 Reranker 测试 |
|
||||
|
||||
---
|
||||
|
||||
## 7. 工作量估算
|
||||
|
||||
| 任务 | 估算 |
|
||||
|------|------|
|
||||
| R.3.1 查询特性分析 | 1 天 |
|
||||
| R.3.2 动态 Reranker | 1 天 |
|
||||
| R.3.3 核心标签系统 | 1 天 |
|
||||
| 测试 + 调试 | 1.5 天 |
|
||||
| **R.3 总计** | **4.5 天** |
|
||||
255
development-doc/plan/rag-update/phase-r-4-advanced.md
Normal file
255
development-doc/plan/rag-update/phase-r-4-advanced.md
Normal file
@@ -0,0 +1,255 @@
|
||||
# Phase R.4：高级特性（可选）
|
||||
|
||||
日期：2026-04-03
|
||||
状态：已规划（可选）
|
||||
工作量：4.5 天
|
||||
|
||||
---
|
||||
|
||||
## 1. 本阶段目的
|
||||
|
||||
探索更高级的 RAG 增强技术。
|
||||
|
||||
> **注意：** 本阶段为可选特性，不影响核心功能。根据实际需求决定是否实施。
|
||||
|
||||
---
|
||||
|
||||
## 2. 核心任务
|
||||
|
||||
### Task R.4.1：语义去重
|
||||
|
||||
**目标：** 消除冗余检索结果
|
||||
|
||||
**新增文件：** `backend/app/services/deduplicator.py`
|
||||
|
||||
```python
|
||||
import numpy as np
|
||||
|
||||
class SemanticDeduplicator:
|
||||
"""语义去重，消除冗余检索结果"""
|
||||
|
||||
DEDUP_THRESHOLD = 0.88 # 余弦相似度阈值
|
||||
|
||||
def deduplicate(
|
||||
self,
|
||||
results: list[SearchResult],
|
||||
embeddings: list[np.ndarray]
|
||||
) -> list[SearchResult]:
|
||||
if len(results) <= 1:
|
||||
return results
|
||||
|
||||
# 计算余弦相似度矩阵
|
||||
n = len(results)
|
||||
similarity_matrix = np.zeros((n, n))
|
||||
|
||||
for i in range(n):
|
||||
for j in range(i + 1, n):
|
||||
sim = self._cosine_similarity(embeddings[i], embeddings[j])
|
||||
similarity_matrix[i][j] = sim
|
||||
similarity_matrix[j][i] = sim
|
||||
|
||||
# 贪心去重
|
||||
keep = [True] * n
|
||||
for i in range(n):
|
||||
if not keep[i]:
|
||||
continue
|
||||
for j in range(i + 1, n):
|
||||
if keep[j] and similarity_matrix[i][j] > self.DEDUP_THRESHOLD:
|
||||
keep[j] = False
|
||||
|
||||
return [r for r, k in zip(results, keep) if k]
|
||||
|
||||
def _cosine_similarity(self, a: np.ndarray, b: np.ndarray) -> float:
|
||||
return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task R.4.2：语义分桶（可选）
|
||||
|
||||
**目标：** 按主题自动组织检索结果
|
||||
|
||||
**新增文件：** `backend/app/services/semantic_bucket.py`
|
||||
|
||||
```python
|
||||
from collections import defaultdict
|
||||
import numpy as np
|
||||
|
||||
class SemanticBucketing:
|
||||
"""语义分桶，按主题自动组织检索结果"""
|
||||
|
||||
async def bucket_by_topic(
|
||||
self,
|
||||
results: list[SearchResult],
|
||||
embeddings: list[np.ndarray]
|
||||
) -> dict[str, list[SearchResult]]:
|
||||
# 使用层次聚类
|
||||
from sklearn.cluster import AgglomerativeClustering
|
||||
|
||||
n_clusters = min(5, len(results))
|
||||
if n_clusters < 2:
|
||||
return {"default": results}
|
||||
|
||||
clusterer = AgglomerativeClustering(n_clusters=n_clusters)
|
||||
labels = clusterer.fit_predict(np.array(embeddings))
|
||||
|
||||
buckets = defaultdict(list)
|
||||
for r, label in zip(results, labels):
|
||||
buckets[f"topic_{label}"].append(r)
|
||||
|
||||
# 按每个桶内最高分排序
|
||||
sorted_buckets = {}
|
||||
for name, items in buckets.items():
|
||||
sorted_items = sorted(items, key=lambda x: x.score, reverse=True)
|
||||
sorted_buckets[name] = sorted_items
|
||||
|
||||
return sorted_buckets
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task R.4.3：EPA 分析设计（探索）
|
||||
|
||||
**目标：** 语义空间投影分析方案设计
|
||||
|
||||
```python
|
||||
class EPAModule:
|
||||
"""
|
||||
EPA: Embedding Projection Analysis
|
||||
|
||||
分析向量在语义空间中的投影，识别：
|
||||
- 逻辑深度 (Logic Depth): 意图聚焦程度
|
||||
- 熵 (Entropy): 信息散乱程度
|
||||
- 共振 (Resonance): 跨域关联程度
|
||||
|
||||
注意：此模块为高级特性，复杂度高，建议后续探索。
|
||||
"""
|
||||
|
||||
def project(self, vector: np.ndarray) -> dict:
|
||||
"""
|
||||
返回语义投影结果：
|
||||
- logic_depth: 0~1, 高=意图聚焦
|
||||
- entropy: 0~1, 高=信息散乱
|
||||
- resonance: 跨域共振程度
|
||||
- dominant_axes: 主要语义轴
|
||||
"""
|
||||
raise NotImplementedError("EPA 模块探索中")
|
||||
|
||||
def detect_cross_domain_resonance(self, vector: np.ndarray) -> dict:
|
||||
"""
|
||||
检测跨域共振：
|
||||
- 当查询同时触及多个正交语义轴时触发
|
||||
- 返回共振强度和涉及的主要领域
|
||||
"""
|
||||
raise NotImplementedError("EPA 模块探索中")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. 新增测试
|
||||
|
||||
```python
|
||||
# backend/tests/services/test_deduplicator.py
|
||||
|
||||
class TestSemanticDeduplicator:
|
||||
def test_deduplicate_similar_results(self):
|
||||
dedup = SemanticDeduplicator()
|
||||
|
||||
results = [
|
||||
SearchResult(chunk_id="1", score=0.9, ...),
|
||||
SearchResult(chunk_id="2", score=0.85, ...),
|
||||
SearchResult(chunk_id="3", score=0.8, ...),
|
||||
]
|
||||
embeddings = [
|
||||
np.array([0.1, 0.2, 0.3]),
|
||||
np.array([0.11, 0.21, 0.31]), # 与第一个高度相似
|
||||
np.array([0.9, 0.8, 0.7]), # 与第一个不相似
|
||||
]
|
||||
|
||||
deduped = dedup.deduplicate(results, embeddings)
|
||||
assert len(deduped) < len(results) # 应该去掉一些重复结果
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. 验收标准
|
||||
|
||||
- [ ] 语义去重测试通过
|
||||
- [ ] 语义分桶原型完成（可选）
|
||||
- [ ] EPA 分析方案设计完成（可选实现）
|
||||
|
||||
---
|
||||
|
||||
## 5. 变更文件清单
|
||||
|
||||
| 文件 | 操作 | 说明 |
|
||||
|------|------|------|
|
||||
| `backend/app/services/deduplicator.py` | 新增 | 语义去重 |
|
||||
| `backend/app/services/semantic_bucket.py` | 新增（可选） | 语义分桶 |
|
||||
| `backend/tests/services/test_deduplicator.py` | 新增 | 去重测试 |
|
||||
|
||||
---
|
||||
|
||||
## 6. 工作量估算
|
||||
|
||||
| 任务 | 估算 | 状态 |
|
||||
|------|------|------|
|
||||
| R.4.1 语义去重 | 1.5 天 | 必须 |
|
||||
| R.4.2 语义分桶 | 2 天 | 可选 |
|
||||
| R.4.3 EPA 设计 | 1 天 | 可选 |
|
||||
| **R.4 总计（必须）** | **1.5 天** | |
|
||||
| **R.4 总计（含可选）** | **4.5 天** | |
|
||||
|
||||
---
|
||||
|
||||
## 7. EPA 分析详细设计（供后续参考）
|
||||
|
||||
### 7.1 核心概念
|
||||
|
||||
EPA (Embedding Projection Analysis) 受 VCPToolBox TagMemo V6 启发，用于分析查询向量在语义空间中的投影特征。
|
||||
|
||||
### 7.2 关键指标
|
||||
|
||||
| 指标 | 定义 | 计算方式 |
|
||||
|------|------|----------|
|
||||
| Logic Depth | 意图聚焦程度 | 通过计算投影熵值判断 |
|
||||
| Entropy | 信息散乱程度 | 向量分布的熵 |
|
||||
| Resonance | 跨域共振 | 查询跨越多个语义轴的程度 |
|
||||
|
||||
### 7.3 动态 Beta 公式
|
||||
|
||||
```
|
||||
β = σ(L · log(1 + R) - S · noise_penalty)
|
||||
```
|
||||
|
||||
- L: Logic Depth
|
||||
- R: Resonance
|
||||
- S: 噪音程度
|
||||
- σ: 归一化函数
|
||||
|
||||
### 7.4 残差金字塔
|
||||
|
||||
对查询向量进行多级剥离：
|
||||
|
||||
1. 首轮匹配 → 获取主要语义
|
||||
2. 计算残差 → 提取被掩盖的微弱信号
|
||||
3. 递归搜索 → 直到 90% 能量被解释
|
||||
|
||||
### 7.5 LIF 脉冲扩散
|
||||
|
||||
模拟神经元的脉冲传导：
|
||||
|
||||
1. 种子节点激活
|
||||
2. 沿共现矩阵向外扩散（2跳限制）
|
||||
3. 阈值过滤噪音
|
||||
4. 涌现拓扑关联
|
||||
|
||||
---
|
||||
|
||||
## 8. 风险与注意事项
|
||||
|
||||
| 风险 | 影响 | 缓解措施 |
|
||||
|------|------|----------|
|
||||
| EPA 实现复杂度高 | 高 | Phase R.4 可选，暂不实现 |
|
||||
| 聚类计算开销 | 中 | 限制聚类数量，使用高效算法 |
|
||||
| 去重阈值调参 | 中 | 提供配置项，允许用户调整 |
|
||||
601
development-doc/plan/rag-update/phase-r-rag-upgrade.md
Normal file
601
development-doc/plan/rag-update/phase-r-rag-upgrade.md
Normal file
@@ -0,0 +1,601 @@
|
||||
# Phase R：RAG 系统升级专项
|
||||
|
||||
日期：2026-04-03
|
||||
状态：已规划
|
||||
借鉴来源：VCPToolBox TagMemo V6 架构
|
||||
|
||||
---
|
||||
|
||||
## R.0 当前现状与目标
|
||||
|
||||
### R.0.1 当前 Jarvis RAG 架构
|
||||
|
||||
```
|
||||
用户上传文档 → DocumentService (解析/分块) → ChromaDB (向量存储) → KnowledgeService (检索)
|
||||
```
|
||||
|
||||
**核心文件：**
|
||||
- `backend/app/services/document_service.py` - 文档上传/解析/分块
|
||||
- `backend/app/services/knowledge_service.py` - ChromaDB 向量检索/混合检索
|
||||
- `backend/app/models/document.py` - Document/DocumentChunk 数据模型
|
||||
|
||||
### R.0.2 当前能力矩阵
|
||||
|
||||
| 能力 | 状态 | 说明 |
|
||||
|------|------|------|
|
||||
| 多格式文档解析 | ✅ | PDF/MD/TXT/DOCX/CSV/XLSX |
|
||||
| 结构化分块 | ✅ | 基于标题层级、表格、段落 |
|
||||
| 向量检索 | ✅ | ChromaDB 语义相似度 |
|
||||
| 关键词检索 | ✅ | SQL LIKE |
|
||||
| 混合检索 | ✅ | 向量 + 关键词加权 |
|
||||
| Rerank | ✅ | 语义分*0.7 + 关键词*0.2 + 标题*0.1 |
|
||||
| 上下文丰富 | ✅ | 自动获取前/后 chunk |
|
||||
|
||||
### R.0.3 当前短板
|
||||
|
||||
| 短板 | 严重程度 | 影响 |
|
||||
|------|----------|------|
|
||||
| 无重叠分块 | 🟡 中 | 跨块边界信息丢失 |
|
||||
| 单索引架构 | 🟡 中 | 无法按知识类型/重要性分层 |
|
||||
| 无动态权重 | 🟡 中 | 检索策略静态，不适配查询类型 |
|
||||
| 无 Tag/标签系统 | 🟡 中 | 无法利用语义标签增强检索 |
|
||||
| 无懒加载机制 | 🟢 低 | 大量文档时内存占用高 |
|
||||
| 无遗忘机制 | 🟢 低 | 存储无限增长 |
|
||||
|
||||
### R.0.4 VCPToolBox TagMemo 核心借鉴
|
||||
|
||||
```
|
||||
日记文件变化 → TextChunker(Token感知分块85%+10%重叠)
|
||||
→ EmbeddingUtils(并发批量向量化)
|
||||
→ SQLite(元数据) + VexusIndex(Rust HNSW向量索引)
|
||||
```
|
||||
|
||||
**TagMemo V6 检索流程：**
|
||||
```
|
||||
Query → EPA分析(逻辑深度L/共振R) → 残差金字塔 → TagBoost(β动态权重)
|
||||
→ LIF脉冲扩散(2跳) → 向量融合 → VexusIndex搜索
|
||||
```
|
||||
|
||||
### R.0.5 目标架构
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ User Query │
|
||||
└─────────────────────────┬───────────────────────────────────┘
|
||||
│
|
||||
┌───────────┴───────────┐
|
||||
│ Query Analyzer │ ← R.3 新增
|
||||
│ (查询特性分析) │
|
||||
└───────────┬───────────┘
|
||||
│
|
||||
┌────────────────┼────────────────┐
|
||||
▼ ▼ ▼
|
||||
┌─────────┐ ┌───────────┐ ┌──────────────┐
|
||||
│ Default │ │ Important │ │ Code/Meeting │
|
||||
│ Collection│ │ Collection │ │ Collections │
|
||||
└────┬─────┘ └─────┬─────┘ └──────┬───────┘
|
||||
│ │ │
|
||||
└──────────────────┼─────────────────┘
|
||||
▼
|
||||
┌───────────────────────────┐
|
||||
│ Dynamic Reranker │ ← R.3 新增
|
||||
│ (Core Tag Boost + 动态权重)│
|
||||
└───────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────┐
|
||||
│ Search Result │
|
||||
└───────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## R.1 Token 感知分块优化
|
||||
|
||||
**目标：** 解决跨块边界信息丢失问题，实现精确的 token 计数和重叠分块
|
||||
|
||||
### R.1.1 核心任务
|
||||
|
||||
#### Task R.1.1.1：集成 tiktoken
|
||||
|
||||
```python
|
||||
# services/chunker.py (新增)
|
||||
import tiktoken
|
||||
|
||||
class TokenAwareChunker:
|
||||
"""Token 感知分块器，85% 安全边界 + 10% 重叠"""
|
||||
|
||||
def __init__(self, max_tokens: int = 8000, overlap_ratio: float = 0.1):
|
||||
self.encoding = tiktoken.get_encoding("cl100k_base")
|
||||
self.safe_max = int(max_tokens * 0.85)
|
||||
self.overlap_tokens = int(self.safe_max * overlap_ratio)
|
||||
|
||||
def count_tokens(self, text: str) -> int:
|
||||
return len(self.encoding.encode(text))
|
||||
```
|
||||
|
||||
#### Task R.1.1.2：实现智能断句
|
||||
|
||||
```python
|
||||
BREAK_POINTS = ['\n', '。', '！', '？', '，', '；', '：', ' ', '\t']
|
||||
|
||||
def find_best_breakpoint(text: str, max_pos: int) -> int:
|
||||
"""在 max_pos 附近找到最佳断点（标点/空白处）"""
|
||||
for i in range(max_pos - 1, max(0, max_pos - 200), -1):
|
||||
if text[i] in BREAK_POINTS:
|
||||
return i + 1
|
||||
return max_pos
|
||||
```
|
||||
|
||||
#### Task R.1.1.3：实现重叠分块
|
||||
|
||||
```python
|
||||
def chunk_with_overlap(self, text: str) -> list[dict]:
|
||||
"""带重叠的分块器，上一块末尾作为下一块开头"""
|
||||
sentences = self._split_sentences(text)
|
||||
chunks = []
|
||||
current_chunk = ""
|
||||
current_tokens = 0
|
||||
|
||||
for sentence in sentences:
|
||||
sentence_tokens = self.count_tokens(sentence)
|
||||
|
||||
if sentence_tokens > self.safe_max:
|
||||
# 超长句子强制分割
|
||||
forced = self._force_split_long_text(sentence)
|
||||
chunks.extend(forced)
|
||||
continue
|
||||
|
||||
if current_tokens + sentence_tokens > self.safe_max:
|
||||
chunks.append({"content": current_chunk.strip()})
|
||||
# 创建重叠部分
|
||||
current_chunk = self._create_overlap(sentences, current_tokens)
|
||||
current_tokens = self.count_tokens(current_chunk)
|
||||
|
||||
current_chunk += sentence
|
||||
current_tokens += sentence_tokens
|
||||
|
||||
if current_chunk.strip():
|
||||
chunks.append({"content": current_chunk.strip()})
|
||||
|
||||
return chunks
|
||||
```
|
||||
|
||||
### R.1.2 验收标准
|
||||
|
||||
- [ ] tiktoken 正确集成，token 计数误差 < 1%
|
||||
- [ ] 超长句子不在词汇中间断开
|
||||
- [ ] 重叠分块保证上下文连续性
|
||||
- [ ] 单元测试覆盖率 > 80%
|
||||
|
||||
### R.1.3 变更文件
|
||||
|
||||
| 文件 | 操作 | 说明 |
|
||||
|------|------|------|
|
||||
| `services/chunker.py` | 新增 | Token 感知分块器 |
|
||||
| `services/document_service.py` | 修改 | 集成新的分块器 |
|
||||
| `tests/test_chunker.py` | 新增 | 分块器单元测试 |
|
||||
|
||||
### R.1.4 工作量估算
|
||||
|
||||
| 任务 | 估算 |
|
||||
|------|------|
|
||||
| R.1.1.1 tiktoken 集成 | 0.5 天 |
|
||||
| R.1.1.2 智能断句 | 0.5 天 |
|
||||
| R.1.1.3 重叠分块 | 1 天 |
|
||||
| 测试 + 调试 | 1 天 |
|
||||
| **R.1 总计** | **3 天** |
|
||||
|
||||
---
|
||||
|
||||
## R.2 多索引架构
|
||||
|
||||
**目标：** 按知识类型/重要性分层，支持懒加载和 LRU 淘汰
|
||||
|
||||
### R.2.1 核心任务
|
||||
|
||||
#### Task R.2.1.1：设计 Collection 分离策略
|
||||
|
||||
```python
|
||||
# services/multi_index.py (新增)
|
||||
class MultiIndexManager:
|
||||
"""多索引管理器，按知识类型分离"""
|
||||
|
||||
INDEX_STRATEGIES = {
|
||||
"default": {"name": "user_{user_id}_default", "description": "通用文档"},
|
||||
"important": {"name": "user_{user_id}_important", "description": "重要文档(1.2x加权)"},
|
||||
"code": {"name": "user_{user_id}_code", "description": "代码片段"},
|
||||
"meeting": {"name": "user_{user_id}_meeting", "description": "会议记录"},
|
||||
}
|
||||
|
||||
def get_collection(self, user_id: str, index_type: str = "default"):
|
||||
name = self.INDEX_STRATEGIES[index_type]["name"].format(user_id=user_id)
|
||||
return self.chroma_client.get_or_create_collection(name=name)
|
||||
```
|
||||
|
||||
#### Task R.2.1.2：实现懒加载 + LRU TTL
|
||||
|
||||
```python
|
||||
import time
|
||||
from threading import Lock
|
||||
|
||||
class LazyIndexLoader:
|
||||
"""懒加载索引，支持 TTL 淘汰"""
|
||||
|
||||
def __init__(self, ttl_seconds: int = 7200):
|
||||
self._cache = {}
|
||||
self._last_used = {}
|
||||
self._lock = Lock()
|
||||
self._ttl = ttl_seconds
|
||||
|
||||
def get_or_load(self, key: str, loader_fn) -> Any:
|
||||
with self._lock:
|
||||
if key in self._cache:
|
||||
self._last_used[key] = time.time()
|
||||
return self._cache[key]
|
||||
|
||||
value = loader_fn()
|
||||
self._cache[key] = value
|
||||
self._last_used[key] = time.time()
|
||||
return value
|
||||
|
||||
def sweep(self):
|
||||
"""清理过期索引"""
|
||||
now = time.time()
|
||||
expired = [k for k, t in self._last_used.items() if now - t > self._ttl]
|
||||
for k in expired:
|
||||
del self._cache[k]
|
||||
del self._last_used[k]
|
||||
```
|
||||
|
||||
#### Task R.2.1.3：实现重要性感知检索
|
||||
|
||||
```python
|
||||
async def retrieve_with_importance(
|
||||
self,
|
||||
query: str,
|
||||
user_id: str,
|
||||
importance_threshold: float = 0.0,
|
||||
top_k: int = 5,
|
||||
) -> list[SearchResult]:
|
||||
"""重要性感知检索，优先返回高重要性文档"""
|
||||
|
||||
# 1. 从 default 索引检索
|
||||
default_results = await self.retrieve(query, user_id, top_k=top_k * 2)
|
||||
|
||||
# 2. 从 important 索引检索
|
||||
important_results = await self.retrieve(
|
||||
query, user_id,
|
||||
collection_name=f"user_{user_id}_important",
|
||||
top_k=top_k
|
||||
)
|
||||
|
||||
# 3. 合并，重要文档加权
|
||||
scored = []
|
||||
for r in default_results:
|
||||
scored.append((r.score * 0.8, r))
|
||||
for r in important_results:
|
||||
scored.append((r.score * 1.2, r)) # 重要文档 1.2x
|
||||
|
||||
scored.sort(key=lambda x: x[0], reverse=True)
|
||||
return [r for _, r in scored[:top_k]]
|
||||
```
|
||||
|
||||
### R.2.2 验收标准
|
||||
|
||||
- [ ] 多 Collection 创建成功
|
||||
- [ ] 懒加载索引生效（访问时加载，不访问不加载）
|
||||
- [ ] TTL 淘汰机制工作（2小时无访问自动卸载）
|
||||
- [ ] 重要性感知检索加权生效
|
||||
|
||||
### R.2.3 变更文件
|
||||
|
||||
| 文件 | 操作 | 说明 |
|
||||
|------|------|------|
|
||||
| `services/multi_index.py` | 新增 | 多索引管理器 |
|
||||
| `services/knowledge_service.py` | 修改 | 集成多索引支持 |
|
||||
| `models/document.py` | 修改 | 增加 importance 字段 |
|
||||
| `tests/test_multi_index.py` | 新增 | 多索引单元测试 |
|
||||
|
||||
### R.2.4 工作量估算
|
||||
|
||||
| 任务 | 估算 |
|
||||
|------|------|
|
||||
| R.2.1.1 Collection 分离策略 | 1 天 |
|
||||
| R.2.1.2 懒加载 + LRU | 1 天 |
|
||||
| R.2.1.3 重要性感知检索 | 0.5 天 |
|
||||
| 测试 + 调试 | 1.5 天 |
|
||||
| **R.2 总计** | **4 天** |
|
||||
|
||||
---
|
||||
|
||||
## R.3 动态权重增强
|
||||
|
||||
**目标：** 根据查询特性动态调整检索策略，支持核心标签加权
|
||||
|
||||
### R.3.1 核心任务
|
||||
|
||||
#### Task R.3.1.1：实现查询特性分析
|
||||
|
||||
```python
|
||||
# services/query_analyzer.py (新增)
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
|
||||
@dataclass
|
||||
class QueryProfile:
|
||||
logic_depth: float # 逻辑深度 (0-1): 意图明确程度
|
||||
is_code_related: bool # 是否代码相关
|
||||
is_table_related: bool # 是否表格相关
|
||||
keyword_density: float # 关键词密度
|
||||
is_conversational: bool # 是否对话式查询
|
||||
|
||||
class QueryAnalyzer:
|
||||
CODE_KEYWORDS = {'code', 'function', 'class', 'api', 'python', 'js', 'bug'}
|
||||
TABLE_KEYWORDS = {'table', 'sheet', 'excel', 'csv', 'column', 'row', '数据', '统计'}
|
||||
|
||||
def analyze(self, query: str) -> QueryProfile:
|
||||
words = set(re.findall(r'\w+', query.lower()))
|
||||
|
||||
return QueryProfile(
|
||||
logic_depth=self._calc_logic_depth(query),
|
||||
is_code_related=bool(words & self.CODE_KEYWORDS),
|
||||
is_table_related=bool(words & self.TABLE_KEYWORDS),
|
||||
keyword_density=len(words) / max(len(query), 1),
|
||||
is_conversational=self._is_conversational(query),
|
||||
)
|
||||
```
|
||||
|
||||
#### Task R.3.1.2：实现动态 Reranker
|
||||
|
||||
```python
|
||||
# services/dynamic_reranker.py (新增)
|
||||
class DynamicReranker:
|
||||
def rerank(self, query: str, results: list[SearchResult]) -> list[SearchResult]:
|
||||
profile = QueryAnalyzer().analyze(query)
|
||||
|
||||
# 根据查询类型调整权重
|
||||
weights = self._get_weights(profile)
|
||||
beta = self._calc_beta(profile)
|
||||
|
||||
scored = []
|
||||
for r in results:
|
||||
score = r.score * weights["semantic"]
|
||||
score += self._keyword_score(query, r.content) * weights["keyword"]
|
||||
score += self._title_score(query, r.document_title) * weights["title"]
|
||||
|
||||
# 表格内容加分
|
||||
if profile.is_table_related:
|
||||
meta = json.loads(r.metadata_ or "{}")
|
||||
if meta.get("content_type") == "table_schema":
|
||||
score += 0.25
|
||||
|
||||
score *= beta
|
||||
scored.append((score, r))
|
||||
|
||||
scored.sort(key=lambda x: x[0], reverse=True)
|
||||
return [r for _, r in scored]
|
||||
|
||||
def _get_weights(self, profile: QueryProfile) -> dict:
|
||||
if profile.is_code_related:
|
||||
return {"semantic": 0.55, "keyword": 0.35, "title": 0.10}
|
||||
elif profile.is_table_related:
|
||||
return {"semantic": 0.50, "keyword": 0.30, "title": 0.20}
|
||||
elif profile.is_conversational:
|
||||
return {"semantic": 0.85, "keyword": 0.10, "title": 0.05}
|
||||
else:
|
||||
return {"semantic": 0.70, "keyword": 0.20, "title": 0.10}
|
||||
```
|
||||
|
||||
#### Task R.3.1.3：实现核心标签系统
|
||||
|
||||
```python
|
||||
# 在 models/document.py 中增加 tags 字段
|
||||
class DocumentChunk(Base):
|
||||
tags = Column(JSON, default=list) # ["重要", "代码", "架构"]
|
||||
is_core = Column(Boolean, default=False) # 是否核心切片
|
||||
|
||||
# services/core_tag_search.py (新增)
|
||||
class CoreTagAwareSearch:
|
||||
CORE_BOOST_FACTOR = 1.33 # 33% 加权
|
||||
|
||||
async def search(self, query: str, user_id: str,
|
||||
core_tags: list[str] = None) -> list[SearchResult]:
|
||||
results = await self.base_search(query, user_id)
|
||||
|
||||
if core_tags:
|
||||
for r in results:
|
||||
meta = json.loads(r.metadata_ or "{}")
|
||||
chunk_tags = meta.get("tags", [])
|
||||
|
||||
if any(tag in chunk_tags for tag in core_tags):
|
||||
r.score *= self.CORE_BOOST_FACTOR
|
||||
|
||||
return sorted(results, key=lambda x: x.score, reverse=True)
|
||||
```
|
||||
|
||||
### R.3.2 验收标准
|
||||
|
||||
- [ ] 查询特性分析准确（代码/表格/对话式识别）
|
||||
- [ ] 动态权重根据查询类型调整
|
||||
- [ ] 核心标签检索加权 1.33x
|
||||
- [ ] Rerank 集成测试通过
|
||||
|
||||
### R.3.3 变更文件
|
||||
|
||||
| 文件 | 操作 | 说明 |
|
||||
|------|------|------|
|
||||
| `services/query_analyzer.py` | 新增 | 查询特性分析 |
|
||||
| `services/dynamic_reranker.py` | 新增 | 动态 Reranker |
|
||||
| `services/core_tag_search.py` | 新增 | 核心标签检索 |
|
||||
| `services/knowledge_service.py` | 修改 | 集成动态权重 |
|
||||
| `models/document.py` | 修改 | 增加 tags/is_core 字段 |
|
||||
| `tests/test_dynamic_reranker.py` | 新增 | 动态 Reranker 测试 |
|
||||
|
||||
### R.3.4 工作量估算
|
||||
|
||||
| 任务 | 估算 |
|
||||
|------|------|
|
||||
| R.3.1.1 查询特性分析 | 1 天 |
|
||||
| R.3.1.2 动态 Reranker | 1 天 |
|
||||
| R.3.1.3 核心标签系统 | 1 天 |
|
||||
| 测试 + 调试 | 1.5 天 |
|
||||
| **R.3 总计** | **4.5 天** |
|
||||
|
||||
---
|
||||
|
||||
## R.4 高级特性（可选）
|
||||
|
||||
**目标：** 探索更高级的 RAG 增强技术
|
||||
|
||||
### R.4.1 Task R.4.1.1：语义去重
|
||||
|
||||
```python
|
||||
class SemanticDeduplicator:
|
||||
DEDUP_THRESHOLD = 0.88
|
||||
|
||||
def deduplicate(self, results, embeddings) -> list:
|
||||
"""消除冗余检索结果"""
|
||||
# 计算余弦相似度矩阵
|
||||
# 贪心去重
|
||||
...
|
||||
```
|
||||
|
||||
### R.4.2 Task R.4.2.1：语义分桶
|
||||
|
||||
```python
|
||||
class SemanticBucketing:
|
||||
async def bucket_by_topic(self, results, embeddings) -> dict:
|
||||
"""按主题自动组织检索结果"""
|
||||
# 使用聚类算法
|
||||
...
|
||||
```
|
||||
|
||||
### R.4.3 Task R.4.3.1：EPA 分析（探索）
|
||||
|
||||
```python
|
||||
class EPAModule:
|
||||
"""
|
||||
EPA: Embedding Projection Analysis
|
||||
高复杂度，Phase R.4 探索
|
||||
"""
|
||||
pass # 暂不实现
|
||||
```
|
||||
|
||||
### R.4.4 验收标准
|
||||
|
||||
- [ ] 语义去重测试通过
|
||||
- [ ] 语义分桶原型完成
|
||||
- [ ] EPA 分析方案设计完成（可选实现）
|
||||
|
||||
### R.4.5 工作量估算
|
||||
|
||||
| 任务 | 估算 |
|
||||
|------|------|
|
||||
| R.4.1.1 语义去重 | 1.5 天 |
|
||||
| R.4.2.1 语义分桶 | 2 天 |
|
||||
| R.4.3.1 EPA 设计 | 1 天 |
|
||||
| **R.4 总计（可选）** | **4.5 天** |
|
||||
|
||||
---
|
||||
|
||||
## R.5 阶段总结与产出
|
||||
|
||||
### R.5.1 完整实施路径
|
||||
|
||||
```
|
||||
R.0 ──────────────────────────────────────────────────────────────┐
|
||||
│ 现状与目标 │
|
||||
│ - 当前架构分析 │
|
||||
│ - 短板识别 │
|
||||
│ - VCPToolBox 借鉴点 │
|
||||
└────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
R.1 ──────────────────────────────────────────────────────────────┐
|
||||
│ Token 感知分块优化 │
|
||||
│ - tiktoken 集成 │
|
||||
│ - 智能断句 │
|
||||
│ - 重叠分块 │
|
||||
│ │
|
||||
│ 工作量: 3 天 │
|
||||
└────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
R.2 ──────────────────────────────────────────────────────────────┐
|
||||
│ 多索引架构 │
|
||||
│ - Collection 分离策略 │
|
||||
│ - 懒加载 + LRU TTL │
|
||||
│ - 重要性感知检索 │
|
||||
│ │
|
||||
│ 依赖: R.1 │
|
||||
│ 工作量: 4 天 │
|
||||
└────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
R.3 ──────────────────────────────────────────────────────────────┐
|
||||
│ 动态权重增强 │
|
||||
│ - QueryAnalyzer │
|
||||
│ - DynamicReranker │
|
||||
│ - CoreTagAwareSearch │
|
||||
│ │
|
||||
│ 依赖: R.1 │
|
||||
│ 工作量: 4.5 天 │
|
||||
└────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
R.4 ──────────────────────────────────────────────────────────────┐
|
||||
│ 高级特性 (可选) │
|
||||
│ - 语义去重 │
|
||||
│ - 语义分桶 │
|
||||
│ - EPA 分析设计 │
|
||||
│ │
|
||||
│ 工作量: 4.5 天（可选） │
|
||||
└────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### R.5.2 总工作量估算
|
||||
|
||||
| Phase | 工作量 |
|
||||
|-------|--------|
|
||||
| R.1 | 3 天 |
|
||||
| R.2 | 4 天 |
|
||||
| R.3 | 4.5 天 |
|
||||
| R.4（可选） | 4.5 天 |
|
||||
| **R.1-R.3 必须** | **11.5 天** |
|
||||
| **R.1-R.4 含可选** | **16 天** |
|
||||
|
||||
### R.5.3 产出清单
|
||||
|
||||
| 产出 | 对应 Phase |
|
||||
|------|-----------|
|
||||
| `services/chunker.py` | R.1 |
|
||||
| `services/multi_index.py` | R.2 |
|
||||
| `services/query_analyzer.py` | R.3 |
|
||||
| `services/dynamic_reranker.py` | R.3 |
|
||||
| `services/core_tag_search.py` | R.3 |
|
||||
| `models/document.py` 更新 | R.2, R.3 |
|
||||
| 单元测试 > 80% | R.1, R.2, R.3 |
|
||||
| 集成测试通过 | R.1, R.2, R.3 |
|
||||
|
||||
### R.5.4 与 Phase 1-5 的关系
|
||||
|
||||
| Phase | RAG 协作内容 |
|
||||
|-------|-------------|
|
||||
| Phase 1 | 基础加固：Task Schema 追踪 RAG 任务 |
|
||||
| Phase 2 | 协作：RAG 任务可分解给 Librarian Agent |
|
||||
| Phase 3 | 动态：支持多索引动态选择 |
|
||||
| Phase 4 | 可视化：RAG 检索过程可视化 |
|
||||
| Phase 5 | 高级：EPA 分析、语义分桶 |
|
||||
| **Phase R** | **独立 RAG 升级路径，可与 Phase 1-5 并行推进** |
|
||||
|
||||
---
|
||||
|
||||
## R.6 风险与注意事项
|
||||
|
||||
| 风险 | 影响 | 缓解措施 |
|
||||
|------|------|----------|
|
||||
| Token 计数不准确 | 低 | 使用 tiktoken 精确计数，多次验证 |
|
||||
| 索引分离后检索复杂 | 中 | 提供统一检索接口，隐藏内部逻辑 |
|
||||
| 动态权重调参困难 | 中 | 提供配置项，允许用户调整 |
|
||||
| EPA 实现复杂度高 | 高 | Phase R.4 可选，暂不实现 |
|
||||
173
development-doc/plan/tool-update/README.md
Normal file
173
development-doc/plan/tool-update/README.md
Normal file
@@ -0,0 +1,173 @@
|
||||
# Jarvis Tools 升级计划索引
|
||||
|
||||
本目录用于存放 Jarvis 工具系统的分阶段升级规划文档。
|
||||
|
||||
## 文档说明
|
||||
|
||||
| 文件 | 说明 |
|
||||
|------|------|
|
||||
| `README.md` | 总览、阶段关系、实施顺序 |
|
||||
| `phase-t-0-current-state.md` | 当前现状、问题、目标架构、VCPToolBox 借鉴 |
|
||||
| `phase-t-1-manifest-system.md` | Manifest 驱动系统 |
|
||||
| `phase-t-2-tool-registry.md` | 工具注册中心 |
|
||||
| `phase-t-3-tool-implementation.md` | 核心工具实现 |
|
||||
| `phase-t-4-advanced.md` | 高级特性（多运行时/Agent协作） |
|
||||
| `checklist.md` | 执行清单 |
|
||||
|
||||
## 推荐阅读顺序
|
||||
|
||||
1. 先读 `phase-t-0-current-state.md`
|
||||
2. 再按顺序阅读 phase t-1 ~ t-4
|
||||
3. 实施时严格按阶段推进
|
||||
4. 参考 `checklist.md` 进行任务追踪
|
||||
|
||||
---
|
||||
|
||||
## 总体升级原则
|
||||
|
||||
1. **Manifest 驱动** - 声明式工具定义，热插拔
|
||||
2. **标准契约** - 统一的调用格式和返回结构
|
||||
3. **多运行时** - 支持 Python/JS/原生
|
||||
4. **类型安全** - Pydantic Schema 验证
|
||||
5. **可观测性** - 调用日志、耗时统计
|
||||
|
||||
---
|
||||
|
||||
## 阶段总览图
|
||||
|
||||
```
|
||||
T.0 ──────────────────────────────────────────────────────────────┐
|
||||
│ 现状与目标 │
|
||||
│ - 当前工具系统分析 │
|
||||
│ - 短板识别 │
|
||||
│ - VCPToolBox 工具系统借鉴 │
|
||||
└────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
T.1 ──────────────────────────────────────────────────────────────┐
|
||||
│ Manifest 驱动系统 │
|
||||
│ - 工具 manifest 定义 │
|
||||
│ - 标准化契约 │
|
||||
│ - Schema 验证 │
|
||||
│ │
|
||||
│ 核心文件: tools/manifests/, tools/schemas/ │
|
||||
│ 工作量: 3 天 │
|
||||
└────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
T.2 ──────────────────────────────────────────────────────────────┐
|
||||
│ 工具注册中心 │
|
||||
│ - 工具发现机制 │
|
||||
│ - 动态注册 │
|
||||
│ - 工具描述生成 │
|
||||
│ │
|
||||
│ 核心文件: tools/registry.py │
|
||||
│ 依赖: T.1 │
|
||||
│ 工作量: 2 天 │
|
||||
└────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
T.3 ──────────────────────────────────────────────────────────────┐
|
||||
│ 核心工具实现 │
|
||||
│ - 文件操作工具 │
|
||||
│ - 搜索工具 │
|
||||
│ - 网页抓取工具 │
|
||||
│ - 任务管理工具 │
|
||||
│ │
|
||||
│ 核心文件: tools/implementations/ │
|
||||
│ 依赖: T.2 │
|
||||
│ 工作量: 5 天 │
|
||||
└────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
T.4 ──────────────────────────────────────────────────────────────┐
|
||||
│ 高级特性 │
|
||||
│ - 多运行时支持 │
|
||||
│ - Agent 间协作 │
|
||||
│ - 定时任务 │
|
||||
│ │
|
||||
│ 核心文件: tools/runtime/, agents/tools/ │
|
||||
│ 依赖: T.3 │
|
||||
│ 工作量: 4 天 │
|
||||
└────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## VCPToolBox 工具系统核心借鉴
|
||||
|
||||
| 借鉴点 | 实现位置 | 难度 |
|
||||
|--------|---------|------|
|
||||
| Manifest 驱动 | T.1 | 🟢 低 |
|
||||
| 标准化契约 | T.1 | 🟢 低 |
|
||||
| configSchema 配置 | T.1 | 🟢 低 |
|
||||
| 工具注册中心 | T.2 | 🟡 中 |
|
||||
| 动态发现 | T.2 | 🟡 中 |
|
||||
| 文件操作工具 | T.3 | 🟢 低 |
|
||||
| 搜索工具 | T.3 | 🟡 中 |
|
||||
| 网页抓取 | T.3 | 🟡 中 |
|
||||
| 多运行时支持 | T.4 | 🟡 中 |
|
||||
| Agent 间协作 | T.4 | 🟡 中 |
|
||||
|
||||
---
|
||||
|
||||
## 实施顺序
|
||||
|
||||
```
|
||||
T.0 → T.1 → T.2 → T.3 → T.4
|
||||
│ │ │ │ │
|
||||
│ │ │ │ └── 多运行时/Agent协作
|
||||
│ │ │ └── 核心工具
|
||||
│ │ └── 注册中心
|
||||
│ └── Manifest系统
|
||||
└── 现状与目标
|
||||
```
|
||||
|
||||
**注意：** T.1 是基础，后续阶段都依赖 T.1。
|
||||
|
||||
---
|
||||
|
||||
## 文件变更追踪
|
||||
|
||||
| Phase | 新增文件 | 修改文件 |
|
||||
|-------|---------|---------|
|
||||
| T.1 | `tools/manifests/*.yaml`, `tools/schemas/` | `pyproject.toml` |
|
||||
| T.2 | `tools/registry.py`, `tools/base.py` | `services/agent_service.py` |
|
||||
| T.3 | `tools/implementations/*.py` | `tools/registry.py` |
|
||||
| T.4 | `tools/runtime/`, `agents/tools/` | `agents/graph.py` |
|
||||
|
||||
---
|
||||
|
||||
## 与 Agent Phase 1-5 的关系
|
||||
|
||||
| Agent Phase | Tools 协作内容 |
|
||||
|-------------|---------------|
|
||||
| Phase 1 | Task Schema 追踪工具调用 |
|
||||
| Phase 2 | 工具可委托给执行 Agent |
|
||||
| Phase 3 | 动态选择最优工具 |
|
||||
| Phase 4 | 工具调用可视化 |
|
||||
| Phase 5 | 多 Agent 工具协作 |
|
||||
| **Phase T** | **工具系统升级，与 Phase 1-5 协同** |
|
||||
|
||||
---
|
||||
|
||||
## 注意事项
|
||||
|
||||
| 注意事项 | 说明 |
|
||||
|---------|------|
|
||||
| T.1 是基础 | T.2-T.4 都依赖 T.1 的 Manifest 系统 |
|
||||
| 兼容性优先 | 保持现有 Agent 工具调用方式 |
|
||||
| 安全第一 | 严格权限控制，防止滥用 |
|
||||
| 测试优先 | 每个工具都要配套测试 |
|
||||
|
||||
---
|
||||
|
||||
## 总工作量
|
||||
|
||||
| Phase | 工作量 |
|
||||
|-------|--------|
|
||||
| T.1 | 3 天 |
|
||||
| T.2 | 2 天 |
|
||||
| T.3 | 5 天 |
|
||||
| T.4 | 4 天 |
|
||||
| **总计** | **14 天** |
|
||||
251
development-doc/plan/tool-update/checklist.md
Normal file
251
development-doc/plan/tool-update/checklist.md
Normal file
@@ -0,0 +1,251 @@
|
||||
# Tools 升级执行清单
|
||||
|
||||
本清单用于追踪 Tools 升级计划的执行进度。
|
||||
|
||||
---
|
||||
|
||||
## 总进度
|
||||
|
||||
| Phase | 名称 | 状态 | 工作量 |
|
||||
|-------|------|------|--------|
|
||||
| T.0 | 现状与目标 | ✅ 完成 | - |
|
||||
| T.1 | Manifest 驱动系统 | ⬜ 待开始 | 3 天 |
|
||||
| T.2 | 工具注册中心 | ⬜ 待开始 | 2 天 |
|
||||
| T.3 | 核心工具实现 | ⬜ 待开始 | 5 天 |
|
||||
| T.4 | 高级特性 | ⬜ 待开始 | 4 天 |
|
||||
| **总计** | | | **14 天** |
|
||||
|
||||
---
|
||||
|
||||
## Phase T.1：Manifest 驱动系统
|
||||
|
||||
### 目标
|
||||
建立 Jarvis 的 Manifest 驱动工具系统，定义标准化工件声明。
|
||||
|
||||
### 任务清单
|
||||
|
||||
#### Schema 定义
|
||||
- [ ] 创建 `tools/schemas/manifest.py`
|
||||
- [ ] 定义 `ToolManifest` Schema
|
||||
- [ ] 定义 `ToolType` 枚举
|
||||
- [ ] 定义 `RuntimeType` 枚举
|
||||
- [ ] 定义 `InvocationCommand` Schema
|
||||
- [ ] 创建 `tools/schemas/tool_call.py`
|
||||
- [ ] 定义 `ToolCallRequest` Schema
|
||||
- [ ] 定义 `ToolCallResponse` Schema
|
||||
- [ ] 定义 `ToolExecutionLog` Schema
|
||||
|
||||
#### 验证器
|
||||
- [ ] 创建 `tools/schemas/validator.py`
|
||||
- [ ] 实现 `validate_manifest` 函数
|
||||
- [ ] 实现 `validate_tool_call` 函数
|
||||
- [ ] 实现错误类
|
||||
|
||||
#### 配置系统
|
||||
- [ ] 创建 `tools/configs/loader.py`
|
||||
- [ ] 实现 `ConfigLoader` 类
|
||||
- [ ] 实现配置缓存
|
||||
- [ ] 实现配置重载
|
||||
|
||||
#### Manifest 文件
|
||||
- [ ] 创建 `tools/manifests/file_operator.yaml`
|
||||
- [ ] 创建 `tools/manifests/web_search.yaml`
|
||||
- [ ] 创建其他工具 Manifest
|
||||
|
||||
#### 测试
|
||||
- [ ] 单元测试
|
||||
|
||||
### 产出文件
|
||||
- `tools/schemas/manifest.py`
|
||||
- `tools/schemas/tool_call.py`
|
||||
- `tools/schemas/validator.py`
|
||||
- `tools/configs/loader.py`
|
||||
- `tools/manifests/*.yaml`
|
||||
|
||||
### 验收
|
||||
- [ ] Schema 验证正常工作
|
||||
- [ ] 配置加载器正常工作
|
||||
- [ ] Manifest 文件格式正确
|
||||
- [ ] 单元测试通过
|
||||
|
||||
---
|
||||
|
||||
## Phase T.2：工具注册中心
|
||||
|
||||
### 目标
|
||||
实现工具的动态注册、发现和管理。
|
||||
|
||||
### 任务清单
|
||||
|
||||
#### 注册中心
|
||||
- [ ] 创建 `tools/registry.py`
|
||||
- [ ] 实现 `ToolMetadata` dataclass
|
||||
- [ ] 实现 `ToolRegistry` 类
|
||||
- [ ] 实现注册/注销方法
|
||||
- [ ] 实现查询方法
|
||||
- [ ] 实现统计方法
|
||||
|
||||
#### 工具发现
|
||||
- [ ] 创建 `tools/discovery.py`
|
||||
- [ ] 实现 `ToolDiscovery` 类
|
||||
- [ ] 实现自动发现
|
||||
- [ ] 实现热重载
|
||||
|
||||
#### 描述生成
|
||||
- [ ] 创建 `tools/description.py`
|
||||
- [ ] 实现 AI 友好描述生成
|
||||
- [ ] 实现工具列表生成
|
||||
|
||||
#### 权限控制
|
||||
- [ ] 创建 `tools/permissions.py`
|
||||
- [ ] 实现 `ToolPermission` 枚举
|
||||
- [ ] 实现 `ToolPermissionChecker` 类
|
||||
|
||||
#### LangChain 集成
|
||||
- [ ] 创建 `tools/langchain_adapter.py`
|
||||
- [ ] 实现适配器
|
||||
- [ ] 集成到 Agent
|
||||
|
||||
#### 测试
|
||||
- [ ] 单元测试
|
||||
|
||||
### 产出文件
|
||||
- `tools/registry.py`
|
||||
- `tools/discovery.py`
|
||||
- `tools/description.py`
|
||||
- `tools/permissions.py`
|
||||
- `tools/langchain_adapter.py`
|
||||
|
||||
### 验收
|
||||
- [ ] 注册中心正常工作
|
||||
- [ ] 工具发现正常工作
|
||||
- [ ] 权限检查正常工作
|
||||
- [ ] LangChain 适配器正常工作
|
||||
- [ ] 单元测试通过
|
||||
|
||||
---
|
||||
|
||||
## Phase T.3：核心工具实现
|
||||
|
||||
### 目标
|
||||
实现文件操作、搜索、网页抓取等核心工具。
|
||||
|
||||
### 任务清单
|
||||
|
||||
#### 文件操作工具
|
||||
- [ ] 创建 `tools/implementations/file_operator.py`
|
||||
- [ ] 实现 `FileOperator` 类
|
||||
- [ ] 实现 read_file
|
||||
- [ ] 实现 write_file
|
||||
- [ ] 实现 list_directory
|
||||
- [ ] 实现 search_files
|
||||
- [ ] 实现路径安全检查
|
||||
- [ ] 实现多格式支持（PDF/DOCX/XLSX）
|
||||
|
||||
#### 搜索工具
|
||||
- [ ] 创建 `tools/implementations/web_search.py`
|
||||
- [ ] 实现 `WebSearch` 类
|
||||
- [ ] 实现 search 方法
|
||||
- [ ] 实现 deep_search 方法
|
||||
|
||||
#### 网页抓取工具
|
||||
- [ ] 创建 `tools/implementations/web_fetch.py`
|
||||
- [ ] 实现 `WebFetch` 类
|
||||
- [ ] 实现 fetch 方法
|
||||
- [ ] 实现 screenshot 方法
|
||||
|
||||
#### 任务管理工具
|
||||
- [ ] 创建 `tools/implementations/task_manager.py`
|
||||
- [ ] 实现 `TaskManager` 类
|
||||
- [ ] 实现任务 CRUD
|
||||
|
||||
#### Manifest 绑定
|
||||
- [ ] 更新 Manifest 文件
|
||||
- [ ] 注册到工具中心
|
||||
|
||||
#### 测试
|
||||
- [ ] 单元测试
|
||||
|
||||
### 产出文件
|
||||
- `tools/implementations/file_operator.py`
|
||||
- `tools/implementations/web_search.py`
|
||||
- `tools/implementations/web_fetch.py`
|
||||
- `tools/implementations/task_manager.py`
|
||||
|
||||
### 验收
|
||||
- [ ] 文件操作工具正常工作
|
||||
- [ ] 搜索工具正常工作
|
||||
- [ ] 网页抓取工具正常工作
|
||||
- [ ] 任务管理工具正常工作
|
||||
- [ ] 单元测试通过
|
||||
|
||||
---
|
||||
|
||||
## Phase T.4：高级特性
|
||||
|
||||
### 目标
|
||||
实现多运行时支持、Agent 协作和定时任务。
|
||||
|
||||
### 任务清单
|
||||
|
||||
#### 运行时系统
|
||||
- [ ] 创建 `tools/runtime/base.py`
|
||||
- [ ] 定义 `BaseRuntime` 抽象基类
|
||||
- [ ] 创建 `tools/runtime/python_runtime.py`
|
||||
- [ ] 创建 `tools/runtime/js_runtime.py`
|
||||
- [ ] 创建 `tools/runtime/native_runtime.py`
|
||||
- [ ] 创建 `tools/runtime/manager.py`
|
||||
|
||||
#### Agent 协作
|
||||
- [ ] 创建 `agents/tools/collaboration.py`
|
||||
- [ ] 定义 `CollaborationMessage`
|
||||
- [ ] 实现 `CollaborationProtocol` 类
|
||||
- [ ] 实现请求/响应机制
|
||||
|
||||
#### 定时任务
|
||||
- [ ] 创建 `tools/scheduler.py`
|
||||
- [ ] 实现 `ScheduledTask`
|
||||
- [ ] 实现 `ToolScheduler` 类
|
||||
- [ ] 实现多种调度类型
|
||||
|
||||
#### 测试
|
||||
- [ ] 单元测试
|
||||
|
||||
### 产出文件
|
||||
- `tools/runtime/*.py`
|
||||
- `agents/tools/collaboration.py`
|
||||
- `tools/scheduler.py`
|
||||
|
||||
### 验收
|
||||
- [ ] 多运行时正常工作
|
||||
- [ ] Agent 协作正常工作
|
||||
- [ ] 定时任务正常工作
|
||||
- [ ] 单元测试通过
|
||||
|
||||
---
|
||||
|
||||
## 完成标准
|
||||
|
||||
- [ ] 所有 Phase T.1-T.4 任务完成
|
||||
- [ ] 所有单元测试通过
|
||||
- [ ] API 文档更新完成
|
||||
- [ ] 部署验证通过
|
||||
|
||||
---
|
||||
|
||||
## 风险与注意事项
|
||||
|
||||
| 风险 | 影响 | 缓解措施 |
|
||||
|------|------|----------|
|
||||
| 多运行时复杂性 | 开发成本增加 | 优先实现 Python 运行时 |
|
||||
| JS 运行时依赖 Node | 部署环境要求 | 提供降级方案 |
|
||||
| 定时任务精度 | 任务延迟 | 使用专业调度库 |
|
||||
| 安全漏洞 | 系统风险 | 严格权限控制 |
|
||||
|
||||
---
|
||||
|
||||
## 更新日志
|
||||
|
||||
| 日期 | Phase | 变更内容 |
|
||||
|------|-------|----------|
|
||||
| 2026-04-04 | T.0 | 创建文档 |
|
||||
192
development-doc/plan/tool-update/phase-t-0-current-state.md
Normal file
192
development-doc/plan/tool-update/phase-t-0-current-state.md
Normal file
@@ -0,0 +1,192 @@
|
||||
# Phase T.0：Tools 现状与目标
|
||||
|
||||
日期：2026-04-04
|
||||
状态：已完成
|
||||
借鉴来源：VCPToolBox Plugin 系统
|
||||
|
||||
---
|
||||
|
||||
## 1. 本阶段目的
|
||||
|
||||
本文件用于统一背景认知，明确：
|
||||
|
||||
- Jarvis 当前工具系统处于什么水平
|
||||
- 主要短板是什么
|
||||
- 为什么要升级
|
||||
- 升级后的目标形态是什么
|
||||
- VCPToolBox 给我们什么启发
|
||||
|
||||
---
|
||||
|
||||
## 2. 当前 Jarvis Tools 架构
|
||||
|
||||
### 2.1 核心流程
|
||||
|
||||
```
|
||||
Agent 决策 → 工具调用 → LLM API → 返回结果
|
||||
```
|
||||
|
||||
### 2.2 当前工具实现
|
||||
|
||||
Jarvis 当前的工具以 LangChain Tools 形式存在：
|
||||
|
||||
```python
|
||||
# agents/tools/
|
||||
├── base.py # 工具基类
|
||||
├── file_tools.py # 文件操作
|
||||
├── search_tools.py # 搜索
|
||||
└── web_tools.py # 网页相关
|
||||
```
|
||||
|
||||
### 2.3 当前工具列表
|
||||
|
||||
| 工具 | 功能 | 实现方式 |
|
||||
|------|------|---------|
|
||||
| `read_file` | 读取文件 | Python |
|
||||
| `write_file` | 写入文件 | Python |
|
||||
| `run_python` | 执行代码 | Python |
|
||||
| `search_knowledge` | 知识库检索 | LangChain |
|
||||
| `search_web` | 联网搜索 | API |
|
||||
|
||||
### 2.4 当前问题
|
||||
|
||||
| 问题 | 影响 |
|
||||
|------|------|
|
||||
| 硬编码工具 | 新增工具需改代码 |
|
||||
| 无 manifest | 无法热插拔 |
|
||||
| 无标准化契约 | 调用格式不统一 |
|
||||
| 无配置分离 | 敏感信息易泄露 |
|
||||
| 无类型安全 | 验证缺失 |
|
||||
| 无权限控制 | 安全隐患 |
|
||||
|
||||
---
|
||||
|
||||
## 3. VCPToolBox 工具系统分析
|
||||
|
||||
### 3.1 六大插件类型
|
||||
|
||||
```javascript
|
||||
const PLUGIN_TYPES = {
|
||||
static: "静态占位符，自动注入系统提示词",
|
||||
synchronous: "同步执行，stdio 协议",
|
||||
asynchronous: "异步执行，后台处理",
|
||||
service: "持续运行服务",
|
||||
hybridservice: "混合服务（Agent间通讯）",
|
||||
messagePreprocessor: "消息预处理"
|
||||
};
|
||||
```
|
||||
|
||||
### 3.2 Manifest 标准契约
|
||||
|
||||
```javascript
|
||||
{
|
||||
"manifestVersion": "1.0.0",
|
||||
"name": "PluginName",
|
||||
"displayName": "中文显示名",
|
||||
"description": "功能描述",
|
||||
"pluginType": "synchronous",
|
||||
"version": "1.0.0",
|
||||
|
||||
"entryPoint": {
|
||||
"type": "nodejs",
|
||||
"command": "node Plugin.js",
|
||||
"timeout": 300000
|
||||
},
|
||||
|
||||
"communication": {
|
||||
"protocol": "stdio",
|
||||
"timeout": 300000
|
||||
},
|
||||
|
||||
"configSchema": {
|
||||
"API_KEY": {
|
||||
"type": "string",
|
||||
"description": "API 密钥"
|
||||
}
|
||||
},
|
||||
|
||||
"capabilities": {
|
||||
"invocationCommands": [
|
||||
{
|
||||
"commandIdentifier": "CommandName",
|
||||
"description": "详细描述+调用格式示例",
|
||||
"example": "调用示例"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 3.3 核心设计理念
|
||||
|
||||
1. **声明式** - 工具通过 manifest 声明，框架自动发现
|
||||
2. **标准化** - 统一的调用协议（stdio/websocket）
|
||||
3. **可配置** - configSchema 声明配置项
|
||||
4. **可观测** - 调用日志、超时控制
|
||||
5. **安全隔离** - 沙箱执行、权限控制
|
||||
|
||||
### 3.4 关键文件
|
||||
|
||||
| 文件 | 作用 |
|
||||
|------|------|
|
||||
| `Plugin.js` | 插件加载与执行引擎 |
|
||||
| `plugin-manifest.json` | 插件声明契约 |
|
||||
| `config.env` | 插件配置 |
|
||||
| `routes/` | API 路由层 |
|
||||
|
||||
---
|
||||
|
||||
## 4. 目标架构
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Tool Manifests │
|
||||
│ - YAML/JSON 声明式定义 │
|
||||
│ - 版本管理 │
|
||||
│ - Schema 验证 │
|
||||
└─────────────────────────┬───────────────────────────────────┘
|
||||
│
|
||||
┌─────────────────────────┴───────────────────────────────────┐
|
||||
│ Tool Registry │
|
||||
│ - 动态发现 │
|
||||
│ - 权限控制 │
|
||||
│ - 调用统计 │
|
||||
└─────────────────────────┬───────────────────────────────────┘
|
||||
│
|
||||
┌─────────────────────────┴───────────────────────────────────┐
|
||||
│ Tool Executor │
|
||||
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
|
||||
│ │ Python RT │ │ JS RT │ │ Native RT │ │
|
||||
│ └─────────────┘ └─────────────┘ └─────────────┘ │
|
||||
└─────────────────────────┬───────────────────────────────────┘
|
||||
│
|
||||
┌─────────────────────────┴───────────────────────────────────┐
|
||||
│ Tool Output │
|
||||
│ - 类型化返回 │
|
||||
│ - 错误处理 │
|
||||
│ - 调用日志 │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. 借鉴点映射
|
||||
|
||||
| VCPToolBox 借鉴点 | Jarvis 实现位置 | 优先级 |
|
||||
|-------------------|---------------|--------|
|
||||
| Manifest 驱动 | `tools/manifests/` | 🟢 高 |
|
||||
| 标准化契约 | `tools/schemas/` | 🟢 高 |
|
||||
| configSchema | `tools/base.py` | 🟢 高 |
|
||||
| 工具注册中心 | `tools/registry.py` | 🟡 中 |
|
||||
| 动态发现 | `tools/discovery.py` | 🟡 中 |
|
||||
| 调用日志 | `tools/logging.py` | 🟡 中 |
|
||||
| 超时控制 | `tools/executor.py` | 🟡 中 |
|
||||
| 多运行时 | `tools/runtime/` | 🟡 中 |
|
||||
|
||||
---
|
||||
|
||||
## 6. 本阶段产出要求
|
||||
|
||||
- [x] 团队对 Jarvis 当前工具系统和目标方向达成一致
|
||||
- [x] VCPToolBox 工具系统借鉴点已映射到具体 Phase
|
||||
- [x] 后续 phase 文档能够在这个认知基础上展开
|
||||
484
development-doc/plan/tool-update/phase-t-1-manifest-system.md
Normal file
484
development-doc/plan/tool-update/phase-t-1-manifest-system.md
Normal file
@@ -0,0 +1,484 @@
|
||||
# Phase T.1：Manifest 驱动系统
|
||||
|
||||
日期：2026-04-04
|
||||
状态：待开始
|
||||
依赖：T.0（已完成）
|
||||
|
||||
---
|
||||
|
||||
## 1. 本阶段目的
|
||||
|
||||
建立 Jarvis 的 Manifest 驱动工具系统：
|
||||
|
||||
- 定义工具 Manifest Schema
|
||||
- 实现 Schema 验证
|
||||
- 创建核心工具的 Manifest 文件
|
||||
- 实现配置分离
|
||||
|
||||
---
|
||||
|
||||
## 2. Manifest Schema 设计
|
||||
|
||||
### 2.1 目录结构
|
||||
|
||||
```
|
||||
backend/app/tools/
|
||||
├── manifests/ # 工具 manifest
|
||||
│ ├── file_operator.yaml
|
||||
│ ├── search.yaml
|
||||
│ └── web_fetch.yaml
|
||||
├── schemas/ # Schema 定义
|
||||
│ ├── __init__.py
|
||||
│ ├── manifest.py # Manifest Schema
|
||||
│ ├── tool_call.py # 工具调用 Schema
|
||||
│ └── config.py # 配置 Schema
|
||||
└── configs/ # 配置分离
|
||||
└── .tool.example # 工具配置模板
|
||||
```
|
||||
|
||||
### 2.2 ToolManifest Schema
|
||||
|
||||
```python
|
||||
# tools/schemas/manifest.py
|
||||
from pydantic import BaseModel, Field
|
||||
from typing import Optional, List, Dict, Any
|
||||
from enum import Enum
|
||||
|
||||
|
||||
class ToolType(str, Enum):
|
||||
"""工具类型"""
|
||||
SYNC = "sync" # 同步执行
|
||||
ASYNC = "async" # 异步执行
|
||||
SERVICE = "service" # 持续服务
|
||||
|
||||
|
||||
class RuntimeType(str, Enum):
|
||||
"""运行时类型"""
|
||||
PYTHON = "python"
|
||||
JAVASCRIPT = "javascript"
|
||||
NATIVE = "native"
|
||||
|
||||
|
||||
class InvocationCommand(BaseModel):
|
||||
"""调用命令定义"""
|
||||
name: str = Field(..., description="命令名称")
|
||||
description: str = Field(..., description="命令描述（给 AI 看）")
|
||||
parameters: Optional[Dict[str, Any]] = Field(
|
||||
default=None,
|
||||
description="参数 JSON Schema"
|
||||
)
|
||||
required: Optional[List[str]] = Field(
|
||||
default=None,
|
||||
description="必需参数列表"
|
||||
)
|
||||
example: Optional[str] = Field(
|
||||
default=None,
|
||||
description="调用示例"
|
||||
)
|
||||
|
||||
|
||||
class ToolManifest(BaseModel):
|
||||
"""工具 Manifest"""
|
||||
manifest_version: str = Field(
|
||||
default="1.0.0",
|
||||
description="Manifest 版本"
|
||||
)
|
||||
name: str = Field(..., description="工具名称（英文，唯一）")
|
||||
display_name: str = Field(..., description="显示名称（中文）")
|
||||
description: str = Field(..., description="工具描述")
|
||||
author: Optional[str] = Field(default=None, description="作者")
|
||||
version: str = Field(default="1.0.0", description="版本号")
|
||||
|
||||
# 执行配置
|
||||
type: ToolType = Field(default=ToolType.SYNC, description="工具类型")
|
||||
runtime: RuntimeType = Field(default=RuntimeType.PYTHON, description="运行时")
|
||||
entry: str = Field(..., description="执行入口（文件路径或命令）")
|
||||
timeout: int = Field(default=30000, description="超时时间（毫秒）")
|
||||
|
||||
# 配置
|
||||
config_schema: Optional[Dict[str, Any]] = Field(
|
||||
default=None,
|
||||
description="配置项 Schema"
|
||||
)
|
||||
|
||||
# 能力
|
||||
commands: List[InvocationCommand] = Field(
|
||||
default_factory=list,
|
||||
description="可用命令列表"
|
||||
)
|
||||
|
||||
# 元数据
|
||||
tags: Optional[List[str]] = Field(default=None, description="标签")
|
||||
dependencies: Optional[List[str]] = Field(default=None, description="依赖工具")
|
||||
enabled: bool = Field(default=True, description="是否启用")
|
||||
|
||||
class Config:
|
||||
use_enum_values = True
|
||||
```
|
||||
|
||||
### 2.3 ToolCall Schema
|
||||
|
||||
```python
|
||||
# tools/schemas/tool_call.py
|
||||
from pydantic import BaseModel, Field
|
||||
from typing import Optional, Dict, Any, List
|
||||
from datetime import datetime
|
||||
|
||||
|
||||
class ToolCallRequest(BaseModel):
|
||||
"""工具调用请求"""
|
||||
tool_name: str = Field(..., description="工具名称")
|
||||
command: str = Field(..., description="命令名称")
|
||||
parameters: Dict[str, Any] = Field(default_factory=dict, description="参数")
|
||||
timeout: Optional[int] = Field(default=None, description="超时时间")
|
||||
context: Optional[Dict[str, Any]] = Field(
|
||||
default=None,
|
||||
description="上下文信息"
|
||||
)
|
||||
|
||||
|
||||
class ToolCallResponse(BaseModel):
|
||||
"""工具调用响应"""
|
||||
status: str = Field(..., description="状态: success/error")
|
||||
result: Optional[Any] = Field(default=None, description="执行结果")
|
||||
error: Optional[str] = Field(default=None, description="错误信息")
|
||||
message: Optional[str] = Field(default=None, description="AI 友好消息")
|
||||
base64: Optional[str] = Field(default=None, description="Base64 数据")
|
||||
duration_ms: Optional[int] = Field(default=None, description="执行耗时")
|
||||
timestamp: datetime = Field(default_factory=datetime.utcnow)
|
||||
|
||||
|
||||
class ToolExecutionLog(BaseModel):
|
||||
"""工具执行日志"""
|
||||
id: str
|
||||
tool_name: str
|
||||
command: str
|
||||
parameters: Dict[str, Any]
|
||||
status: str
|
||||
duration_ms: int
|
||||
error: Optional[str]
|
||||
user_id: Optional[str]
|
||||
agent_id: Optional[str]
|
||||
created_at: datetime
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Manifest 示例
|
||||
|
||||
### 3.1 file_operator.yaml
|
||||
|
||||
```yaml
|
||||
manifest_version: "1.0.0"
|
||||
name: file_operator
|
||||
display_name: 文件操作器
|
||||
description: 强大的文件系统操作工具，支持读写、搜索、下载等功能
|
||||
author: Jarvis
|
||||
version: "1.0.0"
|
||||
|
||||
type: sync
|
||||
runtime: python
|
||||
entry: tools/implementations/file_operator.py
|
||||
timeout: 30000
|
||||
|
||||
config_schema:
|
||||
allowed_directories:
|
||||
type: string
|
||||
description: 允许操作的目录列表，逗号分隔
|
||||
default: ""
|
||||
max_file_size:
|
||||
type: integer
|
||||
description: 最大文件大小（字节）
|
||||
default: 10485760
|
||||
|
||||
commands:
|
||||
- name: read_file
|
||||
description: |
|
||||
读取指定路径文件的内容。支持 PDF、DOCX、XLSX 等格式自动解析。
|
||||
参数:
|
||||
- filePath (必需): 文件绝对路径
|
||||
- encoding (可选): 编码格式，默认 utf8
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
filePath:
|
||||
type: string
|
||||
description: 文件绝对路径
|
||||
encoding:
|
||||
type: string
|
||||
default: utf8
|
||||
required: [filePath]
|
||||
|
||||
- name: write_file
|
||||
description: |
|
||||
将内容写入文件。如果文件存在，自动创建新文件避免覆盖。
|
||||
参数:
|
||||
- filePath (必需): 文件绝对路径
|
||||
- content (必需): 文件内容
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
filePath:
|
||||
type: string
|
||||
content:
|
||||
type: string
|
||||
required: [filePath, content]
|
||||
|
||||
- name: list_directory
|
||||
description: |
|
||||
列出目录内容。
|
||||
参数:
|
||||
- directoryPath (必需): 目录绝对路径
|
||||
- showHidden (可选): 是否显示隐藏文件
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
directoryPath:
|
||||
type: string
|
||||
showHidden:
|
||||
type: boolean
|
||||
default: false
|
||||
required: [directoryPath]
|
||||
|
||||
- name: search_files
|
||||
description: |
|
||||
递归搜索匹配模式的文件。
|
||||
参数:
|
||||
- searchPath (必需): 搜索起始目录
|
||||
- pattern (必需): 文件模式，如 *.txt
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
searchPath:
|
||||
type: string
|
||||
pattern:
|
||||
type: string
|
||||
required: [searchPath, pattern]
|
||||
|
||||
tags: [file, system, essential]
|
||||
enabled: true
|
||||
```
|
||||
|
||||
### 3.2 search.yaml
|
||||
|
||||
```yaml
|
||||
manifest_version: "1.0.0"
|
||||
name: web_search
|
||||
display_name: 联网搜索
|
||||
description: 语义级并发搜索引擎，支持多源搜索和结果聚合
|
||||
author: Jarvis
|
||||
version: "1.0.0"
|
||||
|
||||
type: sync
|
||||
runtime: python
|
||||
entry: tools/implementations/web_search.py
|
||||
timeout: 60000
|
||||
|
||||
config_schema:
|
||||
api_key:
|
||||
type: string
|
||||
description: 搜索引擎 API 密钥
|
||||
required: true
|
||||
max_results:
|
||||
type: integer
|
||||
description: 最大返回结果数
|
||||
default: 10
|
||||
|
||||
commands:
|
||||
- name: search
|
||||
description: |
|
||||
执行语义级搜索。
|
||||
参数:
|
||||
- query (必需): 搜索关键词
|
||||
- max_results (可选): 最大结果数
|
||||
- sources (可选): 搜索源列表
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
query:
|
||||
type: string
|
||||
max_results:
|
||||
type: integer
|
||||
default: 10
|
||||
sources:
|
||||
type: array
|
||||
items:
|
||||
type: string
|
||||
required: [query]
|
||||
|
||||
- name: deep_search
|
||||
description: |
|
||||
深度搜索，带摘要生成。
|
||||
参数:
|
||||
- query (必需): 研究主题
|
||||
- keywords (必需): 关键词列表
|
||||
parameters:
|
||||
type: object
|
||||
properties:
|
||||
query:
|
||||
type: string
|
||||
keywords:
|
||||
type: array
|
||||
items:
|
||||
type: string
|
||||
required: [query, keywords]
|
||||
|
||||
tags: [search, web, research]
|
||||
enabled: true
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Schema 验证
|
||||
|
||||
### 4.1 验证器
|
||||
|
||||
```python
|
||||
# tools/schemas/validator.py
|
||||
from pydantic import ValidationError
|
||||
from tools.schemas.manifest import ToolManifest
|
||||
|
||||
|
||||
def validate_manifest(data: dict) -> ToolManifest:
|
||||
"""验证 Manifest 数据"""
|
||||
try:
|
||||
return ToolManifest(**data)
|
||||
except ValidationError as e:
|
||||
raise ManifestValidationError(str(e))
|
||||
|
||||
|
||||
def validate_tool_call(data: dict) -> ToolCallRequest:
|
||||
"""验证工具调用请求"""
|
||||
from tools.schemas.tool_call import ToolCallRequest
|
||||
try:
|
||||
return ToolCallRequest(**data)
|
||||
except ValidationError as e:
|
||||
raise ToolCallValidationError(str(e))
|
||||
|
||||
|
||||
class ManifestValidationError(Exception):
|
||||
"""Manifest 验证错误"""
|
||||
pass
|
||||
|
||||
|
||||
class ToolCallValidationError(Exception):
|
||||
"""工具调用验证错误"""
|
||||
pass
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. 配置分离
|
||||
|
||||
### 5.1 工具配置模板
|
||||
|
||||
```yaml
|
||||
# tools/configs/.tool.example
|
||||
# 文件操作器
|
||||
file_operator:
|
||||
allowed_directories: ""
|
||||
max_file_size: 10485760
|
||||
|
||||
# 联网搜索
|
||||
web_search:
|
||||
api_key: ""
|
||||
max_results: 10
|
||||
```
|
||||
|
||||
### 5.2 配置加载器
|
||||
|
||||
```python
|
||||
# tools/configs/loader.py
|
||||
import yaml
|
||||
from pathlib import Path
|
||||
from typing import Dict, Any
|
||||
|
||||
|
||||
class ConfigLoader:
|
||||
"""工具配置加载器"""
|
||||
|
||||
def __init__(self, config_dir: Path):
|
||||
self.config_dir = config_dir
|
||||
self._cache: Dict[str, Any] = {}
|
||||
|
||||
def load(self, tool_name: str) -> Dict[str, Any]:
|
||||
"""加载指定工具的配置"""
|
||||
if tool_name in self._cache:
|
||||
return self._cache[tool_name]
|
||||
|
||||
config_file = self.config_dir / f"{tool_name}.yaml"
|
||||
if not config_file.exists():
|
||||
return {}
|
||||
|
||||
with open(config_file) as f:
|
||||
config = yaml.safe_load(f) or {}
|
||||
|
||||
self._cache[tool_name] = config
|
||||
return config
|
||||
|
||||
def reload(self, tool_name: str) -> Dict[str, Any]:
|
||||
"""重新加载配置"""
|
||||
if tool_name in self._cache:
|
||||
del self._cache[tool_name]
|
||||
return self.load(tool_name)
|
||||
|
||||
def get(self, tool_name: str, key: str, default: Any = None) -> Any:
|
||||
"""获取配置项"""
|
||||
config = self.load(tool_name)
|
||||
return config.get(key, default)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. 实现步骤
|
||||
|
||||
| 步骤 | 任务 | 优先级 |
|
||||
|------|------|--------|
|
||||
| 1 | 创建目录结构 | 🟢 高 |
|
||||
| 2 | 实现 ToolManifest Schema | 🟢 高 |
|
||||
| 3 | 实现 ToolCall Schema | 🟢 高 |
|
||||
| 4 | 实现 Schema 验证器 | 🟢 高 |
|
||||
| 5 | 创建配置加载器 | 🟢 高 |
|
||||
| 6 | 创建 file_operator.yaml | 🟢 高 |
|
||||
| 7 | 创建 search.yaml | 🟡 中 |
|
||||
| 8 | 创建其他工具 Manifest | 🟡 中 |
|
||||
| 9 | 单元测试 | 🟡 中 |
|
||||
|
||||
---
|
||||
|
||||
## 7. 核心文件变更
|
||||
|
||||
| 文件 | 变更 |
|
||||
|------|------|
|
||||
| `tools/__init__.py` | 模块初始化 |
|
||||
| `tools/schemas/__init__.py` | Schema 导出 |
|
||||
| `tools/schemas/manifest.py` | 新增 |
|
||||
| `tools/schemas/tool_call.py` | 新增 |
|
||||
| `tools/schemas/validator.py` | 新增 |
|
||||
| `tools/configs/loader.py` | 新增 |
|
||||
| `tools/manifests/file_operator.yaml` | 新增 |
|
||||
| `tools/manifests/search.yaml` | 新增 |
|
||||
|
||||
---
|
||||
|
||||
## 8. 工作量估算
|
||||
|
||||
| 任务 | 工作量 |
|
||||
|------|--------|
|
||||
| Schema 定义 | 1 天 |
|
||||
| 验证器 | 0.5 天 |
|
||||
| 配置加载器 | 0.5 天 |
|
||||
| Manifest 文件 | 0.5 天 |
|
||||
| 单元测试 | 0.5 天 |
|
||||
| **总计** | **3 天** |
|
||||
|
||||
---
|
||||
|
||||
## 9. 验收标准
|
||||
|
||||
- [ ] ToolManifest Schema 可正确验证 Manifest
|
||||
- [ ] ToolCall Schema 可正确验证调用请求
|
||||
- [ ] 配置加载器可正确加载配置
|
||||
- [ ] Manifest 文件格式正确
|
||||
- [ ] Schema 验证器可捕获错误
|
||||
- [ ] 单元测试覆盖核心逻辑
|
||||
476
development-doc/plan/tool-update/phase-t-2-tool-registry.md
Normal file
476
development-doc/plan/tool-update/phase-t-2-tool-registry.md
Normal file
@@ -0,0 +1,476 @@
|
||||
# Phase T.2：工具注册中心
|
||||
|
||||
日期：2026-04-04
|
||||
状态：待开始
|
||||
依赖：T.1（待完成）
|
||||
|
||||
---
|
||||
|
||||
## 1. 本阶段目的
|
||||
|
||||
建立 Jarvis 的工具注册中心：
|
||||
|
||||
- 工具动态发现
|
||||
- 工具注册与管理
|
||||
- 工具描述生成
|
||||
- 调用统计与监控
|
||||
|
||||
---
|
||||
|
||||
## 2. 工具注册中心架构
|
||||
|
||||
### 2.1 核心类
|
||||
|
||||
```python
|
||||
# tools/registry.py
|
||||
from typing import Dict, List, Optional, Callable
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime
|
||||
import asyncio
|
||||
|
||||
|
||||
@dataclass
|
||||
class ToolMetadata:
|
||||
"""工具元数据"""
|
||||
name: str
|
||||
display_name: str
|
||||
description: str
|
||||
version: str
|
||||
author: Optional[str] = None
|
||||
tags: List[str] = field(default_factory=list)
|
||||
dependencies: List[str] = field(default_factory=list)
|
||||
enabled: bool = True
|
||||
registered_at: datetime = field(default_factory=datetime.utcnow)
|
||||
|
||||
# 统计
|
||||
call_count: int = 0
|
||||
error_count: int = 0
|
||||
total_duration_ms: int = 0
|
||||
|
||||
@property
|
||||
def avg_duration_ms(self) -> int:
|
||||
if self.call_count == 0:
|
||||
return 0
|
||||
return self.total_duration_ms // self.call_count
|
||||
|
||||
@property
|
||||
def error_rate(self) -> float:
|
||||
if self.call_count == 0:
|
||||
return 0.0
|
||||
return self.error_count / self.call_count
|
||||
|
||||
|
||||
class ToolRegistry:
|
||||
"""工具注册中心"""
|
||||
|
||||
def __init__(self):
|
||||
self._tools: Dict[str, ToolMetadata] = {}
|
||||
self._executors: Dict[str, Callable] = {}
|
||||
self._configs: Dict[str, dict] = {}
|
||||
self._lock = asyncio.Lock()
|
||||
|
||||
# === 注册方法 ===
|
||||
|
||||
async def register(
|
||||
self,
|
||||
manifest_path: str,
|
||||
executor: Callable,
|
||||
config: Optional[dict] = None,
|
||||
) -> ToolMetadata:
|
||||
"""注册工具"""
|
||||
from tools.schemas.validator import validate_manifest
|
||||
import yaml
|
||||
|
||||
with open(manifest_path) as f:
|
||||
data = yaml.safe_load(f)
|
||||
|
||||
manifest = validate_manifest(data)
|
||||
|
||||
metadata = ToolMetadata(
|
||||
name=manifest.name,
|
||||
display_name=manifest.display_name,
|
||||
description=manifest.description,
|
||||
version=manifest.version,
|
||||
author=manifest.author,
|
||||
tags=manifest.tags or [],
|
||||
dependencies=manifest.dependencies or [],
|
||||
enabled=manifest.enabled,
|
||||
)
|
||||
|
||||
async with self._lock:
|
||||
self._tools[manifest.name] = metadata
|
||||
self._executors[manifest.name] = executor
|
||||
if config:
|
||||
self._configs[manifest.name] = config
|
||||
|
||||
return metadata
|
||||
|
||||
async def unregister(self, name: str) -> bool:
|
||||
"""注销工具"""
|
||||
async with self._lock:
|
||||
if name in self._tools:
|
||||
del self._tools[name]
|
||||
del self._executors[name]
|
||||
self._configs.pop(name, None)
|
||||
return True
|
||||
return False
|
||||
|
||||
async def enable(self, name: str) -> None:
|
||||
"""启用工具"""
|
||||
async with self._lock:
|
||||
if name in self._tools:
|
||||
self._tools[name].enabled = True
|
||||
|
||||
async def disable(self, name: str) -> None:
|
||||
"""禁用工具"""
|
||||
async with self._lock:
|
||||
if name in self._tools:
|
||||
self._tools[name].enabled = False
|
||||
|
||||
# === 查询方法 ===
|
||||
|
||||
async def get(self, name: str) -> Optional[ToolMetadata]:
|
||||
"""获取工具元数据"""
|
||||
return self._tools.get(name)
|
||||
|
||||
async def get_executor(self, name: str) -> Optional[Callable]:
|
||||
"""获取工具执行器"""
|
||||
return self._executors.get(name)
|
||||
|
||||
async def get_config(self, name: str) -> dict:
|
||||
"""获取工具配置"""
|
||||
return self._configs.get(name, {})
|
||||
|
||||
async def list_all(self) -> List[ToolMetadata]:
|
||||
"""列出所有工具"""
|
||||
return list(self._tools.values())
|
||||
|
||||
async def list_enabled(self) -> List[ToolMetadata]:
|
||||
"""列出已启用的工具"""
|
||||
return [t for t in self._tools.values() if t.enabled]
|
||||
|
||||
async def list_by_tag(self, tag: str) -> List[ToolMetadata]:
|
||||
"""按标签筛选工具"""
|
||||
return [t for t in self._tools.values() if tag in t.tags]
|
||||
|
||||
async def search(self, query: str) -> List[ToolMetadata]:
|
||||
"""搜索工具"""
|
||||
query = query.lower()
|
||||
return [
|
||||
t for t in self._tools.values()
|
||||
if query in t.name.lower()
|
||||
or query in t.description.lower()
|
||||
or query in t.display_name.lower()
|
||||
]
|
||||
|
||||
# === 统计方法 ===
|
||||
|
||||
async def record_call(
|
||||
self,
|
||||
name: str,
|
||||
duration_ms: int,
|
||||
error: bool = False,
|
||||
) -> None:
|
||||
"""记录调用"""
|
||||
async with self._lock:
|
||||
if name in self._tools:
|
||||
tool = self._tools[name]
|
||||
tool.call_count += 1
|
||||
tool.total_duration_ms += duration_ms
|
||||
if error:
|
||||
tool.error_count += 1
|
||||
|
||||
async def get_stats(self) -> dict:
|
||||
"""获取统计信息"""
|
||||
tools = list(self._tools.values())
|
||||
return {
|
||||
"total_tools": len(tools),
|
||||
"enabled_tools": sum(1 for t in tools if t.enabled),
|
||||
"total_calls": sum(t.call_count for t in tools),
|
||||
"total_errors": sum(t.error_count for t in tools),
|
||||
"avg_error_rate": sum(t.error_rate for t in tools) / len(tools) if tools else 0,
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. 工具发现机制
|
||||
|
||||
### 3.1 自动发现
|
||||
|
||||
```python
|
||||
# tools/discovery.py
|
||||
from pathlib import Path
|
||||
from typing import List
|
||||
|
||||
|
||||
class ToolDiscovery:
|
||||
"""工具自动发现"""
|
||||
|
||||
def __init__(self, manifest_dir: Path):
|
||||
self.manifest_dir = manifest_dir
|
||||
|
||||
def discover(self) -> List[Path]:
|
||||
"""发现所有 Manifest 文件"""
|
||||
manifests = list(self.manifest_dir.glob("**/*.yaml"))
|
||||
manifests.extend(self.manifest_dir.glob("**/*.yml"))
|
||||
manifests.extend(self.manifest_dir.glob("**/*.json"))
|
||||
return manifests
|
||||
|
||||
def discover_by_tag(self, tag: str) -> List[Path]:
|
||||
"""按标签发现"""
|
||||
# 读取所有 manifest，筛选标签
|
||||
pass
|
||||
|
||||
async def hot_reload(self, registry: ToolRegistry) -> None:
|
||||
"""热重载工具"""
|
||||
for manifest_path in self.discover():
|
||||
# 重新注册
|
||||
pass
|
||||
```
|
||||
|
||||
### 3.2 启动时注册
|
||||
|
||||
```python
|
||||
# tools/loader.py
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
async def load_all_tools(registry: ToolRegistry) -> None:
|
||||
"""加载所有工具"""
|
||||
manifest_dir = Path(__file__).parent / "manifests"
|
||||
|
||||
discovery = ToolDiscovery(manifest_dir)
|
||||
|
||||
for manifest_path in discovery.discover():
|
||||
tool_name = manifest_path.stem
|
||||
|
||||
# 加载 executor
|
||||
executor = load_executor(manifest_path)
|
||||
|
||||
# 加载配置
|
||||
config = load_config(tool_name)
|
||||
|
||||
# 注册
|
||||
await registry.register(manifest_path, executor, config)
|
||||
|
||||
|
||||
def load_executor(manifest_path: Path) -> Callable:
|
||||
"""加载工具执行器"""
|
||||
import yaml
|
||||
|
||||
with open(manifest_path) as f:
|
||||
manifest = yaml.safe_load(f)
|
||||
|
||||
# 根据运行时类型加载
|
||||
runtime = manifest.get("runtime", "python")
|
||||
|
||||
if runtime == "python":
|
||||
return load_python_executor(manifest)
|
||||
elif runtime == "javascript":
|
||||
return load_js_executor(manifest)
|
||||
else:
|
||||
return load_native_executor(manifest)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. 工具描述生成
|
||||
|
||||
### 4.1 AI 友好的工具描述
|
||||
|
||||
```python
|
||||
# tools/description.py
|
||||
from typing import Dict, List
|
||||
|
||||
|
||||
def generate_tool_description(manifest: dict) -> str:
|
||||
"""生成 AI 友好的工具描述"""
|
||||
lines = [
|
||||
f"## {manifest['display_name']}",
|
||||
f"{manifest['description']}",
|
||||
"",
|
||||
"### 可用命令:",
|
||||
]
|
||||
|
||||
for cmd in manifest.get("commands", []):
|
||||
lines.append(f"#### {cmd['name']}")
|
||||
lines.append(cmd["description"])
|
||||
lines.append("")
|
||||
|
||||
if cmd.get("example"):
|
||||
lines.append("**示例:**")
|
||||
lines.append(f"```\n{cmd['example']}\n```")
|
||||
lines.append("")
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def generate_tools_for_llm(registry: ToolRegistry) -> str:
|
||||
"""生成给 LLM 的工具列表"""
|
||||
tools = registry.list_enabled()
|
||||
|
||||
sections = ["## 可用工具\n"]
|
||||
|
||||
for tool in tools:
|
||||
manifest = load_manifest(tool.name)
|
||||
sections.append(generate_tool_description(manifest))
|
||||
sections.append("\n---\n")
|
||||
|
||||
return "\n".join(sections)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. 权限控制
|
||||
|
||||
### 5.1 工具权限
|
||||
|
||||
```python
|
||||
# tools/permissions.py
|
||||
from enum import Enum
|
||||
from typing import Set
|
||||
|
||||
|
||||
class ToolPermission(str, Enum):
|
||||
"""工具权限"""
|
||||
EXECUTE = "tool:execute"
|
||||
CONFIGURE = "tool:configure"
|
||||
ENABLE = "tool:enable"
|
||||
DISABLE = "tool:disable"
|
||||
|
||||
|
||||
class ToolPermissionChecker:
|
||||
"""工具权限检查"""
|
||||
|
||||
def __init__(self):
|
||||
self._user_permissions: Dict[str, Set[ToolPermission]] = {}
|
||||
self._tool_roles: Dict[str, Set[str]] = {} # tool_name -> required_roles
|
||||
|
||||
def set_user_permissions(
|
||||
self,
|
||||
user_id: str,
|
||||
permissions: Set[ToolPermission],
|
||||
) -> None:
|
||||
"""设置用户权限"""
|
||||
self._user_permissions[user_id] = permissions
|
||||
|
||||
def set_tool_roles(
|
||||
self,
|
||||
tool_name: str,
|
||||
required_roles: Set[str],
|
||||
) -> None:
|
||||
"""设置工具所需角色"""
|
||||
self._tool_roles[tool_name] = required_roles
|
||||
|
||||
def can_execute(self, user_id: str, tool_name: str) -> bool:
|
||||
"""检查用户是否可以执行工具"""
|
||||
# 检查全局权限
|
||||
if ToolPermission.EXECUTE in self._user_permissions.get(user_id, set()):
|
||||
return True
|
||||
|
||||
# 检查工具特定角色
|
||||
required_roles = self._tool_roles.get(tool_name, set())
|
||||
if not required_roles:
|
||||
return True
|
||||
|
||||
# TODO: 检查用户是否有所需角色
|
||||
return False
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. 集成到 Agent
|
||||
|
||||
### 6.1 LangChain 集成
|
||||
|
||||
```python
|
||||
# tools/langchain_adapter.py
|
||||
from typing import List, Optional
|
||||
from langchain.tools import BaseTool
|
||||
from pydantic import BaseModel
|
||||
|
||||
|
||||
class LangChainToolAdapter:
|
||||
"""LangChain 工具适配器"""
|
||||
|
||||
def __init__(self, registry: ToolRegistry):
|
||||
self.registry = registry
|
||||
|
||||
def to_langchain_tools(self) -> List[BaseTool]:
|
||||
"""转换为 LangChain 工具"""
|
||||
tools = []
|
||||
|
||||
for metadata in self.registry.list_enabled():
|
||||
executor = self.registry.get_executor(metadata.name)
|
||||
config = self.registry.get_config(metadata.name)
|
||||
|
||||
tool = self._create_tool(metadata, executor, config)
|
||||
tools.append(tool)
|
||||
|
||||
return tools
|
||||
|
||||
def _create_tool(
|
||||
self,
|
||||
metadata: ToolMetadata,
|
||||
executor: Callable,
|
||||
config: dict,
|
||||
) -> BaseTool:
|
||||
"""创建单个 LangChain 工具"""
|
||||
# 根据 manifest 创建工具
|
||||
pass
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. 实现步骤
|
||||
|
||||
| 步骤 | 任务 | 优先级 |
|
||||
|------|------|--------|
|
||||
| 1 | 实现 ToolRegistry 类 | 🟢 高 |
|
||||
| 2 | 实现 ToolDiscovery | 🟢 高 |
|
||||
| 3 | 实现工具描述生成 | 🟡 中 |
|
||||
| 4 | 实现权限检查 | 🟡 中 |
|
||||
| 5 | 实现 LangChain 适配器 | 🟡 中 |
|
||||
| 6 | 集成到 Agent | 🟢 高 |
|
||||
| 7 | 单元测试 | 🟡 中 |
|
||||
|
||||
---
|
||||
|
||||
## 8. 核心文件变更
|
||||
|
||||
| 文件 | 变更 |
|
||||
|------|------|
|
||||
| `tools/registry.py` | 新增 |
|
||||
| `tools/discovery.py` | 新增 |
|
||||
| `tools/description.py` | 新增 |
|
||||
| `tools/permissions.py` | 新增 |
|
||||
| `tools/langchain_adapter.py` | 新增 |
|
||||
| `tools/__init__.py` | 更新导出 |
|
||||
|
||||
---
|
||||
|
||||
## 9. 工作量估算
|
||||
|
||||
| 任务 | 工作量 |
|
||||
|------|--------|
|
||||
| ToolRegistry | 0.5 天 |
|
||||
| ToolDiscovery | 0.5 天 |
|
||||
| 描述生成 | 0.3 天 |
|
||||
| 权限检查 | 0.3 天 |
|
||||
| LangChain 适配 | 0.3 天 |
|
||||
| 集成 | 0.5 天 |
|
||||
| 单元测试 | 0.5 天 |
|
||||
| **总计** | **2.5 天** |
|
||||
|
||||
---
|
||||
|
||||
## 10. 验收标准
|
||||
|
||||
- [ ] ToolRegistry 可正确注册/注销工具
|
||||
- [ ] ToolDiscovery 可发现所有 Manifest
|
||||
- [ ] 工具描述生成正确
|
||||
- [ ] 权限检查正常工作
|
||||
- [ ] LangChain 工具可正常转换
|
||||
- [ ] Agent 可使用注册的工具
|
||||
- [ ] 单元测试通过
|
||||
@@ -0,0 +1,586 @@
|
||||
# Phase T.3：核心工具实现
|
||||
|
||||
日期：2026-04-04
|
||||
状态：待开始
|
||||
依赖：T.2（待完成）
|
||||
|
||||
---
|
||||
|
||||
## 1. 本阶段目的
|
||||
|
||||
实现 Jarvis 的核心工具：
|
||||
|
||||
- 文件操作工具
|
||||
- 搜索工具
|
||||
- 网页抓取工具
|
||||
- 任务管理工具
|
||||
|
||||
---
|
||||
|
||||
## 2. 文件操作工具
|
||||
|
||||
### 2.1 实现
|
||||
|
||||
```python
|
||||
# tools/implementations/file_operator.py
|
||||
import os
|
||||
import shutil
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
from typing import Optional, List, Dict, Any
|
||||
|
||||
|
||||
class FileOperator:
|
||||
"""文件操作工具"""
|
||||
|
||||
def __init__(self, config: dict):
|
||||
self.allowed_dirs = self._parse_allowed_dirs(
|
||||
config.get("allowed_directories", "")
|
||||
)
|
||||
self.max_file_size = config.get("max_file_size", 10 * 1024 * 1024)
|
||||
|
||||
def _parse_allowed_dirs(self, dirs_str: str) -> Optional[List[str]]:
|
||||
"""解析允许目录"""
|
||||
if not dirs_str:
|
||||
return None
|
||||
return [d.strip() for d in dirs_str.split(",") if d.strip()]
|
||||
|
||||
def _check_path(self, path: str) -> bool:
|
||||
"""检查路径是否允许"""
|
||||
if not self.allowed_dirs:
|
||||
return True
|
||||
resolved = Path(path).resolve()
|
||||
return any(
|
||||
str(resolved).startswith(allowed)
|
||||
for allowed in self.allowed_dirs
|
||||
)
|
||||
|
||||
async def read_file(
|
||||
self,
|
||||
filePath: str,
|
||||
encoding: str = "utf-8",
|
||||
) -> Dict[str, Any]:
|
||||
"""读取文件"""
|
||||
if not self._check_path(filePath):
|
||||
return {"status": "error", "error": "路径不在允许范围内"}
|
||||
|
||||
path = Path(filePath)
|
||||
|
||||
if not path.exists():
|
||||
return {"status": "error", "error": "文件不存在"}
|
||||
|
||||
if path.stat().st_size > self.max_file_size:
|
||||
return {"status": "error", "error": "文件过大"}
|
||||
|
||||
# 根据扩展名处理
|
||||
suffix = path.suffix.lower()
|
||||
if suffix in [".pdf", ".docx", ".xlsx", ".xls", ".csv"]:
|
||||
return await self._read_binary_file(path)
|
||||
|
||||
try:
|
||||
content = path.read_text(encoding=encoding)
|
||||
return {"status": "success", "result": content}
|
||||
except Exception as e:
|
||||
return {"status": "error", "error": str(e)}
|
||||
|
||||
async def _read_binary_file(self, path: Path) -> Dict[str, Any]:
|
||||
"""读取二进制文件"""
|
||||
suffix = path.suffix.lower()
|
||||
|
||||
if suffix == ".pdf":
|
||||
return await self._read_pdf(path)
|
||||
elif suffix in [".docx", ".doc"]:
|
||||
return await self._read_docx(path)
|
||||
elif suffix in [".xlsx", ".xls"]:
|
||||
return await self._read_xlsx(path)
|
||||
elif suffix == ".csv":
|
||||
return await self._read_csv(path)
|
||||
|
||||
return {"status": "error", "error": "不支持的文件格式"}
|
||||
|
||||
async def write_file(
|
||||
self,
|
||||
filePath: str,
|
||||
content: str,
|
||||
) -> Dict[str, Any]:
|
||||
"""写入文件"""
|
||||
if not self._check_path(filePath):
|
||||
return {"status": "error", "error": "路径不在允许范围内"}
|
||||
|
||||
path = Path(filePath)
|
||||
|
||||
# 如果文件存在，自动创建新文件名
|
||||
if path.exists():
|
||||
path = self._get_unique_path(path)
|
||||
|
||||
try:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(content, encoding="utf-8")
|
||||
return {
|
||||
"status": "success",
|
||||
"result": f"文件已保存: {path.name}",
|
||||
"path": str(path),
|
||||
}
|
||||
except Exception as e:
|
||||
return {"status": "error", "error": str(e)}
|
||||
|
||||
def _get_unique_path(self, path: Path) -> Path:
|
||||
"""获取唯一路径"""
|
||||
if not path.exists():
|
||||
return path
|
||||
|
||||
stem = path.stem
|
||||
suffix = path.suffix
|
||||
parent = path.parent
|
||||
counter = 1
|
||||
|
||||
while True:
|
||||
new_path = parent / f"{stem}({counter}){suffix}"
|
||||
if not new_path.exists():
|
||||
return new_path
|
||||
counter += 1
|
||||
|
||||
async def list_directory(
|
||||
self,
|
||||
directoryPath: str,
|
||||
showHidden: bool = False,
|
||||
) -> Dict[str, Any]:
|
||||
"""列出目录"""
|
||||
if not self._check_path(directoryPath):
|
||||
return {"status": "error", "error": "路径不在允许范围内"}
|
||||
|
||||
path = Path(directoryPath)
|
||||
|
||||
if not path.exists():
|
||||
return {"status": "error", "error": "目录不存在"}
|
||||
|
||||
if not path.is_dir():
|
||||
return {"status": "error", "error": "不是目录"}
|
||||
|
||||
items = []
|
||||
for item in path.iterdir():
|
||||
if not showHidden and item.name.startswith("."):
|
||||
continue
|
||||
items.append({
|
||||
"name": item.name,
|
||||
"type": "directory" if item.is_dir() else "file",
|
||||
"size": item.stat().st_size if item.is_file() else None,
|
||||
})
|
||||
|
||||
return {"status": "success", "result": items}
|
||||
|
||||
async def search_files(
|
||||
self,
|
||||
searchPath: str,
|
||||
pattern: str,
|
||||
**options,
|
||||
) -> Dict[str, Any]:
|
||||
"""搜索文件"""
|
||||
import fnmatch
|
||||
|
||||
if not self._check_path(searchPath):
|
||||
return {"status": "error", "error": "路径不在允许范围内"}
|
||||
|
||||
path = Path(searchPath)
|
||||
if not path.exists():
|
||||
return {"status": "error", "error": "路径不存在"}
|
||||
|
||||
case_sensitive = options.get("caseSensitive", False)
|
||||
file_type = options.get("fileType", "all")
|
||||
include_hidden = options.get("includeHidden", False)
|
||||
|
||||
results = []
|
||||
for item in path.rglob("*"):
|
||||
if not include_hidden and item.name.startswith("."):
|
||||
continue
|
||||
|
||||
if not fnmatch.fnmatch(item.name, pattern):
|
||||
continue
|
||||
|
||||
if file_type == "file" and item.is_dir():
|
||||
continue
|
||||
if file_type == "directory" and item.is_file():
|
||||
continue
|
||||
|
||||
results.append(str(item))
|
||||
|
||||
return {"status": "success", "result": results[:100]} # 限制结果数
|
||||
```
|
||||
|
||||
### 2.2 Manifest 绑定
|
||||
|
||||
```python
|
||||
# tools/implementations/__init__.py
|
||||
from tools.implementations.file_operator import FileOperator
|
||||
|
||||
|
||||
def create_file_operator_executor(config: dict):
|
||||
"""创建文件操作执行器"""
|
||||
operator = FileOperator(config)
|
||||
|
||||
async def execute(command: str, parameters: dict) -> dict:
|
||||
if command == "read_file":
|
||||
return await operator.read_file(**parameters)
|
||||
elif command == "write_file":
|
||||
return await operator.write_file(**parameters)
|
||||
elif command == "list_directory":
|
||||
return await operator.list_directory(**parameters)
|
||||
elif command == "search_files":
|
||||
return await operator.search_files(**parameters)
|
||||
else:
|
||||
return {"status": "error", "error": f"未知命令: {command}"}
|
||||
|
||||
return execute
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. 搜索工具
|
||||
|
||||
### 3.1 实现
|
||||
|
||||
```python
|
||||
# tools/implementations/web_search.py
|
||||
import asyncio
|
||||
from typing import Dict, Any, List, Optional
|
||||
|
||||
|
||||
class WebSearch:
|
||||
"""联网搜索工具"""
|
||||
|
||||
def __init__(self, config: dict):
|
||||
self.api_key = config.get("api_key")
|
||||
self.max_results = config.get("max_results", 10)
|
||||
|
||||
async def search(
|
||||
self,
|
||||
query: str,
|
||||
max_results: Optional[int] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""执行搜索"""
|
||||
try:
|
||||
# 实现搜索逻辑
|
||||
results = await self._do_search(
|
||||
query,
|
||||
max_results or self.max_results,
|
||||
)
|
||||
return {"status": "success", "result": results}
|
||||
except Exception as e:
|
||||
return {"status": "error", "error": str(e)}
|
||||
|
||||
async def _do_search(self, query: str, limit: int) -> List[dict]:
|
||||
"""实际搜索"""
|
||||
# TODO: 接入搜索 API
|
||||
return []
|
||||
|
||||
async def deep_search(
|
||||
self,
|
||||
query: str,
|
||||
keywords: List[str],
|
||||
) -> Dict[str, Any]:
|
||||
"""深度搜索"""
|
||||
try:
|
||||
# 并发执行多个搜索
|
||||
tasks = [
|
||||
self._do_search(kw, 5)
|
||||
for kw in [query] + keywords
|
||||
]
|
||||
results = await asyncio.gather(*tasks)
|
||||
|
||||
# 聚合结果
|
||||
aggregated = self._aggregate_results(results)
|
||||
|
||||
return {"status": "success", "result": aggregated}
|
||||
except Exception as e:
|
||||
return {"status": "error", "error": str(e)}
|
||||
|
||||
def _aggregate_results(self, results: List[List[dict]]) -> dict:
|
||||
"""聚合搜索结果"""
|
||||
# TODO: 实现结果聚合
|
||||
return {"summary": "聚合结果", "sources": []}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. 网页抓取工具
|
||||
|
||||
### 4.1 实现
|
||||
|
||||
```python
|
||||
# tools/implementations/web_fetch.py
|
||||
import asyncio
|
||||
from typing import Dict, Any, Optional
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
@dataclass
|
||||
class FetchResult:
|
||||
"""抓取结果"""
|
||||
url: str
|
||||
title: Optional[str]
|
||||
content: str
|
||||
images: List[str]
|
||||
links: List[str]
|
||||
status: int
|
||||
|
||||
|
||||
class WebFetch:
|
||||
"""网页抓取工具"""
|
||||
|
||||
def __init__(self, config: dict):
|
||||
self.timeout = config.get("timeout", 30)
|
||||
self.user_agent = config.get(
|
||||
"user_agent",
|
||||
"Mozilla/5.0 (compatible; Jarvis/1.0)"
|
||||
)
|
||||
|
||||
async def fetch(
|
||||
self,
|
||||
url: str,
|
||||
include_images: bool = True,
|
||||
) -> Dict[str, Any]:
|
||||
"""抓取网页"""
|
||||
try:
|
||||
result = await self._do_fetch(url, include_images)
|
||||
return {
|
||||
"status": "success",
|
||||
"result": {
|
||||
"url": result.url,
|
||||
"title": result.title,
|
||||
"content": result.content,
|
||||
"images": result.images,
|
||||
"status": result.status,
|
||||
}
|
||||
}
|
||||
except Exception as e:
|
||||
return {"status": "error", "error": str(e)}
|
||||
|
||||
async def _do_fetch(
|
||||
self,
|
||||
url: str,
|
||||
include_images: bool,
|
||||
) -> FetchResult:
|
||||
"""实际抓取"""
|
||||
import httpx
|
||||
|
||||
async with httpx.AsyncClient(timeout=self.timeout) as client:
|
||||
response = await client.get(
|
||||
url,
|
||||
headers={"User-Agent": self.user_agent},
|
||||
)
|
||||
response.raise_for_status()
|
||||
|
||||
# TODO: 解析 HTML 提取内容
|
||||
return FetchResult(
|
||||
url=url,
|
||||
title=None,
|
||||
content=response.text,
|
||||
images=[],
|
||||
links=[],
|
||||
status=response.status_code,
|
||||
)
|
||||
|
||||
async def screenshot(
|
||||
self,
|
||||
url: str,
|
||||
) -> Dict[str, Any]:
|
||||
"""截取网页截图"""
|
||||
# TODO: 接入截图服务
|
||||
return {"status": "error", "error": "未实现"}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. 任务管理工具
|
||||
|
||||
### 5.1 实现
|
||||
|
||||
```python
|
||||
# tools/implementations/task_manager.py
|
||||
from typing import Dict, Any, List, Optional
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime
|
||||
from enum import Enum
|
||||
|
||||
|
||||
class TaskStatus(str, Enum):
|
||||
PENDING = "pending"
|
||||
RUNNING = "running"
|
||||
COMPLETED = "completed"
|
||||
FAILED = "failed"
|
||||
|
||||
|
||||
@dataclass
|
||||
class Task:
|
||||
"""任务"""
|
||||
id: str
|
||||
name: str
|
||||
description: str
|
||||
status: TaskStatus = TaskStatus.PENDING
|
||||
created_at: datetime = field(default_factory=datetime.utcnow)
|
||||
scheduled_at: Optional[datetime] = None
|
||||
result: Optional[Any] = None
|
||||
error: Optional[str] = None
|
||||
|
||||
|
||||
class TaskManager:
|
||||
"""任务管理工具"""
|
||||
|
||||
def __init__(self, config: dict):
|
||||
self._tasks: Dict[str, Task] = {}
|
||||
|
||||
async def create_task(
|
||||
self,
|
||||
name: str,
|
||||
description: str,
|
||||
scheduled_at: Optional[datetime] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""创建任务"""
|
||||
import uuid
|
||||
|
||||
task_id = str(uuid.uuid4())[:8]
|
||||
task = Task(
|
||||
id=task_id,
|
||||
name=name,
|
||||
description=description,
|
||||
scheduled_at=scheduled_at,
|
||||
)
|
||||
self._tasks[task_id] = task
|
||||
|
||||
return {
|
||||
"status": "success",
|
||||
"result": {
|
||||
"id": task_id,
|
||||
"name": task.name,
|
||||
"status": task.status.value,
|
||||
}
|
||||
}
|
||||
|
||||
async def list_tasks(
|
||||
self,
|
||||
status: Optional[str] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""列出任务"""
|
||||
tasks = list(self._tasks.values())
|
||||
|
||||
if status:
|
||||
tasks = [t for t in tasks if t.status.value == status]
|
||||
|
||||
return {
|
||||
"status": "success",
|
||||
"result": [
|
||||
{
|
||||
"id": t.id,
|
||||
"name": t.name,
|
||||
"status": t.status.value,
|
||||
"created_at": t.created_at.isoformat(),
|
||||
}
|
||||
for t in tasks
|
||||
]
|
||||
}
|
||||
|
||||
async def get_task(self, task_id: str) -> Dict[str, Any]:
|
||||
"""获取任务"""
|
||||
task = self._tasks.get(task_id)
|
||||
if not task:
|
||||
return {"status": "error", "error": "任务不存在"}
|
||||
|
||||
return {
|
||||
"status": "success",
|
||||
"result": {
|
||||
"id": task.id,
|
||||
"name": task.name,
|
||||
"description": task.description,
|
||||
"status": task.status.value,
|
||||
"result": task.result,
|
||||
"error": task.error,
|
||||
}
|
||||
}
|
||||
|
||||
async def complete_task(
|
||||
self,
|
||||
task_id: str,
|
||||
result: Any,
|
||||
) -> Dict[str, Any]:
|
||||
"""完成任务"""
|
||||
task = self._tasks.get(task_id)
|
||||
if not task:
|
||||
return {"status": "error", "error": "任务不存在"}
|
||||
|
||||
task.status = TaskStatus.COMPLETED
|
||||
task.result = result
|
||||
|
||||
return {"status": "success"}
|
||||
|
||||
async def fail_task(
|
||||
self,
|
||||
task_id: str,
|
||||
error: str,
|
||||
) -> Dict[str, Any]:
|
||||
"""标记任务失败"""
|
||||
task = self._tasks.get(task_id)
|
||||
if not task:
|
||||
return {"status": "error", "error": "任务不存在"}
|
||||
|
||||
task.status = TaskStatus.FAILED
|
||||
task.error = error
|
||||
|
||||
return {"status": "success"}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. 实现步骤
|
||||
|
||||
| 步骤 | 任务 | 优先级 |
|
||||
|------|------|--------|
|
||||
| 1 | 实现 FileOperator | 🟢 高 |
|
||||
| 2 | 实现 WebSearch | 🟡 中 |
|
||||
| 3 | 实现 WebFetch | 🟡 中 |
|
||||
| 4 | 实现 TaskManager | 🟡 中 |
|
||||
| 5 | 创建 Manifest 文件 | 🟢 高 |
|
||||
| 6 | 注册到工具中心 | 🟢 高 |
|
||||
| 7 | 单元测试 | 🟡 中 |
|
||||
|
||||
---
|
||||
|
||||
## 7. 核心文件变更
|
||||
|
||||
| 文件 | 变更 |
|
||||
|------|------|
|
||||
| `tools/implementations/__init__.py` | 新增 |
|
||||
| `tools/implementations/file_operator.py` | 新增 |
|
||||
| `tools/implementations/web_search.py` | 新增 |
|
||||
| `tools/implementations/web_fetch.py` | 新增 |
|
||||
| `tools/implementations/task_manager.py` | 新增 |
|
||||
| `tools/manifests/file_operator.yaml` | 更新 |
|
||||
| `tools/manifests/web_search.yaml` | 新增 |
|
||||
| `tools/manifests/web_fetch.yaml` | 新增 |
|
||||
| `tools/manifests/task_manager.yaml` | 新增 |
|
||||
|
||||
---
|
||||
|
||||
## 8. 工作量估算
|
||||
|
||||
| 任务 | 工作量 |
|
||||
|------|--------|
|
||||
| FileOperator | 1.5 天 |
|
||||
| WebSearch | 1 天 |
|
||||
| WebFetch | 1 天 |
|
||||
| TaskManager | 0.5 天 |
|
||||
| Manifest + 注册 | 0.5 天 |
|
||||
| 单元测试 | 0.5 天 |
|
||||
| **总计** | **5 天** |
|
||||
|
||||
---
|
||||
|
||||
## 9. 验收标准
|
||||
|
||||
- [ ] FileOperator 可正确读写文件
|
||||
- [ ] FileOperator 支持多种格式解析
|
||||
- [ ] FileOperator 路径安全检查正常
|
||||
- [ ] WebSearch 可执行搜索
|
||||
- [ ] WebFetch 可抓取网页
|
||||
- [ ] TaskManager 可管理任务
|
||||
- [ ] 所有工具注册到工具中心
|
||||
- [ ] 单元测试通过
|
||||
642
development-doc/plan/tool-update/phase-t-4-advanced.md
Normal file
642
development-doc/plan/tool-update/phase-t-4-advanced.md
Normal file
@@ -0,0 +1,642 @@
|
||||
# Phase T.4：高级特性
|
||||
|
||||
日期：2026-04-04
|
||||
状态：待开始
|
||||
依赖：T.3（待完成）
|
||||
|
||||
---
|
||||
|
||||
## 1. 本阶段目的
|
||||
|
||||
实现 Jarvis 工具系统的高级特性：
|
||||
|
||||
- 多运行时支持（Python/JS/原生）
|
||||
- Agent 间协作
|
||||
- 定时任务
|
||||
|
||||
---
|
||||
|
||||
## 2. 多运行时支持
|
||||
|
||||
### 2.1 运行时架构
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Runtime Manager │
|
||||
│ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │
|
||||
│ │ Python │ │ JS │ │ Native │ │ WASM │ │
|
||||
│ │ Runtime │ │ Runtime │ │ Runtime │ │ Runtime │ │
|
||||
│ └────┬────┘ └────┬────┘ └────┬────┘ └────┬────┘ │
|
||||
│ │ │ │ │ │
|
||||
│ └─────────────┴──────┬──────┴─────────────┘ │
|
||||
│ │ │
|
||||
│ ┌────────┴────────┐ │
|
||||
│ │ Tool Executor │ │
|
||||
│ │ (统一接口) │ │
|
||||
│ └─────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### 2.2 运行时基类
|
||||
|
||||
```python
|
||||
# tools/runtime/base.py
|
||||
from abc import ABC, abstractmethod
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
|
||||
class BaseRuntime(ABC):
|
||||
"""运行时基类"""
|
||||
|
||||
@abstractmethod
|
||||
async def execute(
|
||||
self,
|
||||
entry: str,
|
||||
command: str,
|
||||
parameters: Dict[str, Any],
|
||||
timeout: int,
|
||||
) -> Dict[str, Any]:
|
||||
"""执行工具"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
async def validate(self, entry: str) -> bool:
|
||||
"""验证工具是否可用"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def get_name(self) -> str:
|
||||
"""获取运行时名称"""
|
||||
pass
|
||||
```
|
||||
|
||||
### 2.3 Python 运行时
|
||||
|
||||
```python
|
||||
# tools/runtime/python_runtime.py
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
from tools.runtime.base import BaseRuntime
|
||||
|
||||
|
||||
class PythonRuntime(BaseRuntime):
|
||||
"""Python 运行时"""
|
||||
|
||||
def __init__(self):
|
||||
self._executors: Dict[str, Callable] = {}
|
||||
|
||||
def get_name(self) -> str:
|
||||
return "python"
|
||||
|
||||
async def validate(self, entry: str) -> bool:
|
||||
path = Path(entry)
|
||||
return path.exists() and path.suffix == ".py"
|
||||
|
||||
async def execute(
|
||||
self,
|
||||
entry: str,
|
||||
command: str,
|
||||
parameters: Dict[str, Any],
|
||||
timeout: int,
|
||||
) -> Dict[str, Any]:
|
||||
# 动态加载并执行
|
||||
# 或者通过 subprocess 调用
|
||||
pass
|
||||
```
|
||||
|
||||
### 2.4 JavaScript 运行时
|
||||
|
||||
```python
|
||||
# tools/runtime/js_runtime.py
|
||||
import asyncio
|
||||
import subprocess
|
||||
import json
|
||||
from tools.runtime.base import BaseRuntime
|
||||
|
||||
|
||||
class JavaScriptRuntime(BaseRuntime):
|
||||
"""JavaScript 运行时"""
|
||||
|
||||
def __init__(self):
|
||||
self.node_path = "node" # 可配置
|
||||
|
||||
def get_name(self) -> str:
|
||||
return "javascript"
|
||||
|
||||
async def validate(self, entry: str) -> bool:
|
||||
# 检查 node 是否可用
|
||||
try:
|
||||
result = await asyncio.create_subprocess_exec(
|
||||
self.node_path, "--version",
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
)
|
||||
return result.returncode == 0
|
||||
except:
|
||||
return False
|
||||
|
||||
async def execute(
|
||||
self,
|
||||
entry: str,
|
||||
command: str,
|
||||
parameters: Dict[str, Any],
|
||||
timeout: int,
|
||||
) -> Dict[str, Any]:
|
||||
# 通过 stdio 调用 Node.js 脚本
|
||||
input_data = json.dumps({
|
||||
"command": command,
|
||||
"parameters": parameters,
|
||||
})
|
||||
|
||||
process = await asyncio.create_subprocess_exec(
|
||||
self.node_path, entry,
|
||||
stdin=asyncio.subprocess.PIPE,
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
|
||||
stdout, stderr = await asyncio.wait_for(
|
||||
process.communicate(input=input_data.encode()),
|
||||
timeout=timeout / 1000,
|
||||
)
|
||||
|
||||
if process.returncode != 0:
|
||||
return {
|
||||
"status": "error",
|
||||
"error": stderr.decode(),
|
||||
}
|
||||
|
||||
return json.loads(stdout.decode())
|
||||
```
|
||||
|
||||
### 2.5 原生运行时
|
||||
|
||||
```python
|
||||
# tools/runtime/native_runtime.py
|
||||
import asyncio
|
||||
import subprocess
|
||||
from tools.runtime.base import BaseRuntime
|
||||
|
||||
|
||||
class NativeRuntime(BaseRuntime):
|
||||
"""原生二进制运行时"""
|
||||
|
||||
def get_name(self) -> str:
|
||||
return "native"
|
||||
|
||||
async def validate(self, entry: str) -> bool:
|
||||
from pathlib import Path
|
||||
path = Path(entry)
|
||||
return path.exists() and path.stat().st_mode & 0o111
|
||||
|
||||
async def execute(
|
||||
self,
|
||||
entry: str,
|
||||
command: str,
|
||||
parameters: Dict[str, Any],
|
||||
timeout: int,
|
||||
) -> Dict[str, Any]:
|
||||
# 调用原生可执行文件
|
||||
args = [entry, command] + self._format_args(parameters)
|
||||
|
||||
process = await asyncio.create_subprocess_exec(
|
||||
*args,
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
|
||||
stdout, stderr = await asyncio.wait_for(
|
||||
process.communicate(),
|
||||
timeout=timeout / 1000,
|
||||
)
|
||||
|
||||
if process.returncode != 0:
|
||||
return {
|
||||
"status": "error",
|
||||
"error": stderr.decode(),
|
||||
}
|
||||
|
||||
return {
|
||||
"status": "success",
|
||||
"result": stdout.decode(),
|
||||
}
|
||||
|
||||
def _format_args(self, parameters: Dict[str, Any]) -> list:
|
||||
"""格式化参数"""
|
||||
args = []
|
||||
for key, value in parameters.items():
|
||||
args.extend([f"--{key}", str(value)])
|
||||
return args
|
||||
```
|
||||
|
||||
### 2.6 运行时管理器
|
||||
|
||||
```python
|
||||
# tools/runtime/manager.py
|
||||
from tools.runtime.base import BaseRuntime
|
||||
from tools.runtime.python_runtime import PythonRuntime
|
||||
from tools.runtime.js_runtime import JavaScriptRuntime
|
||||
from tools.runtime.native_runtime import NativeRuntime
|
||||
|
||||
|
||||
class RuntimeManager:
|
||||
"""运行时管理器"""
|
||||
|
||||
def __init__(self):
|
||||
self._runtimes: Dict[str, BaseRuntime] = {
|
||||
"python": PythonRuntime(),
|
||||
"javascript": JavaScriptRuntime(),
|
||||
"native": NativeRuntime(),
|
||||
}
|
||||
|
||||
def get_runtime(self, name: str) -> BaseRuntime:
|
||||
return self._runtimes.get(name)
|
||||
|
||||
async def execute(
|
||||
self,
|
||||
runtime_name: str,
|
||||
entry: str,
|
||||
command: str,
|
||||
parameters: Dict[str, Any],
|
||||
timeout: int,
|
||||
) -> Dict[str, Any]:
|
||||
runtime = self.get_runtime(runtime_name)
|
||||
if not runtime:
|
||||
return {
|
||||
"status": "error",
|
||||
"error": f"未知运行时: {runtime_name}",
|
||||
}
|
||||
|
||||
return await runtime.execute(entry, command, parameters, timeout)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Agent 间协作
|
||||
|
||||
### 3.1 协作协议
|
||||
|
||||
```python
|
||||
# agents/tools/collaboration.py
|
||||
from typing import Dict, Any, Optional, List
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from enum import Enum
|
||||
|
||||
|
||||
class MessageType(str, Enum):
|
||||
REQUEST = "request" # 请求协作
|
||||
RESPONSE = "response" # 响应结果
|
||||
PROGRESS = "progress" # 进度更新
|
||||
CANCEL = "cancel" # 取消请求
|
||||
|
||||
|
||||
@dataclass
|
||||
class CollaborationMessage:
|
||||
"""协作消息"""
|
||||
id: str
|
||||
type: MessageType
|
||||
from_agent: str
|
||||
to_agent: str
|
||||
content: Any
|
||||
metadata: Dict[str, Any]
|
||||
timestamp: datetime = None
|
||||
|
||||
def __post_init__(self):
|
||||
if self.timestamp is None:
|
||||
self.timestamp = datetime.utcnow()
|
||||
|
||||
|
||||
class CollaborationProtocol:
|
||||
"""Agent 协作协议"""
|
||||
|
||||
def __init__(self):
|
||||
self._pending_requests: Dict[str, CollaborationMessage] = {}
|
||||
self._handlers: Dict[str, callable] = {}
|
||||
|
||||
def register_handler(self, tool_name: str, handler: callable) -> None:
|
||||
"""注册工具处理器"""
|
||||
self._handlers[tool_name] = handler
|
||||
|
||||
async def request_collaboration(
|
||||
self,
|
||||
from_agent: str,
|
||||
to_agent: str,
|
||||
tool_name: str,
|
||||
parameters: Dict[str, Any],
|
||||
timeout: int = 30000,
|
||||
) -> Dict[str, Any]:
|
||||
"""请求协作"""
|
||||
import uuid
|
||||
|
||||
request_id = str(uuid.uuid4())
|
||||
|
||||
message = CollaborationMessage(
|
||||
id=request_id,
|
||||
type=MessageType.REQUEST,
|
||||
from_agent=from_agent,
|
||||
to_agent=to_agent,
|
||||
content={
|
||||
"tool": tool_name,
|
||||
"parameters": parameters,
|
||||
},
|
||||
metadata={"timeout": timeout},
|
||||
)
|
||||
|
||||
self._pending_requests[request_id] = message
|
||||
|
||||
# 发送请求
|
||||
await self._send_message(message)
|
||||
|
||||
# 等待响应
|
||||
try:
|
||||
response = await self._wait_for_response(
|
||||
request_id,
|
||||
timeout,
|
||||
)
|
||||
return response
|
||||
except TimeoutError:
|
||||
return {
|
||||
"status": "error",
|
||||
"error": "协作请求超时",
|
||||
}
|
||||
|
||||
async def handle_request(
|
||||
self,
|
||||
message: CollaborationMessage,
|
||||
) -> CollaborationMessage:
|
||||
"""处理协作请求"""
|
||||
tool_name = message.content["tool"]
|
||||
parameters = message.content["parameters"]
|
||||
|
||||
handler = self._handlers.get(tool_name)
|
||||
if not handler:
|
||||
return CollaborationMessage(
|
||||
id=str(uuid.uuid4()),
|
||||
type=MessageType.RESPONSE,
|
||||
from_agent=message.to_agent,
|
||||
to_agent=message.from_agent,
|
||||
content={
|
||||
"status": "error",
|
||||
"error": f"未知工具: {tool_name}",
|
||||
},
|
||||
metadata={},
|
||||
)
|
||||
|
||||
try:
|
||||
result = await handler(**parameters)
|
||||
return CollaborationMessage(
|
||||
id=str(uuid.uuid4()),
|
||||
type=MessageType.RESPONSE,
|
||||
from_agent=message.to_agent,
|
||||
to_agent=message.from_agent,
|
||||
content={"status": "success", "result": result},
|
||||
metadata={},
|
||||
)
|
||||
except Exception as e:
|
||||
return CollaborationMessage(
|
||||
id=str(uuid.uuid4()),
|
||||
type=MessageType.RESPONSE,
|
||||
from_agent=message.to_agent,
|
||||
to_agent=message.from_agent,
|
||||
content={"status": "error", "error": str(e)},
|
||||
metadata={},
|
||||
)
|
||||
|
||||
async def _send_message(self, message: CollaborationMessage) -> None:
|
||||
"""发送消息"""
|
||||
# TODO: 实现消息发送（WebSocket/消息队列）
|
||||
pass
|
||||
|
||||
async def _wait_for_response(
|
||||
self,
|
||||
request_id: str,
|
||||
timeout: int,
|
||||
) -> Dict[str, Any]:
|
||||
"""等待响应"""
|
||||
# TODO: 实现等待逻辑
|
||||
pass
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. 定时任务
|
||||
|
||||
### 4.1 定时任务服务
|
||||
|
||||
```python
|
||||
# tools/scheduler.py
|
||||
import asyncio
|
||||
from typing import Dict, Any, Callable, Optional
|
||||
from datetime import datetime, timedelta
|
||||
from dataclasses import dataclass, field
|
||||
from enum import Enum
|
||||
|
||||
|
||||
class ScheduleType(str, Enum):
|
||||
ONCE = "once" # 单次
|
||||
INTERVAL = "interval" # 间隔
|
||||
CRON = "cron" # Cron 表达式
|
||||
|
||||
|
||||
@dataclass
|
||||
class ScheduledTask:
|
||||
"""定时任务"""
|
||||
id: str
|
||||
name: str
|
||||
schedule_type: ScheduleType
|
||||
schedule_value: str # 时间/间隔/cron
|
||||
tool_name: str
|
||||
parameters: Dict[str, Any]
|
||||
enabled: bool = True
|
||||
last_run: Optional[datetime] = None
|
||||
next_run: Optional[datetime] = None
|
||||
run_count: int = 0
|
||||
callback: Optional[Callable] = field(default=None)
|
||||
|
||||
|
||||
class ToolScheduler:
|
||||
"""工具定时调度器"""
|
||||
|
||||
def __init__(self):
|
||||
self._tasks: Dict[str, ScheduledTask] = {}
|
||||
self._running = False
|
||||
self._loop_task = None
|
||||
|
||||
async def schedule(
|
||||
self,
|
||||
name: str,
|
||||
schedule_type: ScheduleType,
|
||||
schedule_value: str,
|
||||
tool_name: str,
|
||||
parameters: Dict[str, Any],
|
||||
callback: Optional[Callable] = None,
|
||||
) -> str:
|
||||
"""创建定时任务"""
|
||||
import uuid
|
||||
|
||||
task_id = str(uuid.uuid4())[:8]
|
||||
|
||||
task = ScheduledTask(
|
||||
id=task_id,
|
||||
name=name,
|
||||
schedule_type=schedule_type,
|
||||
schedule_value=schedule_value,
|
||||
tool_name=tool_name,
|
||||
parameters=parameters,
|
||||
callback=callback,
|
||||
)
|
||||
|
||||
task.next_run = self._calculate_next_run(task)
|
||||
self._tasks[task_id] = task
|
||||
|
||||
# 启动调度器
|
||||
if not self._running:
|
||||
await self.start()
|
||||
|
||||
return task_id
|
||||
|
||||
def _calculate_next_run(self, task: ScheduledTask) -> datetime:
|
||||
"""计算下次运行时间"""
|
||||
now = datetime.utcnow()
|
||||
|
||||
if task.schedule_type == ScheduleType.ONCE:
|
||||
return datetime.fromisoformat(task.schedule_value)
|
||||
|
||||
elif task.schedule_type == ScheduleType.INTERVAL:
|
||||
seconds = int(task.schedule_value)
|
||||
return now + timedelta(seconds=seconds)
|
||||
|
||||
elif task.schedule_type == ScheduleType.CRON:
|
||||
# 解析 cron 表达式
|
||||
# TODO: 实现 cron 解析
|
||||
return now + timedelta(hours=1)
|
||||
|
||||
return now
|
||||
|
||||
async def start(self) -> None:
|
||||
"""启动调度器"""
|
||||
self._running = True
|
||||
self._loop_task = asyncio.create_task(self._run_loop())
|
||||
|
||||
async def stop(self) -> None:
|
||||
"""停止调度器"""
|
||||
self._running = False
|
||||
if self._loop_task:
|
||||
self._loop_task.cancel()
|
||||
|
||||
async def _run_loop(self) -> None:
|
||||
"""调度循环"""
|
||||
while self._running:
|
||||
now = datetime.utcnow()
|
||||
|
||||
for task in self._tasks.values():
|
||||
if not task.enabled:
|
||||
continue
|
||||
|
||||
if task.next_run and task.next_run <= now:
|
||||
await self._execute_task(task)
|
||||
|
||||
await asyncio.sleep(1) # 每秒检查一次
|
||||
|
||||
async def _execute_task(self, task: ScheduledTask) -> None:
|
||||
"""执行任务"""
|
||||
# 调用工具
|
||||
executor = get_executor(task.tool_name)
|
||||
result = await executor(
|
||||
command=task.parameters.get("command"),
|
||||
parameters=task.parameters,
|
||||
)
|
||||
|
||||
# 更新状态
|
||||
task.last_run = datetime.utcnow()
|
||||
task.run_count += 1
|
||||
|
||||
# 计算下次运行
|
||||
if task.schedule_type != ScheduleType.ONCE:
|
||||
task.next_run = self._calculate_next_run(task)
|
||||
else:
|
||||
task.enabled = False
|
||||
|
||||
# 调用回调
|
||||
if task.callback:
|
||||
await task.callback(task, result)
|
||||
|
||||
async def cancel(self, task_id: str) -> bool:
|
||||
"""取消任务"""
|
||||
if task_id in self._tasks:
|
||||
del self._tasks[task_id]
|
||||
return True
|
||||
return False
|
||||
|
||||
async def list_tasks(self) -> list:
|
||||
"""列出所有任务"""
|
||||
return [
|
||||
{
|
||||
"id": t.id,
|
||||
"name": t.name,
|
||||
"type": t.schedule_type.value,
|
||||
"enabled": t.enabled,
|
||||
"next_run": t.next_run.isoformat() if t.next_run else None,
|
||||
"run_count": t.run_count,
|
||||
}
|
||||
for t in self._tasks.values()
|
||||
]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. 实现步骤
|
||||
|
||||
| 步骤 | 任务 | 优先级 |
|
||||
|------|------|--------|
|
||||
| 1 | 实现运行时基类 | 🟢 高 |
|
||||
| 2 | 实现 Python 运行时 | 🟢 高 |
|
||||
| 3 | 实现 JS 运行时 | 🟡 中 |
|
||||
| 4 | 实现原生运行时 | 🟡 中 |
|
||||
| 5 | 实现运行时管理器 | 🟢 高 |
|
||||
| 6 | 实现协作协议 | 🟡 中 |
|
||||
| 7 | 实现定时调度器 | 🟡 中 |
|
||||
| 8 | 单元测试 | 🟡 中 |
|
||||
|
||||
---
|
||||
|
||||
## 6. 核心文件变更
|
||||
|
||||
| 文件 | 变更 |
|
||||
|------|------|
|
||||
| `tools/runtime/__init__.py` | 新增 |
|
||||
| `tools/runtime/base.py` | 新增 |
|
||||
| `tools/runtime/python_runtime.py` | 新增 |
|
||||
| `tools/runtime/js_runtime.py` | 新增 |
|
||||
| `tools/runtime/native_runtime.py` | 新增 |
|
||||
| `tools/runtime/manager.py` | 新增 |
|
||||
| `agents/tools/collaboration.py` | 新增 |
|
||||
| `tools/scheduler.py` | 新增 |
|
||||
|
||||
---
|
||||
|
||||
## 7. 工作量估算
|
||||
|
||||
| 任务 | 工作量 |
|
||||
|------|--------|
|
||||
| 运行时基类 | 0.5 天 |
|
||||
| Python 运行时 | 0.5 天 |
|
||||
| JS 运行时 | 0.5 天 |
|
||||
| 原生运行时 | 0.5 天 |
|
||||
| 运行时管理器 | 0.5 天 |
|
||||
| 协作协议 | 1 天 |
|
||||
| 定时调度器 | 0.5 天 |
|
||||
| 单元测试 | 0.5 天 |
|
||||
| **总计** | **4 天** |
|
||||
|
||||
---
|
||||
|
||||
## 8. 验收标准
|
||||
|
||||
- [ ] Python 运行时可正常执行工具
|
||||
- [ ] JS 运行时可通过 stdio 调用
|
||||
- [ ] 原生运行时可执行二进制
|
||||
- [ ] 运行时管理器正确路由
|
||||
- [ ] 协作协议可正常请求/响应
|
||||
- [ ] 定时调度器可按计划执行任务
|
||||
- [ ] 单元测试通过
|
||||
Reference in New Issue
Block a user