2026-03-26 12:13:36 +08:00
|
|
|
|
# 记忆检索:异步 API 与 Celery 同步路径
|
|
|
|
|
|
|
|
|
|
|
|
## 两条路径
|
|
|
|
|
|
|
|
|
|
|
|
| 路径 | 入口 | 检索能力 |
|
|
|
|
|
|
|------|------|----------|
|
2026-04-03 11:43:16 +08:00
|
|
|
|
| **异步(HTTP / MemoirService)** | `MemoryService.retrieve` → `HybridRetriever` → `evidence.retrieve_evidence_bundle_async` | **向量(pgvector)** chunks;facts / timeline 按 **query ILIKE**,无命中则 **fallback** 最近条;rolling + ILIKE **摘要**;**stories**(标题/摘要匹配) |
|
|
|
|
|
|
| **同步(Celery)** | `retrieve_evidence_sync`(注入 `get_embedding_provider()` → `evidence.retrieve_evidence_bundle_sync`) | **向量** chunks + 同上元数据;与异步路径对齐 |
|
2026-03-27 16:01:28 +08:00
|
|
|
|
|
2026-04-03 11:43:16 +08:00
|
|
|
|
证据组装在 `app/features/memory/evidence.py`;`memory/repo` 提供原子查询(chunk 向量、facts/timeline 搜索、摘要列表等),story 合并在 evidence 层完成。
|
2026-03-26 12:13:36 +08:00
|
|
|
|
|
2026-04-03 11:43:16 +08:00
|
|
|
|
## 依赖 embedding
|
2026-03-26 12:13:36 +08:00
|
|
|
|
|
2026-04-03 11:43:16 +08:00
|
|
|
|
- 未配置 `ZHIPU_API_KEY`(或 provider `_client` 为空)时,chunk 检索为空列表,仍会返回 facts/timeline/summaries/stories(按 query ILIKE)。
|
|
|
|
|
|
- 日志:`HybridRetriever` / `retrieve_evidence_bundle_sync` 在无 provider 或空向量时会打 warning。
|
2026-03-26 12:13:36 +08:00
|
|
|
|
|
2026-03-27 16:01:28 +08:00
|
|
|
|
## 空 query
|
|
|
|
|
|
|
|
|
|
|
|
- 默认:`relevant_*` 均为空(与历史行为一致)。
|
2026-04-03 11:43:16 +08:00
|
|
|
|
- 若设置 `memory_evidence_empty_query_include_rolling=true`:返回**无 chunk**,但含 **rolling 摘要**、最近 facts / timeline(用于「浏览」模式)。
|
2026-03-27 16:01:28 +08:00
|
|
|
|
|
|
|
|
|
|
## 富化(ingest 后 LLM)
|
|
|
|
|
|
|
feat(eval): memoir A/B chapter judging and eval-web parity with dialogue
- Judge baseline excerpt and library chapter separately; build_memoir_compare_summary for gate, nine-dim and leaf deltas.
- Memoir SSE chapter payload: baseline_judge, compare_summary, baseline_judge_error.
- MemoirJudgeOutput: loose score coercion and post-validate clamp; memoir judge prompt caps from settings.
- app-eval-web: two-column MemoirScoreCard layout, MemoirCompareSummary, chapter blocks and CSS.
- Add memoir_compare_summary, log_events, celery_log_context, memoir_pipeline_progress; tests and migration 0014.
- Misc: memory/evidence and enrichment paths, task/orchestrator updates, internal-eval docs, env examples.
2026-04-10 10:23:43 +08:00
|
|
|
|
- `memory_enrichment_enabled`(默认 `true`):ingest 成功并 **commit** 后,通过 `schedule_memory_enrichment` 将任务投递到 **`CELERY_MEMORY_ENRICHMENT_QUEUE`**(默认 `memory_idle`),在 worker 上 **单次 LLM 调用**产出 **会话摘要(`MemorySummary` session)+ 结构化事实(`MemoryFact`)**;`false` 时不投递。
|
|
|
|
|
|
- ingest 路径 **不再**维护滚动摘要(rolling)与 **时间线表**(`timeline_events`)的物化;检索中的 `timeline_hints` 依赖既有数据(若有)或为空;空 query 下「浏览」模式若开启 `memory_evidence_empty_query_include_rolling`,仅当库内仍有历史 rolling 行时才会出现。
|
|
|
|
|
|
- 异步 `MemoryService.ingest` 与同步 `ingest_transcript_sync` 均 **不**在请求/任务热路径内内联 LLM 富化;回忆录 Phase1 在 DB commit 后调用 `schedule_enrichment_for_sources`(与 `memoir_correlation_id` 观测一致)。
|
|
|
|
|
|
- Worker 须消费该队列(例如 `-Q celery,memory_idle`),否则任务会堆积。
|
2026-03-27 16:01:28 +08:00
|
|
|
|
- `memory_enrichment_max_chars`:截断送入 LLM 的文本长度。
|
2026-04-03 11:43:16 +08:00
|
|
|
|
- Ingest 写入 **embedding**(best-effort);历史 FTS 列 `content_tsv` 已由迁移 `0007_drop_chunk_content_tsv` 删除。
|
feat(eval): memoir A/B chapter judging and eval-web parity with dialogue
- Judge baseline excerpt and library chapter separately; build_memoir_compare_summary for gate, nine-dim and leaf deltas.
- Memoir SSE chapter payload: baseline_judge, compare_summary, baseline_judge_error.
- MemoirJudgeOutput: loose score coercion and post-validate clamp; memoir judge prompt caps from settings.
- app-eval-web: two-column MemoirScoreCard layout, MemoirCompareSummary, chapter blocks and CSS.
- Add memoir_compare_summary, log_events, celery_log_context, memoir_pipeline_progress; tests and migration 0014.
- Misc: memory/evidence and enrichment paths, task/orchestrator updates, internal-eval docs, env examples.
2026-04-10 10:23:43 +08:00
|
|
|
|
- 叙事阶段 `retrieve_evidence_sync` **不等待**富化完成;证据随富化渐进变丰富。
|
2026-03-27 16:01:28 +08:00
|
|
|
|
|
2026-03-26 12:13:36 +08:00
|
|
|
|
## Celery 任务中的顺序
|
|
|
|
|
|
|
feat(eval): memoir A/B chapter judging and eval-web parity with dialogue
- Judge baseline excerpt and library chapter separately; build_memoir_compare_summary for gate, nine-dim and leaf deltas.
- Memoir SSE chapter payload: baseline_judge, compare_summary, baseline_judge_error.
- MemoirJudgeOutput: loose score coercion and post-validate clamp; memoir judge prompt caps from settings.
- app-eval-web: two-column MemoirScoreCard layout, MemoirCompareSummary, chapter blocks and CSS.
- Add memoir_compare_summary, log_events, celery_log_context, memoir_pipeline_progress; tests and migration 0014.
- Misc: memory/evidence and enrichment paths, task/orchestrator updates, internal-eval docs, env examples.
2026-04-10 10:23:43 +08:00
|
|
|
|
`process_memoir_segments`(`app/tasks/memoir_tasks.py`)在**同一任务**内先执行批量 ingest(`ingest_transcripts_batch_sync` 并 `commit`),再富化入队与 `MemoirOrchestrator`、派发 Phase2。Phase2 内 `retrieve_evidence_sync` 能看到**本批刚写入**的 memory chunks(无竞态),前提是 embedding API 已成功写入向量;富化 Summary/Facts 可能稍后才就绪。
|
2026-03-26 12:13:36 +08:00
|
|
|
|
|
2026-03-27 16:01:28 +08:00
|
|
|
|
章节分类上,若模型返回 **none** 或命中零散档案启发式,Story 侧会统一落入 **`summary` 章节**并继续叙事落库,与「本批 transcript 已进 memory」一致,避免误以为内容被丢弃。
|
|
|
|
|
|
|
2026-03-26 12:13:36 +08:00
|
|
|
|
## Evidence 与叙事 Prompt
|
|
|
|
|
|
|
2026-03-27 16:01:28 +08:00
|
|
|
|
`format_evidence_chunks_for_prompt` 拼接 chunks、**摘要(若有)**、facts、timeline、**故事摘要(若有)**;模型应把摘录视为参考材料,非本段口述。
|