feat(api): 拆分章节物化与 Story 后处理,并加固 Redis 锁与腾讯 ASR

回忆录 Story 流水线(同步)
- 同步路径仅写入 Story 与章节关联,改为 mark_chapter_dirty_sync,不再内联 compose
- 物化由 Celery recompose_chapter 异步完成;compose 不变量与异常时保留 dirty 的语义在 repo 中补充说明
- Evidence:大批次时降低 top_k;路由候选 story 携带 char_count/version_count;append 超长/版本过多时强制新开 story
- 叙事 prompt:relevant_chunks 去重,减少重复证据噪声
- 叙事回退与忠实度 gate:返回 fallback 类型并记录结构化日志(含耗时、JSON 有效性等)

Post-commit 与任务编排
- 新增 post_commit.enqueue_story_post_commit_effects:统一派发 generate_story_image(Redis 去重)、延迟 recompose_chapter、可选 memory compaction
- memoir_tasks / story_service / story_image_tasks 改为调用 post-commit 入口;主图回填后按关联章节重算并调度物化与 compacs(锁委托、Redis 单例、ASR to_thread)
- 更新 test_narrative_pipeline 以适配 _apply_narrative_fallbacks 返回值
This commit is contained in:
Kevin
2026-03-30 11:53:04 +08:00
parent e884409410
commit aac484463d
15 changed files with 775 additions and 144 deletions

View File

@@ -673,18 +673,58 @@ def format_narrative_user_content(oral_text: str, evidence_text: str = "") -> st
)
def _normalize_evidence_line(s: str) -> str:
return re.sub(r"\s+", " ", (s or "").strip().lower())
def dedupe_evidence_chunk_rows(chunks: list) -> list:
"""
对 relevant_chunks 做稳定去重:按归一化后长度降序 + 原下标,单遍包含判定;
复杂度 O(n log n);输出按原顺序中保留条目的相对顺序稳定。
"""
extracted: list[tuple[int, str, object]] = []
for i, c in enumerate(chunks):
content = (
c.get("content", "") if isinstance(c, dict) else getattr(c, "content", "")
)
t = (content or "").strip()
if not t:
continue
extracted.append((i, t, c))
if len(extracted) <= 1:
return [x[2] for x in extracted]
extracted.sort(
key=lambda x: (-len(_normalize_evidence_line(x[1])), x[0]),
)
kept_norms: list[str] = []
kept: list[tuple[int, object]] = []
for orig_idx, text, c in extracted:
n = _normalize_evidence_line(text)
dup = False
for kn in kept_norms:
if len(n) <= len(kn) and n in kn:
dup = True
break
if not dup:
kept_norms.append(n)
kept.append((orig_idx, c))
kept.sort(key=lambda x: x[0])
return [x[1] for x in kept]
def format_evidence_chunks_for_prompt(evidence: dict) -> str:
"""将 retrieve_evidence / retrieve_evidence_sync 结果格式化为简短文本,供叙事 prompt 使用。
包含 chunks、摘要若有、confirmed facts、timeline、故事摘要若有
"""
chunks = evidence.get("relevant_chunks") or []
chunks = dedupe_evidence_chunk_rows(chunks[:10])
summaries = evidence.get("relevant_summaries") or []
facts = evidence.get("relevant_facts") or []
timeline = evidence.get("timeline_hints") or []
stories = evidence.get("relevant_stories") or []
parts: list[str] = []
for c in chunks[:10]:
for c in chunks:
content = (
c.get("content", "") if isinstance(c, dict) else getattr(c, "content", "")
)