diff --git a/.github/workflows/README.md b/.github/workflows/README.md index d2076ff..7cdf159 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -1,325 +1,18 @@ -# GitHub Actions CI/CD 工作流配置说明 +# GitHub Actions 说明 -## 工作流列表 +## API：Docker Build and Deploy -| 工作流 | 说明 | 文档 | -|--------|------|------| -| Docker Build and Deploy | API 镜像构建与部署 | 见下文 | -| Android Release Build | Android APK 构建与发布 | 见下文 | -| App Expo Deploy | app-expo Web 构建与发布 | [docs/app-expo-deploy.md](../../docs/app-expo-deploy.md) | +- **工作流文件**： [docker-build-deploy.yml](docker-build-deploy.yml) +- **测试 job**：在构建镜像前于 `api/` 下执行 `uv sync --dev` 与 `pytest`。 +- **Secrets**：预发 `STAGING_*`、生产 `PROD_*`、镜像 `ALIYUN_CR_*` — 详见 [SETUP.md](SETUP.md)。 +- **分支 / Tag**：`main` → 预发；语义化 tag `v*.*.*` → 生产；路径过滤为 `api/**` 与本 workflow。 ---- +头部注释与 `docker-build-deploy.yml` 内说明为最新权威描述。 -## 概述 +## Android Release -本工作流实现了自动化的 Docker 镜像构建和部署流程： +构建签名 APK：见本文件历史版本中 Android 小节或仓库内 app 模块文档（若已迁移）。 -1. **构建阶段**：当代码推送到指定分支时，自动构建 Docker 镜像 -2. **推送阶段**：将构建好的镜像推送到阿里云容器镜像服务（ACR） -3. **部署阶段**：通过 SSH 连接到远程服务器，拉取最新镜像并重建容器 +## App Expo Deploy -## 工作流触发条件 - -- 推送到 `main`、`master` 或 `develop` 分支 -- 仅当 `api/` 目录或 `.github/workflows/` 目录有变更时触发 -- 支持手动触发（workflow_dispatch） - -## 配置步骤 - -### 1. 配置 GitHub Secrets - -在 GitHub 仓库设置中添加以下 Secrets： - -#### 必需配置 - -- **SSH_PRIVATE_KEY**: SSH 私钥，用于连接到远程服务器 - ```bash - # 生成 SSH 密钥对（如果还没有） - ssh-keygen -t ed25519 -C "github-actions" - - # 将私钥添加到 GitHub Secrets - cat ~/.ssh/id_ed25519 # 复制内容到 SSH_PRIVATE_KEY - - # 将公钥添加到远程服务器的 ~/.ssh/authorized_keys - cat ~/.ssh/id_ed25519.pub | ssh user@your-server "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys" - ``` - -- **SSH_USER**: SSH 用户名（例如：`root`、`ubuntu`、`deploy`） -- **SSH_HOST**: 远程服务器 IP 地址或域名（例如：`192.168.1.100`、`example.com`） - -#### 阿里云容器镜像服务配置（必需） - -- **ALIYUN_CR_USERNAME**: 阿里云容器镜像服务用户名（例如：`zaikunxu`） -- **ALIYUN_CR_PASSWORD**: 阿里云容器镜像服务密码 - -#### 可选配置 - -- **SSH_PORT**: SSH 端口（默认：`22`） -- **DEPLOY_PATH**: 远程服务器上的部署目录（默认：`/opt/life-echo`） - -### 2. 容器镜像仓库配置 - -本工作流使用**阿里云容器镜像服务（ACR）**作为镜像仓库。 - -#### 镜像地址 - -- 镜像仓库地址：`crpi-u2903xccyzd6nqnc.cn-shanghai.personal.cr.aliyuncs.com` -- 命名空间：`huaga` -- 镜像名称：`lifecho-api` -- 完整镜像路径：`crpi-u2903xccyzd6nqnc.cn-shanghai.personal.cr.aliyuncs.com/huaga/lifecho-api:latest` - -#### 配置说明 - -工作流已配置为使用阿里云容器镜像服务，无需修改工作流文件。只需在 GitHub Secrets 中配置： - -- `ALIYUN_CR_USERNAME`: 阿里云容器镜像服务用户名 -- `ALIYUN_CR_PASSWORD`: 阿里云容器镜像服务密码 - -#### 在远程服务器上配置 Docker 登录 - -确保远程服务器可以拉取私有镜像，需要在远程服务器上执行： - -```bash -docker login crpi-u2903xccyzd6nqnc.cn-shanghai.personal.cr.aliyuncs.com \ - --username=zaikunxu \ - --password=57ucV,g4LF2cqm8 -``` - -或者将登录信息添加到工作流的部署步骤中（已自动配置）。 - -### 3. 远程服务器准备 - -确保远程服务器已安装： - -- Docker -- Docker Compose - -```bash -# 安装 Docker（Ubuntu/Debian） -curl -fsSL https://get.docker.com -o get-docker.sh -sudo sh get-docker.sh - -# Docker Compose V2（命令为 docker compose，带空格） -# 多数 Docker 安装已包含插件；若提示找不到 compose： -# sudo apt-get update && sudo apt-get install -y docker-compose-plugin -``` - -### 4. 首次部署准备 - -在远程服务器上创建部署目录： - -```bash -mkdir -p /opt/life-echo/api -``` - -确保 SSH 用户可以访问该目录并执行 Docker 命令。 - -## 工作流执行流程 - -### 构建阶段（build-and-push） - -1. 检出代码 -2. 设置 Docker Buildx -3. 登录到容器镜像仓库 -4. 提取元数据（标签、标签） -5. 构建并推送 Docker 镜像 - -### 部署阶段（deploy） - -1. 设置 SSH 连接 -2. 登录到容器镜像仓库（在远程服务器上） -3. 拉取最新镜像 -4. 停止现有容器 -5. 更新 docker-compose.yml 中的镜像标签 -6. 启动新容器 -7. 清理旧镜像 -8. 验证部署状态 - -## 镜像标签规则 - -- `main`/`master` 分支 → `latest` -- 其他分支 → 使用分支名作为标签 -- 同时会生成基于 commit SHA 的标签 - -## 故障排查 - -### 1. SSH 连接失败 - -- 检查 `SSH_PRIVATE_KEY` 是否正确配置 -- 确认远程服务器的 SSH 服务正在运行 -- 验证 SSH 公钥已添加到远程服务器的 `~/.ssh/authorized_keys` - -### 2. Docker 登录失败 - -- 检查 `ALIYUN_CR_USERNAME` 和 `ALIYUN_CR_PASSWORD` 是否正确配置 -- 确认阿里云容器镜像服务的账号密码是否正确 -- 检查远程服务器是否可以访问阿里云容器镜像服务 - -### 3. 镜像拉取失败 - -- 确认远程服务器可以访问容器镜像仓库 -- 检查镜像标签是否正确 -- 验证网络连接 - -### 4. 容器启动失败 - -- 检查 docker-compose.yml 文件是否正确 -- 查看容器日志：`docker compose logs` -- 确认环境变量配置正确 - -## 手动触发 - -可以通过 GitHub Actions 界面手动触发工作流： - -1. 进入仓库的 Actions 标签页 -2. 选择 "Docker Build and Deploy" 工作流 -3. 点击 "Run workflow" 按钮 - -## 自定义配置 - -### 修改触发分支 - -编辑 `.github/workflows/docker-build-deploy.yml`： - -```yaml -on: - push: - branches: - - your-branch-name -``` - -### 修改镜像名称 - -编辑工作流文件中的 `IMAGE_NAME` 和 `REGISTRY_NAMESPACE` 环境变量： - -```yaml -env: - IMAGE_NAME: your-image-name - REGISTRY_NAMESPACE: your-namespace -``` - -### 修改部署路径 - -在 GitHub Secrets 中设置 `DEPLOY_PATH`，或直接修改工作流文件中的默认值。 - -## 注意事项 - -1. **安全性**：确保 SSH 私钥和密码等敏感信息只存储在 GitHub Secrets 中，不要提交到代码仓库 -2. **权限**：确保 SSH 用户在远程服务器上有执行 Docker 命令的权限 -3. **备份**：部署前会自动备份 docker-compose.yml 文件（`.bak` 后缀） -4. **回滚**：如果部署失败，可以使用备份文件恢复 - -## 相关文件 - -- 工作流文件：`.github/workflows/docker-build-deploy.yml` -- Dockerfile：`api/Dockerfile` -- Docker Compose：`api/docker-compose.yml` - ---- - -# Android Release 工作流 - -## 概述 - -自动构建生产环境签名 APK 并发布到 GitHub Releases： - -1. **构建阶段**：编译 Release APK，使用生产签名配置 -2. **发布阶段**：创建 GitHub Release，将 APK 作为附件上传 - -## 工作流触发条件 - -- **Tag 推送**：推送 `v*` 格式的标签时自动触发（如 `v1.0.0`、`v1.2.3-beta`） -- **手动触发**：通过 GitHub Actions 界面手动触发，可选指定版本号 - -## 配置步骤 - -### 1. 配置 GitHub Secrets - -在 GitHub 仓库设置中添加以下 Secrets： - -| Secret | 说明 | 示例 | -|--------|------|------| -| `ANDROID_KEYSTORE_BASE64` | keystore 文件的 Base64 编码 | 见下方生成方法 | -| `ANDROID_KEY_ALIAS` | 密钥别名 | `suiyueshishu` | -| `ANDROID_KEY_PASSWORD` | 密钥密码 | | -| `ANDROID_STORE_PASSWORD` | keystore 密码 | | - -#### 生成 ANDROID_KEYSTORE_BASE64 - -```bash -# 在本地项目 app-android 目录下执行 -base64 -i release-keystore.jks | pbcopy # macOS，结果已复制到剪贴板 - -# Linux -base64 -w 0 release-keystore.jks -``` - -将输出内容粘贴到 GitHub Secrets 的 `ANDROID_KEYSTORE_BASE64` 中。 - -### 2. 版本号管理 - -- **Tag 触发**：从 tag 名自动提取版本号（如 `v1.0.0` → `versionName = "1.0.0"`） -- **手动触发**：使用输入的 `version_name`，或使用 `build.gradle.kts` 中的默认值 -- **versionCode**：自动使用 `GITHUB_RUN_NUMBER` 递增 - -### 3. 发布流程 - -#### 方式一：通过 Tag 自动发布 - -```bash -# 打标签并推送 -git tag v1.0.0 -git push origin v1.0.0 - -# 或者一步完成 -git tag v1.0.0 && git push origin v1.0.0 -``` - -工作流将自动： -1. 构建签名 Release APK -2. 上传 APK 为 GitHub Artifact -3. 创建 GitHub Release（附带 APK 和自动生成的更新日志） - -#### 方式二：手动触发 - -1. 进入仓库的 Actions 标签页 -2. 选择 "Android Release Build" 工作流 -3. 点击 "Run workflow" 按钮 -4. （可选）填写版本号 -5. 点击 "Run workflow" - -手动触发时仅构建 APK 并上传为 Artifact，不会创建 GitHub Release。 - -## 工作流执行流程 - -1. 检出代码（完整历史） -2. 确定版本号（Tag / 手动输入 / build.gradle.kts 默认值） -3. 设置 JDK 17 + Gradle 缓存 -4. 解码签名 keystore + 生成 keystore.properties -5. 覆盖 build.gradle.kts 中的版本号 -6. 执行 `./gradlew assembleRelease` -7. 上传 APK 为 Artifact（保留 30 天） -8. （Tag 触发时）生成 Release Notes 并创建 GitHub Release - -## APK 命名规则 - -APK 文件命名格式：`岁月时书_v{版本号}_release.apk` - -示例：`岁月时书_v1.0.0_release.apk` - -## 故障排查 - -### 1. 签名失败 - -- 检查 `ANDROID_KEYSTORE_BASE64` 是否正确生成（不能有换行或空格） -- 确认 `ANDROID_KEY_ALIAS`、`ANDROID_KEY_PASSWORD`、`ANDROID_STORE_PASSWORD` 与 keystore 匹配 - -### 2. 构建失败 - -- 检查 Actions 日志中的 Gradle 错误输出 -- 确认本地 `./gradlew assembleRelease` 可以成功 - -### 3. Release 创建失败 - -- 确认工作流有 `contents: write` 权限 -- 检查 Tag 名称是否以 `v` 开头 +见仓库 `docs/` 下相关说明（若存在）。 diff --git a/.github/workflows/SETUP.md b/.github/workflows/SETUP.md index 09ac141..e61a9dd 100644 --- a/.github/workflows/SETUP.md +++ b/.github/workflows/SETUP.md @@ -1,83 +1,44 @@ -# 快速配置指南 +# Docker API 部署 — Secrets 快速清单 -## GitHub Secrets 配置清单 +与 [docker-build-deploy.yml](docker-build-deploy.yml) 保持一致。在 **Settings → Secrets and variables → Actions** 中配置（仓库级 Secrets）。 -在 GitHub 仓库的 **Settings → Secrets and variables → Actions** 中添加以下 Secrets： +## 必需 -### 必需配置 +| Secret | 说明 | +|--------|------| +| `STAGING_SSH_PRIVATE_KEY` | 预发机 SSH 私钥全文 | +| `STAGING_SSH_HOST` | 预发机主机名或 IP | +| `STAGING_SSH_USER` | SSH 用户名 | +| `STAGING_SSH_PORT` | SSH 端口（默认 `22`） | +| `STAGING_DEPLOY_PATH` | 预发机上的部署目录 | +| `PROD_SSH_PRIVATE_KEY` | 生产机 SSH 私钥（可与预发不同） | +| `PROD_SSH_HOST` | 生产机主机 | +| `PROD_SSH_USER` | 生产 SSH 用户 | +| `PROD_SSH_PORT` | 生产 SSH 端口 | +| `PROD_DEPLOY_PATH` | 生产部署目录 | +| `ALIYUN_CR_USERNAME` | 阿里云 ACR 用户名 | +| `ALIYUN_CR_PASSWORD` | 阿里云 ACR 密码 | -1. **SSH_PRIVATE_KEY** - - 描述：SSH 私钥，用于连接到远程服务器 - - 生成方法： - ```bash - ssh-keygen -t ed25519 -C "github-actions" - cat ~/.ssh/id_ed25519 # 复制全部内容 - ``` - - 将公钥添加到远程服务器： - ```bash - cat ~/.ssh/id_ed25519.pub | ssh user@your-server "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys" - ``` +> **Tag 部署**：推送 `v*.*.*`（如 `v1.2.0`）时使用 `PROD_*`。**main 分支推送**使用 `STAGING_*`。 -2. **SSH_USER** - - 描述：SSH 用户名 - - 示例：`root`、`ubuntu`、`deploy` +## 触发条件 -3. **SSH_HOST** - - 描述：远程服务器 IP 或域名 - - 示例：`192.168.1.100`、`example.com` +- `push` 到 `main`：改动了 `api/**` 或 `.github/workflows/**` 时，先跑 **API tests**（`uv sync --dev` + `pytest`），再构建镜像并部署预发。 +- `push` tag `v*.*.*`：同上路径过滤；部署生产。 +- **workflow_dispatch**：可选手动指定 ref。 -4. **ALIYUN_CR_USERNAME** - - 描述：阿里云容器镜像服务用户名 - - 值：`zaikunxu` +仓库内需存在 **`api/.env.staging`** / **`api/.env.production`**（供部署 job 校验与上传）；勿将真实密钥提交到公开分支。 -5. **ALIYUN_CR_PASSWORD** - - 描述：阿里云容器镜像服务密码 - - 值：`57ucV,g4LF2cqm8` +## 本地验证 SSH -### 可选配置 - -6. **SSH_PORT** - - 描述：SSH 端口（默认：22） - - 示例：`22`、`2222` - -7. **DEPLOY_PATH** - - 描述：远程服务器部署目录（默认：/opt/life-echo） - - 示例：`/opt/life-echo`、`/home/deploy/life-echo` - -## 镜像信息 - -- **镜像仓库地址**：`crpi-u2903xccyzd6nqnc.cn-shanghai.personal.cr.aliyuncs.com` -- **命名空间**：`huaga` -- **镜像名称**：`lifecho-api` -- **完整镜像路径**：`crpi-u2903xccyzd6nqnc.cn-shanghai.personal.cr.aliyuncs.com/huaga/lifecho-api:latest` - -## 验证配置 - -配置完成后，推送代码到 `main`、`master` 或 `develop` 分支，工作流会自动触发。 - -或者手动触发： -1. 进入仓库的 **Actions** 标签页 -2. 选择 **Docker Build and Deploy** 工作流 -3. 点击 **Run workflow** 按钮 - -## 常见问题 - -### 1. 如何查看工作流执行日志？ -- 进入 **Actions** 标签页，点击对应的运行记录查看详细日志 - -### 2. 如何测试 SSH 连接？ ```bash -ssh -i ~/.ssh/id_ed25519 user@your-server +ssh -i ~/.ssh/your_key -p 22 user@your-host ``` -### 3. 如何测试 Docker 登录？ +## 本地验证 registry 登录 + ```bash -docker login crpi-u2903xccyzd6nqnc.cn-shanghai.personal.cr.aliyuncs.com \ - --username=zaikunxu \ - --password=57ucV,g4LF2cqm8 +docker login --username= --password-stdin ``` -### 4. 部署失败怎么办？ -- 检查 GitHub Actions 日志中的错误信息 -- 确认远程服务器可以访问阿里云容器镜像服务 -- 验证 SSH 连接和 Docker 权限 +（密码从控制台或密钥管理读取，勿写入文档。） diff --git a/.github/workflows/docker-build-deploy.yml b/.github/workflows/docker-build-deploy.yml index a9c2d3d..8023d10 100644 --- a/.github/workflows/docker-build-deploy.yml +++ b/.github/workflows/docker-build-deploy.yml @@ -46,8 +46,28 @@ env: FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true jobs: + test: + name: API tests + runs-on: ubuntu-latest + permissions: + contents: read + steps: + - uses: actions/checkout@v5 + + - name: Install uv + uses: astral-sh/setup-uv@v5 + with: + version: "0.7.3" + + - name: Sync deps and run pytest + working-directory: api + run: | + uv sync --dev + uv run pytest --tb=short -q + build-and-push: name: Build and Push Docker Image + needs: test runs-on: ubuntu-latest permissions: contents: read diff --git a/README.md b/README.md index 8ba2dd4..36e435e 100644 --- a/README.md +++ b/README.md @@ -21,17 +21,23 @@ ``` life-echo/ -├── api/ # 后端服务（FastAPI） -│ ├── agents/ # LangChain Agent（对话引导、记忆整理） -│ ├── database/ # 数据库模型和连接（PostgreSQL） -│ ├── routers/ # API 路由（REST + WebSocket） -│ ├── services/ # 业务服务（ASR、TTS、PDF、LLM） -│ ├── tasks/ # Celery 后台任务 -│ ├── docs/ # API 详细文档 -│ ├── main.py # FastAPI 应用入口 -│ └── README.md # API 服务文档 -│ -└── docs/ # 项目文档 +├── api/ # 后端（FastAPI，uv + Alembic） +│ ├── app/ +│ │ ├── main.py # 对外 API 入口 +│ │ ├── internal_main.py # 内部评测 API 入口 +│ │ ├── core/ # 配置、DB、日志、Redis、LLM 调用等 +│ │ ├── ports/ # 供应商能力协议（ASR/TTS/LLM/…） +│ │ ├── adapters/ # ports 的具体实现（腾讯、OpenAI、Whisper 等） +│ │ ├── agents/ # 对话与回忆录编排 Agent +│ │ ├── features/ # 按域拆分：auth、conversation、memoir、memory、evaluation… +│ │ └── tasks/ # Celery 任务入口 +│ ├── alembic/ # 数据库迁移 +│ ├── tests/ # pytest +│ ├── main.py # uvicorn 兼容入口 +│ └── README.md +├── app-expo/ # 移动端（Expo Router + React Native） +├── app-eval-web/ # 内部评测 Web（Vite） +└── docs/ # 设计与运维文档 ``` ## 🚀 快速开始 diff --git a/api/alembic/versions/0009_chapter_evidence_snapshot_and_message_memory_trace.py b/api/alembic/versions/0009_chapter_evidence_snapshot_and_message_memory_trace.py new file mode 100644 index 0000000..277581a --- /dev/null +++ b/api/alembic/versions/0009_chapter_evidence_snapshot_and_message_memory_trace.py @@ -0,0 +1,51 @@ +"""Chapter evidence_bundle_json + conversation_messages memory trace + +Revision ID: 0009_ce_bundle_mem_trace (short id: alembic_version.version_num is VARCHAR(32)) +Revises: 0008_eval_regression_platform +""" + +from typing import Sequence, Union + +import sqlalchemy as sa +from sqlalchemy.dialects import postgresql + +from alembic import op + +revision: str = "0009_ce_bundle_mem_trace" +down_revision: Union[str, None] = "0008_eval_regression_platform" +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def _has_column(table: str, column: str) -> bool: + bind = op.get_bind() + ins = sa.inspect(bind) + return any(c["name"] == column for c in ins.get_columns(table)) + + +def upgrade() -> None: + if not _has_column("chapters", "evidence_bundle_json"): + op.add_column( + "chapters", + sa.Column( + "evidence_bundle_json", + postgresql.JSON(astext_type=sa.Text()), + nullable=True, + ), + ) + if not _has_column("conversation_messages", "memory_retrieval_trace_json"): + op.add_column( + "conversation_messages", + sa.Column( + "memory_retrieval_trace_json", + postgresql.JSON(astext_type=sa.Text()), + nullable=True, + ), + ) + + +def downgrade() -> None: + if _has_column("conversation_messages", "memory_retrieval_trace_json"): + op.drop_column("conversation_messages", "memory_retrieval_trace_json") + if _has_column("chapters", "evidence_bundle_json"): + op.drop_column("chapters", "evidence_bundle_json") diff --git a/api/alembic/versions/0010_chapter_evidence_snapshots_tables.py b/api/alembic/versions/0010_chapter_evidence_snapshots_tables.py new file mode 100644 index 0000000..008fe4b --- /dev/null +++ b/api/alembic/versions/0010_chapter_evidence_snapshots_tables.py @@ -0,0 +1,140 @@ +"""Chapter evidence snapshots + normalized links (Phase C) + +Revision ID: 0010_ce_snapshots (short id: alembic_version.version_num is VARCHAR(32)) +Revises: 0009_ce_bundle_mem_trace +""" + +from typing import Sequence, Union + +import sqlalchemy as sa +from sqlalchemy.dialects import postgresql + +from alembic import op + +revision: str = "0010_ce_snapshots" +down_revision: Union[str, None] = "0009_ce_bundle_mem_trace" +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def _has_table(name: str) -> bool: + bind = op.get_bind() + return sa.inspect(bind).has_table(name) + + +def _has_column(table: str, column: str) -> bool: + bind = op.get_bind() + return any(c["name"] == column for c in sa.inspect(bind).get_columns(table)) + + +def upgrade() -> None: + if not _has_table("chapter_evidence_snapshots"): + op.create_table( + "chapter_evidence_snapshots", + sa.Column("id", sa.String(), nullable=False), + sa.Column("chapter_id", sa.String(), nullable=False), + sa.Column("user_id", sa.String(), nullable=False), + sa.Column("version_no", sa.Integer(), nullable=False), + sa.Column("schema_version", sa.Integer(), nullable=False, server_default="1"), + sa.Column("segment_ids", postgresql.JSON(astext_type=sa.Text()), nullable=True), + sa.Column("conversation_ids", postgresql.JSON(astext_type=sa.Text()), nullable=True), + sa.Column("story_ids", postgresql.JSON(astext_type=sa.Text()), nullable=True), + sa.Column("memory_chunk_ids", postgresql.JSON(astext_type=sa.Text()), nullable=True), + sa.Column("memory_fact_ids", postgresql.JSON(astext_type=sa.Text()), nullable=True), + sa.Column("timeline_event_ids", postgresql.JSON(astext_type=sa.Text()), nullable=True), + sa.Column("summary_ids", postgresql.JSON(astext_type=sa.Text()), nullable=True), + sa.Column("notes", postgresql.JSON(astext_type=sa.Text()), nullable=True), + sa.Column( + "captured_at", + sa.DateTime(timezone=True), + nullable=False, + server_default=sa.text("now()"), + ), + sa.ForeignKeyConstraint( + ["chapter_id"], + ["chapters.id"], + name="chapter_evidence_snapshots_chapter_id_fkey", + ondelete="CASCADE", + ), + sa.ForeignKeyConstraint( + ["user_id"], + ["users.id"], + name="chapter_evidence_snapshots_user_id_fkey", + ), + sa.PrimaryKeyConstraint("id", name="chapter_evidence_snapshots_pkey"), + sa.UniqueConstraint( + "chapter_id", + "version_no", + name="uq_chapter_evidence_snapshots_chapter_version", + ), + ) + op.create_index( + "ix_chapter_evidence_snapshots_chapter_id", + "chapter_evidence_snapshots", + ["chapter_id"], + ) + op.create_index( + "ix_chapter_evidence_snapshots_user_id", + "chapter_evidence_snapshots", + ["user_id"], + ) + + if not _has_table("chapter_evidence_links"): + op.create_table( + "chapter_evidence_links", + sa.Column("id", sa.String(), nullable=False), + sa.Column("chapter_id", sa.String(), nullable=False), + sa.Column("evidence_type", sa.String(), nullable=False), + sa.Column("evidence_id", sa.String(), nullable=False), + sa.Column("role", sa.String(), nullable=True), + sa.Column("weight", sa.Float(), nullable=True), + sa.Column( + "created_at", + sa.DateTime(timezone=True), + server_default=sa.text("now()"), + nullable=False, + ), + sa.ForeignKeyConstraint( + ["chapter_id"], + ["chapters.id"], + name="chapter_evidence_links_chapter_id_fkey", + ondelete="CASCADE", + ), + sa.PrimaryKeyConstraint("id", name="chapter_evidence_links_pkey"), + ) + op.create_index( + "ix_chapter_evidence_links_chapter_id", + "chapter_evidence_links", + ["chapter_id"], + ) + + if not _has_column("chapters", "current_evidence_snapshot_id"): + op.add_column( + "chapters", + sa.Column("current_evidence_snapshot_id", sa.String(), nullable=True), + ) + op.create_foreign_key( + "fk_chapters_current_evidence_snapshot_id", + "chapters", + "chapter_evidence_snapshots", + ["current_evidence_snapshot_id"], + ["id"], + ondelete="SET NULL", + ) + + +def downgrade() -> None: + if _has_column("chapters", "current_evidence_snapshot_id"): + op.drop_constraint( + "fk_chapters_current_evidence_snapshot_id", + "chapters", + type_="foreignkey", + ) + op.drop_column("chapters", "current_evidence_snapshot_id") + if _has_table("chapter_evidence_links"): + op.drop_index("ix_chapter_evidence_links_chapter_id", table_name="chapter_evidence_links") + op.drop_table("chapter_evidence_links") + if _has_table("chapter_evidence_snapshots"): + op.drop_index("ix_chapter_evidence_snapshots_user_id", table_name="chapter_evidence_snapshots") + op.drop_index("ix_chapter_evidence_snapshots_chapter_id", table_name="chapter_evidence_snapshots") + op.drop_table("chapter_evidence_snapshots") diff --git a/api/alembic/versions/0011_dialogue_lineage_message_provenance.py b/api/alembic/versions/0011_dialogue_lineage_message_provenance.py new file mode 100644 index 0000000..c4f891e --- /dev/null +++ b/api/alembic/versions/0011_dialogue_lineage_message_provenance.py @@ -0,0 +1,122 @@ +"""Dialogue lineage: segment/memory/story/chapter evidence message-grade provenance. + +Revision ID: 0011_dialogue_lineage +Revises: 0010_ce_snapshots +""" + +from typing import Sequence, Union + +import sqlalchemy as sa +from sqlalchemy.dialects import postgresql + +from alembic import op + +revision: str = "0011_dialogue_lineage" +down_revision: Union[str, None] = "0010_ce_snapshots" +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def _has_column(table: str, column: str) -> bool: + bind = op.get_bind() + return any(c["name"] == column for c in sa.inspect(bind).get_columns(table)) + + +def upgrade() -> None: + if not _has_column("segments", "user_message_id"): + op.add_column( + "segments", + sa.Column( + "user_message_id", + sa.String(), + sa.ForeignKey("conversation_messages.id", ondelete="SET NULL"), + nullable=True, + ), + ) + op.create_index( + "ix_segments_user_message_id", + "segments", + ["user_message_id"], + unique=False, + ) + if not _has_column("segments", "lineage_json"): + op.add_column( + "segments", + sa.Column( + "lineage_json", + postgresql.JSON(astext_type=sa.Text()), + nullable=True, + ), + ) + + if not _has_column("memory_sources", "lineage_json"): + op.add_column( + "memory_sources", + sa.Column( + "lineage_json", + postgresql.JSON(astext_type=sa.Text()), + nullable=True, + ), + ) + if not _has_column("memory_sources", "primary_user_message_id"): + op.add_column( + "memory_sources", + sa.Column("primary_user_message_id", sa.String(), nullable=True), + ) + op.create_index( + "ix_memory_sources_primary_user_message_id", + "memory_sources", + ["primary_user_message_id"], + unique=False, + ) + + if not _has_column("story_versions", "lineage_json"): + op.add_column( + "story_versions", + sa.Column( + "lineage_json", + postgresql.JSON(astext_type=sa.Text()), + nullable=True, + ), + ) + + if not _has_column("chapter_evidence_snapshots", "message_lineage_json"): + op.add_column( + "chapter_evidence_snapshots", + sa.Column( + "message_lineage_json", + postgresql.JSON(astext_type=sa.Text()), + nullable=True, + ), + ) + + if not _has_column("chapters", "source_lineage_json"): + op.add_column( + "chapters", + sa.Column( + "source_lineage_json", + postgresql.JSON(astext_type=sa.Text()), + nullable=True, + ), + ) + + +def downgrade() -> None: + if _has_column("chapters", "source_lineage_json"): + op.drop_column("chapters", "source_lineage_json") + if _has_column("chapter_evidence_snapshots", "message_lineage_json"): + op.drop_column("chapter_evidence_snapshots", "message_lineage_json") + if _has_column("story_versions", "lineage_json"): + op.drop_column("story_versions", "lineage_json") + if _has_column("memory_sources", "primary_user_message_id"): + op.drop_index( + "ix_memory_sources_primary_user_message_id", table_name="memory_sources" + ) + op.drop_column("memory_sources", "primary_user_message_id") + if _has_column("memory_sources", "lineage_json"): + op.drop_column("memory_sources", "lineage_json") + if _has_column("segments", "lineage_json"): + op.drop_column("segments", "lineage_json") + if _has_column("segments", "user_message_id"): + op.drop_index("ix_segments_user_message_id", table_name="segments") + op.drop_column("segments", "user_message_id") diff --git a/api/alembic/versions/0012_memory_fact_timeline_lineage_json.py b/api/alembic/versions/0012_memory_fact_timeline_lineage_json.py new file mode 100644 index 0000000..32eda80 --- /dev/null +++ b/api/alembic/versions/0012_memory_fact_timeline_lineage_json.py @@ -0,0 +1,50 @@ +"""Lineage snapshot on memory_facts and timeline_events (enrichment denorm). + +Revision ID: 0012_mem_fact_tl_lineage +Revises: 0011_dialogue_lineage +""" + +from typing import Sequence, Union + +import sqlalchemy as sa +from sqlalchemy.dialects import postgresql + +from alembic import op + +revision: str = "0012_mem_fact_tl_lineage" +down_revision: Union[str, None] = "0011_dialogue_lineage" +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def _has_column(table: str, column: str) -> bool: + bind = op.get_bind() + return any(c["name"] == column for c in sa.inspect(bind).get_columns(table)) + + +def upgrade() -> None: + if not _has_column("memory_facts", "lineage_json"): + op.add_column( + "memory_facts", + sa.Column( + "lineage_json", + postgresql.JSON(astext_type=sa.Text()), + nullable=True, + ), + ) + if not _has_column("timeline_events", "lineage_json"): + op.add_column( + "timeline_events", + sa.Column( + "lineage_json", + postgresql.JSON(astext_type=sa.Text()), + nullable=True, + ), + ) + + +def downgrade() -> None: + if _has_column("timeline_events", "lineage_json"): + op.drop_column("timeline_events", "lineage_json") + if _has_column("memory_facts", "lineage_json"): + op.drop_column("memory_facts", "lineage_json") diff --git a/api/app/adapters/asr/tencent_asr.py b/api/app/adapters/asr/tencent_asr.py index 87d7335..0620910 100644 --- a/api/app/adapters/asr/tencent_asr.py +++ b/api/app/adapters/asr/tencent_asr.py @@ -2,7 +2,9 @@ import asyncio import base64 + from app.core.logging import get_logger +from app.ports.asr import ASRTranscriptionError logger = get_logger(__name__) @@ -39,7 +41,9 @@ class TencentASRProvider: async def transcribe(self, audio: bytes, format: str = "m4a") -> str: client = self._get_client() if not client: - return "转写失败: 腾讯云 ASR 客户端未初始化（请检查密钥与依赖）" + raise ASRTranscriptionError( + "Tencent ASR client not initialized (check credentials)" + ) try: from tencentcloud.asr.v20190614 import models @@ -64,10 +68,11 @@ class TencentASRProvider: req.VoiceFormat, err, ) - return ( - "转写失败: 腾讯云返回空文本（常见原因：采样率与 16k_zh 不匹配、" - "格式不受支持或音频无效；请确认客户端为 16kHz 单声道 m4a）" + raise ASRTranscriptionError( + "Tencent ASR empty Result (check sample rate / format / audio)" ) + except ASRTranscriptionError: + raise except Exception as e: logger.error("Tencent ASR transcribe failed: {}", e, exc_info=True) - return f"转写失败: {e}"[:500] + raise ASRTranscriptionError(f"Tencent ASR transcribe failed: {e!s}") from e diff --git a/api/app/adapters/asr/whisper_local.py b/api/app/adapters/asr/whisper_local.py index 846666b..a64a0a5 100644 --- a/api/app/adapters/asr/whisper_local.py +++ b/api/app/adapters/asr/whisper_local.py @@ -9,6 +9,7 @@ import tempfile from typing import Any, Iterable from app.core.logging import get_logger +from app.ports.asr import ASRTranscriptionError logger = get_logger(__name__) @@ -104,7 +105,7 @@ class WhisperASRProvider: # 与 v1.1.0 相同的单次 transcribe；推理放线程池，避免阻塞 asyncio（tag 上为同步调用）。 self._load_model() if not self._model: - return "" + raise ASRTranscriptionError("Whisper model not loaded") model = self._model @@ -182,9 +183,11 @@ class WhisperASRProvider: logger.warning("Whisper decode_audio 回退失败: {}", ex) return text + except ASRTranscriptionError: + raise except Exception as e: logger.error("Whisper transcribe failed: {}", e) - return "" + raise ASRTranscriptionError(f"Whisper transcribe failed: {e!s}") from e finally: if tmp_path and os.path.exists(tmp_path): try: diff --git a/api/app/adapters/image_gen/liblib_provider.py b/api/app/adapters/image_gen/liblib_provider.py index b13f335..b307713 100644 --- a/api/app/adapters/image_gen/liblib_provider.py +++ b/api/app/adapters/image_gen/liblib_provider.py @@ -6,8 +6,6 @@ Feature 通过 port ImageGenerator 使用，本模块仅被 app.adapters.image_g import base64 import hmac import logging - -from app.core.logging import get_logger import re import time import uuid @@ -17,6 +15,7 @@ from urllib.parse import urlparse import httpx from app.core.config import settings +from app.core.logging import get_logger logger = get_logger(__name__) diff --git a/api/app/adapters/sms/tencent.py b/api/app/adapters/sms/tencent.py index 0076e40..357988b 100644 --- a/api/app/adapters/sms/tencent.py +++ b/api/app/adapters/sms/tencent.py @@ -1,7 +1,5 @@ """Tencent Cloud SMS adapter — implements SmsSender port.""" -from app.core.logging import get_logger - from tencentcloud.common import credential from tencentcloud.common.exception.tencent_cloud_sdk_exception import ( TencentCloudSDKException, @@ -9,6 +7,8 @@ from tencentcloud.common.exception.tencent_cloud_sdk_exception import ( from tencentcloud.sms.v20210111 import models as sms_models from tencentcloud.sms.v20210111 import sms_client +from app.core.logging import get_logger + logger = get_logger(__name__) CODE_EXPIRE_MINUTES = 5 diff --git a/api/app/adapters/storage/tencent_cos.py b/api/app/adapters/storage/tencent_cos.py index 3613e46..8423ad4 100644 --- a/api/app/adapters/storage/tencent_cos.py +++ b/api/app/adapters/storage/tencent_cos.py @@ -1,10 +1,10 @@ """Tencent COS adapter — implements ObjectStorage port.""" -from app.core.logging import get_logger - from qcloud_cos import CosConfig, CosS3Client from qcloud_cos.cos_exception import CosClientError, CosServiceError +from app.core.logging import get_logger + logger = get_logger(__name__) diff --git a/api/app/adapters/tts/openai_tts.py b/api/app/adapters/tts/openai_tts.py index c114715..c62fc32 100644 --- a/api/app/adapters/tts/openai_tts.py +++ b/api/app/adapters/tts/openai_tts.py @@ -1,27 +1,38 @@ """OpenAI TTS adapter — implements TTSProvider port.""" +import asyncio from io import BytesIO from openai import OpenAI +from app.core.logging import get_logger + +logger = get_logger(__name__) + class OpenAITTSProvider: def __init__(self, api_key: str, model: str = "tts-1"): self._client = OpenAI(api_key=api_key) if api_key else None self._model = model + def _synthesize_sync(self, text: str, voice: str) -> bytes: + if not self._client: + return b"" + response = self._client.audio.speech.create( + model=self._model, + voice=voice, + input=text, + ) + buf = BytesIO() + for chunk in response.iter_bytes(): + buf.write(chunk) + return buf.getvalue() + async def synthesize(self, text: str, voice: str = "alloy") -> bytes: if not self._client: return b"" try: - response = self._client.audio.speech.create( - model=self._model, - voice=voice, - input=text, - ) - buf = BytesIO() - for chunk in response.iter_bytes(): - buf.write(chunk) - return buf.getvalue() - except Exception: + return await asyncio.to_thread(self._synthesize_sync, text, voice) + except Exception as e: + logger.warning("OpenAI TTS synthesize failed: {}", e) return b"" diff --git a/api/app/agents/chat/agent_turn.py b/api/app/agents/chat/agent_turn.py index faa0adb..dfa4769 100644 --- a/api/app/agents/chat/agent_turn.py +++ b/api/app/agents/chat/agent_turn.py @@ -3,7 +3,7 @@ from __future__ import annotations from dataclasses import dataclass -from typing import List +from typing import Any, List @dataclass(frozen=True) @@ -12,3 +12,4 @@ class AgentChatTurn: messages: List[str] skip_tts: bool = False + memory_retrieval_trace: dict[str, Any] | None = None diff --git a/api/app/agents/chat/interview_agent.py b/api/app/agents/chat/interview_agent.py index 96f5a48..bcee84c 100644 --- a/api/app/agents/chat/interview_agent.py +++ b/api/app/agents/chat/interview_agent.py @@ -12,17 +12,17 @@ from app.agents.chat.agent_turn import AgentChatTurn from app.agents.chat.helpers import format_history_string, get_history_with_window from app.agents.chat.personas import normalize_interview_persona from app.agents.chat.prompt_context import ChatPromptContext -from app.agents.chat.stage_detection import keyword_fallback_primary_stage from app.agents.chat.prompts_conversation import ( SLOT_NAME_MAP, get_opening_prompt, ) -from app.agents.state_schema import MemoirStateSchema from app.agents.chat.reply_limits import ( nonempty_segments_or_fallback, segments_from_llm_response, truncate_chat_segments, ) +from app.agents.chat.stage_detection import keyword_fallback_primary_stage +from app.agents.state_schema import MemoirStateSchema from app.core.agent_logging import ( agent_span, log_agent_payload, @@ -92,6 +92,8 @@ class InterviewAgent: background_voice: str = "default", normalized_user_message: Optional[str] = None, occupation: str = "", + profile_birth_year: int | None = None, + profile_era_place: str = "", ) -> AgentChatTurn: """生成状态感知的访谈回复，不持久化（由 Orchestrator 负责）""" if not self.llm: @@ -136,6 +138,8 @@ class InterviewAgent: memory_evidence_text=memory_evidence_text, background_voice=background_voice, occupation=occupation, + profile_birth_year=profile_birth_year, + profile_era_place=profile_era_place, ) system_prompt = ctx.guided_system_prompt() messages: List[Any] = [SystemMessage(content=system_prompt)] diff --git a/api/app/agents/chat/orchestrator.py b/api/app/agents/chat/orchestrator.py index ac6496b..f3fec02 100644 --- a/api/app/agents/chat/orchestrator.py +++ b/api/app/agents/chat/orchestrator.py @@ -13,15 +13,15 @@ from app.agents.chat.agent_turn import AgentChatTurn from app.agents.chat.helpers import get_history_with_window from app.agents.chat.interview_agent import InterviewAgent from app.agents.chat.profile_agent import ProfileAgent -from app.agents.state_schema import MemoirStateSchema -from app.core.agent_logging import agent_summary_enabled, log_agent_detail -from app.core.logging import get_logger from app.agents.chat.stage_detection import ( detect_primary_life_stage, life_stage_display_name, ) +from app.agents.state_schema import MemoirStateSchema +from app.core.agent_logging import agent_summary_enabled, log_agent_detail from app.core.config import settings from app.core.dependencies import get_llm_provider +from app.core.logging import get_logger from app.features.conversation.input_normalize import normalize_chat_input_for_agent from app.features.memoir.state_service import get_or_create_state, switch_stage @@ -48,28 +48,35 @@ async def _fetch_interview_memory_evidence( db: AsyncSession, user_id: str, user_message: str, -) -> str: - """按本轮用户话检索记忆，格式化为短文本；失败或未启用时返回空串。""" +) -> tuple[str, dict | None]: + """按本轮用户话检索记忆：格式化短文本 + 可入库 trace（稳定 id）。""" from app.core.dependencies import get_embedding_provider from app.features.memory.evidence_format import format_evidence_chunks_for_prompt + from app.features.memory.retrieval_trace import ( + chat_memory_retrieval_trace_from_bundle, + ) from app.features.memory.service import MemoryService if not settings.chat_memory_retrieval_enabled: logger.debug( "event=chat_memory_retrieval_skip reason=disabled user_id={}", user_id ) - return "" + return "", None msg = (user_message or "").strip() if not msg: logger.debug( "event=chat_memory_retrieval_skip reason=empty user_id={}", user_id ) - return "" + return "", None try: emb = get_embedding_provider() ms = MemoryService(db, embedding_provider=emb) - bundle = await ms.retrieve(user_id, msg, top_k=settings.chat_memory_top_k) + top_k = settings.chat_memory_top_k + bundle = await ms.retrieve(user_id, msg, top_k=top_k) bd = bundle.model_dump() + trace = chat_memory_retrieval_trace_from_bundle( + bd, top_k=top_k, query_len=len(msg) + ) text = format_evidence_chunks_for_prompt(bd) t = (text or "").strip() if not t: @@ -77,7 +84,7 @@ async def _fetch_interview_memory_evidence( "event=memory_evidence_for_prompt user_id={} formatted_chars=0", user_id, ) - return "" + return "", trace max_c = settings.chat_memory_evidence_max_chars if len(t) > max_c: t = t[: max_c - 3] + "..." @@ -86,14 +93,14 @@ async def _fetch_interview_memory_evidence( user_id, len(t), ) - return t + return t, trace except Exception as e: try: await db.rollback() except Exception as rollback_error: logger.warning("访谈记忆检索失败后回滚也失败: {}", rollback_error) logger.warning("访谈记忆检索失败: {}", e) - return "" + return "", None class ChatOrchestrator: @@ -197,12 +204,15 @@ class ChatOrchestrator: conversation_id, len(responses), ) - return AgentChatTurn(messages=responses, skip_tts=False) + return AgentChatTurn( + messages=responses, skip_tts=False, memory_retrieval_trace=None + ) except Exception as e: logger.error(f"资料收集处理失败: {e}", exc_info=True) return AgentChatTurn( messages=["不好意思刚才没接住，你再说一遍好吗？"], skip_tts=False, + memory_retrieval_trace=None, ) # --- 正式访谈模式 --- @@ -262,10 +272,17 @@ class ChatOrchestrator: background_voice = infer_background_voice(user.occupation) occupation = user.occupation or "" - memory_evidence_text = await _fetch_interview_memory_evidence( + memory_evidence_text, mem_trace = await _fetch_interview_memory_evidence( db, user_id, normalized_user_message ) + profile_birth_year = user.birth_year if user else None + profile_era_place = "" + if user: + profile_era_place = ( + (user.birth_place or user.grew_up_place or "").strip() + ) + turn = await self.interview_agent.generate_response_with_state( conversation_id=conversation_id, user_message=user_message, @@ -276,6 +293,8 @@ class ChatOrchestrator: background_voice=background_voice, normalized_user_message=normalized_user_message, occupation=occupation, + profile_birth_year=profile_birth_year, + profile_era_place=profile_era_place, ) if agent_summary_enabled(): logger.info( @@ -287,6 +306,12 @@ class ChatOrchestrator: len(turn.messages), turn.skip_tts, ) + if mem_trace is not None: + return AgentChatTurn( + messages=turn.messages, + skip_tts=turn.skip_tts, + memory_retrieval_trace=mem_trace, + ) return turn async def extract_profile_from_message( @@ -349,6 +374,8 @@ class ChatOrchestrator: background_voice: str = "default", normalized_user_message: str | None = None, occupation: str = "", + profile_birth_year: int | None = None, + profile_era_place: str = "", ) -> AgentChatTurn: """委托 InterviewAgent 生成访谈回复（持久化由调用方负责）。""" return await self.interview_agent.generate_response_with_state( @@ -361,6 +388,8 @@ class ChatOrchestrator: background_voice=background_voice, normalized_user_message=normalized_user_message, occupation=occupation, + profile_birth_year=profile_birth_year, + profile_era_place=profile_era_place, ) def detect_user_stage(self, user_message: str) -> str: diff --git a/api/app/agents/chat/profile_agent.py b/api/app/agents/chat/profile_agent.py index b03a0b5..23d9263 100644 --- a/api/app/agents/chat/profile_agent.py +++ b/api/app/agents/chat/profile_agent.py @@ -14,17 +14,17 @@ from app.agents.chat.prompts_profile import ( get_profile_followup_prompt, get_profile_greeting_prompt, ) +from app.agents.chat.reply_limits import ( + nonempty_segments_or_fallback, + segments_from_llm_response, + truncate_chat_segments, +) from app.agents.chat.schemas import ProfileExtractionOutput from app.core.agent_logging import agent_span, log_agent_payload, log_agent_summary from app.core.config import settings from app.core.dependencies import get_llm_provider from app.core.llm_call import allm_json_call from app.core.logging import get_logger -from app.agents.chat.reply_limits import ( - nonempty_segments_or_fallback, - segments_from_llm_response, - truncate_chat_segments, -) logger = get_logger(__name__) diff --git a/api/app/agents/chat/prompt_context.py b/api/app/agents/chat/prompt_context.py index 32ae21c..481cd7c 100644 --- a/api/app/agents/chat/prompt_context.py +++ b/api/app/agents/chat/prompt_context.py @@ -20,6 +20,8 @@ class ChatPromptContext: memory_evidence_text: str = "" background_voice: str = "default" occupation: str = "" + profile_birth_year: int | None = None + profile_era_place: str = "" def guided_system_prompt(self) -> str: """用户原话仅以对话历史 + HumanMessage 注入模型。""" @@ -36,4 +38,6 @@ class ChatPromptContext: memory_evidence_text=self.memory_evidence_text, background_voice=self.background_voice, occupation=self.occupation, + profile_birth_year=self.profile_birth_year, + profile_era_place=self.profile_era_place, ) diff --git a/api/app/agents/chat/prompts.py b/api/app/agents/chat/prompts.py index 0320e34..3a58104 100644 --- a/api/app/agents/chat/prompts.py +++ b/api/app/agents/chat/prompts.py @@ -4,6 +4,13 @@ Chat 模块提示词：用户资料收集 + 对话访谈 from app.agents.chat.output_rules import chat_output_rules +# Conversation prompts（对话访谈） +from app.agents.chat.prompts_conversation import ( + SLOT_NAME_MAP, + get_guided_conversation_prompt, + get_opening_prompt, +) + # Profile prompts（用户资料收集） from app.agents.chat.prompts_profile import ( PROFILE_FIELD_NAMES, @@ -14,13 +21,6 @@ from app.agents.chat.prompts_profile import ( get_profile_greeting_prompt, ) -# Conversation prompts（对话访谈） -from app.agents.chat.prompts_conversation import ( - SLOT_NAME_MAP, - get_guided_conversation_prompt, - get_opening_prompt, -) - __all__ = [ "chat_output_rules", "PROFILE_FIELD_NAMES", diff --git a/api/app/agents/chat/prompts_conversation.py b/api/app/agents/chat/prompts_conversation.py index 68d9091..dc7d40c 100644 --- a/api/app/agents/chat/prompts_conversation.py +++ b/api/app/agents/chat/prompts_conversation.py @@ -9,11 +9,11 @@ from app.agents.chat.background_voice import ( normalize_background_voice, ) from app.agents.chat.occupation_context import get_occupation_chat_hint +from app.agents.chat.output_rules import chat_output_rules from app.agents.chat.personas import ( get_interview_persona_tone_hint, normalize_interview_persona, ) -from app.agents.chat.output_rules import chat_output_rules from app.agents.stage_constants import CHAT_STAGES, STAGE_DISPLAY_ZH, STAGE_ERA_HINTS from app.core.config import settings @@ -44,25 +44,18 @@ SLOT_NAME_MAP = { } -def _compact_era_hint(current_stage: str, user_profile_context: str) -> str: - """单行时代联想，可选附在进度后。""" - if not user_profile_context: - return "" - - birth_year = None - birth_place = "" - for line in user_profile_context.split("\n"): - if "出生年份" in line: - try: - birth_year = int(line.split("：")[1].strip().replace("年", "")) - except (ValueError, IndexError): - pass - if "出生地" in line or "成长地" in line: - birth_place = line.split("：")[1].strip() if "：" in line else "" - +def _compact_era_hint( + current_stage: str, + *, + birth_year: int | None = None, + era_place: str = "", +) -> str: + """单行时代联想，可选附在进度后。出生年与地点由调用方从用户资料结构化传入。""" if not birth_year: return "" + birth_place = (era_place or "").strip() + age_range = STAGE_ERA_HINTS.get(current_stage, (0, 30)) era_start = birth_year + age_range[0] era_end = birth_year + age_range[1] @@ -230,6 +223,8 @@ def get_guided_conversation_prompt( memory_evidence_text: str = "", background_voice: str = "default", occupation: str = "", + profile_birth_year: Optional[int] = None, + profile_era_place: str = "", ) -> str: """生成状态感知的对话提示词；用户原话仅以 HumanMessage 传入，不写入本 system 文本。""" persona_key = normalize_interview_persona(persona) @@ -285,7 +280,11 @@ def get_guided_conversation_prompt( ) era_line = "" if settings.chat_era_context_enabled: - era_line = _compact_era_hint(active_stage, user_profile_context) + era_line = _compact_era_hint( + active_stage, + birth_year=profile_birth_year, + era_place=profile_era_place, + ) if user_jumped: topic_desc = ( diff --git a/api/app/agents/chat/prompts_profile.py b/api/app/agents/chat/prompts_profile.py index ff1998a..16fc66c 100644 --- a/api/app/agents/chat/prompts_profile.py +++ b/api/app/agents/chat/prompts_profile.py @@ -6,7 +6,6 @@ from typing import Dict, List, Optional from app.agents.chat.output_rules import chat_output_rules - PROFILE_FIELD_NAMES = { "birth_year": "出生年份", "birth_place": "出生地", diff --git a/api/app/agents/image_prompt/orchestrator.py b/api/app/agents/image_prompt/orchestrator.py index a920779..ba618e9 100644 --- a/api/app/agents/image_prompt/orchestrator.py +++ b/api/app/agents/image_prompt/orchestrator.py @@ -8,9 +8,8 @@ from __future__ import annotations from typing import Any, Optional -from app.features.memoir.memoir_images.settings import MemoirImageSettings - from app.agents.image_prompt.prompt_agent import PromptGenerationAgent +from app.features.memoir.memoir_images.settings import MemoirImageSettings class ImagePromptOrchestrator: diff --git a/api/app/agents/memoir/batch_phase1_prep.py b/api/app/agents/memoir/batch_phase1_prep.py index 5d855bc..307acc6 100644 --- a/api/app/agents/memoir/batch_phase1_prep.py +++ b/api/app/agents/memoir/batch_phase1_prep.py @@ -9,8 +9,8 @@ from typing import Any, Dict, List from app.agents.memoir.prompts import get_batch_memoir_phase1_prep_prompt from app.agents.memoir.schemas import BatchPhase1LLMOutput -from app.agents.state_schema import MemoirStateSchema from app.agents.stage_constants import STAGE_SLOT_KEYS +from app.agents.state_schema import MemoirStateSchema from app.core.config import settings from app.core.llm_call import LLMCallError, llm_json_call from app.core.logging import get_logger diff --git a/api/app/agents/memoir/classification_agent.py b/api/app/agents/memoir/classification_agent.py index f04a519..e0245d9 100644 --- a/api/app/agents/memoir/classification_agent.py +++ b/api/app/agents/memoir/classification_agent.py @@ -13,6 +13,8 @@ import re from dataclasses import dataclass from typing import Any +from pydantic import ValidationError + from app.agents.memoir.prompts import get_chapter_classification_json_prompt from app.agents.memoir.schemas import ClassificationOutput from app.agents.stage_constants import ( @@ -22,7 +24,7 @@ from app.agents.stage_constants import ( ) from app.core.config import settings from app.core.json_utils import extract_json_payload -from app.core.llm_call import llm_json_call +from app.core.llm_call import LLMCallError, llm_json_call from app.core.logging import get_logger logger = get_logger(__name__) @@ -159,7 +161,7 @@ class ClassificationAgent: ) if category in CHAPTER_CATEGORIES: return ChapterClassifyResult(category=category, llm_said_none=False) - except Exception as e: + except (LLMCallError, ValidationError, ValueError, KeyError) as e: logger.warning("ClassificationAgent LLM 章节分类失败: {}", e) stage = _detect_stage(text, fallback_stage) diff --git a/api/app/agents/memoir/narrative_agent.py b/api/app/agents/memoir/narrative_agent.py index f34131c..04dba01 100644 --- a/api/app/agents/memoir/narrative_agent.py +++ b/api/app/agents/memoir/narrative_agent.py @@ -7,13 +7,13 @@ from __future__ import annotations from typing import Any, Dict, Optional -from app.agents.stage_constants import CHAPTER_CATEGORIES from app.agents.memoir.prompts import ( get_creative_title_json_prompt, get_narrative_json_prompt, get_narrative_merge_json_prompt, ) from app.agents.memoir.schemas import MemoirTitleOutput +from app.agents.stage_constants import CHAPTER_CATEGORIES from app.core.config import settings from app.core.langchain_llm import invoke_json_object from app.core.llm_call import llm_json_call diff --git a/api/app/agents/memoir/orchestrator.py b/api/app/agents/memoir/orchestrator.py index ee723ab..d6be64b 100644 --- a/api/app/agents/memoir/orchestrator.py +++ b/api/app/agents/memoir/orchestrator.py @@ -16,11 +16,11 @@ from app.agents.memoir.batch_phase1_prep import ( ) from app.agents.memoir.classification_agent import ( ClassificationAgent, + _looks_like_fragment_only, ) from app.agents.memoir.classification_agent import ( _detect_stage as detect_stage_from_keywords, ) -from app.agents.memoir.classification_agent import _looks_like_fragment_only from app.agents.memoir.extraction_agent import ExtractionAgent, ExtractionResult from app.agents.stage_constants import normalize_chapter_category, normalize_chat_stage from app.agents.state_schema import MemoirStateSchema diff --git a/api/app/agents/memoir/prompts.py b/api/app/agents/memoir/prompts.py index 9af6330..9c65d41 100644 --- a/api/app/agents/memoir/prompts.py +++ b/api/app/agents/memoir/prompts.py @@ -10,10 +10,7 @@ from typing import Optional from app.agents.chat.background_voice import get_background_voice_narrative_block from app.agents.chat.occupation_context import get_occupation_narrative_hint from app.agents.stage_constants import STAGE_ERA_HINTS, STAGE_SLOT_KEYS -from app.features.memory.evidence_format import ( - dedupe_evidence_chunk_rows, - format_evidence_chunks_for_prompt, -) +from app.features.memory.evidence_format import format_evidence_chunks_for_prompt def _memoir_fidelity_core_rules() -> str: @@ -581,6 +578,3 @@ def format_narrative_user_content(oral_text: str, evidence_text: str = "") -> st "【仅供参考的相关记忆摘录（非本段口述；不得把其中具体事实写成本轮亲历经历，仅可作主题衔接）】\n" f"{ev}" ) - - -# dedupe_evidence_chunk_rows / format_evidence_chunks_for_prompt 见 app.features.memory.evidence_format diff --git a/api/app/agents/memoir/story_route_payload.py b/api/app/agents/memoir/story_route_payload.py index ed1e7e3..bda7ad0 100644 --- a/api/app/agents/memoir/story_route_payload.py +++ b/api/app/agents/memoir/story_route_payload.py @@ -9,7 +9,7 @@ from __future__ import annotations import json import re from datetime import timezone -from typing import Any, TYPE_CHECKING +from typing import TYPE_CHECKING, Any if TYPE_CHECKING: from app.core.config import Settings diff --git a/api/app/core/json_utils.py b/api/app/core/json_utils.py index 5dd870c..64423d3 100644 --- a/api/app/core/json_utils.py +++ b/api/app/core/json_utils.py @@ -21,8 +21,15 @@ def extract_json_payload(raw_response: str | None) -> str: return cleaned start = cleaned.find("{") - end = cleaned.rfind("}") - if start != -1 and end != -1 and end > start: - return cleaned[start : end + 1].strip() + if start == -1: + return cleaned + depth = 0 + for i, ch in enumerate(cleaned[start:], start): + if ch == "{": + depth += 1 + elif ch == "}": + depth -= 1 + if depth == 0: + return cleaned[start : i + 1].strip() return cleaned diff --git a/api/app/core/middleware.py b/api/app/core/middleware.py index 5c28993..9bdcdc0 100644 --- a/api/app/core/middleware.py +++ b/api/app/core/middleware.py @@ -4,10 +4,11 @@ HTTP 中间件：request_id 注入。 import uuid -from app.core.logging import logger from starlette.middleware.base import BaseHTTPMiddleware from starlette.requests import Request +from app.core.logging import logger + class RequestIdMiddleware(BaseHTTPMiddleware): """Inject request_id into request.state and response headers, bind to loguru context.""" diff --git a/api/app/core/redis.py b/api/app/core/redis.py index 1710b9d..e46a4da 100644 --- a/api/app/core/redis.py +++ b/api/app/core/redis.py @@ -34,7 +34,17 @@ class RedisService: ) await self._client.ping() logger.info("Redis 连接成功") - logger.debug("Redis 连接 URL: {}", self.redis_url) + try: + from urllib.parse import urlparse + + p = urlparse(self.redis_url) + logger.debug( + "Redis 连接 host={} port={}", + p.hostname or "", + p.port or "", + ) + except Exception: + logger.debug("Redis 已连接（URL 解析省略）") except Exception as e: logger.error("Redis 连接失败: {}", e) raise diff --git a/api/app/core/task_tracker.py b/api/app/core/task_tracker.py index 972a92e..710bf8e 100644 --- a/api/app/core/task_tracker.py +++ b/api/app/core/task_tracker.py @@ -3,10 +3,10 @@ """ import json -from app.core.logging import get_logger from datetime import datetime, timezone -from typing import Any, Dict, List, Optional +from typing import Any, Dict, List +from app.core.logging import get_logger from app.core.redis import redis_service logger = get_logger(__name__) diff --git a/api/app/core/text_normalize.py b/api/app/core/text_normalize.py index bf3e738..9a65b59 100644 --- a/api/app/core/text_normalize.py +++ b/api/app/core/text_normalize.py @@ -6,9 +6,9 @@ import json import re from typing import Any +from app.core.json_utils import extract_json_payload from app.core.langchain_llm import invoke_json_object from app.core.logging import get_logger -from app.core.json_utils import extract_json_payload logger = get_logger(__name__) diff --git a/api/app/features/auth/service.py b/api/app/features/auth/service.py index f412350..eced006 100644 --- a/api/app/features/auth/service.py +++ b/api/app/features/auth/service.py @@ -2,16 +2,19 @@ import random import secrets import uuid from datetime import datetime, timedelta, timezone + from sqlalchemy.ext.asyncio import AsyncSession from app.core.db import utc_now from app.core.security import ( create_access_token, - create_refresh_token as generate_refresh_token_str, get_token_expires_at, hash_password, verify_password, ) +from app.core.security import ( + create_refresh_token as generate_refresh_token_str, +) from app.features.auth import repo from app.features.auth.models import RefreshToken, SmsVerificationCode from app.features.user.models import User diff --git a/api/app/features/content/router.py b/api/app/features/content/router.py index 7cd8aa3..c4d1f11 100644 --- a/api/app/features/content/router.py +++ b/api/app/features/content/router.py @@ -7,24 +7,14 @@ from typing import List from fastapi import APIRouter from fastapi.responses import HTMLResponse -from pydantic import BaseModel + +from app.features.content.schemas import FAQResponse router = APIRouter(tags=["content"], responses={404: {"description": "资源不存在"}}) _STATIC_DIR = Path(__file__).resolve().parent.parent.parent.parent / "static" -# ── Schemas ────────────────────────────────────────────────── - - -class FAQResponse(BaseModel): - id: str - question: str - answer: str - category: str - order: int - - # ── FAQ data ───────────────────────────────────────────────── FAQS = [ diff --git a/api/app/features/conversation/history_store.py b/api/app/features/conversation/history_store.py index 99a344a..1762971 100644 --- a/api/app/features/conversation/history_store.py +++ b/api/app/features/conversation/history_store.py @@ -3,6 +3,7 @@ from __future__ import annotations import uuid +from dataclasses import dataclass from datetime import datetime, timedelta, timezone from typing import Any @@ -22,6 +23,14 @@ logger = get_logger(__name__) AI_RESPONSE_SEGMENT_JOIN = "[SPLIT]" +@dataclass(frozen=True) +class HumanAiTurnIds: + """Durable ids for one user + assistant pair in conversation_messages.""" + + human_message_id: str + assistant_message_id: str + + def _utc_now() -> datetime: return datetime.now(timezone.utc) @@ -89,7 +98,8 @@ class ConversationHistoryStore: audio_duration_seconds: int | None, tts_audio_urls: list[str] | None, segment_id: str | None, - ) -> str | None: + memory_retrieval_trace: dict | None = None, + ) -> HumanAiTurnIds | None: if not responses: return None human_ts = user_message_timestamp or _utc_now() @@ -120,13 +130,17 @@ class ConversationHistoryStore: tts_audio_urls=tts_audio_urls if tts_audio_urls else None, segment_id=segment_id, created_at=ai_ts, + memory_retrieval_trace_json=memory_retrieval_trace, ) repo.add_conversation_message(human, self._db) repo.add_conversation_message(ai, self._db) await self._touch_conversation(conversation_id, occurred_at=ai_ts) await self._db.commit() await self._sync_redis_best_effort(conversation_id) - return ai.id + return HumanAiTurnIds( + human_message_id=str(human.id), + assistant_message_id=str(ai.id), + ) async def attach_ai_tts_audio_urls( self, diff --git a/api/app/features/conversation/input_normalize.py b/api/app/features/conversation/input_normalize.py index 2c3eb91..edb6bb3 100644 --- a/api/app/features/conversation/input_normalize.py +++ b/api/app/features/conversation/input_normalize.py @@ -10,8 +10,8 @@ from __future__ import annotations from typing import Any from app.core.config import settings -from app.core.text_normalize import apply_oral_rules, llm_normalize_text from app.core.logging import get_logger +from app.core.text_normalize import apply_oral_rules, llm_normalize_text logger = get_logger(__name__) diff --git a/api/app/features/conversation/lineage_schemas.py b/api/app/features/conversation/lineage_schemas.py new file mode 100644 index 0000000..34e1383 --- /dev/null +++ b/api/app/features/conversation/lineage_schemas.py @@ -0,0 +1,182 @@ +"""Canonical dialogue lineage (message-grade provenance) for memory, story, chapter, eval.""" + +from __future__ import annotations + +from typing import Any, Sequence + +from pydantic import BaseModel, Field, field_validator + +LINEAGE_SCHEMA_VERSION = 1 + + +class DialogueTurnRef(BaseModel): + """One interview beat: user message paired with assistant reply (if any).""" + + user_message_id: str + assistant_message_id: str | None = None + + +class DialogueLineage(BaseModel): + """ + Multi-turn provenance. `turns` order is chronological within the contributing slice. + """ + + schema_version: int = Field(default=LINEAGE_SCHEMA_VERSION, ge=1) + conversation_id: str + turns: list[DialogueTurnRef] = Field(default_factory=list) + primary_user_message_id: str | None = None + segment_ids: list[str] = Field(default_factory=list) + + @field_validator("turns") + @classmethod + def _non_empty_user_ids(cls, v: list[DialogueTurnRef]) -> list[DialogueTurnRef]: + for t in v: + if not (t.user_message_id or "").strip(): + raise ValueError("turn.user_message_id must be non-empty") + return v + + def model_dump_json_safe(self) -> dict[str, Any]: + return self.model_dump(mode="json") + + @classmethod + def for_single_turn( + cls, + *, + conversation_id: str, + user_message_id: str, + assistant_message_id: str | None, + segment_ids: list[str] | None = None, + ) -> DialogueLineage: + return cls( + conversation_id=conversation_id, + turns=[ + DialogueTurnRef( + user_message_id=user_message_id, + assistant_message_id=assistant_message_id, + ) + ], + primary_user_message_id=user_message_id, + segment_ids=list(segment_ids or []), + ) + + +def parse_dialogue_lineage(raw: Any) -> DialogueLineage | None: + if raw is None: + return None + if isinstance(raw, DialogueLineage): + return raw + if not isinstance(raw, dict): + return None + try: + return DialogueLineage.model_validate(raw) + except Exception: + return None + + +def primary_user_message_id_from_lineage(raw: Any) -> str | None: + ln = parse_dialogue_lineage(raw) + if ln is None: + return None + if ln.primary_user_message_id: + return ln.primary_user_message_id + if ln.turns: + return ln.turns[0].user_message_id + return None + + +def merge_dialogue_lineages( + lineages: Sequence[DialogueLineage | dict | None], + *, + conversation_id: str, + segment_ids_ordered: list[str] | None = None, +) -> DialogueLineage | None: + """Ordered union of turns; dedupe by user_message_id (first occurrence wins).""" + turns_out: list[DialogueTurnRef] = [] + seen_user: set[str] = set() + segments_accum: list[str] = [] + + for raw in lineages: + ln = parse_dialogue_lineage(raw) + if ln is None: + continue + for sid in ln.segment_ids: + if sid and sid not in segments_accum: + segments_accum.append(sid) + for t in ln.turns: + uid = t.user_message_id.strip() + if not uid or uid in seen_user: + continue + seen_user.add(uid) + turns_out.append( + DialogueTurnRef( + user_message_id=uid, + assistant_message_id=t.assistant_message_id, + ) + ) + + if segment_ids_ordered: + for sid in segment_ids_ordered: + if sid and sid not in segments_accum: + segments_accum.append(sid) + + if not turns_out: + return None + + return DialogueLineage( + conversation_id=conversation_id, + turns=turns_out, + primary_user_message_id=turns_out[0].user_message_id, + segment_ids=segments_accum, + ) + + +def aggregate_lineage_from_segments( + segments: Sequence[Any], + *, + conversation_id_fallback: str | None = None, +) -> dict[str, Any] | None: + """ + Build merged lineage dict from ORM segments (expects .id, .conversation_id, + .lineage_json, optional .user_message_id). + """ + if not segments: + return None + conv0 = conversation_id_fallback or getattr( + segments[0], "conversation_id", None + ) or "" + if not conv0: + lj0 = getattr(segments[0], "lineage_json", None) + if isinstance(lj0, dict) and lj0.get("conversation_id"): + conv0 = str(lj0["conversation_id"]) + if not conv0: + return None + + lineages: list[DialogueLineage | dict | None] = [] + seg_ids_order: list[str] = [] + for seg in segments: + sid = str(getattr(seg, "id", "") or "") + if sid: + seg_ids_order.append(sid) + lj = getattr(seg, "lineage_json", None) + if lj: + lineages.append(lj) + else: + um = getattr(seg, "user_message_id", None) + if um: + lineages.append( + DialogueLineage.for_single_turn( + conversation_id=str( + getattr(seg, "conversation_id", None) or conv0 + ), + user_message_id=str(um), + assistant_message_id=None, + segment_ids=[sid] if sid else None, + ) + ) + + merged = merge_dialogue_lineages( + lineages, conversation_id=str(conv0), segment_ids_ordered=seg_ids_order + ) + if merged is None: + return None + return merged.model_dump_json_safe() diff --git a/api/app/features/conversation/models.py b/api/app/features/conversation/models.py index a8811b9..20a97b2 100644 --- a/api/app/features/conversation/models.py +++ b/api/app/features/conversation/models.py @@ -58,6 +58,12 @@ class Segment(Base): skip_narrative = Column(Boolean, default=False, server_default="false") agent_response = Column(Text, nullable=True) tts_audio_urls = Column(JSON, nullable=True) + # 用户轮次 durable message id（与 lineage_json 同步；便于查询） + user_message_id = Column( + String, ForeignKey("conversation_messages.id", ondelete="SET NULL"), nullable=True + ) + # DialogueLineage JSON（schema 见 conversation.lineage_schemas） + lineage_json = Column(JSON, nullable=True) conversation = relationship("Conversation", back_populates="segments") @@ -77,6 +83,8 @@ class ConversationMessage(Base): tts_audio_urls = Column(JSON, nullable=True) segment_id = Column(String, ForeignKey("segments.id"), nullable=True) created_at = Column(DateTime(timezone=True), default=utc_now) + # 本轮（与用户句配对）助手生成前检索到的 memory 证据 id 快照；Phase 8 可追溯 + memory_retrieval_trace_json = Column(JSON, nullable=True) conversation = relationship("Conversation", back_populates="messages") segment = relationship("Segment", foreign_keys=[segment_id]) diff --git a/api/app/features/conversation/router.py b/api/app/features/conversation/router.py index bdfaab8..7064c07 100644 --- a/api/app/features/conversation/router.py +++ b/api/app/features/conversation/router.py @@ -102,4 +102,6 @@ async def organize_conversation( raise except Exception as e: logger.exception("提交整理任务失败: {}", e) - raise HTTPException(status_code=500, detail=f"提交整理任务失败: {str(e)}") + raise HTTPException( + status_code=500, detail="提交整理任务失败，请稍后重试" + ) from e diff --git a/api/app/features/conversation/ws/pipeline.py b/api/app/features/conversation/ws/pipeline.py index c9c78b7..a25a74d 100644 --- a/api/app/features/conversation/ws/pipeline.py +++ b/api/app/features/conversation/ws/pipeline.py @@ -27,6 +27,7 @@ from app.features.conversation.history_store import ( AI_RESPONSE_SEGMENT_JOIN, ConversationHistoryStore, ) +from app.features.conversation.lineage_schemas import DialogueLineage from app.features.conversation.models import Conversation, Segment from app.features.conversation.ws.connection_manager import manager from app.features.conversation.ws.message_types import MessageType @@ -37,6 +38,7 @@ from app.features.conversation.ws.profile_collector import ( ) from app.features.memoir.background_runner import BackgroundTaskRunner from app.features.user.models import User +from app.ports.asr import ASRTranscriptionError logger = get_logger(__name__) @@ -492,7 +494,16 @@ async def process_audio_segment( conversation_id, segment_index, ) - transcript_text = await _transcribe_long_audio(audio_bytes, fmt="m4a") + try: + transcript_text = await _transcribe_long_audio(audio_bytes, fmt="m4a") + except ASRTranscriptionError as e: + logger.warning( + "ASR 转写失败 segment_index={} conversation_id={}: {}", + segment_index, + conversation_id, + e, + ) + transcript_text = "" await manager.send_message( conversation_id, { @@ -511,12 +522,12 @@ async def process_audio_segment( if _is_transcribe_failure(transcript_text): detail = (transcript_text or "").strip() - if detail.startswith("转写失败"): - user_msg = f"分段 {segment_index} {detail}" - elif not detail: - user_msg = f"分段 {segment_index} 转写失败：未识别到内容（请检查后端 ASR 配置）" + if not detail: + user_msg = ( + f"分段 {segment_index} 未识别到语音内容，请重试或检查麦克风与网络" + ) else: - user_msg = f"分段 {segment_index} 转写失败：{detail[:400]}" + user_msg = f"分段 {segment_index} 语音识别失败，请稍后再试" await manager.send_message( conversation_id, { @@ -607,7 +618,7 @@ async def process_audio_segment( { "type": MessageType.ERROR, "data": { - "message": f"分段处理失败: {str(e)}", + "message": "语音分段处理遇到问题，请重试", "segment_index": segment_index, }, "timestamp": datetime.now(timezone.utc).isoformat(), @@ -677,7 +688,7 @@ async def process_user_message( segment.agent_response = AI_RESPONSE_SEGMENT_JOIN.join(responses) _mark_conversation_active(conversation) - ai_msg_id = await store.record_human_ai_turn( + turn_ids = await store.record_human_ai_turn( conversation_id=conversation_id, user_message=user_message, responses=responses, @@ -687,8 +698,11 @@ async def process_user_message( audio_duration_seconds=audio_dur, tts_audio_urls=None, segment_id=segment.id, + memory_retrieval_trace=getattr( + turn, "memory_retrieval_trace", None + ), ) - if not ai_msg_id: + if not turn_ids: logger.warning( "process_user_message: 无有效助手段落（responses 为空），conversation_id={} segment_id={}", conversation_id, @@ -707,6 +721,23 @@ async def process_user_message( ) return + lineage = DialogueLineage.for_single_turn( + conversation_id=conversation_id, + user_message_id=turn_ids.human_message_id, + assistant_message_id=turn_ids.assistant_message_id, + segment_ids=[str(segment.id)], + ) + await db.execute( + update(Segment) + .where(Segment.id == segment.id) + .values( + user_message_id=turn_ids.human_message_id, + lineage_json=lineage.model_dump(mode="json"), + ) + ) + await db.commit() + + ai_msg_id = turn_ids.assistant_message_id tts_epoch_start = _tts_epoch_value(conversation_id) n = len(responses) for i, response_text in enumerate(responses): @@ -779,7 +810,7 @@ async def process_user_message( conversation_id, { "type": MessageType.ERROR, - "data": {"message": f"生成回应失败: {str(e)}"}, + "data": {"message": "生成回应时遇到问题，请稍后再试"}, "timestamp": datetime.now(timezone.utc).isoformat(), }, ) diff --git a/api/app/features/conversation/ws/router.py b/api/app/features/conversation/ws/router.py index 0518449..b03d573 100644 --- a/api/app/features/conversation/ws/router.py +++ b/api/app/features/conversation/ws/router.py @@ -5,15 +5,14 @@ WebSocket 路由：实时对话通信 import asyncio import base64 -import uuid from datetime import datetime, timezone from fastapi import WebSocket, WebSocketDisconnect, status from starlette.websockets import WebSocketState from app.agents.chat.background_voice import infer_background_voice -from app.agents.stage_constants import STAGE_TO_ORDER from app.agents.chat.prompts_profile import format_user_profile_context +from app.agents.stage_constants import STAGE_TO_ORDER from app.core.db import AsyncSessionLocal from app.core.dependencies import get_asr_provider from app.core.logging import get_logger diff --git a/api/app/features/evaluation/eval_trace_format.py b/api/app/features/evaluation/eval_trace_format.py new file mode 100644 index 0000000..cdb11ed --- /dev/null +++ b/api/app/features/evaluation/eval_trace_format.py @@ -0,0 +1,275 @@ +"""将证据闭包格式化为评审可读文本，并记录截断/丢弃区块（可审计）。""" + +from __future__ import annotations + +from app.features.conversation.models import Segment +from app.features.evaluation.eval_trace_schemas import ( + ChapterEvidenceBundle, + EvidenceFormatMeta, + FormattedMemoirEvidence, + StoryEvidenceBundle, +) +from app.features.memory.models import ( + MemoryChunk, + MemoryFact, + MemorySummary, + TimelineEvent, +) + +# 与 judge_service._MEMOIR_EVIDENCE_MAX 对齐：访谈与结构化证据分预算，避免总长失控 +_MEMOIR_TRANSCRIPT_CAP = 12_000 +_MEMOIR_STRUCTURED_CAP = 12_000 + + +def _approx_tokens(chars: int) -> int: + return max(0, chars // 4) + + +def _segment_message_id_header(seg: Segment) -> str: + um: str | None = None + am: str | None = None + lj = getattr(seg, "lineage_json", None) + if isinstance(lj, dict): + turns = lj.get("turns") + if isinstance(turns, list) and turns: + t0 = turns[0] + if isinstance(t0, dict): + um = str(t0.get("user_message_id") or "").strip() or None + am = str(t0.get("assistant_message_id") or "").strip() or None + if um is None: + raw_um = getattr(seg, "user_message_id", None) + if raw_um: + um = str(raw_um) + parts: list[str] = [] + if um: + parts.append(f"user_msg={um}") + if am: + parts.append(f"assistant_msg={am}") + return " · ".join(parts) if parts else "" + + +def build_segment_transcript( + segments: list[Segment], + ai_by_segment: dict[str, str], +) -> str: + """按 segment 绑定的局部访谈块（用户句 + AI 回复）。""" + blocks: list[str] = [] + for i, seg in enumerate(segments, start=1): + uid = str(seg.id) + user_txt = (seg.user_input_text or "").strip() + ai_txt = (ai_by_segment.get(uid) or seg.agent_response or "").strip() + id_extra = _segment_message_id_header(seg) + head = ( + f"### Segment {i} · id={uid} · conversation={seg.conversation_id}" + + (f" · {id_extra}" if id_extra else "") + ) + body_u = f"用户: {user_txt}" if user_txt else "用户: （空）" + body_a = f"AI: {ai_txt}" if ai_txt else "AI: （无日志/无 agent_response）" + blocks.append(f"{head}\n{body_u}\n{body_a}") + return "\n\n".join(blocks) + + +def build_structured_evidence_text( + *, + chunks: list[MemoryChunk], + facts: list[MemoryFact], + events: list[TimelineEvent], + summaries: list[MemorySummary], + max_chars: int = _MEMOIR_STRUCTURED_CAP, +) -> tuple[str, bool, list[str]]: + """ + 结构化记忆证据块；返回 (text, truncated, dropped_section_tags)。 + """ + parts: list[str] = [] + dropped: list[str] = [] + used = 0 + truncated = False + + def _add_section(title: str, body: str) -> None: + nonlocal used, truncated + block = f"{title}\n{body}".strip() + if not block: + return + if used + len(block) + 2 > max_chars: + truncated = True + dropped.append(title.strip("【】").split("·")[0].strip()) + return + parts.append(block) + used += len(block) + 2 + + if chunks: + lines = [] + for c in chunks: + snippet = (c.content or "").strip() + if len(snippet) > 1200: + snippet = snippet[:1200] + "…" + lines.append(f"- chunk `{c.id}`: {snippet}") + _add_section("【记忆片段 chunks】", "\n".join(lines)) + if facts: + lines = [] + for f in facts: + subj = (f.subject or "").strip() + pred = (f.predicate or "").strip() + lines.append( + f"- fact `{f.id}` ({f.fact_type}): {subj} · {pred}".strip(" ·") + ) + _add_section("【记忆事实 facts】", "\n".join(lines)) + if events: + lines = [] + for e in events: + lines.append( + f"- timeline `{e.id}`: {e.title} ({e.event_year or e.event_date or ''})" + ) + if e.description: + desc = (e.description or "").strip() + if len(desc) > 400: + desc = desc[:400] + "…" + lines.append(f" {desc}") + _add_section("【时间线 timeline】", "\n".join(lines)) + if summaries: + lines = [] + for s in summaries: + body = (s.content or "").strip() + if len(body) > 2000: + body = body[:2000] + "…" + lines.append(f"- summary `{s.id}` ({s.summary_type}): {body}") + _add_section("【摘要 summaries】", "\n".join(lines)) + + return "\n\n".join(parts).strip(), truncated, dropped + + +def evidence_summary_line( + *, + lineage_tier: str, + segment_n: int, + conv_n: int, + chunk_n: int, + fact_n: int, + tl_n: int, + sum_n: int, + notes: list[str], +) -> str: + bits = [ + f"tier={lineage_tier}", + f"segments={segment_n}", + f"conversations={conv_n}", + f"chunks={chunk_n}", + f"facts={fact_n}", + f"timeline={tl_n}", + f"summaries={sum_n}", + ] + if notes: + bits.append("notes=" + "; ".join(notes[:3])) + return "; ".join(bits) + + +def format_chapter_for_judge( + bundle: ChapterEvidenceBundle, + *, + transcript: str, + chunks: list[MemoryChunk], + facts: list[MemoryFact], + events: list[TimelineEvent], + summaries: list[MemorySummary], +) -> FormattedMemoirEvidence: + t_cap = _MEMOIR_TRANSCRIPT_CAP + s_cap = _MEMOIR_STRUCTURED_CAP + dropped: list[str] = [] + truncated = False + + t_in = transcript.strip() + if len(t_in) > t_cap: + truncated = True + dropped.append("source_transcript_tail") + t_in = t_in[:t_cap] + "\n\n…（原始对话证据已截断）" + + struct, s_trunc, s_drop = build_structured_evidence_text( + chunks=chunks, + facts=facts, + events=events, + summaries=summaries, + max_chars=s_cap, + ) + if s_trunc: + truncated = True + dropped.extend(s_drop) + + meta = EvidenceFormatMeta( + truncated=truncated, + dropped_sections=sorted(set(dropped)), + included_token_estimate=_approx_tokens(len(t_in) + len(struct)), + transcript_chars_included=len(t_in), + structured_evidence_chars_included=len(struct), + ) + summary = evidence_summary_line( + lineage_tier=bundle.lineage_tier, + segment_n=len(bundle.segment_ids), + conv_n=len(bundle.conversation_ids), + chunk_n=len(bundle.memory_chunk_ids), + fact_n=len(bundle.memory_fact_ids), + tl_n=len(bundle.timeline_event_ids), + sum_n=len(bundle.summary_ids), + notes=bundle.notes, + ) + return FormattedMemoirEvidence( + source_transcript=t_in, + structured_evidence=struct, + format_meta=meta, + evidence_summary=summary, + ) + + +def format_story_for_judge( + bundle: StoryEvidenceBundle, + *, + transcript: str, + chunks: list[MemoryChunk], + facts: list[MemoryFact], + events: list[TimelineEvent], + summaries: list[MemorySummary], +) -> FormattedMemoirEvidence: + t_cap = _MEMOIR_TRANSCRIPT_CAP + s_cap = _MEMOIR_STRUCTURED_CAP + dropped: list[str] = [] + truncated = False + + t_in = transcript.strip() + if len(t_in) > t_cap: + truncated = True + dropped.append("source_transcript_tail") + t_in = t_in[:t_cap] + "\n\n…（原始对话证据已截断）" + + struct, s_trunc, s_drop = build_structured_evidence_text( + chunks=chunks, + facts=facts, + events=events, + summaries=summaries, + max_chars=s_cap, + ) + if s_trunc: + truncated = True + dropped.extend(s_drop) + + meta = EvidenceFormatMeta( + truncated=truncated, + dropped_sections=sorted(set(dropped)), + included_token_estimate=_approx_tokens(len(t_in) + len(struct)), + transcript_chars_included=len(t_in), + structured_evidence_chars_included=len(struct), + ) + summary = evidence_summary_line( + lineage_tier=bundle.lineage_tier, + segment_n=len(bundle.segment_ids), + conv_n=len(bundle.conversation_ids), + chunk_n=len(bundle.memory_chunk_ids), + fact_n=len(bundle.memory_fact_ids), + tl_n=len(bundle.timeline_event_ids), + sum_n=len(bundle.summary_ids), + notes=bundle.notes, + ) + return FormattedMemoirEvidence( + source_transcript=t_in, + structured_evidence=struct, + format_meta=meta, + evidence_summary=summary, + ) diff --git a/api/app/features/evaluation/eval_trace_repo.py b/api/app/features/evaluation/eval_trace_repo.py new file mode 100644 index 0000000..cb05733 --- /dev/null +++ b/api/app/features/evaluation/eval_trace_repo.py @@ -0,0 +1,319 @@ +"""评测取证 repo：所有查询显式携带 user_id，避免跨租户串数据。""" + +from __future__ import annotations + +from sqlalchemy import or_, select +from sqlalchemy.ext.asyncio import AsyncSession +from sqlalchemy.orm import joinedload + +from app.features.conversation.models import Conversation, ConversationMessage, Segment +from app.features.memoir.models import Chapter, ChapterStoryLink +from app.features.memory.models import ( + MemoryChunk, + MemoryFact, + MemorySource, + MemorySummary, + TimelineEvent, +) +from app.features.story.models import Story, StoryEvidenceLink + + +def normalize_source_segment_ids(raw: object) -> list[str]: + """chapter.source_segments：历史为 list[str]。""" + if not raw: + return [] + if isinstance(raw, list): + out: list[str] = [] + for x in raw: + s = str(x).strip() + if s: + out.append(s) + # 保序去重 + seen: set[str] = set() + deduped: list[str] = [] + for s in out: + if s not in seen: + seen.add(s) + deduped.append(s) + return deduped + return [] + + +async def get_chapter_for_eval_trace( + db: AsyncSession, *, user_id: str, chapter_id: str +) -> Chapter | None: + stmt = ( + select(Chapter) + .where(Chapter.id == chapter_id, Chapter.user_id == user_id) + .options(joinedload(Chapter.current_evidence_snapshot)) + ) + result = await db.execute(stmt) + return result.unique().scalar_one_or_none() + + +async def get_story_for_eval_trace( + db: AsyncSession, *, user_id: str, story_id: str +) -> Story | None: + stmt = ( + select(Story) + .where(Story.id == story_id, Story.user_id == user_id) + .options(joinedload(Story.evidence_links)) + ) + result = await db.execute(stmt) + return result.unique().scalar_one_or_none() + + +async def list_chapter_ids_for_story( + db: AsyncSession, *, user_id: str, story_id: str +) -> list[str]: + stmt = ( + select(ChapterStoryLink.chapter_id) + .join(Chapter, ChapterStoryLink.chapter_id == Chapter.id) + .where(ChapterStoryLink.story_id == story_id, Chapter.user_id == user_id) + ) + result = await db.execute(stmt) + return list(result.scalars().all()) + + +async def fetch_segments_for_user( + db: AsyncSession, *, user_id: str, segment_ids: list[str] +) -> list[Segment]: + if not segment_ids: + return [] + stmt = ( + select(Segment) + .join(Conversation, Segment.conversation_id == Conversation.id) + .where( + Segment.id.in_(segment_ids), + Conversation.user_id == user_id, + Conversation.deleted_at.is_(None), + ) + .order_by(Segment.created_at) + ) + result = await db.execute(stmt) + rows = list(result.scalars().all()) + order = {sid: i for i, sid in enumerate(segment_ids)} + return sorted(rows, key=lambda s: order.get(s.id, 9999)) + + +async def fetch_turn_refs_for_segments( + db: AsyncSession, *, user_id: str, segment_ids: list[str] +) -> dict[str, dict[str, str | None]]: + """ + segment_id -> { user_message_id, assistant_message_id }（按 message created_at 取首条配对）。 + """ + if not segment_ids: + return {} + human_stmt = ( + select(ConversationMessage.segment_id, ConversationMessage.id) + .join(Conversation, ConversationMessage.conversation_id == Conversation.id) + .where( + ConversationMessage.segment_id.in_(segment_ids), + ConversationMessage.role == "human", + Conversation.user_id == user_id, + Conversation.deleted_at.is_(None), + ) + .order_by(ConversationMessage.created_at) + ) + ai_stmt = ( + select(ConversationMessage.segment_id, ConversationMessage.id) + .join(Conversation, ConversationMessage.conversation_id == Conversation.id) + .where( + ConversationMessage.segment_id.in_(segment_ids), + ConversationMessage.role == "ai", + Conversation.user_id == user_id, + Conversation.deleted_at.is_(None), + ) + .order_by(ConversationMessage.created_at) + ) + h_result = await db.execute(human_stmt) + u_map: dict[str, str] = {} + for seg_id, mid in h_result.all(): + if seg_id and str(seg_id) not in u_map: + u_map[str(seg_id)] = str(mid) + a_result = await db.execute(ai_stmt) + a_map: dict[str, str] = {} + for seg_id, mid in a_result.all(): + if seg_id and str(seg_id) not in a_map: + a_map[str(seg_id)] = str(mid) + out: dict[str, dict[str, str | None]] = {} + for sid in segment_ids: + ss = str(sid) + out[ss] = { + "user_message_id": u_map.get(ss), + "assistant_message_id": a_map.get(ss), + } + return out + + +async def fetch_ai_messages_for_segments( + db: AsyncSession, *, user_id: str, segment_ids: list[str] +) -> dict[str, str]: + """segment_id -> AI 消息正文（优先 durable log）。""" + if not segment_ids: + return {} + stmt = ( + select(ConversationMessage.segment_id, ConversationMessage.content) + .join(Conversation, ConversationMessage.conversation_id == Conversation.id) + .where( + ConversationMessage.segment_id.in_(segment_ids), + ConversationMessage.role == "ai", + Conversation.user_id == user_id, + Conversation.deleted_at.is_(None), + ) + .order_by(ConversationMessage.created_at) + ) + result = await db.execute(stmt) + out: dict[str, str] = {} + for seg_id, content in result.all(): + if seg_id and seg_id not in out: + out[str(seg_id)] = str(content or "") + return out + + +async def fetch_memory_closure_for_conversations( + db: AsyncSession, *, user_id: str, conversation_ids: list[str] +) -> tuple[list[str], list[str], list[str], list[str]]: + """ + 返回 (chunk_ids, fact_ids, timeline_event_ids, summary_ids)，均限定 user_id。 + 路径：MemorySource(conversation_id) -> chunks；facts by source_chunk_id； + timeline by memory_source_id；summaries 仅 rolling + 与会话 chunk 有交集的（轻量近似）。 + """ + if not conversation_ids: + return [], [], [], [] + conv_set = list({c for c in conversation_ids if c}) + + src_stmt = select(MemorySource).where( + MemorySource.user_id == user_id, + MemorySource.conversation_id.in_(conv_set), + ) + src_result = await db.execute(src_stmt) + sources = list(src_result.scalars().all()) + source_ids = [s.id for s in sources] + if not source_ids: + return [], [], [], [] + + ch_stmt = select(MemoryChunk).where( + MemoryChunk.user_id == user_id, + MemoryChunk.source_id.in_(source_ids), + MemoryChunk.is_excluded.is_(False), + ) + ch_result = await db.execute(ch_stmt) + chunks = list(ch_result.scalars().all()) + chunk_ids = [c.id for c in chunks] + if not chunk_ids: + fact_rows: list[MemoryFact] = [] + else: + f_stmt = select(MemoryFact).where( + MemoryFact.user_id == user_id, + MemoryFact.source_chunk_id.in_(chunk_ids), + or_(MemoryFact.status.is_(None), MemoryFact.status != "stale"), + ) + f_result = await db.execute(f_stmt) + fact_rows = list(f_result.scalars().all()) + fact_ids = [f.id for f in fact_rows] + + te_stmt = select(TimelineEvent).where( + TimelineEvent.user_id == user_id, + TimelineEvent.memory_source_id.in_(source_ids), + ) + te_result = await db.execute(te_stmt) + ev_rows = list(te_result.scalars().all()) + timeline_ids = [e.id for e in ev_rows] + + sum_stmt = ( + select(MemorySummary) + .where(MemorySummary.user_id == user_id) + .order_by(MemorySummary.updated_at.desc()) + .limit(12) + ) + sum_result = await db.execute(sum_stmt) + summaries = list(sum_result.scalars().all()) + chunk_set = set(chunk_ids) + summary_ids: list[str] = [] + for sm in summaries: + if sm.summary_type == "rolling": + summary_ids.append(sm.id) + continue + scids = sm.source_chunk_ids or [] + if isinstance(scids, list) and chunk_set.intersection({str(x) for x in scids}): + summary_ids.append(sm.id) + + return chunk_ids, fact_ids, timeline_ids, summary_ids + + +async def load_chunks_by_ids( + db: AsyncSession, *, user_id: str, chunk_ids: list[str] +) -> list[MemoryChunk]: + if not chunk_ids: + return [] + stmt = select(MemoryChunk).where( + MemoryChunk.user_id == user_id, + MemoryChunk.id.in_(chunk_ids), + MemoryChunk.is_excluded.is_(False), + ) + result = await db.execute(stmt) + return list(result.scalars().all()) + + +async def load_facts_by_ids( + db: AsyncSession, *, user_id: str, fact_ids: list[str] +) -> list[MemoryFact]: + if not fact_ids: + return [] + stmt = select(MemoryFact).where( + MemoryFact.user_id == user_id, + MemoryFact.id.in_(fact_ids), + or_(MemoryFact.status.is_(None), MemoryFact.status != "stale"), + ) + result = await db.execute(stmt) + return list(result.scalars().all()) + + +async def load_timeline_by_ids( + db: AsyncSession, *, user_id: str, event_ids: list[str] +) -> list[TimelineEvent]: + if not event_ids: + return [] + stmt = select(TimelineEvent).where( + TimelineEvent.user_id == user_id, + TimelineEvent.id.in_(event_ids), + ) + result = await db.execute(stmt) + return list(result.scalars().all()) + + +async def load_summaries_by_ids( + db: AsyncSession, *, user_id: str, summary_ids: list[str] +) -> list[MemorySummary]: + if not summary_ids: + return [] + stmt = select(MemorySummary).where( + MemorySummary.user_id == user_id, + MemorySummary.id.in_(summary_ids), + ) + result = await db.execute(stmt) + return list(result.scalars().all()) + + +def story_link_ids_by_type(links: list[StoryEvidenceLink]) -> tuple[ + list[str], list[str], list[str], list[str] +]: + chunks: list[str] = [] + facts: list[str] = [] + timelines: list[str] = [] + summaries: list[str] = [] + for ln in links: + et = (ln.evidence_type or "").strip() + eid = (ln.evidence_id or "").strip() + if not eid: + continue + if et == "chunk": + chunks.append(eid) + elif et == "fact": + facts.append(eid) + elif et == "timeline_event": + timelines.append(eid) + elif et == "summary": + summaries.append(eid) + return chunks, facts, timelines, summaries diff --git a/api/app/features/evaluation/eval_trace_schemas.py b/api/app/features/evaluation/eval_trace_schemas.py new file mode 100644 index 0000000..39c88f5 --- /dev/null +++ b/api/app/features/evaluation/eval_trace_schemas.py @@ -0,0 +1,65 @@ +"""可追溯评测：Chapter/Story 证据闭包模型（strict / partial / fallback）。""" + +from __future__ import annotations + +from typing import Literal + +from pydantic import BaseModel, Field + +LineageTier = Literal["strict", "partial", "fallback"] + + +class EvidenceFormatMeta(BaseModel): + """Prompt 裁剪与可审计元数据（formatter 产出）。""" + + truncated: bool = False + dropped_sections: list[str] = Field(default_factory=list) + included_token_estimate: int = 0 + transcript_chars_included: int = 0 + structured_evidence_chars_included: int = 0 + + +class ChapterEvidenceBundle(BaseModel): + """章节评测证据闭包（Phase A：source_segments 为主链，标记 partial）。""" + + user_id: str + chapter_id: str + segment_ids: list[str] = Field(default_factory=list) + conversation_ids: list[str] = Field(default_factory=list) + memory_chunk_ids: list[str] = Field(default_factory=list) + memory_fact_ids: list[str] = Field(default_factory=list) + timeline_event_ids: list[str] = Field(default_factory=list) + summary_ids: list[str] = Field(default_factory=list) + lineage_tier: LineageTier + notes: list[str] = Field(default_factory=list) + augmented_with_chapter_context: bool = False + # DialogueLineage.model_dump() — user/assistant message ids (multi-turn) + dialogue_lineage: dict | None = None + + +class StoryEvidenceBundle(BaseModel): + """故事评测证据闭包：StoryEvidenceLink 优先，缺失则回退 chapter source_segments。""" + + user_id: str + story_id: str + segment_ids: list[str] = Field(default_factory=list) + conversation_ids: list[str] = Field(default_factory=list) + memory_chunk_ids: list[str] = Field(default_factory=list) + memory_fact_ids: list[str] = Field(default_factory=list) + timeline_event_ids: list[str] = Field(default_factory=list) + summary_ids: list[str] = Field(default_factory=list) + lineage_tier: LineageTier + notes: list[str] = Field(default_factory=list) + augmented_with_chapter_context: bool = False + story_link_evidence_count: int = 0 + fallback_chapter_ids: list[str] = Field(default_factory=list) + dialogue_lineage: dict | None = None + + +class FormattedMemoirEvidence(BaseModel): + """供 `EvalJudgeService` 组 prompt 的格式化结果 + 汇总说明。""" + + source_transcript: str = "" + structured_evidence: str = "" + format_meta: EvidenceFormatMeta = Field(default_factory=EvidenceFormatMeta) + evidence_summary: str = "" diff --git a/api/app/features/evaluation/eval_trace_service.py b/api/app/features/evaluation/eval_trace_service.py new file mode 100644 index 0000000..c886614 --- /dev/null +++ b/api/app/features/evaluation/eval_trace_service.py @@ -0,0 +1,516 @@ +"""组装 Chapter/Story 评测证据闭包并格式化为评审输入。""" + +from __future__ import annotations + +from typing import Literal + +from sqlalchemy.ext.asyncio import AsyncSession + +from app.features.conversation import repo as conversation_repo +from app.features.conversation.lineage_schemas import aggregate_lineage_from_segments +from app.features.evaluation.eval_trace_format import ( + build_segment_transcript, + format_chapter_for_judge, + format_story_for_judge, +) +from app.features.evaluation.eval_trace_repo import ( + fetch_ai_messages_for_segments, + fetch_memory_closure_for_conversations, + fetch_segments_for_user, + get_chapter_for_eval_trace, + get_story_for_eval_trace, + list_chapter_ids_for_story, + load_chunks_by_ids, + load_facts_by_ids, + load_summaries_by_ids, + load_timeline_by_ids, + normalize_source_segment_ids, + story_link_ids_by_type, +) +from app.features.evaluation.eval_trace_schemas import ( + ChapterEvidenceBundle, + FormattedMemoirEvidence, + StoryEvidenceBundle, +) +from app.features.memoir.chapter_evidence_snapshot import ( + EVIDENCE_SNAPSHOT_SCHEMA_VERSION, +) +from app.features.memoir.models import Chapter +from app.features.story.models import Story, StoryVersion + +_MAX_EVIDENCE_CONVERSATIONS = 8 +_MAX_EVIDENCE_TRANSCRIPT_CHARS = 16_000 + + +def _segments_in_order(segments: list, segment_ids: list[str]) -> list: + order = {str(sid): i for i, sid in enumerate(segment_ids)} + return sorted(segments, key=lambda s: order.get(str(s.id), 9999)) + + +def _trim_fallback_transcript(text: str) -> str: + s = (text or "").strip() + if len(s) <= _MAX_EVIDENCE_TRANSCRIPT_CHARS: + return s + return f"{s[:_MAX_EVIDENCE_TRANSCRIPT_CHARS]}\n\n…（访谈证据已截断）" + + +async def fallback_user_transcript_evidence(db: AsyncSession, user_id: str) -> str: + """legacy：最近若干会话全文（仅作 fallback，调用方须声明 tier=fallback）。""" + conversations = await conversation_repo.get_user_conversations(user_id, db) + if not conversations: + return "" + parts: list[str] = [] + for conv in reversed(conversations[:_MAX_EVIDENCE_CONVERSATIONS]): + rows = await conversation_repo.get_conversation_messages(str(conv.id), db) + blocks: list[str] = [] + for row in rows: + role = str(row.role or "").lower() + body = (row.content or "").strip() + if not body: + continue + label = "用户" if role == "human" else "AI" + blocks.append(f"{label}: {body}") + transcript = "\n\n".join(blocks) + if transcript: + parts.append(f"## 会话 {str(conv.id)}\n{transcript}") + return _trim_fallback_transcript("\n\n".join(parts)) + + +class EvalTraceService: + def __init__(self, db: AsyncSession) -> None: + self._db = db + + async def _story_dialogue_lineage( + self, + st: Story, + segments: list, + segment_ids_ordered: list[str], + ) -> dict | None: + if getattr(st, "current_version_id", None): + ver = await self._db.get(StoryVersion, st.current_version_id) + if ver and isinstance(getattr(ver, "lineage_json", None), dict): + lj = ver.lineage_json + if lj.get("turns"): + return lj + if segments and segment_ids_ordered: + ordered = _segments_in_order(segments, segment_ids_ordered) + conv_ids = sorted( + {str(s.conversation_id) for s in segments if s.conversation_id} + ) + return aggregate_lineage_from_segments( + ordered, conversation_id_fallback=conv_ids[0] if conv_ids else None + ) + return None + + def _chapter_closure_tier( + self, + *, + segment_ids_resolved: list[str], + chunk_ids: list[str], + fact_ids: list[str], + tl_ids: list[str], + sum_ids: list[str], + ) -> Literal["strict", "partial", "fallback"]: + has_seg = bool(segment_ids_resolved) + has_mem = bool(chunk_ids or fact_ids or tl_ids or sum_ids) + if has_seg and has_mem: + return "strict" + if has_seg: + return "partial" + if has_mem: + return "partial" + return "fallback" + + async def build_chapter_bundle(self, user_id: str, chapter: Chapter) -> ChapterEvidenceBundle: + notes: list[str] = [] + live_segment_ids = normalize_source_segment_ids( + getattr(chapter, "source_segments", None) + ) + + row = getattr(chapter, "current_evidence_snapshot", None) + row_has_closure = bool( + (row and (row.segment_ids or [])) + or (row and (row.memory_chunk_ids or row.memory_fact_ids or row.timeline_event_ids or row.summary_ids)) + ) + if ( + row is not None + and str(row.user_id) == str(user_id) + and str(row.chapter_id) == str(chapter.id) + and int(row.schema_version or 0) == EVIDENCE_SNAPSHOT_SCHEMA_VERSION + and row_has_closure + ): + segment_ids = [ + str(x) for x in (row.segment_ids or []) if str(x).strip() + ] + conv_ids = sorted( + {str(x) for x in (row.conversation_ids or []) if str(x).strip()} + ) + chunk_ids = [ + str(x) for x in (row.memory_chunk_ids or []) if str(x).strip() + ] + fact_ids = [str(x) for x in (row.memory_fact_ids or []) if str(x).strip()] + tl_ids = [ + str(x) for x in (row.timeline_event_ids or []) if str(x).strip() + ] + sum_ids = [str(x) for x in (row.summary_ids or []) if str(x).strip()] + notes.extend([str(x) for x in (row.notes or []) if x]) + notes.append("evidence_from_chapter_evidence_snapshot_table") + tier = self._chapter_closure_tier( + segment_ids_resolved=segment_ids, + chunk_ids=chunk_ids, + fact_ids=fact_ids, + tl_ids=tl_ids, + sum_ids=sum_ids, + ) + if live_segment_ids and set(live_segment_ids) != set(segment_ids): + notes.append("live_source_segments_differ_from_snapshot_reconcile_in_pipeline") + dlg = getattr(row, "message_lineage_json", None) + return ChapterEvidenceBundle( + user_id=user_id, + chapter_id=str(chapter.id), + segment_ids=segment_ids, + conversation_ids=conv_ids, + memory_chunk_ids=chunk_ids, + memory_fact_ids=fact_ids, + timeline_event_ids=tl_ids, + summary_ids=sum_ids, + lineage_tier=tier, + notes=notes, + dialogue_lineage=dlg if isinstance(dlg, dict) else None, + ) + + snap = getattr(chapter, "evidence_bundle_json", None) + snap_uid = str(snap.get("user_id") or "") if isinstance(snap, dict) else "" + snap_has_closure = bool( + (isinstance(snap, dict) and (snap.get("segment_ids") or [])) + or ( + isinstance(snap, dict) + and ( + snap.get("memory_chunk_ids") + or snap.get("memory_fact_ids") + or snap.get("timeline_event_ids") + or snap.get("summary_ids") + ) + ) + ) + use_snap = ( + isinstance(snap, dict) + and int(snap.get("schema_version") or 0) == EVIDENCE_SNAPSHOT_SCHEMA_VERSION + and str(snap.get("chapter_id") or "") == str(chapter.id) + and (not snap_uid or snap_uid == str(user_id)) + and snap_has_closure + ) + + if use_snap and isinstance(snap, dict): + segment_ids = [str(x) for x in (snap.get("segment_ids") or []) if str(x).strip()] + conv_ids = sorted( + {str(x) for x in (snap.get("conversation_ids") or []) if str(x).strip()} + ) + chunk_ids = [str(x) for x in (snap.get("memory_chunk_ids") or []) if str(x).strip()] + fact_ids = [str(x) for x in (snap.get("memory_fact_ids") or []) if str(x).strip()] + tl_ids = [str(x) for x in (snap.get("timeline_event_ids") or []) if str(x).strip()] + sum_ids = [str(x) for x in (snap.get("summary_ids") or []) if str(x).strip()] + notes.extend([str(x) for x in (snap.get("notes") or []) if x]) + notes.append("evidence_from_chapter_evidence_bundle_json_column") + tier = self._chapter_closure_tier( + segment_ids_resolved=segment_ids, + chunk_ids=chunk_ids, + fact_ids=fact_ids, + tl_ids=tl_ids, + sum_ids=sum_ids, + ) + if live_segment_ids and set(live_segment_ids) != set(segment_ids): + notes.append("live_source_segments_differ_from_snapshot_reconcile_in_pipeline") + snap_dlg = snap.get("message_lineage_json") if isinstance(snap, dict) else None + return ChapterEvidenceBundle( + user_id=user_id, + chapter_id=str(chapter.id), + segment_ids=segment_ids, + conversation_ids=conv_ids, + memory_chunk_ids=chunk_ids, + memory_fact_ids=fact_ids, + timeline_event_ids=tl_ids, + summary_ids=sum_ids, + lineage_tier=tier, + notes=notes, + dialogue_lineage=snap_dlg if isinstance(snap_dlg, dict) else None, + ) + + segment_ids = live_segment_ids + if not segment_ids: + notes.append("no_source_segments") + notes.append("fallback_lineage_transcript_pending") + return ChapterEvidenceBundle( + user_id=user_id, + chapter_id=str(chapter.id), + segment_ids=[], + conversation_ids=[], + lineage_tier="fallback", + notes=notes, + dialogue_lineage=None, + ) + + segments = await fetch_segments_for_user( + self._db, user_id=user_id, segment_ids=segment_ids + ) + resolved_seg_ids = [s.id for s in segments] or segment_ids + if len(segments) < len(segment_ids): + notes.append("some_segments_missing_or_foreign_user") + conv_ids = sorted({str(s.conversation_id) for s in segments if s.conversation_id}) + chunk_ids, fact_ids, tl_ids, sum_ids = await fetch_memory_closure_for_conversations( + self._db, user_id=user_id, conversation_ids=conv_ids + ) + tier = self._chapter_closure_tier( + segment_ids_resolved=resolved_seg_ids, + chunk_ids=chunk_ids, + fact_ids=fact_ids, + tl_ids=tl_ids, + sum_ids=sum_ids, + ) + if tier == "partial": + notes.append( + "chapter_source_segments_union_semantics=partial_lineage_until_snapshot" + ) + elif tier == "strict": + notes.append("chapter_lineage_strict_segments_plus_memory_closure") + segs_ord = _segments_in_order(segments, resolved_seg_ids) + dlg_live = aggregate_lineage_from_segments( + segs_ord, conversation_id_fallback=conv_ids[0] if conv_ids else None + ) + return ChapterEvidenceBundle( + user_id=user_id, + chapter_id=str(chapter.id), + segment_ids=resolved_seg_ids, + conversation_ids=conv_ids, + memory_chunk_ids=chunk_ids, + memory_fact_ids=fact_ids, + timeline_event_ids=tl_ids, + summary_ids=sum_ids, + lineage_tier=tier, + notes=notes, + dialogue_lineage=dlg_live, + ) + + async def format_chapter_bundle( + self, bundle: ChapterEvidenceBundle + ) -> tuple[FormattedMemoirEvidence, ChapterEvidenceBundle]: + """若 tier=fallback，调用方应先将要并入 transcripts 写入 session；此处只负责 segment 路径。""" + if bundle.lineage_tier == "fallback": + ft = await fallback_user_transcript_evidence(self._db, bundle.user_id) + notes = list(bundle.notes) + notes.append("used_legacy_recent_conversations_transcript") + bundle = bundle.model_copy(update={"notes": notes}) + formatted = format_chapter_for_judge( + bundle, + transcript=ft, + chunks=[], + facts=[], + events=[], + summaries=[], + ) + return formatted, bundle + + segs = await fetch_segments_for_user( + self._db, user_id=bundle.user_id, segment_ids=bundle.segment_ids + ) + ai_map = await fetch_ai_messages_for_segments( + self._db, user_id=bundle.user_id, segment_ids=[s.id for s in segs] + ) + transcript = build_segment_transcript(segs, ai_map) + chunks = await load_chunks_by_ids( + self._db, user_id=bundle.user_id, chunk_ids=bundle.memory_chunk_ids + ) + facts = await load_facts_by_ids( + self._db, user_id=bundle.user_id, fact_ids=bundle.memory_fact_ids + ) + events = await load_timeline_by_ids( + self._db, user_id=bundle.user_id, event_ids=bundle.timeline_event_ids + ) + summaries = await load_summaries_by_ids( + self._db, user_id=bundle.user_id, summary_ids=bundle.summary_ids + ) + formatted = format_chapter_for_judge( + bundle, + transcript=transcript, + chunks=chunks, + facts=facts, + events=events, + summaries=summaries, + ) + return formatted, bundle + + async def build_story_bundle(self, user_id: str, story_id: str) -> StoryEvidenceBundle: + st = await get_story_for_eval_trace(self._db, user_id=user_id, story_id=story_id) + if not st: + return StoryEvidenceBundle( + user_id=user_id, + story_id=story_id, + lineage_tier="fallback", + notes=["story_not_found"], + dialogue_lineage=None, + ) + + links = list(st.evidence_links or []) + lc, lf, lt, ls = story_link_ids_by_type(links) + notes: list[str] = [] + chapter_ids = await list_chapter_ids_for_story( + self._db, user_id=user_id, story_id=str(st.id) + ) + + if lc or lf or lt or ls: + # 结构化以 link 为准；会话级 transcript 尝试从挂靠章节 source_segments 收缩 + seg_ids: list[str] = [] + conv_ids: list[str] = [] + for cid in chapter_ids: + ch = await get_chapter_for_eval_trace( + self._db, user_id=user_id, chapter_id=cid + ) + if not ch: + continue + seg_ids.extend(normalize_source_segment_ids(ch.source_segments)) + # 保序去重 + seen_s: set[str] = set() + dedup_seg: list[str] = [] + for s in seg_ids: + if s not in seen_s: + seen_s.add(s) + dedup_seg.append(s) + segments = await fetch_segments_for_user( + self._db, user_id=user_id, segment_ids=dedup_seg + ) + conv_ids = sorted({str(s.conversation_id) for s in segments if s.conversation_id}) + if dedup_seg and not segments: + notes.append("chapter_segment_ids_unresolved") + if conv_ids: + notes.append("transcript_from_chapter_source_segments") + else: + notes.append("no_chapter_segments_for_transcript_context") + bound_transcript = bool(segments) + story_tier: Literal["strict", "partial", "fallback"] = "strict" + if (lc or lf or lt or ls) and not bound_transcript: + notes.append("structured_evidence_without_bound_transcript") + story_tier = "partial" + dlg = await self._story_dialogue_lineage(st, segments, dedup_seg) + return StoryEvidenceBundle( + user_id=user_id, + story_id=str(st.id), + segment_ids=[s.id for s in segments] or dedup_seg, + conversation_ids=conv_ids, + memory_chunk_ids=lc, + memory_fact_ids=lf, + timeline_event_ids=lt, + summary_ids=ls, + lineage_tier=story_tier, + notes=notes, + augmented_with_chapter_context=bool(chapter_ids), + story_link_evidence_count=len(links), + fallback_chapter_ids=chapter_ids, + dialogue_lineage=dlg, + ) + + # 无 StoryEvidenceLink：由章节 source_segments 推导 partial；再不行则 fallback + seg_ids = [] + conv_ids: list[str] = [] + for cid in chapter_ids: + ch = await get_chapter_for_eval_trace( + self._db, user_id=user_id, chapter_id=cid + ) + if not ch: + continue + seg_ids.extend(normalize_source_segment_ids(ch.source_segments)) + seen_s = set() + dedup_seg = [] + for s in seg_ids: + if s not in seen_s: + seen_s.add(s) + dedup_seg.append(s) + if dedup_seg: + segments = await fetch_segments_for_user( + self._db, user_id=user_id, segment_ids=dedup_seg + ) + conv_ids = sorted({str(s.conversation_id) for s in segments if s.conversation_id}) + chunk_ids, fact_ids, tl_ids, sum_ids = ( + await fetch_memory_closure_for_conversations( + self._db, user_id=user_id, conversation_ids=conv_ids + ) + ) + notes.append("fallback_lineage_no_story_evidence_links") + notes.append("augmented_with_chapter_context") + dlg2 = await self._story_dialogue_lineage(st, segments, dedup_seg) + return StoryEvidenceBundle( + user_id=user_id, + story_id=str(st.id), + segment_ids=[s.id for s in segments] or dedup_seg, + conversation_ids=conv_ids, + memory_chunk_ids=chunk_ids, + memory_fact_ids=fact_ids, + timeline_event_ids=tl_ids, + summary_ids=sum_ids, + lineage_tier="partial", + notes=notes, + augmented_with_chapter_context=True, + story_link_evidence_count=0, + fallback_chapter_ids=chapter_ids, + dialogue_lineage=dlg2, + ) + + notes.append("no_story_evidence_links_and_no_chapter_segments") + notes.append("fallback_lineage_transcript_pending") + dlg3 = await self._story_dialogue_lineage(st, [], []) + return StoryEvidenceBundle( + user_id=user_id, + story_id=str(st.id), + lineage_tier="fallback", + notes=notes, + story_link_evidence_count=0, + fallback_chapter_ids=chapter_ids, + dialogue_lineage=dlg3, + ) + + async def format_story_bundle( + self, bundle: StoryEvidenceBundle + ) -> tuple[FormattedMemoirEvidence, StoryEvidenceBundle]: + if bundle.lineage_tier == "fallback": + ft = await fallback_user_transcript_evidence(self._db, bundle.user_id) + notes = list(bundle.notes) + notes.append("used_legacy_recent_conversations_transcript") + bundle = bundle.model_copy(update={"notes": notes}) + formatted = format_story_for_judge( + bundle, + transcript=ft, + chunks=[], + facts=[], + events=[], + summaries=[], + ) + return formatted, bundle + + segs = await fetch_segments_for_user( + self._db, user_id=bundle.user_id, segment_ids=bundle.segment_ids + ) + ai_map = await fetch_ai_messages_for_segments( + self._db, user_id=bundle.user_id, segment_ids=[s.id for s in segs] + ) + transcript = build_segment_transcript(segs, ai_map) + + chunks = await load_chunks_by_ids( + self._db, user_id=bundle.user_id, chunk_ids=bundle.memory_chunk_ids + ) + facts = await load_facts_by_ids( + self._db, user_id=bundle.user_id, fact_ids=bundle.memory_fact_ids + ) + events = await load_timeline_by_ids( + self._db, user_id=bundle.user_id, event_ids=bundle.timeline_event_ids + ) + summaries = await load_summaries_by_ids( + self._db, user_id=bundle.user_id, summary_ids=bundle.summary_ids + ) + formatted = format_story_for_judge( + bundle, + transcript=transcript, + chunks=chunks, + facts=facts, + events=events, + summaries=summaries, + ) + return formatted, bundle diff --git a/api/app/features/evaluation/execution_service.py b/api/app/features/evaluation/execution_service.py index d1a196e..b505e9c 100644 --- a/api/app/features/evaluation/execution_service.py +++ b/api/app/features/evaluation/execution_service.py @@ -16,6 +16,7 @@ from app.features.evaluation.candidate_runner import ( EvalCandidateRunner, simple_memoir_from_transcript, ) +from app.features.evaluation.eval_trace_service import EvalTraceService from app.features.evaluation.gate_report_service import gate_result_to_details from app.features.evaluation.gating_service import compute_gate from app.features.evaluation.judge_service import EvalJudgeService @@ -30,7 +31,6 @@ logger = get_logger(__name__) _MAX_JUDGE_MARKDOWN_CHARS = 20_000 _MAX_EVAL_CHAPTERS = 30 _MAX_EVAL_STORIES = 40 -_MAX_EVIDENCE_CONVERSATIONS = 8 _MAX_EVIDENCE_TRANSCRIPT_CHARS = 16_000 @@ -74,43 +74,6 @@ def _trim_evidence_text(text: str, max_chars: int = _MAX_EVIDENCE_TRANSCRIPT_CHA return f"{s[:max_chars]}\n\n…（访谈证据已截断）" -def _dialogue_transcript_from_pairs(pairs: list[tuple[str, str]]) -> str: - parts: list[str] = [] - for role, content in pairs: - body = (content or "").strip() - if not body: - continue - label = "用户" if role == "human" else "AI" - out = assistant_text_for_eval_display(body) if role != "human" else body - parts.append(f"{label}: {out}") - return "\n\n".join(parts) - - -async def _conversation_transcript_for_eval( - db: AsyncSession, conversation_id: str -) -> str: - from app.features.conversation import repo as conversation_repo - - rows = await conversation_repo.get_conversation_messages(conversation_id, db) - return _dialogue_transcript_from_pairs( - [(str(row.role or "").lower(), str(row.content or "")) for row in rows] - ) - - -async def _user_transcript_evidence(db: AsyncSession, user_id: str) -> str: - from app.features.conversation import repo as conversation_repo - - conversations = await conversation_repo.get_user_conversations(user_id, db) - if not conversations: - return "" - parts: list[str] = [] - for conv in reversed(conversations[:_MAX_EVIDENCE_CONVERSATIONS]): - transcript = await _conversation_transcript_for_eval(db, str(conv.id)) - if transcript: - parts.append(f"## 会话 {str(conv.id)}\n{transcript}") - return _trim_evidence_text("\n\n".join(parts)) - - async def execute_eval_run( db: AsyncSession, *, @@ -118,6 +81,14 @@ async def execute_eval_run( case: EvalCase, version: EvalVersion, ) -> None: + fresh_run = await db.get(EvalRun, run.id) + if not fresh_run: + return + if (fresh_run.status or "").lower() == "completed": + logger.info("eval run skip already completed run_id={}", fresh_run.id) + return + run = fresh_run + if not settings.eval_execution_enabled: await eval_repo.update_run( db, @@ -227,34 +198,39 @@ async def execute_eval_run( memoir_md = simple_memoir_from_transcript(utterances, replies) source_transcript = _trim_evidence_text(full_transcript) reference_memoir = (case.reference_memoir_markdown or "").strip() + synthetic_notes = ( + "本项为 replay 合成的短 memoir：证据闭包仅为重放对话 transcript（无 library artifact lineage）。" + f" turns={len(utterances)}" + ) mem_out = await judge.judge_memoir( memoir_markdown=memoir_md, source_transcript=source_transcript, + structured_evidence=( + "（结构化记忆证据：自动化 replay 路径未绑定用户 memory chunk/fact/timeline/summary。）" + ), reference_memoir_markdown=reference_memoir, - evidence_notes="严格按文档核对真实性、覆盖率、可追溯性；以原始访谈为主，参考基线仅作辅助。", + evidence_notes=synthetic_notes, ) chapter_entries: list[dict[str, Any]] = [] story_entries: list[dict[str, Any]] = [] uid = (case.source_user_id or "").strip() - source_conversation_id = (case.source_conversation_id or "").strip() - evidence_transcript = source_transcript - if source_conversation_id: - try: - conversation_evidence = await _conversation_transcript_for_eval( - db, source_conversation_id - ) - if conversation_evidence: - evidence_transcript = _trim_evidence_text(conversation_evidence) - except Exception as e: - logger.warning("eval source conversation evidence skipped: {}", e) - elif uid: - try: - user_evidence = await _user_transcript_evidence(db, uid) - if user_evidence: - evidence_transcript = user_evidence - except Exception as e: - logger.warning("eval user transcript evidence skipped: {}", e) + trace_svc = EvalTraceService(db) + + def _library_evidence_notes( + lineage_tier: str, + evidence_summary: str, + truncated: bool, + dropped: list[str], + ) -> str: + drops = ",".join(dropped[:12]) if dropped else "" + return ( + "library artifact 评审：以证据闭包为准；若 lineage 为 fallback 或不足须保守打分。" + f" lineage_tier={lineage_tier}；summary={evidence_summary}；" + f" prompt_truncated={truncated}；dropped_sections={drops or 'none'}。" + " 单章节/单故事节选；跨篇上下文不足写入 insufficient_evidence。" + ) + if uid: from app.features.memoir.repo import get_chapters_for_memoir_list from app.features.story.repo import get_stories_for_user @@ -268,13 +244,19 @@ async def execute_eval_run( if not body: continue md = f"# 章节：{ch.title}\n\n{_clip_md_for_judge(body)}" + cb = await trace_svc.build_chapter_bundle(uid, ch) + formatted, cb2 = await trace_svc.format_chapter_bundle(cb) + fm = formatted.format_meta cj = await judge.judge_memoir( memoir_markdown=md, - source_transcript=evidence_transcript, + source_transcript=formatted.source_transcript, + structured_evidence=formatted.structured_evidence, reference_memoir_markdown=reference_memoir, - evidence_notes=( - "这是用户现有章节的严格评审；真实性、覆盖率、可追溯性必须对照原始访谈证据。" - " 评审范围：单章节节选；跨全书连贯性仅在与证据一致时评估，否则保守打分并在 insufficient_evidence 说明。" + evidence_notes=_library_evidence_notes( + cb2.lineage_tier, + formatted.evidence_summary, + fm.truncated, + fm.dropped_sections, ), ) chapter_entries.append( @@ -282,6 +264,10 @@ async def execute_eval_run( "id": ch.id, "title": ch.title, "order_index": ch.order_index, + "lineage_tier": cb2.lineage_tier, + "evidence_summary": formatted.evidence_summary, + "evidence_trace": cb2.model_dump(), + "format_meta": fm.model_dump(), "judge": cj.model_dump() if cj else None, } ) @@ -295,13 +281,19 @@ async def execute_eval_run( if not body: continue md = f"# 故事：{st.title}\n\n{_clip_md_for_judge(body)}" + sb = await trace_svc.build_story_bundle(uid, str(st.id)) + formatted, sb2 = await trace_svc.format_story_bundle(sb) + fm = formatted.format_meta sj = await judge.judge_memoir( memoir_markdown=md, - source_transcript=evidence_transcript, + source_transcript=formatted.source_transcript, + structured_evidence=formatted.structured_evidence, reference_memoir_markdown=reference_memoir, - evidence_notes=( - "这是用户现有故事的严格评审；真实性、覆盖率、可追溯性必须对照原始访谈证据。" - " 评审范围：单故事节选；跨篇章关联若证据不足须保守并在 insufficient_evidence 说明。" + evidence_notes=_library_evidence_notes( + sb2.lineage_tier, + formatted.evidence_summary, + fm.truncated, + fm.dropped_sections, ), ) story_entries.append( @@ -309,6 +301,10 @@ async def execute_eval_run( "id": st.id, "title": st.title, "stage": st.stage, + "lineage_tier": sb2.lineage_tier, + "evidence_summary": formatted.evidence_summary, + "evidence_trace": sb2.model_dump(), + "format_meta": fm.model_dump(), "judge": sj.model_dump() if sj else None, } ) @@ -352,6 +348,7 @@ async def execute_eval_run( bundle: dict[str, Any] = { "conversation_judge": conv_out.model_dump() if conv_out else None, "memoir_judge": mem_out.model_dump() if mem_out else None, + "synthetic_memoir_judge": mem_out.model_dump() if mem_out else None, "chapters": chapter_entries, "stories": story_entries, "judge_meta": { @@ -359,10 +356,14 @@ async def execute_eval_run( "memoir_synthetic_ok": mem_out is not None, "memoir_synth_scores_n": len(synth_scores), "memoir_library_scores_n": len(library_scores), + "synthetic_memoir_lineage_tier": "replay_transcript_only", + "synthetic_memoir_evidence_summary": ( + f"replay_turns={len(utterances)};structured_memory=unbound" + ), "memoir_aggregate_rule": ( - "synth_plus_library_weighted_mean" + "synthetic_memoir_judge_plus_library_memoir_judge_weighted_mean" if synth_scores and library_scores - else ("synthetic_only" if synth_scores else "library_only") + else ("synthetic_memoir_only" if synth_scores else "library_memoir_only") ), }, } @@ -422,38 +423,52 @@ async def _finalize_experiment_gate(db: AsyncSession, experiment_id: str) -> Non async def execute_experiment_full(experiment_id: str) -> None: - async with AsyncSessionLocal() as db: - exp = await eval_repo.get_experiment(db, experiment_id) - if not exp: - return - await eval_repo.update_experiment(db, exp, status="running") - await db.commit() + from app.core.redis_lock import acquire_redis_lock, release_redis_lock - cases = await eval_repo.list_cases(db, str(exp.regression_set_id)) - base_v = await eval_repo.get_version(db, str(exp.baseline_version_id)) - cand_v = await eval_repo.get_version(db, str(exp.candidate_version_id)) - if base_v is None or cand_v is None: - await eval_repo.update_experiment( - db, - exp, - status="failed", - error_message="version 不存在", - completed_at=datetime.now(timezone.utc), - ) + lock_key = f"lock:eval_experiment:{experiment_id}" + lock_handle = acquire_redis_lock(lock_key, ttl_seconds=7200) + if lock_handle is None: + logger.warning( + "eval experiment already running or lock busy experiment_id={}", + experiment_id, + ) + return + + try: + async with AsyncSessionLocal() as db: + exp = await eval_repo.get_experiment(db, experiment_id) + if not exp: + return + await eval_repo.update_experiment(db, exp, status="running") await db.commit() - return - for case in cases: - for side, ver in ("baseline", base_v), ("candidate", cand_v): - run = await eval_repo.get_run(db, experiment_id, str(case.id), side) - if not run: - run = await eval_repo.create_run( - db, - experiment_id=experiment_id, - case_id=str(case.id), - side=side, - ) + cases = await eval_repo.list_cases(db, str(exp.regression_set_id)) + base_v = await eval_repo.get_version(db, str(exp.baseline_version_id)) + cand_v = await eval_repo.get_version(db, str(exp.candidate_version_id)) + if base_v is None or cand_v is None: + await eval_repo.update_experiment( + db, + exp, + status="failed", + error_message="version 不存在", + completed_at=datetime.now(timezone.utc), + ) await db.commit() - await execute_eval_run(db, run=run, case=case, version=ver) + return - await _finalize_experiment_gate(db, experiment_id) + for case in cases: + for side, ver in ("baseline", base_v), ("candidate", cand_v): + run = await eval_repo.get_run(db, experiment_id, str(case.id), side) + if not run: + run = await eval_repo.create_run( + db, + experiment_id=experiment_id, + case_id=str(case.id), + side=side, + ) + await db.commit() + await execute_eval_run(db, run=run, case=case, version=ver) + + await _finalize_experiment_gate(db, experiment_id) + finally: + release_redis_lock(lock_handle) diff --git a/api/app/features/evaluation/judge_manual_service.py b/api/app/features/evaluation/judge_manual_service.py index 8816558..d772888 100644 --- a/api/app/features/evaluation/judge_manual_service.py +++ b/api/app/features/evaluation/judge_manual_service.py @@ -15,7 +15,10 @@ from app.features.evaluation.errors import ( EvaluationBadRequestError, EvaluationNotFoundError, ) +from app.features.evaluation.eval_trace_service import EvalTraceService from app.features.evaluation.judge_service import EvalJudgeService +from app.features.evaluation.schemas import MemoirSectionBaselineOut +from app.features.evaluation.session_catalog_service import SessionCatalogService from app.features.evaluation.transcript_for_judge import ( assistant_text_for_eval_display, format_eval_turn_block, @@ -23,8 +26,6 @@ from app.features.evaluation.transcript_for_judge import ( format_session_messages_with_turn_labels, pair_session_messages_to_turns, ) -from app.features.evaluation.schemas import MemoirSectionBaselineOut -from app.features.evaluation.session_catalog_service import SessionCatalogService from app.features.evaluation.user_export_fixtures import read_user_export_fixture from app.features.memoir.repo import get_chapters_for_memoir_list from app.features.story.repo import get_stories_for_user @@ -34,8 +35,6 @@ logger = get_logger(__name__) _MAX_JUDGE_MARKDOWN_CHARS = 20_000 _MAX_EVAL_CHAPTERS = 30 _MAX_EVAL_STORIES = 40 -_MAX_EVIDENCE_CONVERSATIONS = 8 -_MAX_EVIDENCE_TRANSCRIPT_CHARS = 16_000 _PRIOR_TRANSCRIPT_MAX_CHARS = 8000 @@ -75,13 +74,6 @@ def _clip_md_for_judge(text: str, max_chars: int = _MAX_JUDGE_MARKDOWN_CHARS) -> return f"{s[:max_chars]}\n\n…（已截断供评审）" -def _trim_evidence_text(text: str, max_chars: int = _MAX_EVIDENCE_TRANSCRIPT_CHARS) -> str: - s = (text or "").strip() - if len(s) <= max_chars: - return s - return f"{s[:max_chars]}\n\n…（访谈证据已截断）" - - async def _conversation_transcript_for_manual( db: AsyncSession, conversation_id: str ) -> str: @@ -89,18 +81,6 @@ async def _conversation_transcript_for_manual( return format_session_messages_with_turn_labels(rows) -async def _user_transcript_evidence(db: AsyncSession, user_id: str) -> str: - conversations = await conversation_repo.get_user_conversations(user_id, db) - if not conversations: - return "" - parts: list[str] = [] - for conv in reversed(conversations[:_MAX_EVIDENCE_CONVERSATIONS]): - transcript = await _conversation_transcript_for_manual(db, str(conv.id)) - if transcript: - parts.append(f"## 会话 {str(conv.id)}\n{transcript}") - return _trim_evidence_text("\n\n".join(parts)) - - def _normalize_title_key(title: str) -> str: t = (title or "").strip().lower() t = re.sub(r"^#+\s*", "", t) @@ -364,7 +344,17 @@ class EvalJudgeManualService: judge_llm = get_eval_judge_langchain_llm() judge = EvalJudgeService(judge_llm) baselines = list(baseline_sections or []) - evidence_transcript = await _user_transcript_evidence(self._db, uid) + trace_svc = EvalTraceService(self._db) + + def _chapter_evidence_notes( + lineage_tier: str, evidence_summary: str, truncated: bool, dropped: list[str] + ) -> str: + drops = ",".join(dropped[:12]) if dropped else "" + return ( + "严格按文档打分；真实性、事实覆盖率、可追溯性以本章节绑定的证据闭包为准。" + f" lineage_tier={lineage_tier}；evidence_summary={evidence_summary}；" + f" prompt_truncated={truncated}；dropped_sections={drops or 'none'}" + ) chapter_results: list[dict[str, Any]] = [] try: @@ -383,12 +373,19 @@ class EvalJudgeManualService: if baseline_excerpt: md += f"## 导出基线（节选）\n\n{baseline_excerpt}\n\n" md += f"## 当前成稿\n\n{_clip_md_for_judge(body)}" + cb = await trace_svc.build_chapter_bundle(uid, ch) + formatted, cb2 = await trace_svc.format_chapter_bundle(cb) + fm = formatted.format_meta cj = await judge.judge_memoir( memoir_markdown=md, - source_transcript=evidence_transcript, + source_transcript=formatted.source_transcript, + structured_evidence=formatted.structured_evidence, reference_memoir_markdown=baseline_excerpt, - evidence_notes=( - "严格按文档打分；真实性、事实覆盖率、可追溯性必须优先对照该用户历史访谈证据。" + evidence_notes=_chapter_evidence_notes( + cb2.lineage_tier, + formatted.evidence_summary, + fm.truncated, + fm.dropped_sections, ), ) chapter_results.append( @@ -397,6 +394,10 @@ class EvalJudgeManualService: "title": ch.title, "order_index": ch.order_index, "baseline_title": bl.title if bl else None, + "lineage_tier": cb2.lineage_tier, + "evidence_summary": formatted.evidence_summary, + "evidence_trace": cb2.model_dump(), + "format_meta": fm.model_dump(), "judge": cj.model_dump() if cj else None, } ) @@ -411,11 +412,18 @@ class EvalJudgeManualService: if not body: continue md = f"# 故事：{st.title}\n\n{_clip_md_for_judge(body)}" + sb = await trace_svc.build_story_bundle(uid, str(st.id)) + formatted, sb2 = await trace_svc.format_story_bundle(sb) + fm = formatted.format_meta sj = await judge.judge_memoir( memoir_markdown=md, - source_transcript=evidence_transcript, - evidence_notes=( - "严格按文档打分；真实性、事实覆盖率、可追溯性必须优先对照该用户历史访谈证据。" + source_transcript=formatted.source_transcript, + structured_evidence=formatted.structured_evidence, + evidence_notes=_chapter_evidence_notes( + sb2.lineage_tier, + formatted.evidence_summary, + fm.truncated, + fm.dropped_sections, ), ) story_results.append( @@ -423,6 +431,10 @@ class EvalJudgeManualService: "id": st.id, "title": st.title, "stage": st.stage, + "lineage_tier": sb2.lineage_tier, + "evidence_summary": formatted.evidence_summary, + "evidence_trace": sb2.model_dump(), + "format_meta": fm.model_dump(), "judge": sj.model_dump() if sj else None, } ) diff --git a/api/app/features/evaluation/judge_service.py b/api/app/features/evaluation/judge_service.py index bc2267f..be0869f 100644 --- a/api/app/features/evaluation/judge_service.py +++ b/api/app/features/evaluation/judge_service.py @@ -91,28 +91,45 @@ def _build_memoir_judge_prompt( *, memoir_markdown: str, source_transcript: str = "", + structured_evidence: str = "", reference_memoir_markdown: str = "", evidence_notes: str = "", ) -> str: """Assemble an evidence-aware memoir judging prompt.""" source = (source_transcript or "").strip() + struct = (structured_evidence or "").strip() reference = (reference_memoir_markdown or "").strip() notes = (evidence_notes or "").strip() sections = [ MEMOIR_JUDGE_INSTRUCTIONS, "", - "【证据与输入顺序】以下区块按优先级给出：评审说明（若有）→ 原始访谈证据 → 参考基线（若有）→ 待评成稿。**真实性相关细项必须以原始访谈证据为准。**", + "【证据与输入顺序】以下区块按优先级给出：" + "评审说明（若有）→ 原始访谈/对话证据（segment 绑定）→ 结构化记忆证据（chunk/fact/timeline/summary）" + "→ 参考基线（若有）→ 待评成稿。**真实性、覆盖率、可追溯性以「artifact 绑定证据闭包」为准**；" + "若存在 `lineage_tier=fallback` 或证据不足，须保守打分并写 `insufficient_evidence`。", "", ] if notes: sections.extend(["【评审说明】", notes[:1200], ""]) if source: - sections.extend(["【原始访谈/证据】", source[:_MEMOIR_EVIDENCE_MAX], ""]) + sections.extend(["【原始访谈/对话证据】", source[:_MEMOIR_EVIDENCE_MAX], ""]) else: sections.extend( [ - "【原始访谈/证据】", - "无可用原始访谈证据。对于记忆忠实度、事实准确性、事实覆盖率、记忆可追溯性，必须保守打分，不得凭空高分。", + "【原始访谈/对话证据】", + "无可用局部对话证据。对于记忆忠实度、事实准确性、事实覆盖率、记忆可追溯性，必须保守打分，不得凭空高分。", + "", + ] + ) + if struct: + sections.extend( + ["【结构化记忆证据】", struct[:_MEMOIR_EVIDENCE_MAX], ""] + ) + else: + sections.extend( + [ + "【结构化记忆证据】", + "（本 artifact 未绑定或未解析到 chunk/fact/timeline/summary 证据。）", "", ] ) @@ -268,12 +285,14 @@ class EvalJudgeService: *, memoir_markdown: str, source_transcript: str = "", + structured_evidence: str = "", reference_memoir_markdown: str = "", evidence_notes: str = "", ) -> MemoirJudgeOutput | None: result = await self.judge_memoir_result( memoir_markdown=memoir_markdown, source_transcript=source_transcript, + structured_evidence=structured_evidence, reference_memoir_markdown=reference_memoir_markdown, evidence_notes=evidence_notes, ) @@ -284,6 +303,7 @@ class EvalJudgeService: *, memoir_markdown: str, source_transcript: str = "", + structured_evidence: str = "", reference_memoir_markdown: str = "", evidence_notes: str = "", ) -> JudgeCallResult[MemoirJudgeOutput]: @@ -292,6 +312,7 @@ class EvalJudgeService: prompt = _build_memoir_judge_prompt( memoir_markdown=memoir_markdown, source_transcript=source_transcript, + structured_evidence=structured_evidence, reference_memoir_markdown=reference_memoir_markdown, evidence_notes=evidence_notes, ) diff --git a/api/app/features/evaluation/repo.py b/api/app/features/evaluation/repo.py index 0e3c105..f4f6620 100644 --- a/api/app/features/evaluation/repo.py +++ b/api/app/features/evaluation/repo.py @@ -6,6 +6,7 @@ import uuid from typing import Any from sqlalchemy import select +from sqlalchemy.dialects.postgresql import insert as pg_insert from sqlalchemy.ext.asyncio import AsyncSession from app.features.evaluation.models import ( @@ -270,8 +271,10 @@ async def add_turn( judge_scores_json: dict[str, Any] | None, judge_rationale: str | None, ) -> EvalRunTurn: - row = EvalRunTurn( - id=_id(), + """插入或更新同 (run_id, turn_index) 的轮次，避免 Celery 重试时 UniqueViolation。""" + tid = _id() + ins = pg_insert(EvalRunTurn).values( + id=tid, run_id=run_id, turn_index=turn_index, user_utterance=user_utterance, @@ -280,8 +283,27 @@ async def add_turn( judge_scores_json=judge_scores_json, judge_rationale=judge_rationale, ) - db.add(row) + stmt = ins.on_conflict_do_update( + constraint="uq_eval_run_turn_index", + set_={ + "user_utterance": ins.excluded.user_utterance, + "assistant_reply": ins.excluded.assistant_reply, + "duration_ms": ins.excluded.duration_ms, + "judge_scores_json": ins.excluded.judge_scores_json, + "judge_rationale": ins.excluded.judge_rationale, + }, + ) + await db.execute(stmt) await db.flush() + res = await db.execute( + select(EvalRunTurn) + .where( + EvalRunTurn.run_id == run_id, + EvalRunTurn.turn_index == turn_index, + ) + .limit(1) + ) + row = res.scalar_one() return row diff --git a/api/app/features/evaluation/router.py b/api/app/features/evaluation/router.py index 0a0a69c..453582a 100644 --- a/api/app/features/evaluation/router.py +++ b/api/app/features/evaluation/router.py @@ -27,8 +27,8 @@ from app.features.evaluation.importers.user_export_markdown import ( ) from app.features.evaluation.internal_auth import InternalEvalAuth from app.features.evaluation.judge_manual_service import EvalJudgeManualService -from app.features.evaluation.presenters import case_out, run_out from app.features.evaluation.memoir_readiness_service import MemoirReadinessService +from app.features.evaluation.presenters import case_out, run_out from app.features.evaluation.replay_service import ReplayConversationService from app.features.evaluation.schemas import ( CaseCreate, @@ -46,12 +46,12 @@ from app.features.evaluation.schemas import ( ManualJudgeConversationStreamBody, ManualJudgeMemoirBody, ManualJudgeMemoirOut, + MemoirPhase1ReadyOut, MemoirSectionBaselineOut, RegressionSetCreate, RegressionSetOut, ReplayBootstrapBody, ReplayBootstrapOut, - MemoirPhase1ReadyOut, ReplayConversationBody, ReplayConversationOut, SessionDialogueOut, diff --git a/api/app/features/evaluation/rubrics/memoir_v1.py b/api/app/features/evaluation/rubrics/memoir_v1.py index 90f4bef..3945e87 100644 --- a/api/app/features/evaluation/rubrics/memoir_v1.py +++ b/api/app/features/evaluation/rubrics/memoir_v1.py @@ -6,7 +6,7 @@ _MEMOIR_CHARTER = """ ## 评审总原则（必须遵守） -- **证据层级**：以【原始访谈/证据】为最高优先级判定真实性与覆盖；【参考基线/导出成稿】仅辅助对照，不得以基线对错代替证据对错。 +- **证据层级**：以【原始访谈/对话证据】+【结构化记忆证据】共同为 artifact 绑定闭包；二者缺一不可时不等于「无证据」——须区分「证据未进 prompt（见评审说明中的截断）」与「数据库确无 lineage」。【参考基线/导出成稿】仅辅助对照，不得以基线对错代替证据对错。 - 只依据输入中可核对的文字评分；不得臆测用户人生经历。 - **缺少原文证据不等于「写得好」**：无证据或证据极短时，`mem_fidelity`、`mem_factual_coverage`、`mem_traceability` 等须保守，并在 `insufficient_evidence` 说明。 - **文笔与结构不得抵消事实问题**：`lang_*`、`narr_*` 高分不得与明显编造、张冠李戴并存。 @@ -76,6 +76,7 @@ total_score, major_strengths, major_issues, insufficient_evidence, evidence_refs, confidence, rationale `evidence_refs`：数组，每项为对象，字段 `dimension`（上列 mem_* / info_* 等英文名之一）、`turn_index`（无对话轮次用 -1）、`snippet`（≤120 字引文或定位）。 +若输入证据中包含 `dialogue_lineage.turns`，可在 `snippet` 中引用对应轮的 `user_message_id` / `assistant_message_id` 作为可机读定位（与口语引文可同时出现）。 `confidence`：0 到 1 之间小数。 diff --git a/api/app/features/memoir/chapter_evidence_snapshot.py b/api/app/features/memoir/chapter_evidence_snapshot.py new file mode 100644 index 0000000..2c29397 --- /dev/null +++ b/api/app/features/memoir/chapter_evidence_snapshot.py @@ -0,0 +1,246 @@ +"""章节证据闭包：统一计算（评测与生产共用）+ Phase C 表持久化（快照行 + chapter_evidence_links）。""" + +from __future__ import annotations + +import uuid +from datetime import datetime, timezone + +from sqlalchemy import delete, func, select +from sqlalchemy.orm import Session, joinedload + +from app.core.logging import get_logger +from app.features.conversation.lineage_schemas import aggregate_lineage_from_segments +from app.features.conversation.models import Conversation, Segment +from app.features.memoir.models import ( + Chapter, + ChapterEvidenceLink, + ChapterEvidenceSnapshot, +) +from app.features.memory.repo import fetch_memory_closure_for_conversations_sync + +EVIDENCE_SNAPSHOT_SCHEMA_VERSION = 1 + +logger = get_logger(__name__) + + +def _normalize_segment_ids(raw: object) -> list[str]: + if not raw or not isinstance(raw, list): + return [] + out: list[str] = [] + for x in raw: + s = str(x).strip() + if s: + out.append(s) + seen: set[str] = set() + deduped: list[str] = [] + for s in out: + if s not in seen: + seen.add(s) + deduped.append(s) + return deduped + + +def _story_ids_ordered(chapter: Chapter) -> list[str]: + links = sorted( + list(getattr(chapter, "story_links", None) or []), + key=lambda lnk: getattr(lnk, "order_index", 0), + ) + out: list[str] = [] + for ln in links: + sid = getattr(ln, "story_id", None) + if sid: + out.append(str(sid)) + return out + + +def build_chapter_evidence_closure_payload_sync( + session: Session, chapter: Chapter +) -> dict: + """ + 唯一闭包计算入口：由 `refresh_chapter_evidence_snapshot_sync` 与评测侧（经 JSON 镜像） + 共用同一套 segment / conversation / memory 推导逻辑。 + """ + uid = str(chapter.user_id) + segment_ids = _normalize_segment_ids(chapter.source_segments) + story_ids = _story_ids_ordered(chapter) + segs: list = [] + + if not segment_ids: + conv_ids: list[str] = [] + chunk_ids, fact_ids, tl_ids, sum_ids = [], [], [], [] + notes = [ + "no_source_segments", + "snapshot_materialized", + ] + else: + stmt = ( + select(Segment) + .join(Conversation, Segment.conversation_id == Conversation.id) + .where( + Segment.id.in_(segment_ids), + Conversation.user_id == uid, + Conversation.deleted_at.is_(None), + ) + ) + segs = list(session.execute(stmt).scalars().all()) + conv_ids = sorted({str(s.conversation_id) for s in segs if s.conversation_id}) + chunk_ids, fact_ids, tl_ids, sum_ids = ( + fetch_memory_closure_for_conversations_sync(session, uid, conv_ids) + if conv_ids + else ([], [], [], []) + ) + notes = ["snapshot_materialized"] + if len(segs) < len(segment_ids): + notes.append("some_segment_ids_unresolved_or_foreign_user") + + message_lineage_json = None + if segs: + order_map = {sid: i for i, sid in enumerate(segment_ids)} + segs_ordered = sorted(segs, key=lambda s: order_map.get(str(s.id), 9999)) + message_lineage_json = aggregate_lineage_from_segments( + segs_ordered, + conversation_id_fallback=conv_ids[0] if conv_ids else None, + ) + + return { + "schema_version": EVIDENCE_SNAPSHOT_SCHEMA_VERSION, + "captured_at": datetime.now(timezone.utc).isoformat(), + "chapter_id": str(chapter.id), + "user_id": uid, + "segment_ids": segment_ids, + "conversation_ids": conv_ids, + "story_ids": story_ids, + "memory_chunk_ids": chunk_ids, + "memory_fact_ids": fact_ids, + "timeline_event_ids": tl_ids, + "summary_ids": sum_ids, + "notes": notes, + "message_lineage_json": message_lineage_json, + } + + +# 旧名保留，避免外部 import 断裂 +build_chapter_evidence_snapshot_sync = build_chapter_evidence_closure_payload_sync + + +def _replace_chapter_evidence_links_sync( + session: Session, *, chapter_id: str, payload: dict +) -> None: + session.execute( + delete(ChapterEvidenceLink).where(ChapterEvidenceLink.chapter_id == chapter_id) + ) + for cid in payload.get("memory_chunk_ids") or []: + session.add( + ChapterEvidenceLink( + id=str(uuid.uuid4()), + chapter_id=chapter_id, + evidence_type="chunk", + evidence_id=str(cid), + role="primary", + ) + ) + for fid in payload.get("memory_fact_ids") or []: + session.add( + ChapterEvidenceLink( + id=str(uuid.uuid4()), + chapter_id=chapter_id, + evidence_type="fact", + evidence_id=str(fid), + role="supporting", + ) + ) + for tid in payload.get("timeline_event_ids") or []: + session.add( + ChapterEvidenceLink( + id=str(uuid.uuid4()), + chapter_id=chapter_id, + evidence_type="timeline_event", + evidence_id=str(tid), + role="supporting", + ) + ) + for sid in payload.get("summary_ids") or []: + session.add( + ChapterEvidenceLink( + id=str(uuid.uuid4()), + chapter_id=chapter_id, + evidence_type="summary", + evidence_id=str(sid), + role="background", + ) + ) + + +def refresh_chapter_evidence_snapshot_sync(session: Session, chapter_id: str) -> bool: + """写入新版本快照行、替换 evidence_links、更新 Chapter 当前指针；镜像 evidence_bundle_json。""" + stmt = ( + select(Chapter) + .where(Chapter.id == chapter_id) + .options(joinedload(Chapter.story_links)) + ) + ch = session.execute(stmt).unique().scalar_one_or_none() + if not ch: + return False + payload = build_chapter_evidence_closure_payload_sync(session, ch) + + max_v = session.execute( + select(func.coalesce(func.max(ChapterEvidenceSnapshot.version_no), 0)).where( + ChapterEvidenceSnapshot.chapter_id == chapter_id + ) + ).scalar() + next_v = int(max_v or 0) + 1 + cap_at = datetime.now(timezone.utc) + snap = ChapterEvidenceSnapshot( + id=str(uuid.uuid4()), + chapter_id=str(ch.id), + user_id=str(ch.user_id), + version_no=next_v, + schema_version=int(payload.get("schema_version") or EVIDENCE_SNAPSHOT_SCHEMA_VERSION), + segment_ids=list(payload.get("segment_ids") or []), + conversation_ids=list(payload.get("conversation_ids") or []), + story_ids=list(payload.get("story_ids") or []), + memory_chunk_ids=list(payload.get("memory_chunk_ids") or []), + memory_fact_ids=list(payload.get("memory_fact_ids") or []), + timeline_event_ids=list(payload.get("timeline_event_ids") or []), + summary_ids=list(payload.get("summary_ids") or []), + notes=list(payload.get("notes") or []), + message_lineage_json=payload.get("message_lineage_json"), + captured_at=cap_at, + ) + session.add(snap) + session.flush() + _replace_chapter_evidence_links_sync(session, chapter_id=str(ch.id), payload=payload) + ch.current_evidence_snapshot_id = snap.id + ch.evidence_bundle_json = payload + if payload.get("message_lineage_json") is not None: + ch.source_lineage_json = payload.get("message_lineage_json") + session.flush() + return True + + +def refresh_chapter_evidence_snapshot_with_retry_sync( + session: Session, chapter_id: str +) -> bool: + """ + 同 `refresh_chapter_evidence_snapshot_sync`，失败时整体再试 1 次（共 2 次）。 + 日志前缀 `evidence_snapshot_refresh_failed` 便于检索。 + """ + last_exc: Exception | None = None + for attempt in range(2): + try: + return refresh_chapter_evidence_snapshot_sync(session, chapter_id) + except Exception as e: + last_exc = e + logger.warning( + "evidence_snapshot_refresh_failed attempt={} chapter_id={}: {}", + attempt + 1, + chapter_id, + e, + ) + if last_exc: + logger.warning( + "evidence_snapshot_refresh_failed exhausted chapter_id={}: {}", + chapter_id, + last_exc, + ) + return False diff --git a/api/app/features/memoir/memoir_images/parser.py b/api/app/features/memoir/memoir_images/parser.py index bb68fae..b7e9d5a 100644 --- a/api/app/features/memoir/memoir_images/parser.py +++ b/api/app/features/memoir/memoir_images/parser.py @@ -2,9 +2,9 @@ import json import re from typing import Any +from app.core.json_utils import extract_json_payload from app.features.memoir.asset_resolver import strip_image_placeholders -from app.core.json_utils import extract_json_payload from .schema import IMAGE_STATUS_PENDING PLACEHOLDER_RE = re.compile( diff --git a/api/app/features/memoir/memoir_images/prompting.py b/api/app/features/memoir/memoir_images/prompting.py index 7c518ef..d907640 100644 --- a/api/app/features/memoir/memoir_images/prompting.py +++ b/api/app/features/memoir/memoir_images/prompting.py @@ -3,10 +3,10 @@ import re from typing import Any, Optional from app.core.config import settings +from app.core.json_utils import extract_json_payload from app.core.langchain_llm import invoke_json_object from app.core.logging import get_logger -from app.core.json_utils import extract_json_payload from .settings import MemoirImageSettings logger = get_logger(__name__) diff --git a/api/app/features/memoir/models.py b/api/app/features/memoir/models.py index 4066088..61c59e5 100644 --- a/api/app/features/memoir/models.py +++ b/api/app/features/memoir/models.py @@ -3,6 +3,7 @@ from sqlalchemy import ( Boolean, Column, DateTime, + Float, ForeignKey, Integer, String, @@ -34,11 +35,20 @@ class Chapter(Base): is_new = Column(Boolean, default=True) is_active = Column(Boolean, default=True) source_segments = Column(JSON, nullable=True) + # 物化当前 canonical 对应的证据闭包（segment / story / memory ids），供评测与审计；见 memoir/chapter_evidence_snapshot + evidence_bundle_json = Column(JSON, nullable=True) + # Phase C：指向当前生效的 chapter_evidence_snapshots 行（版本链可审计） + current_evidence_snapshot_id = Column( + String, + ForeignKey("chapter_evidence_snapshots.id", ondelete="SET NULL"), + nullable=True, + ) # story-backed 章节：story 变更后标 True，由 Celery 重组 canonical_markdown markdown_compose_dirty = Column(Boolean, default=False, nullable=False) markdown_composed_at = Column(DateTime(timezone=True), nullable=True) # 与 canonical 同一生成时机物化；无签名 URL，读时 hydrate reading_segments_json = Column(JSON, nullable=True) + source_lineage_json = Column(JSON, nullable=True) user = relationship("User", back_populates="chapters") book = relationship("Book", back_populates="chapters") @@ -69,6 +79,22 @@ class Chapter(Base): back_populates="chapter", cascade="all, delete-orphan", ) + evidence_snapshots = relationship( + "ChapterEvidenceSnapshot", + back_populates="chapter", + foreign_keys="ChapterEvidenceSnapshot.chapter_id", + cascade="all, delete-orphan", + ) + evidence_links = relationship( + "ChapterEvidenceLink", + back_populates="chapter", + cascade="all, delete-orphan", + ) + current_evidence_snapshot = relationship( + "ChapterEvidenceSnapshot", + foreign_keys=[current_evidence_snapshot_id], + post_update=True, + ) class MemoirImage(Base): @@ -185,6 +211,60 @@ class ChapterStoryLink(Base): story = relationship("Story", back_populates="chapter_links") +class ChapterEvidenceSnapshot(Base): + """章节证据闭包版本快照（一行 = 一次物化；current 指针在 Chapter 上）。""" + + __tablename__ = "chapter_evidence_snapshots" + + id = Column(String, primary_key=True) + chapter_id = Column( + String, + ForeignKey("chapters.id", ondelete="CASCADE"), + nullable=False, + index=True, + ) + user_id = Column(String, ForeignKey("users.id"), nullable=False, index=True) + version_no = Column(Integer, nullable=False) + schema_version = Column(Integer, nullable=False, default=1) + segment_ids = Column(JSON, nullable=True) + conversation_ids = Column(JSON, nullable=True) + story_ids = Column(JSON, nullable=True) + memory_chunk_ids = Column(JSON, nullable=True) + memory_fact_ids = Column(JSON, nullable=True) + timeline_event_ids = Column(JSON, nullable=True) + summary_ids = Column(JSON, nullable=True) + notes = Column(JSON, nullable=True) + message_lineage_json = Column(JSON, nullable=True) + captured_at = Column(DateTime(timezone=True), default=utc_now) + + chapter = relationship( + "Chapter", + back_populates="evidence_snapshots", + foreign_keys=[chapter_id], + ) + + +class ChapterEvidenceLink(Base): + """与 StoryEvidenceLink 对称：章节当前态绑定的结构化记忆 id（按快照刷新时整批替换）。""" + + __tablename__ = "chapter_evidence_links" + + id = Column(String, primary_key=True) + chapter_id = Column( + String, + ForeignKey("chapters.id", ondelete="CASCADE"), + nullable=False, + index=True, + ) + evidence_type = Column(String, nullable=False) + evidence_id = Column(String, nullable=False) + role = Column(String, nullable=True) + weight = Column(Float, nullable=True) + created_at = Column(DateTime(timezone=True), default=utc_now) + + chapter = relationship("Chapter", back_populates="evidence_links") + + class Book(Base): __tablename__ = "books" diff --git a/api/app/features/memoir/repo.py b/api/app/features/memoir/repo.py index 49fe4cc..7e9e16b 100644 --- a/api/app/features/memoir/repo.py +++ b/api/app/features/memoir/repo.py @@ -7,8 +7,12 @@ from sqlalchemy.ext.asyncio import AsyncSession from sqlalchemy.orm import Session, joinedload from app.core.db import utc_now +from app.core.logging import get_logger from app.features.asset.models import Asset from app.features.memoir.asset_resolver import collect_asset_ids_for_chapter +from app.features.memoir.chapter_evidence_snapshot import ( + refresh_chapter_evidence_snapshot_with_retry_sync, +) from app.features.memoir.chapter_markdown_compose import ( materialize_chapter_markdown_from_loaded_chapter, ) @@ -26,6 +30,8 @@ from app.features.memoir.reading_segment_materialize import ( from app.features.story.models import Story from app.features.story.time_hints import life_sort_key_parts +logger = get_logger(__name__) + async def get_current_book(user_id: str, db: AsyncSession) -> Book | None: stmt = ( @@ -51,6 +57,7 @@ async def get_chapters_for_memoir_list( .where(Chapter.user_id == user_id) .options( joinedload(Chapter.images), + joinedload(Chapter.current_evidence_snapshot), joinedload(Chapter.story_links) .joinedload(ChapterStoryLink.story) .joinedload(Story.image_intents), @@ -71,6 +78,7 @@ async def get_chapter_by_id(chapter_id: str, db: AsyncSession) -> Chapter | None .where(Chapter.id == chapter_id) .options( joinedload(Chapter.images), + joinedload(Chapter.current_evidence_snapshot), joinedload(Chapter.story_links) .joinedload(ChapterStoryLink.story) .joinedload(Story.image_intents), @@ -173,6 +181,18 @@ async def append_chapter_compose_version_async( chapter.markdown_composed_at = utc_now() chapter.reading_segments_json = build_reading_segments_snapshot(chapter) + def _snap(sess: Session) -> None: + refresh_chapter_evidence_snapshot_with_retry_sync(sess, str(chapter.id)) + + try: + await db.run_sync(_snap) + except Exception as e: + logger.warning( + "evidence_snapshot_refresh_failed async compose path chapter_id={}: {}", + chapter.id, + e, + ) + def append_chapter_compose_version_sync( session: Session, @@ -267,6 +287,7 @@ def compose_chapter_from_story_links_sync(session: Session, chapter_id: str) -> return False md = materialize_chapter_markdown_from_loaded_chapter(chapter) append_chapter_compose_version_sync(session, chapter, md) + refresh_chapter_evidence_snapshot_with_retry_sync(session, str(chapter.id)) return True diff --git a/api/app/features/memoir/state_service.py b/api/app/features/memoir/state_service.py index a74b668..331e86e 100644 --- a/api/app/features/memoir/state_service.py +++ b/api/app/features/memoir/state_service.py @@ -10,11 +10,11 @@ from sqlalchemy import select from sqlalchemy.ext.asyncio import AsyncSession from sqlalchemy.orm import Session -from app.agents.state_schema import MemoirStateSchema, SlotData, default_state from app.agents.stage_constants import ( chat_bucket, normalize_chat_stage, ) +from app.agents.state_schema import MemoirStateSchema, SlotData, default_state from app.core.config import settings from app.features.memoir.models import MemoirState as MemoirStateModel diff --git a/api/app/features/memoir/story_pipeline_sync.py b/api/app/features/memoir/story_pipeline_sync.py index 974ea04..6b73e7d 100644 --- a/api/app/features/memoir/story_pipeline_sync.py +++ b/api/app/features/memoir/story_pipeline_sync.py @@ -20,12 +20,6 @@ from app.agents.memoir.prompts import ( format_evidence_chunks_for_prompt, format_narrative_user_content, ) -from app.agents.stage_constants import ( - CATEGORY_TO_CHAT_STAGE, - CHAPTER_CATEGORIES, - CHAT_STAGES, - STAGE_TO_ORDER, -) from app.agents.memoir.story_route_agent import ( APPEND_FIRST_CHAPTER_CATEGORIES, PLAN_BATCH_MAX_SEGMENTS, @@ -33,20 +27,30 @@ from app.agents.memoir.story_route_agent import ( StoryRouteAgent, default_append_target_story_id, ) +from app.agents.stage_constants import ( + CATEGORY_TO_CHAT_STAGE, + CHAPTER_CATEGORIES, + CHAT_STAGES, + STAGE_TO_ORDER, +) from app.agents.state_schema import MemoirStateSchema from app.core.config import settings from app.core.dependencies import get_embedding_provider from app.core.logging import get_logger +from app.features.conversation.lineage_schemas import aggregate_lineage_from_segments +from app.features.memoir.chapter_evidence_snapshot import ( + refresh_chapter_evidence_snapshot_with_retry_sync, +) from app.features.memoir.cover_eligibility import chapter_needs_cover_enqueue from app.features.memoir.memoir_images.settings import MemoirImageSettings from app.features.memoir.models import Chapter -from app.features.memoir.narrative_to_markdown import narrative_to_markdown from app.features.memoir.narrative_safety import ( body_contains_prompt_artifact, evidence_leakage_heuristic, evidence_scene_anchor_leak, strip_evidence_for_overlap_check, ) +from app.features.memoir.narrative_to_markdown import narrative_to_markdown from app.features.memoir.oral_normalize import ( apply_oral_rules, normalize_oral_for_memoir, @@ -56,17 +60,111 @@ from app.features.memoir.repo import ( reorder_chapter_story_links_by_life_order_sync, ) from app.features.memory.repo import retrieve_evidence_sync -from app.features.story.models import Story +from app.features.story.models import Story, StoryVersion from app.features.story.sync_write import ( append_story_version_sync, count_story_versions_sync, create_story_with_version_sync, ensure_chapter_story_link_sync, list_active_stories_for_user_sync, + replace_story_evidence_links_sync, ) logger = get_logger(__name__) + +def _dialogue_lineage_dict_for_segment_ids( + category_segments: list, + segment_ids: list[str], +) -> dict | None: + """Merge DialogueLineage from contributing segments (memoir batch unit order).""" + if not segment_ids or not category_segments: + return None + order = {str(sid): i for i, sid in enumerate(segment_ids)} + picked = [s for s in category_segments if str(getattr(s, "id", "")) in order] + picked.sort(key=lambda s: order[str(s.id)]) + conv_fb: str | None = None + if picked: + conv_fb = getattr(picked[0], "conversation_id", None) + if not conv_fb: + for s in picked: + c = getattr(s, "conversation_id", None) + if c: + conv_fb = str(c) + break + return aggregate_lineage_from_segments( + picked, + conversation_id_fallback=str(conv_fb) if conv_fb else None, + ) + + +def _evidence_link_ids(evidence: dict) -> tuple[list[str], list[str], list[str], list[str]]: + """从 retrieve_evidence_sync 结果提取稳定 ID 列表。""" + chunks: list[str] = [] + for c in evidence.get("relevant_chunks") or []: + if isinstance(c, dict) and c.get("id"): + chunks.append(str(c["id"])) + facts: list[str] = [] + for f in evidence.get("relevant_facts") or []: + if isinstance(f, dict) and f.get("id"): + facts.append(str(f["id"])) + timelines: list[str] = [] + for e in evidence.get("timeline_hints") or []: + if isinstance(e, dict) and e.get("id"): + timelines.append(str(e["id"])) + summaries: list[str] = [] + for s in evidence.get("relevant_summaries") or []: + if isinstance(s, dict) and s.get("id"): + summaries.append(str(s["id"])) + return chunks, facts, timelines, summaries + + +def _story_prompt_meta_for_lineage( + evidence: dict, + *, + memoir_correlation_id: str | None, + top_k: int, +) -> dict: + c, f, t, s = _evidence_link_ids(evidence) + return { + "memoir_retrieval": { + "correlation_id": memoir_correlation_id, + "top_k": top_k, + "chunk_ids": c, + "fact_ids": f, + "timeline_event_ids": t, + "summary_ids": s, + } + } + + +def _persist_story_lineage_sync( + session: Session, + *, + story_id: str, + version: StoryVersion, + evidence: dict, + memoir_correlation_id: str | None, + top_k: int, + dialogue_lineage: dict | None = None, +) -> None: + """写入 StoryEvidenceLink + 本版本 prompt_meta（可审计检索闭包）。""" + c, f, t, s = _evidence_link_ids(evidence) + replace_story_evidence_links_sync( + session, + story_id=story_id, + chunk_ids=c, + fact_ids=f, + timeline_event_ids=t, + summary_ids=s, + ) + version.prompt_meta = _story_prompt_meta_for_lineage( + evidence, memoir_correlation_id=memoir_correlation_id, top_k=top_k + ) + if dialogue_lineage: + version.lineage_json = dialogue_lineage + + # 标题中若出现下列多字履历表述，则必须在 hay（正文+口述+传入标题的 slots）中逐字出现，否则剔除无果片段或降级占位 _MEMOIR_TITLE_HAY_GROUNDING_PHRASES: tuple[str, ...] = ( "晋升旅长", @@ -515,6 +613,7 @@ def _ensure_chapter_record( ) chapter.is_new = True session.flush() + refresh_chapter_evidence_snapshot_with_retry_sync(session, str(chapter.id)) return chapter @@ -526,6 +625,8 @@ def _run_batch_plan_writes( chapter: Chapter, chapter_category: str, evidence_text: str, + evidence: dict, + evidence_top_k: int, slot_snippets: dict[str, str], user_id: str, user_profile: str, @@ -659,10 +760,23 @@ def _run_batch_plan_writes( ) if target_story_id: - append_story_version_sync(session, str(target_story_id), md) - dispatch_ids.add(str(target_story_id)) + sid_s = str(target_story_id) + ver = append_story_version_sync(session, sid_s, md) + dlg = _dialogue_lineage_dict_for_segment_ids( + category_segments, list(unit.segment_ids) + ) + _persist_story_lineage_sync( + session, + story_id=sid_s, + version=ver, + evidence=evidence, + memoir_correlation_id=memoir_correlation_id, + top_k=evidence_top_k, + dialogue_lineage=dlg, + ) + dispatch_ids.add(sid_s) ensure_chapter_story_link_sync( - session, chapter_id=str(chapter.id), story_id=str(target_story_id) + session, chapter_id=str(chapter.id), story_id=sid_s ) sid_log = target_story_id is_append = True @@ -690,6 +804,21 @@ def _run_batch_plan_writes( ) sid_log = st.id is_append = False + if st.current_version_id: + ver0 = session.get(StoryVersion, st.current_version_id) + if ver0: + dlg = _dialogue_lineage_dict_for_segment_ids( + category_segments, list(unit.segment_ids) + ) + _persist_story_lineage_sync( + session, + story_id=str(st.id), + version=ver0, + evidence=evidence, + memoir_correlation_id=memoir_correlation_id, + top_k=evidence_top_k, + dialogue_lineage=dlg, + ) elapsed = time.perf_counter() - t0 logger.info( @@ -865,6 +994,8 @@ def run_story_pipeline_for_category_batch( chapter=chapter, chapter_category=chapter_category, evidence_text=evidence_text, + evidence=evidence, + evidence_top_k=top_k, slot_snippets=slot_snippets, user_id=user_id, user_profile=user_profile, @@ -995,11 +1126,26 @@ def run_story_pipeline_for_category_batch( do_append = target_story_id is not None + dlg_single = _dialogue_lineage_dict_for_segment_ids( + category_segments, + [str(s.id) for s in category_segments], + ) + if do_append: - append_story_version_sync(session, str(target_story_id), md) - dispatch_ids.add(str(target_story_id)) + sid_s = str(target_story_id) + ver = append_story_version_sync(session, sid_s, md) + _persist_story_lineage_sync( + session, + story_id=sid_s, + version=ver, + evidence=evidence, + memoir_correlation_id=memoir_correlation_id, + top_k=top_k, + dialogue_lineage=dlg_single, + ) + dispatch_ids.add(sid_s) ensure_chapter_story_link_sync( - session, chapter_id=str(chapter.id), story_id=str(target_story_id) + session, chapter_id=str(chapter.id), story_id=sid_s ) sid_log = target_story_id is_append = True @@ -1027,6 +1173,18 @@ def run_story_pipeline_for_category_batch( ) sid_log = st.id is_append = False + if st.current_version_id: + ver0 = session.get(StoryVersion, st.current_version_id) + if ver0: + _persist_story_lineage_sync( + session, + story_id=str(st.id), + version=ver0, + evidence=evidence, + memoir_correlation_id=memoir_correlation_id, + top_k=top_k, + dialogue_lineage=dlg_single, + ) elapsed = time.perf_counter() - t0 logger.info( @@ -1055,6 +1213,7 @@ def run_story_pipeline_for_category_batch( reorder_chapter_story_links_by_life_order_sync(session, str(chapter.id)) mark_chapter_dirty_sync(session, str(chapter.id)) session.flush() + refresh_chapter_evidence_snapshot_with_retry_sync(session, str(chapter.id)) image_settings = MemoirImageSettings.from_env() needs_cover = image_settings.enabled and chapter_needs_cover_enqueue(chapter) diff --git a/api/app/features/memory/chunker.py b/api/app/features/memory/chunker.py index 6dd4e4f..0e6fda3 100644 --- a/api/app/features/memory/chunker.py +++ b/api/app/features/memory/chunker.py @@ -1,6 +1,5 @@ """Transcript chunker — split raw text into retrieval-ready chunks.""" -import re def chunk_transcript( diff --git a/api/app/features/memory/enrichment.py b/api/app/features/memory/enrichment.py index 61e10ab..ad9e795 100644 --- a/api/app/features/memory/enrichment.py +++ b/api/app/features/memory/enrichment.py @@ -13,11 +13,16 @@ from sqlalchemy.ext.asyncio import AsyncSession from sqlalchemy.orm import Session from app.core.logging import get_logger +from app.features.memory.enrichment_pipeline import ( + dedupe_key, + normalize_object_json, + normalize_subject, +) from app.features.memory.extractor import ( extract_facts_from_transcript_async, extract_facts_from_transcript_sync, ) -from app.features.memory.models import MemoryChunk, MemorySummary +from app.features.memory.models import MemoryChunk, MemorySource, MemorySummary from app.features.memory.repo import ( create_memory_fact, create_memory_fact_sync, @@ -36,20 +41,20 @@ from app.features.memory.summarizer import ( generate_session_summary_async, generate_session_summary_sync, ) -from app.features.memory.enrichment_pipeline import ( - dedupe_key, - normalize_object_json, - normalize_subject, -) -from app.features.user.models import User from app.features.memory.timeline import ( build_timeline_events_from_facts_async, build_timeline_events_from_facts_sync, ) +from app.features.user.models import User logger = get_logger(__name__) +def _lineage_snapshot_from_source(source: MemorySource | None) -> dict | None: + raw = getattr(source, "lineage_json", None) if source else None + return raw if isinstance(raw, dict) and raw else None + + def _resolve_llm_sync() -> Any | None: try: from app.core.dependencies import get_llm_provider_fast @@ -81,6 +86,8 @@ def enrich_memory_after_ingest_sync( chunks = list_chunks_for_source_sync(session, source_id) if not chunks: return + src_row = session.get(MemorySource, source_id) + lineage_snapshot = _lineage_snapshot_from_source(src_row) chunk_texts = [c.content for c in chunks] chunk_ids = [c.id for c in chunks] numbered = "\n\n".join( @@ -143,6 +150,7 @@ def enrich_memory_after_ingest_sync( confidence=float(f.get("confidence") or 0.75), source_chunk_id=scid, status="confirmed", + lineage_json=lineage_snapshot, ) inserted.append( { @@ -169,6 +177,7 @@ def enrich_memory_after_ingest_sync( description=ev.get("description"), source_fact_ids=ev.get("source_fact_ids") or None, memory_source_id=source_id, + lineage_json=lineage_snapshot, ) @@ -199,6 +208,8 @@ async def enrich_memory_after_ingest_async( chunks = list(result.unique().scalars().all()) if not chunks: return + src_row = await db.get(MemorySource, source_id) + lineage_snapshot = _lineage_snapshot_from_source(src_row) chunk_texts = [c.content for c in chunks] chunk_ids = [c.id for c in chunks] numbered = "\n\n".join( @@ -265,6 +276,7 @@ async def enrich_memory_after_ingest_async( confidence=float(f.get("confidence") or 0.75), source_chunk_id=scid, status="confirmed", + lineage_json=lineage_snapshot, ) inserted.append( { @@ -291,4 +303,5 @@ async def enrich_memory_after_ingest_async( description=ev.get("description"), source_fact_ids=ev.get("source_fact_ids") or None, memory_source_id=source_id, + lineage_json=lineage_snapshot, ) diff --git a/api/app/features/memory/evidence.py b/api/app/features/memory/evidence.py index 7a34605..0ae25a1 100644 --- a/api/app/features/memory/evidence.py +++ b/api/app/features/memory/evidence.py @@ -127,9 +127,9 @@ async def fetch_evidence_metadata_async( def _empty_query_bundle_sync(session: Session, user_id: str, top_k: int) -> dict: """空 query 时的「浏览」降级：rolling 摘要 + 事实/时间线 fallback。""" - from app.features.memory.models import MemorySummary from sqlalchemy import select + from app.features.memory.models import MemorySummary from app.features.memory.repo import ( get_facts_for_user_sync, get_timeline_events_for_user_sync, diff --git a/api/app/features/memory/models.py b/api/app/features/memory/models.py index 9ebcdc1..1311a43 100644 --- a/api/app/features/memory/models.py +++ b/api/app/features/memory/models.py @@ -29,6 +29,8 @@ class MemorySource(Base): captured_at = Column(DateTime(timezone=True), nullable=True) status = Column(String, default="active") conversation_id = Column(String, ForeignKey("conversations.id"), nullable=True) + lineage_json = Column(JSON, nullable=True) + primary_user_message_id = Column(String, nullable=True) created_at = Column(DateTime(timezone=True), default=utc_now) chunks = relationship( "MemoryChunk", back_populates="source", cascade="all, delete-orphan" @@ -80,6 +82,7 @@ class MemoryFact(Base): status = Column( String, default="candidate" ) # candidate / confirmed / rejected / stale (chunk excluded / superseded) + lineage_json = Column(JSON, nullable=True) created_at = Column(DateTime(timezone=True), default=utc_now) @@ -99,6 +102,7 @@ class TimelineEvent(Base): description = Column(Text, nullable=True) person_refs = Column(JSON, nullable=True) source_fact_ids = Column(JSON, nullable=True) + lineage_json = Column(JSON, nullable=True) created_at = Column(DateTime(timezone=True), default=utc_now) diff --git a/api/app/features/memory/repo.py b/api/app/features/memory/repo.py index adc011b..582171d 100644 --- a/api/app/features/memory/repo.py +++ b/api/app/features/memory/repo.py @@ -33,6 +33,8 @@ def create_source_sync( raw_text: str | None = None, conversation_id: str | None = None, captured_at: datetime | None = None, + lineage_json: dict | None = None, + primary_user_message_id: str | None = None, ) -> MemorySource: """Create a memory source (sync). Caller must commit.""" source = MemorySource( @@ -41,6 +43,8 @@ def create_source_sync( source_type=source_type, raw_text=raw_text, conversation_id=conversation_id, + lineage_json=lineage_json, + primary_user_message_id=primary_user_message_id, captured_at=captured_at or datetime.now(timezone.utc), ) session.add(source) @@ -55,6 +59,8 @@ async def create_source( raw_text: str | None = None, conversation_id: str | None = None, captured_at: datetime | None = None, + lineage_json: dict | None = None, + primary_user_message_id: str | None = None, ) -> MemorySource: """Create a memory source. Caller must commit.""" source = MemorySource( @@ -63,6 +69,8 @@ async def create_source( source_type=source_type, raw_text=raw_text, conversation_id=conversation_id, + lineage_json=lineage_json, + primary_user_message_id=primary_user_message_id, captured_at=captured_at or datetime.now(timezone.utc), ) db.add(source) @@ -433,6 +441,70 @@ def list_summaries_for_evidence_sync( ] +def fetch_memory_closure_for_conversations_sync( + session: Session, user_id: str, conversation_ids: list[str] +) -> tuple[list[str], list[str], list[str], list[str]]: + """ + 同步版 memory 闭包（与 evaluation.eval_trace_repo.fetch_memory_closure_for_conversations 对齐）。 + """ + if not conversation_ids: + return [], [], [], [] + conv_set = list({c for c in conversation_ids if c}) + + src_stmt = select(MemorySource).where( + MemorySource.user_id == user_id, + MemorySource.conversation_id.in_(conv_set), + ) + sources = list(session.execute(src_stmt).scalars().all()) + source_ids = [s.id for s in sources] + if not source_ids: + return [], [], [], [] + + ch_stmt = select(MemoryChunk).where( + MemoryChunk.user_id == user_id, + MemoryChunk.source_id.in_(source_ids), + MemoryChunk.is_excluded.is_(False), + ) + chunks = list(session.execute(ch_stmt).scalars().all()) + chunk_ids = [c.id for c in chunks] + if not chunk_ids: + fact_rows: list[MemoryFact] = [] + else: + f_stmt = select(MemoryFact).where( + MemoryFact.user_id == user_id, + MemoryFact.source_chunk_id.in_(chunk_ids), + or_(MemoryFact.status.is_(None), MemoryFact.status != "stale"), + ) + fact_rows = list(session.execute(f_stmt).scalars().all()) + fact_ids = [f.id for f in fact_rows] + + te_stmt = select(TimelineEvent).where( + TimelineEvent.user_id == user_id, + TimelineEvent.memory_source_id.in_(source_ids), + ) + ev_rows = list(session.execute(te_stmt).scalars().all()) + timeline_ids = [e.id for e in ev_rows] + + sum_stmt = ( + select(MemorySummary) + .where(MemorySummary.user_id == user_id) + .order_by(MemorySummary.updated_at.desc()) + .limit(12) + ) + summaries = list(session.execute(sum_stmt).scalars().all()) + chunk_set = set(chunk_ids) + summary_ids: list[str] = [] + for sm in summaries: + if sm.summary_type == "rolling": + summary_ids.append(sm.id) + continue + scids = sm.source_chunk_ids or [] + if isinstance(scids, list) and chunk_set.intersection({str(x) for x in scids}): + summary_ids.append(sm.id) + + return chunk_ids, fact_ids, timeline_ids, summary_ids + + def retrieve_evidence_sync( session: Session, user_id: str, @@ -580,6 +652,7 @@ def create_memory_fact_sync( confidence: float, source_chunk_id: str | None, status: str = "confirmed", + lineage_json: dict | None = None, ) -> MemoryFact: row = MemoryFact( id=_new_id(), @@ -591,6 +664,7 @@ def create_memory_fact_sync( confidence=confidence, source_chunk_id=source_chunk_id, status=status, + lineage_json=lineage_json, ) session.add(row) return row @@ -607,6 +681,7 @@ async def create_memory_fact( confidence: float, source_chunk_id: str | None, status: str = "confirmed", + lineage_json: dict | None = None, ) -> MemoryFact: row = MemoryFact( id=_new_id(), @@ -618,6 +693,7 @@ async def create_memory_fact( confidence=confidence, source_chunk_id=source_chunk_id, status=status, + lineage_json=lineage_json, ) db.add(row) return row @@ -675,6 +751,7 @@ def create_timeline_event_sync( person_refs: list | None = None, source_fact_ids: list[str] | None = None, memory_source_id: str | None = None, + lineage_json: dict | None = None, ) -> TimelineEvent: row = TimelineEvent( id=_new_id(), @@ -686,6 +763,7 @@ def create_timeline_event_sync( description=description, person_refs=person_refs, source_fact_ids=source_fact_ids, + lineage_json=lineage_json, ) session.add(row) return row @@ -702,6 +780,7 @@ async def create_timeline_event( person_refs: list | None = None, source_fact_ids: list[str] | None = None, memory_source_id: str | None = None, + lineage_json: dict | None = None, ) -> TimelineEvent: row = TimelineEvent( id=_new_id(), @@ -713,6 +792,7 @@ async def create_timeline_event( description=description, person_refs=person_refs, source_fact_ids=source_fact_ids, + lineage_json=lineage_json, ) db.add(row) return row diff --git a/api/app/features/memory/retrieval_trace.py b/api/app/features/memory/retrieval_trace.py new file mode 100644 index 0000000..f53751a --- /dev/null +++ b/api/app/features/memory/retrieval_trace.py @@ -0,0 +1,35 @@ +"""将 memory 检索 bundle 压成可入库的轻量 trace（仅稳定 id，限长）。""" + +from __future__ import annotations + +from typing import Any + + +def _capped_ids(items: list[Any] | None, *, cap: int = 100) -> list[str]: + out: list[str] = [] + for x in items or []: + if len(out) >= cap: + break + if isinstance(x, dict) and x.get("id"): + out.append(str(x["id"])) + return out + + +def chat_memory_retrieval_trace_from_bundle( + bundle: dict, + *, + top_k: int, + query_len: int, +) -> dict: + """供 conversation_messages.memory_retrieval_trace_json 使用。""" + return { + "schema_version": 1, + "source": "chat_memory_retrieval", + "top_k": top_k, + "query_len": query_len, + "chunk_ids": _capped_ids(bundle.get("relevant_chunks")), + "fact_ids": _capped_ids(bundle.get("relevant_facts")), + "timeline_event_ids": _capped_ids(bundle.get("timeline_hints")), + "summary_ids": _capped_ids(bundle.get("relevant_summaries")), + "story_ids": _capped_ids(bundle.get("relevant_stories")), + } diff --git a/api/app/features/memory/service.py b/api/app/features/memory/service.py index b2667b9..c9c3fcf 100644 --- a/api/app/features/memory/service.py +++ b/api/app/features/memory/service.py @@ -13,7 +13,6 @@ from sqlalchemy.ext.asyncio import AsyncSession from app.core.logging import get_logger from app.features.memory.chunker import chunk_transcript -from app.features.memory.schemas import EvidenceBundle from app.features.memory.repo import ( create_chunk, create_curation_action, @@ -22,6 +21,8 @@ from app.features.memory.repo import ( set_memory_fact_status, update_chunk_embedding, ) +from app.features.conversation.lineage_schemas import primary_user_message_id_from_lineage +from app.features.memory.schemas import EvidenceBundle from app.ports.embedding import EmbeddingProvider logger = get_logger(__name__) @@ -38,7 +39,12 @@ class MemoryService: self._embedding = embedding_provider async def ingest_transcript( - self, user_id: str, conversation_id: str, transcript: str + self, + user_id: str, + conversation_id: str, + transcript: str, + *, + lineage_json: dict | None = None, ) -> str: """ Ingest conversation transcript into memory. @@ -48,12 +54,19 @@ class MemoryService: if not transcript or not transcript.strip(): raise ValueError("transcript cannot be empty") + primary_mid = ( + primary_user_message_id_from_lineage(lineage_json) + if lineage_json + else None + ) source = await create_source( self._db, user_id=user_id, source_type="transcript", raw_text=transcript.strip(), conversation_id=conversation_id, + lineage_json=lineage_json, + primary_user_message_id=primary_mid, ) chunks_text = chunk_transcript(transcript.strip()) @@ -212,6 +225,8 @@ def ingest_transcript_sync( user_id: str, conversation_id: str, transcript: str, + *, + lineage_json: dict | None = None, ) -> str: """ Sync transcript ingest for Celery tasks. @@ -229,12 +244,17 @@ def ingest_transcript_sync( if not transcript or not transcript.strip(): raise ValueError("transcript cannot be empty") + primary_mid = ( + primary_user_message_id_from_lineage(lineage_json) if lineage_json else None + ) source = create_source_sync( session, user_id=user_id, source_type="transcript", raw_text=transcript.strip(), conversation_id=conversation_id, + lineage_json=lineage_json, + primary_user_message_id=primary_mid, ) session.flush() @@ -283,7 +303,9 @@ def ingest_transcript_sync( vectors_written += 1 update_chunk_embedding_sync(session, chunk_id, emb) if settings.memory_enrichment_enabled: - from app.features.memory.enrichment import enrich_memory_after_ingest_sync + from app.features.memory.enrichment import ( + enrich_memory_after_ingest_sync, + ) enrich_memory_after_ingest_sync( session, user_id, source.id, llm=None diff --git a/api/app/features/payment/alipay_client.py b/api/app/features/payment/alipay_client.py index d3966a1..8f4b0f7 100644 --- a/api/app/features/payment/alipay_client.py +++ b/api/app/features/payment/alipay_client.py @@ -2,17 +2,17 @@ 支付宝 OpenAPI 封装（从 payment 迁入 app） """ -from app.core.logging import get_logger -from typing import Dict, Optional +from typing import Dict +from app.core.logging import get_logger from app.features.payment.payment_config import AlipayConfig -from app.features.payment.schemas import NotifyResult, PaymentResult, PaymentStatus from app.features.payment.payment_exceptions import ( PaymentConfigError, PaymentCreateError, PaymentNotifyError, PaymentQueryError, ) +from app.features.payment.schemas import NotifyResult, PaymentResult, PaymentStatus logger = get_logger(__name__) diff --git a/api/app/features/payment/order_service.py b/api/app/features/payment/order_service.py index d4ad0a9..b77f044 100644 --- a/api/app/features/payment/order_service.py +++ b/api/app/features/payment/order_service.py @@ -3,9 +3,7 @@ """ import asyncio -from app.core.logging import get_logger import time -import traceback import uuid from datetime import timedelta @@ -14,6 +12,7 @@ from sqlalchemy import select from sqlalchemy.ext.asyncio import AsyncSession from app.core.db import utc_now +from app.core.logging import get_logger from app.features.payment.models import Order from app.features.payment.schemas import ( CreateOrderResponse, diff --git a/api/app/features/payment/payment_config.py b/api/app/features/payment/payment_config.py index c6f4e4c..a10db56 100644 --- a/api/app/features/payment/payment_config.py +++ b/api/app/features/payment/payment_config.py @@ -2,9 +2,10 @@ 支付模块配置（从 payment 迁入 app，从 app.core.config.settings 读取） """ -from app.core.logging import get_logger from dataclasses import dataclass, field +from app.core.logging import get_logger + logger = get_logger(__name__) diff --git a/api/app/features/payment/payment_facade.py b/api/app/features/payment/payment_facade.py index a915cf7..420991e 100644 --- a/api/app/features/payment/payment_facade.py +++ b/api/app/features/payment/payment_facade.py @@ -2,14 +2,14 @@ 统一支付服务门面（从 payment 迁入 app） """ -from app.core.logging import get_logger from typing import Dict, Optional -from app.features.payment.payment_config import PaymentConfig -from app.features.payment.wechat_client import WeChatPayClient +from app.core.logging import get_logger from app.features.payment.alipay_client import AlipayClient +from app.features.payment.payment_config import PaymentConfig +from app.features.payment.payment_exceptions import PaymentError from app.features.payment.schemas import NotifyResult, PaymentResult, PaymentStatus -from app.features.payment.payment_exceptions import PaymentError, PaymentConfigError +from app.features.payment.wechat_client import WeChatPayClient logger = get_logger(__name__) diff --git a/api/app/features/payment/router.py b/api/app/features/payment/router.py index bc1a668..b598d1c 100644 --- a/api/app/features/payment/router.py +++ b/api/app/features/payment/router.py @@ -1,10 +1,10 @@ -from app.core.logging import get_logger from typing import List from fastapi import APIRouter, Depends, HTTPException, Request from fastapi.responses import PlainTextResponse from app.core.dependencies import get_current_user +from app.core.logging import get_logger from app.features.payment.deps import get_payment_order_service from app.features.payment.order_service import PaymentOrderService from app.features.payment.schemas import ( diff --git a/api/app/features/payment/schemas.py b/api/app/features/payment/schemas.py index 51823e4..bcbf3d0 100644 --- a/api/app/features/payment/schemas.py +++ b/api/app/features/payment/schemas.py @@ -1,6 +1,6 @@ """支付模块 Pydantic 模型定义（从 payment 迁入 app）""" -from typing import Any, Dict, Optional +from typing import Dict, Optional from pydantic import BaseModel diff --git a/api/app/features/payment/service.py b/api/app/features/payment/service.py index dce0c94..ebb0c50 100644 --- a/api/app/features/payment/service.py +++ b/api/app/features/payment/service.py @@ -3,8 +3,8 @@ """ from app.features.payment.payment_config import PaymentConfig -from app.features.payment.payment_facade import PaymentService from app.features.payment.payment_exceptions import PaymentConfigError, PaymentError +from app.features.payment.payment_facade import PaymentService __all__ = [ "PaymentService", diff --git a/api/app/features/payment/wechat_client.py b/api/app/features/payment/wechat_client.py index 298867c..3268628 100644 --- a/api/app/features/payment/wechat_client.py +++ b/api/app/features/payment/wechat_client.py @@ -3,19 +3,19 @@ """ import json -from app.core.logging import get_logger import os import time from typing import Dict +from app.core.logging import get_logger from app.features.payment.payment_config import WeChatPayConfig -from app.features.payment.schemas import NotifyResult, PaymentResult, PaymentStatus from app.features.payment.payment_exceptions import ( PaymentConfigError, PaymentCreateError, PaymentNotifyError, PaymentQueryError, ) +from app.features.payment.schemas import NotifyResult, PaymentResult, PaymentStatus logger = get_logger(__name__) diff --git a/api/app/features/plan/catalog.py b/api/app/features/plan/catalog.py new file mode 100644 index 0000000..4cdf5ed --- /dev/null +++ b/api/app/features/plan/catalog.py @@ -0,0 +1,67 @@ +"""套餐目录：单一事实来源，供 plan service 与 quota 共用（避免 plan↔quota 循环依赖）。""" + +from app.core.config import settings +from app.features.plan.schemas import PlanResponse + +ENABLE_TEST_PLAN = (settings.enable_test_plan or "").lower() in ("1", "true", "yes") + +AVAILABLE_PLANS = [ + PlanResponse( + id="free", + name="free", + display_name="免费体验版", + price=0.0, + currency="CNY", + features=["500 轮对话", "1 个章节整理", "完整回忆录生成流程"], + max_conversations=500, + max_chapters=1, + is_popular=False, + ), + PlanResponse( + id="pro", + name="pro", + display_name="Pro 版", + price=88.0, + currency="CNY", + features=["2000 轮对话", "无章节限制", "完整回忆录生成"], + max_conversations=2000, + is_popular=True, + ), + PlanResponse( + id="pro_plus", + name="pro_plus", + display_name="Pro+ 版", + price=288.0, + currency="CNY", + features=["10000 轮对话", "无章节限制", "完整回忆录生成", "长期创作无忧"], + max_conversations=10000, + is_popular=False, + ), +] + +TEST_PLAN = PlanResponse( + id="test", + name="test", + display_name="一分钱测试版", + price=0.01, + currency="CNY", + features=["无限对话", "无限章节整理", "仅用于开发环境测试支付"], + is_popular=False, +) + + +def get_plans_for_api() -> list[PlanResponse]: + if ENABLE_TEST_PLAN: + return AVAILABLE_PLANS + [TEST_PLAN] + return list(AVAILABLE_PLANS) + + +def get_plan_by_type(subscription_type: str) -> PlanResponse: + if subscription_type == "premium": + subscription_type = "pro" + if subscription_type == "test": + return TEST_PLAN if ENABLE_TEST_PLAN else AVAILABLE_PLANS[0] + for plan in AVAILABLE_PLANS: + if plan.id == subscription_type: + return plan + return AVAILABLE_PLANS[0] diff --git a/api/app/features/plan/service.py b/api/app/features/plan/service.py index 2454664..b793ef1 100644 --- a/api/app/features/plan/service.py +++ b/api/app/features/plan/service.py @@ -1,74 +1,25 @@ """Plan service — 套餐定义与查询。""" -from typing import List, Optional - -from app.core.config import settings +from app.features.plan.catalog import ( + AVAILABLE_PLANS, + ENABLE_TEST_PLAN, + TEST_PLAN, + get_plan_by_type, + get_plans_for_api, +) from app.features.plan.schemas import CurrentPlanResponse, PlanResponse from app.features.quota.service import QuotaService from app.features.user.models import User -ENABLE_TEST_PLAN = (settings.enable_test_plan or "").lower() in ("1", "true", "yes") - -AVAILABLE_PLANS = [ - PlanResponse( - id="free", - name="free", - display_name="免费体验版", - price=0.0, - currency="CNY", - features=["500 轮对话", "无章节限制", "完整回忆录生成流程"], - max_conversations=500, - is_popular=False, - ), - PlanResponse( - id="pro", - name="pro", - display_name="Pro 版", - price=88.0, - currency="CNY", - features=["2000 轮对话", "无章节限制", "完整回忆录生成"], - max_conversations=2000, - is_popular=True, - ), - PlanResponse( - id="pro_plus", - name="pro_plus", - display_name="Pro+ 版", - price=288.0, - currency="CNY", - features=["10000 轮对话", "无章节限制", "完整回忆录生成", "长期创作无忧"], - max_conversations=10000, - is_popular=False, - ), +__all__ = [ + "AVAILABLE_PLANS", + "ENABLE_TEST_PLAN", + "TEST_PLAN", + "PlanService", + "get_plan_by_type", + "get_plans_for_api", ] -TEST_PLAN = PlanResponse( - id="test", - name="test", - display_name="一分钱测试版", - price=0.01, - currency="CNY", - features=["无限对话", "无限章节整理", "仅用于开发环境测试支付"], - is_popular=False, -) - - -def get_plans_for_api() -> list[PlanResponse]: - if ENABLE_TEST_PLAN: - return AVAILABLE_PLANS + [TEST_PLAN] - return list(AVAILABLE_PLANS) - - -def get_plan_by_type(subscription_type: str) -> Optional[PlanResponse]: - if subscription_type == "premium": - subscription_type = "pro" - if subscription_type == "test": - return TEST_PLAN if ENABLE_TEST_PLAN else AVAILABLE_PLANS[0] - for plan in AVAILABLE_PLANS: - if plan.id == subscription_type: - return plan - return AVAILABLE_PLANS[0] - class PlanService: def __init__(self, quota_service: QuotaService): diff --git a/api/app/features/quota/service.py b/api/app/features/quota/service.py index 8cc9b56..fbb2ffb 100644 --- a/api/app/features/quota/service.py +++ b/api/app/features/quota/service.py @@ -2,6 +2,8 @@ 配额检查业务逻辑。 「对话轮数」的定义：每条用户发出的消息（Segment 表的记录数）计为 1 轮。 + +限额与 [app.features.plan.catalog](catalog) 中的套餐定义一致，单一事实来源。 """ from sqlalchemy import func, select @@ -9,36 +11,9 @@ from sqlalchemy.ext.asyncio import AsyncSession from app.features.conversation.models import Conversation, Segment from app.features.memoir.models import Chapter +from app.features.plan.catalog import get_plan_by_type from app.features.quota.schemas import QuotaCheckResponse -PLAN_QUOTAS = { - "free": { - "max_conversations": 50, - "max_chapters": 1, - "max_words": None, - }, - "pro": { - "max_conversations": 2000, - "max_chapters": None, - "max_words": None, - }, - "pro_plus": { - "max_conversations": 10000, - "max_chapters": None, - "max_words": None, - }, - "premium": { - "max_conversations": None, - "max_chapters": None, - "max_words": None, - }, - "test": { - "max_conversations": None, - "max_chapters": None, - "max_words": None, - }, -} - async def get_segment_count(user_id: str, db: AsyncSession) -> int: """获取用户已消耗的对话轮数（= 该用户所有 Segment 记录数）。""" @@ -71,8 +46,8 @@ def check_can_send_message( segment_count: int, ) -> tuple[bool, str]: """检查用户是否还能发送消息（对话轮数）。返回 (是否允许, 提示信息)。""" - quotas = PLAN_QUOTAS.get(subscription_type, PLAN_QUOTAS["free"]) - max_conv = quotas.get("max_conversations") + plan = get_plan_by_type(subscription_type) + max_conv = plan.max_conversations if max_conv is None: return True, "" if segment_count >= max_conv: @@ -88,8 +63,8 @@ def check_can_submit_organize( chapter_count: int, ) -> tuple[bool, str]: """检查是否可以提交整理任务（生成新章节）。免费版仅允许 1 个章节。""" - quotas = PLAN_QUOTAS.get(subscription_type, PLAN_QUOTAS["free"]) - max_ch = quotas.get("max_chapters") + plan = get_plan_by_type(subscription_type) + max_ch = plan.max_chapters if max_ch is None: return True, "" if chapter_count >= max_ch: @@ -123,11 +98,10 @@ class QuotaService: async def check(self, user_id: str, subscription_type: str) -> QuotaCheckResponse: """检查用户配额使用情况。""" - quotas = PLAN_QUOTAS.get(subscription_type, PLAN_QUOTAS["free"]) + plan = get_plan_by_type(subscription_type) segment_count, chapter_count = await self.get_usage(user_id) - max_conversations = quotas.get("max_conversations") - max_chapters = quotas.get("max_chapters") - max_words = quotas.get("max_words") + max_conversations = plan.max_conversations + max_chapters = plan.max_chapters remaining_conversations = None remaining_chapters = None diff --git a/api/app/features/story/models.py b/api/app/features/story/models.py index ab194da..34394e9 100644 --- a/api/app/features/story/models.py +++ b/api/app/features/story/models.py @@ -7,12 +7,12 @@ Story Layer 数据模型。 """ from sqlalchemy import ( + JSON, Column, DateTime, Float, ForeignKey, Integer, - JSON, String, Text, ) @@ -97,6 +97,7 @@ class StoryVersion(Base): String, ForeignKey("story_versions.id", ondelete="SET NULL"), nullable=True ) prompt_meta = Column(JSON, nullable=True) + lineage_json = Column(JSON, nullable=True) created_at = Column(DateTime(timezone=True), default=utc_now) story = relationship("Story", back_populates="versions", foreign_keys=[story_id]) diff --git a/api/app/features/story/post_commit.py b/api/app/features/story/post_commit.py index 60ca944..71068f4 100644 --- a/api/app/features/story/post_commit.py +++ b/api/app/features/story/post_commit.py @@ -6,9 +6,9 @@ enqueue 失败不回滚已提交数据，仅记录日志；依赖后续触发或 from __future__ import annotations +import threading from dataclasses import dataclass, field from datetime import datetime, timezone -import threading from typing import Any, cast import redis diff --git a/api/app/features/story/repo.py b/api/app/features/story/repo.py index ad5261a..738da76 100644 --- a/api/app/features/story/repo.py +++ b/api/app/features/story/repo.py @@ -1,7 +1,6 @@ """Story repository — Story, StoryVersion, StoryEvidenceLink data access.""" import uuid -from datetime import datetime, timezone from sqlalchemy import delete, or_, select from sqlalchemy.ext.asyncio import AsyncSession diff --git a/api/app/features/story/sync_write.py b/api/app/features/story/sync_write.py index 5f3e8bc..ccfaea8 100644 --- a/api/app/features/story/sync_write.py +++ b/api/app/features/story/sync_write.py @@ -12,13 +12,17 @@ from datetime import datetime, timezone from sqlalchemy import delete, func, select from sqlalchemy.orm import Session, joinedload -from app.core.db import utc_now from app.core.logging import get_logger +from app.features.memoir import repo as memoir_repo from app.features.memoir.asset_resolver import strip_asset_image_refs_from_markdown from app.features.memoir.models import ChapterStoryLink -from app.features.memoir import repo as memoir_repo from app.features.story.image_intent_extractor import extract_primary_image_intent -from app.features.story.models import Story, StoryImageIntent, StoryVersion +from app.features.story.models import ( + Story, + StoryEvidenceLink, + StoryImageIntent, + StoryVersion, +) from app.features.story.time_hints import apply_infer_story_time_start_to_model logger = get_logger(__name__) @@ -108,6 +112,61 @@ def _extract_and_store_image_intent_sync( ) +def replace_story_evidence_links_sync( + session: Session, + *, + story_id: str, + chunk_ids: list[str], + fact_ids: list[str], + timeline_event_ids: list[str], + summary_ids: list[str], +) -> None: + """以当前生成所用的检索闭包覆盖 story 证据关联（artifact 当前态绑定）。""" + session.execute( + delete(StoryEvidenceLink).where(StoryEvidenceLink.story_id == story_id) + ) + for cid in chunk_ids: + session.add( + StoryEvidenceLink( + id=str(uuid.uuid4()), + story_id=story_id, + evidence_type="chunk", + evidence_id=cid, + role="primary", + ) + ) + for fid in fact_ids: + session.add( + StoryEvidenceLink( + id=str(uuid.uuid4()), + story_id=story_id, + evidence_type="fact", + evidence_id=fid, + role="supporting", + ) + ) + for tid in timeline_event_ids: + session.add( + StoryEvidenceLink( + id=str(uuid.uuid4()), + story_id=story_id, + evidence_type="timeline_event", + evidence_id=tid, + role="supporting", + ) + ) + for sid in summary_ids: + session.add( + StoryEvidenceLink( + id=str(uuid.uuid4()), + story_id=story_id, + evidence_type="summary", + evidence_id=sid, + role="background", + ) + ) + + def create_story_with_version_sync( session: Session, *, @@ -115,6 +174,7 @@ def create_story_with_version_sync( title: str, canonical_markdown: str, stage: str | None = None, + prompt_meta: dict | None = None, ) -> Story: md = strip_asset_image_refs_from_markdown(canonical_markdown or "") story = Story( @@ -134,6 +194,7 @@ def create_story_with_version_sync( markdown_snapshot=md, actor_type="ai", source_type="generate", + prompt_meta=prompt_meta, ) session.add(version) session.flush() @@ -154,6 +215,7 @@ def append_story_version_sync( *, actor_type: str = "ai", source_type: str = "generate", + prompt_meta: dict | None = None, ) -> StoryVersion: story = session.get(Story, story_id) if not story: @@ -170,6 +232,7 @@ def append_story_version_sync( actor_type=actor_type, source_type=source_type, parent_version_id=parent_id, + prompt_meta=prompt_meta, ) session.add(version) session.flush() diff --git a/api/app/features/story/time_hints.py b/api/app/features/story/time_hints.py index 2027e19..e9061e7 100644 --- a/api/app/features/story/time_hints.py +++ b/api/app/features/story/time_hints.py @@ -8,7 +8,6 @@ import re from datetime import datetime from typing import Any - _YEAR_IN_TITLE = re.compile(r"(?:^|[\s·，,])(?P(?:19|20)\d{2})(?:年|\s|·|$)") _YEAR_ANYWHERE = re.compile(r"(?:19|20)\d{2}") diff --git a/api/app/features/tasks/deps.py b/api/app/features/tasks/deps.py index d5d3203..f9a5316 100644 --- a/api/app/features/tasks/deps.py +++ b/api/app/features/tasks/deps.py @@ -1,6 +1,5 @@ """Tasks feature 依赖：提供 get_tasks_service。""" -from fastapi import Depends from app.features.tasks.service import TasksService diff --git a/api/app/internal_main.py b/api/app/internal_main.py index c0c34a4..3f0bd72 100644 --- a/api/app/internal_main.py +++ b/api/app/internal_main.py @@ -1,9 +1,9 @@ """ 内部回归评测 API 入口：与 app.main 进程隔离部署。 -启动示例（在 api/ 目录）:: +启动示例（在 api/ 目录；端口与 INTERNAL_EVAL_PORT 一致，development.sh 默认 7999）:: - uv run uvicorn app.internal_main:internal_app --host 0.0.0.0 --port 8001 + uv run uvicorn app.internal_main:internal_app --host 0.0.0.0 --port 7999 """ from __future__ import annotations @@ -56,7 +56,7 @@ register_exception_handlers(internal_app) @internal_app.get("/", include_in_schema=False, response_class=HTMLResponse) async def internal_eval_landing(): - """浏览器打开 :8001 根路径时提示：界面在 Vite（默认 5174），本进程仅为 API。""" + """浏览器打开内部评测 API 根路径时提示：界面在 Vite（默认 5174），本进程仅为 API。""" docs_hint = ( '

OpenAPI 文档 /docs

' if settings.internal_eval_enable_docs diff --git a/api/app/ports/asr.py b/api/app/ports/asr.py index d333b9a..b0efce4 100644 --- a/api/app/ports/asr.py +++ b/api/app/ports/asr.py @@ -3,6 +3,10 @@ from typing import Protocol, runtime_checkable +class ASRTranscriptionError(Exception): + """ASR 失败时抛出；`str(e)` 适合写入日志，面向客户端的文案由 pipeline 统一处理。""" + + @runtime_checkable class ASRProvider(Protocol): async def transcribe(self, audio: bytes, format: str = "m4a") -> str: diff --git a/api/app/tasks/evaluation_tasks.py b/api/app/tasks/evaluation_tasks.py index 29e88d3..afb3de8 100644 --- a/api/app/tasks/evaluation_tasks.py +++ b/api/app/tasks/evaluation_tasks.py @@ -11,8 +11,14 @@ from app.core.logging import get_logger logger = get_logger(__name__) -@shared_task(name="evaluation.run_experiment") -def run_eval_experiment_task(experiment_id: str) -> None: +@shared_task( + bind=True, + name="evaluation.run_experiment", + max_retries=1, + soft_time_limit=1800, + time_limit=2400, +) +def run_eval_experiment_task(self, experiment_id: str) -> None: from app.features.evaluation.execution_service import execute_experiment_full logger.info("evaluation task start experiment_id={}", experiment_id) diff --git a/api/app/tasks/memoir_tasks.py b/api/app/tasks/memoir_tasks.py index fad4573..a85d3e2 100644 --- a/api/app/tasks/memoir_tasks.py +++ b/api/app/tasks/memoir_tasks.py @@ -18,7 +18,6 @@ from app.agents.chat.background_voice import infer_background_voice from app.agents.chat.prompts_profile import format_user_profile_context from app.agents.memoir import MemoirOrchestrator from app.agents.stage_constants import normalize_chapter_category -from app.agents.state_schema import MemoirStateSchema, default_state from app.core.chapter_pipeline_lock import ( acquire_chapter_pipeline_lock as _acquire_chapter_lock, ) @@ -34,8 +33,6 @@ from app.core.memoir_pipeline_trace import ( new_memoir_correlation_id, ) from app.features.conversation.models import Conversation, Segment - -from app.tasks.celery_app import celery_app from app.features.memoir.cover_eligibility import ( chapter_needs_cover_enqueue, ) @@ -64,6 +61,7 @@ from app.features.memoir.story_pipeline_sync import ( run_story_pipeline_for_category_batch, ) from app.features.user.models import User +from app.tasks.celery_app import celery_app logger = get_logger(__name__) _REDIS_CLIENTS: dict[bool, redis.Redis] = {} @@ -597,27 +595,37 @@ def process_memoir_phase1(self, user_id: str, segment_ids: List[str]): ) return {"status": "no_segments"} - conv_id = getattr(segments[0], "conversation_id", None) or "" - transcript = "\n\n".join(seg.user_input_text or "" for seg in segments) - if transcript.strip(): + for seg in segments: + conv_id = getattr(seg, "conversation_id", None) or "" + text = (seg.user_input_text or "").strip() + if not text: + continue try: from app.features.memory.service import ingest_transcript_sync - source_id = ingest_transcript_sync(db, user_id, conv_id, transcript) + ln = getattr(seg, "lineage_json", None) + lineage_payload = ln if isinstance(ln, dict) else None + source_id = ingest_transcript_sync( + db, + user_id, + conv_id, + text, + lineage_json=lineage_payload, + ) logger.info( "event=memory_transcript_ingested user_id={} task_id={} " - "source_id={} conversation_id={} transcript_chars={} " - "segment_count={}", + "source_id={} conversation_id={} segment_id={} transcript_chars={}", user_id, task_id, source_id, conv_id, - len(transcript), - len(segments), + seg.id, + len(text), ) except Exception as e: logger.warning( - "Memory ingest 跳过: {} exc_type={}", + "Memory ingest 跳过 segment_id={}: {} exc_type={}", + getattr(seg, "id", ""), e, type(e).__name__, ) diff --git a/api/development.sh b/api/development.sh index 5eadd48..a869a19 100755 --- a/api/development.sh +++ b/api/development.sh @@ -28,7 +28,7 @@ LIFE_ECHO_WITH_INTERNAL_EVAL="${LIFE_ECHO_WITH_INTERNAL_EVAL:-0}" # 若 :8000 已由其他 development 实例占用，仅附加 :8001 + 前端（需自备同一份 Celery/主站） EVAL_ATTACH_ONLY="${EVAL_ATTACH_ONLY:-0}" INTERNAL_EVAL_HOST="${INTERNAL_EVAL_HOST:-0.0.0.0}" -INTERNAL_EVAL_PORT="${INTERNAL_EVAL_PORT:-8001}" +INTERNAL_EVAL_PORT="${INTERNAL_EVAL_PORT:-7999}" START_EVAL_WEB="${START_EVAL_WEB:-1}" OPEN_EVAL_WEB="${OPEN_EVAL_WEB:-1}" EVAL_WEB_PORT="${EVAL_WEB_PORT:-5174}" @@ -405,6 +405,7 @@ start_eval_web() { ( cd "${EVAL_WEB_DIR}" VITE_EVAL_API_KEY="${api_key}" \ + VITE_EVAL_PROXY_TARGET="http://127.0.0.1:${INTERNAL_EVAL_PORT}" \ npm run dev -- --host 127.0.0.1 --port "${EVAL_WEB_PORT}" "${vite_extra[@]}" ) & EVAL_WEB_PID=$! diff --git a/api/docs/internal-eval.md b/api/docs/internal-eval.md index db8ccd2..995d3ee 100644 --- a/api/docs/internal-eval.md +++ b/api/docs/internal-eval.md @@ -89,6 +89,21 @@ VITE_EVAL_API_BASE=http://127.0.0.1:8001 VITE_EVAL_API_KEY=与上同 npm run dev | `POST` | `/internal/api/evaluation/judge/memoir-chapters` | body：`{ "user_id", "baseline_sections"? }`，Chapter/Story 分项 | | `GET` | `/internal/api/evaluation/users/{user_id}/memoir-snapshot` | 只读章节与故事正文快照 | +## 回忆录评审：可追溯证据闭包（lineage） + +**产品与 tier 口径（strict / partial / fallback）、synthetic vs library 分表、PM 对齐规则、backlog** 见同目录 **[traceable-memoir-lineage.md](./traceable-memoir-lineage.md)**。 + +手动 `/judge/memoir-chapters` 与自动化 `eval_runs.judge_bundle_json` 已按 **artifact 绑定证据** 组 prompt，而不再默认拼接「最近 N 个会话全文」： + +- **`lineage_tier`**：`strict` / `partial` / `fallback`（章节：**有可解析 transcript 链 + 结构化记忆为 strict**；**仅有结构化记忆、无绑定 segment/transcript = partial**，与标注口径一致）。故事侧以 `StoryEvidenceLink` 与章节推导为主；`fallback` = 显式降级最近会话 transcript，避免静默当 strict。 +- **`evidence_trace`**：bundle 完整 JSON（segment / conversation / chunk / fact / timeline / summary、`notes` 等）。内审计一般够用；若需按类型深链 UI 再排期。 +- **`format_meta`**：`truncated`、`dropped_sections`、`included_token_estimate` 等，区分「prompt 裁掉」与「库中无 lineage」。 +- **生产侧**：叙事流水线在每次 Story 写入后覆盖 `story_evidence_links`，并在当前 `story_versions.prompt_meta.memoir_retrieval` 写入本轮检索到的稳定 id（见 `story_pipeline_sync._persist_story_lineage_sync`）。 +- **章节快照 Phase C**：`chapter_evidence_snapshots` + `chapter_evidence_links`，`chapters.current_evidence_snapshot_id` 指向当前版本；`evidence_bundle_json` 仍为镜像。评测读取顺序：表快照 → JSON → 现场 `source_segments`（不一致时 `notes` 提示）。刷新见 `memoir/chapter_evidence_snapshot.py`。历史库可选 `uv run python scripts/backfill_chapter_evidence_snapshots.py`（旧数据不强制）。 +- **对话 memory trace（Phase 八）**：访谈路由下，`conversation_messages.memory_retrieval_trace_json` 在配对 **AI** 消息上写入本轮 `HybridRetriever` 命中的 chunk/fact/timeline/summary/story 等 id（见 `memory/retrieval_trace.py`）。 + +历史数据可无 link：评测仍可用 partial/fallback 跑通；可选离线 backfill 须在 job 中显式打标，不冒充 strict。 + ## Fixture 详情扩展 `GET /internal/api/evaluation/fixtures/user-exports/{filename}` 在原有 `turns` 外增加： diff --git a/api/docs/traceable-memoir-lineage.md b/api/docs/traceable-memoir-lineage.md new file mode 100644 index 0000000..c2d16b2 --- /dev/null +++ b/api/docs/traceable-memoir-lineage.md @@ -0,0 +1,41 @@ +# 回忆录可追溯证据（产品与内评口径） + +本文与 PM、标注、工程共用：**旧库数据不要求为评测专门 backfill**；新写入走统一闭包与快照表。口径不清会导致反复对齐成本，变更 tier 规则时请同步改 `EvalTraceService._chapter_closure_tier` / Story 侧等价逻辑与本文。 + +## lineage_tier：strict / partial / fallback + +| 档位 | 含义（章节） | 含义（故事，概要） | +|------|----------------|-------------------| +| **strict** | 既有可解析的访谈 **segment**（可绑定 transcript），又有从对应会话闭包得到的 **结构化记忆**（chunk / fact / timeline / summary 等任一非空）。 | Story 上以 `StoryEvidenceLink` 等为主链解析出 segment + 结构化记忆均存在。 | +| **partial** | 有可解析的 **segment / transcript 链**，但只有结构化记忆为空，或仅有结构化记忆而 **无** 可绑定的 segment（**与 PM/标注对齐：仅有结构化、无 transcript = partial**）。 | 能从章节 `source_segments` 等推导出一侧证据但闭包不完整。 | +| **fallback** | 无法从 artifact 构建足够闭包（例如无 segment 且无法走库内链路），评测侧 **显式** 降级为「最近若干会话」等粗粒度 transcript；须在结果 `notes` / `evidence_trace` 中可见，避免静默当 strict 用。 | + +**说明**：`partial` 不是「质量差」，而是「血缘不完整仍可评审」；`fallback` 是「链路断裂时的保守降级」，评审 prompt 与 gate 解读需区别对待。 + +## 自动化评测：synthetic memoir vs library artifact（分表心理模型） + +同一次 `eval_run` 里可能同时存在两类「回忆录」分数，语义不同，**勿混为一谈**： + +- **Synthetic（replay 合成短文）**：由 case 的 replay 对话现场拼出的短 markdown，证据闭包仅为 **重放 transcript**，不绑定用户库里的 memory chunk / fact / timeline / summary。`judge_meta.synthetic_memoir_lineage_tier` 等为 `replay_transcript_only` 一类标记。 +- **Library（库内章节 / 故事）**：真实 `Chapter` / `Story` artifact，使用 `EvalTraceService` 组装的 evidence bundle（含 `lineage_tier`、`evidence_trace`）。 + +聚合规则见 `judge_bundle_json.judge_meta.memoir_aggregate_rule`（例如合成与 library 均有分时的加权方式）。对 PM 汇报时请分项展示，避免只报一个「回忆录分」。 + +## 内评 JSON：`evidence_trace` 是否够用？ + +当前 `evidence_trace` 为 `ChapterEvidenceBundle` / `StoryEvidenceBundle` 的 **完整序列化**：含 `segment_ids`、`conversation_ids`、各类 memory id、`lineage_tier`、`notes`、`augmented_with_chapter_context` 等。**一般内审计够**：可按 id 去 DB 或日志反查。 + +若需 **按 artifact 类型展开为可点击深链 / 批量导出**，属于体验增强，可单独排期（eval-web 已支持章节级折叠展示 id 列表）。 + +## Phase C：`chapter_evidence_snapshots` 与 `chapter_evidence_links` + +- **快照表**：一行对应一次物化闭包（`version_no` 递增）；`chapters.current_evidence_snapshot_id` 指向当前生效行。 +- **链接表**：与 `StoryEvidenceLink` 对称，按快照刷新时 **整批替换** 章节侧结构化记忆 id，便于审计与扩展。 +- **评测消费顺序**：`current_evidence_snapshot`（表）→ `evidence_bundle_json`（JSON 镜像，兼容）→ 现场用 `source_segments` 计算（与 live 不一致时 `notes` 提示）。 +- **旧数据**：可不迁；新流水线写入会同时更新表与 JSON 镜像。 + +## 技术债（backlog，不阻塞发版） + +1. **统一闭包计算**：生产快照与 `EvalTraceService` 已共用 `build_chapter_evidence_closure_payload_sync`；Story / 其它路径若仍有重复推导，应收敛到同一入口，避免双份实现漂移。 +2. **扩展 memory trace**：除当前访谈检索外，其它入口若向模型喂 memory，评估是否同样写入 `memory_retrieval_trace_json`（或等价 trace），以便 partial / strict 判定与事后审计一致。 +3. **canonical 与 `source_segments` union 冲突**：若线上冲突案例增多，再评估独立快照表外的「版本级 link」或更强约束；当前 Phase C 已降低仅依赖单列 JSON 的风险。 diff --git a/api/scripts/backfill_chapter_evidence_snapshots.py b/api/scripts/backfill_chapter_evidence_snapshots.py new file mode 100644 index 0000000..32e4322 --- /dev/null +++ b/api/scripts/backfill_chapter_evidence_snapshots.py @@ -0,0 +1,57 @@ +#!/usr/bin/env python3 +"""回填 chapters.evidence_bundle_json（Phase 九：历史 chapter 无快照时可批量物化）。 + +用法（需在已执行 Alembic 0009+ 的库上）:: + + cd api && uv run python scripts/backfill_chapter_evidence_snapshots.py + cd api && uv run python scripts/backfill_chapter_evidence_snapshots.py --user-id +""" + +from __future__ import annotations + +import argparse + +from sqlalchemy import select + +from app.features.auth import models as _auth_models # noqa: F401 +from app.features.conversation import models as _conv_models # noqa: F401 +from app.features.memory import models as _memory_models # noqa: F401 +from app.features.memoir import models as _memoir_models # noqa: F401 +from app.features.payment import models as _payment_models # noqa: F401 +from app.features.story import models as _story_models # noqa: F401 +from app.features.user import models as _user_models # noqa: F401 + +from app.core.db import SessionLocal +from app.features.memoir.chapter_evidence_snapshot import ( + refresh_chapter_evidence_snapshot_sync, +) +from app.features.memoir.models import Chapter + + +def main() -> None: + p = argparse.ArgumentParser() + p.add_argument("--user-id", default="", help="仅刷新该用户的章节；默认全表") + p.add_argument("--limit", type=int, default=0, help="最多处理条数，0 表示不限制") + args = p.parse_args() + + uid = (args.user_id or "").strip() + session = SessionLocal() + n_ok = 0 + try: + stmt = select(Chapter.id) + if uid: + stmt = stmt.where(Chapter.user_id == uid) + if args.limit > 0: + stmt = stmt.limit(args.limit) + ids = list(session.execute(stmt).scalars().all()) + for cid in ids: + if refresh_chapter_evidence_snapshot_sync(session, str(cid)): + n_ok += 1 + session.commit() + print(f"refreshed_snapshots={n_ok} chapter_rows={len(ids)}") + finally: + session.close() + + +if __name__ == "__main__": + main() diff --git a/api/scripts/backfill_segment_dialogue_lineage.py b/api/scripts/backfill_segment_dialogue_lineage.py new file mode 100644 index 0000000..d148b72 --- /dev/null +++ b/api/scripts/backfill_segment_dialogue_lineage.py @@ -0,0 +1,86 @@ +#!/usr/bin/env python3 +"""回填 segments.user_message_id / lineage_json（由既有 conversation_messages 配对）。 + +用法:: + + cd api && uv run python scripts/backfill_segment_dialogue_lineage.py + cd api && uv run python scripts/backfill_segment_dialogue_lineage.py --limit 500 +""" + +from __future__ import annotations + +import argparse + +from sqlalchemy import select +from sqlalchemy.orm import Session + +from app.features.auth import models as _auth_models # noqa: F401 +from app.features.conversation import models as _conv_models # noqa: F401 +from app.features.memory import models as _memory_models # noqa: F401 +from app.features.memoir import models as _memoir_models # noqa: F401 +from app.features.payment import models as _payment_models # noqa: F401 +from app.features.story import models as _story_models # noqa: F401 +from app.features.user import models as _user_models # noqa: F401 + +from app.core.db import SessionLocal +from app.features.conversation.lineage_schemas import DialogueLineage +from app.features.conversation.models import Conversation, ConversationMessage, Segment + + +def _first_message_for_segment( + session: Session, *, segment_id: str, role: str +) -> ConversationMessage | None: + stmt = ( + select(ConversationMessage) + .where( + ConversationMessage.segment_id == segment_id, + ConversationMessage.role == role, + ) + .order_by(ConversationMessage.created_at.asc(), ConversationMessage.id.asc()) + .limit(1) + ) + return session.execute(stmt).scalar_one_or_none() + + +def main() -> None: + p = argparse.ArgumentParser() + p.add_argument("--limit", type=int, default=0, help="最多处理 segment 条数，0 不限制") + args = p.parse_args() + + session = SessionLocal() + n = 0 + try: + stmt = ( + select(Segment) + .join(Conversation, Segment.conversation_id == Conversation.id) + .where( + Segment.lineage_json.is_(None), + Conversation.deleted_at.is_(None), + ) + .order_by(Segment.created_at.asc()) + ) + if args.limit > 0: + stmt = stmt.limit(args.limit) + segments = list(session.execute(stmt).scalars().all()) + for seg in segments: + hum = _first_message_for_segment(session, segment_id=str(seg.id), role="human") + if not hum: + continue + ai = _first_message_for_segment(session, segment_id=str(seg.id), role="ai") + ln = DialogueLineage.for_single_turn( + conversation_id=str(seg.conversation_id), + user_message_id=str(hum.id), + assistant_message_id=str(ai.id) if ai else None, + segment_ids=[str(seg.id)], + ) + seg.user_message_id = str(hum.id) + seg.lineage_json = ln.model_dump(mode="json") + n += 1 + session.commit() + print(f"updated_segments={n} scanned={len(segments)}") + finally: + session.close() + + +if __name__ == "__main__": + main() diff --git a/api/tests/conftest.py b/api/tests/conftest.py new file mode 100644 index 0000000..9a408e9 --- /dev/null +++ b/api/tests/conftest.py @@ -0,0 +1,39 @@ +"""pytest 共享fixtures 与约定入口（可按 backend-testing-strategy 逐步扩展）。""" + +from __future__ import annotations + +import uuid +from datetime import datetime, timezone +from typing import Callable + +import pytest + +from app.features.user.models import User + + +@pytest.fixture +def unique_phone() -> str: + """避免与测试库中已存在手机号冲突（11 位）。""" + return f"138{uuid.uuid4().int % 100_000_000:08d}" + + +@pytest.fixture +def make_test_user(unique_phone: str) -> Callable[..., User]: + """工厂：构造仅用于响应序列化的 User 行（未入库）。""" + + def _make( + *, + user_id: str | None = None, + phone: str | None = None, + nickname: str = "测试用户", + ) -> User: + return User( + id=user_id or str(uuid.uuid4()), + phone=phone or unique_phone, + password_hash="x", + nickname=nickname, + subscription_type="free", + created_at=datetime.now(timezone.utc), + ) + + return _make diff --git a/api/tests/evaluation/test_eval_trace_repo.py b/api/tests/evaluation/test_eval_trace_repo.py new file mode 100644 index 0000000..de6ffb4 --- /dev/null +++ b/api/tests/evaluation/test_eval_trace_repo.py @@ -0,0 +1,10 @@ +from app.features.evaluation.eval_trace_repo import normalize_source_segment_ids + + +def test_normalize_source_segment_ids_list() -> None: + assert normalize_source_segment_ids(["a", "b", "a"]) == ["a", "b"] + + +def test_normalize_source_segment_ids_empty() -> None: + assert normalize_source_segment_ids(None) == [] + assert normalize_source_segment_ids([]) == [] diff --git a/api/tests/evaluation/test_eval_trace_service_dialogue_lineage.py b/api/tests/evaluation/test_eval_trace_service_dialogue_lineage.py new file mode 100644 index 0000000..bf7ec10 --- /dev/null +++ b/api/tests/evaluation/test_eval_trace_service_dialogue_lineage.py @@ -0,0 +1,48 @@ +"""EvalTraceService 在 chapter 快照路径返回 dialogue_lineage。""" + +from __future__ import annotations + +from types import SimpleNamespace +from unittest.mock import MagicMock + +import pytest +from sqlalchemy.ext.asyncio import AsyncSession + +from app.features.evaluation.eval_trace_service import EvalTraceService +from app.features.memoir.chapter_evidence_snapshot import EVIDENCE_SNAPSHOT_SCHEMA_VERSION + + +@pytest.mark.asyncio +async def test_build_chapter_bundle_dialogue_lineage_from_snapshot() -> None: + msg_ln = { + "schema_version": 1, + "conversation_id": "cv1", + "turns": [ + {"user_message_id": "um-99", "assistant_message_id": "as-99"}, + ], + } + snap = SimpleNamespace( + user_id="u1", + chapter_id="ch1", + schema_version=EVIDENCE_SNAPSHOT_SCHEMA_VERSION, + segment_ids=["s1"], + conversation_ids=["cv1"], + memory_chunk_ids=["mk1"], + memory_fact_ids=[], + timeline_event_ids=[], + summary_ids=[], + notes=[], + message_lineage_json=msg_ln, + ) + chapter = SimpleNamespace( + id="ch1", + user_id="u1", + source_segments=["s1"], + current_evidence_snapshot=snap, + evidence_bundle_json=None, + ) + db = MagicMock(spec=AsyncSession) + svc = EvalTraceService(db) + bundle = await svc.build_chapter_bundle("u1", chapter) + assert bundle.dialogue_lineage == msg_ln + assert bundle.dialogue_lineage["turns"][0]["user_message_id"] == "um-99" diff --git a/api/tests/test_auth_flow_http.py b/api/tests/test_auth_flow_http.py new file mode 100644 index 0000000..a07395b --- /dev/null +++ b/api/tests/test_auth_flow_http.py @@ -0,0 +1,100 @@ +"""Auth HTTP 契约：注册 / 登录 / 刷新 / 受保护 /me（依赖注入 mock）。""" + +from __future__ import annotations + +from unittest.mock import AsyncMock, MagicMock + +import pytest +from fastapi import FastAPI +from httpx import ASGITransport, AsyncClient + +from app.core.dependencies import get_current_user +from app.features.auth.deps import get_auth_service +from app.features.auth.router import router as auth_router +from app.features.auth.service import AuthService + + +@pytest.fixture +def auth_app(make_test_user) -> FastAPI: + app = FastAPI() + app.include_router(auth_router) + + mock_service = MagicMock(spec=AuthService) + mock_service.register = AsyncMock( + return_value={"access_token": "access-reg", "refresh_token": "refresh-reg"} + ) + mock_service.login = AsyncMock( + return_value={"access_token": "access-login", "refresh_token": "refresh-login"} + ) + mock_service.refresh_tokens = AsyncMock( + return_value={"access_token": "access-new", "refresh_token": "refresh-new"} + ) + + app.dependency_overrides[get_auth_service] = lambda: mock_service + app.dependency_overrides[get_current_user] = lambda: make_test_user() + + app.state._mock_auth_service = mock_service + return app + + +@pytest.mark.asyncio +async def test_register_login_refresh_me(auth_app: FastAPI, unique_phone: str) -> None: + transport = ASGITransport(app=auth_app) + async with AsyncClient(transport=transport, base_url="http://test") as ac: + reg = await ac.post( + "/api/auth/register", + json={ + "phone": unique_phone, + "password": "secret12", + "nickname": "T", + "agreed_to_terms": True, + }, + ) + assert reg.status_code == 201 + body = reg.json() + assert body["access_token"] == "access-reg" + + login = await ac.post( + "/api/auth/login", + json={ + "phone": unique_phone, + "password": "secret12", + "agreed_to_terms": True, + }, + ) + assert login.status_code == 200 + assert login.json()["access_token"] == "access-login" + + ref = await ac.post( + "/api/auth/refresh", + json={"refresh_token": "refresh-login"}, + ) + assert ref.status_code == 200 + assert ref.json()["access_token"] == "access-new" + + me = await ac.get( + "/api/auth/me", + headers={"Authorization": "Bearer any"}, + ) + assert me.status_code == 200 + assert me.json()["nickname"] == "测试用户" + + svc: MagicMock = auth_app.state._mock_auth_service + svc.register.assert_awaited_once() + svc.login.assert_awaited_once() + svc.refresh_tokens.assert_awaited_once_with(refresh_token="refresh-login") + + +@pytest.mark.asyncio +async def test_me_without_auth_returns_401() -> None: + app = FastAPI() + app.include_router(auth_router) + + mock_service = MagicMock(spec=AuthService) + app.dependency_overrides[get_auth_service] = lambda: mock_service + # 不覆盖 get_current_user — 使用真实 OAuth2，无 header 时应 401 + + transport = ASGITransport(app=app) + async with AsyncClient(transport=transport, base_url="http://test") as ac: + r = await ac.get("/api/auth/me") + assert r.status_code == 401 diff --git a/api/tests/test_conversation_history_turn_ids.py b/api/tests/test_conversation_history_turn_ids.py new file mode 100644 index 0000000..9fd2183 --- /dev/null +++ b/api/tests/test_conversation_history_turn_ids.py @@ -0,0 +1,69 @@ +"""对话落库返回人/助 message id，供 segment lineage 配对。""" + +from __future__ import annotations + +from unittest.mock import AsyncMock, MagicMock + +import pytest +from sqlalchemy.ext.asyncio import AsyncSession + +from app.features.conversation.history_store import HumanAiTurnIds + + +@pytest.mark.asyncio +async def test_record_human_ai_turn_returns_both_message_ids(monkeypatch) -> None: + conv_id = "conv-1" + captured: list[object] = [] + + class FakeMsg: + def __init__(self, **kwargs) -> None: + for k, v in kwargs.items(): + setattr(self, k, v) + + class _FakeRepo: + @staticmethod + def add_conversation_message(msg: object, db) -> None: + captured.append(msg) + + monkeypatch.setattr( + "app.features.conversation.history_store.ConversationMessage", + FakeMsg, + ) + + db = MagicMock(spec=AsyncSession) + db.commit = AsyncMock() + db.refresh = AsyncMock() + + import app.features.conversation.history_store as hs_mod + + orig_repo = hs_mod.repo + hs_mod.repo = _FakeRepo # type: ignore[misc] + try: + from app.features.conversation import history_store as hs + + store = hs.ConversationHistoryStore(db) + store._sync_redis_best_effort = AsyncMock() # type: ignore[method-assign] + store._touch_conversation = AsyncMock() # type: ignore[method-assign] + + out = await store.record_human_ai_turn( + conv_id, + "hello", + ["reply a", "reply b"], + user_message_timestamp=None, + is_from_voice=False, + voice_session_id=None, + audio_duration_seconds=None, + tts_audio_urls=None, + segment_id="seg-1", + memory_retrieval_trace=None, + ) + finally: + hs_mod.repo = orig_repo # type: ignore[misc] + + assert isinstance(out, HumanAiTurnIds) + assert len(captured) == 2 + assert captured[0].role == "human" + assert captured[1].role == "ai" + assert captured[0].segment_id == "seg-1" + assert out.human_message_id == captured[0].id + assert out.assistant_message_id == captured[1].id diff --git a/api/tests/test_dialogue_lineage_memory_ingest.py b/api/tests/test_dialogue_lineage_memory_ingest.py new file mode 100644 index 0000000..a1b85bb --- /dev/null +++ b/api/tests/test_dialogue_lineage_memory_ingest.py @@ -0,0 +1,63 @@ +"""ingest_transcript_sync 将 lineage_json 传入 create_source。""" + +from __future__ import annotations + +from types import SimpleNamespace + +from app.features.memory.service import ingest_transcript_sync + + +def test_ingest_transcript_sync_passes_lineage(monkeypatch) -> None: + captured: dict = {} + + class FakeSession: + commit_calls = 0 + + def commit(self) -> None: + self.commit_calls += 1 + + def flush(self) -> None: + pass + + def begin_nested(self): + from contextlib import nullcontext + + return nullcontext() + + def fake_create_source(session, **kwargs): + captured.update(kwargs) + return SimpleNamespace(id="src-1") + + monkeypatch.setattr( + "app.core.dependencies.get_embedding_provider", lambda: None + ) + monkeypatch.setattr( + "app.features.memory.repo.create_source_sync", + fake_create_source, + ) + monkeypatch.setattr( + "app.features.memory.repo.create_chunk_sync", + lambda *a, **k: SimpleNamespace(id=f"ch-{k.get('chunk_index')}"), + ) + monkeypatch.setattr("app.core.config.settings.memory_enrichment_enabled", False) + + lineage = { + "schema_version": 1, + "conversation_id": "c9", + "turns": [ + {"user_message_id": "um-1", "assistant_message_id": "as-1"}, + ], + "primary_user_message_id": "um-1", + } + + fake_session = FakeSession() + sid = ingest_transcript_sync( + fake_session, + "u1", + "c9", + "hello there", + lineage_json=lineage, + ) + assert sid == "src-1" + assert captured.get("lineage_json") == lineage + assert captured.get("primary_user_message_id") == "um-1" diff --git a/api/tests/test_json_and_memory_utils.py b/api/tests/test_json_and_memory_utils.py index ee640e5..4643a0f 100644 --- a/api/tests/test_json_and_memory_utils.py +++ b/api/tests/test_json_and_memory_utils.py @@ -24,6 +24,11 @@ def test_extract_json_payload_strips_markdown_fence() -> None: assert '"a"' in extract_json_payload(raw) +def test_extract_json_payload_balanced_nested_braces() -> None: + raw = 'noise {"outer": {"inner": 1}} trailing' + assert extract_json_payload(raw) == '{"outer": {"inner": 1}}' + + def test_normalize_llm_category_strips_quotes() -> None: assert _normalize_llm_category('"childhood"') == "childhood" assert _normalize_llm_category("`beliefs`") == "beliefs" diff --git a/api/tests/test_judge_service.py b/api/tests/test_judge_service.py index e45ffdb..b0fd6fe 100644 --- a/api/tests/test_judge_service.py +++ b/api/tests/test_judge_service.py @@ -101,8 +101,9 @@ def test_build_memoir_prompt_includes_source_and_reference_evidence() -> None: ) assert "【评审说明】" in prompt - assert "【原始访谈/证据】" in prompt + assert "【原始访谈/对话证据】" in prompt assert "用户: 我后来去了深圳工作。" in prompt + assert "【结构化记忆证据】" in prompt assert "【参考基线/导出成稿】" in prompt assert "【当前回忆录正文】" in prompt @@ -112,5 +113,6 @@ def test_build_memoir_prompt_requires_conservative_scoring_without_evidence() -> memoir_markdown="# 当前正文\n他后来去了南方。" ) - assert "无可用原始访谈证据" in prompt + assert "无可用局部对话证据" in prompt assert "必须保守打分" in prompt + assert "【结构化记忆证据】" in prompt diff --git a/api/tests/test_lineage_schemas.py b/api/tests/test_lineage_schemas.py new file mode 100644 index 0000000..05980d9 --- /dev/null +++ b/api/tests/test_lineage_schemas.py @@ -0,0 +1,90 @@ +"""conversation.lineage_schemas 单元测试。""" + +from __future__ import annotations + +from types import SimpleNamespace + +from app.features.conversation.lineage_schemas import ( + DialogueLineage, + aggregate_lineage_from_segments, + merge_dialogue_lineages, + primary_user_message_id_from_lineage, +) + + +def test_for_single_turn_sets_primary() -> None: + ln = DialogueLineage.for_single_turn( + conversation_id="c1", + user_message_id="u1", + assistant_message_id="a1", + segment_ids=["s1"], + ) + assert ln.primary_user_message_id == "u1" + assert ln.turns[0].assistant_message_id == "a1" + assert ln.segment_ids == ["s1"] + + +def test_primary_user_message_id_from_lineage_dict() -> None: + d = { + "schema_version": 1, + "conversation_id": "c", + "turns": [{"user_message_id": "x", "assistant_message_id": None}], + } + assert primary_user_message_id_from_lineage(d) == "x" + + +def test_merge_dialogue_lineages_dedupes_user_messages() -> None: + a = DialogueLineage.for_single_turn( + conversation_id="c1", + user_message_id="u1", + assistant_message_id="a1", + segment_ids=["s1"], + ) + b = DialogueLineage.for_single_turn( + conversation_id="c1", + user_message_id="u1", + assistant_message_id="a2", + segment_ids=["s2"], + ) + c = DialogueLineage.for_single_turn( + conversation_id="c1", + user_message_id="u2", + assistant_message_id="a3", + segment_ids=["s3"], + ) + m = merge_dialogue_lineages([a, b, c], conversation_id="c1") + assert m is not None + assert [t.user_message_id for t in m.turns] == ["u1", "u2"] + + +def test_aggregate_lineage_from_segments_orders_and_merges() -> None: + segs = [ + SimpleNamespace( + id="s2", + conversation_id="conv", + lineage_json=DialogueLineage.for_single_turn( + conversation_id="conv", + user_message_id="u2", + assistant_message_id="a2", + segment_ids=["s2"], + ).model_dump(mode="json"), + user_message_id=None, + ), + SimpleNamespace( + id="s1", + conversation_id="conv", + lineage_json=DialogueLineage.for_single_turn( + conversation_id="conv", + user_message_id="u1", + assistant_message_id="a1", + segment_ids=["s1"], + ).model_dump(mode="json"), + user_message_id=None, + ), + ] + ordered = ["s1", "s2"] + order_map = {sid: i for i, sid in enumerate(ordered)} + segs.sort(key=lambda s: order_map[str(s.id)]) + out = aggregate_lineage_from_segments(segs, conversation_id_fallback="conv") + assert out is not None + assert [t["user_message_id"] for t in out["turns"]] == ["u1", "u2"] diff --git a/api/tests/test_memory_compaction.py b/api/tests/test_memory_compaction.py index 0535711..3c4928c 100644 --- a/api/tests/test_memory_compaction.py +++ b/api/tests/test_memory_compaction.py @@ -188,11 +188,21 @@ def test_finalize_clears_stale_deadline_when_not_extended(monkeypatch) -> None: def test_ingest_transcript_sync_populates_embeddings(monkeypatch) -> None: + class _NestedTransaction: + def __enter__(self): + return self + + def __exit__(self, *_exc): + return False + class FakeSession: def __init__(self) -> None: self.flush_calls = 0 self.commit_calls = 0 + def begin_nested(self): + return _NestedTransaction() + def flush(self) -> None: self.flush_calls += 1 diff --git a/api/tests/test_ws_auth_connect.py b/api/tests/test_ws_auth_connect.py new file mode 100644 index 0000000..e4b2e9e --- /dev/null +++ b/api/tests/test_ws_auth_connect.py @@ -0,0 +1,20 @@ +"""WebSocket / 配额：轻量契约测试。""" + +from __future__ import annotations + +from unittest.mock import AsyncMock, MagicMock + +import pytest + +from app.features.conversation.ws.quota_guard import check_ws_quota +from app.features.quota.service import QuotaService + + +@pytest.mark.asyncio +async def test_check_ws_quota_delegates_to_quota_service() -> None: + qs = MagicMock(spec=QuotaService) + qs.check_can_send_message = AsyncMock(return_value=(True, "")) + ok, msg = await check_ws_quota(qs, "user-1", "free") + assert ok is True + assert msg == "" + qs.check_can_send_message.assert_awaited_once_with("user-1", "free") diff --git a/app-eval-web/src/components/DiffTable.tsx b/app-eval-web/src/components/DiffTable.tsx index 1769267..d90d65a 100644 --- a/app-eval-web/src/components/DiffTable.tsx +++ b/app-eval-web/src/components/DiffTable.tsx @@ -1,6 +1,22 @@ import { useState } from "react"; import type { EvalRunOut } from "../types"; import { JsonPreview } from "./JsonPreview"; +import { MemoirScoreSummary } from "./ScoreCard"; + +function isRecord(v: unknown): v is Record { + return typeof v === "object" && v !== null && !Array.isArray(v); +} + +function judgeBundleHasMemoirLists(bundle: unknown): bundle is Record { + if (!isRecord(bundle)) return false; + const lists = [ + bundle.chapter_results, + bundle.chapters, + bundle.story_results, + bundle.stories, + ]; + return lists.some((x) => Array.isArray(x) && x.length > 0); +} type PairRow = { caseId: string; @@ -124,7 +140,23 @@ export function DiffTable({ runs }: { runs: EvalRunOut[] }) {
baseline
{row.bRun ? ( - + <> + {judgeBundleHasMemoirLists(row.bRun.judge_bundle_json) ? ( + <> +
+ 回忆录证据 / 章节评审 +
+ + + ) : null} +
+ 完整 run +
+ + ) : ( — )} @@ -132,7 +164,23 @@ export function DiffTable({ runs }: { runs: EvalRunOut[] }) {
candidate
{row.cRun ? ( - + <> + {judgeBundleHasMemoirLists(row.cRun.judge_bundle_json) ? ( + <> +
+ 回忆录证据 / 章节评审 +
+ + + ) : null} +
+ 完整 run +
+ + ) : ( — )} diff --git a/app-eval-web/src/components/ScoreCard.tsx b/app-eval-web/src/components/ScoreCard.tsx index 06af6e2..b0ce9ec 100644 --- a/app-eval-web/src/components/ScoreCard.tsx +++ b/app-eval-web/src/components/ScoreCard.tsx @@ -12,6 +12,168 @@ function isRecord(v: unknown): v is Record { return typeof v === "object" && v !== null && !Array.isArray(v); } +function stringifyIdList(raw: unknown, maxVisible: number): string[] { + if (!Array.isArray(raw)) return []; + const xs = raw + .map((x) => String(x).trim()) + .filter(Boolean); + return xs.slice(0, maxVisible); +} + +function DialogueLineageTurnsTable({ dlg }: { dlg: Record }) { + const turnsRaw = dlg.turns; + if (!Array.isArray(turnsRaw) || turnsRaw.length === 0) return null; + const rows = turnsRaw.slice(0, 80).map((t, idx) => { + if (!isRecord(t)) return null; + const u = typeof t.user_message_id === "string" ? t.user_message_id : "—"; + const a = + typeof t.assistant_message_id === "string" + ? t.assistant_message_id + : "—"; + return ( + + + {idx + 1} + + + {u} + + + {a} + + + ); + }); + const filtered = rows.filter(Boolean); + if (!filtered.length) return null; + const total = turnsRaw.length; + const more = total > filtered.length ? `（展示前 ${filtered.length} / ${total} 轮）` : ""; + return ( +
+ + 对话血缘 dialogue_lineage.turns + + {more ? ( + + {more} + + ) : null} + + + + + + + + + {filtered} +
#user_message_id + assistant_message_id +
+
+ ); +} + +/** 章节 / 故事行内：折叠展示 evidence_trace 中的 id 列表（内审计） */ +function EvidenceTraceFold({ trace }: { trace: unknown }) { + if (trace == null || !isRecord(trace)) return null; + const groups: { key: string; label: string }[] = [ + { key: "segment_ids", label: "Segments" }, + { key: "conversation_ids", label: "Conversations" }, + { key: "memory_chunk_ids", label: "Memory chunks" }, + { key: "memory_fact_ids", label: "Facts" }, + { key: "timeline_event_ids", label: "Timeline" }, + { key: "summary_ids", label: "Summaries" }, + ]; + const anyIds = groups.some( + (g) => stringifyIdList(trace[g.key], 200).length > 0, + ); + const dlgRaw = trace.dialogue_lineage; + const dlg = isRecord(dlgRaw) ? dlgRaw : null; + const hasDialogue = + dlg != null && + Array.isArray(dlg.turns) && + dlg.turns.some( + (t) => + isRecord(t) && + typeof t.user_message_id === "string" && + t.user_message_id.length > 0, + ); + const notesRaw = trace.notes; + const notes = Array.isArray(notesRaw) + ? notesRaw.map((n) => String(n)).filter(Boolean) + : []; + if (!anyIds && notes.length === 0 && !hasDialogue) return null; + return ( +
+ + 证据溯源（evidence_trace） + +
+ {typeof trace.lineage_tier === "string" ? ( +
+ lineage_tier{" "} + {trace.lineage_tier} +
+ ) : null} + {groups.map(({ key, label }) => { + const ids = stringifyIdList(trace[key], 80); + if (!ids.length) return null; + const total = Array.isArray(trace[key]) ? trace[key].length : ids.length; + const more = total > ids.length ? ` …共 ${total} 条` : ""; + return ( +
+ + {label} + + + {ids.join("\n")} + {more} + +
+ ); + })} + {dlg ? : null} + {notes.length ? ( +
+ notes +
    + {notes.map((n, i) => ( +
  • {n}
    • + ))} +
+
+ ) : null} +
+
+ ); +} + /** 对话自动评分：总分 + 五维摘要 + 理由 + 可展开 JSON */ export function ScoreCard({ kind, @@ -74,13 +236,37 @@ export function ScoreCard({ ); } +/** 手工评审 API 用 chapter_results/story_results；自动化 run 的 judge_bundle_json 用 chapters/stories。 */ +function pickMemoirChapterList(data: Record): unknown[] { + const manual = data.chapter_results; + const exec = data.chapters; + if (Array.isArray(manual) && manual.length > 0) return manual; + if (Array.isArray(exec)) return exec; + return Array.isArray(manual) ? manual : []; +} + +function pickMemoirStoryList(data: Record): unknown[] { + const manual = data.story_results; + const exec = data.stories; + if (Array.isArray(manual) && manual.length > 0) return manual; + if (Array.isArray(exec)) return exec; + return Array.isArray(manual) ? manual : []; +} + /** 回忆录评审结果：列表章节分 + 原始 JSON */ -export function MemoirScoreSummary({ data }: { data: unknown }) { +export function MemoirScoreSummary({ + data, + showRawJson = true, +}: { + data: unknown; + /** 为 false 时仅渲染结构化章节/故事块（供 DiffTable 等外层再贴完整 run JSON） */ + showRawJson?: boolean; +}) { if (!isRecord(data)) { return

暂无结果

; } - const chapters = data.chapter_results; - const stories = data.story_results; + const chapters = pickMemoirChapterList(data); + const stories = pickMemoirStoryList(data); return (
{Array.isArray(chapters) && chapters.length > 0 ? ( @@ -94,11 +280,41 @@ export function MemoirScoreSummary({ data }: { data: unknown }) { {String(c.title ?? c.chapter_title ?? `章节 ${i + 1}`)} - {typeof c.total_score === "number" ? ( - - {c.total_score.toFixed(1)} 分 + {typeof c.lineage_tier === "string" ? ( + + {c.lineage_tier} ) : null} + {(() => { + const j = isRecord(c.judge) ? c.judge : null; + const sc = + j && typeof j.total_score === "number" + ? j.total_score + : null; + return sc != null ? ( + + {sc.toFixed(1)} 分 + + ) : null; + })()} + {typeof c.evidence_summary === "string" && + c.evidence_summary ? ( +
+ {c.evidence_summary} + {c.format_meta && + isRecord(c.format_meta) && + c.format_meta.truncated === true ? ( + · 已截断 + ) : null} +
+ ) : null} + ) : ( {JSON.stringify(c)} @@ -117,9 +333,26 @@ export function MemoirScoreSummary({ data }: { data: unknown }) { {isRecord(s) ? ( <> {String(s.title ?? `故事 ${i + 1}`)} - {typeof s.total_score === "number" - ? ` · ${s.total_score.toFixed(1)}` + {typeof s.lineage_tier === "string" + ? ` · ${s.lineage_tier}` : ""} + {(() => { + const j = isRecord(s.judge) ? s.judge : null; + const sc = + j && typeof j.total_score === "number" + ? j.total_score + : null; + return sc != null ? ` · ${sc.toFixed(1)} 分` : ""; + })()} + {typeof s.evidence_summary === "string" && s.evidence_summary ? ( +
+ {s.evidence_summary} +
+ ) : null} + ) : null} @@ -127,7 +360,7 @@ export function MemoirScoreSummary({ data }: { data: unknown }) {
) : null} - + {showRawJson ? : null}
); } diff --git a/app-eval-web/src/config.ts b/app-eval-web/src/config.ts index 3700675..500e712 100644 --- a/app-eval-web/src/config.ts +++ b/app-eval-web/src/config.ts @@ -2,16 +2,20 @@ export const envApiBase = ( import.meta.env.VITE_EVAL_API_BASE as string | undefined )?.trim() ?? ""; -/** 开发无 VITE_EVAL_API_BASE：相对路径走 Vite proxy → :8001。生产未配 env 则直连 8001。 */ +const devProxyTarget = + (import.meta.env.VITE_EVAL_PROXY_TARGET as string | undefined)?.trim() || + "http://127.0.0.1:7999"; + +/** 开发无 VITE_EVAL_API_BASE：相对路径走 Vite proxy（默认 :7999，与 development.sh 一致）。 */ export const apiBase = - envApiBase || (import.meta.env.DEV ? "" : "http://127.0.0.1:8001"); + envApiBase || (import.meta.env.DEV ? "" : "http://127.0.0.1:7999"); export const apiKey = (import.meta.env.VITE_EVAL_API_KEY as string | undefined)?.trim() ?? ""; export const apiBaseHint = apiBase === "" - ? "（开发）请求经 Vite 代理到 http://127.0.0.1:8001" + ? `（开发）请求经 Vite 代理到 ${devProxyTarget}` : `直连 ${apiBase}`; export const SESSION_LIST_POLL_MS = 4000; diff --git a/app-eval-web/src/vite-env.d.ts b/app-eval-web/src/vite-env.d.ts index d0e280c..ce438da 100644 --- a/app-eval-web/src/vite-env.d.ts +++ b/app-eval-web/src/vite-env.d.ts @@ -1,8 +1,10 @@ /// interface ImportMetaEnv { - readonly VITE_EVAL_API_BASE: string; - readonly VITE_EVAL_API_KEY: string; + readonly VITE_EVAL_API_BASE?: string; + readonly VITE_EVAL_API_KEY?: string; + /** 仅开发：与 vite.config 中 /internal 代理 target 一致，用于侧栏提示 */ + readonly VITE_EVAL_PROXY_TARGET?: string; } interface ImportMeta { diff --git a/app-eval-web/vite.config.ts b/app-eval-web/vite.config.ts index 03ee18a..cb961ac 100644 --- a/app-eval-web/vite.config.ts +++ b/app-eval-web/vite.config.ts @@ -1,23 +1,32 @@ import react from "@vitejs/plugin-react"; +import { loadEnv } from "vite"; import { defineConfig } from "vitest/config"; /** - * 开发时可将 VITE_EVAL_API_BASE 留空，前端请求 /internal/... 由 Vite 转发到 8001， - * 避免连错端口、CORS 或浏览器策略导致看似 404。 + * 开发时可将 VITE_EVAL_API_BASE 留空，前端请求 /internal/... 由 Vite 代理转发。 + * 默认与 api/development.sh 中 INTERNAL_EVAL_PORT（默认 7999）一致。 + * 覆盖：VITE_EVAL_PROXY_TARGET=http://127.0.0.1:8001 npm run dev */ -export default defineConfig({ - plugins: [react()], - server: { - port: 5174, - proxy: { - "/internal": { - target: "http://127.0.0.1:8001", - changeOrigin: true, +export default defineConfig(({ mode }) => { + const env = loadEnv(mode, process.cwd(), ""); + const proxyTarget = + (env.VITE_EVAL_PROXY_TARGET || "").trim() || + "http://127.0.0.1:7999"; + + return { + plugins: [react()], + server: { + port: 5174, + proxy: { + "/internal": { + target: proxyTarget, + changeOrigin: true, + }, }, }, - }, - test: { - environment: "node", - include: ["src/**/*.test.ts"], - }, + test: { + environment: "node", + include: ["src/**/*.test.ts"], + }, + }; });