fix calculation logic, fix api endpoint

This commit is contained in:
zaiun xu
2026-04-15 09:01:45 +08:00
parent 670e6308a5
commit 45de318461
10 changed files with 158 additions and 94 deletions

View File

@@ -10,7 +10,7 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
cd "$SCRIPT_DIR"
SESSION_ROOT="/home/ubuntu/data/fish/2016-1-22-last"
FISH_NAME="fish1"
FISH_NAME="fish9"
fish_dir="${SESSION_ROOT}/${FISH_NAME}/"
OUT_PARENT="output_weight_estimator"
save_out="${OUT_PARENT}/${FISH_NAME}"

View File

@@ -691,13 +691,12 @@ def _predict_folder_impl(
effective_top_by_length = False
selected_topk = selected_topk_by_weight
preds_g_topk = [float(it["predicted_weight_g"]) for it in selected_topk]
use_max_instead_of_mean = len(candidates_for_avg) < 5
# Always arithmetic mean over selected top-K (matches _topk_mean_prediction_g_for_display
# and the printed top{k}_avg= line). Do not use max when N<5 — that made pred_weight_g
# disagree with "top{k}_avg by length" in logs.
use_max_instead_of_mean = False
if preds_g_topk:
avg_g_topk = (
float(np.max(preds_g_topk))
if use_max_instead_of_mean
else float(np.mean(preds_g_topk))
)
avg_g_topk = float(np.mean(preds_g_topk))
else:
avg_g_topk = float(avg_g_all)
num_used_for_avg = len(selected_topk)
@@ -1483,7 +1482,6 @@ def main() -> None:
)
if args.remove_outliers and summary.get("num_outliers_removed", 0) > 0:
print(f"Outliers removed: {summary['num_outliers_removed']} (method={args.outlier_method}, field={args.outlier_field})")
top_label = _summary_top_label(summary, args.top_k, args.top_by_length)
tk = int(summary.get("top_k") or args.top_k)
topk_g = summary.get("avg_topk_mean_pred_g")
topk_sel = summary.get("avg_topk_mean_pred_selection") or "pred"
@@ -1492,8 +1490,9 @@ def main() -> None:
by = "length" if topk_sel == "by_length" else "pred"
topk_extra = f" | top{tk}_avg={float(topk_g):.2f} g by {by}"
mcol = _mean_column_g_for_log(summary)
# First column is mean over ALL post-filter candidates (not top-K); do not label as top-K.
print(
f"Average predicted weight {top_label}: "
f"Mean all candidates (after filters): "
f"{mcol:.2f} g, {mcol / 1000.0:.4f} kg{topk_extra}"
)
_print_max_weight_after_filter(summary)

View File

@@ -1,19 +1,4 @@
# Fish API
FastAPI 网关:分块接收 **SVO2**FishMeasure**MP4**FishAction在后台调用现有脚本对外提供文档约定的 **GET** 轮询接口,并托管预览视频 URL。
## 依赖环境
- Python 3.11+uv 可管理 3.13 等版本)
- **FishMeasure**:需与本机 `FishMeasure/` 一致的环境ZED SDK、`pyzed`、CUDA、YOLO/SAM 权重等)
- **FishAction**:需可运行 `FishAction/predict_video_x3d_3class.py`PyTorch、PyTorchVideo、checkpoint
若 FishMeasure 与 FishAction 使用不同虚拟环境,可设置:
- `PYTHON_FISH_MEASURE` — 运行 `predict_weigth_from_svo2.py` 的解释器路径
- `PYTHON_FISH_ACTION` — 运行 `predict_video_x3d_3class.py` 的解释器路径
**单一 Conda 环境**:若 FishMeasure 与 FishAction 已与网关停在同一个 env例如仓库根目录 [`packaging/conda-fishserver.yaml`](../packaging/conda-fishserver.yaml) 定义的 `fishserver`),则**不要**设置上述两个变量,子进程会使用当前 `uvicorn` 的 Python。可用 [`scripts/start_fresh.sh`](../scripts/start_fresh.sh)(清空后启动)或 [`scripts/start_no_fresh.sh`](../scripts/start_no_fresh.sh)(保留缓存)启动。
## 配置(环境变量)
@@ -37,55 +22,10 @@ cd fish_api
uv sync
# 可选:包含 httpx便于本地用 FastAPI TestClient 做冒烟测试
# uv sync --group dev
bash start_fresh.sh # 默认仅重置 client_id 投递进度,保留 SQLite 历史与快照
./scripts/start_fresh.sh # 默认仅重置 client_id 投递进度,保留 SQLite 历史与快照
# CLEAR_SQLITE_DATABASE=1 bash start_fresh.sh # 需要时才彻底清 SQLite
# 或uv run uvicorn app.main:app --host 0.0.0.0 --port 8000需自行 prestart
```
OpenAPI`http://127.0.0.1:8000/docs`
## 对外 GET由其它系统轮询
- `GET /api/v1/biomass/real/camera/` — 每次 GET 消费**该客户端**下一条未投递的称重快照SQLite **按客户端独立游标**);无新数据时 `data.result` 为空,响应头 `X-Fish-Biomass-New: 0`
- `GET /api/v1/biomass/health/result/` — 同上,行为 / 健康快照队列
**客户端区分**:请求头 `X-Fish-Client-Id: <字符串>` 或查询参数 `client_id=<字符串>`**优先头**)。未携带时等价于 `default`,与旧版「全局一条游标」行为一致。不同 `client_id` 各自从当前队列消费,互不影响。
`result` 中每条鱼含算法字段:`id`(跟踪 ID`weight`g`length`mm。不可交付或失败的推理不会进入客户端队列。
## 流式输入(分块上传)
1. `POST /api/v1/ingest/svo/session``POST /api/v1/ingest/mp4/session``session_id`
2. `PUT /api/v1/ingest/{svo|mp4}/session/{session_id}?offset=0`(多次)
- **追加语义**`offset` 必须等于当前已写入字节数(从 0 开始顺序上传)
3. `POST /api/v1/ingest/{svo|mp4}/session/{session_id}/finalize``202 Accepted`,后台开始跑对应算法
示例(小文件一次性,`$API_KEY` 可选):
```bash
BASE=http://127.0.0.1:8000
H=()
# H=(-H "X-API-Key: your-secret")
sid=$(curl -sS "${H[@]}" -X POST "$BASE/api/v1/ingest/svo/session" | jq -r .session_id)
curl -sS "${H[@]}" -T sample.svo2 -X PUT "$BASE/api/v1/ingest/svo/session/$sid?offset=0"
curl -sS "${H[@]}" -X POST "$BASE/api/v1/ingest/svo/session/$sid/finalize"
curl -sS "$BASE/api/v1/biomass/real/camera/"
```
MP4 将 `svo` 换成 `mp4`,本地文件换成 `clip.mp4`,轮询 `GET /api/v1/biomass/health/result/`
**说明**`curl -T` 发送 PUT 时 offset 为 0 且一次性传完整文件适合单块场景;多块时请自行递增 `offset`
## 行为与健康映射
- X3D 输出 `feeding` / `normal` / `scared` → 中文 **吃饵** / **正常游行** / **惊吓**
- 健康:`scared`**不健康**,其余 → **健康**(启发式,可后续换专用模型)
## 视频 URL
FishMeasure 跑完后在输出目录查找 `*preview*.mp4`,复制到 `MEDIA_ROOT/`,文件名为 `{UTC时间戳}_{svo_stem}_left.mp4` / `_right.mp4`(每次测量不覆盖;仅一个预览文件时可能左右 URL 指向同一逻辑源经 SBS 拆分)。确保 `PUBLIC_BASE_URL` 与前端/文档中的域名端口一致。
## Weight Rule (Current)
最终体重 `pred_weight_g` 由以下规则链决定(按优先级从高到低):
@@ -93,7 +33,7 @@ FishMeasure 跑完后在输出目录查找 `*preview*.mp4`,复制到 `MEDIA_RO
1. **440g 全池均值保护**(规则 B`avg_g_filtered`(所有 candidates 均值)> `--mean-pool-fallback-max-if-over-g`(默认 440g`pred_weight_g = max_predicted_weight_g_after_filter``pred_weight_rule = "max_after_filter_high_mean_pool_over_g"`
2. **400g mean-all fallback**(规则 A`--average-all-after-filter` 开启时):若全池 mean > `--average-all-fallback-max-if-mean-over-g`(默认 400g`pred_weight_g = max_predicted_weight_g_after_filter``pred_weight_rule = "max_after_filter_high_mean_all"`
3. **`--average-all-after-filter`**(默认关):全部 candidates 均值作为最终值,`pred_weight_rule = "mean_all_filtered"`
4. **Top-K 聚合**(默认路径):按 `--top-by-length`(默认开)选 top-K 帧,candidates < 5 用 max 否则用 mean`pred_weight_rule = "top_k_aggregate"`
4. **Top-K 聚合**(默认路径):按 `--top-by-length`(默认开)选 top-K 帧,对选中帧的预测重量取**算术平均**(与日志中 `top{k}_avg` 一致)`pred_weight_rule = "top_k_aggregate"`
DGCNN 明细中同时输出 `mean_all_pred_g_after_filters``avg_topk_mean_pred_g` 等供对比参考。
@@ -103,16 +43,22 @@ DGCNN 明细中同时输出 `mean_all_pred_g_after_filters`、`avg_topk_mean_pre
调用 [`app/services/measure.py`](app/services/measure.py) 中的 `run_measure_batch_subprocess`,配置与 `fish_api/.env` 相同(`get_settings()`)。
**必须在 `fish_api` 下执行 `python -m app...`,或从仓库根用下面脚本**;若在仓库根直接运行 `python -m app.measure_debug_cli`,会因找不到 `app` 包报错(`ModuleNotFoundError: No module named 'app'`)。
```bash
# 方式 A仓库根推荐
bash scripts/measure_debug.sh --fish-id 14
# 方式 B先进入 fish_api
cd fish_api
uv sync
# 默认MEASURE_WATCH_DIR/fish{N}/ 下所有 .svo2 → 输出到 MEASURE_OUTPUT_ROOT/fish{N}(默认 fish_api/.data/measure_output/fish{N}
uv run python -m app.measure_debug_cli --fish-id 1
# 或等价入口
# 或等价入口(须在 fish_api 目录)
uv run fish-measure-debug --fish-id 1
# 指定 SVO 目录或输出目录
# 指定 SVO 目录或输出目录(在 fish_api 目录下)
uv run python -m app.measure_debug_cli --batch-folder /path/to/fish1 --fish-id 1 --output-root /path/to/out
```

View File

@@ -2,7 +2,7 @@
FishMeasure / FishAction 子进程不连接、不依赖本库;它们只读写各自文件(如 measure_output 下
weight_prediction.json、临时 pred.json 等),由 fish_api 在子进程结束后读文件并写入本表。
预览视频在 media_rootstart_fresh 会清空 measure_output、media、ingest 临时目录。
预览视频在 media_rootstart_fresh 默认仅重置投递游标,设置 CLEAR_* 环境变量可按需清空各目录。
"""
from __future__ import annotations
@@ -718,7 +718,7 @@ def _safe_rm_tree(path: Path) -> None:
def clear_runtime_compute_dirs(settings: Settings) -> None:
"""清空 FishMeasure / FishAction 运行时目录、托管预览、ingest 临时文件(保留目录本身)。
与 remove_sqlite_database_files 一起在启动脚本中调用,使两条流水线重启后均重新计算
注意start_fresh 默认不再调用此函数。仅供手动或脚本显式调用
"""
for base in (
settings.measure_output_root,

View File

@@ -1,4 +1,4 @@
"""启动前清空状态:默认重置客户端游标,保留 SQLite 历史快照
"""启动前清空状态:默认**仅**重置客户端投递游标,其余全部保留。
由 start_fresh.sh 在 uvicorn 之前调用。
- 默认保留 SQLite 历史数据,仅重置 client_id 投递游标fresh 语义)
@@ -6,6 +6,10 @@
- 默认保留 measure_output 以复用中间步骤(点云等)
- 设置 CLEAR_MEASURE_OUTPUT=1 清空测量输出目录
- 设置 CLEAR_ACTION_OUTPUT=1 清空行为输出目录
- 默认保留 media_root预览视频和 stream_tmp_dir避免视频链接 404
- 设置 CLEAR_MEDIA=1 清空媒体目录
- 设置 CLEAR_STREAM_TMP=1 清空流临时目录
- 设置 CLEAR_LEGACY_JSON=1 清除旧版 JSON 状态文件
"""
from __future__ import annotations
@@ -65,21 +69,31 @@ def run_prestart_fresh() -> None:
else:
print(f"[prestart-fresh] kept action_output: {s.action_output_root}", flush=True)
# 始终清空媒体根目录和流临时目录(这些是可以重新生成的
_safe_rm_tree(s.media_root)
_safe_rm_tree(s.stream_tmp_dir)
print(
f"[prestart-fresh] cleared media and stream_tmp: "
f"media={s.media_root}, stream_tmp={s.stream_tmp_dir}",
flush=True,
)
# 媒体和流临时目录:默认保留(避免重置游标后视频链接 404
clear_media = os.environ.get("CLEAR_MEDIA", "").strip() in ("1", "true", "yes")
clear_stream_tmp = os.environ.get("CLEAR_STREAM_TMP", "").strip() in ("1", "true", "yes")
if clear_media:
_safe_rm_tree(s.media_root)
print(f"[prestart-fresh] cleared media: {s.media_root}", flush=True)
else:
print(f"[prestart-fresh] kept media for reuse: {s.media_root}", flush=True)
if clear_stream_tmp:
_safe_rm_tree(s.stream_tmp_dir)
print(f"[prestart-fresh] cleared stream_tmp: {s.stream_tmp_dir}", flush=True)
else:
print(f"[prestart-fresh] kept stream_tmp for reuse: {s.stream_tmp_dir}", flush=True)
# 清理旧版 JSON 状态文件(数据已迁移到 SQLite
if s.measure_watch_use_state_file and s.measure_watch_dir is not None:
_rm_legacy_json(s.measure_watch_dir / ".fishmeasure_watch_processed.json")
clear_legacy_json = os.environ.get("CLEAR_LEGACY_JSON", "").strip() in ("1", "true", "yes")
if s.action_watch_use_state_file and s.action_watch_dir is not None:
_rm_legacy_json(s.action_watch_dir / ".fishaction_watch_processed.json")
if clear_legacy_json:
if s.measure_watch_use_state_file and s.measure_watch_dir is not None:
_rm_legacy_json(s.measure_watch_dir / ".fishmeasure_watch_processed.json")
if s.action_watch_use_state_file and s.action_watch_dir is not None:
_rm_legacy_json(s.action_watch_dir / ".fishaction_watch_processed.json")
print("[prestart-fresh] done.", flush=True)

View File

@@ -73,8 +73,6 @@ async def get_real_camera(
"video_left": m.video_left,
"video_right": m.video_right,
}
if m.calculation_log:
payload["calculation_log"] = m.calculation_log
return JSONResponse(
content={

View File

@@ -62,6 +62,26 @@ def _predict_weigth_from_svo2_extra_args(settings: Settings) -> List[str]:
# PLY 复用控制:由 .env 中的 MEASURE_REUSE_EXISTING_CLOUDS 决定
if not settings.measure_reuse_existing_clouds:
out.append("--no-reuse-existing-clouds")
# 体重聚合规则
out.extend(["--weight-top-k", str(settings.measure_weight_top_k)])
if settings.measure_weight_top_by_length:
out.append("--weight-top-by-length")
else:
out.append("--no-weight-top-by-length")
out.extend(["--weight-length-switch-mm", str(settings.measure_weight_length_switch_mm)])
out.extend(["--weight-max-length-mm", str(settings.measure_weight_max_length_mm)])
out.extend(["--weight-min-length-width-ratio", str(settings.measure_weight_min_length_width_ratio)])
if settings.measure_weight_average_all_after_filter:
out.append("--weight-average-all-after-filter")
else:
out.append("--no-weight-average-all-after-filter")
out.extend(["--weight-average-all-fallback-max-if-mean-over-g", str(settings.measure_weight_avg_all_fallback_max_g)])
out.extend(["--weight-mean-pool-fallback-max-if-over-g", str(settings.measure_weight_mean_pool_fallback_max_g)])
if settings.measure_weight_remove_outliers:
out.append("--weight-remove-outliers")
out.extend(["--weight-outlier-method", settings.measure_weight_outlier_method])
return out

View File

@@ -66,7 +66,7 @@ class Settings(BaseSettings):
fish_measure_root: Path = fish_repo_root() / "FishMeasure"
fish_action_root: Path = fish_repo_root() / "FishAction"
#: FishMeasure 推理输出(与 SQLite、媒体缓存同属 fish_api/.data启动脚本清空)
#: FishMeasure 推理输出(与 SQLite、媒体缓存同属 fish_api/.data启动脚本默认保留,设置 CLEAR_MEASURE_OUTPUT=1 可清空)
measure_output_root: Path = fish_repo_root() / "fish_api" / ".data" / "measure_output"
#: 体重推算过程等调试文本写入目录(默认 fish_api/.data/logs/measure。**MEASURE_DEBUG_LOG_DIR**
measure_debug_log_dir: Path = Field(
@@ -78,7 +78,7 @@ class Settings(BaseSettings):
default=True,
validation_alias=AliasChoices("MEASURE_DEBUG_LOG_WRITE", "measure_debug_log_write"),
)
#: FishAction 侧预留目录(与 measure 对称;当前推理多为临时文件,仍随启动清空)
#: FishAction 侧预留目录(与 measure 对称;启动脚本默认保留,设置 CLEAR_ACTION_OUTPUT=1 可清空)
action_output_root: Path = Field(default_factory=_default_action_output_root)
python_fish_measure: str = ""
@@ -160,6 +160,78 @@ class Settings(BaseSettings):
),
)
# ── 体重聚合规则(传给 predict_weigth_from_svo2.py → test_dgcnn_weight_estimator.py ──
#: DGCNN top-K 帧数,传给 ``--weight-top-k``。**MEASURE_WEIGHT_TOP_K**
measure_weight_top_k: int = Field(
default=5,
ge=1,
validation_alias=AliasChoices("MEASURE_WEIGHT_TOP_K", "measure_weight_top_k"),
)
#: 按长度选 top-K传给 ``--weight-top-by-length``。**MEASURE_WEIGHT_TOP_BY_LENGTH**
measure_weight_top_by_length: bool = Field(
default=True,
validation_alias=AliasChoices(
"MEASURE_WEIGHT_TOP_BY_LENGTH", "measure_weight_top_by_length"
),
)
#: top-K 按长度选时,若 K 个平均长度 > 此值则切为按重量选,传给 ``--weight-length-switch-mm``。**MEASURE_WEIGHT_LENGTH_SWITCH_MM**
measure_weight_length_switch_mm: float = Field(
default=319.0,
validation_alias=AliasChoices(
"MEASURE_WEIGHT_LENGTH_SWITCH_MM", "measure_weight_length_switch_mm"
),
)
#: 几何过滤length > 此值的帧排除,传给 ``--weight-max-length-mm``0 关闭)。**MEASURE_WEIGHT_MAX_LENGTH_MM**
measure_weight_max_length_mm: float = Field(
default=400.0,
validation_alias=AliasChoices(
"MEASURE_WEIGHT_MAX_LENGTH_MM", "measure_weight_max_length_mm"
),
)
#: 几何过滤PCA 长/宽 < 此值的帧排除,传给 ``--weight-min-length-width-ratio``0 关闭)。**MEASURE_WEIGHT_MIN_LENGTH_WIDTH_RATIO**
measure_weight_min_length_width_ratio: float = Field(
default=1.5,
validation_alias=AliasChoices(
"MEASURE_WEIGHT_MIN_LENGTH_WIDTH_RATIO", "measure_weight_min_length_width_ratio"
),
)
#: 全池均值模式,传给 ``--weight-average-all-after-filter``。**MEASURE_WEIGHT_AVERAGE_ALL_AFTER_FILTER**
measure_weight_average_all_after_filter: bool = Field(
default=False,
validation_alias=AliasChoices(
"MEASURE_WEIGHT_AVERAGE_ALL_AFTER_FILTER", "measure_weight_average_all_after_filter"
),
)
#: 全池均值 > 此值时改用 max规则 A传给 ``--weight-average-all-fallback-max-if-mean-over-g``0 关闭)。**MEASURE_WEIGHT_AVG_ALL_FALLBACK_MAX_G**
measure_weight_avg_all_fallback_max_g: float = Field(
default=400.0,
validation_alias=AliasChoices(
"MEASURE_WEIGHT_AVG_ALL_FALLBACK_MAX_G", "measure_weight_avg_all_fallback_max_g"
),
)
#: 全池 candidates 均值 > 此值时改用 max规则 B, 440g 保护),传给 ``--weight-mean-pool-fallback-max-if-over-g``0 关闭)。**MEASURE_WEIGHT_MEAN_POOL_FALLBACK_MAX_G**
measure_weight_mean_pool_fallback_max_g: float = Field(
default=440.0,
validation_alias=AliasChoices(
"MEASURE_WEIGHT_MEAN_POOL_FALLBACK_MAX_G", "measure_weight_mean_pool_fallback_max_g"
),
)
#: 异常值剔除开关,传给 ``--weight-remove-outliers``。**MEASURE_WEIGHT_REMOVE_OUTLIERS**
measure_weight_remove_outliers: bool = Field(
default=False,
validation_alias=AliasChoices(
"MEASURE_WEIGHT_REMOVE_OUTLIERS", "measure_weight_remove_outliers"
),
)
#: 异常值剔除方法iqr / zscore传给 ``--weight-outlier-method``。**MEASURE_WEIGHT_OUTLIER_METHOD**
measure_weight_outlier_method: str = Field(
default="iqr",
validation_alias=AliasChoices(
"MEASURE_WEIGHT_OUTLIER_METHOD", "measure_weight_outlier_method"
),
)
#: 非空时由 fish_api 在后台持续扫描该目录中的新 MP4 并跑 FishAction与 ingest 共用 SQLite 最新结果)
action_watch_dir: Optional[Path] = None
action_watch_poll_interval: float = Field(default=2.0, ge=0.1)

View File

@@ -1,6 +1,6 @@
#!/usr/bin/env bash
# 默认重置 client_id 投递游标后启动 Fish APIuvicorn保留 SQLite 历史快照
# 默认保留 measure_output 中间步骤(点云等)以加速重新处理
# 默认**仅**重置 client_id 投递游标后启动 Fish APIuvicorn其余全部保留。
# SQLite 历史快照、measure_output、media预览视频、stream_tmp 均保留
#
# bash fish_api/start_fresh.sh
# PORT=8001 HOST=0.0.0.0 bash fish_api/start_fresh.sh
@@ -12,6 +12,10 @@
# CLEAR_MEASURE_OUTPUT=1 bash fish_api/start_fresh.sh
# CLEAR_MEASURE_OUTPUT=1 CLEAR_ACTION_OUTPUT=1 bash fish_api/start_fresh.sh
#
# 强制清空媒体/流临时目录:
# CLEAR_MEDIA=1 bash fish_api/start_fresh.sh
# CLEAR_STREAM_TMP=1 bash fish_api/start_fresh.sh
#
# 首次使用请先cd fish_api && uv sync
#
set -euo pipefail

11
scripts/measure_debug.sh Executable file
View File

@@ -0,0 +1,11 @@
#!/usr/bin/env bash
# 从仓库根目录运行单条鱼测量(与 fish_api / measure-watch 同一子进程逻辑)。
# 用法bash scripts/measure_debug.sh --fish-id 14
# 若在 fish_api 目录下可直接python -m app.measure_debug_cli --fish-id 14
set -euo pipefail
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$ROOT/fish_api"
if command -v uv >/dev/null 2>&1; then
exec uv run python -m app.measure_debug_cli "$@"
fi
exec "${PYTHON:-python3}" -m app.measure_debug_cli "$@"