FishServer/FishMeasure/docs/ideas.md

Fish Weight Prediction – Ideas

Multi-modal weight prediction (image + point cloud)

Idea: Instead of using only PointNet++ to predict weight from the 3D point cloud, use two separate models and combine their outputs:

1. Image-based weight predictor – A model that predicts fish weight from 2D images (e.g., RGB frames from the video).
2. Point cloud–based weight predictor – The existing PointNet++ model that predicts weight from 3D point clouds.

Then combine the two feature representations (or predictions) to produce a final weight estimate. This could be done by:

• Concatenating features from both encoders and feeding them into a small fusion head.
• Averaging or otherwise combining the two predictions (e.g., weighted average).
• Using a learned fusion module that decides how much to trust each modality.

Rationale: 2D images and 3D point clouds provide complementary information. Images capture texture, color, and fine visual details; point clouds capture geometry and scale. Combining both may improve robustness when one modality is noisy or incomplete (e.g., poor depth, occlusion, or low-quality segmentation).