Join us at 10:00 am (PDT) on Thursday, October 15th for the next meeting of the PNAMP Fish Monitoring Work Group (FMWG). During the meeting, we will update participants on current tasks, hear a presentation about innovative fish monitoring, and provide you with opportunities to engage regional managers and experts. These meetings are an opportune time to discuss tasks and get input from members on where they see importance or could use facilitated support.

Tech Talk: Towards interactive, language-guided tracking of salmon in sonar video

Presenter: Kai Van Brunt, Research Assistant in the Beery Lab at Massachusetts Institute of Technology (MIT) (coauthors are Justin Kay, PhD student; and Sara Beery, Associate Professor, MIT)

Presentation description:

Sonar video cameras are deployed underwater in rivers throughout the Pacific Northwest and Alaska to count the number of salmon migrating each year to spawn, informing the setting of fishing quotas and enabling monitoring of population response to conservation actions. Stakeholders have begun incorporating computer vision-based processing to reduce the time needed to process sonar video into salmon counts. This processing involves a specialized detection model and tracker that tracks fish as they enter and exit the field of view. While these specialized models perform well at river locations that appear in training data, their performance is brittle under environmental shift, causing prediction errors at new deployments and in changing environments.

Correcting erroneous tracking predictions involves collecting new ground truth annotations and re-training tracking models, which is time-consuming and requires machine learning and data science expertise. We evaluated the potential of a new approach to working with imperfect tracking predictions that addresses these problems via an interactive prediction correction workflow taking place as a conversation with a multimodal large language model, which processes video and text jointly and responds with the requested tracks, or refines its earlier predictions based on the user’s comments. Instead of requiring manual labeling from scratch, users guide the model to produce better tracks; instead of requiring fine-tuning, the model updates its predictions in real-time conditioned on the user’s correction prompts. Our experiments with training a model for this task demonstrate that this workflow is possible, but there is much room for improvement.

Agenda

• 10:00 – Welcome and Introductions
• 10:05 – Task Updates
• 10:30 – Tech Talk
• 11:10 – Q&A
• 11:30 – Adjourn

Click link below to join on October 15th

Never been to a FMWG meeting? This meeting is a perfect time to learn about the project, join in, and help guide future work.

Missed a previous FMWG meeting? Check out our YouTube page for presentations or task meetings.

The PNAMP Fish Monitoring Work Group (FMWG), meets quarterly to discuss task progress, share expertise through guest presentations, and support collaboration, communication, and coordination among fish monitoring practitioners for effective monitoring and efficient data sharing. The FMWG is also interested in your work; if you would like to present your innovative fish monitoring ideas or any recent publications at a future meeting please contact Meg Dethloff, mdethloff@psmfc.org