Tracking a beach volleyball game from one phone camera
· 2 min read
What you see above
Given a video, the software finds and tracks the ball and four players, works out every moment the ball was touched, by whom, and if it was a pass, hit, dig, etc.. It also detects point boundaries, meaning when a point starts (a serve) and when it ends.
Traditional computer vision in 2026
In 2026 we'd hope to just give a video clip to a multimodal llm like Gemini and ask it to output a timeline of coordinates for the players, the ball, and each contact between player and ball in a 2d plane. This is something a human coach could do with decent accuracy.
Gemini was able to output some plausible movements, but they weren't accurate and filled in a lot of gaps with hallucination of ball and player trajectories. So in the path from traditional computer vision to multimodal LLMs, I'd put us somewhere around 70%.
The approach
For this tracker, I got the best results with a mix of strategies.
Neural nets to find things
Meta's Segment Anything Model 3 prompted with the text "person" gives persistent player tracks and pixel masks. I used VballNet, an adaptation of TrackNetV4 that was designed for indoor volleyball to track the ball.
YOLO11-pose gave wrist keypoints, which I use in conjunction with player locations to improve prediction of who actually made contact with the ball.
Classical estimation to reason about them
A Kalman filter helps process the tracking data in image space. Dynamic programming is used for re-acquiring the ball after it's lost, by building a graph where each frame of the video + detected ball location is a node and edges are decided by physically plausible movement. The physically-plausible movement is the hard part, but you can use body heights, the net, lines, etc. as reference points to attempt to ground in physics.
Luckily we don't need to be perfect, just to have our "most likely" candidate be correct. This strategy was also used for assigning contacts to players.
A VLM for the semantics
Gemini 3.1 Pro gives timestamps for when a serve starts a point and when the point is over. The other task Gemini was given was to classify the types of touches (bump set, hand set, dig, hit) but this wasn't very accurate - perhaps due to the low quality footage.
Next steps
If your goal was to have robust tracking today, you'd probably want to set up multiple cameras. That said, if a human can review footage from a single phone and estimate positions, players, hit types, etc., why can't an AI model do the same?
The next step for this work is to iteratively test more tasks with the multimodal LLMs, increasing the end-to-end model's responsibility. Until Gemini 4 does this in one shot.