Google’s AVA dataset raises the bar for identifying human actions in videos


Today Google announced a new labeled dataset of human actions taking place in videos. That may sound obscure, but it’s a big deal for anyone working to solve problems in computer vision.

If you’ve been following along, you’ve noticed the significant uptick in companies building products and services that act as a second pair of human eyes. Video detectors like Matroid, security systems like Lighthouse and even autonomous cars benefit from an understanding of what’s going on inside a video and that understanding is born on the back of good labeled datasets for training and benchmarking.

Google’s AVA is short for atomic visual actions. In contrast to other datasets, it takes things up a notch by offering multiple labels for bounding boxes within relevant scenes. This adds more detail in complex scenes and makes for a more rigorous challenge for existing models.

In its blog post, Google does a great job explaining what makes human actions so difficult to classify. Actions, unlike static objects, unfold over time — simply put, there’s more uncertainty to solve for. A picture of someone running could actually just be a picture of someone jumping, but over time, as more and more frames are added, it becomes clear what is really happening. You can imagine how complicated things could get with two people interacting in a scene.

AVA consists of over 57 thousand video segments labeled with 96 thousand labeled humans and 210 thousand total labels. The video segments, pulled from public YouTube videos, are each three seconds long. These segments were then labeled manually using a potential list of 80 action types like walking, kicking or hugging.

If you’re interested in tinkering, you can find the full dataset here. Google first explained its efforts to create AVA in an paper that was published on ArXiv back in May and updated in July. Initial experimentation covered in that paper showed that Google’s dataset was incredibly difficult for existing classification techniques — displayed below as the contrast between performance on the older JHMDB dataset and performance on the new AVA dataset.

Source link