Improving the accuracy of automatic tennis video annotation by high level grammar

Automatic interpretation of videos in general and sports videos in particular relies heavily on the use of contextual information. The contextual information is encapsulated ill form of grammars capturing the permissible relationships between the static and dynamic objects of the video content. The grammar is invariably multilevel, characterising the spatiotemporal relationships of pixels at the lowest level of analysis, and governing the interaction of objects and the evolution of concepts at the highest level of abstraction. An automatic video analysis system has been developed that applies multilevel contextual reasoning to understand and annotate tennis video. The system that exploits grammatical rules up to the point level achieves an impressive accuracy of 87.5%. We demonstrate that by adding an extra level in the contextual hierarchy which imposes constraints on the evolution of tennis video at the game level, further performance improvements call be gained.