Abstract: Video content is strongly associated with time series and has a strong logical structure. Shot
semantic is a kind of basic unit for understanding video content. From the point view of user cognition,
among shot semantics, there are various context information hidden rather than explicit temporal
association, such as logical and structural association. Obviously, it is important to describe these
context information in an reasonable manner. Firstly, this paper presents a label tree with context label
to represent the structured context as characterization model of video semantic context. Within the label
tree, each shot semantic in a shot semantic sequence is taken as a leaf node and all inner nodes with
context label is adopted to represent the inter-dependencies among its child nodes. More important, its
hierarchical structure, corresponding to the hierarchical model of video content, leads to significant
information gain for video content understanding. Furthermore, it is tough to construct a hierarchical
video semantic context label tree from the shot semantic sequence, which needs to bridge from
sequence space to tree structure space. Then, according to the combined feature of shot semantic sequence and video semantic label tree, this paper uses an SVM-Struct analysis to construct structural
function and loss function for the semantic context and implement the construction of video semantic
context label tree. The experimental results show that video semantic context label tree has a better
characterization ability in many aspects. And SVM-Struct driven analysis ensures the characterization
ability of video semantic label tree with high precision, recall and F1 rate.

Key words: video semantic context label tree, SVM-Struct, semantic context, structure data, video semantic
annotation