Comparing Evaluation Protocols on the KTH Dataset

Human action recognition has become a hot research topic, and a lot of algorithms have been proposed Most of researchers evaluated their performances on the KTH dataset. but there is no unified standard how to evaluate algorithms on this dataset Different researchers have employed different test setups, so the comparison is not accurate. lair or complete In order to know how much difference there is when different experimental setups are used, we take our own spatiotemporal MoSIFT feature as an example to assess its performance on the KTH dataset using different test scenarios and different partitioning of the data In all experiments, support vector machine (SVM) with a chi-square kernel is adopted First, we evaluate performance changes resulting from differing vocabulary sizes of the codebook, and then decide on a suitable vocabulary size of codebook Then, we train the models using different training dataset partitions, and test the performances one the corresponding held-out test sets Experiments show that the best performance of MoSIF7 can reach 96 33% on the KTH dataset When different n-fold cross-validation methods are used, there can be up to 10 67% difference in the result And when different dataset segmentations are used (such as KTH1 and KTH2), the difference in results can be up to 5 8% absolute In addition, the performance changes dramatically when different scenarios are used in the training and test dataset When training on KTH1 S1+S2+S3+S4 and testing on KTH I SI and S3 scenarios, the performance can reach 97 33% and 89 33% respectively This paper shows how different test configurations can skew results, even on standard data set The recommendation is to use a simple leave-one-out as the most easily replicable clear-cut partitioning