The Design and Implementation of Vector Autoregressive Model and Structural Vector Autoregressive Model Based on Spark

VAR (Vector Auto-regressive) model is a kind of commonly used econometric-model. It is used to estimate the dynamic relationship of the endogenous variables without any prior constraints. Since VAR is one of the most easily operated models to deal with the analysis and prediction of multiple related economic indicators, more and more attention has been paid by economists in two decades. However, with the increasing of data size, the individual computer has encountered its processing bottleneck. Meanwhile, the advantages of the distributed computing cluster have begun to show obvious strength, such as Hadoop, Spark, and so on. Due to the lack of VAR related model on Spark, MLlib, we developed approaches of VAR and SVAR (Structural Vector Auto-regression) model in Spark and Hadoop cluster. Meanwhile, SGD (Stochastic Gradient Descent) algorithm has been applied after the data processing. To verify the approaches, different sizes of data are used for model testing in different platform, including R and Spark cluster. According to the comparison of the response time of different data size in both platform, the experiment results have shown that the developed methods are simple and efficient in big data environment.