Large-scale text processing pipeline with Apache Spark

被引：0

作者：

Svyatkovskiy, A. ^{[1
,2
,3
]}

Imai, K. ^{[1
]}

Kroeger, M. ^{[1
]}

Shiraito, Y. ^{[1
]}

机构：

[1] Princeton Univ, Princeton, NJ 08544 USA

[2] Princeton Univ, Dept Polit, Princeton, NJ 08544 USA

[3] Princeton Univ, Ctr Stat & Machine Learning, Princeton, NJ 08544 USA

来源：

2016 IEEE INTERNATIONAL CONFERENCE ON BIG DATA (BIG DATA) | 2016年

关键词：

Spark; Avro; Spark ML; Spark GraphFrames; INNOVATIONS; DIFFUSION; STATES;

D O I：

暂无

中图分类号：

TP [自动化技术、计算机技术];

学科分类号：

0812 ;

摘要：

In this paper, we evaluate Apache Spark for a data-intensive machine learning problem. Our use case focuses on policy diffusion detection across the state legislatures in the United States over time. Previous work on policy diffusion has been unable to make an all-pairs comparison between bills due to computational intensity. As a substitute, scholars have studied single topic areas. We provide an implementation of this analysis workflow as a distributed text processing pipeline with Spark dataframes and Scala application programming interface. We discuss the challenges and strategies of unstructured data processing, data formats for storage and efficient access, and graph processing at scale.

引用

页码：3928 / 3935

页数：8