A Study on Big Data Collecting and Utilizing Smart Factory Based Grid Networking Big Data Using Apache Kafka

In the Smart Factory environment of the 4th industrial revolution, much data is generated from equipment, IoT sensors, and a wide range of manufacturing systems. As manufacturing sites are scattered around the world, information exchange between geographically remote factories is ever more necessary. Also, higher quality and effective management can be achieved by integrating and analyzing the collected and refined data and deriving organic results in the ever-rapidly changing manufacturing environment. However, as the main factory consists of a separate network with much data generated, it is highly difficult to gather all data into one and refine it. The most widely used method of data gathering at present has an architecture where data is linked through integration of the centrally configured solutions for data gathering and linkage. In other words, legacy systems most commonly used in manufacturing sites such as ERP, MES, WMS, etc. use the central system called ESB or EAI, to collect data with the SOA method for inter-system data linkage and collection and pass it on to another legacy system. The centralized method is not suitable for gathering and converging data generated from dozens or hundreds of different factories that are regionally dispersed or made up of independent networks and are also extremely vulnerable in terms of security and safety. This article aims to investigate how to stably and effectively exchange and collect data in geographically remote, independent networks using Apache Kafka, one of the big data ecosystems, and how to enable easy analysis of such data so that users can effectively utilize it.