Applying Big Data technologies to high tech manufacturing

The systematic analysis of ever-increasing data collection presents companies with ever-greater challenges. Many manufacturing organizations simply lack the know-how to handle Big Data projects and the corresponding data analysis right. Therefore one simply follows the current trends and buzz words and adopts approaches which are currently en vogue. This approach often leads to less successful projects and several regularly reoccurring patterns of misconceptions can be identified. This paper highlights some of these unsuccessful patterns and introduces some of the work done in the PRO-OPT SMART-DATA research project. The innovation in this data analysis approach is the combination of traditional statistical methods with new Big Data and AI analysis techniques applied to high tech manufacturing. Being able to align process data with the complete metrology data provides amazing new insights into the manufacturing. Furthermore, we will introduce a new visualization technique specifically suited for domains with high amounts of categorical data like semiconductor, photovoltaics, electronics and such. This paper will show how the combination of the statistical data analysis system Cornerstone in conjunction with Apache Spark(1) and Apache Cassandra(2) provides a good basis for engineering analytics of massive data amounts. By properly nesting the solid mathematical methods in Cornerstone with big data-appropriate infrastructure such as Apache Spark and, in our case, Apache Cassandra, many new analytics issues can be addressed. Analyzes that used to be inefficient due to the sheer volume of data in classically modeled schema's can now be performed through appropriate big-table modeling and provide the ability to provide completely new insights into production data. Those directly impacted the manufacturing procedures and improved the products quality and reliability. Experiences gained in the project impacted the upcoming VDI/VDE guideline 3714(3) to be published later in the year 2019.