DLToDW: Transferring Relational and NoSQL Databases from a Data Lake

Over the past decade, digital transformation has led to the evolution of databases towards Big Data. A need to collect and analyze data from different sources has emerged. At the same time, traditional decision support systems are unable to meet the growing needs of modern businesses to integrate and analyze a wide variety of generated data. As a result, most organizations need to convert their data stored in relational systems to NoSQL or "Not only SQL" systems that are based on flexible models and schemas. Our work is part of a medical application that must allow health professionals to analyze complex data for decision making. We propose mechanisms to extract data from a Data Lake and store them in a NoSQL Data Warehouse. This will allow to perform, in a second time, decisional analysis facilitated by the features offered by NoSQL systems (richness of data structures, query language, access performances). In this article, we present a process for ingesting data from a Data Lake into a Data Warehouse. The ingestion consists, first, in transferring relational and NoSQL DBs extracted from the Data Lake into a single NoSQL DB (the Data Warehouse), second, in merging so-called "similar" classes and third, in converting the links into references between objects. To automate this process, we used the Model Driven Architecture (MDA) which provides a schema transformation environment. From the physical schemas describing a Data Lake, we propose transformation rules that allow to create a Data Warehouse stored under a document-oriented NoSQL system. An experimentation has been performed for a medical application. © 2022, The Author(s), under exclusive licence to Springer Nature Singapore Pte Ltd.