An In-memory Database Implementation Technique based on Separation of Management, Computation and Storage

Heterogeneous storage/computing platform has been main-stream high performance computing platform with the support of multicore processors， big memory and non-volatile memory techniques. The traditional database engines are co-designed for storage and compute，the emerging databases employ separation of storage and compute and pushdown compute techniques for novel distributed storage infrastructure. This paper introduces a novel in-memory database implementation based on separation of manage， compute and storage technique， based on separation of storage and compute，it further separates the dataset into meta dataset and value dataset according to the characteristics of database schema，data distribution and workload. The unified query engine is divided into meta data management engine，computing engine and storage engine. The meta data with semantic information management is abstracted as independent management layer， the non-semantic value storage and compute are abstracted as compute and storage layers，and the compute-intensive workload is further defined as compute layer，the data-intensive workload is defined as storage layer， the compute layer and storage layer can be combined or separated according to different hardware configurations. The implementation of in-memory database is designed as following levels：1）schema optimization， separating value and meta data in database to choose different storage and compute strategies according to the inner data features； 2） data model optimization， the Fusion OLAP model supports the high performance multidimensional compute on relational storage model；3）algorithm optimizations，using surrogate key index and vector index to support the optimal vector join and vector aggregation for higher OLAP performance；4）system design optimizations，the layered database engine separates the mange and compute，storage and compute，and pushdown multidimensional compute to storage layer. The experimental results show that the separation of manage，compute and storage model can flexibly support hybrid CPU-GPU computing platform，hybrid DRAM-PM （Persistent Memory）storage platform and external storage platform， by employing the open-source in-memory column store Arrow as data storage engine for database and pushing down multidimensional compute to Arrow storage engine，the OLAP implementation proves to be equal performance as OLAP implementation co-designed for storage and compute in Star Schema Benchmark， the OLAP performance outperforms the leading in-memory databases Hyper，OmniSciDB and Arrow based GPU database PG-Strom. © 2023 Science Press. All rights reserved.