INGEST - A SIMPLE PROGRAM FOR PERFORMING DISTRIBUTED RELATIONAL DATABASE OPERATIONS

The Hubble Space Telescope (HST) and ground system produce approximately 600 gigabytes of data per year in the form of many types of datasets. Because of the formidable size of the total data stream, the datasets are impractical to manage with a conventional database system. Therefore, they are archived onto an optical disk juke-box system. A smaller HST Catalog exists to describe the archived data and manage access to the HST archived data. When the Catalog keywords are queried, the resulting records point to Me names stored in the archive. This allows users to request datasets by their descriptive keywords instead of file names. The Catalog is populated by data from several external databases via the Ingest program. Ingest normalizes and/or joins rows of multiple external database tables and writes the new rows to the HST Catalog. Secondarily, Ingest parses data values, translates data values, and creates row identifiers for each row to be written to the HST Catalog. The Ingest process is driven by translation tables residing in the object database which can be altered on-the-fly. This paper describes the design of the HST Catalog Data Ingest program in more detail. Ingest has proven to be a powerful tool in the HST environment where functional requirements are many, database structures are enormous and both evolve rapidly.