Data management pipeline and hardware facilities for J-PAS and J-PLUS surveys archiving and processing

The Observatorio Astrofisico de Javalambre (OAJ) is designed to carry out large surveys. It consists of two main telescopes: JST/T250, a 2.5m telescope with a FoV of 3 deg diameter, and JAST/T80, a 80cm with a 2 deg FoV. JST/T250 will be devoted during the first 6 years of operation to complete the Javalambre-PAU Astronomical Survey (J-PAS). It is a photometric survey in a system of 54 filters of 14nm width (from similar to 350 nm to similar to 900 nm) plus several broad band filters covering a total area of 8500 square degrees. The JAST/T80 will start operations during 2014 and will be devoted to perform the J-PLUS survey, covering in 2 years the same area as J-PAS using a system of 12 filters. Both surveys will produce similar to 1 PB of raw data, a single data release of the final combined mosaics is similar to 850 TB. The archive and processing will be done in a dedicated data center (UPAD) that will be equipped with > 3 PB to store the data produced by the surveys. This contribution presents the software and hardware architecture designed to store and process the data. The processing pipeline is executed in a computing cluster under a batch queuing system. In a first stage it is devoted to correct instrumental signature on the science images, to perform astrometric and photometric calibration, and the computation of individual image catalogs. In a second stage, the pipeline performs the combination of the mosaics in each sky position and computation of final catalogs. The processing software is connected with a management database to store persistent information about the pipeline operations done on each frame. The information in the final objects' catalogs used by the scientific community will be ingested in a science database. Once the J-PAS survey is finished the information stored in database will be similar to 20 TB. Regarding the storage system, it will combine disk and tape technologies. The disk storage system will have capacity to store the data that is accessed by the pipeline, providing a high bandwidth to feed several hundreds of concurrent processes. The tape library will store and archive the raw data and earlier data releases with lower access frequency. The storage and processing system for the J-PLUS data is already deployed. It contains the job scheduling nodes, database and service nodes for the whole UPAD. For J-PLUS the disk archive have a capacity of similar to 100 TB. It contains four processing nodes, and two nodes to manage the science databases using a clustered SQL database engine.