High performance data deployment and processing

High data rates from modern telescopes are requiring high performance computing. New large survey projects in particular, like the Dark Energy Survey (DES) and the Large Synoptic Survey Telescope (LSST), are placing high demands on computing and data management systems not only in their need for fast turn-around or real-time processing, but in the expectation of multiple reprocessings of the collection as software and algorithms improve. In these applications, I/O and data movement can often incur the largest performance overhead. Even when high network bandwidth is available, filling the network pipe to its capacity is non-trivial. We present a prototype system for a data deployment tool that can make intelligent use of high-performance data transfer software to move data in a fast and efficient way. It accomplishes this by asking the application not only what data is needed and where, but also for what purpose (e.g. last nights observations on that cluster for calibration). We are developing this tool to support automated processing systems for DES and LSST.