A Lightweight, Fault-Tolerant Middleware for a Subaru Telescope Second Generation Observation Control System

Subaru Telescope is developing a second-generation Observation Control System that specifically addresses some of the shortcomings of the current Subaru OCS. Two areas of concern are complexity and failure handling. The current system has over 1000 dedicated OCS processes spread across a dozen hosts and provides nothing in the way of automated failover. Furthermore, manual failover is so fraught with difficulty that it is rarely attempted. Our Generation 2 OCS is written almost entirely in Python and builds upon a Subaru-developed middleware based on the XML-RPC protocol. This framework offers the following benefits: has very few dependencies outside of standard Python provides a nearly seamless remote proxy object-oriented interface provides optional user/password authentication and/or SSL encryption is extremely simple to use from client applications is connectionless, and assists transparent failover of communications and distributed services on a set of hosts has reasonable performance for a wide range of needs allows multiple language bindings for dynamic languages, requires no interface stub files The "back end" (service side) of the OCS is nearing completion, and has already been used successfully during two separate OCS engineering runs. It is comprised of only a couple dozen processes, and provides automated failover capabilities on a rack of commodity x86 Linux servers. We provide an overview of the middleware use, its design and its failover capabilities. Some data on the performance of communications using the middleware protocol is included.