Distributed watchpoints: Debugging large modular robot systems

Distributed systems frequently exhibit properties of interest which span multiple entities. These properties cannot easily be recognized from any single entity, but can be readily detected by combining the knowledge of multiple entities. Testing for distributed properties is especially important in debugging or verifying software for modular robots. We have developed a technique we call distributed watchpoint triggers which can efficiently recognize distributed conditions. Our watchpoint description language can handle a variety of temporal, spatial and logical properties spanning multiple robots. In this paper we present the specification language, describe the distributed online mechanism for detecting distributed conditions in a running system and evaluate the performance of our implementation.