Multi-layer Stream Orchestration with Flange

With advances in embedded sensor hardware and the resulting deluge of measurement data, it has become increasingly clear that corresponding advances in data management are necessary at every level of the data pipeline to keep pace with the copious output of sensors at the edge. Filtering and processing data in-flight from edge to core presents an attractive possibility. By using resources within the network fabric itself, data can be reduced to a manageable level before reaching the core. Such drastic changes in data processing methodology do present new challenges and exacerbate old ones. Manual configuration and optimization at the fog scale is error prone. This paper discusses the Flange DSL and our in-flight processing framework InLocus, as a solution for driving fog configuration and management with declarative network topology assertions. By distributing configurable processing agents within the network fabric and modeling the network as a graph, the Flange compiler is capable of resolving terse, general flow assertions into network configurations at the device level. By chaining many such network functions together, the network topology can be transformed into a single virtual computation unit. Unlike previous approaches, the processing overlay topology is compiled automatically from the requirements of the task and the state of the underlying network. Each hop in the network provides an opportunity to operate on the data as it passes through, resulting in transformed data at the destination. The Flange language provides a propositional-logic based structure for selecting network components and asserting desired properties onto them. By using flow predicates, a program may define arbitrarily long chains of operations which are compiled into instructions to network agents that enact real behaviors on the network. The Flange compiler itself performs on moderately sized network graphs; compiling optimal placement schemes in under half a minute for graphs below 2,000 vertices when a solution exists. This performance scales to graphs in excess of 10,000 vertices, resolving in just over 3 minutes. The realization phase demonstrates faster performance; routing and computation results can be observed in the network in as little as one second over representative wide-area topologies. The fast iteration cycle provides developers with the ability to quickly identify processing or network issues in development or deployment.