Automatic Hierarchical Reinforcement Learning for Reusing Service Process Fragments

Prevailing research trend is to use Web services for data publishing and sharing among organizations, but existing works often fall short of service reuse. Developing efficient solutions to achieve composite services has drawn significant attention in services computing. Services and service process fragments reuse is critical to improve the efficiency of software development and economize on human and material resources, meanwhile Reinforcement Learning (RL) is one commonly used approach in services computing. However, in service composition and service process fragments (SPFs) reusing scenarios, traditional RL methods cannot guarantee good efficiency for large-scale service processes construction problems. In this paper, we present a novel SPF reusing framework that combines automatic Hierarchical Reinforcement Learning (HRL) and extended Cocke-Kasami-Younger (CKY) algorithm. This framework has the ability to reuse any granularity of SPFs. We firstly get action models and trajectories by means of analysis on historical service process fragments. Furthermore, the "Causal Analysis" identifies the causal relationships among the actions in a trajectory, i.e. returning a causally annotated trajectory (CAT). Then, we utilize the SPF-Hierarchy algorithm to discover a coherent task hierarchy for each service process fragment. Finally, we map the hierarchy obtained from the previous stage to the HRL-CKY algorithm, which can fulfill the reuse and retrieval of any granularity of SPFs. The effectiveness and robustness of our approach are evaluated through a set of experiments.