Dynamic scheduling for cloud manufacturing with uncertain events by hierarchical reinforcement learning and attention mechanism

Cloud manufacturing provides a platform for many-to-many scheduling of consumer tasks assigned to service providers. The dynamics and uncertainties of the cloud environment pose stringent requirements on the real-time performance and generalizability of scheduling algorithms. Moreover, the continuous variations in environmental states, task scale, and service statuses further complicate decision-making. However, existing dynamic scheduling methods, primarily developed to address static environments and constant scales, fall short of addressing the escalating volatility and complexity of real-world scheduling. To achieve near-real-time decision-making, a deep hierarchical reinforcement learning framework incorporating attention mechanisms and pointer networks is proposed for multi-objective dynamic scheduling in cloud manufacturing. This framework divides the scheduling problem into three subproblems (optimization objectives, manufacturing tasks, and service selection) and leverages a hierarchical structure to realize a three-step scheduling decision- making process. The proposed framework comprises three encoder-decoder-based agents, each corresponding to a subproblem and collaborating to achieve the overall decision. The agents utilize the multi-head attention mechanism to extract inter-task and inter-service relationships, enhancing decision precision and environmental adaptability. Additionally, the pointer network is incorporated into each agent, endowing the proposed framework with generalizability when inserting new tasks (or services) or removing existing ones. Experimental results across nine dynamic scenarios demonstrate that our framework outperforms five deep reinforcement learning algorithms and three meta-heuristics in terms of scheduling performance and runtime. Results from six out-of-training-scale instances further indicate that our framework exhibits superior generalization and scalability.