Neuro-dynamic programming for optimal control of macroscopic fundamental diagram systems

The macroscopic fundamental diagram (MFD) can effectively reduce the spatial dimension involved in dynamic optimization of traffic performance for large-scale networks. Solving the Hamilton-Jacobi-Bellman (HJB) equation takes center stage in yielding solutions to the optimal control problem. At the core of solving the HJB equation is the value function that represents choosing a sequence of actions to optimize the system performance. However, this problem generally becomes intractable for possible discontinuities in the solution and the curse of dimensionality for systems with all but modest dimension. To address these challenges, a neural network is used to approximate the value function to obtain the optimal controls through policy iteration. Furthermore, a saturated operator is embedded in the neural network approximator to handle the difficulty caused by the control and state constraints. This policy iteration can be implemented as an iterative data-driven technique that integrates with the model-based optimal design based on real-time observations. Numerical experiments are conducted to show that the neuro-dynamic programming approach can achieve optimization goals while stabilizing the system by regulating the traffic state to the desired uncongested equilibrium.