A Deep Reinforcement Learning-Based Distributed Service Offloading Method for Edge Computing Empowered Internet of Vehicles

The increasing number of vehicles, along with the development of the fifth-generation (5G) wireless communication technology, has made the interconnections between vehicles and other objects (e.g., pedestrians, infrastructures, and service platforms) become a reality, which forms a novel networking paradigm: the Internet of Vehicles (IoV). In the IoV, due to the rapid speed of the vehicles, services such as route recommendation and collision warning are required to be satisfied in time. Thanks to the birth of edge computing, which deploys resources (e.g., computation, storage, and bandwidth) at the side close to the users, thereby reducing the transmission latency and alleviating the network load, service providers can efficiently serve users with low-latency services by introducing edge computing into the IoV. Nevertheless, since the edge servers are often limited with insufficient resources, problems such as overload would occur if all the services requested by the IoV users are offloaded to the edge servers for executing, which will significantly slow down the processing speed and reduce the quality of service (QoS) provided by the edge servers. Therefore, how to allocate the limited computation and bandwidth resources of the edge servers to the IoV services and determine the offloading destinations of the services to serve the IoV users with low-latency services still remains enormous challenge. Toward this end, an end-edge-cloud collaborative computing framework for 5G-enabled IoV is proposed in this paper. Based on this framework, a distributed service offloading method with asynchronous advantage actor-critic (A3C), named D-SOAC, is developed to figure out the optimal service offloading strategy. Specifically, by leveraging the deep spatio-temporal residual network (ST-ResNet), D-SOAC predicts the future service requirements from the IoV users in each road segment firstly and sends them to the local edge server deployed in the road segment. Secondly, through combining the local future service requirements with the local network condition (e.g., transmission power and channel gain) and the local resource condition (e.g., remaining computation resources and bandwidth resources of the local edge server) into local system states, each edge server feeds the local system state into the local actor network to obtain the preliminary service offloading strategy. Technically, to avoid dimension explosion of action space in A3C, a multi-output actor network is introduced. Thirdly, based on the temporal difference (TD) error, the local critic network evaluates the preliminary offloading strategy and calculates its parameter gradient, which further guides the gradient ascent of the local actor network for gradient accumulation. After the accumulation of the parameter gradient, the local network pushes the accumulated gradient to the global network in the cloud center for parameter updating and pulls the updated global network parameters back to the local networks afterward, thereby collaborating with the global network in optimizing the preliminary service offloading strategy steadily and obtaining the optimal service offloading strategy. Eventually, extensive experimental evaluations of D-SOAC are conducted based on a big real-world service requirements dataset. The experiment results demonstrate that D-SOAC decreases the average service latency by 0.4% to 20.4% compared with four exiting service offloading methods in different IoV environments, proving the effectiveness and efficiency of D-SOAC. © 2021, Science Press. All right reserved.