M2T: A Framework of Spatial Scene Description Text Generation based on Multi-source Knowledge Graph Fusion

Natural language is an effective tool for humans to describe things, with diversity and ease of dissemination, and can contain human spatial cognitive results. How to use natural language to describe geographic spatial scenes has always been an important research direction in spatial cognition and geographic information science, providing important application values in personalized unmanned tour guides, blind navigation, virtual space scene interpretation, and so on. The essence of natural language description of geographic spatial scenes is the process of transforming the two-dimensional vector of geographic space into a one-dimensional vector in word space. Traditional models perform well in handling spatial relationships, but are somewhat inadequate in natural language description: (1) spatial relationship description models are one-way descriptions of the environment by humans, without considering the impact of the environment on the description; (2) spatial scenes emphasize traversal-based descriptions of spatial relationships, where each set of spatial relationships is equally weighted, which is inconsistent with the varying attention paid by humans to geographic entities and spatial relationships in the environment; (3) the spatial relationship calculation of traditional models is a static description of a single scene, which is difficult to meet the requirement of dynamic description of continuous scenes in practical applications; (4) the natural language style of traditional models is mechanical, lacking necessary knowledge support. This article proposes a spatial scene natural language generation framework Map2Text (M2T) that integrates multiple knowledge graphs. The framework establishes knowledge graphs for spatial relationships, language generation style, and spatial attention, respectively, and realizes the fusion of multiple knowledge graphs and the generation of natural language descriptions of spatial scenes within a unified framework. The spatial scene description knowledge graph solves the pruning problem of traversing spatial relationships, and establishes the relationship between spatial scenes by building a spatial relationship graph, supporting continuous expression of spatial scenes; the natural language style knowledge graph establishes the relationship between spatial expression and language style, achieving diversified language styles that are appropriate for spatial natural language expression; the spatial attention knowledge graph captures the nuances of natural language spatial expression by establishing an attention matrix based on the interaction state between the subject and object of the spatial scene. An experimental prototype system designed based on the Beijing Forbidden City demonstrates that the system-generated results are close to human travel notes, with more complete content coverage and more diverse styles, verifying the effectiveness of the M2T framework and demonstrating the potential value of natural language description of spatial scenes. © 2023 Journal of Geo-information Science. All rights reserved.