Knowledge-centric Natural Language Processing: Acquisition, Representation, and Reasoning

Abstract

Despite the great success achieved by statistical learning theory for building intelligent systems in recent decades, there is still a knowledge gap between what machines and humans know about the real world. And this gap is increasingly larger when the latest neural models become astonishingly bigger (e.g., recent large-scale pre-trained language models). This knowledge gap leads to severe challenges on existing neural models, such as the lack of generalizability, robustness, interpretability, and will inevitably prevent the machine from getting human trust.

To bridge the knowledge gap, a fundamental question to ask is how we can ground the knowledge in machine's head into a similar knowledge space from humans, and teach the machine to develop a similar thinking and reasoning process. By doing so, humans could easily understand machine's rationales and trust them better. Targeting this fundamental question, in this proposal, we present our research efforts and plans towards knowledge-centric natural language processing where external world knowledge needs to be acquired, represented and reasoned for better natural language understanding. Therefore, we present a wide spectrum of challenges to solve, generally including three directions, 1) how to automatically construct structured knowledge from a large text corpus, 2) how to learn neural knowledge representations, and finally, 3) how to build more interpretable and transparent neuro-symbolic systems with knowledge reasoning ability.

This proposal demonstrates our established achievements along with the previous three directions. The introduction first elaborates on our motivation and research vision to construct a holistic and systematic view of knowledge-centric natural language processing. We then describe our contributions distributed into these three directions in each chapter separately. For \textit{knowledge acquisition}, we study extracting structured knowledge (e.g., synonyms, relations) from the text corpus that can be leveraged to build up a better knowledge space. We propose a setting where only corpus-level co-occurrence statistics are available where privacy and personal information can be better preserved. Our proposed framework can fully utilize the surface form and global context information for advanced performance. For \textit{knowledge representation}, we focus on graph representation learning and propose to learn better representations for node pairs for pairwise prediction tasks on graphs, such as link prediction. Our proposed method encourages the interaction between local contexts and would generate more interpretable results. For \textit{knowledg reasoning}, we present two works. The first one presents a self-interpretable framework for medical relation prediction that can generate human-intuitive rationales to explain neural prediction. It relied on a recall and recognition process inspired by the human memory theory of cognitive science. We verify the trustworthiness of generated rationales by conducting a human evaluation of the medical expert. The second one focuses on commonsense reasoning for better word representation learning in which an explicit reasoning module runs over a commonsense knowledge graph to perform multi-hop reasoning. The learned vector representations can be used to benefit downstream tasks and show the reasoning steps as interpretations. 

In the last chapter, we discuss several future works towards the knowledge-centric natural language process. Particularly, we plan to investigate the transferability between structured and unstructured knowledge, and we use the multi-hop reasoning on question answering as an instance to investigate how knowledge is transferred between heterogeneous knowledge sources. Secondly, with the popularity of pre-trained language models, we are interested in directly exploring and eliciting knowledge from them, specifically, the commonsense knowledge contained in neural parameters. We propose a quantification framework with the idea of evaluation by generation to better measure and extract knowledge. To the end, we envision that explicit knowledge and reasoning should be the necessary components towards the next generation of artificial intelligence.