Language Models are Open Knowledge Graphs (Paper Explained)

Introduction

In today's discussion, we delve into the paper titled "Language Models are Open Knowledge Graphs" by Chiang Wang, Wang Xiao Liu, and Don Song. On a high level, this paper proposes a method to construct knowledge graphs automatically using pre-trained language models and a corpus, without any training or human supervision involved. This paper aims to show that standard knowledge graph construction tasks, typically performed manually or semi-manually, can be done using a simple forward pass of the corpus through a pre-trained language model, followed by entity and relation extraction.

Introduction

Knowledge graphs are structured objects that contain entities (nouns) and relations (verbs connecting the entities). Typically, knowledge graphs are built manually by experts, but this paper introduces a method to construct them automatically using pre-trained language models. Unlike traditional machine learning models that require training, this method uses a language model to approximate the construction of knowledge graphs in an unsupervised manner, thus promising to bridge the gap between language models and knowledge graph communities.

Constructing Knowledge Graphs

Step 1: Extract Candidates

1. Use Spacey: The first step is to run Spacey, a standard NLP tool, to extract noun phrases or noun chunks. This process gives us the head and tail of potential facts.

2. Example Extraction: For the sentence "Dylan is a songwriter", Spacey identifies "Dylan" and "songwriter" as the head and tail, respectively.

Step 2: Identify Relationships

Following the extraction of noun phrases, the method identifies the relation phrases that link the head and the tail. This step exploits the attention matrix of the language model. The process involves:

1. Determine Header and Tail: The task is reduced to locating a string span between the head and tail that describes their relation.

2. Dynamic Programming Approach: The model navigates the sentence using attention scores to identify the sequence of words forming the relation.

Step 3: Map to Knowledge Graph Schema

1. Entity Linking: Using pre-existing methods, each discovered entity is mapped to an entry in the knowledge graph schema.

2. Relation Mapping: Relations are similarly mapped to the existing schema.

The goal is to extract candidate facts and map them correctly into the knowledge graph schema using entity and relation linkers.

Results and Observations

• The paper claims that larger and deeper language models yield higher quality knowledge graphs.
• Interestingly, the results show that the system performs better when using the attention matrix from the last layer of the language model rather than from middle layers.
• The extraction and mapping of unmapped facts revealed a significant number of facts not present in existing schemas, suggesting that the model can uncover new information.

Unmapped Facts

The model can sometimes identify new entities and relations that were not part of the existing schema. However, these unmapped facts are mixed with errors introduced by limitations in subsystems like Spacey and entity linkers.

Criticisms

While the paper is innovative, it has some limitations:

• The assumptions made about the construction process may lead to a low recall rate.
• The discovered unmapped facts often reflect errors in subsystems more than genuinely new relations.
• The claim that language models contain structured knowledge could be interpreted as a consequence of their understanding of grammar and statistical associations between words.

Conclusion

In summary, while the paper presents a novel approach to knowledge graph construction, combining traditional NLP tools with pre-trained language models, it highlights both the potential and the current limitations of such an approach.

Keywords

• Language Models
• Knowledge Graphs
• BERT
• GPT-2
• Spacey
• Entity Linking
• Relation Extraction
• Dynamic Programming
• Attention Matrix

FAQ

Q: What are knowledge graphs? A: Knowledge graphs are structured data repositories that contain entities and the relations connecting them, usually constructed manually by experts.

Q: How does this method differ from traditional knowledge graph construction? A: Unlike traditional methods that involve manual or semi-manual processes, this method uses pre-trained language models to automatically construct knowledge graphs without any training.

Q: What tools and models are used in the proposed method? A: The method uses Spacey for extracting noun phrases and pre-trained language models like BERT and GPT-2 for identifying relations. Entity and relation linkers are used for mapping the facts to a schema.

Q: What are the main steps in this method? A: The main steps include extracting noun phrases using Spacey, identifying relationship phrases using attention matrices, and mapping the extracted candidates to a knowledge graph schema using entity and relation linkers.

Q: What are unmapped facts, and how are they handled? A: Unmapped facts are those which are not found in the existing schema. These can be new relations or entities discovered in the corpus but not previously included in the schema.

Q: What are the limitations of this approach? A: The approach has a low recall rate due to its restrictive assumptions. There are also limitations in the pre-existing entity and relation linkers, which can result in errors.