Neo4j Knowledge Graphs and Google Generative AI

Hi, my name is Ben Lackey. In this guide, I'm going to show you the innovative work Neo4j, the leader in graph databases, is doing with Google's new generative AI features in Vertex AI.

Introduction

Neo4j is at the forefront of harnessing the power of graph databases to manage complex data relationships. In collaboration with Google's generative AI, Neo4j is revolutionizing how businesses extract and utilize data from various sources.

Emerging Architecture

An architecture pattern is increasingly emerging among our customers. Here's how it operates:

Data Ingestion: Customers gather data from multiple sources, both unstructured (e.g., flat files, data lakes) and structured (e.g., traditional SQL databases, XML, JSON).
Entity and Relationship Extraction: Generative AI is used to automatically extract entities and relationships from the ingested data.
Knowledge Graph Creation: Using Cipher, the query language for Neo4j, generative AI automatically creates queries to insert the extracted entities and relationships into Neo4j.

Utilizing Generative AI

Our customers are leveraging generative AI in two main ways:

Data Import: Generative AI creates Cipher queries to write extracted data into Neo4j.
Data Querying: Generative AI rephrases user questions into Cipher queries, runs them against the database, and converts results back into natural language.

Practical Application – Human Capital Management (HCM)

In this demo, we applied this general architecture to the HCM domain, focusing on processing resumes:

Data Sources: Unstructured text files containing resumes.
Entity and Relationship Extraction: Using Google’s generative AI, entities (e.g., names, companies, skills) and relationships (e.g., where a person has worked) are identified.
Knowledge Graph: Generated data is inserted into Neo4j, forming a comprehensive knowledge graph.

Demo Walkthrough

GitHub Repository

We utilized a publicly available GitHub repository for this demonstration. The repository includes a notebook to handle the left side (data extraction) and a user interface to handle the right side (data querying).

Vertex AI Managed Notebook

The notebook, running on Google’s Vertex AI, processes resumes to extract and format the data. Given the extensive processing time, we pre-ran most of the notebook for the demo.

Neo4j Browser

By examining the resulting data in the Neo4j browser, you can:

View nodes such as "Company," "Education," and "People."
Explore relationships, for instance, identifying all individuals with expertise in a particular skill.

Streamlit Application

For user interaction, we created a web app using Streamlit, deployed on a Google Compute Engine virtual machine. The UI allows users to:

Ask Questions: Users query the knowledge graph using natural language.
View Results: Generative AI converts the questions into Cipher queries, retrieves results, and presents them in human-readable formats.

Conclusion

This system effectively converts unstructured data into a structured and queryable knowledge graph, leveraging the powerful combination of Neo4j and Google's generative AI. It simplifies complex data extraction and querying, making sophisticated data utilization accessible to non-technical users.

If you have any questions, please reach out to me at ben.lackey@neo4j.com.

Thank you!

Keywords

Neo4j
Graph Databases
Vertex AI
Generative AI
Knowledge Graph
Entity Extraction
Cipher
Human Capital Management
Streamlit
Data Query

FAQ

Q: What is Neo4j?
A: Neo4j is a graph database management system that helps manage complex data relationships efficiently.

Q: How does Vertex AI integrate with Neo4j?
A: Vertex AI is used to automatically extract data entities and relationships and create Cipher queries to insert data into Neo4j.

Q: What are some practical applications of this technology?
A: This technology can be used in various domains like Human Capital Management, Supply Chain Management, financial services for anti-money laundering, and more.

Q: How is unstructured data processed in this system?
A: Generative AI processes unstructured data to extract relevant entities and relationships, which are then inserted into Neo4j to form a structured knowledge graph.

Q: Can non-technical users interact with the knowledge graph?
A: Yes, a Streamlit-based web application allows users to query the knowledge graph using natural language, and generative AI translates these queries into technical Cipher queries.

Q: How accurate are the AI-generated queries?
A: The AI-generated queries are accurate thanks to prompt engineering and model fine-tuning, ensuring relevant and precise extraction and querying.