From the Modern Data Stack to Knowledge Graphs by Bob Muglia

Introduction

Good morning, everyone. Welcome to the Knowledge Graph Conference here in beautiful New York at the Cornell Tech campus, and a warm welcome to those watching online worldwide. Today, I want to delve deep into the evolution of the modern data stack, tracing its origins and projecting its future, particularly towards the concept of knowledge graphs, with a focus on relational knowledge graphs.

The Modern Data Stack: A Historical Overview

The modern data stack emerged to address a plethora of analytic challenges faced by thousands of companies. Essentially, it has become a service-driven approach to data analytics, leveraging the scalability and cost-efficiency of public cloud environments. It is characterized by SQL data modeling, which is paramount for performing comprehensive analytics.

Key Components of the Modern Data Stack

The modern data stack forms a closed loop, beginning and ending with applications that users interact with. It involves various stages:

Data Pipeline: Data is generated from various applications, including SaaS, bespoke software, and log data. Tools and services like Matillion and Python have been instrumental in building pipelines to channel this data into cloud services for analytics.
Data Warehousing: Storage solutions exist in various formats, from data lakes to warehouses, with SQL serving as the go-to language for querying data.
Predictive Analytics: Advanced analytics often utilize procedural languages like Python but face challenges in integrating disparate tools effectively.
Reverse ETL: This final step feeds insights derived from analytics back into the applications in use, completing the data lifecycle.

Despite the success of the modern data stack, challenges remain—particularly around governance, data management, and business modeling.

Governance Challenges and Data Modeling

Governance is a primary challenge within the modern data stack, with companies seeking effective ways to manage complex environments involving multiple vendors. Key capabilities needed include:

Data Catalogs for inventory.
Modeling solutions, largely addressed by tools like dbt.
Data observability for quality control.
Solutions for data privacy and compliance.

However, SQL's limitations in representing business concepts necessitate additional modeling approaches, often resulting in ad-hoc solutions.

The Road to Knowledge Graphs

The modern data stack's evolution leads to a promising direction: knowledge graphs. A knowledge graph integrates data in a manner that represents the complex relationships of entities, which traditional SQL lacks the ability to effectively model.

The Definition and Structure of Knowledge Graphs

A knowledge graph can be defined as a database that models business concepts alongside their relationships and rules. This new relational knowledge graph system brings forth a unified, sophisticated method of data querying and analytics.

Advancements in Algorithms and Techniques

The need for effective governance and modeling has raised awareness of gaps in current SQL databases, particularly around algorithmic design. The transition from binary joins to more complex joining techniques allows for richer data manipulation.

Through an innovative approach called the relational knowledge graph system, we can utilize normalized relational data structures, enabling more powerful analytic queries than ever before.

Looking Forward

While SQL remains a foundational tool, the future of analytics lies in integrating knowledge graphs into the modern data stack. This shift allows business analysts to interact with models directly, fundamentally altering analytics and driving business decisions.

As we stand on the cusp of this transformation, the possibilities for innovation and advancement are tantalizing.

Thank you for being here, and I look forward to engaging in this exciting future with all of you.

Keywords

Modern Data Stack
Cloud Analytics
Data Pipeline
Predictive Analytics
SQL Modeling
Knowledge Graphs
Data Governance
Relational Knowledge Graphs
Data Observability
Business Modeling

FAQ

What is the modern data stack?
- The modern data stack refers to a combination of tools and processes that facilitate data analytics through a service-oriented approach in cloud environments.
What role does SQL play in the modern data stack?
- SQL serves as the primary language for querying and modeling data within the modern data stack, but it has limitations in business modeling.
What are knowledge graphs?
- Knowledge graphs are databases designed to model complex relationships between business entities, integrating data in a manner that is not possible with traditional SQL.
Why is governance important in data analytics?
- Governance is crucial in ensuring data quality, privacy, compliance, and accessibility across complex data environments involving multiple vendors.
What innovations are on the horizon with relational knowledge graphs?
- Relational knowledge graphs promise to broaden analytical capabilities, enabling richer data apps, graph analytics, and novel use cases such as relational machine learning.