Charting the Course: The Latest in AI & Data Science with Fabric | FabCon 2024

Introduction

Welcome to the latest in AI and Data Science with Microsoft Fabric! In this session, Nelly Gson, head of the product team for Synapse Data Science and AI, along with her colleague Oren Orbe, product manager on the same team, shared an array of updates and features designed to enhance productivity for both data scientists and data engineers.

Overview of the Session

The session covered a significant amount of content focusing on new features in data science within the Fabric ecosystem. The agenda included:

• Data Science Features: An overview of recent enhancements.
• Customer Journey: A spotlight on Nebraska Furniture Mart's experience using data science in Fabric.
• AI Features: A discussion on AI advancements and how they integrate with data workflows.

Data Science in Fabric

The conversation opened with an insightful discussion on end-to-end data science workflows within Fabric. Nelly explained how data scientists, analysts, and engineers can leverage the unique offerings of Fabric to streamline their processes. With powerful features like:

• Semantic Link: Allows users to tap into BP semantic models easily, facilitating data discovery.
• Lakehouse Notebooks: Providing a simple interface for complex data exploration and modeling.
• Data Wrangler: A low-code tool for data preparation and transformation, now integrated with Spark for handling larger datasets.

Oren then elaborated on the need for robust data preparation, sharing that data wrangling tools now allow users to perform data operations visually, generating code automatically and ensuring a seamless workflow.

Machine Learning Lifecycle Enhancement

The integration of MLflow in Fabric significantly enhances the ML lifecycle, enabling capabilities such as:

• Model Tracking: With built-in experiment tracking and model versioning.
• Batch Scoring: Using Synapse ML to facilitate scalable batch predictions.
• Real-time Endpoints: Currently in private preview, allowing for model deployment and easy request-response exchanges.

The pair highlighted a successful customer case with Nebraska Furniture Mart, emphasizing how they reduced processing time dramatically by utilizing Fabric's capabilities to streamline their demand forecasting.

Sneak Peeks Into Future Features

Towards the latter part of the session, attendees were treated to live demonstrations of upcoming features, including:

• Automated Machine Learning with Flamel: A lightweight library for accelerated model training.
• Real-time API Endpoints: Making deployment easy and flexible without complex configurations.

AI Experiences and Co-Pilot Integration

As part of Microsoft’s investment in AI, several co-pilot capabilities were discussed, emphasizing how they facilitate user interaction with the data within Fabric. Using natural language queries, users can now interact with robust datasets to extract insights.

Generative AI Development

The presentation outlined future developments with a focus on generative AI. One such feature showcased the ability for users to create AI skills that query both structured and unstructured data in Fabric, paving the way for deeper insights across various organizational data sources.

Summary

The session concluded with a reminder of the ongoing private previews for various AI features and calls to action for attendees to engage with new capabilities as they roll out. Attendees were encouraged to connect for further insight and explore future fabric capabilities in their workflows.

Keywords

• Microsoft Fabric
• Data Science
• AI Integration
• Semantic Link
• Lakehouse Notebooks
• Data Wrangler
• MLflow
• Real-time Endpoints
• Generative AI
• Co-Pilot

FAQ

What are the key features of data science in Fabric?

The key features include Semantic Link, Lakehouse Notebooks, and Data Wrangler, all designed to facilitate data preparation and streamline machine learning workflows.

How does MLflow integrate with Fabric?

MLflow serves as a backbone for tracking experiments and managing the machine learning lifecycle within Fabric, making it easy to save, compare, and manage different model versions.

What is the significance of real-time endpoints?

Real-time endpoints, currently in private preview, allow users to deploy models simply and efficiently, enabling prompt predictions through a URL.

How can co-pilot features assist users?

Co-pilot features enhance productivity by allowing users to interact with data using natural language queries, facilitating insights and reducing the time required to generate results.

What future developments were mentioned in the session?

Future developments include deeper integrations of generative AI capabilities for dynamic querying of both structured and unstructured data, empowering users to interact with their entire dataset.