Power Platform Form Processing with AI Builder and Power Automate Desktop RPA

Introduction

Hey everybody, this is Steve Woodward. Have you ever had a scenario where you have hundreds of forms filled out by people, either handwritten and scanned in or digitally filled, and then you had to enter that into another system? If so, this article is for you. We're diving into how you can use Power Platform's AI Builder service to automate this tedious task. I'll walk you through creating a form processing model, extracting data, and using Power Automate Desktop RPA to populate another system. This approach can save you a significant amount of time with low-code solutions.

Demo Overview

First, let's get into the demo. Imagine you've sent out a Word document to everyone in your organization asking them to fill it out for a company event. You receive responses in different formats: some digitally filled, some handwritten and scanned. Now, you have a pile of files and need to process them efficiently.

Sample Files Demonstration

Digital Form Example: Somebody filled it out in Word, saved it as a PDF, and sent it.
Handwritten Example: Another person wrote it by hand, scanned it, and sent it as a PDF.

Your task is to automate the extraction of data from these forms, populate a web application form, and eventually export the data to Excel.

Running the Power Automate Flow

To achieve this, we use a Power Automate flow, specifying the file type and file itself. The flow processes the file using AI Builder to extract the required fields, then sends the data to a Power Automate Desktop RPA process to populate the web application form automatically.

Creating the AI Builder Model

Here's a detailed breakdown:

Navigate to AI Builder: Go to AI Builder and select "Form Processing".
Model Naming: Name your new model, e.g., Food Sign-Up Model.
Define Field Names: Provide the field names expected in your forms.
Upload Sample Data: Create a collection of both digital and handwritten form samples.
Analyzing Sample Data: AI Builder will analyze your uploads. Assist it in identifying the corresponding fields.
Training the Model: After tagging fields across multiple forms, train the model.

Testing and Publishing

Test the trained model by uploading a form and validating field extraction. Once satisfied, publish the model to make it available in PowerApps and Power Automate.

Integrating with Power Automate Desktop

In Power Automate Desktop:

Web Recording: Simulate filling out the web form using web recorder.
Creating Variables: Create input variables for each field.
Replacing Hard-Coded Values: Replace hard-coded text with variables.
Handling Extra Spaces: Implement regex in Power Automate Desktop to handle extra spaces in handwritten fields.
Final Flow Configuration: Set up the final flow logic that integrates with AI Builder.

Run the desktop flow to ensure it populates the web form correctly with extracted data.

Complete Power Automate Flow

The final Power Automate flow includes:

Manual Trigger: Start the flow manually with document type and file.
AI Builder Call: Process the document and extract fields.
Invoking Desktop Flow: Pass extracted data to Power Automate Desktop.
Sending Email: Send an email summarizing the extracted data.

Conclusion

By following these steps, you can streamline the processing of various form types using Microsoft's Power Platform, AI Builder, and Power Automate Desktop. For further details and code samples, visit my GitHub repository.

Keywords

Power Platform
AI Builder
Power Automate Desktop
Form Processing
Automation
Low-Code Solutions

FAQ

What is the AI Builder in Microsoft Power Platform?

AI Builder is a service in Microsoft Power Platform that enables users to add artificial intelligence capabilities to their workflows and apps without having to write code.

How does Power Automate Desktop help in form processing?

Power Automate Desktop helps by automating web interactions and other processes on your computer, making it easier to input data extracted from forms into other applications.

Can AI Builder handle both digital and handwritten forms?

Yes, AI Builder can process both digitally filled out forms and handwritten, scanned forms.

How do I handle spaces in extracted data from handwritten forms?

Use regular expressions in Power Automate Desktop to replace unwanted spaces in the extracted text.

Where can I find the demo samples and code?

You can find all the demo samples and code on Steve Woodward's GitHub repository, which is linked in the article.