Power Platform : How to extract Form document data using AI Builder and Microsoft Flow

Introduction

In our day to day business processes, we can use Artificial Intelligence to advance business operations. The AI Builder (preview feature) is the new Power Platform feature that allows us to use Artificial Intelligence to easily automate our business processes and predict outcomes. AI Builder currently supports four types of AI model, Form processing model, Text classification model, Binary classification model, and Object detection model. Each model serves a different purpose, we can choose the type of model based on our requirement.

In this blog, we will illustrate how to create and use Form processing model with the help of a real-world business scenario.

What is Form processing model?

Form processing model as the name suggest, processes the forms or documents that have data in key-value pair. It extracts the document data and makes it available for further use. It uses the existing sample documents to analyze the new documents and extract information. The accuracy of outcome of this model depends on the sample documents that we used to learn the model.

Scenario:

A renowned construction company uses CRM to manage their business process. For every new work the sales person creates a new Project entity record in CRM. Contractors (Not CRM users) assigned to that project, analyze the work and prepare a quote and that quote get attached to the project record as notes attachment. Sales person then manually downloads the attachment, reads the quote and updates the quote details on Project entity. They have to do this every time when any new quote document is attached to the record.

This process is quite time consuming and requires human intervention to get the most recent quote information on the project.

Here, we can make use of Form processing AI model to automate this process. Given below are the steps to automate this process using AI Builder.

1. Enable AI Builder: Make sure that AI builder is available and enabled for the environment. Sign in to Power Platform Admin center. In the admin center go to Environments -> <select environment> -> Settings -> Features.

In Features page check if Create AI models in PowerApps is enabled.

2.Create Form Processing AI Model: Sign in to https://web.powerapps.com. Go to ‘AI Builder’ -> Build -> Form processing.

This will open the below screen. Give name for the model. We can see this model requires sample documents. We can see the examples of document. Click on ‘Examples’ to see the example of sample document.

3. Prepare sample documents: Go through Form processing AI model requirements and limitations. The documents that you have must meet these requirements otherwise model will fail with the errors. So if you ever face error while analysing the model make sure that your sample documents meet all the requirements. As an example we have taken below Quote report (PDF format) which the contractor fills up and uploads against the project.

We have 5 different versions (data) of the same report. We will use these five pdf files as a sample data for the Form processing model.

4. Analyze Documents: Click on ‘Add Documents’ button, select all the sample pdf files and upload it.

Once all the documents are uploaded close the window and click on ‘Analyze’ button. This will save the model and start analyzing the documents.

This process will take few minutes and will ask you to select a field. Analyze steps, analyze the structure of the document, identifies the key value pair, table data and expose are fields that we need to select in the next step.

5. Select form fields: Here we need to select data that we want to extract from the document. Click on ‘Process Quote Model form’, this will open the below screen with all possible data that the Model can extract.

We can either select all the fields or we can select those fields that are needed. Here we have selected the Total Amount and Prepared By field. Click on ‘Done’ button to save the field selection for model. Click on ‘Next’ button to train the model using uploaded sample documents.

6. Train, Test and Publish Model: Click on ‘Train’ button to train the Form processing model.

Below screen shows model training is completed. Click on ‘Go to the details page’ button. This will take us on the Model page.

You can also navigate to the model details page by clicking on ‘Model’ -> <click on Model from the list of model>

We can test the model by using ‘Quick test’ button. Upload the new document and check if it extracts the information that we wanted or not. Finally click on Publish button to publish the model.

7. Use Form processing Model to automate business process using Microsoft Flow:

Create a new solution in Microsoft flow and then create a new Flow under that solution and setup it to trigger on creation of notes for Project entity. Here we will consider Opportunity as the Project entity. We have ‘Quotation information’ section on Opportunity/Project entity which has two fields ‘Total Amount’ and ‘Prepared By’ field.

Our flow will trigger on any upload of notes attachment on project entity, then it will call and pass the uploaded document to Form Processing Model and finally, will read and update the response returned by Model on opportunity fields.

Add a ‘Predict – Common Data Service (Current environment)’ action and select our newly created Form Processing Model.

In request payload pass JSON string in below format;

{

“base64Encoded”: “EXPRESSION”,

“mimeType”: “image/jpeg”

}

This expression must return document content in base64 format. In case of CRM notes document, we can directly pass Document field. The expression must be enclosed by base64() based on which connector the files comes from. We can have SharePoint connector or any other connector which will give files to be passed to AI Model. The Mime type also depends on the type of document. If the document type is pdf then pass ‘application/pdf’ and if document type is image then pass correct mime type ‘image/jpeg’

Now save the Flow and test it by clicking on ‘Test’ button. Check the logs of Flow and copy the result of Predict action and past it in code editor i.e. either Visual Studio or Visual Code.

Remove “\” characters from output that we copied from Predict action.

The output will have result for the fields that we selected while creating a Model. Here you can see the value for Total Amount and Prepared By.

Add Parse JSON action, select ‘Response Payload’ in the content and paste the JSON output copied in code editor (JSON string after removing “\” character) in the Schema.

Now we are ready to use the response of Form Processing model to update the quote details in CRM. Add ‘Update record’ action of Common data service and use the values from Parse JSON action.

body(‘Parse_JSON’)?[‘predictionOutput’]?[‘labels’]?[‘Prepared by ‘]?[‘value’]

body(‘Parse_JSON’)?[‘predictionOutput’]?[‘labels’]?[‘Total Amount’]?[‘value’]

Note: To use AI Builder Models in a flow we must create flow in solution.

Conclusion

In this way, we can use Form Processing AI model of AI builder to process the documents, analyze it and extract data from it. Based on requirement we can adjust the Microsoft Flow to when it should trigger, call AI model to get data from the document and use that data.

70% of global 2000 companies apply gamification to improve productivity and returns!

Gamifics365 – Spin the magic of games within Microsoft Dynamics 365 CRM to improve user adoption, enhance productivity, and achieve company goals!