To benefit most from this blog and the Purview APIs in general, you should have a meta-data driven ETL process as this will make it easy to create the dataframe. We will follow the example of a simple ETL process.įirstly, create a pandas dataframe like below. Now, authenticate to your Purview Account using oauth: def authenticatePurview(account_name, tenantId, clientId, clientSecret):Ĭlient = authenticatePurview(purviewAccount,TENANT_ID, CLIENT_ID, CLIENT_SECRET)īy passing the Azure Purview resource name, Azure tenant ID and client ID, and secret of the service principal, we create a PurviewClient object to interact with the Purview APIs. See the Microsoft documentation for a detailed explanation of the above steps. Assign the following roles to the service principal: Then, from within the Purview governance portal, select the root collection in the collections menu and select Role assignments. The first step is to connect to your Azure Purview account using Python (3.6+):įrom the Azure portal, create a service principal and generate a secret (make a note of these).

We will use PyApacheAtlas library to work with the Azure Purview APIs.

Lineage is lost when using other tools like Azure Functions, Databricks notebooks, or SQL stored procedures. Lineage can only be tracked out of the box when using tools such as Data Factory, Power BI, and Azure Data Share. The aim of this blog is to explain how to create custom Purview processes, enabling you to add lineage from processes that are not tracked out of the box.Īs covered in this blog, Azure Purview can help with understanding the lineage of your data, offering visibility of how and where data is moving within your data estate.

Snapshot in azure