In a time when data has become more and more prevalent, managing your sources and information regarding them becomes pivotal as well. Many people spend a lot of time looking for the right data and get the right value out of it. How wonderful would it be to bridge this gap between IT and business, by having a dictionary of knowledge of your data estate? And dig into it by performing a semantic search over your metadata, or navigate the data lineage?
Azure Purview as Data Management tool
An exciting Azure Service to manage information regarding data sources is Azure Purview. It is used for data management and data governance. It is a service to consolidate and centralize information of your data which is stored on-premise, in multicloud or as software-as-a-service. This will enable organizations to have a top view of the data landscape, perform data discovery, data classification and establish end-to-end lineage. Two core features:
- Azure Purview can scan all your data sources fully automated. While scanning, built-in and custom classifiers can identify the type of data existing in your sources and provide the right classification to it. Making it easy to quickly find specific types of data, including sensitive data.
- Azure Purview allows any user to easily search and find data assets, making use of familiar key terms. These terms can be familiar for any level within the organization, whether the terms are technical based on meta-data or aligned to business and it’s processes.
With Azure Purview, we can offer both business & technical stakeholders an overview of all sources collected in Azure Purview provided via a Data Map. These can be organized and structured in separate collections depending on the type of organization.
The goal is to help organizations better manage, understand and use data. Making it easy to collaborate and launch data initiatives which will occur more and more at a high velocity since Azure Purview will encourage and entice anyone involved with data within the organization.
From search term to Power BI Dashboard
Today's challenge: With increasing amount of data and data sources, chances are that you have a hard time finding your relevant data. Especially in organizations with thousands of sources and millions of tables and files. A business analyst will not be able to look for Sales data in an efficient way.
With Azure Purview, a business analyst can just search for a all sources related to a key term e.g. by entering “Sales Revenue” in the Azure Purview portal. By narrowing down the result list, the analyst will quickly identify relevant tables containing revenue data. For instance a Synapse table containing all information of Revenue by Customer.
When drilling down on the revenue table, more information appears, such as when it is created, modified, even a label that might emphasize that this data is confidential and will not be visible to everyone.
Azure Purview allows to drill-down & to show all fields of a table incl. its datatypes, glossary terms and the classification labels as well, in case of specific fields such as EmailAddress, Names, ProductCode.
The analyst might wonder where the data comes from. In that case the lineage view will give a clear overview from which source system and which tables the data originates. Even the Azure Data Factory pipeline used to prepare and transform the data is shown in the lineage, resulting in the Revenue by Customer table the analyst just found.
A step further is even possible to find lineage to Power BI reports and dashboards to explore and find which reports are exactly making use of this table. If more detail is needed, a drilldown to the Power BI lineage itself is possible as well, showing other sources and tables which take part in the Power BI Dashboard as well.
Insights for Data officers
Today's challenge: In times of data compliance & GDPR, knowing what is happening with data within an organization, has become very important as well yet extremely challenging.
With the Insights view in Azure Purview, a holistic view of all the data in the Data Map is provided. Find out which data is sensitive, where data is distributed, where it moves, who it’s shared with.
In the glossary view, the top terms across all sources are shown, emphasizing the ones with the highest occurrence.
Even the classifications are visualized to provide an overview of all classified and sensitive data across the organization. By clicking View more, it is possible to search for data based on a specific classification.
In this case the EU GPS Coordinates might be interesting to analyze and see where this data comes from.
By drilling further down, the source is found. Now the Data Officer can investigate further and find out if this data is effectively stored securely and is following compliance.
There seem to be many beneficial features with the new promising generation of Azure’s Data Catalog, called Azure Purview.
Azure Purview will help to consolidate, centralize and manage information of your data estate. It will ease knowledge sharing between existing and new team members, but also across functional teams! Finally, Azure Purview will help with the alignment of end-2-end processes, overarching departmental needs for data. Resulting in people successfully discovering the right data sets and unlock their full potential in an efficient, timely and elegant manner.
For more insights & research, visit our element61 Knowledgebase at www.element61.be