DSS - Data Science & Strategie

If you have worked with data in Python before, you are probably familiar with the following line of code: 

In this insight we focus on one of the leading tools: MLflow and how MLflow can facilitate and improve your ML lifecycle.

There were many new exciting announcements at the Microsoft Ignite 2020. One of them was the Power BI connection to Azure Databricks.

In this insight I will guide you through what Power BI dataflows are all about and how you can easily setup a "date" Power BI dataflow, which you can reuse in all your datasets.

What is Azure Synapse Analytics Azure Synapse Analytics is the Azure SQL Datawarehouse rebranded. Azure Synapse Analytics v2 (workspaces incl. Azure Synapse Studio) is still in preview. This version of Azure Synapse Analytics integrates existing and new analytical services together to bring the enterprise DWH and the big analytical workloads together.   What are the Azure Synapse Components

On June 18th 2020, Databricks announced that the Apache Spark 3.0 release is available as part of their new Databricks Runtime 7.0. Apache Spark 3.0 itself got released only 8 days prior on the 10th of June. We really love these quick innovation cycles and are happy Databricks is on top of their game in acting very fast here.

It’s not easy to work with incremental data in a data lake. If you would want to transform only the data files that just entered your data lake, you would need a notification service, a message queue and/or a batch trigger all to just get the incremental files. This can be done with Auto Loader. What is Auto Loader Auto Loader is a Spark feature that allows this out-of-the box. It allows to incrementally load the data as soon as it lands on the cloud storage. Auto Loader is a rather new feature and a very simple add-on in your existing Spark jobs & processes.

As the official documentation is not covering this, we have built a guide on how to create an Azure Machine Learning pipeline and how to run this pipeline on an Azure Databricks compute.

On 13 May 2020, the NYC Apache Airflow Meetup hosted a virtual event entitled “What’s coming in Airflow 2.0”. Being big fans of Airflow at element61, we were curious to find out what changes are to be expected in this long-awaited Airflow version.