Knowledge base

It’s not easy to work with incremental data in a data lake. If you would want to transform only the data files that just entered your data lake, you would need a notification service, a message queue and/or a batch trigger all to just get the incremental files. This can be done with Auto Loader. What is Auto Loader Auto Loader is a Spark feature that allows this out-of-the box. It allows to incrementally load the data as soon as it lands on the cloud storage. Auto Loader is a rather new feature and a very simple add-on in your existing Spark jobs & processes.

As the official documentation is not covering this, we have built a guide on how to create an Azure Machine Learning pipeline and how to run this pipeline on an Azure Databricks compute.

On 13 May 2020, the NYC Apache Airflow Meetup hosted a virtual event entitled “What’s coming in Airflow 2.0”. Being big fans of Airflow at element61, we were curious to find out what changes are to be expected in this long-awaited Airflow version.

In recent years many different ML lifecycle tools have been developed. Kubeflow is one of these tools, offering promising features in the field of creating and deploying ML pipelines. We outline how to run it on Azure

AI beating world champions in complex strategy games such as chess or Go? Have you ever wondered how this is possible? In this insight, we outline the underlying training technology, namely Reinforcement Learning step-by-step.

Are we on the eve of a breakthrough for predictive analytics within the Office of Finance?Will covid19 speed up embracing predictive analytics supported & automated forecast and budget reporting? Contribute to the success of the business, act as the trusted advisor to the CEO,…  are some of the credo’s that CFO’s try to live up to over the past couple of years.

Power BI Premium provides dedicated resources to run Power BI in an organization. It can be a great cost saver to try and use this dedicated capacity as efficiently as possible without paying for unneeded resources. This is where performance tracing can help us analyze system usage and help make informed decisions about scaling up or down.

As data professionals, our role is to extract insight, build AI models and present our findings to users through dashboards, API’s and reports. Although the development phase is often the most time-consuming part of a project, automating jobs and monitoring them is essential to generate value over time. At element61, we’re fond of Azure Data Factory and Airflow for this purpose.