Why? Since data pipelines are far behind the efficiency of the software industry, tools are needed to bring the design up to speed and to guarantee data quality. This optimisation and performance improvement can happen with tools such as dbt and...
What is dbt? dbt is an open-source transformation tool that aims to simplify the work of the analytic engineer in the data pipeline workflow. It specifically implements only the T in the ETL process. The greatest feature is that it focuses on...
If you have worked with data in Python before, you are probably familiar with the following line of code: import pandas as pd The Pandas library is widely seen as the de-facto standard library for data science in Python and rightly so. It offers a...
What is Azure Synapse Analytics? Azure Synapse Analytics is the Azure SQL Datawarehouse rebranded. Azure Synapse Analytics v2 (workspaces incl. Azure Synapse Studio) is still in preview. This version of Azure Synapse Analytics integrates existing...
On June 18th 2020, Databricks announced that the Apache Spark 3.0 release is available as part of their new Databricks Runtime 7.0. Apache Spark 3.0 itself got released only 8 days prior on the 10th of June. We really love these quick innovation...
It’s not easy to work with incremental data in a data lake. If you would want to transform only the data files that just entered your data lake, you would need a notification service, a message queue and/or a batch trigger all to just get the...
As the official documentation is not covering this, we have built a guide on how to create an Azure Machine Learning pipeline and how to run this pipeline on an Azure Databricks compute.
At element61 we love to work with your data. Our philosophy is providing you a Modern Data Platform in which you can do real-time analytics, big data batch analytics as well as apply Machine Learning and AI.
