Setting up a Data Platform in the Cloud means spinning up various Cloud Components and configuring them all to work together as a Modern Data Platform. Infrastructure as Code means that we script all configurations and creations of all platform components. This means that, rather than creating them manually by clicking in the User Portal, we automate the creation, configurations and updates to any Data Platform component.
Figure: Infrastructure as Code is code which scripts the creation of VMs or other Cloud Components
Why Infrastructure as Code?
Imagine you have a Data Platform with a Data Lake, various Event Hubs, a Big Data Cluster, a Docker registry and that all configurations were manually configured through the User Portal. How would you ever deploy this setup to Production?
Infrastructure as Code reduces time and provides security to your Data Platform setup.
Major benefits include:
•Ability to track all changes on Infrastructure level
•Faster deploys from Dev to Production given no manual actions are required
•Ability to instantly deploy elsewhere - e.g. create an Acceptance environment
What will we discuss in this session
Infrastructure as Code is provided by all major Clouds including Azure, AWS and Google. To do Infrastructure as Code, you can
- either use the syntax specific to a Cloud (every Cloud has its own infrastructure syntax)
- or use a generic language such as Terraform
We will share three different solutions for Infrastructure as Code:
-
ARM
Azure Resource Manager allows you to create Azure resources using a JSON-template. It allows the creation, update and deletion of resources.
-
Azure CLI
Azure Command Line Interface (CLI) is the command line plug-in to create, update and configure resources through command line code in either Bash, cmd or Linux.
-
Terraform
Terraform is an infrastructure provisioning tool created by Hashicorp. It allows you to describe your Infrastructure as Code, create “execution plans” that outline exactly what will happen when you run your code, build a graph of your resources, and automate changes with minimal human interaction. Terraform uses its own domain-specific language (DSL) and is thus cloud-agnostic allowing you to automate infrastructure stacks from multiple cloud service providers simultaneously and integrate other third-party services.