What is Infrastructure as Code
Setting up a Data Platform in the Cloud means spinning up various Cloud Components and configuring them all to work together as a Modern Data Platform.Infrastructure as Code means that we script all configurations and creations of all platform components. This means that rather than creating them manually by clicking in the User Portal, we automate the creation, configurations, and updates to any Data Platform component.
Figure: Infrastructure as Code is code that scripts the creation of VMs or other Cloud Components
Why Infrastructure as Code
Imagine you have a Data Platform with a Data Lake, various Event Hubs, a Big Data Cluster, and a Docker registry, and all configurations were manually configured through the User Portal. How would you ever deploy this setup to Production?
Infrastructure as Code reduces time and provides security to your Data Platform setup.
Major benefits include:
- Ability to track all changes on an Infrastructure level
- Faster deploys from Dev to Production given no manual actions are required
- Ability to instantly deploy elsewhere - e.g. create an Acceptance environment
Which technologies allow this
Infrastructure as Code is provided by all major Clouds including Azure, AWS, and Google. To do Infrastructure as Code, you can
- either use the syntax specific to a Cloud (every Cloud has its infrastructure syntax)
- or use a generic language such as Terraform
element61 has experience with the following tools:
-
ARM
Azure Resource Manager allows you to create Azure resources using a JSON-template. It allows the creation, update and deletion of resources.
-
Azure CLI
Azure Command Line Interface (CLI) is the command line plug-in to create, update and configure resources through command line code in either Bash, cmd, or Linux.
-
Terraform
Terraform is an infrastructure provisioning tool created by Hashicorp. It allows you to describe your Infrastructure as Code, create “execution plans” that outline exactly what will happen when you run your code, build a graph of your resources, and automate changes with minimal human interaction. Terraform uses its domain-specific language (DSL) and is thus cloud-agnostic allowing you to automate infrastructure stacks from multiple cloud service providers simultaneously and integrate other third-party services.
When to use Infrastructure as Code
We recommend our customers to use Infrastructure as Code for any project or setup, small or big. It saves time and secures your setup to be future-proof.
Our Data Engineers at element61 are trained to use Infrastructure as Code and can help you and your colleagues absorb this best practice.
Contact us to get started!