What is DevOps
- Coding – taking a DevOps mindset means collaborating on code development and reviews
- Building – supporting continuous integration tools and builds
- Testing – embedding continuous testing tools that validate code written
- Packaging – providing an artifact repository or application pre-deployment staging
- Releasing – supporting release approvals and continuous deployment
- Configuring – providing infrastructure configuration including infrastructure as code support
- Monitoring – enabling applications performance monitoring
Why is DevOps important for a Big Data Platform
When setting up a Big Data Platform we use code to provide infrastructure and define data transformations. Similarly to application development, this code needs to be built collaboratively, be tested, and be released periodically. DevOps perfectly applies to Big Data and AI development.
element61 implements DevOps in all its Data Platform implementations providing
- the tools and setup for collaborative code development, automated testing, infrastructure-as-code, and continuous integration and deployment.
- the training of your team in using and benefiting from DevOps
What tools support DevOps
DevOps isn't necessarily supported by 1 tool. Depending on the tools you already use (e.g. Github or Bitbucket) you might need to add tools or you might already have all the components you need.
When building an Azure Data Platform, we recommend using Azure DevOps for organizations with no DevOps yet present. For all other scenarios tools like CircleCI are the most popular.
Azure DevOps services is a DevOps toolbox providing Code versioning, Agile tooling, Build and Release functionalities, Test plans and Azure Artifacts.
It has a strength in providing (1) an end-to-end toolkit including all DevOps best practices (e.g. use of .yaml-files) and (2) tight integration with Azure
Getting started
Data Engineers at element61 are experienced in using DevOps in all projects. Contact us to get started or get some advice