What is DevOps?
- Coding – taking a DevOps mindset means collaborating on code development and reviews
- Building – supporting continuous integration tools and builds
- Testing – embedding continuous testing tools that validates code written
- Packaging – providing an artifact repository or application pre-deployment staging
- Releasing – supporting release approvals and continuous deployment
- Configuring – providing infrastructure configuration including infrastructure as code support
- Monitoring – enabling applications performance monitoring
Why is DevOps important for a Big Data Platform?
When setting up a Big Data Platform we use code to provide infrastructure and define data transformations. Similarly to application development, this code needs to be built collaboratively, be tested and be released periodically. DevOps perfectly applies to Big Data and AI development.
element61 implements DevOps in all its Data Platform implementations providing
- the tools and setup for collaborative code development, automated testing, infrastructure-as-code and continuous integration and deployment.
- the training of your team in using and benefiting from DevOps
What tools support DevOps?
DevOps isn't necessarily supported by 1 tool. Depending on the tools you already use (e.g. Github or Bitbucket) you might need to add tools or you might already have all the components you need.
When building an Azure Data Platform, we recommend to use Azure DevOps for organizations with no DevOps yet present. For all other scenario's tools like CircleCI are most popular.
Azure DevOps services is a DevOps toolbox providing Code versioning, Agile tooling, Build and Release functionalities, Test plans and Azure Artifacts.
It has a strength in providing (1) an end-to-end toolkit including all DevOps best-practices (e.g. use of .yaml-files) and (2) tight integration with Azure
Data Engineers at element61 are experienced in using DevOps in all projects. Contact us to get started or get some advice