What is DevOps?
- Coding – Taking a DevOps mindset means collaborating on code development and reviews
- Building – supporting continuous integration tools and builds
- Testing – embedding continuous testing tools that validates code written
- Packaging – providing an artifact repository or application pre-deployment staging
- Releasing – supporting release approvals and continuous deployment
- Configuring – providing infrastructure configuration including infrastructure as code support
- Monitoring – enabling applications performance monitoring
Why is DevOps important for a Big Data Platform?
When setting-up a Big Data Platform we use code to provide infrastructure and define data transformations. Similar to application development, this code needs to be build collaboratively, be tested, be released periodically. DevOps perfectly applies to Big Data and AI developmet.
element61 implements DevOps in all its Data Platform implementations providing
- the tools and set-up for collaborative code development, automated testing, infrastructure-as-code and continuous integration and deployment.
- the training of your team in using and benefiting from DevOps
What tools support DevOps?
DevOps isn't necessarily supported by 1 tool. Depending your the tools you already use (e.g. Github or Bitbucket) you might need to add tools or you might already have all the components you need.
When building an Azure Data Platform, we recommend to use Azure DevOps for organization with no DevOps yet available. For all other scenario's tools like CircleCI are most popular.
Azure DevOps services is DevOps toolbox providing Code versioning, Agile tooling, Build and Release functionalities, Test plans and Azure Artifacts.
It has a strenght in (1) providing an end-to-end toolkit including all DevOps best-practices (e.g. use of .yaml-files), (2) tight integration with Azure
Data Engineers at element61 are experienced in using DevOps in all projects. Contact us to get started or get some advice