What is DevOps?

DevOps is a set of practices which automate the processes between software development (Dev) and IT teams (Ops). The objective of using DevOps is to shorten the development lifecycle and have frequent updates of tools and applications in line with business priorities. 
 
DevOps isn't a software nor a framework; it's a mindset or way-of-working. The processes which DevOps helps automate and the practices it suggests are the following:
  • Coding – taking a DevOps mindset means collaborating on code development and reviews
  • Building – supporting continuous integration tools and builds
  • Testing – embedding continuous testing tools that validates code written 
  • Packaging – providing an artifact repository or application pre-deployment staging
  • Releasing – supporting release approvals and continuous deployment
  • Configuring – providing infrastructure configuration including infrastructure as code support
  • Monitoring – enabling applications performance monitoring

Why is DevOps important for a Big Data Platform?

When setting up a Big Data Platform we use code to provide infrastructure and define data transformations. Similarly to application development, this code needs to be built collaboratively, be tested and be released periodically. DevOps perfectly applies to Big Data and AI development.

element61 implements DevOps in all its Data Platform implementations providing

  • the tools and setup for collaborative code development, automated testing, infrastructure-as-code and continuous integration and deployment.
  • the training of your team in using and benefiting from DevOps

What tools support DevOps?

DevOps isn't necessarily supported by 1 tool. Depending on the tools you already use (e.g. Github or Bitbucket) you might need to add tools or you might already have all the components you need.

When building an Azure Data Platform, we recommend to use Azure DevOps for organizations with no DevOps yet present. For all other scenario's tools like CircleCI are most popular.

DevOps

Azure DevOps services is a DevOps toolbox providing Code versioning, Agile tooling, Build and Release functionalities, Test plans and Azure Artifacts.
It has a strength in providing (1) an end-to-end toolkit including all DevOps best-practices (e.g. use of .yaml-files) and (2) tight integration with Azure

Getting started

Data Engineers at element61 are experienced in using DevOps in all projects. Contact us to get started or get some advice