As element61 we have done already tons of data warehouse & data platform migrations. Migrations from SQL Server to Azure Cloud, from SAS to Azure Cloud, from Datastage to Azure Cloud, from Qlik to Azure Cloud, etc. In these migrations, our customers choose together with us a migration strategy being
- “Refactoring” means we don’t just copy the code but rather rebuild the ETL in the Cloud considering the latest business context, updated requirements & thus changed assumptions. We don’t just take the code as-is & ‘run it’ but rather re-analyse the requirement, re-factor or even re-write the cloud & do this domain-by-domain.
- The alternative is a “Lift & shift” meaning we truly aim to run the existing logic & code in the target environment without significant changes (of course, an “optimize” phase can follow later to optimize & tune the code to changing business requirements). A Lift & shift is often chosen for big data warehouses with a significant legacy. While refactoring would then quickly take years, companies often aim for faster turnaround times and thus aim for a Lift & shift done in often <1 year.
In a Lift & shift scenario, but also a refactoring scenario, a lot of code needs to be analyzed & translated to new syntax (e.g., new SQL variant, new ETL software tool, etc.). While all of this can be done manually, an alternative way of migrating is automation where code is automatically translated to target syntax. A lot of time is spent translating the existing code from its original state to the desired language of the new cloud technology.
Rather than doing this manually, this can be done through software that takes the legacy ETL code & automatically translates the code to the target ETL tool: translating Datastage into Databricks jobs, translating legacy SQL jobs into Spark SQL jobs, etc.
Who is Next Pathway & why do we collaborate?
While for many organizations a migration is a one-time effort, some companies specialize in this code migration automation. A leading company in this domain is Next Pathway. Next Pathway is a cloud migration company that focuses on the Software development of automated solutions to enhance and accelerate the migration of your data environment/applications to the cloud. In this Cloud architecture, they support all recent architectures incl. a Lakehouse architecture leveraging ETL processes to extract and transform your data on top of a Data Lake & thus including a variety of technologies like Azure Databricks.
With Next Pathway providing the ‘engine’ for translation, our element61 teams at the customer can migrate domains & code a lot faster. Rather than spending time migrating SQL syntax, our teams focus on integrating the translated code, technically & functionally validating the code & running the code performantly in a best-practice modern data platform.
element61 and Next Pathway complement one another by staying close to & hands-on with the customer in the migration process (element61 teams) while at the same time benefiting from a high degree of automation of all syntax, code & config (Next Pathway team). In an iterative process providing source system objects and artifacts & receiving back translated code for validation, migrations can run in a step-by-step approach taking weeks or months rather than years.
What does a migration process or project look like?
Combined, element61 & Next Pathway can and will assist you during all phases of your migration path. With a proven track record, we complement one another both with each one’s own focused and specific expertise.
As a first step in the migration, we need to plan the migration and crucial to the adoption of the automation process, we leverage the Shift Analyzer of Next Pathway. The Shift Analyzer reads the source system artifacts & code and tries to identify the automation potential by automatically performing an assessment of your existing environment and workloads. The outcome of this Analyzer is automation rates, i.e., what percentage of your environment can be automated during the migration & a list of complexities spotted in the code to consider (if any). After identifying and categorizing all your code objects, an inventory is made up, including the migration rates, to provide an estimation of which fraction can be automated and which fraction will require additional custom development. The Analyzer report allows all teams involved to have a good understanding of the migration at hand, the manual efforts still required & the degree of automation that can be expected.
Next, we run the migration. Often iteratively, the code is translated with the aid of the Shift Translator, a tool that allows the automation of the code translation. It reduces manual efforts and interventions by developers and has self-learning capabilities, meaning that the more it sees, the smarter and more accurate it will become for further translation purposes. In phased deliveries aligned & agreed with the customer, the code is migrated.
Before implementation in the customer’s environment, we leverage the Shift Tester algorithm. This tester ensures that your new code will be reliable, stable, and accurate by validating it through automatically generated test cases. These cases will test both happy-flow paths as well as edge scenarios to make sure all cases will be covered and validated. The Shift Tester will create its own valid test data, to guarantee use cases are tested on reliable underlying data.
Once tested, the code is delivered – in iterations – to element61 and the customer team. From this point onwards, element61 will implement the code in the cloud environment of the customer and functionally test it more in-depth the code & ETL to guarantee high customer data quality and a smooth integration in the customer environment. Here the code is run to actual customer data, the performance of a job is quantified and element61 thus ensures that a technically and functionally validated ETL process is set up at the customer side.
The element61 teams work end-to-end with the customer including – if relevant – front-end migration of connections (e.g., connecting Power BI to the new Azure Cloud dataset), training of users (leveraging our element61 Academy) and tuning/optimizing jobs where need be.
Which migrations are supported?
Any migration is eligible as the automated engine supports a lot of both sources & targets of migration. A few important ones are:
- Teradata
- IBM DataStage
- Netezza
- Hadoop
- SQL Server
- IBM DB2
- Informatica
Combining element61 & Next Pathway, we can support automated automation of these into a broad range of target cloud environments. The most important targets for us combined are Databricks Lakehouse & Azure Synapse Analytics/Fabric.
What’s the business case?
Automating (parts of) the migration process offers a lot of benefits compared to rewriting all of the code manually.
- First of all, there is a time advantage that could be achieved. Manually rewriting code requires a considerable amount of effort for developers, both in understanding the existing code as well as translating it to the desired language.
- Next to that, there is a significant chance of making errors or writing the wrong logic, as it is a manual process performed by human beings. By letting algorithms do the work, the chance of human mistakes will decrease significantly, leading to more correct and more efficient cloud environments.
- Finally, by automating the validation process, targeted and in-depth tests will be performed on your code. This allows you to thoroughly test all of the objects with complex cases and will accelerate your quality assurance testing.
With a proven track record, element61 & Next Pathway complement one another with their own focus and specific expertise. We look forward to working with you on your migration trajectory. Do not hesitate to contact us for more information.
