Recognizing the scale of this challenge, one of the organization’s data specialists began seeking external support.
Insight had actively been working with the financial services team at Microsoft to explore opportunities to support the client’s former parent company. When Microsoft heard about the subsidiary’s migration project, they recommended Insight for the job.
A groundbreaking shift to HDInsight
In July of 2020, our team began working with the client’s VP of enterprise data and analytics. A discovery session was conducted to assess the legacy environment and the size of the clusters to be migrated. Over the next few months, Insight analysts and program managers worked around the clock to establish a comprehensive understanding of the organization’s needs and build a strong foundation for success.
With the initial assessment finalized, a complete analysis of the BDS data environment was presented back to the client, along with a project plan tailored to the requisite timeline.
Based on the need for rapid execution and the desire to maximize the scalability of the data environment, Insight partnered with Microsoft to propose a two-phase migration approach. Initially, to meet the required timeline, we recommended migrating from SAP to Azure HDInsight — a feat which, to date, had never been accomplished. This would require minimal changes to the work previously completed by the client’s IT team and would allow Insight’s team to focus on the migration and security of the SAP data, as well as the cluster setup and administration. A second phase was also planned to finalize the migration to Databricks, which offered a simplified architecture that would further decrease both the cost and complexity of the existing Hadoop environment.
Architects and engineers began executing the migration in mid-October. Due to the financial nature of the organization’s data, rigorous protocols were implemented and thoroughly tested to ensure the information remained secure from source to destination. Once the data had been successfully transferred to the new environment, the team began standing up two Hadoop clusters to serve the needs of both the client’s actuarial and data science teams — one Hive LLAP cluster and one Spark cluster.
Data was migrated from SAP storage to Azure for consumption through the new Hadoop clusters. Both were designed to maximize the scalability of the HDInsight environment, automatically scaling down to a small number of nodes during times of low demand and scaling up when workloads hit. This elasticity improved performance while dramatically reducing costs compared to the legacy system.
Delivering greater value for less
The project came to a successful close on schedule by the end of the year, enabling the client to avoid hefty fines — and making Insight’s team the first to successfully complete an SAP BDS to HDInsight migration.
The shift has reduced monthly cloud expenses by nearly 80% and saved the client more than $750K per month. In addition to reduced costs, the environment has dramatically improved visibility and productivity for data teams.
The success of this project has led to an ongoing relationship and created new opportunities for our team to assist with the organization’s greater transformation goals, including new initiatives with DevOps, data analytics and SharePoint®.