Introduction
This is Abe from the Lakehouse Department of the GLB Division. I wrote an article summarizing the content of the session based on reports from Mr. Ichimura participating in Data + AI SUMMIT2023 (DAIS).
This time, AT&T will give a talk titled "Building and Managing a Data Platform for a Delta Lake that Exceeds 13 Petabytes," which will introduce the business benefits of building and managing a Delta Lake data platform that exceeds 13 petabytes, as well as the challenges of migrating to the cloud. Petabytes and Has Thousands of Users: AT&T's Story. Speakers of the talk were Praveen Vemulapalli, Technology Director, Chief Data Office, AT&T, and Jegadeesan Pugazhenthi (JP), Lead Big Data Engineer, AT&T CDO.
Let's take a look at the contents of the lecture!
Cloud Migration Background and Choices
Decided to move to the cloud
AT&T decided to move to the cloud to build and manage Delta Lake's 13+ petabyte data platform. This included cost efficiency, workload management, and security concerns, but they determined that overcoming them would yield business benefits.
Selecting Microsoft as Cloud Provider for Data AI Workloads
AT&T chose Microsoft as its cloud provider for its data and AI workloads. This will allow AT&T to efficiently build and manage Delta Lake's data platform.
Cloud migration challenges and solutions
Challenge 1: Data Migration
Data migration is one of the big challenges of cloud migration. In particular, migrating data volumes exceeding 13 petabytes is expected to be time-consuming and costly.
Solution
AT&T has adopted the following methods to streamline data migration:
- Data classification: By classifying and prioritizing data according to importance and access frequency, efficient migration becomes possible.
- Data Compression: Data compression can reduce migration time and costs.
- Parallelism: Parallelizing data migration allows efficient data migration.
Issue 2: Security
Security is a key issue in cloud migration. In particular, as laws and regulations regarding data protection and privacy are becoming stricter, appropriate measures are required.
Solution
AT&T has adopted the following security measures:
- Data encryption: Encrypting data protects against unauthorized access and ensures compliance with privacy regulations.
- Access Control: Implementing strict access control policies can prevent unauthorized access to sensitive data.
- Regular security audits: Conducting regular security audits helps identify and address potential system vulnerabilities.
In conclusion, AT&T's story of building and managing over 13 petabytes of Delta Lake data platform presents challenges and solutions in cloud migration. By overcoming these challenges, we believe that enterprises can enjoy the benefits of cloud-based data platforms and drive innovation in each industry.
Conclusion
He explained the content of the session based on reports from members participating in DAIS on site. During the DAIS period, articles related to the sessions will be posted on the special site below, so please take a look.
https://www.ap-com.co.jp/data_ai_summit-2023/
Thank you for your continued support!
Translated by Johann
