Building and Managing a Data Platform for a Delta Lake that Exceeds 13 Petabytes and Has Thousands of Users: AT&T's Story

Introduction

This is Abe from the Lakehouse Department of the GLB Division. I wrote an article summarizing the content of the session based on reports from Mr. Ichimura participating in Data + AI SUMMIT2023 (DAIS).

This time, AT&T will give a talk titled "Building and Managing a Data Platform for a Delta Lake that Exceeds 13 Petabytes," which will introduce the business benefits of building and managing a Delta Lake data platform that exceeds 13 petabytes, as well as the challenges of migrating to the cloud. Petabytes and Has Thousands of Users: AT&T's Story. Speakers of the talk were Praveen Vemulapalli, Technology Director, Chief Data Office, AT&T, and Jegadeesan Pugazhenthi (JP), Lead Big Data Engineer, AT&T CDO.

Let's take a look at the contents of the lecture!

Cloud Migration Background and Choices

Decided to move to the cloud

AT&T decided to move to the cloud to build and manage Delta Lake's 13+ petabyte data platform. This included cost efficiency, workload management, and security concerns, but they determined that overcoming them would yield business benefits.

Selecting Microsoft as Cloud Provider for Data AI Workloads

AT&T chose Microsoft as its cloud provider for its data and AI workloads. This will allow AT&T to efficiently build and manage Delta Lake's data platform.

Cloud migration challenges and solutions

Challenge 1: Data Migration

Data migration is one of the big challenges of cloud migration. In particular, migrating data volumes exceeding 13 petabytes is expected to be time-consuming and costly.

Solution

AT&T has adopted the following methods to streamline data migration:

1. Data classification: By classifying and prioritizing data according to importance and access frequency, efficient migration becomes possible.
2. Data Compression: Data compression can reduce migration time and costs.
3. Parallelism: Parallelizing data migration allows efficient data migration.

Issue 2: Security

Security is a key issue in cloud migration. In particular, as laws and regulations regarding data protection and privacy are becoming stricter, appropriate measures are required.

Solution

AT&T has adopted the following security measures:

1. Data encryption: Encrypting data protects against unauthorized access and ensures compliance with privacy regulations.
2. Access Control: Implementing strict access control policies can prevent unauthorized access to sensitive data.
3. Regular security audits: Conducting regular security audits helps identify and address potential system vulnerabilities.

In conclusion, AT&T's story of building and managing over 13 petabytes of Delta Lake data platform presents challenges and solutions in cloud migration. By overcoming these challenges, we believe that enterprises can enjoy the benefits of cloud-based data platforms and drive innovation in each industry.

Conclusion

He explained the content of the session based on reports from members participating in DAIS on site. During the DAIS period, articles related to the sessions will be posted on the special site below, so please take a look.

https://www.ap-com.co.jp/data_ai_summit-2023/

Thank you for your continued support!

Translated by Johann