Databricks As Code: How to Effectively Automate a Secure Lakehouse Using Terraform for Resource Provisioning

Introduction

This is Abe from the Lakehouse Department of the GLB Division. I wrote an article summarizing the content of the session based on the report by Gibo, who is participating in Data + AI SUMMIT2023 (DAIS) on site.

Articles about the session at DAIS are summarized on the special site below. I would appreciate it if you could see this too.

www.ap-com.co.jp

Let's get down to business!

Rivian data utilization and introduction of Databricks Lakehouse

This time, I'm featuring a talk on how to effectively automate a secure Lakehouse using Databricks. Rivian engineer Vadivel Selvaraj and manager Jason Shiverick spoke.

Aimed at data engineers, data architects, data scientists, and business leaders.

Collection and utilization of vehicle data

Rivian collects terabytes of data per day from 32,000 vehicles. This data is used for purposes such as:

Improved product design
Fleet health monitoring
Improved customer experience

By utilizing these data, Rivian aims to improve business efficiency and competitiveness.

Data integration with Databricks Lakehouse

Rivian uses Databricks Lakehouse to aggregate data and integrate it into Delta Lake. This reduces architectural complexity and makes data easier to manage and analyze.

Databricks Lakehouse has the following features.

Architecture that combines data warehouse and data lake functionality
Enables high-speed data processing and analysis
Enhanced security and data governance

Automate resource provisioning with Terraform

This talk introduced how to effectively automate a secure Lakehouse using Databricks. Among them, automation of resource provisioning using Terraform was taken up.

Terraform has the following features.

You can manage your infrastructure with code
Compatible with multiple cloud providers
Create, update, and delete resources automatically

Terraform enables efficient resource provisioning for Databricks Lakehouse. This can help accelerate data utilization and grow your business.

Summary

Rivian leverages vast amounts of vehicle data effectively with Databricks Lakehouse. In addition, by automating resource provisioning using Terraform, we support the promotion of data utilization and business growth. In the future, optimization of data utilization and introduction of new technologies are expected.

Conclusion

This content based on reports from members on site participating in DAIS sessions. During the DAIS period, articles related to the sessions will be posted on the special site below, so please take a look.

Translated by Johann