Introduction
This is Johann from the Global Engineering Department of the GLB Division.
In this post, we will discuss the presentation "Delta Sharing: The Key Data Mesh Enabler" by Francesco Pizzolon, a Data Engineering Director at Avanade. With over 12 years of experience in data engineering across various industries such as banking, retail, transportation, energy, and the public sector, Francesco is well-equipped to share his insights on data mesh and Delta Sharing. The theme of this presentation is "Delta Sharing: The Key Data Mesh Enabler," aiming to explain the key enablers of data mesh. The target audience includes technical professionals interested in Data & AI, as well as CIOs and business leaders of companies building data platforms. This blog post is divided into two parts, and this is the second part. In the first part, we introduced a case study of building a data platform using data mesh and Delta Sharing. In this second part, we will discuss the comparison between centralized data platforms and data mesh, the importance of standardization when choosing a data mesh, the advantages of Delta as a data format standard, and the reasons for adopting Delta Sharing as a data sharing standard. Let's dive right into the topic!
Choosing a Data Platform and the Advantages of Delta
Comparing Centralized Data Platforms and Data Mesh
There are two approaches to choosing a data platform: centralized and data mesh. Centralized data platforms provide centrally managed data, ensuring data quality and security. However, this approach may limit data usage and sharing. On the other hand, data mesh manages data in a distributed manner, promoting data usage and sharing. In a data mesh, data owners can freely manage their data and share it with other teams and organizations. This makes data usage easier and fosters innovation.
The Importance of Standardization When Choosing a Data Mesh
Standardization is crucial when implementing a data mesh. By using standardized data formats and APIs, data usage and sharing become easier. Additionally, standardization ensures data quality and security. Key points of data standardization include:
Unifying data formats
Standardizing APIs
Ensuring data quality
Securing data
Choosing Delta as a Data Format Standard and Its Advantages
Delta is gaining attention as a data format standard for data mesh due to the following advantages:
Schema evolution: Delta can flexibly accommodate schema changes, allowing for rapid adaptation to data evolution.
Transaction support: Delta supports ACID transactions, ensuring data consistency.
Fast query execution: Delta improves query execution speed through data compression and partitioning.
Open format: Delta is an open format, making it compatible with various data processing tools.
To realize a data mesh, a data format standard like Delta is essential. By leveraging Delta, data usage and sharing become easier, enabling effective operation of data platforms.
Reasons to Adopt Delta Sharing as a Data Sharing Standard
Delta Sharing is gaining attention as an indispensable element for the success of data mesh. Data mesh aims to provide an environment where data can be easily shared and utilized, requiring standardized data sharing. Delta Sharing should be adopted as a data sharing standard for the following reasons:
Open source: Delta Sharing is open source, allowing anyone to freely use, modify, and distribute it. This lowers the barrier to data sharing, enabling more people to participate in data mesh.
Data compatibility: Delta Sharing supports various data sources and formats, offering high data compatibility. This makes data handling easier, improving the convenience of data mesh.
Security: Delta Sharing excels in security aspects such as access control and encryption. This ensures data safety and enhances the reliability of data mesh.
Success Metrics and Current State of Data Mesh Platforms
To measure the success of data mesh platforms, the following metrics are important:
Data usage rate: This metric indicates how much data provided by the data mesh platform is being utilized. The higher the usage rate, the more valuable the data mesh platform becomes.
Data quality: The higher the quality of data provided by the data mesh platform, the more reliable the data becomes, increasing the value of the data mesh platform.
Data accessibility: The easier it is to access data provided by the data mesh platform, the more valuable the platform becomes.
Currently, data mesh platforms are still in development, and few platforms are said to meet these metrics. However, with the proliferation of technologies like Delta Sharing, the success of data mesh platforms is becoming more realistic.
Future Outlook: Introducing Marketplaces and Developing Machine Learning Data Products
Two key elements are considered important for the future of data mesh:
Introducing marketplaces: By incorporating marketplaces into data mesh platforms, data transactions become easier, and the value of data becomes more apparent. This improves the convenience of data mesh platforms and promotes data usage.
Developing machine learning data products: Developing machine learning data products on data mesh platforms expands the range of data usage. This increases the value of data mesh platforms and promotes data usage.
To realize these outlooks, data sharing standards like Delta Sharing play a crucial role. The continued proliferation of Delta Sharing is expected to accelerate the success of data mesh.
Summary
In this blog post, we discussed the choice of data platforms and the advantages of Delta, as well as the reasons for adopting Delta Sharing as a data sharing standard. Data mesh and Delta Sharing are essential elements for promoting data usage and sharing, accelerating innovation. By utilizing these technologies, effective operation of data platforms becomes possible. As data mesh and Delta Sharing continue to gain traction, data usage is expected to be promoted, and the value of data platforms will increase. We encourage you to explore data mesh and Delta Sharing to advance your data usage efforts.
Conclusion
This content based on reports from members on site participating in DAIS sessions. During the DAIS period, articles related to the sessions will be posted on the special site below, so please take a look.
Translated by Johann
Thank you for your continued support!
