Data Platform Modernization with a Data Mesh Architecture

Preface

In order to quantify the scale of their operations, Lazans shared a short video that clarified the organization’s core beliefs and targets. This video, made in Portuguese and subtitled in English, emphasized the fact that Bradesco is chosen daily by over 70 million Brazilians.

The challenge for large organizations like Bradesco is to constantly evolve, adapt, and grow while maintaining reliability and trust with millions of customers, in an environment where the technology doesn't stay the same for long. This constant quest for growth and evolution is inseparable from the company's efforts towards data and AI.

Moving towards the theme of the session, Lazans introduced us to the principles of modernizing data platforms and the concept of data mesh architecture. This is the main focus of the session and Bradesco is employing this architecture to improve data ingestion and quality, aiming to reap maximum benefits.

Lazans concluded the intro with an invitation for us to embark on a journey to unlock the potential of data and AI. As participants in this session, we are ready to learn, absorb, and share the knowledge we have gained so far, and we are looking forward to exploring the infinite possibilities of data and AI together.

In essence, the session started by comprehensively introducing the role of digital transformation and modern data platforms, and the role of data mesh architecture. The session got off to an intriguing start, thanks to Lazans' expertise and Bradesco's commitment to unlocking potential from data and AI. We are looking forward to delving deeper into the topic in the next session.

Modernizing Data Platforms: Challenges and Transition to Data Mesh Architecture

Have you ever processed data generated in various formats like batch, stream, structured, unstructured, etc., through more than 150 data repositories? Moreover, the client response layer consists of solutions and initiatives at various levels. All these predominantly exist on-premise, posing numerous challenges.

Firstly, the cost of data scaling is skyrocketing. Then, there are issues related to growth. Expansion always requires synchronization and demands consistent resource allocation and time investment. Lastly, there is a time element. Projects span over extended periods and sometimes take 1-2 years to finish.

As the diversity in technology increases, the environment becomes more complex. This complexity requires frequent containerization efforts and adds great complexity to data pipeline development.

To address these issues, many companies are turning their attention to Data Mesh Architecture. Data Mesh is a new paradigm of data architecture that delegates large-scale data projects to different business domains. Each domain functions as an independent product and optimizes the scaling, management, and development of data.

However, there can be a sense of not knowing where to start. This article focuses on where to begin dealing with challenges that come with transition to data mesh. Let's dig deeper.

As data generated in diverse formats like batch, stream, structured, unstructured, etc., gets processed through more than 150 data repositories, a series of challenges have emerged. Client response layer consists of solutions and initiatives at various levels functioning as an on-premise environment.

Cost of data scaling is skyrocketing. Growth presents challenges, demanding constant synchronization, consistent investment of resources, and time. Moreover, the time frame for projects extends into the long term with the incorporation of new changes taking from 1-2 years.

The environment is becoming more complex as the diversity of technology increases. Constant containerization is required, significantly increasing the complexity of developing data pipelines.

As a solution to these existing challenges, many organizations are shifting their focus towards the Data Mesh Architecture. Data Mesh, an emerging data architecture paradigm, delegates large-scale data projects to different business domains. Each domain performs as an autonomous product, efficiently optimizing data scaling, management, and development.

However, knowing exactly where to start dealing with challenges that come with transitioning to data mesh can be difficult. This article focuses on the starting points to clarify these challenges. Let's delve into the specifics and explore concretely.

Evolution and Cooperation of Data Platform: A Case Study of Bradesco

Bradesco, one of the largest private banks in Brazil, boasting over 75 million customers and a business history of over 80 years, has made significant strides in modernizing its data platform. This article will take a closer look at the steps Bradesco has taken in promoting the evolution and collaboration of its data platform.

Leap for Cloud: The Foundation for Digital Transformation

In 2021, Bradesco launched a program called 'Leap for Cloud', putting a spotlight on modernizing digital platforms and customer governance. Subsequently, a corporate project with Microsoft began in September 2022, and by the end of December 2022, landing zones, including network and security systems, were established.

By August 2023, part of the technical team successfully delivered over 2500 histories across all environments. Advancing further into September 2023, all teams were directed to adopt the new architecture into their projects. This indicated the beginning of a new data platform project in January 2023.

The New Data Platform

The new data platform project officially started in January 2023 and was officially recognized as Bradesco's new data platform in June of the same year. With the modernization of this platform, interactive business domains emerged and communication between platforms was enhanced.

This modernization is a testament to Bradesco's continuing effort towards evolution and collaboration of data platforms, and brought about a stepping stone to elevate their business to the next level. It strongly suggests that modernizing data platforms is an essential requirement for the progression and growth of any business.

Through these efforts, Bradesco provides valuable insights into how data platforms can evolve and collaborate. This depicts an appealing story of how an organization has been able to transform its data infrastructure to respond to the advancements in technology and the increased business demands, offering a blueprint to other organizations following the same path.

Implementation of Ingestio Manager and MLOps

Bradesco bank, one of the largest private banks in Latin America, with more than 75 million customers and over 80 years of experience in the Brazilian banking industry, is managing its existing voluminous corporate data by focusing on data ingestion with the use of technologies like Ingestio Manager and Machine Learning Operations (MLOps). The bank is persistently working towards modernizing its data platform. Let's delve into the specifics of those technologies.

Enhanced Error Handling Capabilities

Ingestio Manager comes equipped with robust error handling capabilities, allowing for re-running of tests as required. Notably, this platform supports standardized tests for conducting activities such as monitoring and transforming data fields, including numerical and time data.

Data Dashboard Visualization

Furthermore, Ingestio Manager provides a data dashboard that includes entry data, errors, and reprocessing information. This allows data administrators to monitor the status of data in real-time, facilitating prompt and efficient responses.

File Management

Bradesco shared an example of managing files using Ingestio Manager. You can define the domain each dataset belongs to and the type of file using the section highlighted in yellow on the left side of the dashboard. This feature proves especially handy in identifying file types, such as EDCD compilation files for mainframes. Detailed information about each file is displayed on the right side of the dashboard.

Performance Demonstration

Ingestio Manager continues to evolve even after implementation. Just last month, it assisted in uploading over 37,000 files to the platform, demonstrating its effectiveness and usefulness in data management.

MLOps Implementation

Bradesco is further enhancing its data management efficiency by incorporating MLOps, allowing for automation of processes like updating algorithms, preparing data, training machine learning models, etc.

To summarize, the implementation of Ingestio Manager and MLOps at Bradesco symbolizes an investment in cutting-edge technology to effectively achieve data management centralization and quality improvement. They signify a commitment to continuously adopt and evaluate new technologies to propel business growth.

The session emphasizes the necessity of adopting Data Mesh Architecture to modernize modern data platforms for effective cost management and data product development. Though new challenges loom, eagerness and passion to jointly shape the future of data were expressed

About the special site during DAIS

This year, we have prepared a special site to report on the session contents and the situation from the DAIS site! We plan to update the blog every day during DAIS, so please take a look.

www.ap-com.co.jp