Databricks’ Journey: Driving Business Transformation with Data & GenAI

Preface

This session revolves around Databricks' efforts to drive business transformation. We particularly delve into two key themes - initiating internal changes and elevating data literacy. By utilizing their unique Data Intelligence Platform along with the GenAI feature, we introduce the process of how Databricks has overcome challenges and established a proprietary design philosophy.

Databricks’ Transformative Journey

The journey starts from Databricks itself. By incorporating its own platform internally, it has instigated business transformations across the organization. This project was led by Deepa Gopinath, the Senior Vice President and Deputy CIO of Databricks, along with Romit Jadhwani, who oversees the company's data and analytics.

The Big Impact of A Small Team

Despite being a relatively smaller unit of around 130 staff members among a total of 7,000, the IT team of Databricks plays an extremely crucial role in the company's business transformation.

The IT organization of Databricks is modelled on a product-oriented mindset. Here, IT Product Managers form close partnerships with individual business functions like HR, finance, marketing, sales, customer support, and product edge. This enables them to prepare an IT delivery roadmap, focusing on business objectives and thus facilitating business operations.

This marked the genesis of Databricks' business transformation journey. In the forthcoming articles focusing on reflecting upon this journey, we will delve deeper into how Databricks achieved internal transformation and how they enhanced data literacy.

Unifying Data Management and Quality with Databricks

This section focuses on Databricks' approach to establishing a solid foundation for reliable data by creating what they call Gold Schemas and Gold Tables. Essentially, these are verified, reliable data sets that function as primary feeds into tables and the BI layer.

One challenge that Databricks faced was maintaining user experience - specifically, they aimed to provide direct data insights within tools with which the users are acquainted. The solution to this challenge was reflexively inserting crucial insights into the tools that users interact with, through ETL.

They also made efforts to facilitate the deployment of models and features, the details of which will be discussed later. At first glance, their work may seem focused only on improving data management and data quality. However, underneath these efforts is Databricks' ultimate goal of providing data insights to the end-users and hence, bringing about a transformation in business. This attests to their far-reaching vision.

Through the discussed case studies, we have seen how Databricks leverages data and AI innovatively to drive changes in the industry.

Databricks' Journey: Propelling Business Transformation Using Data and GenAI

Proactive Workflow and Sales Efficiency

In leveraging the capabilities of their Data Intelligence Platform and GenAI, Databricks is driving business transformation. Participants in this session can understand the intricacies of their journey in overcoming various challenges in establishing a specific design philosophy. In this particular section, we focus on the theme of "proactive workflows and sales efficiency," and introduce their approach, the processes, and the lessons learnt along the way.

Firstly, the session emphasized the importance of having a structured metrics and measurement strategy. This strategy enabled building a structural view of the metrics tree hence achieving consistent processes.

From a business perspective, it was highlighted as very essential to understand who the end-users are. Databricks has a product management organization, which through its representatives has been actively collaborating with the end-users. By conducting comprehensive user research and interviews, it provides insights on how users interact with data, thus contributing significantly in shaping their site experience.

Lastly, the session spoke about the significance of the role of governance. In the context of Databricks' products, it was stated that there are several out-of-the-box features available to initiate governance right from the inception of the schema to the configuration of data lakes.

The second case study addressed the idea of "considerations beyond data." Here it was confirmed that the data actually exists in the lakehouse.

Drawing from these insights, it became clear that the emphasis on building proactive workflows and focusing on sales efficiency greatly contributed to Databricks' business transformation and excites its future development.

AI Governance and Hybrid Strategy in Databricks

Having embarked on the journey of propelling business transformation, Databricks brought about a major pivot in their internal procedures by leveraging their Data Intelligence Platform and GenAI. At the heart of this transformation lies the integration of AI governance and a hybrid strategy. We delve into these elements to understand their strategy in depth and gain insights from their experiences.

Adoption of AI Governance

To effectively harness the power of GenAI, Databricks incorporated AI governance into the crux of their strategy. This initiative played a crucial role in managing potential regulatory and security issues at the implementation stage, and ensuring that the designed solution was in complete alignment with the target requirements.

Collaboration was the key. The legal, privacy, and security teams of Databricks worked together and embedded AI governance into all its trial projects. This not only provided a robust environment for the development of solutions but also ensured safe roll-outs.

Hybrid Strategy for Evaluation

Strategically as a valuable move, Databricks adopted a hybrid strategy. This tactic included the adoption of readily available Software as a Service (SaaS) solutions in the market, and simultaneously, the development of proprietary solutions addressing unmet needs in the open market. Hence, Databricks only invests effort to internally develop when adequate market solutions tailored to specific requirements are lacking.

The introduction of AI governance and a hybrid strategy became vital elements for the successful integration of GenAI at Databricks, bringing about a significant enhancement in operational efficiency. The evolution of these strategies and their effects will be further explored in future sessions. The transformation of business driven by data intelligence, the potential of AI, and adept strategy further propels the concept of transforming business through leveraging information.

Embracing the Journey with Databricks: Driving Business Transformation through Data & GenAI

The session provided a comprehensive overview of the process that Databricks utilized their Data Intelligence Platform and GenAI to drive business transformation. This section focuses specifically on the aspect of "Improving Productivity through GenAI."

Improving Productivity through GenAI

The session introduced the AI-driven outbound prospecting system developed by Databricks. This system handles everything through AI, achieving results equivalent to human involvement while demonstrating communication response speeds and opening rates on par with human interaction. The utilization of GenAI in this manner brought a significant boost to Databricks' productivity.

Discussion also extended to user experience and field productivity. Databricks developed a data intelligence-enhanced tool, Field Sidekick, to improve their own product. Specifically, when sales representatives prepare for client meetings, they can use this tool to quickly review previous meeting summaries, current ongoing issues, and recommended next steps, thereby enhancing the efficiency of meeting preparation.

Conclusion

Data and AI are integral tools in today's business setup. As a leader in this field, Databricks is constantly exploring new ways to optimize business processes and increase productivity by leveraging data and AI. By utilizing GenAI, they streamlined their sales operations and daily processes. Their efforts have greatly benefited their clients, setting an example of strategies in this area, offering insights and learnings.

About the special site during DAIS

This year, we have prepared a special site to report on the session contents and the situation from the DAIS site! We plan to update the blog every day during DAIS, so please take a look.

www.ap-com.co.jp