What's New with Data Sharing and Collaboration

Preface

In the inaugural segment of this "Latest in Data Sharing and Collaboration” session, we delve into the complexities of data sharing and how AI-powered and Databricks is revolutionizing this field.

The modern business landscape is heavily reliant on various types of data and applications, resulting in inherent complexities and high costs related to sharing and collaboration. To execute crucial business initiatives, data teams need the capacity to securely share their data and AI assets beyond organizational boundaries and leverage them effectively.

Regrettably, the current market often offers fragmented and limited solutions, hindering enterprises from reaching their full potential. Delivery methods can greatly vary between vendors, leading to inconsistencies in data integration and causing redundant data assets, outdated data assets, and an increase in privacy and security risks.

Adding further complexity, companies using different platforms face additional hurdles. Collaboration becomes a prime issue, as sharing is impaired among businesses not using the same vendor. Lastly, the limitation of sharable data types, especially in regards to unstructured data, means diverse data sets (AI models, volumes, code, etc.) become unsharable, limiting the complete utilization of AI within businesses.

The Ideal Environment for Data and AI Sharing and Collaboration

The ideal setting for sharing and collaboration would be a platform that promotes an open approach to maximize an organization's reach. This would be a flexible and open unified data sharing and collaboration solution. A major feature of this is to streamline the integration of data and AI, which in turn promotes innovation.

The capability to securely perform live sharing of data and AI is essential for driving innovation. To enable dynamic engagement with customers and partners, a friendly environment promoting open, cross-platform collaboration is requisite. These characteristics pave the way for a robust cross-platform ecosystem.

Finally, a collaborative environment enabling the sharing of tables, volumes, AI models, and notebooks promotes an ambiance where everyone can fully leverage data and AI.

Unveiling Databricks' Data and AI Sharing Solution

Databricks introduces the “Databricks Data Intelligence Platform” - which offers the most secure, open, and flexible ecosystem for data and AI sharing and collaboration. This platform unleashes a broad range of possibilities for the entire organization to fully utilize its data and AI.

Built on a Lakehouse, the Data Intelligence Engine provides an open and unified infrastructure for all data types. A significant aspect of Databricks Data Intelligence Platform is data sharing and collaboration, vastly simplified by leveraging Delta Sharing. This tool waives the constraints of specific vendors and enables easy sharing of data and assets beyond organizational boundaries.

Additionally, the support functionalities of the platform's Unity Catalog facilitate effective, centralized governance. Databricks excels by pushing forward effective sharing and collaboration of data and AI.

In conclusion, this article explored the recent advances in data and AI sharing and collaboration. By harnessing the power of Databricks, we can overcome the inherent challenges accompanying data and AI sharing and achieve swifter, more effective innovation.

The Growth of Partnerships and Ecosystem in Data Sharing and Collaboration

Organizations are increasingly recognizing the value of a secure, open ecosystem for data and AI sharing, and Databricks is following suit. In the past two years since its public outreach began, Databricks has shown considerable growth and progress, particularly in the area of Delta Sharing.

In the past year, the Delta Sharing ecosystem has grown impressively pervasive. At present, over 16,000 users are utilizing Delta Sharing, transferring data and AI assets across various clouds, platforms, and regions. This indicates a 300% increase in active monthly Delta sharing from the previous year.

Apart from this, the diversity in utilization across platforms reflects Databricks’ commitment to an open ecosystem. Particularly interesting is that 40% of Delta Sharing now connects to non-Databricks platforms through open connectors within the Delta Sharing Framework. These platforms include industry standards such as Apache Spark, Excel, Pandas, Power BI, and Tableau, among others.

Pursuing leadership in the data analytics realm, Databricks has recently initiated new and expanded strategic partnerships with major industry players. The purpose of these partnerships is to simplify and enhance data sharing and cooperation.

As we delve deeper into the realm of data and AI, Databricks' innovative initiatives are paving new paths when organizations seek fresh approaches to optimize operations, providing valuable guidance. Through Databricks' groundbreaking efforts in data and AI sharing, we can catch a glimpse of the future of data analytics and collaboration.

Innovative and Cost-Efficient Solutions by Databricks

In the era of machine learning and AI, organizations are often called upon to effectively manage and collaboratively process massive volumes of data, especially in handling unstructured data and semi-structured data. Databricks, a leading platform in the industry, provides smart solutions with inventive, efficient strategies for these challenges.

One of Databricks' highlights is the newly added object type, "Volume," to the Unity Catalog. This feature accommodates the processing and management of various data formats, including non-tabular data such as images, audio, videos, PDF files, and more. The introduction of Volume permits data sharing without creating duplicates, encouraging efficient use of multimodal data and fostering productive collaborations. The potential of these innovations lies in their ability to significantly accelerate tasks involving machine learning or AI.

Databricks doesn’t stop there. It extended the functionality of its popular "View Sharing" feature by leveraging its commitment to continuous innovation. Previously, this tool allowed for the assignment of fine-access control at the column or row level to providers, which, however, was only applicable to recipients within the Databricks platform. Breaking these barriers, Databricks now extends the ability of "View Sharing" to non-Databricks recipients as well - a crucial move in expanding the platform's relevance and scope.

These developments introduced at Databricks' latest session reaffirm the company’s investment in providing innovative, seamlessly integrated, cost-efficient solutions consistently for data collaboration and sharing. The ongoing addition of these advanced features is invaluable for organizations navigating the evolving landscape of machine learning and AI.

Latest on Databricks Marketplace and Cleanroom

Exciting updates have recently been shared about the Databricks Marketplace and Cleanroom. Just one year from its general availability (GA), the Databricks Marketplace has seen phenomenal growth. The diversity of available products has expanded, enabling the discovery, procurement, and sale of data and AI products of all kinds. Private exchanges have also been launched to help providers reach their target consumers.

Until now, most marketplaces offered mainly tabular data products. This was a significant limitation for consumers seeking comprehensive analytics and AI solutions. However, thanks to data sharing innovations and the Databricks Marketplace, access within the marketplace to both proprietary AI models and open-source AI models is now possible.

Launch partner Jon Snow Labs has already released 60 cutting-edge AI models ready for use. Moreover, Databricks is continuously curating and publicizing popular open-source foundational models like the LAMA3 model within the marketplace.

The benefits of the new Databricks Marketplace aren't limited to just generic AI models. Models tailored to specific needs are also possible, allowing businesses to optimize value and promote innovation in data analytics.

As we look ahead, the Databricks Marketplace promises to continue its innovation and supply intriguing developments. Stay tuned for upcoming trends.

Conclusion

Through these updates, the Databricks Marketplace and Cleanroom have significantly evolved, driving further innovation and collaboration in data and AI development. Meeting customer needs while demonstrating the ability to push innovation towards the future, these advancements can help businesses achieve more specific goals and maximize the utilization and advancement of data analytics. As we eagerly anticipate the future evolution of the Databricks Marketplace and Cleanroom, we strive to optimize the use of existing products and services.

About the special site during DAIS

This year, we have prepared a special site to report on the session contents and the situation from the DAIS site! We plan to update the blog every day during DAIS, so please take a look.

www.ap-com.co.jp