Ray on Apache Spark™ Part 1

Ray on Apache Spark™: Scaling AI and Python Applications Made Easy with a Unified Distributed Framework

​ I'm Johann from the Global Engineering Department of the GLB Division. I wrote an article summarizing the content of the session based on the report by Gibo, who is participating in Data + AI SUMMIT2023 (DAIS) on site.

This time I'll be covering Ray, an open source unified distributed framework. The talk explained how to simplify the scaling of AI and Python applications and how to make them accessible to anyone without expertise in distributed systems. This blog consists of two parts, and this time we will deliver the first part. ​

What is Ray?

​

Ray is an open source unified distributed framework with the following features: ​

• Simplifies scaling AI and Python applications

• Accessible to anyone without distributed system expertise

• Highly flexible and scalable

Simplify scaling of AI and Python applications

​ Ray was developed with the goal of simplifying the scaling of AI and Python applications. With conventional distributed systems, it was often difficult to scale and required specialized knowledge, but using Ray has the following advantages. ​

1. Distributed processing can be easily implemented with a simple API

2. Flexible scaling and easy resizing of clusters

3. Fault-tolerant and automatically recovers from failures ​

   Accessible to anyone without distributed system expertise

   ​ Ray is designed to be accessible to anyone without expertise in distributed systems. This has the following advantages: ​

4. Even if you are not a distributed system expert, you can easily implement distributed processing.

5. System performance can be maintained even if the number of users increases

6. Make distributed systems accessible to more people, fostering innovation

Latest concepts and features

​ Ray is being developed with the latest concepts and features. Below is an example. ​

• Ray Serve: Ability to easily deploy and scale services

• Ray Tune: A function for efficient hyperparameter tuning

• Ray RLlib: Features that make it easy to implement and evaluate reinforcement learning algorithms

​ By using these functions, developers can build and operate distributed systems more efficiently. ​

Summary

​ Ray is an open source, unified distributed framework designed to simplify scaling AI and Python applications and make it accessible to anyone without distributed systems expertise. It also incorporates the latest concepts and features, making it a very attractive framework for developers. In the next part, Part 2, we will explore Ray on Apache Spark™ in detail. looking forward to!

Conclusion

This content based on reports from members on site participating in DAIS sessions. During the DAIS period, articles related to the sessions will be posted on the special site below, so please take a look.

Translated by Johann

https://www.ap-com.co.jp/data_ai_summit-2023/

Thank you for your continued support!