Spark [Driver and Executor] Memory Management Deep Dive

Summary

This video script delves into Spark's memory management for driver and executor processes. It explains the JVM process, heap memory, and non-JVM overhead memory, emphasizing the 10% overhead rule for container memory allocation. The script further breaks down JVM heap memory into reserved, Spark, and user memory, detailing their ratios and purposes. It also discusses the dynamic sharing between storage and executor memory pools, the importance of understanding these allocations to prevent 'out of memory' exceptions, and the minimum memory requirements for executors relative to reserved memory.

Takeaways

• 🌟 When submitting a Spark application, it goes to the master node which executes the driver and executor containers with specific memory allocations.
• 🔍 JVM processes in Spark consist of Heap memory for JVM operations and non-JVM overhead memory used by non-JVM processes.
• 📏 Non-JVM overhead memory is calculated as 10% of the actual container memory or a minimum of 384 MB, whichever is higher.
• 🔢 The 'spark.driver.memory' and 'spark.executor.memory' configurations determine the memory allocation for driver and executor containers, respectively.
• 🧩 JVM Heap memory is further divided into reserved memory, Spark memory, and user memory, with specific ratios and purposes.
• 📊 Reserved memory is a fixed allocation for Spark's internal operations, while Spark memory and user memory can be adjusted based on requirements.
• 💾 Spark memory is used for tasks like caching and is divided into storage memory pool and executor memory pool, which can be dynamically shared.
• 🔄 Storage memory pool is used for caching and persisting data, while executor memory pool is used for data processing activities.
• 🛑 If the executor memory is less than 1.5 times the reserved memory, Spark will fail with an error message prompting a larger heap size.
• 🛠 User memory is utilized for user-defined functions (UDFs), broadcast variables, and other objects not managed by Spark's internal operations.
• ⚠️ Understanding the memory allocation and usage is crucial for troubleshooting issues like Spark out-of-memory exceptions.

Q & A

What happens when an application is submitted to a Spark cluster?

When an application is submitted, it goes to the master node which then executes the driver and executor containers with specified resources like CPU and memory.

What is the purpose of JVM Heap memory and non-JVM process overhead memory in Spark containers?

JVM Heap memory is managed by the JVM for the application's use, while non-JVM process overhead memory is used by the container process for internal operations and is typically 10% of the actual container memory or a minimum of 384 MB.

What is the minimum non-JVM overhead memory allocated to a container if it's assigned 4GB?

If a container is assigned 4GB, the minimum non-JVM overhead memory allocated is 400MB, based on the 10% rule or the minimum threshold of 384MB.

How is the memory allocated to an executor in Spark?

The executor memory allocation in Spark follows the same principle as the driver, with 10% of the executor's assigned memory going to overhead and the rest divided into JVM and user memory.

What are the three types of memory within JVM Heap memory in Spark?

Within JVM Heap memory, there are reserved memory, Spark memory, and user memory. Reserved memory is for Spark's internal use, Spark memory is for caching and computation, and user memory is for user-defined functions and other user operations.

What is the default ratio between Spark memory and user memory?

By default, the ratio is 60% for Spark memory and 40% for user memory, but this is configurable through the 'spark.memory.fraction' setting.

How does Spark divide Spark memory into storage memory pool and executor memory pool?

Spark memory is divided into a storage memory pool for caching and persisting data, and an executor memory pool for data processing activities, with a default ratio of 50/50.

What is the purpose of the storage memory pool in Spark?

The storage memory pool is used for caching and persisting data, such as RDDs, DataFrames, and Datasets, and is dynamically shared with the executor memory pool based on workflow demands.

What is the purpose of the executor memory pool in Spark?

The executor memory pool is used for data frame compute operations like joins, aggregations, shuffles, and transformations, and can dynamically share memory with the storage memory pool when needed.

What could cause a Spark application to throw an 'Executor out of memory' exception?

An 'Executor out of memory' exception could be caused by several reasons, including shortage of overhead memory, insufficient storage pool, executor pool, or user memory.

What is the minimum executor memory requirement in relation to reserved memory?

The executor memory should be at least 1.5 times the reserved memory to avoid a 'please use a larger heap size' error. For example, if reserved memory is 300MB, the executor memory should be at least 450MB.