Detailed explanation of MaxCompute Spark resource usage optimization

Introduction to This article mainly explains MaxCompute Spark resource optimization. The purpose is to guide users to better optimize the use of Spark job resources, maximize the use of resources, and reduce costs while ensuring the normal running of Spark tasks.

Author of this article: Wu Shujie, Alibaba Cloud Intelligent Development Engineer

1 Overview

This article mainly explains MaxCompute Spark resource tuning. The purpose is to guide users to better optimize the use of Spark job resources, maximize the use of resources, and reduce costs under the premise of ensuring the normal running of Spark tasks.

2. Sensor

• Sensor provides a visual way to monitor the running Spark process. Each worker (Executor) and master (Driver) has its own status monitoring diagram, which can be found in Logview, as shown in the following figure:

• After opening the Sensor, you can see that the following figure provides the CPU and memory usage of the Driver/Executor during its life cycle:
• • cpu\_plan/mem\_plan (blue line) represents the amount of CPU and memory plan requested by the user
  • Users can intuitively see the CPU utilization during task running from the cpu\_usage graph
  • mem\_usage represents the memory usage during task running, which is the sum of mem\_rss and page cache, see below for details

• Memory Metrics
• • mem\_rss represents the resident memory occupied by the process. This part of the memory is the actual memory used by the Spark task to run. It usually requires user attention. If the memory exceeds the amount of memory requested by the user, OOM may occur, leading to Driver /Executor process terminated. In addition, the curve can also be used to guide users to optimize memory. If the actual usage is far less than the amount of user applications, memory applications can be reduced, resources can be maximized, and costs can be reduced.
  • mem\_cache (page\_cache) is used to cache the data in the disk to the memory, thereby reducing disk I/O operations. It is usually managed by the system. If the physical machine has sufficient memory, then mem\_cache may be used a lot. You don't need to care about the allocation and recovery of the memory.

3. Resource parameter tuning

（1）Executor Cores

• Related parameters: spark.executor.cores
• • The number of cores of each Executor, that is, the number of tasks that can run simultaneously in each Executor
  • The maximum parallelism of Spark tasks is num-executors * executor-cores
  • When a Spark task is executed, a CPU core can only execute at most one task at a time. If the number of CPU cores is sufficient, generally speaking, these tasks can be executed relatively quickly and efficiently. Also note that the memory of each Executor is shared by multiple Tasks. If a single Executor has too many cores and too little memory, OOM may also occur.

（2）Executor Num

• Related parameters: spark.executor.instances
• • This parameter is used to set the total number of Executor processes used by the Spark job to execute
  • Generally, users can decide how many Executors they need to apply for based on the complexity of the task.
  • In addition, it should be noted that if the Executor disk space is insufficient, or some Executor OOM problems occur, you can reduce the number of cores of a single Executor and increase the number of instances of Executor to ensure that the overall parallelism of the task remains unchanged, while reducing the risk of task failure.

（3）Executor Memory

• Related parameters: spark.executor.memory
• • This parameter is used to set the memory of each Executor process. The size of Executor memory often directly determines the performance of Spark jobs, and JVM OOM is more common in Executor.

• Related parameter 2: spark.executor.memoryOverhead
• • Set the off-heap memory for the Executor, which is mainly used for the overhead of JVM itself, strings, NIO Buffer, etc. Note that this part of memory is not used for calculations, and user code and spark cannot be directly operated.
  • If this value is not set, the default is spark.executor.memory * 0.10, and the minimum is 384 MB
• The manifestations of Executor insufficient memory:
• • Cannot allocate memory appears in the Executor log (Logview->a Worker->StdErr)

• • It appears in the first line of Logview result at the end of the task: The job has been killed by "OOM Killer", please check your job's memory usage.
  • The memory usage is found to be very high in the Sensor
  • Java.lang.OutOfMemoryError: Java heap space appears in the Executor log
  • GC overhead limit exceeded appears in the Executor log
  • Frequent GC information found in Spark UI
  • The indirect manifestation of OOM may appear: some Executors have errors such as No route to host: workerd********* / Could not find CoarseGrainedScheduler
• Possible causes and solutions:
• • Limit the parallelism of executors and reduce the cores: multiple tasks running at the same time will share the memory of an Executor, which reduces the memory that can be used by a single task. Lowering the parallelism can alleviate the memory pressure and increase the memory of a single Executor.
  • Increase the number of partitions and reduce the load of each executor
  • Consider the problem of data skew, because the data skew leads to insufficient memory for one task, and enough memory for other tasks
  • If the above-mentioned Cannot allocate memory or The job has been killed by "OOM Killer", please check your job's memory usage occurs, this situation is usually due to insufficient system memory, and some off-heap memory can be appropriately added to alleviate the memory. Pressure, usually setting spark.executor.memoryOverhead to 1g/2g is enough

（4）Driver Cores

• Related parameters spark.driver.cores
• • Usually Driver Cores do not need to be too large, but if the tasks are more complex (such as too many Stages and Tasks) or too many Executors (Driver needs to communicate with each Executor and maintain a heartbeat), the CPU utilization rate in the Sensor is very high. , Then you may need to increase the Driver Cores appropriately
  • Also note that when running Spark tasks in Yarn-Cluster mode, you cannot directly set the driver resource configuration (core/memory) in the code, because this parameter is required when the JVM starts, so you need to pass the --driver-memory command line option Or set it in the spark-defaults.conf file/Dataworks configuration item.

（5）Driver Memory

• Related parameter 1: spark.driver.memory
• • Set the heap memory for the driver, similar to the executor
• Related parameter 2: spark.driver.maxResultSize
• • Represents the limit of the total size of the results of each Spark action (for example, collect). The default is 1g. If the total size exceeds this limit, the job will be aborted. If the value is high, OOM may occur in the Driver. Therefore, the user needs to set an appropriate value according to the actual situation of the job.
• Related parameter 3: spark.driver.memoryOverhead
• • Set the off-heap memory for the driver, similar to the executor

• The memory of the driver usually does not need to be too large. If the driver has insufficient memory, it is usually because the driver has collected too much data. If you need to use the collect operator to pull all the RDD data to the driver for processing, you must ensure that the driver is The memory is large enough.
• Manifestations:
• • The Spark application becomes unresponsive or stops directly
  • The Driver OutOfMemory error was found in the Driver log (Logview->Master->StdErr)
  • Frequent GC information found in Spark UI
  • The memory usage is found to be very high in the Sensor
  • Cannot allocate memory appears in the driver log
• Possible causes and solutions:
• • The code may use the collect operation to collect an overly large data set to the Driver node
  • Create an oversized array in the code, or load an oversized data set to the Driver process summary
  • SparkContext and DAGScheduler are all running on the Driver side. The stage segmentation corresponding to rdd also runs on the Driver side. If the program written by the user has too many steps and splits too many stages, this part of the information consumes the memory of the Driver. At this time, you need to increase the size of the Driver. RAM. Sometimes if there are too many stages, there will even be a stack overflow on the Driver side

(6) Local disk space

• Related parameters: spark.hadoop.odps.cupid.disk.driver.device\_size:
• • This parameter represents the amount of disk space requested for a single Driver or Executor, the default value is 20g, and the maximum support is 100g
  • Shuffle data and BlockManager overflow data are stored on disk
• The manifestations of insufficient disk space:
• • No space left on device error was found in the log of Executor/Driver
• solution:
• • The easiest way is to directly increase more disk space and increase spark.hadoop.odps.cupid.disk.driver.device\_size
  • If the error still occurs after increasing to 100g, it may be due to data skew, the data is concentrated in some blocks during the shuffle or cache process, or the amount of shuffle data of a single Executor is indeed too large, you can try:
• • • Repartition the data to solve the problem of data skew
    • Reduce the task concurrency of a single Executor
    • Reduce table reading concurrent spark.hadoop.odps.input.split.size
    • Increase the number of executors spark.executor.instances
• requires attention:
• • Also because the disk needs to be mounted before the JVM starts, this parameter must be configured in the spark-defaults.conf file or the configuration item of dataworks, and cannot be configured in the user code
  • In addition, it should be noted that the unit of this parameter is g, g cannot be omitted
  • In many cases, because the user configuration location is wrong or the unit g is not included, the parameter does not actually take effect, and the task still fails to run.

4. Summary

The above mainly introduces the problem of insufficient resources that may be encountered during the use of MaxCompute Spark and the corresponding solutions. In order to maximize the use of resources, it is first recommended to apply for a single worker resource at a ratio of 1:4, that is, 1 core: 4 gb memory, if OOM occurs, you need to check the log and Sensor to locate the problem initially, and then perform corresponding optimization and resource adjustments. It is not recommended to set too many cores for a single Executor. Generally, a single Executor is relatively safe in 2-8 cores. If it exceeds 8, it is recommended to increase the number of instances. Appropriately increasing off-heap memory (reserving some memory resources for the system) is also a common tuning method, which can usually solve many OOM problems in practice. Finally, users can refer to the official document https://spark.apache.org/docs/2.4.5/tuning.html , which contains more memory tuning techniques, such as gc optimization, data serialization, etc.

5.MaxCompute Spark related articles

MaxCompute Spark usage and FAQ

How does Spark On MaxCompute access Phonix data

How to use MaxCompute Spark to read and write

Spark runs in

Maxcompute Spark job management and control tool—Cupid Console

Comparative analysis of MaxCompute Spark and Spark SQL and precautions for use

Copyright Statement: content of this article is contributed spontaneously by Alibaba Cloud real-name registered users. The copyright belongs to the original author. The Alibaba Cloud Developer Community does not own its copyright, and does not assume corresponding legal responsibilities. For specific rules, please refer to the "Alibaba Cloud Developer Community User Service Agreement" and the "Alibaba Cloud Developer Community Intellectual Property Protection Guidelines". If you find suspected plagiarism in this community, fill in the infringement complaint form to report it. Once verified, the community will immediately delete the suspected infringing content.