spark参数设置

spark参数设置

SET spark.driver.memory=32G;
SET spark.executor.memory=16G;
SET spark.dynamicAllocation.minExecutors=100;

error

shuffle fetch wait time is:xxx ms greater than max wait 1800000 ms, throw exception

原因: shuffle超时. 作业认为shuffle卡住, 故报错退出

解决方案: 如果作业shuffle时间不超过1800s, 请直接重试. 否则可以调整spark.shuffle.maxFetchWaitTime参数, 提高阈值(默认1800s).

SET spark.shuffle.maxFetchWaitTime=7200s