Flink yarn-session模式下延长taskmanager驻留时间

Flink yarn-session模式下延长taskmanager驻留时间,第1张

Flink yarn-session模式下延长taskmanager驻留时间

resourcemanager.taskmanager-timeout 默认30000毫秒 taskmanager无任务运行就会被回收,下次提交任务时又会重新向yarn申请container启动taskmanager 耗时很大

resourcemanager.taskmanager-timeout 增加为86400000 保留一天

SlotManagerConfiguration.fromConfiguration(){
        final Time taskManagerTimeout =
                Time.milliseconds(
                        configuration.getLong(ResourceManagerOptions.TASK_MANAGER_TIMEOUT));
}
public DeclarativeSlotManager(){
 new TaskExecutorManager(
                                slotManagerConfiguration.getDefaultWorkerResourceSpec(),
                                slotManagerConfiguration.getNumSlotsPerWorker(),
                                slotManagerConfiguration.getMaxSlotNum(),
                                slotManagerConfiguration.isWaitResultConsumedBeforeRelease(),
                                slotManagerConfiguration.getRedundantTaskManagerNum(),
                                slotManagerConfiguration.getTaskManagerTimeout(),
                                scheduledExecutor,
                                executor,
                                resourceActions);
}

TaskExecutorManager(){
        taskManagerTimeoutsAndRedundancyCheck =
                scheduledExecutor.scheduleWithFixedDelay(
                        () ->
                                mainThreadExecutor.execute(
                                        this::checkTaskManagerTimeoutsAndRedundancy),
                        0L,
                        taskManagerTimeout.toMilliseconds(),
                        TimeUnit.MILLISECONDS);
}
taskManagerTimeout.toMilliseconds() 检查一次 checkTaskManagerTimeoutsAndRedundancy  taskmanager空闲时就释放

欢迎分享,转载请注明来源:内存溢出

原文地址: https://outofmemory.cn/zaji/5679171.html

(0)
打赏 微信扫一扫 微信扫一扫 支付宝扫一扫 支付宝扫一扫
上一篇 2022-12-17
下一篇 2022-12-17

发表评论

登录后才能评论

评论列表(0条)

保存