resourcemanager.taskmanager-timeout 默认30000毫秒 taskmanager无任务运行就会被回收,下次提交任务时又会重新向yarn申请container启动taskmanager 耗时很大
resourcemanager.taskmanager-timeout 增加为86400000 保留一天
SlotManagerConfiguration.fromConfiguration(){ final Time taskManagerTimeout = Time.milliseconds( configuration.getLong(ResourceManagerOptions.TASK_MANAGER_TIMEOUT)); } public DeclarativeSlotManager(){ new TaskExecutorManager( slotManagerConfiguration.getDefaultWorkerResourceSpec(), slotManagerConfiguration.getNumSlotsPerWorker(), slotManagerConfiguration.getMaxSlotNum(), slotManagerConfiguration.isWaitResultConsumedBeforeRelease(), slotManagerConfiguration.getRedundantTaskManagerNum(), slotManagerConfiguration.getTaskManagerTimeout(), scheduledExecutor, executor, resourceActions); } TaskExecutorManager(){ taskManagerTimeoutsAndRedundancyCheck = scheduledExecutor.scheduleWithFixedDelay( () -> mainThreadExecutor.execute( this::checkTaskManagerTimeoutsAndRedundancy), 0L, taskManagerTimeout.toMilliseconds(), TimeUnit.MILLISECONDS); } taskManagerTimeout.toMilliseconds() 检查一次 checkTaskManagerTimeoutsAndRedundancy taskmanager空闲时就释放
欢迎分享,转载请注明来源:内存溢出
评论列表(0条)