ERROR TaskSchedulerImpl: Ignoring update with state FINISHED

ERROR TaskSchedulerImpl: Ignoring update with state FINISHED ERROR TaskSchedulerImpl: Ignoring update with state FINISHED

ERROR TaskSchedulerImpl: Ignoring update with state FINISHED for TID 870 because its task set is gone (this is likely the result of receiving duplicate task finished status updates) or its executor has been marked as failed

解决方案

加长executor的心跳时间

--confspark.network.timeout=10000000 
--conf spark.executor.heartbeatInterval=10000000 

参考

在https://mallikarjuna_g.gitbooks.io/spark/content/spark-taskschedulerimpl.html里有提到：If the TaskSetManager for tid could not be found (in taskIdToTaskSetManager registry), you should see the following ERROR message in the logs:

ERROR Ignoring update with state [state] for TID [tid] because its task set is gone (this is likely the result of receiving duplicate task finished status updates)