2021-11-16Hadoop集群问题记录

2021-11-16Hadoop集群问题记录 项目场景：运行WordCount单词记数案例 问题描述：在运行wordcount单词计数程序时报错：错误: 找不到或无法加载主类 org.apache.hadoop.mapreduce.v2.app.MRAppMaster

2021-11-16 10:06:25,707 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.150.129:8032
2021-11-16 10:06:26,291 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1637026326069_0009
2021-11-16 10:06:26,536 INFO input.FileInputFormat: Total input files to process : 1
2021-11-16 10:06:26,794 INFO mapreduce.JobSubmitter: number of splits:1
2021-11-16 10:06:26,973 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1637026326069_0009
2021-11-16 10:06:26,974 INFO mapreduce.JobSubmitter: Executing with tokens: []
2021-11-16 10:06:27,165 INFO conf.Configuration: resource-types.xml not found
2021-11-16 10:06:27,166 INFO resource.ResourceUtils: Unable to find ‘resource-types.xml’.
2021-11-16 10:06:27,231 INFO impl.YarnClientImpl: Submitted application application_1637026326069_0009
2021-11-16 10:06:27,272 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1637026326069_0009/
2021-11-16 10:06:27,273 INFO mapreduce.Job: Running job: job_1637026326069_0009
2021-11-16 10:06:30,308 INFO mapreduce.Job: Job job_1637026326069_0009 running in uber mode : false
2021-11-16 10:06:30,308 INFO mapreduce.Job: map 0% reduce 0%
2021-11-16 10:06:30,320 INFO mapreduce.Job: Job job_1637026326069_0009 failed with state FAILED due to: Application application_1637026326069_0009 failed 2 times due to AM Container for appattempt_1637026326069_0009_000002 exited with exitCode: 1
Failing this attempt.Diagnostics: [2021-11-16 10:06:29.931]Exception from container-launch.
Container id: container_1637026326069_0009_02_000001
Exit code: 1

[2021-11-16 10:06:29.934]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
错误: 找不到或无法加载主类 org.apache.hadoop.mapreduce.v2.app.MRAppMaster

[2021-11-16 10:06:29.937]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
错误: 找不到或无法加载主类 org.apache.hadoop.mapreduce.v2.app.MRAppMaster

For more detailed output, check the application tracking page: http://master:8088/cluster/app/application_1637026326069_0009 Then click on links to logs of each attempt.
. Failing the application.
2021-11-16 10:06:30,337 INFO mapreduce.Job: Counters: 0
[root@master mapreduce]# hadoop jar hadoop-mapreduce-examples-3.1.4.jar wordcount /a.txt /output11
2021-11-16 10:07:17,595 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.150.129:8032
2021-11-16 10:07:18,235 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1637026326069_0010
2021-11-16 10:07:18,456 INFO input.FileInputFormat: Total input files to process : 1
2021-11-16 10:07:18,611 INFO mapreduce.JobSubmitter: number of splits:1
2021-11-16 10:07:18,867 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1637026326069_0010
2021-11-16 10:07:18,868 INFO mapreduce.JobSubmitter: Executing with tokens: []
2021-11-16 10:07:19,012 INFO conf.Configuration: resource-types.xml not found
2021-11-16 10:07:19,012 INFO resource.ResourceUtils: Unable to find ‘resource-types.xml’.
2021-11-16 10:07:19,076 INFO impl.YarnClientImpl: Submitted application application_1637026326069_0010
2021-11-16 10:07:19,120 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1637026326069_0010/
2021-11-16 10:07:19,120 INFO mapreduce.Job: Running job: job_1637026326069_0010
2021-11-16 10:07:20,145 INFO mapreduce.Job: Job job_1637026326069_0010 running in uber mode : false
2021-11-16 10:07:20,146 INFO mapreduce.Job: map 0% reduce 0%
2021-11-16 10:07:20,164 INFO mapreduce.Job: Job job_1637026326069_0010 failed with state FAILED due to: Application application_1637026326069_0010 failed 2 times due to AM Container for appattempt_1637026326069_0010_000002 exited with exitCode: 1
Failing this attempt.Diagnostics: [2021-11-16 10:07:19.816]Exception from container-launch.
Container id: container_1637026326069_0010_02_000001
Exit code: 1

原因分析：yarn-site.xml的yarn.application.classpath 的问题

1.可能是yarn-site.xml这个文件的配置问题，这个文件中没有配置 yarn.application.classpath 2.或者这个文件配置有误

---

解决方案：

cd /usr/local/hadoop-3.1.4/etc/hadoop
vim yarn-site.xml

如果没有，就添加 yarn.application.classpath 这个配置文件

yarn.application.classpath 
 
/usr/local/hadoop-3.1.4/etc/hadoop:/usr/local/hadoop-3.1.4/share/hadoop/common/lib/*:/usr/local/hadoop-3.1.4/share/hadoop/common/*:/usr/local/hadoop-3.1.4/share/hadoop/hdfs:/usr/local/hadoop-3.1.4/share/hadoop/hdfs/lib/*:/usr/local/hadoop-3.1.4/share/hadoop/hdfs/*:/usr/local/hadoop-3.1.4/share/hadoop/mapreduce/lib/*:/usr/local/hadoop-3.1.4/share/hadoop/mapreduce/*:/usr/local/hadoop-3.1.4/share/hadoop/yarn:/usr/local/hadoop-3.1.4/share/hadoop/yarn/lib/*:/usr/local/hadoop-3.1.4/share/hadoop/yarn/*
 

检查这个配置文件

如果有 就检查一下是否有误，比如hadoop-3.1.4中间有个“-”,还有就是需要注意，这一串代码是连起来的，并不是分开的
注意：集群的每个节点都需要修改
解决完成后就重启yarn或者hadoop集群
重启后再次运行相关MapReduce项目 问题已解决！

记录学习遇到的问题，安装Hadoop的方式不一样，解决方法可能也会因此改变，此方法仅供参考