建一个做扰运运行wordcount的类，先不纯梁管他什么意思，代码如下
[java] view plain copy
/**
* Project: hadoop
*
* File Created at 2012-5-21
* $Id$
*/
package seee.you.app
import java.io.IOException
import java.util.StringTokenizer
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.Path
import org.apache.hadoop.io.IntWritable
import org.apache.hadoop.io.LongWritable
import org.apache.hadoop.io.Text
import org.apache.hadoop.mapreduce.Job
import org.apache.hadoop.mapreduce.Mapper
import org.apache.hadoop.mapreduce.Reducer
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat
public class WordCount {
public static class TokenizerMapper extends Mapper<LongWritable, Text, Text, IntWritable>{
private final static IntWritable one = new IntWritable(1)
private Text word = new Text()
public void map(LongWritable key, Text value, Context context)
throws IOException, InterruptedException {
StringTokenizer itr = new StringTokenizer(value.toString())
while (itr.hasMoreTokens()) {
word.set(itr.nextToken())
context.write(word, one)
}
}
}
public static class IntSumReducer extends Reducer<Text, IntWritable, Text, IntWritable>{
private IntWritable result = new IntWritable()
public void reduce(Text key, Iterable<IntWritable>values, Context context)
throws IOException, InterruptedException {
int sum = 0
for (IntWritable val : values) {
sum += val.get()
}
result.set(sum)
context.write(key, result)
}
}
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration()
if (args.length != 2) {
System.err.println("Usage: wordcount ")
System.exit(2)
}
Job job = new Job(conf, "word count")
job.setJarByClass(WordCount.class)
job.setMapperClass(TokenizerMapper.class)
job.setReducerClass(IntSumReducer.class)
job.setMapOutputKeyClass(Text.class)
job.setMapOutputValueClass(IntWritable.class)
job.setOutputKeyClass(Text.class)
job.setOutputValueClass(IntWritable.class)
FileInputFormat.addInputPath(job, new Path(args[0]))
FileOutputFormat.setOutputPath(job, new Path(args[1]))
System.exit(job.waitForCompletion(true) ? 0 : 1)
}
}
这时候右键run on hadoop
这时候不幸的是，报错了，错误信息如下：
[java] view plain copy
12/05/23 19:38:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
12/05/23 19:38:51 ERROR security.UserGroupInformation: PriviledgedActionException as:yongkang.qiyk cause:java.io.IOException: Failed to set permissions of path: \tmp\hadoop-yongkang\mapred\staging\yongkang.qiyk-1840800210\.staging to 0700
Exception in thread "main" java.io.IOException: Failed to set permissions of path: \tmp\hadoop-yongkang\mapred\staging\yongkang.qiyk-1840800210\.staging to 0700
at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:682)
at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:655)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:116)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:856)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:850)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:850)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:500)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530)
at seee.you.app.WordCount.main(WordCount.java:80)
错误信息很明显了，at org.apache.hadoop.fs.FileUtil.checkReturnValue(FileUtil.java:682) 这一行的方法报错了
网上查到这是由于0.20.203.0以后的版本的权限认证引起的，只有去掉才行
修改hadoop源代码，去除权限认证，修改FileUtil.java的checkReturnValue方法，如下：
[java] view plain copy
private static void checkReturnValue(boolean rv, File p,
FsPermission permission
) throws IOException {
// if (!rv) {
// throw new IOException("Failed to set permissions of path: " + p +
// " to " +
// String.format("%04o", permission.toShort()))
// }
}
去掉这一行后，需要重新编译打包下，打包成功之后，可以将hadoop-core-1.0.2.jar拷贝到hadoop根目录下，eclipse中重新导入下即可（我用的这个1.0.2是从网上下载的修改好的，比较省事）
这时重新运行下实例，运行实例需要配置下arguments参数，我的配置如下：
run一下，结果如下，说明已经成功了
[java] view plain copy
WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
****hdfs://10.16.110.7:9000/user/yongkang/test-in
INFO input.FileInputFormat: Total input paths to process : 0
INFO mapred.JobClient: Running job: job_local_0001
INFO mapred.Task: Using ResourceCalculatorPlugin : null
INFO mapred.LocalJobRunner:
INFO mapred.Merger: Merging 0 sorted segments
INFO mapred.Merger: Down to the last merge-pass, with 0 segments left of total size: 0 bytes
INFO mapred.LocalJobRunner:
INFO mapred.Task: Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting
INFO mapred.LocalJobRunner:
INFO mapred.Task: Task attempt_local_0001_r_000000_0 is allowed to commit now
INFO output.FileOutputCommitter: Saved output of task 'attempt_local_0001_r_000000_0' to /user/yongkang/test-out6
INFO mapred.JobClient: map 0% reduce 0%
INFO mapred.LocalJobRunner: reduce >reduce
INFO mapred.Task: Task 'attempt_local_0001_r_000000_0' done.
INFO mapred.JobClient: map 0% reduce 100%
INFO mapred.JobClient: Job complete: job_local_0001
INFO mapred.JobClient: Counters: 10
INFO mapred.JobClient: File Output Format Counters
INFO mapred.JobClient: Bytes Written=0
INFO mapred.JobClient: FileSystemCounters
INFO mapred.JobClient: FILE_BYTES_READ=8604
INFO mapred.JobClient: FILE_BYTES_WRITTEN=51882
INFO mapred.JobClient: Map-Reduce Framework
INFO mapred.JobClient: Reduce input groups=0
INFO mapred.JobClient: Combine output records=0
INFO mapred.JobClient: Reduce shuffle bytes=0
INFO mapred.JobClient: Reduce output records=0
INFO mapred.JobClient: Spilled Records=0
INFO mapred.JobClient: Total committed heap usage (bytes)=5177344
INFO mapred.JobClient: Reduce input records=0
1.找到examples例子我们需要局搭尘找打这个例子的位置：首先需要找到你的hadoop文件夹，然后依照下面路径：
/hadoop/share/hadoop/mapreduce
第二步：
我们需要需要做一下运行需要的工作，比如输入输出路径，上传什么文件等。
1.先在HDFS创建几个数据目录：
1.hadoop fs -mkdir -p /data/wordcount
2.hadoop fs -mkdir -p /output/
2.目录/data/wordcount用来存放Hadoop自带的WordCount例子的数据文件，运行这个MapReduce任务的结果输出到/output/wordcount目录中。
首先新建文件inputWord：
1.vi /usr/inputWord
新建完毕，查看内容：
将本地文件上传到HDFS中：
可以查看上传后桐禅的文件情况，执行如下命令：
1.hadoop fs -ls /data/wordcount
可以看到上传到HDFS中的文件。
登录到Web控制台，访问链接可以看枝升到任务记录情况。
欢迎分享，转载请注明来源：内存溢出
微信扫一扫
支付宝扫一扫
评论列表（0条）