file:usrlocalhiveiotmp89de7dfe-8f26-4a6e-87b4-bacd16c2b2c8hive

file:/usr/local/hive/iotmp/89de7dfe-8f26-4a6e-87b4-bacd16c2b2c8/hive

1、发现问题

场景：当多个sql脚本任务同时在运行的时候经常报如下错误。

具体报错：FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. File file:/usr/local/hive/iotmp/89de7dfe-8f26-4a6e-87b4-bacd16c2b2c8/hive_2021-11-05_05-06-07_555_3392062193706637485-1/-local-10004/HashTable-Stage-2 does not exist

INFO - MapredLocal task succeeded INFO - Launching Job 1 out of 1 INFO - Number of reduce tasks is set to 0 since there's no reduce operator INFO - Cannot run job locally: Input Size (= 6505604400) is larger than hive.exec.mode.local.auto.inputbytes.max (= 134217728) INFO - java.io.FileNotFoundException: File file:/usr/local/hive/iotmp/89de7dfe-8f26-4a6e-87b4-bacd16c2b2c8/hive_2021-11-05_05-06-07_555_3392062193706637485-1/-local-10004/HashTable-Stage-2 does not exist INFO - at org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:429) INFO - at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1515) INFO - at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1555) INFO - at org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:574) INFO - at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:319) INFO - at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:151) INFO - at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199) INFO - at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100) INFO - at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2183) INFO - at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1839) INFO - at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1526) INFO - at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237) INFO - at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227) INFO - at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233) INFO - at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184) INFO - at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403) INFO - at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336) INFO - at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:474) INFO - at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:490) INFO - at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:793) INFO - at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759) INFO - at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686) INFO - at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) INFO - at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) INFO - at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) INFO - at java.lang.reflect.Method.invoke(Method.java:498) INFO - at org.apache.hadoop.util.RunJar.run(RunJar.java:221) INFO - at org.apache.hadoop.util.RunJar.main(RunJar.java:136) INFO - Job Submission failed with exception 'java.io.FileNotFoundException(File file:/usr/local/hive/iotmp/89de7dfe-8f26-4a6e-87b4-bacd16c2b2c8/hive_2021-11-05_05-06-07_555_3392062193706637485-1/-local-10004/HashTable-Stage-2 does not exist)' INFO - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. File file:/usr/local/hive/iotmp/89de7dfe-8f26-4a6e-87b4-bacd16c2b2c8/hive_2021-11-05_05-06-07_555_3392062193706637485-1/-local-10004/HashTable-Stage-2 does not exist

2、分析问题

        看报错可能是hive-site配置的有问题，查看hive配置文件hive-site.xml发现下面四个参数都含有/usr/local/hive/iotmp目录，怀疑是不是因为相同目录相互影响导致的。那么看下hive官网对这四个参数的定义描述。

hive.exec.local.scratchdir

hive.downloaded.resources.dir

hive.querylog.location

hive.server2.logging.operation.log.location

看下配置文件这四个参数的配置：

 
    hive.exec.local.scratchdir
    /usr/local/hive/iotmp
    Local scratch space for Hive jobs
  
  
    hive.downloaded.resources.dir
    /usr/local/hive/iotmp
    Temporary local directory for added resources in the remote file system.
  
  
    hive.querylog.location
    /usr/local/hive/iotmp
    Location of Hive run time structured log file
  
  
    hive.server2.logging.operation.log.location
    ${/usr/local/hive/iotmp}/${system:user.name}/operation_logs
    Top level directory where operation logs are stored if logging functionality is enabled
  

官网参数解释

官网地址：Configuration Properties - Apache Hive - Apache Software Foundation

hive.exec.scratchdir

hive.exec.scratchdir
Default Value: /tmp/${user.name} in Hive 0.2.0 through 0.8.0; /tmp/hive-${user.name} in Hive 0.8.1 through 0.14.0; or /tmp/hive in Hive 0.14.0 and later
Added In: Hive 0.2.0; default changed in 0.8.1 and in 0.14.0 with HIVE-6847  and HIVE-8143 
Scratch space for Hive jobs. This directory is used by Hive to store the plans for different map/reduce stages for the query as well as to stored the intermediate outputs of these stages.

hive.exec.local.scratchdir

hive.exec.local.scratchdir
Default Value: /tmp/${user.name}
Added In: Hive 0.10.0 with HIVE-1577 
Scratch space for Hive jobs when Hive runs in local mode.  Also see hive.exec.scratchdir .

hive.querylog.location

hive.querylog.location
Hive 实时查询日志所在的目录，如果该值为空，将不创建实时的查询日志。  
默认值：'/tmp/${user.name}' 

hive.downloaded.resources.dir

 
    hive.downloaded.resources.dir
    
    /user/local/hive/tmp/${hive.session.id}_resources
    Temporary local directory for added resources in the remote file system.
  

        从官网可以看到hive.exec.scratchdir、hive.exec.local.scratchdir、hive.querylog.location这三个参数的默认值是一样，并没有看到我想要的参数默认应该不一样的结果。则继续分析推理。先按照自己的想法把参数配置成不一样的值试一下。

3、解决问题

把这几个参数改成不同的目录(是否可行有待观察)

    hive.exec.scratchdir
    /tmp/hive
    HDFS root scratch dir for Hive jobs which gets created with write all (733) permission. For each connecting user, an HDFS scratch dir: ${hive.exec.scratchdir}/<username> is created, with ${hive.scratch.dir.permission}.


    hive.exec.local.scratchdir
    /usr/local/hive/iotmp


    hive.downloaded.resources.dir
    /usr/local/hive/iotmp/${hive.session.id}_resources


    hive.querylog.location
    /usr/local/hive/iotmp/root


    hive.server2.logging.operation.log.location
    /usr/local/hive/iotmp/root/operation_logs

改完观察了一个星期发现并没有再报错。

4、结论

在hive配置文件hive-site.xml中把这四个参数改成不同目录可以解决问题。

hive.exec.local.scratchdir

hive.downloaded.resources.dir

hive.querylog.location

hive.server2.logging.operation.log.location