【FAQ】HIVE分区表，字段类型变更后产生has an incompatible Parquet schema for column ‘table.field‘. Column type: xxxx

【FAQ】HIVE分区表，字段类型变更后产生has an incompatible Parquet schema for column ‘table.field‘. Column type: xxxx 异常

1、常见异常

StatementCallback; uncategorized SQLException for SQL [select * from `xxx` where time= '20190404']; SQL state [HY000]; error code [0]; File 'hdfs://xxx/xxx/time=20190404/xxxxxxxxxx' has an incompatible Parquet schema for column 'xxx.xxx'. Column type: STRING, Parquet schema: optional int64 xxx [i:5 d:1 r:0] ; nested exception is java.sql.SQLException: File hdfs://xxx/xxx/time=20190404/xxxxxxxxxx has an incompatible Parquet schema for column 'xxx.xxx'. Column type: STRING, Parquet schema: optional int64 xxx [j:9 x:6 b:1]

StatementCallback; uncategorized SQLException for SQL [select * from xxx where time= ‘20190404’]; SQL state [HY000]; error code [0]; File ‘hdfs://xxx/xxx/time=20190404/xxxxxxxxxx’ has an incompatible Parquet schema for column ‘xxx.xxx’. Column type: STRING, Parquet schema: optional int64 xxx [i:5 d:1 r:0] ; nested exception is java.sql.SQLException: File hdfs://xxx/xxx/time=20190404/xxxxxxxxxx has an incompatible Parquet schema for column ‘xxx.xxx’. Column type: STRING, Parquet schema: optional int64 xxx [j:9 x:6 b:1]

2、常见异常

java.lang.UnsupportedOperationException: Unimplemented type:LongType

java.lang.UnsupportedOperationException: Unimplemented type:LongType

异常根因即各个分区文件scheme属性不一致导致。多见于用户误 *** 作，变更字段属性时未考虑历史分区数据如何刷新。

解决方案

原则即重写异常分区文件。可以drop也可以overwrite，依据实际情况选择。

• 查分区

show PARTITIONS xxx;

20111107
20111108
20111109
20111110
20111111
20111112
20111113
20111114
20111115

• 验数据

select * from xxx  where p_date= 'xxx';

• 刷数据

insert overwrite xxx select * from xxx  where p_date= 'xxx';

• 查结构

show create table xxx;

• 验全表

select * from xxx;