【FAQ】HIVE分区表,字段类型变更后产生has an incompatible Parquet schema for column ‘table.field‘. Column type: xxxx

【FAQ】HIVE分区表,字段类型变更后产生has an incompatible Parquet schema for column ‘table.field‘. Column type: xxxx,第1张

【FAQ】HIVE分区表,字段类型变更后产生has an incompatible Parquet schema for column ‘table.field‘. Column type: xxxx 异常

1、常见异常

StatementCallback; uncategorized SQLException for SQL [select * from `xxx` where time= '20190404']; SQL state [HY000]; error code [0]; File 'hdfs://xxx/xxx/time=20190404/xxxxxxxxxx' has an incompatible Parquet schema for column 'xxx.xxx'. Column type: STRING, Parquet schema: optional int64 xxx [i:5 d:1 r:0] ; nested exception is java.sql.SQLException: File hdfs://xxx/xxx/time=20190404/xxxxxxxxxx has an incompatible Parquet schema for column 'xxx.xxx'. Column type: STRING, Parquet schema: optional int64 xxx [j:9 x:6 b:1]

StatementCallback; uncategorized SQLException for SQL [select * from xxx where time= ‘20190404’]; SQL state [HY000]; error code [0]; File ‘hdfs://xxx/xxx/time=20190404/xxxxxxxxxx’ has an incompatible Parquet schema for column ‘xxx.xxx’. Column type: STRING, Parquet schema: optional int64 xxx [i:5 d:1 r:0] ; nested exception is java.sql.SQLException: File hdfs://xxx/xxx/time=20190404/xxxxxxxxxx has an incompatible Parquet schema for column ‘xxx.xxx’. Column type: STRING, Parquet schema: optional int64 xxx [j:9 x:6 b:1]

2、常见异常

java.lang.UnsupportedOperationException: Unimplemented type:LongType

java.lang.UnsupportedOperationException: Unimplemented type:LongType

异常根因即各个分区文件scheme属性不一致导致。多见于用户误 *** 作,变更字段属性时未考虑历史分区数据如何刷新。

解决方案

原则即重写异常分区文件。可以drop也可以overwrite,依据实际情况选择。

  • 查分区
show PARTITIONS xxx;

20111107
20111108
20111109
20111110
20111111
20111112
20111113
20111114
20111115

  • 验数据
select * from xxx  where p_date= 'xxx';
  • 刷数据
insert overwrite xxx select * from xxx  where p_date= 'xxx';
  • 查结构
show create table xxx;
  • 验全表
select * from xxx;

欢迎分享,转载请注明来源:内存溢出

原文地址: http://outofmemory.cn/zaji/5671953.html

(0)
打赏 微信扫一扫 微信扫一扫 支付宝扫一扫 支付宝扫一扫
上一篇 2022-12-17
下一篇 2022-12-16

发表评论

登录后才能评论

评论列表(0条)

保存