org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to toAttribute on unresolve

org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to toAttribute on unresolve,第1张

org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to toAttribute on unresolve

执行形如下列的spark sql:

select xxx from TABLENAME WHERe x=1 having ConDITION = 1

报错:

org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to toAttribute on unresolved object, tree: ArrayBuffer(a).*
	at org.apache.spark.sql.catalyst.analysis.Star.toAttribute(unresolved.scala:245)
	at org.apache.spark.sql.catalyst.plans.logical.Project$$anonfun$output.apply(basicLogicalOperators.scala:52)
	at org.apache.spark.sql.catalyst.plans.logical.Project$$anonfun$output.apply(basicLogicalOperators.scala:52)
	at scala.collection.TraversableLike$$anonfun$map.apply(TraversableLike.scala:234)
	at scala.collection.TraversableLike$$anonfun$map.apply(TraversableLike.scala:234)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
	at scala.collection.immutable.List.map(List.scala:296)
	at org.apache.spark.sql.catalyst.plans.logical.Project.output(basicLogicalOperators.scala:52)
	at org.apache.spark.sql.hive.HiveAnalysis$$anonfun$apply.applyOrElse(HiveStrategies.scala:160)
	at org.apache.spark.sql.hive.HiveAnalysis$$anonfun$apply.applyOrElse(HiveStrategies.scala:148)

原因:
https://spark.apache.org/docs/2.4.0/sql-migration-guide-upgrade.html#upgrading-from-spark-sql-23-to-24

spark sql在2.3和2.4的处理逻辑不一样,
在2.3中如果没有GROUP BY,则HAVINg会当成WHERe进行处理
但2.4中就会报上述错误。


可参考的解决方案:
1.改sql语句,加上group by
2. 启动的时候设置spark.sql.legacy.parser.havingWithoutGroupByAsWhere 为true ,即 ./bin/spark-sql --conf spark.sql.legacy.parser.havingWithoutGroupByAsWhere =true

欢迎分享,转载请注明来源:内存溢出

原文地址: http://outofmemory.cn/zaji/5116540.html

(0)
打赏 微信扫一扫 微信扫一扫 支付宝扫一扫 支付宝扫一扫
上一篇 2022-11-17
下一篇 2022-11-17

发表评论

登录后才能评论

评论列表(0条)

保存