Saturday, June 25, 2016

Weird Spark bug?


1.5.0-cdh5.5.0

scala> df.filter("ad_market_id = 4 and event_date = '2016-05-23'").show
+----------+------------+
|event_date|ad_market_id|
+----------+------------+
+----------+------------+


scala> df.filter("ad_market_id = 4").filter("event_date = '2016-05-23'").show
+----------+------------+
|event_date|ad_market_id|
+----------+------------+
+----------+------------+


scala> df.filter("ad_market_id = 4").orderBy("event_date").filter("event_date = '2016-05-23'").show
+----------+------------+
|event_date|ad_market_id|
+----------+------------+
|2016-05-23|           4|
+----------+------------+

1 comment:

Zen by Ren said...

Fixed in later versions.