apache spark - DataFrame join optimization - Stack OverflowI am trying to effectively join two DataFrames, one of which is large and the second is a bit smaller. Is there a way to avoid all this shuffling? I cannot set autoBroadCastJoinThreshold, because it supports only Integers - and the table I am trying to broadcast is slightly bigger than integer number of bytes. Is there a way to force broadcast ignoring this variable?
Spark SQL - 3 common joins (Broadcast hash join, Shuffle Hash join, Sort merge join) explained
pyspark.sql module — PySpark 2.2.0 documentation
