Skip to content

NXCALS-1849 Fix spark session properties parsing

#Merge request template @acc-logging-team - please have a look for approval.

This MR addresses an issue reported by Jiri Kral and it's related with the expected properties order that we provide via application properties files to our applications.

Currently, if one changes the spark submit deploy mode property on those property files, the changes won't be taken into account. This is because, our spark context properties class, expects the key as spark.properties.spark.submit.deployMode, but we are providing it wrongly as spark.submit.deployMode.

Moreover, this MR removes the redundant spark.sql.warehouse.dir from spark session builder, to get rid of some long lasting "TODOs".

Finally, by the introduction of SparkSessionModifier functional interface, we can effectively, modify an already constructed SparkSession, I've easily removed the duplicated spark configuration from the compaction project, with no change on behavior.

Closes NXCALS-1849

Merge request reports

Loading