Skip to content

NXCALS-6237 Possibly_setting_PYSPARK_PYTHON_too_late_in_python

Jakub Wozniak requested to merge NXCALS-6237_Possibly_setting_PYSPARK_PYTHON_too_late_in_python into develop

Merge request template

COMPATIBILITY NOTE

Please make sure that this change is BACKWARDS COMPATIBLE and a new server will work correctly with clients using old APIs versions.

It may be that data sent in the new format will break the old clients.

In case of any potential incompatibility we have to inform our clients and schedule release & deployment with them.

@acc-logging-team - please have a look for approval.

This is removing the necessity of setting up the PYSPARK_PYTHON by the user. We just do it up-front in our code before calling any Spark imports. It seems to work fine (additional settings of that variable are removed from everywhere).

UPDATE: Second run of integration tests actually failed in local mode. Not sure why it passed the first time... I guess we need to figure out if we run in local/yarn mode before setting this...

Closes NXCALS-6237

Edited by Jakub Wozniak

Merge request reports