Skip to content

NXCALS-3624 Filter out null values from VariableQuery

Grigorios Avgitidis requested to merge feature-NXCALS-3624-variable-query-filter-nulls into develop

#Merge request template @acc-logging-team - please have a look for approval.

Given the query:

        Dataset<Row> dataset = DataQuery.builder(sparkSession)
                .byVariables()
                .system("CMW")
                .startTime("2018-12-28 00:00:00.000").endTime("2018-12-29 00:00:00.000")
                .variable("TCL.4L5.B2:EXPACQ_CURR_ENERGY")
                .buildDataset();

before:

2020-03-23 16:12:12.129 [WARN ] [main] Utils - Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting 'spark.debug.maxToStringFields' in SparkEnv.conf.
+------------+----------------+-------------------+--------------------+
|nxcals_value|nxcals_entity_id|   nxcals_timestamp|nxcals_variable_name|
+------------+----------------+-------------------+--------------------+
|     7864200|           51986|1545955225280801000|TCL.4L5.B2:EXPACQ...|
|     7864200|           51986|1545955360284446000|TCL.4L5.B2:EXPACQ...|
|     7864200|           51986|1545955378284211000|TCL.4L5.B2:EXPACQ...|
|        null|           51986|1545955549288000000|TCL.4L5.B2:EXPACQ...|
|     7864200|           51986|1545955582290150000|TCL.4L5.B2:EXPACQ...|
+------------+----------------+-------------------+--------------------+

after:

2020-03-23 16:13:21.959 [WARN ] [main] Utils - Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting 'spark.debug.maxToStringFields' in SparkEnv.conf.
+------------+----------------+-------------------+--------------------+
|nxcals_value|nxcals_entity_id|   nxcals_timestamp|nxcals_variable_name|
+------------+----------------+-------------------+--------------------+
|     7864200|           51986|1545955225280801000|TCL.4L5.B2:EXPACQ...|
|     7864200|           51986|1545955360284446000|TCL.4L5.B2:EXPACQ...|
|     7864200|           51986|1545955378284211000|TCL.4L5.B2:EXPACQ...|
|     7864200|           51986|1545955582290150000|TCL.4L5.B2:EXPACQ...|
|     7864200|           51986|1545955733297009000|TCL.4L5.B2:EXPACQ...|
+------------+----------------+-------------------+--------------------+

Closes NXCALS-3624

Edited by Grigorios Avgitidis

Merge request reports