Skip to content

NXCALS-5954 Add new StructTypes with non-nullable ArrayTypes to SparkRowToImmutableDataConverter.typeMap

Viktor Hyhyniak requested to merge fix-NXCALS-5954-backport-data-converter-from-hbase-when-null into develop

Merge request template

During Spark Row to ImmutableData conversion if no data was found for the field, we search for the required StructType in the SparkRowToImmutableDataConverter.typeMap. Though the StructTypes coming from HDFS and HBase are different: the latter contains non-nullable ArrayTypes in the structure. The current content of typeMap had only Structs with nullable ArrayTypes. This were resulting in the error being thrown during extraction from HBase if the field value was null for the record. This fix should add StructTypes with non-nullable ArrayTypes that are returned by HBase.

@acc-logging-team - please have a look for approval.

Please describe the merge request here.

Closes NXCALS-5954

Edited by Viktor Hyhyniak

Merge request reports