Skip to content

NXCALS-3567: HBaseDatasetCreator can now extract datasets from a collection of HBaseResources

#Merge request template @acc-logging-team - please have a look for approval.

Query

        Dataset<Row> dataset = DataQuery.builder(sparkSession)
                .byEntities().system("CMW")
                .startTime("2010-08-09 01:30:46.000").endTime("2020-08-09 01:30:45.000")
                .entity()
                .keyValue("device", "RPCAB.361.BR4.RXSK6L4")
                .keyValue("property", "SUB")
                .buildDataset();

        dataset.printSchema();

Result:

root
 |-- EVENT_GROUP: integer (nullable = true)
 |-- I_DIFF_MA: integer (nullable = true)
 |-- I_EARTH_PCNT: double (nullable = true)
 |-- I_ERR_MA: integer (nullable = true)
 |-- I_MEAS: float (nullable = true)
 |-- I_REF: float (nullable = true)
 |-- STATE_OP: string (nullable = true)
 |-- STATE_PC: string (nullable = true)
 |-- STATE_PLL: string (nullable = true)
 |-- STATE_VS: string (nullable = true)
 |-- ST_ADC_A: string (nullable = true)
 |-- ST_ADC_B: string (nullable = true)
 |-- ST_ADC_C: string (nullable = true)
 |-- ST_ADC_D: string (nullable = true)
 |-- ST_DCCT_A: string (nullable = true)
 |-- ST_DCCT_B: string (nullable = true)
 |-- ST_FAULTS: string (nullable = true)
 |-- ST_LATCHED: string (nullable = true)
 |-- ST_UNLATCHED: string (nullable = true)
 |-- ST_WARNINGS: string (nullable = true)
 |-- V_MEAS: float (nullable = true)
 |-- V_REF: float (nullable = true)
 |-- __record_timestamp__: long (nullable = true)
 |-- __record_version__: long (nullable = true)
 |-- acqStamp: long (nullable = true)
 |-- class: string (nullable = true)
 |-- cyclestamp: long (nullable = true)
 |-- device: string (nullable = true)
 |-- property: string (nullable = true)
 |-- selector: string (nullable = true)
 |-- nxcals_entity_id: long (nullable = true)

total values:8274053

Closes NXCALS-3567

Edited by Grigorios Avgitidis

Merge request reports