Skip to content
Snippets Groups Projects

[MONIT-2774] Update Apache Spark to version 3.4

Merged [MONIT-2774] Update Apache Spark to version 3.4
monit2774 into master
1 unresolved thread
Merged Guillermo Facundo Colunga requested to merge monit2774 into master
1 unresolved thread

This merge request updates the Apache Spark version to 3.4.0, for this:

  • States that the value of kafka idempotence to be false. As of kafka-clients version 3.2.0 this flag takes the value true, while in our brokers (version 2.8) this flag is set to false. In order to be able to write to the cluster, it is therefore necessary to explicitly indicate that it takes the value false.
  • Modifies the GitLab continuous integration configuration file to find the artifact in the path that is generated with the new version.
  • Modifies the Apache Spark version value to 3.4.0.
  • Updates the versions of the various dependencies to be compatible with Apache Spark 3.4.0.

Related JIRA Ticket: https://its.cern.ch/jira/browse/MONIT-2774

Edited by Guillermo Facundo Colunga

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
build.sbt
1 1 name := "monit-utilities"
2 2
3 version := "3.1"
3 version := "3.4"
4 4
5 scalaVersion := "2.11.11"
5 scalaVersion := "2.12.17"
6 6
7 val sparkVersion = "2.3.2"
7 val sparkVersion = "2.4.0"
  • Guillermo Facundo Colunga added 1 commit

    added 1 commit

    • 6b238126 - [MONIT-2774] Hotfix Apache Spark version 2.4.0 to 3.4.0

    Compare with previous version

  • Borja Garrido Bear approved this merge request

    approved this merge request

  • Guillermo Facundo Colunga mentioned in commit 8bf61a87

    mentioned in commit 8bf61a87

  • Guillermo Facundo Colunga merged

    merged

    • Please register or sign in to reply