Skip to content
Snippets Groups Projects

nexus-cern-operator

Helm operator to deploy Nexus3 OSS instances in CERN environment. Provides the Nexus application template for the OKD4 app-catalogue.

How to deploy an instance of Nexus

Nexus instances use OpenShift Persistent Volumes with automatic backup for artifact storage.

  1. Create (if you don't have one already) an e-group whose members will be granted admin privileges on the new instance. Make sure that the creator of the instance is member of that group! Note that only direct members of the group will be taken into account (nested groups are NOT currently supported).
  2. Create a web site from the Web Services portal using type "Application from a catalogue".
  3. From the new site's management page in the Web Services portal, follow the link to "Access project through OKD console".
  4. Click Add, select Operator backed under Developer catlog, select Nexus.
  5. Set required values: hostname, storage capacity (this can be extended later), initial admin e-group from step 1
  6. Provisioning takes a few minutes

The configured e-group has full admin permissions on the provisioned Nexus instance. The default configuration is explained below. See Nexus3 help pages to configure Nexus and create artifact repositories.

NB: it is only possible to deploy a single instance of Nexus per project. A second instance would fail provisioning.

Service Level for Nexus data

Nexus instances use OpenShift Persistent Volumes with automatic backup for artifact storage.

Protection against accidental data deletion or modification

OpenShift Persistent Volume includes a backup policy, this means that, even if the Nexus instance gets deleted, the information will persist for 30 more days. See backup policy.

Nexus database

Nexus uses an OrientDB database to store configuration. This database is backed up and saved in the instance's Persistent Volumes every evening. The database backup files are then backed up as regular files during automatic backup (together with Nexus artifact files).

Security updates

Security vulnerabilities are announced in https://support.sonatype.com/hc/en-us/sections/203012668-Security-Advisories (follow the article for notifications).

Security updates will be deployed centrally and restart all existing instances, with a short downtime (typically a few minutes). Deployment of security updates will be announced on the IT SSB.

Initial configuration

Nexus instances are pre-configured for the CERN environment.

Authentication

A SSO proxy is in charge of performing user authentication, but takes no authorization decision. The following authentication methods are implemented:

  • for the Nexus web interface and API: SSO authentication with Keycloak. No anonymous access is allowed. For API access in scripts, use auth-get-sso-token
  • for repository access:
    • anonymous access if explicitly enabled in Nexus configuration
    • basic authentication with local Nexus users
    • certain repository types use token authentication instead

NB: due to the design of the Nexus web interface, it is not possible to re-authenticate automatically with SSO when the Nexus session expires. This means that once the user session expires, Nexus will complain that the user is not authenticated anymore and users need to refresh the page (typically pressing F5 or the reload button) in their web browser to re-authenticate. Nexus sessions are configured to expire after 4h by default but this can be adjusted as necessary.

Local users

CERN accounts are used to access the Nexus web UI with CERN SSO.

For automated processed such as CI/CD pipelines or to use CLI tools like maven or curl, which don't use SSO but basic HTTP authentication, create local Nexus users to upload (or download if anonymous access is disabled) artifacts.

You have to login to Nexus with a user who has admin rights. Access the Nexus configuration settings by clicking the cog icon (top left corner between the logo and the component search field). In the left administration panel locate the "Users" element it is inside the Security section. Once the form loads click on the "create local user button". Fill out the required form and click on the "create local user" button.

Using CERN accounts rather than local users

By default CERN accounts can only be used to access the web UI with CERN SSO.

It is possible to use CERN accounts to upload or download artifacts using basic authentication, typically when using CLI tools like maven or curl. This requires additional configuration to verify CERN username/passwords against LDAP servers. Nexus requires a CERN Service Account for such authenticated LDAP accesss.

  1. Login to Nexus with a user who has admin rights.
  2. Navigate to Administration -> Security -> SSL Certificates
  3. Click Load certificate -> Paste PEM
  4. Download the CERN Root Certification Authority 2 certificate, convert it to PEM format (e.g. curl --silent 'https://cafiles.cern.ch/cafiles/certificates/CERN%20Root%20Certification%20Authority%202.crt' | openssl x509 -inform DER -outform PEM), copy/paste the PEM contents and click Add certificate to Truststore
  5. Navigate to Administration -> Security -> LDAP
  6. Edit the default CERN xLDAP (anonymous LDAP) connection. Do not create a new one as we want to keep the existing User and Group configuration.
  7. Change LDAP settings as indicated below
  8. Click Verify connection to ensure settings are correct

LDAP Settings for authenticated LDAP:

  • Name: CERN Active Directory
  • LDAP server address: ldaps://cerndc.cern.ch:636
  • Use the Nexus Trustore: yes
  • Search base DN: dc=cern,dc=ch (no change)
  • Authentication method: Simple authentication
  • Username or DN: CN=<your service account>,OU=users,OU=Organic Units,DC=cern,DC=ch
  • Change password: set your service account password

NB: local users can still be used next to CERN accounts.

Authorization

Authorization decisions take place in Nexus via roles. E-groups can be used by Mapping External Groups to Nexus Roles.

E-groups are available under the preconfigured "CERN xLDAP" external group provider.

Important note: only direct e-group members are taken into account. Members of nested e-groups (groups included in a group) are currently ignored.

Scripting is enabled

In order to manage the Nexus configuration for integration with the CERN environment, it is necessary for the Nexus scripting API to be enabled. This results in a warning in the Nexus console's System Status:

Scripting: Enabling and running scripts is not recommended as this bypasses security checks and can cause your Nexus instance to be vulnerable to existing and future attacks. We recommend using alternate ways to automate the configuration of this instance where possible.

As of April 2022 there is no known way to achieve the required configuration without scripting enabled, and it is not possible to disable scripting in instances provisioned by this operator.

Design considerations

Nexus instances are deployed and managed by a Helm operator deploying the nexus-cern Helm chart.

The nexus-cern Helm chart deploys the following components:

NB: no attempt is being made for the chart to support deploying multiple instances in the same namespace.

Nexus administrator cookbook

Invoke API

For general API access, use auth-get-sso-token for authentication with a CERN account.

It is also possible to use the OpenShift CLI to bypass SSO authentication and use Nexus's local admin account. E.g. this runs one of the groovy scripts installed by this project to delete a repo with name central:

# forward a local port to nexus, bypassing SSO (process runs in the background, remember to kill it when done)
oc port-forward deploy/nexus-repository-manager 8081 &
adminpwd=$(oc get secret/nexus-admin-password -o go-template='{{index .data "nexusAdminPassword"}}' | base64 -d)
curl http://admin:${adminpwd}@localhost:8081/service/rest/v1/script/delete_repo/run -X POST -v -d '{"name": "central"}' -H 'Content-Type: text/plain'

Slow Maven artifact downloads

It can take long for Maven to download all artifact from Nexus for each build. When using GitLab CI, consider caching Maven artifacts: https://stackoverflow.com/questions/37785154/how-to-enable-maven-artifact-caching-for-gitlab-ci-runner

Restore previous version of Nexus database

Follow this documentation, note that this procedure has to be executed by the admin of the Nexus instance. $data-dir in our case is set to /nexus-data/

Fix error "nexus data directory already in use: /nexus-data"

Problem when a running nexus pod abruptly dies. In this case a replica count of 1 makes the deployment to start a new pod but it then crash-loops because of some "lock" files. In this case we will have to remove two lock files and re-deploy the nexus instance.

Problem:

id: cannot find name for user ID <user_id>
Nexus data directory already in use: /nexus-data

Solution:

oc scale dc/nexus --replicas=0 -n <namespace>
oc debug dc/nexus -n <namespace>
cd /nexus-data/
# remove 2 lock files
rm lock tmp/nexus-ehcache/.lock
exit
oc scale dc/nexus --replicas=1 -n <namespace>

Increase PV (PermanentVolume) storage size

To increase the size of the volume, go to https://app-catalogue.cern.ch/, then in the Administrator tab, and under Operators, click on Installed Operators, then click on the Nexus tab. Go to the Custom Resource and edit the field spec.nexus-repository-manager.persistence.storageSize by setting a different value.

Pay attention to not set a value higher than the quota (by default 100 GiB). To know more about quotas refer to user docs.

Removing content from a S3 bucket with soft-deletion enabled

Only for Nexus sites created before April 2020 Soft-deletion preserves deleted objects, so special steps are required to wipe the contents of buckets where this was enabled. Follow the steps below:

Install awscli if needed and list the buckets:

yum install awscli
aws configure
# AWS Access Key ID [None]: <key here>
# AWS Secret Access Key [None]: <secret here>
# Default region name [None]:
# Default output format [None]:
aws --endpoint-url=http://s3.cern.ch s3api list-buckets

Execute the following command, have in mind that you will have to run the following script adding the bucket name (e.g. sh remove_bucket_content.sh my_bucket_to_remove):

#!/bin/bash

bucket=$1

set -e

echo "Removing all versions from $bucket"

versions=`aws --endpoint-url=http://s3.cern.ch s3api list-object-versions --bucket $bucket |jq '.Versions'`
markers=`aws --endpoint-url=http://s3.cern.ch s3api list-object-versions --bucket $bucket |jq '.DeleteMarkers'`

echo "removing files"
for version in $(echo "${versions}" | jq -r '.[]? | @base64'); do
    version=$(echo ${version} | base64 --decode)
    key=`echo $version | jq -r .Key`
    versionId=`echo $version | jq -r .VersionId `
    aws --endpoint-url=http://s3.cern.ch s3api delete-object --bucket $bucket --key '$key' --version-id='$versionId'
done

echo "removing delete markers"
for marker in $(echo "${markers}" | jq -r '.[]? | @base64'); do
    marker=$(echo ${marker} | base64 --decode)

    key=`echo $marker | jq -r .Key`
    versionId=`echo $marker | jq -r .VersionId `
    aws --endpoint-url=http://s3.cern.ch s3api delete-object --bucket $bucket --key '$key' --version-id='$versionId'
done
echo "Manually execution to FORCE THE DELETION of the bucket:"
echo "s3cmd --config=S3CONFIG.cfg rb s3://$bucket --force --recursive"

JAVA package deploy (to) and retrieve (from) Nexus

Once you have set up your repositories you can easily get your packages via CURL or in case you want to link your Nexus repository as a source repository for dependency packages you have to configure your JAVA compiler tool (for example Maven).

Please note that on Nexus 3 hasn't got indexes on the repository. This means from your development framework you can not list the uploaded packages. You have to know by heart the version number, group id and artifact id which you want to use.

Also the current Nexus 3 API is lacking of implemented features like give me the latest package using a curl command. Etc.

CURL download RELEASE packages

NB: if anonymous access is enabled then it is not necessary to provide username/password.

>curl -u $NEXUS_REPO_USER:$NEXUS_REPO_PW -o $DESTINATION_PACKAGE_NAME "$HOST_NAME/$REPOSITORY_NAME/$PACKAGE_GROUP_ID/$ARTIFACT_ID/$ARTIFACT_VERSION/$JAVA_PACAKAGE" -L
  • $NEXUS_REPO_USER: authentication user name (local Nexus user) for example MyUser
  • $NEXUS_REPO_PW: authentication pw (local Nexus user) for example MyPw
  • $HOST_NAME: https://mynexus.web.cern.ch/repository
  • $REPOSITORY_NAME: my-awesome-repository
  • $PACKAGE_GROUP_ID: slash separated "my/custom/group/id"
  • $DESTINATION_PACKAGE_NAME: for example my-package.jar (if you don't want to keep the version in the name)
  • $JAVA_PACKAGE: for example my-package-1.0.jar
  • $ARTIFACT_ID: for example my-package
  • $ARTIFACT_VERSION: for example 1.0
>curl -u MyUser:MyPw -o my-package.jar "https://mynexus.web.cern.ch/repository/my-awesome-repository/my/custom/group/id/my-package/1.0/my-package-1.0.jar" -L

CURL download SNAPSHOT packages

Nexus stores the SNAPSHOTS differently than the releases. These are the differences:

  • $JAVA_PACKAGE: for example my-package-1.0-20190321.164448-1.jar (there is a timestamp at the end !!!)
  • $ARTIFACT_VERSION: for example 1.0-SNAPSHOT
>curl -u MyUser:MyPw -o my-package.jar "https://mynexus.web.cern.ch/repository/my-awesome-repository/my/custom/group/id/my-package/1.0-SNAPSHOT/my-package-1.0-20190321.164448-1.jar" -L

Maven configuration

Maven build comes with a lots of good features and plugins. It gives a good standardized way to build your JAVA packages. With Nexus 3 we advise you to use the default maven deploy plugin. With that the maven snapshot and release package deployment can be easily separated after compilation. The compiler will make the separation itself based on the version number, in case the version number ends with "-SNAPSHOT" that will be a snapshot release (In case you follow the Maven version numbering). For authentication you can separate the credentials into a config file. This gives you the flexibility to separation sensitive information from the package itself.

Sample configuration files (pom.xml and settings.xml) can be founded under the maven-setting-templates folder. I advise you to use the snapshot profile for local development and leave the final package build for the CI/CD.

Maven gives the following nice ways for repositories:

  • create a global maven setting file if you want to do a maven (in CentOs usually located under /home/$YourUser/.m2/settings.xml) This is good for developers, in case you specify there the settings. You can execute the build from your framework and you don't have to link manually the settings via command line argument. Please note that you have to enable the right profile (snapshot or release like the "my-profile" in the activeProfiles)!

  • command line argument which contains the custom maven settings With -s you can link a settings.xml file and with -P you can enable the right profile (snapshot or release). This approach is good for CI/CD pipelines.

>mvn clean compile test package deploy -s ${MAVEN_SETTINGS_LOCATION} -P snapshot-build-profile

Operator development and debugging

Updating Nexus

In October 2023 Sonatype discontinued support for the the "Nexus3 Repository Manager" Helm chart. Sonatype now only provides a Helm chart for Nexus-HA, which requires a Nexus Pro license. However we still use the last version of the deprecated chart (v64.2, Feb 2024, vendored as a dependency in helm-charts/nexus-cern/charts/). This last version was unzipped into the nexus-cern chart's dependencies to allow further modifications.

To update the version of Nexus deployed by our operator, proceed as follows:

Deploying chart manually for testing

In OKD4 environment it is required to unset the security context so OKD4 can enforce its UID constraints. Setting the security context to null in values.yaml is not sufficient in removing it, we also need to set the Helm value to null via --set.

In operator deployment we use value overrides for this.

When developing/testing, use something like this:

helm upgrade --namespace test-nexus --install nexus1 helm-charts/nexus-cern --set cern-auth-proxy.route.hostname=nexus1.apptest.cern.ch --set initialNexusConfiguration.initialAdminEgroup=it-service-vcs --set nexus-repository-manager.nexus.securityContext=null

Recipe: how to deploy configuration changes to all instances

I.e. deploy changes to LDAP configuration, update backup script etc. to all instances. Another example is changing the Remote User header when migrating to OKD4 and new SSO proxy.

E.g. during the upgrade to 3.19, some changes in the Nexus object model made it necessary to modify the database backup script.

But the script needs to be updated on all existing instances.

The process would be as follows:

  • update the database backup rotation script in init-nexus-config-parameters-configmap.yaml.
  • deploy the new version of the operation. The updated configmap will be used for new instances, but nothing changes yet for existing instances.
  • for each instance:
    • run oc debug job/nexus-init-config to get an interactive container similar to the one used to pre-configure instances.
    • initialize environment:
      NEXUS_SCRIPT_API_URL="${NEXUS_BASE_URL}/service/rest/v1/script"
      NEXUS_AUTH="admin:$(oc get secret/nexus-admin-password -o json | jq -r .data.nexusAdminPassword | base64 -d)"
      function invoke_groovy_script() { ... # copy/paste from nexus-init-config.sh
    • run the relevant part of setup_database_backup from nexus-init-config.sh to re-create the Nexus task with the new script found in DATABASE_BACKUP_ROTATION_SCRIPT.

License

GPLv3

Copyright

Copyright 2022 CERN. This software is distributed under the terms of the GNU General Public Licence version 3 (GPL Version 3), copied verbatim in the file LICENSE.md. In applying this licence, CERN does not waive the privileges and immunities granted to it by virtue of its status as an Intergovernmental Organization or submit itself to any jurisdiction.

Groovy scripts: Copyright the nexus-oss Ansible playbook authors https://github.com/ansible-ThoTeam/nexus3-oss