Skip to content
Snippets Groups Projects

Add locmap documentation

Merged Add locmap documentation
locmap into master
All threads resolved!
Merged Ben Morrice requested to merge locmap into master
All threads resolved!
Compare
and
2 files
+ 98
− 0
Compare changes
  • Side-by-side
  • Inline
Files
2
docs/locmap/locmap.md 0 → 100644
+ 97
− 0
# locmap
[`locmap`](https://gitlab.cern.ch/linuxsupport/rpms/locmap/locmap) ("**Lo**(cal) **C**(onfiguration) with **Ma**(sterless) **P**(uppet)") is a python application that facilitates the ability to manage "non datacentre" systems with CERN configuration.
Puppet modules are provided via RPMs by the linux team and can be found [here](https://gitlab.cern.ch/linuxsupport/rpms/locmap/). The naming prefix for puppet modules is "puppet-$modulename".
`locmap` is available on el8 and el9 through the [potd8el-stable](https://linuxsoft.cern.ch/internal/repos/potd8el-stable) or [potd9el-stable](https://linuxsoft.cern.ch/internal/repos/potd9el-stable) repositories (potd stands for "puppet on the desktop") and a [locmap-release](https://gitlab.cern.ch/linuxsupport/rpms/releases/locmap-release) RPM to configure these repositories exists in the CERN repo.
Previously, `locmap` was supported on CC7 and was available directly through the "CERN" repository
## Updating puppet modules
We use a mix of modules that are written by the linux team: termed 'custom', and modules that are provided by the configuration management team (it-puppet-module...) for which we call them 'upstream'.
For 'upstream' modules, some can be used "as is" where as others require some small patches to provide functionality on locmap systems
Note: moving away from 'custom' modules and using only 'upstream' modules is preferred and is an ongoing process
### Updating 'custom' modules
A standard git workflow is required here: clone, hack, test, commit, push, git tag, etc
An example of a 'custom' module is [puppet-zoom](https://gitlab.cern.ch/linuxsupport/rpms/locmap/puppet-zoom) which exists because there way to install/configure zoom in 'upstream' modules
Another example of a 'custom' module is [puppet-sudo](https://gitlab.cern.ch/linuxsupport/rpms/locmap/puppet-sudo) which exists due to legacy 'reasons' (This is a prime candidate to move to 'upstream' modules)
### Updating 'upstream' modules
Historically we needed to watch "Puppet CRM" to keep up with changes, this was annoying.
As some upstream modules require tweaks to support `locmap` functionality, manually rebasing and then reapplying said tweaks was also annoying
To ease the pain of managing these patches and to keep an eye on updates, a script has been written that checks for changes and applies patches automatically. This script is aptly named [locmap_updater](https://gitlab.cern.ch/linuxsupport/cronjobs/locmap_updater)
The good news is that we don't need to do much to update upstream modules. locmap_updater will create a MR if there is a change in an upstream module, and will email us about it. Essentially if the tests have passed, we just need to merge and git tag.
The `README.md` for each module that is ['managed'](https://gitlab.cern.ch/linuxsupport/cronjobs/locmap_updater/-/blob/master/prod.variables.sh?ref_type=heads#L4) by locmap_updater includes a note stating so
In the event that locmap_updater fails to apply a patch, we'll also be emailed. Some manual troubleshooting would be required in this case.
As an example, here is the [patch](https://gitlab.cern.ch/linuxsupport/cronjobs/locmap_updater/-/blob/master/patches/eosclient?ref_type=heads) that locmap_updater applies for 'it-puppet-module-eosclient' for it to be usable on `locmap` systems
Note: working with 'upstream' to add locmap functionality directly (thus not requiring a patch) is an ongoing process
### Functional tests
All `locmap` puppet modules have a stage in their gitlab pipeline to run a functional test. The stage is image-ci based, which spawns a fresh VM and executes the [`locmap-functional-tests.sh`](https://gitlab.cern.ch/linuxsupport/rpms/locmap/locmap-functional-tests) script
`locmap-functional-tests.sh` optionally accepts a list of modules to test. The script first installs `locmap`, then installs the latest version of locmap puppet modules (done via git clone / rpm-build), enables these modules, then runs `locmap --configure all`. Finally, some basic checks are performed to confirm/deny that `locmap`/module did the right thing.
## "Special" functionality
Some logic of configuration is not provided through `locmap`, but through other complimentary tooling. Here are some examples:
* [`lpadmincern`](https://gitlab.cern.ch/linuxsupport/rpms/lpadmincern) looks up ldap to retrieve printers, as such the associated [puppet-lpadmin](https://gitlab.cern.ch/linuxsupport/rpms/locmap/puppet-lpadmin/-/blob/master/src/code/manifests/init.pp) is pathetically basic.
* [`useraddcern`](https://gitlab.cern.ch/linuxsupport/rpms/useraddcern) has historically been used to add "CERN" local accounts with data from ldap with the home directory being "/afs". As per a request from the storage team, from el8 the default for home directories is now "/home"
* This change has had some friction within CERN communities and as such there is a `--directory` flag that uses the old "/afs" home directory behaviour
* `locmap` calls `useraddcern` when configuring the afs module, but not with the `--directory` flag. If users wish to have the "old" behaviour, they need to follow the information documented below in [First boot integration][#first-boot-integration]
## Locmap automatic update functionality
Even though `locmap` is masterless, there is functionality to provide the ability for `locmap` to reconfigure modules when there is a change (RPM update)
`locmap` has two command line flags `--enable_autoreconfigure` / `--disable_autoreconfigure` which control how `locmap` behaves when a `puppet-$module` rpm is updated. Both flags control the `autoreconfigure` key in `/etc/locmap/locmap.conf`
Each `locmap` `puppet-$module` rpm has an autoreconfigure %post script that will execute `locmap --configure $module` if `autoreconfigure` is `True` and if that module is 'enabled'.
This option is currently disabled by default (the introduction of el10 would be a good time to change the default)
## First boot integration
For systems that are installed with the group "Software Developer workstation (CERN Recommended Setup)" - the package [`cern-anaconda-addon`](https://gitlab.cern.ch/linuxsupport/rpms/cern-anaconda-addon/) is also installed
`cern-anaconda-addon` provides a first boot GUI screen that the user must pass through before the system is usable. One of the options in `cern-anaconda-addon` is to "customize my system for CERN". Should this option be selected, `cern-anaconda-addon` will install `locmap-release`, then install `locmap`, enable [default modules](https://gitlab.cern.ch/linuxsupport/rpms/cern-anaconda-addon/-/blob/master/src/cern_customizations/constants.py?ref_type=heads#L6) and finally run `locmap --configure`
Another option available to the user is "Enable automatic reconfiguration". If this option is selected, `locmap` is configured with the `--enable_autoreconfigure` [feature]
If the user selects "Enable automatic system updates", `cern-anaconda-addon` will install [`dnf-autoupdate`](https://gitlab.cern.ch/linuxsupport/rpms/yum-autoupdate)
There are two options (radio buttons) to configure how to define user home directories (local /home or /afs). If the user chooses to have a /afs home, `LOCMAP_HOMEDIRECTORY_LOCAL=False` is defined in `/etc/sysconfig/locmap-initialsetup`. `useraddcern` has logic to check for the presence of this key.
### Development
As `cern-anaconda-addon` usually runs on first boot via the anaconda `initial-setup-gui` implementation, development was historically difficult.
The `podman` based helper script [`adev.sh`](https://gitlab.cern.ch/linuxsupport/rpms/cern-anaconda-addon/-/tree/master/utils?ref_type=heads) has been written to assist development. As `initial-setup` is graphical and requires a functional `systemd` installation, running through `podman` requires a bit of juggling. To achieve this, `podman` runs as root and the containers spawned are `--privileged`.
Images created by the script have an entry point of `/sbin/init` and have both `rsyslog` and `sshd` running as enabled services. The script communicates to the pod via ssh with X forwarding for the graphical component. `rsyslog` ensures that `/var/log/messages` is populated which can be useful whilst debugging.
## Monitoring
Some opensearch dashboards are available to provide limited insights into how `locmap` is used at CERN:
[Locmap - Current Installations (EL8/EL9)](https://os-linux.cern.ch/dashboards/goto/2c769ff92a162ecab691989d089fbc70?security_tenant=internal)
This dashboard is generated from the number of unique hosts that are retrieving the repomd.xml file for the potd repositories, thus showing an up-to-date view of how many hosts at least have `locmap-release` installed.
[Locmap - Live data](https://os-linux.cern.ch/dashboards/goto/fa09d0a1862e32313ec293fe9d466022?security_tenant=internal)
This dashboard is generated by the number of unique hosts that hit "checkin" on potd paths. Such a call is made to linuxsoft every time `locmap` is invoked with `--configure`. We are abusing the log instrastrucutre a bit here, and this could be done better. See [here](https://gitlab.cern.ch/linuxsupport/rpms/locmap/locmap/-/blob/master/src/locmap/scripts/cli.py?ref_type=heads#L180) for more details.