Skip to content
Snippets Groups Projects
Commit ad4a13f1 authored by Alex Iribarren's avatar Alex Iribarren
Browse files

Merge branch 'onboard' into 'master' 

Update docs

See merge request !132
parents e531c645 4cf5e14e
No related branches found
No related tags found
1 merge request!132Update docs
Pipeline #10993722 passed
Stage: .pre
Stage: scan-policies
Stage: pages
Stage: deploy
Hide whitespace changes
Inline Side-by-side
docs/distributions/snapshots.md
+ 8
− 9
View file @ ad4a13f1
...@@ -17,11 +17,11 @@ applies to all of them, with specific changes for each one being listed separate ...@@ -17,11 +17,11 @@ applies to all of them, with specific changes for each one being listed separate
## Snapshot Creation ## Snapshot Creation
Snapshot creation is handled by various Nomad jobs ([CS8](https://gitlab.cern.ch/linuxsupport/cronjobs/stream8_snapshots/), [AlmaLinux](https://gitlab.cern.ch/linuxsupport/cronjobs/alma_snapshots/), [RHEL](https://gitlab.cern.ch/linuxsupport/cronjobs/rhel_snapshots/)). Snapshot creation is handled by various Nomad jobs ([AlmaLinux](https://gitlab.cern.ch/linuxsupport/cronjobs/alma_snapshots/), [RHEL](https://gitlab.cern.ch/linuxsupport/cronjobs/rhel_snapshots/)).
These jobs run automatically every day at 8am. They create daily snapshot directories These jobs run automatically every day at 8am. They create daily snapshot directories
in `{centos,alma,rhel}/*-snapshots/` and maintain the symlinks from `{8,9,s8}` (ie. prod), in `{alma,rhel}/*-snapshots/` and maintain the symlinks from `{8,9}` (ie. prod),
`{8,9,s8}-testing`, etc. to the appropriate daily snapshot. The `-testing` symlinks `{8,9}-testing`, etc. to the appropriate daily snapshot. The `-testing` symlinks
point to that day's snapshot (see note), and the production (ie. non -testing) point to that day's snapshot (see note), and the production (ie. non -testing)
ones point to the previous Wednesday's snapshot. Production symlinks are changed on ones point to the previous Wednesday's snapshot. Production symlinks are changed on
Wednesdays and ensure packages spend at least 7 days and at most 13 days in testing. Wednesdays and ensure packages spend at least 7 days and at most 13 days in testing.
...@@ -71,7 +71,7 @@ source RPM for the new upstream version (which you can find on our mirror) and c ...@@ -71,7 +71,7 @@ source RPM for the new upstream version (which you can find on our mirror) and c
version. Copy the changes from the upstream package into ours, adapting repository paths, etc. version. Copy the changes from the upstream package into ours, adapting repository paths, etc.
You can check previous commits to see examples. You can check previous commits to see examples.
You can also use [autopatch script](https://gitlab.cern.ch/morrice/autopatch) that will suggest the changes that you need to apply. You can also use [autopatch script](https://gitlab.cern.ch/morrice/autopatch) that will suggest the changes that you need to apply.
Once the new version of a release package is built and tagged as stable, it will be picked up Once the new version of a release package is built and tagged as stable, it will be picked up
the next time the Nomad job runs. If you were proactive enough and did this the day you got the next time the Nomad job runs. If you were proactive enough and did this the day you got
...@@ -106,7 +106,7 @@ The last command (`git push origin release8.9-1`) will provide a URL that can be ...@@ -106,7 +106,7 @@ The last command (`git push origin release8.9-1`) will provide a URL that can be
9. Re-launch the snapshot, in this case for `alma8`, by clicking on `Force launch` at [`https://lxsoftadm.cern.ch:4646/ui/jobs/prod_alma8_snapshots@default?desc=true&sort=submitTime`](https://lxsoftadm.cern.ch:4646/ui/jobs/prod_alma8_snapshots@default?desc=true&sort=submitTime) 9. Re-launch the snapshot, in this case for `alma8`, by clicking on `Force launch` at [`https://lxsoftadm.cern.ch:4646/ui/jobs/prod_alma8_snapshots@default?desc=true&sort=submitTime`](https://lxsoftadm.cern.ch:4646/ui/jobs/prod_alma8_snapshots@default?desc=true&sort=submitTime)
### Highlight packages ### Highlight packages
The highlight just means that it will be presented more prominently in the subject of the email. The highlight just means that it will be presented more prominently in the subject of the email.
The only thing special about kernels is that there is machinery to automatically build the kmod-openafs module. The only thing special about kernels is that there is machinery to automatically build the kmod-openafs module.
- [linux loves afs](https://mattermost.web.cern.ch/it-dep/channels/linux-afs) boot will send a message about the kmod - [linux loves afs](https://mattermost.web.cern.ch/it-dep/channels/linux-afs) boot will send a message about the kmod
- The test email will mention the kernel and kmod - The test email will mention the kernel and kmod
...@@ -120,7 +120,7 @@ You can see all the packages that changed between two dates using the `snapshotd ...@@ -120,7 +120,7 @@ You can see all the packages that changed between two dates using the `snapshotd
command, which is provided by [lxdist-build](https://gitlab.cern.ch/linuxsupport/lxdist-build/-/tree/master/bin): command, which is provided by [lxdist-build](https://gitlab.cern.ch/linuxsupport/lxdist-build/-/tree/master/bin):
```bash ```bash
[build@lxsoftadm01] snapshotdiff -d s8 -s 20200701 -e 20200709 [build@lxsoftadm01] snapshotdiff -d 8 -s 20200701 -e 20200709
Calculating diff between 20200701 and 20200709... Calculating diff between 20200701 and 20200709...
apache-sshd 2.5.1-1.el8 apache-sshd 2.5.1-1.el8
...@@ -159,8 +159,8 @@ The recommended method of updating the links is to ssh into an lxsoftadm machine ...@@ -159,8 +159,8 @@ The recommended method of updating the links is to ssh into an lxsoftadm machine
and run: and run:
```bash ```bash
# cd /mnt/data1/dist/cern/centos # cd /mnt/data1/dist/cern/alma
# echo "20200709" > .forcemove.s8 # echo "20240709" > .forcemove.8
``` ```
Once that file is created, rerun the cronjob in Nomad. It will see Once that file is created, rerun the cronjob in Nomad. It will see
...@@ -169,7 +169,6 @@ file. The job will also send the usual emails and update the web site. ...@@ -169,7 +169,6 @@ file. The job will also send the usual emails and update the web site.
| Distribution | Force-move file name | | Distribution | Force-move file name |
|--------------|---------------------------------------------| |--------------|---------------------------------------------|
| CS8 | `/mnt/data1/dist/cern/centos/.forcemove.s8` |
| ALMA8 | `/mnt/data1/dist/cern/alma/.forcemove.8` | | ALMA8 | `/mnt/data1/dist/cern/alma/.forcemove.8` |
| ALMA9 | `/mnt/data1/dist/cern/alma/.forcemove.9` | | ALMA9 | `/mnt/data1/dist/cern/alma/.forcemove.9` |
| RHEL8 | `/mnt/data1/dist/cern/rhel/.forcemove.8` | | RHEL8 | `/mnt/data1/dist/cern/rhel/.forcemove.8` |
......
docs/onboarding/index.md
+ 6
− 4
View file @ ad4a13f1
...@@ -9,11 +9,13 @@ This can be considered a good starting point for any new member of the team, thu ...@@ -9,11 +9,13 @@ This can be considered a good starting point for any new member of the team, thu
## Mattermost ## Mattermost
Most internal discussions take place in the ~lxsoft-admins mattermost channel. Most internal discussions take place in the [~lxsoft-admins](https://mattermost.web.cern.ch/it-dep/channels/lxsoft-admins) mattermost channel. We also have a channel [~lxsoft-alerts](https://mattermost.web.cern.ch/it-dep/channels/lxsoft-alerts) which is where automated system messages are recorded.
We also have a channel ~lxsoft-alerts which is where automated system messages are recorded. We also have a channel with 2nd level [~linux-team](https://mattermost.web.cern.ch/it-dep/channels/linux-team), [~linux-afs](https://mattermost.web.cern.ch/it-dep/channels/linux-afs) to receive alerts and discuss with the AFS team, [~linux-config-collaboration](https://mattermost.web.cern.ch/it-dep/channels/linux-config-collaboration) for discussions with the Config team and [~opensearch-for-linux](https://mattermost.web.cern.ch/it-dep/channels/opensearch-for-linux) for notifications related to our OpenSearch instance.
Both channels are private, thus you will need to be invited by an existing member. These channels are all private, thus you will need to be invited by an existing member.
You should also join the public [~linux](https://mattermost.web.cern.ch/it-dep/channels/linux) channel, where users sometimes report incidents.
## Rota ## Rota
...@@ -27,7 +29,7 @@ You won't be assigned rota weeks right away, but please go to the tool to [edit ...@@ -27,7 +29,7 @@ You won't be assigned rota weeks right away, but please go to the tool to [edit
[`cern-linuxsupport-access`](https://gitlab.cern.ch/linuxsupport/rpms/cern-linuxsupport-access) is a tool that can be used to enable/disable root access to the machine by CERN Linux Support personnel. [`cern-linuxsupport-access`](https://gitlab.cern.ch/linuxsupport/rpms/cern-linuxsupport-access) is a tool that can be used to enable/disable root access to the machine by CERN Linux Support personnel.
Hence, create a merge request with your public SSH key and your Kerberos principal. An example of what to modify exactly can be seen in this [commit](https://gitlab.cern.ch/linuxsupport/rpms/cern-linuxsupport-access/-/commit/61fdb57f1723ea5a2d38fb751dc50b988cdda445). The merge request will have to be accepted by a current member of the team, and the new package will have to be promoted to production. Hence, create a merge request with your public SSH key and your Kerberos principal. If people have left the linux team recently and their SSH key is still in this package, now would be a good time to clean that up as well. An example of what to modify exactly can be seen in this [commit](https://gitlab.cern.ch/linuxsupport/rpms/cern-linuxsupport-access/-/commit/61fdb57f1723ea5a2d38fb751dc50b988cdda445). The merge request will have to be accepted by a current member of the team, and the new package will have to be promoted to production.
### Update `cern-linuxsupport-access` on non puppet managed hosts ### Update `cern-linuxsupport-access` on non puppet managed hosts
......
docs/rota/tasks.md
+ 6
− 7
View file @ ad4a13f1
...@@ -4,7 +4,7 @@ The Linux team is in charge of providing a supported Linux distribution for CERN ...@@ -4,7 +4,7 @@ The Linux team is in charge of providing a supported Linux distribution for CERN
Starting with RHEL 7, CERN decided to join the CentOS community and help them rather than duplicating the work for rebuilding the same sources from Red Hat. CC7 (CERN CentOS 7) is just a simple respin of CentOS 7 with additional CERN repositories and staged updates. Starting with RHEL 7, CERN decided to join the CentOS community and help them rather than duplicating the work for rebuilding the same sources from Red Hat. CC7 (CERN CentOS 7) is just a simple respin of CentOS 7 with additional CERN repositories and staged updates.
Alma Linux 8/9, CS8 (CentOS Stream 8) follow upstream as closely as possible, with automated daily snapshots and minimal rebuilds. AlmaLinux and Red Hat follow upstream as closely as possible, with automated daily composes (snapshots) and minimal rebuilds.
Our main method of communication as a team is via the Mattermost channel [~lxsoft-admins](https://mattermost.web.cern.ch/it-dep/channels/lxsoft-admins). As a member of the team, **you are expected to follow that channel closely**. There's not a lot of small talk there, generally we're discussing how to solve a particular issue. Even if you're not involved in that specific issue, reading the messages helps us all learn from one another and stay up-to-date with the state of the services we run. Our main method of communication as a team is via the Mattermost channel [~lxsoft-admins](https://mattermost.web.cern.ch/it-dep/channels/lxsoft-admins). As a member of the team, **you are expected to follow that channel closely**. There's not a lot of small talk there, generally we're discussing how to solve a particular issue. Even if you're not involved in that specific issue, reading the messages helps us all learn from one another and stay up-to-date with the state of the services we run.
...@@ -16,14 +16,13 @@ You need to attend the incredibly useful [Ticket Review Meeting](https://indico. ...@@ -16,14 +16,13 @@ You need to attend the incredibly useful [Ticket Review Meeting](https://indico.
The main tasks as a release manager are : The main tasks as a release manager are :
- Keep an eye on the automation for [CC7](https://gitlab.cern.ch/linuxsupport/cronjobs/centos7_release), [CS8](https://gitlab.cern.ch/linuxsupport/cronjobs/stream8_snapshots) and [AlmaLinux](https://gitlab.cern.ch/linuxsupport/cronjobs/alma_snapshots). Every day, several emails will be sent to `lxsoft-admins` notifying you of the progress, and the full logs can also be seen in [Kibana](https://es-linux.cern.ch/?security_tenant=internal). - Keep an eye on the automation for [ELS7](https://gitlab.cern.ch/linuxsupport/cronjobs/els7_release), [AlmaLinux](https://gitlab.cern.ch/linuxsupport/cronjobs/alma_snapshots) and [Red Hat](https://gitlab.cern.ch/linuxsupport/cronjobs/rhel_snapshots). Every day, several emails will be sent to `lxsoft-admins` notifying you of the progress, and the full logs can also be seen in [Kibana](https://os-linux.cern.ch/?security_tenant=internal).
- Should the automation fail or stop for any reason (new packages that need to be rebuilt, etc.), your job is to make sure the situation is corrected and the automation can continue (either that day, by rerunning the cronjob, or the next). - Should the automation fail or stop for any reason (new packages that need to be rebuilt, etc.), your job is to make sure the situation is corrected and the automation can continue (either that day, by rerunning the cronjob, or the next).
- For CC7, production updates should happen on Thursday morning, and for CS8 and Alma8/9 they should happen on Wednesday morning. - For AlmaLinux and RHEL, production updates should happen on Wednesday morning and testing updates should happen every weekday.
- Manually, these updates may be stopped or done earlier depending on issues in the upstream packages or urgent security fixes. Always check with the rest of the team before doing so. - Manually, these updates may be stopped or done earlier depending on issues in the upstream packages or urgent security fixes. Always check with the rest of the team before doing so.
- Stay up-to-date with the latest Red Hat and CentOS updates. Notifications for particularly interesting packages [will be sent](https://gitlab.cern.ch/linuxsupport/cronjobs/centos_rss) to the Mattermost channel [~lxsoft-alerts](https://mattermost.web.cern.ch/it-dep/channels/lxsoft-alerts), but from time to time there will be other interesting updates that are not [on the list](https://gitlab.cern.ch/linuxsupport/cronjobs/centos_rss/-/blob/master/centos_rss/prod.packages.yml). - Stay up-to-date with the latest Red Hat and CentOS updates. Notifications for particularly interesting packages [will be sent](https://gitlab.cern.ch/linuxsupport/cronjobs/package_alerts) to the Mattermost channel [~lxsoft-alerts](https://mattermost.web.cern.ch/it-dep/channels/lxsoft-alerts), but from time to time there will be other interesting updates that are not [on the list](https://gitlab.cern.ch/linuxsupport/cronjobs/package_alerts/-/blob/master/package_alerts/prod.packages.yml).
- Notifications about ["dangerous packages"](https://linuxops.web.cern.ch/distributions/cs8/#-release-packages-the-dangerous-rpms) need to be acted upon as soon as possible. Not doing so incurrs a debt of croissants for the rest of the team. - Notifications about ["dangerous packages"](/distributions/snapshots/#-release-packages-the-dangerous-rpms) need to be acted upon as soon as possible. Not doing so incurs a debt of croissants for the rest of the team.
- This also includes keeping track of new Red Hat point releases and [publishing them](../support/redhat.md). - Once a month on the first of the month, new [Openstack](https://gitlab.cern.ch/linuxsupport/koji-image-build) and Docker images ([Alma9](https://gitlab.cern.ch/linuxsupport/alma9-base), [Alma8](https://gitlab.cern.ch/linuxsupport/alma8-base)) will be created by some Gitlab scheduled jobs. Make sure the process works and all the tests pass. Promote the resulting images to production.
- Once a month on the first of the month, new [Openstack](https://gitlab.cern.ch/linuxsupport/koji-image-build) and Docker images ([CC7](https://gitlab.cern.ch/linuxsupport/cc7-base), [CS8](https://gitlab.cern.ch/linuxsupport/cs8-base), [Alma9](https://gitlab.cern.ch/linuxsupport/alma9-base), [Alma8](https://gitlab.cern.ch/linuxsupport/alma8-base)) will be created a Gitlab scheduled jobs. Make sure the process works and all the tests pass. Promote the resulting images to production.
- All alerts need to be investigated and understood/fixed in a timely manner by the rota person. The rota person should also notify the rest of the team of the root cause and solution for the alerts. - All alerts need to be investigated and understood/fixed in a timely manner by the rota person. The rota person should also notify the rest of the team of the root cause and solution for the alerts.
## Services ## Services
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment