Skip to content
Snippets Groups Projects
Commit bfdee7bf authored by Laurent Petre's avatar Laurent Petre
Browse files

Update the general infrastructure documentation

parent 9a26f132
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
docs/expert/infrastructure/index.md
+ 39
− 231
View file @ bfdee7bf
# Infrastructure # Infrastructure
## Service accounts ## Service account & e-groups
``` Whenever a user account is required, the common service account is used:
cmsgemonline
```
## EOS storage
An EOS project space has been created to serve web pages, host software
repositories, and perform various other custodial tasks. The root of the
project space is:
``` console
/eos/project/c/cmsgemonline/
```
## YUM repository
Software (firmware) artifacts are deployed as RPMs to an architecture
dependent `yum` repository, hosted in the `cmsgemdaq` website. The `yum`
repository and RPM packages are signed with a GPG key
(`0x5079CAAD2B12DD62`).
## Network topology
### Local network
For local network setup for 904 labs, and recommended for teststands,
several guidelines are provided. If you use the
:cpp`setupMachine.sh script<gemctp7user:setup_machine>`{.interpreted-text
role="func"}, you will be prompted for how to set up your local network,
but it will follow these basic principles:
- μTCA shelves and devices are on a `192.168.0.0/16` network.
- MCHs (NAT, not VadaTech) are assigned IP addresses of
`192.168.<shelf ID>.10`
!!! note
The IP address assignment may need to be done manually, if the
MCH was not previously configured by one of the GEM DAQ expert
team.
- AMC13s use their hard-coded IP address assignments based on the
device S/N
!!! note
Assignment via the `sysmgr` is possible, but has not been
reliably deployed
- CTP7s in the shelves assigned geographic IP addresses of the
form `192.168.<shelf ID>.<40+slot ID>`
- Host aliases are set up in `/etc/hosts` to allow for name
construction in software, e.g., `gem-shelf01-amc02`
- GLIBs (and other "emulator-like" devices) are managed through
`docker` container services, connected to the local μTCA network
via a `macvlan` interface.
#### Firewall rules
If using the firewall, several rules should be enabled. The commands
below assume a `systemd` based system running `firewalld` and configured
using `firewall-cmd`.
!!! tip
If your network policy allows it, you may also disable `firewalld`.
``` bash
systemctl firewalld disable
```
The rules below assume that **all** μTCA devices are connected on the
local. The rest of the rules depend on whether:
- The CTP7s/AMCs are configured to update software
`via a package manager<gemos-ctp7-smart-rule>`{.interpreted-text
role="ref"}
- The setup `is running RCMS<gemos-rcms-rule>`{.interpreted-text
role="ref"}
- The setup
`is running xDAQ applications<gemos-xdaq-rule>`{.interpreted-text
role="ref"}
- There is a
`single PC connected to the local network<gemos-control-hub-rule>`{.interpreted-text
role="ref"}, but other machines in the lab need access to the setup
hardware.
##### Local network device rule
1. Configure the interface (in this example, the `enp2s0` device) that
the communicates on the local μTCA network to belong to the
`trusted` zone
``` bash
sudo firewall-cmd --zone=trusted --add-interface=enp2s0 --permanent
sudo firewall-cmd --zone=trusted --add-interface=enp2s0
```
1. Reload the `firewalld` service to apply the rules
``` bash
sudo firewall-cmd --reload
```
##### CTP7 smart package manager rule
If the CTP7 is configured to pull packages from the `yum` repository * `cmsgemonline`
with the `smart` package manager, a connection from the CTP7 on the
local netowrk to the site hosting the `yum` repository needs to be made.
On the `sysmgr` PC execute the following commands (adapted for your
specific setup):
- Create port forward from the repository host to the local machine on Otherwise, memberhsips and authorizations are managed through various e-groups:
ports `10000+<host port>`
``` bash * `cms-gem-online`
sudo firewall-cmd --zone=public --add-forward-port=port=10080:proto=tcp:toport=80:toaddr=<repo host IP> --permanent * `cms-gem-online-admins`
sudo firewall-cmd --zone=public --add-forward-port=port=10443:proto=tcp:toport=443:toaddr=<repo host IP> --permanent * `cms-gem-online-b904int-users`
sudo firewall-cmd --zone=public --add-forward-port=port=10443:proto=tcp:toport=443:toaddr=<repo host IP> * `cms-gem-online-devs`
sudo firewall-cmd --zone=public --add-forward-port=port=10080:proto=tcp:toport=80:toaddr=<repo host IP> * `cms-gem-online-experts`
``` * `cms-gem-online-notifications`
- Create a port forward between the external facing NIC and the local While the usage of each of those groups should be clear based on their name, it is worth mentioning `cms-gem-online` as the main e-groups for general announcements and basic permissions.
network NIC
``` bash
sudo firewall-cmd --zone=trusted --add-forward-port=port=10080:proto=tcp:toport=10080:toaddr=<local network IP> --permanent
sudo firewall-cmd --zone=trusted --add-forward-port=port=10443:proto=tcp:toport=10443:toaddr=<local network IP> --permanent
sudo firewall-cmd --zone=trusted --add-forward-port=port=10443:proto=tcp:toport=10443:toaddr=<local network IP>
sudo firewall-cmd --zone=trusted --add-forward-port=port=10080:proto=tcp:toport=10080:toaddr=<local network IP>
```
- Allow the requests coming in on the local netowrk to pass through to ## EOS storage
the external facing network
``` bash
sudo firewall-cmd --zone=trusted --add-masquerade --permanent
sudo firewall-cmd --zone=trusted --add-masquerade
```
- Reload the `firewalld` service to apply the rules An EOS project space has been created to serve web pages, host software
repositories, and perform various other custodial tasks. The root of the
project space is:
``` bash
sudo firewall-cmd --reload
``` ```
/eos/project-c/cmsgemonline/
##### RCMS applications rule
The following will allow the RCMS interface to be visible if the machine
itself is reachable
``` bash
sudo firewall-cmd --zone=public --add-port=10000/tcp --permanent
sudo firewall-cmd --zone=public --add-port=10000/tcp
sudo firewall-cmd --reload
``` ```
##### xdaq applications rule Notable locations are:
The following will allow the hyperdaq pages of your `xdaq` applications
to be visible if the machin is reachable.
- `jobcontrol` application | Path | Notes |
|--------------------|--------------------------------------------------------------|
``` bash | `gem-csc-trigger` | Data shared with the `cms-gem-csc-trigger-taskforce` e-group |
sudo firewall-cmd --zone=public --add-port=39999/tcp --permanent | `public` | Data shared with the `cms-gem-online` e-group |
sudo firewall-cmd --zone=public --add-port=39999/tcp | `public/runs` | Configuration information for Global runs |
``` | `www` | Data shared with the `wwweos` account |
| `www/cmsgemonline` | [cmsgemonline](https://cmsgemonline.web.cern.ch/) website |
| `www/cmsgemos` | [cmsgemos](https://cmsgemos.web.cern.ch) website |
- "Standard" GEM applications
``` bash ## Network topology
sudo firewall-cmd --zone=public --add-port=20000-20600/tcp
sudo firewall-cmd --zone=public --add-port=20000-20600/tcp --permanent
```
- As always, reload `firewalld` to ensure the rules are updated
``` bash ### Local network
sudo firewall-cmd --reload
```
##### `control_hub` machine rule For local network setup for 904 labs, and recommended for teststands, several guidelines are provided.
The following will allow the requests to the `control_hub` (if - μTCA shelves and devices are on a `192.168.0.0/16` network.
configured) to succeed. - MCHs (NAT, not VadaTech) are assigned IP addresses of `192.168.<shelf ID>.10`
!!! note !!! note
This shouldn't be necessary, if all devices connected to the local The IP address assignment may need to be done manually, if the
network put that device in the "trusted" zone, as above. MCH was not previously configured by one of the GEM DAQ expert
team.
``` bash - AMC13s use their hard-coded IP address assignments based on the device S/N
sudo firewall-cmd --zone=trusted --add-port=10203/tcp --permanent
sudo firewall-cmd --zone=trusted --add-port=10203/tcp
```
In principle, if you have only a `control_hub` PC, which routes all
traffic to the local μTCA network, and all other DAQ machines are not
connected to the local network, you should enable the second set of
rules (i.e., in the `public` zone). A network setup such as this would
also require a significant number of other changes for CTP7 based,
direct connection systems, as you should route **all** traffic to the
local network through this PCs NIC. A setup such as this has not been
fully implemented at any test stands, but instructions could be prepared
if the need arises.
!!! warning !!! note
This hasn't been verified by the GEM DAQ team, but may be necessary in Assignment via the `sysmgr` is possible, but has not been reliably deployed
the case where you have one PC managing the local devices, and other
machines not on the local network, but can talk to the PC on its
standard NIC.
``` bash - CTP7s in the shelves assigned geographic IP addresses of the form `192.168.<shelf ID>.<40+slot ID>`
sudo firewall-cmd --zone=public --add-port=10203/tcp --permanent
sudo firewall-cmd --zone=public --add-port=10203/tcp
```
- As always, reload `firewalld` to ensure the rules are updated - Host aliases are set up in `/etc/hosts` to allow for name construction in software, e.g., `gem-shelf01-amc02`
``` bash
sudo firewall-cmd --reload
```
### P5/904 technical network ### P5/904 technical network
...@@ -233,21 +70,6 @@ technical network is managed by the CMS sysadmins. ...@@ -233,21 +70,6 @@ technical network is managed by the CMS sysadmins.
Physical machines are named by their rack/slot identifiers, virtual Physical machines are named by their rack/slot identifiers, virtual
machines based on their physical location. machines based on their physical location.
------------------------------------------------------------------------
GEM machine alias rack location Notes
--------------------- -------------------------- -----------------------
`bridge-s2e01-23` `s2c17-22-01` `control_hub` and
`ctrl-s2c17-22-01` `sysmgr` host
`gem-dqm01` `s2g18-34-01`
`gem-dqm01` `s2g18-34-01`
`gemvm-xdaq15` `kvm-s3562-1-ip151-107`
`gemvm-v3daq` `kvm-s3562-1-ip151-74`
------------------------------------------------------------------------
The μTCA shelf identifier is given by the rack U-slot number The μTCA shelf identifier is given by the rack U-slot number
corresponding to the bottom of the shelf, and all devices therein corresponding to the bottom of the shelf, and all devices therein
inherit this base hostname. For example, currently we have two μTCA inherit this base hostname. For example, currently we have two μTCA
...@@ -263,21 +85,7 @@ for the T1 and the other for the T2, and these are aliased to ...@@ -263,21 +85,7 @@ for the T1 and the other for the T2, and these are aliased to
the μTCA shelf will have aliases `amc-s2e01-23-XX`, were `XX` is the the μTCA shelf will have aliases `amc-s2e01-23-XX`, were `XX` is the
slot number. slot number.
!!! note When registering a new AMC card at P5, a Jira ticket should be
To be consistent with the GEM teststands, additional aliases are added
of the form `gem-shelfXX-amcYY`, and currently the mapping is the
follwoing:
-----------------------------------
GEM μTCA shelf rack location
----------------- -----------------
`gem-shelf01` `s2e01-23`
`gem-shelf02` `s2e01-14`
-----------------------------------
When registering a new AMC card at P5, a
`ticket<gemos-sysadmin-support>`{.interpreted-text role="ref"} should be
opened with the sysadmins to tell them the "birdname", but the opened with the sysadmins to tell them the "birdname", but the
geographic slot-based identifiers should already be registered if the geographic slot-based identifiers should already be registered if the
shelf was previously registered. shelf was previously registered.
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment