Many actions on the card requie the Remote Procedure Call (RPC) service to be running and owned by the `gemuser` account. This service is run as a system process called `rpcsvc`. To check if this process is running login to the CTP7's linux image an execute:
```bash
eagleXX:~$ ps | grep rpcsvc
```
Example output should look like:
```bash
eagle60:~$ ps | grep rpcsvc
10994 gemuser 4040 S rpcsvc
14004 gemuser 2796 S grep rpcsvc
```
If you only see a line that says `grep rpcsvc` then the `rpcsvc` service is *not* running. Additionally if you see that `rpcsvc` is running but it is not owned by the `gemuser` account then it will be configured with the *wrong* library and must be killed (either by the sysadmin using the `root` account or from a `$USER` of the other account). Note you might find that multiple lines are returned which show multiple `rpcsvc` services. If these are all owned by `gemuser` then this just indicates that the `rpcsvc` service is running and that one or more open `rpc` connections exist between the card and the DAQ PC.
Sometimes you may need to restart the `rpcsvc` (typically after updating the `rpcmodules` on the card), to do this execute:
```bash
killall rpcsvc && rpcsvc
```
##### RPC Modules and the LMDB
When a command to interface with the front-end hardware is sent from the DAQ machine this typically uses either the micro hardware access library (`uhal`) built on top of `ipbus` or the cross hardware access library (`xhal`) built on top of `rpcsvc` (and using `TCP/IP` communication). An RPC message is sent to the CTP7 and the Zynq processor on the CTP7 will load a function that is defined in a shared object library (`*.so`) file under `/mnt/persistent/rpcmodules`. These shared object libraries are referred to as `rpcmodules`. For more detailes please see the [README of the ctp7_mopdules](https://github.com/cms-gem-daq-project/ctp7_modules/blob/develop/README.md) repository.
On a production machine these will be the following set (although more may be present in the future):
```bash
$ ll /mnt/persistent/rpcmodules/
-rwxr-xr-x 1 root root 49093 Oct 9 16:48 amc.so
-rwxr-xr-x 1 root root 140180 Oct 9 16:48 calibration_routines.so
-rwxr-xr-x 1 root root 95769 Oct 9 16:48 daq_monitor.so
-rwxr-xr-x 1 root root 13917 Oct 9 16:48 extras.so
With the user editable area being `/mnt/persistent/gemuser/rpcmoduleTesting/`.
The LMDB is a database that the `rpcmodules` query when functions are launched on the CTP7. The `rpcmodules` are segmented into what are referred to as local or remote/non-local modules. The local version has `someStringLocal()` always in it's function name and is only ever executed from the CTP7. The non-local module receives the RPC Message from the host DAQ PC, calls the corresponding local function(s) and returns and RPC response to the host machine. Note that no terminal output appears on the host machine during the execution of either the local or non-local `rpcmodules`. Instead the developer will have configured the functions to log actions on the CTP7's log file which is available on the CTP7 at:
```
/var/log/messages
```
Note that this file is *not* found under the `/mnt/persistent` partition and is re-written everytime the card reboots. To ensure log information is not lost the DAQ PC is configured to have a version of this file found under:
```
/var/log/remote/eagleXX/messages.log
```
Where `eagleXX` is the network alias of the CTP7 (e.g. `eagle64`). This file is written to disk on the DAQ PC and will persist through crashes/reboots of the CTP7 (so no information will be lost). You can see the most recent information found in the log file via:
```bash
tail-25 /var/log/messages
```
This will display the last 25 lines in the log file on the CTP7 (executed as `gemuser` on the CTP7). Use the `tail` command similarly to view the log on the DAQ PC.
##### Restarting `syslogd`
In some weird cases `syslogd` will not be running after the CTP7 boots. This will cause the log file:
