Commit 57dad328 authored by Laurent Petre's avatar Laurent Petre
Browse files

Update the developers instruction for PostgreSQL

parent 9d55e0e7
Pipeline #2960185 passed with stages
in 8 minutes and 1 second
......@@ -54,7 +54,7 @@ lint:
- poetry version ${GITVER_PEP440}
- poetry install
# Code formatting - Black
- poetry run black --check -l 100 gdh/ gemanalysis/ gemos/
- poetry run black --check -l 100 gdh gemanalysis gemos
# Linter - Pylint
- mkdir test
- poetry run pylint --output-format=text gdh gemanalysis gemos | tee test/pylint.log || poetry run pylint-exit $?
......
# Developing GEM Online Analysis
# Developers' guide
* [Development Setup](#setup)
* [Running Tests](#tests)
* [Code Style Guidelines](#rules)
* [Writing Documentation](#documentation)
[[_TOC_]]
## <a name="setup"> Development Setup
### Development requirements
## Development setup
The recommended way of development is under a virtual environment managed through `peotry`.
Same instructions are applicable to non-virtual environments too, but it is neither advised nor supported.
The recommended way of development is under a virtual environment managed through `peotry` on an official GEM development machine.
Similar instructions are applicable to non-virtual environments and personal machines too, but it is neither advised nor supported.
Use at your own risk.
First of all, you need to have `python` version `~3.8.6` available on your development PC.
Ensuring this is your own responsibility, as different operation systems are supplied with different python interpreters and different methods are used to install modern python interpreter alongside with the system one.
### Requirements
First of all, you need to have `python` version `~3.8.6` available on your development machine.
Ensuring this is your own responsibility, as different operation systems are supplied with different Python interpreters and different methods are used to install modern Python interpreter alongside with the system one.
On CERN's CC7 `lxplus` machines the recommended way is to [source an appropriate software collection](https://cern.service-now.com/service-portal?id=kb_article&n=KB0000730).
On official GEM development PCs `python3.8` is already installed.
On official GEM development machines `python3.8` is already installed.
#### Peotry tips
In addition, a `PostgreSQL` server version `10` must be running.
On official GEM development machines this is also taken care of for you by the sysadmins, every user is provided a personal database.
The `peotry` package is installed on the GEM development machines and can be enabled as follow:
### Quick start
```
source scl_source enable rh-python38 # Optional
export PATH=/opt/peotry/bin:$PATH
```
0. Before starting read this guide further to get familiar with our development practices and tools
1. Set up your environment (to be done every time you start a new session).
On official GEM development machines, this can be resumed with the following commands:
``` shell
source scl_source enable rh-git218 # Newer Git release (optional)
source scl_source enable rh-python38 # Python 3.8 release (optional)
export PATH=/opt/peotry/bin:$PATH # Poetry
```
2. Clone the repository `git clone https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis.git && cd cmsgemos-analysis`
3. Install the `cmsgemos-analysis` project along with all its dependencies in a virtual environment `peotry install`
4. You are ready... if you have completed point 0 :wink:
If you are not using the GEM development machines, you can install `peotry` for your user as follow:
## Development guidelines
### Coding style
```shell
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/install-poetry.py | python3 -
export PATH=$HOME/.local/bin:$PATH
```
We are following WebKit [Code Style Guideline](https://webkit.org/code-style-guidelines/) for `C++` and [PEP8](https://www.python.org/dev/peps/pep-0008/) standard for Python code.
We also find very useful [C++ Core Guidelines](https://github.com/isocpp/CppCoreGuidelines/) by Bjarne Stroustrup and Herb Sutter and strongly encourage you to read them too.
Please note that the virtual environments and caches created by `peotry` are stored in your personal home folder.
In some setups, it is desired to move their locations to other directories.
This can be achieved via `peotry` configuration, either per project or global to your user:
You can check and fix your code formatting through the usage of `Black`:
``` shell
poetry config [--local] virtualenvs.in-project true # Store the virtual environment under .venv in the project
poetry config [--local] cache-dir <my-path-with-more-storage> # Modify the cache location
poetry run black --check -l 100 gdh gemanalysis gemos
```
In order to avoid prefixing all commands with `peotry run`, one can enter into a sub-shell in which the virtual environment is enabled:
The CI will also ensure your code is properly formatted before merging any of your changes.
``` shell
peotry shell
### Documentation writing
We use the [sphinx](https://www.sphinx-doc.org/en/master/usage/quickstart.html) with read-the-docs theme for `python` code documentation.
The additional necessary dependencies are installed as part of the developments packages.
*Note: `nbsphinx` uses [pandoc](https://pandoc.org/installing.html) to convert markdown cells.
If you don't have it installed in your system, you should install it manually.*
The documentation can be generated using the following command:
```shell
poetry run sphinx-build -b html doc dist/doc
```
### Installation instructions
Generated files are placed in `dist/doc`.
* Clone the repository `git clone https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis.git && cd cmsgemos-analysis`
* Install the `cmsgemos-analysis` project along with all its dependencies in a virtual environment `peotry install`
This section will be completed later with precise documentation style guide, for the moment follow the examples from the code itself.
## <a name="tests"> Running Tests
### Checking whether your code is up to project standard
## Running tests
Use [*pylint*](https://www.pylint.org/) to check if your code quality is good.
Use [pylint](https://www.pylint.org/) to check if your code quality is good.
``` shell
poetry run pylint gdh gemanalysis gemos
......@@ -84,37 +94,50 @@ This command will show you a link to `localhost` which you will have to use. Cop
2. On your local PC setup an ssh port forwarding:
```shell
ssh -vL 8889:gem904daq01.cern.ch:8889 -oProxyCommand="ssh <your_lxplus_name>@lxplus7.cern.ch -W %h:%p" <your_lxplus_name>@gem904daq01.cern.ch
ssh -vL 8889:gem904daq04.cern.ch:8889 -oProxyCommand="ssh <your_lxplus_name>@lxplus7.cern.ch -W %h:%p" <your_lxplus_name>@gem904daq04.cern.ch
```
3. Open the link you've copied in step 1 in your browser. Enjoy!
## <a name="rules"> Code Style Guidelines
We are following WebKit [Code Style Guideline](https://webkit.org/code-style-guidelines/) for `C++` and [PEP8](https://www.python.org/dev/peps/pep-0008/) standard for python code.
We also find very useful [C++ Core Guidelines](https://github.com/isocpp/CppCoreGuidelines/) by Bjarne Stroustrup and Herb Sutter and strongly encourage you to read them too.
## Tips
### Peotry
You can check and fix your code formatting through the usage of `Black`:
If you are not using the GEM development machines, you can install `peotry` for your user as follow:
``` shell
poetry run black --check -l 100 gdh/ gemanalysis/ gemos/
```shell
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/install-poetry.py | python3 -
export PATH=$HOME/.local/bin:$PATH
```
The CI will also ensure your code is properly formatted before merging your changes.
Please note that the virtual environments and caches created by `peotry` are stored in your personal home folder.
In some setups, it is desired to move their locations to other directories.
This can be achieved via `peotry` configuration, either per project or global to your user:
## <a name="documentation"> Writing Documentation
``` shell
poetry config [--local] virtualenvs.in-project true # Store the virtual environment under .venv in the project
poetry config [--local] cache-dir <my-path-with-more-storage> # Modify the cache location
```
We use the [sphinx](https://www.sphinx-doc.org/en/master/usage/quickstart.html) with read-the-docs theme for `python` code documentation.
The additional necessary dependencies are installed as part of the developments packages.
In order to avoid prefixing all commands with `peotry run`, one can enter into a sub-shell in which the virtual environment is enabled:
*Note:* `nbsphinx` *uses* [*pandoc*](https://pandoc.org/installing.html) *to convert markdown cells. If you don't have it installed in your system, you should install it manually.*
``` shell
peotry shell
```
The documentation can be generated using the following command:
### PostgreSQL
```bash
poetry run sphinx-build -b html doc dist/doc
On official GEM development machines, every user is provided a personal database with the following parameters:
```
host: localhost
port: 5432
user: ${USER}
password: ${USER}
database: ${USER}
```
Generated files are placed in `dist/doc`.
From your account, it is possible to connect directly to your database without any password with `psql`.
This section will be completed later with precise documentation style guide, for the moment follow the examples from the code itself.
#### `psql` cheatsheet
* Quit the client: `\q`
* List the tables in the current database: `\dt+`
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment