From 0ec8821cbd6c3d5bac15ed832cc980987821cdc9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Louis=20R=C3=A9gnier?= <louis.regnier@cern.ch> Date: Fri, 5 Jul 2024 17:25:43 +0200 Subject: [PATCH 1/5] Newcomers: use the CERN VOMS as the default provider The HellasGrid service is not available anymore, use the CERN VOMS instead. --- docs/developers/newcomers.md | 36 +++++++++++++++++------------------- 1 file changed, 17 insertions(+), 19 deletions(-) diff --git a/docs/developers/newcomers.md b/docs/developers/newcomers.md index 3ffcd135..51b6ce5d 100644 --- a/docs/developers/newcomers.md +++ b/docs/developers/newcomers.md @@ -139,12 +139,12 @@ export X509_USER_KEY=$HOME/Certs/userkey.pem ### Join a VO -If you are a developer, join the `dteam` VO. To do this, install the p12 you got +If you are a developer, join the CERN `dteam` VO. To do this, install the p12 you got before into your browser as a user certificate, and go to -[https://voms2.hellasgrid.gr:8443/voms/dteam/](https://voms2.hellasgrid.gr:8443/voms/dteam/). +[https://dteam-auth.cern.ch/start-registration](https://dteam-auth.cern.ch/start-registration). Your browser will likely not trust the certificate of this host. You can either accept the warning, or install the -[HellasGrid Root CA](http://www.hellasgrid.gr/ca/) beforehand. +[CERN Root CA](https://ca.cern.ch/cafiles/) beforehand Follow the instructions and wait to be accepted into the VO. It is recommended to assign your request to somebody from your home institute. @@ -197,15 +197,13 @@ SFTP: ``` $ sftp <login>@lxplus.cern.ch -sftp> cd /etc/vomses -sftp> get dteam* -Fetching /etc/vomses/dteam-voms2.hellasgrid.gr to dteam-voms2.hellasgrid.gr -/etc/vomses/dteam-voms2.hellasgrid.gr 100% 112 135.3KB/s 00:00 +sftp> get /etc/vomses/dteam-voms-dteam-auth.cern.ch +Fetching /etc/vomses/dteam-voms-dteam-auth.cern.ch to dteam-voms-dteam-auth.cern.ch +dteam-voms-dteam-auth.cern.ch 100% 105 196.3KB/s 00:00 -sftp> cd /etc/grid-security/vomsdir/dteam -sftp> get * -Fetching /etc/grid-security/vomsdir/dteam/voms2.hellasgrid.gr.lsc to voms2.hellasgrid.gr.lsc -/etc/grid-security/vomsdir/dteam/voms2.hellasgrid.gr.lsc 100% 129 148.6KB/s 00:00 +sftp> get /etc/grid-security/vomsdir/dteam/voms-dteam-auth.cern.ch.lsc +Fetching /etc/grid-security/vomsdir/dteam/voms-dteam-auth.cern.ch.lsc to voms-dteam-auth.cern.ch.lsc +voms-dteam-auth.cern.ch.lsc 100% 102 260.6KB/s 00:00 sftp> quit ``` @@ -214,9 +212,9 @@ Move said files in the following locations: ```bash $ sudo mkdir /etc/vomses -$ sudo mv dteam-voms2.hellasgrid.gr /etc/vomses/ -$ sudo mkdir /etc/grid-security/vomsdir/dteam -$ sudo mv voms2.hellasgrid.gr.lsc /etc/grid-security/vomsdir/dteam/ +$ sudo mv dteam-voms-dteam-auth.cern.ch /etc/vomses/ +$ sudo mkdir -p /etc/grid-security/vomsdir/dteam +$ sudo mv voms-dteam-auth.cern.ch.lsc /etc/grid-security/vomsdir/dteam/ ``` Congratulations, you will be now able to create your **VOMS Proxy**: @@ -229,26 +227,26 @@ If everything has worked properly, you will get an output similar to the one below: ``` -Contacting voms2.hellasgrid.gr:15004 [/C=GR/O=HellasGrid/OU=hellasgrid.gr/CN=voms2.hellasgrid.gr] "dteam"... +Contacting voms-dteam-auth.cern.ch:443 [/DC=ch/DC=cern/OU=computers/CN=dteam-auth.cern.ch] "dteam"... Remote VOMS server contacted succesfully. -Created proxy in /tmp/x509up_u141868. +Created proxy in /tmp/x509up_u$(id -u). -Your proxy is valid until Mon Jul 26 13:10:42 CEST 2021 +Your proxy is valid until Mon Jul 26 13:10:42 CEST 2024 ``` #### Considerations on proxy certificates `voms-proxy-init` created a temporary certificate/key pair (with a limited lifetime), signed with your own private key and with an extension signed by -HellasGrid certifying your belonging to the `dteam` VO. +the CERN, certifying your belonging to the `dteam` VO. These proxies are useful as you don't need to move around your private key. More so, if they are exposed, the harm is reduced as they are limited in scope and lifetime. -The default location for the proxy is `/tmp/x509up_u${UID}`. You can override it +The default location for the proxy is `/tmp/x509up_u$(id -u)`. You can override it with the environment `X509_USER_PROXY`. You may see an error that looks like this after while: -- GitLab From f4d2add0ca94af00751440ffab5b9066663dbd76 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Louis=20R=C3=A9gnier?= <louis.regnier@cern.ch> Date: Thu, 25 Jul 2024 17:24:55 +0200 Subject: [PATCH 2/5] Newcomers: move from CentOS7 to Alma9 --- docs/developers/newcomers.md | 64 +++++++++++++----------------------- 1 file changed, 23 insertions(+), 41 deletions(-) diff --git a/docs/developers/newcomers.md b/docs/developers/newcomers.md index 51b6ce5d..ee953d83 100644 --- a/docs/developers/newcomers.md +++ b/docs/developers/newcomers.md @@ -90,7 +90,7 @@ Project: FTS Database type: MySQL8 ``` For the virtual machine, create an OpenStack compute node under your personal project -using the CC7 image and set up some form of authentication (e.g.: ssh keys). +using the Alma9 image and set up some form of authentication (e.g.: ssh keys). Now we can ssh into the server and start working! @@ -166,7 +166,7 @@ After having set up properly your certificates, you will be ready to install the packages required for the creation of a **VOMS Client**: ```bash -$ sudo yum install voms-clients-java +$ sudo dnf install voms-clients-java ``` Specifically, for the **ca-policy-egi-core** meta-package, you will have to @@ -187,7 +187,7 @@ enabled=1 After that, you will be able to install the meta-package **ca-policy-egi-core**: ```bash -$ sudo yum install ca-policy-egi-core +$ sudo dnf install ca-policy-egi-core ``` Last but not least, you will also need a few more files for both **vomses** and @@ -264,7 +264,7 @@ CRLs allow you to check whether a certificate, although valid, has been revoked in the meantime. You can update CRLs with the following command: ```bash -$ sudo yum install fetch-crl +$ sudo dnf install fetch-crl $ sudo fetch-crl -v ``` @@ -285,7 +285,7 @@ way as with standard UNIX tools. Let's pick `https://eospublic.cern.ch/eos/opstest/dteam/` as the storage endpoint for our tests. ```bash -$ sudo yum install python3-gfal2-util +$ sudo dnf install python3-gfal2-util # Note: Use your own user name $ gfal-mkdir "https://eospublic.cern.ch/eos/opstest/dteam/${USER}" @@ -311,7 +311,7 @@ Submitting an FTS transfer is as easy as follows: ```bash # You may need the FTS3 production repo first # Look at 'Server from existing packages locally' section -$ sudo yum install fts-rest-client +$ sudo dnf install fts-rest-client ``` ```bash @@ -376,7 +376,7 @@ so that's why we explain how to run the tests _before_ how to modify the code. #### Download the dependencies: ```bash -$ yum install -y git python3 python3-devel openssl-devel swig gcc gcc-c++ python36-m2crypto ca-policy-lcg voms-clients-java +$ dnf install -y git python3 python3-devel openssl-devel swig gcc gcc-c++ python3-m2crypto ca-policy-lcg voms-clients-java ``` #### Clone the "tests" repository: @@ -462,7 +462,7 @@ the wait. You can get the certificate directly on a CERN machine using the following commands: ```bash -$ yum -y install cern-get-certificate +$ dnf -y install cern-get-certificate $ cern-get-certificate --autoenroll --grid $ cern-get-certificate --status --grid ``` @@ -505,34 +505,16 @@ $ firewall-cmd --permanent --zone=public --add-port=8449/tcp #### Oracle MySQL Database setup -You can install a local Oracle MySQL database using the following script. It is recommended to put the following instructions in a shell script file e.g. `install-oracle-mysql.sh`. Then simply execute the script. - -> **Note** Don't forget to give execution rights to the above mentioned shell script. +You can install a local Oracle MySQL database using the following command: ```bash -#/bin/sh - -rm -rf /root/mysql -mkdir /root/mysql -wget https://dev.mysql.com/get/mysql80-community-release-el7-7.noarch.rpm --output-document=/root/mysql/mysql80-community-release-el7-7.noarch.rpm -EXPECTED_MD5=659400f9842fffb8d64ae0b650f081b9 -ACTUAL_MD5=`md5sum /root/mysql/mysql80-community-release-el7-7.noarch.rpm | awk '{print $1;}'` -if test "${EXPECTED_MD5}" = "${ACTUAL_MD5}"; then - echo "The md5 checksum of is correct" -else - echo "The md5 checksum of is not correct" - exit 1 -fi - -yum localinstall -y /root/mysql/mysql80-community-release-el7-7.noarch.rpm -yum install -y mysql-community-server +dnf install -y mysql mysql-server ``` You can enable and start the mysql daemon by using the following commands: ```bash -$ systemctl enable mysqld -$ systemctl start mysqld +$ systemctl enable --now mysqld ``` You can login to the database with root user by using the default password for the first time. You need to hit this command and then enter the default password. @@ -580,16 +562,16 @@ repeated here. First we need to install the FTS and DMC production repositories: ```bash -$ wget https://fts-repo.web.cern.ch/fts-repo/fts3-el7.repo -O /etc/yum.repos.d/fts3-el7.repo -$ wget https://fts-repo.web.cern.ch/fts-repo/fts3-depend-el7.repo -O /etc/yum.repos.d/fts3-depend-el7.repo -$ wget https://dmc-repo.web.cern.ch/dmc-repo/dmc-el7.repo -O /etc/yum.repos.d/dmc-el7.repo +$ curl https://fts-repo.web.cern.ch/fts-repo/fts3-el9.repo -o /etc/yum.repos.d/fts3-el9.repo +$ curl https://fts-repo.web.cern.ch/fts-repo/fts3-depend.repo -o /etc/yum.repos.d/fts3-depend-el9.repo +$ curl https://dmc-repo.web.cern.ch/dmc-repo/dmc-el9.repo -o /etc/yum.repos.d/dmc-el9.repo $ yum install centos-release-scl-rh ``` ```bash -$ yum install fts-server fts-monitoring fts-rest-server fts-rest-client -$ yum install fts-server-selinux fts-monitoring-selinux fts-rest-server-selinux -$ yum install fts-mysql +$ dnf install fts-server fts-monitoring fts-rest-server fts-rest-client +$ dnf install fts-server-selinux fts-monitoring-selinux fts-rest-server-selinux +$ dnf install fts-mysql ``` We also need `ca-policy-egi-core` and `fetch-crl` but if you have followed this @@ -692,9 +674,9 @@ Installation is currently laid out well in the `README.md` of the project, but repeating here... ```bash -# Install rh-python36-mod_wsgi -$ yum-config-manager --enable centos-sclo-rh -$ yum install -y rh-python36-mod_wsgi +# Install python3-mod_wsgi +$ dnf config-manager --set-enabled crb +$ dnf install -y python3-mod_wsgi $ git clone https://gitlab.cern.ch/fts/fts-rest-flask.git $ cd fts-rest-flask/ @@ -767,7 +749,7 @@ $ git clone https://gitlab.cern.ch/fts/fts3.git $ cd fts3/ # Install dependencies for building FTS3 binaries -$ yum-builddep packaging/rpm/fts.spec +$ dnf builddep packaging/rpm/fts.spec $ mkdir cmake-build/ $ cd cmake-build/ @@ -799,7 +781,7 @@ $ git clone https://gitlab.cern.ch/fts/fts-monitoring.git $ cd fts-monitoring/ # Install dependencies for the Web Monitoring component -$ yum-builddep packaging/rpm/fts-monitoring.spec +$ dnf builddep packaging/rpm/fts-monitoring.spec # Mimic the RPM installation steps $ \cp -rf src/* /usr/share/fts3web/ @@ -827,7 +809,7 @@ $ git clone https://gitlab.cern.ch/dmc/gfal2.git $ cd gfal2/ # Install dependencies for building Gfal2 binaries -$ yum-builddep packaging/rpm/gfal2.spec +$ dnf builddep packaging/rpm/gfal2.spec $ mkdir cmake-build/ $ cd cmake-build/ -- GitLab From 68ad71c48ff9bdb92cff64adb03bc0cb4cdd8b3b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Louis=20R=C3=A9gnier?= <louis.regnier@cern.ch> Date: Thu, 25 Jul 2024 16:21:56 +0200 Subject: [PATCH 3/5] Newcomers: Move "Running Tests" category after "Build from Sources" --- docs/developers/newcomers.md | 191 +++++++++++++++++------------------ 1 file changed, 95 insertions(+), 96 deletions(-) diff --git a/docs/developers/newcomers.md b/docs/developers/newcomers.md index ee953d83..6f97eb21 100644 --- a/docs/developers/newcomers.md +++ b/docs/developers/newcomers.md @@ -363,100 +363,6 @@ the job id into the search box in the top right corner, and hit enter. ![Web Monitoring Job View](webmon-job-view.png "Job View") -## Running (Integration) Tests - -When you have your own development instance, and you start changing things, not -only will you need to be able to submit simple transfers, but you will also need -to run the existing tests to make sure there are no regressions. - -Besides, you will also need to add your own tests when fixing bugs or adding -features. This comes (or should come) before any commit, release, or anything, -so that's why we explain how to run the tests _before_ how to modify the code. - -#### Download the dependencies: - -```bash -$ dnf install -y git python3 python3-devel openssl-devel swig gcc gcc-c++ python3-m2crypto ca-policy-lcg voms-clients-java -``` - -#### Clone the "tests" repository: - -```bash -$ git clone "https://gitlab.cern.ch/fts/fts-tests.git" -$ cd "fts-tests" -``` - -#### Set up the Python environment - -We can now use `./run-helper.sh` to prepare a Python virtual environment, set -everything up and run the pytest tests for us. We will, however, need to set some -environment variables first. - -```bash -export FTS3_ENABLE_MSG=False # Stops sending info to broker -export BROKER="dashb-mb.cern.ch" # If above is true, send broker messages here -export FTS3_HOST="fts3-devel.cern.ch" # Host to test on -export SHORT_RUN=1 # Skip some tests -export NO_SIGNALS=1 # Skip additional tests for error signals -``` - -You can see there is a `run-helper.sh` that prepares a virtualenv and run all -tests for you. Remember you must have a valid proxy before. - -If we want to run one manually, we can craft a virtual environment ourselves. We -still need to have the above environment variables. This code is from -the [gitlab-CI](https://gitlab.cern.ch/fts/fts-tests/-/blob/develop/.gitlab-ci.yml) pipeline, -which automatically runs these tests. These pipelines are a good place to look -to find how things are done if documentation is lacklustre. - -```bash -$ python3 -m venv "/tmp/my-fts-tests" -$ source "/tmp/my-fts-tests/bin/activate" -$ pip install --upgrade pip -$ pip install "git+https://gitlab.cern.ch/fts/fts-rest-flask.git@develop" -$ pip install "pytest" -$ pip install "git+https://github.com/jasonrbriggs/stomp.py@pre-python2-removal" -``` - -If everything goes well, this should work: - -```bash -# Remember! You must have created your proxy before -$ fts-rest-whoami -s "https://fts3-devel.cern.ch:8446" -``` -#### Run the simplest test - -```bash -$ ./fts_simple.py -``` - -The output should look something like - -``` -INFO Submitted a new job with id aa94f800-0100-11e8-a0d4-02163e0170e3 -INFO Waiting for aa94f800-0100-11e8-a0d4-02163e0170e3 -INFO Poll aa94f800-0100-11e8-a0d4-02163e0170e3 -INFO Submitted a new job with id adbd2db8-0100-11e8-a65a-02163e00a077 -INFO Waiting for adbd2db8-0100-11e8-a65a-02163e00a077 -INFO Poll adbd2db8-0100-11e8-a65a-02163e00a077 -. ----------------------------------------------------------------------- -Ran 2 tests in 10.569s - -OK -``` - -If everything is OK, congratulations! You now know how to verify if your -instance is properly set up, and write and run tests, which are the foundations -to start closing tickets, or cleaning up code. - -You should spend some time reading the tests to understand what they are doing. -Anytime you see a use case, or permutation missing, do not hesitate to add it. - -These tests are run on git pushes in the gitlab CI pipeline automatically, so you can -always look at the result there, rather than running it yourself if you don't mind -the wait. - #### Obtain a root certificate on a CERN machine You can get the certificate directly on a CERN machine using the following commands: @@ -549,7 +455,7 @@ By Default, Oracle uses its own special Authentication method whose plugin is no ## Server from existing packages locally -Now that we have tested on a development server and ran tests, we're ready to start +Now that we have tested on a development server, we're ready to start the server on our own machine and test against it. To set up the entire system, we need the Server, REST Server, database and @@ -770,6 +676,100 @@ $ /usr/sbin/fts_qos -r -n $ systemctl restart fts-server fts-qos ``` +### Running (Integration) Tests + +When you have your own development instance, and you start changing things, not +only will you need to be able to submit simple transfers, but you will also need +to run the existing tests to make sure there are no regressions. + +Besides, you will also need to add your own tests when fixing bugs or adding +features. This comes (or should come) before any commit, release, or anything. + +#### Download the dependencies: + +```bash +$ dnf install -y git python3 python3-devel openssl-devel swig gcc gcc-c++ python3-m2crypto ca-policy-lcg voms-clients-java +``` + +#### Clone the "tests" repository: + +```bash +$ git clone "https://gitlab.cern.ch/fts/fts-tests.git" +$ cd "fts-tests" +``` + +#### Set up the Python environment + +We can now use `./run-helper.sh` to prepare a Python virtual environment, set +everything up and run the pytest tests for us. We will, however, need to set some +environment variables first. + +```bash +export FTS3_ENABLE_MSG=False # Stops sending info to broker +export BROKER="dashb-mb.cern.ch" # If above is true, send broker messages here +export FTS3_HOST="fts3-devel.cern.ch" # Host to test on +export SHORT_RUN=1 # Skip some tests +export NO_SIGNALS=1 # Skip additional tests for error signals +``` + +You can see there is a `run-helper.sh` that prepares a virtualenv and run all +tests for you. Remember you must have a valid proxy before. + +If we want to run one manually, we can craft a virtual environment ourselves. We +still need to have the above environment variables. This code is from +the [gitlab-CI](https://gitlab.cern.ch/fts/fts-tests/-/blob/develop/.gitlab-ci.yml) pipeline, +which automatically runs these tests. These pipelines are a good place to look +to find how things are done if documentation is lacklustre. + +```bash +$ python3 -m venv "/tmp/my-fts-tests" +$ source "/tmp/my-fts-tests/bin/activate" +$ pip install --upgrade pip +$ pip install "git+https://gitlab.cern.ch/fts/fts-rest-flask.git@develop" +$ pip install "pytest" +$ pip install "git+https://github.com/jasonrbriggs/stomp.py@pre-python2-removal" +``` + +If everything goes well, this should work: + +```bash +# Remember! You must have created your proxy before +$ fts-rest-whoami -s "https://fts3-devel.cern.ch:8446" +``` +#### Run the simplest test + +```bash +$ ./fts_simple.py +``` + +The output should look something like + +``` +INFO Submitted a new job with id aa94f800-0100-11e8-a0d4-02163e0170e3 +INFO Waiting for aa94f800-0100-11e8-a0d4-02163e0170e3 +INFO Poll aa94f800-0100-11e8-a0d4-02163e0170e3 +INFO Submitted a new job with id adbd2db8-0100-11e8-a65a-02163e00a077 +INFO Waiting for adbd2db8-0100-11e8-a65a-02163e00a077 +INFO Poll adbd2db8-0100-11e8-a65a-02163e00a077 +. +---------------------------------------------------------------------- +Ran 2 tests in 10.569s + +OK +``` + +If everything is OK, congratulations! You now know how to verify if your +instance is properly set up, and write and run tests, which are the foundations +to start closing tickets, or cleaning up code. + +You should spend some time reading the tests to understand what they are doing. +Anytime you see a use case, or permutation missing, do not hesitate to add it. + +These tests are run on git pushes in the gitlab CI pipeline automatically, so you can +always look at the result there, rather than running it yourself if you don't mind +the wait. + + ### FTS Web Monitoring The Web Monitoring component is composed of Python Django code and static files. @@ -827,4 +827,3 @@ $ \cp -fv /etc/gfal2.d.saved/* /etc/gfal2.d/ At this point, not only do you have your own development FTS instance, but you can also modify the source code and have the changes deployed. -Happy development! -- GitLab From ada832a10a21dde1121725db68f7f07bebe9fe24 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Louis=20R=C3=A9gnier?= <louis.regnier@cern.ch> Date: Thu, 25 Jul 2024 17:25:06 +0200 Subject: [PATCH 4/5] Newcomers: use Oracle MySQL for local database, remove MariaDB references Oracle MySQL is used in production, suggest it also for development, to have similar behaviors/environnements. --- docs/developers/newcomers.md | 95 +++++++++++++++++------------------- 1 file changed, 44 insertions(+), 51 deletions(-) diff --git a/docs/developers/newcomers.md b/docs/developers/newcomers.md index 6f97eb21..f69d1453 100644 --- a/docs/developers/newcomers.md +++ b/docs/developers/newcomers.md @@ -409,50 +409,6 @@ $ firewall-cmd --zone=public --add-port=8449/tcp $ firewall-cmd --permanent --zone=public --add-port=8449/tcp ``` -#### Oracle MySQL Database setup - -You can install a local Oracle MySQL database using the following command: - -```bash -dnf install -y mysql mysql-server -``` - -You can enable and start the mysql daemon by using the following commands: - -```bash -$ systemctl enable --now mysqld -``` - -You can login to the database with root user by using the default password for the first time. You need to hit this command and then enter the default password. - -```bash -$ mysql -u root -p -``` - -Now you need to set up the root password: - -```sql -> ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'Password'; -``` - -You can use following command to create a user in Oracle MySQL database so that native MySQL authentication is used. -By Default, Oracle uses its own special Authentication method whose plugin is not installed by default during Oracle MySQL installation process. So, to solve this problem hit this command in mysql. - -```sql ----Create the database -> CREATE DATABASE fts3db; ---- Check user already exists or not -> SELECT User FROM mysql.user; ----Create user -> CREATE USER 'USER'@'%' IDENTIFIED BY 'UserPassword'; ----Alter the authentication method for the user -> ALTER USER 'USER'@'%' IDENTIFIED WITH mysql_native_password BY 'UserPassword'; ---- GRant the privilages to the user -> GRANT ALL PRIVILEGES ON fts3db.* TO 'USER'@'%' -``` - -> **Note** You may need to set the configuration in the files /etc/fts3/fts3config and /etc/fts3/fts3restconfig to have the appropriate connection strings (explained in next section) - ## Server from existing packages locally Now that we have tested on a development server, we're ready to start @@ -504,15 +460,52 @@ Ensure ports `8446` for REST and `8449` for Web Monitoring are open. We can now set up the database. +#### Oracle MySQL Database setup + +You can install and run a local Oracle MySQL database using the following commands: + +```bash +$ dnf install -y mysql mysql-server +$ systemctl enable --now mysqld +``` + +You can login to the database with root user by using the default password +for the first time. You need to hit this command and then enter the default password. + ```bash -$ yum install mariadb -$ mysql -h <db-hostname> -P <database-port> -u admin -p -MySQL [(none)]> CREATE DATABASE fts3db; -MySQL [(none)]> CREATE USER 'fts3user'@'%' IDENTIFIED BY 'yourSECUREpassword'; -MySQL [(none)]> GRANT ALL PRIVILEGES ON fts3db.* TO 'fts3user'@'%' +$ mysql -u root -p +``` + +Now you need to set up the root password: + +```sql +> ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'Your Password'; +``` -$ mysql -h <db-hostname> -P <database-port> -u fts3user --password='yourSECUREpassword' fts3db -MySQL [fts3db]> source /usr/share/fts-mysql/fts-schema-8.0.0.sql +You can use following command to create a user in Oracle MySQL database so that native MySQL authentication is used. +By Default, Oracle uses its own special Authentication method whose plugin is not installed by default during Oracle +MySQL installation process. So, to solve this problem hit this command in mysql. + +```sql +---Create the database +> CREATE DATABASE fts3db; +--- Check user already exists or not +> SELECT User FROM mysql.user; +---Create user +> CREATE USER 'fts3user'@'%' IDENTIFIED BY 'Your Password'; +---Alter the authentication method for the user +> ALTER USER 'fts3user'@'%' IDENTIFIED WITH mysql_native_password BY 'Your Password'; +--- GRant the privilages to the user +> GRANT ALL PRIVILEGES ON fts3db.* TO 'fts3user'@'%' +``` + +```bash +$ mysql -h <db-hostname> -P <database-port> -u fts3user --password='Your Password' fts3db +``` + +```sql +---Apply FTS SQL schema +> source /usr/share/fts-mysql/fts-schema-8.0.0.sql ``` As an alternative to setting up a local database, if you have a CERN account, you can request a -- GitLab From 4ad99c59d84078f9b8d252711bf9d4849614a615 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Louis=20R=C3=A9gnier?= <louis.regnier@cern.ch> Date: Thu, 25 Jul 2024 17:47:13 +0200 Subject: [PATCH 5/5] Newcomers: remove trailing spaces --- docs/developers/newcomers.md | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/docs/developers/newcomers.md b/docs/developers/newcomers.md index f69d1453..a50f9fef 100644 --- a/docs/developers/newcomers.md +++ b/docs/developers/newcomers.md @@ -32,7 +32,7 @@ Repository: https://gitlab.cern.ch/fts/fts-rest-flask Polls the database for work to do, transferring files via the _swiss army knife_ that is [Gfal2](https://dmc-docs.web.cern.ch/dmc-docs/gfal2/gfal2.html). Submissions are in the form of "Please transfer file X from Source to -Destination". Moving the file is done preferably via a 'Third party copy', +Destination". Moving the file is done preferably via a 'Third party copy', where the data goes directly from the source to the destination instead of streaming via the FTS server. However, this is not always possible, in which case the FTS server can fallback to streaming the data. @@ -46,7 +46,7 @@ Repository: https://gitlab.cern.ch/fts/fts3 ### Web Monitoring component -A web interface to monitor transfers on a particular server. +A web interface to monitor transfers on a particular server. Example: https://fts3-devel.cern.ch:8449/fts3/ftsmon/#/ Repository: https://gitlab.cern.ch/fts/fts-monitoring @@ -104,7 +104,7 @@ if you don't belong to a You can skip this section if you know what I am talking about, and you already have your certificate. -The procedure to do this depends on your home institute. +The procedure to do this depends on your home institute. #### For CERN account holders @@ -418,7 +418,7 @@ To set up the entire system, we need the Server, REST Server, database and Web Monitoring component. Much of this can be found in the [Quick Start](../install/quick.md) guide, repeated here. -> **Note**: this section onwards assumes you are running as root +> **Note**: this section onwards assumes you are running as root > or have all the necessary permissions First we need to install the FTS and DMC production repositories: @@ -549,27 +549,27 @@ You can now submit transfer jobs to your own server with if you're running the transfer command from the same host as the `fts-rest-flask` server. -By this point, you should have your own FTS development instance, +By this point, you should have your own FTS development instance, installed and configured from RPMs, up and running. ## Building / Running from source code It is advisable to make a dedicated folder (e.g.: `/workspace`) where to keep all projects (e.g.: `/workspace/fts-rest-flask`). -Do not use the `/root` directory as other system users won't have access, +Do not use the `/root` directory as other system users won't have access, which can cause hard to track bugs. -The installation instructions are often in the Readme of each project. +The installation instructions are often in the Readme of each project. Ignore any references to vagrant, as these are outdated. ### FTS-REST-Flask (server) The FTS-REST-Flask project is written in Python3, uses the Flask framework -and relies on Apache to handle the HTTP request and load the Flask module. +and relies on Apache to handle the HTTP request and load the Flask module. The goal of the development installation is to have Apache execute the Python code straight from code repository. -Installation is currently laid out well in the `README.md` of the project, +Installation is currently laid out well in the `README.md` of the project, but repeating here... ```bash @@ -623,14 +623,14 @@ $ systemctl restart httpd ### FTS Server -The FTS3 Server is a C++ daemon, using CMake for compilation and installation. +The FTS3 Server is a C++ daemon, using CMake for compilation and installation. The software must be compiled and installed in order to deploy the latest changes. #### Notes on compiler version Building the FTS3 Server requires a modern compiler with support for C++17 features. On older platforms, such as CC7, the default compiler is too old. To be able to compile, -one must use a modern development toolset. +one must use a modern development toolset. Example for CC7: ```bash @@ -786,7 +786,7 @@ $ cp -v conf/fts3firewalld/ftsmon.xml /usr/lib/firewalld/services/ $ vim /etc/fts3web/fts3web.ini # Ensures empty ssl.conf -$ if [ -f /etc/httpd/conf.d/ssl.conf ] ; then +$ if [ -f /etc/httpd/conf.d/ssl.conf ] ; then echo -n > /etc/httpd/conf.d/ssl.conf fi ``` @@ -794,7 +794,7 @@ fi ### Gfal2 (bonus) Gfal2 is a low-level C/C++ client library to interact with Grid Storages. Using a plugin -architecture, it supports multiple protocols. The Gfal2 build system is managed via CMake. +architecture, it supports multiple protocols. The Gfal2 build system is managed via CMake. Given the close relation between FTS and Gfal2, it's useful to have the installation detailed here. ```bash -- GitLab