JAliEn setup merge requestshttps://gitlab.cern.ch/jalien/jalien-setup/-/merge_requests2024-02-16T20:06:42+01:00https://gitlab.cern.ch/jalien/jalien-setup/-/merge_requests/24Update table definition and add config file to allow jobs to run without cont...2024-02-16T20:06:42+01:00Maria-Elena MihailescuUpdate table definition and add config file to allow jobs to run without containersSigned-off-by: Elena Mihailescu <maria.mihailescu@upb.ro>Signed-off-by: Elena Mihailescu <maria.mihailescu@upb.ro>https://gitlab.cern.ch/jalien/jalien-setup/-/merge_requests/23Implement new container setup2024-02-02T13:50:00+01:00Kalana WijethungaImplement new container setupThis PR redesign the current container setup to be in sync with the original JAliEn GridThis PR redesign the current container setup to be in sync with the original JAliEn Gridhttps://gitlab.cern.ch/jalien/jalien-setup/-/merge_requests/22Upgrade jalien-setup to Ubuntu22.042024-01-11T21:50:08+01:00Maria-Elena MihailescuUpgrade jalien-setup to Ubuntu22.04This PR upgrades the `jalien-setup` infrastructure. Moving the `jalien-base` image from `ubuntu:18.04` to `ubuntu:22.04` updated multiple services (mysql was upgraded to 8, ldap to OpenLDAP2.5, XRootD and HTCondor were upgraded as well)....This PR upgrades the `jalien-setup` infrastructure. Moving the `jalien-base` image from `ubuntu:18.04` to `ubuntu:22.04` updated multiple services (mysql was upgraded to 8, ldap to OpenLDAP2.5, XRootD and HTCondor were upgraded as well).
Some of the major changes this PR brings:
- Images: update `jalien-base` to `ubuntu:22.04` (Ubuntu 22.04), schedd to `htcondor/cm:23.2.0-el8` (AlmaLinux 8), workers to `htcondor/execute:23.2.0-el8` (AlmaLinux 8)
- MySQL (8): split set database password from create procedures/functions; replace `GRANT ALL PRIVILEGES ON` with `GRANT ALL ON`
- LDAP: update `ldapadd` call to the corresponding syntax for OpenLDAP2.5 (replace `-h` & `-p` with `-H` and URI); replace HDB with MDB module
- XRootD: update to `xrootd-5.6.0`
- HTCondor: set `CE` as submitter, `schedd` as CentralManagement and `workers` as executors, add custom config for submitter, add possibility to install all EL8 packages for worker nodes
- JCentral: add `-n` to `entr`
- `optimiser.sh`: add jalien user to `QUEUE_USER`
- docker compose: add health check for CE
- Makefile: build base image with `--progress=plain` to better spot errors
Signed-off-by: Elena Mihailescu <maria.mihailescu@upb.ro>https://gitlab.cern.ch/jalien/jalien-setup/-/merge_requests/21Copy CERT-GridCA.pem in $X509_CERT_DIR2023-11-22T11:24:20+01:00Maria-Elena MihailescuCopy CERT-GridCA.pem in $X509_CERT_DIRIf `CERT-GridCA.pem` is not found in `$X509_CERT_DIR`,
`alien.py` ignores it.
Signed-off-by: Elena Mihailescu <maria.mihailescu@upb.ro>If `CERT-GridCA.pem` is not found in `$X509_CERT_DIR`,
`alien.py` ignores it.
Signed-off-by: Elena Mihailescu <maria.mihailescu@upb.ro>https://gitlab.cern.ch/jalien/jalien-setup/-/merge_requests/20Fix partition in JOBAGENT entry and log all mysql queries2023-11-20T13:12:07+01:00Maria-Elena MihailescuFix partition in JOBAGENT entry and log all mysql queriesWhen creating a JOBAGENT using the `optimiser.sh` script, the partition is set to `,,`, and the jobs are not scheduled. This PR replaces `,,` with `%`.
Additionally, for further debugging purposes, `optimiser.sh` contains two `mysql` co...When creating a JOBAGENT using the `optimiser.sh` script, the partition is set to `,,`, and the jobs are not scheduled. This PR replaces `,,` with `%`.
Additionally, for further debugging purposes, `optimiser.sh` contains two `mysql` commands to activate query logging. Default, these two commands are commented.
Signed-off-by: Elena Mihailescu <maria.mihailescu@upb.ro>https://gitlab.cern.ch/jalien/jalien-setup/-/merge_requests/19Add test entries in PRIORITY and SITEQUEUES2023-09-04T11:30:21+02:00Maria-Elena MihailescuAdd test entries in PRIORITY and SITEQUEUESThese entries are needed when testing job resubmission.
Signed-off-by: Elena Mihailescu <maria.mihailescu@upb.ro>These entries are needed when testing job resubmission.
Signed-off-by: Elena Mihailescu <maria.mihailescu@upb.ro>https://gitlab.cern.ch/jalien/jalien-setup/-/merge_requests/18Add QUEUE_CPU in jalien-setup2023-08-23T09:19:54+02:00Maria-Elena MihailescuAdd QUEUE_CPU in jalien-setupSigned-off-by: Elena Mihailescu <maria.mihailescu@upb.ro>Signed-off-by: Elena Mihailescu <maria.mihailescu@upb.ro>https://gitlab.cern.ch/jalien/jalien-setup/-/merge_requests/17Change column type for queueId & update QUEUEPROC structure2023-08-08T11:14:01+02:00Maria-Elena MihailescuChange column type for queueId & update QUEUEPROC structureChange column type for `queueId` from `int` to `bigint(20)` and update `QUEUEPROC` structure.
Signed-off-by: Elena Mihailescu <maria.mihailescu@upb.ro>Change column type for `queueId` from `int` to `bigint(20)` and update `QUEUEPROC` structure.
Signed-off-by: Elena Mihailescu <maria.mihailescu@upb.ro>https://gitlab.cern.ch/jalien/jalien-setup/-/merge_requests/16Update htcondor docker image version and revert the job agent from optimiser.sh2023-03-20T16:06:10+01:00Maria-Elena MihailescuUpdate htcondor docker image version and revert the job agent from optimiser.shThis PR brings two contributions:
- it brings back the previous Job Agent since the current site sonar constraints do not match the running workers. If one needs to test the site sonar constraints feature, one needs to edit the "optimise...This PR brings two contributions:
- it brings back the previous Job Agent since the current site sonar constraints do not match the running workers. If one needs to test the site sonar constraints feature, one needs to edit the "optimiser.sh" file and uncomment/update the commented line.
- it updates the htcondor docker image to a newer one (from `8.9.6-el7` to `10.0.0-el7`). The previous htcondor image was 3 years old and generated the following errors each 5 minutes:
```
03/10/23 19:42:00 SECMAN: FAILED: Received "DENIED" from server for user condor_pool@localhost.localdomain using method PASSWORD.
03/10/23 19:42:00 ERROR: SECMAN:2010:Received "DENIED" from server for user condor_pool@localhost.localdomain using method PASSWORD.
03/10/23 19:42:00 Failed to start non-blocking update to <192.168.0.5:9618>.
```
I've tested the proposed configuration as follows:
- build and start the infrastructure
- run `./optimiser.sh` on `JCentral-dev_1`
- submit a test job
- check that the job is executed
- check that htcondor (`worker1_1`, `worker2_1` and `JCentral-dev-CE_1`) does not generates the previously mentioned error.
Signed-off-by: Elena Mihailescu <maria.mihailescu@upb.ro>https://gitlab.cern.ch/jalien/jalien-setup/-/merge_requests/15Integrate Site Sonar with the replica setup2022-12-08T13:50:37+01:00Kalana WijethungaIntegrate Site Sonar with the replica setupThis PR integrates the latest site sonar version with the replica setupThis PR integrates the latest site sonar version with the replica setupMaksim Melnik StoretvedtMaksim Melnik Storetvedthttps://gitlab.cern.ch/jalien/jalien-setup/-/merge_requests/14Add healthchecks for JCentral-dev, dependencies between containers and wait f...2022-06-16T09:32:23+02:00Maria-Elena MihailescuAdd healthchecks for JCentral-dev, dependencies between containers and wait function for CreateLDAP.shThis PR adds the following:
- health checks for JCentral: check if mySQL and LDAP started
- dependencies between containers: CE and SE does not start until JCentral is considered healthy
- wait function in `bash-setup/CreateLDAP.sh`: the...This PR adds the following:
- health checks for JCentral: check if mySQL and LDAP started
- dependencies between containers: CE and SE does not start until JCentral is considered healthy
- wait function in `bash-setup/CreateLDAP.sh`: the previous way of waiting for the LDAP service to fully start is to `sleep` for 2 seconds. However, sometimes, this does not work since the LDAP services takes more time to start. Thus, I've replace the `sleep 2` command with a function that waits until LDAP is fully functional before continuing the setup.
Signed-off-by: Elena Mihailescu <maria-elena.mihailescu@cern.ch>https://gitlab.cern.ch/jalien/jalien-setup/-/merge_requests/13Move .j to .alien2022-06-07T17:03:30+02:00Maksim Melnik StoretvedtMove .j to .alienMaksim Melnik StoretvedtMaksim Melnik Storetvedthttps://gitlab.cern.ch/jalien/jalien-setup/-/merge_requests/12Changed the engine of INDEXTABLE to InnoDB2022-04-06T16:54:54+02:00Iuliana BrinzoiChanged the engine of INDEXTABLE to InnoDBThis allows the table to support transactions, rolling back in case of an error.This allows the table to support transactions, rolling back in case of an error.https://gitlab.cern.ch/jalien/jalien-setup/-/merge_requests/11[mySQL] Add missing fields for processes.SITEQUEUES2022-04-22T14:17:51+02:00Maria-Elena Mihailescu[mySQL] Add missing fields for processes.SITEQUEUESSigned-off-by: Elena Mihailescu <maria.mihailescu@upb.ro>
Added orphan_pfns and orphan_pfns_0 to all databases and missing fields from the processes.SITEQUEUES table as per @grigoras suggestion.
This PR solves the mySQL errors from jcen...Signed-off-by: Elena Mihailescu <maria.mihailescu@upb.ro>
Added orphan_pfns and orphan_pfns_0 to all databases and missing fields from the processes.SITEQUEUES table as per @grigoras suggestion.
This PR solves the mySQL errors from jcentral-dev.log.https://gitlab.cern.ch/jalien/jalien-setup/-/merge_requests/10Added INDEXTABLE_UPDATE table2022-01-13T09:14:31+01:00Iuliana BrinzoiAdded INDEXTABLE_UPDATE tableAdded INDEXTABLE_UPDATE table that stores the timestamp when the
INDEXTABLE has last been updated. It can be used to determine
if an update of the INDEXTABLE cache is required.Added INDEXTABLE_UPDATE table that stores the timestamp when the
INDEXTABLE has last been updated. It can be used to determine
if an update of the INDEXTABLE cache is required.https://gitlab.cern.ch/jalien/jalien-setup/-/merge_requests/9Centos container2021-11-25T14:04:44+01:00Haakon Andre Reme-NessCentos containerChanged distro to Centos 7 (JCental running ubuntu because of mysql)Changed distro to Centos 7 (JCental running ubuntu because of mysql)https://gitlab.cern.ch/jalien/jalien-setup/-/merge_requests/8Gsoc2020 phase3 integration2020-08-21T15:44:03+02:00Nikola HardiGsoc2020 phase3 integrationhttps://gitlab.cern.ch/jalien/jalien-setup/-/merge_requests/7Split certs setup2020-08-10T13:04:36+02:00Nikola HardiSplit certs setupKeep the credentials in separate subdirectories.Keep the credentials in separate subdirectories.https://gitlab.cern.ch/jalien/jalien-setup/-/merge_requests/6GSoC2020 phase 2 integration: XRootD based storage element2020-07-29T09:56:40+02:00Nikola HardiGSoC2020 phase 2 integration: XRootD based storage element- XRootD container and setup scripts
- Catalog databases updated
- More CI fixes- XRootD container and setup scripts
- Catalog databases updated
- More CI fixeshttps://gitlab.cern.ch/jalien/jalien-setup/-/merge_requests/5Override TJAlien upstream for the moment2020-07-28T19:03:29+02:00Nikola HardiOverride TJAlien upstream for the moment