Troubles with the cta kubernetes environment

assigned to @jleduc

added Continuous integration in CTA-old label

Is there a way we could ssh on your openstack VM?

Maybe the easiest would be to add some of us in /root/.k5login?

I've added you, you should be able to login. The machine is called ctadevmiguel

Looks like you need a deep cleanup of your docker subsystem:

[root@ctadevmiguel ~]# docker system df
TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              10                  5                   3.061 GB            2.798 GB (91%)
Containers          23                  21                  916.2 MB            30.27 kB (0%)
Local Volumes       15                  11                  69.37 MB            0 B (0%)``

Around 4GB should be taken by docker but at the OS level it is more than 22GB...

I have already seen this on our gitlab runners and I had to clean them at one point.

I propose that you delete your kubernetes namespace when you are done for today and I can try to clean you docker subsystem.

And you give some feedback tomorrow morning when you recreate your namespace.

Hi @jleduc

Thanks for the help.

I've run this script to clean everything up:

cd ~/CTA/continuousintegration/orchestration/;   sudo ./delete_instance.sh -n cta -D
sudo docker rm -f $(sudo docker ps -a -q)
sudo docker rmi -f $(sudo docker images -q)
sudo docker volume rm -f $(sudo docker volume ls -q)

Hi @mvelosob, did you change CTA/continuousintegration/orchestration/tests/client_ar.sh by any chance? Because in the output it is referring to line 479 in this file (scroll to the right):

Cancelling prepare for files in /eos/ctaeos/preprod/ae6d3b9a-6ec4-4f32-b37b-59a16055f998/0 using 100 processes (prepare_abort).../root/client_ar.sh: line 479: [: -ne: unary operator expected

Ah, wait, the problem is that ${requestsTotal} is not set. This is the result of this command returning nothing:

requestsTotal=`admin_cta --json sq | jq 'map(select (.mountType == "RETRIEVE") | .queuedFiles | tonumber) | add'`

I changed no script, the error was already present when everything was working. Probably need to call wc -l with the output of admin_cta --json sq | jq 'map(select (.mountType == "RETRIEVE") | .queuedFiles | tonumber) | add' :)

OK I had a go at it this morning.

Some minor PV cleanup issues

When finished with a run and deleting the namespace discarding the logs with: ./delete_instance.sh -n <namespace> -D leaves plenty of files in logs and stg pvs. These must be manually cleaned before spawning a new namespace.

These commands will clean up all unused logs and stg persistent volumes:

kubectl get pv -o json |jq -r '.items[] |select(.status.phase != "Bound") | select(.metadata.labels.type == "logs") | .spec.hostPath.path' | xargs -itoto bash -c 'rm -fr toto/* toto/.*'
kubectl get pv -o json |jq -r '.items[] |select(.status.phase != "Bound") | select(.metadata.labels.type == "stg") | .spec.hostPath.path' | xargs -itoto bash -c 'rm -fr toto/* toto/.*'

After this all pvs should be in available status and you should be able to start the next environment with no leftover (at least from this side).

Specific issues if `cta-frontend` crashes

When instantiating CTA@4907767543b7860cf64eece3d315dfa32235b3e5 with: ./run_systemtest.sh -n toto -s tests/archive_retrieve.sh -d internal_postgres.yaml

Running cta-admin dr ls crashes the cta frontend:

[root@ctadevmiguel orchestration]# kubectl -n toto exec ctacli -- cta-admin dr ls
210907 11:18:30 604 ssi_Pb::Request: pid:591 tid:140200244033280 ProcessResponseData(): fatal error from XRootD framework
[ERROR] Socket error
library drive host desired request status since vid tapepool vo files data MB/s session priority activity age reason disk_system_name reserved_bytes 
210907 11:18:30 604 ssi_Shutdown: Unprovision: /ctafrontend@ctafrontend.toto.svc.cluster.local:10955 error; 14 [ERROR] Invalid session

I reproduced it on 2 different kubernetes boxes on the same commit.

Frontend crashes, generates core dumps and backtrace automatically in the logs persistent volume allocated for the namespace. This should not be small amount of data and the logs pv has a max size of 2GiB.

Later on, all xrdcp from client pod will fail and each of the 10k client archive, 10k client retrieve will redirect Dump level log to ERROR_DIR defined like this:

# Create directory for xrootd error reports
ERROR_DIR="/dev/shm/$(basename ${EOS_DIR})"

Its run location is in the logs of the run/create instance command: echo "$(date +%s): ERROR_DIR=${ERROR_DIR}". The second issue with this is that /dev/shm is in memory: Dump level messages can eat quite a lot of space there especially if the frontend crashes early: the earlier the crash the more error dump messages, the more space consumed in /dev/shm.

At this point this is quite bad but kubernetes doesn't make any difference: the frontend crashed and logging as hard as CI would kill anything...

We could fix client_ar.sh to stop sending too many ERROR messages in /dev/shm: Its goal is to capture and save client side error messages on the flight but if all message fail because somethis is broken that badly... There is no point going further...

I will have a loot at what makes sense here.