Unkown harbor replication delay
User complained that GPN->TN replication took 40 minutes.
registry.cern.ch:
registry-tn.cern.ch:
From prometheus metrics, everything looks good. Some delay but with a maximum of 4minutes...
DB logs for registry.cern.ch, look weird tho:
SELECT log_id, job_uuid, creation_time, content
FROM public.job_log
WHERE content LIKE '%pma-dpoc-orbits-analysis-module%'
ORDER BY log_id DESC LIMIT 100
"log_id" "job_uuid" "creation_time" "content"
6290861 "9cb967dc09093afa5ff1d47f" "2023-02-28 12:28:13.887848" "2023-02-28T12:28:13Z [INFO] [/controller/replication/transfer/image/transfer.go:125]: client for source registry [type: harbor, URL: http://harbor-core:80, insecure: true] created
2023-02-28T12:28:13Z [INFO] [/controller/replication/transfer/image/transfer.go:135]: client for destination registry [type: harbor, URL: https://registry-tn.cern.ch, insecure: false] created
2023-02-28T12:28:13Z [INFO] [/controller/replication/transfer/image/transfer.go:168]: copying acc/pma-dpoc-orbits-analysis-module:[latest](source registry) to acc/pma-dpoc-orbits-analysis-module:[latest](destination registry)...
2023-02-28T12:28:13Z [INFO] [/controller/replication/transfer/image/transfer.go:196]: copying acc/pma-dpoc-orbits-analysis-module:latest(source registry) to acc/pma-dpoc-orbits-analysis-module:latest(destination registry)...
2023-02-28T12:28:13Z [INFO] [/controller/replication/transfer/image/transfer.go:343]: pulling the manifest of artifact acc/pma-dpoc-orbits-analysis-module:latest ...
2023-02-28T12:28:13Z [INFO] [/controller/replication/transfer/image/transfer.go:349]: the manifest of artifact acc/pma-dpoc-orbits-analysis-module:latest pulled
2023-02-28T12:28:13Z [INFO] [/controller/replication/transfer/image/transfer.go:212]: the artifact acc/pma-dpoc-orbits-analysis-module:latest already exists on the destination registry, skip
2023-02-28T12:28:13Z [INFO] [/controller/replication/transfer/image/transfer.go:190]: copy acc/pma-dpoc-orbits-analysis-module:[latest](source registry) to acc/pma-dpoc-orbits-analysis-module:[latest](destination registry) completed
"
6290860 "5f1dc6fa0990c0c5d5af2f64" "2023-02-28 12:28:13.889858" "2023-02-28T12:28:13Z [INFO] [/controller/replication/transfer/image/transfer.go:125]: client for source registry [type: harbor, URL: http://harbor-core:80, insecure: true] created
2023-02-28T12:28:13Z [INFO] [/controller/replication/transfer/image/transfer.go:135]: client for destination registry [type: harbor, URL: https://registry-tn.cern.ch, insecure: false] created
2023-02-28T12:28:13Z [INFO] [/controller/replication/transfer/image/transfer.go:168]: copying acc/pma-dpoc-orbits-analysis-module:[latest](source registry) to acc/pma-dpoc-orbits-analysis-module:[latest](destination registry)...
2023-02-28T12:28:13Z [INFO] [/controller/replication/transfer/image/transfer.go:196]: copying acc/pma-dpoc-orbits-analysis-module:latest(source registry) to acc/pma-dpoc-orbits-analysis-module:latest(destination registry)...
2023-02-28T12:28:13Z [INFO] [/controller/replication/transfer/image/transfer.go:343]: pulling the manifest of artifact acc/pma-dpoc-orbits-analysis-module:latest ...
2023-02-28T12:28:13Z [INFO] [/controller/replication/transfer/image/transfer.go:349]: the manifest of artifact acc/pma-dpoc-orbits-analysis-module:latest pulled
2023-02-28T12:28:13Z [INFO] [/controller/replication/transfer/image/transfer.go:212]: the artifact acc/pma-dpoc-orbits-analysis-module:latest already exists on the destination registry, skip
2023-02-28T12:28:13Z [INFO] [/controller/replication/transfer/image/transfer.go:190]: copy acc/pma-dpoc-orbits-analysis-module:[latest](source registry) to acc/pma-dpoc-orbits-analysis-module:[latest](destination registry) completed
"
6289799 "d18d313cbde64b786154546d" "2023-02-28 11:33:34.397966" "2023-02-28T11:32:22Z [INFO] [/controller/replication/transfer/image/transfer.go:125]: client for source registry [type: harbor, URL: http://harbor-core:80, insecure: true] created
2023-02-28T11:32:22Z [INFO] [/controller/replication/transfer/image/transfer.go:135]: client for destination registry [type: harbor, URL: https://registry-tn.cern.ch, insecure: false] created
2023-02-28T11:32:22Z [INFO] [/controller/replication/transfer/image/transfer.go:168]: copying acc/pma-dpoc-orbits-analysis-module:[latest](source registry) to acc/pma-dpoc-orbits-analysis-module:[latest](destination registry)...
2023-02-28T11:32:22Z [INFO] [/controller/replication/transfer/image/transfer.go:196]: copying acc/pma-dpoc-orbits-analysis-module:latest(source registry) to acc/pma-dpoc-orbits-analysis-module:latest(destination registry)...
2023-02-28T11:32:22Z [INFO] [/controller/replication/transfer/image/transfer.go:343]: pulling the manifest of artifact acc/pma-dpoc-orbits-analysis-module:latest ...
2023-02-28T11:32:23Z [INFO] [/controller/replication/transfer/image/transfer.go:349]: the manifest of artifact acc/pma-dpoc-orbits-analysis-module:latest pulled
2023-02-28T11:32:23Z [WARNING] [/controller/replication/transfer/image/transfer.go:223]: the same name artifact acc/pma-dpoc-orbits-analysis-module:latest exists on the destination registry and the ""override"" is set to true, continue...
2023-02-28T11:32:23Z [INFO] [/controller/replication/transfer/image/transfer.go:270]: copying the blob sha256:9d42d89b54a694c641d4cae19ce08bfda08916a44c8a0555b7ad265b28a7458f(the 1th running)...
2023-02-28T11:32:23Z [INFO] [/controller/replication/transfer/image/transfer.go:296]: the blob sha256:9d42d89b54a694c641d4cae19ce08bfda08916a44c8a0555b7ad265b28a7458f already exists on the destination registry, skip
2023-02-28T11:32:23Z [INFO] [/controller/replication/transfer/image/transfer.go:272]: copy the blob sha256:9d42d89b54a694c641d4cae19ce08bfda08916a44c8a0555b7ad265b28a7458f completed
2023-02-28T11:32:23Z [INFO] [/controller/replication/transfer/image/transfer.go:270]: copying the blob sha256:9d523c5f8c040f981ec28fcc9260d5035de78500fe2f2270f473c55b60135642(the 1th running)...
2023-02-28T11:32:23Z [INFO] [/controller/replication/transfer/image/transfer.go:296]: the blob sha256:9d523c5f8c040f981ec28fcc9260d5035de78500fe2f2270f473c55b60135642 already exists on the destination registry, skip
2023-02-28T11:32:23Z [INFO] [/controller/replication/transfer/image/transfer.go:272]: copy the blob sha256:9d523c5f8c040f981ec28fcc9260d5035de78500fe2f2270f473c55b60135642 completed
2023-02-28T11:32:23Z [INFO] [/controller/replication/transfer/image/transfer.go:270]: copying the blob sha256:e465150e4b9f8e3af011c0f5fea25e882b8de9001465eb9baab887a7045dfca1(the 1th running)...
2023-02-28T11:32:43Z [INFO] [/controller/replication/transfer/image/transfer.go:272]: copy the blob sha256:e465150e4b9f8e3af011c0f5fea25e882b8de9001465eb9baab887a7045dfca1 completed
2023-02-28T11:32:43Z [INFO] [/controller/replication/transfer/image/transfer.go:270]: copying the blob sha256:34d7a9bf03535fc2cff3094d9c0cc20afa8f557d09801ff49e70230e96440db9(the 1th running)...
2023-02-28T11:33:28Z [INFO] [/controller/replication/transfer/image/transfer.go:272]: copy the blob sha256:34d7a9bf03535fc2cff3094d9c0cc20afa8f557d09801ff49e70230e96440db9 completed
2023-02-28T11:33:28Z [INFO] [/controller/replication/transfer/image/transfer.go:270]: copying the blob sha256:1f0e907a17ea9b315120b3d9a16d56a49857eeecb8380a5617358fc278729bfa(the 1th running)...
2023-02-28T11:33:33Z [INFO] [/controller/replication/transfer/image/transfer.go:296]: the blob sha256:1f0e907a17ea9b315120b3d9a16d56a49857eeecb8380a5617358fc278729bfa already exists on the destination registry, skip
2023-02-28T11:33:33Z [INFO] [/controller/replication/transfer/image/transfer.go:272]: copy the blob sha256:1f0e907a17ea9b315120b3d9a16d56a49857eeecb8380a5617358fc278729bfa completed
2023-02-28T11:33:33Z [INFO] [/controller/replication/transfer/image/transfer.go:270]: copying the blob sha256:6302f71f9c2cd1b3378864aa2153c7cb9922bd2fa3a0803692e8ce42fd91041b(the 1th running)...
2023-02-28T11:33:33Z [INFO] [/controller/replication/transfer/image/transfer.go:296]: the blob sha256:6302f71f9c2cd1b3378864aa2153c7cb9922bd2fa3a0803692e8ce42fd91041b already exists on the destination registry, skip
2023-02-28T11:33:33Z [INFO] [/controller/replication/transfer/image/transfer.go:272]: copy the blob sha256:6302f71f9c2cd1b3378864aa2153c7cb9922bd2fa3a0803692e8ce42fd91041b completed
2023-02-28T11:33:33Z [INFO] [/controller/replication/transfer/image/transfer.go:372]: pushing the manifest of artifact acc/pma-dpoc-orbits-analysis-module:latest ...
2023-02-28T11:33:34Z [INFO] [/controller/replication/transfer/image/transfer.go:384]: the manifest of artifact acc/pma-dpoc-orbits-analysis-module:latest pushed
2023-02-28T11:33:34Z [INFO] [/controller/replication/transfer/image/transfer.go:239]: copy acc/pma-dpoc-orbits-analysis-module:latest(source registry) to acc/pma-dpoc-orbits-analysis-module:latest(destination registry) completed
2023-02-28T11:33:34Z [INFO] [/controller/replication/transfer/image/transfer.go:190]: copy acc/pma-dpoc-orbits-analysis-module:[latest](source registry) to acc/pma-dpoc-orbits-analysis-module:[latest](destination registry) completed
"
6289798 "bfa1bb70410401fe25cd78fc" "2023-02-28 11:33:24.929852" "2023-02-28T11:32:22Z [INFO] [/controller/replication/transfer/image/transfer.go:125]: client for source registry [type: harbor, URL: http://harbor-core:80, insecure: true] created
2023-02-28T11:32:22Z [INFO] [/controller/replication/transfer/image/transfer.go:135]: client for destination registry [type: harbor, URL: https://registry-tn.cern.ch, insecure: false] created
2023-02-28T11:32:22Z [INFO] [/controller/replication/transfer/image/transfer.go:168]: copying acc/pma-dpoc-orbits-analysis-module:[latest](source registry) to acc/pma-dpoc-orbits-analysis-module:[latest](destination registry)...
2023-02-28T11:32:22Z [INFO] [/controller/replication/transfer/image/transfer.go:196]: copying acc/pma-dpoc-orbits-analysis-module:latest(source registry) to acc/pma-dpoc-orbits-analysis-module:latest(destination registry)...
2023-02-28T11:32:22Z [INFO] [/controller/replication/transfer/image/transfer.go:343]: pulling the manifest of artifact acc/pma-dpoc-orbits-analysis-module:latest ...
2023-02-28T11:32:23Z [INFO] [/controller/replication/transfer/image/transfer.go:349]: the manifest of artifact acc/pma-dpoc-orbits-analysis-module:latest pulled
2023-02-28T11:32:23Z [WARNING] [/controller/replication/transfer/image/transfer.go:223]: the same name artifact acc/pma-dpoc-orbits-analysis-module:latest exists on the destination registry and the ""override"" is set to true, continue...
2023-02-28T11:32:23Z [INFO] [/controller/replication/transfer/image/transfer.go:270]: copying the blob sha256:9d42d89b54a694c641d4cae19ce08bfda08916a44c8a0555b7ad265b28a7458f(the 1th running)...
2023-02-28T11:32:23Z [INFO] [/controller/replication/transfer/image/transfer.go:296]: the blob sha256:9d42d89b54a694c641d4cae19ce08bfda08916a44c8a0555b7ad265b28a7458f already exists on the destination registry, skip
2023-02-28T11:32:23Z [INFO] [/controller/replication/transfer/image/transfer.go:272]: copy the blob sha256:9d42d89b54a694c641d4cae19ce08bfda08916a44c8a0555b7ad265b28a7458f completed
2023-02-28T11:32:23Z [INFO] [/controller/replication/transfer/image/transfer.go:270]: copying the blob sha256:9d523c5f8c040f981ec28fcc9260d5035de78500fe2f2270f473c55b60135642(the 1th running)...
2023-02-28T11:32:23Z [INFO] [/controller/replication/transfer/image/transfer.go:296]: the blob sha256:9d523c5f8c040f981ec28fcc9260d5035de78500fe2f2270f473c55b60135642 already exists on the destination registry, skip
2023-02-28T11:32:23Z [INFO] [/controller/replication/transfer/image/transfer.go:272]: copy the blob sha256:9d523c5f8c040f981ec28fcc9260d5035de78500fe2f2270f473c55b60135642 completed
2023-02-28T11:32:23Z [INFO] [/controller/replication/transfer/image/transfer.go:270]: copying the blob sha256:e465150e4b9f8e3af011c0f5fea25e882b8de9001465eb9baab887a7045dfca1(the 1th running)...
2023-02-28T11:32:40Z [INFO] [/controller/replication/transfer/image/transfer.go:272]: copy the blob sha256:e465150e4b9f8e3af011c0f5fea25e882b8de9001465eb9baab887a7045dfca1 completed
2023-02-28T11:32:40Z [INFO] [/controller/replication/transfer/image/transfer.go:270]: copying the blob sha256:34d7a9bf03535fc2cff3094d9c0cc20afa8f557d09801ff49e70230e96440db9(the 1th running)...
2023-02-28T11:33:22Z [INFO] [/controller/replication/transfer/image/transfer.go:272]: copy the blob sha256:34d7a9bf03535fc2cff3094d9c0cc20afa8f557d09801ff49e70230e96440db9 completed
2023-02-28T11:33:22Z [INFO] [/controller/replication/transfer/image/transfer.go:270]: copying the blob sha256:1f0e907a17ea9b315120b3d9a16d56a49857eeecb8380a5617358fc278729bfa(the 1th running)...
2023-02-28T11:33:24Z [INFO] [/controller/replication/transfer/image/transfer.go:272]: copy the blob sha256:1f0e907a17ea9b315120b3d9a16d56a49857eeecb8380a5617358fc278729bfa completed
2023-02-28T11:33:24Z [INFO] [/controller/replication/transfer/image/transfer.go:270]: copying the blob sha256:6302f71f9c2cd1b3378864aa2153c7cb9922bd2fa3a0803692e8ce42fd91041b(the 1th running)...
2023-02-28T11:33:24Z [INFO] [/controller/replication/transfer/image/transfer.go:296]: the blob sha256:6302f71f9c2cd1b3378864aa2153c7cb9922bd2fa3a0803692e8ce42fd91041b already exists on the destination registry, skip
2023-02-28T11:33:24Z [INFO] [/controller/replication/transfer/image/transfer.go:272]: copy the blob sha256:6302f71f9c2cd1b3378864aa2153c7cb9922bd2fa3a0803692e8ce42fd91041b completed
2023-02-28T11:33:24Z [INFO] [/controller/replication/transfer/image/transfer.go:372]: pushing the manifest of artifact acc/pma-dpoc-orbits-analysis-module:latest ...
2023-02-28T11:33:24Z [INFO] [/controller/replication/transfer/image/transfer.go:384]: the manifest of artifact acc/pma-dpoc-orbits-analysis-module:latest pushed
2023-02-28T11:33:24Z [INFO] [/controller/replication/transfer/image/transfer.go:239]: copy acc/pma-dpoc-orbits-analysis-module:latest(source registry) to acc/pma-dpoc-orbits-analysis-module:latest(destination registry) completed
2023-02-28T11:33:24Z [INFO] [/controller/replication/transfer/image/transfer.go:190]: copy acc/pma-dpoc-orbits-analysis-module:[latest](source registry) to acc/pma-dpoc-orbits-analysis-module:[latest](destination registry) completed
"
SELECT jl.log_id, jl.job_uuid, jl.creation_time, jl.content
FROM public.job_log AS jl
WHERE jl.content LIKE '%pma-dpoc-orbits-analysis-module%' AND
jl.content LIKE '%replication%'
ORDER BY jl.log_id DESC LIMIT 10;
"log_id" "job_uuid" "creation_time"
6291853 "97f5bd253ed68818c8b1ed3a" "2023-02-28 13:40:25.020341"
6291852 "1fab5b53538176a42d32ef74" "2023-02-28 13:40:24.845721"
6290861 "9cb967dc09093afa5ff1d47f" "2023-02-28 12:28:13.887848"
6290860 "5f1dc6fa0990c0c5d5af2f64" "2023-02-28 12:28:13.889858"
6289799 "d18d313cbde64b786154546d" "2023-02-28 11:33:34.397966"
6289798 "bfa1bb70410401fe25cd78fc" "2023-02-28 11:33:24.929852"
6288727 "fd32b82cf60cce9b6d5db0c9" "2023-02-28 10:52:59.792441"
6288703 "cffd893d5d25f3a3dd16cfb9" "2023-02-28 10:52:59.921696"
6288582 "23ea394e50f8713895cc4dd5" "2023-02-28 10:23:10.297311"
6288581 "fe6637ba33a8948cd7c32a68" "2023-02-28 10:23:10.274004"
Edited by Diogo Filipe Tomas Guerra