oais-platform issueshttps://gitlab.cern.ch/digitalmemory/oais-platform/-/issues2024-03-26T16:31:46+01:00https://gitlab.cern.ch/digitalmemory/oais-platform/-/issues/141Add New CDS as source2024-03-26T16:31:46+01:00Panna Georgina LiptakAdd New CDS as sourcePanna Georgina LiptakPanna Georgina Liptakhttps://gitlab.cern.ch/digitalmemory/oais-platform/-/issues/140Periodic task max limit2024-02-27T16:00:03+01:00Panna Georgina LiptakPeriodic task max limitFound an issue on preserve-qa where 4 periodic `check_am_status` tasks were running for multiple days.
Possible solutions:
- Limit the number of repetitions for a periodic task (consider the scenario where we are waiting for AM to proc...Found an issue on preserve-qa where 4 periodic `check_am_status` tasks were running for multiple days.
Possible solutions:
- Limit the number of repetitions for a periodic task (consider the scenario where we are waiting for AM to process a bigger batch - difficult to estimate - add possibility for manual trigger?)
- For admin users add a tab to see all periodic tasks (it's a postgresql table - oais_db.django_celery_beat_periodictask) and delete them potentially.
Manual workaround:
- Go to postgresql pod terminal
- `psql`
- `\c oais_db` (connect to DB)
- `select * from django_celery_beat_periodictask;` (list active periodic tasks)
- `delete from django_celery_beat_periodictask;` (where ...) (remove them if they are not needed)
- Restart celery podPanna Georgina LiptakPanna Georgina Liptakhttps://gitlab.cern.ch/digitalmemory/oais-platform/-/issues/139Sort out permissions across all components2024-02-27T16:00:56+01:00Panna Georgina LiptakSort out permissions across all componentsPanna Georgina LiptakPanna Georgina Liptakhttps://gitlab.cern.ch/digitalmemory/oais-platform/-/issues/138Feature: Batch AM Upload2024-03-27T15:18:27+01:00Panna Georgina LiptakFeature: Batch AM UploadGoal: From the Archives page allow to select multiple records with state as SIP and Upload them all to Archivematica in a batch.
Code was merged with: https://gitlab.cern.ch/digitalmemory/oais-platform/-/issues/137
Changes: On the fro...Goal: From the Archives page allow to select multiple records with state as SIP and Upload them all to Archivematica in a batch.
Code was merged with: https://gitlab.cern.ch/digitalmemory/oais-platform/-/issues/137
Changes: On the frontend add option to select multiple entries on Archives page and if they are all SIPs the Upload all to AM is a possible action - call the already existing functionality for each.
Blocker: Archivematica sometimes gives "Stale file handle issue" and fails.
Workaround solution for the blocker:
Deploy 1 MCP client pod, increase the CPU request and limit, deploy multiple containers inside the same pod.
Code change:
- AM Task sometimes waits in queue between transfer and digest and AM returns status code 400 (Bad Request, because there is no active transfer or ingest for the given ID).
- In this scenario we check for already executed jobs for the task id and if there is any then we know it's in progress, but waiting in queue not failed (when a task is truly failed in return status FAILED).
Testing:
- When we start 10 AM upload at the same time usually this scenario occurs. We can verify from the logs.
- This approach prevented the process from failing 5 times during the test on dm-luteus
- ![image](/uploads/f577404d920409e86cf9a83edb22e2d8/image.png)
- ![image](/uploads/8979b0dbb303a3f793362c8f0f8c3ab2/image.png)Panna Georgina LiptakPanna Georgina Liptakhttps://gitlab.cern.ch/digitalmemory/oais-platform/-/issues/136Additional metadata for local files2023-08-28T11:04:56+02:00Antonio Vivaceantonio.vivace@cern.chAdditional metadata for local filesIf some local files are not uploaded to any proper digital repository (reasons?), we can think about providing a way to the user to add some basic metadata and add it to the SIP that will be created.If some local files are not uploaded to any proper digital repository (reasons?), we can think about providing a way to the user to add some basic metadata and add it to the SIP that will be created.https://gitlab.cern.ch/digitalmemory/oais-platform/-/issues/133Enable HTTPS for the dev nginx container2023-08-15T10:15:37+02:00Antonio Vivaceantonio.vivace@cern.chEnable HTTPS for the dev nginx containerThis may be required for e.g. serving the file picker widget
Make it optional/easy to disable/skip config when not neededThis may be required for e.g. serving the file picker widget
Make it optional/easy to disable/skip config when not neededhttps://gitlab.cern.ch/digitalmemory/oais-platform/-/issues/132Handle invalid response from codimd2023-07-12T11:48:08+02:00Antonio Vivaceantonio.vivace@cern.chHandle invalid response from codimdWhen searching / harvesting from Codimd, the token can expire/be invalid but a server error is raised.
Handle it more precisely reporting to the user an error while contacting codimd (offline, bad response, or token invalid?)When searching / harvesting from Codimd, the token can expire/be invalid but a server error is raised.
Handle it more precisely reporting to the user an error while contacting codimd (offline, bad response, or token invalid?)https://gitlab.cern.ch/digitalmemory/oais-platform/-/issues/130Imcomplete/inaccurate routes annotations2023-07-04T15:39:09+02:00Antonio Vivaceantonio.vivace@cern.chImcomplete/inaccurate routes annotationsThere are some routes for which it may be worth think better on how to annotate/add an explicit serializer for the responses they can return (e.g. see /logout)There are some routes for which it may be worth think better on how to annotate/add an explicit serializer for the responses they can return (e.g. see /logout)https://gitlab.cern.ch/digitalmemory/oais-platform/-/issues/129Route collisions2023-11-15T17:15:16+01:00Antonio Vivaceantonio.vivace@cern.chRoute collisionsThere are a couple of routes with possible collisions (generic / route vs /{id}) which should be fixed/clarified:
- "search_retrieve" has collisions [('/api/search/{source}/', 'get'), ('/api/search/{source}/{recid}/', 'get')]
- "upload_...There are a couple of routes with possible collisions (generic / route vs /{id}) which should be fixed/clarified:
- "search_retrieve" has collisions [('/api/search/{source}/', 'get'), ('/api/search/{source}/{recid}/', 'get')]
- "upload_jobs_retrieve" has collisions [('/api/upload/jobs/', 'get'), ('/api/upload/jobs/{id}/', 'get')]
Is this an issue? See: https://docs.djangoproject.com/en/4.2/topics/http/urls/https://gitlab.cern.ch/digitalmemory/oais-platform/-/issues/128Consider running celery workers under other UID2023-07-04T11:35:51+02:00Antonio Vivaceantonio.vivace@cern.chConsider running celery workers under other UIDIt's a matter of adding `--uid=nobody --gid=nogroup` to the celery worker command.
See: https://stackoverflow.com/questions/59651428/runtimewarningyoure-running-the-worker-with-superuser-privilegesthis-is-absol
Celery is saying
```
/...It's a matter of adding `--uid=nobody --gid=nogroup` to the celery worker command.
See: https://stackoverflow.com/questions/59651428/runtimewarningyoure-running-the-worker-with-superuser-privilegesthis-is-absol
Celery is saying
```
/usr/local/lib/python3.7/site-packages/celery/platforms.py:841: SecurityWarning: You're running the worker with superuser privileges: this is
oais_celery absolutely not recommended!
```https://gitlab.cern.ch/digitalmemory/oais-platform/-/issues/127OIDC callback state not found in session `oidc_states`!2023-06-20T11:39:01+02:00Antonio Vivaceantonio.vivace@cern.chOIDC callback state not found in session `oidc_states`!This happens when using the container feature of browsers (the login changes container)
This should be kept in mind when going through sentry errorsThis happens when using the container feature of browsers (the login changes container)
This should be kept in mind when going through sentry errorshttps://gitlab.cern.ch/digitalmemory/oais-platform/-/issues/126requirements-dev.txt should only contain the additional packages needed durin...2023-07-04T00:34:07+02:00Antonio Vivaceantonio.vivace@cern.chrequirements-dev.txt should only contain the additional packages needed during developmente.g.: blacke.g.: blackhttps://gitlab.cern.ch/digitalmemory/oais-platform/-/issues/125docker: add container for building the react app2023-11-15T17:04:07+01:00Antonio Vivaceantonio.vivace@cern.chdocker: add container for building the react appSame as https://gitlab.cern.ch/digitalmemory/oais-platform/-/issues/29 ?Same as https://gitlab.cern.ch/digitalmemory/oais-platform/-/issues/29 ?https://gitlab.cern.ch/digitalmemory/oais-platform/-/issues/124Indico needs a token for public records2023-11-15T14:09:52+01:00Antonio Vivaceantonio.vivace@cern.chIndico needs a token for public recordsThis shouldn't happen, as a token is not needed for public records
Probably a regression. Add a test for this
Is this still an issue? This is visible without any tokens:
https://indico.cern.ch/export/event/1324995.jsonThis shouldn't happen, as a token is not needed for public records
Probably a regression. Add a test for this
Is this still an issue? This is visible without any tokens:
https://indico.cern.ch/export/event/1324995.jsonhttps://gitlab.cern.ch/digitalmemory/oais-platform/-/issues/123Add a new route returning permissions2022-12-02T11:29:18+01:00Antonio Vivaceantonio.vivace@cern.chAdd a new route returning permissionsWe need a new API route returning what the user can and cannot do. There's no need to expose the specific permission nodes, some simplified version will work too. The frontend will then use this route to print some modals / info boxes ab...We need a new API route returning what the user can and cannot do. There's no need to expose the specific permission nodes, some simplified version will work too. The frontend will then use this route to print some modals / info boxes about what it can be done.https://gitlab.cern.ch/digitalmemory/oais-platform/-/issues/122Indico: search should return contributions and subcontributions too2022-11-29T19:40:08+01:00Antonio Vivaceantonio.vivace@cern.chIndico: search should return contributions and subcontributions tooAlso event notes? We could have multiple shortcuts/endpoints for thisAlso event notes? We could have multiple shortcuts/endpoints for thishttps://gitlab.cern.ch/digitalmemory/oais-platform/-/issues/121Add limitations2023-11-15T14:26:04+01:00Antonio Vivaceantonio.vivace@cern.chAdd limitationsE.g. Limit number of staged records to 10 ?
Consider: on search you can do 50 hit/page and Select All -> but not possible to have that many records in staged? Do we want to archive every content under given user? Should it be limited at...E.g. Limit number of staged records to 10 ?
Consider: on search you can do 50 hit/page and Select All -> but not possible to have that many records in staged? Do we want to archive every content under given user? Should it be limited at the moment of checking the box?https://gitlab.cern.ch/digitalmemory/oais-platform/-/issues/119The "delete" api endpoint is misleading2023-11-15T14:27:38+01:00Antonio Vivaceantonio.vivace@cern.chThe "delete" api endpoint is misleadingWe use the delete endpoint to **remove** an archive from the staged area. We should keep the "delete" word for when we actually trigger a destructive action.
maybe we can use /staged/remove? or /remove? staged/promote?We use the delete endpoint to **remove** an archive from the staged area. We should keep the "delete" word for when we actually trigger a destructive action.
maybe we can use /staged/remove? or /remove? staged/promote?https://gitlab.cern.ch/digitalmemory/oais-platform/-/issues/118Reuse logic for unstaging a single archive or more than one2022-10-12T15:21:35+02:00Antonio Vivaceantonio.vivace@cern.chReuse logic for unstaging a single archive or more than oneThey are currently two different methods/routes. Reuse the same logic.They are currently two different methods/routes. Reuse the same logic.https://gitlab.cern.ch/digitalmemory/oais-platform/-/issues/117Remove staged archives from archives list2022-10-12T15:52:48+02:00Kostas ChelakisRemove staged archives from archives list