webservices issueshttps://gitlab.cern.ch/groups/webservices/-/issues2023-08-07T10:10:15+02:00https://gitlab.cern.ch/webservices/webeos/webeos-documentation/-/issues/13Add help redirect error pages2023-08-07T10:10:15+02:00Caetan Tojeiro CarpenteAdd help redirect error pages[RQF2372055](https://cern.service-now.com/nav_to.do?uri=u_request_fulfillment.do?sys_id=e6bb8f93979cf5505e4173ae2153afc5)[RQF2372055](https://cern.service-now.com/nav_to.do?uri=u_request_fulfillment.do?sys_id=e6bb8f93979cf5505e4173ae2153afc5)Caetan Tojeiro CarpenteCaetan Tojeiro Carpentehttps://gitlab.cern.ch/webservices/web-redirector-v2/-/issues/35Add cronjob for removing db entries marked for deletion2023-03-30T11:23:21+02:00Dimitra Chatzichrysoudimitra.chatzichrysou@cern.chAdd cronjob for removing db entries marked for deletionhttps://gitlab.cern.ch/webservices/web-redirector-v2/-/issues/34Take SQL dumps from database2022-12-21T11:17:21+01:00Jack Henscheljack.henschel@cern.chTake SQL dumps from databaseDBOD does not take "regular" SQL dumps of the database.
When we have these (in a PVC), it's much easier to restore the database in case of an incident.DBOD does not take "regular" SQL dumps of the database.
When we have these (in a PVC), it's much easier to restore the database in case of an incident.https://gitlab.cern.ch/webservices/web-redirector-v2/-/issues/30Add preview endpoint2022-10-24T11:20:14+02:00Jack Henscheljack.henschel@cern.chAdd preview endpointAn endpoint where the user can see to which site the redirect goes (similar to `curl -I https://cern.ch/xyz`).
It could for example be `https://cern.ch/preview/xyz` -> show redirect link in HTMLAn endpoint where the user can see to which site the redirect goes (similar to `curl -I https://cern.ch/xyz`).
It could for example be `https://cern.ch/preview/xyz` -> show redirect link in HTMLhttps://gitlab.cern.ch/webservices/cern-search/cern-search-admin-docs/-/issues/2[#2] Add documentation on support development for clients2022-08-10T15:26:58+02:00Estela Martinez Calo[#2] Add documentation on support development for clients**Add documentation about how to redeploy a new image, when a client provides new mappings or changes to mappings:**
1) When clients push a new MR with a new mapping, we need to redeploy the changes for them.
For that we need to clic...**Add documentation about how to redeploy a new image, when a client provides new mappings or changes to mappings:**
1) When clients push a new MR with a new mapping, we need to redeploy the changes for them.
For that we need to click the manual steps in the MR to build and push the image to the Paas registry. The image will be available with a tag corresponding to the slug of the MR branch.
If the deployment config (worker and web) is already associated with that image an automatic rollout is done. \
For the first time, a manual change to point to that new image is needed: For that edit the deployment config of worker and web and select the tag of the image you just created (wait for the manual jobs in the MR to finish).
2) When the deployment is finished, we need to wipe the data in the instance, in order for clients to be able to test with an empty instance.
We need to wipe DB records and ES indexes/mappings and re-init the instance. \
For ES everything is documented already. \
For DB we have 2 options:
- wipe everything if the client does not mind creating new API Tokens
- alternatively we can truncate tables but leave user/token data
For the second option we have documentation.
For the first option:
- Drop the tables, using: `db drop`
- Recreate them using `db create`
- Finally run `invenio files location default ${DEFAULT_RECORDS_FILES_LOCATION} --default` to finish the DB setup
**Document the process and adapt existing documentation to reflect this process.**
- Add new page: called `Deploy new Image and Mappings (and support for DEV)`
- On that page add explanation above, step 1) and 2) and on step 2) link to existing page https://cern-search-admin.docs.cern.ch/operations/truncate_instance/
- Update page https://cern-search-admin.docs.cern.ch/operations/truncate_instance/ with the missing information
**Additionally**, copy warning section "First deployment of an instance" from https://cern-search-internal.docs.cern.ch/manual-deployment/openshift/ to helm deployment https://cern-search-internal.docs.cern.ch/deployment/helm-infra/ > New instance instructions > `4. Init Invenio configuration`Estela Martinez CaloEstela Martinez Calohttps://gitlab.cern.ch/webservices/cern-search/cern-search-ui/-/issues/43Add support for 2 builds: production and staging2022-06-13T19:01:39+02:00Carina AntunesAdd support for 2 builds: production and stagingCurrently we are build for dev-cern-search-ui.
When we want to deploy prod, we need to build tags to env production and master to env development.Currently we are build for dev-cern-search-ui.
When we want to deploy prod, we need to build tags to env production and master to env development.https://gitlab.cern.ch/webservices/cern-search/web-crawler/-/issues/29Create a baseline to compare improvements for accelconf and dfs sites crawlers2022-06-08T09:21:37+02:00Carina AntunesCreate a baseline to compare improvements for accelconf and dfs sites crawlers- Document parameters used and outcome of resources found and needed
- what paramenters, eg depth
- how many hours it took
- memory/cpu usage/needed
- how many results were found
- Document improvements that could be done for t...- Document parameters used and outcome of resources found and needed
- what paramenters, eg depth
- how many hours it took
- memory/cpu usage/needed
- how many results were found
- Document improvements that could be done for the crawler
- Audit job status: sites seeded, success/failed, % sites seeded/sites crawled,
- Audit nr of pages per site
- Audit page crawled state: ie success/failed, ignored due to depth limit, depth, last crawled date
- Audit files crawled state: ie success/failed, ignored due to size or depth, size, depth, las crawled dateCarina AntunesNoor Afshan FathimaCarina Antuneshttps://gitlab.cern.ch/webservices/cern-search/cern-search-ui/-/issues/42Support for craco 6.x in @semantic-ui-react/craco-less2022-03-31T15:50:01+02:00Estela Martinez CaloSupport for craco 6.x in @semantic-ui-react/craco-lessNot urgent.
We are using outdated versions of react-scripts (3.4.4 vs v5), craco and craco-less.
More details https://github.com/Semantic-Org/Semantic-UI-React/issues/4301. \
We need to bump and update dependencies eventually. For now i...Not urgent.
We are using outdated versions of react-scripts (3.4.4 vs v5), craco and craco-less.
More details https://github.com/Semantic-Org/Semantic-UI-React/issues/4301. \
We need to bump and update dependencies eventually. For now it's running, even though it's not listed as one of the recommended compatible releases between them.
RN running:
- craco 5.9.0
- semantic-ui-react/craco-less 1.2.6
- react-scripts 3.4.4
Related to https://github.com/Semantic-Org/Semantic-UI-React/issues/4310Carina AntunesEstela Martinez CaloCarina Antuneshttps://gitlab.cern.ch/webservices/cern-search/web-crawler/-/issues/28Bump all dependencies to latest version2022-03-31T14:04:23+02:00Noor Afshan FathimaBump all dependencies to latest version- Scrapy and other dependencies (comments in setup file)
- Python to 3.8
- change to DNF instead of using yum- Scrapy and other dependencies (comments in setup file)
- Python to 3.8
- change to DNF instead of using yumhttps://gitlab.cern.ch/webservices/cern-search/web-crawler/-/issues/27Move from pip to poetry2022-03-31T13:53:07+02:00Noor Afshan FathimaMove from pip to poetryDescription:
Tasks:
- [ ] Change docker, docker-compose file
- [ ] pyproject.toml file
- [ ] Read about setup.py . needed /not needed? why?Description:
Tasks:
- [ ] Change docker, docker-compose file
- [ ] pyproject.toml file
- [ ] Read about setup.py . needed /not needed? why?https://gitlab.cern.ch/webservices/cern-search/web-crawler/-/issues/25Add login behind SSO and crawler webpages with ACL = LOA2022-03-24T10:42:58+01:00Carina AntunesAdd login behind SSO and crawler webpages with ACL = LOASupport new SSO only, and CERN login only for nowSupport new SSO only, and CERN login only for nowhttps://gitlab.cern.ch/webservices/cern-search/cern-search-infra/-/issues/13Add crawler to helm deployment2022-04-20T13:28:23+02:00Carina AntunesAdd crawler to helm deploymentDescription: TBADescription: TBAhttps://gitlab.cern.ch/webservices/cern-search/web-crawler/-/issues/23Indexing improvements2022-03-24T14:13:51+01:00Carina AntunesIndexing improvementsIdentify improvements to be done
- Avoiding Useless Content
- Avoid Spammish pages
- Avoid inappropriate sites? report site is already in place
- Avoid dev/QA sites that are hosted as official sites.Identify improvements to be done
- Avoiding Useless Content
- Avoid Spammish pages
- Avoid inappropriate sites? report site is already in place
- Avoid dev/QA sites that are hosted as official sites.https://gitlab.cern.ch/webservices/cern-search/web-crawler/-/issues/22Implement auditing2022-03-17T16:17:41+01:00Carina AntunesImplement auditinghttps://gitlab.cern.ch/webservices/cern-search/web-crawler/-/issues/21Investigate auditing options and needs2022-03-17T16:52:41+01:00Carina AntunesInvestigate auditing options and needsEtcd vs ELK stack
Considerations:
- Audit: url, from and to urls, hash of content, timestamp, success state, error cause
- Possible usecases:
- avoid reindexing same results for speed eg key exists and timestamp ==, or hash of cont...Etcd vs ELK stack
Considerations:
- Audit: url, from and to urls, hash of content, timestamp, success state, error cause
- Possible usecases:
- avoid reindexing same results for speed eg key exists and timestamp ==, or hash of content ==?
- avoid reindexing same content for different LOA, eg lightweight < cern, and if content for both is the same, then update acl list of document instead of sending in a duplicate (use hash of content)
- Latency, scalability and effort to implement/integrate
- Python clients
- Deploy or hosted at cern
- Search and visualization, eg percentage of failures, ease to get list of failed endpoints for retry in a different jobhttps://gitlab.cern.ch/webservices/cern-search/cern-search-ui/-/issues/41[#41]Move help from header to footer2022-03-17T16:06:24+01:00Carina Antunes[#41]Move help from header to footerEstela Martinez CaloEstela Martinez Calohttps://gitlab.cern.ch/webservices/cern-search/cern-search-ui/-/issues/40[#40]Add login/logout button and directory button2022-07-29T12:28:06+02:00Carina Antunes[#40]Add login/logout button and directory buttonDirectory button can be linked, login/logout behaviour will be added in another ticket.
Same as https://webservices-portal.web.cern.ch/
Related to https://gitlab.cern.ch/webservices/cern-search/cern-search-ui/-/merge_requests/41Directory button can be linked, login/logout behaviour will be added in another ticket.
Same as https://webservices-portal.web.cern.ch/
Related to https://gitlab.cern.ch/webservices/cern-search/cern-search-ui/-/merge_requests/41Estela Martinez CaloEstela Martinez Calohttps://gitlab.cern.ch/webservices/cern-search/cern-search-ui/-/issues/39Edit content on help page2022-03-16T10:37:16+01:00Carina AntunesEdit content on help pageContent on new help page was taken from the old search as-is and does not apply to the new search.
Update the content with helpful and correct content.
New rules/tips can be extracted from the help syntax and the user docs: https://cer...Content on new help page was taken from the old search as-is and does not apply to the new search.
Update the content with helpful and correct content.
New rules/tips can be extracted from the help syntax and the user docs: https://cern-search.docs.cern.ch/usage/operations/#advanced-queries.
Inspiration can be also taken from https://inveniordm.web.cern.ch/help/search which has the same syntax as us.
Tasks:
- Test and study the queries and prepare a codimd with suggested new help tips: Estimate: 1 day
- Change UI not to use cards in the help page + Add the new content: Estimate: 1 dayEstela Martinez CaloEstela Martinez Calohttps://gitlab.cern.ch/webservices/cern-search/cern-search-admin-docs/-/issues/1Migrate from openshift to webeos2022-03-01T16:21:22+01:00Carina AntunesMigrate from openshift to webeos
Follow https://how-to.docs.cern.ch/gitlabpagessite/migration/
Site: https://cern-search-admin.docs.cern.ch/ \
Openshift: https://openshift.cern.ch/console/project/cern-search-admin-docs
After creation under the webservices-portal reas...
Follow https://how-to.docs.cern.ch/gitlabpagessite/migration/
Site: https://cern-search-admin.docs.cern.ch/ \
Openshift: https://openshift.cern.ch/console/project/cern-search-admin-docs
After creation under the webservices-portal reassign the project to `crdeoliv` and add `CernSearch-Administrators` as admins.
Context: Check epic for more detailsEstela Martinez CaloEstela Martinez Calohttps://gitlab.cern.ch/webservices/cern-search/cern-search-ui/-/issues/37Header2022-02-24T15:00:36+01:00Carina AntunesHeaderCheck the header in https://notifications.web.cern.ch/. It must look like this (size, color, font etc from CERN guidelines)Check the header in https://notifications.web.cern.ch/. It must look like this (size, color, font etc from CERN guidelines)