cern-search-rest-api issueshttps://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues2019-11-25T10:20:49+01:00https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/92Profile create record changes2019-11-25T10:20:49+01:00Carina AntunesProfile create record changesProfile create record changes due to https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/issues/85.
If negative impacts is found make files with buckets configurable.Profile create record changes due to https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/issues/85.
If negative impacts is found make files with buckets configurable.https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/89test: ui/test_ui::test_view is failling2019-11-14T16:54:33+01:00Carina Antunestest: ui/test_ui::test_view is failling```
__________________________________________________________________________________________ test_view ___________________________________________________________________________________________
appctx = <Flask 'invenio'>, base_clien...```
__________________________________________________________________________________________ test_view ___________________________________________________________________________________________
appctx = <Flask 'invenio'>, base_client = <FlaskClient <Flask 'invenio'>>
@pytest.mark.unit
def test_view(appctx, base_client):
> resp = base_client.post("/account/settings/applications/", follow_redirects=True)
tests/ui/test_ui.py:17:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/werkzeug/test.py:1039: in post
return self.open(*args, **kw)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/testing.py:227: in open
follow_redirects=follow_redirects,
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/werkzeug/test.py:1017: in open
response, new_location, environ, buffered=buffered
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/werkzeug/test.py:948: in resolve_redirect
return self.open(builder, as_tuple=True, buffered=buffered)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/testing.py:227: in open
follow_redirects=follow_redirects,
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/werkzeug/test.py:993: in open
response = self.run_wsgi_app(environ.copy(), buffered=buffered)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/werkzeug/test.py:884: in run_wsgi_app
rv = run_wsgi_app(self.application, environ, buffered=buffered)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/werkzeug/test.py:1119: in run_wsgi_app
app_rv = app(environ, start_response)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/app.py:2463: in __call__
return self.wsgi_app(environ, start_response)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/app.py:2449: in wsgi_app
response = self.handle_exception(e)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/app.py:1866: in handle_exception
reraise(exc_type, exc_value, tb)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/_compat.py:39: in reraise
raise value
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/app.py:2446: in wsgi_app
response = self.full_dispatch_request()
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/app.py:1951: in full_dispatch_request
rv = self.handle_user_exception(e)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/app.py:1820: in handle_user_exception
reraise(exc_type, exc_value, tb)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/_compat.py:39: in reraise
raise value
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/app.py:1949: in full_dispatch_request
rv = self.dispatch_request()
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/app.py:1935: in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask_security/decorators.py:230: in wrapper
return f(*args, **kwargs)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask_security/views.py:89: in login
**_ctx('login'))
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask_security/core.py:528: in render_template
return render_template(*args, **kwargs)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/templating.py:140: in render_template
ctx.app,
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/templating.py:120: in _render
rv = template.render(context)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/jinja2/asyncsupport.py:76: in render
return original_render(self, *args, **kwargs)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/jinja2/environment.py:1008: in render
return self.environment.handle_exception(exc_info, True)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/jinja2/environment.py:780: in handle_exception
reraise(exc_type, exc_value, tb)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/jinja2/_compat.py:37: in reraise
raise value.with_traceback(tb)
/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/invenio_oauthclient/templates/invenio_oauthclient/login_user.html:11: in top-level template code
{%- from "invenio_oauthclient/_macros.html" import oauth_button %}
/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/invenio_accounts/templates/invenio_accounts/login_user.html:11: in top-level template code
{% from "invenio_accounts/_macros.html" import render_field, form_errors %}
/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/invenio_theme/templates/invenio_theme/page_cover.html:13: in top-level template code
{%- set body_css_classes=['cover-page'] %}
/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/invenio_theme/templates/invenio_theme/page.html:12: in top-level template code
{%- block head %}
/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/invenio_theme/templates/invenio_theme/page.html:47: in block "head"
{%- block css %}
/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/invenio_theme/templates/invenio_theme/page.html:48: in block "css"
{{ webpack['theme.css']}}
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/jinja2/environment.py:411: in getitem
return obj[argument]
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/werkzeug/local.py:378: in <lambda>
__getitem__ = lambda x, i: x._get_current_object()[i]
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/werkzeug/local.py:307: in _get_current_object
return self.__local()
/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/flask_webpackext/proxies.py:22: in <lambda>
lambda: current_app.extensions['flask-webpackext'].manifest)
/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/flask_webpackext/ext.py:88: in manifest
join(self.app.static_folder, path))
/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/flask_webpackext/manifest.py:66: in load
super(JinjaManifestLoader, self).load(filepath)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <flask_webpackext.manifest.JinjaManifestLoader object at 0x7f270187df98>, filepath = '/opt/rh/rh-python36/root/usr/var/instance/static/dist/manifest.json'
def load(self, filepath):
"""Load a manifest from a file."""
> with open(filepath) as fp:
E FileNotFoundError: [Errno 2] No such file or directory: '/opt/rh/rh-python36/root/usr/var/instance/static/dist/manifest.json'
/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pywebpack/manifests.py:212: FileNotFoundError
```https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/87global: expose health status2019-11-06T10:51:34+01:00Pablo Panero Vazquezglobal: expose health statusExpose health status to the outside. This would allow clients to check if their search instance is up and running.
Need to aggregate the `/health/{es,db,uwsig}` (web app) + `/ping` (nginx) endpoint responses.Expose health status to the outside. This would allow clients to check if their search instance is up and running.
Need to aggregate the `/health/{es,db,uwsig}` (web app) + `/ping` (nginx) endpoint responses.https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/81Lock all openshift dependencies2019-10-18T12:17:12+02:00Carina AntunesLock all openshift dependenciesWithout pinning images will be pulled using the very latest version at build time.
To avoid an unstable app state, avoid things breaking with new dependencies releases and avoid confusion on supported environments lock all images versio...Without pinning images will be pulled using the very latest version at build time.
To avoid an unstable app state, avoid things breaking with new dependencies releases and avoid confusion on supported environments lock all images versions.
Current unpined versions:[nginx](https://gitlab.cern.ch/webservices/cern-search/cern-search-openshift/blob/master/rest-api/application.yaml#L64) and [redis](https://gitlab.cern.ch/webservices/cern-search/cern-search-openshift/blob/master/rest-api/services.yaml#L42).
Update docs with the images versions supported.
Check if any other exists. (web-ui)https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/80Investigation: Wrong python dependencies in docker base image2021-07-14T15:48:37+02:00Carina AntunesInvestigation: Wrong python dependencies in docker base imageInvestigate/ Validate if docker base images takes into account the correct python version and dependencies specified in pipefile.lock.
CI step: `build_base_image`
[Image](gitlab-registry.cern.ch/webservices/cern-search/cern-search-rest...Investigate/ Validate if docker base images takes into account the correct python version and dependencies specified in pipefile.lock.
CI step: `build_base_image`
[Image](gitlab-registry.cern.ch/webservices/cern-search/cern-search-rest-api/cern-search-rest-api-base:f813c09e1f92c96f7eb71b2d43c076388be57680) in question was added in this commit:
https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/commit/d4f39ce652d32cfab94912b0eb2b74d993c2d325
The image install dependencies with python 3.6 event though the pipfile.lock specifies 3.4:
* [Pipfile.lock](https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/blob/d4f39ce652d32cfab94912b0eb2b74d993c2d325/Pipfile.lock#L8)
* [Base Python Version](https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/blob/d4f39ce652d32cfab94912b0eb2b74d993c2d325/Dockerfile-base#L10)https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/75docs: index creation/modification2019-09-26T08:19:17+02:00Pablo Panero Vazquezdocs: index creation/modificationDocument in the admin docs how to create/modify JSONSchemas / ES Mappings (rebuild image, run ``invenio index`` command, etc.)Document in the admin docs how to create/modify JSONSchemas / ES Mappings (rebuild image, run ``invenio index`` command, etc.)https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/74deployment: implement versioning on base image2021-07-14T15:47:52+02:00Pablo Panero Vazquezdeployment: implement versioning on base imageWhen building the api image, it will fail if we have changed/updated the image of the base (dependencies). In order to avoid hard crashes or blocking/forcing people to update. We need some sort of versioning on the base image.
Might aff...When building the api image, it will fail if we have changed/updated the image of the base (dependencies). In order to avoid hard crashes or blocking/forcing people to update. We need some sort of versioning on the base image.
Might affect https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/issues/53 and https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/issues/40https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/73deployment: create cookiecutter2019-09-24T08:23:02+02:00Pablo Panero Vazquezdeployment: create cookiecutterTo allow other entities to deploy a Search instance in a configurable well we can provide a [Cookiecutter](https://cookiecutter.readthedocs.io/en/latest/) template.To allow other entities to deploy a Search instance in a configurable well we can provide a [Cookiecutter](https://cookiecutter.readthedocs.io/en/latest/) template.https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/72docs: add screenshots to obtain token workflow2019-09-20T08:39:38+02:00Pablo Panero Vazquezdocs: add screenshots to obtain token workflowThe documentation explains how to obtain a token, but it is not clear. Add screenshots to guide the user. link: http://cernsearchdocs.web.cern.ch/cernsearchdocs/usage/permissions/The documentation explains how to obtain a token, but it is not clear. Add screenshots to guide the user. link: http://cernsearchdocs.web.cern.ch/cernsearchdocs/usage/permissions/https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/71api: validate document before indexing2019-09-20T15:25:47+02:00Pablo Panero Vazquezapi: validate document before indexingValidate the documents for ES inconsistencies / wrong or characters before indexing. This will allow to fail gracefully and tell the user there is (and maybe where) and error in his/her query. Otherwise the error will be produced at Elas...Validate the documents for ES inconsistencies / wrong or characters before indexing. This will allow to fail gracefully and tell the user there is (and maybe where) and error in his/her query. Otherwise the error will be produced at Elasticsearch cluster level which might produce undesired effects.
Invenio-Indexer users the [client](https://github.com/inveniosoftware/invenio-indexer/blob/master/invenio_indexer/api.py#L125) that comes from [Invenio-Search/ext/current_search_client](https://github.com/inveniosoftware/invenio-search/blob/master/invenio_search/ext.py#L19). This latter one is a proxy to the Elasticsearch client. Its [``index``](https://elasticsearch-py.readthedocs.io/en/master/api.html#elasticsearch.Elasticsearch.index) ([code](https://github.com/elastic/elasticsearch-py/blob/master/elasticsearch/client/__init__.py#L324)) function does not perform any validation, whilst [``save`` from Elasticsearch-dsl](https://elasticsearch-dsl.readthedocs.io/en/6.4.0/api.html?highlight=save#elasticsearch_dsl.Document.save) ([code](https://github.com/elastic/elasticsearch-dsl-py/blob/ec9f9c014a38e880d2b7a1a0364ee459d3fcae8a/elasticsearch_dsl/document.py#L417)) does.https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/70add infrastructure monitoring and alerting2021-07-14T15:21:15+02:00Pablo Panero Vazquezadd infrastructure monitoring and alertingApplication monitoring is done through Sentry as in [Invenio Training](https://github.com/inveniosoftware/training/tree/v3.1/14-deployment-monitoring/).
- [ ] Kibana: Add Kibana for application monitoring.
- [ ] Add infrastructure monit...Application monitoring is done through Sentry as in [Invenio Training](https://github.com/inveniosoftware/training/tree/v3.1/14-deployment-monitoring/).
- [ ] Kibana: Add Kibana for application monitoring.
- [ ] Add infrastructure monitoring:
- [ ] Database monitoring (Related to https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/issues/56).
- [ ] Elasticsearch monitoring.
- [ ] OpenShift application (e.g. connections open, pods up, memory consumed, etc.)
- [ ] Redis cache.https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/61documentation: integration issues2021-07-14T15:14:43+02:00Pablo Panero Vazquezdocumentation: integration issuesFrom EDMS integration:
- Use of escaping characters: when/when not/which ones.From EDMS integration:
- Use of escaping characters: when/when not/which ones.https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/59edms: index backward slash2019-09-13T16:12:26+02:00Pablo Panero Vazquezedms: index backward slash- When querying all special characters must be escaped [docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#_reserved_characters).
- When indexing, they just need to be encoded.
Howeve...- When querying all special characters must be escaped [docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#_reserved_characters).
- When indexing, they just need to be encoded.
However, \ breaks the indexing. There is an strange case when putting quotes right after (e.g. "field":"value \""), but the indexed value will also contain that quote.
Needs debugging.https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/56database: improve connection handling and monitoring2021-07-14T15:13:36+02:00Pablo Panero Vazquezdatabase: improve connection handling and monitoringTake as base info: https://indico.cern.ch/event/773969/contributions/3351846/attachments/1814268/2967399/Deployment_and_monitoring.pdf
Tutorial here:https://github.com/inveniosoftware/training/tree/master/14-deployment-monitoringTake as base info: https://indico.cern.ch/event/773969/contributions/3351846/attachments/1814268/2967399/Deployment_and_monitoring.pdf
Tutorial here:https://github.com/inveniosoftware/training/tree/master/14-deployment-monitoringhttps://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/54deployment: separate web from API nodes2021-07-14T15:12:43+02:00Pablo Panero Vazquezdeployment: separate web from API nodesCurrently both wsgi apps (wsgi_api and wsgi_ui) are running in the same container/processes. They should be separated so their behaviour is isolated.
An example configuration for uWSGI could be the one posted [here](https://github.com/i...Currently both wsgi apps (wsgi_api and wsgi_ui) are running in the same container/processes. They should be separated so their behaviour is isolated.
An example configuration for uWSGI could be the one posted [here](https://github.com/inveniosoftware/troubleshooting/issues/51) by fenekku.https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/51permissions: refactor permission factory2019-10-17T16:59:24+02:00Pablo Panero Vazquezpermissions: refactor permission factoryAs done in ``Invenio-Records-Permissions`` for ``Invenio-App-RDM``, a permission factory should return a ``Permissions`` (set of ``Needs`` and ``Roles``). Then the ``need_permission`` decorator of ``invenio-records-rest`` would do the ch...As done in ``Invenio-Records-Permissions`` for ``Invenio-App-RDM``, a permission factory should return a ``Permissions`` (set of ``Needs`` and ``Roles``). Then the ``need_permission`` decorator of ``invenio-records-rest`` would do the check for permissions and accept/reject the request.https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/47Integrate CDS into CERN Search2019-09-13T16:13:33+02:00Pablo Panero VazquezIntegrate CDS into CERN SearchThere might be more than one possibility:
- CDS to push content to CERN Search
- Crawl CDS
- Since both have Elasticsearch underneath, agree on a data-model and use [cross-cluster search](https://www.elastic.co/guide/en/elasticsearch/re...There might be more than one possibility:
- CDS to push content to CERN Search
- Crawl CDS
- Since both have Elasticsearch underneath, agree on a data-model and use [cross-cluster search](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cross-cluster-search.html)
As discussed IRL with CDS managers, this might happen in the far future but is at the moment in the lowest priority.https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/45Add support for suggestions before search2019-09-13T16:10:50+02:00Pablo Panero VazquezAdd support for suggestions before searchInvenio-Records_rest already provides this functionality [1]
From the ES side:
- Completion suggesters require to assign manually the keywords that the ``suggest`` query would match.
- An alternative way is to implement an n-gram analy...Invenio-Records_rest already provides this functionality [1]
From the ES side:
- Completion suggesters require to assign manually the keywords that the ``suggest`` query would match.
- An alternative way is to implement an n-gram analyzer and then query with the standard one. Adding some fuzzyness or wildcard to it would improve its performance.
- Term suggesters are a good alternative but more computing intense.
- Phrase suggesters can be used to build a "did you mean" functionality.
[1] Suggesters: https://invenio-records-rest.readthedocs.io/en/latest/usage.html#suggesters
https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/44Add support for highlighting on search results2021-07-14T15:05:46+02:00Pablo Panero VazquezAdd support for highlighting on search resultsBy adding a custom search factory, which CSaS already has ([search.py#79](https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/blob/master/cern_search_rest_api/modules/cernsearch/search.py#L79)), highlighting can be implem...By adding a custom search factory, which CSaS already has ([search.py#79](https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/blob/master/cern_search_rest_api/modules/cernsearch/search.py#L79)), highlighting can be implemented via elasticsearch (Check the elasticsearch dsl).
[1] Advance customization: https://invenio-records-rest.readthedocs.io/en/latest/usage.html#advanced-customizationhttps://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/41[MyCERN] Research the use of Semantic search2019-09-13T16:13:45+02:00Pablo Panero Vazquez[MyCERN] Research the use of Semantic searchResearch about the use of Semantic search on elasticsearch (e.g. "Where can I...", "Today's events...")Research about the use of Semantic search on elasticsearch (e.g. "Where can I...", "Today's events...")