cern-search-rest-api issueshttps://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues2019-09-13T16:13:33+02:00https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/47Integrate CDS into CERN Search2019-09-13T16:13:33+02:00Pablo Panero VazquezIntegrate CDS into CERN SearchThere might be more than one possibility:
- CDS to push content to CERN Search
- Crawl CDS
- Since both have Elasticsearch underneath, agree on a data-model and use [cross-cluster search](https://www.elastic.co/guide/en/elasticsearch/re...There might be more than one possibility:
- CDS to push content to CERN Search
- Crawl CDS
- Since both have Elasticsearch underneath, agree on a data-model and use [cross-cluster search](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cross-cluster-search.html)
As discussed IRL with CDS managers, this might happen in the far future but is at the moment in the lowest priority.https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/92Profile create record changes2019-11-25T10:20:49+01:00Carina AntunesProfile create record changesProfile create record changes due to https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/issues/85.
If negative impacts is found make files with buckets configurable.Profile create record changes due to https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/issues/85.
If negative impacts is found make files with buckets configurable.https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/89test: ui/test_ui::test_view is failling2019-11-14T16:54:33+01:00Carina Antunestest: ui/test_ui::test_view is failling```
__________________________________________________________________________________________ test_view ___________________________________________________________________________________________
appctx = <Flask 'invenio'>, base_clien...```
__________________________________________________________________________________________ test_view ___________________________________________________________________________________________
appctx = <Flask 'invenio'>, base_client = <FlaskClient <Flask 'invenio'>>
@pytest.mark.unit
def test_view(appctx, base_client):
> resp = base_client.post("/account/settings/applications/", follow_redirects=True)
tests/ui/test_ui.py:17:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/werkzeug/test.py:1039: in post
return self.open(*args, **kw)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/testing.py:227: in open
follow_redirects=follow_redirects,
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/werkzeug/test.py:1017: in open
response, new_location, environ, buffered=buffered
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/werkzeug/test.py:948: in resolve_redirect
return self.open(builder, as_tuple=True, buffered=buffered)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/testing.py:227: in open
follow_redirects=follow_redirects,
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/werkzeug/test.py:993: in open
response = self.run_wsgi_app(environ.copy(), buffered=buffered)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/werkzeug/test.py:884: in run_wsgi_app
rv = run_wsgi_app(self.application, environ, buffered=buffered)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/werkzeug/test.py:1119: in run_wsgi_app
app_rv = app(environ, start_response)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/app.py:2463: in __call__
return self.wsgi_app(environ, start_response)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/app.py:2449: in wsgi_app
response = self.handle_exception(e)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/app.py:1866: in handle_exception
reraise(exc_type, exc_value, tb)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/_compat.py:39: in reraise
raise value
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/app.py:2446: in wsgi_app
response = self.full_dispatch_request()
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/app.py:1951: in full_dispatch_request
rv = self.handle_user_exception(e)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/app.py:1820: in handle_user_exception
reraise(exc_type, exc_value, tb)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/_compat.py:39: in reraise
raise value
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/app.py:1949: in full_dispatch_request
rv = self.dispatch_request()
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/app.py:1935: in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask_security/decorators.py:230: in wrapper
return f(*args, **kwargs)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask_security/views.py:89: in login
**_ctx('login'))
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask_security/core.py:528: in render_template
return render_template(*args, **kwargs)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/templating.py:140: in render_template
ctx.app,
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/templating.py:120: in _render
rv = template.render(context)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/jinja2/asyncsupport.py:76: in render
return original_render(self, *args, **kwargs)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/jinja2/environment.py:1008: in render
return self.environment.handle_exception(exc_info, True)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/jinja2/environment.py:780: in handle_exception
reraise(exc_type, exc_value, tb)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/jinja2/_compat.py:37: in reraise
raise value.with_traceback(tb)
/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/invenio_oauthclient/templates/invenio_oauthclient/login_user.html:11: in top-level template code
{%- from "invenio_oauthclient/_macros.html" import oauth_button %}
/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/invenio_accounts/templates/invenio_accounts/login_user.html:11: in top-level template code
{% from "invenio_accounts/_macros.html" import render_field, form_errors %}
/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/invenio_theme/templates/invenio_theme/page_cover.html:13: in top-level template code
{%- set body_css_classes=['cover-page'] %}
/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/invenio_theme/templates/invenio_theme/page.html:12: in top-level template code
{%- block head %}
/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/invenio_theme/templates/invenio_theme/page.html:47: in block "head"
{%- block css %}
/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/invenio_theme/templates/invenio_theme/page.html:48: in block "css"
{{ webpack['theme.css']}}
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/jinja2/environment.py:411: in getitem
return obj[argument]
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/werkzeug/local.py:378: in <lambda>
__getitem__ = lambda x, i: x._get_current_object()[i]
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/werkzeug/local.py:307: in _get_current_object
return self.__local()
/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/flask_webpackext/proxies.py:22: in <lambda>
lambda: current_app.extensions['flask-webpackext'].manifest)
/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/flask_webpackext/ext.py:88: in manifest
join(self.app.static_folder, path))
/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/flask_webpackext/manifest.py:66: in load
super(JinjaManifestLoader, self).load(filepath)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <flask_webpackext.manifest.JinjaManifestLoader object at 0x7f270187df98>, filepath = '/opt/rh/rh-python36/root/usr/var/instance/static/dist/manifest.json'
def load(self, filepath):
"""Load a manifest from a file."""
> with open(filepath) as fp:
E FileNotFoundError: [Errno 2] No such file or directory: '/opt/rh/rh-python36/root/usr/var/instance/static/dist/manifest.json'
/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pywebpack/manifests.py:212: FileNotFoundError
```https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/87global: expose health status2019-11-06T10:51:34+01:00Pablo Panero Vazquezglobal: expose health statusExpose health status to the outside. This would allow clients to check if their search instance is up and running.
Need to aggregate the `/health/{es,db,uwsig}` (web app) + `/ping` (nginx) endpoint responses.Expose health status to the outside. This would allow clients to check if their search instance is up and running.
Need to aggregate the `/health/{es,db,uwsig}` (web app) + `/ping` (nginx) endpoint responses.https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/81Lock all openshift dependencies2019-10-18T12:17:12+02:00Carina AntunesLock all openshift dependenciesWithout pinning images will be pulled using the very latest version at build time.
To avoid an unstable app state, avoid things breaking with new dependencies releases and avoid confusion on supported environments lock all images versio...Without pinning images will be pulled using the very latest version at build time.
To avoid an unstable app state, avoid things breaking with new dependencies releases and avoid confusion on supported environments lock all images versions.
Current unpined versions:[nginx](https://gitlab.cern.ch/webservices/cern-search/cern-search-openshift/blob/master/rest-api/application.yaml#L64) and [redis](https://gitlab.cern.ch/webservices/cern-search/cern-search-openshift/blob/master/rest-api/services.yaml#L42).
Update docs with the images versions supported.
Check if any other exists. (web-ui)https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/75docs: index creation/modification2019-09-26T08:19:17+02:00Pablo Panero Vazquezdocs: index creation/modificationDocument in the admin docs how to create/modify JSONSchemas / ES Mappings (rebuild image, run ``invenio index`` command, etc.)Document in the admin docs how to create/modify JSONSchemas / ES Mappings (rebuild image, run ``invenio index`` command, etc.)https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/73deployment: create cookiecutter2019-09-24T08:23:02+02:00Pablo Panero Vazquezdeployment: create cookiecutterTo allow other entities to deploy a Search instance in a configurable well we can provide a [Cookiecutter](https://cookiecutter.readthedocs.io/en/latest/) template.To allow other entities to deploy a Search instance in a configurable well we can provide a [Cookiecutter](https://cookiecutter.readthedocs.io/en/latest/) template.https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/72docs: add screenshots to obtain token workflow2019-09-20T08:39:38+02:00Pablo Panero Vazquezdocs: add screenshots to obtain token workflowThe documentation explains how to obtain a token, but it is not clear. Add screenshots to guide the user. link: http://cernsearchdocs.web.cern.ch/cernsearchdocs/usage/permissions/The documentation explains how to obtain a token, but it is not clear. Add screenshots to guide the user. link: http://cernsearchdocs.web.cern.ch/cernsearchdocs/usage/permissions/https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/71api: validate document before indexing2019-09-20T15:25:47+02:00Pablo Panero Vazquezapi: validate document before indexingValidate the documents for ES inconsistencies / wrong or characters before indexing. This will allow to fail gracefully and tell the user there is (and maybe where) and error in his/her query. Otherwise the error will be produced at Elas...Validate the documents for ES inconsistencies / wrong or characters before indexing. This will allow to fail gracefully and tell the user there is (and maybe where) and error in his/her query. Otherwise the error will be produced at Elasticsearch cluster level which might produce undesired effects.
Invenio-Indexer users the [client](https://github.com/inveniosoftware/invenio-indexer/blob/master/invenio_indexer/api.py#L125) that comes from [Invenio-Search/ext/current_search_client](https://github.com/inveniosoftware/invenio-search/blob/master/invenio_search/ext.py#L19). This latter one is a proxy to the Elasticsearch client. Its [``index``](https://elasticsearch-py.readthedocs.io/en/master/api.html#elasticsearch.Elasticsearch.index) ([code](https://github.com/elastic/elasticsearch-py/blob/master/elasticsearch/client/__init__.py#L324)) function does not perform any validation, whilst [``save`` from Elasticsearch-dsl](https://elasticsearch-dsl.readthedocs.io/en/6.4.0/api.html?highlight=save#elasticsearch_dsl.Document.save) ([code](https://github.com/elastic/elasticsearch-dsl-py/blob/ec9f9c014a38e880d2b7a1a0364ee459d3fcae8a/elasticsearch_dsl/document.py#L417)) does.https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/70add infrastructure monitoring and alerting2021-07-14T15:21:15+02:00Pablo Panero Vazquezadd infrastructure monitoring and alertingApplication monitoring is done through Sentry as in [Invenio Training](https://github.com/inveniosoftware/training/tree/v3.1/14-deployment-monitoring/).
- [ ] Kibana: Add Kibana for application monitoring.
- [ ] Add infrastructure monit...Application monitoring is done through Sentry as in [Invenio Training](https://github.com/inveniosoftware/training/tree/v3.1/14-deployment-monitoring/).
- [ ] Kibana: Add Kibana for application monitoring.
- [ ] Add infrastructure monitoring:
- [ ] Database monitoring (Related to https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/issues/56).
- [ ] Elasticsearch monitoring.
- [ ] OpenShift application (e.g. connections open, pods up, memory consumed, etc.)
- [ ] Redis cache.https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/59edms: index backward slash2019-09-13T16:12:26+02:00Pablo Panero Vazquezedms: index backward slash- When querying all special characters must be escaped [docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#_reserved_characters).
- When indexing, they just need to be encoded.
Howeve...- When querying all special characters must be escaped [docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#_reserved_characters).
- When indexing, they just need to be encoded.
However, \ breaks the indexing. There is an strange case when putting quotes right after (e.g. "field":"value \""), but the indexed value will also contain that quote.
Needs debugging.https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/56database: improve connection handling and monitoring2021-07-14T15:13:36+02:00Pablo Panero Vazquezdatabase: improve connection handling and monitoringTake as base info: https://indico.cern.ch/event/773969/contributions/3351846/attachments/1814268/2967399/Deployment_and_monitoring.pdf
Tutorial here:https://github.com/inveniosoftware/training/tree/master/14-deployment-monitoringTake as base info: https://indico.cern.ch/event/773969/contributions/3351846/attachments/1814268/2967399/Deployment_and_monitoring.pdf
Tutorial here:https://github.com/inveniosoftware/training/tree/master/14-deployment-monitoringhttps://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/54deployment: separate web from API nodes2021-07-14T15:12:43+02:00Pablo Panero Vazquezdeployment: separate web from API nodesCurrently both wsgi apps (wsgi_api and wsgi_ui) are running in the same container/processes. They should be separated so their behaviour is isolated.
An example configuration for uWSGI could be the one posted [here](https://github.com/i...Currently both wsgi apps (wsgi_api and wsgi_ui) are running in the same container/processes. They should be separated so their behaviour is isolated.
An example configuration for uWSGI could be the one posted [here](https://github.com/inveniosoftware/troubleshooting/issues/51) by fenekku.https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/51permissions: refactor permission factory2019-10-17T16:59:24+02:00Pablo Panero Vazquezpermissions: refactor permission factoryAs done in ``Invenio-Records-Permissions`` for ``Invenio-App-RDM``, a permission factory should return a ``Permissions`` (set of ``Needs`` and ``Roles``). Then the ``need_permission`` decorator of ``invenio-records-rest`` would do the ch...As done in ``Invenio-Records-Permissions`` for ``Invenio-App-RDM``, a permission factory should return a ``Permissions`` (set of ``Needs`` and ``Roles``). Then the ``need_permission`` decorator of ``invenio-records-rest`` would do the check for permissions and accept/reject the request.https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/45Add support for suggestions before search2019-09-13T16:10:50+02:00Pablo Panero VazquezAdd support for suggestions before searchInvenio-Records_rest already provides this functionality [1]
From the ES side:
- Completion suggesters require to assign manually the keywords that the ``suggest`` query would match.
- An alternative way is to implement an n-gram analy...Invenio-Records_rest already provides this functionality [1]
From the ES side:
- Completion suggesters require to assign manually the keywords that the ``suggest`` query would match.
- An alternative way is to implement an n-gram analyzer and then query with the standard one. Adding some fuzzyness or wildcard to it would improve its performance.
- Term suggesters are a good alternative but more computing intense.
- Phrase suggesters can be used to build a "did you mean" functionality.
[1] Suggesters: https://invenio-records-rest.readthedocs.io/en/latest/usage.html#suggesters
https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/41[MyCERN] Research the use of Semantic search2019-09-13T16:13:45+02:00Pablo Panero Vazquez[MyCERN] Research the use of Semantic searchResearch about the use of Semantic search on elasticsearch (e.g. "Where can I...", "Today's events...")Research about the use of Semantic search on elasticsearch (e.g. "Where can I...", "Today's events...")https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/39Investigate the use of the ranking API along with matomo analytics (And ML)2019-10-03T11:10:36+02:00Pablo Panero VazquezInvestigate the use of the ranking API along with matomo analytics (And ML)Document ranking can potentially be replaced by the Ranking API along with Matomo analytics values.
- [ES Ranking API](https://www.elastic.co/guide/en/elasticsearch/reference/6.2/search-rank-eval.html)
- [ES Learn to Rank](https://elast...Document ranking can potentially be replaced by the Ranking API along with Matomo analytics values.
- [ES Ranking API](https://www.elastic.co/guide/en/elasticsearch/reference/6.2/search-rank-eval.html)
- [ES Learn to Rank](https://elasticsearch-learning-to-rank.readthedocs.io/en/latest/)https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/38Investigate supporting bulk operations2019-09-13T16:40:27+02:00Pablo Panero VazquezInvestigate supporting bulk operationshttps://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/37Automate promoted results generation2019-09-13T16:18:40+02:00Pablo Panero VazquezAutomate promoted results generationPromoted results are generated by querying directly the API, images are stored in the source website/server. What happens if it fails? Should we generate a store for those images?
Automate the whole process with a script?Promoted results are generated by querying directly the API, images are stored in the source website/server. What happens if it fails? Should we generate a store for those images?
Automate the whole process with a script?https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/36test new invenio library of Elasticsearch ACLs2019-09-13T16:14:00+02:00Pablo Panero Vazqueztest new invenio library of Elasticsearch ACLsTest [invenio-explicit-acls](https://github.com/oarepo/invenio-explicit-acls)Test [invenio-explicit-acls](https://github.com/oarepo/invenio-explicit-acls)