cern-search-rest-api issueshttps://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues2019-09-13T16:13:33+02:00https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/47Integrate CDS into CERN Search2019-09-13T16:13:33+02:00Pablo Panero VazquezIntegrate CDS into CERN SearchThere might be more than one possibility:
- CDS to push content to CERN Search
- Crawl CDS
- Since both have Elasticsearch underneath, agree on a data-model and use [cross-cluster search](https://www.elastic.co/guide/en/elasticsearch/re...There might be more than one possibility:
- CDS to push content to CERN Search
- Crawl CDS
- Since both have Elasticsearch underneath, agree on a data-model and use [cross-cluster search](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cross-cluster-search.html)
As discussed IRL with CDS managers, this might happen in the far future but is at the moment in the lowest priority.https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/112Bug: Multiple entries in table ObjectVersion with is_head=True2022-01-14T11:47:05+01:00Carina AntunesBug: Multiple entries in table ObjectVersion with is_head=TruePossibly a race condition since it affects a very small percentage of files eg. 0.1%
Leads to invalid state in those specific docs in which updates are not possible and return a 500 error. Breaks in the code in the queries of type `one_...Possibly a race condition since it affects a very small percentage of files eg. 0.1%
Leads to invalid state in those specific docs in which updates are not possible and return a 500 error. Breaks in the code in the queries of type `one_or_none` on the table ObjectVersion (key, bucket_id, is_head=True)
https://cern-search-sentry.web.cern.ch/searchaas/edms-search-prod/issues/97112/?query=one_or_noneCarina AntunesCarina Antuneshttps://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/111Bump tika (log4j vulnerability)2022-01-11T19:15:19+01:00Carina AntunesBump tika (log4j vulnerability)Version 2.0.0 forward breaks metadata in unpack/all (From 1.27) - Bug report: https://issues.apache.org/jira/browse/TIKA-3624
Block: Release 2.2.1Version 2.0.0 forward breaks metadata in unpack/all (From 1.27) - Bug report: https://issues.apache.org/jira/browse/TIKA-3624
Block: Release 2.2.1Carina AntunesCarina Antuneshttps://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/110Fix sorting order2022-03-17T16:04:33+01:00Carina AntunesFix sorting orderCheck indico which is fixedCheck indico which is fixedhttps://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/108Investigate support for bulk ingestion2021-07-14T16:49:54+02:00Carina AntunesInvestigate support for bulk ingestionhttps://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/107Investigate how to modularize Client mappings and configs2021-07-14T16:45:08+02:00Carina AntunesInvestigate how to modularize Client mappings and configsSimilar to ES
https://gitlab.cern.ch/ai/it-puppet-hostgroup-it_es/-/blob/qa/data/hostgroup/it_es/search/pub7.yaml
or https://gitlab.cern.ch/it-elasticsearch-projectSimilar to ES
https://gitlab.cern.ch/ai/it-puppet-hostgroup-it_es/-/blob/qa/data/hostgroup/it_es/search/pub7.yaml
or https://gitlab.cern.ch/it-elasticsearch-projecthttps://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/106[ASR] Add new client config2021-10-25T14:16:23+02:00Carina Antunes[ASR] Add new client configNoor Afshan FathimaNoor Afshan Fathimahttps://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/104Integrate invenio-index-migrator2022-03-17T16:06:05+01:00Carina AntunesIntegrate invenio-index-migratorAllow migration of indexes with no downtime
https://github.com/inveniosoftware/invenio-index-migrator/blob/master/tests/conftest.py#L54Allow migration of indexes with no downtime
https://github.com/inveniosoftware/invenio-index-migrator/blob/master/tests/conftest.py#L54https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/103Support post record/:id with external id2021-07-15T16:51:35+02:00Carina AntunesSupport post record/:id with external idCarina AntunesCarina Antuneshttps://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/102pin elasticsearch py2021-07-14T16:54:56+02:00Carina Antunespin elasticsearch pyDetails: https://mattermost.web.cern.ch/it-dep/pl/h86gj4o3oif8dpzi8ap4x8boeoDetails: https://mattermost.web.cern.ch/it-dep/pl/h86gj4o3oif8dpzi8ap4x8boeohttps://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/99Implement invenio-files-processor module with Tika processor2020-01-16T10:30:23+01:00Carina AntunesImplement invenio-files-processor module with Tika processorImplement invenio-files-processor module with Tika processor
https://github.com/inveniosoftware/invenio-files-processor/pull/11Implement invenio-files-processor module with Tika processor
https://github.com/inveniosoftware/invenio-files-processor/pull/11https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/98Allow parsing from binary streams in tika python2020-01-16T10:28:56+01:00Carina AntunesAllow parsing from binary streams in tika pythonAllow parsing from binary streams
https://github.com/chrismattmann/tika-python/pull/269Allow parsing from binary streams
https://github.com/chrismattmann/tika-python/pull/269https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/97[file-indexing] Add file processing queue + worker2020-01-16T09:40:50+01:00Carina Antunes[file-indexing] Add file processing queue + workerAdd file processing queue + Create celery worker to read from queue + Send file_uploaded to Queue
More info:
https://invenio-celery.readthedocs.io/en/latest/Add file processing queue + Create celery worker to read from queue + Send file_uploaded to Queue
More info:
https://invenio-celery.readthedocs.io/en/latest/Carina AntunesCarina Antuneshttps://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/96Investigate max capacity/ load test2019-12-05T15:25:30+01:00Carina AntunesInvestigate max capacity/ load testhttps://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/95Decision: File Indexing2020-01-20T09:42:33+01:00Carina AntunesDecision: File IndexingCarina AntunesCarina Antuneshttps://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/92Profile create record changes2019-11-25T10:20:49+01:00Carina AntunesProfile create record changesProfile create record changes due to https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/issues/85.
If negative impacts is found make files with buckets configurable.Profile create record changes due to https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/issues/85.
If negative impacts is found make files with buckets configurable.https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/90Logging: Debug logs broken inside docker / oc2019-11-21T11:24:03+01:00Carina AntunesLogging: Debug logs broken inside docker / ocFlags don't seem to work. Investigate and fix.Flags don't seem to work. Investigate and fix.https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/89test: ui/test_ui::test_view is failling2019-11-14T16:54:33+01:00Carina Antunestest: ui/test_ui::test_view is failling```
__________________________________________________________________________________________ test_view ___________________________________________________________________________________________
appctx = <Flask 'invenio'>, base_clien...```
__________________________________________________________________________________________ test_view ___________________________________________________________________________________________
appctx = <Flask 'invenio'>, base_client = <FlaskClient <Flask 'invenio'>>
@pytest.mark.unit
def test_view(appctx, base_client):
> resp = base_client.post("/account/settings/applications/", follow_redirects=True)
tests/ui/test_ui.py:17:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/werkzeug/test.py:1039: in post
return self.open(*args, **kw)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/testing.py:227: in open
follow_redirects=follow_redirects,
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/werkzeug/test.py:1017: in open
response, new_location, environ, buffered=buffered
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/werkzeug/test.py:948: in resolve_redirect
return self.open(builder, as_tuple=True, buffered=buffered)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/testing.py:227: in open
follow_redirects=follow_redirects,
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/werkzeug/test.py:993: in open
response = self.run_wsgi_app(environ.copy(), buffered=buffered)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/werkzeug/test.py:884: in run_wsgi_app
rv = run_wsgi_app(self.application, environ, buffered=buffered)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/werkzeug/test.py:1119: in run_wsgi_app
app_rv = app(environ, start_response)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/app.py:2463: in __call__
return self.wsgi_app(environ, start_response)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/app.py:2449: in wsgi_app
response = self.handle_exception(e)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/app.py:1866: in handle_exception
reraise(exc_type, exc_value, tb)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/_compat.py:39: in reraise
raise value
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/app.py:2446: in wsgi_app
response = self.full_dispatch_request()
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/app.py:1951: in full_dispatch_request
rv = self.handle_user_exception(e)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/app.py:1820: in handle_user_exception
reraise(exc_type, exc_value, tb)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/_compat.py:39: in reraise
raise value
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/app.py:1949: in full_dispatch_request
rv = self.dispatch_request()
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/app.py:1935: in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask_security/decorators.py:230: in wrapper
return f(*args, **kwargs)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask_security/views.py:89: in login
**_ctx('login'))
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask_security/core.py:528: in render_template
return render_template(*args, **kwargs)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/templating.py:140: in render_template
ctx.app,
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/flask/templating.py:120: in _render
rv = template.render(context)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/jinja2/asyncsupport.py:76: in render
return original_render(self, *args, **kwargs)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/jinja2/environment.py:1008: in render
return self.environment.handle_exception(exc_info, True)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/jinja2/environment.py:780: in handle_exception
reraise(exc_type, exc_value, tb)
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/jinja2/_compat.py:37: in reraise
raise value.with_traceback(tb)
/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/invenio_oauthclient/templates/invenio_oauthclient/login_user.html:11: in top-level template code
{%- from "invenio_oauthclient/_macros.html" import oauth_button %}
/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/invenio_accounts/templates/invenio_accounts/login_user.html:11: in top-level template code
{% from "invenio_accounts/_macros.html" import render_field, form_errors %}
/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/invenio_theme/templates/invenio_theme/page_cover.html:13: in top-level template code
{%- set body_css_classes=['cover-page'] %}
/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/invenio_theme/templates/invenio_theme/page.html:12: in top-level template code
{%- block head %}
/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/invenio_theme/templates/invenio_theme/page.html:47: in block "head"
{%- block css %}
/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/invenio_theme/templates/invenio_theme/page.html:48: in block "css"
{{ webpack['theme.css']}}
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/jinja2/environment.py:411: in getitem
return obj[argument]
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/werkzeug/local.py:378: in <lambda>
__getitem__ = lambda x, i: x._get_current_object()[i]
/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/werkzeug/local.py:307: in _get_current_object
return self.__local()
/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/flask_webpackext/proxies.py:22: in <lambda>
lambda: current_app.extensions['flask-webpackext'].manifest)
/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/flask_webpackext/ext.py:88: in manifest
join(self.app.static_folder, path))
/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/flask_webpackext/manifest.py:66: in load
super(JinjaManifestLoader, self).load(filepath)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <flask_webpackext.manifest.JinjaManifestLoader object at 0x7f270187df98>, filepath = '/opt/rh/rh-python36/root/usr/var/instance/static/dist/manifest.json'
def load(self, filepath):
"""Load a manifest from a file."""
> with open(filepath) as fp:
E FileNotFoundError: [Errno 2] No such file or directory: '/opt/rh/rh-python36/root/usr/var/instance/static/dist/manifest.json'
/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pywebpack/manifests.py:212: FileNotFoundError
```https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/87global: expose health status2019-11-06T10:51:34+01:00Pablo Panero Vazquezglobal: expose health statusExpose health status to the outside. This would allow clients to check if their search instance is up and running.
Need to aggregate the `/health/{es,db,uwsig}` (web app) + `/ping` (nginx) endpoint responses.Expose health status to the outside. This would allow clients to check if their search instance is up and running.
Need to aggregate the `/health/{es,db,uwsig}` (web app) + `/ping` (nginx) endpoint responses.https://gitlab.cern.ch/webservices/cern-search/cern-search-rest-api/-/issues/84global: linux incompatibilities2020-06-11T16:11:40+02:00Pablo Panero Vazquezglobal: linux incompatibilities* Docker-compose version is set to 3.7, the dockerfile is compatible with at leas 2.3 lower.
- Shall we provide support for 2.x? (The whole Invenio framework still users 2.
* `shell` might not be available in all systems.
- Requi...* Docker-compose version is set to 3.7, the dockerfile is compatible with at leas 2.3 lower.
- Shall we provide support for 2.x? (The whole Invenio framework still users 2.
* `shell` might not be available in all systems.
- Require it or avoid using it
- `/scripts/pipenv/requirements.sh` might crash if the command is run in another shell since `[[` is not always available (see [here](https://stackoverflow.com/questions/12230690/string-comparison-in-bash-not-found/12230723))