Commit fb8adef3 authored by Pablo Panero's avatar Pablo Panero
Browse files

Merge branch 'dev' into 'master'

[skip ci] GitHub publishing

See merge request webservices/cern-search/cern-search-rest-api!49
parents 43d371db 6a44810c
......@@ -18,11 +18,28 @@ variables:
### By default, there are 6 stages that we may use:
### Feel free to adapt this to your specific case.
stages:
- build_base
- build
- tag_image
- import_image # This stage is only used when the built image is stored in the GitLab Registry
- deploy
### 'Build Base' stage
### Builds the base docker image and tags it according to the commit message.
build_base_image:
stage: build_base
only:
changes:
- Dockerfile-base
- Pipfile
- Pipfile.lock
tags:
- docker-image-build
script: 'echo "Building base Docker image..."'
variables:
DOCKER_FILE: Dockerfile-base
TO: gitlab-registry.cern.ch/webservices/cern-search/cern-search-rest-api/cern-search-rest-api-base:${CI_COMMIT_SHA}
### 'Build' stage
### Build the image and store it in the registry. It is important that this step
### doesn't override the image the applications are running, as we haven't tested the image yet
......@@ -30,12 +47,19 @@ stages:
build_dev_version:
stage: build
except:
- tags
- master
refs:
- tags
- master
changes:
- Dockerfile-base
- Pipfile
- Pipfile.lock
environment: staging
tags:
- docker-image-build
script: 'echo "Building Dev/QA Docker image..."'
variables:
BUILD_ARG: build_devel=1
########## PROD ##########
build_prod_version:
......@@ -87,6 +111,11 @@ tag_image_prod:
### Import image into OpenShift. Import $CI_COMMIT_TAG if present or 'latest' if not.
import_image_dev:
stage: import_image
except:
changes:
- Dockerfile\-base
- Pipfile
- Pipfile.lock
environment: staging
image: gitlab-registry.cern.ch/paas-tools/openshift-client:latest
script:
......
# How to contribute
Guidelines on how to contribute are explained in the official documentation, they can be found [here](http://cernsearchdocs.web.cern.ch/cernsearchdocs/contributing/).
\ No newline at end of file
This diff is collapsed.
# -*- coding: utf-8 -*-
#
# This file is part of CERN Search.
# Copyright (C) 2018-2019 CERN.
#
# CERN Search is free software; you can redistribute it and/or modify it
# under the terms of the MIT License; see LICENSE file for more details.
# Use CentOS7:
FROM cern/cc7-base
# Install pre-requisites
RUN yum update -y && \
yum install -y epel-release && \
yum install -y \
python-devel \
python-pip \
gcc \
openssl \
npm \
openldap-devel && \
pip install --upgrade pip setuptools wheel
ADD requirements.txt /tmp
RUN pip install -r /tmp/requirements.txt
FROM gitlab-registry.cern.ch/webservices/cern-search/cern-search-rest-api/cern-search-rest-api-base:d1f96a1c3600a9a00e76433b593d0edc0a49e532
ARG build_devel
ENV DEVEL=$build_devel
# CERN Search installation
WORKDIR /code
ADD . /code
ENV INVENIO_INSTANCE_PATH=/usr/local/var/cernsearch/var/cernsearch-instance
ENV LOGO_PATH=/images/cernsearchicon.png
RUN chmod g=u /etc/passwd && \
chmod +x /code/scripts/*.sh && \
sh /code/scripts/create-instance.sh && \
sh /code/scripts/gen-cert.sh && \
chmod +x /code/scripts/patch/oauth_patch.sh && \
sh /code/scripts/patch/oauth_patch.sh && \
mv nginx.crt nginx.key ${INVENIO_INSTANCE_PATH} && \
chgrp -R 0 ${INVENIO_INSTANCE_PATH} && \
chmod -R g=u ${INVENIO_INSTANCE_PATH} &&\
adduser --uid 1000 invenio --gid 0 && \
chown -R invenio:root /code
WORKDIR /${WORKING_DIR}/src
ADD . /${WORKING_DIR}/src
# If env is development, install development dependencies
RUN if [ -n "${DEVEL-}" ]; then pip install -r requirements-devel.txt; fi
# Install CSaS
RUN pip install -e .
# PID File for uWSGI
RUN touch /${WORKING_DIR}/src/uwsgi.pid
RUN chmod 666 /${WORKING_DIR}/src/uwsgi.pid
# Patch auth
RUN sh /${WORKING_DIR}/src/scripts/patch/oauth_patch.sh
# Install UI
USER invenio
RUN invenio collect -v
RUN invenio webpack buildall
# Move static files to instance folder
RUN cp /${WORKING_DIR}/src/static/images/cernsearchicon.png ${INVENIO_INSTANCE_PATH}/static/images/cernsearchicon.png
EXPOSE 5000
# uWSGI configuration
ARG UWSGI_WSGI_MODULE=cern_search_rest_api.wsgi:application
......@@ -47,8 +48,4 @@ ENV UWSGI_PROCESSES ${UWSGI_PROCESSES:-2}
ARG UWSGI_THREADS=2
ENV UWSGI_THREADS ${UWSGI_THREADS:-2}
USER 1000
EXPOSE 5000
CMD ["/bin/sh", "-c", "/code/scripts/manage-user.sh && uwsgi --module ${UWSGI_WSGI_MODULE} --socket 0.0.0.0:${UWSGI_PORT} --master --processes ${UWSGI_PROCESSES} --threads ${UWSGI_THREADS} --stats /tmp/stats.socket"]
\ No newline at end of file
CMD ["/bin/bash", "-c", "uwsgi --module ${UWSGI_WSGI_MODULE} --socket 0.0.0.0:${UWSGI_PORT} --master --processes ${UWSGI_PROCESSES} --threads ${UWSGI_THREADS} --stats /tmp/stats.socket"]
\ No newline at end of file
# -*- coding: utf-8 -*-
#
# This file is part of CERN Search.
# Copyright (C) 2018-2019 CERN.
#
# CERN Search is free software; you can redistribute it and/or modify it
# under the terms of the MIT License; see LICENSE file for more details.
# Use CentOS7:
FROM inveniosoftware/centos7-python:3.6
# Install pre-requisites
RUN yum update -y && \
yum install -y \
gcc \
openssl \
openldap-devel \
https://linuxsoft.cern.ch/cern/centos/7/cern/x86_64/Packages/CERN-CA-certs-20180516-1.el7.cern.noarch.rpm
# CERN Search installation
WORKDIR /${WORKING_DIR}/src
ADD ./Pipfile /${WORKING_DIR}/src/Pipfile
ADD ./Pipfile.lock /${WORKING_DIR}/src/Pipfile.lock
# Install dependencies globally
RUN pipenv install --system --deploy
Installing CERNSearch requires the following services to be running:
- Elasticsearch version>=5.4.0
- PostregSQL/Mariadb/SQLite support
- Redis-server
- RabbitMQ/Another ActiveMQ implementation
Configuration is set to default values (e.g. ip:port pairs) if not, change in Config.py.
In order to start the system run the following commands (just the first time):
- invenio db init
- invenio db create
- invenio index init
- invenio index queue init (Not needed, Bulk indexing support is not yet there)
Then simply start the application with:
- invenio run \[--host 0.0.0.0\]
Note: if you do not have a valid certificate to run over SSL, make sure to have the environment variabel "FLASK_DEBUG"
set to 1.
MIT License
Copyright (C) 2018-2019 CERN.
Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in
the Software without restriction, including without limitation the rights to
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
the Software, and to permit persons to whom the Software is furnished to do so,
subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
\ No newline at end of file
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true
[dev-packages]
[packages]
invenio-access = ">=1.0.0,<1.1.0"
invenio-admin = ">=1.0.0,<1.1.0"
invenio-accounts = ">=1.0.0,<1.1.0"
invenio-app = ">=1.1.0,<1.2.0"
invenio-base = ">=1.0.0,<1.1.0"
invenio-config = ">=1.0.0,<1.1.0"
invenio-db = {version = ">=1.0.0,<1.1.0",extras = ["postgresql", "versioning"]}
invenio-indexer = {version = ">=1.0.0,<1.1.0",extras = ["elasticsearch6"]}
invenio-jsonschemas = ">=1.0.0,<1.1.0"
invenio-logging = ">=1.0.0,<1.1.0"
invenio-records-rest = {version = ">=1.3.0,<1.4.0",extras = ["elasticsearch6"]}
invenio-records = {version = ">=1.0.0,<1.1.0",extras = ["postgresql"]}
invenio-rest = {version = ">=1.0.0,<1.1.0",extras = ["cors"]}
invenio-oauthclient = ">=1.0.0,<1.1.0"
invenio-oauth2server = ">=1.0.0,<1.1.0"
invenio-search = {version = "==1.1.0",extras = ["elasticsearch6"]}
invenio-theme = ">=1.1.0,<1.2.0"
ldap = "<1.1.0,>=1.0.2"
npm = ">=0.1.1"
python-ldap = "<3.2.0,>=3.1.0"
raven = ">=6.9.0,<6.10.0"
redis = ">=2.10.0,<3.0.0"
uwsgi-tools = ">=1.1.1,<1.2.0"
Flask = "*"
uWSGI = ">=2.0.16"
[requires]
python_version = "3.4"
This diff is collapsed.
This diff is collapsed.
#!/usr/bin/python
# -*- coding: utf-8 -*-
# Copyright (C) 2018, CERN
# This software is distributed under the terms of the GNU General Public
# Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING".
# In applying this license, CERN does not waive the privileges and immunities
# granted to it by virtue of its status as Intergovernmental Organization
# or submit itself to any jurisdiction.
#
# This file is part of CERN Search.
# Copyright (C) 2018-2019 CERN.
#
# CERN Search is free software; you can redistribute it and/or modify it
# under the terms of the MIT License; see LICENSE file for more details.
from __future__ import absolute_import, print_function
......
#!/usr/bin/python
# -*- coding: utf-8 -*-
# Copyright (C) 2018, CERN
# This software is distributed under the terms of the GNU General Public
# Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING".
# In applying this license, CERN does not waive the privileges and immunities
# granted to it by virtue of its status as Intergovernmental Organization
# or submit itself to any jurisdiction.
#
# This file is part of CERN Search.
# Copyright (C) 2018-2019 CERN.
#
# CERN Search is free software; you can redistribute it and/or modify it
# under the terms of the MIT License; see LICENSE file for more details.
from __future__ import absolute_import, print_function
import ast
import copy
import os
from flask import request
from invenio_oauthclient.contrib import cern
from invenio_records_rest import config as irr_config
from invenio_records_rest.facets import terms_filter
from .modules.cernsearch.permissions import (record_read_permission_factory,
record_create_permission_factory,
......@@ -25,7 +28,6 @@ def _(x):
"""Identity function used to trigger string extraction."""
return x
# Theme
# =====
THEME_SEARCHBAR = False
......@@ -50,6 +52,7 @@ OAUTHCLIENT_REMOTE_APPS = dict(
# FIXME: Needs to be disable for role base auth in SSO. If not invenio_account/sessions:login_listener will crash
ACCOUNTS_SESSION_ACTIVITY_ENABLED = False
SERVER_NAME = os.getenv("CERN_SEARCH_SERVER_NAME")
# Admin
# =====
......@@ -64,20 +67,12 @@ JSONSCHEMAS_HOST = '0.0.0.0'
# Do not register the endpoints on the UI app."""
JSONSCHEMAS_REGISTER_ENDPOINTS_UI = True
# Indexer
# =======
# TODO use ES central service. Change INDEXER_RECORD_TO_INDEX = 'invenio_indexer.utils.default_record_to_index'
INDEX_PREFIX = os.getenv('CERN_SEARCH_INDEX_PREFIX', 'cernsearch')
INDEXER_DEFAULT_DOC_TYPE = os.getenv('CERN_SEARCH_DEFAULT_DOC_TYPE', 'doc_v0.0.1')
INDEXER_DEFAULT_INDEX = os.getenv('CERN_SEARCH_DEFAULT_INDEX', 'cernsearch-test-doc_v0.0.1')
# Search configuration
# =====================
SEARCH_MAPPINGS = [os.getenv('CERN_SEARCH_INSTANCE', 'cernsearch-test')]
SEARCH_USE_EGROUPS = ast.literal_eval(os.getenv('CERN_SEARCH_USE_EGROUPS', 'True'))
SEARCH_DOC_PIPELINES = ast.literal_eval(os.getenv('CERN_SEARCH_DOC_PIPELINES', '{}'))
# Records REST configuration
# ===========================
......@@ -103,7 +98,8 @@ RECORDS_REST_ENDPOINTS = dict(
},
record_loaders={
'application/json': ('cern_search_rest_api.modules.cernsearch.loaders:'
'csas_loader')
'csas_loader'),
'application/json-patch+json': lambda: request.get_json(force=True)
},
search_class='cern_search_rest_api.modules.cernsearch.search.RecordCERNSearch',
search_index=os.getenv('CERN_SEARCH_INSTANCE', 'cernsearch-test'),
......@@ -111,6 +107,7 @@ RECORDS_REST_ENDPOINTS = dict(
'application/json': ('invenio_records_rest.serializers'
':json_v1_search'),
},
search_factory_imp='cern_search_rest_api.modules.cernsearch.search.csas_search_factory',
max_result_window=10000,
read_permission_factory_imp=record_read_permission_factory,
list_permission_factory_imp=record_list_permission_factory,
......@@ -120,12 +117,38 @@ RECORDS_REST_ENDPOINTS = dict(
)
)
RECORDS_REST_FACETS = {
'cernsearchqa-webservices': {
'aggs': {
'collection': {
'terms': {'field': 'collection'}
}
},
'post_filters': {
'collection': terms_filter("collection")
}
}
}
RECORDS_REST_ELASTICSEARCH_ERROR_HANDLERS = copy.deepcopy(
irr_config.RECORDS_REST_ELASTICSEARCH_ERROR_HANDLERS)
RECORDS_REST_ELASTICSEARCH_ERROR_HANDLERS['mapper_parsing_exception'] = \
'cern_search_rest_api.modules.cernsearch.views:elasticsearch_mapper_parsing_exception_handler'
# App
# ===
RATELIMIT_DEFAULT = os.getenv('CERN_SEARCH_INSTANCE_RATELIMIT', '5000/hour')
APP_HEALTH_BLUEPRINT_ENABLED = False
# CORS
# ====
REST_ENABLE_CORS = True
CORS_SEND_WILDCARD = ast.literal_eval(os.getenv('CERN_SEARCH_CORS_SEND_WILDCARD', 'False'))
CORS_SUPPORTS_CREDENTIALS = ast.literal_eval(os.getenv('CERN_SEARCH_CORS_SUPPORTS_CREDENTIALS', 'True'))
# Flask Security
# ==============
# Avoid error upon registration with email sending
......@@ -135,3 +158,4 @@ SECURITY_CONFIRM_REGISTRATION = False
SECURITY_CONFIRMABLE = False
SECURITY_REGISTERABLE = False # Avoid user registration outside of CERN SSO
SECURITY_RECOVERABLE = False # Avoid user password recovery
SESSION_COOKIE_SECURE = True
#!/usr/bin/python
# -*- coding: utf-8 -*-
# Copyright (C) 2018, CERN
# This software is distributed under the terms of the GNU General Public
# Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING".
# In applying this license, CERN does not waive the privileges and immunities
# granted to it by virtue of its status as Intergovernmental Organization
# or submit itself to any jurisdiction.
\ No newline at end of file
#
# This file is part of CERN Search.
# Copyright (C) 2018-2019 CERN.
#
# CERN Search is free software; you can redistribute it and/or modify it
# under the terms of the MIT License; see LICENSE file for more details.
\ No newline at end of file
#!/usr/bin/python
# -*- coding: utf-8 -*-
# Copyright (C) 2018, CERN
# This software is distributed under the terms of the GNU General Public
# Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING".
# In applying this license, CERN does not waive the privileges and immunities
# granted to it by virtue of its status as Intergovernmental Organization
# or submit itself to any jurisdiction.
\ No newline at end of file
#
# This file is part of CERN Search.
# Copyright (C) 2018-2019 CERN.
#
# CERN Search is free software; you can redistribute it and/or modify it
# under the terms of the MIT License; see LICENSE file for more details.
\ No newline at end of file
#!/usr/bin/python
# -*- coding: utf-8 -*-
# Copyright (C) 2018, CERN
# This software is distributed under the terms of the GNU General Public
# Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING".
# In applying this license, CERN does not waive the privileges and immunities
# granted to it by virtue of its status as Intergovernmental Organization
# or submit itself to any jurisdiction.
#
# This file is part of CERN Search.
# Copyright (C) 2018-2019 CERN.
#
# CERN Search is free software; you can redistribute it and/or modify it
# under the terms of the MIT License; see LICENSE file for more details.
""" Record API"""
from invenio_records.api import Record
......
#!/usr/bin/python
# -*- coding: utf-8 -*-
# Copyright (C) 2018, CERN
# This software is distributed under the terms of the GNU General Public
# Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING".
# In applying this license, CERN does not waive the privileges and immunities
# granted to it by virtue of its status as Intergovernmental Organization
# or submit itself to any jurisdiction.
\ No newline at end of file
#!/usr/bin/python
# -*- coding: utf-8 -*-
# Copyright (C) 2018, CERN
# This software is distributed under the terms of the GNU General Public
# Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING".
# In applying this license, CERN does not waive the privileges and immunities
# granted to it by virtue of its status as Intergovernmental Organization
# or submit itself to any jurisdiction.
\ No newline at end of file
#!/usr/bin/python
# -*- coding: utf-8 -*-
# Copyright (C) 2018, CERN
# This software is distributed under the terms of the GNU General Public
# Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING".
# In applying this license, CERN does not waive the privileges and immunities
# granted to it by virtue of its status as Intergovernmental Organization
# or submit itself to any jurisdiction.
\ No newline at end of file
#!/usr/bin/python
# -*- coding: utf-8 -*-
# Copyright (C) 2018, CERN
# This software is distributed under the terms of the GNU General Public
# Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING".
# In applying this license, CERN does not waive the privileges and immunities
# granted to it by virtue of its status as Intergovernmental Organization
# or submit itself to any jurisdiction.
\ No newline at end of file
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment