merge web-crawler and web-indexer

• Evaluate performance of a Scrapy pipeline to send data directly to the Search API.
• Merge projects (one single app, maybe separate OpenShift projects? Same project different DCs?).
• Remove web-indexer project.