Slow down crawl requests on drupal sites

Francisco Borges Aurindo Barrosrequested to merge
slow-crawl into master

/Closes https://gitlab.cern.ch/webservices/webframeworks-planning/-/issues/793

Composer will enforce the new directive on every robots.txt.

Merge request reports