Slow down crawl requests on drupal sites
/Closes https://gitlab.cern.ch/webservices/webframeworks-planning/-/issues/793
Composer will enforce the new directive on every robots.txt
.
/Closes https://gitlab.cern.ch/webservices/webframeworks-planning/-/issues/793
Composer will enforce the new directive on every robots.txt
.