newsupernova/vendor/magento/module-robots/README.md

The Robots module provides the following functionalities:

• contains a router to match application action class for requests to the robots.txt file;
• allows obtaining the content of the robots.txt file depending on the settings of the current website.