Magento_Robots module

This module provides the following functionalities:

• contains a router to match application action class for requests to the robots.txt file
• allows obtaining the content of the robots.txt file depending on the settings of the current website