Magento 2 Set No Index No Follow

The robots.txt file is used for telling the web crawler which files to index and which not. By default, Magento generates robots.txt, you can update and configure it as per your requirements.

If your website is either staging or development and you wanted to stop crawl your website on the web to prevent duplicate site and content issues then you can easily do this from the Magento 2 admin.

Steps to Manage robots.txt from the Magento admin
Step 1: Log in to your Magento 2 Admin.
Step 2: Navigate to CONTENT > Configuration

Magento Design Configuration
Design Configuration – Magento 2 Admin

Step 3: Edit Design Configuration

Magento 2

Step 4: Navigate to the “Search Engine Robots” tab.
Step 5: Set NOINDEX, NOFOLLOW to “Default Robots” filed, and Save Configuration.

Magento Search Engine Robots
Search Engine Robots – Magento 2 Admin