Shopify gives store owners ability to control robots.txt file better
By Matt Collins Article may include affiliate links
Shopify has rolled out the ability for store owners to edit their robots.txt file — an oft requested feature of the ecommerce platform.
The robots.txt file is a plain text file that typically exists at the root level of a domain (such as domain.com/robots.txt) and contains a set of directives on how robots, often referred to as a spiders, should access and process (or not) a site or select parts of it.
The new features began rolling out June 18, 2021 and should be available for all sites within days.
To be clear, Shopify has long provided its hosted stores with robots.txt files configured with default settings — but never allowed store owners to customize them.
Now, store owners can added a template file named robots.txt.liquid to their theme and use Liquid, documented by Shopify, to add or remove existing or new rules or add a custom one. Users can also optionally add a URL to an XML sitemap (though Shopify automatically generates one at /sitemap.xml, which is widely considered the best practice.
Shopify provides Liquid functionality that will output what it calls the “standard” rules, with the added ability to to use “for” and “if” statements to toggle individual rules or groups on and off.
Shopify notes that it is technically possible to forgo using Liquid at all and simply type your rules in the file as plain text, but it strongly suggests not doing that, likely to ensure that if SEO industry best practices update, Shopify can roll out the change across all its hosted sites.
Being able to edit robots.txt has long been a requested feature of Shopify store owners and has key implications for advanced SEO. It gives store owners a broader, easier to manage way to prevent search engines that respect robots.txt from indexing, caching or following certain parts of a site, files or other criteria.
It does not appear Shopify has added the ability for store owners to easily manage an ads.text file, a file used secure what advertising networks are “authorized” to run ads on a site, though this isn’t likely to become an issue for merchants unless they run third party ads, such as AdSense, on their site to make extra revenue (there’s also a workaround for this issue).
It’s worth noting that robots.txt are technically “directives” but not all spiders or bots will respect them, particularly ones with nefarious purposes. Shopify’s hosting infrastructure should, however, do a decent job of filtering out this type of traffic, especially high volume or known threats.