Meta's New Commercial Terms and Your robots.txt Choices

If you use Instagram, Facebook or WhatsApp for business, your Commercial Terms with Meta change on October 30. The updated text also covers crawling websites associated with a business and using that content in Meta products. How broadly the clause will be read is debated. Which Meta crawlers your site allows is something you can already set today in robots.txt.

What happened

In early October, Meta announced in Meta Business Suite and on a help page that its Commercial Terms will change on October 30, 2026. The update folds the Product Catalog Terms into the Commercial Terms, which Meta says reduces overlap between its agreements.

The preview of the terms defines commercial use broadly. Using ads or marketing messages, selling or listing products, creating or using a product catalog, managing a Page, and using business tools such as Ads Manager or Meta Business Suite all count. The clause at the center of the debate says, in short:

• The business authorizes Meta, or any entity acting on its behalf, to use scrapers, bots, crawlers and similar automated software, and to create test credentials.
• This covers domains, pages and URLs associated with the business’s products or services, or made available through the business’s or users’ use of Meta products. It includes accessing, indexing, caching and analyzing them, and periodically pulling updates.
• Meta may use that content to provide, develop, improve, research, train and maintain the integrity of the products and services covered by the Commercial Terms.

Continuing commercial use after the update means accepting the terms; a business that disagrees may stop. The text doesn’t define when a website counts as “associated.”

What lawyers say, and what Meta says

Nieman Lab asked three legal experts to review the terms for a story published on October 15. According to them, the terms as written could be used to argue that Meta may scrape news sites and train its AI products on that content. They also said the new language gives Meta an opening to argue the terms supersede other blocking instructions. Cornell’s James Grimmelmann said test credentials could technically let Meta sign up as a fake user to get past registration walls. One of the lawyers said it isn’t clear whether a court would enforce the clause, but that publishers should assume it would be enforceable.

Meta’s account is different. A spokesperson told Nieman Lab the update consolidates crawling language that was previously scattered across several terms, that it won’t change how Meta crawls publishers, and that “content owners can still manage how their content is accessed by using robots.txt.” Meta declined to say whether test credentials would bypass paywalls.

The two views don’t exclude each other. Meta is describing its current practice; the lawyers are looking at what the contract text permits. That also leaves open how much legal weight robots.txt rules would carry against the contract. For its own crawlers, the method Meta points to is robots.txt.

What it means for your brand

Nieman Lab focuses on publishers, and the story itself notes that journalism isn’t the only industry affected. An exporter listing products on Instagram and a brand running Meta ads both fall under the terms’ definition of commercial use.

Start by looking at what your site holds: blog posts, reports, guides, areas behind a free sign-up, dealer portals, documents behind a form. Which of these should be open to AI training is a decision for your brand. If your robots.txt has no rules for these crawlers, they can crawl the whole site by default.

What doesn’t change

Meta’s crawlers are not new. Meta’s official crawler documentation already said, before this update, that Meta-ExternalAgent crawls for purposes such as training foundation AI models.

robots.txt doesn’t work as an access lock either. Webflow’s help center notes the file is public and not every bot follows it. Content that must stay private belongs behind a login and access controls. Given the lawyers’ warning about test credentials, areas anyone can join with a free sign-up offer weak protection here.

What you can do

Read the terms

The terms take effect on October 30. Read the preview with your team and get your legal counsel’s view.

State your Meta preferences in robots.txt

Meta’s documentation lists each crawler separately:

• meta-externalagent: crawls for uses such as training foundation AI models or improving products by indexing content directly.
• meta-webindexer: crawls for Meta AI search results. Meta says allowing it helps Meta AI cite and link to your content.
• meta-externalads: crawls for advertising and other business products.
• facebookexternalhit: fetches the title, description and thumbnail of links shared on Facebook, Instagram or Messenger.
• meta-externalfetcher: fetches individual links at a user’s request; Meta says it may bypass robots.txt.

An example that opts out of AI training but keeps Meta AI search and link previews open:

User-agent: meta-externalagent
Disallow: /

User-agent: meta-webindexer
Allow: /

User-agent: facebookexternalhit
Allow: /

Once a crawler has its own group, it no longer follows the rules under User-agent: *. If your general group blocks folders (for example Disallow: /admin/), add the same lines to these groups too.

If you block facebookexternalhit, link previews break. Per Meta’s documentation, if the crawler can’t fetch your content within a few seconds, Facebook can’t display it. A blanket Disallow: / under User-agent: * also shuts it out, along with search engines, unless it has its own group. Meta says changes to robots.txt can take up to 24 hours to take effect.

Set it in Webflow

In Webflow, robots.txt rules go under Site settings > SEO > Indexing; save, then republish the site. Webflow says adding robots.txt requires a Site plan or a paid Workspace plan. A Content-Signal header can also be set there, but Webflow notes it isn’t an accepted standard yet. Meta’s documentation likewise points to robots.txt rather than non-standard formats such as NoAI tags.

Weigh visibility against protection

Blocking every AI crawler can also reduce your chances of appearing in AI answers. You can treat training and search crawlers separately; the example above is one way to do that. Allowing training is also an option. The decision depends on what the content is worth to you and how much visibility you want. We covered AI search visibility (AEO) in our Webflow MCP post and Meta’s AI shopping agents in our Meta Muse post.

Our recommendation at Klyne

We recommend making the robots.txt decision alongside your site’s content map. As a Webflow Certified Partner, we handle robots.txt and indexing settings as part of launch on our Webflow web design projects, and can set your preferences for Meta and other AI crawlers on your existing Webflow site.

This post is not legal advice. Talk to your legal counsel about how the terms affect your organization.

FAQ

Should I shut down my Instagram business account?

That is a business and legal decision. Nieman Lab reports there is no way to opt out of the new terms before October 30. The terms cover commercial use of Meta products as a whole, not just Instagram. Review them with counsel before you decide.

Does robots.txt reliably stop Meta’s crawlers?

No. robots.txt is an instruction to crawlers and doesn’t technically block access. Meta’s own documentation notes that meta-externalfetcher, and facebookexternalhit during security or integrity checks, may bypass it.

If I block meta-externalagent, will I disappear from Meta AI?

robots.txt rules apply per user agent, so a rule for meta-externalagent doesn’t cover meta-webindexer. According to Meta, allowing meta-webindexer helps Meta AI cite your content. Meta also says it uses meta-externalagent to improve products by indexing content directly, but it doesn’t say which products would be affected if you block it.

If you’d like to review which content on your site is open to which crawler, get in touch.