Popular on eTradeWire
- San Jose Holiday Lights Returns to Downtown San Jose for the 2026 Season - 5776
- San Jose Holiday Lights: Group Ticket Discounts for Companies, Schools, Organizations - 2035
- The Mako Girls Announce The "MAKO TAKEOVER CONCERT" Live at Center Stage Vinyl In Atlanta - 1193
- Pleasanton Welcomes The Great Junk Hunt – Discover Unique Vintage Treasures and Holiday Finds! - 733
- Eight Teams Set to Compete in the 2026 MDF Hoop House Tip-Off Classic at Tarrant High School - 610
- Dryer Vent Cleaning New Jersey: Freehold Families Urged to Check Airflow as Laundry Season Begins - 414
- Fivio Foreign Set to Drop on Friday Kingdom Music Single Release "You Make Me" single - 354
- Reggae on the River Returns to County Line June 25 - 27, 2027 - 287
- Okaloosa Recovery Centers Combine Oct. 26: What Niceville, Shalimar and FWB Homeowners Should Know - 255
- Red Carpet, Star Power and Comedy Come to Cranford for "Midlife Women: It's Your Opening Night!" - 223
Robots.txt one line AI training. Anthropic, Microsoft, Google and OpenAI back the concept
eTradeWire News/10853097
How Lab451 Generator handles it: an opt-in training preference for robots.txt. No preference is the default; choose to opt out of the whole site or all but some paths. When you opt out, the per-crawler training groups are written to match.
VICTORIA, British Columbia - eTradeWire -- Lab451 Helps Website Owners Get Ready for the New AI-Training Signal in robots.txt Backed by Microsoft, OpenAI, Google and Anthropic
Free tool generates, verifies and monitors robots.txt, sitemap.xml, llms.txt and llms-full.txt — now with support for the IETF draft '__usage_ai-training' token
One line instead of a growing block list
Until now, a publisher who did not want their content used to train AI models had to add a separate robots.txt rule for every AI crawler, and keep that list current as new crawlers appeared. In late October, the four companies announced they would honor a single shared signal instead: a policy token, '__usage_ai-training', placed in the User-Agent line of an ordinary robots.txt group. The proposal was written by Kevin Kelley, Anthropic's technical lead for standards, and submitted to the Internet Engineering Task Force (IETF) as an Internet-Draft.
More on eTradeWire News
A site that wants to opt out of AI training would add a group such as:
User-Agent: __usage_ai-training
Disallow: /
AI Training, Generate, Verify, Monitor
Availability
More on eTradeWire News
The Lab451 generator and checker are free at https://lab451.org. Monitoring is included in the Pro and Agency plans.
Media contact
Roger Remacle
hello@lab451.org
https://lab451.org
Free tool generates, verifies and monitors robots.txt, sitemap.xml, llms.txt and llms-full.txt — now with support for the IETF draft '__usage_ai-training' token
One line instead of a growing block list
Until now, a publisher who did not want their content used to train AI models had to add a separate robots.txt rule for every AI crawler, and keep that list current as new crawlers appeared. In late October, the four companies announced they would honor a single shared signal instead: a policy token, '__usage_ai-training', placed in the User-Agent line of an ordinary robots.txt group. The proposal was written by Kevin Kelley, Anthropic's technical lead for standards, and submitted to the Internet Engineering Task Force (IETF) as an Internet-Draft.
More on eTradeWire News
- Phoenix Brokers Brad and Sher Kelley Earn Seniors Real Estate Specialist® Designation
- A 35-Year-Old 16-Bit Holy Grail: The Elusive Twinkle Tale Finally Arrives in North America & Europe
- Scottsdale Philharmonic Announces Program for Annual Christmas Concerts
- Soul Singer Teri Tobin Brings an Intimate Holiday Music Experience to Stafford Tonight
- Mesa West Capital Originates $58.7 Million Loan for Industrial Portfolio Refinance in Glendale, AZ
A site that wants to opt out of AI training would add a group such as:
User-Agent: __usage_ai-training
Disallow: /
AI Training, Generate, Verify, Monitor
- AI Training (new): Lab451's generator offers the __usage_ai-training group as an opt-in option clearly labelled "IETF draft". The checker recognizes and validates the token in existing robots.txt files, monitoring alerts when it is added, removed or malformed, and a short explainer page walks site owners through what it means.
- Generate: Build four files from one crawl of a site — robots.txt, sitemap.xml, llms.txt and llms-full.txt — with robots.txt as the entry point that points search engines and AI agents to the other three.
- Verify: Every generated file is checked for dead links and hostnames, unregistered package names, and text hidden from human readers before it is published. The free Checker runs the same validation on files a site already publishes.
- Monitor: Pro and Agency plans watch published files six times a day, send a weekly report by email, and notify the owner immediately when a critical problem appears — such as a robots.txt change that accidentally blocks search engines or drops the AI-training preference.
Availability
More on eTradeWire News
- GIVING TUESDAY: Help Indigenous Families in Guatemala Build Healthy, Self-Reliant Futures
- CITE Research Launches New Website Showcasing PR Surveys and Global Market Research
- founder Technologies opens pre-registration for founder App, an all-in-one platform for founders
- Treasure Coast, Florida Real Estate Forecast 2026-2027
- Dr. Brandy Runyan Brings Athlete Advocacy and Mental Resilience Together Through Black Diamond & Th
The Lab451 generator and checker are free at https://lab451.org. Monitoring is included in the Pro and Agency plans.
Media contact
Roger Remacle
hello@lab451.org
https://lab451.org
Source: Lab451
0 Comments
Latest on eTradeWire News
- Schizophrenia Groups Recommend Just Published Thirty Year Caregiving Struggle
- Pear Sperling Eggan & Daniels, P.C. Explains Michigan's Laws on Handwritten Wills
- BeReal Wear launches its Black Friday event with up to 70% off
- Stay Frosty Enterprises Acquires U.S. Army License!
- iAmBeastNetwork Launches on Roku, Bringing an Independent Indianapolis Media Network
- Lead Empowered TV Joins Eagle-Research as an Affiliate Partner
- Women United Luncheon Celebrates Community Commitment to Early Literacy in Marion County
- Independent Scholar Yogesh Patel Proposes 'Spacenow': A Fifth Dimension Reinterpreting Spacetime
- When the Plant Closes, the Skills Don't: Three Steps for Niagara Bottling and HoMedics Workers
- Matt Connell's Learning Outside the Lines Calls on Educational Leaders to Rethink Broken Pathways
- Restore Childhood to Host Capitol Hill Screening on America's Pandemic School Closures December 8
- Australian Technology Company Introduces RaddeCare+ to Simplify NDIS Provider Management
- WebVM team launches free Linux MicroVM for AI coding agents
- Sell Rolex Oysterquartz London — The Forgotten Reference Making a Quiet Comeback
- Florida Keys Seafood Co. Launches Online Store, Shipping Stone Crab and Spiny Lobster Nationwide
- Hermès Kelly vs Birkin: Which Handbag Is Worth More on Resale?
- AI Can Now Write the Software. But Who Is Checking the Risks?
- Frobb Media Launches a Consultancy With No Discovery Call and SEO Expert in a Box for $19.99 a Month
- PixelPupil Launches English and Spanish Construction Crew Safety Training in the USA
- Contractor Websites HQ Launches Fixed-Price Websites for US Contractors