Why Is robots.txt Telling Crawlers “Come In”?

[53494d_]

This is your robots.txt file:

User-agent: *
Allow: /

Sitemap: https://mattersofsize.com/sitemap.php

Are you serious right now?

 

[53494d_]

Heads up about robots.txt — allows full-site crawling

I noticed the current robots.txt is:

User-agent: *
Allow: /

Sitemap: https://mattersofsize.com/sitemap.php

This effectively invites all crawlers (and scrapers) to index everything that’s publicly reachable.

If that’s intentional, feel free to ignore this. If not, a safer baseline would be something like:

User-agent: *
Disallow: /attachments/
Disallow: /account/
Disallow: /conversations/
Disallow: /search/
Disallow: /members/
Sitemap: https://mattersofsize.com/sitemap.php

• If any attachment/thumb directories aren’t enforced by login, they can be indexed.
• The sitemap gives bots a complete URL list, which is great for SEO but also for scrapers.
• Disallowing sensitive paths reduces leakage without blocking normal indexing of threads.
• Extra SEO note: Indexing profiles, attachments, and other non-thread pages can dilute keyword relevance and reduce ranking for your actual discussion content.

Happy to clarify. Goal here is to protect members content, not to nitpick.

Kind regards,
SIM [53494D]

 

[DLD]

@oldandlively can you do something about this? This is coming from our brother SIM

 

[oldandlively]

I received my directive earlier to propagate SEOs to encourage visits throughout the world. Yes, it was intentional after the site was down for the entire month, started back in mid of December 2024 to practically Feburary of 2025. The propagation was to reindex and rearchived the posts to the master archiving server for future data recovery. When there was no partial or full backups performed, we already saw the collapse of the forum due to missing posts as gaps. Luckily, we manage to capture posts up to October of 2024 before the platform OS crashed. I managed to salvage only 35% of the old posts from the master archivers before Lightning initiated the robots.txt block to minimize bandwidth, not data mining and scavenging. If you want to do a full block again, I have no issues engaging all blocks. But the SEOs and site visits will drop like a brick in the deep ocean.