Why Is robots.txt Telling Crawlers “Come In”?

53494d_

53494d_

New member
This is your robots.txt file:
Code: 
User-agent: *
Allow: /

Sitemap: https://mattersofsize.com/sitemap.php
Are you serious right now?
 
Heads up about robots.txt — allows full-site crawling

I noticed the current robots.txt is:

Code: 
User-agent: *
Allow: /

Sitemap: https://mattersofsize.com/sitemap.php

This effectively invites all crawlers (and scrapers) to index everything that’s publicly reachable.

If that’s intentional, feel free to ignore this. If not, a safer baseline would be something like:

Code: 
User-agent: *
Disallow: /attachments/
Disallow: /account/
Disallow: /conversations/
Disallow: /search/
Disallow: /members/
Sitemap: https://mattersofsize.com/sitemap.php

  • If any attachment/thumb directories aren’t enforced by login, they can be indexed.
  • The sitemap gives bots a complete URL list, which is great for SEO but also for scrapers.
  • Disallowing sensitive paths reduces leakage without blocking normal indexing of threads.
  • Extra SEO note: Indexing profiles, attachments, and other non-thread pages can dilute keyword relevance and reduce ranking for your actual discussion content.

Happy to clarify. Goal here is to protect members content, not to nitpick. 👍

Kind regards,
SIM [53494D]
 
I received my directive earlier to propagate SEOs to encourage visits throughout the world. Yes, it was intentional after the site was down for the entire month, started back in mid of December 2024 to practically Feburary of 2025. The propagation was to reindex and rearchived the posts to the master archiving server for future data recovery. When there was no partial or full backups performed, we already saw the collapse of the forum due to missing posts as gaps. Luckily, we manage to capture posts up to October of 2024 before the platform OS crashed. I managed to salvage only 35% of the old posts from the master archivers before Lightning initiated the robots.txt block to minimize bandwidth, not data mining and scavenging. If you want to do a full block again, I have no issues engaging all blocks. But the SEOs and site visits will drop like a brick in the deep ocean.
 
  • Like
Reactions: DLD
moshka said:
Es gibt Tage, da läuft einfach alles schief, besonders wenn man im regnerischen Vorarlberg festsitzt und schlechte Laune hat. Ich probierte spin mama casino aus, nachdem ich ein Video darüber gesehen hatte, und hoffte auf Besserung. Beim Poker lief es erst mittelmäßig, aber dann kam der große Durchbruch bei den Slots Online. Ich konnte meinen Einsatz vervielfachen und war am Ende des Tages mehr als zufrieden. Solche Momente sind Gold wert und machen den grauen Alltag in den Alpen sofort viel bunter und spannender.
Click to expand...
English if you can
 
You must log in or register to reply here.
Help Users

You haven't joined any rooms.

    You haven't joined any rooms.
    Back
    Top