She visits every house, writes everything in her diary, and when someone asks "accha ladka batao?" she decides whom to suggest first. Pick a page, pick its protection, and watch what she does. Padhle.example is our imaginary PW — every URL here is fictional.
Disallow: /study is a prefix. It also blocks /study-abroad,
/study-material, /studyXYZ. That is what Allow: /study-abroad is for.Disallow: /secret-admin-2026 is handing the chor a map
that says "tijori almari ke peeche hai, please mat dekhna".
| Tool | Aunty analogy | What it does | What it does NOT do |
|---|---|---|---|
| robots.txt Disallow | Board at the gate | Asks polite bots not to crawl. Saves crawl budget (e.g. UTM duplicates). | Doesn't remove from search. Not security. Hides your noindex. |
| noindex (meta or X-Robots-Tag) | "Diary mein mat likhna" note | Removes the page from search results. | Only works if Google is allowed to crawl the page and read the note. |
| nofollow | "Mere padosiyon ke baare mein mat poochna" | Don't follow / pass credit through links on this page. | Doesn't hide this page. |
| Login / auth | Taala | Actually private. The only real protection for private data. | — |
| canonical | "Asli wala woh hai" | Merges duplicates (UTM links, filters) into one URL. | It's a strong hint, not a command. |
| Search Console → Removals | Emergency: aunty, abhi ke abhi page phaado | Hides a URL from results fast, for about 6 months. | Temporary. Fix the real cause (lock or noindex) too. |
// app/robots.ts — note: non-prod does NOT disallow, so bots can SEE the noindex header import type { MetadataRoute } from 'next' const isProd = process.env.APP_ENV === 'production' export default function robots(): MetadataRoute.Robots { if (!isProd) return { rules: { userAgent: '*', allow: '/' } } return { rules: { userAgent: '*', allow: '/', disallow: ['/study/', '/api/'] }, sitemap: 'https://www.padhle.example/sitemap.xml', } } // next.config.ts — every non-prod response carries the note (HTML, PDFs, images…) async headers() { return isProd ? [] : [{ source: '/:path*', headers: [{ key: 'X-Robots-Tag', value: 'noindex, nofollow' }] }] } // app/layout.tsx — belt and braces, in the HTML itself export const metadata = { robots: isProd ? { index: true, follow: true } : { index: false, follow: false } } // …and put staging behind a login (SSO / VPN / Cloudflare Access). The header is the seatbelt. The lock is the lock.