Googlebot · duniya ki sabse badi rishtey wali aunty 👵🔍

She visits every house, writes everything in her diary, and when someone asks "accha ladka batao?" she decides whom to suggest first. Pick a page, pick its protection, and watch what she does. Padhle.example is our imaginary PW — every URL here is fictional.

1 · Discover hears about the house 2 · Crawl visits it 3 · Render reads what's inside 4 · Index writes it in the diary 5 · Rank decides who shows first

1 · Pick a page

2 · What protection does it have?

Rendering:
Who's visiting:

3 · What aunty did

    What people see on Google

    robots.txt tester — with PW's real robots.txt

    Two things everyone gets wrong:
    1. Disallow: /study is a prefix. It also blocks /study-abroad, /study-material, /studyXYZ. That is what Allow: /study-abroad is for.
    2. robots.txt is public. Writing Disallow: /secret-admin-2026 is handing the chor a map that says "tijori almari ke peeche hai, please mat dekhna".

    The whole thing in one table

    ToolAunty analogyWhat it doesWhat it does NOT do
    robots.txt DisallowBoard at the gateAsks polite bots not to crawl. Saves crawl budget (e.g. UTM duplicates).Doesn't remove from search. Not security. Hides your noindex.
    noindex (meta or X-Robots-Tag)"Diary mein mat likhna" noteRemoves the page from search results.Only works if Google is allowed to crawl the page and read the note.
    nofollow"Mere padosiyon ke baare mein mat poochna"Don't follow / pass credit through links on this page.Doesn't hide this page.
    Login / authTaalaActually private. The only real protection for private data.—
    canonical"Asli wala woh hai"Merges duplicates (UTM links, filters) into one URL.It's a strong hint, not a command.
    Search Console → RemovalsEmergency: aunty, abhi ke abhi page phaadoHides a URL from results fast, for about 6 months.Temporary. Fix the real cause (lock or noindex) too.

    Non-production, done properly (Next.js)

    // app/robots.ts — note: non-prod does NOT disallow, so bots can SEE the noindex header
    import type { MetadataRoute } from 'next'
    const isProd = process.env.APP_ENV === 'production'
    
    export default function robots(): MetadataRoute.Robots {
      if (!isProd) return { rules: { userAgent: '*', allow: '/' } }
      return {
        rules: { userAgent: '*', allow: '/', disallow: ['/study/', '/api/'] },
        sitemap: 'https://www.padhle.example/sitemap.xml',
      }
    }
    
    // next.config.ts — every non-prod response carries the note (HTML, PDFs, images…)
    async headers() {
      return isProd ? [] : [{ source: '/:path*', headers: [{ key: 'X-Robots-Tag', value: 'noindex, nofollow' }] }]
    }
    
    // app/layout.tsx — belt and braces, in the HTML itself
    export const metadata = { robots: isProd ? { index: true, follow: true } : { index: false, follow: false } }
    
    // …and put staging behind a login (SSO / VPN / Cloudflare Access). The header is the seatbelt. The lock is the lock.