Indexing

Sitemap

A sitemap is the list of pages you want search engines to crawl. WordPress has generated one at /wp-sitemap.xml since version 5.5, so MetaWrench does not add a second one. It manages the one you already have: what goes in it, what address it lives at, and whether Google and Bing have been told about it.

The tab opens with the sitemap address detected on this site. If you have set a custom address below, it says so.

Settings

Sitemap address

Only change this if something other than WordPress makes your sitemap. Whatever is here is what gets written into robots.txt and submitted to search engines. Leave it at the detected address otherwise.

View sitemap

Opens the live file in a new tab so you can see what search engines see.

Submit to Google / Submit to Bing

Sends the sitemap address to Search Console and Bing Webmaster Tools. Needs the matching account connected under Connections → Reporting; without it, the button tells you so. Submitting is a one-off nudge, not a requirement — both crawlers find sitemaps through robots.txt on their own.

Post types

Every content type on the site, each with a checkbox. Unticked types are left out of the sitemap. Pages, posts and your service type should be on. Media, templates, reusable blocks and anything a page builder adds should be off — they are not pages a customer would land on, and including them tells Google they matter as much as your real content.

Taxonomies

Same idea for categories, tags and any custom groupings. Category archives are usually worth keeping. Tag archives on a small site are thin pages that dilute the sitemap; untick them.

When something looks wrong

The sitemap address shows a 404

WordPress rebuilds its URL rules when you save permalinks. Settings → Permalinks → Save Changes, then reload the sitemap. Nothing else needs changing.

A page is missing from the sitemap

Three things to check: its post type is ticked here; the page is published, not draft or private; and it is not set to noindex on its edit screen under the MetaWrench box. A noindex page is deliberately kept out.

A yellow mismatch notice at the top of the tab

Your robots.txt names a different sitemap address from the one set here. Open the Robots.txt tab and save it; the sitemap line is rewritten to match.

Submit to Google says nothing is connected

Connect Google under Connections → Reporting first. Search Console is one of the accounts that sign-in covers.

Another plugin also generates a sitemap

Two sitemaps at two addresses confuse crawlers. Deactivate the other plugin’s sitemap, or the whole plugin if that is all it does, and let WordPress produce the one MetaWrench manages.

Robots.txt

robots.txt is the file at the root of your site that tells crawlers where they may and may not go. MetaWrench writes it for you from a few settings, so you never edit the file by hand unless you want to. The WordPress admin and core folders are always blocked, and the sitemap line is kept in step with the Sitemap tab automatically.

Settings

Block all search engines

Adds Disallow: / for every crawler, hiding the whole site. The only legitimate use is a staging copy. When it is on, a red warning sits at the top of the tab on every visit, because forgetting it is how live sites vanish from Google. This is the same switch as WordPress’s own “Discourage search engines” setting under Settings → Reading.

Blocked paths

Folders or pages you want crawlers to skip, one per row, starting with a slash: /thank-you/, /internal/. Use + Add path to block for another row and the × to remove one. Blocking a path here does not remove it from Google if it is already indexed; for that, set the page to noindex on its edit screen and wait for a recrawl.

Always blocked

Shown for reference: /wp-admin/ and /wp-includes/. You cannot remove these, and there is no reason to.

Sitemap

Read-only. Shows the address that will be written into the file. Change it on the Sitemap tab.

Advanced — raw editor

Shows the file as it currently exists. Tick Save raw content instead of using the settings above to take over editing by hand. From then on the settings above are ignored and MetaWrench writes exactly what is in the box, including any mistakes. Reset to default hands control back to the settings.

Save robots.txt

Writes the file. View live file opens it to confirm.

When something looks wrong

“Could not write robots.txt — check file permissions”

The web server cannot write to your site’s root folder. Ask your host to make the root writable by the PHP user, or create an empty robots.txt there yourself with permissions 644 and save again.

Changes do not show at /robots.txt

Hard-refresh first; browsers cache this file aggressively. If it is still stale, a caching layer at your host or CDN is serving the old copy — purge it. If the file looks completely different from what you saved, another plugin is also writing robots.txt and winning; turn off its robots feature.

The whole site disappeared from Google

Check this tab for the red warning. If Block all search engines is on, turn it off, save, and request reindexing in Search Console. Recovery takes days to weeks depending on how long it was blocked.

I blocked a folder but its pages are still in results

robots.txt stops crawling, not indexing. Pages already known to Google stay until recrawled, and a blocked page cannot be recrawled, so it can linger. Unblock it, set the page to noindex on its edit screen, wait for Google to see the tag, then block again if you still want to.

AI / llms.txt

llms.txt is a plain-text file at /llms.txt that describes your business for AI tools the way robots.txt and a sitemap describe it for search engines. ChatGPT, Perplexity and Google’s AI Overviews read it to learn who you are, what you do and where, so they cite you correctly instead of guessing.

MetaWrench writes the file from your My Business answers: business name, description, address, phone, website, and the services and service areas you listed. You do not type any of it here.

Settings

The text box

Shows the generated file. Edit it and press Save llms.txt to override the generated version with your own text. Once overridden, changes in My Business no longer flow through to this file until you reset.

Reset to default

Throws away the override and goes back to generating from My Business.

View live file

Opens /llms.txt in a new tab.

When something looks wrong

The business details in the file are wrong

Fix them in My Business, not here. Editing here creates an override, which then stops picking up future My Business changes. If you already overrode it, press Reset to default after fixing My Business.

The file is nearly empty

My Business has not been filled in, or has no services and service areas. Complete the Basics, Location and Services tabs and the file fills itself.

/llms.txt shows a 404

Same cause as a sitemap 404: Settings → Permalinks → Save Changes to rebuild the URL rules.

Indexing defaults

Sets the robots meta tag every page and post gets unless its own edit screen says otherwise. The tag has two halves: whether the page may appear in search results (index / noindex) and whether crawlers may follow the links on it (follow / nofollow). Per-page overrides live in the MetaWrench box on each edit screen.

Settings

Index and Follow

The normal setting. Pages appear in results and their links are followed. Leave it here unless you have a specific reason.

Noindex and Follow

Pages are kept out of results but their links still pass authority. Used for sites that exist only to link somewhere else, which is rare.

Index and Nofollow

Pages appear but their links are not followed. Almost never the right site-wide setting.

Noindex and Nofollow

Hidden from search entirely. Only for a staging or private site.

When something looks wrong

Pages are dropping out of Google

Check this tab first. If it says anything other than Index and Follow, that is why. Set it back, save, and request reindexing of the affected pages in Search Console.

One page should be hidden but the rest indexed

Do not change this tab. Open that page, find the MetaWrench box, and untick Show in search results. The default stays for everything else.

Search Console says “excluded by noindex tag” on pages I want indexed

Either this tab is set to noindex, or those pages have the per-page switch off. Check both. If a second SEO plugin is also active it may be adding its own tag; deactivate it.