Balancing web development and staying on top of SEO can be challenging. Just when you think you’ve solved everything, you return to where you left off only to find that something else isn’t working. I thought this was only an issue in the web development/software development sector, but it happens in SEO too.
I have been continuously making small, agile updates to my digital agency’s website to improve the end user’s UI and deliver as smooth a UX as possible. Along this journey, however, I ended up breaking my sitemap – the very file Google and other search engine crawlers rely on to view and index my site.
What is a sitemap and why is it important?
A sitemap is a list of internal links across your website that tells Google and other search engines about any changes or updates made. It helps identify new or existing pages, posts, or content. As you can see, it is extremely important for your site’s visibility.
Today, for example, I logged into Google Search Console to check some SEO analytics. As I was browsing, I discovered that my sitemaps had stopped working for over a couple of weeks. I was astonished.
Troubleshooting the issue
My first thought was that the server might be blocking XML files or restricting writes for important files such as sitemaps. After logging into the server, I turned off each feature in the firewall section, but still no luck.
The next step was to look at caching, as that can interrupt file access or server requests. I disabled the caching plug-in from the WordPress dashboard, but the issue wasn’t related to caching. Instead of turning off each feature individually, I disabled the whole plug-in for quicker results – but that didn’t work either.
Since I have been using Rank Math for SEO lately, I checked its settings to ensure the generated sitemaps weren’t the issue. They weren’t. I even disabled Rank Math in the WordPress plug-ins dashboard, but again, no success.
Finally, I used Copilot to help check all possibilities. The last resolution on the list was to check permalinks. It dawned on me that resetting permalinks flushes URL rewrites, and since I had been making lots of theme changes (as I mentioned earlier), I realised this would fix the issue. And it did.
What are permalinks?
A permalink (short for permanent link) is the stable, unchanging web address of a specific piece of content on your site – such as webpages, blog posts, categories, and archives.
Permalinks are the backbone of your site’s URL structure. Configuring them correctly improves SEO, enhances user experience, and prevents broken links.
Expanded Section
When troubleshooting SEO issues, one of the most overlooked but critical steps is to check your robots.txt file. This file acts as a set of instructions for search engine crawlers, telling them which areas of your site they can access and which should remain off‑limits.
In WordPress, the robots.txt file is usually auto‑generated if you haven’t created a physical one yourself. You can view it by visiting: www.yourwebsiteurl.com/robots.txt.
This is the default structure, and it’s generally safe. The most important line here is the Sitemap declaration, which points search engines directly to your sitemap. If this line is missing, or if the sitemap URL is broken, crawlers may struggle to index your site properly.
If you don’t see a robots.txt file at all, it means WordPress hasn’t generated one, or server rules are blocking access. In that case, you can create a physical robots.txt file in your site’s root directory or use an SEO plugin like Rank Math or Yoast to generate and customise it.
Always ensure that your robots.txt is not accidentally blocking important files such as /sitemap.xml or /sitemap_index.xml. A single misconfigured line can prevent Google from indexing your content, undoing months of SEO work.
If you have any further question you can always reach-out to us here.
One last thing, you can also check this reference that look like a great read to though it maybe to technical so I put in one less technical for less tech savvy users. You can also read it here.
Leave a Comment