We are cleaning this up far too often. Not once or twice a year, but as a steady stream of work, and almost always with the same key themes.

A brand invests in a redesign or a replatform. The dev agency ships on time. The site is faster, the codebase is modern, the navigation is tidy, and everybody is pleased with themselves at launch. Then organic traffic starts sliding, and it keeps sliding, and six months later the brand is being told it must have been a Google update.

It almost never is. It is usually poor decisions made during the build, with little or no regard for SEO best practice. You know, the channel that usually drives the lion’s share of traffic and revenue. These are just some examples, but critical ones we have started the clean up for a major UK brand who have been sold a website that simply isn’t fit for search.

On this page

• The redirect map nobody owned
• Key content served client-side
• Content cut without any review of what it was earning
• Pages orphaned to keep the menu tidy
• Why this keeps happening
• The AI layer makes all of this more expensive
• What to insist on before the next launch
• The part that frustrates me

The redirect map nobody owned

The most recent one we picked up had 13,844 redirect rules spread across three separate sources, two config files and the CMS, with no single owner and no order of precedence anyone could explain. Inside that set were 702 rules pointing at URLs that no longer resolve, 1,403 chains where one redirect fires into another, and 262 rules that can never fire at all because something earlier in the stack catches the request first.

This can be typical of an enterprise website with a heavy stack, but there are ways to manage this effectively. But at the heart of any migration, should be the clean up and organisation of these key files, as they hold the key to maintained (and potentially improved) organic search performance.

Google’s own migration documentation is pretty clear on this. Use server-side permanent redirects, send the old URL straight to its final destination, and if a chain is unavoidable keep it to three hops at most, because chains add latency and not every user agent will follow a long one. Every hop is a chance for the signal to be diluted or dropped, and a redirect that lands on a dead URL passes nothing at all.

The usual excuse is that the rules were inherited. That is exactly the point. Nobody inherits a redirect map cleanly. It has to be audited, deduplicated, and flattened, and that work needs to happen before launch rather than eighteen months later when someone finally asks why the old category URLs are worth nothing. This an an SEO fundamental and has been for decades, and although the naysayers are telling you that technical SEO has had it’s day, you can see quite clearly below the negative impact that ignoring this has on performance.

Key content served client-side

On one recent audit, 61.9% of the content elements on a core template were client-side rendered. Nearly two-thirds of the page did not exist until JavaScript ran. Headings, body copy, links, and list content, all arriving after hydration.

Google can usually cope with this, bit still tends to result in decreased performance certainly in the short term. It crawls the raw HTML, queues the page, renders it in a headless browser, and indexes what comes back. Slower, less reliable, but generally survivable.

AI crawlers cannot cope with it at all. The Vercel and MERJ analysis of real crawler traffic found that none of the major AI crawlers execute JavaScript. GPTBot and ClaudeBot both download JavaScript files, roughly 11.5% and 23.8% of their requests respectively, and never run them. They fetch the HTML, take what is in it, and leave. Andre Alpar’s great research earlier this summer even suggested that Google’s own AI crawler does not render JavaScript and the only notable ones that do are DeepSeek and Mistral. Which was interesting:

As we know, client side rendered content has serious challenges when it comes to classic search, but it doesn’t stand a chance when it comes to ChatGPT, Claude, and Perplexity et al. Content strategy, advanced technical SEO, entity optimisation and structured data, all these things are pretty pointless if the content isn’t retrievable in the first place.

This is why retrievability sits at the front of the AIR SEO framework.

Brutal culling of content to make things “tidy”

Content pruning is a legitimate tactic and one we have great success with when done pragmatically. By analysing traffic, rankings, conversions, backlinks, and internal link value for every UR, you can remove the dead wood and tackle a range of things from cannibalisation to engagement. But done lazily and just to make things look tidy, and you can all but destroy organic performance.

What usually happens instead is a page count. The new site has a cleaner information architecture, the old site had 400 pages that do not fit it, so those 400 pages go. Nobody checks which ones were ranking, which ones carried the external links, or which ones were quietly picking up long tail demand that added up to real revenue. Sometimes, these pages just needed a bit of love and updating to match the latest volume, intent and demand.

Then the pages that survive are cut down to fit the new template, and the detail that allowed them to rank is designed out in the name of a cleaner layout. Specifications, use cases, technical detail, and answers to the questions buyers actually ask, all reduced to a hero image and a short paragraph in the interests of UX and CRO, or shipping the site on deadline.

That detail is what makes a page answer at passage level, which is what gets it retrieved and cited. Strip it and you lose the ranking and the citation in one fell swoop.

Orphaned pages to facilitate a cleaner menu

The same audit found more than 12,700 orphaned URLs against roughly 3,600 pages reachable by following internal links from the homepage. This was due to a range of reasons from internal links served using client side JavaScript but mainly a new, UX friendly navigation, again built entirely with UX in mind and no consideration for organic performance.

This is almost always cosmetic. The mega menu was too busy, so the categories were trimmed. The related products module was dropped because it looked cluttered. The blog index was capped at recent posts. Each decision is defensible on its own, but together they cut thousands of pages out of the internal link graph.

Internal links are the primary way search engines discover pages, and internal PageRank is absolutely still a think. A page with no internal links pointing at it gets crawled less, receives none of that internal equity, and often ends up sat in Crawled, currently not indexed in Google Search Console. Being in the XML sitemap is generally not enough. A sitemap says a URL exists, an internal link says it matters.

Another example of technical SEO foundations still being vital to organic search visibility and performance across LLMs.

In fact, for AI search it is worse. If a page is not linked and not rendered, there is no realistic path to it being retrieved and cited at all.

Why this keeps happening

This isn’t bad development. These websites look the part and may well improve conversion rate, dwell times and all of the KPIs that matter to those agencies. But what good is that is your traffic has halved (or worse). And if there has been SEO support involved with these projects, then they are unfortunately just not doing their jobs.

A dev agency is measured on shipping to a date, on performance scores, and on how the site looks in a stakeholder review. Nobody is measured on whether the redirect map resolves cleanly, or whether the raw HTML contains the content, or whether every commercial URL still has an internal link pointing at it. Organic performance shows up months later, by which point the build team has moved on and the brand is left explaining a decline it did not cause.

The issue is, that as credible SEOs, we push back on this stuff we become the expensive, opinionated pain in the proverbial. But this is nothing that having good counsel at the start of any migration project doesn’t solve – particularly given the changes that are happening to search. As we always say to our clients, you shouldn’t be approaching a migration looking to maintain performance. Done properly with with SEO, front end dev, back end dev and the brand working together in unison, it should be an opportunity to catapult performance across traditional and AI search.

But we’ve all been there, and just this year, we lost a client for doing our jobs. For holding the line with a dev agency that wanted to ship a clean and tidy site, but one that was going to fall off a cliff in terms of organic visibility. I have gained considerably more work from cleaning up the aftermath elsewhere, but these examples are sad. Mainly for the brands who opt for the bright lights and the cheaper option.

The AI layer makes all of this more expensive

A few years ago these mistakes cost rankings. Now they cost so much more, across a range of surfaces and experiences.

Bot access is also worth watching alongside this. From 15 September 2026, Cloudflare is changing its defaults so that training and agent crawlers are blocked on ad-supported pages for new sites and existing free customers, with search crawlers left alone. There is a trap in there too, because mixed-use crawlers get judged on their strictest applicable rule, so a broad training block can catch a bot you rely on. If you are on another CDN, the same logic applies. Verify what your edge and WAF actually do with each user agent rather than assuming your robots.txt rules are enough.

Retrievability is not just whether the content is in the HTML. It is whether the crawler can reach the URL, is allowed to fetch it, and finds the answer in the source when it does. Break any one of those and the rest of your strategy could be flawed.

What you should insist on before your next migration

Most of this is prevented by a handful of gates that cost days, not months. This is not an exhaustive list my any means, but a good start:

• Parity crawl before and after. Crawl the current site, crawl staging, and do your checks. Every URL that is being dropped needs an SEO friendly solution and implementation.
• One redirect map, one owner, one source. Single hop, server-side, permanent status codes, every destination returning a 200. Crawl the map itself rather than trusting the spreadsheet.
• Server-side rendering for anything you want found. Check the raw HTML, not the rendered DOM. If the copy, headings, and links are not in the rendered HTML, they are not retrievable for anything other than traditional Google Search (and even then, this will be heavily delayed)
• Internal links treated as a deliverable. Every commercial URL needs several internal links from a relevant locations. Tidy navigations are great, but don’t cripple performance due to an over obession with this.
• Content decisions made on data. No URL is removed without someone looking at what it earns in traffic, rankings, links, and revenue.
• A named owner for organic after launch. Someone whose job it is to watch Search Console, log files, and rankings for the first ninety days, with the authority to get things fixed. Very rarely will a migration be perfect, even with the best possible support.

None of that should slow a build down meaningfully of the phasing is correct and recommendations are made efficiently. It just moves the argument to a point where the cost of being right is a week of dev time, rather than a year of lost revenue.

A moral dilemma for credible SEOs

We enjoy this work. Recovering a site that has been mishandled is satisfying, and the wins are usually big and fast because the problems are structural, and bread an butter for any SEO who knows their stuff.

The bitter pill to swallow for the brands involved, is how avoidable it all is. They end up paying for two builds, the one they commissioned and the one that fixes it, and losing serious revenue in between. Every one of the four failures (and there are of course many more) above is caught by a competent pre migration audit and roadmap.

If you are planning a migration or a redesign, ensure you have someone in the room who is accountable for organic performance before the templates are signed off. If you have already launched and the traffic has gone, the diagnosis is usually quicker than people expect, but the action plan is paramount

Whitworth runs technical and AI search audits under the AIR SEO framework, If your traffic dropped after a launch and nobody can tell you why, get in touch.

Sources

• Google Search Central, Site moves with URL changes
• Vercel, The rise of the AI crawler
• TechCrunch, Cloudflare’s new policy pushes AI companies to pay for publishers’ content
• SearchEngineWorld, Do AI Assistants Actually Render Your JavaScript when Grounding? We Put It to the Test