We are all taught early in our technical SEO training, that recapturing a website’s lost authority using 301 redirects is one of the bedrocks of good site maintenance. Even in today’s mobile-first, RankBrain SEO era, this is still very much the case but a mixture of factors has led to some very messy and chaotic practices when it comes to this age-old tech SEO principle.
In fact, one of the most frequent mistakes I see these days from SEO practitioners of all experience levels is the overuse of 301 redirects in the hope that this will propel a website’s organic visibility. Just one of several lazy SEO techniques that are sadly only too common, whether this is due to bad training or just a lack of knowledge.
What we have been left with is the implementation of 301 redirects with absolute abandon. Whilst this may not cripple your site and could guarantee that you catch any leaked authority, there are a number of reasons why this is simply not good practice and could lead to serious website health issues further down the line.
Depending on the scale of your website and the manner in which you are implementing your redirects, the ineffective use of your domain’s crawl budget that the aforementioned tactics can lead to could well be adversely impacting your organic performance.
From site speed issues to the dilution of authority and the aforementioned crawl budget considerations, redirects can be as damaging to your goals as they can be rewarding.
So, before you start compiling vast redirect maps for all your clients or the websites you manage, read on to find out how I would recommend you approach this classic but vital aspect of technical search engine optimisation.
Crawl Error Sources: Establish Where They’re Coming From
First thing’s first and before you even start to think about what you should do with your crawl errors, you should establish just where they are coming from. There is a range of sources you can refer to for this data but remember that not all errors require you to start muddying your htaccess or web config file with unnecessary 301 redirects.
Google Search Console
Although this source provides limited data and only relating to the one search engine, Google Search Console is extremely handy as it will often tell you where your crawl errors emanate from or were linked from. This is not always the case though, so you will need to do your own investigative work should you be given limited information.
If you can then devise a way to stop these errors from occurring in the first place rather than adding these unnecessary redirects, then you’re onto a winner.
This may be fixing some erroneous blog tags or pagination, removing or fixing a hyperlink from your footer or menu or maybe even redirecting an old category page you had forgotten you’d removed from your site.
Each instance will have a different solution and in some cases, there will be no action required at all. Thankfully, Google Search Console now gives you more information than ever about these crawl errors and sixteen months of data will also show you how your crawl errors have been developing (and hopefully improving) over time.
Server Logs
Server log (or web log) analysis seems to be a bit of a buzz term in the world of technical SEO at the moment but it has actually been around since the advent of the web. Before we got nice shiny analytics platforms and the aforementioned Google Search Console suite, server log files were the only way you could get accurate information about the way your site is being crawled. As such, they are still very powerful when analysed correctly and will give you all sorts of actionable data you can utilise for SEO success, particularly with regard to your crawl errors.
Your web server is actually the only place to get a definitive list of every crawl error encountered by every single user agent that arrives at your website. However, whilst utterly comprehensive, the chances are that you will need to sift through a lot of crawl errors and many will make little sense and require no action.
For example, you may well find historic CSS files, scripts and parameter URLs that should certainly not be added to your list of redirects. But you could also find old pages you had no idea ever existed, spider traps or redirect chains and loops you could correct in order to improve crawlability, so ignore this goldmine of SEO data at your peril.
You can usually obtain these server log files directly from your cPanel or equivalent, or alternatively just ask your IT team.
Web Crawl
Whether you’re a staunch Screaming Frogger, loyal Deep Crawler or use SEMRush’s ever-improving web crawler, it doesn’t really matter. All of these tools are more than likely to give you some great insight into crawl errors encountered on your website.
All of these crawlers allow you to emulate a wide range of user agents and execute JavaScript in order to crawl your domain in much the same way that the search engines do, this can often bring back a range of status codes and responses.
Although the crawl errors you get from this method are not as comprehensive as your server logs or Google Search Console, they should be high on your priority list for resolution as they will be encountered by the search engines every time they spider your site.
Redirection should be your last resort for these crawl errors as you are likely to be able to fix or remove them. Adding 301’s will add unnecessary redirects to your site which is not SEO friendly and make life more difficult than it should be for crawlers. If these errors are present at sitewide locations of your site such as your footer or navigation, then you could have a significant issue on your hands. Thankfully, a few quick fixes in these scenarios should see the situation improve quite dramatically.
When Should You Use 301 Redirects?
It would be easy after reading this article to presume that I have some sort of vendetta against redirects for SEO purposes. This couldn’t actually be further from the truth and as discussed, they are one of the most powerful tools in your SEO armoury.
As long as they are not being used recklessly in the hope that they will end up being the silver bullet for your search campaign, then you should be looking to implement 301’s periodically throughout any SEO campaign.
The following are the scenarios in which I would recommend using redirects pragmatically.
Domain Name Change
The big one, of course, is a migration to a new domain. If you have made the bold but often necessary decision to move your brand to a new home, then you will need to redirect all of your URLs to their new destination. This can either be a very simple or a very arduous task depending on how you go about it but is absolutely crucial.
If you are just changing the domain name and the rest of your site is remaining the same structurally, then this shouldn’t take you long and is just a simple find and replace job in Excel.
However, if your domain name change is coinciding with a significant change in site architecture, then you will need to take time to ensure that every authoritative page on your old site is 301 redirected to its counterpart or its nearest equivalent at your new web address.
Get this right and you could see your site’s organic performance increase. Get it wrong and you could plummet in the rankings and suffer a fate that many big brands have experienced following a shoddy migration.
Site Structure Changes
After a change of web address, the scenario in which redirects are probably most effective is when there has been a significant change of website structure. Ideally, these would be mapped out prior to your migration and go live as soon as your new site does. But only too often this isn’t the case.
This is where your server logs can again be a priceless tool and Google Search Console will also tell you key pages that GoogleBot is still attempting to crawl that no longer exist. If these were a prominent part of an old site structure they are almost certain to have legacy and who knows, maybe even external backlinks.
These are prime candidates for 301 redirection, and you should implement these as quickly as possible. You will often see these implemented by developers (or perhaps SEOs) as 302 redirects, which are temporary and not permanent. Whilst there is a chance this may syphon some authority back to your site, the content on your new destination page is unlikely to ever be honoured so make sure you get this right or your indexation (sorry Barry) could suffer.
For more information on which redirects to use in which situation, read this superb article from the team at ContentKing which also has some expert comment from some of the very best tech SEOs in the UK.
Seasonal Pages
If you have pages that have expired because they no longer exist or are seasonal, then you should probably redirect these. For instance, a travel site selling excursions for a particular period will inevitably see pages or products expire unless they have done something quite clever with their URL structure.
If this is the case, then be sure to redirect to your new URLs as these may well see their visibility increase a lot faster. Revise your internal linking accordingly and this should work well for both web crawlers and users.
You may not want to do this for every single product or URL, but you should consider this for any landing pages that were receiving high levels of traffic, enjoying good engagement or attracting backlinks.
HTTPS
If for any reason you have ignored Google’s advice to serve a fully secure website, with a validated SSL certificate and using the HTTPS protocol, then you should, of course, address this before you even start to think about redirect management.
If you have done so, but haven’t redirected all your HTTP URLs to their HTTPS counterpart then I imagine you are sat there looking at some pretty grim SEO visibility or organic traffic reports. Thankfully, implementing this as quickly as possible should see your site recover fairly swiftly.
HTTP to HTTPS redirects are very simple and need no mapping as you are just redirecting each URL to their new home. The following script should suffice if you are using Apache or Nginx using .htacess or .conf.
RewriteEngine On
RewriteCond %{SERVER_PORT} 80
RewriteRule ^(.*)$ https://www.yourdomain.com/$1 [R,L]
Which Crawl Errors Can You Ignore?
Now we know when we should be considering redirects, which URLs can we definitely ignore?
These come in all shapes and sizes but essentially you should sit back and think about whether or not the pages in question are likely to have been regularly crawled, authoritative, canonical URLs.
If not, then the chances are that you should leave this URL to disappear into the ether. If it keeps coming back and popping up in your audits, then the chances are that this is still present in your website’s structure, listed in your XML sitemap or perhaps has some external referring domains. If GoogleBot is still trying to crawl it, then it exists somewhere!
For these cases, you should take steps to reintroduce the URL to your website or consider the correct redirect.
The solution here isn’t rocket science, just take each situation on its own merit and make the call that makes the most sense.
Here are some URLs that in my experience, you won’t need to redirect. You should check for the aforementioned authority signals but by and large, these shouldn’t be clogging up your redirect files and making life more difficult for web crawlers:
- Parameter URLs (Session IDs etc)
- Old Paginated Series
- /feed URLs
- CSS & JS
- Old Tag and/or Author Page
If you have been scratching your head during your technical SEO audits wondering just what you should be doing with your crawl errors and which you should redirect then hopefully this SEO guide has helped.
Should this article have posed more questions than it has answered, then just tweet me and I would be happy to help. Alternatively, just drop me a line.