Musings of a technophile

So my boss where I work noticed that Google was showing 404 error’s in our Google webmaster tools, and that many of those 404′s are on invalid URL’s. With a site as big as ours, I wouldn’t say it would be uncommon to have other sites linking to our content and they mess up the URL in the link, but in any case since we are big on SEO and making sure every user has a good experience on the site, we often work to try and make these 404′s into working URL’s. That being said, the problem URL’s this time around were ones that had characters on the end of the URL. So for instance while our main page is “index.aspx”, we would have 404′s on “index.aspx;” or “index.aspx,”. Obviously the problem is the funky characters on the end that shouldn’t be there, and we needed to remove them. I instantly though URL rewrites would be a good way to fix this, so I wrote up the following script

1
2
3
4
5
6
7
8
9
10
11
12
13
14

<script runat="server">
Private Sub Application_OnBeginRequest(ByVal sender As Object, ByVal e As EventArgs)
        Dim application As HttpApplication = CType(sender, HttpApplication)
        Dim strRegex As String = "\.aspx[;,()!^]+$"
        Dim RegexObj As System.Text.RegularExpressions.Regex = New Regex(strRegex)
        Dim url As String = application.Request.Url.AbsolutePath
        If RegexObj.IsMatch(url) Then
            url = Regex.Replace(url, strRegex, ".aspx" )
            HttpContext.Current.Response.Status = "301 Moved Permanently"
            HttpContext.Current.Response.AddHeader("Location", url)    
        End If        
        HttpContext.Current.RewritePath(url)        
    End Sub
</script>

Now you would just put all of this into your Global.asax file. The Global.asax (for those that don’t know) executes on every request, and therefore every url that is submitted to the site will execute through this function.

Basically this function will take the url passed in, and if it contains one or more of the following characters

;,()!^

after a “.aspx” it will remove them, and then bring the user to the fixed URL. Note that there may be more special characters you can add other than this, but these are all I tested with. Also note that characters such as a period and a colon will not work as they are special URL characters already, and apparently the server will try to render the URL even before calling the OnBeginRequest function. Also note that this does not fix query strings and/or strange characters after a query string.

The function will also throw a 301 permanent redirect with a location of the fixed URL – this is for the search engines so that they know the page is not valid and where to go instead (basically for SEO).

In any case, this may not be the perfect solution (then again, I don’t know what else could be implemented…perhaps an ISAPI filter?), but it does cover most instances or bad URL’s linking to your site.