Andy's Notebook

Web Performance - A Whistlestop Tour

The slides from my recent talk at DigiTalks Cheltenham:

Web Performance - A Whistlestop Tour

View more presentations from Andy Davies

Increasing the TCP Initial Congestion Window on Windows 2008 Server R2

In November 2010, Ben Strong highlighted how Google and Microsoft were cheating on TCP Slow-Start by setting the initial congestion window higher than the RFC value.

Since then there's been quite a few articles discussing the performance benefits of increasing the TCP initial congestion window (initcwnd). Most of the articles showed how to make the changes on Linux but there was no coverage on whether and how it could be increased on a Windows Server.

In one of their recent posts, Tuning initcwnd for optimum performance, the guys at CDN planet mentioned they didn't know how to change the settings on Windows either.

Having trawled through the various options in netsh, I was coming to the conclusion it wasn't possible but Drit Suljoti of Catchpoint pointed me in the direction of MS Hotfix KB2472264 "You cannot customize some TCP configurations by using the netsh command in Windows Server 2008 R2".

After installing this hotfix I was able to change the initial congestion window to 10 (it can be adjusted up to 16 * MSS).

Installing and configuring

1. Request the Hotfix via http://support.microsoft.com/kb/2472264

2. Download and install the relevant version - it's available in x32, x64 and ia64 flavours

3. Check the hotfix is installed ok
   
   Type:

   c:\netsh interface tcp show supplemental
   

   and you should see:

   The TCP global default template is internet
   
   TCP Supplemental Parameters
   ----------------------------------------------
   Minimum RTO (msec)                  : 300
   Initial Congestion Window (MSS)     : 2
   Delayed Ack Timeout (msec)          : 200
   

   If you get the following then the hotfix hasn't been installed:

   The following command was not found: interface tcp show supplemental.
   

4. Update the initial congestion window settings and switch to the custom template
   
   First set the initial congestion window in the custom template:

   c:\netsh interface tcp set supplemental template=custom icw=10
   

   Then tell Windows to use the custom template instead of the standard internet one:

   c:\netsh interface tcp set supplemental template=custom
   

5. Your server should now be running with the new initcwnd value, so pull out your favourite packet sniffer - Wireshark, tcpdump etc. and test.

Performance

So far I've only tested this configuration on an EC2 instance and haven't deployed it in live production environment yet.

Cursory performance tests show a great improvement downloading the default welcome.png that comes with IIS 7.5 - from 11.7s to 2.5s with a server in Singapore and a client in the UK.

I'm not sure I quite believe the 11.7s time so over the next week I'll try some more tests with various size files and post the results.

Before (icw=2):

$ time curl ec2-175-41-175-250.ap-southeast-1.compute.amazonaws.com/welcome.png > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  180k  100  180k    0     0  15912      0  0:00:11  0:00:11 --:--:-- 20785

real	0m11.706s
user	0m0.008s
sys	0m0.015s

After (icw=10):

$ time curl ec2-175-41-175-250.ap-southeast-1.compute.amazonaws.com/welcome.png > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  180k  100  180k    0     0  75765      0  0:00:02  0:00:02 --:--:-- 93690

real	0m2.532s
user	0m0.007s
sys	0m0.015s

Final Notes

If you're thinking of deploying the fix to a production server it's probably worth reading the proposal submitted to the IETF on increasing the default value to 10 as it covers both the advantages and potential drawbacks with an increased default.

Jim Gettys also points out some of the potential issues particularly in relation to the possible effects on network equipment.

Testing how slow third party components affect your page load times

Both Steve Souders and Joshua Bixby have highlighted the effect slow third party components can have on  overall page load time.

Third party components have become common e.g. ads, links for bookmarking, sharing, widgets that pull in content from other sites etc, and the impact on our page load times depends on the quality of service they offer (both how the component is implemented and the reliability/speed of their service)

Steve and Joshua have highlighted approaches that can be used to decouple the components and reduce their impact on the performance of our pages but it's early days and only some providers have implemented them (Sergey Chernyshev has a some thoughts too).

The challenge for web developers is understanding what the effect on page load times could be, when most of the time the providers' networks remain reliable.

 

Charles - a bandwidth limiting proxy

To test the impact providers can have if they perform poorly we need to make the network connections to them unreliable and my tool of choice for this sort of situation is Charles (http://www.charlesproxy.com/)

Charles allows both low bandwidth and high latency to be simulated, either on an all sites or individual site basis.

To enable throttling choose Proxy -> Throttle Settings and check the first box, the same dialog also enables configuration of bandwidth and latency constraints, and the URLs they apply to

On the sites I tested, I throttled using high latency for each provider individually - as an example here's one for addthis.com with the latency set to ten seconds.

I also used the normal waterfall tools (Firebug and WebKit inspector) to see when onload was being fired at, so I could experiment with asynchronous loading to mitigate the effect of the latency.

 

Look out for subtle failures!

One webapp I tested used the widgets from addthis.com and even with long latencies set, the pages appeared to load quickly, but further testing found the tour was broken - clicking the button just replaced the current page with an image

The tour used lightbox to display a series of images, the delay in downloading the javascript from addthis prevented the onload event from being fired and so the lightbox code wasn't being inserted into the page.

Charles works on Windows, Mac and Linux, and has a free trial, so give it a try. Find the components that cause problems for you and raise the issue with their providers.

 

The Need For Speed... why slow page load times matter

About a week ago, I did a presentation at BarCamp Stroud on why page load times matter and things that can be done to improve them.

It was fun to present and share what I'd learnt on a topic I'd spent a lot of time investigating and implementing for a WebApp I run (got quite a few things still left to do)

Here's the presentation for anyone who's interested, and there will probably more as I've already made some notes on other things to cover. 

The Need For Speed

View more presentations from Andy Davies.

.net and ISA Server - reducing bandwidth consumption

Earlier this year I came across some problems with some .net resources not being cached by ISA Server (.net and ISA Server - a match made in hell?)

The problem in a nutshell is that by default ISA Server doesn't cache URLs that have query parameters, and the method .net uses to serve static resources that are bundled in .dlls relies on query parameters.

When .net uses the Microsoft AJAX framework, script elements like the one below are inserted into the page 

<script src="ScriptResource.axd?d=ki7GLYn2P5z-CEtE0SsvKYJhnTEkD13edwjg24cmxWe4aD03WzKyGXQD45nYCFy70&t=633178683580000000" type="text/javascript"></script>

After looking at some of the lastest developments in .net 4.0 I discovered that ScriptManager could be used to serve out static versions of the AJAX framework (even in versions of .net before 4.0).

After a few attempts I came up with this code that takes the ScriptResource.axd references in the page and replaces them with static scripts instead, which are of course generally cachable by ISA Server

<asp:ScriptManager ID="Script" runat="server" EnablePartialRendering="true">
    <Scripts>
        <asp:ScriptReference Name="MicrosoftAjax.js" Assembly="System.Web.Extensions" Path="~/js/ms/MicrosoftAjax.js" />
        <asp:ScriptReference Name="MicrosoftAjaxWebForms.js" Assembly="System.Web.Extensions" Path="~/js/ms/MicrosoftAjaxWebForms.js" />
    <scripts>
<asp:scriptmanager>

(the release i.e. minified, versions of the scripts within the MS AJAX framework have been placed in /js/ms)

By default the scripts won't be served gzipped, or with the long expiry times that ScriptResource normally serves them with but this can be addresses by configuring IIS (which is how it should be in my view)

We're still measuring how much this improves our page load times and how much it reduces our bandwidth consumption but it looks to make a significant difference.

If you're interested Dave Reed has written more on what's possible with ScriptManager in .net 4.0 -  ASP.NET 4.0 ScriptManager Improvements