Opine and Whine

Speeding up Web pages with Expires Headers

2010-11-29T14:49:00.000-07:00

We have an application where I work that is used extensively for business reporting. The audience for these reports include all of the top management of the company, including the CEO. The problem we face is that it’s performance is slow for some of the most important users and they are very vocal about that problem. A few of us have been working on how to speed up the user experience but where do we look first? In this article I will tackle one of the simple solutions we found and how we proved to ourselves that it really did make things faster.

This particular application is all web with no client side installation necessary. The fact that it is a reporting tool is not nearly as important as the fact that it is a full web application with all the benefits and the downsides to go with it. Another important fact is that the users are most often over 2000 miles away in another office and their experience is much worse than the users where the app is hosted. Those remote users however are very important customers and include those who report to the CEO of the company. Suffice it to say that when they complain it is heard loud and clear.

So where does one begin in the effort to increase the speed of a web app? In our situation the network stood out as a primary candidate since the speed was better in the main office than in the remote office. Our network folks did their own analysis and decided to recommend and upgrade from 45 megabits to 1000 megabits (1 gigabit) and we all thought this would be the answer to the problem. This represented a speed increase of over 22 times the speed of our old connection and surely would fix the problem, but as it turned out it did not. The measurements we had been taking every day showed that the speed remained slower for the remote users by a factor of 4-5 so if a user in the main office saw response times of 4 seconds the remote users would see 16-20 seconds for the same page. So we had to start looking elsewhere.

During this same time frame some other colleagues turned me on to the recommendations written up by Yahoo that they called “Best Practices for Speeding Up Your Web Site”. While digging into that information it was revealed that some surprisingly large number of response time issues in web apps were due to client side issues not deep in the tiers of the application server and database. They go on to list out the changes one can make to improve speed and even provide some ways of measuring how bad the problems might be. Armed with this new (to me) information off I went to analyze and make recommendations on how to improve performance.

The first tool I tried was the Yslow addon that works with Firebug and Firefox. This tool gives grades for pages based on the Yahoo recommendations, just like those grades we got in school, A is the best and F is failing. In our case we got a lot of D’s and F’s because of lack of knowledge when setting up the servers.. Some of the recommendations were not possible for us to use because it’s not a public application and we purchased it from a well known vendor so any changes to the internals of the app were beyond our capability. However based on some the failing grades we could make some simple setting changes on the web server and possibly get a speed increase. The problem is how do we know when things have gotten faster and how much faster?

The first rule of experimentation is to limit yourself to only 1 change at time in order to observe the affect of that change. It is tempting to go and make a whole bunch of changes at once but doing this make it next to impossible to tell which of the changes contributed positively to the speed and which ones negatively. This lead to a simple approach of changing just one setting on the web server and then measuring the difference. We chose to start with one of our grades of F on the area of adding expires headers to the content. Yahoo in this case recommends what they call “far future” expires for static content. In other words the content that does not change very often like pictures, icons, css, javascript, etc, should get an expiration date that is a long way out like 10 years. Just to see how this might affect things we made the change on the development environment and run some tests. The procedure seemed simple enough:

Set the setting the web server to add expires header to the static content
Clear the browser cache completely to simulate a new user on the site
Using the to measure the speed and give the grades, load the page the first time and observe the speed
Navigate to a different page
Navigate back to the page in question and observe the speed again.
The difference in speed should be the improved performance.

The problems arose quickly when I found out that most of the users on this particular web app were using Internet Explorer most of the time and most did not know Firefox even existed. I then had to switch my experimentation to another set of tools that work with IE. Since I am focused on free and/or cheap tools I ruled out some that I could not justify the cost. One that stood out was a tool called dynaTrace AJAX Edition. HttpWatch also gets an honorable mention but the full version costs money and the free version is a little to limited for most uses. dynaTrace AJAX Edition also provided grades and recommendations but adds much more detail beyond Yslow and HttpWatch.

I decided to modify my experiment because we have two environments to test on, QA and DEV, so we could use expires header on one environment and leave them off on the other. This would give us side by side comparisons without having make changes over and over again. The second change I made was to define a set of test steps to follow so that we can repeat the same click-stream on each environment, again keeping everything as constant as possible. Another change to my procedure was to make sure to perform the steps at least once on each environment to make sure the application server was primed and ready since the first time using the app after a restart is very slow as the objects get loaded into memory on the server, then all the subsequent requests are faster (a subject of another article). The new procedure looks more like this:

Set the DEV environment to use Expires headers with an expiration of 90 days.
Set the QA environment to the same as PROD using no Expires headers
Restart the servers if necessary
Run through the click-stream script once to wake up the servers manually or with your automated script in Selenium or other automation tool (a subject of another article)
Clear the cache on the browser.
Run through the click-stream script and let dynaTrace AJAX Edition takes it’s measurements.
Repeat the click-stream script again and take a second set of measurements. Optionally close the browser and re-open just to make sure it’s using a permanent cache, not just in memory.

This modified procedure should produce a long response time right after the cache is cleared because the browser needs to download all of the components. The second run through of the click-stream script should produce noticeably different speeds than the first because the cache is primed with non-expiring content and the browser only needs to load those components that do not have Expires headers. The difference between the two speeds will show how much of an improvement Expires headers made. Here is a great article on how the browser cache works “Best Practices on Browser Cacheing”

The results of one of my experiments showed a marked improvement in page times. The tool breaks out a few measurements First Impression time, Onload Time, and Fully loaded time and in each of those I saw in improvement:

	Expires Disabled	Expires Enabled	Delta
Rank	F(0)	A(100)	upgraded from F to A
First Impression time (ms)	2826	839	-1.987 seconds or 337% faster
Onload Time (ms)	3535	1715	-1.820 seconds or 206% faster
Fully Loaded time	3746	1930	-1.816 seconds or 194% faster
Remarks	69 requests, 63 uncached	70 Requests	63 requests cached/

Comparing the results from the two configurations we can see that the user experience is greatly improved by having a primed cache. In practice a user would have slow response times with an empty or expired cache then fast after their cache was loaded with the static content. Using expiration time frames as long as 10 years may not work well with every app which is why we decided to use 90 days in our experiment.

dynaTrace results without Expires headers (sreenshot)

dynaTrace result with Expires headers (screenshot)

Conclusion:
Setting content to use Expires headers can measurably increase the performance of web pages. This increase is limited to return visitors to a page or web site and does not help new visitors as much. In this particular case of a web site that is internal to a company and has nothing but return visitors utilizing cache control headers show a marked increase. There are many other steps that can be taken to improve web site performance but this one represents an easy setting change because it does not require any deep dives into the application code and logic. To know for sure if this setting helps or hinders performance take measurement that reveal the actual response times and compare them with and without the setting enabled.

Why Automated Testing is Broken

2008-12-02T21:46:00.005-07:00

Having been doing automated software testing for almost 10 years now, I am stuck somewhere between intermediate and guru on the subject. As such, I take the liberty to voice what I think based on my experience which has been wide and varied due to my consulting career.

There exists a chasm between the sales pitch for most automated testing tools and the reality they deliver. The chasm is wide mostly because the pressure to sell software is big and being brutally honest in the sales cycle is not conducive to closing deals. The vagaries of software sales is a subject better left to those more qualified than I, but suffice it to say that users are not prepared for what awaits them as the automation effort begins.

It all begins with the training class. The attendees all made it to the QA department from other various pursuits in the business world, from customer service to accounting. Some come with technical skills that include a possible class in programming and most have no scripting/coding experience at all. They feel quite comfortable for the first two days of training because it's all point and click, record and replay, and relatively easy subjects to digest.

Then something terrible happens - an eager student strays away from the "canned" training application and into the murky world of their "real" application that they work with every day. They quickly discover that the script might record but replay is sketchy if it works at all. The inevitable question arises for all the room to hear and from there the situation gets tense. Does the instructor dodge the issue hoping that it fades into oblivion or do they take aim and fire that terribly honest answer "Recording is only the starting point...". If the honest route is taken the discussion gets heated, questions arise as to why this tool was purchased and in some cases that will be the last time the users ever use the automation tool.

This situation is all to often what I have experienced. The QA team expected the automation tool be to be easy, plug and play, no programming necessary. The reality of this situation is that programming is exactly what is required to make the automated test work for more than a few iterations on the first day it was created. This is true for one simple fact, the applications we are paid to test are complex. Complexity in design, in execution, and most importantly the problems they were created so solve in the first place.

If we simply summed a few numbers up each day then a $5 desk calculator would be all the technology necessary to perform the task. Most modern companies today have very complex issues to deal with and therefore the need for software to perform those tasks in a reliable and speedy manner. This complexity is clear in how much time it takes to bring a new team member up to speed on how the system works and how they required to test it. It follows that the instructions we must give to the automated script are summarily complex and require good data in order for them to achieve their task.

In order to succeed with automated testing the personnel that create the scripts must have the same skill set as the developers of the software. Their tasks are nearly identical in every way to the tasks of the programmers on any software development team. This means that automated tests and the process of creating them must be treated and managed as yet another software development life cycle (SDLC). Choose your method of development, waterfall, XP, RAD, Agile (which is preferred here) and formalize it. The automation team should participate in design, code reviews, unit testing, and ultimately a release process.

In the last decade I have seen many automation efforts start up quickly only to be abandoned almost as quickly as they were started. The one common thread with these "failures" are that they have unrealistic expectations of what automated testing will deliver and how quickly. In one case I was on a team that was expected to automate the testing of an enterprise system in about 6 months, however the development of that same system took over 5 years to accomplish due to it's vast complexity. How then could the development of a true regression suite happen in 1/10th the time? The answer is simple, it could not and did not. What could have been done is to automate some of the most tedious and repetitive tasks that would free up the manual testers to do what they do best, find bugs. Over time a momentum can be achieved as the foundation of the test effort becomes more and more automated and hands off.

I will explore some other issues of why automated testing is broken and how to fix it in some future posts. Suffice it to say that the proverbial root of the problem is not in the tools but in the user of the tools. Hiring at least a few skilled programmer types to jump start the automation effort is mandatory for any lasting success. Failing to do so just creates tests that are fresh on day one but begin to rot over time and eventually become useless, sometimes in just a few months.

No Fluff Just Stuff conference was inspiring

2008-11-18T20:26:00.002-07:00

The Rocky Mountain Software Symposium, or more aptly name "No Fluff Just Stuff" took place last weekend. This was my first time to attend and I was very impressed with the quality of the speakers and material covered.

The majority of the topics centered around Java development and what is happening in that world. As a QA professional that does some development I was in a little over my head, but that served only to make me even more inspired by what I saw and heard. Each one of the speakers was very knowledgeable, prepared, and worth listening to. I only wish I could have attended more of the sessions that I missed.

I decided to list here some the juicy nuggets of knowledge that I left the conference with.

First on the list was the talk on Test Driven Design (TDD) by Neal Ford (http://memeagora.blogspot.com/). Neal's presentation was very well done and opened my eyes to the way I should have been doing automated test development all along. (Automation in QA is really development in it's own right, a subject I will write more on later). The way in which the tests are written first, only to fail, then the code is written to answer the problem followed by the test passing is brilliant. I wish I had thought of it before now or at least read into it some time ago. Further into the presentation Neal showed how his code became more and more simple to read and execute while at the same time methods became more focused and specialized. This process made each method self documenting by the fact that they each were named well and did only one thing. I must read more about this as I integrated it into our process for automated test development. Neal himself is clearly knowledgeable and an excellent presenter.

Next most useful talk was by Ken Sipe (http://kensipe.blogspot.com) on Java Memory, Performance and the Garbage Collector. I have been load testing systems using the JVM on the server for many years and the information Ken showed would have been incredibly valuable in that process. In reality the application experts and java pros should have known their stuff about JVM tuning, but they did not and neither did I. Much guessing and voodoo was done to improve performance instead of accurate measurements like Ken showed us. I will be implementing what Ken shared with us and I expect to save a lot of time by either tuning the JVM or ruling it out as a performance bottleneck based on real data.

I attended another one of Neal Ford's talks, this time on code metrics. Again his presentation was very well organized, informative, and inspiring. For years we have been measuring our development work badly but with good intentions. Neal presented some great tools and ideas with an emphasis on not depending too much on one measurement. For instance cyclomatic complexity is tempting to use across the board but he showed that I does not tell the whole story. I especially liked what he said about "code smell" referring to overly complex routines, copy and pasting, and the lack of TDD.

Ken Sipe's talk on Career 2.0 had some very good food for thought. Some of the things he said that stood out were about increasing your digital footprint with blogging, writing, and so on. That footprint works in your favor to advance your career and expose you to others who may be interested in what you have to share. The other concept he talked about that stood out to me, was his statement about how the company managers a responsible for the company but you are responsible for your career. In many cases advancing your career also helps the company but even if it does not it's still up to you to make sure you are becoming more valuable in your career.

Last but not least, the introduction to groovy by Jeff Brown (http://javajeff.blogspot.com) was so inspiring I could hardly sleep. I was thinking about how I could speed up technical testing in the QA world by writing scripts very quickly. Aside from that immediate business need, Groovy promises to be very useful in so many ways that I am still thinking of them many days later. Jeff did a great job doing demos of how to use the tool and I felt like I could download it and get started right away. Once I get a break I will be diving into Groovy.

I highly recommend the NFJS meetings for anyone working with Java in their projects. I plan to attend the next time it comes up again.

2nd Solo

2005-05-28T16:00:00.000-06:00

This day was picture perfect for an early summer flight. The wind was calm, the sky was clear and I had no other obligations to attend to. I made my way to the airplane parked on the flight line and began my preflight inspection. I had already done this once today but in keeping with consistency I did a good inspection anyway. With everything ready on the outside of the plane I climbed in closed the door and began my checklist for starting up the plane. Then it occurred to me that today I am flying by myself and I will be doing maneuvers that I have practiced so many times, but now all one my own. The nervousness starts to build as I check and then recheck the all of the pre-start procedures. Trim set for take off, fuel on both tanks, throttle open slightly, mixture set to rich, carb heat cold, all breakers normal, beacon light on, master switch on, avionics master off, primer set and locked, feet on the brakes, I yell "clear prop" and hit the starter. As the motor comes to life I again think about all the things I am now doing alone and how safe I felt with my instructor in the right seat, until now.

I begin to taxi being extra careful to watch the distance between me and the parked planes and I moving around. Clear to the left, and clear to the right I make my way onto the taxi-way along the runway. I am now listening to the radio traffic to get a feel for who else my be in the pattern and possibly landing when I am hoping to take off. The tower is not in service yet so every pilot must announce where they are and what their intentions are just as I am doing. This system works very well since no one wants and accident to happen. I arrive at the last point on the taxi-way where I begin the pre take off checklist. I have done this many times at this point but this one seems to be the most memorable, now that I am pilot in command and there is no back up. Checklist completed I proceed towards the runway, turn the plane to the right for one last visual check of the traffic pattern to make sure no one is on base or final without their position announcing on the radio. All clear, I roll onto the runway and push the throttle all the way in. As I gain speed I run the next checklist, tach producing power, pressure and temperature in the green, airspeed increasing, and in what seems a few seconds I reach 62 knots and the plane begin to lift off on it's own. No turning back, I am now in the air for my second solo flight.

I follow the routine of climbing and turning in the pattern until I reach the designated altitude of 6,500 feet MSL. Once I make it to midfield I start heading to the south east to the practice area about 15 miles away. I am careful to stay at 6,500 until I reach the town of Bennett because any higher than that the Air Traffic Controllers at DIA will be upset the entered their airspace without clearance. With the practice area approaching I start climbing again to the safe altitude of 1500 feet above the ground (AGL) or 7,000 feet and start planning for my first maneuvers. That nervous feeling hits me again as I realize that I am about to force the airplane to do things that seem a little nuts without my trusty instructor beside me to bail me out if something goes wrong. I do some clearing turns to check for traffic, and then get set up for steep turns. The steep turns went pretty well, so now I am ready for some slow flight. The slow flight went well so now lets see if I can do S-turns. I locate the wind get positioned and the S-turns feel right, not drifting too much, compensating well for the wind, hey this is going well so far. Now it is time for stalls, the maneuver I hate the most.

I get the plane configured to simulate landing, flaps at 30 degrees, carb heat warm, power back to idle, and start pulling back on the yoke. It is so unnatural to intentionally make the plane stop flying and lose all lift but I must do it, and do it well for my test. The plane sort-of stalls but not fully, so I recover and get set up for another one, one that would make an examiner happy. Again pulling back on the yoke the sky is all I see, the plane slows down, and the stall horn starts to whine. Then it happens, the point where the plane buffets and the nose drops rapidly (so does my stomach), I hit the throttle, push the yoke forward, start buiding speed, level out, set the carb heat to cold, and begin retracting the flaps 10 degrees at a time. There it is, my most hated maneuver is complete and I know, it's time to go back to the airport and call it day.

Within 10 miles of the airport I start my procedures for landing. I tune the radio to the ASOS and get the weather to make sure I know what the wind is doing and my altimeter is set right. I make my radio calls to let everyone know where I am and what I am going to do. The wind has shifted so I elect to land the opposite direction on the runway because I have not learned how to do a tailwind landing and today is not the day for that. Another pilot announces he is taking off on runway 26, but I have just announced I am landing on runway 8, which the same runway just a different direction. Did he not hear me? This is head on collision about to happen so I start planning for a go around in case this guy decides to take off anyway. Then the pilot who is following me gets on the radio and makes sure the one the ground knows that the active runway is 8 and there are 4 planes about to land that direction. I am still high enough to see that the guy on ground is turning around, and with my confidence restored I proceed with my landing. The landing is not going well so I opt for a go-around anyway just to be safe and get some practice on that maneuver as well. The next attempt is right on the money and once off the runway I go about parking the plane.

As I sit back and review the day, I realize that I was up there for an hour and It seemed like only 30 minutes. I completed another big milestone on my way to private pilot. I feel a little wiser, more confident, and satisfied with my progress. What a fantastic way to spend a saturday morning.

Blog Beginnings

2005-05-26T12:20:00.000-06:00

Today I am writing my first Blog entry and creating a new outlet for my thoughts and opinions on many things. I have been bending the ears of those close to me for a long time with my ramblings on a wide range of subjects and now I have chosen to write them in this Blog. Now people can volunteer to participate in my discourse and even reply if they so choose. You will find here my writings on just about anything you can think of from technology trends, to philosophy, to politics. I also intend to use some of this space as a place to journal about my latest adventure, becoming a private pilot.

There will be more to come soon.
JJ