Software Testing

Lessons Learned At The DMV: Don’t Skimp On Software Testing

Pinterest LinkedIn Tumblr

I recently bought my son his first car from a private party. The state of California requires registration within 10 days of purchase, which must be done in person at the Department of Motor Vehicles (DMV). That 10-day window is too narrow for acquiring an appointment, which meant that I was going to have to stand in line. A very long line.

Eager to register to avoid paying a penalty, I went to the DMV two days after purchase. I showed up 30 minutes before it opened, thinking I might get ahead of some of the demand. I was wrong.

When I arrived, there was a line wrapped around the building that had to number 300 people. It was 90 minutes before I even got in the building to get a number for the privilege of continuing to wait in a chair. About an hour into that wait, the words were spoken that no motorist waiting at the DMV ever wants to hear: “We’re sorry, but the computer systems statewide are down, and we don’t know when they’ll be up again. You’re welcome to wait.”

What Caused The Outage?

As someone who works in the IT industry, this piqued my interest. I was curious if there was an official announcement on the California DMV Twitter page. There was no mention of the outage there, but I did find their logo amusing: “DMV, Driving Change …” Seems ironic because it’s change that often causes problems within IT.

About 30 minutes after the initial announcement, the staff announced that the queuing system had failed and they were going to begin calling out numbers manually. This article from late February described exactly what I was experiencing. The cause of the February issue in the queuing system was “a high number of attempts to access the central database at the same time.”

Apparently, a new queuing system had been implemented without sufficient capacity for real-world demand, and it continues to cause issues. The employees at the DMV handled it professionally, but I had to wonder how many times they’ve had to manually work around this problem. They seemed to have experience.

Software Development Methods

While picking on the DMV can be all too easy, its challenge isn’t unique. It’s common for applications to be rushed into production and have problems as a result. Without sufficient time for testing and remediation of problems, an organization’s users and customers become manual testers, with predictably bad results.

To understand how to mitigate the risk of unplanned outages caused by insufficiently tested software, let’s look first at how development teams deliver software. Software development typically follows one of two common methods — waterfall or agile, and each one handles software testing differently.

With a traditional waterfall development approach, testing applications is typically performed at the end of the development life cycle, because testing under real-world stress conditions isn’t possible until all components are complete. As the deadline looms for deployment, time left over for the test phase gets reduced and bugs either remain undiscovered or deferred for remediation in future versions.

Many enterprise development organizations have shifted toward agile development and DevOps practices, which makes the feedback that software testing provides part of a continuous integration (CI) loop. With CI, a developer will check out bite-size amounts of code, modify it, test it and reintegrate it often, sometimes on a daily basis. Code that passes testing is then available for deployment whenever business conditions are met.

Agile and DevOps methods can better support fast-moving application development requirements to keep pace with business competitors. But there will remain some older, proven applications that don’t change as often, which will likely remain dedicated to a waterfall method. Regardless of the development method, testing is necessary.

Types Of Software Testing

Software testing isn’t just about whether it works as intended in a lab, though. There are several types of testing necessary to deliver the best user experience:

• Functional testing checks software to ensure that all required features are working as specified in the design documentation. This is performed by providing input and comparing the output against expected results.

• Regression testing is a specialized part of functional testing to ensure that new code works well with pre-existing code. This requires re-running previously completed tests to see if faults have reemerged. As software complexity increases, this can become exponentially more difficult.

• User experience testing is another part of functional testing that seeks to evaluate how users perceive the software. An application may meet all functional requirements but have a poor user experience that impacts usability and, subsequently, adoption.

• Performance and load testing checks software and hardware in the entire system to see if it is capable of handling real-world loads. Software that passes functional testing may not scale to meet the requirements of real-world demand, as I experienced at the DMV.

• Application security testing identifies known vulnerabilities in code before it’s released into production where it can be compromised by attackers. Developers often use open source libraries of code to accelerate their development efforts, but this can introduce vulnerabilities unintentionally. This type of testing is aimed at remediating those unintentional vulnerabilities.

The key point is that in order for the business to gain a return on software investments, multiple types of software testing must be coordinated and managed to deliver an end result that makes users happy and will keep customers coming back to your business.

Business leaders often push for faster deployment of new software to gain competitive advantage, but the introduction of bugs or problems can also cost business opportunities. As legendary UCLA basketball coach John Wooden said, “Be quick but don’t hurry.” It takes an investment of time, people and technology to do testing right. Skimping on software testing is tempting, but if you want to avoid unhappy users, the negative social media they can generate and lost business opportunities, then software testing is a worthwhile investment.