UPDATE: January 30, 2018 The Washington Post published an article with more details: https://www.washingtonpost.com/news/the-switch/wp/2018/01/30/heres-what-went-wrong-with-that-hawaii-missile-alert-the-fcc-says/?utm_term=.84e766af06c6 It is interesting that one of the suggestions I mentioned to prevent this was actually in place, "They then must click 'yes' when the system asks 'Are you sure that you want to send this Alert?'" the article also stated that the application uses “the same language irrespective of whether the message [is] a test or actual alert.” Hmmm.
I live most of the year in Hawaii on the island of Oahu and was woken by this alert last week. Needless to say it was a distressing event that should not have happened. I don’t claim to have any inside information other than what I have read in the paper, but I have managed enough software development projects to know the root cause of this kind of error. I have been contemplating writing about this for the last week, and first of all I want to say, “Do not fire the person that sent this out!” This was user error but based upon a poorly designed system that is all too common in software development projects. For those of you who aren’t in the industry, let me walk you through how this happens:
“Traditional” Approach to Project Management:
Step 1 Business Requirements: A business person or customer decides what they want an application to do and they create a high-level requirement. e,g. “Create an application to allow a user to send out an alert to the phone and television emergency systems. This application should have a dual purpose for both testing the system and for sending actual alerts.”
Step 2 Analysis and Design: Analysts then create a set of system requirements based the business requirement. A designer designs screens to support the application. An architect designs how each piece will work together. They write everything down and get sign-off from their business stakeholder. They then hand it to a software developer to build. Sounds good so far?
Step 3 Development: (This is where it really break down) The developer builds it exactly the way it was written down!
Step 4 Testing: (Final failure point.) The developer hands-off the application to the tester, who tests it to make sure it meets the original business requirement and technical specifications. Once it passes the tests, it is ready to go live.
Again, for those of you who are not in this industry, the above steps probably seem completely reasonable, and many projects are still run this way. The problem is, this approach is why 50% of projects fail. I worked under this model for about 10 years until I experienced a couple of major project failures. Those project failures probably cost more than the above mentioned failure because they were expensive commercial software development projects that built a product “exactly the way it was written down.”
After those experiences, I became an Agile evangelist. For the last 10+ years I have focused on managing, consulting and training on Agile practices. We still have failures but there are far fewer, and if we fail, we fail fast. Also, Agile software development is a true collaboration between the users and the developers. So, for those of you who do not know much about Agile, let me tell you how we (the Agile community) would manage this project:
Agile Approach to Project Management:
Step 1 Business Requirements: This step is pretty much the same - a business person creates a high-level requirement. e,g. “Create an application to allow the user to send out an alert to the phone and television emergency systems. This application should have a dual purpose both for testing the system and for actual alerts.” We call this big requirement an Epic.
Step 2 Analysis and Design – Agile Discovery: (This is where everything changes) There is no analyst or middle-man between the developer and the business. Here is what we do:
- The business person is part of the project and joins the development team.
- At the start of the project we then have a workshop to break that big requirement into smaller requirements called user stories.
- Included in the workshop are the people who will be developing and testing the software, business people, potential users and any interested stakeholders.
- There is a facilitated session where we review the high level requirements and ask the following questions*:
What will the user likely do next?
What mistakes could they make?
What could confuse them?
What additional information might they need?
*Let me be clear, I did not come up with these questions in response that alert failure. It may look like I am using hindsight to fix the problem since asking those questions and addressing them would have obviously prevented the alert from being sent out. The reality is, I took those questions directly from an Agile training I have been delivering for years. I learned that technique from a leading Agile author, Mike Cohn, in his book User Stories Applied: For Agile Software Development. This is a common technique in Agile to ensure an application is well thought through.
- The final difference in this step is that in that requirements meeting the developers are listening, asking questions, and writing notes that make up the specification. There is no second hand information, they design and build the application based upon their conversation with their customer.
Step 3 Agile Development: A key difference in Agile is that requirements are presented as a problem to solve not a specification to be built. It is the developers’ job to come up with a solution. The developers understand the business need since they reviewed the requirements with their customer. They are empowered to ask questions and clarify the design while they build it. They have a business owner or customer representative on the team who is available daily to answer questions. They are asking questions in real time, such as, “It seems like the requirement to test the system and execute a real alert are very different activities and should have a very different workflow and authentication model? Technically, there are many ways we can prevent errors from happening. Can I show you a couple of options?” Again, I am not using hindsight. I worked with systems that had a single critical field that if filled in incorrectly required a lot of work to fix. In those cases developers know the technical, and often simple, best practices to ensure quality of data entry. You may have seen these before even on a regular webpage:
- Please enter your password again (replace the word password with “authorization for an alert”)
- Please type YES if you want to close this screen without saving (replace the words without saving to “send the alert”)
- A more advanced technique I have used in the past is called Double Key Data Entry.This a common technique for ensuring quality of high-risk data entry. Two different people have to enter the same information before the workflow can progress. I used it on a project for entering SSNs since an incorrectly typed SSN caused a new customer to be created instead of updating an existing customer. This prevented a lot of rework. Seems like this would be a great approach for this situation.
Step 4 Agile Testing: Testing is done very differently as well. We test as soon as we build a single functioning piece of software. We don’t wait until we build the entire application. That prevents the high cost of fixing errors. Also, another difference, and probably more critical in this situation, is we write all of our test cases, called acceptance criteria, before we develop anything. That ensures that the developer knows how the product is going to be tested and then used in the real world.
Again, don’t blame the end-user. This happens all the time our industry it’s just that the failures aren’t so public. This is why implementing Agile has become mission critical for many organizations. Implementing Agile to prevent these problems rather than assigning blame is the real lesson learned from this experience.
About the Author, Dan Tousignant
Dan is the president of Cape Project Management, Inc. and is a lifelong project manager. He has embraced Agile as the most effective way to manage software development projects. He has been managing mission critical projects and developing training for over twenty-years and is passionate about improving project team performance.
