Table of Contents

&n>

Foreword xiii

Preface xvii

About the Author xxiii

Chapter 1: Overview 1

1.1 What Is Testing? 1

1.2 Testing and the V Models 2

1.3 What Is a Defect? 5

1.4 Why Is Testing Critical? 7

1.5 The Limitations of Testing 9

1.6 What Is a Testing Pitfall? 10

1.7 Categorizing Pitfalls 11

1.8 Pitfall Specifications 11

Chapter 2: Brief Overviews of the Testing Pitfalls 13

2.1 General Testing Pitfalls 13

2.1.1 Test Planning and Scheduling Pitfalls 13

2.1.2 Stakeholder Involvement and Commitment Pitfalls 14

2.1.3 Management-Related Testing Pitfalls 14

2.1.4 Staffing Pitfalls 15

2.1.5 Test-Process Pitfalls 16

2.1.6 Test Tools and Environments Pitfalls 17

2.1.7 Test Communication Pitfalls 18

2.1.8 Requirements-Related Testing Pitfalls 19

2.2 Test-Type-Specific Pitfalls 20

2.2.1 Unit Testing Pitfalls 20

2.2.2 Integration Testing Pitfalls 20

2.2.3 Specialty Engineering Testing Pitfalls 21

2.2.4 System Testing Pitfalls 22

2.2.5 System of Systems (SoS) Testing Pitfalls 22

2.2.6 Regression Testing Pitfalls 23

Chapter 3: Detailed Descriptions of the Testing Pitfalls 25

3.1 Common Negative Consequences 25

3.2 General Recommendations 26

3.3 General Testing Pitfalls 28

3.3.1 Test Planning and Scheduling Pitfalls 28

3.3.2 Stakeholder Involvement and Commitment Pitfalls 44

3.3.3 Management-Related Testing Pitfalls 51

3.3.4 Staffing Pitfalls 65

3.3.5 Test Process Pitfalls 75

3.3.6 Test Tools and Environments Pitfalls 106

3.3.7 Test Communication Pitfalls 131

3.3.8 Requirements-Related Testing Pitfalls 143

3.4 Test-Type-Specific Pitfalls 164

3.4.1 Unit Testing Pitfalls 164

3.4.2 Integration Testing Pitfalls 169

3.4.3 Specialty Engineering Testing Pitfalls 177

3.4.4 System Testing Pitfalls 206

3.4.5 System of Systems (SoS) Testing Pitfalls 211

3.4.6 Regression Testing Pitfalls 225

Chapter 4: Conclusion 241

4.1 Future Work 241

4.2 Maintaining the Lists of Pitfalls 242

Appendix A: Glossary 243

Appendix B: Acronyms 253

Appendix C: Notes 255

Appendix D: REFERENCES 269

Appendix E: Planning Checklist 271

Index 279

About the Author

Donald G. Firesmith is senior member of technical staff in the Software Solutions Division at the Software Engineering Institute (SEI). There, he helps the U.S. Department of Defense and other agencies acquire large, complex, software-reliant systems. An internationally recognized software and systems engineering expert, he has published books on requirements engineering, architecture engineering, situational method engineering, testing, and object-oriented development.

Reviews

“Firesmith’s collection of actionable practices for real-world, non-trivial testing and the processes in which they’re applied is comprehensive and uniquely valuable. Nothing published about software testing in recent years provides anything like it.” –Robert V. Binder, robertvbinder.com   “Don’s compilation of real-world testing problems, symptoms, and solutions is the most comprehensive available. You can use it early in your project to prevent these problems. Or you can use it at the end, as a ready list of costly lessons you could have avoided, had you used it early on. I’m afraid this book’s publication will undermine a lot of excuses for repeating these mistakes.” –Vince Alcalde, National Australia Bank   “Excellent, Excellent, Excellent! This book should be mandatory reading for anyone involved in product development. Donald’s book addresses the pitfalls that need to be understood and allowed for in all product development verification and validation planning. While the focus of the book is on software projects, most of the pitfalls are equally applicable to any size project that involves both hardware and software.” –Louis S. Wheatcraft, Requirements Experts Inc.   “The potential impact of this book cannot be overstressed. Software systems that are not adequately tested do not adequately evolve. I highly recommend this book as a must-read for people directly involved in the development and management of software- intensive systems.” –Dr. Kenneth E. Nidiffer, Director of Strategic Plans for Government Programs, Software Engineering Institute, Carnegie Mellon University   “Common System and Software Testing Pitfalls identifies realistic testing pitfalls. More importantly, it also identifies solutions for avoiding them on your next project. Every manager should read this book and follow the recommendations.” –Barry Stanly, Enterprise Technology Alliance   “Whether you are a novice tester or a seasoned professional, you will find this book to be a valuable resource. The information on how to identify and prevent problem areas is clear, concise and, most importantly, actionable.” –Allison Yeager, Blackbaud   “First of all, this is great material! It contains probably all of the testing problems I have faced in my career and some that I wasn’t aware of. . . . Thank you for the opportunity to read this superb material!” –Alexandru Cosma, Frequentis   “As a tester, I consider Common System and Software Testing Pitfalls by Donald Firesmith to be a must-read book for all testers and QA engineers.” –Thanh Huynh, LogiGear   “Your book provides very good insight and knowledge. After working in IT for over thirty years, and focusing on software testing the past thirteen years, I still learned more tips and best practices in software testing.” –Tom Zalewski, Texas State Government   “This book is essential for the people in the cyber security business . . . I can see it becoming a classic. Don has done a great job.” –Michael Hom, Compass360 Consulting   “Awesome work. Very mature.” –Alejandro Salado, Kaiser, Threde Gmbh   “All in all, a great document.” –Peter Bolin, Revolution IT Pty Ltd.