Co-ordinating the testing of hundreds or thousands of enterprise applications against a new operating system or patch release has long been the bane of many an application manager's life. But now there is a new approach. Smoke testing can provide a quick, inexpensive, and minimally disruptive way to run your applications through an automated test, verifying that the most essential (or core) functionalities are working as expected. Successfully implemented, it is a crucial part of your Evergreen IT strategy.
While the core principles and outcomes of smoke testing in the Evergreen IT context remain the same as with your initial OS image build testing, it is worth walking through the fundamentals of smoke testing as it pertains to your Evergreen IT strategy. I will also outline the resulting benefits, the required steps, and best practices to perform some testing as part of Evergreen IT Management.
Smoke Testing: Definition & Origin
It is unclear where the name "smoke testing" exactly comes from, but the two most accepted theories are that it refers to the very initial hardware test to see if it starts smoking when turned on, or the smoke that a plumber uses when trying to pinpoint cracks or leaks.
In the software development sense, smoke testing refers to the initial screening of a software build's core features before it goes to the testing and/or quality assurance control teams for further in-depth testing. The reasoning is simple: if the application cannot fulfill its original purpose, e.g., for an eCommerce application this would be to find, add to cart, and purchase a product, there is no need to test any other lesser functionality, such as the ability to easily rate and review products.
It is also often called intake testing, confidence testing, build verification test (BVT), and build acceptance test. Sometimes it is also confused with sanity testing, which is another superficial testing of core functionality, but occurs shortly before releasing the app rather than at the beginning of the testing process.
For software development, smoke testing is a subset of regression testing, but because it is non-exhaustive, it isn't a substitute for further testing. Instead, it should be understood as a gatekeeper to check if a product is ready for further testing. It essentially determines the next level of testing required. Its purpose is to identify any significant bugs and defects as early as possible using the 80/20 rule — the test only covers 20% of the functionality but finds 80% of the problems.
Similarly, in the context of Evergreen IT Management, we will define smoke testing as an 80/20 test including the install, launch and uninstall of an application to ensure that it will run in the new environment (e.g., after a Windows 10 Servicing upgrade). For those applications that require it, further automated or manual functional tests can then be initiated.
Why You Need Smoke Testing As Part Of Evergreen IT Management
Evergreen IT Management requires testing thousands of applications every year. Large organizations might choose to roll out updates to their Office 365 ProPlus clients twice a year and upgrade their Windows 10 annually, which means application testing immediately becomes your number one bottleneck. And that is before we consider any patch testing!
Just do the math: if an enterprise Windows migration touches an average of 1,500 apps, and each test takes 3 hours, you can imagine that it quickly becomes an issue to keep pace with the required change, or you simply have to accept the risk that you cannot physically test that many applications within the allotted time frame.
This is why automated smoke testing is critical for Evergreen IT Management. Compared to full-blown application testing, which often takes hours or days, smoke testing can be carried out in a matter of minutes, saving precious time and resources. Besides, smoke testing:
- Improves efficiencies of your development, testing, and quality assurance teams
- Dramatically reduces the need for manual testing and subsequent intervention
- Eliminates the need for full-blown app testing when running through an Evergreen IT cycle
- Empowers organizations to either test their core or all of their applications with a higher level of confidence
- Significantly accelerates Evergreen IT cycles by efficiently determining the application readiness (and therefore the deployment ring progression)
How To Perform Evergreen IT Smoke Testing
Performing smoke testing and automated functional testing is relatively straight-forward, especially with the right tooling.
First, you have to decide on the criticality of your applications. This will dictate the level of testing required for each application to be considered ready for the upgrade. For those that are low risk or low impact then a smoke test might be the ideal risk mitigation strategy, especially when compared against databases (e.g. desktop analytics) or static analysis tests.
For the next level of applications it is important to define your test cases. These are designed to expose errors in the core features. For a complete test, you will bundle together a manageable number of test cases (usually 20-50 cases). To make your testing repeatable, be sure to create a testing framework that allows you to store and manage your test cases for subsequent test cycles. Now you will need to execute and capture the test before reporting back any possible errors and routing it either back to the packaging team or, in the case of a successful test, to the next step.
Remember to:
- Test early and often (rule number one with any testing!)
- Record all your smoke testing (you will want to have evidence-based testing, e.g., by logging outcomes but also by attaching the video of the entire test for later follow-up if needed)
- Create test cases that are short, precise, and cover your basic functionality
- Automate whenever possible and aim to create a repeatable test case repository to speed up further testing while maintaining consistency
There are many smoke testing tools available in the market and Juriba expects these to make great strides given the immediate requirement for accelerated application testing in large organisations. Whatever tooling you choose, be sure to automate your process as much as possible, including integrating it into your Evergreen IT Management tooling and processes.
