Traditionally, User Acceptance Testing (UAT) is performed at the end of a long software development cycle by the intended audience under real-world conditions. By then, engineering has performed a battery of technical tests to ensure the software works as expected. However, in some cases, engineering's understanding of the business requirements and user needs versus what the user was actually looking for are two very different things. This is where UAT usually comes in.
But UAT is not simply for new software development. It is just as applicable for environmental changes such as an application upgrade or ensuring application functionality works under a different operating system version.
However, many enterprises would rather skip it since this can be a resource-hungry, expensive, and long-winded process. Yet, a well-managed UAT process allows you to find and fix major (or minor) user experience or functionality flaws that could prevent adoption or halt productivity before rolling it out to a larger audience.
So what about today?
With Windows as a service, enterprises are starting to understand the potential impact of the UAT requirement against a multitude of new operating system releases. Microsoft are telling us not to worry - most applications will work just fine. Do you really need User Acceptance Testing as more and more enterprise IT environments are managed in "Business-as-Usual" (BAU) or "Evergreen IT" mode?
Today, I want to take a closer look at why UAT is even more critical today than ever before, what an efficient testing process looks like and what options you have.
Why Is UAT Becoming More Important In The Future?
Today's IT teams have to face completely different challenges compared to 10 to 15 years ago. While back then, we struggled with big-bang OS rollouts every three to four years, we are faced now with a mix of on-premise and cloud-based applications in various lifecycle stages and very fast-paced upgrade timelines. Our entire estate management has become much more fluid, and therefore incredibly complex.
While larger IT projects were overwhelming and almost always over budget and over time, there was an end in sight at some point. You could come up for air once in a while, rethink your strategy, and start fresh. Also, your application discovery, packaging, and testing were done in one more or less efficient fell swoop before you could move on into your deployment phase.
Once the project was complete, standard processes were engaged for UAT of new applications into the environment, but the existing applications did not really require any further thought unless the vendor forced an upgrade. However, with a greater velocity of change and faster upgrade cycles that come with new IT management models such as Windows 10 Branching, application testing, and other migration tasks, it must now be considered an ongoing effort.
What Does An Efficient UAT Process Look Like?
To start your Windows 10 Branching UAT process, you will need to first identify which applications need testing and run through a thorough discovery process to unearth the relevant MSI or source files and any other resources required. You may have a DSL (Definitive Software Library) to help, but if not, your inventory tools such as SCCM will provide a list of what is out there. Once you have identified all the required assets, you will have to prioritize and notify the application owner within your environment.
Traditionally, if you want to perform a User Acceptance Test for a business application in a new environment (e.g., Windows 10) you would then follow this process:
- First, your IT team installs the application on a Windows 10 machine that exactly mirrors real-world conditions (e.g., permissions, authorizations) for the type of business users that will be using this app.
- IT will provide the user with the machine, the log-in credentials and a mechanism to capture the results.
- Then, the end user will open the app, run through usage scenarios (user stories). and record their behavior and outcomes.
- After the test is completed and the results are recorded, the end user will give back the machine.
- IT will wipe it clean and prepare it for the next application test.
Depending on how you structure this process and which testing model you choose to follow, this can take a long time and many resources. It can lead to inconsistent results as it is much based on the willingness and diligence of the tester.
The modern approach to UAT would be to use virtual machines. With the right tooling, you can easily spin up pools of Virtual Machines in parallel, notify your users automatically including all credentials and instructions, and record testing sessions to provide evidence of successful testing. After the test is completed, a workflow can route the app forward (success) or backwards for manual testing (failure), and the virtual machine can be killed off in an instant — resulting in significant time and cost savings!
As you will be testing many applications on an ongoing basis, you will have to think about how you will prioritize which applications to test first and last. There are several options for structuring your Acceptance Testing — i.e., by
- Application owner list
- Application list
- Application volume
- Expected testing difficulty
- How business critical the application is
No matter how you prioritize and organize your application testing, you will need to have the right tooling in place to enable you to identify which applications work (certification), and which require testing.
Impact Of User Acceptance Testing On Windows Branching
To better visualize the impact of (inefficient) User Acceptance Testing on applications and users in a BAU environment, let's have a look at a scenario for Windows 10 Branching. I described above the workflow from an application point of view when I mapped out the general UAT process. Now, we are looking at all of your applications moving to a new operating system release within a timebound deployment window. When you add the users into this view, you can see how the two are intrinsically connected.
As users get onboarded into their default deployment ring, their upgrade schedule is heavily reliant on the application readiness. If your applications are not ready, they need to go back into app owner testing, and all to-be-migrated users come to a standstill. You can easily imagine that determining user candidacy and an efficient (= correctly prioritized) UAT process is critical — otherwise you are running the danger of not getting any deployments on the scoreboard!
What UAT Options Do You Have?
Before you decide which tooling and which testing approach is right for your needs, it is important to know that there are several different levels of testing involved, namely:
- Non-Functional Testing. Non-functional testing only ensures that the application can be installed properly, loads and opens as expected.
- Functional Testing. Functional testing, on the other hand, will test that the application delivered the expected outcome. For example, if I want to test a payment calculator, I want to test here that I can enter all required data points and that the calculated result is correct and displayed in an easy-to-understand way. Of course, this is the simplest example and this gets much more complex when we talk about custom-developed niche applications.
- Performance Testing. Performance testing adds another layer on top of functional and non-functional testing. Here, you will test how much the application impacts the environment it is running in, e.g., CPU usage.
Many tooling vendors have seen this need as an opportunity to fill that gap and provide a solution to ease these testing pains. Depending on their expertise, product portfolio and strategy, they all have a different approach to this:
- Office Compatibility Testing. Applications Readiness, Citrix's AppDNA, and other similarly sophisticated tools will interrogate the application's code even before it gets loaded to discover areas that are most likely to be impacted or broken by any change or any deployment model, such as a new operating system or a new application combination. They identify components that are incompatible with the changes in the operating system, and are more likely to cause user productivity issues. These tools are great for your in-house application compatibility testing.
- Telemetry Data. Microsoft's free Upgrade Readiness (formerly known as Upgrade Analytics) tool allows you to gain insight into Windows 10 compatibility based on millions of telemetry data points. We have written about Upgrade Readiness before, but it is important to point out that many of these data points are provided by consumers, not enterprise users, so there will be a caveat of restricted use to gauge the readiness of your in-house developed and enterprise business applications.
- Software Catalog. A more sophisticated approach is to combine the telemetry data with other aggregated and scrubbed product intelligence, like BDNA's software and hardware catalog Technopedia does. This approach is great for your vendor applications, but is reliant on the speed at which the catalog can be updated for the new release (Telemetry data will be quicker, but possibly not as easy to match to your real applications).
- Automated Testing. If you are testing a lot of apps that expect a similar outcome, you could also build a workflow to automate your testing by using a tool like Selenium — an open-source software testing framework for web applications across multiple browsers and platforms. Getting this set up will take a lot of time from your test resources, but once done, as in software development, the results are likely to accelerate application readiness.
- Workflow-Based Testing. You can also automate your UAT testing with specifically developed tools, like Access Capture, to automate your entire application discovery, packaging, and testing process in one streamlined process. The difference with the above mentioned tool is that you don't build the workflow yourself, but customize it to your needs and it spans your end-to-end process instead of just the UAT component.
Conclusion
No matter which approach or tooling you decide on, it is important to know that the User Acceptance Testing tool market is very immature. While there are many different options, we will have to wait and see which solution approach will rise to the top. In the meantime, enterprises who are looking to invest in a UAT solution have to understand that currently, most vendors offer only very isolated solutions, and it is likely that you will require a combination of tools to cover all your requirements.
Nevertheless, it is essential to the success of your BAU UAT that you understand what and when to test, how to prioritize your application readiness schedule, and subsequently your deployment tasks. Therefore, an essential command and control tool like Juriba's Enterprise IT Transformation Management Tool, Dashworks, is indispensable.
