Desktop Analytics: First Real-World Impressions

To the surprise of many, Microsoft announced on September 6th, 2019, that it will retire its Windows Analytics services in favor of Desktop Analytics. While the reality of Desktop Analytics (DA) replacing Windows Analytics was expected at some point, the timing is surprising as Desktop Analytics is still in Public Preview and the deadline is in just 4.5 months.

To get customers ready, Microsoft is encouraging everyone to start evaluating Desktop Analytics now because new customers will no longer have access to Windows Analytics Upgrade Readiness and Device Health once DA becomes generally available. Customers using “Update Compliance” in their Azure portal already can continue to do so — even after January 2020.

But there are some customer grumbles, some confusion, and a lot of questions about the new suite of cloud services. We have had a chance to work with it for the past four weeks in an enterprise environment with ~15,000 seats to inform both on an upgrade to Windows 10 and an existing update to the 1903 version of Windows 10.

We thought it would be beneficial (especially for those of you who haven't had a chance to evaluate Desktop Analytics yet or who are not sure whether or not it is worth trying) to share our first impression of the Windows Analytics successor as well as some real-world insights and results. Today, we will focus on the readiness aspect of things as this is most important to our customers, and we will cover security, health, and performance in later blog posts in the coming months. Note that screenshots used in this blog post are from Microsoft's generic documentation as for security reasons we cannot show customer data.

A Very Short Desktop Analytics Primer

Since it was first announced at Inspire in September 2018, Desktop Analytics was pegged as the replacement for Windows Analytics. As a reminder, Windows Analytics is a suite of cloud services aimed to provide additional insight when upgrading and updating Windows 10 and is comprised of three separate services: “Upgrade Readiness”, “Update Compliance”, and “Device Health”. It was the successor of first Windows Diagnostic Data and then Upgrade Analytics.

Since then, Desktop Analytics has evolved from simply providing data to creating deployment plans based on the insights, planning and deploying a pilot, and in production devices. That's at least its function in theory — more to the actual results in a minute. 

Essentially, Microsoft sells Desktop Analytics as a service that enables you to:

• Create an application inventory 
• Assess those apps for compatibility with the latest Windows 10 feature updates
• Identify any known compatibility issues (and receive mitigation suggestions based on cloud-enabled data insights)
• Create "optimized" pilot groups representing the entire application and driver estate across a minimal set of devices
• Deploy Windows 10 to pilot and production-managed devices

The idea is to identify and migrate as many no-problem devices as quickly as possible before dealing with devices that require minor intervention before being able to migrate or have problems that might have to be resolved manually. 

How Does Desktop Analytics Work?

Before we dive into the pros and cons, let's take a look at the functionality of Desktop Analytics.

Once you have access and have fulfilled all the prerequisite requirements, you can log into the Desktop Analytics Azure portal. On the home screen you will find three major task categories (see screenshot below):

• Manage (Deployment Plans & Assets),
• Monitor (Security Updates, Feature Upgrades), and
• Global Settings (Global Pilot, Connected Services).

One of the areas users will probably spend the most time in is "Deployment plans" — now a critical component of the tool.

Deployment plans are automatically created lists of devices that should be included in your upgrade based on identified criteria. Each deployment plan includes mitigation recommendations, deployment health assessments, and deployment progress tracking. These deployment plans are informed by collected and analyzed device, application, and driver data as well as user input.

Within the Deployment Plan tab, you will find the following items:

1) Prepare

Identify Importance - In this step, you are asked to assess (based on compatibility and other insights) and to manually edit the applications showing up in the app tab. If you assign "Critical" or "Important", Desktop Analytics will validate the app during the pilot, while apps marked "Not Important" will not be included in any analysis.

Identify Pilot - After you have categorized your applications, Desktop Analytics will now recommend certain devices to be included in the pilot. These machines should theoretically represent the vast majority of your environment. In the "Identify Pilot" stage, you will need to review them and select particular ones (ask DA to replace the non-selected ones) or bulk-select all recommended devices to be added to the pilot. You can also choose to select none of them and ask DA to recalculate. (Remember: this suggestion is only as good as your input!)

Prepare Pilot - Once you have selected which devices to include in the pilot, you can now prepare your pilot for deployment in this step.

2) Pilot

Deployment Status - The deployment status tab gives you an overview of all your pilot devices. This can either be:

• Review in progress
• Ready
• Ready (with remediation)
• Unable
• Not reviewed

You can either get an impression of your overall progress through the pie chart summarizing the status or toggle between the different statuses to get more detail.

Prepare Production - In order to move your devices into production deployment, their upgrade decision has to be set to either "Ready" or "Ready (with remediation)" which in turn will trigger the Configuration Manager phased deployment as Desktop Analytics automatically synchronizes those devices to Configuration Manager deployment plan collections. 

3) Deploy

Deployment Status - Similarly to the Pilot Deployment Status, the Production Deployment Status will provide you with an overview of the deployment status of your devices in your production environment. However, the categories change to:

• Not started
• In progress
• Completed
• Needs attention - Devices (sorted by device name)
• Needs attention - Issues (sorted by issue type)

In our use case, the top three issues we encountered several dozen times were:

• An application is being removed during the upgrade,
• An application needs to evaluate on new operating system before moving on, and 
• An application does not work with new operating system, but won't block the upgrade.

In very few occurrences, we encountered these issues:

• Application blocking upgrade,
• Application does not work with the new operating system and blocks the upgrade,
• Applications with multiple issues, and
• Reinstallation of an application was required after upgrading.

Monitor Health - Last, but not least, you can view, troubleshoot, and fix any potential health issues on the Monitor Health page. You could even change your upgrade decision to "Unable" if needed to prevent any future upgrades on devices with a specific troublesome application.

First Impressions: What Worked? What Could Improve?

Now let's talk about the real-world impressions we got from the tool.

Let's first start with the onboarding experience — which for this particular client was a mixed experience as they encountered some challenges getting their devices to enroll into Desktop Analytics. To be fair, some of these struggles were due to internal technical network limitations when connecting to the Microsoft Cloud services to push that data up. Also, Desktop Analytics is still in Public Preview so some bugs had to be expected, but they worked closely with Microsoft to iron those kinks out eventually.

Secondly, we feel that the interface has improved compared to Windows Analytics. While Windows Analytics already made use of tiles in its UI, the Desktop Analytics interface guides the users through a hierarchical process. However, while Desktop Analytics' interface itself offers some reporting, our customer felt that the Desktop Analytics reporting interface was very restrictive or limited in its functionality as they struggled to get the level of detail they needed.

Only after we hooked it up to Dashworks using our Dashworks Connector for Desktop Analytics, were they able to slice and dice the data the way they needed it. They could then look up any computer and application that Desktop Analytics says is ready or not ready and get into the detail. Although Desktop Analytics has all of that data, you can't get to it at that level through the Desktop Analytics interface. 

• Hardware Inventory Contains Only Onboarded Devices. Desktop Analytics only creates an inventory of all devices that have been onboarded into Desktop Analytics. Once successfully onboarded, it can provide information about whether or not a device meets the target OS hardware requirements, available memory capacity, any BIOS block, and more. But you have to first know what you have in house in order to know which machines to onboard. Also, please note that SCCM version 1902 with update rollup (4500571) is mandatory to provide hardware information now.

• Time-Consuming Manual Assignment Of App Categories. In Desktop Analytics, there are certain attributes of an application that you can change, such as whether you think it's a critical application or whether it's ready for deployment. You can update that in Desktop Analytics, but at the moment, there isn't a bulk way to do that. You have to do it one by one, and with thousands of installed applications in most enterprises, that is a very time-consuming process. A bulk update is not possible.

• Decreased Application Usage Data Visibility. As for applications, the visibility in usage data has reduced compared to its predecessor. Also, please note that the application readiness assessment is created as the Desktop Analytics generated application inventory is compared against Microsoft's telemetry-generated app catalog. Essentially, Microsoft assumes if a certain app run successfully in the same environment for other organizations, it will do the same here.

• Basic Features For Project Command & Control. Whilst the integration with SCCM is useful, DA cannot support common scenarios such as applications being replaced by other, compatible applications, control end user communications, manage scheduling capacity or track readiness tasks outside of the existing application (such as a new device procurement). This makes it quite restrictive for most customers who would revert to third party tools like Juriba's Dashworks or their own internal systems.

In addition, we have seen other clients voice concerns regarding the Intune integration as well as the licensing model.

To run Desktop Analytics, an organization must have one of the following licenses to gain access to the cloud services:

• Windows 10 Enterprise E3 or E5 or Microsoft 365 F1, E3, or E5
• Windows 10 Education A3 or A5 or Microsoft 365 A3 or A5
• Virtual desktop access E3 or E5

For many, that is a big change as Update Compliance and Upgrade Readiness were included at no cost as part of their Windows 10 Pro, Enterprise E1, and Education A1 license. Device Health only required Windows 10 Enterprise/Education with active Software Assurance.

In addition, customers who already bought into Microsoft's Modern IT Management vision and switched to Intune rather than continuing to use SCCM are now left in the rain. While Microsoft claims there will be an integration "in the near future", they have been making that claim since September 2018. Officially, Microsoft states that, at this moment, they cannot share any exact timelines on Intune support, but they have said this: "[We] are actively thinking about the best way to provide Intune customers with Analytics insights and [it] is high on our list after we GA the service." Of course this means that any devices or applications managed in InTune are excluded from any DA workflows.

Conclusion

Due to the limited level of detail you can access through the Desktop Analytics dashboards, the lack of adequate planning algorithms, and the fact that, for most enterprises we work with, a simple pilot wouldn't be sufficient to move into broad deployment, it is difficult to see a large enterprise running an entire migration just using Desktop Analytics. However, let us remember that it is a product in development.

At Juriba, we find that DA becomes a very powerful data source once you combine it with SCCM, Active Directory, HR data, and other location information to scope your upgrade project. Managing that project we would still strongly advise that you look at more functional project command and control tooling.