Windows Search service + Citrix VDA ghost sessions

I recently went through a bunch of OS in-place upgrades from 2016 to Server 2019.  Some time after the upgrade, one of our VDA servers suddenly stopped accepting RDP connection and Citrix Studio is showing that there are over 100+ sessions connected.  Users reported that they cannot launch any applications.  I did a remote PowerShell session and a qwinsta query returned more than 100 disconnected sessions with no usernames.  This is an issue that I saw more often in Microsoft RDS deployment, but not as much in a Citrix environment.

This VDA is part of a delivery Group that has 14 VDA servers that deliver the same set of apps.  I put this VDA on Maintenance mode and restarted it.  When the server came back up, I was able to RDP to the VDA again.  A qwinsta query returned no disconnected session and I was the only user shown as connected.  These kind of ghost sessions usually cleared up after reboots. 

I checked the event logs and saw that the Windows Search service Terminated a million times (652k +)!  Because the windows search service was not running, Citrix Profile Management was also having issue.  I never knew that Citrix Profile Management uses windows search to handle the profiles, but in a way, it makes perfect sense.  The system needs a way to “search” for the profile locations and when windows search service is not working properly, how can it locate all the profiles? It was not able to allow connected/disconnected sessions to sign off, causing 100+ ghost sessions stuck in this VDA.

I launched Services.msc and saw that the service is not running.  I tried to start it manually, and was only greeted by the error message below.

The resolution to this issue was to rename the registry key below and then restart the server. 

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows Search\FileChangeClientConfigs

To be on the safe side, I exported this key first, then rename it to FileChangeClientConfigs.old.  Windows Search service was able to start again after the VDA was restarted, creating/applying a fresh copy of this registry key.