Hi and welcome to this first part about troubleshooting Citrix XenApp. Last November I had the honor to speak at two well-known EUC and Citrix events.

First of all at E2EVC in Rome, Italy, and the day after at DCUG TecCon in Kassel, Germany. The topic of both sessions was “Troubleshoot XenApp with Style”.

I am writing this blog for all of you that couldn´t attend one of these great conferences. This first part is about the Citrix tools because otherwise, it would grow too big if I would integrate all parts. The second part is about some of the Microsoft tools. The third part is about general troubleshooting tools that didn´t fit into the session.

Why Should You Read It?

This post is not about the usual problems you might have with Certificates, Firewalls, or similar things where you might be able to find an answer by googling around or checking the knowledge base.

It is more about the methodology and the tools to use when you are not able to find anything on the whole world wide web.

What About Existing Troubleshooting Guides?

You can check two good troubleshooting guides from Citrix regarding NetScaler, StoreFront, XenApp, and slow logons following these both links:

Troubleshooting Methodology for NetScaler, StoreFront with XenApp and/or XenDesktop

How to Troubleshoot Slow Logons on XenApp

How To Start?

If you are in charge of your Citrix XenApp environment or if you are a consultant arriving at a customer site for troubleshooting purposes you should ask yourself and your customer the following questions to get a picture of the problem:

• What is the problem?
• Who has the problem?
• Since when do you have the problem?
• Can you track down a specific time when the problem happens?
  • Is a specific location affected?
  • Is a specific system affected?
• During which action does it happen?

There are a lot more questions you could ask to get more details about what is going on but you should start simple.

Questions should lead you as near as possible to the component that is creating problems.

Because Citrix XenApp is complex in its nature because of the dependencies to a lot of other infrastructure components you should go on asking about infrastructure details. Ask questions about (not a complete list):

• Active Directory
  • Site configuration
  • Kerberos settings
  • DNS configuration
  • NTP
• Network
  • Routes
  • Load Balancing
• File servers
  • SMB settings
  • Folder redirection
• Citrix infrastructure components
• Application deployment and configuration
  • application configuration
• Group Policies
  • Loopback mode
  • Policies in place
  • WMI filters
• Hypervisors
• Operating system
• other specific configurations
  • hardcoded settings

Things To Do In The Beginning

After you´ve asked all the questions you should be able to get a more precise picture of the problem and after that, it is time to check the technological aspects.

Always start simple.

• Start simple
  • check event logs
  • use Steps Recorder (I´ll explain that in the second part of the blog about Microsoft tools)
    • Gather errors and messages
• Dive deeper
  • use verbose and debug logging
  • gather network traces
• Gather as much information as you can get
• Identify the issue

Try to get as much information as you can get but always keep an eye on the amount of data you might get during verbose and debug logging.

Keep the amount of data you need to analyze as low as possible.

You should try to reproduce the problem to minimize data. If it is not reproducible you will have to go through a lot of data. It is really no fun to analyze 15 GB of ProcessMonitor data. now let´s dive into the tools.

Citrix Supportability Toolkit

The recent Citrix Supportability Toolkit consists of 51 tools that deal with all kinds of problems regarding Citrix infrastructures. I won´t go through all of these here but I will choose some of them that might be useful during troubleshooting of your XenApp environment and I will start from simple to complex

Citrix Health Assistant

The Citrix Health Assistant is a very simple tool that you can copy to your VDA and that you run directly. This is what it looks like:

And as soon as you click on Start a VDA registration check it will check the VDA software installation, the domain membership of your machine, the communication with the required ports, the VDA services, the communication with the DDC, and if your VDA is synced in time.

Use Citrix Health Assistant for a first quick check of your VDA installation.

Now you have a first impression if your VDA is misbehaving or not.

xdping.exe

This is a more advanced command line tool that you can run on the VDA and the DDC.

Again you simply copy the binaries to the target system and you´re good to go. You´ll get a lot more information about the specific machine it is run on.

Like:

• Information and status of Network Interfaces and settings
• Information on Time synchronization and time check for Kerberos authentication
• User information for the logged-on user including
  • User details
  • Authentication type used
  • Group membership
• Machine information
• Environment information
  • Computer name
  • Operating system version
  • Domain
• Information on Services Windows Communication Foundation Endpoints
• Windows Firewall configuration and ports
• Queries local event logs for errors
• Provides client bandwidth and response time

Use xdping.exe to gather additional information about your VDA or Controller.

The information you get will give you a lot more detail than Citrix Health assistant and it might show you the way to a source of failure.

Is Troubleshooting Really Safe When I Use These Tools?

Both tools run on the same system and might deliver different results. Citrix Health Assistant is created intelligently enough to check the Policies hive in the registry for the ListofDDCs value:

You can check this with ProcessMonitor:

xdping.exe shows the option as not configured and tells us that it is not possible to enumerate the list of DDCs:

You can check this with ProcessMonitor:

This might simply be because of the fact that these tools are written by Citrix support guys and they might have started with xdping.exe as a 32-bit tool and never checked it again. Although I would expect this error to be observed by a few customers.

Be carefull… Don´t trust anybody. Even not the tools you use…

Let´s check another tool to gather information about a system.

HDX Monitor 3.x

The next great tool to get an idea of your environment and to start digging into the configured options is HDX Monitor. It can be installed directly on one system via msi or you can run an online install through a browser by visiting: https://cis.citrix.com/hdx/download

After the installation, you are able to choose your local system or a remote system for analysis.

You´ll get an iTunes-style star rating of your system with regard to specific topics.

By clicking on the topics you can dive deeper into the settings. For example Network:

Or for example VDA settings:

Use HDX Monitor to get a quick overview over an environment.

Try a little bit for yourself to get used to the things you can check with this handy tool. That´s enough for the VDA. Now let´s see what you can use to troubleshoot Citrix Receiver.

Citrix Receiver Diagnostics Toolkit

I prefer the Diagnostics Toolkit over How to Enable Logging on Receiver for Windows Using Registry Entries because I had not much luck in my initial tests with Receiver 4.5. The tool is simple to use and only needs to be copied to the VDA or a client with Receiver installed.

Click on Start Tracing and the tool will immediately start collecting data. Now you should reproduce the steps that lead to the problem you have.

You can configure the tool to check for updates and you can set the location where Receiver diagnostic logs are stored as well as the Trace level and the Log size. If you want you can even configure the Event logs (System and Application) and the number of days that should be collected.

If you stop the trace you will find a folder with the Eventlogs, some CDF traces, and a PackingList.csv for further analysis.

The lazy ones can upload the files to Citrix Inside Services (CIS), where the collected files will be analyzed automatically. In my tests, the feedback I got from CIS left room for improvement. Check the log files on your own to find specific hints for problems you might have.

You can dig deeper into the log files by using tools like the Configuration Manager Trace Log Tool that you can download here. It will show you very quickly in which lines to look at by marking lines with warnings in yellow and errors in red.

If you have suggestions for the tool let the creators know. Only by providing feedback, they will have the chance to improve it.

Now let´s have a look at CDF Control that you can use to check the above-created traces or to create traces for specific Citrix modules.

CDF Control

CDFControl is an event tracing controller/consumer, geared towards capturing Citrix Diagnostic Facility (CDF) trace messages that are output from the various Citrix tracing providers.

This is an advanced analytics and debugging tool with a lot of options. If you are experienced with troubleshooting tools it will be self-explanatory after a few tries.

Before you start to trace have a look at Recommendations for Collecting the CDF Traces

After the trace, you are able to filter for Modules, Functions, or for example Errors. The next screenshot set to trace Deliver Controller Services shows us very quickly that the database server is unavailable. This should tell you to call the database guys and tell your boss you´re not guilty 😉

CDF Control is an advanced troubleshooting tool with a lot of potential to resolve errors.

If you need to create CDF control traces during startup check the following support article:

How to Collect a Citrix Diagnostic Facility (CDF) Trace at System Startup

Citrix Scout

Citrix Scout doesn´t seem complex from a user´s point of view but it will gather tons of data that might lead you to the root of a problem.

Scout is there on your Delivery Controllers. Use it!

You can add 10 machines (if you haven´t changed it) for data gathering and that makes it perfect to gather data about machines that might behave differently although configured and installed identically.

After you added your machines to the trace click on CONTINUE and start the trace.

You are now in charge to reproduce the problem and to keep the amount of data low or you can leave it running and wait for the problem to happen. This depends on the problem you are trying to resolve.

Options you can configure are:

• Check for updates
• Report Folder location
• Eventlogs to collect with the number of days
• CDF trace file settings with size and
• Maximum number of machines to collect data from (this will have a huge impact on performance)
• Proxy Server settings

You´ll get an overview of your site and of every machine you entered for the trace.

And you´ll get a folder with a lot of information…

With a lot of logs for Citrix services…

And a lot of Registry entries to look for…

Although this tool creates a lot of data it is perfect to check for differences through machines in your deployment.

Citrix Insight Services (CIS)

The Citrix Call Home feature – you remember that thing that you can enable during installation – if enabled uploads periodically information about your environment to CIS.

After logging in with your Citrix account you get an overview of your uploaded reports and information on your Site. Diagnostic reports will show you issues you have and will give you hints on how to resolve them.

Although it might not be useful to report issues like only one DDC in my site (because it is a home-lab and small) and additionally lacking the hint of only one StoreFront server or an unclustered Site database it gives you an idea where this is going in the future.

In the lower-left corner you will see additional information regarding your environment.

StoreFront Verbose Logging

We have checked the VDAs, Receivers, and Controllers and the last thing that you should check for errors is StoreFront.

The first check leads us to the Eventlog:

If you are not able to see errors you should enable verbose logging. This is done via PowerShell:

If you look at the Trace Folder it looks this way during normal operation:

After enabling verbose logging it will contain additional logs:

Don´t try to open the logs with Notepad because it will not show you data that is readable. Use DebugView from Sysinternals instead. It will look more friendly.

You should disable verbose logging when you´re done:

Set-DSTraceLevel –All –TraceLevel Off

And this is it for the first part of the Troubleshooting tools for XenApp series. In the next part, we will have a look at Microsoft tools you can use. I hope you found it useful.

Cheers,

Sinisa