Archive for October, 2011|Monthly archive page

Mini review: VKernel vScope Explorer

In Virtualization on October 27, 2011 at 20:51

If you like my content please do check out my new blog at ! 


This is my first post in what might become a series 🙂 In these post I  want to give a short review of an interesting piece of software or hardware. Today’s victim is VKernel vScope Explorer.

I found out about this tool through a post on Eric Sloof’s blog and because I got a promotional email from VKernel (apparently I left my email address there for some reason 😉 ).

What does it do?

vScope Explorer is a tool that will visualize and analyze data about your vSphere or Hyper-V environment.

So what I expect it to do is to check for configuration Best Practices and analyze both host, vCenter\SCVMM and guest metrics in order to determine possible bottlenecks and inefficiencies. And of course – pretty pictures with lots of green (or red depending on how hard they want to sell the paid complement – vOperations).

How does it do whatever it does?

vScope Explorer is a virtual appliance with a web interface. You download an OVF and deploy the appliance. It has relatively high system requirements:

  • 4 vCPUs
  • 8 GB of memory
  • 64 GB of storage space

The website says they have a instruction video – but I couldn’t find it. So I just fired it up, went through a small text based setup to configure ip,dns,ntp and http proxy settings and was presented with a login screen. (For those interested the appliance runs SUSE Linux Enterprise Server 11 SP1). Seeing this is a web based tool I decided to stop peeking around in the VM itself and opened the web interface. One note on the ip address – since the tool will connect to vCenter you should ensure the vm can connect to vCenter. And there is a user’s guide included in the download.

On the web interface (which runs on port 80) after accepting the agreement I added our vCenter server and immediately ran into an error:

This gives us a hint that the product is indeed looking into performance metrics in great detail. Since this just a test I changed the logging level and it discovered my vCenter server and I finished the setup. I logged in using the default username and password and was then presented with a nice dialog telling me it would take approximately an hour(5 minutes for 10 vm’s) for the data to be collected. I decided not to add any alarms to vCenter or to install the client plugin at this time btw.

In all honesty – it didn’t even take an hour until the collection was finished. Once it was finished the tool will showed a status screen that defaulted to the VM performance “vScope”. You can then switch to the Host performance vScope, the Capacity,the VM Efficiency or the Datastore Efficiency vScope.

Each Object (a host,a VM or a Datastore grouped together by resource cluster) will have a colour indicating its status (red, yellow or green). On mouse over or when you click the object it will give you some details on why it has a certain status. A red or yellow status can be caused by an inefficient storage allocation, high memory or cpu utilization or on a host level even a projected performance bottleneck or capacity problem with a estimated amount of time until this bottleneck or problem will be reached or occur.

I had a quick look at the status of our environment and all the statuses of the objects seemed plausible. However sometimes issues aren’t really issues – we know we have a lot of wasted space on our datastores – that’s because we need a certain amount of IOPS. There is no way to “override” these checks from the vScope interface. And as I said before – in order to properly solve the actual issues something more elaborate such as the vOperations product will be necessary.

And what do I think about it?

I think this a very nice piece of software – but its only a part of what should be a full virtualization management and monitoring solution. And I think VKernel would agree 😉

It was easy to install, easy to use and easy to interpret. And since you can connect to several vCenter servers (and SCVMM servers) you could provide an high level “single pane glass” overview that’s understandable for just about everyone.But the lack of customization features (and the abundance of red blocks caused by that limition – no one wants to many red blocks…) makes me doubt if vScope can be used as a “Manager Dashboard”

One big plus – its very portable. You download the OVF, deploy it and you have a very nice overview of the general health of your environment or your customer’s.


Exchange Activesync Issue: Device is able to authenticate, however it will not sync.

In Exchange, IOS, Server Activesync on October 26, 2011 at 11:31

We have recently taken our solution for Exchange email on IOS devices into production. We are using client certificate authentication on TMG and we use MobileIron to manage the devices and handle the certificate enrollment on the devices.

We had an issue where our root CA’s crl had expired, which as could be expected led to a situation where no one could sync their email. After tackling that problem one user was still not able to sync. That user had been part of the pilot group so we first cleaned up the certficate clutter for that user account, but he was still not able to sync.

As I mentioned before, we use client certificate authentication on TMG, but no delegation to our CAS servers. And since the authentication on TMG worked as expected for that user we decided to examine the logs on the CAS server.

In IIS logs the following error was logged:

2011-10-25 13:29:25 W3SVC1 *.*.*.* POST /Microsoft-Server-ActiveSync/default.eas User=hartj&DeviceId=Appl*********&DeviceType=iPhone&Cmd=SendMail&SaveInSent=T&Log=V121_LdapC0_LdapL0_RpcC9_RpcL15_Ers1_Pk0_Error:DeviceIsBlockedForThisUser_ 443 *******\[USERNAME] *.*.*.* Apple-iPhone3C1/901.334 403 0 0
This confirmed that the authentication worked, because otherwise the device would not be able to get to the CAS server in the first place. So we turned to Google 🙂 This led us to this excellent article on the german site EAS Authentizirung
This article described blocking access to Activesync based on IMEI or device ID. When the device id is not present in the ActivesyncAllowedDeviceIDs attribute the DeviceIsBlockedForThisUser is logged in the IIS logs and in tthe event viewer:
The article on blocking devices also showed us how to fix this issue – we manually added the DeviceID to the user using the set-casmailbox command:
Set-CasMailbox [USER] –ActiveSyncAllowedDeviceIDs [DEVICEID].
We still have no clue why this happened – either something went wrong while we were troubleshooting the certificate issues or somehow the device partnership was corrupt ( even though we had deleted that manually before with no result).

TEC 2011 Europe Frankfurt: Project Virtual Reality Check

In Citrix, The Experts Conference Europe, Virtualization on October 24, 2011 at 20:37

I was lucky enough to be able to attend the Experts Conference Europe 2011 in Frankfurt last week. In due time all the slide decks and transcripts will hit the web so I refrain from delayed live blogging about all of the sessions. However there was one session (or actually two, the session was split into two parts – but considering the content it could have easily spanned three sessions!) of which both the topic and the presentation really interested me.

The session in question was Project Virtual Reality Check and it’s speaker was Jeroen van der Kamp, CTO for Login Consultants. Project Virtual Reality Check is a joint venture between two Dutch companies, PQR and Login Consultants. Its objective is to find the answers to several questions concerning the performance of virtualized Presentation Virtualization and Desktop Virtualization environments using different hypervisors, hardware and PV/DV technologies.

In order to find those answers they have developed a standard set of benchmarks which they use to find out what the limits are in terms of session (in DV) or guest (in DV) density. All major players in both the PV or Terminal Services and the DV/VDI are being tested – so its Hyper-V v. vSpere v. Xen and XenDesktop v. Vmware View v. vWorkspace etc.

Now the first reason why I attended this session was that I’m currently looking into several technologies that deal with remote offices and remoting. Traditionally Presentation virtualization or VPN have been the two obvious choices to offer users a way to work from home or from a small office. With the advent of VDI, or the rising demands of power users – I’m not getting into the discussion which came first – and the introduction of platforms such as Citrix XenApp/Desktop and vWorkspace where you can have the best of both worlds those choices aren’t that obvious anymore.

In a world of desktop or client connectivity in general you aren’t working with IOPS, CPU ready times or consolidation ratio’s. You are working with people (or as “us” IT people tend to call them “users”). People with subjective preferences, perception and presuppositions.  The first you don’t want to fix, the second you can’t fix and the last will take time and effort and results. So if you are designing such an infrastructure you want to know exactly if, how and why certain design decisions will influence performance – because you will always be juggling directly with client demands (Media content, Choice and Personalization) and limiting factors (Bandwidth, Latency, Cost).

And that is why I think that having independent, falsifiable and full system benchmarks are so important. And that’s exactly what VRC provides – all the specs and “payloads” are known variables and so are the benchmarking tools. Of course, as their own disclaimer states: “All Project VRC test are preformed in a pre-configured lab environment” – so these are not necessarily real life results. But the results will tell you which hypervisor will do what when pushed to the extreme limit. And its just that limit, even though when all prefer to call it optimal utilization, that was one of the main reason to start virtualizing workloads in the first place.

Of course all vendors also supply us with loads of performance information, comparisons and analysis. And some even do a good job. But most of the time the technical sales talk is even worse then the “normal” sales talk because they try to claim legitimacy through statistics. As Brian Madden pointed out during the Virtualization keynote – nothing is easier then lying with numbers.

A side effect of pushing a system to the limit is that you are able to directly identify, test and adjust Best Practices for each platform. So instead of compiling best practices based on problems and solutions in the field you get a great overview of the various best practices and their actual effect on the ability to host more guests or sessions on a piece of hardware.

Jeroen van der Kamp did a terrific job talking us through the results of each of the project phases and their results – one of things that interested me was the fact that in some cases Hyper-V had the upper hand when compared with vSphere and Xen and also the preliminary results of the Antivirus tests which showed that in a VDI environment offloading actually hurt the performance instead of improving it. Quite the contrary of what was claimed in a Tolly report sponsored by Trend Micro…

Monitoring Citrix Netscaler Load Balancers with SCOM 2007 R2 Part III.

In Citrix, Netscaler, Operations Manager on October 21, 2011 at 20:28

This is part three of my series on monitoring Citrix Netscalers with SCOM 2007 R2 ( Part I and Part II).

Now, does it work? And how? As said in my previous post the way the information is presented is a little bit different then with other MP’s. For each SNMP trap sent the pack will raise an alert – and the alert will tell nothing more then the fact that the SNMP trap has been sent 😉 If you make a config change you’ll get an alert that it has been changed..but not what has been changed. That information is shown elsewhere.

To give an example, in the picture below you can see two alerts:

I changed the configuration and saved the running config. This is the information as shown in the config change alert:

To see what has been changed we need to head over to the Events node, here two events are shown:

If you look at the details of the event we can see that a SNMP community reference was added to the Netscalers with “public” as a community string:

Then there is the Health Roll-up of the entity. The health monitored health categories are Availability and Performance. Performance is  based on some SNMP GET based performance metrics of both the appliance and the vservers. Availability is based on the state of the appliance alone and so if a vserver is down the Netscaler Device entity will still be healthy.

And last but not least – the Netscalers themselves also allow you to tune what SCOM will report because you can enable/disable and configure its SNMP traps. You can do this from the GUI by opening System\SNMP\Alarms.

Depending on the type of alarm you can define the alarm and normal thresholds, the time interval and the alarm’s severity. Whatever you configure here will direct influence the way SCOM will report about these events. Of course its also possible to override the rules and or alerts in SCOM but personally I prefer to do this at the source.

I hope this guide will help others to get this MP running in their environment and possibly even convince others to choose this method of monitoring Netscalers.

Monitoring Citrix Netscaler Load Balancers with SCOM 2007 R2 Part II.

In Citrix, Netscaler, Operations Manager on October 20, 2011 at 22:07

This is part two of my series on monitoring Citrix Netscalers with SCOM 2007 R2 ( Part I ).

In the previous post I discussed why we decided to use SCOM to monitor the Netscalers, the MP’s installation and the Netscaler’s configuration. In this post I will discuss discovering the Netscalers in SCOM and the general usage of the MP.


The Netscalers need to be discovered as generic network devices. After they’ve been discovered a scheduled discovery will discover them as Netscaler devices based on their SNMP OID. After that another discovery runs to identify the installed features and modes.

  • Open the SCOM console, choose Administration and start the Discovery wizard.
  • Choose Network Devices
  • Specify an ip range that includes both your NSIP’s.
  • Select SNMP v2, specify your community string and Management Server

  • Now start the discovery, if you’ve configured the Netscaler correctly the wizard will detect two network devices. You will be able to see them both listed under Administration/Network Devices

The discoveries that are ran automatically against all network devices run every 21600 seconds. So you can either wait until it start or override the discovery. The discovery simply discovers all SNMP devices with a certain OID (if included a screenshot of the xml as a reference):

After the Netscalers have been identified as Netscaler Devices they will show up under Monitoring/Citrix Netscaler Devices/All Devices and the following discoveries which are ttargeted at the Citrix NetScaler Device class will start to discover additional classes and some properties to the Citrix Netscaler Device class:

  • Citrix Netscaler Feature Discovery – this will detect all features and their state ( Load Balancing, Access Gateway etc)
  • Citrix Netscaler Mode Discovery – this will detect all modes and their state (L2 versus L3 etc)
  • Citrix Netscaler Device Discovery – this will add the Node State ( Primary/Secondary), Host Name, HA Peer IP and hardware version

This is the point where we ran into some issues. Discovering the Citrix Netscaler Device class went fine but the other classes weren’t discovered at all and the extra attributes weren’t populated. Looking at the evenlogs on the management server I discovered an event with the following error message:

Error Message: 91\2600\Citrix.NetScaler.VirtualServerState.vbs(44, 9) Microsoft VBScript runtime error: ActiveX component can’t create object: ‘SScripting.SNMPManager’

This leads me to the Citrix Knowledge Center article I mentioned earlier ( Case Study: When installing…Error Message ). I downloaded the MP from the Citrix Community page and installed that over the version I had downloaded from MyCitrix and after a reboot the discoveries did identify the modes, features and attributes.

Configuring the MP

When we look at the Monitoring view – the Netscaler MP has 4 main nodes:

  • The root node – this contains an alerts view, a config changes view and events view and a Network Diagram.
  • The Device state node – this shows has two views: Active Devices which lists all the primary nodes and All Devices which shows all nodes.
  • The License & Modes node – this give a state view of all the features and modes as they are configured on each appliance
  • The Performance node – this has a rather large number of performance views

Alerts seems pretty self-explanatory however it is important to note that the alerts contain little information. You’ll know  a rule has triggered an alert but not why. Same goes for the Config Changes. Both will tell you there has been a alert or a config change, but the actual data is in the events view. Here all events (be it triggered alerts or snmp traps or config saves, changes, reboot etc) are logged with all the data provided by the SNMP GET or trap.

The network Diagram was a bit of a disappointment, I would have hoped to see the Vservers and the services in there as well.

License and mode views aren’t to pretty but they do the job, Licenses:

Unfortunatly you’ll need to select a row to see to which appliance it belongs when looking at licenses. The modes view is much better:

The performance views are grouped into several categories, ACL, IP, SSL etc. None of the rules and monitors are enabled by default. Which brings me to a point of criticism – why are all rules and monitor disabled by default and then overidden with an override that’s stored in main Citirx Netscaler MP? Again something that goes against Best Pratices.

Actually most performance counters aren’t active (or have an override by default)when you install the pack – you’ll need to override them one-by-one to be able to get that data into SCOM. This is where a tool such as OverrideExplorer ( I used v3.3. ) can prove to be invaluable, since for each category there are several snmp get rules and in order to fully populate the performance views you’ll need to override all of them.

One clue – when you open the authoring pane in SCOM and limit to the scope to include only the Netscalers you can find the rules needed to each catergory by looking at their name. They will start with the name of the performance view in the monitoring pane and start with a capital. In the picture below you can see all the TCP rules, and if you look at the Override Management Pack you can see I used a custom override pack which means they weren’t enabled by default:

Using this information you can override the performance rules in bulk using Override Explorer.

Then you are ready to go. In the next part I will show the MP in action and show how you can configure and enable/disable the SNMP traps sent by the Netscalers.

Monitoring Citrix Netscaler Load Balancers with SCOM 2007 R2 Part I.

In Citrix, Netscaler, Operations Manager on October 19, 2011 at 19:19


(Part II , Part III)

We recently introduced two Citrix Netscaler clusters into our environment. The first cluster was already running as a Citrix Access Gateway cluster (as an upgrade from our Secure Gateway – needed to support Citrix receiver on IOS devices), we purchased a load balancing license for that cluster and are using it to load balance servers in our DMZ. The other cluster is used to load balance servers in our internal network.

We mainly use the load balancers to create what I call “controlled redundancy”, but we do use it for several critical applications, such as the before mentioned XenApp environment. And one of the key elements in achieving this state of controlled redundancy in my humble opinions is being able to monitor these clusters.

Citrix offers an excellent application to monitor and administer their line of networking products called “Command Center”. But our central monitoring solution is Microsoft SCOM 2007. Of course we could have decided to use both products side-by-side or try to engineer some connector between Command Center and SCOM. But since the number of management task we have to perform on our Netscalers is very small – and the fact that Citrix has a SCOM MP for the Netscalers – we are now managing the two cluster using the GUI and SSH for the time being and installed the SCOM MP.

In this series of posts I am going to show how we installed, configured and tuned the management pack. I’m also going to cover the configuration of the Netscalers and the usage of the Netscaler pack – mainly because its structure is a little different then most standard Microsoft MP’s.

We use vSphere as our virtualization platform so I have no experience with the PRO MP’s that are provided to use SCVMM PRO TIPS  – so all I can say about that is that its unfortunate that there is no comparable feature for vSphere.


The SCOM pack can be downloaded from myctrix if you have the proper licenses associated with your accounts. However – the same pack can also be obtained from the following Citrix Community blog post 🙂

I found that link in this Citrix KB article: – which discusses an issue with this pack and a x64 OS. We actually ran into this issue but more about that later.

Btw both downloads will get you the 2.0 version of the MP – there is a 1.0 version out there for older firmware builds. We have both a classic 9.2 build and a ncore 9.2 build in our environment and we use the 2.0 pack for both.

The installation is pretty straightforward. We do all SNMP based monitoring from a separate management server so it made sense for us to install the MP there. The management pack can do SNMP gets and receive SNMP traps so you’ll have to enable the built-in SNMP service on the management server.

You run the installer and then import the MP into SCOM.  Now its time to configure the Netscalers!

Netscaler Configuration

In order to configure the Netscalers to be monitored by SCOM there are a couple of things you’lll need to configure, but one of things that really bugged me was the fact that in order to properly monitor the cluster I needed to be able to add both nodes to SCOM – which basically means that you have to create your NSIPs in a routed part of your network, which is against Citrix best practices ( or somehow multi-home your management server of course).

So besides configuring your NSIP so that it’s reachable and has SNMP enabled everything you need to configure is in the System\SNMP node of the Netscaler GUI. I’m not familiar with the CLI yet however your just as easily configure it there I guess.

  • First there is the SNMP community:

To monitor the Netscalers only a GET permission is needed, choose Add and input your SNMP string en choose the permission

  • Then you’ll to add the SCOM server(s) or their IP range as SNMP Manager:

Choose Management Host to use a single IP, network for multiple. In our case we have a dedicated VLAN for our monitoring and management servers.

  • Next up are SNMP traps:

This is that part where I ran into some issues – it took me some time to figure out I needed to use Specific as the type instead of Generic. You also need to define the Trap destination and port. Before,I mentioned you needed to use the NSIP to monitor the Netscalers, but that’s only for the SNMP GETS because you are able to set a cluster wide SNIP or MIP as the source address. Minimum severity and Community name are obvious however don’t be fooled by the parenthesis in the Community Name field – you actually have to enter your own string without parenthesis!

That’s most of the configuration on the Netscalers – in the next two parts I’ll discuss discovering the Netscalers, how to tune and configure the monitoring process on both SCOM and the Netscaler and I’ll try to show a little bit about the structure and the usage of the MP – especially because its a little different then your ordinary Microsoft MP.

(Part II , Part III)