Archive for the ‘Citrix’ Category

Netscaler/Citrix Access Gateway and Active Directory nested groups

In Active Directory, Citrix, Netscaler on May 11, 2012 at 21:21

We recently adapted RBAC based on MS Active Directory to manage our infrastructure. We already used AD groups to authenticate and authorize administrators on our Netscaler appliances,  however admins where direct members of the groups defined in the Netscalers. ( Have a look over here to see configure this )

Our new RBAC system uses nested groups – an admin is a member of a role group which is a member of a group authorizing access to a resource. Not every non-MS system is able to “understand” nested groups ( such as Cisco Ironport anti-spam appliances) so you are forced to use some sort of iterative\recursive query to make it work but luckily the Netscalers have a feature called “nested group extraction”.

You can enable nested group extraction when you define a Authentication server. After you choose LDAP as the authentication server type the option is somewhat hidden in the bottom of the dialogue window – but if you flip it open and enter:

  • The nesting level ( default is 2)
  • The group name identifier ( simply the attribute defining the unique name for the group object) which in most situations would be the samAccountName attribute
  • The group search attribute: memberOf
  • And the sub search attribute – here the documentation suggest using the common name – just as in the general server configuration

you will be able to use nested groups to authorize your administrators to perform management tasks on the Netscalers.

You could use an existing LDAP server/policy pair to achieve this – but I would strongly advise to create a separate server/policy pair. The main reason is that when you enable nested group extraction for an authentication server  all users authenticating through that policy/server pair seem to be checked for nested group memberships – even if you don’t use group membership as a factor to authorize your users to access resources…

We found out about that one the hard way – after enabling nested group extraction on our default LDAP policy/server pair certain users where unable to log onto our Citrix environment. This was caused by the fact that they were a member of a group that had “illegal characters” in their common name ( a forward slash “/”) – and with nested group extraction enabled they got an access denied message…

We solved this by using a different policy/server pair for admin authentication/authorization.

SCOM Netscaler pack false positive: No HA hearbeats SNMP trap

In Citrix, Netscaler, Operations Manager, System Center, Uncategorized on December 9, 2011 at 21:55

As I mentioned some time ago we use SCOM to monitor our Netscaler Load Balancers. We ran into an issue where an alert would be raised based on a SNMP trap sent by the Netscaler. Some background info:

– We have an etherchannel (Nortel Avaya SMLT if people are interested) like setup where we use 2 ethernet interfaces on the Netscalers connected to our core switches and we have all our VLAN’s trunked on those ports (including the VLAN where the NSIP’s reside).

– The other ports are not connected – but we had two interfaces enabled so  we could use those to connect to the appliances if the etherchannel config got screwded up

– HA monitoring is only enabled on the channel not on any of the individual ethernet interfaces.

What happened was that traps were sent out saying that both nodes missed HA heartbeats – but when we logged into the Netscaler GUI the HA status was fine. When taking a closer look at the snmp trap data it appeared that no HA heartbeats were seen on the two ethernet ports that were enabled but not connected – even though HA monitoring was disabled on these ports.

We couldn’t really override this because the SNMP trap could only be enabled or disabled for all interfaces, so we disabled the interfaces. The only drawback is that we’ll need to use the console port if we can’t reach the Netscalers through the “etherchanneled” interfaces.

Netscaler Load Balancing: Monitor TMG Webproxy with User Authentication

In Citrix, Netscaler, TMG 2010 on November 22, 2011 at 11:51

We use a Microsoft Forefront Threat Management Gateway 2010 server array as forward proxy servers. Instead of using a autoconfig script, WPAD or the firewall client we use a load balanced VIP on our Netscalers to direct client towards the proxy. The setup is quite simple – a client connects to the VIP on port 8080 and the Netscalers sends the request over to TMG. Because we want the second proxy server to be passive we use a backup VIP instead of two services behind the first VIP.

Now one of the advantages of a hardware load balancer in this scenario over a software based load balancing solution (such as vanilla or TMG integrated MS Network Load Balancing) is that a Netscaler can be configured in such a way that its application and even application performance aware if you want. We were only looking for application awareness – especially because we ran into situations where TMG said it was happy, SCOM said it was happy and there was more then enough cpu, memory, network resources and bandwith to go around – but clients weren’t able to get a single page from the Internet. But TMG has such a special place in my heart that I’ll devote an entire post to it later this week.

Anyway – Netscaler to the rescue.

This is what I wanted to do: build a monitor that retrieves a website through the webproxy server. That’s been done before: How to Configure an HTTP-ECV Health Monitor for Internet Proxy Servers . But that was for an unauthenticated proxy server.It did give some pointers on how to configure it with authentication. And luckily we allow Basic Authentication (using ntlm should be possible I guess using the right perl script) so all seemed well.

First I’d like to point out that I’ve moved from using the GUI to using the CLI to configure things such as new vservers and monitors. I’ve been in a situation twice where a change in the GUI didn’t come through properly – even after saving and refreshing all.

Secondly – the method in the article mentioned above doesn’t work :(.

I tweaked the parameters and headers over and over but either TMG didn’t accept the request or the Netscaler couldn’t find the pattern in the response. I did some tracing with Network Monitor but even when TMG sent back a proper 200 status code the Netscaler said the service was down. But at some point I found another Knowledge center article: How to Configure a NetScaler Monitor to Authenticate with a User Name and Password.

I quote: “Do not use an HTTP-ECV monitor when sending additional headers such as authentication, host, and so on.”.

Wow silly me – how did I ever get that idea…??

Following the article, what I did was this:

add lb monitor Proxy_Monitor TCP-ECV -send “GET HTTP/1.1\r\nProxy-Authorization: Basic Veryintimidatingbase64stringletsnotusepriviligedaccount\r\\r\nCache-control: no-cache\r\n\r\n” -recv 302 -LRTM ENABLED -interval 30


  • The base64 string can be obtained by using Powershell (or from the netscaler CLI – see the article):

function ConvertTo-Base64($string) { $bytes = [System.Text.Encoding]::UTF8.GetBytes($string); $encoded = [System.Convert]::ToBase64String($bytes); return $encoded; } Source

  • You need to use the Proxy Authorization instead of the Authorization header
  • You can set the realm using a header or include it in the username (domain\username:password and then encode with base64)
  • TMG really wants you to give it the full GET, so include the whole url and it wants a host header with the hostname of the destination url
  • We are testing getting a page from Internet – not from our cache so I use a cache-control header
  • The receive string here is not 200 but 302 because that’s the redirect we get when we request (or for that matter).
  • To prevent a failover when a single website is offline for some reason,I’ve made two monitors and bound them to each service, each going to another url and using another user account so that we can prevent an account lockout ruining our day as well. Then by setting the -monThreshold parameter on the service to 1 and giving each monitor a weight of 1 I can ensure that the service is up if one of the monitors is successful.

I hope someone will find this information useful – one small disclaimer: Basic Authentication is not encrypted – just encoded – and therefore basically clear text.

TEC 2011 Europe Frankfurt: Project Virtual Reality Check

In Citrix, The Experts Conference Europe, Virtualization on October 24, 2011 at 20:37

I was lucky enough to be able to attend the Experts Conference Europe 2011 in Frankfurt last week. In due time all the slide decks and transcripts will hit the web so I refrain from delayed live blogging about all of the sessions. However there was one session (or actually two, the session was split into two parts – but considering the content it could have easily spanned three sessions!) of which both the topic and the presentation really interested me.

The session in question was Project Virtual Reality Check and it’s speaker was Jeroen van der Kamp, CTO for Login Consultants. Project Virtual Reality Check is a joint venture between two Dutch companies, PQR and Login Consultants. Its objective is to find the answers to several questions concerning the performance of virtualized Presentation Virtualization and Desktop Virtualization environments using different hypervisors, hardware and PV/DV technologies.

In order to find those answers they have developed a standard set of benchmarks which they use to find out what the limits are in terms of session (in DV) or guest (in DV) density. All major players in both the PV or Terminal Services and the DV/VDI are being tested – so its Hyper-V v. vSpere v. Xen and XenDesktop v. Vmware View v. vWorkspace etc.

Now the first reason why I attended this session was that I’m currently looking into several technologies that deal with remote offices and remoting. Traditionally Presentation virtualization or VPN have been the two obvious choices to offer users a way to work from home or from a small office. With the advent of VDI, or the rising demands of power users – I’m not getting into the discussion which came first – and the introduction of platforms such as Citrix XenApp/Desktop and vWorkspace where you can have the best of both worlds those choices aren’t that obvious anymore.

In a world of desktop or client connectivity in general you aren’t working with IOPS, CPU ready times or consolidation ratio’s. You are working with people (or as “us” IT people tend to call them “users”). People with subjective preferences, perception and presuppositions.  The first you don’t want to fix, the second you can’t fix and the last will take time and effort and results. So if you are designing such an infrastructure you want to know exactly if, how and why certain design decisions will influence performance – because you will always be juggling directly with client demands (Media content, Choice and Personalization) and limiting factors (Bandwidth, Latency, Cost).

And that is why I think that having independent, falsifiable and full system benchmarks are so important. And that’s exactly what VRC provides – all the specs and “payloads” are known variables and so are the benchmarking tools. Of course, as their own disclaimer states: “All Project VRC test are preformed in a pre-configured lab environment” – so these are not necessarily real life results. But the results will tell you which hypervisor will do what when pushed to the extreme limit. And its just that limit, even though when all prefer to call it optimal utilization, that was one of the main reason to start virtualizing workloads in the first place.

Of course all vendors also supply us with loads of performance information, comparisons and analysis. And some even do a good job. But most of the time the technical sales talk is even worse then the “normal” sales talk because they try to claim legitimacy through statistics. As Brian Madden pointed out during the Virtualization keynote – nothing is easier then lying with numbers.

A side effect of pushing a system to the limit is that you are able to directly identify, test and adjust Best Practices for each platform. So instead of compiling best practices based on problems and solutions in the field you get a great overview of the various best practices and their actual effect on the ability to host more guests or sessions on a piece of hardware.

Jeroen van der Kamp did a terrific job talking us through the results of each of the project phases and their results – one of things that interested me was the fact that in some cases Hyper-V had the upper hand when compared with vSphere and Xen and also the preliminary results of the Antivirus tests which showed that in a VDI environment offloading actually hurt the performance instead of improving it. Quite the contrary of what was claimed in a Tolly report sponsored by Trend Micro…

Monitoring Citrix Netscaler Load Balancers with SCOM 2007 R2 Part III.

In Citrix, Netscaler, Operations Manager on October 21, 2011 at 20:28

This is part three of my series on monitoring Citrix Netscalers with SCOM 2007 R2 ( Part I and Part II).

Now, does it work? And how? As said in my previous post the way the information is presented is a little bit different then with other MP’s. For each SNMP trap sent the pack will raise an alert – and the alert will tell nothing more then the fact that the SNMP trap has been sent 😉 If you make a config change you’ll get an alert that it has been changed..but not what has been changed. That information is shown elsewhere.

To give an example, in the picture below you can see two alerts:

I changed the configuration and saved the running config. This is the information as shown in the config change alert:

To see what has been changed we need to head over to the Events node, here two events are shown:

If you look at the details of the event we can see that a SNMP community reference was added to the Netscalers with “public” as a community string:

Then there is the Health Roll-up of the entity. The health monitored health categories are Availability and Performance. Performance is  based on some SNMP GET based performance metrics of both the appliance and the vservers. Availability is based on the state of the appliance alone and so if a vserver is down the Netscaler Device entity will still be healthy.

And last but not least – the Netscalers themselves also allow you to tune what SCOM will report because you can enable/disable and configure its SNMP traps. You can do this from the GUI by opening System\SNMP\Alarms.

Depending on the type of alarm you can define the alarm and normal thresholds, the time interval and the alarm’s severity. Whatever you configure here will direct influence the way SCOM will report about these events. Of course its also possible to override the rules and or alerts in SCOM but personally I prefer to do this at the source.

I hope this guide will help others to get this MP running in their environment and possibly even convince others to choose this method of monitoring Netscalers.

Monitoring Citrix Netscaler Load Balancers with SCOM 2007 R2 Part II.

In Citrix, Netscaler, Operations Manager on October 20, 2011 at 22:07

This is part two of my series on monitoring Citrix Netscalers with SCOM 2007 R2 ( Part I ).

In the previous post I discussed why we decided to use SCOM to monitor the Netscalers, the MP’s installation and the Netscaler’s configuration. In this post I will discuss discovering the Netscalers in SCOM and the general usage of the MP.


The Netscalers need to be discovered as generic network devices. After they’ve been discovered a scheduled discovery will discover them as Netscaler devices based on their SNMP OID. After that another discovery runs to identify the installed features and modes.

  • Open the SCOM console, choose Administration and start the Discovery wizard.
  • Choose Network Devices
  • Specify an ip range that includes both your NSIP’s.
  • Select SNMP v2, specify your community string and Management Server

  • Now start the discovery, if you’ve configured the Netscaler correctly the wizard will detect two network devices. You will be able to see them both listed under Administration/Network Devices

The discoveries that are ran automatically against all network devices run every 21600 seconds. So you can either wait until it start or override the discovery. The discovery simply discovers all SNMP devices with a certain OID (if included a screenshot of the xml as a reference):

After the Netscalers have been identified as Netscaler Devices they will show up under Monitoring/Citrix Netscaler Devices/All Devices and the following discoveries which are ttargeted at the Citrix NetScaler Device class will start to discover additional classes and some properties to the Citrix Netscaler Device class:

  • Citrix Netscaler Feature Discovery – this will detect all features and their state ( Load Balancing, Access Gateway etc)
  • Citrix Netscaler Mode Discovery – this will detect all modes and their state (L2 versus L3 etc)
  • Citrix Netscaler Device Discovery – this will add the Node State ( Primary/Secondary), Host Name, HA Peer IP and hardware version

This is the point where we ran into some issues. Discovering the Citrix Netscaler Device class went fine but the other classes weren’t discovered at all and the extra attributes weren’t populated. Looking at the evenlogs on the management server I discovered an event with the following error message:

Error Message: 91\2600\Citrix.NetScaler.VirtualServerState.vbs(44, 9) Microsoft VBScript runtime error: ActiveX component can’t create object: ‘SScripting.SNMPManager’

This leads me to the Citrix Knowledge Center article I mentioned earlier ( Case Study: When installing…Error Message ). I downloaded the MP from the Citrix Community page and installed that over the version I had downloaded from MyCitrix and after a reboot the discoveries did identify the modes, features and attributes.

Configuring the MP

When we look at the Monitoring view – the Netscaler MP has 4 main nodes:

  • The root node – this contains an alerts view, a config changes view and events view and a Network Diagram.
  • The Device state node – this shows has two views: Active Devices which lists all the primary nodes and All Devices which shows all nodes.
  • The License & Modes node – this give a state view of all the features and modes as they are configured on each appliance
  • The Performance node – this has a rather large number of performance views

Alerts seems pretty self-explanatory however it is important to note that the alerts contain little information. You’ll know  a rule has triggered an alert but not why. Same goes for the Config Changes. Both will tell you there has been a alert or a config change, but the actual data is in the events view. Here all events (be it triggered alerts or snmp traps or config saves, changes, reboot etc) are logged with all the data provided by the SNMP GET or trap.

The network Diagram was a bit of a disappointment, I would have hoped to see the Vservers and the services in there as well.

License and mode views aren’t to pretty but they do the job, Licenses:

Unfortunatly you’ll need to select a row to see to which appliance it belongs when looking at licenses. The modes view is much better:

The performance views are grouped into several categories, ACL, IP, SSL etc. None of the rules and monitors are enabled by default. Which brings me to a point of criticism – why are all rules and monitor disabled by default and then overidden with an override that’s stored in main Citirx Netscaler MP? Again something that goes against Best Pratices.

Actually most performance counters aren’t active (or have an override by default)when you install the pack – you’ll need to override them one-by-one to be able to get that data into SCOM. This is where a tool such as OverrideExplorer ( I used v3.3. ) can prove to be invaluable, since for each category there are several snmp get rules and in order to fully populate the performance views you’ll need to override all of them.

One clue – when you open the authoring pane in SCOM and limit to the scope to include only the Netscalers you can find the rules needed to each catergory by looking at their name. They will start with the name of the performance view in the monitoring pane and start with a capital. In the picture below you can see all the TCP rules, and if you look at the Override Management Pack you can see I used a custom override pack which means they weren’t enabled by default:

Using this information you can override the performance rules in bulk using Override Explorer.

Then you are ready to go. In the next part I will show the MP in action and show how you can configure and enable/disable the SNMP traps sent by the Netscalers.

Monitoring Citrix Netscaler Load Balancers with SCOM 2007 R2 Part I.

In Citrix, Netscaler, Operations Manager on October 19, 2011 at 19:19


(Part II , Part III)

We recently introduced two Citrix Netscaler clusters into our environment. The first cluster was already running as a Citrix Access Gateway cluster (as an upgrade from our Secure Gateway – needed to support Citrix receiver on IOS devices), we purchased a load balancing license for that cluster and are using it to load balance servers in our DMZ. The other cluster is used to load balance servers in our internal network.

We mainly use the load balancers to create what I call “controlled redundancy”, but we do use it for several critical applications, such as the before mentioned XenApp environment. And one of the key elements in achieving this state of controlled redundancy in my humble opinions is being able to monitor these clusters.

Citrix offers an excellent application to monitor and administer their line of networking products called “Command Center”. But our central monitoring solution is Microsoft SCOM 2007. Of course we could have decided to use both products side-by-side or try to engineer some connector between Command Center and SCOM. But since the number of management task we have to perform on our Netscalers is very small – and the fact that Citrix has a SCOM MP for the Netscalers – we are now managing the two cluster using the GUI and SSH for the time being and installed the SCOM MP.

In this series of posts I am going to show how we installed, configured and tuned the management pack. I’m also going to cover the configuration of the Netscalers and the usage of the Netscaler pack – mainly because its structure is a little different then most standard Microsoft MP’s.

We use vSphere as our virtualization platform so I have no experience with the PRO MP’s that are provided to use SCVMM PRO TIPS  – so all I can say about that is that its unfortunate that there is no comparable feature for vSphere.


The SCOM pack can be downloaded from myctrix if you have the proper licenses associated with your accounts. However – the same pack can also be obtained from the following Citrix Community blog post 🙂

I found that link in this Citrix KB article: – which discusses an issue with this pack and a x64 OS. We actually ran into this issue but more about that later.

Btw both downloads will get you the 2.0 version of the MP – there is a 1.0 version out there for older firmware builds. We have both a classic 9.2 build and a ncore 9.2 build in our environment and we use the 2.0 pack for both.

The installation is pretty straightforward. We do all SNMP based monitoring from a separate management server so it made sense for us to install the MP there. The management pack can do SNMP gets and receive SNMP traps so you’ll have to enable the built-in SNMP service on the management server.

You run the installer and then import the MP into SCOM.  Now its time to configure the Netscalers!

Netscaler Configuration

In order to configure the Netscalers to be monitored by SCOM there are a couple of things you’lll need to configure, but one of things that really bugged me was the fact that in order to properly monitor the cluster I needed to be able to add both nodes to SCOM – which basically means that you have to create your NSIPs in a routed part of your network, which is against Citrix best practices ( or somehow multi-home your management server of course).

So besides configuring your NSIP so that it’s reachable and has SNMP enabled everything you need to configure is in the System\SNMP node of the Netscaler GUI. I’m not familiar with the CLI yet however your just as easily configure it there I guess.

  • First there is the SNMP community:

To monitor the Netscalers only a GET permission is needed, choose Add and input your SNMP string en choose the permission

  • Then you’ll to add the SCOM server(s) or their IP range as SNMP Manager:

Choose Management Host to use a single IP, network for multiple. In our case we have a dedicated VLAN for our monitoring and management servers.

  • Next up are SNMP traps:

This is that part where I ran into some issues – it took me some time to figure out I needed to use Specific as the type instead of Generic. You also need to define the Trap destination and port. Before,I mentioned you needed to use the NSIP to monitor the Netscalers, but that’s only for the SNMP GETS because you are able to set a cluster wide SNIP or MIP as the source address. Minimum severity and Community name are obvious however don’t be fooled by the parenthesis in the Community Name field – you actually have to enter your own string without parenthesis!

That’s most of the configuration on the Netscalers – in the next two parts I’ll discuss discovering the Netscalers, how to tune and configure the monitoring process on both SCOM and the Netscaler and I’ll try to show a little bit about the structure and the usage of the MP – especially because its a little different then your ordinary Microsoft MP.

(Part II , Part III)