jmbrinkman

Posts Tagged ‘SCOM’

SCOM Netscaler pack false positive: No HA hearbeats SNMP trap

In Citrix, Netscaler, Operations Manager, System Center, Uncategorized on December 9, 2011 at 21:55

As I mentioned some time ago we use SCOM to monitor our Netscaler Load Balancers. We ran into an issue where an alert would be raised based on a SNMP trap sent by the Netscaler. Some background info:

– We have an etherchannel (Nortel Avaya SMLT if people are interested) like setup where we use 2 ethernet interfaces on the Netscalers connected to our core switches and we have all our VLAN’s trunked on those ports (including the VLAN where the NSIP’s reside).

– The other ports are not connected – but we had two interfaces enabled so  we could use those to connect to the appliances if the etherchannel config got screwded up

– HA monitoring is only enabled on the channel not on any of the individual ethernet interfaces.

What happened was that traps were sent out saying that both nodes missed HA heartbeats – but when we logged into the Netscaler GUI the HA status was fine. When taking a closer look at the snmp trap data it appeared that no HA heartbeats were seen on the two ethernet ports that were enabled but not connected – even though HA monitoring was disabled on these ports.

We couldn’t really override this because the SNMP trap could only be enabled or disabled for all interfaces, so we disabled the interfaces. The only drawback is that we’ll need to use the console port if we can’t reach the Netscalers through the “etherchanneled” interfaces.

SCOM 2007 R2: Monitoring vSphere Shoot Out

In Operations Manager, Virtualization on November 1, 2011 at 20:52

Update: I’ve done a mini-review on SCVMM/SCOM 2012 and vSphere monitoring

We are a Microsoft shop. And a Vmware shop. We use SCOM to monitor everything and vSphere to host all our servers. So you can imagine how crucially important it is for us to properly monitor vSphere. With SCOM. Of course Virtual Center does a great job in giving us basic information about our hypervisor environment and the state of our virtual machines. But without the information about our applications SCOM provides and no real way to relate the two sets of data we really needed a way to get that information into one system.

Of course, there are other monitoring solutions, both for vSphere and for Microsoft applications. But we want to take advantage of our investment in SCOM and we firmly believe that SCOM is the best options to monitor  a 99% Microsoft infrastructure.

We were not the first facing this challenge. Because a challenge it was. We did our best to look at as many options as we could and in the end made a choice based on both functionality and price.

In this post I want to give a short overview of the solutions we looked at and give my personal opinion on each of them.

The contenders

In no particular order:

We also expressed some interest in a management pack created by Bridgeways, but they were very slow to respond to our request for a evaluation and once we got a response the amount of information we had to provide in order to evaluate the pack was so huge we decided it was not worth the effort.

Small disclaimer: we really did our best to give each solution a fair shot, however it could be possible that additional configuration or tweaking would increase the performance or the quality of the data. On the other hand we didn’t take into account how hard to was to actually get the solutions working – because the installation process (especially under Windows 2008) wasn’t always easy though nothing we couldn’t handle.

Round 1: What do they monitor – and how?

All of the solutions work through vCenter, with the exception of QMX which is able to monitor vSphere hosts directly through SNMP and SSH. I guess you could configure Jalasoft or even SCOM itself as a generic SNMP device or build your own sets of monitors and rules but in general you will still need vCenter as a middle man to monitor your hosts.

None of them consists of just a Management Pack – they all need a service running on either a SCOM server or a separate server with access to SCOM. Jalasoft and QMX are frameworks – so its possible to monitor other devices as well which makes it easier to digest that you need to add another component to your monitoring infrastructure – SCVMM could also be used to monitor Hyper-V or to manage vSphere and Hyper-V.

Jalasoft’s Smart MP monitors just vCenter. Hosts are discovered as part of the vCenter server but aren’t represented as separate entities. SCVMM monitors both vCenter, hosts and virtual machines however it will not give you any vSphere specific data such as CPU ready times, Memory swapping etc. During our tests a vSphere host failed and we had fixed the problem before SCVMM alerted us. QMX gives you an afwul lot of options – it can monitor vmware logs, syslogs on the esx servers, esxtop data (my personal favourite) and also give you the possibility to create custom filters on log files to trigger an alert if an entry matching the filter is logged. It also is aware of vCenter alerts en events but I didn’t find any monitor or alerts relating to DRS or HA.

Veeam monitors just about everything that makes vSphere vSphere. Also a lot of work has been put in the knowledge in the alerts as well – and the alerting is really quick and accurate. Therefore Veeam wins this round.

Round 2: Pricing

vSphere is expensive – period. And since vCenter has its own monitoring capabilities it could be hard to justify another large investment. As always its hard to define a ROI on solutions that mitigate risks if it is possible at all. QMX for vSphere is free. Extensions for other devices are not and are generally somewhat more expensive then other solutions (for instance for networking devices) – but I’ll talk more about that in round three.

With Jalasoft you pay per device. If you have one vCenter server, you pay for one device. SCVMM is a a part of the System Center Suite. If you have the proper agreement with Microsoft you get it for “free” once you’ve joined the dark side.

Veeam is so closely aligned with vSphere – they even have (or at least had with vSphere 4.*) the same pricing model. And the price per socket is quite high. But you could ask yourself – if proper monitoring, performance analysis and trend based alerting can increase my consolidation ratio I will be able to host more servers per physical host and need less sockets, less vSphere licenses and less Veeam licenses.

QMX is completly free – except for the OS license for the machine you host it on – so QMX wins this round.

Round 3: Vision, Tactics, Strategy..whatever

This round is about how the solution fits in a management or monitoring vision. So the outcome is going to be very subjective. But hey – when vendors talk about a journey to the cloud they are talking about just that – a vision or even a paradigm if you want about how to manage infrastructure to properly deliver services to users.

If you are virtualizing your infrastructure you are consolidating. So one thing you don’t want to do is to introduce a monitoring server sprawl. Despite the name the current incarnation of the System Center Suite is not at all an organic whole. Still using SCVMM makes sense, especially if you also use Hyper-V in your environment – but you would still need to check vCenter regularly as well because otherwise you are going to miss crucial information about the state of your environment.

Jalasoft and QMX are frameworks. QMX also gives you the possiblity to extend System Center Configuration Manager and has the broadest support for other non-Microsoft platforms and devices. Jalasoft is very network oriented but has a great integration with another add-on to SCOM, Savision LiveMaps.

Veeam – as described in the previous rounds – is very vSphere oriented. It does vSphere, it does it very well, but you will still need something of a framework next to Veeam and SCOM to monitor the other layers of your infrastructure such as your SAN storage or your network.

I put my faith in the frameworks. And I think its inevitable that a solution like Veeam will be built by either Vmware themselves or one of the vendors that offer a monitoring framework at some point in the near future. This round goes to QMX because of the integration with SCCM and the support for just about any non-Windows platform or application out there.

So the winner is..and some final thoughts

I think QMX is the best option available today if you are looking for a solution that is very configurable, affordable and has enough promise for the future to justify investing time and money into making the framework part of your monitoring infrastructure. But….

  • There are other options – vKernel has quite a nice toolset and claims to connect to SCOM – I will be testing that soonish
  • SCVMM 2012 is said to prvoide better vSphere integration and SCOM 2012 is said to have improved network device monitoring. I will look at those two in detail as well and report back with my findings.
  • You could build your own MP – you get get all the relevant data from vCenter using Powershell and SNMP gets and traps
  • SCVMM 2008 has a nasty habit of setting custom properties on your virtual machines – but you can us Powershell (isn’t that ironic) to get rid of those properties – for more info : VCritical article
  • Since Powershell and vSphere are so compatible I’m really surprised that I haven’t found a solution based on just Powershell to link SCOM and vSphere together.

Monitoring Citrix Netscaler Load Balancers with SCOM 2007 R2 Part I.

In Citrix, Netscaler, Operations Manager on October 19, 2011 at 19:19

Introduction

(Part II , Part III)

We recently introduced two Citrix Netscaler clusters into our environment. The first cluster was already running as a Citrix Access Gateway cluster (as an upgrade from our Secure Gateway – needed to support Citrix receiver on IOS devices), we purchased a load balancing license for that cluster and are using it to load balance servers in our DMZ. The other cluster is used to load balance servers in our internal network.

We mainly use the load balancers to create what I call “controlled redundancy”, but we do use it for several critical applications, such as the before mentioned XenApp environment. And one of the key elements in achieving this state of controlled redundancy in my humble opinions is being able to monitor these clusters.

Citrix offers an excellent application to monitor and administer their line of networking products called “Command Center”. But our central monitoring solution is Microsoft SCOM 2007. Of course we could have decided to use both products side-by-side or try to engineer some connector between Command Center and SCOM. But since the number of management task we have to perform on our Netscalers is very small – and the fact that Citrix has a SCOM MP for the Netscalers – we are now managing the two cluster using the GUI and SSH for the time being and installed the SCOM MP.

In this series of posts I am going to show how we installed, configured and tuned the management pack. I’m also going to cover the configuration of the Netscalers and the usage of the Netscaler pack – mainly because its structure is a little different then most standard Microsoft MP’s.

We use vSphere as our virtualization platform so I have no experience with the PRO MP’s that are provided to use SCVMM PRO TIPS  – so all I can say about that is that its unfortunate that there is no comparable feature for vSphere.

Installation

The SCOM pack can be downloaded from myctrix if you have the proper licenses associated with your accounts. However – the same pack can also be obtained from the following Citrix Community blog post 🙂

http://community.citrix.com/pages/viewpage.action?pageId=79463085

I found that link in this Citrix KB article: http://support.citrix.com/article/CTX122844 – which discusses an issue with this pack and a x64 OS. We actually ran into this issue but more about that later.

Btw both downloads will get you the 2.0 version of the MP – there is a 1.0 version out there for older firmware builds. We have both a classic 9.2 build and a ncore 9.2 build in our environment and we use the 2.0 pack for both.

The installation is pretty straightforward. We do all SNMP based monitoring from a separate management server so it made sense for us to install the MP there. The management pack can do SNMP gets and receive SNMP traps so you’ll have to enable the built-in SNMP service on the management server.

You run the installer and then import the MP into SCOM.  Now its time to configure the Netscalers!

Netscaler Configuration

In order to configure the Netscalers to be monitored by SCOM there are a couple of things you’lll need to configure, but one of things that really bugged me was the fact that in order to properly monitor the cluster I needed to be able to add both nodes to SCOM – which basically means that you have to create your NSIPs in a routed part of your network, which is against Citrix best practices ( or somehow multi-home your management server of course).

So besides configuring your NSIP so that it’s reachable and has SNMP enabled everything you need to configure is in the System\SNMP node of the Netscaler GUI. I’m not familiar with the CLI yet however your just as easily configure it there I guess.

  • First there is the SNMP community:

To monitor the Netscalers only a GET permission is needed, choose Add and input your SNMP string en choose the permission

  • Then you’ll to add the SCOM server(s) or their IP range as SNMP Manager:

Choose Management Host to use a single IP, network for multiple. In our case we have a dedicated VLAN for our monitoring and management servers.

  • Next up are SNMP traps:

This is that part where I ran into some issues – it took me some time to figure out I needed to use Specific as the type instead of Generic. You also need to define the Trap destination and port. Before,I mentioned you needed to use the NSIP to monitor the Netscalers, but that’s only for the SNMP GETS because you are able to set a cluster wide SNIP or MIP as the source address. Minimum severity and Community name are obvious however don’t be fooled by the parenthesis in the Community Name field – you actually have to enter your own string without parenthesis!

That’s most of the configuration on the Netscalers – in the next two parts I’ll discuss discovering the Netscalers, how to tune and configure the monitoring process on both SCOM and the Netscaler and I’ll try to show a little bit about the structure and the usage of the MP – especially because its a little different then your ordinary Microsoft MP.

(Part II , Part III)

Running Powershell script as SCOM console task and passing named parameters

In Operations Manager, Powershell, System Center on May 19, 2011 at 09:54

We have a ticketing system for which there is no SCOM connector and we wanted to provide a simple way to forward an alert to the ticketing system by email. We already stumbled upon the Alert Forward Task MP by Cameron Fuller but to add some flexibility I decided to rewrite it using a powershell as the application that is being executed. Contrary to agent task there is no default functionality to specifically run a powershell script to it took some time to figure out how I should call the script from the task and how to pass the needed parameters.

The variables I wanted to get from the alert where the MonitoringObjectName,Name,Description,Severity and Time Raised. I would then use those variables as named paramters to the powershell script that actually sends the mail. This is the code for the powershell script:

Param($managedobject,$name,$description,$time,$severity)
# Variables
$recipient= someone@someone.local
$mailserver = mail@someone.local
$sender= someoneelse@someone.local
$body=@"
<html>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<body>
<h1>$name<br></h1>
<b>Managed Object:</b>$managedobject<br>
<b>Description:</b> $description<br>
<b>Time Raised:</b> $time<br>
<b>Severity:</b> $severity
</body>
</html>
"@
$subject="Operations Manager Alert:"+"$name"
# Let's send an email
Send-MailMessage -ErrorVariable $mailerror -From $sender -To $recipient -SmtpServer $mailserver -Subject $subject -Body $body -BodyAsHtml
# If you enable output on the console task use the clause below to give some output.
#"@
#$message=@"
#The alert: $name ,
#has been forwarded.
#"@
#if ($mailerror)
#    {
#    Write-Host $mailerror
#    }
#else
#    {
#    Write-Host $message
#    }

As you can see it is a rather simple script that takes the input from the alert, builds an html message and uses Send-MailMessage to send the message. If you’d like to show output (error, success) you can uncomment that section and set RequireOutput to true in the XML.

Then I created a console task in the SCOM Authoring console and specificed a command line application, the paramters and working directory. Of course it took me some time to find the right syntax and looking at the xml I noticed something peculiar, all the parameters you enter in the gui are put into one <parameter> element inside the XML. Even when you edit the xml by hand and add each parameter as a separate element and open up the pack in the console and change something like the display name and save it it wraps them all up in one element again. Testing showed that with all the arguments in one element in the XML the task doesn’t work.

Here is the XML snippit of the console task:

<ConsoleTask ID="MCSE.AlertForward" Accessibility="Public" Enabled="true" Target="System!System.Entity" RequireOutput="false" Category="Alert">
<Application>%windir%\System32\WindowsPowerShell\v1.0\powershell.exe</Application>
<Parameters>
<Parameter>-noprofile</Parameter>
<Parameter>"&amp; \\someuncpath\forward-alert.ps1"</Parameter>
<Parameter>-ManagedObject '$MonitoringObjectName$'</Parameter>
<Parameter>-Name '$Name$'</Parameter>
<Parameter>-Description '$Description$'</Parameter>
<Parameter>-Severity '$Severity$'</Parameter>
<Parameter>-Time '$TimeRaised$'</Parameter>
</Parameters>
<WorkingDirectory>\\someuncpath\</WorkingDirectory>
</ConsoleTask>

Easiest way to add this task would be to copy/paste the xml into an existing MP and import the MP into your SCOM environment.

I put the powershell script on a share accessible to all our SCOM operators, used powershell.exe as the application and the script path and the variables from SCOM as arguments.Notice the double quotes around the script path and the single quotes around the alert variables.

You can of course create a more elaborate email, for instance using this excellent script by Tao Yang as an example. (Tao creates his own channel to send email notification and set up subscriptions but the code used to collect the data from SCOM can also be used in a console task).

Persistence is Futile

In Opalis, System Center on November 11, 2010 at 17:43

 

Opalis

In my earlier post I mentioned Opalis. Now what is Opalis? Opalis is an IT process automation tool. It gives you the possibility to visually design workflows that orchestrate, manage and monitor your whole process. By using integration packs Opalis is able to communicate with a host of different systems, vendors and platforms. You can get data out of systems, into systems and base your workflow’s logica on the repsonses you get from those systems.

In the breakout session I attended Opalis was compared to a mainframe run book: a formalization of all the steps involved in a process from start to end. And because of the great interoperability you can start by taking your “informal” processes and putting them into Opalis – no chance in functionality but know you let Opalis handle the execution (for instance calling Powershell), the monitoring/logging (by raising an alert in SCOM if something goes wrong or even creating an incident in Service Manager) and the decision making logic. So instead of incorperating all of that in every script you find in your environment you create a template which you can then reuse for every task.

Opalis itself was a so called third party tool vendor but is now a fully owned subsidiary of Microsoft and has been included in the System Center suite. In later posts I will try to get into the technical details of Opalis and how it relates to Microsoft Cloud management solution.

Computer says Yes

In Service Manager, System Center, Tech Ed on November 10, 2010 at 21:16

In this post I’ll give an overview of SC Service Manager:

Service Manager is an IT service management tool which can provide problem,incident and change management while fully integrating with the other System Center products. You need a CMDB? Connect Service Manager to SCCM and SCOM and you have your CMDB. You want to create an incident if an alert is generated in SCOM? Connect Service Manager to SCOM and there you go. Want to see the same distributed applications in your IT service management tool as you’ve defined in SCOM? Import your existing MP in Service Manager and you’re good to go.

Besides being able to tap into the information provided by SCOM and SCCM Service manager enables you to create work-flows to formalize and/or automate your existing processes. Since existing scripts for common tasks can be included in the workflow you can pick up those pesky scripts and put them into Service Manager so that they are visible, documented and manageable. Combined with Opalis you could take all tasks and scripts (defrags, legacy nt backups, third party config exports) and use Opalis to orchestrate these processes and use Service Manager/SCOM to manage and monitor them. But more on Opalis later.

Eventhough the interface might seem a bit quirky for users accustomed to other IT service management or ticket handling systems the fact that you have all this information about your environment, are able to create logical workflows to for instance create a template for standard changes and are able to automate the change and monitor the change will it is being made in your environment makes this a very powerful tool.

Service Manager uses an extension of the SCOM schema and uses Management Packs just like SCOM does. Out of the box Microsoft provides with MP’s for a knowledge base,change and incident management and they are working with partners to provide things such as asset management.

Microsoft positions Service Manager as the focal point for customer\IT interaction and as a presentational layer to expose and act on information from your data-center.

Service Manager is available for free if you have a System Center Enterprise or Datacenter edition license.