jmbrinkman

Microsoft Server Activesync, Iphone and client certificates issues

In Exchange, Unified Communications on November 16, 2010 at 16:32

At my company we are currently performing a pilot to see if we can offer corporate email to users through an Iphone. We decided to go for a simple setup: one dedicated Exchange 2007 Client Access Server facing the Internet (behind a firewall of course) using HTTPS and client certificates on the Iphones. There are plenty of guides out there that discuss this topic (with or without client certificates and with reverse proxy server infront of the CAS server) so I’m not going to elaborate on that .We did take some standard security measures like using the Microsoft SCW, enabling the Windows Firewall on the external interface and install a virus scanner on the CAS server itself. We were using Iphone 4’s with IOS 4.1 and Exchange 2007 sp1 RU 9.

After we had rolled out the profiles the Iphone’s were syncing fine, but sometimes users weren’t able to connect to the server. One particular symptom that came up every now and then was that messages containing attachments wouldn’t send but would get stuck in the device’s Outbox. Some messages would remain stuck indefinitely while other would send after a certain time period.

On the CAS server itself I noticed the following error in the Application event log:

And in the System Log:

There were also some entries in the httperr1.log:

2010-11-15 22:47:19 109.34.215.23 60140 192.168.2.40 443 HTTP/1.1 POST /Microsoft-Server-ActiveSync?User=MY USERNAME&DeviceId=MYDEVICE&DeviceType=iPhone&Cmd=Ping – 1 Connection_Abandoned_By_AppPool MSExchangeSyncAppPool

At times we would also see Connection_Dropped_By_AppPool MSExchangeSyncAppPool and the same error as above but with the actual send and save command string.

Doing some research (aka using Google/Bing) gave me some information about IIS deadlocks and I found the following suggestions:

– Add another CPU if you have a single CPU VM

– Adjust the machine.config file for the .NET version mentioned in the event log

We tested both and that had no impact.

Additional troubleshooting steps we took were:

– Remove Anti virus, disable Windows Firewall -> No effect whatsoever

– We checked the session time-out on the Firewall, because Direct Push uses very HTTP sessions -> The firewall had a time-out value of 30 minutes and since the Direct Push sessions last about 15 minutes that couldn’t be the cause of our problems either

– Upgraded one of the Iphone’s to the IOS 4.2 GM -> Nada

After that Icontacted PSS in order to jointly investigate the issue. They looked at the logs and we performed a trace but nothing really came up.

Then I decided to have another look myself. I fired up Wireshark, exported the key of the SSL certificate and traced and decrypted the conversations between the device and the CAS server. In the conversations I noticed the following HTTP response:


So apparently the web server had problems with the size of the request. Searching Technet I found this article:

http://technet.microsoft.com/en-us/library/cc737382%28WS.10%29.aspx:

If a client sends a long HTTP request, for example, a POST request, to a Web server running IIS 6.0, the IIS worker process might receive enough data to parse request headers, but not receive the entire request entity body. When the IIS worker process detects that client certificates are required to return data to the client, IIS attempts to renegotiate the client connection. However, the client cannot renegotiate the connection because it is waiting to send the remaining request data to IIS.

If client renegotiation is requested, the request entity body must be preloaded using SSL preload. SSL preload will use the value of the UploadReadAheadSize metabase property, which is used for ISAPI extensions. However, if UploadReadAheadSize is smaller than the content length, an HTTP 413 error is returned, and the connection is closed to prevent deadlock. (Deadlock occurs because a client is waiting to complete sending a request entity, while the server is waiting for renegotiation to complete, but renegotiation requires that the client to be able to send data, which it cannot do).”

I’ve tried enlarging the UploadReachAheadSize to 64k but as could be expected (the attachment was much larger than that) that didn’t help. And just as the article says, increasing this value would create an attack surface on our server. So I followed the link on the bottom of the article to this article:

http://technet.microsoft.com/en-us/library/cc778630%28WS.10%29.aspx:

The SSLAlwaysNegoClientCert property controls SSL client connection negotiations. If this property is set to true, any time SSL connections are negotiated, the server will immediately negotiate a client certificate, preventing an expensive renegotiation. Setting SSLAlwaysNegoClientCert also helps eliminate client certificate renegotiation deadlocks, which may occur when a client is blocked on sending a large request body when a renegotiation request is received.”

I then used the adsutil script to set that value and voila! The messages were sent normally and the errors stopped occuring.

If you want apply either of those settings you should remember to restart the IIS Admin service and not just reset IIS.

I’ve seen several posts on the web dealing with the same issue or at least the same symptom. They might be related to our issue and I think that the UploadReachAheadSize could also affect sending email messages when no client certificates are being used.

Advertisements
  1. Hi Michiel,

    An interesting post. I was the poster on the iis forums thread on this subject. Adding CPUs (4 in the end) to the vm fixed it for us temporarily, but we have added a few more iphones to exchange activesync recently, and the problem has reappeared, although our setup was fine for months.

    We are using IIS 7 on server 2008. I applied the SSLAlwaysNegoClientCert to the default website which was hosting OWA/activesync, which seemed to fix the deadlock problem, but broke autodiscover for our domain connected Outlook 2007 clients as we only have a single CAS server in our environment at present. The symptoms of this for our Outlook 2007 clients, was the fact that we were unable to download the offline address list, and unable to access Out of Office. When running ‘Test E-mail Autoconfiguration’ autodiscover would return an 0x80072F0C error. I include this information for anyone else in a similar situation.

    I don’t know if the SSLAlwaysNegoClientCert can be applied to a single virtual directory in IIS e.g. the microsoft-server-activesync virtual directory, or whether it can only be applied to the entire site.

    So a few of questions for you:

    Did you apply this setting to the entire site hosting OWA/activesync on the CAS server?

    Do you have any idea if the SSLAlwaysNegoClientCert can be applied only to a single virtual directory in IIS?

    Do you have more than one CAS server, so your fix won’t break autodiscover for Outlook clients?

    I see some other people with similar issues seem to be extending the UploadReadAheadSize to 5MB on this thread:

    http://social.technet.microsoft.com/Forums/en-US/exchange2010/thread/52f58b47-b95e-4f44-bb4e-6bd8b1b4eb94/

    But as you mention this isn’t good from a security standpoint.

    Thanks for your post, there is some great detective work in there. You seem to have found the root cause, although the exact still fix eludes me in my environment right now.

    • I haven’t noticed any autodiscover issues yet – however we do have several CAS servers and only one of them is currently being used for Activesync (this will change however if we decide to implement EAS in production so I’m happy your provided this information so we can test with Outlook 2k7/2k10 clients and the SSLAlwaysNegoClientCert set on all CAS servers).

      I did apply it on the site level and according to the Ms article this is the only way to do it.

      http://technet.microsoft.com/en-us/library/cc728324%28WS.10%29.aspx

      There are other ways to work around the security implications of the UploadReadAheadSize without the SSLAlways, an enterprise firewall,IDP device or a reverse proxy could be used to control the traffic going to the CAS servers.

      The reverse proxy solution would also shift the client cert authentication from the CAS to the reverseproxy if you use constrained Kerberos delegation and don’t use HTTPS for the connection from the proxy to the CAS servers.

      In the next phase of our pilot we will deploy MS TMG as a reverse proxy and am planning to blog about it if we run into simular issues there.

      So you could test with the upload size at the desired attachment size and look into a workaround for the increased service of attack.

      On a side note: The technet article actually says you should set both values – I didn’t even set the UploadReadAheadSize and we were able to send large attachments already.

  2. Thanks for your response. We have 2 CAS servers in our test environment. Outlook clients couldn’t pick up autodiscover settings from the one with SSLAlwaysNegoClientCert enabled but could from the one without this setting enabled.

    I will report back when I make further progress.

  3. […] went hand in hand with the introduction of iPhones as mobile mail clients. After some research, this blog linked me to Technet Article cc778630, which […]

  4. Thanks for sharing your thoughts about iphone 5. Regards

  5. […] Microsoft Server Activesync, Iphone and client … – Nov 16, 2010 · I haven’t noticed any autodiscover issues yet – however we do have several CAS servers and only one of them is currently being used for Activesync …… […]

  6. […] Microsoft Server Activesync, Iphone and client … – Nov 16, 2010 · I haven’t noticed any autodiscover issues yet – however we do have several CAS servers and only one of them is currently being used for Activesync …… […]

  7. […] Microsoft Server Activesync, Iphone and client … – Nov 16, 2010 · I haven’t noticed any autodiscover issues yet – however we do have several CAS servers and only one of them is currently being used for Activesync …… […]

  8. […] Microsoft Server Activesync, Iphone and client … – Nov 16, 2010 · I haven’t noticed any autodiscover issues yet – however we do have several CAS servers and only one of them is currently being used for Activesync …… […]

  9. If you are going for most excellent contents like myself, only pay a quick visit this website everyday as it presents feature contents, thanks

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: