Scott Oseychik : PIC

Upcoming scheduled Yahoo / PIC maintenance window - November 17, 2009

Scott Oseychik — Wed, 11 Nov 2009 20:28:00 GMT

Summary:

Yahoo will be performing scheduled maintenance on their OCS Gateway Servers during the following maintenance window:

Confirmed Maintenance Start Date: Tuesday, November 17, 2009, 5:00pm Pacific Time

Projected Maintenance End Date: Tuesday, November 17, 2009, 5:30pm Pacific Time

Users may encounter sporadic access service availability, service disruption, and incorrect presence during this maintenance window.

User Action:

No action needs to be taken by your Office Communications Server administrator(s). Additionally, there will not be any further notification regarding this maintenance activity unless there is a broad disruption of service. In such cases, we will relay outage communication accordingly.

Upcoming scheduled AOL / PIC maintenance window - October 8, 2009

Scott Oseychik — Thu, 08 Oct 2009 02:11:00 GMT

Summary:

America Online will be performing scheduled maintenance on their SIP Access Gateways during the following maintenance window:

Confirmed Maintenance Start Date: 10/08/2009 04:00 ET (24 Hour Time)

Projected Maintenance End Date: 10/08/2009 06:00 ET (24 Hour Time)

There may be sporadic access service availability and users may experience service disruption.

User Action:

Temporary Yahoo! PIC Maintenance: September 24, 2009 – September 25, 2009

Scott Oseychik — Fri, 25 Sep 2009 06:37:00 GMT

Summary:

Yahoo! will be performing maintenance that will affect their OCS gateways. The outage is scheduled to be brief, but as a precautionary measure, the following maintenance window has been declared:

Maintenance Start Date/Time: 9/24/2009, 10:00pm Pacific Time

Maintenance End Date/Time: 9/25/2009, 1:00am Pacific Time

Impact:

Office Communicator users may notice incorrect presence of their Yahoo! Contacts, or may not be able to communicate via PIC with their their Yahoo! contacts.

User Action:

Additional troubleshooting step when encountering PIC issues w/AOL

Scott Oseychik — Fri, 24 Jul 2009 18:48:00 GMT

Summary:

America Online ("AOL") uses Global Server Load Balancing ("GSLB") to maintain their SIP gateway farm. Their configuration uses sub VIPs, one for each of their two datacenters. Whenever AOL performs maintenance (new code, hardware, etc.), they use GSLB to take one datacenter offline while they update it and point all of the SIP traffic to the secondary datacenter.

As with all GSLB systems, DNS caching on the client side can be an issue. For example, this past week (July 16th), AOL replaced some hardware at one of their sites. When doing so, they directed all traffic to their other site around 12am EDT. It takes a maximum of 30 seconds for these changes to be reflected in DNS.

Several of our Office Communications Server customers were having issues connecting via PIC to AOL (through their SIP gateways) as late as 6:30am that morning, but that the issue resolved itself in about an hour's time. This sounds very much like DNS caching at work. That is, the users began using the system at 6:30am and continued to have issues until their DNS was refreshed.

Action Item:

While it is unclear where the caching took place (local host, ISP DNS, local network, or within the OCS topology itself), a good rule of thumb is that whenever an OCS customer is unable to reach AOL's SIP gateways via PIC, the first troubleshooting step should be to initiate a DNS flush (and that it is best to use the sip.oscar.aol.com FQDN instead of a specific IP address when connecting to AOL).

Upcoming scheduled AOL / PIC maintenance window - July 15, 2009 - July 16, 2009

Scott Oseychik — Wed, 15 Jul 2009 19:52:00 GMT

Summary:

America Online will be performing scheduled maintenance on their SIP Access Gateways during the following maintenance window:

Confirmed Maintenance Start Date : 07/15/2009 22:00 ET (24 Hour Time)

Projected Maintenance End Date : 07/16/2009 00:00 ET (24 Hour Time)

There may be sporadic access service availability and users may experience service disruption.

User Action:

Federation and/or PIC may fail against partners using 2048-bit signed Root CAs

Scott Oseychik — Tue, 14 Jul 2009 20:18:00 GMT

Summary:

You may find that OCS 2007 / OCS 2007 R2's Federation and/or PIC fails against partners who do not support Entrust root certificates signed using a 2048-bit public key.

For example, the "broken" certificate chain may resemble the following:

Edge server's certificate -> Entrust L1B chain -> Entrust 2048 Root

Workaround:

For Federated partners who do not support the 2048-bit Root, you can introduce an additional chain certificate which points back to the Entrust 1024-bit root. The chain of authority would then be as follows:

Edge server's certificate -> Entrust L1B chain -> Entrust 2048 chain -> Entrust 1024 Root

To accomplish this, you will be replacing the Entrust 2048 Root certificate with the attached Entrust 2048 chain certificate.

Here are the steps to follow:

1. Start up your MMC console and add the Certificates snap-in for your server's Computer Account.

2. Under "Trusted Root Certification Authorities/Certificates," remove the Entrust.net Certification Authorities (2048) certificate.

3. Ensure you have the Entrust.net Secure Server Certification Authority certificate under the same Trusted Roots folder.

4. Under "Intermediate Certification Authorities/Certificates," import the attached Entrust.net Certification Authorities (2048) chain certificate as follows:

i. Save the attached "2048-to-1024-Cross-Cert.txt" 2048 chain certificate as a *.crt file.

ii. In MMC, expand the Intermediate Certification Authorities folder.

iii. Right-click on Certificates and select All Tasks -> Import

iv. Follow the resulting Certificate Import Wizard to import the 2048 chain certificate into the Intermediate Certification Authorities store.

5. Check to make sure you have two Entrust Certificates under Intermediate Certification Authorities/Certificates: The Entrust Certification Authority - L1B and the Entrust.net Certificate Authority (2048).

Credits:

Many thanks to Jimmy Levesque and Mark Giannotti in ECS Technical Support at Entrust Certificate Services for this information!

Update (December 10, 2009):

We have successfully tested & validated that communicating with AOL via PIC (using a certificate rooted against a CA that is signed with 2048 bits) works properly.

Tweaking OCS for PIC with Windows Live/MSN Messenger

Scott Oseychik — Thu, 18 Jun 2009 17:59:00 GMT

Summary:

Windows Live/MSN Messenger defines both Minimum and Maximum values for Presence subscriptions. If the SUBSCRIBE method uses a value out of this range, it may result in a SIP failure. The minimum value is 3600 seconds and the MAX value is 13000 seconds.

More Information:

MinPresenceSubscriptionExpiry

· Stored in the Active Directory as "msRTCSIP-MinPresenceSubscriptionTimeout"

· The default value is 1200 seconds (20 minutes)

· The minimum value for this property is 300 and the maximum value is 86340

MaxPresenceSubscriptionExpiry

· Stored in the Active Directory as "msRTCSIP-MaxPresenceSubscriptionTimeout"

· The default value is 43200 seconds (12 hours)

· The minimum value for this property is 300 and the maximum value is 86340

Workaround:

If you are encountering SIP interoperability failures via PIC against Windows Live/MSN Messenger, please modify your msRTCSIP-MaxPresenceSubscriptionTimeout and msRTCSIP-MinPresenceSubscriptionTimeout values (either via ADSIEDIT or LDP) in accordance with their guidance (min: 3600, max: 13000)

If you have installed OCS into the System container in the Active Directory, these attributes will be stored at the following location:

CN=Global Settings,CN=RTC Service,CN=Microsoft,CN=System,DC=yourdomain,DC=com

If OCS is installed in the Configuration container, you can locate these values in the Active Directory here:

CN=Global Settings,CN=RTC Service,CN=Services,CN=Configuration,DC=yourdomain,DC=com

Update:

If for any reason the above-mentioned values get reset, they will then fall outside of the default values. Consequently, you'll need to modify the msRTCSIP-DefPresenceSubscriptionTimeout value, as without correcting this to be within range, the OCS Front-End service will not start. Additionally, you will find the following error in the Office Communications Server Event Log:

Log Name:      Office Communications Server
Source:        OCS User Services
Date:          8/20/2009 6:30:31 AM
Event ID:      30929
Task Category: (1006)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      ocs.contoso.com
Description:
Failed to process WMI event to update the User Services Global Settings because the value of Attribute1 cannot be greater than the value of Attribute2.

Attribute1: DefPresenceSubscriptionExpiry Attribute2: MaxPresenceSubscriptionExpiry
Cause: This could happen during upgrades, or in topologies with mixed 1.0 and 2.0 servers or if the Active Directory is modified directly.
Resolution:
Use MMC (or WMI) to update all the values that are out of bounds. The changes will take place without restarting the server.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
    <Provider Name="OCS User Services" />
    <EventID Qualifiers="50158">30929</EventID>
    <Level>2</Level>
    <Task>1006</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2009-08-20T13:30:31.000Z" />
    <EventRecordID>26728</EventRecordID>
    <Channel>Office Communications Server</Channel>
    <Computer>ocs.contoso.com</Computer>
    <Security />
</System>
<EventData>
    <Data>DefPresenceSubscriptionExpiry</Data>
    <Data>MaxPresenceSubscriptionExpiry</Data>
</EventData>
</Event>

Credit to Jason Groves & Greg Anthony for the msRTCSIP-DefPresenceSubscriptionTimeout piece

RESOLVED - Technical inaccuracy in KB 897567 (specific to PIC, LCS/OCS, and Windows Live/MSN Messenger)

Scott Oseychik — Sat, 13 Jun 2009 22:13:00 GMT

Update: As of June 25, 2009, the KB article 897567 has been updated, and now is accurate.

Summary:

As of June 13, 2009, the content of the KB article 897567 contains an inaccuracy. The last octet for the second MSN IP address was improperly transposed. It’s published as 64.4.9.254, but that should be .245.

We are working on getting the Knowledge Base article updated, but in the meantime, wanted to make our customers aware of this matter sooner rather than later. Unfortunately, Microsoft cannot change the infrastructure to remedy this, as the "incorrect" IP address is already in use for another service in the Messenger cloud.

I will update this blog entry once the KB article has been updated to reflect the correct information.

Upcoming scheduled AOL / PIC maintenance window - June 4, 2009

Scott Oseychik — Wed, 03 Jun 2009 22:14:00 GMT

Summary:

America Online will be performing scheduled maintenance on their SIP Access Gateways during the following maintenance window:

Confirmed Maintenance Start Date: 06/04/2009 04:00 ET (24 Hour Time)

Projected Maintenance End Date: 06/04/2009 06:00 ET (24 Hour Time)

There may be sporadic access service availability and users may experience service disruption.

User Action:

Failure to establish PIC connectivity to MSN

Scott Oseychik — Tue, 12 May 2009 20:28:00 GMT

Symptoms:

LCS 2005, OCS 2007, and/or OCS 2007 R2 fail to establish PIC connectivity to MSN after changes were made to the MSN infrastructure on May 9, 2009

Data:

In the logs, you'll see the following data:

TL_ERROR(TF_CONNECTION) [1]08B4.146C::05/12/2009-15:32:27.115.00004bcf (SIPStack,SIPAdminLog::TraceConnectionRecord:1224.idx(157))$$begin_record

LogType: connection

Severity: error

Text: Receive operation on the connection failed

Local-IP: 167.207.100.260:2540

Peer-IP: 65.54.227.249:5061

Peer-FQDN: federation.messenger.msn.com

Connection-ID: 0x375D101

Transport: TLS

Result-Code: 0x80072746 WSAECONNRESET

$$end_record

TL_ERROR(TF_CONNECTION) [1]08B4.146C::05/12/2009-15:32:27.115.00004bde (SIPStack,SIPAdminLog::TraceConnectionRecord:1224.idx(157))$$begin_record

LogType: connection

Severity: error

Text: The connection was closed before TLS negotiation completed. Did the remote peer accept our certificate?

Local-IP: 167.207.100.260:2540

Peer-IP: 65.54.227.249:5061

Peer-FQDN: federation.messenger.msn.com

Connection-ID: 0x375D101

Transport: TLS

$$end_record

Cause:

The old VIP is being cached in your environment somewhere. If you look at the logs, connections are still trying to be made against 65.54.227.249, yet this VIP has been turned off on the MSN side.

Federation.messenger.msn.com is currently being directed to 64.4.9.181, though per Microsoft KB 897567, there are 5 possible IP addresses. At this time the 64.4.9.181 IP address is the only one in service (and the other IP addresses listed in the article will be brought online as load/capacity deems necessary).

Resolution:

Clear the DNS cache (or HOSTS file), as they are resolving to the wrong IP address.

Credit:

Nathan Novak (in MSN; he rocks!)

RESOLVED - Federation with Microsoft is down (aka: "It's not you, it's me.")

Scott Oseychik — Tue, 07 Apr 2009 18:18:00 GMT

We're actively debugging this, and I'll update this post once we have this resolved.

UPDATE (6pm, Eastern): We have identified the issue, and will be making changes this evening to rectify the issue. Things should be resolved by tomorrow morning.

UPDATE 2 (9pm, Eastern): Huge thanks to the RTC Ops, RTC SE, and Core Dev Teams for diving in and helping nail this. In the interest of transparency, this issue is also preventing Microsoft from communicating to our PIC providers (Yahoo, AOL, MSN).

UPDATE 3 (8am, Eastern, 4/08/09): Federation & PIC is now back up and running at Microsoft.

RESOLVED - OCS 2007 R2 PIC fails against AOL

Scott Oseychik — Fri, 03 Apr 2009 21:14:00 GMT

Issue:

Microsoft Office Communicator 2007 R2 in conjunction with Office Communications Server 2007 R2 would intermittently fail to communicate with AOL AIM clients via PIC. Note that this would only reproduce if your OCS 2007 R2 Edge role is running Windows Server 2008 (x64); not Windows Server 2003 (x64).

Resolution:

Essentially, it boils down to tweaking the Windows Server 2008 Edge role to initially establish the SSL dialog using the TLS_RSA_WITH_RC4_128_MD5 cipher suite.

In order to change the cipher suite order, do the following on your Windows Server 2008 (x64) Edge server:

1. Start -> Run -> gpedit.msc -> OK

2. Within the Group Policy Object Editor, expand Computer Configuration, Administrative Templates, Network

3. Under Network, select SSL Configuration, and then double-click on SSL Cipher Suite Order (by default, the SSL Cipher Suite Order is set to "Not Configured")

4. Select the “Enabled” radio button, and in the in the SSL Cipher Suites text box, copy the entire string into Notepad. It should look like the following:

TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P384,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P521,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P384,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P521,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P521,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P521,TLS_DHE_DSS_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES_256_CBC_SHA,TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_RC4_128_MD5,SSL_CK_RC4_128_WITH_MD5,SSL_CK_DES_192_EDE3_CBC_WITH_MD5,TLS_RSA_WITH_NULL_MD5,TLS_RSA_WITH_NULL_SHA

5. The objective here is to move TLS_RSA_WITH_RC4_128_MD5 to be a the front of the list. So, in your Notepad document, find TLS_RSA_WITH_RC4_128_MD5, cut it, navigate to the beginning of your notepad document, and paste TLS_RSA_WITH_RC4_128_MD5. The new order should look like the following:

TLS_RSA_WITH_RC4_128_MD5,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P384,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P521,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P384,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P521,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P521,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P521,TLS_DHE_DSS_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES_256_CBC_SHA,TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA,SSL_CK_RC4_128_WITH_MD5,SSL_CK_DES_192_EDE3_CBC_WITH_MD5,TLS_RSA_WITH_NULL_MD5,TLS_RSA_WITH_NULL_SHA

6. Paste the newly-formatted string back into the text field in the GPO Editor, click OK, then restart your Windows Server 2008 (x64) Edge server for these changes to take effect.

We have verified (and re-verified) these steps work, and can now successfully communicate with AOL AIM clients using Office Communicator 2007 R2 via PIC.

Additional Windows Live Messenger PIC/Federation IP Addresses

Scott Oseychik — Mon, 09 Mar 2009 20:04:00 GMT

Notification

In an effort to provide enhanced capacity and service reliance, Windows Live Messenger will soon be adding additional IP addresses used for PIC/Federation traffic. Some organizations have chosen to restrict this type of traffic to specific IP addresses, as referenced in Microsoft KB 897567 (http://support.microsoft.com/kb/897567). With this in mind we want to give you advanced notice of our intended change if you have configured your enterprise network in this manner.

Please ensure your enterprise firewall configuration is updated with the full list of Windows Live Messenger addresses below on or before Friday, May 8, 2009. Windows Live Messenger will NOT enable any of the additional IP addresses until on or after May 9th(Pacific Time) to ensure your services will not be disrupted by this change.

IP address for Windows Live Messenger PIC/Federation:

65.54.227.249

64.4.9.181

64.4.9.245

65.54.52.53

65.54.52.245

For more information please reference Microsoft KB 897567, or for further assistance, please engage Microsoft Customer Support Services via http://support.microsoft.com.

Upcoming scheduled AOL / PIC maintenance window

Scott Oseychik — Fri, 30 Jan 2009 18:10:00 GMT

Summary:

America Online will be performing scheduled maintenance on their SIP Access Gateways during the following maintenance window:

Confirmed Maintenance Start time: 2009-02-03 04:00 ET

Projected Maintenance End time: 2009-02-03 06:00 ET

There may be sporadic access service availability and users may experience service disruption.

User Action:

PIC stops functioning after installing/renewing your certificate

Scott Oseychik — Tue, 30 Dec 2008 17:00:00 GMT

Summary:

After installing or renewing your DigiCert public certificate for your OCS 2007 Edge server(s), PIC stops functioning properly (however, Remote Access still works correctly).

Cause:

This issue occurs if the certificate issued uses the Certificate Authority listed below:

Root: DigiCert Global CA (2048), Intermediate Root: Entrust.net Certificate Authority (2048)

It will work if DigiCert issues certificate from the Certificate Authority below:

Root: Entrust.net Secure Server Certificate Authority, Intermediate Root: DigiCert Global CA

More Information:

DigiCert is aware of this issue, and is collaborating with Microsoft to ensure our mutual OCS customers experience minimal to no impact. DigiCert has made a change on their end to solve this problem moving forward.

If you are encountering this issue, you will need to reissue and replace any certificates which are issued from the "DigiCert Global CA (2048)" certificate. DigiCert has made a change so your replacement certificate(s) will descend from the correct Entrust.net root certificate for PIC. For help with any part of this process, please engage DigiCert support, either via web chat, phone, or e-mail at support@digicert.com.

After re-applying this certificate to your Edge server, and you still find that your PIC-related issues are still occurring, please restart Edge Front-End services first. Allow me to apologize for this up front; I understand this will require an “emergency service restart change request” for some of you.

If all this fails to resolve the PIC issue, please engage Microsoft Customer Support Services. Premier customers: please leverage your Technical Account Manager to initiate the case creation process.

Please be prepared to supply Edge Server logs, remote access via our EasyAssist applications from MSFT, and we will do our best to investigate and resolve this in a timely manner.

Kudos to Paul Tiemann @ DigiCert for his tenacity & helpfulness ...