Month: November 2010

User Profile Synchronization… the sticky underbelly of SharePoint 2010

In my time working with SharePoint 2010 I have been often amazed at the thought that went into making the product as seamless and clean as possible. It was as though Microsoft listened to all of our gripes and grimaces and attacked them all with a passion to make SharePoint 2010 the best product they ever released. And then there is the User Profiles.

Some of my favorite enhanced features of SharePoint 2010:

1.) Scheduled Automated Password Resets – We spent months getting this right in SharePoint 2007 because our environment was so complex, and then we still had problems from time to time where things just didn’t get set cleanly

2.) No multi-try service starts – In all other versions of Microsoft products that I have worked with we always had multiple fail times before allowing you back to a stopped state, and some services would just hang in a starting state forever until you rebooted.

3.) Implementation of the principle of least privilege for service accounts running services – You can specify a specific managed accounts to run services in order to manage permissions required to execute with the least privilege required to do the job

4.) Windows Service naming convention – Most of the SharePoint specific services have SharePoint in the name (simple right?)

5.) No elevated Active Directory Permissions required to run SharePoint

How did the FIM (Forefront Identity Management) team who implemented the User Profiles section of SharePoint 2010 break each of these enhancements? Let’s take a look:

1.) Scheduled Automated Password Resets –

The User Profile Synchronization Service MUST run as the farm admin account. There is no way to successfully change this. I have tried and it has been a miserable failure.

Unfortunately, the way that they implemented the UPS Service is that it requires that the password be entered upon startup. It does not respect Managed Accounts. This means that you cannot use the automatic password reset capability built into Managed Accounts for your farm admin account. You must set the password to something known so that you can enter it in for the UPS to start up. I haven’t figured out a programmatic way around this yet, but I am on the look out and will post something if I find it.

The best part is that outside of Spencer Harbar’s Bible for UPS, Rational Guide to implementing SharePoint Server 2010 User Profile Synchronization, (which is an absolute must read for anyone who is setting up a SharePoint 2010 farm and wants a chance in hades of getting UPS working) there is NO PLACE that comes right out and tells you this information. I am probably going to post a separate article on this just so that we start to get it out there for people to easily find. The #1 account you would expect to use the automated password changing functionality for, and the FIM team breaks your ability to use it right out of the gate. Hopefully this is going to be addressed in a service pack or an R2 version of the product because it is a thorn in an administrator’s side.

2.) No multi-try service starts –

Once you start the UPS Service it is time to go get a cup of coffee, read the paper, finish the cup of coffee and then go for a 5 mile run. That is unless you are like me and feed the ULS Logs and the Event Viewer hoping not to see one of those awful ERR errors that don’t flag as errors in ULS and don’t show up in the Event Viewer. If you are lucky enough to get the UPS Service to provision cleanly the first time out of the gate (you can thank your lucky stars, or Spencer, dealer’s choice), but if you are like most of us, it is going to take a couple of tries to get it right because you are either cutting corners, trying to be smarter than SharePoint, or just don’t follow directions well.

The bottom line is that the UPS Service tries (in my experience) no less than 10 times before it finally fails out and goes back to stopped, if it gets back to stopped. The more fun version of this story is when you end up in a starting state for a week or so. Then you want to rip what little hair you have left our of your head… but maybe that is just me. I have a good post on how to deal with that situation coming next week.

3.) Implementation of the principle of least privilege for service accounts running services –

This goes back to the farm account running the UPS Service. So much for least privilege…

4.) Windows Service naming convention –

The 2 services that run the UPS Service inside of SharePoint are listed in the Windows Services as “Forefront Identity Manager Service” and “Forefront Identity Manager Synchronization Service”.

5.) No elevated Active Directory Permissions required to run SharePoint –

In order to perform a User Profile Sync you must give the account performing the action the “Replicating Directory Changes” permissions in Active Directory.

I have yet to find a detailed exact explanation of what this permission has rights to do, but based on my testing it appears to be just an elevated permission to allow a deeper level of read. I have not tested to see if it can read from hidden confidential flags yet (an AD schema extension is required that I don’t have in any of my accessible ADs yet), but that will come in the next week or so. The Microsoft TechNet article was just updated to include the details on this in September and can be found at http://technet.microsoft.com/en-us/library/ee721049.aspx (nice credit at the bottom to Spencer here… nice to see that we are all listenting to the same guy that Microsoft is listening to)

Hopefully this helps to shed some light for you, gentle readers, on some areas of SharePoint that have certainly been tripping points for me and many of my colleagues. I still love the product and advocate for it passionately, I just tend to leave out the part about what a pain in the @$$ UPS is.

November 30, 2010
SharePoint 2010 Farm Service Account passwords expired?!?!?!?

Scenario:

Managed service accounts passwords expired. Access to part of Central Administration are no longer accessible. Sites are starting to go down because app pool passwords are managed accounts and have expired.

Soapbox moment:

Firstly, there is NO real excuse for this in SharePoint 2010 because the ability to have this done automagically for you is BUILT-IN, so either your farm admin is so over taxed (usually the case) or incompetent (the two aren’t mutually exclusive).
I take the liberty to say all of this as someone who has had this happen to them, otherwise I wouldn’t be able to write about it, right?

Resolution:

To start, you aren’t going to be able to do anything with SharePoint until you can get the Timer Job Service running again because everything is driven by timer jobs.
Using a credential that has full admin rights to the box and is a Farm admin, change the account the Timer Job Service runs as and start the service. This must be done on all servers in the farm.
Don’t fret, the next things you are going to do is fix this back to the way it should be but you can’t do the next steps without the timer job service running, so just play along.

Go to http://www.yourserver.com/_admin/ManagedAccounts.aspx and edit your farm service account and tell it to change the password now.

Go to http://www.yourserver.com/_admin/FarmCredentialManagement.aspx and select Farm Account. You will see your registered service account (the one that you just changed the password for) and click ok. This will go reset your Timer Job Service account to the registered account which now is active and working.
Next, create a text file called on your server which will be a list, one account per line, of service accounts that you are going to have auto-updated.

Then using the SharePoint 2010 Management Shell interface change the passwords and set to auto change run using this variable script:

foreach ($account in Get-Content driveletter:filename.txt)
{
Set-SPManagedAccount -Identity $account -AutoGeneratePassword -PreExpireDays n#days -Schedule “monthly between n#dayofthemonthvalue hh:mm:ss and n#dayofthemonthvalue hh:mm:ss” -confirm:$false
}

The script that I actually used, without the variables looks like this:

foreach ($account in Get-Content c:managedaccounts.txt)
{
Set-SPManagedAccount -Identity $account -AutoGeneratePassword -PreExpireDays 30 -Schedule “monthly between 7 02:00:00 and 7 03:00:00” -confirm:$false
}

Once this command has completed successfully you will see that your last password change just happened and that your next password change is scheduled. Make sure that if your next scheduled password change isn’t in conflict with a password change minimum group policy that won’t allow passwords to be changed before a minimum number of days or you will end up with some errors in your ULS Logs and some misfired password change attempts.

Lastly, go to http://www.yourserver.com/_admin/FarmCredentialManagement.aspx and walk through all of the farm credentials and let the accounts get synced up. This should re-spin up the app pools and get your users back into the site, but if not, do an IISRESET and things should be back online.

Shout outs

Huge thanks to my partners in crime on this one, Derek Martin and Trent Foley of Slalom Consulting, for helping with the out of the gate perfect PowerShell scripts that are referenced above.

While I was busy figuring out how to break back in to Central Admin, they figured out the proper script to reset the passwords and set the auto change programmatically. This script can be used in advance of this type of shenaniganal activity to ensure that while you are building your farm you get this set right the first time and not have to do it manually (which is often the excuse when you are using 30+ managed accounts in a farm).

November 7, 2010
Random Server hang issues result in a required hard reset

Symptom:

Windows 2008 R2 64bit systems hang at random and require a hard reboot of the system to recover. You can remote to the system via KVM (RDP is not accessible) and even do a CTRL+ALT+DEL, but after the lock screen goes away and tries to give you a login screen… YOU GET NOTHING. Only silence…

Root Cause:

We ended up with a three headed root cause on this set of issues.

1.) Our blades had a bad BIOS version that caused the system to get into an inconsistent state and required a power cycle to get them clear.

2.) The hardware vendor had Data Execution Protection (DEP) turned on at the hardware layer by default.

3.) By default Microsoft has its own version of DEP turned on for all services unless you add in exceptions.

How did we diagnose this beast? Many team members (Dan, Don, Christian, Jim, and Cornè) all weighed in and found part of this along with support from our hardware vendor and Microsoft.

The issues plagued us for several weeks because it was not a predictable failure and there was NOTHING in the logs to correlate the issues together other than a single model of blade server.

Call with Microsoft and the hardware vendor suggested that the Microsoft DEP might be part of the issue as well. Luckily our support level was good enough to get both vendors on the same line and have them work together. Support calls like this are not cheap if you don’t have the agreements in place already.

Resolution

1.) Flash the BIOS with an updated and vendor verified version.
2.) Turning off of the hardware DEP
3.) Setting the Windows DEP to on “for essential Windows programs and services only”

Since making these changes we have not seen reoccurrence of the random system hang issues. I will update this post if things change… but so far, so good!

November 6, 2010
Unable to access VMware Workstation Guest after Host Machine crash

Scenario:

While doing SharePoint 2010 development is now able to be done from within a Windows 7 64bit system and even inside of a Hyper-V VM, many of us in the wide world of computing are still using VMware Workstation, Fusion, Player, or even the old VMware GSX Server to run local development environments. This post deals with how to get back into a VM that seems horribly locked after your workstation hosting your VM crashes hard.

Symptom:

System crashed while a VMware Workstation Guest was running. The VM now reports that it is in use by the current host. Taking ownership of the VM is not possible. All hope is starting to fade…

Solution:

Go to the VM location and delete the .lck files and folders. This will unlock the VM Guest and allow you to boot in. VMware treats this like a hard server reset, and offers safe mode booting at that point. If the system is able to recover cleanly, it gets you back to operational.

Caveat:

Just like anytime you hard reset a server you must be prepared to deal with the consequences. Data loss, data corruption, and complete system failure is possible as a result of such action.

However if you are left with the alternative of complete system rebuild already, it might be worth a shot. Just don’t blame me if your system is toasty afterwards.

November 6, 2010
Windows with Claims User gets access denied to a site they had access to earlier in the day

Scenario:

Small Farm 3 tiered topology using Windows with Claims implementation aggregating AD with a custom LDAP database to create the claims roles.

Symptom:

Users of a SharePoint 2010 site get access denied to a site they could access earlier in the day. As the day goes on, the number of users effected increases. Eventually only users with full control policies can access the farm.

ULS Log error:

An exception occurred in Custom Roles claim provider when calling SPClaimProvider.FillResolve(): The underlying provider failed on Open..

Root Cause:

The 10 hour default session timeout for the user’s claim has been exceeded and the database housing the Role Data is no longer accessible. In this case it was due to an expired SQL account password. Changing the password and updating the connection string or just unchecking the password expiry flag in the SQL account will resolve the issue.

Notes from the field:

There was one easy way to prevent this type of user facing outage. Don’t allow SQL accounts to expire. EVER. They are horrible to diagnose because access to the SQL Server is still operational and access using AD authentication is going to throw you off the scent because the main farm access is still available.

Read Scot Hillier’s blog on “Authorization Failures with Claims-Based Authentication in SharePoint 2010”. Really useful stuff in there about how claims works and extending the timeouts.

November 6, 2010