Problems after Changing Service Accounts

Recently we changed service accounts at one our of client SharePoint environment, this went through without a hitch. Just follow the MSDN article and you should be good to go.

After changing service accounts, we disabled the old service accounts in AD and The SharePoint farm was up and running fine. After 45 days we deleted the service accounts in AD, This is when the problems started. The farm became completely unstable, few of the symptoms were

1. The SharePoint sites were extremely slow to load,
2. Application Server Administration Service Timer Job was struck at initialized and eventually failed,
3. Application Server Timer Job was struck at initialized and eventually failed,
4. The event log was full of errors reading

Synchronization for Shared Services Provider ‘SSP Name’ has failed. The operation will be retried.

Reason: The specified account name is invalid.
Parameter name: account

All these errors made the farm very unstable and eventually the SharePoint farm shut down!

I rebooted the servers and the SharePoint sites came back online for few minutes, but went down again once the timer jobs failed again.

This is when I started digging deep, the diagnostic logs had the following error:

NTAccount ‘Account Name‘ could not be translated to a SID. Exception: System.Security.Principal.IdentityNotMappedException: Some or all identity references could not be translated.     at System.Security.Principal.NTAccount.Translate(IdentityReferenceCollection sourceAccounts, Type targetType, Boolean forceSuccess)     at System.Security.Principal.NTAccount.Translate(Type targetType)     at Microsoft.Office.Server.Utilities.WindowsSecurity.ValidateAccount(NTAccount account, Boolean throwIfInvalid

This account was the old service we replaced before and SharePoint was still trying to convert it into a SID. As it was not able to convert the account into valid, all the timer jobs were failing. This service account still being cached in the SharePoint Configuration cache. So the fix to the problem is to clear the SharePoint configuration Cache.

Even though your service accounts was disabled in AD, SharePoint was  able to convert it into a Valid SID. So the farm was running without any gliches but Once the service accounts were deleted in AD, SharePoint was no longer able to convert the service account into a valid SID. Hence the problems.

Clearing the SharePoint Configuration Cache fixed the problem, I will write a post on how to clear the SharePoint configuration cache soon.

This problem would happen not only when changing service accounts and deleting them at a later time but also when you upgrade the farm to a different domain. SharePoint might still have the old accounts in the cache and it will cause problems. So before deleting accounts, downtime should be planned and farm should be monitored for any anomalies.

Move all SharePoint databases to a different database server with a Different Name

This article primarily refers to the TechNet article about moving SharePoint Databases, I have added my own notes all along. Databases are critical for SharePoint farm to run, Moving databases can take down your farm so please have a roll back plan before you start.

Note: This process requires shutting down the SharePoint Services and will make all sites unavailable to users until the process is complete. Hence, downtime should be planned before starting this process

The following databases need to be moved to complete the process

• Databases for Shared Services Providers (SSPs)
• Search databases for SSPs
• Content databases
• Search database
• Central Administration content database
• Configuration database

This is a procedure for moving all the databases in an Office SharePoint Server 2007 farm to a new database server that has a different name. The new database server must use the same version of SQL Server as the old database server.

Important: Visit “Servers in the farm” under operations in central admin and record all server names running Windows SharePoint Services Database Service.

Note: The following are the minimum required to perform this procedure:

• You must be a member of the Farm Administrators SharePoint group.

• On the computer that is running the SharePoint Central Administration Web site, you must be a member of the Administrators group
• On the database server from which the databases are being moved; you must be a member of the Administrators group and should have at least the db_backupoperator fixed database role.
• On the database server to which the databases are being moved; you must be a member of the Administrators group and you should have at least the db_owner fixed database role.

Note: If you do not have permissions on Database server, please co-ordinate with SQL Server DBA to move the databases to the new server.

Move all databases to a different database server

1. Prepare the new database server by using the Prepare the database servers procedure.
2. Record which Web applications are associated with the SSP by performing the following steps:
   1. On the SharePoint Central Administration Web site, on the Application Management page, in the Office SharePoint Server Shared Services section, click Create or configure this farm’s shared services.
   2. Record the associated Web applications that are listed on the Manage This Farm’s Shared Services page.
3. Record the list of Web Applications, SSP’s and associated Databases by going to the “Perform a Backup” page under operations in Central Admin
4. Stop the farm by performing the following steps:
   1. On the server that is running the Central Administration Web site, in the Services snap-in, stop the following services:
      • Microsoft Single Sign-On service
      • Office Document Conversions Launcher service
      • Office Document Conversions Load Balancer service
      • Office SharePoint Server Search service
      • Windows SharePoint Services Administration service
      • Windows SharePoint Services Search service
      • Windows SharePoint Services Timer service
      • Windows SharePoint Services Tracing service
      • Windows SharePoint Services VSS Writer service
   2. On the server that is running the Central Administration Web site, at the command prompt, type iisreset /stop.
   3. Repeat step 4 on each server in the farm.

Note: If you do have sufficient privileges on the SQL Server, you can Skip steps 5-7 and let your SQL Server DBA team handle the Database Move. The DBA’s can follow backup and restore method as illustrated in Steps 5-7 or DBA’s can detach the Databases in the old server and attach the databases back on the new server.

5. Back up the databases on the source database server.

• Start SQL Server Management Studio and connect to the database server.
• In Object Explorer, expand Databases.
• Right-click the configuration database (usually named SharePoint_Config), point to Tasks, and then click Back Up.
• In the Back Up Database dialog box, in the Source area, select the type of backup that you want to perform from the Backup type list. Click Database.
• In the Backup set area, in the Name text box, type a name or use the default.
• In the Description text box, type a description of the backup.
• Specify how long the backup should be kept, or use the default. When the backup set expires, the backup set can be overwritten by any subsequent backups that have the same name. By default, the setting for the backup set is 0 days so that it never expires.
  1. In the Destination section, specify a location to store the backup set, or use the default.
  2. Click OK to back up the database.
  3. Repeat steps c through j for the remaining databases in the farm.

6. In Windows Explorer, locate the database backup (.bak) files that you want to move, and then copy or move them to the destination server.

• Restore databases on the destination database server.
  1. Start SQL Server Management Studio and connect to the database server.
  2. In Object Explorer, expand Databases.
  3. Right-click the database that you want to restore, point to Tasks, point to Restore, and then click Database.
  4. In the Restore Database dialog box, specify the destination and the source, and then select the backup set or sets that you want to restore.
• The default values for destination and source typically suit most recovery scenarios.
• In the Select a page pane, click Options.
• In the Restore options section, select only Overwrite the existing database. Unless your environment or policies require otherwise, do not select the other options in this section.
• In the Recovery state section:
  • If you have included all the transaction logs that you must restore, select RESTORE WITH RECOVERY.
  • If you must restore additional transaction logs, select RESTORE WITH NORECOVERY.
  • The third option, RESTORE WITH STANDBY, is not used in this scenario.
• Click OK to complete the restore operation.
• Repeat steps c through h for each database that you are restoring.

7. Use SQL Server to copy to the destination server the logons for all service accounts, including SQL Server logins, fixed server roles, fixed database roles, and permissions for the databases.

8. Refer the farm to the new database server.

Note: The renameserver operation does not rename named instances of SQL Server.

For Unnamed/Default Instance:

For unnamed instances of SQL Server, follow the steps below

1. • On the disk on which SharePoint Products and Technologies is installed, change to the following directory: %COMMONPROGRAMFILES%\Microsoft Shared\Web server extensions\12\Bin.
   • Type the following command, and then press ENTER:

stsadm -o renameserver -oldservername <name of the source database server> -newservername <name of the destination database server>

2. Repeat steps a and b for each of the servers in the farm.

For Named Instance:

Use the following procedure if you are using a named instance of SQL Server:

1. Start the SQL Server Native Client Network Utility, click start, Run, type “CLICONFG.EXE” and Click OK.
2. On the General tab, verify that TCP/IP is enabled.
   On the Alias tab, click Add.

The Add Network Library Configuration dialog box appears.

1. • In the Server alias box, enter the name of the current instance of SQL Server i.e. “OLDSERVERNAME”.
   • In the Network libraries area, click TCP/IP.
   • In the Connection parameters area, in the Server name box, enter the new server name and instance to associate with the alias, i.e. “NEWSERVERNAME\SERVERINSTANCE “
   • Uncheck the dynamically determine port and enter the port  number as 2433.

Note: By default named instances uses port 2433, please check with the SQL Team.

************************************************************************

Important Note: Create Alias for the all the server names recorded at the beginning which runs Windows SharePoint Services Database Services

***********************************************************************

2. Repeat steps on all servers in the farm that connect to SQL Server.

9. Start the new farm by performing the following steps:

1. In the Services snap-in, start the following services:
   • Microsoft Single Sign-On service
   • Office Document Conversions Launcher service (optional)
   • Office Document Conversions Load Balancer service (optional)
   • Office SharePoint Server Search service
   • Windows SharePoint Services Administration service
   • Windows SharePoint Services Search service
   • Windows SharePoint Services Timer service
   • Windows SharePoint Services Tracing service
   • Windows SharePoint Services VSS Writer service (optional)
2. At the command prompt, type iisreset /start.
3. Repeat steps on each server in the farm.

10. Test the process has been completed successfully

• By Opening Central Admin Site
• By Opening the Individuals SSP Admin Site
• By Opening Individual Web Applications
• By Opening Individual Site Collections which is not mandatory but would be a good test.

Windows Server 2008 and WSS 3.0

If you are planning to install Windows SharePoint Services (WSS) 3.0 on Windows Server 2008, then you should install at least WSS 3.0 with SP1 executable. You will not be able to install just WSS 3.0, Windows Server 2008 requires a minimum Sp1.

You can get the WSS 3.0 64 bit SP2 here and

WSS 3.0 32 bit Sp2 here

Go to Windows Server 2008 Resource Center for SharePoint Products and Technologies for more information

Volume Shadow Copy error after deleting an Unused SharePoint Service Account

Recently, I deleted an Unused SharePoint Service Account in AD after figuring out we were not using that service account for any Service in SharePoint. This service account was also local admin on the server.  As I did this I could not start the Volume Shadow copy service and whenever I try to start it I would get initiation error message and the application log would have the following error

Event Type:    Error
Event Source:    VSS
Event Category:    None
Event ID:    12289
Date:        6/23/2009
Time:        8:59:06 AM
User:        N/A
Computer:    ServerName
Description:
Volume Shadow Copy Service error: Unexpected error LookupAccountName( NULL, [ServiceAccountName], NULL, p, NULL, p, p).  hr = 0x800706fc.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:

Blah blah Blah…..

I tried rebooting the server, same error reappeared.

I was not able open the tab Shadow Copies  from properties of any disk drive or through diskmgmt.msc

I tried to see if any of the App Pools were using the same account I deleted and if that was causing the problem [If this is the case, simply change the old service account and it should fix the problem], that was not the case!

After research, I found that VSS actually stores the local admin account in the registry key under

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\VssAccessControl

I deleted the entry for the old service account I deleted in AD and VSS started working fine.

Note: This issue might also occur when you move a server to new domain and VSS would have the old domain account. Just update the registry key and it should fix the problem!

Sync Errors after Dropping and Adding SharePoint Databases

We recently upgraded to SP2 in our environment, during the installation we had the drop the content databases through Central Admin and Added them back using STSADM. As we did this we ended up with following sync errors with event id’s 5553 and 7888.

These errors might also occur when you drop and add content databases without doing a preparetomove on it!

I fixed these errors by following simple steps below

• Fist of all I identified the web application that is having Sync issues. (If you do not know which web app  it is, you can identify it with web app GUID from the first error and Comparing it the GUID with the upgrade log in 12 hive)
• The I ran the following the STSADM command with top level site collection url

stsadm -o preparetomove -url [http:\\testsite.abcompany.com\]

• I ran preparetomove command for all the site collections which are part of this web application

stsadm -o preparetomove -url [http:\\testsite.abcompany.com\sites\sitecollectionname]

• After I ran preparetomove command on all site collections in the web application, I ran the sync comman to run the syncronization job

stsadm -o sync

• Just to double check, I ran the following command to check to see if all content db’s are in sync

stsadm -o -sync -listolddatabases 5

• I got a response as “no databases match the criteria for this shared service provider ” which indicates that all my content db’s are in sync.

This fixed my errors in the application log!

Bug with MOSS 2007 SP2 – 180 Days Expiration

Recently MS came out with SP2 for MOSS 2007 and WSS 3.0

WSS 3.0 SP2
http://support.microsoft.com/kb/953338

MOSS 2007 SP2
http://support.microsoft.com/kb/953334

But there is a BUG with SP2, Once you install SP2 on MOSS 2007, product expiration date is activated and your license is set to expire in 180 days i.e. your license is expired in 180 days!

You can check this by going to:
Central Admin
Operations
Convert License Type
Under Current License type you see Office SharePoint Server Trial with Enterprise (or Standard) Client Access license

Fix:
As of 05/26, MS has not come out with a hotfix
You can fix this issue by reentering your product key and your license is converted from trial to enterprise or standard!

Check out the KB from MS

http://support.microsoft.com/kb/971620