How to upgrade the CU level of an On-Premises Exchange server?

This will be a straight forwarded blog post where I’m going to explain on how we can upgrade On-Premises Exchange servers CU level. Upgrading the CU level in general has been a challenging task since it’s not the same as traditional Windows updates. If you are looking for guidance on how to upgrade Exchange servers that don’t have the latest CU level installed. I will be sharing my methodology on how we can do this. This requires careful planning and a solid change management process before we can upgrade the CU level of an Exchange server. During this example, we will be using Exchange 2016.

Quick way to verify the installed Cumulative Update is by running the following command in PowerShell:

function Get-ExchangeCU() {
    Get-ItemProperty "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\*" | ? DisplayName -Match "Microsoft Exchange Server \d{4}" | Select -ExpandProperty DisplayName
}
Get-ExchangeCU

Another preferred way would be to use the Exchange Server Health Checker script. The Exchange Server Health Checker script helps detecting common configuration issues that are known to cause performance issues on an Exchange server and will also verify the latest CU. This script can be found here: https://microsoft.github.io/CSS-Exchange/Diagnostics/HealthChecker/

Microsoft’s official documentation keeps track of all the latest CU level that are available and were released for Exchange servers. This can be found here: https://learn.microsoft.com/en-us/exchange/new-features/build-numbers-and-release-dates?view=exchserver-2019.

We can see that this Exchange server is an Exchange 2016 CU12. This means that this Exchange server has not been upgraded for at least upgraded 3 years.

Best Practices

Microsoft has a set of best practices that they are recommending before upgrading to the latest CU level. This can be found here as well: https://learn.microsoft.com/en-us/exchange/plan-and-deploy/install-cumulative-updates?view=exchserver-2019

  • Always keep your servers as up to date as possible. This especially applies to the installation of a new server.
  • Always install the latest Cumulative Update when creating a new server.
  • There is no need to install the RTM build or previous builds and then upgrade to the latest Cumulative Update. This is because each Cumulative Update is a full build of the product.
  • Reboot the server beforehand.
  • Test the new update in a non-production environment first to avoid any problems in the new update affecting the running production environment.
  • Have a tested and working backup of both the Active Directory and your Exchange Server.
  • Backup any and all customizations. They will not survive the update.
  • Use an elevated command prompt to run the Cumulative Update.
  • Temporarily disable any anti-virus software during the update process.
  • Reboot your server upon completion of the update.
  • Test your Active Directory Replication Health status and make sure that every DC in the domain is replicating fine.

Verify Exchange Server Schema Version

Log in to an Exchange server or Domain Controller that contains the RSAT PowerShell module. Run the following script to gather information of what Exchange Schema version is currently in use. This script can be found here: https://www.alitajran.com/wp-content/uploads/scripts/Get-ADversions.ps1

# Exchange Schema Version
$sc = (Get-ADRootDSE).SchemaNamingContext
$ob = "CN=ms-Exch-Schema-Version-Pt," + $sc
Write-Output "RangeUpper: $((Get-ADObject $ob -pr rangeUpper).rangeUpper)"

# Exchange Object Version (domain)
$dc = (Get-ADRootDSE).DefaultNamingContext
$ob = "CN=Microsoft Exchange System Objects," + $dc
Write-Output "ObjectVersion (Default): $((Get-ADObject $ob -pr objectVersion).objectVersion)"

# Exchange Object Version (forest)
$cc = (Get-ADRootDSE).ConfigurationNamingContext
$fl = "(objectClass=msExchOrganizationContainer)"
Write-Output "ObjectVersion (Configuration): $((Get-ADObject -LDAPFilter $fl -SearchBase $cc -pr objectVersion).objectVersion)"

Another example would be to use dsquery.exe to query the RangeUpper and ObjectVersion. You can compare the object versions you see with the values to verify that Exchange successfully updated Active Directory during the installation.

dsquery * "CN=ms-Exch-Schema-Version-Pt,CN=schema,CN=configuration,DC=Contoso,DC=com" -scope base -at
tr rangeUpper
dsquery * "CN=Microsoft Exchange System Objects,DC=Contoso,DC=com" -scope base -attr objectVersion

Since we are talking about Exchange 2016. I’ve found a great blog post that currently tracks all the Schema Update versions for Exchange. This can be found here: https://www.alitajran.com/exchange-schema-versions/#h-exchange-server-2016-schema-versions. Since I’m using Exchange 2016 CU12, we can see that the objectVersion (Default) has been set to 13236. Our goal is to upgrade to the latest Exchange CU level, which means that if everything goes according to plan. The objectVersion attribute should be updated to 13243.

Verify Exchange Server Health

Great PowerShell script of @practical365 that will perform different Exchange Health Checks to see if everything is running fine. Let’s check this first, before we are going to upgrade to the latest CU level. Save this PowerShell script as Test-ExchangeServerHealth.ps1 and then run it on an Exchange server. It’s good to verify that your Exchange servers are in a healthy state before upgrading. This script can be downloaded here: https://github.com/cunninghamp/Test-ExchangeServerHealth.ps1

At the result, we can see that there are two Exchange servers in the environment. Both of them have passed all the checks, which is a good sign to proceed further.

Run Microsoft Exchange Health Checker script

Run the Microsoft CSS Exchange Health Script, which can be found here: https://microsoft.github.io/CSS-Exchange/Diagnostics/HealthChecker/

The Exchange Server Health Checker script helps detect common configuration issues that are known to cause performance issues and other long running issues that are caused by a simple configuration change within an Exchange Environment. It also helps collect useful information of your server to help speed up the process of common information gathering of your server. It’s similar to the previous script, but it looks at a few different things.

The required permission(s) to run this script are Local Admin on an Exchange server and a member of the Organization Management group. Start reviewing the following checks:

  • Verify if the SSL certificate has been expired or not
  • Verify whether .NET Framework 4.8 is installed or not

Upgrading to the latest Exchange 2016 CU23 level requires to have .NET Framework 4.8 installed. It is good to do this prior to upgrading the CU level.

  • Verify if all Exchange Web App Pools are starting

Install IIS Rewrite Module on Exchange servers

IIS URL Rewrite 2.1 enables Web administrators to create powerful rules to implement URLs that are easier for users to remember and easier for search engines to find. This feature is required when installing the latest Exchange 2016 CU, which happens to be by the time of writing this. CU23. If we don’t do this, it will throw the following an error.

IIS Rewrite Module can be installed here: https://www.iis.net/downloads/microsoft/url-rewrite

Ensure Exchange Servers have installed the latest Windows updates

Make sure that Exchange servers that we’re going to upgrade to the latest CU level have all the latest Windows updates installed. Once the Windows updates have been installed, make sure that the server has been rebooted beforehand.

Upgrading Exchange servers to the latest CU level

By the time of writing this. Exchange 2016 CU23 was the latest version for Exchange 2016. First, let’s download this ISO from the official Microsoft website. See: https://www.microsoft.com/en-us/download/details.aspx?id=104132

Now login to your Exchange server and make sure that the account has Enterprise Admins, Schema Admins, and Local Admin on all the Exchange servers.

  1. Put the HubTransport in draining state

Hub Transport is a role that handles all the mail flow, and we are going to put it in draining state, so it will stop accepting messages.

Run the following command as admin in Exchange Management Shell:

Set-ServerComponentState -Identity "Exchange" -Component HubTransport -State Draining -Requester Maintenance

2. Redirect all the messages that are in the queue to another Exchange server

At this step, we are going to redirect all the messages that are in the queue to another Exchange server. The target Exchange server needs to be typed out as a FQDN and also should NOT be in maintenance mode. A queue is a temporary holding location for messages that are waiting to enter the next stage of processing or delivery to a destination, so this means that all the messages that will now be redirected to Exchange02.contoso.com.

Run the following command as admin in Exchange Management Shell:

Redirect-Message -Server "Exchange" -Target "Exchange02.contoso.com"

3. Run this command if the Exchange server is part of a DAG group. If not, skip to step 8

The Suspend-ClusterNode cmdlet suspends activity on a failover cluster node, that is, pauses the node. Pausing, or suspending, a node is usually done when applying software updates to the node.

Run the following command as admin in Exchange Management Shell:

Suspend-ClusterNode "Exchange"

At this example, we can see that the Exchange server we’re upgrading is not part of the DAG group. This means that we can skip right away to step 6. If it returns any result for you, keep going with step 4 and 5.

4. Disable database copy auto-activation and move the active copy of the database to another DAG

The -DatabaseCopyActivationDisabledAndMoveNow parameter specifies whether to prevent Mailbox databases from being mounted on this server if there are other healthy copies of the databases on other Mailbox servers. It will also move any mounted databases on the server to other DAG members if available.

Run the following command as admin in Exchange Management Shell:

Set-MailboxServer "Exchange" -DatabaseCopyActivationDisabledAndMoveNow $true

5. Block DatabaseCopyAutoActivationPolicy

Ensure that Mailbox databases doesn’t become active.

Run the following command as admin in Exchange Management Shell:

Set-MailboxServer "Exchange" -DatabaseCopyAutoActivationPolicy Blocked

6. Verify if there are any database copy still being mounted

Verify whether there are still any database copies being mounted. This command should return no results.

Run the following command as admin in Exchange Management Shell:

Get-MailboxDatabaseCopyStatus -Server "Exchange" | Where {$_.Status -eq "Mounted"}

7. Check the Transport Queues

The queues should be empty or almost empty. Any e-mails in the queues will have a delay from being delivered when performing this operation.

Get-Queue -Server "Exchange"

8. Put the Exchange Server into maintenance mode

Set-ServerComponentState "Exchange" -Component ServerWideOffline -State Inactive -Requester Maintenance

9. Verify if Exchange server is maintenance mode

Get-ServerComponentState "Exchange" | Select Component, State

10. Restart the Exchange server

Open CMD as an admin and type the following command:

shutdown /r

11. Mount the ExchangeServer2016-x64-CU23 ISO file

First, make sure that you have Enterprise Admins and Schema Admins privileges. The second thing is to go to the location the ISO was mounted.

In this example, we have mounted the ISO now. Everything is stored in the F:\ drive in my example.

Open PowerShell as an admin and in my example. I can see that it has been mounted in the F:\ drive, so I have to go that path.

cd F:\

12. Extending the Active Directory Schema

We only need to do this once, so the next server that we are going to upgrade to the latest CU level. We can skip this part, since we already have extended the schema. Make sure to have Enterprise Admins and Schema Admins privileges when doing this action.

\Setup.exe /IAcceptExchangeServerLicenseTerms_DiagnosticDataOff /PrepareSchema

13. Verify AD Replication status

After Setup finishes extending the schema, you’ll need to wait while Active Directory replicates the changes to all of your domain controllers before you proceed. This is recommended by Microsoft: https://learn.microsoft.com/en-us/exchange/plan-and-deploy/prepare-ad-and-domains?view=exchserver-2019

Run the following command:

repadmin /showrepl

This command displays the GUID of each object that was replicated and its result, which can be helpful to identify what objects are failing to replicate.

14. Prepare Active Directory

After the Active Directory schema has been extended, you can prepare other parts of Active Directory for Exchange. During this step, Exchange will create containers, objects, and other items in Active Directory to store information.

Run the following command in PowerShell:

\Setup.exe /IAcceptExchangeServerLicenseTerms_DiagnosticDataOff /PrepareAD

15. Prepare Active Directory Domains

The final step is to prepare the Active Directory domain where Exchange servers will be installed or where mail-enabled users will be located. This step creates additional containers and security groups and sets the permissions so Exchange can access them.

If you have multiple domains in your Active Directory Forest, you have the following choices in how to prepare them:

  • Prepare all domains in the Active Directory Forest | /PrepareAllDomains
  • Choose the Active Directory domains to prepare | /PrepareDomain

In this example, we will do it for all the domains in the AD Forest.

Run the following command in PowerShell:

\Setup.exe /IAcceptExchangeServerLicenseTerms_DiagnosticDataOff /PrepareAllDomains

16. Install Cumulative Updates via Unattended Setup

Run the following command if you don’t want to avoid sending diagnostic data to Microsoft. If you do want to send diagnostic data to Microsoft, you can use the /IAcceptExchangeServerLicenseTerms_DiagnosticDataON option.

Run the following command in PowerShell:

\Setup.exe /Mode:Upgrade /IAcceptExchangeServerLicenseTerms_DiagnosticDataOFF

It can take a while before everything has been completed, but this how it looks like.

17. Restart the Exchange server again

shutdown /r

18. Run the HealthChecker.ps1 script again and let’s verify if everything succeeded

Run the HealthChecker.ps1 script in the Exchange Management Shell. We can see that we were able to successfully upgrade it to the latest CU level.

19. Remove the Exchange server from maintenance mode

Open Exchange Management Shell and run the following command as admin.

Set-ServerComponentState "Exchange" –Component ServerWideOffline –State Active –Requester Maintenance

20. Remove HubTransport from maintenance mode

Set-ServerComponentState "Exchange" –Component HubTransport –State Active –Requester Maintenance

21. Resume cluster mode and ensure mailbox database is active

Resume-ClusterNode –Name "Exchange"
Set-MailboxServer "Exchange" -DatabaseCopyAutoActivationPolicy Unrestricted
Set-MailboxServer "Exchange" -DatabaseCopyActivationDisabledAndMoveNow $false

22. Re-run the Test-ExchangeServerHealth.ps1 to see that there are no issues

Ensure that the upgraded Exchange server has no issues.

References

One comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s