Exchange 2013

ganeshg1990

Caution ! Exchange 2013 CU18 Security update (OWA Vulnerability)

Upgrading Exchange 2013 CU18 had few challenges when you have too many 3rd party applications running on the exchange servers, In my previous blog you would have seen the show spoiler being identified as Mcafee host intrusion service. Likewise I would also want to bring this to your notice that you should be slightly cautious when installing
Security Update For Exchange Server 2013 CU18 (KB4045655).

Not sure how many went through this caution note on the technet blog where it stresses the importance of running this security update with elevated permissions (run as administrator). I personally experienced this as I ran it just with a double click on the file, post which we had some issues in connecting to our OWA & ECP. Upon investigation its observed that most of the resources weren’t present in the OWA directory.

Here are the known issues reported when this Security patch is installed (ref: https://support.microsoft.com/en-ca/help/4045655/description-of-the-security-update-for-microsoft-exchange-december-12)

  1. We are aware of some reports that Exchange services may remain in a disabled state after you install this security update. If this occurs, the update is installed correctly. However, the service control scripts encounter a problem when they try to return Exchange services to its usual state. To resolve this issue, use Services Manager to restore the startup type to Automatic, and then start the affected Exchange services manually.
  2. When you try to manually install this security update in “normal mode” (not running the update as an administrator) and by double-clicking the update file (.msp), some files are not correctly updated. When this issue occurs, you do not receive an error message or any indication that the security update is not correctly installed. Also, Outlook Web Access (OWA) and the Exchange Control Panel (ECP) may stop working. This issue occurs on servers that are using UAC (user account control). The issue occurs because the security update does not correctly stop certain Exchange-related services. To avoid this issue, run the security update in elevated mode as an administrator. To do this, right click the update file, and then click Run as administrator.

In case if you already installed this, remove the security update from the server completely and post a reboot install the security update again with the elevated permission.

Cheers,
Ganesh G

ganeshg1990

Exchange 2013 CU18 Upgrade issues

download

 

Environment:

Exchange 2013 CU13
.NET Framework 4.7
Windows 2012 R2
Physical Servers
4 Node DAG

Schema Changes:

Active Directory schema changes in Exchange 2013 cumulative updates CU8 and later

No changes have been made to the Active Directory schema in Exchange 2013 from CU8 onwards. The last cumulative update to include schema changes is currently Exchange 2013 CU7.
https://technet.microsoft.com/en-us/library/bb738144%28v=exchg.150%29.aspx#CU7

Confirm the schema versions as listed below,

Exchange

So there are no Schema changes in CU18 as well

Downloads:

1. Download Exchange 2013 CU18 from the link below:
https://www.microsoft.com/en-us/download/details.aspx?id=55955

2. Get .NET Framework 4.6.2 installation packages (Offline Installer)
Download .NET Framework 4.6.2 offline package from the link below:
https://www.microsoft.com/en-us/download/details.aspx?id=53344

3. Security Update For Exchange Server 2013 CU18 (KB4045655)

https://www.microsoft.com/en-us/download/details.aspx?id=56329

Plan:

1. Get the general rechecks done on the exchange servers
2. Microsoft doesn’t recommend .NET Framework 4.7 for CU18 and the same is published in the TechNet blog below,
https://blogs.technet.microsoft.com/exchange/2017/06/13/net-framework-4-7-and-exchange-server/
Hence remove 4.7 from the server and install 4.6.2
Note: In our case our .NET Framework failed back to 4.6.2 (Windows 2012 R2 comes with .NET 4.6.2

3. Turn off all the 3rd party services which can interrupt the installation, Don’t have mercy on any of them.
Be brutal in stopping anything and everything other than Microsoft services,

I am stressing this because, it was a night mare when we had to delay our installation by 8 hours as Mcafee was disrupting the upgrade from nowhere.

Our installation failed with the below error when configuring the transport services,

Processing component ‘Transport Common Configuration’ (Configuring common Transport properties.).
Executing:
$dllFile = join-path $RoleInstallPath “bin\ExSMIME.dll”;
$regsvr = join-path (join-path $env:SystemRoot system32) regsvr32.exe;
start-SetupProcess -Name:”$regsvr” -Args:”/s `”$dllFile`”” -Timeout:120000;

Active Directory session settings for ‘Start-SetupProcess’ are: View Entire Forest: ‘True’, Configuration Domain Controller: “DC”, Preferred Global Catalog: ‘DC’, Preferred Domain Controllers: ‘{ DC }’
User specified parameters: -Name:’C:\windows\system32\regsvr32.exe’ -Args:’/s “E:\Program Files\Microsoft\Exchange Server\V15\bin\ExSMIME.dll”‘ -Timeout:’120000’
Beginning processing start-SetupProcess
Starting: C:\windows\system32\regsvr32.exe with arguments: /s “E:\Program Files\Microsoft\Exchange Server\V15\bin\ExSMIME.dll”
Timed out waiting for process to complete
Timed out waiting for process to complete
Ending processing start-SetupProcess
The following 1 error(s) occurred during task execution:

How we fixed it ?
After several investigation , we identified that the McAfee Host intrusion Prevention was the cause for this time out and the Microsoft Registry server was crashing every time when it reached this stage of upgrade (70%)

PS: McAfee Host intrusion Prevention can be stopped via the Orchestrator or by a elevated permission which has to be provided to you by your respective server management team.

We stopped the McAfee Host intrusion Prevention service and re initiated the upgrade following a reboot.
Upgrade completed successfully & installed the security update for CU18 as well.

Should you have any further questions, please feel free to drop in your questions.

Regards,
Ganesh G

ganeshg1990

Simple Script to get the config file values – Exchange 2013

In a large sized environment, its a tedious job to verify the config files for any specific values, It takes a lot of manual efforts and a utter waste of time.

Just to make it simple, came up with this very basic script that can fasten this task (Still this can be enhanced 🙂 ),

  1. Create a file named Server.txt which should have the list of servers where you would want to check the config files
  2. If you wish you can even change the path as per your convenience.
  3. In this example, we tried looking for the maxRequestLength, MaxDocumentDataSize & MaxRequestLength in the web.config file.
  4. Based on your requirement, you can alter the path, file and values.

Please leave your comments or questions below.

Regards,

Ganesh G

#Script to Simple Script to get the config file values – Exchange 2013

#=========================================================

#Ganesh G

$server = Get-content “C:\Troubleshooting\Server.txt”
Foreach ($Server in $Server)
{
Write-host “$server”
Write-host “From: Exchsrvr\ClientAccess\Sync\web.config”
Select-String “\\$server\d$\Exchsrvr\ClientAccess\Sync\web.config” -pattern “maxRequestLength” | Format-List “Line”
Select-String “\\$server\d$\Exchsrvr\ClientAccess\Sync\web.config” -pattern “MaxDocumentDataSize” | Format-List “Line”
Write-host “From: Exchsrvr\FrontEnd\HttpProxy\sync\web.config”
Select-String “\\$server\d$\Exchsrvr\FrontEnd\HttpProxy\sync\web.config” -pattern “maxRequestLength” | Format-List “Line”
}

ganeshg1990

Mailbox moves – Exchange 2013 Troubleshooting

Having issues during mailbox moves in Exchange 2013?

Few quick ways to check your mailbox move performance

What are the major factors involved/impacted during a mailbox move?

  • Disk IO
  • Network Bandwidth
  • Size of the mailboxes moved
  • No. of active connections on the source and the destination databases
  • MRS throttling

Ideally every individual who is troubleshooting a mailbox moves will have the initial probing done and that has to be in place before proposing a workaround or fix.

If Disk I/Os are not optimal, we might have severe effects on the mailbox moves resulting in read/write latency. Ensure that the disk I/O read/write speed is optimal (0.020 ms is considered optimal), Also ensure the disk drivers are up to date.

In order to get a quick insight about your mailbox move performance make use of the mailbox migration performance analysis script. This script provides you with all the necessary information,

Mailbox migration Performance analysis script

  • It provides the amount of data transferred (in GB)
  • Duration of the migration batch & also the idleness info
  • Best part is that, this also reports latency which will help us take a proactive call to suspend the moves before it hit the overall disk/server performance.
  • Also its easy for reporting and stuff

 

% of data moved ( GB).

clip_image002

Script used to extract the above statistics and the procedure is explained in the below TechNet Blog,

https://blogs.technet.microsoft.com/exchange/2014/03/24/mailbox-migration-performance-analysis/

Good to know information,

In order to gauge the load on the exchange 2013 servers we need to get the number of active connections (OWA, ActiveSync, RPC). I have not covered EWS connections in this blog,

Get-Counter “\MSExchange OWA\Current Unique Users” -ComputerName “Servername1, Servername2”

Get-Counter “\MSExchange RpcClientAccess\User Count” -ComputerName “Servername1, Servername2”

Get-Counter “\MSExchange ActiveSync\Current Requests” -ComputerName “Servername1, Servername2”

Stages of mailbox moves:

Source enumeration  > Initial data sync > Incremental Sync > Finalization

I will post a detailed blog on each stage of the mailbox moves in near future 🙂

Other possible fixes:

  • Modify the MRS config file

MSExchangeMailboxReplication.exe.config, ‘C:\ProgramFiles\Microsoft\Exchange Server\V15\Bin’

MaxActiveMovesPerSourceMDB=”20″

MaxActiveMovesPerTargetMDB=”20″

MaxActiveMovesPerSourceServer=”100″

MaxActiveMovesPerTargetServer=”100″

MaxTotalRequestsPerMRS=”100″

ExportBufferSizeKB=”512″

Also can increase MaxConcurrentMigrations and set that to 100 for a fair mailbox move performance

  • Increase the ExportBufferSizeOverrideKB 

This way we are adding moving more data at a time which is best supported in a network which has high latency networks >100ms

This reduces the initial data sync duration which covers the 70% of the move process

Note: Buffer size cannot go beyond 10240. but its recommended to apply this stage by state and the value has to be tested gradually. Also there are high chances of network chocking.

  • Disable indexing on the source & Destination databases

 

Personally I wouldn’t recommend to do this, as this has other adverse effects 🙂

This is not it, we have a lot more to look at based on the nature of the issue. So drill it down to the core to find the best solution.

Same is also available in Technet Wiki,

IMG_20161113_164927

Ganesh G

ganeshg1990

RBAC – Custom Management Role – Active Sync Mailbox policy

In this post we have a custom management role to restrict users from creating Activesync mailbox policy.

Below are the management role entries for the Recipient Policies Management role (responsible for creating a new active sync mailbox policy)

WP1

Created a new custom RBAC role named “Modified Recipient Policy” and removed the entries which needs to be removed

WP2

WP4

WP3

Then logged in to the exchange server as the test user account (RBACTest) – This user wasn’t able to create a new active sync policy.

Note: This RBAC permission takes effect only after you logout and login (Thank you Andrew for pointing this  🙂 )

WP5

Regards,

Ganesh G

Video Blog available >  https://www.youtube.com/embed/Js9tdRfECus” target=”_blank”>Start Exchange

ganeshg1990

OAB Changes -Exchange 2013

Just a Quick look at the changes made in Exchange 2013 with respect to the Offline address book, 

Older versions of exchange had One Generation server (Mailbox server), But this is discontinued in Exchange 2013, Instead Organization mailbox or Arbitration mailboxes are brought into picture.

  1. Arbitration mailbox are capable of Provisioning the OAB,
  2. Only OAB Version 4 is available,
  3. Outlook 2K7 SP3 RU10 and above are supported client versions to access OAB from Exchange 2013
  4. Mailbox assistance service is responsible for generating the OAB
  5. Offline address book is Stored in Arbitration mailbox
  6. Command to get the list of arbitration mailbox which is capable to be involved in the OAB function,

 

Get-mailbox -arbitration | where {$_.PersistedCapabilities -Like “*oab*”}

 

  1. “OrganizationCapabilityOABGen” marks it as a generation
  2. No FDS – File Distribution service is used in CAS to pull the OAB files
  3. FDS Schedule is not flexible enough
  4. No more generation server

 

Quick work flow,

 

How it works in Exchange 2013 :

 

  • Outlook receives OAB URL from Autodiscover and reaches designated CAS 2013 through OAB URL.
  • Performs initial authentication for OAB.
  • Queries Active Directory and determines the closest Organization Mailbox for the requesting user.
  • Queries Active Directory again to determine the mailbox database hosting the Organization Mailbox.
  • Queries the Active Manager to determine the mailbox server where the mailbox database is active (mounted).
  • Proxies the request to the Mailbox server identified in step 4.
  • Retrieves OAB files and passes them to the client.

 Key points to note:

During the First Request from the client

Arbitration mailbox copies the oab in to the CAS fodler,

Going further all the connections are made to the path where the oab is placed in CAS,

Then in case of rebuild use

 

Update-offlineaddressbook

 

Set-mailbox <Mailbox identifier> -arbitration -oab $true (Create a new arbitration mailbox)

 

Specify a oab for a mailbox

Set-mailbox “mbx name” -Offlineaddressbook <Oab name>

“MSexchangeuSeoab” attribute to be set

 

More detailed information is available in the below links, Feel free to post in your queries here i will try my best to answer your questions, Have a good one !

http://blogs.technet.com/b/exchange/archive/2012/10/26/oab-in-exchange-server-2013.aspx

http://exchangemaster.wordpress.com/2013/04/02/beware-full-oab-downloads-after-installing-1st-exchange-2013-server-in-existing-0710-environment/