New Store, new memory tuning techniques required

One of the more interesting observations that we will soon be able to make is how well the thoroughly overhauled and rewritten Exchange 2013 Information Store functions in production environments. Of course, there’s the small matter of the rest of the deployment prerequisites to be met, such as co-existence with legacy versions, what to do about clients and third party products, and so on. But all of that should not take away from contemplating the Store.

A very good reason has to exist before any software development group will take on the rewrite of a core component. From an Exchange perspective, the good health and reliable functioning of the Store is of primary concern. After all, you don’t want to introduce problems by rewriting code just for fun, to use the latest and greatest programming language or application framework, or for any other reason dreamed up by management (cue thoughts of Dilbert’s curly-haired boss…). This is especially important when considerable effort has been dedicated to upgrading the Store, which Microsoft did pretty successfully in Exchange 2010 through a schema refresh, the introduction of the Database Availability Group, and so on.

The rewrite that the Exchange 2013 developers embarked upon converted the Store into managed code, using C# for the purpose. Despite an ongoing clamour to move Exchange to SQL, the underlying ESE database engine remains in place, simply because it’s the best tool for the job. Above ESE, a set of Store processes run instead of a single monolithic process. Each database is served by a worker process, and all of the worker processes are coordinated by Store.exe. According to Microsoft, the new Store is more tightly integrated with the mailbox replication service (MSExchangeRepl) to ensure that the replication to database copies progresses more smoothly and that database activations are faster.

One of the side-effects of the rewrite is that the Store contains fewer lines of code. When you rewrite code that has been around for a long time, you invariably find code that is no longer required, code that was inserted to handle a long-forgotten condition, or code that might have been required to work around a problem that can no longer occur. And so it was with the Store rewrite. Many lines of unwanted and unused code have been consigned to the wastebasket and Exchange now has a Store that’s been through the equivalent of a programming colonic irrigation.

In practice, you’re probably not going to be aware that a rewrite has happened, unless of course you poke around under the hood and look to see what processes are active on an Exchange server. And then you might see something like what’s shown below, which comes from a multi-role Exchange 2013 server that supports four databases. You can see the central Store process and four worker processes, one for each database. The SharePoint Search component processes shown at the bottom of the screen shot are those that index content in the databases. They’re shown as SharePoint because the Search Foundation (aka FAST) code “belongs” to SharePoint and is shared with Exchange to create a common indexing and search capability across both repositories. The search processes are memory and CPU hungry and can occupy 10% of system resources quite easily.

Store processes running on an Exchange 2013 multi-role server

Which brings us to system tuning. The “old” Store process had an endearing capacity to seize as much memory as it could on a server using a process called dynamic buffer allocation (DBA). The idea being that caching as much data in memory was a good thing because it avoided expensive I/O to disk. Exchange’s DBA scheme worked well, but could cause problems for smaller servers where Exchange was just one of the applications in the mix and it wasn’t good for Exchange to be quite so demanding in terms of memory.

A long time ago, back in the mists of time when Exchange 2000 was the most modern email server you could imagine, Microsoft published KB266768  “How to modify the Store database maximum cache size”. Essentially, you practise brain surgery on Active Directory with ADSIEdit to modify the properties of the Information Store object to set the msExchESEParamCacheSizeMax value for a server, which controls the maximum size of the Store cache. This technique works for Exchange 2000, Exchange 2003, Exchange 2007, and Exchange 2010 (as explained here, it’s slightly different for Exchange 2010 SP1 onwards) and is often applied to servers running Windows Small Business Server and similar configurations.

Unsurprisingly (given the Store rewrite), the ESE cache maximum and minimum size properties don’t exist by default on Exchange 2013 servers. It might be that the new Store is much better at controlling memory demands than the old. Certainly, the Store worker processes appear to demand less memory in total than the older monolithic process. But we’re only at the start of understanding how to manage Exchange 2013 and I’m sure that new ideas and suggestions will appear as time goes by.

Change always brings disruption. It seems like this old technique from Exchange 2000 is no longer viable – unless of course someone knows different?

Follow Tony @12Knocksinna