SQL, OFS, Exchange file system, WinFS, and other murky dealings

Like many people who are interested in Microsoft Exchange, I have followed the ESE versus SQL debate with some interest for as long as it’s been going on, or just about the entire lifetime of the product. ESE stands for Extensible Storage Engine (also known as JET-Blue, the Joint Engine Technology) and is the database engine on which the Exchange Information Store runs.

A recent post by Hal Berenson, a retired Microsoft Distinguished Engineer, illuminated some of the murky corners of the internal debate that occurred between different engineering factions in Microsoft in the late 1990s and early “naughties” as the company grappled with the challenges of unified storage. It’s worth the read, if only to learn about how technology sometimes evolves.

People have asked why Exchange doesn’t use SQL for years. On the surface, it does not make sense for Microsoft to have two competing database engines. Combining all resources to focus on the development and exploitation of a unified database engine would liberate a lot of engineering resources, reduce the need for product support, save money, and accelerate the development of new features, or so the convenient wisdom goes.

Of course, a lot of work would be required to move a product like Exchange from one database engine to another. The usual response from the Exchange development group is that they have experimented with SQL but that ESE continues to be the best choice for the kind of ad-hoc transactions typical of an email server. But technology has a habit of being adaptable to meet different needs, providing sufficient effort is expended, so I guess that it would be possible to create a SQL-based version of Exchange, should that decision be taken. For now, a change does not seem likely as the mantra is “The future is ESE. ESE is easy, SQL squeals like a pig.“, repeated by Microsoft speakers at events such as the late-lamented TEC and the Microsoft Exchange Conference.

But back to the past, it is fascinating to learn about the Object File Store (OFS) and how an early version of Exchange was supposed to move from ESE (described as an “interim store”) to use OFS. An Exchange General Manager (perhaps Brian Valentine) then revealed in email that he thought that Exchange would never move to OFS. Later on, the story evolves to JAWS (Jim Allchin’s Windows Storage), another candidate for the Exchange Store, and then a discussion about how Outlook might have used SQL Server 2000 (clearly not the full-fledged version) as a replacement for the PST structure that is still used today.

Lots of information about internal Microsoft politics is revealed. The struggle between Bob Muglia’s organization (including Exchange) and another (including SQL) is interesting, especially in how it influenced the development of the Exchange File System (ExIFS), a variant of which that shipped with Exchange 2000 and Exchange 2003 to provide the infamous “drive M:” that exposed the contents of the Exchange Store as a drive and encouraged weird behavior such as attempting to take file-level backups through drive M: – and be then disappointed when the backups could not be restored.

Later on, other efforts occurred to try and make development groups co-operate around a single vision of unified storage. SharePoint, described as “idiosyncratic” users of SQL, Outlook, Exchange, and Active Directory seemed to line up to work on a joint approach to move everything to SQL Server and then evolve towards a fully integrated unified storage system (later to become the WinFS project). During this period Exchange dabbled with SQL Server and even told partners that they would move to that platform in the “Kodiak” project. That never happened, possibly because Exchange then decided to wait for a server-based version of WinFS. As Berenson remarks “Apparently they are still waiting.”

As a partner working at Digital, Compaq, and HP, I followed the debate through many product briefings and other more informal (but highly informative) discussions. But outsiders can only ever have partial knowledge of what happens inside Microsoft as the more salubrious details are kept in-house. Hal Berenson’s story fills in a lot of the gaps. I suspect that many who currently work on Exchange don’t realize exactly what happened in the past. Hal’s position in the Microsoft engineering hierarchy equipped him with a unique perspective. I enjoyed reading his piece very much. You might too.

Follow Tony @12Knocksinna