Citer – good service for a change from a car hire company

Car hire companies, especially those who operate at airports, are often criticized for poor service and some pretty tacky habits, such as their desire to charge renters for a full tank of fuel when the rental commences even when the renter knows that they will only drive a few hundred kilometers and will not empty the tank. Or indeed the ever-popular allegation that the fuel tank was not full when the car was returned, necessitating an instant retrospective charge for a “fuel service”.

It’s important to recognize good service when you find it and this year we have rented cars three times from Citer/Enterprise at Nice Airport (NCE) and have received excellent service each time. The staff working at Citer have told us that their work practices have changed since Citer became part of the Enterprise Rent-A-Car organization in November 2011, so this might just be an example where an acquisition results in better customer service.

All is not perfect with the Citer experience as we have often experienced queues at their desks in the Nice Airport rental facility. Part of the reason for the delay in dealing with clients seems to be interminable phone calls made by the agents to all and sundry, seemingly a necessary part of the process to secure cars. But part of the delay is also due to the way that Citer staff walk customers to their car to check it out, explain how the car functions if necessary, and make sure that all existing damage is correctly noted. I’m sure that the Citer staff cover a lot of ground between rental desk and garage over the course of a day, but the impression made on customers is much better than the norm delivered by Citer’s competitors such as Avis and Hertz.

Another nice thing about Citer is that they offer many deals that include a second driver. Most of the other car hire firms that operate in Nice delight in

Citer’s current car fleet in Nice features a lot of Hyundais. We have driven a very nice i30 (it was brand new) but on the other hand, the last time out we had a tatty Lancia Delta that had suffered many dents and scratches in its 28,000 km rental career. I owned a Lancia Delta in 1982-83 and had fond memories of that car but its modern counterpart did not make the same impact and I doubt that we will choose a Delta again.

If you need to rent a car in Nice Airport in the near future, consider trying Citer. I usually use AutoEurope to find the best deals and scan down the (often) hundreds of packages to find the best one that’s available. Not that anything especially compelling will be available soon as the Cote d’Azur holiday season swings into full blast and car rental rates escalate.

Follow Tony @12Knocksinna

Exchange 2013 Inside Out: Mailbox and High Availability makes an appearance

The nice people at O’Reilly Media have posted details of book one of the two-part Exchange 2013 Inside Out set. My book covers the mailbox server and high availability while Paul Robichaux is deep in the process of writing all about the client access server, clients, and other wonderful topics including unified messaging. I’m sure that O’Reilly will get to putting up some details about his book soon.

We’ll soon be making preview chapters available. These are chapters that are in the midst of the editing process. As such, they might contain errors. In fact, I’ll guarantee that they do because, despite several reviews, eradicating errors from text is an ongoing process when the software changes, as Exchange 2013 did recently when CU1 was released. Paul and I are attempting to keep the text updated as Microsoft upgrades Exchange but, as you can imagine, this is not particularly easy.

In any case, it’s nice to see the book entering the final stages. Much of the writing is now done. All that has to happen is updates, fact checking, revisions, indexing, more copy editing, fights with the editors about page counts: the normal kind of thing that happens to bring out a book.

Follow Tony @12Knocksinna

Exchange 2013 CU1 appears

Exchange 2013 CU1 has now appeared. My review of the new release is available on WindowsITPro.com, where I have also posted some notes on the approach needed to update mailbox servers that are members of a Database Availability Group (DAG).

Happy Reading!

Tony

Exchange 2013: Stuck messages in OWA’s Drafts folder and DNS

One of the common things that OWA users notice about Exchange 2013 is that outgoing messages sometimes appear to get “stuck” in the Drafts folder. Not only do messages seem to linger in Drafts, no trace of the outbound messages ever shows up in the Outbox. Or so it seems, but really it’s an urban myth.

Exchange 2013 boasts a new architecture. The hub transport server role is no more and its processing has been subsumed into the mailbox server role. In turn, as the TechNet description of the Exchange 2013 mail flow makes clear, the Mailbox Transport service and the Transport service work together to process messages sent by clients.

Exchange 2013 Mail Flow (source: TechNet)

But how does the Drafts folder come into the picture? Well, OWA clients automatically capture copies of messages as they are being composed and store them in the Drafts folder. When the user issues a sent command, the Mailbox submit agent (running within the Store driver) takes over and processes the outbound message by giving it to either the Transport service running on the same mailbox server or to the Transport server running on another mailbox server. The connection is made via SMTP.

Messages stay in the Drafts folder until they are successfully sent by being processed by the transport service. At this point, items are moved into the Sent Items folder. OWA 2013 behaves in the same way as OWA 2010 – nothing has changed in the way that messages are held in the Drafts folder until dispatch. What might account for user descriptions of items being “stuck” is when a problem occurs somewhere in the transport pipeline that prevents outbound messages being processed.

For instance, items will remain in the Drafts folder if the Store cannot pass them to the transport system. If the transport service is not running on any available server or the mailbox transport service is not running on the mailbox server that hosts the active database for the user’s mailbox, items will stay in the Drafts folder until the services come online and Exchange is able to process outbound items.

Now, the normal state of events is that all of the Exchange services are running along quite happily on the server. Certainly, if a service fails or is not running for some reason, it’s likely that the administrator will notice that this is the case and fix the problem. What else would stop transport being able to process outbound messages and force the Store to keep them in the Drafts folder?

Checking DNS Lookup properties for a server with EAC

Incorrect DNS binding to server NICs is one of the likely culprits. Unless the Exchange 2013 servers know how to route messages, the items stay where they are. Like any email server, Exchange makes heavy use of DNS, so it’s logical that if DNS is not configured properly, then messages are not going to be transported to either internal or external destinations. If users report “stuck” messages, you might just want to take a look at server properties with EAC to make sure that DNS lookups point to the right place (a server can that resolve the lookups). You can also check with EMS by running the Get-TransportService cmdlet to retrieve the ExternalDNSServers and InternalDNSServers properties.

If the server properties reveal that DNS lookups are going walkabout, you’ve just found the problem. On the other hand, if the DNS configuration is correct, you might have to talk to Microsoft Support to see why transport isn’t working as expected. I hear rumblings that Microsoft has improved the way that Exchange interacts with DNS in Exchange 2013 CU1 but we shall have to wait for that release to verify if this is correct.

Outlook processes outbound messages differently because it does use the Outbox folder. Remember that Outlook can be a client for many different versions of Exchange and other email servers. Where OWA keeps messages in the Drafts folder, Outlook continues to do what it has done for years and moves outbound items through the Outbox en route to Sent Items. When messages are stuck in the Outbox, it’s probably due to another factor such as messages being too large for the server to accept. In effect, items in Outlook’s Outbox folder have the same status as items in OWA’s Drafts folder – both are candidates to become outbound items that will be processed by the transport service.

Messages don’t get stuck in the Drafts folder without good reason. It’s not as if Exchange wants to keep messages there. After all, it is an email server after all… and email servers that don’t send messages would not be much good!

Follow Tony @12Knocksinna

Microsoft CVP on the current state of Windows Phone and its competitors

Terry Myerson, who previously ran the Exchange development group, is now the Microsoft Corporate Vice-President (CVP) for Windows Phone. He has always had strong opinions and it came as no surprise that Terry would voice some trenchant views when he was interviewed at the recent Mobile World Congress (the video stuttered badly when I viewed it, but the program is only five minutes long so stick with it).

Among my favorite comments were:

“We’re ahead of iPhone in 7 markets and ahead of Blackberry in 26“. No detail was offered as to where these markets are exactly – the assumption is that these are individual countries. If so, it would be interesting to know where Windows Phone 8 is beating out iPhone. [Update March 28: According to ZDNet, IDC reported that Windows Phone pipped iOS in "Argentina, India, Poland, Russia, South Africa and the Ukraine. The seventh market was a collection of countries, including Croatia, that IDC labels "rest of central and eastern Europe".]

The interviewer put the Windows Phone market share at between 3 and 4% and Terry didn’t disagree. Terry also said that Microsoft had seen tremendous progress over the last nine months and that one billion app downloads had been made from Microsoft’s Store (not much compared to Apple, but a start). The interviewer suggested that BlackBerry was Microsoft’s closest rival, but Terry disagreed, saying that their “sights were higher”.

“Android is a confusing mess” and “iPhone is boring now“. Apparently live tiles make all the difference. I think he is right that the iPhone user interface has started to show its age; he’s also right that the diversity of Android across devices, manufacturers, and software versions can be confusing at times. However, consumers just care whether their phone works and supports the apps that they want to use, which is the huge strength of the iPhone in particular.

The major strength of the WP8 platform was cited as the ability to access the same content across multiple devices, perhaps a reference to SkyDrive. However, the Apple contingent can point to the way that iPad and iPhone share apps, music, and video with Macs using iTunes – data formats that are probably most interesting to consumers whereas the thought of being able to access an Excel worksheet or PowerPoint presentation on SkyDrive is more valuable to the business folks. A more interesting comment came in the reference to the camera and photographic capabilities delivered in Nokia devices, specifically the Lumia 920. Terry also said that WP8 did a better job for lower end phones than low-quality Android devices.

I haven’t looked back since I moved from iPhone to WP and am still happy with the Nokia Lumia 800 (now upgraded to WP 7.8). Nothing in iPhone 5 makes me want to move back and I still can’t get my head around using an Android (which one?). I guess I can stay on the sidelines for a little longer before deciding how to upgrade.

Follow Tony @12Knocksinna

Exchange 2010 SP2 RU6 “can’t delete messages” KB released

Microsoft has released KB2822208 in response to the problems that users reported where they were unable to delete messages when working with Outlook in online mode.

After a great deal of hard word to investigate the possible root causes, Microsoft has determined that the issues lie with attachments generated by third party add-on products for Exchange. The KB mentions two types:

1. Unable to soft delete messages that contain voice mail attachments.
2. Unable to soft delete messages sent from FAX server, printer or scanner which have attachments (such as .PDF).

The problem first emerged when users reported that they couldn’t deal with messages containing PDF attachments generated by multi-function printers, so Microsoft’s KB is in line with real-life experience. The problems with voicemail surfaced soon afterwards and seem to be associated with Cisco Unity rather than Exchange’s own Unified Messaging.

No software fix is available yet, so all you can do is either use Outlook in cached Exchange mode (the delete actions are processed locally and then synchronized back to the server) or perform a “hard delete” by pressing the Shift/Delete combination. This action bypasses the normal processing Outlook does when deleting messages (a soft delete which moves items into the Deleted Items folder) and uses a different code path to avoid the problem. Messages that are hard deleted go into the Recoverable Items folder instead of Deleted Items.

I’m sure that the Exchange team is working hard to figure out a better fix that can be implemented in software. For now all you can do is use the workarounds.

Follow Tony @12Knocksinna

Exchange 2013: Using crimson events to track transaction log truncation within a DAG

Prior to Exchange 2007, transaction log truncation (removal) used to be a pretty simple affair. A successful good backup would truncate the log set or, if you used circular logging, Exchange used a small set of logs to capture transactions and would reuse the files once the transactions in the logs were committed into the database.

The advent of Cluster Continuous Replication (CCR) in Exchange 2007 (plus its variants, LCR and SCR) and more importantly, the Database Availability Group (DAG) in Exchange 2010 required the way that transaction logs are truncated to change. Exchange 2010 breaks the connection between databases and servers and multiple copies can exist within a DAG. Replication of transaction data between servers complicates matters a tad because of a rule that no data should ever be removed if it might be needed. Transaction logs exist to update databases, so it follows that Exchange never truncates logs if the chance exists that the logs might be required to update a database copy.

Some like to operate databases without circular logging. This is fine, as long as you have sufficient disk space to hold the logs. The old best practice of not using circular logging with production databases has evolved as It is quite safe to use circular logging within a DAG, providing that sufficient copies exist to provide a reasonable guarantee of redundancy. Three copies is good; four is even better. And indeed, once database copies are in use, a different form of circular logging called CRCL is used.

TechNet explains the conditions that must exist before transaction logs are truncated by the Replication service:

• The log file must have been successfully backed up, or circular logging must be enabled.
• The log file must be below the checkpoint (the minimum log file required for recovery) for the database.
• All other lagged copies must have inspected the log file.
• All other copies (not lagged copies) must have replayed the log file.

In addition, the article also explains that the transaction logs for the active database copy are never truncated when one or more copies are suspended.  Even more care is taken when a lagged database copy is maintained because the transaction logs are retained for the lagged period and cannot be removed until that period expires and the Replication service replays the logs to update the lagged database copy.

All of this is quite clear, but when you look at the accumulation of transaction logs for a database, you might wonder on what basis the Replication service decides to truncate the logs. Because Exchange 2013 uses quite a big checkpoint depth (100MB), it’s usual to find a hundred or more transaction logs even when circular logging is enabled and the database is essentially quiescent. It’s far removed from the five or six transaction logs that a standalone database enabled for circular logging might use.

Truncation occurs when the “LogTruncator”, a component running inside the Replication service, examines the log set to assess what logs must still be retained. This happens on an ongoing basis and when a database is mounted. Some insight into the decision process can be gained by examining the “TruncationDebug” crimson channel in the Event Viewer. Exchange 2010 began to use crimson events to capture important information about high availability processing; Exchange 2013 captures a lot more information. In the screen shot, you can see that TruncationDebug is under the HighAvailability section. Three interesting events provide some insight into what the Replication service does when it examines logs to decide whether they can be truncated. In sequence, the events are:

• 223: The Replication service gets information from servers that host database copies about what log files they have processed. In the screen shot, the information coming back from the EXSERVER2 server indicates that it wants generation 3557 to be preserved for its database copy. This is the minimum log file required for recovery.
• 224: The Replication service decides what logs can be truncated.
• 299: The Replication service truncates the log stream and removes whatever transaction logs are no longer necessary.

Crimson channel event reporting log truncation

In this instance, the database copy existed on only two servers (definitely not too redundant) and so the Replication service only had to take into account input from the two servers. One (as we’ve seen) advised that it needed generation 3557, so to be safe the Replication Service truncated to generation 3555.

Dumping a transaction log header to check its generation

On a practical level, any transaction logs belonging to generation 3554 or below are removed from the server, so when we look at the transaction log directory, we should see logs for generations 3555 and higher. Of course, Exchange uses a hex numbering scheme for transaction logs, so the log number is DE3, with a full file name of E000000DE3.log. You can validate the generation number for a transaction log by running ESEUTIL with the /ML switch as shown above.

Exchange takes enormous care to make sure that transaction logs are retained until they are no longer required. Given that DAGs can vary so much in construction from simple 2-member implementations right up to the sixteen-member mega-DAGs, it’s obvious that log truncation can be a tricky business. Fortunately the technology works very well in the background. I wish that the same statement could always be made about technology…

Follow Tony @12Knocksinna