Home | Uncategorized | Uh Oh Domino

Uh Oh Domino

0 Flares Twitter 0 Facebook 0 Google+ 0 StumbleUpon 0 Buffer 0 LinkedIn 0 0 Flares ×

It seems that the world is moving to Exchange for email messaging. Unfortunately there are some of us still using Lotus Notes/Domino.

As a messaging product, it seems to me to be reasonably efficient; our Domino servers can support upwards of a thousand users, perhaps 1-2TB of Notes mailboxes. Domino stores the mailboxes as individual files with the .nsf extension. Each of these is opened and held by the nserver.exe task. When using Netbackup with the Notes/Domino agent, the Netbackup client backs up all nsf files on a full backup and the transaction logs and changed nsf files (i.e. those with a new DBID) for an incremental backup. This creates a significant amount of hassle when it comes to performing restores.

A restore is either the full nsf file from the last full backup, or the nsf file plus transaction logs, which are then applied to the nsf file to bring the mailbox up to date. This process is incredibly inefficient because (a) transaction logs contain data for all users and must be scanned for the records relating to the restoring mailbox (b) the transaction logs need to be restored to a temporary file area, which could be considerable (c) the restored logs are disregarded after the restore has completed and so have to be restored again for the next mailbox restore.

So, I’ve been looking at ways to bin Netbackup and improve the backup/restore process. As servers are being rebuilt on Windows 2003 Server, I’ve been looking at VSS (Volume Shadowcopy Services). This is a Windows feature which permits snapshots of file systems to be taken in co-operation with applications and underlying storage. In this instance there isn’t a Lotus Domino provider, so any snapshots taken are dirty (however I did find the dbcache flush command which flushes updates and releases all nsf files). Netapp used to have a product called SnapManager for Lotus Domino which enabled Netapp snapshots of mailboxes using the Domino Backup API. The product has been phased out, as tests performed by Netapp show that dirty snapshots with the security of logs can be used to restore mailboxes successfully. IBM provide trial versions of Domino, so, I’ve downloaded and installed Domino onto one of my test servers under VMware and run the load simulator while taking snapshots with VSS. I’ve also successfully restored a mailbox from a snapshot so there’s no doubting the process works. However my simple task isn’t one of scale. Typical mailboxes are up to 1GB in size and there could be hundreds of active users on a system at any one time. My concern is whether VSS can manage to take snapshots with this level of activity (and not impact the O/S) but also whether the snapshots will be clean or what level of corruption we can expect.

The only way to test this is to implement on a full scale Domino environment and probably with live users. That’s where things could get interesting!

About Chris M Evans

Chris M Evans has worked in the technology industry since 1987, starting as a systems programmer on the IBM mainframe platform, while retaining an interest in storage. After working abroad, he co-founded an Internet-based music distribution company during the .com era, returning to consultancy in the new millennium. In 2009 Chris co-founded Langton Blue Ltd (www.langtonblue.com), a boutique consultancy firm focused on delivering business benefit through efficient technology deployments. Chris writes a popular blog at http://blog.architecting.it, attends many conferences and invitation-only events and can be found providing regular industry contributions through Twitter (@chrismevans) and other social media outlets.
  • Anil Gupta

    Hello Chris,

    Sorry for unrelated comment, couldn’t find your email address on the blog.

    I am attending the SNW in San Diego. I am contacting fellow storage bloggers to inquire about their plans for SNW. If you are attending SNW, will you be interested in attending a
    gathering of fellow storage bloggers? Any thoughts on any such
    gathering are most welcome.

    I recently wrote a blog post on this topic
    http://andirog.blogspot.com/2007/04/say-hello-at-snw.html. I will appreciate any help in spreading the word to fellow bloggers and storage professionals.



  • brett

    Hi Chris

    Interesting post indeed.

    Can you tell me more about how you implemented the VSS/dbcache flush process? I am no Domino expert at all! So sorry for any daft ideas or concepts.

    We backup a number of customer Domino databases using VSS and hit the problem of databases being in cache during the snap-shotting.

    When restored, these databases are then not able to be read:

    “Invalid or non existent document”
    “RRV Bucket is corrupt”
    “Page format is incorrect”

    So I am interested in how effective the dbflush process is – so that we can run it before the VSS snapshoting:

    1) how long would the flushing typically take to happen?
    2) what exactly then causes a database to be re-cached – is it the receipt of an email simply or does the use have to open their client? In otherwords, how could we ensure databases are not in cache when the snapshot takes place?

    As you are clearly and expert, I was wondering if you had any ideas as to fixes to the errors (above)we see when databases are backed up in the crash state?

    Would the tools like fixup and compact that I have read about typically be of any use to make the databases readable?

    Many thanks for any help here.

  • Anonymous


    I just stumbled across this posting (years later!!!) as I seek an answer to this same question… Will Domino EVER be VSS aware? It makes no sense why they don’t take the time to build this in. They already have the API, so just create a VSS provider that works with the API to provide the necessary quiet time for VSS to do it’s thing. Seems like the hard part has already been done with the API, so just call it from the provider.

    Anyway… Given that Domino STILL is not VSS aware, even in their 8.5.x release (amazing) here is another thought/solution to this. Why not create another Domino server that has a full replica of all the key databases in question, and then shutdown the services on this “backup” server every night just before the snapshot kicks off? You could run the most minimal of Domino services on it (ie. no Web, SMTP, Router, etc.) so that the shutdown and restart of the Domino services is as quick as possible. Since the Domino server would effectively be shut down, the data would obviously be in the best state for backup as NOTHING would be open. Yes, this solution costs yet another Domino license, however, given that we typically would be purchasing a backup license of Veritas (or whatever) this may not be so bad. After all, many of us are no longer purchasing backup software and instead relying on VMWare snapshoting and then SAN/NAS snapshoting, so we are already saving loads of money on licensing to begin with!

    I’d be curios of anyone would see any reason this solution would not work?

0 Flares Twitter 0 Facebook 0 Google+ 0 StumbleUpon 0 Buffer 0 LinkedIn 0 0 Flares ×