Connecting On-Premises Enterprises to Windows Azure Active Directory

Although it's possible to use the new Windows Azure Infrastructure Services to build a datacenter that resides completely in the cloud, it's also well-suited to acting as an extension to your on-premises datacenter (or as a secondary datacenter). However, these types of deployments require special planning, especially when it comes to utilizing Active Directory for authentication and authorization.

Microsoft has offered Windows Azure since early 2010. But the April release of Windows Azure Infrastructure Services now allows IT to use the cloud service to deploy VMs and apps designed to run on Windows Server, including SQL Server, SharePoint, and other apps and infrastructure (see the June 2013 feature, "Deploy VMs in Windows Azure," for more on Windows Azure Infrastructure Services).

If you're planning to use Windows Azure as an extension of your datacenter, it makes sense to create a hybrid Active Directory forest in which domain controllers exist on-premises and in the cloud. The reason for this is many server applications require Active Directory access, and you really don't want apps running in the cloud to have to consult an on-premises DC every time they need to perform an operation. Not only is doing so inefficient, but a WAN failure could cause an app to malfunction due to its inability to contact an on-premises DC. As such, it's important to extend your Active Directory to the cloud. Microsoft is enabling that through its new Windows Azure Active Directory (WAAD).

On the surface, the idea of extending your existing Active Directory forest to Windows Azure seems deceptively simple. After all, Windows Azure makes use of VMs running exactly the same Windows OSes that can be run on-premises. Besides, organizations routinely extend Active Directory forests to off-site datacenters all the time, so why should extending an Active Directory forest to a Windows Azure cloud be any different?

Creating a hybrid Active Directory by using a mixture of on-premises and cloud-based DCs is no different than building a multi-site Active Directory forest. As is the case with many things in IT, the devil is in the details.

Prerequisites to Deployment

As I guide you through the process of deploying WAAD, I'll presume you already have an on-premises directory in place. I'll also assume you have a basic working knowledge of Active Directory and DNS.

Just as important, your on-premises network must include an externally accessible VPN server. You should provision this VPN server with a static IP address that's publicly accessible. It requires a VPN in order to establish connectivity between the servers that are hosted on Windows Azure and your on-premises servers.

Throughout the course of my deployment evaluation, I established a virtual network on Windows Azure, but this virtual network wasn't externally accessible. The easiest way to establish connectivity between the on-premises network and the Windows Azure virtual network is to make use of your on-premises VPN.

Set up a Hybrid Active Directory

The first step in the process is to open the Active Directory Sites and Services tool and create a new WAAD site. To do so, right-click on the Sites container and choose the New Site command from the shortcut menu. When the New Object - Site dialog box appears, enter "Azure" as the site name, select the DEFAULTIPSITELINK and click OK (see Figure 1). Upon doing so, you should see a message indicating the new site has been created. Click OK to clear this message.



Build a replica DC on a VM running on top of Windows Azure. Set this up with the presumption that Windows Server is already installed on the replica DC and the replica DC has been assigned an IPv4 address. Make note of the address that you're using for your replica DC.

Go back to the on-premises DC and, from within the Active Directory Sites and Services console, right-click on the Subnets folder and choose the New Subnet command from the resulting shortcut menu. When the New-Object Subnet dialog box appears, enter the subnet in the Prefix dialog box. The prefix must be entered in Classless Inter-Domain Routing (CIDR) notation (for example, 157.54.208.0/20). Select the Azure site object before clicking OK.

So far you've configured the on-premises Active Directory forest to recognize a new site, but haven't yet established communication between the two sites. Furthermore, the WAAD instance is dependent on DNS. Typically the on-premises DNS server used by Active Directory isn't externally accessible. This is a problem because the replica DC hosted on Windows Azure will need access to the on-premises DNS.

The first step in making the server externally accessible is to register the on-premises Active Directory-integrated DNS server with WAAD. This allows you to associate the DNS server with the Windows Azure virtual network. To accomplish this, you can open the Windows Azure Management Portal and click on the Networks link found in the navigation pane. Then click the New button and select the Networks | Virtual Network | Register DNS Server options (see Figure 2). Enter the name of your on-premises DNS server into the name field and then enter the DNS server's IP address into the DNS Server IP Address field.



You're also going to need to have WAAD integrated with a DNS server. Therefore, click the New button and navigate to Networks | Virtual Network | Register DNS Server. Now, enter the name and IP address that you'll use for your cloud-based DNS server.

Establish Connectivity

Once you've registered the on-premises and the cloud-based DNS servers, the next task is to provide connectivity between the two networks. This is done by establishing site-to-site VPN connectivity between the on-premises network and the Windows Azure virtual network.

From the Windows Azure Management Portal, click on the Networks link and then click on the New button. Next, click on Networks | Virtual Network | Custom Create. Windows Azure will launch the Create a Virtual Network Wizard. Enter a name for the virtual network and then select the appropriate affinity group.

At this point, you must click the arrow icon to move to the Address Space and Subnets page. Click on the CIDR button and then use the Add Address Space button (and, optionally, the Add Subnet button) to define the address space and any required subnets for the virtual network (Azure, which you named earlier).

Windows Server 2012 deduplication vs. backup software dedupe

Now that deduplication has become a mainstream technology, IT pros are forced to choose between competing deduplication solutions. One question that I have been asked several times lately is whether it is better to use Windows Server 2012's native file system deduplication or to leave the deduplication process to your backup software.

As with so many other things in IT, there is not a clear-cut answer that holds true 100% of the time. There are, however, a number of factors that can help you determine which deduplication solution is better for your organization.

What are your deduplication goals?

The first consideration that needs to be taken into account is your deduplication goals. Backup software products that perform source deduplication do so with the goal of decreasing the volume of data that needs to be transmitted to the backup server. This type of deduplication typically does not decrease the server's use of primary storage, but rather decreases the volume of data that is transmitted to and stored on the backup.

Windows Server 2012's native deduplication is also source-based, but its primary goal is to decrease space consumption on the server's primary storage. This type of deduplication does not do anything to reduce the volume of data that is being backed up (at least not by itself).

Of course, this raises the question of whether it is possible to use Windows Server 2012's native file system deduplication in conjunction with backup software in an effort to get deduplicated primary storage and a deduplicated backup. Generally speaking, the answer is yes. You just have to make sure that the backup software is fully compatible with Windows Server 2012. In some cases, backup software might even be able to take advantage of the fact that Windows has already deduplicated data so that the backup software does not have to perform its own source-side deduplication. In order to do so, the backup agent would have to be Windows-deduplication-aware, which means that it would have to perform a block-level backup. It would also have to know to back up changed data and the Windows changed chunk store container.

Scalability

Another major consideration is scalability. Enterprise-class backup software is usually designed with the assumption that vast amounts of data will need to be deduplicated. Conversely, Windows Server 2012's native file system deduplication is better suited to small and medium-sized organizations.

A Windows Server 2012 deduplication job requires 1 CPU core and about 250 MB of memory. Given these resources, Windows is able to deduplicate a single volume at a time and can process roughly about 100 GB of data per hour, or 2 terabytes (TB) per day.

Needless to say, this capacity will likely prove to be too small for larger organizations. Windows servers that contain additional CPU and memory resources can deduplicate multiple volumes simultaneously, but running parallel deduplication jobs does nothing to increase the per-volume deduplication throughput.

What type of data is being deduplicated?

The type of data that is being deduplicated plays a major factor in whether it will be better to use native deduplication or backup software deduplication. Regardless of which approach you use, some data will inevitably deduplicate better than other data. Unique data, of course, cannot be deduplicated.

Source-side deduplication that is baked into a backup agent does not typically place restrictions on the types of data that you can and cannot deduplicate. The reason for this is that backup software does not usually alter the original data. Instead, it focuses on removing redundancy before the data is sent to the backup server.

Windows Server 2012's deduplication does alter the original data. That being the case, there are some types of data that are poor candidates for deduplication. Specifically, Microsoft recommends that you do not deduplicate Hyper-V hosts, Exchange Servers, SQL Servers, WSUS servers or volumes containing files that are 1 TB in size or larger. The essence of this recommendation is that volumes containing large amounts of locked data (such as a database) or rapidly changing data tend not to be good candidates for deduplication.

In some cases, using Windows 2012's native deduplication feature might not even be an option. If you have servers that are running older versions of Windows, you will have to rely on your backup software to perform deduplication. Even if your servers are running Windows Server 2012, Windows is not capable of deduplicating boot volumes or system volumes. Many servers contain only a single volume, which means that you probably won't be able to use Windows' native deduplication capabilities for some servers, even if they are running Windows Server 2012.

The verdict

So, is it better to use Windows Server 2012's native file system deduplication, or should you use your backup software's deduplication capabilities instead? If your main goal is to optimize the backup process, then you will usually be better off using the backup software's deduplication feature. The backup agent's deduplication engine is dedicated to optimizing the backup process, whereas the native Windows deduplication engine is more concerned with storage optimization. However, in some cases, they can be used together.

Microsoft ReFS backup

When Microsoft first introduced NTFS back in the '90s, many administrators unexpectedly discovered that they were unable to back up computers that used the new file system.

Although this might seem like a lesson in ancient history, history does sometimes repeat itself.

When Microsoft released Windows Server 2012, they introduced the Resilient File System (ReFS). ReFS is Microsoft's first new file system in well over a decade (unless you count minor revisions to legacy file systems).

As was the case with NTFS so long ago, blindly implementing ReFS can have consequences. As such, it is critically important to perform lab testing prior to implementing ReFS in a production environment.

There are many different ways that you can test the ReFS file system, but some of your initial testing should be geared toward making sure that ReFS works with your backups.

Before you even begin the testing process, it is a good idea to contact your backup vendor and make sure that its software supports ReFS backups. The vendor may tell you that your backup software needs to be updated with a patch. Some backup vendors have allegedly told customers that ReFS backups work, but are not officially supported.

More often however, vendors have stated that ReFS support will be added in the next release. For example, Symantec's Backup Exec 2012 does not support the ReFS file system. Symantec has stated that ReFS support will be added in Backup Exec 2012 r2. This release will also add support for other Windows Server 2012 features, such as Hyper-V and Windows Server 2012 failover clusters.

Still other vendors take a completely different approach to supporting backups of the ReFS file system. Commvault has stated that change journal and data classification scan methods are not supported on ReFS volumes, due to issues with their agent. They recommend performing a classic scan instead.

Regardless of the backup software you are using, it is important to determine your backup vendor's policy on backing up ReFS volumes.

After checking with the backup vendor, the first and most basic test that you should perform is a simple check to make sure that your backup software is able to make a backup of a ReFS volume. Assuming that your backup software is able to back up a ReFS volume, the next step is obviously to perform a restore test. However, there is more to verifying ReFS compatibility than simple restore and done testing.

When you are initially testing your backup software's compatibility with the ReFS volume, I recommend that you perform the tests using file server data. The reason for this recommendation is that file data commonly contains a very diverse set of permissions. Restoring file server data to a lab server will help you to verify that the permissions remain intact throughout the backup and recovery process.

It is also a good idea to talk to your backup vendor and determine the maximum file size and maximum volume size that your backup software can accommodate.

The reason for this is that ReFS offers greater scalability than NTFS. The theoretical maximum file size on an ReFS volume is 16 exabytes, with a maximum theoretical volume size of 1 yottabyte. Organizations are unlikely to have such large files and volumes today (especially given the hardware limitations), but data may eventually grow larger than what an NTFS volume could accommodate and this might cause the backup software to break, unless you have verified the software's scalability limits ahead of time.
Conclusion

After reading this article, I'm sure that some of you are probably wondering what the odds are that your backup software will be compatible with ReFS. The odds really just depend on how old your backup software is and how the software works. ReFS is actually based on NTFS and accessing the file system does not require any additional APIs. This means that most backup software will probably work fine with ReFS. However, there are certain NTFS features that are not supported on ReFS volumes (such as User Data Transactions and extended attributes), and this could potentially cause problems with some backup applications. Thorough backup testing is the only way to guarantee that your backup software will work properly with ReFS.

Integrating e-discovery process with traditional backup

Here in the United States, we have something of an alphabet soup of federal regulations that impact IT pros in various industries. These regulations include HIPAA, FISMA and SOX, and the list goes on and on. Although each of these sets of regulations is unique, most of them impose some sort of data retention and e-discovery requirement.

E-discovery, or the task of searching for specific data for legal purposes, is one of those tasks that sounds simple enough but, in practice, has proven to be quite complicated. One of the things that make e-discovery so tough is that data exists in a variety of forms. For example, an organization might have data stored on a file server, on an Exchange Server, in a SQL Server database and in a SharePoint document library. Of course, that's just the organization's current data. There is also data stored in the organization's backups and archives.

In the past, there was really no good way of searching across multiple data repositories. A backup vendor, for example, might provide a utility that allows administrators to search the backups, but the interface probably will not be able to search for data that has not yet been backed up. Likewise, search results might lack granularity. For instance, a search for the term Contoso might return file names including the word Contoso or documents created by a user named Contoso, but the results might not examine document contents or might not look at individual e-mail messages. Simply put, backup indexes are good for locating data that needs to be restored but, by and large, they are not intended as an e-discovery solution.

Both native and third-party, application-specific e-discovery products have existed for quite some time. Microsoft's Exchange Server 2013, for example, includes a native e-discovery tool that is capable of locating Exchange Server data and SharePoint data, but it does not provide the ability to search other data sources, including data that has been backed up. More recently, however, some vendors have begun integrating e-discovery and backup software products.

One particularly good example of such a product is Quest's Recovery Manager for Exchange. Recovery Manager for Exchange is specifically designed to provide Exchange Server related e-discovery capabilities (although it also supports Lotus Notes). It is not designed to examine non-messaging data. The thing that makes the product so interesting, however, is the fact that a search query can be simultaneously run against live data (an Exchange Server database), archive data (an archive database, an offline database, or a PST file) and data that has been backed up.

The software has built-in support for Windows Backup, Backup Exec, Veritas NetBackup and EMC Legato Networker. It can natively access and query backups made using these applications. Although this list of backup products is somewhat limited, the software can easily be configured to work with dozens of other backup applications. In these cases, Recovery Manager for Exchange works with the backup software to perform e-discovery, rather than bypassing the backup software and natively querying the backups.

Quest is not the only vendor whose products are designed to interact with backup applications. For example, Symantec offers an e-discovery product called Enterprise Vault Collector, which works with its archive platform. EMC offers an e-discovery product called Kazeon, and CommVault offers consulting focused on designing an e-discovery-friendly backup/storage architecture using Simpana. However, what makes Recovery Manager for Exchange unique is the number of backup types and backup applications that the product is designed to work with.

Deduplication and data lifecycle management

While there is no denying that deduplication can be a tremendously helpful tool for backup administrators, there have been certain areas in which deduplication has historically proven to be inefficient.

For example, many deduplication solutions have ignored the fact that some organizations perform backups that are more sophisticated than a basic backup, which merely creates a redundant copy of production data. Larger organizations may use multiple tiers of backup storage, in which data is retained for varying lengths of time. Additionally, business continuity requirements may mean that some of the backed up or archived data may reside off-premises, either in an alternate data center or in the cloud.

To see how the deduplication process works in such a situation, imagine that an organization creates disk-based backups to an on-premises backup server which stores 30 days' worth of backups. Then, the backup server's contents are replicated to an off-site backup appliance and workflows are in place to move aging backup data onto less expensive storage. Let's say 120 days' worth of backups are stored in an off-site data center and that a full two years' worth of backups are stored in the cloud.

Data is deduplicated as it is written to the on-premises backup server. The replica backup server is a mirror of the primary backup server, so deduplicated data can be sent to the replica without the need for rehydration.

However, when data is sent to is the off-site, long-term data repositories things start to get messy. The off-site storage is not a mirror to the backup server, so the backup server's contents cannot simply be replicated. Instead, the data must be rehydrated before it can be sent to the off-site storage. And, because the data is being sent off site, it will likely need to be deduplicated at the source side prior to being sent over the wire. In essence, deduplicated data is being rehydrated only to be deduplicated once again.

Because of these inefficiencies, backup and deduplication vendors have developed various solutions to make the process of moving data to a long-term repository more efficient. CommVault, for example, provides a technology known as Deduplication Accelerated Streaming Hash, or DASH as they like to call it.

DASH makes it possible to move deduplicated data across storage tiers without the need for rehydration. DASH provides the means for creating deduplication-aware secondary copies of your data. The copy process can be either disk-read optimized or network-read optimized.

For a disk-read optimized copy, signatures are read from the source disk's metadata and then sent to the destination media agent, which compares the signatures to the signatures on the destination deduplication database.

If the signature already exists in the destination database, then the data also exists in the destination, so there is no need to transmit the data again. Instead, only the signature references are transmitted. If, on the other hand, no signature is found, then the data is presumed to be new and so the new data is transmitted to the destination, and the destination database is updated.

FalconStor takes a somewhat similar approach with its File Interface Deduplication System (FDS) solution. The product makes use of a global deduplication repository, which makes WAN-optimized replication to remote datacenters possible.

Network-read optimized copy operations are ideal for low bandwidth environments, but they are more I/O intensive than disk read optimized copy operations. The process works similarly to a disk read optimized copy, except that a disk read optimized copy reads signatures from the primary disk's metadata. A network optimized copy operation unravels the data on the primary disk and generates signatures to send to the destination media agent for comparison.

Hyper-V Backups

Although Hyper-V backups are relatively straightforward, there are at least four serious gotchas that administrators must be on the lookout for. These gotchas might not be obvious at the time that the backup is created, but can prevent you from being able to restore the backups that you have created.

1. There is more to a VM than just a virtual hard disk

By far, the most common mistake that organizations make when performing a Hyper-V backup is to treat a VM backup as if it were a physical server backup by installing a backup agent within the virtual machine. The reason this approach is problematic is a backup agent that is installed inside the virtual machine can only backup the virtual machine's contents. Essentially, this means that the backup agent is backing up the contents of one or more virtual hard disk files. However, there is much more to a virtual machine than just the virtual hard disks.

Virtual machines are made up of many different components, including things like virtual hard disks, virtual machine configurations and snapshots. If you only backup the virtual machine contents then it will be impossible to restore the virtual machine as a whole. This means losing access to the virtual machine configuration and to any snapshots that might have been created.

There are situations in which guest-level backups are warranted (such as when the virtual machine is running an operating system that is not supported for host-level backups), but guest-level backups should be the exception rather than the rule.

2. VM contents and configurations matter

Hyper-V backups make use of the Volume Shadow Copy Services. In doing so, the Hyper-V VSS writer communicates with the VSS writer inside each individual virtual machine to facilitate the backup process. By using this method, Hyper-V is able to ensure that each VM is backed up in a reliable manner while also ensuring application consistency inside the individual VMs.

In order for the Volume Shadow Copy Services to be used to back up the virtual machines, each VM must meet a number of criteria including:

    The VM must be fully enlightened. This means that the Integration Services must be installed and that the Backup Integration Service must be enabled.
    Each VM must be configured to store snapshots on the same volume as its virtual hard disks.
    The volumes within each VM must be configured as basic disks (as opposed to dynamic disks) and must be formatted using NTFS.
    The VMs must be in a running state.

If a VM fails to meet even one of these criteria, then Hyper-V will resort to using the saved state backup method instead. This means momentarily placing the VM into a saved state, creating a VSS snapshot and then returning the VM to its normal state. Although this backup method works, it does cause an interruption in service.

3. A single VM backup failure can be catastrophic

Perhaps the most important thing to know about Hyper-V backups is that the backup process's dependency on the Volume Shadow Copy Services leads to one major weakness in the backup process. A backup can only be created after the VSS writer has been notified that VSS snapshots have been successfully created for each virtual machine. Therefore, if the VSS snapshot process fails for even a single virtual machine, then the entire backup will fail. Microsoft does not support creating partial backups of Hyper-V.

4. Pass-through disks need special consideration

The last critically important thing that you need to know about Hyper-V backups is that pass-through disks and iSCSI disks need special attention. These types of disk are not visible to the parent partition (the host operating system). As such, they cannot be backed up by the Hyper-V VSS writer. The only way to back up these types of disks is from within the virtual machine itself.