Recently, I’ve posted two articles in a multipart series demonstrating how to create Azure virtual networks and peer those Vnets using Azure Resource Manager (ARM) templates written in JSON. Soon, I’ll be posting more samples, for example, how to create Network Security Groups (NSGs) and User Defined Routes (UDRs) in Vnets and associate those resources with the peered virtual networks.
The most important characteristic of the samples I’ve posted so far is that they allow for deployment in multiple Azure subscriptions. Most of Microsoft’s ARM template samples assume a single subscription. That’s fine for many cases, but my client wanted to use Azure subscriptions as a billing container to manage cost allocation (they offer their product to multiple customers). This is an ideal use of Azure subscriptions and Azure makes it easy to do so, in fact Azure seems to promote it, by allowing multiple subscriptions as well as subscription types (for example, EA and Dev/Test). Subscription-as-top-level (billing) container is kind of a no-brainer for cost reporting and charge back.
But the subscription isn’t the only required logical container. There’s a second required container, inside the subscription that contains actual Azure resources like VMs, Vnets, etc. This second container is an Azure Resource Manager resource group (RG).
Microsoft defines RGs as “…simply a logical construct that groups multiple resources together so they can be managed as a single entity. For example, resources that share a similar lifecycle…” should be managed (and, for example, deleted) together.” In other words, an RG isn’t a thing itself — it’s just the thing that contains other (more useful) Azure resources.
So, the resources in Azure subscriptions are themselves grouped into Azure resource groups. Just think of nesting Russian dolls. It’s no big deal to work with a multi-level container hierarchy — in fact, it’s quite common and elegant…
…until you hit a limit on the number of containers a container may contain. The Azure documentation states that the number of nested or linked templates that a template may deploy to is limited to five:
You can deploy to only five resource groups in a single deployment. Typically, this limitation means you can deploy to one resource group specified for the parent template, and up to four resource groups in nested or linked deployments.
I wondered if this was a so-called hard-limit on nested ARM templates or one Microsoft could change on request. So I opened an Azure support request. The (disappointing) answer is that it’s a hard limit on nested templates.
In the IaaS deployment I automated for a client, we chose what I call a “Vnet-forward” or “Vnet-first” design for the infrastructure. IOW, we have Vnets for app services, Vnets for clients, Vnets for security functions, Vnets for appliances and so on. Client Vnets are in different subscriptions, which necessitate different resource groups.
I wrote ARM templates that automate creation of these resources in the required resource groups in different subscriptions. Since the templates can work with any number of input parameters, it would be possible to define all required infrastructure in a single set of parameters and deploy it all in one pass. But there’s a problem. After the fifth client Vnet or fifth peering or fifth UDR, we hit the limitation of five resource groups per deployment.
I was, in a word, shocked at this discovery. And I was nonplussed at the (apparently) arbitrary limit of five resource groups in a single deployment. Surely Microsoft understood that customers would want to both use subscriptions freely to structure their environments while at the same time not needing to split up their deployments into artificial groups of five-at-a-time.
When you deploy a template using PowerShell and/or the Azure CLI, you specify the context — the Azure tenant and subscription — as well as the resource group to which the deployment is made. If you need, as shown in the code snippet below, to do something in a resource group in a different subscription, you must use code a nested or linked template using the type Microsoft.Resources/deployments in an “outer template” and code the actual resource (here in subscription VnetBSubscriptionId and VnetBResourceGroup) as a template that is the property of Microsoft.Resources/deployments.
{
"$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json",
"contentVersion": "1.0.0.0",
"parameters": {},
"variables": {},
"resources": [
{
"apiVersion": "2017-05-10",
"type": "Microsoft.Resources/deployments",
"subscriptionId": "[parameters(VnetBSubscriptionId')",
"resourceGroup": "[parameters('VnetBResourceGroup')",
"properties": {
"mode": "Incremental",
"template": {
"$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json",
"contentVersion": "1.0.0.0",
"variables": {},
"resources": [
{
"apiVersion": "2017-10-01",
"type": "Microsoft.Network/virtualNetworks/virtualNetworkPeerings"
}
]
}
}
}
Not only is this hard and inelegant, by running the deployment via the CLI and/or PowerShell, you have just used one of the five possible RGs that can be deployed to. So, the practical limit is actually four resource groups in a single deployment.
Aside from why Microsoft did something that can only be described as short-sighted, the real question is, “Does it matter?”
One could argue that, no, it doesn’t matter because this will force a more bite-sized and logical deployment process. Since there’s no limit on the number of times you can deploy to a resource group, this isn’t a real limitation. (BTW, there is a limit of 800 on the number of deployments that will be kept in the records of the resource group).
But I think that forcing grouping, whether the infrastructure naturally adapts to grouping in small bites or, as in my use case, forcing the infrastructure into arbitrary groupings is a major flaw in Azure Resource Manager.
What do you think? I’d appreciate your comments on this limitation.
Leave a Reply