New AWS CloudFormation Functionality Addresses Programmability, Reusability

Allowing you to programmatically describe and provision your infrastructure, AWS CloudFormation is a critical service in the AWS toolbox for anyone building infrastructure and applications in AWS. To further automate CloudFormation, we have created dozens of custom scripts that generate CloudFormation templates and create Service Catalogs. Having worked with CloudFormation at all scales — from startups to Fortune organizations — we’ve encountered two CloudFormation challenges that we’re excited to see new CloudFormation capabilities address: Programmability and Reusability. These new features introduce streamlined workflows that allow us to abandon our custom tooling and in today’s article, we share three particular examples with you of how CloudFormation’s new capabilities can help reduce your customization efforts, too.

The new AWS CloudFormation functionality extends CloudFormation by enabling you to build and publish resource providers in CloudFormation. A resource provider has resource type specification and handlers that control API interactions with underlying AWS or third-party services. These interactions may include create, read, update, delete and list (CRUDL) operations for resources. Resource providers are used to model and provision resources using CloudFormation. For example, AWS::EC2::Instance is a resource type by Amazon EC2 provider. You can use this type to model and provision an EC2 instance using CloudFormation.

Now operators can build user-defined/custom resource providers to model and provision AWS or non-AWS resources using CloudFormation. And, APN Partners can use the new capabilities to build resource providers to natively support AWS and non-AWS resources with CloudFormation.

AWS CloudFormation now has two key components which are central to the examples we’ll share with you today:

Reusability
Reusability is a key principle of software engineering and similarly a key to scaling DevOps initiatives in organizations. Before, reusing modules was unwieldy. A common way to do it was to put CloudFormation templates in S3 and reuse them as nested stacks. But nested stacks can get clunky. The new CloudFormation functionality addresses both aspects of reusability; it creates a way to define a controlled interface to make sure we can blackbox the module and it provides a well-defined distribution mechanism.

A key component for scaling DevOps is to create high-level, reusable modules to make patterns readily available to application teams. This allows them to focus on writing the business logic that adds value to the enterprise. With a good mechanism for reusability, we can create higher-level modules — e.g. for services.

Programmability
The other new, big feature is that CloudFormation now allows complete programmability. This opens the door to many possibilities, including smarter templates with more complete logic and the ability to create custom functions for external resources.

Updates
While the above two use-cases unlock a lot of value, the biggest challenge for large enterprises is the long-term maintenance of these modules. As needs change, high-level modules evolve and we need a strategy for upgrading existing deployed modules. In line with the lean principle of reducing buffering, CloudFormation will now deploy the currently deployed version across the board on the deployed modules. This causes some upfront pain but reduces the long-term technical debt that plagues many enterprise systems.

Examples
Here are some example use-cases based on the hurdles we’ve seen in deploying Infrastructure as Code (IaC).

Creation of EC2 instances
When we create EC2 instances, we follow best practices around similar security and monitoring. So we also have to create security groups and instance recovery alarms. Using the new CloudFormation functions, we’d be able to create a module that can be shared across the organization that includes these best practices. In addition, without the new programmability features, it was difficult to create a security group set to the minimum permissions because the number of ports would need to be hard-coded. The new programmability features allow iterating over a list, thus increasing the reusability of the module.

Self-sizing VPCs
One of the most tedious parts of creating a VPC is allocating space for each of the subnets. We’ve created tooling that we use with advanced customers that will allow us to connect with an IPAM, get a CIDR block, partition that into the correct size subnets, and perform the CIDR math to get the subnet CIDRs, and use that to deploy the VPC and subnets.

Service deployment with external resources
Taking further the concept of a high-level module that combines the building blocks AWS provides, we can launch complete services. For example, a microservice for deploying to ECS with a Jenkins pipeline for code and external resources. Not only are we able to create the module, but we can also combine the external resources seamlessly.

The new CloudFormation capabilities extend CloudFormation in important ways, allowing for greater programmability and reusability — key tenets of software engineering and enterprise-wide DevOps. Moreover, its streamlined workflows help reduce customization, allowing us to spend more time on other strategic initiatives.

*This was originally written by Flux7 Inc., which has become Flux7, an NTT DATA Services Company as of December 30, 2019