Update: A lot of what's in this blog is dated and wrong, do not use this (2023+) as any sort of guide for publishing constructs. With V2 of the CDK most of this is incorrect now.

Let's talk about construct versioning in the AWS CDK (specifically, Typescript). If you've worked with the Typescript API for the CDK, you've likely run against an error like this:

Types of property 'node' are incompatible.
    Type 'import("/Users/matthewbonig/projects/cdk-sandbox/versioning-example/node_modules/@aws-cdk/core/lib/construct-compat").ConstructNode' is not assignable to type 'import("/Users/matthewbonig/projects/cdk-sandbox/versioning-example/node_modules/@aws-cdk/aws-lambda-nodejs/node_modules/@aws-cdk/core/lib/construct-compat").ConstructNode'.
        Types have separate declarations of a private property 'host'.

new NodejsFunction(this, 'Lambda', { ...

This occurs when you have different versions of the CDK libraries installed, different from the 'core' library:

{
  "dependencies": {
    "@aws-cdk/aws-lambda": "1.89.0",
    "@aws-cdk/aws-lambda-nodejs": "^1.90.1",
    "@aws-cdk/core": "1.89.0",
    "source-map-support": "^0.5.16"
  }
}

This is an especially easy place to get stuck at since NPM will default the '^' caret versioning and install the latest version of a library. This can be troublesome if you're working with a CDK module that is under the experimental banner and future versions could introduce API changes, which would cause your build to break and time spent fixing that.

projen will allow you to specifically address this issue with an optional cdkVersionPinning option, which removes the caret from your package.json and enforces a specific version to be used for all your libraries. Very handy and one of the reasons I use it on my constructs.

While I think most developers are familiar with the problem with versioning and have adjusted their development workflow accordingly, there's special care to be taken when you are publishing your own constructs.

Let's take a look at some specific scenarios around CDK version pinning (not using semver matching, e.g 1.97.0) in the package.json. In each case we'll look at how CDK pinning your own code (pinned or not) vs the third-party construct's use of version pinning (pinned or not) affects the ability to install deps and synth some CDK code.

We'll also see how each scenario is affected by using NPM or Yarn.

Scenario 1 - You're Pinned && Construct is Not Pinned

In this scenario we have our own CDK code that is version pinned at 1.89.0 (no semver matching). We'll leverage a third-party construct that is using a semver matching on the CDK dependencies ^1.88.0.

We have this package.json dependencies:

{
  "dependencies": {
    "@aws-cdk/aws-lambda": "1.89.0",
    "@aws-cdk/aws-lambda-nodejs": "1.89.0",
    "@aws-cdk/core": "1.89.0",
    "cdk-ecr-image-scan-notify": "^0.0.150",
    "source-map-support": "^0.5.16"
  }
}

Using npm i and a cdk synth works without error when synth'ing the following Stack:

export class VersioningExampleStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    new NodejsFunction(this, 'Lambda', {
      entry: path.join(__dirname, 'placeholder.handler.js')
    });

    new EcrImageScanNotify(this, 'Notify', {
      webhookUrl: '',
      channel: ''
    })
  }
}

However, if I use Yarn, I get an error during a yarn install:

error An unexpected error occurred: "expected hoisted manifest for \"cdk-ecr-image-scan-notify#@aws-cdk/core#@aws-cdk/cx-api\"".

Packages aren't installed and nothing will synth.

I can get a yarn install to work, but only if I'm willing to go up to the latest version of the CDK to match what the construct resolved with semver (1.97.0 was latest at time of writing):

{
  "dependencies": {
    "@aws-cdk/aws-lambda": "1.97.0",
    "@aws-cdk/aws-lambda-nodejs": "1.97.0",
    "@aws-cdk/core": "1.97.0",
    "cdk-ecr-image-scan-notify": "^0.0.150",
    "source-map-support": "^0.5.16"
  }
}

This means that if I want to use this third-party construct I'd always have to be on the latest version. I generally recommend people try to keep up with the CDK version as much as possible, but, I don't think that should be a requirement to use a third-party construct. Since there are quite a few APIs in the CDK that are under active development (and may have breaking changes) pinning can prevent unexpected breaks in the build process. Sure, you could also switch to using NPM but that feels like a worse requirement to shove onto consumers of the construct.

Scenario 2 - You're Not Pinned && Construct is Not Pinned

However, let's see what happens if you have your package.json with semversioning the CDK:

{
  "dependencies": {
    "@aws-cdk/aws-lambda": "^1.88.0",
    "@aws-cdk/aws-lambda-nodejs": "^1.88.0",
    "@aws-cdk/core": "^1.88.0",
    "cdk-ecr-image-scan-notify": "^0.0.150",
    "source-map-support": "^0.5.16"
  }
}

Now in both scenarios a npm install and yarn install will both produce a good install and let you synth. As a consumer you might be using stable APIs and can do this.

Scenario 3 - You're Pinned && Construct is Pinned

What about the case when your code is using pinned versions of the CDK and your construct is using the same pinned versions:

{
  "dependencies": {
    "@aws-cdk/aws-lambda": "1.89.0",
    "@aws-cdk/aws-lambda-nodejs": "1.89.0",
    "@aws-cdk/core": "1.89.0",
    "@matthewbonig/nightynight": "1.89.2",
    "source-map-support": "^0.5.16"
  }
}

In this case, the nightynight construct is versioned at 1.89.2 but it uses a pinned 1.89.0 for its dependencies.

Both NPM and Yarn have no problems getting dependencies installed and the synth works as expected.

Scenario 4 - You're Not Pinned && Construct is Pinned

Finally, what happens if you don't want to pin your CDK modules but a construct does?

{
  "dependencies": {
    "@aws-cdk/aws-lambda": "^1.89.0",
    "@aws-cdk/aws-lambda-nodejs": "^1.89.0",
    "@aws-cdk/core": "^1.89.0",
    "@matthewbonig/nightynight": "1.89.2",
    "source-map-support": "^0.5.16"
  }
}

Well, now we can install with either NPM or Yarn, but we'll get the first error again because of the @aws-cdk/core mismatch:

import * as cdk from '@aws-cdk/core';
import { NodejsFunction } from "@aws-cdk/aws-lambda-nodejs";
import * as path from "path";
import { NightyNightForEc2 } from "@matthewbonig/nightynight";

export class VersioningExampleStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    new NodejsFunction(this, 'Lambda', {
      entry: path.join(__dirname, 'placeholder.handler.js')
    });

    new NightyNightForEc2(this, 'NightyNight', {
      instanceId: 'asdf'
    })
  }
}

  Type 'VersioningExampleStack' is not assignable to type 'Construct'.
    Types of property 'node' are incompatible.
      Type 'import("/Users/matthewbonig/projects/cdk-sandbox/versioning-example/node_modules/@aws-cdk/core/lib/construct-compat").ConstructNode' is not assignable to type 'import("/Users/matthewbonig/projects/cdk-sandbox/versioning-example/node_modules/@matthewbonig/nightynight/node_modules/@aws-cdk/core/lib/construct-compat").ConstructNode'.
        Types have separate declarations of a private property 'host'.

      new NightyNightForEc2(this, 'NightyNight', {

Summary

How I publish my constructs

Daniel Schroeder has an excellent blog post here that explains how you can avoid a lot of these issues. I recommend you read that. While I'm going to keep this post up for the information in it I do not recommend you go through pinned dependencies on your constructs.

Seeing all of this I decided that I'm going to publish my constructs for each of version of the CDK I want to support. Using semver CDK versions would certainly ease my workload, but would put restrictions on the consumers (either use NPM or use the latest version of the CDK) that I don't like.

Early on with these constructs I was the only one likely using them, so I only published new versions as I needed them.

However, that's not being a good steward. I wanted to be better. I knew I needed a system that would allow me to easily publish new versions of my constructs with each new version of the CDK. I could do this manually, but felt I should have it automated and wasn't keeping up with it when it was a manual task (you'll see a lot of missing versions in NightyNight).

I also need to make it easy to know which version of my constructs you'd need to install with the version of the CDK you're using. I could use some arbitrary versioning of my construct (e.g. 0.0.1, 1.0.0, 2.1.3, etc.) but then I'd need to list out in the README what version of the CDK each of my constructs is using. Anyone consuming those constructs would need to take that extra step and that felt rather gross. Again, I want to make it as easy as possible for consumers to use my constructs (or else they probably won't).

To that end, my constructs are published with a 'mostly matching' version. Meaning, I published NightyNight as 1.89.2 and it uses 1.89.0 CDK deps. The .2 is there because I had to republish the package twice to work out some bugs. When version 1.90.0 of the CDK was released the CDK versions were bumped and the construct was republished under the new version, with no other changes, at version 1.90.0**. In this case the construct versioning matches the major and minor versions of the CDK it's using, and the minor version's semantic meaning is moved to the .patch value.

** Well, I didn't have any automation in place and didn't actually publish a 1.90.0 version of NightyNight.

Automating the publishing of CDK constructs

Alright, so let's talk about automating this. I'll start with how I did it manually:

1. Create a new branch called bump/$CDK_VERSION
2. Update the CDK version passed to projen
3. Re-synth the project using projen
4. Commit changes from projen
5. Create a PR
6. Merge to trunk
7. Create a Tag/Release for the new version

I am starting by automating the first 5 steps. Sometime in the future I'll handle #6 and #7.

Creating a Branch Automatically

To create a branch I need to know when a new version of the CDK is published. Thanks to the Construct Catalog this is really easy. A few months ago, I PR'd a change to the Construct Catalog where I could subscribe to new constructs being published to the catalog. Up until now I was just sending those messages to my email and would review them when I saw the @aws-cdk/core package get published.

So, I set off to build a Lambda Function that would also subscribe to that same SNS Topic and then create a branch if it saw the @aws-cdk/core package get published, using that published version in the branch name:

export const handler = async (event: any) => {
  console.log(JSON.stringify(event, null, 2));

  // if we're using an SNS message, let's flatten the message to the 'event'
  if (event?.Records[0]?.Sns?.Message) {
    event = JSON.parse(event?.Records[0]?.Sns?.Message);
  }

  const {
    name,
    version,
  } = event; // e.g. {"name":"@aws-cdk/core","version":"1.97.0","url":"https://awscdk.io/packages/@aws-cdk/core@1.97.0/"}

  // we're going to base this check off of the @aws-cdk/pipelines module as it seeems to be published after all other packages
  if (name !== '@aws-cdk/pipelines') {
    console.info('Ignoring the publishing of construct, as it\'s not the CDK Core');
    return;
  }
  // let's make sure we have a version
  if (!version) {
    console.error('You gotta supply an event.version');
    return;
  }

  const workDir = path.join('/tmp', 'autobrancher');
  // ensure the workdir exists
  if (!fs.existsSync(workDir)) {
    fs.mkdirSync(workDir);
  }
  // setup simple-git
  const options: Partial<SimpleGitOptions> = {
    baseDir: workDir,
    binary: 'git',
    maxConcurrentProcesses: 6,
  };
  const git: SimpleGit = simpleGit(options);

  try {
    const branchName = `bump/${version}`;
    const secretString = await getSecretString();
    const repoName = getRepoName();
    const clonedPath = path.join(workDir, repoName!);

    // get and setup the deploy key needed to push back to the repo
    const deployKeyFileName = path.join(workDir, `deploy_keys_${new Date().valueOf()}`);
    writeDeployKey(deployKeyFileName, secretString);

    const GIT_SSH_COMMAND = `ssh -i ${deployKeyFileName} -o StrictHostKeyChecking=no`;
    await git.env({
      ...process.env,
      GIT_SSH_COMMAND,
    });
    console.log('Cloning repo');
    await git.clone(repository!);
    await git.cwd(clonedPath);

    console.log(`Creating new branch ${branchName}`);
    await git.checkoutLocalBranch(branchName);
    console.log('Pushing new branch');
    await git.push('origin', branchName);
    console.log('Branch pushed!');
  } catch (err) {
    console.error('An error happened:', err);
    throw err;
  } finally {
    fs.rmdirSync(workDir, { recursive: true });
  }
};

This code will check out the current construct's code from the trunk branch and then create a new branch with the proper name (e.g. bump/1.97.0) and push it back to the repository.

However, it only took the next version of the CDK being published to find a problem with this. It turns out the @aws-cdk/core package gets published before the rest of the packages are published. The next step in the workflow, a Github Action tried to build the new branch and failed because the other @aws-cdk/* packages hadn't been published yet.

There are a couple ways I could work around this. First thought was to add a delay into the pushing of the branch. However, that's an ugly solution with a lot of fragility. I'd likely have to guess on the delay and if I got it wrong I' d have to wait for another cycle of the CDK publishing to know if I got a better value. Also, that delay could easily need to be more than 15 minutes and that's Lambda's longest timeout.

Another option would be to watch not for the @aws-cdk/core package to be published but whatever the LAST package in the @aws-cdk/* packages to be published. This felt like a better answer, and the @aws-cdk/pipelines package appears to be last.

Everything Else

The rest of the steps can be handled with a Github Action on the branch push. First though, I had to update my constructs so that it was easy to have the Github Action dictate the CDK version projen uses. I chose an environment variable:

const cdkVersion = process.env.CDK_VERSION || '1.97.0';
const project = new AwsCdkConstructLibrary({
  cdkVersion: cdkVersion,
  ...
});

Then I set up a Github Action that handles steps 2->5:

name: cdkbump
on:
  push:
    branches:
      bump/**

jobs:
  build:
    runs-on: ubuntu-latest
    env:
      CI: "false"
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Extract branch name
        shell: bash
        run: echo "##[set-output name=branch;]$(echo ${GITHUB_REF#refs/heads/bump/})"
        id: extract_branch
      - name: Synthesize project files
        run: CDK_VERSION=${{ steps.extract_branch.outputs.branch }} npx projen
      - name: Set git identity
        run: |-
          git config user.name "Auto-bump"
          git config user.email "github-actions@github.com"
      - name: Commit and push changes (if any)
        run: 'git diff --exit-code || ((git add package.json yarn.lock .projen/deps.json) && (git commit -m "chore: bumping CDK version" && git push origin))'
      - name: pull-request
        uses: repo-sync/pull-request@v2
        with:
          destination_branch: "master"
          github_token: ${{ secrets.PUSHABLE_GITHUB_TOKEN }}
    container:
      image: jsii/superchain

Notice specifically:

on:
  push:
    branches:
      bump/**

and

- name: Extract branch name
  shell: bash
  run: echo "##[set-output name=branch;]$(echo ${GITHUB_REF#refs/heads/bump/})"
  id: extract_branch
- name: Synthesize project files
  run: CDK_VERSION=${{ steps.extract_branch.outputs.branch }} npx projen

The Github Action is triggered from this push and away it goes updating the dependencies of the construct.

Great, a lot of automation set up and it greatly reduces the amount of manual work that needs to be done! When a new version of the CDK is published a new branch will be created by the Lambda Function and then pushed to the repo. The Github Action then runs and projen regenerates the dependencies to the new version of the CDK. Additionally, that GH Action also creates a new Pull Request. The only manual steps left are to review the PR and merge it and then create a new Release/Tag. I'm ok with that being manual for now but eventually I'd like to automate that as well.

Conclusion

Since CDK constructs have a strong dependency on the CDK version there is some extra work needed if you're going to publish your constructs. I've chosen a versioning model that creates the most flexibility for the consumer but means I have to jump through some hoops. However, automation can solve many of those problems.

If you'd like to have a similar setup then you'd first need to submit a PR to allow your own AWS account to subscribe to the SNS topic from the Construct Catalog, you'd want your PR to add your account number here. The Autobrancher microservice CDK code can be found in this CDK App.