Sharing is caring: Sharing dependencies and code using AWS Lambda Layers

Sharing dependencies and code using AWS Lambda Layers

Lambda functions are dependent on a lot of other modules or dependencies which are bundled with the zip package created for every lambda. For example, if 20 lambda functions share a set of common dependencies, the deployment zip package for each lambda contains them as well which results in large package size to be uploaded and hence slower deployment times.

In addition to this, the most significant risk one might face is breaching AWS Lambda Limits of function code storage (Code storage limit exceeded) per region which is 75GB at the time of writing this post.

Is there a way by which we can share a set of common dependencies between various lambda functions? 🤔

Sharing is caring is a common phrase but has a valuable meaning when it comes to sharing common dependencies. AWS Lambda has a feature called AWS Lambda Layers which allows us to include common dependencies and code that can be shared between Lambda functions and even other AWS accounts. Now, you can put common components in a ZIP file and upload it as a Lambda Layer. Your function code doesn’t need to be changed and can reference the libraries in the layer as it would typically do.

Advantages of Lambda Layers

• Makes your function code smaller and you are more focused on what you want to build
• Speed up deployments, because less code must be packaged and uploaded, and re-use dependencies
• Enforce separation of concerns, between dependencies and your custom business logic
• Easier updates, Lambda Layers, can be versioned to manage updates, and each version is immutable
• Avoid AWS Account Limit of 75GB for Lambda code storage by re-using the common dependencies using layers

How can we use Lambda Layers?

There are two perspectives to using Lambda Layers:

1. Create/Publish a layer that can be used by other lambda functions
2. Using a specific version of the layer in your function

Serverless Framework version 1.34.0 or greater provides support for both publishing and using Lambda Layers with your functions.

Create/Publish a Lambda Layer

The Lambda Layers are created as a part of a serverless application. Create a serverless project for your layer

serverless create --template aws-nodejs --path hellolayers

The layer created in this serverless application will contain a set of common dependencies shared between different lambda functions.

Layers are defined under the layers element in the serverless.yml file and contain the following properties:

• Name of layer: In this case common-node-libs
• path: This property is a path to a directory that will be zipped up and published as your layer. In this case, it’s the node-libs directory where a package.json file resides which contains the common node dependencies to be shared across lambdas
• description: Description of the contents of the layer

Your serverless.yml file should look like:

Create a directory node-libs and add a package.json file to it which contains the common dependencies to be shared. Your directory structure should look like

Run npm install in the node-libs directory to create the node_modules folder to be packaged in this layer

Now it’s time to publish the layer

serverless deploy —-verbose

On successful execution of the above command, you can see the created layer in the AWS console. Note down the ARN of the layer as you will be using it in defining the lambdas using this layer.

Layers are installed in /opt directory in the order you provided in the serverless.yml file. In the above example, the contents under the node-libs directory will be extracted under the /opt directory, i.e. /opt/node_modules. Order is important because layers are all extracted under the same path, so each layer can potentially overwrite the previous one.

You can find the source for this layer at https://github.com/chaubes/hellolayers

Using Lambda Layer in a function

Now that we have successfully published our Lambda layer, its time to use it in the lambda functions of our serverless applications.

Make the following changes to your serverless.yml file:

• Add a new environment variable COMMON_LIB_LAYER_ARN storing the Version ARN of the layer being used i.e. arn:aws:lambda:{{AWS_REGION}}:{{AWS_ACCOUNT_ID}}:layer:common-node-libs:1. The Version ARN of the layer will be referred from all the lambda functions using it.
• Add a new environment variable NODE_PATH and set its value to “./:/opt/node_modules” to make sure that the node dependencies are imported from app’s node_modules directory as well as from /opt/node_modules directory.
• Make sure that the package element should have the exclude attribute with node_modules/** explicitly listed to make sure that the app’s node_modules folder and its contents are not packaged in the serverless zip and unnecessarily increase the size of lambda code. The dependencies required for lambda execution will be picked from layers referenced by lambda functions.
• For every lambda definitions, specify the version ARN of the layer being used under the layers element. The value for the version ARN should be derived from the environment variable created in serverless.yml file, i.e. COMMON_LIB_LAYER_ARN. You can add up to 5 layers under the layers element in the lambda function definition.

After making the above changes, your serverless.yml file should look like:

Deploy this serverless application

serverless deploy —-verbose

The critical thing to note is that the lambda function with deployment package of size 1.02KB has 33.72MB of node dependencies available to be utilised. AWS Lambda Layers significantly reduce the package size of the lambda functions when they share a set of common dependencies.

You can find the source for the above sample serverless application using Layers @ https://github.com/chaubes/example-lambda-using-layers

FAQs

What if I need to add a new dependency only in one of the serverless applications as other applications do not require it. How can I package this new dependency along with reference to layers used by the lambda functions?

In this scenario, you can specify the include element in the serverless.yml file under the package element and have the folder for the new dependency under the node_modules directory to be explicitly included while still excluding other dependencies which are already present in the layer. For example, if you need to add the new dependency of axios in one of your serverless applications, you can specify the package element as below in your serverless.yml file.

package:
  include:
    - node_modules/axios/**
  exclude:
    - .git/**
    - .vscode/**
    - .serverless/**
    - .idea/**
    - node_modules/**

This will make sure that that the axios library will be included while packaging the serverless application.

How can I add a new common dependency which needs to be shared by most of the serverless applications?

You can update the existing layer by adding the new common dependency. When you publish the updated layer, you get a new version of the layer being created. The older version remains as is. This way you can update the version ARN of the layer in the serverless applications where you require the new shared dependency.

Why have you created a new serverless application for layers, can’t we define layers inside an existing serverless application?

You can define the layer in your existing serverless application, but every time you deploy that serverless application, a new layer version is created. This is due to limitations with CloudFormation. So, in order to keep the layer version increments limited to layer updates only, its best to keep layer and the lambda functions in separate serverless applications. This is important as changes to lambda functions should not increment the version of layers used by them.

---

• Next Post →