OpenTelemetry and AWS Lambda

Many organizations are developing distributed applications based on a serverless microservice architecture, which includes AWS Lambda and other managed AWS services.

Organizations dealing with distributed architecture face a variety of operational challenges, including how to resolve availability and performance issues quickly. This means that when organizations attempt to monitor these applications, they want to understand the application’s health and pinpoint services that are impacted due to a performance bottleneck or an error rate increase that can affect their customers.

However, not all organizations can afford to make large code modifications in their current pipelines to add greater observability, despite the fact that organizations want deeper visibility into the behavior of their systems.

In this walkthrough, we explain how to send telemetry data from AWS Lambda Nods.js functions to backends using OpenTelemetry and Lambda wrappers, without having to change a line of code.

The code on this page was uploaded to a GitHub repository, which you can use to view the whole project if needed.

Project Setup

Let’s create a new directory for our project, and run the following command inside of it:

npm init -y

Then add the following dependencies to the resulting package.json:

"dependencies": {
  "@opentelemetry/api": "^1.1.0",
  "@opentelemetry/auto-instrumentations-node": "^0.30.0",
  "@opentelemetry/exporter-trace-otlp-http": "^0.28.0",
  "@opentelemetry/instrumentation": "^0.28.0",
  "@opentelemetry/sdk-trace-base": "^1.2.0",
  "@opentelemetry/sdk-trace-node": "^1.2.0"
}

After adding the required OpenTelemetry dependencies, run the following command:

npm install

Adding a Lambda wrapper

This file contains all the OpenTelemetry logic, which will enable tracing. So let’s add the following to a new file called lambda-wrapper.js:

const api = require("@opentelemetry/api");
const { BatchSpanProcessor } = require("@opentelemetry/sdk-trace-base");
const {
  OTLPTraceExporter
} = require("@opentelemetry/exporter-trace-otlp-http");
const { NodeTracerProvider } = require("@opentelemetry/sdk-trace-node");
const { registerInstrumentations } = require("@opentelemetry/instrumentation");
const {
  getNodeAutoInstrumentations
} = require("@opentelemetry/auto-instrumentations-node");
api.diag.setLogger(new api.DiagConsoleLogger(), api.DiagLogLevel.ALL);
const provider = new NodeTracerProvider();
const collectorOptions = {
  url: "<backend_url>"
};
const spanProcessor = new BatchSpanProcessor(
  new OTLPTraceExporter(collectorOptions)
);
provider.addSpanProcessor(spanProcessor);
provider.register();
registerInstrumentations({
  instrumentations: [
    getNodeAutoInstrumentations({
      "@opentelemetry/instrumentation-aws-lambda": {
        disableAwsContextPropagation: true
      }
    })
  ]
});

Replace <backend_url> with the URL of your favorite backend to send all tracing data to it.

Note that disableAwsContextPropagation is set to true. The reason for this is that the Lambda instrumentation tries to use the X-Ray context headers by default, this results in a non-sampled context, which creates a NonRecordingSpan.

More details can be found in the instrumentation documentation.

Adding a simple handler

Let’s add a simple handler that will serve as a Lambda function that contains a simple GET request. Create a file called handler.js:

"use strict";
const http = require("http");
module.exports.hello = async (event) => {
  http.get("http://aws.amazon.com");
  return {
    statusCode: 200,
    body: "Success!"
  };
};

Deployment

There are multiple ways of deploying, but we will be using Serverless Framework for ease of use.

Let’s create a file called serverless.yml:

service: lambda-otel-native
frameworkVersion: '3'provider:
  name: aws
  runtime: nodejs14.x
  region: '<your-region>'
  environment:
    NODE_OPTIONS: --require lambda-wrapperfunctions:
  lambda-otel-test:
    handler: handler.hello

For OpenTelemetry to work properly, lambda-wrapper.js must be included before any other file. That’s why we have added the environment variable NODE_OPTIONS:–require lambda-wrapper which preloads the wrapper at startup.

Note if you are not using Serverless Framework to deploy your Lambda function, you must manually add this environment variable using the AWS Console UI.

Finally, run the following command to deploy the project to AWS:

serverless deploy

Running the Lambda function

We can now invoke the newly deployed Lambda function by using the AWS Console UI, and we should expect to see spans related to the invocation of the Lambda function.

The AWS console UI for invoking the Lambda function

Visiting the backend to view our traces

The backend should receive traces produced by OpenTelemetry from our Lambda function!