Adds OpenTelemetry distributed tracing support to nginx.

Supported propagation types:

• W3C - default
• b3

• OS: Linux. Test suite currently runs on Ubuntu 24.04, Alpine 3.20, Amazon Linux 2 & 2023.
• Nginx
• Nginx modules
  • ngx_http_upstream_module (proxy_pass)
  • ngx_http_fastcgi_module (fastcgi_pass)

2. opentelemetry-cpp - opentelemetry-cpp needs to be built with position independent code, e.g.:

cmake -DCMAKE_POSITION_INDEPENDENT_CODE=ON ..

mkdir build
cd build
cmake -DNGINX_VERSION=1.27.3 ..
make

Download the .so file from the latest GitHub Action run or follow the instructions above to build. Then modify nginx.conf, or see the example

load_module /path/to/otel_ngx_module.so;

http {
  opentelemetry_service_name "nginx-proxy";
  opentelemetry_otlp_traces_endpoint "http://collector:4318/v1/traces"

  server {
    listen 80;
    server_name otel_example;

    root /var/www/html;

    location = / {
      opentelemetry_operation_name my_example_backend;
      opentelemetry_propagate;
      proxy_pass http://localhost:3501/;
    }

    location = /b3 {
      opentelemetry_operation_name my_other_backend;
      opentelemetry_propagate b3;
      # Adds a custom attribute to the span
      opentelemetry_attribute "req.time" "$msec";
      proxy_pass http://localhost:3501/;
    }

    location ~ \.php$ {
      root /var/www/html/php;
      opentelemetry_operation_name php_fpm_backend;
      opentelemetry_propagate;
      fastcgi_pass localhost:9000;
      include fastcgi.conf;
    }
  }
}

To use other environment variables defined in the specification, must add the "env" directive.

env OTEL_EXPORTER_OTLP_HEADERS;

http {
  .
  .
  .
}

Enable or disable OpenTelemetry (default: enabled).

• required: false
• syntax: opentelemetry on|off
• block: http, server, location

Service name for the nginx instance (default: uknown:nginx).

• required: false
• syntax: opentelemetry_service_name <name>
• block: http

Chooses between simple and batch span processor (default: batch).

• required: false
• syntax: opentelemetry_span_processor simple|batch
• block: http

OTLP HTTP traces endpoint. (default: http://localhost:4318/v1/traces).

• required: false
• syntax: opentelemetry_otlp_traces_endpoint <endpoint>
• block: http

Set the operation name when starting a new span.

• required: false
• syntax: opentelemetry_operation_name <name>
• block: http, server, location

Chooses the traces sampler. (default: parentbased_always_on).

• required: false
• syntax: opentelemetry_traces_sampler always_on|always_off|traceidratio|parentbased_always_on|parentbased_always_off|parentbased_traceidratio
• block: http

Chooses the trace sampling ratio between 0.0 and 1.0 when a ratio based sampler is active. (default: 1.0).

• required: false
• syntax: opentelemetry_traces_sampler_ratio <value>
• block: http

Enables or disables using spans from incoming requests as parent for created ones. (default: enabled).

• required: false
• syntax: opentelemetry_trust_incoming_spans on|off
• block: http, server, location

Adds a custom attribute to the span. It is possible to access nginx variables, e.g. opentelemetry_attribute "my.user.agent" "$http_user_agent".

• required: false
• syntax: opentelemetry_attribute <key> <value>
• block: http, server, location

Enable propagation of distributed tracing headers, e.g. traceparent. When no parent trace is given, a new trace will be started. The default propagator is W3C.

The same inheritance rules as proxy_set_header apply, which means this directive is applied at the current configuration level if and only if there are no proxy_set_header directives defined on a lower level.

• required: false
• syntax: opentelemetry_propagate or opentelemetry_propagate b3 or opentelemetry_propagate b3multi
• block: http, server, location

Enables the capturing of request and response headers. (default: off).

• required: false
• syntax: opentelemetry_capture_headers on|off
• block: http, server, location

Sets the captured header value to [REDACTED] for all headers where the name matches the given regex (case insensitive).

• required: false
• syntax: opentelemetry_sensitive_header_names <regex>
• block: http, server, location

Sets the captured header value to [REDACTED] for all headers where the value matches the given regex (case insensitive).

• required: false
• syntax: opentelemetry_sensitive_header_values <regex>
• block: http, server, location

No span will be created for URIs matching the given regex (case insensitive).

• required: false
• syntax: opentelemetry_ignore_paths <regex>
• block: http, server, location

Only applicable when batch span processor is selected. Chooses the span batch exporting interval in milliseconds. (default: 5000)

• required: false
• syntax: opentelemetry_bsp_schedule_delay_millis <value>
• block: http

Only applicable when batch span processor is selected. Chooses the span export batch size. (default: 512)

• required: false
• syntax: opentelemetry_bsp_max_export_batch_size <value>
• block: http

Only applicable when batch span processor is selected. Chooses the span exporter queue size. (default: 2048)

• required: false
• syntax: opentelemetry_bsp_max_queue_size <value>
• block: http

List of exported attributes and their corresponding nginx variables if applicable:

• http.status_code
• http.method
• http.target
• http.flavor
• http.host - Host header value
• http.scheme - $scheme
• http.server_name - From the server_name directive
• http.user_agent - User-Agent header value
• http.request.header.* - The request headers (except Host and User-Agent)
• http.response.header.* - The response headers
• net.host.port - $server_port
• net.peer.ip - $remote_addr
• net.peer.port - $remote_port

The following nginx variables are set by the instrumentation:

• opentelemetry_context_traceparent - W3C trace context, e.g.: 00-0af7651916cd43dd8448eb211c80319c-b9c7c989f97918e1-01
• opentelemetry_context_b3 - Trace context in the B3 format. Only set when using opentelemetry_propagate b3.
• opentelemetry_sampled - does current Span records information, "1" or "0"
• opentelemetry_trace_id - Trace Id of the current span
• opentelemetry_span_id - Span Id of the current span

This can be used to add Server-Timing header:

add_header Server-Timing "traceparent;desc=\"$opentelemetry_context_traceparent\"";

Dependencies:

• Elixir
• Docker
• Docker Compose

In case you don't have elixir locally installed, you can run the mix commands inside a container:

docker run -it --rm -v $(pwd):/otel -v /var/run/docker.sock:/var/run/docker.sock -e TEST_ROOT=$(pwd)/test -w /otel elixir:1.11-alpine sh
apk --no-cache add docker-compose docker-cli

cd test/instrumentation
docker build -t otel-nginx-test/nginx -f ../Dockerfile ../..
docker build -t otel-nginx-test/express-backend -f ../backend/simple_express/Dockerfile ../backend/simple_express
mix test

Make sure your nginx is compiled with --with-compat (nginx -V).