Configurability of OpenTelemetry instrumentation

[trentm]

🚀 Feature Proposal

Native OpenTelemetry instrumentation is being added as part of #2267
From review of part of that work:

It would be good to provide some configurability of the OTel instrumentation.
Similar to how the older diagnostics events can be configured via TransportOptions.diagnostic, there are a couple things that might be nice to be configurable:

1. A way to disable the OTel instrumentation. Users of an OTel SDK will be somewhat used to the ability to disable particular instrumentations. Usually this config var is called enabled, in the context of an instrumentation.
2. A way to suppress child HTTP spans that will be created under the Elasticsearch spans. There are a few existing OTel instrumentations that have this same situation, e.g.: instrumentation-aws-sdk, instrumentation-mongoose. Typically this config var is a boolean called suppressInternalInstrumentation.

AFAIK there aren't any other Node.js libraries that have native OTel instrumentation yet, so there isn't prior art on exactly what to name OTel-related config. Perhaps having this in @elastic/transport:

export interface TransportOptions {
  opentelemetry?: {
    enabled?: boolean;
    suppressInternalInstrumentation?: boolean
  }
  // ...

and the equivalent in interface ClientOptions in @elastic/elasticsearch.

I'm suggesting they get grouped under a "opentelemetry." namespace, because there could be requests for more configuration options as well. For example, see the instrumentation-aws-sdk and instrumentation-mongoose links above.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove the stale label, or leave a comment, or this will be closed in 14 days.

[elrob]

Just came across the issue of not seemingly being able to disable the elasticsearch instrumentation.
It looks like the java lib has the option to disable with OTEL_INSTRUMENTATION_ELASTICSEARCH_ENABLED env var. (ref)
I tried this with no success with the node/js lib.

[JoshMock]

Semi-related: #2583

[trentm]

@JoshMock Another thing to consider, is being able to disable dynamically. We are looking at central config support for our EDOT SDKs and one of the considered central config options is a way to disable instrumentations once the app is running.

However, I'm fine with that being a separate issue (let me know). Mostly I think there is a clear need for being able to statically disable the native OTel instrumentation, so getting something for that is a good first step.

Also let me know if you'd like an API proposal for what dynamic config might look like. I don't know if the ES Client already has one or more methods for re-configuring on the fly.

[trentm]

Updated ref to Java ES client configurability of its OTel instrumentation: https://www.elastic.co/docs/reference/elasticsearch/clients/java/setup/opentelemetry#_configuring_the_opentelemetry_instrumentation

[JoshMock]

I don't know if the ES Client already has one or more methods for re-configuring on the fly.

It doesn't support changing the client's base options on the fly once instantiated. However, you can set a lot of options on a per request basis, by passing a TransportRequestOptions object as the second argument to any API function. I don't think it would be difficult to add an OTel toggle option to that object, but I'll have to look more closely before verifying that.