Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "otlp/gateway", "error": "rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 503 (Service Unavailable); transport: received unexpected content-type \"text/html; charset=UTF-8\

[plrrag]

Hi everyone,

I'm having problems with my opentelemetry infrastructure.
I have an OTAgent on server B which needs to send metrics to server A which works as OTGateway. The agent constantly receives the error

Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "otlp/gateway", "error": "rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 503 ( Service Unavailable); transport: received unexpected content-type "text/html; charset=UTF-8"", "interval": "19.948732417s"}
2024-07-02T16:31:46.786+0200 error zapgrpc/zapgrpc.go:210 [transport] Client received GoAway with error code ENHANCE_YOUR_CALM and debug data equal to ASCII "too_many_pings".

I already have 4 agents that send data to the Gateway and none have problems, the 5th and last server to monitor has the indicated problem.
The IP addresses are on two different classes but from the agent, via telnet, I reach the gateway at the specified port 4320

This is the GATEWAY config, the ipaddress is 172.X.Y.Z

receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4320
http:
endpoint: 0.0.0.0:4319

processors:
batch:
timeout: 1s

resource:
attributes:
- key: test-key
value: "gateway"
action: insert

attributes/trace:
actions:
- key: db.statement_type
action: delete
- key: db.system
action: delete
- key: internal.span.format
action: delete
- key: network.protocol.version
action: delete
- key: span.kind
action: delete
- key: http.route
action: delete

'exporters' specify configurations for certain exporters.

See the README for more information on the exporters available for configuration.

exporters:

The logging exporter; This exporter logs to stdout.

For more information on configuring the logging exporter, refer to the documentation here:

https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/loggingexporter

logging:
verbosity: detailed
sampling_initial: 5
sampling_thereafter: 5

prometheus:
endpoint: "0.0.0.0:9191"
namespace: Gateway
metric_expiration: 5m
enable_open_metrics: true
resource_to_telemetry_conversion:
enabled: true

otlp/jaeger: # Jaeger supports OTLP directly. The default port for OTLP/gRPC is 4317
endpoint: 0.0.0.0:4317
tls:
insecure: true

'service' specifies how to construct the data pipelines using the configurations above.

service:
pipelines:
traces:
receivers: [otlp]
processors: [batch,resource,attributes/trace]
exporters: [logging]

metrics:
  receivers: [otlp]
  processors: [batch]
  exporters: [prometheus]

This is the AGENT config, the server IPADDRESS is 10.X.Y.Z

receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317

  http:
    endpoint: 0.0.0.0:4318

hostmetrics:
collection_interval: 60s
scrapers:
cpu:
metrics:
system.cpu.utilization:
enabled: true
disk:
filesystem:
memory:
network:

processors:
batch:
timeout: 60s

#rielabora le metriche e trace aggiungendo come attributo l'hostname
resource:
attributes:
- key: hostname
action: insert
value: "serverB"

exporters:
otlp/gateway:
endpoint: <IP GATEWAY ON 172.X.Y.Z>:4320
tls:
insecure: true
keepalive:
time: 300 # Intervallo tra ping in secondi (5 minuti)
timeout: 20 # Timeout per la risposta del ping in secondi
permit_without_stream: true # Permettere ping anche senza stream attivo

prometheus:
endpoint: "0.0.0.0:9191"
namespace: Gateway
metric_expiration: 5m
enable_open_metrics: true
resource_to_telemetry_conversion:
enabled: true

logging:
verbosity: detailed
sampling_initial: 5
sampling_thereafter: 5

service:
pipelines:
traces:
receivers: [otlp]
processors: [batch,resource]
exporters: [logging]

metrics:
  receivers: [otlp,hostmetrics]
  processors: [batch,resource]
  exporters: [otlp/gateway,prometheus]
  #exporters: [prometheus]

Do you have any suggestions?

[trask]

hi @plrrag, I'd suggest asking in the https://github.com/open-telemetry/opentelemetry-collector repo