[Azure] Add AMQP-over-WebSockets support to Event Hub integrations - #20559
Conversation
✅ Elastic Docs Style Checker (Vale)No issues found on modified lines! The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale. |
TL;DR
Remediation
Investigation detailsRoot CauseThis is a configuration/docs sync failure (not a runtime code bug). The lint step compares generated README content against the committed file and detected drift. In this PR, the template and rendered docs changed near the file end, including EOF/newline handling:
The Buildkite check reports the generated output wants extra trailing lines that are not present in the committed README. Evidence
Verification
Follow-upIf this exact mismatch reappears after regeneration, compare end-of-file newline/trailing blank-line handling between What is this? | From workflow: PR Buildkite Detective Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not. |
The new `transport` option is read only by the v2 Event Hub processor (`CreateEventHubConsumerClient` is called exclusively from `v2_input.go`; the v1 input uses the legacy azure-event-hubs-go SDK and ignores it). azure_ai_foundry, azure_openai, azure_functions and azure_app_service never emitted `processor_version`, so Beats applied its default, which is v1 on 8.19, 9.1 and 9.2. Selecting AMQP-over-WebSockets on those versions silently kept dialing AMQP on 5671/5672. Add a `processor_version` var defaulting to v2 and emit it from the agent templates, matching azure and azure_logs. Also fix the docs: name the processor version requirement explicitly and correct the supported agent versions per package (azure_ai_foundry cannot be installed before 9.2.4, azure_openai has no 9.1.x). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Bring azure_ai_foundry, azure_openai, azure_functions and azure_app_service
in line with azure and azure_logs by exposing the rest of the processor v2
settings: processor_update_interval, processor_start_position,
partition_receive_timeout, partition_receive_count and migrate_checkpoint.
migrate_checkpoint is emitted unconditionally rather than behind an
`{{#if}}` guard so that setting it to false is honoured; a guarded bool
renders nothing when false and the Beats default (true) would apply.
endpoint_suffix is deliberately left out: it is not an input option, only
a fragment of the storage_account_connection_string that azure and
azure_logs assemble by hand. These packages let Beats build that string,
and Beats derives the suffix from authority_host, which they already
expose.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
🚀 Benchmarks reportPackage
|
| Data stream | Previous EPS | New EPS | Diff (%) | Result |
|---|---|---|---|---|
logs |
1709.4 | 1308.9 | -400.5 (-23.43%) | 💔 |
To see the full report comment with /test benchmark fullreport
There was a problem hiding this comment.
Pull request overview
This PR adds AMQP-over-WebSockets support (and optional proxy routing via HTTPS_PROXY) to the Azure Event Hubs–based integrations listed in the description, by introducing a transport option (amqp default, websocket for port 443) and exposing processor v2 tuning/migration settings where they were previously unavailable. It also updates package docs/changelogs and bumps package versions accordingly.
Changes:
- Add
transportvar (AMQP vs AMQP-over-WebSockets) and render it inazure-eventhubinput templates. - Expose processor v2 configuration options (including
processor_versiondefaulting tov2andmigrate_checkpoint) in Azure OpenAI / Azure Functions / Microsoft Foundry / Azure App Service. - Update documentation (firewall + proxy requirements) and bump package versions with changelog entries.
Reviewed changes
Copilot reviewed 28 out of 28 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/azure_openai/manifest.yml | Bump package version to ship new Event Hubs transport/processor v2 options. |
| packages/azure_openai/docs/README.md | Document AMQP-over-WebSockets and proxy requirements for Event Hubs ingestion. |
| packages/azure_openai/data_stream/logs/manifest.yml | Add processor v2 vars + transport var to the logs data stream configuration. |
| packages/azure_openai/data_stream/logs/agent/stream/azure-eventhub.yml.hbs | Render processor v2 vars + transport into the generated azure-eventhub input config. |
| packages/azure_openai/changelog.yml | Add release notes for the new transport/proxy/processor v2 options. |
| packages/azure_openai/_dev/build/docs/README.md | Propagate docs updates into the built docs output. |
| packages/azure_logs/manifest.yml | Bump version and add transport var to Custom Azure Logs integration settings. |
| packages/azure_logs/docs/README.md | Update firewall guidance; document AMQP-over-WebSockets and proxy behavior. |
| packages/azure_logs/changelog.yml | Add release notes for transport/proxy support. |
| packages/azure_logs/agent/input/input.yml.hbs | Render transport into the generated azure-eventhub input config. |
| packages/azure_logs/_dev/build/docs/README.md | Propagate docs updates into the built docs output. |
| packages/azure_functions/manifest.yml | Bump package version to include transport/proxy + processor v2 settings. |
| packages/azure_functions/docs/README.md | Document AMQP-over-WebSockets and proxy requirements + processor v2 settings. |
| packages/azure_functions/data_stream/functionapplogs/manifest.yml | Add processor v2 vars + transport var to the Function App Logs data stream. |
| packages/azure_functions/data_stream/functionapplogs/agent/stream/azure-eventhub.yml.hbs | Render processor v2 vars + transport into the azure-eventhub stream config. |
| packages/azure_functions/changelog.yml | Add release notes for the new transport/proxy/processor v2 options. |
| packages/azure_functions/_dev/build/docs/README.md | Propagate docs updates into the built docs output. |
| packages/azure_app_service/manifest.yml | Bump version and add processor v2 vars + transport at the package var level. |
| packages/azure_app_service/docs/README.md | Document AMQP-over-WebSockets and proxy requirements. |
| packages/azure_app_service/data_stream/app_service_logs/agent/stream/azure-eventhub.yml.hbs | Render processor v2 vars + transport into the azure-eventhub stream config. |
| packages/azure_app_service/changelog.yml | Add release notes for the new transport/proxy/processor v2 options. |
| packages/azure_app_service/_dev/build/docs/README.md | Propagate docs updates into the built docs output. |
| packages/azure_ai_foundry/manifest.yml | Bump package version to include transport/proxy + processor v2 settings. |
| packages/azure_ai_foundry/docs/README.md | Document AMQP-over-WebSockets and proxy requirements. |
| packages/azure_ai_foundry/data_stream/logs/manifest.yml | Add processor v2 vars + transport var to the logs data stream configuration. |
| packages/azure_ai_foundry/data_stream/logs/agent/stream/azure-eventhub.yml.hbs | Render processor v2 vars + transport into the generated azure-eventhub input config. |
| packages/azure_ai_foundry/changelog.yml | Add release notes for the new transport/proxy/processor v2 options. |
| packages/azure_ai_foundry/_dev/build/docs/README.md | Propagate docs updates into the built docs output. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
|
✅ All changelog entries have the correct PR link. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.
Suppressed comments (4)
packages/azure_openai/data_stream/logs/agent/stream/azure-eventhub.yml.hbs:51
migrate_checkpointis a boolean flag, but wrapping it in{{#if migrate_checkpoint}}means the config line is omitted when the user sets it tofalse. That makes it impossible to explicitly disable checkpoint migration (it will fall back to the input default). Render the key unconditionally so bothtrueandfalseare respected.
{{#if migrate_checkpoint}}
migrate_checkpoint: {{migrate_checkpoint}}
{{/if}}
packages/azure_functions/data_stream/functionapplogs/agent/stream/azure-eventhub.yml.hbs:51
migrate_checkpointis a boolean flag, but wrapping it in{{#if migrate_checkpoint}}means the config line is omitted when the user sets it tofalse. That makes it impossible to explicitly disable checkpoint migration (it will fall back to the input default). Render the key unconditionally so bothtrueandfalseare respected.
{{#if migrate_checkpoint}}
migrate_checkpoint: {{migrate_checkpoint}}
{{/if}}
packages/azure_app_service/data_stream/app_service_logs/agent/stream/azure-eventhub.yml.hbs:51
migrate_checkpointis a boolean flag, but wrapping it in{{#if migrate_checkpoint}}means the config line is omitted when the user sets it tofalse. That makes it impossible to explicitly disable checkpoint migration (it will fall back to the input default). Render the key unconditionally so bothtrueandfalseare respected.
{{#if migrate_checkpoint}}
migrate_checkpoint: {{migrate_checkpoint}}
{{/if}}
packages/azure_ai_foundry/data_stream/logs/agent/stream/azure-eventhub.yml.hbs:51
migrate_checkpointis a boolean flag, but wrapping it in{{#if migrate_checkpoint}}means the config line is omitted when the user sets it tofalse. That makes it impossible to explicitly disable checkpoint migration (it will fall back to the input default). Render the key unconditionally so bothtrueandfalseare respected.
{{#if migrate_checkpoint}}
migrate_checkpoint: {{migrate_checkpoint}}
{{/if}}
💚 Build Succeeded
History
|
|
Package azure_ai_foundry - 0.13.0 containing this change is available at https://epr.elastic.co/package/azure_ai_foundry/0.13.0/ |
|
Package azure_app_service - 1.2.0 containing this change is available at https://epr.elastic.co/package/azure_app_service/1.2.0/ |
|
Package azure_functions - 0.16.0 containing this change is available at https://epr.elastic.co/package/azure_functions/0.16.0/ |
|
Package azure_logs - 0.7.0 containing this change is available at https://epr.elastic.co/package/azure_logs/0.7.0/ |
|
Package azure_openai - 1.15.0 containing this change is available at https://epr.elastic.co/package/azure_openai/1.15.0/ |
Closes #20570
Proposed commit message
Add AMQP-over-WebSockets transport support to the Azure Event Hub integrations (Azure OpenAI, Azure Functions, Microsoft Foundry, Azure App Service, and Custom Azure Logs).
transportdefaults toamqp;websockettunnels AMQP over HTTPS port443, enabling Event Hubs traffic through restrictive firewalls andHTTPS_PROXYproxies.Because
transportis a processor v2 option, this also exposes the processor v2 settings in the four packages that lacked them and flips their default processor version from v1 to v2 (matchingazure, #16955). On upgrade,migrate_checkpoint(defaulttrue) migrates v1 checkpoints once, so consumption resumes where v1 left off.This was requested for exactly these five integrations: Azure OpenAI, Azure Functions, Microsoft Foundry, Azure App Service, and Custom Azure Logs (which already used processor v2). Existing kibana constraints already require a stack supporting
transport(8.19.10 / 9.1.10 / 9.2.4+, per package), so no constraint changes are needed.Changes
transportselector (amqpdefault;websocketon port443) in eachazure-eventhubinput.processor_version,migrate_checkpoint,processor_update_interval,processor_start_position,partition_receive_timeout,partition_receive_count) and default to v2 for Azure OpenAI, Azure Functions, Microsoft Foundry, and Azure App Service; Custom Azure Logs already had them.HTTPS_PROXYenvironment variable and that it requireswebsockettransport.Impact on existing installations
The four packages above move from processor v1 to v2 on upgrade; the automatic v1→v2 checkpoint migration preserves the consumer position.
Verification
elastic-package buildandelastic-package checkfor all five packages.git diff --check.SVR00002; that dashboard is unchanged.Related issues