Unable to use repository-azure plugin to connect to Azure storage account in Gov cloud

Elastic Stack Elasticsearch
1

We successfully use the repository-azure plugin to use azure blob storage as a snapshot repo and have been doing this for quite a while in multiple elasticsearch clusters on Azure AKS in their commercial cloud.

We have 2 elasticsearch clusters in Azure Gov and we recently noticed that neither is able to make use of the azure blob storage when configured as a snapshot repo.

We are getting the following error:

{
  "name": "ResponseError",
  "message": "repository_verification_exception\n\tCaused by:\n\t\tblob_storage_exception: blob_storage_exception: If you are using a StorageSharedKeyCredential, and the server returned an error message that says 'Signature did not match', you can compare the string to sign with the one generated by the SDK. To log the string to sign, pass in the context key value pair 'Azure-Storage-Log-String-To-Sign': true to the appropriate method call.\nIf you are using a SAS token, and the server returned an error message that says 'Signature did not match', you can compare the string to sign with the one generated by the SDK. To log the string to sign, pass in the context key value pair 'Azure-Storage-Log-String-To-Sign': true to the appropriate generateSas method call.\nPlease remember to disable 'Azure-Storage-Log-String-To-Sign' before going to production as this string can potentially contain PII.\nStatus code 403, \"<?xml version=\"1.0\" encoding=\"utf-8\"?><Error><Code>AuthorizationFailure</Code><Message>This request is not authorized to perform this operation.\nRequestId:93eaa554-101e-0015-0ff0-b95d77000000\nTime:2025-04-30T16:55:49.0032266Z</Message></Error>\"\n\tRoot causes:\n\t\tblob_storage_exception: blob_storage_exception: If you are using a StorageSharedKeyCredential, and the server returned an error message that says 'Signature did not match', you can compare the string to sign with the one generated by the SDK. To log the string to sign, pass in the context key value pair 'Azure-Storage-Log-String-To-Sign': true to the appropriate method call.\nIf you are using a SAS token, and the server returned an error message that says 'Signature did not match', you can compare the string to sign with the one generated by the SDK. To log the string to sign, pass in the context key value pair 'Azure-Storage-Log-String-To-Sign': true to the appropriate generateSas method call.\nPlease remember to disable 'Azure-Storage-Log-String-To-Sign' before going to production as this string can potentially contain PII.\nStatus code 403, \"<?xml version=\"1.0\" encoding=\"utf-8\"?><Error><Code>AuthorizationFailure</Code><Message>This request is not authorized to perform this operation.\nRequestId:93eaa554-101e-0015-0ff0-b95d77000000\nTime:2025-04-30T16:55:49.0032266Z</Message></Error>\""
}

We have made sure to set azure.client.default.endpoint_suffix: "core.usgovcloudapi.net" on the gov instances.

We are configuring elasticsearch to use the account and key for this storage account. I have confirmed that I can manually access this storage account using the key with curl but for whatever reason elasticsearch is not able to do this successfully. I suspect it has something to do with the repository-azure plugin and Gov but I don't yet have any way of proving this other than it doesn't work via elasticsearch but works fine for me manually.

This is on 8.15.3 running on k8s.

I have filed a bug but it was quickly closed unfortunately:

any help would be much appreciated

2

I can't comment on Azure Gov regions, but I've encountered this exact error on an Azure commercial region due to the Firewall config on the blob storage side.

3

I was hopeful but unfortunately it doesn't look like it's that simple. I set the firewall to "Enable from all networks" and it doesn't seem to making a difference, elaticsearch still can't communicate with the azure blob storage.