[Tables] Delete Entity Returning Unauthorized, Query Entities Not

• Package Name: azure-data-tables (reproducable in azure-cosmosdb-tables)
• Package Version: 12.0.0b3
• Operating System: Windows 10
• Python Version: 3.8.6

Describe the bug
After creating a TableClient from azure.data.tables.aio, a call to table_client.query_entities is successful. Just a few lines later, a call to table_client.delete_entity is not, and returns a ClientAutheniticationError: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.

I've tried this with azure.cosmosdb.table as well: Using similar calls, the query call is successful, but the delete call returns an AzureSigningError: Incorrect padding.

I have verified that the key is 88 characters long, ends with ==, and is DIRECTLY copied from the storage account in question. It obviously works for the query_entities call.

To Reproduce
Steps to reproduce the behavior:

1. Create an Azure Storage Account, add a table to it, and name the table "Locator". Add a few entities with a column called LocatorHash, with any string value you like.
2. Run the following code:

    from azure.data.tables.aio import TableClient
    from azure.core import MatchingConditions

    table_name = 'Locator'
    email_hash = '...' #Can be any string value contained in the LocatorHash column
    table_client = TableClient(account_url='https://<YOURACCOUNTNAME>.table.core.windows.net', table_name=table_name, credential=os.environ['STORAGE_ACCOUNT_ACCESS_KEY'])
    async with table_client:
        locator_entities = table_client.query_entities(filter=f"LocatorHash eq '{email_hash}'")
        coroutines = []
        async for locator_entity in locator_entities:
            logging.info(f"Deleting record: {locator_entity}")
            # error occurs on this line
            await table_client.delete_entity(row_key=locator_entity['RowKey'], partition_key=locator_entity['PartitionKey'], etag=locator_entity['_metadata']['etag'], match_condition=MatchConditions.Unconditionally)

Expected behavior
Entities are deleted rather than receiving an AuthError.

Screenshots

When using azure.data.tables.aio

When using azure.cosmosdb.table -- note, had to edit code to be synchronous

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @southpolesteve, @zfoster

[annatisch]

Thanks @sejohnson-at-griffis - Could you please try removing the etag and match_condition parameters from the delete call? Unconditionally is the default behaviour, so it should work as expected.
I have a suspicion this error is because the etag value is not a valid combination with Unconditionally - and rather than catching it we maybe sending the extra header to the service and causing request signing issues....

Thanks @sejohnson-at-griffis - Could you please try removing the etag and match_condition parameters from the delete call? Unconditionally is the default behaviour, so it should work as expected.
I have a suspicion this error is because the etag value is not a valid combination with Unconditionally - and rather than catching it we maybe sending the extra header to the service and causing request signing issues....

Hey there,

Thanks for the suggestion. Unfortunately, same error. I've actually tried just about every combination of those parameters I can think of... and I can't think of a reason the etag would cause an auth error anyway, as it's not part of the auth header in the first place, right?

I'm also struggling to find the actual documentation for this library anywhere. All I can find is a brief one-page document.

I tried using a batched execution mode as well (code will follow). The code here actually completes execution successfully... but does not delete the entities from the table! (????) Is it just pretending to delete them and calling it a day?

Notes:
I have to build a list out of the return from query_entities because query_entities returns an "AsyncItemPaged", which I cannot filter or iterate over except with an async for loop.

Build a set (distinct list) of PartitionKeys (as batches can only cover one PartitionKey at a time).

For each PartitionKey in this batch, delete the entities belonging to it.

async with table_client:
        # try:
        locator_entities = table_client.query_entities(filter=f"LocatorHash eq '{email_hash}'")
        locator_entity_list = []
        async for locator_entity in locator_entities:
            locator_entity_list.append(locator_entity)
        for key in { entity['PartitionKey'] for entity in locator_entity_list }:
            batch = table_client.create_batch()
            for locator_entity in filter(lambda x: x['PartitionKey'] == key, locator_entity_list):
                delete_entity = {}
                delete_entity['row_key'] = locator_entity['RowKey']
                delete_entity['partition_key'] = locator_entity['PartitionKey']
                batch.delete_entity(row_key=locator_entity['RowKey'], partition_key=locator_entity['PartitionKey'])
            table_client.send_batch(batch)

[seankane-msft]

Here is the documentation for the async library: https://docs.microsoft.com/en-us/python/api/azure-data-tables/azure.data.tables.aio.tableclient?view=azure-python-preview

Here is the documentation for the sync library: https://docs.microsoft.com/en-us/python/api/azure-data-tables/azure.data.tables?view=azure-python-preview

I worked with your initial code and I was able to confirm that deletes were happening. Here is the full snippet that I was able to get working:

from azure.data.tables.aio import TableClient
from azure.core import MatchConditions
from azure.core.exceptions import ResourceExistsError
import random, string, asyncio, logging

def random_string(N=50):
    return ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(N))

async def delete_entities():
    base_entity = {
        "PartitionKey": "pk001",
        "RowKey": "rk000",
        "LocatorHash": ""
    }

    table_name = 'Locator'
    email_hash = "" #Can be any string value contained in the LocatorHash column
    table_client = TableClient(account_url=f'https://{account_name}.table.core.windows.net', table_name=table_name, credential=credential)
    try:
        await table_client.create_table()
    except ResourceExistsError:
        pass

    for i in range(25):
        try:
            new_entity = base_entity.copy()
            new_entity["LocatorHash"] = random_string()
            new_entity["RowKey"] += str(i)
            await table_client.upsert_entity(new_entity)
            email_hash = new_entity["LocatorHash"]
        except ResourceExistsError:
            pass

    async with table_client:
        f = f"LocatorHash eq '{email_hash}'"
        locator_entities = table_client.query_entities(filter=f)

        async for locator_entity in locator_entities:
            print(f"Deleting record: {locator_entity['PartitionKey']}, {locator_entity['RowKey']}")
            # error occurs on this line
            await table_client.delete_entity(row_key=locator_entity['RowKey'], partition_key=locator_entity['PartitionKey'], etag=locator_entity['_metadata']['etag'], match_condition=MatchConditions.Unconditionally)

async def main():
    loop = asyncio.new_event_loop()
    asyncio.set_event_loop(asyncio.new_event_loop())
    loop = asyncio.get_event_loop()
    loop.run_until_complete(await delete_entities())

if __name__ == "__main__":
    asyncio.run(delete_entities())

It's possible I'm misunderstanding this code, in which case please clarify for me. Otherwise, this code is functional and I was able to confirm in the Storage Explorer that the last inserted entity was then deleted.

[seankane-msft]

In addition @sejohnson-at-griffis can you confirm the environment variable is correctly stored? I've accidentally stored env vars with quotes around them and that will cause issues on authentication.

[seankane-msft]

batch.delete_entity(row_key=locator_entity['RowKey'], partition_key=locator_entity['PartitionKey'])
table_client.send_batch(batch)

@sejohnson-at-griffis In this last line the send_batch needs an await before it, that is likely the reason the async batching was not working either.

@seankane-msft

Looks like adding the "await" operator to the batch call worked -- the records were successfully deleted. I'd prefer not to have to use the acrobatics required to group the query return into a set of partition keys, but I still can't get the original method (the same one you used) to work. Not sure why. Obviously my key/environment variable is working correctly if I can use the batch calls...

I've even directly copied your code from async with table_client down to the delete call. Same error. Notably, the error still doesn't occur until the delete call -- the query completes successfully.