From f193e92572cf839e90f15485e6c8bf2b19c3d317 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Wed, 7 Oct 2020 18:36:54 +0200 Subject: [PATCH 1/9] chore(deps): update dependency google-cloud-bigtable to v1.5.1 (#152) --- samples/beam/requirements.txt | 2 +- samples/hello/requirements.txt | 2 +- samples/instanceadmin/requirements.txt | 2 +- samples/metricscaler/requirements.txt | 2 +- samples/quickstart/requirements.txt | 2 +- samples/snippets/filters/requirements.txt | 2 +- samples/snippets/reads/requirements.txt | 2 +- samples/snippets/writes/requirements.txt | 2 +- samples/tableadmin/requirements.txt | 2 +- 9 files changed, 9 insertions(+), 9 deletions(-) diff --git a/samples/beam/requirements.txt b/samples/beam/requirements.txt index 3628e62e5..a5ff4714c 100644 --- a/samples/beam/requirements.txt +++ b/samples/beam/requirements.txt @@ -1,3 +1,3 @@ apache-beam==2.24.0 -google-cloud-bigtable==1.5.0 +google-cloud-bigtable==1.5.1 google-cloud-core==1.4.2 \ No newline at end of file diff --git a/samples/hello/requirements.txt b/samples/hello/requirements.txt index 185b56b7e..770185a05 100644 --- a/samples/hello/requirements.txt +++ b/samples/hello/requirements.txt @@ -1,2 +1,2 @@ -google-cloud-bigtable==1.5.0 +google-cloud-bigtable==1.5.1 google-cloud-core==1.4.2 diff --git a/samples/instanceadmin/requirements.txt b/samples/instanceadmin/requirements.txt index f9c658397..23b27ea70 100755 --- a/samples/instanceadmin/requirements.txt +++ b/samples/instanceadmin/requirements.txt @@ -1 +1 @@ -google-cloud-bigtable==1.5.0 +google-cloud-bigtable==1.5.1 diff --git a/samples/metricscaler/requirements.txt b/samples/metricscaler/requirements.txt index 471385ad0..f0ff9c764 100644 --- a/samples/metricscaler/requirements.txt +++ b/samples/metricscaler/requirements.txt @@ -1,2 +1,2 @@ -google-cloud-bigtable==1.5.0 +google-cloud-bigtable==1.5.1 google-cloud-monitoring==1.1.0 diff --git a/samples/quickstart/requirements.txt b/samples/quickstart/requirements.txt index f9c658397..23b27ea70 100644 --- a/samples/quickstart/requirements.txt +++ b/samples/quickstart/requirements.txt @@ -1 +1 @@ -google-cloud-bigtable==1.5.0 +google-cloud-bigtable==1.5.1 diff --git a/samples/snippets/filters/requirements.txt b/samples/snippets/filters/requirements.txt index 0e902be53..ac34bb1f7 100755 --- a/samples/snippets/filters/requirements.txt +++ b/samples/snippets/filters/requirements.txt @@ -1,2 +1,2 @@ -google-cloud-bigtable==1.5.0 +google-cloud-bigtable==1.5.1 snapshottest==0.6.0 \ No newline at end of file diff --git a/samples/snippets/reads/requirements.txt b/samples/snippets/reads/requirements.txt index 0e902be53..ac34bb1f7 100755 --- a/samples/snippets/reads/requirements.txt +++ b/samples/snippets/reads/requirements.txt @@ -1,2 +1,2 @@ -google-cloud-bigtable==1.5.0 +google-cloud-bigtable==1.5.1 snapshottest==0.6.0 \ No newline at end of file diff --git a/samples/snippets/writes/requirements.txt b/samples/snippets/writes/requirements.txt index 67be6a0ef..168f72e91 100755 --- a/samples/snippets/writes/requirements.txt +++ b/samples/snippets/writes/requirements.txt @@ -1 +1 @@ -google-cloud-bigtable==1.5.0 \ No newline at end of file +google-cloud-bigtable==1.5.1 \ No newline at end of file diff --git a/samples/tableadmin/requirements.txt b/samples/tableadmin/requirements.txt index f9c658397..23b27ea70 100755 --- a/samples/tableadmin/requirements.txt +++ b/samples/tableadmin/requirements.txt @@ -1 +1 @@ -google-cloud-bigtable==1.5.0 +google-cloud-bigtable==1.5.1 From 992d641aaa718d8dbdb0c9fed138e42a8f485b24 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Wed, 7 Oct 2020 19:06:04 +0200 Subject: [PATCH 2/9] chore(deps): update dependency google-cloud-core to v1.4.3 (#153) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [google-cloud-core](https://togithub.com/googleapis/python-cloud-core) | patch | `==1.4.2` -> `==1.4.3` | --- ### Release Notes
googleapis/python-cloud-core ### [`v1.4.3`](https://togithub.com/googleapis/python-cloud-core/blob/master/CHANGELOG.md#​143-httpswwwgithubcomgoogleapispython-cloud-corecomparev142v143-2020-10-06) [Compare Source](https://togithub.com/googleapis/python-cloud-core/compare/v1.4.2...v1.4.3)
--- ### Renovate configuration :date: **Schedule**: At any time (no schedule defined). :vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied. :recycle: **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. :no_bell: **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/python-bigtable). --- samples/beam/requirements.txt | 2 +- samples/hello/requirements.txt | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/samples/beam/requirements.txt b/samples/beam/requirements.txt index a5ff4714c..5ec8ddb2c 100644 --- a/samples/beam/requirements.txt +++ b/samples/beam/requirements.txt @@ -1,3 +1,3 @@ apache-beam==2.24.0 google-cloud-bigtable==1.5.1 -google-cloud-core==1.4.2 \ No newline at end of file +google-cloud-core==1.4.3 \ No newline at end of file diff --git a/samples/hello/requirements.txt b/samples/hello/requirements.txt index 770185a05..3360cf133 100644 --- a/samples/hello/requirements.txt +++ b/samples/hello/requirements.txt @@ -1,2 +1,2 @@ google-cloud-bigtable==1.5.1 -google-cloud-core==1.4.2 +google-cloud-core==1.4.3 From 02783630c28de3d1bc3e17be67c6fa87e8e64ef0 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Thu, 8 Oct 2020 18:10:26 +0200 Subject: [PATCH 3/9] chore(deps): update dependency google-cloud-monitoring to v2 (#154) --- samples/metricscaler/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/samples/metricscaler/requirements.txt b/samples/metricscaler/requirements.txt index f0ff9c764..aa1a6e220 100644 --- a/samples/metricscaler/requirements.txt +++ b/samples/metricscaler/requirements.txt @@ -1,2 +1,2 @@ google-cloud-bigtable==1.5.1 -google-cloud-monitoring==1.1.0 +google-cloud-monitoring==2.0.0 From ff3c57773ed34c41490f19871c4cc12ddc196cc0 Mon Sep 17 00:00:00 2001 From: Tres Seaver Date: Wed, 11 Nov 2020 17:11:20 -0500 Subject: [PATCH 4/9] chore: clean up synth replacements (#161) - Remove those which no longer match. - Apply '_GAPIC_LIBRARY_VERSION' tweak from PR #150. Closes #155. Closes #156. --- synth.py | 50 +++++++++++++++++++------------------------------- 1 file changed, 19 insertions(+), 31 deletions(-) diff --git a/synth.py b/synth.py index 8a2fed1c7..21100c749 100644 --- a/synth.py +++ b/synth.py @@ -45,40 +45,28 @@ s.move(library / "google/cloud/bigtable_admin_v2") s.move(library / "tests") -s.replace( - [ - "google/cloud/bigtable_admin_v2/gapic/bigtable_instance_admin_client.py", - "google/cloud/bigtable_admin_v2/gapic/bigtable_table_admin_client.py", - ], - "'google-cloud-bigtable-admin'", - "'google-cloud-bigtable'", -) - -s.replace( - "google/**/*.py", - "from google\.cloud\.bigtable\.admin_v2.proto", - "from google.cloud.bigtable_admin_v2.proto", -) +# Work around non-standard installations -s.replace( - ["google/cloud/bigtable_admin_v2/__init__.py"], - " __doc__ = bigtable_instance_admin_client." - "BigtableInstanceAdminClient.__doc__\n", - " __doc__ = (\n" - " bigtable_instance_admin_client.BigtableInstanceAdminClient." - "__doc__)\n", -) - -s.replace( - ["google/cloud/bigtable_v2/gapic/bigtable_client.py"], - "if ``true_mutations`` is empty, and at most\n\n\s*100000.", - "if ``true_mutations`` is empty, and at most 100000.", -) +admin_clients = [ + "google/cloud/bigtable_admin_v2/gapic/bigtable_instance_admin_client.py", + "google/cloud/bigtable_admin_v2/gapic/bigtable_table_admin_client.py", +] s.replace( - ["google/cloud/bigtable_v2/gapic/bigtable_client.py"], - "if ``false_mutations`` is empty, and at most\n\n\s*100000.", - "if ``false_mutations`` is empty, and at most 100000.", + admin_clients, + """\ +_GAPIC_LIBRARY_VERSION = pkg_resources.get_distribution\( + 'google-cloud-bigtable-admin', +\).version +""", + """\ +try: + _GAPIC_LIBRARY_VERSION = pkg_resources.get_distribution( + "google-cloud-bigtable" + ).version +except pkg_resources.DistributionNotFound: # pragma: NO COVER + _GAPIC_LIBRARY_VERSION = None +""" ) # ---------------------------------------------------------------------------- From 44932cb8710e12279dbd4e9271577f8bee238980 Mon Sep 17 00:00:00 2001 From: MF2199 <38331387+mf2199@users.noreply.github.com> Date: Thu, 12 Nov 2020 12:08:21 -0500 Subject: [PATCH 5/9] feat: Backup Level IAM (#160) --- google/cloud/bigtable/backup.py | 52 ++++++++++++++++ tests/unit/test_backup.py | 107 ++++++++++++++++++++++++++++++++ 2 files changed, 159 insertions(+) diff --git a/google/cloud/bigtable/backup.py b/google/cloud/bigtable/backup.py index 03a1c894e..291ac783a 100644 --- a/google/cloud/bigtable/backup.py +++ b/google/cloud/bigtable/backup.py @@ -21,6 +21,7 @@ BigtableTableAdminClient, ) from google.cloud.bigtable_admin_v2.types import table_pb2 +from google.cloud.bigtable.policy import Policy from google.cloud.exceptions import NotFound from google.protobuf import field_mask_pb2 @@ -392,3 +393,54 @@ def restore(self, table_id): """ api = self._instance._client.table_admin_client return api.restore_table(self._instance.name, table_id, self.name) + + def get_iam_policy(self): + """Gets the IAM access control policy for this backup. + + :rtype: :class:`google.cloud.bigtable.policy.Policy` + :returns: The current IAM policy of this backup. + """ + table_api = self._instance._client.table_admin_client + args = {"resource": self.name} + response = table_api.get_iam_policy(**args) + return Policy.from_pb(response) + + def set_iam_policy(self, policy): + """Sets the IAM access control policy for this backup. Replaces any + existing policy. + + For more information about policy, please see documentation of + class `google.cloud.bigtable.policy.Policy` + + :type policy: :class:`google.cloud.bigtable.policy.Policy` + :param policy: A new IAM policy to replace the current IAM policy + of this backup. + + :rtype: :class:`google.cloud.bigtable.policy.Policy` + :returns: The current IAM policy of this backup. + """ + table_api = self._instance._client.table_admin_client + response = table_api.set_iam_policy(resource=self.name, policy=policy.to_pb()) + return Policy.from_pb(response) + + def test_iam_permissions(self, permissions): + """Tests whether the caller has the given permissions for this backup. + Returns the permissions that the caller has. + + :type permissions: list + :param permissions: The set of permissions to check for + the ``resource``. Permissions with wildcards (such as '*' + or 'storage.*') are not allowed. For more information see + `IAM Overview + `_. + `Bigtable Permissions + `_. + + :rtype: list + :returns: A List(string) of permissions allowed on the backup. + """ + table_api = self._instance._client.table_admin_client + response = table_api.test_iam_permissions( + resource=self.name, permissions=permissions + ) + return list(response.permissions) diff --git a/tests/unit/test_backup.py b/tests/unit/test_backup.py index 2f263dffd..0285d668b 100644 --- a/tests/unit/test_backup.py +++ b/tests/unit/test_backup.py @@ -734,6 +734,113 @@ def test_restore_success(self): backup=self.BACKUP_NAME, ) + def test_get_iam_policy(self): + from google.cloud.bigtable.client import Client + from google.cloud.bigtable_admin_v2.gapic import bigtable_table_admin_client + from google.iam.v1 import policy_pb2 + from google.cloud.bigtable.policy import BIGTABLE_ADMIN_ROLE + + credentials = _make_credentials() + client = Client(project=self.PROJECT_ID, credentials=credentials, admin=True) + + instance = client.instance(instance_id=self.INSTANCE_ID) + backup = self._make_one(self.BACKUP_ID, instance, cluster_id=self.CLUSTER_ID) + + version = 1 + etag = b"etag_v1" + members = ["serviceAccount:service_acc1@test.com", "user:user1@test.com"] + bindings = [{"role": BIGTABLE_ADMIN_ROLE, "members": members}] + iam_policy = policy_pb2.Policy(version=version, etag=etag, bindings=bindings) + + table_api = mock.create_autospec( + bigtable_table_admin_client.BigtableTableAdminClient + ) + client._table_admin_client = table_api + table_api.get_iam_policy.return_value = iam_policy + + result = backup.get_iam_policy() + + table_api.get_iam_policy.assert_called_once_with(resource=backup.name) + self.assertEqual(result.version, version) + self.assertEqual(result.etag, etag) + + admins = result.bigtable_admins + self.assertEqual(len(admins), len(members)) + for found, expected in zip(sorted(admins), sorted(members)): + self.assertEqual(found, expected) + + def test_set_iam_policy(self): + from google.cloud.bigtable.client import Client + from google.cloud.bigtable_admin_v2.gapic import bigtable_table_admin_client + from google.iam.v1 import policy_pb2 + from google.cloud.bigtable.policy import Policy + from google.cloud.bigtable.policy import BIGTABLE_ADMIN_ROLE + + credentials = _make_credentials() + client = Client(project=self.PROJECT_ID, credentials=credentials, admin=True) + + instance = client.instance(instance_id=self.INSTANCE_ID) + backup = self._make_one(self.BACKUP_ID, instance, cluster_id=self.CLUSTER_ID) + + version = 1 + etag = b"etag_v1" + members = ["serviceAccount:service_acc1@test.com", "user:user1@test.com"] + bindings = [{"role": BIGTABLE_ADMIN_ROLE, "members": sorted(members)}] + iam_policy_pb = policy_pb2.Policy(version=version, etag=etag, bindings=bindings) + + table_api = mock.create_autospec( + bigtable_table_admin_client.BigtableTableAdminClient + ) + client._table_admin_client = table_api + table_api.set_iam_policy.return_value = iam_policy_pb + + iam_policy = Policy(etag=etag, version=version) + iam_policy[BIGTABLE_ADMIN_ROLE] = [ + Policy.user("user1@test.com"), + Policy.service_account("service_acc1@test.com"), + ] + + result = backup.set_iam_policy(iam_policy) + + table_api.set_iam_policy.assert_called_once_with( + resource=backup.name, policy=iam_policy_pb + ) + self.assertEqual(result.version, version) + self.assertEqual(result.etag, etag) + + admins = result.bigtable_admins + self.assertEqual(len(admins), len(members)) + for found, expected in zip(sorted(admins), sorted(members)): + self.assertEqual(found, expected) + + def test_test_iam_permissions(self): + from google.cloud.bigtable.client import Client + from google.cloud.bigtable_admin_v2.gapic import bigtable_table_admin_client + from google.iam.v1 import iam_policy_pb2 + + credentials = _make_credentials() + client = Client(project=self.PROJECT_ID, credentials=credentials, admin=True) + + instance = client.instance(instance_id=self.INSTANCE_ID) + backup = self._make_one(self.BACKUP_ID, instance, cluster_id=self.CLUSTER_ID) + + permissions = ["bigtable.backups.create", "bigtable.backups.list"] + + response = iam_policy_pb2.TestIamPermissionsResponse(permissions=permissions) + + table_api = mock.create_autospec( + bigtable_table_admin_client.BigtableTableAdminClient + ) + table_api.test_iam_permissions.return_value = response + client._table_admin_client = table_api + + result = backup.test_iam_permissions(permissions) + + self.assertEqual(result, permissions) + table_api.test_iam_permissions.assert_called_once_with( + resource=backup.name, permissions=permissions + ) + class _Client(object): def __init__(self, project=TestBackup.PROJECT_ID): From 6d597a1e5be05c993c9f86beca4c1486342caf94 Mon Sep 17 00:00:00 2001 From: Tres Seaver Date: Thu, 12 Nov 2020 15:58:51 -0500 Subject: [PATCH 6/9] feat: add 'timeout' arg to 'Table.mutate_rows' (#157) Also, call data client's 'mutate_rows' directly -- do *not* scribble on its internal API wrappers. See: https://github.com/googleapis/python-bigtable/issues/7#issuecomment-715538708 Closes #7 --- google/cloud/bigtable/table.py | 67 ++++++----- tests/unit/test_table.py | 196 +++++++++++++++++++++++---------- 2 files changed, 172 insertions(+), 91 deletions(-) diff --git a/google/cloud/bigtable/table.py b/google/cloud/bigtable/table.py index 950a8c3fe..35ca43d24 100644 --- a/google/cloud/bigtable/table.py +++ b/google/cloud/bigtable/table.py @@ -20,9 +20,9 @@ from google.api_core.exceptions import NotFound from google.api_core.exceptions import RetryError from google.api_core.exceptions import ServiceUnavailable +from google.api_core.gapic_v1.method import DEFAULT from google.api_core.retry import if_exception_type from google.api_core.retry import Retry -from google.api_core.gapic_v1.method import wrap_method from google.cloud._helpers import _to_bytes from google.cloud.bigtable.backup import Backup from google.cloud.bigtable.column_family import _gc_rule_from_pb @@ -625,7 +625,7 @@ def yield_rows(self, **kwargs): ) return self.read_rows(**kwargs) - def mutate_rows(self, rows, retry=DEFAULT_RETRY): + def mutate_rows(self, rows, retry=DEFAULT_RETRY, timeout=DEFAULT): """Mutates multiple rows in bulk. For example: @@ -656,17 +656,23 @@ def mutate_rows(self, rows, retry=DEFAULT_RETRY): the :meth:`~google.api_core.retry.Retry.with_delay` method or the :meth:`~google.api_core.retry.Retry.with_deadline` method. + :type timeout: float + :param timeout: number of seconds bounding retries for the call + :rtype: list :returns: A list of response statuses (`google.rpc.status_pb2.Status`) corresponding to success or failure of each row mutation sent. These will be in the same order as the `rows`. """ + if timeout is DEFAULT: + timeout = self.mutation_timeout + retryable_mutate_rows = _RetryableMutateRowsWorker( self._instance._client, self.name, rows, app_profile_id=self._app_profile_id, - timeout=self.mutation_timeout, + timeout=timeout, ) return retryable_mutate_rows(retry=retry) @@ -1058,27 +1064,20 @@ def _do_mutate_retryable_rows(self): # All mutations are either successful or non-retryable now. return self.responses_statuses - mutate_rows_request = _mutate_rows_request( - self.table_name, retryable_rows, app_profile_id=self.app_profile_id - ) + entries = _compile_mutation_entries(self.table_name, retryable_rows) data_client = self.client.table_data_client - inner_api_calls = data_client._inner_api_calls - if "mutate_rows" not in inner_api_calls: - default_retry = (data_client._method_configs["MutateRows"].retry,) - if self.timeout is None: - default_timeout = data_client._method_configs["MutateRows"].timeout - else: - default_timeout = timeout.ExponentialTimeout(deadline=self.timeout) - data_client._inner_api_calls["mutate_rows"] = wrap_method( - data_client.transport.mutate_rows, - default_retry=default_retry, - default_timeout=default_timeout, - client_info=data_client._client_info, - ) + + kwargs = {} + if self.timeout is not None: + kwargs["timeout"] = timeout.ExponentialTimeout(deadline=self.timeout) try: - responses = data_client._inner_api_calls["mutate_rows"]( - mutate_rows_request, retry=None + responses = data_client.mutate_rows( + self.table_name, + entries, + app_profile_id=self.app_profile_id, + retry=None, + **kwargs ) except (ServiceUnavailable, DeadlineExceeded, Aborted): # If an exception, considered retryable by `RETRY_CODES`, is @@ -1260,8 +1259,8 @@ def _create_row_request( return message -def _mutate_rows_request(table_name, rows, app_profile_id=None): - """Creates a request to mutate rows in a table. +def _compile_mutation_entries(table_name, rows): + """Create list of mutation entries :type table_name: str :param table_name: The name of the table to write to. @@ -1269,29 +1268,29 @@ def _mutate_rows_request(table_name, rows, app_profile_id=None): :type rows: list :param rows: List or other iterable of :class:`.DirectRow` instances. - :type: app_profile_id: str - :param app_profile_id: (Optional) The unique name of the AppProfile. - - :rtype: :class:`data_messages_v2_pb2.MutateRowsRequest` - :returns: The ``MutateRowsRequest`` protobuf corresponding to the inputs. + :rtype: List[:class:`data_messages_v2_pb2.MutateRowsRequest.Entry`] + :returns: entries corresponding to the inputs. :raises: :exc:`~.table.TooManyMutationsError` if the number of mutations is - greater than 100,000 - """ - request_pb = data_messages_v2_pb2.MutateRowsRequest( - table_name=table_name, app_profile_id=app_profile_id + greater than the max ({}) + """.format( + _MAX_BULK_MUTATIONS ) + entries = [] mutations_count = 0 + entry_klass = data_messages_v2_pb2.MutateRowsRequest.Entry + for row in rows: _check_row_table_name(table_name, row) _check_row_type(row) mutations = row._get_mutations() - request_pb.entries.add(row_key=row.row_key, mutations=mutations) + entries.append(entry_klass(row_key=row.row_key, mutations=mutations)) mutations_count += len(mutations) + if mutations_count > _MAX_BULK_MUTATIONS: raise TooManyMutationsError( "Maximum number of mutations is %s" % (_MAX_BULK_MUTATIONS,) ) - return request_pb + return entries def _check_row_table_name(table_name, row): diff --git a/tests/unit/test_table.py b/tests/unit/test_table.py index c99cd6591..4469846b1 100644 --- a/tests/unit/test_table.py +++ b/tests/unit/test_table.py @@ -20,14 +20,14 @@ from google.api_core.exceptions import DeadlineExceeded -class Test___mutate_rows_request(unittest.TestCase): +class Test__compile_mutation_entries(unittest.TestCase): def _call_fut(self, table_name, rows): - from google.cloud.bigtable.table import _mutate_rows_request + from google.cloud.bigtable.table import _compile_mutation_entries - return _mutate_rows_request(table_name, rows) + return _compile_mutation_entries(table_name, rows) @mock.patch("google.cloud.bigtable.table._MAX_BULK_MUTATIONS", new=3) - def test__mutate_rows_too_many_mutations(self): + def test_w_too_many_mutations(self): from google.cloud.bigtable.row import DirectRow from google.cloud.bigtable.table import TooManyMutationsError @@ -41,13 +41,15 @@ def test__mutate_rows_too_many_mutations(self): rows[0].set_cell("cf1", b"c1", 2) rows[1].set_cell("cf1", b"c1", 3) rows[1].set_cell("cf1", b"c1", 4) + with self.assertRaises(TooManyMutationsError): self._call_fut("table", rows) - def test__mutate_rows_request(self): + def test_normal(self): from google.cloud.bigtable.row import DirectRow + from google.cloud.bigtable_v2.proto import bigtable_pb2 - table = mock.Mock(name="table", spec=["name"]) + table = mock.Mock(spec=["name"]) table.name = "table" rows = [ DirectRow(row_key=b"row_key", table=table), @@ -55,25 +57,26 @@ def test__mutate_rows_request(self): ] rows[0].set_cell("cf1", b"c1", b"1") rows[1].set_cell("cf1", b"c1", b"2") + result = self._call_fut("table", rows) - expected_result = _mutate_rows_request_pb(table_name="table") - entry1 = expected_result.entries.add() - entry1.row_key = b"row_key" - mutations1 = entry1.mutations.add() - mutations1.set_cell.family_name = "cf1" - mutations1.set_cell.column_qualifier = b"c1" - mutations1.set_cell.timestamp_micros = -1 - mutations1.set_cell.value = b"1" - entry2 = expected_result.entries.add() - entry2.row_key = b"row_key_2" - mutations2 = entry2.mutations.add() - mutations2.set_cell.family_name = "cf1" - mutations2.set_cell.column_qualifier = b"c1" - mutations2.set_cell.timestamp_micros = -1 - mutations2.set_cell.value = b"2" + Entry = bigtable_pb2.MutateRowsRequest.Entry - self.assertEqual(result, expected_result) + entry_1 = Entry(row_key=b"row_key") + mutations_1 = entry_1.mutations.add() + mutations_1.set_cell.family_name = "cf1" + mutations_1.set_cell.column_qualifier = b"c1" + mutations_1.set_cell.timestamp_micros = -1 + mutations_1.set_cell.value = b"1" + + entry_2 = Entry(row_key=b"row_key_2") + mutations_2 = entry_2.mutations.add() + mutations_2.set_cell.family_name = "cf1" + mutations_2.set_cell.column_qualifier = b"c1" + mutations_2.set_cell.timestamp_micros = -1 + mutations_2.set_cell.value = b"2" + + self.assertEqual(result, [entry_1, entry_2]) class Test__check_row_table_name(unittest.TestCase): @@ -162,27 +165,49 @@ def _get_target_client_class(): def _make_client(self, *args, **kwargs): return self._get_target_client_class()(*args, **kwargs) - def test_constructor_w_admin(self): - credentials = _make_credentials() - client = self._make_client( - project=self.PROJECT_ID, credentials=credentials, admin=True - ) - instance = client.instance(instance_id=self.INSTANCE_ID) + def test_constructor_defaults(self): + instance = mock.Mock(spec=[]) + table = self._make_one(self.TABLE_ID, instance) + + self.assertEqual(table.table_id, self.TABLE_ID) + self.assertIs(table._instance, instance) + self.assertIsNone(table.mutation_timeout) + self.assertIsNone(table._app_profile_id) + + def test_constructor_explicit(self): + instance = mock.Mock(spec=[]) + mutation_timeout = 123 + app_profile_id = "profile-123" + + table = self._make_one( + self.TABLE_ID, + instance, + mutation_timeout=mutation_timeout, + app_profile_id=app_profile_id, + ) + self.assertEqual(table.table_id, self.TABLE_ID) - self.assertIs(table._instance._client, client) - self.assertEqual(table.name, self.TABLE_NAME) + self.assertIs(table._instance, instance) + self.assertEqual(table.mutation_timeout, mutation_timeout) + self.assertEqual(table._app_profile_id, app_profile_id) - def test_constructor_wo_admin(self): - credentials = _make_credentials() - client = self._make_client( - project=self.PROJECT_ID, credentials=credentials, admin=False + def test_name(self): + table_data_client = mock.Mock(spec=["table_path"]) + client = mock.Mock( + project=self.PROJECT_ID, + table_data_client=table_data_client, + spec=["project", "table_data_client"], ) - instance = client.instance(instance_id=self.INSTANCE_ID) + instance = mock.Mock( + _client=client, + instance_id=self.INSTANCE_ID, + spec=["_client", "instance_id"], + ) + table = self._make_one(self.TABLE_ID, instance) - self.assertEqual(table.table_id, self.TABLE_ID) - self.assertIs(table._instance._client, client) - self.assertEqual(table.name, self.TABLE_NAME) + + self.assertEqual(table.name, table_data_client.table_path.return_value) def _row_methods_helper(self): client = self._make_client( @@ -620,8 +645,11 @@ def test_read_row_still_partial(self): with self.assertRaises(ValueError): self._read_row_helper(chunks, None) - def test_mutate_rows(self): + def _mutate_rows_helper( + self, mutation_timeout=None, app_profile_id=None, retry=None, timeout=None + ): from google.rpc.status_pb2 import Status + from google.cloud.bigtable.table import DEFAULT_RETRY from google.cloud.bigtable_admin_v2.gapic import bigtable_table_admin_client table_api = mock.create_autospec( @@ -633,21 +661,78 @@ def test_mutate_rows(self): ) instance = client.instance(instance_id=self.INSTANCE_ID) client._table_admin_client = table_api - table = self._make_one(self.TABLE_ID, instance) + ctor_kwargs = {} - response = [Status(code=0), Status(code=1)] + if mutation_timeout is not None: + ctor_kwargs["mutation_timeout"] = mutation_timeout + + if app_profile_id is not None: + ctor_kwargs["app_profile_id"] = app_profile_id - mock_worker = mock.Mock(return_value=response) - with mock.patch( + table = self._make_one(self.TABLE_ID, instance, **ctor_kwargs) + + rows = [mock.MagicMock(), mock.MagicMock()] + response = [Status(code=0), Status(code=1)] + instance_mock = mock.Mock(return_value=response) + klass_mock = mock.patch( "google.cloud.bigtable.table._RetryableMutateRowsWorker", - new=mock.MagicMock(return_value=mock_worker), - ): - statuses = table.mutate_rows([mock.MagicMock(), mock.MagicMock()]) + new=mock.MagicMock(return_value=instance_mock), + ) + + call_kwargs = {} + + if retry is not None: + call_kwargs["retry"] = retry + + if timeout is not None: + expected_timeout = call_kwargs["timeout"] = timeout + else: + expected_timeout = mutation_timeout + + with klass_mock: + statuses = table.mutate_rows(rows, **call_kwargs) + result = [status.code for status in statuses] expected_result = [0, 1] - self.assertEqual(result, expected_result) + klass_mock.new.assert_called_once_with( + client, + self.TABLE_NAME, + rows, + app_profile_id=app_profile_id, + timeout=expected_timeout, + ) + + if retry is not None: + instance_mock.assert_called_once_with(retry=retry) + else: + instance_mock.assert_called_once_with(retry=DEFAULT_RETRY) + + def test_mutate_rows_w_default_mutation_timeout_app_profile_id(self): + self._mutate_rows_helper() + + def test_mutate_rows_w_mutation_timeout(self): + mutation_timeout = 123 + self._mutate_rows_helper(mutation_timeout=mutation_timeout) + + def test_mutate_rows_w_app_profile_id(self): + app_profile_id = "profile-123" + self._mutate_rows_helper(app_profile_id=app_profile_id) + + def test_mutate_rows_w_retry(self): + retry = mock.Mock() + self._mutate_rows_helper(retry=retry) + + def test_mutate_rows_w_timeout_arg(self): + timeout = 123 + self._mutate_rows_helper(timeout=timeout) + + def test_mutate_rows_w_mutation_timeout_and_timeout_arg(self): + mutation_timeout = 123 + timeout = 456 + self._mutate_rows_helper(mutation_timeout=mutation_timeout, timeout=timeout) + def test_read_rows(self): from google.cloud._testing import _Monkey from google.cloud.bigtable.row_data import PartialRowsData @@ -1424,21 +1509,18 @@ def test_callable_no_retry_strategy(self): row_3 = DirectRow(row_key=b"row_key_3", table=table) row_3.set_cell("cf", b"col", b"value3") - response = self._make_responses( - [self.SUCCESS, self.RETRYABLE_1, self.NON_RETRYABLE] - ) + worker = self._make_worker(client, table.name, [row_1, row_2, row_3]) - with mock.patch("google.cloud.bigtable.table.wrap_method") as patched: - patched.return_value = mock.Mock(return_value=[response]) + response_codes = [self.SUCCESS, self.RETRYABLE_1, self.NON_RETRYABLE] + response = self._make_responses(response_codes) + data_api.mutate_rows = mock.MagicMock(return_value=[response]) - worker = self._make_worker(client, table.name, [row_1, row_2, row_3]) - statuses = worker(retry=None) + statuses = worker(retry=None) result = [status.code for status in statuses] - expected_result = [self.SUCCESS, self.RETRYABLE_1, self.NON_RETRYABLE] + self.assertEqual(result, response_codes) - client._table_data_client._inner_api_calls["mutate_rows"].assert_called_once() - self.assertEqual(result, expected_result) + data_api.mutate_rows.assert_called_once() def test_callable_retry(self): from google.cloud.bigtable.row import DirectRow From ffb166fccaa7936f4f126564151d6b1cade22b12 Mon Sep 17 00:00:00 2001 From: WhiteSource Renovate Date: Mon, 16 Nov 2020 19:49:38 +0100 Subject: [PATCH 7/9] chore(deps): update dependency apache-beam to v2.25.0 (#158) --- samples/beam/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/samples/beam/requirements.txt b/samples/beam/requirements.txt index 5ec8ddb2c..565c39c39 100644 --- a/samples/beam/requirements.txt +++ b/samples/beam/requirements.txt @@ -1,3 +1,3 @@ -apache-beam==2.24.0 +apache-beam==2.25.0 google-cloud-bigtable==1.5.1 google-cloud-core==1.4.3 \ No newline at end of file From df2336415027d0b93a19a1020104b8c7ff07c01b Mon Sep 17 00:00:00 2001 From: Yoshi Automation Bot Date: Mon, 16 Nov 2020 10:50:17 -0800 Subject: [PATCH 8/9] chore(python): use BUILD_SPECIFIC_GCLOUD_PROJECT for samples (#148) https://github.com/googleapis/python-talent/blob/ef045e8eb348db36d7a2a611e6f26b11530d273b/samples/snippets/noxfile_config.py#L27-L32 `BUILD_SPECIFIC_GCLOUD_PROJECT` is an alternate project used for sample tests that do poorly with concurrent runs on the same project. Source-Author: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com> Source-Date: Wed Sep 30 13:06:03 2020 -0600 Source-Repo: googleapis/synthtool Source-Sha: 9b0da5204ab90bcc36f8cd4e5689eff1a54cc3e4 Source-Link: https://github.com/googleapis/synthtool/commit/9b0da5204ab90bcc36f8cd4e5689eff1a54cc3e4 --- .kokoro/samples/python3.6/common.cfg | 6 ++++++ .kokoro/samples/python3.7/common.cfg | 6 ++++++ .kokoro/samples/python3.8/common.cfg | 6 ++++++ synth.metadata | 6 +++--- 4 files changed, 21 insertions(+), 3 deletions(-) diff --git a/.kokoro/samples/python3.6/common.cfg b/.kokoro/samples/python3.6/common.cfg index dd6620136..f71693fca 100644 --- a/.kokoro/samples/python3.6/common.cfg +++ b/.kokoro/samples/python3.6/common.cfg @@ -13,6 +13,12 @@ env_vars: { value: "py-3.6" } +# Declare build specific Cloud project. +env_vars: { + key: "BUILD_SPECIFIC_GCLOUD_PROJECT" + value: "python-docs-samples-tests-py36" +} + env_vars: { key: "TRAMPOLINE_BUILD_FILE" value: "github/python-bigtable/.kokoro/test-samples.sh" diff --git a/.kokoro/samples/python3.7/common.cfg b/.kokoro/samples/python3.7/common.cfg index 6ee44dbb9..5fa465fda 100644 --- a/.kokoro/samples/python3.7/common.cfg +++ b/.kokoro/samples/python3.7/common.cfg @@ -13,6 +13,12 @@ env_vars: { value: "py-3.7" } +# Declare build specific Cloud project. +env_vars: { + key: "BUILD_SPECIFIC_GCLOUD_PROJECT" + value: "python-docs-samples-tests-py37" +} + env_vars: { key: "TRAMPOLINE_BUILD_FILE" value: "github/python-bigtable/.kokoro/test-samples.sh" diff --git a/.kokoro/samples/python3.8/common.cfg b/.kokoro/samples/python3.8/common.cfg index cc909eb20..f3a6fa7ec 100644 --- a/.kokoro/samples/python3.8/common.cfg +++ b/.kokoro/samples/python3.8/common.cfg @@ -13,6 +13,12 @@ env_vars: { value: "py-3.8" } +# Declare build specific Cloud project. +env_vars: { + key: "BUILD_SPECIFIC_GCLOUD_PROJECT" + value: "python-docs-samples-tests-py38" +} + env_vars: { key: "TRAMPOLINE_BUILD_FILE" value: "github/python-bigtable/.kokoro/test-samples.sh" diff --git a/synth.metadata b/synth.metadata index 7206c46f8..a87cbb407 100644 --- a/synth.metadata +++ b/synth.metadata @@ -4,7 +4,7 @@ "git": { "name": ".", "remote": "https://github.com/googleapis/python-bigtable.git", - "sha": "3861f6b0552e431a1fc7aa872c4d293ca129c28c" + "sha": "9a61c57a73d8b55d59799e2b78ced03b07660fb8" } }, { @@ -19,14 +19,14 @@ "git": { "name": "synthtool", "remote": "https://github.com/googleapis/synthtool.git", - "sha": "f3c04883d6c43261ff13db1f52d03a283be06871" + "sha": "9b0da5204ab90bcc36f8cd4e5689eff1a54cc3e4" } }, { "git": { "name": "synthtool", "remote": "https://github.com/googleapis/synthtool.git", - "sha": "f3c04883d6c43261ff13db1f52d03a283be06871" + "sha": "9b0da5204ab90bcc36f8cd4e5689eff1a54cc3e4" } } ], From 584a822c647092a5084b6322f76c3d5ae9d5557f Mon Sep 17 00:00:00 2001 From: "release-please[bot]" <55107282+release-please[bot]@users.noreply.github.com> Date: Mon, 16 Nov 2020 14:25:22 -0500 Subject: [PATCH 9/9] chore: release 1.6.0 (#164) Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com> --- CHANGELOG.md | 8 ++++++++ setup.py | 2 +- 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f46bcc42f..94bdbdf53 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,14 @@ [1]: https://pypi.org/project/google-cloud-bigtable/#history +## [1.6.0](https://www.github.com/googleapis/python-bigtable/compare/v1.5.1...v1.6.0) (2020-11-16) + + +### Features + +* add 'timeout' arg to 'Table.mutate_rows' ([#157](https://www.github.com/googleapis/python-bigtable/issues/157)) ([6d597a1](https://www.github.com/googleapis/python-bigtable/commit/6d597a1e5be05c993c9f86beca4c1486342caf94)), closes [/github.com/googleapis/python-bigtable/issues/7#issuecomment-715538708](https://www.github.com/googleapis//github.com/googleapis/python-bigtable/issues/7/issues/issuecomment-715538708) [#7](https://www.github.com/googleapis/python-bigtable/issues/7) +* Backup Level IAM ([#160](https://www.github.com/googleapis/python-bigtable/issues/160)) ([44932cb](https://www.github.com/googleapis/python-bigtable/commit/44932cb8710e12279dbd4e9271577f8bee238980)) + ### [1.5.1](https://www.github.com/googleapis/python-bigtable/compare/v1.5.0...v1.5.1) (2020-10-06) diff --git a/setup.py b/setup.py index 0f3c9cd82..0c98fee99 100644 --- a/setup.py +++ b/setup.py @@ -22,7 +22,7 @@ name = 'google-cloud-bigtable' description = 'Google Cloud Bigtable API client library' -version = "1.5.1" +version = "1.6.0" # Should be one of: # 'Development Status :: 3 - Alpha' # 'Development Status :: 4 - Beta'