BigTable: Modify system test for new GAPIC code #5302

aneepct · 2018-05-07T16:31:30Z

Modify system test for new GAPIC code

…thub.com/zakons/google-cloud-python into feature/gapic

…into feature/read_rows_retry

aneepct · 2018-05-07T16:32:59Z

@jonparrott @crwilcox I am getting error on create_cluster on system tests.

grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.INVALID_ARGUMENT, Error in field 'clusters' : Need to have at least one cluster map element in CreateInstanceRequest.)>

bigtable/google/cloud/bigtable/instance.py

+        )
+        clusters = []
+        clusters.append(instance_pb2.Cluster(name=cluster_path,
+                                             location=location_path))


bigtable/tests/system.py

@@ -36,6 +36,7 @@
 from test_utils.system import unique_resource_id


+PROJECT_ID = 'grass-clump-479'


bigtable/tests/system.py

@@ -79,19 +80,19 @@ def setUpModule():
        credentials = EmulatorCreds()
        Config.CLIENT = Client(admin=True, credentials=credentials)
    else:
-        Config.CLIENT = Client(admin=True)
+        Config.CLIENT = Client(project=PROJECT_ID, admin=True)


bigtable/tests/system.py


-        EXISTING_INSTANCES[:] = instances
+        EXISTING_INSTANCES[:] = instances_response.instances


tseaver · 2018-05-08T16:45:38Z

@aneepct Also, can you please rebase against the current master to dump all the redundant commits?

zakons · 2018-05-10T04:27:56Z

@jonparrott @tseaver Hoping this closes the loop on a rather large series of merges on master to support GAPIC. Together with #5275, #5315 and #5283 - which add support for app profile, truncate table and initial splits on top of the GAPIC interface - this should complete the series of changes for the python beta. All have been updated to resolve the review issues found. Please let us know if any issues remain so we can turn these around quickly. Thanks again for the excellent reviews.

bigtable/google/cloud/bigtable/client.py

@@ -140,8 +140,7 @@ def _table_data_client(self):
        :rtype: :class:`.bigtable_v2.BigtableClient`
        :returns: A BigtableClient object.
        """
-        return bigtable_v2.BigtableClient(channel=self._channel,
-                                          credentials=self._credentials)
+        return bigtable_v2.BigtableClient(channel=self._channel)


bigtable/google/cloud/bigtable/client.py

@@ -156,7 +155,7 @@ def _table_admin_client(self):
        if not self._admin:
            raise ValueError('Client is not an admin client.')
        return bigtable_admin_v2.BigtableTableAdminClient(
-            channel=self._channel, credentials=self._credentials)
+            channel=self._channel)


bigtable/google/cloud/bigtable/client.py

@@ -171,7 +170,7 @@ def _instance_admin_client(self):
        if not self._admin:
            raise ValueError('Client is not an admin client.')
        return bigtable_admin_v2.BigtableInstanceAdminClient(
-            channel=self._channel, credentials=self._credentials)
+            channel=self._channel)


bigtable/tests/system.py

@@ -378,7 +380,7 @@ def test_read_large_cell_limit(self):
        row = self._table.row(ROW_KEY)
        self.rows_to_delete.append(row)

-        number_of_bytes = 10 * 1024 * 1024
+        number_of_bytes = 10 * 1024
        data = b'1' * number_of_bytes  # 10MB of 1's.


bigtable/google/cloud/bigtable/instance.py

+        location = self._client._instance_admin_client.location_path(
+            self._client.project, self._cluster_location_id)
+        cluster = instance_pb2.Cluster(name=cluster_name, location=location,
+                                       serve_nodes=DEFAULT_SERVE_NODES)


bigtable/tests/unit/test_client.py

        project = 'PROJECT'
        client = self._make_one(
-            project=project, credentials=credentials, admin=True)
+            project=project, channel=channel, credentials=credentials,
+            admin=True)


bigtable/tests/unit/test_client.py

-        channel = _make_channel()
-        client = self._make_one(project=self.PROJECT, channel=channel)
+        credentials = _make_credentials()
+        client = self._make_one(project=self.PROJECT, credentials=credentials)


bigtable/tests/unit/test_cluster.py

+            google.auth.credentials.Scoped):
+        pass
+
+    return mock.Mock(spec=_CredentialsWithScopes)


aneepct · 2018-05-15T11:48:23Z

@tseaver Please review the recent changes

aneepct · 2018-05-17T04:16:27Z

@tseaver Thank you for all your help.

crwilcox · 2018-05-23T16:33:57Z

@aneepct the tests are failing after this merge. nox -s "system(py='3.6')" run from the bigtable directory will show the failures.

Here is a circle build failing if you want to see what the errors are. https://circleci.com/gh/GoogleCloudPlatform/google-cloud-python/6646

aneepct · 2018-05-25T04:52:41Z

@crwilcox The system test already fixed on different PR #5315

Luke Sneeringer and others added 30 commits January 4, 2018 10:29

Provide new auto-generated layer for Bigtable.

3edca4c

Merge branch 'master' into bigtable-regen

a677959

Add retry for read rows

d540349

Fix line lengths

4aa5702

Add parameter start_inclusive to _create_row_request

f7d4c69

Update unit test case for yield rows retry

28892ce

Fix line lengths

8d141f0

Edit table.py as requested changes

902f11b

Added code review changes from @jonparrott.

4afabb1

Fix parameter name on _RetryableReadRows class

79698b1

Add retry for Deadline Exceeded on read rows

9b778be

Fix line over-indented

cafc5ad

Fix line over-indented

f974752

Merge commit '3edca4c3299e85d4346dae49232d540a2d34002c' of https://gi…

f7242f4

…thub.com/zakons/google-cloud-python into feature/gapic

Merge commit 'a67795951bc9f490265eda7cf511e17d2cc70921' of https://gi…

180d1d5

…thub.com/zakons/google-cloud-python into feature/gapic

Working on gapic integration

f2e39e0

Refactore yield_rows retry

13adfd6

1. Moved last_scanned_row_key into YieldRowsData.

ca69a2f

Fix line over-indented

02c154e

Fix row_data.py

b6da7fa

Change row.py as requested

f1a2d92

Create new test case for yield_rows iterator failure

65c1c4b

Fix for start_inclusive on retry read rows.

170a173

Merge branch 'master' of https://github.com/zakons/google-cloud-python …

5da3fd3

…into feature/read_rows_retry

Change bigtable_pb2 imports to use from gapic library.

f7a58b5

Merge branch 'feature/read_rows_retry' of https://github.com/zakons/g…

c8f664b

…oogle-cloud-python into feature/gapic

Working on gapic

d5f0d98

Integrate bigtable GAPIC client

32df7bd

Refactoring unit tests

77ec950

Refactor to use GAPIC client

efbf2f5

googlebot added the cla: yes This human has signed the Contributor License Agreement. label May 7, 2018

crwilcox mentioned this pull request May 7, 2018

Grpcio pre system tests #5304

Merged

aneepct added 3 commits May 8, 2018 07:40

Remove unused imports

b0646af

Added ClientWithProject on Client

316e59c

Add credentials on unit test cases

1118493

tseaver suggested changes May 8, 2018

View reviewed changes

tseaver added the api: bigtable Issues related to the Bigtable API. label May 8, 2018

aneepct added 9 commits May 9, 2018 07:13

Add ClientWithProject on Client

6e9f95e

Rebase with Master

0f880b1

Rebase with master and add pull client with project

3de1ca3

Make requested changes

4765e3f

Remove unused import

5f03036

Fixing system teest

17f6ac3

Fix instance and system test

b45004f

Create test case for updated instace

c3c9857

Update instance and system tests

35d3199

tseaver suggested changes May 10, 2018

View reviewed changes

aneepct added 3 commits May 11, 2018 12:27

Fixed required changes

8749b94

Fix py lint errors

efef394

Move _make_credentials to _testing

9e2df00

tseaver approved these changes May 16, 2018

View reviewed changes

tseaver merged commit 1b4cf40 into googleapis:master May 16, 2018

crwilcox mentioned this pull request May 23, 2018

Fix Failing BigTable System tests #5362

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BigTable: Modify system test for new GAPIC code #5302

BigTable: Modify system test for new GAPIC code #5302

aneepct commented May 7, 2018

aneepct commented May 7, 2018 •

edited

Loading

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

tseaver commented May 8, 2018

zakons commented May 10, 2018

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

aneepct commented May 15, 2018

aneepct commented May 17, 2018

crwilcox commented May 23, 2018

aneepct commented May 25, 2018

		@@ -36,6 +36,7 @@
		from test_utils.system import unique_resource_id


		PROJECT_ID = 'grass-clump-479'


		EXISTING_INSTANCES[:] = instances
		EXISTING_INSTANCES[:] = instances_response.instances

BigTable: Modify system test for new GAPIC code #5302

BigTable: Modify system test for new GAPIC code #5302

Conversation

aneepct commented May 7, 2018

aneepct commented May 7, 2018 • edited Loading

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

tseaver commented May 8, 2018

zakons commented May 10, 2018

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

This comment was marked as spam.

aneepct commented May 15, 2018

aneepct commented May 17, 2018

crwilcox commented May 23, 2018

aneepct commented May 25, 2018

aneepct commented May 7, 2018 •

edited

Loading