ReadTimeout errors

[uatach]

I keep receiving requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='storage.googleapis.com', port=443): Read timed out. (read timeout=60) when trying to write files to storage.

I've tried adding the timeout value when opening the file:
Client().bucket('my_bucket').blob('my_blob').open(mode='wb', timeout=3600)

Investigating the traceback:

 File "/usr/local/lib/python3.9/site-packages/google/cloud/storage/fileio.py", line 411, in _upload_chunks_from_buffer
    upload.transmit_next_chunk(transport)

I've found that the method transmit_next_chunk can receive a timeout param that is not being set.

[shamyjun22]

I keep receiving requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='storage.googleapis.com', port=443): Read timed out. (read timeout=60) when trying to write files to storage.

I've tried adding the timeout value when opening the file: Client().bucket('my_bucket').blob('my_blob').open(mode='wb', timeout=3600)

Investigating the traceback:

 File "/usr/local/lib/python3.9/site-packages/google/cloud/storage/fileio.py", line 411, in _upload_chunks_from_buffer
    upload.transmit_next_chunk(transport)

I've found that the method transmit_next_chunk can receive a timeout param that is not being set.

@uatach, have you resolved this issue? I keep on receiving this exception.

[uatach]

@shamyjun22 I've added some retry logic and it was enough, I haven't tried the new package version yet.

[shamyjun22]

Hi @uatach, I encounter this exception when trying to upload file to Google Cloud Storage:
1.
Code: ReadTimeout
Type: <class 'requests.exceptions.ReadTimeout'>
Message: HTTPSConnectionPool(host='storage.googleapis.com', port=443): Read timed out. (read timeout=60

Code: ConnectionError
Type: <class 'requests.exceptions.ConnectionError'>
Message: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

Does it mean that I was able to upload the file successfully, but I just don't receive any response from where I am uploading to?

Also, if it wont bother, can you provide a snippet of code on how you resolve it?

Thanks in advance.

[uatach]

I encounter this exception when trying to upload file to Google Cloud Storage

@shamyjun22 are you trying to upload many files, like in parallel or really fast? I've felt that several GCP python libs have some problem dealing with simultaneous connections but I still haven't been able to pin the problem down... I have been adding some sleep calls just to avoid these problems

Does it mean that I was able to upload the file successfully, but I just don't receive any response from where I am uploading to?

I don't think so

Also, if it wont bother, can you provide a snippet of code on how you resolve it?

Sure, but as I said is a simple retry logic:

blob = Client().bucket('bucket_name').blob('blob_name')

while True:
  try:
    with blob.open('w') as fp:
      fp.write('data')
  except:
    continue
  else:
    break

[shamyjun22]

@uatach, thank you for your response. By the way, i am not uploading many files in parallel at one time, but i am uploading one file every second interval because I need realtime data.

This is how I structure my code in python, maybe you can have suggestion the way it is structure

# import other libraries needed here

from google.cloud import storage

# setup cloud storage client here
client = storage.Client.from_service_account_json('service_account_here')
bucket = client.get_bucket('bucket_name_here')

def upload_to_storage_function_here(data):
    try:
        # do stuff here to upload file like the blob path
        # blob upload here
    except:
    # catch exception here

async def main():
    while True:
        # fetch data from source
        # manipulate data
        
        upload_to_storage_function_here(data)
        
        if (sec == datetime.now(manila_timezone).strftime("%S")):
            time.sleep(max(0, 1 - (datetime.now(manila_timezone) - timeStart).total_seconds() % 1))
                    
if __main__ == '__main__':
    asyncio.run(main())

Thanks.

[olejorgenb]

This is not directly on topic, but can we not expect to GCS to handle blob requests within 1 minute in the general case? We're seeing these errors much more frequently (for small blobs) than I would expect (I mean - there's no guarantees in life), so wondering if it's likely something else on our end, or if GCS just is unreliable?