Skip to content

Conversation

@Kontinuation
Copy link
Member

Did you read the Contributor Guide?

Is this PR related to a JIRA ticket?

What changes were proposed in this PR?

Fix writing rasters to S3 on EMR by re-using the FileSystem object instead of creating new instances. Re-using the FileSystem instance is a common pattern, while creating new FileSystem instances is prone to errors, since some implementations hold transient state required by follow-up operations, and that transient states cannot be accessed by newly created instances.

How was this patch tested?

Passing existing tests, testing manually on emr-7.1.0.

Did this PR include necessary documentation updates?

  • No, this PR does not affect any public API so no need to change the documentation.

@Kontinuation Kontinuation marked this pull request as ready for review June 20, 2024 16:47
@Kontinuation Kontinuation requested a review from jiayuasu as a code owner June 20, 2024 16:47
@jiayuasu
Copy link
Member

@Kontinuation Does this re-introduce this bug?: #1236

I guess not, because this PR does not close hfs connection ever? Is this OK?

@Kontinuation
Copy link
Member Author

@Kontinuation Does this re-introduce this bug?: #1236

I guess not, because this PR does not close hfs connection ever? Is this OK?

It is OK. It is a common pattern to acquire a lazily created FileSystem instance and leave it open, so that it could be reused for other filesystem operations.

@jiayuasu jiayuasu added the bug label Jun 21, 2024
@jiayuasu jiayuasu added this to the sedona-1.6.1 milestone Jun 21, 2024
@jiayuasu jiayuasu merged commit 1196823 into apache:master Jun 21, 2024
Kontinuation added a commit to Kontinuation/sedona that referenced this pull request Oct 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants