[GH-2566] Temporarily set pyspark < 4.1.0

[jiayuasu]

Did you read the Contributor Guide?

• Yes, I have read the Contributor Rules and Contributor Development Guide

Is this PR related to a ticket?

• Yes, and the PR name follows the format [GH-2566] my subject. Closes python build extension fails on pyspark 4.1 + windows #2566

What changes were proposed in this PR?

This pull request updates the version constraints for the pyspark dependency in the python/pyproject.toml file to prevent compatibility issues with future pyspark releases. The new constraint ensures that only versions from 3.4.0 up to, but not including, 4.1.0 are allowed.

How was this patch tested?

Did this PR include necessary documentation updates?

• Yes, I am adding a new API. I am using the current SNAPSHOT version number in vX.Y.Z format.
• Yes, I have updated the documentation.
• No, this PR does not affect any public API so no need to change the documentation.