Parallelize pdo tests (dblib, firebird, oci, odbc, pgsql) #12395

kocsismate · 2023-10-10T07:40:41Z

It becomes possible by using different SQL symbols in all tests.

jorgsowa · 2023-10-10T10:05:13Z

There is still one PR waiting for review with a similar goal: #11879

kocsismate · 2023-10-10T11:24:37Z

There is still one PR waiting for review with a similar goal: #11879

Oh nice, thanks for pointing this out! It's just mere luck that I didn't yet include the MySQL implementation in my PR ^^

kocsismate · 2023-10-10T11:27:10Z

Now that I added --CLEAN-- sections without IF EXISTS clauses to ext/pdo/tests/ , SQLite tests started to fail because they run in-memory. Neither simply adding IF EXIST work, because older Oracle servers don't support the DROP TABLE IF EXISTS ... construct. That's why I'll have to apply some fix (either special case the oracle implementation, or skip the cleanup for sqlite - I lean towards the latter one).

ext/pdo/tests/bug_36798.phpt

ext/pdo/tests/bug_43130.phpt

ext/pdo_dblib/tests/bug_47588.phpt

ext/pdo_firebird/tests/bug_64037.phpt

ext/pdo_firebird/tests/bug_74462.phpt

ext/pdo_firebird/tests/rowCount.phpt

ext/pdo_oci/tests/bug54379.phpt

ext/pdo_pgsql/tests/bug62479.phpt

ext/pdo_pgsql/tests/bug62498-32bit.phpt

ext/pdo/tests/bug_36798.phpt

ext/pdo_odbc/tests/long_columns.phpt

ext/pdo_sqlite/tests/pdo_sqlite_createcollation.phpt

It becomes possibly by using different symbols in all tests.

Girgias · 2023-10-16T23:27:35Z

Main remain issues seems to be Firebird on Windows

Girgias · 2023-10-18T00:04:05Z

Still slightly broken

Windows:

 =====================================================================
BORKED TEST SUMMARY
---------------------------------------------------------------------

Warning: PDO::exec(): SQLSTATE[HY000]: General error in D:\a\php-src\php-src\ext\pdo\tests\pdo_test.inc on line 90
 [D:\a\php-src\php-src\ext\pdo\tests\pecl_bug_5217.phpt]

Fatal error: Uncaught PDOException: SQLSTATE[HY000]: General error in D:\a\php-src\php-src\ext\pdo_firebird\tests\ddl.clean.php:5
Stack trace:
#0 D:\a\php-src\php-src\ext\pdo_firebird\tests\ddl.clean.php(5): PDO->exec('DROP TRIGGER dd...')
#1 {main}
  thrown in D:\a\php-src\php-src\ext\pdo_firebird\tests\ddl.clean.php on line 5
 [D:\a\php-src\php-src\ext\pdo_firebird\tests\ddl.phpt]

Warning: PDO::exec(): SQLSTATE[42601]: Syntax error: 7 ERROR:  syntax error at or near "EXIST"
LINE 1: DROP TABLE IF EXIST test36727
                      ^ in D:\a\php-src\php-src\ext\pdo_pgsql\tests\bug36727.clean.php on line 4
 [D:\a\php-src\php-src\ext\pdo_pgsql\tests\bug36727.phpt]

Warning: PDO::exec(): SQLSTATE[42601]: Syntax error: 7 ERROR:  syntax error at or near "EXIST"
LINE 1: DROP TABLE IF EXIST test43925
                      ^ in D:\a\php-src\php-src\ext\pdo_pgsql\tests\bug43925.clean.php on line 4
 [D:\a\php-src\php-src\ext\pdo_pgsql\tests\bug43925.phpt]
=====================================================================

=====================================================================
FAILED TEST SUMMARY
---------------------------------------------------------------------
via [D:\a\php-src\php-src\ext\pdo_firebird\tests\common.phpt]
	FIREBIRD PDO Common: transactions [ext\pdo_firebird\tests\pdo_017.phpt]
=====================================================================

And Linux CI:

=====================================================================
BORKED TEST SUMMARY
---------------------------------------------------------------------

Parse error: syntax error, unexpected double-quote mark, expecting ")" in /home/runner/work/php-src/php-src/ext/pdo_oci/tests/pdo_oci_attr_autocommit_2.clean.php on line 4
 [/home/runner/work/php-src/php-src/ext/pdo_oci/tests/pdo_oci_attr_autocommit_2.phpt]
=====================================================================

ext/pdo/tests/pdo_test.inc

As I suppose there is still some concurrency issue.

ext/pdo_firebird/tests/ddl.phpt

Co-authored-by: Jorg Adam Sowa <[email protected]>

Girgias · 2023-10-21T11:23:34Z

This also happened on a previous CI run:

001+ Fatal error: Uncaught PDOException: SQLSTATE[HY000]: General error: 20018 Transaction (Process ID 59) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction. [20018] (severity 13) [select *
002+ from information_schema.columns ic1
003+ cross join information_schema.columns ic2
004+ cross join information_schema.columns ic3] in /home/runner/work/php-src/php-src/ext/pdo_dblib/tests/bug_50755.php:12
005+ Stack trace:
006+ #0 /home/runner/work/php-src/php-src/ext/pdo_dblib/tests/bug_50755.php(12): PDOStatement->fetch()
007+ #1 {main}
008+   thrown in /home/runner/work/php-src/php-src/ext/pdo_dblib/tests/bug_50755.php on line 12
========DONE========
FAIL PDO_DBLIB: Out of memory on large recordsets [ext/pdo_dblib/tests/bug_50755.phpt]

Girgias

So the following test start a transaction without closing it properly:

ext/pdo/tests/bug_34630.phpt

Maybe the ddl and pdo017 tests need to be marked as conflicting.

ext/pdo/tests/pdo_017.phpt

kocsismate · 2023-10-24T19:25:46Z

ext/pdo/tests/bug_34630.phpt

Hmm, interesting idea, but it looks like the transaction is related to OCI8 only. :/

Maybe the ddl and pdo017 tests need to be marked as conflicting.

Yeah, I had already re-aadded the CONFLICTS file for firebird. Doing so apparently fixed a bunch of failures.

Maybe just skip the test for now on Firebird, as it might be a bug in the driver... (this skip is taken from pdo039.phpt)

Yeah, I tend to do so soon, but I'd still prefer to find out the root cause even though these random failures drive me nuts.. I thought parallelization will be easy to do... :/

kocsismate · 2023-10-24T21:48:45Z

I had enough, I xfailed it...

Girgias · 2023-10-28T11:36:09Z

Merged as is, thank you!

iluuu1994 · 2023-10-29T11:29:25Z

It seems like this broke the CircleCI build. Can somebody take a look?
https://app.circleci.com/pipelines/github/php/php-src/183/workflows/2a3d0a5d-c5f2-40bb-9856-dfc830984744/jobs/171

Girgias · 2023-10-29T15:05:16Z

I'm having a look

iluuu1994 · 2023-11-08T09:20:35Z

ODBC frequently deadlocks in nightly. https://github.com/php/php-src/actions/runs/6792494456/job/18465897192 Can we re-unparallelize this, at least for CI only? I don't think there's currently an option for this, maybe allow CONFLICTS to be a PHP section and then opt-into parallelized execution using an env var?

Girgias · 2023-11-08T13:02:05Z

ODBC frequently deadlocks in nightly. https://github.com/php/php-src/actions/runs/6792494456/job/18465897192 Can we re-unparallelize this, at least for CI only? I don't think there's currently an option for this, maybe allow CONFLICTS to be a PHP section and then opt-into parallelized execution using an env var?

I think just readding the CONFLICT file is better for the moment.

iluuu1994 · 2023-11-09T10:26:23Z

I re-added the CONFLICTS files. Feel free to remove them if you can improve the stability, or maybe rather just enable parallelization for local development.

kocsismate · 2023-11-09T17:02:56Z

Thanks Ilija, I wasn't able to add the conflicts file yesterday. I am fed up fixing these locking issues so I am fine with the current circumvention.

kocsismate requested review from iluuu1994 and Girgias October 10, 2023 07:40

github-actions bot added Extension: pdo (core) Extension: pdo_dblib Extension: pdo_firebird Extension: pdo_oci Extension: pdo_odbc Extension: pdo_pgsql labels Oct 10, 2023

kocsismate force-pushed the pdo-test-parallel branch 3 times, most recently from 6738c98 to e69e7fa Compare October 10, 2023 09:27

Girgias reviewed Oct 10, 2023

View reviewed changes

github-actions bot added the Extension: pdo_sqlite label Oct 12, 2023

kocsismate force-pushed the pdo-test-parallel branch 2 times, most recently from eb9955d to 25820bb Compare October 12, 2023 12:44

Girgias reviewed Oct 14, 2023

View reviewed changes

ext/pdo/tests/bug_36798.phpt Outdated Show resolved Hide resolved

ext/pdo_odbc/tests/long_columns.phpt Show resolved Hide resolved

ext/pdo_sqlite/tests/pdo_sqlite_createcollation.phpt Show resolved Hide resolved

kocsismate force-pushed the pdo-test-parallel branch 2 times, most recently from ef94333 to fbc9c85 Compare October 16, 2023 21:11

kocsismate added 3 commits October 16, 2023 23:41

Parallelize pdo tests (dblib, firebird, oci, odbc, pgsql)

fca4593

It becomes possibly by using different symbols in all tests.

Address review comments

0e789bb

Fix tests

dcfa0c8

kocsismate force-pushed the pdo-test-parallel branch from fbc9c85 to dcfa0c8 Compare October 16, 2023 21:41

kocsismate force-pushed the pdo-test-parallel branch from 2d7ab2a to b0aea91 Compare October 17, 2023 20:11

Really fix tests

b7ec7b0

kocsismate force-pushed the pdo-test-parallel branch from b0aea91 to b7ec7b0 Compare October 17, 2023 20:42

kocsismate force-pushed the pdo-test-parallel branch from 394386a to 586e2ea Compare October 19, 2023 19:37

Girgias reviewed Oct 20, 2023

View reviewed changes

ext/pdo/tests/pdo_test.inc Show resolved Hide resolved

kocsismate added 3 commits October 20, 2023 20:56

Try to fix character encoding

b663919

Readd conflicts to firebird

9de4ea5

As I suppose there is still some concurrency issue.

Fixes

6f1c8b9

jorgsowa reviewed Oct 21, 2023

View reviewed changes

ext/pdo_firebird/tests/ddl.phpt Outdated Show resolved Hide resolved

Update ext/pdo_firebird/tests/ddl.phpt

0ea3181

Co-authored-by: Jorg Adam Sowa <[email protected]>

kocsismate added 2 commits October 21, 2023 18:49

Fixes

44fdd25

Fix firebird?

13760c2

kocsismate force-pushed the pdo-test-parallel branch from e2c8274 to 13760c2 Compare October 22, 2023 21:23

Girgias reviewed Oct 22, 2023

View reviewed changes

ext/pdo/tests/pdo_017.phpt Show resolved Hide resolved

kocsismate requested a review from TimWolla as a code owner October 24, 2023 19:26

kocsismate force-pushed the pdo-test-parallel branch from 3036cfe to 771a21b Compare October 24, 2023 20:47

Another attempt to fix Firebird + debug

298d60e

kocsismate force-pushed the pdo-test-parallel branch from 771a21b to 298d60e Compare October 24, 2023 21:48

Use different notification channel names in pdo_pgsql tests

191227f

kocsismate force-pushed the pdo-test-parallel branch from b11d97b to 191227f Compare October 28, 2023 09:31

Girgias merged commit f4a5db3 into php:master Oct 28, 2023

kocsismate deleted the pdo-test-parallel branch October 29, 2023 09:41

Girgias mentioned this pull request Oct 29, 2023

PDO: Fix parrallel conflicts? #12552

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parallelize pdo tests (dblib, firebird, oci, odbc, pgsql) #12395

Parallelize pdo tests (dblib, firebird, oci, odbc, pgsql) #12395

kocsismate commented Oct 10, 2023

jorgsowa commented Oct 10, 2023

kocsismate commented Oct 10, 2023

kocsismate commented Oct 10, 2023 •

edited

Loading

Girgias commented Oct 16, 2023

Girgias commented Oct 18, 2023

Girgias commented Oct 21, 2023

Girgias left a comment

kocsismate commented Oct 24, 2023

kocsismate commented Oct 24, 2023

Girgias commented Oct 28, 2023

iluuu1994 commented Oct 29, 2023

Girgias commented Oct 29, 2023

iluuu1994 commented Nov 8, 2023

Girgias commented Nov 8, 2023

iluuu1994 commented Nov 9, 2023

kocsismate commented Nov 9, 2023

Parallelize pdo tests (dblib, firebird, oci, odbc, pgsql) #12395

Parallelize pdo tests (dblib, firebird, oci, odbc, pgsql) #12395

Conversation

kocsismate commented Oct 10, 2023

jorgsowa commented Oct 10, 2023

kocsismate commented Oct 10, 2023

kocsismate commented Oct 10, 2023 • edited Loading

Girgias commented Oct 16, 2023

Girgias commented Oct 18, 2023

Girgias commented Oct 21, 2023

Girgias left a comment

Choose a reason for hiding this comment

kocsismate commented Oct 24, 2023

kocsismate commented Oct 24, 2023

Girgias commented Oct 28, 2023

iluuu1994 commented Oct 29, 2023

Girgias commented Oct 29, 2023

iluuu1994 commented Nov 8, 2023

Girgias commented Nov 8, 2023

iluuu1994 commented Nov 9, 2023

kocsismate commented Nov 9, 2023

kocsismate commented Oct 10, 2023 •

edited

Loading