PostgreSQL Build Farm Log
Details for system "hamster" failure at stage RecoveryCheck, snapshot taken 2016-07-24 16:00:07
| System Information |
Farm member |
Branch |
OS |
Compiler |
Architecture |
Owner |
| hamster |
master |
Arch Linux ARM testing |
gcc 4.7.2 |
armv6l |
michael [ a t ] otacoo.com |
Configuration summary
This file was created by PostgreSQL configure 9.6beta3, which was
generated by GNU Autoconf 2.69. Invocation command line was
$ ./configure --enable-cassert --enable-debug --enable-nls --enable-tap-tests \
--with-perl --with-tcl --with-openssl --with-includes=/usr/include/et \
--prefix=/home/buildfarm/data/buildroot/HEAD/inst \
--with-pgport=5678
hostname = hamster
uname -m = armv6l
uname -r = 4.4.14-2-ARCH
uname -s = Linux
uname -v = #1 Sat Jul 2 20:40:32 MDT 2016
PATH: /usr/bin/core_perl
PATH: /usr/bin
PATH: /bin
========================================================
$Script_Config = {
'target' => 'http://www.pgbuildfarm.org/cgi-bin/pgstatus.pl',
'mail_events' => {
'change' => [],
'fail' => [],
'green' => [],
'all' => []
},
'base_port' => 5678,
'use_vpath' => undef,
'alerts' => {},
'steps_completed' => 'SCM-checkout Configure Make Check Contrib TestModules Install ContribInstall TestModulesInstall pg_upgradeCheck test-decoding-check BinInstallCheck',
'scm' => 'git',
'orig_env' => {
'OLDPWD' => 'xxxxxx',
'_' => 'xxxxxx',
'SHELL' => '/bin/sh',
'LOGNAME' => 'buildfarm',
'PATH' => '/usr/bin:/bin',
'PWD' => 'xxxxxx',
'USER' => 'buildfarm',
'HOME' => '/home/buildfarm',
'SHLVL' => 'xxxxxx'
},
'keep_error_builds' => 0,
'scm_url' => undef,
'make' => 'make',
'git_keep_mirror' => 1,
'script_version' => 'REL_4.17',
'web_script_version' => 'REL_4.17',
'current_ts' => 1469384098,
'ccache_failure_remove' => undef,
'tar_log_cmd' => undef,
'animal' => 'hamster',
'config_opts' => [
'--enable-cassert',
'--enable-debug',
'--enable-nls',
'--enable-tap-tests',
'--with-perl',
'--with-tcl',
'--with-openssl',
'--with-includes=/usr/include/et'
],
'use_git_cvsserver' => undef,
'trigger_exclude' => qr/\/doc\/|\.po/,
'git_ignore_mirror_failure' => 1,
'extra_config' => {
'DEFAULT' => [
'log_line_prefix = \'[%c:%l] \'',
'log_connections = \'true\'',
'log_disconnections = \'true\'',
'log_statement = \'all\'',
'fsync = off',
'stats_temp_directory= \'/home/buildfarm/data/stats_temp\''
]
},
'build_root' => '/home/buildfarm/data/buildroot',
'invocation_args' => [
'--config',
'build-farm.conf',
'HEAD'
],
'config_env' => {
'CC' => 'ccache gcc'
},
'trigger_include' => undef,
'core_file_glob' => 'core*',
'upgrade_target' => 'http://www.pgbuildfarm.org/cgi-bin/upgrade.pl',
'make_jobs' => undef,
'optional_steps' => {},
'modules' => [
'TestUpgrade',
'TestDecoding'
],
'using_msvc' => undef,
'locales' => [
'C'
],
'build_env' => {
'PATH' => '/usr/bin/core_perl:/usr/bin:/bin',
'CCACHE_DIR' => '/home/buildfarm/data/ccache'
},
'scmrepo' => '/home/buildfarm/data/postgres.git/',
'force_every' => {},
'module_versions' => {
'PGBuild::SCM' => 'REL_4.17',
'PGBuild::Modules::TestDecoding' => 'REL_4.17',
'PGBuild::WebTxn' => 'REL_4.17',
'PGBuild::Modules::TestUpgrade' => 'REL_4.17',
'PGBuild::Options' => 'REL_4.17'
},
'aux_path' => '/home/buildfarm/data/buildfarm'
};
Log
Last file mtime in snapshot: Sun Jul 24 00:30:03 2016 GMT
===================================================
rm -rf '/home/buildfarm/data/buildroot/HEAD/pgsql.build'/tmp_install
/usr/bin/mkdir -p '/home/buildfarm/data/buildroot/HEAD/pgsql.build'/tmp_install/log
make -C '../../..' DESTDIR='/home/buildfarm/data/buildroot/HEAD/pgsql.build'/tmp_install install >'/home/buildfarm/data/buildroot/HEAD/pgsql.build'/tmp_install/log/install.log 2>&1
rm -rf /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log
cd . && TESTDIR='/home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery' PATH="/home/buildfarm/data/buildroot/HEAD/pgsql.build/tmp_install/home/buildfarm/data/buildroot/HEAD/inst/bin:$PATH" LD_LIBRARY_PATH="/home/buildfarm/data/buildroot/HEAD/pgsql.build/tmp_install/home/buildfarm/data/buildroot/HEAD/inst/lib:$LD_LIBRARY_PATH" PGPORT='65678' PG_REGRESS='/home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/../../../src/test/regress/pg_regress' prove -I ../../../src/test/perl/ --verbose t/*.pl
t/001_stream_rep.pl ........
1..4
ok 1 - check streamed content on standby 1
ok 2 - check streamed content on standby 2
ok 3 - Read-only queries on standby 1
ok 4 - Read-only queries on standby 2
ok
t/002_archiving.pl .........
1..1
ok 1 - check content from archives
ok
#
# Looks like you planned 7 tests but ran 1.
# Looks like your test exited with 25 just after 1.
t/003_recovery_targets.pl ..
1..7
ok 1 - check standby content for immediate target
Dubious, test returned 25 (wstat 6400, 0x1900)
Failed 6/7 subtests
t/004_timeline_switch.pl ...
1..1
ok 1 - check content of standby 2
ok
t/005_replay_delay.pl ......
1..1
ok 1 - Check that standby applies WAL only after replication delay
ok
t/007_sync_rep.pl ..........
1..8
ok 1 - old syntax of synchronous_standby_names
ok 2 - asterisk in synchronous_standby_names
ok 3 - 2 synchronous standbys
ok 4 - 2 sync, 1 potential and 1 async
ok 5 - num_sync exceeds the num of potential sync standbys
ok 6 - asterisk comes before another standby name
ok 7 - multiple standbys having the same priority are chosen as sync
ok 8 - potential standby found earlier in array is promoted to sync
ok
Test Summary Report
-------------------
t/003_recovery_targets.pl (Wstat: 6400 Tests: 1 Failed: 0)
Non-zero exit status: 25
Parse errors: Bad plan. You planned 7 tests but ran 1.
Files=6, Tests=16, 1150 wallclock secs ( 0.82 usr 0.07 sys + 229.94 cusr 55.39 csys = 286.22 CPU)
Result: FAIL
make: *** [Makefile:17: check] Error 1
================== pgsql.build/src/test/recovery/tmp_check/log/001_stream_rep_master.log ===================
LOG: database system was shut down at 2016-07-25 02:56:49 JST
LOG: MultiXact member wraparound protections are now enabled
LOG: database system is ready to accept connections
LOG: autovacuum launcher started
LOG: statement: CREATE TABLE tab_int AS SELECT generate_series(1,1002) AS a
LOG: statement: SELECT pg_current_xlog_location() <= write_location FROM pg_stat_replication WHERE application_name = 'standby_1';
LOG: received immediate shutdown request
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
LOG: database system is shut down
================== pgsql.build/src/test/recovery/tmp_check/log/001_stream_rep_standby_1.log ===================
LOG: database system was interrupted; last known up at 2016-07-25 02:57:06 JST
LOG: entering standby mode
LOG: redo starts at 0/2000028
LOG: consistent recovery state reached at 0/2000130
LOG: database system is ready to accept read only connections
LOG: started streaming WAL from primary at 0/3000000 on timeline 1
LOG: statement: SELECT pg_last_xlog_replay_location() <= write_location FROM pg_stat_replication WHERE application_name = 'standby_2';
LOG: statement: SELECT count(*) FROM tab_int
LOG: statement: INSERT INTO tab_int VALUES (1)
ERROR: cannot execute INSERT in a read-only transaction
STATEMENT: INSERT INTO tab_int VALUES (1)
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
FATAL: could not receive data from WAL stream: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
LOG: invalid record length at 0/30255F0: wanted 24, got 0
FATAL: could not connect to the primary server: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
LOG: received immediate shutdown request
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
LOG: database system is shut down
================== pgsql.build/src/test/recovery/tmp_check/log/001_stream_rep_standby_2.log ===================
LOG: database system was interrupted while in recovery at log time 2016-07-25 02:57:06 JST
HINT: If this has occurred more than once some data might be corrupted and you might need to choose an earlier recovery target.
LOG: entering standby mode
LOG: redo starts at 0/2000028
LOG: consistent recovery state reached at 0/3000000
LOG: database system is ready to accept read only connections
LOG: started streaming WAL from primary at 0/3000000 on timeline 1
LOG: could not send data to client: Broken pipe
FATAL: connection to client lost
LOG: statement: SELECT count(*) FROM tab_int
LOG: statement: INSERT INTO tab_int VALUES (1)
ERROR: cannot execute INSERT in a read-only transaction
STATEMENT: INSERT INTO tab_int VALUES (1)
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
FATAL: could not receive data from WAL stream: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
LOG: invalid record length at 0/30255F0: wanted 24, got 0
FATAL: could not connect to the primary server: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/tmp/1WeJae2Q0G/.s.PGSQL.63028"?
LOG: received immediate shutdown request
LOG: database system is shut down
================== pgsql.build/src/test/recovery/tmp_check/log/002_archiving_master.log ===================
LOG: database system was shut down at 2016-07-25 03:00:02 JST
FATAL: the database system is starting up
LOG: MultiXact member wraparound protections are now enabled
LOG: autovacuum launcher started
LOG: database system is ready to accept connections
LOG: statement: CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a
LOG: statement: SELECT pg_current_xlog_location();
LOG: statement: SELECT pg_switch_xlog()
LOG: statement: INSERT INTO tab_int VALUES (generate_series(1001,2000))
LOG: received immediate shutdown request
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
LOG: archiver process (PID 20328) exited with exit code 1
LOG: statistics collector process (PID 20329) was terminated by signal 9: Killed
LOG: database system is shut down
================== pgsql.build/src/test/recovery/tmp_check/log/002_archiving_standby.log ===================
LOG: database system was interrupted; last known up at 2016-07-25 03:00:32 JST
LOG: entering standby mode
FATAL: the database system is starting up
FATAL: the database system is starting up
LOG: restored log file "000000010000000000000002" from archive
FATAL: the database system is starting up
FATAL: the database system is starting up
LOG: redo starts at 0/2000028
LOG: consistent recovery state reached at 0/20000F8
LOG: database system is ready to accept read only connections
cp: cannot stat '/home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/archives/000000010000000000000003': No such file or directory
cp: cannot stat '/home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/archives/000000010000000000000003': No such file or directory
cp: cannot stat '/home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/archives/000000010000000000000003': No such file or directory
cp: cannot stat '/home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/archives/000000010000000000000003': No such file or directory
cp: cannot stat '/home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/archives/000000010000000000000003': No such file or directory
cp: cannot stat '/home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/archives/000000010000000000000003': No such file or directory
cp: cannot stat '/home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/archives/000000010000000000000003': No such file or directory
cp: cannot stat '/home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/archives/000000010000000000000003': No such file or directory
cp: cannot stat '/home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/archives/000000010000000000000003': No such file or directory
cp: cannot stat '/home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/archives/000000010000000000000003': No such file or directory
cp: cannot stat '/home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/archives/000000010000000000000003': No such file or directory
cp: cannot stat '/home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/archives/000000010000000000000003': No such file or directory
cp: cannot stat '/home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/archives/000000010000000000000003': No such file or directory
cp: cannot stat '/home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/archives/000000010000000000000003': No such file or directory
LOG: restored log file "000000010000000000000003" from archive
cp: cannot stat '/home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/archives/000000010000000000000004': No such file or directory
cp: cannot stat '/home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/archives/000000010000000000000004': No such file or directory
LOG: statement: SELECT '0/30255A0'::pg_lsn <= pg_last_xlog_replay_location()
cp: cannot stat '/home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/archives/000000010000000000000004': No such file or directory
cp: cannot stat '/home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/archives/000000010000000000000004': No such file or directory
LOG: statement: SELECT count(*) FROM tab_int
cp: cannot stat '/home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/archives/000000010000000000000004': No such file or directory
cp: cannot stat '/home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/archives/000000010000000000000004': No such file or directory
cp: cannot stat '/home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/archives/000000010000000000000004': No such file or directory
cp: cannot stat '/home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/archives/000000010000000000000004': No such file or directory
cp: cannot stat '/home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/archives/000000010000000000000004': No such file or directory
cp: cannot stat '/home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/archives/000000010000000000000004': No such file or directory
cp: cannot stat '/home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/archives/000000010000000000000004': No such file or directory
cp: cannot stat '/home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/archives/000000010000000000000004': No such file or directory
cp: cannot stat '/home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/archives/000000010000000000000004': No such file or directory
cp: cannot stat '/home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/archives/000000010000000000000004': No such file or directory
cp: cannot stat '/home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/archives/000000010000000000000004': No such file or directory
cp: cannot stat '/home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/archives/000000010000000000000004': No such file or directory
cp: cannot stat '/home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/archives/000000010000000000000004': No such file or directory
cp: cannot stat '/home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/archives/000000010000000000000004': No such file or directory
cp: cannot stat '/home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/archives/000000010000000000000004': No such file or directory
cp: cannot stat '/home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/archives/000000010000000000000004': No such file or directory
LOG: received immediate shutdown request
FATAL: could not restore file "000000010000000000000004" from archive: child process was terminated by signal 3: Quit
LOG: database system is shut down
================== pgsql.build/src/test/recovery/tmp_check/log/003_recovery_targets_master.log ===================
LOG: database system was shut down at 2016-07-25 03:02:21 JST
LOG: MultiXact member wraparound protections are now enabled
LOG: autovacuum launcher started
LOG: database system is ready to accept connections
LOG: statement: CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a
LOG: statement: SELECT pg_current_xlog_location();
LOG: statement: INSERT INTO tab_int VALUES (generate_series(1001,2000))
LOG: statement: SELECT txid_current()
LOG: statement: SELECT pg_current_xlog_location();
LOG: statement: INSERT INTO tab_int VALUES (generate_series(2001,3000))
LOG: statement: SELECT now()
LOG: statement: SELECT pg_current_xlog_location();
LOG: statement: INSERT INTO tab_int VALUES (generate_series(3001,4000))
LOG: statement: SELECT pg_current_xlog_location();
LOG: statement: SELECT pg_create_restore_point('my_target');
LOG: restore point "my_target" created at 0/303BBB0
STATEMENT: SELECT pg_create_restore_point('my_target');
LOG: statement: SELECT pg_switch_xlog()
LOG: received immediate shutdown request
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
LOG: archiver process (PID 20451) exited with exit code 1
LOG: database system is shut down
================== pgsql.build/src/test/recovery/tmp_check/log/003_recovery_targets_standby_1.log ===================
LOG: database system was interrupted; last known up at 2016-07-25 03:02:33 JST
LOG: entering standby mode
LOG: restored log file "000000010000000000000002" from archive
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
LOG: could not send data to client: Broken pipe
LOG: redo starts at 0/2000028
LOG: consistent recovery state reached at 0/2000130
LOG: recovery stopping after reaching consistency
LOG: recovery has paused
HINT: Execute pg_xlog_replay_resume() to continue.
LOG: database system is ready to accept read only connections
LOG: statement: SELECT '0/15121E0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT count(*) FROM tab_int
LOG: received immediate shutdown request
LOG: database system is shut down
================== pgsql.build/src/test/recovery/tmp_check/log/003_recovery_targets_standby_2.log ===================
LOG: database system was interrupted; last known up at 2016-07-25 03:02:33 JST
LOG: entering standby mode
LOG: restored log file "000000010000000000000002" from archive
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
FATAL: the database system is starting up
LOG: redo starts at 0/2000028
LOG: consistent recovery state reached at 0/2000130
LOG: database system is ready to accept read only connections
LOG: restored log file "000000010000000000000003" from archive
LOG: recovery stopping after commit of transaction 548, time 2016-07-25 03:03:23.646193+09
LOG: recovery has paused
HINT: Execute pg_xlog_replay_resume() to continue.
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT '0/30122D0'::pg_lsn <= pg_last_xlog_replay_location()
LOG: received immediate shutdown request
LOG: database system is shut down
================== pgsql.build/src/test/recovery/tmp_check/log/004_timeline_switch_master.log ===================
LOG: database system was shut down at 2016-07-25 03:06:52 JST
LOG: MultiXact member wraparound protections are now enabled
LOG: database system is ready to accept connections
LOG: autovacuum launcher started
LOG: statement: CREATE TABLE tab_int AS SELECT generate_series(1,1000) AS a
LOG: statement: SELECT pg_current_xlog_location();
LOG: received immediate shutdown request
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
LOG: statistics collector process (PID 20733) was terminated by signal 9: Killed
LOG: database system is shut down
================== pgsql.build/src/test/recovery/tmp_check/log/004_timeline_switch_standby_1.log ===================
LOG: database system was interrupted; last known up at 2016-07-25 03:07:05 JST
LOG: entering standby mode
FATAL: the database system is starting up
FATAL: the database system is starting up
LOG: redo starts at 0/2000028
LOG: consistent recovery state reached at 0/2000130
LOG: database system is ready to accept read only connections
LOG: started streaming WAL from primary at 0/3000000 on timeline 1
LOG: statement: SELECT '0/3025570'::pg_lsn <= pg_last_xlog_replay_location()
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
FATAL: could not receive data from WAL stream: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
LOG: invalid record length at 0/3025570: wanted 24, got 0
FATAL: could not connect to the primary server: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
FATAL: could not connect to the primary server: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/tmp/ZVLQd1G5um/.s.PGSQL.52931"?
FATAL: could not connect to the primary server: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/tmp/ZVLQd1G5um/.s.PGSQL.52931"?
FATAL: could not connect to the primary server: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/tmp/ZVLQd1G5um/.s.PGSQL.52931"?
LOG: received promote request
LOG: redo done at 0/3025538
LOG: last completed transaction was at log time 2016-07-25 03:08:18.648515+09
LOG: selected new timeline ID: 2
LOG: archive recovery complete
LOG: MultiXact member wraparound protections are now enabled
LOG: database system is ready to accept connections
LOG: autovacuum launcher started
LOG: statement: SELECT pg_is_in_recovery() <> true
LOG: statement: INSERT INTO tab_int VALUES (generate_series(1001,2000))
LOG: statement: SELECT pg_current_xlog_location();
LOG: received immediate shutdown request
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
LOG: database system is shut down
================== pgsql.build/src/test/recovery/tmp_check/log/004_timeline_switch_standby_2.log ===================
LOG: database system was interrupted; last known up at 2016-07-25 03:07:05 JST
LOG: entering standby mode
LOG: redo starts at 0/2000028
LOG: consistent recovery state reached at 0/2000130
LOG: database system is ready to accept read only connections
LOG: started streaming WAL from primary at 0/3000000 on timeline 1
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
FATAL: could not receive data from WAL stream: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
LOG: invalid record length at 0/3025570: wanted 24, got 0
FATAL: could not connect to the primary server: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
LOG: received fast shutdown request
LOG: aborting any active transactions
LOG: shutting down
LOG: database system is shut down
LOG: database system was shut down in recovery at 2016-07-25 03:08:46 JST
LOG: entering standby mode
LOG: redo starts at 0/2000028
LOG: consistent recovery state reached at 0/3025570
LOG: database system is ready to accept read only connections
LOG: invalid record length at 0/3025570: wanted 24, got 0
LOG: started streaming WAL from primary at 0/3000000 on timeline 1
LOG: replication terminated by primary server
DETAIL: End of WAL reached on timeline 1 at 0/3025570.
LOG: fetching timeline history file for timeline 2 from primary server
LOG: new target timeline is 2
LOG: restarted WAL streaming at 0/3000000 on timeline 2
LOG: statement: SELECT '0/3037488'::pg_lsn <= pg_last_xlog_replay_location()
LOG: statement: SELECT count(*) FROM tab_int
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
FATAL: could not receive data from WAL stream: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
LOG: invalid record length at 0/3037488: wanted 24, got 0
FATAL: could not connect to the primary server: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/tmp/ZVLQd1G5um/.s.PGSQL.52932"?
LOG: received immediate shutdown request
LOG: database system is shut down
================== pgsql.build/src/test/recovery/tmp_check/log/005_replay_delay_master.log ===================
LOG: database system was shut down at 2016-07-25 03:09:38 JST
LOG: MultiXact member wraparound protections are now enabled
LOG: database system is ready to accept connections
LOG: autovacuum launcher started
LOG: statement: CREATE TABLE tab_int AS SELECT generate_series(1, 10) AS a
LOG: statement: INSERT INTO tab_int VALUES (generate_series(11, 20))
LOG: statement: SELECT pg_current_xlog_location()
LOG: received immediate shutdown request
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
LOG: database system is shut down
================== pgsql.build/src/test/recovery/tmp_check/log/005_replay_delay_standby.log ===================
LOG: database system was interrupted; last known up at 2016-07-25 03:09:41 JST
LOG: entering standby mode
LOG: redo starts at 0/2000028
LOG: consistent recovery state reached at 0/20000F8
LOG: database system is ready to accept read only connections
LOG: started streaming WAL from primary at 0/3000000 on timeline 1
LOG: statement: SELECT (pg_last_xlog_replay_location() - '0/3001E00'::pg_lsn) >= 0
LOG: statement: SELECT (pg_last_xlog_replay_location() - '0/3001E00'::pg_lsn) >= 0
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
FATAL: could not receive data from WAL stream: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
LOG: invalid record length at 0/3001E38: wanted 24, got 0
LOG: received immediate shutdown request
LOG: database system is shut down
================== pgsql.build/src/test/recovery/tmp_check/log/007_sync_rep_master.log ===================
LOG: database system was shut down at 2016-07-25 03:11:29 JST
LOG: MultiXact member wraparound protections are now enabled
LOG: database system is ready to accept connections
LOG: autovacuum launcher started
LOG: using stale statistics instead of current ones because stats collector is not responding
LOG: statement: ALTER SYSTEM SET synchronous_standby_names = 'standby1,standby2';
LOG: received SIGHUP, reloading configuration files
LOG: parameter "synchronous_standby_names" changed to "standby1,standby2"
LOG: statement: SELECT application_name, sync_priority, sync_state FROM pg_stat_replication ORDER BY application_name;
LOG: statement: ALTER SYSTEM SET synchronous_standby_names = '*';
LOG: received SIGHUP, reloading configuration files
LOG: parameter "synchronous_standby_names" changed to "*"
LOG: statement: SELECT application_name, sync_priority, sync_state FROM pg_stat_replication ORDER BY application_name;
LOG: standby "standby2" is now a synchronous standby with priority 1
LOG: statement: ALTER SYSTEM SET synchronous_standby_names = '2(standby1,standby2,standby3)';
LOG: received SIGHUP, reloading configuration files
LOG: parameter "synchronous_standby_names" changed to "2(standby1,standby2,standby3)"
LOG: statement: SELECT application_name, sync_priority, sync_state FROM pg_stat_replication ORDER BY application_name;
LOG: standby "standby1" is now a synchronous standby with priority 1
LOG: statement: SELECT application_name, sync_priority, sync_state FROM pg_stat_replication ORDER BY application_name;
LOG: statement: ALTER SYSTEM SET synchronous_standby_names = '6(standby4,standby0,standby3,standby2)';
LOG: received SIGHUP, reloading configuration files
LOG: parameter "synchronous_standby_names" changed to "6(standby4,standby0,standby3,standby2)"
LOG: statement: SELECT application_name, sync_priority, sync_state FROM pg_stat_replication ORDER BY application_name;
LOG: statement: ALTER SYSTEM SET synchronous_standby_names = '2(standby1,*,standby2)';
LOG: standby "standby4" is now a synchronous standby with priority 1
LOG: received SIGHUP, reloading configuration files
LOG: parameter "synchronous_standby_names" changed to "2(standby1,*,standby2)"
LOG: statement: SELECT application_name, sync_priority, sync_state FROM pg_stat_replication ORDER BY application_name;
LOG: statement: ALTER SYSTEM SET synchronous_standby_names = '2(*)';
LOG: received SIGHUP, reloading configuration files
LOG: parameter "synchronous_standby_names" changed to "2(*)"
LOG: statement: SELECT application_name, sync_priority, sync_state FROM pg_stat_replication ORDER BY application_name;
LOG: statement: SELECT application_name, sync_priority, sync_state FROM pg_stat_replication ORDER BY application_name;
LOG: received immediate shutdown request
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
LOG: database system is shut down
================== pgsql.build/src/test/recovery/tmp_check/log/007_sync_rep_standby1.log ===================
LOG: database system was interrupted; last known up at 2016-07-25 03:11:44 JST
LOG: entering standby mode
LOG: redo starts at 0/2000028
LOG: consistent recovery state reached at 0/2000130
LOG: database system is ready to accept read only connections
LOG: started streaming WAL from primary at 0/3000000 on timeline 1
LOG: received fast shutdown request
FATAL: terminating walreceiver process due to administrator command
LOG: aborting any active transactions
LOG: shutting down
LOG: database system is shut down
LOG: database system was shut down in recovery at 2016-07-25 03:13:54 JST
LOG: entering standby mode
LOG: redo starts at 0/2000028
LOG: consistent recovery state reached at 0/3000060
LOG: invalid record length at 0/3000060: wanted 24, got 0
LOG: database system is ready to accept read only connections
LOG: started streaming WAL from primary at 0/3000000 on timeline 1
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
FATAL: could not receive data from WAL stream: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
FATAL: could not connect to the primary server: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/tmp/KY7LQTZCSC/.s.PGSQL.49703"?
LOG: received immediate shutdown request
LOG: database system is shut down
================== pgsql.build/src/test/recovery/tmp_check/log/007_sync_rep_standby2.log ===================
LOG: database system was interrupted; last known up at 2016-07-25 03:11:44 JST
LOG: entering standby mode
LOG: redo starts at 0/2000028
LOG: consistent recovery state reached at 0/2000130
LOG: database system is ready to accept read only connections
LOG: started streaming WAL from primary at 0/3000000 on timeline 1
LOG: could not send data to client: Broken pipe
FATAL: connection to client lost
LOG: received fast shutdown request
FATAL: terminating walreceiver process due to administrator command
LOG: aborting any active transactions
LOG: shutting down
LOG: database system is shut down
LOG: database system was shut down in recovery at 2016-07-25 03:13:55 JST
LOG: entering standby mode
LOG: redo starts at 0/2000028
LOG: consistent recovery state reached at 0/3000060
LOG: invalid record length at 0/3000060: wanted 24, got 0
LOG: database system is ready to accept read only connections
LOG: started streaming WAL from primary at 0/3000000 on timeline 1
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
FATAL: could not receive data from WAL stream: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
FATAL: could not connect to the primary server: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/tmp/KY7LQTZCSC/.s.PGSQL.49703"?
LOG: received immediate shutdown request
LOG: database system is shut down
================== pgsql.build/src/test/recovery/tmp_check/log/007_sync_rep_standby3.log ===================
LOG: database system was interrupted; last known up at 2016-07-25 03:11:44 JST
LOG: entering standby mode
LOG: redo starts at 0/2000028
LOG: consistent recovery state reached at 0/2000130
LOG: database system is ready to accept read only connections
LOG: started streaming WAL from primary at 0/3000000 on timeline 1
LOG: received fast shutdown request
FATAL: terminating walreceiver process due to administrator command
LOG: aborting any active transactions
LOG: shutting down
LOG: database system is shut down
LOG: database system was shut down in recovery at 2016-07-25 03:13:56 JST
LOG: entering standby mode
LOG: redo starts at 0/2000028
LOG: consistent recovery state reached at 0/3000060
LOG: invalid record length at 0/3000060: wanted 24, got 0
LOG: database system is ready to accept read only connections
LOG: started streaming WAL from primary at 0/3000000 on timeline 1
LOG: received fast shutdown request
FATAL: terminating walreceiver process due to administrator command
LOG: aborting any active transactions
LOG: shutting down
LOG: database system is shut down
================== pgsql.build/src/test/recovery/tmp_check/log/007_sync_rep_standby4.log ===================
LOG: database system was interrupted; last known up at 2016-07-25 03:11:44 JST
LOG: entering standby mode
LOG: redo starts at 0/2000028
LOG: consistent recovery state reached at 0/2000130
LOG: database system is ready to accept read only connections
LOG: started streaming WAL from primary at 0/3000000 on timeline 1
LOG: could not send data to client: Broken pipe
FATAL: connection to client lost
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
FATAL: could not receive data from WAL stream: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
LOG: invalid record length at 0/3000060: wanted 24, got 0
LOG: received immediate shutdown request
LOG: database system is shut down
================== pgsql.build/src/test/recovery/tmp_check/log/regress_log_001_stream_rep ===================
# Checking port 63027
# Found free port 63027
Name: master
Data directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_YTra/pgdata
Backup directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_YTra/backup
Archive directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_YTra/archives
Connection string: port=63027 host=/tmp/1WeJae2Q0G
Log file: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/001_stream_rep_master.log
# Running: initdb -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_YTra/pgdata -A trust -N
The files belonging to this database system will be owned by user "buildfarm".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_YTra/pgdata ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... No usable system locales were found.
Use the option "--debug" to see details.
ok
Sync to disk skipped.
The data directory might become corrupt if the operating system crashes.
Success. You can now start the database server using:
pg_ctl -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_YTra/pgdata -l logfile start
# Running: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/../../../src/test/regress/pg_regress --config-auth /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_YTra/pgdata
### Starting node "master"
# Running: pg_ctl -w -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_YTra/pgdata -l /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/001_stream_rep_master.log start
waiting for server to start.... done
server started
# Postmaster PID for node "master" is 20238
# Taking pg_basebackup my_backup from node "master"
# Running: pg_basebackup -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_YTra/backup/my_backup -p 63027 -x
# Backup finished
# Checking port 63028
# Found free port 63028
Name: standby_1
Data directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_1_EUaV/pgdata
Backup directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_1_EUaV/backup
Archive directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_1_EUaV/archives
Connection string: port=63028 host=/tmp/1WeJae2Q0G
Log file: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/001_stream_rep_standby_1.log
# Initializing node "standby_1" from backup "my_backup" of node "master"
### Enabling streaming replication for node "standby_1"
### Starting node "standby_1"
# Running: pg_ctl -w -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_1_EUaV/pgdata -l /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/001_stream_rep_standby_1.log start
waiting for server to start.... done
server started
# Postmaster PID for node "standby_1" is 20250
# Taking pg_basebackup my_backup from node "standby_1"
# Running: pg_basebackup -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_1_EUaV/backup/my_backup -p 63028 -x
# Backup finished
# Checking port 63029
# Found free port 63029
Name: standby_2
Data directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_2_7DVH/pgdata
Backup directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_2_7DVH/backup
Archive directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_2_7DVH/archives
Connection string: port=63029 host=/tmp/1WeJae2Q0G
Log file: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/001_stream_rep_standby_2.log
# Initializing node "standby_2" from backup "my_backup" of node "standby_1"
### Enabling streaming replication for node "standby_2"
### Starting node "standby_2"
# Running: pg_ctl -w -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_2_7DVH/pgdata -l /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/001_stream_rep_standby_2.log start
waiting for server to start..... done
server started
# Postmaster PID for node "standby_2" is 20265
standby 1: 1002
ok 1 - check streamed content on standby 1
standby 2: 1002
ok 2 - check streamed content on standby 2
psql:<stdin>:1: ERROR: cannot execute INSERT in a read-only transaction
ok 3 - Read-only queries on standby 1
psql:<stdin>:1: ERROR: cannot execute INSERT in a read-only transaction
ok 4 - Read-only queries on standby 2
### Stopping node "master" using mode immediate
# Running: pg_ctl -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_YTra/pgdata -m immediate stop
waiting for server to shut down.... done
server stopped
# No postmaster PID
### Stopping node "standby_1" using mode immediate
# Running: pg_ctl -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_1_EUaV/pgdata -m immediate stop
waiting for server to shut down.... done
server stopped
# No postmaster PID
### Stopping node "standby_2" using mode immediate
# Running: pg_ctl -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_2_7DVH/pgdata -m immediate stop
waiting for server to shut down... done
server stopped
# No postmaster PID
================== pgsql.build/src/test/recovery/tmp_check/log/regress_log_002_archiving ===================
# Checking port 50892
# Found free port 50892
Name: master
Data directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/pgdata
Backup directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/backup
Archive directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/archives
Connection string: port=50892 host=/tmp/PvstYXhOjM
Log file: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/002_archiving_master.log
# Running: initdb -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/pgdata -A trust -N
The files belonging to this database system will be owned by user "buildfarm".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/pgdata ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... No usable system locales were found.
Use the option "--debug" to see details.
ok
Sync to disk skipped.
The data directory might become corrupt if the operating system crashes.
Success. You can now start the database server using:
pg_ctl -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/pgdata -l logfile start
# Running: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/../../../src/test/regress/pg_regress --config-auth /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/pgdata
### Enabling WAL archiving for node "master"
### Starting node "master"
# Running: pg_ctl -w -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/pgdata -l /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/002_archiving_master.log start
waiting for server to start..... done
server started
# Postmaster PID for node "master" is 20321
# Taking pg_basebackup my_backup from node "master"
# Running: pg_basebackup -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/backup/my_backup -p 50892 -x
# Backup finished
# Checking port 50893
# Found free port 50893
Name: standby
Data directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_QhTp/pgdata
Backup directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_QhTp/backup
Archive directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_QhTp/archives
Connection string: port=50893 host=/tmp/PvstYXhOjM
Log file: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/002_archiving_standby.log
# Initializing node "standby" from backup "my_backup" of node "master"
### Enabling WAL restore for node "standby"
### Starting node "standby"
# Running: pg_ctl -w -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_QhTp/pgdata -l /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/002_archiving_standby.log start
waiting for server to start........ done
server started
# Postmaster PID for node "standby" is 20352
ok 1 - check content from archives
### Stopping node "master" using mode immediate
# Running: pg_ctl -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_u4p_/pgdata -m immediate stop
waiting for server to shut down..... done
server stopped
# No postmaster PID
### Stopping node "standby" using mode immediate
# Running: pg_ctl -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_QhTp/pgdata -m immediate stop
waiting for server to shut down.... done
server stopped
# No postmaster PID
================== pgsql.build/src/test/recovery/tmp_check/log/regress_log_003_recovery_targets ===================
# Checking port 65142
# Found free port 65142
Name: master
Data directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_XLiy/pgdata
Backup directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_XLiy/backup
Archive directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_XLiy/archives
Connection string: port=65142 host=/tmp/fGvtYMOjqA
Log file: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/003_recovery_targets_master.log
# Running: initdb -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_XLiy/pgdata -A trust -N
The files belonging to this database system will be owned by user "buildfarm".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_XLiy/pgdata ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... No usable system locales were found.
Use the option "--debug" to see details.
ok
Sync to disk skipped.
The data directory might become corrupt if the operating system crashes.
Success. You can now start the database server using:
pg_ctl -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_XLiy/pgdata -l logfile start
# Running: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/../../../src/test/regress/pg_regress --config-auth /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_XLiy/pgdata
### Enabling WAL archiving for node "master"
### Starting node "master"
# Running: pg_ctl -w -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_XLiy/pgdata -l /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/003_recovery_targets_master.log start
waiting for server to start.... done
server started
# Postmaster PID for node "master" is 20445
# Taking pg_basebackup my_backup from node "master"
# Running: pg_basebackup -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_XLiy/backup/my_backup -p 65142 -x
# Backup finished
# Checking port 65143
# Found free port 65143
Name: standby_1
Data directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_1_Vlvv/pgdata
Backup directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_1_Vlvv/backup
Archive directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_1_Vlvv/archives
Connection string: port=65143 host=/tmp/fGvtYMOjqA
Log file: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/003_recovery_targets_standby_1.log
# Initializing node "standby_1" from backup "my_backup" of node "master"
### Enabling WAL restore for node "standby_1"
### Starting node "standby_1"
# Running: pg_ctl -w -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_1_Vlvv/pgdata -l /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/003_recovery_targets_standby_1.log start
waiting for server to start........ done
server started
# Postmaster PID for node "standby_1" is 20489
ok 1 - check standby content for immediate target
### Stopping node "standby_1" using mode immediate
# Running: pg_ctl -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_1_Vlvv/pgdata -m immediate stop
waiting for server to shut down... done
server stopped
# No postmaster PID
# Checking port 65144
# Found free port 65144
Name: standby_2
Data directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_2_V61Z/pgdata
Backup directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_2_V61Z/backup
Archive directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_2_V61Z/archives
Connection string: port=65144 host=/tmp/fGvtYMOjqA
Log file: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/003_recovery_targets_standby_2.log
# Initializing node "standby_2" from backup "my_backup" of node "master"
### Enabling WAL restore for node "standby_2"
### Starting node "standby_2"
# Running: pg_ctl -w -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_2_V61Z/pgdata -l /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/003_recovery_targets_standby_2.log start
waiting for server to start........ done
server started
# Postmaster PID for node "standby_2" is 20510
#
Timed out while waiting for standby to catch up at t/003_recovery_targets.pl line 36.
### Stopping node "master" using mode immediate
# Running: pg_ctl -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_XLiy/pgdata -m immediate stop
waiting for server to shut down... done
server stopped
# No postmaster PID
### Stopping node "standby_2" using mode immediate
# Running: pg_ctl -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_2_V61Z/pgdata -m immediate stop
waiting for server to shut down... done
server stopped
# No postmaster PID
# Looks like you planned 7 tests but ran 1.
# Looks like your test exited with 25 just after 1.
================== pgsql.build/src/test/recovery/tmp_check/log/regress_log_004_timeline_switch ===================
# Checking port 52931
# Found free port 52931
Name: master
Data directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_MJFs/pgdata
Backup directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_MJFs/backup
Archive directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_MJFs/archives
Connection string: port=52931 host=/tmp/ZVLQd1G5um
Log file: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/004_timeline_switch_master.log
# Running: initdb -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_MJFs/pgdata -A trust -N
The files belonging to this database system will be owned by user "buildfarm".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_MJFs/pgdata ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... No usable system locales were found.
Use the option "--debug" to see details.
ok
Sync to disk skipped.
The data directory might become corrupt if the operating system crashes.
Success. You can now start the database server using:
pg_ctl -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_MJFs/pgdata -l logfile start
# Running: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/../../../src/test/regress/pg_regress --config-auth /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_MJFs/pgdata
### Starting node "master"
# Running: pg_ctl -w -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_MJFs/pgdata -l /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/004_timeline_switch_master.log start
waiting for server to start.... done
server started
# Postmaster PID for node "master" is 20727
# Taking pg_basebackup my_backup from node "master"
# Running: pg_basebackup -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_MJFs/backup/my_backup -p 52931 -x
# Backup finished
# Checking port 52932
# Found free port 52932
Name: standby_1
Data directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_1_osjk/pgdata
Backup directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_1_osjk/backup
Archive directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_1_osjk/archives
Connection string: port=52932 host=/tmp/ZVLQd1G5um
Log file: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/004_timeline_switch_standby_1.log
# Initializing node "standby_1" from backup "my_backup" of node "master"
### Enabling streaming replication for node "standby_1"
### Starting node "standby_1"
# Running: pg_ctl -w -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_1_osjk/pgdata -l /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/004_timeline_switch_standby_1.log start
waiting for server to start...... done
server started
# Postmaster PID for node "standby_1" is 20741
# Checking port 52933
# Found free port 52933
Name: standby_2
Data directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_2_nVkl/pgdata
Backup directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_2_nVkl/backup
Archive directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_2_nVkl/archives
Connection string: port=52933 host=/tmp/ZVLQd1G5um
Log file: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/004_timeline_switch_standby_2.log
# Initializing node "standby_2" from backup "my_backup" of node "master"
### Enabling streaming replication for node "standby_2"
### Starting node "standby_2"
# Running: pg_ctl -w -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_2_nVkl/pgdata -l /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/004_timeline_switch_standby_2.log start
waiting for server to start.... done
server started
# Postmaster PID for node "standby_2" is 20756
### Stopping node "master" using mode immediate
# Running: pg_ctl -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_MJFs/pgdata -m immediate stop
waiting for server to shut down....... done
server stopped
# No postmaster PID
### Promoting node "standby_1"
# Running: pg_ctl -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_1_osjk/pgdata -l /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/004_timeline_switch_standby_1.log promote
server promoting
### Restarting node "standby_2"
# Running: pg_ctl -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_2_nVkl/pgdata -w -l /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/004_timeline_switch_standby_2.log restart
waiting for server to shut down... done
server stopped
waiting for server to start.... done
server started
# Postmaster PID for node "standby_2" is 20778
ok 1 - check content of standby 2
### Stopping node "standby_1" using mode immediate
# Running: pg_ctl -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_1_osjk/pgdata -m immediate stop
waiting for server to shut down.... done
server stopped
# No postmaster PID
### Stopping node "standby_2" using mode immediate
# Running: pg_ctl -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_2_nVkl/pgdata -m immediate stop
waiting for server to shut down... done
server stopped
# No postmaster PID
================== pgsql.build/src/test/recovery/tmp_check/log/regress_log_005_replay_delay ===================
# Checking port 53380
# Found free port 53380
Name: master
Data directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_LPK7/pgdata
Backup directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_LPK7/backup
Archive directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_LPK7/archives
Connection string: port=53380 host=/tmp/ckgPk_5w5A
Log file: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/005_replay_delay_master.log
# Running: initdb -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_LPK7/pgdata -A trust -N
The files belonging to this database system will be owned by user "buildfarm".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_LPK7/pgdata ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... No usable system locales were found.
Use the option "--debug" to see details.
ok
Sync to disk skipped.
The data directory might become corrupt if the operating system crashes.
Success. You can now start the database server using:
pg_ctl -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_LPK7/pgdata -l logfile start
# Running: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/../../../src/test/regress/pg_regress --config-auth /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_LPK7/pgdata
### Starting node "master"
# Running: pg_ctl -w -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_LPK7/pgdata -l /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/005_replay_delay_master.log start
waiting for server to start.... done
server started
# Postmaster PID for node "master" is 20817
# Taking pg_basebackup my_backup from node "master"
# Running: pg_basebackup -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_LPK7/backup/my_backup -p 53380 -x
# Backup finished
# Checking port 53381
# Found free port 53381
Name: standby
Data directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_sfyC/pgdata
Backup directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_sfyC/backup
Archive directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_sfyC/archives
Connection string: port=53381 host=/tmp/ckgPk_5w5A
Log file: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/005_replay_delay_standby.log
# Initializing node "standby" from backup "my_backup" of node "master"
### Enabling streaming replication for node "standby"
### Starting node "standby"
# Running: pg_ctl -w -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_sfyC/pgdata -l /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/005_replay_delay_standby.log start
waiting for server to start.... done
server started
# Postmaster PID for node "standby" is 20836
ok 1 - Check that standby applies WAL only after replication delay
### Stopping node "master" using mode immediate
# Running: pg_ctl -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_LPK7/pgdata -m immediate stop
waiting for server to shut down..... done
server stopped
# No postmaster PID
### Stopping node "standby" using mode immediate
# Running: pg_ctl -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby_sfyC/pgdata -m immediate stop
waiting for server to shut down... done
server stopped
# No postmaster PID
================== pgsql.build/src/test/recovery/tmp_check/log/regress_log_007_sync_rep ===================
# Checking port 49703
# Found free port 49703
Name: master
Data directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_vikz/pgdata
Backup directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_vikz/backup
Archive directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_vikz/archives
Connection string: port=49703 host=/tmp/KY7LQTZCSC
Log file: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/007_sync_rep_master.log
# Running: initdb -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_vikz/pgdata -A trust -N
The files belonging to this database system will be owned by user "buildfarm".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_vikz/pgdata ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... No usable system locales were found.
Use the option "--debug" to see details.
ok
Sync to disk skipped.
The data directory might become corrupt if the operating system crashes.
Success. You can now start the database server using:
pg_ctl -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_vikz/pgdata -l logfile start
# Running: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/../../../src/test/regress/pg_regress --config-auth /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_vikz/pgdata
### Starting node "master"
# Running: pg_ctl -w -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_vikz/pgdata -l /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/007_sync_rep_master.log start
waiting for server to start.... done
server started
# Postmaster PID for node "master" is 20876
# Taking pg_basebackup master_backup from node "master"
# Running: pg_basebackup -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_vikz/backup/master_backup -p 49703 -x
# Backup finished
# Checking port 49704
# Found free port 49704
Name: standby1
Data directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby1_3d1D/pgdata
Backup directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby1_3d1D/backup
Archive directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby1_3d1D/archives
Connection string: port=49704 host=/tmp/KY7LQTZCSC
Log file: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/007_sync_rep_standby1.log
# Initializing node "standby1" from backup "master_backup" of node "master"
### Enabling streaming replication for node "standby1"
### Starting node "standby1"
# Running: pg_ctl -w -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby1_3d1D/pgdata -l /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/007_sync_rep_standby1.log start
waiting for server to start.... done
server started
# Postmaster PID for node "standby1" is 20891
# Checking port 49705
# Found free port 49705
Name: standby2
Data directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby2_rUuj/pgdata
Backup directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby2_rUuj/backup
Archive directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby2_rUuj/archives
Connection string: port=49705 host=/tmp/KY7LQTZCSC
Log file: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/007_sync_rep_standby2.log
# Initializing node "standby2" from backup "master_backup" of node "master"
### Enabling streaming replication for node "standby2"
### Starting node "standby2"
# Running: pg_ctl -w -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby2_rUuj/pgdata -l /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/007_sync_rep_standby2.log start
waiting for server to start..... done
server started
# Postmaster PID for node "standby2" is 20901
# Checking port 49706
# Found free port 49706
Name: standby3
Data directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby3_XH2t/pgdata
Backup directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby3_XH2t/backup
Archive directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby3_XH2t/archives
Connection string: port=49706 host=/tmp/KY7LQTZCSC
Log file: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/007_sync_rep_standby3.log
# Initializing node "standby3" from backup "master_backup" of node "master"
### Enabling streaming replication for node "standby3"
### Starting node "standby3"
# Running: pg_ctl -w -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby3_XH2t/pgdata -l /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/007_sync_rep_standby3.log start
waiting for server to start.... done
server started
# Postmaster PID for node "standby3" is 20914
### Reloading node "master"
# Running: pg_ctl -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_vikz/pgdata reload
server signaled
ok 1 - old syntax of synchronous_standby_names
### Reloading node "master"
# Running: pg_ctl -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_vikz/pgdata reload
server signaled
ok 2 - asterisk in synchronous_standby_names
### Stopping node "standby1" using mode fast
# Running: pg_ctl -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby1_3d1D/pgdata -m fast stop
waiting for server to shut down.... done
server stopped
# No postmaster PID
### Stopping node "standby2" using mode fast
# Running: pg_ctl -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby2_rUuj/pgdata -m fast stop
waiting for server to shut down.... done
server stopped
# No postmaster PID
### Stopping node "standby3" using mode fast
# Running: pg_ctl -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby3_XH2t/pgdata -m fast stop
waiting for server to shut down... done
server stopped
# No postmaster PID
### Starting node "standby2"
# Running: pg_ctl -w -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby2_rUuj/pgdata -l /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/007_sync_rep_standby2.log start
waiting for server to start.... done
server started
# Postmaster PID for node "standby2" is 20938
### Starting node "standby3"
# Running: pg_ctl -w -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby3_XH2t/pgdata -l /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/007_sync_rep_standby3.log start
waiting for server to start.... done
server started
# Postmaster PID for node "standby3" is 20948
### Reloading node "master"
# Running: pg_ctl -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_vikz/pgdata reload
server signaled
ok 3 - 2 synchronous standbys
### Starting node "standby1"
# Running: pg_ctl -w -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby1_3d1D/pgdata -l /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/007_sync_rep_standby1.log start
waiting for server to start.... done
server started
# Postmaster PID for node "standby1" is 20963
# Checking port 49707
# Found free port 49707
Name: standby4
Data directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby4_OrBF/pgdata
Backup directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby4_OrBF/backup
Archive directory: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby4_OrBF/archives
Connection string: port=49707 host=/tmp/KY7LQTZCSC
Log file: /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/007_sync_rep_standby4.log
# Initializing node "standby4" from backup "master_backup" of node "master"
### Enabling streaming replication for node "standby4"
### Starting node "standby4"
# Running: pg_ctl -w -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby4_OrBF/pgdata -l /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/log/007_sync_rep_standby4.log start
waiting for server to start..... done
server started
# Postmaster PID for node "standby4" is 20976
ok 4 - 2 sync, 1 potential and 1 async
### Reloading node "master"
# Running: pg_ctl -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_vikz/pgdata reload
server signaled
ok 5 - num_sync exceeds the num of potential sync standbys
### Reloading node "master"
# Running: pg_ctl -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_vikz/pgdata reload
server signaled
ok 6 - asterisk comes before another standby name
### Reloading node "master"
# Running: pg_ctl -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_vikz/pgdata reload
server signaled
ok 7 - multiple standbys having the same priority are chosen as sync
### Stopping node "standby3" using mode fast
# Running: pg_ctl -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby3_XH2t/pgdata -m fast stop
waiting for server to shut down.... done
server stopped
# No postmaster PID
ok 8 - potential standby found earlier in array is promoted to sync
### Stopping node "master" using mode immediate
# Running: pg_ctl -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_master_vikz/pgdata -m immediate stop
waiting for server to shut down... done
server stopped
# No postmaster PID
### Stopping node "standby1" using mode immediate
# Running: pg_ctl -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby1_3d1D/pgdata -m immediate stop
waiting for server to shut down... done
server stopped
# No postmaster PID
### Stopping node "standby2" using mode immediate
# Running: pg_ctl -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby2_rUuj/pgdata -m immediate stop
waiting for server to shut down... done
server stopped
# No postmaster PID
### Stopping node "standby4" using mode immediate
# Running: pg_ctl -D /home/buildfarm/data/buildroot/HEAD/pgsql.build/src/test/recovery/tmp_check/data_standby4_OrBF/pgdata -m immediate stop
waiting for server to shut down... done
server stopped
# No postmaster PID
