pg_upgrade: Add --swap for faster file transfer.

This new option instructs pg_upgrade to move the data directories
from the old cluster to the new cluster and then to replace the
catalog files with those generated for the new cluster.  This mode
can outperform --link, --clone, --copy, and --copy-file-range,
especially on clusters with many relations.

However, this mode creates many garbage files in the old cluster,
which can prolong the file synchronization step.  To handle that,
we use "initdb --sync-only --no-sync-data-files" for file
synchronization, and we synchronize the catalog files as they are
transferred.  We assume that the database files transferred from
the old cluster were synchronized prior to upgrade.

This mode also complicates reverting to the old cluster, so we
recommend restoring from backup upon failure during or after file
transfer.  We did consider teaching pg_upgrade how to generate a
revert script for such failures, but we decided against it due to
the rarity of failing during file transfer, the complexity of
generating the script, and the potential for misusing the script.

The new mode is limited to clusters located in the same file
system.  With some effort, we could probably support upgrades
between different file systems, but this mode is unlikely to offer
much benefit if we have to copy the files across file system
boundaries.

It is also limited to upgrades from version 10 or newer.  There are
a few known obstacles for using swap mode to upgrade from older
versions.  For example, the visibility map format changed in v9.6,
and the sequence tuple format changed in v10.  In fact, swap mode
omits the --sequence-data option in its uses of pg_dump and instead
reuses the old cluster's sequence data files.  While teaching swap
mode to deal with these kinds of changes is surely possible (and we
may have to deal with similar problems in the future, anyway), it
doesn't seem worth the effort to support upgrades from
long-unsupported versions.

Reviewed-by: Greg Sabino Mullane <[email protected]>
Reviewed-by: Bruce Momjian <[email protected]>
Reviewed-by: Robert Haas <[email protected]>
Discussion: https://postgr.es/m/Zyvop-LxLXBLrZil%40nathan

Author: nathan-bossart

doc/src/sgml/ref/pgupgrade.sgml

@@ -244,7 +244,8 @@ PostgreSQL documentation
       <listitem>
        <para>
         Copy files to the new cluster.  This is the default.  (See also
-        <option>--link</option> and <option>--clone</option>.)
+        <option>--link</option>, <option>--clone</option>,
+        <option>--copy-file-range</option>, and <option>--swap</option>.)
        </para>
       </listitem>
      </varlistentry>
@@ -262,6 +263,32 @@ PostgreSQL documentation
       </listitem>
      </varlistentry>
 
+     <varlistentry>
+      <term><option>--swap</option></term>
+      <listitem>
+       <para>
+        Move the data directories from the old cluster to the new cluster.
+        Then, replace the catalog files with those generated for the new
+        cluster.  This mode can outperform <option>--link</option>,
+        <option>--clone</option>, <option>--copy</option>, and
+        <option>--copy-file-range</option>, especially on clusters with many
+        relations.
+       </para>
+       <para>
+        However, this mode creates many garbage files in the old cluster, which
+        can prolong the file synchronization step if
+        <option>--sync-method=syncfs</option> is used.  Therefore, it is
+        recommended to use <option>--sync-method=fsync</option> with
+        <option>--swap</option>.
+       </para>
+       <para>
+        Additionally, once the file transfer step begins, the old cluster will
+        be destructively modified and therefore will no longer be safe to
+        start.  See <xref linkend="pgupgrade-step-revert"/> for details.
+       </para>
+      </listitem>
+     </varlistentry>
+
      <varlistentry>
       <term><option>--sync-method=</option><replaceable>method</replaceable></term>
       <listitem>
@@ -530,6 +557,10 @@ NET STOP postgresql-&majorversion;
      is started.  Clone mode also requires that the old and new data
      directories be in the same file system.  This mode is only available
      on certain operating systems and file systems.
+     Swap mode may be the fastest if there are many relations, but you will not
+     be able to access your old cluster once the file transfer step begins.
+     Swap mode also requires that the old and new cluster data directories be
+     in the same file system.
     </para>
 
     <para>
@@ -889,6 +920,32 @@ psql --username=postgres --file=script.sql postgres
 
         </itemizedlist></para>
       </listitem>
+
+      <listitem>
+       <para>
+        If the <option>--swap</option> option was used, the old cluster might
+        be destructively modified:
+
+        <itemizedlist>
+         <listitem>
+          <para>
+           If <command>pg_upgrade</command> aborts before reporting that the
+           old cluster is no longer safe to start, the old cluster was
+           unmodified; it can be restarted.
+          </para>
+         </listitem>
+
+         <listitem>
+          <para>
+           If <command>pg_upgrade</command> has reported that the old cluster
+           is no longer safe to start, the old cluster was destructively
+           modified.  The old cluster will need to be restored from backup in
+           this case.
+          </para>
+         </listitem>
+        </itemizedlist>
+       </para>
+      </listitem>
      </itemizedlist></para>
    </step>
   </procedure>

src/bin/pg_upgrade/TESTING

@@ -20,13 +20,13 @@ export oldinstall=...otherversion/	(old version's install base path)
 See DETAILS below for more information about creation of the dump.
 
 You can also test the different transfer modes (--copy, --link,
---clone, --copy-file-range) by setting the environment variable
+--clone, --copy-file-range, --swap) by setting the environment variable
 PG_TEST_PG_UPGRADE_MODE to the respective command-line option, like
 
 	make check PG_TEST_PG_UPGRADE_MODE=--link
 
-The default is --copy.  Note that the other modes are not supported on
-all operating systems.
+The default is --copy.  Note that not all modes are supported on all
+operating systems.
 
 DETAILS
 -------

src/bin/pg_upgrade/check.c

@@ -709,7 +709,34 @@ check_new_cluster(void)
 			check_copy_file_range();
 			break;
 		case TRANSFER_MODE_LINK:
-			check_hard_link();
+			check_hard_link(TRANSFER_MODE_LINK);
+			break;
+		case TRANSFER_MODE_SWAP:
+
+			/*
+			 * We do the hard link check for --swap, too, since it's an easy
+			 * way to verify the clusters are in the same file system.  This
+			 * allows us to take some shortcuts in the file synchronization
+			 * step.  With some more effort, we could probably support the
+			 * separate-file-system use case, but this mode is unlikely to
+			 * offer much benefit if we have to copy the files across file
+			 * system boundaries.
+			 */
+			check_hard_link(TRANSFER_MODE_SWAP);
+
+			/*
+			 * There are a few known issues with using --swap to upgrade from
+			 * versions older than 10.  For example, the sequence tuple format
+			 * changed in v10, and the visibility map format changed in 9.6.
+			 * While such problems are not insurmountable (and we may have to
+			 * deal with similar problems in the future, anyway), it doesn't
+			 * seem worth the effort to support swap mode for upgrades from
+			 * long-unsupported versions.
+			 */
+			if (GET_MAJOR_VERSION(old_cluster.major_version) < 1000)
+				pg_fatal("Swap mode can only upgrade clusters from PostgreSQL version %s and later.",
+						 "10");
+
 			break;
 	}
 

src/bin/pg_upgrade/controldata.c

@@ -751,7 +751,7 @@ check_control_data(ControlData *oldctrl,
 
 
 void
-disable_old_cluster(void)
+disable_old_cluster(transferMode transfer_mode)
 {
 	char		old_path[MAXPGPATH],
 				new_path[MAXPGPATH];
@@ -766,10 +766,17 @@ disable_old_cluster(void)
 				 old_path, new_path);
 	check_ok();
 
-	pg_log(PG_REPORT, "\n"
-		   "If you want to start the old cluster, you will need to remove\n"
-		   "the \".old\" suffix from %s/global/pg_control.old.\n"
-		   "Because \"link\" mode was used, the old cluster cannot be safely\n"
-		   "started once the new cluster has been started.",
-		   old_cluster.pgdata);
+	if (transfer_mode == TRANSFER_MODE_LINK)
+		pg_log(PG_REPORT, "\n"
+			   "If you want to start the old cluster, you will need to remove\n"
+			   "the \".old\" suffix from %s/global/pg_control.old.\n"
+			   "Because \"link\" mode was used, the old cluster cannot be safely\n"
+			   "started once the new cluster has been started.",
+			   old_cluster.pgdata);
+	else if (transfer_mode == TRANSFER_MODE_SWAP)
+		pg_log(PG_REPORT, "\n"
+			   "Because \"swap\" mode was used, the old cluster can no longer be\n"
+			   "safely started.");
+	else
+		pg_fatal("unrecognized transfer mode");
 }

src/bin/pg_upgrade/dump.c

@@ -52,9 +52,11 @@ generate_old_dump(void)
 		snprintf(log_file_name, sizeof(log_file_name), DB_DUMP_LOG_FILE_MASK, old_db->db_oid);
 
 		parallel_exec_prog(log_file_name, NULL,
-						   "\"%s/pg_dump\" %s --no-data %s --sequence-data --quote-all-identifiers "
+						   "\"%s/pg_dump\" %s --no-data %s %s --quote-all-identifiers "
 						   "--binary-upgrade --format=custom %s --no-sync --file=\"%s/%s\" %s",
 						   new_cluster.bindir, cluster_conn_opts(&old_cluster),
+						   (user_opts.transfer_mode == TRANSFER_MODE_SWAP) ?
+						   "" : "--sequence-data",
 						   log_opts.verbose ? "--verbose" : "",
 						   user_opts.do_statistics ? "" : "--no-statistics",
 						   log_opts.dumpdir,

src/bin/pg_upgrade/file.c

@@ -434,7 +434,7 @@ check_copy_file_range(void)
 }
 
 void
-check_hard_link(void)
+check_hard_link(transferMode transfer_mode)
 {
 	char		existing_file[MAXPGPATH];
 	char		new_link_file[MAXPGPATH];
@@ -444,8 +444,16 @@ check_hard_link(void)
 	unlink(new_link_file);		/* might fail */
 
 	if (link(existing_file, new_link_file) < 0)
-		pg_fatal("could not create hard link between old and new data directories: %m\n"
-				 "In link mode the old and new data directories must be on the same file system.");
+	{
+		if (transfer_mode == TRANSFER_MODE_LINK)
+			pg_fatal("could not create hard link between old and new data directories: %m\n"
+					 "In link mode the old and new data directories must be on the same file system.");
+		else if (transfer_mode == TRANSFER_MODE_SWAP)
+			pg_fatal("could not create hard link between old and new data directories: %m\n"
+					 "In swap mode the old and new data directories must be on the same file system.");
+		else
+			pg_fatal("unrecognized transfer mode");
+	}
 
 	unlink(new_link_file);
 }

src/bin/pg_upgrade/info.c

@@ -490,7 +490,7 @@ get_rel_infos_query(void)
 					  "  FROM pg_catalog.pg_class c JOIN pg_catalog.pg_namespace n "
 					  "         ON c.relnamespace = n.oid "
 					  "  WHERE relkind IN (" CppAsString2(RELKIND_RELATION) ", "
-					  CppAsString2(RELKIND_MATVIEW) ") AND "
+					  CppAsString2(RELKIND_MATVIEW) "%s) AND "
 	/* exclude possible orphaned temp tables */
 					  "    ((n.nspname !~ '^pg_temp_' AND "
 					  "      n.nspname !~ '^pg_toast_temp_' AND "
@@ -499,6 +499,8 @@ get_rel_infos_query(void)
 					  "      c.oid >= %u::pg_catalog.oid) OR "
 					  "     (n.nspname = 'pg_catalog' AND "
 					  "      relname IN ('pg_largeobject') ))), ",
+					  (user_opts.transfer_mode == TRANSFER_MODE_SWAP) ?
+					  ", " CppAsString2(RELKIND_SEQUENCE) : "",
 					  FirstNormalObjectId);
 
 	/*

src/bin/pg_upgrade/option.c

@@ -62,6 +62,7 @@ parseCommandLine(int argc, char *argv[])
 		{"sync-method", required_argument, NULL, 4},
 		{"no-statistics", no_argument, NULL, 5},
 		{"set-char-signedness", required_argument, NULL, 6},
+		{"swap", no_argument, NULL, 7},
 
 		{NULL, 0, NULL, 0}
 	};
@@ -228,6 +229,11 @@ parseCommandLine(int argc, char *argv[])
 				else
 					pg_fatal("invalid argument for option %s", "--set-char-signedness");
 				break;
+
+			case 7:
+				user_opts.transfer_mode = TRANSFER_MODE_SWAP;
+				break;
+
 			default:
 				fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
 						os_info.progname);
@@ -325,6 +331,7 @@ usage(void)
 	printf(_("  --no-statistics               do not import statistics from old cluster\n"));
 	printf(_("  --set-char-signedness=OPTION  set new cluster char signedness to \"signed\" or\n"
 			 "                                \"unsigned\"\n"));
+	printf(_("  --swap                        move data directories to new cluster\n"));
 	printf(_("  --sync-method=METHOD          set method for syncing files to disk\n"));
 	printf(_("  -?, --help                    show this help, then exit\n"));
 	printf(_("\n"

src/bin/pg_upgrade/pg_upgrade.c

@@ -170,12 +170,14 @@ main(int argc, char **argv)
 
 	/*
 	 * Most failures happen in create_new_objects(), which has completed at
-	 * this point.  We do this here because it is just before linking, which
-	 * will link the old and new cluster data files, preventing the old
-	 * cluster from being safely started once the new cluster is started.
+	 * this point.  We do this here because it is just before file transfer,
+	 * which for --link will make it unsafe to start the old cluster once the
+	 * new cluster is started, and for --swap will make it unsafe to start the
+	 * old cluster at all.
 	 */
-	if (user_opts.transfer_mode == TRANSFER_MODE_LINK)
-		disable_old_cluster();
+	if (user_opts.transfer_mode == TRANSFER_MODE_LINK ||
+		user_opts.transfer_mode == TRANSFER_MODE_SWAP)
+		disable_old_cluster(user_opts.transfer_mode);
 
 	transfer_all_new_tablespaces(&old_cluster.dbarr, &new_cluster.dbarr,
 								 old_cluster.pgdata, new_cluster.pgdata);
@@ -212,8 +214,10 @@ main(int argc, char **argv)
 	{
 		prep_status("Sync data directory to disk");
 		exec_prog(UTILITY_LOG_FILE, NULL, true, true,
-				  "\"%s/initdb\" --sync-only \"%s\" --sync-method %s",
+				  "\"%s/initdb\" --sync-only %s \"%s\" --sync-method %s",
 				  new_cluster.bindir,
+				  (user_opts.transfer_mode == TRANSFER_MODE_SWAP) ?
+				  "--no-sync-data-files" : "",
 				  new_cluster.pgdata,
 				  user_opts.sync_method);
 		check_ok();

src/bin/pg_upgrade/pg_upgrade.h

@@ -262,6 +262,7 @@ typedef enum
 	TRANSFER_MODE_COPY,
 	TRANSFER_MODE_COPY_FILE_RANGE,
 	TRANSFER_MODE_LINK,
+	TRANSFER_MODE_SWAP,
 } transferMode;
 
 /*
@@ -391,7 +392,7 @@ void		create_script_for_old_cluster_deletion(char **deletion_script_file_name);
 
 void		get_control_data(ClusterInfo *cluster);
 void		check_control_data(ControlData *oldctrl, ControlData *newctrl);
-void		disable_old_cluster(void);
+void		disable_old_cluster(transferMode transfer_mode);
 
 
 /* dump.c */
@@ -423,7 +424,7 @@ void		rewriteVisibilityMap(const char *fromfile, const char *tofile,
 								 const char *schemaName, const char *relName);
 void		check_file_clone(void);
 void		check_copy_file_range(void);
-void		check_hard_link(void);
+void		check_hard_link(transferMode transfer_mode);
 
 /* fopen_priv() is no longer different from fopen() */
 #define fopen_priv(path, mode)	fopen(path, mode)