[PATCH] Add loongarch native checksum implementation.

Hi,

This patch tries to add loongarch native crc32 check with crcc.*
instructions to postgresql.

The patch is tested on my Loongson 3A5000 machine with Loong Arch Linux
and GCC 13.1.0 / clang 16.0.0 with

- default ./configure
- default meson setup

See:

[1]:
https://loongson.github.io/LoongArch-Documentation/LoongArch-Vol1-EN.html#crc-check-instructions
[2]:
https://gcc.gnu.org/onlinedocs/gcc/LoongArch-Base-Built-in-Functions.html
[3]:
https://github.com/llvm/llvm-project/blob/release/16.x/clang/include/clang/Basic/BuiltinsLoongArch.def#L36-L39

On Thu, Jun 8, 2023 at 12:24 PM YANG Xudong <yangxudong(at)ymatrix(dot)cn> wrote:
>
> This patch tries to add loongarch native crc32 check with crcc.*
> instructions to postgresql.
>
> The patch is tested on my Loongson 3A5000 machine with Loong Arch Linux
> and GCC 13.1.0 / clang 16.0.0 with
>
> - default ./configure
> - default meson setup

I took a quick look at this, and it seems mostly in line with other
architectures we support for CRC. I have a couple questions and comments:

configure.ac:

+AC_SUBST(CFLAGS_CRC)

This seems to be an unnecessary copy-paste. I think we only need one, after
all checks have run.

meson.build

+ if cc.links(prog, name: '__builtin_loongarch_crcc_w_b_w,
__builtin_loongarch_crcc_w_h_w, __builtin_loongarch_crcc_w_w_w, and
__builtin_loongarch_crcc_w_d_w without -march=loongarch64',
+ args: test_c_args)
+ # Use LoongArch CRC instruction unconditionally
+ cdata.set('USE_LOONGARCH_CRC32C', 1)
+ have_optimized_crc = true
+ elif cc.links(prog, name: '__builtin_loongarch_crcc_w_b_w,
__builtin_loongarch_crcc_w_h_w, __builtin_loongarch_crcc_w_w_w, and
__builtin_loongarch_crcc_w_d_w with -march=loongarch64',
+ args: test_c_args + ['-march=loongarch64'])
+ # Use LoongArch CRC instruction unconditionally

For x86 and Arm, if it fails to link without an -march flag, we allow for a
runtime check. The flags "-march=armv8-a+crc" and "-msse4.2" are for
instructions not found on all platforms. The patch also checks both ways,
and each one results in "Use LoongArch CRC instruction unconditionally".
The -march flag here is general, not specific. In other words, if this only
runs inside "+elif host_cpu == 'loongarch64'", why do we need both with
-march and without?

Also, I don't have a Loongarch machine for testing. Could you show that the
instructions are found in the binary, maybe using objdump and grep? Or a
performance test?

In the future, you may also consider running the buildfarm client on a
machine dedicated for testing. That will give us quick feedback if some
future new code doesn't work on this platform. More information here:

https://wiki.postgresql.org/wiki/PostgreSQL_Buildfarm_Howto

--
John Naylor
EDB: http://www.enterprisedb.com

Attached a new patch with fixes based on the comment below.

On 2023/6/13 18:26, John Naylor wrote:
>
> On Thu, Jun 8, 2023 at 12:24 PM YANG Xudong <yangxudong(at)ymatrix(dot)cn
> <mailto:yangxudong(at)ymatrix(dot)cn>> wrote:
> >
> > This patch tries to add loongarch native crc32 check with crcc.*
> > instructions to postgresql.
> >
> > The patch is tested on my Loongson 3A5000 machine with Loong Arch Linux
> > and GCC 13.1.0 / clang 16.0.0 with
> >
> > - default ./configure
> > - default meson setup
>
> I took a quick look at this, and it seems mostly in line with other
> architectures we support for CRC. I have a couple questions and comments:
>
> configure.ac <http://configure.ac>:
>
> +AC_SUBST(CFLAGS_CRC)
> > This seems to be an unnecessary copy-paste. I think we only need one,
> after all checks have run.
>

Removed the extra line.

> meson.build
>
> +  if cc.links(prog, name: '__builtin_loongarch_crcc_w_b_w,
> __builtin_loongarch_crcc_w_h_w, __builtin_loongarch_crcc_w_w_w, and
> __builtin_loongarch_crcc_w_d_w without -march=loongarch64',
> +      args: test_c_args)
> +    # Use LoongArch CRC instruction unconditionally
> +    cdata.set('USE_LOONGARCH_CRC32C', 1)
> +    have_optimized_crc = true
> +  elif cc.links(prog, name: '__builtin_loongarch_crcc_w_b_w,
> __builtin_loongarch_crcc_w_h_w, __builtin_loongarch_crcc_w_w_w, and
> __builtin_loongarch_crcc_w_d_w with -march=loongarch64',
> +      args: test_c_args + ['-march=loongarch64'])
> +    # Use LoongArch CRC instruction unconditionally
>
> For x86 and Arm, if it fails to link without an -march flag, we allow
> for a runtime check. The flags "-march=armv8-a+crc" and "-msse4.2" are
> for instructions not found on all platforms. The patch also checks both
> ways, and each one results in "Use LoongArch CRC instruction
> unconditionally". The -march flag here is general, not specific. In
> other words, if this only runs inside "+elif host_cpu == 'loongarch64'",
> why do we need both with -march and without?
>

Removed the elif branch.

> Also, I don't have a Loongarch machine for testing. Could you show that
> the instructions are found in the binary, maybe using objdump and grep?
> Or a performance test?
>

The output of the objdump command `objdump -dS
../postgres-build/tmp_install/usr/local/pgsql/bin/postgres | grep -B 30
-A 10 crcc` is attached.

Also the output of make check is attached.

I run a simple test program to compare the performance of
pg_comp_crc32c_loongarch and pg_comp_crc32c_sb8 on my test machine. The
result is that pg_comp_crc32c_loongarch is over 2x faster than
pg_comp_crc32c_sb8.

> In the future, you may also consider running the buildfarm client on a
> machine dedicated for testing. That will give us quick feedback if some
> future new code doesn't work on this platform. More information here:
>
> https://wiki.postgresql.org/wiki/PostgreSQL_Buildfarm_Howto
> <https://wiki.postgresql.org/wiki/PostgreSQL_Buildfarm_Howto>
>

I will contact the loongson community
(https://github.com/loongson-community) to see if they are able to
provide some machine for buildfarm or not.

> --
> John Naylor
> EDB: http://www.enterprisedb.com <http://www.enterprisedb.com>

--
YANG Xudong

On Wed, Jun 14, 2023 at 9:20 AM YANG Xudong <yangxudong(at)ymatrix(dot)cn> wrote:
>
> Attached a new patch with fixes based on the comment below.

Note: It's helpful to pass "-v" to git format-patch, to have different
versions.

> > For x86 and Arm, if it fails to link without an -march flag, we allow
> > for a runtime check. The flags "-march=armv8-a+crc" and "-msse4.2" are
> > for instructions not found on all platforms. The patch also checks both
> > ways, and each one results in "Use LoongArch CRC instruction
> > unconditionally". The -march flag here is general, not specific. In
> > other words, if this only runs inside "+elif host_cpu == 'loongarch64'",
> > why do we need both with -march and without?
> >
>
> Removed the elif branch.

Okay, since we've confirmed that no arch flag is necessary, some other
places can be simplified:

--- a/src/port/Makefile
+++ b/src/port/Makefile
@@ -98,6 +98,11 @@ pg_crc32c_armv8.o: CFLAGS+=$(CFLAGS_CRC)
pg_crc32c_armv8_shlib.o: CFLAGS+=$(CFLAGS_CRC)
pg_crc32c_armv8_srv.o: CFLAGS+=$(CFLAGS_CRC)

+# all versions of pg_crc32c_loongarch.o need CFLAGS_CRC
+pg_crc32c_loongarch.o: CFLAGS+=$(CFLAGS_CRC)
+pg_crc32c_loongarch_shlib.o: CFLAGS+=$(CFLAGS_CRC)
+pg_crc32c_loongarch_srv.o: CFLAGS+=$(CFLAGS_CRC)

This was copy-and-pasted from platforms that use a runtime check, so should
be unnecessary.

+# If the intrinsics are supported, sets pgac_loongarch_crc32c_intrinsics,
+# and CFLAGS_CRC.

+# Check if __builtin_loongarch_crcc_* intrinsics can be used
+# with the default compiler flags.
+# CFLAGS_CRC is set if the extra flag is required.

Same here -- it seems we don't need to set CFLAGS_CRC at all. Can you
confirm?

> > Also, I don't have a Loongarch machine for testing. Could you show that
> > the instructions are found in the binary, maybe using objdump and grep?
> > Or a performance test?
> >
>
> The output of the objdump command `objdump -dS
> ../postgres-build/tmp_install/usr/local/pgsql/bin/postgres | grep -B 30
> -A 10 crcc` is attached.

Thanks for confirming.

--
John Naylor
EDB: http://www.enterprisedb.com

Updated the patch based on the comments.

On 2023/6/15 18:30, John Naylor wrote:
>
> On Wed, Jun 14, 2023 at 9:20 AM YANG Xudong <yangxudong(at)ymatrix(dot)cn
> <mailto:yangxudong(at)ymatrix(dot)cn>> wrote:
> >
> > Attached a new patch with fixes based on the comment below.
>
> Note: It's helpful to pass "-v" to git format-patch, to have different
> versions.
>

Added v2

> > > For x86 and Arm, if it fails to link without an -march flag, we allow
> > > for a runtime check. The flags "-march=armv8-a+crc" and "-msse4.2" are
> > > for instructions not found on all platforms. The patch also checks both
> > > ways, and each one results in "Use LoongArch CRC instruction
> > > unconditionally". The -march flag here is general, not specific. In
> > > other words, if this only runs inside "+elif host_cpu ==
> 'loongarch64'",
> > > why do we need both with -march and without?
> > >
> >
> > Removed the elif branch.
>
> Okay, since we've confirmed that no arch flag is necessary, some other
> places can be simplified:
>
> --- a/src/port/Makefile
> +++ b/src/port/Makefile
> @@ -98,6 +98,11 @@ pg_crc32c_armv8.o: CFLAGS+=$(CFLAGS_CRC)
>  pg_crc32c_armv8_shlib.o: CFLAGS+=$(CFLAGS_CRC)
>  pg_crc32c_armv8_srv.o: CFLAGS+=$(CFLAGS_CRC)
>
> +# all versions of pg_crc32c_loongarch.o need CFLAGS_CRC
> +pg_crc32c_loongarch.o: CFLAGS+=$(CFLAGS_CRC)
> +pg_crc32c_loongarch_shlib.o: CFLAGS+=$(CFLAGS_CRC)
> +pg_crc32c_loongarch_srv.o: CFLAGS+=$(CFLAGS_CRC)
>
> This was copy-and-pasted from platforms that use a runtime check, so
> should be unnecessary.
>

Removed these lines.

> +# If the intrinsics are supported, sets pgac_loongarch_crc32c_intrinsics,
> +# and CFLAGS_CRC.
>
> +# Check if __builtin_loongarch_crcc_* intrinsics can be used
> +# with the default compiler flags.
> +# CFLAGS_CRC is set if the extra flag is required.
>
> Same here -- it seems we don't need to set CFLAGS_CRC at all. Can you
> confirm?
>

We don't need to set CFLAGS_CRC as commented. I have updated the
configure script to make it align with the logic in meson build script.

> > > Also, I don't have a Loongarch machine for testing. Could you show that
> > > the instructions are found in the binary, maybe using objdump and grep?
> > > Or a performance test?
> > >
> >
> > The output of the objdump command `objdump -dS
> > ../postgres-build/tmp_install/usr/local/pgsql/bin/postgres  | grep -B 30
> > -A 10 crcc` is attached.
>
> Thanks for confirming.
>
> --
> John Naylor
> EDB: http://www.enterprisedb.com <http://www.enterprisedb.com>

Is there any other comment?

If the patch looks OK, I would like to update its status to ready for
committer in the commitfest.

Thanks!

On 2023/6/16 09:28, YANG Xudong wrote:
> Updated the patch based on the comments.
>
> On 2023/6/15 18:30, John Naylor wrote:
>>
>> On Wed, Jun 14, 2023 at 9:20 AM YANG Xudong <yangxudong(at)ymatrix(dot)cn
>> <mailto:yangxudong(at)ymatrix(dot)cn>> wrote:
>>  >
>>  > Attached a new patch with fixes based on the comment below.
>>
>> Note: It's helpful to pass "-v" to git format-patch, to have different
>> versions.
>>
>
> Added v2
>
>>  > > For x86 and Arm, if it fails to link without an -march flag, we
>> allow
>>  > > for a runtime check. The flags "-march=armv8-a+crc" and
>> "-msse4.2" are
>>  > > for instructions not found on all platforms. The patch also
>> checks both
>>  > > ways, and each one results in "Use LoongArch CRC instruction
>>  > > unconditionally". The -march flag here is general, not specific. In
>>  > > other words, if this only runs inside "+elif host_cpu ==
>> 'loongarch64'",
>>  > > why do we need both with -march and without?
>>  > >
>>  >
>>  > Removed the elif branch.
>>
>> Okay, since we've confirmed that no arch flag is necessary, some other
>> places can be simplified:
>>
>> --- a/src/port/Makefile
>> +++ b/src/port/Makefile
>> @@ -98,6 +98,11 @@ pg_crc32c_armv8.o: CFLAGS+=$(CFLAGS_CRC)
>>   pg_crc32c_armv8_shlib.o: CFLAGS+=$(CFLAGS_CRC)
>>   pg_crc32c_armv8_srv.o: CFLAGS+=$(CFLAGS_CRC)
>>
>> +# all versions of pg_crc32c_loongarch.o need CFLAGS_CRC
>> +pg_crc32c_loongarch.o: CFLAGS+=$(CFLAGS_CRC)
>> +pg_crc32c_loongarch_shlib.o: CFLAGS+=$(CFLAGS_CRC)
>> +pg_crc32c_loongarch_srv.o: CFLAGS+=$(CFLAGS_CRC)
>>
>> This was copy-and-pasted from platforms that use a runtime check, so
>> should be unnecessary.
>>
>
> Removed these lines.
>
>> +# If the intrinsics are supported, sets
>> pgac_loongarch_crc32c_intrinsics,
>> +# and CFLAGS_CRC.
>>
>> +# Check if __builtin_loongarch_crcc_* intrinsics can be used
>> +# with the default compiler flags.
>> +# CFLAGS_CRC is set if the extra flag is required.
>>
>> Same here -- it seems we don't need to set CFLAGS_CRC at all. Can you
>> confirm?
>>
>
> We don't need to set CFLAGS_CRC as commented. I have updated the
> configure script to make it align with the logic in meson build script.
>
>>  > > Also, I don't have a Loongarch machine for testing. Could you
>> show that
>>  > > the instructions are found in the binary, maybe using objdump and
>> grep?
>>  > > Or a performance test?
>>  > >
>>  >
>>  > The output of the objdump command `objdump -dS
>>  > ../postgres-build/tmp_install/usr/local/pgsql/bin/postgres  | grep
>> -B 30
>>  > -A 10 crcc` is attached.
>>
>> Thanks for confirming.
>>
>> --
>> John Naylor
>> EDB: http://www.enterprisedb.com <http://www.enterprisedb.com>

On Wed, Jul 5, 2023 at 9:16 AM YANG Xudong <yangxudong(at)ymatrix(dot)cn> wrote:
>
> Is there any other comment?

It's only been a few weeks since the last patch, and this is not an urgent
bugfix, so there is no reason to ping the thread. Feature freeze will
likely be in April of next year.

Also, please don't top-post (which means: quoting an entire message, with
new text at the top) -- it clutters our archives.

Before I look at this again: Are there any objections to another CRC
implementation for the reason of having no buildfarm member?

--
John Naylor
EDB: http://www.enterprisedb.com

Hi, i have a loongarch machine runing Loongnix-server (based on redhat/centos, it has gcc-8.3 on it), i am trying to test buildfarm on it, when i edit build-farm.conf it seems to need animal and secret to connect the buildfarm server.
then i go to https://buildfarm.postgresql.org/cgi-bin/register-form.pl, and registered the buildfarm a week ago, but didn't receive any response. so what else i need to do next.

> -----Original Messages-----
> From: "YANG Xudong" <yangxudong(at)ymatrix(dot)cn>
> Send time:Wednesday, 07/05/2023 10:15:51
> To: "John Naylor" <john(dot)naylor(at)enterprisedb(dot)com>
> Cc: pgsql-hackers(at)lists(dot)postgresql(dot)org, wengyanqing(at)ymatrix(dot)cn, wanghao(at)ymatrix(dot)cn
> Subject: Re: [PATCH] Add loongarch native checksum implementation.
>
> Is there any other comment?
>
> If the patch looks OK, I would like to update its status to ready for
> committer in the commitfest.
>
> Thanks!
>
> On 2023/6/16 09:28, YANG Xudong wrote:
> > Updated the patch based on the comments.
> >
> > On 2023/6/15 18:30, John Naylor wrote:
> >>
> >> On Wed, Jun 14, 2023 at 9:20 AM YANG Xudong <yangxudong(at)ymatrix(dot)cn
> >> <mailto:yangxudong(at)ymatrix(dot)cn>> wrote:
> >>  >
> >>  > Attached a new patch with fixes based on the comment below.
> >>
> >> Note: It's helpful to pass "-v" to git format-patch, to have different
> >> versions.
> >>
> >
> > Added v2
> >
> >>  > > For x86 and Arm, if it fails to link without an -march flag, we
> >> allow
> >>  > > for a runtime check. The flags "-march=armv8-a+crc" and
> >> "-msse4.2" are
> >>  > > for instructions not found on all platforms. The patch also
> >> checks both
> >>  > > ways, and each one results in "Use LoongArch CRC instruction
> >>  > > unconditionally". The -march flag here is general, not specific. In
> >>  > > other words, if this only runs inside "+elif host_cpu ==
> >> 'loongarch64'",
> >>  > > why do we need both with -march and without?
> >>  > >
> >>  >
> >>  > Removed the elif branch.
> >>
> >> Okay, since we've confirmed that no arch flag is necessary, some other
> >> places can be simplified:
> >>
> >> --- a/src/port/Makefile
> >> +++ b/src/port/Makefile
> >> @@ -98,6 +98,11 @@ pg_crc32c_armv8.o: CFLAGS+=$(CFLAGS_CRC)
> >>   pg_crc32c_armv8_shlib.o: CFLAGS+=$(CFLAGS_CRC)
> >>   pg_crc32c_armv8_srv.o: CFLAGS+=$(CFLAGS_CRC)
> >>
> >> +# all versions of pg_crc32c_loongarch.o need CFLAGS_CRC
> >> +pg_crc32c_loongarch.o: CFLAGS+=$(CFLAGS_CRC)
> >> +pg_crc32c_loongarch_shlib.o: CFLAGS+=$(CFLAGS_CRC)
> >> +pg_crc32c_loongarch_srv.o: CFLAGS+=$(CFLAGS_CRC)
> >>
> >> This was copy-and-pasted from platforms that use a runtime check, so
> >> should be unnecessary.
> >>
> >
> > Removed these lines.
> >
> >> +# If the intrinsics are supported, sets
> >> pgac_loongarch_crc32c_intrinsics,
> >> +# and CFLAGS_CRC.
> >>
> >> +# Check if __builtin_loongarch_crcc_* intrinsics can be used
> >> +# with the default compiler flags.
> >> +# CFLAGS_CRC is set if the extra flag is required.
> >>
> >> Same here -- it seems we don't need to set CFLAGS_CRC at all. Can you
> >> confirm?
> >>
> >
> > We don't need to set CFLAGS_CRC as commented. I have updated the
> > configure script to make it align with the logic in meson build script.
> >
> >>  > > Also, I don't have a Loongarch machine for testing. Could you
> >> show that
> >>  > > the instructions are found in the binary, maybe using objdump and
> >> grep?
> >>  > > Or a performance test?
> >>  > >
> >>  >
> >>  > The output of the objdump command `objdump -dS
> >>  > ../postgres-build/tmp_install/usr/local/pgsql/bin/postgres  | grep
> >> -B 30
> >>  > -A 10 crcc` is attached.
> >>
> >> Thanks for confirming.
> >>
> >> --
> >> John Naylor
> >> EDB: http://www.enterprisedb.com <http://www.enterprisedb.com>
>

本邮件及其附件含有龙芯中科的商业秘密信息，仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制或散发）本邮件及其附件中的信息。如果您错收本邮件，请您立即电话或邮件通知发件人并删除本邮件。
This email and its attachments contain confidential information from Loongson Technology , which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this email in error, please notify the sender by phone or email immediately and delete it.

> On 6 Jul 2023, at 09:14, huchangqi <huchangqi(at)loongson(dot)cn> wrote:
>
> Hi, i have a loongarch machine runing Loongnix-server (based on redhat/centos, it has gcc-8.3 on it), i am trying to test buildfarm on it, when i edit build-farm.conf it seems to need animal and secret to connect the buildfarm server.
> then i go to https://buildfarm.postgresql.org/cgi-bin/register-form.pl, and registered the buildfarm a week ago, but didn't receive any response. so what else i need to do next.

Thanks for volunteering a buildfarm animal! The registration is probably just
pending due to it being summer and things slow down, but I've added Andrew
Dunstan who is the Buildfarm expert on CC:.

--
Daniel Gustafsson

On 2023/7/6 15:14, huchangqi wrote:
> Hi, i have a loongarch machine runing Loongnix-server (based on redhat/centos, it has gcc-8.3 on it), i am trying to test buildfarm on it, when i edit build-farm.conf it seems to need animal and secret to connect the buildfarm server.
> then i go to https://buildfarm.postgresql.org/cgi-bin/register-form.pl, and registered the buildfarm a week ago, but didn't receive any response. so what else i need to do next.
>
>

Is it possible to provide a build farm instance for new world ABI of
loongarch also by loongson? It will be really appreciated.

Thanks!

>
>> -----Original Messages-----
>> From: "YANG Xudong" <yangxudong(at)ymatrix(dot)cn>
>> Send time:Wednesday, 07/05/2023 10:15:51
>> To: "John Naylor" <john(dot)naylor(at)enterprisedb(dot)com>
>> Cc: pgsql-hackers(at)lists(dot)postgresql(dot)org, wengyanqing(at)ymatrix(dot)cn, wanghao(at)ymatrix(dot)cn
>> Subject: Re: [PATCH] Add loongarch native checksum implementation.
>>
>> Is there any other comment?
>>
>> If the patch looks OK, I would like to update its status to ready for
>> committer in the commitfest.
>>
>> Thanks!
>>
>> On 2023/6/16 09:28, YANG Xudong wrote:
>>> Updated the patch based on the comments.
>>>
>>> On 2023/6/15 18:30, John Naylor wrote:
>>>>
>>>> On Wed, Jun 14, 2023 at 9:20 AM YANG Xudong <yangxudong(at)ymatrix(dot)cn
>>>> <mailto:yangxudong(at)ymatrix(dot)cn>> wrote:
>>>>  >
>>>>  > Attached a new patch with fixes based on the comment below.
>>>>
>>>> Note: It's helpful to pass "-v" to git format-patch, to have different
>>>> versions.
>>>>
>>>
>>> Added v2
>>>
>>>>  > > For x86 and Arm, if it fails to link without an -march flag, we
>>>> allow
>>>>  > > for a runtime check. The flags "-march=armv8-a+crc" and
>>>> "-msse4.2" are
>>>>  > > for instructions not found on all platforms. The patch also
>>>> checks both
>>>>  > > ways, and each one results in "Use LoongArch CRC instruction
>>>>  > > unconditionally". The -march flag here is general, not specific. In
>>>>  > > other words, if this only runs inside "+elif host_cpu ==
>>>> 'loongarch64'",
>>>>  > > why do we need both with -march and without?
>>>>  > >
>>>>  >
>>>>  > Removed the elif branch.
>>>>
>>>> Okay, since we've confirmed that no arch flag is necessary, some other
>>>> places can be simplified:
>>>>
>>>> --- a/src/port/Makefile
>>>> +++ b/src/port/Makefile
>>>> @@ -98,6 +98,11 @@ pg_crc32c_armv8.o: CFLAGS+=$(CFLAGS_CRC)
>>>>   pg_crc32c_armv8_shlib.o: CFLAGS+=$(CFLAGS_CRC)
>>>>   pg_crc32c_armv8_srv.o: CFLAGS+=$(CFLAGS_CRC)
>>>>
>>>> +# all versions of pg_crc32c_loongarch.o need CFLAGS_CRC
>>>> +pg_crc32c_loongarch.o: CFLAGS+=$(CFLAGS_CRC)
>>>> +pg_crc32c_loongarch_shlib.o: CFLAGS+=$(CFLAGS_CRC)
>>>> +pg_crc32c_loongarch_srv.o: CFLAGS+=$(CFLAGS_CRC)
>>>>
>>>> This was copy-and-pasted from platforms that use a runtime check, so
>>>> should be unnecessary.
>>>>
>>>
>>> Removed these lines.
>>>
>>>> +# If the intrinsics are supported, sets
>>>> pgac_loongarch_crc32c_intrinsics,
>>>> +# and CFLAGS_CRC.
>>>>
>>>> +# Check if __builtin_loongarch_crcc_* intrinsics can be used
>>>> +# with the default compiler flags.
>>>> +# CFLAGS_CRC is set if the extra flag is required.
>>>>
>>>> Same here -- it seems we don't need to set CFLAGS_CRC at all. Can you
>>>> confirm?
>>>>
>>>
>>> We don't need to set CFLAGS_CRC as commented. I have updated the
>>> configure script to make it align with the logic in meson build script.
>>>
>>>>  > > Also, I don't have a Loongarch machine for testing. Could you
>>>> show that
>>>>  > > the instructions are found in the binary, maybe using objdump and
>>>> grep?
>>>>  > > Or a performance test?
>>>>  > >
>>>>  >
>>>>  > The output of the objdump command `objdump -dS
>>>>  > ../postgres-build/tmp_install/usr/local/pgsql/bin/postgres  | grep
>>>> -B 30
>>>>  > -A 10 crcc` is attached.
>>>>
>>>> Thanks for confirming.
>>>>
>>>> --
>>>> John Naylor
>>>> EDB: http://www.enterprisedb.com <http://www.enterprisedb.com>
>>
>
>
> 本邮件及其附件含有龙芯中科的商业秘密信息，仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制或散发）本邮件及其附件中的信息。如果您错收本邮件，请您立即电话或邮件通知发件人并删除本邮件。
> This email and its attachments contain confidential information from Loongson Technology , which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this email in error, please notify the sender by phone or email immediately and delete it.

Many thanks to huchangqi. Now we have loongarch64 support for both old
world ABI and new world ABI on the buildfarm!

-------- Forwarded Message --------
Subject: Re: [PATCH] Add loongarch native checksum implementation.
Date: Tue, 25 Jul 2023 15:51:43 +0800
From: huchangqi <huchangqi(at)loongson(dot)cn>
To: YANG Xudong <yangxudong(at)ymatrix(dot)cn>

Both cisticola and nuthatch are on the buildfarm now。

cisticola is "old world ABI".
https://buildfarm.postgresql.org/cgi-bin/show_history.pl?nm=cisticola&br=HEAD

nuthatch is "new world ABI".
https://buildfarm.postgresql.org/cgi-bin/show_history.pl?nm=nuthatch&br=HEAD

----------
Best regards,
huchangqi

On 2023/7/5 15:11, John Naylor wrote:
>
> Before I look at this again: Are there any objections to another CRC
> implementation for the reason of having no buildfarm member?

It is possible to try this patch on buildfarm now, I guess?

On Wed, Jul 05, 2023 at 02:11:02PM +0700, John Naylor wrote:
> Also, please don't top-post (which means: quoting an entire message, with
> new text at the top) -- it clutters our archives.
>
> Before I look at this again: Are there any objections to another CRC
> implementation for the reason of having no buildfarm member?

The performance numbers presented upthread for the CRC computations
are kind of nice in this environment, but honestly I have no idea how
much this architecture is used. Perhaps that's only something in
China? I am not seeing much activity around that in Japan, for
instance, and that's really close.

Anyway, based on today's state of the buildfarm, we have a buildfarm
member named cisticola that should be able to test this new CRC
implementation, so I see no problem in applying this stuff now if you
think it is in good shape.
--
Michael

On Wed, Jul 26, 2023 at 12:16:28PM +0900, Michael Paquier wrote:
> On Wed, Jul 05, 2023 at 02:11:02PM +0700, John Naylor wrote:
>> Before I look at this again: Are there any objections to another CRC
>> implementation for the reason of having no buildfarm member?
>
> [ ... ]
>
> Anyway, based on today's state of the buildfarm, we have a buildfarm
> member named cisticola that should be able to test this new CRC
> implementation, so I see no problem in applying this stuff now if you
> think it is in good shape.

IMHO we should strive to maintain buildfarm coverage for all the
instrinsics used within Postgres, if for no other reason than to ensure
future changes do not break those platforms.

--
Nathan Bossart
Amazon Web Services: https://aws.amazon.com

On 2023/7/26 11:16, Michael Paquier wrote> The performance numbers
presented upthread for the CRC computations
> are kind of nice in this environment, but honestly I have no idea how
> much this architecture is used. Perhaps that's only something in
> China? I am not seeing much activity around that in Japan, for
> instance, and that's really close.

The architecture is pretty new (to open source ecosystem). The support
of it in Linux kernel and GCC were released last year.

Here is a site about the status of major open source project support of
it. (Chinese only)

https://areweloongyet.com/

On Wed, Jul 26, 2023 at 8:25 AM YANG Xudong <yangxudong(at)ymatrix(dot)cn> wrote:
>
> Many thanks to huchangqi. Now we have loongarch64 support for both old
> world ABI and new world ABI on the buildfarm!

Glad to hear it!

On Wed, Jul 26, 2023 at 10:16 AM Michael Paquier <michael(at)paquier(dot)xyz>
wrote:
>
> The performance numbers presented upthread for the CRC computations
> are kind of nice in this environment, but honestly I have no idea how
> much this architecture is used. Perhaps that's only something in
> China? I am not seeing much activity around that in Japan, for
> instance, and that's really close.

That was my impression as well. My thinking was, we can give the same
treatment that we gave Arm a number of years ago (which is now quite
mainstream).

> Anyway, based on today's state of the buildfarm, we have a buildfarm
> member named cisticola that should be able to test this new CRC
> implementation, so I see no problem in applying this stuff now if you
> think it is in good shape.

I believe there was just a comment that needed updating, so I'll do that
and push within a few days.

--
John Naylor
EDB: http://www.enterprisedb.com

On Fri, Jun 16, 2023 at 8:28 AM YANG Xudong <yangxudong(at)ymatrix(dot)cn> wrote:
> > +# If the intrinsics are supported, sets
pgac_loongarch_crc32c_intrinsics,
> > +# and CFLAGS_CRC.
> >
> > +# Check if __builtin_loongarch_crcc_* intrinsics can be used
> > +# with the default compiler flags.
> > +# CFLAGS_CRC is set if the extra flag is required.
> >
> > Same here -- it seems we don't need to set CFLAGS_CRC at all. Can you
> > confirm?
> >
>
> We don't need to set CFLAGS_CRC as commented. I have updated the
> configure script to make it align with the logic in meson build script.

(Looking again at v2)

The compilation test is found in c-compiler.m4, which still has all logic
for CFLAGS_CRC, including saving and restoring the old CFLAGS. Can this
also be simplified?

I diff'd pg_crc32c_loongarch.c with the current other files, and found it
is structurally the same as the Arm implementation. That's logical if
memory alignment is important.

/*
- * ARMv8 doesn't require alignment, but aligned memory access is
- * significantly faster. Process leading bytes so that the loop below
- * starts with a pointer aligned to eight bytes.
+ * Aligned memory access is significantly faster.
+ * Process leading bytes so that the loop below starts with a pointer
aligned to eight bytes.

Can you confirm the alignment requirement -- it's not clear what the
intention is since "doesn't require" wasn't carried over. Is there any
documentation (or even a report in some other context) about aligned vs
unaligned memory access performance?

--
John Naylor
EDB: http://www.enterprisedb.com

Thanks for the comment. I have updated the patch to v3. Please have a look.

On 2023/8/7 19:01, John Naylor wrote:
>
> On Fri, Jun 16, 2023 at 8:28 AM YANG Xudong <yangxudong(at)ymatrix(dot)cn
> <mailto:yangxudong(at)ymatrix(dot)cn>> wrote:
> > > +# If the intrinsics are supported, sets
> pgac_loongarch_crc32c_intrinsics,
> > > +# and CFLAGS_CRC.
> > >
> > > +# Check if __builtin_loongarch_crcc_* intrinsics can be used
> > > +# with the default compiler flags.
> > > +# CFLAGS_CRC is set if the extra flag is required.
> > >
> > > Same here -- it seems we don't need to set CFLAGS_CRC at all. Can you
> > > confirm?
> > >
> >
> > We don't need to set CFLAGS_CRC as commented. I have updated the
> > configure script to make it align with the logic in meson build script.
>
> (Looking again at v2)
>
> The compilation test is found in c-compiler.m4, which still has all
> logic for CFLAGS_CRC, including saving and restoring the old CFLAGS. Can
> this also be simplified?

Fixed the function in c-compiler.m4 by removing the function argument
and the logic of handling CFLAGS and CFLAGS_CRC.

>
> I diff'd pg_crc32c_loongarch.c with the current other files, and found
> it is structurally the same as the Arm implementation. That's logical if
> memory alignment is important.
>
>   /*
> - * ARMv8 doesn't require alignment, but aligned memory access is
> - * significantly faster. Process leading bytes so that the loop below
> - * starts with a pointer aligned to eight bytes.
> + * Aligned memory access is significantly faster.
> + * Process leading bytes so that the loop below starts with a pointer
> aligned to eight bytes.
>
> Can you confirm the alignment requirement -- it's not clear what the
> intention is since "doesn't require" wasn't carried over. Is there any
> documentation (or even a report in some other context) about aligned vs
> unaligned memory access performance?

It is in the official document that the alignment is not required.

https://github.com/loongson/la-softdev-convention/blob/master/la-softdev-convention.adoc#74-unaligned-memory-access-support

However, I found this patch in LKML that shows great performance gain
when using aligned memory access similar to this patch.

https://lore.kernel.org/lkml/20230410115734(dot)93365-1-wangrui(at)loongson(dot)cn/

So I guess using aligned memory access is necessary and I have updated
the comment in the code.

>
> --
> John Naylor
> EDB: http://www.enterprisedb.com <http://www.enterprisedb.com>

On Tue, Aug 8, 2023 at 10:07 AM YANG Xudong <yangxudong(at)ymatrix(dot)cn> wrote:

> On 2023/8/7 19:01, John Naylor wrote:

> > The compilation test is found in c-compiler.m4, which still has all
> > logic for CFLAGS_CRC, including saving and restoring the old CFLAGS. Can
> > this also be simplified?
>
> Fixed the function in c-compiler.m4 by removing the function argument
> and the logic of handling CFLAGS and CFLAGS_CRC.

Looks good to me. It seems that platforms capable of running Postgres only
support 64 bit. If that ever changes, the compiler intrinsic test (with 8
byte CRC input) should still gate that well enough in autoconf, I believe,
so in v4 I added a comment to clarify this. The Meson build checks hostcpu
first for all platforms, and the patch is consistent with surrounding code.
In the attached 0002 addendum, I change a comment in configure.ac to
clarify "override" is referring to the runtime check for x86 and Arm, and
that LoongArch doesn't need one.

> > Can you confirm the alignment requirement -- it's not clear what the
> > intention is since "doesn't require" wasn't carried over. Is there any
> > documentation (or even a report in some other context) about aligned vs
> > unaligned memory access performance?
>
> It is in the official document that the alignment is not required.
>
>
https://github.com/loongson/la-softdev-convention/blob/master/la-softdev-convention.adoc#74-unaligned-memory-access-support
>
>
> However, I found this patch in LKML that shows great performance gain
> when using aligned memory access similar to this patch.
>
> https://lore.kernel.org/lkml/20230410115734(dot)93365-1-wangrui(at)loongson(dot)cn/
>
> So I guess using aligned memory access is necessary and I have updated
> the comment in the code.

Okay, so it's not "necessary" in the sense that it's illegal, so I'm
thinking we can just re-use the Arm comment language, as in 0002.

v4 0001 is the same as v3, but with a draft commit message. I will squash
and commit this week, unless there is additional feedback.

--
John Naylor
EDB: http://www.enterprisedb.com

On 2023/8/8 14:38, John Naylor wrote:
>
> It seems that platforms capable of running Postgres
> only support 64 bit.

I think so.

> > So I guess using aligned memory access is necessary and I have updated
> > the comment in the code.
>
> Okay, so it's not "necessary" in the sense that it's illegal, so I'm
> thinking we can just re-use the Arm comment language, as in 0002.

Yes. I think it is similar to Arm.

> v4 0001 is the same as v3, but with a draft commit message. I will
> squash and commit this week, unless there is additional feedback.

Looks good to me. Thanks for the additional patch.

On Tue, Aug 8, 2023 at 2:22 PM YANG Xudong <yangxudong(at)ymatrix(dot)cn> wrote:
>
> On 2023/8/8 14:38, John Naylor wrote:
>
> > v4 0001 is the same as v3, but with a draft commit message. I will
> > squash and commit this week, unless there is additional feedback.
>
> Looks good to me. Thanks for the additional patch.

I pushed this with another small comment change. Unfortunately, I didn't
glance at the buildfarm beforehand -- it seems many members are failing an
isolation check added by commit fa2e87494, including both loongarch64
members. I'll check back periodically.

https://buildfarm.postgresql.org/cgi-bin/show_status.pl

--
John Naylor
EDB: http://www.enterprisedb.com

On Thu, Aug 10, 2023 at 10:35 AM John Naylor
<john(dot)naylor(at)enterprisedb(dot)com> wrote:
>
> On Tue, Aug 8, 2023 at 2:22 PM YANG Xudong <yangxudong(at)ymatrix(dot)cn> wrote:
> >
> > On 2023/8/8 14:38, John Naylor wrote:
> >
> > > v4 0001 is the same as v3, but with a draft commit message. I will
> > > squash and commit this week, unless there is additional feedback.
> >
> > Looks good to me. Thanks for the additional patch.
>
> I pushed this with another small comment change. Unfortunately, I didn't glance at the buildfarm beforehand -- it seems many members are failing an isolation check added by commit fa2e87494, including both loongarch64 members. I'll check back periodically.
>
> https://buildfarm.postgresql.org/cgi-bin/show_status.pl
>

In MSVC build, on doing: perl mkvcbuild.pl after this commit, I am
facing the below error:
Generating configuration headers...
undefined symbol: USE_LOONGARCH_CRC32C at src/include/pg_config.h line
718 at ../postgresql/src/tools/msvc/Mkvcbuild.pm line 872.

Am I missing something or did the commit miss something?

--
With Regards,
Amit Kapila.

On Thu, Aug 10, 2023 at 03:56:37PM +0530, Amit Kapila wrote:
> In MSVC build, on doing: perl mkvcbuild.pl after this commit, I am
> facing the below error:
> Generating configuration headers...
> undefined symbol: USE_LOONGARCH_CRC32C at src/include/pg_config.h line
> 718 at ../postgresql/src/tools/msvc/Mkvcbuild.pm line 872.
>
> Am I missing something or did the commit miss something?

Yes, the commit has missed the addition of USE_LOONGARCH_CRC32C in
Solution.pm. If you want to be consistent with pg_config.h.in, you
could add it just after USE_LLVM, for instance.
--
Michael

On Thu, Aug 10, 2023 at 5:54 PM Michael Paquier <michael(at)paquier(dot)xyz> wrote:
>
> On Thu, Aug 10, 2023 at 03:56:37PM +0530, Amit Kapila wrote:
> > In MSVC build, on doing: perl mkvcbuild.pl after this commit, I am
> > facing the below error:
> > Generating configuration headers...
> > undefined symbol: USE_LOONGARCH_CRC32C at src/include/pg_config.h line
> > 718 at ../postgresql/src/tools/msvc/Mkvcbuild.pm line 872.
> >
> > Am I missing something or did the commit miss something?
>
> Yes, the commit has missed the addition of USE_LOONGARCH_CRC32C in
> Solution.pm. If you want to be consistent with pg_config.h.in, you
> could add it just after USE_LLVM, for instance.

Oops, fixing now...

--
John Naylor
EDB: http://www.enterprisedb.com

On Thu, Aug 10, 2023 at 5:07 PM John Naylor
<john(dot)naylor(at)enterprisedb(dot)com> wrote:
>
> On Thu, Aug 10, 2023 at 5:54 PM Michael Paquier <michael(at)paquier(dot)xyz> wrote:
> >
> > On Thu, Aug 10, 2023 at 03:56:37PM +0530, Amit Kapila wrote:
> > > In MSVC build, on doing: perl mkvcbuild.pl after this commit, I am
> > > facing the below error:
> > > Generating configuration headers...
> > > undefined symbol: USE_LOONGARCH_CRC32C at src/include/pg_config.h line
> > > 718 at ../postgresql/src/tools/msvc/Mkvcbuild.pm line 872.
> > >
> > > Am I missing something or did the commit miss something?
> >
> > Yes, the commit has missed the addition of USE_LOONGARCH_CRC32C in
> > Solution.pm. If you want to be consistent with pg_config.h.in, you
> > could add it just after USE_LLVM, for instance.
>
> Oops, fixing now...
>

It is fixed now. Thanks!

--
With Regards,
Amit Kapila.