[ruby/openssl] Fix test_create_with_mac_iter accidently setting keytype not maciter

This test was accidentally passing the value 2048 into the keytype
parameter of PKCS12_create, not the mac_iter parameter (because it had
one too many `nil`s in the call). This value is invalid, and will make
OpenSSL perform an out-of-bounds read which is caught when compiling
with ASAN.

This commit fixes the tests, and also adds some validation to
PKCS12.create to make sure any keytype passed is actually valid. Since
there only two valid keytype constants, and the whole feature is an
export-grade crypto era thing only ever supported by old MSIE, it seems
far more likely that code in the whild is using keytype similarly by
mistake rather than as intended. So this validation might catch that.

https://github.com/ruby/openssl/commit/47028686d2

1 files changed, 30 insertions, 1 deletions

diff --git a/test/openssl/test_pkcs12.rb b/test/openssl/test_pkcs12.rb
index e6b91b52af..faf26c9e3e 100644
--- a/test/openssl/test_pkcs12.rb
+++ b/test/openssl/test_pkcs12.rb
@@ -159,7 +159,6 @@ module OpenSSL
         DEFAULT_PBE_PKEYS,
         DEFAULT_PBE_CERTS,
         nil,
-        nil,
         2048
       )
 
@@ -178,6 +177,36 @@ module OpenSSL
       end
     end
 
+    def test_create_with_keytype
+      OpenSSL::PKCS12.create(
+        "omg",
+        "hello",
+        @mykey,
+        @mycert,
+        [],
+        DEFAULT_PBE_PKEYS,
+        DEFAULT_PBE_CERTS,
+        nil,
+        nil,
+        OpenSSL::PKCS12::KEY_SIG
+      )
+
+      assert_raise(ArgumentError) do
+        OpenSSL::PKCS12.create(
+          "omg",
+          "hello",
+          @mykey,
+          @mycert,
+          [],
+          DEFAULT_PBE_PKEYS,
+          DEFAULT_PBE_CERTS,
+          nil,
+          nil,
+          2048
+        )
+      end
+    end
+
     def test_new_with_no_keys
       # generated with:
       #   openssl pkcs12 -certpbe PBE-SHA1-3DES -in <@mycert> -nokeys -export