aws(学习笔记第四十课) iot-msk-pipeline

aws(学习笔记第四十课) iot-msk-pipeline

• 使用 iot和msk

学习内容：

• 使用 iot和msk

1. 整体架构

1.1 代码链接

• 代码链接(iot-msk-lambda-pipeline)

1.2 代码调整

• 建立cluster的时候，kafka_version=msk.KafkaVersion.V3_4_0,，需要指定高版本

   # MSK cluster with SASL/SCRAM authentication
      self.cluster = msk.Cluster(self, "Cluster",
          cluster_name="iotCluster",
          kafka_version=msk.KafkaVersion.V3_4_0,
          vpc=vpc,
          encryption_in_transit=msk.EncryptionInTransitConfig(
              client_broker=msk.ClientBrokerEncryption.TLS
          ),
          client_authentication=msk.ClientAuthentication.sasl(
              scram=True
          ),
      )
  

• 生成scram secret

  # 1. 创建Secrets Manager密码（在CDK中）
          scram_secret = secretsmanager.Secret(
              self, "MSKScramSecret",
              secret_name="AmazonMSK_iotCluster_demo",
              generate_secret_string=secretsmanager.SecretStringGenerator(
                  secret_string_template=json.dumps({
       "username": "admin"}),
                  generate_string_key="password",
                  exclude_characters='"@/\\',  # 避免特殊字符冲突
                  password_length=16
              )
          )
  

• 将生成scram secret指定到MSK Cluster，client_authentication的secret设定

  # MSK cluster with SASL/SCRAM authentication
          self.cluster = msk.Cluster(self, "Cluster",
              cluster_name="iotCluster",
              kafka_version=msk.KafkaVersion.V3_4_0,
              vpc=vpc,
              encryption_in_transit=msk.EncryptionInTransitConfig(
                  client_broker=msk.ClientBrokerEncryption.TLS
              ),
              client_authentication=msk.ClientAuthentication.sasl(
                  scram=True,
                  secret=scram_secret
              ),
  

• 指定client的kafka version的时候，需要指定高版本

  constants = {
       
      "KAFKA_DOWNLOAD_VERSION": "kafka_2.13-4.0.0",
      "KAFKA_VERSION": "4.0.0",
      "KAFKA_CLIENT_INSTANCE": "t2.xlarge",
      "MSK_TOPIC": "topic",
      "IOT_TOPIC": 'iot/topic'
  }
  

• 伴随升版，requirement.txt也需要改修

  aws-cdk-lib==2.101.0
  constructs>=10.0.0,<11.0.0
  aws-cdk.aws-msk-alpha==2.101.0a0
  boto3>=1.28.0
  

• 新版本Kafka（2.2+）已弃用–zookeeper参数，改用–bootstrap-server
  instance.user_data.add_commands这里需要改成如下，需要的client-ssl.properties在下面作成。

  ./bin/kafka-topics.sh \
    --create \
    --bootstrap-server b-1.iotcluster.g9od7s.c4.kafka.ap-northeast-1.amazonaws.com:9096 \
    --topic topic \
    --partitions 3 \
    --replication-factor 2 \
    --command-config client-ssl.properties
  

• export KAFKA_HEAP_OPTS=“-Xmx1G -Xms512M”
• 创建Java Keystore (JKS) 格式的truststore
  其中，AmazonRootCA1.pem文件需要执行
  wget https://www.amazontrust.com/repository/AmazonRootCA1.pem进行获得。

  keytool -importcert \
    -alias AmazonMSKCA \
    -file AmazonRootCA1.pem \
    -keystore kafka.client.truststore.jks \
    -storepass 123456 \
    -noprompt
  

• 生成client-ssl.properties

    cat << EOF > client-ssl.properties
    bootstrap.servers=b-1.iotcluster.g9od7s.c4.kafka.ap-northeast-1.amazonaws.com:9096
    security.protocol=SSL
    ssl.truststore.lo