We've reorganized our navigation structure to align directly with your operational workflows. See the Google SecOps release notes for more information.

Collect Amazon VPC Transit Gateway Flow Logs

Supported in:

Google secops SIEM

This document explains how to ingest Amazon VPC Transit Gateway Flow Logs into Google Security Operations using Amazon S3 or Amazon Kinesis Data Firehose. Transit Gateway Flow Logs capture detailed network traffic metadata (source and destination IPs, ports, protocols, byte and packet counts, actions) across your Transit Gateway attachments. This data enables security monitoring, anomaly detection, and compliance auditing within Google SecOps.

Before you begin

Make sure you have the following prerequisites:

A Google SecOps instance.
Privileged access to AWS with permissions to:
- Create and manage Transit Gateway flow logs
- Create S3 buckets (for S3 export option)
- Create IAM users, policies, and roles
- Configure Kinesis Data Firehose delivery streams (for Firehose option)
- Access CloudWatch Logs and create subscription filters (for Firehose option)

Option - Export to Amazon S3

Enable Transit Gateway Flow Logs (to S3)

Open the Amazon VPC console at https://console.aws.amazon.com/vpc/.
In the navigation pane, choose Transit gateways.
Select the checkboxes for one or more transit gateways and choose Actions > Create flow log.
For Destination, select Send to an Amazon S3 bucket.
For S3 bucket ARN, enter the ARN of an existing S3 bucket (for example, arn:aws:s3:::tgw-flowlogs-bucket).

Note: You can include a subfolder path in the ARN (for example, arn:aws:s3:::tgw-flowlogs-bucket/flow-logs/).
For Log record format, select the format for the flow log record:
- AWS default format - includes all version 2 through version 6 fields.
- Custom format - select only the fields you need.
For Log file format, select Text (default) or Parquet.
For Hive-compatible S3 prefix, optionally select Enable to use Hive-compatible prefixes for partitioning.
For Partition logs by time, select Every 1 hour (60 min) or Every 24 hours (1440 min).
(Optional) Choose Add new tag to apply tags to the flow log.
Click Create flow log.

Create an Amazon S3 bucket

Open the Amazon S3 console.
Click Create bucket.
Provide the following configuration details:
- Bucket name: Enter a unique name (for example, tgw-flowlogs-bucket)
- AWS Region: Select the same Region where your Transit Gateway resides
Click Create bucket.

Create an IAM user with access to Amazon S3

Open the IAM console.
Click Users > Add user.
Enter a user name (for example, chronicle-tgw-s3-reader).
Select Programmatic access.
Click Next: Permissions.
Choose Attach existing policies directly.
Search for and select the AmazonS3ReadOnlyAccess policy.
Click Next: Tags > Next: Review > Create user.
Copy the Access Key ID and Secret Access Key for the Google SecOps feed configuration.

Configure a feed in Google SecOps to ingest Amazon VPC Transit Gateway Flow Logs (S3)

Go to SIEM Settings > Feeds.
Click Add New Feed.
In the Feed name field, enter a name for the feed (for example, AWS Transit Gateway Flow Logs - S3).
Select Amazon S3 as the Source type.
Select Amazon VPC Transit Gateway Flow Logs as the Log type.
Click Next.
Specify values for the following input parameters:
- S3 URI: The bucket URI (for example, s3://tgw-flowlogs-bucket/flow-logs/)
Note: Include the trailing slash (/) at the end of the S3 URI.
- Source deletion option: Select the deletion option according to your preference:
  - Never: Never deletes any files after transfers.
  - Delete transferred files: Deletes files after successful transfer.
  - Delete transferred files and empty directories: Deletes files and empty directories after successful transfer.
- Maximum File Age: Include files modified in the last number of days (default is 180 days)
- Access Key ID: The IAM user access key with access to the S3 bucket
- Secret Access Key: The IAM user secret key with access to the S3 bucket
- Asset namespace: The asset namespace
- Ingestion labels: The label to be applied to the events from this feed
Click Next.
Review the feed configuration and click Submit.

Option - Stream via Amazon Kinesis Data Firehose

Enable Transit Gateway Flow Logs (to CloudWatch Logs)

Open the Amazon VPC console at https://console.aws.amazon.com/vpc/.
In the navigation pane, choose Transit gateways.
Select the checkboxes for one or more transit gateways and choose Actions > Create flow log.
For Destination, select Send to CloudWatch Logs.
For Destination log group, choose an existing log group or enter a new name to create one (for example, /aws/tgw/flowlogs).

Note: If the destination log group does not yet exist, entering a new name creates it automatically.
For IAM role, specify a role that has permissions to publish logs to CloudWatch Logs.
For Log record format, select the format for the flow log record:
- AWS default format - includes all version 2 through version 6 fields.
- Custom format - select only the fields you need.
(Optional) Choose Add new tag to apply tags to the flow log.
Click Create flow log.

Create a feed in Google SecOps to ingest Amazon VPC Transit Gateway Flow Logs (Firehose)

Go to SIEM Settings > Feeds.
Click Add New Feed.
In the Feed name field, enter a name for the feed (for example, AWS Transit Gateway Flow Logs - Firehose).
Select Amazon Data Firehose as the Source type.
Select Amazon VPC Transit Gateway Flow Logs as the Log type.
Click Next.
Specify values for the following input parameters:
- Split delimiter: Enter \n
- Asset namespace: The asset namespace
- Ingestion labels: The label to be applied to the events from this feed
Click Next.
Review the feed configuration and click Submit.
Click Generate Secret Key to generate a secret key to authenticate this feed.
Copy and save the secret key. You cannot view this secret again.
Go to the Details tab.
Copy the feed endpoint URL from the Endpoint Information field.
Click Done.

Create an API key for the Amazon Data Firehose feed

Go to the Google Cloud console Credentials page at https://console.cloud.google.com/apis/credentials.
Click Create credentials, and then select API key.
Click Edit API key to restrict the key.
Under API restrictions, select Restrict key.
Search for and select Google SecOps API.
Click Save.
Copy and save the API key.

Construct the endpoint URL

Append the API key to the feed endpoint URL in the following format:
```
<FEED_ENDPOINT_URL>?key=<API_KEY>
```

Replace the following:

<FEED_ENDPOINT_URL>: The feed endpoint URL
<API_KEY>: The API key

Example:

https://malachiteingestion-pa.googleapis.com/v2/unstructuredlogentries:batchCreate?key=AIzaSyD...

Save this full URL for the next step.

Create IAM policy for Firehose

In the AWS Console, go to IAM > Policies > Create policy > JSON tab.

Paste the following policy JSON:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "firehose:PutRecord",
                "firehose:PutRecordBatch"
            ],
            "Resource": "arn:aws:firehose:<REGION>:<ACCOUNT_ID>:deliverystream/<DELIVERY_STREAM_NAME>"
        }
    ]
}

Replace the following:
- <REGION>: Your AWS Region (for example, us-east-1)
- <ACCOUNT_ID>: Your AWS account ID (12-digit number)
- <DELIVERY_STREAM_NAME>: Your Firehose delivery stream name (you will create this in the next step)
Name the policy TGWFlowLogsToFirehosePolicy (for example) and click Create policy.

Create IAM role for CloudWatch Logs

Go to IAM > Roles > Create role.

Select Custom trust policy and paste:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "logs.<REGION>.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Replace <REGION> with your AWS Region.
Click Next.
Search for and select the policy TGWFlowLogsToFirehosePolicy.
Click Next.
Name the role TGWFlowLogsToFirehoseRole (for example) and click Create role.

Create Kinesis Data Firehose delivery stream

In the AWS Console, go to Kinesis > Data Firehose > Create delivery stream.
Provide the following configuration details:
- Source and destination:
  - Source: Select Direct PUT or other sources
  - Destination: Select HTTP endpoint
- Delivery stream name:
  - Delivery stream name: Enter a name (for example, tgw-flowlogs-to-secops)
- HTTP endpoint destination:
  - HTTP endpoint URL: Enter the full endpoint URL you constructed (feed endpoint + API key)
  - Content encoding: Select GZIP
- Custom HTTP headers:
  - Click Add custom HTTP header
  - Header name: Enter X-Goog-Chronicle-Auth
  - Header value: Enter the secret key you saved from the feed creation step
- Backup settings:
  - Source record backup in Amazon S3: Select Failed data only
  - S3 bucket: Select an existing bucket or create a new one for failed records
- Buffer hints:
  - Buffer size: Enter 1 MiB
  - Buffer interval: Enter 60 seconds
Click Create delivery stream.
Wait for the delivery stream status to change to Active.

Subscribe CloudWatch Log Group to Firehose

In the AWS Console, go to CloudWatch > Logs > Log groups.
Select the Transit Gateway flow log group (for example, /aws/tgw/flowlogs).
Click the Subscription filters tab.
Click Create > Create Amazon Kinesis Data Firehose subscription filter.
Provide the following configuration details:
- Destination: Select delivery stream tgw-flowlogs-to-secops
- Grant permission: Select role TGWFlowLogsToFirehoseRole
- Subscription filter name: Enter a name (for example, tgw-flowlogs-to-chronicle)
- Log format: Select Other
- Subscription filter pattern: Leave empty to send all events
Click Start streaming.

UDM mapping table

Log Field	UDM Mapping	Logic
`packets_lost_blackhole_label`	`additional.fields`	Merged
`packets_lost_mtu_exceeded_label`	`additional.fields`	Merged
`packets_lost_no_route_label`	`additional.fields`	Merged
`packets_lost_ttl_expired_label`	`additional.fields`	Merged
`start_time_label`	`additional.fields`	Merged
`tcp_flags_label`	`additional.fields`	Merged
`tgw_attachment_id_label`	`additional.fields`	Merged
`tgw_id_label`	`additional.fields`	Merged
`tgw_pair_attachment_id_label`	`additional.fields`	Merged
`type_label`	`additional.fields`	Merged
`end_time`	`metadata.event_timestamp`	Parsed as `UNIX`
`event_type`	`metadata.event_type`	Directly mapped
`account_id`	`metadata.product_log_id`	Directly mapped
`version`	`metadata.product_version`	Directly mapped
`flow_direction`	`network.direction`	Mapped: `(?i)ingress` → `INBOUND`, `(?i)egress` → `OUTBOUND`
`bytes`	`network.sent_bytes`	Directly mapped
`packets`	`network.sent_packets`	Directly mapped
`tgw_src_eni`	`principal.asset.asset_id`	Directly mapped
`srcaddr`	`principal.asset.ip`	Merged
`srcaddr`	`principal.ip`	Merged
`region`	`principal.location.country_or_region`	Directly mapped
`tgw_src_az_id`	`principal.location.country_or_region`	Directly mapped
`srcport`	`principal.port`	Directly mapped
`pkt_src_aws_service_label`	`principal.resource.attribute.labels`	Merged
`tgw_src_az_id_label`	`principal.resource.attribute.labels`	Merged
`tgw_src_subnet_id`	`principal.resource.name`	Directly mapped
`tgw_src_vpc_id`	`principal.resource.product_object_id`	Directly mapped
`resource_type`	`principal.resource.resource_subtype`	Directly mapped
`tgw_src_vpc_account_id`	`principal.user.userid`	Directly mapped
`security_result_action`	`security_result.action`	Merged
`protocol_label`	`security_result.detection_fields`	Merged
`tgw_dst_eni`	`target.asset.asset_id`	Directly mapped
`dstaddr`	`target.asset.ip`	Merged
`dstaddr`	`target.ip`	Merged
`tgw_dst_az_id`	`target.location.country_or_region`	Directly mapped
`dstport`	`target.port`	Directly mapped
`pkt_dst_aws_service_label`	`target.resource.attribute.labels`	Merged
`tgw_dst_subnet_id`	`target.resource.name`	Directly mapped
`tgw_dst_vpc_id`	`target.resource.product_object_id`	Directly mapped
`tgw_dst_vpc_account_id`	`target.user.userid`	Directly mapped
N/A	`metadata.product_name`	Constant: `AWS VPC Transit Gateway`
N/A	`metadata.vendor_name`	Constant: `AWS`
N/A	`network.direction`	Constant: `INBOUND`

Change Log

View the Change Log for this parser

Need more help? Get answers from Community members and Google SecOps professionals.