[Azure Logs] Add a missing pattern for AzureFirewallNetworkRule

[zmoog]

What does this PR do?

Add a new pattern to the grok processor to better support the AzureFirewallNetworkRule in the Azure Firewall logs integration.

UPDATE: the log sample with the unsupported message format was kindly provided by a user through a support request.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

$ elastic-package test pipeline -v
2023/02/24 18:24:09 DEBUG Enable verbose logging
Run pipeline tests for the package
--- Test results for package: azure - START ---
╭─────────┬─────────────────────┬───────────┬──────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM         │ TEST TYPE │ TEST NAME                                    │ RESULT │ TIME ELAPSED │
├─────────┼─────────────────────┼───────────┼──────────────────────────────────────────────┼────────┼──────────────┤
│ azure   │ activitylogs        │ pipeline  │ test-activitylogs-edgecases.log              │ PASS   │   7.783584ms │
│ azure   │ activitylogs        │ pipeline  │ test-activitylogs-identity.log               │ PASS   │   3.462458ms │
│ azure   │ activitylogs        │ pipeline  │ test-activitylogs-raw.log                    │ PASS   │   2.379167ms │
│ azure   │ application_gateway │ pipeline  │ test-application-gateway-raw.log             │ PASS   │   2.918042ms │
│ azure   │ auditlogs           │ pipeline  │ test-audit-logs-edgecases.log                │ PASS   │   2.879084ms │
│ azure   │ auditlogs           │ pipeline  │ test-audit-logs-sample.log                   │ PASS   │       4.05ms │
│ azure   │ auditlogs           │ pipeline  │ test-auditlogs-raw.log                       │ PASS   │   2.088667ms │
│ azure   │ eventhub            │ pipeline  │ test-eventhub-raw.log                        │ PASS   │   6.043167ms │
│ azure   │ firewall_logs       │ pipeline  │ test-applicationrules-raw.log                │ PASS   │    4.46675ms │
│ azure   │ firewall_logs       │ pipeline  │ test-dnsproxyrules-raw.log                   │ PASS   │   3.203083ms │
│ azure   │ firewall_logs       │ pipeline  │ test-networkrules-raw.log                    │ PASS   │   4.660541ms │
│ azure   │ firewall_logs       │ pipeline  │ test-sdh3075-raw.log                         │ PASS   │   2.771666ms │
│ azure   │ identity_protection │ pipeline  │ test-rickyusers-raw.log                      │ PASS   │   1.857708ms │
│ azure   │ identity_protection │ pipeline  │ test-userriskevents-raw.log                  │ PASS   │   2.080458ms │
│ azure   │ platformlogs        │ pipeline  │ test-platformlogs-edgecases.log              │ PASS   │   2.438166ms │
│ azure   │ platformlogs        │ pipeline  │ test-platformlogs-identity-raw.log           │ PASS   │   1.899667ms │
│ azure   │ platformlogs        │ pipeline  │ test-platformlogs-invalid-raw.log            │ PASS   │   2.592041ms │
│ azure   │ platformlogs        │ pipeline  │ test-platformlogs-kube.log                   │ PASS   │   2.055083ms │
│ azure   │ platformlogs        │ pipeline  │ test-platformlogs-raw.log                    │ PASS   │   1.964459ms │
│ azure   │ platformlogs        │ pipeline  │ test-platformlogs-remote-raw.log             │ PASS   │   1.945416ms │
│ azure   │ provisioning        │ pipeline  │ test-provisioninglogs-raw.log                │ PASS   │   2.383958ms │
│ azure   │ signinlogs          │ pipeline  │ test-managed-identity-sample.log             │ PASS   │  35.266958ms │
│ azure   │ signinlogs          │ pipeline  │ test-managed-identity.log                    │ PASS   │   2.970292ms │
│ azure   │ signinlogs          │ pipeline  │ test-non-interactive-user-sample.log         │ PASS   │  10.061875ms │
│ azure   │ signinlogs          │ pipeline  │ test-non-interactive-user-signin.log         │ PASS   │      3.539ms │
│ azure   │ signinlogs          │ pipeline  │ test-non-interactive-user.log                │ PASS   │   3.488542ms │
│ azure   │ signinlogs          │ pipeline  │ test-service-principal-signinlogs-sample.log │ PASS   │  12.283958ms │
│ azure   │ signinlogs          │ pipeline  │ test-service-principal.log                   │ PASS   │   2.783292ms │
│ azure   │ signinlogs          │ pipeline  │ test-signinlogs-raw.log                      │ PASS   │   2.709583ms │
│ azure   │ signinlogs          │ pipeline  │ test-signinlogs-sample.log                   │ PASS   │   2.958875ms │
│ azure   │ springcloudlogs     │ pipeline  │ test-springcloudlogs-edgecases.log           │ PASS   │   2.487958ms │
│ azure   │ springcloudlogs     │ pipeline  │ test-springcloudlogs-raw.log                 │ PASS   │   2.125166ms │
╰─────────┴─────────────────────┴───────────┴──────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: azure - END   ---
Done

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-02-24T23:04:36.262+0000

• Duration: 13 min 19 sec

Test stats 🧪

Test	Results
Failed	0
Passed	125
Skipped	0
Total	125

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (10/10)	💚
Files	86.364% (19/22)	👎 -13.636
Classes	86.364% (19/22)	👎 -13.636
Methods	83.333% (155/186)	👎 -6.958
Lines	84.935% (2802/3299)	👎 -11.065
Conditionals	100.0% (0/0)	💚

[aspacca]

@zmoog , just curious, how were we made aware about the fact that this pattern exists as well and we need to suppor it?

I'm wondering if we can exploit further the origin of such knowledge to cover even more patterns :)

[aspacca]

how were we made aware about the fact that this pattern exists as well and we need to suppor it?

answering myself after an hint from the test filename: https://github.com/elastic/sdh-beats/issues/3075

[zmoog]

@aspacca, exactly, the user kindly shared a sample log with the unsupported format.

I tried the canonical resources like:

• https://learn.microsoft.com/en-us/azure/firewall/logs-and-metrics#diagnostic-logs
• https://learn.microsoft.com/en-us/azure/azure-monitor/reference/

But I need help finding a complete list of existing formats.

I am now asking previous authors/maintainers, the Azure community, and people in Azure about existing formats and how to stay updated for future changes.

Thank you for taking the time to review the PR! 🙇

[elasticmachine]

Package azure - 1.5.11 containing this change is available at https://epr.elastic.co/search?package=azure

[aspacca]

I just got the doubt if patter will remain from %external address% to %internal address% and Rule will change to AllowInternetIn, in case of inbound connections, or the from/to will be flipped

we don't set network.direction (https://www.elastic.co/guide/en/ecs/current/ecs-network.html#field-network-direction) and if we want to do according to the above we might have different ways to do it

[zmoog]

Good point! I understood this is a source and destination pattern without implied internal or external concepts, but I may be wrong.

I'm unsure, so let me bring @ebeahan into this conversation. Eric created this integration, and there's a good chance he has the Azure Firewall domain knowledge to sort this out. 😇

[ebeahan]

Even if the NAT traffic was coming out-to-in instead of in-to-out, the mapping should still be valid since the fields would still accurately represent the source and destination as logged by the firewall.

Some security products have a concept, like HOME_NET, to set what's considered inside vs. outside a firewall's perimeter and from that determine if traffic is inbound/outbound/etc. AFAIK Azure Firewall doesn't have that type of configuration.