network_traffic using invalid ECS values

[jsoriano]

[0] parsing field value failed: field "event.category"'s value "network_traffic" is not one of the allowed values (authentication, configuration, database, driver, email, file, host, iam, intrusion_detection, malware, network, package, process, registry, session, threat, web)

Part of #3016.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[efd6]

I will do this after #3157 is merged.

[efd6]

This was fixed by #3343.

[jsoriano]

@efd6 I have updated the PR testing this and I see that there are still several failures in network_traffic, could you please take another look?
https://beats-ci.elastic.co/blue/organizations/jenkins/Ingest-manager%2Fintegrations/detail/PR-3017/6/tests

[efd6]

It looks like these come from the beats, though it's difficult to see where since it's not coming from packetbeat and the report doesn't give any context because they are system tests. I'll send a fix to auditbeat and filebeat that are writing that value in, but will also try to replay the equivalent of https://github.com/elastic/integrations/pull/3343/files#diff-62eb4ec7064da37d476f9421d1bc3c862d8bf2f584b16a22de5b74a1f36b6f55 on each.

The sample events don't show this value, so I don't know what is going on there. This is explained in the next comment.

[efd6]

This does not fail locally (8.2.1-SNAPSHOT). The 'network_traffic' term was removed from packetbeat in elastic/beats#20556 which was not backported to 7.x, so presumably the tests that jenkins are running are on a version prior to that — I have confirmed that the tests fail on 7.17.0.

@andrewkroh What is your position on the fix for this since the removal was not backported to 7.x and the change is marked as breaking?

[jsoriano]

@efd6 an option for the integration side would be to add a processor in the ingest pipeline to fix the wrong values from old versions of Beats. This "compatibility layer" wouldn't require any change in Beats.