Skip to content

crowdstrike.fdr: Increase field limits in FDR data-streams to avoid unindexed ECS fields #13056

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Mar 12, 2025

Conversation

kcreddy
Copy link
Contributor

@kcreddy kcreddy commented Mar 11, 2025

Proposed commit message

Crowdstrike FDR data stream can collect more than 1000 fields
and the data stream explicitly defines more than 500 fields.
In such cases, the total_fields.limit is automatically updated to 
10000[1] from the default 1000. 
Having such dynamic resizing could be detrimental when some of 
these explicitly mapped fields are later removed and the limit goes 
back to 1000. If it goes back to 1000, then it could end up not 
indexing all fields, even the ECS fields.

This PR sets the field limit to 2000 to facilitate 
indexing of all fields, especially ECS.

Ref: 
- [1] https://www.elastic.co/guide/en/integrations-developer/current/general-guidelines.html#_field_limits

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Current (1.54.0) version:
GET .ds-logs-crowdstrike.fdr-default-2025.03.11-000001/_settings

{
  ".ds-logs-crowdstrike.fdr-default-2025.03.11-000001": {
    "settings": {
      "index": {
        "mapping": {
          "total_fields": {
            "limit": "10000",
            "ignore_dynamic_beyond_limit": "true"
          },
          "ignore_malformed": "true"
        },

With new PR change
GET .ds-logs-crowdstrike.fdr-default-2025.03.11-000001/_settings

{
  ".ds-logs-crowdstrike.fdr-default-2025.03.11-000001": {
    "settings": {
      "index": {
        "mapping": {
          "total_fields": {
            "limit": "2000",
            "ignore_dynamic_beyond_limit": "true"
          },
          "ignore_malformed": "true"
        },

Related issues

Screenshots

@elastic-vault-github-plugin-prod

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

@elasticmachine
Copy link

💚 Build Succeeded

Copy link

@kcreddy kcreddy self-assigned this Mar 11, 2025
@kcreddy kcreddy added enhancement New feature or request Integration:crowdstrike CrowdStrike Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Mar 11, 2025
@kcreddy kcreddy marked this pull request as ready for review March 11, 2025 14:27
@kcreddy kcreddy requested a review from a team as a code owner March 11, 2025 14:27
@elasticmachine
Copy link

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

Copy link
Contributor

@efd6 efd6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Behaviour change confirmed.

@kcreddy kcreddy merged commit f1747ab into elastic:main Mar 12, 2025
7 checks passed
@elastic-vault-github-plugin-prod

Package crowdstrike - 1.55.0 containing this change is available at https://epr.elastic.co/package/crowdstrike/1.55.0/

flexitrev pushed a commit that referenced this pull request Mar 20, 2025
…nindexed ECS fields (#13056)

Crowdstrike FDR data stream can collect more than 1000 fields
and the data stream explicitly defines more than 500 fields.
In such cases, the total_fields.limit is automatically updated to 
10000[1] from the default 1000. 
Having such dynamic resizing could be detrimental when some of 
these explicitly mapped fields are later removed and the limit goes 
back to 1000. If it goes back to 1000, then it could end up not 
indexing all fields, even the ECS fields.

This PR sets the field limit to 2000 to facilitate 
indexing of all fields, especially ECS.

Ref: 
- [1] https://www.elastic.co/guide/en/integrations-developer/current/general-guidelines.html#_field_limits
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request Integration:crowdstrike CrowdStrike Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants