[ML] Improve performance of closing files before spawning #2424
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Before spawning new processes from the `controller` we close all open file descriptors except for stdin, stdout and stderr. Previously this was done by checking every possible file descriptor to see if it was open, but this is very expensive if the file descriptor limit is high. During a lookback a large number of `normalize` processes get started, and this could lead to significant CPU usage by the `controller` process. This change makes the file closure code learn the highest open file descriptor each time it is used, and work on the basis that no more than 10 files will be opened in between calls to it. This significantly reduces `controller` CPU usage on machines that have high file descriptor limits and run a lot of `normalize` processes.
droberts195
added a commit
to droberts195/ml-cpp
that referenced
this pull request
Dec 5, 2022
Before spawning new processes from the `controller` we close all open file descriptors except for stdin, stdout and stderr. Previously this was done by checking every possible file descriptor to see if it was open, but this is very expensive if the file descriptor limit is high. During a lookback a large number of `normalize` processes get started, and this could lead to significant CPU usage by the `controller` process. This change makes the file closure code learn the highest open file descriptor each time it is used, and work on the basis that no more than 10 files will be opened in between calls to it. This significantly reduces `controller` CPU usage on machines that have high file descriptor limits and run a lot of `normalize` processes. Backport of elastic#2424
droberts195
added a commit
to droberts195/ml-cpp
that referenced
this pull request
Dec 5, 2022
Before spawning new processes from the `controller` we close all open file descriptors except for stdin, stdout and stderr. Previously this was done by checking every possible file descriptor to see if it was open, but this is very expensive if the file descriptor limit is high. During a lookback a large number of `normalize` processes get started, and this could lead to significant CPU usage by the `controller` process. This change makes the file closure code learn the highest open file descriptor each time it is used, and work on the basis that no more than 10 files will be opened in between calls to it. This significantly reduces `controller` CPU usage on machines that have high file descriptor limits and run a lot of `normalize` processes. Backport of elastic#2424
droberts195
added a commit
that referenced
this pull request
Dec 5, 2022
Before spawning new processes from the `controller` we close all open file descriptors except for stdin, stdout and stderr. Previously this was done by checking every possible file descriptor to see if it was open, but this is very expensive if the file descriptor limit is high. During a lookback a large number of `normalize` processes get started, and this could lead to significant CPU usage by the `controller` process. This change makes the file closure code learn the highest open file descriptor each time it is used, and work on the basis that no more than 10 files will be opened in between calls to it. This significantly reduces `controller` CPU usage on machines that have high file descriptor limits and run a lot of `normalize` processes. Backport of #2424
droberts195
added a commit
that referenced
this pull request
Dec 8, 2022
Before spawning new processes from the `controller` we close all open file descriptors except for stdin, stdout and stderr. Previously this was done by checking every possible file descriptor to see if it was open, but this is very expensive if the file descriptor limit is high. During a lookback a large number of `normalize` processes get started, and this could lead to significant CPU usage by the `controller` process. This change makes the file closure code learn the highest open file descriptor each time it is used, and work on the basis that no more than 10 files will be opened in between calls to it. This significantly reduces `controller` CPU usage on machines that have high file descriptor limits and run a lot of `normalize` processes. Backport of #2424
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Before spawning new processes from the
controllerwe close all open file descriptors except for stdin, stdout and stderr.Previously this was done by checking every possible file descriptor to see if it was open, but this is very expensive if the file descriptor limit is high. During a lookback a large number of
normalizeprocesses get started, and this could lead to significant CPU usage by thecontrollerprocess.This change makes the file closure code learn the highest open file descriptor each time it is used, and work on the basis that no more than 10 files will be opened in between calls to it. This significantly reduces
controllerCPU usage on machines that have high file descriptor limits and run a lot ofnormalizeprocesses.