SysTrack builds the Health Score in stages, starting with 15-second checks on each agent and ending with a 30-day average. This article describes each stage. For the Health Score formula itself, see Understanding the Health Score.
15-Second Checks
The agent checks most health metrics every 15 seconds, using the performance counter value from that interval, and compares it directly against the configured threshold for that metric.
EXAMPLE: To affect the Health Score, the average CPU usage for a 15-second interval must be greater than 90%. If the CPU usage spikes to 90% for 1 second, but stays low at 1% for 14 seconds, then the average CPU usage would only be 7%. In this example, there is no Health Score impact.
Windows events, such as application errors, do not use this 15-second cycle. SysTrack counts an event as an impact when an alarm was triggered on the event and the level of the event is Error, Warning, or Critical.
The One-Minute Rule
The agent groups the four 15-second checks for each metric into one-minute intervals. Any single 15-second sample over the threshold marks the whole minute as impacted for that category. SysTrack does not require all four samples to fail.
SysTrack counts each impacted minute for the system and for each health category that was over its threshold in that minute.
For the system, an impacted minute counts once, even when more than one category was over its threshold in that minute. These minutes make up the total impacted minutes that the Health Score uses.
For each health category, SysTrack tracks two values:
Impacted minutes are the minutes in which the category was over its threshold. A minute shared by two categories adds 1 minute to each of them. As a result, the impacted minutes of all categories can add up to more than the total for the system.
Impact cost is the share of each impacted minute that SysTrack assigns to the category. The impact costs of all categories add up to the total for the system.
When more than one category is over its threshold in the same minute, SysTrack splits that minute evenly among them:
Category impact cost = 1 / Number of active categories
EXAMPLE: In one 10-minute bucket, CPU affected 2 active minutes and Latency affected 1 active minute, with 1 minute of overlap. The total for the system is 2 impacted minutes, because the overlapping minute counts once. The impacted minutes by category are 2 for CPU and 1 for Latency, which add up to 3. The overlapping minute splits evenly, 0.5 to each category, so the impact cost is 1 + 0.5 = 1.5 for CPU and 0.5 for Latency. The impact costs add up to 2, which matches the total.
Faults, blue screens, and kernel panics work differently:
A fault, from an Application Error event, sets a 3-minute countdown. The current minute and the next 2 minutes each count as impacted.
A blue screen on Windows, or a kernel panic on macOS and Linux, adds a 10-minute impact.
10-Minute Buckets
The agent adds up impacted minutes in 10-minute buckets. Each minute adds at most 1 impacted minute, so a bucket holds at most 10 impacted minutes.
EXAMPLE: In one 10-minute bucket, Latency affected 8 active minutes and CPU affected 7 active minutes, with 5 minutes of overlap. Every minute of the bucket was affected: 3 minutes by Latency only, 2 minutes by CPU only, and 5 minutes by both. Each minute adds 1, so the bucket has 10 impacted minutes. The impacted minutes of the two categories add up to 8 + 7 = 15. In each of the 5 overlapping minutes, each category gets an impact cost of 0.5. The impact cost is 3 + (5 × 0.5) = 5.5 for Latency and 2 + (5 × 0.5) = 4.5 for CPU, which add up to 10.
The agent writes a bucket to its local database only when the bucket had impact. A bucket with no impact produces no record.
For another example, see Health Score and Experience Index Calculation Examples.
Daily Totals
Once a day, each agent totals its health impact minutes by category for that calendar day and uploads the total to the master. Each daily record includes, per system, the active minutes, the total impact minutes, the Health Score, and the minutes for each of the 13 health impact categories.
The 30-Day Weighted Average
SysTrack averages the daily Health Scores of a system over a trailing 30-day window, using only the days that have a health record. This average is weighted by active minutes: a day with more active minutes counts more toward the average than a day with fewer active minutes. It is not a simple mean of the daily scores.
Health Score and Experience Index
The Experience Index follows the same idea as the Health Score: it compares impact with active time. It uses named sensors and 5-minute buckets instead of health categories and one-minute checks. When the Experience Index is enabled, it replaces the Health Score view in Reliability Engineering.
For more information, see Experience Index.