Thresholds Rules

This section provides the information about the metrics processing rules and generation of new Acure objects - Thresholds.

• Threshold Rules management
  • Threshold Rule creation
  • Activate/Stop Threshold Rules
  • Threshold Rule deletion
  • Threshold Rules export/import
• Threshold calculator (processor)
• Section access rights

Event generation by thresholds from metrics with the aim of further passing through processing event pipeline gives the opportunity to fully process metrics in Acure.

Thresholds are generated from the collected metrics in a special Acure processor - mcs-thresholds-service, which runs calculation rules with a certain frequency. The result of the processor's work can be:

• Opening a new threshold
• Closing an existing threshold
• Confirming the threshold

When a threshold opens or closes, special events are generated as a result, which then come to the main Acure event processing path.

The opening events are sent to the ThresholdsProcessor Automation Scenario type for binding threshold to a CI.

The process of Threshold binding is performed automatically in a special Automation Scenario type using scenarios developed by users with an indication of the rules for binding according to the labels of the received metrics.

A Threshold, like a Signal, is a short-lived object that has a start and end time.

⚠️Important note

It is not allowed to open closed thresholds in Acure.

Threshold Rules management

Threshold Rules section for threshold management was added to the Data Collection (ETL) on the left sidebar menu.

In order to work with the threshold rules, open the appropriate section.

Threshold Rule creation

To create a new threshold calculation rule, click + New rule button in the top-right corner of the page.

Fill in the rule creation form:

• Owner

  Workgroup to which the threshold calculation rule belongs

• Name

  Name of the rule must be unique within the Workgroup

• Frequency

• Description (optional)

• Import

  Code of the threshold calculation rule in base64 format (for more details, see Export/import of threshold rules

After creating a rule for calculating thresholds, it should be configured:

1. Rule settings

   • Calculation frequency

     The time interval that determines how often the rule will be calculated.

     By default: 10 min.

2. Metrics Query

   • Data Streams

     You should select at least one Data stream - the source of the metrics.

   • Filter Query

     Request for metric values in MetricsQL (PromQL)

     For example:

     aggregator_openapi_v2_regeneration_count{endpoint="https", instance="10.18.0.100:6443", job="apiserver"}

     The more detailed the set of labels is, the more accurate the identification of a particular vector will be.

     If you simply specify the name of the metric: aggregator_openapi_v2_regeneration_count, then the request will also be valid. However instead of one vector, a set of several vectors will be received.

     💡Tip

     If it is necessary to make changes to the Query, you should stop the Rule first in order to make all the created Thresholds closed. Otherwise, opened Thresholds will remain in the system, and new ones will not be able to overwrite them.

   • Evaluation Window

     The time period in which the aggregation function of metric values will be calculated. The window allows you to exclude abrupt changes in the metric value, reacting only to changes over a longer period of time.

   • Aggregation (function)

     • Last - the last value of the vector for calculation is used.
     • Average - the average value of the vector is taken.
     • Max - the maximum value of the vector is used.
     • Min - the minimum value of the vector is used.
     • Sum - all values of the vector are summed.

     💡Tip

     All aggregation functions apply to time series within a given evaluation window.

   The entered query can be checked by pushing the Run Query button. All time series matching the query will be loaded.

   

   3. Conditions

   

   The user can set conditions for the calculation threshold level in order to classify the obtained metric values according to the degree of criticality.

   Several severity levels are available for setting conditions:

   • Fatal
   • Critical
   • Major
   • Warning
   • Info

   By default, OK status is set for newly created threshold rules.

   4. Threshold Settings

   

   • Autoclose

     The time after which the threshold will be automatically closed if threshold confirmation events are not received.

     The countdown is on from the moment the threshold is opened/confirmed.

   • Threshold Name

     Macros are allowed (specified in double curly braces).

     • Rule.Id - Rule ID
     • Rule.Name - Rule Name
     • Rule.Description - Rule Description
     • Rule.CreatedAt - date of the threshold rule creation
     • Rule.FilterQuery - the Filter Query field value
     • Rule.CheckIntervalSeconds - the interval for requesting metrics in seconds
     • Rule.EvaluationWindowSeconds - the Evaluation Window field value in seconds
     • Rule.AutoCloseTimeSeconds - the Autoclose field value
     • Rule.AggregationFunction - the aggregation function specified for the rule
     • Rule.StreamIds - selected Streams for the rule
     • Rule.Conditions - list of the configured conditions
     • Rule.OwnerWorkGroupId - ID of the Workgroup-owner of the threshold rule
     • Rule.OwnerWorkGroupName - Name of the Workgroup-owner of the threshold rule
     • Condition.Function - the comparison function specified for the triggered rule
     • Condition.Value - with what value the comparison was made
     • Condition.Level - triggered condition level
     • Condition.LevelNumber - digital level value
     • Metric.Value - the metric value when calculating the threshold
     • Metric.Labels - metric labels
     • Metric.Hash - unique hash of the metric
     • Metric.Name - metric Name
     • Metric.Labels.<metric name> - allows to specify the name of the metric label

     💡Tip

     If there is no value, the macro is ignored.

   • Annotations

     Additional threshold fields that will be assigned to it when it is created

Activate/Stop Threshold Rules

Activation

After setting up the threshold calculation rule you can run it and start analyzing the metric series entering the system.

Click the Activate button in the top-right corner to start the calculation under this rule.

Immediately after the rule is activated, metric series matching the request will be checked and thresholds will be generated in accordance with the thresholds creating conditions.

By default, thresholds are created without being bound to any CI.

In order to see the generated thresholds, activate the corresponding filter setting - "Without CI - Show" on the Thresholds tab of the SM Maps section.

Stop

In order to stop the threshold calculation under a certain rule, you need to Stop it with the corresponding button in the rule settings.

When the rule is stopped by default the user is prompted to close all previously opened thresholds for this rule.

Threshold Rule deletion

Deleting threshold calculation rules is available in the rule settings, in the additional menu.

⚠️Important note

After deleting the calculation rule, all thresholds created under this rule will be closed.

Threshold Rules export/import

For convenience of setting up the logic of processing metrics, the following functions are available to users:

• Cloning - creating a copy of the threshold rule in the current Userspace.

  

• Export - export threshold rule settings to the base64 format.

  

• Import - import settings from base64 format to the current rule

  ⚠️Important note

  All current rule settings will be replaced with the values from the imported rule.

  

Threshold calculator (processor)

The calculation of the rule is executed in the time interval specified by the user in the rule. During the calculation process, a request is made to the metrics storage, according to the selected data streams and the indicated request.

Each request is wrapped in an aggregation function specified by the user in the rule settings.

For each received time series, a hash identifier is generated, which is used to check the presence of a previously created threshold for this time series.

After executing the rule expression, the result is compared with the table of current open thresholds by a hash identifier:

• If there is no threshold with such a hash identifier, a new threshold is created.
• If there is a threshold with such a hash identifier and the criticality level matches, the threshold lifetime is updated.
• If there is a threshold with such an identifier and their levels do not match, the old threshold is closed and a new one with a different criticality level is opened.

Section access rights

Using the Workgroup role model, the manager can set up certain user roles that will be available to:

• View - viewing of all rules within the Workgroup is available.
• Edit - full access to manage Threshold Rules.
• Prohibit - access to viewing Thresholds Rules is denied.