Docs

# Data streams

This guide describes methods of connecting different data sources (including logs) to the Acure system.

Image

For quick access to the functionality you are interested in, use the navigation:

# Adding a Data Stream

  1. Go to the section Data Collection→Data Streams via the side menu.
  2. Click the Add stream button in the upper right corner of the screen.
  3. Fill in the fields:
    • Owner - Workgroup, which will own the data stream.
    • Configuration template - select the required configuration template (Configuration templates).
    • Name (unique within Workgroup).
    • Description (optional).
  4. Click the Add button - the configuration page of the added data stream will open.
  5. On the tab Settings of data stream, fill in the configuration parameters according to the selected configuration template.

    ⚠️ The stream will not run without filling in the configuration parameters of the selected template.

  6. On the Access tab, if necessary, set additional permissions for the workgroup to View or Edit the created data stream.

Reference

The integration setup depends on the Configuration Template of the data stream you have chosen. Go to connection examples in the Solutions section for details.

# Starting and stopping a Data Stream

# Start/Stop Data Stream

  1. Go to the section Data Streams via the side menu.
  2. Select the required stream.
  3. Take advantage of the auxiliary menu under the icon οΈ™ to start/stop the data stream.

You can also start or stop a stream from the stream settings page:

  1. Go to the stream setup page.
  2. In the upper right corner, click the Start/Stop button.

πŸ’‘ When the stream starts, the filling of the fields in the stream tab Configuration is checked. The stream will not be started if the fields are empty.

# Data stream settings

  1. Open the Settings tab of the required data stream.
  2. On this tab you can change:
    • Name
    • Owner
    • Description
    • Configuration parameters
    • External system URL
  3. Unavailable for changing:
    • API-key
  4. The configuration template can be changed in the Configuration tab.
  5. To save the changes, click the "Save" button

The "Save" button can be disabled in two cases:

  • If the user has not changed anything in the "Settings" tab.
  • After changing the parameters, there are validation errors in the fields in the "General settings" block on the "Settings" tab.

Image

# Data storage

The user has the ability to define the storage time of events for the selected stream.

Image

Event storage time is specified in days from one day to infinity. By default, created data streams have a retention time of "Infinity".

When this setting is enabled for streams that have receivd events from a date not included in the retention interval, the user receives a message with the option to confirm or cancel the action.

Events from [date of the first event in the stream] to [date of the first event that did not fall into the storage interval] will be deleted from the system without the possibility of restoring them.

To enable automatic deletion of data older than a specified number of days:

  1. Go to the "Settings" tab of the required data stream
  2. In the block "Data storage" select the parameter "during" from the drop-down list Store primary events.
  3. Specify the number of days to store events.
  4. Click the "Save" button.

# Data stream configuration

The built-in Acure configuration templates are composed of Tasks and Handlers

# Tasks

In the Tasks block under the template selection element, a list of preset tasks for the current template is presented.

For templates: Ntopng, Anystream, Prometheus, No template there are no preset tasks.

Attention

⚠️ Execution of Task scripts is performed by Acure Agents.

Guidelines for connecting Agents

Image

# Adding a task

You can add your own tasks to the preset tasks.

  1. Click the "Add Task" button.
  2. Select "New task" or select a specified task from the list.
  3. To assign a task, go to the code editing window on the right (YAML).
  4. Write a script for the task.
  5. Assign an agent by choosing the coordinator tag through which this task will be performed.

    ⚠️ The tag SharedAgents is used to execute jobs on the Acure system agent.

    ⚠️ Please note that the execution of jobs with the run command on the system agent is prohibited by internal security policy.

  6. Set the frequency of the task launch by selecting the launch interval.

    ⚠️ Please note that the coordinator distributes tasks every 5sec, this time will be added to the start interval.

  7. Click the "Save" button.

# Editing a task

You can only edit your own tasks.

Built-in tasks are view-only and can be used as examples.

To edit your assignment:

  1. Hover the mouse cursor over the task and click Editor.
  2. In the Script version list, select Draft to unlock the editor window.
  3. Change the job code in the (YAML) code editor window.
  4. Click the Save button.

# Deleting a task

Only added tasks can be deleted. Tasks defined by a configuration template cannot be deleted.

To delete them, click on the trash can icon opposite the task name - the object will be deleted.

# Handlers

The Acure users can write their own handlers for incoming data streams.

To start working with handlers, go to the Configuration tab on the page of the selected data stream.

Image

If necessary, change Configuration template by selecting the one you need from the list of available templates.

⚠️ Modifying the configuration template removes all native handlers.

# Adding a handler

You can add your own handlers to the predefined handlers.

  1. Click the "+Add Handler" button.
  2. Select a predefined handler from the list or create your own "+New handler".
  3. To write a handler, go to the code edit window on the right (Lua IDE).
  4. Write a script for processing the incoming stream..
  5. Click the "Save" button.

# Editing a handler

You can only edit your own handlers. Built-in handlers are view-only and can be used as examples.

To edit a handler:

  1. Move the mouse cursor over the handler and click Edit.
  2. In the Script version list, select Draft to unlock the editor window.
  3. Modify the handler script in the Lua code editor window.
  4. Click the Save button.

# Deleting a handler

You can delete only added handlers. The handlers defined by the configuration template cannot be deleted.

To delete them, click on the trash icon near with handler name and the object will be deleted.

# Deleting a Data Stream

  1. Go to the Data Collection→Data Streams section in the side menu.
  2. Find the required data stream.
  3. Take advantage of the auxiliary menu under the icon οΈ™ to delete the data stream.

You can also delete a data stream from the stream setup page:

  1. Find the required stream and go to the stream setup page.
  2. In the upper right corner, click the Delete button.

# Self-monitoring

Image

# Monitoring incoming events

The user has access to the function of self-controlling the state of data streams.

To enable checking for primary events in a stream, follow these steps:

  1. Go to Data Collection→Data Streams via the main menu.
  2. Find the required stream.
  3. Click the "Settings" tab of the data stream.
  4. In the Stream Monitoring block, activate the switch "Send error primary event if any primary event hasn't been aggregated for more than N hours".

    By default, this feature is disabled for all data streams.

  5. Set the interval for checking events in the corresponding field (in hours). Only integer values ​​are available for input.
  6. Save the data stream settings.

Checking for primary events in a data stream works like this:

  1. Runs periodically in an interval set by the user.
  2. The presence of primary events on the stream in this time interval is checked, without taking into account self-monitoring events and events sent by stream errors.
  3. If there are primary events on the thread, no action is taken.
  4. If there are no primary events in the thread, a service primary event is generated and sent to the data stream:
{
    "source": {
        "monqStreamControl": {
            "statusValue": "Warning",
            "alertMessage": "Events don't arrive on the [NAME] stream for more than [INTERVAL]",
            "alertType": "Stream",
            "alertSourceName": "Self-monitoring"
        }
    }
}

# Monitoring tasks and handlers

If errors occur in the operation of handlers or tasks of a data stream, it is possible to send corresponding events to this data stream.

Please note that the generation of error events is directly related to the launch interval of Tasks of the corresponding Data Stream.

For example, if a Task is set to run every 5 seconds, and it is impossible to complete this task, then an error event will be generated every 5 seconds and sent to the corresponding Data Stream.

To enable this feature:

  1. Go to Data Collection→Data Streams via the main menu.
  2. Find the required stream.
  3. Click the "Settings" tab of the data stream.
  4. In the Stream Monitoring block, activate the switch "Send primary events for the stream errors".

    By default, this feature is disabled for all data streams.

  5. Save the data stream settings.

An example of an event that will be sent to the stream:

{
    "source": {
        "monqStreamControl": {
            "statusValue": "Error",
            "alertMessage": "[Error message]",
            "alertType": "AgentTask",
            "alertSourceName": "[Task/handler NAME]"
        }
    }
}

Possible values for the "alertType" field:

  • "AgentTask" - an error in the agent task.
  • "StreamHandler" - an error in the stream handler.

# Operations with Data Streams

# Searching for data streams

  1. Go to the Data Collection→Data Streams section in the side menu.
  2. In the upper left corner, in the Search field enter some text, and the search will be performed according to the following parameters:
    • Stream name,
    • Stream description,
    • Stream ID,
    • Stream Owner,
    • Stream Configuration template.

# Actions with data streams

  1. Click the οΈ™ icon next to the required Data Stream and select the required action:
    • Start/stop stream
    • Copy Stream link
    • Copy Stream ID
    • Copy API-key
    • Open the panel with Data Stream's collected data (User guide)
    • Open External System (the item is available when the field External System URL is filled).

# Sorting data streams

  1. Go to the Data Collection→Data Streams section in the side menu.
  2. Click on one of the column headings:
    • Stream
    • Status
    • Owner
    • Configuration template
    • Per Day
    • Average
  3. Each click on the heading performs:
    • Sort ascending
    • Sort descending
    • Disable sorting by this column

# Configuring access rights to a data stream

  1. Go to the Data Collection→Data Streams section in the side menu.
  2. Find the required data stream.
  3. Open the Access tab of the data stream.
  4. In the field Select Workgroups select the workgroups you want to allow access.
  5. Select View or Edit.
  6. Click Grant Access to grant the access rights.
  7. If necessary, turn on Share with all workgroups, including the future ones to provide view access for all workgroups.

    πŸ’‘ Share with all workgroups, including the future ones can only be granted by a Userspace Administrator.

# Data stream statistics

On the Statistics tab, users have access to information about the events (logs) and metrics collected in the data stream.

Information is presented in the form of histograms with statistical indicators.

# Histogram of Events and Logs

The histogram of Events and Logs displays the amount of data received through the Data Stream for the selected period of time with the following indicators:

  • Amount of data for the selected period
  • Average amount of data for the selected period = amount of data for the period / per number of timeslots of the period
  • The maximum amount of data for the selected period = the maximum amount of one of the time intervals of the period
  • The minimum amount of data for the selected period = the minimum amount of one of the time intervals of the period

image

# Histogram of Metrics

Histogram of Metrics displays the quantity of Metrics collected form the Data stream for the selected period of time with the following indicators:

  • Quantity for the selected period
  • Average quantity for the selected period = number for the period / per number of timeslots of the period
  • The maximum quantity for the selected period = the maximum amount of one of the time intervals of the period
  • The minimum quantity for the selected period = the minimum quantity of one of the time intervals of the period

image

# Definition of terms

# Status of a Data Stream

Integral indicator of the state of the data stream. Has the following states:

  • Ok - data stream is collecting events,
  • Unknown - data stream has no incoming data,
  • Problem - data stream contains an error message.

It is calculated based on the execution statuses of the handlers and tasks of the agent of the corresponding data stream. It is assigned by the worst value of the script statuses.

πŸ’‘ In addition to the status value, the time of its last calculation is displayed (the time is updated even if the status has not changed).

# Average data volume

The arithmetic mean of the amount of data received by the data stream over the last 30 days (if the period of operation of the stream is less than 30 days, then for the entire period of operation of the stream).

# Data volume over today

The total amount of data received by the data stream for the current day.