Data streams

This guide describes methods of connecting different data sources (including logs) to the Acure system.

For quick access to the functionality you are interested in, use the navigation:

• Adding a Data Stream
• Starting and stopping a Data Stream
  • Start/Stop Data Stream
• Data stream settings
  • Data storage
• Data stream configuration
  • Tasks
  • Handlers
• Self-monitoring
  • Monitoring incoming events
  • Monitoring tasks and handlers
• Operations with Data Streams
  • Data Streams Filtering
  • Searching for data streams
  • Actions with data streams
  • Sorting data streams
• Configuring access rights to a data stream
• Data stream statistics
  • Histogram of Events and Logs
  • Histogram of Metrics
• Definition of terms
  • Status of a Data Stream
  • Average data volume
  • Data volume over today
• Data Streams exporting
• Deleting Data Streams

Adding a Data Stream

1. Go to the section Data Collection (ETL) - Data Streams via the side menu.
2. Click the Add stream button in the upper right corner of the screen.
3. Fill in the fields:
   • Owner - Workgroup, which will own the data stream.
   • Configuration template - select the required configuration template (Configuration templates).
   • Name (unique within Workgroup).
   • Description (optional).
   • Import - the code exported in base64 format (read more about Data Stream export).
4. Click the Create button - the configuration page of the added data stream will open.
5. On the tab Settings of data stream, fill in the configuration parameters according to the selected configuration template.

   ⚠️ The stream will not run without filling in required parameters of the selected template.

6. On the Access tab, if necessary, set additional permissions for other workgroups to View events or Edit configuration of the current data stream.

Reference

Futher data stream setup depends on the Configuration Template of the data stream you have chosen and on the requirements of the data source. Go to the Integrations section to explore examples of data stream configurations for various data sources.

Starting and stopping a Data Stream

Start/Stop Data Stream

1. Go to the section Data Collection (ETL) - Data Streams via the side menu.
2. Select the required data stream.
3. Take advantage of the auxiliary menu under the icon ︙ to start/stop the data stream.

You can also start or stop a data stream on the stream configuration page:

1. Go to the stream setup page.
2. In the upper right corner, click the Start/Stop button.

💡 When the stream starts, the filling of the fields in the stream tab Configuration is checked. The stream will not be started if the fields are empty.

Data stream settings

1. Open the Settings tab of the required data stream.
2. On this tab you can change:
   • Name
   • Owner
   • Description
   • Configuration parameters
   • External system URL
3. Unavailable for changing:
   • API-key
4. The configuration template can be changed in the Configuration tab.
5. To save the changes, click the "Save" button

The "Save" button can be disabled in two cases:

• If the user has not changed anything in the "Settings" tab.
• After changing the parameters, there are validation errors in the fields in the "General settings" block on the "Settings" tab.

Data storage

If you need to save disk space, specify the event retention time for the selected data stream.

Event storage time is specified in days from one day to infinity. By default, created data streams have a retention time of "Infinity".

If an incorrect value is specified, the value of 365 days is set.

When this setting is enabled for streams that have receivd events from a date not included in the retention interval, the user receives a message with the option to confirm or cancel the action.

Events from [date of the first event in the stream] to [date of the first event that did not fall into the storage interval] will be deleted from the system without the possibility of restoring them. Data stream events that are subject to deletion will be removed within an hour.

To enable automatic deletion of data older than a specified number of days:

1. Go to the "Settings" tab of the required data stream
2. In the block "Data storage" select the parameter "during" from the drop-down list Store primary events.
3. Specify the number of days to store events.
4. Click the "Save" button.

Data stream configuration

On the Configuration tab of the data stream, the user can configure their own Tasks for data collection and Handlers - rules for preprocessing the collected data.

Tasks

In the Tasks block under the template selection element, a list of preset tasks for the current template is presented.

For templates: Ntopng, Prometheus, Default there are no preset tasks.

Attention

⚠️ Task scripts lanches are performed by Acure Agents.

Guidelines for connecting Agents

Task creation

Users can add their own tasks in addition to the pre-installed tasks.

To add a new task, follow the next steps:

1. Click the "Add Task" button.
2. Select "New task" or select a specified task from the list.
3. To write a scenario of a new task, go to the code editing window on the right (YAML).
4. Write a script for the task.
5. Select the "Agent Label" on which this task will be executed.

   ⚠️ The tag SharedAgents is used to execute jobs on the Acure system agent (microservice pl-acure-agent).

   ⚠️ Please note that the launch of tasks with the run command on the system agent is prohibited by internal security policy. Image

6. Set the frequency of the task launch by using the CRON format.

   ⚠️ Automated launch is switched off by default when adding a task.

7. Click the "Save" button.

Editing a task

You can only edit your own tasks.

Built-in tasks are view-only.

To edit your assignment:

1. Hover the mouse cursor over the task and click Editor.
2. At the right top corner in the Script version field, select Draft to create a new version of the task.
3. Make necessary edits to the script code using the code editor window (YAML).
4. Click the Make Executable button.

Manual Launch

To manually run (debug) a task script, use the Manual Launch button in the task schedule settings.

Disabling a Task

You can disable the execution of a task by setting the Launch option to No.

Deleting a task

To delete a previously added task, click on the trash can icon next to the task name - the task will be removed. This action is irreversible and does not require additional confirmation.

⚠️ You can only delete your own tasks. Tasks defined by the configuration template cannot be deleted.

Handlers

The Acure users can write their own handlers for incoming data streams.

To start working with handlers, go to the Configuration tab on the page of the selected data stream.

If necessary, change Configuration template by selecting the one you need from the list of available templates.

⚠️ Modifying the configuration template removes all native handlers.

Adding a handler

You can add your own handlers to the predefined handlers.

1. Click the "+Add Handler" button.
2. Select a predefined handler from the list or create your own "+New handler".
3. To write a handler, go to the code edit window on the right (Lua IDE).
4. Write a script for processing the incoming stream..
5. Click the "Save" button.

Editing a handler

You can only edit your own handlers. Built-in handlers are view-only and can be used as examples.

To edit a handler:

1. Move the mouse cursor over the handler and click Edit.
2. In the Script version list, select Draft to unlock the editor window.
3. Modify the handler script in the Lua code editor window.
4. Click the Save button.

Deleting a handler

You can delete only added handlers. The handlers defined by the configuration template cannot be deleted.

To delete them, click on the trash icon near with handler name and the object will be deleted.

Self-monitoring

Monitoring incoming events

The user has access to the function of self-controlling the state of data streams.

To enable checking for primary events in a stream, follow these steps:

1. Go to Data Collection→Data Streams via the main menu.
2. Find the required stream.
3. Click the "Settings" tab of the data stream.
4. In the Stream Monitoring block, activate the switch "Send error primary event if any primary event hasn't been aggregated for more than N hours".

   By default, this feature is disabled for all data streams.

5. Set the interval for checking events in the corresponding field (in hours). Only integer values ​​are available for input.
6. Save the data stream settings.

Checking for primary events in a data stream works like this:

1. Runs periodically in an interval set by the user.
2. The presence of primary events on the stream in this time interval is checked, without taking into account self-monitoring events and events sent by stream errors.
3. If there are primary events on the thread, no action is taken.
4. If there are no primary events in the thread, a service primary event is generated and sent to the data stream:

{
    "source": {
        "monqStreamControl": {
            "statusValue": "Warning",
            "alertMessage": "Events don't arrive on the [NAME] stream for more than [INTERVAL]",
            "alertType": "Stream",
            "alertSourceName": "Self-monitoring"
        }
    }
}

Monitoring tasks and handlers

If errors occur in the operation of handlers or tasks of a data stream, it is possible to send corresponding events to this data stream.

Please note that the generation of error events is directly related to the launch interval of Tasks of the corresponding Data Stream.

For example, if a Task is set to run every 5 seconds, and it is impossible to complete this task, then an error event will be generated every 5 seconds and sent to the corresponding Data Stream.

To enable this feature:

1. Go to Data Collection→Data Streams via the main menu.
2. Find the required stream.
3. Click the "Settings" tab of the data stream.
4. In the Stream Monitoring block, activate the switch "Send primary events for the stream errors".

   By default, this feature is disabled for all data streams.

5. Save the data stream settings.

An example of an event that will be sent to the stream:

{
    "source": {
        "monqStreamControl": {
            "statusValue": "Error",
            "alertMessage": "[Error message]",
            "alertType": "AgentTask",
            "alertSourceName": "[Task/handler NAME]"
        }
    }
}

Possible values for the "alertType" field:

• "AgentTask" - an error in the agent task.
• "StreamHandler" - an error in the stream handler.

Operations with Data Streams

Data Streams Filtering

Data streams can be filtered based on the following parameters:

• Owner
  • Workgroup to which the data stream belongs
• State
  • Active
  • Stopped
• Health status
  • OK
  • Unknown
  • Error

Image

Filter settings are not saved after navigating to another page.

Searching for data streams

1. Go to the Data Collection→Data Streams section in the side menu.
2. In the upper left corner, in the Search field enter some text, and the search will be performed according to the following parameters:
   • Stream name,
   • Stream description,
   • Stream ID,
   • Stream Owner,
   • Stream Configuration template.

Actions with data streams

1. Click the ︙ icon next to the required Data Stream and select the required action:
   • Start/stop stream
   • Copy Stream link
   • Copy Stream ID
   • Copy API-key
   • Open the panel with Data Stream's collected data (User guide)
   • Open External System (the item is available when the field External System URL is filled).

Sorting data streams

1. Go to the Data Collection→Data Streams section in the side menu.
2. Click on one of the column headings:
   • Stream
   • Status
   • Owner
   • Configuration template
   • Per Day
   • Average
3. Each click on the heading performs:
   • Sort ascending
   • Sort descending
   • Disable sorting by this column

Configuring access rights to a data stream

1. Go to the Data Collection→Data Streams section in the side menu.
2. Find the required data stream.
3. Open the Access tab of the data stream.
4. In the field Select Workgroups select the workgroups you want to allow access.
5. Select View or Edit.
6. Click Grant Access to grant the access rights.
7. If necessary, turn on Share with all workgroups, including the future ones to provide view access for all workgroups.

   💡 Share with all workgroups, including the future ones can only be granted by a Userspace Administrator.

Data stream statistics

On the Statistics tab, users have access to information about the events (logs) and metrics collected in the data stream.

Information is presented in the form of histograms with statistical indicators.

Histogram of Events and Logs

The histogram of Events and Logs displays the amount of data received through the Data Stream for the selected period of time with the following indicators:

• Amount of data for the selected period
• Average amount of data for the selected period = amount of data for the period / per number of timeslots of the period
• The maximum amount of data for the selected period = the maximum amount of one of the time intervals of the period
• The minimum amount of data for the selected period = the minimum amount of one of the time intervals of the period

Histogram of Metrics

Histogram of Metrics displays the quantity of Metrics collected form the Data stream for the selected period of time with the following indicators:

• Quantity for the selected period
• Average quantity for the selected period = number for the period / per number of timeslots of the period
• The maximum quantity for the selected period = the maximum amount of one of the time intervals of the period
• The minimum quantity for the selected period = the minimum quantity of one of the time intervals of the period

Definition of terms

Status of a Data Stream

Integral indicator of the state of the data stream. Has the following states:

• Ok - data stream is collecting events,
• Unknown - data stream has no incoming data,
• Problem - data stream contains an error message.

It is calculated based on the execution statuses of the handlers and tasks of the agent of the corresponding data stream. It is assigned by the worst value of the script statuses.

💡 In addition to the status value, the time of its last calculation is displayed (the time is updated even if the status has not changed).

Average data volume

The arithmetic mean of the amount of data received by the data stream over the last 30 days (if the period of operation of the stream is less than 30 days, then for the entire period of operation of the stream).

Data volume over today

The total amount of data received by the data stream for the current day.

Data Streams exporting

System users can export configured data streams in the base64 format.

The exported data stream code contains the following information:

• Settings tab
  • Configuration parameters
    • All unprotected parameters and their values
    • All protected parameters without their values
• Custom parameters
  • All unprotected parameters and their values
  • All protected parameters without their values
• Storage settings
• Self-monitoring settings
• Configuration tab
  • Data stream tasks
    • Task names
    • Task scripts
    • Agent labels
    • Schedule launch settings

Other data stream parameters are not included in the exported data and require additional configuration after import.

To export a data stream, while on the page of all data streams or on the card of the selected data stream, use the additional menu and select "Export".

Image

Deleting Data Streams

1. Go to the Data Collection (ETL) - Data Streams section through the main menu.
2. Find the necessary data stream.
3. Use the ellipsis menu ︙ for deleting the corresponding data stream.

You can also delete a data stream from the data stream Settings page:

1. Find the necessary data stream and go to the data stream Settings page.
2. Click Delete in the top right corner.