Excited about your brand new Elasticsearch SIEM setup, but trying to figure out which data to start with? Look no further than the landing page of the SIEM section in Kibana.
Notice the two categories:
These are reasonably self explanatory, but the idea is that with this SIEM capability, you bridge the connections between the hosts and the network to reveal emergent properties of malicious activity, and build attack chains more efficiently than looking at the data separately.
More information on the Elasticsearch SIEM and the use of different beats to ingest the additionally compatible events the other Elasticsearch SIEM guides is here. In this guide, you will install, configure, and ingest logs from the most prominent category in the host data section, Auditbeats. You can find more information on the Elasticsearch SIEM and the use of different beats to ingest the additionally compatible events the other Elasticsearch SIEM guides here.
The Auditbeat module from Elasticsearch is an agent that is loaded on to an endpoint, Linux, MacOS, or Windows that uses different modules to provide events to the Elasticsearch SIEM.
The events that are compatible with the Elasticsearch SIEM are shown on the overview page, but not all modules and datasets work on all endpoint OS systems.
Here is an overview:
|File Integrity||Windows, Linux, MacOS|
|System - Host||Windows, LInux, MacOS|
|System - Process||Windows, Linux, MacOS|
|System - Login||Linux Only|
|System - Package||Linux Only|
|System - Socket||Linux Only|
|System - User||Linux Only|
To gather information from a previous Elasticsearch SIEM installation, you will need the exposed IP of the Elasticsearch and the Kibana service found in the services respective YAML configuration files.
Assuming the elastic instance was set up to expose 192.168.218.139 and the firewall set rules to allow Elasticsearch (9200) and Kibana (5601) to listen externally, as shown in the elasticsearch setup guide, these will be the endpoint settings used for the Auditbeat configuration in this guide as well.
Though not all features are supported on Windows and MacOS, the file integrity, process, and host monitoring capabilities are significant. To get started with audit beats on Windows, first navigate to the Auditbeat downloads page within your browser of choice and select the correct distro from the list.
For this guide, I am installing Auditbeats on a 64bit Windows 10 system and chose that respective package for download.
Make sure to also download the
sha file for integrity checking and then open up Powershell, navigating to the download location of the two files.
An alternative command line is
Open Powershell and navigate to the download location, hash the downloaded zip file, and print the contents from the
sha integrity file and ensure the two values match before continuing.
1 2 3 4 5
cd c:/Users/Globo-Control/Downloads/ (get-file -algorythim SHA512 auditbeat).hash get-content auditbeat.sha
The results should be two hashes printed that match and look something like this:
With verified installation media downloaded from Elasticsearch, it is time to install the Auditbeat client.
Open Windows Explorer to the location containing the Auditbeat zip file, right-click the file, and select extract all.
Choose the location to extract the contents of the archive and click extract.
Then verify successful extraction of folder and files to the desired location.
You can install Auditbeat as a service by running the supplied
ps1 file from an elevated command prompt. It will set up an automatic start service and use the default
auditbeat.yml config file.
Note: Install service Powershell module for later
Before running, you need to let Auditbeat know where to send the data. In this case, it is the Elasticsearch SIEM set up previously.
auditbeat.yml file located in the extracted folder with Powershell ISE using the run as admin option. Powershell ISE is the choice here because it will display the yaml text in a easily readable and editable format while providing a Powershell console to run the commands that will verify the configurations were properly implemented.
Once you have Powershell ISE open, select the file menu button, then open. Navigate to the extracted Auditbeat folder location and change the file type selector to Any File
You should now see the
auditbeat.yml file available for selection.
Now that the file is open, you need to scroll down until you see the
Kibana section,which will have the default values commented out and look like this:
# from the line that currently says
localhost:5601 and change that value to equal
http://192.168.218.139:5601, to the listening address of your respective Kibana node.
Do not forget to add the http:// url prefix to the Kibana address when using an external instance.
The results should look like this:
Next, continue scrolling to the
Output section of the configuration file to the first subheading named
Similar to the Kibana section configuration, you will want to remove the
# comment character and replace the
localhost:9200 value with the value of your Elasticsearch node's listening address. In this case, it will be
Note: This config differs from the kibana configuration in that it does not require the http:// prefix.
Now save the configuration, but do not close the Powershell ISE Window.
In the bottom of the Powershell ISE window, run the following command to change to the extracted Auditbeat directory and run the setup.
1 2 3
PS> cd C:\auditbeat-7.3.2-windows-x86_64 PS> .\auditbeat.exe -e -c .\auditbeat.yml setup
Once run, the the Auditbeat program will run checks agains the config and the environment and reach out to the Elasticsearch and Kibana node configured to create an index and index template, pattern, policy, and upload pre-built dashboards.
A successful run will finish with
Loaded Dashboards and return you to the command prompt.
With the setup successfully complete, it is time to run Auditbeats for the first time. It is always good troubleshooting practice to run the Auditbeats executable not only from the command line, but with the
-e option, which tells the program to output to standard out, allowing you to see the activity and troubleshoot problems as they arise.
Once you are happy with the configuration and stability of the Auditbeats program, you can then add it as a service that runs automatically with the start of system.
Run Auditbeats with following command:
PS> .\auditbeat.exe -e
As it runs, you can see the successful connections to Elasticsearch and Kibana, but once initialized, you can also see the JSON output of the metrics of events sent to the Elasticsearch SIEM with the
[monitoring] event category shown at the bottom of the console print out.
The moment you have been waiting for is here. You will finally get your hands on that juicy Auditbeat host data. Though you trust the console output from the client, browse to the Kibana node and click the SIEM window to verify successful transmission and collection of data.
In the hosts section of the Elasticsearch SIEM, you now have one host populated and listed by name in the
All Hosts card resulting from the default configuration for the system module to include the host dataset.
Event Data is also included as the output of the file integrity module.
Uncommon Processes is populated from the process data set of the system module.
You probably noticed that the authentication and IP data that is shown in the hosts card is missing, and if you took a look at the overview page you would see that under the Auditbeat audit, login, package, user, and socket modules there is still a
0, indicating that Elasticsearch SIEM has not received any of this type of data within the given time period. This makes sense because those are the exact modules that are not compatible with the Windows operating system. To get data from the rest of the modules, you need to install and configure the Auditbeat agent on a Linux OS device.
No reason to leave any data analysis profit on the table. Following the same installation process and centOS 7 system as the Filebeats program in the Setting Up Elasticsearch SIEM guide, you will be able to leverage all the available modules.
First open a terminal and use the following commands to download the rpm and install the Auditbeat 7.3.2 agent.
sudo curl -L -O https://artifacts.elastic.co/downloads/beats/auditbeat/auditbeat-7.3.2-amd64.rpm
sudo rpm -vi auditbeat-7.3.2-x86_64.rpm
Then edit the
auditbeat.yml file located in
sudo nano /etc/auditbeat/auditbeat.yml
The configuration file has some different modules available, but otherwise is identical to the configuration file for Windows. Navigate to the Kibana section and change it to the external listening address of the Kibana node, and then do the same in the Elasticsearch section under output.
Press ctrl+x, then y, then return/enter to save the file from nano.
Now, just as in Windows, you need to run the setup. The Linux version of the Auditbeat agent will have different index patterns and dashboard to upload because it supports many more modules and datasets than the agent for MacOS or Windows.
sudo auditbeat -e setup
Successful setup will again end with the
Loaded Dashboards text and return you to the command prompt.
Next, use screen to create a new terminal session, and run Auditbeat with the
-e flag to monitor the progress and look for the `monitoring] event category to indicate successful transmission of events to the Elasticsearch SIEM.
1 2 3
sudo screen sudo auditbeat -e
As the Auditbeat agent initializes, you may see indications that the
auditd module does not have any rules specified but also failed to receive audit events. This is normal and you will deal with this later.
Once you see the indication that events from the other modules are being sent, open up the browser to the Kibana node and the Elasticsearch SIEM host window and behold!
The second host is listed in the
All Hosts card, and you now have data populating the
User Authentications and
Unique IPs section. If you pivot to the SIEM overview tab, you can see there are events in every single category of Auditbeat data sources.
You might notice in this environment the Auditbeat Audit category only has one event, which stands out due to the earlier errors you may have also seen in the
If you did not have those errors, and if you have more than one event in the Audit beat, then disregard the following instructions.
Take a look at this one event from the
auditd module. You can see that it is simply an event saying that it failed to start this module and that an audit process was already running. This is confirmed on the centOS 7 endpoint being used in this guide but may vary depending on distribution.
Also, remember that in the logs generated to standard out from the
-e flag, when this executable was run on the Linux device mentioned that no
auditd rules were set.
Back in the centOS7 device running Auditbeat, press ctrl+c to exit the process to terminal, and open up the configuration file.
sudo nano /etc/auditbeat/auditbeat.yml
Notice the modules sections, beginning with
file integrity. This is the module that scans for changes in files and folder locations that you are concerned with. The values can be changed and customized, and the options for this module can be found in detail on Elastic's website here.
Below, you can see the system module running now with every possible data set, which was not the case on the Windows system due to the reduced compatibility and supported features. More options for the system module can be found here.
All of the modules are configured by default on Linux systems with the exception of the
auditd module, which requires you to either write custom rules or uncomment (remove the
#) rules that are already in the configuration.
Once you have uncommented the rules, you have eliminated the first error that was recognized for the
auditd module during the running of the Auditbeat process. Save the file from nano and exit by pressing ctrl+x, then y then return.
Back at the terminal, you now need to stop the process that is currently running monitoring audit events. This process, known as
auditd, is run on the local OS in centOS 7 by default to record audit logs to disk. You have to stop this service, disable it, and optimize the disk space usage by sending the audit events to devel null, so that the
auditd module can run in its place.
service auditd status
Confirm that the
auditd status is indeed running and is the reason that you cannot run the
auditd module from Auditbeat.
Then stop the service and disable it to prevent it from starting automatically at startup.
> sudo service auditd stop
sudo systemctl disable auditd
Note: If you are using a debian-based linux, you may need to run
chkconfig auditd off to disable the auditd service permanently.
Now optimize the OS and disk preventing the audit socket from writing logs to disk.
sudo systemctl mask systemd-journald-audit.socket
With the OS taken care of, you need to also make sure you have your bases covered with the
auditd module configuration. Start by showing the audit rules to ensure the rules were properly uncommented and recognized.
sudo auditbeat show auditd-rules
Check the status of the
sudo auditbeat show auditd-status
Test your configuration file changes.
sudo auditbeat -e test config
And with the okay from Auditbeat itself, it is time to re-run.
sudo auditbeat -e
This time, you should not see any errors for
auditd, and once you see good events in the
monitor category, shift over to kiban and the Elasticserch SIEM overview tab.
Now you can see that there are 73 messages from
auditd, which makes more sense from something that should be actively sending events and alerts from the kernel.
Just like that, you have working data from every Auditbeat module for the Elasticsearch SIEM. You can view them in the host tab, network (which includes Auditbeat socket), or in the form of a custom timeline of select events. The SIEM has optimized queries and parsing for these log formats, which can be seen in the search bar.
And outside of the SIEM, there are custom visualizations and dashboards for each of the Auditbeat modules and datasets.