Author avatar

Aaron Rosenmund

Setup Audit Events with Elasticsearch SIEM

Aaron Rosenmund

  • Jan 20, 2020
  • 20 Min read
  • 3,236 Views
  • Jan 20, 2020
  • 20 Min read
  • 3,236 Views
Security
Elasticsearch

Introduction to Auditbeat with Elasticsearch SIEM

Excited about your brand new Elasticsearch SIEM setup, but trying to figure out which data to start with? Look no further than the landing page of the SIEM section in Kibana.

elastic-overview-blank

Notice the two categories:

  • Host Events
  • Network Events

These are reasonably self explanatory, but the idea is that with this SIEM capability, you bridge the connections between the hosts and the network to reveal emergent properties of malicious activity, and build attack chains more efficiently than looking at the data separately.

More information on the Elasticsearch SIEM and the use of different beats to ingest the additionally compatible events the other Elasticsearch SIEM guides is here. In this guide, you will install, configure, and ingest logs from the most prominent category in the host data section, Auditbeats. You can find more information on the Elasticsearch SIEM and the use of different beats to ingest the additionally compatible events the other Elasticsearch SIEM guides here.

Technology

ProductVersionLink
Filebeat7.3.2downloads

Overview

The Auditbeat module from Elasticsearch is an agent that is loaded on to an endpoint, Linux, MacOS, or Windows that uses different modules to provide events to the Elasticsearch SIEM.

The events that are compatible with the Elasticsearch SIEM are shown on the overview page, but not all modules and datasets work on all endpoint OS systems.

Here is an overview:

ModuleCompatible OS
AuditdLinux Only
File IntegrityWindows, Linux, MacOS
System - HostWindows, LInux, MacOS
System - ProcessWindows, Linux, MacOS
System - LoginLinux Only
System - PackageLinux Only
System - SocketLinux Only
System - UserLinux Only

Gather Information from Elasticsearch and Decide What to Monitor

To gather information from a previous Elasticsearch SIEM installation, you will need the exposed IP of the Elasticsearch and the Kibana service found in the services respective YAML configuration files.

Assuming the elastic instance was set up to expose 192.168.218.139 and the firewall set rules to allow Elasticsearch (9200) and Kibana (5601) to listen externally, as shown in the elasticsearch setup guide, these will be the endpoint settings used for the Auditbeat configuration in this guide as well.

Install, Configure, and Run Auditbeat on Windows

Downloading Auditbeat on Windows

Though not all features are supported on Windows and MacOS, the file integrity, process, and host monitoring capabilities are significant. To get started with audit beats on Windows, first navigate to the Auditbeat downloads page within your browser of choice and select the correct distro from the list.

windows-download-page

For this guide, I am installing Auditbeats on a 64bit Windows 10 system and chose that respective package for download.

Make sure to also download the sha file for integrity checking and then open up Powershell, navigating to the download location of the two files.

auditbeat-windows-sha-files

Open Powershell and navigate to the download location, hash the downloaded zip file, and print the contents from the sha integrity file and ensure the two values match before continuing.

1
2
3
4
5
cd c:/Users/Globo-Control/Downloads/

(get-file -algorythim SHA512  auditbeat).hash

get-content  auditbeat.sha

The results should be two hashes printed that match and look something like this:

windows-hashes

Install

With verified installation media downloaded from Elasticsearch, it is time to install the Auditbeat client.

Open Windows Explorer to the location containing the Auditbeat zip file, right-click the file, and select extract all.

select-extract-all

Choose the location to extract the contents of the archive and click extract.

click-extract

Then verify successful extraction of folder and files to the desired location.

auditbeat-files-windows

You can install Auditbeat as a service by running the supplied ps1 file from an elevated command prompt. It will set up an automatic start service and use the default auditbeat.yml config file. Note: Install service Powershell module for later .

Configure Auditbeat for Windows

Before running, you need to let Auditbeat know where to send the data. In this case, it is the Elasticsearch SIEM set up previously.

Open the auditbeat.yml file located in the extracted folder with Powershell ISE using the run as admin option. Powershell ISE is the choice here because it will display the yaml text in a easily readable and editable format while providing a Powershell console to run the commands that will verify the configurations were properly implemented.

Once you have Powershell ISE open, select the file menu button, then open. Navigate to the extracted Auditbeat folder location and change the file type selector to Any File . You should now see the auditbeat.yml file available for selection.

auditbeat-yml-file-open

Now that the file is open, you need to scroll down until you see the Kibana section,which will have the default values commented out and look like this:

kibana-yml-section

Remove the # from the line that currently says localhost:5601 and change that value to equal http://192.168.218.139:5601, to the listening address of your respective Kibana node. Do not forget to add the http:// url prefix to the Kibana address when using an external instance. The results should look like this: kibana-section-after

Next, continue scrolling to the Output section of the configuration file to the first subheading named Elasticsearch.

elasticsearch-yml-before

Similar to the Kibana section configuration, you will want to remove the # comment character and replace the localhost:9200 value with the value of your Elasticsearch node's listening address. In this case, it will be 192.168.218.139:9200.

elasticsearch-yml-before

Note: This config differs from the kibana configuration in that it does not require the http:// prefix.

Now save the configuration, but do not close the Powershell ISE Window.

Running Auditbeat

In the bottom of the Powershell ISE window, run the following command to change to the extracted Auditbeat directory and run the setup.

1
2
3
PS> cd C:\auditbeat-7.3.2-windows-x86_64

PS> .\auditbeat.exe -e -c .\auditbeat.yml setup

run-auditbeat-setup

Once run, the the Auditbeat program will run checks agains the config and the environment and reach out to the Elasticsearch and Kibana node configured to create an index and index template, pattern, policy, and upload pre-built dashboards.

A successful run will finish with Loaded Dashboards and return you to the command prompt. successful-aduitbeats-setup

With the setup successfully complete, it is time to run Auditbeats for the first time. It is always good troubleshooting practice to run the Auditbeats executable not only from the command line, but with the -e option, which tells the program to output to standard out, allowing you to see the activity and troubleshoot problems as they arise.

Once you are happy with the configuration and stability of the Auditbeats program, you can then add it as a service that runs automatically with the start of system.

Run Auditbeats with following command:

1
PS> .\auditbeat.exe -e

As it runs, you can see the successful connections to Elasticsearch and Kibana, but once initialized, you can also see the JSON output of the metrics of events sent to the Elasticsearch SIEM with the [monitoring] event category shown at the bottom of the console print out.

auditbeat-first-run

Verify Elasticsearch SIEM Data Collection

The moment you have been waiting for is here. You will finally get your hands on that juicy Auditbeat host data. Though you trust the console output from the client, browse to the Kibana node and click the SIEM window to verify successful transmission and collection of data.

In the hosts section of the Elasticsearch SIEM, you now have one host populated and listed by name in the All Hosts card resulting from the default configuration for the system module to include the host dataset.

windows-beat-siem

Event Data is also included as the output of the file integrity module.

windows-beat-siem-2

Uncommon Processes is populated from the process data set of the system module.

windows-beat-siem-process

You probably noticed that the authentication and IP data that is shown in the hosts card is missing, and if you took a look at the overview page you would see that under the Auditbeat audit, login, package, user, and socket modules there is still a 0, indicating that Elasticsearch SIEM has not received any of this type of data within the given time period. This makes sense because those are the exact modules that are not compatible with the Windows operating system. To get data from the rest of the modules, you need to install and configure the Auditbeat agent on a Linux OS device.

Auditbeat on Linux

No reason to leave any data analysis profit on the table. Following the same installation process and centOS 7 system as the Filebeats program in the Setting Up Elasticsearch SIEM guide, you will be able to leverage all the available modules.

Downloading and Installing Auditbeat Linux

First open a terminal and use the following commands to download the rpm and install the Auditbeat 7.3.2 agent.

1
 sudo curl -L -O https://artifacts.elastic.co/downloads/beats/auditbeat/auditbeat-7.3.2-amd64.rpm

download-auditbeat-rpm

1
sudo rpm -vi auditbeat-7.3.2-x86_64.rpm

install-audtibeats

Configuring Auditbeat Linux

Then edit the auditbeat.yml file located in /etc/auditbeat/*.

1
sudo nano /etc/auditbeat/auditbeat.yml

The configuration file has some different modules available, but otherwise is identical to the configuration file for Windows. Navigate to the Kibana section and change it to the external listening address of the Kibana node, and then do the same in the Elasticsearch section under output.

Press ctrl+x, then y, then return/enter to save the file from nano.

Running Auditbeat on Linux

Now, just as in Windows, you need to run the setup. The Linux version of the Auditbeat agent will have different index patterns and dashboard to upload because it supports many more modules and datasets than the agent for MacOS or Windows.

1
sudo auditbeat -e setup

Successful setup will again end with the Loaded Dashboards text and return you to the command prompt.

linux-auditbeat-setup

Next, use screen to create a new terminal session, and run Auditbeat with the -e flag to monitor the progress and look for the `monitoring] event category to indicate successful transmission of events to the Elasticsearch SIEM.

1
2
3
sudo screen

sudo auditbeat -e

As the Auditbeat agent initializes, you may see indications that the auditd module does not have any rules specified but also failed to receive audit events. This is normal and you will deal with this later. auditbeat-linux-run-1

Once you see the indication that events from the other modules are being sent, open up the browser to the Kibana node and the Elasticsearch SIEM host window and behold!

linux-auditbeat-run1

The second host is listed in the All Hosts card, and you now have data populating the User Authentications and Unique IPs section. If you pivot to the SIEM overview tab, you can see there are events in every single category of Auditbeat data sources.

linux-win-overview-1

Fixing the Auditd Module

You might notice in this environment the Auditbeat Audit category only has one event, which stands out due to the earlier errors you may have also seen in the auditd module.

If you did not have those errors, and if you have more than one event in the Audit beat, then disregard the following instructions.

Take a look at this one event from the auditd module. You can see that it is simply an event saying that it failed to start this module and that an audit process was already running. This is confirmed on the centOS 7 endpoint being used in this guide but may vary depending on distribution.

audtit-failure

Also, remember that in the logs generated to standard out from the -e flag, when this executable was run on the Linux device mentioned that no auditd rules were set.

Back in the centOS7 device running Auditbeat, press ctrl+c to exit the process to terminal, and open up the configuration file.

1
sudo nano /etc/auditbeat/auditbeat.yml

Notice the modules sections, beginning with file integrity. This is the module that scans for changes in files and folder locations that you are concerned with. The values can be changed and customized, and the options for this module can be found in detail on Elastic's website here.

Below, you can see the system module running now with every possible data set, which was not the case on the Windows system due to the reduced compatibility and supported features. More options for the system module can be found here.

file-integrity

All of the modules are configured by default on Linux systems with the exception of the auditd module, which requires you to either write custom rules or uncomment (remove the #) rules that are already in the configuration.

linux-audit-rules

Once you have uncommented the rules, you have eliminated the first error that was recognized for the auditd module during the running of the Auditbeat process. Save the file from nano and exit by pressing ctrl+x, then y then return.

Back at the terminal, you now need to stop the process that is currently running monitoring audit events. This process, known as auditd, is run on the local OS in centOS 7 by default to record audit logs to disk. You have to stop this service, disable it, and optimize the disk space usage by sending the audit events to devel null, so that the auditd module can run in its place.

Run:

1
service auditd status

auditd-status

Confirm that the auditd status is indeed running and is the reason that you cannot run the auditd module from Auditbeat.

Then stop the service and disable it to prevent it from starting automatically at startup.

1
> sudo service auditd stop

stop-auditd

1
sudo systemctl disable auditd

disable-auditd

Note: If you are using a debian-based linux, you may need to run chkconfig auditd off to disable the auditd service permanently.

Now optimize the OS and disk preventing the audit socket from writing logs to disk.

1
sudo systemctl mask systemd-journald-audit.socket

stopped-journeld

With the OS taken care of, you need to also make sure you have your bases covered with the auditd module configuration. Start by showing the audit rules to ensure the rules were properly uncommented and recognized.

1
sudo auditbeat show auditd-rules

auditd-rules

Check the status of the auditd module.

1
sudo auditbeat show auditd-status

auditd-status

Test your configuration file changes.

1
sudo auditbeat -e test config

test-config-linux-ok

And with the okay from Auditbeat itself, it is time to re-run.

1
sudo auditbeat -e

This time, you should not see any errors for auditd, and once you see good events in the monitor category, shift over to kiban and the Elasticserch SIEM overview tab.

Now you can see that there are 73 messages from auditd, which makes more sense from something that should be actively sending events and alerts from the kernel.

73-auditd

Just like that, you have working data from every Auditbeat module for the Elasticsearch SIEM. You can view them in the host tab, network (which includes Auditbeat socket), or in the form of a custom timeline of select events. The SIEM has optimized queries and parsing for these log formats, which can be seen in the search bar.

search-capes

And outside of the SIEM, there are custom visualizations and dashboards for each of the Auditbeat modules and datasets. auditd-dashboard

16