Hostname Tracking in Splunk

Keeping track of the relationship between hostnames and IP addresses over time gives important context during incident response or any other forensic activity. This article will walk through one way to accomplish this in Splunk.

Time-based looks are a great way to track artifacts that change over time. This is just one example of how to leverage them.

The Problem

It may be challenging to look back in your logs and identify what a hostname was for the IP 192.168.24.20 three months ago. We could create a dashboard to correlate the DHCP events (or another data source) around the time of the event and output the hostname at that time. But what if there was another way? A way to automatically output the hostname during that time to the event during search time.

Solution Walk-through

The following details how to accomplish this using time-based lookups in Splunk.

Figure 3: Expected outcome

What we will need

1. A data source that provides mappings from hostname to IP addresses.
   • Examples:
     • CrowdStrike Device logs
     • DHCP logs
     • DNS logs
     • etc…
2. A KVStore lookup.
3. A scheduled search to update the hostname mappings.

Data Source

The data source choice will come down to what is available in your environment. We are seeking a data source that provides accurate IP to Hostname mapping. For this example, I will leverage a built-in external command to query my DNS servers in real time for the hostname of an IP.

This method will put a significant load on your DNS servers in large environments. It is recommended to use an existing data source in Splunk to correlate the hostname.

KVStore lookup

Splunk KVStores are a great way to store content that changes frequently. Compared to CSV lookups, KVStores can handle large amounts of records efficiently and are well-suited for this use case.

KVStores are typically configured from configurations files, but an app like the Splunk App for Lookup File Editing can be used to configure them from the Splunk web interface.

Configuration files

collections.conf

[zts_ip_hostname_tracker_collection]
field.clientip              = string
field.clienthost            = string
field._time                 = time
accelerated_fields.clientip = {"clientip": 1}

In this case we are keeping it simple and only storing the IP, hostname, and the time. Update the stanza’s name, zts_ip_hostname_tracker_collection to the name you would like to use.

transforms.conf

[zts_ip_hostname_tracker]
external_type = kvstore
collection    = zts_ip_hostname_tracker_collection
fields_list   = _key, _time, clientip, clienthost
time_field    = _time

Define the lookup and add the time_field to equal the time field in the collections.conf file. Again, you can change the stanza zts_ip_hostname_tracker to a name you would prefer.

Scheduled Search

To populate the KVStore lookup and to keep it up to date, we need to create a search using our data source of choice.

In my case, I will be using firewall data that I will then be performing a reverse IP lookup against my DNS servers.

Base Search

index=firewall (`zts_local_ip(src)` OR `zts_local_ip(dest)`) _index_earliest=-6m@m _index_latest=-1m@m

zts_local_ip(1): Search macro that expands to my internal IP ranges.

$ip$ IN("10.0.0.0/8","172.16.0.0/12","192.168.0.0/16") 

The _index_earliest/latest is used to look at that event index time rather than the event timestamp.

Define fields and table

| eval 
    local_src=case(cidrmatch("10.0.0.0/8", src), src, cidrmatch("172.16.0.0/12", src), src, cidrmatch("192.168.0.0/16", src), src),
    local_dest=case(cidrmatch("10.0.0.0/8", dest), dest, cidrmatch("172.16.0.0/12", dest), dest, cidrmatch("192.168.0.0/16", dest), dest),
    clientip=mvappend(local_src, local_dest)
| stats max(_time) as _time count by clientip

local_src and local_dest are being defined to collect the internal IP in the event. clientip then combines those fields into one field. The stats command tables each client’s ip with the latest time.

DNS lookup

| lookup dnslookup clientip OUTPUT clienthost
| stats max(_time) as _time by clientip clienthost

dnslookup is a built-in external command that will take in a clientip and output clienthost. This will query the configured DNS servers of the OS. We then pipe the results into stats to get a list of IPs and hostnames by the latest time.

Output to our lookup

| outputlookup append=true zts_ip_hostname_tracker

Using the outputlookup command, we write the results to our KVStore file zts_ip_hostname_tracker. Be sure to update the lookup name to the one you chose. append=true is set to not overwrite the lookup but to add to it each time the search runs.

Full example

index=firewall (`zts_local_ip(src)` OR `zts_local_ip(dest)`) _index_earliest=-6m@m _index_latest=-1m@m
| eval 
    local_src=case(cidrmatch("10.0.0.0/8", src), src, cidrmatch("172.16.0.0/12", src), src, cidrmatch("192.168.0.0/16", src), src),
    local_dest=case(cidrmatch("10.0.0.0/8", dest), dest, cidrmatch("172.16.0.0/12", dest), dest, cidrmatch("192.168.0.0/16", dest), dest),
    clientip=mvappend(local_src, local_dest)
| stats max(_time) as _time count by clientip
| lookup dnslookup clientip OUTPUT clienthost
| stats max(_time) as _time by clientip clienthost
| outputlookup append=true zts_ip_hostname_tracker

Example output

clientip	clienthost	_time
10.0.10.100	myhost34	2023-10-30 12:22:48.605
192.168.24.154	myhost12	2023-10-30 12:10:32.891
192.168.24.20	myhost77	2023-10-30 12:22:49.343

Schedule

Save your search as a Report and run over “All Time” (since we have our _index_earliest/latest set). In my environment, I set this to run every five minutes with the Highest Priority to ensure it always runs when it needs to. Determine the best window for your search to run in your environment.

Optimize

Now that we have our search creating records in our lookup table. We can see that this lookup table will grow exceptionally large very quickly. Here are two strategies for keeping your lookup size from growing too large.

1. Create a retention policy based on time.
   • You can create a secondary search that looks at your lookup table and “cleans” out old records by time. For example, after 90 days, the record will be removed.
2. (Definitely do this) Update the search we just created to write to the lookup only when the hostname has changed.

Optimize scheduled search

All we need to do to write to the lookup table only when there is a change of hostname is to perform an additional lookup and do the check.

| lookup zts_ip_hostname_tracker clientip OUTPUT clienthost as previous_hostname
| where NOT clienthost==previous_hostname

Here, we are just looking up the existing hostname and comparing it to the one queried. The “where” condition will then remove any events with a match, allowing us to write to the lookup only when there is a change or new record.

Full example with change

index=firewall (`zts_local_ip(src)` OR `zts_local_ip(dest)`) _index_earliest=-6m@m _index_latest=-1m@m
| eval 
    local_src=case(cidrmatch("10.0.0.0/8", src), src, cidrmatch("172.16.0.0/12", src), src, cidrmatch("192.168.0.0/16", src), src),
    local_dest=case(cidrmatch("10.0.0.0/8", dest), dest, cidrmatch("172.16.0.0/12", dest), dest, cidrmatch("192.168.0.0/16", dest), dest),
    clientip=mvappend(local_src, local_dest)
| stats max(_time) as _time count by clientip
| lookup dnslookup clientip OUTPUT clienthost
| lookup zts_ip_hostname_tracker_slim clientip OUTPUT clienthost as previous_hostname
| where NOT clienthost==previous_hostname
| stats max(_time) as _time by clientip clienthost
| outputlookup append=true zts_ip_hostname_tracker

This slight change will keep your lookups from growing out of control. However, you will still want to keep an eye on the size of your lookups. If they grow too large, you can begin experiencing performance issues. See the Docs for more information.

Create Auto-lookup

See Splunk Documentation for more information on automatic lookups.

Now that we have our lookup table file populated with hostname information, we can begin applying it to our data for the hostname to be outputted during search time automatically.

props.conf

[bro:conn:json]
LOOKUP-zts_src_ip_hostname_tracker = zts_ip_hostname_tracker clientip AS src OUTPUT clienthost AS src_hostname
LOOKUP-zts_dest_ip_hostname_tracker = zts_ip_hostname_tracker clientip AS dest OUTPUT clienthost AS dest_hostname

In this example, the src and dest fields are being targeted for automatic enrichment with a new field to appear as either src_hostname or dest_hostname. Keep in mind you can change all of the names and even target additional fields. Also, note that this will only work for the bro:conn:json sourcetype.

If you wanted this to apply to all sourcetypes to automatically enrich fields for all data sources, you can change the stanza name to default:

[default]
LOOKUP-zts_src_ip_hostname_tracker = zts_ip_hostname_tracker clientip AS src OUTPUT clienthost AS src_hostname
LOOKUP-zts_dest_ip_hostname_tracker = zts_ip_hostname_tracker clientip AS dest OUTPUT clienthost AS dest_hostname

Conclusion

With just a small amount of setup, you can enrich your data with hostnames that correspond to the timeframe you are looking at. This way, you can search for an event from a few months ago and see what the hostname for that IP was at that time. This will provide great context during investigations and save time and effort to track down a hostname from a specific period.