ThreatBlockr 2.0 Syslog Export Configuration and Formats

This document describes the capabilities and formatting of the syslog export features built into Bandura Cyber’s ThreatBlockr 2.0 software, which powers our ThreatBlockr devices. This document was written to be compatible with our official release “version 1” syslog export functionality, which was first released as Bandura Cyber ThreatBlockr 2.0 software build 48. So, if you are on an older build, update your Bandura Cyber ThreatBlockr device first before continuing. As of the latest update to this document, build 59 is the recommended build.

 

Earlier builds will likely not match the descriptions in this document, so it is important to update your software to ensure compatibility. It is possible that later builds will still match the descriptions in this document, but if you have a later build, you should consult the software release notes which can be viewed at our Global Management Center (GMC) at https://gmc.banduracyber.com, to be sure that nothing has changed, or ask our Customer Success team if a newer version of this document is available.

Why Syslog Exports?

 

Syslog exports are an industry-standard and time-proven way of exporting data in a concise, standards-based manner. Bandura Cyber’s syslog export format is compliant to RFC-5424. This ensures seamless integration alongside any number of external tools, including popular security information and event management (SIEM) tools, such as Splunk and IBM QRadar, as well as popular data analytics tools like Gravwell.

 

This document is not written with any particular SIEM tool in mind. Instead, this document is meant to focus on the comprehensive data contained in our syslog exports, enabling our logs to be parsed in an intelligent way by any tool that can ingest RFC-compliant syslog exports.

Configuring the Bandura Cyber ThreatBlockr’s Syslog Export

 

To configure the syslog export features of your ThreatBlockr device, the general ordered flow is shown in the following image:

 

 

After logging into your Bandura Cyber ThreatBlockr's secure web interface, select Logging > External Syslog (step 1). Click the green/white plus sign in the top right (step 2) to add a new syslog server. You can add as many of these as you want - you aren’t limited to just one. This can be useful if you purposely want to send different log information to different business systems.

 

Choose the IP address (step 3) of your target system, and generally, you’ll use 514 for the UDP port (step 4), which is the typical listening port most systems use for syslog ingest. Feel free to change it if you need to.

 

You can then select the logs that you want to send - either a subset or all of them (steps 5 and 6). In the example above, we have configured things to send “everything” by enabling the IPv4, DNS, Syslog, and Audit checkboxes, and for the packet logs, we’ve set the Resource Group, Verdict, and Direction to All. In a subsequent section, we’ll describe the various configuration options in significantly more detail.

 

When you’ve configured it to your liking, click create (step 7), and at that point, your ThreatBlockr will immediately start sending the logs you’ve chosen as they are generated from that point forward in real-time, to the system of your choosing.

 

Multiple Targets

 

The ThreatBlockr supports as many external syslog targets as you’d like. That is, you can selectively filter the logs of your choosing and send them to one particular SIEM or business tool, and send the same or different filtered configuration to a different SIEM or business tool. This can be quite useful in situations where you need to be concerned with ingest rates or cost implications of data ingest on various target systems. This capability is also useful when you have multiple external critical business systems that need live, real-time access to our logs.

RFC-5424 Compliant Headers

 

All of our export logs are RFC-5424 compliant. We use an intelligent comma-separated key=value scheme after the RFC-compliant header, to simplify ingest and parsing by target SIEM and business systems.

 

Our log messages are compliant to RFC-5424 and resemble:

 

<nn>V YYYY-MM-DDTHH:MM:SS.sssZ hostname bctifw - type_log - data

 

The leading <nn> is defined in RFC-5424 and represents the priority value. Generally, many syslog parsers will automatically parse and store this field, even though its value is limited in our application given the intelligent way we’ve done our data modeling, which we’ll discuss in more detail soon.

 

The immediately following V is a version number. Currently, all of our logs are built-out as version 1, with our official version 1 being first released with software build 48. Previous releases all used version 1 for purposes of RFC compliance, but note that build 48 is the first officially support version 1 and may differ from prior releases. Subsequent releases (post build 48) will maintain adherence to version compatibility.

 

The UTC timestamp format shown is standardized and well-known; most SIEM and related business tools are able to natively parse it. A populated actual value might be: 2020-08-14T14:37:11.650Z.

 

The hostname quantity will be replaced by the hostname of the source ThreatBlockr device.. Note that setting a unique hostname on each of your ThreatBlockr devices will allow for simple filtering and correlation across devices. To change a ThreatBlockr hostname for uniqueness, you can set the hostname on your ThreatBlockr from Settings > General, like this:

 

 

The value bctifw is a static value that we use to uniquely identify our logs in the APP-NAME field of the RFC 5424 header. Using that field as part of any specific query might simplify the query and improve overall performance as a top-level filter.

 

The type_log quantity provides for a simple text filter that can be used in your target system to know that you are operating on a particular log type. Currently, the four possible values are packet_log, dns_log, system_log and audit_log.

 

And last but not least, the trailing comma-separated key=value data is dependent on the type_log quantity. The various formats and log configurations are described in detail in the remainder of this document.

Packet Logs - Configuration

 

The packet log exports are generally the logs that you will be most interested in keeping track of. They track all connections and pseudo-connection traffic. They provide a simple, centralized way to get ground-truth on whether a connection was allowed or denied, including why those decisions were made, with full attribution to the various supported geo-mapping, ASNs, threat lists, denied lists, allowed lists, and so on, in comprehensive fashion.

 

Regardless of whether a connection is tagged as allowed or denied, we always perform a full security analysis with the threat intelligence information made available from our cloud-based GMC software as a service. That means that even when a connection might be allowed because it’s IP address was manually placed on an allowed list, the syslog entry will still contain detailed attribution information about the country, ASN, and its presence on any or all of the configured lists at the time the decision was made.

 

That capability provides a fantastic way to do security triage in true real time, across any number of ThreatBlockr devices, by configuring them to send their syslog export data to suitable external systems of your choosing, whether those are on-premise systems or cloud-based systems.

 

The IPv4 packet logs can get quite chatty, and are responsible for the vast majority of the generated data. Because of this, we’ve added powerful filtering features to the packet log export configurator as briefly described in the previous section. We’ll talk a bit more about them below, in conjunction with corresponding imagery:

 

 

Resource Group refers to either the collection of All of the resource groups on a given ThreatBlockr device, or you can isolate them. For example, if you have a DEFAULT OUTBOUND resource group configured and you want to send just the IPv4 connection logs associated to that DEFAULT OUTBOUND to a particular SIEM, you can click in the Resource Group field and select that entry from a dropdown that will appear, and might resemble:

 

 

 

Verdict selects whether you want to export allowed (ALLOW) connections, denied (DENY) connections, or both (All) and is also selectable via a simple dropdown:

 

 

Direction selects the connection direction that you wish to export, which is either inbound (IN) , outbound (OUT), or both (All), selectable by its own dropdown:

 

 

The collection of those three things - Resource Group, Verdict, and Direction - are combined with boolean AND logic. That is, in the following example, only outbound connections that have been denied with respect to the DEFAULT OUTBOUND policy would be exported to the IPv4 packet log:

 

 

But let’s say that, in addition to exporting information about your policy-specific outbound blocks, you also wanted to log, to that same target system (for, perhaps, correlation and cybersecurity analysis), any connection that happened to also be registered on a threat list, irrespective of whether or not it was filtered by the AND logic that we just configured?  We can achieve that by leveraging the second layer of filters which represent a boolean logic OR section:

 

 

Checking one or more of the Denied, Allowed, and Threat list checkboxes in that boolean logic section will result in any IPv4 log that is associated with the selected list types to be exported as well, independent of the boolean AND clause from the first section. Note that a log is exported only once, so if a particular log matches both sections it will still only appear in your target system one time.

 

For scenarios where it isn’t feasible to send everything to a given SIEM, this layered scheme provides a best-practices approach to maximizing the use of a target SIEM system that may be limited by its ingest rate and/or cost considerations.

 

As a final example, here is one typical configuration we often see, where any blocks (independent of resource group and direction) are conveyed, plus anything that was allowed that happened to be associated to one or more of the list types:

 

 

Effectively, in that example, everything is exported except for standard allowed traffic not appearing on any list.

Packet Logs - Format Descriptions

 

Following the RFC-5424 compliant header, the data section of our packet logs contain rich information about both allowed and denied connections and pseudo-connections. Before we describe the detailed data formats, we’ll look at a few real-life examples.

 

Here’s a log that details that the device allowed a UDP request to Google’s well known secondary DNS server at 8.8.4.4 on port 53. Note that the list scenario here has been contrived to show all possible allowed and denied list variants for both active and inactive lists (but no threat list variants):

 

<30>1 2020-08-18T00:51:59.294Z ThreatBlockr1 bctifw - packet_log - action=allowed, direction=outbound, group="Local Office Outbound", proto=UDP, country="UNITED STATES", as_num=15169, as_name="Google Inc.", reason=allowedlist, src=10.0.0.95, dst=8.8.4.4, src_port=39113, dst_port=53, dl_active="Manual Denied List", dl_inactive="Manual Denied List Inactive", al_active="Local Office", al_inactive="Local Office Inactive,Bandura Curated DNS"

 

Here’s a separate example of a log that was denied because it was on a Proofpoint threat list, categorized as a known-malicious proxy server, and its threat score of 92 was above the associated policy’s configured threshold of 90, which resulted in the connection being blocked. It did not show up on any other list:

 

<29>1 2020-08-18T00:57:12.401Z ThreatBlockr1 bctifw - packet_log - action=denied, direction=outbound, group="Local Office Outbound", proto=TCP, country="NETHERLANDS", as_num=14061, as_name="Digital Ocean, Inc.", reason=threatlist, src=10.0.0.95, dst=198.211.120.59, src_port=58802, dst_port=443, flags="SYN", tl="Proofpoint", tl_category="PROXY", tl_threshold="90", tl_score="92"

 

Note that fields that have no bearing on a particular connection are not populated. For example, if no denied lists are associated with an entry, then keys relating to denied lists are not created for that entry. Put another way, parsers should not expect any given key to be populated. Instead, if a key is not present, it simply did not apply for that particular record. This keeps the data transfer sizes minimized, which is important since many SIEMs charge on amount of data ingested, or related factors.

 

It is critical that your parsing engines operate based on the named values and not their positions, as key=value position order is not guaranteed from software release to release.

 

The following table describes the various packet log key value pair constructs.

 

 

packet_log Key	Description
action	This represents the connection/packet verdict. If this reads allowed, the connection was allowed, and if it reads denied, it will have been blocked.
direction	This represents the determined direction of the initial connection (or pseudo-connection, in the case of connectionless protocols such as UDP). The value will be either inbound or outbound. Note that this is the direction of the connection and not necessarily the packet orientation. That is, a connection is outbound if a protected IP initiates the conversation. It is inbound if a protected IP is the target of an external initiator. It’s similar to a phone call: although I might call you on the phone, meaning that it’s an outbound connection from my perspective. During that connection, there are both inbound and outbound packets, regardless of the established and reported connection direction, which is always set for the duration of that particular conversation.
lsh	This represents whether or not loose state handling was employed in determining the connection direction. Generally, if we see a pure SYN packet, we can definitively determine the connection direction. If, on the other hand, we do not see the originating SYN packet, we use a loose packet handling scheme to predict the packet direction. This can happen when a unit is first installed, or if it is rebooted. In the event that we used such a scheme, this key will appear in the log and will be set to true.
group	This is the resource group responsible for the verdict. It is enclosed in double quotes.
proto	This is the identified protocol as determined from the packet’s header information. Most often it is TCP or UDP, but the full gamut of layer 2 and layer 3 protocols are observed by the device. Generally, all layer 2 traffic that does not encapsulate an IPv4 packet is always bridged through, whereas the IPv4 traffic is evaluated against our always-on real-time country, ASN, and threat/denied/allowed list intelligence.
country	This is the country attributable to the external endpoint IP address. This information comes from our always-up-to-date geolocation intelligence for all IPs across the globe. It is enclosed in double quotes.
as_num	This is the unique asynchronous subscriber number (ASN) id as determined from our always-up-to-date ASN intelligence for all networks across the globe. Normally this would only be used to cross-index into a separate SIEM database for advanced reporting. It is generally ignored by a parser in favor of human-readable as_name, as noted below.
as_name	This is the ASN’s physical name. An example might be “Google Inc.” or “CloudFlare, Inc.” or “Orange S.A.” and so on. It is enclosed in double quotes.
reason	The reason key provides the primary reason for the verdict. Values can be allowedlist, asn, country, deniedlist, policy, and threatlist. For details about the decision making process, the reader is referred to the ThreatBlockr 2.0 flowchart in the separate document Bandura Cyber - Policy Enforcement and our user manuals.
src	This is the source IP address from the perspective of the connection orientation. For example, if a protected IP initiates an outbound connection to a Google server, then the protected IP is the source IP. If, on the other hand, an external IP initiates a connection to a protected IP, then the connection is inbound, and the source IP is the external IP.
dst	This is the destination IP address from the perspective of the connection orientation. For example, if a protected IP initiates an outbound connection to a Google server, then the Google server is the destination. If, on the other hand, an external IP initiates a connection to a protected IP, then the connection is inbound, and the destination IP is the protected IP.
src_port	This is the source port associated with the source IP of a TCP connection or UDP pseudo-connection.
dst_port	This is the destination port associated with the destination IP of a TCP connection or UDP pseudo-connection.
flags	For TCP connections, this key lists the associated TCP flags for the particular connection/packet used for evaluation. It is always enclosed in double quotes (since multiple comma-separated TCP flags can be associated to one log, such as perhaps “RST,ACK”).
tl	Displays, as a double quoted comma separated value list, any and all threat lists associated with the IP address(es) associated with the connection.
tl_category	This lists all associated threat list categories for the packet as a double quoted comma-separated list. It is in lockstep-order to the values in the tl_threshold key.
tl_score	This lists all associated actual scores as a double quoted comma-separated list, where the order is identical to the ordering of the tl_category key for simple parsing correlation. Note that it is common for the scores to be the same for multiple category entries.
tl_threshold	This lists all associated threshold scores as a double quoted comma-separated list, where the order is identical to the ordering of the tl_category key for simple parsing correlation.
dl_active	Displays, as a double quoted comma separated value list, any and all active (enabled) denied lists associated with the resource group and policy responsible for the verdict.
dl_inactive	Displays, as a double quoted comma separated value list, any and all inactive (disabled) denied lists associated with the resource group and policy responsible for the verdict.
al_active	Displays, as a double quoted comma separated value list, any and all active (enabled) allowed lists associated with the resource group and policy responsible for the verdict.
al_inactive	Displays, as a double quoted comma separated value list, any and all inactive (disabled) allowed lists associated with the resource group and policy responsible for the verdict.

 

DNS Logs - Format Descriptions

 

Following the RFC-5424 compliant header, the data section of our DNS logs contain rich information about outbound unencrypted DNS requests and whether they were allowed or blocked with respect to port 53 activity.

 

Here’s an example unencrypted DNS request made to an internal gateway, attempting to resolve www.google.com, which was allowed:

 

<30>1 2020-08-18T00:53:17.431Z ThreatBlockr1 bctifw - dns_log - action=allowed, proto=UDP, reason=policy, src=10.0.0.95, dst=10.0.0.1, domain=www.google.com

 

Here’s another example of an unencrypted DNS request against Google’s primary DNS server 8.8.8.8 also attempting to resolve google.com that was also allowed:

 

<30>1 2020-08-18T00:28:53.648Z ThreatBlockr1 bctifw - dns_log - action=allowed, proto=UDP, reason=policy, src=10.0.0.95, dst=8.8.8.8, domain=google.com

 

In this next example, an unencrypted DNS request that was attempted against Google’s primary DNS server 8.8.8.8, but the request was intercepted and denied by the Bandura Cyber ThreatBlockr since the domain requested was on a manually crafted denied list:

 

<29>1 2020-08-18T00:28:53.959Z ThreatBlockr1 bctifw - dns_log - action=denied, proto=UDP, reason=deniedlist, src=10.0.0.95, dst=8.8.8.8, domain=amazonfoo.cn, dl_active="CN amazon"

 

Now we’ll look at an example of an unencrypted DNS request to resolve banduracyber.com, which was on an allowed domain list that was enabled and also a separate allowed list that had not been enabled (and so it reports them individually as being present on both of the lists):

 

<30>1 2020-08-18T00:28:54.119Z ThreatBlockr1 bctifw - dns_log - action=allowed, proto=UDP, reason=allowedlist, src=10.0.0.95, dst=8.8.8.8, domain=banduracyber.com, al_active="Bandura Corporate", al_inactive="Bandura Corporate Inactive"

 

Here’s one final example of another domain lookup that was denied; this one relates to a fake coronavirus site that has been associated to cybercrime as populated by multiple world-class DomainTools denied lists:

 

<29>1 2020-08-18T00:28:54.284Z ThreatBlockr1 bctifw - dns_log - action=denied, proto=UDP, reason=deniedlist, src=10.0.0.95, dst=8.8.8.8, domain=100bellscoronaviruspandemiclockdown.golf, dl_active="COVID-19- DomainTools- 70,DomainTools,COVID-19- DomainTools- 99"

 

Note that fields that have no bearing on a particular connection are not populated. For example, if no denied lists are associated with an entry, then keys relating to denied lists are not created for that entry. Put another way, parsers should not expect any given key to be populated. Instead, if a key is not present, it simply did not apply for that particular record. This keeps the data transfer sizes minimized, which is important since many SIEMs charge on amount of data ingested, or related factors.

 

It is critical that your parsing engines operate based on the named values and not their positions, as key=value position order is not guaranteed from software release to release.

 

The following table describes the various packet log key value pair constructs.

 

 

dns_log Key	Description
action	This represents the verdict. If this reads allowed, the outbound unencrypted DNS request was allowed, and if it reads denied, it will have been blocked. A blocked DNS request will result in the requester not receiving a mapped numeric IP address for the requested domain.
proto	This is the identified protocol as determined from the packet’s header information. Currently, this value will always show as UDP, since we only analyze DNS traffic riding over UDP on port 53.
reason	The reason key provides the primary reason for the verdict. Values can be allowedlist, deniedlist, or policy.
src	This is the source (requesting) protected IP address from the perspective of the unencrypted DNS request.
dst	This is the external destination IP address from the perspective of the unencrypted DNS request. For example, when a protected IP initiates an unencrypted DNS request to a Google DNS server, then the Google DNS server is the destination.
domain	The domain requested for resolution as part of the unencrypted DNS request.
dl_active	Displays, as a double quoted comma separated value list, any and all active (enabled) denied domain lists.
dl_inactive	Displays, as a double quoted comma separated value list, any and all inactive (disabled) denied domain lists.
al_active	Displays, as a double quoted comma separated value list, any and all active (enabled) allowed domain lists.
al_inactive	Displays, as a double quoted comma separated value list, any and all inactive (disabled) allowed domain lists.

 

System Logs and Audit Logs - Format Descriptions

 

Following the RFC-5424 compliant header, the data sections of our System logs and Audit logs contain information mostly useful for our Customer Success team when troubleshooting low-level problems or configuration issues. The formatting for the data section is generally free-form, and specific to internal system components. We recommend that if a SIEM is storing our system and audit log types, to store the data sections of those types as free-form text strings.

 

Generally, you do not need to be concerned with messages that show up as system or audit logs unless our Customer Success team instructs you to investigate them during a troubleshooting session.