Page tree

Contents

We have written some scripts that you can use to get your ThreatSTOP block lists.

Download the scripts here.

The script queries the ThreatSTOP DNS server and stores the results in a file that PF can use to build a table. The resulting file is a list of IP addresses in CIDR format. You can then create a rule that will block access to and from the table.

If this is a new device, please allow up to 15 minutes for our systems to be updated.

Prerequisites

Setting up ThreatSTOP for OpenBSD systems requires a connection to the internet from your OpenBSD device, and a pair of Perl libraries and their dependencies. These are available as binary packages for OpenBSD.

Pinging www.threatstop.com with the following command:

ping www.threatstop.com

will verify if you are able to reach the Internet from your OpenBSD device. If you are able to reach the Internet, then the following setups will help meet the ThreatSTOP prerequisites and then install the ThreatSTOP scripts.

To install the modules run the following as root:

export PKG_PATH=http://ftp5.usa.openbsd.org/pub/OpenBSD/`uname -r`/packages/`machine -a`/

pkg_add p5-libwww p5-LWP-Protocol-https wget

This will install the LWP Perl module and any dependencies it defines, as well as wget and its dependencies.

Install

Before running the script it needs to be downloaded and installed. The following command will download the file and install it for you:

wget ftp://ftp.threatstop.com/pub/threatstop-pf.tar.gz && tar xzvf threatstop-pf.tar.gz && cd ./threatstop-pf && cp threatstop.conf.example threatstop.conf && ./install.sh

This will download the ThreatSTOP configuration files, extract them, create a clean copy of the threatstop.conf file, and run the installation script automatically. The script copies the files threatstop-pf.sh, sendlog.sh, loguploadclient.pl, ptr.sed, and apl.sed to /usr/local/sbin. It also copies the file threatstop.conf file to /usr/local/etc.

Configuration

After the installation some modifications to the /usr/etc/threatstop.conf file will be required. Change over to the directory and open the configuration file using the following command:

cd /usr/local/etc

vi threatstop.conf

Modify the file to match the example below.

ThreatSTOP Configuration file

# Final location of the files PF will use for the tables
OUT_DIR="/var/db"

# ThreatSTOP DNS Servers
# You can add or remove entries. Each server must be separated by a space
DNS_SERVERS="192.124.129.42"

# The block and/or allow list

BLOCK_LIST=<block list name>..threatstop.local
ALLOW_LIST=<allow list name>..threatstop.local

# The name of the files pf will use for the tables
BLOCK_FILE="$OUT_DIR/tsblock.txt"
ALLOW_FILE="$OUT_DIR/tsallow.txt"

# The name of the table for PF
BLOCK_TABLE="ThreatSTOP"
ALLOW_TABLE="ThreatSTOP-Allow"

# Automatically reload PF after the files are created
RELOAD_PF="YES"

#
# Options for the sendlog.sh script
#

# Location and name of the PF log file
PFLOG="/var/log/pflog.0"

# Name of the log file to upload. File is created
# and deleted by the sendlog.sh script
logfile="/tmp/pflog.txt"

# URL to upload the logs to
url=https://www.threatstop.com/cgi-bin/logupload.pl

This will set the proper configuration data so that your device can pull policy lists from your ThreatSTOP account.

Testing

Note:

When used, this script must be run from the original installation location for the scripts (that is $HOME/threatstop-pf)

Before running the main script to get your block lists and create the files, we need to make sure that your block lists are ready. We have included the test.sh script that you can run to make sure everything is ready. If the test does not work within an hour of creating your device, please contact support for assistance.

./test.sh

DNS Server 192.124.129.42 is ready.

 

The test was successful. You are now ready to run the threatstop-pf.sh script.

Check Crontab for the added jobs

Verify that two ThreatSTOP jobs have been added to cron (threatstop-pf.sh and sendlog.sh). The command to do this is:

crontab -l

This will list the contents of crontab, and should contain the following entries:

#ThreatSTOP update
36    */2    *    *    *    /usr/local/sbin/threatstop-pf.sh
0     1      *    *    *    /usr/local/sbin/sendlog.sh

If these entries are present we're set to add tables and rules.

Main Script

After making sure the test script works, you can now run the main script, threatstop-pf.sh. The script gets your block lists and creates the files that PF can use to populate a table. The files are saved to /var/db.
In order for PF to load any updates to the ThreatSTOP lists, PF needs to flush and reload the table. In the configuration file, there is an option to reload PF after the block lists have been downloaded. The commands that are run to do this are:

# threatstop-pf.sh

Starting ThreatSTOP v1.14-01 update on Fri Aug 12 07:42:05 PDT 2016


Using DNS Server 64.87.26.147

Processing allow list <allow list name>..threatstop.local

Processing block list <block list name>..threatstop.local

 

Completed getting all the lists

Adding 26424 blocked addresses

Adding 3 allowed addresses

Reloading pf...done

 

Finished ThreatSTOP update at Fri Aug 12 07:42:15 PDT 2016

Run Length: 0 hour(s) 0 minute(s) 10 second(s)

This will create the files tsblock.txt and tsallow.txt in the /var/db directory.

Configure PF

Bridge Configuration

OpenBSD PF can be configured to run as either a router or as a bridge, with some minor modifications to the router setup. To setup a PF bridge a second network interface will need to be added to the network interfaces. To do this:

  1. We'll need to create a bridged pair of network interfaces. To do this:
    1. At the command prompt enter:

       vi /etc/hostname.bridge0

    2. Next we need to add the interfaces, and link them.

      add interface em1
      add interface em2
      up

    3. Save the file using :wq
  2. In the /etc/pf.conf file, use the configuration to the left, but append the following to the front of your configuration:

    PUBLIC_IF=bridge0

    and after the configuration append:

    pass out on bridge0 all
    pass in on $PUBLIC_IF

    Additionally you will need to omit the following line:

    pass out quick on em0 inet from 192.168.160.0/24 to any nat-to em0

  3. Now that the configuration is complete, interface em0 will connect to your management device, em1 and em2 will need to each connect to different networks, such as your trusted and untrusted environments.

Once the files containing the addresses to block are created, you will need to configure PF to use the file as a table. The base of these instructions provide OpenBSD PF as a router, the sidebar to the right explains how to configure OpenBSD as a bridge. In either case in the /etc/pf.conf file, add the table definition:

table <ThreatSTOP> persist file "/var/db/tsblock.txt"
table <ThreatSTOP-Allow> persist file "/var/db/tsallow.txt"

With the table defined, you can create the rules to block traffic to and from the addresses in the table. We recommend that the rules to block the ThreatSTOP table be placed before any rules that allow incoming or outgoing traffic. The "quick" directive tells PF to treat the rule as the last matching rule. Any rules after it are not evaluated.

block drop in log quick from <ThreatSTOP>
block drop out log quick to <ThreatSTOP>
pass out quick on em0 inet from 192.168.160.0/24 to any nat-to em0
pass in quick from <ThreatSTOP-Allow>
pass out quick to <ThreatSTOP-Allow>
set limit { states 20000, frags 5000, src-nodes 10000, tables 50000, table-entries 1000000 }

After you make the changes to the /etc/pf.conf file, PF will need to be reloaded to read the updated configuration:

/sbin/pfctl -f /etc/pf.conf

To update the the policy immediately:

/usr/local/sbin/threatstop-pf.sh

To view the addresses in the table, run the command:

# pfctl -T show -t ThreatSTOP

# pfctl -T show -t ThreatSTOP-Allow

To view the updated rules, run the command:

/sbin/pfctl -s rules

Automating Block List Updates

In order to have the block lists updated hourly the sendlog.sh script we will need to modify the /etc/newsyslog.conf file so that the /var/log/pflog is rotated every night at 12:00 AM. In the newsyslog.conf file, to do this we will need to change:

  • From:

    /var/log/pflog 600 3 * $D0 ZB "pkill -HUP -u root -U root -t - -x pflogd"

  • To:

    /var/log/pflog 600 24 * 1 ZB "pkill -HUP -u root -U root -t - -x pflogd"

This will maintain backup logs for the previous 24 hours, as well as rotate the logs hourly and then upload the most recent log to ThreatSTOP for processing and analysis.

Sending Your Logs

We have a log parsing feature where we can take your firewall log, parse it and compare the source and destination IP addresses to what is in our database. You can then login to our website and see the results. This allows you, and us, to see how effective we are in protecting your network infrastructure.
The log file written by PF is in binary format and must be converted before it is sent to ThreatSTOP. We have included a script that converts the log to plain text and uploads the file to the secure ThreatSTOP website.
The installation script has already configured your system to run the sendlog.sh script every night at 1:00 AM.
If you would like to upload a log now, run the following command as root:

/usr/local/sbin/sendlog.sh
Log file uploaded successfully

Testing and Validation

As a last step, we'll test that threats are actively being blocked. To do this you'll need to place a device behind your OpenBSD PF firewall, and attempt to browse out to our testing page. To do this:

  1. On the OpenBSD firewall verify that the validation record is loaded using the following command:

    pfctl -T show -t ThreatSTOP | grep 64.87.3.133

  2. Place a network device behind the firewall.
  3. Open a web browser and attempt to visit: bad.threatstop.com from the secured device.
    If blocking is working the page will time out, if it is not happening the ThreatSTOP logo will appear.
  4. Confirm that the blocked communication attempt was written to pflog using the following command:

    tcpdump -n -e -ttt -r /var/log/pflog | grep 64.8.3.133

Restore to Previous State

If you decide to return to your pre-ThreatSTOP configuration, you will need to perform the following actions to disable and remove ThreatSTOP from your system:

  1. Stop the VM from updating the firewall by deleting the user crontab:

    crontab -r

  2. #       $OpenBSD: pf.conf,v 1.54 2014/08/23 05:49:42 deraadt Exp $
    #
    # See pf.conf(5) and /etc/examples/pf.conf
    set skip on lo


    block return    # block stateless traffic
    pass            # establish keep-state


    # By default, do not permit remote connections to X11


    block return in on ! lo0 proto tcp to port 6000:6010
    table <ThreatSTOP> persist file "/var/db/tsblock.txt"
    table <ThreatSTOP-Allow> persist file "/var/db/tsallow.txt"


    pass in quick from <ThreatSTOP-Allow>
    pass out quick to <ThreatSTOP-Allow>
    block drop in log quick from <ThreatSTOP>
    block drop out log quick to <ThreatSTOP>

    Remove the ThreatSTOP address groups from the policies using them (or delete the policies completely). This can be done by removing the entries from the /etc/pf.conf file. In the example to the right removing the lines containing pass, block, and ThreatSTOP will remove the policies. While deleting the lines with table, and ThreatSTOP will delete the address groups.

  3. After making the changes to the .pf.conf file you will need to sync the firewall back into place with the command:

    /sbin/pfctl -f /etc/pf.conf

The Files

  • threatstop-pf.sh: The main script. It downloads the block lists and creates the files PF uses to populate a table.
  • loguploadclient.pl: Perl script that uploads the log file
  • sendlog.sh: Converts the PF log file to plain text and calls loguploadclient.pl to upload the script.
  • install.sh: Installation script.
  • test.sh: Script to run a quick test to make sure the block lists are ready to be downloaded.

  • apl.sed: Supporting sed script to parse DNS APL query results.
  • prt.sed: Supporting sed script to parse DNS PTR query results.
  • threatstop.conf.example: Example configuration file.