April 02, 2015

Automatic Malware IP Filters for NfSen


Below are my plain text notes for adding crontab based automatic malware filters in my CentOS based nfsen.

This was done with nfsen 1.3.6p1 and nfdump 1.6.6 -- i have not yet upgraded to any newer versions which may may be different.

Note that this takes into account my setup's file-structure -- yours may differ.

###############################
NFSEN NETFLOW AUTOMATED FILTERS
###############################

###############################
HOW TO
###############################
For each of the following names: "Malware-Domain-List", "Hostile_IPs", "ZeusBotNet_CC" (if you change the names, you will have to change the scripts)
Create new Profile
Group under "malware"
Description "Crontab enabled automatic filter" (and whatever other info you like to add, maybe the URLs from the scripts below)
no start date
no end date
default max size
default expire
1:1 channels
Shadow Profile
Sources: select all the sources you like.
Filter: temporarily use "not any"
[Create]
This will create a "blank" filter for each of your sources.
Now Create the following scripts, mark executable and run-once manually; Afterward, add them to crontab.
note: The *-filter.txt files (created by the gui) should be marked writable.


###############################
 ✓ root@netflow: /usr/local/nfsen/profiles-stat/malware $ find ./ -name "*.sh"
###############################
./Malware-Domain-List/import-list.sh
./Hostile_IPs/import-list.sh
./ZeusBotNet_CC/import-list.sh


###############################
✓ root@netflow: /usr/local/nfsen/profiles-stat/malware $ cat Hostile_IPs/import-list.sh
###############################
#!/bin/bash

export PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin

cd /usr/local/nfsen/profiles-stat/malware/Hostile_IPs

printf "IP in [\n" > temp.txt
wget -qO- http://www.autoshun.org/files/shunlist.csv | tail -n +2 | awk -F, '{print $1}' >> temp.txt
printf "]\n" >> temp.txt

for f in *-filter.txt ; do
   cp temp.txt $f
done

rm temp.txt

#-rw-rw-r--. 1 apache apache *-filter.txt

###############################
 ✓ root@netflow: /usr/local/nfsen/profiles-stat/malware $ cat ./Malware-Domain-List/import-list.sh
###############################
#!/bin/bash

export PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin

cd /usr/local/nfsen/profiles-stat/malware/Malware-Domain-List

printf "IP in [\n" > temp.txt
wget -qO- http://www.malwaredomainlist.com/hostslist/ip.txt >> temp.txt
printf "]\n" >> temp.txt

for f in *-filter.txt ; do
   cp temp.txt $f
done

rm temp.txt

#-rw-rw-r--. 1 apache apache *-filter.txt


###############################
 ✓ root@netflow: /usr/local/nfsen/profiles-stat/malware $ cat ./ZeusBotNet_CC/import-list.sh
###############################
#!/bin/bash

export PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin

cd /usr/local/nfsen/profiles-stat/malware/ZeusBotNet_CC

printf "IP in [\n" > temp.txt
wget --no-check-certificate -qO- https://zeustracker.abuse.ch/blocklist.php?download=badips | tail -n +7  >> temp.txt
printf "]\n" >> temp.txt

for f in *-filter.txt ; do
   cp temp.txt $f
done

rm temp.txt

#-rw-rw-r--. 1 apache apache *-filter.txt


###############################
 ✓ root@netflow: /usr/local/nfsen/profiles-stat/malware $ crontab -l | tail -n 4
 ###############################
0 * * * * /usr/local/nfsen/profiles-stat/malware/Hostile_IPs/import-list.sh
0 * * * * /usr/local/nfsen/profiles-stat/malware/Malware-Domain-List/import-list.sh
0 * * * * /usr/local/nfsen/profiles-stat/malware/ZeusBotNet_CC/import-list.sh




---
As Always, Good Luck!

Please comment or tip me or use any/all of my affiliate links; Thank YOU!

You can thank me with bitcoin.    



8 comments:

  1. Thank you for these. They work great. However I had to modify the Malware-Domain-List script. They are putting ^M characters at the end of each IP in the ip.txt file. I suggest modifying the wget line to:

    wget -qO- http://www.malwaredomainlist.com/hostslist/ip.txt | sed -e 's/^M//g' >> temp.txt

    To get the special character, use in vi.

    Again, thanks for posting these filters.

    ReplyDelete
    Replies
    1. Edit: use ctrl+v ctrl+m in vi. Post stripped that out for some reason.

      Delete
    2. Interesting, I would have never noticed. Thank you!

      Delete
    3. I only noticed because of errors in syslog. Thanks again. Cheers!

      Delete
    4. Mine did not have the ^M. I verified via the sed replace command, then compared their md5 sums. Each were identical. However, i thank you still for anyone else affected by this.

      Delete
    5. Whoa! That's super interesting. I'll download it on a couple of other servers and take a look. Also ^M is equivalent to \r\n, but I'm sure you already knew that.

      Delete
  2. Tested in OpenBSD, same return character in that file for me.

    # wget http://www.malwaredomainlist.com/hostslist/ip.txt
    --2016-04-07 08:56:33-- http://www.malwaredomainlist.com/hostslist/ip.txt
    Resolving www.malwaredomainlist.com (www.malwaredomainlist.com)... 143.215.130.61
    Connecting to www.malwaredomainlist.com (www.malwaredomainlist.com)|143.215.130.61|:80... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 22420 (22K) [text/plain]
    Saving to: 'ip.txt.1'

    ip.txt.1 100%[==========================================================================================================================================>] 21.89K --.-KB/s in 0.07s

    2016-04-07 08:56:34 (310 KB/s) - 'ip.txt.1' saved [22420/22420]

    # cat ip.txt | sed -e 's/^M//g' > ip2.txt
    # md5 ip.txt > ip.md5 && md5 ip2.txt > ip2.md5
    # diff ip.md5 ip2.md5
    1c1
    < MD5 (ip.txt) = afa5d64e01ddcf49062db4622af85be2
    ---
    > MD5 (ip2.txt) = 2d88abb5f4b2bdd82ae3d58ad9f23752

    When typing sed in terminal, it's not ^M (caret M), it's ctrl+v/ctrl+m. Could that be our difference there?

    ReplyDelete
  3. nope, same result. CentOS 6.7 no ctrl-v ctrl-m character on mine.

    ReplyDelete

Comments, Suggestions or "Thank you's" Invited! If you have used this info in any way, please comment below and link/link-back to your project (if applicable). Please Share.
I accept Bitcoin tips of ANY amount to: 1GS3XWJCTWU7fnM4vfzerrVAxmnMFnhysL
I accept Litecoin tips of ANY amount to: LTBvVxRdv2Lz9T41UzqNrAVVNw4wz3kKYk