Below are my plain text notes for adding crontab based automatic malware filters in my CentOS based nfsen.
This was done with nfsen 1.3.6p1 and nfdump 1.6.6 -- i have not yet upgraded to any newer versions which may may be different.
Note that this takes into account my setup's file-structure -- yours may differ.
###############################
NFSEN NETFLOW AUTOMATED FILTERS
###############################
###############################
HOW TO
###############################
For each of the following names: "Malware-Domain-List", "Hostile_IPs", "ZeusBotNet_CC" (if you change the names, you will have to change the scripts)
Create new Profile
Group under "malware"
Description "Crontab enabled automatic filter" (and whatever other info you like to add, maybe the URLs from the scripts below)
no start date
no end date
default max size
default expire
1:1 channels
Shadow Profile
Sources: select all the sources you like.
Filter: temporarily use "not any"
[Create]
This will create a "blank" filter for each of your sources.
Now Create the following scripts, mark executable and run-once manually; Afterward, add them to crontab.
note: The *-filter.txt files (created by the gui) should be marked writable.
###############################
✓ root@netflow: /usr/local/nfsen/profiles-stat/malware $ find ./ -name "*.sh"
###############################
./Malware-Domain-List/import-list.sh
./Hostile_IPs/import-list.sh
./ZeusBotNet_CC/import-list.sh
###############################
✓ root@netflow: /usr/local/nfsen/profiles-stat/malware $ cat Hostile_IPs/import-list.sh
###############################
#!/bin/bash
export PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
cd /usr/local/nfsen/profiles-stat/malware/Hostile_IPs
printf "IP in [\n" > temp.txt
wget -qO- http://www.autoshun.org/files/shunlist.csv | tail -n +2 | awk -F, '{print $1}' >> temp.txt
printf "]\n" >> temp.txt
for f in *-filter.txt ; do
cp temp.txt $f
done
rm temp.txt
#-rw-rw-r--. 1 apache apache *-filter.txt
###############################
✓ root@netflow: /usr/local/nfsen/profiles-stat/malware $ cat ./Malware-Domain-List/import-list.sh
###############################
#!/bin/bash
export PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
cd /usr/local/nfsen/profiles-stat/malware/Malware-Domain-List
printf "IP in [\n" > temp.txt
wget -qO- http://www.malwaredomainlist.com/hostslist/ip.txt >> temp.txt
printf "]\n" >> temp.txt
for f in *-filter.txt ; do
cp temp.txt $f
done
rm temp.txt
#-rw-rw-r--. 1 apache apache *-filter.txt
###############################
✓ root@netflow: /usr/local/nfsen/profiles-stat/malware $ cat ./ZeusBotNet_CC/import-list.sh
###############################
#!/bin/bash
export PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
cd /usr/local/nfsen/profiles-stat/malware/ZeusBotNet_CC
printf "IP in [\n" > temp.txt
wget --no-check-certificate -qO- https://zeustracker.abuse.ch/blocklist.php?download=badips | tail -n +7 >> temp.txt
printf "]\n" >> temp.txt
for f in *-filter.txt ; do
cp temp.txt $f
done
rm temp.txt
#-rw-rw-r--. 1 apache apache *-filter.txt
###############################
✓ root@netflow: /usr/local/nfsen/profiles-stat/malware $ crontab -l | tail -n 4
###############################
0 * * * * /usr/local/nfsen/profiles-stat/malware/Hostile_IPs/import-list.sh
0 * * * * /usr/local/nfsen/profiles-stat/malware/Malware-Domain-List/import-list.sh
0 * * * * /usr/local/nfsen/profiles-stat/malware/ZeusBotNet_CC/import-list.sh
---
As Always, Good Luck!
Please comment or tip me or use any/all of my affiliate links; Thank YOU!
You can thank me with bitcoin.
Thank you for these. They work great. However I had to modify the Malware-Domain-List script. They are putting ^M characters at the end of each IP in the ip.txt file. I suggest modifying the wget line to:
ReplyDeletewget -qO- http://www.malwaredomainlist.com/hostslist/ip.txt | sed -e 's/^M//g' >> temp.txt
To get the special character, use in vi.
Again, thanks for posting these filters.
Edit: use ctrl+v ctrl+m in vi. Post stripped that out for some reason.
DeleteInteresting, I would have never noticed. Thank you!
DeleteI only noticed because of errors in syslog. Thanks again. Cheers!
DeleteMine did not have the ^M. I verified via the sed replace command, then compared their md5 sums. Each were identical. However, i thank you still for anyone else affected by this.
DeleteWhoa! That's super interesting. I'll download it on a couple of other servers and take a look. Also ^M is equivalent to \r\n, but I'm sure you already knew that.
DeleteTested in OpenBSD, same return character in that file for me.
ReplyDelete# wget http://www.malwaredomainlist.com/hostslist/ip.txt
--2016-04-07 08:56:33-- http://www.malwaredomainlist.com/hostslist/ip.txt
Resolving www.malwaredomainlist.com (www.malwaredomainlist.com)... 143.215.130.61
Connecting to www.malwaredomainlist.com (www.malwaredomainlist.com)|143.215.130.61|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 22420 (22K) [text/plain]
Saving to: 'ip.txt.1'
ip.txt.1 100%[==========================================================================================================================================>] 21.89K --.-KB/s in 0.07s
2016-04-07 08:56:34 (310 KB/s) - 'ip.txt.1' saved [22420/22420]
# cat ip.txt | sed -e 's/^M//g' > ip2.txt
# md5 ip.txt > ip.md5 && md5 ip2.txt > ip2.md5
# diff ip.md5 ip2.md5
1c1
< MD5 (ip.txt) = afa5d64e01ddcf49062db4622af85be2
---
> MD5 (ip2.txt) = 2d88abb5f4b2bdd82ae3d58ad9f23752
When typing sed in terminal, it's not ^M (caret M), it's ctrl+v/ctrl+m. Could that be our difference there?
nope, same result. CentOS 6.7 no ctrl-v ctrl-m character on mine.
ReplyDelete