Showing posts with label sflow. Show all posts
Showing posts with label sflow. Show all posts

April 02, 2015

Automatic Malware IP Filters for NfSen


Below are my plain text notes for adding crontab based automatic malware filters in my CentOS based nfsen.

This was done with nfsen 1.3.6p1 and nfdump 1.6.6 -- i have not yet upgraded to any newer versions which may may be different.

Note that this takes into account my setup's file-structure -- yours may differ.

###############################
NFSEN NETFLOW AUTOMATED FILTERS
###############################

###############################
HOW TO
###############################
For each of the following names: "Malware-Domain-List", "Hostile_IPs", "ZeusBotNet_CC" (if you change the names, you will have to change the scripts)
Create new Profile
Group under "malware"
Description "Crontab enabled automatic filter" (and whatever other info you like to add, maybe the URLs from the scripts below)
no start date
no end date
default max size
default expire
1:1 channels
Shadow Profile
Sources: select all the sources you like.
Filter: temporarily use "not any"
[Create]
This will create a "blank" filter for each of your sources.
Now Create the following scripts, mark executable and run-once manually; Afterward, add them to crontab.
note: The *-filter.txt files (created by the gui) should be marked writable.


###############################
 ✓ root@netflow: /usr/local/nfsen/profiles-stat/malware $ find ./ -name "*.sh"
###############################
./Malware-Domain-List/import-list.sh
./Hostile_IPs/import-list.sh
./ZeusBotNet_CC/import-list.sh


###############################
✓ root@netflow: /usr/local/nfsen/profiles-stat/malware $ cat Hostile_IPs/import-list.sh
###############################
#!/bin/bash

export PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin

cd /usr/local/nfsen/profiles-stat/malware/Hostile_IPs

printf "IP in [\n" > temp.txt
wget -qO- http://www.autoshun.org/files/shunlist.csv | tail -n +2 | awk -F, '{print $1}' >> temp.txt
printf "]\n" >> temp.txt

for f in *-filter.txt ; do
   cp temp.txt $f
done

rm temp.txt

#-rw-rw-r--. 1 apache apache *-filter.txt

###############################
 ✓ root@netflow: /usr/local/nfsen/profiles-stat/malware $ cat ./Malware-Domain-List/import-list.sh
###############################
#!/bin/bash

export PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin

cd /usr/local/nfsen/profiles-stat/malware/Malware-Domain-List

printf "IP in [\n" > temp.txt
wget -qO- http://www.malwaredomainlist.com/hostslist/ip.txt >> temp.txt
printf "]\n" >> temp.txt

for f in *-filter.txt ; do
   cp temp.txt $f
done

rm temp.txt

#-rw-rw-r--. 1 apache apache *-filter.txt


###############################
 ✓ root@netflow: /usr/local/nfsen/profiles-stat/malware $ cat ./ZeusBotNet_CC/import-list.sh
###############################
#!/bin/bash

export PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin

cd /usr/local/nfsen/profiles-stat/malware/ZeusBotNet_CC

printf "IP in [\n" > temp.txt
wget --no-check-certificate -qO- https://zeustracker.abuse.ch/blocklist.php?download=badips | tail -n +7  >> temp.txt
printf "]\n" >> temp.txt

for f in *-filter.txt ; do
   cp temp.txt $f
done

rm temp.txt

#-rw-rw-r--. 1 apache apache *-filter.txt


###############################
 ✓ root@netflow: /usr/local/nfsen/profiles-stat/malware $ crontab -l | tail -n 4
 ###############################
0 * * * * /usr/local/nfsen/profiles-stat/malware/Hostile_IPs/import-list.sh
0 * * * * /usr/local/nfsen/profiles-stat/malware/Malware-Domain-List/import-list.sh
0 * * * * /usr/local/nfsen/profiles-stat/malware/ZeusBotNet_CC/import-list.sh




---
As Always, Good Luck!

Please comment or tip me or use any/all of my affiliate links; Thank YOU!

You can thank me with bitcoin.    



May 30, 2013

Install nfsight plugin for nfsen on CentOS 6


Install nfsight plugin for nfsen netflow/sflow collector/analyser on CentOS 6 (and probably RedHat 6)

This was done with nfsen 1.3.6p1 and nfdump 1.6.6 -- i have not yet upgraded to any newer versions which may may be different.

This was one of the most difficult installations I've done to date.  I had attempted to do so at least twice if not thrice without success.  The key was to know about the chgrp command which I had not used before.  Also mysql was not something I was overly familiar with.  I attempted this task knowing I had failed in the past, but I also knew I had gained more Linux experience since those attempts.  I was very excited to succeed this time.

Here are my notes, because quite frankly I have not found any hints online other than the official installation guide which is bare minimum.

Please comment, especially if you find errors or know better solutions.  Enjoy.

Edit: newer nfsight version available: nfsight-beta-20140905.tgz ; just replace references.

######################################################
### Install nfsight plugin for nfsen on CentOS 6.4 ###
### http://sourceforge.net/p/nfsight/home/Nfsight/ ###
### Prerequisite: nfsen/nfdump already operational ###
######################################################

#################################################################
### Your nfsen and webserver directories may indeed be different.
### Apply settings as they pertain to your directory structure.
###
### My server's configuration:
### nfsen installed to /usr/local/nfsen
### nfsen website installed to /var/www/html/nfsen
### website owner=root, group=apache
###
### All commands performed as root.
#################################################################

### install prerequisites
yum install mysql mysql-server perl-DBI perl-DBD-MySQL php-mysql

### download and untar nfsight v.20130323
cd ~
wget http://sourceforge.net/projects/nfsight/files/nfsight-beta-20130323.tgz/download
tar xvfz nfsight-beta-20130323.tgz
cd ~/nfsight-beta-20130323

### following http://sourceforge.net/p/nfsight/wiki/Installation/

### copy nfsight.pm to plugins directory
cp ~/nfsight-beta-20130323/backend/nfsight.pm /usr/local/nfsen/plugins/

### make nfsight data directory and set rights
mkdir /var/www/html/nfsen/plugins/nfsight
chgrp -R apache /var/www/html/nfsen/plugins/nfsight

### make nfsight website directory
mkdir /var/www/html/nfsen/nfsight

### copy frontend to nfsight website directory
cp -R ~/nfsight-beta-20130323/frontend/* /var/www/html/nfsen/nfsight/

### set rights to nfsight website
chgrp -R apache /var/www/html/nfsen/nfsight/

### unsure if necessary - set write permissions for apache
chmod g+w /var/www/html/nfsen/nfsight/
chmod g+w /var/www/html/nfsen/plugins/nfsight
### if nothing else apache must write config.php
### touch /var/www/html/nfsen/nfsight/config.php
### chmod g+w /var/www/html/nfsen/nfsight/config.php

### install mysql service and start it 
yum install chkconfig 
chkconfig mysqld on
service mysqld start

### first time setup of mysql
/usr/bin/mysql_secure_installation


#################################################################
### create nfsight database -- probably overkill but this worked
#################################################################
mysql -u root -p 
Enter password: 

mysql> CREATE DATABASE nfsight

mysql> GRANT ALL PRIVILEGES ON nfsight.* TO root@'%' IDENTIFIED BY 'password';
Query OK, 0 rows affected (0.00 sec)

mysql> grant all privileges on nfsight.* TO root@localhost IDENTIFIED BY 'password';
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON nfsight.* TO 'root'@'%' WITH GRANT OPTION;
Query OK, 0 rows affected (0.00 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

mysql> quit
Bye

#################################################################
### launch http://yourserver/nfsight/installer.php
### 
### setup as you desire including proper paths:
### URL = /nfsight/
### Path to data files = /var/www/html/nfsen/plugins/nfsight
#################################################################

#################################################################
### settings can be edited later without installer.php: 
### nano /var/www/html/nfsen/nfsight/config.php
### nano /var/www/html/nfsen/nfsight/detail.php
#################################################################

#################################################################
### edit detail.php to include proper paths:
### /bin/grep
### /bin/cat
### /usr/bin/pcv
#################################################################
nano /var/www/html/nfsen/nfsight/detail.php


#################################################################
### edit /usr/local/nfsen/etc/nfsen.conf
### add settings as output by installer.php
### 
### your setting may vary:
#################################################################
@plugins = (
[ '*', 'nfsight' ],
);

%PluginConf = (\
nfsight => {
            path => "/var/www/html/nfsen/plugins/nfsight",
                expiration => "180",
                processing_timer => "",
                network => {
                        "10.0.0.0" => "8",
                },
                scanner_limit => "5",
                bidirectional_via_nfdump => "",
                print_int_scanner => "1",
                print_ext_scanner => "1",
                print_int_client => "0",
                print_ext_client => "0",
                print_int_server => "1",
                print_ext_server => "0",
                print_int_invalid => "0",
                print_ext_invalid => "0",
                sql_host => "localhost",
                sql_port => "3306",
                sql_user => "root",
                sql_pass => "password",
                sql_db => "nfsight",
        },

);
#################################################################

### restart nfsen
/usr/local/nfsen/bin/nfsen stop
/usr/local/nfsen/bin/nfsen start
/usr/local/nfsen/bin/nfsen status

### edit cron jobs as decribed from installer.php
crontab -e
06 * * * *  /usr/bin/wget --no-check-certificate -q -O - http://management:aggregate@127.0.0.1/nfsight/aggregate.php


#################################################################
### optional: install picviz
#################################################################
### picviz 0.6-8 has only one dependency issue in CentOS 6.4
### picviz 0.6-8 requires libev available in the EPEL repository
### What is EPEL?: https://fedoraproject.org/wiki/EPEL
#################################################################

### Install the EPEL repository 
### This is the 64-bit version, you may need to locate the 32-bit version
rpm -Uvh http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

### install picviz prerequisite libev from EPEL
yum install libev

### install picviz 0.6-8 64-bit
wget https://depots.global-sp.net/CentOS/6/x86_64/picviz-0.6-8.el6.x86_64.rpm
rpm -i picviz-0.6-8.el6.x86_64.rpm 
wget https://depots.global-sp.net/CentOS/6/x86_64/picviz-plugin-pngcairo-0.6-8.el6.x86_64.rpm
rpm -i picviz-plugin-pngcairo-0.6-8.el6.x86_64.rpm 

### fix errors decribed by nfsight regarding picviz
chmod g+w /var/www/html/nfsen/nfsight/cache
chmod g+x /var/www/html/nfsen/nfsight/bin/biflow2picviz.pl

---------------

Please consider crypto tipping: