Z-Car

Some useful custom SpamAssassin rules and settings – modify your own

 

I have a Linux server that I use for managing web sites, development work, etc.  In addition, I use it to host a mail server for two domains that I have had since 1993.  Because of their age, and the number of email addresses that were used on them over the years, they receive a LOT of Spam.  In order to manage this huge volume of SPAM, I use SpamAssassin, and a ton of custom rules in Postfix to minimize the amount of spam that ultimately reached my inbox.  

Here is a collection of score changes, and custom filters that I use.  These change on a fairly regular basis, as the Spammers are always making changes.  I will try and keep this updated fairly often.

If you have any questions on how to write a rule, or have a suggestion for a good one, leave a comment or email.  

score RAZOR2_CHECK 5
score BAYES_999 1.0
score BAYES_00 -4
score T_FREEMAIL_DOC_PDF 2
score DIGEST_MULTIPLE 5
score MPART_ALT_DIFF 5
score RCVD_IN_MSPIKE_L5 4
score URIBL_BLACK 5
score URIBL_DBL_SPAM 5
score DCC_CHECK 5
score PYZOR_CHECK 5

mimeheader ZIP_ATTACHED Content-Type =~ /zip|xls|docm|doc/i
describe ZIP_ATTACHED email contains a zip file attachment
score ZIP_ATTACHED 4.5
header CUSTOM_PHP_ID_SPAM X-PHP-Originating-Script =~ /class.php/
score CUSTOM_PHP_ID_SPAM 5

header CUSTOM_UNQ_ID_SPAM X-MC-Unique =~ /randcase/
score CUSTOM_UNQ_ID_SPAM 5

header CUSTOM_UA_ID_SPAM User-Agent =~ /Mutt/
score CUSTOM_UA_ID_SPAM 5

rawbody CUSTOM_GMAIL_SPAM /style\=\”color\:\#245dc1\;text\-decoration\:none\;/
score CUSTOM_GMAIL_SPAM 4

rawbody CUSTOM_WHATSAP_SPAM /background\:\#d9d9d9\;font\-family\:arial\;font\-weight\:normal\;font\-size\:11px\;color\:\#808080\;/
score CUSTOM_WHATSAP_SPAM 4

rawbody CUSTOM_FEDEX_SPAM /style\=\”text\-decoration\:none\;color\:\#4d148c\;\” alt\=\”Privacy policy\” title\=\”Privacy policy/
score CUSTOM_FEDEX_SPAM 4

body CUSTOM_FARGO_SPAM /FARGO\, ND 58103/
score CUSTOM_FARGO_SPAM 4

rawbody CUSTOM_INLINE_IMAGE /src=”cid:/
score CUSTOM_INLINE_IMAGE 5.5

rawbody CUSTOM_TRACKING_CODE /img src=”(.*)\.us(.*)\?email/
score CUSTOM_TRACKING_CODE 5.5

rawbody CUSTOM_ENDS_IN_GUID /[A-Za-z0-9]{8}[A-Za-z0-9]{4}[A-Za-z0-9]{4}[A-Za-z0-9]{4}[A-Za-z0-9]{12}$/
score CUSTOM_ENDS_IN_GUID 7.5

rawbody CUSTOM_ALT_IN_GUID /alt=”[A-Za-z0-9]{8}[A-Za-z0-9]{4}[A-Za-z0-9]{4}[A-Za-z0-9]{4}[A-Za-z0-9]{12}”/
score CUSTOM_ALT_IN_GUID 0.5

 


How do you extract POST form data and file uploads in node.js?

 

laptop_shutterstock_119563954

One of the most common tasks in creating a Node web server is handling GET and POST calls.  GET calls are fairly straightforward, however POST calls are more involved.  POST form data and POST file uploads provide the ability for a web client to upload enormous amounts of data to your server.  Proper safeguards must be put into place to insure that your server does not crash if it runs out of resources required to handle a POST of unlimited size.

Many of the common Node frameworks, like Express, provide an abstraction layer that allows you to more easily access POST data.  However, if you are rolling your own web server in Node, you will need to handle this with your own code.  The example below will allow you to do so.

var http = require('http');
var querystring = require('querystring');

function receivePost(request, response, callback) {
    var queryData = "";
    if(typeof callback !== 'function') return null;
    request.on('data', function(data) {
         queryData += data;
         if(queryData.length > 6e6) { // limit uploaded data to 6M
             queryData = "";
             response.writeHead(413, {'Content-Type': 'text/plain'}).end();
              // Kill connection if over 6M of data received
             request.connection.destroy();
         }
     });

    request.on('end', function() {
        // Parse the final result and save as request.post
        request.post = querystring.parse(queryData);
        callback();
    });
}

http.createServer(function(request, response) {
    if(request.method == 'POST') {
        receivePost(request, response, function() {
            console.log(request.post);
            
            // This section is where you would process the result of receivePost

            response.writeHead(200, "OK", {'Content-Type': 'text/plain'});
            response.end();
        });
    } else {
        response.writeHead(200, "OK", {'Content-Type': 'text/plain'});
        response.end();
    }

}).listen(3000);

Wherever you are handling POST requests, just simply pass the request and response fields to receivePost.  This function will collect the incoming data, and it to a working variable, and insure that it does not exceed 6MB of data (easily changed if you want to allow more or less).  When it returns, request.post will contain the POST data object that you can then manipulate and process as required.


How to proxy web sockets using Apache 2.14 and Node

pretty-hot-girl-working-on-a-laptop

I have an application that I am working on that requires the use of socket.io.  In most implementations, folks are using Node.JS to provide API access on the server-side.   However, many of us still use Apache for serving up static content, and using Node as the API server, facilitating this by using Apache’s proxy capability (mod-proxy).

However, when using the socket.io library, it becomes quickly apparent that a straight-forward proxy will not work.  If you watch how socket.io connects, it connects to a single URL, both using http polling, and then attempting to switch to web sockets if possible, using parameters passed on the query string to control this behavior.

websocket meme

In our implementation, we configure the client to use a specific socket path, which we can filter on in our proxy configuration.  As a bonus, we can also use this socket path to load-balance web socket traffic if required.

Socket.io setup in client side JavaScript.

var socketpath = ‘/chat_api/socket.io’;
var url = ‘/’;

socket = io.connect(url, {
path: socketpath
});

In the Apache config file, we configure ProxyPass to pass all normal http traffic to the node server, API calls (/api) going to one Node port, and Chat API calls (/chat_api)going to another port.  Our rewrite condition specifically looks for web socket traffic by matching the query parameter transport as matching websocket.  In this case, we do a Rewrite to ws:// node chat server.

Apache http.conf

RewriteEngine On

RewriteCond %{REQUEST_URI} ^/chat_api/socket.io [NC]
RewriteCond %{QUERY_STRING} transport=websocket [NC]
RewriteRule “^/chat_api/socket.io” “ws://localhost:4001/socket.io/” [P,L]

ProxyPass /api http://localhost:4000/api
ProxyPassReverse /api http://localhost:4000/api

ProxyPass /chat_api http://localhost:4001
ProxyPassReverse /chat_api http://localhost:4001

In my limited testing, I was ONLY able to get this to reliably work using Apache 2.4.16.  I believe that this is due to a bug in earlier versions that will not allow you to redirect to a web socket  (ws://).   I was forced to compile and install the most recent version available when I wrote this.

One of the most often recommended solutions is to attempt to use proxy_wstunnel and back-port it to an older version of Apache.  I tried this method, and was unsuccessful in getting a stable configuration.  Your milage may vary…


A simple way to monitor for any changes in your Node.js application and magically restart the server

computer girl success

In most of my Node.js application servers, I add a simple section of code at the end which watches specific application files, and terminates the server if any are changed.  In tandem with PM2, this allows the code to restart automatically when either pushing a new version, or directly making changes to the code.

There are other applications like nodemon that provide similar functionality, however I feel that PM2 is a better overall process management system.  This capability was not available, so I just added it directly into my applications.

In my application, I perform a simple process.exit(), however you may prefer to raise a signal in your application so you can gracefully shutdown.

console.log('Starting filename watch');

function restartServer(event, filename) {
    console.log('event is: ' + event);
    if (filename) {
        console.log('filename provided: ' + filename);
    } else {
        console.log('filename not provided');
    }
    process.exit(0);
}

fs.watch('jobs_processor.js', restartServer);
fs.watch('lib/hercle_jobs/lib/hercle_jobs.js', restartServer);
fs.watch('lib/hercle_jobs_v2.js', restartServer);
fs.watch('lib/ims_utility.js', restartServer);

How to Rotate Logs using PM2 Process Manager for Node.js

girl reading service manual

Pm2 is a great utility for keeping your Node production applications running.  It  provides high-level process management, and consolidates all your logs into one common location.  In addition, you can view those logs in a real-time manner using the PM2 logs command.

I now use PM2 in all new Node installations, heavily depending on its ability to capture all Node output into easily organized logs.  One feature lacking is the built-in ability to rotate your log files.  This can create some pretty large files if you are not careful.  Because PM2 does not accept a signal to rotate the log, you need to use the copytruncate capability of logrotate.   Simply add the following in a file called PM2 in /etc/logrotate.d. You can change the frequency (daily, weekly, etc),  number of copies to keep (rotate 5), and you can use the compress option if you want the logs to be compressed when rotated.  I usually skip this step, I prefer to easily be able to access the files without having to uncompress first.

Let me know if you have any other handy tips for using PM2.

/root/.pm2/logs/*.log {
daily
rotate 5
missingok
notifempty
sharedscripts
copytruncate
dateext
}

 


Node.js – fs.open / fs.writeFile, mode, file permissions, umask, and how they interact

Girl Reading Book
While working with Node.js recently, I ran into a situation with the mode parameter used in several functions; fs.open, fs.appendFile, fs.writeFile, etc. When trying to create a new file and specifying a specific mode parameter to allow user and group writes, I found that the file would get created, however would have the wrong permissions.  After much experimenting, I realized that the Linux server’s umask was getting applied, and preventing the creation of a file with the permissions I wanted.

The work around turned out to be fairly simple, although it does create a slight security hole, and should only be used in cases where you can control access to the application.

I simply wrapped the file create calls with the following code :

oldmask = process.umask(newmask);
fs.open(path, flags, [mode], callback)
process.umask(oldmask);

This only needs be performed when the open (or other method) will create a new file.  Let me know if you have experienced a similar issue, and if you have found other solutions to resolve.