Z-Car

Unable to renew using LetsEncrypt Certbot-Auto with CRON – Fail

Recently I had a fail when trying to automate the renewal of my LetsEncrypt SSL certificates using a CRON job, I experienced the following error when running “certbot-auto renew” :

Error: couldn’t get currently installed version for //.local/share/letsencrypt/bin/letsencrypt:
//.local/share/letsencrypt/lib64/python2.6/site-packages/cryptography/__init__.py:26: DeprecationWarning: Python 2.6 is no longer supported by the Python core team, please upgrade your Python. A future version of cryptography will drop support for Python 2.6
  DeprecationWarning
An unexpected error occurred:
ContextualVersionConflict: (setuptools 0.9.8 (/.local/share/letsencrypt/lib/python2.6/site-packages), Requirement.parse(‘setuptools>=1.0′), set([‘certbot’]))

At first glance, this failure would appear to have something to do with Python 2.6 and the cryptography site-package.  However, this actually just turns out to be a warning.  The unexpected error below is what causes the issue.  If you look closely at the error (couldn’t get currently installed version for //.local/share/letsencrypt/bin/letsencrypt) line and the line starting with ContextualVersionConflict, you will see that it is trying to access (/.local/share/letsencrypt/lib/python2.6/site-packages).  In both cases, the issue is with //.local/, when running as a cronjob, this needs to point to /root/.local.  There are a couple ways to fix this.

One, you can edit your crontab and edit HOME=/ to HOME=/root/
Two, you can create a wrapper script that changes your HOME directory before calling certbot-auto renew
Lastly, you can choose to do what I did and directly edit the certbot-auto script.   I hard-coded the values for HOME and XDG_DATA_HOME.  This has the disadvantage of getting wiped out if you ever update that script, however it also guarantees that no matter how that script is executed, it will always be pointing to the correct location.

To edit the script, add the following lines :
HOME=/root/
XDG_DATA_HOME=/root/.local/share

before the line that has :
VENV_NAME=”letsencrypt”

Leave a comment if this solved your problem, or if you are experiencing any other strange issues running certbot-auto from a cron job.


Installing LetsEncrypt Certbot on Amazon Linux Server

The LetsEncrypt Certbot is an awesome tool for getting and maintaining SSL certificates for your sites.  In most cases, it is extremely easy to setup and maintain.  However, installing it on Amazon’s proprietary Linux server in EC2, it often fails.  These are the steps I took to get it to install properly in my environment.  

wget https://dl.eff.org/certbot-auto
chmod a+x ./certbot-auto
sudo yum install libffi-devel
sudo yum install openssl-devel
sudo chown -R ec2-user:ec2-user .well-known/
./certbot-auto –no-self-upgrade –no-bootstrap

After performing these commands, you should now be able to use Certbot as you would normally.

Certbot will place your completed SSL certs in the following paths.

  • Certificate: /etc/letsencrypt/live/YOUR_WEBSITE_HERE/cert.pem
  • Full Chain: /etc/letsencrypt/live/YOUR_WEBSITE_HERE/fullchain.pem
  • Private Key: /etc/letsencrypt/live/YOUR_WEBSITE_HERE/privkey.pem

To setup auto renew, you will need to add the following line to your crontab file.  Make sure you are running as root when you do so using sudo su.

Adding this  line will run the renewal process twice per day.

0 2,14 * * * /home/ec2-user/certbot-auto renew

 If you have any issues, leave a message in the comments so we can help.


Some useful custom SpamAssassin rules and settings – modify your own

 

I have a Linux server that I use for managing web sites, development work, etc.  In addition, I use it to host a mail server for two domains that I have had since 1993.  Because of their age, and the number of email addresses that were used on them over the years, they receive a LOT of Spam.  In order to manage this huge volume of SPAM, I use SpamAssassin, and a ton of custom rules in Postfix to minimize the amount of spam that ultimately reached my inbox.  

Here is a collection of score changes, and custom filters that I use.  These change on a fairly regular basis, as the Spammers are always making changes.  I will try and keep this updated fairly often.

If you have any questions on how to write a rule, or have a suggestion for a good one, leave a comment or email.  

score RAZOR2_CHECK 5
score BAYES_999 1.0
score BAYES_00 -4
score T_FREEMAIL_DOC_PDF 2
score DIGEST_MULTIPLE 5
score MPART_ALT_DIFF 5
score RCVD_IN_MSPIKE_L5 4
score URIBL_BLACK 5
score URIBL_DBL_SPAM 5
score DCC_CHECK 5
score PYZOR_CHECK 5

mimeheader ZIP_ATTACHED Content-Type =~ /zip|xls|docm|doc/i
describe ZIP_ATTACHED email contains a zip file attachment
score ZIP_ATTACHED 4.5
header CUSTOM_PHP_ID_SPAM X-PHP-Originating-Script =~ /class.php/
score CUSTOM_PHP_ID_SPAM 5

header CUSTOM_UNQ_ID_SPAM X-MC-Unique =~ /randcase/
score CUSTOM_UNQ_ID_SPAM 5

header CUSTOM_UA_ID_SPAM User-Agent =~ /Mutt/
score CUSTOM_UA_ID_SPAM 5

rawbody CUSTOM_GMAIL_SPAM /style\=\”color\:\#245dc1\;text\-decoration\:none\;/
score CUSTOM_GMAIL_SPAM 4

rawbody CUSTOM_WHATSAP_SPAM /background\:\#d9d9d9\;font\-family\:arial\;font\-weight\:normal\;font\-size\:11px\;color\:\#808080\;/
score CUSTOM_WHATSAP_SPAM 4

rawbody CUSTOM_FEDEX_SPAM /style\=\”text\-decoration\:none\;color\:\#4d148c\;\” alt\=\”Privacy policy\” title\=\”Privacy policy/
score CUSTOM_FEDEX_SPAM 4

body CUSTOM_FARGO_SPAM /FARGO\, ND 58103/
score CUSTOM_FARGO_SPAM 4

rawbody CUSTOM_INLINE_IMAGE /src=”cid:/
score CUSTOM_INLINE_IMAGE 5.5

rawbody CUSTOM_TRACKING_CODE /img src=”(.*)\.us(.*)\?email/
score CUSTOM_TRACKING_CODE 5.5

rawbody CUSTOM_ENDS_IN_GUID /[A-Za-z0-9]{8}[A-Za-z0-9]{4}[A-Za-z0-9]{4}[A-Za-z0-9]{4}[A-Za-z0-9]{12}$/
score CUSTOM_ENDS_IN_GUID 7.5

rawbody CUSTOM_ALT_IN_GUID /alt=”[A-Za-z0-9]{8}[A-Za-z0-9]{4}[A-Za-z0-9]{4}[A-Za-z0-9]{4}[A-Za-z0-9]{12}”/
score CUSTOM_ALT_IN_GUID 0.5

 


How do you extract POST form data and file uploads in node.js?

 

laptop_shutterstock_119563954

One of the most common tasks in creating a Node web server is handling GET and POST calls.  GET calls are fairly straightforward, however POST calls are more involved.  POST form data and POST file uploads provide the ability for a web client to upload enormous amounts of data to your server.  Proper safeguards must be put into place to insure that your server does not crash if it runs out of resources required to handle a POST of unlimited size.

Many of the common Node frameworks, like Express, provide an abstraction layer that allows you to more easily access POST data.  However, if you are rolling your own web server in Node, you will need to handle this with your own code.  The example below will allow you to do so.

var http = require('http');
var querystring = require('querystring');

function receivePost(request, response, callback) {
    var queryData = "";
    if(typeof callback !== 'function') return null;
    request.on('data', function(data) {
         queryData += data;
         if(queryData.length > 6e6) { // limit uploaded data to 6M
             queryData = "";
             response.writeHead(413, {'Content-Type': 'text/plain'}).end();
              // Kill connection if over 6M of data received
             request.connection.destroy();
         }
     });

    request.on('end', function() {
        // Parse the final result and save as request.post
        request.post = querystring.parse(queryData);
        callback();
    });
}

http.createServer(function(request, response) {
    if(request.method == 'POST') {
        receivePost(request, response, function() {
            console.log(request.post);
            
            // This section is where you would process the result of receivePost

            response.writeHead(200, "OK", {'Content-Type': 'text/plain'});
            response.end();
        });
    } else {
        response.writeHead(200, "OK", {'Content-Type': 'text/plain'});
        response.end();
    }

}).listen(3000);

Wherever you are handling POST requests, just simply pass the request and response fields to receivePost.  This function will collect the incoming data, and it to a working variable, and insure that it does not exceed 6MB of data (easily changed if you want to allow more or less).  When it returns, request.post will contain the POST data object that you can then manipulate and process as required.


How to proxy web sockets using Apache 2.14 and Node

pretty-hot-girl-working-on-a-laptop

I have an application that I am working on that requires the use of socket.io.  In most implementations, folks are using Node.JS to provide API access on the server-side.   However, many of us still use Apache for serving up static content, and using Node as the API server, facilitating this by using Apache’s proxy capability (mod-proxy).

However, when using the socket.io library, it becomes quickly apparent that a straight-forward proxy will not work.  If you watch how socket.io connects, it connects to a single URL, both using http polling, and then attempting to switch to web sockets if possible, using parameters passed on the query string to control this behavior.

websocket meme

In our implementation, we configure the client to use a specific socket path, which we can filter on in our proxy configuration.  As a bonus, we can also use this socket path to load-balance web socket traffic if required.

Socket.io setup in client side JavaScript.

var socketpath = ‘/chat_api/socket.io’;
var url = ‘/’;

socket = io.connect(url, {
path: socketpath
});

In the Apache config file, we configure ProxyPass to pass all normal http traffic to the node server, API calls (/api) going to one Node port, and Chat API calls (/chat_api)going to another port.  Our rewrite condition specifically looks for web socket traffic by matching the query parameter transport as matching websocket.  In this case, we do a Rewrite to ws:// node chat server.

Apache http.conf

RewriteEngine On

RewriteCond %{REQUEST_URI} ^/chat_api/socket.io [NC]
RewriteCond %{QUERY_STRING} transport=websocket [NC]
RewriteRule “^/chat_api/socket.io” “ws://localhost:4001/socket.io/” [P,L]

ProxyPass /api http://localhost:4000/api
ProxyPassReverse /api http://localhost:4000/api

ProxyPass /chat_api http://localhost:4001
ProxyPassReverse /chat_api http://localhost:4001

In my limited testing, I was ONLY able to get this to reliably work using Apache 2.4.16.  I believe that this is due to a bug in earlier versions that will not allow you to redirect to a web socket  (ws://).   I was forced to compile and install the most recent version available when I wrote this.

One of the most often recommended solutions is to attempt to use proxy_wstunnel and back-port it to an older version of Apache.  I tried this method, and was unsuccessful in getting a stable configuration.  Your milage may vary…


A simple way to monitor for any changes in your Node.js application and magically restart the server

computer girl success

In most of my Node.js application servers, I add a simple section of code at the end which watches specific application files, and terminates the server if any are changed.  In tandem with PM2, this allows the code to restart automatically when either pushing a new version, or directly making changes to the code.

There are other applications like nodemon that provide similar functionality, however I feel that PM2 is a better overall process management system.  This capability was not available, so I just added it directly into my applications.

In my application, I perform a simple process.exit(), however you may prefer to raise a signal in your application so you can gracefully shutdown.

console.log('Starting filename watch');

function restartServer(event, filename) {
    console.log('event is: ' + event);
    if (filename) {
        console.log('filename provided: ' + filename);
    } else {
        console.log('filename not provided');
    }
    process.exit(0);
}

fs.watch('jobs_processor.js', restartServer);
fs.watch('lib/hercle_jobs/lib/hercle_jobs.js', restartServer);
fs.watch('lib/hercle_jobs_v2.js', restartServer);
fs.watch('lib/ims_utility.js', restartServer);