Z-Car

How do you extract POST form data and file uploads in node.js?

 

laptop_shutterstock_119563954

One of the most common tasks in creating a Node web server is handling GET and POST calls.  GET calls are fairly straightforward, however POST calls are more involved.  POST form data and POST file uploads provide the ability for a web client to upload enormous amounts of data to your server.  Proper safeguards must be put into place to insure that your server does not crash if it runs out of resources required to handle a POST of unlimited size.

Many of the common Node frameworks, like Express, provide an abstraction layer that allows you to more easily access POST data.  However, if you are rolling your own web server in Node, you will need to handle this with your own code.  The example below will allow you to do so.

var http = require('http');
var querystring = require('querystring');

function receivePost(request, response, callback) {
    var queryData = "";
    if(typeof callback !== 'function') return null;
    request.on('data', function(data) {
         queryData += data;
         if(queryData.length > 6e6) { // limit uploaded data to 6M
             queryData = "";
             response.writeHead(413, {'Content-Type': 'text/plain'}).end();
              // Kill connection if over 6M of data received
             request.connection.destroy();
         }
     });

    request.on('end', function() {
        // Parse the final result and save as request.post
        request.post = querystring.parse(queryData);
        callback();
    });
}

http.createServer(function(request, response) {
    if(request.method == 'POST') {
        receivePost(request, response, function() {
            console.log(request.post);
            
            // This section is where you would process the result of receivePost

            response.writeHead(200, "OK", {'Content-Type': 'text/plain'});
            response.end();
        });
    } else {
        response.writeHead(200, "OK", {'Content-Type': 'text/plain'});
        response.end();
    }

}).listen(3000);

Wherever you are handling POST requests, just simply pass the request and response fields to receivePost.  This function will collect the incoming data, and it to a working variable, and insure that it does not exceed 6MB of data (easily changed if you want to allow more or less).  When it returns, request.post will contain the POST data object that you can then manipulate and process as required.


How to proxy web sockets using Apache 2.14 and Node

pretty-hot-girl-working-on-a-laptop

I have an application that I am working on that requires the use of socket.io.  In most implementations, folks are using Node.JS to provide API access on the server-side.   However, many of us still use Apache for serving up static content, and using Node as the API server, facilitating this by using Apache’s proxy capability (mod-proxy).

However, when using the socket.io library, it becomes quickly apparent that a straight-forward proxy will not work.  If you watch how socket.io connects, it connects to a single URL, both using http polling, and then attempting to switch to web sockets if possible, using parameters passed on the query string to control this behavior.

websocket meme

In our implementation, we configure the client to use a specific socket path, which we can filter on in our proxy configuration.  As a bonus, we can also use this socket path to load-balance web socket traffic if required.

Socket.io setup in client side JavaScript.

var socketpath = ‘/chat_api/socket.io’;
var url = ‘/’;

socket = io.connect(url, {
path: socketpath
});

In the Apache config file, we configure ProxyPass to pass all normal http traffic to the node server, API calls (/api) going to one Node port, and Chat API calls (/chat_api)going to another port.  Our rewrite condition specifically looks for web socket traffic by matching the query parameter transport as matching websocket.  In this case, we do a Rewrite to ws:// node chat server.

Apache http.conf

RewriteEngine On

RewriteCond %{REQUEST_URI} ^/chat_api/socket.io [NC]
RewriteCond %{QUERY_STRING} transport=websocket [NC]
RewriteRule “^/chat_api/socket.io” “ws://localhost:4001/socket.io/” [P,L]

ProxyPass /api http://localhost:4000/api
ProxyPassReverse /api http://localhost:4000/api

ProxyPass /chat_api http://localhost:4001
ProxyPassReverse /chat_api http://localhost:4001

In my limited testing, I was ONLY able to get this to reliably work using Apache 2.4.16.  I believe that this is due to a bug in earlier versions that will not allow you to redirect to a web socket  (ws://).   I was forced to compile and install the most recent version available when I wrote this.

One of the most often recommended solutions is to attempt to use proxy_wstunnel and back-port it to an older version of Apache.  I tried this method, and was unsuccessful in getting a stable configuration.  Your milage may vary…


A simple way to monitor for any changes in your Node.js application and magically restart the server

computer girl success

In most of my Node.js application servers, I add a simple section of code at the end which watches specific application files, and terminates the server if any are changed.  In tandem with PM2, this allows the code to restart automatically when either pushing a new version, or directly making changes to the code.

There are other applications like nodemon that provide similar functionality, however I feel that PM2 is a better overall process management system.  This capability was not available, so I just added it directly into my applications.

In my application, I perform a simple process.exit(), however you may prefer to raise a signal in your application so you can gracefully shutdown.

console.log('Starting filename watch');

function restartServer(event, filename) {
    console.log('event is: ' + event);
    if (filename) {
        console.log('filename provided: ' + filename);
    } else {
        console.log('filename not provided');
    }
    process.exit(0);
}

fs.watch('jobs_processor.js', restartServer);
fs.watch('lib/hercle_jobs/lib/hercle_jobs.js', restartServer);
fs.watch('lib/hercle_jobs_v2.js', restartServer);
fs.watch('lib/ims_utility.js', restartServer);

How to Rotate Logs using PM2 Process Manager for Node.js

girl reading service manual

Pm2 is a great utility for keeping your Node production applications running.  It  provides high-level process management, and consolidates all your logs into one common location.  In addition, you can view those logs in a real-time manner using the PM2 logs command.

I now use PM2 in all new Node installations, heavily depending on its ability to capture all Node output into easily organized logs.  One feature lacking is the built-in ability to rotate your log files.  This can create some pretty large files if you are not careful.  Because PM2 does not accept a signal to rotate the log, you need to use the copytruncate capability of logrotate.   Simply add the following in a file called PM2 in /etc/logrotate.d. You can change the frequency (daily, weekly, etc),  number of copies to keep (rotate 5), and you can use the compress option if you want the logs to be compressed when rotated.  I usually skip this step, I prefer to easily be able to access the files without having to uncompress first.

Let me know if you have any other handy tips for using PM2.

/root/.pm2/logs/*.log {
daily
rotate 5
missingok
notifempty
sharedscripts
copytruncate
dateext
}

 


Node.js – fs.open / fs.writeFile, mode, file permissions, umask, and how they interact

Girl Reading Book
While working with Node.js recently, I ran into a situation with the mode parameter used in several functions; fs.open, fs.appendFile, fs.writeFile, etc. When trying to create a new file and specifying a specific mode parameter to allow user and group writes, I found that the file would get created, however would have the wrong permissions.  After much experimenting, I realized that the Linux server’s umask was getting applied, and preventing the creation of a file with the permissions I wanted.

The work around turned out to be fairly simple, although it does create a slight security hole, and should only be used in cases where you can control access to the application.

I simply wrapped the file create calls with the following code :

oldmask = process.umask(newmask);
fs.open(path, flags, [mode], callback)
process.umask(oldmask);

This only needs be performed when the open (or other method) will create a new file.  Let me know if you have experienced a similar issue, and if you have found other solutions to resolve.


How to escape Mongo keys using Node.js in a Flash

Dale is Flash Gordon's constant companion in his adventures, as well as his one true love.

Dale is Flash Gordon’s constant companion in his adventures, as well as his one true love.

Don’t be scared, dealing with Mongo is not that difficult, especially with some help from Flash.   While some people know of Mongo as a fictional planet where the comic  and movie serials of Flash Gordon takes place, ruled by a tyrant named Ming the Merciless, who governs with an iron hand, others are familiar with Mongo as a cross-platform document-oriented database system.

Many first-time users of  Mongo quickly find out that it does not allow insertion of keys with a dot (.) or dollar sign ($).  While understandable, this would seem to defeat the purpose of having a free-form document that is not rigidly defined.   Luckily, there is an easy way to address this issue in Node.js.  Below is a simple code snippet that I wrote which will escape the dot (.) or dollar sign ($) in any of your keys.

Just pass the document into the function, and your object will be escaped.  The dot (.) will be replaced with _dot_ and the ampersand (&) will be replaced with _amp_.  If required, you can change these to whatever you need.  If you need to unescape, you can easily swap the key.replace params.  I will leave this as an exercise to be completed by the reader…

function escapeKeys(obj) {
    if (!(Boolean(obj) && typeof obj == 'object'
      && Object.keys(obj).length > 0)) {
        return false;
    }
    Object.keys(obj).forEach(function(key) {
        if (typeof(obj[key]) == 'object') {
            escapeKeys(obj[key]);
        } else {
            if (key.indexOf('.') !== -1) {
                var newkey = key.replace(/\./g, '_dot_');
                obj[newkey] = obj[key];
                delete obj[key];
            }
            if (key.indexOf('$') !== -1) {
                var newkey = key.replace(/\$/g, '_amp_');
                obj[newkey] = obj[key];
                delete obj[key];
            }

        }
    });
    return true;
}
Syk is ruled by the witch-queen Azura, who is feared by everyone on Mongo.

Syk is ruled by the witch-queen Azura, who is feared by everyone on Mongo.

MongoDB is an open-source document database, and the leading NoSQL database.

MongoDB is an open-source document database, and the leading NoSQL database.