Z-Car

How to Rotate Logs using PM2 Process Manager for Node.js

girl reading service manual

Pm2 is a great utility for keeping your Node production applications running.  It  provides high-level process management, and consolidates all your logs into one common location.  In addition, you can view those logs in a real-time manner using the PM2 logs command.

I now use PM2 in all new Node installations, heavily depending on its ability to capture all Node output into easily organized logs.  One feature lacking is the built-in ability to rotate your log files.  This can create some pretty large files if you are not careful.  Because PM2 does not accept a signal to rotate the log, you need to use the copytruncate capability of logrotate.   Simply add the following in a file called PM2 in /etc/logrotate.d. You can change the frequency (daily, weekly, etc),  number of copies to keep (rotate 5), and you can use the compress option if you want the logs to be compressed when rotated.  I usually skip this step, I prefer to easily be able to access the files without having to uncompress first.

Let me know if you have any other handy tips for using PM2.

/root/.pm2/logs/*.log {
daily
rotate 5
missingok
notifempty
sharedscripts
copytruncate
dateext
}

 


Node.js – fs.open / fs.writeFile, mode, file permissions, umask, and how they interact

Girl Reading Book
While working with Node.js recently, I ran into a situation with the mode parameter used in several functions; fs.open, fs.appendFile, fs.writeFile, etc. When trying to create a new file and specifying a specific mode parameter to allow user and group writes, I found that the file would get created, however would have the wrong permissions.  After much experimenting, I realized that the Linux server’s umask was getting applied, and preventing the creation of a file with the permissions I wanted.

The work around turned out to be fairly simple, although it does create a slight security hole, and should only be used in cases where you can control access to the application.

I simply wrapped the file create calls with the following code :

oldmask = process.umask(newmask);
fs.open(path, flags, [mode], callback)
process.umask(oldmask);

This only needs be performed when the open (or other method) will create a new file.  Let me know if you have experienced a similar issue, and if you have found other solutions to resolve.


How to escape Mongo keys using Node.js in a Flash

Dale is Flash Gordon's constant companion in his adventures, as well as his one true love.

Dale is Flash Gordon’s constant companion in his adventures, as well as his one true love.

Don’t be scared, dealing with Mongo is not that difficult, especially with some help from Flash.   While some people know of Mongo as a fictional planet where the comic  and movie serials of Flash Gordon takes place, ruled by a tyrant named Ming the Merciless, who governs with an iron hand, others are familiar with Mongo as a cross-platform document-oriented database system.

Many first-time users of  Mongo quickly find out that it does not allow insertion of keys with a dot (.) or dollar sign ($).  While understandable, this would seem to defeat the purpose of having a free-form document that is not rigidly defined.   Luckily, there is an easy way to address this issue in Node.js.  Below is a simple code snippet that I wrote which will escape the dot (.) or dollar sign ($) in any of your keys.

Just pass the document into the function, and your object will be escaped.  The dot (.) will be replaced with _dot_ and the ampersand (&) will be replaced with _amp_.  If required, you can change these to whatever you need.  If you need to unescape, you can easily swap the key.replace params.  I will leave this as an exercise to be completed by the reader…

function escapeKeys(obj) {
    if (!(Boolean(obj) && typeof obj == 'object'
      && Object.keys(obj).length > 0)) {
        return false;
    }
    Object.keys(obj).forEach(function(key) {
        if (typeof(obj[key]) == 'object') {
            escapeKeys(obj[key]);
        } else {
            if (key.indexOf('.') !== -1) {
                var newkey = key.replace(/\./g, '_dot_');
                obj[newkey] = obj[key];
                delete obj[key];
            }
            if (key.indexOf('$') !== -1) {
                var newkey = key.replace(/\$/g, '_amp_');
                obj[newkey] = obj[key];
                delete obj[key];
            }

        }
    });
    return true;
}
Syk is ruled by the witch-queen Azura, who is feared by everyone on Mongo.

Syk is ruled by the witch-queen Azura, who is feared by everyone on Mongo.

MongoDB is an open-source document database, and the leading NoSQL database.

MongoDB is an open-source document database, and the leading NoSQL database.

 


Database Admin Controller for Code Igniter and Ext JS 4

girl working on laptop

I have been using Code Igniter and Ext JS 4 for several months.  In that time, I have created a couple basic templates to display MySQL tables and perform CRUD on those tables.  In addition, I have wanted to create some basic admin screens for the Core Igniter configured databases.  In my configuration, I have multiple databases defined, using Code Igniter’s built-in support for multiple database access.  So, given all of that, I have created a controller that will display all Code Igniter configured databases, list the contained tables, and allow you to display and edit those tables.

Right now the basic template supports simple pagination, as well as Adding, Deleting, and Updating records.  While fairly full-featured, it does not yet handle remote sorting.  In addition, I am sure you will find some minor display issues with data fields that are not your standard integer or varchar.  I will look to address these later.  As an added bonus, the template references Ext JS remotely, so you do not need to install Ext JS 4 for these admin screens to work.  If you are interested in giving Ext JS 4 a try, this is a great way to explore with it.

One feature of this controller is that the panel for the table admin is dynamically generated.  This will allow you to save the created JS file, and modify it to suit your needs.  You can easily create very custom screens from this generated template.

If you would like to see updates, or have any issues, leave me a message here and I will try to address them.

You can download the zip file here.

To install, just add the enclosed files to your controllers and views directory and make sure you have a valid database configured.


MySQL Removing duplicate rows – Part II

hot-girl-link
Using ALTER IGNORE TABLE [TABLENAME] ADD UNIQUE INDEX `UNIQUE_INDEX` ([FIELDNAME]) to remove duplicate rows in a table is a fast an efficient process, however on large tables where the physical size is larger than server memory, the ALTER statement can take a long time to run in a production environment.

If you need to remove duplicates on a very large table (we recently used this on a table of 77 million rows), try this method :

delete t1 from table t1, table t2
where t1.duplicate_field= t2.duplicate_field (add more if need ie. and t1.duplicate_field2=t2.duplicate_field2)
and t1.unique_field > t2.unique_field
and breakup into ranges to run faster

If you use an auto-incrementing ID field as the primary key, use this as your unique field, and in the Where clause to run on a range of records to break into smaller operations.


MySQL How to delete duplicate records and rows of data

15647301_s

The fastest and easiest way to delete duplicate records is my issuing a very simple command.

alter ignore table [tablename] add unique index `unique_index` ([fieldname])

What this does is create a unique index on the field that you do not want to have any duplicates. The ignore syntax instructs MySQL to not stop and display an error when it hits a duplicate. This is much easier than dumping and reloading a table.

This also will work, but is not as elegant:

delete from [tablename] where fieldname in (select a.[fieldname] from
(select [fieldname] from [tablename] group by [fieldname] having count(*) > 1 ) a )