25 Dec 2011

List all installed and available Perl modules

Code below provides list of modules which perl interpreter can find in runtime.

Code

#!/usr/bin/perl
#
# Author:  Tomasz Gaweda
# Date:    2011.12.23 17:00
# Purpose: Print all modules installed 
#          in perl distribution
#
# Usage:   perl showmodules.pl

use strict;
use warnings;
use File::Find;

my @modules;
my $IgnorePatt = join("|",@INC) ;#$IgnorePatt =~ s/\|\.// ;
find(
 sub {
  my $f = $File::Find::name;
  if ( $f =~ s/^(${IgnorePatt})\/(.*?)\.pm$/$2/ ) 
  {
   $f =~ s/[\/\\]/::/g;
   push @modules, $f;
  }
 }
 , grep { -d } @INC
);
print "There are [".scalar(@modules)."] modules that perl can locate on runtime...\n";
print "Printing List in Minute\n" ;
sleep 3;
print join("\n", sort{ lc $a cmp lc $b} @modules)."\n";

Sample output

There are [2346] modules that perl can locate on runtime...
Printing List in Minute
AcidRip::acidrip
AcidRip::interface
AcidRip::messages
AcidRip::signals
AE
Algorithm::C3
Algorithm::Diff
Algorithm::Diff::XS
Algorithm::DiffOld
Algorithm::Merge
Alien::Package
...

11 Dec 2011

Encoding files "on the fly" with Blowfish using Perl

Some time ago friend of mine asked me if I could write him an CGI script which encodes his backup file and allows to download it using HTTP protocol.

He is using some kind of VPS so my CGI app had some limitations:

  • file must be encoded "on the fly" to save disk space (no temporary files)
  • encoding should be memmory efficient - because VPS hasn't got very much RAM and backup files are quite large
  • encoding should be fast - so the best solution would be using XS module

After some research I've created this two scripts:

Warning: Scripts depends on external Perl modules which are available at CPAN and in Debian repository.
apt-get install libcrypt-cbc-perl libcrypt-blowfish-perl

Blowfish CGI encoding using Perl

#!/usr/bin/perl

use strict;
use warnings;

use CGI;
use HTML::Entities;
use Crypt::CBC;
use File::Spec;

my $as_cgi = 1; # if run from apachce set to 1 if CLI 0

my $key = "ala_ma_kota.123";
my $fName = './large_file.bin';

my $chunkSize = 1024;
my $out_FName = 'out.bin';



if ( $as_cgi ) {
 print "Content-Type:application/octet-stream; name=\"".$out_FName."\"\r\n";
 print "Content-Disposition: attachment; filename=\"".$out_FName."\"\r\n\n";
}

my $buffer = '';
my $cipher = Crypt::CBC->new( 
   -key    => $key,
   -cipher => 'Blowfish',
);

die "Unable to find file $fName\n" if not -e $fName;


open(ENCODEME, '<:raw', $fName) or die "$!";
binmode ENCODEME;
$cipher->start('encrypting');
while( read(ENCODEME,$buffer,$chunkSize) ) {
 print $cipher->crypt($buffer);
}
print $cipher->finish;
close(ENCODEME);

Blowfish decoding file using Perl

 #!/usr/bin/perl

# apt-get install libcrypt-cbc-perl
# apt-get install libcrypt-blowfish-perl

use strict;
use warnings;

use Crypt::CBC;
use File::Spec;

my $key = "ala_ma_kota.123";
my $chunkSize = 1024;

my $fName = 'zz.bin';

my $buffer = '';
my $cipher = Crypt::CBC->new( 
   -key    => $key,
   -cipher => 'Blowfish',
);

#$fName = File::Spec->catfile($base_dir, $fName);
die "Unable to find file $fName\n" if not -e $fName;


open(ENCODEME, '<:raw', $fName) or die "$!";
binmode ENCODEME;
$cipher->start('decrypting');
while( read(ENCODEME,$buffer,$chunkSize) ) {
 print $cipher->crypt($buffer);
}
print $cipher->finish;
close(ENCODEME);

13 Oct 2011

Using perl to extract files from large directory structure

When I work on Windows I use Activesite Active Perl to do some stuff. Recently I had to copy about 60 files from large directory structure to a new place (about 800 files in nasty directory structure). It was rather boring stuff to do, so I used perl to finish job because it would be pretty easy task to do using hashes (I had names of files given in text file). Unfortunately I had some problems on completion of job, because there was a problem with encoding of file path name on Windows.

Below I provide simple script demonstration how to handle national characters in path using perl on Windows.

#!/usr/bin/perl
 
use strict;
use warnings;
 
use utf8;
use Encoding;
 
# Win32::Codepage::Simple is available in ActiveState Perl
use Win32::Codepage::Simple qw(get_codepage get_acp);
use File::Find;
use File::Copy;
use File::Path;
use File::Basename;
 
binmode STDOUT, ":utf8";
binmode STDERR, ":utf8";
 
my $dir  = "./copy_from_here/";
my $dest = "./copy_here/";
my %filter = ( 'grzegrzółka.doc' => 1, 'słoń i łoś.avi' => 1);
 
# Polish Windows 7 :)
my $encoding = get_codepage();
if ($encoding eq '1250') {
   $encoding = 'cp1250';
}
 
find(
       { wanted =>
               sub {
 
               return if not -f $File::Find::name;
 
               my $mfname = $encoding ? 
                              decode($encoding, $File::Find::name) 
                            : $File::Find::name;
               return if not exists $filter{$mfname};
 
               my $nfile = basename($File::Find::name);
               if ( not -e $dest.'/'.$nfile ) {
                       copy($fInName, $dest.'/'.$nfile) or die "File cannot be copied.";
               } else {
                       print "File exists: ".$nfile."\n";
               }
 
               },
       follow => 1, no_chdir => 1, depth => 1
       }
       , $dir
);

More information about national characters in path on Windows

9 Apr 2011

How to manage and disable bash history

You can change Bash history behaviour changing environment variables. Here is simple reference about this settings.

20 Mar 2011

Arrays in PostgreSQL quick example

What is an array in database?

PostgreSQL allows columns of a table to be defined as variable-length multidimensional arrays. Arrays of any built-in or user-defined base type, enum type, or composite type can be created. Arrays are not supper efficient way of doing things, but they are sometimes handy.

Create some test data

Suppose we have table like this:

CREATE TABLE survey_test (
 id SERIAL,
 gender CHAR(1),
 school CHAR(2), -- can be: LO, TH, ZW
 q_9 VARCHAR(10) -- can be: [A-J]{0,10}
);

Where we store this data:

Securing Linux: few tips for good configuration of SSHD

Most people install sshd daemon using command apt-get install open-sshd, and then forgets about making it more secure. In this article I will point out few sshd options which you should have in your sshd config when going to production.

1 Feb 2011

NWCHem on Ubuntu Linux

This article has been written as instruction for my friend how to run NWChem.

What is NWChem?

From Wikipedia:

NWChem is an ab initio computational chemistry software package which also includes quantum chemical and molecular dynamics functionality. It was designed to run on high-performance parallel supercomputers as well as conventional workstation clusters. It aims to be scalable both in its ability to treat large problems efficiently, and in its usage of available parallel computing resources.

Why should I use NWChem?

Because it can handle:
  • Biomolecules, nanostructures, and solid-state
  • From quantum to classical, and all combinations
  • Gaussian basis functions or plane-waves
  • Scaling from one to thousands of processors
  • Properties and relativity

Before NWChem compilation

apt-get install build-essential gfortran

NWChem compilation

tar -xvzf ../Nwchem-6.0.tar.gz 
cd nwchem-6.0/
export NWCHEM_TOP=`pwd`
cd src/
export FC=gfortran
export NWCHEM_TARGET=LINUX64
make nwchem_config NWCHEM_MODULES=all
make
cd ../bin/LINUX64/
./nwchem

After running nwchem you probably get segfault (are you sure you have X windows session available?). Installation can be done using make install command executed in nwchem-6.0/src/

Additional notes

  • Use Intel compiler when you have Pentium4/Xeon (+30-40% performance gain) or PentiumIII/AMD (+10-20% performance gain).
  • pgf77 (Portland Group Compiler) is not fully supported
  • GNU fortran compiler (g77) on x64 is not encouraged because of the NWChem default use of 64-bit integers on 64-bit architectures

31 Jan 2011

"What time is it?" Using rdate/ntpdate to set time on Linux server

ntpdate

Ntpdate sets the local date and time on your machine by polling the Network Time Protocol (NTP - RFC 1305) server given as the server arguments to determine the correct time. The accuracy and reliability of ntpdate depends on the number of servers and the number of polls each time it is run and the interval between runs. Ntpdate can be run manually as necessary to set the host clock, or it can be run from the host startup script to set the clock at boot time. Using ntpdate with cron scripts is no substitute for the NTP daemon, which uses sophisticated algorithms to maximize accuracy and reliability while minimizing resource use. Finally, since ntpdate does not discipline the host clock frequency as does ntpd, the accuracy using ntpdate is limited.

Installation of ntpdate is simple apt-get install ntpdate.

Usage of ntpdate

ntpdate -t 3 pl.pool.ntp.org
hwclock --systohc

List of polish ntp servers

  • ntp.certum.pl
  • ntp1.tp.pl
  • ntp1.net.icm.edu.pl
  • vega.cbk.poznan.pl
Here is official list of NTP startum one time servers.

rdate

Rdate is a tool for querying the current time from a network server and, optionally, setting the system time (for detailed protocol specification see rdate RFC - RFC 868). Rdate uses the Time Protocol which is generally considered obsolete and has been replaced by the Network Time Protocol (NTP). Rate is generally used only on systems where NTP is not available, or in specialized circumstances where it is required that system time be set correctly as soon as possible during initial setup, before services which may be vulnerable to time-changes.

Installation of rdate is simple apt-get install rdate.

Usage of rdate

rdate -s vega.cbk.poznan.pl
hwclock --systohc

List of polish rdate servers

  • task.gda.pl
  • vega.cbk.poznan.pl

24 Jan 2011

Apache2 - create password protected directory

My friend requested article about password protected access to public directories using .htpasswd file. Configuration of this is quite simple. All you have to do is make sure that you have .htaccess and AuthConfig override enabled (on Ubuntu this is default AFAIK) files support enabled (somewhere in your /etc/apache2/apache.conf should be AllowOverride AuthConfig directive).

23 Jan 2011

Creating ASCII ART MOTD

MOTD or "Message of The Day" in most Unix systems is displayed after successful login, but before execution of user shell. MOTD is used to send a common message to all system users (after login), in a more efficient manner than sending them all an e-mail message. MOTD message is stored in /etc/motd file.

MOTD is not only used in Unix/Linux but it become a common feature of the online services like IRC and game clients.

How to create MOTD for your Linux box?

The easiest way to create nice MOTD is to use linux_logo application, which can be installed using apt-get install linuxlogo (warning: it also changes default MOTD!).

Usage of linux_logo application

NOTE: My blogger template destroyed ASCII ART below, sorry :(

$ linux_logo  -k -a -b -L 26 -F "Host: #H (#V).\\n CPU: #T\\n RAM: #R"
              .-. 
        .-'``(|||) 
     ,`\ \    `-`.               88                         88 
    /   \ '``-.   `              88                         88 
  .-.  ,       `___:    88   88  88,888,  88   88  ,88888, 88888  88   88 
 (:::) :        ___     88   88  88   88  88   88  88   88  88    88   88 
  `-`  `       ,   :    88   88  88   88  88   88  88   88  88    88   88 
    \   / ,..-`   ,     88   88  88   88  88   88  88   88  88    88   88 
     `./ /    .-.`      '88888'  '88888'  '88888'  88   88  '8888 '88888' 
        `-..-(   ) 
              `-` 


Host: jambia (2.6.32-27-generic).
 CPU: Intel(R) Core(TM) Duo CPU      T2350  @ 1.86GHz
 RAM: 2GB

Get All Available logos in linux_logo package

Available logos (can be displayed using linux_logo -L list):
Available Built-in Logos:
 Num Type Ascii Name  Description
 1 Classic Yes aix  AIX Logo
 2 Classic Yes bsd  FreeBSD Logo
 3 Banner Yes bsd_banner FreeBSD Logo
 4 Classic Yes irix  Irix Logo
 5 Banner Yes solaris  The Default Banner Logos
 6 Banner Yes banner-simp Simplified Banner Logo
 7 Banner Yes banner  The Default Banner Logo
 8 Classic Yes classic-nodots The Classic Logo, No Periods
 9 Classic Yes classic-simp Classic No Dots Or Letters
 10 Classic Yes classic  The Default Classic Logo
 11 Classic Yes core  Core Linux Logo
 12 Banner Yes debian_banner Debian Banner (white) 
 13 Classic Yes debian_old Debian Old Penguin Logos 
 14 Classic Yes debian  Debian Swirl Logos
 15 Classic Yes gnu_linux Classic GNU/Linux
 16 Banner Yes mandrake_banner Mandrake(TM) Linux Banner
 17 Banner Yes mandrake Mandrakelinux(TM) Banner
 18 Banner Yes mandriva Mandriva(TM) Linux Banner
 19 Banner Yes pld  PLD Linux banner
 20 Banner Yes redhat  RedHat Banner (white) 
 21 Banner Yes slackware Slackware Logo
 22 Banner Yes sme  SME Server Banner Logo
 23 Banner Yes sourcemage_ban Source Mage GNU/Linux banner
 24 Banner Yes sourcemage Source Mage GNU/Linux large
 25 Banner Yes suse  SUSE Logo
 26 Banner Yes ubuntu  Ubuntu Logo

Do "linux_logo -L num" where num is from above to get the appropriate logo.

Checking out all available logos

for x in `seq 1 26` ; do 
 clear ; 
 linux_logo  -k -a -b -L $x -F "Host: #H (#V).\\n CPU: #T\\n RAM: #R" ; 
 sleep 2 ; 
done

Creating custom MOTD logo

FIGlet is a computer program that generates text banners, in a variety of typefaces, composed of letters made up of conglomerations of smaller ASCII characters. Installation of FIGlet is really simple apt-get install figlet.

Creating more personalized MOTD using FIGlet

$ figlet -f slant jambia
       _                 __    _      
      (_)___ _____ ___  / /_  (_)___ _
     / / __ `/ __ `__ \/ __ \/ / __ `/
    / / /_/ / / / / / / /_/ / / /_/ / 
 __/ /\__,_/_/ /_/ /_/_.___/_/\__,_/  
/___/                                 

My MOTD (linux_logo + FIGlet)

              .-
        .-'``(|||)
     ,`\ \    `-`.           _                 __    _
    /   \ '``-.   `         (_)___ _____ ___  / /_  (_)___ _
  .-.  ,       `___:       / / __ `/ __ `__ \/ __ \/ / __ `/
 (:::) :        ___       / / /_/ / / / / / / /_/ / / /_/ /
  `-`  `       ,   :   __/ /\__,_/_/ /_/ /_/_.___/_/\__,_/
    \   / ,..-`   ,   /___/
     `./ /    .-.`
        `-..-(   )
              `-`

Host: jambia (2.6.32-27-generic).
 CPU: Intel(R) Core(TM) Duo CPU      T2350  @ 1.86GHz
 RAM: 2GB

Additional references:

19 Jan 2011

Group and count files by type in Perl (using file magic numbers)

Perl File::Find - passing parameters to "wanted" callback - basics

Using object oriented libraries with File::Find module is not so obvious like it should be. But there is a way to do it in simple way - use closures in Perl. Now you probably think, "stop the bla bla talk, and show me code!", so here it is:

my $prefix = "Found ";
find(
   {
      wanted => sub {
         print $prefix.$File::Find::name."\n";
      },
      follow   => 1,
      no_chdir => 1,
   },
   "." # dir to search
); # find

How to determine file type ignoring extension?

Answer is simple - using magic numbers.

What is magic number

The term magic number is a constant numerical or text value used to identify a file format or protocol.

How connect file type with magic number in Perl?

There is variety of modules implementing this, for sample application presented below I have used File::Type module. Installation of this module on Ubuntu is extreme easy: apt-get install libfile-type-perl.

15 Jan 2011

The regular expressions guides

Wikipedia defines regular expressions like this: In computing, a regular expression, also referred to as regex or regexp, provides a concise and flexible means for matching strings of text, such as particular characters, words, or patterns of characters. A regular expression is written in a formal language that can be interpreted by a regular expression processor, a program that either serves as a parser generator or examines text and identifies parts that match the provided specification.

Regular expressions - theory

Online regular expressions tester

Putting website into maintenance mode

When you have dynamic website with loyal users you should provide at least decent support and service for them. One of things you have to do when working on startup is maintenance of service, adding new features and so on .... How to do it right?

How to create good maintenance mode?

First of all do not let your users to lost some of their activity on site, when you are putting your website into maintenance mode you should ensure that all GET/POST request will be served when website will be back online.

How to do it? There are many ways to achieve that, prefer using .htaccess with additional php file to keep user state.

Here is draft of my solution (insecure draft) - before using it you should tune code below to your needs, this article should be treated like inspiration, not a solution to your problems.

Before performing this operations make sure that you have mod_rewrite enabled:

sudo a2enmod rewrite # enables mod rewrite
The .htaccess maintenance mode file:
## make sure nobody gets the htaccess files
<Files ~ "^[\._]ht">
   Order allow,deny
   Deny from all
   Satisfy All
</Files>

ErrorDocument 503 /maint.php
<IfModule mod_rewrite.c>
   RewriteEngine On
   RewriteBase /
   RewriteCond %{REQUEST_URI}  !^/maint.php$
   RewriteRule .* /maint.php [NC,L]
   ErrorDocument 503 /maint.php
</IfModule>
The maint.php file:
<html>
<head>
<title>Site is under maintenance</title>
</head>
<body>
<?php 
   #echo '<pre>'.print_r($_SERVER, 1).'</pre>';
   if ( $_SERVER['REQUEST_METHOD'] == 'GET' ) {
      echo '<a href="'.$_SERVER['REQUEST_URI'].'">Reload</a>' ;
   } else if ( $_SERVER['REQUEST_METHOD'] == 'POST' ) {
      echo '<pre>POST\n\n'.print_r($_POST, 1).'</pre>';
   }
?>
</body>
</html>
For more information about mod_rewrite check out:

12 Jan 2011

Cron tip: prevent running multiple copies of the same job in HA environment

All cron jobs should prevent themselves from started in multiple copies!

This can be done using file locks (do not just create files, normal files left after script crash can prevent it from starting again and executing). File locks can also serialize jobs from many servers or even limit concurrency between machines.

In High Availability environments where are many servers, jobs must be run on certain time to ensure that jobs should be queued on all servers in cluster. But there is catch - when we configure our environment there can be situation where on every machine one copy of our script will be run on every server - that is bad. To prevent this all servers should have one shared filesystem where you can store your lockfiles and acquire lock for processes.

File locks from Bash

  • flock -xn /var/lock/run.sh.lock -c run.sh
  • flock -x /var/lock/run.sh.lock -c run.sh
  • flock -x -w 30 /var/lock/run.sh.lock -c run.sh

man flock

-x, -e, --exclusive
Obtain an exclusive lock, sometimes called a write lock. This is the default.
-n, --nb, --nonblock
Fail (with an exit code of 1) rather than wait if the lock can‐ not be immediately acquired.
-w, --wait, --timeout seconds
Fail (with an exit code of 1) if the lock cannot be acquired within seconds seconds. Decimal fractional values are allowed.

9 Jan 2011

Quick dump or backup all databases in MySQL server

Dump all databases to one file

mysqldump -u root -pPASSWORD --all-databases | gzip > database_`date -I`.sql.gz

Dump all MySQL databases to many separate files (one file for database)

You can use this script mydump2many.sh.

8 Jan 2011

Sending emails from Perl via SMTP server

Email inbox could be a good place to store many informations including logs from your services. Common usage of email may be sending notification about your service failures. Email systems are based on a store-and-forward model. Email servers accept, forward, deliver and store messages. So users don't need be online simultaneously for exchange messages, online presence is only required for time that takes to send or receive messages. So when you send some message it will be delivered to recipient sooner or later.

Dependicies (Debian/Ubuntu)

Script below depends on Mail::Sender module, which may be installed using apt-get install libmail-sender-perl.

Sending emails from Perl


#!/usr/bin/perl

use strict;
use warnings;

use Mail::Sender;

my $smtp = 'smtp.poczta.onet.pl';
my $smtp_port = 587;

my $from = 'src@poczta.onet.pl';
my $pass = 'password';

my $dmail = 'dest@mail.com';
my $subj = 'Hello';
my $msg = 'MSG!';


my $sender = new Mail::Sender {
        smtp => $smtp,
        from => $from,
        auth => 'LOGIN',
        port => $smtp_port,
        authid => $from,
        authpwd => $pass,
 on_errors => 'undef',

# debug_level => 4,
# debug => \*STDERR,
} ;

$sender->MailMsg({to => $dmail,subject => $subj,msg => $msg }) 
 or die "Unable to send mail: ".$Mail::Sender::Error ."\n";

Other modules/additional references

Perl daemon: the hard way

From Wikipedia: In Unix and other computer multitasking operating systems, a daemon is a computer program that runs in the background, rather than under the direct control of a user; they are usually initiated as background processes. Typically daemons have names that end with the letter "d": for example, syslogd.

The common method for a process to become a daemon involves:

  • Dissociating from the controlling tty
  • Becoming a session leader
  • Becoming a process group leader
  • Staying in the background by forking
  • Setting the root directory ("/") as the current working directory so that the process will not keep any directory in use that may be on a mounted file system (allowing it to be unmounted).
  • Changing the umask to 0 to allow open(), creat(), et al. calls to provide their own permission masks and not to depend on the umask of the caller
  • Closing all inherited open files at the time of execution that are left open by the parent process, including file descriptors 0, 1 and 2 (stdin, stdout, stderr). Required files will be opened later.
  • Using a logfile, the console, or /dev/null as stdin, stdout, and stderr
In Perl are couple ways to write daemon (TMTOWTDI), you can use and mix already tested modules and have work done in hours instead of days.

Modules which could make writing daemon in Perl much easier

But sometimes there are no possibility to use modules, so you have to make all by yourself. Code below shows how to do it. This code was not tested in production environment it is only effect of evening coding. This code is also provided for download daemon.pl.