Posts

Showing posts from 2011

List all installed and available Perl modules

Code below provides list of modules which perl interpreter can find in runtime. Code#!/usr/bin/perl # # Author: Tomasz Gaweda # Date: 2011.12.23 17:00 # Purpose: Print all modules installed # in perl distribution # # Usage: perl showmodules.pl use strict; use warnings; use File::Find; my @modules; my $IgnorePatt = join("|",@INC) ;#$IgnorePatt =~ s/\|\.// ; find( sub { my $f = $File::Find::name; if ( $f =~ s/^(${IgnorePatt})\/(.*?)\.pm$/$2/ ) { $f =~ s/[\/\\]/::/g; push @modules, $f; } } , grep { -d } @INC ); print "There are [".scalar(@modules)."] modules that perl can locate on runtime...\n"; print "Printing List in Minute\n" ; sleep 3; print join("\n", sort{ lc $a cmp lc $b} @modules)."\n";Sample outputThere are [2346] modules that perl can locate on runtime... Printing List in Minute AcidRip::acidrip AcidRip::interface AcidRip::messages AcidRip::signals AE Algorithm::C3 Algorithm::Diff …

Encoding files "on the fly" with Blowfish using Perl

Some time ago friend of mine asked me if I could write him an CGI script which encodes his backup file and allows to download it using HTTP protocol.He is using some kind of VPS so my CGI app had some limitations:file must be encoded "on the fly" to save disk space (no temporary files)encoding should be memmory efficient - because VPS hasn't got very much RAM and backup files are quite largeencoding should be fast - so the best solution would be using XS moduleAfter some research I've created this two scripts: Warning: Scripts depends on external Perl modules which are available at CPAN and in Debian repository. apt-get install libcrypt-cbc-perl libcrypt-blowfish-perlBlowfish CGI encoding using Perl#!/usr/bin/perl use strict; use warnings; use CGI; use HTML::Entities; use Crypt::CBC; use File::Spec; my $as_cgi = 1; # if run from apachce set to 1 if CLI 0 my $key = "ala_ma_kota.123"; my $fName = './large_file.bin'; my $chunkSize = 1024; my $ou…

Using perl to extract files from large directory structure

When I work on Windows I use Activesite Active Perl to do some stuff. Recently I had to copy about 60 files from large directory structure to a new place (about 800 files in nasty directory structure). It was rather boring stuff to do, so I used perl to finish job because it would be pretty easy task to do using hashes (I had names of files given in text file). Unfortunately I had some problems on completion of job, because there was a problem with encoding of file path name on Windows.Below I provide simple script demonstration how to handle national characters in path using perl on Windows.#!/usr/bin/perl use strict; use warnings; use utf8; use Encoding; # Win32::Codepage::Simple is available in ActiveState Perl use Win32::Codepage::Simple qw(get_codepage get_acp); use File::Find; use File::Copy; use File::Path; use File::Basename; binmode STDOUT, ":utf8"; binmode STDERR, ":utf8"; my $dir = "./copy_from_here/"; my $dest = "./copy_here/…

How to manage and disable bash history

You can change Bash history behaviour changing environment variables. Here is simple reference about this settings.

Arrays in PostgreSQL quick example

Image
What is an array in database?PostgreSQL allows columns of a table to be defined as variable-length multidimensional arrays. Arrays of any built-in or user-defined base type, enum type, or composite type can be created. Arrays are not supper efficient way of doing things, but they are sometimes handy.Create some test dataSuppose we have table like this:CREATE TABLE survey_test ( id SERIAL, gender CHAR(1), school CHAR(2), -- can be: LO, TH, ZW q_9 VARCHAR(10) -- can be: [A-J]{0,10} );Where we store this data:

Securing Linux: few tips for good configuration of SSHD

Most people install sshd daemon using command apt-get install open-sshd, and then forgets about making it more secure. In this article I will point out few sshd options which you should have in your sshd config when going to production.

NWCHem on Ubuntu Linux

This article has been written as instruction for my friend how to run NWChem.What is NWChem?From Wikipedia: NWChem is an ab initio computational chemistry software package which also includes quantum chemical and molecular dynamics functionality. It was designed to run on high-performance parallel supercomputers as well as conventional workstation clusters. It aims to be scalable both in its ability to treat large problems efficiently, and in its usage of available parallel computing resources. Why should I use NWChem? Because it can handle: Biomolecules, nanostructures, and solid-stateFrom quantum to classical, and all combinationsGaussian basis functions or plane-wavesScaling from one to thousands of processorsProperties and relativityBefore NWChem compilationapt-get install build-essential gfortranNWChem compilationtar -xvzf ../Nwchem-6.0.tar.gz cd nwchem-6.0/ export NWCHEM_TOP=`pwd` cd src/ export FC=gfortran export NWCHEM_TARGET=LINUX64 make nwchem_config NWCHEM_MODULES=all ma…

"What time is it?" Using rdate/ntpdate to set time on Linux server

ntpdateNtpdate sets the local date and time on your machine by polling the Network Time Protocol (NTP - RFC 1305) server given as the server arguments to determine the correct time. The accuracy and reliability of ntpdate depends on the number of servers and the number of polls each time it is run and the interval between runs. Ntpdate can be run manually as necessary to set the host clock, or it can be run from the host startup script to set the clock at boot time. Using ntpdate with cron scripts is no substitute for the NTP daemon, which uses sophisticated algorithms to maximize accuracy and reliability while minimizing resource use. Finally, since ntpdate does not discipline the host clock frequency as does ntpd, the accuracy using ntpdate is limited.Installation of ntpdate is simple apt-get install ntpdate.Usage of ntpdatentpdate -t 3 pl.pool.ntp.org hwclock --systohcList of polish ntp serversntp.certum.plntp1.tp.plntp1.net.icm.edu.plvega.cbk.poznan.plHere is official list of…

Apache2 - create password protected directory

My friend requested article about password protected access to public directories using .htpasswd file. Configuration of this is quite simple. All you have to do is make sure that you have .htaccess and AuthConfig override enabled (on Ubuntu this is default AFAIK) files support enabled (somewhere in your /etc/apache2/apache.conf should be AllowOverride AuthConfig directive).

Creating ASCII ART MOTD

MOTD or "Message of The Day" in most Unix systems is displayed after successful login, but before execution of user shell. MOTD is used to send a common message to all system users (after login), in a more efficient manner than sending them all an e-mail message. MOTD message is stored in /etc/motd file.MOTD is not only used in Unix/Linux but it become a common feature of the online services like IRC and game clients.How to create MOTD for your Linux box?The easiest way to create nice MOTD is to use linux_logo application, which can be installed using apt-get install linuxlogo (warning: it also changes default MOTD!).Usage of linux_logo applicationNOTE: My blogger template destroyed ASCII ART below, sorry :($ linux_logo -k -a -b -L 26 -F "Host: #H (#V).\\n CPU: #T\\n RAM: #R" .-. .-'``(|||) ,`\ \ `-`. 88 88 / \ '``-. ` 88 88 .-. , …

Group and count files by type in Perl (using file magic numbers)

Perl File::Find - passing parameters to "wanted" callback - basicsUsing object oriented libraries with File::Find module is not so obvious like it should be. But there is a way to do it in simple way - use closures in Perl. Now you probably think, "stop the bla bla talk, and show me code!", so here it is:my $prefix = "Found "; find( { wanted => sub { print $prefix.$File::Find::name."\n"; }, follow => 1, no_chdir => 1, }, "." # dir to search ); # find How to determine file type ignoring extension?Answer is simple - using magic numbers.What is magic numberThe term magic number is a constant numerical or text value used to identify a file format or protocol.How connect file type with magic number in Perl?There is variety of modules implementing this, for sample application presented below I have used File::Type module. Installation of this module on Ubuntu is extreme easy: apt-get ins…

The regular expressions guides

Wikipedia defines regular expressions like this: In computing, a regular expression, also referred to as regex or regexp, provides a concise and flexible means for matching strings of text, such as particular characters, words, or patterns of characters. A regular expression is written in a formal language that can be interpreted by a regular expression processor, a program that either serves as a parser generator or examines text and identifies parts that match the provided specification.Regular expressions - theoryPerldoc page about regular expressionsGuide to regular expressionsParsing HTML using Regular Expressions - but i don't think it is good ideaHow to design regular expressions parser - tech paperOnline regular expressions testerPHP RegEXP testerJS RegEXP testerFlex3 RegEXP

Putting website into maintenance mode

When you have dynamic website with loyal users you should provide at least decent support and service for them. One of things you have to do when working on startup is maintenance of service, adding new features and so on .... How to do it right? How to create good maintenance mode?First of all do not let your users to lost some of their activity on site, when you are putting your website into maintenance mode you should ensure that all GET/POST request will be served when website will be back online.How to do it? There are many ways to achieve that, prefer using .htaccess with additional php file to keep user state.Here is draft of my solution (insecure draft) - before using it you should tune code below to your needs, this article should be treated like inspiration, not a solution to your problems.Before performing this operations make sure that you have mod_rewrite enabled:sudo a2enmod rewrite # enables mod rewrite The .htaccess maintenance mode file: ## make sure nobody gets t…

Cron tip: prevent running multiple copies of the same job in HA environment

All cron jobs should prevent themselves from started in multiple copies!This can be done using file locks (do not just create files, normal files left after script crash can prevent it from starting again and executing). File locks can also serialize jobs from many servers or even limit concurrency between machines.In High Availability environments where are many servers, jobs must be run on certain time to ensure that jobs should be queued on all servers in cluster. But there is catch - when we configure our environment there can be situation where on every machine one copy of our script will be run on every server - that is bad. To prevent this all servers should have one shared filesystem where you can store your lockfiles and acquire lock for processes.File locks from Bashflock -xn /var/lock/run.sh.lock -c run.shflock -x /var/lock/run.sh.lock -c run.shflock -x -w 30 /var/lock/run.sh.lock -c run.shman flock-x, -e, --exclusiveObtain an exclusive lock, sometimes called a write loc…

Quick dump or backup all databases in MySQL server

Dump all databases to one filemysqldump -u root -pPASSWORD --all-databases | gzip > database_`date -I`.sql.gzDump all MySQL databases to many separate files (one file for database) You can use this script mydump2many.sh.

Sending emails from Perl via SMTP server

Email inbox could be a good place to store many informations including logs from your services. Common usage of email may be sending notification about your service failures. Email systems are based on a store-and-forward model. Email servers accept, forward, deliver and store messages. So users don't need be online simultaneously for exchange messages, online presence is only required for time that takes to send or receive messages. So when you send some message it will be delivered to recipient sooner or later. Dependicies (Debian/Ubuntu)Script below depends on Mail::Sender module, which may be installed using apt-get install libmail-sender-perl. Sending emails from Perl #!/usr/bin/perl use strict; use warnings; use Mail::Sender; my $smtp = 'smtp.poczta.onet.pl'; my $smtp_port = 587; my $from = 'src@poczta.onet.pl'; my $pass = 'password'; my $dmail = 'dest@mail.com'; my $subj = 'Hello'; my $msg = 'MSG!'; my $sender =…

Perl daemon: the hard way

From Wikipedia: In Unix and other computer multitasking operating systems, a daemon is a computer program that runs in the background, rather than under the direct control of a user; they are usually initiated as background processes. Typically daemons have names that end with the letter "d": for example, syslogd. The common method for a process to become a daemon involves:Dissociating from the controlling ttyBecoming a session leaderBecoming a process group leaderStaying in the background by forkingSetting the root directory ("/") as the current working directory so that the process will not keep any directory in use that may be on a mounted file system (allowing it to be unmounted).Changing the umask to 0 to allow open(), creat(), et al. calls to provide their own permission masks and not to depend on the umask of the callerClosing all inherited open files at the time of execution that are left open by the parent process, including file descriptors 0, 1 and 2 (…