Retrieve web pages using perl

Here is the sample script to fetch web pages using perl.

#!/usr/local/bin/perl -w
use strict;
use LWP 5.64;
use URI;

my $browser = LWP::UserAgent->new;
$browser->cookie_jar({});

my @browserHeaders = (
'User-Agent' => 'Mozilla/4.0; (compatible; MSIE 6.0; Windows NT 5.1; en-US)',
'Accept-Language' => 'en-US',
);

my $response;
$response = $browser->get("http://vishnuagrawal.blogspot.com",@browserHeaders);

if ($response->is_success) {
print $response->content;
} else {
print $response->error_as_HTML, "\n";
print $response->status_line, "\n";
}

print $response->code(), "\n";
print $response->message() , "\n";
print $response->header('content-type'), "\n";

my $url = URI->new("http://www.softwareqa.com");
$response = $browser->get($url,@browserHeaders);

print $response->content;

Find the day diff between two days

Following is the shell script solution for getting number of days between two dates:

#/bin/sh
date1=`date +%s -d $1`
date2=`date +%s -d $2`
diff=`expr $date2 - $date1`
diff_days=`expr $diff / \( 60 '*' 60 '*' 24 \)`
echo $diff_days


[vishnu@ shell]# ./daydiff.sh 2010-08-02 2010-09-15
44
[vishnu@ shell]#

Here is the perl solution for the same:

#!/bin/perl
use strict;
use warnings;
use Time::Local;

my $date1 = $ARGV[0];
my $date2 = $ARGV[1];

my @formatted_date1 = split(/-/, $date1);
my @formatted_date2 = split(/-/, $date2);

my $local_date1 = timelocal(0, 0, 0, $formatted_date1[2], $formatted_date1[1], $formatted_date1[0]);
my $local_date2 = timelocal(0, 0, 0, $formatted_date2[2], $formatted_date2[1], $formatted_date2[0]);

my $diffSeconds = $local_date2- $local_date1;
my $diffDays = $diffSeconds / (60 * 60 * 24);
print "Day diff :: $diffDays\n";

[vishnu@ shell]# ./daydiff.pl 2010-08-02 2010-09-15
Day diff :: 43
[vishnu@ shell]#

Here is modified version of above shell script, if you want to find out time difference in seconds

#!/bin/bash

d1=`date +%s -d "$1"`
d2=`date +%s -d "$2"`
((diff_sec=d2-d1))
echo "Diff Seconds : $diff_sec"

[vishnu@ shell]#./date.sh "2011-07-19 22:44:34" "2011-07-20 02:04:14"
Diff Seconds : 11980
[vishnu@ shell]#

Run sql query from command line

We all know to run sql scripts (.sql files) from command prompt but sometimes we need to run a sql query from command prompt. It is sometimes necessary when we have to fetch a query result from a script (shell/perl) and process it. By following way, we can run a sql query from command prompt.

echo "select * from dual;" | sqlplus -S user/pwd\@sid

Send Email from perl script

Sometimes we do not have enough privileges to install perl modules on the host system and we have to work with ideal installation of perl. In that case, simpler way to send emails using perl is to use sendmail command. Following function can be used to send emails in perl.

sub sendEmail
{
($from, $to, $subject, $message) = @_;
my $sendmail = '/usr/lib/sendmail';
open(MAIL, "|$sendmail -oi -t");
print MAIL "From: $from\n";
print MAIL "To: $to\n";
print MAIL "Subject: $subject\n\n";
print MAIL "$message\n";
close(MAIL);
}

above function can be used as below:

sendEmail("vishnu\@test.com", "vishnu\@mytest.com", "Test sendEmail.", "Testing sendEemail function.");

Compare two files line by line

If your are looking for comparing two files(may be any CSV file) and wanted to find out which liens are common or unique to files. See below script:

Publish at Scribd or explore others: Internet & Technolog Research perl

PERL round function

Following is the round function in PERL, which you can put into your script and actually call to round numbers instead of cutting them with ceil or cut.

In the following round function, you can specify the number to round and the number of digits to show after the decimal.

sub round {
 my $number = shift || 0;
 my $dec = 10 ** (shift || 0);
 return int( $dec * $number + .5 * ($number <=> 0)) / $dec;
}

$result = round(123.4567,3);

it would return 123.457

Perl : Get filename or directory name from an absolute path

Use Perl's File module to get Filename or directory name from an absolute path. Below is an example of how to do this:

#!/usr/bin/perl
use File::Basename;

print getDirPath("/mnt/users/vishnu/blog/PerlPost.html");
print getFileNameWithoutExt("/mnt/users/vishnu/blog/PerlPost.html");
print getFileName("/mnt/users/vishnu/blog/PerlPost.html");

sub getDirPath() {
return dirname($_[0]);
}

sub getFileNameWithoutExt() {
my $file = basename($_[0]);
$file =~ s/\.[^.]*$//;
return $file;
}

sub getFileName() {
return basename($_[0]);
}


Output of above program would be:
/mnt/users/vishnu/blog
PerlPost
PerlPost.html

Perl error: Can't locate module in @INC

If you get a Perl error message like Can't locate module.pm in @INC, this message means that the Perl module you're trying to include (like the module named module) can't be found in Perl's include path, which is represented by the variable named @INC.

Perl uses an @INC variable that contains the directories it will search in when you specify that you want to include a module, but you can't look at it like an environment variable. Although Perl doesn't have something like the CLASSPATH environment variable you can easily look at, you can print the @INC variable with a simple one-line command. The following Perl command shows the directories that Perl looks in when it tries to load modules:

perl -e 'print join("\n", @INC);'

Including a perl module your @INC include path

There are different ways by which you can add your perl module in @INC include path
1. Specify it in PERL5LIB environment variable
export PERL5LIB=path

2. Perl takes -I option where you can specify the additional directories where it should search for modules.
perl -I path myscript.pl

3. Modify your Perl program to find the module by adding below line near the top of Perl programs. This simple line of code tells your Perl program to add this directory to it's @INC search directory.
use lib "path"

Perl: FileDiff utility

I have 2 csv files (pipe seperated), one column (basically an Id field) is common in both the files. I need to compare both the files on the basis of this common field and print the lines which has common Id, Lines with id which is only in file1 and vice-versa. I thought up for writing perl script for this task. See below script.

Publish at Scribd or explore others: perl Technology

Group rows of a csv file with sum of column values

You have heard about Group functionality in Microsoft Excel, which group the rows on a common value and does the sum of other field. Same thing can be done in unix using awk very easily. See below example:

[ ~]# cat test.csv
a,5
b,10
a,4
c,8
b,9
b,4
[ ~]# awk -F"," '{sum[$1] += $2 } END { for(n in sum)print n"=" sum[n] }' test.csv
a=9
b=23
c=8

Perl version of above script can be seen here:

Perl: Date formatting

If you need to convert Date from YYYY-MM-DD format to MM/DD/YYYY format, use below perl code.

$str="2008-10-13";
$str =~s/^(\d{4})\-(\d{2})\-(\d{2})/$2\/$3\/$1/;
print $str;

will output:
10/13/2008

Perl : Formatting a number to currency format

sub formatCurrency {
my $number = sprintf "%.2f", shift @_;
# Add one comma each time through the do-nothing loop
1 while $number =~ s/^(-?\d+)(\d\d\d)/$1,$2/;
# Put the dollar sign in the right place
$number =~ s/^(-?)/$1\$/;
return $number;
}

Make INSERT statement syntax from a csv file using perl

I need to read a csv file and insert the data from csv to oracle. I created below perl script which will read the csv file line by line and will create INSERT statement accordingly.

#!/usr/bin/perl

$filename = "test.csv";
open(INFILE, "<", $filename)
or die "Can't open $filename for reading: $!\n";
my(@lines) = <INFILE>;
my($line1) = $lines[0];
my($line2) = $lines[1];

print "$line1 \n";
print "$line2 \n";

close(INFILE);

my($tablename) = "mytable";
my($createquery) = "Create table " . $tablename . "(";

my(@var1) = split("\t", $line1);
my(@var2) = split("\t", $line2);
my($i) = 0;
for $item1 (@var1){
chomp($item1);
$createquery = $createquery . $item1 . " ";
$data = $var2[$i++];
print "$item1\t : $data\n";
if(checkData($data) eq "int"){
$createquery = $createquery . "number(10,0), ";
}
elsif(checkData($data) eq "float"){
$createquery = $createquery . "number(10,2), ";
}
else{
$createquery = $createquery . "varchar2(50), ";
}
}
$createquery = substr($createquery,0, -2);
$createquery = $createquery . ");";
print "$createquery\n";

$i = 0;
for $dataline (@lines){
if ( $i == 0){
$i = 1;
next;
}
chomp($dataline);
my(@dataitem) = split("\t", $dataline);
my($insertquery) = "INSERT INTO " . $tablename . " VALUES (";
print "$dataitem\n";
for $items (@dataitem){
if( (checkData($items) eq "int") || (checkData($items) eq "float") ){
$insertquery = $insertquery . $items .",";
}
else{
$insertquery = $insertquery . "\"" . $items . "\"" .",";
}
}
$insertquery = substr($insertquery, 0, -1);
$insertquery = $insertquery . ")";
print "$insertquery\n";
}

sub checkData(){
if(@_[0] =~ /^-?\d+$/){
return "int";
}
elsif(@_[0] =~ /^-?(?:\d+(?:\.\d*)?|\.\d+)$/){
return "float";
}
else{
return "string";
}
}

Make CREATE TABLE syntax from a csv file using perl

I have a csv file and i have to make the CREATE TABLE statement (which later will be used to create an Oracle table) by reading first and second line of the csv file. First line in the csv file is column headers and the second line is actual data. By reading the first line, i will create column name and by reading the second line, i'll decide the data type for the column.
Below is the perl script for the same: (i am assuming that data in the csv file is tab seperated)

#!/usr/bin/perl

$filename = "test.csv";
open(INFILE, "<", $filename) or die "Can't open $filename for reading: $!\n";
@lines = <INFILE>;
$line1 = $lines[0];
$line2 = $lines[1];

print "$line1 \n";
print "$line2 \n";

close(INFILE);

$tablename = "mytable";
$query = "Create table " . $tablename . "(";

@var1 = split("\t", $line1);
@var2 = split("\t", $line2);
$i = 0;
for $item1 (@var1){
chomp($item1);
$query = $query . $item1 . " ";
$data = $var2[$i++];
print "$item1\t : $data\n";
if($data =~ /^-?\d+$/){
$query = $query . "number(10,0), ";
}
elsif ($data =~ /^-?(?:\d+(?:\.\d*)?|\.\d+)$/){
$query = $query . "number(10,2), ";
}
else{
$query = $query . "varchar2(50), ";
}
}
$query = substr($query,0, -2);
$query = $query . ");";
print "$query\n";

Read data from a csv file using perl (when the column count is not known)

I need to read a csv file, and process each and every column data (i do not know the column count before processing the file)

#!/usr/local/bin/perl

if (scalar(@ARGV) < 1)
{
print "Please Specify the file name to be read \n";
exit
}

$file = $ARGV[0];

open FILE, "< $file" or die "Can't open test.csv : $!";
while(<FILE>){
chop;
my(@columns)= split("\t");
my($colnum)=$#columns;
# print "Number of Column in this row is : $colnum\n";
foreach $col(@columns){
print "$col \t";
}
print "\n";
}
close FILE;

Read data from a csv file using perl (when the column count is known)

I need to read a csv file, and process each and every column data (i know the column count before processing the file)

#!/usr/local/bin/perl

open FILE, "< test.csv" or die "Can't open test.csv : $!";
while(){
chomp;
($col1,$col2,$col3,$col4) = split ("\t");
print "$col1 \t";
print "$col2 \t";
print "$col3 \t";
print "$col4 \t";
print "\n";
}
close FILE;

Check datatype of a argument in perl

I need to pick few lines from a particular file and check the data type of values of a particular column. Following is the perl script for the same:

#!/usr/bin/perl

$filename = "test.csv";
open(INFILE, "<", $filename) or die "Can't open $filename for reading: $!\n";
@lines = <INFILE>;
$line1 = $lines[0];
$line2 = $lines[1];

print "$line1 \n";
print "$line2 \n";

close(INFILE);

@var1 = split("\t", $line2);
for $item (@var1){
if($item =~ /\D/){
print "$item\t : Has Non Digits\n";
}
elsif ($item =~ /^\d+$/){
print "$item\t : Whole Number\n";
}
elsif ($item =~ /^-?\d+$/){
print "$item\t : Integer\n";
}
elsif ($item =~ /^[+-]?\d+$/){
print "$item\t : +/- Integer\n";
}
elsif ($item =~ /^-?\d+\.?\d*$/){
print "$item\t : Real Number\n";
}
elsif ($item =~ /^-?(?:\d+(?:\.\d*)?|\.\d+)$/){
print "$item\t : Float Number\n";
}
else{
print "$item\t : NAN\n";
}
}

Print particular line/s from a file using perl

Method 1:

#!/usr/bin/perl

$filename = "test.csv";
open(INFILE, "<", $filename)
or die "Can't open $filename for reading: $!\n";

$line_number = 1;
while () {
$line = $_;
last if $. == $line_number;
}
if ($. != $line_number) {
die "Didn't find line $line_number in $filename\n";
}
print;

close(INFILE);



Method 2:

#!/usr/bin/perl

$filename = "test.csv";
open(INFILE, "<", $filename)
or die "Can't open $filename for reading: $!\n";
@lines = ;
$line1 = $lines[0];
$line2 = $lines[1];

print "$line1 \n";
print "$line2 \n";

close(INFILE);

Read data from a file in perl

Below is the sample perl script, if you want to read some data from a file and pass it to as a argument in your perl script.

In my case, I have some object ids in a file and for each id I have to run oocheck command.

#!/usr/local/bin/perl

open FILE, "testoids" or die $!;

while ()

{

chomp $_;

systemCmd("oocheck -id $_");

}

close(FILE);

sub systemCmd {

my($cmd) = @_;

print("$cmd\n");

(system($cmd) == 0) || die ("cmd $cmd failed");

}

Read properties from ant build file using perl

I want to read some properties from my ant build file in my perl script. I wrote a sample perl script for it. below is the script: