Bash: read csv file

Here is the script to read csv file in shell script.

#!/bin/bash
FILENAME=test.csv
OLDSEP=$IFS
IFS=,

[ ! -f $FILENAME ] && { echo "File not found : $FILENAME"; exit 99; }

while read field1 field2 field3 field4
do
    echo "$field1 : $field2 : $field3 : $field4"
done < $FILENAME

IFS=$OLDSEP

Cannot remove directory: File exists

I was trying to remove a directory and got error: "cannot remove directory 'logs/': File exists"

$chmod -R 777 logs/
$rm -rf logs/
$rm: cannot remove directory 'logs/': File exists

Checked the logs folder content and found out that there was .nfs*** file in this, when I manually deleted this file, it was recreated immediately and the timestamp of the file was not current, it was same of which I just deleted.

$ls -lrta
total 20K
drwxrwxrwx 3 test vishnu 4.0K May  3 03:27 ..
-rwxrwxr-x 1 test vishnu  11K May  3 03:27 .nfs29682
drwxrwxr-x 2 test vishnu 4.0K Jun  4 23:17 .

To solve this issue, we need to find out which process is using this file and delete that process.

$/usr/sbin/fuser -u logs/.*
logs/.:    
logs/..:    
logs/.nfs29682:     23142m(test)

23142 is the process which is using this directory, we need to kill it.

kill -9 23142

now delete the directory, it will be deleted successfully.

$rm -rf logs/
$

Automatically login a ssh session using putty

Putty is used to connect to remote server and for day to day tasks developer login into remote machine using putty. User have to input username/password for every login. This process can be automated to save some time.

1. Downlaod PuTTY

2. Download PuTTYgen

3. Run PuTTYgen

4. Select SSH-2 DSA as the "Type of Key" to generate

5. Click on generate button and move your mouse around so that randomness is generated

6. Click on “Save Private Key” and save it to your computer

7. Copy the entire content inside the box to your clipboard (this is your generated public key).

8. Login to your SSH server using putty (by using user name and password)

9. Create file ~/.ssh/authorized_keys and put your public key on a single line (from step 7)

10. Make this file readable (chmod +r )

11. Open PuTTY, navigate to Connection->Data and fill in the auto-login username

12. Navigate to Connection->SSH->Auth and under Private-key, browse to the file you had saved earlier on your computer. (from step 6)

You are done. Now whenever you login into your remote machine, it will login automatically.

Unix: Get a file's name, extension and directory name

file="/usr/local/bin/test.sh"

# get extension; everything after last '.'
ext=${file##*.}
# will return 'sh'

# everything after last '/'
basename=${file##*/}
# will return 'test.sh', similar to 'basename'

# everything before last '/'
basename=${file%/*}
# will return '/usr/local/bin', similar to 'dirname'

# back one directory
basename=${file%/*/*}
# will return '/usr/local'

Solaris: colorize stuff

If you want to colorize directory listing in solaris, just add the following alias in your .bashrc

alias ls='ls -h --color=auto'

If you want to colorize vim editor, add below in your .bashrc

export TERM=dtterm (for solaris)
export TERM=xterm (for linux)

Change the case of file contents

If you want to change case (either lower or upper case) of contents of a file use the tr command.

Change contents of filename to lower case
cat filename | tr "[:upper:]" "[:lower:]" > newfilename
or
cat filename | tr '[A-Z]' '[a-z]' > newfilename

Change contents of filename to upper case
cat filename | tr "[:lower:]" "[:upper:]" > newfilename
or
cat filename | tr '[a-z]' '[A-Z]' > newfilename

Sorting a csv file on a particular field

I have below csv file, which have around 5k lines in following format:

ABC,12345,test.com,56780,0.00,200.65,0.00,0.00,200.65
XYZ,54311,tset.com,69540,0.00,102.32,0.00,0.00,102.32
..
...

I wanted to sort this file on field6 so that i could extract my desired lines from the csv file. I did use below command which sorted my csv file on field no 6:

sort -n -r -t, +6 -7 vishnu.csv

-n Specifies numeric field
-r Reverse sorting
-t specifies seperator (here , is a seperator)
+6, -7 specifies column position

sorting the csv file if separator is tab
sort -t $'\t' -k5 -nr filename

compare two sorted files in unix

If you wanted to compare 2 sorted files then use comm command. comm command compares two sorted files line by line. This command reads the File1 and File2 as parameter and writes a three column output to standard output.

First Column: Lines that are only in File1
Second Column: Lines that are only in File2
Third Column: Lines that are in both File1 and File2

Syntax:

comm [-1] [-2] [-3 ] file1 file2

-1 Suppresses the display of the first column (lines in File1)
-2 Suppresses the display of the second column (lines in File2)
-3 Suppresses the display of the third column (lines both in File1 and File2)

Killing a defunct process

A defunct (or "zombie") process is a process that isn't running, isn't eligible to run, and takes up no system resources. It's actually a process that has exited, but its parent has not called wait() in order to find out its exit status.

defunct processes can't be killed since they are already dead. To make them disappear you have to kill their parent process.

Find the parent process id of the defunct process and then kill that parent process:

ps -fe | grep defunctprocess | awk '{print $3}'

kill -9 parentprocessid

Reverse the contents of a file

If you wanted to reverse the contents of a file, use below perl script:


#!/usr/bin/perl

$filename = "a.java";
open(INFILE, "<", $filename)
or die "Can't open $filename for reading: $!\n";
my(@content) = <INFILE>;
close(INFILE);
while (@content) {
my($line) = pop(@content);
chomp $line;
print reverse($line)."\n";
}

Reversing a file

You have a file and you wanted to reverse it. How to do it? Guys...... its really very easy.. Use below commands:


tail -r filename
or
cat -n filename | sort -nr | cut -c8-

You can achieve the same by writing a perl script. See below perl script:


#!/usr/bin/perl

$filename = "a.java";
open(INFILE, "<", $filename)
or die "Can't open $filename for reading: $!\n";
my(@content) = <INFILE>;
close(INFILE);
while (@content) {
my($line) = pop(@content);
chomp $line;
print "$line \n";
}

Linux time command

The time command runs the specified program command with the given arguments. When command finishes, time writes a message to standard output giving timing statistics about this program run.

See below example:

[vishnu@solaris /mnt/usr/vishnu] time myProgram arg1

real 24m10.951s

user 6m2.390s

sys 0m15.705s

[vishnu@solaris /mnt/usr/vishnu]

Above command outputs the time taken by myProgram.

· real - Elapsed time from beginning to end of program

· user - time used by the program itself and any library subroutines it calls

· sys- time used by the system calls invoked by the program (directly or indirectly)

Read latest file in vi editor

In my daily routine on solaris box, i have to read the latest file in vi. For this i have to run below 2 commands:

ls -ltr
vi filename

I get the latest file name with ls -ltr command (last file in the list) and then vi to open this file.

I was sick of using 2 commands for this simple operation, so i thought of using a single command using combination of commands; following are the commands which I used for this operation:

vi `ls -ltr|tail -1|awk -F" " '{print $9}'` (includes directory listing)
vi `ls -ltr|grep -v ^d|tail -1|awk -F" " '{print $9}'` (ignores directory listing)
or
vi `ls -ltr|tail -1|cut -d":" -f2|cut -d" " -f2` (includes directory listing)
vi `ls -ltr|grep -v ^d|tail -1|cut -d":" -f2|cut -d" " -f2` (ignores directory listing)

vi Editor Commands

The VI editor is a screen-based editor used by many Unix users. The VI editor has powerful features to aid programmers, but many beginning users avoid using VI because the different features overwhelm them.

General startup
        To use vi: vi
        To exit vi and save changes: ZZ or :wq
        To exit vi without saving changes: :q!
        To enter vi command mode: [esc]
        Read the original file back in so that you can start over.: :e!


Counts
        A number preceding any vi command tells vi to repeat that command that many times.

Deleting
        x       delete character under cursor
        X       delete character before cursor
        dd      delete line under cursor
        dw      delete word under cursor
        db      delete word before cursor
        D       delete to the end of line from the current cursor position
        d^      delete from current cursor position to the beginning of the line
        d$      delete from current cursor position to the end of the line

Copy/Paste Text
        yy     copies line
        P      paste copied data before cursor
        p      paste copied data after cursor

Find Commands
        ?       finds a word going backwards
        /       finds a word going forwards
        f       finds a character on the line under the cursor going forward
        F       finds a character on the line under the cursor going backwards
        t       find a character on the current line going forward and stop one character before it
        T       find a character on the current line going backward and stop one character before it
        ;       repeat last f, F, t, T
        ,       repeat last f, F, t, T going backwards
        n       repeat last search given by ‘/’ or ‘?’
        N       repeat last search given by ‘/’ or ‘?’ going backwards

Miscellaneous Commands
        .       repeat last command
        u       undoes last command issued
        U       undoes all commands on one line
        xp      deletes first character and inserts after second (swap)
        J       join current line with the next line
        ^G      display current line number
        %       if at one parenthesis, will jump to its mate
        mx      mark current line with character x
        'x      find line marked with character x
        ^^      Go back to last file 

READING/WRITING FILES
        :r filename    Copies filename after cursor in file currently editing
        :n      Start editing next file in list
        :rew    rewind file list, start editing 1st file on argument list
        :w      Saves the current file without quitting

MOVING
        :#      Move to line #
        :$      Move to last line of file


Character/Line Formatting
        ~      Switch the case of the character under cursor
        <      Shifts the line upto where to the left by one shiftwidth
        <<     Shifts the current line to the left, and can be specified with a count
        >      Shifts the line upto where to the right by one shiftwidth
        >>     Shifts the current line to the right, and can be specified with a count
        J      Join the current line with the next one.

SHELL ESCAPE
        :!'CMD'      Executes CMD as a shell command
        :!sh         Fork shell in vi; hit ctrl-d to go back to vi

INSERTING
        r      replace character under cursor with next character typed
        R      keep replacing character until [esc] is hit
        i      insert before cursor
        I      insert from the beginning of line
        a      append after cursor
        A      append at end of line
        o      open line below cursor and enter append mode
        O      open line above cursor and enter append mode
        c      change until . "cc" changes the current line
        C      change to the end of line from the current cursor position
        s      substitute one character under the cursor and go into insert mode
        S      change an entire line


Screen Movement
        G      move to the last line in the file
        xG     move to line x
        z+     move current line to top of screen
        z      move current line to the middle of screen
        z-     move current line to the bottom of screen
        ^F     move forward one screen
        ^B     move backward one line
        ^D     move forward one half screen
        ^U     move backward one half screen
        ^R     redraw screen (does not work with vt100 type terminals)
        ^L     redraw screen (does not work with Televideo terminals)

Cursor Movement
        h      move left
        j      move down
        k      move up
        l      move right
        [return]    move to the beginning of next line
        $      last column on the current line
        0      move cursor to the first column on the current line
        ^      move cursor to the first nonblank column on the current line
        w      move to the beginning of the previous word or punctuation
        W      move past the next space
        b      move to the beginning of the previous word or punctuation mark
        B      move to the beginning of the previous word, ignores punctuation
        e      end of next word or punctuation mark
        E      end of next word, ignoring punctuation
        H      move cursor to the top of the screen
        M      move cursor to the middle of the screen
        L      move cursor to the bottom of the screen
        ^H     move cursor one space to the left
        ^J     move cursor down one line in same column
        ^M     move to the first character on the next line
        ^N     move cursor down one line in same column
        ^P     move cursor up one line in same column
        %      move cursor to the matching parenthesis or brace
        (      move cursor to the beginning of a sentence
        )      move cursor to the beginning of the next sentence
        {      move cursor to the preceding paragraph
        }      move cursor to the next paragraph
        |      move cursor to the column specified by the count
        +      move cursor to the first non-whitespace character in the next line
        -      move cursor to the first non-whitespace character in the previous line
        _      move cursor to the first non-whitespace character in the current line

Check all arguments of a running process

Many Java processes have long command lines, and as as a regular user, I may wish to determine that my java processes are running with the correct arguments or not.
unfortunately with the ps (/usr/bin/ps) command, users can only see the first 80 characters of command line, rest are ignored. So the user can not see the all command line arguments.

Solution:
1. Use pargs argument (ex. ps -eaf | pargs -a <pid>)
or
2. Use /usr/ucb/ps command ( /usr/ucb/ps -awwx | grep <pid>)

GNU Screen utility

Screen, this command might not be well known to many people but its one of those programs that you just can’t stop using once you’ve started. Previously I used VNC to connect the remote servers, but now i have stopped using it and use SCREEN on a regular daily basis and it’s really easy to use.
Screen program provides the following functionality:

• Remote terminal session management (detaching or sharing terminal sessions)
• Unlimited windows (unlike the hardcoded number of Linux virtual consoles)
• Scrollback buffer (not limited to video memory like Linux virtual consoles)
• Copy/paste between windows
• Split terminal (horizontally) into multiple regions
• Locking other users out of terminal
• Screen is an easy way to allow processes to continue running after the session is terminated, if you lose connection screen will save your spot

Following is the content of my ~/.screenrc file
************************************************************
startup_message off
vbell off
caption always “%{= bb}%{+b w}%n %h %=%t %c”
hardstatus alwayslastline “%-Lw%{= BW}%50>%n%f* %t%{-}%+Lw%<” activity “Activity in %t(%n)” shell -/bin/bash
************************************************************
Screen Commands
(screen) Start screen
(screen -S vishnu) Create a screen session with the name vishnu
(screen -r vishnu) reconnect to the session with the name vishnu
(screen -x ) Connect to an existing screen session
(screen -d ) detaching a screen session
(screen -r) reattaching the screen session
(Ctrl+a c ) New window
(Ctrl+a n ) Next Window
(Ctrl+a p ) Previous Window
(Ctrl+a ” ) Select window from list
(Ctrl+a Ctrl+a) Previous window viewed
(Ctrl+a <0-9> ) Select the numbered window
(Ctrl+a A ) Set window title
(Ctrl+a K ) Kill window
(Ctrl+a d ) Detach screen from terminal
(Ctrl+a x ) Lock Session
(Ctrl+a : ) Goto screen command prompt
(Ctrl+a ? ) Show key binding/command names
(Ctrl+s ) Pause the output on screen
(Ctrl+q ) Resume the output on screen
(Ctrl+a :escape ^Ww) Change key binding to w character
(Ctrl-a * ) List all currently attached displays. (displays)
(Ctrl-a Ctrl\) Kill all windows and terminate screen. (quit)
(Ctrl-a w ) List all windows. (windows)
(Ctrl-a h ) Write contents of the current window to the file hardcopy.n. (hardcopy)
(Ctrl-a H ) Begin/end logging of the current window to the file screenlog.n. (log)
(Ctrl-a ' ) Prompt for window name or number to switch to. (select)

Linux Cut command

Cut Command
------------

Divide a file into several parts (columns)

syntax:
cut [-b] [-c] [-f] list [-n] [-d delim] [-s] [file]

Examples:
--------

1. Let say you have a file test.txt which has colon(:) seperated data

406378:Sales:Itorre:Jan
031762:Marketing:Nasium:Jim
636496:Research:Ancholie:Mel
396082:Sales:Jucacion:Ed

If you want to print first set of data from each row, you can use cut command as follow:

cut -d":" -f1 test.txt

If you want to print just columns 1 to 6 of each line (the employee serial numbers), use the -c1-6 flag, as in this command

cut -c1-6 test.txt

Linux Tips

Repeat a command (yes 'date;sleep 5' | ksh)
Replace newline with comma (cut -d, -f1 vishnu.csv | tr '\n' ',')
Get some specific string from each line (grep "uid%3D" <fileName> |awk -F"uid%3D" ` {print $2}` | cut -d"%" -f1)
Pull first n characters of each line from a file (cut -c1-n file.txt > newfile.txt)
Count total number of lines in all specific files under a directory (find . -type f -name '*.as' -o -name '*.mxml' -o -name '*.java'| xargs cat |wc -l)
Find number of occurrences of a text in a file (grep text fileName |wc -l)
Display the top most process utilizing most CPU (top -b 1)
Show the working directory of a process ? (pwdx pid )
Display the parent/child tree of a process ? (ptree pid )
Display the no.of active established connections to localhost ? (netstat -a | grep EST)
How to create null file ? (cat /dev/null > filename1)
Display top ten largest files/directories ? (du -sk * | sort -nr | head)
Display disk usage (du -h)
How to save man pages to a file ? (man | col -b > filename )
Display the files in the directory by file size ? (ls -ltr | sort -nr -k 5)
Display the processes, which are running under yourusername ( ps -aef | grep username)
Display the all files recursively with path under current directory ? ( find . -depth -print)
Display the Disk Usage of file sizes under each directory in currentDirectory ? (du -k . | sort -nr)
List the files in current directory sorted by size ? (ls -l | grep ^- | sort -nr)

Special Shell Script Variables

There are some variables which are set internally by the shell and which are available to the user:

$1 - $9   These variables are the positional parameters.

$0        The name of the command currently being executed.

$#        The number of positional arguments given to this invocation of the shell.

$?        The exit status of the last command executed is given as a decimal string.
       When a command completes successfully, it returns the exit status of 0
       (zero), otherwise it returns a non-zero exit status.

$$        The process number of this shell - useful for including in filenames, to
       make them unique.

$!        The process id of the last command run in the background.

$-        The current options supplied to this invocation of the shell.

$*        A string containing all the arguments to the shell, starting at $1.

$@@       Same as above, except when quoted.