Bash: read csv file

Here is the script to read csv file in shell script.

#!/bin/bash
FILENAME=test.csv
OLDSEP=$IFS
IFS=,

[ ! -f $FILENAME ] && { echo "File not found : $FILENAME"; exit 99; }

while read field1 field2 field3 field4
do
    echo "$field1 : $field2 : $field3 : $field4"
done < $FILENAME

IFS=$OLDSEP

Cannot remove directory: File exists

I was trying to remove a directory and got error: "cannot remove directory 'logs/': File exists"

$chmod -R 777 logs/
$rm -rf logs/
$rm: cannot remove directory 'logs/': File exists

Checked the logs folder content and found out that there was .nfs*** file in this, when I manually deleted this file, it was recreated immediately and the timestamp of the file was not current, it was same of which I just deleted.

$ls -lrta
total 20K
drwxrwxrwx 3 test vishnu 4.0K May  3 03:27 ..
-rwxrwxr-x 1 test vishnu  11K May  3 03:27 .nfs29682
drwxrwxr-x 2 test vishnu 4.0K Jun  4 23:17 .

To solve this issue, we need to find out which process is using this file and delete that process.

$/usr/sbin/fuser -u logs/.*
logs/.:    
logs/..:    
logs/.nfs29682:     23142m(test)

23142 is the process which is using this directory, we need to kill it.

kill -9 23142

now delete the directory, it will be deleted successfully.

$rm -rf logs/
$

Remove a file whose name begins with "-"

Due to a bug in our product the log file was being created with the name "-server.log" (instead of "host-server.log"). Once the issue was fixed, i was trying to remove the file. Tried single quote/double quote/escape, but nothing worked. Finally after googling, found solution for it. Since the file name begins with the "-", all the unix commands treats the file name itself as a parameter to the command. To make it work put -- or ./ before the file name.

example:
rm -- -server.log
rm ./-server.log

Change the case of file contents

If you want to change case (either lower or upper case) of contents of a file use the tr command.

Change contents of filename to lower case
cat filename | tr "[:upper:]" "[:lower:]" > newfilename
or
cat filename | tr '[A-Z]' '[a-z]' > newfilename

Change contents of filename to upper case
cat filename | tr "[:lower:]" "[:upper:]" > newfilename
or
cat filename | tr '[a-z]' '[A-Z]' > newfilename

Memory usage by process

Solaris pmap command provides the memory usage of a process. This command tells you how much of memory a particular process is using and how much of there particular threads/heap are using. It's a very useful command to debug cases where you have to find that who is consuming more memory in your process.

pmap <PID>

example$ pmap 102905
102905: sh
00010000 192K r-x-- /usr/bin/ksh
00040000 8K rwx-- /usr/bin/ksh
00042000 40K rwx-- [ heap ]
FF180000 664K r-x-- /usr/lib/libc.so.1
FF236000 24K rwx-- /usr/lib/libc.so.1
FF23C000 8K rwx-- /usr/lib/libc.so.1
FF250000 8K rwx-- [ anon ]
FF260000 16K r-x-- /usr/lib/en_US.ISO8859-1.so.2
FF272000 16K rwx-- /usr/lib/en_US.ISO8859-1.so.2
FF280000 560K r-x-- /usr/lib/libnsl.so.1
FF31C000 32K rwx-- /usr/lib/libnsl.so.1
FF324000 32K rwx-- /usr/lib/libnsl.so.1
FF340000 16K r-x-- /usr/lib/libc_psr.so.1
FF350000 16K r-x-- /usr/lib/libmp.so.2
FF364000 8K rwx-- /usr/lib/libmp.so.2
FF380000 40K r-x-- /usr/lib/libsocket.so.1
FF39A000 8K rwx-- /usr/lib/libsocket.so.1
FF3A0000 8K r-x-- /usr/lib/libdl.so.1
FF3B0000 8K rwx-- [ anon ]
FF3C0000 152K r-x-- /usr/lib/ld.so.1

Sorting a csv file on a particular field

I have below csv file, which have around 5k lines in following format:

ABC,12345,test.com,56780,0.00,200.65,0.00,0.00,200.65
XYZ,54311,tset.com,69540,0.00,102.32,0.00,0.00,102.32
..
...

I wanted to sort this file on field6 so that i could extract my desired lines from the csv file. I did use below command which sorted my csv file on field no 6:

sort -n -r -t, +6 -7 vishnu.csv

-n Specifies numeric field
-r Reverse sorting
-t specifies seperator (here , is a seperator)
+6, -7 specifies column position

sorting the csv file if separator is tab
sort -t $'\t' -k5 -nr filename

Set Terminal Title

If you want the Terminal window title to be set automatically to the remote zone you have logged into, add this to your .bashrc file on the remote machine.


# Terminal settings
PS1="[\u@\h \W]# "

if [ -n "$PS1" ]; then
echo -n -e "\033]0;`hostname`\007"
fi

Linux Cut command

Cut Command
------------

Divide a file into several parts (columns)

syntax:
cut [-b] [-c] [-f] list [-n] [-d delim] [-s] [file]

Examples:
--------

1. Let say you have a file test.txt which has colon(:) seperated data

406378:Sales:Itorre:Jan
031762:Marketing:Nasium:Jim
636496:Research:Ancholie:Mel
396082:Sales:Jucacion:Ed

If you want to print first set of data from each row, you can use cut command as follow:

cut -d":" -f1 test.txt

If you want to print just columns 1 to 6 of each line (the employee serial numbers), use the -c1-6 flag, as in this command

cut -c1-6 test.txt

Linux Tips

Repeat a command (yes 'date;sleep 5' | ksh)
Replace newline with comma (cut -d, -f1 vishnu.csv | tr '\n' ',')
Get some specific string from each line (grep "uid%3D" <fileName> |awk -F"uid%3D" ` {print $2}` | cut -d"%" -f1)
Pull first n characters of each line from a file (cut -c1-n file.txt > newfile.txt)
Count total number of lines in all specific files under a directory (find . -type f -name '*.as' -o -name '*.mxml' -o -name '*.java'| xargs cat |wc -l)
Find number of occurrences of a text in a file (grep text fileName |wc -l)
Display the top most process utilizing most CPU (top -b 1)
Show the working directory of a process ? (pwdx pid )
Display the parent/child tree of a process ? (ptree pid )
Display the no.of active established connections to localhost ? (netstat -a | grep EST)
How to create null file ? (cat /dev/null > filename1)
Display top ten largest files/directories ? (du -sk * | sort -nr | head)
Display disk usage (du -h)
How to save man pages to a file ? (man | col -b > filename )
Display the files in the directory by file size ? (ls -ltr | sort -nr -k 5)
Display the processes, which are running under yourusername ( ps -aef | grep username)
Display the all files recursively with path under current directory ? ( find . -depth -print)
Display the Disk Usage of file sizes under each directory in currentDirectory ? (du -k . | sort -nr)
List the files in current directory sorted by size ? (ls -l | grep ^- | sort -nr)

Soalris: Kill a process which is using a particular port number

Today i came across a problem in solaris. The problem was that while starting my application server, it was throwing an error "Address already in use".

My app server is a java process and there are many other java process which are running on my zone. But the issue is, how may i know that which java process is using that particular port?

I followed following steps:
1. List all the java process running on my zone ( ps -eaf |grep vagrawal| grep java )
2. Go through each java process and check if it using that particular port ( pfiles $pid|grep 1182 )
(here $pid is the process id of the java process and 1182 is the port number of which i am looking for)


Above method works fine but it is bit a long process, as i have to run step 2 for all java processes, so i ran a folowing command on my console:

for i in `ps -e|awk '{print $1}'`; do echo $i; pfiles $i 2>/dev/null | grep 'port: 1188'; done

Above command/script will list out all the process ID and will tell if any process is using port 1188

Now i have process ID of the process which is occupying my port, and i can kill that by kill -9 pid