Unix Tip: Working with CSV file in Unix

One of the unix problem I worked recently, thought its worth sharing.
The problem required to query over csv files in unix, like we query database using sql.
So, these commands work quite like SQL. :)

Requirement:

To get the first record for where 2nd column is null, that is, emp_name has been left blank for the below input file content.

Input File Name:

test_data.csv [1]

Input File Content:

employee_id,emp_name,emp_age
234,Ram,27
235,Shyam,21
236,Mohan,40
237,,25

Command:

sed 's/,,/,NULL,/g;' test_data.csv | awk -F, '$2 ~/NULL/'  | sed 's/,NULL,/,,/g;' | head -n 1

Description:

- sed 's/,,/,NULL,/g;' test_data.csv : replaces all empty values with the string "NULL"
- awk -F, '$2 ~/NULL/' : finds all the values where 2nd column is NULL, -F option is used for defining field separator
- sed 's/,NULL,/,,/g;' : reverts back all the NULL to empty values
- head -n 1 : prints only the first record, if you remove this you will get all the records. [2]

Other examples:

1) Get all employees age greater then 20

sed 's/,,/,NULL,/g;' test_data.csv | awk -F, '$3 > 20' | sed 's/,NULL,/,,/g;'

2) Get all employees whose employee_id is equal of 234

sed 's/,,/,NULL,/g;' test_data.csv | awk -F, '$1 == 234' | sed 's/,NULL,/,,/g;'

3) Get all employees whose employee_id is less than or equal to 235

sed 's/,,/,NULL,/g;' test_data.csv | awk -F, '$1 <= 235' | sed 's/,NULL,/,,/g;'

--------------------------------------------------------------------------------------------
[1] CSV: A CSV file contains comma separated values and can be easily created/edited with Microsoft Excel, almost like normal excel files.
[2] In case we don’t need entire row, we can also use cut command here to get only one field or a few fields

Unix Tip : Unix command to find the files modified 'n' number of days ago

Find is the command for searching files in a unix folder. Probably, this is one such command which you might need quite often.

And, while working on a project, with VCS ( version control ), you often need to find the files that have been modified today, in last couple days and so on. So, this is one the way find command comes handy. It finds the files recursively as well.

Examples:

# replaces n with number of days here
> find . -mtime –n 

# A minus (-) before n indicates number of days ago 

# i.e. include those file which falls between today and nth day
> find . -mtime n    

# display list of files modified exactly ON the nth day, from today
> find . -mtime +n 

# A plus (+) before n indicates file modified even prior to nth day

In case you need to find the files that have been modified few minutes ago, or some 'm' minutes ago, you can use the option mmin instead of mtime.

------
Reference: http://linux.about.com/od/commands/l/blcmdl1_find.htm

ANT Deploy Tasks using Cargo

I wrote a post some time back on how to stop and start tomcat container using ANT.

Well, while trying to find something about Maven, I found this CARGO thing which can help you achieving the same, in fact better results. 

Interestingly, it has support to do similar task from Maven as well from JUnit as well.

Here is a link to  a sample example for Ant task

http://cargo.codehaus.org/Ant+support#Antsupport-examples

A similar example for doing it via Maven :

http://cargo.codehaus.org/Maven2+plugin

How to display/print current time using Ant

At times, while juggling between lots of things a time, you may forget when the deployment happended or did you actually deployed since you did the last code change!

This small 4 liners can facilitate you with that feature. You can know the last deployed time by looking at the console/terminal.

<tstamp>
    <format property="IST_TIME" pattern="HH:mm:ss:sss zzz" locale="en,In"/>
 </tstamp>
<echo> Build Completed At: ${IST_TIME}</echo>

Add it the last target of your build script that is going to be executed, change the locale to your respective locale, to time in your timezone and you are good to go.

A similar example can be found in my previous post.

       

How to restart (stop/start) the tomcat using Ant

In case you are using tomcat on your local machine and using ant script to deploy your project war to tomcat webserver, this build script can come handy.

Please note, this may not work on your machine just by doing a copy paste. You may need to do a bit of change in the build.properties file to change the parameters there and also little bit of change in the build.xml to suit your exact requirement.

When I wrote this script, this build.xml was kept in the project home directory, which is normally the case for most of the web projects.

Our Goal:

• Our goal here is to stop the tomcat if already running, then copy the war file to tomcat's webapps directory and after that start the tomcat again.
• In addition, skip the tomcat stop, if it is not running already. Just copy the war file to tomcat's webapps directory and after that start the tomcat again.

Steps this script does:

• The ant script, first start executing the target named, tomcat-start (which is meant to start the tomcat).
• As you can see the depends attribute,  tomcat-start, depends on  deploy-local, so it executes the deploy-local
• deploy-local in turn depends on tomcat-stop.
• tomcat-stop depends on the check-port.
• what check-port does is, it sets the property, "tomcat.running". Sets this parameter to true if it the tomcat is running on localhost, port 8080.
• Please note that the port can differ depending on your tomcat configuration/installation. So, you might need to change this port to the one on which your tomcat runs.
• Once this is done, the tomcat-stop, stops the tomcat, if "tomcat.running" sets to true.
• deploy-local, copies the war file to tomcat's webapps directory.
• Once the war is copied, tomcat-start starts the tomcat, and you app is ready to be tested on your local machine.

build.properties:

 project.name=MyProject  
 project.dir=C\://My_WorkSpace//MyProject  
 #Tomcat Local setting  
 tomcat.home=C\://Program Files (x86)//Apache Software Foundation//Tomcat 5.0  

 

build.xml

<?xml version="1.0" encoding="UTF-8"?>
<project name="${project.name}" default="tomcat-start" basedir=".">
 <!-- previous default task : tomcat-start -->
 <property file="build.properties" />
 <property environment="env" />


  
 <!--stopping tomcat  , if it is found running -->
 
 <target name="tomcat-stop" if="tomcat.running" depends="check-port" >
  <echo message="Tomcat is running...stopping it"/> 
     <java jar="${tomcat.home}/bin/bootstrap.jar" fork="true" >
         <jvmarg value="-Dcatalina.home=${tomcat.home}"/>
         <arg line="stop"/>
     </java>
 </target>
 

 <!-- Check whether Tomcat is running -->

   <target name="check-port" description="Check whether Tomcat is running">
     <echo message="Checking whether Tomcat is running"/>
     <condition property="tomcat.running">
       <socket server="localhost" port="8080"/> 
     </condition>
   </target>


 <!--    deploy-local: delete and copy ear --> 
 
 <target name="deploy-local" depends="tomcat-stop">
  
  <delete>
   <fileset dir="${tomcat.home}/webapps">
       <include name="${project.name}.war"/>
   </fileset>
  </delete>
  
  <delete dir="${tomcat.home}/webapps/${project.name}"/>
  
  <copy todir="${tomcat.home}/webapps">
   <fileset dir="${dist.dir}">
    <include name="**/${project.name}.war" />
   </fileset>
  </copy>
  
  <tstamp>
   <format property="TODAY_INDIA" pattern="HH:mm:ss:sss zzz" locale="en,In"/>
  </tstamp>
  <echo> Build Completed At: ${TODAY_INDIA}</echo>
  
      
 </target>
 
 <!-- start tomcat, it depends on deploy-local -->
 
 <target name="tomcat-start" depends="deploy-local">
       <java jar="${tomcat.home}/bin/bootstrap.jar" fork="true">
           <jvmarg value="-Dcatalina.home=${tomcat.home}"/>
           <arg line="start"/>
       </java>
 </target>
  
 
</project>

Related Links:

• http://techfindouts.blogspot.com/2013/08/ant-tasks-using-cargo.html

 

replace vs replaceAll in Java (Difference between replace and replace All)

Sometimes we do some silly mistake unintentionally.

Assuming replace() will replace first instance of String found in the entire String, and replaceAll() will replace all the instances of the Strings, is one of them.

When I realized this mistake of mine few years back, I curiously wanted to know what my fellow developers think about it.Interestingly, most of them were living with the same misconception.In fact they are really good Java programmers.
 
Hence, don't go by the name, Java has some other meaning tagged to it.

Both replace() and replaceAll() replaces all the occurrence found. However, there is difference in the type of arguments they accept.

replace() works on CharSequence, where as replaceAll(), accepts regex, so you can do a regular expression based pattern matching. The later is more flexible, but might result in a bit of performance overhead of compiling the regex and then do a matching. For smaller String replacements, you might not need to think much.  


 Quoting Java Doc:

String replace(CharSequence target, CharSequence replacement)
          Replaces each substring of this string that matches the literal target sequence with the specified literal replacement sequence.
 String replaceAll(String regex, String replacement)
          Replaces each substring of this string that matches the given regular expression with the given replacement.
 String replaceFirst(String regex, String replacement)
          Replaces the first substring of this string that matches the given regular expression with the given replacement.
          

PS: CharSequence is one of the interface implemented by String class.

If you see the Java Doc for String class, String implements Serializable, CharSequence and Comparable<String>.
So, dont get confused by CharSequence, normally you can pass String as input to replace() function because of above reason.

And yes, for repacling only the first occurance of a char sequence, you can use replaceFirst(String regex, String replacement) method of String class.
           

How to Convert a File Content in a Java String? A Simple Example Using Guava API...

Recently in a project I got into this task of converting an entire file into string. After googling a bit, I got this "guava" API to help. There are other APIs available from Apache etc, but I found Guava easy to use and have been working quite nice so far.

Although writing your own would not be much difficult either, but then why to re-invent the wheel! :)

You can use this for few doing some other interesting stuff as well like Collection, cahcing etc. Details you can find at Guava's home page.

If you are using Maven you can update the dependencies in your project by using snippet given at
http://code.google.com/p/guava-libraries/wiki/UseGuavaInYourBuild

Else, you can download the JAR from its home page : http://code.google.com/p/guava-libraries/


So, once you have added the dependencies/jar you can play with it. Here is a demo code for converting  a file's content into a Java String:

package com.demo.fileops;

import java.io.File;
import java.io.FileNotFoundException;

import java.io.IOException;
import java.nio.charset.Charset;

import com.google.common.io.Files;

public class FiletoStringGuavaStyle{
 
 String fileContent = null;

 /** A function to convert File Content into String **/
 public static String loadFileIntoString() throws IOException{ 
  
  String path = "C:"+ File.separator + "Users" +  File.separator  + "myTextFile.txt";  // location -> C:/Users/myTextFile.txt
  
  this.fileContent = Files.toString(new File(path), Charset.defaultCharset()); // use of Guava here
  
  System.out.println("fileContent=" + this.fileContent);
   
  return this.fileContent; 
 }
 
 public static void main(String args[]){
  
  String content = FiletoStringGuavaStyle.loadFileIntoString() ; 
 }

} 

Unix Tip for cutting last field properly : Use tr to cut the last field

Issue:Cut command giving wrong results on cutting last field

While processing data from a delimited file in bash/shell script using cut command, you might run into peculiar issue. The last field does not cut properly.

The output of cutting the last field will give you some junk character and it might break your code to something quite unexpected.

If you don't know whats causing this, the situation can be very tricky and frustrating.

So, the secret is the file was edited in window mode and it adds carriage return character. So, now you the root cause, the solution is not very difficult either. 

#1. You can use unix to edit the command. This may not help when you are using CSV files with MS excel.

#2. Alternatively, Copy-paste or edit the file in EditPlus or similar text editor, and edit and save in unix mode.

#3. Use tr command to trim the '\r' character. (tr -d '\r'). You should anyway do it to be on the safer side even if you are doing one of the above step. Just in case, some of your friend wanna use your script.

Here is a demo script, showing the example how to use it in a bash script:

#! /bin/bash
#Purpose: Demo Script

 
 echo "Started processing the input file: $1"
 
 INPUT_FILE_NAME=$1
 cat /dev/null > $2

 count=0

 while read LINE
 do
    let count++
  
    cust_id=`echo $LINE | cut -d"," -f1`
    itemName =`echo $LINE | cut -d"," -f2`     
    paymentStatus =`echo $LINE | cut -d"," -f3`
    invoice=`echo $LINE |tr -d '\r'| cut -d"," -f4` 
  
   echo "Record:'$count': cust_id='$cust_id':itemName='$itemName':paymentStatus='$paymentStatus':invoice='$invoice'" 
     
  
 done < $INPUT_FILE_NAME
 
 echo "Finished processing the input file: $1"

fi