unix commands cheat sheet

 

Vi Usage

 

Note - you need to be in command mode to run these commands(Press Esc key to go to command mode)

Below command will find PM_LINK_BK and replace it with PM_LINK

%s/PM_LINK_BK/PM_LINK/g

 

Below command will find PM_LINK

%/PM_LINK           

Press letter n to repeat the search to go to next result.

 

Below command will undo the last change

Just press letter u             

 

Below command will undo all changes made in the current line

Just press letter U         

 

Similarly x, d are for delete,  h j k l are for moving forward or up and down.

 

kill all the process of user -  username

ps U <username> | cut -d " " -f 1 | xargs kill

 

use this command to order the largest files for validation

 ls -l -S | sort -k 5 -n | awk ' { printf("%10d %s\n",$5,$9) } '

or 

simply run 

ls -lS 

in the folder - it will sort the files based on size

get unique first field delimited by “,” in the file load_dig_in_market_digtatz.bad
(similar to something like select distinct f1 from tablename)

 cut -d, -f1 load_dig_in_market_digtatz.bad | uniq

 

 

Clear history

cat /dev/null > ~/.bash_history && history -c && exit -- clean history

 

Find number of CPUs

grep -c ^processor /proc/cpuinfo  -- number of cpu

 

Find memory

free -m ----- memory

or

file - /proc/meminfo

cat /proc/meminfo

 

Find OS version (multiple command for different linux distributions)

cat /etc/os-release

or

lsb_release –a

or

hostnamectl

 

Find command

find in cac.properties file starting with current dir with the word valvcsspi001vm

find . -name cac.properties 2> /dev/null -exec grep valvcsspi001vm {} \;

 

TimeOut command

Times out the command if its not success in specified seconds

When times out – it return 124

timeout 10 <command>

 

 

 

scp command

copy files from one server to another

scp <localfile> user@remotehost:/<folder-to-be-copied>/

 

scp -i /home/mapr/.ssh/id_rsa /dfs/impower_dig/vlh/vlh_00014-8a9ccab8-ac1f-4776-8c58-8eb9149da9bd.c000_20190610.snappy.parquet valmarketsvc@sa1x-hadoop-p1.hchc.local:/mapr/mapr-clp1.hchc.local/valassis_data/impower_dig/vlh/.

 

command to run sh script from any server to <serverB> using user <username>

ssh -l <username> <serverB> /tda/data/epsilon/scripts/getFile.sh

A good example of timeout and scp

timeout 100 scp -i /home/mapr/.ssh/id_rsa /dfs/impower_dig/vlh/vlh_00014-8a9ccab8-ac1f-4776-8c58-8eb9149da9bd.c000_20190610.snappy.parquet valmarketsvc@sa1x-hadoop-p1.hchc.local:/mapr/mapr-clp1.hchc.local/valassis_data/impower_dig/vlh/.

And check the status of command.

 

Awk Usage

 

Extract 6th field from xref_0002.txt file , grab the txt containing SO , distinct and count.

(similar to something like  select count(distinct f6) from tablename

                                                 where f6 like ‘%DO%’)

 

awk -F, '{print $6}' xref_0002.txt |  grep -i SO | uniq | wc -l

 

 

just grep ooxie.distro process

 

[mapr@valvcshad002vm ~]$ ps -eo comm,pid,etimes | awk '/oozie.distro/ {print $3, $1,$2}'

2 oozie.distro 57876

 

Kill all java processes

ps ax | grep java | grep -v 'grep' | cut -d '?' -f1 | xargs kill -9