Text processing

wc

• Word count
• flags
  • c: total bytes
  • m: total characters
  • l: total lines
  • w: total words

# show count of lines, words, bytes
> echo "Hello World" | wc
1       2      12
 
# count lines
> printf "line1\nline2\n" | wc -l
2

cat

• flags
  • A: show non-printable characters
  • n: show numbered lines

# save stdin into a file
# type in stdin and press Ctrl+D
cat > file.txt
 
# show non-printable characters
cat -A file.txt
 
# show numbered lines
cat -n file.txt

sort

• flags
  • f: ignore case
  • r: reverse order
  • n: numeric sort
  • k: for tabular data, based on given column

# sort stdin and save into a file
sort > file.txt
 
# sort multiple files and merge
sort file1.txt file2.txt
 
# sort multiple files and merge to save in a new file
sort file1.txt file2.txt > merged.txt
 
# sort by largest file
du -s /usr/share/* | sort -nr

uniq

• must be used with sorted data
• flags
  • i: ignore case
  • c: output duplicates with count

sort foo.txt | uniq

cut

• extract section of text from line
• flags
  • c: line ranges
  • f: for tabular data, based on given column
  • d: specify field delimiter (default is tab)

# print third column of tabular data
cut -f 3 distros.txt
 
# print lines 7 to 10
cut -c 7-10 distros.txt

paste

• add one or more columns of text files

# The final output will contain dates column first then columns from distro-versions
paste distro-dates.txt distro-versions.txt

join

• performs join based on key on multiple tabular data
• uses whitespace as delimiter

# both text files have first column as key which will be used to perform join
join distro-key-names.txt distro-key-vernums.txt

tac

• reverse of cat
• concatenates files in reverse order

# reverse chronological order of logs (latest first)
tail /var/log/backup.log | tac

comm

• compare files

comm file1.txt file2.txt

diff

• compare files
• more complex than comm
• flags
  • c: context format
  • u: unified format

diff file1.txt file2.txt

patch

• apply changes from diff

diff -Naur file1.txt file2.txt > patchfile.txt
# applies patch to old file (file1.txt)
# file1.txt will now match with file2.txt
patch < patchfile.txt

tr

• Transliterate characters
• letter to letter mapping

# convert lowercase to uppercase
> echo "lowercase letters" | tr a-z A-Z
LOWERCASE LETTERS
 
# convert all chars to A
> echo "lowercase letters" | tr [:lower:] A
AAAAAAAAA AAAAAAA
 
# remove repeated a or b
> echo "aaabbbccc" | tr -s ab
abccc
 
# perform ROT13 encoding
> echo "secret text" | tr a-zA-Z n-za-mN-ZA-M
frperg grkg

sed

• Stream editor
  • started with commands
  • s/regexp/replacement/: substitution
  • y/set1/set2: transliteration
  • a: append
  • d: delete
  • i: insert
  • p: print

> echo "front" | sed 's/front/back/'
back

awk

• created by Bell Labs scientists
  • A – Alfred Aho (co-author of the classic compilers “Dragon Book”)
  • W – Peter Weinberger (known for his early work on UNIX databases)
  • K – Brian Kernighan (co-author of the first book on the C language, “K&R C”)
• awk is a Turing complete language
• Pattern scanning and processing language

# print specific columns
> echo "John Doe 25" | awk '{print $1, $3}'
John 25

aspell

• interactive spelling checker

aspell check foo.txt

Text Formatting

nl

• adds numbering to lines

nl distros.txt

fold

• performs word wrap

# each line with width 12 chars
> echo "The quick brown fox jumps over the lazy dog." | fold -w 12
The quick br
own fox jump
s over the l
azy dog.
 
# don't break words
> echo "The quick brown fox jumps over the lazy dog." | fold -w 12 -s
The quick
brown fox
jumps over
the lazy
dog.

fmt

• simple formatter

# formats comments
fmt -w 50 -p '# ' fmt-code.txt

pr

• paginate text
• used to convert text files for printing
• adds header and footer for page

# page length: 15 lines
# page width: 65 columns
pr -l 15 -w 65 distros.txt

printf

• Formatted Print
• originally developed in C language

> printf "I formatted the string: %s\n" foo
I formatted the string: foo
 
# save result in variable
> printf -v result "I formatted the string: %s\n" foo
> echo "$result"
I formatted the string: foo