Shell scripting - cheatsheet

Here’s a cheatsheet for shell scripting including some tips and tricks, where and how to use it.
Introduction
There are many different shells out there - sh, bash, zsh, fish and csh just to name some popular ones. Each has its variances and scripts that work on bash may not work on sh or fish. These snippets use code that targets my personal shell choice zsh but TRY to remain POSIX compatible.
Please note: This cheatsheet is not meant for beginners. You should always try to understand what is happening before you copy and paste.
Basics
Header line
#!/usr/bin/env bash
#!/bin/sh
#!/usr/bin/env zsh
suppress all output (including errors)
echo "suppressed" 2>&1 > /dev/null
if then else (see if then else reference )
# [[...]] would be non POSIX, even if it is preferred for bash/zsh!
if [ -d "${DIR}" ]; then
echo "${DIR} exists"
elif [ -d "${DIR2}" ]; then
echo "${DIR2} exists"
else
echo "${DIR} and ${DIR2} missing"
fi
# short version
[ -d "${DIR}" ] && ( echo "${DIR} exists" ) || ( echo "${DIR} missing" );
# if NOT (negation)
[ ! -d "${DIR}" ] && ( echo "${DIR} missing" );
case
day=$(date +%w)
case "${day}" in
0|\
7) echo "Weekend - yay!"
;;
6) echo "Almost made it!"
;;
*)
echo "Oh no..."
;;
esac
loops - for, while, continue and break
for i in 1 2 3 4 5; do
echo "$i"
done
i=0
while true; do
i=$((i+1))
if [ $i -lt 2 ] && (continue)
echo $i
[ $i -gt 3 ] && (break)
done
functions
function append {
echo "${1}${2}" # returning strings only with echo
return 0 # real return value stored in $?
}
VALUE=$(append "123" "456")
echo $?
Files
write output to files
#truncate file (empty file is kept)
> /tmp/test.txt
# overwrite existing contents
echo "test-overwrite" > /tmp/file.txt
# append content
echo "test-append" >> /tmp/file.txt
# append multiple lines
cat <<EOF >> /tmp/file.txt
multiple
lines
EOF
# overwrite contents with multiple lines and sudo permissions
sudo tee /tmp/file.txt > /dev/null <<EOT
multiple
lines
EOT
# append multiple lines with sudo permissions
sudo tee -a /tmp/file.txt > /dev/null <<EOT
multiple
lines
EOT
search file contents
# search for pattern in file
grep 'pattern' file
# search for pattern recursively in my_directory
grep -R 'pattern' my_directory
# search for multiple patterns
grep 'pattern1\|pattern2' file
# search for NON matches (invert search)
grep -v 'pattern' file
# search without printing found matches
grep -q 'pattern' file && (echo "pattern found")
inplace editiing with sed
# comment out matching line (dtparam=audio=on => #dtparam=audio=on)
FILE_TO_REPLACE="/boot/config.txt"
sed -i 's/^dtparam=audio=on/#dtparam=audio=on/g' "${FILE_TO_REPLACE}"
# replace matching line (value=hello => value=replaced)
FILE_TO_REPLACE="/tmp/test.txt"
sed -i 's/^value=.*/value=replaced/g' "${FILE_TO_REPLACE}"
# append line AFTER match (dtoverlay=hifiberry-dac after #dtparam=audio=on)
FILE_TO_REPLACE="/boot/config.txt"
sed -i '/^#dtparam=audio=on/a dtoverlay=hifiberry-dac/' "${FILE_TO_REPLACE}"
# insert a line BEFORE match (dtoverlay=hifiberry-dac before #dtparam=audio=on)
FILE_TO_REPLACE="/boot/config.txt"
sed -i '/^#dtparam=audio=on/i dtoverlay=hifiberry-dac/' "${FILE_TO_REPLACE}"
# insert multiline before a match (using `\n`)
FILE_TO_REPLACE="/boot/config.txt"
sed -i '/^#dtparam=audio=on/i dtoverlay=hifiberry-dac\n#another=value/' "${FILE_TO_REPLACE}"
mkdir if not exists
[ -d "${DIR}" ] || (mkdir -p "${DIR}")
change dir, run command and change back
cd /tmp/ && touch "testfile.txt" && cd -
loop over files in directory
SAVEIFS=$IFS
IFS=$(echo -en "\n\b")
FILES=/data/*
for f in $FILES; do
echo "$f"
done
IFS=$SAVEIFS # restore $IFS
loop over files with find
find . -print0 | while read -d $'\0' file; do
echo -v "$file"
done
find duplicate files
# linux only (unix uniq works different)
# skip 0 byte files (! -empty)
# only files, not dirs (-type f)
# create md5 checksum (md5sum {})
# sort by md5 checksum
# compare first 32 chars (only hash, exclude filename) of only duplicate lines and all of them (uniq -w32 -dD)
find "." ! -empty -type f -exec md5sum {} + | sort | uniq -w32 -dD
Math
Arithmetic (increment / decrement / calculate)
# simple operations (only integers)
i=0
i=$((i+1))
i=$((i-1))
# more complex (including floats)
var=$(echo "1+2"|bc)
String manipulation
concat strings
ASTRING="a-string-with-$VARIABLE"
heredoc simple
cat <<EOF
testing
EOF
heredoc with pipes
(
cat <<EOF
testing
EOF
) | grep 'testing'
match text with heredoc
# (result: 1 for match, 0 for no match)
grep -oc 'yes' <<< "Available: yes"
squeeze multiple spaces
tr -s ' ' <<< "this is a test"
left pad a string (with spaces)
PADLEN=15
echo "to_pad" | sed -e :a -e "s/^.\{1,$PADLEN\}\$/ &/;ta" ;
Misc
Count lines (e.g. jpg files in dir)
ls -l *.jpg | wc -l
find examples
use a function for find -exec
export -f my_function
find . -name '*.jpg' -exec bash -c 'myfunction "$0"' {} \;
find multiple extensions
find . -iregex '.*\.\(jpg\|svg\|gif\|png\|jpeg\)$'
find invert match (e.g. files that are not mp3)
find . -name '*' -type f -not -path '*.mp3'
Utility reference
| Name | Description | Example |
|---|---|---|
awk | pattern scanning and string manipulation (complex!) | awk '{print}' cooking.txt |
cat | concatenate files and print on the standard output | cat/tmp/file.txt |
cp | copy files and directories | cp a.txt b.txt |
curl | transfer a URL. | curl -O http://example.com/file.jpg |
cut | cut out selected portions of each line of a file | cut -d ' ' -f 2 <<< "Hello World" |
diff | compare files line by line | diff a.txt b.txt |
du | estimate file space usage | du -sh * |
echo | display a line of text | echo "test" |
find | search for files in a directory hierarchy | find . -name '*.mp3 |
grep | Print lines matching a pattern | grep 'search' file.txt |
kill | send a signal to a process | kill -HUP 801 |
ls | list directory contents | ls -a /tmp |
mv | move files and directories | move a.txt b.txt |
paste | merge lines of files | paste id.txt name.txt email.txt |
patch | apply a diff file to an original | patch < insert_lines.patch |
printf | format and print data | printf "Open: %s\nClosed: %s\n" "34" "65" |
rsync | file transfer utility | rsync -a src/ destination/ |
rm | remove files or directories | rm /tmp/file-to-remove.txt |
sed | stream editor (e.g. for string replacements) | sed 's/Hello/Hi/' <<< "Hello World" |
seq | print a sequence of numbers | seq 5 |
sleep | delay for a specified amount of time | sleep 2 |
sort | sort lines of text files | sort file.txt |
tee | read from input and write to output and files | `ls |
tr | translate characters | tr -s 'Hello' 'Hi' <<< 'Hi' |
uname | print system information | uname -a |
wc | print newline, word, and byte counts for each file | `ls -l |
wget | The non-interactive network downloader. | wget http://example.com/file.jpg |
which | locate a program file in the user’s path | which sh |
xargs | append arguments when pipe support is missing | `ls *.txt |
if then else reference
For a complete reference see man test.
| Parameter | Description |
|---|---|
[ -e "${FILE}" ] | file exists |
[ -s "${FILE}" ] | file exists and not empty |
[ -w "${FILE}" ] | file is writable by current user |
[ -d "${DIR}" ] | file is directory and exists |
[ -h "${LINK}" ] | file is symlink and exists |
[ -n "${STR}" ] | string is NOT empty |
[ -z "${STR}" ] | string is empty |
[ "${STR1}" = "${STR2}" ] | string STR1 equals STR2 |
[ "${STR1}" != "${STR2}" ] | string STR1 NOT equals STR2 |
[ ${INT1} -eq "${INT2}" ] | integer INT1 equals INT2 |
[ ${INT1} -ne "${INT2}" ] | integer INT1 NOT equals INT2 |
[ ${INT1} -lt "${INT2}" ] | integer INT1 lower than INT2 |
[ ${INT1} -le "${INT2}" ] | integer INT1 lower or equal than INT2 |
[ ${INT1} -gt "${INT2}" ] | integer INT1 greater than INT2 |
[ ${INT1} -ge "${INT2}" ] | integer INT1 greater or equal than INT2 |
