My most used Shell Scripts – Part 1

Introduction

This post has my most used Shell scripts for configuring favorite environment setting on Linux server such as short hands for some frequently used functions, time date format, prompt appearance.

Concepts

What is shell script?

A shell script is a computer program, including a set of instructions, designed to be run by the Unix/Linux shell to interact with the operating system.

It can be used to automate a series of tasks, such as setting up the environment, running commands, manipulating files and directories, and performing other system-level operations.

Shell scripts are typically written using a text editor and saved with a .sh file extension. They can be executed by running the script file in the shell environment using the command-line interpreter. Shell scripts can be written in various shell programming languages, including Bash, Csh, Ksh, and Zsh.

What are different shell programming languages?

There are various shell programming languages available, but some of the most commonly used ones are:

Bash (Bourne-Again SHell) - Bash is the most widely used shell on Linux and macOS systems. It is an enhanced version of the original Bourne shell (sh) and includes many additional features and improvements.
Csh (C Shell) - Csh is a Unix shell that is similar to the C programming language. It includes features such as command-line editing, job control, and aliases.
Ksh (Korn Shell) - Ksh is a Unix shell that is similar to the Bourne shell (sh) but includes many additional features and improvements. It includes features such as command-line editing, history, and job control.
Zsh (Z Shell) - Zsh is a Unix shell that is similar to Bash and Ksh, but includes many additional features and improvements. It includes features such as improved tab completion, history, and spelling correction.

How does Linux know to interpret a shell script with correct shell programming language?

When a shell script is executed, the operating system uses the shebang (#!) at the beginning of the file to determine which shell program should be used to interpret the script.

#!/usr/bin/env zsh

When a shebang is specified as #!/usr/bin/env zsh, it means that the system should use the zsh shell program to interpret and execute the script, regardless of where the zsh program is installed on the system.

The env command is used to search the system's PATH environment variable for the zsh program and use it to execute the script. This approach is more flexible than using the absolute path to the zsh program (#!/bin/zsh), as it allows the script to be executed on systems where zsh is installed in a different location.

This shebang line is commonly used for shell scripts that are intended to be portable across different systems, as it avoids hard-coding the path to the shell program and instead relies on the PATH environment variable to locate the appropriate shell interpreter.

References:

This page explains the differences between bash and z-script.

Some common confusions about shell scripts

Single quote '

In shell script, a single quote (') is used to enclose a string literal. When a string is enclosed in single quotes, all characters within the quotes are treated literally, without any special meaning.

For example, if you have a variable $name that contains a string with a space, you can enclose it in single quotes to preserve the space:

name="John Smith"
echo 'Hello, '$name'!'

In this example, the single quotes around the string literal 'Hello, ' and the variable $name ensure that the space between "John" and "Smith" is preserved in the output.

Double quote "

When passing a parameter to a script with double quotes (""), the shell will interpret the contents of the double quotes as a single argument, even if it contains spaces or special characters. This is known as word splitting

./script.sh "hello world"

The OS will interpret hello world as a single word when running script.sh

Dollar sign $

$* represents all the positional parameters as a single string, with each parameter separated by the first character in the IFS (Internal Field Separator) variable. By default, IFS is set to whitespace, so $* will separate the positional parameters with a space. For example

$ set -- "hello" "world"
$ echo "$*"
hello world

here IFS is set to white space. Thus $* return a string as hello world

$@, on the other hand, represents all the positional parameters as separate strings. Each parameter is quoted and treated as a separate argument.

$ set -- "hello" "world"
$ echo "$@"
hello world

hello and world are 2 different strings.

$0 returns the name of the current program
$? return the status of the last run command
$ when placed after a command makes the command runs in backgroud process. Thus we can make parallel processes.

Wildcards extension:

In shell script, a wildcard is a special character or sequence of characters that can be used to represent a group of filenames or characters in a command. Wildcards are often used in combination with commands like ls, cp, mv, and rm.

Here are some of the most common wildcards and what they represent:

* (asterisk): Represents any sequence of characters, including none. For example, *.txt would match all files in the current directory that end with .txt, while file*.txt would match all files in the current directory that start with file and end with .txt.
? (question mark): Represents any single character. For example, file?.txt would match all files in the current directory that start with file, followed by any single character, and end with .txt.
[ ] (bracket expression): Represents any single character in a specified set. For example, [aeiou] would match any single vowel, while [0-9] would match any single digit.
[! ] (negated bracket expression): Represents any single character not in a specified set. For example, [!aeiou] would match any single character that is not a vowel.
** (double asterisk): Represents any number of subdirectories. For example, **/*.txt would match all files in the current directory and any subdirectories that end with .txt.

Dot-source operator:

In shell script, the dot-source operator is . (dot) followed by a space. This operator is used to "source" (i.e., read and execute) a script file in the current shell environment.

When you use the dot-source operator to source a script file, any functions, variables, or aliases defined in the script file are made available in the current shell environment.

For example, let's say you have a script file named myscript.sh that defines a function named greet:

#!/bin/bash

function greet {
    echo "Hello, $1!"
}

To make the greet function available in the current shell environment, you can source the myscript.sh file using the dot-source operator like this:

. myscript.sh

Now, you can call the greet function from the command line like this:

greet John

This will output Hello, John! because the greet function is now defined in the current shell environment.

Some common shell scripts

Environment config files

How does Linux read environmental variables?

Linux reads environmental variables from a variety of sources, including:

Shell startup files: When a user logs into a shell, the shell reads a series of startup files, including /etc/profile, ~/.bash_profile, ~/.bashrc, and others. These files can contain definitions for environmental variables.
System configuration files: The system configuration files, such as /etc/environment, can contain definitions for environmental variables that are set for all users on the system => define global persistent environment
User-specific configuration files: Each user can have their own configuration files that set environmental variables, such as ~/.bashrc, ~/.profile, and ~/.bash_profile.
Command-line options: Some programs allow users to set environmental variables through command-line options. For example, the env command can be used to set environmental variables for a single command.
Other processes: Some processes, such as systemd, can set environmental variables that are inherited by child processes.

The common files that stores environment variables are:

/bin/bash
       The bash executable
/etc/profile
       The systemwide initialization file, executed for login shells
~/.bash_profile
       The personal initialization file, executed for login shells
~/.bashrc
       The individual per-interactive-shell startup file
~/.bash_logout
       The individual login shell cleanup file, executed when a login shell exits
~/.inputrc
       Individual readline initialization file

Once environmental variables are defined, they can be accessed and modified by shell scripts and other programs. Environmental variables can be accessed using the $VARIABLE_NAME syntax, where VARIABLE_NAME is the name of the variable. For example, to access the value of the PATH environmental variable, you would use $PATH.

Environmental variables can be modified using the export command in a shell script. For example, to add a new directory to the PATH environmental variable, you could use the following command:

export PATH=$PATH:/path/to/new/directory

Setting alias for frequently used commands

Short hand for ls, cd, applying environment, and other commands:

# short hand for basic commands
alias   ls='ls --color=auto'
alias   ls='ls -Fh'
alias   dir=ls
alias   ll='ls -lh'
alias   la='ls -ah'
alias   lt='ls -ltr'
alias   ..='cd ..'
alias   ...='cd ../..'
alias   ....='cd ../../..'
# short hand to set env
alias   vv='vim ~/.vimrc'
alias   sv='source ~/.vimrc'
alias   vz='vim ~/.zshrc'
alias   sz='source ~/.zshrc'
# quickly find files and directories
alias   ff='find . -type f -name'
alias   fd='find . -type d -name'
# print the path with each dir on a separated line
alias   path='echo -e ${PATH//:/\\n}'
# print the current date
alias   now='date +%T'
alias   nowdate='date +"%d-%m-%Y"'
alias   tmux='LC_CTYPE=en_CA.UTF-8 TERM=screen tmux -2'
alias   summon_t='tmux attach'
# Add timestamp for output

Short hand for rpm commands:

# -i: install, -v: verbose, -h: display # while installation is in progress
allias install='rpm -ivh' 
# -q: query for alreay installed package
allias query='rpm -q'  
# -U: upgrade package
alias query='rpm -Uvh'

When using the script, I could run something like this:

install Linux_2.10.20

Configuring the environment

Control the behavior of the shell:

### shell options ###############################

setopt AUTO_LIST
setopt AUTO_MENU
setopt AUTO_PUSHD
setopt ALWAYS_TO_END
setopt CORRECT
unsetopt MENU_COMPLETE
setopt AUTO_CD
setopt RM_STAR_SILENT
setopt BASH_AUTO_LIST

setopt APPEND_HISTORY           # all Z shells append to the same history file
setopt HIST_IGNORE_DUPS         # history ignores subsequent duplicate entries
HISTFILE=$HOME/.history.$HOST   # command history location
SAVEHIST=20000                  # number of commands to store in history file
HISTORYSIZE=20000               # number of commands to store in shell history

Define characters as part of a word:

Since some automation scripts may have commands that contain characters outside the default set of word characters, I use the following to defines characters as part of the words

export WORDCHARS='*?_[]~=&;!#$%^(){}'

Define a function to add timestamp for each function:

# Add timestamp for output
# Usage: command | addts
addts() {
  while IFS= read -r line; do
    echo "[$(date +%Y-%m-%d\ %H:%M:%S)] $line"
  done
}

The IFS=read -r line syntax tells the loop to read each line into the line variable and strip any leading/trailing whitespace.

For each line of input, the function uses the date command to generate a timestamp in the format [YYYY-MM-DD| HH:MM:SS], where YYYY-MM-DD is the year, month, and day, and HH:MM:SS is the hour, minute, and second. The +%Y-%m-%d| %H:%M:%S option specifies the format of the timestamp.

Personalize the prompt

# enable command completion
autoload -Uz compinit
compinit

# enable colors and prompt themes
autoload -U colors && colors

# Prompt settings
autoload -Uz promptinit
promptinit

setopt PROMPT_SUBST       # allow funky stuff in prompt
color="green"
if [ "$USER" = "root" ]; then
  color="red"   # root is red, user is green
fi;

if [ "$HOST" = "$HOST_NAME" ]; then
    PROMPT="%F{magenta}%*%f %F{green}%B[%172<...<%~]%f%b
%F{yellow}%n%f@%F{cyan}%m%F{white} %# "

    precmd () {
        print ""
        print -Pn "\e]0;%n@%m ( $WORKSPACE_NAME ) - $PWD\a"
    }

    # set current Project and DB repos
    export PROJECT_CURRENT=/home/myproject
    export DB_CURRENT=/home/mydatabase
else
    PROMPT="%F{magenta}%*%f %F{green}%B[%172<...<%~]%f%b
%F{yellow}%n%f@%F{cyan}%m%F{white} %# "

    precmd () {
        vcs_info
        print ""
        print -Pn "\e]0;%n@%m ( $WORKSPACE_NAME ) - $PWD\a"
    }
fi

Define some options for Version Control System (VCS-info):

# VCS info
 ###
autoload -Uz vcs_info
zstyle ':vcs_info:*' enable git svn                   #display version ctrl info for both git and svn
zstyle ':vcs_info:*' check-for-changes true           #check for changes in the repository when the prompt is generated
zstyle ':vcs_info:*' format "%c %u [%b] %s:%r"        #apply format that has name, user, number of commits, current repository, current version

Set up the workspace for a project:


#!/usr/bin/env zsh
# Set up workspace for the project

if [ -f "$MYPROJECT_DIR" ]; then
  . $MYPROJECT_DIR
elif [ ! -d "$MYPROJECT_ROOT" ]; then
  # If neither MYPROJECT_DIR nor MYPROJECT_ROOT are set, then
  # try to set it to the 2nd-order parent directory of this script file
  MYPROJECT_ROOT=$(dirname "$0")/..
fi

So if the script file is save to MyProjectScript.sh, and I run the following command:

. MyProjectScript.sh Myproject

The project folder path MYPROJECT_ROOT will be set to the current directory of MyProjectScript.sh/Myproject/

Note that the dot operator means source. It run source command that reads and runs MyProjectScript.