Introduction
The shell script programming (aka shell scripting), is a powerful skill for automating tasks in a command-line environment. It's great for streamlining repetitive tasks and managing system processes.Various shell
There are various types of shells, each with its own features and capabilities. Here are some of the commonly used types of shells:- Bourne Shell (sh): The original Bourne shell, developed by Stephen Bourne. It is a simple and efficient shell, but it lacks some of the features found in more modern shells. The Bourne shell is a command-line interpreter or shell that was the original Unix shell.
- Bash (Bourne Again SHell): Bash is one of the most popular and widely used shells. It is an enhanced version of the original Bourne shell, providing additional features like command history, job control, and improved scripting capabilities. Bash is the default shell for many Unix-like operating systems.
- Zsh (Z Shell): Zsh is an extended Bourne shell with features from Bash, Korn shell, and others. It includes advanced tab completion, spelling correction, and improved scripting capabilities. Zsh is highly customizable and user-friendly.
- PowerShell: Developed by Microsoft, PowerShell is a task automation framework and shell. It is especially prevalent in Windows environments but is also available for Linux and macOS.
- Korn Shell (ksh): The Korn shell was developed by David Korn as a superset of the Bourne shell with additional features. It includes features like command-line editing, history, and associative arrays. There are two main versions: ksh88 and ksh93.
- Dash: Dash is a minimalistic POSIX-compliant shell, designed to be lightweight and fast. It is often used as the default system shell on some Unix-like systems for scripts and system utilities.
- Ash (Almquist Shell): Ash is a lightweight shell that aims to be POSIX-compliant. It is often used in embedded systems and is known for its small memory footprint.
- Fish (Friendly Interactive SHell): Fish is designed to be user-friendly and interactive. It provides features like syntax highlighting, autosuggestions, and a consistent command syntax. While not POSIX-compliant, it offers a unique and intuitive user experience.
Some common terminologies
Scripting involves a variety of terminology that is commonly used across different programming and scripting languages. Here are some key terms:- Script: A script is a series of commands or instructions written in a scripting language to be executed by a runtime environment.
- Interpreter: An interpreter is a program that reads and executes scripts line by line. It interprets the script's instructions without the need for compilation.
- Syntax: Syntax refers to the rules governing the structure of statements in a script. Correct syntax is essential for the script to be properly interpreted and executed.
- Variable: A variable is a named storage location in which data can be stored and retrieved. Variables are used to store information that may change during the script's execution.
- Data Types: Data types define the type of data that a variable can hold, such as integers, strings, or booleans.
- Conditional Statements: Conditional statements (e.g.,
if
,else
,elif
) allow the script to make decisions based on certain conditions. - Loop: A loop is a control structure that allows a set of instructions to be repeated multiple times. Common loop types include
for
andwhile
loops. - Function: A function is a reusable block of code that performs a specific task. Functions help in organizing and modularizing code.
- Arguments/Parameters: Arguments or parameters are values that can be passed to a function or a script. They provide input to the code.
- Comments: Comments are non-executable lines in a script used to provide explanations, notes, or documentation for the code. They are ignored during execution.
- Debugging: Debugging is the process of identifying and fixing errors (bugs) in a script. This often involves using tools to trace and analyze the script's execution.
- Shebag (#!)The shebang is a special character sequence (e.g.,
`#!/bin/bash`
) at the beginning of a script that specifies the interpreter to be used for executing the script. - Scripting Languages: Scripting languages are programming languages designed for scripting, often with a focus on ease of use and rapid development. Examples include Bash, Python, Perl, and Ruby.
- Environment Variables: Environment variables are variables outside the script that store configuration settings, paths, or other information relevant to the execution environment.
Execution:
- Create a file using command ===>
`vi filename.sh`
- To make the text file executable ===>
`chmod 755 filename.sh`
- To run the script ===>
`./filename.sh`
- A tab (to create a space) in between the script like:
echo "Hello `tab` World"
will show a space asHello `tab` World
- But when there is just space placed, but not tab between
`Hello World`
, it will not show any space.
`chmod u+rx myscript.sh`
command is used to change the permissions of a file in a Unix-like operating system, such as Linux or macOS. Let's break down the components of this command:
chmod`
: Stands for "change mode" and is the command used to change file permissions.u+rx
: This specifies the changes to be made to the permissions of the file. Here's what each part means:u
: Refers to the user (owner) of the file.+rx
: Adds read (`r`) and execute (`x`) permissions.
`myscript.sh`
This is the name of the file for which you want to change the permissions. Replace it with the actual name of your script.
- read(r): Allows the owner to read the contents of the file.
- Execute (`x`): Allows the owner to execute the file as a program.
Example scripts:
Action | Description | Script |
---|---|---|
echo | Outputs text or variables to the terminal. |
|
read | Reads input from the user and assigns it to a variable (The `-p` option is used to specify a prompt that will be displayed to the user. In this case, the prompt is "Enter your name: ") |
|
if | Conditional statement for branching based on a condition. |
|
for | Looping construct for iterating over a sequence of values. |
|
while | Looping construct that continues as long as a specified condition is true. |
|
case | Case statement for conditional branching based on pattern matching. |
|
function | Declares and defines a function for code modularity. |
|
ls | Lists files and directories in a directory. |
|
cp | Copies files or directories. |
|
mv | Moves or renames files or directories. |
|
rm | Removes files or directories. |
|
touch | Creates an empty file or updates the access and modification times of a file. |
|
grep | Searches for a pattern in a file or input stream. |
|
sed | Stream editor for filtering and transforming text. |
|
awk | Text processing tool for pattern scanning and processing. |
|
- Unix system:
`#!/bin/sh`
: The first line tells Unix that the file is to be executed by/bin/sh
. - Perl: script may start with the line `#!/usr/bin/perl` to tell your interactive shell that the program which follows should be executed by perl. For Bourne shell programming, we shall stick to
`#!/bin/sh`
.
Variables
In shell scripting, variables are used to store and manipulate data. The concept of variables - a symbolic name for a chunk of memory to which we can assign values, read and manipulate its contents. Here are the key aspects of variables in shell scripting:
- Variable Declaration and Assignment:
Syntax:
variable_name=value
Example:
greeting="Hello, World!"
- Variable Naming Rules:
- Variable names are case-sensitive.
- They can consist of letters, numbers, and underscores.
- The first character must be a letter or an underscore.
- Accessing Variable Values:
#!/bin/sh echo $greeting
- Special Variables:
- Positional Parameters: $0, $1, $2, ..., represent the script name and its arguments.
- Number of Arguments: $# gives the number of command-line arguments.
- All Arguments: $@ or $* represents all command-line arguments.
- Exit Status: $? holds the exit status of the last executed command.
- Read Input into a Variable:
read -p "Enter your name: " username
- Variable Concatenation:
fullname="$firstname $lastname"
- Variable Scope:
export myvar="some value"
- Unsetting Variables:
unset myvar
- Using Variables in Commands:
count=5 echo "There are $count apples."
These are fundamental concepts when working with variables in shell scripting. They provide a way to store, retrieve, and manipulate data within your scripts, making them dynamic and adaptable.
Wildcards:
- Copy all the files from /tmp/a into /tmp/b
$ cp /tmp/a/* /tmp/b/
$ cp /tmp/a/*.txt /tmp/b/
$ cp /tmp/a/*.html /tmp/b/
- rename all .txt files to .bak
$ mv *.txt *.bak
Escape Characters:
Escape characters in bash scripting are used to represent characters that are difficult to type or might have a special meaning in the context of a script. They are preceded by a backslash (\) to indicate that the following character should be treated differently. Here are some common escape characters in bash scripting along with examples:- Newline (
`\n`
): Represents a new line.echo -e "Line 1\nLine 2"
Output:
Line 1
Line 2 - Tab (
`\t
): Represents a horizontal tab.$ echo -e "Column 1\tColumn 2"
Output: Column 1 Column 2 - Backspace (
`\b`
): Moves the cursor one character to the left.$echo -e "Hello\bWorld"
Output: HellWorld - Carriage Return (
`\r`
): Moves the cursor to the beginning of the line.$echo -e "123\rABC"
Output: ABC - Single Quote (
`\'`
): Escapes a single quote within a single-quoted string.$echo 'It '\''s a single quote'
Output: It 's a single quote - Double Quote (
`\"`
): Escapes a double quote within a double-quoted string.$echo "She said, \"Hello!\""
Output: She said, "Hello!" - Dollar Sign (
`\$`
): Escapes the dollar sign to prevent variable expansion.$echo "The cost is \$10"
Output: The cost is $10 - Backslash (
`\\`
): Escapes a backslash.$echo "This is a backslash: \\"
Output: This is a backslash: \ -
$ echo Hello World
Output: Hello World -
$ echo "Hello World"
Output: Hello World -
$ echo "Hello "World""
Output: Hello "World" - Characters (*, ', etc):
$ echo *
Output: LICENSE README.md Shell-scripting-tutorials.ipynb all_args.sh count_args.sh exit_status.sh my-script-0.sh my-script-1.sh my-script-2.sh my-script-3.sh my-script-4.sh myvar2.sh myvar3.sh var.sh var2.sh var3.sh var4.sh test.txt
(basically all files in the current directory)$ echo *txt
Output: test.txt
(*txt means all files ending in txt)$ echo "*"
Output: *
( * in double quotes, is interpreted literally.)$ echo *txt
Output: test.txt$ echo "*txt"
Output: *txt
(the same applies, but we have appended txt to the string) -
$echo "A quote is \", backslash is \\, backtick is \`."
Output: A quote is ", backslash is \, backtick is `. -
$echo "A few spaces are and dollar is \$. \$X is ${X}."
Output: A few spaces are and dollar is $. $X is 5. - Some other examples:
echo "The cost is $10" ==> The cost is 0
echo 'The cost is $10' ==> The cost is $10
echo "The cost is \$10" ==> The cost is $10
Loop
Most languages have the concept of loops: If we want to repeat a task twenty times, we don't want to have to type in the code twenty times, with maybe a slight change each time. In Bash scripting, you can use various types of loops to iterate through a set of commands or statements. The two primary types of loops are the for loop and the while loop. Here are examples of both:1. 'for' loop:
The`for`
loop is used when you know in advance how many times you want to execute a set of statements.
- Numeric Range:
Output:#!/bin/bash
for i in {1..5}
do
echo "Iteration $i"
done
Iteration 1
Iteration 2
Iteration 3
Iteration 4
Iteration 5 - Iterate Over Items in an Array:
Output:#!/bin/bash fruits=("Apple" "Banana" "Orange") for fruit in "${fruits[@]}" do echo "Fruit: $fruit" done
Fruit: Apple
Fruit: Banana
Fruit: Orange
2. `while` Loop:
Thewhile
loop is used when you want to execute a set of statements as long as a condition is true.
#!/bin/bash
counter=5
while [ $counter -gt 0 ]
do
echo "Countdown: $counter"
((counter--))
done
- `-gt`: This is a comparison operator that stands for "greater than."
- `0`: This is the value that $counter is being compared to.
Countdown: 5
Countdown: 4
Countdown: 3
Countdown: 2
Countdown: 1
Remember to make your script executable using
chmod +x script.sh
and then execute it with ./script.sh
assuming the script is named script.sh
.
Test
In shell scripting, the`test`
command is often used to evaluate conditional expressions. Alternatively, you can use square brackets `[ ]`
as a synonym for `test`
. Test is most often invoked indirectly via the if and while statements. It is also the reason you will come into difficulties
if you create a program called test and try to run it, as this shell builtin will be called instead of your program!
The syntax for if...then...else...
is:
#!/bin/bash
if [ ... ]
then
# if-code
else
# else-code
fi
Note that fi
is if
backwards! This is used again later with case and esac. Also, be aware of the syntax - the "if [ ... ]
" and the "then
"
commands must be on different lines. Alternatively, the semicolon ";
" can separate them:
#!/bin/bash
if [ ... ]; then
# do something
fi
You can also use the elif, like this
#!/bin/bash
if [ something ]; then
echo "Something"
elif [ something_else ]; then
echo "Something else"
else
echo "None of the above"
fi
Examples:
- String Comparison:
#!/bin/bash str1="hello" str2="world" if [ "$str1" = "$str2" ]; then echo "Strings are equal" else echo "Strings are not equal" fi
- Numeric Comparison:
#!/bin/bash num1=5 num2=10 if [ "$num1" -eq "$num2" ]; then echo "Numbers are equal" else echo "Numbers are not equal" fi
- File Existence:
#!/bin/bash file_path="/path/to/some/file.txt" if [ -e "$file_path" ]; then echo "File exists" else echo "File does not exist" fi
- Logical AND:
#!/bin/bash num1=5 num2=10 if [ "$num1" -gt 0 ] && [ "$num2" -lt 20 ]; then echo "Both conditions are true" else echo "At least one condition is false" fi
The case statement
A`case`
statement in Bash scripting is used for conditional branching based on pattern matching.
It is often used as an alternative to multiple `if-elif-else`
statements when you have multiple
conditions to check. Its syntax is really quite simple:
Example:
#!/bin/bash
echo "Enter a fruit: "
read fruit
case "$fruit" in
"apple")
echo "Selected fruit: Apple"
;;
"banana" | "plantain")
echo "Selected fruit: Banana or Plantain"
;;
"orange")
echo "Selected fruit: Orange"
;;
*)
echo "Unknown fruit"
;;
esac
Bash command references
Command | Description | Example |
---|---|---|
& | Run the previous command in the background | ls & |
&& | Logical AND | if [ "$foo" -ge "0" ] && [ "$foo" -le "9" ] |
|| | Logical OR | if [ "$foo" -lt "0" ] || [ "$foo" -gt "9" ] |
^ | Start of line | grep "^foo" |
$ | End of line | grep "foo$" |
= | String equality (cf. -eq) | if [ "$foo" = "bar" ] |
! | Logical NOT | if [ "$foo" != "bar" ] |
$$ | PID of current shell | echo "my PID = $$" |
$! | PID of last background command | ls & echo "PID of ls = $!" |
$? | Exit status of last command | ls ; echo "ls returned code $?" |
$0 | Name of current command (as called) | echo "I am $0" |
$1 | Name of current command's first parameter | echo "My first argument is $1" |
$9 | Name of current command's ninth parameter | echo "My ninth argument is $9" |
$@ | All of current command's parameters (preserving whitespace and quoting) | echo "My arguments are $@" |
$* | All of current command's parameters (not preserving whitespace and quoting) | echo "My arguments are $*" |
-eq | Numeric Equality | if [ "$foo" -eq "9" ] |
-ne | Numeric Inequality | if [ "$foo" -ne "9" ] |
-lt | Less Than | if [ "$foo" -lt "9" ] |
-le | Less Than or Equal | if [ "$foo" -le "9" ] |
-gt | Greater Than | if [ "$foo" -gt "9" ] |
-ge | Greater Than or Equal | if [ "$foo" -ge "9" ] |
-z | String is zero length | if [ -z "$foo" ] |
-n | String is not zero length | if [ -n "$foo" ] |
-nt | Newer Than | if [ "$file1" -nt "$file2" ] |
-d | Is a Directory | if [ -d /bin ] |
-f | Is a File | if [ -f /bin/ls ] |
-r | Is a readable file | if [ -r /bin/ls ] |
-w | Is a writable file | if [ -w /bin/ls ] |
-x | Is an executable file | if [ -x /bin/ls ] |
( ... ) | Function definition | function myfunc() { echo hello } |
References
Some other interesting things to know:
- Visit my website on For Data, Big Data, Data-modeling, Datawarehouse, SQL, cloud-compute.
- Visit my website on Data engineering