Bash Special Variables

1. Overview

Bash comes with some special built-in variables. They come in handy when we want to control the execution flow of a Bash script.

We can only read their values and never assign to them.

In this tutorial, we’re going to see how and when to use them.

2. Special Positional Variables

First, let’s explain what positional variables are. We use them to reference arguments passed to a shell script or a function.

The basic syntax is ${N}, where N is a digit. We can omit the curly braces if N is a single digit.

However, for readability and consistency reasons, we’re going to use curly brace notation in our examples.

Let’s see how this looks like in practice by creating a function called positional_variables:

function positional_variables(){
    echo "Positional variable 1: ${1}"
    echo "Positional variable 2: ${2}"
}

Then, we can invoke positional_variables, passing it two parameters:

$ positional_variables "one" "two"
Positional variable 1: one
Positional variable 2: two

So far, there’s nothing special about this handling.

Let’s try to reference an argument that was not passed:

function positional_variables(){
    echo "Positional variable 1: ${1}"
    echo "Positional variable 2: ${2}"
    echo "Positional variable 3: ${3}"
}

Bash considers it unassigned:

$ positional_variables "one" "two"
Positional variable 1: one
Positional variable 2: two
Positional variable 3:

2.1. All Arguments

Let’s see what happens if we change the digit with the special @ character:

function positional_variables(){
    echo "Positional variables with @: ${@}"
}

The ${@} construct expands to all the parameters passed to our function:

$ positional_variables "one" "two"
Positional variables with @: one two

Additionally, we can iterate over them, similar to an array:

function positional_variables(){
    echo "Positional variables with @: ${@}"
    for element in ${@} 
    do
        echo ${element}
    done
}

Let’s run it and see what we get:

$ positional_variables "one" "two"
Positional variables with @: one two
one
two

We can also use the ${∗} construct to achieve the same result:

function positional_variables(){
    echo "Positional variables with @: ${@}"
    echo "Positional variables with *: ${*}"
}

Let’s take a look at the output:

$ positional_variables "one" "two"
Positional variables with @: one two
Positional variables with *: one two

At first look, the output is the same, right? Not exactly.

Let’s change the IFS variable and rerun it:

function positional_variables(){
    IFS=";"
    echo "Positional variables with @: ${@}"
    echo "Positional variables with *: ${*}"
}

Now, the output is:

$ positional_variables "one" "two"
Positional variables with @: one two
Positional variables with *: one;two

The difference here is the join between the input arguments.

When using ${∗}, the parameters are expanded to ${1}c${2} and so on, where c is the first character set in IFS.

Getting all inputs at once is very useful for input validation.

Let’s consider a simple utility function that checks if specific options are passed:

function login(){
    if [[ ${@} =~ "-user" && ${@} =~ "-pass" ]]; then
        echo "Yehaww"
    else
        echo "Bad Syntax. Usage: -user [username] -pass [password] required"
    fi
}

In this snippet, we check all our function arguments for two specific flags.

Let’s focus on what happens differently with each call:

$ login -user "one" -pass "two" 
Yehaww
$ login -pass "two" -user "one" 
Yehaww
$ login "one" "two"
Bad Syntax. Usage: -user [username] -pass [password] required

Notice that the first two calls succeed regardless of the order of our inputs.

This allows the user to pass the arguments in any order but is a very rudimentary check:

$ login -user -pass
Yehaww

Notice our condition passes, although we send just the two strings. We’ll see next how to handle this.

2.2. Number of Arguments

Most input validation sequences first check the number of parameters.

Let’s use the ${#} construct to obtain the number of inputs:

function login(){
    if [[ ${#} < 4 ]]; then
        echo "Bad Syntax. Usage: -user [username] -pass [password] required"
        return
    fi
    echo "Passed argument number check" 
    # previous checks omitted
}

In this example, we verify that our function was called with the minimum number of arguments.

This way, we can fail fast, and we don’t waste time processing other validations:

$ login
Bad syntax. Usage: -user [username] -pass [password] required

However, we can still bypass this check:

$ login -user -pass "one" "two"
Passed argument check
Yehaww

Let’s make this more robust:

function login(){
    # previous checks omitted 
    while [[ ${#} > 0 ]]; do
        case $1 in
            -user) 
            user=${2};
            shift;;
            -pass)
            pass=${2};
            shift;;
            *)
            echo "Bad Syntax. Usage: -user [username] -pass [password] required"; 
            return;;
        esac
        shift
    done
    echo "User=${user}"
    echo "Pass=${pass}"
}

This looks a bit more complex. Let’s break it down.

We use a while loop and check if the first positional variable matches any of our options.

If it does, then we extract the value from the second positional variable.

After that, we use the shift built-in to move across our while loop.

Let’s see it in action:

$ login -pass "one" -user "two"
Passed argument check
Yehaww
User=two
Pass=one

Now let’s try to change the input order:

$ login -pass -user "two" "one"
Passed argument check
Yehaww
Bad Syntax. Usage: -user [username] -pass [password] required

2.3. Name of Script

Let’s make our error handling more expressive by adding the script name in our function:

function login(){
    if [[ ${#} < 4 ]]; then 
        echo "Bad Syntax. Usage: ${0} -user [username] -pass [password] required" 
        return 
    fi
    # previous checks omitted
}

We run it and see an error message:

$ login
Bad syntax. Usage: ./shell.sh -user [username] -pass [password] required

Let’s see what we did. We extracted our script name using the ${0} positional variable.

Notice that it evaluates to the script name (that contains our function) and not the function name.

3. Process and Job Variables

Sometimes, in complex scripts, we need the execution status of other commands to proceed further.

Additionally, we might need to execute long-running tasks in the background and check if they are finished.

Lucky for us, Bash offers some neat tricks to handle such situations. Let’s take a look at them.

3.1. Execution Status of the Last Command

Remember our previous login example? Let’s improve it a bit:

function login(){
    if [[ ${#} < 4 ]]; then
        echo "Bad syntax. Usage: ${0} -user [user] -pass [pass] required"
        return 15
    fi 
    if [[ ${@} =~ "-user" && ${@} =~ "-pass" ]]; then
        echo "Yehaww"
    else
        echo "Bad Syntax. Usage: -user [username] -pass [password] required"
        return 16
    fi
}

We return two error codes: 16 and 15.

Let’s use them to distinguish between errors in the calling code:

function check_login(){
    login
    login_rc=${?}
    if [[ $login_rc == 15 ]];then
        echo "Insufficient parameters to login function"
    elif [[ $login_rc == 16 ]];then
        echo "Parameters -user and -pass not sent to login function"
    elif [[ $login_rc == 0 ]];then
        echo "Everthing is awesome ... proceeding"
    fi
}

We use the ${?} construct to retrieve the execution status of the last command.

In this case, the login function will fail on the first check:

$ check_login
Bad syntax. Usage: ./shell.sh -user [username] -pass [password] required
Insufficient parameters to login function

Most importantly, we use an intermediate variable to check the return codes.

Let’s see what happens if we don’t:

function check_login(){
    login "one" "two"
    if [[ ${?} == 15 ]];then
        echo "Insufficient parameters to login function"
    elif [[ ${?} == 16 ]];then
        echo "Parameters -user and -pass not sent to login function"
    elif [[ ${?} == 0 ]];then
        echo "Everthing is awesome ... proceeding"
    fi
}

Let’s rerun the function and see the output:

$ check_login
Bad Syntax. Usage: -user [username] -pass [password] required

So what happened here?

Let’s add some verbosity with the set -x and rerun it:

function check_login(){
    set -x
    login one two
    # previous checks
}

In this case, the last command exit status is overwritten by each comparison:

$ check_login
+ login one two
+ [[ 2 == 0 ]]
+ [[ 2 == 1 ]]
+ [[ one two =~ -user ]]
+ echo 'Bad Syntax. Usage: -user [username] -pass [password] required'
Bad Syntax. Usage: -user [username] -pass [password] required
+ return 16
+ [[ 16 == 15 ]]
+ [[ 1 == 16 ]]
+ [[ 1 == 0 ]]

At first, the exit status is 16 as we expect it. After the first comparison (which fails), the exit status is 1.

This is correct since it represents the executed comparison exit code and returns a non-zero value.

3.2. Background Job Process Id

Let’s go back to our login function and assume it takes a while to execute:

function login() {
    echo "Sleeping for 3 seconds"
    sleep 3s
    # previous checks omitted
}

Now let’s call it as an asynchronous command, using the & control operator:

function check_login() {
    login &
    login_rc=${?}
    # previous checks omitted
}

Let’s run it and see what happens:

$ check_login
Everthing is awesome ... proceeding
Sleeping for 3 seconds
$ [original script finished]
...[after 3 seconds]
Bad syntax. Usage: ./shell.sh -user [username] -pass [password] required

Our calling function finished before the asynchronous login function returned. Also, our exit code checks became useless.

Luckily, we can use the ${!} construct to get our background process id and wait before continuing:

function check_login(){
    login &
    wait ${!}
    # previous checks omitted
}

Let’s take a look at the output:

$ check_login
Sleeping for 3 seconds
Bad syntax. Usage: ./shell.sh -user [username] -pass [password] required
Insufficient parameters to login function

This is a naive implementation. It makes no sense to launch a long-running process asynchronously and immediately wait for it to finish.

Let’s put it to better use and implement a simple parallel download of three Linux kernel versions:

function download_linux(){
    declare -a linux_versions=("5.7" "5.6.16" "5.4.44")
    declare -a commands
    mkdir linux
    for version in ${linux_versions[@]}
        do 
            curl -so ./linux/kernel-${version}.tar.xz -L \
            "https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-${version}.tar.xz" &
            echo "Running with pid ${!}"
            commands+=(${!})
        done   
    for pid in ${commands[@]}
        do
            echo "Waiting for pid ${pid}"
            wait $pid
        done
}

First, we launch the curl utility asynchronously for each of the kernel versions. Then, we retain the background process id.

Lastly, we wait for each background process to finish before we exit the function.

Let’s see what it prints:

$ download_linux
Running with pid 5699
Running with pid 5700
Running with pid 5701
Waiting for pid 5699
Waiting for pid 5700
Waiting for pid 5701

This takes a while to complete since each version has ~100MB in size.

We can examine in a different terminal, during the download, the contents of our linux destination folder:

$ ls linux/
kernel-5.4.44.tar.xz  kernel-5.6.16.tar.xz  kernel-5.7.tar.xz

3.3. Current Shell Process Id

So far, we’ve seen the background process id.

We can also identify the current shell process id using the ${$} construct:

function login(){
    echo "Current shell process id ${$}"
    echo "Sleeping for 3 seconds"
    sleep 3s
    # previous checks omitted
}

We use again our login function:

$ login
Current shell process id 6575
Sleeping for 3 seconds
Bad syntax. Usage: ./shell.sh -user [username] -pass [password] required

But what happens if we call our function asynchronously? Let’s see:

function check_login() {
    echo "Calling shell process id ${$}"
    login &
    wait ${!}
    # previous checks omitted
}

We get the output:

$ check_login
Calling shell process id 6772
Current shell process id 6772
Sleeping for 3 seconds
# same output as before

The ${$} expands to the calling shell process id. As a result, the shell process id returned by our background job is the same as the one in the main script.

We can write it to a PID file. Let’s put our login and check_login functions in a script:

#!/bin/bash
function login() { 
    # same body as before
}
function check_login() { 
    echo "${$}" > shell.pid
    login & 
    wait ${!} 
    # previous checks omitted 
}
check_login

Then, let’s implement a rudimentary watchdog script that checks if our main script is running or not:

#!/bin/bash
function watchdog(){
    pid=`cat ./shell.pid`
    if [[ -e /proc/$pid ]];then
        echo "Login still running"
    else 
        echo "Login is finished"
    fi
}
watchdog

First, we read the process id, and then we check if it exists in the virtual proc filesystem.

Let’s run our login script and, in a separate terminal, our watchdog script:

$ ./shell.sh
Current shell process id 7549
Sleeping for 3 seconds

$ ./watchdog.sh 
Login still running

These mechanisms are common on daemons, for shutdown or restart handling.

4. Other Special Variables

We’ve saved the most exotic built-in variables as last.  Let’s see what they’re about.

4.1. Current Shell Options

In a previous example, we used the set built-in with the –x option to enable verbosity.

We can print the current shell options using the ${-} construct.

Let’s rerun our check_login function:

function check_login() {
    echo "Before shell options ${-}"
    set -x
    echo "After shell options ${-}"
    # same body as before
}

Let’s see the options before and after setting the verbosity:

$ check_login
Before shell options hB
+ echo 'After shell options hxB'
After shell options hxB
# same debug output as before

By default, our shell has the h and B flags set. The h option enables creating a hash table of the recently executed commands for faster lookup, while the B option enables the brace expansion mechanism.

When we set the -x flag, we instruct Bash to perform command tracing.

Of course, there are many other options we can further explore in the manual.

4.2. The Underscore Variable

Let’s now take a look at the most intriguing built-in special variable: ${_}.

Depending on the context, it has different meanings, and that’s why it is a bit harder to grasp.

Let’s start a new terminal and see its current value:

$ echo ${_}
true

When we start a new shell, this expands to the last argument of the last command executed.

So, where does the true value come from then? The magic behind it is in our ~/.bashrc file on the last line:

$ cat ~/.bashrc
# For some news readers it makes sense to specify the NEWSSERVER variable here
#export NEWSSERVER=your.news.server
# some other comments
test -s ~/.alias && . ~/.alias || true

This can vary from one Linux distribution to another.

Let’s add a new command in there and spawn a new terminal:

test -s ~/.alias && . ~/.alias || true
echo lorem ipsum

Now let’s take a look at the prompt and see what ${_} expands to:

lorem ipsum
$ echo ${_}
ipsum

Since we modified our .bashrc, every shell we spawn will first prompt our dummy text.

When Bash expands the special underscored variable, it will populate it with the second argument of our echo.

In general, this kind of construct is useful when using single arguments commands:

$ mkdir test && cd ${_}
test $

But what happens if we launch a script instead? Let’s look at a simple one-liner:

#!/bin/bash
echo "This is underscore in our script before: ${_}"

Let’s run it from a terminal and pass a parameter to it:

$ ./shell.sh lorem
This is underscore in our script before: ./shell.sh

Now the special variable is expanded to the absolute path of the script file.

Inside the script, it behaves similar to our first scenario:

#!/bin/bash
echo "This is underscore in our script before: ${_}"
echo "demo1234"
echo "This is underscore in our script after: ${_}"

Let’s see what it outputs:

$ ./shell.sh
This is underscore in our script before: ./shell.sh
demo1234
This is underscore in our script after: demo1234

On the other hand, a general source of confusion with Bash is the existence of an environment variable with the valid name <_>.

Let’s try this hack in a new terminal:

$ env -i _=lorem/ipsum bash -c 'echo "First time [${_}]" && echo "Second time [${_}]"'
First time [lorem/ipsum]
Second time [First time [lorem/ipsum]]

Well, that’s a bit misleading. Let’s explain a bit about what happened here. We first launched a new environment with the env command.

Then, we set a new value to the environment variable <_>, which was picked on our first echo call.

After that, on the second echo call, Bash replaced its value with the last argument of our first echo call.

Finally, the underscore variable also has a special meaning in the context of Bash MAILPATH.

5. Conclusion

In this tutorial, we’ve seen the Bash special variables in action. First, we looked at parameter expansions and explained when and how to use them.

Next, we jumped on the process and job-related variables. We saw how to get the execution status of the last command and also how to wait for background processes.

Finally, we saw how to list the shell options and the different behaviors of the underscore special variable.