The Shell, the Whole Shell, and Nothing but the Shell

| Comments

A recent turn of events left me debugging an environment with a shell and little to nothing else. There were some log files in memory on a server that had somehow gotten into a state where it appeared to have no binaries. Essentially, the network continued to work, but the only access we had to the server was via and already open login shell on the serial console. Lucky for us it happened to be bash

How well do you know your shell builtins? Knowing what was built in in this situation was the difference between losing the state of the machine by rebooting into a live environment and being able to investigate without destroying that state.

If you find yourself in a similar situation, here are some helpful hints.

Changing directories

Think about cd. All it is doing is setting the current process’s working directory. Obviously, this is a shell builtin, so we’re doing okay.

Listing files

First discoveries: ls is not a shell builtin. One of those things you likely know, but have already developed the nervous habit of just typing ls to pass the time. Let’s take care of that first by creating a simple little function.

1

ls() { echo *; }

echo is a shell builtin. Globbing is also a shell builtin.

Examining files

Now we can find our files, but how do we look at them? less and more and cat are all binaries. Read and while aren’t.

1
2
3
4
5
6
7

cat() {
    for file in "$@"; do
        while read line; do
            echo $line
        done < "$file"
    done
}

You can also implement head, tac, and tail fairly easily, though I think tac will require having the entire file in memory. tail would require a linear scan, but not the memory, as you only need to keep a lookbehind buffer sufficient to print the lines you care about.

Filtering files

All right, let’s take a quick look in that file and see if we can match our carefully constructed regular expression using grep.

# grep
bash: grep: command not found
# builtin grep
bash: builtin: grep: not a shell builtin

Darn. Can we do this one in shell?

1
2
3
4
5
6
7
8
9

grep () {
    regex="$1"
    shift
    for file in "$@"; do
        while read line; do
            [[ $line =~ $regex ]] && echo $line
        done < "$file"
    done
}

Cool. Does it work?

# grep '^[a-z]wesome$' words
awesome

Note that it won’t work for pipes as written.

Running processes

Since /proc is still mounted, implementing a rudimentary ps is fairly straightforward.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

ps() {
    printf "  PID\t STATE\t CMD\n"
    for pdir in /proc/[0-9]*; do
        pdir_pid=${pdir##*/}
        cmd=$(cat "${pdir}/comm")
        # man 5 proc
        read pid comm state ppid pgrp session tty_nr tpgid flags minflt \
            cminflt majflt cmajflt utime stime cutime cstime priority nice \
            num_threads itrealvalue starttime vsize rss rsslim startcode \
            endcode startstack kstkesp kstkeip signal blocked sigignore \
            sigcatch wchan nswap cnswap exit_signal processor rt_priority \
            policy delayacct_blkio_ticks guest_time cguest_time \
            < /proc/$pdir_pid/stat
        printf "$pid\t $state\t $cmd\n"
    done
}

Getting files across the network

My initial assertion was that the log files we needed could be transferred over the network. The secret here is to use the builtin /dev/tcp syntactic sugar for making outbound network connections.

On the server side, you’ll want netcat any simple network listener:

1
2
3
4
5

receive_file() {
    local file
    file="$1"
    nc -l -p 9988 | tee "$file"
}

On our client side:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

send_file() {
    local server="$1"
    local port="$2"
    local file="$3"
    if [ -z $server ]; then
        echo "no server" >&2
        return
    fi
    if [ -z $port ]; then
        echo "no port" >&2
        return
    fi
    if [ -z $file ]; then
        echo "no file" >&2
        return
    fi
    cat "${file}" > /dev/tcp/$server/$port
}

Remember, you might be in an enviornment where you don’t have nsswitch or resolv, so you likely need to use an IP address for the server.

As for why this works even when /dev doesn’t exist - /dev/tcp (and /dev/udp) is interpreted by the shell itself. For bash, you will need to have a copy compiled with --enable-net-redirections. For zsh, there is the zsh/net/tcp module that allows you to do similar things with a builtin ztcp command.

Closing remarks

While this situation itself wasn’t contrived, these examples were documented in a contrived environment. I created a chroot on Debian Jessie with only the following files:

/bin
/bin/bash
/lib
/lib/x86_64-linux-gnu
/lib/x86_64-linux-gnu/ld-2.18.so
/lib/x86_64-linux-gnu/libtinfo.so.5
/lib/x86_64-linux-gnu/libtinfo.so.5.9
/lib/x86_64-linux-gnu/libdl-2.18.so
/lib/x86_64-linux-gnu/libdl.so.2
/lib/x86_64-linux-gnu/libc.so.6
/lib64
/lib64/ld-linux-x86-64.so.2
/usr/share/dict/words
/proc

I then mounted a proc filesytem and chrooted to make the examples. Should I keep working on these and put them on github? Call the project builtouts?

Keep in mind, the best parts of the above post are unfortunately not POSIX, namely the =~ operator and /dev/tcp.

Comments