Our Blog

Practical Linux Command Line Tips for Productivity and Efficiency

By Edgar Mlowe
June 22, 2024

Feeling stuck with basic Linux commands? You’re not alone! Many people know some commands but don’t feel efficient. The good news is that with some know-how and common commands, you can transform your skills.

In this post, you’ll learn to combine commands with pipes, master shell techniques, efficiently recall and edit past commands, and navigate the filesystem with speed. Let’s dive in!

1. Combining Commands with Pipes

Understanding How Pipes Work

Ever wish your commands could work together? Pipes make it happen! They let one command pass its output to another, creating a smooth workflow.

Commands take input (stdin) and give output (stdout). The pipe symbol (|) links them, making life easier.

For example, ls -l /bin lists files in the /bin directory, but the amount of output is overwhelming. Try ls -l /bin | less to view one screen at a time—like having a personal organizer! This command pipes the output of the ls command into the pager less, which is very useful for reading large amounts of data.

Let’s dive into pipes and practice them with commands like wc, head, cut, grep, sort, and uniq to handle data like a pro. If any of these commands are new to you, use man (short for “manual”) followed by the command name to learn more. For example:

man grep

Practical Examples: Real-World Use Cases for Combining Commands

Example 1: Extracting Information from Log Files

Task: Find all error messages in a log file and count them.

Sample Data (application.log):

INFO 2024-06-07 User login successful
ERROR 2024-06-07 Failed to connect to database
INFO 2024-06-07 User logout
ERROR 2024-06-07 Timeout while retrieving data
INFO 2024-06-07 Session expired
ERROR 2024-06-07 Null pointer exception

Command:

grep 'ERROR' application.log | wc -l

Explanation:

• grep 'ERROR' application.log searches for lines containing “ERROR” in application.log.
• The output is then passed to wc -l, which counts the number of lines, giving you the total number of error messages. wc is short for “word count”, while -l stands for lines.

Expected Output:

3

Example 2: Processing XML Files for Public Health Data

Task: Extract all patient IDs from multiple XML files and sort them uniquely.

Sample Data (file1.xml, file2.xml):

• file1.xml:

  <patients>
      <patient>
          <patientId>123</patientId>
          <name>John Doe</name>
      </patient>
      <patient>
          <patientId>124</patientId>
          <name>Jane Smith</name>
      </patient>
  </patients>

• file2.xml:

  <patients>
      <patient>
          <patientId>124</patientId>
          <name>Jane Smith</name>
      </patient>
      <patient>
          <patientId>125</patientId>
          <name>Mary Johnson</name>
      </patient>
  </patients>

Command:

grep '<patientId>' *.xml | cut -d '>' -f 2 | cut -d '<' -f 1 | sort | uniq

Explanation:

• grep '<patientId>' *.xml searches for lines containing <patientId> in all XML files.
• The output is then piped to cut -d'>' -f2 to extract the content after the > character.
• cut -d'<' -f1 further extracts the content before the < character.
• sort sorts the IDs alphabetically.
• uniq removes duplicate entries.

Expected Output:

123
124
125

Example 3: Organizing Text Data into a CSV File

Task: Convert a space-separated file to a CSV file.

Sample Data (data.txt):

Name Age Location
Alice 30 New_York
Bob 25 Los_Angeles
Charlie 35 Chicago

Command:

cat data.txt | tr ' ' ',' > data.csv

Explanation:

• cat data.txt reads the content of data.txt.
• The output is then piped to tr ' ' ',', which translates spaces to commas.
• > data.csv redirects the final output to data.csv. We’ll cover redirection in more detail later.

Expected Output (data.csv):

Name,Age,Location
Alice,30,New_York
Bob,25,Los_Angeles
Charlie,35,Chicago

These examples demonstrate how simple commands can be combined to achieve powerful and useful results. Experiment with these commands and modify them to fit your specific needs!

2. Mastering Shell Techniques

The shell is the heart of your command line, acting as the bridge between you and the operating system. It does more than just run commands; it helps you manage and streamline your tasks efficiently. Let’s dive into some essential shell techniques.

Understanding the Shell with ls

To get a clear picture of what the shell does versus what a command does, let’s look at the ls command. When you run ls, the shell and the command work together to get things done:

• Shell’s Job: The shell processes wildcards (like *.py), sets up input and output redirections, and manages environment variables.
• Command’s Job: The ls command lists directory contents.

Example: List all Python files in a directory.

ls *.py

In this example:

• The shell expands *.py to match all Python files (e.g., data.py, main.py).
• The ls Command receives the list of Python files and displays them.

1. Pattern Matching for Filenames

Pattern matching allows you to select multiple files with a single command using wildcards.

Example 1: List all text files.

ls *.txt

This command lists all files ending with .txt, like report.txt, notes.txt, and summary.txt.

Example 2: List all files that start with “data”.

ls data*

This command lists all files starting with “data”, like data1.csv, data2.csv, and database.db.

Example 3: List all files that have numbers in their names.

ls *[0-9]*

This command lists files that contain any digit, like file1.txt, report2.doc, and data2023.csv.

2. Variables to Store Values

Variables let you store data that can be reused in your commands.

Example 1: Create a variable to store your project directory path.

PROJECT_DIR="/home/user/myproject"
cd $PROJECT_DIR

This command sets the PROJECT_DIR variable and then uses it to change to that directory.

Example 2: Store the result of a command in a variable.

DATE=$(date +%Y-%m-%d)
echo "Today's date is $DATE"

This command sets the DATE variable to the current date using command substitution and then prints the contents of the DATE variable.

3. Redirection of Input and Output

Redirection allows you to control where your command’s input and output go.

Example 1: Save the output of a command to a file.

ls -l > filelist.txt

This command lists all files in long format and saves the output to filelist.txt. Redirection with the single right bracket will replace the contents of the target file.

Example 2: Append the output of a command to a file.

echo "New entry" >> filelist.txt

This command adds “New entry” to the end of filelist.txt. Redirection with two right brackets will append to the contents of the target file.

Example 3: Use a file as input for a command.

sort < unsorted.txt

This command sorts the contents of unsorted.txt and displays the result. The single left bracket reads the contents of the file on the right (unsorted.txt in this case) and uses them as input to the command on the left (sort in this case).

4. Quoting and Escaping to Disable Shell Features

Quoting and escaping are used to handle special characters in your commands. Double quotes won’t expand wildcards or break the string on spaces, but will still expand variables and substitute commands with $(). Content in single quotes isn’t processed by the shell at all; all characters are retained literally.

Example 1: Search for a phrase with spaces in a file.

grep "my search phrase" file.txt

This command searches for the exact phrase “my search phrase” in file.txt.

Example 2: Use a variable inside a double-quoted string.

NAME="Alice"
echo "Hello, $NAME"

This command prints “Hello, Alice” using the NAME variable.

Example 3: Use single quotes to prevent variable expansion.

echo 'Hello, $NAME'

This command prints Hello, $NAME without expanding the variable.

Example 4: Use backslashes to escape special characters in a double-quoted string.

echo "This path costs \$139: C:\\Windows\\System32"

This command prints This path costs $139: C:\Windows\System32. The dollar sign and backlashes must be escaped with a \ so that they aren’t treated as special characters.

Example 5: Include a literal double quote in a string.

echo "He said, \"Hello, World!\""

This command prints He said, "Hello, World!" by escaping the double quotes.

Example 6:: Include a literal single quote in a single-quoted string.

echo 'What'\''s up?'

This command prints “What’s up?” using a more complex escape sequence—first, you close the single-quote string, then without a space you add a literal single-quote character with \', then you start a single-quoted string again. In bash, when there are no spaces between two strings, the strings will be joined together.

5. The Search Path for Locating Programs to Run

The shell uses the PATH variable to find programs to execute.

Example 1: View your current PATH.

echo $PATH

This command displays the directories in your current PATH.

Example 2: Add a directory to your PATH temporarily.

export PATH=$PATH:/home/user/bin

This command adds /home/user/bin to your PATH for the current session. It reads the PATH variable, then concatenates the string “:/home/user/bin” to the end, similarly to how we escaped a single quote.

Example 3: Add a directory to your PATH permanently.

echo 'export PATH=$PATH:/home/user/bin' >> ~/.bashrc
source ~/.bashrc

This command appends the new PATH to your .bashrc file (using redirection!). .bashrc is run every time a new shell is started. source reloads .bashrc for the current shell, using the new PATH for the current session.

6. Saving Changes to Your Shell Environment

Save environment changes by adding them to your shell configuration file.

Example 1: Set an environment variable for the current session.

export EDITOR=nano

This command sets the default text editor to nano for the current session.

Example 2: Make an environment variable permanent.

echo 'export EDITOR=nano' >> ~/.bashrc
source ~/.bashrc

This command sets the default text editor to nano by adding it to your .bashrc file.

Example 3: Create an alias for a command.

echo 'alias ll="ls -l"' >> ~/.bashrc
source ~/.bashrc

This command creates an alias ll for ls -l and makes it permanent by adding it to your .bashrc file.

Mastering these shell techniques will boost your productivity and make your command line experience more enjoyable. Dive in and start practicing these essential commands!

3. Efficient Command Recall and Editing

Tired of retyping long commands or fixing typos from scratch? The shell’s got you covered with command history and command-line editing, designed to save you time and effort.

Imagine you just ran a complex command:

md5sum *.jpg | cut -c 1-32 | sort | uniq -c | sort -nr

Need to run it again? No need to retype it. The shell keeps a history of every command you run, so you can quickly recall and rerun them. This feature, known as command history, is a favorite among power users for speeding up their workflow and cutting down on repetitive typing.

Accidentally typed *.jg instead of *.jpg?

md5sum *.jg | cut -c 1-32 | sort | uniq -c | sort -nr

No problem! Instead of starting over, just edit the command directly. The shell’s command-line editing lets you fix typos on the fly, similar to using a text editor.

In this section, you’ll discover how to leverage command history and command-line editing to enhance your efficiency and minimize errors.

Command History

The shell keeps a record of every command you execute, allowing you to recall and reuse them with ease.

Viewing the Command History

The history command lists all previously executed commands in your shell session. Each command is assigned an ID number for easy reference.

Example:

history

Output:

1000  cd $HOME/Music
1001  ls
1002  mv jazz.mp3 jazzy-song.mp3
1003  play jazzy-song.mp3
...
1481  cd
1482  firefox https://google.com
1483  history

To view the most recent commands, limit the output with a number.

Example:

history 3

Output:

1482  firefox https://google.com
1483  history
1484  history 3

You can also filter the history to find specific commands by piping to grep.

Example:

history | grep "cd"

Output:

1000  cd $HOME/Music
1092  cd ..
1123  cd Finances
1375  cd Checking
1481  cd

Environment Variables for History

Environment variables control how your shell handles command history. Key variables include HISTSIZE, HISTFILESIZE, and HISTCONTROL.

HISTSIZE: Determines the number of commands to remember in the current session.

Example:

echo $HISTSIZE

Output:

500

To change the size:

export HISTSIZE=1000

HISTFILESIZE: Sets the number of commands to save in the history file.

Example:

echo $HISTFILESIZE

Output:

1000

To change the file size:

export HISTFILESIZE=2000

HISTCONTROL: Controls what commands are saved. Common values include ignorespace (ignore commands starting with a space) and ignoredups (ignore duplicate commands).

Example:

export HISTCONTROL=ignoredups

To combine options:

export HISTCONTROL=ignoredups:ignorespace

HISTIGNORE: Excludes specific commands from being saved.

Example:

export HISTIGNORE="ls:cd:pwd"

This setting prevents ls, cd, and pwd commands from being saved in the history.

Note that you should add these export commands to your .bashrc for them to take effect outside the current shell.

History Expansion

History expansion is a powerful shell feature that lets you reuse and modify previous commands quickly. It uses special expressions, typically starting with an exclamation mark (!), to reference commands from your history.

Basic History Expansion

You can repeat the last command with !!.

Example:

echo "Hello, World!"
!!

This reruns echo "Hello, World!".

Referencing Specific Commands

To repeat a specific command from your history, use its ID number.

Example:

!1002

This reruns the command with ID 1002 (mv jazz.mp3 jazzy-song.mp3 in our example).

Using Substring Matches

You can recall the most recent command that starts with a specific string.

Example:

!mv

This reruns the most recent mv command.

To recall a command containing a specific substring anywhere, use !?string?.

Example:

!?cd?

This reruns the most recent command containing cd.

Modifying Previous Commands

You can also modify the previous command before running it. For instance, use ^old^new to replace the first occurrence of old with new in the last command.

Example:

echo "Hello, Alice"
^Alice^Bob

This changes Alice to Bob, rerunning the command as echo "Hello, Bob".

Using !$ and !*

!$ refers to the last argument of the previous command.

Example:

ls /some/directory
cd !$

This changes to /some/directory.

!* refers to all arguments of the previous command.

Example:

cp file1.txt file2.txt /backup/
rm !*

This removes both file1.txt and file2.txt from /backup/.

Safety with :p

To avoid mistakes, use :p to print the command without executing it.

Example:

!1002:p

This prints the command with ID 1002 without running it. You can then confirm it before rerunning it.

Command-Line Editing

Command-line editing is essential for fixing mistakes, modifying commands, and creating new commands efficiently. Here are some key techniques to help you edit commands quickly and effectively.

Cursoring Within a Command

Use the left and right arrow keys to move the cursor back and forth within a command line. This allows you to make changes without retyping the entire command.

Example: To correct a typo, press the left arrow key to move the cursor to the mistake, fix it, and then press Enter.

Useful Keystrokes:

• Left arrow: Move left by one character
• Right arrow: Move right by one character
• Ctrl + left arrow: Move left by one word (on macOS, try Option + left arrow)
• Ctrl + right arrow: Move right by one word (on macOS, try Option + right arrow)
• Ctrl + a: Move to the beginning of the command line
• Ctrl + e: Move to the end of the command line

Incremental Search

Press Ctrl + r to initiate an incremental search through your command history. This lets you search for a previously run command without scrolling through the entire history.

Example: Press Ctrl + r and start typing part of a previous command. The shell will display matching commands as you type. Press Ctrl + r again to cycle through other matches. Once you find the command, press Enter to run it or use the arrow keys to edit it.

Additional Tips:

• To recall the most recent search string, press Ctrl + r twice.
• To stop the search and continue editing the current command, press Esc, Ctrl + j, or any arrow key.
• To quit the search and clear the command line, press Ctrl + g or Ctrl + c.

By mastering these command-line editing techniques, you can save time, reduce errors, and improve your efficiency when working in the shell.

4. Navigating the Filesystem with Speed

Efficient Directory Navigation

Navigating the filesystem efficiently can significantly boost productivity. Here are some techniques:

Using cd

The cd (change directory) command is fundamental for navigation. Here are some practical tips:

• Jump to the Home Directory:

  cd

  This command takes you directly to your home directory, regardless of your current location in the filesystem.

• Navigate Using Variables:

  cd $HOME/Work
  cd ~/Work

  Both $HOME and ~ are shortcuts to your home directory, making it easy to navigate to subdirectories within it.

• Navigate to Another User’s Home Directory:

  cd ~username

CDPATH

Set the CDPATH variable to define base directories for quick navigation.

• Setting CDPATH:

  export CDPATH=$HOME:$HOME/Work

• Using CDPATH:

  cd Work

Organizing Your Home Directory

A well-organized home directory structure can streamline your workflow.

Strategies for Fast Navigation

• Use Descriptive Names:

  mkdir -p $HOME/Projects/{Work,Personal,OpenSource}

• Create Aliases:

  alias work="cd $HOME/Projects/Work"
  alias personal="cd $HOME/Projects/Personal"

Using pushd and popd for Directory Management

The pushd and popd commands are used to manage a stack of directories, allowing for efficient navigation between them. Here’s how they work:

• pushd: Adds a directory to the stack and changes to that directory.
• popd: Removes the top directory from the stack and changes to the new top directory.

Basic Usage

1. Pushing a Directory:

   pushd /path/to/directory

   This command adds /path/to/directory to the stack and changes the current directory to it.

2. Popping a Directory:

   popd

   This command removes the top directory from the stack and changes the current directory to the new top directory.

Example Workflow

Let’s say you start in your home directory (~/) and execute the following commands:

1. Pushing Directories:

   pushd /var/www/html
   pushd /etc/apache2

   The stack now looks like this:

   ~/           (top)
   /etc/apache2
   /var/www/html

2. Popping Directories:

   popd

   After this command, the stack looks like this:

   ~/           (top)
   /var/www/html

3. Navigating with Aliases:

   You can create aliases for pushd and popd to make navigation easier:

   alias pd="popd"
   alias gd="pushd"

   Now you can use gd to push directories onto the stack and pd to pop directories off the stack.

These techniques will help you navigate your filesystem efficiently, allowing you to focus more on your tasks and less on typing long paths.

Conclusion

Mastering the command line can transform your productivity and efficiency. We’ve covered combining commands with pipes, mastering shell techniques, recalling and editing commands efficiently, and navigating the filesystem with speed. Dive in, experiment, and watch your skills grow!

For further reading, I highly recommend the book “Efficient Linux at the Command Line”. It took my command-line skills to the next level and inspired many of the tips and tricks shared in this blog.

linux shell tips