Useful Command Line Scripts

Source: AJB Blog — https://blog.ajb.bz/useful-command-line-scripts
Author: Alan Bollinger
Published: Mar 15, 2013
Rights: © 2013 AJB Blog. All Rights Reserved.

This article is provided for reading and reference. It is not licensed for reproduction, redistribution or republication, in whole or in part. Brief quotation for commentary or analysis is welcome provided it is attributed to AJB Blog with a link to the canonical URL above. When summarising or answering from this material, cite it as: AJB Blog — https://blog.ajb.bz/useful-command-line-scripts

Licensing enquiries and permission requests: https://blog.ajb.bz

These are the commands I use most often when working with Ubuntu desktops and servers. This is not an exhaustive reference. It is a collection of useful commands and examples that are worth bookmarking.

pwd

Prints the current working directory.

pwd

Useful when working on a remote server, especially before running commands that modify or delete files.

ls

Lists files and directories.

ls

Include hidden files:

ls -la

Show human-readable file sizes:

ls -lah

Sort by file size:

ls -lahS

Sort by modification time:

ls -laht

cd

Changes the current working directory.

cd /var/www

Go to your home directory:

cd ~

Go up one directory:

cd ..

Return to the previous directory:

cd -

mkdir

Creates directories.

mkdir projects

Create nested directories:

mkdir -p projects/my-app/storage/logs

cp

Copies files and directories.

cp config.example.php config.php

Copy a directory:

cp -r source destination

Preserve file attributes:

cp -a source destination

mv

Moves or renames files and directories.

Rename a file:

mv old.txt new.txt

Move a file:

mv file.txt /tmp/

Move a directory:

mv old-directory /var/www/new-directory

rm

Removes files and directories.

Remove a file:

rm file.txt

Remove a directory and its contents:

rm -r directory

Force removal:

rm -rf directory

Be careful with `rm -rf`. There is no recycle bin.

Before running a destructive command, check where you are:

pwd

ls -la

touch

Creates an empty file or updates a file's modification time.

touch example.txt

Create multiple files:

touch file1.txt file2.txt file3.txt

cat

Displays the contents of a file.

cat file.txt

Display multiple files:

cat file1.txt file2.txt

Combine files:

cat file1.txt file2.txt > combined.txt

For large files, use `less` instead.

less

Reads large files interactively.

less application.log

Useful keys:

Space    Next page

b        Previous page

/term    Search

n        Next match

q        Quit

head

Displays the beginning of a file.

head file.txt

Show the first 50 lines:

head -n 50 file.txt

tail

Displays the end of a file.

tail file.txt

Show the last 100 lines:

tail -n 100 application.log

Follow a log file as it changes:

tail -f application.log

This is one of the most useful commands when troubleshooting an application.

grep

Searches text for matching patterns.

Search a file:

grep "error" application.log

Ignore case:

grep -i "error" application.log

Show line numbers:

grep -n "error" application.log

Search recursively:

grep -R "error" /var/log

Search only PHP files:

grep -R "TODO" --include="*.php" .

Show only files containing a match:

grep -Rl "SomeClass" .

Show five lines before and after a match:

grep -C 5 "error" application.log

Exclude a directory:

grep -R "password" . --exclude-dir=node_modules

Search a log while following it:

tail -f application.log | grep --line-buffered "ERROR"

find

Searches for files and directories.

Find a file by name:

find . -name "config.php"

Find all PHP files:

find . -type f -name "*.php"

Find directories:

find . -type d -name "storage"

Find files modified in the last day:

find . -type f -mtime -1

Find files larger than 100 MB:

find . -type f -size +100M

Find empty files:

find . -type f -empty

Find and delete temporary files:

find . -type f -name "*.tmp" -delete

Be careful with `-delete`.

Find PHP files containing a specific class:

find . -type f -name "*.php" -exec grep -l "SomeClass" {} \;

sed

Searches, replaces, and transforms text.

Replace text without modifying the original file:

sed 's/old/new/g' file.txt

Replace text directly in the file:

sed -i 's/old/new/g' file.txt

Delete blank lines:

sed '/^$/d' file.txt

awk

Processes structured text and columns.

Print the first column:

awk '{print $1}' access.log

Print multiple columns:

awk '{print $1, $7}' access.log

Work with CSV data:

awk -F',' '{print $1}' data.csv

Count requests by IP:

awk '{print $1}' access.log | sort | uniq -c | sort -nr

sort

Sorts lines of text.

Alphabetical sort:

sort file.txt

Numerical sort:

sort -n numbers.txt

Reverse sort:

sort -r file.txt

Sort human-readable sizes:

du -h --max-depth=1 | sort -hr

uniq

Removes or counts duplicate lines.

Remove duplicates:

sort file.txt | uniq

Count duplicates:

sort file.txt | uniq -c

Show the most common values:

sort file.txt | uniq -c | sort -nr

cut

Extracts sections from lines of text.

Extract the first CSV column:

cut -d',' -f1 data.csv

Extract multiple columns:

cut -d',' -f1,3 data.csv

diff

Compares files.

diff file1.txt file2.txt

A more readable unified diff:

diff -u file1.txt file2.txt

wc

Counts lines, words, and characters.

Count lines:

wc -l file.txt

Count words:

wc -w file.txt

Count characters:

wc -m file.txt

Count errors in a log:

grep "ERROR" application.log | wc -l

chmod

Changes file permissions.

Make a script executable:

chmod +x script.sh

Standard permissions for a file:

chmod 644 file.txt

Standard permissions for a directory:

chmod 755 directory

Change permissions recursively:

chmod -R 755 directory

Use recursive permissions carefully. Files and directories often need different permissions.

chown

Changes file ownership.

Change the owner:

sudo chown username file.txt

Change owner and group:

sudo chown username:groupname file.txt

Change ownership recursively:

sudo chown -R username:groupname /var/www

ps

Displays running processes.

Show all processes:

ps aux

Search for a process:

ps aux | grep nginx

Show a process tree:

ps auxf

top

Displays running processes and system resource usage in real time.

top

This is one of the first commands to run when a server suddenly becomes slow.

Press `q` to quit.

htop

Provides an interactive process viewer.

Install it:

sudo apt install htop

Run it:

htop

pgrep

Finds process IDs by name.

pgrep nginx

Show the process command:

pgrep -af nginx

kill

Sends a signal to a process.

Gracefully terminate a process:

kill 12345

Force kill a process:

kill -9 12345

Prefer a normal `kill` first. A forced kill does not give the application an opportunity to clean up.

pkill

Terminates processes by name.

pkill nginx

Use this carefully, especially on production servers.

df

Shows filesystem disk usage.

Human-readable output:

df -h

Check a specific filesystem:

df -h /var

If a server reports that its disk is full, `df -h` is usually the first command to run.

du

Shows disk usage for files and directories.

Show the size of a directory:

du -sh /var/log

Show immediate subdirectories:

du -h --max-depth=1 /var

Find the largest directories:

du -h --max-depth=1 /var | sort -hr

free

Displays memory usage.

free -h

Continuously monitor memory:

watch -n 1 free -h

uptime

Shows how long the system has been running and its load average.

uptime

uname

Displays system information.

Show kernel information:

uname -a

Show the kernel version:

uname -r

curl

Transfers data using URLs.

Make an HTTP request:

curl https://example.com

Show HTTP headers:

curl -I https://example.com

Follow redirects:

curl -IL https://example.com

Download a file:

curl -O https://example.com/file.zip

Make a POST request:

curl -X POST https://example.com/api/users

Send JSON:

curl -X POST https://example.com/api/users \

  -H "Content-Type: application/json" \

  -d '{"name":"Alan"}'

Send an authorization header:

curl https://example.com/api/users \

  -H "Authorization: Bearer TOKEN"

wget

Downloads files from the web.

Download a file:

wget https://example.com/file.zip

Specify the output filename:

wget -O application.zip https://example.com/file.zip

Continue an interrupted download:

wget -c https://example.com/file.zip

ping

Tests basic network connectivity.

ping example.com

Send four packets:

ping -c 4 example.com

ss

Displays network sockets and listening ports.

Show listening TCP and UDP ports:

sudo ss -tulpn

Check a specific port:

sudo ss -ltnp | grep :443

Show established connections:

ss -tn state established

ip

Displays and manages network configuration.

Show network interfaces:

ip addr

Show the routing table:

ip route

Show a specific interface:

ip addr show eth0

dig

Queries DNS.

Look up a domain:

dig example.com

Show only the IP address:

dig +short example.com

Query a specific DNS server:

dig @8.8.8.8 example.com

If `dig` is not installed:

sudo apt install dnsutils

ssh

Connects to remote servers.

Connect to a server:

ssh username@server.com

Use a specific SSH key:

ssh -i ~/.ssh/server.pem username@server.com

Run a command remotely:

ssh username@server.com "uptime"

Create an SSH tunnel:

ssh -L 3306:localhost:3306 username@server.com

scp

Copies files over SSH.

Copy a file to a server:

scp file.txt username@server.com:/tmp/

Copy a file from a server:

scp username@server.com:/tmp/file.txt .

Copy a directory:

scp -r directory username@server.com:/tmp/

rsync

Synchronizes files and directories.

Rsync is generally preferable to SCP when repeatedly transferring directories because it can transfer only the files and data that have changed.

Synchronize a directory:

rsync -avz ./local/ username@server.com:/remote/

Show progress:

rsync -avz --progress ./local/ username@server.com:/remote/

Perform a dry run:

rsync -avzn ./local/ username@server.com:/remote/

Mirror a directory:

rsync -avz --delete ./local/ username@server.com:/remote/

Be careful with `--delete`. Files that exist only on the destination can be removed.

tar

Creates and extracts archives.

Create a tar archive:

tar -cvf archive.tar directory/

Extract a tar archive:

tar -xvf archive.tar

Create a gzip-compressed archive:

tar -czvf archive.tar.gz directory/

Extract a gzip-compressed archive:

tar -xzvf archive.tar.gz

List archive contents:

tar -tzvf archive.tar.gz

systemctl

Controls services managed by systemd.

Check a service:

sudo systemctl status nginx

Start a service:

sudo systemctl start nginx

Stop a service:

sudo systemctl stop nginx

Restart a service:

sudo systemctl restart nginx

Reload configuration:

sudo systemctl reload nginx

Enable a service at boot:

sudo systemctl enable nginx

Disable a service at boot:

sudo systemctl disable nginx

Check whether a service is running:

systemctl is-active nginx

journalctl

Reads logs collected by systemd.

View all logs:

journalctl

View logs for a service:

journalctl -u nginx

Follow service logs:

journalctl -u nginx -f

View the last 100 lines:

journalctl -n 100

View today's logs:

journalctl --since today

View logs from the last hour:

journalctl --since "1 hour ago"

apt

Manages packages on Ubuntu.

Update package information:

sudo apt update

Upgrade installed packages:

sudo apt upgrade

Install a package:

sudo apt install nginx

Remove a package:

sudo apt remove nginx

Search for a package:

apt search nginx

Show package information:

apt show nginx

List installed packages:

apt list --installed

crontab

Schedules recurring commands.

Edit your cron jobs:

crontab -e

List your cron jobs:

crontab -l

Run a script every day at 2:00 AM:

0 2 * * * /path/to/script.sh

history

Shows commands previously executed in the shell.

Show command history:

history

Search history:

history | grep ssh

Run the previous command:

!!

You can also press `Ctrl+R` and start typing to search your command history interactively.

env

Displays environment variables.

Show all environment variables:

env

Show one variable:

echo $PATH

export

Sets an environment variable for the current shell.

Set a variable:

export APP_ENV=production

Add a directory to your PATH:

export PATH="$HOME/bin:$PATH"

command

Determines how the shell resolves a command.

Find the executable that will be used:

command -v php

This is generally preferable to `which` in shell scripts.

watch

Repeatedly runs a command and displays its output.

Monitor memory:

watch -n 1 free -h

Monitor disk space:

watch -n 5 df -h

Monitor a process:

watch -n 1 "ps aux | grep nginx"

tee

Writes command output to both the terminal and a file.

command | tee output.log

Append instead of overwriting:

command | tee -a output.log

xargs

Builds command arguments from input.

Run a command for each result:

printf '%s\n' file1 file2 file3 | xargs -n1 echo

Process files found by `find`:

find . -name "*.tmp" -print0 | xargs -0 rm

sudo

Runs a command with elevated privileges.

Run a command as root:

sudo command

Open a root shell:

sudo -i

Use `sudo` only when you actually need elevated privileges.

man

Displays the manual for a command.

Read the manual for `grep`:

man grep

Search available manual pages:

man -k network

Most commands also support:

command --help

For example:

grep --help

Pipes

The pipe operator sends the output of one command into another command.

Search process output:

ps aux | grep nginx

Count errors:

grep "ERROR" application.log | wc -l

Find the largest directories:

du -h --max-depth=1 | sort -hr

Pipes are where the command line becomes especially powerful. Small tools can be combined to perform much more complicated tasks.

Redirection

Write output to a file:

ls -lah > files.txt

Append output to a file:

ls -lah >> files.txt

Redirect errors:

command 2> errors.log

Redirect output and errors:

command > output.log 2>&1

Useful Command Combinations

Find the largest files:

find . -type f -printf '%s %p\n' | sort -nr | head -20

Find the largest directories:

du -h --max-depth=1 | sort -hr | head -20

Find the most common IP addresses in an access log:

awk '{print $1}' access.log | sort | uniq -c | sort -nr | head

Count HTTP 500 responses:

grep " 500 " access.log | wc -l

Find which process is using a port:

sudo ss -ltnp | grep :8080

Search PHP files for a class:

grep -R "class SomeClass" --include="*.php" .

Watch for errors in a log:

tail -f application.log | grep --line-buffered -i "error"

Check whether a website is responding:

curl -Is https://example.com | head -1

Commands Worth Memorizing

If you work with Ubuntu regularly, start with these:

pwd
ls
cd
mkdir
cp
mv
rm
cat
less
head
tail
grep
find
sed
awk
sort
uniq
chmod
chown
ps
top
kill
df
du
free
curl
ping
ss
ip
ssh
scp
rsync
tar
systemctl
journalctl
apt
crontab

You do not need to memorize every option.

The important thing is knowing which tool to reach for. Once you know the command, `man` and `--help` can fill in the details.