Alchemist.Digital

Foreword

For further studying I highly recommend purchasing and accessing the following: 

CompTIA Linux+ Powered by Linux Professional Institute Study Guide: Exam LX0-103 and Exam LX0-104 3rd Edition
by Christine Bresnahan and Richard Blum

Exam Review for CompTIA Linux+ (LX0-103) and LPI LPIC-1 (101-400)
By Andrew Mallett

Command Line

Exam Objectives

• 103.1 – Work on the command line
• 103.2 – Process text streams using filters
• 103.4 – Use streams, pipes, and redirects
• 103.7 – Search text files using regular expressions

Shells

A shell is a program that accepts and interprets text-mode commands and provides an interface to the system.

The most commonly used shell is bash (GNU Bourne Again Shell), and it is most emphasized on the exam.

There are two types of default shells:

1. Default interactive shell – The shell program a user uses to enter commands, run programs/scripts, etc.
2. Default system shell – Used by the Linux system to run system shell scripts (typically at start-up). 

The /bin/sh file is a pointer to the system’s default system shell, which is normally /bin/bash for Linux.

To access a shell you can either log into Linux via text-mode, or load a GUI such as terminal, xterm, konsole, etc. Either option will bring you to the default interactive shell. 

To use a shell you simply type a command, possibly add options and/or a path to it, and then the computer executes the command.

Internal and External Commands

Most commands are external but some commands are internal (built-in) to the shell itself. 

The most common internal commands are:

To determine if a command is internal or not, you can use the type command:
$ type <command>
$ type type
type is a shell builtin

Some commands also have a duplicate external command in addition to the internal command of the same name. To check for duplicates:
$ type -a <command>
$ type -a pwd
pwd is a shell builtin
pwd is /bin/pwd

Internal commands take precedence over external commands when both are present. In order to access an external command you must provide the complete external command path (ex. /usr/bin/time rather than time).

When a command is ran the shell checks for an internal command first. If none is found, it will check for external command within any of the directories listed in the $PATH environment variable.

If a command is not internal or present on the $PATH, it can be run by providing a full path name on the command line.

In order for any file to be ran it must have the executable bit marked on its file permissions (ex. -rwxr--r--).

To adjust a file’s permissions you can use the chmod command:
$ chmod <options> <file>
$ chmod +x script.sh

BASH Navigation

Filenames can be up to 255 characters in length on most filesystems.

The tilde (~) character can be used as a shortcut to represent the current user’s home directory path (ex. ~ translates to /home/jeff when logged in as the user named jeff).

When bash is invoked as an interactive login shell it reads commands from a files /etc/profile. After that, it will attempt to read another file at ~/.bash_profile, ~/.bash_login, and ~/.profile (in that order).

When an interactive shell that is not a login shell is started, bash reads and executes commands from a ~/.bashrc file, if it exists. 

When a login shell exits, bash processes any commands from files at ~/.bash_logout and /etc/bash.bash_logout, if those files exist.

Pressing the tab key will cause the shell to try to fill out the rest of the command or filename. If there are multiple possible matches, it will display them when you press the tab key a second time. This is called command completion, which is supported in bash and tcsh. 

The bash shell provides many hotkeys for navigation, based on the Emacs editor:

History

The shell keeps a log of every command you type, called history. To access this history:
$ history

The following hotkeys can be used for navigating the history:

To retrieve and execute the last command in history:
$ !!

To run the most recent command in history starting with a particular string:
$ !<characters>

To re-use the first argument of the previous command:
$ !^

To re-use the last argument of the previous command:
$ !$

To re-use all arguments from the previous command:
$ !*

To execute a particular item within history:
$ !<number>
$ !210

Note: To display what any of these shortcut expressions translate to, add :p to the end. For example, $ !*:p.

The bash history file is stored at ~/.bash_history  (i.e. within your user’s home directory).

The bash history does not store what you type in response to other prompts made by commands or programs. Because of this, it is advisable to not enter sensitive information (passwords, SSNs, etc.) as an argument to a command, but rather enter it when prompted by the command being run. 

Environment Variables

Environment variables hold data that can be referred to by variable name.

These variables (and their data) are available to all programs, including the shell itself.

All environment variables can be displayed with:
$ env

To view the particular data an environment variable is holding:
$ echo $<env_name>

To assign an environment variable:
$ <env_name>="<env_value>"
$ export <env_name>

Alternatively, you can do it in one line:
$ export <env_name>="<env_value>"

To delete an environment variable:
$ unset <env_name>

Some common environment variables:

Getting Help

Linux provides several commands to lookup information on other commands: 

By default, man uses the less pager to display the manual pages. 

To use another pager with man:
$ man -P <path_to_pager> <command>

Most Linux distributions provide a whatis database, which contains short descriptions of each man page and keywords for searches.

To create or update a whatis database:
$ makewhatis

To search man pages by keyword:
$ man -k "<keyword1 keyword2 ... keywordN>"

Man pages can be broken down into several sections based on number:

Man generally returns the lowest-numbered section, but you can force it to display a particular section number with:
$ man <section_num> <command>

Less Pager Navigation

The less pager provides several hotkeys to navigate with:

Streams, Pipes, and Redirection

In order to move input and output between programs and/or files we use streams, pipes, and redirection.

Linux treats input and output from programs as a stream — which is a data entity that can be manipulated.

Input and output streams can be redirected from or to other sources, including files. Similarly, output from one program can be piped to another program as its input.

File Descriptors 

Linux handles all objects as files — including a program’s input and output stream.

To identify a particular file object, Linux uses file descriptors:

Redirects

Input and output can be redirected by using specific operators following a command and its options:

Pipes

Data pipes (a.k.a. pipelines) can be used to send STDOUT from one program to the STDIN of another program:
$ first_program | second_program

Using a pipe is functionally equivalent to:
$ first_program > temp_file && second_program < temp_file

Tee

The tee command is often used with pipes to split STDOUT so that it’s both displayed on screen and sent to a file:
$ echo $PATH | tee path.txt
/usr/lib64/qt-3.3/bin:/usr/local/bin:[...]

Note: tee will overwrite files unless you pass the -a option to it.

Generating Commands

The xargs command passes its STDIN as the STDIN to whichever command you wish:
$ xargs [options] [command [initial-arguments]]

By default, xargs delimits its input by spaces and newlines. To force a particular character to use for delimiting, use the -d option:
$ find / -user jeff | xargs -d "\n" rm

Enclosing a command within backticks (`) works similarly to xargs:
$ rm `find ./ -user jeff`

Since backticks can sometimes not work very well in complex situations, enclosing commands with a $() is generally preferred:
$ rm $(find ./ -user jeff)

Text Filters

The cat, join, and paste commands all join files end-to-end based on fields within the file, or by merging on a line-by-line basis.

Cat / Tac

The cat command is short for the word concatenate. It combines the contents of files together and sends the data to STDOUT:
$ cat first.txt second.txt > combined.txt

It can also be used to display the contents of a file to the screen when using cat without a redirect:
$ cat hello_world.txt
hello world!

Common options for cat:

tac is the same as cat, but it reverses the order of lines in the output.

Join

The join command combines two files by matching their contents on a delimited field specified by the user:
$ join file1.txt file2.txt
006 Trevelyan, Alec deceased
007 Bond, James active

By default, join uses a space character as the field delimiter. However, the delimiter can be changed with the -t option:
$ join -t “,” file1.txt file2.txt

The join command also defaults to using the first delimited field for matching. This can be changed by using an option corresponding to the file number (ex. -1, -2, -3, …, -99) followed by the field number to use:
$ join -1 3 -2 2 file1.txt file2.txt

To specify the output  format, use the -o option.

Paste

The paste command merges files line-by-line, separating the lines from each file with tabs:
$ paste file1.txt file2.txt
file1_line1    file2_line1
file1_line2    file2_line2

File Transforming Commands

Commands for transforming the contents of files doesn’t actually change their content, but you can redirect their STDOUT to a new file or pipe it to another command.

Sort

By default, the sort command will sort files by the first field using case-sensitive ASCII values.

Common options for sort:

Delete Duplicate Lines

The uniq command removes duplicate lines within a file:
$ sort shakespeare.txt | uniq

Split

The split command can split a file into two or more files, which receive an automatically generated alphabetic suffix. 

By default, split will create output files in 1,000-line chunks, with names ending in aa, ab, ac, etc.

Common options for split:

split works on STDIN if no input file is specified.

Translate Characters

The tr command changes individual characters from STDIN:
$ tr [options] SET1 [SET2]

SET1 is a group of characters to be replaced by SET2 (if supplied).

Because tr uses STDIN, redirection must be used to input from a file:
$ tr d D < file1.txt

tr supports a number of shortcuts to represent various character groups:

Common options for tr:

Convert Tabs and Spaces

The expand command converts tab characters into spaces. Each tab becomes 8 spaces by default, but this can be changed with the -t or --tabs= options.

The unexpand command converts multiple spaces into tab characters. Just like expand, unexpand can use the -t or --tabs= options to specify the number of characters per tab.

Display Files as Byte Data

The od (octal dump) command can be used to display a file in an unambiguous byte data format, such as:
octal (base 8), hexadecimal (base 16), decimal (base 10), and even ASCII with escaped control characters.

By default, od will output as octal when called without options:
$ od file1.txt
0000000 032465 026465 031462 033471 072440 066156 071551 062564
0000020 005144 032465 026465 030465 033061 066040 071551 062564
0000040 005144 032465 026465 034467 034462 066040 071551 062564
[…]

The first field on each line is an index of the file in octal. The second line begins at octal 20 (16 in decimal) bytes into the file. The remaining numbers represent the bytes in the file.

File Formatting

The fmt, nl, and pr commands are used to reformat text within a file.

Fmt

fmt can be used to clean up files with long line lengths, irregular line lengths, or other problems.

By default, fmt will attempt to clean up paragraphs assumed to be delimited by two or more blank lines or by changes in indentation. The new paragraphs will be formatted to no more than 75 characters wide.

The width of formatted paragraphs can be changed with the -<width>, -w <width>, and --width=<width> options.

NL

nl adds number lines to a file (similar to cat -b).

By default, nl starts each new page on line 1. The -p or --no-renumber option will prevent the line number reset on each page.

-n <format> or --number-format=<format> can be used to change the numbering format:

The header, body, and footer of a page can be styled with -h <style>, -b <style>, and -f <style>. 
Alternatively --header-numbering=<style>, --body-numbering=<style>, and --footer-numbering=<style>:

An example of using nl to add line numbers to a script:
$ nl -b a script.js > numbered-script.txt

The page delimiter can be set with -d=<code> or --section-delimiter=<code>. 
<code> represents the character that identifies the new page.

PR

pr can be used to prepare a plain-text file for printing by adding headers, footers, page breaks, etc.

The pr command itself does not handle printing, but its output is often piped to the lpr command to actually handle printing.

By default, pr assumes an 80-character line length with a monospaced font; and outputs the original text with headers that include the current date, time, filename, and page number. 

Common options for pr:

An example of using pr to print a double-spaced and numbered version of a configuration file at /etc/profile:
$ cat -n /etc/profile | pr -d | lpr

Note: The pr command was built around assumptions for printer capabilities back in the 80s. The command can still be useful today, but GNU Enscript (http://www.codento.com/people/mtr/genscript/) is preferred. This program has many of the same features as pr, but it generates PostScript output that can take better advantage of modern printers and their features.

File Viewing

Head

head echos the first 10 lines of one or more files to STDOUT.

If more than one file is passed to head, it will precede each file’s output with a header to identify it.

Common options for head:

Tail

tail echos the last 10 lines of one or more files to STDOUT.

Common options for tail:

Note: head and tail can be combined to display or extract portions of a file. For example, if you want to display lines 11 – 15 of a file named sample.txt, you could extract the first 15 lines with head and display the last five of those with tail:
$ head -n 15 sample.txt | tail -n 5

Less / More

less is the successor to more (the joke being: “less is more“).

less enables you to read through a file one screen at a time:
$ less <filename>

To navigate the less pager:

Note: less cannot be used in a pipe, except as the final command.

Note 2: more cannot page backwards through a file.

File Summarizing

Cut

cut extractions portions of input lines and displays them on STDOUT.

Common options for cut:

<list> can be a single number (ex. 4), a closed range of numbers (ex. 2-4), or an open range of numbers (ex. -4 or 4-). 

An example of using cut to extract the hardware address of a wireless adapter:
$ ifconfig wlp2s0 | grep ether | cut -d " " -f 10
e4:a4:71:66:24:ad

WC

wc produces a word count, line count, or byte count for a file:
$ wc file.txt
308    2343    15534  file.txt

The above output shows that file.txt contains 308 lines (308 newline characters), 2343 words, and 15534 bytes.

Common options for wc:

Note: Character and byte counts are typically identical on most ASCII files, unless multibyte character encoding is used.

Regular Expressions

Regular Expressions (RegEx) are tools for describing or matching patterns in text:

There are two forms of regular expressions: basic and extended.

More information on RegEx can be found with:
$ man 7 regex

Grep

grep searches for files that contain a specific string, returning the name of the file and a line of context for the match (if the file is a text file):
$ grep [options] <regexp> [files]

Common options for grep:

An example of using grep with a regular expression:
$ grep -r eth0 /etc/*

The above command will recursively search for the pattern eth0 within all files of all directories inside of /etc, and any file found to match that string will be printed to STDOUT.

To find all files that contain a match for eth0 and eth1, the above command can be rewritten as:
$ grep -r eth[01] /etc/*

An example of using extended regular expressions with grep:
$ grep -E "(alchemist\.digital|mr2\.run).*169" /etc/*

The above command will search all files within the /etc directory for anything containing alchemist.digital or mr2.run with the number 169 on the same line.

Note: Sometimes shell quoting is necessary to ensure grep has a chance to use certain characters in a regular expression before the shell attempts to parse them for its own purpose (ex. |, *, etc.)

Sed

sed directly modifies a file’s contents and sends the changes to STDOUT.

There are two forms of syntax for sed:
$ sed [options] -f <script-file> [input-file]
$ sed [options] '<script-text>' [input-file]

[input-file] is the name of the file you want to modify. 

The contents of <script-file> or <script-text> are a set of commands that sed will perform. 

By default, commands operate on the entire file. However, an address (line number) can be provided to ensure the command only operates on the desired lines.

Addresses can be provided as either a single number or a range of two numbers separated by a comma.

Note: Addresses can also be provided as a regex pattern when using the -e option with sed.

Common commands used with sed:

An example of using sed for text replacement:
$ sed 's/2012/2013/' cal-2012-txt > cal-2013.txt

Note: If no input file is specified, sed can use STDIN.

Convert Between Unix and DOS Line Endings

Unix (and Linux) use a single line feed character (ASCII 0x0a, sometimes represented as \n) as the end of a line in a file.

DOS and Windows use the combination of a carriage return (ASCII 0x0d or \r) and a line feed (i.e. \r\n).

To convert files from one system to another there is special-purpose programs such as dos2unix and unix2dos:
$ dos2unix file.txt 

Alternatives

Since not all distributions have these utilities, an alternative to convert from DOS to Unix would be:
$ tr -d [[backslash backslash]]r < dosfile.txt > unixfile.txt

The above tr command is run with the -d (delete) option to remove all carriage returns (\r) using the escaped literal backslash in the set. The tr command’s input is the dosfile.txt file, and the output is redirected to the unixfile.txt file.

An alternative to convert from Unix to DOS style would be:
$ sed s/$/"\r"/ unixfile.txt > dosfile.txt

The above command uses the regexp of $ for the end of line, and replaces it with a \r (carriage return) in its place. The input for the command is the unixfile.txt and the redirected output goes into the dosfile.txt.

Note: Another option is to have a text editor save the file using a different file-type setting, but not all editors support this feature/option.

Command Line Essentials

Things to know without lookup:

• Summarize features that Linux shells offer to speed up command entry:
  • Command history enables you to get an earlier command that is identical or similar to the one you want to enter.
  • Tab completion reduces typing effort by letting the shell finish long command and file names.
  • Command-line editing lets you edit a retrieved command or change a typo before submitting the command.
• Describe the purpose of the man command:
  • man displays the manual page for a keyword (command, filename, system call, etc.) that you type.
  • The information provided is a succinct summary that’s useful for reference or to learn about exact command options and/or features.
• Explain the purpose of environment variables:
  • Environment variables store small pieces of data — program options, information about the machine, etc. 
  • This information can be read by programs and used to modify program behavior in a way that’s appropriate for the current environment.
• Describe the difference between standard output and standard error:
  • STDOUT carries normal program output, whereas STDERR carries high-priority output, such as error messages. 
  • The two can be redirected independently of one another.
• Explain the purpose of pipes:
  • Pipes tie programs together by feeding the STDOUT from the first program into the STDIN of the next program.
  • They can be used to link together a series a simple programs in order to perform more complex tasks than they could do individually.
• Describe the filter commands:
  • Simple filter commands allow the manipulation of text.
  • These commands accomplish tasks such as: combining files, transforming the data in files, formatting text, displaying text, and summarizing data.
• Summarize the structure of regular expressions:
  • Regular expressions are strings that describe other strings.
  • They can contain normal alphanumeric characters, which match the exact same characters in the string they are describing. They can also contain several special symbols and symbol sets that match multiple different characters. 
  • Overall, regular expressions are a powerful pattern-matching tool used by many Linux programs.

Command Line Study and Review

• How to check environment variables (hint: env, echo $<name>), how to set environment variables (hint: export <name>=”<value>”, or <name>=”<value>” -> export <name>), how to delete environment variables (hint: unset <name>), and the common environment variables (hint: $TERM, etc).
• How to change shells and set the default shell.
• The difference between the default interactive shell and the default system shell (hint: /bin/sh points to the system shell).
• The difference between internal and external commands, which one takes precedence if both exist (hint: internal), how to determine if a command is internal / built-in (hint: builtin command) or external (hint: type command), and how to tell if both internal and external commands exist (hint: type -a) . 
• What xterm, terminal, and konsole are and how they differ.
• What the system’s path is (hint: $PATH), how it stores its values (hint: colon-delimited list of directories), how to check its values, how to set its values, and how to run commands not on the path (hint: ./<program_name> or /<full_path>/<program_name>).
• Which permission / bit needs to be set in order to run a program / script (hint: executable bit), and how to change permissions (hint: chmod).
• The length filenames can be (hint: 255 characters on most systems).
• The shell shortcuts like command completion (tab complete) and history.
• How to navigate history:
  • Ctrl+R to search backwards and Ctrl+S to search forward and cycle through matches.
  • Ctrl+Q to quit a search (and stty -ixon to prevent hanging).
  • Ctrl+G to terminate a search.
  • !! to show and execute the last command used.
  • !<number> to execute that command from history.
  • history -c option to clear history.
• Where the history is stored (hint: ~/.bash_history), and how to prevent passwords and sensitive data from being saved in history (hint: only enter it as a response to a prompt, not as part of the command itself). 
• How to navigate bash:
  • Ctrl+A and Ctrl+E for start and end of line.
  • Ctrl+D or Del to delete characters under the cursor.
  • Backspace to delete characters to the left of the cursor.
  • Ctrl+K to delete all text from the cursor to end of line.
  • Ctrl+X -> Backspace to delete all text from the cursor to start of line.
  • Ctrl+T to transpose (swap) the character under the cursor with the one before it.
  • Esc -> T to transpose the two words before / under the cursor.
  • Esc -> U for UPPERCASE, Esc -> L for lowercase, and Esc -> C for Capitalize Case.
  • Ctrl+X -> Ctrl+E to open a full editor (defined in $FCEDIT or $EDITOR, defaults to Emacs).
  • ‘set -o vi’ to set a vi-like mode in bash.
• Where the bash configuration files are (hint: ~/.bashrc and /etc/profile) and how to set them.
• How to find help documents and information on commands and options (hint: man, info, and help), how to search based on keyword (hint: man -k “<keywords>”), how to establish a keyword database (hint: the whatis database and the makewhatis command), and how to change the pager that displays the information (hint: man -P <path_to_pager> <command>).
• The sections of manual pages:
  • 1 – Executable programs and shell commands.
  • 2 – System calls provided by the kernel.
  • 3 – Library calls provided by program libraries.
  • 4 – Device files (usually stored in /dev).
  • 5 – File formats.
  • 6 – Games.
  • 7 – Miscellaneous (macro packages, conventions, etc.).
  • 8 – System administration commands (programs run mostly or exclusively by root).
  • 9 – Kernel routines.
• How to navigate the less pager:
  • Spacebar to move forward a page.
  • Esc -> V to move back a page.
  • Arrow keys to move up and down one line.
  • / to search for text.
  • Q to quit.
• How Linux treats all objects (hint: as files), what file descriptors are available (hint: 0, 1, 2), how to redirect input (STDIN) / output (STDOUT) / error  (STDERR) streams (hint: >, >>, 2>, 2>>, &>, <, <<, <>), how to use a here document / heredoc (hint: command << eof_marker), and how to discard STDERR messages (hint: 2> /dev/null). 
• How to use pipes to redirect and chain pipelines from multiple programs together (hint: prog1 | prog2).
• How to send STDOUT to both the screen and to another file or program (hint: tee).
• How to build new commands programmatically (hint: xargs), and what other options are available (hint: `backticks`  and $() enclosures ).
• How to locate files (hint: find).
• How to find particular patterns of text within files (hint: grep).
• How to combine files (hint: cat), join files on a common field (hint: join), merge lines together (hint: paste), sort the content of files (hint: sort), delete duplicate lines (hint: uniq), and split files into multiple files (hint: split).
• How to convert between tabs and spaces (hint: expand, unexpand), convert a file to octal / hexadecimal / escaped ASCII (hint: od), and change characters in a file (hint: tr).
• How to format a file (hint: fmt), add line numbers (hint: nl, cat -b), how to prepare for printing (hint: pr), and how to print (hint: lpr).
• How to quickly view files without opening them in an editor (hint: head, tail).
• How to extract lines out of input (hint: cut).
• How to count the number of lines or size of a file (hint: wc).
• How to use Regular Expressions / regexp / regex:
  • Bracket expression [abc]
  • Range expression [0-9]
  • Any single character (.)
  • Start (^) and end ($) of line 
  • 0+ times (*), 1+ times (+), 0/1 time (?) 
  • Multiple possible matches ( this|that ).
  • Subexpressions .*(\.digital)?
  • Escaping special characters ( \|, etc. ). 
• How to replace the contents of a file directly from the command line (hint: sed).
• Commands to know:
  • uname
  • cd
  • rm
  • pwd
  • echo
  • time
  • set
  • exit
  • logout
  • type
  • builtin
  • chmod
  • env
  • export
  • unset
  • man
  • info
  • less
  • more
  • find
  • tee
  • xargs
  • cat / tac
  • join
  • paste
  • sort
  • tr
  • expand / unexpand
  • grep 
  • sed
  • head
  • tail
  • cut
  • wc
  • man 7 regex
  • ps