• Learn what kinds of problems are suited to shell scripts
• Review the most commonly used Unix commands that are useful in shell scripts.
• Write simple shell scripts using the Bourne, Korn or Bash shells

• log in and get a command line window (any shell)
• run basic commands, navigate directories
• use simple I/O redirection and pipes
• use a text editor (any one)
• look up details of command usage in man pages

• A text file containing commands which could have been typed directly into the shell.

  There is no difference in syntax between interactive command line use and placing the commands in a file. Some commands are only useful when used interactively (e.g. command line history recall) and other commands are too complex to use interactively.

   

• The shell itself has limited capabilities -- the power comes from using it as a "glue" language to combine the standard Unix utilities, and custom software, to produce a tool more useful than the component parts alone.

   

• Any shell can be used for writing a shell script. To allow for this, the first line of every script is:
  #!/path/to/shell (e.g. #!/bin/ksh).

  The #! characters tell the system to locate the following pathname, start it up and feed it the rest of the file as input. Any program which can read commands from a file can be started up this way, as long as it recognizes the # comment convention. The program is started, and then the script file is given to it as an argument. Because of this, the script must be readable as well as executable. Examples are perl, awk, tcl and python.

   

• Any file can be used as input to a shell by using the syntax:
  ksh myscript

   

• If the file is made executable using chmod, it becomes a new command and available for use (subject to the usual $PATH search).
  chmod +x myscript

   1: #!/bin/sh
   2: date
   3: pwd
   4: du -k

• Combine lengthy and repetitive sequences of commands into a single, simple command.

   

• Generalize a sequence of operations on one set of data, into a procedure that can be applied to any similar set of data.

  (e.g. apply the same analysis to every data file on a CD, without needing to repeat the commands)

   

• Create new commands using combinations of utilities in ways the original authors never thought of.

   

• Simple shell scripts might be written as shell aliases, but the script can be made available to all users and all processes. Shell aliases apply only to the current shell.

   

• Wrap programs over which you have no control inside an environment that you can control.

  e.g. set environment variables, switch to a special directory, create or select a configuration file, redirect output, log usage, and then run the program.

   

• Create customized datasets on the fly, and call applications (e.g. matlab, sas, idl, gnuplot) to work on them, or create customized application commands/procedures.

   

• Rapid prototyping (but avoid letting prototypes become production)

• System boot scripts (/etc/init.d)

   

• System administrators, for automating many aspects of computer maintenance, user account creation etc.

   

• Application package installation tools

  Other tools may create fancier installers (e.g. tcl/tk), but can not be assumed to be installed already. Shell scripts are used because they are very portable. Some software comes with a complete installation of the tool it wants to use (tcl/tk/python) in order to be self contained, but this leads to software bloat.

   

• Application startup scripts, especially unattended applications (e.g. started from cron or at)

   

• Any user needing to automate the process of setting up and running commercial applications, or their own code.

sh

aka "Bourne" shell, written by Steve Bourne at AT&T Bell Labs for Unix V7 (1979). Small, simple, and (originally) very few internal commands, so it called external programs for even the simplest of tasks. It is always available on everything that looks vaguely like Unix.

 

csh

The "C" shell. (Bill Joy, at Berkeley). Many things in common with the Bourne shell, but many enhancements to improve interactive use. The internal commands used only in scripts are very different from "sh", and similar (by design) to the "C" language syntax.

 

tcsh

The "TC" shell. Freely available and based on "csh". It has many additional features to make interactive use more convenient.

We use it as the default interactive shell for new accounts on all of our public systems.
Not many people write scripts in [t]csh. See Csh Programming Considered Harmful by Tom Christiansen for a discussion of problems with programming csh scripts.

 

ksh

The "Korn" shell, written by David Korn of AT&T Bell Labs (now AT&T Research). Written as a major upgrade to "sh" and backwards compatible with it, but has many internal commands for the most frequently used functions. It also incorporates many of the features from tcsh which enhance interactive use (command line history recall etc.).

It was slow to gain acceptance because earlier versions were encumbered by AT&T licensing. This shell is now freely available on all systems, but sometimes not installed by default on "free" Unix. There are two major versions. ksh88 was the version incorporated into AT&T SVR4 Unix, and may still be installed by some of the commercial Unix vendors. ksh93 added more features, primarily for programming, and better POSIX compliance.

 

POSIX 1003.2 Shell Standard.

Standards committees worked over the Bourne shell and added many features of the Korn shell (ksh88) and C shell to define a standard set of features which all compliant shells must have.

On most systems, /bin/sh is now a POSIX compliant shell. Korn shell and Bash are POSIX compliant, but have many features which go beyond the standard. On Solaris, the POSIX/XPG4 commands which differ slightly in behaviour from traditional SunOS commands are located in /usr/xpg4/bin

 

bash

The "Bourne again" shell. Written as part of the GNU/Linux Open Source effort, and the default shell for Linux and Mac OS-X. It is a functional clone of sh, with additional features to enhance interactive use, add POSIX compliance, and partial ksh compatability.

 

zsh

A freeware functional clone of sh, with parts of ksh, bash and full POSIX compliance, and many new interactive command-line editing features. It was installed as the default shell on early MacOSX systems.

• Parse lines by whitespace, search for external commands using $PATH.
• Can run a shell script using shellname scriptfile, or run a single command using shellname -c "command"
• Pass expanded command line arguments to programs; get exit status back.
• Pass environment variables to programs.
• Expand filename wildcards using []*?. Each shell has some additional wildcard metacharacters, but these are common to all shells.
• Standard I/O redirection and piping with <,>,>>,|
• A few internal functions (cd)
• Backgrounding commands with &
• Quoting rules: "double quotes" protect most things, but allow $var interpretation; 'single quotes' protect all metacharacters from interpretation.
• Home directory expansion using ~user (except for sh)

   

• # comments
• Command substitution using `command` (backtics)
• Expand variables using $varname syntax
• Conditional execution using && and ||
• Line continuation with "\"

• Syntax of all the flow control constructs and conditional tests.
• Syntax for string manipulation inside of scripts
• Syntax for arithmetic manipulation inside of scripts
• Syntax for setting local variables (used only in the script) and environment variables (which are passed to child processes). setenv vs export
• Syntax for redirecting I/O streams other than stdin/stdout
• Login startup files (.cshrc and .login, vs .profile) and default options
• Reading other shell scripts into the current shell (source filename, vs . filename)
• Handling of signals (interrupts)

• what is installed already - many other scripting languages are not available by default
• what similar code already exists
• what you are most familiar with and can use most efficiently. Your time is always more expensive than computer cycles.

• awk

  A pattern matching and data (text and numeric) manipulation tool. Predates perl. Installed on all Unix systems. Often used in combination with shell scripts.

• perl

  The most used scripting language for Web CGI applications and system administration tasks. Perl is harder to learn, and is uusually installed by default now. It is more efficient and has an enormous library of functions available. You could use Perl for almost all scripting tasks, but the syntax is very different to the shell command line

• python

  An object-oriented scripting language. Commonly installed by default on modern systems.

• tcl/tk

  Tool Command Language. Another general purpose scripting language. The "tk" component is a scripted interface to standard X-windows graphical components, so the combination is often used to create graphical user interfaces.

  Ksh93 can be extended by linking to shared libraries providing additional internal commands. One example of an extended shell is tksh which incorporates Tcl/Tk with ksh and allows generation of scripts using both languages. It can be used for prototyping GUI applications.

The philosophy of separate Unix tools each performing a single operation was followed closely by the designers of the original shell, so it had very few internal commands and used external tools for very trivial operations (like echo and [). Ksh and bash internally performs many of the basic string and numeric manipulations and conditional tests. Occasional problems arise because the internal versions of some commands like echo are not fully compatible with the external utility they replaced.

The action taken every time a shell needs to run an external program is to locate the program (via $PATH), fork(), which creates a second copy of the shell, adjust the standard input/output for the external program, and exec(), which replaces the second shell with the external program. This process is computationally expensive (relatively), so when the script does something trivial many times over in a loop, it saves a lot of time if the function is handled internally.

# as the first non-whitespace character on a line

flags the line as a comment, and the rest of the line is completely ignored. Use comments liberally in your scripts, as in all other forms of programming.

 

\ as the last character on a line

causes the following line to be logically joined before interpretation. This allows single very long commands to be entered in the script in a more readable fashion. You can continue the line as many times as needed.

This is actually just a particular instance of \ being to escape, or remove the special meaning from, the following character.

 

; as a separator between words on a line

is interpreted as a newline. It allows you to put multiple commands on a single line. There are few occasions when you must do this, but often it is used to improve the layout of compound commands.

   1: #!/bin/ksh
   2: # For the purposes of display, parts of the script have 
   3: # been rendered in glorious technicolor.
   4: ## Some comments are bold to flag special sections
   5: 
   6: # Line numbers on the left are not part of the script.
   7: # They are just added to the HTML for reference.
   8: 
   9: # Built-in commands and keywords (e.g. print) are in blue
  10: # Command substitutions are purple. Variables are black
  11: print "Disk usage summary for $USER on `date`"
  12: 
  13: # Everything else is red - mostly that is external 
  14: # commands, and the arguments to all of the commands.
  15: print These are my files       # end of line comment for print
  16: # List the files in columns
  17: ls -C
  18: # Summarize the disk usage
  19: print
  20: print Disk space usage
  21: du -k 
  22: exit 0

The exit status is an integer 0-255. Conventionally 0=success and any other value indicates a problem. Think of it as only one way for everything to work, but many possible ways to fail. If the command was terminated by a signal, the value is 128 plus the signal value.

Wildcards may be used in the directory parts of a pathname as well as the filename part. If no files match the wildcard, it is left unchanged. Wildcards are not full regular expressions. Sed, grep, awk etc. work with more flexible (and more complex) string matching operators.

*

Match zero or more characters.

?

Match any single character

[...]

Match any single character from the bracketed set. A range of characters can be specified with [ - ]

[!...]

Match any single character NOT in the bracketed set.

• An initial "." in a filename does not match a wildcard unless explicitly given in the pattern. In this sense filenames starting with "." are hidden. A "." elsewhere in the filename is not special.

   

• Pattern operators can be combined

   

srcfile=dataset1

Creates (if it didn't exist) a variable named "srcfile" and sets it to the value "dataset1". If the variable already existed, it is overwritten. Variables are treated as text strings, unless the context implies a numeric interpretation. You can make a variable always be treated as a number. Note there must be no spaces around the "=".

set

Display all the variables currently set in the shell

unset srcfile

Remove the variable "srcfile"

srcfile=

Give the variable a null value, (not the same as removing it).

export srcfile

Added srcfile to the list of variables which will be made available to external program through the environment. If you don't do this, the variable is local to this shell instance.

export

List all the variables currently being exported - this is the environment which will be passed to external programs.

$srcfile

Prefacing the variable name with $ causes the value of the variable to be substituted in place of the name.

${srcfile}

If the variable is not surrounded by whitespace (or other characters that can't be in a name), the name must be surrounded by "{}" braces so that the shell knows what characters you intend to be part of the name.

Example:

datafile=census2000
# Tries to find $datafile_part1, which doesn't exist
echo $datafile_part1.sas 
# This is what we intended
echo ${datafile}_part1.sas  

${datafile-default}

Substitute the value of $datafile, if it has been defined, otherwise use the string "default". This is an easy way to allow for optional variables, and have sensible defaults if they haven't been set. If datafile was undefined, it remains so.

${datafile=default}

Similar to the above, except if datafile has not been defined, set it to the string "default".

${datafile+default}

If variable datafile has been defined, use the string "default", otherwise use null. In this case the actual value $datafile is not used.

${datafile?"error message"}

Substitute the value of $datafile, if it has been defined, otherwise display datafile: error message. This is used for diagnostics when a variable should have been set and there is no sensible default value to use.

Placing a colon (:) before the operator character in these constructs has the effect of counting a null value the same as an undefined variable. Variables may be given a null value by setting them to an empty string, e.g. datafile= .
Example: echo ${datafile:-mydata.dat}
Echo the value of variable datafile if it has been set and is non-null, otherwise echo "mydata.dat".

$USER, $LOGNAME

Preset to the currently logged-in username.

$PATH

The list of directories that will be searched for external commands. You can change this in a script to make sure you get the programs you intend, and don't accidentally get other versions which might have been installed.

$TERM

The terminal type in which the shell session is currently executing. Usually "xterm" or "vt100". Many programs need to know this to figure out what special character sequences to send to achieve special effects.

$PAGER

If set, this contains the name of the program which the user prefers to use for text file viewing. Usually set to "more" or "less" or something similar. Many programs which need to present multipage information to the user will respect this setting (e.g. man). This isn't actually used by the shell itself, but shell scripts should honour it if they need to page output to the user.

$EDITOR

If set, this contains the name of the program which the user prefers to use for text file editing. A program which needs to have the user manually edit a file might choose to start up this program instead of some built-in default (e.g. "crontab -e". This also determines the default command-line-editing behaviour in interactive shells.

$PWD

Always set the current working directory (readonly)

$OLDPWD

The previous directory (before the most recent cd command). However, changing directories in a script is often dangerous.

$? (readonly)

Set to the exit status of the last command run, so you can test success or failure. Every command resets this so it must be saved immediately if you want to use it later.

$-

Set to the currently set options flags.

$IFS

Internal Field Separators: the set of characters (normally space and tab) which are used to parse a command line into separate arguments. This may be set by the user for special purposes, but things get very confusing if it isn't changed back.

$$ (readonly)

Set to the process ID of the current shell - useful in making unique temporary files, e.g. /tmp/$0.$$

$PPID (readonly)

Set to the process ID of the parent process of this shell - useful for discovering how the script was called.

$! (readonly)

Set to the process ID of the last command started in background - useful for checking on background processes.

$SECONDS (readonly)

Integer number of seconds since this shell was started. Can be used for timing commands.

$RANDOM

Every time it is valuated, $RANDOM returns a random integer in the range 0-32k. RANDOM may be set to "seed" the random number generator.

$LINENO (readonly)

Always evaluates to the current line number of the script being executed - useful for debugging.

These are often filenames, but can be interpreted by the script in any way. Options are often specified using the "-flag" convention used by most Unix programs, and a ksh command getopts is available to help parse them.

$0, $1, $2, ... $9

The first 9 arguments are made available directly as $1-$9. To access more than 9, use shift, or $*, $@. The variable $0 contains the name of the script itself.

${10}, ${11}, ...

Positional arguments greater than 9 are set by ksh and bash. Remember to use braces to refer to them.

shift

discard $1 and renumber all the other variables. "shift N" will shift N arguments at once.

$#

contains the number of arguments that were set (not including $0).

$*

contains all of the arguments in a single string, with one space separating them.

$@

similar to $*, but if used in quotes, it effectively quotes each argument and keeps them separate. If any argument contains whitespace, the distinction is important.

   1: #!/bin/sh
   2: #
   3: # Check positional argument handling
   4: echo "Number of arguments: $#"
   5: echo "\$0 = $0"
   6: 
   7: echo "Loop over \$*"
   8: for a in $*; do
   9:    echo \"$a\"
  10: done
  11: 
  12: echo "Loop over \"\$@\""
  13: for a in "$@"; do
  14:    echo \"$a\"
  15: done

Пред. - След. »