Shell scripting
Rough Notes
file1 > file2 > file3 > /dev/null
: Send content offile1,file2
tofile3
.- backslashes escape special characters.
tar czf
: To zip a directory,czf
means compress, zip and file.du
: Check disk usage.- Wildcards: Things like
*.pdf
will expand topdf1.pdf pdf2.pdf
and so on. wc
: prints words, characters and lines for list of files, e.g.wc file1.txt file2.txt
.- To push output from shell to a file, put
> outputfilename
at the end. cat
: prints contents of files one after the other.sort
: output sorted contents of file.head
:head -k filename
outputs first k lines in given file.- To use output of one command as input of another command, use the pipe like
command 1 | command 2
. - UNIX philosophy is to create simple tools that do one job well and work well with each other.
- Pipes and filters model of shell programming.
-rwxrwxrwx
here - means its a file, if its d its a directory. The first 3 rwx refer to the permissions the owner has, next 3 is the permissions the group owner permissions, last 3 is for everyone else. rwx stands for read, write, execute. For directories, x means being able to traverse the directory but not look at its direct contents, i.e. they can see contents of subdirectories inside that directory but not what is inside the directory.- Use ; to use multiple commands in the same line.
cat > filename
gives user prompt, the input of which goes into the file (use ^D i.e. Control-D to finish), useful for creating quick shell scripts.grep
stands for global/regular expression/print. It finds and prints lines in files that match the given regexp pattern. Some flags include -w for only whole word matches, -n to print line number as well.find
can be used to find files instead of lines.- find and grep are commonly used together.
ps
lists running processes.- To run a process in the background, put & at the end. To bring it to the foreground use
fg
. set
shows variables in the shell. The value of shell variables are strings. It is up to programs to convert them to other types like int when necessary. The PATH variable's value defines the directories that the shell looks for runnable programs.echo
prints its arguments, using with $ to print out values of variables, e.g. to print the value of the variablePATH
, useecho $PATH
.- Assigning a value to a variable only changes the variable's value in the shell it is run on. To solve this, one can use the
export
command, often placed in the.bashrc
file which is executed when the shell starts. It is also common to usealias
inside this file. ssh
,scp
. Can run commands directly viassh user@remote 'command'
. Here first we need to separate the two communication channels: server to user and user to server, then public key encryption is used in each channel.- The
2>
can be used to redirected stderror, since>
will not redirect stderror to files. To redirect stdout and stderror, one can do for e.g.command > stdoutputfile 2> stderrorfile
. To redirect both to the same file, use&>
. Also,1>>, 2>>
(1> same as >) will append (and create if a file does not exist). - Pipes and filters work by using a single program on some input, however we might want to run the same program separately for each input. Loops can help here. For e.g. to zip all files with a certain filetype in the directory, the following loop will do:
for file in *.pdb; do zip $file.zip $file; done
. If such a command returns some output, we can pipe extra commands on them. Merge PDFs via
gs -q -sPAPERSIZE=letter -dNOPAUSE -dBATCH -sDEVICE=pdfwrite -sOutputFile=output.pdf file1.pdf file2.pdf
du -h -x -d 1
to list size of each directories in the current directory.du --inodes -h -x -d 1
to list number of files within each directory in the current directory (add| sort -nr
to sort by descending order - note that this is not robust since it would put 900 as higher than 900K for example).ps -U $USER --no-headers -o rss | awk '{ sum+=$1} END {print int(sum/1024) "MB"}'
to see memory consumption of current user - helpful when in clusters.Compressing PDFs:
gs -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dPDFSETTINGS=/ebook -dNOPAUSE -dQUIET -dBATCH -sOutputFile=output.pdf input.pdf
-dPDFSETTINGS Option | Description |
---|---|
-dPDFSETTINGS=/screen | Has a lower quality and smaller size. (72 dpi) |
-dPDFSETTINGS=/ebook | Has a better quality, but has a slightly larger size (150 dpi) |
-dPDFSETTINGS=/printer | Output is of a printer type quality (300 dpi) |
-dPDFSETTINGS=/default | Selects the output which is useful for multiple purposes. Can cause large PDFS. |
Multiple patterns can be passed to
find
to narrow down results. E.g. to search for PDF files with 2023 in the filename:find . -name "*.pdf" -not -name "*2023*"