Text Processing with Awk

Awk is a powerful unix utility.
$ awk -F":" '{ print "username: " $1 "\t\tuid:" $3" }' /etc/passwd
-F=Field Seperator
$1=first field seperated by ":"

Awk can use a program which is a file with awk instructions.
$ awk -f program.awk inputfile

A sample awk program file program.awk is as follows:
1- BEGIN {
2- FS=":"
3- x=0
4- }
5- { print $1 }
6- $1 ~ /regex/ { x++}
7- END {
8- print x
9- }

BEGIN is the place for commands to be executed before inputfile
is being processed. Here we set FS(field seperator) to : and
initiate variable x to 0.
Line 5 prints all matching 1st columns.
In Line 6 if $1 is similar to /regex/ then variable x is raised by 1.
At the END after processing input file variable x is printed to output.
FS,RS(Record Seperator),OFS(Output Field Seperator),ORS and
NF(Number of Fields) are other special variables in awk.
Reference: http://www.ibm.com/developerworks/linux/library/l-awk3.html

Curl and Wget For Proxying

Curl has many proxy options.Simply
curl -x proxyhost:port -U proxyuser(if any) www.example.com
It has also socks proxy support which wget doesnt
curl --socks5 127.0.0.1:9999 www.example.com

When it comes to wget
http_proxy = http://proxyhost:port/
use_proxy = on
wait = 15
suchsettings on .wgetrc makes it work with the proxy.

Regex help

Regex 11 metachars: [,?,*,$,^,\,.,+,(,),|
Character classes : [A-Z] matches a single character in range A-Z.
[^A-Z] matches a character that is not in between A and Z.
Metachars in a chearcter class are : [,\,^.- and need to be escaped by \ when try to match themselves.