by Dave Taylor
So far you've spent a lot of your time learning about complex and sophisticated languages for working with the Internet. All of them are powerful approaches to developing Common Gateway Interface (CGI) solutions for the World Wide Web. If you've been around the UNIX system for a while, though, you already know the majority of what it takes to make a powerful programming environment from your day-to-day command-line usage: the UNIX shell. This chapter explores the capabilities of the shell and looks at how it can be the ideal solution for a variety of simple to sophisticated Web back-end processes.
First, you can admit it: The Web, and all that it has spawned, is ugly and pretty much a huge hack. It's somewhat miraculous that it all works as well as it does and that it appears so seamless, when underneath, it is weird and remarkably inconsistent. Not just the markup language itself, mind you, but the actual protocol and even the way that the server identifies the type of information being sent to the client is the result of this combination of software and luck.
It all works similar to the depiction in Figure 24.1.
Figure 24.1 : The basic client/server communication.
The simplest case is when the client program (the Web browser) asks for a specific file using the file's address, or Uniform Resource Locator (URL). The server looks for the file, and if it finds it, sends back the contents of the specified file. If not, an error occurs and you get an ugly one-liner such as 404 File not found.
However, instead of the server sending back the contents of a file, the request can instead be "run this program on the server and send the output." Better yet, what if the program itself could output information in HTML so that the results are attractive and consistent? That's what interactivity is all about on the Web.
To have a program, rather than a file, be the target of a URL, you need to have that program live on the server and output HTML code. At its simplest, a shell script like the following does the trick:
#!/bin/sh -f echo "Content-type: text/html" echo "" echo "<HTML><BODY>" echo "<H1>Hello World</H1>" echo "</BODY></HTML>" exit 0
| NOTE |
Web servers typically execute programs only in certain directories on the server, most commonly cgi-bin (Common Gateway Interface Binaries). Drop your programs into that directory on your server if that's how your server is configured, or you have another possibility. All scripts must have the .cgi suffix, even if they're shell scripts, but they can live anywhere in the file system. Check with the person who built the server-or, for your own system, look in the config files-to find out how the system is installed |
After you've figured out the necessary naming scheme for your server, you're ready to roll with simple programs; change the ownership of the script using the UNIX chmod command to ensure that the script is executable, and figure out the full URL including your host name (such as http://www.intuitive.com/script.cgi). Any output that your program emits is sent to the browser (the client program) on the other end of the wire.
You can write a simple output-only CGI script that shows what time it is and who is logged on to the server itself by using the UNIX date and who commands, all wrapped up with standard HTML commands (see Listing 24.1).
Listing 24.1. Providing the time and a list of users to the client.
#!/bin/sh -f echo "Content-type: text/html" echo "" echo -n "<h1>Welcome to " hostname echo -n " It's " date echo "</h1>" echo "Here's Who is Logged In Right Now:" echo "<BLOCKQUOTE><PRE>" who echo "</PRE></BLOCKQUOTE>" echo "</BODY></HTML>" exit 0
The output looks like the following:
Welcome to www.intuitive.com It's Fri Jul 26 14:06:41 CUT 1996
Here's Who is Logged In Right Now:
sol pts/2 Jul 06 00:48 (server.iadfw.net)
taylor pts/5 Jul 26 13:56 (ws3.hostname.com)
This script isn't too gorgeous, but it's quite utilitarian and
serves as a basis for some interesting status scripts and such.
| NOTE |
In addition to the standard HTML formatting instructions output from your script, the first two lines of output are critical to the Web browser's understanding of what to do with the information in the output: the two lines must be "Content-type: text/html" followed by a blank line. If you omit this, you'll see "document has no data" error messages when you try to access the scripts from a browser |
The request for a URL from the Web browser comes wrapped in a package that contains various snippets of useful information. When that information gets to the CGI program, it becomes part of the calling environment. Listing 24.2 shows a simple script that lets you see what you've got.
Listing 24.2. Obtaining browser information.
#!/bin/sh -f echo "Content-type: text/html" echo "" echo "<PRE>" env || printenv # SVR4 or BSD - skip errors echo "</PRE>" exit 0
The output from the server when I connect via the Web shows what kind of client I'm using, from what system I'm connected, and lots of other interesting information, if you look closely.
DOCUMENT_ROOT=/home/httpd/htdocs GATEWAY_INTERFACE=CGI/1.1 HTTP_ACCEPT=*/*, image/gif, image/x-xbitmap, image/jpeg HTTP_USER_AGENT=Mozilla/2.1N (Macintosh; I; PPC) PATH=/bin:/usr/bin:/usr/ucb:/usr/bsd:/usr/local/bin QUERY_STRING= REMOTE_ADDR=205.149.165.109 REMOTE_HOST=ws3.hostname.com REQUEST_METHOD=GET SCRIPT_NAME= /test.cgi SERVER_NAME=www.intuitive.com SERVER_PORT=80 SERVER_PROTOCOL=HTTP/1.0 SERVER_SOFTWARE=NCSA/1.4.2
Notice that there's no remote login name or e-mail address in this collection of information. The best you can get automatically from a connection (today) is the user's remote host name (variable REMOTE_HOST) and the type of browser that the client is running (variable HTTP_USER_AGENT). You can extract the specific browser with a couple of simple UNIX tools:
x="`echo $HTTP_USER_AGENT | sed 's:/: :g'`" browser="`echo $x | cut -d\ -f1`"
The sed invocation replaces the slash that separates the browser from its version ID, so it has arguments separated by spaces. Then, cut gives you the first field (-f1) in the argument as variable browser.
There are a variety of different browsers on the Net today, but a small number represent a large percentage of the user population. Here are some of the most common user agent identification strings:
HTTP_USER_AGENT=Lynx/2-4-2 libwww/2.14 HTTP_USER_AGENT=Microsoft Internet Explorer/4.40.308 (Windows 95) HTTP_USER_AGENT=Mozilla/0.96 Beta (Windows) HTTP_USER_AGENT=Mozilla/1.1N (Macintosh; I; PPC) HTTP_USER_AGENT=Mozilla/1.22 (Windows; I; 32bit) HTTP_USER_AGENT=NCSA Mosaic for the X Window System/2.4 libwww/2.12 modified HTTP_USER_AGENT=NetCruiser/V2.00 HTTP_USER_AGENT=PRODIGY-WB/1.3e HTTP_USER_AGENT=Spyglass Mosaic/1.0 libwww/2.15_Spyglass
| TIP |
Mozilla is the internal name for Netscape's Navigator and has crept out in various releases of the browser |
Being able to automatically identify the browser software within your script offers a terrific capability: browser-sensitive pages.
Suppose you want to have two versions of the same Web page, one for people with graphic capabilities and another for those using the utilitarian Lynx text-only Web browser. Listing 24.3 shows how you can do it.
Listing 24.3. Producing browser-dependent Web pages.
#!/bin/sh -f echo "Content-type: text/html" echo "" x="`echo $HTTP_USER_AGENT | sed 's:/: :g'`" browser="`echo $x | cut -d\ -f1`" if [ $browser = "Lynx" ] ; then cat $text_only_home_page else echo $graphical_home_page fi exit 0
One drawback to this approach is that you must maintain two parallel HTML pages with the same basic information. Web page development tools that can help with this problem are just starting to show up on the market; you could also use m4 or a similar UNIX macro processor to allow #ifdef-style conditionals within a single version of the file that would automatically split into separate browser-specific documents.
One common use for the Web is to provide information that's available for the public and simultaneously provide other information only for people in the same institution or type of business. There's a very straightforward way to accomplish this by using the REMOTE_HOST environment variable. Its value is computer.domain.top-level-domain-for example, www4.intuitive.com. To extract the relevant information, use a pair of UNIX commands:
TOPMOST="`echo $REMOTE_HOST | rev | cut -d. -f1 | rev`" DOMAIN="`echo $REMOTE_HOST | rev | cut -d. -f1,2 | rev`"
Note the double use of the obscure UNIX rev command, which reverses characters in the line. By reversing the remote domain name (to moc.evitiutni.4www), you can get the first field only (-f1 to cut) as the topmost domain and the first two fields as the regional domain. Then, flip it back with another call to rev, and you're ready to use the domain name.
How could you use this? The following shows an example:
echo "<h2>Our Favorite Links</h2>" if [ $TOPMOST = "com" ] ; then cat commercial_links elif [ $TOPMOST = "edu" ] ; then cat educational_links else cat other_links fi
The rage in Web site promotion is advertising, and this tool gives you the capability to tailor a portion of your page-perhaps to your advertisers-based on their domain. For example, you can list different sets of advertisements or sponsors for students and commercial users (which includes subscribers America Online, the Microsoft Network, and all the dial-up services, too).
When a browser connects to a server, its host name is sent as the environment variable REMOTE_HOST. Knowing that, you can write a shell script to run on a server and indicate the speed of the connection between the visitor and the server system. Listing 24.4 shows how that would look.
Listing 24.4. Checking the client/server connection.
#!/bin/sh -f echo "Content-type: text/html" echo "" echo "<HTML>" echo "<H1>Ping info to host $REMOTE_HOST:</h1>" echo "<BLOCKQUOTE><PRE>" ping -c 10 $REMOTE_HOST echo "</HTML>" exit 0
If you connect to this script via the Web, you might see the following output:
Ping info to host test.intuitive.com:
PING test.intuitive.com (205.149.165.109): 56 data bytes
64 bytes from 205.149.165.109: icmp_seq=0 ttl=47 time=351 ms
64 bytes from 205.149.165.109: icmp_seq=1 ttl=47 time=286 ms
64 bytes from 205.149.165.109: icmp_seq=2 ttl=47 time=310 ms
64 bytes from 205.149.165.109: icmp_seq=3 ttl=47 time=293 ms
64 bytes from 205.149.165.109: icmp_seq=4 ttl=47 time=291 ms
64 bytes from 205.149.165.109: icmp_seq=5 ttl=47 time=293 ms
64 bytes from 205.149.165.109: icmp_seq=6 ttl=47 time=302 ms
64 bytes from 205.149.165.109: icmp_seq=7 ttl=47 time=284 ms
64 bytes from 205.149.165.109: icmp_seq=8 ttl=47 time=289 ms
64 bytes from 205.149.165.109: icmp_seq=9 ttl=47 time=297 ms
--- test.intuitive.com ping statistics ---
10 packets transmitted, 10 packets received, 0% packet loss
round-trip min/avg/max = 284/299/351 ms
Because ping computes an average round-trip time (see the last line of the preceding output), you could even add some smarts to your script that would let you take different actions based on the speed of the connection between the server and the browser system.
The first step is to extract the average round-trip packet speed:
ping -c 10 $REMOTE_HOST > /tmp/pingme.$$
average="`tail -1 /tmp/pingme.$$ | awk -F/ '{ print $4 }'`"
Now average has the average ping speed, in milliseconds, which you can then use as a gauge to deliver different information based on connection speed. The following is a very simple example of different graphics that you might include on your page based on whether the client has a fast or slow connection (fast connections get larger images in this case). Listing 24.5 shows how you could write a script to exploit this idea.
Listing 24.5. Delivering graphics based on connection speed.
if [ $average -lt 100 ] ; then echo "<img src=hi-rez.jpg>" elif [ $average -lt 200 ] ; then echo "<img src=low-rez.jpg>" else echo "<img src=black+white.gif>" fi
Users who connect over a slow line see the black-and-white GIF format, but users with a reasonable speed connection see a color graphic. However, if your users connect to the same site over a really fast line, they are surprised to see a beautiful, high-resolution JPEG image.
What if you want to specify what machine you want to ping? It turns out that there's another environment variable that can be sent from the browser to the server called QUERY_STRING. If you simply specify the URL of a particular CGI program or script and add a ?, anything following that question mark-up to the first space-is sent as the value of the QUERY_STRING variable.
Listing 24.6 shows how you could easily modify the ping script to check the connection between the server and any arbitrary system on the Internet.
Listing 24.6. Using ping for a specific machine.
#!/bin/sh -f echo "Content-type: text/html" echo "" echo "<HTML>" if [ "$QUERY_STRING" = "" ] ; then echo "<h1>no query string? No host to check</h1>" else echo "<H1>Ping info to host $QUERY_STRING:</h1>" echo "<blockquote><pre>" ping -c 10 $QUERY_STRING echo "</pre></blockquote>" fi echo "</HTML>" exit 0
Suppose the script in Listing 24.6 is called query.cgi and that it lives on test.intuitive.com. The URL for that particular script is then http://test.intuitive.com/query.cgi. Now you can actually send it some arguments-in this case, the host name of a machine to check-by appending the desired system name to the URL itself. For example, within Netscape Navigator, you can click the Open URL button within your browser and type
http://test.intuitive.com/cgi-bin/query.sh?pipeline.com
The following output is shown as the contents of that page:
Ping info to host pipeline.com:
PING pipeline.com (198.80.32.3): 56 data bytes
64 bytes from 198.80.32.3: icmp_seq=0 ttl=244 time=39 ms
64 bytes from 198.80.32.3: icmp_seq=1 ttl=244 time=36 ms
64 bytes from 198.80.32.3: icmp_seq=2 ttl=244 time=41 ms
64 bytes from 198.80.32.3: icmp_seq=3 ttl=244 time=42 ms
64 bytes from 198.80.32.3: icmp_seq=4 ttl=244 time=29 ms
64 bytes from 198.80.32.3: icmp_seq=5 ttl=244 time=32 ms
64 bytes from 198.80.32.3: icmp_seq=6 ttl=244 time=31 ms
64 bytes from 198.80.32.3: icmp_seq=7 ttl=244 time=30 ms
64 bytes from 198.80.32.3: icmp_seq=8 ttl=244 time=32 ms
64 bytes from 198.80.32.3: icmp_seq=9 ttl=244 time=31 ms
--- pipeline.com ping statistics ---
10 packets transmitted, 10 packets received, 0% packet loss
round-trip min/avg/max = 29/34/42 ms
As you can see, Pipeline in New York has a much, much faster TCP packet turn-around speed with the server than the test workstation used in the previous example (34 milliseconds compared to 299 milliseconds); that's almost 10 times faster!
Being able to add arguments to a URL as a way to pass information is useful, but it's somewhat limited. After all, what you really want to have on your Web pages are boxes and checklists-places where users can specify information and then press a Do It or Submit button and have that information quickly relayed to the waiting CGI script.
You include such elements with the <FORM> HTML tag set within your documents, as shown in Listing 24.7.
Listing 24.7. Creating a form to reach your script.
<HTML> <h1>How fast is your connection?</h1> <hr> <FORM METHOD=get ACTION="http://test.intuitive.com/query.cgi"> Look for? <input type=string name=ping> <P> <input type=submit value="ping this host"> </form> </HTML>
Most of this script is typically straightforward Web markup until you get to the <FORM> section. The FORM tag has two attributes: the mechanism by which the information should be transmitted to the server and the URL of the CGI program that should receive the information. In this case, you can see that the ACTION specifies that the remote script is referenced as
http://test.intuitive.com/query.cgi
and that the METHOD is Get.
The two basic METHODs for sending information from a browser to a CGI script are Get and Post. Get is the easiest to work with because the information is all tucked neatly into the QUERY_STRING environment variable, but it has some serious size limitations. Instead, complex Web forms invariably use Post, which sends all the information as standard input to the CGI script, allowing an arbitrary amount of data to flow from the browser to the CGI script.
Every input field from an HTML form is sent as a name=value
pair with multiple pairs separated by an ampersand. In the case
of this particular form, the <input type=string>
HTML tag produces a small box within which the user can type the
name of a system to ping. That information is actually
sent to the CGI script as ping=whatever-they-typed (that's
what the name=ping does in that <INPUT>
tag). There's a variety of different input type fields,
including those shown in Table 24.1.
| Input type | Meaning |
| string | One line of text requested from the browser |
| password | One line of text-not echoed as typed |
| radio | One of a set of radio buttons |
| checkbox | A yes/no checkbox |
| submit | The submit or do it button |
| reset | The reset or restore default values button |
Many good references are available both online and in books at your local bookstore, so I won't belabor the point here.
Now you have a simple Web page that prompts for a host to ping and presents an action button right below the prompt that users can click to have their information sent to the script and acted upon. This is shown in Figure 24.2.
Figure 24.2 : A simple form-based Web page inviting input.
How does this variation of the CGI script look? It's surprisingly similar to the last version you saw; indeed, all you need to add is the capability to extract the host name from its name=value form, as shown in Listing 24.8.
Listing 24.8. ping script that accepts input.
#!/bin/sh -f
# modified to accept name=value pairs...
echo "Content-type: text/html"
echo ""
echo "<HTML>"
if [ "$QUERY_STRING" = "" ] ; then
echo "<h1><I>no query string? No host to check</i></h1>"
else
host="`echo $QUERY_STRING | awk -F= '{print $2}'`"
echo "<H1>Ping Info to Host $host:</h1>"
echo "<blockquote><PRE>"
ping -c 10 $host
echo "</pre></blockquote>"
fi
echo "</HTML>"
exit 0
You use the awk program to split the information received at the =, which works fine for a single value. However, if you move to a multiple-variable script, a more sophisticated technique is required.
With this HTML page and the script shown in Listing 24.8, users can now pop over to the Web site and ping any host on the network with wild and merry abandon. Entering pipeline.com and pressing the Submit button even produces results identical to those shown earlier.
You can apply the same technique you used for ping to another simple UNIX command-one that would be useful to access from within the Web environment: the finger command. This is an interesting command because its behavior is dependent on the type of information that you give it; use it to finger a name, and it searches for everyone with that information in the password file, showing you all the results. Give it a remote host name instead-in the form @hostname.com-and it tells you who is logged in to that machine. Use a fully qualified e-mail address-user@hostname.com-and it tells you, the requestor, about that person if it can connect to the machine.
Listing 24.9 shows an HTML file that's a quick and simple finger front end. You'll see that it's remarkably similar to the ping page.
Listing 24.9. Form front end for finger queries.
<HTML> <h1>Finger:</h1><h2>find out about users or computers</h2> <hr> <form method=get action="http://test.intuitive.com/cgi- bin/finger.sh"> Look for? <input type=string name=finger> <P> <blockquote><font size=+1> <I>Try an email address for a specific user, or just the '@hostname' format to see who is using a particular computer on the net</i> </font></blockquote><P> <CENTER><input type=submit value="look up this user or host"> </CENTER> </form> </html>
This time, you've also included some helpful information for the user in the form of a brief italicized comment below the input box, as you can see in Figure 24.3.
Figure 24.3 : Simple Web front end for the finger command.
The script at the other end looks like Listing 24.10.
Listing 24.10. The finger CGI script.
#!/bin/sh -f
# Finger user or user@host or @host
echo "Content-type: text/html"
echo ""
echo "<HTML>"
if [ "$QUERY_STRING" = "" ] ; then
echo "<h1><I>no user or host to check?</i></h1>"
else
value="`echo $QUERY_STRING | awk -F= '{print $2}'`"
echo "<H1>Finger information for $value:</h1>"
echo "<blockquote><PRE>"
finger $value
echo "</pre></blockquote>"
fi
echo "</HTML>"
exit 0
In the spirit of good coding, you again include some error checking-this time, the CGI script can produce an error message (as proper HTML, of course) if you hit the script without going through the HTML page. (If you leave the box blank, it sends finger=, so there's still some data.) If the client does give the script something, it shows you, the client, the results of the finger command run on the server with the information you've specified.
Take a quick look at some of the possible output formats. Note in these examples that the $value sent also appears as part of the output. As an interface rule, this is a great bit of positive user feedback, allowing users to verify that what they sent was processed accurately.
First, what happens if the client doesn't specify anything?
Finger information for :
Login Name TTY Idle When Site Info
taylor Dave Taylor p0 2 Thu 12:57
What happens if you specify @usenix.org to see who might be logged in there?
Finger information for @usenix.org:
[usenix.org]
Login Name TTY Idle When Where
zanna Zanna Knight KA 1:46 Thu 09:00 Zanna's Mac:8.23
toni Toni Veglia co 13: Tue 14:59
diane Diane DeMartini p6 8 Thu 08:44 131.106.3.16:0.0
ellie Ellie Young p7 1:22 Thu 09:04 boss:0.0
toni Toni Veglia pb Thu 09:12 131.106.3.17:0.0
carolyn Carolyn Carr pe 1:26 Thu 09:20 bigx:0.0
ah Alain Henon q0 12 Thu 09:29 131.106.3.29
zanna Zanna Knight q3 59 Thu 09:48 131.106.3.20:0.0
eileen Eileen Curtis q5 36 Thu 10:04 131.106.3.19:0.0
The Usenix Assocation has a lot of people and a lot of things going on. You can also pick someone and submit that person's name @usenix.org to find out more about him or her:
Finger information for zanna@usenix.org:
[usenix.org]
Login name: zanna In real life: Zanna Knight
Directory: /staff/zanna Shell: /bin/csh
On since Dec 7 09:00:41 on KA0.0 from Zanna's Mac:8.23
1 hour 45 minutes Idle Time
No unread mail
No Plan.
Login name: zanna In real life: Zanna Knight
Directory: /staff/zanna Shell: /bin/csh
On since Dec 7 09:48:06 on ttyq3 from 131.106.3.20:0.0
58 minutes Idle Time
Here, Zanna is actually logged in on two different lines, which is why you see two entries for her in this output. Notice that the top entry indicates that she's actually connected from a Macintosh, too. Ah, the things you can glean when you poke around on the networkÉ
One nuance of CGI response that's quite cool is if your program emits a line "Location: url" with some valid URL for a page anywhere on the network, the connection to your server is instantly replaced with a connection to the specified other server. Suppose you're at www.intuitive.com and you have a CGI script elsewhere.sh in the cgi-bin directory. The script elsewhere.sh proves to be a tiny script that looks like this:
#!/bin/sh -f echo "Content-type: text/html" echo "Location: http://www.tntMedia.com/" echo "" exit 0
You connect to the URL http://www.intuitive.com/elsewhere.cgi
and what's returned is actually http://www.tntMedia.com/.
The net result of invoking this CGI script is that you'll be looking
at the home page of TNT Media at www.tntMedia.com.
| NOTE |
The Location: line is output in addition to the Content-Type: header. Notice that the blank line still must appear after these lines |
What can you do with this? The most obvious answer is a random forwarding service, a so-called URL roulette, where you connect to this URL, and it randomly picks another URL from a file and sends you there instead-somewhat like the following pseudo code:
randomurl = `pick-random-line-from urllist` print "Location: " $randomurl
More interestingly, you can combine what I've been discussing to make a page that takes people to the home page of their browsers, as shown in Listing 24.11.
Listing 24.11. Jumping to the right home page.
#!/bin/sh -f
echo "Content-type: text/html"
x="`echo $HTTP_USER_AGENT | sed 's:/: :g'`"
browser="`echo $x | cut -d\ -f1`"
case $browser in
Cello ) loc=http://www.law.cornell.edu/cello/cellotop.html; ;;
Spyglass ) loc=http://www.spyglass.com/three/index.html; ;;
Lynx ) loc=http://www.cc.ukans.edu/about_lynx/about_lynx.html; ;;
NCSA ) loc=http://www.ncsa.uiuc.edu/SDG/Software/; ;;
Mozilla ) loc=http://www.netscape.com/; ;;
NetCruiser ) loc=http://www.netcom.com/faq/; ;;
Microsoft )
loc=http://www.windows.microsoft.com/windows/ie/ie.htm; ;;
* ) loc=http://www.yahoo.com/; ;;
esac
echo Location: $loc
echo ""
exit 0
If you don't know what to do with their browsers, you send 'em to Yahoo!
This chapter shows only the tip of the proverbial iceberg for shell-based CGI scripting. In particular, any time you either display pages based on processing, without user input, or process only a single variable of information, a shell script is probably the fastest and easiest solution available within UNIX.
Other scripting languages are available on UNIX-notably TK, TCL, and Python-and they have very specific capabilities that make them useful for UNIX programming but not for CGI work.