November 11, 2003 Edition

By Amit "Prototyped" Gurdasani, Kyle "greenfly" Rankin

Welcome back to Linux.Ars. This week, we've covered in gory detail how someone tried to get a backdoor into the Linux development kernel's source code. Also, we're reintroducing you to the excellent quasi-lightweight window manager Enlightenment, as well as the X11 extensions aXe and XDamage, both of which should give eye-candy connoisseurs something to cheer about. We'll also show you how to do things with files in entire directory hierarchies. Finally, we'll look at the ROX Filer, a RISC OS inspired file manager for X11 that's pretty nifty to use.

 

Backdoor attempt in the Linux Kernel

Last week saw an alarming occurrence: someone attempted to insert a backdoor into the Linux 2.6 kernel code available from the BitKeeper-to-CVS gateway which provides users of the CVS source control system the ability to access the latest kernel source code tree. This tree is generated from the BitKeeper source control system which holds the source code, and where changes to the code are merged. Larry McVoy of BitMover, the makers of BitKeeper, announced on November 5 that someone had made modifications to code in the CVS repository.

The unauthorized modification was discovered when the CVS source code tree was being synchronized with the BitKeeper tree. The CVS logs indicated that renowned kernel programmer David S. Miller of Red Hat made the changes, but he denied having done so. Closer inspection revealed that someone had modified the C function sys_wait4(), the code that dealt with a particular system call from a user's task, to elevate the task's privilege level to superuser level — current->uid = 0 — when the system call was made in a certain manner. This was made to look like a somewhat subtle programmer error, so that it would not be easy to detect, and would allow for very-nasty and widespread local exploits.

The primary credit for having found and curtailed the exploit goes to the BitKeeper team, who were concerned enough about the security of the Linux kernel that they even put in checks to ensure that after the synchronization, the CVS tree looked identical to the BitKeeper source that it was copied from. Linus Torvalds explained why this sort of modification would have been noticed very quickly, had it been done on the original BitKeeper-managed source code.

We believe that it is a testament to the resilience of the open-source development community that, despite the subtle and serious nature of the exploit attempt, it was found, fixed, investigated and reported to the public within twenty hours' time. Indeed, there was little danger of the code making it into an official kernel release, since releases are cut from the BitKeeper repository, not the CVS repository derived from it.

As of now, it is not known who was responsible for inserting the exploit. The CVS gateway has been taken down for forensics.

 

Enlightenment 0.16.6 released

Users of the Enlightenment window manager were given a treat (and perhaps a surprise) on November 5, 2003 as Enlightenment 0.16.6 was released. Many Enlightenment users had long assumed that any development on the DR16 series was finished, since Enlightenment 0.16.5 was released on October 27, 2000, and all talk since that time had been on the development of the still-to-be-completed DR17 series.

So, why the new release? While Enlightenment has long supported both GNOME and KDE, new releases of both desktop environments were having problems with Enlightenment compatibility, specifically related to the support of Extended Window Manager Hints. As these problems emerged, patches were posted on the e-devel mailing list, and plans for a new release were made, with Mandrake (Not to be confused with the Linux distribution) posting the first 0.16.6 pre-releases.

In June of 2003 Kim Woelders assumed the role as DR16 maintainer and had been working since to incorporate and test all the new patches that were now being submitted. The significant new features in E 0.16.6 include:

Image:Windowgroup.jpg

Setting persistent properties of a number of Eterm windows

Enlightenment has always had as its goal giving users as much control of their desktop as possible, while adding plenty of eye candy. Even though Enlightenment 16 is over 3 years old, it still has many features other popular window managers have not yet incorporated. One can see this with a quick look at the DR16 feature list. Some notable features are proper Xinerama support (for multiple-display systems), per-application based session and property management (window memory, i.e. the ability to keep track of the size, shape, position and decoration of application windows between sessions), Virtual and Multiple Desktops, and highly-configurable window borders.

Advanced users might appreciate the power and flexibility found in the Enlightenment eesh tool, which allows users to perform window manager events and queries on the command line. Using eesh you can write scripts that move, raise, lower, iconify or scroll your windows. Greenfly has used eesh to create a "Quake console" that lowers or raises a terminal via a keybinding, no matter which window has focus.

So, how do you get it? Packages for many Linux distributions can be found on the Enlightenment download page and, depending on the distribution, new packages might already be available for automatic update (E 0.16.6 packages have now entered Debian unstable, for instance).

Image:Windowmemory.jpg

Selecting properties of a Galeon window to remember

 

Developer's corner

This week, we've covered two emerging extensions to the X Window System that are intended to provide mechanisms for things like window alpha blending and intelligent drawing updates. This should prove interesting for developers and desktop users alike, since it will allow for both eye candy and more serious applications, like accessibility for the visually impaired.

Both these extensions have been designed by Keith Packard, who also spearheaded the development of the Render and Rotate-and-Resize (RnR) extensions in XFree86 versions 4.0.2 and later. Bill Haneman from Sun, known for his work on the GNOME Accessibility Project, has also contributed to the project, which should allow for better screen magnifiers and the like.

 

Apportion X extension

The Apportion X Extension (aXe) enables a program external to the X11 display server to be able to render window hierarchies (windows with their widgets, controls and the like; a special case is the root window, which is an ancestor of all windows on the display) in off-screen memory, and then allow an external program (say, your window manager) to decide how to show them. This should allow for effective magnification and contrast effects (hence the interest of accessibility developers), drop shadows on menus, transparency/translucency effects, animations, and the like. Best of all, all these things can occur while the windows are being updated (see below). (As of the deadline for this week's edition, the link to the protocol description on the aXe page seems not to be working, so you can view Google's cached version of it instead.)

 

XDamage Extension

The XDamage Extension allows applications to track when a window obscures them, or when a window is moved across them, etc. Normally, when an obscured window is uncovered, the application that owns it receives an Expose event, whereupon it can repaint the window. The XDamage extension provides the ability to notify an application whenever windows it wants to monitor undergo a redraw, and some portion of them changes. This provides a powerful mechanism for "live" effects, animations, etc., since applications that cause the effects (i.e., a translucent window, a window with a drop shadow, a screensaver, a "live" shrinking/twisting animation during iconization of a window, etc.) can ask to be notified whenever windows that they are obscuring are updated, so that they can themselves ask to be redrawn. In conjunction with the Render extension (to allow translucency/alpha blending of windows) and the aXe extension (for other effects), this extension will allow true alpha blending to come to XFree86, now that an event model exists to support notifications of updates of other windows.

 

TTT: Tools, Tips and Tweaks

In this issue, we introduce bash scripting and regular expressions by demonstrating two techniques: one that searches and replaces text in files in a whole directory tree, and another that bold-faces keywords in the output of grep.

 

A search-and-replace tool for directory trees

Once in a while, we need to search for a phrase in a bunch of files and replace it with another. For instance, an editor might be informed of a consistent misspelling in the next issue of the magazine she's editing. A product's name might be changed on a web site. A company might be acquired or renamed, and marketing and technical documentation might need to be updated. In such a case, it's handy to have a tool around that you can use to automate this.

Here's a bash function that can search and replace in files within an entire directory tree:

replace () {
    SEARCH="`echo \"$1\" | sed -r 's/([\^\.\[\$\(\)\|\*\+\?\{])/\\\\\1/g'`"
    REPLACE="`echo \"$2\" | sed -r 's/([\^\.\[\$\(\)\|\*\+\?\{])/\\\\\1/g'`"
    PATHNAME="`dirname \"$3\"`"
    FILEPATTERN="`basename \"$3\"`"
    find $PATHNAME -name "$FILEPATTERN" -type f -exec sh -c \
"sed \"s/$SEARCH/$REPLACE/g\" \"{}\" > \"{}.S_R\"; exec mv \"{}.S_R\" \"{}\"" \;
}

This can be invoked as in the example below:

$ replace 'ACME Graphics, Inc.' 'ACME Imaging Corp.' 'Reports/2003/Q4/*.{txt,log}'

We should note that all these parameters, especially the third, may contain symbols that bash will try to expand when they are typed on the command line. This is because, unlike DOS and Windows, UNIX shells do globbing, or wildcard expansion, themselves. The solution is to surround all these patterns with single quotes when invoking the bash function, so it will consider the entire contents of the single quote pair a literal string.

Note that the last parameter must contain at least a file pattern because of the way this function works, so it is not enough to specify just '/home/amitg/Reports', for instance; we must specify '/home/amitg/Reports/*' instead.

If we place the function in our ~/.bashrc, the function will be created every time we start a new bash shell.

 

Revisiting find

Let's explain the core functionality of the function we showed you in the previous section. It depends on two tools, find and sed. We've covered the tool find in passing before, but we think that it deserves proper treatment. Not only is find useful in searching for files named a certain way, but it is also an excellent tool to operate on them.

As an example, suppose we have a directory hierarchy full of files of all kinds. We want to search for the term "ACME Graphics, Inc." in only the files ending with .txt or .log, and replace it with "ACME Imaging Corp.", but only in files that are older than a week ago. Say we have tons of these files, so going through each one with a text editor is going to be a lot of tedious work. Thankfully, find is very good at tedious work like this.

find has the ability to take boolean search criteria. If you mention some criteria in a row, such as -name '*.txt' -mtime 7, only files that match both criteria will be operated on. If you mention something like '(' -name '*.txt' -o -name '*.log' ')', if either criterion is satisfied, the file will be operated on.

The -exec option lets you specify a command to be run that can take the matched file name as a parameter, specified by {}, and terminated using ';'. This command will check all files ending in .txt for spelling errors, for instance:

$ find lysdexic -name \*.txt -type f -exec ispell {} \;</pre?
 
(The backslashes are to tell the shell not to expand the *, and not to treat the semicolon as a command separator.
 
The sed command can be used to do the actual search-and-replace in the contents of each file. We'll introduce it a bit better later, but for now, let us mention that the command sed -r 's/''search/replace''/g' ''filename'' will search the file ''filename'' for ''search'' and then replace it with ''replace'', then outputs the text to standard output. This can be ''redirected'' into another file, which can then be moved back. Redirection is accomplished by starting a ''subshell'' that is meant to run just the search-and-replace command, and then using the greater-than sign to indicate that the output goes into another file. Putting all these together gets us:
 
<pre>$ find reports \( -name \*.txt -o -name \*.log \) -type f -mtime 7 \
  -exec sh -c "sed -r 's/ACME Graphics, Inc\./ACME Imaging Corp\./g' {} > {}.new; exec mv {}.new {}" \;

This will faithfully search through everything under the directory reports and replace all instances of ACME Graphics, Inc. in files ending in .txt or .log more than a week old with ACME Imaging Corp.

 

Bold-facing search keywords

We think that few people will say that grep is not a useful tool, especially the GNU version of the tool. If you have large amounts of text to search through, using grep is quick and effective. fgrep ("fast grep"), also invoked as grep -F in the case of GNU grep, is a variant that will only look for fixed strings, rather than the patterns ("regular expressions") that grep will look for.

GNU grep and its variants have the ability to output a number of lines of context along with lines that carry the keyword being searched for. The -A option specifies the number of contextual lines to output after a line containing the searched-for keyword, and the -B option specifies the number of such lines to output before the lines found with the keyword in them.

For instance, searching through the COPYING file that comes with the Linux kernel source for version 2.4.22 for the phrase "free software" yields this:

$ fgrep -i -A1 -B1 -m5 "free software" ~/linux/COPYING 
 of the kernel, and does *not* fall under the heading of "derived work".
 Also note that the GPL below is copyrighted by the Free Software
 Foundation, but the instance of code that it refers to (the Linux
--
 
 Copyright (C) 1989, 1991 Free Software Foundation, Inc.
                       59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
--
software--to make sure the software is free for all its users.  This
General Public License applies to most of the Free Software
Foundation's software and to any other program whose authors commit to
using it.  (Some other Free Software Foundation software is covered by
the GNU Library General Public License instead.)  You can apply it to
--
 
  When we speak of free software, we are referring to freedom, not
price.  Our General Public Licenses are designed to make sure that you

(The -i option tells grep that the provided search term is case-insensitive; for instance, "free software" will be accepted as well as "Free Software" as well as "FREE SOFTWARE" — were it not provided, only the exact term "free software", including capitalization, would be matched. The -m5 option limits the report to only the first five matches; we didn't want to flood you with unnecessary listings.)

The problem here is evident in the third set of results—it is not so easy to find exactly where the term "Free Software" occurs in that set of results, since fgrep will band together lines that are within the bounds of -A and -B that carry the searched-for pattern. This is where ANSI color codes come in.

 

ANSI color codes

ANSI color code is an informal name given to those escape sequences that most terminals and terminal emulators recognize as instructions to output subsequent information in a certain color or style (e.g. boldface, underline, inverse video). These originated in the work of the ANSI X3 Committee, which produced the X3.64 standard in 1979 that specified control codes to be obeyed by compliant terminals. So far, no one has implemented the standard fully, but things like these color control codes as well as various cursor and terminal setting control codes are widely accepted. (If you are interested, the ECMA has a newer version of the standard, ECMA 48 — also accepted as ISO/IEC 6429:1992 — [ http://www.ecma-international.org/publications/standards/Ecma-048.htm available for download] free of charge. A summary of the codes is also available.)

ANSI color codes look like this: <ESC> [ colornumber ; colornumber ; … colornumber m

The control code for boldfacing is 1, and specifying 0 as a control code causes all styles and color settings to return to default. So, we can take fgrep's output and put the boldfacing code before it, and have it return to normal style after the keyword. The tool to do this is sed, the stream editor.

 

sed to the rescue

So we type

$ fgrep -m5 -i -A1 -B1 "free software" ~/linux/COPYING | sed -r 's/(free software)/^[[1m\1^[[0m/gi'
 
 of the kernel, and does *not* fall under the heading of "derived work".
 Also note that the GPL below is copyrighted by the Free Software
 Foundation, but the instance of code that it refers to (the Linux
--
 
 Copyright (C) 1989, 1991 Free Software Foundation, Inc.
                       59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
--
software--to make sure the software is free for all its users.  This
General Public License applies to most of the Free Software
Foundation's software and to any other program whose authors commit to
using it.  (Some other Free Software Foundation software is covered by
the GNU Library General Public License instead.)  You can apply it to
--
 
  When we speak of free software, we are referring to freedom, not
price.  Our General Public Licenses are designed to make sure that you

The ^[ is an escape character, ASCII 27, inserted at a bash prompt by hitting Ctrl-V Esc.

Now let us explain the parameter we provided to sed. Providing regular expressions in the form of s/pattern/replace/options to sed will cause sed to take standard input (by default, input from the keyboard, but we have piped output from grep into sed here) and transform the input. This involves matching bits of text using the regular expression we have indicated as pattern, and replacing the matched text with transformed text according to rules specified where we have indicated replace. This can be just replacement text, or it can consist of back-references like \1, which indicates that the text matching the pattern in the first ( ) pair should be reproduced. The options modify the nature of the match, as we will see in a moment.

So in this case, the pattern free software is matched, and then replaced with the matched text surrounded by control codes that will boldface it. By default, sed will stop with the first replacement on a line, but the g option modifies this behavior to keep looking until the end of the line. Also, since our text contained both the uppercased Free Software and the all-lowercase free software, and we wanted to match both occurrences (hence the -i option to fgrep), we provided the i option to sed, too.

 

Putting the pieces together

We can make this a bash function, so we don't have to type out the sed command every time. We put this in our .bashrc:

bgrep () {
    SEARCH="`echo \"$1\" | sed -r 's/([\^\.\[\$\(\)\|\*\+\?\{])/\\\\\1/g'`"
    fgrep -i "$1" $2 $3 $4 $5 $6 $7 $8 $9 | sed -r "s/($SEARCH)/^[[1m\1^[[0m/gi"
}

In case you're wondering why that SEARCH line is there, it cleans up the search phrase so that sed does not interpret any symbols in it specially as part of the pattern to match, but treats them as literal.

We edit the file using VI or some other editor that can be used to insert the escape character (in this case by hitting Ctrl-V then Esc), then we either log out and in again, or in a pre-existing shell where we want to use it, we run source ~/.bashrc.

The way this function has been written, the first parameter needs to be the text to search for, so any options like -i, -A, -B or -m need to be specified after it.

$ bgrep "free software" ~/linux/COPYING -m5 -A1 -B1
 
 of the kernel, and does *not* fall under the heading of "derived work".
 Also note that the GPL below is copyrighted by the Free Software
 Foundation, but the instance of code that it refers to (the Linux
--
 
 Copyright (C) 1989, 1991 Free Software Foundation, Inc.
                       59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
--
software--to make sure the software is free for all its users.  This
General Public License applies to most of the Free Software
Foundation's software and to any other program whose authors commit to
using it.  (Some other Free Software Foundation software is covered by
the GNU Library General Public License instead.)  You can apply it to
--
 
  When we speak of free software, we are referring to freedom, not
price.  Our General Public Licenses are designed to make sure that you

 

Cool app of the week

A file manager is one of the pillars of a decent desktop environment. One of the first things a user learns to do on his computer is to create, move, copy, list, open, rename, delete, and otherwise handle files, and a good file manager can help to make these tasks easy, quick and efficient. The file manager we introduce to you in this issue, ROX Filer, is among the lighter-weight modern file managers for the X Window System.

ROX Filer's user interface is inspired by the RISC OS that ran on Acorn Archimedes computers. Starting it up, you get a clean GTK+ 2.0 window with icons and file names. Any images that ROX recognizes will have thumbnails generated and used as their icons. On top is a standard toolbar that lets you, among other things, navigate upward through the file system hierarchy, access "bookmarked" locations in the file system as well as a history, resize the icons for a better look at the previews, sort files according to the criterion of your choice (name, type, date, size, owner or group; ascending or descending), etc.

Image:Rox-thumbs-small.gif

ROX Filer at startup

ROX is pretty simple to use — double-clicking a file attempts to open it if ROX knows how, and if it doesn't, pops up a message box asking you to set a run action. Double-clicking a folder will open it in the same window, using the same view settings.

Actions such as renaming a file or folder, setting an icon on an item or a class of items, or setting a run action are done via the contextual menu that pops up on a right click. This contextual menu actually encompasses everything that you can do with ROX Filer. It also lets you set ROX's options, as well as allowing you to modify your view of the folder — how to sort the files, how much detail to show, how big the icons should be, etc.

Image:Rox-set-icon.gif

Icons in ROX Filer can be set arbitrarily

ROX Filer lets you set icons arbitrarily. Any file or folder can have a custom icon set, or you can choose to set the icon for all items in a MIME type (what ROX Filer uses to determine the type of a file) by just right-clicking any one and setting its icon. Similarly, it is possible to set the run action for any MIME type or its category using this right-click menu.

ROX Filer enforces the one-window-per-folder rule. This prevents any confusion on the user's part, so she does not need to decide whether or not a window is a duplicate of another, so whether it should be closed or not. Unfortunately, it does not seem to preserve window size or view information, both of which are important for spatial cognition and manipulation of files.

For quick navigation through the filesystem, one can hit the / key, and then type and tab-complete his way through the filesystem. The tab completion is very convenient, and rather unusual in a file manager.

Image:Rox-goto.gif

Go places quickly with ROX Filer

Unfortunately, navigating among the icons in a window by typing the first letter of the file you are attempting to locate isn't supported in the regular view — however, this is easily accomplished by using the location bar (Indeed, this mechanism is more powerful, since you can type more letters in the name, and the selector will move to the most appropriate icon thus far).

Image:Rox-tab-completion.gif

Keyboard navigation in the window is convenient

ROX Filer supports XDND quite well, which means that you are able to drag and drop objects to and from most GTK+, GNOME and KDE applications easily. ROX itself makes much use of the drag-and-drop paradigm, for instance, in setting icons. It also supports creating scripts and symbolic links in a "Send To" menu, where you can specify target applications and scripts that can process files.

ROX can draw a desktop with icons, too, and can also be configured to pass mouse clicks on the desktop through to the window manager. ROX calls the desktop a "pinboard". You can start this up by invoking rox -p=default. The icons on the desktop do not reside in any directory on the hard disk, but rather, they exist as items in an XML file. Consequently, you also cannot move items to the desktop, but only create references to them. The pinboard can also be configured to show minimized windows' icons. Setting the background image is easy — right-click somewhere on the pinboard, choose "Backdrop", and then drag and drop an image of your choice into the dialog box provided (similar to the Set Icon dialog box shown above). There is a nifty history/navigation menu provided, too.

The file manager is highly configurable. Colors used for file names can be configured based on file attributes. The spacing between icons can be adjusted. Options to make life convenient are provided, such as resizing icons automatically based on the number of files in the folder, and resizing the window automatically to fit the icons it displays. There's even a section on changing ROX's interactions with misbehaving applications to get them to interoperate.

There is also the ability to show a panel. Running rox -l=default creates a sidebar on the left edge of the screen that carries icons for folders and any launchers that you care to create by dragging and dropping application executables onto it. Similarly, rox -r=default will create the panel on the right edge of the screen, rox -t=default will create a panel along the top edge, and rox -b=default will create one along the bottom edge.

In all, we enjoyed using ROX Filer, and found it to be a good lightweight alternative file manager.

 

/dev/random

This article is in need of attention.

Please improve it and then remove this notice.

[javascript:function getJD (b) { var p = /^(([0-9]+)\/)?(([0-9]+)\/)?(([0-9])+)$/; var r = p.exec(b); return (new Date (r [5] == undefined ? 1995 : (r [5] < 1000 ? Number (r [5]) + 1900 : r [5]), (r [2] == undefined ? 1 : r [2]) - 1, r [4] == undefined ? 1 : r [4]) - new Date (-4713, 10, 25).valueOf ()) / 86400000; } var q; var b; var e; q = prompt ('Enter keywords to search for:'); if (q) { b = prompt ('Enter start date (e.g. 10/23/2002, 10/2002, 1/1995, 1998 or 96). Leave blank for 1/1/1995:'); if (! b) { b = '1/1/1995'; } b = getJD (b); e = prompt ('Enter end date, or leave blank for present:'); var tomorrow = new Date (new Date ().valueOf () + 86400000); if (! e) { e = tomorrow.getDate () + '/' + (tomorrow.getMonth () + 1) + '/' + tomorrow.getYear (); } e = getJD (e); window.location.href = 'http://www.google.com/search?q=daterange:' + b + '-' + e + '+' + escape (q); } else void(0); month/date/year dates] or by [javascript:function getJD (b) { var p = /^(([0-9]+)\/)?(([0-9]+)\/)?(([0-9])+)$/; var r = p.exec(b); return (new Date (r [5] == undefined ? 1995 : (r [5] < 1000 ? Number (r [5]) + 1900 : r [5]), (r [4] != undefined ? r [4] : r [2] == undefined ? 1 : r [2]) - 1, r [4] == undefined ? 1 : r [2] == undefined ? 1 : r [2].valueOf ()) - new Date (-4713, 10, 25).valueOf ()) / 86400000; } var q; var b; var e; q = prompt ('Enter keywords to search for:'); if (q) { b = prompt ('Enter start date (e.g. 23/10/2002, 10/2002, 1/1995, 1998 or 96). Leave blank for 1/1/1995:'); if (! b) { b = '1/1/1995'; } b = getJD (b); e = prompt ('Enter end date, or leave blank for present:'); var tomorrow = new Date (new Date ().valueOf () + 86400000); if (! e) { e = tomorrow.getDate () + '/' + (tomorrow.getMonth () + 1) + '/' + tomorrow.getYear (); } e = getJD (e); window.location.href = 'http://www.google.com/search?q=daterange:' + b + '-' + e + '+' + escape (q); } else void(0); day/month/year]. (These links require JavaScript and are known not to work with Konqueror.)