Geek Pit

I recently had the need to install Linux on an older PC - a generic PIII 1Ghz with 384MB of RAM. I’ve long been a fan of Debian for older hardware - the installer allows you to install a base system of about 400MB which you can tweak and add onto after-the-fact. I find this much easier than installing a complete desktop and removing loads of uneccesary bloat afterwards. I’m going to show you how to proceed with such a minimal install of Debian Etch and get X and a few lightweight apps installed.

To get started, during the installation, select your network mirror as usual, but do not select anything during the software selection phase. By default, you will have “Desktop Environment” and “Standard System” selected. Un-select both and proceed with the rest of the install. Here is what the default looks like:

Proceed with the rest of the installation as normal; it will be very fast, since only a basic console system is installed. Log into your bare-bones desktop as root and edit (with vi or nano) /etc/apt/sources.list to remove the Debian CD (or DVD) line at the top. You should have just a couple of lines left:
deb http://ftp.debian.org/debian/ etch main deb http://security.debian.org/ etch/updates main contrib
Now let’s install a few basics - X, a lightweight window manager, GNU Emacs and some tools for text-mode and graphical browsing. Feel free to substitute your own window manager here, I prefer icewm as it has the “standard” keybindings I’m used to for a desktop.
aptitude update aptitude upgrade aptitude install ssh openssl wget elinks less screen xserver-xorg \ xresprobe icewm icewm-themes exim4 menu \ build-essential emacs21 iceweasel rxvt
You may want to leave out emacs21 and replace it with vim or nvi. Remove build-essential if you won’t be compiling any software. Iceweasel is just Mozilla Firefox in a Debian package. rxvt is a lightweight replacement for xterm, and menu is a Debian-specific package that keeps your window manager applications menu updated as packages are installed and removed.

If you want to play that music collection, install mpg321 and aumix, console-based mp3 player and volume control, respectively. For image manipulation, imagemagick is a suite of powerful command-line image viewing and editing tools. For printing, install CUPS and associated print filters:
aptitude install cupsys cupsys-client foomatic-filters-ppds
Then open a web browser to http://localhost:631 to add a printer. For mail, give mutt and getmail a try: aptitude install mutt getmail4
Remember that in Debian, extended package documentation is in /usr/share/doc/<package name>, for example /usr/share/doc/getmail4. A helpful tip - a lot of package documentation is stored in compressed form - you can read it with zless, which will uncompress it on the fly.

During the Xorg server configuration, you can probably accept the defaults to all of the questions - with xresprobe installed, critical things like screen resolution and monitor timings will be auto-detected. When all this is done, log out and back in as a non-root user, and type “startx” at the prompt. Here is a look at the spartan desktop:

If you need to, you can redo the Xorg configuration with this:
dpkg-reconfigure -phigh xserver-xorg
If you would rather install a graphical login manager, you can choose from gdm, xdm or kdm, or do a package search with apt-cache search “display manager”. I recommend xdm, as it has the fewest dependencies.

That’s pretty much it. At this point, you still have far less than a gigabyte of software and OS installed, and your desktop will seem positively snappy, even on slow hardware.

As I related previously, I just upgraded to Wordpress from Blogger. I’m using Wordpress 2.2 under Debian Etch. There were a few snags in the upgrade, so I’d like to tell you about my solutions to them.

Preparing for the Jump

The first thing I found was that Wordpress 2.2 has a nice new Blogger import feature, but it requires that you are using the “new” Blogger (called “Beta” for quite some time), not the old. Since I had some Blogger template modifications, I had been resisting the urge to upgrade. This forced my hand, however. I used the Blogger control panel to upgrade my template, then added back my modifications for shaded blockquote and code boxes. I also had previously added some javascript code that handled “Read More..” links in posts, but I left this out of the upgraded template.

Authentication Failures

Next, I logged into my Wordpress control panel, and went to Manage->Import->Blogger. There was an “Authorize” link, which I clicked on, but the first time I did this it failed Google authentication with the message “We were not able to gain access to your account. Try starting over”. When I clicked on “Clear account information” and tried again, the same error would be displayed. After a little digging on the Wordpress support forums, I found a couple of solutions. One is to edit the PHP file responsible for the import, the other is to use wordpress.com as a waypoint for imports to your own installation. I chose the former, since this is my own server and editing a file is much quicker than the other method. The file is wp-admin/import/blogger.php, you’ll find it relative to your web root, so if you have a site hosted in /var/www/foo, the file will be /var/www/foo/wp-admin/import/blogger.php. Edit this file and search for the text Host: www2.blogger.com (around line 84), and change it to Host: www.blogger.com. That’s it. Now the importer worked like a charm. One note - this was fixed in Wordpress version 2.3, so if you are using that version or higher, you should be fine. The importer also won’t change your Blogger blog in any way, it just reads the data, so it’s safe to experiment with if you’re not sure Wordpress is for you.

Post Formatting

The next issue related to the fact that I edited my blog posts using Emacs (I now use the Firefox extension It’s All text to integrate more smoothly with Emacs). Anyway, in text and HTML modes where you are using Emacs’ auto-fill, Emacs inserts soft newlines, which are not true newlines, but used just to display formatting on-screen. Blogger’s HTML post editor ignores these linebreaks in pasted text (as it should), but the Wordpress editor does not; it converts these pseudo-linebreaks to real (hard) newlines. The end result was that every one of my imported posts had bizarre formatting when it was displayed on-screen - this even broke a bunch of hyperlinks where the link text spanned a line boundary.

After more digging, I found a Wordpress plugin that was tailor-made to deal with this problem - Alex King’s WP Unformatted plugin. Once I had this installed in my wp-content/plugins directory, I could then set a custom field in each new post with a key of sponge and a value of 1.


The effect of this is to cause Wordpress to ignore the linebreaks in pasted text. This works great for new posts, but what about my 100-some-odd existing posts? This is where GUI interfaces really fall down - it would take me forever to manually set the required custom field in each of my posts. Not to be deterred, I jumped into the MySQL shell and started poking around the database used for the blog. I found that the “geek_postmeta” table (’geek_’ is my database prefix, yours will be different, and is specified in your wp-config.php file) held each post’s custom field data. I drilled in on one of the posts where I had set the ’sponge’ field manually. Notice the row with the meta_key of sponge and the meta_value of 1:

mysql> describe geek_postmeta; +------------+--------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +------------+--------------+------+-----+---------+----------------+ | meta_id | bigint(20) | NO | PRI | NULL | auto_increment | | post_id | bigint(20) | NO | MUL | 0 | | | meta_key | varchar(255) | YES | MUL | NULL | | | meta_value | longtext | YES | | NULL | | +------------+--------------+------+-----+---------+----------------+ 4 rows in set (0.00 sec) mysql> select * from geek_postmeta where post_id = 122; +---------+---------+-------------------+--------------------...-+ | meta_id | post_id | meta_key | meta_value ... | +---------+---------+-------------------+--------------------...-+ | 356 | 122 | blogger_blog | blog.unixlore.net ... | | 357 | 122 | blogger_author | Doug ... | | 358 | 122 | blogger_permalink /2007/11/great-fire... | | 405 | 122 | sponge | 1 ... | +---------+---------+-------------------+--------------------...-+ 4 rows in set (0.00 sec) mysql>
I needed one more piece of information to proceed - the post ID’s of all my posts. The post ID is the number displayed to the left of each post in the Wordpress admin panel, but you can also use the MySQL shell to get the start and end post ID’s: mysql> select MIN(ID) from geek_posts; +---------+ | MIN(ID) | +---------+ | 3 | +---------+ 1 row in set (0.00 sec) mysql> select MAX(ID) from geek_posts; +---------+ | MAX(ID) | +---------+ | 141 | +---------+ 1 row in set (0.00 sec)
I was now ready to script a solution; this one is in Perl, feel free to substitute your favorite language:

#!/usr/bin/perl use DBI; use strict; use warnings; my ($sth,$sql); my $dbh = DBI->connect('DBI:mysql:wordpress:localhost','user','pass'); for (my $i=3; $i<=141;++$i) { next if ($i == 122 || $i == 129 || $i == 123); $sql = "INSERT INTO geek_postmeta (post_id,meta_key,meta_value)\ VALUES ('$i','sponge','1')"; print "Executing SQL: INSERT INTO geek_postmeta \ (post_id,meta_key,meta_value) VALUES ('$i','sponge','1')\n"; $sth = $dbh->prepare($sql); $sth->execute(); }
You need to modify a few things to use this code:

• You need to specify your database username and password in the DBI->connect line.
• You need to change the table prefix geek_ to match your own in the line that begins $sql = "INSERT INTO geek_postmeta…
• You need to edit the starting and ending post ID’s in the for loop. Mine were 3 and 141, respectively.
• You may or may not need a line like next if ($i == 122 || $i == 129 || $i == 123). I used this to skip the post ID’s where I had already set the custom field manually. Just edit it appropriately or comment it out if you haven’t manually changed any posts.

Permalink Structure

Another issue I ran across had to do with permalinks. Blogger was using the following link structure:

http://blog.unixlore.net/year/month/post-name.html
This was important to me because I have a few posts that get quite a bit of traffic, and I did not want users to get 404 errors from broken links. I duplicated the Blogger permalink structure by specifying a custom format in the Wordpress control panel, using the format specifier /%year%/%monthnum%/%postname%.html.


I also had to create an .htaccess file in my web root with the following contents (this was displayed on-screen once I set the permalink format. Your installation may be able to create the file for you if permissions are set appropriately).

RewriteEngine On RewriteBase / RewriteCond %{REQUEST_FILENAME} !-f RewriteCond %{REQUEST_FILENAME} !-d RewriteRule . /index.php [L]
After all this, there were a few posts where the permalinks did not match, most notably where the word ‘a’ was in the post title (Wordpress includes the ‘a’, Blogger does not). You can individually edit a post’s permalink in Wordpress by changing the Post Slug from the post editor as needed.

The Blogroll

The final issue I came across was how to transfer my blogroll and lists of links I had in my Blogger sidebar. These were specified as HTML unordered lists in my Blogger template:

<ul> <li><a href="http://blog.unixlore.net/2006/...</li> <li><a href="http://blog.unixlore.net/2006/...</li> <li><a href="http://blog.unixlore.net/2006/...</li> <li><a href="http://blog.unixlore.net/2006/...</li> </ul>
The easiest way to transfer these lists over is to place a text widget into the site’s sidebar using the control panel (Go to Presentation->Widgets), then edit that specific text widget and just paste the unordered list directly into the widget’s text area, adding an appropriate title. If you want to be able to categorize your links, you’ll need to convert the list into OPML first (I used this site to convert my list of HTML links into OPML). Copy/paste the OPML into a file, then import it via Blogroll->Import Links. The OPML for the above list of links looks like this:

<opml version="1.1"> <head> <title/> <dateModified/> <ownerName>mmpower</ownerName> <ownerEmail>foo@example.com</ownerEmail> </head> <body> <outline type="link" url="http://blog.unixlore.net/... /> <outline type="link" url="http://blog.unixlore.net/... /> <outline type="link" url="http://blog.unixlore.net/... /> <outline type="link" url="http://blog.unixlore.net/... /> </body> </opml>
That’s pretty much everything. Probably the hardest part was dealing with the unformatted posts; if you use the various visual post editors, you probably won’t run into that problem.

I’ve just migrated the original Blogger blog over to Wordpress - I’ve been using Wordpress for a while, and am pretty comfortable with it. It’s also nice to get away from shared hosting, this blog is being hosted on my own server.

I’m going to put something up shortly with the details of how the migration went, there were definitely some hiccups; hopefully the details will help others looking to get away from Blogger. Anyway, everything seems to be running well now, let me know if I missed any links, and be sure to update your feeds to the new feed address: http://blog.unixlore.net/feed.

I’ve been taking a break from blogging lately (as you can see from my last post). Blogs are interesting things. I’m not sure blogging itself was a very new or even ingenious idea; I think advances in search, coupled with ideas like tagging and the accessibility of a pure browser-based content medium made the idea of blogging more appealing (You mean someone might actually read what I write?). I had a PHP-Nuke-based pseudo-blog back in 2000-2001; I used it as a way to distribute software and howtos. There were certainly many online journals prior to that, they just weren’t called blogs.

From a purely utilitarian standpoint, Usenet is a much nicer medium for conversations (Blogger’s posting model is particularly bad). However, if you want to get noticed and perhaps make some money from what you write, blogs are it. Blogging has also changed the political landscape forever. I came across this decent comparison of blogs and Usenet recently (noted with some irony that I’m linking to a blog and not a Usenet post).

Anyway, just some musings, I’ll have some more posts up shortly…

It’s a common sysadmin task to want to change permissions on all the files and subdirectories under a top-level directory. You could just use the ‘-R’ switch to chmod, but what if your files and directories need different permissions? One scenario that comes up is with shared directories - you have a directory tree that has to be writable by users in a specific group. To do this you want to set the group ID bit on all the directories, so that files created by an individual user are always writable by the entire group (this is numeric permission mode 2775). We want regular files to just have permissions 664.

Find

So we first need a way to differentiate files and directories - one easy way is with the find command, which as a bonus will also recurse into subdirectories for us. Here’s our first crack at a solution - let’s assume we have changed to the top-level directory we are interested in already:

find . -type f -exec chmod 664 {} \; find . -type d -exec chmod 2775 {} \;
A word of warning - don’t try something like “find . -type f | chmod 664 *” - chmod will ignore its standard input and change the permissions on all the files in the current directory. This is easily fixable by just re-running chmod, but it would be a disaster if you were trying to delete only certain files or directories. Anyway, in the command above, the “-type f” and “-type d” output just files and just directories, respectively. The “-exec” will execute the given command on each file or directory produced by find. The special construct “{}” is a placeholder for the current argument, as output by find. These commands will work, but they are very slow on large directory trees, since the chmod is operating on one file or directory at a time. We could try to improve the speed by feeding the entire output of find to chmod:

chmod 664 $(find . -type f) chmod 2775 $(find . -type d)

Xargs

These last two commands will work fine until we have more than a few dozen files or directories in total - if we do, we’ll get the error “/bin/chmod: Argument list too long”. That’s a cue that we should be using xargs, a very useful command that will submit its input in manageable chunks to the specified command. Here is our next try:

find . -type f | xargs chmod 664 find . -type d | xargs chmod 2775
This is better - the errors about the command line being too long will go away, and this will work, most of the time. But what happens if we have directories or filenames with spaces, quotes or other special characters in them? This comes up quite a bit when you have transferred files from Windows filesystems - the end result will be that xargs will mangle its input, and the command will fail with an error like xargs: unmatched single quote; by default quotes are special to xargs unless you use the -0 option. The error leads us in the right direction, the solution is to use a couple of options to find and xargs that go together: -print0 and -0.

-print0: Find option that prints the full filename to standard output, terminated by a null character instead of a newline.

-0: Xargs option that says input is terminated by a null character, rather than a newline, and all characters are treated literally (even quotes or backslashes).

Here is our final attempt with find and xargs:

find . -type f -print0 | xargs -0 chmod 664 find . -type d -print0 | xargs -0 chmod 2775
This will work for us all the time, no matter what special characters comprise file or directory names.

Perl

There are some versions of find that don’t support the “-print0″ switch. On these systems, you may be able to use a Perl solution:

perl -MFile::Find -e 'find sub { -f && chmod 0664, $_; \ -d && chmod 02775, $_ },"."'
The find procedure exported by the the File::Find module takes two arguments - a callback subroutine and a list of directories. It will recursively descend into the list of supplied directories (in this case just the current directory “.”), and run the callback subroutine on each file or directory found. The subroutine in this case is an anonymous one, given by “sub { -f && chmod 0664, $_; -d && chmod 02775, $_ }”. It first tests whether the current argument is a regular file, if it is it performs the required “chmod 664″. It then tests whether the current argument is a directory, and as you might expect, performs the required “chmod 2775″. The variable “$_” represents the current argument, in this case whatever the current file or directory name is. Note also that the numeric permissions must always have a leading zero so that the Perl interpreter knows they are octal numbers.

This solution has the advantage of working on any Unix system that has Perl installed, since File::Find is a core Perl module.

I was curious about how fast each solution ran, here are the timings on a directory tree with 9105 files and 370 directories: time find . -type f -exec chmod 664 {} \; real 0m15.687s user 0m5.676s sys 0m9.877s time find . -type f -print0 | xargs -0 chmod 664 real 0m0.132s user 0m0.036s sys 0m0.080s time perl -MFile::Find -e 'find sub { -f && chmod 0664, $_; },"."' real 0m0.151s user 0m0.080s sys 0m0.056s time perl -MFile::Find -e 'find sub { -f && chmod 0664, $_; \ -d && chmod 02775, $_ },"."' real 0m0.160s user 0m0.064s sys 0m0.076s

The Perl solution was surprisingly fast, very much comparable to the xargs solution. When you consider that the last Perl solution timed tests for both files and directories at once, it is faster than running two xargs commands in a row.

Some humor from WTF-d00d.com - Bourne shell server pages. Classic:

The basic idea behind all server page technologies is this: rather than writing code that generates an HTML document on-the-fly by writing it out as a series of print statements, you start with a “skeleton” HTML document and embed the code right inside it. Voila! Instead of having a tangled, unreadable, unmaintainable mess of HTML embedded in source code, you have a tangled, unreadable, unmaintainable mess of source code embedded in HTML.

Bourne Shell Server Pages are ordinary ASCII text files, with the special extension .shit, which denotes “Shell-Interpreted Template.” The result of invoking the page compiler on a .shit file, is, naturally, a shell script.

and yet…the minimalist in me thinks this might be a good idea…

Didier Stevens wanted to see if people would click on an ad that offered to infect them with a virus. Short version, they did.

Mark Pilgrim expresses his frustrations with Amazon’s new ebook reader and DRM.

More humor - what if Gmail had been designed by Microsoft?.

Finally, you can run multiple HTTPS sites off of one IP address with OpenSSL and TLS extensions. You can also do this with mod_gnutls.

I just came across a great Firefox extension called "It’s All Text!". Any HTML textarea you see while browsing gets a little edit button on the bottom right corner - clicking it launches your favorite editor (the frst time you use it, it brings you to the preferences screen). For me, that’s GNU Emacs.

To use it with Emacs, just add (server-start) to your .emacs and use /usr/bin/emacsclient as your editor in the preferences dialog. Now when you click on the ‘edit’ button, you’ll get a new, empty Emacs buffer to type in. When you are done, type C-x # to close the buffer and get back to the browser. You’ll see the contents of the Emacs buffer in the text window. Made a mistake? Clicking ‘edit’ a second or subsequent time will copy whatever is in the textarea into your editor once again.

This Code Goes to Eleven asks if adding namespaces to PHP can save it. That question presupposes that PHP is in need of saving - for better or worse, I think PHP is far too widely used at this point to be in danger of extinction. But yes, the lack of proper namespaces in PHP is a royal pain for anything outside of a trivial script.

You have to worry when Bruce Schneier wonders if the NSA put a backdoor in the new PRNG standard.

Gene Simmons is an idiot. Perhaps he should speak to one of these gentlemen. They seem to be doing quite well, despite the evil music downloaders.

Emacs fans who still like to read printed manuals, the GNU Emacs manual for the latest version 22 is finally out in paperback.

TechRepublic talks about alternative Linux desktops.

OFB.biz has a good series of articles on desktop FreeBSD.

I very infrequently install X11/Xorg on any servers, unless I’m doing an install for a client and they ask for it. My most common server install is a base installation of Debian stable that weighs in at about 300MB. I always thought there was no need for a graphical display on a server, for the standard reasons:

• The X server uses resources better devoted to key server processes
• There are security implications to having the additional libraries and binaries on a system
• The command line is much more efficient when you need to get something done

Of course, you can leave out the X server, and just install the needed X clients. SSH works great with its built-in X forwarding. But you still have a potential security problem to deal with on the server itself - local privilege escalation from an insecure X binary, for example.

It seems things have been changing lately. Memory and CPU are more plentiful, so resources are not as much of a concern as they were even five years ago. Default installs from the commercial Linux vendors install a full-blown graphical desktop, as much as they still offer the choice of a minimal installation. Security will always be an issue, but SELinux and AppArmor ease the concerns for buffer overflows and privilege escalation. And there are some useful graphical tools with features that would be hard to replicate from a shell - Red Hat’s virtual machine manager comes to mind. I still refuse to install X on servers, mainly because I’m habituated to years of shell use (hell, even on my desktops I spend a disproportionate amount of time in a terminal or Emacs buffer). There just seems to be less reason not to install X these days, apart from personal preference.

So I’m wondering, do you install X on your servers, or recommend it for your clients or employer? If so, why?

There’s an interesting article at ITWire on whether or not a small business afford not to run Linux. The conclusion of the author is that small business should be running Linux, both on the desktop and server. One part of the article caught my eye:

I copped some flack from the Windows crowd for some comments in the prequel to this story in which I expressed my dismay at how slow my highly configured computer ran under Windows Home Server…Apparently, this was my fault according to those who serve Redmond. I should have configured and optimised my computer correctly, chosen my security package more wisely, so I was just an idiot and a dumbass who didn’t know what he was talking about…Believe it or not, like most people who use computers for work, I don’t have time to fiddle around to optimise my computers and network.

This is understandable, small-business owners don’t have time to waste tweaking server and desktop settings to get something usable. They want something that works out-of-the-box. Next comes this:

…how come when I partitioned my disk and installed a dual boot Ubuntu 7.10 system without any special tweaking, only then, when I had Linux up and running, did my computer give me the sort of performance I expected from the hardware?

This was the surprising part - for years, Windows advocates have picked on Linux for the need to configure and tweak it - largely true. It’s only been the last year or so that Linux distributions like Ubuntu and Fedora have garnered enough hardware and video support to make installation and configuration pretty painless. Witness the automatic printer configuration in Ubuntu 7.10, for example, or the automatic X-configuration that happens now under Xorg.