scottlu.com

Managing ZFS Snapshots

noreply@blogger.com (scottlu) — Sat, 25 Apr 2009 07:30:00 +0000

Recently I've been using ZFS for home backup, for the impressive data integrity and painless snapshot features it has built in. I've put together an OpenSolaris system and migrated my data to it, and came to the point of deciding how to manage snapshots.

Creating a snapshot with zfs is simple, but zfs doesn't help you decide when a given snapshot isn't needed anymore. For this I've created a simple script. Give the script your daily / weekly / monthly / quarterly / yearly snapshot retention policy and your snapshot list, and it will tell you which snapshots from the list you can safely delete.

This makes it simple to create and maintain snapshots with a retention policy from your own scripts. For various reasons, it is difficult to do this with the auto-snapshot service that is part of OpenSolaris.

Read more and download here.

Installing Debian on a MSI Wind PC

noreply@blogger.com (scottlu) — Sun, 25 Jan 2009 22:28:00 +0000

I've been putting together a new backup box using the MSI Wind PC. I ran into a couple of issues that I'll mention here - maybe it will save someone else some time!

My approach was to boot from a USB key, and install the appropriate network install (netinst) .iso. This installs a basic system, and from there you're off to the races. The first set of issues for me was discovering how to create a bootable USB key, and what to put on the key. The second set of issues surround the RealTek 8111C network controller. The default Lenny kernel (2.6.26-1) has a driver r8169 built in, but it just doesn't work reliably, most notably the network device would hang for me when doing large transfers. I ended up going to the Realtek website, and getting the source code for the latest driver, which is r8168 (they clearly rolled back the driver), compiling it and modifying my install to use it. This is very easy to do. I'll list the key points below.

To boot from USB key and install the latest Lenny netinst.iso:

1. Follow these instructions:

http://www.debian-administration.org/articles/446

But for boot.img.gz, use this image:

http://http.us.debian.org/debian/dists/lenny/main/installer-i386/current/images/hd-media/boot.img.gz

and for the netinst.iso, use this image:

http://cdimage.debian.org/cdimage/lenny_di_rc1/i386/iso-cd/debian-testing-i386-netinst.iso

2. Mark the USB partition as bootable, if it isn't already. You can do this with /sbin/parted.

3. Install a master boot record:

install-mbr /dev/xxx (where xxx is the usb device)

install-mbr is part of the mbr package on Debian.

Now plug your USB key into your Wind PC, and boot. If the hard drive does not have a bootable image on it yet, the bios will boot from your USB key. If it does, you'll need to go into bios setup and tell it to boot from USB first, before the hard drive.

2.6.26-1 has the r8169 driver built in, so Debian installer will detect your network and get an IP address over DHCP. I was able to go through the install without problems (including installing software from the repository). However if you use it for any length of time, you will encounter this bug where the network device simply stops working. You need the latest version of the driver. Once you are done with your install, follow the directions listed here:

http://ubuntuforums.org/showthread.php?t=1022411

I suggest a small modification to the instructions: build the driver before removing the old one, because you'll find you will need to install components to build correctly, like kernel headers, and possibly make. After building and before installing, remove the current r8169 driver, install the new one, and proceed as usual.

Cheap backup box vs. Drobo

noreply@blogger.com (scottlu) — Sun, 25 Jan 2009 21:46:00 +0000

Recently I needed to add another backup destination to my pool. On the search for how to build a cheap Linux box, I found these components and have put one together:

- MSI Wind PC, comes with Intel Atom cpu, for $140 from newegg
- 2gig memory for $27 from newegg
- 1T drive for $120 from newegg

For a total of $287. For comparison, the cheapest standalone, network addressable drobo is $430 (for the drobo, which is a USB device), and $190 for the droboshare (a network device that the drobo plugs in to). That comes to $620. Clearly when it comes to the drobo, you're paying a hefty premium.

Why use a drobo? I have borrowed one that I am experimenting with to understand the tradeoffs. It has hung i/o on my Linux box twice(!), so I am now trying it with a droboshare. It specializes in ease of use and handling hard drive failure gracefully, at the tradeoff of higher cost, less flexibility, and less transparency (it is a magic box and your files are stored in an opaque format, for example).

An advantage of the Linux box over the drobo is flexibility. I can install the necessary software on it to use it remotely in a secure way, for example. Drobo has added "droboapps" but the capability is limited, requires building tools with a special toolchain, and there are hardware limitations, like not a lot of memory to play with. With my Linux box, I have 2gig for $27.

Another advantage of the Linux box is use of a standard file system for storing files (ext3 for example). If I run into trouble with a corrupt disk, it is comforting to know it is in a public format. With drobo, the file system is opaque.

From my point of view, the Linux box with a single drive is also simpler than a drobo. The drobo is emulating a hard drive, with private firmware and 2+ drives. It's a neat idea, but it is more complex, and there are more things to go wrong that are less easy to troubleshoot. A linux box with a single drive is a very standard set up.

The biggest selling point of a drobo (for backup) in my opinion, is the ease of use. No hats with propellers on them required. However if you're evaluating ease of use that highly and you're on a Mac, Time Machine is even easier.

fwd: a tool for peer to peer port forwarding

noreply@blogger.com (scottlu) — Thu, 15 May 2008 06:00:00 +0000

Today I am releasing fwd, a tool for port forwarding between peers, through the firewalls and NATs separating the peers. Fwd achieves this without the need for firewall configuration. The only requirement is that each peer is running an instance of fwd.

To set up a forwarding session, fwd negotiates a peer to peer tunnel between the local and remote peers using a technique called Interactive Connectivity Establishment (ICE). Forwarded data is sent through this tunnel.

Fwd is cross platform and runs on Mac, Linux, and Windows. Read more and download fwd here.

Fast 2D graphics w/OpenGL ES

noreply@blogger.com (scottlu) — Wed, 23 Apr 2008 16:30:00 +0000

Say you have an existing 2D game with a frame rate that generates new frames totally in game code, and you are porting to the increasing popular OpenGL ES v1.1 api. You can read about OpenGL ES from the public specification here:

http://www.khronos.org/opengles/spec/

Let's assume, hypothetically, that your game already has optimized 2D graphics code, and only needs a way to get a composed frame to the screen quickly (this is the case for Warfare Incorporated). There are two overall approaches:

1. Turn all your graphics objects into OpenGL texture objects, then compose your frame by texture mapping these into a color buffer.

2. Keep all your optimized 2D graphics code. When a frame is ready, turn it into a texture with OpenGL ES, and texture map this into a color buffer.

There are pros and cons to each of these. #1 will mean a lot of code change: if you have optimized 2D graphics code already, it's possible your graphics are stored in a custom format, and that drawing a particular object is done in a custom way. Porting the format, and the drawing for every object over to OpenGL ES api is a lot of work. However, it will likely be the fastest executing if the device your game is running on has a graphics accelerator. It is also likely to take the least amount of battery life.

Approach #2 is the least amount of work: it means you keep all your 2D graphics code and data formats. When a composed frame is ready you "upload" it to OpenGL ES, and have it draw that to the screen. The problem with this is that uploading a texture is often slow. First, regular texture objects aren't designed to dynamically update. When a texture is uploaded it is converted into a gpu specific native format that is optimized for fast texture mapping. This "conversion" is quite slow. Putting this texture conversion into the middle of your game's frame rate won't work.

Thankfully, OpenGL ES has created the concept of a "PBuffer". A PBuffer is both a surface for rendering onto, that can also be used as a pixel buffer for a texture. It is assumed that PBuffers dynamically update, so using a PBuffer as a texture is designed to be inexpensive. They key thing for your 2D game, is that uploading a new texture into your PBuffer is also fairly inexpensive, using glTexSubImage2D().

Once your frame is composed in your PBuffer, you'll want to draw it onto your windowed surface. OpenGL ES v1.1 has a nice "extension api" called glDrawTex(). This is the fastest and easiest way of drawing a textured quad. Use this to draw your PBuffer contents to your window surface. Before you call glDrawTex(), be sure to set the clipping rectangle with glTexParameter / GL_TEXTURE_CROP_RECT_OES. Finally, your platform specific swap buffers api will copy your color buffer to the associated native window. This whole process will give you decent frame rates with minimal change to your existing game. To recap:

1. Use glOrtho to set up a parallel projection, useful for your 2D game.
2. Create a PBuffer surface and associate it with a texture name. Use powers of 2 dimensions.
3. Use glTexSubImage to update only the parts of the PBuffer that are changing (worst case, the whole frame).
4. Once the PBuffer is ready, draw it to your window surface by setting the cropping rect and calling glDrawTex.
5. Swap buffers. Goto 3.

I will post more information about this approach in June.

Using Leopard's RSS-fed picture screen saver

noreply@blogger.com (scottlu) — Tue, 04 Mar 2008 22:55:00 +0000

I have a lot of pictures - actually it's my wife who takes them all. I mainly ensure they are accessible, safely backed up on and off site, and useful in various ways. They are stored / accessed from a home Linux machine. So when my wife asked to see them on her Mac in a screen saver, I was pleased to discover that Leopard had an RSS based picture screen saver built in. This should be a snap I thought. First thing - our pictures are simply stored on the Linux machine's file system in a large directory tree. I needed a way to 1> serve an RSS feed, and 2> make the pictures in the feed accessible from http. In the process of writing a script to do this I found out a few tidbits about Leopard's RSS screen saver:

- The first thing I did was make a single RSS feed of all the pictures - 60K or so. This just didn't work. (a) For some reason, the screen saver takes forever to validate the large RSS document I enter. It literally hung and after 5 minutes I killed it. (b) every picture in the RSS feed is read right away, rather than reading the picture just before it is displayed. I decided a smaller number of images (< 1000) randomly chosen every 24 hours from the archive, would be a better approach.

- The screen saver ignores the EXIF orientation setting. Modern cameras know the orientation of the photo because it is recorded in the image itself how the camera was being held at the time the picture was taken. I added support to orient the image properly. Doing this required storing the normalized image in a local cache (because I didn't want to modify the original). While doing this, I added support for resizing to a desired size. There is no need to make the image larger than the screen it is displayed on, and resizing makes the file sizes much smaller than the original typically. This is helpful since the screen saver downloads and caches all the items in the RSS feed, so smaller images use less disk space on the Mac.

- Originally I named the pictures in the normalized image cache by index. When the images were changed, the same names were used since it was index based. This caused a problem because the screen saver cached the images by name. So now the images are named based on md5 sum, ensuring uniqueness.

- Finally, I noticed that using an enclosure in the RSS item for the picture caused the RSS screen saver to log a warning message to the console utility about not being able to find the "img node". I changed the script to not use enclosures.

BuildRss is the result, a simple script that works around the Leopard's RSS picture screen saver's various behaviors. I run this script nightly from a cron job to update with a new batch of pictures. You can download the script from here.

Link-Backup v 0.8

noreply@blogger.com (scottlu) — Sun, 24 Dec 2006 17:43:00 +0000

New version of Link-Backup released, with a few enhancements:

v0.8 12/24/2006 scottlu
- allow backup of any file while it is changing
- added --verbose logging to tree building
- minor --verify command fix

Link-Backup can be found here.

Link-Backup v 0.7

noreply@blogger.com (scottlu) — Sun, 03 Sep 2006 17:50:00 +0000

v 0.7 09/02/2006 scottlu
- Ignore pipe, socket, and device file types
- Added --ssh-i to select ssh id file to use (see ssh -i) (Damien Mascord)
- Added --ssh-C to perform ssh compression (see ssh -C) (David Precious)
- Added --ssh-p to specify remote port (see ssh -p) (David Precious)

A friendly user reported a failed backup, and on close inspection it turned out that link-backup was attempting to backup atypical file types such as pipes, sockets, and devices. Now link-backup knows how to ignore those types of files.

Added --ssh-C on request to compress the transfer stream. If you're mainly backing up already compressed files such as pictures and music, --ssh-C is likely not to be a win. For other types of files, it'll save some bandwidth.

Added --ssh-i and ssh-p on request. Good options, thanks for the suggestions and code!

Get Link-Backup here.

Link-Backup v 0.6

noreply@blogger.com (scottlu) — Sun, 18 Jun 2006 08:34:00 +0000

v 0.6 06/17/2006 scottlu
- Ignore broken symlinks and other failed stats during filelist creation
- Added --lock, which ensures only one backup to a given dest can occur at a time
- Released viewlb.cgi, a web based backup viewer.

Special thanks to Joe Beda for the lock feature and backup viewer.

Get Link-Backup here.

Link-Backup v 0.5

noreply@blogger.com (scottlu) — Sat, 15 Apr 2006 23:11:00 +0000

v 0.5 04/15/2006 scottlu

- Added 'latest' link from Joe Beda (thanks Joe!). Now your latest backup is linked from "latest" so it's easy to access.
- Fixed --verify. It wasn't specifying the remote machine correctly.

Read about Link-Backup here.

Link-Backup v 0.4

noreply@blogger.com (scottlu) — Sun, 19 Dec 2004 17:41:00 +0000

I've released Link-Backup v 0.4 with a number of new capabilities, the primary being "incremental backup". Now you can specify the number of minutes to run with --minutes. If the backup doesn't complete in this time, it'll resume where it left off next time it is run again.

Instead of hard-linking between trees, now trees hard link to a central "catalog" of unique files. The catalog has none of the structure of the trees and in a way is simply a large md5(+file stats) to inode lookup table. This approach allows simple incremental backup: the catalog is updated independent from tree creation. Once the catalog update is complete, a tree is created that hardlinks to the catalog.

Other new features include being able to calculate a "diff" between the source and dest, and then a way to update the catalog of the dest with this diff. This is useful if you are doing remote backup and want to take a particularly large set of new files to the destination by hand (or by laptop). Sometimes my DSL can't keep up with the inflow of pictures!

Note: the v0.4 backup format isn't compatible with v0.3! Contact me if you want the v0.3 to v0.4 archive converter.

From the history:

v 0.4 11/14/2004 scottlu
- Changed a central catalog design with backup trees hardlinking to the catalog.
This way catalog updating can be incremental.
- Removed filemaps - not required any longer
- Changed logging to occur in the catalog as well as backups. Changed log parsing
methods accordingly
- Added incremental backup feature --minutes
- Make md5hash calculation incremental so a timeout doesn't waste time
- Created 0.3-0.4.py for 0.3 to 0.4 upgrading
- Added --showfiles, shows differences between src and dst
- Added --catalogonly, updates catalog only, doesn't create tree
- Added --filelist, specifies file list to use instead of tree
- Removed --rmempty
- Added --verbose

Backup

noreply@blogger.com (scottlu) — Fri, 10 Sep 2004 16:01:00 +0000

I have a very large picture archive that needs backup "off-site". Week to week this archive sees a lot of action, whether it be picture addition or reorganization. The pictures are organized by directory, and directories are often moved or renamed. Nothing ever gets deleted. I wanted:

- daily backups so I could go back and see the archive on a particular day.
- hard-links to the previous backup so that only "new" files use disk space.
- graceful handling of moved, renamed, and duplicate files without additional storage or transfer bandwidth.
- backup format of a directory tree that can be directly copied when needed.
- remote backups using ssh.

I didn't find exactly this tool so I created Link-Backup ("lb"). lb keeps track of file stats and content and is able to find and hard-link to identical files independent of filename or path. A side benefit of this is that it'll find and hard-link identical files within your archive. In addition, if there is a duplicate file with different stats (can't be hard-linked) lb will perform a copy at the destination instead of transferring a new copy. lb requires python to be installed at both the source and destination. It sends itself to the remote end, so it only needs to be installed where it is invoked.

Jail vs. UML

noreply@blogger.com (scottlu) — Sun, 05 Sep 2004 22:02:00 +0000

One can never be too careful (paranoid?) when exposing outward facing services to the world. It's been shown time and again there are many creative ways to breach a server, especially when the hacker can replicate the runtime environment of the target and has the time and incentive. A careful approach and most importantly ongoing maintenance are both key to trouble free success. Even so there is no perfect system, only countermeasures.

Earlier this summer I set out to evaluate a hosting OS for this server. There is nothing on this machine that makes it an interesting target, but even so I couldn't bring myself to expose a server without some heavy duty OS provided protection mechanism. I ruled out chroot mainly because there are better protection mechanisms available that don't have some of the documented problems with chroot, such as means of escaping the chroot environment, and the non-partitioning of OS services that have no bearing on the chrooted environment.

I narrowed the evaluation down to two protection mechanisms: FreeBSD Jails vs User Mode Linux (UML). As the name implies, the jail feature in FreeBSD allows the creation of a partitioned execution environment in which system services and state outside the jail are not visible inside the jail. UML allows an instance of linux to run as a user mode process, on linux - a virtualized linux instance. Both of these have the possibility of hosting an http server in a throw-away runtime environment without exposing the native OS.

I first evaluated UML and took note of these issues:

- need to select / build the root image to be hosted
- need to rebuild hosting kernel with skas patch
- need to rebuild hosted kernel to disallow module loading
- need to configure tun/tap device for tunneling the UML's ip traffic
- need a way to start / stop the UML gracefully on boot / shutdown
- lcalls not virtualized inside UML (show stopper?)

These are addressable however my main observations are that UML is not optimized for service hosting, and that it has significant administrative complexity. On the otherhand it looks very useful for OS development and testing purposes. I expect that as UML matures and tools flourish these issues will get addressed. A Google search shows "virtualized server" hosting companies offering UML hosts even though the lcall issue exists. Perhaps it is not a big deal - these companies could have either directly resolved the lcall issue in the hosting kernel, or maybe they just don't expect their customers to be bad guys.

FreeBSD Jails on the other hand offer benefits that UMLs do not. There is technically no OS virtualization, instead the OS recognizes jailed processes and only offers them services available in that jail. A jailed process cannot "see" outside of the jail (unless there is a bug in the jail implementation - a major caveat). A few immediate benefits of this technique:

- There is no separate kernel image to build and maintain.
- The ip network infrastructure is already in place in the native kernel. The jail only needs to be assigned an ip address.
- There is an easy to use existing mechanism in FreeBSD 5.2 to start and stop jails gracefully.

When it comes to service hosting, my observations are that the Jail feature is designed for this use whereas UML isn't specifically, is more mature having been authored in 1999, and has lower administrative complexity than UML.