scottlu.com

fwd: a tool for peer to peer port forwarding

2008-05-14T23:00:00.000-07:00

Today I am releasing fwd, a tool for port forwarding between peers, through the firewalls and NATs separating the peers. Fwd achieves this without the need for firewall configuration. The only requirement is that each peer is running an instance of fwd.

To set up a forwarding session, fwd negotiates a peer to peer tunnel between the local and remote peers using a technique called Interactive Connectivity Establishment (ICE). Forwarded data is sent through this tunnel.

Fwd is cross platform and runs on Mac, Linux, and Windows. Read more and download fwd here.

Fast 2D graphics w/OpenGL ES

2008-04-23T09:30:00.000-07:00

Say you have an existing 2D game with a frame rate that generates new frames totally in game code, and you are porting to the increasing popular OpenGL ES v1.1 api. You can read about OpenGL ES from the public specification here:

http://www.khronos.org/opengles/spec/

Let's assume, hypothetically, that your game already has optimized 2D graphics code, and only needs a way to get a composed frame to the screen quickly (this is the case for Warfare Incorporated). There are two overall approaches:

1. Turn all your graphics objects into OpenGL texture objects, then compose your frame by texture mapping these into a color buffer.

2. Keep all your optimized 2D graphics code. When a frame is ready, turn it into a texture with OpenGL ES, and texture map this into a color buffer.

There are pros and cons to each of these. #1 will mean a lot of code change: if you have optimized 2D graphics code already, it's possible your graphics are stored in a custom format, and that drawing a particular object is done in a custom way. Porting the format, and the drawing for every object over to OpenGL ES api is a lot of work. However, it will likely be the fastest executing if the device your game is running on has a graphics accelerator. It is also likely to take the least amount of battery life.

Approach #2 is the least amount of work: it means you keep all your 2D graphics code and data formats. When a composed frame is ready you "upload" it to OpenGL ES, and have it draw that to the screen. The problem with this is that uploading a texture is often slow. First, regular texture objects aren't designed to dynamically update. When a texture is uploaded it is converted into a gpu specific native format that is optimized for fast texture mapping. This "conversion" is quite slow. Putting this texture conversion into the middle of your game's frame rate won't work.

Thankfully, OpenGL ES has created the concept of a "PBuffer". A PBuffer is both a surface for rendering onto, that can also be used as a pixel buffer for a texture. It is assumed that PBuffers dynamically update, so using a PBuffer as a texture is designed to be inexpensive. They key thing for your 2D game, is that uploading a new texture into your PBuffer is also fairly inexpensive, using glTexSubImage2D().

Once your frame is composed in your PBuffer, you'll want to draw it onto your windowed surface. OpenGL ES v1.1 has a nice "extension api" called glDrawTex(). This is the fastest and easiest way of drawing a textured quad. Use this to draw your PBuffer contents to your window surface. Before you call glDrawTex(), be sure to set the clipping rectangle with glTexParameter / GL_TEXTURE_CROP_RECT_OES. Finally, your platform specific swap buffers api will copy your color buffer to the associated native window. This whole process will give you decent frame rates with minimal change to your existing game. To recap:

1. Use glOrtho to set up a parallel projection, useful for your 2D game.
2. Create a PBuffer surface and associate it with a texture name. Use powers of 2 dimensions.
3. Use glTexSubImage to update only the parts of the PBuffer that are changing (worst case, the whole frame).
4. Once the PBuffer is ready, draw it to your window surface by setting the cropping rect and calling glDrawTex.
5. Swap buffers. Goto 3.

I will post more information about this approach in June.

Using Leopard's RSS-fed picture screen saver

2008-03-04T14:55:00.000-08:00

I have a lot of pictures - actually it's my wife who takes them all. I mainly ensure they are accessible, safely backed up on and off site, and useful in various ways. They are stored / accessed from a home Linux machine. So when my wife asked to see them on her Mac in a screen saver, I was pleased to discover that Leopard had an RSS based picture screen saver built in. This should be a snap I thought. First thing - our pictures are simply stored on the Linux machine's file system in a large directory tree. I needed a way to 1> serve an RSS feed, and 2> make the pictures in the feed accessible from http. In the process of writing a script to do this I found out a few tidbits about Leopard's RSS screen saver:

- The first thing I did was make a single RSS feed of all the pictures - 60K or so. This just didn't work. (a) For some reason, the screen saver takes forever to validate the large RSS document I enter. It literally hung and after 5 minutes I killed it. (b) every picture in the RSS feed is read right away, rather than reading the picture just before it is displayed. I decided a smaller number of images (< 1000) randomly chosen every 24 hours from the archive, would be a better approach.

- The screen saver ignores the EXIF orientation setting. Modern cameras know the orientation of the photo because it is recorded in the image itself how the camera was being held at the time the picture was taken. I added support to orient the image properly. Doing this required storing the normalized image in a local cache (because I didn't want to modify the original). While doing this, I added support for resizing to a desired size. There is no need to make the image larger than the screen it is displayed on, and resizing makes the file sizes much smaller than the original typically. This is helpful since the screen saver downloads and caches all the items in the RSS feed, so smaller images use less disk space on the Mac.

- Originally I named the pictures in the normalized image cache by index. When the images were changed, the same names were used since it was index based. This caused a problem because the screen saver cached the images by name. So now the images are named based on md5 sum, ensuring uniqueness.

- Finally, I noticed that using an enclosure in the RSS item for the picture caused the RSS screen saver to log a warning message to the console utility about not being able to find the "img node". I changed the script to not use enclosures.

BuildRss is the result, a simple script that works around the Leopard's RSS picture screen saver's various behaviors. I run this script nightly from a cron job to update with a new batch of pictures. You can download the script from here.

Link-Backup v 0.8

2006-12-24T09:43:00.000-08:00

New version of Link-Backup released, with a few enhancements:

v0.8 12/24/2006 scottlu
- allow backup of any file while it is changing
- added --verbose logging to tree building
- minor --verify command fix

Link-Backup can be found here.

Link-Backup v 0.7

2006-09-03T10:50:00.000-07:00

v 0.7 09/02/2006 scottlu
- Ignore pipe, socket, and device file types
- Added --ssh-i to select ssh id file to use (see ssh -i) (Damien Mascord)
- Added --ssh-C to perform ssh compression (see ssh -C) (David Precious)
- Added --ssh-p to specify remote port (see ssh -p) (David Precious)

A friendly user reported a failed backup, and on close inspection it turned out that link-backup was attempting to backup atypical file types such as pipes, sockets, and devices. Now link-backup knows how to ignore those types of files.

Added --ssh-C on request to compress the transfer stream. If you're mainly backing up already compressed files such as pictures and music, --ssh-C is likely not to be a win. For other types of files, it'll save some bandwidth.

Added --ssh-i and ssh-p on request. Good options, thanks for the suggestions and code!

Get Link-Backup here.

Link-Backup v 0.6

2006-06-18T01:34:00.000-07:00

v 0.6 06/17/2006 scottlu
- Ignore broken symlinks and other failed stats during filelist creation
- Added --lock, which ensures only one backup to a given dest can occur at a time
- Released viewlb.cgi, a web based backup viewer.

Special thanks to Joe Beda for the lock feature and backup viewer.

Get Link-Backup here.

Link-Backup v 0.5

2006-04-15T16:11:00.000-07:00

v 0.5 04/15/2006 scottlu

- Added 'latest' link from Joe Beda (thanks Joe!). Now your latest backup is linked from "latest" so it's easy to access.
- Fixed --verify. It wasn't specifying the remote machine correctly.

Read about Link-Backup here.

Link-Backup v 0.4

2004-12-19T09:41:00.000-08:00

I've released Link-Backup v 0.4 with a number of new capabilities, the primary being "incremental backup". Now you can specify the number of minutes to run with --minutes. If the backup doesn't complete in this time, it'll resume where it left off next time it is run again.

Instead of hard-linking between trees, now trees hard link to a central "catalog" of unique files. The catalog has none of the structure of the trees and in a way is simply a large md5(+file stats) to inode lookup table. This approach allows simple incremental backup: the catalog is updated independent from tree creation. Once the catalog update is complete, a tree is created that hardlinks to the catalog.

Other new features include being able to calculate a "diff" between the source and dest, and then a way to update the catalog of the dest with this diff. This is useful if you are doing remote backup and want to take a particularly large set of new files to the destination by hand (or by laptop). Sometimes my DSL can't keep up with the inflow of pictures!

Note: the v0.4 backup format isn't compatible with v0.3! Contact me if you want the v0.3 to v0.4 archive converter.

From the history:

v 0.4 11/14/2004 scottlu
- Changed a central catalog design with backup trees hardlinking to the catalog.
This way catalog updating can be incremental.
- Removed filemaps - not required any longer
- Changed logging to occur in the catalog as well as backups. Changed log parsing
methods accordingly
- Added incremental backup feature --minutes
- Make md5hash calculation incremental so a timeout doesn't waste time
- Created 0.3-0.4.py for 0.3 to 0.4 upgrading
- Added --showfiles, shows differences between src and dst
- Added --catalogonly, updates catalog only, doesn't create tree
- Added --filelist, specifies file list to use instead of tree
- Removed --rmempty
- Added --verbose

Backup

2004-09-10T09:01:00.000-07:00

I have a very large picture archive that needs backup "off-site". Week to week this archive sees a lot of action, whether it be picture addition or reorganization. The pictures are organized by directory, and directories are often moved or renamed. Nothing ever gets deleted. I wanted:

- daily backups so I could go back and see the archive on a particular day.
- hard-links to the previous backup so that only "new" files use disk space.
- graceful handling of moved, renamed, and duplicate files without additional storage or transfer bandwidth.
- backup format of a directory tree that can be directly copied when needed.
- remote backups using ssh.

I didn't find exactly this tool so I created Link-Backup ("lb"). lb keeps track of file stats and content and is able to find and hard-link to identical files independent of filename or path. A side benefit of this is that it'll find and hard-link identical files within your archive. In addition, if there is a duplicate file with different stats (can't be hard-linked) lb will perform a copy at the destination instead of transferring a new copy. lb requires python to be installed at both the source and destination. It sends itself to the remote end, so it only needs to be installed where it is invoked.

Jail vs. UML

2004-09-05T15:02:00.000-07:00

One can never be too careful (paranoid?) when exposing outward facing services to the world. It's been shown time and again there are many creative ways to breach a server, especially when the hacker can replicate the runtime environment of the target and has the time and incentive. A careful approach and most importantly ongoing maintenance are both key to trouble free success. Even so there is no perfect system, only countermeasures.

Earlier this summer I set out to evaluate a hosting OS for this server. There is nothing on this machine that makes it an interesting target, but even so I couldn't bring myself to expose a server without some heavy duty OS provided protection mechanism. I ruled out chroot mainly because there are better protection mechanisms available that don't have some of the documented problems with chroot, such as means of escaping the chroot environment, and the non-partitioning of OS services that have no bearing on the chrooted environment.

I narrowed the evaluation down to two protection mechanisms: FreeBSD Jails vs User Mode Linux (UML). As the name implies, the jail feature in FreeBSD allows the creation of a partitioned execution environment in which system services and state outside the jail are not visible inside the jail. UML allows an instance of linux to run as a user mode process, on linux - a virtualized linux instance. Both of these have the possibility of hosting an http server in a throw-away runtime environment without exposing the native OS.

I first evaluated UML and took note of these issues:

- need to select / build the root image to be hosted
- need to rebuild hosting kernel with skas patch
- need to rebuild hosted kernel to disallow module loading
- need to configure tun/tap device for tunneling the UML's ip traffic
- need a way to start / stop the UML gracefully on boot / shutdown
- lcalls not virtualized inside UML (show stopper?)

These are addressable however my main observations are that UML is not optimized for service hosting, and that it has significant administrative complexity. On the otherhand it looks very useful for OS development and testing purposes. I expect that as UML matures and tools flourish these issues will get addressed. A Google search shows "virtualized server" hosting companies offering UML hosts even though the lcall issue exists. Perhaps it is not a big deal - these companies could have either directly resolved the lcall issue in the hosting kernel, or maybe they just don't expect their customers to be bad guys.

FreeBSD Jails on the other hand offer benefits that UMLs do not. There is technically no OS virtualization, instead the OS recognizes jailed processes and only offers them services available in that jail. A jailed process cannot "see" outside of the jail (unless there is a bug in the jail implementation - a major caveat). A few immediate benefits of this technique:

- There is no separate kernel image to build and maintain.
- The ip network infrastructure is already in place in the native kernel. The jail only needs to be assigned an ip address.
- There is an easy to use existing mechanism in FreeBSD 5.2 to start and stop jails gracefully.

When it comes to service hosting, my observations are that the Jail feature is designed for this use whereas UML isn't specifically, is more mature having been authored in 1999, and has lower administrative complexity than UML.