Hacking & Bodging a Git Hook + Vagrant + WP-CLI + Bash Local to Dev Database Transfer

Ever since I started using Git to push local website changes to a development server, I’ve been vaguely irritated about dealing with the database in the same manner. For a long time, I used interconnect/it’s Search Replace DB for this side of things, but I was always wondering if I could somehow increase laziness time by automating the process. One hungover Sunday, plus a couple of hours on Monday, and one hacked and bodged success.

This isn’t a “How to do yer Git + Vagrant + local to dev” thing, nor is it a copy-paste, “Works for me!” party. Nonetheless, provided you’re using git-push, and are comfortable with WP-CLI or MySQL command-line, Bash, and generally thrashing small bits of code around, in principle it would work in any situation. And I do feel kinda dirty throwing all this into Git post-receive, but whatever seems to work.

So, here’s what I wanted to do:

  1. Do all my commits, and run git-push and throw all the file changes to the dev server.
  2. Combine that with a dump of the database, then get it to the dev server.
  3. Use something at the other end to import the database, and search-replace strings
  4. Clean up after at both ends.

When I first looked into this, it seemed using the pre-commit Git hook was the most common approach, dumping the database and adding it to the commit. I didn’t want to do this, for a couple of reasons: I do a lot of commits, and the majority have no database component; I wasn’t looking to version control the database; All I wanted to do was push local to dev database with the changed files. Looks like a job for pre-push hook.

Earlier this year, I started using Vagrant, so the first issue was how to dump the database from there. I do commits from the local folder, rather than SSH-ing into the VM, so mysqldump is not going to work without first getting into the VM. Which brought its own set of weirdnesses, and this was the point when I decided to flop over to WP-CLI, the WordPress command-line tool.

I often find solutions to this sort of thing are dependant on the combination of software and commands being used. I use mysqldump on its own all the time, but here, I needed to use Git to set the path for where the database would be dumped to — because git hooks are in a sub-directory of the git folder — and that, in combination with dumping the database inside the VM while within a Git command running from the local folder (yeah, probably should just do all my git via SSH), and hurling it at a remote server, means sometimes things that work in isolation get cranky. And this is a hack/bodge, so I went with:

  1. Set up paths for the database dump with Git, ’cos Git is running this show.
  2. SSH into the Vagrant box.
  3. WP-CLI dump the database to a gzipped file.
  4. SCP that up to the dev server.
  5. Delete all that on the local server, ’cos I’m tidy.

That’s half of it done. I’ve got my pushes working, the database file is up on the dev server, the local server is all cleaned up, so now it’s time for the other end.

In this case, I was doing it for a site on DreamHost, who conveniently give all kinds of fun command-line access, plus WP-CLI on their shared servers. Once Git has finished checking out the new file changes in post-receive, it’s time for frankly bodging it.

My current usual setup is a bare repository on the dev server, which checks out to the development website directory. This means neither the uploaded database, nor WP-CLI and the WordPress root are in the same place as the running hook. No big deal, just use –path=. The next thing though, is cleaning up post-import. Strings to be changed all over the place, like local URLs swapped to dev. And for that we have, wp search-replace, which is an awful lot like Search Replace DB. At the dev end then:

  1. Set up paths again, this time it’s WP-CLI running the show.
  2. Unzip the database then import it.
  3. Do database stuff like search-replace strings, and delete transients.
  4. Delete that uploaded database file on the dev server, ’cos I’m tidy.

I was looking at all this late last night, all those repeating lines of ‘wp search-replace’ and I thought, “That looks like a job for an array.” Which led me down the tunnel of Bash arrays, associative arrays, “How can I actually do ‘blah’, ’cos bash seems to be kinda unwilling here?” and finally settling on not quite what I wanted, but does the job. Also, bash syntax always looks like it’s cursing and swearing.

The pre-push hook:

#!/bin/sh

# a pre-push hook to dump the database to a folder in the repo's root directory, upload it to the dev server, then delete when finished

echo '***************************************************************'
echo 'preparing to back up database'
echo '***************************************************************'

# set up some variables, to keep things more readable later on
# backup_dir is relative to git hooks, i.e. 2 directories higher, so use git to set it

ROOT="$(git rev-parse --show-toplevel)"
BACKUP_DIR="$ROOT/.database"
DB_NAME="database"

# check there is a database backup directory, make it if it doesn't exist then cd to it

if [ ! -d "$BACKUP_DIR" ]; then
mkdir "$BACKUP_DIR"
cd "$BACKUP_DIR"
else
cd "$BACKUP_DIR"
fi

# cos this is vagrant, first ssh into it. there will be a password prompt
# using EOF to write the commands in bash, rather than in ssh quotation marks

ssh -t vagrant@172.17.0.10 << EOF

# cd to the new databases folder. this is absolute, cos is vm and not local folder
cd "/var/www/user/domain.tld/.database" 

# then export the database with wp-cli and gzip it
wp db export --add-drop-table - | gzip -9 > $DB_NAME.sql.gz

# exit ssh
exit

# bail out of eof
EOF

# scp the backup directory and database to dev server
scp -r $BACKUP_DIR user@domain.tld:~/

# remove that backup directory so it's not clogging up git changes
rm -r $BACKUP_DIR

echo '***************************************************************'
echo 'all done, finishing up git push stuff'
echo '***************************************************************'

The post-receive hook:

#!/bin/sh

echo '***************************************************************'
echo 'post-receive is working. checking out pushed changes.'
echo '***************************************************************'

# check out the received changes from local to the dev site
git --work-tree=/home/user/dev.domain.tld  --git-dir=/home/user/.repo.git checkout -f


# import the database with wp-cli
echo '***************************************************************'
echo 'starting database import'
echo '***************************************************************'

# setting up some paths
# on some webhosts, e.g. all-inkl, setting the alias to wp-cli.phar is required, uncomment and set if needed
# alias wp='/path/to/.wp-cli/wp-cli.phar'

# the path to wp-config, needed for wp-cli
WP_PATH="/home/user/dev.domain.tld/wordpress"
# database directory, created in git pre-push
DB_DIR="/home/user/.database"

# check there is a database directory
if [ -d "$DB_DIR" ]; then

	# then check it for sql.gz files
	DB_COUNT=`ls -1 $DB_DIR/*.sql.gz 2>/dev/null | wc -l` 

	# if there is exactly 1 database, proceed
	if [ $DB_COUNT == 1 ]; then

		#grab the db name, this way the db name isn't hardcoded
		DB_NAME=$(basename $DB_DIR/*)

		echo 'importing the database'
		echo '***************************************************************'

		# unzip the database, then import it with wp-cli
		gunzip < $DB_DIR/$DB_NAME | wp db import - --path=$WP_PATH

		# clear the transients
		wp transient delete --all --path=$WP_PATH

		# run search replace on the main strings needing to be updated
		# make an array of strings to be searched for and replaced
		search=(
			"local.domain.tld:8443"
			"local.domain.tld"
			"/var/www/user/"
		)
		replace=(
			"dev.domain.tld"
			"dev.domain.tld"
			"/home/user/"
		)

		#loop through the array and spit it into wp search-replace
		for (( i=0; i < ${#search[@]}; ++i )); do
			eval wp search-replace --all-tables --precise \"${search[i]}\" \"${replace[i]}\" --path=$WP_PATH
		done

		# any other wp-cli commands to run
		wp option update blogname "blog name" --path=$WP_PATH

		# delete the backup directory, so there's no junk lying around
		rm -rf $DB_DIR
	
	else
	
		echo 'database was not found'
		echo '***************************************************************'
	
	fi

else

	echo 'database folder was not found'
	echo '***************************************************************'

fi

echo '***************************************************************'
echo 'all done'
echo '***************************************************************'

What else? Dunno. It’s pretty rough, but basically proves something I didn’t find an example of all combined into one: that you can use git hooks to push the database and file changes at the same time, and automate the local-to-dev database transfer process. Is this the best way to do it? Nah, it’s majorly bodgy, and would have to be tailored for each server setup, and I’m not even sure doing such things in a git hook is advisable, even if it works. It does demonstrate that each step of the process can be automated — irrespective of how shonky your setup is — and provided you account for that and your own coding proclivities, there’s multiple ways of doing the same thing.

(edit, a day later.)
I decided to throw this into ‘production’, testing it on a development site I had to create on webhost I’m not so familiar with but who do provide the necessities (like SSH and Let’s Encrypt). Two things happened.

First, WP-CLI didn’t work at all in the post-receive script, even while it did if I ran commands directly in Terminal (or iTerm as I’m currently using). After much messing about, and trying a bunch of things it turned out that this was an issue of “has to be tailored for each server setup”, in this case adding an alias to wp-cli.phar.

Second, having a preference for over-compensation while automating, it occurred to me that I’d made some assumptions, like there’d only be one database file in the uploaded directory, and that hardcoding the filename — which was one of those “I’ll fix that later” things — had morphed into technical debt. So, feeling well competent in Bash today, I decided for the “make sure there’s actually a database folder, then check there’s actually a sql.gz file in it, and there’s only one of them, then get the name of that file, and use it as a variable”. I often wonder how much of this is too much, but trying to cover the more obvious possible bollocks seems reasonably sensible.

Both of these have been rolled into the code above. And as always, it occurs to me already there’s better — ‘better’ — ways to do this, like in pre-push, piping the database directly to the dev server with SSH, or simultaneously creating a separate, local database backup, or doing it all in SQL commands.

Git Hook + Vagrant + WP-CLI + Bash Local to Dev Database Transfer

5-Character Dev Environment

Messing with my .bash_profile this afternoon, post-diving into Laravel and Git (which I’ve been doing much of the last week), I realised I could boot my entire dev environment with 5 letters. Fewer, if I wanted.

So instead of going to the Dock, clicking each of the icons, going to each and faffing around, I could at least boot them all, and set off some commands in Terminal (or ITerm2 as I’m now using).

Weirdly, until Justine gave me an evening of command-line Git learning, and wanted my .bash_profile, “Like so,” I hadn’t realised you could do stuff like that, despite amusing myself with all manner of shell scripts. Now I know what’s possible, I’m over-achieving in efficient laziness.

What’s missing is:

  • Opening multiple windows in ITerm or Terminal and running a different command in each (I don’t want to boot multiple instances of an app).
  • Setting off a menu action in an opened app, e.g. in Transmit going to my work drive.
  • Extending it to boot the environment and then a specific project, e.g. “devup laravel” would open my laravel installation in each of the apps, like opening the database in Sequel Pro; cd-ing to the laravel folder after automatic SSH-ing into my Vagrant box, and so on.

Some of these are probably uncomplicated, but this was a 30-minute experiment that turned out to be very useful.

5-character dev environment
5-character dev environment

Website rsync Backups the Time Machine Way

Continuing my recent rash of stupid coding, after Spellcheck the Shell Way, I decided for Website rsync Backups the Time Machine Way.

For a few years now, I’ve been using a bash script I bodged together that does incremental-ish backups of my websites using the rather formidable rsync. This week I’ve been working for maschinentempel.de, helping get frohstoff.de‘s WooCommerce shop from Trabant to Hoonage. Which required repeated backing up of the entire site and database, and made me realise the shoddiness of my original backup script.

I thought, “Wouldn’t it be awesome, instead of having to make those stupid ‘backup.blah’ folders, to let the script create a time-stamped folder like Time Machine for each backup, and use the most recent backup for the rsync hard links link destination?” Fukken wouldn’t it, eh?

Creating time-stamped folders was easy. Using the most recent backup folder — which has the most recent date, and in standard list view on my Mac, the last folder in a list — was a little trickier. Especially because once a new folder was created to backup into, that previously most recent was now second to last. tail and head feels hilariously bodgy, but works? Of course it does.

Bare bones explaining: The script needs to be in a folder with another folder called ‘backups’, and a text file called ‘excludes.txt’.  Needs to be given chmod +x to make it executable, and generally can be re-bodged to work on any server you can ssh into. Much faster, more reliable, increased laziness, time-stamped server backups.

#!/bin/sh
# ---------------------------------------------------------------
# A script to manually back up your entire website
# Backup will include everything from the user directory up
# excludes.txt lists files and folders not backed up
# Subsequent backups only download changes, but each folder is a complete backup
# ---------------------------------------------------------------
# get the folder we're in
this_dir="`dirname \"$0\"`"
# set the folder in that to backup into
backup_dir="$this_dir/backups"
# cd to that folder
echo "******************"
echo "cd-ing to $backup_dir"
echo "******************"
cd "$backup_dir" || exit 1
# make a new folder with timestamp
time_stamp=$(date +%Y-%m-%d-%H%M%S)
mkdir "$backup_dir/${backuppath}supernaut-${time_stamp}"
echo "created backup folder: supernaut-${time_stamp}"
echo "******************"
# set link destination for hard links to previous backup
# this gets the last two folders (including the one just made)
# and then the first of those, which is the most recent backup
link_dest=`ls | tail -2 | head -n 1`
echo "hardlink destination: $link_dest"
echo "******************"
# set rsync backup destination to the folder we just made
backup_dest=`ls | tail -1`
echo "backup destination: $backup_dest"
echo "******************"
# run rsync to do the backup via ssh with passwordless login
rsync -avvzc --hard-links --delete --delete-excluded --progress --exclude-from="$this_dir/excludes.txt" --link-dest="$backup_dir/$link_dest" -e ssh username@supernaut.info:~/ "$backup_dir/$backup_dest"
echo "******************"
echo "Backup complete"
echo "******************"
#------------------------------------------------
# info on the backup commands:
# -a --archive archive mode; same as -rlptgoD (no -H)
# -r --recursive recurse into directories
# -l --links copy symlinks as symlinks
# -p --perms preserve permissions
# -t --times preserve times
# -g --group preserve group
# -o --owner preserve owner (super-user only)
# -D same as --devices --specials
# --devices preserve device files (super-user only)
# --specials preserve special files
# -v --verbose increase verbosity - can increment for more detail i.e. -vv -vvv
# -z --compress compress file data during the transfer
# -c --checksum skip based on checksum, not mod-time & size – SLOWER
# -H --hard-links preserve hard links
# --delete delete extraneous files from dest dirs
# --delete-excluded also delete excluded files from dest dirs
# --progress show progress during transfer
# --exclude-from=FILE read exclude patterns from FILE – one file or folder per line
# --link-dest=DIR hardlink to files in DIR when unchanged – set as previous backup
# -e --rsh=COMMAND specify the remote shell to use – SSH
# -n --dry-run show what would have been transferred

Spellcheck the Shell Way

I was reading this awesome book (about which I shall soon blog) and there was this moment of, “Fark! What a brilliant line!” like I actually said that ’cos it was so good, followed by, “Fark! Spelling mistake of spacecraft’s name!” And I thought wouldn’t a good way to deal with spellchecking (besides my favourite cmd-;) be to take the entire text, do something fancy command-line to it, and output all the words alphabetically by frequency. Then you could just spellcheck that file, find the weird words, go back to the original document and correct the shit out of them. So I did. Brilliant!

# take a text and output all the words alphabetically by frequency
# spaces replaced with line breaks, lowercase everything, punctuation included (apostrophe in ascii \047)
# http://unix.stackexchange.com/questions/39039/get-text-file-word-occurrence-count-of-all-words-print-output-sorted
# http://tldp.org/LDP/abs/html/textproc.html
# http://donsnotes.com/tech/charsets/ascii.html
find . -name "foo.txt" -exec cat {} \; | tr ' ' '\012' | tr A-Z a-z | tr -cd '\012[a-z][0-9]\047' | grep -v "^\s*$" | sort | uniq -c | sort -bnr

Mavericks? I would have called it 10.9 Anarchists myself

Mainly it was because iCal and Address Book lose that utterly vile skeuomorphic stitched leather look, and also realising my afternoon was slightly free, and I’d downloaded the 5.31GB of OS X 10.9 Mavericks, and I was dead lusty for all the new stuff, so 40 minutes later or so I was booting on my venerable (but definitely alive) 2008 MacBook Pro into the first non-cat OS in 12 1/2 years. (Actually a bit more because I was messing around with pre-release versions even before that).

And that was easy, wasn’t it? (Besides needing to reindex my Mail which caused 20 or so emails from years ago to try and send themselves until I mashed the ‘off’ button for Wi-Fi). Tabs in the Finder? Nice! Not sure I’ll use Tags, mainly due to having a decade of junk on my laptop already organised. iCal’s new look and the Day view are especially pleasing (though adding notes is still not entirely possible with keyboard). Safari’s Inspector has been given a new set of clothes. iBooks! Awesome! Really brilliant that it’s finally on Macs. And Maps! My short play with it hasn’t revealed whether it can replace my current map choice for tracking my training rides, but the 3D view of around here makes the trees look like Krynoids from Doctor Who: Seeds of Doom.

Important stuff like Apache, PHP, and MySQL worked almost immediately: the former just needing its httpd-vhosts.conf file updated; the mid needing the former’s httpd.conf edited to load the PHP module and the last working without a problem. And that is the easiest setup for my localhost environment ever.

I also had to buy Little Snitch, which no longer worked, but considering how much I use it, it’s €30 very well-spent – especially considering 10.9 was free. Oh, and iWork, Aperture, bunch of other stuff also updated. Pity I can’t afford one of the new MacBook Pros.

cookie monster

Yeah, I wouldn’t recommend doing this. It’s old. Probably doesn’t work anymore anyway.

I don’t like cookies so much. The persistent browser types, with expiry dates of 2031 that cause a trail of my identity to be left across Google and other sites. And I don’t like how poorly Safari manages them. Even a checkbox option would be better, to keep the ones I need or don’t mind and to delete all the others instead of manually having to go through them all.

SafariPlus used to do this perfectly, from within the browser, unobtrusively. But since 10.5 and Safari 3, it hasn’t worked. So I changed to Cocoa Cookie, a separate small utility. I had to go to my Applications folder to find it, but was still quick and… then it stopped working, it would open without the window showing, caused much weirdness with Safari’s cookies since the latest version and…

uuuhhh… annoyance.

I found a Perl script a couple of days ago. I suppose it could also be done in AppleScript, and should really learn how to write in that, but it’s rather perfect. Well it runs from Terminal also, and requires some editing, but…
#!/usr/bin/perl

use strict;
use warnings;
use File::Slurp;

### Edit this to your liking (put a pipe character between two words)
my $keepCookiesWith = "gaydargirls|culturedcode|supernaut|dreamhost";

### Put your OS X short username here (there should be a directory with the same name under /Users)
my $userName = "francesdath";

### ### ### Don't edit beneath this line unless you know some Perl
my $path = "/Users/francesdath/Library/Cookies/Cookies.plist";
my @date = localtime();
my $date = sprintf("%04d%02d%02d", $date[5] + 1900, $date[4] + 1, $date[3]);
my $cookies = read_file($path);
rename ($path, $path . "." . $date);

open(WH, ">$path");
print WH <<EOF;
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/
PropertyList-1.0.dtd">
<plist version="1.0">
<array>
EOF
while ($cookies =~ m#(\s*<dict>.+?</dict>)#gs)
{
my $cookie = $1;
if ($cookie =~ /$keepCookiesWith/)
{
print WH $cookie;
}
}
print WH <<EOF;

</array>
</plist>
EOF
close (WH);

So, copy it into your favourite text editor, save as something memorable like ‘SafariCookiecleaner.pl’ and put it somewhere out of the way but not forgettable. You can add new cookies to be saved based on any attribute that appears in the Safari cookie window. I tend to use the domain names, like ‘macosxhints’, though I have one Google cookie I like to keep, with the name ‘PREF’, so I added that also (unfortunately YouTube has one with the same name…). Set the ‘username’ to your Home Folder (probably what appears in the SideBar in the Finder), and then set ‘my $path = “”;’ to ‘/Users/yourhomefolder/Library/Cookies/Cookies.plist’ which is the path to Safari’s cookies file.

Then open Terminal, and change to the directory where you’ve put it, change the permissions to 755 and then run it (quit Safari first).

Well like this:
cd /drag/the/folder/containing/the/script/into/Terminal
chmod 755 SafariCookieCleaner.pl
./SafariCookieCleaner.pl

Open Safari, look in cookies in Preferences and the ones you like should still be there. It makes a backup of the cookies file, so at worst nothing is irreparable. And it makes you look all UNIX by opening Terminal.

Things…

I started talking about Things by saying,

For a very long time my way of organising bits and pieces was, for short things in a hurry creating a new folder and sticking the information in its name, or for longer a note in TextEdit, or dragging bits of text or links or whatever to the desktop, to end up being dealt with later, or accumulating in folders called ‘detritus’ and ‘blllrrrblllrrr…’. Somewhat adequate, but not particularly. And as I quite enjoy the interplay between iCal and Mail (and my phone), I’ve always wanted something a little more… ummm… useful.

… and then veered off to talk about MailTags and MailActOn for quite a bit, before deciding to write a completely separate post on Things

I’ve tried various project or task management software before, but Merlin which I would unhesitatingly use for managing projects is completely not appropriate for daily blllrrr… stuff. And a lot of the ethos of GTD which underlies the practical workings of many applications is a bit too dogmatic for me. Or failing that causes so much time to be spent in the Getting, that things don’t get done. Getting Busywork Done might be a better name.

So I decided anyway, bored while erotikputzenarbeitsdurchsuchend, to play a little with Things and see if it was anything I could find useful. I’m not in the habit of idly playing with software too much, and Things is barely out of beta and very much still in development with many additions yet to occur. But what’s more important for me is how it feels to use. And it’s rather pretty.

Familiar enough to be Mac-like with a sidebar containing five general areas, Collect, Focus which is split into Today Next Scheduled and Someday, then Active Projects, Areas, and Logbook, Trash, then a main window which lists things either individually or combined into a project, or separately for recurring scheduled items. mmm, rather difficult to explain the interface, because it’s more in how I use it that it finds itself.

While it’s not immediately apparent visually, Areas are like overarching projects and Projects are sub-projects within an Area. Though it doesn’t have to be used like this, I find the nesting of projects suits how I think and also comes from my use of Merlin. Then individual items, To Do’s, tasks can be created or stored in the Inbox or Today or Next (or Scheduled for later, or Someday for unlikely to be done before spring), which I think of as items in a project.

So for an example of how I use this, I have an Area called Computer, which has two Projects in it, Blog and Computer Maintenance, several uncategorised items and one recurring task, a weekly scheduled reminder to back-up my data. Items that will appear in Today have a yellow star and a bar showing how many days left until due, which becomes red when it says, ‘! day left’. Projects have a blue bar at the left end showing how many items are contained, and scheduled items are set apart and below, indented with a dropdown menu on the left for the schedule. Weirdly this also puts a task in the main list of the project…

Knowing my tendency to let things unravel and not be attended to, I assign due dates for everything. Things does this firstly with a calendar, and secondly with an option to Show in Today either on the day or as many days previous as I like. Or I can place a project or To Do in Scheduled and set a date when it becomes active. Up until then it’s hidden from my default view.

The difference between throwing files and folders all around, or putting them as iCal Events, and having an ordered list of stuff is that I can have an active, timed flow. These things need to be done now, these things soon, don’t worry about those and these are deeply overdue which is why you are unemployed and broke.

Tagging is something I first started with on my blog, rewriting a script so blogging client ecto’s Keywords would build a list of tags to search my blog with. Then of course was MailTags. I rather like the idea though am torn between the need to tag and the need to physically group stuff together.

Things tags are fairly standard, but the ability to build nested tag groups, say a main tag called Location and sub-tags, Home and Away… is very useful, and also the built-in groups for priority, time, and difficulty. I especially like – though am lazy in using – filtering via multiple tags and sorting lists by tags, and that Projects and Areas can have overarching tags that are applied to their contents.

The difficulty in using iCal for everything for me is that I have things which happen at certain times I’d like to do, such as dance class, and things which can be done whenever this week or in the future. Supposedly the latter are To Dos, but I don’t ascribe to a clear delineation between the two. Being freelance means most of my work is done when I decide, rather than at fixed, immutable times. A genuine To Do for me would barely be worth registering as I tend to remember stuff quite well. It’s more about the organising of that stuff that I’m concerned with. So in a way Things is just making neat piles out of already serviceable mess.

Largely I’ve avoided iCal’s To Do’s. They are messily implemented, difficult to edit and view, and… uuhhh… enforce the annoyance of Event / To Do hegemony, a distinctly American obsession with labeling and ordering and structuring even if all this kills whatever life there was in living.

Things more-or-less is an iCal To Do editor. Or it helps me to think of it as that. I create Areas that correspond to iCal Calendars, and within those Areas, various Projects where the actual To Dos that appear in iCal mostly for me are found. Well, that’s if you have iCal syncing turned on for it.

In Things preferences, by choosing Custom from the list, it brings up all my iCal Calendars, in pretty colour too. Then I can choose a calendar to sync with either a group of tags or Projects and Areas, or both. Under Options I can show the tags in the iCal Todo, and also the Projects in the To Do Title. All of this turns a To Do for ‘Back-Up Weekly’ into ‘Computer Maintenance: Back-Up Weekly @Computer@Cleaning’, the first part being the Project, the last the tags. And this with a weekly scheduled recurrence.

Which is nice.

Mainly because I put off backing up until it’s been a month or more…

Things’ New To Do or Quick Entry are quite straightforward, a Title, Tags field, notes Due Date and Where to place it, in any of the Projects, Areas or Focuses. Possibly too simple. Setting an alarm would be nice, or setting a time rather than just a date. But the notes section is as good or better than iCal. Dragging any kind of file or folder into this area creates a link. Well, one that doesn’t remain working if the file is moved but that might just be my overly abused system.

What I really like, and why Things will likely remain on my laptop is I can assign a key combination, in my case command-control-z which brings up the Quick Entry pane in any application and stays there while I copy useful stuff into it.

And for MailTags, which is where this all began, this is quite excellent.

One of the things you can do with Mail Rules is run an AppleScript. One of the things you can do with an AppleScript is tell Applications to do things.

So when I press my MailTags ‘Things Create Task’ keys, control-z, it runs an AppleScript which brings up the Things Quick Entry pane then goes through the email and fills the previously mentioned fields with information from the email, MailTags tags included, and a link to the email itself. Find the script here. There’s other too for non-MailTags users.

Other than remembering to change my key commands, and upgrading MailTags to the latest version because of some weirdness, using this is so simple. I made a folder called Scripts in my ~/Library/Mail, to keep everything together and now have a rather serviceable and useful discussion between Mail and Things. Which makes me want MailTags to do the same for New Events.

How does it look on my phone?

I don’t have an iPhone, (Things also exists as an iPhone app) but my now venerable Sony Ericsson k750c does a fairly good job of keeping things synced between iCal Events in the calendar, Adress Book in the… uhhh contacts, and To Do’s buried in Tasks. Just as they are formatted in Things and iCal as ‘Project: description @tag’, so do they appear on my phone, with the addition of red ‘!’ for things important or late.

Some improvements I would enjoy very much, and I suppose a philosophical musing on applications making you do things in certain ways and imagining how an application might be built that I could drag stuff around to make it work the way I wanted… Nested Projects are hugely important. Syncing with iCal Events, and I don’t care how much this is not of the ethos of Getting Things Done, but I suppose if you want to limit potential users to autocratic geek-zealots then… I mean to say even Merlin has a calendar view, and no it’s not doubling up on applications. To have the same finesse over Events as Things gives over To Do’s would be really nice. I’d still use iCal, it just means how I might do things more coherently could change.

The Logbook needs work, where finished stuff is archived. This mainly though is a slight need for the interface to develop more. And some key commands are… mmm unpredictable. And some general ill-defined strangeness… The database is in my favourite XML, which makes for ease of doing other stuff with it, like web-based task management for groups (People currently suck, a placeholder rather than an AddressBook-synced useable aspect), and reliable back-ups (yet to turn up in the file menu). Though the forums, wiki, blog, twitter and developer contact are all rather special and that makes up for a lot.

I realised while writing this that MailTags has long been a part of my daily computer use, and Things is still… we are coming to terms with each other. That I can look at it now and see what I need to do tonight and tomorrow that is pressing, makes it undeniably useful,. And perhaps this is it. Anything that I take the time and care to put into Things is something that is important. I’m allowed to arrange my stuff in such fluid combinations and views that what remains important is this, not the application’s determination of what is. It makes use of iCal’s to Do’s, which I never really used, and if I really want, I can drag a To Do from the Items window in iCal into the calendar and make it an event… mmm mindless doubling up?

I guess also, would I take the time to learn another task manager, especially when none are as pretty to look at or use, to find if it was capable of my decidedly evasive habits?

…no.

things today view and quick entry pane
things today view and quick entry pane
things tags and nested tags
things tags and nested tags
things quick entry in mail with mailtags and applescript
things quick entry in mail with mailtags and applescript

mailtags & …

For a very long time my way of organising bits and pieces was, for short things in a hurry creating a new folder and sticking the information in its name, or for longer a note in TextEdit, or dragging bits of text or links or whatever to the desktop, to end up being dealt with later, or accumulating in folders called ‘detritus’ and ‘blllrrrblllrrr…’. Somewhat adequate, but not particularly. And as I quite enjoy the interplay between iCal and Mail (and my phone), I’ve always wanted something a little more… ummm… useful.

Actually, iCal and Mail don’t really talk to each other much, even in 10.5, and iCal has a revolting and clumsy interface design for viewing or editing Events and To Dos. And the To Dos are rather useless, better to keep them in the calendar to remind me of things needing doing.

But mostly useful enough that I manage with organising life and projects with these.

Except for my colourful messages. I can’t remember when I started using MailTags and MailActOn, but it’s been long enough I’d cry if it vanished. For several reasons it’s indispensable for managing my emails. I have six addresses, for various reasons, some for mailing lists only, some for blog stuff only, one that I know I can access anywhere. And then I have a multitude of reasons for sending or receiving, that I do need to refer to, like all the emails relating to monadologie, that are also not from the ANAT mailing list. And an endless list of white email headers does not agree with my decidedly visual approach to my laptop world.

Oh and I like doing almost everything through the keyboard also.

With MailTags, I have a bunch of keywords for common things, like ‘reply’ for messages I need to uuhh… reply to, and ‘jobs’ for when I am looking for erotik cleaning work, then I have a bunch of projects, for each of the works I’ve made, and a few other on-going things that I can group together. And each of these has a different pretty colour. Except ‘reply’, which is a garish, demanding yellow.

If I was to view the raw source of a message I’d tagged, I’d see something like this at the end of the header:
X-Mailtags-Version: 2.0
X-Mailtags: {
"mailTagsProject" : ["project monadologie"],
"mailTagsKeyWords" : ["to archive"]}
X-Keywords: to_archive
X-Project: project monadologie

Which is ummm… not especially astounding, except it means I can search from anywhere via Spotlight and find everything for monadologie, including all my emails.

Which is nice.

But mostly I love MailTags because of what it does that Mail and iCal don’t really do so well together. So, I have a calendar in iCal called ‘monadologie’ and it’s a rather astronomical shade of dark, cold blue. And I have a MailTags Project and Mail Rule with the same name and the same colour.

When I receive an email for something ‘monadologie’-related I need to do, like a meeting, firstly I tag the email, control-m, and it turns a delightful deep blue. This is done in the background by earlier creating a new Mail Rule in Preferences, that sets the colour and then tags for the message, and is controlled by MailActOn. Besides turning to arctic white of Mail’s inbox into something hallucinatory, I can find individual messages and create separate groups of messages just by looking at their colour (or sorting by Keyword or Project…).

If you like reading XML formatted Apple .plists, then in Mail’s MessageRules.plist this creates an entry which looks like this (minus sundry empty tags):

<key>MailTagsProject</key>
<string>project monadologie</string>

<key>RuleId</key>
<string>99ABA089-2490-4E83-B755-7EC8E45E6758</string>
<key>RuleName</key>
<string>Act-On: m | project monadologie</string>

Which looks a lot like the stuff in the message above. What it means is, I’ve created a Rule in Mail Preferences Rules pane with Description ‘Act-On: m | project monadologie’, which Performs the Following Actions ‘Set Color’ of ‘Background’ to ‘dark cold blue’ and Set MailTags Project’ to ‘project monadologie’. I’ve also (actually before I created this rule), created a Project in the MailTags pane called ‘project monadologie’ and because I’m obsessive, given it the same colour.

When I’m viewing a message and press the MailActOn Menu Key ‘ ` ‘ it brings up the MailActOn Menu and I tag the message by pressing ‘m’ (which is the ‘Act-on: m’ part of the above rule. Everything to the right of the ‘|’ is the name of the Project or Keyword in MailTags), or select it from the menu. Or I use command-m which does the same without bringing up the menu.

Now I have a more useful email. It relates to the project visually across my computer, it can be found in searches in Mail or Spotlight or in the Finder, I can make a Smart Mailbox for emails with those tags, and keep track of replies and priorities without having to sift through or rely on the inbox.

Then I add a New Event in iCal. I click (because I haven’t found a keyboard shortcut) ‘New Event’ in the MailTags pane and up pops a window where I enter a bunch of relevant stuff from the email, save it and a new Event in iCal is created with a link to the email, and a note in the MailTags panel allowing me to edit or view the calendar entry.

If I were to look at the raw source of my message again, it would now look like this:
X-Mailtags-Version: 2.0
X-Keywords: to_archive
X-Project: project monadologie
X-Mailtags: {
"mailTagsICalEvents" : [{
"location" : "berlin",
"uuid" : "26E471BE-2722-470A-8294-479666921995",
"notes" : "Lie about my age",
"title" : "Daniel Audition",
"endDate" : "@2009-01-17 21:00:00 +0100@",
"iCalReferences" : {
"B34ED321-81A0-4710-BF44-4E3AF50D5224" : "D4471F6C-3F26-4593-AA0A-7FF510F31AC4"},
"alarms" : [{
"triggerInterval" : -5,
"type" : "soundAlarm"}],
"calendar" : "monadologie",
"startDate" : "@2009-01-17 20:00:00 +0100@"}],
"mailTagsProject" : ["project monadologie"],
"mailTagsKeyWords" : ["to archive"]}

All of which you don’t need to know, except when I sync iCal with my phone, I now have all the information from my Mail message in a calendar event, in this case with an alarm, without having to leave Mail.

Then if I want to I can have a smart mailbox with the criteria set to include only mails with the tag ‘reply’, or search for emails with extra ‘Keywords’ and ‘Projects’ items now in the search toolbar. Or do other fun stuff. Like make new tasks in Things using all MailTags and AppleScript.

Mostly it’s about tagging my email with useful information that allows me to deal with things both within Mail and across my computer, in iCal or Spotlight or wherever I happen to need to group arbitrary files and folders together under a common project or task or search. And it does this without needing to physically assign a folder or location for them. For someone who uses the keyboard to get around, it’s a joy. I can archive messages to folders and tag them on the way without drag-and-drop, and find them again, irrespective of where they’ve ended up. I can quickly get through a queue of new mail, filtering it into my abstract collection of projects and keywords so I know what I need to do with it. And I get to look at an inbox full of pretty colours.

It would be nice if there were more keyboard shortcuts, say for making a New Event. Also it would be handy if it automatically populated a new event with relevant information from the email. I suppose I could write an AppleScript for that also, though. And admittedly I don’t really use this to its fullest ability either.

Well, this post was really supposed to be about Things as well, but I’ve realised in writing it properly deserves its own post, just as MailTags has suddenly got its own.

mailtags keywords and projects
mailtags keywords and projects
mailtags act-on and mail rules
mailtags act-on and mail rules
applying a new rule and mailtags panel
applying a new rule and mailtags panel
mailtags pane for editing a new event for ical
mailtags pane for editing a new event for ical

things with bits

For all of you with Apple computers, iPods, iPhones, wireless internet, other bits and pieces of hardware, a need for various design things, anything else you can think of with a laptop or desktop, I’m finally offering what some of you have already enjoyed – Mac support and design services – from my years of fiddling around inside Macs.

THINGS WITH PIXELS

design

video – from documentation to dvd, web
design – print and web
web – sites, blogs, web2.0, social networking

mac stuff

software and hardware support
networks, security, ipods iphones mobile sync
getting things done
backing up your data
help it’s not working
data recovery

learn to do it yourself…

contact

frances d’ath
frances@francesdath.info
+49 1522 63 53 740

costs

call out: €15 to your place
mac stuff hourly rate: €20
design hourly rate: €20
longer projects by agreement
66% off if i think i can’t solve it

negotiable for: students, unemployed, low income, dancers

7 years experience living inside macs

things with bits
things with bits