Georg, with whom I worked on co-writing The Station, asked me if I’d like to do another piece of co-writing with him, this time an opera libretto. I said yes (duh!). Last Friday, we had a three-way chat with Henry Vega, the composer, about Alan Turing, neural networks, science fiction, queer stuff, and all, for a sharp hour (Georg’s good like that with his one-hour meetings).

Today I spent a couple of hours (after some dipping of toes last night) in installing TensorFlow-Char-RNN, a “a character level language model using multilayer Recurrent Neural Network,” as made wildly lovable by Janelle Shane of Letting neural networks be weird. That involved installing TensorFlow. I went for the direct MacOS approach (after toying with either a Vagrant VM or Docker container) of the Virtualenv flavour. Plus Python 3. And pip. Dependencies. We have them.

A bit of faffing around, and out is spat a ‘Shakespeare’:

t ‘vkdwsa avf
neu irot rS
, mvuaeea giCsouo aed renat rs
;iiweszteseooiiWhe thrr l st !htt :hsre

I mean, I was expecting a single, long ‘aaaaaaaaaaaaaaaaaaaaaa’, so this was progress.

More faffing, fans to 6000rpm, CPU to 500%, and some short while later, ‘Shakespeare’!

Before we proceed any further,
Or each doth now foul branch with thy preser’d up
Young to devise me him;
But in my jewities rebeeve me to this,
Your soul than daggers and breeding
some abrother Arms
What will be pronound with a husband; he’s beauty much or a slaughter,
But I’ll wring my false find than how ill.

Nailed it.

TensorFlow-Char-RNN before and after on ~204,000 words of Shakespeare
TensorFlow-Char-RNN before and after on ~204,000 words of Shakespeare

Hacking & Bodging a Git Hook + Vagrant + WP-CLI + Bash Local to Dev Database Transfer

Ever since I started using Git to push local website changes to a development server, I’ve been vaguely irritated about dealing with the database in the same manner. For a long time, I used interconnect/it’s Search Replace DB for this side of things, but I was always wondering if I could somehow increase laziness time by automating the process. One hungover Sunday, plus a couple of hours on Monday, and one hacked and bodged success.

This isn’t a “How to do yer Git + Vagrant + local to dev” thing, nor is it a copy-paste, “Works for me!” party. Nonetheless, provided you’re using git-push, and are comfortable with WP-CLI or MySQL command-line, Bash, and generally thrashing small bits of code around, in principle it would work in any situation. And I do feel kinda dirty throwing all this into Git post-receive, but whatever seems to work.

So, here’s what I wanted to do:

  1. Do all my commits, and run git-push and throw all the file changes to the dev server.
  2. Combine that with a dump of the database, then get it to the dev server.
  3. Use something at the other end to import the database, and search-replace strings
  4. Clean up after at both ends.

When I first looked into this, it seemed using the pre-commit Git hook was the most common approach, dumping the database and adding it to the commit. I didn’t want to do this, for a couple of reasons: I do a lot of commits, and the majority have no database component; I wasn’t looking to version control the database; All I wanted to do was push local to dev database with the changed files. Looks like a job for pre-push hook.

Earlier this year, I started using Vagrant, so the first issue was how to dump the database from there. I do commits from the local folder, rather than SSH-ing into the VM, so mysqldump is not going to work without first getting into the VM. Which brought its own set of weirdnesses, and this was the point when I decided to flop over to WP-CLI, the WordPress command-line tool.

I often find solutions to this sort of thing are dependant on the combination of software and commands being used. I use mysqldump on its own all the time, but here, I needed to use Git to set the path for where the database would be dumped to — because git hooks are in a sub-directory of the git folder — and that, in combination with dumping the database inside the VM while within a Git command running from the local folder (yeah, probably should just do all my git via SSH), and hurling it at a remote server, means sometimes things that work in isolation get cranky. And this is a hack/bodge, so I went with:

  1. Set up paths for the database dump with Git, ’cos Git is running this show.
  2. SSH into the Vagrant box.
  3. WP-CLI dump the database to a gzipped file.
  4. SCP that up to the dev server.
  5. Delete all that on the local server, ’cos I’m tidy.

That’s half of it done. I’ve got my pushes working, the database file is up on the dev server, the local server is all cleaned up, so now it’s time for the other end.

In this case, I was doing it for a site on DreamHost, who conveniently give all kinds of fun command-line access, plus WP-CLI on their shared servers. Once Git has finished checking out the new file changes in post-receive, it’s time for frankly bodging it.

My current usual setup is a bare repository on the dev server, which checks out to the development website directory. This means neither the uploaded database, nor WP-CLI and the WordPress root are in the same place as the running hook. No big deal, just use –path=. The next thing though, is cleaning up post-import. Strings to be changed all over the place, like local URLs swapped to dev. And for that we have, wp search-replace, which is an awful lot like Search Replace DB. At the dev end then:

  1. Set up paths again, this time it’s WP-CLI running the show.
  2. Unzip the database then import it.
  3. Do database stuff like search-replace strings, and delete transients.
  4. Delete that uploaded database file on the dev server, ’cos I’m tidy.

I was looking at all this late last night, all those repeating lines of ‘wp search-replace’ and I thought, “That looks like a job for an array.” Which led me down the tunnel of Bash arrays, associative arrays, “How can I actually do ‘blah’, ’cos bash seems to be kinda unwilling here?” and finally settling on not quite what I wanted, but does the job. Also, bash syntax always looks like it’s cursing and swearing.

The pre-push hook:


# a pre-push hook to dump the database to a folder in the repo's root directory, upload it to the dev server, then delete when finished

echo '***************************************************************'
echo 'preparing to back up database'
echo '***************************************************************'

# set up some variables, to keep things more readable later on
# backup_dir is relative to git hooks, i.e. 2 directories higher, so use git to set it

ROOT="$(git rev-parse --show-toplevel)"

# check there is a database backup directory, make it if it doesn't exist then cd to it

if [ ! -d "$BACKUP_DIR" ]; then
mkdir "$BACKUP_DIR"

# cos this is vagrant, first ssh into it. there will be a password prompt
# using EOF to write the commands in bash, rather than in ssh quotation marks

ssh -t vagrant@ << EOF

# cd to the new databases folder. this is absolute, cos is vm and not local folder
cd "/var/www/user/domain.tld/.database" 

# then export the database with wp-cli and gzip it
wp db export --add-drop-table - | gzip -9 > $DB_NAME.sql.gz

# exit ssh

# bail out of eof

# scp the backup directory and database to dev server
scp -r $BACKUP_DIR user@domain.tld:~/

# remove that backup directory so it's not clogging up git changes

echo '***************************************************************'
echo 'all done, finishing up git push stuff'
echo '***************************************************************'

The post-receive hook:


echo '***************************************************************'
echo 'post-receive is working. checking out pushed changes.'
echo '***************************************************************'

# check out the received changes from local to the dev site
git --work-tree=/home/user/dev.domain.tld  --git-dir=/home/user/.repo.git checkout -f

# import the database with wp-cli
echo '***************************************************************'
echo 'starting database import'
echo '***************************************************************'

# setting up some paths
# on some webhosts, e.g. all-inkl, setting the alias to wp-cli.phar is required, uncomment and set if needed
# alias wp='/path/to/.wp-cli/wp-cli.phar'

# the path to wp-config, needed for wp-cli
# database directory, created in git pre-push

# check there is a database directory
if [ -d "$DB_DIR" ]; then

	# then check it for sql.gz files
	DB_COUNT=`ls -1 $DB_DIR/*.sql.gz 2>/dev/null | wc -l` 

	# if there is exactly 1 database, proceed
	if [ $DB_COUNT == 1 ]; then

		#grab the db name, this way the db name isn't hardcoded
		DB_NAME=$(basename $DB_DIR/*)

		echo 'importing the database'
		echo '***************************************************************'

		# unzip the database, then import it with wp-cli
		gunzip < $DB_DIR/$DB_NAME | wp db import - --path=$WP_PATH

		# clear the transients
		wp transient delete --all --path=$WP_PATH

		# run search replace on the main strings needing to be updated
		# make an array of strings to be searched for and replaced

		#loop through the array and spit it into wp search-replace
		for (( i=0; i < ${#search[@]}; ++i )); do
			eval wp search-replace --all-tables --precise \"${search[i]}\" \"${replace[i]}\" --path=$WP_PATH

		# any other wp-cli commands to run
		wp option update blogname "blog name" --path=$WP_PATH

		# delete the backup directory, so there's no junk lying around
		rm -rf $DB_DIR
		echo 'database was not found'
		echo '***************************************************************'


	echo 'database folder was not found'
	echo '***************************************************************'


echo '***************************************************************'
echo 'all done'
echo '***************************************************************'

What else? Dunno. It’s pretty rough, but basically proves something I didn’t find an example of all combined into one: that you can use git hooks to push the database and file changes at the same time, and automate the local-to-dev database transfer process. Is this the best way to do it? Nah, it’s majorly bodgy, and would have to be tailored for each server setup, and I’m not even sure doing such things in a git hook is advisable, even if it works. It does demonstrate that each step of the process can be automated — irrespective of how shonky your setup is — and provided you account for that and your own coding proclivities, there’s multiple ways of doing the same thing.

(edit, a day later.)
I decided to throw this into ‘production’, testing it on a development site I had to create on webhost I’m not so familiar with but who do provide the necessities (like SSH and Let’s Encrypt). Two things happened.

First, WP-CLI didn’t work at all in the post-receive script, even while it did if I ran commands directly in Terminal (or iTerm as I’m currently using). After much messing about, and trying a bunch of things it turned out that this was an issue of “has to be tailored for each server setup”, in this case adding an alias to wp-cli.phar.

Second, having a preference for over-compensation while automating, it occurred to me that I’d made some assumptions, like there’d only be one database file in the uploaded directory, and that hardcoding the filename — which was one of those “I’ll fix that later” things — had morphed into technical debt. So, feeling well competent in Bash today, I decided for the “make sure there’s actually a database folder, then check there’s actually a sql.gz file in it, and there’s only one of them, then get the name of that file, and use it as a variable”. I often wonder how much of this is too much, but trying to cover the more obvious possible bollocks seems reasonably sensible.

Both of these have been rolled into the code above. And as always, it occurs to me already there’s better — ‘better’ — ways to do this, like in pre-push, piping the database directly to the dev server with SSH, or simultaneously creating a separate, local database backup, or doing it all in SQL commands.

Git Hook + Vagrant + WP-CLI + Bash Local to Dev Database Transfer


Ars Electronica

Wednesday in Linz. I had to pick up some requirements for S.J Norman’s Rest Area and pick up Kali Rose from the train station, all the way from Amsterdam to be the person in the bed in the van of Rest Area. I had an hour to kill, and had planned Sunday for mediæval art (did not happen), so went to Ars Electronica.

I’m rethinking my museum-ing, or at least for the moment not taking hundreds of photos, editing and blogging scores. These are simply things I liked and felt motivated enough to photograph. There’s so much in the museum, and much of it is temporal, interactive, and 3-dimensional; photography doesn’t serve these well.

I sent “Your unreadable text message to +43 664 1788374” for Stefan Tiefengraber’s your unerasable text. My phone autocorrected. It was pushed to the shredder then sat there, unshredded. The pink of the Biolab was so, so, very hot, florescent, candy, neon pink, rendered as something less than all those and fuchsia by my camera. Markus Reibe’s Protected Areas 2 is the closest thing to recognisable, non-interactive, 2-dimensional art I saw, on a wall documenting the history of what was digital / new media / computer art, and what I just call art these days. The atrium of Ars Electronica was bus yellow and grey cement. If I had time, I would have spent hours here with hundreds of photos.

5-Character Dev Environment

Messing with my .bash_profile this afternoon, post-diving into Laravel and Git (which I’ve been doing much of the last week), I realised I could boot my entire dev environment with 5 letters. Fewer, if I wanted.

So instead of going to the Dock, clicking each of the icons, going to each and faffing around, I could at least boot them all, and set off some commands in Terminal (or ITerm2 as I’m now using).

Weirdly, until Justine gave me an evening of command-line Git learning, and wanted my .bash_profile, “Like so,” I hadn’t realised you could do stuff like that, despite amusing myself with all manner of shell scripts. Now I know what’s possible, I’m over-achieving in efficient laziness.

What’s missing is:

  • Opening multiple windows in ITerm or Terminal and running a different command in each (I don’t want to boot multiple instances of an app).
  • Setting off a menu action in an opened app, e.g. in Transmit going to my work drive.
  • Extending it to boot the environment and then a specific project, e.g. “devup laravel” would open my laravel installation in each of the apps, like opening the database in Sequel Pro; cd-ing to the laravel folder after automatic SSH-ing into my Vagrant box, and so on.

Some of these are probably uncomplicated, but this was a 30-minute experiment that turned out to be very useful.

5-character dev environment
5-character dev environment


Why, Yes, Chinese Grinding & Boring Equipment Company Spam Email, I am Interested.

Dear Lei Zhang of Fujian Nan’an Boreway Machinery Co., Ltd. in Shuitou Town, Nan’an City, Fujian Province, China. We have never met. But you emailed me on my blog email this morning. Your email was beautiful. I sadly have no use for boring and grinding heads as I sadly have no boring or grinding equipment. I wish I did. I saw the attached photograph of bush hammer rollers and thought to myself, “This is good and useful spam. All spam should be this good and useful. Imagine if it was, I would buy things.” So, thank you, Lei Zhang, you have done the unique. In all my years of receiving spam, in all those tens or hundreds of thousands of impersonal and unwanted emails, yours is the first that’s made me say, “I wish I had a need for this, I would buy immediately.” It’s true, if I had spare cash, I’d buy some bush hammer rollers just because they look brilliant — and I’ve never even seen them before. Because of your email, I learned something new, which is what is best in life. You must think I’m being sarcastic and mocking you, but I’m not. Yours is genuinely the best unsolicited email I’ve ever received.

Bush Hammer Rollers from Fujian Nan'an Boreway Machinery Co., Ltd.

Reading: Becky Chambers — A Closed and Common Orbit

I think I’m far too hard and cynical a person to be the audience of Becky Chambers’ novels, like them though I do. I wrote at length about her debut novel, The Long Way to a Small, Angry Planet and plenty of that holds true for A Closed and Common Orbit. I think this novel isn’t as successful though, perhaps because it alternates between only two characters and tried to build parallels between them that don’t really hold up.

The hard, cynic side of me also finds the general tenor of the characters flattened by a pervasive, apologetic niceness. There’s a scene early on where one of the two main characters, Pepper, at this time around ten-years old, escapes the slave scrap recycling plant she was born into and flees across the endless junkyard surface of the planet until stumbling by chance close enough to a destined-to-be-junked spacecraft she is rescued by the ship’s AI. So here’s a kid who’s obviously traumatised, dehydrated and malnourished — and we later learn the ship knows exactly what kind of planet and child this is — yet the AI spends pages before apologising for not flipping into emergency mode and doing triage, which the AI does not a little ineffectually. It’s a general over-caring niceness that ends up reading pathetic and monotonous, and grates against my “harden the fuck up” tendencies. Which may be my failure. “Always check the equipment for sensor error first.” As Iain Banks said.

Against me here, I wonder if the kind of world Chambers proposes is not a little of utopian, queer North American communities, and for people whose lives are made legible in such places, this novel might be really fulfilling to read, to see themselves represented in worlds which they yearn to live. And maybe if I’d been born 15 or 20 years later, coming of age in the LiveJournal and tumblr eras, I’d feel the same.

But I wasn’t.

But I like her novels enough to keep reading — even though I skipped a few pages out of boredom. I’d like to think she’s going to keep writing, have those glorious jumps in maturity and adroitness that happen to writers as they get a full handle on what they’re doing, cos for all my crapulous, old bitterness — which is going, “Frances, you’d fukkin hate being crew on their ship, haaate.” — I like reading her.

Becky Chambers — A Closed and Common Orbit
Becky Chambers — A Closed and Common Orbit

Bookmark Archaeology


I was cleaning out my browser bookmarks last night, first time in years, bookmarks going back to the early-’00s, thousands of them. I opened them in batches, every one, to see if I wanted to keep them. Hundreds, thousands of dead sites, no longer found, no longer existing. All that history and culture vanished as if it never was, only the link and title in my bookmarks proving they once existed, and once I deleted that …

Code Stupidity


I got sick of the tiny, Web1.0 images everywhere here, a hangover from the earliest days of supernaut, so I decided — ’cos I like visuality & pix — to make small, big. I thought it would be easy. Little did I know I also create and add to the pile of Technical Debt. So: most single images in the recent past are now huge-ified, 666px wide; recent image galleries which are not full of diverse image ratios are now evenly splitting the Number of the Beast. Older images and galleries should be retaining their previous diminutiveness, but who knows, 13 years of blog is difficult to homogenise. Mostly I got distracted with how to make portrait images not blow out of the available browser window space, which turns out to be a kinda traumatising process I didn’t achieve. Plus how to Lazy Load srcsets by preg_replacing the new WordPress caption shortcode. OMFG, Frances, WTF? All of which makes me think it might be time for yet another supernaut refresh. So much code. So many images. So much …

Reading: Becky Chambers — The Long Way to a Small, Angry Planet

It’s not like the days when Charlie Jane Anders was running io9 and her monthly roundup of all things skiffy getting published pretty much guaranteed at least one book I’d stick in my reading list — I suddenly realise I’ve gone off on a tangent here — but that monthly summary has returned or reinvigorated itself, and with the arrival of The Root and Fusion under the Gawker Gizmodo Media banner, I could hope that io9 might similarly get the love it deserves and be de-subdomained from, because it is one of the best sci-fi/fantasy/speculative fiction/etc websites around.

Which is a long way of saying I’m pretty sure I heard about Becky Chambers’ The Long Way to a Small, Angry Planet there, probably when it came out late-2015, but didn’t put it on my Must Eventually Buy list until a few months ago. I’m going through another phase of random experimentation with new writers, and she seemed to pass my rather strict interpretation of the Bechdel Test. And now I’ve read this, and yes, she does.

It’s a light read, in the sense that unlike say, Alasdair Reynold’s Revenger, we don’t have entire space ship crews annihilated just as we’ve begun to care for them, nor do the protagonists come out the other side morally terrifying. Almost all the story takes place on their moderately sized, ramshackle construction ship as they move ever core-wards in the galaxy. And the story, the actual story from which all those things we’re told are crucial come, narrative tension and arcs, conflict, and so on, all this is more like the background staging through which they move. What’s in fact the story is a group of individuals — well, for the most part individuals — let’s just say a small mob who we get to know as they live and work their daily lives.

I was thinking it owes something to Firefly, which is one of those series that’s either hugely pivotal in people’s sci-fi evolution, or entirely baffling. A more recent comparison might be Mass Effect. Either way, it owes a lot to fan fiction set in these universes. It also owes a lot to current critical discussions on identity — a word I’m very ambivalent about at the moment, and have been trying selfhood as a rickety replacement, not sure it’s much better, but the problem is with English (and English-influenced) language and its fixation on describing the world in a highly rigid manner going back to the Enlightenment — and you can’t easily think outside language.

In a lot of science-fiction set in the future — in writers who are actively trying to work through this stuff — I find that where we are currently around language, identity, selfhood, what constitutes personhood or a person, these massive discussions we’re having amongst ourselves and fighting against others who would deny us, are carried over into a future hundreds or thousands of years away. Or maybe it’s just a future where gender neutral ze / hir is used isn’t one I really aspire to. Perhaps also because this again proposes a future in which Anglo-American culture is dominant, something interestingly that Firefly tried to modulate with its use of Chinese language. And given English has a singular they (which is used in the novel), spoken Mandarin has nǐ, Cantonese has 佢 keoi5, Persian has او (yes, I’m imagining a future where Cantonese and Persian is in the galaxy), on and on, I feel like ze / hir is kinda redundant at best (plus I’m not a fan of Kate Bernstein). So on one hand I liked the novel and Chambers for working with this, and on the other, a far future where we’re still struggling with early-21st century identity is probably not a future we’d have survived to live in. Which is maybe to say, Chambers could be a lot more deliberate in thinking these ideas through to far more interesting and developed states.

Then I realise I haven’t said much about the story itself, like a review and all, where you get familiarised with a synopsis and a bit of who’s who. A crew of multi-planetary species mostly vaguely humanoid, one who I decided looks like a sloth, another a tardigrade with chin tentacles, another like Vastra the Silurian from Doctor Who, another who reminds me of Jewel the mechanic from Firefly, a ship artificial intelligence like Cortana from Halo (or pretty much any recent sci-fi with a ship A.I.); a hyperspace ship like a well-loved junkyard with modules and sections bolted on, one of which is a garden and kitchen, dining, hanging out area; the lives and relationships of this crew I both could imagine hanging out with and find their lack of boundaries a little off-putting. That’s not a review. You can find those everywhere. So, yes, despite my truculence, I read it and enjoyed it, enough I’ll read the sequel / offshoot A Closed and Common Orbit.

Becky Chambers — The Long Way to a Small, Angry Planet
Becky Chambers — The Long Way to a Small, Angry Planet