Getting alerts about outdated Time Machine backups

Our household uses Apple’s Time Machine backup system as one of our backup methods. Various Macs are set to run their backups at night (we use TimeMachineEditor to schedule backups so they don’t take up local bandwidth or computing resources during the day) and everything goes to a Synology NAS (basically following the steps in this great guide from 9to5Mac).

But, Time Machine doesn’t necessarily complete a successful backup for every machine every night. Maybe the Mac is turned off or traveling away from home. Maybe the backup was interrupted for some reason. Maybe the backup destination isn’t on or working correctly. Most of these conditions can self-correct within a day or two.

But, I wanted to have a way to be notified if any given Mac had not successfully completed a backup after a few days so that I could take a look. If, say, three days go by without a successful backup, I start to get nervous. I also didn’t want to rely on any given Mac’s human owner/user having to notice this issue and remember to tell me about it.

I decided to handle this by having a local script on each machine send a daily backup status to a centralized database, accomplished through a new endpoint on my custom webhook server. The webhook call stores the status update locally in a database table. Another script runs daily to make sure every machine is current, and sends a warning if anyone is running behind.

Here are the details in case it helps anyone else.

Continue reading Getting alerts about outdated Time Machine backups

Personal banking needs an API

My washer and dryer can tell my smart watch when they are done washing and drying. A voice assistant in my kitchen can update my grocery list. Documents in selected folders on my laptop can be synced to data centers around the world in an instant.

These things are possible through the use of APIs, which most every user-facing service, tool, device and ecosystem out there these days seems to understand are an essential part of their offering. APIs give users, developers and partners a way to build new things on top of the thing you already offer. They give people flexibility to integrate a service, tool or device into their lives in a way that makes sense for them. APIs help encourage wide adoption and extensibility.

The industry that seems to be far behind in offering powerful APIs to end users? Personal banking, and related billing systems for utilities and credit card companies.

When I want to check the current balance on my personal checking account, I have to follow a multi-step process in a web browser or mobile app.

When I want to get the latest PDF copy of a bill from my mobile phone carrier, it’s something like 10 clicks across three different websites.

When I want to initiate a bill payment on a credit card provider that doesn’t support automatic drafting from my bank, it’s a similarly long process.

The other day a rep from a utility told me I had to call them to request a form be mailed to me so I could fill it out and mail it back to them, just in order to set up automatic payments from a bank account.

And when I want to be notified about certain kinds of activity from these institutions, I have to log in to each one to go through their proprietary grid of checkboxes and verification methods to set up push alerts or text messages…if they offer notifications at all.

Indeed, accessing and working with my personal financial information is one of the most cumbersome, high-friction, analog things I do any more. Personal banking and bill payment feels like swimming in mud compared to the light speed of most of the rest of the information economy.

Why is there so much friction in personal banking and financial transactions?

Continue reading Personal banking needs an API

How long does it take between when a plugin update is released and when auto-updates install it on your WordPress site?

Auto-updates for WordPress themes and plugins were released this year in WordPress version 5.5. They allow WordPress site owners to opt-in to automatically have new versions of plugins and themes installed when they are released, without any intervention from the site owner.

If you use auto-updates, one question might be on your mind:

How long will it take between when the author of a plugin releases a new version and when that new version is installed on your WordPress site?

This question is vital for site owners and managers. Especially in scenarios when new plugin or theme versions contain critical security fixes, time is of the essence to avoid possible unauthorized access to your WordPress site.

To get to the answer, let’s first review how plugin and theme releases happen.

The Plugin and Theme Release Process

When a plugin or theme author is ready to make an update to their software, they upload those changes to the directory on WordPress.org. This is where the code for their theme or plugin is hosted publicly.

Most theme and plugin authors also indicate the release of non-trivial changes by increasing the version number associated with their plugin. Maybe it’s a small “point release” like going from version 1.1 to version 1.2, or maybe it’s a major release like going from version 3.0 to version 4.0. The change in version number lets everyone know that there’s new functionality and fixes available. It’s a convenient way to refer to how software has changed over time.

Once the updated software and version number change is live on WordPress.org, it’s immediately in effect for new installations of that plugin or theme. Anyone downloading and installing a plugin or theme from that directory will now be using the latest code made available by the author.

But what about existing sites that already have that theme or plugin installed? How do they learn about the new changes and new version?

How WordPress Sites Discover Updates

You might think it happens through a “push notification” sent to your site from WordPress.org. But the WordPress.org systems would have to contact thousands or maybe millions of sites to tell them about an update to a single plugin. That’s just not practical.

Continue reading How long does it take between when a plugin update is released and when auto-updates install it on your WordPress site?

How I built and launched a SaaS app in one month

In June of this year, I had an idea for a software-as-a-service (Saas) application that I wanted to build and launch. On July 1st, I committed the first lines of code to a git repo for the app. After about 200 hours of development time, on August 10th I launched the first version of the application into the world, ready for users and subscribers.

The service is called WP Lookout and you can read more about the service itself and why I built it on wplookout.com. This post is about the process, tools and services I used to build and launch a SaaS application in what felt like a relatively short period of time.

Here’s the short list of resources I used along the way if you just want to explore them without further commentary:

  • Laravel for the application framework
  • Laravel Homestead for my local development environment
  • PhpStorm for software development
  • Laravel Spark for scaffolding the SaaS subscription and account management
  • Stripe for subscription payment processing
  • Laravel Nova for an administrative dashboard
  • Git and GitHub for tracking code development, to-do items and feature branches
  • Upwork for hiring and paying a software code reviewer and consultant
  • Slack for coordinating with my contractor
  • TermsFeed for generating privacy policy and terms of service documents
  • Amazon Web Services for application hosting
  • Laravel Vapor for managing AWS setup and deployment
  • WordPress for building the WP Lookout marketing website
  • Matomo for privacy-focused analytics
  • HelpScout for managing user support interactions
  • MailerLite for handling new user onboarding and marketing automation

I’m not going to go in-depth on all of these tools, as some of them are pretty simple and self-evident in their value. Others are just magical and deserving of some additional observations. Here are some things that stood out and what I learned along the way:

Continue reading How I built and launched a SaaS app in one month

Cloning a WP Engine site to a local VVV development environment

When doing custom WordPress theme and plugin development for a site that’s already launched, I find it is essential to get my local development environment set up as close as possible to the live environment where the site is hosted. This minimizes headaches and unexpected problems that come when making any updated code live.

Here are the steps I use to clone the database, theme, plugin and media/uploads from a site hosted on WP Engine into a locally-hosted development environment powered by VVV. This method allows me to develop and test new functionality against recent site content before I deploy it to a staging environment for final testing.

The usual warning applies: you should make sure you know what each of these steps is actually doing and adjust them to fit your specific setup. Some of these actions could result in data loss, so please use them at your own risk.

Okay, here we go:

  1. Make sure any custom theme and plugin code you’ll be working with is under source control, probably using these instructions from WP Engine, and up to date locally. Make sure that those directories are not accidentally included in the .gitignore file for your repo. (I handle this by default ignoring everything in, say, the plugins directory with /wp-content/plugins/* and then adding exclusion rules for each directory I’m working on, e.g. !/wp-content/plugins/my-custom-plugin*.)
  2. Initiate a backup checkpoint on the environment you want to clone. If you’re okay using the slightly out of date one that WP Engine automatically makes every morning, that’s fine.
  3. Initiate a download of the backup checkpoint, choosing the “partial” option and checking “Entire database,” “Plugins” and “Themes.” Leave “Media uploads” and “Everything else” unchecked.
  4. Use the WP Engine SSH gateway to rsync the media folder from the WP Engine environment to your local environment. Here’s an example command for a “mystaging” environment…you will want to try your version with the dry run -n flag first to make sure it is going to do what you want!
    rsync -n -rlD -vz --size-only --delete mystaging:sites/mystaging/wp-content/uploads/ /Users/chris/vvv/www/mysite/public_html/wp-content/uploads/
    

    This should delete any local media files not in the WP Engine environment, and copy everything else down that isn’t already there. It could take a while if it’s the first time you’re running it, but subsequent runs should only grab new uploads to the site.

  5. Unzip the downloaded backup checkpoint on your local system.
  6. Move the mysql.sql file into a directory that will be accessible from your VVV Vagrant shell: mv ~/Downloads/wp-content/mysql.sql /Users/chris/vvv/www/mysite/
  7. Update your local plugins directory from the downloaded backup. Again, please use the -n flag with your version of this command first to preview the results:
    rsync -rlD -vz --size-only --delete --exclude=query-monitor ~/Downloads/wp-content/plugins/ /Users/chris/vvv/www/mysite/public_html/wp-content/plugins/
    

    Note that this command also excludes removing the query-monitor plugin, which I have installed locally but not in the WP Engine environment. That way it can stay ready to activate without reinstalling.

  8. Repeat the above step for the themes directory if need be.
  9. Resolve any discrepancies, things git notes as changes that need to be committed, missing files, etc. There shouldn’t be any but it’s worth checking.
  10. SSH into your VVV box: vagrant ssh
  11. Make sure your WP dev site and the WP Engine site are running the same version of WordPress core.
  12. Go to the directory where the site lives, e.g. cd /srv/www/mysite/public_html/.
  13. Clear out the existing database (again, assuming you don’t have any locally staged changed that you would care about): wp db reset.
  14. Import the database from the WP Engine environment: wp db import ../mysql.sql
  15. Update any content or configuration references to the site’s hostname, noting that you may want to run this with the --dry-run flag first: wp search-replace '//mystaging.com' '//mysite.test' --precise --recurse-objects --all-tables --skip-columns=guid.
  16. If you need to put any of your plugins in local development mode, now’s the time, e.g. wp config set JETPACK_DEV_DEBUG true --raw. And, if you  were using a local development plugin like Query Monitor, reactivate it: wp plugin install query-monitor --activate

That’s it! In only 16 steps you should have a fully up to date copy of your site’s data available for local development and testing. Obviously much of the above could be scripted (with appropriate safeguards) to save time and reduce human error in the process.

I hope you find this helpful. If you have improvements to suggest or your own fun methods of cloning environments, please share in the comments.

Generate an RSS feed from a Twitter user timeline

I needed to generate a valid RSS feed from a Twitter user’s timeline, but only for tweets that matched a certain pattern. Here’s how I did it using PHP.

First, I added the dependency on the TwitterOAuth library by Abraham Williams:

$ composer require abraham/twitteroauth

This library will handle all of my communication and authentication with Twitter’s API. Then I created a read-only app in my Twitter account and securely noted the four key authentication items I would need, the consumer API token and secret, and the access token and secret.

Now, I can quickly bring recent tweets from my target Twitter user into a PHP variable:

require "/path/to/vendor/autoload.php" ;
use Abraham\TwitterOAuth\TwitterOAuth;

$consumerKey       = "your_key_goes_here"; // Consumer Key
$consumerSecret    = "your_secret_goes_here"; // Consumer Secret
$accessToken       = "your_token_goes_here"; // Access Token
$accessTokenSecret = "your_token_secret_goes_here"; // Access Token Secret

$twitter_username    = 'wearrrichmond';

$connection = new TwitterOAuth( $consumerKey, $consumerSecret, $accessToken, $accessTokenSecret );

// Get the 10 most recent tweets from our target user, excluding replies and retweets
$statuses = $connection->get(
	'statuses/user_timeline',
	array(
		"count" => 10,
		"exclude_replies" => true,
		'include_rts' => false,
		'screen_name' => $twitter_username,
		'tweet_mode' => 'extended',
	)
);

My specific use case is that my local public school system doesn’t publish an RSS feed of news updates on its website, but it does tweet those updates with a somewhat standard pattern: the headline of the announcement, possibly followed by an at-mention and/or image, and then including a link back to a PDF file on their website that lives in a certain directory. I wanted to capture these items for use on another site I created to aggregate local news headlines into one place, and it mostly relies on the presence of an RSS feed.

So, I only want to use the tweets that match this pattern in the custom RSS feed. Here’s that snippet:

// For each tweet returned by the API, loop through them
foreach ( $statuses as $tweet ) {

	$permalink = '';
	$title     = '';

	// We only want tweets with URLs
	if ( ! empty( $tweet->entities->urls ) ) {

		// Look for a usable permalink that matches our desired URL pattern, and use the last (or maybe only) one
		foreach ( $tweet->entities->urls as $url ) {

			if ( false !== strpos( $url->expanded_url, 'rcs.k12.in.us/files', 0 ) ) {
				$permalink = $url->expanded_url;

			}
		}

		// If we got a usable permalink, go ahead and fill out the rest of the RSS item
		if ( ! empty( $permalink ) ) {

			// Set the title value from the Tweet text
			$title = $tweet->full_text;

			// Remove links
			$title = preg_replace( '/\bhttp.*\b/', '', $title );

			// Remove at-mentions
			$title = preg_replace( '/\@\w+\b/', '', $title );

			// Remove whitespace at beginning and end
			$title = trim( $title );

			// TODO: Add the item to the feed here

		}
	}
}

Now we have just the tweets we want, ready to add to an RSS feed. We can use the included SimplePie library to do this. In my case, the final output is written to an output text file, which another part of my workflow can then query.

Here’s the final result all put together:

<?php

/**
 * Generate an RSS feed from a Twitter user's timeline
 * Chris Hardie <chris@chrishardie.com>
 */

require "/path/to/vendor/autoload.php" ;
use Abraham\TwitterOAuth\TwitterOAuth;

$consumerKey       = "your_key_goes_here"; // Consumer Key
$consumerSecret    = "your_secret_goes_here"; // Consumer Secret
$accessToken       = "your_token_goes_here"; // Access Token
$accessTokenSecret = "your_token_secret_goes_here"; // Access Token Secret

$twitter_username    = 'wearrrichmond';
$rss_output_filename = '/path/to/www/rcs-twitter.rss';

$connection = new TwitterOAuth( $consumerKey, $consumerSecret, $accessToken, $accessTokenSecret );

// Get the 10 most recent tweets from our target user, excluding replies and retweets
$statuses = $connection->get(
	'statuses/user_timeline',
	array(
		"count" => 10,
		"exclude_replies" => true,
		'include_rts' => false,
		'screen_name' => $twitter_username,
                'tweet_mode' => 'extended',
	)
);

$xml = new SimpleXMLElement( '<rss/>' );
$xml->addAttribute( 'version', '2.0' );
$channel = $xml->addChild( 'channel' );

$channel->addChild( 'title', 'Richmond Community Schools' );
$channel->addChild( 'link', 'http://www.rcs.k12.in.us/' );
$channel->addChild( 'description', 'Richmond Community Schools' );
$channel->addChild( 'language', 'en-us' );

// For each tweet returned by the API, loop through them
foreach ( $statuses as $tweet ) {

	$permalink = '';
	$title     = '';

	// We only want tweets with URLs
	if ( ! empty( $tweet->entities->urls ) ) {

		// Look for a usable permalink that matches our desired URL pattern, and use the last (or maybe only) one
		foreach ( $tweet->entities->urls as $url ) {

			if ( false !== strpos( $url->expanded_url, 'rcs.k12.in.us/files', 0 ) ) {
				$permalink = $url->expanded_url;

			}
		}

		// If we got a usable permalink, go ahead and fill out the rest of the RSS item
		if ( ! empty( $permalink ) ) {

			// Set the title value from the Tweet text
			$title = $tweet->full_text;

			// Remove links
			$title = preg_replace( '/\bhttp.*\b/', '', $title );

			// Remove at-mentions
			$title = preg_replace( '/\@\w+\b/', '', $title );

			// Remove whitespace at beginning and end
			$title = trim( $title );

			$item = $channel->addChild( 'item' );
			$item->addChild( 'link', $permalink );
			$item->addChild( 'pubDate', date( 'r', strtotime( $tweet->created_at ) ) );
			$item->addChild( 'title', $title );

			// For the description, include both the original Tweet text and a full link to the Tweet itself
			$item->addChild( 'description', $tweet->full_text . PHP_EOL . 'https://twitter.com/' . $twitter_username . '/status/' . $tweet->id_str . PHP_EOL );

		}
	}
}

$rss_file = fopen( $rss_output_filename, 'w' ) or die ("Unable to open $rss_output_filename!" );
fwrite( $rss_file, $xml->asXML() );
fclose( $rss_file );

exit;

Here’s the same thing as a gist on GitHub.

I set this script up to run via cronjob every hour, which gives me a regularly updated feed based on the Twitter account’s activity.

Several ways this could be improved include:

  • Better escaping and sanitizing of the data that comes back from Twitter
  • Make the filtering of the Tweets more tolerant to changes in the target user’s Tweet structure
  • Genericizing the functionality to support querying multiple Twitter accounts and generating multiple corresponding output feeds
  • Fixing Twitter so that RSS feeds of user timelines are offered on the platform again

If you find this helpful or have a variation on this concept that you use, let me know in the comments!

Updated July 14 2020 to add the use of “tweet_mode = extended” in the API connection and to replace the references to tweet text with “full_text”, as apparently it is not the default in the Twitter API to use the 240-character version of tweets.

Running WordPress cron on a multisite instance

For a long time I used the WP Cron Control plugin and an associated cron job to make sure that scheduled actions on my WordPress multisite instance were executed properly. (You should never rely on event execution that is triggered by visits to your website, the WordPress default, IMHO.) But after the upgrade to WordPress 5.4 I noticed that some of my scheduled events in WordPress were not firing on time, sometimes delayed by 10-20 minutes. I did some troubleshooting and got as far as suspecting a weird interaction between that plugin and WordPress 5.4, but never got to the bottom of it.

When I reluctantly went in search of a new solution, I decided to try using WP CLI cron commands, executed via my server’s own cron service. Ryan Hellyer provided most of what I needed in this helpful post, and I extended it a bit for my own purposes.

Here’s the resulting script that I use:

#!/bin/bash

# This script runs all due events on every WordPress site in a Multisite install, using WP CLI.
# To be run by the "www-data" user every minute or so.
#
# Thanks https://geek.hellyer.kiwi/2017/01/29/using-wp-cli-run-cron-jobs-multisite-networks/

PATH_TO_WORDPRESS="/path/to/wordpress"
DEBUG=false
DEBUG_LOG=/var/log/wp-cron

if [ "$DEBUG" = true ]; then
        echo $(date -u) "Running WP Cron for all sites." >> $DEBUG_LOG
fi

for URL in = $(wp site list --field=url --path="/path/to/wordpress" --deleted=0 --archived=0)
do
        if [[ $URL == "http"* ]]; then
                if [ "$DEBUG" = true ]; then
                        echo $(date -u) "Running WP Cron for $URL:" >> $DEBUG_LOG
                        wp cron event run --due-now --url="$URL" --path="$PATH_TO_WORDPRESS" >> $DEBUG_LOG
                else
                        wp cron event run --quiet --due-now --url="$URL" --path="$PATH_TO_WORDPRESS"
                fi
        fi
done

Then, in my system crontab:

# Run WordPress Cron For All Sites
*/2 * * * * www-data /bin/bash /path/to/bin/run-wp-cli-cron-for-sites.bash

Yes, I run cron every 2 minutes; there are some sites I operate that require very precise execution times in order to be useful. One implication is that this solution does not scale up very well; if the total execution time of all cron jobs across all sites exceeds 2 minutes, I could quickly run into situations where duplicate jobs are running trying to do the same thing, and that could be bad for performance or worse.

Generic ‘send to Slack’ shell script

On any given server I maintain, I like to set up a generic “send a message to Slack” shell script that can be called from any other tool or service running on that machine. With it I can log information of interest to a Slack channel for reading or maybe action.

Here’s what send-to-slack.sh usually looks like:

#!/bin/bash -e

message=$1

[ ! -z "$message" ] && curl -X POST -H 'Content-type: application/json' --data "{
              \"text\": \"${message}\"
      }" https://hooks.slack.com/services/12345/67890/abcdefghijklmnop

That last line has the “incoming webhook URL” provided by Slack when you set Slack up to receive messages via incoming webhooks, something that is included in even their most basic/free tier.

Running the script and sending a message to the channel is as simple as $ sh send-to-slack.sh 'My message goes here' and the result is what you would expect:

Once that’s in place and tested, I can call the script from wherever I want on that server. Other shell scripts. Custom WordPress functions. Cron jobs. And so on.

There are many other ways this could be customized or extended. It’s worth noting that this is not necessarily a fully secure way to do things if you have untrusted users who can control the input to the script and the message that gets output…please remember to sanitize your inputs and escape your outputs!

 

Put all those email newsletters in an RSS feed

The other day someone told me that they think blogging is dead.

I tried to suppress the sounds of existential pain emanating from deep within my soul, but it still hurt.

Blogging is far from dead, but I also recognize that email newsletters are all the hotness right now when it comes to getting your written thoughts in front of someone else. And I recognize that if you want to follow some kinds of updates from some kinds of people or organizations, you’re going to have to do the email thing.

For a while, I used email filters to manage this issue, dutifully creating or updating them in my setup each time I cringe-fully subscribed to a new email newsletter list after searching in vain for an RSS feed subscribe button. Then I would let them all go into just the right email folder (or maybe even still my inbox) so I could read them when I was in the mood to read blog-posts-as-email-messages on a given subject.

Ugh.

I didn’t like that this approach still created a kind of additional email “to do” burden on me, leaving me with folders to sort, search through and clear out. Newsletter content is usually not actionable or time-sensitive. What I really wanted was to treat all those email newsletter messages like blog post headlines in a separate kind of reader app, available to be read at my leisure. YOU KNOW, LIKE AN RSS FEED READER.

So here’s my current setup:

  1. newsletter emails go to a dedicated email alias configured at my mail provider, and that’s what I subscribe to lists with
  2. those messages are forwarded to a Zapier-powered recipe that converts them into items on a custom generated RSS feed
  3. I subscribe to the RSS feed in my feed reader, Feedly.

Now I can browse the headlines when I want to, read some items and gloss over others, and my email inbox is no longer crowded with articles that aren’t necessarily actionable or time-sensitive for me.

A few things I could do to tweak this setup further:

  • Right now all of my email newsletters go into a single RSS feed. For better categorization and readability, I could break these out into individual feeds.
  • The translation of HTML-only emails (another annoying thing about the email newsletter age) doesn’t always work well into the RSS feed format as supported by Zapier. I haven’t really explored a fix for this but it hasn’t affected me much so far.

Also note that Zapier’s pricing structure is such that depending on the number of incoming messages you have, you might need to upgrade to a paid plan.

That’s it. My email inbox has benefitted greatly from this setup, and I hope yours will too.

New WordPress and WooCommerce plugin: Harmonizely Booking Product

I’ve released a new, free plugin for WordPress and WooCommerce, Harmonizely Booking Product. The plugin creates a new WooCommerce product type that allows you to sell access to scheduled appointments on your calendar, using Harmonizely.

Here’s a quick video to show you how it works:

Disclaimer: I am not affiliated with Harmonizely and they did not ask or pay me to create this plugin, I’m just a fan of the service who wanted to create more ways to use it within the WordPress ecosystem. This post does contain some referral links where I may receive a small percentage of any sales that might result from readers clicking through.

There are a growing number of options to handle appointment scheduling, and if you’re in some field where people schedule things with you a lot (consultant, agency, counselor, accountant, lawyer, healthcare professional) I hope you’re looking at those tools to save you some time. One of the main reasons I like and settled on Harmonizely is because they support the open CalDAV standard for calendar connections and syncing, where as many other services only support Google Calendar or other proprietary connections. (This is especially important to me as a part of advocating for an open web.)

I also like Harmonizely because the service is simple and fast, they regularly release improvements and new features, they have a small and responsive team, and they’ve made their product roadmap public and interactive. Their basic tool is free and they have very affordable pricing for an upgraded version.

Creating this plugin to work with WooCommerce means that anyone who has an existing WooCommerce-powered store can add booking functionality in and keep using their existing payment methods, plugins and other settings. I can imagine a content creator who already sells access to video courses or other educational resources might enjoy being able to let users schedule a quick call with them for a small fee, too. Or maybe someone who offers troubleshooting services of some kind can now give their customers a quick way to pay for and schedule an appointment. There are lots of possibilities, and WooCommerce offers tons of flexibility so you can integrate with Stripe, Paypal, Square and other payment processors.

If you want to sell access to your time through a website, I hope you’ll take a look at Harmonizely, WooCommerce, and this new Harmonizely Booking Product plugin. If you have questions or need help, you can submit a support message or open a GitHub issue.

Enjoy!