Cloudflare helper scripts for nginx and ufw

I’ve been making more use of Cloudflare as a DNS and proxy/DoS protection service. I tend to use it by default now when I set up any kind of public-facing web application that might attract non-trivial traffic, store sensitive information, or where performance is a factor.

When spinning up a new server on an infrastructure platform like Digital Ocean, it’s almost certain that the IP address assigned to it is going to immediately see traffic from bad actors using automated attempts to find a web-based exploit. Even if you put Cloudflare in front of a web service and lock things down, connection attempts directly to the IP address will bypass their proxy and get through.

That’s where this first script comes in handy. Instead of allowing all tcp network activity to ports 80 and 443 to get through, I want to only allow traffic from known Cloudflare IP addresses. We can do this using the Ubuntu ufw firewall and Cloudflare’s published IP address blocks.

GadElKareem shared a script that I’ve adapted a bit to make use of ufw’s application profiles:

#!/usr/bin/env bash

set -euo pipefail

# lock it
PIDFILE="/tmp/$(basename "${BASH_SOURCE[0]%.*}.pid")"
exec 200>${PIDFILE}
flock -n 200 || ( echo "${BASH_SOURCE[0]} script is already running. Aborting . ." && exit 1 )
PID=$$
echo ${PID} 1>&200

cd "$(dirname $(readlink -f "${BASH_SOURCE[0]}"))"
CUR_DIR="$(pwd)"

wget -q https://www.cloudflare.com/ips-v4 -O /tmp/cloudflare-ips-v4
wget -q https://www.cloudflare.com/ips-v6 -O /tmp/cloudflare-ips-v6

for cfip in `cat /tmp/cloudflare-ips-v4`; do /usr/sbin/ufw allow from $cfip to any app "Nginx Full" comment "cloudflare"; done
for cfip in `cat /tmp/cloudflare-ips-v6`; do /usr/sbin/ufw allow from $cfip to any app "Nginx Full" comment "cloudflare"; done

It basically downloads known Cloudflare IPv4 and IPv6 addresses and then adds ufw rules to allow traffic from those addresses. I run this on a cron job like so:

# Refresh cloudflare IPs in ufw
0 7 * * * /root/bin/cloudflare-only-web-ports-ufw.sh >/dev/null 2>&1

The final step is to remove any ufw rules that allow traffic through to ports 80 and 443 for any source IP. This kind of rule may or may not be in place as a part of your existing server configuration.

The end result is that any web connection attempts not from Cloudflare will not be allowed through.

A second challenge with Cloudflare is making sure anything you do with visitor IP addresses found in access logs is using the actual original visitor IP instead of the Cloudflare proxy IP. An example would be using fail2ban to block accesses from a host that has tried too many times to gain unauthorized access to a server.

Fortunately, Cloudflare passes through the original visitor IP as a header, so we just have to make use of that in our logs. This script, originally shared by ergin on GitHub, will download the published Cloudflare IP addresses and generate an nginx-friendly configuration file that adjusts the “real IP” of the visitor for logging and other purposes:

#!/bin/bash

CLOUDFLARE_FILE_PATH=/etc/nginx/conf.d/cloudflare-realips.conf

echo "#Cloudflare" > $CLOUDFLARE_FILE_PATH;
echo "" >> $CLOUDFLARE_FILE_PATH;

echo "# - IPv4" >> $CLOUDFLARE_FILE_PATH;
for i in `curl -s https://www.cloudflare.com/ips-v4/`; do
    echo "set_real_ip_from $i;" >> $CLOUDFLARE_FILE_PATH;
done

echo "" >> $CLOUDFLARE_FILE_PATH;
echo "# - IPv6" >> $CLOUDFLARE_FILE_PATH;
for i in `curl -s https://www.cloudflare.com/ips-v6/`; do
    echo "set_real_ip_from $i;" >> $CLOUDFLARE_FILE_PATH;
done

echo "" >> $CLOUDFLARE_FILE_PATH;
echo "real_ip_header CF-Connecting-IP;" >> $CLOUDFLARE_FILE_PATH;

#test configuration and reload nginx
nginx -t && systemctl reload nginx

I run this on cron like so:

# Refresh cloudflare IPs and reload nginx
30 4 * * * /root/bin/cloudflare-ip-whitelist-sync.sh >/dev/null 2>&1

Because the config file is output in the /conf.d/ directory in the nginx main directory, the default nginx config will pick up its contents without further action.

Continuing with the example of using fail2ban to block unwanted traffic from abusive hosts, you can set up a set of custom Cloudflare ban/unban actions in a file like /etc/fail2ban/action.d/cloudflare.conf, using this version adapted from the original by Mike Rushton:

[Definition]
actionstart =
actionstop =
actioncheck =

actionban = curl -s -X POST -H 'X-Auth-Email: <cfuser>' -H 'X-Auth-Key: <cftoken>' \
            -H 'Content-Type: application/json' -d '{ "mode": "block", "configuration": { "target": "ip", "value": "<ip>" } }' \
            https://api.cloudflare.com/client/v4/user/firewall/access_rules/rules

actionunban = curl -s -o /dev/null -X DELETE -H 'X-Auth-Email: <cfuser>' -H 'X-Auth-Key: <cftoken>' \
              https://api.cloudflare.com/client/v4/user/firewall/access_rules/rules/$(curl -s -X GET -H 'X-Auth-Email: <cfuser>' -H 'X-Auth-Key: <cftoken>' \
              'https://api.cloudflare.com/client/v4/user/firewall/access_rules/rules?mode=block&configuration_target=ip&configuration_value=<ip>&page=1&per_page=1' | tr -d '\n' | cut -d'"' -f6)

[Init]
cftoken =
cfuser =

and then in the jail.local file you can set this up as the default action:

cfemail=<your cloudflare@email here.com>
cfapikey=<your cloudflare API key here>

# Define CF ban action without mailing
action_cf = cloudflare[cfuser="%(cfemail)s", cftoken="%(cfapikey)s"]

# Set default action to Cloudflare ban/unban
action = %(action_cf)s

The end result is that bad actors never get past Cloudflare’s proxy protection to attempt additional foolishness directly on your server.

Thanks to all of the folks who created the original version of these scripts. If you have suggestions for improvement or your own fun Cloudflare tooling, please share!

What are AWS hosting costs using Laravel Vapor?

When I was researching tools and services for launching a SaaS app, I was pretty clear that I wanted to use Laravel Vapor to manage the Amazon Web Services deployment. The main mystery about that decision was what it would actually cost to have a Vapor-managed deployment on AWS for the size of my application and my expected usage levels.

I found a few articles and blog posts about that topic (the most helpful was Cost & Performance optimization in Laravel Vapor) but, as is the case with AWS hosting in general, there was no clear formula that would lead me to a precise monthly hosting cost for a brand new web application.

In hopes that it helps someone else in a similar situation, I’d like to add one more data point to the mix. Here are some details about what it’s costing me (so far) to host my Laravel-powered application on AWS as managed by Vapor.

Vapor itself is $39/month. This cost does not change if you use Vapor across multiple projects, so your per-project cost can go down over time if you plan to launch more than one project. Some people have raised eyebrows at this baseline cost but as anyone who has ever had to manage their own hosting server infrastructure and worry about upgrades, security issues, configuration management, etc. knows, it feels like a great deal. I wrote about that more in my other post on launching a SaaS business, but this sentiment remains true:

It felt like the magical world of cloud hosting that was always promised but never quite delivered had finally become reality. Even a month later I’m still constantly amazed by it. Huge kudos to the Vapor team.

Now, on to AWS itself. I’m currently paying approximately $1.00 per day for AWS services, and the monthly bill ends up being about $33.00.

Continue reading What are AWS hosting costs using Laravel Vapor?

Cloning a WP Engine site to a local VVV development environment

When doing custom WordPress theme and plugin development for a site that’s already launched, I find it is essential to get my local development environment set up as close as possible to the live environment where the site is hosted. This minimizes headaches and unexpected problems that come when making any updated code live.

Here are the steps I use to clone the database, theme, plugin and media/uploads from a site hosted on WP Engine into a locally-hosted development environment powered by VVV. This method allows me to develop and test new functionality against recent site content before I deploy it to a staging environment for final testing.

The usual warning applies: you should make sure you know what each of these steps is actually doing and adjust them to fit your specific setup. Some of these actions could result in data loss, so please use them at your own risk.

Okay, here we go:

  1. Make sure any custom theme and plugin code you’ll be working with is under source control, probably using these instructions from WP Engine, and up to date locally. Make sure that those directories are not accidentally included in the .gitignore file for your repo. (I handle this by default ignoring everything in, say, the plugins directory with /wp-content/plugins/* and then adding exclusion rules for each directory I’m working on, e.g. !/wp-content/plugins/my-custom-plugin*.)
  2. Initiate a backup checkpoint on the environment you want to clone. If you’re okay using the slightly out of date one that WP Engine automatically makes every morning, that’s fine.
  3. Initiate a download of the backup checkpoint, choosing the “partial” option and checking “Entire database,” “Plugins” and “Themes.” Leave “Media uploads” and “Everything else” unchecked.
  4. Use the WP Engine SSH gateway to rsync the media folder from the WP Engine environment to your local environment. Here’s an example command for a “mystaging” environment…you will want to try your version with the dry run -n flag first to make sure it is going to do what you want!
    rsync -n -rlD -vz --size-only --delete mystaging:sites/mystaging/wp-content/uploads/ /Users/chris/vvv/www/mysite/public_html/wp-content/uploads/
    

    This should delete any local media files not in the WP Engine environment, and copy everything else down that isn’t already there. It could take a while if it’s the first time you’re running it, but subsequent runs should only grab new uploads to the site.

  5. Unzip the downloaded backup checkpoint on your local system.
  6. Move the mysql.sql file into a directory that will be accessible from your VVV Vagrant shell: mv ~/Downloads/wp-content/mysql.sql /Users/chris/vvv/www/mysite/
  7. Update your local plugins directory from the downloaded backup. Again, please use the -n flag with your version of this command first to preview the results:
    rsync -rlD -vz --size-only --delete --exclude=query-monitor ~/Downloads/wp-content/plugins/ /Users/chris/vvv/www/mysite/public_html/wp-content/plugins/
    

    Note that this command also excludes removing the query-monitor plugin, which I have installed locally but not in the WP Engine environment. That way it can stay ready to activate without reinstalling.

  8. Repeat the above step for the themes directory if need be.
  9. Resolve any discrepancies, things git notes as changes that need to be committed, missing files, etc. There shouldn’t be any but it’s worth checking.
  10. SSH into your VVV box: vagrant ssh
  11. Make sure your WP dev site and the WP Engine site are running the same version of WordPress core.
  12. Go to the directory where the site lives, e.g. cd /srv/www/mysite/public_html/.
  13. Clear out the existing database (again, assuming you don’t have any locally staged changed that you would care about): wp db reset.
  14. Import the database from the WP Engine environment: wp db import ../mysql.sql
  15. Update any content or configuration references to the site’s hostname, noting that you may want to run this with the --dry-run flag first: wp search-replace '//mystaging.com' '//mysite.test' --precise --recurse-objects --all-tables --skip-columns=guid.
  16. If you need to put any of your plugins in local development mode, now’s the time, e.g. wp config set JETPACK_DEV_DEBUG true --raw. And, if you  were using a local development plugin like Query Monitor, reactivate it: wp plugin install query-monitor --activate

That’s it! In only 16 steps you should have a fully up to date copy of your site’s data available for local development and testing. Obviously much of the above could be scripted (with appropriate safeguards) to save time and reduce human error in the process.

I hope you find this helpful. If you have improvements to suggest or your own fun methods of cloning environments, please share in the comments.

WP Engine is a great web host for WordPress developers

I’ve been aware of WP Engine’s WordPress hosting offerings for quite a while now, but I only recently had a chance to dive deeply into the features and benefits they offer to WordPress developers, and I was really impressed.

(Disclaimer: I am not affiliated with WP Engine and am not being compensated by them in any way for this review. But this post contains some referral links where I may receive a small percentage of any sales that result from readers clicking through, and where readers may receive a discount on their purchase.)

Some of the things about WP Engine that stood out to me as really helpful and awesome for WordPress developers:

  1. Super-fast, comprehensive site backup snapshots and cloning. The ability to quickly make a copy of an entire production site (with a large DB and tons of media) to a staging version of that site, or just to a backup snapshot, is a huge benefit. Being able to do it at a click of a button without messing around with export/import tools, find-replace operations or similar command line intervention is just awesome, and enables all sorts of other development best practices when it comes to testing changes and having a safety net for production updates. It’s SO fast, usually completing within a minute or two, so you can make backups/clones all day long without delay. It’s better than any other site backup or environment cloning tool I’ve used in the WordPress hosting space.
  2. Deep integration with git repo management. Although the instructions and interface for setting it up needs a little expanding and polishing, the WP Engine makes it really easy to set up a git repo for a given hosting environment, where changes pushed to its main branch are quickly deployed to the associated environment. They’ve thought through the complexities of exclusions and co-existing with WordPress-initiated core/plugin/theme updates. Add in the GitHub Action to deploy to a WP Engine environment and you’ve got a really sweet development and deployment pipeline setup, all using industry best practices.
  3. Fast and powerful SSH command line access, optimized for security and WP CLI operations. WP Engine seems to understand that command line operations are an essential tool in a WordPress developer or site manager’s toolkit, and they make it really straightforward to use.
  4. Robust system status monitoring and reporting. Whereas some hosts update their system status page well after an impacting event, WP Engine seems to have theirs wired up to show a closer-to-realtime status, and that makes all the difference in not wasting time when troubleshooting or reacting to problems. I also really appreciate that they offer email, Slack and webhook-based notifications for status events, offering endless possibilities with integrating platform events into your development tools and workflows.
  5. Thoughtful tools for keeping WordPress current and secure. WP Engine clearly understands the importance of keeping WordPress core up to date and making sure no insecure plugins or themes are in place any longer than is absolutely necessary. While I think responsibility for these things generally falls to a developer and not the host, I appreciate that they’ve invested in infrastructure here, and I’m sure it benefits them and their support operations in the long run too.
  6. Great support, great communication. Whenever I’ve used the WP Engine support chat they’ve been fast, knowledgeable and straight to the point without being curt. If a question or issue needs input from another internal team, they seem to be able to do that quickly and without any resistance. Their documentation is generally well-written and organized.

In the project I was working on where I finally got to see these features directly in action, I had evaluated a variety of hosts including SiteGround, Pressable, WordPress.com Business, and WP Engine. I picked WP Engine for the above reasons and others, including their focus on WordPress-specific performance optimization.

To be clear, I’m not saying WP Engine is the best WordPress host for every use case, or even most use cases out there. Whether you’re a non-technical WordPress site owner looking for something simple and low-cost, or an enterprise-level site needing something that scales for Superbowl-level traffic with commensurate high-touch support, there are lots of great options out there that might be a better fit. (Having been a part of Automattic/WordPress.com/WordPress.com VIP and seeing the incredible investment in scalable infrastructure there, I know the details really matter at those different ends of the spectrum; I still frequently recommend their offerings too.)

But for a WordPress developer or small development team deploying custom theme and plugin code to a high-traffic site and wanting great WordPress-specific tools, systems and people to support them in that, WP Engine really stands out as worth a look.