Tag: RSS

An example of why RSS is useful and important

Let’s say I want to be an informed, engaged local citizen. Let’s say my town has a website set up where they post local updates and alerts that I might care about.

Specifically, let’s say there was a water main break a few days ago that led to everyone in the town being asked to boil water, and that I wanted to know when that boil advisory was over.

Sure, the information is right there on their website, so one option is to just reload the website all day and wait for an alert to pop up:

Screenshot of an alert on a city website advising that a boil advisory has been lifted

But what if my goal and wish is to have access to that information in a programmatic way? Such as, say, RSS, the technology that has long allowed websites to share information with each other in a format that other websites and software can read?

Once I have that, I could read the alert in a news reader, I could have it piped to a Slack or Discord channel, I could make a little notification appear on my computer desktop, make the lights flash using IFTTT or Zapier, or whatever was most useful to me. The steps would be:

  1. Get RSS feed link.
  2. Do whatever I want to with the information in the feed.

But this town does not offer an RSS feed on its website. Not for alerts, not for regular community news, nothing. (To make matters worse, it’s powered by web software that appears aimed at helping smaller towns and cities have a web presence, so we know this missing feature is now being repeated in communities across the world!)

What are my other options?

Continue reading An example of why RSS is useful and important

Unlocking email content into RSS feeds redux with WordPress and Postie

As a part of some local journalism projects I’m exploring, I wanted to have a way to get information that is being emailed around (press releases, newsletters) into a publicly accessible RSS feed.

I’ve already explored this general “unlock email into an RSS feed” workflow using Zapier but Zapier’s limitations around translating HTML email messages into useful RSS entries led me to explore other options. For a while now I’ve been using Feedly’s paid feature that lets you receive email at a custom address and puts the content into your feed reading experience, and that’s actually been a good solution for me as an individual. (I made sure to set up an address at a domain I control and aliased that to Feedly’s provided address, in case I want to move to another solution later.)

But if we want to help a given audience have better access to information that’s only available in email but is intended to be public, I don’t think it scales well to ask them all to subscribe to the same email newsletter, or to all sign up for a Feedly paid plan. And yet so many organizations continue to use email as a way to distribute information, often instead of a website, and it doesn’t scale well to beg each of them to start (or go back to?) publishing their updates on a website with an RSS feed.

I started looking at using Mailparser for a more generalized solution. Receive the emails, have Mailparser extract the information into a structured, API-queryable format, and then download that information (including any attachments and images) and put it up on a publicly accessible URL somewhere. I knew I needed a way to organize the information coming in according to the email address of the person who sent it, so I’d have to build a small application that managed that categorization during the publishing process.

And then I realized I was basically getting into CMS territory. Publishing text and media on a website. Organizing content by categories and authors. Searchable and sortable. Yeah, I know a tool that already does all that stuff really well: WordPress.

But was I still going to need to build a glue application to process the emails and create WordPress posts?

I was aware of Jetpack’s post by email feature and I think it could work well for some scenarios, but I wanted something a little more purpose built. I did a little bit of Googling and found Postie, a WordPress plugin that has great features for bringing emails into WordPress posts. I exclaimed many words of delight upon finding it, and continued to be impressed as I looked through the thoughtful documentation, the developer-oriented options for extending and customizing it, and the active support and maintenance that goes into it. The WordPress community is amazing that way. I sent the developer a donation.

So, here’s the new workflow I would use:

  • Email is sent to an alias at my custom domain, which goes to a free email provider with IMAP access.
  • Every half hour, Postie goes out and checks for new email messages and creates pending WordPress posts.
  • I get a notification in a Slack channel about the WordPress post, and can publish it if appropriate.
  • The WordPress site provides a built in RSS feed of “emails” as posts on the site.

Amazing! But I still wanted a way to organize the incoming emails and resulting posts based on sender, without using Postie’s default method of creating WordPress users for each sender.

So, I created a custom taxonomy, Sources, with some term meta fields that allow me to associate email addresses and to select whether posts from that source should be pending or published by default. (Of course, email can be forged so it’s never safe to depend on the value of the “From” address in an email to authenticate anything important.)

Here are some code snippets used to accomplish this in a custom WordPress plugin I set up to complement Postie.

Continue reading Unlocking email content into RSS feeds redux with WordPress and Postie

Creating a personalized, private RSS feed for users in Laravel

In building WP Lookout I had the need to create user-specific, private RSS feeds as a way for users to get updates and information specific to their account. Here’s how I did it.

I started with the laravel-feed package from Spatie, which allows you to easily generate RSS feeds, either from a model’s records or from some other method that you define. I found that someone has proposed a feature and submitted a PR to add support for signed routes, which is Laravel’s way of adding a hash key to a given URL so that it can’t be accessed if it’s been modified, perfect for something like a user-specific RSS feed. But the Spatie folks decided not to include the feature, so I had to figure out how to do it on my own.

First, I added the package to my project and published the related config file:

$ composer require spatie/laravel-feed
$ php artisan vendor:publish \
    --provider="Spatie\Feed\FeedServiceProvider" \
    --tag="config"

Then, I added a user-specific feed identifier to the User model:

$ php artisan make:migration add_feed_id_to_users_table --table=users

The migration looked like this in part:

public function up()
{
    Schema::table('users', function (Blueprint $table) {
        //
        $table->string('feed_id');
    });

    DB::statement('UPDATE users SET feed_id = ...');
}

The UPDATE statement there is used to set the initial value of the feed identifier for existing users. An alternative would have been to make the field nullable and then set it elsewhere. For new users, I set the feed identifier as users are created, via the boot() method in the User model:

Continue reading Creating a personalized, private RSS feed for users in Laravel

Quick hack to monitor the health of a collection of AWS services

My SaaS app WP Lookout uses Amazon Web Services for hosting. I’ve had plenty of monitoring set up using both internal and external tools notifications, and these mostly output to a combination of a dedicated Slack channel and/or a Pushover notification that goes to my mobile device.

The one thing I didn’t have in place until recently was monitoring for the health of the specific group of AWS services the app depends on. With AWS offering many services spread across many parts of the world (as illustrated in the Very Long Service Health Dashboard), I wasn’t about to start monitoring all of the AWS infrastructure just to see if my particular app was affected.

They do offer a “Personal Health Dashboard” that’s supposedly more tailored to incidents affecting my AWS account, but I find it unintuitive and confusing to use, especially when it comes to setting up notifications to external services like Slack. In my online searching I found various services and tools that supposedly integrated the PHD into more standard monitoring and alerting options, but they were also either too complicated or seemed to require additional levels of paid AWS support plans that I didn’t want to take on.

So I decided to keep things simple and use the tried and true technology for polling information about external resources: RSS!

Continue reading Quick hack to monitor the health of a collection of AWS services

Generate an RSS feed from a Twitter user timeline

I needed to generate a valid RSS feed from a Twitter user’s timeline, but only for tweets that matched a certain pattern. Here’s how I did it using PHP.

First, I added the dependency on the TwitterOAuth library by Abraham Williams:

$ composer require abraham/twitteroauth

This library will handle all of my communication and authentication with Twitter’s API. Then I created a read-only app in my Twitter account and securely noted the four key authentication items I would need, the consumer API token and secret, and the access token and secret.

Now, I can quickly bring recent tweets from my target Twitter user into a PHP variable:

require "/path/to/vendor/autoload.php" ;
use Abraham\TwitterOAuth\TwitterOAuth;

$consumerKey       = "your_key_goes_here"; // Consumer Key
$consumerSecret    = "your_secret_goes_here"; // Consumer Secret
$accessToken       = "your_token_goes_here"; // Access Token
$accessTokenSecret = "your_token_secret_goes_here"; // Access Token Secret

$twitter_username    = 'wearrrichmond';

$connection = new TwitterOAuth( $consumerKey, $consumerSecret, $accessToken, $accessTokenSecret );

// Get the 10 most recent tweets from our target user, excluding replies and retweets
$statuses = $connection->get(
	'statuses/user_timeline',
	array(
		"count" => 10,
		"exclude_replies" => true,
		'include_rts' => false,
		'screen_name' => $twitter_username,
		'tweet_mode' => 'extended',
	)
);

My specific use case is that my local public school system doesn’t publish an RSS feed of news updates on its website, but it does tweet those updates with a somewhat standard pattern: the headline of the announcement, possibly followed by an at-mention and/or image, and then including a link back to a PDF file on their website that lives in a certain directory. I wanted to capture these items for use on another site I created to aggregate local news headlines into one place, and it mostly relies on the presence of an RSS feed.

So, I only want to use the tweets that match this pattern in the custom RSS feed. Here’s that snippet:

// For each tweet returned by the API, loop through them
foreach ( $statuses as $tweet ) {

	$permalink = '';
	$title     = '';

	// We only want tweets with URLs
	if ( ! empty( $tweet->entities->urls ) ) {

		// Look for a usable permalink that matches our desired URL pattern, and use the last (or maybe only) one
		foreach ( $tweet->entities->urls as $url ) {

			if ( false !== strpos( $url->expanded_url, 'rcs.k12.in.us/files', 0 ) ) {
				$permalink = $url->expanded_url;

			}
		}

		// If we got a usable permalink, go ahead and fill out the rest of the RSS item
		if ( ! empty( $permalink ) ) {

			// Set the title value from the Tweet text
			$title = $tweet->full_text;

			// Remove links
			$title = preg_replace( '/\bhttp.*\b/', '', $title );

			// Remove at-mentions
			$title = preg_replace( '/\@\w+\b/', '', $title );

			// Remove whitespace at beginning and end
			$title = trim( $title );

			// TODO: Add the item to the feed here

		}
	}
}

Now we have just the tweets we want, ready to add to an RSS feed. We can use the included SimplePie library to do this. In my case, the final output is written to an output text file, which another part of my workflow can then query.

Here’s the final result all put together:

<?php

/**
 * Generate an RSS feed from a Twitter user's timeline
 * Chris Hardie <chris@chrishardie.com>
 */

require "/path/to/vendor/autoload.php" ;
use Abraham\TwitterOAuth\TwitterOAuth;

$consumerKey       = "your_key_goes_here"; // Consumer Key
$consumerSecret    = "your_secret_goes_here"; // Consumer Secret
$accessToken       = "your_token_goes_here"; // Access Token
$accessTokenSecret = "your_token_secret_goes_here"; // Access Token Secret

$twitter_username    = 'wearrrichmond';
$rss_output_filename = '/path/to/www/rcs-twitter.rss';

$connection = new TwitterOAuth( $consumerKey, $consumerSecret, $accessToken, $accessTokenSecret );

// Get the 10 most recent tweets from our target user, excluding replies and retweets
$statuses = $connection->get(
	'statuses/user_timeline',
	array(
		"count" => 10,
		"exclude_replies" => true,
		'include_rts' => false,
		'screen_name' => $twitter_username,
                'tweet_mode' => 'extended',
	)
);

$xml = new SimpleXMLElement( '<rss/>' );
$xml->addAttribute( 'version', '2.0' );
$channel = $xml->addChild( 'channel' );

$channel->addChild( 'title', 'Richmond Community Schools' );
$channel->addChild( 'link', 'http://www.rcs.k12.in.us/' );
$channel->addChild( 'description', 'Richmond Community Schools' );
$channel->addChild( 'language', 'en-us' );

// For each tweet returned by the API, loop through them
foreach ( $statuses as $tweet ) {

	$permalink = '';
	$title     = '';

	// We only want tweets with URLs
	if ( ! empty( $tweet->entities->urls ) ) {

		// Look for a usable permalink that matches our desired URL pattern, and use the last (or maybe only) one
		foreach ( $tweet->entities->urls as $url ) {

			if ( false !== strpos( $url->expanded_url, 'rcs.k12.in.us/files', 0 ) ) {
				$permalink = $url->expanded_url;

			}
		}

		// If we got a usable permalink, go ahead and fill out the rest of the RSS item
		if ( ! empty( $permalink ) ) {

			// Set the title value from the Tweet text
			$title = $tweet->full_text;

			// Remove links
			$title = preg_replace( '/\bhttp.*\b/', '', $title );

			// Remove at-mentions
			$title = preg_replace( '/\@\w+\b/', '', $title );

			// Remove whitespace at beginning and end
			$title = trim( $title );

			$item = $channel->addChild( 'item' );
			$item->addChild( 'link', $permalink );
			$item->addChild( 'pubDate', date( 'r', strtotime( $tweet->created_at ) ) );
			$item->addChild( 'title', $title );

			// For the description, include both the original Tweet text and a full link to the Tweet itself
			$item->addChild( 'description', $tweet->full_text . PHP_EOL . 'https://twitter.com/' . $twitter_username . '/status/' . $tweet->id_str . PHP_EOL );

		}
	}
}

$rss_file = fopen( $rss_output_filename, 'w' ) or die ("Unable to open $rss_output_filename!" );
fwrite( $rss_file, $xml->asXML() );
fclose( $rss_file );

exit;

Here’s the same thing as a gist on GitHub.

I set this script up to run via cronjob every hour, which gives me a regularly updated feed based on the Twitter account’s activity.

Several ways this could be improved include:

  • Better escaping and sanitizing of the data that comes back from Twitter
  • Make the filtering of the Tweets more tolerant to changes in the target user’s Tweet structure
  • Genericizing the functionality to support querying multiple Twitter accounts and generating multiple corresponding output feeds
  • Fixing Twitter so that RSS feeds of user timelines are offered on the platform again

If you find this helpful or have a variation on this concept that you use, let me know in the comments!

Updated July 14 2020 to add the use of “tweet_mode = extended” in the API connection and to replace the references to tweet text with “full_text”, as apparently it is not the default in the Twitter API to use the 240-character version of tweets.

Put all those email newsletters in an RSS feed

The other day someone told me that they think blogging is dead.

I tried to suppress the sounds of existential pain emanating from deep within my soul, but it still hurt.

Blogging is far from dead, but I also recognize that email newsletters are all the hotness right now when it comes to getting your written thoughts in front of someone else. And I recognize that if you want to follow some kinds of updates from some kinds of people or organizations, you’re going to have to do the email thing.

For a while, I used email filters to manage this issue, dutifully creating or updating them in my setup each time I cringe-fully subscribed to a new email newsletter list after searching in vain for an RSS feed subscribe button. Then I would let them all go into just the right email folder (or maybe even still my inbox) so I could read them when I was in the mood to read blog-posts-as-email-messages on a given subject.

Ugh.

I didn’t like that this approach still created a kind of additional email “to do” burden on me, leaving me with folders to sort, search through and clear out. Newsletter content is usually not actionable or time-sensitive. What I really wanted was to treat all those email newsletter messages like blog post headlines in a separate kind of reader app, available to be read at my leisure. YOU KNOW, LIKE AN RSS FEED READER.

So here’s my current setup:

  1. newsletter emails go to a dedicated email alias configured at my mail provider, and that’s what I subscribe to lists with
  2. those messages are forwarded to a Zapier-powered recipe that converts them into items on a custom generated RSS feed
  3. I subscribe to the RSS feed in my feed reader, Feedly.

Now I can browse the headlines when I want to, read some items and gloss over others, and my email inbox is no longer crowded with articles that aren’t necessarily actionable or time-sensitive for me.

A few things I could do to tweak this setup further:

  • Right now all of my email newsletters go into a single RSS feed. For better categorization and readability, I could break these out into individual feeds.
  • The translation of HTML-only emails (another annoying thing about the email newsletter age) doesn’t always work well into the RSS feed format as supported by Zapier. I haven’t really explored a fix for this but it hasn’t affected me much so far.

Also note that Zapier’s pricing structure is such that depending on the number of incoming messages you have, you might need to upgrade to a paid plan.

That’s it. My email inbox has benefitted greatly from this setup, and I hope yours will too.