You can use current awareness tools to watch information flow past in real time or save it for later. I do a lot of news trapping and, while I use alert services with RSS feeds, sometimes I want to capture something to review later. I had been using Scraperwiki for this because they supported 2 free scrapes and I could point it at Twitter and then return for my results. They have lost access to Twitter so I needed an alternative. Here’s my solution using Twitter, Google Drive and IFTTT.
Create an RSS Feed on Twitter
The first step is to create your output from Twitter. You can use pretty much any keywords. I typically want to capture the conversation from a conference or tweet chat, so I use the hashtag: #ILTA14 is one I used recently. In your Twitter account, click on the gear wheel at the top right to access the menu from which you can select Settings. At the bottom of the Settings menu are Widgets. You want to create a new widget. I tend to use the Create New Search Widget option. There is a video tutorial here from Amit Agarwal, which is where I learned this.
Once you have the widget created, follow the rest of Amit’s instructions. He has made his Google Script available to create RSS from your Google Drive account. His instructions are pretty clear on how to create your first one, which is to take the URL of your new script (created when you use Publish > Deploy as Web App) and your Twitter widget ID and put them together.
Two notes. First, I cut and pasted the HTML that Twitter provided for the widget into a blank text file in Notepad. I couldn’t select just the widget ID while it was in the Twitter page. Once in Notepad, I could delete everything but the ID and then paste in the script URL. Second, to see your script URL a second time, just return to Publish > Deploy as Web app. You’ll see that, second time around, the script/app URL is now visible at the top of the box. You can access the script from your Google Drive account (if you get a request to Connect Google Script, go ahead and accept and then click the script again).
You can see my Google Drive above, with the script for the RSS generator and the IFTTT folder I’ll be referring to below.
RSS Twitter is Too Noisy
I use RSS as my primary information source, following a load of feeds and Tiny Tiny RSS as my news reader. A Twitter-based RSS feed can become unmanageable, though, if it has a lot of activity. In fact, the whole reason I don’t follow these hashtags in a Twitter client like Hootsuite or using a saved search in Twitter is that there’s more happening than I want to follow live.
If you want to use the RSS as a news alert, you can just add it to your RSS feed reader. I wanted to save it for later, however, so now I turned to IFTTT.
IFTTT RSS to Google Drive Spreadsheet
IFTTT is a service that uses the if-then programming concept. If one thing happens, it triggers another. IFTTT has a trigger for Twitter but it is limited to your Twitter account. In this case, I wanted to use my new RSS feed, created above, as my trigger. You can see the two recipes I have below. The first one is the RSS one (see the orange icon) which grabs my #ILTA14 feed. The second one is my Twitter account, where I create an archive of my own tweets for posterity.
I won’t belabor the IFTTT recipe setup. It is pretty easy. You paste in the URL of your RSS feed (the script URL + the Twitter widget ID from above) and can then take all of the defaults. If you don’t already have an IFTTT folder in your Google Drive, it will create one, and any subfolders it needs. Be patient at this point. Until the recipe is triggered the first time, nothing will happen on Google Drive. It is not a realtime connection. In the case of this recipe, I had to wait nearly 30 minutes before the first tweets were recorded, even though they’d been happening from the instant I created the recipe.
By piping the tweets into Google Drive, I can save them as a spreadsheet. This is the spreadsheet with the #ILTA14 results in it, if you want to take a look. Once it’s in a spreadsheet, I can do so many more things with it:
- exclude RTs, for example, so I only see original tweets – this is particularly useful for Twitter chats, where a lot of lurkers may be retweeting messages but not a whole lot is actually being said
- filter by particular speakers/tweeters
- sort by date
- store copies of all of the links that were shared
Just as I used to do with Scraperwiki, this is a set and forget tool. It is not going to be useful if you need minute-by-minute alerts. However, I have found it is good for known hashtags – like conferences – and also when I am starting a new project and want to trawl for concepts or terminology but don’t need to watch them in realtime.