Finding interior terms or “sub-trends” on Twitter

After a discussion with some of the other interhacktives, two things became readily apparent:

• Being able to search for words accompanying trends on Twitter would be really useful
• Currently, support for “sub-trend” searches is fairly lacking

As an example of the first point: imagine you’re at a conference full of Twitter users, and everyone’s using the hashtag #fakeEvent to talk to each other. This works perfectly fine if everyone’s always in the same room, but what if the conference’s structure was such that attendees had to choose from a variety of talks happening concurrently? Say folks in one talk started using the hashtag “#foo” (to use a long-standing programming convention) to talk about that individual session, while those attending another talk used hashtag “#bar” (to use another), and these were added to the global #fakeEvent hashtag (Example: “@soAndSo says Twitter’s a waste of time… #foo #fakeEvent”). If you’re in either talk and know what the hashtag is, or the global #fakeEvent hashtag isn’t moving very fast, there isn’t a problem. But what if you’re watching an event’s Twitter traffic from afar and don’t know what the additional hashtags are about, or the global hashtag is moving too quickly to find relevant information about a specific aspect of that event? Another example of where this might be true is the worldwide Occupy protests — finding the global hashtag isn’t very difficult, but drilling down into specific subject matter becomes significantly moreso.

I thus began looking for ways to find the words that accompany a specific hashtag. While some tools have limited support for this out of the box (Specifically Twazzup, which is pretty cool unto itself), there just isn’t the level of effectiveness allowed by a paid, high-level marketing tool like the $150 per month PeopleBrowsr. A StackExchange question I posted hasn’t yielded anything in the way of tools specifically designed for this purpose, either.

However, thanks to the ever-helpful Tony Hirst (who you should all most certainly follow at @psychemedia for all things related to R and data visualization), a Yahoo! Pipes tutorial on how to accomplish exactly this is now available, with the end result available here.

While Tony’s Pipe is a really good start towards a tool such as the one I’m envisioning, it falls short in a few places. It’s limited on some level by the Yahoo! Pipes API, which caps the number of requests at 200 (This can be expanded to 1500 if you clone and modify the Pipe, but even that might not be sufficient for a really popular hashtag), and can’t seem to filter out ampersand characters. It would also be great if there was a full dictionary of predicates and other words to always ignore (“a,” “if”, “the”, “but”, et cetera), plus maybe a way to view the popularity of sub-trends over time (I.e., “which were the most active sub-trends at #fakeEvent?”). A way to ignore words preceded by “@” would also be useful, so as to filter out users who get a lot of responses in a specific hashtag and thus become a sub-trend unto themselves. Finally, a way to limit words to only those preceded by a hash character would probably also help, so as to only get sub-hashtag hashtags.

That’s not to complain about the solution Tony set up — it’s really well thought-out and I’ve already begun using it — but rather, to point out an interesting project idea for any reasonably-talented RESTful programmer with an afternoon to kill.

Know of a good search tool for terms inside hashtags? Did I miss something? Let me know in the comments below!