Extending Yahoo! Pipes with Google App Engine

In my Uploads and Favorites pipe I wanted to turn the timestamps into so called "humanized dates", like these six examples:

23 seconds ago
1 minute ago
4 hours ago
2 days ago
7 months ago
1 year ago

This could not be done by using only Yahoo! Pipes.

Google App Engine to the rescue

Yahoo! Pipes can be extended using a web service, which accepts the feed items as json, and outputs them (after changes) as json. I used this blog article Using Google App Engine to Extend Yahoo! Pipes as a starting point. Django framework has date humanizing functions coded in Python, and I borrowed those functions from there. Then I created a new Google App Engine application "humandate", and made files app.yaml and humandate.py.

app.yaml

Data provided by Pastebin.com - Download Raw - See Original

humandate.py

Data provided by Pastebin.com - Download Raw - See Original

I put these two files in a directory called "humandate" under my Google App Engine directory, with favicon.ico (the GAE default icon), and the development environment is ready. I now can deploy the application using "./appcfg.py update humandate/" or test it locally using "./dev_appserver.py humandate/", then the app runs at http://localhost:8080.

Testing the date humanizer

I made also an url based interface to "humandate" in addition to Yahoo! Pipes json interface. Url based interface works by giving a unix timestamp as "unixtime" parameter, like this. This made testing the date humanizer easier.

Using the service in Yahoo! Pipes

This service expects items to have y:published.utime field present. Service reads it from every item, turns it into a humanized date, and makes a new field "when" containing the humanized date. Then the service returns the items back to pipes.

See also my previous article about the four youtube pipes I've made (one of them uses this humandate, and one uses it indirectly). And have fun with Google App Engine and Yahoo! Pipes.

Yahoo! Pipes, pipes, and more pipes for YouTube

Lately I've been playing around with Yahoo! Pipes and learned some new things. I already presented the YouTube Activity Feed, which I've cleaned up a little. And I've made three more pipes.


YouTube Activity Feed

Pipe ID: 58c841d14337ba4fbf693abd9701dc49
Pipe Web Address: http://pipes.yahoo.com/mspotilas/youtubeactivityfeed

Here's my previous article about this feed. YouTube API 2.0 does provide activity feeds, but you need to register an application to get developer key. My idea was to read the same info from user's channel feed page in html, parse the html and republish as a feed, and YouTube Activity Feed pipe does that.

As html parsing is quite complex task, if YouTube changes the feed page structure a lot, this pipe might break. I'll probably try to fix it if that happens.


YouTube Likes Feed

Pipe ID: a58d86555f7eb396e73d58ff50edc8f1
Pipe Web Address: http://pipes.yahoo.com/mspotilas/youtubelikesfeed

Same idea as in Youtube Activity Feed, but it gets only "liked videos" from the channel feed page. It is a bit simpler to parse only the likes. Also with this pipe, if YouTube changes the feed page a lot, the pipe might break.


YouTube Uploads and Favourites

Pipe ID: 0c2ee991c8d94ed0ceb19865c252e047
Pipe Web Address: http://pipes.yahoo.com/mspotilas/youtubeuploadsfavoritesfeed

YouTube has gdata feeds for user uploads and user favorites.

http://gdata.youtube.com/feeds/base/users/USERNAME/uploads?alt=rss&v=2
and
http://gdata.youtube.com/feeds/base/users/USERNAME/favorites?alt=rss&v=2

This pipe combines these two feeds, descriptions will be for example "2 days ago USERNAME favorited" or "1 hour ago USERNAME uploaded". The tricky part was to get this humanized date "2 days ago" derived from feed item's timestamp. It could not be done in Pipes only, so I extended its functionality with a web service I coded into Google AppEngine. It goes through the list, converts "y:published.utime" to humanized date like "2 days ago" and stores it in field "when" and returns the list back to pipes. Here's an article about the web service: Extending Yahoo! Pipes with Google App Engine.

This pipe should be robust, as it uses feeds as source, not page html.


YouTube Feed

Pipe ID: 73c5ee4a46e1837d908deaa60b1858d2
Pipe Web Address: http://pipes.yahoo.com/mspotilas/youtubefeed

This fourth pipe simply combines Uploads and Favourites and Likes Feed. Structure is very simple. Output is something similar to YouTube Activity feed, but it does not contain comments, playlist additions nor channel subscriptions. But sometimes simpler is better.


All pipes are compatible with my gadget which works also on Opera (Yahoo! Pipes badge does not). Just change the pipe ID (_id) to use a different pipe.

Feel free to examine these pipes, you can also make clones which you can modify. Comments are welcome, too. :)

Another gadget for YouTube Activity Feed, works with Opera

Few days ago I finished making my first Yahoo! Pipe, which collects info from YouTube channel feed page, and republishes that info. Yesterday I noticed, that Yahoo! Pipes badge output does not support Opera – with Opera, nothing is shown. It is strange that Yahoo does not fix this, maybe it is because Pipes is a free product. About 3 % of my blogs readers use Opera.

I knew I could make a gadget that works on Opera, too. Yesterday evening I coded first a javascript version, and today morning a little tidier jQuery version.

Demo

Installation is simple: add an HTML/Javascript gadget and put the following code in it:

Data provided by Pastebin.com - Download Raw - See Original

You can leave out the first line, if your template already loads jQuery. Make necessary configuration changes (change YouTube user name, for example). I did only very basic CSS, you can of course modify it to your liking. Code can be used also outside Blogger (in any html page).

Yahoo! Pipe to show YouTube activity feed in Blogger, WordPress, etc.

I've had this idea for some time: somehow republish YouTube channel activity feed page as an RSS feed. So you could get a view of your latest YouTube activity in your blog, for example. This week I finally implemented a version of this using Yahoo! Pipes. This is a completely new gadget for Blogger, I have not seen anything like this before. You can use the pipe gadget output on WordPress and other internet pages, too. And you can get the RSS address for the pipe to connect it with various services and/or applications.

Link to the pipe: pipes.yahoo.com/mspotilas/youtubeactivityfeed.

And here's the "badge output" of that pipe from my youtube feed page, with 5 items:

This can be put into a sidebar gadget, like I've done in YABTB demo blog. The code to put inside an HTML/Javascript gadget is simple:

Data provided by Pastebin.com - Download Raw - See Original

You may also use the "Get as a Badge" link when the pipe is run in the pipe's page, it offers an automated gadget installation into Blogger blogs, WordPress blogs and others.

Update May 21th: Because Badge output does not work with Opera browser, you might consider using instead a gadget made by me, which works with Opera.

Parameters of the pipe:

user: youtube user name,
max-results: maximum number of items returned,
allow: only accept entries that have this word in description, for example "liked" to get only likes or "uploaded" to get only uploads,
disallow: don't allow entries that have this word in description, for example "commented" to filter out commenting activity.

So at minimum you just have to change the user parameter value from "mspotilas" to any youtube user name you want (it does not have to be yours, try for example "NBC"). In my YABTB demo blog, to show what can be done with CSS styles, I also changed background colors, adjusted font sizes and margins, changed thumbnail to right side, changed thumbnail proportions (size), and hid the footer (powered by and get this), like this:

Data provided by Pastebin.com - Download Raw - See Original

There is also a YouTube RSS feed, but... YouTube gdata feed for YouTube channel includes only uploads. No likes, favourites, comments, etc. Link to YouTube user's RSS feed of uploads is:

http://gdata.youtube.com/feeds/base/users/youtubeusername/uploads?alt=rss&v=2&orderby=published

For example my YouTube channel's RSS feed is:

gdata.youtube.com/feeds/base/users/mspotilas/uploads?alt=rss&v=2&orderby=published

This is my very first Yahoo Pipe, so the implementation might be a bit clumsy. But it works, at least in my tests. :) You can clone the pipe and modify it according to your needs, or use this pipe if it suits you as is. If you like it, or find some bugs or something to improve, please tell me.

Update May 20th: I found out that Opera browser is not supported by Yahoo! Pipes. :( I think I could make a gadget for the pipe which would work in Opera, too, but I'm not sure if I will. About 3 % of my blog's viewers do use Opera.
Update May 21th: Here's the gadget I made, works with Opera, too.

Top Commentators Gadget with avatars

I had a request of this gadget in one blog comment. There are some top commentators gadgets around, most relying on a Yahoo pipe. I knew it can be done with just javascript, with no pipes. Demo: it is installed on this blog, look at the right sidebar. Hope you like it. :)

Features

You can configure the maximum number of top commentators, and minimum number of comments one must have to get on the list (to weed out the 1 comment commentators, if wanted, for example). Own nick and a list of other nicks (for example "Anonymous") can be excluded from the list. Output strings can be translated.

The Script, installing

Add a HTML/Javascript gadget to your blog, and paste the following code into it.

Data provided by Pastebin.com - Download Raw - See Original

Script configuration

There are two CSS definitions at first, which can be changed to your liking. After that there comes javascript code and configurable variables. Here is a list of the variables and a short explanation.

maxTopCommenters: how big the list is at maximum
minComments: how many comments must top commentator have at least
numDays: from how many days (ex. 30), or 0 from "all the time" (max 500 comments)
excludeMe: true, if own comments excluded
excludeUsers: array of usernames to exclude
maxUserNameLength: if 0, don't cut; if > 4, cut lenghty usernames

txtTopLine: specifies, what is output on each line. This text can have simple variables, which are: [#], [image], [user], and [count]. [#] is substituted by position on the list, [image] by commentator's avatar image, [user] by commentator name and [count] by number of comments. Text can also contain html.
txtNoTopCommenters: what to display, if list is empty
txtAnonymous: "Anonymous" username localized, or empty if English text "Anonymous" is to be used
getTitles: if true, fetch titles from post feed; if false, titles are generated from url
blogger favico [B]
urlMyAvatar: especially if trueAvatars = false, set here the url to your avatar image
urlMyProfile: if you don't have a profile gadget on page, put your profile url here!
urlAnoAvatar: anonymous avatar, you can change from mystery man to a custom one, if you want, possibly this
urlNoAvatar: comment feed offers the icon (Blogger logo) for those Bloggers who have not set their profile image. You can override the icon with this setting (default: ).

cropAvatar: crop avatar to square (true) or stretch (false) to square
sizeAvatar: size of avatar in x and y direction, pixels

If you have very much comments (thousands and thousands of them), I suggest you to use this as "top commentators of 30 days" gadget or something like that (set numDays variable to 30), so that it won't take too long to load and calculate all the comments.

June 5th 2012: Small CSS update: Blogger changed the paddings of .profile-name-link, that's why I added this line (line 3.):
.top-commenter-line .profile-name-link {padding-left:0;}

August 1st 2017: To fix avatars, please change lines 48 and 49 (article code is updated).