Extending Yahoo! Pipes with Google App Engine

In my Uploads and Favorites pipe I wanted to turn the timestamps into so called "humanized dates", like these six examples:

23 seconds ago
1 minute ago
4 hours ago
2 days ago
7 months ago
1 year ago

This could not be done by using only Yahoo! Pipes.

Google App Engine to the rescue

Yahoo! Pipes can be extended using a web service, which accepts the feed items as json, and outputs them (after changes) as json. I used this blog article Using Google App Engine to Extend Yahoo! Pipes as a starting point. Django framework has date humanizing functions coded in Python, and I borrowed those functions from there. Then I created a new Google App Engine application "humandate", and made files app.yaml and humandate.py.

app.yaml

Data provided by Pastebin.com - Download Raw - See Original

humandate.py

Data provided by Pastebin.com - Download Raw - See Original

I put these two files in a directory called "humandate" under my Google App Engine directory, with favicon.ico (the GAE default icon), and the development environment is ready. I now can deploy the application using "./appcfg.py update humandate/" or test it locally using "./dev_appserver.py humandate/", then the app runs at http://localhost:8080.

Testing the date humanizer

I made also an url based interface to "humandate" in addition to Yahoo! Pipes json interface. Url based interface works by giving a unix timestamp as "unixtime" parameter, like this. This made testing the date humanizer easier.

Using the service in Yahoo! Pipes

This service expects items to have y:published.utime field present. Service reads it from every item, turns it into a humanized date, and makes a new field "when" containing the humanized date. Then the service returns the items back to pipes.

See also my previous article about the four youtube pipes I've made (one of them uses this humandate, and one uses it indirectly). And have fun with Google App Engine and Yahoo! Pipes.

Yahoo! Pipes, pipes, and more pipes for YouTube

Lately I've been playing around with Yahoo! Pipes and learned some new things. I already presented the YouTube Activity Feed, which I've cleaned up a little. And I've made three more pipes.


YouTube Activity Feed

Pipe ID: 58c841d14337ba4fbf693abd9701dc49
Pipe Web Address: http://pipes.yahoo.com/mspotilas/youtubeactivityfeed

Here's my previous article about this feed. YouTube API 2.0 does provide activity feeds, but you need to register an application to get developer key. My idea was to read the same info from user's channel feed page in html, parse the html and republish as a feed, and YouTube Activity Feed pipe does that.

As html parsing is quite complex task, if YouTube changes the feed page structure a lot, this pipe might break. I'll probably try to fix it if that happens.


YouTube Likes Feed

Pipe ID: a58d86555f7eb396e73d58ff50edc8f1
Pipe Web Address: http://pipes.yahoo.com/mspotilas/youtubelikesfeed

Same idea as in Youtube Activity Feed, but it gets only "liked videos" from the channel feed page. It is a bit simpler to parse only the likes. Also with this pipe, if YouTube changes the feed page a lot, the pipe might break.


YouTube Uploads and Favourites

Pipe ID: 0c2ee991c8d94ed0ceb19865c252e047
Pipe Web Address: http://pipes.yahoo.com/mspotilas/youtubeuploadsfavoritesfeed

YouTube has gdata feeds for user uploads and user favorites.

http://gdata.youtube.com/feeds/base/users/USERNAME/uploads?alt=rss&v=2
and
http://gdata.youtube.com/feeds/base/users/USERNAME/favorites?alt=rss&v=2

This pipe combines these two feeds, descriptions will be for example "2 days ago USERNAME favorited" or "1 hour ago USERNAME uploaded". The tricky part was to get this humanized date "2 days ago" derived from feed item's timestamp. It could not be done in Pipes only, so I extended its functionality with a web service I coded into Google AppEngine. It goes through the list, converts "y:published.utime" to humanized date like "2 days ago" and stores it in field "when" and returns the list back to pipes. Here's an article about the web service: Extending Yahoo! Pipes with Google App Engine.

This pipe should be robust, as it uses feeds as source, not page html.


YouTube Feed

Pipe ID: 73c5ee4a46e1837d908deaa60b1858d2
Pipe Web Address: http://pipes.yahoo.com/mspotilas/youtubefeed

This fourth pipe simply combines Uploads and Favourites and Likes Feed. Structure is very simple. Output is something similar to YouTube Activity feed, but it does not contain comments, playlist additions nor channel subscriptions. But sometimes simpler is better.


All pipes are compatible with my gadget which works also on Opera (Yahoo! Pipes badge does not). Just change the pipe ID (_id) to use a different pipe.

Feel free to examine these pipes, you can also make clones which you can modify. Comments are welcome, too. :)

Google App Engine, Python 2.7, and lxml

Google App Engine (GAE) lets you build and run applications on Google's infrastructure. I have done two applications with GAE and Python (the programming language). The previous version of my tool to get blogger avatars (avafavico) used regular expressions to parse the Blogger profile page for the profile photo, or user's first contributed blog's favico, if no profile photo is found. As you might know, using regular expressions may not be the best solution. It would be better to parse html page document object model (DOM) and get results there. That way the code is not so sensitive to possible changes in the page html code. Of course I did the regular expressions so, that they should work with different html.

Previously I used GAE Python 2.5 environment, which was the default and supported version. On February 27th Python 2.7 became fully supported. GAE with Python 2.7 contains more external libraries than version 2.5, one of those libraries being lxml.

I tried to search for different solutions and examples, but there were not many. With GAE Python 2.5, one can use BeautifulSoup, but there are some issues (problematic 3.1.0 version, uncertain development future of BS, etc.). And there is minidom, but it may not handle broken html well. Blogger profile page should not have broken html, but you never know. lxml is definitely better and faster, supports XPath, etc.

The day before yesterday I updated the GAE app to use Python 2.7 and lxml. There were none to some examples about using python27, lxml and GAE, so I'll show you here a working example. First I started modifying the file app.yaml, there I changed runtime to python27, added "threadsafe" (false), and added (latest) lxml in libraries section. Increased version number to 5. Now app.yaml looks like this:

Data provided by Pastebin.com - Download Raw - See Original

In blogava.py I added "from lxml import etree" and then used etree functions instead of regular expressions to find things in html. Here's how DOM tree is constructed, variable "result" contains page html as a string, and then XPath is applied to the tree, like this:

>>> tree = etree.HTML(result)
>>> r=tree.xpath("//img[@id='profile-photo']/@src")

Here find from the tree the first img tag, which id is set to "profile-photo", and get that tag's src attribute. In the full script, if no id='profile-photo' is not found, then try to search for the first image that has class "photo". If both fails, search for first "contributed-to" blog, and use it's favicon, if that is not found, use Blogger favicon. And here is the blogava.py source file:

Data provided by Pastebin.com - Download Raw - See Original

This new version of avafavico has been up and running for two days. I'm very pleased that I got lxml working with GAE, all in all it was quite easy. Hope this example is useful to someone. If it helped you, please leave a comment. :)

YouTube/JSONP oEmbed service "oembed-js"

In the previous post I introduced my oembed-js service and an application using it: YouTube embeds in Blogger comments.

Oembed-js is a Google Appengine application coded in Python. It relays YouTube oembed calls and offers results also in JSONP format, so javascript applications can use it.

Parameters

url: the (youtube) url to embed (best to be urlencoded)
maxwidth: max width of embedded object (optional)
maxheight: max height of embedded object (optional)
format: json/jsonp or xml (optional, default:json/jsonp)
callback or jsonp: the javascript callback function, required, if jsonp format is wanted
callID: a call ID that is passed through to response (optional)

Sample call:

http://oembed-js.appspot.com/?callback=foo&callID=123&url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DGMra5Wh51oU

Response

YouTube oEmbed response in json(p) or xml format, including field html (see oembed specs), and:

callID: if callID was given, it is passed through (can identify the response)
url: pass through the url that is being embedded
videoid: contains 11 char long YouTube videoid, if single video embed
playlist: contains playlist id, if playlist embed

Sample response:

foo({"provider_url": "http://www.youtube.com/", "version": "1.0", "title": "Sviatoslav Richter in Kiev, 1958 - Pictures at an Exhibition", "url": "http://www.youtube.com/watch?v=GMra5Wh51oU", "type": "video", "videoid": "GMra5Wh51oU", "thumbnail_width": 480, "height": 344, "width": 459, "html": "<iframe width=\"459\" height=\"344\" src=\"http://www.youtube.com/embed/GMra5Wh51oU?fs=1&feature=oembed\" frameborder=\"0\" allowfullscreen></iframe>", "author_name": "FirstPublicChannel", "callID": "123", "provider_name": "YouTube", "thumbnail_url": "http://i4.ytimg.com/vi/GMra5Wh51oU/hqdefault.jpg", "thumbnail_height": 360, "author_url": "http://www.youtube.com/user/FirstPublicChannel"})

Another demo how to use it

Enter YouTube URL (video or playlist):

 

This demo and its source is also here in this jsFiddle.

Update March 27th 2012: Now oembed-js supports youtu.be urls (YouTube oembed service does not support them). Also now supports videos which have embedding turned off (YouTube oembed does not support these either).

Allow users embed YouTube videos in Blogger comments

Some time ago while googling something about video embeds I came across with oEmbed specification. This simple API allows websites to display embedded content from oEmbed providers. After some inspecting I used the oEmbed in a Python script to turn youtube links in a text into embedded videos. Worked nicely.

oEmbed returns data in JSON or XML format. That is ok with server side languages like php and Python. But for javascript this is a problem. If there is a problem, usually solution is found, and I found Embedly service, which is unfortunately not free.

With no intention to invest any money, and hands itching for another Google Appengine Python app, I saw an opportunity. To make a service that relays YouTube's oEmbed results to javascript format (JSONP). And all with memory caching etc. so it should be able to handle some traffic. After some coding it was ready and working http://oembed-js.appspot.com/?url=[youtube url to embed][&callback=foo] But now where to use it...

In Wordpress blogs, when commenting an article, user can just write/paste youtube url into the comment text, and it will be turned into a video embed. Surprise, not in Blogger. For some blogs this is a bummer. Now with my new, free, javascript/jQuery compatible GAE service oembed-js this became possible in Blogger. Of course we need to hack the blog first a bit. :)

See the demo blog!

You can decide, whether you want to turn all raw youtube links (watch and playlist links) into embeds, which is the default, or only those, that use [video=youtube_url] tag. You can also change the tag name, if you want, maybe you like youtube or embed more. If you have lazy loading installed in your blog, by default this script will use it, and pages with many videos load much faster. Installation is simple, open template html for editing, find </head> and paste this code before it:

Data provided by Pastebin.com - Download Raw - See Original

This script does not work in dynamic views, but it works with Blogger native comments, my threaded comments, and also Blogger threaded comments. Hope you like it. I'll tell you later about the oembed‑js service API (application programming interface), so it is easier for you to develop own applications.

References:
- oEmbed specification
- YouTube API blog: oEmbed support

Update 27th March 2012: Line #74 updated, to handle better the case of only a YouTube url as comment text in new Blogger threaded comments (you do not need to update if you use videotag, or don't use Blogger threaded comments).

URLtinyfy - a JSONP API for tinyurl.com, goo.gl, etc.

Shortened urls, of blog posts for example, maybe useful in some places (like Twitter). I started first making a hack for Blogger which would offer the reader shortened url(s) for the posts. I came across ggl-shortener and json-tinyurl, which could be used to make goo.gl and tinyurl.com short urls in client side (javascript). But these both Google App Engine applications suffer from overrunning quota, especially json-tinyurl, but sometimes also ggl-shortener.

I think they both simply forward every request to the url shortener services, which always results in url fetch call, and url fetches are quite limited per day in Google appengine. My hands were already itching to make a new appengine app and to excercise my new Python skills, and I thought I could do the url shorting a bit cleverer. And at least have (for my url shortener hack) a service that is not yet, hopefully ever, suffering from quota problems.

Presenting urltinyfy

So I created a new appspot app, urltinyfy, which offers more or less the same APIs as ggl-shortener and json-tinyurl, at least should be compatible. As a bonus there is also support for safe.mn, although they have their own jsonp API so it is a bit redundant. I also coded bit.ly support, but because it needs bitly API key, and they have own jsonp API, I left it out from the "production version". Safe.mn and bit.ly was mainly implemented because I wanted to see if and how it could be done.

My application uses memory cache, so if shortened url is already fetched, it is returned from the cache and no url fetch is needed. This should lessen the possibility to run out of free quota (of url fetches).

I have coded this entirely from a scratch, I have not used any code from ggl-shortener or json-tinyurl. I'm not sure if I'm going to publish the source code of this application, we'll see about that. :)

Parameters

url:  the url to shorten (best to be urlencoded). Only mandatory parameter.
callback or jsonp: the callback function, if jsonp is used
unify: if 1, json will include tinyurl and longurl fields for all services (easy to use, independent of the service), and will pass through title and callID parameters, if found

Usage

If you now use ggl-shortener or json-tinyurl, you should be able to switch to my app with minimal changes, by changing the urls in your javascript like this:

change http://ggl-shortener.appspot.com/?url=... to http://urltinyfy.appspot.com/googl?url=...
change http://json-tinyurl.appspot.com/?url=... to http://urltinyfy.appspot.com/tinyurl?url=...

As alternative "syntax", http://urltinyfy.appspot.com/googl?url=... is the same as http://urltinyfy.appspot.com/?service=googl&url=..., and accordingly with other services.

If you want javascript examples how to use this, do google searches for "ggl-shortener" and "json-tinyurl". I might use this in some url shortening hack later on.


Problems? Suggestions? Please leave a comment.

My Google App Engine Python application evolved

Couple of weeks ago I made my first Google App Engine application, using a new language for me, Python. It is made to fetch Blogger user avatars using Blogger userID. It worked/works ok, but I did some improvements to code, some structuring and formatting, generalized tag searching from html, etc.

Yesterday, as a deviation of making the comment bubbles, I added a completely new function to that application: a tool to make gradient PNG data URIs, which can be used in CSS, for example. The avatar part of the application functions just like before. I will tell you more about the gradient PNG data URI tool later in another article, and the comment bubbles, too, but here's the revised code (in my versioning it is version number 4):

Data provided by Pastebin.com - Download Raw

Again, if you have any ideas for the code, please comment. There was a minor html bug (which did not affect functionality) in the previous version, also one "None" comparison made with "==", unnecessary parenthesis, etc, and nobody told me! :)

Updates:
2011 Dec 14th, v4.01: Load also the new default icon from "www.blogger.com"
2011 Dec 18th, v4.02: Increased timeout from 5 to 15 in loading blogger profile pages, also handle download errors.
2012 March 14th, newest version 5 here.

Google App Engine Python tool to get Blogger avatar

Blogger avatar, or Profile image, is hard or even impossible to get using javascript. In data feeds there is only profile page link, but not the image on that profile page.

To get the image, one should open the profile page, parse the html, find profile image url, and use/load it. This cannot be done in Javascript, because profile page is in different domain (blogger.com). So I did it with Google App Engine. That was quite a trip: completely new programming environment and I did not know much about Python, but I managed to get done what I wanted. Phew!

My profile page: http://www.blogger.com/profile/05137522196636058302
My avatar image: http://avafavico.appspot.com/?userid=05137522196636058302

You can try it on your user id (or any other's) in this post, if you like.

Blogger userid:  

I was going to use the avatar in javascript application as 16x16 picture, so I shrank the Profile pictures to 32x32 to minimize net traffic. Avafavico is just a name... First I thought it could also return favicons to urls that are not blogger profile, but then decided not to reinvent (http://www.google.com/s2/favicons?domain=) the wheel, after that the working name was BlogAva, but that domain was taken.

Here is my Python code. I don't write Python well, so any suggestions how to optimize or make the code better, are welcome! :) Thanks.

Data provided by Pastebin.com - Download Raw

A newer version of this Python application is here.

I've used this online application/tool in my newest recent comments gadget, which can be seen on this blog. I'll tell you another day how that is done.