Google App Engine, Python 2.7, and lxml

Google App Engine (GAE) lets you build and run applications on Google's infrastructure. I have done two applications with GAE and Python (the programming language). The previous version of my tool to get blogger avatars (avafavico) used regular expressions to parse the Blogger profile page for the profile photo, or user's first contributed blog's favico, if no profile photo is found. As you might know, using regular expressions may not be the best solution. It would be better to parse html page document object model (DOM) and get results there. That way the code is not so sensitive to possible changes in the page html code. Of course I did the regular expressions so, that they should work with different html.

Previously I used GAE Python 2.5 environment, which was the default and supported version. On February 27th Python 2.7 became fully supported. GAE with Python 2.7 contains more external libraries than version 2.5, one of those libraries being lxml.

I tried to search for different solutions and examples, but there were not many. With GAE Python 2.5, one can use BeautifulSoup, but there are some issues (problematic 3.1.0 version, uncertain development future of BS, etc.). And there is minidom, but it may not handle broken html well. Blogger profile page should not have broken html, but you never know. lxml is definitely better and faster, supports XPath, etc.

The day before yesterday I updated the GAE app to use Python 2.7 and lxml. There were none to some examples about using python27, lxml and GAE, so I'll show you here a working example. First I started modifying the file app.yaml, there I changed runtime to python27, added "threadsafe" (false), and added (latest) lxml in libraries section. Increased version number to 5. Now app.yaml looks like this:

Data provided by Pastebin.com - Download Raw - See Original

In blogava.py I added "from lxml import etree" and then used etree functions instead of regular expressions to find things in html. Here's how DOM tree is constructed, variable "result" contains page html as a string, and then XPath is applied to the tree, like this:

>>> tree = etree.HTML(result)
>>> r=tree.xpath("//img[@id='profile-photo']/@src")

Here find from the tree the first img tag, which id is set to "profile-photo", and get that tag's src attribute. In the full script, if no id='profile-photo' is not found, then try to search for the first image that has class "photo". If both fails, search for first "contributed-to" blog, and use it's favicon, if that is not found, use Blogger favicon. And here is the blogava.py source file:

Data provided by Pastebin.com - Download Raw - See Original

This new version of avafavico has been up and running for two days. I'm very pleased that I got lxml working with GAE, all in all it was quite easy. Hope this example is useful to someone. If it helped you, please leave a comment. :)

YouTube/JSONP oEmbed service "oembed-js"

In the previous post I introduced my oembed-js service and an application using it: YouTube embeds in Blogger comments.

Oembed-js is a Google Appengine application coded in Python. It relays YouTube oembed calls and offers results also in JSONP format, so javascript applications can use it.

Parameters

url: the (youtube) url to embed (best to be urlencoded)
maxwidth: max width of embedded object (optional)
maxheight: max height of embedded object (optional)
format: json/jsonp or xml (optional, default:json/jsonp)
callback or jsonp: the javascript callback function, required, if jsonp format is wanted
callID: a call ID that is passed through to response (optional)

Sample call:

http://oembed-js.appspot.com/?callback=foo&callID=123&url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DGMra5Wh51oU

Response

YouTube oEmbed response in json(p) or xml format, including field html (see oembed specs), and:

callID: if callID was given, it is passed through (can identify the response)
url: pass through the url that is being embedded
videoid: contains 11 char long YouTube videoid, if single video embed
playlist: contains playlist id, if playlist embed

Sample response:

foo({"provider_url": "http://www.youtube.com/", "version": "1.0", "title": "Sviatoslav Richter in Kiev, 1958 - Pictures at an Exhibition", "url": "http://www.youtube.com/watch?v=GMra5Wh51oU", "type": "video", "videoid": "GMra5Wh51oU", "thumbnail_width": 480, "height": 344, "width": 459, "html": "<iframe width=\"459\" height=\"344\" src=\"http://www.youtube.com/embed/GMra5Wh51oU?fs=1&feature=oembed\" frameborder=\"0\" allowfullscreen></iframe>", "author_name": "FirstPublicChannel", "callID": "123", "provider_name": "YouTube", "thumbnail_url": "http://i4.ytimg.com/vi/GMra5Wh51oU/hqdefault.jpg", "thumbnail_height": 360, "author_url": "http://www.youtube.com/user/FirstPublicChannel"})

Another demo how to use it

Enter YouTube URL (video or playlist):

 

This demo and its source is also here in this jsFiddle.

Update March 27th 2012: Now oembed-js supports youtu.be urls (YouTube oembed service does not support them). Also now supports videos which have embedding turned off (YouTube oembed does not support these either).

Allow users embed YouTube videos in Blogger comments

Some time ago while googling something about video embeds I came across with oEmbed specification. This simple API allows websites to display embedded content from oEmbed providers. After some inspecting I used the oEmbed in a Python script to turn youtube links in a text into embedded videos. Worked nicely.

oEmbed returns data in JSON or XML format. That is ok with server side languages like php and Python. But for javascript this is a problem. If there is a problem, usually solution is found, and I found Embedly service, which is unfortunately not free.

With no intention to invest any money, and hands itching for another Google Appengine Python app, I saw an opportunity. To make a service that relays YouTube's oEmbed results to javascript format (JSONP). And all with memory caching etc. so it should be able to handle some traffic. After some coding it was ready and working http://oembed-js.appspot.com/?url=[youtube url to embed][&callback=foo] But now where to use it...

In Wordpress blogs, when commenting an article, user can just write/paste youtube url into the comment text, and it will be turned into a video embed. Surprise, not in Blogger. For some blogs this is a bummer. Now with my new, free, javascript/jQuery compatible GAE service oembed-js this became possible in Blogger. Of course we need to hack the blog first a bit. :)

See the demo blog!

You can decide, whether you want to turn all raw youtube links (watch and playlist links) into embeds, which is the default, or only those, that use [video=youtube_url] tag. You can also change the tag name, if you want, maybe you like youtube or embed more. If you have lazy loading installed in your blog, by default this script will use it, and pages with many videos load much faster. Installation is simple, open template html for editing, find </head> and paste this code before it:

Data provided by Pastebin.com - Download Raw - See Original

This script does not work in dynamic views, but it works with Blogger native comments, my threaded comments, and also Blogger threaded comments. Hope you like it. I'll tell you later about the oembed‑js service API (application programming interface), so it is easier for you to develop own applications.

References:
- oEmbed specification
- YouTube API blog: oEmbed support

Update 27th March 2012: Line #74 updated, to handle better the case of only a YouTube url as comment text in new Blogger threaded comments (you do not need to update if you use videotag, or don't use Blogger threaded comments).

Youtube videos lazy load, improved style

I styled my previous youtube lazy load hack a bit so that the video description text is easier to read. Before it looked like this (this is an image, so no hover effect):

HTML needed to produce this is:

<div style="text-align:center;"><a class="youtube-lazy-link" href="http://www.youtube.com/watch?v=SFiWfrLEqPw" style="width: 420px;height:315px;" title="Click to play!"><b>Mozart Fantasy in C minor</b> played by me</a></div>

With the new style, it looks like this (actual embed):

Mozart Fantasy in C minor played by me

Here's the code, place it before </head>. If you had previous version installed, you replace it with this code. And as always, you can edit CSS styles to your needs.

Data provided by Pastebin.com - Download Raw - See Original

This script is installed on this blog.

Pastebin embeds temporarily broken

I use pastebin to store most of my hacks' source codes and to embed them into this blog. Pastebin was today updated to version 3.1, and it broke the embeds. On my blog's first page the sidebar had fallen to the bottom of the page because of an imbalance in div elements' opening and closing tags. I fixed the first page, but on consequent pages the sidebar might still drop, and embeds contain this 'ss="javascript">' bug. And the code is not syntax highlighted. I notified this to pastebin admin, hope it gets fixed soon. Codes can still be copied like before by clicking the "Download Raw" link.

Update 3 hours later: Pastebin embeds fixed.

Automatic image caption from img title

I'm administering a Blogspot blog where some of the content uses img title tag to store image captions. Title tags are normally shown in a tooltip window, when user hovers the mouse over the image. There are some scripts around that display the image caption from image's title tag, but I decided to make my own. To get it quickly done, I used jQuery. This code goes before </head> tag, remove first line if you have jQuery already loaded.

Data provided by Pastebin.com - Download Raw - See Original

Now you can use the following html markup on the page:

<img class="caption" title="Image title" src="./pics/pic.jpg" />

and it will dynamically be turned into something like this:

<div class="caption-wrap">
<img class="caption" title="Image title" src="./pics/pic.jpg" />
<p class="caption-text">Image title</p>
</div>

Use the classname "caption" and a title tag for those images you wish to autocaption.

Alignment and left/right margins are copied from img to the wrapping div, so horizontal positioning of the title paragraph is (should be) the same for basic image alignment options (centered or float left/right).

Here's a link to an example article with many auto-generated image captions, the blog is in Finnish, article is about prohibition: Nobel-palkitut ja huippupoliitikot vaativat huumeiden laillistamista.

Blogger post editor's "compose mode" has possibility to set image captions. But I think compose mode is not reliable, and so I mainly use the HTML editing mode. Also the HTML markup is much simpler using title tags and this script.

If you use this, you can adjust the script and the styles for your needs. This script can also be used outside Blogger.


Update January 21st 2013: Changed the script to use $(window).load() function instead of $(document).ready(), which makes sure that outerWidth function always returns the image width correctly. Line 31 was changed.

Hack to expand/collapse Blogger sidebar gadgets

Previously I made the tabbed gadgets hack, which can save space in blog's sidebar. This hack serves the same purpose, but a bit differently. With this hack you can make your sidebar gadgets vertically collapsible, using jQuery. I have used it on this blog on some gadgets, and here is...

a demo blog.

You can choose which gadgets you make collapsible, and make them either closed or open initially. You can also make gadget groups, from which only one gadget may be open at one time (in this blog the search gadgets, and in demo blog the profile gadget and "read more" gadget).

To install, you need to have some computing skills and know your way around. Edit your blog's template, and paste the following code before </head>. If you have jQuery already loaded somewhere in your template, you can leave out the first line.

Data provided by Pastebin.com - Download Raw - See Original

Then you need to edit the window.load function at the end of the script. There you should add calls to cedgAddOpen or cedgAddClosed for all those gadgets that you want to make collapsible (and open or closed by default). Use gadget ID, preceded with #, or gadget title, as parameter. Using "#ID" is preferred method. You can easily see the IDs of the blog's gadgets at end of the template, when widget templates are not expanded.

To make a gadget group, pass several ID parameters to cedgAddOpen. The first ID's gadget will be opened initially, others closed. Example: cedgAddOpen("#Profile1", "#HTML1"); You can call cedgAddClosed again for the first gadget's ID if you want it initially closed, too.

Note: if you use also tabbed gadgets, you cannot collapse gadgets that are made into tabs.

As you can see in the CSS, the little plus and minus signs are implemented as base64 images. You can change them if you want, and they can be external images, too. Also other styles, and the gadget spacing, can be easily changed. You may also want to style the gadget titles (H2 element), I use a gradient there, here are instructions.