I’d like to see similar steps forward from the German DFG and the likes, or my university. I guess most of the faculty at UdK does not even know what OA means, and who needs to if there’s no research anyway ;-)

subparts.TAGCLOUD = RECORDS
subparts.TAGCLOUD {
tables = tt_content
source = 444
dontCheckPid = 1
}

That’s it. The plugin content is rendered without USER_INT fiddling or COA tricks, configuration is dummy proof with Flexforms which seem to be more popular than TS configuration anyway. You can also include link lists, search forms, user login or normal content etc. like this, all nicely editable by your average users.

Here’s a quick howto using Python and the BeautifulSoup HTML/XML parser.

First off, we choose a Youtube video page, like this one and stuff it into the BeautifulSoup parser:

#!/usr/bin/env python
from urllib2 import urlopen
from BeautifulSoup import BeautifulSoup
import re # we need regular expressions later

monty_vid = urlopen('http://youtube.com/watch?v=Xe1a1wHxTyo')
page = BeautifulSoup(monty_vid)

print page.prettify()

The last line pretty-prints the HTML you just retrieved, so you can check if it’s an existing page or a 404. Next, we’d like to extract some meta data like title, description and tags for the video. Luckily, those are provided in the HTML head as meta tags, in order, so we can extract the content attribute from those. The result object is a list with elements that act like dictionaries:

meta = {}
meta['title'] = page.head(’meta’)[0]['content']
meta['description'] = page.head(’meta’)[1]['content']
meta['tags'] = page.head(’meta’)[2]['content'].split(’, ‘)

Notice that all the extracted strings are Unicode, and we made a list of tags by splitting the string. Next up, we want the number of views and the number of ratings. Luckily, the former is available in a span tag with a dedicated class which we can retrieve with the following search on the document body:

views = page.body('span',"viewCount")[0].string

The number of ratings is not readily marked up, but available as a string like “55 ratings”, so we need another technique — pattern matching within a certain div:

numratings = page.body('div', id='defaultRatingMessage',    
                        text=re.compile('ratings'))[0].string.split()[0]

The body() method with the text parameter gives us all tags in the named div that match our simple pattern, from which we extract the first part with split()[0]. Finally, we do not only want the number of ratings, but the rating itself. The rating is not available as a number in the document, but indicated by 0 to 5 star images which we can count:

rating = len(page.body('img','rating icn_star_full_19x20png'))

Of course, there are dozens of different ways to query the HTML for some values, but these work with the current layout. Since you have to update your scraping script with every change, why bother with optimal queries?

(via Tobias Lütke)

There are some books and articles available specifically for beginners or switchers from SPSS, Stata or SAS, but the most valuable source is a collection of common use cases, including data management and graphs, freely available as Quick-R.

(via Dataninja)