The reliable and valid measurement of issue salience is a key problem of agenda setting research. While exposure, awareness and salience are usually investigated using surveys, the behavioral consequences of those processes, like follow-up communication or information seeking, lend themselves very well to observation. Many theoretical models of agenda setting incorporate these concepts, but only few empirical studies exist that actually measure salience-driven behavior. We propose a new method for measuring aggregate issue salience by analyzing data from search queries typed into Google. We illustrate this approach with a case study from the 2005 Bundestag elections in Germany, focussing on the fiscal expert Paul Kirchhof of Angela Merkel’s campaign team. Using both survey and online observation data, we find substantial correlations between the two longitudinal measures of issue salience.

I’d like to see similar steps forward from the German DFG and the likes, or my university. I guess most of the faculty at UdK does not even know what OA means, and who needs to if there’s no research anyway ;-)

subparts.TAGCLOUD = RECORDS
subparts.TAGCLOUD {
tables = tt_content
source = 444
dontCheckPid = 1
}

That’s it. The plugin content is rendered without USER_INT fiddling or COA tricks, configuration is dummy proof with Flexforms which seem to be more popular than TS configuration anyway. You can also include link lists, search forms, user login or normal content etc. like this, all nicely editable by your average users.

Here’s a quick howto using Python and the BeautifulSoup HTML/XML parser.

First off, we choose a Youtube video page, like this one and stuff it into the BeautifulSoup parser:

#!/usr/bin/env python
from urllib2 import urlopen
from BeautifulSoup import BeautifulSoup
import re # we need regular expressions later

monty_vid = urlopen('http://youtube.com/watch?v=Xe1a1wHxTyo')
page = BeautifulSoup(monty_vid)

print page.prettify()

The last line pretty-prints the HTML you just retrieved, so you can check if it’s an existing page or a 404. Next, we’d like to extract some meta data like title, description and tags for the video. Luckily, those are provided in the HTML head as meta tags, in order, so we can extract the content attribute from those. The result object is a list with elements that act like dictionaries:

meta = {}
meta['title'] = page.head('meta')[0]['content']
meta['description'] = page.head('meta')[1]['content']
meta['tags'] = page.head('meta')[2]['content'].split(', ')

Notice that all the extracted strings are Unicode, and we made a list of tags by splitting the string. Next up, we want the number of views and the number of ratings. Luckily, the former is available in a span tag with a dedicated class which we can retrieve with the following search on the document body:

views = page.body('span',"viewCount")[0].string

The number of ratings is not readily marked up, but available as a string like “55 ratings”, so we need another technique — pattern matching within a certain div:

numratings = page.body('div', id='defaultRatingMessage',    
                        text=re.compile('ratings'))[0].string.split()[0]

The body() method with the text parameter gives us all tags in the named div that match our simple pattern, from which we extract the first part with split()[0]. Finally, we do not only want the number of ratings, but the rating itself. The rating is not available as a number in the document, but indicated by 0 to 5 star images which we can count:

rating = len(page.body('img','rating icn_star_full_19x20png'))

Of course, there are dozens of different ways to query the HTML for some values, but these work with the current layout. Since you have to update your scraping script with every change, why bother with optimal queries?