Tracking Hamilton City Council meetings is a challenge. They often cancel and defer meetings, they announce meetings at the last-minute, and sometimes they don’t announce their meetings.
I’ve pondered creating a webspider to check the various Council and committee webpages for changes. I’ve done many spiders in the past. I run them on my desktop, they alert me of changes, and I go look at the page to find the changes.
It’s not the most efficient system, but it worked in the past for me. It doesn’t help others.
One of my goals is let open source philosophies guide my journalistic practice. I work to end gatekeeping journalism.
I’ve been inspired by the great work of the Ottawa Citizen’s Glen McGregor who created the amazing Twitter account @LobbyWatcher. LobbyWatcher is the output of a scraper that monitor’s the federal lobbyist registry for communications reports and tweets whenever there is a new filing.
My project’s goal is the same product – just for Hamilton City Council meetings.
Towards this goal, I’m taking on a rather ambitious project:
-
The Scrapers:
-
creating a series of scrapers that decipher the city meeting calendars and report on changes.
-
each month requires its own scraper as there is no mechnical structure to the city website.
-
another set of scrapers to monitor the Council and Committee agenda pages
-
the primary purpose of these scrapers is to monitor for agendas to appear, the video replay to be made available, and if possible, watch for updates to agendas on the agenda pages.
-
The RSS feed
-
Each of the scrapers will output to an RSS feed allowing for people to subscribe in their favourite RSS reader.
-
Google App Engine to push to Twitter
-
Using Google App Engine, I’ll push updates to an automated Twitter account that will tweet each change.
-
A record of crawls
-
Create a webpage – potentially automate posts to my blog – with a record of when the scraper finds changes.
It’s happened in the past that Council has called legitimate closed-door meetings without informing the media. I learn about them by chance – I spend way too much time checking for new agendas and reading them.
This will (hopefully) solve that problem and save me time.
I have a good idea of how I will achieve this project and will likely look for assistance improving efficiency