Over the past few years we’ve seen an increasing number of projects that take the phrase ‘human-computer interaction’ literally (or perhaps turning HCI into human-computer integration), organising tasks done by people and by computers into a unified system. One of the most obvious benefits of crowdsourcing on digital platforms has been the ability to coordinate the distribution and validation of tasks, but now data classified by people through crowdsourcing is being fed into computers to improve machine learning so that computers can learn to recognise images almost as well as we do. I’ve outlined a few projects putting this approach to work below. Of course, this creates new challenges for the future – what do cultural heritage crowdsourcing projects do when all the fun tasks like image tagging and text transcription can be done by computers? After all, Fast Company reports ‘at least one Zooniverse project, Galaxy Zoo Supernova, has already automated itself out of existence’. More positively, assuming we can find compelling reasons for people to spend time with cultural heritage collections, how does machine learning and task coordination free us to fly further?
The Public Catalogue Foundation has taken tags created through Your Paintings Tagger and turned them over to computers. As they explain, the results are impressive. The art of computer image recognition: ‘Using the 3.5 million or so tags provided by taggers, the research team at Oxford ‘educated’ image-recognition software to recognise the top tagged terms. Professor Zisserman explains this is a three stage process. Firstly, gather all paintings tagged by taggers with a particular subject (e.g. ‘horse’). Secondly, use feature extraction processes to build an ‘object model’ of a horse (a set of characteristics a painting might have that would indicate that a horse is present). Thirdly, run this algorithm over the Your Paintings database and rank paintings according to how closely they match this model.’