Crowdsourcing & Citizen Science

Editor's Notes

1. From the different research topics I am involved in today I address some of the results from projects in which we use explicit semantics and user interaction to make crowd knowledge effectively processable for machines\n\n\n*****\nFrom Crowd Knowledge to Machine Knowledge: Use cases with semantics and user interaction in Dutch cultural heritage collections\n\nIn this talk I will discuss several projects, for example with the Dutch national archive for sound and vision or with the Rijksmuseum Amsterdam, where we have experimented with semantics-based technologies and user interaction paradigms to provide systems with additional support for users to turn their lay knowledge into machine-processable knowledge. Turning this crowd knowledge into machine knowledge makes the system more intelligent and thus makes them capitalize on the knowledge assets in the crowds of users.\n\nThe problem our system addresses is concerned with making the massive set of interesting multimedia content available in these cultural heritage institutions accessible for a large community of users.  On the one hand, this content is difficult to find for the average user as they have been indexed by experts (curators, art historians, etc) who use a very specific vocabulary that is unknown to the general audience.  On the other hand, these professionals can no longer  cope with the demand for annotation on the ever growing multimedia content in these collections.\n\nOur solution to both these problem is to exploit crowdsourcing on the web, where there is a lot of specific domain knowledge and processing power available that such archives and museums would gladly incorporate. However, turning this knowledge from the mass of lay users into additional intelligence in content management systems is not a simple challenge for several reasons: (1) the lay users use different vocabularies and are also interested in different aspects of the collection items; (2) the collected user metadata needs to be validated for quality and correctness; (3) crowd-sourcing users need to be continuously supported and motivated in order to supply more and better knowledge in such systems.  I will present our data analysis of the crowdsourced content, our techniques for quality control and incorporating domain semantics, and the results of our attempts to employ different interaction strategies and user interfaces (for example games and simplified interfaces for interactive annotation) to engage and stimulate the users.\n\nDemos:\n-----------\nVideo-tagging game: http://woordentikkertje.manbijthond.nl/\nArt recommender and personalized museum tours: http://www.chip-project.org/demo/ (3rd prize winner of ISWC Semantic Web challenge)\nHistorical events in museum collections: http://agora.cs.vu.nl/demo/\n\nRelevant Papers:\n------------------------\n- On the role of user-generated metadata in audio visual collections, at K-CAP2011\nhttp://portal.acm.org/citation.cfm?id=1999702\n\n- Digital Hermeneutics:  Agora and the Online Understanding of Cultural Heritage, at WebSci2011, (Best paper nominee)\nhttp://www.websci11.org/fileadmin/websci/Papers/116_paper.pdf\n\n- The effects of transparency on trust in and acceptance of a content-based art recommender, UMUAI journal, (Best journal paper for 2008)\nhttp://www.springerlink.com/content/81q34u73mpp58u75/\n\n- Recommendations based on semantically enriched museum collections, Journal of Web Semantics\nhttp://www.sciencedirect.com/science/article/pii/S1570826808000681\n\n- Enhancing Content-Based Recommendation with the Task Model of Classification, at EKAW2010\nhttp://www.springerlink.com/content/p78hl5r283x79r13/\n\nShort bio:\n-----------------\nLora Aroyo is an associate professor at the Web and Media group, at the Department of Computer Science, Free University Amsterdam, The Netherlands. Her research interests are in using semantic web technologies for modeling user interests and context, recommendation systems and personalized access in Web-based applications. Typical example domains are cultural heritage collections, multimedia archives and interactive TV. She has coordinated the research work in the CHIP project on Cultural Heritage Information Personalization (http://chip-project.org). Currently she is a scientific coordinator of the EU Integrated Project NoTube dealing with the integration of Web and TV data with the help of semantics (http://notube.tv), a project leader of VU INTERTAIN Experimental Research Lab initiative (http://www.cs.vu.nl/intertain), and involved in the research on motivational user interaction for video-tagging games in the PrestoPrime project (http://www.prestoprime.org/) and modeling\nhistoric events in the Agora project (http://agora.cs.vu.nl/). She has organized numerous workshops in the areas of personalized access to cultural heritage, e-learning, interactive television, as well as on visual interfaces to the social and semantic web (PATCH, FutureTV, PersWeb, VISSW and DeRIVE). Lora has been actively involved in both the Semantic Web (PC co-chair and conference chair for ESWC2009 and ESWC2010 and PC co-chair for ISWC2011) and the Personalization and User modeling communities (on the editorial board for the UMUAI journal and on the steering committee of UMAP conference).\n\nMore information can be found at:\n-----------------------------------------------\nWebpage: http://www.cs.vu.nl/~laroyo\nSlideshare: http://www.slideshare.net/laroyo\nTwitter: @laroyo\n\n
2. Nowadays av collections are undergoing process of transformation from archives of analog material to large digital (online) data stores, as videos are very much wanted by different types of end users. \n\nFor example, the Netherlands Institute of Sound and Vision archives all radio and TV material broadcasted in the Netherlands (has appr. 700,000 hours radio and television programs available online. \n\nFacilitating a successfully access to av collection items demands quality metadata associated with them.\n\nTraditionally, in AV achives it is the task of professional catalogers to manually describe the videos. Usually, in the process they follow well-defined , well-established guidelines and rules. They also may make use of auxiliary materials like controlled vocabularies, thesauri, and such.\nHowever, as we all know video is medium that is extremely rich in meaning. Directors and screenwriters create entire universes with complex interplay between characters, objects and events. Sometimes they may employ rich and complex abstract symbolic language. This makes that task of describing the meaning of a video as complicated as describing the real worlds. Which is no trivial matter.\nAs a result the process of annotation is tedious, time-consuming and inevitably incomplete. According to some research, it takes approximately 5 times of the duration of the material to annotate it completely. So for example, if we are talking about a documentary that lasts one hour, it will take approximately 5 hours for a cataloger to fully describe it. Furthermore,\n\nConsequently, professional annotations are coarse-grained in a sense that they are referring to the entire video describing prevalent topics. It may happen that catalogers provide more fine-grained, shot-level descriptions for a video. But this is exception of the rule and it is reserved to the most important pieces of the AV collection.\n
3. \n* Search Behavior of Media Professionals at an Audiovisual Archive: A Transaction Log Analysis\nB. Huurnink, L. Hollink, W. van den Heuvel, and M. de Rijke.\n\n
4. \n
5. The problem our system addresses is concerned with making the massive set of interesting multimedia content available in these cultural heritage institutions accessible for a large community of users.  On the one hand, this content is difficult to find for the average user as they have been indexed by experts (curators, art historians, etc) who use a very specific vocabulary that is unknown to the general audience.  On the other hand, these professionals can no longer  cope with the demand for annotation on the ever growing multimedia content in these collections.\n
6. The problem our system addresses is concerned with making the massive set of interesting multimedia content available in these cultural heritage institutions accessible for a large community of users.  On the one hand, this content is difficult to find for the average user as they have been indexed by experts (curators, art historians, etc) who use a very specific vocabulary that is unknown to the general audience.  On the other hand, these professionals can no longer  cope with the demand for annotation on the ever growing multimedia content in these collections.\n\nRecommendation of video fragments\nLinks between video fragments\nLinks to background information\nintegrate the Goal with the general --> merge the two slides --> one slide for RMA and Europeana\n\n
7. The problem our system addresses is concerned with making the massive set of interesting multimedia content available in these cultural heritage institutions accessible for a large community of users.  On the one hand, this content is difficult to find for the average user as they have been indexed by experts (curators, art historians, etc) who use a very specific vocabulary that is unknown to the general audience.  On the other hand, these professionals can no longer  cope with the demand for annotation on the ever growing multimedia content in these collections.\n\nRecommendation of video fragments\nLinks between video fragments\nLinks to background information\nintegrate the Goal with the general --> merge the two slides --> one slide for RMA and Europeana\n\n
8. The problem our system addresses is concerned with making the massive set of interesting multimedia content available in these cultural heritage institutions accessible for a large community of users.  On the one hand, this content is difficult to find for the average user as they have been indexed by experts (curators, art historians, etc) who use a very specific vocabulary that is unknown to the general audience.  On the other hand, these professionals can no longer  cope with the demand for annotation on the ever growing multimedia content in these collections.\n\nRecommendation of video fragments\nLinks between video fragments\nLinks to background information\nintegrate the Goal with the general --> merge the two slides --> one slide for RMA and Europeana\n\n
9. Our solution to both these problem is to exploit crowdsourcing on the web, where there is a lot of specific domain knowledge and processing power available that such archives and museums would gladly incorporate. However, turning this knowledge from the mass of lay users into additional intelligence in content management systems is not a simple challenge for several reasons: (1) the lay users use different vocabularies and are also interested in different aspects of the collection items; (2) the collected user metadata needs to be validated for quality and correctness; (3) crowd-sourcing users need to be continuously supported and motivated in order to supply more and better knowledge in such systems.  I will present our data analysis of the crowdsourced content, our techniques for quality control and incorporating domain semantics, and the results of our attempts to employ different interaction strategies and user interfaces (for example games and simplified interfaces for interactive annotation) to engage and stimulate the users.\n\nunderstanding the user-generated data\ncontextualize the user-generated metadata \n\n\n
10. supplement video data with background knowledge\n introduce the context, rijksmuseum\n the general picture, examples, CH landscape\n \n \n
11. \n
12. \n
13. \n
14. We hebben naar uiteenlopende voorbeelden gekeken binnen de sector en deze geclusterd\n\ncriterium - voorname/leidende rol erfgoed. Uiteraard zijn er ook veel grassroots initiatieven. \n\nEen van mijn favoriete voorbeelden is de International Amateur Scanning league.\nDigitaliseren van historische video’s in het publieke domein.\n\n\nwe komen vervolgens op een indeling in zes categorieen. \n\nwe hebben al goede voorbeelden gezien vandaag - ik zal met name voorbeelden noemen uit categorieen die niet in andere presentaties aan bod komen.\n\n
15. \n
16. Wikipedia as a compelling example --> calculating that creating Wikipedia as it stands today has taken one hundred\nmillion hours of cumulative thought, he juxtaposes this to the astounding 200 billion hours people watch TV in the\nUS alone. 200 billion hours would amount to two thousand Wikipedia projects-worth of free time, annually. \n\n
17. \n
18. three types of users\ncollect: gamers\nmanage: domain experts\nintegrate: collection maintainer\n
19. explain games with purpose\n\nplay on words\n\n
20. \n
21. multi-player game \n 4 channels each continuously streaming videos from a predefined category\n a player starts a game by selecting one of the channels\n plays agains all the players who selected the same channel\n\n
22. multi-player game \n users score points by entering tags (based on the principle of Louis van Ahn’s ESP games\n points are scored when two or more players enter the same tag\n as in the Yahoo! video tag game --> this is done in a period of 10 sec\n each tag is shown together with the score it has earned\n\nThe relevance of the tags is ensured by two factors: independence of players and consensus among players. The players don’t know each other, they are grouped randomly, and the only thing they share at the moment when they are playing the game is the video. So if they enter the same tag within a time frame the changes are the tag is relevant for the video.\n
23. the changes:\n- shorter videos (~3 mins), independent fragments vs. fragmented videos\n- improved scoring (including matches in Cornetto and GTAA; first time tag)\n- game recap with continuous scoring\n- profile page\n\nbots again\n- different handling of users to start playing the game\n\n
24. total # tags 32,248 (25,685 first time tags)\n 19% are one time tags (81% appear more than once)\n \n 257 person names in GTAA, 83 also validated\n 1,661 geo names in GTAA, 666 also validated\n 9,796 validated, but no match in GTAA\n\n
25. \n
26. \n
27. \n
28. First step manual\n\nGoal is to automate it and make it an integral part of Waisda\n\nEventually the output can be expressed in a quality metric\n
29. \n\n
30. How/why did you select those fragments\nHow did you do the classifacation\n
31. OK some statistics. These stats are computed on the dataset that was accumulated by waisda? in the first six months after it went live \n
32. The goal of our first data analysis is to investigate user terminology and compare it to the terminology used by professionals. We de\n
33. Google search - a tag is meaningful if the number of pages returned by Google is positive\nFor about 89% of the tags - not found in vocabs or not verified --> google returned positive # of hits\n200 Sample of the ‘no-hits’ (zero sample)\n200 Sample of the ‘hits’ (pos sample)\n\nThings heard or things seen on the screen --> subtitles (for heard)\n\nComparable with Steve.museum\n
34. Google search - a tag is meaningful if the number of pages returned by Google is positive\nFor about 89% of the tags - not found in vocabs or not verified --> google returned positive # of hits\n200 Sample of the ‘no-hits’ (zero sample)\n200 Sample of the ‘hits’ (pos sample)\n\nThings heard or things seen on the screen --> subtitles (for heard)\n\nComparable with Steve.museum\n
35. Google search - a tag is meaningful if the number of pages returned by Google is positive\nFor about 89% of the tags - not found in vocabs or not verified --> google returned positive # of hits\n200 Sample of the ‘no-hits’ (zero sample)\n200 Sample of the ‘hits’ (pos sample)\n\nThings heard or things seen on the screen --> subtitles (for heard)\n\nComparable with Steve.museum\n
36. Google search - a tag is meaningful if the number of pages returned by Google is positive\nFor about 89% of the tags - not found in vocabs or not verified --> google returned positive # of hits\n200 Sample of the ‘no-hits’ (zero sample)\n200 Sample of the ‘hits’ (pos sample)\n\nThings heard or things seen on the screen --> subtitles (for heard)\n\nComparable with Steve.museum\n
37. Google search - a tag is meaningful if the number of pages returned by Google is positive\nFor about 89% of the tags - not found in vocabs or not verified --> google returned positive # of hits\n200 Sample of the ‘no-hits’ (zero sample)\n200 Sample of the ‘hits’ (pos sample)\n\nThings heard or things seen on the screen --> subtitles (for heard)\n\nComparable with Steve.museum\n
38. Google search - a tag is meaningful if the number of pages returned by Google is positive\nFor about 89% of the tags - not found in vocabs or not verified --> google returned positive # of hits\n200 Sample of the ‘no-hits’ (zero sample)\n200 Sample of the ‘hits’ (pos sample)\n\nThings heard or things seen on the screen --> subtitles (for heard)\n\nComparable with Steve.museum\n
39. Google search - a tag is meaningful if the number of pages returned by Google is positive\nFor about 89% of the tags - not found in vocabs or not verified --> google returned positive # of hits\n200 Sample of the ‘no-hits’ (zero sample)\n200 Sample of the ‘hits’ (pos sample)\n\nThings heard or things seen on the screen --> subtitles (for heard)\n\nComparable with Steve.museum\n
40. Google search - a tag is meaningful if the number of pages returned by Google is positive\nFor about 89% of the tags - not found in vocabs or not verified --> google returned positive # of hits\n200 Sample of the ‘no-hits’ (zero sample)\n200 Sample of the ‘hits’ (pos sample)\n\nThings heard or things seen on the screen --> subtitles (for heard)\n\nComparable with Steve.museum\n
41. non-visual --> 0\nperceptual --> 11 tags (referring to color)\nconceptual --> the rest (describing objects, 30 about scenes only)\n
42. 30% of scenes are conceptual (by Laura Hollink)\n\n
43. 30% of scenes are conceptual (by Laura Hollink)\n\n
44. 30% of scenes are conceptual (by Laura Hollink)\n\n
45. \n
46. genre is influencing the specificity and usefulness\nusers focus on what can be seen or heard\nusers focus on directly perceivable objects - because of the pace\n\n
47. \n
48. Our solution to both these problem is to exploit crowdsourcing on the web, where there is a lot of specific domain knowledge and processing power available that such archives and museums would gladly incorporate. However, turning this knowledge from the mass of lay users into additional intelligence in content management systems is not a simple challenge for several reasons: (1) the lay users use different vocabularies and are also interested in different aspects of the collection items; (2) the collected user metadata needs to be validated for quality and correctness; (3) crowd-sourcing users need to be continuously supported and motivated in order to supply more and better knowledge in such systems.  I will present our data analysis of the crowdsourced content, our techniques for quality control and incorporating domain semantics, and the results of our attempts to employ different interaction strategies and user interfaces (for example games and simplified interfaces for interactive annotation) to engage and stimulate the users.\n\n
49. Our solution to both these problem is to exploit crowdsourcing on the web, where there is a lot of specific domain knowledge and processing power available that such archives and museums would gladly incorporate. However, turning this knowledge from the mass of lay users into additional intelligence in content management systems is not a simple challenge for several reasons: (1) the lay users use different vocabularies and are also interested in different aspects of the collection items; (2) the collected user metadata needs to be validated for quality and correctness; (3) crowd-sourcing users need to be continuously supported and motivated in order to supply more and better knowledge in such systems.  I will present our data analysis of the crowdsourced content, our techniques for quality control and incorporating domain semantics, and the results of our attempts to employ different interaction strategies and user interfaces (for example games and simplified interfaces for interactive annotation) to engage and stimulate the users.\n\n
50. Our solution to both these problem is to exploit crowdsourcing on the web, where there is a lot of specific domain knowledge and processing power available that such archives and museums would gladly incorporate. However, turning this knowledge from the mass of lay users into additional intelligence in content management systems is not a simple challenge for several reasons: (1) the lay users use different vocabularies and are also interested in different aspects of the collection items; (2) the collected user metadata needs to be validated for quality and correctness; (3) crowd-sourcing users need to be continuously supported and motivated in order to supply more and better knowledge in such systems.  I will present our data analysis of the crowdsourced content, our techniques for quality control and incorporating domain semantics, and the results of our attempts to employ different interaction strategies and user interfaces (for example games and simplified interfaces for interactive annotation) to engage and stimulate the users.\n\n\nCan user tags be used to derive topical descriptions for scenes?\n\n
51. \n
52. \n
53. Bridge communities \n crowd\n professionals\n Combine perspectives\n (subjective) user perspective\n (objective) professional perspective\n Link tags to concepts\n Annotations of video scenes \n Quality assurance by moderation\n
54. \n
55. \n
56. \n
57. frames per tag!\nview all frames for a tag\nsorted by frequency\nFrequently occurring tags for scene descriptions\n
58. Selecting a role\n
59. Selecting a role\n
60. Link tags to concepts\n
61. not always easy,\nwordnet: polysemi, interpretations are close\nsome vocabularies have no description\n\nhow do tags relate to a shot\n- the frame is often a bit later\n
62. \n
63. \n
64. publicly available reconciliation APIs: e.g. Freebase, Europeana, Talis (kasabi.com)\n\n
65. \n
66. krippendorff alpha\n\nWe conclude that for this small experiment, with a coverage of 33 out of 36 tags and a total reliability coefficient of 0.91, users can effectively reconcile tags using the selected services.\n\nMore than half of the selected concepts were ranked first (32), while 23 concepts were ranked second or lower. Several (6) selected concepts were even ranked sixth or lower\nThus, often the first concept could be automatically selected, but not always\n\n\n
67. \n
68. \n
69. \n
70. \n
71. \n
72. \n
73. Open Archives Initiative Object Reuse and Exchange (OAI-ORE) defines standards for the description and exchange of aggregations of Web resources. The goal of these standards is to expose the rich content in these aggregations to applications that support authoring, deposit, exchange, visualization, reuse, and preservation. \n
74. \n
75. \n
76. \n
77. - group by type (faceted)\n- concept is uniquely identified (see full name for Juliana and Bernhard)\n- background information\n
78. - group by type (faceted)\n- concept is uniquely identified (see full name for Juliana and Bernhard)\n- background information\n
79. - group by type (faceted)\n- concept is uniquely identified (see full name for Juliana and Bernhard)\n- background information\n
80. - group by type (faceted)\n- concept is uniquely identified (see full name for Juliana and Bernhard)\n- background information\n
81. - group by type (faceted)\n- concept is uniquely identified (see full name for Juliana and Bernhard)\n- background information\n
82. \n
83. \n