Parameterized framework for the analysis of visual quality assessments using crowdsourcing

Anthony Fremuth; Velibor Adzic; Hari Kalva

doi:10.1117/12.2080661

17 March 2015 Parameterized framework for the analysis of visual quality assessments using crowdsourcing

Anthony Fremuth, Velibor Adzic, Hari Kalva

Proceedings Volume 9394, Human Vision and Electronic Imaging XX; 93940C (2015) https://doi.org/10.1117/12.2080661
Event: SPIE/IS&T Electronic Imaging, 2015, San Francisco, California, United States

Abstract

The ability to assess the quality of new multimedia tools and applications relies heavily on the perception of the end user. In order to quantify the perception, subjective tests are required to evaluate the effectiveness of new technologies. However, the standard for subjective user studies requires a highly controlled test environment and is costly in terms of both money and time. To circumvent these issues we are utilizing crowdsourcing platforms such as CrowdFlower and Amazon's Mechanical Turk. The reliability of the results relies on factors that are not controlled and can be considered “hidden”. We are using pre-test survey to collect responses from subjects that reveal some of the hidden factors. Using statistical analysis we build parameterized model allowing for proper adjustments to collected test scores.

Citation Download Citation

Anthony Fremuth, Velibor Adzic, and Hari Kalva "Parameterized framework for the analysis of visual quality assessments using crowdsourcing", Proc. SPIE 9394, Human Vision and Electronic Imaging XX, 93940C (17 March 2015); https://doi.org/10.1117/12.2080661

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available