Skip to Main Content

We use the crowdsourcing data sets compiled by Li, Baba, and Kashima (2017) whose basic information is shown in Table 1. The task is to identify the top-k workers with the highest accuracy only from a sequential access to the accuracy of part of labels given by some workers. Notice that the number of super arms is more than 1010 in all experiments. All data sets are hard instances as Δmin is less than 0.05. We set k=10 and ɛ=0.5. Since SA-Ex and CLUCB-Ex are prohibitive, we compare the other algorithms. SAQM, SA-FOA, and ICB employ uniform allocation strategy.

Table 1:
Real-World Data Sets on Crowdsourcing.
Data SetNumber of TasksNumber of WorkersAverageBestΔmin
IT 25 36 0.54 0.84 0.04 
Medicine 36 45 0.48 0.92 0.03 
Chinese 24 50 0.37 0.79 0.04 
Pokémon 20 55 0.28 1.00 0.05 
English 30 63 0.26 0.70 0.03 
Science 20 111 0.29 0.85 0.05 
Data SetNumber of TasksNumber of WorkersAverageBestΔmin
IT 25 36 0.54 0.84 0.04 
Medicine 36 45 0.48 0.92 0.03 
Chinese 24 50 0.37 0.79 0.04 
Pokémon 20 55 0.28 1.00 0.05 
English 30 63 0.26 0.70 0.03 
Science 20 111 0.29 0.85 0.05 

Note: “Average” and “Best” give the average and the best accuracy rate among the workers, respectively.

Close Modal

or Create an Account

Close Modal
Close Modal