The result is shown in Table 2, which indicates the applicability of our algorithms to the instances with a massive number of super arms. Moreover, all algorithms found the optimal subset of crowdworkers. In all data sets, SA-FOA outperformed the other algorithms. Recall that ICB uses the simplified confidence bound for the gap between two super arms. On the other hand, SA-FOA uses the approximation of the confidence ellipsoids for the gap between two super arms, which results in better performance than ICB. SAQM approximately computes the maximal confidence ellipsoid bound for the reward of one super arm rather than the gap between two super arms, which may result in worse performance than SA-FOA. CLUCB-QM, which employs the same sampling rule as CLUCB and the same stopping rule as SAQM, performed better than CLUCB and SAQM. This result may indicate that an adaptive sampling rule is more desirable than a static sampling rule, and using a confidence ellipsoid is more desirable than considering an individual confidence bound. ME, LUCB, and CLUCB discard the information from $k-1$ arms at each pull, which may cause the unfavorable results. LUCB worked better than CLUCB, since the original version of CLUCB was designed for very general combinatorial constraints, while LUCB was designed only for the top-$k$ setting. Notice that ME is phased adaptive while LUCB is fully adaptive, ME performed poorly in all instances, although ME is the counterpart of LUCB.

Table 2:
Number of Samples ($×102$) on Real-World Crowdsourcing Data Sets.
Data SetSAQMSA-FOACLUCB-QMCLUCBMELUCBICB
IT 62,985 1437 9896 405,313 111,603 91,442 43,773
Medicine 96,174 865 15,678 400,953 139,504 109,124 66,468
Chinese 88,209 1060 19,438 754,439 301,635 129,795 99,424
Pokémon 83,209 328 1994 151,748 331,799 89,674 19,705
English 121,890 1023 31,300 671,274 380,060 117,611 114,406
Science 276,325 1505 100,950 1,825,106 1,292,074 224,494 418,155
Data SetSAQMSA-FOACLUCB-QMCLUCBMELUCBICB
IT 62,985 1437 9896 405,313 111,603 91,442 43,773
Medicine 96,174 865 15,678 400,953 139,504 109,124 66,468
Chinese 88,209 1060 19,438 754,439 301,635 129,795 99,424
Pokémon 83,209 328 1994 151,748 331,799 89,674 19,705
English 121,890 1023 31,300 671,274 380,060 117,611 114,406
Science 276,325 1505 100,950 1,825,106 1,292,074 224,494 418,155

Note: Each value is an average over 10 realizations.

Close Modal