Precision (P), Recall (R) and F1 on the three journals when the text from the methodology section (if available) is ignored. “–” indicates that no articles are classified under method by the coders, and neither does our method predict that method for any articles. Bold indicates better results whereas underline indicates worse results compared to Table 4. For the absolute number of instances for each method, see Figure 5
| Method . | JDoc . | LISR . | JASIS&T . | ||||||
|---|---|---|---|---|---|---|---|---|---|
| P . | R . | F1 . | P . | R . | F1 . | P . | R . | F1 . | |
| Methods from Chu (2015) | |||||||||
| bibliometrics | 0.917 | 1.00 | 0.957 | 1.00 | 0.833 | 0.909 | 0.846 | 0.846 | 0.846 |
| content analysis | 1.00 | 0.462 | 0.632 | 0.444 | 0.363 | 0.400 | 0.500 | 0.667 | 0.571 |
| delphi study | 0 | 0 | 0 | 1.00 | 1.00 | 1.00 | – | – | – |
| ethnography/field study | 0.875 | 0.875 | 0.875 | 0.250 | 1.00 | 0.400 | – | – | – |
| experiment | 1.00 | 0.714 | 0.833 | 1.00 | 0.500 | 0.667 | 0.071 | 0.250 | 0.111 |
| focus group | 0.500 | 0.500 | 0.500 | 0.833 | 0.833 | 0.833 | – | – | – |
| historical method | 1.00 | 1.00 | 1.00 | – | – | – | – | – | – |
| interview | 0.821 | 0.821 | 0.821 | 0.885 | 0.766 | 0.821 | 1.00 | 0.500 | 0.667 |
| observation | 0.429 | 1.00 | 0.600 | 0.600 | 0.429 | 0.500 | 0 | 0 | 0 |
| questionnaire | 0.600 | 0.750 | 0.667 | 0.891 | 0.891 | 0.891 | 0.333 | 0.667 | 0.444 |
| research diary/journal | – | – | – | 0 | 0 | 0 | – | – | – |
| think aloud protocol | – | – | – | 0.500 | 1.00 | 0.667 | – | – | – |
| transaction log analysis | – | – | – | 0 | 0 | 0 | – | – | – |
| webometric | – | – | – | – | – | – | – | – | – |
| theoretical studies | 0.324 | 0.923 | 0.480 | 0.457 | 0.889 | 0.604 | 0.476 | 0.769 | 0.588 |
| New | |||||||||
| agent based modeling | – | – | – | – | – | – | 1.00 | 1.00 | 1.00 |
| annotation | – | – | – | – | – | – | 0 | 0 | 0 |
| classification | 0.167 | 1.00 | 0.286 | – | – | – | 0.467 | 0.583 | 0.519 |
| clustering | 0 | 0 | 0 | – | – | – | 0.833 | 1.00 | 0.909 |
| comparative evaluation | – | – | – | – | – | – | 0 | 0 | 0 |
| document analysis | 1.00 | 0.286 | 0.444 | 1.00 | 0.500 | 0.667 | 0 | 0 | 0 |
| information extraction | – | – | – | – | – | – | 1.00 | 0.250 | 0.400 |
| IR related indexing/ranking/query methods | – | – | – | – | – | – | 0.67 | 0.25 | 0.363 |
| mixed method | 0.667 | 1.000 | 0.800 | – | – | – | – | – | – |
| digital ethnography | 1.00 | 1.00 | 1.00 | – | – | – | – | – | – |
| network analysis | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.750 | 0.333 | 0.462 |
| statistical methods | 0 | 0 | 0 | 0.167 | 1.00 | 0.286 | 0.500 | 0.444 | 0.471 |
| topic modeling | – | – | – | – | – | – | 1.00 | 0.500 | 0.667 |
| user task based study | 0 | 0 | 0 | 0 | 0 | 0 | 1.00 | 0.500 | 0.667 |
| AVERAGE | 0.804 | 0.804 | 0.804 | 0.812 | 0.812 | 0.812 | 0.610 | 0.610 | 0.610 |
| Method . | JDoc . | LISR . | JASIS&T . | ||||||
|---|---|---|---|---|---|---|---|---|---|
| P . | R . | F1 . | P . | R . | F1 . | P . | R . | F1 . | |
| Methods from Chu (2015) | |||||||||
| bibliometrics | 0.917 | 1.00 | 0.957 | 1.00 | 0.833 | 0.909 | 0.846 | 0.846 | 0.846 |
| content analysis | 1.00 | 0.462 | 0.632 | 0.444 | 0.363 | 0.400 | 0.500 | 0.667 | 0.571 |
| delphi study | 0 | 0 | 0 | 1.00 | 1.00 | 1.00 | – | – | – |
| ethnography/field study | 0.875 | 0.875 | 0.875 | 0.250 | 1.00 | 0.400 | – | – | – |
| experiment | 1.00 | 0.714 | 0.833 | 1.00 | 0.500 | 0.667 | 0.071 | 0.250 | 0.111 |
| focus group | 0.500 | 0.500 | 0.500 | 0.833 | 0.833 | 0.833 | – | – | – |
| historical method | 1.00 | 1.00 | 1.00 | – | – | – | – | – | – |
| interview | 0.821 | 0.821 | 0.821 | 0.885 | 0.766 | 0.821 | 1.00 | 0.500 | 0.667 |
| observation | 0.429 | 1.00 | 0.600 | 0.600 | 0.429 | 0.500 | 0 | 0 | 0 |
| questionnaire | 0.600 | 0.750 | 0.667 | 0.891 | 0.891 | 0.891 | 0.333 | 0.667 | 0.444 |
| research diary/journal | – | – | – | 0 | 0 | 0 | – | – | – |
| think aloud protocol | – | – | – | 0.500 | 1.00 | 0.667 | – | – | – |
| transaction log analysis | – | – | – | 0 | 0 | 0 | – | – | – |
| webometric | – | – | – | – | – | – | – | – | – |
| theoretical studies | 0.324 | 0.923 | 0.480 | 0.457 | 0.889 | 0.604 | 0.476 | 0.769 | 0.588 |
| New | |||||||||
| agent based modeling | – | – | – | – | – | – | 1.00 | 1.00 | 1.00 |
| annotation | – | – | – | – | – | – | 0 | 0 | 0 |
| classification | 0.167 | 1.00 | 0.286 | – | – | – | 0.467 | 0.583 | 0.519 |
| clustering | 0 | 0 | 0 | – | – | – | 0.833 | 1.00 | 0.909 |
| comparative evaluation | – | – | – | – | – | – | 0 | 0 | 0 |
| document analysis | 1.00 | 0.286 | 0.444 | 1.00 | 0.500 | 0.667 | 0 | 0 | 0 |
| information extraction | – | – | – | – | – | – | 1.00 | 0.250 | 0.400 |
| IR related indexing/ranking/query methods | – | – | – | – | – | – | 0.67 | 0.25 | 0.363 |
| mixed method | 0.667 | 1.000 | 0.800 | – | – | – | – | – | – |
| digital ethnography | 1.00 | 1.00 | 1.00 | – | – | – | – | – | – |
| network analysis | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.750 | 0.333 | 0.462 |
| statistical methods | 0 | 0 | 0 | 0.167 | 1.00 | 0.286 | 0.500 | 0.444 | 0.471 |
| topic modeling | – | – | – | – | – | – | 1.00 | 0.500 | 0.667 |
| user task based study | 0 | 0 | 0 | 0 | 0 | 0 | 1.00 | 0.500 | 0.667 |
| AVERAGE | 0.804 | 0.804 | 0.804 | 0.812 | 0.812 | 0.812 | 0.610 | 0.610 | 0.610 |