Precision (P), Recall (R) and F1 on the three journals. “–” indicates that no articles are classified under method by the coders; neither does our method predict that method for any articles. For the absolute number of instances for each method, see Figure 5
| Method . | JDoc . | LISR . | JASIS&T . | ||||||
|---|---|---|---|---|---|---|---|---|---|
| P . | R . | F1 . | P . | R . | F1 . | P . | R . | F1 . | |
| Methods from Chu (2015) | |||||||||
| bibliometrics | 0.917 | 1.00 | 0.957 | 1.00 | 0.833 | 0.909 | 0.846 | 0.846 | 0.846 |
| content analysis | 0.857 | 0.462 | 0.600 | 0.462 | 0.462 | 0.462 | 0.500 | 0.667 | 0.571 |
| delphi study | 0 | 0 | 0 | 1.00 | 1.00 | 1.00 | — | — | — |
| ethnography/field study | 0.875 | 0.875 | 0.875 | 0.250 | 1.00 | 0.400 | — | — | — |
| experiment | 1.00 | 0.714 | 0.833 | 0.400 | 0.667 | 0.500 | 0.071 | 0.250 | 0.111 |
| focus group | 0.500 | 0.500 | 0.500 | 0.667 | 0.750 | 0.706 | — | — | — |
| historical method | 1.00 | 1.00 | 1.00 | — | — | — | — | — | — |
| interview | 0.793 | 0.821 | 0.807 | 0.735 | 0.781 | 0.758 | 1.00 | 0.500 | 0.667 |
| observation | 0.375 | 1.00 | 0.545 | 0.444 | 0.500 | 0.471 | 0 | 0 | 0 |
| questionnaire | 0.600 | 0.750 | 0.667 | 0.827 | 0.915 | 0.869 | 0.333 | 0.667 | 0.444 |
| research diary/journal | – | – | – | 0 | 0 | 0 | – | – | – |
| think aloud protocol | – | – | – | 0.250 | 1.00 | 0.400 | – | – | – |
| transaction log analysis | – | – | – | 0 | 0 | 0 | – | – | – |
| webometric | – | – | – | – | – | – | – | – | – |
| theoretical studies | 0.310 | 0.923 | 0.462 | 0.474 | 0.857 | 0.610 | 0.476 | 0.769 | 0.588 |
| New | |||||||||
| agent based modeling | – | – | – | – | – | – | 1.00 | 1.00 | 1.00 |
| annotation | – | – | – | – | – | – | 0 | 0 | 0 |
| classification | 0.143 | 1.00 | 0.250 | – | – | – | 0.467 | 0.583 | 0.519 |
| clustering | 0 | 0 | 0 | – | – | – | 0.833 | 1.00 | 0.909 |
| comparative evaluation | – | – | – | – | – | – | 0 | 0 | 0 |
| document analysis | 1.00 | 0.200 | 0.333 | 1.00 | 0.600 | 0.750 | 0 | 0 | 0 |
| information extraction | – | – | – | – | – | – | 1.00 | 0.250 | 0.400 |
| IR related indexing/ranking/query methods | – | – | – | – | – | – | 0.67 | 0.25 | 0.363 |
| mixed method | 0.667 | 1.000 | 0.800 | – | – | – | – | – | – |
| digital ethnography | 1.00 | 1.00 | 1.00 | – | – | – | – | – | – |
| network analysis | 0 | 0 | 0 | 1.00 | 1.00 | 1.00 | 0.750 | 0.333 | 0.462 |
| statistical methods | 0 | 0 | 0 | 0.167 | 1.00 | 0.286 | 0.500 | 0.444 | 0.471 |
| topic modeling | – | – | – | – | – | – | 1.00 | 0.500 | 0.667 |
| user task based study | 0 | 0 | 0 | 0 | 0 | 0 | 1.00 | 0.500 | 0.667 |
| AVERAGE7 | 0.783 | 0.783 | 0.783 | 0.811 | 0.811 | 0.811 | 0.610 | 0.610 | 0.610 |
| Method . | JDoc . | LISR . | JASIS&T . | ||||||
|---|---|---|---|---|---|---|---|---|---|
| P . | R . | F1 . | P . | R . | F1 . | P . | R . | F1 . | |
| Methods from Chu (2015) | |||||||||
| bibliometrics | 0.917 | 1.00 | 0.957 | 1.00 | 0.833 | 0.909 | 0.846 | 0.846 | 0.846 |
| content analysis | 0.857 | 0.462 | 0.600 | 0.462 | 0.462 | 0.462 | 0.500 | 0.667 | 0.571 |
| delphi study | 0 | 0 | 0 | 1.00 | 1.00 | 1.00 | — | — | — |
| ethnography/field study | 0.875 | 0.875 | 0.875 | 0.250 | 1.00 | 0.400 | — | — | — |
| experiment | 1.00 | 0.714 | 0.833 | 0.400 | 0.667 | 0.500 | 0.071 | 0.250 | 0.111 |
| focus group | 0.500 | 0.500 | 0.500 | 0.667 | 0.750 | 0.706 | — | — | — |
| historical method | 1.00 | 1.00 | 1.00 | — | — | — | — | — | — |
| interview | 0.793 | 0.821 | 0.807 | 0.735 | 0.781 | 0.758 | 1.00 | 0.500 | 0.667 |
| observation | 0.375 | 1.00 | 0.545 | 0.444 | 0.500 | 0.471 | 0 | 0 | 0 |
| questionnaire | 0.600 | 0.750 | 0.667 | 0.827 | 0.915 | 0.869 | 0.333 | 0.667 | 0.444 |
| research diary/journal | – | – | – | 0 | 0 | 0 | – | – | – |
| think aloud protocol | – | – | – | 0.250 | 1.00 | 0.400 | – | – | – |
| transaction log analysis | – | – | – | 0 | 0 | 0 | – | – | – |
| webometric | – | – | – | – | – | – | – | – | – |
| theoretical studies | 0.310 | 0.923 | 0.462 | 0.474 | 0.857 | 0.610 | 0.476 | 0.769 | 0.588 |
| New | |||||||||
| agent based modeling | – | – | – | – | – | – | 1.00 | 1.00 | 1.00 |
| annotation | – | – | – | – | – | – | 0 | 0 | 0 |
| classification | 0.143 | 1.00 | 0.250 | – | – | – | 0.467 | 0.583 | 0.519 |
| clustering | 0 | 0 | 0 | – | – | – | 0.833 | 1.00 | 0.909 |
| comparative evaluation | – | – | – | – | – | – | 0 | 0 | 0 |
| document analysis | 1.00 | 0.200 | 0.333 | 1.00 | 0.600 | 0.750 | 0 | 0 | 0 |
| information extraction | – | – | – | – | – | – | 1.00 | 0.250 | 0.400 |
| IR related indexing/ranking/query methods | – | – | – | – | – | – | 0.67 | 0.25 | 0.363 |
| mixed method | 0.667 | 1.000 | 0.800 | – | – | – | – | – | – |
| digital ethnography | 1.00 | 1.00 | 1.00 | – | – | – | – | – | – |
| network analysis | 0 | 0 | 0 | 1.00 | 1.00 | 1.00 | 0.750 | 0.333 | 0.462 |
| statistical methods | 0 | 0 | 0 | 0.167 | 1.00 | 0.286 | 0.500 | 0.444 | 0.471 |
| topic modeling | – | – | – | – | – | – | 1.00 | 0.500 | 0.667 |
| user task based study | 0 | 0 | 0 | 0 | 0 | 0 | 1.00 | 0.500 | 0.667 |
| AVERAGE7 | 0.783 | 0.783 | 0.783 | 0.811 | 0.811 | 0.811 | 0.610 | 0.610 | 0.610 |