Many research and higher education institutions are interested in their contribution to achieving the United Nation’s Sustainable Development Goals (SDG). Commercial services from Elsevier and Times Higher Education are addressing this by developing bibliometric queries for measuring SDG-related publications and SDG university rankings. However, such services should be evaluated carefully before use due to the challenging nature of interpreting the SDGs, delimiting relevance, and building queries. The aim of this bibliometric study was to build independent queries to find scholarly publications related to SDG 1, SDG 2, SDG 3, SDG 7, SDG 13, and SDG 14 using a consistent method based on SDG targets and indicators (the Bergen approach), and compare sets of publications retrieved by the Bergen and Elsevier approaches. Our results show that approach made a large difference, with little overlap in publications retrieved by the two approaches. We further demonstrate that different approaches can alter resulting country rankings. Choice of search terms, how they are combined, and query structure play a role, related to differing interpretations of the SDGs and viewpoints on relevance. Our results suggest that currently available SDG rankings and tools should be used with caution at their current stage of development.