This study explores how analytical choices of researchers affect the reliability of scientific
findings. Current lack-of-reliability discussions focus on systematic biases. We broaden the lens
to include idiosyncratic decisions in data analysis that lead researchers to diverging results and
conclusions. We coordinated and observed decisions among 73 research-teams as they
independently tested the same hypothesis using the same data. Results show that in this typical
secondary data research situation, the universe of pathways from data to results is so vast that each
analysis was unique in some way. Teams reported divergent findings with contradictory
substantive implications that could not be explained by differences in researchers’ expertise, prior
beliefs, and expectations. This calls for greater humility and clarity in presentation of scientific
findings. Idiosyncratic variation may also be a cause for why many hypotheses remain highly
contested, particularly in large-scale social and behavioral research