Site-specific weed management (SSWM) may reduce herbicide use by identifying weed patches and weed-free areas. However, one major constraint is robust weed detection algorithms that are able to predict weed infestations outside of the training data. This study investigates the predictive power of regression models trained on drone imagery that are used within fields to predict infestations of annual grass weeds in the late growth stages of cereals. The main objective was to identify the optimum sampling strategy for training regression models based on aerial RGB images. The study showed that training based on sampling from the whole range of weed infestations or the extreme values in the field provided better prediction accuracy than random sampling. Prediction models based on vegetation indices (VIs) offered a useful alternative to a more complex random forest machine-learning algorithm. For binary decision-making, linear regression utilizing weed density information resulted in higher accuracy than a logistic regression approach that only relied on information regarding the presence/absence of weeds. Across six fields, the average balanced accuracy based on linear regression was in the range of 75–83%, with the highest accuracy found when the sampling was from the extreme values in the field, and with the lowest accuracy found for random sampling. For future work on training weed prediction models, choosing training sets covering the entire sample space is recommended in favor of random sampling.