Background: Medicinal plants are used to treat various disorders, including diabetes, globally in a range of formulations. While attention has mainly been on the aerial plant parts, there are only a few review studies to date that are focused on the natural constituents present in the plant roots with health benefits. Thus, the present study was performed to review in vivo studies investigating the antidiabetic potential of the natural compounds in plant roots. Methods: We sorted relevant data in 2001–2019 from scientific databases and search engines, including Web of Knowledge, PubMed, ScienceDirect, Medline, Reaxys, and Google Scholar. The class of phytochemicals, plant families, major compounds, active constituents, effective dosages, type of extracts, time of experiments, and type of diabetic induction were described. Results: In our literature review, we found 104 plants with determined antidiabetic activity in their root extracts. The biosynthesis pathways and mechanism of actions of the most frequent class of compounds were also proposed. The results of this review indicated that flavonoids, phenolic compounds, alkaloids, and phytosteroids are the most abundant natural compounds in plant roots with antidiabetic activity. Phytochemicals in plant roots possess different mechanisms of action to control diabetes, including inhibition of α-amylase and α-glucosidase enzymes, oxidative stress reduction, secretion of insulin, improvement of diabetic retinopathy/nephropathy, slow the starch digestion, and contribution against hyperglycemia. Conclusion: This review concludes that plant roots are a promising source of bioactive compounds which can be explored to develop against diabetes and diabetes-related complications. Graphical abstract: [Figure not available: see fulltext.]