During tumor growth the extracellular matrix (ECM) undergoes dramatic remodeling. The normal ECM is degraded and substituted with a tumor-specific ECM, which is often of higher collagen density and increased stiffness. The structure and collagen density of the tumor-specific ECM has been associated with poor prognosis in several types of cancer. However, the reason for this association is still largely unknown. Collagen can promote cancer cell growth and migration, but recent studies have shown that collagens can also affect the function and phenotype of various types of tumor-infiltrating immune cells such as tumor-associated macrophages (TAMs) and T cells. This suggests that tumor-associated collagen could have important immune modulatory functions within the tumor microenvironment, affecting cancer progression as well as the efficacy of cancer immunotherapy. The effects of tumor-associated collagen on immune cells could help explain why a high collagen density in tumors is often correlated with a poor prognosis. Knowledge about immune modulatory functions of collagen could potentially identify targets for improving current cancer therapies or for development of new treatments. In this review, the current knowledge about the ability of collagen to influence T cell activity will be summarized. This includes direct interactions with T cells as well as induction of immune suppressive activity in other immune cells such as macrophages. Additionally, the potential effects of collagen on the efficacy of cancer immunotherapy will be discussed.