Objective: To investigate (i) the proportions of time with irritability and (ii) the association between irritability and affective symptoms and functioning, stress, and quality of life in patients with bipolar disorder (BD) and unipolar depressive disorder (UD). Methods: A total of 316 patients with BD and 58 patients with UD provided self-reported once-a-day data on irritability and other affective symptoms using smartphones for a total of 64,129 days with observations. Questionnaires on perceived stress and quality of life and clinical evaluations of functioning were collected multiple times during the study. Results: During a depressive state, patients with UD spent a significantly higher proportion of time with presence of irritability (83.10%) as compared with patients with BD (70.27%) (p = 0.045). Irritability was associated with lower mood, activity level and sleep duration and with increased stress and anxiety level, in both patient groups (p-values<0.008). Increased irritability was associated with impaired functioning and increased perceived stress (p-values<0.024). In addition, in patients with UD, increased irritability was associated with decreased quality of life (p = 0.002). The results were not altered when adjusting for psychopharmacological treatments. Conclusions: Irritability is an important part of the symptomatology in affective disorders. Clinicians could have focus on symptoms of irritability in both patients with BD and UD during their course of illness. Future studies investigating treatment effects on irritability would be interesting.