Although reduced early physical function after total hip- and knee arthroplasty (THA/TKA) is well-described, the underlying reasons have not been clarified with detailed studies on pathophysiological mechanisms related to recovery, thereby prohibiting advances in rehabilitation. Thus, we aimed to describe early post-THA/TKA physical activity measured by actigraphy and potential underlying pathophysiological mechanisms related to recovery in a well-defined cohort of THA and TKA patients. Daytime-activity was measured from 2 days before until 13 (THA) or 20 (TKA) days after surgery. The primary outcome was individualized recovery in activity, with secondary analyses of activity-intensities and association to the perioperative factors: sex, age, BMI, hemoglobin (hgb), C-reactive protein and postoperative pain. Eighty-one THA/TKA-patients were examined. A large inter-individual variation in early physical activity was found. On a group level, activity was significantly reduced compared to preoperatively the first 2 (THA) or 3 (TKA) weeks after surgery (mean-difference - 64 counts × 103/day, p < 0.001 and - 78 counts × 103/day, p < 0.001, respectively). All activity-intensities were affected with the largest decline in high intense activity. A slight overall improvement in activity was seen during the postoperative phase [THA: 1%/day (SD 2.15); TKA: 0.7%/day (SD 1.04)], but approximately 30% of THA and 20% of TKA patients had reduced and declining activity. Hgb, CRP, BMI (THA) and postoperative pain (TKA) were only weakly associated with impaired physical activity. Physical activity was reduced the first weeks following THA/TKA, but with large inter-individual variations in recovery profiles. No single pathogenic factor was associated with a poor recovery. Early risk stratified interventions are needed in patients on a suboptimal course.