Duffy and his wife Rachel have nine children. The youngest, a girl, has Down syndrome. In September 2019, at age 47, Duffy resigned from Congress around the time of her birth, citing the need to spend more time with his family and to help care for his youngest daughter.

Hurricane Helene destroyed parts of Interstate 40 at the Tennessee/North Carolina line. A special use permit may change the work.