The guys share stories of their families’ incredibly impoverished upbringings