The REAL reason why the ex-Japanese PM was eliminated