In an interview with Variety while at the Toronto Film Festival, Robert Downey Jr. was asked about the possibility of an Iron Man 4.
"There isn't one in the pipe," Downey said. "No, there's no plan for a fourth 'Iron Man.'"
He is only signed on with Marvel and Disney to do two Avengers sequels including Age of Ultron. When asked about letting someone else take over as the armored Avenger.
"I like that the idea is that it would be up to me, like I'm casting director for Marvel," he said.
Downey was in Toronto to promote his new film The Judge along with his co-star Robert Duvall who advised the younger actor: "Don't give it to anyone else."
Here is video of some of the interview done about the movie the Judge.