This follows claims that staff for the charity in Haiti and other countries paid vulnerable people for sex.
In a statement she said that she was “nothing short of horrified” by the allegations.
Oxfam said it was “grateful” for Ms Driver’s commitment and that it was “more committed now than ever to learn from our mistakes”.
The British charity is accused of concealing the findings of an inquiry into claims staff used prostitutes while delivering aid in Haiti in 2011.
The actress, best known for the movies Good Will Hunting, Grosse Point Blank and Hope Springs, said that although she could not continue her 20 years of involvement with Oxfam, she would work against “social and economic injustice”.
She added: “I certainly will not let the abhorrent mistakes of a troubling organization stop me or anyone else from working with good people in this space to support a population of human beings around the world that needs our help.”
The Charity Commission has launched a statutory inquiry into Oxfam – which denies a cover-up – but details of its scope have not yet been released.
Driver is the first of the charity’s celebrity supporters to address the scandal.
Since the allegations came to light, Oxfam’s deputy chief executive, Penny Lawrence, has resigned over the charity’s handling of complaints.
In a separate blow for the charity, Oxfam’s international chairman, Juan Alberto Fuentes, was arrested in Guatemala as part of a corruption investigation relating to his time as the country’s finance minister.