Victor Munoz tells Usmagazine.com that he quit as Suleman's rep Friday night.
"It just got to be too much," he says. "It's pretty much a free-for-all over there right now. They are freaking out right now.
"Not to sound arrogant, but those people depended on me for everything," he continues. "You have no idea what I've had to do for these people.
"Nadya got real greedy. This woman is nuts," he adds. "This I can say: what ultimately destroyed the business arrangement was personal reasons."
Munoz -- who plans to meet with his lawyers Monday to go over the terms of his contract and confidentiality clause -- did not go into further detail.
Suleman's first publicist, Joann Killeen, stepped down after her firm received over 100 graphic e-mail and voicemail threats.
"They'd put me in the wood chipper and throw me in the bottom of the ocean and hope I die," Killeen has said. "We've gotten her through the worst part of it and now they are putting their venom and anger toward us."