Buffett hits back at Clayton Homes lending gripes

Warren Buffett unapologetically defended Berkshire Hathaway's lending unit on Saturday from accusations of predatory lending.

Warren Buffett unapologetically defended Berkshire Hathaway (BRK.A)'s lending unit on Saturday from accusations of predatory lending, telling CNBC in an interview that he's never even received as much as "one letter of complaint" about its practices.

Meanwhile, one of the world's wealthiest men also proved he can win even when he's doesn't try. Buffett told CNBC he wasn't following this year's Kentucky Derby, yet managed to correctly predict the contest's winner hours before the race was even decided.

In April, two published reports implied Clayton Homes might be fleecing struggling home owners, and suggested that buyers were being saddled with high borrowing costs and houses with falling values. Buffett, however, had a different take, and used the occasion of Berkshire's annual meeting to break his silence on the controversy.

"We are not forcing loans on anybody. If they had a loan with us they didn't like, they can pass off and borrow from somebody else," Buffett told CNBC on the sidelines of the company's meeting. "We have 300,000 loans on the books and in the last 3 years I've not received one letter of complaint from anybody."

Read More Working for Buffett like a 'marriage': BNSF chief

At the meeting, Buffett said that many of Clayton's buyers posed higher risks, and charging them higher interest rates was justified.

"It's true that manufactured housing hits the lower end of the market," he said, adding that Clayton actually helps borrowers by providing reasonable loan terms.

He told CNBC that in certain cases, Clayton Homes has worked to modify loan terms for strapped borrowers.

Separately, the billionaire investor also offered more information on why he continues to purchase shares of IBM (IBM).

"This is really off the top of my head but I'm sure we bought maybe 3 million" recently, Buffett said. "I think we have 78 or 79 million now. I bought it because I thought we were getting our money's worth," he added, declining to comment further.

Buffett also steered clear of picking the winner of the Floyd Mayweather and Manny Pacquiao boxing battle royale. The mogul is friendly with Mayweather, but declined to say outright whether the boxer would prevail in what is being billed as one of the biggest boxing matches in years.

Although the CEO said he didn't have a pick for this year's Kentucky Derby, ironically enough he guessed the eventual winner from a random list of top contenders read to him by CNBC.

"American Pharoah," Buffett replied, when CNBC recited a list of horses racing at the Derby, without missing a beat.

--Reuters contributed to this story.

--Reporting by CNBC's Becky Quick; writing by Javier E. David