The upside to market selloff? Mortgage rates dive

As investors flee the stock market and rush to the relative safety of bonds, consumers are seeing at least one benefit: Lower mortgage rates. The average rate on the 30-year fixed conforming loan ($417,000 or less) dropped from 4.5 percent to 4.375 percent for most borrowers and as low as 4.25 percent for those with the highest credit scores, according to Mortgage News Daily.

"We have a broad-based flight to safety in multiple asset classes and on a global scale. Domestic bond markets have only been a supporting player in that drama, but the past six days have been enough to bring rates to their lowest levels in over a month," said Matthew Graham, chief operating officer of Mortgage News Daily.

Still, lower rates don't seem to be helping some of the biggest players in the mortgage business. On Friday morning, Wells Fargo (WFC) and JPMorgan Chase (JPM), which control nearly one-third of the mortgage market today, reported that their origination volumes plummeted in the first quarter of this year. Wells funded $36 billion in home loans, down 67 percent year over year, and JPMorgan's originations fell 68 percent in dollar volume.

Higher rates have plagued the mortgage market since last summer, when the average rate on the 30-year fixed went from 3.5 percent to 4.5 percent. After a refinance boom throughout much of 2012 and 2013, that market dried up.

Applications to refinance a mortgage were down 70 percent last week from a year ago, according to the Mortgage Bankers Association. And applications to purchase a home were down 14 percent. Total mortgage originations in February hit their lowest level in 14 years, according to Black Knight Financial Services.

"The problem is, even though rates are now falling again, so many people have already re-fied in the last 2½ years. If someone is sitting at even 4.5 percent, they're not going to refi even if rates go to 4 percent. If you have a $200,000 mortgage, and rates only move a half percentage point, it's not in your interest to refinance," said Paul Muolo of Inside Mortgage Finance.

Read More Real estate site targets underserved commercial sweet spot

The question now is, will rates continue to fall, or is this a momentary blip?

"They're still in the same fairly narrow 4.25-4.5 percent range that has characterized most of 2014," said Graham. "We'd need to see another day or two like (Thursday) to be able to say something different is happening."

Most had expected rates to surge higher by now, due to the so-called taper by the Federal Reserve , but so far they have not. As part of the taper, the Fed has been reducing the amount of agency mortgage-backed securities it buys.

"Safe-haven buying has outpaced the Fed's QE3 taper, which is why mortgage rates have stayed low. We saw a similar pattern after the end of the first two QE programs, too. Mortgage rates should remain low so long as inflation rates stay low," said Dan Green, publisher of TheMortgageReports.com.

Green added that mortgage rates remain subject to event risks, be they economic or geopolitical. "Rates could cross 5 percent easily between now and June, or fall into the 3s."

Read More Divided housing recovery is all for the 'haves'

While the big banks see their mortgage business fall off, some are shifting strategies.

"They purposely are shedding market share. They seem more than willing to cede business to nonbanks like Quicken," said Muolo, who points out that the banks can make more money on servicing rights, especially on today's better-quality loans. "They're not losing a whole lot of sleep over losing market share in the mortgage business."

Big banks are originating more jumbo loans (loans with balances higher than $417,000), which, given today's tight underwriting, are a very safe bet. Of the 20,000 jumbo loans originated and then securitized since 2010, just two are 60 days past due, according to Fitch Ratings.

"What a lot of banks are telling me now is 'I never want to service a defaulted loan again, which means I won't originate a loan that does default. I'm keeping my credit scores above 700,'" said Paul Miller, a bank analyst at FBR Capital Markets.

Read More Homebuyers face spring sticker shock

Wells Fargo recently announced it would offer loans to borrowers with FICO credit scores as low as 600, but only as FHA loans, those backed with government insurance. These loans are fixed rate and are underwritten extremely carefully. Miller said he expects they will be a very small share of the overall market.

While lower mortgage rates may not do much to juice the refinance market, they could help some potential homebuyers get off the fence and into a home. The housing recovery, however, continues to be plagued by historically low inventory. That, more than rates, is keeping sales low and shrinking affordability, by pushing prices ever higher.

—By CNBC's Diana Olick