It makes sense, given the gains workers have seen in their retirement plans and in home prices in recent years, that they would consider tapping their 401(k) accounts to buy homes. And that is exactly what a growing number of workplace savers are doing, according to a new study from Fidelity Investments.

"Over the past year alone, more than 27,000 investors took loans specifically for the purchase of a home," said Fidelity, which looked at data from workplace retirement plans it runs.

The investment firm said workers who borrowed from their 401(k)s for home purchases tended to borrow more - $23,500 on average - and could be putting themselves at risk of reducing or stopping their retirement contributions.

Millennials who borrow an average of 37 percent of their accounts, or $17,100, might particularly find the loan "a stretch for people who are also taking on a mortgage and might be saddled with student debt," Fidelity said.

To be sure, cash-poor workers who borrow to the max to buy a house can get into trouble quickly if they don't have the reserves to handle emergencies. As any homeowner would attest, the boiler will break and the roof will leak when they least expect it.

And a worker who borrows from his 401(k) and then leaves his job usually has to pay back the loan within a few weeks or face a big tax hit as he is forced to treat the loan as a distribution.

On the other hand, some workers with sizeable retirement funds and little other cash may find it a reasonable way to get into a house. A well-timed loan from a 401(k) plan may help a home buyer qualify for a better mortgage and cut monthly housing costs.

It's a strategy that is "pretty common and not looked at as a negative by lenders," said Chris Jordan, a loan officer in the Silver Spring, Maryland, office of First Home Mortgage.

Here are some considerations.

WHEN TO DO IT

Ironically, the best time to borrow against your 401(k) to pay closing costs or cover a down payment is when you can well afford to, suggests David Hultstrom, a financial adviser in Woodstock, Georgia. Otherwise, you are stretching yourself to the point where you couldn't weather an emergency.

If you lose your job and couldn't repay the 401(k) loan, you would have to take that amount as a distribution. That would cost you income tax and a 10 percent penalty on the amount and it would also leave your retirement plan permanently lighter, as you wouldn't be able to replace that money when you got onto more solid ground.

WHAT WORKS BEST

At the margin, a 401(k) loan could reduce your monthly costs for years to come. Mortgage borrowers who put less than 20 percent down are required to buy mortgage insurance, and that's not cheap.

A family with excellent credit putting $20,000 down on a $400,000 home would be expected to pay an extra $184 a month for that insurance, reports HSH.com, a mortgage research firm.

"You've got to go through all those calculations to see what works best for you," said Keith Gumbinger, a spokesman for HSH.

Furthermore, a mortgage lender might not qualify you for a loan if you had to borrow the down payment from a parent or someone else, but would if you "were just borrowing from yourself," said Jordan.

MATH WINS

The math always wins. Some workers closer to retirement might find themselves retirement-plan heavy with their eyes on a retirement home. To buy it, they might have to sell investments and eat a sizeable capital gains tax. Or a younger worker who has made a lot on stock funds in their 401(k) in recent years might want to temporarily tap that money to establish themselves as a homeowner.

Christopher Van Slyke, an Austin, Texas, fee-only adviser, tells his clients that if they want to buy a house, they should consider contributing less to their retirement accounts while they accumulate a down payment in savings.

He is generally against using retirement money for anything other than retirement, but "life intervenes," he said.

"When you've got a child on the way and you need a home, I tell clients to go ahead and do it." Then he tells them to make repaying the 401(k) loan their top financial priority.