When it comes to the lifelong pursuit of happiness, dieting rarely comes up.

Now a new study using data from over 28,000 adults finds that calorie-restricted diets may come with unexpected mental health costs — especially for men and people with a higher body weight. Researchers from Toronto analyzed six cycles of the National Health and Nutrition Examination Survey from 2007 to 2018, looking at the link between different kinds of diets and depression scores.

Participants were grouped by how they ate: Most, 87%, weren’t dieting, but about 8% were cutting calories, while a smaller percentage followed nutrient-restricted plans or structured eating patterns. Ultimately, those on calorie-restricted diets had depression scores that were, on average, 0.29 points higher than those who weren’t dieting at all. For overweight individuals, those numbers rose even more significantly — by 0.46 points for calorie restriction and 0.61 for nutrient-restricted plans.

Men were especially affected. Restrictive diets showed stronger links to emotional and physical symptoms of depression in men than in women, possibly because men tend to have higher nutritional needs. When people diet outside a clinical setting, meals often lack key nutrients, like glucose or omega-3 fats, that the brain relies on for mood and cognitive function.

In theory, weight loss should make us feel better. But in practice, the stress of chronic restriction, missed nutrients and unmet expectations can come together to create the perfect storm for a decline to our well-being.

After all, how we diet matters — and mental health shouldn’t be starved in the process.