Maybe the thought that you do not desire material goods is harder to be carried on than the unfulfilled desire of having them.
Without the desire for material goods, our usual conduct in the world is seriously affected. The contact with the world is always stronger when it involves our business for sharing a part of it. So, it becomes weaker when we are not longing for material possession.
A weaker contact requires to be compensated by a stronger way of living by ourselves. We are forced to cast out a great part of what is different from us, but interferes with our lives. And there are many things and many persons, too. The power we find in such assumed loneliness is as natural as the naturalness of the way of living for possessing material things. For showing this, it is suffice only to mention that we die alone.
Sometimes, the poverty makes someone to lose his or her desires for material goods. In these cases, we will meet poor and strong persons, in spite of our custom to see them being poor and weak. Nonetheless, their experience of loneliness makes them more skillful in discovering how others simulate philanthropic feelings.