Praise be to Allah, and peace and blessings be upon the Messenger of Allah and his family.
The fact that your mother died before the estate of her son who died when she was still alive was shared out does not cancel out her share of the estate, because the conditions of her being entitled to inherit from him were fulfilled, namely that the testator died when the heir was still alive.
For more information, please see the answer to question no. 196671 .
As that is the case, the estate of the deceased’s son should be shared out, and the mother’s share should be allocated as if she were still alive, which is one-sixth of the entire estate.
Then what she inherited from her son should be added to whatever wealth she left behind, if she left behind anything else. Then what she left behind after she died should be divided among the heirs mentioned: three sons and two daughters, with each male receiving the share of two females.
With regard to the issue of giving up your share of the estate to your brother’s sons, then changing your mind about that, there is nothing wrong with that and there is no sin on you as a result of that, because all that happened was that you intended to give it, and a gift does not become binding unless it is actually received, according to the view of the majority of scholars (may Allah have mercy on them).
And Allah knows best.