He died and left behind a wife, a mother, and his brother’s children; how should the estate be divided?

If someone dies and leaves behind a wife, a mother and his brother’s
children, the estate is to be divided as follows. 

The wife gets one quarter, because there are no descendants who inherit.
Allah, may He be exalted, says (interpretation of the meaning):

“In that
which you leave, their (your wives) share is a fourth if you leave no child;
but if you leave a child, they get an eighth of that which you leave after
payment of legacies that you may have bequeathed or debts”

[an-Nisa’ 4:12]. 

The mother gets one third, because there are no descendants who inherit or
siblings. Allah, may He be exalted, says (interpretation of the meaning):

“For parents, a sixth share of inheritance to each if the
deceased left children; if no children, and the parents are the (only)
heirs, the mother has a third; but if the deceased left brothers or
(sisters), the mother has a sixth”

[an-Nisa’ 4:11]. 

See: al-Mughni by Ibn Qudaamah (6/169). 

Shaykh Ibn ‘Uthaymeen (may Allah have mercy on him) said: She inherits one
sixth if the deceased had descendants who inherit or he had a number of
brothers or sisters or both, because Allah says (interpretation of the
meaning):

“For parents, a sixth share of inheritance to each if the
deceased left children; if no children, and the parents are the (only)
heirs, the mother has a third; but if the deceased left brothers or
(sisters), the mother has a sixth”

[an-Nisa’ 4:11]. 

It
makes no difference whether the siblings are males or females, or both, full
siblings or siblings through the father only or through the mother only, and
it makes no difference whether they are heirs or are barred from inheriting
by the father, as is the apparent meaning of the verse, because Allah has
allocated to the mother one third with the father. Then He says
(interpretation of the meaning): “but if the deceased left brothers or
(sisters), the mother has a sixth” [an-Nisa’ 4:11]. The particle fa
(“but” in the phrase fa in kaana lahu ikhwah…, translated here as
“but if the deceased left brothers (or sisters)…”) indicates that the second
phrase is connected to the first and is to be understood in the light
thereof, and siblings do not inherit if the father is present, yet despite
that Allah allocated one sixth to the mother in this case. This is the view
of the majority of scholars.

End quote from Tasheel al-Faraa’id (1/19) 

Whatever is left of the estate goes to the brother’s children, to the males
only, because the Prophet (blessings and peace of Allah be upon him) said:
“Give the shares of inheritance to those who are entitled to them, and
whatever is left goes to the closest male relative.”

Narrated by al-Bukhaari (6746) and Muslim (1615). 

And Allah knows best.