Swearing by acts of worship is haraam but there is no expiation for it

It is not permissible to swear an oath except by one of the beautiful names of Allah or one of His sublime attributes, because the Prophet (blessings and peace of Allah be upon him) said: “Whoever wants to swear (an oath), let him swear by Allah or else remain silent.” 

Narrated by al-Bukhaari, 2679 

The Prophet (blessings and peace of Allah be upon him) regarded swearing by anyone or anything other than Allah as minor shirk, because it involves venerating something or someone other than Allah, because an oath can only be sworn by something that is venerated. The Prophet (blessings and peace of Allah be upon him) said: “Whoever swears by something other than Allah has committed an act of shirk.”

Narrated by Abu Dawood (3251); classed as saheeh by al-Albaani in Sunan Abi Dawood. 

Swearing an oath by fasting, prayer or any other act of worship is swearing by something other than Allah, so it is haraam. 

It says in Tabyeen al-Haqaa’iq (3/109): 

Swearing by an act of worship is not an oath because it is swearing by something other than Allah, may He be exalted. End quote. 

Ibn al-Humaam said in Fath al-Qadeer (5/71): 

Swearing by acts of worship is swearing by something other then Allah or His attributes, so it is not an oath. End quote. 

The one who has done that has to repent to Allah, regret what he has done and resolve never to go back to it again. 

An oath sworn by something other than Allah is not a binding oath, so it does not have to be fulfilled and no expiation has to be offered for it. 

Ibn Hazm said in al-Muhalla (9/125): 

The one who swears by something other than Allah has not sworn and it is not an oath; it is invalid and nothing has to be done with regard to it except asking Allah for forgiveness and repenting only. End quote. 

Shaykh al-Islam Ibn Taymiyah (may Allah have mercy on) said: 

Swearing by created things such as the Ka‘bah, the kngels, shaykhs, kings, parents, parents’ graves and so on is an oath that is not binding and no expiation has to be offered for it, according to scholarly consensus. Rather it is forbidden according to scholarly consensus, and this prohibition means that it is haraam according to the more sound of the two opinions. End quote. 

Al-Fataawa al-Kubra, 3/222 

Based on this, their is nothing wrong with you doing what you swore not to do if it was permissible, and you do not have to offer any expiation, but you have to repent from this haraam oath. 

And Allah knows best.