He swore to divorce her if she visited her family, and now he has recanted. Does he have to offer kafaarat yameen?

Firstly: 

It is prescribed for the Muslim not to use divorce in arguments between him and his wife. That is because of the bad consequences that result from divorce. Many men take the matter of divorce lightly and every time they have an argument with their wives they swear to divorce them, and every time they have a dispute with their friends they swear to divorce their wives, and so on. This is a kind of playing with the word of Allaah, so how about the one who takes divorce as his habit, so that every time he wants to stop his wife doing something or make her do something, he swears to divorce her! 

Al-Nasaa’i (3401) narrated that Mahmoud ibn Labeed said: The Messenger of Allaah (peace and blessings of Allaah be upon him) was told about a man who had divorced his wife three times in one sitting. He stood up angrily and said, “Is he playing with the Book of Allaah whilst I am still among you?” Then a man stood up and said, “O Messenger of Allaah, shall I not kill him?” Al-Haafiz said: the men of its isnaad are trustworthy. This was classed as saheeh by al-Albaani in Ghaayat al-Maraam, 261. 

Shaykh Ibn ‘Uthaymeen (may Allaah have mercy on him) said: 

Those foolish men who utter words of divorce for every matter, minor or major, are going against the teaching of the Prophet (peace and blessings of Allaah be upon him) who said: “Whoever swears (an oath), let him swear by Allaah or else remain silent.” Narrated by al-Bukhaari, 2679. So if a believer wants to swear an oath, let him swear by Allaah. 

Moreover we should not make a lot of oaths, because Allaah says (interpretation of the meaning): 

“And protect your oaths (i.e. do not swear much)”

[al-Maa’idah 5:89]

The commentaries on this verse may be summed up as saying that what it means is: do not swear a great deal by Allaah. 

But swearing to divorce one’s wife, such as saying, “My wife is divorced if she does such and such,” or “My wife is divorced if she does not do such and such,” or “if I do such and such then my wife is divorced,” or “If you do not do such and such then my wife is divorced,” and so on, is contrary to the teachings of the Prophet (peace and blessings of Allaah be upon him). 

Fataawa al-Mar’ah al-Muslimah, 2/753. 

Secondly: 

As to whether or not divorce takes place as a result of that, the reference point here is the husband’s intention. If he intended to divorce her, then the divorce takes place if his wife does the thing that he swore she should not do. If he did not intend to divorce her, rather he only intended to stop her doing something, then the ruling in this case is the ruling on vows. 

Shaykh Ibn ‘Uthaymeen said: 

The most correct view is that if divorce is used as a vow, in the sense that the intention behind it is to urge someone to do something, or to stop them from doing something, or to prove that someone is telling the truth or lying, or to confirm something, then it comes under the rulings on vows, because Allaah says (interpretation of the meaning): 

“O Prophet! Why do you forbid (for yourself) that which Allaah has allowed to you, seeking to please your wives? And Allaah is Oft‑Forgiving, Most Merciful.

Allaah has already ordained for you (O men) the absolution from your oaths”
[al-Tahreem 66:1-2]

So Allaah has made forbidding something to oneself a vow or oath. 

And the Prophet (peace and blessings of Allaah be upon him) said: “Actions are but by intentions, and every man will have but that which he intended.” Al-Bukhaari, 1.  

This man did not intend to divorce, rather he intended to swear an oath, or something of that nature. So if he breaks his vow, then it is sufficient for him to offer kafaarat yameen. This is the more correct view. 

Fataawa al-Mar’ah al-Muslimah, 2/754 

The Standing Committee was asked about a man who said to his wife, “You will be divorced if you do not come with me,” and she did not go with him. Does this mean that they are divorced? 

They replied: 

If you did not intend that divorce should take place, rather you meant to urge her to go with you, then divorce has not taken place. But you have to offer kafaarat yameen, according to the more sound of the two scholarly opinions. If you did intend that divorce should take place and she did not do as you said, then she has been divorced once. 

Fataawa al-Lajnah al-Daa’imah, 20/86. 

And Allaah knows best.