Fighters from the self-styled Islamic State West Africa Province overran a naval base and another housing troops from a regional force in and around Baga late last month.
The UN has said more than 30,000 people had since sought refuge in the garrison town of Monguno and the Borno state capital, Maiduguri, as the military prepared a fight-back.
A military officer in Maiduguri said: "Our soldiers entered Baga yesterday evening (Wednesday). They are in total control of the town and the terrorists had fled.
"They entered the town at about 7:00 pm (1800 GMT) after taking control of Kuros-Kauwa, which is 15 kilometres (nearly 10 miles) from the town," he told AFP.
The development was confirmed by a resident, who like the officer asked not to be identified for security reasons.
"I left Baga at about 5:30 pm and I met a huge military convoy heading to the town from Kuros-Kauwa," said the local man.
"They entered Baga unchallenged. Boko Haram fighters had not been staying the night in Baga since they learnt soldiers were preparing to launch a massive attack."
Some buildings, including the homes of local politicians, chiefs and community leaders, were burnt down in Baga and neighbouring Doron Baga.
A hospital, clinic and schools were razed, as was the naval base in Fish Dam, he added.
Army spokesman Sani Usman said later in a statement Nigerian special forces had given ISWAP fighters who attacked Baga "a bloody nose" and "cleared" the area "without any resistance".
Usman did not say the troops recaptured the town, only that they had "since linked up with other forces in Baga, where they cleared the naval base and the surrounding areas".
One soldier was killed in the process, he said. Some jihadists were "neutralised" after an attempted attack on troops in Monguno, he added, without elaborating.
Civilians threatened
Baga and Doron Baga have been attacked repeatedly since the insurgency began in 2009. Overall more than 27,000 people have been killed and 1.8 million others are still homeless.
Hundreds of people, if not more, were killed in an attack on the two towns that began on January 3, 2015 that was feared to have been the worst attack in the nearly 10-year conflict.
Security has improved since then as a result of a sustained counter-insurgency but in recent months there have been indications of a resurgence.
Dozens of attacks on military facilities and soldiers have been blamed on or claimed by ISWAP, which split from the faction led by Abubakar Shekau in mid-2016.
On Wednesday, jihadists attacked military positions near Bulabulin village in the Konduga area of Borno, and nearby Delwa but were pushed back, sources said.
Soldiers on Sunday raided the Daily Trust offices in Maiduguri, arresting two reporters and seizing equipment after it reported on pending operations to retake Baga.
On Thursday, the army said Boko Haram fighters had told residents to leave their homes in villages near Maiduguri.
Civilian militia said earlier this week Boko Haram told people in Jakana and Mainok to flee as they were going to launch a major attack.
There was no indication they had done so but on Wednesday troops stepped up stop and search operations on the main road that passes through both locations.
Army spokesman Onyema Nwachukwu described the latest threats as "psychological war" and urged residents to report suspicious activity.