Commanders said they had cleared outlying areas and were within two kilometres of the centre.
They said they hoped to have full control in the next few hours.
Bani Walid and three other loyalist-held towns were given until September 10 to surrender but the anti-Gaddafi forces said they had come under attack.
Fierce fighting has also been reported close to the Gaddafi-held city of Sirte.
The anti-Gaddafi forces had given a briefing in which they said they had had no choice but to respond to the loyalist attacks.
They said they had made quick progress so far but that they were not engaging in a full military assault as they still wanted to minimise casualties.
Anti-Gaddafi forces have been preparing for an attack on Sirte, Col Gaddafi's birthplace.
They said that four people had so far been killed, including three pro-Gaddafi fighters, and a number of people had been injured.
Add new comment