The baby and a 40-year-old man are the only confirmed deaths from the building, a complex of towers whose floors pancaked down onto to each other when the 6.4 magnitude tremor struck just before 4 a.m. (2000 GMT), at the start of a Lunar New Year holiday.
Rescuers mounted hydraulic ladders and a crane to scour the wreckage, plucking more than 120 survivors to safety, with dozens taken to hospital, a fire brigade official said.
Elsewhere in the city of 2 million people, several buildings tilted at alarming angles but a fire department official said rescue efforts were now focused entirely on the apartment block.
"I was watching TV and after a sudden burst of shaking, I heard a boom. I opened my metal door and saw the building opposite fall down," said a 71-year-old neighbor who gave his name as Chang.
A plumber, he said he fetched some tools and a ladder and prised some window bars open to rescue a woman crying for help.
"She asked me to go back and rescue her husband, child, but I was afraid of a gas explosion so I didn't go in. At the time there were more people calling for help, but my ladder wasn't long enough so there was no way to save them."
The quake was centered 27 miles (43 km) southeast of Tainan, at a depth of 23 km (14 miles), the U.S. Geological Survey said. Several aftershocks shook Tainan, Taiwan's Central Weather Bureau said.
RELATED COVERAGE
› Two dead in 17-story collapsed building in Taiwan: city mayor
One elderly woman, wrapped in blankets, was strapped to a board and slowly slid down a ramp to the ground as the cries of those still trapped rang out. Rescuers used dogs and acoustic equipment to pick up signs of life in the rubble.
"There are 60 households in that building," said Tainan fire department information officer Lee Po Min, estimating that there might be about 240 people living there.
One city hospital said 58 people had been brought in, most of them with light injuries. The fire department said a total of 115 people had been taken to hospital from around Tainan.
SEVERAL BUILDINGS DAMAGED
Taiwan President Ma Ying-jeou, speaking to reporters in the capital before leaving for the disaster scene, said authorities were not clear on the extent of the disaster.
"The disaster situation is not very clear yet. We will do our utmost to rescue and secure (survivors)," Ma said.
The quake initially cut power to 168,000 households in Tainan, many of whose residents lived through a massive 1999 tremor that killed about 2,400 people. Later, utility Taipower said power had been restored to all but about 900 households.
Taiwan lies in the seismically active "Pacific Ring of Fire". Television quoted Tainan residents as saying the quake felt worse than the 1999 tremor, centered in central Taiwan.
China's Taiwan Affairs Office, which in is charge of Beijing's relations with the self-ruled island, said China was willing to provide help if needed, Chinese state news agency Xinhua said. Beijing regards Taiwan as a wayward province.
Taiwan's Formosa TV said its reporters could hear the cries of people trapped inside the collapsed apartment tower as firefighters, police and troops swarmed the area. The defense ministry said 810 soldiers had been mobilized for rescue efforts.
Firefighters hosed down part of the building to prevent a fire while others used ladders and a crane to enter upper floors. The building appeared to have collapsed onto the first story where a child's clothes fluttered on a laundry line.
Some bullet train services were suspended to the south of Taiwan as inspections were carried out on the tracks for damage, Taiwan High Speed Rail Corp said in a statement.