The 83-year-old was on his way to visit a nativity scene in Vatican City's central plaza when the woman grabbed his hand and yanked him towards her.
He had reached over a barrier to touch a child and had already turned to walk away when he was drawed back.
The visibly-amazed pontiff stumbled slightly as he moved back towards the barrier and appeared to be in some ache.
He then scolded the woman verbally before slapping her hand twice when she still refused to let go.
Security guards then moved in to pry the woman off him.
The woman had made the sign of the cross as the Pope walked towards her, and was trying to talk to him as she held his hand, although it is not clear what she was saying.
The Pope has now apologised over the incident.
"We lose patience many times," Francis confessed. "It happens to me too. I apologise for the bad example given yesterday."
He was later filmed continuing to shake hands with worshippers and kissing the cheek of a baby passed to his guards by its mother.
'BUILD BRIDGES, NOT WALLS'
At the New Year's Eve evening service at St Peter's Basilica, the Pope told the congregation to practice solidarity and to "build bridges, not walls".
On Christmas Eve, he said the light of Christmas should be allowed to counter the "darkness of human hearts", mentioning in particular unrest in Ukraine, Venezuela, Syria, and a number of countries in Africa.
He also called for compassion for migrants forced by conflicts and economic hardship to "emigrate in the hope of a secure life".
He said that after perilous journeys, they are often "find themselves before walls of indifference" in places where "they might have hoped for a dignified life".
The New Year's incident is not the first time the Pope has rebuffed affection by the faithful.
