Authorities said only 20 people were rescued when the ferry travelling between Niger state and Kebbi state in the north-west broke in two.
Yusuf Birma, head of the National Inland Waterways Authority (Niwa) in Kebbi state told the BBC that 30 bodies have so far been recovered.
"There's no hope of finding any survivors at this moment," he said.
The true death toll may never be known. Some reports said the boat had been carrying up to 200 people, mostly women and children.
Mr Birma said the boat did not load at a designated jetty so there was no manifest to know the exact number onboard.
"It was meant to be carrying between 131 and 150 but certainly not up to the 180 being quoted," he told the BBC.
He said some of the boat operators, not wanting to pay taxes, did not load at the jetties but preferred to pick passengers along the way were they are unregulated.
Another official supervising the rescue operation told AFP news agency that so far 45 bodies had been found.
The state governor tweeted that it was "in deep shock and sorrow, over the unfortunate boat accident which claimed the lives of over 100 passengers".
The waterways authority that regulates water transport in Nigeria said the boat had been overcrowded and went down an hour into its journey.
Authorities are blaming the accident on overloading as the boat was also reportedly carrying bags of sand from a gold mine and motorcycles.
This is the second accident in the region this month, after at least 28 people drowned when their boat capsized two weeks ago in Niger state.
President Muhammadu Buhari described the accident as "devastating", offering his condolences to the victims' families.
Qasimu Umar Wara, a resident of the nearby town of Wara, said more bodies are expected to wash up in the days ahead.
The boat was overloaded, he said: "My brother is among those missing. This is the worst boat accident that has happened in this water."
Boat accidents along the River Niger are common with overcrowding a major cause, as well as collisions with tree trunks under the water.
Sailing at night, not wearing life jackets and overloading boats are criminal offences in Nigeria but ferry operators often disregard the regulations.
Boat skippers and ferry owners are rarely prosecuted for deaths resulting from their actions such as overloading.