Libya's Colonel Muammar Gaddafi is championing the idea, but many African leaders in Accra, Ghana, are wary.
Some of the 50 leaders there fear the issue will push the crises in Zimbabwe, Somalia and Darfur off the agenda.
Zimbabwean President Robert Mugabe said unity was vital to make the continent truly independent of the West, as he spoke to a crowd of cheering Ghanaians.
"Until and unless we put our act together, and look at our resources, organise and start pooling our resources together, and we have lots of them, we will never, ever prosper from any aid from any source outside Africa," President Mugabe said on Sunday in a speech at the tomb of Ghana's first president.
The idea of a single pan-African government was first promoted by Kwame Nkrumah, who led Ghana to independence in 1957, and the Libyan leader has long been an enthusiastic proponent of the idea.
Mr Gaddafi has called for the immediate establishment of a single government, foreign policy and army.
Ghana's President John Kufuor said in his opening speech to the conference that the question of unifying Africa was not in doubt, but the key issue was how to attain it.