They are known simply as “bandits” – heavily armed criminal gangs that have terrorised Nigeria’s rural northwest, killing, kidnapping, forcing people from their homes, and taunting the authorities with their brazenness.

The violence typically involves scores of gunmen on motorbikes sweeping into villages, shooting all the young men they can find on the assumption they belong to local vigilantes, and then carting away livestock and anything else of value.

The raids are increasingly daring. In the past few months, bandits have downed an air force jet; attacked the military’s officer training school; struck a prestige commuter rail service running between the capital, Abuja, and the city of Kaduna; and kidnapped students for ransom from schools and colleges so many times that education is now in peril.

And although bandits aren’t natural ideological bedfellows for jihadist movements, there’s also a persistent fear that al-Qaeda-linked Ansaru and Islamic State in West Africa Province (ISWAP) – the country’s largest extremist group after splitting from Boko Haram – are recruiting among them to expand their influence across the entire north.

Zamfara, one of Nigeria’s poorest states, is at the centre of the banditry. It tops the country’s league table of violent deaths, with 495 reported killings between July and October. That’s far more fatalities than northeastern Borno, where ISWAP and Boko Haram operate.

The insecurity has triggered a food emergency across the northwest, with over 450,000 people fleeing farms and rural markets. The US government’s Famine Early Warning Systems Network, known as FEWS NET, has predicted “catastrophe” levels of hunger in parts of the region, which means famine for some households.

The following outlines what’s gone wrong in the ongoing attempts to resolve the expanding crisis, and offers some recommendations on a way forward.

Roughly 10 military operations have been launched against banditry in the northwest so far, but they have failed to make a dent in the insecurity. The mobile gangs take advantage of a forested, sparsely populated region, and when attacked, simply move elsewhere.

The security stick doesn’t address the root causes of the violence. Land disputes are a key issue: Expanding farms have encroached on the routes and cattle reserves used by pastoralists, resulting in friction – on both sides – when fields are trampled. Local authorities are seen to have failed to fairly adjudicate these conflicts.

Nigeria is also under-policed, so farmers turned to vigilantes – known as Yan Sakai – for protection. But the Yan Sakai have been indiscriminate in their retaliatory violence. Although the bandits do recruit from among pastoralists, the vigilantes’ response has been to target all herder communities, regardless of culpability.

The pastoralists have responded with their own, forest-based – and better-armed – self-defence groups, which over time have become almost indistinguishable from the original bandits.

To try and stem the spreading chaos, some state governments have turned to peace deals to essentially buy off the gunmen. The model is the oil-rich Niger Delta from a decade ago, where militants protesting the government’s exploitation of the area accepted amnesty deals and development programmes to end attacks on oil facilities.

But that approach has had mixed results in the lawless northwest. Zamfara introduced an amnesty and a cash for guns scheme in 2016. Initially, it seemed to work, but it fell apart two years later with the death of warlord Buharin Daji, the lynchpin of the deal.

The violence has forced more than 450,000 people from their homes in five states – Kaduna, Kano, Katsina, Sokoto, and Zamfara.

Zamfara tried again in 2020. This time, the new governor, Bello Matawalle, offered cows in return for guns (to avoid monetising the violence). He also ordered the disarmament of the Yan Sakai, and pledged to construct Rural Grazing Areas – settlements with dams and veterinary services set aside for pastoralists. But the attacks continued, and Matawalle made a U-turn in September, cancelling further dialogue.

A new military offensive was launched in Zamfara in September, combined with a telecommunications blackout, and bans on the sale of fuel in jerry cans to try to blunt the bandits’ mobility. But the measures have simply pushed the bandits into neighbouring states.

It has been the same story in Katsina, the home state of President Muhammadu Buhari, with peace deals never properly sticking. Yet pastoralists in both Zamfara and Katsina also point out that governments have failed to deliver on the development promises they made.

Some states, led by Kaduna and Niger, have taken a far harder line by refusing to negotiate over ransom demands or amnesties. But they have not fared any better, with a string of high-profile student abductions that has forced the closure of schools.

Private mediation has also been tried. The most prominent was by Sheikh Ahmad Gumi, the leader of the influential Izala Islamic sect. He held a series of forest meetings with bandits, arguing that their grievances should be taken seriously and would need the kind of political and financial investment that ended the Delta crisis.

There’s a tendency to see the bandits as an undifferentiated group of ruthless, criminally-minded men, ready to welch on any deal as soon as it’s in their interests to do so. In much of Nigeria, they are referred to as “terrorists”, which clouds conciliation attempts.

Both Matawalle, the Zamfara governor, and Gumi, the cleric, have argued that among the bandits are the self-defence groups that took up arms to defend pastoralist communities against the Yan Sakai – and to protest government neglect – and these are the men the amnesties and promises of development spending aim to reach.

• The New Humanitarian report