Have you ever wondered why the world is full of so many people who are incredibly incompetent at the very thing they are paid to do? If so, a crotchety educationalist called Laurence J Peter may have the answer.

Working as a teacher in Canada the 1940s, Peter had been puzzled by the inept behaviour of his equals and superiors. He had applied to join a new school district, for instance, only to have all his forms and documents returned. There was nothing wrong with the application – rather, the Department of Education told him that they could not accept the package since it had not been registered at the Post Office for safe delivery, despite the fact that it had, of course, already arrived safely. How could someone stupid enough to create that rule have found a place in the Department of Education?

Peter soon saw such foolish behaviour all around him – in politics, journalism, the military and the law. “Occupational incompetence is everywhere,” he later wrote in a best-selling book on the subject.

That book, published in 1969, attempted to explain why that might be. In Peter’s view, most people were promoted based on their current performance, with no real consideration of their capacity to take on greater responsibility. The result is that we may be less good at our current job than the one before. As we climb one, two or three rungs up the ladder, our performance may be so bad that we no longer warrant a further promotion. By this point we’ve reached our limits and fail to improve any more, and so we end-up irritating our colleagues and clients with our inability to do the job.

Eventually, “every employee tends to rise to his [or her] level of incompetence”, he wrote – a law he termed the ‘Peter Principle’.

The work was largely satirical – an academic’s attempt to ‘stick it to the Man’. It’s only recently that organisational scientists and psychologists have tested whether it’s actually true.

High levels of incompetence

Perhaps spurned on by the 2008 financial crisis – and the flawed decision making behind it – much work on the Peter Principle has been conducted within the last decade. Of this handful of studies, the strongest evidence for the theory comes from a recent study of 131 companies (operating in IT, manufacturing and professional services) that all used the same performance management software. This allowed the researchers to mine the (anonymised) date of nearly 39,000 sales workers, 1,553 of whom were promoted to management roles over the six-year study period.

As you might expect, the team found that the best salesmen or women were the ones who tended to be promoted. To assess their aptitude for the new managerial position, the researchers then examined the effect of this move on their team members.

“The managers are in charge of training and allocating and directing their sales employees,” says Kelly Shue at the Yale School of Management. “So, to figure out if someone is a good manager, we basically looked at the extent to which they improve or change the performance of their subordinates.” If the previously high-performing candidates really were competent at the new job, you would hope there would be a rise in the average performance of the team as a whole.

Unfortunately, that’s not what Shue saw. Managers who used to be very high-performing sales workers tended not to bring a significant boost to their colleagues, while the lower performers were often much better at raising the average sales within their team.

Shue was not entirely surprised by the result. “We looked at sales partly because, anecdotally, there's a lot of complaints about the Peter Principle being a problem in that setting,” she says. Indeed, one inspiration was the US version of The Office, in which the protagonist Michael Scott – who used to be a great salesperson – became an absolutely terrible manager.

There are lots of potential reasons why this might occur, says Shue. The personal drive – even aggression – that is needed to boost your own individual sales doesn’t necessarily translate to the people skills necessary to motivate others, for instance.

“It could be the types of experience or the types of people that are drawn to have very high sales and also to work on their own that makes them less effective managers,” she says. Along these lines, Shue found that the sales workers who already had a record of working collaboratively tended to be much better managers.

Although she does not yet have the data, Shue suspects that the problems described by the Peter Principle may also be rife in science, technology and engineering. “The best scientists and engineers – and this includes all those start-up founders – may not be the right person to eventually lead the organisation or to lead team,” she says. “You can see how there's a difference in skills required to be a top scientist versus a top manager of scientists.”

As Lawrence J Peter had originally noted in his book, the rise of incompetent managers is also a common complaint in academia and education. “The best researcher or the person who's best at teaching may not be the best dean of a school,” says Shue.

Who’s the boss?

Given these findings, we might conclude that we should overlook people’s current performance and promote them solely on things like their people skills. Yet that strategy can also come at a price.

The possibility of a promotion is a positive incentive for many people, boosting their individual performance. Removing that motivation could lead to reduced productivity across the workforce. There’s also the sense of disenchantment that comes after an uninspiring colleague has been promoted ahead of you. Shue’s analyses of the sales teams’ data found that high-performing candidates are about 23% more likely to leave a firm if they have been overlooked in favour of a less competent colleague, for instance.

The reality is that we often find it personally reassuring to be managed by someone who has already proven their own competence at a job, as Amanda Goodall at the Cass Business School in London recently found.

Unlike Shue, Goodall did not consider objective measures of performance, but instead examined how employees feel about their bosses – whether they were frustrated with things such as communication difficulties, for example.

Analysing data from a European-wide survey of 28,000 workers, she found that just 13% of people are unhappy with their current supervisor (a surprisingly low number, given how much media emphasises the prevalence of horrible bosses).

Crucially, Goodall found that the most common complaint was that the bosses lacked specialist expertise. The finding tallied with her previous research, which had shown that a boss’s technical competence – essentially, whether they would be able to do your job if necessary – strongly predicted employee’s overall satisfaction.

If you’re managed by someone who doesn’t really understand the systems you are using or the day-to-day tasks to get your job done, they might impose new unnecessary procedures that only waste your time, she says. Or they might struggle to understand how best to support you during a particularly challenging period.

Whatever the specific reason for their discontent, Goodall’s research questions the value of “generalist” managers who move from company to company with no core expertise in a particular field.

“There's been this belief that if you do an MBA or other forms of management training, then that means you're [automatically] a good manager, but all of our evidence shows that that's absolutely not the case,” says Goodall. In healthcare, for instance, “a lot of people think we should let the doctors do the medicine and let the managers run the hospitals, but the research shows that actually that's wrong. You need doctors to lead other doctors, because they understand what they need in terms of being an employee.”

The hiring dilemma

All of this raises a serious dilemma for companies hiring for a vacancy. Focus too much on current job performance and they risk promoting someone who struggles with management strategy; focus too much on other qualities and their lack of technical expertise may demotivate and disenchant the rest of staff. Clearly a fine balance between the two is needed.

One solution might be to change the company hierarchy. Shue, for instance, suggests that more companies consider different kinds of promotions, rather than the typical career ladder in which you move from subordinate to manager. In technology firms, for instance, “you can become Distinguished Engineer or Senior Engineer and basically your performance is recognised in the job title, without a change in the job role.” Even if that doesn’t correspond to a significant pay rise, valuing experience in this way could prevent people from jumping ship to other firms when they fail to be given a management position.

She says that some companies have also started to offer two forms of appraisal – a performance review and a management potential review, which considers more specifically the skills that are necessary for leadership. “It has the benefit of recognising that [leadership] potential and performance are two different dimensions,” she says.

Goodall suggests that the best solution is to invest more in management training “tailored to experts” while moving away from the idea that leadership is a transferable skill across multiple disciplines.

In the meantime, knowledge of the Peter Principle might be useful for each of us personally.

If you frequently find yourself frustrated with your own boss (and their reluctance to give you a promotion) or with your subordinates (and their inability to carry out your instructions) it’s worth considering the possibility that you’ve already reached your “level of incompetence”, in Peter’s words, and you just don’t know it.

That’s not always easy to recognise – as Peter put it, “competence, like truth, beauty, and contact lenses, is in the eye of the beholder” – but a humble awareness may just encourage you to correct your flaws and build the skills that are currently holding you back, or to find a new position where your unique talents are of greater value.

• David Robson is the author of The Intelligence Trap: Why Smart People Make Dumb Mistakes, which explores cutting-edge research on the best ways to improve our decision making and learning. He is @d_a_robson on Twitter.

BBC