Facts, Analysis and Comment.

Regulating Larger Organisations


There have in recent years been a number of catastrophic failures involving large, heavily-regulated organisations, including the 2007 worldwide collapse of financial institutions, subsequent scandals involving major UK banks, the 2010 Gulf of Mexico Oil Spill, the collapse of Enron, Bernie Madoff's Ponzi scheme, and the mis-management of the Mid-Staffordshire Hospitals. Can any general lessons be learned?

It would be a great mistake to seek to avoid all regulatory failure. Although it is clearly necessary to regulate the behaviour of organisations that are in a position to do serious damage to wider society, we must not over-regulate such organisations and so restrict their growth and innovative behaviour which is in turn so important for employment and the wider economy. It is extraordinarily hard to get this balance right. Every significant regulatory failure is followed by calls for stronger regulation, inevitably followed some time later by equally strident calls for regulators to adopt a lighter touch, and be more pragmatic. But the catastrophes listed in the first paragraph of this note were surely so dreadful that they cannot be characterised as the result of risks which were worth taking. (See Note 1 below)

This is accordingly the first in a series of notes which seek to explore these questions in some depth, and consider what might be done to avoid future catastrophes within heavily regulated sectors. In particular, should more attention be paid to the psychology of regulation:- to the behaviour of large organisations and their interaction with those who regulate them? This first note accordingly examines the behaviour of large organisations. Further notes look at ...

    •    the psychology of regulatory institutions, before turning to ...
    •    examples of the problems that can arise because of that psychology, before finally considering ...
    •    lessons that might be learned from the above analysis and examples.

This note argues that it is a mistake to approach the regulation of large organisations as though they are merely supersized versions of individual-led smaller firms. Most senior managers do not know (and some do not want to know) what is going on outside their head offices, nor what risks (financial, environmental and other) being taken by their staff. Internal communications difficulties, compounded by the incentive to report good results, and senior executives' aversion to taking responsibility for errors, mean that senior managers are often the last to learn what is going wrong. Regulators therefore need to take account of organisations' internal communications problems; the existence of herd behaviour and groupthink; and the culture and tensions that are typical of most large organisations.

Principal and Agents

It is particularly important to be aware of the principal-agent problem. This arises when a principal hires an agent to pursue its, the principal's, interests but the agent develops priorities of his/her own. The principal-agent problem is found in most large organisations, including in the way that shareholders and senior executives find it very difficult to ensure that middle managers, foremen etc. work to the corporate agenda as seen from head office. There is a strong tendency for humans to align their goals and behaviour to that of the team or work group around them. As a result, it often seems that corporations are in practice run in the interests of the firm's managers who have grown to regard their firm's economic interests as entirely coincident with their own.

It is for instance almost universally common for middle managers to focus on what they see as the long term good of their factory, office or other small part of the organisation. When asked to find efficiency savings they will fight hard to retain their budget and their staff numbers. They will often enter into an implicit bargain with their teams, allowing the use of material for personal ends, providing generous expense accounts, etc. so as to generate a better (non-confrontational) climate within the team - often characterised as 'high morale'. Such teams resist change - especially if it might lead to greater efficiency (working harder and/or job losses)

There are two consequences for regulators. First, it can be very hard to obtain reliable information from the intermediate levels in any hierarchy. Second, regulators need to be aware that agents and their teams often resist the introduction of (what they see as) tedious protocols aimed at improving quality or safety. Rail and marine accident reports include many examples of such behaviour. It is then quite common for such teams to begin falsifying data, including quality and safety data. New recruits are taught the difference between current working practices and formal instructions, which experienced colleagues characterise as misleading, out-of-date or impractical. It can therefore be very hard for a regulator to find out what is really happening inside a regulated entity, not least because the entity's own senior executives probably do not themselves know what is happening and, if they do, will be very reluctant to admit their partial loss of control.

It is important be aware that teams' failure to comply with safety, financial and other protocols does not necessarily mean that such teams are responding to pressure from above, such as to meet financial or other targets, complete work quickly, maintain production etc. Leaders of rule-bending teams may themselves be keen to work quickly, or to impress seniors with their achievements, or to get home on time, or to avoid a small amount of work running over into the next day. It is truly very difficult for regulators and company directors to know that this is happening unless there are robust and unpredictable inspection arrangements backed up by a strong compliance culture.

The principal-agent problem also means that regulators need to take into account how the middle and lower reaches of an organisation might react to regulatory pressure. The need for change and/or improvement may be accepted by senior executives, but this does not mean that the necessary changes will be accepted or implemented at the working level. The regulatory intervention needs to be designed with this problem in mind.

Here are some examples of the principal-agent problem. (Others may be found here.)

See Jean Tirole's Hierarchies and Bureaucracies: On the Role of Collusion in Organizations for a lengthier discussion of the Principal-Agent problem.

Absence of Honest Feedback

Even if middle managers (the agents) are trying hard to achieve the aims set for them by their seniors (their principals), it is absolutely certain that their communications to their seniors will be less than totally honest or frank. Tim Harford puts in nicely in his book Adapt:

'There is a limit to how much honest feedback most leaders really want to hear; and, because we know this, most of us sugar-coat our opinions whenever we speak to a powerful person. In a deep hierarchy, that process is repeated many times, until the truth is utterly concealed inside a thick layer of sweet-talk. There is some evidence that the more ambitious a person is, the more he will choose to be a yes-man - and with good reason because yes-men tend to be rewarded.

Even when leaders and managers genuinely want honest feedback, they may not receive it. At every stage in a plan, junior managers or petty bureaucrats must tell their superiors what resources they need and what they propose to do with them. There are a number of plausible lies they might choose to tell, including over-promising in the hope of winning influence as go-getters, or stressing the impossibility of a task and the vast resources needed to deliver success, in the hope of providing a pleasant surprise. Actually telling the unvarnished truth is unlikely to be the best strategy in a bureaucratic hierarchy. Even if someone does tell the truth, how is the senior decision-maker the honest opinion from some cynical protestation?

Herd Behaviour, Group-Think and Cognitive Dissonance

"Men, it has been well said, think in herds; it will be seen that they go mad in herds, while they only recover their senses slowly, and one by one." (Charles Mackay)

There is a separate problem as follows. It almost goes without saying that the majority in government, in regulation, and in the financial services industry were supported in their rosy view of the 1990s financial world by the fact that all 'the great and the good' shared their optimistic analysis of corporate behaviour and of the virtues of 'light touch regulation'. This was despite the fact that there were nevertheless a good number of perceptive commentators who did their best to warn of the forthcoming catastrophe. This is a good example of two very common psychological behaviours, often called 'herd behaviour' or 'group-think'.

Herd behaviour is not necessarily irrational. Herds usually run together for a good reason, and there is nothing irrational about humans seeking to watch and learn from what others are doing. Rational investors in an efficient market will therefore produce frenzies and crashes from time to time - but regulators are surely responsible for ensuring that herds to do not crash over cliffs and take lots of innocent (customer/taxpayer) victims with them.

Group-think is more pervasive but also highly undesirable, for it is essentially the unquestioning acceptance of obviously wrong answers simply because it is socially painful to disagree. Large organisations require a good deal of conformity, of course. You can't have every single person asking questions. Decisions have to be made and implemented. And if everyone you know, every newspaper you read, every person you once admired, are all saying the same thing, it takes a real effort of will, and real courage, to argue back. But someone has to do it - and yet it is uncomfortable to be a contrarian. Indeed, dissent can lead to the dissenter becoming the subject of personal and sometimes humiliating attack. Camilla Cavendish (who had run Prime Minister David Cameron's Policy Unit) told the Today Programme in December 2016 that some officials "became very very angry and took it viscerally personally" when it was suggested that there might have been a better way of negotiating with the rest of the EU in advance of the 2016 Brexit referendum. Questioning their strategy was perceived by them as questioning their commitment.

For further information, you might like to read Mannie Sher's paper on The Psychology of Regulation, the first part of which deals with this subject.

The British reluctance to offer overt challenge or criticism doesn't help. Industrialist Sir Denys Henderson was famed for his unforgiving tongue but appears to have had little impact as a non-exec of the Board of Barclays Bank. One scholarly director told him "You are essentially an oratio recta [direct talking] man but Barclays is essentially an oratio obliqua [indirect talking] company." No doubt this contributed to Barclays' subsequent troubles. (The Senior Civil Service, too, suffers from too much indirect talking.)

Margaret Heffernan offers an interesting - almost hilarious - analysis of group-think in her excellent book Wilful Blindness. "I've even heard boards discuss how, and why, they are invulnerable to groupthink, oblivious to the irony inherent in their confidence. ... Dennis Stevenson, then chairman of HBOS, eulogised the outstanding board he chaired [at a time when] everyone knew the bank teetered on the edge of collapse. ... [Lord Stevenson cited as evidence] the fact that, even in this crisis, 'we are as one'. He seemed oblivious to the notion that the unity of his board may have been a contributory factor to the bank's mess in the first place."

The same book introduced me to the related concept of cognitive dissonance. This phrase refers to the stress that we all encounter if we try seriously to consider two incompatible views at the same time. We all therefore fiercely hold onto our preconceived beliefs even in response to intense external pressure. One result is that - as we almost all regard ourselves as good people, and essentially honest - we find it very hard to admit to ourselves that we or our organisation is behaving badly. Senior executives typically therefore push back very hard against any regulator's or other suggestion that their actions maybe doing unnecessary harm, including risking their own business.

Cognitive dissonance almost certainly accounts for the failure of many, including Alan Greenspan - a deep believer in the power of the markets, and a fan of financial instruments such as derivatives - to react to all the warnings that something was going very badly wrong in the period before the 2008 financial crisis.

The MacWhirr Syndrome

There is a separate though closely related syndrome, personified by Joseph Conrad's Captain MacWhirr who did clearly recognise the signs of a devastating oncoming storm but also recognised that his employers would criticise him for delaying their cargo if he sought to sail round it. "Suppose", he says, "I went swinging off my course and came in two days late, and they asked me 'where have you been?' 'Went round to dodge the bad weather,' I would say. 'It must have been dam' bad, they would say. 'Don't know,' I would have to say, 'I've dodged clear of it.'"

The armed forces have often taken similar decisions.  Andrew Lambert, discussing the 1812 loss of HMS Guerriere and the deaths of 15 seamen, noted that "Facing an opponent 50% more powerful in guns, tons and men, [the captain's] only hope of avoiding defeat lay in running away, a tactical choice that would have seen him cashiered, or shot." But the classic case was perhaps the sinking of HMS Victoria in 1893 with the loss of 358 lives, following a collision which resulted from Admiral Markham following an order which he knew to be catastrophic. As another admiral wrote some time later: "Admiral Markham might have refused to [obey the order but he would] have been tried by court martial, and no one would have sympathised with him as it would not have been realised that he had averted a catastrophe".

Modern executives who mimic MacWhirr do so because senior managers and stakeholders are very likely to criticise delay and expense, even if the organisation eventually achieves its objective. Executives therefore prefer to take uncertain and dangerous risks rather than face certain (unjustified) criticism.

There was a classic example in late December 2012 when Shell chose to tow the Arctic drilling rig Kulluk through treacherous waters all the way from Alaska to Seattle in the middle of winter, at least in part to ensure that they avoided a $6m Alaskan tax charge which would otherwise have fallen due on 1 January.  The loss of the rig cost Shell around $200m.

There were also a number of senior financiers who understood that they were running their companies into serious trouble in the years before the 2008 financial crisis, but felt they would face unacceptable criticism if they were to follow a less risky course.  Citibank's Charles Prince famously told the Financial Times that "As long as the music is playing, you've got to get up and dance. We're still dancing." This led the same paper's John Kay to conclude that "The man who held the most powerful position in the financial services industry was the prisoner of his own organisation." And Andrew Ross Sorkin, in Too Big to Fail notes that Lehman's Dick Fuld "had known for years that Lehman Brothers' day of reckoning would come ... But, like everyone else on Wall Street, he couldn't pass up the opportunities."

The MacWhirr syndrome is just as common in politics. US Congress Representative Barney Frank commented, when talking about Hank Paulson's dilemma in the middle of the Lehman crisis: "The problem in politics is this: You don't get any credit for disaster averted ... Going to the voters and saying, 'Boy, things really suck, but you know what? If it wasn't for me, they would suck even worse.' That is not a platform on which anybody has ever gotten elected in the history of the world". And others have noted that if Margaret Thatcher had acted to deter Argentina from invading the Falklands, rather than ordering a taskforce to remove the occupying forces after they had landed, she would probably have been remembered as an unsuccessful one-term Prime Minister.

See also my note on behavioural economics for a further discussion of herd behaviour etc.

Diffuse Responsibility

Knowledge of, and/or responsibility for, problems is often widely shared within large organisations, but it is then often the case that no-one feels responsible for addressing or even highlighting the problem. One example was the poor maintenance and terrible safety record at BP's Texas City Refinery, where an explosion in 2005 killed 15 and injured over 170 more. The problems were readily apparent to most employees and managers but - partly under pressure to save money - no-one felt responsible for doing anything about them.

Much the same was true of Bernie Madoff's Ponzi scheme where quite a few financial professionals had worked out that something fishy was going on, but saw no need to do anything more than avoid dealing with him.

And just about everyone in Westminster knew that Members of Parliament were receiving over-generous expenses payments as recompense for their low salaries, but hardly anyone saw much wrong with this ... until Heather Brooke and the Daily Telegraph published the expenses claims in 2009 - triggering immediate uproar outside Westminster, and a few prosecutions. Indeed, many MPs had previously supported an attempt to exclude their expenses claims from the Freedom of Information Act.

The diffusion of responsibility is exacerbated if there are frequent changes of management. This was part of the problem at Texas City as the over-worked men 'close to the valve' stayed put, whilst their managers generally moved on very rapidly, away from an old refinery which lacked the prestige of many of BP's other locations. Managers were not therefore in post long enough to both understand the issues and then have the time and inclination to do much about them.

Profits or Ethics? - Fiduciary Duty, Caveat Emptor and all that

In the absence of strong regulation, many organisations are not incentivised to behave in a socially responsible manner. A commercial company's primary duty is to generate profits for its shareholders. Non-profits and public sector organisations have their own primary duties, and must also operate in a way which generates income and/or reduces expenditure. In more detail:-

First, Corporate' managers' primary duty is to generate profits for their shareholders, and this fiduciary duty in law (and some would say rightly) often takes precedence over the consequences for the environment, for healthy and fair competition, for customers, for employees, and for wider society if their business were to fail. Companies are now very good at filling their annual reports with information about their CSR (Corporate Social Responsibility), but they seem nowadays typically to exercise much less social responsibility than in previous decades. The US Business Roundtable, for instance, in 1978 listed CSR as one of the four core functions of company Boards. It had disappeared by 1990.

Equally, however, fiduciary duty is (or should be) a long term concept. It is easy enough to maximise short-term profits at the expense of customers and other stakeholders, but directors' and executives' duties are to maximise the value of the company in the longer term. The first paragraph of this note lists a number of examples of where neglect or exploitation of customers, or the environment, has been extremely costly over the longer term.

Farrer & Co (Solicitors)'s advice for the Tax Justice Network in September 2013 makes the point about directors' duty to consider the longer term success of the company very well indeed. Here is an extract:

In circumstances where a director decides that the company will eschew tax avoidance, he or she may do so for reasons that he or she considers to be aligned with the long-term success of the company, including for example:

   (i) the adverse risk profile of tax-structured transactions in the long term,
   (ii) the desirability of investment in public health, education, infrastructure &c in the jurisdictions where the company's employees live and work,
   (iii) the need to foster the company's relationship with tax authorities and with consumers,
   (iv) the impact of tax avoidance on the wider community of taxpayers and users of public services, and
   (v) the desirability of the company maintaining a reputation for high standards of business conduct.

. . Our view that other kinds of impact may constitute a legitimate basis for a director's decision notwithstanding an adverse financial impact may be derived from the clear meaning of the [Companies Act] but it is also supported by the pre-existing case law on the corresponding common law duty . . For example in Re Welfab Engineers Ltd (1990) BCLC 833 the directors were found by Mr Justice Hoffmann . . not to be in breach of their duties when they deliberately sold a real property asset at an undervalue, in circumstances where they did so in order to protect the jobs of the company's employees.

Second, there is a well established business principle that the buyer is responsible for assessing the value and suitability of the thing that they are buying. ('Caveat Emptor' = Buyer Beware!) Although significantly eroded by consumer and other law, it remains the case that many companies feel under no obligation to behave ethically, even when their customers are buying complex financial and other products about which they are inevitably badly informed. This applies big time in dealing rooms. For further thoughts on this subject, see Notes 2 and 3 below.

See also my note on consumer protection.

Executives' Behaviour

Their power almost inevitably isolates senior executives from ordinary people, and even more so from their front line staff and their concerns. Top executives all have great confidence in their own judgement and opinions, and so often become over-optimistic. And the scale of their operations means that they have to think in abstract terms where risks, and even death rates, become mere numbers that have to be managed. This can easily cause whole management teams to characterise as naive or unworldly anyone - and in particular a whistle blower or journalist - who expresses any concern about organisational culture. Lord Browne, BP's Chief Exec at the time of the Texas City disaster, was said by his executive assistant to show 'no passion, no curiosity [and] no interest' in safety. It can be worse if an organisation is highly focused on the power and influence of a single individual, often someone with a heroic leadership style. Their direct reports then spend all their time trying to second guess their hero's wishes, rather than think for themselves or commission analysis which might challenge their leader's views.

At another level, the behaviour of large companies' senior executives has become quite fascinating. One begins to wonder whether big business has succeeded warfare as the most exciting form of competition between human organisations. Some modern boardrooms appear to attract those who would, in previous generations, have sought to command large armies. Instead of invasions, we nowadays have corporate takeovers; instead of gold braid and military honours, we have executive salaries. The leading actors therefore remain the odd combination of hugely ambitious, sometimes inspirational, but too often also highly self-absorbed and disastrously inept. For Haig, Montgomery, Patton and MacArthur substitute Dick Fuld, Fred Goodwin, John Gutfreund, Tony Hayward, Jeffrey Skilling, Bob Diamond, Rajat Gupta? Indeed, I understand that Turkish Islamists rather bitterly note that the mujahids or aspiring warriors of old have become the muteahhits or construction tycoons of today.

It certainly seems to be the case that power changes the behaviour of previously decent men and women. 'Power corrupts ... etc.'. It is perhaps inevitable that powerful CEOs' sense of right and wrong becomes aligned with the norms and expectations of others who are similarly rich and powerful. Neuroscientist and psychologist Ian Robertson goes further and draws attention to research which shows that power increases testosterone levels which in turn increases the uptake of dopamine in the brain, leading to increased egocentricity and reduced empathy (New Scientist 7 July 2012). Power also reduces anxiety and increases appetite for risk.

And yet, as the FT's Lucy Kellaway points out: "Modern CEOs seem to have no [public] opinions, especially not negative ones. If they feel one coming on, they have been trained by their lawyers and PR advisers to suppress it." It is certainly difficult these days to correct the mistakes or improve the behaviour of an organisation without laying oneself open to the charge that previous behaviour was in some sense faulty and so worthy of criticism and/or claims for compensation. But senior managers are also themselves all too often concerned to appear infallible (and un-sue-able) and so organisations therefore get locked into defensive modes, from which they becomes increasingly difficult to extricate themselves - and so the problems get worse. The reputation of BP's Tony Hayward was for instance badly damaged (yet further) when he failed to answer many of the questions put to him by US Congressmen in June 2010.  Barclays Bob Diamond also did himself no favours when responding so blandly to UK politicians' questions in July 2012.


  1. By the end of the financial crisis, the UK taxpayers' support for the banking sector totalled £850 billion, and the cost of the consequential economic recession was estimated by the Bank of England to be up to £7,400 billion (Andrew Haldane speaking in Hong Kong in March 2010).
  2. Harry Markopolos notes that "... the [financial services] industry is based on predator-prey relationships ... If you don't know who the predator is then you are the prey." (See also Michael Lewis' Liar's Poker.) Even Warren Buffett, normally a stickler for corporate ethics, said "If you have to care who's on the other side of the trade, you shouldn't be in the business of insuring bonds" when answering his shareholders' questions about the accusation that Goldman Sachs tricked another large company into insuring sub-prime mortgage securities, despite privately believing them to be a very bad risk.
  3. For a very readable account of the origins of the financial crisis, see Alex Brummer's The Crunch in which he notes that "[One particular] episode, with its blatant exaggerations and deceptions, sums up the complete folly of the sub-prime bonanza in America. Bear Sterns, like so many financial firms, had lost its moral compass.".

Martin Stanley