When the government recently pledged ‘gold standard full fibre’ broadband, a collective cheer went up. For a short while, until we reflected upon the fate of most such announcements, the realities of stuttering broadband, and the prospect of dealing with one of the big providers.

The history of this industry says the ‘gold standard’ will not happen without four major reforms (skip to the end to see them). Just how much of the pledged £1bn will end up in fibre to premises and how much in waste or super-profit? How many of the “up to 2 million homes and businesses” will actually experience another politician’s promise?

My 35 years in and around government has led to a certain cynicism. Announcements are primarily designed to fend off the press, the opposition and campaign groups, and to buy the government time by throwing a new idea to the snapping packs. “Yes, we know our transport/health/schools system is poor,” the Minister will admit. “But don’t worry, we have a plan to fix it.”

It’s a familiar story: fast forward a few years and the policy has been scrapped, it turns out no ‘real’ money was allocated, and the minster and civil servants have now changed. The electorate shuffles on, impotent in the face of such ineffectiveness from on high. There are exceptions of course, just a few.

I interviewed industry insiders who have a combined 100 years of experience of the telecommunications industry to see what is wrong and what might be done.

“Our customer experience has been far worse than in any other country we have lived – Canada, Australia, China and Hong Kong. It took us 4 months to get an engineer to resolve a physical network fault. In another more than 15 hours on the phone to resolve issues with our service profile.” That represents 2 days unpaid work for the unfortunate business or home experiencing this.

BT’s answer to its failings, caused by its lack of investment in fibre and switches and its disorganized and costly installation and repair operation, is to invest in its ‘customer pacification’ programme. Which consumers pay for of course. Strange choice.

One local government official leading the installation of rural broadband spoke at an LGA conference. Malcolm Corbett of the Independent Networks Cooperative Association asked him what it was like working with Gigaclear compared to BT. (Gigaclear has been laying fibre-optic cables under the fields and streets of Britain’s most rural outposts since 2010). He thought for a few seconds then said, “It’s like escaping from an abusive relationship.”

He thought for a few seconds then said, “It’s like escaping from an abusive relationship.” Everyone giggled, smiled, and understood exactly what he meant.

Long gone are the days when BT stood up internationally as world class. The charge sheet runs to several pages. BT is not the only villain in the broadband/mobile industry, just the biggest and most practiced.

Why all these tricks? Why are they allowed? Why can’t broadband be a real market, like retail, with real choice, straight pricing, and products that work? Simples. Is continual suffering relieved by occasional advances the consumers’ lot. Or, can something be done? Let’s see.

The absolutely clear political intent by a most determined prime minister (one Mrs Thatcher) of the 1984 Telecommunications Act was to create real competition (she must be turning in her grave).

This type of privatisation sat between the transfer in ownership from public to private of Rolls Royce and BP, companies already operating in real competitive markets, and the quasi-privatisations of train operating companies that never had and never will live in real markets.

The telecoms industry had all the potential to become a proper market and, through fierce domestic competition, to take a global lead as the rest of the world followed in introducing its own markets.

But this meant rather than milking the national assets it had inherited, BT had to transform from a bureaucratic monopoly into a dynamic high-tech operator. Think Apple rather than Whitehall.

Alas, BT has spent the last 30 years defending all it surveys. Successive governments and regulators have failed. Why?

Well, they simply don’t have the know-how, for a start. The MP and the SNP’s spokesperson for digital, Calum Kerr, with 10 years industry experience, commented recently on his fellow politicians’ ignorance: “There is a lack of knowledge, which means a lack of debate. Policy doesn’t get debated properly or challenged.”

In our system of government, the civil service should compensate for non-specialist ministers but instead elevates ignorance to a virtue by its belief in the ‘clever generalist.’ Senior civil servants jump from one specialist role to another, rarely staying more than three years in one. 10 years is the minimum to compete on industry knowledge.

The regulator does employ specialists but their thinking is dominated by economic theories as the means to market success, when the solution rests with technological know-how and the backbone and nous to spot when they are being taken for a ride and to call a halt with a sledgehammer. Regulation in all fields so often falls down at the enforcement stage. The Chinese regulator employs over 10,000 technologists and has its own R&D set-up.

Ignorance is an ideal growing medium for preferential lobbying. All the major companies and industries deploy as many staff and consultants as it takes against the regulator and against those backbench MPs seeking to represent their constituents’ interests. Smaller rivals, like INCA, cannot compete for policy influence.

Big companies find it easy to bamboozle civil servants and ministers, who have zero knowledge of the technologies and next to no grasp of the intricacies of sharp business practice. At best, the civil servants exist in a theoretical, black and white world, not the grey of real business. Innocents abroad. Any proposed ‘interference’ is met with projections of techno doom. And it is so much easier to shelter behind legal or economic constructs rather than be exposed to the interpersonal conflicts inevitable in dealing with the tricks of this trade.

No one sees it in action, but preferential lobbying is the biggest influence on decisions for this and many other industries. Actually in almost all government decision-making. It was not always like this. But PL has probably been the single most powerful influence on the decline of democracy and the rise of wealth and power inequality over the last 40 years. Several books catalogue its impact, the most recent from the ALTER-EU coalition, which sets out its road map for greater transparency and accountability in the European Union with the publication of a new book: Bursting the Brussels Bubble – the battle to expose corporate lobbying at the heart of the EU. It’s the same in Westminster and Washington.

BT has lobbied forever to retain Openreach. In practice, it is simply not possible for a real market to develop for as long as one provider owns the fixed line network for around half the country. In its continuing impersonation of a chocolate teapot, Ofcom’s latest decision for further separation but continuing ownership of Openreach by BT is another fudge purporting action.

“This is a significant day for phone and broadband users,” said Ofcom chief executive Sharon White today. “The new Openreach will be built to serve all its customers equally, working truly independently and taking investment decisions on behalf of the whole industry – not just BT.”

In practice for the non-BT consumer, the service may get a little better. But, look at Sky where Mr Murdoch owns only 40% through Fox, yet totally dominates and controls it. Openreach will continue to do BT’s bidding. It should have been sold 20 years ago, and owned jointly by all the telecom companies. Or run as an independent non-profit mutual like Welsh Water.

The result of all this lobbying is an out of balance power relationship between us and the big companies. The regulator and government department, DCMS, are supposed to level the playing field, but either don’t or do a bit after years of consumer suffering. Like the gross overseas voice roaming charges, it took the final snapping of the EC regulator’s patience and her legislation, approved by the European parliament in 2007, to call a halt to that consumer abuse. (I do wonder sometimes if there should be a universal crime of abuse covering all of its forms). These charges have been replaced with further trickery: ‘day plans’ that are less costly but still very profitable.

Once again we find the hidden but broken hand of governance at the heart of this failure. The industry performs as well as its poor governance. This is invariably true. Every institution fails: Ofcom, Department for Culture Media and Sport, Government, National Audit Office, House of Commons, House of Lords. And the reason they all fail is that they exist within a system of government that has never ever been designed for this purpose. It’s not their fault, it’s the system they inhabit. The reality is that this has allowed BT to become a national disgrace.

What is to be done?

Short of a much reformed system of government – entirely needed but will take a while to produce – our only hope is Parliament. Setting the terms of governance is or should be a major part of its job. It has to drop its obsequious acceptance of its place as a debating chamber, cross those party lines and grip the system. Hanging onto one’s seat, hoping for a ministerial job, or finding refuge in one’s constituency are insufficient excuse for ducking this issue. Haven’t you noticed, politics isn’t working? The country demands you fulfill your role.

First we need the facts of industry performance. Facts have to be produced independently: self-scoring ministers – or regulators – using rhetorically spun statistics conceal the truth. This is a job for a fully independent Office for National Statistics. Then we will no longer spend years arguing over the scores. We can concentrate on how to improve.

Second, the regulator has to be given strong objectives with all decision making in full view. The Monetary Policy Committee has operated this way since 1997 with members’ views and votes recorded publically. The discipline and learning this produces has resulted in an interest rate regime far superior to its previously politically set method.

The industry should be able to express its preferences but not behind closed doors. All their submissions should be made public and subject to open discussion. No more lobbying in private. They should be rationed to give small companies equal air time.

Third, Ofcom and DCMS have to be staffed and led by specialists with the right experience. This should include tough enforcers. Lance Corporal Jones in Dad’s Army had it right: “They don’t like it up ’em!”

Fourth, Parliament should establish one select committee from both the Commons and the Lords whose remit should be to ensure that all the above is happening and producing results. Its powers should include the appointment of the chief regulator and its board.

Then we need a plan, one that is based on a thorough grasp of practice in other countries, and that sets clear objectives for the services we need, and the actions to get us there.

Theresa May has said corporate governance needs to be strengthened. Indeed. Her Green Paper on industrial strategy has been published. Parts of it look sound. But certainly its broadband element will flounder without the reforms of government governance set out here. Time to act. Then all those flowery promises should actually occur and we can compete.

High quality Broadband is not just about you or me watching The Crown in HD without a single whirring centipede interrupting our viewing pleasure. This is nation critical, this is industrial strategy, this is post-Brexit essential.

The new digital divide between mainly rural and mainly urban areas is locking industry into the South East and limiting growth even in somewhere as wealthy as Gloucestershire.

Business enablement (communications, logistics, roads, rail, and a skilled national workforce) is vital to be successful outside the EU. Wake up Parliament: your country needs you far more than your political party.