They should have titled it How To Buy A Headline.
Manitoba Hydro has submitted a long, long, long report by their hired guns, KPMG, to the Public Utilities Board in their latest effort to undermine the New York consultant who blew the whistle on the shaky, scary way that Hydro is really run.
"KPMG review calls allegations unfounded" blared the headline in the Winnipeg Free Press, written by someone who hadn't read a word of the KPMG report and who was relying for guidance on reporter Mary Agnes Welch, who apparently never read beyond the carefully worded conclusions which told Hydro exactly what Hydro wanted to hear.
Which certainly wasn't "Facts Vindicate Whistleblower", the headline that anyone who actually read the report would agree is an accurate reflection of the contents.
Manitoba Hydro is not being mismanaged, said KPMG, to wild huzzahs from Manitoba Hydro managers. KPMG had been hired to repudiate the whistleblower's allegations -- and they did the job, to Hydro's satisfaction.
The New York Consultant went to the Manitoba government 19 months ago with a variety of concerns about the way Hydro was being run. The corporation, said the whistleblower, was making multi-multi-billion dollar decisions based on data spun out of outdated computers using outmoded software analyzed by a tiny team of good ol' boys. Their "expertise" had already cost Manitoba $1 billion in unnecessary expenses, declared the consultant, and Hydro was risking bankruptcy---after plunging Manitobans into unnecessary blackouts and brownouts. Scary stuff indeed.
KPMG spent more than 100 pages discussing the computer modelling that went into decisions to spend $18 billion on a series of dams to provide power to sell into the U.S. over the next 30 years.
HERMES is the main tool used to support operations scheduling. Modules within HERMES represent the MH system in a significant amount of detail. These modules have been developed and regularly updated over many years and reflect extensive work to calibrate model outputs to actual system performance and thereby continually validate the model.
SPLASH is a simulation tool designed to support MH’s long-range system planning. The output from the SPLASH model provides information used to evaluate the economics of power resource options such as power export marketing contracts, system enhancements and surplus energy rate programs.
SPLASH is also used to support financial forecasting.
SPLASH is also used to support financial forecasting.
Manitoba Hydro's models were "reasonable", declared KPMG.
Reasonable?
That's like asking your best friend if the movie he saw last night was any good. "Ehh. It's okay," he says. Okay? Hardly a resounding endorsement. Sort of like... reasonable.
We didn't have to dig deep to find out why KMPG was so lukewarm.
- "We did not undertake an audit of the models or verify their computational accuracy."
"Rather, we assessed the overall reasonableness of the modeling approach..."
They didn't bother to verify that the models were accurate? What?????
- In fact, KPMG suggested that Manitoba Hydro, maybe, might think about, if they have time, backtesting their models.
- In fact, KPMG suggested that Manitoba Hydro, maybe, might think about, if they have time, backtesting their models.
"In addition to the current validation procedures used for HERMES and SPLASH, MH should consider incorporating backtesting practices to validate its models."
Backtesting---that's when you put in known data and see if the computers spit out something close to the already known answer.
To you, that might seem obvious, but to the geniuses running Manitoba Hydro, not so much. Although somebody did suggest it before.
"Backtesting is a means by which errors in the inputs can be removed in order to verify the appropriateness of the model logic. The Consultant asserts that MH does not back test its HERMES or SPLASH models. Accordingly, the Consultant argues that management decisions and reports based on the outputs of these two models may be flawed." (KPMG Report)
- KPMG also alerted the PUB to the fact that Manitoba Hydro is one good flu away from chaos.
It seems that the utility is run seat-of-the-pants style by a tiny clique of employees, who control the data going in and out of the computer models and whose skill in assessing the data consists of on-the-job expertise. If one is on holiday, another is getting married, another has the sniffles and another gets hit by a bus, Hydro is in deep trouble.
"As HERMES and SPLASH are in-house models and operated by a small group of highly skilled modelers, Manitoba Hydro should provide more formal documentation of the models to preserve their proprietary information and assist new modelers. This will require dedicated additional resources to develop the documentation, but doing so will help mitigate risk in the event of staff turnover."
"The skills required to operate HERMES are derived on-the-job training and experience. The nature of this process is that it has not been highly reliant on formal documentation or training courses. While this is understandable in the circumstances, it creates some risks with respect to knowledge sharing and corporate exposure to the potential loss of key personnel."
"Manitoba Hydro has a relatively small group of highly skilled analysts and negotiators in power sales with deep experience in long-term export sales contracts, and more formal documentation of the pricing analysis will help preserve that experience."
• "Lack of formal documentation, oversight or validation also creates a risk that the overall model is seen as a “black box” to outside stakeholders, both within and outside the company."
Translation: If nobody can understand how you reach your conclusions, there could be trouble, and you don't want that.
• "MH would benefit from more formal documentation and oversight of the modeling process. However, MH must also be aware that such documentation and oversight will require additional resources without necessarily producing an immediate financial return in terms of improved forecasting performance."
Good luck finding the answer by mucking through 162 pages of gobbledygook intersperced with long sections of numbers censored by Hydro and commentary censored by the New York consultant.
KPMG says Hydro is so consumed with the fear of drought that it's extremely conservative in its decisions on how much power to sell the Americans. Since taking a bath in the last big drought in the 2004, they include "curtailment" provisions in sales contracts which allow Hydro to stop sending power to the U.S. if it's needed at home. Hydro even has contracts to buy power from the U.S. which is sold to American customers (if we've deciphered this properly):
"13. For example. PUB order 32/09 states on page 27 (of 48) that “Dependable hydraulic generation for the year 2003/04 was 18500 GWh that being a level significantly below the 21 000 GWh on which MH bases its potential for firm export contracts (after fullfilling the domestic requirement).
Yet MH has not lowered the dependable resource level to 18,500 GWh; rather MH now defines the dependable resource as a multi year historical event (not a one year, event) This effectively rneans that once every fifteen years (the deemed frequency of the 2003/04 drought event) MH will be faced with dependable energy shortfalls comparable to 2003-04 though perhaps in an environment of much higher import prices.
MH has not adequately demonstrated that the Corporation's mean energy forecast adequately reflects this self-imposed additional risk”
In a nutshell, when caught saying it had almost 14 percent more power to sell than it actually produced, Hydro simply redefined how much it needed to set aside for customers in Manitoba.
There, doesn't that leave you reassured?
• "MH would benefit from more formal documentation and oversight of the modeling process. However, MH must also be aware that such documentation and oversight will require additional resources without necessarily producing an immediate financial return in terms of improved forecasting performance."
What about the key question: is Manitoba Hydro betting the farm on its planned mega-sales to the United States? Are we bound by contract to sell power we may not have?
Good luck finding the answer by mucking through 162 pages of gobbledygook intersperced with long sections of numbers censored by Hydro and commentary censored by the New York consultant.
KPMG says Hydro is so consumed with the fear of drought that it's extremely conservative in its decisions on how much power to sell the Americans. Since taking a bath in the last big drought in the 2004, they include "curtailment" provisions in sales contracts which allow Hydro to stop sending power to the U.S. if it's needed at home. Hydro even has contracts to buy power from the U.S. which is sold to American customers (if we've deciphered this properly):
"By purchasing firm import contracts, MH effectively ‘outsources” some of its back-up requirements to external parties. A key advantage of this approach is that the thermal generation thus obtained has more opportunity to be used for purposes other than providing back—up to MH. This reflects a location that is nearer to major US markets. and less subject to import transmission constraints, and which can better take advantage of diversity in demand protiles. (US utilities tend to be summer peaking while MH is a winter peaking utility.). As a consequence. MH is likely to pay lower costs for such back—up generation by outsourcing it than by building thermal generation locally."
KPMG concluded:
"Based on the information presented above, we see no evidence that MH is over committing its firm dependable energy production through the proposed export contracts is and thereby unnecessarily exposing MH to volume risk."
"Based on the information presented above, we see no evidence that MH is over committing its firm dependable energy production through the proposed export contracts is and thereby unnecessarily exposing MH to volume risk."
But then there's this footnote that slipped through the censorship process which reflects the fact that last year the Public Utilities Board redflagged a problem with Hydro's alleged aversion to risk.
"13. For example. PUB order 32/09 states on page 27 (of 48) that “Dependable hydraulic generation for the year 2003/04 was 18500 GWh that being a level significantly below the 21 000 GWh on which MH bases its potential for firm export contracts (after fullfilling the domestic requirement).
Yet MH has not lowered the dependable resource level to 18,500 GWh; rather MH now defines the dependable resource as a multi year historical event (not a one year, event) This effectively rneans that once every fifteen years (the deemed frequency of the 2003/04 drought event) MH will be faced with dependable energy shortfalls comparable to 2003-04 though perhaps in an environment of much higher import prices.
MH has not adequately demonstrated that the Corporation's mean energy forecast adequately reflects this self-imposed additional risk”
In a nutshell, when caught saying it had almost 14 percent more power to sell than it actually produced, Hydro simply redefined how much it needed to set aside for customers in Manitoba.
There, doesn't that leave you reassured?