This page location is:
BPA.gov - Bonneville Power Administration
News & Us
News & Us
Freedom of Information Act
Projects & Initiatives
Finance & Rates
Financial Public Processes
Cost Verification Process
Residential Exchange Program
Surplus Power Sales Reports
Involvement & Outreach
Community & Education
Lands & Community
Bonneville Purchasing Instructions
Buying or Selling Products or Services
Financial Assistance Instructions Manual
How to Pay BPA
Reliability Program and NERC Standards
Freedom of Information Act
Legacy computer system retires after keeping the lights on for 38 years
4/24/2013 12:00 AM
About 100 BPA employees and retirees gathered March 22 at headquarters to bid farewell to the legacy system and its successful retirement. Pictured from left are veteran RODS experts Harry Comins, Bill Ostrander, Jim Dow and Steve Pongracz.
Long before the age of the Internet, Angry Birds and omnipresent digital connectivity, a group of BPA innovators made history.
BPA’s Real-time Operations Dispatch and Scheduling system (aka RODS) came on line in 1974. It was one of the first systems of its kind.
And here’s another bit of history: The computer experts brought in to work on the project included two whiz kids from Seattle, Bill Gates and Paul Allen, co-founders of Microsoft Corporation. (See sidebar below).
RODS was BPA’s first computerized system to manage operations. It did pretty much everything, from automatic generation control to bill settlement to transmission scheduling as well as many other functions that supported hydro operations.
For decades, BPA relied on RODS to manage its critical grid operations. But changes in technology over the years made using the old system an operational risk for the agency.
This February, after 38 years of steadfast service, RODS was officially shut down.
Retiring the old system was no simple task. It took five years of thoughtful planning and coordinated development to transfer RODS’ functions to new applications built on modern platforms.
“Replacing RODS was one of the more urgent and difficult automation efforts BPA has taken on – and it was also very high risk,” says Larry Buttress, acting executive vice president of Internal Business Services. “We needed to replace a mission-critical, monolithic, legacy system that had been in operation for 38 years with a portfolio of more supportable applications. The challenge was to accomplish all this without causing a disruption to critical business operations.”
Fortunately for BPA and the region, the project team was successful in meeting the challenge.
“The RODS Replacement Project was completed on time, $5 million under budget and with no operational disruptions − a tremendous accomplishment,” Buttress says. “The team that worked on this did just an outstanding job!”
To put this achievement in perspective, here’s a look at how RODS was created, the central role it played at BPA and the complexities of retiring the system.
An innovative system to manage the grid
“When RODS was launched in 1974, it was an engineering feat unto itself,” says Andy Macklin from the Information Technology Project Management group, who served as the RODS Retirement Project manager since 2009.
This photo from 1972 shows the original KA-10 mainframe units that ran the RODS system.
“The agency needed a system to manage the grid. But in those days, you couldn’t just go out and buy one. BPA engineers had to essentially invent the system,” Macklin says. “It’s a testament to the ingenuity of BPA staff and contract personnel that RODS was created in the first place – and that it lived 38 years.”
Dozens of BPA employees devoted their whole careers to RODS. Bill Ostrander is one of them. In 1969, he began working on a RODS prototype, along with several others.
“Taking in data from the electric grid, refreshing it each second and making it available to multiple consoles was an unusual application that required unique programming techniques.” Ostrander says.
Initially, the RODS prototype ran on a CDC 1700 mainframe computer, about the size of a standard desk. Although the data storage discs were large, the central processing unit only had 16 kilobytes of memory. (For comparison, 16 KB is about the size of a short, text-only email message you’d send today.) One of the team’s most important innovations was figuring out how to maximize hardware resources using the technology of the day.
From 1989, this shot shows part of the RODS KL-10 mainframe and the large (92 megabyte) disks used to store the operating system, the application programs and the data that formed RODS.
In 1971, BPA awarded a contract to TRW, the large aerospace firm, to write software for the system. TRW brought in computer programming experts that worked closely with BPA engineers. The team took a standard time-sharing system (TOPS-10) and made it do the real-time functions necessary to manage BPA’s power grid operations.
As Ostrander recalls, the system could run 180 different processes at any one time. That took a lot of programming code – 1.2 million lines of code written in FORTRAN to be exact.
The system could do so much. That’s why it played such a critical role in BPA’s business for so long and why it was difficult to replace.
For BPA, the fundamentals of operating the federal hydro system haven’t changed that much over the years. But technology has.
In the 1980s, information systems housed on mainframe computers started migrating to distributed networks. Around that time, BPA realized RODS had a limited shelf life and began attempts to replace it.
In the 1990s, as digital technology evolved and the Internet was becoming a critical part of business operations, several key “pieces” of RODS, including transmission and power scheduling, were replaced with new applications.
RODS moved from being BPA’s only system to serving as a kind of data transfer hub for the agency’s business operations. For example, RODS ensured that data from the scheduling system was transferred to the billing and automated generation control applications. Additionally, many functions and users continued to rely on RODS directly.
“As a result, when the RODS Replacement Planning Project started in November 2007, the task was not only to determine how to replace the functions remaining on RODS, but also how to unravel the intricate data flows,” Macklin says.
First, do no harm
Julie McKenney and Andy Macklin, who each served as project manager of the RODS Replacement effort, are shown at the system’s retirement party. The project’s core team included BPA technical architects Calin Epifanov and Russ Huntley.
“We knew replacing RODS would be difficult. It was like the elephant that no one wanted to tackle,” says Randi Thomas, manager of System Operations.
“BPA had made multiple attempts to replace RODS, but in 2007, the stars aligned,” Thomas says. “Strong cross-agency involvement at many levels and the alignment of executive sponsors, project managers, subject matter experts and RODS users made the project successful this time.”
Step one was a thorough analysis of the system. Julie McKenney, the initial project manager, led a team of six that studied the system and documented its functions. With extensive input from users as well as the employees who had built and maintained RODS over the years, the team created a replacement plan.
“Our guiding principle was to work carefully so we didn’t break anything that would harm BPA’s ability to manage its operations,” McKenney says.
The stakes were high. A system outage could mean millions of dollars in lost power sales and potential fines, not to mention the damage to BPA’s reputation.
Team members spent about 18 months planning − consulting with 125 subject matter experts from 29 BPA organizations − and another four years replacing and retiring core functions of the system.
Deconstructing RODS and shifting to new applications was tricky. The project team compares it to playing Jenga, the game where players try to pull out pieces from a tower of wooden blocks without knocking over the whole stack.
“We had to be really careful removing each part of RODS because many of the functions were interconnected,” says Ostrander. “For every action we took, we needed to ensure users could still conduct their business. It took a concerted effort among all the application programmers and users to accomplish that.”
Unlike Jenga players, the team followed a set sequence for which part would be replaced first, second and so on. Once a new application was developed, it was run in parallel with the old one to avoid loss of data or functionality. This systematic approach prevented the whole structure from crashing down.
To replace the multiple functions of RODS, BPA staff developed four brand new systems from the ground up. Additionally, they made significant modifications to four existing systems as well as many minor changes to other applications.
The team began phasing in RODS replacement systems in 2010. By November 2012, nearly all of BPA’s operations applications had been successfully migrated onto new platforms, essentially leaving RODS behind.
“The cutover in November was a complicated all-night effort involving technical and operational staff from across BPA,” Macklin says. Then in January, the team moved the final piece, one isolated application in Dispatch.
The actual power-down of the legacy system and all its remaining 32 consoles took place Feb. 19 in the Dittmer Control Center.
Today, BPA employs about a dozen systems that do what RODS did, but in highly specialized ways. Ostrander reports that after the initial fine tuning all new systems are subject to, all the replacement systems are performing well.
On March 22, about 100 BPA employees, retirees and former BPA executives gathered to bid farewell to the legacy system and celebrate the retirement project’s success.
“Thanks to the tenacity of everyone involved and the commitment of BPA leadership, we got the job done successfully without interrupting operations. That’s something we’re all proud of,” McKenney says. “But I know for many folks, especially those who worked their whole careers creating, maintaining or replacing portions of the RODS system, it was a bittersweet moment when the plug was pulled.”
As Macklin sees it, “The original business case for RODS retirement was almost entirely founded on risk mitigation and avoidance.
“But it’s my personal hope that the project will be more remembered for all the people who contributed to it and what they created together. Ultimately, this team achieved something that supports the reliability of BPA’s business today as well as its mission in the future.”
Bill and Paul’s excellent adventure
And now, here’s the real story about how Microsoft founders Bill Gates and Paul Allen came to work for BPA on the RODS project.*
When Gates and Allen were teenagers, they attended Lakeside School, an exclusive private school in Seattle, where they shared a common enthusiasm for computers. They learned how to write computer programs in BASIC and helped computerize the school’s payroll system. In 1972, Allen and Gates founded their first company and sold a computer that counted traffic to the city of Seattle.
Paul Allen and Bill Gates in 1981, the year Microsoft was incorporated. (Photo: Microsoft)
Meanwhile, down in Vancouver, Wash., BPA had contracted with TRW to write software for the new RODS system in 1971. The firm underestimated the complexity of the programming effort, and by late 1972, the project was seriously behind schedule. TRW scoured the country for de-bugging experts who had experience with PDP-10 software, and the names of Bill Gates and Paul Allen kept popping up. TRW invited the pair to come in for an interview. Despite their youth, they were offered jobs at $165 a week.
Here’s how authors Stephen Manes and Paul Andrews describe it in their book about Gates:
“So when TRW made its offer, Gates and Allen jumped at it. A paid job with unlimited access to their cherished PDP-10 machines? After years of scrounging cheap or free time from whatever source they could finagle? It sounded like programming nirvana. … The two friends tossed some gear into Allen’s car, drove the 160 miles from Seattle south to the ‘other’ Vancouver, and rolled up to the RODS control center.”
The pair began working on the RODS project in January 1973. Gates got approval from Lakeside to pursue this opportunity as a senior project for his final semester of high school. Allen, who was 20, dropped his studies at the Washington State University to take the job.
In addition to getting paid to do what they loved, the whiz kids also honed their programming skills. TRW had hired some of top programmers in the country for the project. One of the best was John Norton, who became something of a mentor to Gates. Whenever the teenager made a mistake or did sloppy work, Norton would review his code and explain how he could do it more efficiently.
BPA’s Bill Ostrander, who was part of the first development team that began working of RODS in 1969, remembers Gates.
“I didn’t see Bill a lot because he generally worked nights,” Ostrander says. “But I distinctly remember him talking about wanting to develop a system to compete with IBM. And I thought if anybody could do it, this kid could. He was only 17, but he was really smart.”
Gates and Allen continued working on the RODS project through the summer of 1973. They both left BPA that fall − Gates started his freshman year at Harvard and Allen returned to WSU.
In 1975, Gates left Harvard to form Microsoft with Allen. They wrote a version of BASIC to program the first personal computer, the Altair 8800, and planned to develop software for the newly emerging PC market. In 1980, Microsoft began its domination of the microcomputer industry when it licensed the MS-DOS operating system for use in IBM's first personal computer. Microsoft grew to become the world’s largest software maker and one of the world’s most successful companies.
* Sources: “Idea Man” by Paul Allen; “Gates: How Microsoft’s Mogul Reinventing an Industry − and Made Himself the Richest Man in America” by Stephen Manes and Paul Andrews; “Hard Drive: Bill Gates and the Making of the Microsoft Empire” by James Wallace.
Related Articles (by tag)
Lost films chronicle dawn of hydroelectric power in the Northwest
Wednesday, June 11, 2014
Dispatchers and operators get a charge out of discovery
Wednesday, March 26, 2014
Publications feature Big Eddy-Knight, connector shunt
Tuesday, March 25, 2014
Mainzer taps Gendron, Ehli to lead Power and Corporate Strategy
Wednesday, March 05, 2014
BPA Engineer's Historic Photography Featured at Oregon Nikkei Legacy Center
Friday, February 28, 2014
Submit a Comment