Power Plant Recommissioning

To earn the title of Control Engineering’s 2022 System Integrator of the Year, Hargrove Controls & Automation was required to submit detailed documentation of project success stories and customer references to demonstrate our commitment to customer satisfaction. Previously we shared highlights from one of those project success stories in which we innovated a custom simplified batch solution for a chemical company. Here we discuss the details of a project that had previously proven too difficult for the client’s in-house engineers and another system integrator.

As an independent system integrator, Hargrove Controls & Automation Team is well versed on both legacy and new DCS/PLC process controls system technology across a wide spectrum of platforms (Rockwell Automation, Siemens, Emerson, Honeywell and the list of ICS platform experience goes on). As a partner to our clients, we are often challenged to find solutions where others have failed. In this project, our Teammates’ depth of experience was put to the test when our client purchased a non-operational existing facility.

Background

The power plant had been shut down for 5+ years, which posed significant issues when attempting to get the facility back up and running. In addition to the challenge of restarting the power plant, the facility was being repurposed for a different opportunity than it was originally built for which required new modifications and accommodations to be made. The client had tried various routes to recommission the facility on the existing DCS platform and communication network, but they were not able to do so.

Summary of Work Completed

Hargrove Controls & Automation was brought onsite for an extended period of about 10 weeks to provide additional support and expertise. The key to helping the client was the technical depth of the Hargrove Team. Hargrove Teammates understood the client’s system very well and worked within the nuances of the different revisions of the control systems package releases and were able to connect the pieces and get them stably operational.

The work performed included recovering Rockwell Automation system servers, upgrading components in the system servers, recreating their Historian SE system and connecting it to their PlantPAx DCS; specifying diverse hardware and facilitating the orders; installing and configuring the hardware upon arrival, including verifying compatibility; and troubleshooting the DCS HMI system, Powerflex drives, relays, and starters.

The site’s networking heavily utilized DeviceNet and ControlNet, which had to be mapped out and understood properly in order to facilitate effective troubleshooting – which couldn’t be done in-house by the client. Hargrove assisted in the testing of single speed, multi-speed, and variable speed motors, and with the help of instrument field techs, assisted in loop checks, which needed to be done for the 300+ instruments involved in the process onsite. They investigated any issues that arose to determine if they were DCS related or in the field. If they were DCS related, Hargrove worked to resolve the issues. If they were determined to be in the field, Hargrove enabled the client to implement fixes or order new instruments. The Hargrove Team continued to provide remote support to the site as needed.

Benefits

Hargrove’s Team was able to generate substantial cost savings for the client by helping them become stably operational without taking the time or expense to implement a major overhaul of the entire system. The major overhaul could be planned for a future date when the company has achieved an operational capacity that can sustain the economic impact of upgrading the entire control system.

Hargrove Engineers Share Details of the Project Challenges

Project Engineers Ali Badruddin and Mike Howley of Hargrove Controls & Automation share how Hargrove went to great lengths to support a large engineering services client who was in the process of recommissioning a dilapidated power plant in East Texas.

As would be expected with a site that had been idle for years, the equipment was generally in disrepair. “Because of its age, it’s not a modern, up-to-date site and we ran into major hurdles throughout the project. Obviously, communications play an important part in how the power plant functions,” said Badruddin. “The existing equipment was not using the latest Ethernet IP; DeviceNet and older communications technology was embedded into this facility and that was one of the hurdles we faced.”

“It was a large and interesting network,” said Mike Howley, Project Engineer at Hargrove. “The network had Ethernet communication from the PLCs to their SCADA HMIs. The PLC internal devices used ControlNet. The internal PLC remote racks were talking through ControlNet, the VFDs were actually using DeviceNet, and they had a device that converted ControlNet to DeviceNet.”

One motor control center (MCC) room had more than a dozen variable frequency drives (VFDs) on DeviceNet. “Some of them were working, some were not,” said Howley. “We had to figure out why some of these drives were not working. We had to replace one of the ControlNet-to-DeviceNet modules and program it onsite, which was a hurdle.”

Cascading problems can arise with older equipment and protocols. “We were forced to revert backwards to Windows XP because some of the equipment was not compatible with Windows 7. This created a challenge, especially when an older communications protocol like DeviceNet was involved.” said Badruddin.

In addition, the PLC controls were tied to the HMI servers which historicized the data.  “Historical data is important because it shows you which equipment is running well and what needs attention,” noted Badruddin. “The first problem we encountered was that the entire server was locked out.”

Other challenges quickly followed. “There were existing redundant HMI servers and one fell because the air conditioning wasn’t working in the building,” said Howley. “The server failed due to overheating, and when we received the replacement, the operating systems didn’t match.”

The Hargrove Team reviewed the server equipment item by item to discover what was functioning and what needed to be replaced. The procurement and replacement of legacy components required the team to get creative in collaboration with the client. “The level of difficulty can snowball when you are working on older equipment,” said Howley. “We worked with Microsoft to get the right operating system and licenses. Ali made a considerable effort to get the server’s operating system installed and FactoryTalk reinstalled; it was quite a task.”

Badruddin added, “The historian server needed to be rebuilt entirely because it wasn’t functioning. We worked through the hardware and software issues to recreate a functioning historian server. The client now has visibility into their operations and data, which allows them to optimize their process. It was critical for us to accomplish that.”

During this time, the plant was running off electrical generators. The client’s engineers were doing testing related to the power on the site and the power kept going down, requiring endurance from the Hargrove team. “Imagine trying to get an outdated site up and running. You spend your day working and making progress and then in the afternoon the power cuts off,” said Badruddin. “You come back the next morning and you are back to square one; your progress was lost. We fought a lot of that – it was a regular occurrence. This required us to put preventative measures in place to ensure our client’s testing could proceed uninterrupted.”

At project close, the Hargrove team had installed redundant HMI and historian servers and obtained all the necessary licenses so both new and old systems functioned properly and worked together. In addition, the Hargrove team helped to troubleshoot, fix, or replace existing controls equipment. “Thanks to the team’s flexibility and persistence, the client was able to move forward with bringing the power plant back online,” said Badruddin. “It was a big challenge that our team was able to complete successfully.”

The Hargrove team remained available for remote assistance in the days following completion to ensure the client’s sustained success.

For more information on Hargrove Controls & Automation, visit our website or contact us.