This Messy Control Panel Is One Fault Away From a Full Line Stop [Real Plant Panel Review]

How's it going everyone? Vlad here, and welcome back to the next episode in which we're going to tackle some of the industry's most upvoted questions as well as comments.

A panel that has been worked on for years

The first one comes from the username El Compastel Solvino, and he's showing us today's office where he's working. A panel, what looks like underneath a machine. Says, "Hopefully we'll get to do a refurbish soon." You can probably tell by the image it is a massive mess. So let's dig in really quick and see if we can figure out what's going on here.

On the top left, we've got some disconnect blocks, which is fine, circuit breakers. It seems that there's a lack of labels in general. There's some labels that I still see on the cabling, but of course they're not extremely intuitive. It is very difficult to tell what is actually going on.

It looks like we have a Delta VFD, which again gets probably started, and I would assume still works. It seems that there's some relays that were added post production because we can tell that the labels are fairly different from what we see on the rest of the equipment. So we have some Finder relays that are stacked on the right hand side. We have some floating relays. Again, for what reason, it is very difficult to tell.

We do have a PLC that's underneath this entire mess. It is a Panasonic PLC. It is not something that I'm overly familiar with. I'm assuming it is the equivalent of a MicroLogix, so it looks very similar form factor wise. I can't imagine it accomplishes anything overly complex, more so than a couple of discrete digital as well as analog type of inputs and outputs.

That being said, we obviously have a huge mess on our hands. We see that the wiring has been terminated maybe at the very beginning well and was probably inside of the wire trays that we see in the back. But at this point, none of it is contained. We have a random Rockwell Automation power supply. Again, not sure why they paid the extra price for a lot of very different hardware. We have more floating relays. We have what seems to be a safety relay just based on the fact that it is red that's lying on the panel floor. We've got a switch. Someone pointed out this is an unmanaged Stratix 2000 series switch. Behind me, as many of you know, I have a variety of different managed switches.

So the first impression, this is not a new type of an environment. I've seen a lot of these before. This is something very common in many industries. Typically for those of you who are not from the control system side, if you leave panels long enough and you put pressure on your techs or your engineers, whoever is doing the changes or the troubleshooting, then chances are if it's not done properly, if there's just a lack of time perhaps for the work to take place, they will do their best to execute the job, but they will not have the time necessary to make sure that the panel is in a good state or condition.

The second item, of course, is if there's a lot of problems with a specific piece of equipment, the chances of you needing to go and remove those covers that cover the wire trays and troubleshoot and pull on the wires and see the labels that haven't been properly placed is going to be significantly higher.

So this is not a unique situation. I'm assuming that the switch that is floating on the side of the panel was added after the fact, and this could be attributed to the mess of the panel being way too small for expansion. A lot of times when you design electrical panels or cabinets, you want to provision some additional space on DIN rails and just the back faceplate so that you could add and expand should you need to. A lot of times the equipment comes in as is from the machine builder or the OEM, and then the plant wants to add a couple of components, maybe a handshake from the upstream or the downstream equipment. Maybe you want to add the asset onto the network for either remote troubleshooting purposes or data extracting purposes, and you end up having to stick the switch on the side. Would there have been better locations? Maybe. But again, whoever was delegated that task decided to go down this route.

So what would be my general recommendation? At this point, it is just very difficult to maintain this asset. If something were to bring this asset down, we even see a naked cable that is not landed into anything that's sticking out. But if anything was to bring down this asset, it usually takes a long time to troubleshoot. If you bring in somebody new that doesn't know this machine, that doesn't have at least a decade of experience troubleshooting this specific piece of equipment, it's going to take a while for them to figure out what's going on.

I can only assume that the prints, the electrical schematics, are not up to date, so the person working on this machine in the time of downtime is going to struggle. And of course, depending on the cost, it might be worth it just scrapping the entire panel, rebuilding it from scratch, paying someone to create the up to date schematic as built, and then replacing this entire mess with something that makes sense.

There's also some obsolete components. I'm not familiar with the Delta VFDs as well as I am with some other brands. I would assume that this VFD is probably no longer sold by the distributor, so probably want to replace that. The same goes for the PLC. Again, you can look up the part number and figure out if it's obsolete or not. Of course, the switch that is just hanging on the side, it's not recommended. An unmanaged switch is relatively robust, but of course it should be properly mounted. I'm assuming there's quite a bit of heat coming out of this panel once it is closed with everything that is inside of it.

So long story short, I would generally recommend investing in retrofitting this panel. Of course, it depends on the budget. It depends on how the machine is being utilized by the business, and someone needs to do the calculation if it makes sense or not in terms of investment. Maybe this is going to be a discontinued machine that's no longer running in a couple of years, so it could continue running the way it has been, probably for the next couple of years.

A laptop, a trolley, and an SLC 500

The second question comes from the username Cora60, and once again the user writes "today's office" with a picture of a laptop sitting on what usually I would describe as either a trolley or one of those metal desks in a food and beverage facility. So we immediately notice a couple of components from this image, and the joke in our industry is the fact that a lot of times, and in many industries, your desk is whatever you can find at the plant at the time.

Of course he is plugged in via an Ethernet cable. We see the cable hanging over the panel. We can tell that this is an SLC rack. It could be a 5/01, 5/02, 5/03, or 5/05, depending on the setup. If he's going to the PLC directly, it is an SLC 5/05, which is the best model arguably of the SLC. Of course there could be a conversion from RS232 or RS485 onto Ethernet. We don't see the actual PLC. We only see the input and output cards.

Once again, it is a messy panel, but it is not to the extreme that I have seen, so there shouldn't be any issues there. He's able to log in. We can see the green status, so he is connected to the actual PLC. He is making some changes. Again, exactly what, it is difficult to tell. There is a scaling instruction from what I can tell. Input min, input max, scaled min, and scaled max. So that's RSLogix 500 if you haven't played around with the software.

We see some food and beverage or chemical plant type of equipment. This is piping that can be manually redirected. You usually have some prox sensors that tell you the position of these pipes, and then you can run equipment. Of course we don't see any tanks or anything else, so it's hard to say what the process is exactly. Surprised that he's got some water next to him. Maybe it is allowed at that plant. Usually in food and beverage this is not something that you will typically get in with. But once again, if it's commissioning, if it's troubleshooting, exceptions can of course be made.

If you're not in our industry, this is fairly common. You walk up to the control panel. There's going to be no desk for you, so the next best thing is to grab some kind of a table that's used maybe for quality samples. You pull up a chair, and that's pretty much going to be your programming workstation. So great post.

The S7-1500 that will not see the ET 200SP

The next post and comment is of a very interesting nature, and this is something that I've reflected on many times over the years. It comes from the username Goodzal Doktor. And the user says, "Need help with S7-1500," and we have a couple of images. So we have the PLC. I believe this is the smallest version, because they come in this variation and then a much larger one. Then you have the ET 200SP I/O module here on the bottom. And we have a couple of screenshots of TIA Portal. So it says "Module exists," "Error," "Error in lower level component," "Data transfer not possible," "Connection interrupted."

The description says, "So I have this S7-1500 that's almost fifteen years old. For a new project, I bought the new I/O modules. I had problems connecting to the PLC, so I changed the firmware versions, but still can't communicate with ET 200SP module. It's been two days and I'm starting to lose my mind over this."

So the important point here is not the answer to the question. The important learning lesson is if you're coming in as an engineer on a project, and you're working on figuring this out, and you're seeking out help, whether from other peers and colleagues, or if you're going to get in touch with a distributor, maybe the OEM or someone that offers support like a systems integrator, you need to provide a lot more detail than what he has in this specific post.

Of course we can extract all the information we can from the images, but in this case the only thing that's made available to us is we can probably figure out what version of the PLC he is using. We have no idea about the firmware. We don't know which firmware he's using for the remote I/O. If it's an older PLC, in many instances, and I'm not incredibly familiar with where the limit is, you cannot always flash it to the latest revision, so that will be extremely important for troubleshooting purposes.

In this case, the error is in lower level component. Again, I would try removing the I/O. If everything works well, we can eliminate that factor. Of course, then it is the I/O. Then we need to work backwards. Is the I/O compatible with this specific PLC? Do we need to flash the firmware? He says "changed the firmware versions." Is that of the PLC? Is that of the I/O? Is that of the specific I/O modules as opposed to the base PROFINET module in this case? And of course it says the port monitoring was configured. However, there's no physical connection to the port, no cable plugged in. It seems that the cable is plugged in. I can't imagine that the cable is the problem, although I've seen that situation before.

So nothing can be ruled out based on the information given. I would need a lot more details to be able to help out this individual to help troubleshoot what the problem is. And I would assume that if he reaches out to Siemens, either directly or to an SI or a distributor, they will be asking him the same exact information.

The other item to try, of course, is process of elimination. We have a PLC, we have an I/O block. You remove everything, you just connect to the PLC, everything works well. Then you only add the Ethernet component. What I mean by that is you have PROFINET on this module, so what you can do is remove all other I/O. If you remove all other I/O except the Ethernet coupler, you should be able to remote into it via PROFINET and make sure that you can configure and establish the program that way before adding anything else. I know that there's going to be different revisions of the hardware and thus of the software and the firmware, which possibly could make the modules incompatible with the PLC.

A first program

Hi, my name is Vladimir Romanov. I am the founder of Joltek as well as SolisPLC. With a background in electrical engineering and an MBA, and over a decade of experience leading projects in manufacturing and industrial automation, I help engineers, managers, and manufacturers make smarter technical and business decisions, modernize their operations, and build stronger careers. If you're serious about manufacturing, automation, and staying ahead in the industry, subscribe and join the community.

The next post, and I would say more of a comment than anything else, is coming to us from the username Autumnal Coffee. And this user says, "My first official program, it scoots poop around. Pray I don't kill anyone under a big pile of dookie." Hopefully he's joking, but it seems that he is serious. But I would hope that it doesn't harm anyone.

So here we have a screenshot of two screens. And interestingly enough, it looks like this was programmed in RSLogix 500, so this indicates probably an older type of a system that is currently running. I can't tell if this is some kind of a different software, if this was a migration, or it is actually running inside of RSLogix 500. This almost looks like CODESYS, but I can't necessarily tell based on the screenshot.

So we have RSLogix 500. We have a few other softwares on his laptop. We can see that there is some pumps, so pump one, we have pump two, we have the enables. It seems to be right, of course. Again, it's very difficult to say from just looking at one program. He's got the comments in there, which means that he has the latest and greatest software. And we should be curious on what this actually does for that plant.

I think it's very stressful and also very rewarding to have written your first program. In many instances on many of my projects, especially early on, you never have the chance to design the architecture of the software. You're usually making small modifications, or you're maybe reapplying some of the software from an existing control system onto a newer one. So it is very exciting when you finally get a chance in our industry to write something from scratch where you get to decide what the architecture looks like, what the logic looks like. And of course, once you are done and commissioned the machine, which seems like he hasn't yet, he only wrote the software, it is extremely rewarding.

What I have also learned, and this is a side note for maybe this person that has finalized the software, is that once you get into the field and run the actual software on the PLC and control system, you will have to make quite a few adjustments. As you grow in your career and grow in your experience, of course you can have the foresight of certain things changing. But if it is your first program, I think you're excited and you think you're going to download and everything is going to run as expected, and that is extremely unlikely. I'd be very surprised that this person doesn't need to make a lot of changes in the field once the program is actually on the PLC and they realize either the sensors are slightly different, or the I/O is coming in unexpectedly, or there's simply different routines that aren't executing as expected. In any case, it would be great to have an update from this specific person.

A Festo trainer and a Siemens LOGO

We've got another exciting first project from the username grouchystart9660, and we do have a short video. There's a laptop with some, what looks like ladder logic. It looks like there's some I/O laid out. We see this is, if I'm not mistaken, an IFM AS-i bus extender. I don't see the flat ribbon cable coming out, but it could be that. Maybe it's just a power supply. I could be wrong.

This is a skid that you would normally see in community colleges or universities. There's a tank that's being filled with water. I'm assuming there's some pumps and some valves, and he's controlling. There's probably a PID loop that is going to be controlling. And what's the PLC? It's a Siemens LOGO, I'm assuming. That's the small scale PLC on the Siemens side. It is not programmed in TIA Portal, which is why I was a little bit, not necessarily confused, but unfamiliar with the interface.

It is always exciting, like I've mentioned in the previous comment, to see our first project. This is for education purposes. I don't imagine that this is a real environment. We see a lot of banana plugs where you can relocate the I/O, so we don't have a full blown industrial cabinet. We have the test rig, which again, it's difficult to decipher. I'm sure if he is able to share the information, there's probably diagrams and schematics for the water flow as well as what needs to be or can be controlled. But it looks like there's two tanks. There's going to be some sensors on both the top as well as the bottom. So there's probably a sensor for tank full and a sensor for tank empty, and probably an ultrasonic sensor for the level of the tank.

We can also tell that a lot of the equipment is from Festo. There's a Festo label on both of those tanks. I actually had the opportunity to speak with the folks from Festo Didactic not too long ago at Automate. So they provide a lot of these training kits for universities and colleges where students get to play with different I/O. Of course Festo is very known for pneumatics, but they also have many other components. They have everything from valves to valve banks. They have motors and servo drives that can of course actuate pumps. So they do provide quite a complete educational package in these specific instances.

But yeah, once again, I think it's really cool to see some real projects in our industry, because of course of the secrecy of the actual industrial automation space, it is not always possible to capture video footage of the actual company that is doing XYZ process. There's a lot of different regulations, usually from the companies themselves, that don't want to share what's going on behind closed doors.

IT is taking over OT

The last question is very interesting and incredibly topical for the type of work I do under the Joltek consultancy and systems integration, and this comes from the username LoveMyMittens828, and they say:

"IT take over, I think. So I've been a controls engineer for the same company for almost 15 years. When I took over, there was very few network devices. I upgrade when I can to the latest and greatest. Yes, the system was once air gapped, and about six or seven years ago we put in an OPC UA server to push information out for a few services. So my intranet Wi-Fi got expanded to being internet connected all the time. I put in a firewall and kept it updated. I think I'm doing everything rightish, but my background is not IT related in any means. So now our parent company has moved in since our IT person retired, and they are pushing hard and fast all over the place, IT side and OT side, RMM and monitoring, stuff I really don't understand, to be honest. So all of you have been through this, I'm curious as to what I should expect, as in what symptoms that IT is breaking things on the OT side, from the littlest oddity to full shutdown mode. Let me know what your experience has been."

Of course, this is very topical. I have been on both the OT side as well as the IT side. I came through and my experience has predominantly been OT. That being said, I understand the technologies that IT is trying to bring, but I also understand the cautionary tales that could happen when you press too hard or press in the wrong direction.

Generally speaking, the topic of IT and OT convergence is not a new concept. This has been a conversation for as long as I have been in the industry, which has been over a decade. The idea that IT is trying to bring some of their tools, but also some of their benefits, to the plant floor. Now of course the same can be said about the hardware that's coming in today. A lot of the devices are going to be Ethernet compatible, whether that's the EtherNet/IP side for Rockwell Automation or PROFINET for example for Siemens, or any other brands for that matter. But almost everything communicates over an Ethernet based protocol, which means that it's easy to assign an IP address, and easy, I put that a bit in air quotes. You can always assign an IP address. It becomes a lot more complex to manage once you start to have segmentation, once you start identifying different VLANs that need to be implemented, once you need to be cyber secure, as you mentioned there's firewalls that have been put in place, once you need to add certain access control lists to make sure that no bad actors are able to connect to your plant floor.

So where does this generally lead? It needs to be taken on a case by case basis. Is the parent company trying to implement a new SCADA system? Is the parent company trying to extract some of the plant floor data, perhaps because they want to leverage AI, quote unquote? Are they installing quality control software that can once again automate some of the processes? I think it's very difficult to answer where this is going to go.

What I can say is in organizations where IT took over the OT side, a lot of problems started happening in the years to come. And the reason, and again this is just my experience, is of course the requirements, but also the KPIs on the IT side are very different than the KPIs and the requirements for the OT side.

My general perspective is that everything in the manufacturing business at the top level relies on the revenue generated from making the product that comes out the door. If it's palletized, then it gets shipped to the distribution warehouse. If it gets sold directly to the customer, you need to make sure that the lines and production does not stop. And so if you start introducing more and more technologies that can help increase production, that is usually a plus. But if as a result of introducing said technology you now create complexity for your team where there's constant shutdowns, where there's different exposure to cybercrime or ransomware, that starts to create a problem.

So as this gentleman mentioned, in the past this plant was air gapped. If you don't know what air gapped means, it was completely disconnected from the internet. It means that there's no outside influence that could shut down the line. Once you connect to the internet, there is some exposure to the outside forces, and of course you create the possibility of the cyberattacks.

Now, once you start adding other software, so maybe you want to create a VPN where you can remote into some of the assets, there's another layer of complexity. Now you need to manage who's able to communicate with the plant. If you give access to a contractor, are they being secure? So now you have the possibility and also the necessity to manage those credentials.

The next step of course is if you start installing, for example, SCADA software. I've been part of multiple projects where either FactoryTalk View SE or Optix or Ignition are brought in by the IT side, and they don't necessarily understand the requirements, so they will sometimes patch the server upon which one of those softwares is running. And of course production cannot run if the servers that are pushing down the SCADA or the HMI or the control system is not operational or functional. And that is simply because they prioritize cyber security over running the actual manufacturing line.

So what does this mean for someone that's on the OT side?

Number one, you should be having a lot more conversations with the IT team, in this case I'm assuming it's from the corporate side, not at the plant side, and asking them questions as to what they are doing and why are they running these initiatives and what kind of an impact is that going to have on the production. If they don't have the right answers, you need to be marking those as flags and you need to be having conversations usually with the IT managers or the IT director or even VP of IT to understand and to have them understand what your needs are on the OT side. If you've been there for fifteen years, you should be very clearly explaining to them what are some of the risks.

Number two, if you see initiatives that touch OT directly, if you're deploying for example a software that's only collecting maybe data, sure, you need to open up some ports and maybe there's going to be problems on the network, but hopefully it does not affect the actual control system. But if they're going to deploy a SCADA solution that actually has impact on running lines, you need to be present in those conversations. You need to be vetting the people that they bring in to do that work, and you need to be very cautious in how it is being rolled out across the plant. Otherwise you will be penalized for them not installing it appropriately and of course shutting down production.

Number three, I would be very clear as to whom they are bringing in to do that work. I've seen many groups that will be sold by high level consulting groups that I'm not going to name right now, but they bring in fairly inexperienced people that do understand software, do understand IT, but have absolutely no room to speak when it comes to the OT or plant requirements. They don't understand PLCs. They don't understand HMIs. They don't even understand SCADA, and they're trying to make changes to systems that are fairly different than what you would normally see in the IT side. And although they have the good intentions, it's simply the lack of experience that will get them into trouble, and then the entire facility and company will pay for those mistakes.

So number three, make sure that you vet the groups that are going to be working, especially on anything that touches the OT side.

And that's going to be all for us for today. Those are all the questions. If you have any follow ups, if you disagree with some of my comments, don't hesitate to leave me a comment, and I hope to see you next time.