Hi everyone, it’s been a little while since I’ve posted to the site. It’s amazing how busy life can get at times. With a new year upon us I thought, “it’s about time I dusted off the cob webs and post something…”.
So I found myself contemplating what should I write about? I’ve read about so many automation success stories here on the site, the forums, and over on LinkedIn and other social media outlets, that I thought…“why not an automation story that failed!”.
I know, it seems a little pessimistic to start off a new year, let alone a new decade, with a story about project failure, but it’s amazing how you never hear people talking about it.
I’m hoping none of you here have to experience, when an automation project goes wrong, however, the reality is, when you’re developing one-off, custom automation solutions there’s always a risk that it may not meet customer expectations, or flat out fail. In this article I am going to share one of my own personal experiences of a project failure that I was directly involved.
What is project failure and why do projects fail?
Before I dive into my own personal story of project failure, specifically an automation project that failed, I thought I’d do a little digging to see what factors go into the ultimate demise of a project. The resource I found that does a reasonably good job explaining why projects fail, or more appropriately project failure, is on the Project Management Institute’s website or pmi.org. It’s a great article, and I’m sure I won’t do it a lot of justice here, so I’ll refer you to check it out after your done here if you want additional insight.
Basically, PMI suggests that projects most commonly fail because there is a lack of efforts or attention being applied to these seven project performance factors (source: PMI.org).
- Focus on business value, not technical detail. This involves establishing a clear link between the project and the organizations key strategic practices. The project plan needs to cover the planned delivery, the business change required and the means of benefits realization.
- Establish clear accountability for measured results. There must be clear view of the inter-dependencies between the projects, the benefits, and the criteria against which success will be judged. It is necessary to establish a reasonably stable requirement baseline before any other work goes forward. Requirements may still continue to creep. In virtually all projects there will be some degree of “learning what the requirements really are” while building the project product.
- Have consistent processes for managing unambiguous checkpoints. Successful large projects typically have software measurement programs for capturing productivity and quality historical data that can be used to compare it against similar projects in order to judge the validity of schedules, costs, quality, and other project related factors. The lack of effective quality centered mechanisms can be a major contributor to both cost and schedule overruns.
- Have a consistent methodology for planning and executing projects. There should be a detailed plan developed before any release date of a project is announced. Inadequate planning is one of the major reasons why projects spin out of control.
- Include the customer at the beginning of the project and continually involve the customer as things change so that the required adjustments can be made together. It has been observed that successful projects occur when end users (customers) and the project members work as teams, although this is not always possible. Projects are less likely to fail if there are informed customers giving meaningful input during every phase of requirements elicitation, product description and implementation. The customer needs to be asking, “how are the project results used over time and what do I get out of the results?
- Manage and motivate people so that project efforts will experience a zone of optimal performance throughout its life. This involves managing and retaining the most highly skilled and productive people. Knowledge is money. A project team made up of higher paid people with the right specialized skills is worth more per dollar than a group of lower cost people who need weeks or months of training before they can start to be productive.
- Provide the project team members the tools and techniques the need to produce consistently successful projects. The project team must be skilled and experienced with clear defined roles and responsibilities. If not, there must be access to expertise which can benefit those fulfilling the requisite roles.
In my own personal experience with project failure, that I will recount for you shortly, I would say that all seven of these performance factors had an impact to some varying degree.
A little history…
Of course, for confidentiality reasons I will not be using the real names of the company I worked for at the time, or the customer the project was designed. And while it may prove challenging to encapsulate this into words, here it goes…
The company I worked for, ABC Engineering Corporation, was a Tier 1 automation supplier to the big 3 automotive companies of the time, namely, Ford, GM and Chrysler. The company has since gone out of business (this is one example of why), however, I spent nearly 5 years travelling to customer facilities to oversee the installation and perform commissioning and validation of all automation and equipment shipped. I was very young in my engineering career, just out of school.
When I was not at customer sites, my role included all aspects of panel design and programming. While on-site, it was my job to ensure the equipment passed all the functional requirements as defined by the customer and that all quality assurance and safety metrics were achieved. This involved detailed troubleshooting to work through the risks that were not identified during design phases and run-off phases of the project.
The one project failure that stands out for me, was a robotic automation project that I was tasked with installing at one of the big 3 automotive companies, mentioned above, in Atlanta Georgia (plant is also now closed). The reason this particular project stands out for me is because, well quite frankly, it never worked, nor was it ever going to work!
The project, which took approximately one month to do the initial install, saw me flying back and forth to Atlanta every weekend for about 5 months, while the line was not operational to continue the commissioning to try to get it to work.
Some project details…
To outline this project, we utilized two Kawasaki 6-axis articulated robots on either side of the vehicle assembly line with a main “cell level” PLC controller. The task of the robots was to install both the front driver side and passenger side seats in the vehicle so they could be fastened at the next operation by a human operator. As it was, before this installation these seats were being placed in the vehicle by a line operator on either side, which of course was leading to ergonomic repetitive strain type injuries.
The cell had two lines, one carrying the vehicles as mentioned, and another line beside it carrying the seats to be installed in that vehicle, they were synchronized with the seat conveyor slightly ahead of the vehicle they were to be placed in (approximately one car length).
We had a robot on either side of the line mounted to a tracking vehicle (7th axis) that would clamp on to the vehicles lines “clam shell” and free wheel along-side it as it indexed through the cell (after installing we would unclamp and drive the robot vehicle back – VFD controlled motor driven back). I’ve put together a rough layout of the cell above to help visualize.
Basic sequence of operation…
The sequence of operation was as follows:
- The driver/passenger seats would index into the cell slightly ahead of the vehicle they were to be placed.
- A pneumatic (PLC controlled) Fixture Station would positively locate the seats and halt the seat-line via a hard-wired interlock.
- The passenger-side robot would unload the driver seat and place it onto the “Overhead Driver Seat Fixture”.
- Once cleared of the overhead fixture (via software interlocks), the Driver-side Robot would unload its seat from the overhead fixture and at the same time the Passenger-side robot would unload the passenger seat from the seat fixture station.
- Both robots would now be holding their respective seats and waiting in the “pounce” position to clamp up to the vehicle and track. Therefore, it was imperative that all of this material handling took place before the vehicle exited the permitted “clamping window” (this was not accounted for in the original design!).
- Once confirmed clamped and tracking, the robots would proceed to install the seats into the vehicle and the seat fixture station would be released and the seat conveyor turned back on allowing the next pallet of seats to enter the cell.
- After successful installation, the robot vehicles would be motor driven back to the home switch and the process would repeat.
As noted above, there was no contingency built into the system to abort the process if the clamping window was missed other than to stop the vehicle line (via another hard-wired interlock), however, this was not an acceptable solution to the customer for obvious reasons. I still can hear the line supervisors shouting “WHY IS THE LINE STOPPED…THIS IS $10,000 PER MINUTE”. I still have nightmares of line stoppages!
The exact timing escapes me now, however, I do recall that by the 8th or 9th vehicle cycle we would essentially have no choice but to stop the line. With every cycle, there was a small accumulation of cycle time error, error that was not accounted for – it was time to go back to the drawing board!
What they came up with for a fix…
What was eventually designed, built and shipped down to me to remedy this problem was a set of pneumatic turntables that we would place at the ends of either side of the line so in the event we missed the “clamping window” we would drive the robot vehicles with their respective seats down to the end of the track as quickly as possible using the variable speed drives, initially designed to only drive us back to the home position, and place them on the turntables.
Of course, my job was to integrate the program logic necessary to achieve this and all of the necessary robot kinematics, all coupled with the production pressures to do so.
The operators at the exit point of the cell (outside of the perimeter guarding at the turntables), there to secure the seat bolts and make the required seat electrical connections (power seats only), would be able to operate a push-button that would rotate the seats out of the cell. Then by means of a manual pneumatic pick and place system be able to unload the seats from the turntables and manual place them in the vehicle…a band-aid to the problem at best, however, it would save us from stopping the main vehicle line.
The remaining months (week-end travels) I spent optimizing robot positions, program logic and recovery to try to get as much cycle time out of the process as possible, however, the efforts were futile as the required cycle-time was simply unattainable.
The lead design engineers on the project failed to do a proper cycle time analysis during the design phase at the onset of this project. As a result, the project ultimately was shut-down and removed from the plant after tens, if not hundreds of thousands of dollars wasted. Even thought the project was a complete failure, I felt this was an important experience for me in my young engineering career.
Reflecting back now on this project and performing my own post-mortem of sorts, it has instilled in me the importance of gathering and understanding the requirements of any project I embark on, and defining a clear scope BEFORE jumping into the design and build phases.
Project Failure – The Recap
Well I hope you enjoyed reading my little automation project failure story, it seems there are so many successful automation stories out there, I thought it might be interesting to rehash one that wasn’t so successful. While this project was a failure, I still learned a lot from the experience. If you have any automation project failure stories of your own I’m starting a thread in the forums over at the Water Cooler. Please share your own stories automation project failure stories over there!