Author Archive

I actually enjoyed watching the Super Bowl half time show last night – but not for the aging hystrionics of Roger Daltry, Pete Townsend and a bunch of stand-in musicians now calling themselves The Who.  I guess The Who must be considered a “safe” choice as they fit the bill for that sort of large stadium spectacle.  CBS didn’t help matters by showing stock footage of a significantly younger Who during the promotional commercials only emphasizing the contrast when the actual band took the stage. I noticed they didn’t play “My Generation” which now has sort of an ironic overtone doesn’t it?  I wonder if they regret the lyrics “I hope I die before I get old”.  Oh well.  Never the less, it was sort of painful to watch. 

What wasn’t hard for me to watch was the spectacular stage set up.  I did a bit of google sleuthing and wasn’t able to find out who did the conceptual design, engineering or production on it but I thought it was pretty darn cool!  If you didn’t see it – the stage was a circular array of multi-colored lights that had been programmed to for a variety of conventional lighting and video effects.  The fact that it was circular was particularly interesting and I suspect presented some software and hardware challenges as we are most used to thinking of our output displays in rectangular format.  The electronic aspects of the stage were integrated with the Laser Light show and Pyrotechnics to form a pretty compelling visual image.  All-in-all The Who got kind of lost in the production which was just fine by me.

In a previous post I promised to provide my guesses on what factors may have contributed to the recalls Toyota is facing right now.  Here are a few: 

a)            Increasing interwoven system complexity.  What were once purely mechanical system with very intuitive failure modes (hey… that cable broke) have evolved to electro-mechanical systems and now electro-mechanical systems with integrated micro-processor control and firmware.  Failure modes that previously lay entirely within a single discipline now splash messily across functional areas.  Electrical and software problems are particularly bedeviling and notoriously hard to faithfully observe, detect and eradicate. 

b)            Increased reliance on outside suppliers for subsystem engineering.  Once upon a time a company’s engineers would design and meticulously test each and every aspect of a product’s function.  No more.  More often suppliers are brought in to “black box” or “gray box” subsystem leaving the company’s engineers to often function as “system integrators” – a sort of general contractor.  At its best this system can leave the details to subsystem experts.  At its worst, it is a recipe for omissions, mistakes, confusion, finger pointing. 

c)            Platform Engineering.  Toyota is the mastery of assembling a huge variety of unique cars from a building block of components and systems that are fundamentally identical.   At its best, this practice allows for reduced costs, increased development speed and adoption of best practices.  At its worse, systems end up over-designed for some applications or under designed for others.  Change can be slow and improvements forgone because of the interwoven complexity of validating revisions on every instance where a system is used.  Furthermore, as this situation has shown, if design errors do crop up in fundamental components and systems used through the platform the effects and scope can be far-reaching.   I was sharing this theory with a former classmate and he commented on the similarity between platform engineering and the risks associated with a lack of genetic diversity…

‘It is not a trivial thing to lose your genetic variation,” said Dr. Stephen O’Brien, head of the research team. ”Genetic variation exists so ecological pressures can be adapted to.’

http://www.nytimes.com/1985/09/17/science/loss-of-gene-diversity-is-threat-to-cheetahs.html

d)            Finally, I firmly believe a potential culprit is Toyota’s massive recent success combined with relentless plans for expansion.  It seems entirely probable that Toyota tremendous expansion and a well publicized plan for industry domination lead to a degree of rapid growth, internal pressure, hubris, disorganization and errors.

Looks like Toyota is headed for a software fix for the Prius brakes.  Weirdly enough, I feel like I have a personal intersection with Toyota’s alleged ABS brake difficulty.  I own a 1997 Toyota 4-runner.  Several times in the life of that car I have experience a complete loss of braking – always when braking while riding over significant bumps.  The first time it happened was weird… but not overly terrifying.  I quickly assumed the failure was being caused by “resonance” or the unhappy combination of the ABS choosing to brake when a wheel was virtually mid-air and then releasing just as the wheel hit the ground as the wheels bounced over the bumps.  If this explanation is right it could and perhaps should have been solved by tuning the ABS braking frequency to ensure that the natural frequency of the suspension could never match the ABS braking rate.  

Perhaps this explanation is naïve or just plain wrong.  Regardless I am left to wonder if my 1997 car suffers from the same flaws as the 2010 Toyotas.  Is it possible that these cars share the same basic control strategies and algorithms even after all these years???  If so this highlights a weakness of platform engineering and presenting a pretty damning portrait of their ability to spot and eradicate problems.  I’ve never experienced this phenomena on any other car I’ve ever driven.  Alternately perhaps it’s the same engineering team making the same mistakes year after year.  It is ironic that as an engineer I simply wrote this off to a freak combination of speed, bumps and possibly a sloppy -  but not criminal – job of engineering.  The brakes on the 4-runner have always been pretty marginal and even need to be pumped a few times in very cold weather before they’ll work.  Like many engineers I think I’m perhaps too tolerant of design related issues.

Ah for the good old days when “sudden acceleration syndrome” meant a 19 year old kid in his dad’s 1967 Pontiac GTO with a 455 V8, 4-speed and a Saturday night of mischief ahead. Alas, times are different and our national attention once again turns to another stumbling automaker – this time the seemingly impervious juggernaut Toyota withers under the stunning specter of frightening allegations, an unprecedented recall and PR nightmare.

I wanted to blog for a bit about the reactions of an engineer to this situation but full disclosure – I’m an ex-automotive engineer having spent 12 formative years with General Motors’ Saturn which hardly makes me dispassionate or unbiased. And while you might think that leaves me gloating with a sense of schadenfreude in truth I’m far more reserved – as seems to be all my friends who still work in the auto industry.

While there is plenty of mud to be appropriately hurled at Toyota many of us engineers are bracing ourselves for another round of public castigation, misinformation, naiveté and virtually comical pronouncements. Not to necessarily defend Toyota but some cover for their engineers… an automobile is by far and away the most sophisticated “consumer product” in our society. With no disrespect intended to iphones and X-boxes nothing even comes close to the degree of technical complexity, system integration and punishing operating environment seen in today’s modern motor vehicles. The simple fact that my 74 year old mother can slide confidently behind 2000 lbs of rolling sheet metal propelled by 185 horsepower and pull out in a freezing  sleet is alternately a modern miracle and a testimony to the combined skill and collaborative ingenuity of over 100 years of incremental engineering prowess. Well that or terrifying.

But as the degree of technical complexity and refinement have steadily ratcheted up, so have our expectations and the degree of blind trust we place on the engineers behind our cars. So while the general public may be perplexed by – or indignant over Toyota’s recent problems, most engineers I know are a bit more nuanced in their reactions. Perhaps a combination of technical curiosity and there-but-for-the-grace-of-God-go-I.

While the real story is still emerging, my instincts tell me Toyota has a real problem here. I believe there are significant and substantive safety defects in these systems and cars. While hindsight will quite possibly show it to be inadequate their response has been predictable, understandable and frankly perhaps the best that could be expected. We’ll see.  Why did this happen? It’s just supposition but I feel this is situation a perfect storm of several trends sweeping the modern design world combined with some factors unique to Toyota. In my next post I’ll break down my thoughts on what happened – and perhaps a bit of why…