I feel the need…the need for speed…

Besides being a classic quote from the movie, Top Gun, it’s also a mantra of most software development organizations today.  We know that time to market is critical and development cycles are shrinking, so development teams need to make sure that the processes and technology they adopt will not slow them down. The same goes for security. With developers under pressure to deliver more functionality at a rapid pace, security is easily overlooked. It’s handled by ‘that other team’ anyway, right?  Wrong.  Security is everyone’s responsibility in the development lifecycle, including developers. But the question is not why security should be addressed in development, but how it can be addressed efficiently in development.

According to renowned security expert Bruce Schneier, “Assurance is less about developing new security techniques than about using the ones we have.” This statement rings true for development testing technologies such as static analysis. Static analysis is not just about finding quality issues in the code, but also finds common security issues as well. Think buffer overflow. It also integrates with the development and defect tracking workflow, so developers, management, and security teams can quickly identify, assign, and address common security vulnerabilities in code without having to overhaul their process—and without slowing them down.

We are excited to join forces with our friends at Wind River to bring development testing for security to the embedded software development teams building their products on the Wind River platform.  Development teams choose Wind River because the platform is ready to use. For example, it’s already pre-built with industry-leading protocols and security certifications, it optimizes application performance, and works with a large partner ecosystem, including Coverity. You can save precious development time with Wind River, and now with Coverity you can do the same by building security testing into the development process as you are writing code–when issues are easiest and fastest to fix–without leaving your workflow. Security doesn’t have to be painful for development, or slow you down.

We are offering a free evaluation of Coverity Static Analysis, pre-configured for Wind River VxWorks and Wind River Linux and integrated with the Wind River Workbench IDE.  You can sign up here.

Announcing the Coverity Security Research Laboratory

Today we’re excited to formally announce the formation of the Coverity Security Research Laboratory. Romain Gaucher and Chris Valasek are top-notch security researchers who bring a wealth of experience and hard-won security knowledge to tackle the complex challenges of secure development.

The lab will have two primary goals: to perform cutting edge security research and to improve the security capabilities of our products.  It’s commonly understood that developers have a lot to learn about security, and we intend to help with that.  But we also believe that helping the security community better understand development will lead to improved communication, tools, and practices and, ultimately, more secure software.

We believe that Coverity is in a unique position to shape the relationship between security and development. Our customers have traditionally been developers and development organizations, and we’ve learned what it takes to get developers to actually fix problems in software early in the development cycle.  We’ve worked with over 1,100 customers, collaborated with over 250 open source projects, analyzed billions of lines of code and fixed millions of defects.  In our view, this is the kind of scale required to “make a dent” in security. So while we will research individual vulnerabilities and dive deep into specific exploits, an important goal will be to build security knowledge into our products. That way, hard-won lessons from studying how one line of code goes wrong can be applied to billions of others.

The Coverity Security Research Laboratory is a sign of our commitment to the security market. We believe that this is a critical problem for the entire industry, and as such, it falls squarely within our mission to change the way software is developed.

PS. We’re hiring. If you’ve got experience with code review and pen testing, contact us to learn more.

Accelerating Innovation in Automobile Infotainment with Development Testing

This week Coverity announced how one of its customers, Neopost, is advancing software quality and product innovation with Coverity Development Testing. Neopost, a European leader for mailing systems equipment, is constantly driving technological advancements in order to consolidate its market leadership position. Since software is fast becoming the key differentiator for success in this sector, the need to drive software innovation (and therefore quality) even faster, across multiple product lines, is paramount.

None of this is new to automobile manufacturers – they’ve been driving new product innovation with software for years. Much of this has led to improvements in car safety – based on standards such as MISRA and ISO26262 (see the recent Coverity blog on this subject).

But what’s the big shtick for auto companies today? In short: Infotainment. Consumers are demanding more and more of it in their cars: More advanced and detailed information about fuel consumption; More sophisticated and higher-spec entertainment systems (thorough Multimedia and Web-based devices), Better Services (Navigation, Telephony Integration) and even better driver Security (such as night vision and autopilot features). And the result: Modern cars are literally filled to the roof with software code.

Paradoxically, the distractions of in-car infotainment have increasingly become a cause for concern, when it comes to automobile safety. While standards such as ISO26262 are raising the inherent quality and safety of the automobile (including key functions such as brakes, gears, ignition, lights etc), the arms race for Infotainment Innovation is threatening the driver’s ability to stay focused on the real task in hand – that is, to drive the actual car! By way of (perhaps trivial) example, our neighbor practically wrote off our car this weekend when he undocked his iPhone, and subsequently dropped it on the car floor. By moving over to pick up his iPhone, he suddenly turned the wheel, and crashed into our car, which was parked outside on the street. So much for safety on the roads…

That aside, it seems clear that consumer demand for ever more complex in-car Infotainment systems will not abate. Hence innovation in this space will become even more important. And, like with Neopost, it’s the software that determines the must-have from the nice-to-have infotainment features. So before one thinks about upgrading the car for infotainment reasons, it’s worth making sure it’s has Coverity inside. At least that way, your neighbor will have no excuse if he crashes into your car.

Code defects series: The science behind code execution with buffer overflows

Reading through the release notes of most software vendors, you will notice the security patches typically contain fixes to a variety of buffer overflow defects. For example, this is Apple’s Mac OS X update from December.

In this post on understanding defects, we will look at what goes on in a simple buffer overflow and how that allows one to change the flow or execution of a program. For the discussion that follows, assume we are executing on an Intel x86 CPU and the operating system is Linux.

First, let’s understand how a program in memory is organized. Broadly, there are three sections – text, data and stack (Figure 1).

Figure 1 – Organization of a program memory

Text is the read-only section that includes code and instructions on executing the program, and the Data region stores things like the static variables defined.

The Stack is the region that is most relevant to us. It starts at a fixed address and a register (SP) points to the top of the stack. Elements called ‘stack frames’ are PUSH’ed when calling a function, and POP’ed when returning. Among other things, a stack frame contains the value of the instruction pointer when the function is called. This instruction pointer is the key to altering the flow of execution from a buffer overflow.

Let’s first see what the stack looks like with a simple example:

void password (char *buf)
{
 char var[16];
 strcpy(var, buf);
}
void main ()
{
 password(“mypassword”);
 printf(“This should be executed first\n”);
 printf(“This should be executed next\n”);
}

When this program is executed, you will see the following:

This should be executed first
This should be executed next

We compile this code with gcc using the –S option to generate assembly code as output. The following are the relevant parts the instruction code.

In main(), we see:

call    password <- This will push the instruction pointer (IP) on to the stack so that it can be used as a return address (RET)

And in password(), we see this:

pushl   %ebp <- This pushes the frame pointer (EBP) on to the stack

movl    %esp, %ebp <- This copies the stack pointer (SP) onto EBP making it the new frame pointer (SFP)

You will notice some other manipulations in the assembly code, but for our discussion these are the most relevant instruction. When the function is called, this is what the stack looks like:

Notice that just before the buffer var[16] on the stack is SFP, and before it the return address. So if we could modify the var[16], fill it with a value that is larger than the 16 characters allocated so that we change the RET value, we can execute address we fill RET with. password() calls strcpy() which allows us to overflow var[16] and in turn change RET.

Using GDB, I know the address of the second printf() instruction in main().

(gdb) disassemble main
Dump of assembler code for function main:
0x0804840e <main+0>:    lea    0x4(%esp),%ecx
0x08048412 <main+4>:    and    $0xfffffff0,%esp
0x08048415 <main+7>:    pushl  -0x4(%ecx)
0x08048418 <main+10>:   push   %ebp
0x08048419 <main+11>:   mov    %esp,%ebp
0x0804841b <main+13>:   push   %ecx
0x0804841c <main+14>:   sub    $0x4,%esp
0x0804841f <main+17>:   movl   $0x8048514,(%esp)
0x08048426 <main+24>:   call   0x80483f4 <password>
0x0804842b <main+29>:   movl   $0x804851f,(%esp)
0x08048432 <main+36>:   call   0x8048324 <puts@plt> <- Call to first printf()
0x08048437 <main+41>:   movl   $0x804853d,(%esp)
0x0804843e <main+48>:   call   0x8048324 <puts@plt> <- Call to second printf()
0x08048443 <main+53>:   add    $0x4,%esp
0x08048446 <main+56>:   pop    %ecx
0x08048447 <main+57>:   pop    %ebp
0x08048448 <main+58>:   lea    -0x4(%ecx),%esp
0x0804844b <main+61>:   ret
End of assembler dump.
(gdb)

So now it’s a matter of passing a string larger than 16 characters that contains the address 0×0804843e, overflowing the buffer and changing the flow of the program execution.

An altered code execution flow is just one of the problems due to buffer overflows. And knowing the right address and being able to identify the buffer overflow to exploit is not a trivial challenge. But buffer overflows are still bad as at the very least they cause the program execution to halt if the RET address is corrupted. Even when it is not corrupted, you will have unexpected values if the source buffer is larger than the destination buffer.

If you have an interesting sources you refer to when understanding programming issues such as these, let us know about those by posting a comment here. My personal favorite on understanding buffer overflows is this paper titled ‘Smashing the Stack for Fun and Profit’.

Coverity Person of the Year for 2011

Taking a leaf from TIME, at Coverity we looked at the candidates for person of the year for software development in 2011. It has been an interesting year, and we had a strong set of candidates to pick from. Here are the nominees:

Nominee #1: Code Governance Management

If you are not yet convinced about the importance of software, David Kirkpatrick will definitely make you a believer in this great article on Forbes that shows how software is at the core of all industries and companies today. With every company finding itself a software company, it’s clear that building good quality software is at the core of their business success. This in turn highlighted the importance of a key concept in software development – code governance.

This year, Coverity’s customers were introduced to and embraced Coverity Integrity Control, a solution for implementing code governance that leverages the results of code analysis.

Nominee #2: Software Supply Chain Vendors

2011 was the year when development and engineering organizations started taking the software supply chain into consideration. It takes a village to build large, scalable software systems that power medical devices, cell phones, military equipment, aircrafts, and cars. This village includes developers and teams within the organization, open source communities, and third-party vendors – all key contributors.

The challenge of visualizing how these units come together, how they interact, and the amount of risk introduced due to changes in any part of the system were key areas of innovation and research. Andy Chou provided his thoughts on this in a series of posts on EBN Online that are worth a read.

Nominee #3: Technical Debt

For many, the concept of technical debt became an important factor when evaluating the quality of software. Ward Cunningham’s comparison between technical complexity and debt can be quantified by analyzing the complexity metrics of code during development. Using these metrics, development organizations are now designing software plans by taking into account efforts required to keep technical debt at a minimum.

Nominee #4: QA & Security Teams

This year, Coverity released the Coverity Connector for HP ALM, a solution that combines the strengths of code analysis and Application Lifecycle Management workflow. It’s one of the many bricks that need to be laid to build that metaphorical bridge between development and QA. 2011 was a good year, but I predict 2012 to be the banner year for this new level of collaboration between development and QA and security teams.

But finally, when all was said and done, there is just one nominee that stood out as the Coverity Person of the Year – Nominee #5: The Developer.

With every business now powered by software, and with every company turning into a software company, the value of the developer is far more significant than ever before. Software development is at the core of the rise in mobile computing, cloud computing and whatever powers this next generation web. The global economic slowdown did not slow the speed of innovation driven by software, with the developer one of the most productive members of the workforce. Even from an economic perspective, some of the points made in this article on The Rise of Developeronomics highlight the significance of the developer.

For Coverity, the consistent focus throughout the year was to build solutions for the developer. The Coverity Development Testing suite is a key outcome from this focus. Even the rising importance of the software supply chain emphasized how every developer in the chain is critical to the business success. The highly public security breaches throughout the year indicated the need for new ways to address software security and a voice to the fact that software security is best addressed in development, and the developer is the key to doing this.

2012 will bring with it more asks from the developer and the development teams. But if there is one group that is up for that challenge, it is undoubtedly ‘The Developer’.

Traveling Safely over the Holidays

I read an article in the paper the other day that projected 92 million people would be hitting the road this holdiay season. Road trips can be a fun escape filled with beautiful scenery, stops to funky diners, and spontaneous detours. But road trips can also be filled with peril due to inclement weather, bad drivers, and in my own family’s case, three fighting children in the back seat. However, more and more, software failures could be a leading cause of risk on the road.

The average car is expected to contain 300 million lines of code in the next decade up from 10 million lines of code in today’s cars.  Software will control everything from safety critical systems like brakes, power steering, to basic vehicle controls like doors and windows, and sophisticated infotainment systems and telematics. With the exponential growth of software comes a dramatic increase in software defects. The average car is expected to contain up to 150,000 bugs. In late October, Jaguar had to recall 18,000 cars because of a software glitch.

To help control the risk in software, The International Organization for Standardization, ISO, has put forth ISO 26262 for road vehicles functional safety which is in the final stages of review. Read our new white paper Supporting ISO 26262 with Coverity Development Testing to learn how the Coverity Platform can be used to help assure the quality and safety of the software in automobiles.

Make way for the new rockstars… of development testing

Click here to see a short video about Coverity Exchange, 7th December 2011:

The Beatles, Led Zeppelin, Joy Division and Nirvana: Unique, visionary, trendsetters. Rock bands that produced a sound so distinctive, creative and potent – each defining a new era, all of its own. On a similar note, social media has redefined the way the world communicates with itself. One might even argue that social media is to marketing what Joy Division was to Led Zeppelin (please use your imagination on that one – and your comments appreciated!).

Whilst we (at Team Coverity) strive for excellence in both the products we develop and the marketing of our products, we realize that the real rockstars are our customers. What’s more, given the increasing power of social media, we also understand that customers of any technology vendor have an ever more powerful voice in the market (an essential quality for any serious rockstar!)

So on 7th December 2011, Coverity held the first Coverity Exchange event in London, UK. The aim of the event was really to showcase the success that our clients have achieved in the area of development testing. And boy, did they rock! Software development leaders, from both our client base and from companies interested in working with us, came together to hear our clients’ presentations and to discuss industry best practices for software quality in development.

So who are the rockstars (of the new development testing era)?

Vocals and rhythm guitar: Fons Rademakers of CERN. Fons explained how Coverity is helping to ensure the accuracy of the Large Hadron Collider analysis software. Quite timely, since CERN is potentially on the verge of making one of most significant scientific discoveries of the last 50 years. Believe it or not, Coverity is playing a part in that very discovery!

Lead Guitar: Florian Moesch of Draeger Medical. You may have heard of Draeger from the ‘Draegermen’ in one of the Superman films. Draeger make life saving medical devices and use Coverity to ensure the quality of a number of its devices.

Bass: Frank Klosek of Schneider Electric. Frank did a brilliant presentation explaining how Coverity development testing has helped his organization drive significant improvements in quality and achieve Return on Investment in less than 9 months!

Drums: Pierre Sangouard of ipanema Technologies. Wan Optimisation leader, Ipanema, has deployed Coverity across its Embedded Applications to drive quality and time to market. Pierre explained how the company is now planning the rollout of Coverity Development Testing platform across all of its system applications.

Here’s a short overview video of the event, which should hopefully give you some insight into the event: http://www.youtube.com/watch?v=zqL5vjkaYjM

Fancy joining the band? You know where we are!

Woohoo!! I hit the jackpot… or did I?

I often imagine of winning the lottery or striking it big on a slot machine. The issue for me with this dream is that I don’t buy lottery tickets and seldom find myself in a casino, so the chance of me winning big is probably extremely slim. But this wasn’t the case for Behar Merlaku, the headline from the International Business Times says it all, “Behar Merlaku Wins $57M Jackpot, But Casino Claims ‘Software Error.’

Let me summarize the back story. Mr. Merlaku was playing at a slot machine in Austria. He pulls the lever (or most likely hit the button) and suddenly sirens, bells, and whistles go off letting him know he hit the jackpot. He goes to collect his big pay day – $57 Million, but is told by the casino that while the alarms were trigged the slot machine did not actually end up on the jackpot. And while unfortunately there was a software glitch that created this chaos, he does not get his payout. Well, we know where this story is ending up… in court. Lawyers will start accumulating fees and computer developers will be tasked with re-writing code to fix this “computer glitch.” And Regardless of the court outcome, in the end the casinos will be spending dollars – and it won’t be going to a lucky winner.

And you probably know where my mind is going with this story. Well after I dreamed of how I’d spend $57M, I thought about what, if any, type of testing was done during the software development. Could this poor man’s hopes and dreams been spared? And while, I am not sure how this exact situation came about, I do know that this is just one more story that highlights the importance for companies to implement a development testing methodology. Companies need to use all the safety nets possible so that they can detect these types of issues before it happens in the field. Cuz if it were me at that slot machine, in that casino, being told I didn’t really win… well let’s just say I wouldn’t be so sweet.

Just a Teenager in a Gaming Store

One of the great perks of being a part of the product management team at Coverity is the opportunity to regularly visit our customers, many of which are among the world’s most revered enterprises.  On my most recent trip to Asia, I visited a dozen globally recognized gaming, media and consumer electronic giants.  The storied gaming companies in particular, practically transformed me back into a giddy teenager as soon as I stepped inside their offices and recalled my childhood obsession.

As I visited more and more gaming companies, I began to see trends emerge across the industry.  Most strikingly, it became evident that a vast majority of studios now rely on licensing parts of their core technologies from other vendors or employ open source software.  As gaming interactions become more involved, graphics rendering more photorealistic and design ever more complex, it’s no wonder that even the most fiercely independent studios are now coming to the realization that keeping everything proprietary is no longer a realistic approach in today’s competitive market.  Game engines like Cry, Unreal and Unity permeate the gaming industry and beyond, as a matter of fact, just this past June, the Australian Department of Defense announced that they will begin training Navy sailors with software created using the CryEngine, a well known game engine. And it didn’t just end with game engines, I heard about companies leveraging software platforms like Android, being core in the development of a whole slew of products other than phones like computers, media centers, set-top boxes, automotive navigation systems and even applications for financial institutions.

These days with the software supply chain being as complex for so many companies, simply knowing about the defects and risks lurking around in proprietary code is simply not good enough to ensure high quality software products.  In fact, many of the customers I visited told me that they use Coverity to validate code they receive from third parties.  After all, how farfetched is it really to have your phone, television, computer game, microwave and car all crash because of the same defects in shared code sometime in the near future?

Thoughts on defect density

How do you measure software quality?

It’s a simple question with a not-so-simple answer. For instance, depending on who you ask, software quality depends on one or more of the following:

  • The metric or combination of metrics you consider as accurate indicators of quality
  • Whether the software is pre-release or post-release
  • The complexity of the code base
  • The number of issues reported in the field, and the trend
  • Even if there few visible bugs, the amount of technical debt built in

From our experience with over 1100 customers, who are creators of some of the most fascinating products and software of our generation, defect density is consistently ranked as a good indicator of software quality, especially pre-release.

So why defect density? I believe there are a few strong reasons for this:

  1. With the strengths of modern static analysis you can now automatically, quickly, accurately, and consistently analyze your entire codebase, regardless of which stage of development you are in. You don’t have to be dependent on “feature completeness” to understand your defect density.
  2. Finding out your defect density does not require you to change anything in the development workflow – static analysis is typically easy to integrate in your build.
  3. Research shows the pre-release defect density as reported by static analysis is a very accurate indicator of software quality

The obvious question is then, what is a good number to aim for?

Defect density is typically measure per thousand lines of code (KLOC). The Coverity SCAN team has done a study over the last few years on open source projects, and estimate that 1.0 defects/KLOC typically represents a good quality project – a project that is actively evaluating the reported defects, and making fixes. There are several books that have estimates of software quality based on defect density – e.g. the book ‘Code Complete’ by Steve McConnell has a section on error expectations using defect density that provides an example for Microsoft Applications which have about 10 – 20 defects/KLOC during in-house testing and 0.5/KLOC in released product.

There is no silver bullet for measuring software quality, especially when in development pre-release. But there are certainly good indicators, and defect density is a strong indicator that you can use to guide your software development ship in the right direction.