My name's Mike Whalen, and I'd like to welcome you to the Software Testing

Coursera specialization. Software testing is a vibrant and important subject, and I

look forward to sharing it with you.

So our first question may be, why do we test? Why is it hard? It turns out that
writing software is hard, and programmers make mistakes. And if we don't
adequately test our systems, we may end up on front page news. Let me give you
some examples from 2016. First, Yahoo. Yahoo managed to leak the personal
information of 500 million of its users, leading to lawsuits and a big drop in stock
price. British Airways managed to make a mistake with its scheduling system,
leading to cancelled reservations and flights and 92 billion pounds lost in their
stock price. Jeep had to recall 1.5 million cars because it was possible to
remotelyhack into them and disable the steering, the brakes, remotely surveil them,
and do other bad things. The Democratic National Committee had their email
server hacked, causing embarrassing internal emails about the political process to
be exposed and possibly influencing the election. A Bangladeshi bank lost $81
million dollars due to some poor code that allowed hackers to get in and steal the
money. And finally, if you owned a Nest thermometer, if you were using it in
November, you probably were without heat for a couple of days. Because the
software update that they pushed out actually broke the ability to control your
furnace. So there are lots of important reasons why we need to have adequately
tested software. In fact, if we look at the cost overall of software failures in 2016,
it's estimated to be over $1.1 trillion per year according to tricentis.com. And it's
worse than that. It's not all about money. Software can also kill you. So this is a
radiation machine called the Therac-25. And back in a period of time in the 1980s,
it managed to kill around 20 people because of badly designed software. So
software now is important to our finances, it's important to our infrastructure, and
it's important to our lives. And we rely on software testing to help us get it right. To
make sure software performs as expected, we have to do two things well. Those are
verification and validation. And what we're trying to do is make sure that the
software is going to meet the user's needs.

So the first thing that we do is verification. This asks the question, "Are we
building the software right?"

So in this case, we assume we have some requirements and what we're trying to
check is that the software conforms to its requirements. This is, of course, very
important and a lot of the focus of software testing. But, another thing that we have
to do is something called validation. And that is, "are we building the right
product?" So maybe our requirements are wrong, and it doesn't actually meet the
user's needs. So we have to perform a level of validation to make sure at the end of
the process, and actually throughout, that we're building something not only that
meets what we think the product should do, but also our customers think the
product should do. And we'll use testing for both of these tasks. So in order to get
the software right, we have several techniques that we'll be going through in the
class. The first thing we need to do is we need to understand and validate the
software requirements. Whether or not we write them down, in order to build tests
for a piece of software, we first have to understand what it does and what it should
do. And the next thing we want to do is we want to apply multiple V&V, that is
verification and validation techniques, throughout the development cycle. So we're
going to want to apply inspections. We want other people to look over the code we
write. We want to have design discussions to make sure that the code that we're
going to build is going to meet its performance and security requirements. We have
static analysis, so we want to check our program for well-formedness. Things like,
does it have any integer overflows, or null pointer dereferences? And then testing,
which is of course the focus of this course, where we have the software product
itself, and and we're seeing if it does the right thing. And finally, for some kinds of
requirements, we want to have runtime verification. So we think the software is
going to work as intended. But in case we see any anomalous behavior, we want to
be able to shut it down and inspect it. So this course, as I just said, focuses on
software testing. And why are we focusing on this area of verification and
validation rather than all the others? Well, software testing is the only defect
detection technique that can check the whole system. So we can do inspections and
design walk-throughs on models and code. But we don't really know how it's going
to work in its intended environment. And that intended environment not only
includes the software you write but the software you use and the hardware it runs
on. So things like the compiler, the processor, any devices that you interact with,
the network, the linker, the loader, the operating system. There are a whole bunch
of things that have to interact correctly in order for your system to work the way it
should. Also, it's currently the best way to assess some kinds of system behaviors.
Right now, other than testing, we don't have good ways of measuring whole system
performance. So we have to build up a performance testing rig in order to really
assess how the system will perform. Another reason we want to use testing is that
oftentimes it's just necessary for the customer to accept the system. So when you
build something for, say, the United States government or another company, they
not only want the software but they usually want a test suite that demonstrates the
software does what it's supposed to do. Finally, testing is a reasonably good way of
documenting expected system behavior. If you look at Agile methods, they tend to
use tests to write down what the software should do. So these are good things
about testing. But testing is always incomplete. Even simple software has lots and
lots possible tests that you can run against it. And so what we're going to do, what
our goal is for this course, is to make testing effective despite its incompleteness.
Okay, so let's try and examine all of the V&V techniques that we have in our
toolbox that are related to testing on one square. So this is going to get a little bit
complicated, so I'm going to explain it one thing at a time. So on the yellow side,
we have analysis techniques, including potentially testing, that are pessimistic.
They'll say your program is wrong when in fact your program may be right. On the
red side, we have optimistic analysis techniques. They say your program is right,
when in fact it may be wrong in some cases. So as we go towards the center, we
get more accurate results. And along the center line, which is the blue line, we have
something that's exactly accurate. So let me give you some illustrations here. For
optimistic techniques, we have typical testing. So a developer writes two or three
tests, they run it through the program. And the tests pass. And the developer thinks,
hey, my code works. Well, maybe it does and maybe it doesn't. But chances are the
test weren't sufficient to really demonstrate that. So it's very optimistic in terms of
the results you get back. On the pessimistic side, we can think of simplistic
program analysis. So there's a tool called Lint that's design to check whether or not
your program has certain simple errors in it, like null pointer dereference or integer
overflow. So this variable may overflow the maximum integer that's possible to be
stored there. And these kinds of tools tend to be very pessimistic. They'll tell you
that your program has all kinds of errors in it when in fact, most of those are false
warnings. We can also look at this in a vertical scale in terms of how complex the
properties that we want to check are. So in typical testing, you can check quite
complicated properties. You can essentially check to see whether or not your
program meets its requirements along certain code paths. The simplistic program
analysis, on the other hand, is not looking at requirements. They're just looking at
simple well-formedness properties of your program. Did you have any overflows
or pointer dereferences where you shouldn't? Or did you have any allocations of
memory without a free? So these things, while important to make sure your
program runs correctly, don't actually rise to the level of requirements. So if we
look at this vertical scale. At the bottom we have precise analysis of very simple
properties, like the type checking that the C compiler does. And at the top, you
have perfect verification of any property over any program. And we know that this
is not theoretically possible. So there's something called the halting problem that
Turing used to prove that we can't even check to see whether a program will
eventually halt, let alone that it'll meet some other requirement. So in this space,
now we can start to look at different analysis techniques. Some are very
pessimistic, some are very optimistic, some can check simple properties, and some
can check complicated properties. So at the bottom, we have things like Java type
checking, which actually does more than C type checking. It can check to see, for
example, whether or not a variable's been initialized. And when you get up to
fancier languages like Haskell, which is a functional language, you can do even
more with a type checker. And it does it accurately. Then we can go up the scale a
little bit, in terms of the complexity of the properties that we can look at and we
can look at static analysis tools. So these are tools that check your program for
well-formedness in a variety of different ways. And there are two different kinds of
static analysis tools that are used in practice. One are called sound static analysis
tools. And as you might guess, these tools will never say that your program is right
when it's wrong. But they may give you a bunch of false warnings that say your
program may misbehave when in fact it's doing the right thing. On the other side
you have unsound static analysis tools that may allow an incorrect program to pass
but don't provide any false analysis warnings. And surprisingly, in practice, people
tend to prefer the unsound ones. Because they don't have to wade through all the
false warnings. And they know that any warnings at the tool returns are likely to be
real problems. Moving up the chain a little bit, we can go from typical testing to
coverage testing. Where we have specific coverage objectives and metrics that we
have to meet before we call our testing process done. And this makes the testing
much more effective in many cases. If we move up the chain further on the
analysis side, we can do something called model checking. Which is equivalent to
exhaustive testing of your system. And for certain kinds of even complex software,
it's possible, actually, to conclusively verify whether certain properties are true
about that software. And now if we go back to the testing side, we can make the
testing more rigorous by adding tool automation. So we can essentially write many,
many, many more tests for free using automation tools to help us. Which makes the
testing more rigorous. Finally, we can move up further to something called
theorem proving, which is just basically math. So you have an automated tool that
helps you reason about your programming. And with theorem proving, you can
prove things that are extremely complex over very complex software, but it takes a
lot of effort. So what we want is we want access to many tools and techniques. And
we can pick the right tool for the job. Now in this class, we're going to focus on the
testing side. But we're going to give you some insight into how these analysis tools
also work to create an effective verification and validation strategy. So what we're
trying to do for the rest of the course is to teach you techniques to verify and
validate your software cost effectively and rigorously. Thank you for your time.

Question 1: Design and prototype walk- Question 2: Unit tests mapping requirements
throughs with users are examples of: (or User Stories in Agile development) to
A. Validation class behaviors are examples of:
B. Verification A. Verification
B. Validation

Question 3: User interviews to see if Question 4: Refactoring is a technique that

requirements/User Stories match the user's modifies the structure of software to improve
expectations of how the system will perform the design (or some non-functional attribute
are examples of: such as performance) without changing the
A. Validation functionality of the software.
B. Verification Regression testing after refactoring the system
to determine whether it behaves the same as
the non-refactored version is an example of:
A. Validation
B. Verification

Question 5: Which of the following are

strengths of testing? (Check all that apply)
A. It checks the whole system, including
software that you didn't write.
B. It can conclusively determine whether the
software is correct.
C. It documents system behavior.