a lot of books have been written about software testing, and there are University courses about software testing. Hell, it's a full research field, I can't pretend to say anything close to be meaningful about this in one post. Why should I write something about it ?
The answer is simple: to explain (once more) why testing is important, and how to approach the testing phase in ns-3.
First and foremost: there's a ns-3 manual section about testing, and it even explains what are the generic goals of software testing (Correctness, Validation and Verification, Robustness, Performance, and Maintainability). Amazing, isn't it ? (The most amazing part is that nobody seems to read the manual, or even know what's inside it.)
Anyway... to test or not to test, that's the question. Not really, tests are necessary, so there's no question. The question may be about what to test.
Before writing a test, one have to consider that ns-3 i a network simulator. If you're writing tests, you probably wrote a protocol, and a protocol involves two (or more) network entities, exchanging data (packets) with delays and random error / losses.
For a software engineer this is often shocking. Testing is much like any other kind of software test, with the twist that API calls can be deferred, mangled, etc. You transmit 10 and the receiver understands 100.
And this is the first lesson: it's not all about positive tests (i.e., conformance testing), it's instead about negative tests: you have to test what happens when the "normal" operation fails.
You have to test this: A send [x] to B, B replies [f(x)] to A.
But also: A send [x] to B, B understand [x'] and replies [f(x')] to A, which understand [g(x')]. A conversation between deaf, and nothing should go crazy in this (beside the developer, of course).
Ok, this is good, but it's too generic. So... some rules. In order to make an effective test for a protocol you should do the following:
- Static test (format): check that the packet are well formed.
- Positive test (conformance): check that the behaviour you have is the expected one.
- Negative test: try shuffling the packet order, loosing a packet and so on.
- Vulnerability test (the paranoid's one): try messing up with the packet's data. Packet's fields representing lengths are a good start.
Point 1 is important, but you need something able to understand the packet's format. A popular choice is Wireshark. However, also Wireshark has its limits. If the protocol is new, chances are that Wireshark can not "dissect" its packets. Moreover, Wireshark itself may be bugged, or not understand all the protocol's options. E.g., right now Wireshark understands only the ETX metric in RPL. It can't understand the HC metric. In this case, only inspecting the packet byte by byte is possible. terribly boring - but necessary.
There's a last kind of test, the duck test: If it looks like a duck, swims like a duck, and quacks like a duck, then it probably is a duck.
In other words, compare against a real system.
This is the most advanced possible test. And it's one of the most valuable and useless tests at the same time: the comparison against a reference implementation.
Before even thinking to do it, consider the following:
- You need to have an implementation to compare against.
- You need to setup a test environment with both your stack and the "other" stack.
And this is where doubts can arise.
- If you have a working implementation, why the hell did you develop the protocol in the simulator? (false problem, maybe the simulator is the reference implementation, maybe you want to change stuff and in the real system you can't, maybe you need the power of the simulator to perform massive tests).
- Something is not working as expected, who's the bad guy? (welcome to cross-implementation debugging).
- How can I force the "other" system to send me "wrong" packets?
Well, I'm not going to explain all the possible cases, let's just say that testing against a real system is like doing normal tests, just much harder.
BUT, there's the duck.
As is: you're calling your protocol a duck because it behaves like a duck, but... who told you that the thing you're calling a duck is a duck?
If you're comparing against a "bad" example, you'll tune your system to that bad example, and both will be consistent with each other... in doing the bad thing- "Follow my steps" [Marty Feldman in Frankenstein Junior].
The bottom line: tests are important. Do them, and make sure they're covering the most important parts of the protocols. Don't avoid them just because it's hard to do them, do them. Bugs are more than often spotted by tests, because the "example" will show you how the system performs when all is right.
