Network simulations and "realistic" results. The WSN case.

This post is not a rant, it's more a philosophical thing. I hope you don't mind.

Network simulators are widely used tools in academia (from teaching to research) and industry (from research to new technology pre-deployment evaluation). However, I found out that often the simulators are used without thinking about how close to reality the results will be. Leading to disaster (sometimes) and to useless research (more often).

There's a nice presentation from +Sally Floyd (http://www.icir.org/floyd/talks/WNS2-Oct06.pdf) discussing some interesting topics about what's a simulator goal should be. I cannot agree more: the simulator's user should first find out why he/she is using the tool, and then use the appropriate tool.

Now, this is often not done. More than often, a simulator is chosen for one of the following compelling reasons:

1. Everybody in the lab is using it (or, my company always uses it).
2. I already know it, why switch.
3. There's a model in the simulator that seems to suit my need.
4. Etc.

From these points, the most relevant is missing:

• Will the results be useful to what I'm looking for ?

Let's take an example: Wireless Sensor Networks (WSN) and Internet of Things (IoT).

In order to simulate a WSN, you have to have a model, and the model must reflect an actual scenario.

L1/2, the PHY and MAC layers. 

Usually WSNs use IEEE 802.15.4, but it's not the only choice. For example, Body Area Networks can use 802.15.6, or Bluetooth LE.

Even the same protocol (e.g., 802.15.4) have different variants. And for each one something changes. Packet framing, access methods, etc. 802.15.4 in 2.4GHz and in 800MHz are slightly different, and 802.15.4e is completely different.

L3, it's IP...

Nope, it's not "simply" IP. IETF is advocating the use of 6LoWPAN, ZigBee is using a different approach. Other (proprietary) systems use completely different things.

Routing

On this topic, everybody re-invented the wheel. RPL, RIME, LEACH, HERD, [Controlled] Flooding, 

whatever. they're all different.

L4 and above protocols

You'd say that here things should go better... no.

UDP, TCP, HTTP, CoRE/CoAP, OMA variants, ZigBee ones, etc. Name a random one, chances are that somebody is using it.

And now the most astonishing one...

Channel models

It has been proved that simulating any wireless channel is challenging. The model, no matter how good it is, will never exactly capture all the things from a real system, like interference, scattering objects, etc.

To ad complexity to this, let's just say this: two devices, using exactly the same protocol, can have completely different performances. I personally found that 2 devices from 2 different vendors have drastically different performances. One can reach 15m, the other 50m.

This seems a minor detail, but when your scenario is "let's place X devices in the area", you'd like to know how many are need to ensure the network connectivity. Moreover the node's density is driven by the actual phenomena to be observed, not by how crappy their radio device is.

Summarizing: panic. You mis-model one of the above and your simulations will give you some results. False, like a 3$ coin.

However, i said I wasn't king to post a rant. So, here are the good news. What ns-3 can do for you.

1. Channel model: it can be easily changed. It's even possible to use experimental data (but you'll need the BER Vs distance).
2. PHY/MAC: Actually the lr-wpan model is being reviewed. It's implementing the Contiki's NULLMAC, but it's possible (not easy, but possible) to extend the model to mimic other MAC protocols. Mind that this is an extremely important point to make realistic simulations.
3. IP: IPv6 and 6LoWPAN are supported out-of-the-box.
4. Routing: bad news here. I'm working on an RPL implementation, but it's taking AGES. The protocol is very complex. And believe me when I say: actual OSes do not have complete implementations. They just have the bare minimum (and sometimes it's even bugged).
5. L4 and above: UDP is there, upper layers no. However, DCE could come to the rescue here. On the other hand, usually simulating exactly the application layers is out of scope.

I'd say that ns-3 is not yet ready for IoT WSN simulations out of the box, but we're close. Very close.

Have fun simulating, and don't forget to check the models before using them !