Extending simulator to support .pcap tracing

[nondetalle]

.pcap tracing (Issue #87):

• helper/lora-helper: Extending class to support promiscuous traffic sniffing using .pcap files. Implementation follows the one in PointToPointHelper base Ns-3 class. Check EnablePcap() Ns-3 docs for usage;
• model/lora-net-device: Adding .pcap tracing in the gateways' net devices by adding the source callback into Send and Receive functions.

Other fixes:

• model/end-device-lorawan-mac.h: Changing dimension of FCnt header field from 1 octet to 2 according to specifications;
• model/end-device-lorawan-mac.cc: Moving check for the max allowed MACPayload after frame header addition, according to specifications.

EDIT: The two minor fixes moved to other pull request because unrelated

[DvdMgr]

This looks like a great start to implement pcap tracing, thank you for this contribution! I have a couple of questions:

• I had to add DLT_LORATAP = 270 as a DataLinkType in src/network/trace-helper.h. Are there any additional changes to the ns-3 codebase that are required to support this?
• I tested this by calling EnablePcap in complete-network-example.cc, and the resulting file did indeed contain LoraTap packets. However, it seems like a lot of the information is missing - there is no info about the rssi, the sf, source and destination addresses, and so on (which I expected to find based on the LoRaTap definition). Is this expected or am I missing something?

[nondetalle]

Hello @DvdMgr.

• About the first point, I did this sometime ago and so I eventually forgot that I had added DLT_LORATAP as a DataLinkType. Sorry for not mentioning it. To my knowledge this is the only change needed to start working with the DLT_LORATAP .pcap format. Do you have ideas on how this could be managed once the implementation is good enough to be merged? Maybe by adding this information in the module instructions?
• This early implementation of .pcap tracing was useful to me to inspect the content of the MAC header, even if it didn't fit the LoRaTap definition, so I thought it might be a good thing to open a pull request. Your observation makes me realize it may be still too incomplete. I will continue working on this so that it fits the exact format, should I close the pull request for the time being?

As you can see I also added two small fixes. We already discussed the dimension of FCnt via email some time ago, but what do you think about the second one? The regional parameters document (RP002) says that the byte limitation as is defined in the code refers to the entirety of MACPayload field, so I moved the check after LoraFrameHeader is added to any packet.

[DvdMgr]

As for the change to the DataLinkType, I don't see why they wouldn't accept a merge request for this very small change directly in the ns-3 repository! This would ensure our module keeps working with the 'vanilla' ns-3 repository - asking users to change files in ns-3 and providing something that doesn't work out of the box can get messy...

If you have some spare time to keep working on the .pcap implementation, I think the best way to proceed is to keep adding to this pull request (so that I and other can also follow progress and contribute) and merge it once we are satisfied with the overall result.

As for the check on maximum packet size, nice find! I agree that it should take into account the FrameHeader too, since it's defined for the PHY payload.

[nondetalle]

Hello,

I implemented a new header with the necessary fields for .pcap traces to be displayed correctly and moved the small fixes to another pull.

I filled the header using the info from LoraTag. The content of some fields in the new header are still missing (SNR, RSSI related), let me know if you have ideas on how to fill them. Also it is not super easy to understand how this info is encoded, my final interpretation is that something like (dBm value is -139 + rssi) in the documentation means that here rssi is the uint8_t value and this is an explanation of how it will be displayed.

Design remark: I would have liked to be able to detect from inside LoraNetDevice if the traced callback was hooked to a pcap sink, but after investigating the issue it doesn't seem possible, so I settled to adding a flag in the devices activated by EnablePcapInternal() in LoraHelper.

[renzoe]

We may think where we capture the packet. Maybe the MAC layer does not have duty cicle available (or whatever) and will not send this packet. We have to be the closest possible to the PHY layer (Lower-MAC probably) to capture the packet that effectively will go to the medium. See Discussion : #103

[renzoe]

HI @non-det-alle !!
Incredible , thank you for this contribution, I "manually" (...) merged it on my local repo.
I can help you to polish all this.
I will use it mostly on End Device-side. As thing stands, the ED does not capture anything, because I think the functions where the traces are fed are re-defined by end-devices. In short, I had to add the m_promiscSnifferTrace(packet); at different places. To be precise:

• ClassAEndDeviceLorawanMac::SendToPhy
• ClassAEndDeviceLorawanMac::Receive [I did it only if the message is for us.. so it is actually not promiscuous mode :S]

I think we have not so trivial design decisions to make. In order to impact the less possible amount of core-code. And feed the trace cleanly.
At lora-phy all devices could share the code. But will be only promiscuous. In case of a node wanting to capture "unicast" received packets, the appropriate layer to implement the capture is at the "lower" MAC (other example, for sending: after you checked that you respect the duty-cycle restrictions and you will actually modulate this frame).
See: 7e06b92#diff-00db5d5c0bd2658d8a5fbcbf25d6c0cb598b766b44c02281e5717b044205f8b5R124

In any case, thank you @non-det-alle.
I will contact you privately, and we may work together on this, or publicly but in your private fork. In any case, I got this working in one day thanks to your contribution :).

[nondetalle]

Hello @renzoe, I agree with you that the m_promiscSnifferTrace(packet); trace source for downlink packets should be fired at a lower level. The problem is that Ns-3 existing Pcap tracing system is designed for NetDevices (see) and it wouldn't hook the trace sink correctly in other classes (did you actually manage to do this?).
To do this we would need to create a new Pcap helper (more specifically a whole new trace-helper) tailored to our needs.

[renzoe]

HI @non-det-alle ! Thanks for the reply. "Short" answer:

The problem is that Ns-3 existing Pcap tracing system is designed for NetDevices (see) and it wouldn't hook the trace sink correctly in other classes (did you actually manage to do this?).

Yes, I managed, it was you that showed me the way. See how they do it a ns-3 WifiHelper: wifi-helper.cc
Once you have your NetDevice, you ask for his PHY (or MAC in my case), and then you hook. Of course if the device has no PHY there is nothing to sniff :P.

In my dirty way, because I put it at MAC, I am asking for the MAC layer of the Net Device, and finally you have something like this (LoraHelper::EnablePcapInternal of lora-helper.cc)
pcapHelper.HookDefaultSink<LorawanMac> (mac, "PromiscSniffer", file);

Actually, having our own trace-helper is not a bad Idea... we can inherit from the existing one, and also add/redefine the DataLinkType enum to include DLT_LORATAP = 270; problem solved to not depend on the dev branch of ns-3 (still to see if the merge this pull request that will include DLT_LORATAP).

[nondetalle]

Hello,
This is to let you know that DLT_LORATAP has been officially added to the main Ns-3 codebase as one of the possible Pcap filetypes. Rejoice!

[asmanuha]

Dear All,
Thanks for your effort to provide Pcap application so that we can see what's going on in the data transmission. However After I successfully implement Pcap trace in example file "network-server-example.cc" , I found out that there are NO DATA captured in the tracer. I compared the Pcap result gathering from - pointToPoint.EnablePcapAll("first") - in first.cc, and with this lorawan scenario.
Could you advise me how to integrate pcap application, so that the LoraWan DATA appear on Pcap tracer? (For example data send via oneShotHelper or PeriodicSenderHelper).
My goal is that we can able to compare encrypted and un-encrypted lorawan data as a bridge to implement security on LoRaWAN.

Thanks

[DvdMgr]

Hey @non-det-alle, @renzoe - can you provide a quick summary of the current state of this pull request? The branch linked to this PR does not seem to compile, is there a more updated version somewhere that I can test?

[nondetalle]

Hello @DvdMgr,

It's been a bit since I have taken a look so I don't remember, I'll go over it in the weekend and get back to you so we can decide what to do with it.

In the meantime, have you tried it with the latest ns-3-dev? The support for LoRa Pcap files was merged some months ago but may still not be in an official release.

[nondetalle]

After I successfully implement Pcap trace in example file "network-server-example.cc" , I found out that there are NO DATA captured in the tracer. I compared the Pcap result gathering from - pointToPoint.EnablePcapAll("first") - in first.cc, and with this lorawan scenario.
Could you advise me how to integrate pcap application, so that the LoraWan DATA appear on Pcap tracer? (For example data send via oneShotHelper or PeriodicSenderHelper).

@asmanuha sorry, I completely lost your message. I'll check the implementation and get back to you.

[nondetalle]

Hello @DvdMgr,

Sorry for the long pause. I addressed your comment on SNR.

[nondetalle]

I checked the malformed payload error for downlink frames sniffed by devices. After a bit of testing with LinkAdrReq frames, it seems that the Wireshark dissector expects 4 bytes of MIC at the end.
This is in line with specifications, I am not sure how this should be addressed. The issue is present in uplink packets too, where the trailing 4 bytes of the payload are interpreted as MIC.

You can test by adding status->m_reply.payload = Create<Packet> (4); at line 156 of adr-component.cc and activating devices pcap in ADR example. Remember to lower the number of devices to avoid filling up the folder with traces.

[brunocitoni]

Sorry to butt in, but I recently opened an issue (123) about the same MIC problem #123, which is giving rise to slightly incorrect Time-on-Air for packets so I am interested in seeing how this develops!

[renzoe]

Hello @DvdMgr,

Sorry for the long pause. I addressed your comment on SNR.
Thank you very much @non-det-alle ; I am more than sorry for my absence too. I changed works in Aug 2021, and did not have time to continue with this at all. This summer I will have some time. I will be open to discuss/lend a hand. Good catch @brunocitoni about the MIC !

[nondetalle]

Hello,

As mentioned previously by @DvdMgr, I added bidirectional Pcap sniffing (before was only on reception). For packets captured on transmission I put 0 for SNR and RSSI in the respective mandatory Pcap fields, as the sender does not have this information (the LORA_TAP format was designed for sniffing on reception).

Also I improved PcapExample to provide traces of a more interesting scenario. Here, a single device is sending uplinks to a single gateway, and ADR is enabled. With default parameters, for the first 4 frames the device is duty-cycle limited at SF12. Then, after receiving an ADR reconfiguration from the server to SF9, it resumes its 60 seconds periodicity.

I added a second commit with a simple fix to malformed frames: a fake MIC is added at the end of packets before they are sent (both uplink and downlink).

[nondetalle]

I am closing this branch due to lack of replies from maintainers.

[maribu]

#195 tries to do the same as this PR. I didn't really borrow any code of this PR as I sadly spotted this only after I have finished my implementation.

I think #195 might sidestep some of the friction in this PR by doing the bandwidth and SNR stuff as follow up.

[nondetalle]

I agree with @maribu that we should move the work to #195, especially because the code is compatible with the current simulator version. However, I think we should carry over the insights from the discussion above to provide a complete implementation of .pcap tracing. For example, we learned that we are still missing an implementation for MIC trailer (the source of malformed warnings in Wireshark). Thus, before tackling .pcap tracing, we might want to try and merge #126.

[maribu]

For example, we learned that we are still missing an implementation for MIC trailer (the source of malformed warnings in Wireshark).

Ah, I see :)

Another insight is to also capture outgoing frames. I'll add this now and afterwards I'll go through the discussion in detail.

There are somethings added here that #195 is still missing (e.g. filling the SNR with actual values). I have a prototype locally that uses the interference helper to compute that, as there it is easy to take other transmissions as sources for noise into account. I believe this is a good place for that, but the implementation is not really there yet.