OpenDDS vs CycloneDDS

[jorgefm]

Hi,
I've been able to generate the publisher and subscriber for CycloneDDS and OpenDDS.

Running csub against opub or osub against cpub no data is sent or received. Is it normal?

Running osub/opub give some values far from cyclone ones. For instance, in OpenDDS I get:

0.0001429 kS/s 1.1706e-06 Gb/s 0 lost 0 errs 1.293e+06 us90%lat
0.35692 kS/s 0.0029239 Gb/s 0 lost 0 errs 2.2067e+06 us90%lat
0.33844 kS/s 0.0027725 Gb/s 0 lost 0 errs 3.3317e+06 us90%lat
0.30325 kS/s 0.0024842 Gb/s 0 lost 0 errs 4.467e+06 us90%lat
0.34949 kS/s 0.0028631 Gb/s 0 lost 0 errs 5.5945e+06 us90%lat
0.37171 kS/s 0.003045 Gb/s 0 lost 0 errs 6.7326e+06 us90%lat
0.36695 kS/s 0.003006 Gb/s 0 lost 0 errs 7.8621e+06 us90%lat
0.35419 kS/s 0.0029015 Gb/s 0 lost 0 errs 8.9794e+06 us90%lat

and with CycloneDDS I get:

0.00040558 kS/s 3.3225e-06 Gb/s 0 lost 0 errs 807.24 us90%lat
38.684 kS/s 0.3169 Gb/s 0 lost 0 errs 178.16 us90%lat
50.229 kS/s 0.41148 Gb/s 0 lost 0 errs 43.829 us90%lat
58.072 kS/s 0.47572 Gb/s 0 lost 0 errs 1247.2 us90%lat
50.048 kS/s 0.41 Gb/s 0 lost 0 errs 31.506 us90%lat
59.671 kS/s 0.48882 Gb/s 0 lost 0 errs 37.289 us90%lat
60.856 kS/s 0.49853 Gb/s 0 lost 0 errs 29.757 us90%lat
52.746 kS/s 0.43209 Gb/s 0 lost 0 errs 28.396 us90%lat

Is CycloneDDS faster for something special or I need to tune some OpenDDS parameter?

Thanks!

[eboasson]

It used to be that csub/opub and osub/cpub did work, IIRC, otherwise what'd be the point? But I tried it with a version of Cyclone before merging in XTypes support; it may have to do with that. We can have a look at logs.

Beware though: I may be halfway decent at reading Cyclone DDS logs, but I am no good at reading OpenDDS logs. Still, I did manage to spot the problem with the encodings that made me add the bit in opub.cpp about the data representation. Wireshark is also useful.

Your performance numbers match what I have seen and I have not been able to find out how to tweak OpenDDS to perform reasonably. For out-of-the-box performance it is abominable — at least I'd think this repository is pretty "out-of-the-box". After the right tweaks (whatever they may be), I expect it will improve but every indication I have is that Cyclone will then still be faster than OpenDDS (especially so if you also do -DBATCHING=ON).

[jorgefm]

osub.log

Attached the osub log running against cpub when I've enabled:
DCPSTransportDebugLevel=10
DCPSDebugLevel=10

From cpub I don't know how to enable logs but on console I only get:
$ ./cpub
1649336647.956137 [0] dq.builtin: data(builtin, vendor 1.3): 1036c0b:84a98b54:2ae5dd25:301c3 #1: deserialization failed
^C

I'm only interested in OpenDDS because it has C# support (in beta and only Windows) and I have some C# services I would like to communicate to C/C++ services through DDS

Thanks!

[eboasson]

Enabling logs in Cyclone is most easily done by setting the CYCLONEDDS_URI environment variable, e.g.,

CYCLONEDDS_URI="<Tr><C>trace,plist</><Out>cdds.log</><Pack>cdds.pcap</></>" ./cpub

(Yes, it looks a bit like XML, but it isn't quite and there is a closing tag missing. I want a quick way of setting an option temporarily 🙂) I don't quite expect you to make sense of the output, but it contains pretty much all the details one can think of. This gives a detailed trace including the binary dumps of the discovery parameters, and a packet capture file compatible with wireshark.)

But one can already say something: the 301c3 means it is rejecting a response from the "type lookup writer". I think that means you built Cyclone with type discovery enabled. I wholeheartedly support that, but exchanging type objects that contain the type definitions is an area we flagged as most likely to cause interoperability issues. The spec is massive and ambiguous. We were planning to do some more interoperability in this area, but you are ahead of us, it seems.

I suspect that particular problem will go away if build it without type discovery. There are downsides to that, but I think there's a fair chance it allows you to make progress. I do find it interesting to see all the inputs and outputs. (I suppose I could try reproducing it locally as well, but if you can easily provide them ... 🙂)

[jorgefm]

cdds.log.tar.gz

Yes, you were right, the CycloneDDS was compiled with -DBUILD_EXAMPLES=ON -DENABLE_TYPE_DISCOVERY=YES -DENABLE_TOPIC_DISCOVERY=YES

Sure! Attached you'll find the log from cpub... It's compressed because is a 207,8MB file! Do you need the pcap file too?
Now, I'm going to generate the Cyclone without type discovery and try again.

Thanks!

[jorgefm]

The problem was the type discovery!

Now I have cpub/osub with this values:
7.3307e-05 kS/s 6.0053e-07 Gb/s 0 lost 0 errs 1.0689e+06 us90%lat
1.3293 kS/s 0.01089 Gb/s 0 lost 0 errs 15064 us90%lat
1.1049 kS/s 0.0090509 Gb/s 0 lost 0 errs 15241 us90%lat
1.1877 kS/s 0.0097298 Gb/s 0 lost 0 errs 15703 us90%lat
1.1579 kS/s 0.0094857 Gb/s 0 lost 0 errs 13531 us90%lat
1.0458 kS/s 0.008567 Gb/s 0 lost 0 errs 14348 us90%lat
1.0774 kS/s 0.0088265 Gb/s 0 lost 0 errs 16143 us90%lat
1.0414 kS/s 0.0085308 Gb/s 0 lost 0 errs 14500 us90%lat
1.1555 kS/s 0.0094656 Gb/s 0 lost 0 errs 14184 us90%lat

and opub/csub with:
0.00010371 kS/s 8.4958e-07 Gb/s 0 lost 0 errs 3.152e+05 us90%lat
0.18747 kS/s 0.0015357 Gb/s 0 lost 0 errs 1.1314e+06 us90%lat
0.145 kS/s 0.0011878 Gb/s 0 lost 0 errs 3.1045e+06 us90%lat
0.143 kS/s 0.0011714 Gb/s 0 lost 0 errs 4.0906e+06 us90%lat
0.096996 kS/s 0.00079459 Gb/s 0 lost 0 errs 4.089e+06 us90%lat
0.1269 kS/s 0.0010395 Gb/s 0 lost 0 errs 5.0789e+06 us90%lat
0.12786 kS/s 0.0010474 Gb/s 0 lost 0 errs 6.0665e+06 us90%lat

It seems cpub/osub is better in latency values...

Hope it helps!

[eboasson]

Firstly, I am happy to hear the two are talking. I think there is something fishy if the latency is increasing with 1s/s and I suspect there is a problem in limiting the amount of unacknowledged data in opub. The model of the DDSI protocol is that when you publish data, it first goes into the "writer history cache" and then it is forwarded to the readers. If you consistently write faster than the readers can acknowledge the data, the amount of unacknowledged data in that cache builds up. There are resource limits you can set to prevent "too many" samples from being stored in that cache, but that's tricky because it is dependent on the size of the samples. Cyclone is likely doing so much better because it does flow control for streams of messages by default.

Perhaps my attempt at setting that resource limit in opub was not quite right? You could play with that. (Or perhaps try setting -DSLEEP_MS=10 to make it sleep a bit between samples to see if that makes it behave more like one would expect.)

Secondly, I can reproduce the problem with type discovery. There are some discrepancies between the type identifiers that OpenDDS and Cyclone generate (not unexpected given the spec issues). Cyclone then requests the type definition from OpenDDS to determine whether the types are compatible and rejects the response it gets (this is what causes the "deserialization failed" message).

Why exactly it rejects the response, I don't know yet. It could be a difference in interpretation when a certain length header needs to be injected in the CDR blob, or it could perhaps be that OpenDDS says "what type? I don't know that type" even though Cyclone is asking for the definition of the type identifier it received from OpenDDS in the first place ... Or it could be something else still. We'll figure out the cause and then take it from there.

(BTW I had a chat with the others working on Cyclone, we think work on a C# API can start before the summer. A basic API doesn't take much time, so with a bit of luck you won't have to suffer from OpenDDS performance for a long time. From the looks of it, just a few thousand samples' worth 😜)

[jorgefm]

Yes, you're right, playing with SLEEP_MS building OpenDDS tools changes the lantency although the data rate is changed too!

Now opub/csub with SLEEP_MS=10:
0.00010616 kS/s 8.6968e-07 Gb/s 0 lost 0 errs 90464 us90%lat
0.10373 kS/s 0.0008498 Gb/s 0 lost 0 errs 465.75 us90%lat
0.094863 kS/s 0.00077712 Gb/s 0 lost 0 errs 408.33 us90%lat
0.095504 kS/s 0.00078237 Gb/s 0 lost 0 errs 414.13 us90%lat
0.095418 kS/s 0.00078166 Gb/s 0 lost 0 errs 465.71 us90%lat
0.09553 kS/s 0.00078258 Gb/s 0 lost 0 errs 431.94 us90%lat
0.095522 kS/s 0.00078252 Gb/s 0 lost 0 errs 413.8 us90%lat
0.095548 kS/s 0.00078273 Gb/s 0 lost 0 errs 442.32 us90%lat
0.095436 kS/s 0.00078181 Gb/s 0 lost 0 errs 452.86 us90%lat
0.095535 kS/s 0.00078262 Gb/s 0 lost 0 errs 405.61 us90%lat

Glad to hear that C#API is near to be started! I've been taking a look to OpenDDSharp and is basically a wrapper over OpenDDS then I suspect that will be slower because every C# call will be mapped over a C++ call. How do you plan to implement the C# API in CycloneDDS?

[eboasson]

Glad to hear that C#API is near to be started! I've been taking a look to OpenDDSharp and is basically a wrapper over OpenDDS then I suspect that will be slower because every C# call will be mapped over a C++ call. How do you plan to implement the C# API in CycloneDDS?

C# in Cyclone DDS will ultimately be a wrapper around the C API. The overhead of calling from C# into C like this is negligible, but it is of course some overhead. The (de)serialization will (most likely) be done in C#, because that allows efficiently handling the "complex" types like sequences. This is no different from the C++ binding, or indeed, no different from how the Windows API itself is wrapped in .NET.

[jorgefm]

Thanks for the feedback! I'll be waiting for something to test

[eboasson]

I just merged some fixes from @dpotman for type discovery interoperability with OpenDDS as well as a few changes to this repo. With this, type discovery seems to work as intended. That doesn't change the oddly bad performance from OpenDDS.

It doesn't introduce C# support either, but I guess my hoping for the Easter bunny to bring all that was a bit overly optimistic 🙂

[jorgefm]

Ok, I'll try again with type discovery enabled again and check benchmarks still working

[jorgefm]

After a git pull from CycloneDDS I've rebuilt cyclone with -DENABLE_TYPE_DISCOVERY=YES -DENABLE_TOPIC_DISCOVERY=YES and then I've rebuilt cyclone's i11eperf but it's not working. What I get is:

$ ./osub

$ CYCLONEDDS_URI="trace,plist</>cdds.log</>cdds.pcap</></>" ./cpub
1650354021.477834 [0] cpub: using network interface eno1 (udp/10.16.82.165) selected arbitrarily from: eno1, virbr0, docker0, zcctun0
1650354024.700903 [0] dq.builtin: invalid parameter list (vendor 1.3, version 2.4): pid 75 (TYPE_INFORMATION) invalid, input = 84,0,0,0,1,16,0,64,40,0,0,0,36,0,0,0,20,0,0,0,241,54,248,25,67,186,32,102,82,168,18,65,34,192,70,0,85,0,0,0,255,255,255,255,4,0,0,0,0,0,0,0,2,16,0,64,28,0,0,0,24,0,0,0,8,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,0,0,0,0,0,0,0
1650354024.701102 [0] dq.builtin: invalid parameter list (vendor 1.3, version 2.4): pid 75 (TYPE_INFORMATION) invalid, input = 84,0,0,0,1,16,0,64,40,0,0,0,36,0,0,0,20,0,0,0,241,54,248,25,67,186,32,102,82,168,18,65,34,192,70,0,85,0,0,0,255,255,255,255,4,0,0,0,0,0,0,0,2,16,0,64,28,0,0,0,24,0,0,0,8,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,4,0,0,0,0,0,0,0
1650354024.701138 [0] dq.builtin: SPDP (vendor 1.3): invalid qos/parameters
^C

I've attached the zipped log if it helps...
cdds.log.tar.gz

[eboasson]

The "vendor 1.3" means it is rejecting the TYPE_INFORMATION parameter from OpenDDS. @dpotman, would you by any chance have some time to take a look at it?

On a side note: some bits are now set differently in the type information by our IDL compiler (there was a bug in the way it encodes a default try_construct option), and so re-running the IDL compiler would make sense. That should have happen automatically if you also pulled this repo (which also has some changes related to OpenDDS interoperability). It doesn't explain the error you got, I'm afraid.

[dpotman]

It looks like OpenDDS is not sending the complete type identifier in the TYPE_INFORMATION parameter, and Cyclone requires this to be set to a valid hash type identifier (the XTypes spec section 7.6.3.2.1: The field complete.typeid_with_size shall contain the COMPLETE Hash TypeIdentifier of the TopicType and the serialized size of the associated TypeObject.).

The -Gxtypes-complete option was added in i11eperf to set this option for the OpenDDS IDL compiler. With this option set, the generated type support in the i11eperf build directory (i11eperf/build/opendds_generated/i11eperfTypeSupportImpl.cpp) should contain a line:

XTypes::TypeIdentifier(XTypes::EK_COMPLETE, XTypes::EquivalenceHashWrapper(134, 139, 189, 125, 191, 124, 134, 109, 67, 1, 69, 189, 136, 205))

Could you please check if this line is in the generated code in your build?

[jorgefm]

No, the entry that I have is:
XTypes::TypeIdentifier(XTypes::EK_MINIMAL, XTypes::EquivalenceHashWrapper(142, 208, 146, 62, 13, 179, 190, 177, 37, 24, 236, 61, 170, 165))

I'll try to recompile the OpenDDS with that parameter...