CNDB-13303: Made warn log regarding obsolete property less spammy.#2229
CNDB-13303: Made warn log regarding obsolete property less spammy.#2229MSpryszynski merged 7 commits intomainfrom
Conversation
Checklist before you submit for review
|
20c5812 to
5a2f70f
Compare
5a2f70f to
20cf772
Compare
20cf772 to
6a03437
Compare
scottfines
left a comment
There was a problem hiding this comment.
A couple of things:
- There is a race condition in the actual implementation, which I detailed in a comment. It shouldn't be too hard to fix.
- The testing logic relies on
OBSOLETE_PROPERTY_LOG_INTERVAL_MSbeing non-final, and then uses clock timing to verify the logic. From a production standpoint, this isn't a big deal because we aren't going to change the variable anyway (it's effectively final). But from a testing logic it makes me nervous that we may end up adding an unreliable test, depending on clock timing in CI operating systems. It would be preferable to use some type of abstraction which can be swapped out for testing. On option might be to use aClockabstraction, which can allow for logical and controlled clock changes. Another might be to push the actual properties logging into a class which can be configured while testing to not require explicit clock timing.
| private static final Pattern NEGATIVE_PATTERN = Pattern.compile("(0|false|no)"); | ||
|
|
||
| private static final Map<String, Long> OBSOLETE_PROPERTY_LOG_TIME = new ConcurrentHashMap<>(); | ||
| private static long OBSOLETE_PROPERTY_LOG_INTERVAL_MS = 30_000; |
| // map overload | ||
| testAddProperty(new HashMap<String, String>(){{put("k1", "v1");}}, | ||
| new HashMap<String, String>(){{put("k2", "v2");}}, | ||
| testAddProperty(new HashMap<String, String>() |
There was a problem hiding this comment.
So while Java technically allows this, it's a fairly unclear way of constructing maps. A more standard way would be
testAddProperty(Map.of("k1", "v1"),Map.of("k2","v2"));
(using java.util.Map's static factory methods)
| logger.warn("Ignoring obsolete property {}", name); | ||
| { | ||
| long now = System.currentTimeMillis(); | ||
| Long lastLogged = OBSOLETE_PROPERTY_LOG_TIME.get(name); |
There was a problem hiding this comment.
The consequences of this isn't super high (extra log messages), but this isn't thread safe. The line of logic as written is read-modify-write(get(), do stuff, thenput). Because the entire block isn't atomic, it is possible for two messages to concurrently arrive, both read that the last log time was outside the allowed window, and then write a new log message. It's not the end of the world obviously, but if we can fix it we should.
A more thread-safe way to do this would be to write it as
Long lastLogged = OBSOLETE_PROPERTY_LOG_TIME.putIfAbsent(name, now);
if(lastLogged == null || (now - lastLogged) >= OBSOLETE_PROPERTY_LOG_INTERVAL_MS)
logger.warn("Ignoring obsolete property {}", name);
This ensures that the map modification is atomic, so we won't have extra-message problems
| private static final Pattern POSITIVE_PATTERN = Pattern.compile("(1|true|yes)"); | ||
| private static final Pattern NEGATIVE_PATTERN = Pattern.compile("(0|false|no)"); | ||
|
|
||
| private static final Map<String, Long> OBSOLETE_PROPERTY_LOG_TIME = new ConcurrentHashMap<>(); |
There was a problem hiding this comment.
nit: the name is slightly unclear: perhaps OBSOLETE_PROPERTY_LAST_LOG_TIMES?
| logAppender.list.clear(); | ||
|
|
||
| // Wait 100 ms and try again - should log | ||
| Thread.sleep(100); |
There was a problem hiding this comment.
This can be somewhat unreliable--there is actually no guarantee that Thread.sleep(100) will actually wait 100 milliseconds. Usually it will, but sometimes it might not, and if it returns early, this test will fail. The net result is that the test will be treated as flaky (which we definitely don't want :)).
You would either need to guarantee that you wait a full 100ms(by tracking the time spent), or alter your solution design to not rely on system timings.
6a03437 to
4bffb6c
Compare
|
@scottfines Addressed all comments besides Map creation -> I rollbacked reformat changes to avoid possible conflicts during C* rebases in the future. |
scottfines
left a comment
There was a problem hiding this comment.
This looks pretty good--the main concern I had was with the race condition, and that's resolved nicely. I had one last minor comment to get your thoughts on, but otherwise this is looking good
| // Setup custom clock for testing | ||
| testClock = new TestClock(); | ||
| clockField = PropertyDefinitions.class.getDeclaredField("clock"); | ||
| clockField.setAccessible(true); |
There was a problem hiding this comment.
This works, but you could also just create a @VisibleForTesting constructor in PropertyDefinitions that accepts a clock as a param. You'd need to also have a no-args constructor which uses a default clock, but that would allow you to avoid the reflection on this field, and also let you make the Clock final
4bffb6c to
52af837
Compare
scottfines
left a comment
There was a problem hiding this comment.
this looks good to me--once you can wrangle the CI tool to recognize your tests, you should be good to go. Nice work!
| logAppender.start(); | ||
| logger.addAppender(logAppender); | ||
|
|
||
| lastLoggedTimeField = PropertyDefinitions.class.getDeclaredField("OBSOLETE_PROPERTY_LAST_LOG_TIMES"); |
There was a problem hiding this comment.
I don't think you need this any longer, since you are using constructors to get the same level of access as before.
There was a problem hiding this comment.
This field needs to be static, so creating it by constructor is not an option I think
…configuring threshold for repetition per property.
…configuring threshold for repetition per property. * Handled race condition scenario properly
…configuring threshold for repetition per property. * Created testing purpose constructor instead of using reflection in test
|
❌ Build ds-cassandra-pr-gate/PR-2229 rejected by Butler4 regressions found Found 4 new test failures
Found 8 known test failures |



Made warn log regarding obsolete property less spammy by configuring threshold for repetition per property.
What is the issue
There are reoccurring logs "Ignoring obsolete property nodesync" which provide no context other than that this property is still being used.
Fixes https://github.com/riptano/cndb/issues/13303
What does this PR fix and why was it fixed
Implements mechanism that allow logging single obsolete property once per 30s. It should significantly decrease number of logs and still be equally useful as we do have information that somebody is using this obsolete property.