Normalize timestamps in ingested messages to current time if in the future

[dennisoelkers]

Expected Behavior

When ingesting messages, a timestamp that is obviously in the future (maybe with a grace period) should be normalized to the current time of the Graylog node.

Current Behavior

Currently, the timestamp of a messages will be taken as is, if it can be parsed. It is generally fine to trust the client, but it is also very easy to misconfigure (e.g. when using timestamps with a wrong resolution). Also, accidentally ingesting messages with timestamps far in the future will lead to the index range of the index this message ends up in to have an end date that leads to inclusion in far more search queries than necessary - potentially all "last five minute" searches until the end date is reached.

If we worry about backward compatibility/breaking weird setups, we could introduce a config setting to disable this.

Possible Solution

We should - either by default or configurable - normalize the timestamp of a message ingested to the current time, if the timestamp of the message is > now + grace.

Steps to Reproduce (for bugs)

1. Ingest message with future timestamp
2. Rotate indices of index set this message ended up in
3. Check index range of index
4. Check that index will be included in now - 5m queries as well

Context

Your Environment

• Graylog Version:
• Java Version:
• OpenSearch Version:
• MongoDB Version:
• Operating System:
• Browser version:

[tellistone]

IMO high priority to fix this blind spot, because this sort of thing is not good news for warm tier performance: