Add support for TracerDump in TracerFlare #8053
Open
+194
−14
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
BenchmarksStartupParameters
See matching parameters
SummaryFound 15 performance improvements and 15 performance regressions! Performance is the same for 28 metrics, 5 unstable metrics.
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.45.0-SNAPSHOT~0b3278dcb9, baseline=1.46.0-SNAPSHOT~82d80455c5
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.059 s) : 0, 1058674
Total [baseline] (8.625 s) : 0, 8624574
Agent [candidate] (1.094 s) : 0, 1094109
Total [candidate] (8.628 s) : 0, 8627519
section iast
Agent [baseline] (1.185 s) : 0, 1185036
Total [baseline] (9.209 s) : 0, 9209450
Agent [candidate] (1.226 s) : 0, 1226120
Total [candidate] (9.243 s) : 0, 9242985
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.186 s) : 0, 1186377
Total [baseline] (9.163 s) : 0, 9163255
Agent [candidate] (1.231 s) : 0, 1230527
Total [candidate] (9.202 s) : 0, 9202431
section iast_TELEMETRY_OFF
Agent [baseline] (1.182 s) : 0, 1182278
Total [baseline] (9.182 s) : 0, 9181885
Agent [candidate] (1.219 s) : 0, 1218729
Total [candidate] (9.172 s) : 0, 9171718
gantt
title insecure-bank - break down per module: candidate=1.45.0-SNAPSHOT~0b3278dcb9, baseline=1.46.0-SNAPSHOT~82d80455c5
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (716.379 ms) : 0, 716379
BytebuddyAgent [candidate] (694.714 ms) : 0, 694714
GlobalTracer [baseline] (255.756 ms) : 0, 255756
GlobalTracer [candidate] (319.513 ms) : 0, 319513
AppSec [baseline] (56.602 ms) : 0, 56602
AppSec [candidate] (54.762 ms) : 0, 54762
Remote Config [baseline] (737.468 µs) : 0, 737
Remote Config [candidate] (740.095 µs) : 0, 740
Telemetry [baseline] (14.164 ms) : 0, 14164
Telemetry [candidate] (10.645 ms) : 0, 10645
section iast
BytebuddyAgent [baseline] (833.649 ms) : 0, 833649
BytebuddyAgent [candidate] (814.003 ms) : 0, 814003
GlobalTracer [baseline] (246.888 ms) : 0, 246888
GlobalTracer [candidate] (310.991 ms) : 0, 310991
AppSec [baseline] (58.172 ms) : 0, 58172
AppSec [candidate] (58.318 ms) : 0, 58318
Remote Config [baseline] (661.493 µs) : 0, 661
Remote Config [candidate] (630.138 µs) : 0, 630
Telemetry [baseline] (8.916 ms) : 0, 8916
Telemetry [candidate] (7.438 ms) : 0, 7438
IAST [baseline] (21.735 ms) : 0, 21735
IAST [candidate] (20.916 ms) : 0, 20916
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (834.405 ms) : 0, 834405
BytebuddyAgent [candidate] (819.052 ms) : 0, 819052
GlobalTracer [baseline] (247.27 ms) : 0, 247270
GlobalTracer [candidate] (310.024 ms) : 0, 310024
AppSec [baseline] (58.518 ms) : 0, 58518
AppSec [candidate] (57.444 ms) : 0, 57444
Remote Config [baseline] (679.283 µs) : 0, 679
Remote Config [candidate] (617.584 µs) : 0, 618
Telemetry [baseline] (8.873 ms) : 0, 8873
Telemetry [candidate] (7.467 ms) : 0, 7467
IAST [baseline] (21.579 ms) : 0, 21579
IAST [candidate] (21.917 ms) : 0, 21917
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (831.642 ms) : 0, 831642
BytebuddyAgent [candidate] (810.215 ms) : 0, 810215
GlobalTracer [baseline] (246.947 ms) : 0, 246947
GlobalTracer [candidate] (307.921 ms) : 0, 307921
AppSec [baseline] (58.09 ms) : 0, 58090
AppSec [candidate] (58.116 ms) : 0, 58116
Remote Config [baseline] (683.865 µs) : 0, 684
Remote Config [candidate] (635.165 µs) : 0, 635
Telemetry [baseline] (8.723 ms) : 0, 8723
Telemetry [candidate] (7.404 ms) : 0, 7404
IAST [baseline] (21.161 ms) : 0, 21161
IAST [candidate] (20.669 ms) : 0, 20669
Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.45.0-SNAPSHOT~0b3278dcb9, baseline=1.46.0-SNAPSHOT~82d80455c5
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.07 s) : 0, 1069924
Total [baseline] (10.478 s) : 0, 10477814
Agent [candidate] (1.098 s) : 0, 1097621
Total [candidate] (10.492 s) : 0, 10492315
section appsec
Agent [baseline] (1.188 s) : 0, 1188352
Total [baseline] (10.753 s) : 0, 10752679
Agent [candidate] (1.227 s) : 0, 1226959
Total [candidate] (10.705 s) : 0, 10705496
section iast
Agent [baseline] (1.183 s) : 0, 1183160
Total [baseline] (11.003 s) : 0, 11003132
Agent [candidate] (1.231 s) : 0, 1231482
Total [candidate] (10.987 s) : 0, 10987410
section profiling
Agent [baseline] (1.259 s) : 0, 1258983
Total [baseline] (10.849 s) : 0, 10849101
Agent [candidate] (1.32 s) : 0, 1320113
Total [candidate] (10.897 s) : 0, 10896943
gantt
title petclinic - break down per module: candidate=1.45.0-SNAPSHOT~0b3278dcb9, baseline=1.46.0-SNAPSHOT~82d80455c5
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (724.274 ms) : 0, 724274
BytebuddyAgent [candidate] (697.174 ms) : 0, 697174
GlobalTracer [baseline] (258.554 ms) : 0, 258554
GlobalTracer [candidate] (319.574 ms) : 0, 319574
AppSec [baseline] (56.009 ms) : 0, 56009
AppSec [candidate] (54.989 ms) : 0, 54989
Remote Config [baseline] (759.302 µs) : 0, 759
Remote Config [candidate] (731.736 µs) : 0, 732
Telemetry [baseline] (15.142 ms) : 0, 15142
Telemetry [candidate] (11.386 ms) : 0, 11386
section appsec
BytebuddyAgent [baseline] (730.649 ms) : 0, 730649
BytebuddyAgent [candidate] (711.634 ms) : 0, 711634
GlobalTracer [baseline] (253.064 ms) : 0, 253064
GlobalTracer [candidate] (316.626 ms) : 0, 316626
AppSec [baseline] (171.147 ms) : 0, 171147
AppSec [candidate] (167.095 ms) : 0, 167095
Remote Config [baseline] (666.594 µs) : 0, 667
Remote Config [candidate] (646.137 µs) : 0, 646
Telemetry [baseline] (8.169 ms) : 0, 8169
Telemetry [candidate] (7.42 ms) : 0, 7420
IAST [baseline] (19.319 ms) : 0, 19319
IAST [candidate] (19.637 ms) : 0, 19637
section iast
BytebuddyAgent [baseline] (832.675 ms) : 0, 832675
BytebuddyAgent [candidate] (819.604 ms) : 0, 819604
GlobalTracer [baseline] (246.525 ms) : 0, 246525
GlobalTracer [candidate] (310.218 ms) : 0, 310218
AppSec [baseline] (58.119 ms) : 0, 58119
AppSec [candidate] (57.502 ms) : 0, 57502
Remote Config [baseline] (667.53 µs) : 0, 668
Remote Config [candidate] (621.102 µs) : 0, 621
Telemetry [baseline] (8.751 ms) : 0, 8751
Telemetry [candidate] (7.587 ms) : 0, 7587
IAST [baseline] (21.402 ms) : 0, 21402
IAST [candidate] (22.088 ms) : 0, 22088
section profiling
BytebuddyAgent [baseline] (708.158 ms) : 0, 708158
BytebuddyAgent [candidate] (689.659 ms) : 0, 689659
GlobalTracer [baseline] (348.775 ms) : 0, 348775
GlobalTracer [candidate] (435.836 ms) : 0, 435836
AppSec [baseline] (54.734 ms) : 0, 54734
AppSec [candidate] (53.448 ms) : 0, 53448
Remote Config [baseline] (724.276 µs) : 0, 724
Remote Config [candidate] (652.853 µs) : 0, 653
Telemetry [baseline] (8.769 ms) : 0, 8769
Telemetry [candidate] (7.785 ms) : 0, 7785
ProfilingAgent [baseline] (95.574 ms) : 0, 95574
ProfilingAgent [candidate] (93.539 ms) : 0, 93539
Profiling [baseline] (95.599 ms) : 0, 95599
Profiling [candidate] (93.566 ms) : 0, 93566
LoadParameters
See matching parameters
SummaryFound 2 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 16 unstable metrics.
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.45.0-SNAPSHOT~0b3278dcb9, baseline=1.46.0-SNAPSHOT~82d80455c5
dateFormat X
axisFormat %s
section baseline
no_agent (1.358 ms) : 1339, 1378
. : milestone, 1358,
appsec (1.754 ms) : 1730, 1778
. : milestone, 1754,
appsec_no_iast (1.769 ms) : 1746, 1792
. : milestone, 1769,
iast (1.5 ms) : 1475, 1524
. : milestone, 1500,
profiling (1.516 ms) : 1493, 1540
. : milestone, 1516,
tracing (1.469 ms) : 1444, 1494
. : milestone, 1469,
section candidate
no_agent (1.351 ms) : 1331, 1370
. : milestone, 1351,
appsec (1.754 ms) : 1731, 1778
. : milestone, 1754,
appsec_no_iast (1.765 ms) : 1738, 1791
. : milestone, 1765,
iast (1.507 ms) : 1484, 1529
. : milestone, 1507,
profiling (1.536 ms) : 1512, 1559
. : milestone, 1536,
tracing (1.503 ms) : 1478, 1527
. : milestone, 1503,
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.45.0-SNAPSHOT~0b3278dcb9, baseline=1.46.0-SNAPSHOT~82d80455c5
dateFormat X
axisFormat %s
section baseline
no_agent (382.398 µs) : 362, 403
. : milestone, 382,
iast (505.962 µs) : 484, 528
. : milestone, 506,
iast_FULL (744.119 µs) : 722, 766
. : milestone, 744,
iast_GLOBAL (568.417 µs) : 545, 592
. : milestone, 568,
iast_HARDCODED_SECRET_DISABLED (511.601 µs) : 490, 534
. : milestone, 512,
iast_INACTIVE (464.88 µs) : 443, 487
. : milestone, 465,
iast_TELEMETRY_OFF (495.111 µs) : 474, 517
. : milestone, 495,
tracing (456.618 µs) : 436, 477
. : milestone, 457,
section candidate
no_agent (384.477 µs) : 365, 404
. : milestone, 384,
iast (497.722 µs) : 476, 520
. : milestone, 498,
iast_FULL (657.446 µs) : 636, 679
. : milestone, 657,
iast_GLOBAL (526.919 µs) : 505, 549
. : milestone, 527,
iast_HARDCODED_SECRET_DISABLED (495.955 µs) : 474, 517
. : milestone, 496,
iast_INACTIVE (460.452 µs) : 439, 482
. : milestone, 460,
iast_TELEMETRY_OFF (487.029 µs) : 465, 509
. : milestone, 487,
tracing (454.311 µs) : 433, 475
. : milestone, 454,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics. Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.45.0-SNAPSHOT~0b3278dcb9, baseline=1.46.0-SNAPSHOT~82d80455c5
dateFormat X
axisFormat %s
section baseline
no_agent (14.857 s) : 14857000, 14857000
. : milestone, 14857000,
appsec (15.186 s) : 15186000, 15186000
. : milestone, 15186000,
iast (19.062 s) : 19062000, 19062000
. : milestone, 19062000,
iast_GLOBAL (17.845 s) : 17845000, 17845000
. : milestone, 17845000,
profiling (15.094 s) : 15094000, 15094000
. : milestone, 15094000,
tracing (15.005 s) : 15005000, 15005000
. : milestone, 15005000,
section candidate
no_agent (15.567 s) : 15567000, 15567000
. : milestone, 15567000,
appsec (15.017 s) : 15017000, 15017000
. : milestone, 15017000,
iast (18.497 s) : 18497000, 18497000
. : milestone, 18497000,
iast_GLOBAL (17.984 s) : 17984000, 17984000
. : milestone, 17984000,
profiling (14.938 s) : 14938000, 14938000
. : milestone, 14938000,
tracing (14.961 s) : 14961000, 14961000
. : milestone, 14961000,
Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.45.0-SNAPSHOT~0b3278dcb9, baseline=1.46.0-SNAPSHOT~82d80455c5
dateFormat X
axisFormat %s
section baseline
no_agent (1.466 ms) : 1454, 1477
. : milestone, 1466,
appsec (2.358 ms) : 2315, 2401
. : milestone, 2358,
iast (2.106 ms) : 2052, 2160
. : milestone, 2106,
iast_GLOBAL (2.149 ms) : 2095, 2204
. : milestone, 2149,
profiling (1.946 ms) : 1903, 1989
. : milestone, 1946,
tracing (1.945 ms) : 1903, 1987
. : milestone, 1945,
section candidate
no_agent (1.465 ms) : 1454, 1477
. : milestone, 1465,
appsec (2.34 ms) : 2299, 2382
. : milestone, 2340,
iast (2.088 ms) : 2036, 2141
. : milestone, 2088,
iast_GLOBAL (2.131 ms) : 2078, 2184
. : milestone, 2131,
profiling (1.969 ms) : 1926, 2012
. : milestone, 1969,
tracing (1.925 ms) : 1885, 1965
. : milestone, 1925,
|
dd-trace-core/src/main/java/datadog/trace/core/flare/TracerFlareService.java
Outdated
Show resolved
Hide resolved
dd-trace-core/src/main/java/datadog/trace/core/util/TracerDump.java
Outdated
Show resolved
Hide resolved
dd-trace-core/src/main/java/datadog/trace/core/util/TracerDump.java
Outdated
Show resolved
Hide resolved
dd-trace-core/src/main/java/datadog/trace/core/util/TracerDump.java
Outdated
Show resolved
Hide resolved
dd-trace-core/src/main/java/datadog/trace/core/util/TracerDump.java
Outdated
Show resolved
Hide resolved
PerfectSlayer
changed the title
mcculls
reviewed
Dec 12, 2024
mcculls
reviewed
Dec 12, 2024
mcculls
reviewed
Dec 12, 2024
dd-trace-core/src/main/java/datadog/trace/core/PendingTrace.java
Outdated
Show resolved
Hide resolved
mcculls
reviewed
Dec 12, 2024
dd-trace-core/src/main/java/datadog/trace/core/util/TracerDump.java
Outdated
Show resolved
Hide resolved
|
Have you considered tackling this another way, by adding the reporting to |
mhlidd
force-pushed
the
mhlidd/pending_traces
branch
from
4be2999 to
3e4955a
Compare
![datadog-datadog-prod-us1[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
dd-trace-core/src/main/java/datadog/trace/core/PendingTraceBuffer.java
Outdated
Show resolved
Hide resolved
dd-trace-core/src/main/java/datadog/trace/core/PendingTraceBuffer.java
Outdated
Show resolved
Hide resolved
dd-trace-core/src/main/java/datadog/trace/core/PendingTraceBuffer.java
Outdated
Show resolved
Hide resolved
dd-trace-core/src/main/java/datadog/trace/core/PendingTraceBuffer.java
Outdated
Show resolved
Hide resolved
dd-trace-core/src/main/java/datadog/trace/core/PendingTraceBuffer.java
Outdated
Show resolved
Hide resolved
| } | ||
|
|
||
| DumpSupplier supplier = new DumpSupplier(DumpDrain.data); | ||
| buffer.queue.fill(supplier, supplier.data.size()); |
There was a problem hiding this comment.
I wonder if filling the queue back with the original element can't be done when handling the dump element.
It would limit having new element inserted in the same time and messing with the queue order. WDYT?
There was a problem hiding this comment.
If so, does it make sense having the same object implement both Consummer and Supplier?
There was a problem hiding this comment.
Will add this in the new commit!
| if (e instanceof PendingTrace) { | ||
| PendingTrace trace = (PendingTrace) e; | ||
| for (DDSpan span : trace.getSpans()) { | ||
| dumpText.append(span.toString()).append("\n"); |
There was a problem hiding this comment.
'\n' for single char (instead of using a String object)
There was a problem hiding this comment.
Have drain and fill together feels way better! 🙌
Now, about using the collected elements:
- You can declare your comparator as a
private final Comparator<List<Span>> TRACE_BY_START_DATEfield so its meaning will be clear and it will only be allocated once - You will need to figure out how to filter, limit, and map your elements. I would recommend having a look at the Java
StreamAPI - Now you get most of the part in place, try building some tests / triggering your code. You should be able to find some NPE by yourself 😉
| @@ -182,11 +242,18 @@ public void run() { | |||
|
|
|||
| if (pendingTrace instanceof FlushElement) { | |||
| // Since this is an MPSC queue, the drain needs to be called on the consumer thread | |||
| queue.drain(WriteDrain.WRITE_DRAIN); | |||
| queue.drain(WriteDrain.WRITE_DRAIN, 50); | |||
There was a problem hiding this comment.
Hmm I was trying to use the MPSC drain method that takes in a limit to the number of elements that get drained. Do you think that I should avoid doing this and using the Java stream API instead?
dd-trace-core/src/main/java/datadog/trace/core/PendingTraceBuffer.java
Outdated
Show resolved
Hide resolved
dd-trace-core/src/main/java/datadog/trace/core/PendingTraceBuffer.java
Outdated
Show resolved
Hide resolved
PerfectSlayer
force-pushed
the
mhlidd/pending_traces
branch
from
5119ac7 to
3a8b13c
Compare
|
|
||
| public class TraceDumpWriter implements Writer { | ||
|
|
||
| private StringBuilder dumpText; |
There was a problem hiding this comment.
🟠 Code Quality Violation
StringBuilder can lead to memory leaks in long lasting classes (...read more)
StringBuffers and StringBuilders have the potential to grow significantly, which could lead to memory leaks if they are retained within objects with extended lifetimes.
mhlidd
changed the title
|
|
||
| public class TraceDumpJsonExporter { | ||
|
|
||
| private StringBuilder dumpText; |
There was a problem hiding this comment.
🟠 Code Quality Violation
StringBuilder can lead to memory leaks in long lasting classes (...read more)
StringBuffers and StringBuilders have the potential to grow significantly, which could lead to memory leaks if they are retained within objects with extended lifetimes.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What Does This Do
APMAPI-906
Creates a TracerFlare Dump in order to dump traces that have been captured but not yet sent to the agent. This PR introduces a new
DUMP_DRAINelement in order to properly extract the pending traces and sends such traces to aJsonExporterin order to write them into theTracerFlareas Json. These traces will be displayed from oldest first and are currently limited to 50 traces in a dump.Motivation
This PR will help with solving customer issues where traces are captured but not sent to the agent because the root span is never finished/written.
Additional Notes
Contributor Checklist
type:and (comp:orinst:) labels in addition to any usefull labelsclose,fixor any linking keywords when referencing an issue.Use
solvesinstead, and assign the PR milestone to the issueJira ticket: APMAPI-906