# Duplicate Marking Tool Assessment: Picard vs samtools vs sambamba

Based on Alexander Noskov's three-way benchmark ([phase3_three_way](https://github.com/WeTheAgents/rnaseq_data/tree/main/phase3_three_way)) of 8 ENCODE samples on a 16 vCPU / 30 GB RAM server, with threads at 1/4/8/16 and 3 runs each.

All three tools produce identical duplicate counts on all 8 samples. No divergence.

## Performance summary

|                    | Picard              | samtools            | sambamba            |
|--------------------|---------------------|---------------------|---------------------|
| Commands           | 1                   | 4 (pipe)            | 1                   |
| Peak RSS (t=16)    | 19-25 GB            | ~14.5 GB            | 1.7-3.6 GB          |
| Wall time (t=1)    | 1,855s              | 4,600-6,200s        | 1,500-2,200s        |
| Wall time (t=16)   | ~1,855s (no scaling)| 620-870s            | 250-390s            |
| Thread scaling     | None (JVM)          | Good up to t=8      | Good up to t=8      |
| Config needed      | None                | None                | 3 flags + ulimit    |

Thread scaling plateaus above t=8 for both samtools and sambamba (~1.2x gain from 8 to 16). The I/O investigation ruled out disk as the bottleneck - this is intrinsic to the algorithm (memory bandwidth or serialisation).

## samtools markdup is not a simple swap

The benchmark reveals that samtools markdup requires a 4-step pipe:

```bash
samtools sort -n | samtools fixmate -m | samtools sort | samtools markdup -s
```

Name-sort, fixmate, re-sort to coordinate order, then markdup. This is significantly more complex than either Picard (one command) or sambamba (one command).

At single-threaded execution, samtools is 3x slower than Picard (4,600-6,200s vs 1,855s). It only catches up with multithreading. The memory improvement over Picard (~14.5 GB vs 19-25 GB at t=16) is also moderate rather than dramatic.

## sambamba mitigations

sambamba requires two categories of configuration to run reliably:

### Memory tuning flags (trivially handled)

Three CLI arguments prevent OOM on memory-constrained servers:

```
--sort-buffer-size=512
--overflow-list-size=200000
--hash-table-size=131072
```

These can be baked into the module config via `ext.args` or hardcoded in the process definition. No different from how Picard already receives `--ASSUME_SORTED true --REMOVE_DUPLICATES false`. Zero user exposure.

### File descriptor limit (mostly handled by containers)

sambamba creates hundreds of temporary sort files, exceeding the default Linux limit of 1024. The benchmark required `ulimit -n 65536`.

In practice, this is less severe than it first appears:

- **Docker**: default nofile limit is 1,048,576. Non-issue.
- **AWS Batch / cloud executors**: similarly high defaults. Non-issue.
- **Singularity/Apptainer on HPC**: inherits host limits. Many clusters set 4,096-65,536, which is sufficient. Restrictive systems with a hard limit of 1,024 would be a problem.
- **Conda on local**: varies. macOS has unlimited hard limit (fine). Linux depends on distro/admin config.

A defensive `ulimit -n 65536 2>/dev/null || true` in the process script would handle most cases. The risk is restrictive HPC environments where it silently fails and sambamba crashes mid-run with a cryptic file descriptor error. This could be mitigated with an explicit check and clear error message.

## Recommendation

For a stability-first pipeline like nf-core/rnaseq:

**Keep Picard as the default.** It works, everyone knows it, and duplicate marking is rarely a bottleneck relative to alignment. Picard's lack of thread scaling is a known limitation, but it's also predictable - users don't need to think about it.

**samtools is not a compelling replacement.** Despite being a lighter dependency, the 4-step pipe is more complex, it's 3x slower single-threaded, and the memory improvement over Picard is modest. The main argument for it (simpler dependency) doesn't hold when the implementation is more complex.

**sambamba could be offered as an opt-in alternative** behind a parameter like `--markdup_tool sambamba`, but only if the mitigations are fully baked in and the failure mode on restrictive systems is handled gracefully (clear error, not a silent crash). The memory reduction (1.7-3.6 GB vs 19-25 GB) is genuine and would benefit resource-constrained users. However, adding a new tool choice adds a code path to maintain and test, which has a cost in a pipeline that is already complex.

A new `--markdup_tool` parameter is additional complexity in a pipeline that already has a combinatorial problem with aligners, pseudo-aligners, UMI tools, Sentieon, Parabricks, and strandedness modes. Each new axis multiplies the test matrix and the surface area for subtle bugs. The performance benefit needs to be weighed against that maintenance cost.