
Creating a performance gate in a CI environment, preventing significant
performance regressions from being deployed has been a long-standing goal of
dozens of software teams. But measuring in hosted CI runners is a particularly
challenging task, mostly because of noisy neighbors leaking through
virtualization layers.

Still, it's worth the effort. Performance regressions are harder to catch and
more expensive to fix the longer they go unnoticed. Mostly because:

1. Catching issues in production is too late: Once it's deployed, the damage
   (slowdown, cost, UX impact) is already done.
1. Feedback stays close to the code: Catching regressions during development
   reduces costly context switching and rework.
1. Performance regressions can hide functional bugs: A sudden slowdown might
   point to deeper issues, like accidental $\mathcal{O}(n^2)$ logic or misuse of
   an API, which won't necessarily show up in unit tests.

For many teams, the easiest way to get started is by running benchmarks in their
existing CI environment.

## Attempt on GitHub-hosted runners

Let's measure this noise by using various benchmarking suites from popular
performance-focused open-source projects:
[next.js/turbopack](https://github.com/vercel/next.js) by Vercel,
[ruff](https://github.com/astral-sh/ruff) and
[uv](https://github.com/astral-sh/uv) by Astral, and
[reflex](https://github.com/reflex-dev/reflex) by Reflex.

To measure consistency, we'll use the
[coefficient of variation](https://en.wikipedia.org/wiki/Coefficient_of_variation)
which is the standard deviation divided by the mean. This metric is useful since
it helps expressing and comparing the relative dispersion of the results.

Each run is executed on a different machine, simulating real CI conditions.
Within each run, each result is the outcome of multiple executions of the same
benchmark, which is done by benchmarking framework in use.

This approach ensures that:

- We stay close to real-world CI variability (one machine per run),
- But still benefit from statistical robustness within each run (via repeated
  local executions).

Here are the results on GitHub Actions hosted runners after 100 runs for each
benchmark suite:

<VarianceChart
  title="Benchmark Suites"
  benchmarks={["turbopack", "reflex", "uv", "ruff"]}
  githubHostedData={[3.72, 1.79, 2.79, 2.23]}
  style={{ maxWidth: "500px" }}
/>

Overall, the coefficient of variation is 2.66% on GitHub-hosted runners. Well...
it might seem okay but let's see what's the consequence of this number for
incorrectly detected performance changes (both regressions and improvements).

Let's say we want to create a 2% regression performance gate to catch small
regressions in each benchmark. Considering a normal distribution of the results,
we can estimate the chance to observe a false positive.

<Details>
  <Summary>Modeling the false positive probability by performance gate threshold</Summary>

Starting with $CV$, the coefficient of variation, and $\mu$, the mean, we can
model the false positive rate for different performance gate thresholds.

1. **Model one run as a random variable**: $X \sim \mathcal N(\mu,\,\sigma^2)$.

   With $\sigma = \text{CV} \times \mu$, the standard deviation.

2. **Define the performance gate threshold.**

   We observe a false positive (flags a regression or improvement) when the run
   time drifts more than $\pm \Delta\%$ from the mean:

   $$
     \lvert X - \mu\rvert > \Delta \,\mu .
   $$

3. **Standardise to a Z-score.** Divide both sides by $\sigma$ to turn it into a
   standard normal problem:

   $$
     \Bigl\lvert\tfrac{X-\mu}{\sigma}\Bigr\rvert
     \;>\;
     \frac{\Delta \,\mu}{\sigma}
     \;=\;
     \frac{\Delta}{\text{CV}} .
   $$

   Let $Z = (X-\mu)/\sigma \sim \mathcal N(0,1)$.

   The event becomes $|Z| > \Delta/\text{CV}$.

4. **Compute the probability:**

   $$
   P(\text{false alert}) = P\!\bigl(|Z| > \tfrac{\Delta}{\text{CV}}\bigr)
   $$

   Because the standard normal is symmetric:

   $$
   P(\text{false alert}) = 2\bigl[\,1-\Phi\!\bigl(\tfrac{\Delta}{\text{CV}}\bigr)\bigr]
   $$

   where $\Phi$ is the standard normal CDF.

</Details>

A coefficient of variation of 2.66% gives us **45% chance of having a false
positive** with this 2% performance gate. About 1 out of 2 runs would be a false
positive. In an active development pipeline, this level of noise makes the
results completely unreliable, turning performance checks into distractions
rather than signals, and ultimately eroding trust in performance testing.

To determine a more consistent performance gate threshold for the GitHub
runners, we can plot the false positive rate for different values of the
threshold:

<FalsePositiveChart />

Here, to guarantee a 1% false positive we'd have to use a 7% performance gate.
That's not convincing since any smaller performance changes will go unnoticed.
Moreover, those regressions can compound pretty fast since every time a
regression will be merged, the baseline will be higher, allowing further
regressions to go unnoticed.

## Let's try something else

We've been working on isolating CI runners to eliminate the kind of noise that
makes performance testing unreliable, while keeping the costs reasonable. This
work spans both infrastructure and OS-level optimizations. Our
[Macro Runners](https://codspeed.io/docs/instruments/walltime) run on bare-metal
instances in the cloud, with additional stability configurations specifically
designed for high-precision benchmarking.

We'll cover the technical details in an upcoming blog post (linked here once
published), but in short: you can run your existing tests on this infrastructure
with no changes to your benchmarks, and minimal changes in your CI workflow.

We reran the same benchmark suites on these runners, again, 100 times each. The
improvement in variance was immediate:

<VarianceChart
  title="Benchmark Suites"
  benchmarks={["turbopack", "reflex", "uv", "ruff"]}
  githubHostedData={[3.72, 1.79, 2.79, 2.23]}
  codspeedMacroData={[0.74, 0.62, 0.57, 0.26]}
/>

Overall, we measure a **0.56% average variance**, 5 times less than on
GitHub-hosted runners.

Now, with this 2% performance gate, we get a 0.04% chance (1 in 2500 runs) of
having a false alarms instead of 45% on GitHub-hosted runners before.

Consequently, this shifts the false positive curve dramatically:

<FalsePositiveChart withCodSpeed />

Now, we can reach a sub 1% false positive rate with a 1.5% performance gate,
catching finer-grained regressions without overwhelming contributors with false
alarms!

#### Integration

The configuration changes required to run on CodSpeed Macro Runners are minimal
on a GitHub Actions workflow, boiling down to changing the `runs-on` field:

```yaml
jobs:
  benchmarks:
    runs-on: ubuntu-latest # [!code --]
    runs-on: codspeed-macro # [!code ++]
    steps:
      - uses: actions/checkout@v4
      # ...
      - run: <Benchmark command> # [!code --]
      - uses: CodSpeedHQ/action@v3 # [!code ++]
        with: # [!code ++]
          run: <Benchmark command> # [!code ++]
```

Wrapping the benchmark command with the
[CodSpeed upload action](https://github.com/CodSpeedHQ/action) allows collection
and upload the performance results so you can configure performance gates and
see the detailed results in your CodSpeed dashboard:

<Figure>
  <Box maxWidth="500px" border="1px" borderRadius="lg" p={2}>
    ![Example of a CodSpeed dashboard with macro runner
    results](macro-example.png)
  </Box>
  <Caption>
    Example result from `uv` (check it out
    [here](https://codspeed.io/astral-sh/uv/branches/zb%2F083?uri=crates%2Fuv-bench%2Fbenches%2Fuv.rs%3A%3Auv%3A%3Aresolve_warm_jupyter%3A%3Aresolve_warm_jupyter&runnerMode=WallTime))
  </Caption>
</Figure>

## Toward Reliable Performance Checks

If you've tried building continuous benchmarking in CI before and felt like the
noise made it impossible, you're not alone and most CI environments just weren't
built for this kind of precision.

CodSpeed Macro Runners give you a clean foundation to actually rely on your
performance data. **Less noise means fewer false alarms, and fewer false alarms
means you can finally trust small regressions won't slip through unnoticed**.
And just better performance overall means happier users, lower costs, better
developer feedback loops, and less time spent debugging post-deploy slowdowns.

We'll soon publish a deep dive to share how we built this infrastructure, from
choosing the hardware to the stability tweaks we had to layer on top of the OS,
and why these details matter so much when you're chasing < 1% regressions.

In the meantime, feel free to explore live examples we presented here, check out
[the docs](https://codspeed.io/docs/instruments/walltime), or
[reach out](https://codspeed.io/discord) if you want help setting things up.

### Resources

<Details>
  <Summary>Detailed variance results</Summary>

<VarianceChart
  title="next.js/turbopack build benchmarks"
  benchmarks={[
    "build[shiki]",
    "build[mui]",
    "build[lucide-react-all]",
    "build[framer-motion-single]",
    "build[date-fns-all]",
    "build[framer-motion-all]",
    "build[joy]",
    "build[lucide-react-single]",
    "build[date-fns-single]",
  ]}
  githubHostedData={[4.7, 4.5, 4.5, 3.9, 3.8, 3.5, 3.1, 2.8, 2.7]}
  codspeedMacroData={[0.9, 0.4, 2.2, 0.5, 0.6, 0.7, 0.4, 0.5, 0.5]}
  largeLabels
/>

<VarianceChart
  title="reflex benchmarks"
  benchmarks={[
    "evaluate_page[complicated]",
    "get_all_imports[complicated]",
    "compile_page[complicated]",
    "get_all_imports[stateful]",
    "evaluate_page[stateful]",
    "compile_stateful[stateful]",
    "compile_page[stateful]",
    "compile_stateful[complicated]",
  ]}
  githubHostedData={[1.8, 2.1, 1.6, 2.2, 1.8, 1.6, 1.4, 1.8]}
  codspeedMacroData={[0.4, 0.6, 0.4, 0.7, 0.5, 0.8, 0.4, 1.2]}
  largeLabels
/>

<VarianceChart
  title="uv benchmarks"
  benchmarks={[
    "build_platform_tags",
    "wheelname_parsing[long-compatible]",
    "wheelname_parsing[long-incompatible]",
    "wheelname_parsing[short-compatible]",
    "wheelname_parsing[short-incompatible]",
    "resolve_warm_airflow",
    "resolve_warm_jupyter_universal",
    "resolve_warm_jupyter",
  ]}
  githubHostedData={[2.2, 1.3, 1.4, 3.1, 2.7, 1.5, 5.4, 4.7]}
  codspeedMacroData={[0.2, 0.5, 0.6, 0.2, 1.4, 0.4, 0.5, 0.8]}
  largeLabels
/>

<VarianceChart
  title="ruff benchmarks"
  benchmarks={[
    "multithreaded[pydantic]",
    "large[sympy]",
    "small[altair]",
    "small[freqtrade]",
    "medium[colour-science]",
    "small[pydantic]",
    "small[tanjun]",
    "medium[pandas]",
  ]}
  githubHostedData={[1.3, 2.5, 2.2, 2.3, 2.2, 2.2, 2.7, 2.4]}
  codspeedMacroData={[0.8, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.1]}
  largeLabels
/>
</Details>

- [Macro runners and WallTime instrument documentation](https://codspeed.io/docs/instruments/walltime)

- [Are Benchmarks From Cloud CI Services Reliable?](https://bheisler.github.io/post/benchmarking-in-the-cloud/)
  by the author of Criterion.rs

- _All the benchmarking suites are taken in projects using CodSpeed to run the
  benchmarks already:_
  - [next.js/turbopack benchmarks](https://codspeed.io/vercel/next.js)
  - [reflex benchmarks](https://codspeed.io/reflex-dev/reflex)
  - [uv benchmarks](https://codspeed.io/astral-sh/uv?runnerMode=WallTime)
  - [ruff benchmarks](https://codspeed.io/astral-sh/ruff)

### Appendix: Runner specs

<Table>
  <Thead>
    <Tr>
      <Th />
      <Th>GitHub-hosted runners</Th>
      <Th>CodSpeed Macro runners</Th>
    </Tr>
  </Thead>
  <Tbody>
    <Tr>
      <Td>**Image**</Td>
      <Td>Ubuntu 24.04 GitHub Image</Td>
      <Td>Ubuntu 24.04 with additional stability fine-tuning</Td>
    </Tr>
    <Tr>
      <Td>**CPU Architecture**</Td>
      <Td>x86-64</Td>
      <Td>AArch64</Td>
    </Tr>
    <Tr>
      <Td>**CPU**</Td>
      <Td>AMD EPYC 7763 64-Core Processor</Td>
      <Td>AWS Graviton Cortex-A72</Td>
    </Tr>
    <Tr>
      <Td>**Allocated**</Td>
      <Td>4 vCPUs</Td>
      <Td>All 16 CPUs (bare-metal)</Td>
    </Tr>
    <Tr>
      <Td>**RAM**</Td>
      <Td>16GB</Td>
      <Td>32GB</Td>
    </Tr>
    <Tr>
      <Td>**Cost**</Td>
      <Td>\$0.008 / min</Td>
      <Td>\$0.032 / min</Td>
    </Tr>
  </Tbody>
</Table>



CPython 3.13 was released two weeks ago and this release is the most
performance-oriented in some time. After a quick read of the release notes, a
few things stand out for the impact they can have on the performance:

- CPython can now run in **free-threaded mode**, with the global interpreter
  lock (GIL) disabled
- a brand new **just-in-time** (JIT) compiler has been added
- CPython now bundles the `mimalloc` allocator out of the box

Let's focus on the free-threaded mode in this article to see how to leverage
this change and how it can impact the performance of Python applications by
measuring performance with CodSpeed.

> ⏭️ The JIT and `mimalloc` performance will be covered in the next post. Stay
> tuned!

## Free-threaded CPython

Free-threading is an experimental feature in Python 3.13 that allows CPython to
run without the Global Interpreter Lock (GIL). The GIL is a mutex preventing
multiple threads from executing Python bytecode simultaneously. This design
choice has simplified CPython's memory management and made the C API easier to
work with. However, it has also been one of the most significant barriers to
utilizing modern multi-core processors effectively.

#### The Multiprocessing Workaround

The traditional solution has been to use the `multiprocessing` module, which
spawns separate Python processes instead of threads and while this approach
works, it comes with significant limitations:

1. **Memory Overhead**: Each process requires its own Python interpreter
   instance and memory space. For data-intensive applications, this can quickly
   become a bottleneck.

1. **Communication Cost**: Processes can't share memory directly. Data must be
   serialized and deserialized when passed between processes, which adds
   overhead and complexity.

1. **Startup Time**: Creating new processes is significantly slower than
   creating threads, making it impractical for tasks that require frequent
   spawning of workers.

## Real-World Impact: The PageRank Example

To illustrate these limitations, let's consider implementing PageRank, the
algorithm that powered Google's early search engine. PageRank is an ideal
example because it:

1. Is compute-intensive (matrix operations)
1. Works with large datasets (the web graph)
1. Can benefit significantly from parallelization

A naive multithreaded implementation in Python 3.12 or earlier would be
bottlenecked by the GIL during matrix operations, while a multiprocessing
version would struggle with:

- The memory overhead of copying the graph to each process
- The cost of transferring partial results between processes
- The complexity of managing shared state

Before we proceed, it's important to clarify that our focus here isn't on the
specifics of the PageRank algorithm itself but rather on the parallelization so
we won't go into the details of the algorithm itself here.

Let's look at how we would implement this with those different concurrency
models.

### Textbook Implementation (Single-Threaded)

```python
def pagerank_single(matrix: np.ndarray, num_iterations: int) -> np.ndarray:
    """Single-threaded PageRank implementation"""
    size = matrix.shape[0]
    # Initialize scores
    scores = np.ones(size) / size

    for _ in range(num_iterations):
        new_scores = np.zeros(size)
        for i in range(size):
            # Get nodes that point to current node
            incoming = np.where(matrix[:, i])[0]
            for j in incoming:
                # Add score contribution from incoming node
                new_scores[i] += scores[j] / np.sum(matrix[j]) # [!code highlight]
        # Apply damping factor
        scores = (1 - DAMPING) / size + DAMPING * new_scores # [!code highlight]

    return scores
```

The highlighted lines show the two most computationally intensive parts of the
algorithm. The first computes the score contribution from incoming nodes, while
the second applies the damping factor, incorporating the new scores into the
final result.

Parallelizing the first part will be the most beneficial and easy to implement
since we can divide the range and thus efficiently use multiple threads to
compute the `new_scores` array.

### Multithreaded Implementation

For the multithreaded implementation, we'll start by dividing the matrix into
multiple chunks:

```python
chunk_size = size // num_threads
chunks = [(i, min(i + chunk_size, size)) for i in range(0, size, chunk_size)]
```

Each thread will then work on a different chunk of the matrix, updating the new
scores:

```python
def _thread_worker(
    matrix: np.ndarray,
    scores: np.ndarray,
    new_scores: np.ndarray,
    start_idx: int,
    end_idx: int,
    lock: threading.Lock,
):
    size = matrix.shape[0]
    local_scores = np.zeros(size)

    for i in range(start_idx, end_idx):
        incoming = np.where(matrix[:, i])[0]
        for j in incoming:
            local_scores[i] += scores[j] / np.sum(matrix[j])

    with lock: # [!code highlight]
        new_scores += local_scores # [!code highlight]
```

It's important to note that updating the `new_scores` array is done behind a
lock to prevent race conditions. This could potentially become a bottleneck if
the lock is held for too long, but in practice, parallelizing the first part of
the algorithm should provide a significant speedup already.

Finally, we'll feed the chunks to each of the threads:

```python
new_scores = np.zeros(size)
lock = threading.Lock() # [!code focus]
with concurrent.futures.ThreadPoolExecutor(max_workers=num_threads) as executor: # [!code focus]
    # Process chunks in parallel # [!code focus]
    futures = executor.map( # [!code focus]
        lambda args: _thread_worker(*args), # starmap isn't available on ThreadPoolExecutor # [!code focus]
        [ # [!code focus]
            (matrix, scores, new_scores, start_idx, end_idx, lock) # [!code focus]
            for start_idx, end_idx in chunks # [!code focus]
        ], # [!code focus]
    ) # [!code focus]
new_scores = (1 - DAMPING) / size + DAMPING * new_scores
scores = new_scores
```

### Multiprocessing Implementation

Essentially, the `multiprocessing` implementation is extremely similar to the
`threading` one, let's focus on the differences:

- Each worker, will now return the `local_scores` array instead of updating a
  shared `new_scores` array since processes can't share memory directly. The
  local scores will then be aggregated in the main process:

  ```python
  # Combine results
  new_scores = sum(chunk_results)
  ```

  While this should be faster than the threading version, it still incurs the
  overhead of the inter-process communication which can become very significant
  for large datasets.

- Instead of using a `ThreadPoolExecutor`, it will use a `multiprocessing.Pool`.
  The APIs are very similar, but `multiprocessing.Pool` will spawn a pool of
  processes instead of threads:
  ```python
  with multiprocessing.Pool(processes=num_processes) as pool: # [!code focus]
      # Process chunks in parallel # [!code focus]
      chunk_results = pool.starmap(_process_chunk, chunks) # [!code focus]
      # Combine results
      new_scores = sum(chunk_results)
      new_scores = (1 - DAMPING) / size + DAMPING * new_scores
      scores = new_scores
  ```

## Measuring Performance

In order to measure the actual performance changes, let's build a performance
test. First things first, we need to generate some testing data:

```python
def create_test_graph(size: int) -> np.ndarray:
    # Fixed seed  # [!code highlight]
    np.random.seed(0) # [!code highlight]
    # Create random adjacency matrix with ~5 outgoing edges per node
    matrix = np.random.choice([0, 1], size=(size, size), p=[1 - 5/size, 5/size])
    # Find nodes with no outgoing edges
    zero_outdegree = ~matrix.any(axis=1)
    zero_indices = np.where(zero_outdegree)[0]
    # For each node with no outgoing edges, add a random edge
    if len(zero_indices) > 0:
        random_targets = np.random.randint(0, size, size=len(zero_indices))
        matrix[zero_indices, random_targets] = 1

    return matrix
```

As highlighted, we're using a fixed seed to ensure reproducibility from one run
to the next. This is important when comparing the performance of different
implementations. Here we're building some fake connections between pages to
build a realistic graph but the mathematical operations would be exactly the
same with an empty matrix as long as the size is the same.

Next, we'll use
[`pytest-codspeed`](https://github.com/CodSpeedHQ/pytest-codspeed), a `pytest`
plugin to measure the performance of the different implementations with various
parameter and with multiple builds/versions of CPython.

First let's define the benchmark cases:

```python
@pytest.mark.parametrize(
    "pagerank",
    [
        pagerank_single,
        partial(pagerank_multiprocess, num_processes=8),
        partial(pagerank_multithread, num_threads=8),
    ],
    ids=["single", "8-processes", "8-threads"],
)
@pytest.mark.parametrize(
    "graph",
    [
        create_test_graph(100),
        create_test_graph(1000),
        create_test_graph(2000),
    ],
    ids=["XS", "L", "XL"],
)
def test_pagerank(
    benchmark: BenchmarkFixture,
    pagerank: PagerankFunc,
    graph: np.ndarray,
):
    benchmark(pagerank, graph, num_iterations=10)
```

Here we're testing the 3 implementations with 3 different graph sizes. The
`benchmark` fixture is provided by `pytest-codspeed` and will measure the
execution time of the `pagerank` function with the given args.

Then, let's write a GitHub Actions workflow to measure the performance with
various builds of CPython on CodSpeed's infrastructure:

```yaml
on:
  push:
jobs:
  codspeed:
    runs-on: codspeed-macro # requests a CodSpeed Macro runner for the jobs # [!code highlight]
    strategy:
      matrix:
        python-version: ["3.12", "3.13"]
        include:
          - { python-version: "3.13t", gil: "1" } # [!code highlight]
          - { python-version: "3.13t", gil: "0" } # [!code highlight]
    env:
      UV_PYTHON: ${{ matrix.python-version }} # [!code highlight]
    steps:
      - uses: actions/checkout@v4
      - name: Install uv
        uses: astral-sh/setup-uv@v3
      - name: Install CPython & dependencies
        run: uv sync --all-extras
      - name: Run benchmarks
        uses: CodSpeedHQ/action@v3
        env:
          PYTHON_GIL: ${{ matrix.gil }} # [!code highlight]
        with:
          run: uv run pytest --codspeed --codspeed-max-time 10 -vs src/tests.py # [!code highlight]
```

Here we're running the benchmarks with Python 3.12, 3.13, and 3.13 with free
threading support (`3.13t`), both with and without the GIL. Running 3.13 both
with and without free-threading support will allow us to see its impact on the
performance even when the GIL is enabled.

The python builds used are pulled by `uv` directly from
[python-build-standalone](https://github.com/indygreg/python-build-standalone)
built with `GCC 6.3.0 20170516`

The jobs will run on
[CodSpeed Macro](https://codspeed.io/docs/instruments/walltime) runners, which
are ARM64 bare-metal instances with 16 cores and 32GB of RAM, dedicated to each
job.

## Results

<Figure maxW="900px" mx="auto" position="relative">

<ReactECharts
  opts={{ renderer: "svg" }}
  style={{ height: "350px", width: "100%" }}
  option={{
    legend: {
      textStyle: {
        fontFamily: "Inter",
        color: colors.text.principal,
      },
    },
    textStyle: {
      fontFamily: "Inter",
      color: colors.text.principal,
    },
    tooltip: {
      valueFormatter: (value) => `${value} s`,
      backgroundColor: colors.zinc[800],
      textStyle: {
        color: colors.text.principal,
      },
    },
    grid: {
      bottom: 35,
    },
    dataset: {
      source: [
        ["product", "single", "threads", "processes"],
        ["3.12.7", 1.487, 1.845, 2.75],
        ["3.13.0", 1.44, 1.825, 2.636],
        ["3.13.0t GIL", 1.7, 2.3, 6.2],
        ["3.13.0t no GIL", 1.7, 0.417, 6.3],
      ],
    },
    xAxis: {
      type: "category",
    },
    yAxis: {
      axisLabel: {
        formatter: "{value} s", // Customize the unit here
      },
      splitLine: {
        lineStyle: {
          color: colors.zinc[700],
        },
      },
    },
    series: [
      {
        type: "bar",
        seriesLayoutBy: "row",
        itemStyle: {
          color: "#93c47d",
          borderRadius: [5, 5, 0, 0],
        },
      },
      {
        type: "bar",
        seriesLayoutBy: "row",
        itemStyle: {
          color: "#6d9eeb",
          borderRadius: [5, 5, 0, 0],
        },
      },
      {
        type: "bar",
        seriesLayoutBy: "row",
        itemStyle: {
          color: "#8e7cc3",
          borderRadius: [5, 5, 0, 0],
        },
      },
      {
        type: "bar",
        seriesLayoutBy: "row",
        itemStyle: {
          color: "#f6b26b",
          borderRadius: [5, 5, 0, 0],
        },
      },
    ],
  }}
/>
<MeasuredBy
  position="absolute"
  sx={{
    right: { md: 5 },
    bottom: { md: -2 },
    left: { base: "15%", md: "unset" },
    top: { base: "20%", md: "unset" },
  }}
/>

<Caption fontSize="sm">
  Time (in s) for the L graph size, **less is better**
</Caption>

</Figure>

{/* prettier-ignore */}
- Without enabling new build options both <PBuildColor build="3.12">{"3.12"}</PBuildColor> and
  <PBuildColor build="3.13">{"3.13"}</PBuildColor> perform very similarly.
  We also clearly see the limitation of the `multiprocessing` implementation being even
  slower than the single-threaded one due to the overhead of the inter-process communication.

{/* prettier-ignore */}
- As expected the `threading` based implementation is the fastest when running
  <PBuildColor build="3.13t-no-gil">3.13t with no GIL</PBuildColor> and the GIL is effectively not limiting the
  parallel execution of the threads anymore.

{/* prettier-ignore */}
- Still, when running with the free threaded build both <PBuildColor build="3.13t-gil">with</PBuildColor> and
  <PBuildColor build="3.13t-no-gil">without</PBuildColor> the GIL, we see a significant
  slowdown for all other implementations. This is mostly because the free-threaded build
  requires the specializing adaptive interpreter to be disabled, thus clearly decreasing
  the performance of the other implementations. This overhead should be reduced in the
  3.14 release where the specializing adaptive interpreter will be thread-safe and thus
  will be re-enabled. At that point, migrating to the free-threaded build should be a
  no-brainer for a lot of parallel applications and it will be interesting to measure
  the performance changes.

For all other graph sizes, the results are very similar and the conclusions are
the same. From this measurement, we can see that the new free-threaded build of
CPython 3.13 can have a significant impact on the performance of parallel
applications, bringing a very relevant alternative to `multiprocessing`. Still
it's experimental and not yet ready for production use because of the overall
slowdown it introduces, but it's a very promising step in the right direction!

#### Note

This benchmark doesn't include subinterpreters, which is another way to run
Python code parallelly without the GIL introduced in Python 3.12. Subintepreters
proved to be slower than other approaches in most cases, mostly because where
data sharing and inter-worker communication has not been fully solved yet. But
when it's there, it might definitely be a nice alternative to `multiprocessing`.

## Resources

- [Repository with the full code](https://github.com/CodSpeedHQ/python-parallel-pagerank/)

- [What's New In Python 3.13?](https://docs.python.org/3.13/whatsnew/3.13.html)
- [Python experimental support for free threading](https://docs.python.org/3/howto/free-threading-python.html)
- [py-free-threading](https://py-free-threading.github.io/): a centralized
  collection of documentation and trackers around compatibility with
  free-threaded CPython
- [PEP 659 - Specializing Adaptive Interpreter](https://peps.python.org/pep-0659/)
- [Docs on having the Specializing Adaptive Interpreter without the GIL](https://docs.google.com/document/d/1hsV25JSwDb08c6-aHrI_YBHyl_uQKlys0ODihSx_aSw)



The Rust 1.78.0 version was released on May 2, 2024. The
[release announcement](https://blog.rust-lang.org/2024/05/02/Rust-1.78.0.html#compatibility-notes)
mentions that the bundled LLVM version is upgraded to 18, completing the
announced `u128`/`i128`
[change](https://blog.rust-lang.org/2024/03/30/i128-layout-update.html) for x86
architectures. It is mentioned that if your code uses 128-bit integers, **you
may notice runtime performance improvement**. Let's dive into how this can
happen.

### The alignment issue with `u128`/`i128`

> The _alignment_ of a value specifies what addresses are valid to store the
> value at. A value of alignment `n` must only be stored at an address that is a
> multiple of `n` (from the
> [the Rust Reference](https://doc.rust-lang.org/reference/type-layout.html#size-and-alignment))

<Figure maxW="600px" mx="auto">
  ![How alignment works](alignment.excalidraw.png)
</Figure>

Normally, the memory size of a value should always be a multiple of its
alignment to prevent misalignment issues.

However, on x86 architectures, there is an inconsistency with the alignment of
128-bit integers between rustc (the rust compiler) and C conventions:

<Table maxW="500px">
  <Thead>
    <Tr>
      <Th>Data type</Th>
      <Th>Memory size</Th>
      <Th>Alignment</Th>
    </Tr>
  </Thead>
  <Tbody>
    <Tr>
      <Td>i128 (rustc 1.77.0)</Td>
      <Td>16 bytes</Td>
      <Td>8 bytes</Td>
    </Tr>
    <Tr>
      <Td>__int128 (clang 17.0.1)</Td>
      <Td>16 bytes</Td>
      <Td>16 bytes</Td>
    </Tr>
  </Tbody>
</Table>

In Rust < 1.77, 128-bit integers are **8-byte aligned,** whereas the
corresponding C types are **16-byte aligned**. But they all have a size of **16
bytes**.

Let's see how this misalignment can degrade performance.

### Why memory misalignment can lead to worse performance?

Having a memory size greater than its alignment can lead to cases where the
value is stored at an address right at the edge of a memory cache line (which is
usually **64 bytes**), which will cause the value to be stored on **two cache
lines instead of one**.

Let's create an example where this happens with Rust 1.76.0 on an x86_64
machine. Let's look at the following code:

```rust
use type_layout::TypeLayout;

const GAP: usize = 128;

#[derive(TypeLayout, Copy, Clone)]
#[repr(C)]
struct StockAlignment {
    _offset: [u8; GAP - 8], // 1-byte aligned
    data: u128,             // 8-byte aligned
}

pub fn main() {
    println!("Alignment of u128: {}", align_of::<u128>());
    println!("{}", StockAlignment::type_layout());
}
```

Here, we created a `StockAlignment` struct that contains an array `_offset` of
`u8` and an `u128` integer `data`. We use the
[`type_layout` crate](https://docs.rs/type-layout/latest/type_layout/) to be
able to see what the size and alignment of the struct will be.

After we execute the code, we have the following:

```text
Alignment of u128: 8
StockAlignment (size 136, alignment 8)
| Offset | Name      | Size |
| ------ | --------- | ---- |
| 0      | _offset   | 120  |
| 120    | data      | 16   |
```

That means that an instance of `StockAlignment` will span three cache lines of
64 bytes, like so:

<Figure maxW="500px" mx="auto">

![Stock alignment with Rust 1.76.0](stock-alignment-1.76.excalidraw.png)

  <Caption>Each square represents 8 bytes</Caption>
</Figure>

The misalignment problem is visible here. When `data` is accessed, **two cache
lines of 64 bytes are read** from memory. This is effectively loading twice as
much data as needed, since `data` is only 16 bytes, a **single cache line could
have been read** if it was properly aligned.

Let's create a new struct where we enforce `data` and its containing struct to
be correctly (16-bit) aligned. With the proper alignment, this is what we expect
the memory layout will look like:

<Figure maxW="500px" mx="auto">

![Enforced alignment with Rust 1.76.0](enforced-alignment.excalidraw.png)

  <Caption>Enforced alignment with Rust 1.76.0</Caption>
</Figure>

With that alignment, accessing `data` will only load a **single cache line**. To
achieve it, we can write the following code:

```rust
use type_layout::TypeLayout;

const GAP: usize = 128;

#[derive(Copy, Clone)]
#[repr(C, align(16))]
struct AlignedU128(u128);

#[derive(TypeLayout, Copy, Clone)]
#[repr(C)]
struct EnforcedAlignment {
    _offset: [u8; GAP - 8],
    data: AlignedU128,
}

pub fn main() {
    println!("{}", EnforcedAlignment::type_layout());
}
```

Here we created a struct `AlignedU128`, that only stores a single `u128` and
enforce it to be 16-byte aligned. We then use this as the type of `data` in a
new `EnforcedAlignment` struct.

After we execute the code, we have the following:

```text
EnforcedAlignment (size 144, alignment 16)
| Offset | Name      | Size |
| ------ | --------- | ---- |
| 0      | _offset   | 120  |
| 120    | [padding] | 8    |
| 128    | data      | 16   |
```

So we optimized `EnforcedAlignment` to access `data` more efficiently, but it
comes at a cost in memory since it costs 8 more bytes than `StockAlignment`.

### Comparing the performance between the two alignments

To measure the performance of loading `data` with different alignments, we
created performance tests with the
[criterion.rs benchmarking library](https://github.com/bheisler/criterion.rs).

Before running the benchmarks, we allocate two arrays of `8192` elements, one
filled with `StockAlignment` instances, and the other one with
`EnforcedAlignement`. We then create two benchmarks where we access the `data`
property of all the elements of the array:

```rust
let c: &mut Criterion;
let stock_align: Vec<StockAlignment>;

c.bench_function("stock align bench", |b| {
    b.iter_batched(
        cache_fuzzer,
        |_| {
            let mut sum = 0;
            for i in 0..N {
                sum += stock_aligned[i].data;
            }
            sum
        },
        criterion::BatchSize::PerIteration,
    );
});
```

For the other struct, we have the same benchmark code.

<Alert>
  The [`iter_batched`
  function](https://bheisler.github.io/criterion.rs/book/user_guide/timing_loops.html#iter_batchediter_batched_ref)
  allows to pass a setup function as the first argument, with the actual
  benchmark being the second argument. The `cache_fuzzer` function is used to
  clear the cache before each benchmark to avoid the effect of cache hits on the
  benchmark results. This ensures that previous iterations have close to no
  impact on the current one.
</Alert>

To have stable and reproducible results, we use the **CodSpeed runner, providing
a consistent execution environment**.

<PerfDiff
  base={280.4e-6}
  head={250.5e-6}
  name="ordered access"
  description="Changing from stock to enforced alignment"
/>

This notable **performance improvement when using the properly aligned struct**
confirms our initial idea: optimizing the alignment effectively **reduces the
number of cache misses** and thus RAM accesses, ending up with a **positive
impact on performance**.

Here we only tested when accessing elements of the array in an ordered way.
Under the hood, there could be some optimizations loading more memory than
requested from contiguous blocks in our measurements. For example, loading more
memory than requested from contiguous blocks. To test in different situations,
we added two more groups of benchmarks:

- `reversed access`, to simulate access in a reversed order
- `randomized access`, to simulate access in a randomized order

We get the following results:

<PerfDiff
  base={279.8e-6}
  head={249.9e-6}
  name="reversed access"
  description="Changing from stock to enforced alignment"
/>

<PerfDiff
  base={241.5e-6}
  head={219.4e-6}
  name="randomized access"
  description="Changing from stock to enforced alignment"
/>

The performance gains are consistent with the first we observed and thus the
order in which we access the data does not affect the performance much.

<Alert>
  Using the **CodSpeed runner will execute the benchmark code on a virtualized
  CPU** in a controlled CI environment, measuring the CPU instructions and
  turning them back into a human-readable time measurement. Learn more about the
  [CodSpeed performance metric on our
  docs](https://codspeed.io/docs/instruments/cpu/overview).
</Alert>

### Real-world check with Rust 1.78.0

Now that we identified and shown the problem with Rust 1.76.0, let's bump Rust
to 1.78.0 and see what happens.

```toml
// rust-toolchain.toml
[toolchain]
channel = "1.76.0" # [!code --]
channel = "1.78.0" # [!code ++]
```

Let's check the alignment of our previous misaligned struct:

```text
Alignment of u128: 16                   <-- was 8 before
StockAlignment (size 144, alignment 16) <-- size was 136 before, alignment was 8
| Offset | Name      | Size |
| ------ | --------- | ---- |
| 0      | _offset   | 120  |
| 120    | [padding] | 8    |           <-- new
| 128    | data      | 16   |           <-- properly aligned now
```

The inconsistency has indeed been solved. The performance of `stock align`
increased by **10%** and is now equal to `enforced align` since the alignments
are the same.

<Figure maxW="400px" mx="auto">

![CodSpeed report of the bump to 1.78](rust-align-128-1.78.png)

  <Caption>[Check out the performance report of the Rust bump to 1.78.0](https://codspeed.io/art049/rust-align-128/branches/upgrade-rust-1-78)</Caption>
</Figure>

### A note on 1.77

Although it was not announced in the
[release blog post](https://blog.rust-lang.org/2024/03/21/Rust-1.77.0.html), the
[128-bit integers misalignment fix](https://github.com/rust-lang/rust/pull/116672/)
was already included as a part of the
[1.77.0 release](https://github.com/rust-lang/rust/releases/tag/1.77.0). The
reason it was announced later is that there was a related bug linked to 128-bit
integers arguments which was only solved with the
[update to LLVM 18](https://github.com/rust-lang/rust/pull/120055/) completed
with the 1.78.0 release.

[Check out the CodSpeed report of the pull request](https://codspeed.io/art049/rust-align-128/branches/upgrade-rust),
where the performance gain is already visible.

### Performance gains from bumping to 1.78.0 in the ecosystem

We have seen a number of repositories gaining performance increases when
upgrading the toolchain from **1.77.x** **to 1.78.0**, with **performance gains
up to 21%** depending on the benchmark. Those performance changes are not solely
due to the alignment fix, but most probably related to optimizations released
with the new LLVM version.

<Figure maxW="300px" mx="auto" my={4}>

![CodSpeed report of the bump of rust to 1.78.0 on oxc](oxc-rust-1.78.png)

  <Caption>[Improvement on oxc-project/oxc](https://codspeed.io/oxc-project/oxc/branches/renovate/rust-toolchain)</Caption>
</Figure>

<Figure maxW="300px" mx="auto" mt={0} mb={4}>

![CodSpeed report of the bump of rust to 1.78.0 on uv](uv-rust-1.78.png)

  <Caption>[Improvement on astral-sh/uv](https://codspeed.io/astral-sh/uv/branches/samypr100:rust-update-178)</Caption>
</Figure>

### Takeaways

Making sure our data structures have **a proper memory alignment** does
**improves performance by reducing cache misses** and **making memory accesses
faster**. However, it comes with a **higher memory footprint** implied by the
padding added within our memory layouts.

These small performance changes can be hard and tedious to track, but this is
where we can **leverage performance testing continuously in CI environments**,
enabling informed optimization decisions.

[Here is the repository](https://github.com/art049/rust-align-128) containing
the code shown in this article and you can also find its
[CodSpeed performance dashboard here](https://codspeed.io/art049/rust-align-128).

### Resources

- [Rust 1.78.0 release blog](https://blog.rust-lang.org/2024/05/02/Rust-1.78.0.html#compatibility-notes)
- [u128/i128 layout update blog](https://blog.rust-lang.org/2024/03/30/i128-layout-update.html)
- [Memory Alignment and Performance](https://fylux.github.io/2017/07/11/Memory_Alignment/)
- [Size and Alignment docs in Rust Reference](https://doc.rust-lang.org/reference/type-layout.html#size-and-alignment)
- [type_layout crate](https://docs.rs/type-layout/latest/type_layout/)
- [Repository containing the code shown in the blog](https://github.com/art049/rust-align-128)
- [Getting started with Rust on CodSpeed](https://codspeed.io/docs/benchmarks/rust)



While unit tests have become a standard for ensuring that the code performs as
expected, they focus on checking the logic, not the performance. Yet,
**performance regressions can be just as dangerous as functional bugs, putting
your whole software at risk**. This is why performance should be checked early
while testing the code, the icing on the cake being to check it in CI
environments.

## Effortless performance testing with pytest

**Benchmarking** is a way of testing a block of code's performance. It is like a
test case but for performance. It will execute the code and measure how long it
takes to run.

Let's see how to implement that with `pytest`. First, install the
`pytest-codspeed` library to enable the benchmark marker and fixture:

```text
pip install pytest-codspeed
```

Then, you're ready to use the `@pytest.mark.benchmark` marker for measuring
performance. You can use it directly on a single test:

```python
import pytest # [!code ++]
@pytest.mark.benchmark # [!code ++]
def test_my_fn():
    inputs = gen_inputs()
    results = my_fn(inputs)
    assert results == "expected_result"
```

But you can also apply it at the module level by using the global `pytestmark`
variable, effectively enabling performance testing on all the tests contained
within it:

```python
import pytest
pytestmark = pytest.mark.benchmark

# The rest of the test cases are now performance tests as well
```

You can then run the performance tests locally to ensure that everything works:

```bash
pytest tests/ --codspeed
```

## Rely on your CI to avoid Spray and Pray

While being a good starting point, isolated runs do not work for long-term
performance tracking and the risk of missing important performance changes is
just too high. Regressions will surface when you less expect them and it's
essential to **automate these checks into your CI pipeline**. This ensures that
any performance degradation is caught automatically during the development cycle
and builds a history of your codebase performance.

Using the CodSpeed test runner helps a lot to make the measurement extremely
steady. Our runner relies on CPU simulation, enabling us to separate the noisy
neighbours (other VMs, workloads, users) from the precious workload you want to
measure.

A typical setup with the runner in GitHub Actions would be as simple as:

```yaml
- uses: CodSpeedHQ/action@v3
  with:
    run: pytest tests/ --codspeed
```

This setup not only runs your tests but also uploads the results to CodSpeed,
where you can track performance over time.

<Figure maxW="600px" mx="auto">

![Sample pull request report on CodSpeed](sample-pull-request-report.png)

  <Caption>Sample pull request report on CodSpeed</Caption>
</Figure>

## Measuring only what matters

Sometimes, you want more granularity in what is measured. For example, you do
not want to measure the time it took to generate the inputs for calling our
function or the assertions after getting the result; and instead **focus only on
the actual function call**.

We can modify the unit test:

```python
def test_my_fn(benchmark):
    inputs = gen_inputs()
    results = benchmark(my_fn, inputs)
    assert results == "expected_result"
```

This test uses the `benchmark` fixture to only measure the execution time of
`my_fn`. The fixture makes it easy to focus on what matters—how long it takes
your function to execute under test conditions.

<Alert>
  Using the `benchmark` fixture will automatically mark the test as a benchmark,
  without having to use the `pytest.mark.benchmark` marker.
</Alert>

## Optimizing hot paths with Differential Profiling

When you encounter a performance regression, your next mission(if you accept
it🤵) is often to investigate and find what/who/when/why was this issue
introduced in the first place.

This is where differential profiling comes in handy. This allows to compare two
execution profiles to find exactly what changed between two separate
measurements.

The good news is **CodSpeed automatically profiles your benchmark's code while
measuring performance**. So if you spot a regression, you'll have all the data
to investigate:

<Figure>
  <FlameGraph height={150} flameGraphsPath="parse-pr-complex" />
  <Caption>
    Sample flame graph with regressions, improvements, and added code
  </Caption>
</Figure>

## Takeaways

Integrating performance testing into your development process with tools like
`pytest` and CodSpeed fosters a culture of continuous improvement. It ensures
that performance considerations are never an afterthought but a key component of
your software development lifecycle from the ground up.

To see CodSpeed in action, you can check out open-source repositories using the
tool in the [explore page](https://codspeed.io/explore). A lot of them are
actually using the `pytest` integration we just talked about, like
[pydantic](https://codspeed.io/pydantic/pydantic-core) and
[polars](https://codspeed.io/pola-rs/polars)

Last but certainly not least, shout out to
[patrick91](https://twitter.com/patrick91), who pioneered this use case and
whose contributions have made it significantly easier for developers to
incorporate benchmarking into their existing unit tests.

## Resources

- [pytest-codspeed](https://github.com/CodSpeedHQ/pytest-codspeed): the plugin
  for `pytest`
- [@codspeed/action](https://github.com/CodSpeedHQ/action): the GitHub Action to
  run benchmarks and generate flame graphs
- [@codspeed/runner](https://github.com/CodSpeedHQ/runner): the generic CodSpeed
  runner working with various CI providers
- [CodSpeed Docs: Python Integration](https://codspeed.io/docs/benchmarks/python)
- [Pinpoint performance regressions with CI-Integrated differential profiling](https://codspeed.io/blog/pinpoint-performance-regressions-with-ci-integrated-differential-profiling)



One of the easiest ways to track performance regressions in your JavaScript
applications is to use Vitest's `bench` API. It lets you write benchmarks as
easily as unit tests:

```ts
import { bench, describe, test } from "vitest";

describe("fibonacci", () => {
  test("should return 55 when called with 10", () => {
    expect(fibonacci(10)).toBe(55);
  });

  bench("fibo 10", () => {
    fibonacci(10);
  });
});
```

Then you can run the benchmarks with `vitest bench`:

```bash
pnpm vitest bench
```

The problem that we now face is that the results are **inconsistent between
runs**. Running the command multiple times in a CI/CD runner or on your machine
will yield different results. Thus it will be hard to spot performance
regressions with confidence.

This is where using CodSpeed comes in handy. CodSpeed provides a way to **run
benchmarks in a consistent manner**, and thus **detect performance regressions
with confidence**.

We are excited to announce that CodSpeed now has first-class support for Vitest!

## How to use Vitest with CodSpeed

Install the `@codspeed/vitest-plugin` package and `vitest` as dev dependencies:

```bash
pnpm add -D @codspeed/vitest-plugin vitest
```

<Alert>
  The CodSpeed plugin is only compatible with
  [vitest@1.0.0](https://www.npmjs.com/package/vitest/v/1.0.0) and above.
</Alert>

Create or update the `vitest.config.ts` file at the root of your project:

```ts title="vitest.config.ts"
import codspeedPlugin from "@codspeed/vitest-plugin";
import { defineConfig } from "vitest/config";

export default defineConfig({
  plugins: [codspeedPlugin()],
  // ...
});
```

<Alert>
  If you are on a project using `vite` and `vitest`, you can also add the plugin
  to the `vite.config.ts` file.
</Alert>

Let's create a fibonacci function and create some benchmarks on it. Create a new
file `src/fibonacci.ts`:

```ts
export function fibonacci(n: number): number {
  if (n <= 1) return n;
  return fibonacci(n - 1) + fibonacci(n - 2);
}
```

Create a new file `src/fibonacci.bench.ts`:

```ts
import { bench, describe } from "vitest";
import { fibonacci } from "./fibonacci";

describe("fibonacci", () => {
  bench("fibo 10", () => {
    fibonacci(10);
  });

  bench("fibo 15", () => {
    fibonacci(15);
  });
});
```

Now we can run the benchmarks with `vitest bench`:

```bash
$ vitest bench

[CodSpeed] bench detected but no instrumentation found, falling back to default vitest runner
 RUN  v1.0.3 ...

 ✓ src/fibonacci.bench.ts (2) 1521ms
   ✓ fibonacci (2) 1520ms
     name                hz     min     max    mean     p75     p99    p995    p999     rme  samples
   · fibo 10   1,760,687.42  0.0005  1.0279  0.0006  0.0006  0.0007  0.0007  0.0016  ±0.50%   880344   fastest
   · fibo 15     173,932.15  0.0052  0.0594  0.0057  0.0059  0.0064  0.0070  0.0147  ±0.08%    86967


 BENCH  Summary

  fibo 10 - src/fibonacci.bench.ts > fibonacci
    10.12x faster than fibo 15
```

We can see that CodSpeed has been detected and that the benchmarks have been
run. For the moment we can see the default output of Vitest's `bench` API.

## Running the benchmarks in your CI

To generate performance reports, you need to run the benchmarks in your CI. This
allows CodSpeed to detect the CI environment and properly configure the
instrumented environment.

### Enable the repository on CodSpeed

The first step is to log in to CodSpeed and import the repository you want to
track. You can find more information about this in the
[CodSpeed documentation](https://codspeed.io/docs/integrations/providers/overview).

After installing the GitHub app and importing the repository in CodSpeed, you
can proceed to the next step and create a benchmarks workflow.

### Add a CodSpeed GitHub Actions workflow

Open a Pull Request on your repo with the previous changes, and the following
GitHub Actions workflow `.github/workflows/codspeed.yml`. It will run the
benchmarks and report the results to CodSpeed on every push to the `main` branch
and every pull request:

```yaml title=".github/workflows/codspeed.yml"
name: CodSpeed

on:
  push:
    branches:
      - "main"
  pull_request:

permissions: # optional for public repositories
  contents: read # required for actions/checkout
  id-token: write # required for OIDC authentication with CodSpeed

jobs:
  benchmarks:
    name: Run benchmarks
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: pnpm/action-setup@v2
      - uses: actions/setup-node@v3
        with:
          cache: pnpm
      - name: Install dependencies
        run: pnpm install
      - name: Run benchmarks
        uses: CodSpeedHQ/action@v3
        with:
          run: pnpm vitest bench
```

If everything went well, the workflow should run and after it finishes you
should see a comment from the CodSpeed GitHub app on the Pull Request with the
run report.

### Visualizing the benchmarks on the CodSpeed app

By clicking on the CodSpeed comment in the Pull Request, you will be redirected
to the CodSpeed report of the PR. You will see a list of the benchmarks, and
when clicking on the `fibo 10` benchmark its flame graph will appear:

<Figure>
  <FlameGraph height={150} flameGraphsPath="fibo10-vitest" />
  <Caption>
    Flame Graphs component taken from the CodSpeed app. It is fully interactive,
    don't hesitate to play with the buttons!
  </Caption>
</Figure>

And that's it for the installation of CodSpeed with Vitest, now we can explore a
bit more what CodSpeed offers.

## Visualizing a performance change

One of the main features of CodSpeed is to detect and visualize the performance
changes of a pull request.

Let's now say that we are on a more complex project where we have a `parsePr`
function that parses a GitHub pull request. We created a pull request with some
performance improvements, and the following is in the CodSpeed report:

<PerfDiff
  base={1.5e-3}
  head={1.2e-3}
  name="parsePr::short body"
  description="Improve parsing performance"
/>
<Figure>
  <FlameGraph height={150} flameGraphsPath="parse-pr-vitest" />
  <Caption>
    The Diff graph shows the resulting graph of the difference between the head
    and base runs, allowing us to pinpoint where the performance changes are
    inside the code
  </Caption>
</Figure>

We observe a 25% improvement in the overall benchmark, driven by enhancements in
the `parsePr` and `parseIssueFixed` functions. Although the `logMetrics`
function has experienced regression, its impact on the overall benchmark is less
pronounced compared to the observed improvements. Additionally, a new function,
`modifyTitle`, is introduced and denoted by the distinctive blue color.

Thanks to the differential flame graph, it is now really easy to understand at a
glance what code changes impact the performance.

## Wrapping up

And there you have it—CodSpeed now supports Vitest to make performance testing a
breeze for your JavaScript and TypeScript projects! With an easy-to-use bench
API and the magic touch of CodSpeed's consistency, tracking regressions is as
simple as adding a unit test.

Get ready to supercharge your CI with performance reports, flame graphs, and
more!

---

If you enjoyed this article, you can follow us on
[Twitter](https://twitter.com/codspeedhq) to get notified when we publish new
articles and product updates.

## Resources

- [Vitest integration](https://codspeed.io/docs/benchmarks/nodejs/vitest) in the
  CodSpeed documentation
- [@codspeed/vitest-plugin](https://github.com/CodSpeedHQ/codspeed-node/tree/main/packages/vitest-plugin)
  on GitHub
- [Trace Generation](https://codspeed.io/docs/instruments/cpu/trace-generation)
  in the CodSpeed documentation
- [Vitest bench API](https://vitest.dev/api/#bench)



Flame Graphs are a visualization tool that helps you understand how your
software is performing. They display the call stack of a program and the time
spent in each function. They are a powerful tool to quickly identify performance
bottlenecks.

Differential Flame Graphs combine two flame graphs to highlight the differences
between them. They allow you to easily spot performance regressions and
improvements and gain invaluable insights into your software's performance.

Below you can find an example of normal and differential flame graphs following
a change in a codebase. At a glance, we can understand where the code has become
slower. Here it is the `parse_issue_fixed` function as it has the brightest red
color.

<Figure>
  <FlameGraph height={150} flameGraphsPath="parse-pr" />
  <Caption>
    Flame Graphs component taken from the CodSpeed app. It is fully interactive,
    don't hesitate to play with the buttons!
  </Caption>
</Figure>

Let's explore how to read flame graphs and how CodSpeed automates flame graph
generation in your CI pipeline.

<Alert>
  CodSpeed supports Rust, Node.js, Python and generates flame graphs out of the
  box. More languages are coming soon!
</Alert>

## Spotting a performance regression

Let's dive into the previous example.

We have a function `parse_issue_fixed` that parses a GitHub pull request body
and extracts the issue number that it fixes. Given `body = "fixes #123"`, the
function returns `123`. Here `body` can be a multiline string, and the issue
number can be anywhere in the string.

<Alert>
  On GitHub, adding the string `fixes #123` in a pull request body will
  automatically close the issue #123 when the pull request is merged.
</Alert>

We notice that the code is quite long and not easily understandable. We refactor
it to use a regular expression instead. This gives us the following diff:

```python
import re # [!code ++]
# [!code ++]
FIXES_REGEX = re.compile(r"fixes #(\d+)") # [!code ++]

def parse_issue_fixed(body: str) -> int | None:
    prefix = "fixes #" # [!code --]
    index = body.find(prefix) # [!code --]
    if index == -1: # [!code --]
        return None # [!code --]
# [!code --]
    end = start # [!code --]
    while end < len(body) and body[end].isdigit(): # [!code --]
        end += 1 # [!code --]
    return int(body[start:end]) # [!code --]
    match = FIXES_REGEX.search(body) # [!code ++]
    return int(match.group(1)) if match else None # [!code ++]
```

Great, we have a much shorter and more readable function. But did we introduce a
performance regression?

We already have a benchmark in place for the function `parse_pr`, a higher-level
function that parses a GitHub pull request body and calls multiple functions to
retrieve information, including `parse_issue_fixed`. The input is a multiline
string of approximately 50kB. We chose a large input to better understand the
performance characteristics of the function and make sure the function performs
well when parsing large pull request bodies.

Let's check out the flame graphs for the benchmark to analyze the performance
impact of our change. The **Base** flame graph is the one before the change, the
**Head** flame graph is the one after the change, and finally, the **Diff**
flame graph is the difference between the two.

<PerfDiff
  base={1.2e-3}
  head={1.525e-3}
  name="test_parse_pr"
  description="Refactor with regular expression"
/>
<div style={{ marginBottom: "16px" }}>
  <FlameGraph height={150} flameGraphsPath="parse-pr" />
</div>

Here we can see that the `parse_issue_fixed` function is bright red, thus
denoting that it was slower after the change and had the biggest performance
impact on the benchmark. When hovering over the `parse_issue_fixed` frame, we
can see that the performance impact is a _-21.5% regression_ on the overall
benchmark.

So in our use case, using a regular expression will result in slower executions
compared to using `str.find`.

So yes, our change for the sake of readability introduced a performance
regression. We can now make an informed decision on whether we want to keep the
change or not.

Now that we understand how a differential flame graph works and how useful it
is, let's explore how to generate one.

## Generating flame graphs manually is unreliable and time-consuming

The first thing to note is that manually generated flame graphs on a dev machine
are **not consistent**.

Running the same script multiple times with the same input will result in
different flame graphs. Indeed, other processes running on the machine will
impact the execution time of the script and its different functions. This make
it hard to spot performance regressions with confidence.

The second thing to note is that generating flame graphs manually is
**tedious**.

The first tool to generate flame graphs is the eponym FlameGraph by Brendan
Gregg written in Perl. Since then, many other tools have been created in and for
other languages. The general steps are the same:

- Run your script with a profiler and output the profile in a file
- Run the tool on the profile to generate a flame graph svg
- Open the generated svg in your browser and explore the data

This is already a quite heavy process: you have to create a script to isolate
the part of your program you want to generate a flame graph for, then run a
bunch of commands and finally open a file in your browser.

This is why we decided to automate the generation of flame graphs in your CI
pipeline with CodSpeed, and find a way to make sure that the flame graphs are
consistent.

## Automating flame graphs with CodSpeed

Since we already had wrappers of benchmarking libraries in different languages,
we decided to augment them with profiling capabilities.

### Steps to follow to get flame graphs in your CI pipeline

Write benchmarks with one of our wrappers in Rust, Node.js, or Python.

For example, in Python, use our `pytest` extension `pytest-codspeed` (which API
is compatible with `pytest-benchmark`) and add a test marked as a benchmark:

```python
import pytest
from parse_pr import parse_pr

@pytest.mark.benchmark
def test_parse_pr():
    parse_pr(body=very_long_body_string, pr_number=126, title="Refactor some code")
```

Run the benchmarks in the `@codspeed/action` GitHub Action:

```yml
- name: Run benchmarks
  uses: CodSpeedHQ/action@v3
  with:
    run: pytest tests/ --codspeed
```

The action will run the benchmarks on a "virtualized" CPU with Valgrind. Instead
of measuring the execution time, it measures the number of CPU cycles and memory
accesses. Each benchmark is only ran once and it is enough to get **consistent
data**.

This data is then sent to CodSpeed's servers and flame graphs are generated. A
comment is added to the GitHub Pull Request, with a link to the CodSpeed app,
where you can browse the benchmarks and their flame graphs.

And voilà! You now have flame graphs in your CI pipeline, with the benefits of
consistency and automation. Moreover, now that are using CodSpeed, **performance
regressions will automatically be detected and reported** in your pull requests
🎉

<Alert>
  Head out to our [documentation](https://docs.codspeed.io) to view integrations
  for the different languages.
</Alert>

## Going further

Flame graphs are not only useful for spotting performance regressions but they
can also be used to understand the performance impact of a change. Being an
improvement or new calls to a function.

<Figure>
  <FlameGraph height={150} flameGraphsPath="parse-pr-complex" />
  <Caption>
    A more complex flame graph with regressions, improvements, and added code
  </Caption>
</Figure>

If you enjoyed this article, you can follow us on
[Twitter](https://twitter.com/codspeedhq) to get notified when we publish new
articles.

## Resources

- [Trace Generation](https://codspeed.io/docs/instruments/cpu/trace-generation)
  in the CodSpeed documentation
- [pytest-codspeed](https://github.com/CodSpeedHQ/pytest-codspeed), plugin for
  pytest
- [@codspeed/action](https://github.com/CodSpeedHQ/action), GitHub Action to run
  benchmarks and generate flame graphs
- [Flame Graphs](https://www.brendangregg.com/flamegraphs.html) by Brendan Gregg
- [Differential Flame Graphs](https://www.brendangregg.com/blog/2014-11-09/differential-flame-graphs.html)
  by Brendan Gregg
- [Linking a pull request to an issue](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue)
- [Valgrind](https://valgrind.org/)
- [pytest-benchmark](https://github.com/ionelmc/pytest-benchmark)



During the last advent of code, while starting with Rust, I discovered that the
Rust standard library was missing the usual `div_mod`(a.k.a. `div_rem`) function
that would simultaneously return the quotient and remainder. I got confused
since I already used the operator in C++ with
[std::div](https://en.cppreference.com/w/cpp/numeric/math/div) or Python with
[divmod](https://docs.python.org/3/library/functions.html#divmod). It seemed a
basic way of improving performance since both values would come out with the
same `DIV` assembly instruction, nicely avoiding extra CPU cycles.

Looking it up, I found
[an existing RFC](https://github.com/rust-lang/rfcs/issues/1887) created in
2017, but it hasn't received much attention since.

So, do we need a `div_rem` implementation in the standard library? I decided to
investigate.

## A naive approach

Let's naively implement the function:

```rust
pub fn naive_div_rem(a: u32, b: u32) -> (u32, u32) {
    (a / b, a % b)
}
```

To dissect what is executed we're gonna analyze the compilation steps.

<Figure>

![Rust Compilation Chain](compilation-chain.excalidraw.png)

  <Caption>The compilation chain of our Rust program</Caption>
</Figure>

First, let's have a look at the LLVM Intermediate Representation(IR). If you're
not familiar with it, is an extremely low-level programming language close to
assembly but without being tied to specific hardware. It's extremely useful for
compiler writers since it's a common language that can be used to target
multiple architectures. It's also a great tool to understand what the compiler
is doing and how it "interpret" our code.

For the previous code, here is the annotated LLVM IR we can obtain:

```llvm
; Define a function called 'example::naive_div_rem
define { i32, i32 } @example::naive_div_rem(i32 %a, i32 %b) unnamed_addr {
start:
  %_0 = alloca { i32, i32 }, align 4; Allocate memory for our return tuple
  %_4 = icmp eq i32 %b, 0; Check if divisor %b is zero
  br i1 %_4, label %panic, label %bb1; If %b is zero, panic, else continue
bb1:
  %_3 = udiv i32 %a, %b; Perform unsigned division(only get the quotient)
  %_6 = icmp eq i32 %b, 0; Check again for the divisor being zero (for the modulo)
  br i1 %_6, label %panic1, label %bb2; If %b is zero, panic, else continue
bb2:
  %_5 = urem i32 %a, %b; Compute the unsigned remainder
  ; ...
  ; Stores quotient and the remainder in a tuple, load them back,
  ; and constructs a return tuple with the values.
  ; ...
  ret { i32, i32 } %8
; ... omitted panic handlers ...
}
```

We can witness that this code contains redundant checks for the division and
modulo by 0. Also, we can note two operations to get our quotient and remainder:
`udiv` and `urem`. This is expected at this stage since LLVM doesn't have an
instruction combining them but hopefully, the compiler will be able to optimize
this while generating the assembly.

Let's have a look at the next compilation step so we can dive into the assembly
code generated from the IR(`x86_64` with optimizations enabled):

```asm
example::naive_div_rem:
    test    esi, esi  ; Test if esi (divisor) is zero
    je      .LBB0_2   ; Jump to panic block if esi is zero
    mov     eax, edi  ; Move the dividend into eax
    xor     edx, edx  ; Zero out edx to prepare for the division
    div     esi       ; Unsigned division, quotient goes into eax, remainder into edx
    ret
.LBB0_2:
    ; ... "attempt to divide by zero" panic handler ...
```

And voila! We can see that the compiler was able to optimize our code and fused
the two operations into a single `div` instruction. So having a dedicated
implementation wouldn't be that helpful for performance while using `rustc`
since this optimization should almost always be done by the IR compiler.

You might have noticed quite a few things in the assembly code, let's have a
look at them by creating our own assembly version.

## An inline assembly version

```rust
use std::arch::asm;

pub fn asm_div_rem(a: u32, b: u32) -> (u32, u32) {
    let mut tmp: u32 = a;
    let mut remainder: u32 = 0;
    unsafe {
        asm!(
            "div {divisor}",
            inout("eax") tmp,
            inout("edx") remainder,
            divisor = in(reg) b,
            options(pure, nomem, nostack),
        );
    }
    (tmp, remainder)
}
```

A few things to note in this implementation:

- The `div` instruction divides the `eax` register by its operand value. It will
  store the quotient in `eax` and the remainder in `edx`.
- `tmp` holds the dividend before our assembly is executed. It will contain the
  quotient after. This is because its value is tied to the `eax` register and it
  will be used both for input and output thanks to the `inout` operand.
- `remainder` is also passed with the `inout` operand, allowing us to clear the
  `edx` register before the operation is done since `remainder` is 0 before the
  assembly is executed. At the end, the register will hold the remainder value
  that we return in the output tuple.
- The divisor is passed through any general purpose register as a simple input
  value that we pass directly to the `div` instruction.
- Finally, we pass a few flags that will help the compiler to know how our
  inline assembly will behave with `options`. Here: our block has no side
  effect, we're neither reading nor writing to the memory and we don't push
  anything to the stack.

Compiling this, we can see that it ends up a bit more concise than the previous
compiled assembly we built:

```asm
example::asm_div_rem:
    mov     eax, edi ; Move the value of the dividend from EDI to EAX.
    xor     edx, edx ; Zero out EDX to prepare for the division.
    div     rsi      ; Signed divide the dividend in EAX by the value in RSI('b').
    ret              ; Return from the function. quotient -> EAX, remainder -> EDX.
```

It should be faster but we're cheating a bit on rust here since we're not taking
into account the panic check that was done in the previous version.

<PerfDiff
  base={32.7e-9}
  head={30.5e-9}
  name="div_rem"
  description="Switching to an inline assembly version"
/>

It's faster! The overhead of those panic checks is not that dramatic in the end.
Branch prediction is probably contributing a lot to this. It was eluded in the
first IR snippets but in there, the rust compiler hints the rest of the pipeline
that those checks are likely to succeed. The compiler's ability to optimize and
hint that these checks are more likely to pass than fail can lead to
surprisingly efficient code, even when it appears to be doing "much more" than
the assembly version.

## The cost of panic handling: an unchecked version

Now that we built our unchecked version in assembly, we can get quite close to
it by using unchecked div and rem from `std::intrisics`:

```rust
#![feature(core_intrinsics)]
pub unsafe fn unchecked_naive_div_rem(a: u32, b: u32) -> (u32, u32) {
    let quotient = std::intrinsics::unchecked_div(a, b);
    let remainder = std::intrinsics::unchecked_rem(a, b);
    (quotient, remainder)
}
```

And its intermediate representation:

```llvm
; Define a function named 'example::unchecked_naive_div_rem' that takes two arguments
define { i32, i32 } @example::unchecked_naive_div_rem(i32 noundef %a, i32 noundef %b) unnamed_addr {
start:
  %_0 = alloca { i32, i32 }, align 4; Allocate memory for a tuple to store the quotient and remainder
  %quotient = udiv i32 %a, %b; Perform unsigned division of %a by %b
  %remainder = urem i32 %a, %b; Compute the unsigned remainder of %a / %b
  ; ...
  ; Stores quotient and remainder in a tuple, loads them back, and constructs a return tuple with the values.
  ; ...
  ret { i32, i32 } %1
}
```

As you can see this IR doesn't contain anything about overflow checks and the
performance should be close to or even better than our assembly version.

<PerfDiff
  base={30.5e-9}
  head={30.5e-9}
  name="div_rem"
  description="Switching to the unchecked version"
/>

That's exactly the same time our assembly version takes. So let's have a look at
the assembly:

```asm
example::unchecked_naive_div_rem:
    mov     eax, edi
    xor     edx, edx
    div     esi
    ret
```

And it's the same as our assembly version! So bothering writing our assembly
version is not really worth it in the end.

To conclude, there is no performance drawback of not having a single `div_rem`
function since the compiler optimizations will take care of this burden for us.
It might seem odd to write those two operations but unless very specific
optimization killer cases they will be fused as a single assembly instruction.

If you enjoyed this article, you can follow us on
[Twitter](https://twitter.com/codspeedhq) to get notified when we publish new
articles.

### Resources

- [Compiler Explorer](https://godbolt.org/): an amazing tool to visualize the
  assembly and IR generated by the compiler
- [Understanding LLVM IR](https://mukulrathi.com/create-your-own-programming-language/llvm-ir-cpp-api-tutorial/#understanding-llvm-ir)
- [The faker's guide to reading (x86) assembly](https://www.timdbg.com/posts/fakers-guide-to-assembly/)
- [Guide to x86 Assembly](https://www.cs.virginia.edu/~evans/cs216/guides/x86.html)



### Why does Continuous Performance analysis matter?

**Performance is the cornerstone of many problems in software development.**
Application reactivity, infrastructure costs, energy consumption, and carbon
footprint all depend on the performance of the underlying software. **By
performance, we often mean execution speed** but more generally, it refers to
the efficiency or throughput of a system.

Today, most of the performance monitoring is done through **Application
Performance Monitoring** **solutions**(often referred to as APM), provided by
companies such as Datadog, Sentry, Blackfire, and many others. These platforms
bring **several interesting insights about the production environment's
health**: client-side UX monitoring, endpoint latency checks, and even
continuous production profiling.

However, these solutions are monitoring tools; **they are built to check if
everything is okay in production,** and not really to run experiments. They need
real users serving as guinea pigs to experience poor performance to be able to
report anomalies.

<Figure>

![Performance feedback in the **current** Software Development Life-Cycle](SDLC_Traditional.excalidraw.png)

  <Caption>Performance feedback in the <b>current</b> Software Development Life-Cycle</Caption>
</Figure>

So as a developer, to understand the performance impact of my changes, I need to
wait for end users to try out my changes in production!? Then, if something is
wrong, maybe I’ll try to improve it, someday…

In an ideal world, **performance checks should be included way earlier in the
development lifecycle**; just as an additional testing flavor, **nurturing
continuous improvement**.

<Figure>

![Performance feedback in an **ideal** Software Development Life-Cycle](SDLC_Ideal.excalidraw.png)

  <Caption>
    Performance feedback in an <b>ideal</b> Software Development Life-Cycle
  </Caption>
</Figure>

This much shorter feedback loop would **provide visibility with consistent
metrics to the teams while they are building** and not once everything is
already shipped in production environments. Besides, **guesstimating performance
is hard and often plain wrong**. Accurate performance reports help in those
cases and can serve as an educational tool for software developers.

### Building a consistent performance metric

Measuring software performance in various environments to gather reproducible
results is hard. The most basic metric we can think of is to **measure the
execution time but just changing the hardware will produce completely different
data points**. Running a program on a toaster will be significantly slower than
running it on the latest generations of cloud instances. Joking aside,
**merely** **using the same machine at different times will produce different
results** because other unpredictable background tasks are eating up the CPU
time.

One obvious solution to bring consistency in the results is to run the program
in a controlled cloud environment where background running processes are very
limited. Despite a significant improvement in the quality of the results, **this
doesn’t give repeatable measurements either because of the
[noisy neighbor issue](https://www.techtarget.com/searchcloudcomputing/definition/noisy-neighbor-cloud-computing-performance)**.
Basically, physical cloud machines are shared among customers by splitting them
into multiple Virtual Machines. At a software level, those are perfectly
isolated but in the end, they still share some common hardware and **isolation
can’t be perfect**(eg. memory, high-level CPU caches, network interfaces). Since
it’s not possible to predict the workload running along a measurement task,
finding the “truth” in this noise becomes a statistical challenge, and
**building an extremely accurate metric is nearly impossible**.

<Figure>
  <Box sx={{ "& img": { borderRadius: "lg", border: "1px" } }}>
    ![Time measurement for a Fibonacci sequence computation(Python runs from
    GitHub Action)](TimeMeasurementFibo.png)
  </Box>
  <Caption>
    Time measurement for a Fibonacci sequence computation(Python runs from
    GitHub Action)
  </Caption>
</Figure>

When it comes to consistency, working with execution time measurement seems
doomed to failure. So what if we instead decided to dissect exactly what a
virtual machine does to run our program? In this case, running the same
measurement again and again on various machines produces the exact same results
since **the hosted virtual machine always starts with a predefined state and
emulates the same hardware**. We don’t mind the noisy neighbor issue either
because **whatever is running along our measurement doesn’t change the
instructions executed to run our program** in its sandboxed environment.
Furthermore, based on those micro execution steps, it is possible to aggregate
**a time equivalent metric that will be consistent, accurate (detect less than
1% performance changes), and hardware agnostic**: **this is how CodSpeed
works!**

<Figure>
  <Box sx={{ "& img": { borderRadius: "lg", border: "1px" } }}>
    ![CodSpeed Measurement for a Fibonacci sequence computation(Python runs from
    GitHub Action)](CodSpeedMeasurementFibo.png)
  </Box>
  <Caption>
    CodSpeed Measurement for a Fibonacci sequence computation(Python runs from
    GitHub Action)
  </Caption>
</Figure>

### Shifting left with CodSpeed

CodSpeed brings this consistent measurement to the Continuous Integration
environments, enabling performance checks to be included in the software
development lifecycle as early as possible.

On every new feature, the performance is measured and reported directly in the
repository provider as a Pull Request comment. Optionally, status checks can
also be enabled to enforce performance requirements to be satisfied before
merging the new delivery.

<Figure>![Github Integration.png](Github_Integration.png)</Figure>

Furthermore, there is a whole platform available. Breaking down the performance
per branches, commits, or benchmarks and giving you an overview of the upcoming
performance changes. We will give you more details about this in a future blog
post!

But the beta is already opened! 🎉

Here are some of the open-source repositories already using CodSpeed:

- [`pydantic-core`](https://github.com/pydantic/pydantic-core): The core
  validation logic for pydantic, a Python data parsing and validation library.
- [`pathfinding`](https://github.com/samueltardieu/pathfinding): A pathfinding
  library for Rust.
- [`swarmion`](https://github.com/swarmion/swarmion): A set of tools to build
  and deploy type-safe serverless microservices with Typescript.

If you want to learn more, [check out the product](https://codspeed.io) and
[follow us on Twitter](https://twitter.com/codspeedhq) to stay updated!


The CodSpeed Blog

How to Benchmark Rust Code?

How to Benchmark Python Code?

Benchmarking a Go Gin API

Benchmarks in CI: Escaping the Cloud Chaos

State of Python 3.13 Performance: Free-Threading

Rust 1.78: Performance Impact of the 128-bit Memory Alignment Fix

One pytest Marker to Track the Performance of Your Tests

Using Vitest bench to track performance regressions in your CI

Pinpoint performance regressions with CI-Integrated differential profiling

Why Rust doesn't need a standard div_rem: An LLVM tale

Introducing CodSpeed: Continuous Benchmarking

It's Not a Regression
If It's Not in Production

The CodSpeed Blog

It's Not a Regression.css-x3npl{display:inline;background-image:linear-gradient(180deg, #FDB982 0%, #FC6E39 62.5%);color:transparent;-webkit-background-clip:text;background-clip:text;}If It's Not in Production

It's Not a Regression
If It's Not in Production