Skip to content

Commit 4f2a99e

Browse files
authored
Added documentation for how resource limits work and how they are configured (#129)
1 parent 8224abb commit 4f2a99e

File tree

5 files changed

+105
-2
lines changed

5 files changed

+105
-2
lines changed
Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
---
2+
title: "Host Resource Reservations"
3+
description: "Reserving CPU and Memory to ensure GMT has sufficient compute power to orchestrate measurement"
4+
date: 2025-12-15T16:20:15+10:00
5+
weight: 1005
6+
toc: false
7+
---
8+
9+
GMT runs on the host system orchestrating containers. To have this process running smoothly.
10+
GMT reserves a share of the available CPUs and the available Memory for the host system.
11+
12+
This ensures:
13+
14+
- Host does not OOM and measurement fails
15+
- Host does not run into CPU scarcity and metric providers can safely capture all metrics
16+
17+
The setting is configured *per machine* in the `config.yml`
18+
19+
```yml
20+
# config.yml
21+
machine:
22+
...
23+
host_reserved_cpus: 1
24+
host_reserved_memory: 1073741824 # 1 GiB
25+
26+
```
27+
28+
#### Specifications
29+
30+
- `host_reserved_cpus` **[integer]** (Default 1): Value between 1 and CPU_MAX. CPU_MAX is the amount of available compute threads on the system.
31+
- `host_reserved_memory` **[integer]** (Default 0): Value between 0 and MEMORY_MAX. MEMORY_MAX is the amount of available physical memory on the system.
32+
33+
## How to choose good values
34+
35+
### CPU
36+
37+
GMT will always reserve one core. So setting a value for `host_reserved_cpus` < 1 will fail. Typically GMT does not need more than one core, so you only should set this value to a higher value if your system has SMT / Hyper-Threading enabled or your host does a lot of other operations that should run undisturbed by the orchestrated containers.
38+
39+
For the former you should reserve as much cores as the SMT increases the core count. Typically SMT doubles a physical core to two hyper-cores. Thus you should set the reservation to two cores.
40+
41+
For the latter however though this indicates that the machine is a *noisy* measurement machine and we currently do not see any valid case for such a measurement machine ... write us an email if you have a one :)
42+
43+
### Memory
44+
45+
The default value for memory reservations in development is 0. This means Memory can be overcommited. In development this is fine as OOM situations can be resolved manually and it makes development quicker as containers have more memory to work with.
46+
47+
In a cluster setup the value should be derived as follows:
48+
49+
- Start the GMT `client.py`
50+
- Wait 30 seconds until the python process has reserved all memory it needs
51+
- Check `free -m` and read the current *used* memory.
52+
- Add 300 MB and round up to the neareast half GiB (e.g 2.1 GiB (+300 MB) used memory rounds up 2.5 GiB)

content/en/docs/installation/minimum-system-requirements.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,8 @@ toc: true
1111
At least an SSE2-compatible processor is required;
1212
For macOS a 64bit-compatible Intel processor (Core2Duo or newer) or an M1 ARM or newer is required.
1313

14+
The CPU must have at least **two physical threads** available (in case of SMT / Hyperthreading it should be 4 threads).
15+
1416
### Memory
1517

1618
- 1 GB

content/en/docs/measuring/configuration.md

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -109,9 +109,13 @@ For the rest please see [installation →]({{< relref "/docs/cluster/installatio
109109

110110
### machine
111111

112-
If you run locally nothing needs to be configured here. But if you run a *cluster* you must set the base temperature values for the accuracy control to work
112+
If you run locally nothing needs to be configured here.
113113

114-
Please see [cluster installation →]({{< relref "/docs/cluster/installation" >}}) and [accuracy control →]({{< relref "/docs/cluster/accuracy-control" >}})
114+
But if you run a *cluster* you must set the base temperature values for the accuracy control to work as well as configure the host reservation for CPU and memory.
115+
116+
Please see [cluster installation →]({{< relref "/docs/cluster/installation" >}}), [accuracy control →]({{< relref "/docs/cluster/accuracy-control" >}}) and [host resource reservations]({{< relref "/docs/cluster/host-resource-reservations" >}}).
117+
118+
Also see [Resource Limits]({{< relref "/docs/measuring/resource-limits" >}}) to better understand how GMT enforces resource limits on its orchestrated containers.
115119

116120
### measurement
117121

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
---
2+
title : "Resource Limits"
3+
description: ""
4+
date: 2025-12-15T16:48:45+10:00
5+
weight: 426
6+
---
7+
8+
Resource limits are an essential part of any in production deployed container.
9+
10+
GMT enables **and** enforces resource limits by:
11+
12+
- Using `Compose Specification` settings on containers like `mem_limit` and `cpus`
13+
- Auto-Assigning values to these two settings in case they are not set
14+
15+
## Understanding Auto-Assignment
16+
17+
When no resource limits are set GMT will determine how many available CPUs and how much memory is on the host sytem.
18+
19+
### CPUs
20+
21+
GMT will always reserve one core to have processing of the measurement and metric providers running smoothly. This means you cannot run GMT on a system with less than 2 cores.
22+
23+
After reserving one core GMT will assign the available cores *in full* to all other containers. Meaning that CPUs are not dedicated per container, but available to all containers in parallel and scheduling per CPU is done by the native OS scheduler.
24+
25+
If your machine has SMT / Hyper-Threading enabled or you feel you need to reserve more than one core for GMT check out [host resource reservations]({{< relref "/docs/cluster/host-resource-reservations" >}})
26+
27+
### Memory
28+
29+
Per default GMT will not reserve any memory from the host. This is fine for development but can lead to OOM situations in unattended modes like the [cluster mode]({{< relref "/docs/cluster" >}}).
30+
31+
If you feel you want to change that check out [host resource reservations]({{< relref "/docs/cluster/host-resource-reservations" >}}).
32+
33+
Once a value is set, or the value is left at 0, GMT will calculate the total memory on the host system and deduct the configured memory reservation.
34+
35+
Then all containers that have a manual `mem_limit` set will get their memory assigned.
36+
37+
The rest of the memory is then auto-assigned to the rest of the containers in the `usage_scenario.yml` evenly.
38+
39+
If during the process the memory gets exhausted because too much memory was requested manually through `mem_limit` GMT will error.
40+
41+
### Validating
42+
43+
You can see the auto applied values in the *Containers* Tab in the Dashboard
44+
45+
<center><img style="width: 600px;" src="/img/dashboard-containers-tab.webp" alt="Dashboard Container Tab for GMT Measurements"></center>
52.3 KB
Loading

0 commit comments

Comments
 (0)