Skip to content

[doc] Split peer and root CA for trusted certificates; Add purpose #6792

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
minglumlu wants to merge 3 commits into
base: master
Choose a base branch
from
Open

[doc] Split peer and root CA for trusted certificates; Add purpose #6792

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
08d29ff
[doc] Improvements on management of trusted certificates
minglumlu Dec 15, 2025
5a5e2a2
fixup! [doc] Improvements on management of trusted certificates
minglumlu Dec 16, 2025
cedf310
fixup! fixup! [doc] Improvements on management of trusted certificates
minglumlu Dec 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
146 changes: 146 additions & 0 deletions doc/content/design/trusted-certificates.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,146 @@
---
title: Trusted certificates for identity validation in TLS connections
layout: default
design_doc: true
revision: 1
status: draft
---

# Overview

In various use cases, TLS connections are established on the host on which XAPI runs.
When establishing a TLS connection, the peer identity needs to be validated.
This is done using either a root CA certificate to perform certificate chain validation, or a known peer certificate for validation with certificate pinning.
The root CA certificates and peer certificates involved in this process are referred to as trusted certificates.
When a trusted certificate is installed, the local endpoint can validate the peer identity during TLS connection establishment.
Certificate chain validation is a general-purpose, standards-based approach but requires additional steps, such as getting the peer's certificate signed by a CA.
In contrast, certificate pinning offers a quicker way to set up trust in some cases without the overhead of CA signing.
For example, when establishing a TLS connection, the peer endpoint presenting a self-signed server certificate, the local endpoint, after explicit user confirmation, can set up the trust by pinning the server certificate for this peer.
For subsequent connections, the local endpoint validates the peer against the pinned certificate.
This allows the use case to start in quicker and easier way without prior CA signing and without compromising security.

As the unified API for the whole system, XAPI also exposes interfaces for users to install and manage trusted certificates that are used by system components for different purposes.

The base design described in [pool-certificates.md](https://github.com/minglumlu/xen-api/blob/5d1ea1520825d502c57a90a02db476cd7d6a9132/doc/content/design/pool-certificates.md) defines the database, API, and trust store in the filesystem for managing trusted certificates.
This document introduces the following enhancements to that design:

* Explicit separation of root CA certificates and peer certificates:
In the base design, both certificate types share the same database schema, APIs, and are stored together in a single bundle file.
This makes it difficult to determine the appropriate validation approach based on the certificate type.
The improvement introduces a type value to separate root CA certificates and peer certificates explicitly.

* Add a "purpose" attribute for trusted certificates:
According to the base design, only certificates used for internal TLS connections among XAPI processes within a pool are stored separately.
All other trusted certificates are grouped in a single bundle, which may include certificates for multiple purposes.
By introducing a "purpose" attribute, certificates can be organized by their intended use, improving clarity and reducing ambiguity.

# Use Cases
* An XAPI client establishes a TLS connection to an XAPI service.
This case is outside the scope of trusted certificates managed by XAPI and is included here only for completeness.
* An XAPI process on one host initiates a TLS connection to an XAPI process on another host within the same pool.
This case is covered in the base design and is listed here for completeness.
* An XAPI process initiates a TLS connection to an external service, such as an appliance.
This case benefits from the improvements introduced in this design.
* A non-XAPI process (like licensing agent) running on a host managed by XAPI initiates a TLS connection to an external service, such as a License Server.
This case benefits from the improvements introduced in this design as well.


# Changes
## Database schema
The *Certificate* class in database is defined to represent general certificates, including trusted certificates.
One existing class field "type" supports the following enumeration values:
* "ca": trusted certificates including both root CA and peer.
* "host": identity certificate of a host for communication with entities outside the pool.
* "host_internal": identity certificate of a host for communication with other pool members.

Two improvements in this design:
* A new value "peer" is introduced in this design so that the existing "ca" now represents trusted root CA only.
The new "peer" will represent trusted peer certificates.

* A new enumeration type "purpose" is introduced to indicate the intended usage of a trusted certificate.
A new *Certificate* class field "purpose" (a set of values of enumeration type "purpose") will be added to represent all applicable purposes of a trusted certificate.
By default, this set is empty which corresponds to the existing "ca" certificates for general purpose.

## API

### pool.install_ca_certificate

This is an existing API to install a trusted certificate into the pool with its arguments being defined as:
* session (ref session_id): reference to a valid session;
* name (string): the name of the certificate;
* cert (string): the certificate in PEM format.

In this design, it is used to install root CA certificates only.
A new argument "purpose" is appended to specify the purposes of the trusted certificate to be installed. By default it is an empty set.
* session (ref session_id): reference to a valid session;
* name (string): the name of the certificate;
* cert (string): the certificate in PEM format;
* purpose (string list): the purposes of the certificate
Comment on lines +73 to +78
Copy link
Member

@psafont psafont Dec 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would propose to add a new API call, leaving the current one for backwards compatibility purposes:

pool.install_root_certificate session cert purposes
where

  • cert is the certificate in PEM format
  • purposes is a list of purpose enum values

This gets rid of the name parameter. name was needed when the call was intriduced because certificates were not database object. This meant there was no way to shop information about the certificate without downloading it back again, and that there was no way to decide its name in the filesystem.
Because now we can show the fingerprint as well as expiration data to identify certificates, and we can get a unique name that the user does not have to handle, I think removing the name is ideal.

Copy link
Member Author

@minglumlu minglumlu Dec 17, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's great that the existing hidden relation between name and the file name can be removed. But I think the name can be kept as a meaningful identify.
With this in consideration, new APIs pool.install_root_certificate and pool.install_peer_certificate would have the same argument list. It could be one API with an additional type for "root" or "peer", or just two APIs.
I have no preference. Either is fine to me.

Hi @robhoes May I please have your thoughts here?

I think about it again.
Comparing:

  • pool.install_ca_certificate
  • pool.install_root_certificate
  • pool.install_peer_certificate

with

  • pool.install_ca_certificate
  • pool.install_peer_certificate

The latter looks concise. And the appended purpose can keep the backwards compatibility.
But the hidden relation between name and file name can be removed.


### pool.install_peer_certificate
This is a new API introduced in this design with its arguments being defined as:
* session (ref session_id): reference to a valid session;
* name (string): the name of the certificate;
Copy link
Member

@psafont psafont Dec 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think users should be able to decide the filename of these certificates.

Suggested change
* name (string): the name of the certificate;

* cert (string): the certificate in PEM format;
* purpose (string list): the purposes of the certificate.
Copy link
Member

@psafont psafont Dec 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this be a list of DNS hotnames? for example [www.example.com; www1.example.com] This way it's very clear the relationship between the certificate and the server that is trusted.

Otherwise there should be some kind of general mechanism to be able to tell to which server names is each certificate related to.

Copy link
Member Author

@minglumlu minglumlu Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. Since it's the peer, it makes sense to know the particular server names.
The purpose is to make it possible to retrieve both trusted root CA and peer certificate when validating the peer identify for one purpose without getting other unrelated certificates involved.
How about just using the name to store the server names as this info is only used by API client, rather than XAPI?

Copy link
Member

@psafont psafont Dec 17, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that makes sense, we need to be careful with the description of the field name, since it has a different meaning depending on the type of certificate.
Also, this means that if a certificate is used for several domains, it seems like it needs to be duplicated.

Maybe we could do some sort of detection of shared certificate and change the name to have a concatenation of domain names: www.example.com,www1.example.com, but this seems fragile

Copy link
Member

@robhoes robhoes Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The distinction we are making here between "CA certificates" and "peer certificates" as separate types of trusted certificates is because they will be used a little differently when verifying the remote's cert:

  • CA: Verify the chain and check the hostname, using the stunnel options verifyChain and checkHost. The hostname needs to be verified, because there may be many certs for different hosts signed by the same CA.
  • Peer (pinning): Exact match of the certificate, using the stunnel option verifyPeer. The hostname does not need to be checked. This also makes it easier to connect to IP addresses rather than hostnames (as we do for xapi-xapi connections within a pool).

The idea of the purpose is to limit what a trusted certificate is used for, in a way that works for both cases. I think adding hostnames/domains in the mix is making this more complicated for not a lot of benefit?

Copy link
Member Author

@minglumlu minglumlu Dec 18, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With introducing the purpose, the only use pattern of these trusted certificates is to use the bundles for a purpose (a CA bundle or a peer bundle). The server names are not involved in this use pattern.
And the server names are not applicable to CA certificates.
For peer certificates, it looks useful for the clients/users when there are multiple peer certificates installed for the same purpose.
For example, a user may have set up multiple license servers, and for some reason they switch license server back and forth. It would be clear for them to know which peer certificate is for which license server in a quicker way if they can see the server names from the peer certificates.

Maybe we could do some sort of detection of shared certificate ... but this seems fragile

Yeah. I think detection of the sharing is necessary. But updating for sharing seems too complicated.
Say, there is a peer certificate for www.example.com installed already. Now a same one but with a different name www1.example.com is to be installed. Making it return an error will not impact the use case as it is the bundle to be used instead of the server name. And it's still clear to the user as the www.example.com is still there.

Otherwise, I would lean to add CN and SAN fields in Certificate for thorough clearness. But it would not bring a lot of benefit currently.

Copy link
Member

@psafont psafont Dec 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Peer (pinning): Exact match of the certificate, using the stunnel option verifyPeer. The hostname does not need to be checked. This also makes it easier to connect to IP addresses rather than hostnames (as we do for xapi-xapi connections within a pool).

I agree that during verification the hostname does not need to be matched. However, when the xapi host wants to connect to a server, if it knows which certificate to use it can minimize the amount of roundtrips to establish a connection. With the current approach there's an iteration of certificates until one matches. Using nameservers / IPs, the xapi host can trust a single certificate when connecting, instead of a bundle.

This means also means that the xapi host can minimize the amount of certificates are trusted per connection, and be more granular.

And the server names are not applicable to CA certificates.

I agree, it's part of the negative of my proposal

Copy link
Member Author

@minglumlu minglumlu Dec 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the current approach there's an iteration of certificates until one matches.

I think usually this done by the TLS lib so that we can have a big bundle for everything. And the purpose being proposed here is exactly to make it better.

This means also means that the xapi host can minimize the amount of certificates are trusted per connection, and be more granular.

I would not expect that there will be too many certificates in the bundle reduced by purpose. In most cases there will be only one in it. And use of bundle is also a common practice in TLS.

Given we have to combine CA and peer certificates together, the granularity of purpose looks proper.

Copy link
Collaborator

@liulinC liulinC Dec 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The distinction we are making here between "CA certificates" and "peer certificates" as separate types of trusted certificates is because they will be used a little differently when verifying the remote's cert:

  • CA: Verify the chain and check the hostname, using the stunnel options verifyChain and checkHost. The hostname needs to be verified, because there may be many certs for different hosts signed by the same CA.
  • Peer (pinning): Exact match of the certificate, using the stunnel option verifyPeer. The hostname does not need to be checked. This also makes it easier to connect to IP addresses rather than hostnames (as we do for xapi-xapi connections within a pool).

The idea of the purpose is to limit what a trusted certificate is used for, in a way that works for both cases. I think adding hostnames/domains in the mix is making this more complicated for not a lot of benefit?

In case of static IP, the hostname is not pushed to DNS, thus hosts in a pool can not resole each other,
*This also makes it easier to connect to IP addresses rather than hostnames * may explain why xapi-xapi connections within a pool works.


This new API can be used to install trusted peer certificates only.
The "purpose" parameter cannot be empty.

### pool.uninstall_certificate
This is a new API introduced in this design to uninstall a trusted certificate with its arguments being defined as:
* session (ref session_id): reference to a valid session;
* certificate (ref Certificate): reference to the trusted certificate;
* force (bool): remove the database entry even if the file doesn't exist.

### pool.join
Since the trusted certificates managed in this design are pool-wide, any existing trusted certificates on the joining host should be removed during pool.join.
Copy link
Member

@psafont psafont Dec 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that instead of removing them, the certificates should be exchanged, like it happens with pool certificates.

I also think that the ca certificates are exchanges currently, am I wrong?

Copy link
Member Author

@minglumlu minglumlu Dec 17, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, the CA certificates are exchanged in pool.join.
I thought the pool's trust set up should not be changed by a new joiner. It has some implications on the trust set up. Would it be clearer that a new joiner accepts the pool's trust set up and drops its own? Just like a joiner drops its other pool-wide configurations during pool join.

Copy link
Member

@psafont psafont Dec 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought the pool's trust set up should not be changed by a new joiner.

I think it's OK to do so, unless there are filename, or configuration conflicts (think of different WLB servers being used). On Pool join pools A and B have to trust each other, so they also trust whatever they were trusting as well. There's no problem security-related, IMO.

We can be smarter and for each of the purposes see how the related setting is joined, and drop any unused certificates coming from the joiner.

For example:
Pool A uses 3.wlb.example.com as WLB server, stores its peer certificate using wlb as purpose.
Host B uses 1.wlb.example.com as WLB server, stores its peer certificate using wlb as purpose.

When joining, only a single WLB server is used, the configuration of Pool A is used. Host B imports the wlb certificates in Pool A. After the reboot on pool join, Host B checks that now 3.wlb.example.com is used, and the certificate that was used for 1.wlb.example.com can be uninstalled.

Instead, all trusted certificates from the pool will be synchronized to the new host in the pre-join phase.

### Other APIs
The install/uninstall APIs above are not the only ways of managing the trusted certificates.
A particular API, e.g. "pool.set_wlb_url", may also install the trusted certificate used to validate the WLB server on subsequent TLS connections.
However, regardless of the entry point, all trusted certificates must be represented by a *Certificate* database object and stored in the same way described below as if installed by the install APIs.

## Trust store
The trusted certificates are stored in individual hosts' filesystems.
The existing stores defined in the base design are:
| Name | Filesystem location | User-configurable | Used for |
| ---- | ------------------- | ----------------- | -------- |
| Trusted Default | /etc/stunnel/certs/ | yes (using API) | Certificates that users can install for trusting appliances
| Trusted Pool | /etc/stunnel/certs-pool/ | no | Certificates that are managed by the pool for host-to-host communications
| Default Bundle | /etc/stunnel/xapi-stunnel-ca-bundle.pem | no | Bundle of certificates that hosts use to verify appliances (in particular WLB), this is kept in sync with "Trusted Default"
| Pool Bundle | /etc/stunnel/xapi-pool-ca-bundle.pem | no | Bundle of certificates that hosts use to verify other hosts on pool communications, this is kept in sync with "Trusted Pool"

Regarding the "User-configurable", when it is "yes", it means a user can only install and remove the file via APIs; when it is "no", it means the user can't install or remove it even via APIs. In any cases, a user can't change the certificate files directly.

When a trusted certificate is being installed via "pool.install_ca_certificate" but with an empty "purpose",
the trusted certificate will be stored in the existing "Trusted Default" and "Default Bundle" for general purpose.

There is no general‑purpose store for trusted peer certificates, because each peer certificate is specific to a single server and therefore unsuitable for a shared trust bundle for general purpose.

The pool "Trusted Pool" and "Pool Bundle" are for host-to-host TLS communications within a pool. This design doesn't change them.


When the "purpose" is not empty, the stores for the certificates installed via "pool.install_ca_certificate" or "pool.install_peer_certificate" are defined as:
| Name | Filesystem location | User-configurable | Used for |
| ---- | ------------------- | ----------------- | -------- |
| Trusted Peer | /etc/trusted-certs/peer-\<PURPOSE\>/ | yes (using API) | Trusted peer certificates that users can install to validate a peer’s identity when establishing a TLS connection for \<PURPOSE\>
| Trusted CA | /etc/trusted-certs/ca-\<PURPOSE\>/ | yes (using API) | Trusted root CA certificates that users can install to validate a peer’s identity when establishing a TLS connection for \<PURPOSE\>
| Peer Bundle | /etc/trusted-certs/peer-bundle-\<PURPOSE\>.pem | no | Bundle of trusted peer certificates under /etc/trusted-certs/peer-\<PURPOSE\>/ to verify a peer's identity when establishing a TLS connection for \<PURPOSE\>
| CA Bundle | /etc/trusted-certs/ca-bundle-\<PURPOSE\>.pem | no | Bundle of trusted root CA certificates under /etc/trusted-certs/ca-\<PURPOSE\>/ to verify a peer's identity when establishing a TLS connection for \<PURPOSE\>

The filesystem location is derived from the \<PURPOSE\>. Each \<PURPOSE\> string corresponds to a predefined value of the "purpose" type in the database, implemented as predefined constants.
The certificate file names under filesystem locations of "Trusted Peer" and "Trusted CA" should not rely on the "name" field of *Certificate* objects anymore.
Instead, the UUIDs of the *Certificate* objects are used as the file names.

## Precedence order of choosing trust stores
The "Peer Bundle", "CA Bundle", and "Default Bundle" can be directly used when establishing TLS connections.
The endpoint to validate the peer's identity must unambiguously choose only one non-empty bundle from them with the following precedence order:
1. "Peer Bundle"
2. "CA Bundle"
3. "Default Bundle"

No more attempts on remaining bundles when validation with the selected one fails (the server certificate is not trusted by the selected bundle).

For example, if "Peer Bundle" doesn't exist and "CA Bundle" (not empty) is selected to do the validation, the endpoint should not try with "Default Bundle" even when the validation with "CA Bundle" failed.
Loading