Provide API that take in a hash instead of io bytes

[laurentsimon]

Description

The current sign and verify APIs take as input an io[bytes]:

• signing: https://github.com/sigstore/sigstore-python/blob/main/sigstore/sign.py#L119
• verifying: https://github.com/sigstore/sigstore-python/blob/main/sigstore/verify/models.py#L238

There are limitations to expecting an input buffer:

For signing, we may want to sign something like a tree structure, a folder, or something that does not have an obvious array representation and is application specific. In this case it would be useful to let the caller hash the content and pass it to the sign function. This would not work for edDSA signature - because they need the raw text message - but works for ECDSA.

For verification, same limitation. In addition, imagine you want to create a remote service (REST API) that verifies attestations / signatures. The signed message you may be too large to be sent over the network (e.g., a large SBOM), so you may want to let the caller hash the document and send over only the attestation and the hash of the message. There is one limitation to allowing this: it's is more prone to mistakes. There have been cases where a verifier would mistakenly take in the hash provided by an attacker instead of the message they intended to verify. So maybe we could dangerous APIs under a "danger" folder. FWIW, the Tink crypto library puts dangerous functions under a subtle folder.

[woodruffw]

For signing, we may want to sign something like a tree structure, a folder, or something that does not have an obvious array representation and is application specific.

I'm not sure that we want to encourage this on an ecosystem level -- if you want to sign over a tree structure or a directory or something like that, then you should canonicalize that into something that looks like a stream of bytes (e.g., a tar archive) and sign over that. Could you say more about your use case, and why that kind of "directory-to-file" representation wouldn't work?

In this case it would be useful to let the caller hash the content and pass it to the sign function. This would not work for edDSA signature - because they need the raw text message - but works for ECDSA.

FWIW, this is essentially what we do internally (since we current use ECDSA, as you noted) -- we hash the message ahead of time and pass it into the underlying signing APIs as a Prehashed object.

I wouldn't be opposed to exposing that as a public API as well, although I'm hesitant to further introduce public expectations around the fact that our current signature algorithm is defined over the digest rather than the input itself. As you noted, that won't work for EdDSA, and we may want to switch to EdDSA in the near future (although ideally we'd switch to Ed25519-ph, which would avoid that problem).

For verification, same limitation. In addition, imagine you want to create a remote service (REST API) that verifies attestations / signatures. The signed message you may be too large to be sent over the network (e.g., a large SBOM), so you may want to let the caller hash the document and send over only the attestation and the hash of the message. There is one limitation to allowing this: it's is more prone to mistakes. There have been cases where a verifier would mistakenly take in the hash provided by an attacker instead of the message they intended to verify. So maybe we could dangerous APIs under a "danger" folder. FWIW, the Tink crypto library puts dangerous functions under a subtle folder.

This is an understandable use case, although I have similar hesitations around it -- we've had some similar conversations around the Sigstore bundle format choosing to contain digests, which I was concerned about for the same reasons (it's easy to accidentally trust that digest as the signing input rather than retrieving or re-deriving it from somewhere else).

In terms of handling this use case more generally, I wonder if Sigstore as an ecosystem could issue some non-normative guidance around prehashing (which, for ECDSA, would effectively mean double-hashing). In other words, Sigstore could recommend that people do M' = H(M) even before signing for scenarios/use cases where the size of the original message is unacceptably large.

Do you think something like that would be suitable to your use case?

[laurentsimon]

For signing, we may want to sign something like a tree structure, a folder, or something that does not have an obvious array representation and is application specific.

I'm not sure that we want to encourage this on an ecosystem level -- if you want to sign over a tree structure or a directory or something like that, then you should canonicalize that into something that looks like a stream of bytes (e.g., a tar archive) and sign over that. Could you say more about your use case, and why that kind of "directory-to-file" representation wouldn't work?

internal use case where we have a folder containing multiple files. There is an existing set of API used to read files within the folder (it's a framework). We're trying to update the API to support signature verification. So we can't use a tarball because the existing API takes in a folder, so we can't change this :/. We've thought of computing a tarball but it's not reproducible so we can't compute it on the fly to re-construct the folder. Let me know if you have a better idea.

In this case it would be useful to let the caller hash the content and pass it to the sign function. This would not work for edDSA signature - because they need the raw text message - but works for ECDSA.

FWIW, this is essentially what we do internally (since we current use ECDSA, as you noted) -- we hash the message ahead of time and pass it into the underlying signing APIs as a Prehashed object.

I wouldn't be opposed to exposing that as a public API as well, although I'm hesitant to further introduce public expectations around the fact that our current signature algorithm is defined over the digest rather than the input itself. As you noted, that won't work for EdDSA, and we may want to switch to EdDSA in the near future (although ideally we'd switch to Ed25519-ph, which would avoid that problem).

Good to know.

For verification, same limitation. In addition, imagine you want to create a remote service (REST API) that verifies attestations / signatures. The signed message you may be too large to be sent over the network (e.g., a large SBOM), so you may want to let the caller hash the document and send over only the attestation and the hash of the message. There is one limitation to allowing this: it's is more prone to mistakes. There have been cases where a verifier would mistakenly take in the hash provided by an attacker instead of the message they intended to verify. So maybe we could dangerous APIs under a "danger" folder. FWIW, the Tink crypto library puts dangerous functions under a subtle folder.

This is an understandable use case, although I have similar hesitations around it -- we've had some similar conversations around the Sigstore bundle format choosing to contain digests, which I was concerned about for the same reasons (it's easy to accidentally trust that digest as the signing input rather than retrieving or re-deriving it from somewhere else).

In terms of handling this use case more generally, I wonder if Sigstore as an ecosystem could issue some non-normative guidance around prehashing (which, for ECDSA, would effectively mean double-hashing). In other words, Sigstore could recommend that people do M' = H(M) even before signing for scenarios/use cases where the size of the original message is unacceptably large.

Do you think something like that would be suitable to your use case?

No strong opinion from me. If Sigstore wants to do this I'm not against it. But maybe you need a special record type for it otherwise verification won't be compatible across clients?

@haydentherapper from Sigstore side

[woodruffw]

internal use case where we have a folder containing multiple files. There is an existing set of API used to read files within the folder (it's a framework). We're trying to update the API to support signature verification. So we can't use a tarball because the existing API takes in a folder, so we can't change this :/. We've thought of computing a tarball but it's not reproducible so we can't compute it on the fly to re-construct the folder. Let me know if you have a better idea.

Ah yeah, that's thorny -- tar isn't great in terms of reproducibility. Have you seen these docs from the Reproducible Builds folks? I'm not sure if they'll help in your case, but I've used them in the past to make tarballs more reproducible (in my case so that they don't bust the cache they're in).

As another potential option here: would you be able to produce a "receipt"-style input for your folder? For example, you could hash each file in your folder structure, sort them lexicographically by filename, and then use Sigstore to sign the receipt.

This could be something like shasum -a256 --binary's output, e.g.:

digest1 *path/to/file1
digest2 *path/to/file2

This would be a few more steps on the calling side, but it preserves the conceptual interface that "signing" is done over inputs and not their precomputed images 🙂

Not sure if that will work in your case either, but I figured it was worth bringing up.

No strong opinion from me. If Sigstore wants to do this I'm not against it. But maybe you need a special record type for it otherwise verification won't be compatible across clients?

Yeah, that might be helpful -- in principle everything here is compatible with the hashedrekord type, but a separate type identifier might be useful to clients to signal that they should pre-digest (or not) their input rather than assuming that they'll know it from their application's context.

[laurentsimon]

internal use case where we have a folder containing multiple files. There is an existing set of API used to read files within the folder (it's a framework). We're trying to update the API to support signature verification. So we can't use a tarball because the existing API takes in a folder, so we can't change this :/. We've thought of computing a tarball but it's not reproducible so we can't compute it on the fly to re-construct the folder. Let me know if you have a better idea.

Ah yeah, that's thorny -- tar isn't great in terms of reproducibility. Have you seen these docs from the Reproducible Builds folks? I'm not sure if they'll help in your case, but I've used them in the past to make tarballs more reproducible (in my case so that they don't bust the cache they're in).

As another potential option here: would you be able to produce a "receipt"-style input for your folder? For example, you could hash each file in your folder structure, sort them lexicographically by filename, and then use Sigstore to sign the receipt.

We have a reproducible hashing implemented. We would like to sign without double-hashing if possible, which is why I was proposing the API that takes in a hash.

This could be something like shasum -a256 --binary's output, e.g.:

digest1 *path/to/file1
digest2 *path/to/file2

That sort of works but when there are many files, you end up keeping a large content in memory before passing it to the API. Hashing ourselves means we can read one file at at time and keep memory usage lower.

[woodruffw]

That sort of works but when there are many files, you end up keeping a large content in memory before passing it to the API. Hashing ourselves means we can read one file at at time and keep memory usage lower.

I might be misunderstanding, but I don't think this approach prevents you from hashing one-at-a-time: there's a linear overhead here for each filename and file digest, but unless you're processing on the order of 100,000 or more files per directory, the memory usage shouldn't be appreciable on a consumer computer (much less a server).

(You could also avoid the linear memory usage entirely by writing appending to the receipt on disk -- your overhead would then only be 32 bytes for each digest + however long each path is, which should be a rounding error compared to what sigstore-python is doing under the hood. That would also parallelize nicely without appreciable memory overhead.)

Overall, I think I'm hesitant to expose the hash directly as a supported public API, unless (1) other clients also agree that it's a reasonable interface to support, and/or (2) Sigstore makes an official design/policy statement about input/digest semantics (i.e. whether the lack of domain separation here is actually intended, or just an incidental property of the current popular signing algorithms).

[haydentherapper]

Sigstore could provide some guidance on signing a folder or a set of files in a deterministic manner. For example, Golang uses dirhash. Ultimately it'll be up to the signer though, leaving two options: An API for signing the hash, or double-hashing by hashing and signing the hash. The former can be misused obviously, particularly if your scheme for deriving a hash from a folder doesn't have collision resistance, so it's safer to just treat that hash as a blob.

FWIW, the golang APIs for Sigstore do support signing a hash directly, but this isn't exposed in Cosign. I like Laurent's suggestion of supporting this as a low level API but not exposing this as a top-level API without more discussion with other clients.

[woodruffw]

Sigstore could provide some guidance on signing a folder or a set of files in a deterministic manner. For example, Golang uses dirhash. Ultimately it'll be up to the signer though, leaving two options: An API for signing the hash, or double-hashing by hashing and signing the hash. The former can be misused obviously, particularly if your scheme for deriving a hash from a folder doesn't have collision resistance, so it's safer to just treat that hash as a blob.

This would be great!

I just checked, and the (WIP) client specification does include double-hashing as a potential signing input (link). That doesn't solve the "signalling" problem of determining that the input is a digest, though.

[haydentherapper]

To come back to this - @woodruffw, would you be open to a modified or another API that would allow callers to configure the hash algorithm? For our use case, we've thought about dirhash as a standardized approach, but likely will be exploring a few other hashing approaches.

Another option is to do these tests on a fork, and then once we've found an optimal hashing strategy, "standardize" it by adding it as an option to the python library.

[woodruffw]

To come back to this - @woodruffw, would you be open to a modified or another API that would allow callers to configure the hash algorithm? For our use case, we've thought about dirhash as a standardized approach, but likely will be exploring a few other hashing approaches.

Yes! I think modifying the API makes sense: IMO input could be input: IO[bytes] | Prehashed, where Prehashed is a smaller wrapper type for a bytes digest. Or something like that 🙂

[laurentsimon]

Great! I'm willing to send a PR. Do you have any recommendations / requirements / things I should know about besides the comment above?

[woodruffw]

Great! I'm willing to send a PR. Do you have any recommendations / requirements / things I should know about besides the comment above?

Not off the top of my head, I think you're good to go!