candle: health check by queuing on cuda

[michaelfeil]

What does this PR do?

Fixes #775

Fixes a crucial health check, that when a GPU has a driver failure (fallen of the bus etc, or CUDA OOM). TEI technically is very unlikley to OOM during runtime, as it allocates only ~10-20MB more than during warmup in some edge cases. I have never seen it OOM on cuda.
However, if you have a driver issue, #775 fails. To make things worse, the way things are set up in k8s is that once one replica starts erroring at very fast rates, it might consume a lot of requests. (more than healthy replicas, making incidents worse)

On CPU , works great, compiled via

cargo build --release --bin text-embeddings-router -F candle -F dynamic-linking -F http --no-default-features

2025-12-17T19:59:30.262188Z  INFO health:health:health: text_embeddings_backend_candle: backends/candle/src/lib.rs:605: Device Cpu healthcheck passed
2025-12-17T19:59:32.116511Z  INFO health:health:health: text_embeddings_backend_candle: backends/candle/src/lib.rs:605: Device Cpu healthcheck passed
2025-12-17T19:59:33.330219Z  INFO health:health:health: text_embeddings_backend_candle: backends/candle/src/lib.rs:605: Device Cpu healthcheck passed
2025-12-17T19:59:34.256090Z  INFO health:health:health: text_embeddings_backend_candle: backends/candle/src/lib.rs:605: Device Cpu healthcheck passed
2025-12-17T19:59:35.548203Z  INFO health:health:health: text_embeddings_backend_candle: backends/candle/src/lib.rs:605: Device Cpu healthcheck passed

On GPU, works great, compiled via cargo build --release --bin text-embeddings-router -F candle-cuda -F dynamic-linking -F http --no-default-features

2025-12-17T19:52:10.256396Z  INFO text_embeddings_router::http::server: router/src/http/server.rs:1853: Ready
2025-12-17T19:52:18.663691Z  INFO health:health:health: text_embeddings_backend_candle: backends/candle/src/lib.rs:604: Device Cuda(CudaDevice(DeviceId(1))) healthcheck passed
2025-12-17T19:52:22.167088Z  INFO health:health:health: text_embeddings_backend_candle: backends/candle/src/lib.rs:604: Device Cuda(CudaDevice(DeviceId(1))) healthcheck passed
2025-12-17T19:52:34.347053Z  INFO health:health:health: text_embeddings_backend_candle: backends/candle/src/lib.rs:604: Device Cuda(CudaDevice(DeviceId(1))) healthcheck passed
2025-12-17T19:52:36.060119Z  INFO health:health:he

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the documentation guidelines.
• Did you write any new necessary tests? If applicable, did you include or update the insta snapshots?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[OlivierDehaene]

Instead of doing actual work on the device, one could use cudaGetLastError directly and see whether the error is recoverable or not.
If it is not, in my opinion, the runtime should shutdown directly instead of setting itself to un-healthy and waiting for a higher level system to catch its health state and act on it.

To make things worse, the way things are set up in k8s is that once one replica starts erroring at very fast rates, it might consume a lot of requests.

What type of load balancer do you use? You should be able to set it up to remove replicas with high failure rates automatically even without checking their health.

[michaelfeil]

E.g. knative or vanilla k8s (random). Fair comment on runtime.shutdown if its not reverable.