Debug network problems in jobs
Jobs run behind an enterprise firewall. Outbound traffic follows the job’s network policy, and tools inside the job use the CRACI ephemeral certificate authority. A step that cannot reach a destination on CRACI, or rejects its certificate, has run into one or the other. CRACI runners also differ from other providers in CPU, memory, and storage, so a step that fails for one of those reasons is out of scope for this page.
Start with the error
Section titled “Start with the error”| What the job reports | Go to |
|---|---|
A certificate error: x509: certificate signed by unknown authority, unable to get local issuer certificate, CERTIFICATE_VERIFY_FAILED, PKIX path building failed | Certificate errors |
403 Forbidden, a connection reset, an EOF, or a handshake failure with no certificate error | Blocked connections |
The job trace confirms which one it is. A blocked connection is listed there as a proxy violation; a certificate problem produces none, because the connection was allowed and the tool refused to use it.
Successful hostname resolution does not aid in troubleshooting: DNS is also resolved for destinations the policy goes on to block.
Certificate errors
Section titled “Certificate errors”Every job carries a CRACI CA certificate, valid for that job alone, in the system trust store:
| Path | Contents |
|---|---|
/usr/local/share/ca-certificates/craci.crt | The job’s CA certificate on its own |
/etc/ssl/certs/ca-certificates.crt | The system trust bundle, including the job CA |
Steps run with these variables already pointing at it:
| Variable | Value |
|---|---|
SSL_CERT_FILE, REQUESTS_CA_BUNDLE, CURL_CA_BUNDLE, NIX_SSL_CERT_FILE | /etc/ssl/certs/ca-certificates.crt |
NODE_EXTRA_CA_CERTS | /usr/local/share/ca-certificates/craci.crt |
JAVA_TOOL_OPTIONS | A Java trust store holding the same bundle |
Most tooling picks the CA up from one of these with no change to your workflow. A certificate error means the failing tool read none of them:
- It ships its own CA bundle. Append
/usr/local/share/ca-certificates/craci.crtto that bundle, or point the tool at$SSL_CERT_FILE. - It reads a variable not listed above. Set that variable to
/etc/ssl/certs/ca-certificates.crt. - A step in your workflow overrode one of these variables. The tool then reads a trust store with no job CA in it.
Containers the job starts get the same variables, and the trust bundle and Java trust
store are available at the same paths. An image that ships no trust store at all still
needs ca-certificates installed. docker build gets the CA too, with APT configured
separately because it ignores SSL_CERT_FILE. None of it reaches your committed layers.
A client that pins a certificate or public key, or presents a client certificate for mutual TLS, cannot use the CA at all. Those need an exemption.
Blocked connections
Section titled “Blocked connections”Open the run, find the job, and choose View trace. The trace streams while the job runs and stays available afterwards. Proxy violations lists what was blocked and why, and All network accesses contains the full record, including the rule that matched each allowed connection.
Each violation carries a reason:
| Reason | Meaning | Fix |
|---|---|---|
no_match | The destination was identified, but no preset, source, or connection matched it | Add a preset, custom source, or connection rule that covers it |
resolver_unknown | The job connected to an address it never resolved through DNS | Match the destination with cidrs instead of a hostname pattern |
tls_no_match | The TLS client hello carried no server name, or one no rule matched | Add the name to serverNamePatterns, or use cidrs when there is no server name |
invalid_client_hello | The port carries TLS rules, but the client sent something other than TLS | Use a tcp connection for that port |
git_protocol_v1 | The Git client negotiated protocol v1 | Use Git 2.18 or later with protocol.version=2, the default since Git 2.34 |
upstream_known_hosts | The SSH host key was not among the trusted keys | Add the server’s key to upstreamKnownHosts |
missing_upstream_auth | The SSH server did not accept the connection | Use HTTPS with a token, or allow the host with a tcp connection |
dial_failed | The policy allowed the connection; the destination refused it | Check the destination and any allowlist on its side |
dial_timeout | The policy allowed the connection; the destination did not answer | Check reachability from the public internet |
Any other reason is a transport failure rather than a policy decision. Include the trace when you contact support about one.
Control network access from jobs covers the rule syntax for each of these fixes.
Destinations with no hostname
Section titled “Destinations with no hostname”Rules match on names: the TLS server name for tls, the resolved hostname for tcp, the
URL for package sources. A job that dials a literal IP address, reuses a cached address,
or resolves names through its own resolver gives CRACI no name to match, and the
connection is blocked with resolver_unknown or tls_no_match even though the hostname
is allowed.
Match those destinations with a cidrs selector. Jobs reach the internet over IPv4 only,
so cidrs values are IPv4 ranges.
Clients that cannot use the CA
Section titled “Clients that cannot use the CA”Exempt the destination with a tls connection.
- uses: cracicorp/setup@v1 with: network: | connections: - tls: serverNamePatterns: [api.example.com] ports: [443]Exempt as little as possible. An exempt destination is excluded from CRACI’s package detection, so it contributes nothing to the job’s SBOM or to the supply chain analysis built on it. A package source is the worst thing to exempt: see Understand job and cache SBOM completeness.
The audit runner template exempts a whole job. See
Control network access from jobs.