Testlie
Menu

More sandboxes

On this page

Next time: break the next one

Negative paths are the ones nobody tests, because making a real provider fail on cue is hard. Next time turns any captured text, email, API call or webhook into a switch: open it, choose what should go wrong, and the next matching one does. Your code under test changes nothing; it sees exactly what Twilio would send.

From a message

In Activity, select an SMS, an email, an API call or a webhook and choose ⚡ Break the next one. On the SMS page, open a recipient's thread; in an inbox, open a message; then choose ⚡ Next time…. In an API mock's call log, open a call and use its Next time… card. Then pick:

  • What happens: the outcome, with a line saying what your app will see.
  • For: this number or address, a pattern (+97150*, *@bounce.test), every number or address on this account or inbox, or every one in the team.
  • How long: the next one, the next 3 or 5, every other one for an hour (flaky), every match for 15 minutes, or until you disarm it.
  • Only in test run (when the message was tagged): the rule then breaks only traffic with the same X-Testlie-Run, so tests running in parallel are not affected.

Armed rules show above the thread and in the message panel with a ✕ to disarm them. A message a rule broke carries a 💥 injected badge, so nobody chases it as a real failure. Used-up and expired rules disarm themselves.

Outcomes

SMS

These apply to every text the account sends, over the Twilio-compatible API or SMPP, including the code texts Twilio Verify sends.

Outcome Your app sees
Undelivered: unreachable handset 201, then status callbacks queued → sent → undelivered, ErrorCode 30003
Undelivered: unknown handset the same with 30005
Carrier filtered the same with 30007, as when a carrier silently blocks an OTP text
Rejected: invalid number the send call fails: HTTP 400, code 21211; nothing is stored
Rejected: recipient opted out the send call fails: HTTP 400, code 21610
Late delivery sent at once, delivered only after 1 to 60 minutes
Stuck in sent sent, then no final status and no further callback
Duplicate delivered callback delivered normally, but the delivered callback arrives twice

Twilio Verify

Outcome Your app sees
Code not sent: blocked starting the verification answers 403, code 60410; no code exists
Code not sent: max send attempts starting or resending answers 429, code 60203
Code expired at check checking answers 404, code 20404, even with the right code
Right code rejected checking the right code answers 200, status: pending, valid: false, and counts as an attempt

Email

For mail your app sends to an inbox over SMTP (and inbound mail to an inbox address). Recipient outcomes answer at RCPT TO for that address only; message outcomes answer after DATA.

Outcome Your app sees
Recipient rejected: user unknown RCPT TO answers 550 5.1.1; other recipients of the same mail are accepted
Mailbox full RCPT TO answers 452 4.2.2, a temporary failure to retry later
Greylisted RCPT TO answers 451 4.7.1; with the default of one time, the retry is accepted
Rejected as spam after DATA, 554 5.7.1; nothing is stored
Temporary failure after DATA, 421 4.3.0; nothing is stored
Slow accept stored, but the 250 after DATA comes only after 1 to 120 seconds

The X-Testlie-Run header is only known after DATA, so a rule scoped to a run applies to the message outcomes only.

API mocks

For calls to any mock (Stripe, Adyen, Salesforce, Airtable...), matched on METHOD /path, such as POST /v1/payment_intents or * /v1/customers/*. Rules only take calls that present one of the mock's credentials, so someone who knows the URL cannot use them up; the provider does not run for an injected error, so nothing is created on its side.

Outcome Your app sees
Server error 500 with the provider's own error body
Rate limited 429 with the provider's rate-limit error and Retry-After (1 to 3600 seconds)
Unavailable 503 with Retry-After: 30
Not found the provider's 404, as if the object was deleted on their side
Timeout the call hangs for 1 to 60 seconds, then 504; your client's timeout usually fires first
Slow response the normal response, after 1 to 60 seconds
Malformed response 200 with a truncated JSON body: your parsing fails

Injected calls are marked 💥 in the call log and in Activity.

Webhooks

For the webhooks a mock sends to your app (Stripe events, DocuSign Connect, Segment...), matched on the event type, such as payment_intent.succeeded or payment_intent.*. A rule replaces the mock's webhook scenario for the events it matches.

Outcome Your app sees
Dropped nothing; the delivery log shows it as dropped. Your app must notice through polling or reconciliation
Duplicated the same webhook, same event id, twice half a second apart
Late duplicate the webhook, then the same one again after 5 seconds to an hour, as after a lost response
Delayed the webhook only after 1 second to an hour, so later events overtake it
Bad signature the webhook signed with the wrong key: your signature check must reject it

Webhooks are sent by Testlie, not tagged by a request of yours, so run scoping does not apply to them.

A rule wins over failure scenarios and the account's own failure rules for the events it matches; everything else behaves as before.

From tests and CI

bash
curl -X POST https://app.testlie.com/api/next-time \
  -H "Authorization: Bearer $TESTLIE_API_KEY" -H "Content-Type: application/json" \
  -d '{"area":"sms","systemId":"<sms account id>","match":"+15551234567","outcome":"undelivered"}'
Field Meaning
area sms, verify, email, api_mock or webhook
outcome an id from the tables above: undelivered, unknown_handset, filtered, invalid_number, opted_out, late, stuck, duplicate_callback; send_blocked, max_send_attempts, code_expired, wrong_code; user_unknown, mailbox_full, greylisted, rejected_spam, temp_failure, slow_accept; server_error, rate_limited, unavailable, not_found, timeout, slow, malformed; dropped, duplicated, late_duplicate, delayed, bad_signature
systemId the SMS account, the inbox for email, the mock for api_mock and webhook; omit for every one of the team
match glob on the To number, the recipient address, METHOD /path or the webhook event type (* any run of characters, ? one); default *
params late: { "delayMinutes": 1-60 }; slow_accept, slow, delayed: { "delaySeconds" }; rate_limited: { "retryAfterSeconds" }; timeout: { "waitSeconds" }; late_duplicate: { "afterSeconds" }
times how many events it applies to (default 1); null for every match until it expires
every 2 breaks every other match, starting with the first; n one in n
expiresInMinutes default a day (an hour when times is null); null keeps it until disarmed
runId only events tagged with this X-Testlie-Run

GET /api/next-time lists the live rules (?area=, ?systemId=, ?includeDone=true for the last day's used, expired and disarmed ones) with the outcome catalogue. DELETE /api/next-time/<id> disarms one; DELETE /api/next-time?systemId=... disarms all of an account. Arming and disarming need an owner, an admin or a write API key.

With the Node SDK, rules default to the client's run (TESTLIE_RUN), so a CI job only breaks its own traffic:

js
const rule = await testlie.nextTime.failNextSms(smsAccountId, "+15551234567", "filtered");
await signUp("+15551234567");                 // your app sends the OTP; the carrier "filters" it
await expect(page.getByText("Didn't get a code?")).toBeVisible();
await testlie.nextTime.arm({ area: "verify", systemId: smsAccountId, match: "+15551234567", outcome: "wrong_code" });
await testlie.nextTime.failNextEmail(inboxId, "jo@example.com", "user_unknown"); // the welcome mail bounces at RCPT
await testlie.nextTime.failNextCall(stripeMockId, "POST /v1/payment_intents", "timeout"); // checkout must recover
await testlie.nextTime.breakNextWebhook(stripeMockId, "payment_intent.succeeded", "duplicated"); // fulfil once

AI agents connected over MCP use arm_next_time, list_next_time and disarm_next_time.

Limits

At most 50 live rules per team. A rule counts each matching event when it happens, so two parallel sends never both take a "next one" rule.