Vendor Search · Network Growth · A/B test

Trust in “Who am I paying?”

AI-assisted vendor identity matching

At a glance

Summary:: Accounts Payable (AP) users struggle to trust the entities shown in network search, forcing manual verification and delayed first payment. We tested concepts with 5 users and found 3 ways to improve trust in new vendor add experience.
Team:: 1 product manager, 1 product designer, 1 front-end engineer, 1 back-end engineer.

3+ experiments to validate initial hypothesis

50%+ targeted search success rate lift

01Where it got hard

“The number of connections. So often we have multiple choices to select and try to go with the one with the most connections if there are multiples.”

Our users are accounts payable (AP) specialists, bookkeepers and finance staff who pay vendors regularly. When they add a new vendor, they search our payment network for that vendor, but they often don’t trust the matches they see. So they verify each vendor by hand, and the first payment is delayed.

That shows up as a low search success rate: the share of searches where the user picks a vendor from the first page of results. The network had the data to earn their trust; the vendor card showed none of it.

What must the vendor card show, and which signals earn trust?

Legacy vendor search UI: the Add a vendor flow lists matching vendors as near-identical cards, each marked Verified vendor, with a details drawer showing network connections, member-since date and Payment Network ID.

To find out, we gave participants one task: “You need to pay your vendor, Boston Tea, $800.00 by February 28. Find them, add them, and confirm you’ve chosen the right vendor.” We measured how confident they felt in their choice, whether they understood terms like “connections” and “virtual card”, and, in their own words, what they relied on to decide.

Learning #1

They actively distrust vendor listings, and have learned to compensate.

When asked whether they ever have difficulty finding the correct vendor to pay, four of five said yes, citing concrete reasons: duplicate listings with different email addresses, accounts created by staff who have since left, and information that's simply outdated or doesn't match what their contact provided.

Learning #2

They treat connection count as a trust heuristic.

When asked what else they'd want to know before deciding, a participant revealed the mental model: “The number of connections. So often we have multiple choices to select and try to go with the one with the most connections if there are multiples.”

Learning #3

They want freshness, not just accuracy.

Several users described a verification ritual, confirming by email that the account is still active and that the vendor still prefers ePayments. One user wanted a way to “prompt [the vendor] to update their information if we know something is out of date.”

Learning #4

Users want multiple ways to add their new vendor.

Often times APs will drop an invoice or W-9 of the vendor instead of manually typing vendor info to add a vendor.

Learning #5

Many SMBs have vendor policy in place before sending payments.

Larger AP teams have vendor identity policy in place that requires W-9 collection before payment.

Here's a list of data points our users prefer to see in order to validate a new vendor.

Account last updated date.
Account created date.
# of connections/customers.
Vendor preferred payment method.
A way to verify vendor by first and last name (esp. contractors).
A way to verify vendor by searching by Tax ID.

Maze test synthesis board: sticky notes grouped by question — expected next step, understanding of the card's data points, difficulty finding the right vendor, most useful information, and useful-but-missing data.

02The decision I made, and why

Three decisions, one thesis:

Only evidence users actually rely on goes on the vendor card. Every data point had to earn its place in research.
Show the evidence where the decision happens. Verification moved out of a separate drawer and into the chat itself, so users don’t have to open anything to check a match.
The system suggests; the user always makes the final call. The most likely match appears first, but nothing is chosen for them.

Why: trust comes from specific signals, not badges, and the AI never guesses with money. The card also had to fit inside the existing add-vendor flow and work with the data the network actually has (connection counts, payment method, business name and address), in plain language for non-technical finance users.

Each vendor card shows two kinds of evidence:

Standard identity checks: details used to verify any vendor, such as address, Tax ID and the last digits of the bank account.
Network signals only we have: details that tell this vendor’s story, such as how many companies in the network pay them, when the account was created and when it was last active.

03What I tried and dropped

The kitchen-sink card. Round 1 put every data point users had asked for in front of the AP. I cut it back to the signals research showed people actually use.
The verification drawer. Round 1 hid the evidence behind a “details” drawer. By round 3 it was gone, and the evidence sat in the chat.
A generic “Verified vendor” badge. Instead of assuming it worked, we tested it against specific trust signals.
The full A/B test against legacy. The project was cut short before it could run, so we pivoted to three smaller experiments on the legacy flow.

Round 1: every useful data point, a confidence score on the card, and a feedback question so matching could improve once live, but the evidence lived in a drawer.

Round 2: a new way in. APs often start from an invoice or W-9 rather than typing vendor details, so dropping a document became the main entry point; later, the system could pull new invoices straight from the user’s inbox.

04What changed

Behaviour: users could judge a match without leaving the flow, and the feedback question lets matching improve once live.

Numbers, stated as targets: a 50%+ lift in search success rate. To be plain about status: the big A/B test never ran. We launched three incremental experiments on the legacy flow for directional validation:

Payment details. Hypothesis: We believe showing payment details on the cards of VC vendors will raise search success rate, because users can confirm it's the right vendor without leaving the results. Test: cards with payment details vs. without.
Specific trust signals vs. a generic badge. Hypothesis: We believe specific signals, such as last active date and number of connections, will raise search success rate more than a generic “Verified vendor” badge. They show users why a vendor can be trusted, not just that it can. Test: “Verified vendor” badge vs. specific signal tags.
Confidence score with a reason. Hypothesis: We believe a confidence score on the vendor card, with a short reason why it's likely the right vendor, will make users more confident in their pick and raise search success rate. Test: cards with the score and reason vs. without.

The three incremental experiments run on the legacy add-vendor flow, each with its hypothesis, control and test variants.

After the third round, I went back to our interview notes on how APs validate vendors and confirmed two habits with users:

They collect bank details after the bill is approved. If the invoice doesn’t include payment information, they ask the vendor for it at that point, so they don’t pay the wrong account.
They keep vendor records up to date. When a vendor’s details change, they update the existing record rather than adding a new one.

That shaped two next steps:

An optional step to request bank details from the vendor, inside the flow.
A final review screen that shows the vendor’s full, latest details, with what has changed, so users aren’t left wondering.