SAT, also known as Site Acceptance Testing (and often used interchangeably with System Acceptance Testing), can often be a repeat of Factory Acceptance Testing (FAT) test cases. Often enough, it’s a full repeat – only the location at which you’re conducting the tests is different.
Exceptions might be cases that are unique to your client’s environment. For example, you might find yourself adding cases focused on virtualized desktop environments or cases impacted by shared network storage that you might not have had in your development environment. In addition, web deployments might be even more unique at a client’s site, with special cases that test high availability failover scenarios, port routing, or the use of proxies.
Parts vs. the Whole
Think about building a bicycle – as the “builder” of the bike; I can lay a bunch of parts out on the floor. I can count them to ensure I have everything I need. I can do a fit test on the parts to make sure they fit together in ways that I expect. I can check for flaws that might lead to failure either during construction or during early use. All these things I can do as the builder, and I might feel good about where the project is heading.
You can think about SAT in this way – with SAT, we’re testing the parts – the components that ultimately will make the solution. With functional requirements, these parts are the various tools that meet those individual requirements.
Components vs. Capabilities
So here I have all these parts on the floor – at this point, I could bring the “rider” into the shop, point at all the parts laying around, and ask her: “how do you like your bike?” and she’ll probably ask “what bike?”.
Of course, the rider can’t comment until the parts are put together. The rider can’t judge the usefulness of a bike until the parts are put together to become an actual bike. But also, she has in her head an idea of what makes a useful bike. So, hopefully, you cleared that up before you started putting parts together!
Here, we have a nice track bike – it’s simple. It has a single gear; It’s lightweight; It accelerates quickly. It’s also a great way to get yourself killed on steep hills. If your rider wants to ride in the mountains, you’ll need to get her at least 19 more gears. Oh, and in the case of our track bike, she’ll need brakes.
Getting at what is a workable solution is the key. And also, recognize that this is probably going to be an incremental process – you’ll be deploying a number of successive releases, ultimately becoming the end solution. So your user acceptance cases will evolve to include those new capabilities as they become available.
If your bike starts as a single-speed, track-focused torture device, then make sure your User Acceptance Test (UAT) case doesn’t include hills! When the multi-speed drivetrain becomes available, make sure the acceptance criteria change as well. Can she ascend the hill? Can she slow down to safe speeds on the other side? Yes, don’t forget the brakes!
Define the Roles
Not every user is the same – you’ll often be designing a solution that will be used across a bunch of different types of users or roles. Each role will have its own acceptance criteria – and possible specialized tools to meet their needs. In discovery, make sure you define the roles involved: give them names; write a paragraph about what they do, where they work – if Wendy is a great example of your distribution system designer, then use the name “Wendy” in your conversations about the role.
Find the Exit
The collective set of acceptance criteria, taken from the UAT cases from all your user roles, will form your Exit Criteria. Meeting those will tell you that your user has an acceptable solution. There’s going to be nuance here – your users will come away from your UAT with a “wish list”, a list of things that they’d like to see changed. Hopefully, these aren’t defects but rather items to improve in the future. Your exit criteria are there to help keep expectations in check, and to ensure that you can separate the “must haves” from the “nice to haves”. After all, that’s what Version 2.0 is for!