How to Design a Guest Memory Pilot Study

What should a guest-memory pilot establish?

A guest-memory pilot should establish whether a defined team can capture and use reliable guest knowledge with manageable effort, and whether the resulting service improves against an agreed comparison. Adoption, service delivery, guest response and return bookings require different evidence and observation periods.

This is AFG Holdings’ proposed operator study protocol, version 1.0. It develops the existing buyer evaluation guide into a plan that can be specified before a trial starts. No results are implied by the proposed design.

HM Treasury’s guidance on evaluating AI interventions distinguishes experimental, quasi-experimental and theory-based approaches. It concerns public-sector evaluation, not a hospitality product endorsement. [1] AFG applies those distinctions below to a hotel or cruise workflow.

Write a study brief before reviewing outcomes

Choose one decision to investigate. For example: does a current guest request reach the next shift before the next relevant service interaction? Make timely handover availability the primary operational measure if that is the question. Avoid selecting whichever metric improves most after the trial.

Record the participating property or vessel, departments, roles, languages, eligible guests and service types. State the starting workflow, deployment and training dates, comparison method, observation window, exclusions and data-review responsibilities. Name the operator who owns the expansion decision.

Define thresholds for usefulness and acceptable workload with the operator before starting. Include conditions that require correction or suspension, such as repeated wrong-person attribution or access outside the agreed scope. Thresholds are decisions for the intended setting; this paper does not invent a universal passing score.

Match the comparison to the claim

A feasibility study asks whether the workflow can operate

A small initial deployment can examine training, participation, missing records and handover failures. Report these observations with their context. It may justify a larger study without estimating a causal improvement in loyalty.

A comparison study asks what changes with the workflow

Where appropriate, allocate comparable service teams or operating units to introduction at different times or to intervention and comparison conditions. Consider shared staff and information: if the same employees serve both conditions using the new memory, the distinction may be blurred. Keep normal required service in place.

If random allocation is impractical, document how the comparison group was selected and what differences remain. Similar-looking groups may differ in motivation, guest mix or management attention. Before-and-after results from one enthusiastic team can be informative, but other changes may explain the difference.

A difference-in-differences analysis needs defensible assumptions

The Magenta Book’s analytical annex describes comparing change over time in an intervention group with change in a comparison group, examining pre-intervention trends. [2] For a hospitality application, collect enough comparable periods to assess the assumption, record concurrent changes and use appropriate analytical expertise. Two headline percentages alone cannot establish the counterfactual.

Collect service evidence consistently

Use the measurement dictionary to freeze definitions and denominators. Capture both successful and failed handovers. Link action records and direct guest responses when appropriate, while preserving a separate category for missing response. Use the same recording method in the comparison condition where feasible.

Have a reviewer assess a sample of attribution, relevance and corrections against reference evidence. Where practical, conceal the condition from the reviewer. Record disagreements and the rule used to resolve them. A second reviewer can help reveal whether a subjective “usefulness” rating is applied consistently.

Measure the whole workload: entry, clarification, review, briefing and follow-up. Record training and setup separately from recurring effort. If the operation changes during the trial—staffing, menus, room availability or service policy—include that change in the study log.

Document meaningful software or configuration changes as well. If a change alters the workflow under evaluation, identify the affected period rather than silently pooling results from different conditions.

Set the sample and follow-up to fit the outcome

Sample size depends on the primary outcome, expected baseline, smallest worthwhile change, variation and grouping of observations. Several hundred notes from a few stays are not several hundred independent guests. Have the study design account for repeated observations within guests, teams and properties.

A short pilot may reveal whether staff use the workflow and whether a brief arrives on time. It usually cannot settle long-term repeat booking behaviour. Predefine a later observation window for eligible guests, and distinguish completed bookings from stated intentions and predicted rebooking scores.

Report how many eligible guests have completed the full follow-up period. Do not classify guests whose observation window is still open as non-returners. Differences in opportunity to return, such as trip purpose or travel distance, should be considered in the interpretation.

Publish a report that another operator can assess

1. Setting and ownership. Identify the operator’s environment, dates, study sponsor and who reviewed the findings.
2. Method. State the comparison, eligibility rules, sample sizes, missingness and changes from the original plan.
3. Results. Give raw counts, defined rates and uncertainty appropriate to the design. Separate primary from exploratory measures.
4. Experience and effort. Include guest responses, staff workload, failures and corrective work.
5. Limits. Explain what the setting and design cannot establish, including unresolved alternative explanations.
6. Decision. State whether to expand, revise or stop, and which evidence supports that choice.

Publish aggregated findings and methods with appropriate review of what may be disclosed. Inspectable evidence does not require making individual guest histories public. A credible report allows a reader to understand the result without exposing those records.

AFG’s current evidence review separates the existing residence account from this proposed next stage. An operator trial adds value when it answers a question the current record cannot yet settle.

Sources and evidence notes

[1] HM Treasury. Guidance on the Impact Evaluation of AI Interventions.

Official evaluation guidance; choosing an evaluation approach. Applied here to hospitality by AFG. Source reviewed 25 September 2026.

[2] HM Treasury. Magenta Book Annex A: Analytical methods for use within an evaluation.

Official guidance; section A2.7, difference-in-difference. Methodological reference. Source reviewed 25 September 2026.

Related AFG documents

• Buyer evaluation guide
• AFG operator trial approach
• The economics of loyalty