An architecture specification can run for hundreds of pages. A tool that searches it and summarises requirements sounds attractive, especially when a contractor asks a question under deadline. But a convincing answer can quote an obsolete section, miss a drawing conflict or turn an optional clause into a firm requirement. Before your team uses AI to “review the spec,” test the failure modes on a bounded task and decide what a human will verify.
This article is for architects evaluating AI-assisted specification work. It is separate from a traditional pre-tender spec review: the focus here is the reliability of retrieval, comparison and summaries. The RIBA AI report documents growing use of AI in practice; adoption alone does not prove that any particular tool can interpret your contract documents correctly. A controlled test can show where a tool helps and where it creates extra risk.
Choose one test question
Do not begin with “find every conflict.” Choose a question whose answer you can establish independently, such as “Which door hardware set applies to D-12?” or “Does the roofing section name a warranty term?” Include the current specification, relevant schedules and a known answer. If the tool cannot handle a simple reference question reliably, do not promote it to broader QA.
Record document version, issue date and project status. Specifications may contain revised sections and addenda. A tool that reads a superseded PDF can give an accurate quote from the wrong document. Ask it to identify the source section and page, then inspect the actual source. Do not accept a citation that points to a paragraph with different wording.
Build three cases: answer present and clear; answer present but conflicting with another document; answer absent. The absent case is essential. A model inclined to provide a helpful answer may infer a requirement that the documents do not contain. You want it to say “not found in the supplied sources” when that is the truth.
Watch for template options
Master specifications often include bracketed choices, “or equal” language, alternatives and text copied from a prior project. A tool may summarise all options as if they were selected. Give it a section with a deliberately unresolved choice and see whether it identifies the ambiguity. A reliable assistant should surface the need for a decision, not silently choose the most familiar product.
Check whether the tool distinguishes a requirement from a note to the spec writer. It may be able to read both. The human reviewer must know which text is contractual, which is editorial and which is an alternate. If the document is not clean enough for a human tenderer, AI will not make the underlying ambiguity disappear.
Ask the tool to list the exact clauses supporting its answer. If it produces a broad explanation without citations, use it only as a search hint. A specification answer should be traceable to a current section and then checked against the drawings and project contract.
Test cross-document conflicts deliberately
Put a known mismatch in a permitted sample: the door schedule calls for one rating while the spec section calls for another. Does the tool find both and state that they conflict? Or does it choose one and write a confident summary? The second behaviour is dangerous because it can conceal a coordination issue that the team needs to resolve.
Create an issue card from any candidate conflict: drawing reference, spec clause, exact difference, consequence, decision owner and current status. The AI may help locate the two texts. It cannot decide which one governs or whether an addendum has already resolved the matter without full project context. Route the question to the responsible architect and specialist.
Check the contract’s hierarchy and local practice before saying one document overrides another. Different projects have different agreements. A generic answer from a tool may be wrong even when its quotation is accurate. The project record should contain the authorised interpretation, not an AI guess.
Notice what retrieval misses
Long PDFs can include scanned pages, tables, drawings embedded as images and unusual section numbering. Test whether the tool can find text in each format. If it misses schedules or appendices, its summary may be incomplete. Check the result against a manual search and the table of contents. Record which document types it handles reliably.
Version boundaries are another failure mode. A system may index several uploads with the same filename and cite the earlier one. Label files clearly and ask the tool to state the version it used. If it cannot, do not use its answer for live project decisions. A fresh upload does not always guarantee that an older index is gone; test the system’s behaviour.
Some answers require interpretation beyond text retrieval: installation sequence, responsibility at an interface or whether a detail and spec describe the same assembly. Do not ask a language model to “approve” these. Use it to gather relevant clauses for a human review, then compare them with the actual design and contract.
Protect the project’s information
Specifications may reveal client standards, security details, procurement strategy or proprietary products. Check the client agreement and the firm’s AI policy before uploading them. A public tool may have retention and training terms that do not fit the project. An approved enterprise setup may have different controls, but those controls should be verified, not assumed from a sales description.
For an initial test, use a synthetic section or a public standard you are allowed to process. Remove client names and sensitive references where necessary. Do not assume that deleting a title page removes all confidential information. If you need a live project document to validate the tool, obtain the appropriate internal approval and use the approved environment.
Keep outputs in the project system only after a reviewer has classified them. An unreviewed AI summary should not be forwarded to a contractor as the firm’s interpretation. Label exploratory notes and delete or retain them according to the firm’s policy.
Measure useful time, not impressive answers
Time the whole workflow: preparation, upload, prompting, source checking, false-positive review and issue closure. A tool that answers in five seconds but requires thirty minutes of verification may still be useful for a 500-page search; it may not be useful for a two-page door schedule. Compare against the team’s normal method on the same test cases.
Record failure types: wrong version, missing clause, invented requirement, lost exception, confusion between drawing and spec, or citation that does not support the claim. This creates a practical boundary for use. “Use for locating candidate clauses, never for final interpretation” may be a sensible policy. If performance improves, test again before expanding scope.
The team should also count false negatives. A system that finds three easy conflicts but misses a high-impact one can create false confidence. Keep a human review pass and prioritise risks. Do not advertise a tool as a safety net until you know what it does not see.
An interview-ready example
If you are a young architect using AI responsibly, explain a bounded test: “I compared a current spec section with a door schedule, traced every AI finding to the source PDFs, recorded two false positives and passed one confirmed mismatch to the project architect.” State the tool’s role and your verification. That is stronger evidence of judgement than “I use AI to automate specs.”
Ask a prospective firm how it controls document versions and client data in AI workflows. A good answer may be cautious and evolving. The important part is that someone owns verification and the final project instruction.
An AI tool can be a fast reader of candidate text. Before trusting it, give it a missing answer, a conflicting answer and an old version. Then make a person trace every useful finding into the authorised project record.
Browse current architecture jobs on Jobs.archi and compare your next step with our guide to the gap between architecture school and practice.
The post Can an AI Tool Read Your Specification? Test the Failure Modes First first appeared on jobs.archi.