What the AEC AI benchmarks measure, and what they miss
Artificial Intelligence
12
min read
Over the past few weeks we went through every public benchmark, evaluation and vendor study we could find that claims to measure AI on AEC work. We found 99. They come from universities, vendors, engineering firms and a few practitioners, and they range from exam questions to full quantity takeoffs on real drawing sets.
For each one we asked four questions.
What does it test?
How close is that to real work?
Can someone outside the authors check it?
And what do the newest results say?
For each one we checked what was actually released. Where the code was public, we ran it on a handful of benchmarks, and we ran three small spot tests of our own on public data.
This post is the summary. The full list, with links and results, is in the table at the end.
In short
Almost all of it is new. Nearly nine in ten of these benchmarks appeared in 2025 or 2026.
Text in, number out works. Give a model the geometry in words, the tables and a calculator, and it gets the engineering number right. Professional exam questions are also widely solved.
Drawings and 3D models are where it drops. Counting, measuring and tracing across a real sheet, and editing a BIM model without breaking it, are far from solved for a general-purpose model on its own. A harness helps on drawings, but every strong drawing result we found is vendor-reported, and editing a BIM model stays unsolved even for agents.
The setup around the model matters more than the model. With the right inputs, tools and checks, the same model moves from failing to useful. Yet half of all published tests use the raw model only.
The benchmarks closest to real work are mostly built by vendors, and the vendor’s own system usually leads them. Of the 14 vendor benchmarks, nine work on drawings and five are takeoffs.
Only a handful test real work on real data. Of the 99, nine measure real AEC workflows on project data that someone outside can inspect, or come from a practising AEC firm.
None of it tells you what AI will do for your firm. That depends on your documents, conventions, codes and people, and only a test on your own work answers it.
Executive summary
If you don’t have time for the whole article, this is the part that matters for your firm.
Benchmarks for AI on AEC work are multiplying, and they are a useful indication of what is possible, not of what is ready. Text, codes and calculations are largely handled: give a model clean inputs and tools and it gets the answer right. Drawings and 3D models, where most project information actually lives, are not. General-purpose models still fail at counting, measuring and tracing on real sheets, and no system, with or without a harness, reliably edits a BIM model without breaking it.
How your data is structured and fed to the model moves results more than which model you pick.
The benchmarks closest to real project work are mostly built by vendors, and the vendor’s own system usually leads them. Handoff, Kamai and Nomic built three of the most realistic drawing benchmarks, and each one’s system tops the table it published. That doesn’t make the numbers wrong, but it is the opposite of a neutral leaderboard, where one party sets the test and vendors take it on equal terms, the way Artificial Analysis ranks language models. The closest thing in AEC is Nomic’s AEC-Bench, which other vendors, us included, now run their evaluations, but every vendor score on it is self-reported. Vendor benchmarks also cluster on drawings, takeoffs above all, while real projects run on documents, drawings, models and spreadsheets together, and nothing tests that.
So read published scores as a direction, not a verdict. Whether AI works for you depends on your documents, your drafting conventions, the codes you design to, the country you build in and how your data is kept.
A high score on someone else’s floor plans says little about your MEP sets.
For most firms the question is no longer whether to use AI, but where to begin. Start with reading tasks that scored well on real data and that a person can check quickly: answering questions across drawings and specs, cross-reference and detail checks, takeoff support, compliance pre-checks. Leave editing and generation of signed deliverables until the evidence is there.
Then test it on your own work. A small private evaluation on your own documents, marked by the people who would sign the output, is the only benchmark that answers the question that matters: “can we automate this part of our job, here, now?” If you want to run one, AECFoundry can help.
Someone still signs the drawing, the payment and the compliance verdict, so deciding what to automate, and who is accountable when it goes wrong, stays with the firm.
The firms that move first will set the pace, but in a high-consequence industry the ones that last will be those that scale on high-volume workflows they have validated.
The rest of this article is the evidence behind all of this.
The map: what is measured on real project data
We started with a strict filter. A task makes this chart only if it is a real AEC workflow and the benchmark behind it uses real project data that someone outside the authors can inspect, in full, in part or on request. We made one exception: a benchmark built by a practising AEC firm, because a firm testing on its own work has no product to sell. That brought in Thornton Tomasetti’s structural calculation set. Grey task names mark benchmarks where only part of the data is public.
Most of the 99 fall at one of those two hurdles. Exam questions are real but they are not work. Several of the most interesting workflow benchmarks run on fictional projects or on 3D models the authors built themselves. Some vendor studies publish a score on data nobody else can see. The table at the end shows where each one’s data comes from.
Three things to keep in mind when reading the chart. Each task has its own benchmark and its own metric, so compare red and blue on the same task, never across tasks. “No AEC tooling” still allows generic tools: in the structural calculation set every model has a calculator and Python, so the arithmetic is deterministic and the model only chooses formulas and inputs. And human baselines exist on only two of the 13 tasks, so for the other eleven a high score doesn’t say whether the tool beats the person who does that job today.
What the chart shows is that every task on it is a reading task: answer, extract, count, check. None asks the model to change a drawing or a model. The largest gaps between red and blue sit on the tasks that need the most context, such as submittal review, spec-drawing coordination and IFC data extraction. And the empty space matters as much as the shape. There is no benchmark on real project data for MEP drawings, BIM rule checking, reconciling facts across documents, drawings, models and spreadsheets, scheduling or priced estimates.
The landscape
The full list of all 99, with links, data access and headline results, is in the table below.
Benchmark | Who | Year | What it tests | Data | Access | Headline result |
|---|---|---|---|---|---|---|
Knowledge recall and exams (18) | ||||||
Li et al. / DACI Lab | 2026 | Regulatory-clause MCQ, true/false, cloze and short answers; numerical and non-numerical subsets (text) | Chinese GB, CJJ and JGJ standards; clause-to-task generation + review | Planned, not released | Paper reports numerical and cross-clause weaknesses; no independently rerun result | |
Tongji + ECADI/Arcplus | 2025–26 | 23 tasks on a 5-level hierarchy: recall, code-table reading, calculation, compliance checking, tender review, report… | Chinese codes, ECADI documents, licensure exams; two-round engineer… | Full, gated | DeepSeek-R1 best: calculation 78.8, code tables 63.5, compliance 52.5 | |
Tongji | 2024 | Licensure-exam MCQ, 5 cognitive levels (text, zh) | Chinese licensure exams | Not found | Scores fall with cognitive level | |
Guangxi Univ. | 2025 | 4-option MCQ; answer-only vs chain-of-thought; stability over 7 resamples (text, zh) | Registered architect and planner exams, mock exams, university papers | On request | Qwen1.5-14B most accurate, above GPT-4-turbo; CoT adds <3% for 26× the time | |
HIT + CSCEC Digital | 2025 | Single and multi-choice, 8 subjects (text, zh) | National certification exams | Not released | o1 best; weak on maths | |
Tongji | 2025 | Objective + subjective exam items, 13 categories (text, zh) | Qualification exams | Not released | CivilGPT “superior” | |
HK PolyU + WeBank | 2025 | Qualification MCQ and true/false (text, zh) | 24 Chinese qualification banks | Full | Civil: Qwen2.5-7B 63.1%, GPT-4o 53.2% | |
Tsinghua + Zhipu AI + CHN Energy | 2025 | Concepts, application and calculation MCQ (text, zh) | Textbooks, standards, exams; LLM-drafted, ≥3 expert reviewers | Partial | DeepSeek-V3.2 and R1 >0.79; GPT-5 fifth | |
Utah, HKUST, Tongji | 2025 | MCQ, calculation, short answer, cloze over 12 topics (text) | n/s | Not found | Average ~50%; calculation ~13%, codes ~80% | |
Shandong Univ. | 2026 | 9 tasks: geology, design, construction decisions, disaster prediction, code reading (text + image, zh) | Codes, geological reports, papers, disaster records | Not found | Doubao-Seed-1.6 and Qwen3-235B best; MLLMs ~46 average | |
Kent State, LSU | 2025 | Certification MCQ incl. figure and table items (text) | AIC and CMAA study guides | Withheld | Claude 3.7 83%, GPT-4o 82%; figure items ~40% | |
Texas A&M | 2024–26 | ASP/CSP/CHST exam MCQ with prompt variants (text) | Official test banks | Withheld | GPT-4o 84.6%, GPT-3.5 73.8%; prompt choice moves 8–13 points | |
FIU, UF | 2025 | MCQ and true/false on drawing literacy, takeoff, estimating (text only) | Expert-written from textbooks | Full | GPT-4.1 75.4 ± 1.1%; Llama 3.3-70B 65.4% | |
Lingnan Univ. | 2026 | Memorisation to application questions; hybrid auto + human scoring (text) | n/s | Not found | Weak on accuracy and completeness; few-shot inconsistent | |
Clemson et al. | 2026 | Open-ended instruction–response, 7 intent types × 3 tiers (text) | Synthetic: GPT-5-mini generated, GPT-5.1 verified | Full | GPT-5 4.48/5; Claude Sonnet 4 4.31; civil is GPT-5’s weakest field | |
Licensure-exam studies: FE/PE structural, FE civil, FE… | Clemson, U Alberta, U Iowa, B. Potter… | 2023–25 | Pass-the-exam MCQ, text-only (figures transcribed) | NCEES, ARE and textbook banks | Withheld | GPT-4: FE 70.9%, PE 46.2% (2023); ARE 70.4%; FE-civil text items 87% vs image items 42% |
SALUS Safety | 2025 | 4-option MCQ built from safety documents; RAG system vs closed-book LLMs (text) | Proprietary SDS, standards, regulations; expert-validated | Partial | SALUS IQ 94.0%; Claude 4.1 Opus 80.9%; GPT-5 80.3% | |
UpCodes | 2025 | Code-interpretation Q&A (text) | Vendor-written | On request | Copilot 93%; ChatGPT-5 46%; Gemini 32% | |
Codes and compliance processing (8) | ||||||
Lumen-IT | 2026 | Text QA, formula retrieval, bill-of-quantities matching, out-of-scope refusal | Vendor test bank; domain-expert ground truth | Not found | ||
U Alberta | 2025 | Retriever + LoRA QA over the National Building Code of Canada; VLM QA on code-table images | LLM-extracted from NBCC pages | On request | Elasticsearch best retriever; Qwen2.5-VL-3B +105% relative after LoRA on tables | |
U Auckland | 2023–24 | Translate code clauses into rule logic; entity and relation F1 (text) | Expert translations of real NZ regulations | Not released | GPT-3.5 71.1% F1 vs fine-tuned T5 68.4% | |
EU ACCORD consortium | 2024 | Entity and relation extraction from regulations, the step before rule generation (text) | 33 real regulation PDFs: 23 English Building Regulations documents… | Full | None (annotation corpus) | |
NUS, HIT, NovaCITYNETS | 2026 | Regulatory clause to Python code skeleton; tree-edit distance | Real PUB and SCDF rule-interpretation project | Not released | Beats few-shot Claude 4.5 and GPT-5.2 by 24–39% distance reduction | |
Singapore (IMDA) | 2026 | Turn Codes-of-Practice rules into executable checks on BIM models; the same models with and without the ARCHER harness | Singapore Codes of Practice + BIM test models | Not checked | GPT-5.5 0.739 unaided → 0.976 with the harness; DeepSeek-v4-flash 0.498 → 0.954; GPT-5.4 Mini 0.421 → 0.913… | |
Fuchs, Hellin, Borrmann, TUM | 2026 | LLM agents write reusable compliance-check functions that run on IFC models | n/s | Not checked | GLM-4.7 test F1 0.40 overall, 0.21 on the most complex rules; validation 0.54 (Jul 2026) | |
TUM MSc thesis | 2026 | MLLMs check regulatory provisions against rendered BIM scenes | Scenes from public IFC-Bench models | Not released | n/s | |
Reading drawings and documents (17) | ||||||
ContractorOS | 2026 | Drawing QA; raw API model vs model in third-party agent application | Real drawings; estimator-written key | Results only publicly verified | Best raw 82.1%; best agent 93.3%; headline uses 134 answerable items, not all 169 | |
AECFoundry | 2026 | Object counting on floor plans; QA on real sheets: OCR, spatial, comparative, counting (image) | CubiCasa5K plans + real drawings | Full | Counting: Gemini 3.5 Flash 54% exact match; QA: Claude Fable 5 0.901, Qwen 3.7 Plus 0.818 | |
LSU, UIUC | 2026 | MCQ VQA at three reasoning depths incl. quantity takeoff and compliance (image) | Real issued-for-construction structural sets from 6 campus projects… | Partial | Gemini-3-pro 77.2; takeoff subset best 41.7 vs professionals 96.6 | |
Zhejiang Sci-Tech, U Macau | 2026 | VLM understanding of floor-plan CAD drawings (paywalled; unverified) | n/s | Not found | n/s | |
Tsinghua Architecture | 2026 | 17 subtasks: distances, routes, plan–section correspondence, drawing to photo, zoning (MCQ, image) | Internet drawings and photos; architecture-trained annotators | Full | Gemini-3-Pro 77.9; GPT-5.4 53.8; Claude Opus 4.6 53.3 | |
SJTU, SWJTU + SenseTime, ModelBest | 2026 | Short-answer reasoning over research-paper figures (image) | Journal and arXiv figures; LLM-authored items, 3-architect approval | Not released | GPT-5.5 51.7; Claude Opus 4.8 51.1 | |
Tsinghua SIGS, UCL | 2026 | Identity and geometry across plan, section, elevation: judge, select, locate (rendered PDFs) | Published drawing sets + open BIM | Not released | GPT-5.5 82.4% categorical; line-endpoint PCK@.05 22% | |
Harvard GSD, Northeastern | 2026 | Match the same building across plan, photo, elevation (4-choice, image) | Zillow, Realtor, ArchDaily, Dezeen scrapes | Partial | Gemini 3.1 Pro 83.9%; Claude Fable 5 80.2% | |
Kamai | 2026 | Detect doors, windows, fixtures, walls and areas on one sheet; F1 at IoU 0.5 and pixel F1 (PDF) | Real issued-for-construction sheets, redacted; CVAT, one annotator… | Partial | Kamai object F1 0.929 vs best LLM 0.062 (Gemini 3.7 Flash); walls 0.935 vs 0.268 | |
Tongji, Arcplus, Shanghai AI Lab | 2025 | Panoptic symbol spotting, 27 classes (vector SVG + raster) | Completed drawings from design institutes; layer-based auto-labels… | Partial, gated | DPSS PQ 70.6 vs SymPointV2 60.5 | |
Beike | 2026 | Raster plan to vector JSON: validity, IoU, F1 | Beike listing corpus | Claimed open, no link found | FloorplanVLM exterior-wall IoU 0.925 | |
Andon Labs | 2025, 2026 | Apartment photos to 2D floor plan; v2 agent-only with a persistent notepad (photos) | Real-estate listings; truth from listing plans | Withheld | v2: Gemini 4 Argon 0.544, Claude Opus 5.5 0.512; v1: Claude Opus 4 at random | |
BME Budapest | 2026 | VLM room-area estimation vs pixel counting (image) | Sampled from public ResPlan | Not released | VLM MAPE ~43% vs pixel counting 5.7% | |
Tel Aviv, Cornell | 2024 | Building-type retrieval, feature segmentation, plan generation (raster + text) | Wikimedia Commons | Full | Fine-tuned CLIP R@1 1.5% to 11.8% | |
KAUST | 2025–26 | Distance, visibility, paths, placement on JSON layouts (text only) | 1,800 Gemini-generated + 200 HSSD | Partial | Metric tasks 75–95% on synthetic, 35–60% on HSSD; planning 5–45% | |
Practitioner tests: Flowtivity, Businessware | Consultancies | 2025–26 | Quantities from plan images; cell-level schedule extraction | Real projects, undisclosed | Not released | Flowtivity: grok-4.5 0.7% wall error (office job); Businessware: Gemini 2.5 Pro best, GPT-4o 38% |
TU Eindhoven, McDermott, Chalmers | 2025 | One-shot symbol counting and tag reading, incl. revision-cloud occlusion (image) | Synthetic Digitize-PID sheets | Full | Gemini 2.5 Pro 50.5%; Qwen2.5-VL-72B 40.5%; GPT-4o 30%; models hallucinate symbols on empty regions | |
BIM and IFC (11) | ||||||
Joy, Zarei, Nik-Bakht, Concordia | 2026 | RFI text → intent cues → rule/metadata ranking of Revit elements | Real project: 524 RFIs screened to 316 relevant, then 74 with… | Not found | 8/10 experimental exact links; real-project GPT-4.1 family accuracy 60.27% with or without intent | |
Hellin, Borrmann et al., TUM / Georg… | 2024, 2026 | Natural-language questions over IFC: retrieval, aggregation, geometry, incomplete information (IFC to text) | buildingSMART, KIT, BIMserver, RWTH sample models + 6 student models | Full | ||
TUM + Nemetschek | 2026 | Recall project facts from 4,016 prior chat sessions, then live IFC queries | Derived from IFC-Bench; synthetic sessions, human-reviewed | Not released | Mem0 32.4%; no memory 0%; everything <60% | |
Tsinghua Civil | 2026 | 22 templates: extraction, statistics, calculation, defect review (textualised Revit blocks) | Proprietary Revit models; rule-computed answers | On request | Qwen-BIM 14B G-Eval 0.834 vs base 0.689 | |
HUST, TU Berlin | 2025 | Intent recognition + table QA over IFC-derived tables | Open IFC Model Repository | Not released | GPT-4o table QA 91.95%; XLNet intent 99.1% | |
Zhejiang Sci-Tech | 2025 | Natural language to BIMserver query, 12 types | Software examples + GPT augmentation | Full | 87.2% exact, 88.8% execution | |
Rostock, TU Clausthal | 2026 | Edit IFC from instructions: create, update, delete; direct, spatial, topological (IfcOpenShell agent) | Author-made realistic + synthetic models | Full | Gemini 3 Flash 49.5 final; strict solve ≤3.4% for every model | |
Rostock, TU Clausthal | 2026 | Text-to-IFC create, read, update, delete; 5 reasoning categories | Minimal author-made IFC | Full | Claude Opus 4.6 50.2%; GPT-5.4 31.3% | |
TUM / GNI, Nemetschek-funded | 2025 | GUI agents build 3D models in Vectorworks from text, sketches, CubiCasa plans | Author briefs, sketches, CubiCasa5K | Full | BIMgent 32% end-to-end vs 0% for GPT-4o and Claude 3.7 | |
TUM / GNI | 2024–25 | Text to editable BIM via the Vectorworks API; Solibri rule checks + 4 experts | Author-written prompts | Partial | Pass rates mostly >0.95; Claude-sonnet-4 most consistent | |
Michigan Tech | 2026 | Natural language to Revit modelling scripts; reliability index over 4 attributes | Author-defined | Not found | Reliability 0.60 (component) to 0.49 (system); API misuse 48% of failures | |
Engineering calculation and design (13) | ||||||
Liang, Zhou, Kalaleh, Mei, U Alberta | 2025 | Text racking-system description → tools/OpenSeesPy → capacity and safety decisions | Reorganised British Columbia racking projects | Partial | o4-mini MASEB 94.7; SAAB 96.6 | |
Thornton Tomasetti CORE studio | 2026 | Code-based single-answer calculations with calculator and Python tools; strict JSON (text) | “Contributed by Thornton Tomasetti engineers”; drawings replaced by… | Withheld | 95.6% three-way tie: GPT-5.6 Sol XHigh ($3.58), Claude Opus 4.7, Claude Fable 5; 17 of 68 configs ≥90%… | |
CalcTree | 2026 | All five questions of the January 2026 IStructE Chartered Membership exam: scheme, calculations, drawings (DXF), two… | Real exam paper; 29% of 328 human candidates passed that sitting | Not released | All 10 builds passed; mean 72.8 (prescriptive prompt) vs 77.2 (principled prompt); calculations went from… | |
Theodoros Galanos, Aurecon | 2026 | Numeric engineering calculations in a tool loop; 3% tolerance per field (text) | Templated and synthetic, seeded parameters; formulas often in the… | Partial | Grok 4.3 0.89, Grok 4.20 0.87, Kimi K2.6 0.86, GPT-5.2 0.83; 18 models, no Claude | |
Qin, Liao, Lu et al., Tsinghua | 2026 | Requirements to model to OpenSees/PKPM/YJK analysis to code check to report; interactive and multimodal (DXF, image) | Controlled synthetic scenarios, double-solved | Full | GLM-5.2 97.3% end-to-end; orchestration 82.9% vs 22.0% generic; multimodal 53.2% | |
UIUC et al. + Amazon | 2025 | Design tasks graded by simulation: topology optimisation, beam and truss sizing | Expert-authored | Full | Overall o3 34.4%; structural: Gemini 2.5 Pro 50% | |
Sichuan Univ., NTU | 2026 | Specification to OpenSeesPy building model; executability, period, compliance checks | Author-curated | Partial | Claude Sonnet 4.5 41.6 average; strict 18.0 | |
T. Shinohara | 2026 | 90 MCQ + 24 code tasks + 900 agentic OpenSeesPy workflows | Templated (“calibration profile”) | Not found | Reference-agent calibration only: 89.3 / 70.3 / 60.7% | |
Small structural agent sets: SAWPs, Miami/Lehigh beams and… | U Alberta; U Miami, Lehigh; Luo et al. | 2025–26 | Text problem to analysis code; FE-verified design loop | Author-curated | Full | SAWPs: GPT-4o 100%; Miami agents >90–99%; 44-case: compliance 56.8% to 98.6% |
“superhatk” | 2026 | Beam, frame, truss image to JSON model to WASM solver; partial credit | Textbook figures + one entrance-exam problem | Full | Qwen3.5-35B-A3B 40.2%; no closed models | |
U Naples Federico II | 2026 | 300 base tasks expanded to 800 instances, 3 levels (text) | n/s | Restricted | None published | |
UIUC | 2024 | Course problems, true/false and open (text) | UIUC CEE courses | Full | Claude 3.5 Sonnet 67.1% | |
Southeast Univ. et al. | 2026 | MCQ over regulations, calculations, scenes, point clouds | Authored by the team | Claimed open, no link found | Gemini-3-flash 88.8%; most models 75–85% | |
Project workflows and estimating (12) | ||||||
Ashraf M. Mahdy | 2026 | 20 linked scenarios: programme, bids, contract revisions, logistics, reporting, month-ends, claims and valuation | Polaris DC1 Finnish data-campus design-build | Public, external dependency | A released worked baseline, not a controlled model leaderboard | |
ContractorOS | 2026 | Drawing PDFs → quantity lines, raw API and third-party agent-app lanes | Performing-arts centre and supermarket; author-checked estimator key | Withheld drawings | Best app 53.1% (69/130), Cowork + Opus 5.5 | |
ContractorOS | 2026 | Answer from source documents bundled per question into one PDF | Fictional, internally reconciled record; 25 real structural sheets… | Release not found | Best 97.7%; score covers gradeable responses; 16 unanswerable items separately reported | |
Nomic AI + Galanos (Aurecon) | 2026 | Agent in Docker reviews real PDF sets: 9 families from detail review to submittal review; JSONL findings | Public US and Canadian bid sets with expert-injected defects | Full | Nomic Agent 70.6 / 88.3 / 62.0 (intra-sheet, intra-drawing, intra-project); Nomic tools +32.2 on detail… | |
Handoff AI | 2026 | Full per-trade material takeoff from raw PDF: material, quantity, unit | Permissioned real projects; paid estimators + two-stage expert audit | On request | H1 81.6 (coverage 86.1, P@25 78.8) vs humans 77.6 (65.5, 87.9); Claude Fable 5 61.4; GPT-5.6 58.6; H1 leads 5… | |
Civils.ai | 2026 | Priced BoQ from a 287-page issued-for-construction package (CESMM4) | Live scheme; chartered QS reference measured after the runs | Not released | Civils.ai £3.56M vs QS £3.67M, 45 of 45 items within ±3%; Claude −39%, ChatGPT −43% | |
Togal.AI studies: KU, ASC 2026 | Kansas (Togal-hosted); Sadeh et al.… | 2025–26 | Time and quantities vs On-Screen Takeoff and vs contractor takeoff | Real projects | Not released | KU: 70–76% time saved, “most quantities within 5%” after manual correction; ASC: small but significant area… |
Energent.ai | 2026 | Vendor agent run on 5 third-party benchmarks (DrawingVQA, BlueprintSymVL, CircuitSense, CADBench, BenchCAD) | inherits | Energent outputs not released | DrawingVQA 89.1 vs 77.2 published; CircuitSense 69.6 vs 12.3 | |
McGill, UCSB/NVIDIA | 2025 | Agent revises PDF drawings through 46 tools; instruction ambiguity varied | Summarised from real revision documents across 20 projects | Full | Leaderboard: claude-4.5-sonnet 86.5, o3 86.0, gpt-5 85.1; paper: o1 79.9 | |
NTU Taiwan, Cambridge | 2025 | Extract from Primavera .xer; sequence activities; predict missing tasks | 2 real residential schedules | Not released | GPT-4o-mini 91.6% similarity; Mistral-7B 83.7% extraction | |
U New Haven | 2026 | Closed-book vs RAG QA on public specs | CT DOT, OSHA, FHWA | Partial | 56–64% of closed-book answers unacceptable; RAG adds 1.3–1.5 judge points | |
Rishi Srivastava | 2026 | Agents act as a construction CFO or controller across ERP, pay applications, payroll, lien waivers and bank portals | Synthetic: 35 mock applications for one company | Public split (711 tasks); private split for remote scoring | Best open-weight model: 0.67 single attempt, 0.38 when it must succeed five times in a row | |
Generation and creation (8) | ||||||
Salesforce AI Research | 2026 | Renders to Blender construction code with structural-validator feedback | Procedurally generated | Partial | Joint pass (valid and faithful) 7.1%, Claude Opus 4.5; structural pass up to 79.2%, GPT-5 | |
SUTD, A*STAR, NUS | 2023 | Natural-language instructions to floor plan; IoU vs reference | RPLAN + human instructions | Full | T2D 54.3 micro-IoU; image generators ≤15.7 | |
RPLAN bubble-diagram protocol: HouseDiffusion, LLM-RLVR 2026 | SFU; Mila | 2020–26 | Room graph to vector plan; realism, FID, graph compatibility | RPLAN, Asian residential | By request form | HouseDiffusion compatibility 1.5–2.5; LLM-RLVR 0.01–0.15; few-shot GPT-4o and o3 produced no valid plan JSON |
TU Delft, SKKU | 2024 | Access graph + structure to building plan; architect user study | Swiss Dwellings, real | Full | Best MIoU 42.4; MHD compatibility 76.2 | |
Mila | 2024 | JSON plans under numeric constraints; self- and prompt-consistency | RPLAN, ProcTHOR | Full | Numeric consistency 0.9–1.0 but overlapping rooms in 12–53% of outputs | |
Tsinghua, Guangzhou Univ. | 2025 | Architect-rated functionality and flow metric, used as an RLHF reward | RPLAN, Tell2Design, model outputs | Not released | FP-LLaMa + RLHF beats HouseGAN++, HouseDiffusion and Tell2Design in architect evaluation | |
Tsinghua | 2025 | Sketch, diagram or render to editable 3D form with a window-to-wall ratio | Synthetic | Not released | n/s (PDF blocked) | |
Create Streets | 2025 | LLMs choose between paired designs, compared with public polls | UK projects | Not released | LLMs pick the “human-centric” design 90–100% of the time | |
Site and asset imagery (6) | ||||||
Wang et al., U Florida | 2026 | Remote-sensing imagery → 13 quantitative metrics; numerical and MCQ answers | Mapbox imagery aligned with OpenStreetMap-derived labels | Claimed released, link not found | Best footprint-ratio tolerance accuracy 12.7%; tested models are 7–8B, not current closed frontier | |
HKUST Guangzhou | 2026 | Hazard identification from site photos: MCQ + free-form hazard descriptions | 3,000+ expert-verified images from 100K+ industrial records | Full | Gemini-3-Flash 59.4; Qwen3-VL-Plus 59.1; Claude 4.5 Sonnet 54.9; GPT-4o 48.8 | |
UBC, Columbia | 2025 | Captioning, safety-rule-violation VQA, visual grounding of site elements | Real site photos | Full, gated | GPT-4o 5-shot best captions; VLMs high recall, low precision on violations; grounding IoU <20% on abstract… | |
U Houston | 2025–26 | Agent answers inspection questions from ~48 views per bridge and cites images; condition ratings | Real professional inspection reports | Full | Authors’ scene-graph agent beats baselines (numbers not extracted) | |
Hilti, Trimble, Oxford | 2026 | SLAM and localisation against floor plans on an active site | Real construction site, 360° camera + IMU, LiDAR truth | Partial | 62 SLAM and 22 localisation teams; localisation less mature | |
Stanford, Oregon State, ETH, NYU | 2026 | Point-cloud registration across time on evolving sites | Real North American sites | Partial | n/s | |
Building operations and energy modelling (6) | ||||||
Big Ladder, U Arizona, NREL | 2026 | Documents to HPXML/EPC schema; KVOR metric | Public reference buildings | Full | Gemini 2.5 beats GPT-5; EPC easier than HPXML | |
CMU, Colorado School of Mines, LBNL | 2025 | Natural language → SPARQL → building knowledge-graph answers | Practitioner questions + LLM paraphrases; curated SPARQL references | Full core artifacts | Paper’s best F1 0.38 | |
Wooyoung Jung, U Arizona | 2026 | Text → executable SPARQL; six query-pattern types | 81 real/reference + 120 synthetic graphs; code-built queries, LLM… | QA release verified | Paper: retrieval improves exact-match F1 to ~56–65% | |
Wooyoung Jung, U Arizona | 2026 | Resolve equipment → fetch sensor series → compute answer / diagnose injected fault | 9 DOE office/climate configurations, normal + faulted; EnergyPlus… | Data verified | ||
Jiang & Chen | 2026 release; citation says 2027 | Detailed or noisy text prompts → EnergyPlus models | Controlled geometries/HVAC combinations; noisy prompts generated with… | Partial benchmark release | Model-generation artifacts; no independently verified benchmark score here | |
Liu et al., BUCEA + industry | 2026 | State understanding, counterfactual interventions, hourly setpoint control against EnergyPlus 25.1 | DOE medium-office prototype; 5 Chinese climates; simulated… | Release not verified | State accuracy 24.3–59.7%; counterfactual 11.3–56.1%; control composite 0.630–0.736 |
Before 2024 there was almost nothing. The jump follows vision-language models becoming good enough to try on drawings, and agent frameworks making tool-use benchmarks cheap to build.
Who builds them matters as much as when. Universities and public bodies produced 74 of the 99, mostly exam, dataset and model benchmarks. Vendors produced 14, all in 2025 or 2026, and nine of those are about drawings. Practising firms produced two, and independent researchers the rest. The split by discipline is just as uneven.
A quarter of all benchmarks are architectural, mostly floor plans. Structural, civil and MEP have about ten each.
The most useful way we found to sort the 99 is by what goes in and what comes out. Exam questions, drawing Q&A, IFC queries and calculations all sit on the same two axes: what the model is given, and what it has to produce.
Each cell counts benchmarks by what the model is given and what it has to produce. A finished deliverable is something a firm would hand over: on drawings, quantity takeoffs and a priced bill of quantities; on a mixed project record, construction-management and construction-finance work such as programmes, valuations and pay applications, both on fictional projects.
Two patterns stand out. The mass of the field sits in the top-left corner: text in, answer out, and drawings in, answer out. And the two areas that look most like the real job are almost empty. Only five benchmarks give a model a mixed project record (documents, drawings, models and spreadsheets together), and only one of those asks it to check something. Sixteen ask a model to create or edit a drawing or a 3D model, and models mostly fail them.
Layer by layer
Below is the short version. Each layer expands into its own section with the evidence; in this draft they follow in shaded boxes.
Knowledge recall is largely done. Text exam questions score in the high 80s, and the remaining misses are about figures and badly written answer keys.
Codes and regulations are tied to one jurisdiction each, and results jump inside a checking harness. Nothing transfers between codes.
Reading drawings is where raw models struggle most: finding, counting and measuring across a full sheet. Specialist vision and vendor systems do much better.
BIM and IFC: reading works when the model writes and runs code against the file. Editing breaks the relationships that make a BIM model a BIM model.
Engineering calculation is solved when the problem is text and the tools are given. Turning a drawing into an analysable model is not.
Project workflows are the closest to real deliverables, and the gap between generic agents and systems with structured document knowledge is largest here.
Generation produces plans and structures that look valid and often aren’t buildable.
Site imagery produces plausible but wrong answers.
Building operations is early, and depends on how the building’s data is structured.
Knowledge recall: largely done
Codes and regulations: code-dependent and harness-driven
Reading drawings: raw models struggle, vision systems and vendors do better
BIM and IFC: reading works with code, writing breaks the model
Engineering calculation: text plus tools is solved, drawings are not
Project workflows: submittal review is the floor
Generation and creation: valid-looking, not buildable
Site and asset imagery: plausible but wrong
Building operations and energy: early
Can you trust the numbers?
What an outsider can do with each benchmark, from re-scoring every published claim to taking the author’s word.
More than half of the 99 publish a score and nothing an outsider can check. Only two release their data, their scorer and every model’s output, so that anyone can re-score every claim: BIM-Edit and our own AECV-Bench. Most of the results the industry quotes rest on private data.
Openness and realism rarely come together. Of the 22 benchmarks anyone can rerun, only five test a real AEC workflow on real project data, and four of the nine benchmarks on our map are only partly open or not open at all. BIM-Edit, the most open of all, runs on BIM models its authors built themselves.
Vendor benchmarks need one more caution. Even when a vendor publishes its data and scorer, so that you can run other models, or your own, on the same test, you usually can’t run the vendor’s system. Its own score has to be taken on trust. We welcome vendors publishing benchmarks, and we publish ours, but a vendor’s claims about its own product should be read carefully.
Two more things limit trust. Human baselines are rare; where they exist, models beat novices and average participants and fall short of professionals. And most test sets are small and run once, so a single question can move a score by several points and many leaderboard orderings are within noise.
What the results say
Text versus drawings
Each row is one benchmark and the same models: the best score on questions that can be answered from text, against the best score on questions that need the drawing itself.
The pattern repeats across groups who don’t know each other. When the question can be answered from text, models do well. When the answer has to be found, counted, traced or measured on a drawing, scores fall by roughly 35 to 60 points.
The harness effect
By harness we mean everything around the model: how the drawing gets in (as pixels, extracted text or vector geometry), the tools it can call, retrieval, a workflow split into checked steps, and a final verification. Wherever a benchmark tests the same model with and without one, the harness wins, often by a wide margin. The red-to-blue gap on the map is that effect.
Yet half of all benchmarks test the raw model only, including twelve of the seventeen drawing benchmarks and our own AECV-Bench. So most published “AI can’t read drawings” results really say that a raw vision model, given one look at a page, can’t read drawings. That is worth knowing, and it is not the same claim as “AI systems are not capable at that task at the moment”.
Only one system is reported above a human baseline on a real deliverable: Handoff’s takeoff model (81.6* against 77.6 for paid estimators). The result is vendor-run on data available only on request, and an item counts as correct if it is within 25% of the ground truth. Everywhere else, the deliverable layer is far from parity.
The gap
These are the limits of what exists today, and each has a reason.
No benchmark combines the four things that would make a result directly useful to a firm: real project documents, a deliverable as the output, a human baseline scored the same way, and public data and scorer.
Each exists somewhere. None exist together.
Nobody tests reconciliation across formats. AEC runs on documents, drawings, 3D models and spreadsheets at once, and much of the job is keeping them consistent: the door in the schedule matches the door on the plan, the spec matches the submittal, the model matches the drawing after the last revision. Every benchmark we found takes one slice. None tests cross-document consistency, how a change propagates through revisions, or whether a system recovers from a wrong assumption over a project’s life. The test isn’t whether AI gets the answer right once, it’s whether professionals can trust it throughout the project.
Reading is measured; changing is not. Everything on the map is a reading task. Editing and generation, where the model changes what someone will sign, are tested mostly on 3D models and drawings the authors made themselves, and AI models mostly fail them.
Local context is missing. There is no open MEP drawing corpus, no public project-level check against Eurocodes or other non-US codes on a real model, and nothing on RFIs, change orders, claims, scheduling or priced estimates on real jobs. Codes, drafting conventions and drawing standards differ by country and by firm, so a benchmark built in one place rarely transfers to another.
Human baselines are mostly missing. Only a handful of the 99 compare models with people doing the same task under the same scoring, so most scores can’t tell you whether the tool beats the person who does the job today.
None of this is an oversight. A benchmark that reconciles documents, drawings, models and spreadsheets needs real project data in every format, and that is exactly the data firms can least share in public. Such a benchmark will be less open than one built on exam questions, so it is most credible when it comes from an AEC firm, or from someone with nothing to sell. A human baseline is expensive in its own way: you pay experts, not students, to do the work under the same scoring, and it takes their time. Both are real investments, which is why nobody has made them yet.
What a useful benchmark would need
Real, redistributable project documents with a stated chain of rights; public bid and permit sets are the obvious source.
Tasks that need documents, drawings, models and spreadsheets together, including consistency across revisions.
Outputs that are the deliverable or one step below it, scored deterministically, with two simple controls: an agent that does nothing, to show what an empty answer scores, and a run with the drawings removed, to show how much the model actually needed to look at them.
Editing and generation tasks with ground truth taken from the file, plus expert review of whether the edit was the right one.
A human baseline from paid practitioners under the same scorer, commissioned by someone who doesn’t sell the product.
Human approval built into the scoring where a person is accountable. CFAgentBench already does this: it fails an agent that executes a payment without approval, even when the payment is correct.
Every model’s outputs published, repeated runs, and a held-out set split by project.
Tracks for different countries and codes, and open MEP and structural sheets.
What AECFoundry has to do with it
What we see in the benchmarks matches what we build. Text is taken; perception of drawings, models and project data, and connecting them, is not.
We maintain AECV-Bench as an open, vendor-neutral benchmark on real drawings, with every model’s outputs published. Having built a public benchmark, we also help firms build private ones on their own documents, marked by their own people.
Our platform turns drawings, specs and models into structured, connected data and reconciles them, so a model works from what the project actually says. Our result on AEC-Bench, where submittal review is the hardest family and our score sits far above generic agents, is the evidence so far.
If you want to know what AI can do for your firm, the answer is a test on your own documents and data, built for your firm and largely by it. Talk to us about building that.
