US Patent 11,270,105 — Extracting and analyzing information from engineering drawings

Executive summary

Engineering drawings (PCB fabrication prints, architectural plans, wiring diagrams) encode most of their meaning in symbols, connecting lines, and text annotations that vary in orientation, scale, and style from one drawing to the next. Reading them at scale has traditionally required a person. This patent describes a machine-learning pipeline that classifies the symbols (nodes) in a drawing, traces the connections (edges) between them, extracts each one's text and attributes, and checks the result against published standards and domain rules to catch errors a human reviewer would otherwise have to find by hand.

Somesh Jha is one of ten named inventors, credited while at IBM (Toronto). The patent is assigned to International Business Machines Corporation.

The problem with reading drawings automatically

Symbols in a technical drawing rarely appear the same way twice.

The same component can show up rotated, rescaled, or redrawn in a slightly different house style across a set of drawings, which breaks pattern-matching approaches that expect a fixed template. Existing automated extraction tools could locate some of this content, but struggled to stay accurate across those variations. Even when extraction worked, nothing in the pipeline could tell whether the result was actually correct: a wire rated for less current than the annotation implies, a symbol missing a required label, a connection that doesn't match the standard it claims to follow. Catching that kind of error still meant a person reading the whole drawing.

The node pipeline

Classify symbols with a region-based CNN, trained with rotation and scale augmentation so detection isn't tied to one orientation
Extract each symbol's text, color, and other visual attributes
Validate those attributes against stored standards, domain rules, and language models
Remove classified nodes from the image once they're read

The edge pipeline

Apply edge-detection operators and reinforcement learning to trace the connecting lines left after nodes are removed
Extract each connection's text, color, and connectivity information
Validate connections and attributes against the same standards and domain knowledge
Flag, suggest a correction for, or modify attributes that don't check out

The correction component can flag errors, suggest corrections, or modify attributes based on domain knowledge: for example, catching when a wire annotation specifies current exceeding the wire's rated ampacity.

The method, step by step

Six steps, run in sequence on each drawing.

Node classification

An R-CNN identifies and classifies the symbols in the drawing, trained to stay accurate regardless of a symbol's rotation or scale.

Node attribute extraction

Text annotations, colors, and other properties are retrieved for every classified symbol.

Error detection

Extracted attributes are compared against stored standards, domain rules, and language models to flag inconsistencies.

Edge identification

With classified nodes removed from the image, edge-detection operators and reinforcement learning trace the remaining connecting lines.

Edge attribute extraction

Text, color, and connectivity information are retrieved for every identified connection.

Edge validation

The same error-checking process validates connection attributes against domain knowledge.

Key facts

Field	Detail
Patent number	US 11,270,105 B2
Application number	US 16/580,229
Filed	September 24, 2019
Issued	March 8, 2022
Assignee	International Business Machines Corporation
Inventors	10 named, including Somesh Jha