Archived study
Taxonomy cluster study
If we let embeddings and HDBSCAN choose how many failure classes there
are across the 183 categories ingested from DAPLab 9 Patterns, MAST, Microsoft AIRT 2025, OWASP Agentic Top 10, Shah 2026 Agentic Faults and Swiss Cheese Model, do we arrive at SAICA's
manual 11, or something different? Two algorithmic methods were tried.
Text clustering over MiniLM sentence embeddings of the
label + description prose landed on
Ktext = 34, more than SAICA's 11, with
ARI 0.03 and NMI 0.27 against SAICA's labels.
Response clustering over the
(category × tool) bipartite reachability matrix landed on
Kbipartite = 5, fewer than SAICA's
11, with ARI 0.38 and NMI 0.54 against
SAICA's labels. That agreement is built in: the matrix and the labels are both made from the same
crosswalk (see the note above).
UMAP — categories grouped into cluster regions
Each shaded polygon is one cluster's convex hull, colored by partition and labeled with the cluster's proposed name. Dots are the 183 source categories at their UMAP-2D coordinates. Light-gray dots are HDBSCAN noise (cluster = -1) — they get no hull. Switch the coloring to see how the same 183 points get carved up differently by text clustering (34 clusters), bipartite clustering (5 clusters), and SAICA's manual partition (11 modes). Click any dot for its full description and fellow cluster members.
Alluvial — how categories flow between partitions
Three stacked columns, each a vertical inventory of clusters at a different K: text (34) → bipartite (5) → SAICA modes (11); the grey bar of each column holds its noise or unmapped categories. Ribbon thickness = number of categories that belong to both the left and right bar; ribbons inherit their source bar's color. Read it left-to-right. Where bipartite blocks line up with SAICA's modes, it is because both columns are derived from the same crosswalk, so that part of the picture is not evidence for the 11 modes. Hover any bar to light up the ribbons that touch it; hover a ribbon to see its size.
Key takeaways
- Text clustering fragments: 34 clusters, 31 of them with 4 members or fewer; 86 of the 183 categories were left as noise, placed in no cluster.
- Tool-based clustering concentrates: 5 clusters of 14 to 83 categories, 9 categories as noise. This follows from how the matrix is built: categories mapped to the same failure modes get the same row, and the 132 categories with no mapping all get an empty one.
- Fan-in, left → middle: each tool-based cluster draws its members from 6 to 17 text clusters: the text clusters are finer than the tool-based ones.
Clusters for the current coloring
Rows correspond to clusters of the currently-selected partition. Noise points (HDBSCAN id = -1) are listed last.
| Cluster | Size | SAICA top match | Coverage | Top 3 members |
|---|
Further reading
- Exploratory note on this study, with the shuffle test and the other findings of the review:
research/TAXONOMY_CLUSTER_STUDY.md - Raw artifacts:
research/taxonomy_clustering/ - Machine-readable input:
/taxonomy_study.json