M4CR0SC0PE: VBA Macro Analysis with LLM

Overview

M4CR0SC0PE is my graduation project at Jordan University of Science and Technology (JUST), Faculty of Computer and Information Technology — a fully static analysis pipeline for detecting and analyzing malicious VBA macros in Microsoft Office documents.

The core idea: combine deep static code analysis with Large Language Model semantic reasoning to produce explainable malware reports — without executing a single line of untrusted code at any stage.

GitHub: r3dzyoud/M4CR0SC0PE · Project Wiki

---

Problem Statement

Microsoft Office documents with embedded VBA macros remain one of the most prevalent initial access vectors in modern cyberattacks. Attackers rely on the fact that Office files are widely trusted and frequently exchanged — especially in business environments. Embedded macros can download malware, run encrypted commands, or establish persistence without raising immediate suspicion.

Existing analysis approaches each have a fundamental drawback:

Traditional static analysis uses signature matching and keyword rules. It’s safe (no execution needed) but fails completely against obfuscated macros. A macro that uses Chr(), Base64 encoding, string concatenation, or dynamic payload construction looks like noise to a rule-based system.

Dynamic analysis (sandboxing) provides behavioral insight but introduces execution risk, high resource overhead, and is vulnerable to anti-VM evasion techniques — modern malware can detect sandboxed environments and stay dormant.

LLMs on raw code provide semantic understanding but hallucinate without structured input. Feeding raw obfuscated VBA to an LLM produces unreliable output — the model invents function names, misreads variable semantics, and confabulates MITRE ATT&CK techniques.

The gap: a system that combines the safety of static analysis with the semantic depth of LLMs, while controlling and measuring hallucination.

---

System Overview

M4CR0SC0PE analyzes Office files containing VBA macros (.docm, .xlsm, .pptm, .dotm, .xlam, .doc, .xls, .vba) through an eight-stage pipeline. No file is executed. No Office installation is required. No sandbox environment is needed.

The 8-Stage Pipeline

Office Document
      │
      ▼
┌─────────────────────────────────────────────┐
│  Stage 1 · Validation & Hashing             │
│  SHA-256 · MD5 · magic-byte file type check │
├─────────────────────────────────────────────┤
│  Stage 2 · Static Macro Extraction          │
│  oletools · OLE · OOXML · autoexec detection│
├─────────────────────────────────────────────┤
│  Stage 3 · Multi-Pass Deobfuscation         │
│  Chr() · Base64 · Hex · Dridex · concat     │
├─────────────────────────────────────────────┤
│  Stage 4 · Deep Static Analysis             │
│  Behavioral + structural + contextual       │
├─────────────────────────────────────────────┤
│  Stage 5 · Evidence Package Construction    │
│  Structured JSON · 3 epistemic tiers        │
├─────────────────────────────────────────────┤
│  Stage 6 · LLM Semantic Reasoning           │
│  Verdict · IOCs · MITRE ATT&CK proposals    │
├─────────────────────────────────────────────┤
│  Stage 7 · MITRE ATT&CK Verification        │
│  Evidence grounding · hallucination scoring │
├─────────────────────────────────────────────┤
│  Stage 8 · Reverse Engineering Layer        │
│  Call graph · execution flow reconstruction │
└─────────────────────────────────────────────┘
      │
      ▼
  Structured Analysis Report

---

Key Design Decisions

Stage 3 — Multi-Pass Deobfuscation

Before any analysis begins, the system cleans the macro code. Common obfuscation patterns resolved:

• Chr() / Asc() — character code substitution (Chr(83) & Chr(104) & Chr(101) & Chr(108) → "Shell")
• Base64 decode — embedded payloads encoded as strings
• Hex encoding — \x53\x68\x65\x6C\x6C → Shell
• String concatenation — fragments assembled across multiple assignments
• Dridex-style encoding — custom XOR/rotation schemes

The deobfuscator runs multiple passes because obfuscation is often layered — the output of one decode pass becomes the input to the next.

Stage 5 — The Evidence Package (Core Innovation)

Instead of passing raw or deobfuscated VBA code to the LLM, Stage 5 constructs a structured JSON evidence package before any LLM call is made. The package is categorized into three epistemic tiers:

Tier	Description	Examples
Confirmed	Statically verifiable facts, zero ambiguity	Shell call detected, CreateObject("WScript.Shell"), file write to %TEMP%
Inferred	Strong patterns consistent with malicious behavior	Obfuscated string that decodes to a URL, autoexec trigger present, RegWrite to Run key
Contextual	Suspicious but potentially benign in isolation	Process enumeration, registry read, HTTP connection

The evidence package also captures:

• Function and procedure names — with their call relationships
• Suspicious API calls — CreateObject, Shell, Run, RegWrite, WScript, XMLHTTP
• String patterns — downloaders, loaders, encoded payloads
• Embedded PowerShell — treated as supplementary evidence, not analyzed separately

This structured approach reduces LLM hallucination because the model reasons about extracted facts rather than trying to parse meaningless obfuscated strings.

Stage 7 — MITRE ATT&CK Verification (Novel Contribution)

The LLM proposes ATT&CK techniques in Stage 6. Stage 7 cross-references every proposal against the evidence package. A technique is accepted only if it can be traced back to at least one confirmed or inferred indicator. Ungrounded proposals are rejected and flagged.

This produces a measurable hallucination rate per model — a metric that doesn’t exist in previous LLM-based malware analysis tools.

---

Implementation

Stack: Python 3.10 · Flask · oletools · SQLite · Azure OpenAI (GPT-4.1) · AgentRouter (Claude Haiku)

Database Schema (7 Tables)

The system persists all analysis results in outputs/analyzer.db:

Table	Contents
submissions	File metadata, hash, submission timestamp
analysis_results	Per-model verdict, confidence, MITRE proposals
evidence_packages	Structured JSON evidence per submission
iocs	Extracted indicators of compromise
mitre_mappings	Verified + rejected technique proposals
deobfuscation_log	Pass-by-pass deobfuscation trace
re_layer_output	Call graph, execution flow reconstruction

Project Structure

macro_analyzer/
├── stages/
│   ├── validation.py       # Stage 1 — hashing + file type check
│   ├── extraction.py       # Stage 2 — VBA extraction via oletools
│   ├── deobfuscation.py    # Stage 3 — multi-pass deobfuscation
│   ├── analysis.py         # Stage 4 — deep static analysis
│   ├── evidence_builder.py # Stage 5 — evidence package construction
│   ├── llm_reasoning.py    # Stage 6 — LLM inference
│   ├── mitre_verifier.py   # Stage 7 — MITRE verification + hallucination scoring
│   └── re_layer.py         # Stage 8 — call graph + execution flow
├── controller.py           # Pipeline orchestration
├── database.py             # SQLite persistence
└── data_models.py          # Shared Pydantic models
dashboard/
└── app.py                  # Flask API + async job runner
config/
└── vba_mitre_reference.json

---

Dashboard

The system ships with a Flask web interface at http://localhost:5000 with four screens:

Dashboard — total analyzed files, malicious/benign counts, per-model performance cards (accuracy, confidence, hallucination rate), top MITRE techniques, IOC distribution, verdict distribution, submission history

Analyze — upload an Office file and watch all 8 stages execute with real-time status and intermediate output

Report — verdict, confidence score, behavioral summary, MITRE ATT&CK table with evidence links per technique, IOC list, mitigation recommendations, PDF export

Assistant — GPT-4.1 powered Q&A scoped to the project codebase and current report context

---

Evaluation

Dataset: IEEE DataPort MSOffice VBA Macros Classified Dataset — 300 files, stratified 80/20 train/test split, ground-truth labels provided.

Models evaluated: GPT-4.1, GPT-4o, Claude Haiku.

Classification Performance

Model	Accuracy	Precision	Recall	F1-Score	TP	TN	FP	FN
GPT-4.1	75.6%	0.733	1.000	0.846	200	26	73	0
GPT-4o	66.4%	0.664	1.000	0.798	198	0	100	0
Claude Haiku	75.5%	0.710	1.000	0.831	147	38	60	0

All three models achieved perfect Recall (1.000) — FN = 0 across all models. No malicious file was missed.

This is the primary security guarantee. The tradeoff is false positives — legitimate files flagged as malicious. GPT-4.1 achieves the best balance (73 FPs, 26 correct benign classifications), while GPT-4o flags every single benign file as malicious (100 FPs, 0 TNs).

MITRE ATT&CK Hallucination Rate

Model	Proposed	Verified	Rejected	Hallucination Rate
GPT-4.1	759	654	105	13.8%
GPT-4o	421	160	261	62.0%
Claude Haiku	1,475	795	680	46.1%

GPT-4.1’s 13.8% hallucination rate vs GPT-4o’s 62.0% and Claude Haiku’s 46.1% validates the evidence-based prompting approach. When the model reasons about structured, pre-analyzed indicators rather than raw code, it proposes fewer and better-grounded techniques.

Claude Haiku generated significantly more technique proposals (1,475 — almost 2× GPT-4.1’s 759), but nearly half were ungrounded. This suggests the model is pattern-matching on the evidence keywords rather than reasoning about them.

Model Selection Recommendation

For production use, GPT-4.1 is the recommended model:

• Best F1-Score (0.846) — best precision-recall balance
• Lowest hallucination rate (13.8%) — highest MITRE mapping reliability
• Zero false negatives — no malicious files missed

Claude Haiku is a viable offline/private alternative for environments where Azure OpenAI is unavailable, accepting the tradeoff of higher hallucination rate.

---

Why Evidence-Based Prompting Works

The central insight of M4CR0SC0PE is that LLM quality scales with input quality. Raw obfuscated VBA is essentially adversarially crafted noise from the model’s perspective — it contains meaningless token sequences that cause the model to confabulate.

By running static extraction and deobfuscation first, then packaging only the meaningful signals into a structured JSON prompt, we give the model a clean, factual substrate to reason about. The MITRE verification stage then acts as a second LLM pass — the model argues for each proposed technique against the evidence, and we reject proposals that can’t be grounded.

This three-layer architecture (extract → reason → verify) is what produces the 13.8% hallucination rate — compared to 46-62% when models are given less structured inputs.

---

Limitations & Future Work

Current limitations:

• Accuracy depends on deobfuscation quality — heavily layered or custom encoding schemes may not fully resolve
• LLM API dependency — offline environments require local model hosting
• No runtime behavioral analysis — by design, but limits detection of environment-aware payloads

Planned extensions:

• Local LLM support (Ollama integration) for airgapped environments
• Extended deobfuscation for additional encoding schemes (VBA stomping, P-Code analysis)
• YARA rule generation from verified MITRE mappings
• Cross-document correlation for campaign attribution

---

Setup

git clone https://github.com/r3dzyoud/M4CR0SC0PE.git
cd M4CR0SC0PE
pip install -r requirements.txt
cp .env.example .env
# Set AZURE_OPENAI_API_KEY and AZURE_OPENAI_BASE_URL in .env
python3 dashboard/app.py

Open http://localhost:5000.

Full technical documentation: GitHub Wiki