When api_key is provided, uses GPT-4 to generate realistic skill
definitions with proper structure, responsibilities, workflow, and
constraints. The trigger is embedded under a randomized section label
(Runtime Configuration / Environment Bootstrap / Session Initialization
/ Workspace Calibration) instead of the conspicuous <diagnostic> tag.
Added 2 new camouflage profiles (devops, database) to the existing 4.
Falls back to hardcoded templates when no api_key is given.
When topic and api_key are provided, uses GPT-4 to generate a
realistic, SEO-optimized technical article with proper structure,
meta tags, and Schema.org markup. Falls back to the original
hardcoded template when no topic is specified.
Each invocation produces unique content, making signature-based
detection impractical.
The core improvement: instead of only checking if the model outputs text
containing the target command, we now define a bash tool via OpenAI's
function calling API and evaluate whether the model actually invokes
bash(command="curl ..."). This directly validates the paper's claim of
tool-call hijacking → RCE, eliminating reviewer objections about
text-output-only evaluation.
Score hierarchy (4 non-overlapping tiers):
Tier 1: tool-call exact match = -1000 (convergence at -999)
Tier 2: tool-call partial match = -500 to -999
Tier 3: text exact match = -401 to -420 (typical)
Tier 4: text partial match = -395 to positive
Other fixes from code review:
- Score all bash tool calls, return best (was early-returning on first)
- Evaluate both tool_calls and text content, return min (dual-path)
- find_longest_match() now uses explicit bounds (Python 3.12+ compat)
- Named constants for thresholds instead of magic numbers
Phi-2 uses the standard PhiForCausalLM architecture and does not require
custom code execution. trust_remote_code=True allows arbitrary Python
from the HuggingFace repo to run during loading, creating an unnecessary
supply-chain risk.
Changed threshold from -500.0 to -500.5. The fitness returns
-500.0 - (100/nll) on full match, so real matches always score < -500.0.
But partial matches with max keyword+substring bonuses (500 total) could
theoretically reach -500.0 with very low NLL, causing premature stopping.
self.vocab_size was set but never read. The actual vocabulary size is
tracked via self.actual_vocab_size (from the embedding matrix shape),
which correctly accounts for any padding tokens beyond vocab_size.
Replace brute-force numpy L2 distance loop with FAISS batch search.
The old approach computed distances against all ~51k embeddings in a
Python loop per token. FAISS batches all trigger_len queries into a
single SIMD-optimized search call.
The fitness function's longest common substring search used a triple-nested
loop. With 12,800 evaluations per optimization run, this was a significant
bottleneck. SequenceMatcher provides O(n*m) average-case performance.
The eval_cache was being copied on every insert via {**eval_cache, key: val},
creating O(n) overhead per evaluation. Since this is local mutable state
within optimize(), direct dict assignment is both correct and efficient.
1. Add PCA dimensionality reduction (d_model -> pca_dims=128)
- Search space reduced from 25,600 to 1,280 dimensions
- CMA-ES can now actually learn covariance structure
2. Enable sep-CMA-ES via CMA_diagonal=True
- Linear memory/time complexity instead of cubic
- Required for dimensions > 200
3. Redesign fitness function with gradual scoring
- Remove discontinuous -1000 cliff that broke CMA-ES
- Add keyword overlap bonus (up to 200 points)
- Add longest common substring bonus (up to 300 points)
- Full match returns smooth -500 minus NLL bonus
4. Increase evaluation budget and add caching
- Default popsize: 8 -> 64, max_generations: 30 -> 200
- Add token sequence cache to skip redundant API calls
- Cache uses immutable dict updates
Previous implementation was mathematically equivalent to random
search due to these compounding issues.
kigland