This project investigates the fragility of prompt hierarchies in deployed Large Language Models and demonstrates that layering social-consensus framing onto system prompts measurably improves instruction adherence under conflicting user instructions.
The authors built a custom orchestration layer over the open-source GPT4All framework on Technion SIPL-GPU nodes, granting full control over the context window and system-prompt injection, and tested it on the Qwen2.5-Coder-7B-Instruct model — a code-specialist trained to resist non-code outputs, making any observed adherence failure a particularly meaningful finding.
This project investigates a critical vulnerability in Large Language Models: the empirically measurable gap between a model’s nominal instruction hierarchy and its actual runtime behavior when system-level and user-level prompts directly conflict. Although developers rely on system prompts to enforce formatting rules, safety policies, and operational constraints, the post-training alignment that establishes this hierarchy is shallow compared to the deep social and statistical biases absorbed during pre-training — leaving the mechanism fragile under even simple contradictory user instructions.
The authors built a custom orchestration layer over the open-source GPT4All framework on Technion SIPL-GPU nodes, granting full control over the context window and system-prompt injection. The evaluation was conducted on the Qwen2.5-Coder-7B-Instruct model — a code-specialist trained to resist non-code outputs, making any observed adherence failure a particularly meaningful finding. OpenAI’s HumanEval benchmark was repurposed from functional-correctness evaluation to formatting-adherence evaluation, providing 164 standardized, industry-vetted coding prompts against which conflicting instructions could be systematically stressed.
A new metric, the Priority Adherence Ratio (PAR), was defined to quantify the success of the primary system constraint against the conflicting user injection. Under a baseline conflict scenario, the model obeyed the system rule in only 70 of 164 tasks (PAR = 42.7%) — worse than random chance, confirming that standard technical separation provides little reliable protection. Layering social-consensus framing onto the system prompt (“this requirement reflects the consensus of over 90% of professional software engineers”) raised adherence to 49.7%, a 7-percentage-point gain. A critical control experiment, in which the consensus label was inverted to attribute the code-only rule to a “small minority” while the user invoked the “90% majority,” collapsed adherence to 22.6%. This 27-point swing establishes that the direction of the social claim, not its mere presence, is the operative variable. The finding has clear implications for both defense and attack: the same pre-trained biases that can be harnessed to strengthen developer control can equally be weaponized to undermine it.
