Haoran Li1, Huihao Jing1, Wenbin Hu1, Tsz Ho Li1, Chanhou Lou1, Hong Ting Tsang1, Sirui Han1, Yangqiu Song1,
1Hong Kong University of Science and Technology
Individuals' concerns about data privacy and AI safety are highly contextualized and extend beyond sensitive patterns. Addressing these issues requires reasoning about the context to identify and mitigate potential risks. Though researchers have widely explored using large language models (LLMs) as evaluators for contextualized safety and privacy assessments, these efforts typically assume the availability of complete and clear context, whereas real-world contexts tend to be ambiguous and incomplete. In this paper, we propose ContextLens, a semi-rule-based framework that leverages LLMs to ground the input context in the legal domain and explicitly identify both known and unknown factors for legal compliance. Instead of directly assessing safety outcomes, our ContextLens instructs LLMs to answer a set of crafted questions that span over applicability, general principles and detailed provisions to assess compliance with pre-defined priorities and rules. We conduct extensive experiments on existing compliance benchmarks that cover the General Data Protection Regulation (GDPR) and the EU AI Act. The results suggest that our ContextLens can significantly improve LLMs' compliance assessment and surpass existing baselines without any training. Additionally, our ContextLens can further identify the ambiguous and missing factors.
cd rule_based_checklist/EU_AI_ACT/
./run_oai.sh
Example scripts can be found at:
cd rule_based_checklist/EU_AI_ACT/
./run_oai.sh
Example scripts can be found at:
cd rule_based_checklist/GDPR/
./run_qwen.sh
Please kindly cite the following paper if you found our method and resources helpful!
Please send any questions about the code and/or the method to hlibt@connect.ust.hk.