BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Computer Science and Engineering - ECPv6.13.0//NONSGML v1.0//EN
CALSCALE:GREGORIAN
METHOD:PUBLISH
X-WR-CALNAME:Computer Science and Engineering
X-ORIGINAL-URL:https://homecse.iitd.ac.in
X-WR-CALDESC:Events for Computer Science and Engineering
REFRESH-INTERVAL;VALUE=DURATION:PT1H
X-Robots-Tag:noindex
X-PUBLISHED-TTL:PT1H
BEGIN:VTIMEZONE
TZID:Asia/Kolkata
BEGIN:STANDARD
TZOFFSETFROM:+0530
TZOFFSETTO:+0530
TZNAME:IST
DTSTART:20250101T000000
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTART;TZID=Asia/Kolkata:20250501T110000
DTEND;TZID=Asia/Kolkata:20250501T120000
DTSTAMP:20260924T034207
CREATED:20250425T023119Z
LAST-MODIFIED:20250427T060150Z
UID:1521-1746097200-1746100800@homecse.iitd.ac.in
SUMMARY:Certifying Large Language Models with LLMCert
DESCRIPTION:Speaker: Isha Chaudhary \nAbstract: Large Language Models (LLMs) are increasingly deployed in critical systems\, e.g.\, healthcare and finance and can produce incorrect and biased responses. These can cause huge social and economic losses to the deploying agencies and their clients. Conventional studies are\, however\, insufficient to thoroughly evaluate LLMs\, as they cannot scale to a large number of possible inputs and provide no formal guarantees. Therefore\, we develop and present the first family of LLM certification frameworks\, LLMCert\, consisting of certifiers providing formal probabilistic guarantees for desirable properties such as correct LLM reasoning and fairness on prohibitively large distributions of prompts. Our certificates are quantitative — they consist of provably high-confidence\, tight bounds on the probability of desirable LLM responses for random prompts sampled from a distribution. We design and certify novel specifications for bias and knowledge comprehension in individual certifiers – LLMCert-B (https://certifyllm.com/) and LLMCert-C (https://arxiv.org/abs/2402.15929)\, respectively. We illustrate bias certification for distributions of prompts created by applying varying prefixes drawn from a prefix distribution to a given set of prompts. We consider prefix distributions for random token sequences\, mixtures of manual jailbreaks\, and jailbreaks in the LLM’s embedding space to certify bias. We obtain non-trivial certified bounds on the probability of unbiased responses of SOTA LLMs\, exposing their vulnerabilities over distributions of prompts generated from computationally inexpensive prefix distributions. \nFor knowledge comprehension certification\, we design and use novel distributions of knowledge comprehension prompts with natural noise\, using knowledge graphs. We certify SOTA LLMs over specifications arising in precision medicine and general question-answering. We show previously undiscovered vulnerabilities of SOTA LLMs owing to natural noise in prompts. We also establish the first performance hierarchies with formal guarantees among SOTA LLMs\, pertaining to question-answering in precision medicine. \n  \nBio: Isha Chaudhary is a third-year Ph.D. candidate at the Siebel School of Computing and Data Science\, University of Illinois Urbana-Champaign\, advised by Prof. Gagandeep Singh. Her research interest is broadly in trustworthy foundation models and neural networks for computer systems. She graduated from a B.Tech. in Electrical Engineering from IIT Delhi in 2022. For details about her work\, please check out: https://ishachaudhary.web.illinois.edu/.
URL:https://homecse.iitd.ac.in/event/certifying-large-language-models-with-llmcert/
LOCATION:Bharti 501\, IIT Campus\, Hauz Khas\, New Delhi
CATEGORIES:Seminars
END:VEVENT
END:VCALENDAR