In the fictional comic book or TV world including the character of Superman, kryptonite is an element which is his one weakness.  He is otherwise invincible except when in contact with this one element.  Kryptonite has since become a term referring to something which is someone or something else’s weakness or can seriously weaken them.  Someone could say that a food that they are allergic to is their kryptonite.