The verifier used to review Telegram’s MTProto 2.0 protocol was ProVerif.
ProVerif is an automated tool for analyzing cryptographic protocols. It uses a symbolic approach commonly associated with the Dolev–Yao model, in which cryptographic primitives are treated abstractly while an attacker is given broad powers to intercept, modify and replay messages. This lets researchers test whether formal security properties hold under the model’s assumptions.
In their study, University of Udine researchers used ProVerif to analyze MTProto 2.0’s authentication, normal cloud chats, secret chats and re-keying mechanisms. They reported automated proofs for properties such as authentication, integrity, confidentiality and perfect forward secrecy, while also identifying an unknown-key-share vulnerability in the re-keying protocol.
ProVerif is not the same thing as Coq or Isabelle, which are general proof assistants, nor TLA+, which is mainly used for specifying and checking concurrent systems. Another important qualification is that a symbolic ProVerif result does not establish absolute real-world security. The conclusions depend on the model, its abstractions and assumptions about cryptographic operations and implementation behavior.