Which Warcraft III mod inspired League of Legends?

The Warcraft III mod that inspired League of Legends was Defense of the Ancients, commonly shortened to DotA.

Defense of the Ancients was a multiplayer online battle arena custom map created for Warcraft III. The first version was developed by the pseudonymous modder Eul in 2003 using Warcraft III’s World Editor. Later versions, especially DotA Allstars, expanded and popularized the format in which teams of heroes fight through lanes toward an opposing Ancient.

League of Legends was created by Riot Games developers with strong connections to the DotA community. Steve “Guinsoo” Feak, a major DotA Allstars contributor, and Steve “Pendragon” Mescon were involved in Riot’s early development, while designer Steve “Guinsoo” Feak helped carry lessons from the mod into League’s design. Riot co-founder Marc Merrill and Brandon Beck launched the studio around the goal of making a polished, standalone version of the team-based formula.

DotA itself drew inspiration from Aeon of Strife, a StarCraft custom map, so Aeon of Strife is an important ancestor but not the Warcraft III mod named in the question. Castle Fight and Footman Frenzy were separate Warcraft III custom maps.