All these books cover a particular genre of character regardless of the media.
Encyclopedia of Super-Heroes is the best of these and also the book with the most emphasis on comic characters. It has a main section of hero listings, a team section, a foreign section, a minor heroes section, a miknor Golden Age section, and a Dial "H" for Hero identity section. A bit dated now (1985) and not without some odd inaccuracies (Rovin incorrectly claims that a process that removed Giant-Man's powers for a time killed him; Magik is referred to as Colossus's brother). Even so a good comphrensive listing of heroes from all media.
Encyclopedia of Super-Villains in the book that covers the second most number of comic book characters, though a lot of movie serial villains are covered as well. The entries we do get are interesting enough, though some villains if they get mentioned at all only get mentioned as follows: "Like the person mainly covered in this entry, the hero also fought so-and-so". Some odd inaccuracies: Cobra's entry makes a reference to the comic Mr. Hyde's entry, but only the movie Hyde is mentioned there. Why some characters get a full entry and others don't is puzzling (why Power Ring of all the Crime Synduicate? Why Pigeon Person of all the Hostess villains?). Also sometimes powered villains are ignored in favour of non-powered villains: Jmaes Bond's foe Jaws, clearly a super-villain is ignored while some of the more borderline masterminds are covered.
Illustrated Encyclopedia of Cartoon Animals also has a fair bit of comic info, with nice histories on characters who won't find covered so thoroughly in other comic book resources (such as the individual Legion of Super-Pets member). Nice use of background info. Close to EoSV in comic book content.
The other two books include a liberal amount of comic book info and much of the other info could also be found on books on movies and TV and the like. Seeing comic book monsters and space heroes covered in Monsters and Adventure Heroes respectively is always welcomed. Nice additions to your collection as long as you don't mind that comics are one of many media covered.