There are a lot of characters from the books that aren't in the show, and the characters who are in both are usually completely different. I'd start from book 1, because there is an overarching plot that builds from book to book. That said, no matter where you start, the first couple chapters of every book serve as a sort of "catch up" and recap everything up to that point. You'll get tired of all the times you have Harry re-explain to you the basics of casting magic.