How do I know if I am truly prepared for graduate school in mathematics?

Tl;dr: Having taken lots of math classes, I've managed to forget a lot of information. Also, since I'm at a small university, most of my peers are not interested in pure mathematics graduate school, so I am not sure whether I am truly prepared for graduate school. I also feel that my GPA indicates my ability to work hard as opposed to any real ability. How do I know if I am actually ready for graduate school, as opposed to being a big hardworking fish in a small pond?

---

I am currently in my fourth year of undergraduate school at a small public liberal arts university (I transferred in during my sophomore year from community college). I plan on spending another year here in order to graduate with the degrees I want. I came in as a biology major, taking mathematics courses for fun. I ended up getting As in each math course and was hooked. I am now a math major, with minors in biology and chemistry.

I've taken several classes (including Proof/discrete math, Linear Algebra, Real Analysis, ODE, Advanced ODE, Abstract Algebra, Number Theory, Math Methods Operation Research) and some independent learning studies (fractals, Cantor set, some real analysis, measure theory, etc). I'm currently in Complex Variables, Real Analysis - 2, and Mathematical Modeling. In the future I am planning to take Abstract Algebra - 2, partial DEs, and an independent learning study on primality. So far I have a 4.0 GPA in math (3.92 overall).

I am not required to take many of these courses to graduate, but I have taken them because I enjoy math and to broaden what I know. I have pass failed only 2 classes (mathematical modeling and operations research) because I did not have enough time to devote to get the grade I wanted, due to pledging into a fraternity and being ill. I attained a pass of course in these two classes. Plus, these are more applied courses which I did not care as much as the purer courses.

My ultimate goal is to go to graduate school in number theory, mainly focusing on pure mathematics and primality.

My main problem is that over the years, I have taken more math courses than the average undergraduate would at my university and plan to do more. Although I've maintained an A average in all my math courses, I have managed to forget a lot of information I have learned. This mainly has occurred due to the fact that I've focused so hard to maintain this GPA that I often work very hard to get that A and then the knowledge is never used again for the most part. I am very, very afraid that when I attend graduate school, I will not know as much as I should, that I will be subpar, much far behind other applicants who have more mathematics memorized. Granted I have decent grades, it almost feels as if I don't deserve it since I can not recall a lot of the information I knew so well at one point or the other. I want to pursue a career in mathematics as I love it. I am just worried.

Also, aside from one very gifted student, I do not know of any other students in the small math department with grades like mine. Many are pursuing teaching masters and not as interested in pure math. Many of my classmates believe I am intelligent or gifted of some sort, but I do not feel like it. I feel like I should know everything if I attained an A in every course. The problem this is late in the game and I'm not sure what to do. Having a course load so difficult and challenging for the past few years, with taking organic chemistry/biology lectures and labs, with 2-3 math courses each semester, I barely have time to sit there and recall information from earlier courses to jog my memory.

I also don't believe I have the mind like some select few who just know how to do things or apply things. Sure I can work on something for a few hours and eventually get it but it does not just spark into my head to do certain things.

Overall, I honestly do not believe that my GPA is a good representation of what I know, but more how hard I can work. I know the simple answer is to just take the time and review things, learn it again, and use it so it isn't forgotten; but surely I can not do this for all the courses I've taken.