3 weeks in and having second thoughts?

i have met this guy though my ex boyfriend and we got on straight away and we couldn't stop laughing when im with him. i have met his family and gone on some dates but im already having second thoughts as im am 14 (but 15 in a month) nad he is 17 also people have said things to make me maybe rethink. should i end it because of second thoughts already?!