Is the reminders page accessible by all team leaders even if they don't have routing rules set up?

Yes, the reminders page is visible to all team leaders. If the team leader does not have routing rules set up, the reminder notifications will be sent to them.