I think you should definitely let them know. If you figured it out, it's possible someone else has too. Let them know you need to speak with them privately and that it's urgent. The conversation might be awkward, but it's an important one.