The IRK is meant for obscuring who is associated with a bluetooth address (to prevent tracking). This means that you won't want to do whitelisting, or filtering based on the bluetooth address, as it will always be wrong.
I think GapGetPeerBdAddr() will return the random Bd (bluetooth device) Address, as the device can't change it's address when it connects to your peripheral, and if it did so, it would become "trackable" to other devices with a return to its old Bd address.
Based on this thread: Whitelisting with resolvable random address
The resolving of the Bd address is handled by the application in BLE 4.1, but handled by the LL in BLE 4.2. Ultimately, all you need to do is:
Encrypt the connection so only one other device is seeing the transmitted data, and verify the authenticity of the connected device as the authorized user for connecting.
Encrypting the connection is part of the connection, and thus you probably already have it working;
As far as authenticating the remote user, any security username/password variation will work. (Setup the username how you want, then store the password on first bonding/setting up the device. Then upon reconnection, verify user enters accurate password before allowing device access; Otherwise, disconnect after some timeout)