Rate distortion function