The analysis and design of relay protocols is a hot issue in 3GPP Long Term Evolution—Advanced. In this paper, we discuss interference cancellation in a multiuser MIMO environment using Amplify-and-Forward (AF), Decode-and-Forward (DF) and De-Modulate-and-Forward (DMF) as relay protocols, and using Thomilson Harashima Precoding and Dirty Paper Coding as precoding techniques, with Zero-Forcing, Minimum Mean Square Error, Successive Interference Cancellation and Ordered Successive Interference Cancellation detection techniques. By using a combination of classical precoding schemes and detection techniques with weighted matrix, we propose a new interference cancellation technique that is capable of cancelling interference. The interference cancellation is managed by AF, DF and DMF relay node protocols and the interference free codeword is transmitted to the selected User Equipment. The proposed algorithm when used with DMF protocol shows best performance, compared to the conventional system or the no-relay system case, it gives best performance. The observation results shows that DMF protocol gives the best results for BER and Throughput performance in a high interference environment.