This paper addresses the problem of performance analysis based on communication modelling of largescale heterogeneous distributed systems with emphases on enterprise grid computing systems. The study of communication layers is important because the overall performance of a distributed system is often critically hinged on the effectiveness of this part. This model considers processor as well as network heterogeneity of target system. The model is validated through comprehensive simulation, which demonstrates that the proposed model exhibits a good degree of accuracy for various system sizes and under different working conditions. The proposed model is then used to investigate the performance analysis of typical systems.