With the increasing growth and demand for capacity in national, regional, and even metropolitan optical networks, high bit rate fiber and large bandwidth transmission becomes the essential part of communication. The large bandwidth and high bit rate transmission improves the spectral utilization which further improves the system efficiency and reduces overall cost. Recently, the capacity of optical communication system is rising in order to fulfill the demand of data traffic that is increasing rapidly by adopting cost effective solutions on the basis of new technologies. The goal of present work is an analysis of CATV transmission system using different modulation formats is to choose best modulation format for data transmission. So that large number of channels can transmit through the optical fiber and fulfill the demand of bandwidth. The objective of this thesis is to provide simulative performance analysis for different modulation formats like NRZ rectangular, NRZ raised cosine, RZ soliton, RZ super gaussian, RZ raised cosine and RZ rectangular, and analyze the performance at each node with varying bit rate.