High-speed data transmission through secure and dependable means is the direction in which Optical Wireless Communication (OWC) technology rapidly develops. The research develops a simulated OWC system based on QPSK, DPSK, and QAM modulation, which focuses on Q-factor optimization and BER reduction through photodetector evaluation of PIN and APD. The simulation data shows QAM along with a PIN photodiode produces improved performance through elevated Q-factor levels along with reduced BER, thus delivering better results than DPSK and QPSK systems that utilize APD photo detection. The avalanche multiplication property in APD diodes leads to higher gain yet produces excessive noise, which reduces system performance quality. The signal stability is improved significantly because PIN diodes produce lower noise when using QAM. Research confirms that QAM-PIN diodes provide the most effective solution for high-speed OWC operations. The simulation is performed in OptiSystem software with channel attenuation parameters adjusted to represent various seawater conditions.