The complex structure of gas, metals, and dust in the interstellar and circumgalactic medium (ISM and CGM, respectively) in star-forming galaxies can be probed by Ly\(\alpha\) emission and absorption, low-ionization interstellar (LIS) metal absorption, and dust reddening \(E(B-V)\). We present a statistical analysis of the correlations among Ly\(\alpha\) equivalent width (EW\(_{Ly\alpha}\)), LIS equivalent width (EW\(_{LIS}\)), and \(E(B-V)\) in a sample of 159 star-forming galaxies at \(z\sim2.3\). With measurements obtained from individual, deep rest-UV spectra and spectral-energy distribution (SED) modeling, we find that the tightest correlation exists between EW\(_{LIS}\) and \(E(B-V)\), although correlations among all three parameters are statistically significant. These results signal a direct connection between dust and metal-enriched HI gas, and that they are likely co-spatial. By comparing our results with the predictions of different ISM/CGM models, we favor a clumpy ISM/CGM model where dust resides in neutral gas clumps and Ly\(\alpha\) photons escape through the low HI covering fraction/column density intra-clump medium. Finally, we investigate the factors that potentially contribute to the intrinsic scatter in the correlations studied in this work, including metallicity, outflow kinematics, Ly\(\alpha\) production efficiency, and slit loss. Specifically, we find evidence that scatter in the relationship between EW\(_{Ly\alpha}\) and \(E(B-V)\) reflects the variation in metal-to-HI covering fraction ratio as a function of metallicity, and the effects of outflows on the porosity of the ISM/CGM. Future simulations incorporating star-formation feedback and the radiative transfer of Ly\(\alpha\) photons will provide key constraints on the spatial distributions of neutral gas and dust in the ISM/CGM structure.