Ports are facing increasing environmental concerns with their importance to the global economy. Numerous studies indicated sediment quality deterioration in ports; however, the deterioration is not discriminated for each port activity. This study investigated a spectrum of contaminants (metals and organic pollutants) in surface sediments at 20 sampling points in Port Ningbo, China, one of the top five world ports by volume. The spectrum of contaminants (metals and organic pollutants) was quantified following marine sediment quality guidelines of China and USA and surface sediment quality was assessed according to thresholds of the two guidelines. Coupling a categorical matrix of port activities with the matrix of sedimentary contaminants revealed that contaminants were highly associated with the port operations. Ship repair posed a severe chemical risk to sediment. Operations of crude oil and coal loadings were two top activities related to organic pollutants in sediments while port operations of ore and container loadings discharged metals. Among the 20 sampling points, Cu, Zn, Pb, and DDT and its metabolites were the priority contaminants influencing sediment quality. Overall, surface sediments in Port Ningbo had relatively low environmental risks but ship repair is an environmental concern that must be addressed. This study provides a practical approach for port activity-related quality assessment of surface sediments in ports that could be applicable in many world sites.