In this study, we collected data from the National Health and Nutrition Examination Survey (NHANES) for the years 2011–2014. Multiple linear regression and logistic regression were used to analyse the association between nonfood pro- or prebiotic use and cognitive function among elderly Americans. To estimate the potential unobserved results, propensity score matching (PSM) was used to analyse the causal effect. Nonfood pro- or prebiotic use was analysed through the Dietary Supplement Use 30-Day Study. Cognitive function was evaluated by the Digit Symbol Substitution Test (DSST), the Animal Fluency Test (AFT), the Consortium to Establish a Registry for Alzheimer’s Disease (CERAD), and a composite Z-score calculated by summing the Z-scores of three tests. Male participants who used nonfood pro- or prebiotics tended to have higher comprehensive cognitive function (sum.z) with a β-coefficient of 0.64 (95% CI: 0.08–1.19). Probiotics or prebiotics may be a protective factor against cognitive impairment in males, with an odds ratio of 0.08 (95% CI: 0.02–0.29). Furthermore, the average treatment effect for the treated (ATT) with nonfood pro- or prebiotics (0.555) on sum.z in males was statistically significant (p < 0.05). Our research revealed that nonfood pre- or probiotic use was an effective method to improve cognitive function in elderly men from the USA.