This study investigates whether numbers are known by the company they keep. A composite number is one with factors other than $1$ and itself (‘non-trivial’ factors), i.e., not prime. Some numbers are more composite than others. For example, of three proximal numbers on the number line, $10$ and $14$ have only two non-trivial positive factors (one factor pair), while $12$ has four (two pairs). We hypothesized that people mentally represent numbers in factor neighborhoods and that performance on missing-factor problems will be slower for composite products with denser factor neighborhoods. That is, solving $x * 2 = 10$ or $x * 2 = 14$ requires retrieving the sole remaining non-trivial factor ($5$ and $7$, respectively), while solving $x * 2 = 12$ requires inhibiting $3$ and $4$ to retrieve $6$. By contrast, if people do not represent numbers in factor neighborhoods, then performance should be largely driven by the product size, which increases linearly across $10$, $12$, and $14$ (Campbell & Graham, 1985). We tested 50 U.S. undergraduates’ factoring, measuring their time to solve missing-factor problems for composite products ($6$ to $100$). We predicted that response times on missing-factor problems will be sensitive to the number of factors a product has (‘compositeness’), reflecting a neighborhood representation of numbers. After controlling for whether a given factor pair appears in the 12x12 multiplication table, we found a three-way interaction on response time between problem size, compositeness, and the size of the factor ($x$). These findings support our factor neighborhood hypothesis but also suggest that people’s representations of composite numbers are complex, shaped by problem characteristics and math instruction.