Languages commonly use physical properties to discuss non-physical states and events in the world. We investigate the degree to which human associations between physical properties and abstract concepts are culturally specific constructs. To do this, we tested three distinct populations—US adults, US children, and adults from an indigenous group in the lowlands of Bolivia, the Tsimane'—on their associations between the physical concept of weight, and a variety of abstract attributes (e.g., importance, emotional state, moral worth). We find a strong relationship between US and Tsimane' adults, but little-to-no relationship between US children and either adult population. This suggests that the concept of weight plays a similar role in everyday thought across cultures, but takes time to develop. We found that these associations could not be recovered from a simple semantic embedding analysis, suggesting that the cross-cultural connections between physical and abstract attributes may be learned through more than language alone.