Diversity (biological, cultural and social) is an important aspect of, and contributor to, resilience of social-ecological systems. However, diversity in agriculture is increasingly lost, and also in organic agriculture, where diversity is ethically required, there is increasing concern about low diversity. This exploratory study investigated organic farmers’ perception of diversity and which extrinsic and intrinsic factors influence agricultural diversification and agro-biodiversity. To do so, it employed qualitative interviews with organic farmers in Upper Austria, in which leverage points are identified, that might help to develop strategies for improving diversity on farms and in the food system. It is shown that farmers’ connotations with the term diversity are rich and manifold and encompass different aspects and elements of planned and associated biodiversity as well as social and economic aspects. The analysis of influencing factors on several aspects of biodiversity showed that personal, economic and production-technical factors determine agricultural diversity. It is shown that social diversity on the farm, direct marketing and cooperation are leverage points for increasing agricultural diversity on the farm level and thus should be considered and focused on in attempts to increase biological and agricultural diversity on (organic) farms.