The emerging field of graph signal processing (GSP) allows one to transpose classical signal processing operations (e.g., filtering) to signals on graphs. The GSP framework is generally built upon the graph Laplacian, which plays a crucial role in studying graph properties and measuring graph signal smoothness. Here, instead, we propose the graph modularity matrix as the centerpiece of GSP to incorporate knowledge about graph community structure when processing signals on the graph but without the need for community detection. We study this approach in several generic settings, such as filtering, optimal sampling and reconstruction, surrogate data generation, and denoising. Feasibility is illustrated by a small-scale example and a transportation network data set as well as one application in human neuroimaging where community-aware GSP reveals relationships between behavior and brain features that are not shown by Laplacian-based GSP. This work demonstrates how concepts from network science can lead to new, meaningful operations on graph signals.