Current technological developments have led to a large quantity of embedded sensors, connected objects, and their related networks and communication to be present in the transport area involving modern cars, autonomous vehicles (AVs), aircraft, trains, as well as road infrastructures. Various types of signals and connections occurring on the Internet of Vehicles (IoV) are vulnerable to security attacks, which can cause the system to fail with serious consequences on the user's safety. Research on IoV security focuses on securing the communication among nodes. Only a few studies investigate the relationships between security and safety in IoV. Our approach addresses this gap by providing a semantic-based analysis exploring jointly safety and security. We propose an ontology, named SSIoV, - representing both security-safety knowledge - together with axioms and rules. Our aim is to perform reasoning and inferences on security vulnerabilities and their impact on safety risks, on the basis of actual data sets extracted from real scenarios.