I’d know if someone had any access to my phone
This is really a bold claim. How or why makes you so sure of that?
If the attacker/app manages to get some application running in the background as root, how would you know that they had access to your phone?
I ran GrapheneOS on a pixel 5 but ultimately went back to stock.
GrapheneOS was considerably slower on my phone. Apps took a bit longer to loader, but the worst was installing APKs, it takes so much longer compared to stock. Some apps (e.g. revolut) took more than 5 minutes to install, it was crazy.