Unfortunately, without having someone to be accountable, you become that person.  In this case, without a Project Manager, you would be accountable for everything that happens on your project.