Can I ask a contractor to remove his gravel from "my" lawn?

In most states and citys... while you (or I) would not like this, the ground you are talking about does not belong to you, makes no difference who takes care of it.You can call the city and ask if it is legal to dump the gravel on the street like that and maybe get something done. Other than that you are out of luck.