Dust from concrete and mortar would be light grey so a very light layer of the colours from the joints between the bricks and the plaster outside the bricks could be a solution.
It would be a pity to cover those nicely painted paving stones but at least there should be more "dust" in the cracks
Some other thoughts:
1. Bricks broken out from the walls and lying on the street/sidewalk should have more residue from the mortar joints on them. They look a little too clean now.
2. That kind of damage to the building should have created more rubble. An alternative now is to make the remaining rubble look like someone has removed most of it and just a little remains. In that case the left overs should be more regular as if someone has cleared the street and half of the sidewalk.
3. If there had been rubble on the street the joints between the paving stones should be full of dust.
4. The paving stones, street and sidewalk, look wet (washed or by a recent rain) and in that case the heaps of rubble would have less well defined edges. The finer stuff would have washed out over the paving stones and filled in the cracks.
Some inspiration
Civilians clearing rubble. Note how it is shovelled into heaps to clear the road.
A "cleared" street ...
Aleppo, rubble looks more or less the same today ...
Note the size of the heaps. Doing that size is unrealistic in your diorama but you get the general idea. Even if it has been cleared the cleared surfaces should probably be dirtier ...
Another "clean" street ...
Grass growing at the edge of the rubble
Searching for something useful ...
Street which has not been cleared ...
This could be a bit further down from your street corner
"MacFarland standing near automobile in a bombed street where rubble has been piled onto sidewalks in Berlin, Germany in April 1945"
Instead of cleaning the insides of the haouse they now clean the insides of the house off the street
