Since the activity replies are stored in the activity, it makes more sense to remove an activities list so that each ActivityStreamFilter can handle a full activity.
For instance, the RatingActivityStreamFilter must have a complete Activity to remove the rating activities linked to the activity replies.