Feature request: Extend the privatization to improve segmentation

[naoyam]

Here's how the Llama3 RoPE forward fusion is currently segmented (with NVFUSER_ENABLE=resize_scheduler):

There are two resize segments, one with the grey nodes and another with the yellow nodes. The green nodes compose a no-op segment.

One performance problem here is that T1 is used as an input to the yellow segment and T24 is generated as an output after just broadcast and expand, which is then fed into the grey segment. Instead, the grey segment should just directly read T1 and replicate the broadcast and expand ops.

The current status is better than it used to be because the bf16 tensor is used as the segment output and input, which was not the case before the privation of cast ops by #3776. I think we should extend that approach to include other "cheap" ops. In this particular case, extending the forwarding approach may work, but it also has some limitation.