Been hearing lots of complaints from Whedon lately, seems like since he isn't returning he is pretty much disclosing most of what he wanted done and it really is looking like AoU isn't exactly the film he wanted to make? Could all these controls by Marvel be the reason he isn't returning?
In addition to the other reports, here is what he has to say about Loki being in AoU.
"He’s so important to the mythos, and they’re like, 'We can’t get Tom. We can’t make a deal," Whedon said of his initial struggles to get Hiddleston into Avengers: Age of Ultron. "You can have Idris!' I was like, 'Oh, I love Idris! This is great!' And then I talked to Tom, saying, 'Just so you know, I feel bad not telling you, and I would never pressure you, but I really feel like it would be great if you could do this...' And they’re like, 'But we already have Idris!' And again, I had no problem there. Everybody’s in!" It doesn't sound like the studio were that keen on having to pay Hiddleston to reprise the role, but they gave in to Whedon in the end, and the scene sounds great!
"We even had a little reference to the fact that he’s taken the throne, which was Tom doing his Anthony Hopkins impression when Thor says, 'Oh, what would father say?' Then Tom does his Hopkins impression, and Thor’s like, 'That is uncanny!' It’s sort of like his subconscious is telling him that Loki was imitating his father. But he would never make that connection."