Is downhill downhill or downhill?

Scenario #1: Something has gone wrong early in the day. It is not good. The rest of the day promises to only get worse.

I say, “Damn. Everything is going to be downhill from here.”

___________________________________

Scenario #2:

A difficult day suddenly becomes much better when a colleague solves an especially challenging problem for me.

I say, “Wow! You just made my day! Everything’s going to be downhill from here!”

___________________________________

So I ask you:

Which usage us correct? The positive or the negative connotation?

Please don’t say both.