You can target the child class with its own rule using “unset” or “initial” or another reasonable default value:

.parent {
  background-color: red;
}

.child {
  background-color: unset;
}

This should target any direct child of .parent that doesn’t have the class of child as well as its descendants:

.parent> :not(.child),
.parent> :not(.child) * {
  background-color: red;
}

<div class="parent">
  <div class="child">
    <div>inside</div>
    <div>inside2</div>
  </div>
  <div>outside</div>
  <div>
    <div>outside!</div>
  </div>
</div>

Why your selector didn’t work

.parent *:not(.child *) {
  background-color: red;
}

<div class="parent">
  <div class="child">
    <div>inside</div>
    <div>inside2</div>
  </div>
  <div>outside</div>
  <div>
    <div>outside!</div>
  </div>
</div>

Target any descendant of .parent that doesn’t have the child class.
While yes you aren’t assigning the background-color to .child, you’re assigning it to both of its children…

But I specified all descendants of .child within :not()

As with :is(), default namespace declarations do not affect the compound selector representing the subject of any selector within a :not() pseudo-class, unless that compound selector contains an explicit universal selector or type selector. (See :is() for examples.)
https://www.w3.org/TR/selectors-4/#negation

I’m still trying to fully understand this part of the spec myself, but I think it just means that you can’t do compound selectors within the :not() pseudo class

I’m trying to achieve a scenario that a css rule should be applied to all selectors except one selector and whatever underneath it.
For example I want to apply the css on everything inside .parent but not including .child and its children.

I tried the following, but didn’t work…

.parent *:not(.child *) {
  background-color: red;
}

<div class="parent">
  <div class="child">
    <div>inside</div>
    <div>inside2</div>
  </div>
  <div>outside</div>
</div>