A little while back I took a few acting workshops, one of them focused entirely on Improv. A lot of DMing, in my experience is just that; improvising an encounter when your players do exactly what you don’t expect them to do. Anyways, during the workshop we worked on the concept of “Yes, and…”

The idea was that as you role-play a scene from scratch, saying “yes” to a player generated idea opens up new possibilities and allows you to build off of them.

A recent example that comes to mind involves my PCs resuscitating a downed NPC. I had originally intended for the players to stumble across the NPC, have the hapless half-elf mumble some trivial clue about a future encounter, and then die. The party, quite altruistically, refused to let him die, and after 3 Heal checks at 25+ I agreed and ruled that yes, he was stabilized, but he was still unconscious. Not wanting to drag dead weight around, the party left the NPC with the first group of “trustworthy” individuals they could find, which just so happened to be a group of Deathcultists. This opened up the door for me to use the half-elf, now a half-crazed cultist, as a recurring antagonist, much to my PCs’ chagrin (but secret enjoyment).

I started to realize this the first time one of my PCs (who was also a fledgling DM) encouraged the group to forge ahead with little planning simply because he said that the published module wouldn’t contain any encounters that weren’t balanced for their level.

One thing I’ve had a lot of success with is to create an encounter that is intentionally too hard for the PCs. Let them run in and fail, eventually fleeing out bloodied and humbled. I encourage them to rest, and re-think their strategies and plan more, use the group more effectively and work as a team. Then when they go at it again at a later point in the story, I lower the difficulty enough to make it realistic. This does two things. It encourages them to work more as a team, and rewards them for playing more strategically. I know I’ve been successful when I hear the talk after the game and they say things like, “Man, if we had just been more careful the first time”. Little do they know that it was planned for them to fail the first time and now the game seems like a more genuine challenge to everyone, not to be taken for granted.

My other big problem, which has thankfully been overcome, is severely hurting my wife in gear and role playing ability due to wanting to make sure I did not show favoritism. It got so bad, that she was no longer effective, which hurt the party. I have learned to treat her as any other player, and haven’t had any complaints of ignoring or favoring my wife in game. And I have to say, having a spouse who not only supports my gaming habit, but also participates, is a very wonderful experience.

However, I do agree with your main point: the rules as written are a guideline, a starting place. The reason there’s a live DM in that chair is that not every monster or encounter or adventure is going to be perfectly balanced for every party. Things are going to need tweaking, and even well-playtested published adventures are not above this.

My first real experience with 4e was running Keep on the Shadowfell for my party, a party that included two defenders, a leader, and a striker (then later, a second striker). This is not the standard group; there was no controller, and even with only a single leader, there was a lot of healing in the group (paladins make good secondary leaders, and fighters can heal themselves fairly well). Keep on the Shadowfell, while well-balanced from a purely rules perspective, was not designed with this group in mind, and needed to be tweaked.

I discovered this after the first few encounters, which my players completely thumped mercilessly. So, I started tweaking. By the end of the adventure, I’d say that I’d changed at least 50%, if not more, of the encounters in some way or another, and the encounters that were the most memorable were the ones I had changed.

The rules are great. I’ll be the first to say that. I love 4e D&D, it’s easily, hands-down, my favorite version of D&D so far. The rules are not perfect, though. I’ll be the first to say that, as well. They need tweaking from time to time. And I agree with both Mike and Asmor: you can’t let one bad experience discourage you from rolling up your sleeves and tinkering with the works. The tinkering is where all the good stuff is.
Brian Engard´s last [type] ..Here There Be Dragons: Status-Quo Encounters