I've encountered a similar situation before, and it turned out that the brake booster was leaking air. You can try stepping on the brake pedal several times when the car is cold - started, and then hold the pedal down while starting the engine. Under normal circumstances, the pedal will sink a little bit. If it doesn't move at all or sinks all the way to the bottom, there's a high chance that the brake booster is broken.