It's on page 10 of the rulebook. modifiers for a duel are only added at step D : Determine Final Duel Total.
If you look at Step B: Flip Fate card, they say to add your stat, the value of the card and any suits you have to determine your current duel total. While in step D they ask you to add your stat, value of the card, suits and any additional suits or modifiers to get your final duel total