Did you pre carbonate with co2? If so how much?
Next, how long is your beer line?
It's been a while since I've hooked up my nitro setup and I don't have my notes in front of me so I'm going to estimate some of my #'s
You can carbonate with straight co2 to about .8-1.2vols... any more and you will get foam. the whole purpose of the nitro is to knock some of the co2 out of suspension.
Your other option is to carbonate with beer gas. 25%-30%co2/70%-75%nitro. Only the co2 will dissolve into the solution. depending on how could you keep your setup, the beer gas will need to be set at somewhere between 35 and 45psi. Again, I'm estimating since I don't have my notes with me. It also take some experimenting to get it dialed in how you like it.
As for the length of the beer line, since you're forcing it at a much higher pressure, you need a lot more hose... I think mine are at 16ish feet. compare that to my normal beer lines at about 7.5 feet.
Good Luck. Hope this helps.