If people operated on "if IRS said so they must be right", there would not be the concept of "tax dispute" and a "tax court". It's apparent that when IRS says "you owe me, give me more money", people do not just say "okay take my money", they fight it at court. 

and let me give you an example

Valuation victory for taxpayer – irs loses most of its proposed estate and gift tax deficiencies

Estate of Mitchell v. Comm'r., T.C. Memo 2011-94 (4/28/2011) 

This Tax Court Memo illustrates the benefits of good estate planning and valuations. The net result in this case was that the IRS lost most of its proposed $10 million estate and gift tax deficiencies.

James Mitchell, a widower, died a resident of California leaving substantial assets to his teenage sons in trust. This case involves disputes between the estate and the IRS on the valuation of two real property holdings and two paintings. The real property consisted of a beachfront residence and a ranch, both of which the decedent leased to third parties on a long term basis. The use of the relatively long-term leases for both properties was a method to insure keeping the properties in the family and having them be income producing. The paintings were by well known American western artists.

Six days before he died, the decedent gifted a 5% co-tenancy interest in the two real properties to his sons in trust. With respect to the beachfront residence, the estate took a 32% discount on the 5% gifted interest and 19% discount on the estate's 95% interest. With respect to the ranch, the estate took a 40% discount on the 5% gifted interest and a 35% discount on the estate's 95% interest. The Court reviewed the valuations of the real property and paintings and found in favor of the estate.

With respect to the real property valuations, the estate's appraiser used a standard “income capitalization” method (which estimates the present value of anticipated cash flows and the reversionary interest) while the IRS appraisers proposed a unique, never approved by the Courts, valuation approach called the “leased buyout” method (which takes the real property's fee simple value less the amount the landlord would have to pay to buyout a tenant). The Court accepted the estate's appraisals and pointed out that any property that generates income can be valued using the income capitalization approach. The Court rejected that leased buyout approach stating that it is “speculative at best”.

With respect to the painting valuations, the Court accepted the estate's values and determined that the estate's experts at trial were experienced, qualified and reasonable. Specifically, one of the estate's appraisers was well qualified to opine on American western art while the IRS experts had neither expertise nor extensive background in American western art. In addition, an IRS appraiser included private sales in his analysis together with the usual sales at public auctions. The use of private sales, presented without details of such sales, was rejected by the Court. The Tax Court accepted the estate's valuation of one painting at $1.2 million (valued by the IRS at $2.3 million) and the other painting at $750,000 (valued by the IRS at $2 million).



so look to the above example. IRS said your Estate taxes are wrong and you need to pay us 10 Million dollars. What happened in the end? IRS came up with a speculative way to determine the value of the houses and they used an appraiser who had no experience with art to value the paintings. Estate on the other hand had used standard calculation methods and hired experts knowledgeable in art. Therefore IRS's valuation was totally wrong and that Estate was correct.

So how can anyone go with IRS? It's not like they have a perfect track record of being right or fair in their valuations. What makes anyone so sure about IRS being right especially not knowing how they (or MJ Estate) came to any valuation? If we had calculations in front us and the logic behind them, we could have perhaps concluded who probably had the correct calculation. but without that information saying "IRS has to be correct because they are IRS" seems to be unfounded.