After a reboot following an upgrade, my TrueNAS system crashed with the following error:<\/p>\n
scsi host0: Adaptec AIC7XXX EISA\/VLB\/PCI SCSI HBA DRIVER, Rev 7.0\r\n<Adaptec 3960D Ultra160 SCSI adapter>\r\naic7899: Ultra160 Wide Channel A, SCSI id=7, 32\/253 SCBs\r\naic7xxx 0000:05:02.1: enabling device (0000 -> 0003)\r\nDMAR: DRHD: handling fault status reg 3\r\nDMAR: [DMA read NO PASID] Request device [05:00.0] fault addr 0x12174000 [Fault reason 0x02] Present bit in context entry is clear<\/pre>\nThis was followed by a large number of SCB errors in an endless loop.<\/p>\n
<\/p>\n
The problem is fairly straightforward if you know where to look. First, let\u2019s break down the error message:<\/p>\n
– DMAR – DMA remapping
\n– DRHD – DMA Remapping Hardware Unit Definition [1]<\/em><\/p><\/blockquote>\nBecause of the DMAR reference, the error is related to Intel\u2019s IOMMU technology. What does the IOMMU do? When a device accesses memory via DMA, the IOMMU creates a mapping table and keeps track of which device is allowed to access which memory region. Without an IOMMU, a faulty or compromised driver could access memory regions it shouldn\u2019t, potentially causing a segmentation fault or worse.<\/p>\n
The root cause in my case was a PCIe-to-PCI bridge. If you look closely, you can see that there is a bridge device between the actual device and the motherboard:<\/p>\n
lspci -tv\r\n-[0000:00]-+-00.0 Intel Corporation Gemini Lake Host Bridge\r\n...\r\n +-13.0-[01-05]----00.0-[02-05]--+-03.0-[03]--+-00.0 QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA\r\n | | \\-00.1 QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA\r\n | \\-07.0-[04-05]----00.0-[05]--+-02.0 Adaptec AHA-3960D \/ AIC-7899A U160\/m\r\n | \\-02.1 Adaptec AHA-3960D \/ AIC-7899A U160\/m\r\n +-13.1-[06]--\r\n...<\/pre>\nSo what happens?<\/h1>\n
A device attempts to access memory. However, because of the bridge, the request contains the bridge\u2019s device ID instead of the actual device\u2019s ID. You can see this in the original error message: the bridge ID is
05:00.0<\/code>, while the actual PCI devices are05:02.0<\/code> and05:02.1<\/code>.<\/p>\nHow to solve this?<\/h1>\n
Disable the entire Intel IOMMU with a kernel parameter:<\/p>\n
intel_iommu=off<\/p><\/blockquote>\n
How to do this permanently?<\/h1>\n
In truenas you can add a kernel parameter with the following command, so it won’t break the followup upgrades:<\/p>\n
midclt call system.advanced.update '{\"kernel_extra_options\": \"intel_iommu=off\"}'<\/pre>\nLinks:<\/h1>\n
[1]:\u00a0https:\/\/www.kernel.org\/doc\/Documentation\/Intel-IOMMU.txt<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
After a reboot following an upgrade, my TrueNAS system crashed with the following error: scsi host0: Adaptec AIC7XXX EISA\/VLB\/PCI SCSI HBA DRIVER, Rev 7.0 <Adaptec 3960D Ultra160 SCSI adapter> aic7899: Ultra160 Wide Channel A, SCSI id=7, 32\/253 SCBs aic7xxx 0000:05:02.1: … Continue reading