When LR imports any file, it has to build a series of previews, from thumbnail sized up to 1:1, which are used to display the image in the grid or when you zoom in. Normally, it would do this by processing the raw file and applying Adobe's Camera Raw algorithms.
However, when Embedded & Sidecar is selected in the Import dialog, LR builds its previews by copying the previews which the camera "embedded" in the image (you see these on the camera LCD). If there is only a raw file, LR quickly extracts and copies whatever previews the camera stored in the raw file - up to full size in Nikon's case, but not in other raw files such as Fuji. So LR can show Fuji camera-generated previews in the grid, but when you zoom in 1:1 it is forced to process the raw data and build a 1:1 preview. You can avoid this processing by importing Fuji raw+JPEG pairs, allowing LR to build its previews by extracting and copying full size JPEGs.
The settings I've mentioned will only be affecting newly-imported files, not existing ones. But it sounds like you are importing.
Also, It's only in Library where you can see the exact Fuji-written image. In Develop you are processing the raw data, and applying whatever profile you chose (Camera Settings means Adobe's attempt to match Fuji's rendering). I wonder if you're seeing the effect of lens corrections not being applied until Develop has loaded all the raw data.