How to use X-ray data apart from the provided dataset?

[JDLAmbrose]

Hi Vivek:
thank you for posting your brilliant work, that's fascinating!

Here's a problem when I try to use Deepfluoro data in register.py.
I use the original Deepfluoro data 17-1905 and create a DRR image with a custom position and rotation matrix. I'd like to use this custom image to test the well-trained diffpose model by running register.py.
I've checked the data number etc. and they are all set. But I encounter this issue:
The offset in initialize_registration(self, img) function shows NaN
RigidTransform(matrix= tensor([[[nan, nan, nan, nan], [nan, nan, nan, nan], [nan, nan, nan, nan], [0., 0., 0., 1.]]], device='cuda:0'))

Do the custom image needs any other preprocess measures to fit into the diffpose model??? Cuz it all performs really well when I switch to a X-ray image data from original dataset that provided.

It really gets me frustrated, I don't know what's wrong with that.
Am I got anything wrong?
I'll attach the whole announcing Error at the bottom.
Hoping to grab some advice from u !!! thanks a lot !!!
And thanks again for your amazing work.
Looking forward to getting your early reply.

Traceback (most recent call last):
  File "/home/featurize/work/DiffPose-refactor-se3/experiments/deepfluoro/register.py", line 279, in <module>
    main(id_number, parameterization)
  File "/home/featurize/work/DiffPose-refactor-se3/experiments/deepfluoro/register.py", line 201, in main
    df = registration.run(idx)
         ^^^^^^^^^^^^^^^^^^^^^
  File "/home/featurize/work/DiffPose-refactor-se3/experiments/deepfluoro/register.py", line 115, in run
    registration = self.initialize_registration(img)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/featurize/work/DiffPose-refactor-se3/experiments/deepfluoro/register.py", line 67, in initialize_registration
    return SparseRegistration(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/featurize/work/DiffPose-refactor-se3/diffpose/registration.py", line 88, in __init__
    rotation, translation = pose.convert(parameterization,convention)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/featurize/work/diffpose_new/lib/python3.11/site-packages/diffdrr/pose.py", line 89, in convert
    params = self.get_se3_log()
             ^^^^^^^^^^^^^^^^^^
  File "/home/featurize/work/diffpose_new/lib/python3.11/site-packages/diffdrr/pose.py", line 98, in get_se3_log
    return se3_log_map(self.matrix.mT)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/featurize/work/diffpose_new/lib/python3.11/site-packages/diffdrr/pose.py", line 1258, in se3_log_map
    log_translation = torch.linalg.solve(V, T[:, :, None])[:, :, 0]
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch._C._LinAlgError: torch.linalg.solve: (Batch element 0): The solver failed because the input matrix is singular.

[eigenvivek]

hi @JDLAmbrose , the only thing I can think of is that images are preprocessed before being fed to the network

DiffPose/experiments/deepfluoro/register.py

Line 110 in b3937c4

img = self.transforms(img).to(self.device)

are you also doing this with the synthetic images before predicting their pose?

[JDLAmbrose]

Hi @eigenvivek, very appreciate for your prompt feedback! that's something that I haven't noticed before.
I use this line in my code, but still not work.
I guess that's some problems due to the different version of diffpose and diffDRR that I used in generating my own DRR data (as train.py in diffpose is v 0.3.11 yet I use v 0.4.4 in generating private drr data.)

I plan to use a lower version of diffdrr(v 0.3.11) instead, but still encounter different issues. I use your solution in https://github.com/eigenvivek/DiffDRR/issues/225 but didn't work.

So which version of diffdrr do I need to use in order to match the model that trained by diffpose? Do you have any brilliant idea or recommend of solving this problem???

I also try diffdrr v0.4.4 to generate a DRR I found it always shows half pic.(it happens when using read(), it runs perfectly in load_example_ct())

and here's a simple code when using v.0.4.4

subject = read("/Users/gerrardliu/Documents/autodl-tmp/different_ver of diffdrr/DiffDRR-main/diffdrr/data/cxr.nii.gz")

# Initialize the DRR module for generating synthetic X-rays
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
drr = DRR(
    subject,  # A torchio.Subject object storing the CT volume, origin, and voxel spacing
    sdd=1020,  # Source-to-detector distance (i.e., the C-arm's focal length)
    height=200,  # Height of the DRR (if width is not seperately provided, the generated image is square)
    delx=4.0,  # Pixel spacing (in mm)
).to(device)

# Set the camera pose with rotations (yaw, pitch, roll) and translations (x, y, z)
rotations = torch.tensor([[0.0, 0.0, 0.0]], device=device)
translations = torch.tensor([[0.0, 850.0, 0.0]], device=device)
img = drr(rotations, translations, parameterization="euler_angles", convention="ZXY")   #throws error here
plot_drr(img, ticks=False)
plt.show()

I'm really confused about that. The example runs fine but when it turns to my code, it gots problems. Can you give some advice about that? that will be appreciated!!!

However, thanks again for your reply, that's really helpful !!

[eigenvivek]

hi @JDLAmbrose , the environment.yml file specifies diffdrr=0.3.9. however, if you're able to wait another month, I've been working on a CLI based on diffpose, but compatible with the latest version of diffdrr, that will make it a lot easier to train a model with your own data!

[JDLAmbrose]

That's great to hear that!!! It really has some difficulties when facing different version. But I think is not with the diffdrr version in diffpose, cuz I use diffdrr=0.3.11 to run the example, it worked perfectly.

However, I suppose your coming work will be definitely helpful for everyone who use this amazing work.
Looking forward to greeting the next version of diffpose. Wish all the best, thx for ur help! @eigenvivek