Using cuMemGetAccess and cuMemSetAccess in nvbit tool

[CarloRamponi]

Hello,
I would like to set a virtual address range as read-only, by using the cuMemSetAccess function.
In this example, I am trying to set the page containing the function as read-only, but the call to cuMemSetAccess fails with the error: "invalid argument".
I think I am using the right addresses, in terms of alignment and size. I've verified the correctness of it by dumping the PTEs of the device.

uint64_t func_addr = nvbit_get_func_addr(f);
const char *func_name = nvbit_get_func_name(ctx, f);

CUmemAllocationProp prop = {
    .type = CU_MEM_ALLOCATION_TYPE_PINNED,
    .location = {
        .type = CU_MEM_LOCATION_TYPE_DEVICE,
        .id = cuDevice
    }
};

size_t granularity;
cuMemGetAllocationGranularity(
    &granularity,
    &prop,
    CU_MEM_ALLOC_GRANULARITY_MINIMUM
);

// Ensure size matches granularity requirements for the allocation
const std::vector<Instr *> &instrs = nvbit_get_instrs(ctx, f);
size_t size = instrs.back()->getOffset() + instrs.back()->getSize();
size_t padded_size = size + (granularity - (size % granularity));
CUdeviceptr page_ptr = (CUdeviceptr)func_addr & ~(granularity - 1);

if(verbose) {
    printf("Function name: %s\n", func_name);
    printf("Function address: %p\n", (void *)func_addr);
    printf("Function size: %lu\n", size);
    printf("Function padded size: %lu\n", padded_size);
    printf("Function granularity: %lu\n", granularity);
    printf("Function page_ptr: %p\n", (void *)page_ptr);
}

// Set access flag
CUmemAccessDesc accessDesc = {
     .location = {
         .type = CU_MEM_LOCATION_TYPE_DEVICE,
         .id = cuDevice
     },
     .flags = CU_MEM_ACCESS_FLAGS_PROT_READ
};

CUresult res = cuMemSetAccess(page_ptr, padded_size, &accessDesc, 1);
if(res != CUDA_SUCCESS) {
    const char *err_str;
    cuGetErrorString(res, &err_str);
    printf("Failed to set read-only access for function %s\nError: %s\n", func_name, err_str);
}

I'm not sure whether this problem is about using these functions in NVBIT or if I'm doing something wrong, thus I am not sure this is the right place to seek for help.

[atomicapple0]

I ran into a similar issue previously. I believe that earlier versions of cuda toolkit do not support the read only flag. I forget which revision of cuda 12 was this fixed in.

What version of cuda are you running on? Did you try setting memory allocated from cumemcreate to be read only?

[CarloRamponi]

Hello,
I'm currently using CUDA 12.6 on an NVIDIA RTX A500 Laptop GPU (Ampere).
I tried setting the memory allocated from cuMemCreate to be read-only, which works.
It could be because I don't have access to the physical memory handle in that context. I mean the cuMemSetAccess does not require that, but by looking at the interactions between the runtime and the kernel driver it seems like the cuMemSetAccess operation consists in:

1. Freeing the (virtual) memory
2. Re-allocating the virtual memory address range
3. Mapping the old physical memory to the new virtual address range, by setting the updated access flags.

Step 3. requires the physical memory handle.

Or maybe, more simply, I'm not computing the page boundaries correctly, or some of my assumptions do not hold.